Update on GPU crunching, please?

Alez · Post by **Alez** » Mon Oct 07, 2013 12:42 am

@ Janos

07/10/2013 00:31:32 | Einstein@Home | Requesting new tasks for NVIDIA and ATI
07/10/2013 00:31:41 | | Project communication failed: attempting access to reference site
07/10/2013 00:32:09 | | Internet access OK - project servers may be temporarily down.
07/10/2013 00:35:43 | Einstein@Home | Scheduler request failed: Transferred a partial file

Bwaahaahaa....success.

Janos (retired) · Post by **Janos (retired)** » Mon Oct 07, 2013 9:11 am

Superb!

Post by **CycloneRanger** » Mon Oct 07, 2013 1:24 pm

I had already tried revealing hidden files, but didn't realize I needed to restart -- all good now, thanks. Reports to follow.

Alez · Post by **Alez** » Mon Oct 07, 2013 4:04 pm

CycloneRanger wrote:I had already tried revealing hidden files, but didn't realize I needed to restart -- all good now, thanks. Reports to follow.

You shouldn't have needed to restart just shut down windows explorer and then re open, but then again windoze does weird stuff, so what ever works is good

Hope everything else goes smoother.....

Post by **CycloneRanger** » Mon Oct 07, 2013 9:57 pm

Sorry to be such a noob, but, in the patch ...

<app_config>
<app>
<name>solo_collatz</name>
<gpu_versions>
<gpu_usage>1</gpu_usage>
<cpu_usage>1</cpu_usage>
</gpu_versions>
</app>
</app_config>

... if I am running two GPU, should the "1" be a "2"?

And if I have a quad core, how do I employ the unused cores?

Again, sorry for the bother, but so far CC is running exactly as before, with no improvement in time to process WUs.

Alez · Post by **Alez** » Mon Oct 07, 2013 10:59 pm

Boinc treats each GPU individually. You can run different projects on different GPU's at the same time or set both to run Collatz. The optimization works per gpu. If both are set to collatz then it will apply to both.If collatz is only running on 1 gpu then it will only apply to the one running collatz.

to run 2 apps PER GPU you would change the
<gpu_usage>1</gpu_usage> to
<gpu_usage>0.5</gpu_usage>

with collatz this would be a very bad idea as you should already be running a very high gpu load standard.
the term
<cpu_usage>1</cpu_usage> means boinc will retain 1 core for the gpu exclusively. If boinc manager is set to use 100% cpu then it should run 3 cpu apps on the other cores.

This is PER GPU. Boinc treats each gpu individually regardless of what is running on it. So if you are running 2 gpu's on collatz then boinc should reserve 2 cores for the GPU's and have the other 2 cores available for other tasks.
If you changed the line
<cpu_usage>1</cpu_usage> to
<cpu_usage>0.5</cpu_usage> then boinc will reserve 1/2 core PER gpu so you would have 3 cores free to crunch other projects. This setup only affects collatz. This will not be applied to any other projects you crunch.

it's the second part of the optimisation that makes a huge difference.

Post by **CycloneRanger** » Mon Oct 07, 2013 11:09 pm

I am running two GPUs, but am only processing one GPU work unit (using one core), and three CPU work units on the other cores. I don't understand why.

Alez · Post by **Alez** » Mon Oct 07, 2013 11:30 pm

Completely different problem and easy to solve

By default boinc only uses 1 GPU. When you start up Boinc if you look under advanced - event log in the boinc manager I'll bet boinc only reports 1 gpu.

To solve go to c:/program data /boinc in windows explorer.
look for a file called cc_config.xml
if it's there open it with notepad. If it's not then create one with notepad.

the contents of the file should be this ( make sure the syntax is exact )

<cc_config>
<options>
<use_all_gpus>1</use_all_gpus>
</options>
</cc_config>

save this ( note save ,not save as ) the file must be called cc_config.xml but it's not a true .xml file. don't use explorer or double click to open.
After that restart boinc. You should now have 2 gpu's crunching away like eager beavers. They should also both be reported in the boinc event log.

Post by **CycloneRanger** » Mon Oct 07, 2013 11:34 pm

Done and done! So, finally, how come I'm not seeing an increase in completion speed with the optimization? Was that not the point of it all?

Alez · Post by **Alez** » Mon Oct 07, 2013 11:54 pm

This is the part that optimises collatz

c:program data/boinc/projects/boinc.thesonntags.com_collatz
scroll down and find a file called solo_collatz_4.07_windows_x86_64_opencl_ati_100 config file ( THIS IS FOR AMD CARDS )
solo_collatz_4.07_windows_x86_64_cuda50.config ( THIS IS FOR nVIDIA CARDS )

right click on this file and open with notepad

copy the below into it exactly

verbose=0
items_per_kernel=20
kernels_per_reduction=9
threads=8
sleep=1

you need to change the items_per_kernel=14 to something more ( try 16 first) as this massively increases the loading on the gpu. You are running nVidia cards so the file you need to change is called
solo_collatz_4.07_windows_x86_64_cuda50.config

this is for solo collatz. if you are running collatz and mini collatz then you need to do the same for their config files.
collatz_4.07_windows_x86_64_cuda50.config
mini_collatz_4.07_windows_x86_64_cuda50.config

solo collatz is the best as no wingmen, credit straight off, but this works for all. I am also presuming that you are running windows 64 bit. if not then the file names will be the same i think but will not have the _64 in them.

Post by **CycloneRanger** » Tue Oct 08, 2013 12:12 am

I did that early on ... set at 18. Time to complete still exactly the same.

Alez · Post by **Alez** » Tue Oct 08, 2013 12:16 am

did you change the cuda50 configs, not the ATI config as posted as that was for my AMD 7970 card ?
What is the current load on the cards ? may be that 14 is the max that those cards can run. My cards are all gtx660ti's and 7970 which have far more overhead than the 650ti's.

Post by **CycloneRanger** » Tue Oct 08, 2013 4:32 am

I did change the cuda file, yes. Set to 14, they're loaded in the 70% range. I'm running solo only with the 'accept work' option selected.

Alez · Post by **Alez** » Tue Oct 08, 2013 11:34 am

Well that's a good start. We now have the config files written and we have both GPU's running. At 70% we can definitely get more, lets see if we can push up the load on the gpu's.

First note the load, temp, clock freq and time to run a collatz unit.
Temp is the most important. pushing the load will cause the gpu's to get hotter. We need to check that the cards are not downclocking. With nVidia's i don't like to run them much hotter than 80 c.

To control the cards I use afterburner
http://www.guru3d.com/files_details/msi ... nload.html

I also use TechPowerUp GPU-Z to monitor the cards as it works better in a multi card environment
http://www.techpowerup.com/gpuz/

Once you have got a base line edit
solo_collatz_4.07_windows_x86_64_cuda50.config
change items_per_kernel=14 to items_per_kernel=18 and save.
The change should take place when the next Collatz unit starts running. (The values are set at the start of the unit run and can't be altered mid run.)
Once a new unit starts running monitor the gpu. you should see an increase in load and an increase in temp. Monitor the temp and check the core clock is not downclocking.
I believe that the nVidia control panal allows you to select different modes for the cards ( performance etc.) but not 100% sure as trying to run nVidia, ATI and Intel simultaneously with 3 separate programs is hard work, hence afterburner.

Let me know what the values were pre and post change for time, load, temp and freq. You will need to run a few units as the time is a bit variable. Obviously use your own judgement and if you feel the cards are too hot, stop and turn the optimisation back and abort the current running units. this will force the change immediately. Hopefully this will work, if it doesn't at least we will have hopefully some idea of what is happening. If you don't run SLI for games I would simply disable it ( 3d max performance button in nVidia control panel I believe, on =sli, off= no sli ) I have never run it but I know it has no effect in boinc, I also believe it has no ill effect either, but if you don't use it then 1 less variable to consider.
Any hoo, let me know how you get on.

Edit : can you also check the boinc message log on startup and check that there are no red warning notices being displayed.

Post by **CycloneRanger** » Tue Oct 08, 2013 3:30 pm

OK, changed 14 to 18, and so far I've seen a nominal tempt increase (+/- 2degC) and no change in performance -- still running probably mid-30 minutes per cuda task. The card will definitely handle 18 with no downclocking. No red warning notices, everything's crunching as before. I disabled SLI, but that appears to indeed be an independent variable.

Alez · Post by **Alez** » Tue Oct 08, 2013 4:10 pm

Did the load on the GPU go up any ?
The temp going up means it's doing something and I guess the fan speed increased as well. I presume both cards are running similar loads and temps ?
What hz is the core clocks running at ?
is it what they should be running at ?
I have a gtx 650 ( lying in a box just now )which wasn't that great, guess the TI version you have is a different beast

change
items_per_kernel to 20
change
threads=8 to threads=9

that should surely increase the load on the card.

My 660ti's do a unit in 20 odd mins so mid 30's is not bad but we should be able to get an increase. I would like to see the load up to mid 90's and then we can go backwards to a comfortable load that isn't to hot.

Performance

NVIDIA GTX 650 Ti BOOST

768 CUDA Core
1072 MHz Base Clock
1137 MHz Boost Clock
68.6GT/s Texture Fill Rate
Memory
2048 MB, 192 bit GDDR5
6008 MHz (effective)

I presume this is the card. Lots better than a gtx650.
Are they both running at @1100MHz ?
I note these cards are 140 watts each, what power supply do you have ? i would think at least 500 W or the cards wont be happy and be down clocking.

Post by **CycloneRanger** » Tue Oct 08, 2013 4:27 pm

Load is in the 90s, so I think I'm doing as well as I can.

Cards are synced and tracking.
Core clocks at 1175-1200, which is in the expected range.

Looks like we're as done as my EVGAs will let us get with normal parameters.

Thanks for all your patience and assistance!

Alez · Post by **Alez** » Tue Oct 08, 2013 5:10 pm

You are more than welcome.
Probably increased by 20-30% which is what you could expect with your cards and the fact that the app is pretty good anyway unless you are running it on very high powered cards like the 7970. Doesn't sound that much but it will add up over time. Just keep an eye on the temp. You know what to do now if you want to try and go further.
Glad I was able to help you out.

Anything else then post up and either I or someone else will try to help.

Post by **CycloneRanger** » Tue Oct 08, 2013 7:04 pm

i can't help but notice that my credits do not show up on Temujin's BOINC project stats. I am guessing that is because I de-selected collatz and mini-collatz and selected only collatz solo, per advice on one of the sites referenced earlier. Was this a good strategy, or should I select another combo or all three? (I did check 'use other tasks', which produces mini-collatz WUs but not collatz WUs for my CPU, which I find somewhat curious.)

Post by **CycloneRanger** » Tue Oct 08, 2013 7:04 pm

i can't help but notice that my credits do not show up on Temujin's BOINC project stats. I am guessing that is because I de-selected collatz and mini-collatz and selected only collatz solo, per advice on one of the sites referenced earlier. Was this a good strategy, or should I select another combo or all three? (I did check 'use other tasks', which produces mini-collatz WUs but not collatz WUs for my CPU, which I find somewhat curious.)

Post by **CycloneRanger** » Tue Oct 08, 2013 7:05 pm

Sorry for multiple posts - kept getting general error and didn't realize they were going through

Alez · Post by **Alez** » Tue Oct 08, 2013 7:27 pm

Nothing to do with not selecting all projects. Check in your collatz account that the tasks are being credited and not being marked invalid or something. Sometimes it just takes a bit. I have noticed on collatz that you sometimes only get a little or no credit for a few hours and then you get the lot credited.

This is a new project for you ? Sometimes it takes a day or two before boincstats and temujins sites catch up. As long as you are being credited in your Collatz account then all will be well. :)

edit: scratch that, you show on temujins site but are not being credited, on boincstats you do not show at all as a member. Check in your account that you are still a member of TSBT. Everybody below where you were has moved up a place. I think you have accidentally left the team

Post by **CycloneRanger** » Tue Oct 08, 2013 7:35 pm

State: All (201) Ã‚Â· In progress (150) Ã‚Â· Validation pending (4) Ã‚Â· Validation inconclusive (0) Ã‚Â· Valid (45) Ã‚Â· Invalid (0) Ã‚Â· Error (2)
Application: All (201) Ã‚Â· collatz (5) Ã‚Â· mini_collatz (43) Ã‚Â· solo_collatz (153)

They show up everywhere except Temujin's site, where my highlighted name on the collatz tab is red instead of green.

This has gone on for days now; it's not a temporary delay. I'm missing a million credits or so on his site (but not on BAM). The project is not even highlighted in the top table.

Alez · Post by **Alez** » Tue Oct 08, 2013 7:37 pm

You show on temujins site but are not being credited, on boincstats you do not show at all as a member. Check in your account that you are still a member of TSBT. Everybody below where you were has moved up a place. I think you have accidentally left the team Shocked

Post by **CycloneRanger** » Tue Oct 08, 2013 7:40 pm

Well derp! That would help. wouldn't it? Clearly I need more coffee! Thanks, man!

Alez · Post by **Alez** » Tue Oct 08, 2013 7:46 pm

You show up as our newest member now

--- Welcome to the team !!!

always nice to have a new member

Bwaaahaha, sorry man

Janos (retired) · Post by **Janos (retired)** » Wed Oct 09, 2013 9:35 am

Judging by your stats something is working. :)

Post by **CycloneRanger** » Wed Oct 09, 2013 9:35 pm

So, of course, never satisfied with success, I managed to bollix up my TSBT account by changing my e-mail address.

Thank Grid that Mega is working to fix my latest egregious error!

The Scottish Boinc Team

Update on GPU crunching, please?

#16

#17

#18

#19

#20

#21

#22

#23

#24

#25

#26

#27

#28

#29

#30

#31

#32

#33

#34

#35

#36

#37

#38

#39

#40

#41

#42

#43