Limit number of Python jobs

Message boards : Number crunching : Limit number of Python jobs

To post messages, you must log in.

AuthorMessage
Ryan Munro

Send message
Joined: 19 Nov 15
Posts: 16
Credit: 13,977,143
RAC: 4,829
Message 106388 - Posted: 14 Jun 2022, 9:12:12 UTC

Is it possible to limit the number that run at once? on my machine with 32gb ram it runs 11 at a time maxing out the ram, the issue is though that its not even half using my CPU (5950x) which causes a few cores to run really fast and the heat to jump up causing the fans to spin up.
I don't want to stop doing them but limiting to like 4 at any one time would be great.
ID: 106388 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Profile dcdc

Send message
Joined: 3 Nov 05
Posts: 1831
Credit: 119,536,330
RAC: 6,139
Message 106400 - Posted: 16 Jun 2022, 22:23:42 UTC - in response to Message 106388.  

BOINC is in control of how many tasks are run at a time. You can control it by limiting RAM available to BOINC. I think you can also set the max_concurrent value in one of the files too.
ID: 106400 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Bryn Mawr

Send message
Joined: 26 Dec 18
Posts: 391
Credit: 12,083,032
RAC: 4,831
Message 106402 - Posted: 17 Jun 2022, 0:08:19 UTC - in response to Message 106400.  

BOINC is in control of how many tasks are run at a time. You can control it by limiting RAM available to BOINC. I think you can also set the max_concurrent value in one of the files too.


But that is Rosetta overall, not Python specific.

You can switch Python off but, afaik, you cannot say allow a maximum of 2 Python jobs whilst allowing up to 8 Rosetta jobs.
ID: 106402 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
Sid Celery

Send message
Joined: 11 Feb 08
Posts: 2122
Credit: 41,183,435
RAC: 10,025
Message 106404 - Posted: 18 Jun 2022, 23:44:06 UTC - in response to Message 106388.  

Is it possible to limit the number that run at once? on my machine with 32gb ram it runs 11 at a time maxing out the ram, the issue is though that it's not even half using my CPU (5950x) which causes a few cores to run really fast and the heat to jump up causing the fans to spin up.
I don't want to stop doing them but limiting to like 4 at any one time would be great.

I'm running a 5800X with 32Gb RAM and the maximum number of VBox64 tasks I can run is 11 as well, but not very reliably.
I slightly reduced the amount of RAM allocated to Boinc so that only 10 run at once and a lot more reliably - far from perfect though.

The other more obvious way is to go into Options/Computing Preferences and on the Computing tab set "Use at most xx% of the CPUs" to 12.5% for 4 cores of your 32.
You may want to try 25% first to see if that's acceptable to you and tweak down from there if necessary
ID: 106404 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote
rjs5

Send message
Joined: 22 Nov 10
Posts: 273
Credit: 23,005,357
RAC: 5,836
Message 106413 - Posted: 22 Jun 2022, 20:21:15 UTC - in response to Message 106388.  

I play around with app_config.xml and have been able to do some fine tuning.

Try and change the 2 to what you would like to limit it. It will not control

<app_config>
<app> <name> rosetta_python_projects </name> <max_concurrent> 2 </max_concurrent> </app>
</app_config>

Another app_config.xml line I use to control number of active project jobs is the max concurrent line. It may download more but only execute the specified amount. I was surprised that if a WU "starts", the slot is set up and the memory is allocated in Linux.

<project_max_concurrent> 4 </project_max_concurrent>


Is it possible to limit the number that run at once? on my machine with 32gb ram it runs 11 at a time maxing out the ram, the issue is though that its not even half using my CPU (5950x) which causes a few cores to run really fast and the heat to jump up causing the fans to spin up.
I don't want to stop doing them but limiting to like 4 at any one time would be great.
ID: 106413 · Rating: 0 · rate: Rate + / Rate - Report as offensive    Reply Quote

Message boards : Number crunching : Limit number of Python jobs



©2024 University of Washington
https://www.bakerlab.org