tasks with 8 GB memory requirements

Alles zum Projekt yoyo@home
Everything about the project yoyo@home
Nachricht
Autor
biodoc
Taschenrechner
Taschenrechner
Beiträge: 8
Registriert: 25.08.2018 11:55

tasks with 8 GB memory requirements

#1 Ungelesener Beitrag von biodoc » 25.08.2018 12:07

There are tasks with _np_ in the name with extremely large RAM requirements (~8 GB) such as this one: ecm_uc_1535071917_np_161_2900e6_01_289_P2. My 40 thread Xeon processor with 96 GB RAM has been swamped with these WU so I have to reduce CPU% in boinc to 25% just to get through these WUs and even then all my boinc allocated RAM (95%) and part of my swap is needed just to process 10 of these tasks simulataneously. Link to computer: [url][http://www.rechenkraft.net/yoyo/show_ho ... 437331/url]

Properties of one of the WU:

Application
ecm 705.01
Name
ecm_uc_1535071917_np_161_2900e6_01_289_P2
State
Running
Received
Thu 23 Aug 2018 09:37:55 PM EDT
Report deadline
Tue 28 Aug 2018 09:37:52 PM EDT
Estimated computation size
22,334 GFLOPs
CPU time
00:07:41
CPU time since checkpoint
00:07:41
Elapsed time
00:07:42
Estimated time remaining
00:52:13
Fraction done
6.977%
Virtual memory size
7.43 GB
Working set size
7.36 GB

Directory
slots/3
Process ID
25971
Progress rate
54.360% per hour
Executable
ecmwrapper_705.01_x86_64-pc-linux-gnu

biodoc
Taschenrechner
Taschenrechner
Beiträge: 8
Registriert: 25.08.2018 11:55

Re: tasks with 8 GB memory requirements

#2 Ungelesener Beitrag von biodoc » 25.08.2018 12:55

These ECM tasks are running with -maxmem 10000. Is this 10000 MB/10GB?

Is this a mistake? Should I abort these WUs with _np_ in the task name?

Benutzeravatar
yoyo
Vereinsvorstand
Vereinsvorstand
Beiträge: 8045
Registriert: 17.12.2002 14:09
Wohnort: Berlin
Kontaktdaten:

Re: tasks with 8 GB memory requirements

#3 Ungelesener Beitrag von yoyo » 25.08.2018 14:03

This is not a mistake. Some wus, named with P2 at the end are RAM monsters.
HILF mit im Rechenkraft-WiKi, dies gibts zu tun.
Wiki - FAQ - Verein - Chat

Bild Bild

crashtech
Fingerzähler
Fingerzähler
Beiträge: 1
Registriert: 25.08.2018 18:10

Re: tasks with 8 GB memory requirements

#4 Ungelesener Beitrag von crashtech » 25.08.2018 19:54

So there is no concern that contributors that have clients of average resources can't run these WUs, because such machines will crash if this is attempted? Some may decide not to run yoyo at all because of this.
My opinion is that these memory hog WUs should be in their own category, since they perform so much differently than other ecm WUs.

Benutzeravatar
yoyo
Vereinsvorstand
Vereinsvorstand
Beiträge: 8045
Registriert: 17.12.2002 14:09
Wohnort: Berlin
Kontaktdaten:

Re: tasks with 8 GB memory requirements

#5 Ungelesener Beitrag von yoyo » 25.08.2018 21:36

If you don't have the required RAM free, BOINC will not download such workunits. And if they are downloaded they are not started if the RAM is not available.
HILF mit im Rechenkraft-WiKi, dies gibts zu tun.
Wiki - FAQ - Verein - Chat

Bild Bild

biodoc
Taschenrechner
Taschenrechner
Beiträge: 8
Registriert: 25.08.2018 11:55

Re: tasks with 8 GB memory requirements

#6 Ungelesener Beitrag von biodoc » 25.08.2018 23:06

Apparently, I had a worse case scenario for my machine. I was working on ecm tasks exclusively and the RAM monster wus started downloading and they
had the shortest deadline so eventually all my cores were loaded up with 100% monster wus. These tasks grabbed all available RAM and swap file space as defined in boinc manager options. On a 40 thread machine with 96 GB RAM, I reduced CPU% to 25 (10 threads) and the tasks still grabbed all my RAM and swap file space. To avoid thrashing your HDD 24/7, I would set boinc swap file usage to 1% (lowest setting possible). When these tasks start, they use a small amount of RAM but as they process they grab more and more RAM (up to 10 GB). Thrashing my HDD 24/7 is not an option for me.

Why is it scientifically necessary to allow a single task to use 10 GB RAM?

Benutzeravatar
yoyo
Vereinsvorstand
Vereinsvorstand
Beiträge: 8045
Registriert: 17.12.2002 14:09
Wohnort: Berlin
Kontaktdaten:

Re: tasks with 8 GB memory requirements

#7 Ungelesener Beitrag von yoyo » 26.08.2018 06:24

As said, BOINC should handle it. It determines RAM usage and free RAM and starts/suspends workunits depending on it.
Which hostID is it?
HILF mit im Rechenkraft-WiKi, dies gibts zu tun.
Wiki - FAQ - Verein - Chat

Bild Bild

biodoc
Taschenrechner
Taschenrechner
Beiträge: 8
Registriert: 25.08.2018 11:55

Re: tasks with 8 GB memory requirements

#8 Ungelesener Beitrag von biodoc » 26.08.2018 10:25

The host ID is 437331. On that computer, I aborted all ecm tasks but could not get work on another subproject so I moved that machine on to another project.

biodoc
Taschenrechner
Taschenrechner
Beiträge: 8
Registriert: 25.08.2018 11:55

Re: tasks with 8 GB memory requirements

#9 Ungelesener Beitrag von biodoc » 26.08.2018 11:15

Ok, I was able to get ecm tasks just now on host ID 437331. No monster tasks in play and none in que yet.

biodoc
Taschenrechner
Taschenrechner
Beiträge: 8
Registriert: 25.08.2018 11:55

Re: tasks with 8 GB memory requirements

#10 Ungelesener Beitrag von biodoc » 26.08.2018 14:47

I now see tasks in my que with names ending in P2_0. Are these RAM Monster tasks?

It'll take a couple of hours or so before they start loading into my processors.

My boinc manager settings are 100% CPU (40 threads), 90% RAM and 25% swap file. OS is linux mint 19.

Benutzeravatar
yoyo
Vereinsvorstand
Vereinsvorstand
Beiträge: 8045
Registriert: 17.12.2002 14:09
Wohnort: Berlin
Kontaktdaten:

Re: tasks with 8 GB memory requirements

#11 Ungelesener Beitrag von yoyo » 26.08.2018 16:39

Those P2 are RAM monsters. BUT the do not require all 10 GB or RAM. The -maxmem is just to limit the maximum at all. I have some on my systems which uses 2 GB.
HILF mit im Rechenkraft-WiKi, dies gibts zu tun.
Wiki - FAQ - Verein - Chat

Bild Bild

biodoc
Taschenrechner
Taschenrechner
Beiträge: 8
Registriert: 25.08.2018 11:55

Re: tasks with 8 GB memory requirements

#12 Ungelesener Beitrag von biodoc » 26.08.2018 18:18

I've got 6 monsters out of 40 tasks running now and 36GB out of 96 GB being used. No issues at this ratio of monsters/standard ecm tasks.

Antworten

Zurück zu „Number crunching“