cmsearch needs 3814.70 MB

[Trotador]

Hi,

My first post, just to note a change in the RAM requirements as of yesterday evening:

RNA World|Message from server: No work sent
RNA World|Message from server: cmsearch needs 3814.70 MB RAM but only 3151.55 MB is available for use.
RNA World|Message from server: cmcalibrate is not available for your type of computer.

I assume it is for the best of the project but did you realise that no PC with 4 MB of RAM will meet this requirement?, either with a 32 or 64 bit OS

That leaves out most of the systems, mine at least, I have got just one machine with over 4 GB RAM but it is dedicated to another project and can not be shared.

Guess i will have to wait till your requirements decrease if it even occur.

regards,

[yoyo]

I think it is better that you do not get this memory eating wu instead you get it and it crashes because to less memory after hours of running.
yoyo

[Trotador]

It 's OK for me.

Maybe you should consider updating the FAQ.

Regards,

[cisf]

can;t the tasks be separated based on memory requirements, and sent to computers that can manage them ?

make categories like, under 1G mem req. , under 2 G ... and so on ....

the crunching would get done a lot faster !

[MonsieurPoot]

can;t the tasks be separated based on memory requirements, and sent to computers that can manage them ?

At the moment this unfortunately cannot be achieved because currently there is no way to determine exactly how much RAM a workunit is going to require, but the people behind the project are busy looking for a solution.
Until one has been found, 4GB is the mininum amount of memory set in order to prevent too many workunits from failing and being sent out and crunched in vain.

[yoyo]

Exactly it is like that currently:
- old not yet finished wus have a ram requirements of 3GB
- new generated wus have a requirement of 1.3GB if the input file is < 50MB,
- else 4GB.
So it is already separated.
But most of the new wus are really big. So they require 4GB.
yoyo

[cisf]

i might be wrong, but i have noticed that the maximum memory requirement is the "virt" amount in the "top" command (on linux).

ex:

Code: Alles auswählen

./cmsearch cmfile in

Code: Alles auswählen

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 9297 cisf      20   0  997m 116m  784 R   99  6.6   0:03.06 cmsearch 

should be very easy to implement a script to extract the info, and send it to the server side tot decide where the WU cand be sent.

if this is correct (can the people who have working WU's test this ? ) it would be a dirty fix until a proper solution can be found.

the down side wold be that each WU has to be run for about 2 sec. on the server.

[MonsieurPoot]

new generated wus have a requirement of 1.3GB if the input file is < 50MB

Below 50MB in uncompressed form, I presume?

- Poot

[yoyo]

new generated wus have a requirement of 1.3GB if the input file is < 50MB

Below 50MB in uncompressed form, I presume?

Yes.

[Michael H.W. Weber]

i might be wrong, but i have noticed that the maximum memory requirement is the "virt" amount in the "top" command (on linux).

ex:

Code: Alles auswählen

./cmsearch cmfile in

Code: Alles auswählen

 PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 9297 cisf      20   0  997m 116m  784 R   99  6.6   0:03.06 cmsearch 

should be very easy to implement a script to extract the info, and send it to the server side tot decide where the WU cand be sent.

if this is correct (can the people who have working WU's test this ? ) it would be a dirty fix until a proper solution can be found.

the down side wold be that each WU has to be run for about 2 sec. on the server.

Maybe you can explain this a bit more in detail? We run each WU on the server to at least perform a run time estimation. The RAM requirement, however, cannot be extracted by such "mini simulations" as it is not linearly progressing. Instead, the RAM requirement for each WU is yet unpredictable and is oscillating during the WU computation.

Michael.

[cisf]

i see that a ram filter is allready in place, as my pc gets low ram WU's if they are available, so ... problem fixed i guess !?

the "VIRT" expresses the max. amout of memory available from an aplication point of view (id does way more than that, and i don't understand it all).

as i figure it, the rna wu's, (step 1:)does some small computation in the begining (the estimates perhaps?), (step2:) and after that starts filing the memory (it cannot generate more than the "VIRT" shows to the aplication as available);(step3:) after crunching on the data in the memory, it does another smaller work(cpu time and memory req.); and again step 2 and 3, and wu is ready.

well the "virt" for the step 2 is what i was saing can be used as an estimate.

for some aplications VIRT is irelevant as it can show way more memory available to the apication than needed, but in rna i see they are accurate enough. Perhaps a large matrix is defined in the code ?!(and than that could be use as an mem req. estimate )

[fractal]

I understand it is difficult to predict how much memory your application will need. But, rather than "run full speed into a brick wall", may I suggest a slightly different approach.

First of all, respect your donors and their hardware. If you say you want 1GB of ram, DON'T USE MORE THAN 1GB OF RAM!!

If you can accept that, may I suggest a technical solution. Firstly, wrap malloc/free, use one of the existing profiling memory managers or use system dependent code to determine memory used.. Pass the memory estimate to the program so you can test against this limit. Should you find you need more, terminate processing so you can increase the estimate for the next donor who gets the work. The donor who got the underestimated unit should get appropriate credit for the work they did. After all, it is not their fault you underestimated the work. Some care on your part may be needed to ensure people do not send back false reports with fudged cpu times.

I do understand your application was not built with bounded memory in mind, but simply exhausting all memory in a machine when your estimate is wrong is a poor programming practice. Detecting and gracefully recovering while rewarding the donor for their good faith work is a much better practice. I know it is very hard to abort a work unit that is 95% done, 55% done or even 2% done, but that view does not adequate balance the needs of the donor with the needs of the science.