error - exit code 195 (0xc3) on CMSearch

Everything about the project RNA World
Nachricht
Autor
Benutzeravatar
rilian
PDA-Benutzer
PDA-Benutzer
Beiträge: 47
Registriert: 08.02.2010 15:38

error - exit code 195 (0xc3) on CMSearch

#1 Ungelesener Beitrag von rilian » 08.03.2010 17:21

Some WUs are randomly failing after 1-5 hours of computation on win2003 server 32-bit machine

for example

http://www.rnaworld.de/rnaworld/result. ... id=1292206
http://www.rnaworld.de/rnaworld/result. ... id=1290930
http://www.rnaworld.de/rnaworld/result. ... id=1290895
http://www.rnaworld.de/rnaworld/result. ... id=1290886

<core_client_version>6.10.32</core_client_version>
<![CDATA[
<message>
- exit code 195 (0xc3)
</message>
<stderr_txt>
RNA World wrapper v0.04
wrapper: no checkpoint file found
wrapper: running unzip (-o graphics.zip)
wrapper: no checkpoint file found
wrapper: running unzip (-o in.zip)
wrapper: no checkpoint file found
wrapper: running unzip_cpufeat (cmsearch.zip)
wrapper: no checkpoint file found
wrapper: running cmsearch (--forecast 1 cmfile in)
forecast.txt found.
wrapper: no checkpoint file found
wrapper: running cmsearch (-o out cmfile in)
forecast.txt found.
wrapper: windows. no checkpoint image
wrapper: windows. no checkpoint image
wrapper: windows. no checkpoint image
wrapper: windows. no checkpoint image
app exit status: 0x3
17:32:18 (5972): called boinc_finish

</stderr_txt>
]]>


Could not find anything about this status message, except smth (maybe not really related) on calm chaos yoyo@home forum
Last week we extended the number range and run some test workunits with up to 85000 digits. Unfortunately in the first try the work generator was not able to handle such big numbers and greated wrong work units. We fixed it and run all the test work units again and found already some factors during the test. On some clients the big numbers crashed, mostly Windows. If you had ecm work units with exit code 195 (0xc3) and in stderr out you have app exit status: 0xc00000fd you can help to find a solution for it.
I crunch for Ukraine

Benutzeravatar
Michael H.W. Weber
Vereinsvorstand
Vereinsvorstand
Beiträge: 23174
Registriert: 07.01.2002 01:00
Wohnort: Marpurk

Re: error - exit code 195 (0xc3) on CMSearch

#2 Ungelesener Beitrag von Michael H.W. Weber » 08.03.2010 17:41

Most certainly a "lack of memory" error. You use a Quad-Core machine with only two GB of RAM? :o That means, this machine will start 4 instances of RNA World having a maximum of 512 kB of RAM per core available for computation. On top of that the WUs you were processing are amongst the largest available. And what else was running on that machine in parallel to BOINC? What other DC projects were attached? 8)

Michael.
Fördern, kooperieren und konstruieren statt fordern, konkurrieren und konsumieren.

Bild Bild

Benutzeravatar
rilian
PDA-Benutzer
PDA-Benutzer
Beiträge: 47
Registriert: 08.02.2010 15:38

Re: error - exit code 195 (0xc3) on CMSearch

#3 Ungelesener Beitrag von rilian » 08.03.2010 18:09

Michael H.W. Weber hat geschrieben:Most certainly a "lack of memory" error. You use a Quad-Core machine with only two GB of RAM? :o That means, this machine will start 4 instances of RNA World having a maximum of 512 kB of RAM per core available for computation. On top of that the WUs you were processing are amongst the largest available. And what else was running on that machine in parallel to BOINC? What other DC projects were attached? 8)

Michael.
machine has 4 cores (it is Xeon with 2 physical cores and 2 virtual or smth), and 2GB memory, and i think it has "leave apps in memory while suspended" ON, and runs some web services.

beside usual projects (WCG, LHC, pirates which eat no more than 70MB/WU) it had "distributed data mining" project which eats around 120MB RAM per WU ...

Just looked, some DDM WUs also errored :evil: with "unexpected app termination" ... this is the same host, http://ddm.nicoschlitter.de/Distributed ... 8&offset=0 - good thing they exited just after few mins of computation

looks like i have to detach one of projects :puppydogeyes: :(
I crunch for Ukraine

Benutzeravatar
Michael H.W. Weber
Vereinsvorstand
Vereinsvorstand
Beiträge: 23174
Registriert: 07.01.2002 01:00
Wohnort: Marpurk

Re: error - exit code 195 (0xc3) on CMSearch

#4 Ungelesener Beitrag von Michael H.W. Weber » 08.03.2010 19:02

Normally BOINC detects how much memory you have and on the basis of this RNA World does not deliver WUs that are requiring more RAM than you have available. However, as said many times, we can not (yet) estimate exactly how much of RAM each WU will need. Neither for CMC (although here we now have quite reliable experimental data) nor for CMS. So far, CMS was not a problem as long as we had not engaged on the large eukaryotic genomes which, as also said before, are one to two orders of magnitudes (!) larger in size compared to the prokaryotic ones. So, in some cases it is possible and it in fact has occurred already that we underestimated the RAM requirements of the big CMS WUs. As a consequence, the system might just halt and then of course will crash everything that is in progress at the time. That we need to avoid. What we do at present is to check out maximum RAM usage by relying on user bug reports and then better adjust the individual WU settings at the server's end to avoid delivery of these WUs to underequipped machines in the future. It is the only way to go and it will cause some annoyance at the users end (and consequently at our end as well). I hope, however, that there is some understanding for this issue given the early stage of development we are in with this project. :wink:

Michael.
Fördern, kooperieren und konstruieren statt fordern, konkurrieren und konsumieren.

Bild Bild

Ananas
WU-Schieber
WU-Schieber
Beiträge: 1184
Registriert: 27.04.2008 18:37
Wohnort: Nordlichter Köln

Re: error - exit code 195 (0xc3) on CMSearch

#5 Ungelesener Beitrag von Ananas » 08.03.2010 19:09

This error 3 ... it is familiar. We have never figured out what it means.

Maybe you'll remember ...

Code: Alles auswählen

#define eslOK              0    /* no error/success             */
#define eslFAIL            1    /* failure                      */
#define eslEOL             2    /* end-of-line (often normal)   */
#define eslEOF             3    /* end-of-file (often normal)   */
#define eslEOD             4    /* end-of-data (often normal)   */
#define eslEMEM            5    /* malloc or realloc failed     */
... and so on ...
It isn't clear wether it comes from easel or infernal or something totally different. I cannot find any direct exit() throwing 3, not in Infernal and not in Easel, they all use just 0 or 1.

Everything that goes through cm_Fail() or cm_Die() sends an explanation of the failure to stderr - but should only return 1 as well.

This means, that it might be an uncaught memory error, returned by the function that executes the cm... binary. This would be the system() function - it would be interesting how errno is set after returning from system().
vi BOINC/checkin_notes
:1,$s/bug/feature/g
:wq!

Erzaehlen sich Biologen eigentlich Klein-RNA-Witze?

Benutzeravatar
rilian
PDA-Benutzer
PDA-Benutzer
Beiträge: 47
Registriert: 08.02.2010 15:38

Re: error - exit code 195 (0xc3) on CMSearch

#6 Ungelesener Beitrag von rilian » 08.03.2010 19:27

When BOINC Manager suspends WU due "not enough memory", does it count real memory that is eaten by WU, or that "WU memory" parameter from WU config ?

I think RNA WUs were suspended, and then just could not be restored from memory/swap ...
I crunch for Ukraine

Benutzeravatar
rilian
PDA-Benutzer
PDA-Benutzer
Beiträge: 47
Registriert: 08.02.2010 15:38

Re: error - exit code 195 (0xc3) on CMSearch

#7 Ungelesener Beitrag von rilian » 08.03.2010 21:40

hmm, another one exited with exit code 195 on 2-core machine with 2GB ram, after 3 hours of computations

http://www.rnaworld.de/rnaworld/result. ... id=1294048

i am 100% sure there was enough memory this time, at this moment 1,2GB available

well, all tasks of that WU errored with same status.

You can just run that WU in debug mode

thanks for replies, anyway, and good luck with fixing!
Zuletzt geändert von rilian am 08.03.2010 21:45, insgesamt 1-mal geändert.
I crunch for Ukraine

MonsieurPoot

Re: error - exit code 195 (0xc3) on CMSearch

#8 Ungelesener Beitrag von MonsieurPoot » 08.03.2010 21:45

rilian hat geschrieben:i am 100% sure there was enough memory this time
I had a horse once (strange enough sentence) that used up around 2.8 gigs of memory. Those WUs can be pretty demanding.

Benutzeravatar
rilian
PDA-Benutzer
PDA-Benutzer
Beiträge: 47
Registriert: 08.02.2010 15:38

Re: error - exit code 195 (0xc3) on CMSearch

#9 Ungelesener Beitrag von rilian » 08.03.2010 21:48

MonsieurPoot, do you mean "(horse)" WU as in the task name?

just looked, one of those 4 tasks in 1st topic is Horse, 2 Humans and one Cattle :lol:
I crunch for Ukraine

MonsieurPoot

Re: error - exit code 195 (0xc3) on CMSearch

#10 Ungelesener Beitrag von MonsieurPoot » 08.03.2010 21:52

Yup, that's what I meant.
The bigger the creature, the hungrier the WUs are when it comes to memory consumption. :roll2:
Then again: I am choking on an ordinary rat right now. ;)
Zuletzt geändert von MonsieurPoot am 08.03.2010 21:53, insgesamt 1-mal geändert.

Benutzeravatar
rilian
PDA-Benutzer
PDA-Benutzer
Beiträge: 47
Registriert: 08.02.2010 15:38

Re: error - exit code 195 (0xc3) on CMSearch

#11 Ungelesener Beitrag von rilian » 08.03.2010 21:57

Are there any whales in current batch? :lol:

i've opened performance log window on that machine, will report it if any WU will fail again
I crunch for Ukraine

Benutzeravatar
rilian
PDA-Benutzer
PDA-Benutzer
Beiträge: 47
Registriert: 08.02.2010 15:38

Re: error - exit code 195 (0xc3) on CMSearch

#12 Ungelesener Beitrag von rilian » 08.03.2010 23:14

rhesus-monkey WU exited 195-ish

http://www.rnaworld.de/rnaworld/result. ... id=1294197

WU took 170MB RAM on average, nothing special in performance logs, still available memory around 1,2-1,3Gb, no peaks ... :uhoh:
I crunch for Ukraine

Zurück zu „RNA World Discussions (english)“