Long running work unit

[yoyo]

extended

[Michael H.W. Weber]

WONDERFUL after 200 or 300 hours of processing time and reaching almost 30% of calculation, restraining the Mac from rebooting (even though 2 apps already want that, but I was waiting), I came home today and...

Lun 15 avr 15:27:32 2013 | RNA World | Task cms_GA-p[e20-30MB_Lin64s]_1_Drosophila-yakuba_CM000160.lin.EMBL_RF00177_SSU_rRNA_5_1340128624_30357_7 exited with zero status but no 'finished' file
Lun 15 avr 15:27:32 2013 | RNA World | If this happens repeatedly you may need to reset the project.
Lun 15 avr 15:28:05 2013 | RNA World | Restarting task cms_GA-p[e20-30MB_Lin64s]_1_Drosophila-yakuba_CM000160.lin.EMBL_RF00177_SSU_rRNA_5_1340128624_30357_7 using cmsearch version 30 in slot 10

GREAT ! Now 3% completed after 6 fantastic new hours of calculation.

Obviously I just abandoned it, enough calculation hours thrown to the bin. The next so called long running WU will have the same destiny.

Ok so time went by, then another big WU coming, ok let's give it a try, and I had not done a close follow up of this task, I had suspended it a few time (with keep in memory option) and I though it had run maybe 70 hours of calculation, but today I realize it was not the case at all, I searched all the log history and this is what happened (again) :

Sam 11 mai 21:04:05 2013 | RNA World | Starting task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10

Jeu 16 mai 19:51:53 2013 | RNA World | Task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 exited with zero status but no 'finished' file
Jeu 16 mai 19:51:53 2013 | RNA World | If this happens repeatedly you may need to reset the project.
Jeu 16 mai 19:51:53 2013 | RNA World | Restarting task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10

Ven 17 mai 01:58:39 2013 | RNA World | Task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 exited with zero status but no 'finished' file
Ven 17 mai 01:58:39 2013 | RNA World | If this happens repeatedly you may need to reset the project.
Ven 17 mai 01:58:39 2013 | RNA World | Restarting task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10

Ven 17 mai 21:33:39 2013 | RNA World | task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 suspended by user
(note : I have the option to keep in memory when suspended)

Dim 19 mai 01:11:47 2013 | RNA World | work fetch resumed by user
Dim 19 mai 01:11:59 2013 | RNA World | task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 resumed by user
Dim 19 mai 01:12:00 2013 | RNA World | Resuming task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10

Lun 20 mai 09:07:01 2013 | RNA World | Task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 exited with zero status but no 'finished' file
Lun 20 mai 09:07:01 2013 | RNA World | If this happens repeatedly you may need to reset the project.
Lun 20 mai 09:07:01 2013 | RNA World | Restarting task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10
Lun 20 mai 09:15:40 2013 | RNA World | Task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 exited with zero status but no 'finished' file
Lun 20 mai 09:15:40 2013 | RNA World | If this happens repeatedly you may need to reset the project.
Lun 20 mai 09:15:40 2013 | RNA World | Restarting task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10

Mar 21 mai 11:36:30 2013 | RNA World | Task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 exited with zero status but no 'finished' file
Mar 21 mai 11:36:30 2013 | RNA World | If this happens repeatedly you may need to reset the project.
Mar 21 mai 11:36:30 2013 | RNA World | Restarting task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10

Mar 21 mai 18:41:17 2013 | RNA World | Task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 exited with zero status but no 'finished' file
Mar 21 mai 18:41:17 2013 | RNA World | If this happens repeatedly you may need to reset the project.
Mar 21 mai 18:41:17 2013 | RNA World | Restarting task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10

Mar 21 mai 19:56:09 2013 | RNA World | Task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 exited with zero status but no 'finished' file
Mar 21 mai 19:56:09 2013 | RNA World | If this happens repeatedly you may need to reset the project.
Mar 21 mai 19:56:09 2013 | RNA World | Restarting task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10

Mar 21 mai 20:20:34 2013 | RNA World | Task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 exited with zero status but no 'finished' file
Mar 21 mai 20:20:34 2013 | RNA World | If this happens repeatedly you may need to reset the project.
Mar 21 mai 20:20:34 2013 | RNA World | Restarting task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10

Mar 21 mai 20:26:27 2013 | RNA World | Task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 exited with zero status but no 'finished' file
Mar 21 mai 20:26:27 2013 | RNA World | If this happens repeatedly you may need to reset the project.
Mar 21 mai 20:26:27 2013 | RNA World | Restarting task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10

Mar 21 mai 20:31:10 2013 | RNA World | Task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 exited with zero status but no 'finished' file
Mar 21 mai 20:31:10 2013 | RNA World | If this happens repeatedly you may need to reset the project.
Mar 21 mai 20:31:10 2013 | RNA World | Restarting task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10

Mar 21 mai 20:41:12 2013 | RNA World | Task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 exited with zero status but no 'finished' file
Mar 21 mai 20:41:12 2013 | RNA World | If this happens repeatedly you may need to reset the project.
Mar 21 mai 20:41:12 2013 | RNA World | Restarting task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10

Mar 21 mai 21:05:46 2013 | RNA World | Task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 exited with zero status but no 'finished' file
Mar 21 mai 21:05:46 2013 | RNA World | If this happens repeatedly you may need to reset the project.
Mar 21 mai 21:05:46 2013 | RNA World | Restarting task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10

Mar 21 mai 21:11:08 2013 | RNA World | Task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 exited with zero status but no 'finished' file
Mar 21 mai 21:11:08 2013 | RNA World | If this happens repeatedly you may need to reset the project.
Mar 21 mai 21:11:08 2013 | RNA World | Restarting task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 using cmsearch version 30 in slot 10


So VERY sadly :

Mar 21 mai 21:39:41 2013 | RNA World | task cms_GA-p[b-Lin64f-2]_1_Streptococcus-mutans-UA159_AE014133.cir.EMBL_RF00028_Intron_gpI_1358679723_1849_6 aborted by user


I'm done with RNA long WU, really not reliable enough, so many hours of calculation lost in space...

Well, the good news is: checkpointing appears to come up soon. After all.

Michael.

[JeromeC]

This is what I call a hell of a good news !!!

I'll keep an eye on this topic to see when it happens and then maybe will give it ANOTHER try

Thanks for the info.

[biodoc]

Could you please extend these WUs which have a deadline of May 27th?

6321906 422 hours runtime so far and 100% for several days
6330001 422 hours runtime so far and 100% for several days (FInished)
6325696 422 hours runtime and only 21% complete.

Please extend 9321906 and 6325696. The other one finished on time.

thanks!

[yoyo]

extended.
BTW: We have now a list of these monster workunits: http://www.rnaworld.de/rnaworld/y_monster.php

yoyo

[ArcSedna]

I would like an additional extension for this monster workunit!

6032579 : elapsed 2229.5hrs, but only 38% progress.

Thanks.

[yoyo]

extended

[Conan]

Just wondering how long these "Intron" type work units will run past there deadline?

I have This one which is still running 3 days after it passed dead line and into the 4th day after it reached 100%.

It had reached around 230 hours and 100% but then just kept going, it is now up to 314 Hours.

It is still using CPU so it has not failed, just wondering when it will finish.

As it was going to finish before deadline I did not ask for an extension.

Conan

[Michael H.W. Weber]

That WU can be reaactivated and the deadline extended. You just need to come to this forum and tell us in time. Keep it running. These long WUs often run double as long as initially indicated (i.e. after reaching 100% the keep going) but finish successfully.

Michael.

[JeromeC]

but finish successfully.

... or they don't.

(they were not aborted, they failed, see above and my previous posts here)

[Trotador]

This one can go to the Hall of Fame of Monster Hunter

http://www.rnaworld.de/rnaworld/result. ... d=14899590

[yoyo]

Just wondering how long these "Intron" type work units will run past there deadline?

I have This one which is still running 3 days after it passed dead line and into the 4th day after it reached 100%.

It had reached around 230 hours and 100% but then just kept going, it is now up to 314 Hours.

It is still using CPU so it has not failed, just wondering when it will finish.

As it was going to finish before deadline I did not ask for an extension.

Conan

Deadline extended.