Assistance needed - 2 Long-running VMs just failed - Clones

[Jacob Klein]

If I remember correctly, Oracle has also changed something else which is making it difficult to use the latest VirtualBox versions with BOINC.
Christian knows all the details...

Michael.

I'm actually on top of those issues, too. Basically, so far as I know, the situation is this:

Hardened security:
- In VirtualBox 4.3.14, Oracle implemented "hardened security" in their Windows versions of VirtualBox
- It initially caused many VMs to not run correctly, if the user had certain AntiVirus programs installed, or programs that had dll hooks (including uxtheme patching)
- Oracle has tried desperately to "fix" these issues, but even the latest 4.3.20 still interferes. They usually have a stickied open forum thread on each new version of Virtualbox, so users can report compatibility issues.

BOINC VBoxWrapper Process access:
- When 4.3.14 was released, the then-current version of VBoxWrapper could not successfully launch a VM in BOINC.
- Rom fixed that pretty promptly, and so, current versions of VBoxWrapper should be used by BOINC VM projects.

VM priority in Windows:
- When 4.3.14 was released, the hardened security made it impossible to adjust the VM process priorities.
- So, they run at normal instead of below normal. When BOINC runs multiple VMs at normal priority, it can create system sluggishness or worse.
- I have created an Oracle ticket for them to allow us to adjust this again, however Oracle may never do this.

Long story short:
VirtualBox 4.3.14 had "hardened security" for Windows that caused a lot of compatibility issues, but most have been worked out. If VM projects use the latest VBoxWrapper, they should be able to support clients that are using VirtualBox 4.3.14+. I recommend that the projects do that, upgrading app versions if necessary. It is time to move forward.

Regards,
Jacob Klein

[Michael H.W. Weber]

At present, for DC VirtualBox Version 4.3.12 ist best to use because e.g. ATLAS@home and vLHC@home will NOT run with more recent versions. For RNA World, even the new versions should do.

Michael.

[Jacob Klein]

Time to chime in with another update: I completed one of the VMs!

Work Unit: 6330945
Name: cmsvm2_GA-p[e20-30MB_Lin64f]_1_Oryza-sativa-Japonica-Group_CM000142.lin.EMBL_RF00028_Intron_gpI_1349111823_57652
URL: http://www.rnaworld.de/rnaworld/workuni ... id=6330945
... is done!

It took about 39.8 weeks of processing time, according to top within the VM, on a nearly-complete snapshot that I tested with after it was complete. Some of this time was spent running on my overclocked i7-965 XE 3.74GHz, and some of it was spent on my i7-740QM 1.73GHz laptop. It resulted in 2 files, that are now in the possession of Christian. My inspection shows that they appear to be completed correctly!

This is the first monster I've ever tackled, and I'm truly excited to have been helped!

1) (In Progress)
Name: cmsvm2_GA-p[e20-30MB_Lin64f]_1_Drosophila-melanogaster-(fruit-fly)_AE014297.lin.EMBL_RF00028_Intron_gpI_1349111823_13748_9
http://www.rnaworld.de/rnaworld/workuni ... id=6330939
estimated runtime on reference system: 10w 5d 3h 19m 24s (6491964.4413781 s)
forecast 9655215 sec (~16 weeks)
Failed on: 2 Jul 2014, 11:09:24 UTC
Failed at: 8,345,739 sec (~13.8 weeks)
Current Progress.txt: 98.765%
Current runtime: 369,854 mins (~36.7 weeks)
Note: Has a wingman with a completion time of: 18,268,430 sec (~30.2 weeks; probably valid)

2) (Completed)
Name: cmsvm2_GA-p[e20-30MB_Lin64f]_1_Oryza-sativa-Japonica-Group_CM000142.lin.EMBL_RF00028_Intron_gpI_1349111823_57652_12
http://www.rnaworld.de/rnaworld/workuni ... id=6330945
estimated runtime on reference system: 8w 5d 20h 41m 32s (5344892.6310472 s)
forecast 8627070 sec (~14.25 weeks)
Failed on: 2 Jul 2014, 11:09:24 UTC
Failed at: 10,810,220 sec (~17.9 weeks)
Final Progress.txt: 100%
Final runtime: ~401,000 mins (~39.8 weeks)
Note: Has a wingman with a completion time of: 463,101.50 sec (~0.75 weeks; invalid)

3) (In Progress)
Name: cmsvm2_GA-p[e20-30MB_Lin64f]_1_Oryza-sativa-Japonica-Group_CM000147.lin.EMBL_RF00028_Intron_gpI_1349111823_64512_30
http://www.rnaworld.de/rnaworld/workuni ... id=6330855
estimated runtime on reference system: 8w 0d 21h 7m 47s (4914467.536505 s)
forecast 17647160 sec (~29.2 weeks)
Failed on: 20 Nov 2014, 20:11:45 UTC
Failed at: 13,672,580 sec (~22.6 weeks)
Current Progress.txt: 58.8177%
Current runtime: 172,995 mins (~17.2 weeks) * Started running outside of BOINC using a pre-crash snapshot that was saved before "write errors" in VM

4) (In Progress)
Name: cmsvm2_GA-p[e20-30MB_Lin64f]_1_Oryza-sativa-Japonica-Group_CM000147.lin.EMBL_RF00028_Intron_gpI_1349111823_64512_30
http://www.rnaworld.de/rnaworld/workuni ... id=6330855
estimated runtime on reference system: 8w 0d 21h 7m 47s (4914467.536505 s)
forecast 17647160 sec (~29.2 weeks)
Failed on: 20 Nov 2014, 20:11:45 UTC
Failed at: 13,672,580 sec (~22.6 weeks)
Current Progress.txt: 98.765%
Current runtime: 338,004 mins (~33.5 weeks)

[Michael H.W. Weber]

Excellent.

Michael.

[randi]

Hi,

I am running cmsvm_GA-p[b-Lin64f-2]_1_Salmonella-enterica-subsp.-enterica-serovar-Paratyphi-A-str.-ATCC-9150_CP000026.cir.EMBL_RF00028_Intron_gpI_1358679723_1697_3.
I have BOINC Manager 7.4.36 (x64) and VirtualBox 4.3.12. r93733.

Last night I looked at it and progress was over 99% with 18 minutes remaining. When I looked a little bit later, progress had jumped back to 98.765% with 4:31:56 remaining. I ran it all night. In the morning the progress was still 98.765% and the time remaining had increased slightly to 4:39:48. Elapsed time is 372:56:47

Judging from http://www.rnaworld.de/rnaworld/workuni ... id=6341889, this task has had problems.

What should I do? I have run BOINC tasks, including some VM tasks, for many years, but I don't know too much about the inner workings.

Thanks,
Randi

[Jacob Klein]

It is normal for it to go from 99.999, to 98.765, and stay there until completion. If you read the whole thread, you will better understand the deficiency.

If your CPU is still being used by the VM, then you could just let it run until it completes, with no indication of progress.

If you have further questions, please start your own thread.

Thanks,
Jacob

[randi]

OK, thank you.

[Jacob Klein]

You're welcome -- sorry that sounded ruder than I intended Good luck. I think you'll enjoy reading through the thread, and learning about the "98.765%" limitation.

[Jacob Klein]

I have another update -- I completed another one of the VMs!

Work Unit: 6330939
Name: cmsvm2_GA-p[e20-30MB_Lin64f]_1_Drosophila-melanogaster-(fruit-fly)_AE014297.lin.EMBL_RF00028_Intron_gpI_1349111823_13748
URL: http://www.rnaworld.de/rnaworld/workuni ... id=6330939
... is done!

It took about 45.9 weeks of processing time, according to top within the VM, on a nearly-complete snapshot that I tested with after it was complete. This is the second monster I've ever tackled, and I'm thrilled!

Current status of my 4 VM instances:

1) (Completed)
Name: cmsvm2_GA-p[e20-30MB_Lin64f]_1_Drosophila-melanogaster-(fruit-fly)_AE014297.lin.EMBL_RF00028_Intron_gpI_1349111823_13748_9
http://www.rnaworld.de/rnaworld/workuni ... id=6330939
estimated runtime on reference system: 10w 5d 3h 19m 24s (6491964.4413781 s)
forecast 9655215 sec (~16 weeks)
Failed on: 2 Jul 2014, 11:09:24 UTC
Failed at: 8,345,739 sec (~13.8 weeks)
Final Progress.txt: 100%
Final runtime: ~462,730 mins (~45.9 weeks)
Note: Has a wingman with a completion time of: 18,268,430 sec (~30.2 weeks; probably valid)

2) (Completed)
Name: cmsvm2_GA-p[e20-30MB_Lin64f]_1_Oryza-sativa-Japonica-Group_CM000142.lin.EMBL_RF00028_Intron_gpI_1349111823_57652_12
http://www.rnaworld.de/rnaworld/workuni ... id=6330945
estimated runtime on reference system: 8w 5d 20h 41m 32s (5344892.6310472 s)
forecast 8627070 sec (~14.25 weeks)
Failed on: 2 Jul 2014, 11:09:24 UTC
Failed at: 10,810,220 sec (~17.9 weeks)
Final Progress.txt: 100%
Final runtime: ~401,000 mins (~39.8 weeks)
Note: Has a wingman with a completion time of: 463,101.50 sec (~0.75 weeks; invalid)

3) (In Progress)
Name: cmsvm2_GA-p[e20-30MB_Lin64f]_1_Oryza-sativa-Japonica-Group_CM000147.lin.EMBL_RF00028_Intron_gpI_1349111823_64512_30
http://www.rnaworld.de/rnaworld/workuni ... id=6330855
estimated runtime on reference system: 8w 0d 21h 7m 47s (4914467.536505 s)
forecast 17647160 sec (~29.2 weeks)
Failed on: 20 Nov 2014, 20:11:45 UTC
Failed at: 13,672,580 sec (~22.6 weeks)
Current Progress.txt: 94.6052%
Current runtime: 278,253 mins (~27.6 weeks) * Started running outside of BOINC using a pre-crash snapshot that was saved before "write errors" in VM

4) (In Progress)
Name: cmsvm2_GA-p[e20-30MB_Lin64f]_1_Oryza-sativa-Japonica-Group_CM000147.lin.EMBL_RF00028_Intron_gpI_1349111823_64512_30
http://www.rnaworld.de/rnaworld/workuni ... id=6330855
estimated runtime on reference system: 8w 0d 21h 7m 47s (4914467.536505 s)
forecast 17647160 sec (~29.2 weeks)
Failed on: 20 Nov 2014, 20:11:45 UTC
Failed at: 13,672,580 sec (~22.6 weeks)
Current Progress.txt: 98.765%
Current runtime: 429,598 mins (~42.6 weeks)

[Jacob Klein]

I have another (final) update -- I completed the final VMs!

Current status of my 4 VM instances:

1) (Completed)
Name: cmsvm2_GA-p[e20-30MB_Lin64f]_1_Drosophila-melanogaster-(fruit-fly)_AE014297.lin.EMBL_RF00028_Intron_gpI_1349111823_13748_9
http://www.rnaworld.de/rnaworld/workuni ... id=6330939
estimated runtime on reference system: 10w 5d 3h 19m 24s (6491964.4413781 s)
forecast 9655215 sec (~16 weeks)
Failed on: 2 Jul 2014, 11:09:24 UTC
Failed at: 8,345,739 sec (~13.8 weeks)
Final Progress.txt: 100%
Final runtime: ~462,730 mins (~45.9 weeks)
Note: Has a wingman with a completion time of: 18,268,430 sec (~30.2 weeks; probably valid)

2) (Completed)
Name: cmsvm2_GA-p[e20-30MB_Lin64f]_1_Oryza-sativa-Japonica-Group_CM000142.lin.EMBL_RF00028_Intron_gpI_1349111823_57652_12
http://www.rnaworld.de/rnaworld/workuni ... id=6330945
estimated runtime on reference system: 8w 5d 20h 41m 32s (5344892.6310472 s)
forecast 8627070 sec (~14.25 weeks)
Failed on: 2 Jul 2014, 11:09:24 UTC
Failed at: 10,810,220 sec (~17.9 weeks)
Final Progress.txt: 100%
Final runtime: ~401,000 mins (~39.8 weeks)
Note: Has a wingman with a completion time of: 463,101.50 sec (~0.75 weeks; invalid)

3) (Completed)
Name: cmsvm2_GA-p[e20-30MB_Lin64f]_1_Oryza-sativa-Japonica-Group_CM000147.lin.EMBL_RF00028_Intron_gpI_1349111823_64512_30
http://www.rnaworld.de/rnaworld/workuni ... id=6330855
estimated runtime on reference system: 8w 0d 21h 7m 47s (4914467.536505 s)
forecast 17647160 sec (~29.2 weeks)
Failed on: 20 Nov 2014, 20:11:45 UTC
Failed at: 13,672,580 sec (~22.6 weeks)
Final Progress.txt: 100%
Final runtime: (~33 weeks) * Started running outside of BOINC using a pre-crash snapshot that was saved before "write errors" in VM

4) (Completed)
Name: cmsvm2_GA-p[e20-30MB_Lin64f]_1_Oryza-sativa-Japonica-Group_CM000147.lin.EMBL_RF00028_Intron_gpI_1349111823_64512_30
http://www.rnaworld.de/rnaworld/workuni ... id=6330855
estimated runtime on reference system: 8w 0d 21h 7m 47s (4914467.536505 s)
forecast 17647160 sec (~29.2 weeks)
Failed on: 20 Nov 2014, 20:11:45 UTC
Failed at: 13,672,580 sec (~22.6 weeks)
Final Progress.txt: 100%
Final runtime: 429,598 mins (~42.6 weeks)

Note: The results for 3 and 4, do indeed match.