VboxHeadless crash ate work unit

Everything about the project RNA World
Nachricht
Autor
Silverdrake
Mikrocruncher
Mikrocruncher
Beiträge: 17
Registriert: 21.04.2013 22:06

VboxHeadless crash ate work unit

#1 Ungelesener Beitrag von Silverdrake » 17.02.2015 11:37

I was running several programs at once and got a notice that VBoxHeadless had crashed. After I finished what I was doing, I shut down BOINC to check VBox and clean out any extra snapshots and get the WU back on track. Imagine my surprise when VBox came up empty and said that I hadn't created any VMs!

Does a VBoxHeadless crash usually destroy the VM and wipe out the snapshots? If not, is there anything I should look for that might give you data on why it did, this time?

This is really disappointing. I finally got a computer that could run the VM WUs and all I've had are errors. :(

Benutzeravatar
Michael H.W. Weber
Vereinsvorstand
Vereinsvorstand
Beiträge: 23174
Registriert: 07.01.2002 01:00
Wohnort: Marpurk

Re: VboxHeadless crash ate work unit

#2 Ungelesener Beitrag von Michael H.W. Weber » 17.02.2015 12:00

To the best of my knowledge, our VM runs very stable by now. Of course, it makes no sense that the snapshots are deleted upon VM crash. What is possible in rare events is that a system crashes right at the time of writing a snapshot - which then is corrupted. But even here, I would hope (and expect) that the previous snapshot is deleted ONLY after successful writing of the current one. But here, maybe Christian knows more...

Michael.

P.S.: The default snapshot writing interval is set to 10 minutes at minimum - regardless of what your local settings say (as long as this setting is set to less that 10 minutes). You might think of increasing this interval beyond the 10 minutes mark to reduce the I/O load on the system.
Fördern, kooperieren und konstruieren statt fordern, konkurrieren und konsumieren.

Bild Bild

ChristianB
Admin
Admin
Beiträge: 1920
Registriert: 23.02.2010 22:12

Re: VboxHeadless crash ate work unit

#3 Ungelesener Beitrag von ChristianB » 17.02.2015 19:35

Can you provide me with your hostid or rna world username? I'll see if the task got reported and what the error log shows.

Did the VM show in the manager before the crash? It's possible that the RNA VM's do not show there but are fine.

Silverdrake
Mikrocruncher
Mikrocruncher
Beiträge: 17
Registriert: 21.04.2013 22:06

Re: VboxHeadless crash ate work unit

#4 Ungelesener Beitrag von Silverdrake » 18.02.2015 00:09

Username: Silverdrake
User ID: 8407

Computer ID: 25167

Task: 14948503
Work Unit: 6330877


Yes, the VM had been in Vbox manager. I had to go in a few times to delete excess snapshots to get it running again.

In case it's any help:
Files in BOINC/Projects/rnaworld from the time of the crash (all are 1kb in size)
stat_icon
slideshow_01
slideshow_02
slideshow_03
slideshow_04
slideshow_05
slideshow_06
slideshow_07
slideshow_08

ChristianB
Admin
Admin
Beiträge: 1920
Registriert: 23.02.2010 22:12

Re: VboxHeadless crash ate work unit

#5 Ungelesener Beitrag von ChristianB » 18.02.2015 09:58

The VM crashed when it couldn't restore from the last checkpoint. Which in the end was a good thing because the computation was not going very well. It seems to me that your checkpointing intervall somehow got set to 60 seconds and the minimum of 1800 seconds was discarded somehow. The VM was checkpointing 60 seconds after it was started and then doing some computation and eventually checkpointing there. In the last roughly 30 hours before the crash the VM was started 18 times which is very unusual.

I had a report of another odd behaviour involving the checkpoint interval. Which let me to believe that there is side effect in the code that is only experienced with tasks that have run for a while. Unfortunately this will not get more investigation because the windows vboxwrapper was redone some time ago and we first have to make the switch to the new version and see if the problem resurfaces.

I will grant you partial credit for the task and hope you will stick with the project..

Silverdrake
Mikrocruncher
Mikrocruncher
Beiträge: 17
Registriert: 21.04.2013 22:06

Re: VboxHeadless crash ate work unit

#6 Ungelesener Beitrag von Silverdrake » 19.02.2015 12:23

Does each VM set up its own checkpoint interval when it is created? (I don't have any now and don't see anything in VBox manager that would be a general setting for it.)

First, I hope I didn't mess up some setting when I went in to fix the problem I had with multiple snapshots causing it to not run. Second, I wonder if the checkpoint interval decrease might have been causing the excess snapshots. The WU had been running for a while before that problem started cropping up. Might that be an indicator to watch for with the new version (excess snapshots > investigate checkpoint interval)?

Thank you.

Benutzeravatar
Michael H.W. Weber
Vereinsvorstand
Vereinsvorstand
Beiträge: 23174
Registriert: 07.01.2002 01:00
Wohnort: Marpurk

Re: VboxHeadless crash ate work unit

#7 Ungelesener Beitrag von Michael H.W. Weber » 19.02.2015 12:34

If you have the VirtualBox Tools installed, you can manually delete all VB date to clean your system. Then restart RNA World to run our tasks and please do not fiddle around with any settings there.

The checkpointing interval should by default be set to 10 minutes. I think, this interval still is set too short. You can just change the hard disk writing interval in the BOINC manager settings and that interval then should be used by VirtualBox PROVIDED you do not place a value there which is smaller than 10 minutes (in that case, again, 10 minutes will be used reagardless of what you select).
Remeber, I/O is time-consuming. The smaller the checkpoint intervals, the more safe you are in terms of loosing data in case of a crash BUT the increased I/O on the HD in order wo write the snapshots also costs compute time. You need to find out your personal preference. I do checkpoint every half an hour (but will probably increase to 2 hours in the future as my machines run rock stable).

Christian will correct in case I said something wrong. :)

Michael.
Fördern, kooperieren und konstruieren statt fordern, konkurrieren und konsumieren.

Bild Bild

Silverdrake
Mikrocruncher
Mikrocruncher
Beiträge: 17
Registriert: 21.04.2013 22:06

Re: VboxHeadless crash ate work unit

#8 Ungelesener Beitrag von Silverdrake » 21.02.2015 20:53

I don't have VBox tools, so I deinstalled VBox and reinstalled from the BOINC with repair. Also set BOINC's interval to 900 seconds. If it was bleedover from BOINC causing the ultra-short checkpointing, that should take care of it.

Thank you for the tip about I/O eating up so much time. I had thought a shorter interval would be better. On reflection, that's also a lot of run time on my hard drive that isn't necessary.

Zurück zu „RNA World Discussions (english)“