Rechenkraft.net e.V.

Stiwi hat geschrieben: ↑

20.08.2022 10:07

ALso es scheint im Moment wieder deutlich mehr WUs zu geben. ALlerdings haben die scheinbar immer noch netzwerkprobleme wodurch man beim Download manchmal nachhelfen muss

It looks like there was a big batch of OPNG tasks released during Canadian office hours yesterday - and the transient download errors came back with a vengeance. I managed to clear them with multiple transfer retries, but there was another block still struggling when I got up this morning.

They cleared quickly, and there has been no new OPNG work issued since. So it's a volume thing: the download server farm can't handle these big work releases.

The administration team also need to understand the relationship between your servers and our clients. When our clients encounter problems with more than 3 files in succession, they sense a problem and go into extended backoff - quickly reaching several hours. And the OPNG tasks require five files to be downloaded...

So this problem is counter-productive to the research. The more tasks you issue, the slower the downloads: the longer before we can start processing them: and the longer before the results can be returned. (...)

Dear volunteers,

We have taken additional measures to increase the quantity of WUs we can send out, and we have been able to increase the quantity of WUs in flight at any given time. Volunteers should see this reflected on their devices now, and perhaps even over this past week.

We are also relieved to share that the hosting data centre has assigned additional personnel on site to resolve our networking issues, meaning a fix is imminent. We will share with you any further updates we receive from the data centre. The network fix will allow us to bring our remaining servers online, stabilizing and further increasing the WU supply.

Thus, until we are able to deploy all dedicated servers, we must continuously adjust and monitor tasks scheduled in Aurora/Mesos to keep the tasks balanced and the workunits flowing, and so far this process is unduly intensive and sporadic. For example, a recurring job may saturate the scheduler by creating a large number of downstream jobs. This flood of new jobs might then throttle the processing rate of other waiting jobs and thereby interrupt the supply of work. To fix the problem, we would need to temporarily deschedule the parent job, decrease its frequency, or decrease the priority of its children in such a way that does not starve other stages of the pipeline.

Last week, we mentioned that we have begun to investigate concerns over statistics, credit, streaks, and database dumps raised by volunteers. We will have an update on some of these issues next week. We also plan to release a more structured breakdown from the tech team similar to a CHANGELOG starting next week or the week after so that we can increase the frequency and clarity of updates.
(...)
Thanks for your patience and have a great weekend!
-WCG Tech Team