So...ahhh...what's happening in the world of Rosetta?

The main page Server Status hasn't updated since the 17th.

---

Grant
Darwin NT

So...ahhh...what's happening in the world of Rosetta?

The main page Server Status hasn't updated since the 17th.

I didn't realise - mainly through not looking
But going by what's reported at Boincstats, it's still within 9 tasks of the level it's really at (0)

---

I found an update in the only WCG forum I visit
Jun 15, 2026
Here is what I've been doing, and an official post on the lab website "Operational Status" tab should follow soon:
1. Docker/Podman/Virtual Box workunits are now circulating in small batches in the beta30 project, some running on the cancer datasets for Lung and Ovarian using highest scoring large signatures to seed the search space for the optimizer in a promising direction, and with this capability now working I am also going to distribute our Mojo/Modular build for beta30 so we can see if this angle to get general GPU support is going to work (https://mojolang.org/docs/requirements/#gpu-compatibility)
2. The results API switch is currently running in "shadow mode" in production, which just means for every request going to the legacy database another is being sent to the citus coordinator so I can assess parity, get some performance numbers, size the connection pools and limits, and see if Websphere is going to crash for some reason or other which it already did when I first rolled this out. I am going to make this switch soon, the hope is that the massive archive of results that will simply timeout the API right now if large enough will be viewable even for the largest results sets, and that an accurate picture of workunits IN_PROGRESS and otherwise will henceforth be visible on the website and reflect what is currently running on volunteer devices. I want to move the stats rollup, the stats dump, and Results API to citus too, but these are all scary and invasive overhauls, so we'll see if just the Results page can be fixed this way first before tackling the chronic under-reporting of the stats rollup and stats dump by switching out the database.
3. Bulk of the incorrectly "floor" 14.0 credited workunits have been recalculated, seems to be some stragglers and I will do another pass to look for them. Also, the aborts of `_0` and `_1` workunits and many cached resends should start to dissipate this week, and my hope is that a more typical distribution and ratio should fall into place for MCM1 as that happens. I pushed a feeder build with a guard against assigning already canonicalized results, as it seems resends got ahead of the `_0` and `_1` populations? I'm still looking into what I did that caused this. My expectation is that somewhere in the multiple repair passes on results, or when I created additional resends manually by bumping up target_nresults, or when to relieve memory pressure on the nodes and handle 404s from the transitioner bug, server crashes, and botched repair operations, I added a service that sits behind apache, consults the corresponding batch plan protobuf "schema", and just templates out workunits based on the mcm1_create_work code, I managed to create this problem.
4. MCM1 results are not going into a black hole. We run a batch conversion to the Parquet file format for each 24h window (find uploads in the in-memory cache with mtime > tail of previous "assimilated" listing), and those Parquet files go into the "Ducklake" (https://ducklake.select/) data lake. I can query with SQL using DuckDB every single signature uploaded since the move to Nibi with standard SQL. The conversion code is based on the new mcm1_validator_assimilator code that actually unrolls the signatures and compares their contents. I used this data store to grab some of the best 100 gene signatures to test MDMG, the new LibTorch MAM1 application, on Ovarian in the staging environment and it worked and found some decent shorter signatures right away. Also, to "fall through" to searching for missing results that aren't in the local in-memory cache on a node anymore, and to restore correct credit after introducing a disastrous credit amplification bug. We intend to merge the historical results from Jurisica lab servers into this dataset at Nibi, once the S3 object storage feature is made available to us, and expand in multiple directions from there. Using the top historical signatures to seed the search space of MDMG batches, given the articles we publish already establish promising results identified by our lab scientists in the MCM1 data as is, we expect to get our best results to date.

---

Another Forum post from 17th June:

Hopefully the results page will now reflect, with a slight delay for some users with very large result sets, what the BOINC manager already shows. It is possible that some of the repair work for the various issues I introduced over the past several months is affecting credit, validation, etc only made it to the legacy BOINC database, as the thinking when setting this up was that maintaining an authoritative view of credit and final result state in that database, which already has all the API and stats rollup plumbing attached to it would be far less risky and more straightforward than moving the entire results API over to the new database cluster.
Unfortunately, many mistakes were made and I'm sorry for the chaos - I accelerated the multi-year planned upgrade to a distributed architecture because in the new environment, the network attached storage we were provided after our storage server previously serving up SSD pools for the databases and shared filesystem that workunit upload/download/processing relied on was scuttled during the migration, something I did not understand was going to happen until mid-august.
In addition, the Lenovo nodes with large local disks I initially planned the Kafka/Citus/striped workunit partitioning scheme around in prior years could not be connected to the new cloud environment and were shipped back to the lab, so rather than a massive jump in performance to facilitate the GPU throughput we were hoping to ramp up to we ended up with a BOINC database data direcrory pinned at 100% utilization, causing rollbacks and crashes even after I looped most of the state management round trips that used to hit the BOINC database into the Kafka cluster, and batched up absolutely everything about validation and assimilation into a single large batch transaction with backoff/retry logic and hot reloadable intervals at which to try the database.
Hence, when citus v14 was released and resolved some of the issues I had thought were show stoppers for trying to implement postgres support for BOINC and making our capacity scalable, AND BOINC server release 1.6.1 came out around the same time, I decided to try and eliminate our technical debt once and for all and get us off the legacy BOINC build we inherited, spread the IO across N network attached storage volumes behind citus, and further optimize our partitioning scheme to remove Kafka for most of the hot path and just talk locally to the citus/postgres workers instead of having to manage a non-trivial Kafka (Redpanda actually, easier to manage and faster although there are quite a few "not free" enterprise features that encourage going from the open source to licensed version...) cluster in addition to citus and everything else in the stack.

Apologies for such a brief and rambling explanation of our decision-making. Largely this is all my fault for not clarifying well in advance and with due paranoia what the Nibi migration actually meant for our infrastructure, and so began a very long sprint that I am hoping to bring to a positive ending this summer with some part time help joining the team and (I hope...) the critical bugs we incurred testing in production, because shutting down the project would likely be its doom, and fail to uphold obligations we committed to in order to allow the project to survive to this point.

While this is not exactly an "official" statement, it is the perspective of the tech team in brief and I hope it at least provides some answers as to why things had gone from bad to worse and again for such a long time.

---

Lots of tasks at WCG recently, until today when the whole website went down and no tasks can be uploaded.
The homepage is back up, I notice, but nothing has been resolved just yet.

Meanwhile, SiDock, which has been dead for a few months, finally accepted uploaded tasks earlier today/yesterday.

Then, a new message appeared in my Boinc log, reading

16/07/2026 23:16:16 | SiDock@home | Message from server: The project has changed its security key. Please remove and add this project.

Doing that works immediately, then

17/07/2026 0:32:35 | SiDock@home | Fetching scheduler list
17/07/2026 0:32:37 | SiDock@home | Master file download succeeded
17/07/2026 0:32:44 | SiDock@home | Sending scheduler request: Project initialization.
17/07/2026 0:32:44 | SiDock@home | Requesting new tasks for CPU and NVIDIA GPU
17/07/2026 0:32:45 | SiDock@home | Scheduler request completed: got 0 new tasks

So something is happening there slowly

---

WCG is back overnight and providing tasks again.
I'm scanning through the forum to find some info. First this one from 13 July (Monday)

~~~
Jul 13, 2026 4:14:45 PM
Apologies for the sparsity of communications

Outage yesterday, another incoming
There was an outage yesterday affecting about half of our servers, which lost network connectivity on all interfaces.
Volunteers may have noticed the load balancer 503 page when trying to visit the website and some uploads/downloads failing with the familiar HTTP error.
Hosting fixed that issue, however there is a planned outage today and we were told to expect that we will again lose all connectivity to our boxes (believe they are replacing a switch, anecdotal).

Tech Team has tripled in size!
We have two new developers joining the project, their focus will be on web development and API development.
Intent is to resolve the many issues plaguing the website and stats APIs and continue IBM's plan to update specific website pages (they left behind design assets and well-scoped documentation), while I continue to work on the BOINC backend, MAM1, and ARP1.
Currently, I have been working on the MAM1 GPU build by implementing a population-based optimizer from the ensmallen library called "Differential Evolution" to join the existing Simulated Annealing and Particle Swarm Optimization execution modes, and launching multiple folds of cross-validation at once. GPU utilization now provides a decent speedup.
Plan is to:
- do a full analytics pass on signatures obtained so far from testing MAM1
- fix the low credit and poor occupancy of multi-threaded build especially
- use the best signatures we have so far to seed the search space for the next round of MAM1 (and the few workunits we send out that crunch Ovarian workunits from MCM1 since the app is compatible with the data format. This has been very promising in terms of MCC scores)
- release regular batches for all of the MAM1 application versions with these fixes, including GPU builds

I am currently unsure how the cleanup is going to go after the maintenance outage today, or if this outage will materialize at all.

I will review forum posts and aim to post a coherent status update Friday afternoon, July 17th 2026.

---

Jul 16, 2026 2:00:04 PM
Device profile issue may come back over the course of the day, among others.
We were informed yesterday night that the cloud is getting a security update that requires restarting most if not all instances, in addition to the networking outage - hosting said they would make best efforts to live migrate our key infra and avoid a reboot, but can't guarantee it.
Just got the website back up, but one of the postgres node instances (1/6th of the new horizontally scalable partitioned architecture I've been talking about in updates) came back up in an "Error" state that I'm asking hosting about and looking into.

We're meeting with the ARP1 team this morning, I'm asking them to look at the recent handful of results we've received back and validated and if LGT them as well, we'll be able to restart ARP1 after the data center maintenance.

---

I'm scanning through the forum to find some info. First this one from 13 July (Monday)

Currently, I have been working on the MAM1 GPU build by implementing a population-based optimizer from the ensmallen library called "Differential Evolution" to join the existing Simulated Annealing and Particle Swarm Optimization execution modes, and launching multiple folds of cross-validation at once. GPU utilization now provides a decent speedup.

That's interesting. Which language, CUDA? OpenCl??
I cannot find details in the forum

---

Meanwhile, SiDock, which has been dead for a few months, finally accepted uploaded tasks earlier today/yesterday.

Then, a new message appeared in my Boinc log, reading

16/07/2026 23:16:16 | SiDock@home | Message from server: The project has changed its security key. Please remove and add this project.

Doing that works immediately, then

17/07/2026 0:32:35 | SiDock@home | Fetching scheduler list
17/07/2026 0:32:37 | SiDock@home | Master file download succeeded
17/07/2026 0:32:44 | SiDock@home | Sending scheduler request: Project initialization.
17/07/2026 0:32:44 | SiDock@home | Requesting new tasks for CPU and NVIDIA GPU
17/07/2026 0:32:45 | SiDock@home | Scheduler request completed: got 0 new tasks

So something is happening there slowly

SiDock tasks have come down this morning - shocker!

---

I'm scanning through the forum to find some info. First this one from 13 July (Monday)

Currently, I have been working on the MAM1 GPU build by implementing a population-based optimizer from the ensmallen library called "Differential Evolution" to join the existing Simulated Annealing and Particle Swarm Optimization execution modes, and launching multiple folds of cross-validation at once. GPU utilization now provides a decent speedup.

That's interesting. Which language, CUDA? OpenCl??
I cannot find details in the forum

If it's not in the forum for me to quote, I have no idea
Everything said is far above my level of understanding

---