This is about every task generating an own copy of a 400MB database, and I assume that those copies are read only so they will always remain identical. I think that assumption is correct, though I haven't seen this discussed before. (For simplicity I ignore the fact that there actually seem to be two databases, one for rosetta and one for minirosetta.)

Well, if all tasks work with identical copies of a database, it shouldn't be too hard to make them work with a single copy. Disk space is not my main concern here, though it has always annoyed me that this amount of data can easily fill your disk cache if you run several tasks, thus flushing more useful data out. It will make your system less responsive with no gain.

But what really needs to be addressed IMO is the total amount of data written to disk. I was about to return to this project after some weeks of absence, but then it occurred to me that in the meantime I had replaced my HDDs with SSDs so I'd better make some calculations. My result is that the above behavior can easily cause terabytes to be written where some hundred megabytes should be enough. For HDDs this won't matter much but I'm not willing to wear my SSDs down for nothing so I'm off again until there is a solution.

As far as I can see the only problem with a single database is that you can't easily tell when it's no longer needed so it can be deleted. But not deleting it at all, even keeping a few versions, is IMO still a better solution than creating a fresh copy for every task. The problem is not the data kept, it's the data written.

When running 8 work units at a time, with the 8 hour length (so three downloads per core per day), then downloading the database adds 400 x 8 x 3 = 9.6 GB/day. And in operation (not including downloads), I am seeing writes of 11 GB/day for six Rosettas (four of 3.78 and two of 4.07) on my i7-4771 under Win 7 64-bit. So the total writes would be about 20 GB/day on my machine.

Doubling that for running 16 cores is still a relatively small amount, and not a problem for SSD life. And you can run longer work units if you want to reduce it further.

---

And I just checked it on an Ubuntu 16.04 machine, where I am running Rosetta 3.78 on six cores of an i7-3770. The writes (averaged over one hour) are 30 kB_WRTN/s according to iostat. That works out to only 2.6 GB/day. Add to that the downloads, and it comes out to even less than for Windows.

---

My result is that the above behavior can easily cause terabytes to be written where some hundred megabytes should be enough.

Besides the fact, that current SSDs can take petabytes (https://techreport.com/review/27436/the-ssd-endurance-experiment-two-freaking-petabytes), so terabytes are not really something you need to think much about, you can set the target CPU run time in your Rosetta@home preferences to 24 hours. That will cut down the writes to 1/3 of the standard value.

But yes, in general I agree with you, two persistent databases would be nice to have, that would speed up the start up of each WU, specially on systems with HDD.

---

.

I agree with the calculation but not with the conclusion. Perhaps I should explain my thoughts better. What I need is just a storage device for a dedicated crunching machine. Any old HDD (sic) is good enough for that. Space doesn't matter and speed also is not very important. More so, any SSD should be fine. With that in mind I buy small, inexpensive SSDs, if for some reason I decide against a HDD. Unfortunately those small SSDs offer little endurance and often are from less known brands where you can't be sure about the quality. Larger SSDs are better with respect to that but cost a lot more and in the end 98% of space is unused.
I don't care how much I can possibly stress a device before I kill it. On the contrary, I wish to make as sure as possible that I don't, not even accidentally. To achieve that I decided to follow the manufacturer's recommendations, which for the devices in question mostly are between 40 and 70 TB of total writes, if any are given at all. For Intel this is even a hard limit, not sure about others. So to play it safe, and hoping for a 10 year life time, I want to limit writes to few terabytes per year, the lower the better. And now Rosetta@Home comes into play and writes 9.6 GB a day, as per your example which seems realistic to me. That extrapolates to 3.5 TB/year, just by writing the same data over and over again, thousands of times, where once or a few times could be enough. I'm not saying this is going to kill the device, but it certainly makes my plan impossible and increases the risk of failure. Whether the risk is high or low I can't say for sure but I'm not willing to take it. From my point of view it is just not necessary.

Larger SSDs are better with respect to that but cost a lot more and in the end 98% of space is unused.

This space isn't really unused, it's used for wear leveling. More free space on SSD means longer life time, because the data won't be written always to the same few empty cells.

To achieve that I decided to follow the manufacturer's recommendations, which for the devices in question mostly are between 40 and 70 TB of total writes, (...) 10 year life time, (...) 9.6 GB a day (...) That extrapolates to 3.5 TB/year

So that means between 10 and 20 years life time (when we think about the writes), much longer than any HDD will last. And as I said, you can cut that down to 3.2GB/day by changing your rosetta@home preferences which will give you up to 60 years life time. The SSD will very likely fail long time before that for some other random reason and not because of the data written to it by the rosetta application, specially if you plan to buy a cheap one.

---

.

This space isn't really unused, it's used for wear leveling.

There's two aspects of space. More space allows broader wear leveling so more data can be written before cells reach their end of life. And of course more space means being able to store more data at a time. I don't want to buy larger drives for the former when I don't need the latter.

More free space on SSD means longer life time, because the data won't be written always to the same few empty cells.

As I understand it the whole space is used for wear leveling, not only the "free" space. Of course things get easier if cells are known to contain no user data. But when "free" space is already much larger than "used" space you won't gain much from even more.

To achieve that I decided to follow the manufacturer's recommendations, which for the devices in question mostly are between 40 and 70 TB of total writes, (...) 10 year life time, (...) 9.6 GB a day (...) That extrapolates to 3.5 TB/year

So that means between 10 and 20 years life time (when we think about the writes)

I think you misunderstood that. Those 3.5 TB/a are only from always rewriting the DB, which I'm suggesting to avoid. As you can see this already halfway reaches my (you may say self-imposed) limit and of course goes on top of the other write operations which can't be avoided but by themselves may not be a problem. Those 3.5 unnecessary extra TB are my problem. Having said that, maybe there actually is a good reason why things need to be this way. In that case I'd be happy to hear it. My guess is that first it was easier to implement and didn't cause any harm in HDD days, and later "it has always been that way".

much longer than any HDD will last

I just recently replaced an 80 GB HDD. 14 years old, without a single bad sector and total overkill for a BOINC machine. Only problem is, it's PATA, none of my mainboards takes that any more, and a controller costs nearly as much as a cheap SSD. Now the host has a 120 GB SSD with 4 GB used.

The SSD will very likely fail long time before that for some other random reason and not because of the data written to it by the rosetta application

You say it will fail anyway. I say if I'm careful it may not fail while it's still useful.