Multiattach mode disk images

Author	Message
[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,571,020 RAC: 7,185	Message 104952 - Posted: 18 Feb 2022, 21:24:47 UTC I hope Rosetta@Home will soon implement this new function of Boinc's wrapper. This would give a great advantage to the project!! ID: 104952 · Rating: 0 · rate: / Reply Quote

Grant (SSSF) Send message Joined: 28 Mar 20 Posts: 1678 Credit: 17,775,863 RAC: 22,702	Message 104954 - Posted: 18 Feb 2022, 21:45:55 UTC Multiattach mode images can be attached to more than one virtual machine at the same time, even if these machines are running simultaneously. For each virtual machine to which such an image is attached, a differencing image is created. As a result, data that is written to such a virtual disk by one machine is not seen by the other machines to which the image is attached. Each machine creates its own write history of the multiattach image. Technically, a multiattach image behaves identically to an immutable image except the differencing image is not reset every time the machine starts. This mode is useful for sharing files which are almost never written, for instance picture galleries, where every guest changes only a small amount of data and the majority of the disk content remains unchanged. The modified blocks are stored in differencing images which remain relatively small and the shared content is stored only once at the host. Too much to hope for. It would certainly end the massive disk activity needed for Python Tasks by not having to write whole the whole disk image every time a new Task starts, but just the data that has changed from the previous image. ie the the ID number for the new VM, and the new Task data files. All the other files are unchanged (the VM itself & the Python executables & support files). Grant Darwin NT ID: 104954 · Rating: 0 · rate: / Reply Quote

[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,571,020 RAC: 7,185	Message 104956 - Posted: 18 Feb 2022, 22:10:40 UTC - in response to Message 104954. Too much to hope for. All the other files are unchanged (the VM itself & the Python executables & support files). I have over 42 gb of space used by Rosetta and everytime a python wu starts it's a lot of disk usage (computers without SSD....a nightmare). A simple update to boinc infrasctucture solve two problems: space and performances. ID: 104956 · Rating: 0 · rate: / Reply Quote

dcdc Send message Joined: 3 Nov 05 Posts: 1831 Credit: 119,536,330 RAC: 6,139	Message 104999 - Posted: 19 Feb 2022, 13:58:04 UTC This would make a huge difference to throughput, and presumably to the impact on users and to drive wear too. Any idea what is required to implement it? A yet-to-be released version of the BOINC client, or just an updated wrapper file? And what do the project need to do to implement this? ID: 104999 · Rating: 0 · rate: / Reply Quote

computezrmle Send message Joined: 9 Dec 11 Posts: 63 Credit: 9,680,103 RAC: 0	Message 105000 - Posted: 19 Feb 2022, 14:43:39 UTC - in response to Message 104999. The project team would have to provide a slightly modified app_version including: - the new vboxwrapper; Either a self compiled version from the github source or an officially released version (don't know if/when BOINC will provide it independend of a full release). - a new job template without the copy flag (to ensure the client does not copy the vdi to the worker slot) Everything else is done by standard functions VirtualBox has built in for many years. Additional measures Although it's not a must to change the vdi image (AIMNet_vm_v2.vdi) I would suggest to do some cleanups since it still wastes 253 MB in /usr/lib/modules/4.19.0-13-amd64 from an older kernel which is not in use any more and 514 MB in /var/cache/apt/archives which is only required during installation. After the cleanup unused space needs to be overwritten with zeros and the new vdi needs to be compacted with "vboxmanage ... --compact". That way the size shrinks from 6.9 GB to 4.2 GB. I also suggest to switch the VM's virtual disk adapter to sata/IntelAHCI (via vbox_job.xml) and to get rid of forcing a VM snapshot every 10 min. See: https://boinc.berkeley.edu/trac/wiki/VboxApps ID: 105000 · Rating: 0 · rate: / Reply Quote

[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,571,020 RAC: 7,185	Message 105047 - Posted: 20 Feb 2022, 9:53:17 UTC - in response to Message 105000. The project team would have to provide a slightly modified app_version including: - the new vboxwrapper; Either a self compiled version from the github source or an officially released version (don't know if/when BOINC will provide it independend of a full release). - a new job template without the copy flag (to ensure the client does not copy the vdi to the worker slot) Your explanation is very clear ID: 105047 · Rating: 0 · rate: / Reply Quote

[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,571,020 RAC: 7,185	Message 105350 - Posted: 7 Mar 2022, 13:23:47 UTC Any news??? ID: 105350 · Rating: 0 · rate: / Reply Quote

computezrmle Send message Joined: 9 Dec 11 Posts: 63 Credit: 9,680,103 RAC: 0	Message 105351 - Posted: 7 Mar 2022, 13:50:02 UTC - in response to Message 105350. Required steps 1. A premade vboxwrapper executable needs to be created. See: https://github.com/BOINC/boinc/issues/4649 Or The Rosetta team distributes a self compiled vboxwrapper (I doubt they will). 2. The Rosetta team will have to create a new app version based on that vboxwrapper ID: 105351 · Rating: 0 · rate: / Reply Quote

[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,571,020 RAC: 7,185	Message 105353 - Posted: 7 Mar 2022, 21:25:20 UTC - in response to Message 105351. Required steps 1. A premade vboxwrapper executable needs to be created. See: https://github.com/BOINC/boinc/issues/4649 2. The Rosetta team will have to create a new app version based on that vboxwrapper Seems not so difficult... ID: 105353 · Rating: 0 · rate: / Reply Quote

[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,571,020 RAC: 7,185	Message 105728 - Posted: 28 Mar 2022, 8:14:06 UTC - in response to Message 105353. Seems not so difficult... Indeed....they did nothing.... :-( ID: 105728 · Rating: 0 · rate: / Reply Quote

computezrmle Send message Joined: 9 Dec 11 Posts: 63 Credit: 9,680,103 RAC: 0	Message 105729 - Posted: 28 Mar 2022, 8:45:32 UTC - in response to Message 105728. The github issue regarding Step 1 has recently been added to the next BOINC client milestone. That's a good sign. Step 2 is a Rosetta task, so :-(( ID: 105729 · Rating: 0 · rate: / Reply Quote

[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,571,020 RAC: 7,185	Message 105736 - Posted: 28 Mar 2022, 13:10:43 UTC - in response to Message 105729. Last modified: 28 Mar 2022, 13:13:08 UTC The github issue regarding Step 1 has recently been added to the next BOINC client milestone. That's a good sign. Great!! Maybe other projects can use this feature... ID: 105736 · Rating: 0 · rate: / Reply Quote

rjs5 Send message Joined: 22 Nov 10 Posts: 273 Credit: 23,007,065 RAC: 5,928	Message 105746 - Posted: 28 Mar 2022, 19:31:53 UTC - in response to Message 105736. The github issue regarding Step 1 has recently been added to the next BOINC client milestone. That's a good sign. Great!! Maybe other projects can use this feature... You can also download other premade vbox images of the Rosetta environment. https://www.osboxes.org/virtualbox-images/ ID: 105746 · Rating: 0 · rate: / Reply Quote

computezrmle Send message Joined: 9 Dec 11 Posts: 63 Credit: 9,680,103 RAC: 0	Message 105747 - Posted: 28 Mar 2022, 19:48:11 UTC - in response to Message 105746. You can also download other premade vbox images of the Rosetta environment. https://www.osboxes.org/virtualbox-images/ VM images from that website have nothing to do with Rosetta. They just offer plain (Linux) VMs without the required scientific environment. ID: 105747 · Rating: 0 · rate: / Reply Quote

[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,571,020 RAC: 7,185	Message 105937 - Posted: 14 Apr 2022, 16:02:57 UTC - in response to Message 105729. The github issue regarding Step 1 has recently been added to the next BOINC client milestone. That's a good sign. Yes, is passed from "Backlog Prioritized" to "In Progress". ID: 105937 · Rating: 0 · rate: / Reply Quote

computezrmle Send message Joined: 9 Dec 11 Posts: 63 Credit: 9,680,103 RAC: 0	Message 106376 - Posted: 10 Jun 2022, 17:36:38 UTC BOINC recently published vboxwrapper 26204 which introduces the code to deal with differencing images. Downloads: https://boinc.berkeley.edu/dl/vboxwrapper_26204_windows_x86_64.zip https://boinc.berkeley.edu/dl/vboxwrapper_26204_x86_64-apple-darwin.zip https://boinc.berkeley.edu/dl/vboxwrapper_26204_x86_64-pc-linux-gnu It's now up to the Rosetta developers to create new app versions that make use of it. Hint: https://github.com/BOINC/boinc/issues/4602 ID: 106376 · Rating: 0 · rate: / Reply Quote

[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,571,020 RAC: 7,185	Message 106383 - Posted: 11 Jun 2022, 14:00:12 UTC - in response to Message 106376. BOINC recently published vboxwrapper 26204 which introduces the code to deal with differencing images. It's now up to the Rosetta developers to create new app versions that make use of it. Hint: https://github.com/BOINC/boinc/issues/4602 Great news!!! ID: 106383 · Rating: 0 · rate: / Reply Quote

Jim1348 Send message Joined: 19 Jan 06 Posts: 881 Credit: 52,257,545 RAC: 0	Message 106384 - Posted: 11 Jun 2022, 17:07:49 UTC - in response to Message 106376. That would solve a lot of problems at once. Thanks for your efforts. ID: 106384 · Rating: 0 · rate: / Reply Quote

[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,571,020 RAC: 7,185	Message 106387 - Posted: 14 Jun 2022, 7:29:51 UTC Atlas@Home introduced yesterday the new wrapper as beta test ID: 106387 · Rating: 0 · rate: / Reply Quote

[VENETO] boboviz Send message Joined: 1 Dec 05 Posts: 1994 Credit: 9,571,020 RAC: 7,185	Message 106389 - Posted: 14 Jun 2022, 15:31:47 UTC - in response to Message 106387. Last modified: 14 Jun 2022, 15:31:59 UTC Atlas@Home introduced yesterday the new wrapper as beta test Now also CMS@Home and Test4Theory have new versions with the new wrapper (in test on boinc lhc-dev site) ID: 106389 · Rating: 0 · rate: / Reply Quote