Volunteering free build system capacity for science

One early morning a couple of weeks back, a question appeared on our company-internal chat:

Do we want to harness the idle CPU time of our hosts to benefit understanding the COVID-19 virus? Our CI hosts alone would throw in 2640 cores into the pool. https://foldingathome.org/2020/03/15/coronavirus-what-were-doing-and-how-you-can-help-in-simple-terms/

We have a long-standing tradition of initiatives coming from R&D. Indeed, this was no exception. The proposal came from Tony Sarajärvi, a senior engineer on our Continuous Integration team. Within hours, several people were involved in the discussion: SVP of R&D, Tuukka Turunen, asking for an analysis of the impact to our operational capacity; Marko Möttönen, CIO, confirming that the increase in power consumption would be covered by ITs budget. Several engineers trying to uncover any unforeseen pitfalls. By lunch-time the next day, Tuukka gave the all-clear:

Let's start.

Idle CPU time?

Imagine a Formula One race track. Now imagine that there are cars on the track, in idle, waiting for the next race to happen. That's our CI when no one is using it. The basic idea is to let the cars drive even when no one's watching. In other words, let our CI system's CPUs crunch numbers when no one uses it. Okay, it isn't exactly like that, but I think the imagery serves its purpose. For the sake of simplicity, let's say we're trying to help research projects that are looking for a needle in a haystack, by reducing the size of the haystack.

Initially, we started with three build hosts and one formerly retired blade server, totaling 6480 core hours per day. This is a part of our old CI infrastructure and consist of second generation Intel Core processors. The new hardware consists of 33 Dell R430 rack mounted servers with 5th generation Intels. They are dual socket machines with 2 20-core CPUs each, due to hyperthreading doubling the CPU count, it looks like 2x40. Later on, more of those retired blade servers were added to the mix.

This hardware is there to run tests for open source commits that drop by the dozens every day. We create 1.4 million virtual machines per year to test these changes. Depending on our own load, we could exceed 70000 core-hours per day when harnessing the unused CPU capacity of our CI system.

Tony explained how this differs from a typical gaming rig: "Fun fact: One of the 80 core build hosts consumes 112 Watts of power when idling. It consumed 350 Watts when it crunched the numbers. It went through the packet it got in approx 10-12 minutes. I don't know if all of them are of the same size, but my desktop at home crunched some numbers for 6 hours."

We saw the reports that Folding@Home had troubles keeping up with the surge of volunteers, so Tony also set us up to support both Folding@Home and Rosetta@Home. On the following weekend, all of our CI infrastructure participated in folding.

This is how you get started

Rosetta@Home sent out a thank you tweet when they learned that we had joined their program, and others replied that they were considering doing the same. My hope is that this will help you make the leap and join one (or both) of the largest high-performance computing platforms in the world. Installing the client and setting it up on a single computer takes a couple of minutes at most. Deploying to a more complex multi-host system takes a bit more time, but is easily scriptable. The client runs on all the major operating systems, Windows, macOS, and Linux (with packages available for multiple distributions).

You don't need a CI system or a server farm to contribute. Some of the engineers in the Qt Company are participating using their laptops. I suggest you take a look at the websites of the two respective projects, and join the one where you believe you will make the most impact. For a single computer, I'm pretty sure participating in one suffices. If you want to make use of your kick-ass GPU, it looks like Folding@Home is your best option. If you have a few servers set up as build hosts, for example, perhaps you want to consider joining both.

Folding@Home

• Website: https://foldingathome.org/
• Wikipedia entry: https://en.wikipedia.org/wiki/Folding@home
• Our team: https://stats.foldingathome.org/team/242435

BOINC

• Website: https://boinc.bakerlab.org/
• Wikipedia entry: https://en.wikipedia.org/wiki/Berkeley_Open_Infrastructure_for_Network_Computing
• Our team: https://boinc.bakerlab.org/rosetta/team_members.php?teamid=19061

Wrapping up

Witnessing the response to this challenge from within offered me an opportunity to reflect on how I feel about being a part of our team. It feels good to know that we're doing something, especially in this situation where, frankly, it's hard to see what anyone can do. But mostly, I experienced a profound sense of gratefulness for being part of an organization that can respond so fast to an idea from the "bottom", without the need to cut through red tape.

Want to know how it feels? Why don't you join one of the projects and try it on for size?