If you look around Google’s offices in Mountain View, California, you’ll see Windows machines, Chromebooks, Macs, and gLinux desktops. G what, you ask? Well, in addition to relying on Linux for its servers, Google has its own Linux desktop distro.
You can’t get it – damn it! – but for more than a decade, Google has been cooking and eating its own homemade Linux desktop distro. The first version was Goobuntu. (As you guess from the name, it was based on Ubuntu.)
In 2018, Google moved its internal Linux desktop from the Goobuntu to a new Linux distribution, the DebianNamegLinux-based. Why? Because, as Google explained, Ubuntu’s two-year Long Term Support (LTS) release “meant that we had to upgrade every machine in our fleet of over 100,000 devices before the end date. life of the operating system”.
It was a pain. Add to that the tedious need to fully customize engineers’ PCs, and Google decided it was too expensive. Also, “the effort to upgrade our Goobuntu fleet typically took a good chunk of a year. With a two-year support window, there was only one year left before we had to go through the same process again to the next LTS. This whole process has been a huge stressor for our team, as we have received hundreds of bugs with requests for help with critical cases.”
So when Google had had enough, it switched to Debian Linux (but not just vanilla Debian). The company has created a rolling Debian distribution: GLinux Rolling Debian Testing (Rodete). The idea is that users and developers are best served by giving them the latest updates and fixes as they are created and deemed production-ready. These distributions include Arch Linux, Debian-testingand openSUSE Tumbleweed.
For Google, the immediate goal was to get out of the two-year upgrade cycle. Like moving to Continuous Integration/Continuous Deployment (CI/CD) shown, these incremental changes work well. They are also easier to control and undo if something goes wrong.
To make it all work without a lot of blood, sweat, and tears, Google created a new workflow system, Sieve. Every time Sieve spots a new version of a Debian package, it starts a new build. These packages are integrated into package groups because separate packages often need to be upgraded together. Once the entire group has been created, Google runs a virtualized test suite to ensure that no core components and developer workflows are broken. Then each group is tested separately with a full system install, boot, and run of the local test suite. The package builds in minutes, but testing can take up to an hour.
Once done, all new packages are merged with the latest gLinux package pool. Then, when Google decides it’s time to put it into production, the team takes a snapshot of that pool. Finally, he deploys the new version of the fleet. Of course, that’s not just going to offload it to users. Instead, it uses Site Reliability Engineering (SRE) principles such as incremental canarying to ensure nothing goes wrong.
Over the years, Google has improved in this area. Today, thanks to Sieve, the entire gLinux development team consists of a single in-service release engineer position that rotates among team members. There are no major efforts to modernize the fleet. No multi-stage alpha, beta, and general availability (GA) releases.
Even better, with the rolling release schedule, Google can quickly fix security flaws across the entire fleet without compromising stability. Previously, security engineers had to carefully examine each Debian Security Advisory (DSA) to make sure the fix was in place.
Additionally, “Google’s improved test suite and integration testing with key partner teams that run mission-critical development systems also resulted in a more stable experience using a Linux distribution that provides the latest Linux kernel releases. Our strong desire to automate everything in the pipeline has greatly reduced the toil and stress within the team, and now it is also possible for us to report bugs and incompatibilities with other versions of the library while ensuring that Google tools work better within the Linux ecosystem.”
Going forward, the Google team said they will “work more closely with Debian upstream and contribute more to our internal patches to maintain the Debian package ecosystem.”
It all sounds good. But I have two thoughts to share.
First, for some organizations, LTS releases still make sense. If you don’t need the latest and greatest programs for your business, an Ubuntu or Red Hat LTS Linux still makes sense.
Second, and most important: Sieve sounds like a cat’s meow. A program that can automate a streaming delivery production pipeline to the point where it only takes one engineer to maintain a desktop used by over 100,000 users? Sign me up!
Better yet, release the code for Sieve so we can all start producing Linux desktop builds continuously. What about Google? What are you saying?
Copyright © 2022 IDG Communications, Inc.