Inception: VM inside Docker inside KVM – Testing Debian VM installation builds on Travis CI
Back in 2006 I started to write a tool called grml-debootstrap. grml-debootstrap is a wrapper around debootstrap for installing Debian systems. Using grml-debootstrap, it’s possible to install Debian systems from the command line, without having to boot a Debian installer ISO. This is very handy when you’re running a live system (like Grml or Tails) and want to install Debian. It’s as easy as running:
% sudo grml-debootstrap --target /dev/sda1 --grub /dev/sda
I’m aware that grml-debootstrap is used in Continuous Integration/Delivery environments, installing Debian systems several hundreds or even thousands of times each month. Over the time grml-debootstrap gained many new features. For example, since 2011 grml-debootstrap supports installation into VM images:
% sudo grml-debootstrap --vmfile --vmsize 3G --target debian.img
In 2016 we also added (U)EFI support (the target device in this example is a logical device on LVM):
% sudo grml-debootstrap --grub /dev/sdb --target /dev/mapper/debian--server-rootfs --efi /dev/sdb1
As you might imagine, every new feature we add also increases the risk of breaking something™ existing. Back in 2014, I contributed a setup using Packer to build automated machine images, using grml-debootstrap. That allowed me to generate Vagrant boxes with VirtualBox automation via Packer, serving as a base for reproducing customer environments, but also ensuring that some base features of grml-debootstrap work as intended (including backwards compatibility until Debian 5.0 AKA lenny).
The problem of this Packer setup though is, contributors usually don’t necessarily have Packer and VirtualBox (readily) available. They also might not have the proper network speed/bandwidth to run extensive tests. To get rid of those (local) dependencies and make contributing towards grml-debootstrap more accessible (we’re currently working on e.g. systemd-networkd integration), I invested some time at DebCamp at DebConf18.
I decided to give Travis CI a spin. Travis CI is a well known Continuous Integration service in the open source community. Among others, it’s providing Ubuntu Linux environments, either Container-based or as full Virtual Machines, providing us what we need. Working on the Travis CI integration, I started with enabling ShellCheck (which is also available as Debian package, BTW!), serving as lint tool for shell scripts. All of that takes place in an isolated docker container.
To be able to execute grml-debootstrap, we need to install the latest version of grml-debootstrap from Git. That’s where travis.debian.net helps us – it is a hosted service for projects that host their Debian packaging on GitHub to use the Travis CI continuous integration platform to test builds on every update. The result is a Debian package (grml-debootstrap_*.deb) which we can use for installation, ensuring that we run exactly what we will ship to users (including scripts, configuration + dependencies). This also takes place in an isolated docker instance.
Then it’s time to start a Debian/stretch docker container, installing the resulting grml-debootstrap*.deb file from the travis.debian.net container run there. Inside it, we execute grml-debootstrap with its VM installation feature, to install Debian into a qemu.img file. Via qemu-system-x86_64 we can boot this VM file. Finally, goss takes care of testing and validation of the resulting system.
The overall architecture looks like:
So Travis CI is booting a KVM instance on GCE (Google Compute Engine) for us, inside of which we start three docker instances:
- shellcheck (koalaman/shellcheck:stable)
- travis.debian.net (debian:stretch + debian:unstable, controlled via TRAVIS_DEBIAN_DISTRIBUTION)
- VM image installation + validation (debian:stretch)
Inside the debian/stretch docker environment, we install and execute grml-debootstrap. Finally we’re booting it via Qemu/KVM and running tests against it.
An example of such a Travis CI run is available at https://travis-ci.org/grml/grml-debootstrap/jobs/407751811.
Travis CI builds heavily depend on a bunch of external resources, which might result in false negatives in builds, this is something that we might improve by further integrating and using our infrastructure with Jenkins, GitLab etc. Anyway, it serves as a great base to make contributions and refactoring of grml-debootstrap easier.
Thanks to Christian Hofstaedtler + Darshaka Pathirana for for proof-reading this.