Jenkins on-demand slave selection through labels
Problem description: One of my customers had a problem with their Selenium tests in the Jenkins continuous integration system. While Perl’s Test::WebDriver still worked just fine the Selenium tests using Ruby’s selenium-webdriver suddenly reported failures. The problem was caused by Debian wheezy’s upgrade of the Iceweasel web browser. Debian originally shipped Iceweasel version 17.0.10esr-1~deb7u1 in wheezy, but during a security-update version 24.3.0esr-1~deb7u1 was brought in through the wheezy-security channel. Because the selenium tests are used in an automated fashion in a quite large and long-running build pipeline we immediately rolled back to Iceweasel version 17.0.10esr-1~deb7u1 so everything can continue as expected. Of course we wanted to get the new Iceweasel version up and running, but we didn’t want to break the existing workflow while working on it. This is where on-demand slave selection through labels comes in.
Basics: As soon as you’re using Jenkins slaves you can instruct Jenkins to run a specific project on a particular (slave) node. By attaching labels to your slaves you can also use a label instead of a specific node name, providing more flexibility and scalability (to e.g. avoid problems if a specific node is down or you want to scale to more systems). Then Jenkins decides which of the nodes providing the according label should be considered for job execution. In the following screenshot a job uses the ‘selenium’ label to restrict its execution to the slaves providing selenium and currently there are two nodes available providing this label:
TIP 1: Visiting $JENKINS_SERVER/label/$label/ provides a list of slaves that provide that given $label (as well as list of projects that use $label in their configuration), like:
TIP 2: Execute the following script on $JENKINS_SERVER/script to get a list of available labels of your Jenkins system:
import hudson.model.*
labels = Hudson.instance.getLabels()
labels.each{ label -> println label.name }
Solution: In the according customer setup we’re using the swarm plugin (with automated Debian deployment through Grml’s netscript boot option, grml-debootstrap + Puppet) to automatically connect our Jenkins slaves to Jenkins master without any manual intervention. The swarm plugin allows you to define the labels through the -labels command line option.
By using the NodeLabel Parameter plugin we can configure additional parameters in Jenkins jobs: ‘node’ and ‘label’. The ‘label’ parameter allows us to execute the jobs on the nodes providing the requested label:
This is what we can use to gradually upgrade from the old Iceweasel version to the new one by keeping a given set of slaves at the old Iceweasel version while we’re upgrading other nodes to the new Iceweasel version (same for the selenium-server version which we want to also control). We can include the version number of the Iceweasel and selenium-server packages inside the labels we announce through the swarm slaves, with something like:
if [ -r /etc/init.d/selenium-server ] ; then
FLAGS="selenium"
ICEWEASEL_VERSION="$(dpkg-query --show --showformat='${Version}' iceweasel)"
if [ -n "$ICEWEASEL_VERSION" ] ; then
ICEWEASEL_FLAG="iceweasel-${ICEWEASEL_VERSION%%.*}"
EXTRA_FLAGS="$EXTRA_FLAGS $ICEWEASEL_FLAG"
fi
SELENIUM_VERSION="$(dpkg-query --show --showformat='${Version}' selenium-server)"
if [ -n "$SELENIUM_VERSION" ] ; then
SELENIUM_FLAG="selenium-${SELENIUM_VERSION%-*}"
EXTRA_FLAGS="$EXTRA_FLAGS $SELENIUM_FLAG"
fi
fi
Then by using ‘-labels “$FLAGS EXTRA_FLAGS”‘ in the swarm invocation script we end up with labels like ‘selenium iceweasel-24 selenium-2.40.0’ for the slaves providing the Iceweasel v24 and selenium v2.40.0 Debian packages and ‘selenium iceweasel-17 selenium-2.40.0’ for the slaves providing Iceweasel v17 and selenium v2.40.0.
This is perfect for our needs, because instead of using the “selenium” label (which is still there) we can configure the selenium jobs that should continue to work as usual to default to the slaves with the iceweasel-17 label now. The development related jobs though can use label iceweasel-24 and fail as often as needed without interrupting the build pipeline used for production.
To illustrate this here we have slave selenium-client2 providing Iceweasel v17 with selenium-server v2.40. When triggering the production selenium job it will get executed on selenium-client2, because that’s the slave providing the requested labels:
Whereas the development selenium job can point to the slaves providing Iceweasel v24, so it will be executed on slave selenium-client1 here:
This setup allowed us to work on the selenium Ruby tests while not conflicting with any production build pipeline. By the time I’m writing about this setup we’ve already finished the migration to support Iceweasel v24 and the infrastructure is ready for further Iceweasel and selenium-server upgrades.