Last weekend (2010-07-02 – 2010-07-04) nine people met at the FAI developer workshop at Linuxhotel in Essen/Germany. If you can’t remember: FAI is a non-interactive system to install, customize and manage Linux systems and software configurations on computers as well as virtual machines and chroot environments, from small networks to large-scale infrastructures and clusters.
The participants of the FAI meeting:
second row from left to right: Michael Goetze, Michael Prokop, Andreas Schuldei
first row from left to right: Sebastian Hetze, Manuel Hachtkemper, Thomas Lange, Mattias Jansson
missing on the picture: Thomas Neumann (left on sunday midday) and Stephan Hermann (only part-time)
Friday afternoon started with getting to know each other, continuing with discussions all around FAI. On saturday we started to hack on FAI.
Between the hack sessions and discussions the attending people presented their FAI usage and approaches. Some notes from the presentations:
FAI Manager webfrontend / Stephan Hermann
Stephan ‘\sh’ Hermann presented his FAI web frontend which should be released under the GPL license in those days. The frontend uses qooxdoo whereas the backend is based on django, rpc4django and python-tftpy.
A demo video is available at blip.tv. Currently Stephan is searching for a nice name for his FAI management tool – please send suggestions either to him or to the linux-fai-devel mailinglist.
Grml / Michael Prokop
Grml is a Debian based Linux live system specially made for system administrators. Grml uses grml-live for building the ISOs, whereas grml-live itself uses FAI’s dirinstall feature to build the live system. This provides the Grml team with a nice way to autobuild 18 ISOs per day, known as daily.grml.org. Mika also presented Grml’s netscript bootoption and the ethdevice bootoption of live-initramfs which is useful for booting Grml/FAI via PXE.
Host Europe / Michael Goetze
Host Europe uses FAI for installing Debian and Ubuntu (32+64 bit) in the support center. They have ~20 FAI classes and use a Debian lenny NFSROOT as base for all deployed systems. Their main problems with FAI aren’t related to FAI itself, but instead e.g. broadcom NICs with lack of support for it in Lenny’s kernel. They are not using softupdate (yet) and currently use Kickstart for deploying CentOS but are working on deploying CentOS with FAI as well.
LIS AG / Sebastian Hetze
Linux Information Systems AG (LIS AG) are using FAI 3.2.17 and provide a luma and PyQt based GUI to their customers. They use DHCP, LDAP and DDNS for inventory, configuration and deployment.
Mathematical Institute of the University of Bonn / Manuel Hachtkemper
The Mathematical Institute of the University of Bonn uses FAI 3.1.8 and 3.3.5 for managing ~150 systems. They are automatically running softupdates every day, reporting how many hosts actually did run the softupdate and how many didn’t run. The involved failogwatch tool supports two regex files, one for excluding specific hosts and the other one for grepping for known problems in the logs.
Spotify / Andreas Schuldei + Mattias Jansson
Spotify is a peer-to-peer music streaming service and the operating people at Spotify use FAI for deploying the systems. Currently they are using FAI 3.3.3 to deploy ~400 bare metal machines and ~150 virtualised machines. They have their class names in DNS using the txt/Text record entry. They are using a self written prepend_class script to manage dependencies between classes.
University Köln / Thomas Lange
Thomas uses FAI’s trunk version (of course :)), managing ~25 machines with less than 20 FAI classes. He’s not using softupdates as Lenny’s aptitude ignores the hold status of packages (this bug should be fixed for Squeeze).
$COMPANY
One of the big telecommunication providers in Germany uses FAI 3.3.3 for installing their bare-metal and virtual servers, providing Debian, Ubuntu and SLES. They are using Debian NFSROOT as a base for all systems as well and their main problems with FAI wasn’t FAI itself but how to manage installation of virtual machines.
On Saturday evening we had a nice barbecue which included beer and Kölsch *d&r*. ;) On Sunday we continued with discussions and development.
Our work-log of the weekend:
- identified important packages for the Debian/squeeze release
- discussed features that should be available (important packages and bugreports, missing features,…)
- fixed several bugs (wrong exit codes, error handling, variable handling,…)
- discussed FAI packaging for Ubuntu (packages for lucid are available at ppa at launchpad)
- implemented support for grub2 in fai-cd (will be merged soon)
- implemented initial support for retrieving sources with FAI (available in svn’s trunk)
Important decisions made:
- next major release will have version number 4.x
- we want to continue to provide a stable version 3.x (no new features, just bugfixes and maintenance) side-by-side with version 4.x
- deprecated setup-harddisk will be dropped, setup-storage is well established, works fine and is properly maintained by FAI developer Michael Tautschnig (who sadly couldn’t attend the FAI developer meeting)
- a FAQ section will be created on the FAI homepage
We noticed that many FAI users implement their own way how to handle dependency management between classes, we will re-consider how we could provide such a mechanism through FAI’s core. We also noted that it’s important that any self-written scripts used within FAI are fully idempotent and users should be aware of this.
Last but not least – many thanks to the sponsors of the FAI developer workshop 07/2010! The workshop wouldn’t have been possible without our generous sponsors, namely being:
