And you?

Agenda

• Valuable tools + services in Debian
• Common causes in failing systems
• Lessons Learned

Not topic of this talk

Disaster + Security Incidents?

• Disasters due to poor system maintenance (e.g. service downtime)
• Disasters because of security incident
• Disasters because something™️ went wrong
• Reasons that are common when such disasters happen?

Incident related

• debsums (check MD5 sums of installed Debian packages)
• dpkg -V (verify integrity of packages)
• dpkg --audit (perform database sanity + consistency checks)
• busybox-static (statically linked shell that provides BusyBox utilities)
• security-tracker.debian.org (check CVE/package state)
• debian-security (Debian security support coverage checker)

Misc tools + services

• apt changelog $PKG
• bugs.debian.org + bts [show --mbox $BUG]
• codesearch.debian.net
• debcheckout $PKG
  (side note: sadly still not all
  packages available via VCS)

snapshot.debian.org

% debsnap -a amd64 bash 5.1-2+deb11u1
% dpkg-deb -e binary-bash/bash_5.1-2+deb11u1_amd64.deb extract
% grep -w bin/bash extract/md5sums
c283d8a2688baef860c41c49b79a82db  bin/bash

% wget https://snapshot.debian.org/archive/debian/20050312T000000Z/pool/main/g/gcc/libstdc%2B%2B2.10_2.95.2-14_i386.deb
[…]
% dpkg-deb -e libstdc++2.10_2.95.2-14_i386.deb extract
% grep -w usr/lib/libstdc++-3-libc6.1-2-2.10.0.so extract/md5sums
d20155b428849e6fdd12f8d63e7469dc  usr/lib/libstdc++-3-libc6.1-2-2.10.0.so

sources.debian.org

% sha256sum /etc/crontab
e25eb0e3303ea63ac7452139be788efa86d89a11005288e7150c0ac193d7e1f6  /etc/crontab
% curl -s 'https://sources.debian.org/api/sha256/?checksum=e25eb0e3303ea63ac7452139be788efa86d89a11005288e7150c0ac193d7e1f6'| jq .
{
  […]
    {
      "package": "cron",
      "path": "debian/crontab.main",
      "percent_encoded_path": "debian/crontab.main",
      "version": "3.0pl1-197"
    },
    {
      "package": "cron",
      "path": "debian/crontab.main",
      "percent_encoded_path": "debian/crontab.main",
      "version": "3.0pl1-209"
    }
  ],
  "sha256": "e25eb0e3303ea63ac7452139be788efa86d89a11005288e7150c0ac193d7e1f6"
}

Monitoring

• Systems / services not properly monitored
• Systems missing in monitoring (shadow IT)
• No monitoring at all (🤡 IT)

Outdated software

• no longer getting (upstream) support
• … no longer getting security fixes
• … hard to get fixes backported

Insecure configuration

• open network ports (esp. unexpected ones)
• unrestricted access
• usage of weak passwords
• root login via SSH (esp. password based)
• lack of hardening

Not following best practices

• lack of backups (or untested ones)
• lack of network separation
• no principle of least privilege
• no central logging
• no inventory system
• Some common situations of Debian
  users running into problems?

sources.list configuration

Mixing suites, e.g. using stable-security
on old-stable system (AKA #1137002):

This site holds security updates for Debian GNU/Linux.
If you are using apt you can use this entry in /etc/apt/sources.list:
deb http://security.debian.org/debian-security stable-security main

Expired repository keys

user@failure ~ % sudo apt update
[…]
Err:5 https://demo.example.org/custom trixie InRelease
  The following signatures were invalid: EXPKEYSIG BEFORE1FAILS2342 Automatic Signing Key <demo@example.org>
[…]
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: http://demo.example.org/custom trixie InRelease: The following signatures were invalid: EXPKEYSIG BEFORE1FAILS2342 Automatic Signing Key <demo@example.org>
W: Failed to fetch https://demo.example.org/custom/dists/trixie/InRelease  The following signatures were invalid: EXPKEYSIG BEFORE1FAILS2342 Automatic Signing Key <demo@example.org>
W: Some index files failed to download. They have been ignored, or old ones used instead.
user@failure ~ % echo $?
0
user@failure ~ %

Running out of disk space

user@gitlab ~ % sudo apt update
[...]
Get:2 https://security.debian.org/debian-security trixie-security InRelease [43.4 kB]
Err:2 https://security.debian.org/debian-security trixie-security InRelease
  Error writing to file - write (28: No space left on device) [IP: 151.101.194.132 443]
[...]
All packages are up to date.
[...]
Warning: Failed to fetch https://security.debian.org/debian-security/dists/trixie-security/InRelease  Error writing to file - write (28: No space left on device) [IP: 151.101.194.132 443]
Warning: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ce/debian/dists/trixie/InRelease  Error writing to file - write (28: No space left on device) [IP: 172.64.148.245 443]
Warning: Some index files failed to download. They have been ignored, or old ones used instead.
user@gitlab ~ % echo $?
0
user@gitlab ~ %

Upstream ≠ best practices

• Upstream needs to follow our changes (incl. config mgmt)
• No established way to announce (un)planned key changes
• e.g. grafana.com/blog/grafana-security-update-gpg-signing-key-rotation/
• Not everyone has proper monitoring in place…
• … usually failures only noticed once it’s too late

apt-key removal

• apt-key got removed with trixie
• we don’t provide an interface to list active
  (or deprecated) keys
• go through wiki.debian.org/SecureApt,
  trying to understand current state

SHA1 deprecation

SHA1 is not considered secure since 2026-02-01T00:00:00Z

% cat /etc/crypto-policies/back-ends/apt-sequoia.config
[…]
[hash_algorithms]
sha1.second_preimage_resistance = 2026-07-23  # Extend expiry for legacy repos
sha224 = 2026-07-23
[…]

• wiki.debian.org/Teams/Apt/Sha1Removal
• wiki.debian.org/FixKeyWithSha1

Packages from Backports

• Backport packages (usually) lack security support
• Package(s) might disappear from backports
• People are not always aware (or ignore it)
• People might not notice that they might be using unsupported packages

3rd party repositories

• other policies + standards than Debian
• problems related to apt setup + apt-pinning
• conflicting configuration / package files
• unexpected behavior in maintainer scripts

Cleanup of packages

Identify packages without any corresponding repository:

aptitude --disable-columns -F '%p' search '?obsolete'
aptitude --disable-columns -F '%p' search '?narrow(?installed, ?not(?origin(Debian)))'
aptitude --disable-columns -F '%p' search '?narrow(?not(?archive("^[^n][^o][^w].*$")),?version(CURRENT))'
apt list -oapt::cmd::use-format=1 -oapt::cmd::format='${Package}' '~o'
apt-show-versions | grep -v 'uptodate$'
apt-show-versions | grep 'No available'
apt-forktracer

Package tracking

Once a repository gets disabled in apt’s sources.list,
we can no longer tell from which repository an
installed package was coming from

Running out of disk space 1/2

• /boot not always big enough for kernel updates
• problem esp. for systems installed pre-trixie with
  d-i defaults (see #972396)
• Kernel images + initrd getting larger:

-rw-r--r-- 1 root root 27M Jul 18 11:23 /boot/initrd.img-5.10.0-45-amd64
-rw-r--r-- 1 root root 29M Jul 18 11:23 /boot/initrd.img-6.1.0-51-amd64
-rw-r--r-- 1 root root 35M Jul 18 11:23 /boot/initrd.img-6.12.95+deb13-amd64

Running out of disk space 2/2

Or even:

-rw-r--r-- 1 root root 117M Jul 18 11:23 /boot/initrd.img-6.12.95+deb13-amd64

• apt autoremove|autopurge greatly improved the situation!

• apt upgrade nowadays provides useful information:

    Space needed: 188 MB / 797 MB available
    └─ in /boot:  102 MB / 621 MB available

Lack of automated upgrades

• EOL systems with lack of security updates amongst biggest problems
• especially if not properly hardened…
• … more often than not (Securing Debian Manual dates back to <=2017)
• unattended-upgrades not installed + active by default
• debian-security-support not installed by default
• no firewall/packet filter installed + active by default

Standards

• CRA (Cyber Resilience Act)
• NIS2 Directive (network and information security)
• ETSI (European Telecommunications Standards Institute)
• EN 304 626 (Essential cybersecurity requirements for operating systems)
• Debian updates are left to the user for our server based installations

Limitations of unattended-upgrades

• default configuration not ready for external repos
• no coordination amongst group of systems
• failing upgrade on one host?
• further hosts continue and might cause failures

Missing service restarts

• Package update with fix gets installed
• … but service restart might be missing
• Result? Fixed package present but unfixed software being active
• needrestart not installed + active by default

Symbol changes + service reloads

AKA #1108939:

OMD[demo]:/opt/omd/sites/demo/var/log/apache$ cat error_log.1
[…]
apache2: Syntax error on line 126 of /omd/sites/demo/etc/apache/apache.conf: Cannot load /omd/sites/demo/lib/apache/modules/mod_headers.so into server: /omd/sites/demo/lib/apache/modules/mod_headers.so: undefined symbol: ap_set_content_type_ex

% cat /etc/logrotate.d/apache2
/var/log/apache2/*.log {
[…]
        postrotate
                if pgrep -f ^/usr/sbin/apache2 > /dev/null; then
                        invoke-rc.d apache2 reload 2>&1 | logger -t apache2.logrotate
                fi
        endscript
}

Announce EOL

We do not report

… or information like from
debian-security-support
in MOTD or alike

MOTD of Debian

MOTD of Ubuntu

Init + service scripts

• init scripts vs systemd service
• upgrades enabling systemd socket activation
  unconditionally (see e.g. #1141976)
• automatic startup of services after installation
  (might also be unconfigured, forgotten,…)

Switching network stack 1/2

# dpkg -l ifupdown\* | grep '^[ri]'
ii  ifupdown       0.8.44+deb13u1 amd64        high level tools to configure network interfaces
# apt install ifupdown2
[…]
REMOVING:
  ifupdown
[…]
# find /etc/ -name networking\*
/etc/systemd/system/network-online.target.wants/networking.service
/etc/systemd/system/basic.target.wants/networking.service
/etc/systemd/system/multi-user.target.wants/networking.service
/etc/systemd/system/shutdown.target.wants/networking.service
/etc/systemd/system/network.target.wants/networking.service
/etc/default/networking
/etc/init.d/networking
#

Switching network stack 2/2

dh_installsystemd kicking in on cleanup:

# dpkg -l ifupdown\* | grep '^[ri]'
rc  ifupdown       0.8.44+deb13u1 amd64        high level tools to configure network interfaces
ii  ifupdown2      3.0.0-1.3      all          Network Interface Management tool similar to ifupdown
# dpkg --purge ifupdown
(Reading database ... 6924 files and directories currently installed.)
Purging configuration files for ifupdown (0.8.44+deb13u1) ...
# find /etc/ -name networking\*
/etc/default/networking
#

Result? No network after reboot.

Warning and error messages

• people ignore warning messages + problems
• … or don’t understand them
• … or don’t report them

BTW

Recommendations

• provide actionable error messages
• test error paths
• have backwards and forwards compatibility in mind
• behavior change testing
• provide good defaults

Change

“If we want things to stay as they are, things will have to change.”

Machine readable output

• lack of machine-readable output in tools
  (support e.g. XXX status --json)
• gap between what we implement +
  (think) how our users use it in practice

Integration in monitoring

• upstream projects provide monitoring checks
• … which are incomplete
• … without users being aware (e.g. mk_apt)
• users (forced to) implement each their own monitoring scripts
  (see e.g. dsa-check-packages)
• … with other bugs/flaws

General

• we tend to have too many options
  instead of focusing on a single/few
• improving our websites / docs /
  wiki feels harder than necessary
• tools are often implemented + changed
  without having other users in mind
• lack of documentation upfront

Teams

• different people/teams/… not talking
  to each other (often enough)…
• NIMBY (not in my backyard): instead of
  resolving problems problems are
  displaced onto others

Cognitive biases

• Turkey illusion: our IT works perfectly fine!
• Maslow’s hammer: use known tool while different one would be better
• IKEA effect: one places a disproportionately high value on things one created
• Herd mentality: tendency to orient oneself towards the majority
• Status quo bias: change is perceived as a loss