Some useful bits about Linux hardware support and patched Kernel packages
Disclaimer: I started writing this blog post in May 2018, when Debian/stretch was the current stable release of Debian, but published this article in August 2019, so please keep the version information (Debian releases + kernels not being up2date) in mind.
The kernel version of Debian/stretch (4.9.0) didn’t support the RAID controller as present in Lenovo ThinkSystem SN550 blade servers yet. The RAID controller was known to be supported with Ubuntu 18.10 using kernel v4.15 as well as with Grml ISOs using kernel v4.15 and newer. Using a more recent Debian kernel version wasn’t really an option for my customer, as there was no LTS kernel version that could be relied on. Using the kernel version from stretch-backports could have be an option, though it would be our last resort only, since the customer where this applied to controls the Debian repositories in usage and we’d have to track security issues more closely, test new versions of the kernel on different kinds of hardware more often,… whereas the kernel version from Debian/stable is known to be working fine and is less in a flux than the ones from backports. Alright, so it doesn’t support this new hardware model yet, but how to identify the relevant changes in the kernel to have a chance to get it supported in the stable Debian kernel?
Some bits about PCI IDs and related kernel drivers
We start by identifying the relevant hardware:
root@grml ~ # lspci | grep 'LSI.*RAID' 08:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID Tri-Mode SAS3404 (rev 01) root@grml ~ # lspci -s '08:00.0' 08:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID Tri-Mode SAS3404 (rev 01)
Which driver gets used for this device?
root@grml ~ # lspci -k -s '08:00.0'
08:00.0 RAID bus controller: LSI Logic / Symbios Logic MegaRAID Tri-Mode SAS3404 (rev 01)
Subsystem: Lenovo ThinkSystem RAID 530-4i Flex Adapter
Kernel driver in use: megaraid_sas
Kernel modules: megaraid_sas
So it’s the megaraid_sas driver, let’s check some version information:
root@grml ~ # modinfo megaraid_sas | grep version version: 07.703.05.00-rc1 srcversion: 442923A12415C892220D5F0 vermagic: 4.15.0-1-grml-amd64 SMP mod_unload modversions
But how does the kernel know which driver should be used for this device? We start by listing further details about the hardware device:
root@grml ~ # lspci -n -s 0000:08:00.0 08:00.0 0104: 1000:001c (rev 01)
The 08:00.0 describes the hardware slot information ([domain:]bus:device.function), the 0104 describes the class (with 0104 being of type RAID bus controller, also see /usr/share/misc/pci.ids by searching for ‘C 01’ -> ’04`), the (rev 01) obviously describes the revision number. We’re interested in the 1000:001c though. The 1000 identifies the vendor:
% grep '^1000' /usr/share/misc/pci.ids 1000 LSI Logic / Symbios Logic
The `001c` finally identifies the actual model. Having this information available, we can check the mapping of the megaraid_sas driver, using the `modules.alias` file of the kernel:
root@grml ~ # grep -i '1000.*001c' /lib/modules/$(uname -r)/modules.alias alias pci:v00001000d0000001Csv*sd*bc*sc*i* megaraid_sas root@grml ~ # modinfo megaraid_sas | grep -i 001c alias: pci:v00001000d0000001Csv*sd*bc*sc*i*
Bingo! Now we can check this against the Debian/stretch kernel, which doesn’t support this device yet:
root@stretch:~# modinfo megaraid_sas | grep version version: 06.811.02.00-rc1 srcversion: 64B34706678212A7A9CC1B1 vermagic: 4.9.0-6-amd64 SMP mod_unload modversions root@stretch:~# modinfo megaraid_sas | grep -i 001c root@stretch:~#
No match here – bingo²! Now we know for sure that the ID 001c is relevant for us. How do we identify the corresponding change in the Linux kernel though?
The file drivers/scsi/megaraid/megaraid_sas.h of the kernel source lists the PCI device IDs supported by the megaraid_sas driver. Since we know that kernel v4.9 doesn’t support it yet, while it’s supported with v4.15 we can run "git log v4.9..v4.15 drivers/scsi/megaraid/megaraid_sas.h" in the git repository of the kernel to go through the relevant changes. It’s easier to run "git blame drivers/scsi/megaraid/megaraid_sas.h" though – then we’ll stumble upon our ID from before – `0x001C` – right at the top:
[...] 45f4f2eb3da3c (Sasikumar Chandrasekaran 2017-01-10 18:20:43 -0500 59) #define PCI_DEVICE_ID_LSI_VENTURA 0x0014 754f1bae0f1e3 (Shivasharan S 2017-10-19 02:48:49 -0700 60) #define PCI_DEVICE_ID_LSI_CRUSADER 0x0015 45f4f2eb3da3c (Sasikumar Chandrasekaran 2017-01-10 18:20:43 -0500 61) #define PCI_DEVICE_ID_LSI_HARPOON 0x0016 45f4f2eb3da3c (Sasikumar Chandrasekaran 2017-01-10 18:20:43 -0500 62) #define PCI_DEVICE_ID_LSI_TOMCAT 0x0017 45f4f2eb3da3c (Sasikumar Chandrasekaran 2017-01-10 18:20:43 -0500 63) #define PCI_DEVICE_ID_LSI_VENTURA_4PORT 0x001B 45f4f2eb3da3c (Sasikumar Chandrasekaran 2017-01-10 18:20:43 -0500 64) #define PCI_DEVICE_ID_LSI_CRUSADER_4PORT 0x001C [...]
Alright, the relevant change was commit 45f4f2eb3da3c:
commit 45f4f2eb3da3cbff02c3d77c784c81320c733056
Author: Sasikumar Chandrasekaran […]
Date: Tue Jan 10 18:20:43 2017 -0500
scsi: megaraid_sas: Add new pci device Ids for SAS3.5 Generic Megaraid Controllers
This patch contains new pci device ids for SAS3.5 Generic Megaraid Controllers
Signed-off-by: Sasikumar Chandrasekaran […]
Reviewed-by: Tomas Henzl […]
Signed-off-by: Martin K. Petersen […]
diff --git a/drivers/scsi/megaraid/megaraid_sas.h b/drivers/scsi/megaraid/megaraid_sas.h
index fdd519c1dd57..cb82195a8be1 100644
--- a/drivers/scsi/megaraid/megaraid_sas.h
+++ b/drivers/scsi/megaraid/megaraid_sas.h
@@ -56,6 +56,11 @@
#define PCI_DEVICE_ID_LSI_INTRUDER_24 0x00cf
#define PCI_DEVICE_ID_LSI_CUTLASS_52 0x0052
#define PCI_DEVICE_ID_LSI_CUTLASS_53 0x0053
+#define PCI_DEVICE_ID_LSI_VENTURA 0x0014
+#define PCI_DEVICE_ID_LSI_HARPOON 0x0016
+#define PCI_DEVICE_ID_LSI_TOMCAT 0x0017
+#define PCI_DEVICE_ID_LSI_VENTURA_4PORT 0x001B
+#define PCI_DEVICE_ID_LSI_CRUSADER_4PORT 0x001C
[...]
Custom Debian kernel packages for testing
Now that we identified the relevant change, what’s the easiest way to test this change? There’s an easy way how to build a custom Debian package, based on the official Debian kernel but including further patch(es), thanks to Ben Hutchings. Make sure to have a Debian system available (I was running this inside an amd64 system, building for amd64), with according deb-src entries in your apt’s sources.list and enough free disk space, then run:
% sudo apt install dpkg-dev build-essential devscripts fakeroot % apt-get source -t stretch linux % cd linux-* % sudo apt-get build-dep linux % bash debian/bin/test-patches -f amd64 -s none 0001-scsi-megaraid_sas-Add-new-pci-device-Ids-for-SAS3.5-.patch
This generates something like a linux-image-4.9.0-6-amd64_4.9.88-1+deb9u1a~test_amd64.deb for you (next to further Debian packages like linux-headers-4.9.0-6-amd64_4.9.88-1+deb9u1a~test_amd64.deb + linux-image-4.9.0-6-amd64-dbg_4.9.88-1+deb9u1a~test_amd64.deb), ready for installing and testing on the affected system. The Kernel Handbook documents this procedure as well, I just wasn’t aware of this handy `debian/bin/test-patches` so far though.
JFTR: sadly the patch with the additional PCI_DEVICE_ID* was not enough (also see #900349), we seem to need further patches from the changes between v4.9 and v4.15, though this turned up to be no longer relevant for my customer and it’s also working with Debian/buster nowadays.