Debian Lenny PXE Installation on Dell PowerEdge 1950/2950 servers: bnx2 annoyances

Debian Lenny installer brings in several improvements over the old stable Etch version, but for us this has been a big disappointment. Basically as I mentioned in an older post “bnx2 missing from stock Debian 2.6.24 etchnhalf kernel”, starting with kernel 2.6.24, the debian team removed the bnx2 kernel module out of the kernel due to license restrictions placed on the firmware. I am a long time debian fan and I can fully understand the reason why they have done that, but still, if you have several hundred servers that use the Broadcom NetXtreme II NIC (that uses the bnx2 module) then you have a problem. This post is intended to show how we solved this problem and hopefully help others in the same situation.

As I mentioned we have many Dell PE1950 and PE2950 servers and we deploy all new servers using PXE boot install and use preseeding to setup basic configurations (custom partitions, apt mirrors, etc.). This worked fine for some years now with debian Etch; let’s see what happens with lenny installer: after the initial pxe boot, the installation halts with this error:

This tells us that the network hardware needs non-free firmware files to operate, and asks us to load them using a USB stick or a floppy. Even though this solution is out of the question because we could not automate something like that, we did try to use the DRAC virtual media to load the firmware but that didn’t work either. We are performing the installation over a remote console (using a DRAC card) and we are not near the servers while performing the installation, as this is mostly automated. But let’s assume that if we can put the firmware on a floppy (?!) or USB stick we should be able to move on with the installation. Since this was not acceptable for us in the first place we had to find a solution to this issue.

Adding bnx2 firmware to the installer initrd

Debian has removed the firmware from the initrd, but it is possible for anyone to add it back in and have it working again. For this we have to uncompress the initrd.gz, add the bnx2 firmware and save the result back as initrd. Here is an older post that I wrote describing how we can edit the initramfs debian uses.

We uncompress the image using:

mkdir temp
cd temp/
zcat ../initrd.gz | cpio -iv

Download the firmware-bnx2 debian package:

cd ..
wget http://http.us.debian.org/debian/pool/non-free/f/firmware-nonfree/firmware-bnx2_0.14_all.deb

and extract the contents with:

dpkg-deb -x firmware-bnx2_0.14_all.deb bnx2

We need to copy the files under lib/firmware/bnx2-06-4.0.5.fw , bnx2-09-4.0.5.fw and bnx2/usr/share/initramfs-tools/hooks/firmware_bnx2 inside the extracted directory of the initrd and compress it back (from the extracted directory):

find . -print0 | cpio -0 -H newc -ov | gzip -c > ../initrd.gz

Automatically install firmware-bnx2

After this, a new pxe boot using the resulted initrd.gz image will allow the installer to set up the network interface and move on with the installation. Please note that the firmware-bnx2 package will not automatically be installed by the installer leaving the installed system with a problem as it can’t use the bnx2 network interface to go out on the network (and eventually install the package afterwards). So the best way to do this is to install the firmware-bnx2 package during installation; this can be done including in the preseeding file:

d-i pkgsel/include string openssh-server firmware-bnx2

This should install the firmware-bnx2 package automatically and have the system work on the network once it finishes the installation.

Disabling usb-storage in the installer

Even if this is not related to bnx2, another annoyance we encountered was caused by the way the hard drives were detected by the installer. Sometimes this is different from the finished installed system, changing sda to sdb and so on. For example in our preseeding file we would partition and format the first drive, and this would fail like this:

Looking at the console of the installer this shows that sda was detected as scsi removable disk (USB DRAC stuff), while the actual system disk was detected as sdb:

I am sure other people use usb storage during installation ;-) but we don’t need that at all, so I just chose the simplest solution to remove that completely from the initrd (I had one try to disable it using blacklist modules but that didn’t work). From the extracted content of the initrd I removed:

rm -rf lib/modules/2.6.26-1-amd64/kernel/drivers/usb/storage

and created the initrd without the module:

find . -print0 | cpio -0 -H newc -ov | gzip -c > ../initrd.gz

Now the lenny installer is able to run ok on our Dell servers. Hopefully this will be useful to others in the same situation. You can also download the resulted initrd using the above method from here (this is for amd64; if anyone needs it for i386 let me know and I will upload it also).

Update: as requested by email, here is also a i386 initrd image. It should work just fine, but I haven’t tested this as I don’t have any i386 installations available for this. Let me know if all worked as expected…

Update 20090518: apparently several people used the resulted initrd’s, and after lenny’s 5.0.1 upgrade these were no longer working (because of kernel update in the installer). I have fixed this and uploaded updated images for both i386 and amd64. Let me know if you encounter any problems by using them.