Improving the boot times of IT servers with a custom BIOS

One of the big challenges with the use of IT hardware such as computer servers in broadcasting is excessively long boot times of up to ten minutes. In reactive broadcasting environments this may cause problems or be frustrating for operators. Most of the boot time in a server consists of the BIOS boot time plus the boot time of the Operating System. In our case the boot time of the OS is relatively small compared to the BIOS so this article will focus on BIOS changes. A computer BIOS (or something more modern like UEFI) is doing a very complex job setting up hardware and doing things like memory training. However, most BIOSes on servers are written by third-party vendors and arguably there is an incentive for third-party BIOS vendors to add features (that some would see as unnecessary complexity) to their BIOSes which increases boot time. This is where coreboot comes in. Coreboot is an Open Source BIOS implementation (and so much more) with a substantially smaller footprint compared to vendor BIOSes. This allows for faster boot times amongst many other benefits. But its downside is that every motherboard must be added manually, quite a challenge for a company with lots of different motherboards in use. It also only supports entry-level servers for the time being. Servers with Xeon Scalable CPUs are not supported owing to limitations from Intel. Coreboot thankfully supported the X11 series from Supermicro, and we had a similar old server in our lab, although we had an X11SSW-F, a model that was not supported. As these motherboards are similar, after some work understanding the differences between PCI Express (PCIe) slots and USB ports compared to other boards it was possible to write a patch that booted successfully. There was an obvious difference in boot time but it was time to measure. This was done by a cronjob running every five minutes and logging the date of reboot and logging the first message from the Linux kernel in syslog. Note that we were only measuring “warm” boot times and the test was run 15 times.
Vendor BIOS boot time Coreboot BIOS boot time
83±10s 38±5s
Twice as fast! It’s likely there are possible settings changes to both the Vendor BIOS and Coreboot to reduce boot time further. The bigger issue is that in both BIOSes a substantial amount of time is spent in the closed Intel Firmware Support Package that can’t be modified. We also need to check that Coreboot doesn’t affect functionality of servers, so we wrote a script to reboot the server and start up an 8-channel decoder every 30 minutes. After 730 reboots (around two weeks nonstop!), we saw from logs that every single reboot was successful and the decoders started successfully. We also have plans to add coreboot support to more modern motherboards and roll this out to customers on request. Coreboot also has some benefits for debugging PCIe related issues, but more on that another day.