2015-08-28 Librem 13: Weekly BIOS Update

Author: Larry.Moberg@puri.sm

This post gives some details on the Librem 15 rev2 prototype. One challenge with developing BIOS is finding parts that can be reused; coreboot makes heavy reuse of certain pieces of code.

Very Similar

Starting with the chips on the back of the mainboard, the 15 prototype uses the same ene KB3930QF-A1 as the Librem 13, and it is configured to read from an external Macronix MX25L512 SPI flash for firmware. The 15 has this SPI chip on the front side of the board between the DIMMs and the USB3 ports.

The 15 prototype uses a Realtek ALC269 codec via the AC’97. The Librem 13 should have a very similar codec.

On the front (the side visible just by removing the back laptop case), the 15 prototype uses an MX25L6406E SPI flash for the BIOS. The Librem 13 prototype uses a GD25Q64B, but other than the Intel Firmware Descriptor fields for JEDEC ID etc, these chips are interchangeable.

Both CPUs are Broadwell-U. They use the same FSP. They both have the LPC bus exposed on pads.

These similarities help us by reducing the amount of variation between the board subdirs in coreboot and can use the same development rig.

Acceptance Test Matrix

We’ve put together the following tests to validate coreboot builds:

  1. Cold boot: memory controller works.
  2. Cold boot: all installed DRAM is online.
  3. Cold boot: graphics controller works.
  4. Cold boot: SATA controller succeeds.
  5. Cold boot: EC controller responds ok to init code.
  6. Cold boot: LCD backlight turns on.
  7. Cold boot: linux boots ok in text mode.
  8. Cold boot: linux boots ok in framebuffer (boot splash) mode.
  9. Cold boot: X initializes the LCD at full native resolution.
  10. Cold boot: X enables hardware acceleration.
  11. Boot time: Cold boot to grub succeeds in less than a set timeout.
  12. Boot time: Reboot from linux back to linux succeeds in less than a set timeout.
  13. Boot time: Power down succeeds in less than a set timeout.
  14. SeaBIOS test: keyboard works.
  15. Grub test: keyboard works.
  16. Grub test: text mode and framebuffer graphics work.
  17. Cold boot to USB linux succeeds. (We plan to use SeaBIOS for boot device selection, barring major bugs.)
  18. Reboot to USB linux succeeds.
  19. EC test: fan spins.
  20. EC test: holding power for >5 seconds forces a power down.
  21. ACPI test: lid switch works.
  22. ACPI test: power button event received ok.
  23. ACPI test: AC power on/off event received ok.
  24. ACPI+EC+battery test: battery percentage works.
  25. Media keys on keyboard work in linux.
  26. Device tests: internal mic, internal speakers, webcam, webcam mic, wifi, bluetooth, hard drive, SSD, SD card, each USB port, headphone jack.
  27. prime95 (one instance bound to each hyperthread) for a fixed time to test CPU thermal management.
  28. glxgears for a fixed time to test GPU thermal management.
  29. During prime95 test, CPU digital thermal sensor should give reasonable results.
  30. Linux suspend ok.
  31. LCD backlight adjustable in linux.
  32. Linux kernel boot messages should not contain too many errors.

The effort to write Free Software implementations for all binary blobs will continue in parallel.

Secondary items would include further tweaks to PCI IRQ routing, additional ACPI tables, and optimizing battery life/power use.

coreboot changelog – Weeks of 2015-08-10 and 2015-08-17

this report covers commits 1cbef1c to 410f9ad

The vast majority of changes in these two weeks were upstreamed from Chrome OS and cover work on the Intel Skylake chipset and two mainboards based on it.

QEmu and Getac P470 saw a couple of improvements.
On AMD, there were some bugfixes to Fam10h concerning VGA memory and SMM initialization. The latter was in response to the Memory Sinkhole vulnerability, although it is as yet unclear if it even affects AMD.
Finally, an important memory structure used on pre-AGESA AMD code is now also usable outside Cache-as-RAM.
There was more progress on fixing 64bit issues across the codebase.

Our reference compiler was updated to gcc 5.2. This became necessary to support an update to the RISC-V specification.

Our other tools also saw a couple of improvements: ifdtool now works for descriptors on Skylake and newer platforms. cbfstool saw some refactorings that allow us to extend the format. cbmem now emits the accumulated boot time.

In our configuration system, the Kconfig definitions were cleaned up, so that boards don’t define symbols that their code never uses, that Chrome OS capable boards define “MAINBOARD_HAS_CHROMEOS” (which defines the capability) instead of “CHROMEOS” (which defines that this mode should be
used) and that dependencies between Kconfig options become more consistent.
There is a pending commit on gerrit to enforce clean dependencies by making errors out of kconfig’s warnings, that the latter changes prepare for.

On the build system side, it is now possible to build SeaBIOS as part of our build system even with an enabled ccache. The payload config and revision can also be stored in CBFS for better reproducibility. Finally, it’s possible to override the location from where the vboot source code for Chrome OS-style verified boot is taken from.

In libpayload, the non-accelerated memmove implementation now also works with size == 0 (instead of trying to move 4GB), and there were a couple of bug fixes to the DWC2 (some ARM) and XHCI (USB3) controller drivers, including support for the newer XHCI 1.1 specification.

[GSoC] coreboot for ARM64 Qemu – Week #9 #10

In the last post I talked about using aarch64-linux-gnu-gdb and debugging in qemu. In these two weeks I was intensely involved in stepping through gdb, disassembly and in-turn debugging the qemu port. I summarise the major highlights below.

Firstly, the correct instruction to invoke qemu is as follows

./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -bios ~/coreboot/build coreboot.rom -s -S

After invoking gdb, I moved onto tracing the execution of the instructions step by step to determine where and how the code fails. A compendium of the code execution is as follows

gdb) target remote :1234
Remote debugging using :1234
(gdb) set disassemble-next-line on
(gdb) stepi
0x0000000000000980 in ?? ()
=> 0x0000000000000980: 02 00 00 14 b 0x988
(gdb)
0x0000000000000988 in ?? ()
=> 0x0000000000000988: 1a 00 80 d2 mov x26, #0x0                    // #0
(gdb)
0x000000000000098c in ?? ()
=> 0x000000000000098c: 02 00 00 14 b 0x994
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x0000000000000750 in ?? ()
=> 0x0000000000000750: 3f 08 00 71 cmp w1, #0x2

The detailed version can be seen here.

The first sign of error can be seen here, where the instruction is 0 and the address is way off.

0x64672d3337303031 in ?? ()
=> 0x64672d3337303031: 00 00 00 00 .inst 0x00000000 ; undefined

To find insights as to why this is happening, I resorted to tracing in gdb. This can be done by adding the following in the qemu invoke command. This creates a log file in /tmp which can be read to determine suitable information.

-d out_asm,in_asm,exec,cpu,int,guest_errors -D /tmp/qemu.log

Looking at the disassembly, it can be seen that execution of instructions till 0x784 is correct and it goes bonkers immediately after it. Looking at the trace, this is where the code hangs

IN:
0x0000000000000784:  d65f03c0      ret
The ret goes to somewhere bad. So the stack has been blown or it has executed into an area it should have prior to this. Next, I did a objdump on the bootblock.debug file. Relating to the code at this address, it could be determined that the code fails at “ret in 0000000000010758 <raw_write_sctlr>:”
Next up was determining where the stack gets blown or corrupt. For this, while stepping through each instruction, I looked at the stack pointer. The output here shows the details. Everything seems to function correctly till 0x00000000000007a0 (0x00000000000007a0: f3 7b 40 a9 ldp x19, x30, [sp] ), then next is 0x00000000000007a4: ff 43 00 91 add sp, sp, #0x10 . This is where saved pc goes corrupt. This code gets called in the “raw_write_sctlr_current” (using objdump)
From the trace, we have the following information : The ret goes bad at 0000000000010758 <raw_write_sctlr>:
0x0000000000000908:  97fffe06      bl #-0x7e8 (addr 0x120)
0x0000000000000120:  3800a017      sturb w23, [x0, #10]
0x0000000000000124:  001c00d5      unallocated (Unallocated)
Taking exception 1 [Undefined Instruction]
…from EL1
…with ESR 0x2000000
Which is here:
0000000000010908 <arm64_c_environment>:
   10908: 97fffe06  bl 10120 <loop3_csw+0x1b>
   1090c: aa0003f8  mov x24, x0
This finally gave some leads in the qemu debug. There seems be some misalignment in smp_processor_id.
While tracing in gdb, we have
0x0000000000000908 in ?? ()
=> 0x0000000000000908: 06 fe ff 97   bl  0x120
(which is actually bl smp_processor_id (from src/arch/arm64/stage_entry.S))
Under arm64_c_environment (in objdump) we have;
10908:       97fffe06        bl      10120 <loop3_csw+0x1b>
Also in the trace we have
IN:
0x0000000000000908:  97fffe06      bl #-0x7e8 (addr 0x120)

Now loop3_csw is defined at (from objdump)
0000000000010105 <loop3_csw>:

So this + 0x1b = 10120

Thus it wants to branch and link to 0x120 but smp_processor_id is at 121.

smp_processor_id is at (from objdump)
0000000000010121 <smp_processor_id>:

This gives us where the code is failing. Next up is finding out the reason for this misalignment and rectifying it.

 

 

 

2015-08-21 Librem 13: Weekly Progress Update

This post covers the Librem 13 engineering considerations around writing to the SPI flash chip, and how that affects coreboot development. This builds on last week’s notes on the low level I/O for debugging coreboot builds. As always, email questions or comments to: larry.moberg@puri.sm.

The BIOS flash on the Librem 13 is an 8 MiB chip in a SOP-8 package. The EC firmware is located on a separate 64 KiB chip. This link can be used to locate the chips on your mainboard: the BIOS flash is halfway down, near the CPU heatpipe. The EC flash is at the bottom just below the DDR3L module.

Even without any patches specifically for the Librem 13, flashrom can flawlessly read its BIOS flash because it supports the PCH. Unless you write outside the BIOS region, you will not encounter any problems using flashrom to update your BIOS.

Please don’t update the Intel Firmware Descriptor or ME region just yet–all the IFD bits are marked read/write and flashrom is happy to execute a write to that region. The issue arises because the ME writes asynchronously to the ME region. A collision of flashrom and ME writes will corrupt the ME region and may brick the laptop. Purism intends to provide an unlocked ME that respects your freedom, so look for an update on the purism blog.

In-System Programming The BIOS Flash

The BIOS can be written and verified using a SOP-8 clip after closing the two sides of jumper J1 to assert the PCH RSMRST# pin. Warning: reflashing the BIOS risks bricking the laptop until in-system programming and/or soldering a new BIOS chip restores the BIOS to a good state.

EC Flash

The KB3930 datasheet, section 3.1 “Hardware Trap,” gives the method for programming the EC flash:

  1. Disconnect the battery and the A/C power so that the EC is fully powered down.
  2. Pull GPIO23/TP_ISP (pin 42) to GND. Note the internal 40K-ohm pull-up resistor.
  3. Connect the A/C power supply to the board. The EC is now powered up.
  4. TP_ISP sets CS#, SPI_CLK, MISO, and MOSI lines “High-Z”.
  5. Connect a SOP-8 clip and flash programmer and flash the EC firmware.

It should be noted fine-pitch soldering is required to connect to pin 42 on the EC. The KB3930 also supports firmware updates via software. We will be developing a clean room Free Software EC firmware that has all the same functionality as the existing EC firmware.

 

EC Firmware Listing Notes

This listing is still incomplete. Here are some notes:

  1. KB3930-specific Special Function Registers (SFR), SFR bits, and interrupt vectors are labelled like in the datasheet.
  2. Besides the jump tables and switch tables, there is some 8051 code not listed as code. It appears to be unreachable, though in time the execution path to reach these sections may be found.
  3. No A5 opcode. Some 8051 implementations add this opcode, but the KB3930 uses a vanilla 8051 architecture.
  4. This is not Free Software. Users should have the ability to verify their shipped firmware against this listing but Purism’s goal is to provide a Free Software implementation of the EC firmware.

Conclusions

This post covers the steps for coreboot development and recovery. Fortunately, the EC firmware is not shared with the BIOS or ME blob, which makes flashrom’s job easier.

BIOS development is hard. One of the major challenges facing BIOS developers is a lack of accurate, comprehensive documentation for all the hardware coreboot interacts with. The “elephant in the room,” for an Intel-based laptop, is the Management Engine.

[GSoC] End user flash tool – week #13 – summary

Hello!

During week 13 I worked on:

  • writing project documentation
  • bug fixing
  • code cleanup
  • changing debug messages to information popups
DOCUMENTATION

All functions and variables are now documented in javadoc style, I also attached some comments in code, documentation can be generated with doxygen. Besides documenting code I prepared documentation for functional tests which I executed. It contains described test cases and test results.

NEW WORKING CONFIGURATIONS

To make automatic building of coreboot image feature more useful it is necessary to add more data about working configurations. Every added configuration also needs to be tested to check if tool correctly recognizes hardware on target system and builds working coreboot image. I can do this for my hardware, so of course I did, I can also add some fake configurations for testing purposes, but I can’t do this for hardware which I do not have.

Here is description of application, building process and information about data I need to add a working configuration: link. I will be grateful for every configuration which you will send!

GUI IMPROVEMENTS

Tool is targeted mostly for users which are not familiar with coreboot and flashrom details and also dont know how to proceed with building a working coreboot image. Taking it into consideration I changed most debug/error messages to appear in a form of a popup with description what action is needed from application user to proceed or what went wrong.

end_user_flash_tool_dialogs

SUMMARY

End user flash tool is my first experience with coreboot and flashrom. Both of them are not easy projects, especially for someone who does not have great experience with firmware programming, because most code involves serious low level implementations, but in End user flash tool project I have been working always few layers above it because my work involved GUI programming, system programming, integration of external tools like bios_extract or cbfs_tool and adding few features to libflashrom. By doing it I learned a bit how these tools work internally. I am now also more familiar with coreboot itself. This work gave me good basics and smooth entry to the coreboot world. As all of this is interesting and working for such project is very satisfying I want to dive in more and also maintain and extend the tool.

CCCamp 2015

We had an table on the cccamp 2015 in Mildenberg. The cccamp (chaos communication camp) is one of Europe big hacker events every 4 years where several hackers come together and do camping. Everybody could relax a little bit and talk to each other in a nice background. The camp infrastructure contained own wifi, dect, gsm network and a 10 GBit uplink in the middle of nowhere. You could give your tent a 1 Gbit uplink to the internet ;).

Our table was in the BER village, named to the never finished airport near Berlin, Germany. Several people from the coreboot community showed up (CareBear, felix, mue, paulk-collins, tnias, zaolin, […]) and we shared a lot of ideas to each other. In that way we flashed several laptops, replaced some WSON chips with SOIC-8’s. Also we’ve found another bug in the sandybridge ram init, a fix is waiting for merge on gerrit #11248.
paulk-collins came by to talk about a EC open source firmware for the ENE KBxxx embedded controllers.
One of the MAME hackers visited us to get ideas how to port a Dell notebook (ENE KBxxx based EC).
Felix did some work on ME as well other hackers joined him.
Tnias and zaolin started the idea of a raspberry pi doing all the flashing including detection of the device. This could let us drink more Mate while other do the flashing themself.
In the end a thunderstorm reached our tent and we had to evacuate it.

Hopefully everybody can come to Bonn this year.

[GSoC] End user flash tool – week #10 #11 #12

Hello!

During weeks 10, 11 and 12 I worked on:

  • functions for gathering hardware specific data
  • automating process of building coreboot image
  • GUI improvements
GATHERING HARDWARE SPECIFIC DATA

As I mentioned in my last post, if coreboot image should be built automatically then application needs to collect hardware specific data. During last weeks I added functions responsible for:

  • dumping VGABIOS from memory: Some systems (like Lenovo T60 with ATI graphics) require adding VGABIOS dumped from memory to coreboot image, because factory BIOS patches it at runtime, so in certain cases using option rom extracted from factory BIOS may not work
  • getting EDID data: to know what type of display panel is used in system
  • getting motherboard model name: this is mandatory to recognize if we can install coreboot in system or if we have known working configuration
AUTOMATING PROCESS OF BUILDING COREBOOT IMAGE

It is now possible to build and flash coreboot image by clicking few application buttons without taking care which options are correct for system. Of course this is not possible for all hardware configurations, for now this is very limited, but with time database of known good configurations will grow and more users will be able to flash coreboot in such easy way.

flash_tool_auto_tab

Process of automated image building:

  1. Check if working configuration for system is known.
  2. Check if configuration requires additional option rom.
  3. If necessary, add option rom extracted from factory bios or extracted from running system memory.
  4. Build image.
GUI IMPROVEMENTS

I decided to change a look of ‘ROM options’ tab. Previously just raw output of cbfs_tool with rom contents info was redirected to GUI log window. Now it is visible in a form of table, what in my opinion is more readable.

cbfs_tool_GUI

LAST WEEK AND FURTHER PLANS

GSoC “pencils down” date is coming up on Friday, so only few days are left. I want to use this time for:

  • writing project documentation
  • looking for bugs and fix them
  • code cleanup
  • adding new working configurations and testing them (I need your help with it)
  • change debug messages to information popups

What after GSoC? I would like to still contribute, there is place for many improvements and possible extensions.

2015-08-14 Librem 13: Weekly Progress Update

A question coreboot developers are commonly asked is this: “can you port coreboot to my board?”

For my first coreboot post I’d like to show some of the steps required to port coreboot to the Librem 13. In particular, this post is a good example of some of the challenges involved in such a port.

This post is also the first weekly progress update for the Librem 13. Please email me with questions or comments: larry.moberg@puri.sm.

LPC Bus

The Librem 13 has convenient test points for the LPC bus. This allows a bed-of-nails test setup to quickly diagnose problems during manufacturing. But it has the added bonus of facilitating coreboot development.

The earliest coreboot stages are the most important to get right. Debugging using port 0x80 writes on the Librem 13 is possible because port 0x80 writes are configured as LPC writes, which can be traced by connecting to the LPC pins.

And…It’s Gone

BIOS development is hard. I applied a little too much force on the SPI flash chip and tore the solder pads off the board.

Pads...gone

I attached the LPC connection to a test setup and didn’t check using a multimeter before applying power. LAD2 was shorted to LAD3. This immediately bricked the laptop without even releasing any smoke. Remembering to double check for shorts is a tedious but important lesson.
Don't Cross The Streams

The LPC bus wires go under the board. Don’t Cross The Streams!

Why It Matters

Imagine a laptop where the LPC bus is only available by soldering directly to the pins of the EC. Yes, they exist! That level of fine soldering is a significant barrier for future coreboot hackers. (The Librem 13’s external USB ports are all USB 3, which makes an EHCI debug port harder, but the LPC bus is a good substitute.)

Porting coreboot to a new laptop takes a lot of time and work. Even a good laptop design like the Librem 13 where the LPC pads are available still has a non-trivial level of engineering work to get to a Free Software BIOS.

Next week, I’ll document the engineering considerations around writing to the SPI flash chip, and how that affects coreboot development.

[GSoC] End user flash tool – week #7 #8 #9

Hello!

During weeks 7, 8 and 9 I worked on:

  • functions for gathering hardware specific data
  • extending libflashrom
  • GUI improvements
  • testing
GATHERING HARDWARE SPECIFIC DATA

Main purpose of End user flash tool project is to provide an easy way to build and flash coreboot ROM. To achieve it there is a need to collect hardware specific data such as:

  • lspci -nn output: information about all PCI buses and devices in the system, it is possible to recognize a graphic card, its vendor and device codes
  • dump of factory BIOS: it is important to make a copy of factory BIOS in case something will go wrong, but not only in this case, very often there is a need to use a VGABIOS extracted from factory BIOS if particular graphic card or display panel is present in our system, it is the best to make dump two times and then check if files are the same (for example by comparing hashes)
EXTENDING LIBFLASHROM

Sometimes when flashrom probes for all known chips there are multiple chips found. I needed to implement a function which will return all such chips to GUI. Now in this case it is possible to just select which chip is correct by clicking a proper one in dialog box.

choose_chip

GUI IMPROVEMENTS

I decided to change a bit visual design of the app. There is a new (but most important for the project)  tab – ‘Auto’. In this tab it is possible to gather hardware specific data, which will be then used in a process of automatic building of coreboot image and flashing. I also decided to move programmer selection combobox from ‘Flash tab’ to main application window and  add edit text field for parameters.

LOOKING FOR TESTERS

GSoC ends in next week, application is almost done (but will be improved and extended also after GSoC), so this is time for some testing! I would be very grateful if some of you could help me with it. It would be the best if you have Lenovo T60 and external programmer. First there is a need to collect some hardware specific data and then it would be possible to check if application creates working coreboot image basing on this data. So it is not only about testing, but also about making white list of hardware configurations bigger to let more users flash their hardware with coreboot in easy way!

Please contact me on:

  • IRC #flashrom #coreboot: lukaszdm
  • e-mail: lukasz.dmitrowski@gmail.com

Thanks in advance!

coreboot changelog – Weeks of 2015-07-27 and 2015-08-03

This covers commits ef0158ec up to commit 1cbef1c
Development is typically slower during the summer and 2015 is no exception, so the report switches to a biweekly installment for a while.

The last two weeks have seen improvements in our development tools:
coreboot upstream can now build Chrome OS boards with Chrome OS features (verified boot, interaction with Chrome EC, flash based error logging) enabled, and the projects builders at http://qa.coreboot.org/ are now routinely building these configurations alongside the regular default configs for all boards.
The builders now run ‘make what-jenkins does’ (see coreboot/Makefile.inc) instead of a hard-coded set of commands, which provides the community the capability to adapt the test build without admin intervention.
When adding the .config used for building an image into said image, it’s now minimized which gives visibility to the relevant changes to the config compared to the board’s defaults.
Kconfig features a strict mode, which acts as a ‘warnings-as-errors’ equivalent and fails the build if kconfig would emit any warning. Since we still have a couple of those in the tree, it’s not enabled yet.
For users of cscope or ctags, we now have new make targets to create tree-wide indexes (make ctags-project cscope-project).

Reproducible builds got a boost by fixes to the build.h generator script, which can finally emit stable timestamps based on the git revision, instead of the local time.

External payload integration was coalesced within payloads/external, with more work in progress. The integrated SeaBIOS build can now also be used when building with ccache. libpayload gained robustness in different developer environments, being smarter about looking for compilers, configs and include files in all the right places.

On the Free Software side, more microcode blobs were moved to the 3rdparty/blobs repository and one false positive that libreboot’s blob detector tripped over was eliminated, and with a little more progress, it should soon be possible to build from a fully blob-free coreboot tree. Before you get your hopes up, please note that the result may not be very useful on a lot of boards, so more care must be taken.

The effort to make coreboot capable of booting in 64bit mode on x86-64 is still ongoing and saw the integration of more commits.

coreboot should have an easier time again when building on Cygwin and BSD systems.

Skylake was the chipset with the largest amount of work in the 2 weeks, but there was also the addition of a coreboot port for RISC-V’s Spike ISA Simulator, contributions to the AMD Bettong mainboard and its chipset drivers, as well as fixes and cleanups to AMD K8 and Intel i945.

In terms of style, a bunch of extraneous whitespaces, indenting errors and FSF addresses were also dealt with.