Author: jneuschaefer

In less than an hour, Google Summer of Code 2016 will be over (at least for us students). In this blog post, I will describe how you can use coreboot on RISC-V.

You can find the complete list of commits that I made during GSoC with this gerrit query.

The details

Compiling spike, the RISC-V instruction-set-level simulator

Spike, also known as riscv-isa-sim, is the reference implementation of RISC-V, and the only RISC-V platform that is currently known to work with coreboot (QEMU is nominally also supported, but the corresponding coreboot code has not been updated in a while).

First, you need to build and install libfesvr:

• Clone the libfesvr git repository
• run ./configure --prefix="$HOME" && make && make install

Then, you can compile and install spike:

• Clone the spike git repository.
• Apply the patch in this pull request to make console output possible.
• Open riscv/processor.cc in a text editor and find processor_t::get_csr. Add a line that reads case CSR_MTIME: return 0;.
• run ./configure --prefix="$HOME" --with-fesvr="$HOME" && make && make install

Compiling coreboot for RISC-V

• Clone the coreboot git repository, if you haven’t already
• Apply Iru Cai’s patch that updates the toolchain to GCC 6.1. You will currently have to fix a merge conflict when you apply this patch, but it’s an easy one.
• Run make crossgcc-riscv
• Run make menuconfig to configure coreboot. Select Emulation and SPIKE usb riscv in the Mainboard menu.
• Run make
• Run util/riscvtools/make-spike-elf.sh build/coreboot.rom build/coreboot.elf
• Start coreboot by running spike build/coreboot.elf. You should see a few pages of output, ending in Payload not loaded.

Compiling and running Linux

• Clone the riscv-linux git repository and check out the priv-1.9 branch
• Apply this patch that allows linux to be started in machine-mode.
• Download a copy linux 4.6.x from kernel.org, and unpack it. I’ll assume version 4.6.2 is used.
• cd into linux-4.6.2/arch and symlink the arch/riscv directory from riscv-linux
• Back in linux-4.6.2, run make O=build ARCH=riscv defconfig; cd into the newly created build directory.
• Run make ARCH=riscv menuconfig. In the “General Setup” menu of the linux menuconfig, enter path/to/coreboot/util/crossgcc/xgcc/bin/riscv64-elf- as the cross-compiler tool prefix.
• Run make ARCH=riscv vmlinux to compile linux.
• Open vmlinux in a hex editor, such as hexer. Change the 8-byte number at 0x18 to 00 00 00 90 00 00 00 00; Add 00 00 00 90 00 00 00 00 to the numbers at 0x58 and 0x90, to load linux at physical addresses within RAM. It’s unfortunate that I have to recommend this step, but I did not come up with a better fix yet.

Next, you need to add vmlinux to coreboot:

• Return to the coreboot directory.
• Apply the remaining coreboot patches that are tagged riscv.
• In the Payload menu, select An ELF executable payload. Instead of payload.elf, select the vmlinux file.
• Run make and util/riscvtools/make-spike-elf.sh build/coreboot.rom build/coreboot.elf again.
• Run spike build/coreboot.elf. You should now see a Linux boot, at least partially.

Future work

Even though my GSoC is over, coreboot’s support for RISC-V can still be improved, and I intend to fix at least some of the following things:

• As you can see above, running coreboot and linux on RISC-V currently isn’t straight-forward, but involves a few manual steps.
• There are other RISC-V platforms that I’d like to see coreboot on, such as lowRISC.
• Linux does not completely boot, i.e. into userspace. There are still some bugs to be ironed out.
• Automatic testing could be used to detect regressions.
• I only tested coreboot on RISC-V with Linux; support for other operating systems or payloads is welcome.

Acknowledgements

I’d like to thank Ron Minnich and Furquan Shaikh for being good mentors, and everyone in the coreboot community for being helpful and friendly.

This past week, I worked on the virtual memory initialization code of coreboot on RISC-V. The first part of this was to update encoding.h a file that defines constants such as bit masks which are necessary for interacting with RISC-V’s Control and Status Registers. As a result, I also had to change a few files that relied on outdated constants. Then I wrote some code to walk the page table structures, and fixed one or two bugs in the page table setup code. Unfortunately my patches aren’t as finished as I would like them to be.

When I tested my changes with a Linux payload (which I had to edit the ELF headers of, because Linux uses virtual addresses, while cbfstool and the SELF loader use physical addresses), I stumbled upon another strange error: I get a store access fault in the instruction of the trap handler that saves the first register, even if I initialize the stack pointer to a value that’s within the RAM. When the trap handler runs again to handle this store access fault, it runs without any problems. This fault is especially confusing, because machine mode should always be able to access RAM through its physical addresses.

What’s next?

During the next week, I’ll be traveling and won’t be able to work on coreboot. When I return, I will rework my patches so they can be merged, and hopefully understand the aforementioned access fault problem. Properly set-up page tables should bring me a step closer to running Linux on coreboot/RISC-V (without bbl in the middle).

If time permits, I will start porting coreboot to the Nexys 4 DDR devboard.

Since I haven’t posted an update of my coreboot-on-RISC-V work in a while, this will be a slightly longer post.

Week 6

In week 6, I started documenting how to build and boot coreboot on RISC-V, in the coreboot wiki.
It is now a bit outdated, because we’re moving away from using bbl to boot Linux.

Week 7

I wrote some patches: I removed code that used the old Host-Target Interface (HTIF), because it’s deprecated. I submitted an improved version of my workaround for the bug that causes Spike to only execute 5000 instructions in some cases. I informed the coreboot resource management subsystem about the position of the RAM in physical address space, so that the program loader wouldn’t refuse to load segments into RAM. I submitted two patches to fix compiler errors with the new toolchain.

Meanwhile, there were some good news in the RISC-V world:

• The RISC-V project released version 1.9 of the RISC-V Privileged Architecture Specification
• Around the same time, lowRISC released version 0.3 of their RISC-V implementation for the Nexys 4 DDR devboard

Week 8

I submitted a few more patches and started to explore the Nexys4 board. The precompiled bitstream and kernel from the lowRISC version 0.3 tutorial worked without any problems, and after a few days and some help from the lowRISC mailing list, I was able to recompile the lowRISC bitstream.

This week

I discussed the choice of boot medium with the lowRISC developers, and they agreed that a memory-mapped flash would be useful. Once it is implemented, I
can start porting coreboot to lowRISC on the Nexys 4 DDR board. Luckily the Nexys4 already has large enough flash to use for this purpose.

My mentors and I agreed that the switch from machine mode to supervisor mode should be left completely to the payload.

What’s next?

I will continue to work on running coreboot on the Spike emulator. Currently I’m facing the following problems and tasks:

• Linux, when compiled to an ELF file (vmlinux) specifies that it wants to be loaded at the physical address 0x0 and at the virtual address 0xffffffff80000000. Since coreboot’s ELF loading code only looks at the physical address, it refuses to load Linux, since RAM starts at 0x80000000 on RISC-V.
• Low level platform information (most importantly the memory layout) is passed to the firmware (coreboot in this case) as a configuration string, which is dynamically generated by the emulator, in the case of Spike. I still need to implemented a parser for this format, so coreboot can know how much memory is available.
• The RISC-V Privileged Architecture Specification 1.9 specifies that there shall be a page at the top of the virtual address space where the operating system can call a few functions exposed by the firmware (this is the Supervisor Binary Interface).

Week 4

In week 4, I tracked down why coreboot halted after about one line of output. It turned out to be a spike bug, that I wrote up in this bug report, and affect any program that doesn’t have a tohost symbol. As a workaround, I extended my script that turns coreboot.rom into a ELF to also include this symbol.

After some more patches I could run coreboot in spike and get the familiar “Payload not loaded” line.

Week 5

I was now clearly moving towards being able to run linux on spike/coreboot. But there was a problem: The RISC-V linux port requires a working implementation of the Supervisor Binary Interface (SBI), which is a collection of functions that the supervisor (i.e. the linux kernel) can call in the system firmware.

Coreboot has an implementation of the SBI, but it’s probably outdated by now. To get an up-to-date SBI implementation, I decided to use bbl as a payload. When I built bbl with coreboot’s RISC-V toolchain, I noticed that it depends on a libc being installed in two ways:

• The autoconf-generated configure script checks that the C compiler can compile and link a program, which only succeeds if it finds a linker script (riscv.ld) and a crt0.o in the right place.
• bbl relies on the libc headers to declare some common functions and types (it doesn’t use any of the implementations in the libc, though).

The coreboot toolchain script doesn’t, however, install a libc, because coreboot doesn’t need one.

I tweaked the bbl source code until it didn’t need the libc headers, changed the implementation of mcall_console_putchar to use my 8250 UART, got the payload section of bbl (where linux is stored before it’s loaded) out of the way of the CBFS by moving it to 0x81000000 (bbl/bbl.lds is the relevant file for this change), and could finally observe Linux booting in spike, on top of coreboot and bbl. It stops with a kernel panic, though, because it doesn’t have a root filesystem.

Plans for this week

This week I will document my work on the Spike wiki page in the coreboot wiki, so others can run coreboot on spike, too.

Last week, after updating GCC (by applying Iru Cai’s patch) and commenting out uses of outdated instructions and CSRs (most notably eret and the HTIF CSRs), I noticed that coreboot crashed when it tried to access any global variables. This was because the coreboot build system thought coreboot would live near the start of the address space.

I found spike-riscv/memlayout.ld, and adjusted the starting offset. But then I got a linker error:

build/bootblock/arch/riscv/rom_media.o: In function `boot_device_ro': [...]/src/arch/riscv/rom_media.c:26:(.text.boot_device_ro+0x0): relocation truncated to fit: _RISCV_HI20 against `.LANCHOR0'


I played around with the start address and noticed that addresses below 0x78000000 worked, but if I chose an address that was too close to 0x80000000, it broke. This is, in fact, because pointers to global variables were determined with an instruction sequence like lui a0, 0xNNNNN; addi a0, a0, 0xNNN. On 32-bit RISC-V, the LUI instruction loads its argument into the upper 20 bits of a register, and ADDI adds a 12-bit number. On a 64-bit RISC-V system, lui a0, 0x80000 loads 0xffffffff80000000 into a0, because the number is sign extended.

After disassembling some .o files of coreboot and the RISC-V proxy kernel, I finally noticed that I had to use the -mcmodel=medany compiler option, which makes data accesses pc-relative.

Now that coreboot finally ran and could access its data section, I finished debugging the UART block that I promised last week. Coreboot can now print stuff, but it stops running pretty soon.

Plans for this week

This week I will debug why coreboot hangs, and will hopefully get it to boot until the “Payload not found” line again, which worked with an older version of Spike.

Also, Ron Minnich will be giving a talk about coreboot on RISC-V at the coreboot convention in San Francisco, in a few hours.

Last week, I updated my copy of spike (to commit 2fe8a17a), and familiarized myself with the differences between the old and the new version:

• The Host-Target Interface (HTIF) isn’t accessed through the mtohost and mfromhost CSRs anymore. Instead, you have to define two ELF symbols (tohost and fromhost). Usually this is done by declaring two global variables with these names, but since the coreboot build system doesn’t natively produce an ELF file, it would get a little tricky.
• Spike doesn’t implement a classic UART.
• The memory layout is different. The default entry point is now at 0x1000, where spike puts a small ROM, which jumps to the start of the emulated RAM, at 0x80000000. One way to run coreboot is to load it at 0x80000000, but then it can’t catch exceptions: The exception vector is at 0x1010.
• Within spike’s boot ROM, there’s also a text-based “platform tree”, which describes the installed peripherals.

“Why does coreboot need a serial console?”, you may ask. Coreboot uses it to log everything it does (at a configurable level of detail), and that’s quite useful for debugging and development.

Instead of working around the problems with HTIF, I decided to implement a minimal, 8250-compatible UART. I’m not done yet, but the goal is to use coreboot’s existing 8250 driver.

Plans for this week

This week, I will rewrite the bootblock and CBFS code to work with RISC-V’s new memory layout, and make sure that the spike UART works with coreboot’s 8250 UART driver. Booting Linux probably still takes some time.