[GSoC] Wrap-up for Adding QEMU/AArch64 Support to Coreboot

Hello, I’m Asami. It is the time for the final evaluation of GSoC 2019 now. My project is adding QEMU/AArch64 support to coreboot. It would help developers for compatibility testing and make sure that changes to architecture code don’t break current implementations for ARMv8.

Here is my work on Gerrit. CL 33387 is the main patch for this project and it is successfully merged. We can now run coreboot on QEMU/AArch64.

What I’ve Done for My Main Project

Firstly I made a new directory qemu-aarch64 in src/mainboard/emulation/ and I was basically working on in it. In the bootblock stage, I have written custom bootblock code in assembly because an ARM virt machine doesn’t have SRAM, which means I had to relocate code inside ROM to DRAM during the first stage. I also made a memory layout with reference to the QEMU implementation. In the romstage and the ramstage, I registered a custom handler for detecting DRAM size because AArch64 throws a Synchronous External Abort that happens when you try to access something that is not memory.

I wrote documentation for how to use coreboot with QEMU/AArch64. Here is the page: https://doc.coreboot.org/mainboard/emulation/qemu-aarch64.html

Future Work

ARMv8 architecture has an integrated LinuxBios as a payload. However, I can’t run coreboot with it yet. It now causes an error while building. I also need to make sure that coreboot works well with ARM Trusted Firmware. So, I’d like to solve problems and I hope to see “Hello World” with LinuxBios.

Conclusion

I believe I almost succeed in my project. The main code has already merged and I also solved small problems found while working on the main project. The members of coreboot are really great and they, especially my mentor and reviewers, helped me a lot. It was not an easy project for me because I had never experienced to work on coreboot and I had to know the basic code flow of it. I’ve read much code and it made me grown up. Thank you all of the members for such an excellent time.

My Previous Blog Posts

[GSoC] How to Use ARM Trusted Firmware in Coreboot

Hello, I’m Asami. In this article, I’m going to talk about how to use ARM Trusted Firmware in coreboot for ARMv8 (AArch64). ARM Trusted Firmware provides a reference implementation of secure world software for Armv8-A and Armv8-M. You can see the code via https://github.com/ARM-software/arm-trusted-firmware.

Trusted Firmware has 5 steps which are called as BL1, BL2, BL3-1, BL3-2, and BL3-3. BL1 step is similar to the bootblock stage and BL2 is similar to the romstage of coreboot. In the coreboot project, we only use the BL3-1 part that is expected to work on EL3 exception level. The code of BL3-1 will execute just after the ramstage and before the payload when we enable the Trusted Firmware.

How to enable Trusted Firmware

It’s very easy to enable Trusted Firmware on coreboot. You just need to ‘select ARM64_USE_ARM_TRUSTED_FIRMWARE’ in your Kconfig. If you want to run coreboot on QEMU/AArch64, you need to add the ‘select ARM64_USE_ARM_TRUSTED_FIRMWARE’ at src/mainboard/emulation/qemu-aarch64/Kconfig. The next step switches depending on the configuration at src/arch/arm64/boot.c. Here is the code to switch the next step:

// src/arch/arm64/boot.c
static void run_payload(struct prog *prog)
{
	void (*doit)(void *);
	void *arg;

	doit = prog_entry(prog);
	arg = prog_entry_arg(prog);
	u64 payload_spsr = get_eret_el(EL2, SPSR_USE_L);

	if (CONFIG(ARM64_USE_ARM_TRUSTED_FIRMWARE))
		arm_tf_run_bl31((u64)doit, (u64)arg, payload_spsr);
	else
		transition_to_el2(doit, arg, payload_spsr);
}

Why using Trusted Firmware

Coreboot for ARMv8 has 2 options to pass an execution from it to a payload. The first is passing execution to a payload directly and the second one is passing to the BL3-1 code before a payload. You always don’t have to use Trusted Firmware. However, you need to enable Trusted Firmware if you want to run Linux because it expects to work with PSCI. PSCI is an abbreviation of Power State Coordination Interface which is a standard interface for power management that can be used by OS vendors for supervisory software working at different levels of privilege on an ARM device. Coreboot doesn’t have the setup for PSCI but Trusted Firmware does.

Current Status

Unfortunately, QEMU/AArch64 in coreboot doesn’t support Trusted Firmware yet. It means we can’t run Linux with QEMU for ARMv8. I’m now trying to support Trusted Firmware for QEMU/AArch64.

[GSoC] How to Run C Code in Bootblock Stage for QEMU/AArch64

Hi, I’m Asami. My project “adding QEMU/AArch64 support to coreboot” is making good progress. I’ve almost done to write porting code and I’m now writing a new tool to make a FIT payload for QEMU/AArch64. Here is my CL. In this article, I’m going to talk about how to run C code in the bootblock stage.

The way to run C code before DRAM has been initialized is various in the coreboot project. The most famous and used in the x86 system is known as the Cache-As-Ram (CAR). CAR can be set up by 1. enable CPU cache 2. enable the ‘no eviction’ mode 3. change cache mode from write-through to write-back. ‘No-eviction’ means that the CPU doesn’t write any data to DRAM as long as the size of data is less than the CPU cache. Then, you can get all the data from the CPU cache. The implementation for CAR is src/cpu/intel/car/non-evict/cache_as_ram.S.

Another way to run C code especially implemented in ARM system is relocating the bootblock code to SRAM. System on a Chip (SoC) has an ARM CPU and often includes SRAM as a cache. For example, QEMU VExpress machine has SRAM is located at 0x48000000 that we can know in qemu/hw/arm/vexpress.c#L120.

However, what should the system that doesn’t have SRAM do? My target machine, QEMU virt machine, doesn’t have SRAM according to the implementation of qemu/hw/arm/virt.c. In this case, should we initialize DRAM earlier?

The answer is No but it’s only for my project. Because QEMU is not actual hardware so DRAM already works. I just need to relocate the bootblock code to DRAM directly. I believe that other ARM systems that have no SRAM should initialize DRAM earlier.

You can see the relocation code in bootblock_custom.S under review. https://review.coreboot.org/c/coreboot/+/33387

[GSoC] Debug Bootblock Stage for ARMv8 on QEMU

Hello again, I’m Asami. I’ve just finished 4 weeks as a GSoC student. I’m currently debugging the implementation of my main project, which is adding QEMU/AArch64 support. I can see nothing output right now when I start a QEMU with the coreboot.rom that has my implementation. It means there is something wrong before a hardware initialization has finished. In this article, I’m going to talk about what I found while debugging the bootblock for ARMv8.

Code Path of Bootblock Stage

The bootblock is executed just after CPU reset and it is almost written by assembly language. The main task is to set up a C-environment. The basic code path for ARMv8 from the beginning the bootblock to the romstage is:

  1. _start() at src/arch/arm64/armv8/bootblock.S
  2. arm64_init_cpu() at src/arch/arm64/armv8/cpu.S
  3. main() at srclib/bootblock.c
  4. run_romstage() at src/lib/prog_loaders.c
  5. prog_run() at src/lib/prog_ops.c
  6. arch_prog_run() at src/arch/arm64/boot.c
  7. main() at src/arch/arm64/romstage.c // The entry point of the romstage

You can use your custom _start function instead of the common _start function by CONFIG_BOOTBLOCK_CUSTOM=y and adding bootblock-y += bootblock_custom.S which is your custom assembly file.

The Reason Why Execution Stopped inside arm64_init_cpu()

I found that an execution stopped inside the arm64_init_cpu function for some reason. The line that has some problem is mrs x22, sctlr_el3 . MRS instruction can read a system control register and store the value into a general purpose register. So this line means to store the value of SCTLR_EL3 into the X22 register.

According to the “ARM Architecture Reference Manual ARMv8, for ARMv8-A architecture profile”, the purpose of SCTLR_EL3 is

Provides top level control of the system, including its memory system, at EL3. This register is part of the Other system control registers functional group.

Also, SCTLR_EL3 is accessible only from EL3 mode. EL3 is the highest privileged mode that a low-level firmware, including the secure monitor, works on it.

Next, I checked the current mode via mrs x0, CurrentEL. CurrentEL is a register that holds the current exception level. The result of CurrentEL was 0x04, which means the program works on EL1 mode. EL1 is the mode that an operating system kernel typically described as privileged. I didn’t have the right to access SCTLR_EL3. That’s why an execution stopped.

Ideas to Solve EL3 Issue

I considered 2 solutions:

  1. Use only EL1 registers
  2. Run QEMU in EL3

Firstly, I tried to use only EL1 registers. I replaced arm64_init_cpu with arm64_init_cpu_el1 that is a new function I created. Then I replaced SCTLR_EL3 with SCTLR_EL1 and TLBI ALLE3 with TLBI VMALLE1. It seems to work well but still, there was nothing output.

Secondly, I tried to run QEMU in EL3 that is enabled by -machine flag. QEMU can work on EL2 with -machine virtualization=on and EL3 with -machine secure=on to enable EL3. The following command works well for me.

$ qemu-system-aarch64 -bios ./build/coreboot.rom -M virt -cpu cortex-a53 -nographic -smp 1 -machine secure=on

My mentor Raul told me the second solution. Thank you so much!

[GSoC] Common Mistakes for Beginners

Hello everyone. I am Asami and a student for this year’s GSoC project. My project is adding a new mainboard QEMU/AArch64 to make it easier for coreboot developers to support new boards for ARMv8. I’ve already written a small patch to enable building a sample program with libpayload for ARM architecture. Also, I’ve read the implementation of coreboot (main code path) for ARMv7 and QEMU (qemu/hw/arm/vexpress.c). Now, I just created a new CL for my main project and I started to read the implementation of the target machine of AArch64 (qemu/hw/arm/virt.c).

In this article, I’m going to talk about my mistakes when I developed coreboot. I hope it helps for beginners of coreboot development. The target board is QEMU/ARM and the CPU is ARMv7.

“ERROR: Ramstage region _postram_cbfs_cache overlapped by: fallback/payload”

I faced this error when I built coreboot.rom for QEMU/ARM with the coreinfo which is a small informational payload for coreboot. The cause is that the coreinfo doesn’t support ARM architecture and then the payload is compiled as a 32-bit x86.

Make sure that your payload is your target architecture. You need to use other executable files instead of the coreinfo when you want to use architectures other than x86. We provide the libpayload which is a small BSD-licensed static library.

The details of the error is:

$ make
...(omitted)....
W: Written area will abut bottom of target region: any unused space will keep its current contents
CBFS fallback/romstage
CBFS fallback/ramstage
CBFS config 
CBFS revision
CBFS fallback/payload
INFO: Performing operation on 'COREBOOT' region...
ERROR: Ramstage region _postram_cbfs_cache overlapped by: fallback/payload
Makefile.inc:1171: recipe for target 'check-ramstage-overlaps' failed
make: *** [check-ramstage-overlaps] Error 1 

“ERROR: undefined reference to ‘_ttb'” and “ERROR: undefined reference to ‘_ettb'”

This errors might happen when you build coreboot.rom by `make` at root directory. In this case, You need to add TTB() at your memleyout.ld.  

TTB is a translation table base address for MMU. TTBR0 and TTBR1 (TTB registers) hold the start point of TTB. We can put TTB anywhere in memory as long as we store the address to TTBR.

According to the “ARM Architecture Reference Manual ARMv7-A and ARMv7-R edition”, the difference between TTBR0 and TTBR1 is:

(B3-1345) When a PL1&0 stage 1 MMU is enabled, TTBR0 is always used. If TTBR1 is also used then:

– TTBR1 is used for the top part of the input address range

– TTBR0 is used for the bottom part of the input address range

https://static.docs.arm.com/ddi0406/c/DDI0406C_C_arm_architecture_reference_manual.pdf

(B4-1724) TTBCR determines which of the Translation Table Base Registers, TTBR0 or TTBR1, defines the base address for a translation table walk required for the stage 1 translation of a memory access from any mode other than Hyp mode.

https://static.docs.arm.com/ddi0406/c/DDI0406C_C_arm_architecture_reference_manual.pdf

TTBR0 is basically used for user processes and TTBR1 is used for kernel. However, Linux kernel only uses TTBR0 to reduce the time of context switch. (I just heard that Linux kernel starts to use TTBR1 because of security reasons such as Meltdown and Spectre.)

In coreboot, mmu_init() sets TTBR registers in arch/arm/armv7/mmu.c.

Fails to build a sample program with libpayload

We provide the libpayload which is a small BSD-licensed static library for coreboot and we also have a sample program to know how to use it. However, you might fail to build a sample program when you select the ARM architecture as a target with the following errors:

/usr/bin/ld: cannot represent machine `arm'

The reason why this problem happens is Makefile in the sample directory is old dated. So I created a CL to update current architectures that coreboot supports.

Please see the Makefile in https://review.coreboot.org/c/coreboot/+/33287

“Payload not loaded”

“Payload not loaded” happens when the load address of a payload is wrong. The load address should be placed in the RAM place where anyone can use. You can define the load address via CONFIG_LP_BASE_ADDRESS if you use a libpayload.

I created a CL for a sample configuration. Please see the config.emulation-qemu-arm in https://review.coreboot.org/c/coreboot/+/33287

Whole operations for building coreboot.rom with a sample payload for QEMU/ARM are:

1. Build a libc and cross compiler environment.

// In coreboot/payloads/libpayload/
$ make distclean // Always needs when switching a mainboard.
$ cp configs/config.emulation-qemu-arm configs/defconfig // Or you can set up it via 'make menuconfig'
$ make defconfig
$ make
$ make install

2. Build a sample payload hello.elf.

// In coreboot/payloads/libpayload/sample
$ make // Make sure that Makefile is updated by https://review.coreboot.org/c/coreboot/+/33287

3. Build coreboot.rom with a sample payload.

// In coreboot/
$ make distclean // Always needs when switching a mainboard.
$ make menuconfig // or make defconfig
  Select payload “payloads/libpayload/sample/hello.elf”
$ make

Make sure to do ‘make distclean’ before switching your board target

‘make distclean’ removes build artifacts and config files. The default archtecture in coreboot is x86, so you need to do ‘make distclean’ when you want to use other architectures.

Fails to update an existing CL on Gerrit 

Gerrit is a code review tool used in coreboot project. I’m familiar with GitHub and I thought the operations of Gerrit are almost the same with the operations of GitHub, but it weren’t.

On GitHub, developers can create a commit for each update. On the other hand, developers using Gerrit need to amend their commit until it will be merged.

Commands to create a new CL are almost the same with the operations of GitHub:

$ git add <target files>
$ git commit -s
$ git push

Commands to update an existing CL are slightly different:

$ git add <target files>
$ git commit --amend --no-edit

Make sure to leave the “Change-Id” line of the commit message as is.