[GSoC] How to Run C Code in Bootblock Stage for QEMU/AArch64

Hi, I’m Asami. My project “adding QEMU/AArch64 support to coreboot” is making good progress. I’ve almost done to write porting code and I’m now writing a new tool to make a FIT payload for QEMU/AArch64. Here is my CL. In this article, I’m going to talk about how to run C code in the bootblock stage.

The way to run C code before DRAM has been initialized is various in the coreboot project. The most famous and used in the x86 system is known as the Cache-As-Ram (CAR). CAR can be set up by 1. enable CPU cache 2. enable the ‘no eviction’ mode 3. change cache mode from write-through to write-back. ‘No-eviction’ means that the CPU doesn’t write any data to DRAM as long as the size of data is less than the CPU cache. Then, you can get all the data from the CPU cache. The implementation for CAR is src/cpu/intel/car/non-evict/cache_as_ram.S.

Another way to run C code especially implemented in ARM system is relocating the bootblock code to SRAM. System on a Chip (SoC) has an ARM CPU and often includes SRAM as a cache. For example, QEMU VExpress machine has SRAM is located at 0x48000000 that we can know in qemu/hw/arm/vexpress.c#L120.

However, what should the system that doesn’t have SRAM do? My target machine, QEMU virt machine, doesn’t have SRAM according to the implementation of qemu/hw/arm/virt.c. In this case, should we initialize DRAM earlier?

The answer is No but it’s only for my project. Because QEMU is not actual hardware so DRAM already works. I just need to relocate the bootblock code to DRAM directly. I believe that other ARM systems that have no SRAM should initialize DRAM earlier.

You can see the relocation code in bootblock_custom.S under review. https://review.coreboot.org/c/coreboot/+/33387

[GSoC] Debug Bootblock Stage for ARMv8 on QEMU

Hello again, I’m Asami. I’ve just finished 4 weeks as a GSoC student. I’m currently debugging the implementation of my main project, which is adding QEMU/AArch64 support. I can see nothing output right now when I start a QEMU with the coreboot.rom that has my implementation. It means there is something wrong before a hardware initialization has finished. In this article, I’m going to talk about what I found while debugging the bootblock for ARMv8.

Code Path of Bootblock Stage

The bootblock is executed just after CPU reset and it is almost written by assembly language. The main task is to set up a C-environment. The basic code path for ARMv8 from the beginning the bootblock to the romstage is:

  1. _start() at src/arch/arm64/armv8/bootblock.S
  2. arm64_init_cpu() at src/arch/arm64/armv8/cpu.S
  3. main() at srclib/bootblock.c
  4. run_romstage() at src/lib/prog_loaders.c
  5. prog_run() at src/lib/prog_ops.c
  6. arch_prog_run() at src/arch/arm64/boot.c
  7. main() at src/arch/arm64/romstage.c // The entry point of the romstage

You can use your custom _start function instead of the common _start function by CONFIG_BOOTBLOCK_CUSTOM=y and adding bootblock-y += bootblock_custom.S which is your custom assembly file.

The Reason Why Execution Stopped inside arm64_init_cpu()

I found that an execution stopped inside the arm64_init_cpu function for some reason. The line that has some problem is mrs x22, sctlr_el3 . MRS instruction can read a system control register and store the value into a general purpose register. So this line means to store the value of SCTLR_EL3 into the X22 register.

According to the “ARM Architecture Reference Manual ARMv8, for ARMv8-A architecture profile”, the purpose of SCTLR_EL3 is

Provides top level control of the system, including its memory system, at EL3. This register is part of the Other system control registers functional group.

Also, SCTLR_EL3 is accessible only from EL3 mode. EL3 is the highest privileged mode that a low-level firmware, including the secure monitor, works on it.

Next, I checked the current mode via mrs x0, CurrentEL. CurrentEL is a register that holds the current exception level. The result of CurrentEL was 0x04, which means the program works on EL1 mode. EL1 is the mode that an operating system kernel typically described as privileged. I didn’t have the right to access SCTLR_EL3. That’s why an execution stopped.

Ideas to Solve EL3 Issue

I considered 2 solutions:

  1. Use only EL1 registers
  2. Run QEMU in EL3

Firstly, I tried to use only EL1 registers. I replaced arm64_init_cpu with arm64_init_cpu_el1 that is a new function I created. Then I replaced SCTLR_EL3 with SCTLR_EL1 and TLBI ALLE3 with TLBI VMALLE1. It seems to work well but still, there was nothing output.

Secondly, I tried to run QEMU in EL3 that is enabled by -machine flag. QEMU can work on EL2 with -machine virtualization=on and EL3 with -machine secure=on to enable EL3. The following command works well for me.

$ qemu-system-aarch64 -bios ./build/coreboot.rom -M virt -cpu cortex-a53 -nographic -smp 1 -machine secure=on

My mentor Raul told me the second solution. Thank you so much!

[GSoC] Common Mistakes for Beginners

Hello everyone. I am Asami and a student for this year’s GSoC project. My project is adding a new mainboard QEMU/AArch64 to make it easier for coreboot developers to support new boards for ARMv8. I’ve already written a small patch to enable building a sample program with libpayload for ARM architecture. Also, I’ve read the implementation of coreboot (main code path) for ARMv7 and QEMU (qemu/hw/arm/vexpress.c). Now, I just created a new CL for my main project and I started to read the implementation of the target machine of AArch64 (qemu/hw/arm/virt.c).

In this article, I’m going to talk about my mistakes when I developed coreboot. I hope it helps for beginners of coreboot development. The target board is QEMU/ARM and the CPU is ARMv7.

“ERROR: Ramstage region _postram_cbfs_cache overlapped by: fallback/payload”

I faced this error when I built coreboot.rom for QEMU/ARM with the coreinfo which is a small informational payload for coreboot. The cause is that the coreinfo doesn’t support ARM architecture and then the payload is compiled as a 32-bit x86.

Make sure that your payload is your target architecture. You need to use other executable files instead of the coreinfo when you want to use architectures other than x86. We provide the libpayload which is a small BSD-licensed static library.

The details of the error is:

$ make
...(omitted)....
W: Written area will abut bottom of target region: any unused space will keep its current contents
CBFS fallback/romstage
CBFS fallback/ramstage
CBFS config 
CBFS revision
CBFS fallback/payload
INFO: Performing operation on 'COREBOOT' region...
ERROR: Ramstage region _postram_cbfs_cache overlapped by: fallback/payload
Makefile.inc:1171: recipe for target 'check-ramstage-overlaps' failed
make: *** [check-ramstage-overlaps] Error 1 

“ERROR: undefined reference to ‘_ttb'” and “ERROR: undefined reference to ‘_ettb'”

This errors might happen when you build coreboot.rom by `make` at root directory. In this case, You need to add TTB() at your memleyout.ld.  

TTB is a translation table base address for MMU. TTBR0 and TTBR1 (TTB registers) hold the start point of TTB. We can put TTB anywhere in memory as long as we store the address to TTBR.

According to the “ARM Architecture Reference Manual ARMv7-A and ARMv7-R edition”, the difference between TTBR0 and TTBR1 is:

(B3-1345) When a PL1&0 stage 1 MMU is enabled, TTBR0 is always used. If TTBR1 is also used then:

– TTBR1 is used for the top part of the input address range

– TTBR0 is used for the bottom part of the input address range

https://static.docs.arm.com/ddi0406/c/DDI0406C_C_arm_architecture_reference_manual.pdf

(B4-1724) TTBCR determines which of the Translation Table Base Registers, TTBR0 or TTBR1, defines the base address for a translation table walk required for the stage 1 translation of a memory access from any mode other than Hyp mode.

https://static.docs.arm.com/ddi0406/c/DDI0406C_C_arm_architecture_reference_manual.pdf

TTBR0 is basically used for user processes and TTBR1 is used for kernel. However, Linux kernel only uses TTBR0 to reduce the time of context switch. (I just heard that Linux kernel starts to use TTBR1 because of security reasons such as Meltdown and Spectre.)

In coreboot, mmu_init() sets TTBR registers in arch/arm/armv7/mmu.c.

Fails to build a sample program with libpayload

We provide the libpayload which is a small BSD-licensed static library for coreboot and we also have a sample program to know how to use it. However, you might fail to build a sample program when you select the ARM architecture as a target with the following errors:

/usr/bin/ld: cannot represent machine `arm'

The reason why this problem happens is Makefile in the sample directory is old dated. So I created a CL to update current architectures that coreboot supports.

Please see the Makefile in https://review.coreboot.org/c/coreboot/+/33287

“Payload not loaded”

“Payload not loaded” happens when the load address of a payload is wrong. The load address should be placed in the RAM place where anyone can use. You can define the load address via CONFIG_LP_BASE_ADDRESS if you use a libpayload.

I created a CL for a sample configuration. Please see the config.emulation-qemu-arm in https://review.coreboot.org/c/coreboot/+/33287

Whole operations for building coreboot.rom with a sample payload for QEMU/ARM are:

1. Build a libc and cross compiler environment.

// In coreboot/payloads/libpayload/
$ make distclean // Always needs when switching a mainboard.
$ cp configs/config.emulation-qemu-arm configs/defconfig // Or you can set up it via 'make menuconfig'
$ make defconfig
$ make
$ make install

2. Build a sample payload hello.elf.

// In coreboot/payloads/libpayload/sample
$ make // Make sure that Makefile is updated by https://review.coreboot.org/c/coreboot/+/33287

3. Build coreboot.rom with a sample payload.

// In coreboot/
$ make distclean // Always needs when switching a mainboard.
$ make menuconfig // or make defconfig
  Select payload “payloads/libpayload/sample/hello.elf”
$ make

Make sure to do ‘make distclean’ before switching your board target

‘make distclean’ removes build artifacts and config files. The default archtecture in coreboot is x86, so you need to do ‘make distclean’ when you want to use other architectures.

Fails to update an existing CL on Gerrit 

Gerrit is a code review tool used in coreboot project. I’m familiar with GitHub and I thought the operations of Gerrit are almost the same with the operations of GitHub, but it weren’t.

On GitHub, developers can create a commit for each update. On the other hand, developers using Gerrit need to amend their commit until it will be merged.

Commands to create a new CL are almost the same with the operations of GitHub:

$ git add <target files>
$ git commit -s
$ git push

Commands to update an existing CL are slightly different:

$ git add <target files>
$ git commit --amend --no-edit

Make sure to leave the “Change-Id” line of the commit message as is.