[GSoC] coreboot for ARM64 Qemu – Week #9 #10

In the last post I talked about using aarch64-linux-gnu-gdb and debugging in qemu. In these two weeks I was intensely involved in stepping through gdb, disassembly and in-turn debugging the qemu port. I summarise the major highlights below.

Firstly, the correct instruction to invoke qemu is as follows

./aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -bios ~/coreboot/build coreboot.rom -s -S

After invoking gdb, I moved onto tracing the execution of the instructions step by step to determine where and how the code fails. A compendium of the code execution is as follows

gdb) target remote :1234
Remote debugging using :1234
(gdb) set disassemble-next-line on
(gdb) stepi
0x0000000000000980 in ?? ()
=> 0x0000000000000980: 02 00 00 14 b 0x988
(gdb)
0x0000000000000988 in ?? ()
=> 0x0000000000000988: 1a 00 80 d2 mov x26, #0x0                    // #0
(gdb)
0x000000000000098c in ?? ()
=> 0x000000000000098c: 02 00 00 14 b 0x994
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x0000000000000750 in ?? ()
=> 0x0000000000000750: 3f 08 00 71 cmp w1, #0x2

The detailed version can be seen here.

The first sign of error can be seen here, where the instruction is 0 and the address is way off.

0x64672d3337303031 in ?? ()
=> 0x64672d3337303031: 00 00 00 00 .inst 0x00000000 ; undefined

To find insights as to why this is happening, I resorted to tracing in gdb. This can be done by adding the following in the qemu invoke command. This creates a log file in /tmp which can be read to determine suitable information.

-d out_asm,in_asm,exec,cpu,int,guest_errors -D /tmp/qemu.log

Looking at the disassembly, it can be seen that execution of instructions till 0x784 is correct and it goes bonkers immediately after it. Looking at the trace, this is where the code hangs

IN:
0x0000000000000784:  d65f03c0      ret
The ret goes to somewhere bad. So the stack has been blown or it has executed into an area it should have prior to this. Next, I did a objdump on the bootblock.debug file. Relating to the code at this address, it could be determined that the code fails at “ret in 0000000000010758 <raw_write_sctlr>:”
Next up was determining where the stack gets blown or corrupt. For this, while stepping through each instruction, I looked at the stack pointer. The output here shows the details. Everything seems to function correctly till 0x00000000000007a0 (0x00000000000007a0: f3 7b 40 a9 ldp x19, x30, [sp] ), then next is 0x00000000000007a4: ff 43 00 91 add sp, sp, #0x10 . This is where saved pc goes corrupt. This code gets called in the “raw_write_sctlr_current” (using objdump)
From the trace, we have the following information : The ret goes bad at 0000000000010758 <raw_write_sctlr>:
0x0000000000000908:  97fffe06      bl #-0x7e8 (addr 0x120)
0x0000000000000120:  3800a017      sturb w23, [x0, #10]
0x0000000000000124:  001c00d5      unallocated (Unallocated)
Taking exception 1 [Undefined Instruction]
…from EL1
…with ESR 0x2000000
Which is here:
0000000000010908 <arm64_c_environment>:
   10908: 97fffe06  bl 10120 <loop3_csw+0x1b>
   1090c: aa0003f8  mov x24, x0
This finally gave some leads in the qemu debug. There seems be some misalignment in smp_processor_id.
While tracing in gdb, we have
0x0000000000000908 in ?? ()
=> 0x0000000000000908: 06 fe ff 97   bl  0x120
(which is actually bl smp_processor_id (from src/arch/arm64/stage_entry.S))
Under arm64_c_environment (in objdump) we have;
10908:       97fffe06        bl      10120 <loop3_csw+0x1b>
Also in the trace we have
IN:
0x0000000000000908:  97fffe06      bl #-0x7e8 (addr 0x120)

Now loop3_csw is defined at (from objdump)
0000000000010105 <loop3_csw>:

So this + 0x1b = 10120

Thus it wants to branch and link to 0x120 but smp_processor_id is at 121.

smp_processor_id is at (from objdump)
0000000000010121 <smp_processor_id>:

This gives us where the code is failing. Next up is finding out the reason for this misalignment and rectifying it.

 

 

 

[GSoC] coreboot for ARM64 Qemu – Week #8

As I had discussed in my last blog post, currently I am onto the debug of the qemu boot. I was intending to use Valgrind tools to detect various memory managements bugs and use that information for my debug. But sadly the information provided by Valgrind was not of much use since it didn’t deal with the execution stream of the coreboot code in qemu. I ultimately had to turn to gdb and use it for further debug.

This was an initial hiccup, since, as in my last post, building aarch64-linux-gnu-gdb on MacOSX was not straightforward, since there was no direct replacement for the “gdb-multiarch”. I was able to get this done. I discuss some of the basic steps of how to set it up below.

First, we need a couple to packages to build gdb. They are listed below:

expat guile texinfo

Next, download the aarch64-gdb from here. Now, you need to configure CC to gcc (GNU gcc and not the innate symlink to clang). Then proceed to,

$ ./configure --target=aarch64-linux-gnu
$ make
$ make install

If this completes successfully, you would have aarch64-gdb installed on your system correctly. The important thing to remember is to use GNU gcc (>=4.9) and not the innate MacOS gcc.

To run gdb you must

$ aarch64-linux-gnu-gdb

The output looks like this :

GNU gdb (GDB) 7.9
Copyright (C) 2015 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "--host=x86_64-apple-darwin13.3.0 --target=aarch64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".

Now I had gdb working. Then I did started the debug by giving a “-s -S” while invoking qemu. After this, I need to connect to gdb remotely using

(gdb) target remote : 1234

Some of the debug information I received was this :

(gdb) target remote :1234
Remote debugging using :1234
0x0000000000000200 in ?? ()
(gdb) run
The “remote” target does not support “run”.  Try “help target” or “continue”.
(gdb) continue
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x0000000000000200 in ?? ()

On trying single-step execution on gdb, I received :

(gdb) step
Cannot find bounds of current function
An error like this usually seen when we overflow a buffer and corrupt the stack, the proper return address is destroyed. When the debugger tries to figure out which function the address is in, it fails, because the address is not in any of the functions in the program.
On running the simple where on gdb I get [where displays the current line and function and the stack of calls that got you there]
(gdb) where
#0  0x0000000000000200 in ?? ()

After some unscrambling of the source code using information from gdb, we were pointed to some issues under the stage_entry in src/arch/arm64/stage_entry.S. I am onto re-setting those and continuing the debug further now.  

 

 

[GSoC] coreboot for ARM64 Qemu – Week #7

This was a tough week. After having passed the coreboot building stage, I thought my work would be easier now. But I had another thing coming.

As I had mentioned in my last post, I didn’t have any output while booting on qemu. So, the first aim was to get qemu monitor working. After some debug, I was able to get qemu monitor working to print onto my terminal (stdio)
This gave me then following :

qemu: fatal: Trying to execute code outside RAM or ROM at 0x0000000008000000

R00=00000950 R01=ffffffff R02=44000000 R03=00000000
R04=00000000 R05=00000000 R06=00000000 R07=00000000
R08=00000000 R09=00000000 R10=00000000 R11=00000000
R12=00000000 R13=00000000 R14=40010010 R15=08000000
PSR=400001db -Z– A und32
s00=00000000 s01=00000000 d00=0000000000000000
s02=00000000 s03=00000000 d01=0000000000000000
s04=00000000 s05=00000000 d02=0000000000000000
s06=00000000 s07=00000000 d03=0000000000000000
s08=00000000 s09=00000000 d04=0000000000000000
s10=00000000 s11=00000000 d05=0000000000000000
s12=00000000 s13=00000000 d06=0000000000000000
s14=00000000 s15=00000000 d07=0000000000000000
s16=00000000 s17=00000000 d08=0000000000000000
s18=00000000 s19=00000000 d09=0000000000000000
s20=00000000 s21=00000000 d10=0000000000000000
s22=00000000 s23=00000000 d11=0000000000000000
s24=00000000 s25=00000000 d12=0000000000000000
s26=00000000 s27=00000000 d13=0000000000000000
s28=00000000 s29=00000000 d14=0000000000000000
s30=00000000 s31=00000000 d15=0000000000000000
s32=00000000 s33=00000000 d16=0000000000000000
s34=00000000 s35=00000000 d17=0000000000000000
s36=00000000 s37=00000000 d18=0000000000000000
s38=00000000 s39=00000000 d19=0000000000000000
s40=000000 Abort trap: 6

I did some searching, this meant that the bootloader could not be loaded. And realised maybe the ROM qemu is being allotted is not sufficient. The ‘execute outside ram or rom’ is usually a jump to somewhere that qemu does not recognize as ROM/RAM.
Since we expect
CONFIG_BOOTBLOCK_BASE is 0x10000
CONFIG_ROMSTAGE_BASE  is 0x20000
CONFIG_SYS_SDRAM_BASE is 0x1000000
i.e ROM to start at 64k. So I ran qemu by giving a -m 2048M (for testing) and got over this fatal qemu error, but still wasn’t able to get coreboot to boot (no output on serial). This meant, some more debugging was needed.

I started to debug this using gdb. I created a gdb stub in the qemu boot (by using -s -S), but running gdb to connect to it gave me :
(gdb) target remote localhost:1234
Remote debugging using localhost:1234
warning: Architecture rejected target-supplied description
0x40080000 in ?? ()

Which probably meant I will have to have to build a cross gdb (aarch64-linux-gnu-gdb) and use that.
For this, on linux we could have something called gdb-multiarch, but this is not available for macOSX.

I then turned to using Valgrind. There are Valgrind tools available that can help detect many memory management bugs.

This is what I got on valgrind,

==2070== Memcheck, a memory error detector
==2070== Copyright (C) 2002-2013, and GNU GPL’d, by Julian Seward et al.
==2070== Using Valgrind-3.10.1 and LibVEX; rerun with -h for copyright info
==2070== Command: aarch64-softmmu/qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -m 2048 -kernel /Users/naman/gsoc/coreboot2.0/coreboot/build/coreboot.rom
==2070==
–2070– aarch64-softmmu/qemu-system-aarch64:
–2070– dSYM directory is missing; consider using –dsymutil=yes
UNKNOWN __pthread_sigmask is unsupported. This warning will not be repeated.
–2070– WARNING: unhandled syscall: unix:330
–2070– You may be able to write your own handler.
–2070– Read the file README_MISSING_SYSCALL_OR_IOCTL.
–2070– Nevertheless we consider this a bug.  Please report
–2070– it at http://valgrind.org/support/bug_reports.html.
==2070== Warning: set address range perms: large range [0x1053c5040, 0x1253c5040) (undefined)
==2070== Warning: set address range perms: large range [0x239e56040, 0x255e55cc0) (undefined)
==2070== Warning: set address range perms: large range [0x255e56000, 0x2d5e56000) (defined)

2070 set address range perms means that the permissions changed on a particularly large block of memory.
That can happen because when a very large memory allocation or deallocation occurs – a mmap or umap call. Which meant we are leaking some memory, but we need to find where. I read some documentations and believe something called a massif tool (in valgrind) could be used. I am now looking at how to find where this memory gets eaten.

On the target now is getting some answers on valgrind if possible. But if I dont get sufficient leads, I would have to switch to gdb (aarch64 on macOSX) and continue my debugging.

 

[GSoC] coreboot for ARM64 Qemu – Week #6

This week I worked on completing the build and sorting all complications imposed by it. As I talked in the last post, I was facing some issues regarding setting up smp for this port. I solved this issue by adding an assembly file which declared smp_processor_id and then defined it by setting the right registers. I had to do some background reading on arm64 details. This provided me with the information I needed.

Next up, was another hitch. During the build, ‘mmu_enable()’ and ‘arch_secondary_cpu_init()’ function calls are happening for all stages but the definitions for these functions are getting compiled only for ramstage. So this gave recurrent errors since the compiler couldn’t find these definitions. While attempting to sort this, I stumbled across something on the chromium tree. There was a patch which dealt with some of the issues, similar to mine. I had to cherry pick and apply this change.

After debugging and sorting through some more errors, I was finally able to get it to build successfully.

coreboot.rom: 4096 kB, bootblocksize 37008, romsize 4194304, offset 0x90c0 alignment: 64 bytes, architecture: arm64

Name                                       Offset     Type         Size
fallback/romstage               0x90c0     stage        12108
fallback/ramstage               0xc080     stage        17768
config                                    0x10640    raw          2034
revision                                0x10e80    raw          577
(empty)                                 0x11100    null         4124312
      HOSTCC     cbfstool/rmodtool.o
     HOSTCC     cbfstool/rmodule.o
     HOSTCC     cbfstool/rmodtool (link)
The complete build can be found here.
I attempted to boot off on qemu after this,
$qemu-system-aarch64 -machine type=virt -nographic -bios ~/coreboot/build/coreboot.rom
This did not give any output, which meant probably I had to make some changes in the uart set up. I attempted to debug this by adding few printks early in bootblock once the console_init is done. This process ongoing, I hope to get through. Another aspect in question is the bootblock initialisation. The src/arch/arm64/armv8/bootblock_simple.c calls for an appropriate bootblock_cpu_init(). This is another thing I will be working on in the coming days.

[GSoC] coreboot for ARM64 Qemu – Week #4 and #5

From this week I started dealing with the core aspects of aarch64 design. I continued with the process of building the armv8, along with handling the required patching-up, interfacing, hook-ups. In my last post, I had talked about the toolchain building error (in binutils) for arm64 which I was facing on OSX. I had to remove the –enable-gold flag from the binutils. After making this small update, the build_BINUTILS looked like this, and I was able to get the toolchain working.

build_BINUTILS() {
 if [ $TARGETARCH == "x86_64-elf" ]; then
 ADDITIONALTARGET=",i386-elf"
 fi
 CC="$CC" ../binutils-${BINUTILS_VERSION}/configure --prefix=$TARGETDIR \
 --target=${TARGETARCH} --enable-targets=${TARGETARCH}${ADDITIONALTARGET} \
 --disable-werror --disable-nls --enable-lto \
 --enable-plugins --enable-multilibs CFLAGS="$HOSTCFLAGS" || touch .failed
 $MAKE $JOBS || touch .failed
 $MAKE install DESTDIR=$DESTDIR || touch .failed
}

The work had just began after fixing the toolchain. On attempting to building, the faced error I got was :

toolchain.inc:137: *** building bootblock without the required toolchain. Stop.

This was due to certain wrongly configured CONFIG options in the Kconfig. After this stage the initial bring-up of arm64 looked stable. Moving forward, I was met with an error in src/arch/arm64/c_entry.c

src/arch/arm64/c_entry.c: In function ‘arm64_init’: src/arch/arm64/c_entry.c:52:2: error: implicit declaration of function ‘cpu_set_bsp’ [-Werror=implicit-function-declaration] cpu_set_bsp();

The inclusion of necessary files and structures were correct, and I kept getting this error. Furquan ultimately pointed to change 257370  following which, I could get past this. After this, I had to solve another BSD/OSX issue about date in genbuild_h.sh to get my build progressing.

Subsequently, some architectural decisions had to be made for the armv8. In the initial version, I had been banking on cbfs_media based structure in media.c, for creating functions for read, write and map. But the older formulation (in cbfs_core.c and cbfs_core.h) is changed now. In order to keep up, and to maintain uniformity, we decided to handle this as it is handled in armv7, i.e by creating a mapping to the qemu memory mapped space. Another point of discussion for stage loading. It is being brought up similar to armv7 for now. This might change in the future. Also the organisation for UART was finalised. plo11.c is included, as in  src/drivers/uart/Makefile.inc. by setting the DRIVERS_UART_PL011 in armv8 Kconfig.

Next hitch was dealing with SMP. In my proposal, I had suggested that incorporating smp into the emulation could be a long term goal. But since smp is a part of core arm64 logic, this cannot be completely ignored at this stage. I am met with this

coreboot/src/arch/arm64/stage_entry.S:94: undefined reference to `smp_processor_id’
build/arch/arm64/c_entry.romstage.o: In function `seed_stack’:

 

A cpu file which adds smp_processor_id() is needed at this moment, which I am currently working on. Next week’s plan is to get past these (and meet new unforeseen issues 😛 ) and boot off on qemu.

 

 

 

[GSoC] coreboot for ARM64 Qemu – Week #3

For the week 3, I started to build the coreboot image to run on the target arm64 qemu. As soon as I started, I had to face a big red flag, a toolchain building error! The new toolchain (v1.31) failed to build on MacOSX. Despite several attempts, binutils failed to build and I couldn’t get the toolchain running. /buildgcc also failed on OSX due to missing string search for clang. Marc kindly help me out on this by introducing a small patch on this front. I also tried using the actual gcc and g++ (and not the innate gcc and g++ on OSX which are just symlinks to clang) but couldn’t rectify the toolchain build error.  But strangely, when I tried to build an older toolchain (v1.24) on the same machine, it worked. Confused, I decided to move forward with the build with the old toolchain itself.

I had to make some modifications on the src/mainboard/emulation/Kconfig, to bring up the new emulation mainboard. After this, I was able to successfully generate a configuration by running make menuconfig.

Next step was to make and generate the build/coreboot.rom. I have been facing some errors while building which I have been resolving on the way. The plan for this week is to finish the build. I also want to look at solving the toolchain building issue. Stepan said that the new toolchain hasn’t been tested on OSX, so this should provide some useful insights.

[GSoC] coreboot for ARM64 Qemu – Week #2

I spent the previous week working on the architecture of the qemu port. I made an attempt to dive into the internals of memory mapping for armv8. And then formulated an initial memory map structure for the armv8 port. After that I moved into developing some code. The most challenging aspect of this was moving to and fro between the qemu-armv7 existing port and the foundation-armv8 patches (now depreciated) of chromium and extracting the required modifications. My current work is building on qemu-armv7  taking inspiration on some aspects from the foundation armv8 which cater to the alterations required for 64-bit.

I then moved onto developing the default mem_uart which would be used in our emulation. After writing up a backbone for this new port, Marc suggested I push to gerrit and seek some reviews on it there. This was an important change from the development strategy I followed last year, when I did all the development locally and then pushed the end results to gerrit. This year, I would be following a more dynamic approach, with a continuos review-and-modify development cycle.

My plan for this week involves starting to build the firmware written thus far. I will look to load the built firmware in qemu and try getting some output on the console and finally get qemu-debug up and running.

 

[GSoC] coreboot for ARM64 Qemu – Week 1

To begin with the aim of introducing coreboot for arm64 qemu, the first task I had to accomplish was to set up a qemu aarch64 environment to work on. In this post, I will talk about building qemu and then booting a kernel that allows us to begin experimentation with this architecture.

To begin building qemu, we need a few packages:

pkg-config, libfdt-dev

Next,  we need a qemu version which support aarch64, so I installed qemu 2.3.0.  Here you can also do :

sudo apt-get install build-dep qemu

Since I was building it on a mac, I was required to do a brew install qemu (again, v2.3.0). For mac, it is recommended to use actual gcc rather than the existing ‘gcc’ which is symbolic-linked to llvm-gcc (x86_64-apple-darwin13.4.0/4.9.2/). Going with the innate gcc kept giving me pains, so I downloaded gcc 4.9.2, created a manual link and used it for my build. Moving on, we now need some of the source code;

git clone git://git.qemu.org/qemu.git qemu.git
cd qemu.git
./configure --target-list=aarch64-softmmu

The last command will usually return an error, saying DTC (libfdt) not present. The problem is that qemu tries to search for dtc binaries in qemu/dtc. Even if you install dtc using sudo apt-get install device-tree-compiler, we keep getting this error. So probably you need to have the binaries in qemu/dtc. Doing this in the repo will fix it.

git submodule update --init dtc

Then, run the ./configure command again. The output can be found here. We then have to run a make command,

make

This gives the following ouput. After this successful build, we have an executable ./qemu-system-aarch64 in qemu.git/aarch64-softmmu. I then used a prebuilt kernel image that has a combined initial RAM disk (initrd) and a basic root file-system. It can be downloaded from here.

Then finally, we run this kernel in our generated aarch64 system to find the linux boot sequence and eventually a log in prompt.

qemu-system-aarch64 -machine virt -cpu cortex-a57 -machine type=virt -nographic -smp 1 -m 2048 -kernel ~/Downloads/aarch64-linux-3.15rc2-buildroot.img  --append "console=ttyAMA0"

The boot sequence results as

Welcome to Buildroot
buildroot login: root
# ls
# uname -a 
Linux buildroot 3.15.0-rc2ajb-00069-g1aae31c #39 SMP Thu Apr 24 11:48:57 BST 2014 aarch64 GNU/Linux

This gives us an aarch64 qemu environment with linux on which we can begin building coreboot.

With the development platform ready, I now begin my actual work on building coreboot for qemu arm64. For this week, I look at the ( now obsolete ) foundation-armv8 patchset and begin my development. The first task would be to create an appropriate media structure / functions that I would use.

 

[GSoC-2015] Introduction – Qemu Support for ARM64

Moin!

A new post after a long time. I’ll introduce myself again for the benefit of (few? :P) faded memories. I am Naman, a senior year undergraduate from India. I worked on developing alternate resource-efficient CBFS access patterns for ARM SoCs as a part of GSoC-2014. I am fortunate enough to engage as a GSoC student for coreboot this year too. I have had a busy semester but finally, as the summers arrive, I look forward to dive into the riveting world of coreboot hacking!

This summer I would be working to introduce a qemu target for arm64 architecture supporting coreboot. Currently, In order to work on aarch64, users require to have suitable hardware. This project intends to clear away this constraint. We have a growing aarch64 support with the tegra132 port, but no support for arm64 in emulation. Having a qemu target might be helpful for many projects which aim to arm64 work.

Continue reading [GSoC-2015] Introduction – Qemu Support for ARM64

[GSoC-2014] Payload Loading – Success!

This post marks the completion of the payload re-structuring that I did as a second part of the project. The following summarizes the major highlights of the work:

  • We change the ‘struct payload’ ( in the src/include/payload_loader.h) to have a ‘struct cbfs_media’ and cbfs_file_handle. That way we have the two thing we need to read the contents of the file.
  • In the build_self_segment_lists() we use the above data structures to media->read() the payload metadata. 
  • We use the metadata to segregate segments on the basis of their types and form a linked list of segments for reference later.
  •  Then we do mapping and then decompression for the compressed segments and direct reading for the uncompressed segments.

Some of the major issues were: playing with the data_offset and segment offsets; to get src_address to read to.

I was able to get past all the issues and finally got a successful working boot This completes the revamp of stage and payload loading 😀

During the past week I also worked with gerrit; wherein I submitted my patches and received feedback. I am more involved with the development process of coreboot now and leaving aside some minor glitches; the process was very smooth.