u-boot as coreboot payload

U-boot is bootloader on ARMs, PowerPCs and other platforms, it has a nice set of commands and in general it feels like a small operating system. I’m not certainly sure if it is good direction, please feel free to compare with UEFI 😉 but I simply miss it on x86. I work at SYSGO with u-boot in daily basis and even port it to different boards/platforms. The x86 is no easy to init and I think this is the reason why there is only one x86 board in whole u-boot tree. This board is called eNET and it has a AMD ELAN SC520 SOC. But luckily, with coreboot we can init much more x86 boards and this leads to natural conclusion to have the u-boot as the coreboot payload. I would like to share with you part of this “fantastic” hacking journey to make it happen.

It started as a real journey, I was traveling to visit my parents and I thought I might spent some time to investigate this “as payload idea”. You can get u-boot sources from git from here. First I checked how many boards are for x86 and I found only eNET which is the SC520. First objective is to have it at least compiled, which is easy if you worked with the u-boot before. Just type make eNET_config ; make and build it. Of course it did not build, there were some warnings which were treated as errors. I fixed that and I got the u-boot.bin and u-boot ELF at the end. So far so good, now second objective is to load it as the payload. The u-boot runs in first stage from ROM (function suffix _f), then it _dynamically_ relocates itself to the RAM and continues init, at least this is what it does on PowerPC or ARM. My idea was simple, change linking to the RAM location even for ROM stage and make a wrapper around u-boot bin so it is just one big .text ELF which can be loaded as payload.

If you worked with u-boot already you might know that there is a directory with the board configuration file in include/configs and architecture specific stuff can be found (surprisingly) in the arch/x86 directory. I started to read the sources to check if u-boot on x86 is doing same thing and I wanted to change the ROM base and maybe the entry point if it is 16 bit. The link ROM is usually set in config file
or in boards.cfg. In our case it was set in the boards.cfg as SYS_TEXT_BASE=0x19000000. Quite strange address huh? I checked the u-boot.lds in arch/x86 to see how it is linked. After short investigation it turns out that at the end is some 16bit start stub and rest of the flash is mapped elsewhere. The last 64kb constants are hidden in the config.mk with very nice warning:


# DO NOT MODIFY THE FOLLOWING UNLESS YOU REALLY KNOW WHAT YOU ARE DOING!
LDPPFLAGS += -DRESET_SEG_START=0xffff0000
LDPPFLAGS += -DRESET_SEG_SIZE=0x10000
LDPPFLAGS += -DRESET_VEC_LOC=0xfff0
LDPPFLAGS += -DSTART_16=0xf800


Oh well we certainly know what we want to do, not sure about the stuff I was just doing 😉 so I changed to make it end just bellow 16MB (minus ROM size 256KB).


# DO NOT MODIFY THE FOLLOWING UNLESS YOU REALLY KNOW WHAT YOU ARE DOING!
LDPPFLAGS += -DRESET_SEG_START=0xFF0000
LDPPFLAGS += -DRESET_SEG_SIZE=0x10000
LDPPFLAGS += -DRESET_VEC_LOC=0xfff0
LDPPFLAGS += -DSTART_16=0xf800


Now distclean, config and make… surprise! it worked and even it did not produce some monster u-boot.bin (4GB which happens if you do not change everything right ;). Now time to look how the beast is started. There is couple of promissing files like resetvec.S, start16.S and start.S and yes, it starts in this order. The start.S calls early board init and then it requires the architecture to turn on CAR.
The CAR code is in sc520 subdirectory which was reduced by me to:


.globl car_init
car_init:
/*
* Done - We should now have CONFIG_SYS_CAR_SIZE bytes of
* Cache-As-RAM
*/
jmp car_init_ret


The start.S ends with call to C board_init_f, which is in lib/board.c where memory is init, console etc and at the end the relocate gets called and the u-boot continues in the RAM. I simply deleted all memory init and left there only a information that I have 64MB of RAM, which I think is qemu default. The sdram.c is now looking like this:


int dram_init_f(void) {
gd->ram_size = 64*1024*1024;
return 0;
}

int dram_init(void)
{
gd->bd->bi_dram[0].start = 0;
gd->bd->bi_dram[0].size = 64*1024*1024;
return 0;
}


Now the last thing to deal is how to jump to 32bit without forcing the 16-bit mode from coreboot? It turns out that in start.S there is already a small stub just at the start of u-boot.bin. Now I need the wrapper to to wrap the u-boot.bin back to the ELF. I simply hacked following and compiled with nasm.


section .text
global _start
_start:
incbin "u-boot.bin"


And linked this with ld to the -tTex 16MB-256KB. I compiled coreboot with qemu emulation and the wraped u-boot as payload. I took qemu and let it run qemu -serial stdio /dev/null -L . with symlink bios.bin -> build/coreboot.rom. Coreboot phase run it loaded my ELF and … a crash complaing of execution from non-RAM or ROM. QEMU has a nice feature to let you debug everything with -s -S options and remote gdb debugging. I set my breakpoint at the entrypoint and started to debug. It went well functions were even executed all looked fine but the function returns ends up in the 0xffffffff. Why so? Stack ESP seems wrong. Quick check for ESP in start.S


/*
* We now have CONFIG_SYS_CAR_SIZE bytes of Cache-As-RAM (or SRAM,
* or fully initialised SDRAM - we really don't care which)
* starting at CONFIG_SYS_CAR_ADDR to be used as a temporary stack
*/
movl $CONFIG_SYS_INIT_SP_ADDR, %esp


And this is set in eNET.h to:


#define CONFIG_SYS_INIT_SP_ADDR (CONFIG_SYS_CAR_ADDR + \
CONFIG_SYS_CAR_SIZE)


Which is

#define CONFIG_SYS_CAR_ADDR 0x19200000
#define CONFIG_SYS_CAR_SIZE (16 * 1024)


No RAM in there. Quickly fixed to base of 256KB and now it produced even some messages! It complained about the division by zero interrupt! I debugged that again and this interrupt was arriving
just right when interrupts were enabled. Hmm strange. I realized that probably PIC is not set right and I verified this with QEMU monitor “info pic” command. I cut and pasted coreboot PIC init code
and gave second try… (using build version from today as an example)


Jumping to boot code at fc0013
entry = 0x00fc0013
lb_start = 0x00100000
lb_size = 0x0001c000
adjust = 0x17ed4000
buffer = 0x17fb8000
elf_boot_notes = 0x0010cf80
adjusted_boot_notes = 0x17fe0f80

U-Boot 2011.03-00213-gd571c02-dirty (Apr 28 2011 - 01:08:24)

DRAM Configuration:
Bank #0: 00000000 64 MiB
Bank #1: 00000000 0 Bytes
Bank #2: 00000000 0 Bytes
Bank #3: 00000000 0 Bytes
## Unknown flash on Bank 1 - Size = 0x00000000 = 0 MB
## Unknown flash on Bank 2 - Size = 0x00000000 = 0 MB
## Unknown flash on Bank 3 - Size = 0x00000000 = 0 MB
Flash: 0 Bytes
*** Warning - bad CRC, using default environment

In: serial
Out: serial
Err: serial
Net: RTL8139#0
Serck Controls eNET
boot >


Oh wow, I did not thought this was possible 😉 I spent rest of my stay in parents place rewriting the stuff to my own “coreboot” architecture and my own “coreboot” board. While traveling back I wanted to add the IDE, so I can actually load the kernel and see if I can boot something. Added IDE as quite easy, just copied stuff from another boards and added following to the coreboot.h board.


#define CONFIG_SYS_IDE_MAXBUS 1 /* max. 2 IDE busses */
#define CONFIG_SYS_IDE_MAXDEVICE (CONFIG_SYS_IDE_MAXBUS*1) /* max. 2 drives per IDE bus */

#define CONFIG_SYS_ATA_BASE_ADDR CONFIG_SYS_ISA_IO_BASE_ADDRESS /* base address */
#define CONFIG_SYS_ATA_IDE0_OFFSET 0x01F0 /* ide0 offste */
#define CONFIG_SYS_ATA_IDE1_OFFSET 0x0170 /* ide1 offset */
#define CONFIG_SYS_ATA_DATA_OFFSET 0 /* data reg offset */
#define CONFIG_SYS_ATA_REG_OFFSET 0 /* reg offset */
#define CONFIG_SYS_ATA_ALT_OFFSET 0x200 /* alternate register offset */

#define CONFIG_SUPPORT_VFAT
/************************************************************
* ATAPI support (experimental)
************************************************************/
#define CONFIG_ATAPI /* enable ATAPI Support */

/************************************************************
* DISK Partition support
************************************************************/
#define CONFIG_DOS_PARTITION
#define CONFIG_MAC_PARTITION
#define CONFIG_ISO_PARTITION /* Experimental */

#define CONFIG_CMD_IDE
#define CONFIG_CMD_FAT
#define CONFIG_CMD_EXT2


I created 32MB VFAT disk image (no partition) and put there the ubuntu vmlinuz kernel. I figured out quickly how to load it:


fatload ide 0 3000000 bzimage
zboot 3000000


The base is 48MB because I thought I have no protection over the u-boot / legacy areas. I simply fired this commands and received again that execution ends
outside of memory. Bus came to Prague and this was the end. It was like a detective story, too bad I had nearly no free time to continue with that. I had some plans for Friday night but well, it turned out that I will have free time 😉 So, I wanted to check why I have condition. I traced again everything with gdb. It was bit harder because it relocated and I had to load debug symbols to different place (add-symbol-file command). It was failing after copying
the realmode switch trampoline to the 0x7E0. The code was simply not there. I found out quickly that there is something wrong with relocated section address which gets copied and changed the sign from plus to minus.


long realmode_start = (ulong)&__realmode_start - gd->reloc_off;


This has to do with the fact that now the u-boot ROM is at 16MB and relocated to end of 64MB which tells that the relocate function check is different. Also this realmode_start section is loaded to different
address then it is linked (VMA is 0x7c0 and LMA is just after 16MB). I recompiled my code and coreboot image and … this change was not enough and nothing happend. I realized that I need same change for the legacy BIOS emulation which is u-boot. In this section the memory sizing is emulated (not the e820 but only 81 I think) and there is a PCI BIOS emulation. Afterwards, again recompilation and …
SIGSEGV from QEMU!. Very nice! What now… Now it got little boring because I had to step through all stages of bzImage loading and I eventually got into the kernel with identity paging enabled! So far so good! Crash is happening in the real kernel somewhere!

I tried to compile the kernel with early console and with debug stuff but console did not show anything. I verified that u-boot passes parameters to kernel with console=ttyS0,9600 but I suspected that
the crash is happening too early. I found the kernel print buffer address and I used gdb to dump it to file. This is what I was seeing:


Linux version 2.6.32.24 (ruik@ruik) (gcc version 4.4.5 (Ubuntu/Linaro 4.4.4-14ubuntu5) ) #3 Fri Apr 29 19:28:32 CEST 2011
KERNEL supported cpus:
Intel GenuineIntel
AMD AuthenticAMD
NSC Geode by NSC
Cyrix CyrixInstead
Centaur CentaurHauls
Transmeta GenuineTMx86
Transmeta TransmetaCPU
UMC UMC UMC UMC
BIOS-provided physical RAM map:
BIOS-e801: 0000000000000000 - 000000000009f000 (usable)
BIOS-e801: 0000000000100000 - 0000000004000000 (usable)
DMI not present or invalid.
last_pfn = 0x4000 max_arch_pfn = 0x100000
initial memory mapped : 0 - 01400000
init_memory_mapping: 0000000000000000-0000000004000000
0000000000 - 0000400000 page 4k
0000400000 - 0004000000 page 2M
kernel direct mapping tables up to 4000000 @ 7000-b000
64MB LOWMEM available.
mapped low ram: 0 - 04000000
low ram: 0 - 04000000
node 0 low ram: 00000000 - 04000000
node 0 bootmap 00001000 - 00001800
BUG: Int 6: CR2 (null)
EDI (null) ESI (null) EBP 00001800 ESP c1129ef4
EBX c115ad80 EDX 00000006 ECX 0000009f EAX c0001000
err (null) EIP c1147bf6 CS 00000060 flg 00000046
Stack: fffffcc1 35000004 c118e15a c11558ac c1129f2a (null) (null) c115ad80
(null) c1147d6a (null) (null) (null) 00001000 c1114710 (null)
(null) (null) (null) (null) (null) 00004000 c1147eaf (null)
Pid: 0, comm: swapper Not tainted 2.6.32.24 #3
Call Trace:
[] ? hlt_loop+0x0/0x3
[] ? __free+0x6e/0x7f
[] ? mark_bootmem_node+0x92/0x9b
[] ? free_bootmem_node+0x2b/0x2e
[] ? free_bootmem_with_active_regions+0x5a/0x8d
[] ? setup_bootmem_allocator+0x11a/0x140
[] ? setup_arch+0x45d/0x4b0
[] ? reserve_early_overlap_ok+0x3f/0x47
[] ? start_kernel+0x61/0x230
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@


Int 6 is invalid opcode. Why I have invalid opcode while doing some early boot memory manager init? I don’t know. I started to check the kernel to see why. It took me two hours and I found it:


static void __init __free(bootmem_data_t *bdata,
unsigned long sidx, unsigned long eidx)
{
unsigned long idx;

bdebug("nid=%td start=%lx end=%lx\n", bdata - bootmem_node_data,
sidx + bdata->node_min_pfn,
eidx + bdata->node_min_pfn);

if (bdata->hint_idx > sidx)
bdata->hint_idx = sidx;

for (idx = sidx; idx node_bootmem_map))
BUG();
}


Guess how the BUG() is implemented! Yes it emits the ud2 opcode, which is yes you know it – invalid opcode. What is happening here is that the region it tries to reserve is already reserved. But why. I really want to get to sleep now. But this is still soo interesting. There is a bit field defined as pointer which puts there all 1 at the beginning:


memset(bdata->node_bootmem_map, 0xff, mapsize);


First I thought it gets somewhere twice cleared but it turned out that this memset simply refuses to work. The memory is not filled with 0xff !!!! In fact it looks like this after array is done. Dumped with gdb again:


00000000 3f 00 00 00 ff ff ff 00 ff 00 10 00 ef 00 ff ff |?...............|
00000010 00 00 00 00 ff ff 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000100 ff ff ff ff ff ff ff ff ff 1f ff 1f ff 1f ff 1f |................|
00000110 ff ff 3f ff ff ff 3f ff ff ff ff ff ff ff ff ff |..?...?.........|
00000120 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
*
00000140 ec ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
00000150 ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
*
00001000


WTF is going on here? Time to check the MTRR settings in coreboot. There is none. But it seems it is not a problem because qemu just do the regs and no function to it. I want to go sleep! Why this is
happening! I simply deleted the BUG() and let the kernel run. It run for a quite a while and then I got SIGSEGV from QEMU again. Is this QEMU bug???? One interesting fact is that this bitmap is the
second page in memory (virtual address 0xc0001000). In the bright moment I tried to run “mtest” command for this particular physical address (+4KB base) and it failed too! Why why why why!
Do you know that u-boot has build in PCI command?


boot > pci
Scanning PCI devices on bus 0
BusDevFun VendorId DeviceId Device Class Sub-Class
_____________________________________________________________
00.00.00 0x8086 0x1237 Bridge device 0x00
00.01.00 0x8086 0x7000 Bridge device 0x01
00.01.01 0x8086 0x7010 Mass storage controller 0x01
00.01.03 0x8086 0x7113 Bridge device 0x80
00.02.00 0x1013 0x00b8 Display controller 0x00
00.03.00 0x10ec 0x8139 Network controller 0x00


I don’t remember the bright thought I was very tired but I started to check the PCI BARs and yes you know now… The VGA BAR is overlapping the QEMU RAM, making it partly read only. I added the
-vga off and kernel panicked with the “unable to find rootfs” which was perfectly good for me and sleep was ahead.

Does your coreboot + QEMU has same problem? It cost me lot of time to figure it out. The BARS looks good in the print:

PCI: 00:02.0 resource base fc000000 size 2000000 align 25 gran 25 limit febfffff flags 60001200 index 10
PCI: 00:02.0 resource base fe000000 size 1000 align 12 gran 12 limit febfffff flags 60000200 index 14


But for some reason the BAR1 seems to get corrupted, if printed using u-boot.


base address 0 = 0xfe000008
base address 1 = 0x00001000


See the 0x1000, this is exactly address of second page…

PS: btw patrickg told me on IRC that this u-boot ELF can be loaded from coreboot and yes it works 😉

EOF