coreboot

[GSoC-2014] [cbfs_media] Stage 2

As per the plan, we set out to investigate the decompression algorithm that is being employed. Mid-way through, we found something interesting that appealed to us. That was the payload loading process. In our existing architecture, we memory map the entire payload. The selfload()’s current API assumes the payload has already been memory mapped. That’s the bad assumption that needs to change. Even if we investigate and resolve the decompression algorithm, and get a pipelined architecture relying on smaller buffer size, still this mapping will cost resources. Hence this needs to be rectified before we go on the the decompression thingy. So we decided this is what we will be targeting.

I then spent some time trying to break the payload loading and see where we can put some map() saving efforts. Below shows some details of the process:

1. First we locate the payload. Here the process for stage loading happens where we have the following:

default_media->open()

Reading done. size = 24 bytes

Load entry 0x14440 file name (32 bytes)…

Mapping size is equal to 32

Found file:offset = 0x14478, len=95716

CBFS: Found file.

default_media->map(0x14478, 0x1761c)

Mapping size is equal to 95804

CBFS: located payload @ 7ec14298, 95716 bytes.

Thus we map the all the segments with this one big mapping.

2. Loading procedure begins by building the segment list (build_self_segment_list() does this) Here w.r.t. payload_segment_types, we check for proper destination address, file_size etc. After checking for the segment; we do simple aligning by pointing the prev and next pointers appropriately; to reach to the further places where to place the segment; and so on.

3. In load_self_segments() we run a simple for loop covering all segments. The loading that happens is

(i) A PAYLOAD_SEGMENT_CODE

–> Loading segment from rom address 0x7ec14298

code (compression=1)

New segment dstaddr 0x4a000000 memsize 0x39929 srcaddr 0x7ec142d0

filesize 0x175ac

(cleaned up) New segment addr 0x4a000000 size 0x39929 offset 0x7ec142d0 filesize 0x175ac

(ii) Next a PAYLOAD_SEGMENT_ENTRY

–> Loading segment from rom address 0x7ec142b4

Entry Point 0x4a000000

(iii) After this we come to load_self_segments()

First a bounce buffer is created: Bounce Buffer at 7ffcf000, 186192 bytes.

We have one segment that is worked upon; which is compressed hence ulzma(src , dest) reads it.

–>Loading Segment: addr: 0x000000004a000000 memsz: 0x0000000000039929 filesz: 0x00000000000175ac

Post relocation: addr: 0x000000004a000000 memsz: 0x0000000000039929 filesz: 0x00000000000175ac

using LZMA

After that one segment that we see on the logs it says

–> Loaded segments

Hence process complete. In essence we had only 3 segments.

Currently, I am working on a strategy on deciding how to modify the architecture of the API so as to conserve as much sram memory consumption as possible.

[GSoC 2014][cbfs_media] Stage 1 : Mission Accomplished

Firstly, sorry for the delay in posting update on the work. I had been busy getting the design to code and wanted to post after its successful completion.

As I had talked about in the previous post, we did a detailed analysis on the existing read() and map() calls. The original log; with all the extra gibberish removed can be seen here. The first design modification that was done was to remove the mapping done for getting cbfs_header. These were the 0x20 size mappings we see in the log. These were unnecessary and could be done away with. And we did! 😛 This log shows the first optimized build; Stage 1 -> Part 1 ->done.

Now we moved on to the more complex and colossal mappings. A function cbfs_find_file() was created, which returned the absolute data_offset of the file based on the name and type we ask for. Once we have the whereabouts of the file; modifications were made in cbfs_load_stage() to appropriately read() and/or map() various files.

The files are arranged as -> [ cbfs_file ] [ cbfs_stage ] [ data ] <Thanks Aaron for this visualization >

cbfs_find_file() : worked with the cbfs_file to get details about the whereabouts of the file

cbfs_load_stage() : we first read fundamental information about the stage; and then do corresponding map() or read()

Voila!! Stage 1 Complete! 😀

Now, the major issue we have persisting is that the decompression of file data assumes memory mapped access to its contents, and hence is quite inefficient due the that ‘one’ large buffer. SO this is what we tackle next, to be more precise, have a pipelined decompression strategy which would eliminate the need for one large data buffer.

Its getting fascinating to work on the project by the day! Until the next post, signing off.

P.S. Thanks Aaron for helping out with any and every issue I face, and always finding the time to reply, even on sundays! 😀

GSoC 2014 [cbfs_media] Updates

This past week went into looking at the internal working of the cbfs_media interface. Some of the major observations were:

Locations of map() and read() calls
No read() calls at all. Also for the map() calls that were made, there weren’t any unmap() calls.

Size of mappings
The entire cbfs is pulled into the iram. There is a map call which puts about 28KB into the sram, to load romstage. The a10 has an sram of 32KB, hence we are using up most of the necessary ram.
The sequence followed is open() -> map()’s -> close().

Total Resources
Is just the sum of all the mappings, since there are no unmaps to subtract. This gives a benchmark to work upon. Now resource utilization is calculated each time coreboot loads, automatically and progress can tracked.

Now we are giving some thoughts on how to reduce the size of the mappings, one possibility being defining a limit (bound) on its size. What is happening currently, is the size is determined dynamically and hence some mappings are quite large. If we define a bound on it, and then repeat call ‘smaller’ map()s instead of one big one, that could do the job. But this wont always work as the decompression algorithm (LZMA) expects memory-mapped access to the entire compressed buffer. By the end of this week, we hope to strike a workaround this and get a more resource-efficient cbfs interface.

GSoC 2014 [flashrom] Rise like a phoenix

The best plan is no plan at all, then everything goes according to the plan. (Jacek Bukowski)

I am 142% on track according to my proposal so far if you take the quote above seriously. But even if you don’t the progress so far is way better than I would have hoped for when I applied. I made 22 commits so far during the official coding period (starting with 2014-05-19 or within ~16 days), and 49 if you count from the date of project acceptance (2014-04-21, i.e. ~37 days). That’s pretty significant for a project that normally does 100-150 commits per year. Not all of the patches were authored by myself, and some of them were bitrotting for an extended period of time before I revived them. I have also reviewed, rebased and merged patches sent in by foreigners as well as long-term contributors. Even Carl-Daniel awakes periodically from hiatus and helps where he can. So flashrom seems to be pretty vivid again, yay.

Two of the most interesting sets of changes are the eventually committed support for AMD’s Yangtze-based SPI controller (found in Kabini and Tamesh), and finer-grained display of support/test status of hardware. For example we can now clearly indicate if a flash chip is actually a ROM that can only be probed and read. This has become necessary when we added our first ROM chip, Macronix MX23L3254. The results can be seen in the wiki as well as flashrom’s -L output. Besides that there were also various new chips and even two programmers committed with another 1-3 waiting.

It is still a bit early to nail down any fundamental changes I want to tackle, but I think I’ll manage to rewrite the probing algorithm completely this year. I have made one huge step towards that by getting rid of the .probe_timing field of struct flashchip. It was used by some probing functions of parallel chips but there were only about 5 concrete values so they could easily be wrapped into stub functions. This is – as Carl-Daniel correctly stated – not what we normally do… replacing data in tables with code, but in this case I think it is completely justified. The next big step will be to turn the probing loop inside-out. Currently we iterate over all flash chips, filter the ones compatible with the selected programmer, and execute the respective probing function stored. Naturally this creates the same results over and over again because there is only about a dozen different probing functions. That does not only take a lot of unnecessary time but also makes it hard to support more than one probing function. It should be the other way round: we should iterate over all possible probing functions and match the results with the stored flash chips. I will report back how far I got with that in my next post.

cbfs_media [Week 1]

This post covers the complete set-up and building coreboot for cubieboard. Due to lack of documentation for this, I had to spend sometime figuring out the details; hence decided to write it myself to help others in the future.

Step 1: Build Payload

As mentioned here, what we have to do first is to build a payload to use later for coreboot.rom. A suitable ARM payload is the sunxi/uboot. Now, there are two ways to build uboot: natively or from another system. To build from another system, we need to get a suitable toolchain. For that you need to do:

apt-get install gcc-arm-linux-gnueabihf

Too many issues are faced to get this toolchain set up right 😐 A more suitable and convenient method is to build uboot natively from the cubieboard itself (thanks #cubieboard for the tip :P). For this follow the instructions here. tl;dr Clone repository, choose you target board, make (without CROSS_COMPILE). This completes building the payload. NOTE: The correct file to use as the payload is “u-boot”, not u-boot.bin. The “u-boot” file is the non-SPL part of uboot in elf format. The log for successful build can be seen here.

file u-boot
u-boot: ELF 32-bit LSB shared object, ARM, version 1 (SYSV), dynamically linked (uses shared libs), not stripped

2. Step 2: Build coreboot

Before following the instructions on the coreboot/Build_HOWTO, you first need the latest development code; which contains the mmc driver needed to load romstage, etc.

You need to make crossgcc first. This might take a lot of time: be patient 😛 Some missing toolchain errors can arise. Get past them by:

apt-get install bison flex patch
add-apt-repository ppa:linaro-maintainers/toolchain

Once this is done, set your suitable configuration in make menuconfig. Make sure to disable CONFIG_VGA_ROM_RUN (set by default), since it doesnt work for ARM boards. Just make and wait. Your coreboot.rom is ready. 🙂

This image needs to be placed on the SD card:

dd if=build/BOOT0 of=/path/to/sdcard/blockdev bs=1024 seek=8

Now pfff! this is enough to get coreboot up and running! 😀

For the next week; our plan is to Identify locations of the map() and read() calls; and to determine size of each map(). The driver is currently configured to pull the entire cbfs into ram; so we work to reduce size of these mappings.

[Pre-GSoC] Set up Phase – II

Hi All! This is in continuation with the previous blog post regarding set up. This post will contain the brief walk-through the process of setting up the cubieboard.

The first step is to burn the linux image on the micro SD card, and then use this micro SD card to boot the board. The BerryBoot interface is quite user-friendly and we can follow the instructions on it to successfully complete the process. Details for this can be found here.

The latest set of linux distributions that are available can be seen when we power on the board with the micro SD card inserted. This image depicts the screen as visible:

After completing the installation process (takes about 10 minutes) we successfully install Linaro Ubuntu 2012.11

The Home screen looks like this:

Thus, the set-up process is completed. A schematic for the cubieboard can be found here. Also, a cool video with the whole process can be seen here.

It is 20th May, and now its time to get cracking on the summer of code! 😀 My abode for the next few months:

GSoC [early debugging] There is so much source

Not much new or fancy development yet. A time-consuming dive into sources of AMD platforms took place during the last two weeks as I need to fully understand what is going on with the memory management there. We need to find solution for CAR migration on these platforms, and while at it, let’s evolve these boards too for DYNAMIC_CBMEM.

Originally I had scheduled improvements for these platforms for July, but coordination of making changes to vendorcode might become a bottleneck, so I thought I better start early. In this process fifteen (15) or so copies of heap management sources of AGESA boards got combined into one. As a bonus, some family14 and family15 boards are now a bit closer for having S3 suspend/resume support.

I am expecting some more gear for coreboot development arrive later this week, I should then be fully equipped for my GSoC tasks.

[Pre-GSoC] Set-up Phase-I

Hi! This past week went in gearing up and getting ready for the Coding phase to start. I intend to use the community bonding period for set-up of necessary hardware and studying the 1000 SLOC of the existing cbfs_media interface.

I had ordered my cubieboard, the back-bone for this project last week, to avoid any hassle due to unavailability of hardware when the coding phase actually started. I received it this week. Along with it, all other supplements like the micro-SD cards, card-reader, HDMI-cable, etc were gathered. Hardware setup : Check! 🙂

The Cubieboard has 4GB of internal memory (NAND Flash), which comes pre-loaded with Android 4.0.4, on power-on we can see it operating. For our purpose, we would need linux on it. So, the next step is to run Linux on this board. BerryBoot makes this very easy. We use the manual installation using SD card image, found on that link. A very good article that I found, which describes the set-up really well can be found here.

I would write another post next week that will show some of the results of this installation.

Also, I am spending some time reading the existing cbfs_media interface, can be found here. The regions patch by Aaron highlights the direction we would be taking, with the aim of reducing the buffer size for mappings, i.e. the aim of the project, as explained in the previous post.

This Bonding Period = Spending time on the IRC + Initial Set-up + Slogging through source codes 😛 + Writing blogs (a first for me 😀 )

GSoC [early debugging] The very short introduction

Oh dear, what did I get into again. My GSoC 2014 project page gives you an idea of things to expect during this summer of code and seems like I have promised to deliver a lot this time. More like a complete in-circuit-debugging solution of x86 boot firmware over some readily available and low-cost USB hardware. I only scratched the surface with my GSoC 2013 when I did not get much further than a working usbdebug and some intense clean-up and preparation on the CBMEM side.

There has been serious use of usbdebug combined with SerialICE to troubleshoot and/or reverse-engineer proprietary firmwares. Tests have been done to connect GDB stub built into coreboot over BeagleBone even before debug target has initialized RAM. Also other pieces of my project plan have already seen proof-of-concepts but the quality or the flexibility have not reached the requirements to see them in widespread use in coreboot community.

We can see more practical uses for SerialICE if we could connect QEMU, GDB, SerialIce and radare together, and visualize some of the system bus topologies at runtime. With the amount of support CPU and chipset vendors have shown towards open-source firmware development the last years, I consider this as a key part for any further community-driven mainboard ports on coreboot.

GSoC 2014 [flashrom] Do I need to introduce myself?

Hello, my name is Stefan Tauner and I am participating for the third time in GSoC with a flashrom-related project. If you don’t know me then you have to be new here and should write an introduction post instead of me. 🙂 If you really have no idea then please take a brief look at my introductory posts from last year.

I finally managed to finish my master thesis shortly after this year’s accepted GSoC proposals were announced, so this year will be the last time I’ll participate as a student – unless I find time to do it again during my potential PhD project…

Anyway, instead of writing about my goals for this year, I want to share a few graphs related to flashrom development. The first one draws the number of lines added by individual contributors over time. Note that this includes all patches a developer submitted to our Subversion repository even if he is not the original author. The more horizontal a lines is the less active the committer is. Sadly there are a lot of horizontal lines…

I was curious how my hiatus during my thesis work was affecting flashrom development so I created two graphs spanning the last year. This includes not only the final months of my thesis but also last year’s GSoC. One diagram shows the number of average commits per day each month and the other one depicts the emails sent to the flashrom mailing list on average per day each month. The bars for May 2014 are based on only 4 days worth of data, and I really hope that at the end of May it would not show about 25 Emails per day anymore or I’ll have to work on my project from a rehab facility. 😉

In my next post I’ll lay out my plans for this summer. My (in)formal proposal can be found here. I was quite surprised that it has been accepted instead of my second and more focused proposal for a GUI tool to prepare coreboot images. OTOH, after a short trip into C++ land at the end of my thesis, I am quite delighted that I don’t have to wrestle with Qt and the like…