GSoC [infrastructure] : Along the way, something went terribly wrong

I started working with AMD platforms’ infrastructure with high hopes of being able  to better manage the CBMEM setup. While I have a selection of family14 boards to work with, things did not continue so well:

First off, tree had literally thousands of lines of copy-pasted or misplaced AGESA interface code remaining in the tree. A lot of that should have been caught in the reviews, but it appears a few years back the attitude was that if coreboot project was lucky enough to get some patches from an industry partner, the code must be good (as the development was paid for!) and just got rubber-stamped and committed.

Second, the agreement I have for chipset documentation is not open-source friendly. It contains a clause saying all documentation behind the site login is to be used for internal evaluation only. I was well aware of this at the beginning of GSoC and at that time I expected my mentor organisation would be able to get me in contact with right people at AMD to get this fixed. But that never happened. I am also concerned of the little amount of feedback received as essentially nobody in community has fam16kb to test.

So currently I am balancing which parts of my work on AGESA I should and can publish and what I cannot. Furthermore, first evidences that vendor has decided to withdraw from releasing  AGESA sources have appeared for review. In practice there has been zero communication with the coreboot community on this so I anticipate the mistakes that were done with FSP binary blobs will get repeated.

Needless to say the impact this has had on my motivation to further work on AGESA as my efforts are likely to go wasted with any new boards using blobs. I guess this leaves pleanty of opportunities for future positive surprises once we have things like complete timestamps, CBMEM and USBDEBUG consoles and generally any working debug output from AGESA implemented. I attempted to initiate communication around these topics already in fall 2013 without success and it is sad to see the communications between different parties interested in overall tree maintenance have not improved at all since then.

GSoC (board-)status update

Two weeks went past fast. Our industry partners have been kind and I  received some new boards  for coreboot testing.  I think I spent three  workdays on just the initial hardware setups and SPI recovery tools for the boards.

You should already see a posted board status report for Gizmosphere/Gizmo built from coreboot.org toolchain and source tree.  I have Adlink/CoreModule2-GF (aka LiPPERT/Frontrunner-AF) and PCEngines/apu1c4 next on my list to report. Latter board also requires it coreboot sources to be rebased and published.

These are all Agesa Family14 boards, more or less intended for embedded application use. When applicable, I will try to do some demonstration of the capabilities of Sage EDK and SmartProbe as those seem to be somewhat standard commercially available toolkit for AMD Embedded Partners.  It is clear we cannot beat a JTAG based in-circuit debugger with an alternative running over USB, but we may get surprisingly close by improving the command set of the GDB stub we run in ramstage.

Most of my time is still on solving some early memory space mapping issues for AMD platforms. As expected, reviews and test reports there are progressing slowly. My goal with these is to have CBMEM and USB consoles work in romstage too and additionally some boards currently lack S3 resume support and/or have PCI-e reset problems. I expect most of my development to work unmodified for fam15, fam15tn and fam16kb.

As a sidedish there has been more than the typical amount of review work for me on the table. Edward has done an amazing job sorting out the superio spaghetti romcc left us with. There was also a fair amount of discussion around IRQ routing and new Intel FSP ports to look at.

GSoC [early debugging] There is so much source

Not much new or fancy development yet. A time-consuming dive into sources of AMD platforms took place during the last two weeks as I need to fully understand what is going on with the memory management there. We need to find solution for CAR migration on these platforms, and while at it, let’s evolve these boards too for DYNAMIC_CBMEM.

Originally I had scheduled improvements for these platforms for July, but coordination of making changes to vendorcode might become a bottleneck, so I thought I better start early. In this process fifteen (15) or so copies of heap management sources of AGESA boards got combined into one. As a bonus, some family14 and family15 boards are now a bit closer for having S3 suspend/resume support.

I am expecting some more gear for coreboot development arrive later this week, I should then be fully equipped for my GSoC tasks.

GSoC [early debugging] The very short introduction

Oh dear, what did I get into again. My GSoC 2014 project page  gives you an idea of things to expect during this summer of code and seems like I have promised to deliver a lot this time. More like a complete in-circuit-debugging solution of x86 boot firmware over some readily available and low-cost USB hardware. I only scratched the surface with my GSoC 2013 when I did not get much further than a working usbdebug and some intense clean-up and preparation on the CBMEM side.

There has been serious use of usbdebug combined with SerialICE to troubleshoot and/or reverse-engineer proprietary firmwares. Tests have been done to connect GDB stub built into coreboot over BeagleBone even before debug target has initialized RAM.  Also other pieces of my project plan have already seen proof-of-concepts but the quality or the flexibility have not reached the requirements to see them in widespread use in coreboot community.

We can see more practical uses for SerialICE if we could connect QEMU, GDB, SerialIce and radare together, and visualize some of the system bus topologies at runtime. With the amount of support CPU and chipset vendors have shown towards open-source firmware development the last years, I consider this as a key part for any further community-driven mainboard ports on coreboot.

GSoC [early debugging] closure

Time to stop coding was last week and I feel I have barely touched the surface of my original project proposal.

My biggest contribution comes in the form of usbdebug patches: Most mainboards with USB 2.0 EHCI controller should now be able to produce console logging on USB port. More importantly, one has now the option of using some low-cost ARM development kit boards as a replacement for the Net20DC dongle device. Digging into ARM was not really in my original plans and setting up the required environment took more time that I had hoped for.

I had the idea of collecting a trace of PCI configuration in CBMEM. Turned out I first had to do some cleanup on both PCI and CBMEM and while those have been submitted (with a minimal amount of testing) the tracing part I did not start at all. On the other hand, cleanup on CBMEM has enabled timestamp collection and CBMEM console for ramstage for most mainboards and I have a fairly clear vision what needs to be done to extend these to romstages as well.  I think there would be interest to have these features on ARMv7 too, just takes a bit of coordination and access to platform.

On the so-called “coreboot panic room” tasks I did not submit any code. I liked the idea of using some BeagleBone board as a proxy and having the possibility to switch between SerialICE debugging or GDB or pre-OS flash programmer. If time permits, there might even be some integration or interaction with radare coming up next months for SerialICE.

 

GSoC [early debugging] More connectivity

A substantial cleanup on CBMEM initialisation is now under review. Goal is to get timestamps and CBMEM console supported on more/most mainboards, but I do not expect to complete this during the official GSoC period, or within the next two weeks time.

One of the goals I originally had set was to have means to re-program the flash chip from pre-OS environment. It is now clear I will not have time to finish (or even start) this part of the project. The  decision to delay this part was made quite early on, actually. I learned similar work had already been developed as a combination of FILO+libflashrom, and I hope Stefan’s efforts on another GSoC project will help get this code published in near future. I might still try to get the FILO console appear over usbdebug, adding support of USB communication class (CDC / ACM) in libpayload should not be very difficult.

I have saved some of the most interesting and challenging parts last: having SerialICE and GDB run over usbdebug. Hopefully I get to report about those next week.

GSoC [early debugging] Bridging the gap

ARM is now on the table, as I am bridging the coreboot console from USB to gigabit ethernet, using a BeagleBone board equipped with the USB debug device gadget driver. At first I was a bit concerned of all the latency having an USB-to-ethernet bridge software solution on the communication, so it was now good time to do some measurements.

I used daemon called ser2net to redirect coreboot console TTY to TCP (telnet) and enabled timestamping to estimate the time it takes from power-on to entering payload on amd/persimmon with maximum logging (level=spew).

First, connecting with serial UART @115200bps on x86 host this was total of 15 seconds. Of this it spent 4 seconds in AGESA doing some SPI flash operations and during that time there was no console output.

Repeating same using usbdebug on the BeagleBone OTG port, total time was 9 seconds and again 4 seconds was waiting for SPI completion.  I would say we have a rough figure that console output on usbdebug is twice as fast as what super-IO can typically offer.

Driver compilation and patches are updated here: http://www.coreboot.org/EHCI_Gadget_Debug

To use BeagleBone some additional work was needed on coreboot side but BeagleBone Black and other ARM boards where there is no hub between OTG connector and controller should already work.

GSoC [early debugging] USB submission

Seems I am reaching one of my goals of my original GSoC proposal in bringing usbdebug available as a compile option for most the mainboards with compatible chipset.

As I wrote in an earlier blog, it has been more of a refactoring job on existing code rather than creating something entirely new. There are a series of protocol details where I spotted the implementation took some shortcuts and I have attempted to fix those. I also made improvements to better control how the possible debug dongle connection is probed. There is more testing needed there and also it needs to be fixed to not become excessively slow when dongle is not connected or if it is disconnected before OS is loaded.

At the time of writing, my patches have not yet been submitted to master but are available on the gerrit review board. It is likely there will be some minor fixes, so I will not give exact commit hashes one should checkout and merge. In short: checkout and merge the two topic-branches usbdebug-cfg and usbdebug-lib. In addition, for AMD Agesa boards one needs  “AMD AGESA: Place CAR_GLOBAL in BSP stack”.

Now if you only had something to connect it with, I could ask for your help actually testing these changes and finding out if it works! I am still waiting for my BeagleBone Black to arrive to make some fixes to the kernel EHCI debug gadget driver, and the situation with the choice of debug dongles will then improve quite radically. I do have the older BeagleBone and have built custom kernel modules for it and I have started to study how g_dbgp driver interacts with the gadget serial port framework.

What I discovered after picking up a second-hand original BeagleBone was that it does not have its USB port directly connected to the ARM chip, but there is an USB hub chip in between. It might be possible to configure that hub even though our USB requests are limited to a length of max 8 bytes per transaction. The EHCI debug specs do not allow a hub there, but if we can make it work, why not do it?

 

GSoC [early debugging] AGESA woes

I took some days off the project for holiday in mid-July; after that there has been some amount of problem solving / headaches with my main development platform (samsung/lumpy), as the upstream coreboot tree still appears to be broken. There is incompatibility with recent binary blob and even after some local fixes I still lose some early debugging with usbdebug.

SAGE Electronic Engineering has kindly provided me an AMD Persimmon board for coreboot development.  It took a while for me to get it up and running as it turned out even the very basic documentation of the board connectors and jumpers were behind registration and login on AMD website. Board arrived with coreboot + SeaBIOS combination pre-installed. Seems like flashrom utility works and I also have SPI header connector for recovery purposes. So I should be all equipped for my own build and I hope I get lucky with usbdebug on this hardware.

Now I must say I am not a big fan of the AMD vendorcode named AGESA. Things like CBMEM init and CAR setup are done in a way different fashion compared to the implementation without using vendorcode wrappers. It is a heavy reading, and to get early logging via either CBMEM or usbdebug, I will need to master and possibly modify that code too.

On SerialICE parts I have not made the progress the way I originally had planned, meanwhile some cleanups on low-level PCI configuration and minor usbdebug fixes have been merged. There is more to come on that sector. A fairly complex patchset draft on CBMEM, finalizing that should make it possible to enable CBMEM console for all boards, for ramstage at least.

 

GSoC [early debugging] Art of refactor

Your branch is ahead of origin/master by 48 commits.

Yes, I knew this would happen, it has become increasingly difficult to push new work for review on gerrit, as I have dependencies  on existing work waiting for merge. As the pile of un-merged patches increases so does the time I spend with git rebase, so I am hoping for some progress on that side.

My eyes in the local working directory have turned towards SerialICE integration inside coreboot tree. The benefits of this approach are better tree structure, wider hardware support, cache-as-ram and usbdebug.

There are several use-cases to consider:

  1. Compile classic stand-alone SerialICE ROM image with ROMCC, using super-IO and chipset initialisation from coreboot tree.
  2. Compile SerialICE as an alternative romstage with ROMCC, using existing coreboot bootblock added with serial port initialisation.
  3. Compile SerialICE as romstage with GCC and cache-as-ram to use existing usbdebug code and possibly better execution performance.
  4. Add abilility to jump out of SerialICE to regular romstage.

Also for the SerialICE session on debug host we have alternatives:

  1. Execute vendor BIOS image under QEMU.
  2. Execute coreboot image under QEMU.
  3. Execute coreboot image in user-mode under GDB without QEMU.
  4. Execute utils like nvramtool, msrtool, inteltool, superiotool, lspci and setpci remotely.

Now all of the above has been demonstrated before but not adopted. Adopting these widely for all mainboards may not happen during my GSoC, as there is no common function to call to enable a serial-port from romstage. At the minimum I will make some simple example one can follow to get SerialICE running on boards with existing coreboot support.