coreboot

Experiments of mind

The time for writing code is over. The time to design hardware is over. After seven weeks, the beginning has come to an abrupt end. I am severely behind schedule. In week seven I was supposed to implement erase functionality — tell the programmer how to erase the chip. This is not done. On the other hand, I have had code for weeks 8 and 9 almost ready, and just merged most of it last week. So, where am I? Am I ahead or behind schedule?

The fallacy of preemption

One of the requirements for applying as a GSoC coreboot student was to have a fully established, schedule from day[-1]. Establishing this schedule was a great experience, and it allowed me to think in depth about the problem and possible solutions — to a certain degree. I picked the steps I considered logical, in the order which I saw logical. Development is never about writing code in the order in which it will be executed. In this particular case, it was much easier to implement bulk writing without a predefined erase/write strategy, opting instead for a default just-do-it approach.

Why is this approach better than following the schedule, from a development point of view? We have had bulk read partially working for a while now. From the host point of view, reading and writing are symmetrical operations. The bulk of the code (pun definitely intended) is shared between the read and write operation. They both juggle data on the same endpoint. The only difference is the endpoint direction bit. It therefore made sense, once bulk reading was fixed for corner cases, to uses the same code to send data to the programmer. Making the programmer write that data was a matter of a couple of hours. There was no sensible reason to wait an additional two weeks before implementing this last bit.

Software development work is as much about making things work, as it is about the application of programming principles with unquestionable moral authority and correctness. In this case, implementing a trivial extension reusing code fresh in my mind was the preferred approach. Not only did it save me time by not having to re-examine the situation a few weeks from now, it also allows me to have a working program/verify scenario when implementing the erase strategies. As one might imagine, this makes the problem a lot easier. Attempting to preempt and enforce a schedule before the problem is thoroughly explored, occasionally conflicts with best practices of development. With this in mind, I am neither behind, nor ahead of schedule. I am precisely where I need to be.

A matter of experimentation

Most of the infrastructure and code is already in place. Bringing QiProg to completion is no longer an issue of adding functionality through code, but rather completing functionality by connecting the existing code. One issue I discovered after testing the bulk program code was a terrible race condition between read prefetching and the write loop. The prefetch logic incremented the internal address before data arrived. As a result, the new data would get written at the wrong address. Choosing the best solution to the problem is a matter of experimentation.

The “this won’t work because of that” and “what if this” turned into a series of exhausting thought experiments. I have been bugging Peter a lot in the past few days about a series of potential issues. Through tiring thought experimentation, we eventually agreed that the best way to proceed was to abstract a lot more through the API. This is a non-exhaustive list of the decisions we’ve made in the past week:

set_address() is hidden from the API
the internal address range is not exhausted once read or written
read and write operations must not be interdependent, the internal read and write pointers will be distinct (as a side effect, this change also eliminates the race condition depicted above)
set_address() + readn() turns into read(dev, where, n)
All API addresses begin at 0. The programmer translates that into an absolute address
new API call set_chip_size()
new API call to explicitly erase blocks or sectors (to be defined)
implicit erase on write can be enabled or disabled (to be defined)
implicit erase will erase the sector/block right before the first byte of the sector is written
exposing any USB specific dependencies in the API is strictly forbidden

My focus for the remainder of this week will be to shorten this list as much as possible. Once the dependency between read and write is unshackled, I will be able to erase/program/verify my faithful SST 49LF080A. From here, it will be a matter of finalizing and implementing the last obscure bits of the specification.

The state of QiProg for flashrom

As QiProg is still being finalized, implementing it as a flashrom programmer is still a long ways ahead. I do estimate that weeks 11 and 12 will provide ample time to integrate everything into flashrom, hopefully, in time for the 0.9.8 release.

GSoC 2013 [flashrom] blog post #5

The running gag that is not funny at all: new flashrom release will be out very soon! 😛

There are mainly two reasons why it was not released yet. First we have found a regression due to the inclusion of flash chips bigger than 16 MB and then I was working on adding decent support for a new SPI programmer before release. Continue reading GSoC 2013 [flashrom] blog post #5

GSoC (coreboot): Week 5-7 – Redesigning the test-interface-board

In the last blog post I talked about the test-interface-board. There were some concerns about the use of FT232H chip and extra features like voltage and temperature measurement were not required. Keeping in view all the suggestions and improving things on my side I have redesigned the board.

The figure above shows only one target board connected to the test supervision for simplicity. The purple L-block is the test-interface-board. This time, instead of using an integrated flasher I wish to support all existing flashers so I have developed an ICP adapter. An In-Circuit Programming (ICP) adapter contains the firmware flash memory removed from the motherboard and it acts as an electronic change-over switch that connects this flash memory to the motherboard or the programmer as required. It also includes logic-level translation to support as many as possible combinations of programmers and motherboards. The support for programmer voltage is in range 1.2V-5.5V and for motherboard SPI voltage it’s in range 1.65V-3.3V.

As a quick test for VGA initialization I’ve added video detection for Analog video (VGA), DVI and HDMI ports. This checks presence of signals that are only there when video is active. In other words, it checks whether a monitor’s status led would turn green if it’s plugged to the motherboard without actually plugging one. I have prepared schematics, some parts of which are yet to be validated through prototyping. This will all be implemented as an Arduino add-on board (aka Shield) for convenience and the Arduino would act as the controller.

The Serial and Ethernet ports can be connected in the usual way using Serial-to-USB cables, Ethernet cables and hubs. Continue reading GSoC (coreboot): Week 5-7 – Redesigning the test-interface-board

VultureProg command center

It was time to upgrade. I got a new desk last week. For me, a piece of furniture is as boring as watching stainless steel rust (it does rust eventually). A desk, on the other hand, is anathema to replace. I have 20+ wires connected to my workstation going in all sorts of places. I like to keep these wires carefully routed and out of sight, a task made all the more difficult by the lack of any wire management gizmos on new desks. Consequently, after I get a new desk, you are well within your rights to imagine me grabbing my drilling machine, a set of saw cutter bits and an assortment of hole covers — and you need not wait long to see it. Needless to say, getting back up and running has taken a few days, but we are back with a shiny new VultureProg command center (pictured above for your viewing convenience).

The 80-20 rule

In 2011 I used to work developing Android apps. My boss told me “you will spend 20% of the time writing 80% of the code, and you will spend the remaining 80% of the time writing the remaining 20% of the code”. Sadly, QiProg has proven to not be an exception to this rule. Although I have consistently fixed issues and improved the layout of the code, the number of lines of code has been more or less stagnating after the first three weeks of exponential growth. I know how to erase the chip, I know how to program the chip, I know how to read the chip, and I have code to do all that. I just have not yet had the chance to hook it up to the ecosystem.

I am currently working on improving the bulk reading code. There are a few corner cases which are not well handled. As expected, it is taking a lot of time, and I even had to write a special testcase before I started.

Did you want fries with that?

Although we know how to handle every single aspect of the identify/read/erase/write/verify cycle, finding a way of connecting everything together in a simple, elegant, and efficient manner is a different story altogether. We have three API calls for handling erase and write, namely qiprog_set_erase_size, qiprog_set_erase_command, and qiprog_set_write_command. Peter wants everything to work without requiring an explicit erase. In his view, the VultureProg firmware should automatically erase a sector that is being written. Although this would simplify dumb cases where the whole chip or large sections of a chip are written at once, it spells disaster for flashrom’s aggressive optimization. What happens if the first part of a block matches contents, and flashrom decides to write the other part without needing an erase? VultureProg would erase the entire block, then write the second part to the block we never meant to erase. Questions such as this one need to be carefully thought over and elegantly answered. If I blindly start putting all the code together right now, I’ll most certainly have to fix it later.

Postal patronage

I received a very interesting package last week. I had a deal with Idwer to get him rid of a few LPC and FWH flash chips. I was able to take them off his hands for just a few euros, a deal made sweeter by the fact that those chips are no longer being sold anywhere. I also sent Peter a VultureProg board. Since he already has a Stellaris or two, I have just recruited an eager tester.

Can you send me a board?

Short answer, no. Just grab the gerbers and take them to SeeedStudio. They will be able to get boards in your hands for much less than what I pay to ship them. The two SMD capacitors are not hard to assemble. If you still want me to send you an assembled PCB, I will charge $50 (international) or $30 (US). I don’t have the physical time to sit and assemble PCBs by hand, plus, it would be very un-geekish of you to not assemble your own.

GSoC [early debugging] AGESA woes

I took some days off the project for holiday in mid-July; after that there has been some amount of problem solving / headaches with my main development platform (samsung/lumpy), as the upstream coreboot tree still appears to be broken. There is incompatibility with recent binary blob and even after some local fixes I still lose some early debugging with usbdebug.

SAGE Electronic Engineering has kindly provided me an AMD Persimmon board for coreboot development. It took a while for me to get it up and running as it turned out even the very basic documentation of the board connectors and jumpers were behind registration and login on AMD website. Board arrived with coreboot + SeaBIOS combination pre-installed. Seems like flashrom utility works and I also have SPI header connector for recovery purposes. So I should be all equipped for my own build and I hope I get lucky with usbdebug on this hardware.

Now I must say I am not a big fan of the AMD vendorcode named AGESA. Things like CBMEM init and CAR setup are done in a way different fashion compared to the implementation without using vendorcode wrappers. It is a heavy reading, and to get early logging via either CBMEM or usbdebug, I will need to master and possibly modify that code too.

On SerialICE parts I have not made the progress the way I originally had planned, meanwhile some cleanups on low-level PCI configuration and minor usbdebug fixes have been merged. There is more to come on that sector. A fairly complex patchset draft on CBMEM, finalizing that should make it possible to enable CBMEM console for all boards, for ramstage at least.

VultureProg: Equipped for galactic travel

It’s here! And it’s ready for takeoff. The VultureProg PCBs have finally arrived, and it is time to turn VultureProg from a proof-of-concept toy to a serious galactic tool. My major concern was that I could have misrouted one or two connections. The LAD pins are particularly sensitive, as they need to be mapped to sequential GPIO pins, and start from GPIO0, otherwise we need to do bitshitfs in every LPC cycle, killing any hope of decent performance. I was also worried that the particularly tight tolerances could be problematic during manufacture. Everything works as expected. Enough words, let’s see the porn.

vultureprog_shuttle — Brings back memories, doesn’t it?

vultureprog_shuttle_angle — The world’s first fully assembled VultureProg board

From bitstream to reality

I found it interesting to look at the transformation from a spaghetti on the screen to something real, something tangible.

vultureprog_shuttle_sbs — From conception to birth: the complete transformation

No thorough and thoughtful post today. I have a new toy.

QiProg: Expanding the flight range

Today is disappointing. I was expecting to have gotten the first batch of VultureProg PCBs. Having arrived in the US on Tuesday, I did not expect the package to be hovering in New York for most of this week. The need for eating the spaghetti and eliminating the hanging wires is growing more and more urgent with each speed increase. The long wires and insane inductance is already getting in the way of the signals, the lack of physical portability of the test setup is annoying, and the pain of accidentally knocking out a few wires is unbearable. Where are my PCBs ?!!!

High speed tests

The Stellaris is getting faster, much faster than initially predicted. From the pathetic 23 KiB/s read speed during the first tests, it now comfortably does over 450 KiB/s. The emulated LPC bus goes so fast, that I need to lower the CPU’s core clock down to a measly 16MHz to be able to just capture all the details of the waveforms.

Too much impedance

24MHz logic analyzer + Nyquist theorem = 12+MHz LPC clock

We’re pushing a 12MHz signal through 20+ centimeter hanging wires, routing them through a solder-less breadboard, then a socket, with the added load of probe wires. I was getting random errors or bad data, yet as soon as I disconnected the probe wires, everything magically worked. It must have had something to do with signal rise and fall times. On the Stellaris, this was relatively easy to cheat and fix by increasing the drive strength.

Where’s my bandwidth?

450 KiB/s * 1024 * 17 clocks/byte = 8 MHz LPC clock

Something does not add up. We know we’re running at over 12MHz because we can not sample the signal well, yet the throughput is much smaller. The USB on the Stellaris is easily capable of 1 MiB/s. To quote Seconds from Disaster, “when investigators looked at […], what they found shocked them”:

The LPC bus is idle 30% of the time. We’re driving the bus so fast, that loop overhead is not only noticeable, but significant. Killing this overhead has the potential to bring the speed to 600 KiB/s.

Insect season

It’s a hot summer in Houston, with air conditioning running around the clock. Stepping in the hot, wet weather outside results in an instant cascade of sweat. Insects are crawling from every nick and crevice, fire ants are spawning from the underground in huge mounds, and mosquitoes are raising their own deadly army of high-pitched buzzers. The QiProg and VultureProg trees are no different.

I’ve made the executive decision to fix bugs as soon as they are getting in the way. I intentionally avoid the use of the term found. I can find my own damn bugs, however, it’s fixing them that is the problem. I prefer to have all the pieces in place before I polish and shine them.

A ripe testing ground

Out of curiosity, I wanted to see if VultureProg will work on Install’n’Pray operating systems (InP). InP are known for their excellent ability to expose even the smallest, most innocent problems, and expose problems where there are none. If VultureProg can work in an InP environment, it most certainly will be stable in unix-like environments. Although it did not work at first, testing on InP has lead to a number of fixes.

Where next?

We have bulk reading completed, which was initially scheduled for week 6. Weeks 7 and 8 look scary, with a lot of goodies, including program and erase functionality. I’ve decided to peek early into how to program and erase LPC chips. I found flashrom’s jedec.c to be of great help: QiProg knows how to byte-program and erase JEDEC-compliant chips. A lot is still scattered in topic branches here and there, waiting for one’s mercy to merge. Somebody please send me some coffee.

GSoC (coreboot): Week 3 and 4

In the past two weeks I was on vacation and I have been working on what I call “test interface board”. Before I go on to elaborate this I feel there’s a need to discuss the big picture of this project because a lot of things have changed for good reasons and the old terminologies don’t make sense.

Just to remind, my project is centered on building inexpensive and flexible test-rig for the Automated Distributed Firmware Test System described in Quality Assurance Talk by Stefan Reinauer.

A centralized Test Management Server generates test sequences for remotely located systems under test (SUTs) and this includes controlling and monitoring the SUTs and flashing different firmware builds on them. The test management server coordinates with a repository for accessing test builds and for storing test reports. Test reports are the final and useful output of the whole system and these may be accessed using browser by clients from internet.

A Test Supervision Server is a low power computer that acts as a local housekeeper of SUTs for a given physical location. It connects to the Test Management Server using SSH over internet and executes given test sequences by coordinating closely with the SUTs using a Test Interface Board. Programmable power-strips are provided to control power supply to the SUTs from Test Supervision Server.

My work will be confined to the distributed components for now. I have completed the programmable power-strip block. A future add-on to this block could be integrating active power & energy measurement of an SUT for energy efficiency benchmarking. If this is really desirable it could be done after I finish doing the other parts. Right now I’m working on the Test Interface Board.

The Test Interface Board provides necessary hardware interface for connecting Test Supervision Server to an SUT. This is necessary to flash firmware to the ROM, to control power/reset sw, to measure PSU voltages and surface temp. of ICs and to take POST feedback if available. Let’s dive into more details to see how this can be done.

FT232H has a multipurpose serial engine that can be configured as SPI master. FT232H has additional pins that may be used as GPIOs so a GPIO expander may not be needed. The FT232H datasheet states that it offers up to 30mpbs throughput in synchronous serial mode which makes it a fast flashing solution for the given price point (3$). Slave Select (SS) pins can be used to switch between other devices like an ADC that gives voltage and temperature measurements and an optional Feedback microcontroller configured as an SPI slave that gives more information about the SUT. A few GPIO pins can be used to configure Logic Level Translator to ensure compatibility with serial flash of different voltages ranging from 1.8 V to 5V and a GPIO pin will also be used to configure a FET toggle switch (MUX) to electrically detach the serial flash for programming and connecting it back to the motherboard when it’s done.

Notice that I’ve got rid of microcontroller this time. This is because a new microcontroller chip doesn’t necessarily have a bootloader and it needs to be programmed using a dedicated programmer. This adds considerable cost and inconvenience for someone who needs to build only few of these boards. So unless you’re using the optional Feedback module nothing needs to be programmed. Just ordering the board and components and soldering up everything using a 15W iron should be enough to make one of these.

I’m also going to ensure modularity by having small PCBs for each functionality connected to a main-board using headers so that they can be developed independently and used as required. Also, there’s flexibility of choosing temperature probes because it is possible that someone already has good quality probes (that come with professional DMMs).

And a few comments about the ADC I’ve chosen – The ideal choice of ADC for voltage and temperature measurements where the sampling period is large is an integrating ADC. An integrating ADC charges a capacitor from the input signal for a known period of time using an opamp integrator then it discharges that capacitor using a known negative reference voltage. The time it takes to discharge the capacitor is proportional to average value (area under curve) of input signal over sampling period. It’s theoretically simple but it needs use of precision external components and a microcontroller program to work. This is the technique used in professional DMMs (True-RMS) and bench power supplies (for feedback). Delta-Sigma ADCs are common and cheap these days but they don’t average the values over time like integrating ADCs. However, they can provide acceptable accuracy for our application and MCP3208 is a good candidate.

Please see the figure for more details and let me know if there are concerns or suggestions. I’ll post more stuff and schematics in a couple of days.

GSoC 2013 [flashrom] week #4

There is still no release, but we made quite some progress over the weekend. BSDs should work again as well as Windows (32b) – but the fix for 64b Windows will (most probably) not make it into 0.9.7 because we need to rework a bit of infrastructure code to get this right and we are not too sure about possible breakage from the quick hack I posted. We are still trying to verify the effectiveness of the IMC shutdown patch and looking for Dediprog testers with either an EM100 or a very old flash chip without fast reads. I expect the release to happen in the next two days. Now would be a good time to test flashrom’s trunk or my flashrom_0.9.7 branch on github on flash programmers near you. 😉

I would also welcome contributions to our brain storming about possible features of layout files. I decided to post them to the list instead of making a blog post out of it as envisioned in my last post.

DIY SOIC8 ZIF to DIP8 adapters

As mentioned previously I got an ASRock A180-H sample and want to develop Kabini support for flashrom with it. The board features an 8-pin DIP socket which is way easier to work with than a soldered flash chip, but the included DIP chip is the only one I possess. I do own quite a few SOIC chips though and 3 SOIC8 ZIF sockets though. And one of the two hacker spaces in town has these SOIC8 to DIP8 adapter PCBs in its inventory. So… of course I have built a SOIC8 ZIF to DIP8 adapter out of them.

Complete adapters are usually very expensive (>>50$) for no good reason (from a customer’s perspective only), so building such an adapter is the logical conclusion. Instead of the PCBs from futurlec mentioned above there is a very good deal available from adafruit. The cheapest SOIC ZIF sockets I could find are made by Wieson. They have 3 models available – for “200 mil” 8-pin (G6179-10), “209 mil” 8-pin (G6179-200000) and 16 pin (G6179-070000) chips. They are available from siliconkit (US, 2.5$ per 8-pin adapter, 10 pcs minimum), dediprog (Taiwan, 30$, 15 pcs package), and bios-repair.co.uk (UK, 5£) and probably elsewhere. I am using the G6179-10 below – IMHO the difference of the two 8-pin variants is the length of the chip to be inserted, not its width.

Before acquiring a pre-etched PCB I tried to solder a breakout board myself, similar to this tutorial:

Attaching the wires works somewhat (but one needs steady hands and a lot of patience) but there is one major issue: there is just not enough space to fit the SOIC footprint inside the outline of the DIP pins (the DIP rows of the socket on the mainboard are 0.3″/~8mm apart, the SOIC chip including the pins is about the same size), and of course the pitch is so different that one can’t just solder it on top:

One could try solving this by rotating one of the footprints about 90° and routing the connections manually with enameled wires, but I very much prefer PCBs like this one:

Here is the end result with the futurlec PCB which took me a fraction of the time I wrestled with the manual breakout approach: standing_cropped_0

All pictures above are taken by myself and put into public domain.

GSoC 2013 [flashrom] week #3

After enjoying an extended weekend at the customs office the Micron chips finally arrived at my place and I began testing them with flashrom while preparing parts of this post. One has quite a lot of idle time when testing bigger flash chips with a lousy self-made programmer… the execution of flashrom -w with random data took 52m58s on a 16MB chip (N25Q128) to complete (which is actually two complete reads and one complete erase + read + write cycle per block). Oh if only there would be a cheap but awesome open USB programmer… 😉

Carl-Daniel was still missing till I called him at work yesterday, so Continue reading GSoC 2013 [flashrom] week #3