GSoC [early debugging] Art of refactor

Your branch is ahead of origin/master by 48 commits.

Yes, I knew this would happen, it has become increasingly difficult to push new work for review on gerrit, as I have dependencies on existing work waiting for merge. As the pile of un-merged patches increases so does the time I spend with git rebase, so I am hoping for some progress on that side.

My eyes in the local working directory have turned towards SerialICE integration inside coreboot tree. The benefits of this approach are better tree structure, wider hardware support, cache-as-ram and usbdebug.

There are several use-cases to consider:

Compile classic stand-alone SerialICE ROM image with ROMCC, using super-IO and chipset initialisation from coreboot tree.
Compile SerialICE as an alternative romstage with ROMCC, using existing coreboot bootblock added with serial port initialisation.
Compile SerialICE as romstage with GCC and cache-as-ram to use existing usbdebug code and possibly better execution performance.
Add abilility to jump out of SerialICE to regular romstage.

Also for the SerialICE session on debug host we have alternatives:

Execute vendor BIOS image under QEMU.
Execute coreboot image under QEMU.
Execute coreboot image in user-mode under GDB without QEMU.
Execute utils like nvramtool, msrtool, inteltool, superiotool, lspci and setpci remotely.

Now all of the above has been demonstrated before but not adopted. Adopting these widely for all mainboards may not happen during my GSoC, as there is no common function to call to enable a serial-port from romstage. At the minimum I will make some simple example one can follow to get SerialICE running on boards with existing coreboot support.

Cooking with thin spaghetti: The hard side of Vultureprog

One of the reasons I fell in love with the Stellaris Launchpad boards is that they are modularly expandable. This notion is difficult to explain without comparison to STM Discovery boards, which have a row or two of pins on each side. The idea is simple: you hook one end of your wire to the right pin, and the other end to your breadboard, or you design a custom baseboard specific to the Discovery model. Stellaris takes this idea a little further. The layout of the pins is standardized, not just for the Stellaris, but across the family of TI development boards. Enter the Booster Packs: standardized add-on modules for TI boards. These modules are stackable, so it is possible to connect more than one to a single Stellaris board. This is why I wanted to use the Stellaris for this project. It’s much easier to build a booster pack than to tell people how to connect 32 wires; most people have problems connecting four of them to a buspirate. Let’s look at some of the design choices.

Constraints, constraints, constraints

It’s easy to imagine connecting a LPC chip: six wires and power. In reality, the situation is nowhere near as bright. Four ID pins need to be pulled low, reset pins (yes, there is more than one) need to be pulled high, and some pins simply cannot be left floating. Thus, even a simple bus like LPC becomes a nightmare. Without a logic analyzer to tell what works and what does not, the result is frustration and even self-inflicted injuries. Consequently, I wanted to do a few things right from the beginning(TM).

The most important point was to have all pins properly connected with zero wires. Users should not have to worry about what connects to where. Remember, these chips have 32 pins.

I also wanted to support all possible bus types. LPC and FWH are identical hardware-wise, and are not a problem to support concurrently. SPI is also just a few extra traces that lead to a header. On the other hand, having a programmer that also supports parallel mode is a much harder problem. It turns out there are really two “parallel” modes. The first one is ISA, where the chip is accessed via a linear address space. You put the address you want to access on the address pins, handle a couple of handshake lines to tell the chip if you want to read or write, and move the data over a separate 8-bit data bus.

On the other hand, the second “parallel” mode is a real pain. It uses a 2-dimensional address space, where you need to drive a row address, then a column address, and only then access the data. It’s called PP or “parallel programming” mode. Luckily we get a break: PP mode is an auxiliary programming mode specific to some LPC chips. If we support LPC, we don’t need PP. PP goes in the garbage bin (for now).

Now we need an efficient way to connect the GPIOs to the chip. By “efficient” I mean minimizing the number of GPIO accesses, and the number of bitshifts we need to do in firmware. A poorly chosen pinout will result in abysmal performance, as the 80MHz core struggles to shift the correct bit to the correct GPIO. My choice here was limited, as the best I could do was assign successive GPIOs to successive address pins. I spent the entire Sunday looking over chip datasheets and deciding on this “spaghetti recipe”.

Flexibility – a big issue

I also wanted to have the option between a normal PLCC32 socket, or a ZIF socket (AKA clamshell). I was really an idiot for thinking I would have both on the same board. On paper, it looks very straightforward. In reality, adjacent pins are on different hemispheres of the globe, and routing them is well, the tastiest spaghetti you have ever eaten. There was no way I could fit both a clamshell, and a PLCC32 socket. There was no way to route the 32 or so tracks on just 2 layers. So I killed the clamshell, the SPI header, and the LPC header. After a couple of hours of messing with the routing, I always had one or two pins that got cornered.

Even routing a simple PLCC socket proved difficult.

What coffee can do to you

I decided to start over, with all the components in place. Once I reduced the track size to 8 mils, and spacing to 6 mils, I was able to route two tracks between a set of pins. This time, I placed the socket inside the clamshell, and managed to connect the two using just the top layer. I then worked from the booster pack connection to the DIP pins on the same side of the board, again, using only the top layer. Then I started using the bottom layer for DIP pins on the opposite side. After a few hours, Chuck Norris warped space and time to make room for all the tracks:

From here, it was a matter of optimizing the routing, taking care of ground planes and other finishing touches. In the end, we get VultureProg hardware version 0.1:

Don’t let the PRELIMINARY DESIGN warning fool you. There is an infinitesimal possibility I will ever want to go back and revise the design. We have 35 GPIOs. accessible on the Stellaris. Five of them are connected to the on-board LEDs and buttons. The remaining 30 are all used up.

Conclusion

If you are a Kicad user, you can head over to yet another one of my GitHub repositories. If you do not have a way to consume Kicad files, you can look in the doc and gerbers directories. Feel free to feed the gerbers to Mayhew Labs’ 3D Gerber Viewer (hint: you can rotate the board in 3D). With all that being done I ordered the first batch of PCBs from Seed Studio’s Fusion PCB service. Routing is definitely too crammed and painful, but I really wanted something versatile and flexible. Whether it lives up to its design goals in REV 0.1 or REV 0.2 remains to be seen. My money is on REV 0.1 — quite literally.

GSoC [coreboot debugging] Now it is broken, now it is not

I feel I did not make much progress the last week, I realised I wasted two days looking for error in my code and I finally found the error elsewhere. As for preparation to push my developments to review, I had rebased my tree. That is, I had picked up the developments done by other people in to my setup. My mistake. While the error still persists there in the master tree, in the process of recovering my platform I learned that there are two types of SOIC-8 SPI flash chips, ones that fit in the miniature socket I have and ones that are physically too large. The spare chips I had were of the second type and that slowed down my system recovery procedure radically. New flash chips are waiting for pickup in the store now.

This is actually just the situation I want coreboot to deal with better in the future: doing a firmware upgrade without the risk of bricking the device to the point where you need to use an external programmer device to recover. Problem is specifically with laptops, which may take a good hour or so to disassemble and put back together, and with every disassembly the risk of breaking some of those miniature connectors increases.

My plan of having two copies of firmware in the same flash chip image just got a bit more complicated. I learned that with recent platforms using a so-called binary blob for raminit, aka. system-agent binary, it is not possible to do a type of dual-boot-prefix setup I had planned, since one cannot put two system-agent binaries in the same CBFS image. I hope the system-agent build and release process is seriously improved to overcome this issue as badly gone(/done) binary blob upgrade procedure was the root-cause of my troubles the past week.

I have not really had a chance to test pre-OS flashing with FILO (actually the code might not yet be available for me to download). Instead I have attacked the low-level PCI and IO sources to reduce a good two or three copies from coreboot tree, this will help my efforts in the long run with SerialICE integration work.

GSoC (coreboot) Progress till week 2

As you might know my GSoC project is about making a test rig that can make coreboot test systems more accessible to a coreboot test server. This test rig enables coreboot test server to interact with the systems under test (which may be remotely located) in the following ways:

Power supply control (discussed in this post)
power/reset switch control, voltage and temperature readouts, firmware flashing on serial flash (to be done next)
provision for POST feedback (later)

With this project I’m hoping to create an environment where developers will be able to conveniently connect their systems to the coreboot test server for testing at their own place. This is why I like to call it a distributed test environment as it facilitates mass testing without the need to maintain a dedicated testing facility.

So this week I will present a nice and easy solution for power control of the coreboot test systems. I would call this device a ‘programmable power strip’. Before going to the final solution let me first walk you through all the routes that I’ve taken in order to answer some potential questions that may arise. Read on…

GSoC 2013 [flashrom] week #2

This week I have a very important directive to share with you:

Working (and especially debugging) in a methodological way does not only mean that every step should be taken on scientific grounds, but that the order of steps should be in an effective order too.

Why do I mention that? Last week I told you that the nice guys at Sage and AMD have sent me an ASRock Kabini board. I received it the following day (which amazed me quite a bit, because it came from US hence it had to undergo a customs check too…), hooked everything up: an old 300W ATX power supply (way overpowered for a 10W SoC) which I have used for coreboot development in the past, USB keyboard and mouse, network, a USB key with Ubuntu and a power button. I switched the PSU on, pressed the button and the fan began to rotate… for a few hundred milliseconds. WTF? I stripped away the non-essential connections and tried again – no change. I thought it could be the PSU – maybe the load is too small or something. But since I had no other supply easily accessible I decided to look at other possible causes: I checked all jumper settings (and there are quite a few of them) and noted a difference between the docs and the actual board regarding the jumpering of an always-on feature (which was a dead end but seemed very promising first), I cleared CMOS memory, reseated the DIMM etc. I even hooked up the flash chip to my logic analyzer to see if it tries to read commands from it… but there was no single proof of life.

So what do you think, is the board dead?

Did you spot the error I made? I hope you did with the blunt hint in the beginning. 🙂 After pulling another PSU out of an old PC and hooking it up everything was fine. *sigh*

When getting the board up eventually I just did a few quick tests (including the flashrom hack that Wei Hu contributed after some discussion in my previous blog post (oh who would have thought that these blog posts are useful at all!? :P)) and put it aside again for hacking in the weeks to come.

The remaining time was spent again on bringing flashrom up to shape for release, waiting for Carl-Daniel and negotiating with a Micron representative over support for their (i.e Numonyx’ and ST’s) chips in flashrom. It has not been the first time for me to mail back and forth with flash vendors, but it is always quite tedious to explain non-technicians and/or people with no idea about open source what we have to offer and what we need; often language barriers play a role too. For example I tried to explain to a Macronix guy about 3 or 4 times why I can not truthfully fill out the sampling order form completely (i.e. the company field) before I gave up. I can’t remember if I filled out the form in the end or not, but I received the samples eventually. Together with the Micron samples that should arrive this week and other samples I received previously I will soon have more than 1GB of SPI flash space at my desk, yay. 🙂

GSoC 2013 [flashrom] week #1: while(1);

This week I was busy preparing flashrom for the 0.9.7 release and queue up some overdue patches to be merged shortly after. This includes the infamous layout patches which I need to polish a bit since quite some time has passed since they were created and the surrounding conditions have changed a bit. Not only did flashrom evolve quite a bit (the original version of the layout patches were part of my GSoC 2011 contributions(!)), but I have learned a few tricks in the meantime too, I hope. Progress is rather slow because I am waiting for Carl-Daniel’s input to various issues but there is no response. That’s also the reason why I chose the subject for this blog post ;).

When I stumbled over a discussion in #coreboot about the new AMD SoCs (Kabini et al, preliminary BKDG), I discovered that they apparently contain a new flash interface supporting all kind of neat stuff (e.g. Multi I/O). This would match parts of my GSoC project perfectly and so I made a joke by asking who will send me a board. To my pleasant and big surprise I received a private message a few minutes later and a brand new ASRock IMB-A180-H is currently on the way to me. I want to express my gratitude to Martin Roth from Sage who was so kind to arrange this and Sage and AMD for paying for it. 🙂

Lately I’ve been looking at libpayload a bit since it will probably play some role in Kyösti’s project in conjunction with libflashrom. So it is also important that I grasp it before I am working on libflashrom. I got (lib)flashrom to compile locally with a slightly patched version of libpayload (patches pushed upstream of course). I hope to get the changes needed in flashrom out before release (NB: I am talking about the current state of libflashrom not about Nico’s patchset). After that I’ll continue to queue up/refine overdue patches – the main focus will be on Nico’s libflashrom.

Kick-starting with some maintenance

EHCI, USB, LOL, OTG, CBMEM, OMG, CAR. Those have been the topics of my first week of GSoC on the coreboot tree. Dozen or so patches in, same amount waiting on approvals or further actions from me. I was glad to find my mentors with many ideas for refactoring and working actively on reviews. Nice start I would say!

It turned out usbdebug support in coreboot may not be very widely tested, hardware has typically had serial ports available for the same task. With some required bugfixes on cache-as-ram and CBMEM, I now have identical output on usbdebug when compared for CBMEM console and serial console. For my setup, that is. More needs to be done to get AMD boards supported once again. I also get to fix usbdebug receive side to make it a usable pipe for GDB and SerialICE, and I want it to handle USB errors and disconnects gracefully.

On the debugging hardware side things have brightened up quite a bit. While the original Net20DC product is discontinued, I was concerned the only solution is the DIY version pictured on the right. I have then received positive feedback and testing from the community (thanks Denis and Aaron) of using some inexpensive ARM boards as USB debug gadgets. To make them work flawlessy, some modification needs to be done on the USB gadget framework drivers on the kernel side. I should try to find someone already familiar with the gadgets to take this development task as I believe it is of interest for kernel developers too.

Some principal decisions on payloads have been made. I would first add usbdebug support for FILO. I am eagerly waiting for the FILO payload with flashrom to be released, this would gain us methods to program the system flashchip from USB storage, in a pre-os environment.

Hello :)

I’m Ayush Sagar from India and I will be working with coreboot this summer on the project “Test set-up for the coreboot distributed firmware test environment featuring greater extensibility, enhanced automation, concurrent high speed firmware flashing and decentralized operation“ under Google Summer of Code 2013.

I’ve almost completed my graduation in Electrical & Electronics Engineering and by training I’m skilled at developing SCADA applications and ladder logic programs which are used for power system and factory automation. However my interests are widely scattered around physics, electrical engineering and computer science. I have been repairing consumer electronics and computer hardware on component level since a very long time as an earning hobby. It’s quite profitable here even today as most people are reluctant about throwing away their belongings. I’m also passionate about programming but I’m new to free and open source software development.

Continue reading Hello 🙂

Launch control and a brief test-flight

Last time, I promised I would prepare you for a short test-flight. We won’t go into a full orbit, but we will turn on the thrusters, launch, and land safely. This is a real mission. We will actually launch the Stellaris into firmware low-orbit.

The Toolchain

We need a tool to transform the flight plans into something the Stellaris can understand. I will call this The Toolchain. Having a good toolchain (compiler and supporting libraries) will save us a lot of trouble. Well, it will save you a lot of trouble; I already have a decent toolchain. At the very least, the toolchain should be aware of the following:

Hard-float “-mfloat-abi=hard -mfpu=fpv4-sp-d16” compiled libraries
A minimal C library
libnosys

I will be using the summon-arm toolchain, but any arm-none-eabi will do. I won’t go over the details of setting up this or that toolchain. The options are plentiful.

The Debugger

We need a tool to monitor the Stellaris in-flight, and solve any potential issues. Let’s call this tool The Debugger. I will be using OpenOCD since it works for me (TM). You will need version 0.7.0 or newer, with ti-icdi support enabled. I will say no more.

HINT: if you need to build it from source, pass “–enable-maintainer-mode –enable-ti-icdi” to the configure script.

The Flight Plans

This is the fun part. I have been working since last year on Flight Plans for the Stellaris Launchpad. Courtesy of flight engineer Peter Stuge, the flight plans are available on the QiProg website (more on that later).

OK, let’s get them:

$ git clone git://git.stuge.se/vultureprog.git
$ cd vultureprog

And the flight plans for the subsystems:

$ git submodule init
$ git submodule update

Now it’s time to choose our mission:

$ git checkout 841ba3460767eefc2c744ec03c37e7e383cd8485

Now that we have everything we need:

$ make

This should get the Flight Plan primed and ready.

The Launch

Assuming everything went fine so far, we are go for launch. Prepare your Stellaris Launchpad. When you are ready for launch:

$ make flash -C boards/lm4f/stellaris-ek-lm4f120xl/

WARNING! Do not look directly in the thrusters (LEDs) of the Stellaris. Thrusters are operated at full power and may cause temporary loss of vision.

If you see the thrusters (LEDs) alternating, then launch was a success. Congratulations! During the test flight, you can try pressing the buttons on the Stellaris (HINT: they do something).

The Announcements

Peter Stuge has been kind enough to set up a website for QiProg on his own time. It’s http://qiprog.org . Stable code will be pushed to the git repositories at http://git.qiprog.org. Development will be kept on github for the time being, on my vultureprog repository. I will only be pushing stable updates to qiprog.org.

That’s all for now. Mission accomplished.

New coreboot debugging solutions

Hello. I am Kyösti Mälkki and I will be working on improvements for coreboot debugging tools during my GSoC 2013. My GSoC project plan has some details on things to come, but deliverables and schedule there will be fine-tuned next week or so. All in all, my work here should make it a little less painful to port new mainboards to coreboot and make SerialICE use more flexible.

Some feedback I have already received and the discussion we had on #coreboot suggests that some ARM boards are good candidates to be used for debugging x86 targets. Keep tuned in, the success of my work depends heavily on getting test results from a variety of mainboards.