GSoC 2010, TianoCore as a payload :(

It’s been an interesting summer.  It didn’t at all turn out how I expected, but it is what it is.  TianoCore as a software project turned out to be massively more complex than I anticipated when I submitted my proposal, and the level of knowledge required was quite a bit deeper than I expected… it’s one of those cases where I didn’t know what I didn’t know.  I’ll have to talk to a couple of my professors about that, to see if there’s some elective class that explains the things I’ve missed.

Sorry, that’s vague, let me give an example.  Coreboot does it’s thing, hardware initialization, then passes control to the payload.  This seems to be the equivalent of the dreaded “goto”, which is actually pretty cool.  Coreboot doesn’t care what happens next.  So hypothetically, I have some code, anything, I want to use it as a payload.  I compile it, then what?  Well, it depends (as you all know), how was it compiled?  Is it an elf?  PE32?  Something else?  Where exactly is the entry point to this binary blob?  (That’s a rhetorical question, please don’t answer it in the comments.)  You would have thought at some point in one of my classes executable formats would have come up, just as an example.  Or calling conventions.  Or hundreds of other little things that I’d never seen or heard of before I suddenly realized that I needed to understand them.  So that’s what I ended up spending much of the summer on.  Write code… stop and realize what I’m doing doesn’t make sense/won’t work/is the wrong approach, then start over.

One of the things that drew me to coreboot as a project was that as a computer engineering student, I took a lot of classes focusing on the physical side of computing, starting with physics and circuits classes, moving up through logic gates to chip design.  On the other side, programming started at a pretty high level with c++, then worked down, till I got to the computer architecture and operating system classes, and assembly language (not x86, unfortunately).   I would expect that as a “computer engineer” I should understand the whole stack, that the physical, EE stuff and the CS stuff would meet in the middle.  But they haven’t (and they won’t: I’m about to graduate, and there aren’t any crucial classes left to take).  I knew this going into GSoC, and coreboot seemed like the perfect project to fill the gap (and give something back to the open source world that I’ve gotten so much out of).  Well, like I said, the gap turned out to be a lot bigger than I expected.  (To abuse the metaphor a little more, anyone remember when Evel Knievel tried to jump the Snake River Canyon?  That’s kind of how I feel about my summer of code.) Continue reading GSoC 2010, TianoCore as a payload 🙁

GSOC Coreboot Kconfig summary

Although the GSOC 2010 is closed,  my patch about coreboot kconfig which involving the payload kconfig is send to the mail list. There are only two payload are supported.”Filo and coreinfo”. I can add more, but i would like to wait a moment. Because the most important of my project is not how hard it is but how to make the users can use it easily. There are also some consideration should be discussed with the others.

The latest patch which i send now support:

1. use coreboot to load the payload’s kconfig. This feature can be used by “make config PAYLOAD=filo” or “make config PAYLOAD=coreinfo”. Coreboot users can use the coreboot as before too. just “make config” and “make” to manually put payload.elf under coreboot directory. The new method will do all of these things automatically. The method which i use is let the coreboot kconfig detect the macro PAYLOAD, if the variable matches filo/coreinfo, it would load the filo/coreinfo kconfig. It also disable the coreboot kconfig’s payload menu which could avoid conflict.

there is a consideration which should be discussed. There is an sinario,the libpayload may be modified between “make config” and “make”. I did not take this into consideration, because it can be avoid by “make clean” to rebuild the project. If anyone thought it’s improper, please let me know.

2. Most of the payload are using libpayload. Libpayload should be installed to certain path to make sure payload can be compiling correctly. The previous patch which i send are using to let payload can load libpayload automatically, it can configure the libpayload each time. Like enable USB features or not. It’s pretty easy to build the payload. the whole things can be worked as one.

I also posted an new patch about filo to use an new method to do these things. The previous method is trying to use filo’s kconfig program to load libpayload config files. Lots of messages are show the same with filo while filo configuring libpayload. The new patch use a remote method which invoke libpayload’s own kconfig program to buid itself. The patch also looks simpler that previous.

The time is passed much quick. But i learned much about the open source. I reviewed the goals of GSOC

Google Summer of Code has several goals:

  • Create and release open source code for the benefit of all
  • Inspire young developers to begin participating in open source development
  • Help open source projects identify and bring in new developers and committers
  • Provide students the opportunity to do work related to their academic pursuits during the summer (think “flip bits, not burgers”)
  • Give students more exposure to real-world software development scenarios (e.g., distributed development, software licensing questions, mailing-list etiquette)

I am so glad that i can join the open source family before graduation. Coreboot is a wonderful project, from here I can learn much about the X86 systems, firmware things. I also should spend lots of time to learn them. One of my experience is that real-world software should thinking every possibility not only the realization of  features. Marc always told me: thinking about all of the corner cases. That helps a lot. Thanks to google give me this opportunity to join in Coreboot. Thanks to my mentor Marc,  he is a very nice man who knows everything that i want to know He tells me how to join an open source project, how to do real software programming, how to considerate the software. Thanks to QingPei, who helps me a lot during the whole process. He also tells me that: “GSOC is an opportunity to teach new developers rather than a chance to get a pile of free code. It’s a big step for me to learn how to contribute to coreboot. So no matter my last evaluation is passed or not. I will try to do all of these things.

By the way, Any ideas about the coreboot kconfig things is welcome.

There are also some questions that should be completed:

1) What was the final outcome of your project?
well, i am not sure about this. I think i am good, i learned how does coreboot work, how wonderful it is. Although patches are still pending. But the patch is the best way i can realize the feature with my ability.

2) What problems did you encounter and how did you work through them?

The problems which i can not forget is it’s much more hard than i had ever think about low level things. At the beginning, it’s only seems wonderful for me, Coreboot did the all of the legacy system bios things. But after several weeks, i found that the coreboot developer are all genius, it’s too hard to understand the who things. Fortunately i at least know how does coreboot work, i know lots of things that i can not learn from class. i used to call QingPei for 3 hours to let him tell me what’s the coreboot things going on.

3) What did you learn about open source development?

Lots of them, there are two parts. First, the real project programming skills. At least i can read most of the Makefile and understand how does these things doing. I also know pretty much about Kconfig language. By the way, i also learned much about “Bash” “Sed” programming. I learned how to work with linux gcc.there are something more i can not write down one by one. Second, i finally can join an open source project, i knew how does the large open source project are working. It need “singed-off-by” and “ack-by” before checking in the code. the build service will show interesting messaging if there is any error between the building progress.
4) Do you plan to work on other open source projects? Which ones?

Because there are also lots of thing should be done with the coreboot kconfig things. I would like to still my work with coreboot, i hope i can contribute more for coreboot.
5) What did you learn about coreboot?

Pretty much. As i said in question 3.
6) Do you plan to develop or use coreboot in the future?

Sure, i will.
7) What could coreboot do better to help developers in the future?

More manuals for the new developers? Although i thinks the wiki is good enough. If there is more manuals, it will be better.
8) Would you recommend coreboot GSoC to other students?

I will, but i should check if he is smart enough.
9) Do you feel that you passed or failed your GSoC project?

I do not know, it does not matter, as least i learn  pretty much more thank i thought.

Thanks to Google

Thanks to Coreboot

Thanks to Marc Jones

Thanks to Wang Qing Pei

GSoC USB: Conclusion

Now that GSoC is coming to an end, I prepared the patches and pushed the code upstream.

r5691 contains the work done until today, which is what I’ll post as my final result to GSoC, too. Work won’t end on it however, so expect more patches in the future.

OHCI

OHCI works – except for interrupt transfers, which are mostly used (in boot environments at least) for keyboards.

xHCI

That one is more complicated than the other controllers combined, and while I made a couple of stupid mistakes that held me up for longer than I wanted, there are aspects in xHCI that make the bootstrap of the driver harder than I’d like it to be.

Once you got the command and event channels set up, it seems that xHCI provides a neat interface for getting all kinds of status information out of it. The only problem is that setting up these channels seems to be more complicated than the entire bring up of UHCI – at least, that’s where I’m stuck right now.

I’ll get back to it, but I hope that a couple of days of doing something else will help me to finally see the problem.

Conclusion

Doing both drivers was ambitious. While I didn’t have the burden of creating the stack design, and learning USB in the first place, like I had in 2007 (when I worked on UHCI for GSoC), it’s still two specifications to understand, two way of communication between the controller and the host system, and finally two drivers to write.

It was fun, but I’ll be more conservative in choosing my project, and estimating the required effort, next time. It’s much easier (and also more satisfying) to add some tasks when the job is done early than to remove them – especially when you get to strip milestones because the hardware is acting up.

GSoC USB: xHCI part 1

I started implementing the xHCI driver. The first obstacle was to overcome a weird lack of configuration of the card (it’s a PCIe device) by coreboot. First I suspected that something went wrong because it uses a 64bit memory BAR, but then it was just a disabled PCIe bus in the devicetree.cb.

Thanks to Stefan for working out that issue.

However, the last days weren’t wasted, as I read the xHCI spec again and again, to build a mental image of how things interact in that standard.

Now I’m chugging along with implementing the data structures in C that xHCI requires (many more than in the older USB HCI specs)

GSoC USB: mid-term report

I stopped doing weekly reports at some point, for a very simple reason: I had few to report, except maybe my frustration that I couldn’t find the bugs in my OHCI driver that prevented some (but not all) devices from working.

So I spent the last weeks tracking down these issues and reading specs and more specs. Both OHCI and xHCI – the latter because it’s part two of my project, and because I was close to giving up OHCI for now and working on xHCI first. It’s an independent task, so that could have been done easily.

Today, I managed to hunt down the bug. It was a simple fix once I found out what’s up. While this uncovered more problems, I can move forward again.

To prevent me from hanging 4 weeks on the next bug, I’ll start on xHCI nonetheless, while doing OHCI on the sideline. Most of OHCI is done, and I guess I can start pushing code upstream soon – just one feature (interrupt transfers, so keyboards work) and a couple of cleanups are missing.

TianoCore payload, mid-summer status update

Well, it’s hard to believe that the GSoC midterm evaluations are here already.  I guess it’s true what they say, time flies when you’re sitting in a basement in front of a computer all day. If I were to evaluate myself, I’d give myself a barely passing grade based on results –I’m nowhere near where I expected to be this summer.  I think I mentioned before, partly this is due to TianoCore being massively more complicated than I expected when I wrote my project proposal –seriously, I ran cscope on the edk2 branch of TianoCore, and it reported over 160,000 files… the resulting index itself took up half a gig– and partly it’s due to there being so much about sophisticated C usage (and makefiles, and preprocessor directives, and macros, and calling conventions…) that I didn’t understand going into this.  (But, having said that, part of the reason I applied to coreboot was I knew there were a lot of important details that had been glossed over in my classes, things that I needed to know and would be forced to learn if I worked on a close-to-the-hardware project.)

Moving on to the status of my project. In my proposal I assumed a couple weeks to improve the state of TianoCore as a payload, a month or so to write a CBFS driver for TianoCore, and a month or so to write the VGA driver.  It turns out that the state of TianoCore as a payload was not very good, and so that is what I am working on.

Let me try to briefly explain my current approach.  UEFI itself does not initialize the hardware.  Before the UEFI firmware can be run, the system (from a cold boot) has to go through the Platform Initialization stage.  The PI stage is itself made up of the Security stage (the initial booting, and some optional checksums to make sure the image hasn’t been tampered with), the Pre-EFI Initiialization (PEI , where the memory and chipsets are woken up and initialized) and the Driver Execution Environment (DXE, which loads additional drivers, then starts the UEFI).  Coreboot already does most of this work in its own way, so it seems the best strategy would be for a coreboot payload to impersonate one of these stages (each stage is it’s own binary in the firmware volume), provide all the functions and data stuctures that the following stage expects, then jumps to it.  Inserting the payload immediately after the Security stage seems redundant and dangerous (the PEI stage would end up trying to reinitialize hardware that’s already  in use), and after the DXE stage seems too late, because then the payload would have to know how to load DXE-stage binary drivers.  So I’m working on implementing a pseudo-PEI stage, that translates the coreboot provided data structures into the form that the DXE stage expects, and writing a couple dozen functions that the DXE stage expects to be available.  This way we can have a minimal-sized payload that can leverage a separate DXE and UEFI stage compiled directly from the TianoCore codebase (or borrowed from a manufacturer supplied image, like a traditional option ROM).  I think I can have this working by the end of summer. Thoughts?

coreboot porting to jetway PA78VM5 have been 95% successful.

i am  glad to say that the Jetway PA78VM5 mainboard can run coreboot sucessfully. The configuration of Jetway PA78VM5 can be found at PA78VM5.the coreboot+filo can work fine. The kernel began booting, but the only problem is after kernel shows”jumping to **” the serial port stoped showing anything. And i have already set the kernel parameter with “console=ttyS0,115200”

First of all thank olsen provide me this mainboard.

The mainboard have an SPI flash W25X80A, my SF100+testchip SO08 can detect the flash type, but can not erase the flash correctly. After contacted with dediprog engineer. i remove the flash from the board, it seems fine, the programming is fine, but i can not use it with the mainboard unless i can bear removing the flash chip every time i need to rebuild the coreboot. After that, i replace that flash chip with an sst 080b. it worked pretty fine. 🙂

another problem is while the coreboot booting, it stoped while corebooting trying to extract the cbfs files. i debugged this for a long time, finally thanks to patrick, i take his advice remove the $(CBFS_COMPRESS_FLAG).it worked.

the latest problem is that amd famlily10 may have much problem with the current build version.

i should find out what difference between btdc and coreboot public version caused problems.

my next step may focus on this things merge the code, and find out why the kernel did not show the booting message.

i am so glad that coreboot can finally booting the Jetway PA78VM5

progress of libpayload infrastructure and filo

it’s so happy that, filo finally can load libpayload kconfig. i deleted the first line of libpayload Config.in which is “mainmenu”, it still work even if libpayload build itself alone. After this, filo can easy load libpayload kconfig by adding “source ../libpayload/Config.in”. The variable is also blocked me a while for each variable is defined by quotation mark. the makefile should remove these quotation mark before using them, i learned that from coreboot makefile.

the xcompile things of libpayload has already resolved, i found that filo have the same problem which seem the scripts are almostly the same with libpayload, they are need to use the correct directory of coreboot, not themselves.

the problem now is the kconfig command “source” is used to read specified configuration file, this file is always parsed which means that it is parsed at the first beginning, i can not make it get the variable from the forward defined lines. it can only read from ../libpayload. i am thinking if we put filo under coreboot/payloads/, then the path is permanent. We did not need to change this path.