I’ve started to work on the Xen on ARM project by trying to get Xen running on the nice little Samsung Chromebook which run an ARM processor with the virtualization extensions. The Chromebook uses Exynos 5250 dual core platform with 2GB of RAM, which could be perfect for a good demo of Xen on ARM.
But porting Xen to new ARM platform without any useful debug output of some kind was really hard at the beginning. There is no accessible serial port available on the Chromebook and Xen cannot display anything on the screen. But there is a trick, write your log to specific area in RAM, and read it back after a reboot. It is really annoying, does take more time to have any feedback from Xen and you don’t know when you can reboot the machine to get those logs. This is how we did it, until we received an Arndale Development board.
So, we started the development on Chromebook by simply installing a Linux on it (on an SD card). There were few tutorial on internet, so it was not too hard. Then we recompiled a kernel, as it will be needed to integrate dom0 support. And last, we tried to understand how to use the bootloader on the machine to load Xen where we need it to be and execute it, it took some time and many tries. When we finally get something from Xen, we could continue make it work on the Chromebook. We had bugs like loading Xen at the wrong place in RAM or missing information in the device tree. Then it took sometime to load Linux correctly and we had to use some hack to actually find the Linux blob in memory as the default bootloader was not very helpful for that.
We found several problems were a page table was written in the middle of the binary in memory because of too many hack on how to load Linux for dom0. Before starting to work with the Arndale, we’ve been able to run some of Linux code, but fail early in Linux when it tried to load the device tree.
The Arndale board finally arrived, we had to start over, different bootloader and this time there is a serial port!! The Arndale use the same hardware platform, namely, the Exynos 5.
So first thing, how to start Xen on it? It appears to be relatively easy to load your own kernel, Xen here. So let’s hack a serial driver for Xen, in assembly first :-). We found out that the bootloader delivered with the board does not let Xen switch to hypervisor mode, so we had to recompile a u-boot for the board. Not a big deal, Linaro has done some work on the board as well, so we could get u-boot from their source tree and finally have Xen get its hypervisor mode.
After the first bits resolved, we could get along and figured out all the hack/fix needed in Xen in order to run it on this second platform, the first one was a Versatile Express. There were a lot of assumption about the hardware. So to get things running, we just replace those by others because having a proper way to handle different platform take a lot of time.
We had an issue with the time on this platform. The ARM Generic Timer was not working, it was always giving the same time. After trying to use the Multi-Core Timer of the Exynos platform, we realise there was the same, but we had to set a bit in the MCT register in order for the timer to start counting.
After resolving few other issues, we could finally get a prompt! Next, we worked on getting more, like starting few guest.
There is yet another hack for both the Arndale and the Chromebook to get a working network interface or having something on the screen of the Chromebook, it’s a 1:1 mapping of the memory of dom0. This is when the virtual address (seen by dom0) match the physical address (seen by the hardware). Normaly, those two addresses are different for any guest.
To conclude, it has been a lot of fun working on those two. We could get a working demo for both the Arndale and the Chromebook of Xen capabilities. There is still a lot of work ahead of us, but we are getting there.
Much more happened in the latest weeks, during which I concentrated on something different than Xen on ARM or the Chromebook. For instance, there is now a better multi-platform support in Xen and SMP support is getting there as well.
Here is some wiki page on how to get started:
Bringing Xen on the Chromebook and the Arndale Board: the journey so far
