Xen has long history and many features. Sometimes even experienced developers cannot be sure whether their new code is regression-free. To make sure new code doesn’t cause regression, Ian Jackson developed a test framework called OSSTest. In order to make this framework usable for individual ad-hoc testing, standalone mode was introduced. Recently I played with it and found it useful to share my experience with the community.
Basic Requirements
To run OSSTest in standalone mode, you need to have at least two machines: one is controller, which runs OSSTest itself and various other services, the other is test host, which is used to run test jobs.
For the controller you need to have several services setup:
- DNS / DHCP, use to translate IP < -> hostname
- Webserver, provide test box accessible web space, OSSTest will expose configurations / overlays via HTTP and detect test host status via webserver’s access log.
- PXEboot / Tftp, used to provide Debian installers to test host.
- NTP, optional
For the test host you need to:
- enable PXE boot
- connect to PDU if necessary
Step by step setup
git clone OSSTest tree, master branch, we assume you run everything inside osstest.git directory.
Have a look at README / TODO which contains useful information. You can create a global config file in ~/.xen-osstest/config. In standalone mode you can also create standalone.config in osstest.git directory to override global configurations.
In the configuration you can specify DNS name, name server etc.
DnsDomain uk.xensource.com
NetNameservers 10.80.248.2 10.80.16.28 10.80.16.67
HostProp_DhcpWatchMethod leases dhcp3 dhcp.uk.xensource.com:5556
TftpPath /usr/groups/netboot/
DebianNonfreeFirmware firmware-bnx2
DebianSuite squeeze
DebianMirrorHost debian.uk.xensource.com
DebianPreseed= < <‘END’
d-i clock-setup/ntp-server string ntp.uk.xensource.com
END
Debian is the de-facto OS in OSSTest, you can find more info in Osstest/Debian.pm. Here we use Squeeze to build binaries and run test jobs, because there’s a bug in Wheezy’s Xen-tools which breaks xen-image-create, which eventually breaks ts-guest-start. Patches to fix that problem have been posted but not yet merged. Hopefully some day in near future we can use Wheezy to run build jobs and test jobs.
Configure test host in OSSTest config file. There can be multiple test hosts in the config file, but I only have one. Here is what I have:
TestHost kodo4
HostProp_kodo4_DIFrontend newt
HostProp_kodo4_PowerMethod xenuse
HostProp_kodo4_Build_Make_Flags -j16
HostFlags_kodo4 need-firmware-deb-firmware-bnx2
There is detailed explanation of what those paramters mean in README. An interesting one is PowerMethod, which in fact points to a module in OSSTest that controls power cycle of the test box. Have a look at Osstest/PDU for all supported modules. Use manual.pm if your test host is not capable of doing power cycle automatically.
Before actually running OSSTest, you need to make some directories yourself.
- logs: used to store tarballs from build-* jobs
- public_html: expose this directory via HTTP server to the test host
- $TFTPROOT/osstest: OSSTest will write PXE boot information for test hosts here
Make sure test host is able to access contents in public_html, then you can have “WebspaceUrl http://YOUR.CONTROLLER/public_html” in your OSSTest config. Test host will try to fetch all sort of things from there.
Next step will be setting “WebspaceLog /PATCH/TO/WEBSERVER/ACCESS.LOG”. OSSTest watches webserver access log. When test host / guest go to fetch things via HTTP OSSTest gets to know their status. I use Apache2 so I’ve set WebspaceLog to /var/log/apache2/access.log which just works.
Have Debian PXE installers ready. Remember the “DebianSuite” option in your config file? In order to make OSSTest fully functional you will also need to place Debian PXE installers in the right place. You can grab Debian’s PXE installers from any of the Debian archives around the world. And put them under the TFTP you just set up. I would recommend having at least amd64 and i386 in place. Make sure installers are accessible from the test host before proceeding.
By now we’re all set! Next step:
./standalone-reset
This will reset everything in standalone mode and create standalone.db, which includes test jobs and runtime variables. You can peek what’s inside that database with sqlite3.
The first job to run should be a build-* job. That can: 1) verify your setup is correct; 2) generate a bunch of runtime variables for subsequent test-* jobs.
./sg-run-job build-amd64 # WARNING: this will wipe out your test box
If the job pass, you’re all set. You can play around with other jobs. The default setting of jobs is to wipe out test box on every run. If you don’t want that you need to specify OSSTEST_HOST_REUSE=1 as stated in README.
An example of what I run:
./sg-run-job build-amd64-pvops
OSSTEST_HOST_REUSE=1 ./sg-run-job test-amd64-amd64-xl
If you only want to run a specific testcase, you can try OSSTEST_JOB=$JOBNAME ./ts-XXX host=$TESTHOSTNAME.
Customized tree / revisions
By default OSSTest always fetches trees and revisions from Xenbits. You can easily override them with standalone.config.
Say if I want to test a specific revision of Xen, I have:
export REVISION_XEN=c5e9596cd095e3b96a090002d9e6629a980904eb
in my standalone.confg.
You can look at make-flight to know all the interesting environment variables. (Sorry, no document yet)
Writing new testcases
If you’re interested in writing new testcase, you can do that in two simple steps:
- write ts-my-test-case script, you can use any existing testcase as template (they are prefixed with “ts-“)
- modify sg-run-job, which has the information for which testcases to run for a specific job
Do have a look as osstest/TestSupport.pm, in which you can find lots of helpers to accomplish your task.
Writing new test job
The script responsible for making jobs is cs-job-create. When you run OSSTest in standalone mode, it is probably more useful to modify make-flight. You also need to modify sg-run-job to link your new job with testcases.
Hope the above information can help you get started with OSSTest. If you have any problem, do come to Xen-devel and ask.
Some readers may recall the recent announcement of open-sourcing XenRT, and wondering about the relationship between OSSTest and XenRT. Long-term, we expect that XenRT will mature as an open development project and eventually displace OSSTest. But that is not likely to happen for another six months to a year. Developing OSSTest has benefits for the current development, and we hope that getting people involved in writing test cases for OSSTest will be of service in helping people write test cases for XenRT when it becomes available. So we have decided to continue to develop OSSTest until XenRT is ready to replace it.
Have fun! 🙂