Linux 3.14 and PVH

The Linux v3.14 will sport a new mode in which the Linux kernel can run thanks to Mukesh Rathor (Oracle).
Called ‘ParaVirtualized Hardware,’ it allows the guest to utilize many hardware features – while at the same time having no emulated devices. It is the next step in PV evolution, and it is pretty fantastic.
Here is a great blog that explains the background and history in detail at:
The Paravirtualization Spectrum, Part 2: From poles to a spectrum.
The short description is that Xen guests can run as HVM or PV. PV is a mode where the kernel lets the hypervisor program page-tables, segments, etc. With EPT/NPT capabilities in current processors, the overhead of doing this in an HVM (Hardware Virtual Machine) container is much lower than the hypervisor doing it for us. In short, we let a PV guest run without doing page-table, segment, syscall, etc updates through the hypervisor – instead it is all done within the guest container.
It is a hybrid PV – hence the ‘PVH’ name – a PV guest within an HVM container.

The benefits of this – less code to maintainer, faster performance for syscall (no context switches into the hypervisor), less traps on various operations etc. – in short better and faster response.
The code going in will allow users to use it with Xen 4.4 (which as of Jan 2014 is in RC3). From a standpoint of what the guest does compared to the normal PV – it is almost
no different except that it reports itself as an HVM guest (without any PCI devices) and is much faster. Naturally, since we are still working through the kinks of the PVH ABI, there is nothing set in stone.
That means the next version of Xen might not even run with this version of Linux PVH (or vice-versa). It is very experimental and unstable. Naturally we want to make this production quality, and we are working furiously toward that goal.
It helps us immensely if users also try it out so we can track bugs and issues we have not even thought of.
HOW TO USE IT
The only things needed to make this work as PVH are:

  • Get the latest version of Xen and compile/install it. See http://wiki.xen.org/wiki/Compiling_Xen_From_Source for details or http://wiki.xen.org/wiki/Xen_4.4_RC3_test_instructions
  • Get the latest version of Linux , see < a href="http://wiki.xenproject.org/wiki/Mainline_Linux_Kernel_Configs#Configuring_the_Kernel">http://wiki.xenproject.org/wiki/Mainline_Linux_Kernel_Configs#Configuring_the_Kernel for details. The steps are:

    cd $HOME
    git clone git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git
    cd linux
  • Compile with CONFIG_XEN_PVH=y
    • Based on your current distro.
      cp /boot/config-`uname -r `$HOME/linux/.config
      make menuconfig
      Where one will select Processor type and features —> Linux guest support —>Support for running as a PVH guest (NEW)
    • Or from scratch:
      make localmodconfig

      You should see:
      Support for running as a PVH guest (XEN_PVH) [N/y] (NEW)“In case you missed it:

      make menuconfig
      Processor type and features ---; Linux guest support ---;
      Xen guest support (which will now show you:)
      Support for running as a PVH guest (NEW)
    • If you prefer to edit .config, these should be enabled:

      CONFIG_HYPERVISOR_GUEST=y
      CONFIG_PARAVIRT=y
      CONFIG_PARAVIRT_GUEST=y
      CONFIG_PARAVIRT_SPINLOCKS=y
      CONFIG_XEN=y
      CONFIG_XEN_PVH=y

      You will also have to enable the block, network drivers, console, etc which are in different submenus.
    • Install it. Usually doing:

      make modules_install && install

      is suffice. It should generate the initramfs.cpio.gz and the kernel and stash them in /boot directory.
  • Launch it with ‘pvh=1‘ in your guest config (for example):
    extra="console=hvc0 debug kgdboc=hvc0 nokgdbroundup initcall_debug debug"
    kernel="/boot/vmlinuz-3.13+"
    ramdisk="/boot/initramfs-3.13+.cpio.gz"
    memory=1024
    vcpus=4
    name="pvh"
    vif = [ 'mac=00:0F:4B:00:00:68' ] vfb = [ 'vnc=1, vnclisten=0.0.0.0,vncunused=1'] disk=['phy:/dev/sdb1,xvda,w'] pvh=1
    on_reboot="preserve"
    on_crash="preserve"
    on_poweroff="preserve

    using ‘xl’. Xend ‘xm’ does not have PVH support.

It will bootup as a normal PV guest, but ‘xen-detect’ will report it as an HVM guest.
Items that have not been tested extensively or at all:

  • Migration (xl save && xl restore for example).
  • 32-bit guests (won’t even present you with a CONFIG_XEN_PVH option)
  • PCI passthrough
  • Running it in dom0 mode (as the patches for that are not yet in Xen upstream). If you want to try that, you can merge/pull Mukesh’s branch:

    cd $HOME/xen
    git pull git://oss.oracle.com/git/mrathor/xen.git dom0pvh-v7

    And use on the Xen command line the dom0pvh=1 bootup parameter. Remember to recompile and install the new version of Xen. The patches in Linux do not contain the necessary code to setup guests.
  • Memory ballooning
  • Multiple VBDs, NICs, etc.

Things that are broken:

  • CPUID filtering. There are no filtering done at all which means that certain cpuid flags are exposed to the guest. The x2apic will cause a crash if the NMI handler is invoked. The APERF will cause inferior scheduling decisions.
  • Does not work with AMD hardware.
  • Does not work with 32-bit guests.

If you encounter errors, please email with the following (please note that the guest config has “on_reboot=”preserve”, on_crash=”preserve” – which you should have in your guest config to contain the memory of the guest):

  • xl dmesg
  • xl list
  • xenctx -s $HOME/linux/System.map -f -a -C <domain id>
    [xenctx is sometimes found in /usr/lib/xen/bin/xenctx ]
  • The console output from the guest
  • Anything else you can think off.

to xen-devel@lists.xenproject.org
Stash away your vmlinux file (it is too big to send via email) – as we might need it later on.
That is it!
Thank you!

Read more

Xen Project Announces Performance and Security Advancements with Release of 4.19
Aug 05 2024

New release marks significant enhancements in performance, security, and versatility across various architectures.  SAN FRANCISCO – July 31st, 2024 – The Xen Project, an open source project under the Linux Foundation, is proud to announce the release of Xen Project 4.19. This release marks a significant milestone in enhancing performance, security,

Upcoming Closure of Xen Project Colo Facility
Jul 10 2024

Dear Xen Community, We regret to inform you that the Xen Project is currently experiencing unexpected changes due to the sudden shutdown of our colocated (colo) data center facility by Synoptek. This incident is beyond our control and will impact the continuity of OSSTest (the gating Xen Project CI loop)

Xen Summit Talks Now Live on YouTube!
Jun 18 2024

Hello Xen Community! We have some thrilling news to share with you all. The highly anticipated talks from this year’s Xen Summit are now live on YouTube! Whether you attended the summit in person or couldn’t make it this time, you can now access all the insightful presentations

Get ready for Xen Summit 2024!
May 24 2024

With less than 2 weeks to go, are you ready? The Xen Project is gearing up for a summit full of discussions, collaboration and innovation. If you haven’t already done so – get involved by submitting a design session topic. Don’t worry if you can’t attend in person,