Dom0 Memory -- Where It Has Not Gone

If you are upgrading domain 0 Linux kernel from a non-pvops (classic, 2.6.18/2.6.32/etc.) kernel to a pvops one (3.0 or later), you may find that the amount of free memory inside dom0 has decreased significantly.  This is because of changes in the way kernel handles the memory given to it by Xen.  With some updates and configuration changes, the “lost” memory can be recovered.

tl;dr: If you previously had ‘dom0_mem=2G’ as a command line option to Xen, change this to ‘dom0_mem=2G,max:2G’.  If that didn’t help, read on.

What Changed?

When a domain is started, it is provided with:

  1. A number of pages of memory. These are contiguous in the guest’s pseudo-physical address space and start at address 0.
  2. The maximum number of pages the domain is allowed to have (the maximum reservation).  This may be more than the initial number of pages to allow the the guest to balloon up (increase its number of pages).
  3. The e820 memory map.  For dom0, the physical e820 map is used so dom0 can access all BIOS data areas and PCI device MMIO regions.

The principle differences between the new and old kernels are:

  1. The new kernel allocates page tables and struct page’s for all the pages up to the maximum reservation, the older kernels only allocate enough for the current number of pages.  On systems with lots of RAM relative to the number of pages used by dom0 this can result in lots of memory wasted.
  2. The new kernel tries harder to release pages back to Xen that are unusable (because they are behind reserved regions or holes in the memory map).

Diagram showing how the kernel modifies the e820 map and how it releases pages that overlap with holes.
Give Me Back My Memory!
To stop the kernel from wasting memory on page tables etc. for memory it will never use:

  1. Use Xen dom0_mem=max:LLL command line option to set the maximum reservation to LLL (the M and G suffixes can be used for for MiB and GiB) .  e.g., If you previously had dom0_mem=XXXG, change this to dom0_mem=XXXG,max:XXXG.

This requires Xen 4.1.2 or later (which sets the maximum reservation based on the command line) and Linux 3.0.5 or later (which makes use of it)..
To recover the memory released during boot:

  1. Ensure the balloon driver is enabled in the kernel (CONFIG_XEN_BALLOON).
  2. Set the balloon driver’s target to the desired amount of RAM. Either with the mechanism provided by your toolstack or directly with (XXX is the amount of RAM in KiB):

    echo XXX > /sys/devices/system/xen_memory/xen_memory0/target_kb

I’m Lazy, Make it So
A series of patches by Konrad Rzeszutek Wilk will eliminate the need for the balloon driver.  With these patches, the otherwise unusable pages will be moved instead of released.  These patches are expected in 3.5.

Read more

Xen Project Announces Performance and Security Advancements with Release of 4.19
Aug 05 2024

New release marks significant enhancements in performance, security, and versatility across various architectures.  SAN FRANCISCO – July 31st, 2024 – The Xen Project, an open source project under the Linux Foundation, is proud to announce the release of Xen Project 4.19. This release marks a significant milestone in enhancing performance, security,

Upcoming Closure of Xen Project Colo Facility
Jul 10 2024

Dear Xen Community, We regret to inform you that the Xen Project is currently experiencing unexpected changes due to the sudden shutdown of our colocated (colo) data center facility by Synoptek. This incident is beyond our control and will impact the continuity of OSSTest (the gating Xen Project CI loop)

Xen Summit Talks Now Live on YouTube!
Jun 18 2024

Hello Xen Community! We have some thrilling news to share with you all. The highly anticipated talks from this year’s Xen Summit are now live on YouTube! Whether you attended the summit in person or couldn’t make it this time, you can now access all the insightful presentations

Get ready for Xen Summit 2024!
May 24 2024

With less than 2 weeks to go, are you ready? The Xen Project is gearing up for a summit full of discussions, collaboration and innovation. If you haven’t already done so – get involved by submitting a design session topic. Don’t worry if you can’t attend in person,