iomem and shared-memory

iomem is a xl config file option that can be used to map memory into a domU. Typically, it is used to map device memory into a domU as part of device assignment. However it can also be used to setup a cacheable shared memory region between dom0 and a domU. This section explains how to do that.

First, we have to add a reserved-memory node to the host device tree to advertise the special memory region to dom0, so that it won't use it to allocate memory as any other pages. For that, we can make use of the newly introduced "xen,shared-memory-v1" compatible string. For example:

    reserved-memory {
        #address-cells = <0x2>;
        #size-cells = <0x2>;
        ranges;

        xen-shmem@70000000 {
            compatible = "xen,shared-memory-v1";
            reg = <0x0 0x70000000 0x0 0x1000>;
        };
    };

This node tells dom0 that one page at 0x70000000 is to be use as reserved memory. Then, we need to do the same for DomU. We can do that by adding a device tree fragment to the DomU VM config file. The device tree fragment could be for example:

/dts-v1/;


/ {
    /* #*cells are here to keep DTC happy */
    #address-cells = <2>;
    #size-cells = <2>;


    passthrough {
        #address-cells = <2>;
        #size-cells = <2>;
        ranges;


        reserved-memory {
            #address-cells = <2>;
            #size-cells = <2>;
            ranges;


            xen-shmem@70000000 {
                compatible = "xen,shared-memory-v1";
                reg = <0x0 0x70000000 0x0 0x1000>;
            };
        };


        memory {             
            device_type = "memory";
            reg = <0x0 0x70000000 0x0 0x1000>;
        };
    };
};

Similarly to the dom0 example, it tells the domU kernel that the page at 0x70000000 is to be used as reserved memory. Note that we also added the range to a regular memory node, because it is required by device tree that all  reserved-memory ranges are also covered by the regular memory nodes.  We add the device tree fragment to the DomU device tree using the device_tree option in the VM config file, the same way we use it for device assignment:

device_tree = "/root/snippet.dtb"

Finally, we only need to map the page into the DomU address space at the right address, which in this example is 0x70000000. We can do that with the iomem VM config option. It is possible to specify the cacheability of the mapping, "memory" means normal cacheable memory:

iomem = ["0x70000,1@0x70000,memory"]

In this example, we are asking to map one page at physical address 0x70000000 into the guest pseudo-physical address space at 0x70000000. We are also asking to make the mapping a normal cacheable memory mapping.

Cacheable Shared Memory between Linux and Bare-metal guests

Please refer to the following docs in the Xen tree:

docs/man/xl-static-shm-configuration.pod.5

docs/man/xl.cfg.pod.5.in

This feature allows sharing one or more pages in memory across VMs, simply by adding one config option to the VM config file. Memory can be shared as cacheable memory. Here is an example:

# Add following to the config file of VM1: 
static_shm = ["id=ID1, begin=0x40000000, size=0x1000, role=owner"]

# Add following to the config file of VM2: 
static_shm = ["id=ID1, begin=0x48000000, size=0x1000, role=borrower"]

In this example we are sharing one page owned by VM1 with VM2. The owner of the page is the "owner" VM (VM1), while the other VM is called "borrower" (VM2). The memory of Xen VMs always starts at 0x40000000 (guest physical address), so in this example we are sharing the first page of VM1 with VM2. Make sure to choose a valid allocated address of the owner VM.

0x48000000 is the address where the page will appear in VM2. Make sure to choose an free address in VM2. In this example, if we assigned 128M of RAM to VM2, then the shared page will be mapped right at the end of memory.

The shared page is advertised as "reserved-memory" in the guest device tree. This is a sample device tree node of a guest with a static_shm configuration:

reserved-memory {
                #address-cells = <0x2>;
                #size-cells = <0x2>;
                ranges;

                xen-shmem@40000000 {
                        compatible = "xen,shared-memory-v1";
                        id = "ID1";
                        reg = <0x0 0x40000000 0x0 0x1000>;
                };

The guest is free to use the reserved-memory region as it wishes. For instance, in the case of the Linux kernel, it is possible to export the memory as cacheable memory to userspace with a dmabuf driver. See this proof of concept of Linux driver and user-space program:

Xen-dmabuf Linux driver

xen-dmabuf user space program

Dom0-less Shared Memory

  1. Carve out a range of memory to be used for sharing
    You can do that by editing the host device tree, reducing the amount of RAM in the memory node, and adding a separate new "mmio-sram" node as follows:

    --- mpsoc.dts.1 2019-10-31 16:00:35.574817353 -0700
    +++ mpsoc.dts.2 2019-10-31 16:00:24.850379706 -0700
    @@ -1738,6 +1738,11 @@
      
        memory {
            device_type = "memory";
    -       reg = <0x0 0x0 0x0 0x80000000 0x8 0x0 0x0 0x80000000>;
    +       reg = <0x0 0x0 0x0 0x7f000000 0x8 0x0 0x0 0x80000000>;
    +   };
    +
    +   sram@7f000000 {
    +       compatible = "mmio-sram";
    +       reg = <0x0 0x7f000000 0x0 0x1000000>;
        };
     };


    In this example the range 0x7f000000 - 0x80000000 was removed from the memory node and exposed as a special mmio-sram region to be assigned.

    If you don’t want dom0 to get access automatically to the range, add xen,passthrough; under the sram@7f000000 node.

  2. Share the sram region with dom0-less domUs
    Dom0 will get access to the sram region automatically (unless xen,passthrough; was specified). You can map the region to one or more dom0-less domUs by adding an sram node to the dom0-less DomU partial device tree. For instance:

    /dts-v1/;
     
    / {
        #address-cells = <0x2>;
        #size-cells = <0x1>;
     
        gic: gic {
            #interrupt-cells = <0x3>;
            interrupt-controller;
        };
     
        passthrough {
            compatible = "simple-bus";
            ranges;
            #address-cells = <0x2>;
            #size-cells = <0x1>;
     
            sram@7f000000 {
                compatible = "mmio-sram";
                reg = <0x0 0x7f000000 0x0 0x1000000>;
                xen,reg = <0x0 0x7f000000 0x1000000 0x0 0x7f000000>;
                xen,force-assign-without-iommu = <0x1>;
            };
        };
    };


    Note that the xen,reg property is key because it triggers the remapping of the memory range at the same location into the DomU. If you want to map the memory as cacheable is possible to use xen,reg-cacheable instead of xen,reg.

  3. Use the sram region
    When you boot the system, Dom0 and the DomUs will get a mapping of the sram region (0x7f000000 - 0x80000000) automatically. You can use the shared memory by calling `ioremap' in your kernel.