Linux Reserved Memory

Created by Confluence Wiki Admin (Unlicensed)
Last updated: Jun 15, 2020 by Terry O'NealVersion comment
One of the common requirements on Xilinx Zynq SoC/MPSoC based system is to reserve memory regions for special usage, excluding it from the usage of Linux kernel and making it available only for a custom device driver. This feature is covered by the reserved-memory framework and is closely related to the DMA-API and CMA frameworks within the kernel.

This article is intended to showcase and explain some of the use cases available, and has been tested using Petalinux build tool, but can be exported for the Yocto or OSL workflows, as the steps refer only to the DTS file customization and a custom device driver where the memory is allocated.

Table of Contents

Reserved memory

To reserve a memory range from system address space, the reserved-memory node can be used in the device-tree configuration. Each child node defines a specific memory space and can be configured according the different parameters available for the reserved memory nodes as described in the kernel docs. The reserved memory spaces can then be assigned to a specific device driver through the memory-region parameter.

Device-tree nodes within system-top.dts file for a 64-bit Cortex-A53 MPSoC:
   reserved-memory {
      #address-cells = <2>;
      #size-cells = <2>;
      ranges;
 
      reserved: buffer@0 {
         no-map;
         reg = <0x0 0x70000000 0x0 0x10000000>;
      };
   };
 
   reserved-driver@0 {
      compatible = "xlnx,reserved-memory";
      memory-region = <&reserved>;
   };
Or similar device-tree nodes for customization in the more recent Yocto-based Petalinux project-spec/meta-user/recipes-bsp/device-tree/files/system-user.dtsi for a 32-bit Cortex-A9 Zynq:
/include/ "system-conf.dtsi"
/ {
   reserved-memory {
      #address-cells = <1>;
      #size-cells = <1>;
      ranges;
 
      reserved: buffer@0x38000000 {
         no-map;
         reg = <0x38000000 0x08000000>;
      };
   };
 
   reserved-driver@0 {
      compatible = "xlnx,reserved-memory";
      memory-region = <&reserved>;
   };
};

In the device driver, the properties of the memory region can be handled parsing the device tree nodes, and once the physical address and size are known the memory region can be mapped using memremap/ioremap calls. Below the code referred to the reserved memory allocation:
/* Get reserved memory region from Device-tree */
np = of_parse_phandle(dev->of_node, "memory-region", 0);
if (!np) {
  dev_err(dev, "No %s specified\n", "memory-region");
  goto error1;
}
 
rc = of_address_to_resource(np, 0, &r);
if (rc) {
  dev_err(dev, "No memory address assigned to the region\n");
  goto error1;
}
 
lp->paddr = r.start;
lp->vaddr = memremap(r.start, resource_size(&r), MEMREMAP_WB);
dev_info(dev, "Allocated reserved memory, vaddr: 0x%0llX, paddr: 0x%0llX\n", (u64)lp->vaddr, lp->paddr);

As the reserved memory region has been excluded for the common usage by the kernel and marked as no-map, the iomem information (/proc/iomem) shows that the System RAM is less than the amount of memory in the board.
root@plnx_aarch64:~# cat /proc/iomem
00000000-6fffffff : System RAM
  00080000-00b37fff : Kernel code
  011c9000-012b8fff : Kernel data

Once the device is loaded, the allocation can be confirmed:
[  126.191774] reserved-memory reserved-driver@0: Device Tree Probing
[  126.198595] reserved-memory reserved-driver@0: Allocated reserved memory, vaddr: 0xFFFFFF8020000000, paddr: 0x70000000

Reserved memory through DMA API

Commonly the reserved memory spaces are being used with DMA engines, so integrating both frameworks can be useful from the device driver point of view. For that particular purspose the compatible property can be set as shared-dma-pool, generating a DMA memory pool reserved for a particular device driver.
reserved-memory {
      #address-cells = <2>;
      #size-cells = <2>;
      ranges;
 
      reserved: buffer@0 {
         compatible = "shared-dma-pool";
         no-map;
         reg = <0x0 0x70000000 0x0 0x10000000>;
      };
   };
 
   reserved-driver@0 {
      compatible = "xlnx,reserved-memory";
      memory-region = <&reserved>;
   };
This way the device driver only need to use the DMA API in the regular way, but instead of using the default CMA memory pool it will use the reserved memory region for this particular device.
/* Initialize reserved memory resources */
  rc = of_reserved_mem_device_init(dev);
  if(rc) {
    dev_err(dev, "Could not get reserved memory\n");
    goto error1;
  }
 
  /* Allocate memory */
  dma_set_coherent_mask(dev, 0xFFFFFFFF);
  lp->vaddr = dma_alloc_coherent(dev, ALLOC_SIZE, &lp->paddr, GFP_KERNEL);
  dev_info(dev, "Allocated coherent memory, vaddr: 0x%0llX, paddr: 0x%0llX\n", (u64)lp->vaddr, lp->paddr);

Kernel bootlog:
[    0.000000] Reserved memory: created DMA memory pool at 0x0000000070000000, size 256 MiB
[    0.000000] Reserved memory: initialized node buffer@0, compatible id shared-dma-pool
[    0.000000] cma: Reserved 128 MiB at 0x0000000068000000
Kernel log after loading the device driver :
root@plnx_aarch64:~# insmod /lib/modules/4.6.0-xilinx/extra/reserved-memory.ko
[   80.745166] reserved-memory reserved-driver@0: Device Tree Probing
[   80.750183] reserved-memory reserved-driver@0: assigned reserved memory node buffer@0
[   81.220878] reserved-memory reserved-driver@0: Allocated coherent memory, vaddr: 0xFFFFFF8020000000, paddr: 0x70000000

Reserved memory for CMA

Sometimes the reserved memory region does not require to be assigned to an specific device driver and is only intended to have a bigger CMA memory pool than the default one. For that particular use case, an extra property can be used to point to the kernel to use the reserved memory region as the default CMA memory pool.
reserved-memory {
      #address-cells = <2>;
      #size-cells = <2>;
      ranges;
 
      reserved: buffer@0 {
         compatible = "shared-dma-pool";
         reusable;
         reg = <0x0 0x70000000 0x0 0x10000000>;
         linux,cma-default;
      };
   };

The kernel bootlog confirms the custom CMA memory pool allocation:
[    0.000000] Reserved memory: created CMA memory pool at 0x0000000070000000, size 256 MiB
[    0.000000] Reserved memory: initialized node buffer@0, compatible id shared-dma-pool

Petalinux example

These use cases can be tested using the Petalinux build tool following the steps above:
  1. Generate a petalinux project using a BSP package (ZC702 for Zynq-7000 or ZCU102 for Zynq MPSoC)
  2. Modify the dts file to include the reserved memory node
  3. Create a driver module and modify the default content of the driver
  4. Build the project and launch it on QEMU

Related Links