Robots Atlas>ROBOTS ATLAS
Infrastructure

CUDA VMM

2019ActivePublished: 23 September 2026Updated: 23 September 2026Published
Key innovation
Decoupling reservation of a virtual address range from the physical GPU memory backing it, enabling memory to be mapped, remapped and shared without copying data or paying a full-device synchronization cost.
Category
Infrastructure
Abstraction level
Primitive
Operation level
TrainingInferenceServing
Use cases
Expandable GPU buffers and allocations without copying dataCustom GPU allocators and memory poolsReducing memory fragmentation (e.g. PyTorch expandable_segments)Inter-process memory sharing (IPC)Selective peer-to-peer access across multiple GPUsGraphics API interoperability (Vulkan, OpenGL)Training and inference of large models with changing batch size

How it works

1) cuMemGetAllocationGranularity returns the required size granularity (alignment) for a given device. 2) cuMemAddressReserve reserves a contiguous virtual address range (with no physical memory). 3) cuMemCreate creates a handle to physical memory (backing store) whose size is a multiple of the granularity. 4) cuMemMap maps the physical handle into the reserved address range. 5) cuMemSetAccess grants access rights (read/write) for specified devices, which also controls peer-to-peer access. The inverse operations are cuMemUnmap, cuMemAddressFree and cuMemRelease. To grow an allocation, a larger address range is reserved and additional physical handles are mapped in without moving existing data. Cross-process sharing uses cuMemExportToShareableHandle and cuMemImportFromShareableHandle operating on OS handles: file descriptors on Linux and HANDLE / D3DKMT_HANDLE on Windows. CUDA does not support mapping part of a single physical allocation, so sizes must match.

Problem solved

High-level cudaMalloc fuses address reservation with physical allocation, so an existing allocation cannot be grown without reallocating and copying data, and freeing memory forces a full-device synchronization. This makes it hard to build growing buffers, memory pools, and to control fragmentation as well as memory sharing across devices and processes.

Components

cuMemAddressReserveAddress space reservation

Reserves a contiguous range of virtual GPU addresses without backing it with physical memory.

cuMemCreatePhysical memory allocation

Creates a handle to a physical memory allocation (backing store) sized as a multiple of the granularity.

cuMemMapBinding virtual address to physical memory

Maps a physical memory handle into the reserved virtual address range.

cuMemSetAccessPer-device access control

Sets access rights (read/write) to the mapped memory per device, also controlling peer-to-peer access.

cuMemGetAllocationGranularityDetermining size alignment

Returns the required size alignment (granularity); allocation sizes must be a multiple of it.

cuMemExportToShareableHandle / cuMemImportFromShareableHandleCross-process memory sharing

Export and import allocations as OS handles (file descriptor on Linux, HANDLE/D3DKMT_HANDLE on Windows), enabling sharing across processes and with graphics APIs.

Official

Implementation

Reference implementations
NVIDIA CUDA Samples (memMapIPCDrv, vectorAddMMAP)
C++/CUDA ยท NVIDIA
Official
PyTorch CUDACachingAllocator (expandable_segments)
C++/CUDA ยท PyTorch
Official
Implementation pitfalls
Not aligning to allocation granularityMedium

Sizes must be a multiple of cuMemGetAllocationGranularity; CUDA cannot map part of a single physical allocation.

Fix:Round sizes up to the granularity and match sizes when mapping.
Manual lifecycle managementHigh

The developer is responsible for the reserve โ†’ create โ†’ map โ†’ setAccess order and the unmap โ†’ release โ†’ addressFree teardown; mistakes cause leaks or access errors.

Fix:Wrap the operations in a custom allocator/RAII and test error paths.
Missing explicit peer access grantMedium

Another GPU's access to the memory requires an explicit cuMemSetAccess; without it access faults occur.

Fix:Grant access rights for every device that should use the allocation.

Evolution

2019
Introduction of the Low-Level Virtual Memory Management API in CUDA 10.2
Inflection point

CUDA 10.2 adds cuMemCreate, cuMemAddressReserve, cuMemMap and cuMemSetAccess, decoupling address reservation from physical allocation.

2020
NVIDIA publishes the guide "Introducing Low-Level GPU Virtual Memory Management"

The official guide describes usage patterns: growing allocations, memory pools, IPC and selective peer access.

2023
PyTorch introduces VMM-based expandable_segments

PyTorch's caching allocator gains the expandable_segments mode that uses VMM to reduce fragmentation under changing allocation sizes.

Hyperparameters (configurable axes)

Allocation granularityHigh

Physical memory size alignment required by the device; allocation sizes must be a multiple of it.

Shareable handle typeMedium

Type of OS handle used for sharing: POSIX file descriptor (Linux) or HANDLE / D3DKMT_HANDLE (Windows).

Access flagsHigh

Read/write rights set per device via cuMemSetAccess, also governing peer-to-peer access.

Hardware requirements

Primary

The CUDA driver API runs only on NVIDIA GPUs supporting virtual memory management (from CUDA 10.2).