sparse_mmap, membacking: add NUMA-aware memory allocation and mapping#3554
Draft
jstarks wants to merge 1 commit into
Draft
sparse_mmap, membacking: add NUMA-aware memory allocation and mapping#3554jstarks wants to merge 1 commit into
jstarks wants to merge 1 commit into
Conversation
Add cross-platform support for binding memory allocations and file mappings to specific host NUMA nodes. On Linux, this uses mbind(MPOL_BIND) after mmap to set the page allocation policy. On Windows, this passes MemExtendedParameterNumaNode to VirtualAlloc2 and MapViewOfFile3. The new SparseMapping methods (alloc_numa, map_file_numa) take Option<u32> for the NUMA node, falling through to the non-NUMA path when None. This avoids separate code paths at call sites. The NUMA node flows through the full membacking pipeline: RamBackingRequest -> RamBacking -> RegionMappingParams -> MappingParams -> VaMapper, so both private (anonymous) and file-backed (memfd/section) RAM backings get NUMA-bound at the point where pages are actually committed or mapped into the guest memory VA space.
|
This PR modifies files containing For more on why we check whole files, instead of just diffs, check out the Rustonomicon |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds optional host-NUMA-node binding to sparse memory allocations and file mappings, and threads that parameter through the OpenVMM membacking pipeline so RAM can be committed/mapped with NUMA affinity on both Linux and Windows.
Changes:
- Extend
SparseMappingwith NUMA-aware APIs (alloc_numa,map_file_numa) implemented viambind(MPOL_BIND)on Linux andMEM_EXTENDED_PARAMETERon Windows. - Plumb
Option<u32>NUMA node selection through membacking request/backing/mapping parameters intoVaMapper. - Add basic cross-platform tests covering NUMA node 0 success and invalid-node failure.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| support/sparse_mmap/src/windows.rs | Adds Windows NUMA binding support via extended parameters for VirtualAlloc2 / MapViewOfFile3; adds NUMA-aware mapping/allocation methods. |
| support/sparse_mmap/src/unix.rs | Adds Linux NUMA binding via mbind and exposes NUMA-aware allocation/mapping APIs on Unix. |
| support/sparse_mmap/src/lib.rs | Adds tests for NUMA-aware allocation and file mapping. |
| openvmm/membacking/src/region_manager.rs | Carries NUMA node through region mapping parameters into mapping manager requests. |
| openvmm/membacking/src/memory_manager/mod.rs | Adds host_numa_node to RamBackingRequest/RamBacking and passes it down to mapping/allocation. |
| openvmm/membacking/src/memory_manager/device_memory.rs | Updates region mapping calls to provide the new NUMA field (currently None). |
| openvmm/membacking/src/mapping_manager/va_mapper.rs | Uses NUMA-aware file mapping and passes NUMA selection into private-range allocation. |
| openvmm/membacking/src/mapping_manager/manager.rs | Extends MappingParams with a NUMA node field and updates tests accordingly. |
Comment on lines
+633
to
+640
| // with bit `numa_node` set. maxnode is the number of bits (1-based). | ||
| let word_bits = libc::c_ulong::BITS as usize; | ||
| let word_index = numa_node as usize / word_bits; | ||
| let bit_index = numa_node as usize % word_bits; | ||
| let num_words = word_index + 1; | ||
| let mut nodemask = vec![0 as libc::c_ulong; num_words]; | ||
| nodemask[word_index] = 1 << bit_index; | ||
| let maxnode = num_words * word_bits + 1; |
| ) -> Result<(), Error> { | ||
| self.alloc(offset, len)?; | ||
| if let Some(node) = numa_node { | ||
| self.mbind_at(offset, len, node)?; |
Comment on lines
+310
to
+314
| self.mbind_at(offset, len, node)?; | ||
| } | ||
| Ok(()) | ||
| } | ||
|
|
Comment on lines
128
to
138
| self.inner | ||
| .mapping | ||
| .map_file( | ||
| .map_file_numa( | ||
| range.start() as usize, | ||
| range.len() as usize, | ||
| &mappable, | ||
| file_offset, | ||
| writable, | ||
| numa_node, | ||
| ) | ||
| .expect("oom mapping file"); |
Comment on lines
+252
to
+257
| /// Bind this backing's memory to a specific host NUMA node | ||
| /// (Linux: `mbind(MPOL_BIND)`). Incompatible with `private_memory`. | ||
| pub fn host_numa_node(mut self, node: Option<u32>) -> Self { | ||
| self.host_numa_node = node; | ||
| self | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add cross-platform support for binding memory allocations and file mappings to specific host NUMA nodes.
On Linux, this uses mbind(MPOL_BIND) after mmap to set the page allocation policy. On Windows, this passes MemExtendedParameterNumaNode to VirtualAlloc2 and MapViewOfFile3.
The new SparseMapping methods (alloc_numa, map_file_numa) take Option for the NUMA node, falling through to the non-NUMA path when None. This avoids separate code paths at call sites.
The NUMA node flows through the full membacking pipeline: RamBackingRequest -> RamBacking -> RegionMappingParams -> MappingParams -> VaMapper, so both private (anonymous) and file-backed (memfd/section) RAM backings get NUMA-bound at the point where pages are actually committed or mapped into the guest memory VA space.