CXL.morsel: CXL-attached Virtual Memory

Problem

CXL.mem devices are emerging as key players in modern clou and high-performance systems, allowing memory to be disaggregated and shared across multiple systems via the CXL (Compute Express Link) protocol. However, CXL memory expanders are currently meant as simple extensions of host physical memory.

This thesis proposes an approach where a CXL.mem expander does not just offer plain physical memory but virtualizes the memory using an on-device MMU (Memory Management Unit). By allowing the memory expander to handle page-table walks locally, we can significantly reduce the overhead on the host CPU and enhance memory access performance.

This thesis is part of the ongoing effort to establish Morsels as first-class operating-system primitive. Morsels are self-contained virtual-memory that come with their own page-table structure. Thereby, Morsels are easy to share between processes and devices.

Goal

The goal of this thesis is to build an functional prototype for an Morsel-mode within a virtualized CXL.mem device (e.g. within QEMU). For this, we require an concept for on-device MMUs. The thesis will explore how to:

1. Dynamically attaching CXL.mem devices at run-time.
2. Enable on-device page table walks (e.g. page-table trees, hashed page tables)
3. Integrate CXL.morsels with existing kernel implementation

Background: CXL and Memory Virtualization

The CXL standard is a key technology for enabling memory pooling and disaggregation in modern data centers. The CXL.mem protocol allows devices to connect directly to the memory fabric and provide memory to host systems. CXL memory expanders are seen as passive memory providers, leaving the host CPU responsible for managing virtual memory.

A Memory Management Unit (MMU) is a critical component that handles virtual-to-physical memory address translations. In traditional systems, page table walks — the process of translating virtual addresses to physical addresses — are handled by the CPU. By shifting this operation to an MMU on the CXL.mem device itself, it is possible to free the CPU from this task and potentially reduce memory access latency.

Requirements

• Strong C and C++ programming skills.
• Familiarity with memory management concepts, particularly virtual memory.
• Interest in working with hardware-software interactions, including low-level memory management and performance optimization.

Related Work

• [1] M. K. Aguilera, E. Amaro, N. Amit, E. Hunhoff, A. Yelam, and G. Zellweger, “Memory disaggregation: why now and what are the challenges,” SIGOPS Opererating System Review, vol. 57, no. 1, pp. 38–46, Jun. 2023, doi: 10.1145/3606557.3606563.
• [2] S. W. Stark, A. T. Markettos, and S. W. Moore, “How Flexible is CXL’s Memory Protection? Replacing a sledgehammer with a scalpel,” Queue, vol. 21, no. 3, pp. 54--64, Jul. 2023, doi: 10.1145/3606014.
• [3] H. Ham et al., “Low-overhead General-purpose Near-Data Processing in CXL Memory Expanders,” Apr. 30, 2024, arXiv: 2404.19381v2. Accessed: Sep. 19, 2024. [Online]. Available: https://arxiv.org/abs/2404.19381v2