Add AWS EFA support to rdmaproxy - #13882
Open
ayushr2 wants to merge 5 commits into
Open
Conversation
Collaborator
Author
Add support for NV_ESC_EXPORT_TO_DMABUF_FD, which CUDA issues to export GPU memory as a dma-buf file descriptor. This is the mechanism used for GPUDirect RDMA: the returned fd is handed to ibv_reg_dmabuf_mr so a NIC can DMA directly to and from GPU memory. The driver allocates a host dma-buf and returns its fd in the ioctl's FD field; nvproxy wraps that host fd in a sentry file description installed in the application's fd table. The wrapper implements vfs.HostFDProvider so the RDMA device proxy can recover the host fd for registration. This ioctl exposes GPU memory to external devices, so it is gated behind a new privileged driver capability, CapRDMA, and is only permitted (and its seccomp filter only installed) when that capability is enabled. CapRDMA is enabled automatically when RDMA support is requested. Co-authored-by: Alessio Ricci Toniolo <alessio@modal.com>
Document RDMA (InfiniBand/RoCE) support: what it does, how to enable it (--rdmaproxy), GPUDirect via dma-buf, and the current limitations (Mellanox-only, single-container sandboxes, device lifecycle, RoCE shared namespace mode, no checkpoint). Co-authored-by: Alessio Ricci Toniolo <alessio@modal.com>
Add pkg/sentry/devices/rdmaproxy, a passthrough proxy for /dev/infiniband/uverbs* that lets a sandboxed application drive RDMA verbs against the host's ib_uverbs devices. The proxy forwards the modern RDMA_VERBS_IOCTL to a host uverbs FD. Every request is validated against an explicit per-(object, method) schema, and each attribute is translated strictly by its schema-declared ABI type rather than by probing guest memory: input pointers are copied into sentry buffers, output buffers are staged and copied back, object handles are passed through, and file descriptors are translated between the sandbox and the host. Interpreting an attribute's direction from the fixed kernel ABI (additive-only, stable since Linux 4.20) instead of from the guest's memory is what prevents a malicious guest from coercing the proxy into an out-of-bounds copy. DMA buffers referenced by memory-region registrations and by the vendor driver-private CQ/QP data are pinned and mirrored into the sentry's address space so the host kernel can pin_user_pages on them. The mlx5 (ConnectX) driver-private layout lives in a vendor plug-in, pkg/sentry/devices/rdmaproxy/ cxproxy, registered with the core at init time. A new MemoryManager.FindVMARange helper bounds a driver-private DMA buffer whose backing region the guest describes by start address only. Co-authored-by: Alessio Ricci Toniolo <alessio@modal.com>
Register a proxied uverbs char device for each /dev/infiniband/uverbs* device listed in the spec when --rdmaproxy is set. Devices are registered at their host major:minor (from the spec), so the guest device node, its /sys .../dev entry, and the VFS registration all agree without any renumbering. The vendor driver plug-in is selected per device from the PCI driver name in the host sysfs snapshot; the mlx5 (cxproxy) plug-in is linked in via a blank import. Install the rdmaproxy seccomp filters (a new Options.RDMAProxy in the boot filter config, expanded in the precompiled set) whenever RDMA is enabled, and donate the uverbs device FDs through the dev gofer (ShouldExposeRDMADevice in SetupDev, and shouldCreateDeviceGofer). GPUDirect RDMA additionally needs the privileged CapRDMA nvproxy capability to export GPU memory to a dma-buf fd; since there is no container-facing flag for it, enable it implicitly when --rdmaproxy is set. Co-authored-by: Alessio Ricci Toniolo <alessio@modal.com>
Add an efaproxy driver plug-in for AWS Elastic Fabric Adapter (EFA) devices
bound to the host `efa` driver. EFA's driver-private uverbs ABI carries no
userspace buffer pointers on CQ/QP CREATE: queue memory is kernel-allocated
and handed back via mmap keys in the CREATE response, which the core already
forwards to the host FD verbatim. PrepareCreateDMA is therefore a no-op; the
plug-in exists to register the "efa" driver name and to model EFA's MR_QUERY
driver-namespace method (used by libfabric to fetch RDMA read/write
interconnect IDs).
Model the standard AH object's destroy method (UVERBS_OBJECT_AH): EFA (SRD)
creates address handles via the legacy write path and destroys them through
the modern object, which the core must allow.
Extend the virtual RDMA sysfs for libfabric/hwloc topology discovery:
- device/driver symlink and /sys/bus/pci/drivers/<driver> tree, which
libfabric's EFA provider resolves during device discovery.
- Mirror each PCI node's raw config space ("config" file). hwloc (used by
aws-ofi-nccl to build the NCCL topology) reads it to recover the PCI
bridge bus-number registers; without it hwloc cannot reconstruct the
bridge hierarchy and aws-ofi-nccl aborts its topology write.
- Serve the per-port lid_mask_count attribute alongside the other port
attributes libibverbs reads from sysfs; ib_core creates it on every
port.
The sysfs integration tests cover the new driver tree and config mirroring.
Advertise a >= 5.12 kernel release in RDMA-enabled sandboxes so aws-ofi-nccl
enables dmabuf-based GPU memory registration (ibv_reg_dmabuf_mr), which the
proxy handles via the existing REG_DMABUF_MR path. Sandboxes without RDMA
keep the current 4.19 release to limit the bump's blast radius; a later
change can make the newer release the default.
Co-authored-by: Alessio Ricci Toniolo <alessio@modal.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add an efaproxy driver plug-in for AWS Elastic Fabric Adapter (EFA) devices
bound to the host
efadriver. EFA's driver-private uverbs ABI carries nouserspace buffer pointers on CQ/QP CREATE: queue memory is kernel-allocated
and handed back via mmap keys in the CREATE response, which the core already
forwards to the host FD verbatim. PrepareCreateDMA is therefore a no-op; the
plug-in exists to register the "efa" driver name and to model EFA's MR_QUERY
driver-namespace method (used by libfabric to fetch RDMA read/write
interconnect IDs).
Model the standard AH object's destroy method (UVERBS_OBJECT_AH): EFA (SRD)
creates address handles via the legacy write path and destroys them through
the modern object, which the core must allow.
Extend the virtual RDMA sysfs for libfabric/hwloc topology discovery:
libfabric's EFA provider resolves during device discovery.
aws-ofi-nccl to build the NCCL topology) reads it to recover the PCI
bridge bus-number registers; without it hwloc cannot reconstruct the
bridge hierarchy and aws-ofi-nccl aborts its topology write.
attributes libibverbs reads from sysfs; ib_core creates it on every
port.
The sysfs integration tests cover the new driver tree and config mirroring.
Advertise a >= 5.12 kernel release in RDMA-enabled sandboxes so aws-ofi-nccl
enables dmabuf-based GPU memory registration (ibv_reg_dmabuf_mr), which the
proxy handles via the existing REG_DMABUF_MR path. Sandboxes without RDMA
keep the current 4.19 release to limit the bump's blast radius; a later
change can make the newer release the default.
Co-authored-by: Alessio Ricci Toniolo alessio@modal.com