Skip to content

Add RDMA sysfs topology construction for gVisor sandboxes - #13908

Open
copybara-service[bot] wants to merge 1 commit into
masterfrom
test/cl956129599
Open

Add RDMA sysfs topology construction for gVisor sandboxes#13908
copybara-service[bot] wants to merge 1 commit into
masterfrom
test/cl956129599

Conversation

@copybara-service

Copy link
Copy Markdown

Add RDMA sysfs topology construction for gVisor sandboxes

Containers with /dev/infiniband/uverbs* devices need a sysfs surface for
device discovery: libibverbs enumerates /sys/class/infiniband_verbs, and
NCCL builds its hardware topology by resolving /sys/class symlinks with
realpath() and walking the PCI hierarchy.

This adds a --rdmaproxy flag which will gate all RDMA support. For now it
only wires up the sysfs topology: runsc snapshots the host sysfs closure
derived from the spec's uverbs devices (ibdevs, netdevs, PCI ancestor
chains, GPU functions, NUMA nodes) before pivot_root, and the sentry
constructs a path-identical virtual tree: one canonical /sys/devices/pci...
subtree per device, with /sys/class/* and /sys/bus/pci/devices as symlink
farms matching the kernel layout. Per-port dynamic state (RoCE GID tables,
link state, counters) is served live through targeted read-only bind mounts
of the per-ibdev ports subtrees; identity and PCI attributes are a static
snapshot, since the collecting process and the constructed sandbox tree do
not observe the same sysfs mount (the RDMA netdevs move into the sandbox
network namespace only after the sandbox is created).

Each PCI function directory also gets a subsystem symlink into a
synthesized /sys/bus/pci, and every node's bus gets a pci_bus entry: NCCL
classifies a directory as a PCI device by following its subsystem symlink,
and aborts topology discovery without it.

The verbs device proxy that makes RDMA usable end-to-end is not part of
this change; --rdmaproxy is a work in progress and RDMA is not yet
functional.

Co-authored-by: Alessio Ricci Toniolo alessio@modal.com
FUTURE_COPYBARA_INTEGRATE_REVIEW=#13712 from ayushr2:rdma-sysfs 24b199b

@copybara-service copybara-service Bot added the exported Issue was exported automatically label Jul 29, 2026
Containers with /dev/infiniband/uverbs* devices need a sysfs surface for
device discovery: libibverbs enumerates /sys/class/infiniband_verbs, and
NCCL builds its hardware topology by resolving /sys/class symlinks with
realpath() and walking the PCI hierarchy.

This adds a --rdmaproxy flag which will gate all RDMA support. For now it
only wires up the sysfs topology: runsc snapshots the host sysfs closure
derived from the spec's uverbs devices (ibdevs, netdevs, PCI ancestor
chains, GPU functions, NUMA nodes) before pivot_root, and the sentry
constructs a path-identical virtual tree: one canonical /sys/devices/pci...
subtree per device, with /sys/class/* and /sys/bus/pci/devices as symlink
farms matching the kernel layout. Per-port dynamic state (RoCE GID tables,
link state, counters) is served live through targeted read-only bind mounts
of the per-ibdev ports subtrees; identity and PCI attributes are a static
snapshot, since the collecting process and the constructed sandbox tree do
not observe the same sysfs mount (the RDMA netdevs move into the sandbox
network namespace only after the sandbox is created).

Each PCI function directory also gets a subsystem symlink into a
synthesized /sys/bus/pci, and every node's bus gets a pci_bus entry: NCCL
classifies a directory as a PCI device by following its subsystem symlink,
and aborts topology discovery without it.

The verbs device proxy that makes RDMA usable end-to-end is not part of
this change; --rdmaproxy is a work in progress and RDMA is not yet
functional.

Co-authored-by: Alessio Ricci Toniolo <alessio@modal.com>
FUTURE_COPYBARA_INTEGRATE_REVIEW=#13712 from ayushr2:rdma-sysfs 24b199b
PiperOrigin-RevId: 956129599
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

exported Issue was exported automatically

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant