rdma_topo: NUMA-based topologies and multi-PF systems#1779
Open
EdwardSro wants to merge 7 commits into
Open
Conversation
Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com> Signed-off-by: Edward Srouji <edwards@nvidia.com>
Replace the TOPO_NOT_SUPPORTED singleton and has_cx_dma/has_gpu_and_nic
boolean flags with structured alternatives:
* CommandError.Level (ERROR/WARNING) with self-formatting __str__
* TopoNotSupportedError: unrecognized topology (expected on
yet to be supported platforms)
* TopoUnexpectedError: inconsistency in the topology which should not
happen in practice (prompts the user to report the issue)
* DumpParsingError: malformed sysfs dump input
* PCITopo.TopoType enum (INLINE/DMA) replacing the boolean pair
A prior patch introduced a regression where certain topologies that were
previously rejected cleanly by printing the "not supported" message
instead fell through to the inline topology path and raised an unhandled
ValueError.
Drop .supported property from PCITopo and all post-construction checks
done by callers.
Fixes: f8f0f10 ("rdma_topo: Add support for inline topologies")
Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com>
Signed-off-by: Edward Srouji <edwards@nvidia.com>
There is no reason to skip this boolean when False. Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com> Signed-off-by: Edward Srouji <edwards@nvidia.com>
Will be used to group CX NICs with the same SN into NIC boards. Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com> Signed-off-by: Edward Srouji <edwards@nvidia.com>
Add topology model that groups CX PFs into NICs (by shared parent bridge) and NICs into boards (by shared VPD SN). Useful for future system topologies with multiple CX PFs in the same complex. Produce both JSON (to_dict) and human-readable (topo_str) output with deterministic ordering throughout. Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com> Signed-off-by: Edward Srouji <edwards@nvidia.com>
Cover all branches of NVCX_Topo and its nested PF, NIC, and Board types using mock PCIDevice objects. Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com> Signed-off-by: Edward Srouji <edwards@nvidia.com>
Some platforms connect GPUs directly under a CPU root port rather than through a shared CX switch, making the existing Inline and DMA topology detectors inapplicable. Add a new type of complex that groups NICs and GPUs by NUMA node. Add two new root port device IDs mapped to a new vera_rp device type. Move parent/child PCI device linking into __load_devices so the parent relationship is available during topology type detection. Signed-off-by: Vlad Dumitrescu <vdumitrescu@nvidia.com> Signed-off-by: Edward Srouji <edwards@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extend the rdma_topo helper to cover a class of NIC/GPU platforms in which the RDMA NICs and the GPU are each attached directly under a CPU root port rather than through a shared PCIe switch. Prior to this series the tool recognized only:
Grace-Blackwell (GB200) + ConnectX-7, Grace-Vera (Vera Rubin), and similar NIC-on-CPU-root platforms don't match either shape. Attempting to classify them as inline caused rdma_topo topo to abort with an unhandled ValueError traceback when a GPU was directly attached to a Grace root port.
This series introduces a third topology class, NUMA-based, that recognizes those platforms explicitly.
Add infrastructure and tests for grouping CX PFs into a hierarchical NIC / Board model, in preparation for machines that expose more than one CX PF per complex.
The new NUMA-based topology is the first end-to-end consumer of the infrastructure.
The series also overhauls error handling in PCITopo construction, converting several previously unhandled ValueError sites to a typed CommandError so unsupported topologies produce a single friendly line instead of a Python traceback.