Fix GB200 gpu_roce decoding-server validation issues - #683
Merged
vedika-saravanan merged 6 commits intoJul 14, 2026
Conversation
vedika-saravanan
force-pushed
the
relaybp-gpuroce-gb200-fixes
branch
from
July 14, 2026 13:54
017229c to
8e5e9f7
Compare
…ge alignment, private dispatch symbols Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
vedika-saravanan
force-pushed
the
relaybp-gpuroce-gb200-fixes
branch
from
July 14, 2026 13:56
8e5e9f7 to
75780c3
Compare
Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
vedika-saravanan
force-pushed
the
relaybp-gpuroce-gb200-fixes
branch
from
July 14, 2026 14:04
75780c3 to
3de6452
Compare
vedika-saravanan
force-pushed
the
relaybp-gpuroce-gb200-fixes
branch
2 times, most recently
from
July 14, 2026 16:15
a92b03c to
fc0ceaf
Compare
vedika-saravanan
marked this pull request as ready for review
July 14, 2026 16:29
Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
vedika-saravanan
force-pushed
the
relaybp-gpuroce-gb200-fixes
branch
from
July 14, 2026 16:39
fc0ceaf to
df7d51e
Compare
bmhowe23
reviewed
Jul 14, 2026
bmhowe23
reviewed
Jul 14, 2026
Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
Signed-off-by: vedika-saravanan <vsaravanan@nvidia.com>
bmhowe23
approved these changes
Jul 14, 2026
bmhowe23
left a comment
Collaborator
There was a problem hiding this comment.
LGTM once Chuck's threads are marked as resolved.
bmhowe23
added a commit
to bmhowe23/cudaqx
that referenced
this pull request
Jul 15, 2026
…ridge branch Main landed the squashed PR 670 plus NVIDIA#656/NVIDIA#673/NVIDIA#674/NVIDIA#678/NVIDIA#680/NVIDIA#683/NVIDIA#688/ NVIDIA#690/NVIDIA#691/NVIDIA#692. Resolution keeps this branch's design and ports main's content into it: - Naming: main's GpuRoce{Transceiver,Factory,LinkCheck} map 1:1 onto this branch's DeviceGraph* files (the de-transport-ing follow-up requested in the PR 670 review). Main's post-review fixes are ported into the renamed files: ring-size overflow + host-page-size alignment validation, gpu_id resolved from the decoder's cuda_device_id instead of an env var, the factory taking pinned_cuda_device, and the linkcheck exercising the new factory signature. - Schema: per-decoder 'dispatch: host|device_graph' plus the top-level transport section stay; main's per-decoder 'transport:' key does not return. cuda_device_id is adopted as a per-decoder placement knob (config struct + YAML trait), and the hsb script now injects 'dispatch: device_graph' + cuda_device_id and drives the server with QEC_DEVICE_GRAPH_* env and the device_graph READY sentinel. - Features adopted from main: decoder CUDA-device pinning (worker-thread pin via promise, graph-capture pin, hardware_guards.h), per-session decode counters + print_session_stats + QEC_DECODING_SERVER_STATS server-side stats printing, DecodingServer ctor exception safety, virtualized realtime decoder API (NVIDIA#674), CMP0126 fix, CUDA::cudart on the core lib, and the cudevice-archive dedup in the server link. - DecodingSession combines main's set_graph_capture_device with this branch's reserved-SMs decode-graph capture (QEC_DEVICE_GRAPH_RESERVED_SMS). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Ben Howe <bhowe@nvidia.com>
anjbur
added a commit
to anjbur/cudaqx
that referenced
this pull request
Jul 16, 2026
)" This reverts commit 2cdc9df. Signed-off-by: Angela Burton <angelab@nvidia.com>
melody-ren
added a commit
that referenced
this pull request
Jul 17, 2026
…698) ## Background - #690 introduced the cuda_device_id placement knob for GPU decoders. - #678 made worker threads and the inproc graph capture honor the pinned device, and pinned decoder construction. - #683 closed the remaining gap: the gpu_roce decoding-server capture path now pins too. Each fix added a device pin to one more path via its own ad-hoc helper. The "which device" decision ended up duplicated across dispatch, capture, the worker, construction, and the gpu_roce transport. And that is the motivation for this PR. ## What this PR does Basically, every path should be following a single source of truth. Consolidate all of it to: the decoder's `cuda_device_id`, resolved one way (`decode_device_for`) and applied through two sanctioned wrappers — `pin_decode_device` (dispatch/decode) and `capture_graph_pinned` (graph capture) — in `hardware_guards.h.` Every path now derives its device from that one field; none selects a device by any other means. Unpinned (< 0) keeps existing semantics: dispatch no-ops, capture defaults to device 0. Signed-off-by: Melody Ren <melodyr@nvidia.com>
cketcham2333
added a commit
that referenced
this pull request
Jul 24, 2026
…r decoder, mixed host + device_graph dispatch (#682) Built on the companion cuda-quantum PR NVIDIA/cuda-quantum#4915, **now merged upstream** (squash `2a7911f`, 2026-07-23); `.cudaq_version` pins that commit, so this branch builds against stock `NVIDIA/cuda-quantum` main. ## Summary This PR re-architects the realtime decoding server around the CUDA-Q bridge-provider boundary. The server no longer contains any transport code: the wire is named in the deployment YAML (or a `--transport` fallback), loaded at runtime as a `libcudaq-realtime-bridge-<name>.so` provider, and every decoder gets its own ring buffer and its own consumer -- a host dispatcher thread for `dispatch: host` decoders, or the GPU device-graph scheduler for `dispatch: device_graph` decoders, both side by side in one server process. A partner transport library drops in as a single `.so` with zero decoding-server changes. The guiding litmus test for the layering: cudaq-realtime and its transport providers never say "QEC" or "decoder"; the QEC library code never names a wire (UDP, RoCE, hololink). Wire names appear only where deployments are described: YAML configs, launch scripts, and tests. ## Before (main prior to #670; #670 landed the first step) ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ decoding_server (pre-refactor) │ │ compiled-in wire branches, one per transport: │ │ #ifdef udp path #ifdef cpu_roce path gpu_roce fork │ │ udp wrapper calls RendezvousInfo + hsb_fpga GpuRoceTransceiver = │ │ wired directly QP handshakes INLINE hololink bring-up │ │ (byte-exact copy of the FUSED with the QEC │ │ CpuRoceChannel structs) dispatch engine │ │ hand-rolled std::thread around cudaq_host_ring_dispatch_loop │ │ ONE ring buffer + ONE dispatch loop shared by ALL decoders │ │ two transport knobs that had to agree (--transport + per-decoder YAML) │ └──────────────────────────────────────────────────────────────────────────┘ │ link-time: udp/cpu_roce wrappers; DOCA + hololink + HSB + ibverbs ▼ adding a wire = editing this repo; no partner drop-in possible ``` - The server had per-transport code paths: udp and cpu_roce wired directly against transport wrapper headers, and a separate gpu_roce fork that linked hololink/DOCA/HSB/ibverbs into the QEC libraries (`DT_NEEDED` on the server binary). Adding a wire meant editing the server. - Transport selection was two knobs that had to agree (`--transport` on the CLI AND a per-decoder YAML key), with silent misconfiguration when they disagreed. - One ring buffer served all decoders: every RPC funneled through a single dispatcher, and per-decoder endpoints (the topology the product needs: caller routes with `device_id == decoder_id`) were impossible. - The GPU device-graph path (`gpu_roce`) was hololink-only, named after one wire, and could not coexist with host-dispatch decoders in the same process. - The CPU-RoCE rendezvous / HSB-FPGA QP handshake lived in the server itself instead of behind the transport boundary. - #670 (now on main) took the first step -- the gpu_roce engine moved out of the core library behind a weak factory -- but the component still links hololink/DOCA/HSB directly, serves exactly one decoder, is named for one wire, and the per-decoder `transport:` key remains the selection knob. ## After ``` ┌──────────────────────────────────────────────────────────────────────────┐ │ decoding_server (transport-blind) │ │ YAML `dispatch:` picks the ENGINE YAML `transport:` picks the WIRE │ │ (per decoder) (per deployment; --transport is │ │ a fallback, conflict = error) │ │ ONE RING PER DECODER, each with its own consumer: │ │ host ─► dispatcher object over <name> ► libcudaq-realtime- │ │ its provider ring (4869 API) bridge-<name>.so │ │ device_graph ─► DeviceGraphRing- /path.so ► partner library, │ │ Consumer (GPU scheduler) verbatim, zero changes │ │ geometry + READY ring tokens derived FROM the providers (v2 queries) │ └──────────────────────────────────────────────────────────────────────────┘ │ links │ links (weak factory, WHOLE_ARCHIVE) ┌────────────────────┐ ┌────────────────────────────────────────────────┐ │ CQR plugin │ │ DecodingServer core · device-graph component: │ │ (HOST_CALL table) │ │ DeviceGraphTransceiver = dispatch ENGINE only │ └────────────────────┘ └────────────────────────────────────────────────┘ │ links (the ONLY realtime link dependency) ┌──────────────────────────────────────────────────────────────────────────┐ │ libcudaq-realtime.so — bridge loader (iface v2) + dispatcher object │ └──────────────────────────────────────────────────────────────────────────┘ ◌ dlopen ─────────────── runtime plug-in seam ────────────── ◌ dlopen ┌────────────┬──────────────────┬─────────────────┬────────────────────────┐ │ bridge-udp │ bridge-cpu-roce │ bridge-hololink │ partner transport .so │ │ .so │ .so (rendezvous │ .so (built on │ (out of tree, ~9 C │ │ │ + hsb_fpga) │ the HSB rig) │ functions) │ └────────────┴──────────────────┴─────────────────┴────────────────────────┘ ``` - **Bridge-provider-only server.** `decoding_server` speaks only YAML + the `cudaq_bridge_*` C API. Provider libraries resolve by name next to the cudaq-realtime install (`QEC_BRIDGE_PROVIDER_DIR`) or verbatim by path (partner drop-in). The QEC libraries link no transport libraries; the rendezvous/hsb_fpga handshake moved down into the cpu_roce provider (companion cuda-quantum PR). - **Engine vs. wire, each named once.** Per-decoder YAML `dispatch: host|device_graph` picks HOW RPCs execute; the top-level, shape-keyed `transport:` section (`provider`, `args`, plus a `device_graph:` override for the rings that must be GPU-pollable) picks the wire for the whole deployment. `--transport` is a fallback for configs that intentionally leave the wire unspecified (one YAML reused across wires, selected per launch); a YAML that names a provider combined with an explicit `--transport` is a startup error, never a silent precedence decision. All pre-release aliases (`transport:` as a per-decoder key, `--transport=gpu_roce`, `HOLOLINK_*` env fallbacks) are removed -- this component is new this cycle, so there is no compatibility surface to keep. - **One ring per decoder.** The server opens one bridge instance + one ring + one consumer per YAML decoder; ring geometry and endpoint identity come from the provider's v2 queries instead of re-parsed CLI. Readiness publishes every endpoint in one line (`QEC_DECODING_SERVER_READY port=<p0> ... ring<id>=<port>`) before any blocking connect, and shutdown reports per-ring traffic (`QEC_DECODING_SERVER_RING decoder=<id> dispatched=<n>`). Callers route with `cudaq::device_call(decoder_id, ...)` and per-device endpoint args (`udp-port.<id>=<port>`). - **Mixed dispatch in one process.** `DeviceGraphRingConsumer` runs the CUDA-Q device-graph scheduler (self-relaunching GPU dispatch graph + the decoder's captured decode graph) as a ring consumer over any GPU-pollable ring -- hololink DOCA rings on a rig, or pinned+mapped udp rings (`--pinned-rings`) on any CUDA box. `GpuRoceTransceiver` was renamed to what it actually is (`DeviceGraphTransceiver`, the dispatch engine, transport-blind) and now delegates to the ring consumer; the standalone all-device_graph path remains for the HSB flow. - **Device-graph wedge root-caused and fixed.** The scheduler deadlock after the first decode trigger was cooperative-launch co-residency starvation: the decode graph was captured sized for every SM and could never become co-resident with the persistent dispatch graph, so the device-side fire queued forever at `grid.sync()`. `DecodingSession` now captures with `reserved_sms=1` (`QEC_DEVICE_GRAPH_RESERVED_SMS` to raise it on rigs where hololink RX/TX kernels are also resident). The same bug class affects host-path GPU-cooperative decodes when a scheduler is resident -- documented, with the mixed-deployment guidance of a CPU decoder on host rings until the plugin plumbs reservation into its host path. - **Works with the session stats from main.** `QEC_DECODING_SERVER_STATS=1` prints per-decoder counters in the mixed server too; they are host-session counters, so a `device_graph` ring legitimately reports `decodes=0` -- its execution evidence is the trigger diagnostics (`fires == tail_relaunches`) and the per-ring `dispatched=` line. - **Tests and config schema.** New coverage: transport-section parse/round-trip with a mixed host+device_graph decoder set, a two-process decode where the YAML alone names the wire, CLI/YAML conflict rejection, a per-decoder-rings two-process test asserting traffic on each ring, and `app_examples.surface_code-4-yaml-mixed-dispatch` -- the flagship mixed flow as a gated ctest (registered only when the server links the device-graph component; skips without a GPU or the nv-qldpc plugin) asserting scheduler health via the new trigger diagnostics (rc=0, fires == tail_relaunches > 0). The generated JSON config schema (`decoder_config_json_schema()`) advertises the new per-decoder `dispatch:` key and the top-level `transport:` section (the removed per-decoder `transport:` key is gone from the schema too), with a python `jsonschema` test that validates a populated `dispatch`/`transport` document against it and rejects the removed key. The python config bindings expose the new fields as well -- `decoder_config.dispatch`, `multi_decoder_config.transport`, and the `decoder_dispatch` / `transport_config` / `transport_shape_override` types. ## Sample configurations Three representative deployment shapes (YAML config + exact launch line + matching caller config): **Two host decoders, one udp ring per decoder** -- the YAML says nothing about the wire, so `--transport` (default udp) selects it per launch; the READY line publishes every ring's endpoint. ```yaml decoders: - id: 0 type: pymatching # block_size / syndrome_size / H_sparse / ... - id: 1 type: pymatching # ... ``` ``` decoding_server --config=decoders.yaml # QEC_DECODING_SERVER_READY port=<P0> transport=udp ring0=<P0> ring1=<P1> ``` Caller routes per decoder with device-scoped endpoint args: `--cudaq-device-call=udp udp-host=127.0.0.1 udp-port=<P0> udp-port.1=<P1>`. **Mixed dispatch (host CPU decoder + GPU device-graph decoder), runs on any CUDA box** -- the wire lives in the YAML transport section; the shape-keyed override adds `--pinned-rings` only to the device_graph ring so the GPU scheduler can poll it. No `--transport` on the command line (combining both is a startup error). This is exactly what the new gated ctest runs. ```yaml transport: provider: udp device_graph: args: [--pinned-rings] decoders: - id: 0 type: multi_error_lut # ... - id: 1 type: nv-qldpc-decoder dispatch: device_graph # ... ``` ``` decoding_server --config=decoders.yaml ``` **Partner transport drop-in** -- an out-of-tree provider library is named by path, verbatim; its args are forwarded untouched. Zero decoding-server changes. ```yaml transport: provider: /opt/partner/libpartner_bridge.so args: [--lane=3] decoders: - id: 0 type: pymatching # ... ``` ``` decoding_server --config=decoders.yaml ``` ## Validation - Two-process suite 6/6 (udp; includes per-decoder rings and the YAML-transport-section tests), decoding-server core + YAML suites, per-decoder-rings app example. - Re-validated after merging main with #670 landed (#656/#673/#674/#678/#680/#683/#690/#691/#692): two-process suite, DecoderYAMLTest 18/18, per-decoder-rings + mixed-dispatch app tests, python surface_code-1, and the full mixed E2E all green against an nv-qldpc plugin rebuilt for #674's virtualized decoder API (an ABI break for out-of-tree plugin builds). - Full mixed E2E on a WSL2 laptop, two-process, wire named only by the YAML: multi_error_lut on a host ring + nv-qldpc RelayBP behind the GPU device-graph scheduler on a pinned-udp ring -- decode graph fired for every window (trigger rc=0, fires == tail_relaunches), correct corrections, both rings dispatched, clean teardown. The HSB rig is no longer required to exercise the device-graph dispatch path; rig runs remain the validation for the hololink and cpu_roce RDMA data planes. ## Breaking changes vs. main All of these surfaces are new this release cycle, so no deprecation aliases are kept: - The per-decoder `transport:` YAML key and the `DecoderTransport` enum are replaced by per-decoder `dispatch: host|device_graph` plus the top-level `transport:` section. A config using the old key fails to parse loudly. `cuda_device_id` (#690) is kept unchanged and now also places the device_graph rings and scheduler. - The device-graph transceiver's environment surface is `QEC_DEVICE_GRAPH_*` (was `HOLOLINK_*`), and `--transport=gpu_roce` no longer exists: device_graph is a dispatch shape named in the YAML; `--transport` only ever names a wire provider. The HSB test script is updated to match (injects `dispatch: device_graph` + `cuda_device_id`, sets `QEC_DEVICE_GRAPH_*`, waits for `READY device_graph`). - `CpuRoceTransceiver` (the always-throwing placeholder) is removed; cpu_roce is served by its bridge provider. ## Dependencies - **Built on the companion cuda-quantum PR [#4915](NVIDIA/cuda-quantum#4915 ([realtime] pluggable transport providers / bridge interface v2 + per-device device_call sessions), **merged upstream 2026-07-23** (squash `2a7911f`) and itself built on cuda-quantum #4869. `.cudaq_version` pins that squash commit, so this branch builds against stock main. This PR consumes `cudaq_bridge_create_from_library`, the v2 endpoint/geometry queries, per-device sessions and device-scoped channel args, the udp provider's `--pinned-rings`, and `cudaq_dispatch_get_trigger_debug`. - **Built on main with #670 landed** (origin/main is merged into this branch). Relative to landed #670, this PR completes the follow-up its review called for -- removing the server's transport awareness: `GpuRoce{Transceiver,Factory,LinkCheck}` become `DeviceGraph*` (same weak-factory / optional-component structure, now transport-blind), while #670's post-review hardening (ring-size and host-page-alignment validation, `cuda_device_id`-driven GPU placement threaded through the factory) is preserved in the renamed files. - The nv-qldpc decoder plugin and the proprietary cudevice archive are consumed as prebuilt binaries (`CUDAQ_QEC_REALTIME_CUDEVICE_PROPRIETARY_ARCHIVE`); builds without them still compile and the device-graph paths fail at runtime with a clear not-linked error, and the gated ctest is simply not registered. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Signed-off-by: Ben Howe <bhowe@nvidia.com> Signed-off-by: Chuck Ketcham <cketcham@nvidia.com> Co-authored-by: Chuck Ketcham <cketcham@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
cketcham2333
added a commit
that referenced
this pull request
Aug 3, 2026
…er from an FPGA or a QPU kernel (#703) ## What Adds a shipped example (`docs/sphinx/examples/qec/realtime_decoding_demo/`) that compiles against the installed SDK — installed headers/libs only — and exercises the delivered `decoding_server` end-to-end, plus its docs/CI registration and an FPGA RX-ring sizing fix in the unittests driver. The example source is the refactored `surface_code-1.cpp` from #685 (new syndrome format, `decoder_context_from_memory_circuit`, seeded runs), copied with three adaptations: the header comment, the usage line, and a true-width `--save_syndrome` capture so the FPGA playback tool replays exact per-round measurement counts. ## The example One source, built two ways: a generator (decoder config + FPGA syndrome file) and the lowered `-cqr` kernel (`-frealtime-lowering -DQEC_APP_CQR`). One script, two syndrome sources, three decoders (`pymatching`, `multi_error_lut`, `nv-qldpc-decoder`): - `--source qpu-kernel` — the lowered kernel streams syndromes to the server over UDP (no hardware; `QEC_DECODING_SERVER_PORT` routes the device_calls). - `--source fpga` — the delivered playback tool streams syndromes from a real FPGA over RoCE (`cpu_roce` / `gpu_roce`). Runs are seeded (`--seed`, default 42), so results are reproducible. Pass/fail matches the existing surface-code tests: decoder-failure greps, the completion proof, a `num_shots/50` residual logical-error ceiling, an in-process dispatch count of 0 (the two-process test's proof the decode stayed in the server), and a `num_shots*(num_rounds+3)` server dispatch floor. ## CI `lib_qec.yaml` builds the example against the installed SDK (enforcing the installed-headers-only rule) and runs a seeded 200-shot qpu-kernel pymatching decode over UDP loopback on the CPU job — a real decode, no hardware, and deterministic pass/fail. ## FPGA ring fix (`hsb_fpga_decoding_server_test.sh`) Fixes a #683 regression: the playback ring modulus (`--num-pages 512`) no longer matches the cpu_roce server's 64-slot RX ring, starving the test after ~10 shots. Both transports now run a ring equal to the HSB QP's 64-entry receive WQE depth (deeper rings drop frames), with a fail-fast preflight of DOCA's host-page alignment on 16K/64K-page hosts. --------- Signed-off-by: Ben Howe <bhowe@nvidia.com> Signed-off-by: Chuck Ketcham <cketcham@nvidia.com> Co-authored-by: Ben Howe <bhowe@nvidia.com> Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Follow-up fixes on top of the HSB/FPGA
gpu_rocedecoding-server path, found during end-to-end GB200 validation of RelayBP /nv-qldpc-decoderwith the FPGA emulator.Validated after rebase with an 85-shot / 425-frame
gpu_roceRelayBP run against the FPGA emulator, using 512 playback windows. The emulator processed all windows successfully with 425 responses, 0 send errors, and 0 timeouts.Changes:
cuda_device_id, and reconcile it with the transitionalHOLOLINK_GPU_IDsetting.gpu_roceDOCA ring allocation against host page-size alignment, and make the HSB test harness choose an aligned server ring depth while keeping playback window capacity explicit.decoder_rpc_dispatch.cusupport intodecoding_serverwhen present; public builds remain unaffected.decoding_server.gpu_rocelink check verify symbol/link availability without constructing a transceiver or selecting GPU 0.gpu_rocering pages and FPGA/emulator playback window pages.Note: I left this local intentionally because the
HOLOLINK_GPU_IDpath is transitional and PR692 removes it. Sharing the env parser now would add churn and increase conflicts for code that is about to disappear.Runtime / performance impact
na
Self-review checklist
Please confirm each item before requesting review. Check
[x]or strikethrough and explain.
Before requesting review
Scope and size
(if so, an issue has been raised).
Tests
just when it is missing.
EXPECT_*/assertchecks areinsufficient for algorithmic correctness.
Documentation
tracked.
Code style
snake_casevscamelCase) forthe area being modified.
Dependencies
OSRB tickets filed.