gfx: GPU video interop layer (from #2109) - #2121
Open
jcelerier wants to merge 74 commits into
Open
Conversation
jcelerier
added a commit
that referenced
this pull request
Jul 17, 2026
FIX-ENC: - NV12.hpp: UV plane encodes into an R8 target (16-byte-tight rows), was mis-sized producing garbage chroma on readback - YUVPlanar.hpp: p420_10 -> RGBA8-packed 16-bit LE on the Qt<6.10 GL path Tests: test_unit_video_pixel_format (pixfmt plane math + avutil mapping); test_integration_encoder_matrix (every encoder on a real QRhi backend, byte-checked); test_regression_offscreen_teardown (full-app offscreen render + /stop + /exit exits 0 — guards 1228382 #2119 + 32ad555 #2121 teardown UAFs); VideoDecoderTester + video-decoder-sweep.sh; EncoderTester ctest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
jcelerier
added a commit
that referenced
this pull request
Jul 17, 2026
…indow Adding/removing a node (or opening the inspector texture preview) during playback triggers a full gfx-graph rebuild while a real window + vsync clock is live. Three regressions, one family: CRASH (deterministic, ASAN 2/2): a stray DeferredDelete reaching the shared_ptr-owned score::gfx::Window ran 'delete this' on a make_shared interior pointer (invalid free) and, by destroying Window::state, dropped the shared RenderState to RenderList-only ownership so the next Graph::createAllRenderLists freed it under the in-flight rebuild -> use-after- free in ScreenNode::updateGraphicsAPI. Fix: Window::event swallows DeferredDelete (the window is never owned by the QObject tree; the shared_ptr deleter destroys it at the right time). Also make ScreenNode::createOutput idempotent — a rebuild that re-enters it while the window's swapchain is still pending must not make_shared-replace the in-flight window (deliberate api/ sample recreation routes through destroyOutput() first). FREEZE-until-window-move: switching manual -> vsync mode only set the vsync callback; nothing kicked a first frame, so the window's self-perpetuating requestUpdate() chain stayed dead until a platform expose. ScreenNode:: setVSyncCallback now kicks a queued requestUpdate() on the null->non-null transition. UAF: Window::render() now executes a copy of onUpdate — a rebuild driven from inside it can tear down the very std::function being executed. Verified on DISPLAY=:0/xcb under ASAN with the DIAG repro (add/remove 2nd output mid-play): 0 ASAN errors across all phases (was 2/2 crash at add); 461 fps samples, no gap >500ms, 119.6 fps held across the manual->vsync transition (was: frozen). fps-drop symptom (2nd preview output flips whole context to manual rate) is left as a follow-up — it needs the per-output clock coexistence refactor (canDoVSync excluding manual-only preview outputs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
jcelerier
added a commit
that referenced
this pull request
Jul 17, 2026
FIX-ENC: - NV12.hpp: UV plane encodes into an R8 target (16-byte-tight rows), was mis-sized producing garbage chroma on readback - YUVPlanar.hpp: p420_10 -> RGBA8-packed 16-bit LE on the Qt<6.10 GL path Tests: test_unit_video_pixel_format (pixfmt plane math + avutil mapping); test_integration_encoder_matrix (every encoder on a real QRhi backend, byte-checked); test_regression_offscreen_teardown (full-app offscreen render + /stop + /exit exits 0 — guards 1228382 #2119 + 32ad555 #2121 teardown UAFs); VideoDecoderTester + video-decoder-sweep.sh; EncoderTester ctest. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
jcelerier
added a commit
that referenced
this pull request
Jul 17, 2026
…indow Adding/removing a node (or opening the inspector texture preview) during playback triggers a full gfx-graph rebuild while a real window + vsync clock is live. Three regressions, one family: CRASH (deterministic, ASAN 2/2): a stray DeferredDelete reaching the shared_ptr-owned score::gfx::Window ran 'delete this' on a make_shared interior pointer (invalid free) and, by destroying Window::state, dropped the shared RenderState to RenderList-only ownership so the next Graph::createAllRenderLists freed it under the in-flight rebuild -> use-after- free in ScreenNode::updateGraphicsAPI. Fix: Window::event swallows DeferredDelete (the window is never owned by the QObject tree; the shared_ptr deleter destroys it at the right time). Also make ScreenNode::createOutput idempotent — a rebuild that re-enters it while the window's swapchain is still pending must not make_shared-replace the in-flight window (deliberate api/ sample recreation routes through destroyOutput() first). FREEZE-until-window-move: switching manual -> vsync mode only set the vsync callback; nothing kicked a first frame, so the window's self-perpetuating requestUpdate() chain stayed dead until a platform expose. ScreenNode:: setVSyncCallback now kicks a queued requestUpdate() on the null->non-null transition. UAF: Window::render() now executes a copy of onUpdate — a rebuild driven from inside it can tear down the very std::function being executed. Verified on DISPLAY=:0/xcb under ASAN with the DIAG repro (add/remove 2nd output mid-play): 0 ASAN errors across all phases (was 2/2 crash at add); 461 fps samples, no gap >500ms, 119.6 fps held across the manual->vsync transition (was: frozen). fps-drop symptom (2nd preview output flips whole context to manual rate) is left as a follow-up — it needs the per-output clock coexistence refactor (canDoVSync excluding manual-only preview outputs). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
jcelerier
added a commit
that referenced
this pull request
Jul 17, 2026
…ss test Two shipped video-decoder bugs found by the P3-video decode-correctness matrix (both verified under ASAN on llvmpipe): - RGB24/BGR24 rendered ~15 dB too dark: RGB24Decoder's packed R8 data texture was flagged QRhiTexture::sRGB, so the sampler applied the sRGB EOTF to raw bytes on texelFetch. Drop the flag (data texture, not colour). 15 -> 99 dB. - RGBA64LE/BGRA64LE rendered pure black: routed to a half-float RGBA16F texture, but the data is 16-bit UNORM integer -> reinterpreted as halfs -> NaN. QRhi has no 4-channel 16-bit UNORM, so add RGBA64Decoder (R16 x w*4 packed + texelFetch reassembly, mirroring RGB48Decoder). black -> 51 dB. (RGBAF16LE, a genuine half-float format, still uses RGBA16F — unchanged.) Test: test_video_decode_correctness — every software-decodable pixel format encoded as a known pattern (ffprobe-verified pix_fmt), decoded through VideoDecoderTester --expect/--psnr, RGBA readback asserted vs ffmpeg's own decode (per-format PSNR bound) + truncated/garbage fuzz. 43 PASS / 0 XFAIL. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
jcelerier
force-pushed
the
plane/interop
branch
2 times, most recently
from
July 18, 2026 13:37
4498ad7 to
71731df
Compare
jcelerier
force-pushed
the
pland/scene
branch
2 times, most recently
from
July 19, 2026 21:59
bd66e34 to
0da070a
Compare
jcelerier
force-pushed
the
plane/interop
branch
2 times, most recently
from
July 20, 2026 23:09
76333c0 to
ca08bb3
Compare
jcelerier
force-pushed
the
pland/scene
branch
2 times, most recently
from
July 21, 2026 04:55
d3bcb48 to
864d930
Compare
The external-OES rung imported the frame correctly and then sampled nothing.
DmaBufImportCapture created its renderer-facing textures with
QRhiTexture::Flag{}, and DMACaptureInputNode swaps those in over the ones the
decoder allocated -- destroying NV12ExternalOESDecoder's texture, which was the
only one carrying QRhiTexture::ExternalOES.
That flag is not decoration. QGles2Texture::prepareCreate() derives the GL bind
target from the flags alone, and createFrom() routes through it as well, so
without ExternalOES the target is GL_TEXTURE_2D. bindExternal() has already
bound the texture object to GL_TEXTURE_EXTERNAL_OES, and a GL ES texture
object's target is fixed at first bind -- so QRhi's bind at draw time is
invalid and the sampler reads undefined data.
Nothing in the frame-rate figures could show this: the import, the pool
recycling and the publish counter were all working, which is why the rung
measured 57 fps while its output was never pixel-verified in score. grabTo
returns black on that board for every rung (including CPU), so the one check
that would have caught it was unavailable.
Found by auditing the buffer path for copies rather than by testing, and
confirmed against the Qt sources at qrhigles2.cpp:6140 and :6299. Still
unverified on hardware for the same reason it was missed.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
A multi-sensor rig produces one frame per sensor per capture, and the renderer has to bind frames from the same capture or a 360 stitch shows one eye ahead of the other. The capture path could not promise that. Each node owns a private ring and latches whatever was published last, independently: if stream A published this tick and stream B did not, A binds a new frame and B silently re-uses its previous texture, with nothing reporting it. CaptureSyncGroup closes both halves. The producer publishes the whole set at once, so "A published, B did not" stops being representable rather than being detected afterwards. The consumer pins one generation per render pass, because RenderList updates members sequentially and a publish landing between two members' updates would otherwise split them across captures -- a window of microseconds, which is how a one-frame skew ships: rare, unreproducible, and visible only in the stitch. Incomplete captures hold the previous complete one instead of compositing a mismatch, and are counted, along with lapping and the worst intra-capture skew. A rig that is quietly tearing should read as a number rather than as an artefact someone eventually notices in a recording. Slot lifetime moves here too. With a group driving slot choice the renderer no longer consumes each strategy's own publisher, and leaving both to arbitrate would let a strategy hand back a slot the group had just pinned. The retirement rule is BorrowedSlotTracker's, generalised to N members: a capture is queued at the acquisition that displaces it and freed retireDepth acquisitions later. Vendor-neutral on purpose -- it knows nothing about Argus and serves any backend that can hand over N slots belonging to one capture. Not yet wired to one; this is the mechanism plus its tests (14 cases, 56 assertions, app-free). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
Wires CaptureSyncGroup into the capture path. A backend whose device drives several sensors from one capture returns the group they share plus this stream's index; the renderer then binds the slot the group chooses, pinned per render pass by RenderList::frame, instead of whatever this stream published last. Slot selection is a capability, not an assumption. supportsSlotSelection() is asked before a group is attached, and a rung that cannot bind a caller-chosen slot stays on the unsynchronised path with a warning rather than being handed a slot index it would ignore -- binding nothing every frame would look like a dead camera, and binding the wrong frame is worse than an unsynchronised one because it looks correct. In grouped mode the producer publishes the set to the group and must not call ingestFrame(): that path exchanges into the strategy's own publisher and hands the displaced slot straight back, which could be the slot the group just pinned. One owner for slot lifetime, and with a group it is the group. The per-plane SRB rebind is extracted to rebindPlanes() and called from both paths. Leaving it inside the unsynchronised branch would have made the grouped path skip it -- a no-op for single-texture strategies, silently wrong for the double-buffered ones. A backend that returns no group leaves the single-stream path unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
Every other output in this tree ends with a copy. ScreenNode renders the graph into an offscreen target and then blits it into the swapchain (ScaledRenderer::finishFrame); on GLES that pass does not even flip Y, so it is a literal full-frame copy, measured at ~1.2 ms at 3552x3556 and ~0.35 ms at 2560x1440 -- about half a passthrough's entire write traffic. KmsOutputNode has no swapchain, so that copy does not exist to be removed. N scanout buffers are allocated once as GBM bos with GBM_BO_USE_SCANOUT, exported as dma-bufs, imported back as EGLImages and wrapped as QRhi textures with createFrom; the same dma-buf becomes a KMS framebuffer. Each frame the graph renders into one of them and the node atomically flips to it. The modifier is what decides whether this is really zero-copy. KmsDevice already reports what each plane advertises, and that list is what the GBM allocation is constrained to: allocate something the plane cannot scan out and either the kernel refuses the framebuffer or a detiling copy appears where nobody is looking for it. A LINEAR result on a device advertising more is logged rather than accepted silently. Slot rotation via OutputNode::currentRenderTarget(), which exists for exactly this. All slots share one render-pass descriptor, so pipelines built against one stay valid across the rotation -- the reason the objection recorded against doing this for the PipeWire output does not apply: that was about a consumer handing back a different buffer every frame, and here the set is ours and fixed. Flip completions are tracked as an ordered queue rather than a per-slot flag. The kernel delivers events in flip order, so by the time a slot is reused its event has already been read; a per-slot flag would have stalled for an extra event every frame, which is a latency bug that looks like the display being slow. DRM master is required, and a compositor holds it -- the failure says so, and says that a lease would be needed, rather than just producing no output. Leases are not implemented in KmsDevice yet, so the "one output while the editor runs on another screen" case is still out of reach; the appliance case is not. GL only so far. Compiles; not yet run -- verification is writeback capture against vkms, which checks scanout in software and does not go through grabTo (black on the Orin NX, #125). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
KmsOutputNode never created its image input port, so the first thing to touch input[0] read off the end of an empty vector -- a segfault the moment anything tried to connect to it. Found by running it, which is the only way this one shows up: it compiles and links perfectly happily. KmsOutputTest builds a real graph (TexgenNode -> KmsOutputNode) and drives it, reporting the negotiated device/connector/mode/modifier and the flip intervals. The source pattern moves every frame on purpose: a static image would flip identically whether or not the graph was rendering anything, so the run would prove nothing. It uses a plain QGuiApplication rather than the score integration harness, which pulls in the full GUI application and aborts on missing skin resources -- a graph plus an output node needs neither. Verified so far on this workstation: enumeration of all three cards, and the degradation path when DRM master cannot be taken (Xorg holds it, and taking it needs CAP_SYS_ADMIN). The flip loop itself still needs a device we can master -- vkms, a bare VT, or the Jetson. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
The node collapsed every acquireMaster() failure into "a compositor most likely holds it". Against vkms -- a fresh virtual card with no compositor anywhere near it -- that message sends you looking for a client that does not exist. KmsDevice already distinguishes them and they need different fixes: EACCES/EPERM is a privilege/seat problem (CAP_SYS_ADMIN, or run from the active VT), EBUSY is another client genuinely holding master, which would need a DRM lease. Surface lastError() and spell out both. The vkms run this came from reports EACCES, so nothing holds master there and root is enough. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
createRenderState() reads score's Gfx settings through score::AppContext() for one value -- the MSAA sample count -- which made a headless scanout output depend on a full score GUI application being constructed. A harness driving just a graph and this node aborted in ScenarioApplicationPlugin::initialize() reaching for an inspector interface list it had no reason to need. Multisampling is also wrong here on its own terms: resolving a multisampled colour buffer into the scanout buffer is precisely the full-frame copy this output exists to remove. So the node builds its QRhi directly with samples=1. The harness goes back to a plain QGuiApplication accordingly. This does not yet make the node work headless -- Qt's platform plugins either provide no GL (offscreen/minimal) or provide it by taking DRM master themselves (eglfs_kms), which is the resource the node needs. See PLAN-DRM-OUTPUT.md; the fix is for the node to create its own EGL context on the GBM device it already builds. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
The Vulkan import rung is refused whenever the driver reports DRIVER_ID_NVIDIA_PROPRIETARY. Tegra reports that same id, so the gate also fires on a Jetson -- where the raw-Bayer camera path for the 360 rig depends on this import working. Nothing in the tree could tell us whether the gate is justified there or merely inherited from the desktop driver. Measured on the desktop with this probe: a GBM-exported dma-buf imports byte-exact 10/10, while a V4L2/vb2-exported one intermittently reads zeros on scattered pages (32 B to ~1.2 MB per frame). So the driver imports its own buffers correctly and fails on foreign exporters, which is what the gate is really describing. Deliberately standalone -- no Qt, no score headers, one TU -- so it cross-compiles against a Yocto sysroot with only libvulkan and libgbm, and runs on a board that has neither a display nor a compiler. The GBM source is the default because it allocates its own buffer; the V4L2 source needs bypass_mode=0, which stops Argus until it is set back, so it is opt-in. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
The rung was refused whenever the driver reported NVIDIA_PROPRIETARY, which is both too broad and aimed at the wrong thing. Measured with tests/DmaBufImportProbe.cpp on an RTX 4090 (595.84) and an Orin NX (540.4.0): exporter desktop Tegra GBM 10/10 exact 10/10 exact NvBufSurface -- 10/10 exact V4L2 / vb2 intermittent 0/10, ~20.6 MB of 25 MB read as zero Both drivers import their own allocators' buffers exactly and misread ones a foreign device exported, so the axis is the exporter, not the platform and not the driver id. An arch check would be wrong twice over: Tegra behaves like the desktop here, and a Grace-class machine is ARM64 but desktop-like. The foreign-queue ownership acquire is not the answer either -- 20/30 vs 21/30 corrupted over fresh processes with and without it. Recorded in the header, since the previous note claimed the omission was sound for linear modifiers. Default stays ForeignDevice so a producer that says nothing keeps today's behaviour. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
Closing the V4L2 fd while the driver still holds attachments to dma-bufs we are about to drop deadlocks vivid in v4l2_release -- an unkillable D-state process and a device wedged until reboot. REQBUFS(0) after STREAMOFF is the correct teardown and avoids it. vivid cannot serve as a stand-in for a real capture device on this path at all: dma-contig refuses QBUF with EINVAL for every length convention with pitch and size matching exactly, and vmalloc deadlocks as above. Said so in the header so nobody spends the afternoon I just did. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
…ived one
The single-plane path built its PlaneInfo with pitch = width * bytesPerPixel and
handed that to the importer, while the producer's real bytesperline was only
range-checked and then discarded. A padded producer therefore passed validation
and imported with rows 64 bytes too short, shearing the frame progressively
instead of failing. The multi-plane path already refuses a padded slot outright
because it cannot derive plane offsets; only this case was wrong.
This is on the path the 360 rig needs: the Tegra VI reports bytesperline 7168
for a 3552-wide 16-bit raster whose packed width is 7104.
Measured with the probe's new --tight-pitch mode, which imports with width*bpp
while the buffer's real stride is padded -- same buffer, same driver, same
import call, only the pitch differs. GBM pads a 1000x256 ARGB buffer to 4096:
real stride 4096 5/5 byte-exact
tight 4000 0/5, ~743 KB wrong per frame
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
The chroma plane of a 10/16-bit semi-planar frame was imported with 0x36315247
('GR16'), which is not a fourcc any kernel defines. DRM_FORMAT_GR1616 is
fourcc_code('G','R','3','2') = 0x32335247 (drm_fourcc.h:158), so every P010/P016
dma-buf import of plane 1 was asking for a format nothing recognises. The
neighbouring comment called the value "Mesa-defined", asserting a check that had
not happened.
It was wrong in two places, because the constants were duplicated: once as
function-local literals in the capture table and once hardcoded in DRMPrime.
They now live in DrmFourcc.hpp -- which already declared itself the one
authoritative table -- spelled through the shared builder so the characters are
readable, and pinned with static_asserts. The sibling constants R8, GR88 and R16
were checked against the kernel header and are correct.
The static_asserts are the test: the header compiles with the corrected values
and fails to compile when asserted against the previous literal.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
score_static_plugins.hpp registers plug-ins behind
`#if __has_include(<score_addon_foo.hpp>)`. avnd_score_plugin_finalize wrote that
header into CMAKE_BINARY_DIR, which is on every target's include path, so the
guard evaluated true in translation units that never link the addon -- they then
emitted a reference to its constructor and failed at link.
Adding score-addon-synthimi to the tree broke 53 targets this way, all of them
small unit tests. score-addon-videoio escapes only because its header is a
source file rather than generated, so its guard is correctly false elsewhere.
The generated files now live in ${CMAKE_BINARY_DIR}/score_addons/<target>/, and
that directory is a PUBLIC include directory of the addon target: the addon's own
generated .cpp and anything linking it can find the header, nothing else can, and
the __has_include guard finally means what it says.
Measured: 53 failing targets before, 2 after -- and those two fail on an
unrelated libremidi/pipewire symbol.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
The IMX676 on the Orin NX rig delivers V4L2_PIX_FMT_SRGGB10 -- the 'RG10' both /dev/video0 and /dev/video1 enumerate. The vocabulary had Bayer at 8, 12 and 16 bits but nothing at 10, and V4L2PixelFormat mapped SRGGB8 and SRGGB16 only, so that fourcc resolved to Unknown and the camera could not be opened at all. Add the four orders V4L2 defines at this depth rather than only the one this sensor needs: the CFA order decides how a demosaic reads the mosaic, so a GRBG10 sensor silently resolving to Unknown is the same class of bug the explicit 8-bit orders already exist to prevent. They occupy a 16-bit little-endian container, so they are two bytes per sample and map to QRhiTexture::R16 -- the mosaic can be uploaded as one single-channel texture and demosaiced in a shader, with no CPU pass over a 25 MB frame. V4L2 defines the ten significant bits as right-aligned. The Tegra VI left-aligns them instead, which is a sixty-fourfold scale rather than a different layout, so it belongs to the demosaic rather than to a separate enumerator here; mapping SRGGB10 onto BayerRGGB16 would have made this one rig correct and every conforming driver sixty-four times too dark. Unbridged on the AV side, following BayerRG8/BayerRG12: FFmpeg spells Bayer at 8 and 16 bits only, and borrowing the 16-bit twin would lose the significant-bit count on the way back. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
A Bayer sensor delivers one sample per pixel and no decoder unpacked that, so every CFA format resolved to a null decoder however well the capture side negotiated it. The 360 rig's two IMX676 are exactly this: no ISP touches the frames, so the mosaic arrives raw. Bilinear reconstruction, one pass, sampling the mosaic as a single-channel texture with nearest filtering -- linear would blend neighbouring colour sites together before the demosaic can separate them, the same hazard the byte-reassembling decoders already avoid. The CFA order is a parameter rather than a baked constant. A capture that crops to an odd origin flips the phase, and a wrong phase does not look broken: it looks like a colour cast, which is easy to mistake for white balance and chase in the wrong place. `sampleScale` covers a mosaic that does not fill its container: ten or twelve bits right-aligned in a 16-bit lane normalise to a fraction of full scale and need the same rescale Mono10 and Mono12 already carry. Black level, white balance and lens shading are deliberately not here. They are per-sensor corrections rather than part of turning a mosaic into RGB, and the frames will look milky until something applies them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
CaptureSyncGroup::publish() writes every member of a set and bumps the generation in one call, which fits a device that hands over all its sensors from one capture -- Argus' multi-sensor session, an SDI card with several inputs. Two CSI cameras are not that: separate file descriptors, separate threads, separate arrival times, and two threads calling publish() would interleave halves of different captures into one set. Nothing bridged that, so syncGroup() had no implementer and a rig of discrete cameras could not be frame-locked at all. Pairing is by arrival rather than by timestamp. The rig this exists for is frequency-locked, so consecutive arrivals correspond, and its eyes sit a constant offset apart that no matching removes. Timestamps ride along so the group reports the skew that happened rather than the one the hardware promised. A member that outruns its partners displaces its own previous offer and gets that slot straight back: holding it would starve the driver of buffers, which presents as a stall rather than as the drop it is. Only complete rows are published. Publishing partial ones does not keep the live members going, which is what it looks like it would do -- take() serves only the newest complete set, so a partial row advances the generation while the newest complete one stands still, and once that gap reaches the ring depth the last good set is condemned as lapped and every member goes dark. A stalled rig therefore holds, and shows up as displacedFrames() climbing at the frame rate. The test for this asserted the opposite until it was run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
The binding sat between the zero-copy ladder and the CPU fallback, guarded on the strategy the ladder had chosen. When every zero-copy rung failed the ladder left that null, so the whole block was skipped and the fallback installed its strategy afterwards -- the stream joined no group, and because the "cannot bind a chosen slot" warning lived inside the same guard, it said nothing either. A rig whose members fell back to CPU therefore ran unsynchronised in silence, which is the failure this code was written to make impossible: it looks exactly like a working rig until two members disagree about what they are showing. Observed on a two-device rig where the dma-buf rung could not sample the driver's fourcc: no warning, and the member reporting a generation that climbed with its own frame counter rather than the group's. With the binding moved after the fallback the same run prints "rung V4L2-CPU cannot bind a chosen slot". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
A dma-buf imported on Tegra cannot be sampled as GL_TEXTURE_2D. Measured on
an Orin NX against a V4L2 export of the IMX676, with TegraDmaBufProbe added
here so it is re-runnable rather than a story:
R16 IMPORTED 3552x3556 TEXTURE_2D=err TEXTURE_EXTERNAL_OES=OK
R8 IMPORTED 7168x3556 TEXTURE_2D=err TEXTURE_EXTERNAL_OES=OK
ABGR IMPORTED 1776x3556 TEXTURE_2D=err TEXTURE_EXTERNAL_OES=OK
eglCreateImage accepts all three -- R16, R8, RG88 and GR32 are all in this
driver's importable list -- so the refusal is the bind target alone, which is
what the per-plane branch reports as "cannot sample fourcc ... as a 2D
texture" before declining the whole rung. The consequence was capture falling
back to staging 25 MB per frame out of uncached V4L2 pages, which does not
hold 30 fps at 3552x3556.
Two facts the probe settled, both load-bearing for the shader:
- the sample arrives in .r. An R16 external image reads back (v,0,0,1).
- the GPU sees what the CPU wrote: 1024 of 1024 sampled points matched the
CPU's view of the same pages, mean absolute error 0.50, which is the
8-bit quantisation of a 16-bit value and nothing else. Tegra does not
have the coherency gap that makes the desktop read zeros out of a
foreign dma-buf, so no flush and no gate are needed here. (The existing
NVIDIA gate is Vulkan-only and never applied to this path.)
texelFetch does not exist for samplerExternalOES, so the neighbourhood is
gathered by normalised coordinate with mat.texSz standing in for textureSize.
NEAREST is mandatory rather than preferred: LINEAR blends adjacent colour
sites before the demosaic can separate them.
toDrmFourcc had no Bayer row at all, so a mosaic -- byte-identical to the
greyscale of the same depth -- resolved to 0 and could not name itself to the
importer. Added in the to-DRM direction only: coming back, R8/R16 stay
Mono8/Mono16, since the fourcc cannot say which of the five enumerators
sharing it was meant.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
DmaBufSlotDesc documents planeCount == 0 as "derive it", and says it is the right convention for a V4L2-style producer that hands out one buffer and fills nothing else in. The external branch then passed that zero straight to importExternal, whose first guard is planeCount == 0, so the import was refused before EGL was ever called -- for exactly the producer class the external path exists to serve. On the Orin NX this read as "external EGL import refused slot 0 fourcc 20363152" and dropped capture to the CPU rung, which the same run measured at 91-104 ms of CPU per frame and 14 fps at 3552x3556. The refusal was not EGL's. TegraDmaBufProbe imports that fourcc from that device successfully, and gains an A/B here showing the modifier attribute is not involved either: an explicit DRM_FORMAT_MOD_LINEAR and an omitted modifier are both accepted, which is what ruled that hypothesis out before it was acted on. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
simple_texture_input_device carries one root node with one texture parameter,
and its make_child() returns {}, so a rig of N sensors could only be N
separate devices that a user has to keep configured consistently by hand --
matching resolutions, matching rates, and a member index typed twice.
This adds the multi-stream form: a root that holds no parameter of its own
and one child per stream, each child carrying the same texture parameter the
single-stream node has, bound to its own score::gfx::Node. A rig is then one
device addressed rig:/cam0, rig:/cam1.
Streams are added up front rather than created on demand, because each one
has to be handed the gfx node it renders; make_child() stays refused, which
is also what keeps a stray OSC address from conjuring a stream with no node
behind it.
This is the shape DMACaptureBackend::SyncMembership was written for -- "a
backend whose device drives several sensors from one capture returns the group
they all share, plus this stream's index in it" -- and the same structure the
per-sensor black level and white balance controls will hang off.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
The decoder has to be chosen before the ladder runs, because the strategy config points at its textures. A backend that asked for the whole-frame external image therefore already holds a decoder only that rung can feed, and when the rung declined, that decoder sampled a texture nothing ever uploaded to: a black frame from a path reporting itself engaged, on every host-staged fallback. The node now asks the backend whether its decoder depends on that rung, and if the ladder fell through to CPU staging, drops the request, remakes the decoder and repoints the strategy config at the new textures. An unnecessary copy is recoverable; a silently black fallback is not. Also here, two things that belong with it: deviceToJson() no longer takes the application down. A device tree can hold parameters the preset serializer cannot read -- a gfx device's texture parameters are the case that found this -- and it signals that by throwing, which nothing caught. Asking a graphics device for its tree from a script killed the process. It now reports which device and why, and returns nothing. X11Shot.cpp: the Jetson image ships no screenshot tool at all (xwd, import, scrot, xfce4-screenshooter, gnome-screenshot, ffmpeg all absent), so what score put on screen could not be checked -- and a rendering failure was inferred from a log line for hours as a direct result. XGetImage to a binary PPM, reading the visual's channel masks rather than assuming a byte order. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
The dlopen'd libv4l2 entry points lived as a private class inside CameraDevice.v4l2.cpp. The control tree needs the same three symbols, and a second copy of a dlopen singleton is how two copies drift apart -- the same argument that moved the V4L2 fourcc table into interop/V4L2PixelFormat, which these two files already share. Moved as-is except for one change: the constructor asserted every symbol resolved, which is a crash on a machine that has no libv4l2 and an assert away from being no check at all in release. It now reports `available()` so each caller can decide. Capture depends on the library's format emulation and declines without it; controls do not, and fall back to the raw syscall. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
Two unrelated things become settings under a video device: what the driver publishes -- discovered at runtime and different on every camera -- and what score itself offers, such as the Video process's scale mode or the demosaic's own corrections. They share no vocabulary, so this takes what they do have in common, a name and a type and a domain and something to do on write, and builds the nodes from that. Deliberately free of V4L2 so the score-side group is not obliged to describe itself as a fake driver control. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
A raw sensor frame is not a picture. The demosaic turns a mosaic into RGB and stops, deliberately, because everything after it is per-sensor -- the pedestal, the gains that make grey grey under this light, the curve that makes linear samples look right on a display. Until now there was nowhere to put any of it, so an IMX676 rendered green, dark and uncorrectable. Every correction is the identity by default, including the linear transfer curve. That curve is why a raw frame looks dark, and defaulting it to sRGB would have been the bigger improvement and the wrong call: it would silently restyle every existing project. It is offered, not imposed. The corrections live in the material block rather than baked into the shader, so moving a slider does not rebuild a pipeline. The block is a superset of VideoMaterialUBO with identical leading fields: a decoder whose shader declares only the short block reads the same buffer correctly, while extending the shared struct would have forced every renderer that allocates it -- both video paths included -- to grow in lockstep or bind one too small for its own shader. Both demosaicers share one copy of the maths, because a correction that differed between the host-staged and the external-image path would show up as the picture changing when the capture rung changed, which reads as a capture bug rather than a shader one. Scale mode arrives the same way: `mat.scale` already multiplies the quad in the vertex shader and the capture node simply never set it, so the frame was always drawn 1:1 regardless of the viewport. Publication is a generation counter, not a lock-free struct: the renderer pays one relaxed load per frame and only takes the lock when something moved. Reading the fields individually as atomics would let it see half of one setting and half of the next -- a visible colour flash while dragging two sliders. CaptureAdjustTest covers the maths against a CPU reference that is written as the specification the GLSL mirrors, the UBO offsets the shader reads through, the ordering that makes black level meaningful before gain, degenerate settings that must not produce NaN, and the slot under a concurrent writer. 944 assertions. It caught one real defect while being written: an unrecognised scale-mode name silently reset the fit to Original instead of leaving it alone. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
CaptureAdjustTest covers the reference and the UBO packing, which is everything except the part that draws. The two implementations are written to be the same maths, and "written to be" is the sort of claim that rots quietly: a divergence would surface as a colour shift on a camera, months later, and look like a capture bug. This runs the actual adjustCapture GLSL over 160 colours for each of eight settings and compares every channel against adjustCaptureReference. Measured on a desktop GL context: identity is bit-exact, and the worst delta anywhere else is 1/255 -- 8-bit quantisation, not arithmetic. Two things it deliberately does not do. It builds its QRhi directly instead of calling createRenderState, which reaches for score::AppContext() and so needs a whole application booted -- resources, settings, audio backend -- to compare two implementations of a fragment shader; the first attempt did that and died in resource loading. And it skips, exit 0, when no RHI can be created: a headless box with no GL cannot say anything about a shader, and reporting that as a failure would train people to ignore it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
take() serves only the newest complete set, so a capture published while the render thread was busy can never be chosen. Its slots were still lent to the group, and nothing released them: the producer lost a buffer per skipped capture until it had none left to lend and the rig stalled. Release them at pin time. They need no retirement delay, unlike the capture that was bound -- no member ever sampled them. That reads the skipped captures back out of the ring, so the ring now has to be deep enough to still hold them. Sized to the width of the return mask rather than to render latency: a member cannot lend a slot it has not got back, and there are at most that many slots, so an unreturned capture is always still resident however far behind the renderer falls. What cannot be recovered is counted rather than guessed at. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
A backend offering a group cannot see the renderer decline it, and the renderer does decline: a rung that cannot bind a caller-chosen slot leaves the stream on the unsynchronised path. It has to know, because the two paths disagree about who owns a slot. Ungrouped, the strategy's publisher decides when one may go back to the device. Grouped, the group decides, and a backend still asking the publisher gives the device back the very frame the group has just bound -- silently, and only visible as a rare tear. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
The unit tests each cover one side of the handover, and the slot leak lived in between them: the correlator gave its offers back correctly, the group retired what it had bound correctly, and captures that fell between the two were held by neither. This drives the loop the V4L2 rig actually runs -- dequeue, offer, latch, requeue whatever comes back -- with the producers deliberately outrunning the renderer. Without the fix, member 0 runs out of buffers on the sixth pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_014rZgzE8JjWvHDtaVUhxpLE
The generic sweeps hand every tester one fixed producer, but 35 of the .fs testers carry an explicit `Wire:` clause naming the chain they expect, and not one of the 40 .cs testers declares an OUTPUTS block -- they emit geometry or images, so an Image sink can never see what they wrote. Three harnesses cover what that leaves untested: - ShaderSweepCSFGeometry: the raster sweep inverted. The producer varies over a fixed rasterizer (raw-raster-basic, which the corpus itself names as the consumer), skipping any .cs without a geometry RESOURCE so the image half stays with the image sweep. - ShaderSweepWired: one hand-written fixture per constructible `Wire:` clause. - ShaderSweepScene: the threedim producers the scene testers need, built through oscr::GfxNode -- SceneFlattener is score::gfx::ScenePreprocessorNode, which was always constructible; what it lacked was an ossia::scene_spec source, and a Crousti-wrapped halp producer is exactly that. The fixture gains what these need: addNode() plus node port accessors for engine/Crousti nodes, bufferIn/bufferOut (there was no way to address a Types::Buffer port at all, which is why no uniform_input tester had ever been driven), a CableType on wire() with a wireFeedback() helper -- Graph.cpp's no_delay_edges filter keeps only the Immediate kinds, so an Immediate self-edge makes the graph cyclic -- and render() now pumps the Message so Crousti nodes see their controls. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Cables csf-storage-rw.cs (a read_write storage RESOURCE, so a Types::Buffer output backed by a StorageBuffer) into each uniform_input consumer and asserts the graph builds, renders and reads back on every available backend. Covers all three consumers deliberately. isf-persistent-uniform-input and isf-multipass-uniform-input go through RenderedISFNode, which is what crashed; binding-uniform-input goes through SimpleRenderedISFNode, which never did, so a guard built only on it passes with or without the fix and guards nothing. Verified both directions: with the fix, 18 assertions pass on Vulkan and OpenGL; with the usage check disabled, both backends SIGSEGV. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The GPU video interop layer from #2109, on top of the scene rework (#2120). These commits are already granular and land cleanly on the scene state. With this PR, the score-plugin-gfx tree is byte-identical to #2109's, rebased onto current master.
Stacked on #2120; retargets to master as the stack merges.
Commits
Validation
ctest: 17/17.🤖 Generated with Claude Code