OpenNOW native · Qt
- Owned RTSPS. Mjolnir video. Bundle audio and input. Media Foundation plus tearing present. Raw Input thread for relative mouse. Client-side type-7 tuning. No captured host mouse-settings frame. Native CloudMatch forces 8-bit 4:2:0. Audio has no jitter buffer and no RED. Mic is advertised, then refused.
+ Owned RTSPS. Mjolnir video. Bundle audio and input. Media Foundation plus tearing present. Raw Input thread for relative mouse. Client-side type-7 tuning. No captured host mouse-settings frame. Native CloudMatch forces 8-bit 4:2:0. Audio has no jitter buffer and no RED. There is no microphone capture or send path.
Official GFN · this machine
@@ -257,7 +257,7 @@ Decode and present
Audio
Read the long form in audio.md.
Official concealment is a timestamped buffer plus RED. OpenNOW concealment is “drop the packet.” WASAPI underrun stops the client and raises preroll. That is a hole, not a fade.
- ANNOUNCE still writes rtcMicOnNativeBundle:1. Core then refuses NVST mic. There is no send path. Official mic this session was 20 ms, 2 ch, 16 kbps, RED 3.
+ ANNOUNCE still writes rtcMicOnNativeBundle:1 for compatibility, but the runtime has no microphone capture, encoder, queue or send path. Official mic this session was 20 ms, 2 ch, 16 kbps, RED 3.
diff --git a/docs/streamer-comparison/mouse-input.md b/docs/streamer-comparison/mouse-input.md
index 69281460e..db2ab3055 100644
--- a/docs/streamer-comparison/mouse-input.md
+++ b/docs/streamer-comparison/mouse-input.md
@@ -105,22 +105,22 @@ NVST wrap is `COMMAND_REMOTE_INPUT` `0x0206`. Absolute encode injects flags `0x0
1. **Host curve.** Official disables session accel and pins speed 10 on every focus change. The type 10 frame itself has not been captured. This is still the first thing to match if you want official look.
2. **Client curve.** OpenNOW can scale type 7 before send. Official logs imply raw counts.
3. **i16 split.** A large HID burst becomes several type-7 packets. Official may keep a wider delta.
-4. **Foreground filter.** OpenNOW requires the SDL HWND to be foreground. Overlay is special-cased. A real Qt steal drops all mouse. Official async thread is also focus-scoped, but it restarts on window state 19 and resends alt settings.
+4. **Foreground filter.** The embedded Windows capture binds Raw Input ownership to the Qt window and pauses capture while an overlay owns input. Official's async thread is also focus-scoped, but it restarts on window state 19 and resends alternate settings.
5. **Binary cursor mode.** Official has visible-unlocked, visible-warped, and hidden-locked. OpenNOW has hidden versus visible. Custom bitmap id 0 stays visible.
-6. **SDL still grabs** while Raw Input owns motion. Official may keep lock inside `RawInputController`.
+6. **Qt owns pointer state** while the embedded Raw Input controller owns relative motion. Official may keep lock inside `RawInputController`.
7. **Drain stalls.** Encode and SCTP share the NVST event thread. A hitch batches then bursts. The 300 ms unordered lifetime exists because ordered PR-SCTP already felt like delayed catch-up.
## Where things live
- Raw Input. `native/opennow-streamer/crates/opennow-streamer-platform/src/windows_raw_input.rs`
-- SDL grab and cursor. `native/opennow-streamer/crates/opennow-streamer-platform/src/output.rs`
+- Embedded capture ownership. `native/opennow-streamer/crates/opennow-streamer-platform/src/embedded_input.rs`
- Tuning and packet. `native/opennow-streamer/crates/opennow-streamer-core/src/lib.rs` `tune_relative_mouse`, `captured_input_packet`
- NVST encode and channels. `native/opennow-streamer/crates/opennow-streamer-transport/src/nvst_input.rs`
-- Launch env. `native/opennow-core/src/streamer.rs`
+- Qt input bridge. `opennow-qt/src/NativeStreamRuntime.cpp`
## Gotchas
-`owned_window_rejects_hit_testing` is the D3D child HWND used when video is embedded. Qt's real path presents on the SDL HWND, which is a popup that takes focus. That child is not what eats mouse.
+The Qt path has no SDL presenter HWND. The shell passes its own native window handle to the embedded capture controller and submits Qt/controller events directly through the FFI.
Absolute `MOUSE_MOVE_ABSOLUTE` reports never become type 5. Tablets and some virtual mice produce no relative motion.
diff --git a/locales/en.json b/locales/en.json
index ffa0c9499..0c0519282 100644
--- a/locales/en.json
+++ b/locales/en.json
@@ -2600,6 +2600,19 @@
"s_qt_runtime_229": "Fullscreen",
"s_qt_runtime_230": "Shows live stream statistics",
"s_qt_runtime_231": "Starts the next session with compact stats",
- "s_qt_runtime_232": "Stream controls are available once the session is live."
+ "s_qt_runtime_232": "Stream controls are available once the session is live.",
+ "s_qt_runtime_233": "The embedded media runtime could not start",
+ "s_qt_runtime_234": "The embedded media runtime rejected a command",
+ "s_qt_runtime_235": "Preparing the embedded native media runtime…",
+ "s_qt_runtime_236": "The core could not prepare the native stream context",
+ "s_qt_runtime_237": "This stream control is unavailable",
+ "s_qt_runtime_238": "The embedded media runtime returned an invalid handshake",
+ "s_qt_runtime_239": "Native-owned NVST media transport is active",
+ "s_qt_runtime_240": "Stream stopped",
+ "s_qt_runtime_241": "Native media runtime failed",
+ "s_qt_runtime_242": "Recording failed",
+ "s_qt_runtime_243": "The embedded media runtime stopped",
+ "s_qt_runtime_244": "The core returned an invalid embedded stream context",
+ "s_qt_runtime_245": "The embedded media runtime could not start the stream"
}
}
diff --git a/native/opennow-core/src/main.rs b/native/opennow-core/src/main.rs
index 142f69a85..cb7564b14 100644
--- a/native/opennow-core/src/main.rs
+++ b/native/opennow-core/src/main.rs
@@ -523,6 +523,13 @@ fn dispatch(method: &str, params: &Value, core: &AppCore) -> DispatchResult {
.map(|value| (value.clone(), Some(("streamer.changed", value))))
.map_err(streamer_error)
}
+ "streamer.prepare" => {
+ let settings = core.settings.lock().expect("settings poisoned").all();
+ core.streamer
+ .prepare_embedded(params, &settings)
+ .map(|value| (value, None))
+ .map_err(streamer_error)
+ }
"streamer.status.get" => Ok((core.streamer.status(), None)),
"streamer.stop" => core
.streamer
diff --git a/native/opennow-core/src/streamer.rs b/native/opennow-core/src/streamer.rs
index 52c84ff0b..1b73d1170 100644
--- a/native/opennow-core/src/streamer.rs
+++ b/native/opennow-core/src/streamer.rs
@@ -298,6 +298,29 @@ impl StreamerService {
ensure_codec_available(&detection["capabilities"], codec)
}
+ pub fn prepare_embedded(
+ &self,
+ params: &Value,
+ settings: &Value,
+ ) -> Result {
+ let session = params["session"]
+ .as_object()
+ .map(|_| params["session"].clone())
+ .ok_or_else(|| invalid("streamer.prepare requires a ready session"))?;
+ let status = session["status"].as_i64().unwrap_or_default();
+ if !matches!(status, 2 | 3) {
+ return Err(invalid(
+ "CloudMatch session is not ready for NVST media attachment",
+ ));
+ }
+ let mut context = streamer_context(session, settings);
+ context["surface"] = Value::Null;
+ Ok(json!({
+ "protocolVersion": STREAMER_PROTOCOL_VERSION,
+ "context": context
+ }))
+ }
+
pub fn start(&self, params: &Value, settings: &Value) -> Result {
self.reap_finished();
let mut worker = self.worker.lock().expect("streamer worker poisoned");
@@ -1632,6 +1655,30 @@ mod tests {
assert_eq!(context_resolution(&context), (1920, 1080));
}
+ #[test]
+ fn embedded_prepare_returns_nvst_context_without_a_native_surface() {
+ let service = StreamerService::new();
+ let prepared = service
+ .prepare_embedded(
+ &json!({
+ "session": {
+ "sessionId": "session-one",
+ "status": 2,
+ "signalingUrl": "wss://server.nvidiagrid.net/nvst/"
+ }
+ }),
+ &json!({"codec":"auto","transportMode":"webrtc","resolution":"1920x1080"}),
+ )
+ .expect("embedded context");
+
+ assert_eq!(prepared["protocolVersion"], STREAMER_PROTOCOL_VERSION);
+ assert_eq!(prepared["context"]["session"]["sessionId"], "session-one");
+ assert_eq!(prepared["context"]["settings"]["codec"], "H264");
+ assert_eq!(prepared["context"]["settings"]["transportMode"], "nvst");
+ assert_eq!(prepared["context"]["surface"], Value::Null);
+ assert!(service.worker.lock().expect("streamer worker").is_none());
+ }
+
#[test]
fn microphone_child_state_is_exposed_without_losing_the_message() {
let state = Arc::new(Mutex::new(Snapshot::default()));
diff --git a/native/opennow-streamer/Cargo.lock b/native/opennow-streamer/Cargo.lock
index bc251e068..4f9319df0 100644
--- a/native/opennow-streamer/Cargo.lock
+++ b/native/opennow-streamer/Cargo.lock
@@ -1634,15 +1634,23 @@ dependencies = [
"opennow-streamer-transport",
"rustls",
"serde_json",
- "str0m",
"tungstenite",
]
+[[package]]
+name = "opennow-streamer-ffi"
+version = "0.2.0"
+dependencies = [
+ "opennow-streamer-core",
+ "opennow-streamer-platform",
+ "opennow-streamer-protocol",
+ "serde_json",
+]
+
[[package]]
name = "opennow-streamer-platform"
version = "0.2.0"
dependencies = [
- "audiopus_sys",
"base64",
"image",
"libc",
diff --git a/native/opennow-streamer/Cargo.toml b/native/opennow-streamer/Cargo.toml
index dadfed52b..ddc26f4af 100644
--- a/native/opennow-streamer/Cargo.toml
+++ b/native/opennow-streamer/Cargo.toml
@@ -2,6 +2,7 @@
members = [
"crates/opennow-streamer",
"crates/opennow-streamer-core",
+ "crates/opennow-streamer-ffi",
"crates/opennow-streamer-platform",
"crates/opennow-streamer-platform-linux",
"crates/opennow-streamer-platform-macos",
diff --git a/native/opennow-streamer/README.md b/native/opennow-streamer/README.md
index 7da24da3c..f57c9bfbd 100644
--- a/native/opennow-streamer/README.md
+++ b/native/opennow-streamer/README.md
@@ -1,57 +1,43 @@
-# OpenNOW Native Streamer v2
+# OpenNOW native streamer
-This workspace is the clean replacement for the former GStreamer-based native streamer.
+This workspace implements the native GeForce NOW NVST runtime. It owns RTSPS negotiation, the dedicated Mjolnir SRTP video socket, the NVST ICE/DTLS/SCTP bundle used for audio, RTCP and input, bounded media queues, platform decode/audio output, source-stream Matroska recording and GPU frame publication. It does not implement the browser WebRTC offer/answer or trickle-ICE protocol and it has no microphone upstream path.
-The workspace owns the local process protocol, lifecycle state machine, GeForce NOW RTSPS/NVST negotiation, standards-based WebRTC transport, media sockets, media output path, microphone upstream and source-stream Matroska recording. The UI supplies a complete CloudMatch session context once; the streamer reserves its own bundle/Mjolnir sockets and performs OPTIONS, DESCRIBE, SETUP, ANNOUNCE, PLAY, keepalive, and TEARDOWN itself. OpenH264, Opus, SDL, and the Linux FFmpeg codec stack are compiled into the executable; GStreamer and external codec processes are not used.
-
-The executable retains the versioned JSON-lines process contract used by OpenNOW while the app shell is migrated away from Electron. It does not load or redistribute NVIDIA client libraries.
+The Qt application loads `opennow-streamer-ffi` as an in-process shared library. Qt supplies one complete CloudMatch session context; the embedded engine reserves its bundle and Mjolnir sockets and performs OPTIONS, DESCRIBE, SETUP, ANNOUNCE, PLAY, keepalive and TEARDOWN. The runtime does not load or redistribute NVIDIA client libraries.
## Crates
-- `opennow-streamer-protocol`: versioned local IPC DTOs.
-- `opennow-streamer-core`: session lifecycle and command routing.
-- `opennow-streamer-transport`: ICE, DTLS-SRTP, RTP/RTCP, and SCTP data channels.
-- `opennow-streamer-platform`: bounded media queues, OpenH264/Opus decode, SDL audio/video output, zero-reencode H.264/H.265/AV1 plus Opus Matroska recording, and shell-neutral native surface ownership.
-- `opennow-streamer-platform-windows`: strictly separated Media Foundation H.264/HEVC/AV1 hardware and system-software decode, D3D11 NV12/P010 presentation, and WASAPI PCM output for Windows x64 and ARM64.
-- `opennow-streamer-platform-macos`: VideoToolbox H.264/HEVC hardware decode, zero-copy IOSurface/Metal presentation, and CoreAudio output.
-- `opennow-streamer`: process entry point.
-
-## Checks
-
-```sh
-cargo test --manifest-path native/opennow-streamer/Cargo.toml
-cargo build --manifest-path native/opennow-streamer/Cargo.toml --release
-```
+- `opennow-streamer-protocol`: versioned local command and session DTOs.
+- `opennow-streamer-core`: NVST lifecycle, command routing, media feedback and recording.
+- `opennow-streamer-transport`: Mjolnir SRTP plus the NVST-required ICE/DTLS/SCTP, RTCP and input implementation.
+- `opennow-streamer-platform`: bounded media queues, decode/audio output, recording and GPU-frame publication.
+- `opennow-streamer-platform-{windows,macos,linux}`: platform decoders and native GPU texture producers.
+- `opennow-streamer-ffi`: bounded C ABI used in process by Qt.
+- `opennow-streamer`: a development JSON-lines host for the same engine; Qt packages do not include it.
-The legacy Electron build wrapper still checks protocol-version parity during migration. The Qt build compiles and bundles the same executable directly:
+## Qt GPU integration
-```sh
-npm --prefix opennow-stable run native:build
-```
+The Qt path does not create an SDL video window or a child streamer process. `NativeStreamRuntime` owns the embedded Rust handle, and `StreamVideoItem` drives the GPU-only FFI from Qt's render thread:
-## Platform integration
+1. Qt lends the current QRhi native graphics objects to the runtime.
+2. Platform decode publishes a native GPU frame into a one-frame, drop-stale mailbox.
+3. Qt acquires the latest opaque frame token and asks Rust to record conversion and synchronization into the same QRhi command buffer that will render it.
+4. Qt imports the returned RGBA8 native texture and samples it in the scene graph, so QML overlays compose above video normally.
+5. Qt releases the token after GPU use and shuts down the scene-graph binding on the render thread before QRhi teardown.
-The SDL presentation window is created hidden on the process main thread. Windows supports Media Foundation over either native D3D11 or a D3D12-backed D3D11-on-12 device and uses WASAPI for audio. Hardware mode enumerates only hardware MFTs. Software mode keeps bundled OpenH264/SDL for H.264 and can use registered D3D11-aware software MFTs for HEVC/AV1. Automatic mode performs the same class-preserving fallback after startup or unrecoverable device loss. `OPENNOW_NATIVE_VIDEO_BACKEND=software` forces that software path, while `d3d12` and `d3d11` select the corresponding hardware graphics path. Auto prefers D3D12 when its complete decode, presentation, and audio probe succeeds. Renderer surface rectangles are already physical pixels and are never scaled again by `deviceScaleFactor`. On Windows the Qt launch contract reparents the presenter HWND as an input-capable child of the Qt top-level window. Linux/X11 and Linux/Wayland use a separate compositor-managed SDL window aligned to the shell's stream region; that window owns native keyboard, mouse, relative-pointer, and cursor handling. macOS also uses a standalone, resizable SDL/AppKit window because AppKit cannot embed a view across processes. VideoToolbox decodes hardware-supported H.264, HEVC and AV1 into IOSurface-backed pixel buffers and Metal renders directly into that window's SDL-managed `CAMetalLayer`; the window owns keyboard, mouse, relative-pointer, and cursor handling. Fullscreen and other shell UI actions are emitted to Core/Qt; native presentation follows the surface geometry supplied by the shell and does not mutate fullscreen state. AV1 format configuration is derived from the first sequence-header keyframe and reconfigured atomically when that header changes. The macOS streamer is a regular LaunchServices application with its own menu bar, Dock identity, process coalition, and AppKit activation lifecycle. The executable runs `MainThreadHost` on its real main thread; FIFO IPC, NVST transport (including its WebRTC-compatible ICE/DTLS/SCTP control bundle), recording and codec work run on named workers. This is required by AppKit and is checked with `pthread_main_np()` on macOS.
+The FFI exposes no CPU image, encoded-frame callback, swap chain, window or Qt object. See `crates/opennow-streamer-ffi/README.md` for ownership and threading details.
-The app and native child select the same Linux window system. `OPENNOW_NATIVE_WINDOW_SYSTEM=x11|wayland` records that launch contract and `SDL_VIDEODRIVER` selects the matching SDL backend. On a native Wayland session the stream opens in its own resizable window; on X11 it follows the Qt stream surface. Linux decode supports H.264, HEVC/H.265, and AV1 through Vulkan Video, CUDA/NVDEC, or FFmpeg software decode; native VA-API and V4L2 provide additional H.264 paths. Vulkan presentation is preferred, with SDL NV12 presentation as an independent fallback so decoder acceleration remains active if the Vulkan window path fails.
+## Packaging
-Linux production builds statically include FFmpeg, its H.264/HEVC/AV1 decoders, Opus, OpenH264, SDL, and their C/C++ support runtimes. They do not require system FFmpeg, GStreamer, Opus, SDL, VA-API, X11, Wayland, or Vulkan-loader libraries. Only the Linux base ABI and the selected display/audio/GPU driver interfaces remain system responsibilities. Building the bundled stack requires a C/C++ toolchain, CMake, Make, NASM, pkg-config, and Git. Native VA-API remains opt-in with `OPENNOW_NATIVE_LINUX_VAAPI=1` because it would add a host `libva` runtime dependency. `OPENNOW_NATIVE_VIDEO_BACKEND=auto|vulkan|cuda|nvdec|vaapi|v4l2|ffmpeg|software` controls decoder selection; `auto` prefers CUDA/NVDEC on NVIDIA, then Vulkan Video, VA-API, V4L2, and bundled FFmpeg software.
+The Qt CMake build always compiles `opennow-streamer-ffi` with Cargo's release profile and links the resulting shared library to `opennow-qt`. CPack installs that library beside the Qt executable and `opennow-core`; there is no separate streamer executable, helper application or native video window in the Qt package. Linux packages enable the bundled FFmpeg fallback while optional GPU driver interfaces remain dynamically discovered.
-## macOS validation
+The standalone `opennow-streamer` binary remains a development host and is not evidence for the packaged Qt presentation path.
-Run the checks natively on each architecture rather than treating a Linux cross-check as macOS proof:
+## Checks
```sh
-rustup target add aarch64-apple-darwin x86_64-apple-darwin
-OPENNOW_NATIVE_STREAMER_TARGET="$(rustc -vV | sed -n 's/^host: //p')" \
-OPENNOW_NATIVE_STREAMER_PLATFORM_KEY="darwin-$(uname -m | sed 's/arm64/arm64/;s/x86_64/x64/')" \
-npm --prefix opennow-stable run native:build
+cargo fmt --manifest-path native/opennow-streamer/Cargo.toml --all -- --check
+cargo clippy --manifest-path native/opennow-streamer/Cargo.toml --workspace --all-targets -- -D warnings
+cargo test --manifest-path native/opennow-streamer/Cargo.toml --workspace
```
-The CI package matrix runs this build on both Apple Silicon and Intel macOS runners before producing the DMG and ZIP. Release validation must additionally inspect the app bundle, verify its signature, launch the packaged child executable, and confirm fallback behavior on real hardware.
-
-The generated helper lives at `bin/darwin-/OpenNOWStreamer.app/Contents/MacOS/opennow-streamer`. Both that nested application and its executable are ad-hoc signed for local/unsigned builds; the release signing pass signs nested code inside-out before sealing the outer OpenNOW application.
-
-## Runtime validation
-
-Unit tests synthesize and decode video and Opus frames, verify drop-oldest queue behavior, and exercise pause/stop lifecycle. Linux hardware validation generates H.264, HEVC, and AV1 keyframes and decodes each through Vulkan Video and CUDA/NVDEC. Release validation must additionally run an authorized live session on Windows, Linux/X11, Linux/Wayland, Intel macOS, and Apple Silicon to validate NVST interoperability, A/V timing, Windows child-HWND behavior, paired-window ordering on the other platforms, native relative input, and device-specific audio output.
+Release validation must additionally run authorized live sessions on Windows, Linux/X11, Linux/Wayland, Intel macOS and Apple Silicon to validate NVST interoperability, native GPU import, audio, input, recovery and device-loss behavior.
diff --git a/native/opennow-streamer/crates/opennow-streamer-core/Cargo.toml b/native/opennow-streamer/crates/opennow-streamer-core/Cargo.toml
index 5a8516a77..69e824dcf 100644
--- a/native/opennow-streamer/crates/opennow-streamer-core/Cargo.toml
+++ b/native/opennow-streamer/crates/opennow-streamer-core/Cargo.toml
@@ -14,6 +14,3 @@ opennow-streamer-transport = { path = "../opennow-streamer-transport" }
rustls.workspace = true
serde_json.workspace = true
tungstenite.workspace = true
-
-[dev-dependencies]
-str0m.workspace = true
diff --git a/native/opennow-streamer/crates/opennow-streamer-core/src/lib.rs b/native/opennow-streamer/crates/opennow-streamer-core/src/lib.rs
index 878636af8..e3298c84e 100644
--- a/native/opennow-streamer/crates/opennow-streamer-core/src/lib.rs
+++ b/native/opennow-streamer/crates/opennow-streamer-core/src/lib.rs
@@ -1,6 +1,6 @@
use std::collections::HashMap;
use std::net::UdpSocket;
-use std::sync::mpsc::{Receiver, Sender};
+use std::sync::mpsc::{Receiver, Sender, SyncSender, TrySendError};
use std::sync::{Arc, Mutex, MutexGuard, OnceLock};
use std::thread::{self, JoinHandle};
use std::time::{Duration, Instant};
@@ -8,21 +8,19 @@ use std::time::{Duration, Instant};
use base64::Engine as _;
use base64::engine::general_purpose::STANDARD as BASE64;
use opennow_streamer_platform::{
- CapturedInput, CapturedInputQueue, CapturedInputSample, EncodedFrame, EncodedMicrophoneQueue,
- MediaCodec, MediaColorQuality, MediaControl, MediaFeedback, MediaRuntime, MediaRuntimeControl,
- MediaSession, MediaSink, MediaStreamConfig, MediaVideoCodec, MicrophoneCapture, PushOutcome,
- RecordingSummary, StreamShortcutAction, StreamShortcutBindings, record_matroska,
+ CapturedInput, CapturedInputQueue, CapturedInputSample, EncodedFrame, MediaCodec,
+ MediaColorQuality, MediaControl, MediaFeedback, MediaRuntime, MediaRuntimeControl,
+ MediaSession, MediaSink, MediaStreamConfig, MediaVideoCodec, PushOutcome, RecordingSummary,
+ StreamShortcutAction, StreamShortcutBindings, embedded_video_backends, record_matroska,
supports_audio_decode, supports_audio_output, video_backends,
};
use opennow_streamer_protocol::{
Capabilities, Command, PROTOCOL_VERSION, SessionContext, error, event, response,
};
use opennow_streamer_transport::{
- EncodedMicrophoneFrame, NegotiatedVideoCodec, NvstDropReason, NvstReceiveEvent,
- NvstReceiverState, NvstRecovery, NvstUdpReceiverControl, NvstUdpReceiverSession,
- PreferredVideoTransport, ReservedNvstBundle, SharedNvstFeedback, TransportControl,
- TransportEvent, TransportSession, negotiate, reserve_nvst_mjolnir_udp_socket,
- select_preferred_video_transport, spawn_nvst_mjolnir_receiver,
+ NvstDropReason, NvstReceiveEvent, NvstReceiverState, NvstRecovery, NvstUdpReceiverControl,
+ NvstUdpReceiverSession, ReservedNvstBundle, SharedNvstFeedback, parse_nvst_video_handoff,
+ reserve_nvst_mjolnir_udp_socket, spawn_nvst_mjolnir_receiver,
spawn_nvst_udp_receiver_with_socket,
};
use serde_json::{Value, json};
@@ -33,11 +31,44 @@ use nvst_rtsp::{ActiveNvstRtspSession, prepare_owned_nvst};
pub use opennow_streamer_transport::{EncodedMediaFrame, MediaConsumer};
+#[derive(Clone)]
+pub struct EventSender {
+ inner: EventSenderInner,
+}
+
+#[derive(Clone)]
+enum EventSenderInner {
+ Unbounded(Sender),
+ Bounded(SyncSender),
+}
+
+impl EventSender {
+ fn unbounded(sender: Sender) -> Self {
+ Self {
+ inner: EventSenderInner::Unbounded(sender),
+ }
+ }
+
+ pub fn bounded(sender: SyncSender) -> Self {
+ Self {
+ inner: EventSenderInner::Bounded(sender),
+ }
+ }
+
+ fn send(&self, value: Value) -> Result<(), ()> {
+ match &self.inner {
+ EventSenderInner::Unbounded(sender) => sender.send(value).map_err(|_| ()),
+ EventSenderInner::Bounded(sender) => match sender.try_send(value) {
+ Ok(()) => Ok(()),
+ Err(TrySendError::Full(_) | TrySendError::Disconnected(_)) => Err(()),
+ },
+ }
+ }
+}
+
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum State {
Idle,
- Prepared,
- Negotiating,
Connected,
}
@@ -107,21 +138,18 @@ impl NvstSessionResources for ActiveNvstResources {
pub struct Engine {
lifecycle: Arc>,
- transport: Option,
nvst_transport: Option,
nvst_mjolnir_transport: Option,
reserved_nvst_bundle: Option,
nvst_hole_punch_socket: Option,
nvst_rtsp: Option,
- events: Sender,
+ events: EventSender,
media_consumer: Option,
media_runtime: Option,
media_session: Option,
media_worker: Option>,
media_feedback: Option>,
feedback_worker: Option>,
- microphone_capture: Option,
- microphone_enabled: bool,
recording_worker: Option>>,
}
@@ -134,13 +162,16 @@ struct Lifecycle {
impl Engine {
pub fn new(events: Sender) -> Self {
+ Self::with_event_sender(EventSender::unbounded(events))
+ }
+
+ pub fn with_event_sender(events: EventSender) -> Self {
Self {
lifecycle: Arc::new(Mutex::new(Lifecycle {
state: State::Idle,
context: None,
generation: 0,
})),
- transport: None,
nvst_transport: None,
nvst_mjolnir_transport: None,
reserved_nvst_bundle: None,
@@ -153,20 +184,28 @@ impl Engine {
media_worker: None,
media_feedback: None,
feedback_worker: None,
- microphone_capture: None,
- microphone_enabled: false,
recording_worker: None,
}
}
+ pub fn embedded(events: EventSender) -> Self {
+ Self::with_event_sender(events)
+ }
+
pub fn with_media_consumer(events: Sender, media_consumer: MediaConsumer) -> Self {
+ Self::with_media_consumer_and_event_sender(EventSender::unbounded(events), media_consumer)
+ }
+
+ pub fn with_media_consumer_and_event_sender(
+ events: EventSender,
+ media_consumer: MediaConsumer,
+ ) -> Self {
Self {
lifecycle: Arc::new(Mutex::new(Lifecycle {
state: State::Idle,
context: None,
generation: 0,
})),
- transport: None,
nvst_transport: None,
nvst_mjolnir_transport: None,
reserved_nvst_bundle: None,
@@ -179,20 +218,24 @@ impl Engine {
media_worker: None,
media_feedback: None,
feedback_worker: None,
- microphone_capture: None,
- microphone_enabled: false,
recording_worker: None,
}
}
pub fn with_media_runtime(events: Sender, media_runtime: MediaRuntime) -> Self {
+ Self::with_media_runtime_and_event_sender(EventSender::unbounded(events), media_runtime)
+ }
+
+ pub fn with_media_runtime_and_event_sender(
+ events: EventSender,
+ media_runtime: MediaRuntime,
+ ) -> Self {
Self {
lifecycle: Arc::new(Mutex::new(Lifecycle {
state: State::Idle,
context: None,
generation: 0,
})),
- transport: None,
nvst_transport: None,
nvst_mjolnir_transport: None,
reserved_nvst_bundle: None,
@@ -205,12 +248,14 @@ impl Engine {
media_worker: None,
media_feedback: None,
feedback_worker: None,
- microphone_capture: None,
- microphone_enabled: false,
recording_worker: None,
}
}
+ pub fn with_embedded_media_runtime(events: EventSender, media_runtime: MediaRuntime) -> Self {
+ Self::with_media_runtime_and_event_sender(events, media_runtime)
+ }
+
pub fn handle(&mut self, command: Command) -> (Vec, bool) {
let id = command.id.clone();
let result = match command.kind.as_str() {
@@ -219,9 +264,6 @@ impl Engine {
"nvst-unbind" => self.nvst_unbind(command),
"nvst-send" => self.nvst_send(command),
"start" => self.start(command),
- "offer" => self.offer(command),
- "remote-ice" => self.remote_ice(command),
- "input" => self.input(command),
"input-paused" => self.set_paused(command),
"surface" => self.update_surface(command),
"stats-toggle" => Ok(vec![
@@ -238,7 +280,6 @@ impl Engine {
json!({"action":"toggle-fullscreen", "source":"command"}),
),
]),
- "microphone-toggle" => self.toggle_microphone(command),
"anti-afk-pulse" => self.anti_afk_pulse(command),
"recording-start" => self.start_recording(command),
"recording-stop" => self.stop_recording(command),
@@ -276,17 +317,20 @@ impl Engine {
format!("Native streamer requires protocol {PROTOCOL_VERSION}"),
));
}
- let backends = video_backends();
+ let backends = if self
+ .media_runtime
+ .as_ref()
+ .is_some_and(MediaRuntime::is_embedded)
+ {
+ embedded_video_backends()
+ } else {
+ video_backends()
+ };
let media_ready = self.media_runtime.is_some();
let video_ready = media_ready && backends.iter().any(|backend| backend.available);
let capabilities = Capabilities {
protocol_version: PROTOCOL_VERSION,
backend: "native",
- fallback_reason: (!media_ready)
- .then_some("Native streamer requires an in-process decoded media runtime"),
- supports_offer_answer: media_ready,
- supports_remote_ice: media_ready,
- supports_local_ice: media_ready,
supports_input: media_ready,
supports_video_decode: video_ready,
supports_video_present: video_ready,
@@ -295,14 +339,12 @@ impl Engine {
supports_owned_nvst_negotiation: media_ready,
video_backends: backends,
};
- let mut ready = json!({
+ let ready = json!({
"id": command.id,
"type": "ready",
"processId": std::process::id(),
"capabilities": capabilities,
});
- ready["capabilities"]["microphoneDevices"] =
- json!(MicrophoneCapture::device_names().unwrap_or_default());
Ok(vec![ready])
}
@@ -485,29 +527,23 @@ impl Engine {
format!("Session context is not serializable: {context_error}"),
)
})?;
- let explicit_nvst = transport_context
- .pointer("/settings/transportMode")
- .and_then(Value::as_str)
- .is_some_and(|mode| mode.eq_ignore_ascii_case("nvst"))
- || transport_context.get("nvstVideo").is_some()
- || transport_context.get("nvstTransport").is_some();
- let (nvst_config, fallback_note) =
- match select_preferred_video_transport(&transport_context) {
- PreferredVideoTransport::Nvst(config) => (Some(*config), None),
- PreferredVideoTransport::WebRtcFallback(reason) if explicit_nvst => {
- return Err(error(
- Some(&command.id),
- "invalid-nvst-handoff",
- format!("Explicit NVST transport is invalid: {reason:?}"),
- ));
- }
- PreferredVideoTransport::WebRtcFallback(reason) => (
- None,
- Some(format!(
- "NVST unavailable; using WebRTC fallback: {reason:?}"
- )),
- ),
- };
+ let nvst_config = match parse_nvst_video_handoff(&transport_context) {
+ Ok(Some(config)) => Some(config),
+ Ok(None) => {
+ return Err(error(
+ Some(&command.id),
+ "nvst-handoff-required",
+ "Native streaming requires an NVST handoff",
+ ));
+ }
+ Err(reason) => {
+ return Err(error(
+ Some(&command.id),
+ "invalid-nvst-handoff",
+ format!("NVST transport is invalid: {reason}"),
+ ));
+ }
+ };
let nvst_bundle_available = nvst_config
.as_ref()
.is_some_and(|config| config.remote_dtls_fingerprint().is_some());
@@ -515,9 +551,6 @@ impl Engine {
.as_ref()
.is_some_and(|config| config.audio_track().is_some());
- if let Some(transport) = self.transport.take() {
- transport.stop();
- }
if let Some(transport) = self.nvst_transport.take() {
transport.stop();
}
@@ -680,11 +713,7 @@ impl Engine {
let mut lifecycle = lock_lifecycle(&self.lifecycle);
lifecycle.generation = lifecycle.generation.wrapping_add(1);
lifecycle.context = Some(context);
- lifecycle.state = if nvst_events.is_some() {
- State::Connected
- } else {
- State::Prepared
- };
+ lifecycle.state = State::Connected;
lifecycle.generation
};
if let Some(nvst_events) = nvst_events {
@@ -751,341 +780,17 @@ impl Engine {
"status",
json!({
"status": "ready",
- "message": if self.nvst_transport.is_some() {
- "NVST authenticated H.264 receive path initialized"
- } else if self.media_runtime.is_some() {
- "H.264 video and Opus audio media path initialized"
- } else {
- "Native WebRTC session prepared"
- }
+ "message": "NVST authenticated media path initialized"
}),
));
- if let Some(note) = fallback_note {
- let _ = self
- .events
- .send(event("log", json!({ "level": "debug", "message": note })));
- }
let mut start_response = response(command.id, "ok");
- let using_nvst = self.nvst_transport.is_some();
- start_response["transport"] =
- Value::String(if using_nvst { "nvst" } else { "webrtc" }.to_owned());
- start_response["capabilities"] = if using_nvst {
- json!({
- "supportsOfferAnswer": false,
- "supportsRemoteIce": false,
- "supportsLocalIce": false,
- "supportsInput": nvst_bundle_available,
- "supportsAudioDecode": nvst_audio_negotiated && supports_audio_decode(),
- "supportsAudioOutput": nvst_audio_negotiated && supports_audio_output(),
- })
- } else {
- json!({
- "supportsOfferAnswer": self.media_runtime.is_some(),
- "supportsRemoteIce": self.media_runtime.is_some(),
- "supportsLocalIce": self.media_runtime.is_some(),
- "supportsInput": self.media_runtime.is_some(),
- "supportsAudioDecode": self.media_runtime.is_some() && supports_audio_decode(),
- "supportsAudioOutput": self.media_runtime.is_some() && supports_audio_output(),
- })
- };
- Ok(vec![start_response])
- }
-
- fn offer(&mut self, command: Command) -> Result, Value> {
- if self.nvst_transport.is_some() {
- return Err(error(
- Some(&command.id),
- "nvst-video-active",
- "NVST video is active; do not negotiate a WebRTC media offer for this session",
- ));
- }
- let offer_sdp = command.sdp.as_deref().ok_or_else(|| {
- error(
- Some(&command.id),
- "missing-sdp",
- "Offer command does not include SDP",
- )
- })?;
- let offered_context = command
- .context
- .map(|context| parse_context(Some(context), &command.id))
- .transpose()?;
- if let Some(context) = &offered_context {
- validate_context(context, &command.id)?;
- }
- let (context, generation) = {
- let mut lifecycle = lock_lifecycle(&self.lifecycle);
- if lifecycle.state == State::Idle {
- return Err(error(
- Some(&command.id),
- "not-started",
- "Start must be sent before offer",
- ));
- }
- if lifecycle.state != State::Prepared {
- return Err(invalid_state(
- &command.id,
- "offer",
- lifecycle.state,
- "Prepared",
- ));
- }
- let Some(stored_context) = lifecycle.context.as_ref() else {
- lifecycle.state = State::Idle;
- return Err(error(
- Some(&command.id),
- "invalid-state",
- "Prepared lifecycle is missing its session context",
- ));
- };
- let stored_session_id = stored_context.session.session_id.clone();
- if let Some(context) = offered_context {
- if context.session.session_id != stored_session_id {
- return Err(error(
- Some(&command.id),
- "session-mismatch",
- "Offer context does not match the prepared session",
- ));
- }
- lifecycle.context = Some(context);
- }
- let Some(context) = lifecycle.context.clone() else {
- lifecycle.state = State::Idle;
- return Err(error(
- Some(&command.id),
- "invalid-state",
- "Prepared lifecycle is missing its session context",
- ));
- };
- lifecycle.state = State::Negotiating;
- (context, lifecycle.generation)
- };
- let Some(media_consumer) = self.media_consumer.clone() else {
- let mut lifecycle = lock_lifecycle(&self.lifecycle);
- if lifecycle.generation == generation && lifecycle.state == State::Negotiating {
- lifecycle.state = State::Prepared;
- }
- return Err(error(
- Some(&command.id),
- "media-consumer-unavailable",
- "No in-process encoded media consumer is configured",
- ));
- };
- let threshold = partial_reliable_threshold(offer_sdp).unwrap_or(300);
- let video_codec = match media_stream_config(&context).codec {
- MediaVideoCodec::H264 => NegotiatedVideoCodec::H264,
- MediaVideoCodec::H265 => NegotiatedVideoCodec::H265,
- MediaVideoCodec::Av1 => NegotiatedVideoCodec::Av1,
- };
- let (transport_events, receiver) = std::sync::mpsc::channel();
- let negotiated = negotiate(
- offer_sdp,
- &context.session,
- video_codec,
- threshold,
- transport_events,
- media_consumer,
- )
- .map_err(|transport_error| {
- let mut lifecycle = lock_lifecycle(&self.lifecycle);
- if lifecycle.generation == generation && lifecycle.state == State::Negotiating {
- lifecycle.state = State::Prepared;
- }
- error(
- Some(&command.id),
- transport_error.code(),
- transport_error.to_string(),
- )
- })?;
- let output = self.events.clone();
- let lifecycle = self.lifecycle.clone();
- let transport_control = negotiated.session.control();
- let media_feedback = self.media_feedback.take();
- let captured_input = self
- .media_session
- .as_ref()
- .map(MediaSession::captured_input);
- let microphone_packets = match context
- .settings
- .get("microphoneMode")
- .and_then(Value::as_str)
- .unwrap_or("disabled")
- {
- "voice-activity" if self.media_runtime.is_some() => {
- let configured_device = context
- .settings
- .get("microphoneDeviceId")
- .and_then(Value::as_str)
- .filter(|value| !value.trim().is_empty());
- let capture = MicrophoneCapture::start(configured_device).or_else(|configured_error| {
- if configured_device.is_none() {
- return Err(configured_error);
- }
- let _ = self.events.send(event(
- "log",
- json!({"level":"warn","message":"Configured microphone is unavailable; using the system default"}),
- ));
- MicrophoneCapture::start(None).map_err(|fallback_error| {
- format!("configured microphone failed ({configured_error}); default microphone failed ({fallback_error})")
- })
- });
- match capture {
- Ok(capture) => {
- let packets = capture.packets();
- self.microphone_capture = Some(capture);
- self.microphone_enabled = true;
- Some(packets)
- }
- Err(message) => {
- let _ = self.events.send(event(
- "microphone-state",
- json!({"state":"unavailable","enabled":false,"message":message}),
- ));
- None
- }
- }
- }
- "push-to-talk" => {
- let _ = self.events.send(event(
- "microphone-state",
- json!({
- "state":"unavailable",
- "enabled":false,
- "message":"Push-to-talk requires dynamic shortcut negotiation and remains disabled"
- }),
- ));
- None
- }
- _ => None,
- };
- let shortcut_runtime = self.media_runtime.clone();
- let feedback_worker = thread::Builder::new()
- .name("opennow-media-events".to_owned())
- .spawn(move || {
- forward_session_events(
- &output,
- &lifecycle,
- generation,
- WebrtcSessionResources {
- transport_events: receiver,
- media_feedback,
- captured_input,
- microphone_packets,
- shortcut_runtime,
- transport: transport_control,
- },
- );
- });
- self.feedback_worker = Some(match feedback_worker {
- Ok(worker) => worker,
- Err(spawn_error) => {
- negotiated.session.stop();
- let mut lifecycle = lock_lifecycle(&self.lifecycle);
- if lifecycle.generation == generation && lifecycle.state == State::Negotiating {
- lifecycle.state = State::Prepared;
- }
- drop(lifecycle);
- self.stop_media_resources();
- return Err(error(
- Some(&command.id),
- "media-worker-failed",
- spawn_error.to_string(),
- ));
- }
+ start_response["transport"] = Value::String("nvst".to_owned());
+ start_response["capabilities"] = json!({
+ "supportsInput": nvst_bundle_available,
+ "supportsAudioDecode": nvst_audio_negotiated && supports_audio_decode(),
+ "supportsAudioOutput": nvst_audio_negotiated && supports_audio_output(),
});
- self.transport = Some(negotiated.session);
- let _ = self.events.send(event(
- "local-ice",
- json!({ "candidate": negotiated.local_candidate }),
- ));
- Ok(vec![json!({
- "id": command.id,
- "type": "answer",
- "answer": { "sdp": negotiated.answer_sdp },
- })])
- }
-
- fn remote_ice(&self, command: Command) -> Result, Value> {
- if self.nvst_transport.is_some() {
- return Err(error(
- Some(&command.id),
- "nvst-remote-ice-unsupported",
- "NVST owns its negotiated ICE bundle and does not accept remote-ice commands",
- ));
- }
- let state = lock_lifecycle(&self.lifecycle).state;
- if !matches!(state, State::Negotiating | State::Connected) {
- return Err(invalid_state(
- &command.id,
- "remote-ice",
- state,
- "Negotiating or Connected",
- ));
- }
- let transport = self.transport.as_ref().ok_or_else(|| {
- error(
- Some(&command.id),
- "transport-not-ready",
- "No active WebRTC transport",
- )
- })?;
- let candidate = command.candidate.as_ref().ok_or_else(|| {
- error(
- Some(&command.id),
- "missing-candidate",
- "Remote ICE command is empty",
- )
- })?;
- transport
- .add_remote_candidate(candidate)
- .map_err(|transport_error| {
- error(
- Some(&command.id),
- transport_error.code(),
- transport_error.to_string(),
- )
- })?;
- Ok(vec![response(command.id, "ok")])
- }
-
- fn input(&self, command: Command) -> Result, Value> {
- let state = lock_lifecycle(&self.lifecycle).state;
- if state != State::Connected {
- return Err(invalid_state(
- &command.id,
- "input",
- state,
- "Connected with an initialized input channel",
- ));
- }
- let input = command
- .input
- .as_ref()
- .ok_or_else(|| error(Some(&command.id), "missing-input", "Input command is empty"))?;
- let bytes = BASE64
- .decode(&input.payload_base64)
- .map_err(|decode_error| {
- error(Some(&command.id), "invalid-input", decode_error.to_string())
- })?;
- let send_result = if let Some(transport) = self.nvst_transport.as_ref() {
- transport.send_input(bytes, input.partially_reliable)
- } else if let Some(transport) = self.transport.as_ref() {
- transport.send_input(bytes, input.partially_reliable)
- } else {
- return Err(error(
- Some(&command.id),
- "transport-not-ready",
- "No active media transport",
- ));
- };
- send_result.map_err(|transport_error| {
- error(
- Some(&command.id),
- transport_error.code(),
- transport_error.to_string(),
- )
- })?;
- Ok(vec![response(command.id, "ok")])
+ Ok(vec![start_response])
}
fn set_paused(&self, command: Command) -> Result, Value> {
@@ -1170,9 +875,6 @@ impl Engine {
lifecycle.state = State::Idle;
was_active
};
- if let Some(transport) = self.transport.take() {
- transport.stop();
- }
if let Some(transport) = self.nvst_transport.take() {
transport.stop();
}
@@ -1195,8 +897,6 @@ impl Engine {
fn stop_media_resources(&mut self) {
let _ = self.stop_recording_inner();
- self.microphone_capture = None;
- self.microphone_enabled = false;
if self.media_runtime.is_some() {
self.media_consumer = None;
if let Some(session) = self.media_session.take() {
@@ -1300,36 +1000,6 @@ impl Engine {
}
}
- fn toggle_microphone(&mut self, command: Command) -> Result, Value> {
- let capture = self.microphone_capture.as_ref().ok_or_else(|| {
- error(
- Some(&command.id),
- "microphone-unavailable",
- "Microphone was not armed when this WebRTC session started",
- )
- })?;
- self.microphone_enabled = !self.microphone_enabled;
- capture.set_enabled(self.microphone_enabled);
- let message = if self.microphone_enabled {
- "Microphone is streaming"
- } else {
- "Microphone is muted"
- };
- let _ = self.events.send(event(
- "microphone-state",
- json!({
- "state":if self.microphone_enabled { "ready" } else { "disabled" },
- "enabled":self.microphone_enabled,
- "message":message
- }),
- ));
- Ok(vec![json!({
- "id":command.id,
- "type":"microphone-state",
- "enabled":self.microphone_enabled,
- })])
- }
-
fn anti_afk_pulse(&self, command: Command) -> Result, Value> {
let state = lock_lifecycle(&self.lifecycle).state;
if state != State::Connected {
@@ -1344,8 +1014,6 @@ impl Engine {
let bytes = captured_input_packet(input, 0);
if let Some(transport) = self.nvst_transport.as_ref() {
transport.send_input(bytes, false)
- } else if let Some(transport) = self.transport.as_ref() {
- transport.send_input(bytes, false)
} else {
Err(opennow_streamer_transport::TransportError::Closed)
}
@@ -1392,14 +1060,6 @@ impl Drop for Engine {
}
}
-fn partial_reliable_threshold(sdp: &str) -> Option {
- sdp.lines().find_map(|line| {
- line.trim()
- .strip_prefix("a=ri.partialReliableThresholdMs:")
- .and_then(|value| value.trim().parse().ok())
- })
-}
-
fn parse_context(context: Option, id: &str) -> Result {
let context = context.ok_or_else(|| {
error(
@@ -1439,18 +1099,6 @@ fn validate_context(context: &SessionContext, id: &str) -> Result<(), Value> {
"Session context settings and shortcuts must be objects",
));
}
- if context
- .session
- .ice_servers
- .iter()
- .any(|server| server.urls.is_empty() || server.urls.iter().any(|url| url.trim().is_empty()))
- {
- return Err(error(
- Some(id),
- "invalid-context",
- "Every ICE server requires at least one non-empty URL",
- ));
- }
if let Some(endpoint) = &context.session.media_connection_info {
if endpoint.ip.trim().is_empty() || endpoint.port == 0 || endpoint.port > u16::MAX.into() {
return Err(error(
@@ -1505,61 +1153,20 @@ fn lock_lifecycle(lifecycle: &Mutex) -> MutexGuard<'_, Lifecycle> {
.unwrap_or_else(|poisoned| poisoned.into_inner())
}
-fn forward_transport_event(
- output: &Sender,
- lifecycle: &Mutex,
- generation: u64,
- transport_event: TransportEvent,
-) {
- {
- let mut lifecycle = lock_lifecycle(lifecycle);
- if lifecycle.generation != generation {
- return;
- }
- match &transport_event {
- TransportEvent::Connected => lifecycle.state = State::Connected,
- TransportEvent::Disconnected(_) => {
- lifecycle.context = None;
- lifecycle.state = State::Idle;
- }
- _ => {}
- }
- }
- let value = match transport_event {
- TransportEvent::Connected => event(
- "status",
- json!({ "status": "streaming", "message": "ICE, DTLS-SRTP, and RTP connected" }),
- ),
- TransportEvent::Disconnected(message) => {
- event("status", json!({ "status": "stopped", "message": message }))
- }
- TransportEvent::InputReady(protocol_version) => event(
- "input-ready",
- json!({ "protocolVersion": protocol_version }),
- ),
- TransportEvent::InputUnavailable(reason) => {
- event("input-unavailable", json!({ "reason": reason }))
- }
- TransportEvent::MicrophoneReady => event(
- "microphone-state",
- json!({"state":"ready","enabled":true,"message":"Microphone is streaming"}),
- ),
- TransportEvent::MicrophoneUnavailable(reason) => event(
- "microphone-state",
- json!({"state":"unavailable","enabled":false,"message":reason}),
- ),
- TransportEvent::Log(message) => {
- event("log", json!({ "level": "warn", "message": message }))
- }
- };
- let _ = output.send(value);
-}
-
fn forward_shortcut_action(
- output: &Sender,
+ output: &EventSender,
runtime: Option<&MediaRuntime>,
action: StreamShortcutAction,
) {
+ if action == StreamShortcutAction::TogglePointerLock
+ && runtime.is_some_and(MediaRuntime::is_embedded)
+ {
+ let _ = output.send(event(
+ "shortcut-action",
+ json!({"action":action.protocol_name(), "source":"keyboard"}),
+ ));
+ return;
+ }
let control = match action {
StreamShortcutAction::ToggleStats => None,
StreamShortcutAction::ToggleFullscreen => None,
@@ -1593,7 +1200,7 @@ struct NvstSessionEventResources {
}
fn forward_nvst_session_events(
- output: &Sender,
+ output: &EventSender,
lifecycle: &Mutex,
generation: u64,
event_resources: NvstSessionEventResources,
@@ -1714,7 +1321,7 @@ fn forward_nvst_session_events(
}
fn forward_nvst_event(
- output: &Sender,
+ output: &EventSender,
lifecycle: &Mutex,
generation: u64,
resources: &R,
@@ -1849,7 +1456,7 @@ fn forward_nvst_event(
}
fn attempt_nvst_recovery(
- output: &Sender,
+ output: &EventSender,
lifecycle: &Mutex,
generation: u64,
resources: &R,
@@ -1890,7 +1497,7 @@ fn attempt_nvst_recovery(
}
fn emit_nvst_terminal(
- output: &Sender,
+ output: &EventSender,
lifecycle: &Mutex,
generation: u64,
resources: &R,
@@ -1941,7 +1548,7 @@ impl NvstMediaFeedbackState {
}
fn forward_nvst_media_feedback(
- output: &Sender,
+ output: &EventSender,
lifecycle: &Mutex,
generation: u64,
resources: &R,
@@ -2348,7 +1955,7 @@ fn media_stream_config(context: &SessionContext) -> MediaStreamConfig {
}
fn consume_encoded_media(
- output: &Sender,
+ output: &EventSender,
receiver: Receiver,
sink: MediaSink,
) {
@@ -2392,252 +1999,6 @@ fn consume_encoded_media(
}
}
-struct WebrtcSessionResources {
- transport_events: Receiver,
- media_feedback: Option>,
- captured_input: Option>,
- microphone_packets: Option>,
- shortcut_runtime: Option,
- transport: TransportControl,
-}
-
-fn forward_session_events(
- output: &Sender,
- lifecycle: &Mutex,
- generation: u64,
- resources: WebrtcSessionResources,
-) {
- let WebrtcSessionResources {
- transport_events,
- media_feedback,
- captured_input,
- microphone_packets,
- shortcut_runtime,
- transport,
- } = resources;
- let mut drop_reports = HashMap::new();
- let input_origin = Instant::now();
- let mut input_available = false;
- let mut microphone_available = false;
- loop {
- if let Some(feedback) = media_feedback.as_ref() {
- while let Ok(feedback) = feedback.try_recv() {
- forward_media_feedback(
- output,
- lifecycle,
- generation,
- &transport,
- feedback,
- &mut drop_reports,
- );
- }
- }
- if let Some(captured_input) = captured_input.as_ref() {
- if !input_available {
- captured_input.clear();
- } else if captured_input.take_overflowed() {
- let _ = output.send(event(
- "error",
- json!({
- "code":"native-input-capture-overflow",
- "message":"Native input capture queue overflowed; input was paused to prevent stuck controls"
- }),
- ));
- input_available = false;
- } else {
- for _ in 0..32 {
- let Some(sample) = captured_input.take_sample() else {
- break;
- };
- if matches!(sample.input, CapturedInput::Guide) {
- let _ = output.send(event("overlay-request", json!({"source":"gamepad"})));
- continue;
- }
- if matches!(sample.input, CapturedInput::Screenshot) {
- let _ =
- output.send(event("screenshot-request", json!({"source":"keyboard"})));
- continue;
- }
- if matches!(sample.input, CapturedInput::RecordingToggle) {
- let _ = output.send(event(
- "recording-toggle-request",
- json!({"source":"keyboard"}),
- ));
- continue;
- }
- if let CapturedInput::Shortcut(action) = &sample.input {
- forward_shortcut_action(output, shortcut_runtime.as_ref(), *action);
- continue;
- }
- let captured = sample
- .captured_at
- .checked_duration_since(input_origin)
- .unwrap_or_default();
- let timestamp_us = u64::try_from(captured.as_micros()).unwrap_or(u64::MAX);
- if transport
- .send_input(captured_input_packet(sample.input, timestamp_us), false)
- .is_err()
- {
- input_available = false;
- break;
- }
- }
- }
- }
- if let Some(microphone_packets) = microphone_packets.as_ref() {
- if !microphone_available {
- microphone_packets.clear();
- } else {
- let dropped = microphone_packets.take_dropped();
- if dropped > 0 {
- let _ = output.send(event(
- "log",
- json!({
- "level":"warn",
- "message":format!("Dropped {dropped} microphone packets under load")
- }),
- ));
- }
- for _ in 0..4 {
- let Some(packet) = microphone_packets.take() else {
- break;
- };
- let _ = transport.send_microphone(EncodedMicrophoneFrame {
- payload: packet.payload,
- captured_at_us: packet.captured_at_us,
- audio_level_db: packet.audio_level_db,
- voice_activity: packet.voice_activity,
- });
- }
- }
- }
- match transport_events.recv_timeout(Duration::from_millis(5)) {
- Ok(transport_event) => {
- match &transport_event {
- TransportEvent::InputReady(_) => input_available = true,
- TransportEvent::InputUnavailable(_) | TransportEvent::Disconnected(_) => {
- input_available = false
- }
- TransportEvent::MicrophoneReady => microphone_available = true,
- TransportEvent::MicrophoneUnavailable(_) => microphone_available = false,
- _ => {}
- }
- let disconnected = matches!(transport_event, TransportEvent::Disconnected(_));
- forward_transport_event(output, lifecycle, generation, transport_event);
- if disconnected {
- break;
- }
- }
- Err(std::sync::mpsc::RecvTimeoutError::Timeout) => {}
- Err(std::sync::mpsc::RecvTimeoutError::Disconnected) => break,
- }
- }
-}
-
-fn forward_media_feedback(
- output: &Sender,
- lifecycle: &Mutex,
- generation: u64,
- transport: &TransportControl,
- feedback: MediaFeedback,
- drop_reports: &mut HashMap<&'static str, QueueDropReport>,
-) {
- if lock_lifecycle(lifecycle).generation != generation {
- return;
- }
- match feedback {
- MediaFeedback::VideoFrameAccepted { .. } => {}
- MediaFeedback::PlaybackStarted { backend } => {
- let _ = output.send(event(
- "status",
- json!({
- "event": "first-frame",
- "backend": backend,
- "status": "streaming",
- "message": format!("{backend} presented the first video frame")
- }),
- ));
- }
- MediaFeedback::BackendFallback { from, to, reason } => {
- let _ = output.send(event(
- "log",
- json!({
- "event": "backend-fallback",
- "fromBackend": from,
- "toBackend": to,
- "reason": reason,
- "level": "warn",
- "message": format!("{from} startup failed; using {to}: {reason}")
- }),
- ));
- }
- MediaFeedback::RequestKeyframe { mid, reason } => {
- let request_result = transport.request_keyframe(mid);
- let _ = output.send(event(
- "log",
- json!({
- "event": "keyframe-request",
- "reason": reason,
- "level": if request_result.is_ok() { "info" } else { "warn" },
- "message": format!("Requested a video keyframe: {reason}")
- }),
- ));
- }
- MediaFeedback::DecoderError { codec, message } => {
- let _ = output.send(event(
- "error",
- json!({
- "event": "decoder-error",
- "codec": codec,
- "code": "media-decode-error",
- "message": format!("{codec} decoder error: {message}")
- }),
- ));
- }
- MediaFeedback::OutputError { message } => {
- let _ = output.send(event(
- "error",
- json!({ "event": "output-error", "code": "media-output-error", "message": message }),
- ));
- }
- MediaFeedback::DeviceLost {
- subsystem,
- recovered,
- message,
- } => {
- let _ = output.send(event(
- "log",
- json!({
- "event": "device-state",
- "subsystem": subsystem,
- "recovered": recovered,
- "level": if recovered { "info" } else { "warn" },
- "message": message.unwrap_or_else(|| format!(
- "{subsystem} device {}",
- if recovered { "recovered" } else { "was lost" }
- ))
- }),
- ));
- }
- MediaFeedback::QueueDropped { media, count } => {
- if let Some(dropped) = record_queue_drop(drop_reports, media, count) {
- let _ = output.send(event(
- "log",
- json!({
- "event": "queue-dropped",
- "media": media,
- "count": dropped,
- "level": "debug",
- "message": format!(
- "Low-latency {media} queues dropped {dropped} stale samples/frames"
- )
- }),
- ));
- }
- }
- }
-}
-
struct QueueDropReport {
dropped: usize,
started: Instant,
@@ -2667,8 +2028,6 @@ mod tests {
use std::net::UdpSocket;
use std::sync::atomic::{AtomicUsize, Ordering};
use std::time::Instant;
- use str0m::media::{Direction, MediaKind};
- use str0m::{Candidate, RtcConfig};
fn command(value: Value) -> Command {
serde_json::from_value(value).expect("command")
@@ -2693,19 +2052,6 @@ mod tests {
})
}
- fn synthetic_offer() -> String {
- opennow_streamer_transport::install_crypto();
- let mut offerer = RtcConfig::new().build(Instant::now());
- offerer.add_local_candidate(
- Candidate::host("127.0.0.1:49152".parse().expect("candidate address"), "udp")
- .expect("local candidate"),
- );
- let mut change = offerer.sdp_api();
- change.add_media(MediaKind::Video, Direction::SendOnly, None, None, None);
- let (offer, _pending) = change.apply().expect("synthetic offer");
- offer.to_sdp_string()
- }
-
fn lifecycle_state(engine: &Engine) -> State {
lock_lifecycle(&engine.lifecycle).state
}
@@ -2713,11 +2059,7 @@ mod tests {
#[test]
fn shell_shortcuts_emit_typed_actions() {
let (sender, receiver) = std::sync::mpsc::channel();
- forward_shortcut_action(&sender, None, StreamShortcutAction::ToggleMicrophone);
- let action = receiver.recv().expect("shortcut action");
- assert_eq!(action["type"], "shortcut-action");
- assert_eq!(action["action"], "toggle-microphone");
-
+ let sender = EventSender::unbounded(sender);
forward_shortcut_action(&sender, None, StreamShortcutAction::ToggleStats);
let action = receiver.recv().expect("stats shortcut action");
assert_eq!(action["type"], "shortcut-action");
@@ -2820,6 +2162,7 @@ mod tests {
#[test]
fn decoder_keyframe_feedback_routes_to_nvst_pli_handle() {
let (sender, receiver) = std::sync::mpsc::channel();
+ let sender = EventSender::unbounded(sender);
let lifecycle = connected_lifecycle();
let resources = TestNvstResources::default();
let mut state = NvstMediaFeedbackState::new(true);
@@ -2849,6 +2192,7 @@ mod tests {
#[test]
fn accepted_video_keyframe_routes_pacing_feedback_and_resets_recovery_budget() {
let (sender, _receiver) = std::sync::mpsc::channel();
+ let sender = EventSender::unbounded(sender);
let lifecycle = connected_lifecycle();
let resources = TestNvstResources::default();
let mut state = NvstMediaFeedbackState::new(true);
@@ -2874,6 +2218,7 @@ mod tests {
#[test]
fn accepted_video_frames_emit_bounded_shell_telemetry() {
let (sender, receiver) = std::sync::mpsc::channel();
+ let sender = EventSender::unbounded(sender);
let lifecycle = connected_lifecycle();
let resources = TestNvstResources::default();
let mut state = NvstMediaFeedbackState::new(true);
@@ -3139,6 +2484,7 @@ mod tests {
#[test]
fn nvst_recovery_is_attempted_once_with_pli() {
let (sender, receiver) = std::sync::mpsc::channel();
+ let sender = EventSender::unbounded(sender);
let lifecycle = connected_lifecycle();
let resources = TestNvstResources::default();
let mut recovery_attempts = 0;
@@ -3170,6 +2516,7 @@ mod tests {
#[test]
fn repeated_packet_gaps_request_keyframes_without_stopping_the_session() {
let (sender, receiver) = std::sync::mpsc::channel();
+ let sender = EventSender::unbounded(sender);
let lifecycle = connected_lifecycle();
let resources = TestNvstResources::default();
let mut recovery_attempts = 0;
@@ -3203,6 +2550,7 @@ mod tests {
#[test]
fn transient_media_backpressure_requests_keyframe_without_stopping_session() {
let (sender, receiver) = std::sync::mpsc::channel();
+ let sender = EventSender::unbounded(sender);
let lifecycle = connected_lifecycle();
let resources = TestNvstResources::default();
let mut recovery_attempts = 0;
@@ -3230,6 +2578,7 @@ mod tests {
#[test]
fn exhausted_nvst_recovery_stops_every_leg_and_emits_terminal_status() {
let (sender, receiver) = std::sync::mpsc::channel();
+ let sender = EventSender::unbounded(sender);
let lifecycle = connected_lifecycle();
let resources = TestNvstResources::default();
let mut recovery_attempts = 0;
@@ -3277,6 +2626,7 @@ mod tests {
#[test]
fn assembled_keyframe_does_not_reset_recovery_episode_budget() {
let (sender, _receiver) = std::sync::mpsc::channel();
+ let sender = EventSender::unbounded(sender);
let lifecycle = connected_lifecycle();
let resources = TestNvstResources::default();
let mut recovery_attempts = 1;
@@ -3311,16 +2661,17 @@ mod tests {
}));
let (responses, _) = engine.handle(command);
assert_eq!(responses[0]["type"], "ready");
- assert_eq!(responses[0]["capabilities"]["supportsOfferAnswer"], false);
- assert_eq!(responses[0]["capabilities"]["supportsVideoPresent"], false);
- }
-
- #[test]
- fn extracts_partial_reliable_threshold() {
- assert_eq!(
- partial_reliable_threshold("v=0\r\na=ri.partialReliableThresholdMs:250\r\n"),
- Some(250),
+ assert!(
+ responses[0]["capabilities"]
+ .get("supportsOfferAnswer")
+ .is_none()
);
+ assert!(
+ responses[0]["capabilities"]
+ .get("supportsRemoteIce")
+ .is_none()
+ );
+ assert_eq!(responses[0]["capabilities"]["supportsVideoPresent"], false);
}
#[test]
@@ -3416,33 +2767,7 @@ mod tests {
}
#[test]
- fn start_validates_stores_context_and_prepares_session() {
- let (sender, receiver) = std::sync::mpsc::channel();
- let mut engine = Engine::new(sender);
- let context = synthetic_context("synthetic-session", json!([]));
- let start = command(json!({
- "id": "start",
- "type": "start",
- "context": context,
- }));
- let (responses, _) = engine.handle(start);
-
- assert_eq!(responses[0]["type"], "ok");
- assert_eq!(responses[0]["transport"], "webrtc");
- let lifecycle = lock_lifecycle(&engine.lifecycle);
- assert_eq!(lifecycle.state, State::Prepared);
- let stored = serde_json::to_value(lifecycle.context.as_ref().expect("stored context"))
- .expect("serializable stored context");
- assert_eq!(stored["session"]["sessionId"], "synthetic-session");
- assert_eq!(stored["session"]["syntheticExtension"], "preserved");
- assert_eq!(stored["syntheticContextExtension"], true);
- drop(lifecycle);
- let status = receiver.recv().expect("ready status");
- assert_eq!(status["status"], "ready");
- }
-
- #[test]
- fn valid_nvst_handoff_starts_udp_video_and_bypasses_webrtc_offer_negotiation() {
+ fn valid_nvst_handoff_starts_udp_video_and_rejects_removed_offer_command() {
let (sender, receiver) = std::sync::mpsc::channel();
let (media_sender, _media_receiver) = std::sync::mpsc::sync_channel(4);
let mut engine = Engine::with_media_consumer(sender, media_sender);
@@ -3464,13 +2789,20 @@ mod tests {
assert_eq!(responses[0]["type"], "ok");
assert_eq!(responses[0]["transport"], "nvst");
- assert_eq!(responses[0]["capabilities"]["supportsOfferAnswer"], false);
- assert_eq!(responses[0]["capabilities"]["supportsRemoteIce"], false);
+ assert!(
+ responses[0]["capabilities"]
+ .get("supportsOfferAnswer")
+ .is_none()
+ );
+ assert!(
+ responses[0]["capabilities"]
+ .get("supportsRemoteIce")
+ .is_none()
+ );
assert_eq!(responses[0]["capabilities"]["supportsInput"], false);
assert_eq!(responses[0]["capabilities"]["supportsAudioDecode"], false);
assert_eq!(lifecycle_state(&engine), State::Connected);
assert!(engine.nvst_transport.is_some());
- assert!(engine.transport.is_none());
assert!(receiver.try_iter().any(|message| {
message["type"] == "status"
&& message["message"]
@@ -3482,9 +2814,8 @@ mod tests {
"id": "offer",
"type": "offer",
"context": context,
- "sdp": synthetic_offer(),
})));
- assert_eq!(responses[0]["code"], "nvst-video-active");
+ assert_eq!(responses[0]["code"], "unknown-command");
let (responses, _) = engine.handle(command(json!({
"id": "stop",
@@ -3496,7 +2827,7 @@ mod tests {
}
#[test]
- fn explicit_invalid_nvst_handoff_fails_closed_without_webrtc_fallback() {
+ fn explicit_invalid_nvst_handoff_fails_closed() {
let (sender, _receiver) = std::sync::mpsc::channel();
let mut engine = Engine::new(sender);
let mut context = synthetic_context("invalid-nvst-session", json!([]));
@@ -3513,7 +2844,6 @@ mod tests {
assert_eq!(responses[0]["code"], "invalid-nvst-handoff");
assert_eq!(lifecycle_state(&engine), State::Idle);
- assert!(engine.transport.is_none());
assert!(engine.nvst_transport.is_none());
}
@@ -3532,7 +2862,6 @@ mod tests {
assert_eq!(responses[0]["code"], "missing-rtsps-endpoint");
assert_eq!(lifecycle_state(&engine), State::Idle);
- assert!(engine.transport.is_none());
assert!(engine.nvst_transport.is_none());
}
@@ -3559,7 +2888,7 @@ mod tests {
}
#[test]
- fn start_rejects_invalid_and_duplicate_sessions() {
+ fn start_rejects_invalid_contexts_and_missing_nvst_handoffs() {
let (sender, _receiver) = std::sync::mpsc::channel();
let mut engine = Engine::new(sender);
let invalid = command(json!({
@@ -3575,241 +2904,13 @@ mod tests {
assert_eq!(responses[0]["code"], "invalid-context");
assert_eq!(lifecycle_state(&engine), State::Idle);
- for id in ["first", "duplicate"] {
- let start = command(json!({
- "id": id,
- "type": "start",
- "context": synthetic_context("synthetic-session", json!([])),
- }));
- let (responses, _) = engine.handle(start);
- if id == "first" {
- assert_eq!(responses[0]["type"], "ok");
- } else {
- assert_eq!(responses[0]["code"], "invalid-state");
- }
- }
- }
-
- #[test]
- fn offer_negotiates_directly_with_configured_ice_services() {
- let (sender, _receiver) = std::sync::mpsc::channel();
- let (media_sender, _media_receiver) = std::sync::mpsc::sync_channel(4);
- let mut engine = Engine::with_media_consumer(sender, media_sender);
- let context = synthetic_context(
- "synthetic-session",
- json!([{
- "urls": ["stun:stun.synthetic.invalid:3478", "turn:turn.synthetic.invalid:3478"],
- "username": "synthetic-user",
- "credential": "synthetic-credential"
- }]),
- );
- let (responses, _) = engine.handle(command(json!({
- "id": "start",
- "type": "start",
- "context": context.clone(),
- })));
- assert_eq!(responses[0]["type"], "ok");
-
let (responses, _) = engine.handle(command(json!({
- "id": "offer",
- "type": "offer",
- "context": context,
- "sdp": synthetic_offer()
- })));
- assert_eq!(responses[0]["type"], "answer");
- assert!(
- responses[0]["answer"]["sdp"]
- .as_str()
- .is_some_and(|sdp| sdp.contains("m=video") && !sdp.contains("m=video 0"))
- );
- assert_eq!(lifecycle_state(&engine), State::Negotiating);
- }
-
- #[test]
- fn offer_fails_typed_when_no_in_process_media_consumer_exists() {
- let (sender, _receiver) = std::sync::mpsc::channel();
- let mut engine = Engine::new(sender);
- let context = synthetic_context("synthetic-session", json!([]));
- let (responses, _) = engine.handle(command(json!({
- "id": "start",
- "type": "start",
- "context": context.clone(),
- })));
- assert_eq!(responses[0]["type"], "ok");
-
- let (responses, _) = engine.handle(command(json!({
- "id": "offer",
- "type": "offer",
- "context": context,
- "sdp": "v=0\r\n"
- })));
-
- assert_eq!(responses[0]["code"], "media-consumer-unavailable");
- assert_eq!(lifecycle_state(&engine), State::Prepared);
- }
-
- #[test]
- fn prepared_session_negotiates_synthetic_offer_for_typed_media_consumer() {
- let (sender, receiver) = std::sync::mpsc::channel();
- let (media_sender, _media_receiver) = std::sync::mpsc::sync_channel(4);
- let mut engine = Engine::with_media_consumer(sender, media_sender);
- let context = synthetic_context("synthetic-session", json!([]));
- let (responses, _) = engine.handle(command(json!({
- "id": "start",
- "type": "start",
- "context": context.clone(),
- })));
- assert_eq!(responses[0]["type"], "ok");
-
- let (responses, _) = engine.handle(command(json!({
- "id": "offer",
- "type": "offer",
- "context": context,
- "sdp": synthetic_offer()
- })));
-
- assert_eq!(responses[0]["type"], "answer");
- assert!(
- responses[0]["answer"]["sdp"]
- .as_str()
- .is_some_and(|sdp| sdp.contains("m=video") && !sdp.contains("m=video 0"))
- );
- assert_eq!(lifecycle_state(&engine), State::Negotiating);
- assert!(
- receiver
- .try_iter()
- .any(|value| value["type"] == "local-ice")
- );
- }
-
- #[test]
- fn disconnect_clears_context_and_stale_disconnect_cannot_clear_new_session() {
- let (sender, receiver) = std::sync::mpsc::channel();
- let mut engine = Engine::new(sender.clone());
- let (responses, _) = engine.handle(command(json!({
- "id": "first",
- "type": "start",
- "context": synthetic_context("first-session", json!([])),
- })));
- assert_eq!(responses[0]["type"], "ok");
- let first_generation = lock_lifecycle(&engine.lifecycle).generation;
- forward_transport_event(
- &sender,
- &engine.lifecycle,
- first_generation,
- TransportEvent::Disconnected("synthetic disconnect".to_owned()),
- );
- {
- let lifecycle = lock_lifecycle(&engine.lifecycle);
- assert_eq!(lifecycle.state, State::Idle);
- assert!(lifecycle.context.is_none());
- }
-
- let (responses, _) = engine.handle(command(json!({
- "id": "second",
- "type": "start",
- "context": synthetic_context("second-session", json!([])),
- })));
- assert_eq!(responses[0]["type"], "ok");
- forward_transport_event(
- &sender,
- &engine.lifecycle,
- first_generation,
- TransportEvent::Disconnected("late stale disconnect".to_owned()),
- );
- let lifecycle = lock_lifecycle(&engine.lifecycle);
- assert_eq!(lifecycle.state, State::Prepared);
- assert_eq!(
- lifecycle
- .context
- .as_ref()
- .map(|value| value.session.session_id.as_str()),
- Some("second-session")
- );
- drop(lifecycle);
-
- let events = receiver.try_iter().collect::>();
- assert!(events.iter().any(|value| value["status"] == "stopped"));
- assert!(
- !events
- .iter()
- .any(|value| value["message"] == "late stale disconnect")
- );
- }
-
- #[test]
- fn encoded_media_consumer_is_typed_in_process_and_preserves_arc_payload() {
- let (sender, receiver) = std::sync::mpsc::channel();
- let (media_sender, media_receiver) = std::sync::mpsc::sync_channel(4);
- let mut engine = Engine::with_media_consumer(sender, media_sender);
- let (responses, _) = engine.handle(command(json!({
- "id": "start",
- "type": "start",
- "context": synthetic_context("synthetic-session", json!([])),
- })));
- assert_eq!(responses[0]["type"], "ok");
- let payload: Arc<[u8]> = Arc::from([1_u8, 2, 3]);
- engine
- .media_consumer
- .as_ref()
- .expect("media consumer")
- .send(EncodedMediaFrame {
- mid: "video-0".to_owned(),
- codec: "H264".to_owned(),
- payload: payload.clone(),
- rtp_timestamp: 90_000,
- clock_rate_hz: 90_000,
- channels: None,
- received_at_us: 1_500,
- keyframe: true,
- contiguous: true,
- })
- .expect("frame delivery");
-
- let frame = media_receiver.recv().expect("encoded frame");
- assert!(Arc::ptr_eq(&frame.payload, &payload));
- assert_eq!(frame.rtp_timestamp, 90_000);
- assert_eq!(frame.clock_rate_hz, 90_000);
- assert_eq!(frame.received_at_us, 1_500);
- assert!(
- receiver
- .try_iter()
- .all(|value| value["type"] != "encoded-media")
- );
- }
-
- #[test]
- fn unapplied_commands_are_rejected_and_stop_clears_context() {
- let (sender, _receiver) = std::sync::mpsc::channel();
- let mut engine = Engine::new(sender);
- let (responses, _) = engine.handle(command(json!({
- "id": "surface",
- "type": "surface",
- "surface": {}
- })));
- assert_eq!(responses[0]["code"], "unsupported-command");
-
- let (responses, _) = engine.handle(command(json!({
- "id": "start",
+ "id": "missing-nvst",
"type": "start",
"context": synthetic_context("synthetic-session", json!([])),
})));
- assert_eq!(responses[0]["type"], "ok");
- let (responses, _) = engine.handle(command(json!({
- "id": "shortcuts",
- "type": "update-shortcuts",
- "shortcuts": { "stopStream": "Ctrl+Alt+Q" }
- })));
- assert_eq!(responses[0]["code"], "unsupported-command");
- let (responses, _) = engine.handle(command(json!({
- "id": "stop",
- "type": "stop",
- "reason": "synthetic test complete"
- })));
- assert_eq!(responses[0]["type"], "ok");
- let lifecycle = lock_lifecycle(&engine.lifecycle);
- assert_eq!(lifecycle.state, State::Idle);
- assert!(lifecycle.context.is_none());
+ assert_eq!(responses[0]["code"], "nvst-handoff-required");
+ assert_eq!(lifecycle_state(&engine), State::Idle);
}
#[test]
diff --git a/native/opennow-streamer/crates/opennow-streamer-ffi/Cargo.toml b/native/opennow-streamer/crates/opennow-streamer-ffi/Cargo.toml
new file mode 100644
index 000000000..79b4f1935
--- /dev/null
+++ b/native/opennow-streamer/crates/opennow-streamer-ffi/Cargo.toml
@@ -0,0 +1,24 @@
+[package]
+name = "opennow-streamer-ffi"
+version.workspace = true
+edition.workspace = true
+license.workspace = true
+rust-version.workspace = true
+
+[lib]
+crate-type = ["cdylib", "rlib"]
+
+[features]
+default = []
+linux-vaapi = ["opennow-streamer-platform/linux-vaapi"]
+linux-ffmpeg = ["opennow-streamer-platform/linux-ffmpeg"]
+linux-ffmpeg-bundled = ["opennow-streamer-platform/linux-ffmpeg-bundled"]
+
+[dependencies]
+opennow-streamer-core = { path = "../opennow-streamer-core" }
+opennow-streamer-platform = { path = "../opennow-streamer-platform" }
+opennow-streamer-protocol = { path = "../opennow-streamer-protocol" }
+serde_json.workspace = true
+
+[dev-dependencies]
+opennow-streamer-platform = { path = "../opennow-streamer-platform", features = ["test-runtime"] }
diff --git a/native/opennow-streamer/crates/opennow-streamer-ffi/README.md b/native/opennow-streamer/crates/opennow-streamer-ffi/README.md
new file mode 100644
index 000000000..266124aaa
--- /dev/null
+++ b/native/opennow-streamer/crates/opennow-streamer-ffi/README.md
@@ -0,0 +1,34 @@
+# OpenNOW streamer FFI
+
+This crate exposes `opennow-streamer-core::Engine` as a C-compatible in-process library. It is an integration boundary only: it does not start a child process, read or write standard streams, create an SDL/native window, or depend on Qt.
+
+## Contract
+
+- Initialize every `OpenNowStreamerConfig` field, set `abi_version` to `OPENNOW_STREAMER_FFI_ABI_VERSION`, and set `struct_size` to `sizeof(OpenNowStreamerConfig)`.
+- `opennow_streamer_create` writes one owned opaque handle to `output`. On failure it writes `NULL` when the supplied config header is readable.
+- `opennow_streamer_send` copies one serialized JSON protocol command before returning. `OPENNOW_STREAMER_OK` means the bounded command queue accepted it, not that the protocol command succeeded. Parse errors and protocol results arrive at `response_callback`.
+- Values returned directly by `Engine::handle` go to `response_callback`. Unsolicited engine events go to `event_callback`. Each byte slice is UTF-8 JSON and is valid only for the duration of that callback.
+- `frame_available_callback` is separate from the protocol callbacks. It only asks the shell to schedule a render; it carries no frame or protocol payload and may run on a decoder thread.
+- Response and event callbacks each run serially on their own dispatcher thread, so the two callbacks may overlap. They must return promptly and must not re-enter this handle. `user_data` must remain valid until `opennow_streamer_destroy` returns.
+- `opennow_streamer_destroy` consumes the handle exactly once, waits for the engine and both callback queues to drain, and then returns. No call may race with destroy. A null handle is rejected; reusing a destroyed pointer is caller-side undefined behavior.
+- Every exported function catches Rust panics before they can unwind through the C ABI. A worker-thread panic closes the command queue.
+
+All three queues are bounded. Command submission returns `OPENNOW_STREAMER_QUEUE_FULL` rather than blocking. Responses backpressure the engine worker so an accepted command's response is retained. Unsolicited events use a drop-newest policy when their queue is full because the engine's event path cannot block latency-sensitive transport workers.
+
+## GPU frame lifecycle
+
+The graphics API is GPU-only. It exposes no window, swap chain, `QWindow`, CPU image, pixel buffer, or encoded-video callback.
+
+1. On the QQuick render thread, call `opennow_streamer_set_graphics_context` with the versioned native objects borrowed from the current QRhi.
+2. When `frame_available_callback` schedules a frame, call `opennow_streamer_acquire_latest_frame`. The bounded mailbox contains one pending frame: publishing a newer frame releases the stale pending frame. A successful acquisition transfers one retained reference into the opaque token.
+3. After QRhi has opened the frame and provided its native command buffer, but **before** `QQuickRhiItem` calls `beginPass`, call `opennow_streamer_record_frame`. Conversion and synchronization are encoded into that exact command stream. The function never creates, submits, commits, or waits for another command buffer. It returns one producer-owned RGBA8 GPU texture for the selected in-flight slot.
+4. Import and sample that texture inside the item's render pass. Keep the token until QRhi has finished every GPU use of the slot, then call `opennow_streamer_release_frame`. A token records at most once and must be released exactly once.
+5. During scene-graph invalidation, release tokens first and call `opennow_streamer_scene_graph_shutdown` on the bound render thread before QRhi destroys its native objects. Destroy rejects a still-active scene graph instead of dropping GPU state from the wrong thread.
+
+Changing the graphics context clears the one-frame mailbox and advances its epoch. Previously acquired tokens stay releasable but become stale and cannot record against the replacement context. Graphics calls are bound to the thread that installed the active context; a new scene graph can bind a different thread after shutdown.
+
+## Integration boundary
+
+The public C constructor creates the embedded engine used by Qt. GPU producers publish through the platform runtime's `GraphicsFramePublisher`; the FFI owns the mailbox, token epochs, thread checks, and C ownership boundary while platform decoders own native texture creation and same-command-stream conversion.
+
+There is no callback cancellation or timeout. A callback that never returns will eventually backpressure responses and will make destroy wait indefinitely. Dropped unsolicited events are not yet summarized with an overflow event. The header is handwritten and must be validated by each C/C++ consumer's compile-time layout assertions.
diff --git a/native/opennow-streamer/crates/opennow-streamer-ffi/build.rs b/native/opennow-streamer/crates/opennow-streamer-ffi/build.rs
new file mode 100644
index 000000000..65313b1f0
--- /dev/null
+++ b/native/opennow-streamer/crates/opennow-streamer-ffi/build.rs
@@ -0,0 +1,15 @@
+fn main() {
+ match std::env::var("CARGO_CFG_TARGET_OS").as_deref() {
+ Ok("linux") => {
+ println!("cargo:rustc-link-arg=-Wl,-soname,libopennow_streamer_ffi.so");
+ }
+ Ok("macos") => {
+ println!("cargo:rustc-link-arg=-Wl,-install_name,@rpath/libopennow_streamer_ffi.dylib");
+ }
+ Ok("windows") if std::env::var("CARGO_CFG_TARGET_ENV").as_deref() == Ok("msvc") => {
+ println!("cargo:rustc-link-arg=/DELAYLOAD:mfplat.dll");
+ println!("cargo:rustc-link-lib=delayimp");
+ }
+ _ => {}
+ }
+}
diff --git a/native/opennow-streamer/crates/opennow-streamer-ffi/include/opennow_streamer_ffi.h b/native/opennow-streamer/crates/opennow-streamer-ffi/include/opennow_streamer_ffi.h
new file mode 100644
index 000000000..410a9518b
--- /dev/null
+++ b/native/opennow-streamer/crates/opennow-streamer-ffi/include/opennow_streamer_ffi.h
@@ -0,0 +1,215 @@
+#ifndef OPENNOW_STREAMER_FFI_H
+#define OPENNOW_STREAMER_FFI_H
+
+#include
+#include
+#include
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+#define OPENNOW_STREAMER_FFI_ABI_VERSION 2u
+#define OPENNOW_STREAMER_GRAPHICS_CONTEXT_VERSION 1u
+#define OPENNOW_STREAMER_RENDER_COMMAND_VERSION 1u
+
+#define OPENNOW_STREAMER_GRAPHICS_API_D3D11 1u
+#define OPENNOW_STREAMER_GRAPHICS_API_VULKAN 2u
+#define OPENNOW_STREAMER_GRAPHICS_API_METAL 3u
+
+#define OPENNOW_STREAMER_LOCAL_ACTION_GUIDE 1u
+#define OPENNOW_STREAMER_LOCAL_ACTION_SCREENSHOT 2u
+#define OPENNOW_STREAMER_LOCAL_ACTION_RECORDING_TOGGLE 3u
+
+typedef struct OpenNowStreamer OpenNowStreamer;
+typedef struct OpenNowStreamerFrame OpenNowStreamerFrame;
+
+typedef void (*OpenNowStreamerCallback)(
+ const uint8_t *bytes,
+ size_t length,
+ void *user_data);
+
+typedef void (*OpenNowStreamerFrameAvailableCallback)(void *user_data);
+
+typedef struct OpenNowStreamerConfig {
+ uint32_t abi_version;
+ size_t struct_size;
+ size_t command_queue_capacity;
+ size_t response_queue_capacity;
+ size_t event_queue_capacity;
+ size_t max_command_bytes;
+ OpenNowStreamerCallback response_callback;
+ OpenNowStreamerCallback event_callback;
+ OpenNowStreamerFrameAvailableCallback frame_available_callback;
+ OpenNowStreamerCallback cursor_callback;
+ void *user_data;
+} OpenNowStreamerConfig;
+
+/*
+ * Borrowed native objects from one live QRhi. The shell retains ownership.
+ *
+ * D3D11: device is ID3D11Device and queue is its immediate ID3D11DeviceContext.
+ * Vulkan: instance, physical_device, device, queue, and queue_family_index are required.
+ * Metal: device is id and queue is id.
+ */
+typedef struct OpenNowStreamerGraphicsContext {
+ uint32_t version;
+ size_t struct_size;
+ uint32_t graphics_api;
+ void *instance;
+ void *physical_device;
+ void *device;
+ void *queue;
+ uint32_t queue_family_index;
+} OpenNowStreamerGraphicsContext;
+
+/*
+ * One shell-owned in-flight slot and the current QRhi native command buffer.
+ * record_frame writes conversion/synchronization commands into this command stream; it never
+ * creates, commits, submits, or waits for another command buffer.
+ */
+typedef struct OpenNowStreamerRecordCommand {
+ uint32_t version;
+ size_t struct_size;
+ void *command_buffer;
+ uint32_t frame_slot;
+} OpenNowStreamerRecordCommand;
+
+typedef struct OpenNowStreamerFrameInfo {
+ uint32_t width;
+ uint32_t height;
+ uint64_t sequence;
+ uint64_t presentation_time_ns;
+} OpenNowStreamerFrameInfo;
+
+/*
+ * One RGBA8 GPU texture populated by record_frame. On D3D and Metal, resource is the native
+ * texture pointer encoded as uint64_t and resource_view is zero. On Vulkan, resource is VkImage
+ * and resource_view is VkImageView. The producer owns both handles; the frame token retains their
+ * backing slot until release.
+ */
+typedef struct OpenNowStreamerRecordedFrame {
+ uint64_t resource;
+ uint64_t resource_view;
+ uint32_t graphics_api;
+ uint32_t width;
+ uint32_t height;
+ uint32_t frame_slot;
+ uint64_t generation;
+ uint64_t presentation_time_ns;
+} OpenNowStreamerRecordedFrame;
+
+typedef enum OpenNowStreamerStatus {
+ OPENNOW_STREAMER_OK = 0,
+ OPENNOW_STREAMER_NULL_POINTER = 1,
+ OPENNOW_STREAMER_INVALID_CONFIG = 2,
+ OPENNOW_STREAMER_MESSAGE_TOO_LARGE = 3,
+ OPENNOW_STREAMER_QUEUE_FULL = 4,
+ OPENNOW_STREAMER_CLOSED = 5,
+ OPENNOW_STREAMER_NO_FRAME = 6,
+ OPENNOW_STREAMER_GRAPHICS_UNAVAILABLE = 7,
+ OPENNOW_STREAMER_WRONG_THREAD = 8,
+ OPENNOW_STREAMER_STALE_FRAME = 9,
+ OPENNOW_STREAMER_RENDER_FAILED = 10,
+ OPENNOW_STREAMER_SCENE_GRAPH_ACTIVE = 11,
+ OPENNOW_STREAMER_FRAME_ALREADY_RECORDED = 12,
+ OPENNOW_STREAMER_PANIC = 255
+} OpenNowStreamerStatus;
+
+OpenNowStreamerStatus opennow_streamer_create(
+ const OpenNowStreamerConfig *config,
+ OpenNowStreamer **output);
+
+OpenNowStreamerStatus opennow_streamer_send(
+ const OpenNowStreamer *handle,
+ const uint8_t *bytes,
+ size_t length);
+
+OpenNowStreamerStatus opennow_streamer_submit_key(
+ const OpenNowStreamer *handle,
+ uint16_t virtual_key,
+ uint16_t modifiers,
+ bool pressed);
+
+OpenNowStreamerStatus opennow_streamer_submit_mouse_relative(
+ const OpenNowStreamer *handle,
+ int16_t delta_x,
+ int16_t delta_y);
+
+OpenNowStreamerStatus opennow_streamer_submit_mouse_absolute(
+ const OpenNowStreamer *handle,
+ uint16_t x,
+ uint16_t y,
+ uint16_t width,
+ uint16_t height);
+
+OpenNowStreamerStatus opennow_streamer_submit_mouse_button(
+ const OpenNowStreamer *handle,
+ uint8_t button,
+ bool pressed);
+
+OpenNowStreamerStatus opennow_streamer_submit_mouse_wheel(
+ const OpenNowStreamer *handle,
+ int16_t delta_x,
+ int16_t delta_y);
+
+OpenNowStreamerStatus opennow_streamer_submit_gamepad(
+ const OpenNowStreamer *handle,
+ uint8_t controller_id,
+ uint16_t bitmap,
+ uint16_t buttons,
+ uint8_t left_trigger,
+ uint8_t right_trigger,
+ int16_t left_stick_x,
+ int16_t left_stick_y,
+ int16_t right_stick_x,
+ int16_t right_stick_y);
+
+OpenNowStreamerStatus opennow_streamer_submit_local_action(
+ const OpenNowStreamer *handle,
+ uint32_t action);
+
+OpenNowStreamerStatus opennow_streamer_set_capture_active(
+ const OpenNowStreamer *handle,
+ bool active,
+ bool relative_mouse,
+ uintptr_t window_handle,
+ bool *raw_input_active);
+
+OpenNowStreamerStatus opennow_streamer_set_graphics_context(
+ const OpenNowStreamer *handle,
+ const OpenNowStreamerGraphicsContext *context);
+
+OpenNowStreamerStatus opennow_streamer_acquire_latest_frame(
+ const OpenNowStreamer *handle,
+ OpenNowStreamerFrame **output,
+ OpenNowStreamerFrameInfo *info);
+
+/*
+ * Call after QRhi has opened its frame and supplied the native command buffer, but BEFORE
+ * QQuickRhiItem begins its render pass. The conversion may open its own offscreen pass or encode
+ * barriers, which is invalid inside the item's pass. After this returns, import/sample output's
+ * GPU texture in the QQuickRhiItem pass on the same command stream.
+ */
+OpenNowStreamerStatus opennow_streamer_record_frame(
+ const OpenNowStreamer *handle,
+ const OpenNowStreamerFrame *frame,
+ const OpenNowStreamerRecordCommand *command,
+ OpenNowStreamerRecordedFrame *output);
+
+OpenNowStreamerStatus opennow_streamer_release_frame(OpenNowStreamerFrame *frame);
+
+/*
+ * Call from the render thread during sceneGraphInvalidated/releaseResources, after releasing all
+ * frame tokens and before QRhi destroys the native objects from the graphics context.
+ */
+OpenNowStreamerStatus opennow_streamer_scene_graph_shutdown(
+ const OpenNowStreamer *handle);
+
+OpenNowStreamerStatus opennow_streamer_destroy(OpenNowStreamer *handle);
+
+#ifdef __cplusplus
+}
+#endif
+
+#endif
diff --git a/native/opennow-streamer/crates/opennow-streamer-ffi/src/lib.rs b/native/opennow-streamer/crates/opennow-streamer-ffi/src/lib.rs
new file mode 100644
index 000000000..a3ecef9a6
--- /dev/null
+++ b/native/opennow-streamer/crates/opennow-streamer-ffi/src/lib.rs
@@ -0,0 +1,1573 @@
+use std::ffi::c_void;
+use std::panic::{AssertUnwindSafe, catch_unwind};
+use std::ptr;
+use std::sync::Arc;
+use std::sync::mpsc::{Receiver, SyncSender, TrySendError, sync_channel};
+use std::thread::{self, JoinHandle};
+
+use opennow_streamer_core::{Engine, EventSender};
+use opennow_streamer_platform::{
+ CapturedInput, CapturedInputQueue, EmbeddedInputCapture, EmbeddedLocalAction, GraphicsApi,
+ GraphicsContext, GraphicsFramePublisher, GraphicsFrameToken, GraphicsRecordCommand,
+ GraphicsRecordedFrame, GraphicsRuntimeError, RenderThreadGraphics,
+ create_embedded_runtime_with_input,
+};
+use opennow_streamer_protocol::{Command, error};
+use serde_json::Value;
+
+pub const OPENNOW_STREAMER_FFI_ABI_VERSION: u32 = 2;
+const DEFAULT_MAX_COMMAND_BYTES: usize = 1024 * 1024;
+const MAX_QUEUE_CAPACITY: usize = 4096;
+const MAX_COMMAND_BYTES: usize = 16 * 1024 * 1024;
+
+pub type OpenNowStreamerCallback =
+ Option;
+pub type OpenNowStreamerFrameAvailableCallback =
+ Option;
+pub type OpenNowStreamerCursorCallback =
+ Option;
+
+#[repr(C)]
+#[derive(Clone, Copy)]
+pub struct OpenNowStreamerConfig {
+ pub abi_version: u32,
+ pub struct_size: usize,
+ pub command_queue_capacity: usize,
+ pub response_queue_capacity: usize,
+ pub event_queue_capacity: usize,
+ pub max_command_bytes: usize,
+ pub response_callback: OpenNowStreamerCallback,
+ pub event_callback: OpenNowStreamerCallback,
+ pub frame_available_callback: OpenNowStreamerFrameAvailableCallback,
+ pub cursor_callback: OpenNowStreamerCursorCallback,
+ pub user_data: *mut c_void,
+}
+
+#[repr(i32)]
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum OpenNowStreamerStatus {
+ Ok = 0,
+ NullPointer = 1,
+ InvalidConfig = 2,
+ MessageTooLarge = 3,
+ QueueFull = 4,
+ Closed = 5,
+ NoFrame = 6,
+ GraphicsUnavailable = 7,
+ WrongThread = 8,
+ StaleFrame = 9,
+ RenderFailed = 10,
+ SceneGraphActive = 11,
+ FrameAlreadyRecorded = 12,
+ Panic = 255,
+}
+
+pub const OPENNOW_STREAMER_GRAPHICS_CONTEXT_VERSION: u32 = 1;
+pub const OPENNOW_STREAMER_RENDER_COMMAND_VERSION: u32 = 1;
+pub const OPENNOW_STREAMER_GRAPHICS_API_D3D11: u32 = 1;
+pub const OPENNOW_STREAMER_GRAPHICS_API_VULKAN: u32 = 2;
+pub const OPENNOW_STREAMER_GRAPHICS_API_METAL: u32 = 3;
+pub const OPENNOW_STREAMER_LOCAL_ACTION_GUIDE: u32 = 1;
+pub const OPENNOW_STREAMER_LOCAL_ACTION_SCREENSHOT: u32 = 2;
+pub const OPENNOW_STREAMER_LOCAL_ACTION_RECORDING_TOGGLE: u32 = 3;
+
+#[repr(C)]
+#[derive(Debug, Clone, Copy)]
+pub struct OpenNowStreamerGraphicsContext {
+ pub version: u32,
+ pub struct_size: usize,
+ pub graphics_api: u32,
+ pub instance: *mut c_void,
+ pub physical_device: *mut c_void,
+ pub device: *mut c_void,
+ pub queue: *mut c_void,
+ pub queue_family_index: u32,
+}
+
+#[repr(C)]
+#[derive(Debug, Clone, Copy)]
+pub struct OpenNowStreamerRecordCommand {
+ pub version: u32,
+ pub struct_size: usize,
+ pub command_buffer: *mut c_void,
+ pub frame_slot: u32,
+}
+
+#[repr(C)]
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct OpenNowStreamerFrameInfo {
+ pub width: u32,
+ pub height: u32,
+ pub sequence: u64,
+ pub presentation_time_ns: u64,
+}
+
+#[repr(C)]
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct OpenNowStreamerRecordedFrame {
+ pub resource: u64,
+ pub resource_view: u64,
+ pub graphics_api: u32,
+ pub width: u32,
+ pub height: u32,
+ pub frame_slot: u32,
+ pub generation: u64,
+ pub presentation_time_ns: u64,
+}
+
+pub struct OpenNowStreamerFrame {
+ token: GraphicsFrameToken,
+}
+
+#[derive(Clone, Copy)]
+struct Callback {
+ function: OpenNowStreamerCallback,
+ user_data: usize,
+}
+
+#[derive(Clone, Copy)]
+struct FrameAvailableCallback {
+ function: OpenNowStreamerFrameAvailableCallback,
+ user_data: usize,
+}
+
+impl FrameAvailableCallback {
+ fn invoke(self) {
+ let Some(function) = self.function else {
+ return;
+ };
+ unsafe {
+ function(self.user_data as *mut c_void);
+ }
+ }
+}
+
+impl Callback {
+ fn invoke(self, value: &Value) {
+ let Some(function) = self.function else {
+ return;
+ };
+ let Ok(bytes) = serde_json::to_vec(value) else {
+ return;
+ };
+ unsafe {
+ function(bytes.as_ptr(), bytes.len(), self.user_data as *mut c_void);
+ }
+ }
+
+ fn invoke_bytes(self, bytes: &[u8]) {
+ let Some(function) = self.function else {
+ return;
+ };
+ unsafe {
+ function(bytes.as_ptr(), bytes.len(), self.user_data as *mut c_void);
+ }
+ }
+}
+
+enum WorkerCommand {
+ Send(Vec),
+ Destroy,
+}
+
+pub struct OpenNowStreamer {
+ commands: Option>,
+ worker: Option>,
+ response_dispatcher: Option>,
+ event_dispatcher: Option>,
+ cursor_dispatcher: Option>,
+ max_command_bytes: usize,
+ graphics: RenderThreadGraphics,
+ frame_publisher: GraphicsFramePublisher,
+ input: EmbeddedInputCapture,
+}
+
+impl OpenNowStreamer {
+ fn create(
+ config: OpenNowStreamerConfig,
+ captured_input: Arc,
+ engine_factory: impl FnOnce(EventSender, GraphicsFramePublisher, SyncSender>) -> Engine
+ + Send
+ + 'static,
+ exit_hook: impl FnOnce() + Send + 'static,
+ ) -> Result {
+ validate_config(&config)?;
+ let callback = |function| Callback {
+ function,
+ user_data: config.user_data as usize,
+ };
+ let frame_available = FrameAvailableCallback {
+ function: config.frame_available_callback,
+ user_data: config.user_data as usize,
+ };
+ let (graphics, frame_publisher) =
+ RenderThreadGraphics::new(move || frame_available.invoke());
+ let engine_frame_publisher = frame_publisher.clone();
+ let (commands, command_receiver) = sync_channel(config.command_queue_capacity);
+ let (responses, response_receiver) = sync_channel(config.response_queue_capacity);
+ let (events, event_receiver) = sync_channel(config.event_queue_capacity);
+ let (cursor_updates, cursor_receiver) = sync_channel(config.event_queue_capacity);
+ let response_dispatcher = spawn_dispatcher(
+ "opennow-ffi-responses",
+ response_receiver,
+ callback(config.response_callback),
+ )?;
+ let event_dispatcher = match spawn_dispatcher(
+ "opennow-ffi-events",
+ event_receiver,
+ callback(config.event_callback),
+ ) {
+ Ok(dispatcher) => dispatcher,
+ Err(status) => {
+ drop(responses);
+ let _ = response_dispatcher.join();
+ return Err(status);
+ }
+ };
+ let cursor_dispatcher =
+ match spawn_cursor_dispatcher(cursor_receiver, callback(config.cursor_callback)) {
+ Ok(dispatcher) => dispatcher,
+ Err(status) => {
+ drop(responses);
+ drop(events);
+ let _ = response_dispatcher.join();
+ let _ = event_dispatcher.join();
+ return Err(status);
+ }
+ };
+ let worker = match thread::Builder::new()
+ .name("opennow-ffi-engine".to_owned())
+ .spawn(move || {
+ let _ = catch_unwind(AssertUnwindSafe(|| {
+ run_engine(
+ command_receiver,
+ responses,
+ EventSender::bounded(events),
+ move |events| {
+ engine_factory(events, engine_frame_publisher, cursor_updates)
+ },
+ );
+ }));
+ exit_hook();
+ }) {
+ Ok(worker) => worker,
+ Err(_) => {
+ drop(commands);
+ let _ = response_dispatcher.join();
+ let _ = event_dispatcher.join();
+ let _ = cursor_dispatcher.join();
+ return Err(OpenNowStreamerStatus::Closed);
+ }
+ };
+ Ok(Self {
+ commands: Some(commands),
+ worker: Some(worker),
+ response_dispatcher: Some(response_dispatcher),
+ event_dispatcher: Some(event_dispatcher),
+ cursor_dispatcher: Some(cursor_dispatcher),
+ max_command_bytes: if config.max_command_bytes == 0 {
+ DEFAULT_MAX_COMMAND_BYTES
+ } else {
+ config.max_command_bytes
+ },
+ graphics,
+ frame_publisher,
+ input: EmbeddedInputCapture::new(captured_input),
+ })
+ }
+
+ fn send(&self, bytes: &[u8]) -> OpenNowStreamerStatus {
+ if bytes.len() > self.max_command_bytes {
+ return OpenNowStreamerStatus::MessageTooLarge;
+ }
+ let Some(commands) = self.commands.as_ref() else {
+ return OpenNowStreamerStatus::Closed;
+ };
+ match commands.try_send(WorkerCommand::Send(bytes.to_vec())) {
+ Ok(()) => OpenNowStreamerStatus::Ok,
+ Err(TrySendError::Full(_)) => OpenNowStreamerStatus::QueueFull,
+ Err(TrySendError::Disconnected(_)) => OpenNowStreamerStatus::Closed,
+ }
+ }
+
+ fn shutdown(&mut self) {
+ if let Some(commands) = self.commands.take() {
+ let _ = commands.send(WorkerCommand::Destroy);
+ }
+ if let Some(worker) = self.worker.take() {
+ let _ = worker.join();
+ }
+ if let Some(dispatcher) = self.response_dispatcher.take() {
+ let _ = dispatcher.join();
+ }
+ if let Some(dispatcher) = self.event_dispatcher.take() {
+ let _ = dispatcher.join();
+ }
+ if let Some(dispatcher) = self.cursor_dispatcher.take() {
+ let _ = dispatcher.join();
+ }
+ }
+
+ #[cfg(test)]
+ fn frame_publisher(&self) -> GraphicsFramePublisher {
+ self.frame_publisher.clone()
+ }
+}
+
+impl Drop for OpenNowStreamer {
+ fn drop(&mut self) {
+ self.shutdown();
+ }
+}
+
+fn validate_config(config: &OpenNowStreamerConfig) -> Result<(), OpenNowStreamerStatus> {
+ let valid_capacity = |capacity| (1..=MAX_QUEUE_CAPACITY).contains(&capacity);
+ if config.abi_version != OPENNOW_STREAMER_FFI_ABI_VERSION
+ || config.struct_size < size_of::()
+ || !valid_capacity(config.command_queue_capacity)
+ || !valid_capacity(config.response_queue_capacity)
+ || !valid_capacity(config.event_queue_capacity)
+ || config.max_command_bytes > MAX_COMMAND_BYTES
+ || config.response_callback.is_none()
+ {
+ return Err(OpenNowStreamerStatus::InvalidConfig);
+ }
+ Ok(())
+}
+
+fn spawn_dispatcher(
+ name: &str,
+ receiver: Receiver,
+ callback: Callback,
+) -> Result, OpenNowStreamerStatus> {
+ thread::Builder::new()
+ .name(name.to_owned())
+ .spawn(move || {
+ while let Ok(value) = receiver.recv() {
+ callback.invoke(&value);
+ }
+ })
+ .map_err(|_| OpenNowStreamerStatus::Closed)
+}
+
+fn spawn_cursor_dispatcher(
+ receiver: Receiver>,
+ callback: Callback,
+) -> Result, OpenNowStreamerStatus> {
+ thread::Builder::new()
+ .name("opennow-ffi-cursor".to_owned())
+ .spawn(move || {
+ while let Ok(bytes) = receiver.recv() {
+ callback.invoke_bytes(&bytes);
+ }
+ })
+ .map_err(|_| OpenNowStreamerStatus::Closed)
+}
+
+fn run_engine(
+ commands: Receiver,
+ responses: SyncSender,
+ events: EventSender,
+ engine_factory: impl FnOnce(EventSender) -> Engine,
+) {
+ let mut engine = engine_factory(events);
+ while let Ok(command) = commands.recv() {
+ let bytes = match command {
+ WorkerCommand::Send(bytes) => bytes,
+ WorkerCommand::Destroy => break,
+ };
+ let command: Command = match serde_json::from_slice(&bytes) {
+ Ok(command) => command,
+ Err(parse_error) => {
+ if responses
+ .send(error(None, "invalid-command", parse_error.to_string()))
+ .is_err()
+ {
+ break;
+ }
+ continue;
+ }
+ };
+ let (messages, keep_running) = engine.handle(command);
+ for message in messages {
+ if responses.send(message).is_err() {
+ return;
+ }
+ }
+ if !keep_running {
+ break;
+ }
+ }
+}
+
+fn graphics_context(
+ context: OpenNowStreamerGraphicsContext,
+) -> Result {
+ if context.version != OPENNOW_STREAMER_GRAPHICS_CONTEXT_VERSION
+ || context.struct_size < size_of::()
+ {
+ return Err(OpenNowStreamerStatus::InvalidConfig);
+ }
+ let api = match context.graphics_api {
+ OPENNOW_STREAMER_GRAPHICS_API_D3D11 => GraphicsApi::D3d11,
+ OPENNOW_STREAMER_GRAPHICS_API_VULKAN => GraphicsApi::Vulkan,
+ OPENNOW_STREAMER_GRAPHICS_API_METAL => GraphicsApi::Metal,
+ _ => return Err(OpenNowStreamerStatus::InvalidConfig),
+ };
+ Ok(GraphicsContext {
+ api,
+ instance: context.instance as usize,
+ physical_device: context.physical_device as usize,
+ device: context.device as usize,
+ queue: context.queue as usize,
+ queue_family_index: context.queue_family_index,
+ })
+}
+
+fn render_command(
+ command: OpenNowStreamerRecordCommand,
+) -> Result {
+ if command.version != OPENNOW_STREAMER_RENDER_COMMAND_VERSION
+ || command.struct_size < size_of::()
+ {
+ return Err(OpenNowStreamerStatus::InvalidConfig);
+ }
+ Ok(GraphicsRecordCommand {
+ command_buffer: command.command_buffer as usize,
+ frame_slot: command.frame_slot,
+ })
+}
+
+fn recorded_frame(
+ api: GraphicsApi,
+ frame: GraphicsRecordedFrame,
+) -> Result {
+ if frame.resource == 0 || frame.width == 0 || frame.height == 0 {
+ return Err(OpenNowStreamerStatus::RenderFailed);
+ }
+ Ok(OpenNowStreamerRecordedFrame {
+ resource: frame.resource,
+ resource_view: frame.resource_view,
+ graphics_api: match api {
+ GraphicsApi::D3d11 => OPENNOW_STREAMER_GRAPHICS_API_D3D11,
+ GraphicsApi::Vulkan => OPENNOW_STREAMER_GRAPHICS_API_VULKAN,
+ GraphicsApi::Metal => OPENNOW_STREAMER_GRAPHICS_API_METAL,
+ },
+ width: frame.width,
+ height: frame.height,
+ frame_slot: frame.frame_slot,
+ generation: frame.generation,
+ presentation_time_ns: frame.presentation_time_ns,
+ })
+}
+
+fn graphics_status(error: GraphicsRuntimeError) -> OpenNowStreamerStatus {
+ match error {
+ GraphicsRuntimeError::InvalidContext(_) | GraphicsRuntimeError::InvalidRenderCommand(_) => {
+ OpenNowStreamerStatus::InvalidConfig
+ }
+ GraphicsRuntimeError::SceneGraphInactive => OpenNowStreamerStatus::GraphicsUnavailable,
+ GraphicsRuntimeError::WrongThread => OpenNowStreamerStatus::WrongThread,
+ GraphicsRuntimeError::NoFrame => OpenNowStreamerStatus::NoFrame,
+ GraphicsRuntimeError::StaleFrame => OpenNowStreamerStatus::StaleFrame,
+ GraphicsRuntimeError::FrameAlreadyRecorded => OpenNowStreamerStatus::FrameAlreadyRecorded,
+ GraphicsRuntimeError::RecordFailed(_) => OpenNowStreamerStatus::RenderFailed,
+ }
+}
+
+fn ffi_status(body: impl FnOnce() -> OpenNowStreamerStatus) -> OpenNowStreamerStatus {
+ catch_unwind(AssertUnwindSafe(body)).unwrap_or(OpenNowStreamerStatus::Panic)
+}
+
+#[unsafe(no_mangle)]
+/// Creates one engine handle owned by the caller.
+///
+/// # Safety
+///
+/// `config` must point to a readable configuration whose first `struct_size` bytes remain valid
+/// for this call. `output` must point to writable storage for one handle pointer. Callback pointers
+/// and `user_data` must remain valid until destroy returns.
+pub unsafe extern "C" fn opennow_streamer_create(
+ config: *const OpenNowStreamerConfig,
+ output: *mut *mut OpenNowStreamer,
+) -> OpenNowStreamerStatus {
+ match catch_unwind(AssertUnwindSafe(|| {
+ if config.is_null() || output.is_null() {
+ return OpenNowStreamerStatus::NullPointer;
+ }
+ unsafe {
+ output.write(ptr::null_mut());
+ }
+ let abi_version = unsafe { ptr::addr_of!((*config).abi_version).read() };
+ let struct_size = unsafe { ptr::addr_of!((*config).struct_size).read() };
+ if abi_version != OPENNOW_STREAMER_FFI_ABI_VERSION
+ || struct_size < size_of::()
+ {
+ return OpenNowStreamerStatus::InvalidConfig;
+ }
+ let config = unsafe { config.read() };
+ let captured_input = Arc::new(CapturedInputQueue::default());
+ let runtime_input = Arc::clone(&captured_input);
+ match OpenNowStreamer::create(
+ config,
+ captured_input,
+ move |events, frames, cursor_updates| {
+ let cursor_update = Arc::new(move |bytes: Vec| {
+ let _ = cursor_updates.try_send(bytes);
+ });
+ Engine::with_embedded_media_runtime(
+ events,
+ create_embedded_runtime_with_input(frames, runtime_input, Some(cursor_update)),
+ )
+ },
+ || {},
+ ) {
+ Ok(handle) => {
+ unsafe {
+ output.write(Box::into_raw(Box::new(handle)));
+ }
+ OpenNowStreamerStatus::Ok
+ }
+ Err(status) => status,
+ }
+ })) {
+ Ok(status) => status,
+ Err(_) => OpenNowStreamerStatus::Panic,
+ }
+}
+
+#[unsafe(no_mangle)]
+/// Submits one keyboard transition to the embedded input queue.
+///
+/// # Safety
+///
+/// `handle` must be null or point to a live engine handle that is not being destroyed.
+pub unsafe extern "C" fn opennow_streamer_submit_key(
+ handle: *const OpenNowStreamer,
+ virtual_key: u16,
+ modifiers: u16,
+ pressed: bool,
+) -> OpenNowStreamerStatus {
+ ffi_status(|| {
+ let Some(handle) = (unsafe { handle.as_ref() }) else {
+ return OpenNowStreamerStatus::NullPointer;
+ };
+ handle.input.submit(CapturedInput::Key {
+ virtual_key,
+ modifiers,
+ pressed,
+ });
+ OpenNowStreamerStatus::Ok
+ })
+}
+
+#[unsafe(no_mangle)]
+/// Submits one relative mouse movement to the embedded input queue.
+///
+/// # Safety
+///
+/// `handle` must be null or point to a live engine handle that is not being destroyed.
+pub unsafe extern "C" fn opennow_streamer_submit_mouse_relative(
+ handle: *const OpenNowStreamer,
+ delta_x: i16,
+ delta_y: i16,
+) -> OpenNowStreamerStatus {
+ ffi_status(|| {
+ let Some(handle) = (unsafe { handle.as_ref() }) else {
+ return OpenNowStreamerStatus::NullPointer;
+ };
+ handle
+ .input
+ .submit(CapturedInput::MouseMove { delta_x, delta_y });
+ OpenNowStreamerStatus::Ok
+ })
+}
+
+#[unsafe(no_mangle)]
+/// Submits one absolute mouse position to the embedded input queue.
+///
+/// # Safety
+///
+/// `handle` must be null or point to a live engine handle that is not being destroyed.
+pub unsafe extern "C" fn opennow_streamer_submit_mouse_absolute(
+ handle: *const OpenNowStreamer,
+ x: u16,
+ y: u16,
+ width: u16,
+ height: u16,
+) -> OpenNowStreamerStatus {
+ ffi_status(|| {
+ let Some(handle) = (unsafe { handle.as_ref() }) else {
+ return OpenNowStreamerStatus::NullPointer;
+ };
+ if width == 0 || height == 0 {
+ return OpenNowStreamerStatus::InvalidConfig;
+ }
+ handle.input.submit(CapturedInput::MouseAbsolute {
+ x,
+ y,
+ width,
+ height,
+ });
+ OpenNowStreamerStatus::Ok
+ })
+}
+
+#[unsafe(no_mangle)]
+/// Submits one mouse-button transition to the embedded input queue.
+///
+/// # Safety
+///
+/// `handle` must be null or point to a live engine handle that is not being destroyed.
+pub unsafe extern "C" fn opennow_streamer_submit_mouse_button(
+ handle: *const OpenNowStreamer,
+ button: u8,
+ pressed: bool,
+) -> OpenNowStreamerStatus {
+ ffi_status(|| {
+ let Some(handle) = (unsafe { handle.as_ref() }) else {
+ return OpenNowStreamerStatus::NullPointer;
+ };
+ if !(1..=5).contains(&button) {
+ return OpenNowStreamerStatus::InvalidConfig;
+ }
+ handle
+ .input
+ .submit(CapturedInput::MouseButton { button, pressed });
+ OpenNowStreamerStatus::Ok
+ })
+}
+
+#[unsafe(no_mangle)]
+/// Submits one mouse-wheel movement to the embedded input queue.
+///
+/// # Safety
+///
+/// `handle` must be null or point to a live engine handle that is not being destroyed.
+pub unsafe extern "C" fn opennow_streamer_submit_mouse_wheel(
+ handle: *const OpenNowStreamer,
+ delta_x: i16,
+ delta_y: i16,
+) -> OpenNowStreamerStatus {
+ ffi_status(|| {
+ let Some(handle) = (unsafe { handle.as_ref() }) else {
+ return OpenNowStreamerStatus::NullPointer;
+ };
+ handle
+ .input
+ .submit(CapturedInput::MouseWheel { delta_x, delta_y });
+ OpenNowStreamerStatus::Ok
+ })
+}
+
+#[unsafe(no_mangle)]
+/// Submits one gamepad state snapshot to the embedded input queue.
+///
+/// # Safety
+///
+/// `handle` must be null or point to a live engine handle that is not being destroyed.
+pub unsafe extern "C" fn opennow_streamer_submit_gamepad(
+ handle: *const OpenNowStreamer,
+ controller_id: u8,
+ bitmap: u16,
+ buttons: u16,
+ left_trigger: u8,
+ right_trigger: u8,
+ left_stick_x: i16,
+ left_stick_y: i16,
+ right_stick_x: i16,
+ right_stick_y: i16,
+) -> OpenNowStreamerStatus {
+ ffi_status(|| {
+ let Some(handle) = (unsafe { handle.as_ref() }) else {
+ return OpenNowStreamerStatus::NullPointer;
+ };
+ if controller_id >= 4 {
+ return OpenNowStreamerStatus::InvalidConfig;
+ }
+ handle.input.submit(CapturedInput::Gamepad {
+ controller_id,
+ bitmap,
+ buttons,
+ left_trigger,
+ right_trigger,
+ left_stick_x,
+ left_stick_y,
+ right_stick_x,
+ right_stick_y,
+ });
+ OpenNowStreamerStatus::Ok
+ })
+}
+
+#[unsafe(no_mangle)]
+/// Submits one shell-local action to the embedded input queue.
+///
+/// # Safety
+///
+/// `handle` must be null or point to a live engine handle that is not being destroyed.
+pub unsafe extern "C" fn opennow_streamer_submit_local_action(
+ handle: *const OpenNowStreamer,
+ action: u32,
+) -> OpenNowStreamerStatus {
+ ffi_status(|| {
+ let Some(handle) = (unsafe { handle.as_ref() }) else {
+ return OpenNowStreamerStatus::NullPointer;
+ };
+ let action = match action {
+ OPENNOW_STREAMER_LOCAL_ACTION_GUIDE => EmbeddedLocalAction::Guide,
+ OPENNOW_STREAMER_LOCAL_ACTION_SCREENSHOT => EmbeddedLocalAction::Screenshot,
+ OPENNOW_STREAMER_LOCAL_ACTION_RECORDING_TOGGLE => EmbeddedLocalAction::RecordingToggle,
+ _ => return OpenNowStreamerStatus::InvalidConfig,
+ };
+ handle.input.submit_local_action(action);
+ OpenNowStreamerStatus::Ok
+ })
+}
+
+#[unsafe(no_mangle)]
+/// Changes native input capture state for the Qt window.
+///
+/// # Safety
+///
+/// `handle` must be null or point to a live engine handle that is not being destroyed.
+/// `raw_input_active` must be null or point to writable storage for one `bool`.
+pub unsafe extern "C" fn opennow_streamer_set_capture_active(
+ handle: *const OpenNowStreamer,
+ active: bool,
+ relative_mouse: bool,
+ window_handle: usize,
+ raw_input_active: *mut bool,
+) -> OpenNowStreamerStatus {
+ ffi_status(|| {
+ if handle.is_null() || raw_input_active.is_null() {
+ return OpenNowStreamerStatus::NullPointer;
+ }
+ let raw = unsafe { &*handle }
+ .input
+ .set_active(active, relative_mouse, window_handle);
+ unsafe {
+ raw_input_active.write(raw);
+ }
+ OpenNowStreamerStatus::Ok
+ })
+}
+
+#[unsafe(no_mangle)]
+/// Binds the caller's live graphics objects to the current scene-graph render thread.
+///
+/// Repeating the call with an identical context is idempotent. A changed context invalidates any
+/// queued or acquired frame from the previous context.
+///
+/// # Safety
+///
+/// `handle` must be live and must not race with destroy. `context` must point to a readable
+/// versioned context. Its native graphics objects must remain live until scene-graph shutdown,
+/// and this call must run on the thread that will acquire and record frames.
+pub unsafe extern "C" fn opennow_streamer_set_graphics_context(
+ handle: *const OpenNowStreamer,
+ context: *const OpenNowStreamerGraphicsContext,
+) -> OpenNowStreamerStatus {
+ match catch_unwind(AssertUnwindSafe(|| {
+ if handle.is_null() || context.is_null() {
+ return OpenNowStreamerStatus::NullPointer;
+ }
+ let version = unsafe { ptr::addr_of!((*context).version).read() };
+ let struct_size = unsafe { ptr::addr_of!((*context).struct_size).read() };
+ if version != OPENNOW_STREAMER_GRAPHICS_CONTEXT_VERSION
+ || struct_size < size_of::()
+ {
+ return OpenNowStreamerStatus::InvalidConfig;
+ }
+ let context = match graphics_context(unsafe { context.read() }) {
+ Ok(context) => context,
+ Err(status) => return status,
+ };
+ unsafe { &*handle }
+ .graphics
+ .initialize(context)
+ .map_or_else(graphics_status, |()| OpenNowStreamerStatus::Ok)
+ })) {
+ Ok(status) => status,
+ Err(_) => OpenNowStreamerStatus::Panic,
+ }
+}
+
+#[unsafe(no_mangle)]
+/// Acquires the newest pending GPU frame and transfers one retained reference to the caller.
+///
+/// # Safety
+///
+/// `handle` must be live. `output` and `info` must point to writable storage. The call must run on
+/// the bound render thread. A successful token must be released exactly once.
+pub unsafe extern "C" fn opennow_streamer_acquire_latest_frame(
+ handle: *const OpenNowStreamer,
+ output: *mut *mut OpenNowStreamerFrame,
+ info: *mut OpenNowStreamerFrameInfo,
+) -> OpenNowStreamerStatus {
+ match catch_unwind(AssertUnwindSafe(|| {
+ if handle.is_null() || output.is_null() || info.is_null() {
+ return OpenNowStreamerStatus::NullPointer;
+ }
+ unsafe {
+ output.write(ptr::null_mut());
+ info.write(OpenNowStreamerFrameInfo::default());
+ }
+ let token = match unsafe { &*handle }.graphics.acquire_latest() {
+ Ok(token) => token,
+ Err(error) => return graphics_status(error),
+ };
+ let frame_info = token.info();
+ unsafe {
+ info.write(OpenNowStreamerFrameInfo {
+ width: frame_info.width,
+ height: frame_info.height,
+ sequence: frame_info.sequence,
+ presentation_time_ns: frame_info.presentation_time_ns,
+ });
+ output.write(Box::into_raw(Box::new(OpenNowStreamerFrame { token })));
+ }
+ OpenNowStreamerStatus::Ok
+ })) {
+ Ok(status) => status,
+ Err(_) => OpenNowStreamerStatus::Panic,
+ }
+}
+
+#[unsafe(no_mangle)]
+/// Records the acquired frame into the caller's current GPU command stream.
+///
+/// # Safety
+///
+/// `handle` and `frame` must be live and belong to the same streamer. `command` must describe a
+/// writable command buffer owned by the bound graphics context. This call must happen before the
+/// QQuickRhiItem render pass begins. `output` receives textures that can then be sampled inside the
+/// item pass. All native objects must remain valid for the call, which must run on the bound render
+/// thread.
+pub unsafe extern "C" fn opennow_streamer_record_frame(
+ handle: *const OpenNowStreamer,
+ frame: *const OpenNowStreamerFrame,
+ command: *const OpenNowStreamerRecordCommand,
+ output: *mut OpenNowStreamerRecordedFrame,
+) -> OpenNowStreamerStatus {
+ match catch_unwind(AssertUnwindSafe(|| {
+ if handle.is_null() || frame.is_null() || command.is_null() || output.is_null() {
+ return OpenNowStreamerStatus::NullPointer;
+ }
+ unsafe {
+ output.write(OpenNowStreamerRecordedFrame::default());
+ }
+ let version = unsafe { ptr::addr_of!((*command).version).read() };
+ let struct_size = unsafe { ptr::addr_of!((*command).struct_size).read() };
+ if version != OPENNOW_STREAMER_RENDER_COMMAND_VERSION
+ || struct_size < size_of::()
+ {
+ return OpenNowStreamerStatus::InvalidConfig;
+ }
+ let command = match render_command(unsafe { command.read() }) {
+ Ok(command) => command,
+ Err(status) => return status,
+ };
+ let handle = unsafe { &*handle };
+ let recorded = match handle.graphics.record(&unsafe { &*frame }.token, command) {
+ Ok(recorded) => recorded,
+ Err(error) => return graphics_status(error),
+ };
+ let api = match handle.frame_publisher.context() {
+ Some(lease) => lease.context().api,
+ None => return OpenNowStreamerStatus::GraphicsUnavailable,
+ };
+ let recorded = match recorded_frame(api, recorded) {
+ Ok(recorded) => recorded,
+ Err(status) => return status,
+ };
+ unsafe {
+ output.write(recorded);
+ }
+ OpenNowStreamerStatus::Ok
+ })) {
+ Ok(status) => status,
+ Err(_) => OpenNowStreamerStatus::Panic,
+ }
+}
+
+#[unsafe(no_mangle)]
+/// Releases one acquired frame token.
+///
+/// # Safety
+///
+/// `frame` must be a live token returned by acquire and must be released exactly once. Release may
+/// run on any thread and remains valid after scene-graph shutdown.
+pub unsafe extern "C" fn opennow_streamer_release_frame(
+ frame: *mut OpenNowStreamerFrame,
+) -> OpenNowStreamerStatus {
+ match catch_unwind(AssertUnwindSafe(|| {
+ if frame.is_null() {
+ return OpenNowStreamerStatus::NullPointer;
+ }
+ drop(unsafe { Box::from_raw(frame) });
+ OpenNowStreamerStatus::Ok
+ })) {
+ Ok(status) => status,
+ Err(_) => OpenNowStreamerStatus::Panic,
+ }
+}
+
+#[unsafe(no_mangle)]
+/// Invalidates queued frames and releases all scene-graph-owned graphics state.
+///
+/// Acquired tokens remain releasable but cannot be recorded. The handle can later bind a new
+/// context, including from a replacement render thread.
+///
+/// # Safety
+///
+/// `handle` must be live and the call must run on the currently bound render thread. It must not
+/// race with another graphics operation or destroy.
+pub unsafe extern "C" fn opennow_streamer_scene_graph_shutdown(
+ handle: *const OpenNowStreamer,
+) -> OpenNowStreamerStatus {
+ match catch_unwind(AssertUnwindSafe(|| {
+ if handle.is_null() {
+ return OpenNowStreamerStatus::NullPointer;
+ }
+ unsafe { &*handle }
+ .graphics
+ .shutdown()
+ .map_or_else(graphics_status, |()| OpenNowStreamerStatus::Ok)
+ })) {
+ Ok(status) => status,
+ Err(_) => OpenNowStreamerStatus::Panic,
+ }
+}
+
+#[unsafe(no_mangle)]
+/// Copies and queues one serialized protocol command.
+///
+/// # Safety
+///
+/// `handle` must be a live handle returned by create and must not race with destroy. When `length`
+/// is nonzero, `bytes` must identify a readable allocation of at least `length` bytes.
+pub unsafe extern "C" fn opennow_streamer_send(
+ handle: *const OpenNowStreamer,
+ bytes: *const u8,
+ length: usize,
+) -> OpenNowStreamerStatus {
+ match catch_unwind(AssertUnwindSafe(|| {
+ if handle.is_null() || (bytes.is_null() && length != 0) {
+ return OpenNowStreamerStatus::NullPointer;
+ }
+ let handle = unsafe { &*handle };
+ let bytes = if length == 0 {
+ &[]
+ } else {
+ unsafe { std::slice::from_raw_parts(bytes, length) }
+ };
+ handle.send(bytes)
+ })) {
+ Ok(status) => status,
+ Err(_) => OpenNowStreamerStatus::Panic,
+ }
+}
+
+#[unsafe(no_mangle)]
+/// Stops the engine, drains callbacks, and consumes the handle.
+///
+/// # Safety
+///
+/// `handle` must be a live handle returned by create. It must be passed to destroy exactly once,
+/// with no concurrent or subsequent access through the pointer.
+pub unsafe extern "C" fn opennow_streamer_destroy(
+ handle: *mut OpenNowStreamer,
+) -> OpenNowStreamerStatus {
+ match catch_unwind(AssertUnwindSafe(|| {
+ if handle.is_null() {
+ return OpenNowStreamerStatus::NullPointer;
+ }
+ if unsafe { &*handle }.graphics.is_active() {
+ return OpenNowStreamerStatus::SceneGraphActive;
+ }
+ let mut handle = unsafe { Box::from_raw(handle) };
+ handle.shutdown();
+ OpenNowStreamerStatus::Ok
+ })) {
+ Ok(status) => status,
+ Err(_) => OpenNowStreamerStatus::Panic,
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use std::sync::Arc;
+ use std::sync::atomic::{AtomicUsize, Ordering};
+ use std::sync::{Condvar, Mutex};
+ use std::time::{Duration, Instant};
+
+ use opennow_streamer_platform::{GraphicsFrame, GraphicsFrameInfo, create_test_runtime};
+ use serde_json::json;
+
+ use super::*;
+
+ #[derive(Default)]
+ struct CallbackMessages {
+ values: Mutex>,
+ changed: Condvar,
+ frames_available: AtomicUsize,
+ }
+
+ impl CallbackMessages {
+ fn wait_for_id(&self, id: &str) -> Value {
+ let deadline = Instant::now() + Duration::from_secs(5);
+ let mut values = self.values.lock().expect("callback values");
+ loop {
+ if let Some(value) = values.iter().find(|value| value["id"] == id) {
+ return value.clone();
+ }
+ let timeout = deadline.saturating_duration_since(Instant::now());
+ assert!(!timeout.is_zero(), "timed out waiting for response {id}");
+ let (next, result) = self
+ .changed
+ .wait_timeout(values, timeout)
+ .expect("callback wait");
+ values = next;
+ assert!(!result.timed_out(), "timed out waiting for response {id}");
+ }
+ }
+ }
+
+ unsafe extern "C" fn collect_response(bytes: *const u8, length: usize, user_data: *mut c_void) {
+ let bytes = unsafe { std::slice::from_raw_parts(bytes, length) };
+ let Ok(value) = serde_json::from_slice(bytes) else {
+ return;
+ };
+ let messages = unsafe { &*(user_data as *const CallbackMessages) };
+ messages.values.lock().expect("callback values").push(value);
+ messages.changed.notify_all();
+ }
+
+ unsafe extern "C" fn collect_frame_available(user_data: *mut c_void) {
+ let messages = unsafe { &*(user_data as *const CallbackMessages) };
+ messages.frames_available.fetch_add(1, Ordering::Relaxed);
+ }
+
+ fn test_config(messages: &CallbackMessages) -> OpenNowStreamerConfig {
+ OpenNowStreamerConfig {
+ abi_version: OPENNOW_STREAMER_FFI_ABI_VERSION,
+ struct_size: size_of::(),
+ command_queue_capacity: 4,
+ response_queue_capacity: 4,
+ event_queue_capacity: 4,
+ max_command_bytes: 4096,
+ response_callback: Some(collect_response),
+ event_callback: None,
+ frame_available_callback: Some(collect_frame_available),
+ cursor_callback: None,
+ user_data: ptr::from_ref(messages).cast_mut().cast(),
+ }
+ }
+
+ fn create_with_test_runtime(
+ messages: &CallbackMessages,
+ ) -> (
+ OpenNowStreamer,
+ opennow_streamer_platform::TestMediaRuntimeHost,
+ ) {
+ let (host, runtime) = create_test_runtime();
+ let shutdown_runtime = runtime.clone();
+ let handle = OpenNowStreamer::create(
+ test_config(messages),
+ runtime.captured_input(),
+ move |events, _frames, _cursor| Engine::with_embedded_media_runtime(events, runtime),
+ move || shutdown_runtime.shutdown(),
+ )
+ .expect("FFI handle");
+ (handle, host)
+ }
+
+ #[derive(Default)]
+ struct RecordedCommands {
+ values: Mutex>,
+ drops: AtomicUsize,
+ }
+
+ struct TestGraphicsFrame {
+ sequence: u64,
+ recorded: Arc,
+ panic_on_record: bool,
+ }
+
+ impl GraphicsFrame for TestGraphicsFrame {
+ fn info(&self) -> GraphicsFrameInfo {
+ GraphicsFrameInfo {
+ width: 2560,
+ height: 1440,
+ sequence: self.sequence,
+ presentation_time_ns: 8_333_333 * self.sequence,
+ }
+ }
+
+ fn record(
+ &self,
+ context: GraphicsContext,
+ command: GraphicsRecordCommand,
+ ) -> Result {
+ assert!(!self.panic_on_record, "injected render panic");
+ self.recorded
+ .values
+ .lock()
+ .expect("recorded commands")
+ .push((context, command));
+ Ok(GraphicsRecordedFrame {
+ resource: 0xfeed,
+ resource_view: 0xbeef,
+ width: 2560,
+ height: 1440,
+ frame_slot: command.frame_slot,
+ generation: self.sequence,
+ presentation_time_ns: 8_333_333 * self.sequence,
+ })
+ }
+ }
+
+ impl Drop for TestGraphicsFrame {
+ fn drop(&mut self) {
+ self.recorded.drops.fetch_add(1, Ordering::Relaxed);
+ }
+ }
+
+ fn ffi_graphics_context() -> OpenNowStreamerGraphicsContext {
+ OpenNowStreamerGraphicsContext {
+ version: OPENNOW_STREAMER_GRAPHICS_CONTEXT_VERSION,
+ struct_size: size_of::(),
+ graphics_api: OPENNOW_STREAMER_GRAPHICS_API_VULKAN,
+ instance: ptr::dangling_mut(),
+ physical_device: ptr::dangling_mut(),
+ device: ptr::dangling_mut(),
+ queue: ptr::dangling_mut(),
+ queue_family_index: 5,
+ }
+ }
+
+ fn ffi_render_command() -> OpenNowStreamerRecordCommand {
+ OpenNowStreamerRecordCommand {
+ version: OPENNOW_STREAMER_RENDER_COMMAND_VERSION,
+ struct_size: size_of::(),
+ command_buffer: ptr::dangling_mut(),
+ frame_slot: 2,
+ }
+ }
+
+ fn graphics_test_handle(messages: &CallbackMessages) -> OpenNowStreamer {
+ OpenNowStreamer::create(
+ test_config(messages),
+ Arc::new(CapturedInputQueue::default()),
+ |events, _frames, _cursor| Engine::embedded(events),
+ || {},
+ )
+ .expect("FFI handle")
+ }
+
+ #[test]
+ fn typed_submit_api_routes_only_captured_events_to_the_rust_queue() {
+ let messages = Box::new(CallbackMessages::default());
+ let mut handle = graphics_test_handle(&messages);
+ handle.input.set_active(true, false, 0);
+
+ assert_eq!(
+ unsafe { opennow_streamer_submit_key(&handle, 0x57, 0x02, true) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(
+ unsafe { opennow_streamer_submit_mouse_relative(&handle, -12, 34) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(
+ unsafe { opennow_streamer_submit_mouse_absolute(&handle, 20, 30, 640, 360) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(
+ unsafe { opennow_streamer_submit_mouse_button(&handle, 5, true) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(
+ unsafe { opennow_streamer_submit_mouse_wheel(&handle, 0, -120) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(
+ unsafe {
+ opennow_streamer_submit_gamepad(&handle, 2, 0x0404, 0x1001, 4, 5, -6, 7, -8, 9)
+ },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(
+ unsafe {
+ opennow_streamer_submit_local_action(&handle, OPENNOW_STREAMER_LOCAL_ACTION_GUIDE)
+ },
+ OpenNowStreamerStatus::Ok
+ );
+
+ let queue = handle.input.queue();
+ assert_eq!(
+ queue.take(),
+ Some(CapturedInput::Key {
+ virtual_key: 0x57,
+ modifiers: 0x02,
+ pressed: true,
+ })
+ );
+ assert_eq!(
+ queue.take(),
+ Some(CapturedInput::MouseMove {
+ delta_x: -12,
+ delta_y: 34,
+ })
+ );
+ assert_eq!(
+ queue.take(),
+ Some(CapturedInput::MouseAbsolute {
+ x: 20,
+ y: 30,
+ width: 640,
+ height: 360,
+ })
+ );
+ assert_eq!(
+ queue.take(),
+ Some(CapturedInput::MouseButton {
+ button: 5,
+ pressed: true,
+ })
+ );
+ assert_eq!(
+ queue.take(),
+ Some(CapturedInput::MouseWheel {
+ delta_x: 0,
+ delta_y: -120,
+ })
+ );
+ assert_eq!(
+ queue.take(),
+ Some(CapturedInput::Gamepad {
+ controller_id: 2,
+ bitmap: 0x0404,
+ buttons: 0x1001,
+ left_trigger: 4,
+ right_trigger: 5,
+ left_stick_x: -6,
+ left_stick_y: 7,
+ right_stick_x: -8,
+ right_stick_y: 9,
+ })
+ );
+ assert_eq!(queue.take(), Some(CapturedInput::Guide));
+ assert_eq!(queue.take(), None);
+
+ handle.shutdown();
+ }
+
+ #[test]
+ fn typed_submit_api_validates_boundaries_without_crossing_the_ffi() {
+ let messages = Box::new(CallbackMessages::default());
+ let mut handle = graphics_test_handle(&messages);
+ assert_eq!(
+ unsafe { opennow_streamer_submit_mouse_absolute(&handle, 0, 0, 0, 1) },
+ OpenNowStreamerStatus::InvalidConfig
+ );
+ assert_eq!(
+ unsafe { opennow_streamer_submit_mouse_button(&handle, 6, true) },
+ OpenNowStreamerStatus::InvalidConfig
+ );
+ assert_eq!(
+ unsafe { opennow_streamer_submit_gamepad(&handle, 4, 0, 0, 0, 0, 0, 0, 0, 0) },
+ OpenNowStreamerStatus::InvalidConfig
+ );
+ assert_eq!(
+ unsafe { opennow_streamer_submit_local_action(&handle, u32::MAX) },
+ OpenNowStreamerStatus::InvalidConfig
+ );
+ assert_eq!(
+ unsafe { opennow_streamer_submit_key(ptr::null(), 0, 0, false) },
+ OpenNowStreamerStatus::NullPointer
+ );
+ handle.shutdown();
+ }
+
+ #[test]
+ fn graphics_ffi_acquires_records_and_releases_the_latest_frame() {
+ let messages = Box::new(CallbackMessages::default());
+ let mut handle = graphics_test_handle(&messages);
+ let context = ffi_graphics_context();
+ assert_eq!(
+ unsafe { opennow_streamer_set_graphics_context(&handle, &context) },
+ OpenNowStreamerStatus::Ok
+ );
+ let publisher = handle.frame_publisher();
+ let lease = publisher.context().expect("graphics context lease");
+ let recorded = Arc::new(RecordedCommands::default());
+ publisher
+ .publish(
+ lease,
+ Arc::new(TestGraphicsFrame {
+ sequence: 42,
+ recorded: Arc::clone(&recorded),
+ panic_on_record: false,
+ }),
+ )
+ .expect("publish frame");
+ assert_eq!(messages.frames_available.load(Ordering::Relaxed), 1);
+
+ let mut token = ptr::null_mut();
+ let mut info = OpenNowStreamerFrameInfo::default();
+ assert_eq!(
+ unsafe { opennow_streamer_acquire_latest_frame(&handle, &mut token, &mut info) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert!(!token.is_null());
+ assert_eq!(
+ info,
+ OpenNowStreamerFrameInfo {
+ width: 2560,
+ height: 1440,
+ sequence: 42,
+ presentation_time_ns: 349_999_986,
+ }
+ );
+ let command = ffi_render_command();
+ let mut output = OpenNowStreamerRecordedFrame::default();
+ assert_eq!(
+ unsafe { opennow_streamer_record_frame(&handle, token, &command, &mut output) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(output.graphics_api, OPENNOW_STREAMER_GRAPHICS_API_VULKAN);
+ assert_eq!(output.width, 2560);
+ assert_eq!(output.height, 1440);
+ assert_eq!(output.frame_slot, 2);
+ assert_eq!(output.generation, 42);
+ assert_eq!(output.resource, 0xfeed);
+ assert_eq!(output.resource_view, 0xbeef);
+ assert_eq!(output.presentation_time_ns, 349_999_986);
+ assert_eq!(
+ unsafe { opennow_streamer_record_frame(&handle, token, &command, &mut output) },
+ OpenNowStreamerStatus::FrameAlreadyRecorded
+ );
+ assert_eq!(recorded.drops.load(Ordering::Relaxed), 0);
+ assert_eq!(
+ unsafe { opennow_streamer_release_frame(token) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(recorded.drops.load(Ordering::Relaxed), 1);
+
+ let values = recorded.values.lock().expect("recorded commands");
+ assert_eq!(values.len(), 1);
+ assert_eq!(values[0].0.api, GraphicsApi::Vulkan);
+ assert_eq!(values[0].0.device, context.device as usize);
+ assert_eq!(values[0].0.queue, context.queue as usize);
+ assert_eq!(values[0].1.command_buffer, command.command_buffer as usize);
+ assert_eq!(values[0].1.frame_slot, 2);
+ drop(values);
+
+ assert_eq!(
+ unsafe { opennow_streamer_scene_graph_shutdown(&handle) },
+ OpenNowStreamerStatus::Ok
+ );
+ handle.shutdown();
+ }
+
+ #[test]
+ fn scene_graph_shutdown_invalidates_tokens_without_leaking_them() {
+ let messages = Box::new(CallbackMessages::default());
+ let mut handle = graphics_test_handle(&messages);
+ let context = ffi_graphics_context();
+ assert_eq!(
+ unsafe { opennow_streamer_set_graphics_context(&handle, &context) },
+ OpenNowStreamerStatus::Ok
+ );
+ let publisher = handle.frame_publisher();
+ let recorded = Arc::new(RecordedCommands::default());
+ publisher
+ .publish(
+ publisher.context().expect("graphics context lease"),
+ Arc::new(TestGraphicsFrame {
+ sequence: 1,
+ recorded: Arc::clone(&recorded),
+ panic_on_record: false,
+ }),
+ )
+ .expect("publish frame");
+ let mut token = ptr::null_mut();
+ let mut info = OpenNowStreamerFrameInfo::default();
+ assert_eq!(
+ unsafe { opennow_streamer_acquire_latest_frame(&handle, &mut token, &mut info) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(
+ unsafe { opennow_streamer_scene_graph_shutdown(&handle) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(
+ unsafe {
+ opennow_streamer_record_frame(
+ &handle,
+ token,
+ &ffi_render_command(),
+ &mut OpenNowStreamerRecordedFrame::default(),
+ )
+ },
+ OpenNowStreamerStatus::GraphicsUnavailable
+ );
+ assert_eq!(recorded.drops.load(Ordering::Relaxed), 0);
+ assert_eq!(
+ unsafe { opennow_streamer_release_frame(token) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(recorded.drops.load(Ordering::Relaxed), 1);
+ handle.shutdown();
+ }
+
+ #[test]
+ fn graphics_ffi_contains_panics_from_frame_recording() {
+ let messages = Box::new(CallbackMessages::default());
+ let mut handle = graphics_test_handle(&messages);
+ assert_eq!(
+ unsafe { opennow_streamer_set_graphics_context(&handle, &ffi_graphics_context()) },
+ OpenNowStreamerStatus::Ok
+ );
+ let publisher = handle.frame_publisher();
+ let recorded = Arc::new(RecordedCommands::default());
+ publisher
+ .publish(
+ publisher.context().expect("graphics context lease"),
+ Arc::new(TestGraphicsFrame {
+ sequence: 1,
+ recorded,
+ panic_on_record: true,
+ }),
+ )
+ .expect("publish frame");
+ let mut token = ptr::null_mut();
+ let mut info = OpenNowStreamerFrameInfo::default();
+ assert_eq!(
+ unsafe { opennow_streamer_acquire_latest_frame(&handle, &mut token, &mut info) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(
+ unsafe {
+ opennow_streamer_record_frame(
+ &handle,
+ token,
+ &ffi_render_command(),
+ &mut OpenNowStreamerRecordedFrame::default(),
+ )
+ },
+ OpenNowStreamerStatus::Panic
+ );
+ assert_eq!(
+ unsafe { opennow_streamer_release_frame(token) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(
+ unsafe { opennow_streamer_scene_graph_shutdown(&handle) },
+ OpenNowStreamerStatus::Ok
+ );
+ handle.shutdown();
+ }
+
+ #[test]
+ fn destroy_rejects_an_active_scene_graph_without_consuming_the_handle() {
+ let messages = Box::new(CallbackMessages::default());
+ let handle = Box::into_raw(Box::new(graphics_test_handle(&messages)));
+ assert_eq!(
+ unsafe { opennow_streamer_set_graphics_context(handle, &ffi_graphics_context()) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(
+ unsafe { opennow_streamer_destroy(handle) },
+ OpenNowStreamerStatus::SceneGraphActive
+ );
+ assert_eq!(
+ unsafe { opennow_streamer_scene_graph_shutdown(handle) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert_eq!(
+ unsafe { opennow_streamer_destroy(handle) },
+ OpenNowStreamerStatus::Ok
+ );
+ }
+
+ #[test]
+ fn hello_round_trips_through_the_response_callback_with_test_media_runtime() {
+ let messages = Box::new(CallbackMessages::default());
+ let (mut handle, host) = create_with_test_runtime(&messages);
+ let command = serde_json::to_vec(&json!({
+ "id": "hello-1",
+ "type": "hello",
+ "protocolVersion": 5
+ }))
+ .expect("hello command");
+
+ assert_eq!(handle.send(&command), OpenNowStreamerStatus::Ok);
+ let response = messages.wait_for_id("hello-1");
+ assert_eq!(response["type"], "ready");
+ assert!(
+ response["capabilities"]
+ .get("supportsOfferAnswer")
+ .is_none()
+ );
+ assert!(response["capabilities"].get("microphoneDevices").is_none());
+
+ handle.shutdown();
+ host.join().expect("test media runtime");
+ }
+
+ #[test]
+ fn public_constructor_installs_the_production_in_process_media_runtime() {
+ let messages = Box::new(CallbackMessages::default());
+ let config = test_config(&messages);
+ let mut handle = ptr::null_mut();
+ assert_eq!(
+ unsafe { opennow_streamer_create(&config, &mut handle) },
+ OpenNowStreamerStatus::Ok
+ );
+ assert!(!handle.is_null());
+ let command = serde_json::to_vec(&json!({
+ "id": "hello-production",
+ "type": "hello",
+ "protocolVersion": 5
+ }))
+ .expect("hello command");
+ assert_eq!(
+ unsafe { &*handle }.send(&command),
+ OpenNowStreamerStatus::Ok
+ );
+ let response = messages.wait_for_id("hello-production");
+ assert!(
+ response["capabilities"]
+ .get("supportsOfferAnswer")
+ .is_none()
+ );
+ assert_eq!(response["capabilities"]["supportsAudioDecode"], true);
+ assert_eq!(response["capabilities"]["supportsAudioOutput"], true);
+ assert_eq!(
+ unsafe { opennow_streamer_destroy(handle) },
+ OpenNowStreamerStatus::Ok
+ );
+ }
+
+ #[test]
+ fn shutdown_responds_and_closes_the_command_queue() {
+ let messages = Box::new(CallbackMessages::default());
+ let (mut handle, host) = create_with_test_runtime(&messages);
+ let command = br#"{"id":"shutdown-1","type":"shutdown","reason":"ffi test"}"#;
+
+ assert_eq!(handle.send(command), OpenNowStreamerStatus::Ok);
+ let response = messages.wait_for_id("shutdown-1");
+ assert_eq!(response["type"], "ok");
+ let deadline = Instant::now() + Duration::from_secs(5);
+ while handle.send(br#"{"id":"late","type":"hello"}"#) != OpenNowStreamerStatus::Closed {
+ assert!(Instant::now() < deadline, "command queue did not close");
+ thread::yield_now();
+ }
+
+ handle.shutdown();
+ host.join().expect("test media runtime");
+ }
+}
diff --git a/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/format.rs b/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/format.rs
index dbee142a2..db5b6aed1 100644
--- a/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/format.rs
+++ b/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/format.rs
@@ -220,6 +220,7 @@ pub struct VulkanVideoFrame {
device_context: usize,
lock_queue: Option,
unlock_queue: Option,
+ cpu_nv12_fallback: Option Result> + Send + Sync>>,
owner: Arc,
}
@@ -249,10 +250,28 @@ impl VulkanVideoFrame {
device_context,
lock_queue,
unlock_queue,
+ cpu_nv12_fallback: None,
owner,
}
}
+ pub fn with_cpu_nv12_fallback(
+ mut self,
+ fallback: Arc Result> + Send + Sync>,
+ ) -> Self {
+ self.cpu_nv12_fallback = Some(fallback);
+ self
+ }
+
+ pub fn download_nv12(&self) -> Result> {
+ self.cpu_nv12_fallback.as_ref().ok_or_else(|| {
+ Error::unavailable(
+ crate::Subsystem::Vulkan,
+ "Vulkan frame has no CPU NV12 fallback",
+ )
+ })?()
+ }
+
pub fn validate(&self) -> Result<()> {
if self.instance == 0
|| self.physical_device == 0
@@ -478,4 +497,51 @@ mod tests {
};
assert!(frame.validate().is_err());
}
+
+ #[test]
+ fn vulkan_frame_cpu_fallback_is_lazy_and_typed_as_nv12_planes() {
+ let calls = Arc::new(std::sync::atomic::AtomicUsize::new(0));
+ let fallback_calls = Arc::clone(&calls);
+ let frame = VulkanVideoFrame::new(
+ 1,
+ 2,
+ 3,
+ vec![0],
+ 0,
+ 0,
+ vec![VulkanImage {
+ image: 4,
+ format: 1,
+ width: 4,
+ height: 4,
+ layout: 1,
+ access: 0,
+ semaphore: 5,
+ semaphore_value: 1,
+ queue_family: 0,
+ }],
+ 0,
+ None,
+ None,
+ Arc::new(()),
+ )
+ .with_cpu_nv12_fallback(Arc::new(move || {
+ fallback_calls.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
+ Ok(vec![
+ FramePlane {
+ data: Arc::from(vec![0_u8; 16]),
+ stride: 4,
+ rows: 4,
+ },
+ FramePlane {
+ data: Arc::from(vec![0_u8; 8]),
+ stride: 4,
+ rows: 2,
+ },
+ ])
+ }));
+ assert_eq!(calls.load(std::sync::atomic::Ordering::Relaxed), 0);
+ assert_eq!(frame.download_nv12().unwrap().len(), 2);
+ assert_eq!(calls.load(std::sync::atomic::Ordering::Relaxed), 1);
+ }
}
diff --git a/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/frame_producer.rs b/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/frame_producer.rs
new file mode 100644
index 000000000..c66177af1
--- /dev/null
+++ b/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/frame_producer.rs
@@ -0,0 +1,1871 @@
+use std::collections::{HashMap, hash_map::Entry as HashMapEntry};
+use std::io::Cursor;
+use std::os::fd::RawFd;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::{Arc, Mutex, Weak};
+use std::time::Duration;
+
+use ash::vk::{self, Handle};
+
+use crate::{
+ DecodedVideoFrame, DmaBufFrame, DmaBufPlane, Error, FramePlane, PixelFormat, Result, Subsystem,
+ VulkanVideoFrame,
+};
+
+const DRM_FORMAT_NV12: u32 = u32::from_le_bytes(*b"NV12");
+const DRM_FORMAT_R8: u32 = u32::from_le_bytes(*b"R8 ");
+const DRM_FORMAT_GR88: u32 = u32::from_le_bytes(*b"GR88");
+const DECODE_WAIT_TIMEOUT: Duration = Duration::from_secs(1);
+const NV12_VERTEX_SHADER: &[u8] = include_bytes!("../shaders/nv12.vert.spv");
+const NV12_FRAGMENT_SHADER: &[u8] = include_bytes!("../shaders/nv12.frag.spv");
+const DEFAULT_FRAME_SLOTS: u32 = 3;
+const MAX_FRAME_SLOTS: u32 = 8;
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct VulkanRenderDevice {
+ pub instance: usize,
+ pub physical_device: usize,
+ pub device: usize,
+ pub queue_family: u32,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct RecordedGpuFrame {
+ pub slot: u32,
+ pub generation: u64,
+ pub image: u64,
+ pub image_view: u64,
+ pub width: u32,
+ pub height: u32,
+ pub timestamp_us: u64,
+}
+
+#[derive(Clone)]
+pub struct LinuxGpuFrameProducer {
+ state: Arc>,
+ sequence: Arc,
+ slot_count: u32,
+}
+
+struct SharedProducerState {
+ render: Option,
+ producer: Option,
+}
+
+pub struct LinuxGpuFrame {
+ frame: DecodedVideoFrame,
+ producer: LinuxGpuFrameProducer,
+ sequence: u64,
+}
+
+impl LinuxGpuFrameProducer {
+ pub fn new(slot_count: u32) -> Result {
+ if slot_count == 0 || slot_count > MAX_FRAME_SLOTS {
+ return Err(Error::InvalidFormat(
+ "embedded Vulkan frame slot count must be between 1 and 8".to_owned(),
+ ));
+ }
+ Ok(Self {
+ state: Arc::new(Mutex::new(SharedProducerState {
+ render: None,
+ producer: None,
+ })),
+ sequence: Arc::new(AtomicU64::new(0)),
+ slot_count,
+ })
+ }
+
+ pub fn frame(&self, frame: DecodedVideoFrame) -> Result {
+ frame.validate()?;
+ Ok(LinuxGpuFrame {
+ frame,
+ producer: self.clone(),
+ sequence: self.sequence.fetch_add(1, Ordering::Relaxed) + 1,
+ })
+ }
+}
+
+impl LinuxGpuFrame {
+ pub fn width(&self) -> u32 {
+ self.frame.format.width
+ }
+
+ pub fn height(&self) -> u32 {
+ self.frame.format.height
+ }
+
+ pub fn sequence(&self) -> u64 {
+ self.sequence
+ }
+
+ pub fn presentation_time_ns(&self) -> u64 {
+ self.frame.timestamp_us.saturating_mul(1_000)
+ }
+
+ /// Records conversion before Qt begins its item render pass.
+ ///
+ /// # Safety
+ ///
+ /// `render` and `command_buffer` have the requirements documented by
+ /// [`LinuxFrameProducer::new_with_slots`] and
+ /// [`LinuxFrameProducer::record_frame`].
+ pub unsafe fn record(
+ &self,
+ render: VulkanRenderDevice,
+ command_buffer: usize,
+ frame_slot: u32,
+ ) -> Result {
+ let mut state = self
+ .producer
+ .state
+ .lock()
+ .unwrap_or_else(|poison| poison.into_inner());
+ if state.render != Some(render) {
+ state.producer = None;
+ state.producer = Some(unsafe {
+ LinuxFrameProducer::new_with_slots(render, self.producer.slot_count)?
+ });
+ state.render = Some(render);
+ }
+ unsafe {
+ state
+ .producer
+ .as_mut()
+ .expect("producer initialized")
+ .record_frame(self.frame.clone(), command_buffer, frame_slot)
+ }
+ }
+}
+
+impl VulkanRenderDevice {
+ fn validate(self) -> Result<()> {
+ if self.instance == 0 || self.physical_device == 0 || self.device == 0 {
+ return Err(Error::InvalidFormat(
+ "Qt Vulkan render device contains a null handle".to_owned(),
+ ));
+ }
+ Ok(())
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PreparedVulkanImage {
+ pub image: u64,
+ pub format: i32,
+ pub width: u32,
+ pub height: u32,
+ pub old_layout: i32,
+ pub old_access: u64,
+ pub source_queue_family: u32,
+ pub render_queue_family: u32,
+}
+
+#[derive(Debug)]
+pub struct PreparedVulkanFrame {
+ pub images: Vec,
+ source: Arc,
+}
+
+impl PreparedVulkanFrame {
+ pub fn retain_source(&self) -> &Arc {
+ &self.source
+ }
+}
+
+#[derive(Debug)]
+pub struct CpuNv12Frame {
+ pub luma: FramePlane,
+ pub chroma: FramePlane,
+ source: Arc,
+}
+
+impl CpuNv12Frame {
+ pub fn retain_source(&self) -> &Arc {
+ &self.source
+ }
+}
+
+#[derive(Debug)]
+pub enum PreparedLinuxFrame {
+ Vulkan(PreparedVulkanFrame),
+ DmaBuf(Arc),
+ Cpu(CpuNv12Frame),
+}
+
+impl PreparedLinuxFrame {
+ pub fn timestamp_us(&self) -> u64 {
+ match self {
+ Self::Vulkan(frame) => frame.source.timestamp_us,
+ Self::DmaBuf(frame) => frame.source.timestamp_us,
+ Self::Cpu(frame) => frame.source.timestamp_us,
+ }
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+struct DmaBufKey {
+ device: u64,
+ inode: u64,
+ width: u32,
+ height: u32,
+ modifier: u64,
+ luma_offset: usize,
+ luma_pitch: usize,
+ chroma_offset: usize,
+ chroma_pitch: usize,
+}
+
+pub struct ImportedNv12Frame {
+ pub image: u64,
+ pub luma_view: u64,
+ pub chroma_view: u64,
+ pub modifier: u64,
+ pub external_queue_family: u32,
+ pub render_queue_family: u32,
+ image_handle: vk::Image,
+ luma_view_handle: vk::ImageView,
+ chroma_view_handle: vk::ImageView,
+ memory: vk::DeviceMemory,
+ device: ash::Device,
+ source: Arc,
+}
+
+impl ImportedNv12Frame {
+ pub fn retain_source(&self) -> &Arc {
+ &self.source
+ }
+}
+
+impl std::fmt::Debug for ImportedNv12Frame {
+ fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ formatter
+ .debug_struct("ImportedNv12Frame")
+ .field("image", &self.image)
+ .field("luma_view", &self.luma_view)
+ .field("chroma_view", &self.chroma_view)
+ .field("modifier", &self.modifier)
+ .finish_non_exhaustive()
+ }
+}
+
+impl Drop for ImportedNv12Frame {
+ fn drop(&mut self) {
+ unsafe {
+ self.device
+ .destroy_image_view(self.chroma_view_handle, None);
+ self.device.destroy_image_view(self.luma_view_handle, None);
+ self.device.destroy_image(self.image_handle, None);
+ self.device.free_memory(self.memory, None);
+ }
+ }
+}
+
+pub struct LinuxFrameProducer {
+ _entry: ash::Entry,
+ instance: ash::Instance,
+ physical_device: vk::PhysicalDevice,
+ device: ash::Device,
+ render: VulkanRenderDevice,
+ dmabuf_import_supported: bool,
+ imported: HashMap>,
+ renderer: Option,
+ slots: Vec>,
+ generation: u64,
+}
+
+struct RendererResources {
+ render_pass: vk::RenderPass,
+ descriptor_set_layout: vk::DescriptorSetLayout,
+ pipeline_layout: vk::PipelineLayout,
+ pipeline: vk::Pipeline,
+ descriptor_pool: vk::DescriptorPool,
+ descriptor_sets: Vec,
+ sampler: vk::Sampler,
+}
+
+struct GpuImage {
+ image: vk::Image,
+ memory: vk::DeviceMemory,
+ view: vk::ImageView,
+}
+
+struct CpuUploadResources {
+ luma: GpuImage,
+ chroma: GpuImage,
+ staging_buffer: vk::Buffer,
+ staging_memory: vk::DeviceMemory,
+ initialized: bool,
+}
+
+struct FrameSlotResources {
+ width: u32,
+ height: u32,
+ output: GpuImage,
+ output_initialized: bool,
+ framebuffer: vk::Framebuffer,
+ cpu: Option,
+ owned_input_views: Vec,
+ frame: Option,
+}
+
+impl LinuxFrameProducer {
+ /// Adopts Qt's Vulkan handles without taking ownership of them.
+ ///
+ /// The producer and every prepared frame must be dropped before Qt destroys
+ /// the corresponding QRhi. Calls must remain on the QRhi render thread.
+ ///
+ /// # Safety
+ ///
+ /// All handles must belong to the same live Vulkan device and remain valid
+ /// for the lifetime described above.
+ pub unsafe fn new(render: VulkanRenderDevice) -> Result {
+ unsafe { Self::new_with_slots(render, DEFAULT_FRAME_SLOTS) }
+ }
+
+ /// # Safety
+ ///
+ /// This has the same requirements as [`Self::new`]. `slot_count` must be
+ /// Qt's maximum number of in-flight QRhi frame slots.
+ pub unsafe fn new_with_slots(render: VulkanRenderDevice, slot_count: u32) -> Result {
+ render.validate()?;
+ if slot_count == 0 || slot_count > MAX_FRAME_SLOTS {
+ return Err(Error::InvalidFormat(
+ "embedded Vulkan frame slot count must be between 1 and 8".to_owned(),
+ ));
+ }
+ let entry = unsafe { ash::Entry::load() }.map_err(|error| {
+ Error::unavailable(Subsystem::Vulkan, format!("load Vulkan entry: {error}"))
+ })?;
+ let instance = unsafe {
+ ash::Instance::load(
+ entry.static_fn(),
+ vk::Instance::from_raw(render.instance as u64),
+ )
+ };
+ let physical_device = vk::PhysicalDevice::from_raw(render.physical_device as u64);
+ let device = unsafe {
+ ash::Device::load(
+ instance.fp_v1_0(),
+ vk::Device::from_raw(render.device as u64),
+ )
+ };
+ let required = [
+ ash::khr::external_memory::NAME,
+ ash::khr::external_memory_fd::NAME,
+ ash::ext::external_memory_dma_buf::NAME,
+ ash::ext::image_drm_format_modifier::NAME,
+ ];
+ let available = unsafe { instance.enumerate_device_extension_properties(physical_device) }
+ .map_err(|error| vk_error("enumerate Qt Vulkan device extensions", error))?;
+ let dmabuf_import_supported = required.iter().all(|required| {
+ available.iter().any(|extension| {
+ let name = unsafe { std::ffi::CStr::from_ptr(extension.extension_name.as_ptr()) };
+ name == *required
+ })
+ });
+ Ok(Self {
+ _entry: entry,
+ instance,
+ physical_device,
+ device,
+ render,
+ dmabuf_import_supported,
+ imported: HashMap::new(),
+ renderer: None,
+ slots: (0..slot_count).map(|_| None).collect(),
+ generation: 0,
+ })
+ }
+
+ /// Encodes NV12-to-RGBA into Qt's current Vulkan command buffer.
+ ///
+ /// The command buffer must be recording but outside a render pass. This
+ /// method never begins, ends, submits, or waits for Qt's command buffer.
+ /// Qt may reuse a slot only after its previous frame has completed.
+ ///
+ /// # Safety
+ ///
+ /// `command_buffer` must belong to the adopted device and be in the
+ /// recording state on the QRhi render thread.
+ pub unsafe fn record_frame(
+ &mut self,
+ frame: DecodedVideoFrame,
+ command_buffer: usize,
+ slot: u32,
+ ) -> Result {
+ if command_buffer == 0 {
+ return Err(Error::InvalidFormat(
+ "Qt supplied a null Vulkan command buffer".to_owned(),
+ ));
+ }
+ let slot_index = usize::try_from(slot)
+ .ok()
+ .filter(|index| *index < self.slots.len())
+ .ok_or_else(|| Error::InvalidFormat(format!("invalid QRhi frame slot {slot}")))?;
+ let width = frame.format.width;
+ let height = frame.format.height;
+ let timestamp_us = frame.timestamp_us;
+ let color_matrix = frame.format.color_matrix;
+ let full_range = frame.format.color_range == crate::ColorRange::Full;
+ let prepared = self.prepare(frame)?;
+ self.ensure_renderer()?;
+ self.ensure_slot(slot_index, width, height)?;
+ let command_buffer = vk::CommandBuffer::from_raw(command_buffer as u64);
+ let (luma_view, chroma_view) =
+ self.prepare_input(slot_index, command_buffer, &prepared, width, height)?;
+ self.update_descriptors(slot_index, luma_view, chroma_view);
+ unsafe {
+ self.record_conversion(
+ slot_index,
+ command_buffer,
+ &prepared,
+ width,
+ height,
+ color_matrix,
+ full_range,
+ )?;
+ }
+ self.generation = self.generation.wrapping_add(1);
+ let generation = self.generation;
+ let slot_resources = self.slots[slot_index]
+ .as_mut()
+ .expect("frame slot initialized");
+ slot_resources.frame = Some(prepared);
+ Ok(RecordedGpuFrame {
+ slot,
+ generation,
+ image: slot_resources.output.image.as_raw(),
+ image_view: slot_resources.output.view.as_raw(),
+ width,
+ height,
+ timestamp_us,
+ })
+ }
+
+ pub fn prepare(&mut self, frame: DecodedVideoFrame) -> Result {
+ frame.validate()?;
+ if frame.format.pixel_format != PixelFormat::Nv12 {
+ return Err(Error::InvalidFormat(format!(
+ "embedded Linux presentation requires NV12, received {:?}",
+ frame.format.pixel_format
+ )));
+ }
+ let frame = Arc::new(frame);
+ let mut decode_waited = frame.vulkan.is_none();
+ if let Some(vulkan) = frame.vulkan.as_ref() {
+ decode_waited = self.wait_for_decode(vulkan).is_ok();
+ if vulkan.device == self.render.device && decode_waited {
+ let images = vulkan
+ .images
+ .iter()
+ .map(|image| PreparedVulkanImage {
+ image: image.image,
+ format: image.format,
+ width: image.width,
+ height: image.height,
+ old_layout: image.layout,
+ old_access: image.access,
+ source_queue_family: image.queue_family,
+ render_queue_family: self.render.queue_family,
+ })
+ .collect();
+ return Ok(PreparedLinuxFrame::Vulkan(PreparedVulkanFrame {
+ images,
+ source: frame,
+ }));
+ }
+ }
+ let mut gpu_error = None;
+ if decode_waited && self.dmabuf_import_supported && frame.dmabuf.is_some() {
+ match self.import_dmabuf(Arc::clone(&frame)) {
+ Ok(imported) => return Ok(PreparedLinuxFrame::DmaBuf(imported)),
+ Err(error) => gpu_error = Some(error),
+ }
+ }
+ if frame.planes.len() == 2 {
+ return Ok(PreparedLinuxFrame::Cpu(CpuNv12Frame {
+ luma: frame.planes[0].clone(),
+ chroma: frame.planes[1].clone(),
+ source: frame,
+ }));
+ }
+ if let Some(vulkan) = frame.vulkan.as_ref() {
+ let planes = vulkan.download_nv12()?;
+ if planes.len() != 2 {
+ return Err(Error::InvalidFormat(
+ "Vulkan CPU fallback did not produce two NV12 planes".to_owned(),
+ ));
+ }
+ return Ok(PreparedLinuxFrame::Cpu(CpuNv12Frame {
+ luma: planes[0].clone(),
+ chroma: planes[1].clone(),
+ source: frame,
+ }));
+ }
+ if let Some(error) = gpu_error {
+ return Err(error);
+ }
+ Err(Error::unavailable(
+ Subsystem::Vulkan,
+ "decoded frame cannot be synchronized with Qt's Vulkan device and has no CPU NV12 fallback",
+ ))
+ }
+
+ fn wait_for_decode(&self, frame: &VulkanVideoFrame) -> Result<()> {
+ let (semaphores, values): (Vec<_>, Vec<_>) = timeline_waits(frame)?
+ .into_iter()
+ .map(|(semaphore, value)| (vk::Semaphore::from_raw(semaphore), value))
+ .unzip();
+ let wait = vk::SemaphoreWaitInfo::default()
+ .semaphores(&semaphores)
+ .values(&values);
+ let source_instance;
+ let source_device;
+ let device = if frame.device == self.render.device {
+ &self.device
+ } else {
+ source_instance = unsafe {
+ ash::Instance::load(
+ self._entry.static_fn(),
+ vk::Instance::from_raw(frame.instance as u64),
+ )
+ };
+ source_device = unsafe {
+ ash::Device::load(
+ source_instance.fp_v1_0(),
+ vk::Device::from_raw(frame.device as u64),
+ )
+ };
+ &source_device
+ };
+ unsafe { device.wait_semaphores(&wait, DECODE_WAIT_TIMEOUT.as_nanos() as u64) }
+ .map_err(|error| vk_error("host-wait for Vulkan decoder frame", error))
+ }
+
+ fn import_dmabuf(&mut self, source: Arc) -> Result> {
+ let dmabuf = source.dmabuf.as_ref().expect("DMA-BUF checked by caller");
+ let (object_index, luma, chroma) = nv12_dmabuf_layout(dmabuf)?;
+ let object = dmabuf.objects[object_index];
+ let (device, inode) = fd_identity(object.fd)?;
+ let key = DmaBufKey {
+ device,
+ inode,
+ width: source.format.width,
+ height: source.format.height,
+ modifier: object.format_modifier,
+ luma_offset: luma.offset,
+ luma_pitch: luma.pitch,
+ chroma_offset: chroma.offset,
+ chroma_pitch: chroma.pitch,
+ };
+ if let Some(imported) = self.imported.get(&key).and_then(Weak::upgrade) {
+ return Ok(imported);
+ }
+ self.imported
+ .retain(|_, imported| imported.strong_count() != 0);
+ let imported = Arc::new(import_nv12_dmabuf(
+ &self.instance,
+ self.physical_device,
+ &self.device,
+ self.render.queue_family,
+ object.fd,
+ object.size,
+ object.format_modifier,
+ luma,
+ chroma,
+ source,
+ )?);
+ self.imported.insert(key, Arc::downgrade(&imported));
+ Ok(imported)
+ }
+
+ fn ensure_renderer(&mut self) -> Result<()> {
+ if self.renderer.is_some() {
+ return Ok(());
+ }
+ let attachment = [vk::AttachmentDescription::default()
+ .format(vk::Format::R8G8B8A8_UNORM)
+ .samples(vk::SampleCountFlags::TYPE_1)
+ .load_op(vk::AttachmentLoadOp::DONT_CARE)
+ .store_op(vk::AttachmentStoreOp::STORE)
+ .stencil_load_op(vk::AttachmentLoadOp::DONT_CARE)
+ .stencil_store_op(vk::AttachmentStoreOp::DONT_CARE)
+ .initial_layout(vk::ImageLayout::COLOR_ATTACHMENT_OPTIMAL)
+ .final_layout(vk::ImageLayout::COLOR_ATTACHMENT_OPTIMAL)];
+ let color_reference = [vk::AttachmentReference::default()
+ .attachment(0)
+ .layout(vk::ImageLayout::COLOR_ATTACHMENT_OPTIMAL)];
+ let subpasses = [vk::SubpassDescription::default()
+ .pipeline_bind_point(vk::PipelineBindPoint::GRAPHICS)
+ .color_attachments(&color_reference)];
+ let render_pass = unsafe {
+ self.device.create_render_pass(
+ &vk::RenderPassCreateInfo::default()
+ .attachments(&attachment)
+ .subpasses(&subpasses),
+ None,
+ )
+ }
+ .map_err(|error| vk_error("create embedded NV12 render pass", error))?;
+ let bindings = [0, 1].map(|binding| {
+ vk::DescriptorSetLayoutBinding::default()
+ .binding(binding)
+ .descriptor_type(vk::DescriptorType::COMBINED_IMAGE_SAMPLER)
+ .descriptor_count(1)
+ .stage_flags(vk::ShaderStageFlags::FRAGMENT)
+ });
+ let descriptor_set_layout = unsafe {
+ self.device.create_descriptor_set_layout(
+ &vk::DescriptorSetLayoutCreateInfo::default().bindings(&bindings),
+ None,
+ )
+ }
+ .map_err(|error| vk_error("create embedded NV12 descriptor layout", error))?;
+ let layouts = [descriptor_set_layout];
+ let push_ranges = [vk::PushConstantRange::default()
+ .stage_flags(vk::ShaderStageFlags::FRAGMENT)
+ .size(size_of::() as u32)];
+ let pipeline_layout = unsafe {
+ self.device.create_pipeline_layout(
+ &vk::PipelineLayoutCreateInfo::default()
+ .set_layouts(&layouts)
+ .push_constant_ranges(&push_ranges),
+ None,
+ )
+ }
+ .map_err(|error| vk_error("create embedded NV12 pipeline layout", error))?;
+ let vertex_words = ash::util::read_spv(&mut Cursor::new(NV12_VERTEX_SHADER))
+ .map_err(|error| Error::backend(Subsystem::Vulkan, error.to_string()))?;
+ let fragment_words = ash::util::read_spv(&mut Cursor::new(NV12_FRAGMENT_SHADER))
+ .map_err(|error| Error::backend(Subsystem::Vulkan, error.to_string()))?;
+ let vertex_module = unsafe {
+ self.device.create_shader_module(
+ &vk::ShaderModuleCreateInfo::default().code(&vertex_words),
+ None,
+ )
+ }
+ .map_err(|error| vk_error("create embedded NV12 vertex shader", error))?;
+ let fragment_module = unsafe {
+ self.device.create_shader_module(
+ &vk::ShaderModuleCreateInfo::default().code(&fragment_words),
+ None,
+ )
+ }
+ .map_err(|error| vk_error("create embedded NV12 fragment shader", error))?;
+ let stages = [
+ vk::PipelineShaderStageCreateInfo::default()
+ .stage(vk::ShaderStageFlags::VERTEX)
+ .module(vertex_module)
+ .name(c"main"),
+ vk::PipelineShaderStageCreateInfo::default()
+ .stage(vk::ShaderStageFlags::FRAGMENT)
+ .module(fragment_module)
+ .name(c"main"),
+ ];
+ let vertex_input = vk::PipelineVertexInputStateCreateInfo::default();
+ let input_assembly = vk::PipelineInputAssemblyStateCreateInfo::default()
+ .topology(vk::PrimitiveTopology::TRIANGLE_LIST);
+ let viewport_state = vk::PipelineViewportStateCreateInfo::default()
+ .viewport_count(1)
+ .scissor_count(1);
+ let rasterization = vk::PipelineRasterizationStateCreateInfo::default()
+ .polygon_mode(vk::PolygonMode::FILL)
+ .cull_mode(vk::CullModeFlags::NONE)
+ .front_face(vk::FrontFace::COUNTER_CLOCKWISE)
+ .line_width(1.0);
+ let multisample = vk::PipelineMultisampleStateCreateInfo::default()
+ .rasterization_samples(vk::SampleCountFlags::TYPE_1);
+ let blend_attachments = [vk::PipelineColorBlendAttachmentState::default()
+ .color_write_mask(vk::ColorComponentFlags::RGBA)];
+ let blend =
+ vk::PipelineColorBlendStateCreateInfo::default().attachments(&blend_attachments);
+ let dynamic_states = [vk::DynamicState::VIEWPORT, vk::DynamicState::SCISSOR];
+ let dynamic = vk::PipelineDynamicStateCreateInfo::default().dynamic_states(&dynamic_states);
+ let pipeline_info = [vk::GraphicsPipelineCreateInfo::default()
+ .stages(&stages)
+ .vertex_input_state(&vertex_input)
+ .input_assembly_state(&input_assembly)
+ .viewport_state(&viewport_state)
+ .rasterization_state(&rasterization)
+ .multisample_state(&multisample)
+ .color_blend_state(&blend)
+ .dynamic_state(&dynamic)
+ .layout(pipeline_layout)
+ .render_pass(render_pass)];
+ let pipeline = unsafe {
+ self.device
+ .create_graphics_pipelines(vk::PipelineCache::null(), &pipeline_info, None)
+ }
+ .map_err(|(_, error)| vk_error("create embedded NV12 pipeline", error))?[0];
+ unsafe {
+ self.device.destroy_shader_module(fragment_module, None);
+ self.device.destroy_shader_module(vertex_module, None);
+ }
+ let sampler = unsafe {
+ self.device.create_sampler(
+ &vk::SamplerCreateInfo::default()
+ .mag_filter(vk::Filter::LINEAR)
+ .min_filter(vk::Filter::LINEAR)
+ .mipmap_mode(vk::SamplerMipmapMode::NEAREST)
+ .address_mode_u(vk::SamplerAddressMode::CLAMP_TO_EDGE)
+ .address_mode_v(vk::SamplerAddressMode::CLAMP_TO_EDGE)
+ .address_mode_w(vk::SamplerAddressMode::CLAMP_TO_EDGE),
+ None,
+ )
+ }
+ .map_err(|error| vk_error("create embedded NV12 sampler", error))?;
+ let pool_sizes = [vk::DescriptorPoolSize::default()
+ .ty(vk::DescriptorType::COMBINED_IMAGE_SAMPLER)
+ .descriptor_count(self.slots.len() as u32 * 2)];
+ let descriptor_pool = unsafe {
+ self.device.create_descriptor_pool(
+ &vk::DescriptorPoolCreateInfo::default()
+ .max_sets(self.slots.len() as u32)
+ .pool_sizes(&pool_sizes),
+ None,
+ )
+ }
+ .map_err(|error| vk_error("create embedded NV12 descriptor pool", error))?;
+ let set_layouts = vec![descriptor_set_layout; self.slots.len()];
+ let descriptor_sets = unsafe {
+ self.device.allocate_descriptor_sets(
+ &vk::DescriptorSetAllocateInfo::default()
+ .descriptor_pool(descriptor_pool)
+ .set_layouts(&set_layouts),
+ )
+ }
+ .map_err(|error| vk_error("allocate embedded NV12 descriptor sets", error))?;
+ self.renderer = Some(RendererResources {
+ render_pass,
+ descriptor_set_layout,
+ pipeline_layout,
+ pipeline,
+ descriptor_pool,
+ descriptor_sets,
+ sampler,
+ });
+ Ok(())
+ }
+
+ fn ensure_slot(&mut self, slot: usize, width: u32, height: u32) -> Result<()> {
+ if self.slots[slot]
+ .as_ref()
+ .is_some_and(|resources| resources.width == width && resources.height == height)
+ {
+ return Ok(());
+ }
+ if let Some(resources) = self.slots[slot].take() {
+ self.destroy_slot(resources);
+ }
+ let output = create_gpu_image(
+ &self.instance,
+ self.physical_device,
+ &self.device,
+ width,
+ height,
+ vk::Format::R8G8B8A8_UNORM,
+ vk::ImageUsageFlags::COLOR_ATTACHMENT | vk::ImageUsageFlags::SAMPLED,
+ )?;
+ let renderer = self.renderer.as_ref().expect("renderer initialized");
+ let attachments = [output.view];
+ let framebuffer = unsafe {
+ self.device.create_framebuffer(
+ &vk::FramebufferCreateInfo::default()
+ .render_pass(renderer.render_pass)
+ .attachments(&attachments)
+ .width(width)
+ .height(height)
+ .layers(1),
+ None,
+ )
+ }
+ .map_err(|error| vk_error("create embedded RGBA framebuffer", error))?;
+ self.slots[slot] = Some(FrameSlotResources {
+ width,
+ height,
+ output,
+ output_initialized: false,
+ framebuffer,
+ cpu: None,
+ owned_input_views: Vec::new(),
+ frame: None,
+ });
+ Ok(())
+ }
+
+ fn prepare_input(
+ &mut self,
+ slot: usize,
+ command_buffer: vk::CommandBuffer,
+ prepared: &PreparedLinuxFrame,
+ width: u32,
+ height: u32,
+ ) -> Result<(vk::ImageView, vk::ImageView)> {
+ let resources = self.slots[slot].as_mut().expect("slot initialized");
+ resources.frame = None;
+ for view in resources.owned_input_views.drain(..) {
+ unsafe { self.device.destroy_image_view(view, None) };
+ }
+ match prepared {
+ PreparedLinuxFrame::Vulkan(frame) => {
+ let (luma, chroma) = direct_image_views(&self.device, &frame.images)?;
+ resources.owned_input_views.extend([luma, chroma]);
+ Ok((luma, chroma))
+ }
+ PreparedLinuxFrame::DmaBuf(frame) => Ok((
+ vk::ImageView::from_raw(frame.luma_view),
+ vk::ImageView::from_raw(frame.chroma_view),
+ )),
+ PreparedLinuxFrame::Cpu(frame) => {
+ if resources.cpu.is_none() {
+ resources.cpu = Some(create_cpu_upload_resources(
+ &self.instance,
+ self.physical_device,
+ &self.device,
+ width,
+ height,
+ )?);
+ }
+ let cpu = resources.cpu.as_mut().expect("CPU resources initialized");
+ upload_cpu_nv12(&self.device, command_buffer, cpu, frame, width, height)?;
+ Ok((cpu.luma.view, cpu.chroma.view))
+ }
+ }
+ }
+
+ fn update_descriptors(
+ &self,
+ slot: usize,
+ luma_view: vk::ImageView,
+ chroma_view: vk::ImageView,
+ ) {
+ let renderer = self.renderer.as_ref().expect("renderer initialized");
+ let luma = [vk::DescriptorImageInfo::default()
+ .sampler(renderer.sampler)
+ .image_view(luma_view)
+ .image_layout(vk::ImageLayout::SHADER_READ_ONLY_OPTIMAL)];
+ let chroma = [vk::DescriptorImageInfo::default()
+ .sampler(renderer.sampler)
+ .image_view(chroma_view)
+ .image_layout(vk::ImageLayout::SHADER_READ_ONLY_OPTIMAL)];
+ let writes = [
+ vk::WriteDescriptorSet::default()
+ .dst_set(renderer.descriptor_sets[slot])
+ .dst_binding(0)
+ .descriptor_type(vk::DescriptorType::COMBINED_IMAGE_SAMPLER)
+ .image_info(&luma),
+ vk::WriteDescriptorSet::default()
+ .dst_set(renderer.descriptor_sets[slot])
+ .dst_binding(1)
+ .descriptor_type(vk::DescriptorType::COMBINED_IMAGE_SAMPLER)
+ .image_info(&chroma),
+ ];
+ unsafe { self.device.update_descriptor_sets(&writes, &[]) };
+ }
+
+ #[allow(clippy::too_many_arguments)]
+ unsafe fn record_conversion(
+ &mut self,
+ slot: usize,
+ command_buffer: vk::CommandBuffer,
+ prepared: &PreparedLinuxFrame,
+ width: u32,
+ height: u32,
+ color_matrix: crate::ColorMatrix,
+ full_range: bool,
+ ) -> Result<()> {
+ let resources = self.slots[slot].as_mut().expect("slot initialized");
+ let renderer = self.renderer.as_ref().expect("renderer initialized");
+ let mut acquire = Vec::new();
+ match prepared {
+ PreparedLinuxFrame::Vulkan(frame) => {
+ acquire.extend(frame.images.iter().map(|image| {
+ let source_family = if image.source_queue_family == self.render.queue_family {
+ vk::QUEUE_FAMILY_IGNORED
+ } else {
+ image.source_queue_family
+ };
+ vk::ImageMemoryBarrier::default()
+ .old_layout(vk::ImageLayout::from_raw(image.old_layout))
+ .new_layout(vk::ImageLayout::SHADER_READ_ONLY_OPTIMAL)
+ .src_queue_family_index(source_family)
+ .dst_queue_family_index(if source_family == vk::QUEUE_FAMILY_IGNORED {
+ vk::QUEUE_FAMILY_IGNORED
+ } else {
+ self.render.queue_family
+ })
+ .src_access_mask(vk::AccessFlags::from_raw(image.old_access as u32))
+ .dst_access_mask(vk::AccessFlags::SHADER_READ)
+ .image(vk::Image::from_raw(image.image))
+ .subresource_range(image_color_range())
+ }));
+ }
+ PreparedLinuxFrame::DmaBuf(frame) => {
+ acquire.push(
+ vk::ImageMemoryBarrier::default()
+ .old_layout(vk::ImageLayout::GENERAL)
+ .new_layout(vk::ImageLayout::SHADER_READ_ONLY_OPTIMAL)
+ .src_queue_family_index(vk::QUEUE_FAMILY_EXTERNAL)
+ .dst_queue_family_index(self.render.queue_family)
+ .src_access_mask(vk::AccessFlags::MEMORY_WRITE)
+ .dst_access_mask(vk::AccessFlags::SHADER_READ)
+ .image(vk::Image::from_raw(frame.image))
+ .subresource_range(image_color_range()),
+ );
+ }
+ PreparedLinuxFrame::Cpu(_) => {}
+ }
+ acquire.push(
+ vk::ImageMemoryBarrier::default()
+ .old_layout(if resources.output_initialized {
+ vk::ImageLayout::SHADER_READ_ONLY_OPTIMAL
+ } else {
+ vk::ImageLayout::UNDEFINED
+ })
+ .new_layout(vk::ImageLayout::COLOR_ATTACHMENT_OPTIMAL)
+ .src_access_mask(if resources.output_initialized {
+ vk::AccessFlags::SHADER_READ
+ } else {
+ vk::AccessFlags::empty()
+ })
+ .dst_access_mask(vk::AccessFlags::COLOR_ATTACHMENT_WRITE)
+ .image(resources.output.image)
+ .subresource_range(image_color_range()),
+ );
+ unsafe {
+ self.device.cmd_pipeline_barrier(
+ command_buffer,
+ vk::PipelineStageFlags::ALL_COMMANDS,
+ vk::PipelineStageFlags::FRAGMENT_SHADER
+ | vk::PipelineStageFlags::COLOR_ATTACHMENT_OUTPUT,
+ vk::DependencyFlags::empty(),
+ &[],
+ &[],
+ &acquire,
+ );
+ self.device.cmd_begin_render_pass(
+ command_buffer,
+ &vk::RenderPassBeginInfo::default()
+ .render_pass(renderer.render_pass)
+ .framebuffer(resources.framebuffer)
+ .render_area(vk::Rect2D::default().extent(vk::Extent2D { width, height })),
+ vk::SubpassContents::INLINE,
+ );
+ self.device.cmd_set_viewport(
+ command_buffer,
+ 0,
+ &[vk::Viewport::default()
+ .width(width as f32)
+ .height(height as f32)
+ .max_depth(1.0)],
+ );
+ self.device.cmd_set_scissor(
+ command_buffer,
+ 0,
+ &[vk::Rect2D::default().extent(vk::Extent2D { width, height })],
+ );
+ self.device.cmd_bind_pipeline(
+ command_buffer,
+ vk::PipelineBindPoint::GRAPHICS,
+ renderer.pipeline,
+ );
+ self.device.cmd_bind_descriptor_sets(
+ command_buffer,
+ vk::PipelineBindPoint::GRAPHICS,
+ renderer.pipeline_layout,
+ 0,
+ &[renderer.descriptor_sets[slot]],
+ &[],
+ );
+ let constants = ConversionConstants {
+ texture_scale: [1.0, 1.0],
+ color_matrix: match color_matrix {
+ crate::ColorMatrix::Bt601 => 0,
+ crate::ColorMatrix::Bt709 => 1,
+ crate::ColorMatrix::Bt2020 => 2,
+ },
+ full_range: u32::from(full_range),
+ };
+ let constants = std::slice::from_raw_parts(
+ (&constants as *const ConversionConstants).cast::(),
+ size_of::(),
+ );
+ self.device.cmd_push_constants(
+ command_buffer,
+ renderer.pipeline_layout,
+ vk::ShaderStageFlags::FRAGMENT,
+ 0,
+ constants,
+ );
+ self.device.cmd_draw(command_buffer, 3, 1, 0, 0);
+ self.device.cmd_end_render_pass(command_buffer);
+ }
+ let mut release = vec![
+ vk::ImageMemoryBarrier::default()
+ .old_layout(vk::ImageLayout::COLOR_ATTACHMENT_OPTIMAL)
+ .new_layout(vk::ImageLayout::SHADER_READ_ONLY_OPTIMAL)
+ .src_access_mask(vk::AccessFlags::COLOR_ATTACHMENT_WRITE)
+ .dst_access_mask(vk::AccessFlags::SHADER_READ)
+ .image(resources.output.image)
+ .subresource_range(image_color_range()),
+ ];
+ match prepared {
+ PreparedLinuxFrame::Vulkan(frame) => {
+ release.extend(frame.images.iter().map(|image| {
+ let destination_family =
+ if image.source_queue_family == self.render.queue_family {
+ vk::QUEUE_FAMILY_IGNORED
+ } else {
+ image.source_queue_family
+ };
+ vk::ImageMemoryBarrier::default()
+ .old_layout(vk::ImageLayout::SHADER_READ_ONLY_OPTIMAL)
+ .new_layout(vk::ImageLayout::from_raw(image.old_layout))
+ .src_queue_family_index(if destination_family == vk::QUEUE_FAMILY_IGNORED {
+ vk::QUEUE_FAMILY_IGNORED
+ } else {
+ self.render.queue_family
+ })
+ .dst_queue_family_index(destination_family)
+ .src_access_mask(vk::AccessFlags::SHADER_READ)
+ .dst_access_mask(vk::AccessFlags::MEMORY_READ)
+ .image(vk::Image::from_raw(image.image))
+ .subresource_range(image_color_range())
+ }));
+ }
+ PreparedLinuxFrame::DmaBuf(frame) => release.push(
+ vk::ImageMemoryBarrier::default()
+ .old_layout(vk::ImageLayout::SHADER_READ_ONLY_OPTIMAL)
+ .new_layout(vk::ImageLayout::GENERAL)
+ .src_queue_family_index(self.render.queue_family)
+ .dst_queue_family_index(vk::QUEUE_FAMILY_EXTERNAL)
+ .src_access_mask(vk::AccessFlags::SHADER_READ)
+ .dst_access_mask(vk::AccessFlags::MEMORY_READ | vk::AccessFlags::MEMORY_WRITE)
+ .image(vk::Image::from_raw(frame.image))
+ .subresource_range(image_color_range()),
+ ),
+ PreparedLinuxFrame::Cpu(_) => {}
+ }
+ unsafe {
+ self.device.cmd_pipeline_barrier(
+ command_buffer,
+ vk::PipelineStageFlags::COLOR_ATTACHMENT_OUTPUT
+ | vk::PipelineStageFlags::FRAGMENT_SHADER,
+ vk::PipelineStageFlags::ALL_COMMANDS,
+ vk::DependencyFlags::empty(),
+ &[],
+ &[],
+ &release,
+ );
+ }
+ resources.output_initialized = true;
+ Ok(())
+ }
+
+ fn destroy_slot(&self, mut resources: FrameSlotResources) {
+ resources.frame = None;
+ unsafe {
+ for view in resources.owned_input_views.drain(..) {
+ self.device.destroy_image_view(view, None);
+ }
+ if let Some(cpu) = resources.cpu.take() {
+ self.device.destroy_buffer(cpu.staging_buffer, None);
+ self.device.free_memory(cpu.staging_memory, None);
+ destroy_gpu_image(&self.device, cpu.chroma);
+ destroy_gpu_image(&self.device, cpu.luma);
+ }
+ self.device.destroy_framebuffer(resources.framebuffer, None);
+ destroy_gpu_image(&self.device, resources.output);
+ }
+ }
+}
+
+#[repr(C)]
+struct ConversionConstants {
+ texture_scale: [f32; 2],
+ color_matrix: u32,
+ full_range: u32,
+}
+
+impl Drop for LinuxFrameProducer {
+ fn drop(&mut self) {
+ for resources in std::mem::take(&mut self.slots).into_iter().flatten() {
+ self.destroy_slot(resources);
+ }
+ if let Some(renderer) = self.renderer.take() {
+ unsafe {
+ self.device
+ .destroy_descriptor_pool(renderer.descriptor_pool, None);
+ self.device.destroy_sampler(renderer.sampler, None);
+ self.device.destroy_pipeline(renderer.pipeline, None);
+ self.device
+ .destroy_pipeline_layout(renderer.pipeline_layout, None);
+ self.device
+ .destroy_descriptor_set_layout(renderer.descriptor_set_layout, None);
+ self.device.destroy_render_pass(renderer.render_pass, None);
+ }
+ }
+ self.imported.clear();
+ }
+}
+
+fn direct_image_views(
+ device: &ash::Device,
+ images: &[PreparedVulkanImage],
+) -> Result<(vk::ImageView, vk::ImageView)> {
+ if images.len() == 1 {
+ let image = vk::Image::from_raw(images[0].image);
+ let luma = create_image_view(
+ device,
+ image,
+ vk::Format::R8_UNORM,
+ vk::ImageAspectFlags::PLANE_0,
+ )?;
+ let chroma = match create_image_view(
+ device,
+ image,
+ vk::Format::R8G8_UNORM,
+ vk::ImageAspectFlags::PLANE_1,
+ ) {
+ Ok(view) => view,
+ Err(error) => {
+ unsafe { device.destroy_image_view(luma, None) };
+ return Err(error);
+ }
+ };
+ return Ok((luma, chroma));
+ }
+ if images.len() != 2 {
+ return Err(Error::InvalidFormat(
+ "Vulkan NV12 frame must contain one multiplanar or two plane images".to_owned(),
+ ));
+ }
+ let luma = create_image_view(
+ device,
+ vk::Image::from_raw(images[0].image),
+ vk::Format::from_raw(images[0].format),
+ vk::ImageAspectFlags::COLOR,
+ )?;
+ let chroma = match create_image_view(
+ device,
+ vk::Image::from_raw(images[1].image),
+ vk::Format::from_raw(images[1].format),
+ vk::ImageAspectFlags::COLOR,
+ ) {
+ Ok(view) => view,
+ Err(error) => {
+ unsafe { device.destroy_image_view(luma, None) };
+ return Err(error);
+ }
+ };
+ Ok((luma, chroma))
+}
+
+fn create_gpu_image(
+ instance: &ash::Instance,
+ physical_device: vk::PhysicalDevice,
+ device: &ash::Device,
+ width: u32,
+ height: u32,
+ format: vk::Format,
+ usage: vk::ImageUsageFlags,
+) -> Result {
+ let image = unsafe {
+ device.create_image(
+ &vk::ImageCreateInfo::default()
+ .image_type(vk::ImageType::TYPE_2D)
+ .format(format)
+ .extent(vk::Extent3D {
+ width,
+ height,
+ depth: 1,
+ })
+ .mip_levels(1)
+ .array_layers(1)
+ .samples(vk::SampleCountFlags::TYPE_1)
+ .tiling(vk::ImageTiling::OPTIMAL)
+ .usage(usage)
+ .sharing_mode(vk::SharingMode::EXCLUSIVE)
+ .initial_layout(vk::ImageLayout::UNDEFINED),
+ None,
+ )
+ }
+ .map_err(|error| vk_error("create embedded GPU image", error))?;
+ let requirements = unsafe { device.get_image_memory_requirements(image) };
+ let memory_type = find_memory_type(
+ instance,
+ physical_device,
+ requirements.memory_type_bits,
+ vk::MemoryPropertyFlags::DEVICE_LOCAL,
+ )?;
+ let memory = unsafe {
+ device.allocate_memory(
+ &vk::MemoryAllocateInfo::default()
+ .allocation_size(requirements.size)
+ .memory_type_index(memory_type),
+ None,
+ )
+ }
+ .map_err(|error| vk_error("allocate embedded GPU image", error))?;
+ if let Err(error) = unsafe { device.bind_image_memory(image, memory, 0) } {
+ unsafe {
+ device.free_memory(memory, None);
+ device.destroy_image(image, None);
+ }
+ return Err(vk_error("bind embedded GPU image", error));
+ }
+ let view = match create_image_view(device, image, format, vk::ImageAspectFlags::COLOR) {
+ Ok(view) => view,
+ Err(error) => {
+ unsafe {
+ device.free_memory(memory, None);
+ device.destroy_image(image, None);
+ }
+ return Err(error);
+ }
+ };
+ Ok(GpuImage {
+ image,
+ memory,
+ view,
+ })
+}
+
+fn create_cpu_upload_resources(
+ instance: &ash::Instance,
+ physical_device: vk::PhysicalDevice,
+ device: &ash::Device,
+ width: u32,
+ height: u32,
+) -> Result {
+ let luma = create_gpu_image(
+ instance,
+ physical_device,
+ device,
+ width,
+ height,
+ vk::Format::R8_UNORM,
+ vk::ImageUsageFlags::TRANSFER_DST | vk::ImageUsageFlags::SAMPLED,
+ )?;
+ let chroma = create_gpu_image(
+ instance,
+ physical_device,
+ device,
+ width / 2,
+ height / 2,
+ vk::Format::R8G8_UNORM,
+ vk::ImageUsageFlags::TRANSFER_DST | vk::ImageUsageFlags::SAMPLED,
+ )?;
+ let size = u64::from(width) * u64::from(height) * 3 / 2;
+ let buffer = unsafe {
+ device.create_buffer(
+ &vk::BufferCreateInfo::default()
+ .size(size)
+ .usage(vk::BufferUsageFlags::TRANSFER_SRC)
+ .sharing_mode(vk::SharingMode::EXCLUSIVE),
+ None,
+ )
+ }
+ .map_err(|error| vk_error("create NV12 upload buffer", error))?;
+ let requirements = unsafe { device.get_buffer_memory_requirements(buffer) };
+ let memory_type = find_memory_type(
+ instance,
+ physical_device,
+ requirements.memory_type_bits,
+ vk::MemoryPropertyFlags::HOST_VISIBLE | vk::MemoryPropertyFlags::HOST_COHERENT,
+ )?;
+ let memory = unsafe {
+ device.allocate_memory(
+ &vk::MemoryAllocateInfo::default()
+ .allocation_size(requirements.size)
+ .memory_type_index(memory_type),
+ None,
+ )
+ }
+ .map_err(|error| vk_error("allocate NV12 upload buffer", error))?;
+ unsafe { device.bind_buffer_memory(buffer, memory, 0) }
+ .map_err(|error| vk_error("bind NV12 upload buffer", error))?;
+ Ok(CpuUploadResources {
+ luma,
+ chroma,
+ staging_buffer: buffer,
+ staging_memory: memory,
+ initialized: false,
+ })
+}
+
+fn upload_cpu_nv12(
+ device: &ash::Device,
+ command_buffer: vk::CommandBuffer,
+ resources: &mut CpuUploadResources,
+ frame: &CpuNv12Frame,
+ width: u32,
+ height: u32,
+) -> Result<()> {
+ let luma_len = width as usize * height as usize;
+ let total_len = luma_len + luma_len / 2;
+ let mapped = unsafe {
+ device.map_memory(
+ resources.staging_memory,
+ 0,
+ total_len as u64,
+ vk::MemoryMapFlags::empty(),
+ )
+ }
+ .map_err(|error| vk_error("map NV12 upload buffer", error))?;
+ unsafe {
+ copy_plane_rows(&frame.luma, mapped.cast(), width as usize, height as usize);
+ copy_plane_rows(
+ &frame.chroma,
+ mapped.cast::().add(luma_len),
+ width as usize,
+ height as usize / 2,
+ );
+ device.unmap_memory(resources.staging_memory);
+ }
+ let source_stage = if resources.initialized {
+ vk::PipelineStageFlags::FRAGMENT_SHADER
+ } else {
+ vk::PipelineStageFlags::TOP_OF_PIPE
+ };
+ let source_access = if resources.initialized {
+ vk::AccessFlags::SHADER_READ
+ } else {
+ vk::AccessFlags::empty()
+ };
+ let old_layout = if resources.initialized {
+ vk::ImageLayout::SHADER_READ_ONLY_OPTIMAL
+ } else {
+ vk::ImageLayout::UNDEFINED
+ };
+ let to_transfer = [resources.luma.image, resources.chroma.image].map(|image| {
+ vk::ImageMemoryBarrier::default()
+ .old_layout(old_layout)
+ .new_layout(vk::ImageLayout::TRANSFER_DST_OPTIMAL)
+ .src_access_mask(source_access)
+ .dst_access_mask(vk::AccessFlags::TRANSFER_WRITE)
+ .image(image)
+ .subresource_range(image_color_range())
+ });
+ unsafe {
+ device.cmd_pipeline_barrier(
+ command_buffer,
+ source_stage,
+ vk::PipelineStageFlags::TRANSFER,
+ vk::DependencyFlags::empty(),
+ &[],
+ &[],
+ &to_transfer,
+ );
+ device.cmd_copy_buffer_to_image(
+ command_buffer,
+ resources.staging_buffer,
+ resources.luma.image,
+ vk::ImageLayout::TRANSFER_DST_OPTIMAL,
+ &[vk::BufferImageCopy::default()
+ .buffer_offset(0)
+ .image_subresource(
+ vk::ImageSubresourceLayers::default()
+ .aspect_mask(vk::ImageAspectFlags::COLOR)
+ .layer_count(1),
+ )
+ .image_extent(vk::Extent3D {
+ width,
+ height,
+ depth: 1,
+ })],
+ );
+ device.cmd_copy_buffer_to_image(
+ command_buffer,
+ resources.staging_buffer,
+ resources.chroma.image,
+ vk::ImageLayout::TRANSFER_DST_OPTIMAL,
+ &[vk::BufferImageCopy::default()
+ .buffer_offset(luma_len as u64)
+ .image_subresource(
+ vk::ImageSubresourceLayers::default()
+ .aspect_mask(vk::ImageAspectFlags::COLOR)
+ .layer_count(1),
+ )
+ .image_extent(vk::Extent3D {
+ width: width / 2,
+ height: height / 2,
+ depth: 1,
+ })],
+ );
+ let to_sample = [resources.luma.image, resources.chroma.image].map(|image| {
+ vk::ImageMemoryBarrier::default()
+ .old_layout(vk::ImageLayout::TRANSFER_DST_OPTIMAL)
+ .new_layout(vk::ImageLayout::SHADER_READ_ONLY_OPTIMAL)
+ .src_access_mask(vk::AccessFlags::TRANSFER_WRITE)
+ .dst_access_mask(vk::AccessFlags::SHADER_READ)
+ .image(image)
+ .subresource_range(image_color_range())
+ });
+ device.cmd_pipeline_barrier(
+ command_buffer,
+ vk::PipelineStageFlags::TRANSFER,
+ vk::PipelineStageFlags::FRAGMENT_SHADER,
+ vk::DependencyFlags::empty(),
+ &[],
+ &[],
+ &to_sample,
+ );
+ }
+ resources.initialized = true;
+ Ok(())
+}
+
+unsafe fn copy_plane_rows(plane: &FramePlane, destination: *mut u8, row_bytes: usize, rows: usize) {
+ for row in 0..rows {
+ unsafe {
+ std::ptr::copy_nonoverlapping(
+ plane.data.as_ptr().add(row * plane.stride),
+ destination.add(row * row_bytes),
+ row_bytes,
+ );
+ }
+ }
+}
+
+fn create_image_view(
+ device: &ash::Device,
+ image: vk::Image,
+ format: vk::Format,
+ aspect: vk::ImageAspectFlags,
+) -> Result {
+ unsafe {
+ device.create_image_view(
+ &vk::ImageViewCreateInfo::default()
+ .image(image)
+ .view_type(vk::ImageViewType::TYPE_2D)
+ .format(format)
+ .subresource_range(
+ vk::ImageSubresourceRange::default()
+ .aspect_mask(aspect)
+ .level_count(1)
+ .layer_count(1),
+ ),
+ None,
+ )
+ }
+ .map_err(|error| vk_error("create embedded image view", error))
+}
+
+fn destroy_gpu_image(device: &ash::Device, image: GpuImage) {
+ unsafe {
+ device.destroy_image_view(image.view, None);
+ device.destroy_image(image.image, None);
+ device.free_memory(image.memory, None);
+ }
+}
+
+fn find_memory_type(
+ instance: &ash::Instance,
+ physical_device: vk::PhysicalDevice,
+ memory_type_bits: u32,
+ required: vk::MemoryPropertyFlags,
+) -> Result {
+ let properties = unsafe { instance.get_physical_device_memory_properties(physical_device) };
+ (0..properties.memory_type_count)
+ .find(|index| {
+ memory_type_bits & (1 << index) != 0
+ && properties.memory_types[*index as usize]
+ .property_flags
+ .contains(required)
+ })
+ .ok_or_else(|| Error::unavailable(Subsystem::Vulkan, "no compatible Vulkan memory type"))
+}
+
+fn image_color_range() -> vk::ImageSubresourceRange {
+ vk::ImageSubresourceRange::default()
+ .aspect_mask(vk::ImageAspectFlags::COLOR)
+ .level_count(1)
+ .layer_count(1)
+}
+
+fn timeline_waits(frame: &VulkanVideoFrame) -> Result> {
+ let mut waits = HashMap::::new();
+ for image in &frame.images {
+ if image.semaphore == 0 || image.semaphore_value == 0 {
+ return Err(Error::unavailable(
+ Subsystem::Vulkan,
+ "Vulkan decoder frame has no host-waitable timeline semaphore",
+ ));
+ }
+ match waits.entry(image.semaphore) {
+ HashMapEntry::Occupied(mut entry) => {
+ *entry.get_mut() = (*entry.get()).max(image.semaphore_value);
+ }
+ HashMapEntry::Vacant(entry) => {
+ entry.insert(image.semaphore_value);
+ }
+ }
+ }
+ let mut waits = waits.into_iter().collect::>();
+ waits.sort_unstable_by_key(|(semaphore, _)| *semaphore);
+ Ok(waits)
+}
+
+fn nv12_dmabuf_layout(frame: &DmaBufFrame) -> Result<(usize, DmaBufPlane, DmaBufPlane)> {
+ let (luma, chroma) = if frame.layers.len() == 1
+ && frame.layers[0].format == DRM_FORMAT_NV12
+ && frame.layers[0].planes.len() >= 2
+ {
+ (frame.layers[0].planes[0], frame.layers[0].planes[1])
+ } else if frame.layers.len() >= 2
+ && frame.layers[0].format == DRM_FORMAT_R8
+ && frame.layers[1].format == DRM_FORMAT_GR88
+ && !frame.layers[0].planes.is_empty()
+ && !frame.layers[1].planes.is_empty()
+ {
+ (frame.layers[0].planes[0], frame.layers[1].planes[0])
+ } else {
+ return Err(Error::unavailable(
+ Subsystem::Vulkan,
+ "unsupported DMA-BUF NV12 plane layout",
+ ));
+ };
+ if luma.object_index != chroma.object_index {
+ return Err(Error::unavailable(
+ Subsystem::Vulkan,
+ "disjoint multi-object NV12 DMA-BUF import is not supported",
+ ));
+ }
+ Ok((luma.object_index, luma, chroma))
+}
+
+fn fd_identity(fd: RawFd) -> Result<(u64, u64)> {
+ let mut metadata = std::mem::MaybeUninit::::zeroed();
+ if unsafe { libc::fstat(fd, metadata.as_mut_ptr()) } != 0 {
+ return Err(Error::backend(
+ Subsystem::Vulkan,
+ format!(
+ "fstat on DMA-BUF failed: {}",
+ std::io::Error::last_os_error()
+ ),
+ ));
+ }
+ let metadata = unsafe { metadata.assume_init() };
+ Ok((metadata.st_dev, metadata.st_ino))
+}
+
+#[allow(clippy::too_many_arguments)]
+fn import_nv12_dmabuf(
+ instance: &ash::Instance,
+ physical_device: vk::PhysicalDevice,
+ device: &ash::Device,
+ render_queue_family: u32,
+ fd: RawFd,
+ object_size: usize,
+ modifier: u64,
+ luma: DmaBufPlane,
+ chroma: DmaBufPlane,
+ source: Arc,
+) -> Result {
+ if luma.offset >= object_size || chroma.offset >= object_size {
+ return Err(Error::InvalidFormat(
+ "DMA-BUF plane offset exceeds its object size".to_owned(),
+ ));
+ }
+ let plane_layouts = [
+ vk::SubresourceLayout {
+ offset: luma.offset as u64,
+ size: object_size.saturating_sub(luma.offset) as u64,
+ row_pitch: luma.pitch as u64,
+ array_pitch: 0,
+ depth_pitch: 0,
+ },
+ vk::SubresourceLayout {
+ offset: chroma.offset as u64,
+ size: object_size.saturating_sub(chroma.offset) as u64,
+ row_pitch: chroma.pitch as u64,
+ array_pitch: 0,
+ depth_pitch: 0,
+ },
+ ];
+ let mut external = vk::ExternalMemoryImageCreateInfo::default()
+ .handle_types(vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT);
+ let mut drm_modifier = vk::ImageDrmFormatModifierExplicitCreateInfoEXT::default()
+ .drm_format_modifier(modifier)
+ .plane_layouts(&plane_layouts);
+ let image_info = vk::ImageCreateInfo::default()
+ .push_next(&mut external)
+ .push_next(&mut drm_modifier)
+ .flags(vk::ImageCreateFlags::MUTABLE_FORMAT)
+ .image_type(vk::ImageType::TYPE_2D)
+ .format(vk::Format::G8_B8R8_2PLANE_420_UNORM)
+ .extent(vk::Extent3D {
+ width: source.format.width,
+ height: source.format.height,
+ depth: 1,
+ })
+ .mip_levels(1)
+ .array_layers(1)
+ .samples(vk::SampleCountFlags::TYPE_1)
+ .tiling(vk::ImageTiling::DRM_FORMAT_MODIFIER_EXT)
+ .usage(vk::ImageUsageFlags::SAMPLED)
+ .sharing_mode(vk::SharingMode::EXCLUSIVE)
+ .initial_layout(vk::ImageLayout::UNDEFINED);
+ let image = unsafe { device.create_image(&image_info, None) }
+ .map_err(|error| vk_error("create DMA-BUF NV12 image", error))?;
+ let requirements = unsafe { device.get_image_memory_requirements(image) };
+ let fd_loader = ash::khr::external_memory_fd::Device::new(instance, device);
+ let mut fd_properties = vk::MemoryFdPropertiesKHR::default();
+ if let Err(error) = unsafe {
+ fd_loader.get_memory_fd_properties(
+ vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT,
+ fd,
+ &mut fd_properties,
+ )
+ } {
+ unsafe { device.destroy_image(image, None) };
+ return Err(vk_error("query DMA-BUF memory properties", error));
+ }
+ let memory_bits = requirements.memory_type_bits & fd_properties.memory_type_bits;
+ let properties = unsafe { instance.get_physical_device_memory_properties(physical_device) };
+ let memory_type = (0..properties.memory_type_count)
+ .find(|index| memory_bits & (1 << index) != 0)
+ .ok_or_else(|| {
+ Error::unavailable(Subsystem::Vulkan, "no memory type can import DMA-BUF")
+ })?;
+ let imported_fd = unsafe { libc::fcntl(fd, libc::F_DUPFD_CLOEXEC, 0) };
+ if imported_fd < 0 {
+ unsafe { device.destroy_image(image, None) };
+ return Err(Error::backend(
+ Subsystem::Vulkan,
+ format!(
+ "duplicate DMA-BUF failed: {}",
+ std::io::Error::last_os_error()
+ ),
+ ));
+ }
+ let mut import = vk::ImportMemoryFdInfoKHR::default()
+ .handle_type(vk::ExternalMemoryHandleTypeFlags::DMA_BUF_EXT)
+ .fd(imported_fd);
+ let mut dedicated = vk::MemoryDedicatedAllocateInfo::default().image(image);
+ let allocation = vk::MemoryAllocateInfo::default()
+ .push_next(&mut import)
+ .push_next(&mut dedicated)
+ .allocation_size(requirements.size.max(object_size as u64))
+ .memory_type_index(memory_type);
+ let memory = match unsafe { device.allocate_memory(&allocation, None) } {
+ Ok(memory) => memory,
+ Err(error) => {
+ unsafe {
+ libc::close(imported_fd);
+ device.destroy_image(image, None);
+ }
+ return Err(vk_error("import DMA-BUF memory", error));
+ }
+ };
+ if let Err(error) = unsafe { device.bind_image_memory(image, memory, 0) } {
+ unsafe {
+ device.free_memory(memory, None);
+ device.destroy_image(image, None);
+ }
+ return Err(vk_error("bind DMA-BUF image memory", error));
+ }
+ let luma_view = match create_plane_view(
+ device,
+ image,
+ vk::Format::R8_UNORM,
+ vk::ImageAspectFlags::PLANE_0,
+ ) {
+ Ok(view) => view,
+ Err(error) => {
+ unsafe {
+ device.free_memory(memory, None);
+ device.destroy_image(image, None);
+ }
+ return Err(error);
+ }
+ };
+ let chroma_view = match create_plane_view(
+ device,
+ image,
+ vk::Format::R8G8_UNORM,
+ vk::ImageAspectFlags::PLANE_1,
+ ) {
+ Ok(view) => view,
+ Err(error) => {
+ unsafe {
+ device.destroy_image_view(luma_view, None);
+ device.free_memory(memory, None);
+ device.destroy_image(image, None);
+ }
+ return Err(error);
+ }
+ };
+ Ok(ImportedNv12Frame {
+ image: image.as_raw(),
+ luma_view: luma_view.as_raw(),
+ chroma_view: chroma_view.as_raw(),
+ modifier,
+ external_queue_family: vk::QUEUE_FAMILY_EXTERNAL,
+ render_queue_family,
+ image_handle: image,
+ luma_view_handle: luma_view,
+ chroma_view_handle: chroma_view,
+ memory,
+ device: device.clone(),
+ source,
+ })
+}
+
+fn create_plane_view(
+ device: &ash::Device,
+ image: vk::Image,
+ format: vk::Format,
+ aspect: vk::ImageAspectFlags,
+) -> Result {
+ let range = vk::ImageSubresourceRange::default()
+ .aspect_mask(aspect)
+ .level_count(1)
+ .layer_count(1);
+ let create = vk::ImageViewCreateInfo::default()
+ .image(image)
+ .view_type(vk::ImageViewType::TYPE_2D)
+ .format(format)
+ .subresource_range(range);
+ unsafe { device.create_image_view(&create, None) }
+ .map_err(|error| vk_error("create DMA-BUF plane image view", error))
+}
+
+fn vk_error(context: &'static str, error: vk::Result) -> Error {
+ Error::backend(Subsystem::Vulkan, format!("{context}: {error:?}"))
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::{DmaBufLayer, DmaBufObject, VulkanImage};
+
+ fn vulkan_frame(images: Vec) -> VulkanVideoFrame {
+ VulkanVideoFrame::new(1, 2, 3, vec![0], 0, 0, images, 0, None, None, Arc::new(()))
+ }
+
+ #[test]
+ fn timeline_waits_deduplicate_images_and_keep_latest_value() {
+ let frame = vulkan_frame(vec![
+ VulkanImage {
+ image: 10,
+ format: 1,
+ width: 4,
+ height: 4,
+ layout: 1,
+ access: 1,
+ semaphore: 22,
+ semaphore_value: 7,
+ queue_family: 0,
+ },
+ VulkanImage {
+ image: 11,
+ format: 2,
+ width: 2,
+ height: 2,
+ layout: 1,
+ access: 1,
+ semaphore: 22,
+ semaphore_value: 9,
+ queue_family: 0,
+ },
+ ]);
+ assert_eq!(timeline_waits(&frame).unwrap(), vec![(22, 9)]);
+ }
+
+ #[test]
+ fn frame_without_timeline_signal_cannot_be_rendered_unsynchronized() {
+ let frame = vulkan_frame(vec![VulkanImage {
+ image: 10,
+ format: 1,
+ width: 4,
+ height: 4,
+ layout: 1,
+ access: 1,
+ semaphore: 0,
+ semaphore_value: 0,
+ queue_family: 0,
+ }]);
+ assert!(timeline_waits(&frame).is_err());
+ }
+
+ #[test]
+ fn dmabuf_layout_accepts_single_and_split_nv12_layers() {
+ let object = DmaBufObject {
+ fd: 0,
+ size: 4096,
+ format_modifier: 12,
+ };
+ let luma = DmaBufPlane {
+ object_index: 0,
+ offset: 0,
+ pitch: 64,
+ };
+ let chroma = DmaBufPlane {
+ object_index: 0,
+ offset: 2048,
+ pitch: 64,
+ };
+ let packed = DmaBufFrame::new(
+ vec![object],
+ vec![DmaBufLayer {
+ format: DRM_FORMAT_NV12,
+ planes: vec![luma, chroma],
+ }],
+ Arc::new(()),
+ );
+ assert_eq!(nv12_dmabuf_layout(&packed).unwrap(), (0, luma, chroma));
+
+ let split = DmaBufFrame::new(
+ vec![object],
+ vec![
+ DmaBufLayer {
+ format: DRM_FORMAT_R8,
+ planes: vec![luma],
+ },
+ DmaBufLayer {
+ format: DRM_FORMAT_GR88,
+ planes: vec![chroma],
+ },
+ ],
+ Arc::new(()),
+ );
+ assert_eq!(nv12_dmabuf_layout(&split).unwrap(), (0, luma, chroma));
+ }
+
+ #[test]
+ fn dmabuf_layout_rejects_disjoint_objects() {
+ let frame = DmaBufFrame::new(
+ vec![
+ DmaBufObject {
+ fd: 0,
+ size: 4096,
+ format_modifier: 0,
+ },
+ DmaBufObject {
+ fd: 1,
+ size: 4096,
+ format_modifier: 0,
+ },
+ ],
+ vec![DmaBufLayer {
+ format: DRM_FORMAT_NV12,
+ planes: vec![
+ DmaBufPlane {
+ object_index: 0,
+ offset: 0,
+ pitch: 64,
+ },
+ DmaBufPlane {
+ object_index: 1,
+ offset: 0,
+ pitch: 64,
+ },
+ ],
+ }],
+ Arc::new(()),
+ );
+ assert!(nv12_dmabuf_layout(&frame).is_err());
+ }
+}
diff --git a/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/lib.rs b/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/lib.rs
index 09f40ec3e..a465da16b 100644
--- a/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/lib.rs
+++ b/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/lib.rs
@@ -13,6 +13,8 @@ mod error;
#[cfg(target_os = "linux")]
mod format;
#[cfg(all(target_os = "linux", feature = "vulkan"))]
+mod frame_producer;
+#[cfg(all(target_os = "linux", feature = "vulkan"))]
mod presentation;
#[cfg(target_os = "linux")]
mod queue;
@@ -37,6 +39,12 @@ pub use format::{
VideoCodec, VulkanImage, VulkanVideoFrame,
};
#[cfg(all(target_os = "linux", feature = "vulkan"))]
+pub use frame_producer::{
+ CpuNv12Frame, ImportedNv12Frame, LinuxFrameProducer, LinuxGpuFrame, LinuxGpuFrameProducer,
+ PreparedLinuxFrame, PreparedVulkanFrame, PreparedVulkanImage, RecordedGpuFrame,
+ VulkanRenderDevice,
+};
+#[cfg(all(target_os = "linux", feature = "vulkan"))]
pub use presentation::{NativeSurface, VulkanPresenter};
#[cfg(target_os = "linux")]
pub use session::{
diff --git a/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/video/ffmpeg.rs b/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/video/ffmpeg.rs
index f14a63a52..0cedefb29 100644
--- a/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/video/ffmpeg.rs
+++ b/native/opennow-streamer/crates/opennow-streamer-platform-linux/src/video/ffmpeg.rs
@@ -1,5 +1,5 @@
use std::ptr;
-use std::sync::{Arc, OnceLock};
+use std::sync::{Arc, Mutex, OnceLock};
use ffmpeg::codec;
use ffmpeg::ffi;
@@ -210,23 +210,23 @@ impl FfmpegDecoder {
}
fn convert_frame(&mut self, decoded: &frame::Video) -> Result {
+ let mut direct_vulkan = None;
if self.mode == FfmpegMode::Vulkan
&& self
.wanted_hw_format
.as_deref()
.is_some_and(|selection| Pixel::from(selection.pixel_format) == decoded.format())
{
- if let Ok(frame) = map_vulkan_frame_direct(decoded, self.last_timestamp_us) {
- if !self.zero_copy_active {
- eprintln!("Vulkan Video same-device zero-copy enabled");
- self.zero_copy_active = true;
- }
- return Ok(frame);
- }
+ direct_vulkan = map_vulkan_frame_direct(decoded, self.last_timestamp_us)
+ .ok()
+ .and_then(|frame| frame.vulkan);
match map_vulkan_frame_to_dmabuf(decoded, self.last_timestamp_us) {
- Ok(frame) => {
+ Ok(mut frame) => {
+ frame.vulkan = direct_vulkan.clone();
if !self.zero_copy_active {
- eprintln!("Vulkan Video zero-copy enabled through DRM PRIME/DMA-BUF");
+ eprintln!(
+ "Vulkan Video frames expose same-device and DRM PRIME/DMA-BUF paths"
+ );
self.zero_copy_active = true;
}
return Ok(frame);
@@ -346,7 +346,7 @@ impl FfmpegDecoder {
},
],
dmabuf: None,
- vulkan: None,
+ vulkan: direct_vulkan,
timestamp_us,
})
}
@@ -431,6 +431,7 @@ fn map_vulkan_frame_direct(
"failed to retain Vulkan Video frame for presentation",
));
}
+ let cpu_fallback = vulkan_cpu_nv12_fallback(decoded)?;
let device = unsafe {
VulkanVideoFrame::new(
(*vulkan_device).inst as usize,
@@ -455,7 +456,8 @@ fn map_vulkan_frame_direct(
}),
Arc::new(retained),
)
- };
+ }
+ .with_cpu_nv12_fallback(cpu_fallback);
let output_format = StreamFormat {
width,
height,
@@ -479,6 +481,74 @@ fn map_vulkan_frame_direct(
Ok(output)
}
+fn vulkan_cpu_nv12_fallback(
+ decoded: &frame::Video,
+) -> Result Result> + Send + Sync>> {
+ let mut retained = frame::Video::empty();
+ if unsafe { ffi::av_frame_ref(retained.as_mut_ptr(), decoded.as_ptr()) } < 0 {
+ return Err(Error::backend(
+ Subsystem::Ffmpeg,
+ "failed to retain Vulkan frame for CPU fallback",
+ ));
+ }
+ let retained = Mutex::new(retained);
+ Ok(Arc::new(move || {
+ let source = retained.lock().unwrap_or_else(|poison| poison.into_inner());
+ let mut transferred = frame::Video::empty();
+ let transfer_result =
+ unsafe { ffi::av_hwframe_transfer_data(transferred.as_mut_ptr(), source.as_ptr(), 0) };
+ if transfer_result < 0 {
+ return Err(ffmpeg_error(
+ "Vulkan frame CPU fallback download failed".to_owned(),
+ transfer_result,
+ ));
+ }
+ let width = transferred.width().max(1);
+ let height = transferred.height().max(1);
+ let converted;
+ let nv12 = if transferred.format() == Pixel::NV12 {
+ &transferred
+ } else {
+ let mut scaler = Scaler::get(
+ transferred.format(),
+ width,
+ height,
+ Pixel::NV12,
+ width,
+ height,
+ ScaleFlags::BILINEAR,
+ )
+ .map_err(|error| {
+ Error::backend(
+ Subsystem::Ffmpeg,
+ format!("Vulkan CPU fallback conversion setup failed: {error}"),
+ )
+ })?;
+ let mut output = frame::Video::empty();
+ scaler.run(&transferred, &mut output).map_err(|error| {
+ Error::backend(
+ Subsystem::Ffmpeg,
+ format!("Vulkan CPU fallback NV12 conversion failed: {error}"),
+ )
+ })?;
+ converted = output;
+ &converted
+ };
+ Ok(vec![
+ FramePlane {
+ data: Arc::from(nv12.data(0).to_vec()),
+ stride: nv12.stride(0),
+ rows: height as usize,
+ },
+ FramePlane {
+ data: Arc::from(nv12.data(1).to_vec()),
+ stride: nv12.stride(1),
+ rows: height as usize / 2,
+ },
+ ])
+ }))
+}
+
fn map_vulkan_frame_to_dmabuf(
decoded: &frame::Video,
fallback_timestamp_us: u64,
diff --git a/native/opennow-streamer/crates/opennow-streamer-platform-linux/tests/linux_api.rs b/native/opennow-streamer/crates/opennow-streamer-platform-linux/tests/linux_api.rs
index 3e6e8669c..a650fd9a7 100644
--- a/native/opennow-streamer/crates/opennow-streamer-platform-linux/tests/linux_api.rs
+++ b/native/opennow-streamer/crates/opennow-streamer-platform-linux/tests/linux_api.rs
@@ -10,6 +10,10 @@ use std::sync::Arc;
assert_impl_all!(LinuxSession: Send);
#[cfg(feature = "vulkan")]
+assert_impl_all!(opennow_streamer_platform_linux::LinuxGpuFrame: Send, Sync);
+#[cfg(feature = "vulkan")]
+assert_impl_all!(opennow_streamer_platform_linux::LinuxGpuFrameProducer: Send, Sync);
+#[cfg(feature = "vulkan")]
assert_not_impl_any!(opennow_streamer_platform_linux::NativeSurface<'static>: Send, Sync);
#[test]
@@ -28,3 +32,57 @@ fn linux_compile_target_is_supported() {
let _: fn() -> opennow_streamer_platform_linux::CapabilityReport =
opennow_streamer_platform_linux::probe_capabilities;
}
+
+#[cfg(feature = "vulkan")]
+#[test]
+fn embedded_gpu_producer_bounds_qrhi_frame_slots() {
+ use opennow_streamer_platform_linux::LinuxGpuFrameProducer;
+
+ assert!(LinuxGpuFrameProducer::new(1).is_ok());
+ assert!(LinuxGpuFrameProducer::new(8).is_ok());
+ assert!(LinuxGpuFrameProducer::new(0).is_err());
+ assert!(LinuxGpuFrameProducer::new(9).is_err());
+}
+
+#[cfg(feature = "vulkan")]
+#[test]
+fn embedded_gpu_frames_keep_sequence_and_presentation_time() {
+ use opennow_streamer_platform_linux::{
+ ChromaLocation, ColorMatrix, ColorRange, DecodedVideoFrame, FramePlane,
+ LinuxGpuFrameProducer, PixelFormat,
+ };
+
+ let producer = LinuxGpuFrameProducer::new(3).unwrap();
+ let decoded = DecodedVideoFrame {
+ format: StreamFormat {
+ width: 4,
+ height: 4,
+ pixel_format: PixelFormat::Nv12,
+ color_range: ColorRange::Limited,
+ color_matrix: ColorMatrix::Bt709,
+ chroma_location: ChromaLocation::Left,
+ },
+ planes: vec![
+ FramePlane {
+ data: Arc::from(vec![16_u8; 16]),
+ stride: 4,
+ rows: 4,
+ },
+ FramePlane {
+ data: Arc::from(vec![128_u8; 8]),
+ stride: 4,
+ rows: 2,
+ },
+ ],
+ dmabuf: None,
+ vulkan: None,
+ timestamp_us: 42,
+ };
+ let first = producer.frame(decoded.clone()).unwrap();
+ let second = producer.frame(decoded).unwrap();
+
+ assert_eq!(first.sequence(), 1);
+ assert_eq!(second.sequence(), 2);
+ assert_eq!(first.presentation_time_ns(), 42_000);
+ assert_eq!((first.width(), first.height()), (4, 4));
+}
diff --git a/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/format.rs b/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/format.rs
index cba06b050..b13dcd887 100644
--- a/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/format.rs
+++ b/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/format.rs
@@ -478,6 +478,20 @@ pub struct BackendConfig {
pub queues: QueueLimits,
}
+#[derive(Clone, Debug, PartialEq)]
+pub struct EmbeddedBackendConfig {
+ pub video: VideoFormat,
+ pub audio: AudioFormat,
+ pub queues: QueueLimits,
+}
+
+impl EmbeddedBackendConfig {
+ pub(crate) fn validate(&self) -> Result<(), FormatError> {
+ self.audio.validate()?;
+ self.queues.validate()
+ }
+}
+
impl BackendConfig {
pub(crate) fn validate(&self) -> Result<(), FormatError> {
self.audio.validate()?;
diff --git a/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/lib.rs b/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/lib.rs
index f96b0d165..683634b09 100644
--- a/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/lib.rs
+++ b/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/lib.rs
@@ -40,10 +40,10 @@ mod macos;
pub use failure::{BackendFailure, BackendSubsystem, VideoDecodeLoss};
pub use format::{
- AudioFormat, Av1Format, BackendConfig, BorrowedNsView, BorrowedNsWindow, FrameTiming,
- H264Format, H264Framing, H264ParameterSets, H265Format, H265ParameterSets, OwnedOverlayConfig,
- QueueLimits, RendererRect, ScreenRect, SurfaceTarget, VideoColorSpace, VideoFormat,
- WindowSurfaceConfig,
+ AudioFormat, Av1Format, BackendConfig, BorrowedNsView, BorrowedNsWindow, EmbeddedBackendConfig,
+ FrameTiming, H264Format, H264Framing, H264ParameterSets, H265Format, H265ParameterSets,
+ OwnedOverlayConfig, QueueLimits, RendererRect, ScreenRect, SurfaceTarget, VideoColorSpace,
+ VideoFormat, WindowSurfaceConfig,
};
pub use lifecycle::BackendState;
@@ -56,7 +56,8 @@ pub(crate) const fn overlay_should_be_ordered(
#[cfg(target_os = "macos")]
pub use macos::{
- BackendError, BackendStats, MacOsBackend, NativeSurfaceHandle, StreamSink, SubmitOutcome,
+ AdoptedMetalContext, BackendError, BackendStats, EmbeddedFrameProducer, MacOsBackend,
+ MetalFrame, MetalRecordedFrame, NativeSurfaceHandle, StreamSink, SubmitOutcome,
activate_stream_application, probe_av1_hardware, probe_h264_hardware, probe_h265_hardware,
pump_app_events,
};
diff --git a/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/macos/embedded.rs b/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/macos/embedded.rs
new file mode 100644
index 000000000..f6bd567cc
--- /dev/null
+++ b/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/macos/embedded.rs
@@ -0,0 +1,585 @@
+use std::collections::HashMap;
+use std::ffi::c_void;
+use std::ptr::{self, NonNull};
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::{Arc, Mutex};
+
+use objc2::Message;
+use objc2::rc::Retained;
+use objc2::runtime::ProtocolObject;
+use objc2_core_foundation::CFRetained;
+use objc2_core_video::{
+ CVMetalTexture, CVMetalTextureCache, CVMetalTextureGetTexture, CVPixelBufferGetHeightOfPlane,
+ CVPixelBufferGetIOSurface, CVPixelBufferGetPixelFormatType, CVPixelBufferGetPlaneCount,
+ CVPixelBufferGetWidthOfPlane, kCVPixelFormatType_420YpCbCr8BiPlanarFullRange,
+ kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange,
+ kCVPixelFormatType_420YpCbCr10BiPlanarFullRange,
+ kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange,
+};
+use objc2_foundation::NSString;
+use objc2_metal::{
+ MTLCommandBuffer, MTLCommandEncoder, MTLDevice, MTLLibrary, MTLLoadAction, MTLPixelFormat,
+ MTLPrimitiveType, MTLRenderCommandEncoder, MTLRenderPassDescriptor,
+ MTLRenderPipelineDescriptor, MTLRenderPipelineState, MTLStorageMode, MTLStoreAction,
+ MTLTexture, MTLTextureDescriptor, MTLTextureUsage, MTLViewport,
+};
+
+use crate::format::VideoColorSpace;
+
+use super::mailbox::LatestMailbox;
+use super::video::DecodedFrame;
+use super::{BackendError, Counters};
+
+const MAX_RETAINED_FRAME_SLOTS: usize = 8;
+
+fn validate_frame_slot(frame_slot: u32) -> Result<(), BackendError> {
+ if frame_slot < MAX_RETAINED_FRAME_SLOTS as u32 {
+ Ok(())
+ } else {
+ Err(BackendError::Metal(format!(
+ "Metal frame slot {frame_slot} exceeds the eight-slot limit"
+ )))
+ }
+}
+
+const SHADER_SOURCE: &str = r#"
+#include
+using namespace metal;
+
+struct VertexOut {
+ float4 position [[position]];
+ float2 texcoord;
+};
+
+struct ConversionParameters {
+ uint color_space;
+ uint sample_format;
+ uint full_range;
+};
+
+vertex VertexOut embedded_video_vertex(uint vertex_id [[vertex_id]]) {
+ const float2 positions[3] = { float2(-1.0, -1.0), float2(3.0, -1.0), float2(-1.0, 3.0) };
+ const float2 texcoords[3] = { float2(0.0, 1.0), float2(2.0, 1.0), float2(0.0, -1.0) };
+ VertexOut out;
+ out.position = float4(positions[vertex_id], 0.0, 1.0);
+ out.texcoord = texcoords[vertex_id];
+ return out;
+}
+
+fragment float4 embedded_video_fragment(
+ VertexOut in [[stage_in]],
+ texture2d luma [[texture(0)]],
+ texture2d chroma [[texture(1)]],
+ constant ConversionParameters ¶meters [[buffer(0)]]) {
+ constexpr sampler linear_sampler(coord::normalized, address::clamp_to_edge, filter::linear);
+ float y = luma.sample(linear_sampler, in.texcoord).r;
+ float2 cbcr = chroma.sample(linear_sampler, in.texcoord).rg;
+ if (parameters.full_range == 0) {
+ if (parameters.sample_format == 0) {
+ y = (y - (16.0 / 255.0)) * (255.0 / 219.0);
+ cbcr -= float2(0.5);
+ } else {
+ y = (y - (64.0 / 1023.0)) * (1023.0 / 876.0);
+ cbcr -= float2(512.0 / 1023.0);
+ }
+ } else {
+ cbcr -= parameters.sample_format == 0
+ ? float2(128.0 / 255.0)
+ : float2(512.0 / 1023.0);
+ }
+ float3 rgb;
+ if (parameters.color_space == 0) {
+ rgb = float3(
+ y + 1.596027 * cbcr.y,
+ y - 0.391762 * cbcr.x - 0.812968 * cbcr.y,
+ y + 2.017232 * cbcr.x);
+ } else {
+ rgb = float3(
+ y + 1.792741 * cbcr.y,
+ y - 0.213249 * cbcr.x - 0.532909 * cbcr.y,
+ y + 2.112402 * cbcr.x);
+ }
+ return float4(saturate(rgb), 1.0);
+}
+"#;
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+enum BiPlanarFormat {
+ Nv12VideoRange,
+ Nv12FullRange,
+ P010VideoRange,
+ P010FullRange,
+}
+
+impl BiPlanarFormat {
+ fn from_pixel_format(pixel_format: u32) -> Option {
+ if pixel_format == kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange {
+ Some(Self::Nv12VideoRange)
+ } else if pixel_format == kCVPixelFormatType_420YpCbCr8BiPlanarFullRange {
+ Some(Self::Nv12FullRange)
+ } else if pixel_format == kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange {
+ Some(Self::P010VideoRange)
+ } else if pixel_format == kCVPixelFormatType_420YpCbCr10BiPlanarFullRange {
+ Some(Self::P010FullRange)
+ } else {
+ None
+ }
+ }
+
+ const fn plane_formats(self) -> (MTLPixelFormat, MTLPixelFormat) {
+ match self {
+ Self::Nv12VideoRange | Self::Nv12FullRange => {
+ (MTLPixelFormat::R8Unorm, MTLPixelFormat::RG8Unorm)
+ }
+ Self::P010VideoRange | Self::P010FullRange => {
+ (MTLPixelFormat::R16Unorm, MTLPixelFormat::RG16Unorm)
+ }
+ }
+ }
+
+ const fn parameters(self, color_space: VideoColorSpace) -> ConversionParameters {
+ ConversionParameters {
+ color_space: match color_space {
+ VideoColorSpace::Bt601 => 0,
+ VideoColorSpace::Bt709 => 1,
+ },
+ sample_format: match self {
+ Self::Nv12VideoRange | Self::Nv12FullRange => 0,
+ Self::P010VideoRange | Self::P010FullRange => 1,
+ },
+ full_range: match self {
+ Self::Nv12VideoRange | Self::P010VideoRange => 0,
+ Self::Nv12FullRange | Self::P010FullRange => 1,
+ },
+ }
+ }
+}
+
+#[repr(C)]
+#[derive(Clone, Copy)]
+struct ConversionParameters {
+ color_space: u32,
+ sample_format: u32,
+ full_range: u32,
+}
+
+#[derive(Clone, Copy, Debug)]
+pub struct AdoptedMetalContext {
+ pub device: *mut c_void,
+ pub command_buffer: *mut c_void,
+}
+
+#[derive(Clone, Copy, Debug, Eq, PartialEq)]
+pub struct MetalRecordedFrame {
+ pub texture: *mut c_void,
+ pub width: u32,
+ pub height: u32,
+ pub frame_slot: u32,
+ pub generation: u64,
+ pub presentation_time_ns: u64,
+}
+
+pub struct MetalFrame {
+ frame: DecodedFrame,
+ state: Arc>>,
+ counters: Arc,
+ sequence: u64,
+}
+
+impl MetalFrame {
+ pub fn width(&self) -> u32 {
+ u32::try_from(CVPixelBufferGetWidthOfPlane(&self.frame.image, 0)).unwrap_or(0)
+ }
+
+ pub fn height(&self) -> u32 {
+ u32::try_from(CVPixelBufferGetHeightOfPlane(&self.frame.image, 0)).unwrap_or(0)
+ }
+
+ pub const fn sequence(&self) -> u64 {
+ self.sequence
+ }
+
+ pub fn presentation_time_ns(&self) -> u64 {
+ u64::try_from(self.frame.timestamp_100ns.max(0))
+ .unwrap_or(0)
+ .saturating_mul(100)
+ }
+
+ /// Records IOSurface plane sampling and YUV-to-RGBA conversion into a retained slot texture.
+ ///
+ /// The function creates no command queue and does not commit or wait for the command buffer.
+ /// It must run before Qt begins the render pass that samples the returned texture.
+ ///
+ /// # Safety
+ ///
+ /// Both pointers must identify live Metal objects from the same Qt QRhi frame, and the command
+ /// buffer must be open for encoding. `frame_slot` may be reused only when Qt has retired its
+ /// previous GPU work for that slot.
+ pub unsafe fn record(
+ &self,
+ adopted: AdoptedMetalContext,
+ frame_slot: u32,
+ ) -> Result {
+ if adopted.device.is_null() || adopted.command_buffer.is_null() {
+ return Err(BackendError::Metal(
+ "Qt supplied a null Metal device or command buffer".into(),
+ ));
+ }
+ validate_frame_slot(frame_slot)?;
+ let device_ptr = NonNull::new(adopted.device)
+ .expect("validated device")
+ .cast::>();
+ let command_buffer_ptr = NonNull::new(adopted.command_buffer)
+ .expect("validated command buffer")
+ .cast::>();
+ let device = unsafe { device_ptr.as_ref() };
+ let command_buffer = unsafe { command_buffer_ptr.as_ref() };
+
+ let mut state = self
+ .state
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner());
+ let device_identity = device as *const _ as *const () as usize;
+ if state
+ .as_ref()
+ .is_none_or(|state| state.device_identity != device_identity)
+ {
+ *state = Some(MetalState::new(device, device_identity)?);
+ }
+ let recorded = state.as_mut().expect("initialized Metal state").record(
+ self.frame.clone(),
+ command_buffer,
+ frame_slot,
+ )?;
+ self.counters
+ .video_metal_submitted
+ .fetch_add(1, Ordering::Relaxed);
+ Ok(MetalRecordedFrame {
+ presentation_time_ns: self.presentation_time_ns(),
+ ..recorded
+ })
+ }
+}
+
+#[derive(Clone)]
+pub struct EmbeddedFrameProducer {
+ mailbox: Arc>,
+ state: Arc>>,
+ sequence: Arc,
+ counters: Arc,
+}
+
+impl EmbeddedFrameProducer {
+ pub(super) fn new(mailbox: Arc>, counters: Arc) -> Self {
+ Self {
+ mailbox,
+ state: Arc::new(Mutex::new(None)),
+ sequence: Arc::new(AtomicU64::new(0)),
+ counters,
+ }
+ }
+
+ pub fn acquire_latest(&self) -> Option {
+ let frame = self.mailbox.take()?;
+ let sequence = self.sequence.fetch_add(1, Ordering::AcqRel) + 1;
+ Some(MetalFrame {
+ frame,
+ state: Arc::clone(&self.state),
+ counters: Arc::clone(&self.counters),
+ sequence,
+ })
+ }
+
+ pub fn clear(&self) -> bool {
+ self.mailbox.clear()
+ }
+
+ /// Releases device-bound caches after Qt has invalidated and drained its scene graph.
+ pub fn release_graphics_resources(&self) {
+ *self
+ .state
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner()) = None;
+ }
+
+ pub(super) fn mailbox(&self) -> &Arc> {
+ &self.mailbox
+ }
+
+ pub(super) fn counters(&self) -> &Arc {
+ &self.counters
+ }
+}
+
+struct SlotResources {
+ output: Retained>,
+ _frame: DecodedFrame,
+ _luma_cv_texture: CFRetained,
+ _chroma_cv_texture: CFRetained,
+ _luma_texture: Retained>,
+ _chroma_texture: Retained>,
+}
+
+struct MetalState {
+ device_identity: usize,
+ _device: Retained>,
+ pipeline: Retained>,
+ texture_cache: CFRetained,
+ slots: HashMap,
+ generation: u64,
+}
+
+unsafe impl Send for MetalState {}
+
+impl MetalState {
+ fn new(
+ device: &ProtocolObject,
+ device_identity: usize,
+ ) -> Result {
+ let device = device.retain();
+ let source = NSString::from_str(SHADER_SOURCE);
+ let library = device
+ .newLibraryWithSource_options_error(&source, None)
+ .map_err(|error| BackendError::Metal(error.localizedDescription().to_string()))?;
+ let vertex = library
+ .newFunctionWithName(&NSString::from_str("embedded_video_vertex"))
+ .ok_or_else(|| BackendError::Metal("embedded_video_vertex shader is missing".into()))?;
+ let fragment = library
+ .newFunctionWithName(&NSString::from_str("embedded_video_fragment"))
+ .ok_or_else(|| {
+ BackendError::Metal("embedded_video_fragment shader is missing".into())
+ })?;
+ let descriptor = MTLRenderPipelineDescriptor::new();
+ descriptor.setVertexFunction(Some(&vertex));
+ descriptor.setFragmentFunction(Some(&fragment));
+ let attachments = descriptor.colorAttachments();
+ let attachment = unsafe { attachments.objectAtIndexedSubscript(0) };
+ attachment.setPixelFormat(MTLPixelFormat::RGBA8Unorm);
+ let pipeline = device
+ .newRenderPipelineStateWithDescriptor_error(&descriptor)
+ .map_err(|error| BackendError::Metal(error.localizedDescription().to_string()))?;
+
+ let mut cache_ptr = ptr::null_mut();
+ let status = unsafe {
+ CVMetalTextureCache::create(None, None, &device, None, NonNull::from(&mut cache_ptr))
+ };
+ if status != 0 {
+ return Err(BackendError::AppleApi {
+ api: "CVMetalTextureCacheCreate",
+ status,
+ });
+ }
+ let cache_ptr = NonNull::new(cache_ptr).ok_or(BackendError::AppleApi {
+ api: "CVMetalTextureCacheCreate",
+ status: -1,
+ })?;
+ Ok(Self {
+ device_identity,
+ _device: device,
+ pipeline,
+ texture_cache: unsafe { CFRetained::from_raw(cache_ptr) },
+ slots: HashMap::with_capacity(3),
+ generation: 0,
+ })
+ }
+
+ fn record(
+ &mut self,
+ frame: DecodedFrame,
+ command_buffer: &ProtocolObject,
+ frame_slot: u32,
+ ) -> Result {
+ if CVPixelBufferGetPlaneCount(&frame.image) != 2 {
+ return Err(BackendError::Metal(
+ "VideoToolbox returned a non-bi-planar pixel buffer".into(),
+ ));
+ }
+ if CVPixelBufferGetIOSurface(Some(&frame.image)).is_none() {
+ return Err(BackendError::Metal(
+ "VideoToolbox returned a pixel buffer without IOSurface backing".into(),
+ ));
+ }
+ let format =
+ BiPlanarFormat::from_pixel_format(CVPixelBufferGetPixelFormatType(&frame.image))
+ .ok_or_else(|| {
+ BackendError::Metal("VideoToolbox returned neither NV12 nor P010".into())
+ })?;
+ let width = CVPixelBufferGetWidthOfPlane(&frame.image, 0);
+ let height = CVPixelBufferGetHeightOfPlane(&frame.image, 0);
+ let chroma_width = CVPixelBufferGetWidthOfPlane(&frame.image, 1);
+ let chroma_height = CVPixelBufferGetHeightOfPlane(&frame.image, 1);
+ let (luma_format, chroma_format) = format.plane_formats();
+ let luma_cv_texture = self.make_plane_texture(&frame, luma_format, width, height, 0)?;
+ let chroma_cv_texture =
+ self.make_plane_texture(&frame, chroma_format, chroma_width, chroma_height, 1)?;
+ let luma_texture = CVMetalTextureGetTexture(&luma_cv_texture)
+ .ok_or_else(|| BackendError::Metal("failed to get Metal luma texture".into()))?;
+ let chroma_texture = CVMetalTextureGetTexture(&chroma_cv_texture)
+ .ok_or_else(|| BackendError::Metal("failed to get Metal chroma texture".into()))?;
+
+ let output = self
+ .slots
+ .remove(&frame_slot)
+ .map(|resources| resources.output)
+ .filter(|texture| texture.width() == width && texture.height() == height)
+ .map_or_else(
+ || {
+ let descriptor = unsafe {
+ MTLTextureDescriptor::texture2DDescriptorWithPixelFormat_width_height_mipmapped(
+ MTLPixelFormat::RGBA8Unorm,
+ width,
+ height,
+ false,
+ )
+ };
+ descriptor.setStorageMode(MTLStorageMode::Private);
+ descriptor.setUsage(
+ MTLTextureUsage::RenderTarget | MTLTextureUsage::ShaderRead,
+ );
+ self._device
+ .newTextureWithDescriptor(&descriptor)
+ .ok_or_else(|| {
+ BackendError::Metal("failed to create embedded RGBA texture".into())
+ })
+ },
+ Ok,
+ )?;
+
+ let render_pass = MTLRenderPassDescriptor::renderPassDescriptor();
+ let attachments = render_pass.colorAttachments();
+ let attachment = unsafe { attachments.objectAtIndexedSubscript(0) };
+ attachment.setTexture(Some(&output));
+ attachment.setLoadAction(MTLLoadAction::DontCare);
+ attachment.setStoreAction(MTLStoreAction::Store);
+ let encoder = command_buffer
+ .renderCommandEncoderWithDescriptor(&render_pass)
+ .ok_or_else(|| BackendError::Metal("failed to create embedded Metal encoder".into()))?;
+ encoder.setRenderPipelineState(&self.pipeline);
+ unsafe {
+ encoder.setFragmentTexture_atIndex(Some(&luma_texture), 0);
+ encoder.setFragmentTexture_atIndex(Some(&chroma_texture), 1);
+ }
+ let parameters = format.parameters(frame.color_space);
+ unsafe {
+ encoder.setFragmentBytes_length_atIndex(
+ NonNull::from(¶meters).cast::(),
+ std::mem::size_of_val(¶meters),
+ 0,
+ );
+ }
+ encoder.setViewport(MTLViewport {
+ originX: 0.0,
+ originY: 0.0,
+ width: width as f64,
+ height: height as f64,
+ znear: 0.0,
+ zfar: 1.0,
+ });
+ unsafe { encoder.drawPrimitives_vertexStart_vertexCount(MTLPrimitiveType::Triangle, 0, 3) };
+ encoder.endEncoding();
+
+ let texture = Retained::as_ptr(&output).cast_mut().cast::();
+ self.slots.insert(
+ frame_slot,
+ SlotResources {
+ output,
+ _frame: frame,
+ _luma_cv_texture: luma_cv_texture,
+ _chroma_cv_texture: chroma_cv_texture,
+ _luma_texture: luma_texture,
+ _chroma_texture: chroma_texture,
+ },
+ );
+ self.generation = self.generation.wrapping_add(1);
+ Ok(MetalRecordedFrame {
+ texture,
+ width: u32::try_from(width).unwrap_or(u32::MAX),
+ height: u32::try_from(height).unwrap_or(u32::MAX),
+ frame_slot,
+ generation: self.generation,
+ presentation_time_ns: 0,
+ })
+ }
+
+ fn make_plane_texture(
+ &self,
+ frame: &DecodedFrame,
+ pixel_format: MTLPixelFormat,
+ width: usize,
+ height: usize,
+ plane: usize,
+ ) -> Result, BackendError> {
+ let mut texture_ptr = ptr::null_mut();
+ let status = unsafe {
+ CVMetalTextureCache::create_texture_from_image(
+ None,
+ &self.texture_cache,
+ &frame.image,
+ None,
+ pixel_format,
+ width,
+ height,
+ plane,
+ NonNull::from(&mut texture_ptr),
+ )
+ };
+ if status != 0 {
+ return Err(BackendError::AppleApi {
+ api: "CVMetalTextureCacheCreateTextureFromImage",
+ status,
+ });
+ }
+ let texture_ptr = NonNull::new(texture_ptr).ok_or(BackendError::AppleApi {
+ api: "CVMetalTextureCacheCreateTextureFromImage",
+ status: -1,
+ })?;
+ Ok(unsafe { CFRetained::from_raw(texture_ptr) })
+ }
+}
+
+impl Drop for MetalState {
+ fn drop(&mut self) {
+ self.slots.clear();
+ self.texture_cache.flush(0);
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::{BiPlanarFormat, MAX_RETAINED_FRAME_SLOTS, validate_frame_slot};
+ use objc2_core_video::{
+ kCVPixelFormatType_420YpCbCr8BiPlanarFullRange,
+ kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange,
+ kCVPixelFormatType_420YpCbCr10BiPlanarFullRange,
+ kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange,
+ };
+
+ #[test]
+ fn accepts_nv12_and_p010_ranges_only() {
+ assert_eq!(
+ BiPlanarFormat::from_pixel_format(kCVPixelFormatType_420YpCbCr8BiPlanarVideoRange),
+ Some(BiPlanarFormat::Nv12VideoRange)
+ );
+ assert_eq!(
+ BiPlanarFormat::from_pixel_format(kCVPixelFormatType_420YpCbCr8BiPlanarFullRange),
+ Some(BiPlanarFormat::Nv12FullRange)
+ );
+ assert_eq!(
+ BiPlanarFormat::from_pixel_format(kCVPixelFormatType_420YpCbCr10BiPlanarVideoRange),
+ Some(BiPlanarFormat::P010VideoRange)
+ );
+ assert_eq!(
+ BiPlanarFormat::from_pixel_format(kCVPixelFormatType_420YpCbCr10BiPlanarFullRange),
+ Some(BiPlanarFormat::P010FullRange)
+ );
+ assert_eq!(BiPlanarFormat::from_pixel_format(0), None);
+ }
+
+ #[test]
+ fn retained_slot_bound_matches_public_embedded_limit() {
+ assert_eq!(MAX_RETAINED_FRAME_SLOTS, 8);
+ assert!(validate_frame_slot(0).is_ok());
+ assert!(validate_frame_slot(7).is_ok());
+ assert!(validate_frame_slot(8).is_err());
+ }
+}
diff --git a/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/macos/mailbox.rs b/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/macos/mailbox.rs
new file mode 100644
index 000000000..3f9e78402
--- /dev/null
+++ b/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/macos/mailbox.rs
@@ -0,0 +1,55 @@
+use std::sync::Mutex;
+
+pub(super) struct LatestMailbox {
+ value: Mutex>,
+}
+
+impl LatestMailbox {
+ pub(super) fn new() -> Self {
+ Self {
+ value: Mutex::new(None),
+ }
+ }
+
+ pub(super) fn replace(&self, value: T) -> bool {
+ self.value
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner())
+ .replace(value)
+ .is_some()
+ }
+
+ pub(super) fn take(&self) -> Option {
+ self.value
+ .lock()
+ .unwrap_or_else(|poisoned| poisoned.into_inner())
+ .take()
+ }
+
+ pub(super) fn clear(&self) -> bool {
+ self.take().is_some()
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::LatestMailbox;
+
+ #[test]
+ fn replaces_stale_value_and_consumes_latest_once() {
+ let mailbox = LatestMailbox::new();
+ assert!(!mailbox.replace(1));
+ assert!(mailbox.replace(2));
+ assert_eq!(mailbox.take(), Some(2));
+ assert_eq!(mailbox.take(), None);
+ }
+
+ #[test]
+ fn clear_reports_whether_a_value_was_discarded() {
+ let mailbox = LatestMailbox::new();
+ assert!(!mailbox.clear());
+ mailbox.replace(3);
+ assert!(mailbox.clear());
+ assert!(!mailbox.clear());
+ }
+}
diff --git a/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/macos/mod.rs b/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/macos/mod.rs
index 91d13f89a..08c16d0ac 100644
--- a/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/macos/mod.rs
+++ b/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/macos/mod.rs
@@ -1,4 +1,6 @@
mod audio;
+mod embedded;
+mod mailbox;
mod presentation;
mod surface;
mod video;
@@ -14,16 +16,21 @@ use thiserror::Error;
use crate::failure::{BackendFailure, FailureReporter, VideoDecodeLoss};
use crate::format::{
- AudioFormat, Av1Format, BackendConfig, FormatError, FrameTiming, H264Format, H264Framing,
- H265Format, RendererRect, ScreenRect, VideoFormat, access_unit_to_avcc,
+ AudioFormat, Av1Format, BackendConfig, EmbeddedBackendConfig, FormatError, FrameTiming,
+ H264Format, H264Framing, H265Format, RendererRect, ScreenRect, VideoFormat,
+ access_unit_to_avcc,
};
use crate::lifecycle::{BackendState, Lifecycle};
use crate::queue::{BoundedQueue, PushResult};
use self::audio::AudioPipeline;
+pub use self::embedded::{
+ AdoptedMetalContext, EmbeddedFrameProducer, MetalFrame, MetalRecordedFrame,
+};
+use self::mailbox::LatestMailbox;
use self::presentation::PresenterHandle;
use self::surface::SurfaceOwner;
-use self::video::{DecodedFrame, VideoDecoder};
+use self::video::{DecodedFrameOutput, VideoDecoder};
const MAX_OPUS_PACKET_BYTES: usize = 1_275;
@@ -371,7 +378,7 @@ impl StreamSink {
}
let replacement = VideoDecoder::new(
&format,
- self.shared.video_queue.clone(),
+ self.shared.video_output.clone(),
Arc::clone(&self.shared.counters),
Arc::clone(&self.shared.failures),
self.shared.video_frames_in_flight,
@@ -387,7 +394,7 @@ impl StreamSink {
}
let previous = decoder.replace(replacement);
drop(previous);
- let discarded = self.shared.video_queue.clear();
+ let discarded = self.shared.video_output.clear();
self.shared
.counters
.video_frames_dropped
@@ -444,7 +451,7 @@ struct Shared {
paused: AtomicBool,
counters: Arc,
failures: Arc,
- video_queue: Arc>,
+ video_output: DecodedFrameOutput,
video: Mutex>,
audio: Mutex>,
presenter: Mutex>,
@@ -465,6 +472,10 @@ impl Shared {
.unwrap_or_else(|poisoned| poisoned.into_inner())
.take();
drop(decoder);
+ let discarded = self.video_output.clear();
+ self.counters
+ .video_frames_dropped
+ .fetch_add(discarded as u64, Ordering::Relaxed);
if let Some(audio) = self
.audio
.lock()
@@ -489,6 +500,7 @@ impl Shared {
pub struct MacOsBackend {
shared: Arc,
surface: Option,
+ frame_producer: Option,
_main_thread_only: PhantomData>,
}
@@ -500,6 +512,7 @@ impl MacOsBackend {
let counters = Arc::new(Counters::default());
let failures = Arc::new(FailureReporter::default());
let video_queue = Arc::new(BoundedQueue::new(config.queues.decoded_video_frames));
+ let video_output = DecodedFrameOutput::PresentationQueue(Arc::clone(&video_queue));
let presenter = PresenterHandle::start(
surface.metal_layer(),
surface.presentation_visibility(),
@@ -509,7 +522,7 @@ impl MacOsBackend {
)?;
let video = VideoDecoder::new(
&config.video,
- Arc::clone(&video_queue),
+ video_output.clone(),
Arc::clone(&counters),
Arc::clone(&failures),
config.queues.video_frames_in_flight,
@@ -526,7 +539,7 @@ impl MacOsBackend {
paused: AtomicBool::new(false),
counters,
failures,
- video_queue,
+ video_output,
video: Mutex::new(Some(video)),
audio: Mutex::new(Some(audio)),
presenter: Mutex::new(Some(presenter)),
@@ -538,6 +551,94 @@ impl MacOsBackend {
Ok(Self {
shared,
surface: Some(surface),
+ frame_producer: None,
+ _main_thread_only: PhantomData,
+ })
+ }
+
+ /// Starts VideoToolbox and CoreAudio for a shell-owned Qt/Metal renderer.
+ ///
+ /// This path creates no AppKit object, SDL window, `CAMetalLayer`, `CVDisplayLink`, Metal
+ /// device, or Metal command queue. Decoded IOSurface-backed frames are retained in a
+ /// latest-frame mailbox until [`EmbeddedFrameProducer::acquire_latest`] transfers them to Qt's
+ /// render thread.
+ pub fn start_embedded(config: EmbeddedBackendConfig) -> Result {
+ Self::start_embedded_inner(config, None:: bool>)
+ }
+
+ /// Starts embedded output and forwards each newest retained frame to a shell mailbox.
+ ///
+ /// The publisher runs on VideoToolbox's callback thread and must return promptly. Returning
+ /// `true` indicates that the shell replaced an older unconsumed frame.
+ pub fn start_embedded_with_publisher(
+ config: EmbeddedBackendConfig,
+ publish: impl Fn(MetalFrame) -> bool + Send + Sync + 'static,
+ ) -> Result {
+ Self::start_embedded_inner(config, Some(publish))
+ }
+
+ fn start_embedded_inner(
+ config: EmbeddedBackendConfig,
+ publish: Option bool + Send + Sync + 'static>,
+ ) -> Result {
+ config.validate()?;
+ let counters = Arc::new(Counters::default());
+ let failures = Arc::new(FailureReporter::default());
+ let mailbox = Arc::new(LatestMailbox::new());
+ let frame_producer = EmbeddedFrameProducer::new(mailbox, Arc::clone(&counters));
+ let frame_available = publish.map(|publish| {
+ let producer = frame_producer.clone();
+ Arc::new(move || {
+ if let Some(frame) = producer.acquire_latest() {
+ let replaced = publish(frame);
+ if replaced {
+ let counters = producer.counters();
+ counters
+ .video_decoded_queue_dropped
+ .fetch_add(1, Ordering::Relaxed);
+ counters
+ .video_frames_dropped
+ .fetch_add(1, Ordering::Relaxed);
+ }
+ }
+ }) as Arc
+ });
+ let video_output = DecodedFrameOutput::EmbeddedMailbox {
+ mailbox: Arc::clone(frame_producer.mailbox()),
+ frame_available,
+ };
+ let video = VideoDecoder::new(
+ &config.video,
+ video_output.clone(),
+ Arc::clone(&counters),
+ Arc::clone(&failures),
+ config.queues.video_frames_in_flight,
+ )?;
+ let audio = AudioPipeline::start(
+ config.audio,
+ config.queues.opus_packets,
+ config.queues.pcm_milliseconds,
+ Arc::clone(&counters),
+ Arc::clone(&failures),
+ )?;
+ let shared = Arc::new(Shared {
+ lifecycle: Lifecycle::running(),
+ paused: AtomicBool::new(false),
+ counters,
+ failures,
+ video_output,
+ video: Mutex::new(Some(video)),
+ audio: Mutex::new(Some(audio)),
+ presenter: Mutex::new(None),
+ video_frames_in_flight: config.queues.video_frames_in_flight,
+ opus_packets: config.queues.opus_packets,
+ pcm_milliseconds: config.queues.pcm_milliseconds,
+ max_video_access_unit_bytes: config.queues.max_video_access_unit_bytes,
+ });
+ Ok(Self {
+ shared,
+ surface: None,
+ frame_producer: Some(frame_producer),
_main_thread_only: PhantomData,
})
}
@@ -552,6 +653,10 @@ impl MacOsBackend {
self.surface.as_ref().map(SurfaceOwner::native_handle)
}
+ pub fn frame_producer(&self) -> Option {
+ self.frame_producer.clone()
+ }
+
/// Updates a supplied-window child surface in renderer-relative, top-left AppKit points.
///
/// This method is available only for `SurfaceTarget::NsWindow` and returns
@@ -613,14 +718,16 @@ impl MacOsBackend {
}
pub fn set_paused(&mut self, paused: bool) -> Result<(), BackendError> {
- let _main_thread = MainThreadMarker::new().ok_or(BackendError::MainThreadRequired)?;
+ if self.surface.is_some() && MainThreadMarker::new().is_none() {
+ return Err(BackendError::MainThreadRequired);
+ }
if self.shared.lifecycle.state() != BackendState::Running {
return Err(BackendError::Stopped);
}
if paused {
self.shared.paused.store(true, Ordering::Release);
}
- let discarded = self.shared.video_queue.clear();
+ let discarded = self.shared.video_output.clear();
self.shared
.counters
.video_frames_dropped
diff --git a/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/macos/video.rs b/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/macos/video.rs
index 55566e896..1fc44f0d5 100644
--- a/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/macos/video.rs
+++ b/native/opennow-streamer/crates/opennow-streamer-platform-macos/src/macos/video.rs
@@ -27,16 +27,57 @@ use crate::failure::FailureReporter;
use crate::format::{Av1Format, FrameTiming, H264Format, H265Format, VideoColorSpace, VideoFormat};
use crate::queue::{BoundedQueue, PushResult};
+use super::mailbox::LatestMailbox;
use super::{BackendError, Counters};
+#[derive(Clone)]
pub(super) struct DecodedFrame {
pub(super) image: CFRetained,
pub(super) color_space: VideoColorSpace,
pub(super) minimum_frame_duration_seconds: f64,
+ pub(super) timestamp_100ns: i64,
}
// The callback retains the CVImageBuffer and no code mutates it after publication to the queue.
unsafe impl Send for DecodedFrame {}
+unsafe impl Sync for DecodedFrame {}
+
+#[derive(Clone)]
+pub(super) enum DecodedFrameOutput {
+ PresentationQueue(Arc>),
+ EmbeddedMailbox {
+ mailbox: Arc>,
+ frame_available: Option>,
+ },
+}
+
+impl DecodedFrameOutput {
+ fn publish(&self, frame: DecodedFrame) -> bool {
+ match self {
+ Self::PresentationQueue(queue) => matches!(
+ queue.push_drop_oldest(frame),
+ PushResult::Replaced(_) | PushResult::Closed(_)
+ ),
+ Self::EmbeddedMailbox {
+ mailbox,
+ frame_available,
+ } => {
+ let replaced = mailbox.replace(frame);
+ if let Some(frame_available) = frame_available {
+ frame_available();
+ }
+ replaced
+ }
+ }
+ }
+
+ pub(super) fn clear(&self) -> usize {
+ match self {
+ Self::PresentationQueue(queue) => queue.clear(),
+ Self::EmbeddedMailbox { mailbox, .. } => usize::from(mailbox.clear()),
+ }
+ }
+}
struct InFlight {
count: AtomicUsize,
@@ -59,7 +100,7 @@ impl InFlight {
}
struct CallbackContext {
- queue: Arc>,
+ output: DecodedFrameOutput,
counters: Arc,
failures: Arc,
in_flight: Arc,
@@ -80,7 +121,7 @@ unsafe impl Send for VideoDecoder {}
impl VideoDecoder {
pub(super) fn new(
format: &VideoFormat,
- queue: Arc>,
+ output: DecodedFrameOutput,
counters: Arc,
failures: Arc,
maximum_in_flight: usize,
@@ -91,7 +132,7 @@ impl VideoDecoder {
maximum: maximum_in_flight,
});
let mut callback_context = Box::new(CallbackContext {
- queue,
+ output,
counters,
failures,
in_flight: Arc::clone(&in_flight),
@@ -120,14 +161,15 @@ impl VideoDecoder {
let io_surface_properties_key = unsafe { kCVPixelBufferIOSurfacePropertiesKey };
let hardware_decoder_key =
unsafe { kVTVideoDecoderSpecification_RequireHardwareAcceleratedVideoDecoder };
- let destination_attributes = make_dictionary(&[
- (cf_ptr(pixel_format_key), cf_ptr(&*pixel_format_number)),
+ let mut destination_entries = vec![
(cf_ptr(metal_compatibility_key), cf_ptr(true_value)),
(
cf_ptr(io_surface_properties_key),
cf_ptr(&*empty_properties),
),
- ])?;
+ ];
+ destination_entries.insert(0, (cf_ptr(pixel_format_key), cf_ptr(&*pixel_format_number)));
+ let destination_attributes = make_dictionary(&destination_entries)?;
let decoder_specification =
make_dictionary(&[(cf_ptr(hardware_decoder_key), cf_ptr(true_value))])?;
@@ -271,7 +313,7 @@ unsafe extern "C-unwind" fn decompression_callback(
status: i32,
_info_flags: VTDecodeInfoFlags,
image_buffer: *mut CVImageBuffer,
- _presentation_time_stamp: CMTime,
+ presentation_time_stamp: CMTime,
presentation_duration: CMTime,
) {
let Some(context) = NonNull::new(output_refcon.cast::()) else {
@@ -285,16 +327,14 @@ unsafe extern "C-unwind" fn decompression_callback(
image,
color_space: context.color_space,
minimum_frame_duration_seconds: frame_duration_seconds(presentation_duration),
+ timestamp_100ns: time_to_100ns(presentation_time_stamp),
};
context
.counters
.video_decoded
.fetch_add(1, Ordering::Relaxed);
context.failures.video_decode_succeeded();
- if matches!(
- context.queue.push_drop_oldest(frame),
- PushResult::Replaced(_) | PushResult::Closed(_)
- ) {
+ if context.output.publish(frame) {
context
.counters
.video_frames_dropped
@@ -329,9 +369,26 @@ fn frame_duration_seconds(duration: CMTime) -> f64 {
}
}
+fn time_to_100ns(time: CMTime) -> i64 {
+ if time.timescale <= 0 {
+ return 0;
+ }
+ i128::from(time.value)
+ .saturating_mul(10_000_000)
+ .checked_div(i128::from(time.timescale))
+ .and_then(|value| i64::try_from(value).ok())
+ .unwrap_or_else(|| {
+ if time.value.is_negative() {
+ i64::MIN
+ } else {
+ i64::MAX
+ }
+ })
+}
+
#[cfg(test)]
mod tests {
- use super::frame_duration_seconds;
+ use super::{frame_duration_seconds, time_to_100ns};
use objc2_core_media::{CMTime, CMTimeFlags};
#[test]
@@ -344,6 +401,17 @@ mod tests {
};
assert!((frame_duration_seconds(duration) - 1.0 / 120.0).abs() < f64::EPSILON);
}
+
+ #[test]
+ fn converts_core_media_time_to_cross_platform_100ns_units() {
+ let time = CMTime {
+ value: 90_000,
+ timescale: 90_000,
+ flags: CMTimeFlags(1),
+ epoch: 0,
+ };
+ assert_eq!(time_to_100ns(time), 10_000_000);
+ }
}
fn create_format_description(
diff --git a/native/opennow-streamer/crates/opennow-streamer-platform-windows/src/lib.rs b/native/opennow-streamer/crates/opennow-streamer-platform-windows/src/lib.rs
index 2281fc953..3aff6f5f5 100644
--- a/native/opennow-streamer/crates/opennow-streamer-platform-windows/src/lib.rs
+++ b/native/opennow-streamer/crates/opennow-streamer-platform-windows/src/lib.rs
@@ -6,6 +6,11 @@ mod queue;
#[cfg(windows)]
mod windows;
+#[cfg(windows)]
+pub use windows::{
+ AdoptedD3d11Context, D3d11Frame, D3d11FrameProducer, D3d11FrameSubmitter, D3d11RecordedFrame,
+};
+
use std::sync::atomic::{AtomicU8, AtomicU64, Ordering};
use std::sync::{Arc, Mutex, mpsc};
use std::thread::JoinHandle;
diff --git a/native/opennow-streamer/crates/opennow-streamer-platform-windows/src/windows/decoder.rs b/native/opennow-streamer/crates/opennow-streamer-platform-windows/src/windows/decoder.rs
index c163c8c03..fca183c15 100644
--- a/native/opennow-streamer/crates/opennow-streamer-platform-windows/src/windows/decoder.rs
+++ b/native/opennow-streamer/crates/opennow-streamer-platform-windows/src/windows/decoder.rs
@@ -33,7 +33,11 @@ use crate::{
VideoCodec, VideoFormat, VideoPixelFormat, WindowsDecoderMode,
};
-use super::graphics::Graphics;
+pub(super) trait DecoderDevice {
+ fn device_manager(&self) -> &::windows::Win32::Media::MediaFoundation::IMFDXGIDeviceManager;
+ fn adapter_luid(&self) -> Result;
+ fn video_format(&self) -> VideoFormat;
+}
pub(super) struct DecodedVideoFrame {
pub(super) texture: ID3D11Texture2D,
@@ -56,8 +60,8 @@ pub(super) struct Decoder {
}
impl Decoder {
- pub(super) fn probe(
- graphics: &Graphics,
+ pub(super) fn probe(
+ graphics: &G,
codec: VideoCodec,
mode: WindowsDecoderMode,
) -> Result<(), String> {
@@ -73,8 +77,8 @@ impl Decoder {
Ok(())
}
- pub(super) fn new(
- graphics: &Graphics,
+ pub(super) fn new(
+ graphics: &G,
format: VideoFormat,
mode: WindowsDecoderMode,
) -> Result {
@@ -360,22 +364,21 @@ impl Drop for Decoder {
}
}
-fn configure_transform(
+type ConfiguredTransform = (
+ IMFTransform,
+ Option,
+ u32,
+ u32,
+ bool,
+ VideoPixelFormat,
+ bool,
+);
+
+fn configure_transform(
activation: &IMFActivate,
- graphics: &Graphics,
+ graphics: &G,
format: VideoFormat,
-) -> Result<
- (
- IMFTransform,
- Option,
- u32,
- u32,
- bool,
- VideoPixelFormat,
- bool,
- ),
- String,
-> {
+) -> Result {
unsafe {
let transform: IMFTransform = activation
.ActivateObject()
diff --git a/native/opennow-streamer/crates/opennow-streamer-platform-windows/src/windows/embedded.rs b/native/opennow-streamer/crates/opennow-streamer-platform-windows/src/windows/embedded.rs
new file mode 100644
index 000000000..2b9ef2177
--- /dev/null
+++ b/native/opennow-streamer/crates/opennow-streamer-platform-windows/src/windows/embedded.rs
@@ -0,0 +1,934 @@
+use std::collections::{HashMap, VecDeque};
+use std::ffi::c_void;
+use std::mem::ManuallyDrop;
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::{Arc, Mutex};
+use std::thread::{self, ThreadId};
+
+use ::windows::Win32::Foundation::{LUID, RECT};
+use ::windows::Win32::Graphics::Direct3D11::{
+ D3D11_BIND_RENDER_TARGET, D3D11_BIND_SHADER_RESOURCE, D3D11_TEX2D_VPIV, D3D11_TEX2D_VPOV,
+ D3D11_TEXTURE2D_DESC, D3D11_USAGE_DEFAULT, D3D11_VIDEO_FRAME_FORMAT_PROGRESSIVE,
+ D3D11_VIDEO_PROCESSOR_CONTENT_DESC, D3D11_VIDEO_PROCESSOR_FORMAT_SUPPORT_OUTPUT,
+ D3D11_VIDEO_PROCESSOR_INPUT_VIEW_DESC, D3D11_VIDEO_PROCESSOR_INPUT_VIEW_DESC_0,
+ D3D11_VIDEO_PROCESSOR_OUTPUT_VIEW_DESC, D3D11_VIDEO_PROCESSOR_OUTPUT_VIEW_DESC_0,
+ D3D11_VIDEO_PROCESSOR_STREAM, D3D11_VIDEO_USAGE_OPTIMAL_SPEED, D3D11_VPIV_DIMENSION_TEXTURE2D,
+ D3D11_VPOV_DIMENSION_TEXTURE2D, ID3D11Device, ID3D11DeviceContext, ID3D11Texture2D,
+ ID3D11VideoContext, ID3D11VideoContext1, ID3D11VideoDevice, ID3D11VideoProcessor,
+ ID3D11VideoProcessorEnumerator, ID3D11VideoProcessorEnumerator1, ID3D11VideoProcessorInputView,
+ ID3D11VideoProcessorOutputView,
+};
+use ::windows::Win32::Graphics::Dxgi::Common::{
+ DXGI_COLOR_SPACE_RGB_FULL_G22_NONE_P709, DXGI_COLOR_SPACE_YCBCR_FULL_G22_LEFT_P709,
+ DXGI_COLOR_SPACE_YCBCR_STUDIO_G22_LEFT_P709, DXGI_FORMAT, DXGI_FORMAT_AYUV, DXGI_FORMAT_NV12,
+ DXGI_FORMAT_P010, DXGI_FORMAT_R8G8B8A8_UNORM, DXGI_FORMAT_Y410, DXGI_RATIONAL,
+ DXGI_SAMPLE_DESC,
+};
+use ::windows::Win32::Graphics::Dxgi::IDXGIDevice;
+use ::windows::Win32::Media::MediaFoundation::{
+ IMFDXGIDeviceManager, MF_VERSION, MFCreateDXGIDeviceManager, MFSTARTUP_LITE, MFShutdown,
+ MFStartup,
+};
+use ::windows::Win32::System::Com::{
+ CO_MTA_USAGE_COOKIE, CoDecrementMTAUsage, CoIncrementMTAUsage,
+};
+use ::windows::core::{IUnknown, Interface};
+
+use crate::queue::BoundedQueue;
+use crate::{
+ ADAPTIVE_VIDEO_QUEUE_CAPACITY, BackendError, BackendEvent, EncodedVideoFrame, PushOutcome,
+ Subsystem, VideoChromaFormat, VideoFormat, VideoPixelFormat, WindowsDecoderMode,
+};
+
+use super::decoder::{DecodedVideoFrame, Decoder, DecoderDevice};
+
+const MAX_DECODE_STEPS_PER_RECORD: usize = ADAPTIVE_VIDEO_QUEUE_CAPACITY * 2;
+const MAX_FRAME_SLOTS: usize = 8;
+
+/// Borrowed Qt D3D11 objects used by the embedded frame producer.
+///
+/// Both pointers must identify the D3D11 device and its immediate context from
+/// Qt's adopted QRhi. The producer takes its own COM references and never
+/// assumes ownership of either incoming reference.
+#[derive(Debug, Clone, Copy)]
+pub struct AdoptedD3d11Context {
+ pub device: *mut c_void,
+ pub immediate_context: *mut c_void,
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct D3d11RecordedFrame {
+ pub texture: *mut c_void,
+ pub width: u32,
+ pub height: u32,
+ pub frame_slot: u32,
+ pub generation: u64,
+ pub presentation_time_ns: u64,
+}
+
+#[derive(Clone)]
+pub struct D3d11FrameSubmitter {
+ encoded: Arc>,
+ events: Arc>,
+}
+
+impl D3d11FrameSubmitter {
+ pub fn submit_video(&self, frame: EncodedVideoFrame) -> Result {
+ frame.validate()?;
+ let outcome = self
+ .encoded
+ .push_or_clear_on_overflow(frame)
+ .map_err(|_| BackendError::WorkerDisconnected)?;
+ if outcome == PushOutcome::DroppedOldest {
+ let _ = self
+ .events
+ .push(BackendEvent::QueueOverflow(Subsystem::VideoDecode));
+ let _ = self.events.push(BackendEvent::KeyFrameRequired);
+ }
+ Ok(outcome)
+ }
+}
+
+struct EmbeddedMediaRuntime {
+ mta_cookie: usize,
+ media_foundation_started: bool,
+}
+
+impl EmbeddedMediaRuntime {
+ fn initialize() -> Result {
+ unsafe {
+ let mta_cookie =
+ CoIncrementMTAUsage().map_err(|error| format!("CoIncrementMTAUsage: {error}"))?;
+ if let Err(error) = MFStartup(MF_VERSION, MFSTARTUP_LITE) {
+ let _ = CoDecrementMTAUsage(mta_cookie);
+ return Err(format!("MFStartup: {error}"));
+ }
+ Ok(Self {
+ mta_cookie: mta_cookie.0 as usize,
+ media_foundation_started: true,
+ })
+ }
+ }
+}
+
+impl Drop for EmbeddedMediaRuntime {
+ fn drop(&mut self) {
+ unsafe {
+ if self.media_foundation_started {
+ let _ = MFShutdown();
+ }
+ let _ = CoDecrementMTAUsage(CO_MTA_USAGE_COOKIE(self.mta_cookie as *mut c_void));
+ }
+ }
+}
+
+struct FrameSlot {
+ texture: ID3D11Texture2D,
+ output_view: ID3D11VideoProcessorOutputView,
+}
+
+struct ProcessorResources {
+ input_width: u32,
+ input_height: u32,
+ input_format: DXGI_FORMAT,
+ output_width: u32,
+ output_height: u32,
+ enumerator: ID3D11VideoProcessorEnumerator,
+ processor: ID3D11VideoProcessor,
+ input_views: HashMap<(usize, u32), ID3D11VideoProcessorInputView>,
+ slots: Vec,
+}
+
+struct AdoptedResources {
+ device: ID3D11Device,
+ video_device: ID3D11VideoDevice,
+ video_context: ID3D11VideoContext,
+ video_context_1: Option,
+ manager: IMFDXGIDeviceManager,
+ format: VideoFormat,
+ generation: u64,
+ processor: Option,
+}
+
+impl AdoptedResources {
+ unsafe fn new(adopted: AdoptedD3d11Context, format: VideoFormat) -> Result {
+ if adopted.device.is_null() || adopted.immediate_context.is_null() {
+ return Err("Qt supplied a null D3D11 device or immediate context".to_owned());
+ }
+ let device = unsafe { clone_interface::