Device screen mirroring & remote control — cross-platform feature spec

[jamesarich]

Device Screen Mirroring & Remote Control — Feature Specification

Status: draft · protocol shipped in meshtastic/protobufs#1054 · reference device implementation meshtastic/firmware#11681 · reference client meshtastic/Meshtastic-Android#6987 · prior art meshtastic/web#224

The key words MUST, MUST NOT, SHOULD, SHOULD NOT, and MAY are to be interpreted as in RFC 2119. "Device" means the Meshtastic node firmware; "client" means any locally connected app (Android, desktop, iOS, web, python) speaking the phone API over BLE, USB serial, or TCP.

1. Overview

A client can view the device's screen live and drive its UI remotely. The device streams snapshots of its framebuffer to the local client; the client renders them and sends input events back. On devices that paint a monochrome UI onto a color panel, a small palette sideband lets the client reproduce the panel's true colors at monochrome bandwidth.

Goals

• Remote support: see what the user's screen says without being there.
• Headless and pocketed nodes: operate a device whose screen/buttons are inaccessible.
• Deterministic UI testing: pixel-exact screen assertions without cameras.

Non-goals (this revision)

• Mirroring LVGL/MUI color UIs (RGB565). The wire format reserves fields for it (§4.3 rects); no device implements it yet.
• Mirroring a remote node's screen over the mesh. Frames ride FromRadio, which never crosses the mesh (§8).
• Audio, video rates, or lossless guarantees. This is a best-effort snapshot stream at UI cadence (~1 fps idle).

2. Capability discovery — DisplayInfo

DeviceMetadata.display (field 15), sent during the connection handshake:

Field	Meaning
width, height	Panel dimensions in pixels
format	The DisplayFrame.Format frames from this device will use (currently always MONO_VLSB)
panel_class	OLED, LCD, TFT, EINK, HUB75, or PANEL_CLASS_UNSPECIFIED (runtime-selected panels, e.g. Linux-native)
has_touch	Panel accepts touch input

• Absence of display means the device has no screen or runs pre-feature firmware. Clients SHOULD show mirroring UI opportunistically in that case, but MUST NOT rely on its presence.
• Clients SHOULD adapt to panel_class == EINK: prefer one-shot frame requests over continuous mirroring, and expect multi-second refresh latency (§6.5).
• Clients SHOULD only offer tap-to-touch when has_touch is true (§5.2).

3. Session model

3.1 Arming

Two AdminMessage verbs control the stream. Both are local-connection-only: a device receiving them over the mesh, or a build without a display, ignores them.

• get_display_frame_request = 50 (bool): one-shot. Forces one frame on the next redraw, even if the screen is unchanged. Classified as an admin request (no session passkey needed). There is no AdminMessage response; the frame arrives out-of-band as FromRadio.display_frame.
• set_display_mirror = 51 (bool): continuous mirroring on/off. false is meaningful — unlike most bool admin verbs this is not a trigger. The first frame arrives immediately on enable and acts as the acknowledgement. Not persisted across reboot; clients MUST re-arm after reconnect.

3.2 Lifecycle

• The device disarms and frees all mirror state when a client connection closes. Current firmware keeps a single global arm flag: any local client's disconnect disarms every client's stream (known limitation; per-client refcounted arming is the planned follow-up).
• Clients MUST disable mirroring when their mirror UI is dismissed and SHOULD treat a disconnect as an implicit disarm (reflect it in UI state rather than assuming the stream survives).
• Clients MUST NOT present a previously received frame as live after a disconnect or device change; clear held frames when a session ends.

4. Wire protocol

All messages ride the existing phone API (FromRadio, ≤512 bytes encoded per message). Both streams use the same completion rule: offset + payload length == total completes the unit.

4.1 FromRadio.display_frame — DisplayFrame (tag 20)

One chunk of a framebuffer snapshot:

Field	Semantics
width, height	Full display dimensions (constant across a device's frames)
format	Pixel encoding of data. MONO_VLSB = 1: 1 bpp, vertical LSB-first pages — byte index x + (y / 8) * width, bit index y % 8. FORMAT_UNSPECIFIED = 0 MUST NOT be sent
frame_id	Constant across one frame's chunks; increments per captured frame; wraps at uint32; restarts from 1 on reboot. Treat any change as "a new frame", never as ordering
offset, total_size	Byte position of this chunk in the frame buffer, and the full buffer size
data	≤384 bytes of framebuffer
rect_x/y/width/height	Reserved for future partial-rect updates (unset today; see §9)
palette_signature	The DisplayPalette that colorizes this frame; 0 = render monochrome

Guarantees a client MAY rely on:

• Chunks of one frame arrive contiguously (no other display_frame interleaves; display_palette messages may) and in offset order — FromRadio is a reliable ordered stream. Reassembly needs no reordering buffer.
• A frame whose streaming has begun is always drained to completion, even if mirroring is disabled mid-frame.
• Only changed framebuffers are streamed (the device diffs snapshots), plus one forced frame per one-shot request or enable. A palette change with unchanged pixels also produces a new frame (§4.2).

Client reassembly requirements:

• offset == 0 MUST start a new frame (this is also the reboot/torn-capture recovery path). Geometry (width/height) MUST be taken from the first chunk; a later chunk disagreeing MUST drop the partial frame.
• A chunk that is not the exact contiguous continuation (frame_id, offset, total_size all matching expectations) MUST drop the partial frame and await the next offset == 0.
• Clients MUST validate before allocating: format supported, width > 0, height > 0, width * ceil(height / 8) == total_size, and a sanity cap on total_size (reference: 16 KiB — a 320×240 1 bpp frame is 9600 bytes).

4.2 FromRadio.display_palette — DisplayPalette (tag 21)

Color devices paint the 1 bpp UI through a table of colored regions. Streaming that table lets a client render a color-accurate mirror at monochrome bandwidth.

Field	Semantics
signature	Identity of this palette; changes whenever the region table or theme changes. DisplayFrame.palette_signature references it
default_on_color, default_off_color	RGB565 for set/clear pixels outside all regions. Authoritative on the region_offset == 0 chunk
region_offset, region_total	Chunking by region index; offset + regions length == total completes the palette
regions[≤16]	ColorRegion { x, y, width, height, on_color, off_color }, in table order

• All colors are RGB565 in logical bit layout (RRRRRGGGGGGBBBBB) — the device byte-swaps from panel order before sending.
• Precedence: a region with a higher table index overrides lower-indexed ones where they overlap, regardless of chunk boundaries.
• Palettes are sent only when the signature changes (per client), before the frames that reference them. Monochrome devices never send one.
• A client holding partial chunks of a signature that no longer matches incoming chunks MUST discard them. Clients SHOULD cap accepted region_total (reference: 512; firmware's own table caps at 48).
• Rendering rule per pixel: find the highest-indexed region containing (x, y); use its on_color/off_color by the pixel's bit; otherwise the defaults. A frame referencing a palette the client doesn't hold yet SHOULD render monochrome and SHOULD be re-rendered when the palette arrives (frames self-heal to color).
• RGB565 → 8-bit expansion SHOULD use bit replication (r8 = (r5 << 3) | (r5 >> 2)), not plain scaling.

4.3 Size budget

DisplayFrame encodes ≤447 bytes and DisplayPalette ≤440, both under the 512-byte MAX_TO_FROM_RADIO_SIZE cap. A 128×64 frame is 3 chunks (~1 KB); a 320×240 1 bpp frame is 25 chunks (~10 KB). At the BaseUI idle cadence of 1 fps this is well within BLE throughput; clients need no flow control beyond normal FromRadio reads.

5. Remote input

Input reuses the pre-existing AdminMessage.send_input_event = 27 (InputEvent { event_code, kb_char, touch_x, touch_y }), which injects into the firmware's InputBroker — the same path physical buttons use, so it works on every device UI unmodified.

5.1 Event codes (firmware input_broker_event)

Code	Event	Client mapping guidance
10	SELECT	OK button / Enter / Space / touch long-press
11	SELECT_LONG	OK long-press
17–20	UP, DOWN, LEFT, RIGHT	D-pad / arrow keys / swipe on the mirror
24	CANCEL	—
27	BACK	Back button / Esc / Backspace
28	USER_PRESS	Touch tap, with coordinates

5.2 Touch forwarding

On has_touch devices, clients SHOULD map pointer input on the mirrored image to panel coordinates (floor(px / rendered_size * panel_size), clamped) and send:

• tap → USER_PRESS (28) with touch_x/touch_y — matching what physical touch drivers emit;
• long-press → SELECT (10) with coordinates — likewise matching the physical driver.

5.3 Input transport & pacing

• Input SHOULD be sent on an immediate/priority path, never queued behind bulk admin traffic.
• Hold-to-repeat SHOULD use ~500 ms initial delay, then ~10 Hz — deliberately slower than OS typematic so events cannot outrun the radio's UI rendering. Repeat directions only; never auto-repeat SELECT/BACK.

6. Client UX requirements & guidance

Derived from TV-remote / emulator-overlay / accessibility research; the Android client is the reference.

1. Directional cluster: a circular 5-way ring (four 90° wedges around a center OK disc, hit-test by angle, dead band between disc and ring) is the preferred layout; a well-spaced cross of ≥48 dp targets is an acceptable minimum. BACK sits below the cluster. Every control MUST be reachable by assistive tech as a discrete button (per-wedge semantics; gesture surfaces are duplicates, never the only route).
2. Keyboard capture (pointer-based platforms): capturing focus on the mirror maps arrows/Enter/Esc per §5.1. Capture MUST be visibly indicated and MUST have an obvious exit (toggle control and/or click-again). Key-up events of captured keys MUST be consumed so they don't leak into the surrounding UI.
3. Swipe-to-navigate: a cardinal swipe on the mirror sends one direction event; sub-threshold movements (reference: 48 dp) are ignored.
4. Rendering: render each frame once into a native bitmap at 1:1 and upscale with nearest-neighbor filtering (crisp pixels, no interpolation seams). Never redraw per-pixel per UI frame.
5. E-ink: steer users toward one-shot Refresh; don't imply live cadence.
6. State honesty: the mirror toggle reflects connection state; disconnect resets it (§3.2); a stale frame is never displayed as live.

7. Device implementation requirements

For a firmware (or simulator) implementing the device side:

• Capture point: snapshot the framebuffer after each committed UI frame, from the thread that draws it. Diff against the last sent snapshot; stream only changes. On palette-driven devices, capture the region table at paint time — before any per-frame clearing — and stamp frames with the signature captured with that snapshot, not the live one.
• Multi-client: each client connection keeps its own drain cursors (frame (id, offset), palette (signature, region_offset)) against the single shared snapshot, so coexisting clients receive complete units independently. A frame captured while a client is mid-drain restarts that client at offset 0 of the new frame.
• Priority: mirror data drains at the lowest phone-API priority — mesh packets, notifications, and sync replay always outrank pixels. Palette chunks precede frame chunks so the first rendered frame can be colored.
• Redaction: builds with display redaction (lockdown) MUST mirror the redacted screen, never the underlying content. Under client access control, frames and palettes MUST only be delivered to authorized clients (same bar as mesh packets).
• Memory: allocate nothing until armed; free on disarm/disconnect; account allocations in the platform's memory audit. Refuse panels whose 1 bpp size exceeds internal cursor width rather than truncating.

8. Security & privacy

• Screen contents are operator content (messages, positions, node names). They are confined to the local link by construction: frames ride FromRadio, which has no mesh path.
• The admin verbs act only from the local connection (from == 0); remote nodes cannot arm a mirror, and a passkey-authorized remote admin still cannot receive frames.
• Known gap (owned in firmware#11681): under client access control, an unauthorized local client can arm/disarm (control-plane DoS) though never read frames. Fixed by per-client arming.

9. Future work

• Color/MUI streaming: Format.RGB565 + the reserved rect_* fields turn the same message into a dirty-rect stream fed by LVGL's flush callback — additive, no wire break. Viable over USB/TCP; needs throttling over BLE.
• Per-client refcounted arming (replaces the global flag). The per-client cursors it needs already exist.
• Palette cache: clients MAY keep the last few palettes keyed by signature so a one-shot frame taken across a theme change still colorizes.
• Device bezels: client-side cosmetic frames from the design repo's device art, keyed by hw_model, with a canonical screen-viewport rect per asset.
• Firmware native test suite (test_screen_mirror): chunk reassembly, mid-drain restart, palette signature changes incl. theme-only, memaudit balance.

10. Verification status

End-to-end verified on real hardware against the reference client (Android/desktop, shared KMP code):

• RAK4631 (WisMesh Pocket) — 128×64 OLED, monochrome mirror, D-pad/keyboard remote control.
• LILYGO T-Deck (BaseUI build) — 320×240 ST7789, color-accurate mirror via palette streaming, tap-to-touch, theme-recolor propagation.
• Build-verified additionally on Heltec V3, Heltec Mesh Node T114 (ST7789/nRF52), and the Linux-native target (coloring-disabled TFT combination).

[jamesarich]

MUI (device-ui) mirroring — phase-0 spike findings (2026-08-31)

Tested on a LILYGO T-Deck running the t-deck-tft build with display.displaymode = COLOR (MUI/LVGL live):

• Serial coexistence works. MUI is itself an in-process phone-API client (PacketAPI : PhoneAPI), and a USB-serial client completed a full config handshake and operated normally while MUI stayed live and interactive on the glass. Embedded-MUI mirroring over serial is therefore in scope.
• BLE is structurally excluded on embedded MUI: PacketAPI::runOnce starves MUI into programmingMode whenever config.bluetooth.enabled is set (the config flag, not an active connection), so there is never a live MUI to mirror while a BLE client could watch.
• Portduino MUI has no gate at all (#ifndef ARCH_PORTDUINO): meshtasticd + native-tft serves TCP clients while MUI runs — remote HMI for Linux nodes and deterministic MUI screen assertions in CI are unconditionally viable.
• As specified (§9), MUI builds stream no frames today (no BaseUI framebuffer); the same t-deck-tft build under displaymode: DEFAULT renders BaseUI and mirrors in color correctly.

Phase-1 sketch: tap LVGL's flush_cb in device-ui → stream dirty rects as DisplayFrame with Format.RGB565 and the reserved rect_* fields (no palette needed); client composites rects into its persistent bitmap. Initial full-frame sync ≈150 KB (~2 s serial, instant TCP), dirty rects tiny thereafter. Notes: requires an upstream device-ui change (zip-pinned lib), and USE_PACKET_API is build-incompatible with MESHTASTIC_PHONEAPI_ACCESS_CONTROL, so MUI mirroring stays out of scope for hardened builds.

[jamesarich]

MUI mirroring — phase 1 shipped (2026-08-31)

Live full-color MUI mirroring works end-to-end on a T-Deck over USB serial: device-ui's LVGL flush callback now feeds a flush observer (native little-endian RGB565 dirty rects, pre-panel-swap), the firmware queues rects in a PSRAM pool sized for a full repaint (192 KB — undersizing this was the one field bug: a partial pool can never complete the arming sync) and streams them as DisplayFrame Format.RGB565 chunks on the reserved rect fields; the client composites into a persistent canvas. Arming triggers a full LVGL repaint (active screen plus top/system overlay layers) so a new client synchronizes completely, then updates flow as tiny dirty rects.

Branches: device-ui jamesarich/device-ui@screen-mirror-poc (flush observer + requestFullRefresh), firmware/android on the existing screen-mirror-poc branches (the firmware registration is gated on MESHTASTIC_MUI_MIRROR until the device-ui change merges and the vendored pin moves).

Confirmed gap → next spike: remote input does not reach MUI — send_input_event injects into the firmware InputBroker, but device-ui reads its own input drivers. The client now renders MUI mirrors view-only with a note. The input seam under design: an injection API in device-ui (virtual LVGL pointer indev + key dispatch, queued off-thread and drained on the LVGL thread), bridged from an InputBroker observer in firmware — the exact sibling of the flush observer.

[jamesarich]

MUI remote control — phase 2 validated (2026-08-31)

Remote control now works on MUI, verified objectively without a camera: injecting AdminMessage.send_input_event USER_PRESS at (35,105) navigates the T-Deck's LVGL UI to Group Channels, captured before/after through a headless mirror client.

Root cause of the initial failure (worth knowing for any client work): MUI builds never construct an InputBroker — Modules.cpp skips it, along with SystemCommands and buzzer feedback, whenever display.displaymode == COLOR. So send_input_event hit AdminModule's null-broker guard and vanished; remote control was silently inert on every MUI device, and an observer-on-InputBroker bridge (my first design) could never attach. AdminModule now offers the event to a MUI injector first and falls through to the broker on BaseUI builds.

device-ui seam (jamesarich/device-ui@screen-mirror-poc): InputDriver::injectTouch/injectKey statics, callable from any thread, feeding two always-present virtual LVGL devices created in the previously-empty base init() — a pointer whose read callback holds PRESSED for a requested duration (taps, and synthesized long presses) and a keypad bound to the default group, which init now guarantees exists so injected keys can navigate widgets even on boards with no physical input.

Note for spec §5: firmware's input_broker_event codes and LVGL's LV_KEY_* disagree — LVGL has RIGHT=19/LEFT=20, the broker has LEFT=19/RIGHT=20. The mapping is by name in the injector; anyone bridging these elsewhere must cross-map.

Tooling: the validation used a small headless mirror client (arms mirroring over the phone API, reassembles MONO_VLSB frames and RGB565 dirty rects, writes PNGs, injects input) — the camera-free UI-assertion path this feature was partly motivated by. Parked at notes/mirror-client.py in the workspace.

Known rough edge: arming immediately after boot can miss the full-screen sync (the repaint request lands before MUI finishes initialising), leaving the client compositing incremental rects over black until enough of the screen redraws. Arming after the UI settles delivers a complete frame.

[jamesarich]

Hardening pass — protocol rules clients must implement (2026-08-31)

A third adversarial review over all three branches turned up several behaviours that are wire-visible, so they belong in the spec rather than in any one client. Everything below is now implemented on screen-mirror-poc (protobufs#1054, firmware#11681, android#6987).

Validating the RGB565 rect stream

total_size and the rect geometry are device-controlled, so a client must validate before it allocates:

• Latch the geometry from the chunk at offset == 0. A continuation chunk that carries different rect_x/y/width/height must not be allowed to transpose the blit of a rect already in flight.
• A continuation must match the buffer actually allocated, not merely its own total_size claim — check it against the allocation, the latched frame_id, and the running offset.
• Bounds-check by subtraction (rect_x <= width - rect_width), never rect_x + rect_width <= width: the addition overflows for hostile values.
• A rect that omits rect_width or rect_height covers the full panel in that axis. Zero means "all", not "empty".
• rect_width * rect_height * 2 must equal total_size exactly, and the whole rect must fit a sane panel bound.

One rect consumer at a time

The firmware serves MUI rects to a single owner — the first client to drain the queue claims it, and the claim is released on disarm. A second concurrent client will not receive rects. This is deliberate: dirty rects are stateful (they only make sense composited onto a canvas that saw every earlier rect), so fanning the same queue to two clients would give both a corrupt canvas. Mono BaseUI frames are self-contained and have no such restriction.

Backpressure is a full resync, not a dropped rect

When the device's rect pool overflows, it discards the entire backlog and queues a full refresh rather than dropping the newest rect. Dropping any single rect would leave the client's canvas permanently wrong with no way to detect it; a full repaint is self-healing. Clients must therefore tolerate a sudden full-panel rect burst at any point in the stream, not just at arming.

Panel dimensions come from LVGL

The MUI panel size is read from lv_display_get_horizontal_resolution / ..._vertical_resolution at registration, not from the variant header — a variant's nominal panel size and the resolution LVGL is actually driving can differ (rotation, an offset window). DeviceMetadata.display reports what LVGL reports.

Input handling

• Injection is opt-in in device-ui. The virtual LVGL indevs are only created when the firmware explicitly enables injection, so a device-ui build with no mirroring support pays nothing and behaves exactly as before.
• Clients should not forward modified keys. Ctrl/Cmd/Alt chords belong to the host — a mirror that swallows Cmd+Q or Ctrl+C to send it to the radio is a trap. Forward plain keys and the navigation set only.