feat(screen): stream the display framebuffer to local clients (DisplayFrame mirroring) - #11681
Draft
jamesarich wants to merge 15 commits into
Draft
feat(screen): stream the display framebuffer to local clients (DisplayFrame mirroring)#11681jamesarich wants to merge 15 commits into
jamesarich wants to merge 15 commits into
Conversation
…rame chunks ScreenMirror snapshots the 1bpp OLEDDisplay buffer after each committed frame (updateUiFrame), diffing against the last sent snapshot so only changed frames stream. Armed via the new AdminMessage set_display_mirror (continuous) / get_display_frame_request (one-shot); chunks drain through PhoneAPI's STATE_SEND_PACKETS chain like XModem, so BLE, serial and TCP clients all receive them. The lockdown path mirrors the LOCKED frame, preserving display redaction. Excludable with MESHTASTIC_EXCLUDE_SCREEN_MIRROR. Bumps the protobufs submodule to the DisplayFrame PoC commit and regenerates nanopb sources (FromRadio_size stays 510 <= 512). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Gate frame delivery on getAdminAuthorized under access control (screen pixels are operator content, same rule as mesh packets); move the drain to lowest getFromRadio/available priority behind the replay drain; give each PhoneAPI its own (frameId, offset) cursor via ScreenMirror::copyChunk so coexisting BLE/serial/TCP clients each receive complete frames; disarm and free the snapshot on client close and on setMirror(false), with memaudit accounting and a geometry-change realloc guard; honor the local-connection-only contract for both admin verbs (mp.from == 0); demote the per-frame onNotify log to TRACE; hoist the guard into a derived HAS_SCREEN_MIRROR in configuration.h. Regenerates protos for the reviewed contract (FORMAT_UNSPECIFIED = 0, MONO_VLSB = 1, width/height int_size:16). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mirroring ScreenMirror captures the color-region table with each snapshot when the frame signature changes (regions byte-swapped from panel order to logical RGB565, defaults = White / theme body background) and streams it as FromRadio.display_palette chunks ahead of frames, with per-client (signature, region) cursors like the frame drain. Frames reference the palette via palette_signature; monochrome-only builds send 0 and no palettes. Also classify USE_TFTDISPLAY/HAS_SPI_TFT panels (T-Deck) as TFT in DeviceMetadata.display. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…frame TFTDisplay/HUB75 consume and then clear the per-frame color-region table inside display(), so the post-update capture always photographed an empty table and mirrors rendered monochrome. The drivers now hand ScreenMirror the palette (signature, defaults, regions — all panel-byte-order, swapped to logical RGB565 in the store) right before clearing, and the mirror stamps frames with the signature of the palette they were painted with. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Guard the diff-path palette capture with GRAPHICS_TFT_COLORING_ENABLED too (env:native defines USE_TFTDISPLAY without coloring, so the site referenced an undeclared identifier and broke the CI/test env); test HAS_TOUCHSCREEN by value, not definedness — it defaults to 0, so every screened device reported touch; stamp frames with the signature captured WITH the snapshot and treat a palette-only change as frame-worthy, so theme recolors reach the client without pixel churn and mid-drain captures stay coherent; compute the frame size in 32 bits and refuse panels past the uint16 chunk cursors; make setMirror idempotent (no disconnect log spam); report PanelClass UNSPECIFIED on portduino where the panel is runtime-selected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This was referenced Aug 31, 2026
Contributor
Author
|
Cross-platform feature spec (client-agnostic wire contract, reassembly rules, input mapping, UX and device requirements): meshtastic/design#142 |
Contributor
⚡ Try this PR in the Web FlasherNote Building this pull request… the flash button, badges and supported-board |
On HAS_TFT builds, tftSetup registers device-ui's new flush observer: every LVGL dirty rect arrives pre-byte-swap (native RGB565) and queues into ScreenMirror's bounded rect pool (96 KB, 32 rects), draining through the existing display_frame chunker with the rect fields and per-rect frame_ids. Arming or a one-shot triggers a full repaint through device-ui so a new client synchronizes the whole screen; queue overflow drops rects and requests one resync repaint when drained. Spike scope: the rect queue is single-consumer (per-client cursors cover mono frames only). Requires the device-ui flush-observer branch; the vendored pin is overridden locally and intentionally left uncommitted until device-ui merges. Regenerates protos for Format.RGB565. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A 320x240 RGB565 sync is ~150 KB; the 96 KB pool overflowed mid-repaint and the resync loop could never deliver a complete first frame — clients composited incremental rects over black. 192 KB (PSRAM-first on ESP32) holds a full repaint plus concurrent updates, and 64 rect headers cover icon-sized bursts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…UI_MIRROR The vendored device-ui pin predates the flush observer; without the gate this branch cannot build t-deck-tft in CI. The define is set locally alongside the device-ui override until the device-ui change merges and the pin moves. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A lazily attached observer (inputBroker is created after tftSetup) maps remote send_input_event traffic onto device-ui's injection seam: broker directions become LV_KEY_* (note the deliberate LEFT/RIGHT cross-map — the broker codes were modeled on LVGL keys but those two are swapped), SELECT with coordinates synthesizes a long press, USER_PRESS taps, BACK/CANCEL map to ESC, and kb_char passes through. Gated with the MUI mirror define until the device-ui side merges. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tBroker MUI builds never construct an InputBroker — Modules.cpp skips it (along with SystemCommands and buzzer feedback) whenever displaymode is COLOR — so AdminMessage.send_input_event hit AdminModule's null-broker guard and died: remote control was silently inert on every MUI device. AdminModule now offers the event to graphics::muiInjectInputEvent first, which maps it onto device-ui's virtual LVGL devices and reports whether it consumed it, falling through to the broker on BaseUI builds. Verified on a T-Deck with a headless mirror client: injecting USER_PRESS at (35,105) navigates MUI to Group Channels. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
getDeviceMetadata derived display geometry from the BaseUI screen object, which MUI builds never create — so color-UI devices advertised no display at all, and clients could not tell they were touch-capable. Ask LVGL for the logical resolution instead (already rotated) and report RGB565 / TFT / has_touch. Verified on a T-Deck: 320x240, RGB565, TFT, touch true. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Vertical becomes encoder rotation (what actually moves focus in an LVGL group) and horizontal becomes the slider keys, deliberately inverted, so remote control matches what the device's own trackball driver emits. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Four defects from review. The idempotence guard added with the wake work skipped cleanup when a one-shot request disarmed, stranding the 192 KB rect pool and leaving the mirror armed for a departed client — cleanup is now unconditional and only the log is guarded. The rect FIFO has a single shared cursor, so it now records the connection that claims it rather than splitting each frame between coexisting BLE and serial clients. Panel dimensions come from LVGL at registration instead of growing as a running max of rect extents, which contradicted both the wire contract and DeviceMetadata. And on overflow the backlog is dropped and a repaint requested immediately, because dropping only the newest rect wedged the pool until the queue drained and thrashed the resync — the reason a fresh arm often never delivered a full frame. Also: notify only on the empty-to-non-empty transition rather than per LVGL flush, log a failed pool allocation, keep PSRAM out of the internal memaudit budget, and enable device-ui injection explicitly at setup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The previous commit swept in two working-tree files that exist only to build the MUI mirror against a local device-ui checkout: a symlink:// lib_deps pointing at a path on one machine, and the MESHTASTIC_MUI_MIRROR define on t-deck-tft. Neither resolves anywhere else, so t-deck-tft could not build. Restores the pinned device-ui archive and the stock variant flags. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
setMirror cleared muiOwner outside any guard, so every board without a TFT failed to compile — the member only exists on the MUI path. The assignment was redundant anyway: freeSnapshotLocked() releases the pool and the ownership claim together. The guards were also drawn on HAS_TFT while the code they protect needs a device-ui carrying the flush observer, so a color build that never opts in still carried the rect queue in .bss. Replaces the ad-hoc HAS_SCREEN_MIRROR && defined(MESHTASTIC_MUI_MIRROR) pairs with a single derived HAS_MUI_MIRROR next to HAS_SCREEN_MIRROR, and gates the queue, the pool, the input seam and their declarations on it. Verified on all three build shapes: t-deck-tft with the opt-in, picomputer-s3 (TFT, opted out) and heltec-v3 (no TFT). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Streams the device's screen to its locally connected client and completes the remote-control story started by
send_input_event(meshtastic/protobufs#702). Prior art: meshtastic/web#224. Draft until meshtastic/protobufs#1054 merges; the submodule pointer will move to protobufs master after. Android client: meshtastic/Meshtastic-Android#6987.How it works
graphics::ScreenMirrorsnapshots the 1bppOLEDDisplayframebuffer after each committed frame (single hook atupdateUiFrame's exits, so the lockdown build mirrors the redacted LOCKED frame, never operator content). Amemcmpdiff gates capture: a static screen streams nothing.PhoneAPI's existinggetFromRadiochain at lowest priority — mesh packets, notifications and the satellite replay always outrank pixels — with a per-client cursor (frameId,offset), so coexisting BLE/serial/TCP clients each receive complete frames.TFTDisplay/HUB75Displayhand the mirror their color-region table at paint time (immediately beforeclearTFTColorRegions()), byte-swapped to logical RGB565. Palettes stream asdisplay_palettechunks keyed by the existing region-table signature and re-send only when layout/theme changes; frames carry the signature of the palette they were painted with (coherent even across a mid-drain theme change, which also counts as frame-worthy so recolors propagate without pixel churn).DeviceMetadata.display(DisplayInfo) reports geometry,PanelClassand touch during the handshake.mp.from != 0is dropped): frames rideFromRadioand can never cross the mesh.MESHTASTIC_PHONEAPI_ACCESS_CONTROLgetAdminAuthorizedbar as mesh packets)MESHTASTIC_EXCLUDE_SCREEN_MIRRORbuildHAS_SCREEN_MIRROR), verbs ignoredCost
Zero allocation until armed: snapshot (1 KB on 128×64, 9.6 KB on 320×240 1bpp) + 576 B palette table on arm, freed on disarm/disconnect,
memaudit-tracked. ~2 KB flash when enabled; excluded builds pay only the nanopbFromRadiounion growth from the new oneof members (encodedFromRadio_sizestays 510 ≤ 512).Known limitations (deliberate for this round)
DisplayFrame's reserved rect fields are the path.test_screen_mirrornative suite (chunk reassembly, mid-drain restart, palette signature changes, memaudit balance) is sketched and queued as a follow-up commit.🤝 Attestations
t-deck(BaseUI) envnative-macos,heltec-mesh-node-t114(ST7789),heltec-v3— build verification incl. the coloring-disabled TFT combination🤖 Generated with Claude Code