feat(debug): live device screen mirror with remote control (protobufs #1054) - #6987
Draft
jamesarich wants to merge 13 commits into
Draft
feat(debug): live device screen mirror with remote control (protobufs #1054)#6987jamesarich wants to merge 13 commits into
jamesarich wants to merge 13 commits into
Conversation
Consumes the DisplayFrame PoC protobufs (2.99.0-screen-mirror-poc-SNAPSHOT from mavenLocal, useMavenLocal committed on this branch): DisplayMirrorManager reassembles FromRadio.display_frame chunks into MirrorFrame snapshots, AdminController gains setDisplayMirror + sendInputEvent, and the Debug panel gains a Mirror tab rendering the device's 1bpp framebuffer live with a D-pad driving the firmware InputBroker. PoC: tab strings deliberately unlocalized. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Cap the mirror canvas at 512.dp (4x for a 128px OLED) instead of filling the window width, fold the frame info into the toggle row, and pair Down/Back so the controls fit without scrolling on desktop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Manager: validate format (MONO_VLSB only) and geometry (width/height must describe exactly total_size bytes) so malformed device input can't reach aspectRatio and crash; unit tests for reassembly, drops, restarts. AdminController: setDisplayMirror/sendInputEvent become non-suspend immediate sends (teardown-safe, input never queues behind bulk admin) and gain requestDisplayFrame. Mirror tab: render each frame once into a 1:1 ImageBitmap scaled with FilterQuality.None (kills the fractional-scale seams), gate controls on connection state, reset the toggle on disconnect, stop the stream on tab dispose and ViewModel clear, add a one-shot Refresh, inject AdminController/ConnectionStateProvider instead of RadioController. Dispatch: display_frame test coverage incl. revoked-session. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Populates the new DisplayInfo during the handshake — dimensions from the live display device, MONO_VLSB frame format, PanelClass from the panel defines (EINK/HUB75/TFT/LCD/OLED), has_touch from HAS_TOUCHSCREEN — so clients can gate mirroring UI and adapt to the panel. Absent on display-less builds. Regenerates protos (DisplayInfo, DisplayFrame rect fields reserved for dirty-rect streaming). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace the flat button stack with the researched control surface: cross-shaped 5-way cluster (56dp circular keys, 8dp gaps, Material Symbols chevrons with contentDescriptions), directions auto-repeat on hold (500ms then 10Hz — slow enough for the radio to render), OK long-press sends SELECT_LONG, Back sits below per TV-remote convention. The mirror itself becomes an input surface: click to capture the keyboard (arrows / Enter / Esc, with a visible focus ring and hint) and swipe in a cardinal direction to navigate. Consumes the new DeviceMetadata.display: shows an e-ink hint steering toward one-shot Refresh. Adds keyboard_arrow_left/right icons (Material Symbols rounded, transposed from the existing up glyph). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
DisplayMirrorManager reassembles FromRadio.display_palette chunks (signature-keyed, defaults captured on the offset-0 chunk) into a MirrorPalette; frames carry palette_signature and the renderer resolves each pixel through the region table (last overlapping region wins, matching firmware precedence) with RGB565 conversion, falling back to monochrome when the referenced palette hasn't arrived. Renderer split into MirrorFrameRenderer.kt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Controls sit beside the mirror when the content width allows (>=760dp, BoxWithConstraints) and stack below it otherwise; tapping the mirror now toggles keyboard capture instead of only acquiring it, with the hint text naming the way out. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Circular D-pad ring replaces the cross cluster (wedge hit-testing with a radial dead band, hold-to-repeat, per-wedge semantics, haptics), keyboard capture becomes an explicit chip with tap-to-toggle, and taps on touch devices forward real panel coordinates (long-press = SELECT, matching the physical touch driver). Manager gains reset() (wired to ViewModel creation and disconnect), a palette region cap, and first-chunk geometry pinning; renderer uses bit-replicated RGB565 and run-length row drawing with the pixel resolver extracted pure and tested; KeyUp of captured keys is consumed; ViewModel moved to its own file; all strings localized; the stray .bak test file is gone. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
|
Important Draft PR not reviewedDraft PRs are not automatically reviewed by default.
To automatically review draft PRs, update your CodeRabbit configuration: reviews:
auto_review:
drafts: trueThanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Collaborator
Author
|
Cross-platform feature spec (client-agnostic wire contract, reassembly rules, input mapping, UX and device requirements): meshtastic/design#142 |
|
DisplayMirrorManager reassembles RGB565 rect chunks (sequence, geometry, and panel-bound validation) and composites them into a persistent little-endian RGB565 canvas emitted per completed rect; the renderer gains a true-color branch using the same run-length row drawing. Mono frames are untouched. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RGB565 frames identify LVGL/MUI devices, whose UIs read their own input drivers rather than the InputBroker remote events inject into — so the mirror hides the D-pad, keyboard, swipe, and touch affordances and says so, until an input-injection path exists. Utility helpers and the input codes move to DisplayMirrorUtils.kt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
device-ui now accepts injected input (virtual LVGL pointer + group-attached keypad bridged from the InputBroker), so the view-only treatment goes away: the D-pad navigates MUI widgets, taps forward panel coordinates, and the keyboard types. The mirror_view_only string stays for the future capability-flag path. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The mirror's keyboard capture only mapped navigation keys and silently dropped everything else, so text entry did nothing: printable characters never left the client even though the firmware already injects kb_char. Characters now go out as ANYKEY with the code point, Backspace goes as a character (so text fields delete rather than navigating back, which stays Esc's job), and the handling moves into a Modifier.remoteKeyInput extension. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Third adversarial review pass over the client side. Rect reassembly moves out of DisplayMirrorManagerImpl into MirrorRectCompositor: the manager was carrying two unrelated state machines and 14 methods. Every rect field is now validated before anything is allocated (total_size is device-controlled), geometry is latched from the chunk at offset 0 so a later chunk cannot transpose the blit, continuations must match the buffer that was actually allocated rather than their own claim, and bounds use the subtraction form because rect_x + rect_width overflows for hostile values. A rect that omits width or height covers the full panel in that axis. remoteKeyInput no longer swallows modified keys, so Cmd+Q, Ctrl+C and Alt+Tab stay with the desktop instead of being forwarded to the device. MirrorFrameRenderer tracks colorized runs as a raw Int with a sentinel rather than a boxed Color?, and MirrorFrame.hashCode includes the format so a mono and an RGB565 frame of the same size cannot collide. Drops mirror_view_only, orphaned since MUI gained remote input. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a live Mirror tab to the Debug panel: the connected device's screen rendered in real time — in its actual theme colors on color-TFT devices — with full remote control. Verified on hardware against RAK4631 (mono OLED) and LILYGO T-Deck (color, touch).
🌟 New Features
FromRadio.display_framechunks reassembled into frames and rendered once per frame into a 1:1ImageBitmap, upscaled with nearest-neighbor filtering (crisp device pixels). Color devices stream a signature-keyedDisplayPalette; the renderer applies per-region RGB565 colors with the firmware's last-region-wins precedence, and a frame arriving before its palette self-heals to color when the palette lands.SELECT_LONG, haptics, per-wedge semantics for screen readers); desktop keyboard capture on the mirror (chip- or click-toggled, visible focus ring; arrows/Enter/Esc,KeyUpconsumed so keys never leak to the scroll container); swipe-to-navigate on the image, and onhas_touchdevices tap/long-press forwards real panel coordinates viasend_input_event.DeviceMetadata.displaygates behavior — e-ink devices get a hint steering toward one-shot Refresh; touch capability enables tap-to-touch.Changes
core/repository:DisplayMirrorManager(+MirrorFrame/MirrorPalette),AdminController.setDisplayMirror/requestDisplayFrame/sendInputEvent— non-suspend immediate sends so input never queues behind bulk admin traffic and lifecycle teardown can always stop the stream.core/data: reassembly state machine — contiguous-offset chunks only, geometry pinned at the first chunk, format/size/region-cap validation before any allocation,reset()so a previous device's last screen is never shown as live. Unit-tested (frames, palettes, drops, restarts, caps).core/service: admin senders viasendAdminImmediate.feature/settings: the Mirror tab (DisplayMirror.kt,DisplayMirrorViewModel.kt,DpadRing.kt,MirrorFrameRenderer.kt) — stream stopped on tab dispose, ViewModel clear, and disconnect; toggle tracks connection state. Renderer resolver extracted pure and unit-tested (region precedence, RGB565 bit-replication).core/resources), two Material Symbols chevrons added.🤖 Generated with Claude Code