-
-
Notifications
You must be signed in to change notification settings - Fork 737
feat(macos-control): add macOS AX control MCP tool #357
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
almirchowdhury
wants to merge
1
commit into
wonderwhy-er:main
from
almirchowdhury:pr/macos-ax-control-clean
Closed
Changes from all commits
Commits
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,49 @@ | ||
| #!/usr/bin/env bash | ||
| set -euo pipefail | ||
|
|
||
| ROOT_DIR="$(cd "$(dirname "$0")" && pwd)" | ||
| HELPER_DIR="$ROOT_DIR/native/macos-ax-helper" | ||
| OUT_DIR="$ROOT_DIR/bin/macos" | ||
|
|
||
| if [[ "$(uname -s)" != "Darwin" ]]; then | ||
| echo "build-macos-helper.sh is macOS-only" | ||
| exit 1 | ||
| fi | ||
|
|
||
| mkdir -p "$OUT_DIR" | ||
|
|
||
| build_arch() { | ||
| local arch="$1" | ||
| local out_name="$2" | ||
|
|
||
| echo "Building macos-ax-helper for $arch..." | ||
| if swift build --package-path "$HELPER_DIR" -c release --arch "$arch"; then | ||
| local built="$HELPER_DIR/.build/apple/Products/Release/macos-ax-helper" | ||
|
|
||
| if [[ ! -f "$built" ]]; then | ||
| built="$HELPER_DIR/.build/release/macos-ax-helper" | ||
| fi | ||
|
|
||
| if [[ ! -f "$built" ]]; then | ||
| echo "Could not find built helper for $arch" | ||
| return 1 | ||
| fi | ||
|
|
||
| cp "$built" "$OUT_DIR/$out_name" | ||
| chmod +x "$OUT_DIR/$out_name" | ||
| echo "Wrote $OUT_DIR/$out_name" | ||
| else | ||
| echo "Warning: build failed for $arch" | ||
| return 1 | ||
| fi | ||
| } | ||
|
|
||
| status=0 | ||
| build_arch arm64 macos-ax-helper-darwin-arm64 || status=1 | ||
| build_arch x86_64 macos-ax-helper-darwin-x64 || status=1 | ||
|
|
||
| if [[ $status -ne 0 ]]; then | ||
| echo "One or more architectures failed. If running on Apple Silicon, x64 build may require extra toolchains." | ||
| fi | ||
|
|
||
| exit $status |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,20 @@ | ||
| # Known Limitations | ||
|
|
||
| ## Accessibility path | ||
|
|
||
| - macOS-only: AX tools return unsupported platform on non-macOS hosts. | ||
| - Requires explicit Accessibility permission for the MCP host process. | ||
| - AX trees differ by app and can be incomplete or unstable in custom UI frameworks. | ||
| - Stable IDs are best-effort hashes of visible properties; some dynamic UIs can invalidate them quickly. | ||
| - Stale-ID fallback depends on prior element signatures and may fail in heavily changing layouts. | ||
|
|
||
| ## Electron/CDP path | ||
|
|
||
| - Requires app/browser launched with remote debugging endpoint enabled. | ||
| - Current CDP tool set is intentionally minimal: attach/eval/disconnect. | ||
| - DOM interaction helpers are not yet first-class MCP tools (can be done via JavaScript in `electron_debug_eval`). | ||
|
|
||
| ## Packaging/build | ||
|
|
||
| - Native helper binaries are architecture-specific (`darwin-arm64`, `darwin-x64`). | ||
| - Building both architectures from one machine may require additional toolchain support. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,141 @@ | ||
| # macOS Control in Desktop Commander MCP | ||
|
|
||
| This guide covers the macOS Accessibility (AX) and Electron debugger control features added to Desktop Commander MCP. | ||
|
|
||
| ## Overview | ||
|
|
||
| The implementation has two control paths: | ||
|
|
||
| 1. Accessibility path (native apps and Electron UI chrome) | ||
| - Uses a native Swift helper (`macos-ax-helper`) to query and act on AX elements. | ||
| - Supports finding/clicking elements, typing, keypress, waiting for UI state, and batching actions. | ||
|
|
||
| 2. Debugger path (Electron/Chromium web content) | ||
| - Uses Chrome DevTools Protocol (CDP) over WebSocket. | ||
| - Supports attach, JavaScript evaluation, and disconnect. | ||
|
|
||
| ## New MCP tools | ||
|
|
||
| Accessibility tools: | ||
| - `macos_ax_status` | ||
| - `macos_ax_list_apps` | ||
| - `macos_ax_find` | ||
| - `macos_ax_click` | ||
| - `macos_ax_type` | ||
| - `macos_ax_key` | ||
| - `macos_ax_activate` | ||
| - `macos_ax_wait_for` | ||
| - `macos_ax_batch` | ||
|
|
||
| Electron debug tools: | ||
| - `electron_debug_attach` | ||
| - `electron_debug_eval` | ||
| - `electron_debug_disconnect` | ||
|
|
||
| ## Native helper contract | ||
|
|
||
| `macos-ax-helper` reads one JSON request from `stdin` and returns one JSON response to `stdout`. | ||
|
|
||
| Request shape: | ||
| ```json | ||
| { | ||
| "requestId": "optional-id", | ||
| "command": "list_elements", | ||
| "args": {} | ||
| } | ||
| ``` | ||
|
|
||
| Success response: | ||
| ```json | ||
| { | ||
| "ok": true, | ||
| "data": {}, | ||
| "meta": {"requestId": "optional-id", "durationMs": 12} | ||
| } | ||
| ``` | ||
|
|
||
| Error response: | ||
| ```json | ||
| { | ||
| "ok": false, | ||
| "error": { | ||
| "code": "PERMISSION_DENIED", | ||
| "message": "Accessibility permissions not granted", | ||
| "details": {} | ||
| }, | ||
| "meta": {"requestId": "optional-id", "durationMs": 9} | ||
| } | ||
| ``` | ||
|
|
||
| Current helper commands: | ||
| - `status` | ||
| - `list_apps` | ||
| - `list_elements` | ||
| - `click` | ||
| - `type_text` | ||
| - `press_key` | ||
| - `activate` | ||
| - `wait_for` | ||
|
|
||
| ## Build helper binaries | ||
|
|
||
| From repo root: | ||
|
|
||
| ```bash | ||
| ./build-macos-helper.sh | ||
| ``` | ||
|
|
||
| Expected outputs: | ||
| - `bin/macos/macos-ax-helper-darwin-arm64` | ||
| - `bin/macos/macos-ax-helper-darwin-x64` | ||
|
|
||
| If only one architecture can be built on your machine, the script exits non-zero and prints guidance. | ||
|
|
||
| ## Permissions setup | ||
|
|
||
| Grant Accessibility permission to the process running Desktop Commander MCP. | ||
|
|
||
| macOS path: | ||
| - System Settings -> Privacy & Security -> Accessibility | ||
|
|
||
| Use `macos_ax_status` to verify permission state and process identification details. | ||
|
|
||
| ## Suggested usage flow | ||
|
|
||
| 1. `macos_ax_status` | ||
| 2. `macos_ax_list_apps` | ||
| 3. `macos_ax_find` (app + text/role) | ||
| 4. `macos_ax_click` (prefer by `id`) | ||
| 5. `macos_ax_wait_for` when UI transitions are asynchronous | ||
| 6. `macos_ax_batch` for multi-step flows | ||
|
|
||
| For Electron debug workflows: | ||
| 1. Launch target with remote debugging enabled | ||
| 2. `electron_debug_attach` | ||
| 3. `electron_debug_eval` | ||
| 4. `electron_debug_disconnect` | ||
|
|
||
| ## Test steps | ||
|
|
||
| 1. Build TypeScript: | ||
| ```bash | ||
| npm run build | ||
| ``` | ||
|
|
||
| 2. Run focused tests: | ||
| ```bash | ||
| node test/test-macos-control.js | ||
| node test/test-electron-debug.js | ||
| ``` | ||
|
|
||
| 3. Optional: run the full test suite: | ||
| ```bash | ||
| npm test | ||
| ``` | ||
|
|
||
| ## Optimization opportunities | ||
|
|
||
| - Keep a persistent helper process to avoid process spawn overhead per AX call. | ||
| - Add element path locators (`AXParent` chain) for stronger stale-ID recovery. | ||
| - Add incremental snapshot mode to avoid full tree scans on repeated `find`. | ||
| - Add richer CDP APIs (`DOM.querySelector`, click/type helpers) on top of `electron_debug_eval`. |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Duplicate
.build/entry..build/appears twice (Lines 61 and 63). Remove the duplicate.🔧 Proposed fix
.build/ .claude -.build/ bin/macos/macos-ax-helper-darwin-arm64📝 Committable suggestion
🤖 Prompt for AI Agents