Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .github/workflows/dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,11 @@ jobs:
run: python ./test/unit_test/keyboard/hotkey_test.py

# Mouse tests
# These three drive the real mouse and the real exit path on a hosted
# runner with no desk in front of it, so a failure says more about the
# runner's session than about the change under test. They are demo
# scripts (CLAUDE.md: the *_test.py files run on import), not the CI
# gate -- that is pytest-headless in quality.yml.
- name: Test Mouse Module
run: python ./test/unit_test/mouse/mouse_test.py
continue-on-error: true
Expand Down
30 changes: 19 additions & 11 deletions .github/workflows/platform-smoke.yml
Original file line number Diff line number Diff line change
Expand Up @@ -16,30 +16,38 @@ jobs:
matrix:
# arm64 is not a rounding error on the desktop any more, and the
# dependency set is where it shows. macos-14 is already arm64;
# ubuntu-22.04-arm adds Linux, and it passes.
# ubuntu-22.04-arm adds Linux, and windows-11-arm is back.
#
# windows-11-arm is deliberately absent, and it was measured rather
# than assumed. Two dependencies have no win_arm64 wheel, and both
# have to go before the runner is worth adding back:
# windows-11-arm installs a smaller dependency set than every other
# square, and that is deliberate rather than accidental. Two packages
# publish no win_arm64 wheel, so pyproject.toml marks them off this
# one platform:
#
# opencv-python — no win_arm64 wheel in any version, so pip falls
# back to building from source and CMake cannot
# opencv-python — no win_arm64 wheel in any version, so pip fell
# back to building from source and CMake could not
# configure for ARM64. Twelve minutes, then failure.
# cryptography — wheels stop at 46.0.3; 46.0.4 onwards ship none.
# Our floor is >=48.0.1 and that is a security floor
# (GHSA-537c-gmf6-5ccf), so it cannot be lowered.
#
# Neither is a CI problem to work around — the package genuinely
# cannot be installed on Windows arm64 today. Re-check without a
# runner, in about ten seconds:
# Nothing in the package imports either one at import time, so what
# this square proves is real: the install succeeds and the stable API
# works. Image matching, action signing, ACME and encrypted recording
# do not work there, and Progress.md says so. Re-check upstream in
# about ten seconds, no runner required:
#
# pip install --dry-run --only-binary=:all: --platform win_arm64 \
# --python-version 3.12 --target /tmp/probe \
# 'opencv-python>=4.8,<6' 'cryptography>=48.0.1'
#
# Recorded in Progress.md; add the runner back when both resolve.
os: [windows-2022, ubuntu-22.04, macos-14, ubuntu-22.04-arm]
# When both resolve, drop the markers from pyproject.toml.
os: [windows-2022, ubuntu-22.04, macos-14, ubuntu-22.04-arm, windows-11-arm]
python-version: ["3.10", "3.14"]
exclude:
# CPython's official Windows arm64 builds start at 3.11, so
# setup-python has no 3.10 interpreter to fetch on this runner.
- os: windows-11-arm
python-version: "3.10"
runs-on: ${{ matrix.os }}
steps:
- uses: actions/checkout@v4
Expand Down
5 changes: 5 additions & 0 deletions .github/workflows/stable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,11 @@ jobs:
run: python ./test/unit_test/keyboard/hotkey_test.py

# Mouse tests
# These three drive the real mouse and the real exit path on a hosted
# runner with no desk in front of it, so a failure says more about the
# runner's session than about the change under test. They are demo
# scripts (CLAUDE.md: the *_test.py files run on import), not the CI
# gate -- that is pytest-headless in quality.yml.
- name: Test Mouse Module
run: python ./test/unit_test/mouse/mouse_test.py
continue-on-error: true
Expand Down
12 changes: 12 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,18 @@ only when documented here with a migration path.

### Added

- **Windows on arm64 installs.** `opencv-python`, `cryptography` and
`je_open_cv` now carry the environment marker
`sys_platform != 'win32' or platform_machine != 'ARM64'`, because none of
the three publishes a `win_arm64` wheel and `pip install je_auto_control`
therefore failed on that platform before any of this code ran. Every other
platform resolves exactly the same dependency set as before. On Windows
arm64, the features that need those wheels — `find_image*`, the OpenCV
`screenshot()`, the secret vault, action-file encryption, ACME/TLS and
encrypted recording — raise a `RuntimeError` or `ImportError` naming the
missing wheel rather than a bare `ModuleNotFoundError`. Python 3.11 is the
floor there, since CPython publishes no official Windows arm64 build for
3.10.
- **The macOS recorder works.** `record()`, `stop_record()`,
`stop_record_timeline()`, the `AC_record*` commands, the `ac_record_*` MCP
tools and `je_auto_control record` all run on macOS now; they used to refuse
Expand Down
124 changes: 95 additions & 29 deletions Progress.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,53 +50,69 @@

---

## Windows arm64 裝不起來——是兩個上游,不是一個
## Windows arm64:裝得起來了,但少了影像與加密

`BLOCKED` — 上游(`opencv-python` 沒有 win_arm64 wheel;`cryptography` 在安全下限之上也沒有)
`TODO` — 上游仍未發 wheel(`opencv-python``cryptography`),但安裝本身不再是卡點

`windows-11-arm` 加進 `platform-smoke.yml` 的矩陣跑了一次
結果是實測而不是推測:**opencv-python 並沒有發
win_arm64 wheel**,pip 回退到從原碼建,CMake 在 ARM64 上
configure 不起來,花了十二分鐘失敗。所以那一格已從矩陣
移除,並把原因寫在 workflow 的註解裡
這一項曾經是 `BLOCKED`,而那個判斷只對一半。上游確實沒有發 wheel
這件事到今天(2026-08-20)重新實測依舊成立;但「裝不起來」卡的不是程式,
是 `pyproject.toml` 無條件要求那兩個套件。實測:把 `cryptography`、`cv2`、
`je_open_cv`、`numpy`、`PIL` 五個全擋掉之後,`import je_auto_control`、executor、
MCP 工具表、`cli`、`api.generate_code`、`api.create_failure_bundle` **全部照常跑**

門面已經不在 module scope import OpenCV 了(見
[WHATS_NEW.md](WHATS_NEW.md)),但這裡卡的不是 import
而是 **pip 裝不起來**:那幾個套件仍列在 `pyproject.toml`
的 `dependencies`,`pip install -e .` 第一步就會去建它們。
所以修法是一個 PEP 508 環境標記,三個相依共用同一個:

### 2026-08-20 重新實測:當初只數到一半
```
sys_platform != 'win32' or platform_machine != 'ARM64'
```

`windows-11-arm` 已經回到 `platform-smoke.yml` 的矩陣(只跑 3.14,CPython 的
官方 win-arm64 build 從 3.11 才有)。其他平台拿到的東西一個位元都沒變。

### 還沒有答案的:Windows arm64 上這些功能不能用

不必開 runner——`pip` 可以替別的平台解析,十秒就給出答案
裝得起來不等於功能齊。該平台上以下四組會拋帶提示的錯誤,而不是默默失效

| 依賴 | win_arm64 | 實測 |
| 功能 | 缺的是 | 錯誤形式 |
| --- | --- | --- |
| `opencv-python>=4.8,<6` | **沒有** | 任何版本都沒有,pip 回的是 `from versions: none`。`je_open_cv` 自己是純 Python,但它相依 opencv-python,所以一起卡。 |
| 影像比對、截圖轉 BGR、螢幕錄影 | `opencv-python`/`je_open_cv` | `utils/cv2_utils/optional.py` 的 `require_cv2()`/`require_je_open_cv()` 拋 `RuntimeError` |
| 動作檔加密(`action_signing`) | `cryptography` | `_fernet_types()` 拋 `RuntimeError`(簽章本身是 HMAC,不受影響) |
| 秘密金庫(`${secrets.NAME}`) | `cryptography` | 同上 |
| ACME/TLS 發證、加密錄影 | `cryptography` | 模組層 `ImportError` 轉述(照 `webrtc_transport` 慣例) |

這四組在 arm64 上能不能回來,**完全取決於上游**:

| 依賴 | win_arm64 | 實測(2026-08-20) |
| --- | --- | --- |
| `opencv-python>=4.8,<6` | **沒有** | 任何版本都沒有,pip 回的是 `from versions: none`。`je_open_cv` 自己是純 Python,但相依 opencv-python,所以一起卡——標記也必須一起下。 |
| `cryptography>=48.0.1` | **沒有** | wheel 只出到 **46.0.3**,46.0.4 起上游就不再發 win_arm64。而 `>=48.0.1` 是 347ec1e 為了 GHSA-537c-gmf6-5ccf(high)訂的**安全下限**,不能為了 arm64 降回去。 |
| `pillow==12.3.0` | 有 | `pillow-12.3.0-cp3xx-win_arm64.whl` 一直都在。**原本這裡寫「把 OpenCV/Pillow 移到 extra」,Pillow 那半是猜的,它從來不是卡點。** |
| `mss`/`defusedxml`/`je_open_cv` | 有 | 純 Python。 |
| `pillow==12.3.0` | 有 | `pillow-12.3.0-cp3xx-win_arm64.whl` 一直都在。**曾經被寫成卡點,那是猜的,它從來不是。** |
| `mss`/`defusedxml` | 有 | 純 Python。這三個加上 Pillow 就是 arm64 實際裝到的全部。 |
| `PySide6==6.11.1`/`qt-material==2.17` | 有 | `[gui]` extra 在 arm64 上裝得起來。 |
| `aiortc` | **沒有** | 卡在傳遞相依 `google-crc32c`,與本專案的選擇無關;`av` 自己有 wheel。 |

**所以原本那句「把 OpenCV/Pillow 移到 optional extra,arm64 就能只裝輸入的部分」
是不成立的**——就算 OpenCV 移走,`cryptography` 還是會把 `pip install` 擋在同一個
地方,而它的下限是安全下限,沒有往下讓的空間。要真的讓 arm64 裝得起來,**兩個都得
離開必裝集合**;`cryptography` 今天被六個模組用到(`acme_v2`、`tls_acme`、
`action_signing`、`secrets`、`remote_desktop` 的加密錄影),那是比 OpenCV 更大的
相容性決定。沒有人要求之前不做。

重驗指令(不需要 arm64 機器,也不需要 runner):

```bash
pip install --dry-run --only-binary=:all: --platform win_arm64 --python-version 3.12 --target /tmp/probe 'opencv-python>=4.8,<6' 'cryptography>=48.0.1'
```

兩行 `ERROR: No matching distribution` 就是現況。哪天其中一行不見了,就是上游發了
wheel,那時把 `windows-11-arm` 加回 `platform-smoke.yml` 的矩陣。
兩行 `ERROR: No matching distribution` 就是現況。**哪天其中一行不見了,就把
`pyproject.toml` 上那個標記拿掉**(三行一起),
`test/unit_test/headless/test_arm64_dependency_markers.py` 會帶著你改完。

**Linux arm64 是好的**——`ubuntu-22.04-arm` 兩個 Python 版本
都綠,macOS 本來就是 arm64。所以卡住的只有 Windows
這一個組合。
注意一個驗證上的陷阱:**`pip --platform` 不會換掉 marker 的評估環境**,
它只影響 wheel 相容性標籤,所以拿本機做 `--dry-run` **驗不到標記的效果**(兩個
套件依舊會被要求)。能驗的是兩件事:直接評估 marker(上面那支測試在做的),
以及 `windows-11-arm` 那一格自己綠。

### 一個刻意的取捨:cv2 只包兩扇門

`cv2` 在 33 個檔、共 76 句 import,全部是函式內 lazy。這次**只**在兩個大家一定會
經過的門換成 `require_cv2()`/`require_je_open_cv()`:`wrapper/auto_control_screen.py`(截圖)
與 `utils/cv2_utils/template_detection.py`(樣板比對)。其餘七十幾句維持原樣,在 arm64 上
會得到 `ModuleNotFoundError: No module named 'cv2'`。全包一輪是大面積 diff,且對呼叫端
並沒有多提供可以行動的資訊——哪天語意不足再說。

## Wayland:剩下的都不是「缺一台機器」

Expand Down Expand Up @@ -181,3 +197,53 @@ capability enum 值與 variadic `ei_seat_bind_capabilities`、event-type enum

**緩解**:驗不到的擷取部分有逃生門——`JE_AUTOCONTROL_WAYLAND_CAPTURE_COMMAND` 讓操作者
直接指定自己的擷取指令(`{output}` 會被換成暫存 PNG 路徑),優先於所有偵測。

---

## libei 的 `ei_unref` 在半開交握上會 SIGSEGV

`BLOCKED` — 上游(libei 1.3.901)

`linux_wayland/libei.py` 的 `_teardown` **刻意每個行程漏一個 context 與一個 fd**,
因為對一個還沒完成交握的 handle 呼叫 `ei_unref` 會直接 SIGSEGV。
這是在驅動使用者桌面的函式庫裡的 crash,所以寧可漏也不能當。

**這條本來就該在這裡。** 兩支 verify 腳本都會印 `*** REVISIT ***` 並叫讀者
來翻 `Progress.md`,而這裡一直什麼都沒寫:

- `docker/libei_verify.py`:「The workaround in `LibeiBackend._teardown` can probably go」
- `docker/eis_verify.py`:「`ei_unref` now SEGFAULTS on a live context too」

重驗方式就是跑那兩支腳本(`eis-verification` job 已經在跑);哪天 banner 不再
出現,就把 `_teardown` 的迴避拿掉。形狀與 arm64 那條一樣:卡上游、有一行重驗。

---

## 三個 Qt thread-marshal 測試永久跳過中

`TODO` — 需要子行程隔離,做法已知

`test/unit_test/headless/test_r3_gui_thread_marshal.py` 裡三個測試被無條件 skip:
`test_panel_signals_expose_file_received`、`test_webrtc_received_file_marshaled_to_gui`、
`test_thumbnail_poll_thread_is_reaped`。skip 理由自己寫著「needs subprocess
isolation (see test_actions_menu_gui) … skip until then」——那句「until then」就是這條。

跟 `CLAUDE.md` §Testing 記的 0xC0000409 `__fastfail` 是同一個家族:
worker→GUI 的 teardown 在共用的 pytest 行程裡把整個監獸帶走。
`test_actions_menu_gui.py` 已經示範過解法(把建 widget 的部分丟進子行程),
這三個只是還沒改過去。沒改之前,這三條路徑沒有任何回歸保護。

---

## 兩個講好要爬、還沒爬的門檻

`TODO` — 兩者都寫在 `pyproject.toml` 的註解裡,但不在任何待辦清單上

- **覆蓋率**:`fail_under = 35`,註解寫著「Raise toward 70 as legacy modules are
brought under the stable API contract」。目標是 70,今天是 35,中間沒有計畫。
- **mypy 範圍**:CI 只型別檢查兩條路徑(`quality.yml` 的
`mypy je_auto_control/api je_auto_control/utils/failure_bundle`)。註解寫著
「followed legacy modules are analysed for signatures but not reported until they
join the contract」——同樣是講好要擴、還沒擴。

兩者都不是一次做得完的事,但放在這裡至少讓「下一步是什麼」有一個地方可寫。
14 changes: 13 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,19 @@ Optional extras, installed only when you need them:
| `fuzzy` / `locale` | `rapidfuzz` matching, `babel` locale parsing |
| `s3` / `audio` | S3 artifact store, system volume control |

**Requirements:** Python ≥ 3.10. On Linux, install build prerequisites first:
**Windows on arm64** installs and runs, minus what upstream cannot ship
there: neither `opencv-python` nor `cryptography` publishes a `win_arm64`
wheel. So `find_image*`, `screenshot()` (the OpenCV/BGR one — the Pillow
capture still works), the secret vault, action-file encryption, ACME/TLS
and encrypted recording each raise a message naming the missing wheel
instead of failing obscurely. Mouse, keyboard, screen size, window
management, the accessibility tree, the action executor, the MCP/REST/TCP
servers and the GUI all work — measured, not assumed. Every other platform
is unaffected.

**Requirements:** Python ≥ 3.10 (≥ 3.11 on Windows arm64, which is where
CPython's official builds for it start). On Linux, install build
prerequisites first:

```bash
sudo apt-get install cmake libssl-dev
Expand Down
10 changes: 9 additions & 1 deletion README/README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,15 @@ pip install je_auto_control[gui] # 加上 PySide6 桌面应用
| `fuzzy` / `locale` | `rapidfuzz` 模糊匹配、`babel` 区域解析 |
| `s3` / `audio` | S3 制品存储、系统音量控制 |

**系统需求:** Python ≥ 3.10。Linux 请先安装构建依赖:
**Windows arm64** 装得起来也跑得起来,少的是上游在那里发不出来的那些:
`opencv-python` 与 `cryptography` 都没发 `win_arm64` wheel。所以 `find_image*`、
`screenshot()`(OpenCV/BGR 那一支——Pillow 截图仍可用)、密钥金库、动作文件加密、
ACME/TLS 与加密录影会抛出指名缺哪个 wheel 的错误,而不是难以追查的失败。
鼠标、键盘、屏幕尺寸、窗口管理、无障碍树、动作执行器、MCP/REST/TCP 服务器
与 GUI 都正常——这是实测的,不是推论的。其他平台不受影响。

**系统需求:** Python ≥ 3.10(Windows arm64≥ 3.11,CPython 官方构建从那里开始)。
Linux 请先安装构建依赖:

```bash
sudo apt-get install cmake libssl-dev
Expand Down
10 changes: 9 additions & 1 deletion README/README_zh-TW.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,15 @@ pip install je_auto_control[gui] # 加上 PySide6 桌面應用程式
| `fuzzy` / `locale` | `rapidfuzz` 模糊比對、`babel` 地區解析 |
| `s3` / `audio` | S3 產出物儲存、系統音量控制 |

**系統需求:** Python ≥ 3.10。Linux 請先安裝建置前置套件:
**Windows arm64** 裝得起來也跑得起來,少的是上游在那裡發不出來的那些:
`opencv-python` 與 `cryptography` 都沒發 `win_arm64` wheel。所以 `find_image*`、
`screenshot()`(OpenCV/BGR 那一支——Pillow 截圖仍可用)、秘密金庫、動作檔加密、
ACME/TLS 與加密錄影會拋出指名缺哪個 wheel 的錯誤,而不是難以追查的失敗。
滑鼠、鍵盤、螢幕尺寸、視窗管理、無障礙樹、動作執行器、MCP/REST/TCP 伺服器
與 GUI 都正常——這是實測的,不是推論的。其他平台不受影響。

**系統需求:** Python ≥ 3.10(Windows arm64≥ 3.11,CPython 官方建置從那裡開始)。
Linux 請先安裝建置前置套件:

```bash
sudo apt-get install cmake libssl-dev
Expand Down
Loading
Loading