Perf/vom hover probe opt in - #169
Merged
Merged
Conversation
Every tool.observe moved the real cursor across the page to look for hover-only menus and tooltips. Measured against real Chrome, that cost a median of 2.5s and up to 4.7s per observation — 73% to 99% of total observe wall clock — and returned 3 hits across 12 pages. A positive control confirms the probes work; the yield on real pages simply does not justify putting them on the default path. observe now probes only when asked, via `probe_hover` (`--probe-hover`). Agents that know which element hides a menu should keep using `bsk hover <ref>`, which is both cheaper and precise. A held hover latch still suppresses probing, since probing would move the cursor off the element the caller is deliberately holding. Correctness fixes that apply whether or not probing is requested: - Hover probing ran between the DOM snapshot and the accessibility tree fetch, so a hover that opened a menu left one half of an observation describing the page before the change and the other half after it. Both chains now run after all capture is complete. - The overlay bypass toggled once per chain and swallowed restore failures, which could leave the agent overlay hidden for the rest of the tab's life. It is now reference counted across both chains, and a failed restore is reported instead of discarded. - Observations report whether they hovered the page and whether that revealed content the static tree lacks, so callers can judge how far the live page may have drifted from the returned snapshot. Probe budgets were checked at the top of the loop, so a candidate could start with 1ms remaining and still run a full settle window, overshooting by up to 634ms. Candidates are now admitted only if they fit. Raising MAX_HOVER_PROBE_MS to 2600 is part of that fix rather than a relaxation: 2000 was never the real ceiling, and enforcing it literally would have silently dropped a candidate the positive control depends on. Tooltip misses are cached too, with a shorter TTL than hits, so repeat observations of the same page no longer re-probe every unlabelled control from scratch.
The previous wording asked the agent to notice when a page "looks like" it hides content behind hover. That is not something an agent reading a text tree can judge: hover-only content is invisible by definition, and the marker that used to point at it came from the probing that is now off by default. Replaces it with a condition the agent can actually evaluate — an expected control is missing and no [has-submenu] marker points at a trigger — and places it on the escalation ladder ahead of get-html and screenshot, which is where that dead end is currently reached. Also documents [has-submenu]/[expanded] as the primary hover signal. It comes from aria-haspopup in the accessibility tree, costs nothing, and is unaffected by probing being off, so it covers menus that are marked up correctly.
main rewrote SKILL.md into a leaner core (#167) while this branch documented the opt-in --probe-hover flag against the old structure. Keep main's rewritten prose and re-apply the probe-hover guidance on top of it: the [has-submenu] / [expanded] markers, when reaching for --probe-hover is warranted, and its slot in the page-reading escalation ladder ahead of get-html/screenshot. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
背景
#122 引入语义图的同时，在
tool.observe上挂了两条主动 hover 探测链路，且默认对每一次observe 都执行：
probeHoverSurfaces—— 扫描 CSS:hover规则，对候选元素真实移动鼠标，diff 前后 DOM 找出被揭示的浮层
probeTooltipNames—— 对没有可访问名称的交互控件真实 hover，读取 tooltip 补名字这两条链路的代价此前从未被测量过，参数是凭直觉调的。本 PR 先补测量，再据此改造。
测量结论（改造的依据）
真实 Chrome 152 + WebSocket 实现的
CdpRunner，直接调用未经修改的生产代码，埋点只加在 CDP 传输层。视口 1280×800，每个 URL 独立导航后静置 2.5s。
代价：12 个真实页面，hover 探测给 observe 增加中位数 2468 ms、最大 4730 ms，
占 observe 墙钟时间的 73%–99%（唯一例外是 github PR files 页，那里开销来自
32751 个 DOM 节点本身）。
产出：
probeHoverSurfacesprobeTooltipNames阳性对照是刻意构造的页面（CSS
:hover三级子菜单 + 三个无名图标按钮配[role=tooltip]），两条链路都正常命中。所以这不是 bug，是产出率问题：代码在它设计的页面上工作正常，
但这类页面在真实 web 上罕见。
链路 B 零命中的原因是合理的 —— 它只针对无可访问名称的控件，而 stackoverflow、github
这类站点的图标按钮基本都带
aria-label，在候选阶段就被正确排除了。它真正花大钱的场景是「有很多无名控件但都没有 tooltip」：news.ycombinator 与 stackoverflow 均触发满额 8 个候选、
耗时 2273 ms / 2306 ms、命中 0。
结论：探测能力本身没问题，问题是把它强加给了每一次 observe。
改动方案
1. 默认关闭，改为显式 opt-in
ObserveParams.probe_hover（#[serde(default)]，默认false）与 CLI flag
bsk observe --probe-hoverdispatcher.ts中conditionalSurfaceProbe: !hasHoverLatchForScope(...)改为
params.probe_hover === true && !hasHoverLatchForScope(...)这里有个语义变化值得单独说：hover latch 从「默认开关」降级为「opt-in 之上的抑制器」。
它原本的作用是「调用方正显式持有 hover 时不要探测」，现在依然如此 —— 因为探测会把光标
移开调用方正按着的元素 —— 只是它不再是决定探不探的主开关。
2. 正确性修复（无论是否 opt-in 都生效）
这三条是探测链路本身的缺陷，与开关无关：
captureViewModel内部执行，位置在 DOM 快照之后、AX 树抓取之前。一次打开菜单的 hover 会让同一份观察结果里 DOM 描述变更前的页面、
AX 描述变更后的页面。现改为
probeHoverSurfaces从captureViewModel移出并导出，由
observation.ts在 DOM 与 AX 均采集完成后调用，与链路 B 相邻。CapturedViewModel.surfaceProbes随之删除，探测结果改为显式参数传递，不再是 capture 的隐藏输出。
.catch(() => undefined)，一旦恢复失败，agent overlay 会在该 tab 的余下生命周期里一直处于隐藏状态。
新增
withOverlayBypass（hover-perception.ts）：按 tab 引用计数，两条链路共享一次overlay 切换；恢复失败走
onRestoreFailure（默认console.error）而非丢弃，且失败时仍释放计数，不会把 overlay 永久钉死。
CaptureVomObservationResult.hoverProbe与
ObserveResult.hover_probe，报告本次是否 hover 过页面、以及 hover 是否揭示了静态树中没有的内容，让调用方能判断返回的文本与实时页面可能偏离了多远。
3. 延迟与预算
实测链路 A 预算写 2000 ms、实际最高跑到 2634 ms，超出 634 ms —— 正好是一个候选的
完整开销（600 ms 睡眠 + CDP 往返）。新增
ProbeBudget.canAfford(estimatedMs)，两条链路都在开始候选前判断是否放得下。
MAX_HOVER_PROBE_MS2000 → 2600。这是上一条的配套，不是放宽：2000 从来不是真实上限，旧代码的事实上限就是 ~2.6 s。改成预估式检查后若仍写 2000，等于把预算收紧 600 ms、
少探一个候选（阳性对照因此掉了唯一一次命中，见下）。2600 是把旧代码的实际开销写成
可执行的约束：候选吞吐不变，而边界第一次变得诚实。
storeName只在命中时调用，未命中不记录。实测阳性对照第二次调用862 ms → 0.3 ms（缓存有效），而真实页面全部未命中，第二次调用 wikipedia 568 → 584 ms、
excalidraw 283 → 300 ms，完全没有收益 —— 同一页面反复 observe 会反复全额支付。
现在未命中也记录（
name: null，TTL 60 s；命中仍为 5 min）。4. Agent 指引（
skill/SKILL.md）探测默认关闭后，需要给 agent 一个可判定的触发条件。旧文案是「页面看起来把内容藏在
hover 后面时」—— 这不是一个读文本树的 agent 能判断的事，hover-only 内容按定义就是不可见的。
改为：
[has-submenu]/[expanded]记为首要 hover 信号。它来自 AX 树的aria-haspopup，零成本、始终可见、不受探测开关影响，覆盖标记规范的菜单 —— 此时直接
bsk hover <ref>，比探测更快也更精确
--probe-hover改为失败驱动：预期的控件找不到、且没有[has-submenu]指向任何触发器时才用get-html与screenshot之前，因为那正是目前走进死胡同的位置测试结果
性能：默认路径对默认路径
同一套 harness、同一批 URL、同样 settle 2.5 s。对比的是改造前默认开探测
与改造后默认关探测。
中位数 2636 ms → 130 ms（约 20 倍，-95%），平均值 2806 ms → 484 ms。
两点需要说明，避免选择性呈现：
jd.com前后都是 ~47 ms。该页在 harness 中稳定走 capture 回退路径，观察本身近乎为空，既不受益也不受损。保留在表中。
github-pr改造后仍有 3528 ms，但基线的「探测关闭」列本就是 3779 ms。这个页面的开销来自32751 个 DOM 节点 / 62901 个 AX 节点本身，不是 hover，不在本次改动的射程内。
可重复性：同一份代码独立重跑一轮，逐页差值在 -52 ms ~ +104 ms 之间，
中位数 130 ms / 108 ms，倍率 20x / 24x。结论不依赖单次运行的网络波动。
能力未退化：阳性对照
12 个页面的 opt-in 路径中位数 2610 ms，与改造前的默认路径持平 ——
探测行为本身没有被削弱，只是不再强加给每一次 observe。
阳性对照抓到的一个回归（已修）
第一次复测时链路 A 命中从 1 掉到 0。原因是预估式预算在保持
MAX_HOVER_PROBE_MS = 2000的前提下，把可探候选数从 4 压到 3，而该 fixture 上唯一命中的恰好是第 4 个（评分最低的）候选。
这说明「预算检查从事后改为事前」不是纯粹的正确性修复，它会真实地收紧探测预算。
把常量同步调整到旧代码的事实上限 2600 后，命中数恢复为 1，实际耗时仍在预算内。
这条只有真实浏览器 + 阳性对照才能发现，单元测试与 mock 都不会暴露它。
新增回归测试
observation.test.ts「hovers the page only after the accessibility tree has been captured」——断言首次
Input.dispatchMouseEvent晚于DOMSnapshot.captureSnapshot与Accessibility.getFullAXTreeobservation.test.ts「runs conditional surface probing only when observe opts in」——覆盖默认关闭 / 显式开启两条路径与
hover_probe上报hover-perception.test.ts（新文件）—— 覆盖 overlay 引用计数、异常路径恢复、恢复失败上报、失败后不钉死 overlay，以及预估式预算不超发
name-enrichment.test.ts「remembers controls that revealed no tooltip」—— 覆盖负缓存验证矩阵
apps/extension测试packages/vom测试tsc --noEmitbsk-protocol单测tools_ipc（本次改了参数）