Skip to content

fix(desktop): take the macOS queue off the Desktop Tests critical path - #95852

Draft
posthog[bot] wants to merge 1 commit into
masterfrom
posthog-self-driving/perfci-cut-the-desktop-tests-p95-tail-baa1ca
Draft

fix(desktop): take the macOS queue off the Desktop Tests critical path#95852
posthog[bot] wants to merge 1 commit into
masterfrom
posthog-self-driving/perfci-cut-the-desktop-tests-p95-tail-baa1ca

Conversation

@posthog

@posthog posthog Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

Problem

  • Desktop engineers wait up to 37 minutes for Desktop Tests. Almost all of that wait is queueing, not testing.
  • The macOS job's own run time is flat. Its wait for a depot-macos-26 slot is what moves, from one second to sixty minutes. depot-macos-26 is a fixed pool with FIFO queuing, and five desktop workflows share it.
  • That is why p95 roughly doubled while p50 stayed flat: two run shapes, not a uniform slowdown.
  • The Linux e2e job waited on the macOS job, so the whole macOS queue sat on its critical path.
  • The web E2E suite drives a plain SPA in stock Chromium. It held a slot in the fixed macOS pool for no reason.

Measured on the integration-test job, over sampled successful runs either side of 2026-09-03:

Run shape Wait for a macOS slot Job run time
Uncontended 1s - 61s 5m14s - 5m30s
Contended 8m24s - 60m38s 5m28s - 5m50s
The suspected cold turbo cache is not the cause

The report suspected that dda7614 left build:deps running cold on macOS. Per-step timings say otherwise. Build packages took 8-13s before that commit and 6-13s after, so it never went cold. Depot Cache serves turbo on these runners, which is what the commit assumed. No cache is restored here.

Changes

  • Web E2E now runs on its own Linux job. It reports its own pass or fail, and no longer holds a slot in the macOS pool.
  • The live-model e2e job now starts as soon as the unit job passes, instead of queueing behind macOS. A red tree still never reaches the gateway.
  • With one suite left on macOS, the continue-on-error handling and the trailing verdict step are gone. Mechanical.
  • Desktop Tests Pass gates on the new web-test job. Mechanical.

Before:

flowchart LR
    changes --> unit-test
    changes --> integration-test
    unit-test --> e2e
    integration-test --> e2e
    integration-test -. holds a macOS slot .-> webe2e[web E2E suite]
    e2e --> pass[Desktop Tests Pass]
    integration-test --> pass
    style integration-test fill:#F54E00,color:#fff
    style webe2e fill:#F54E00,color:#fff
Loading

After:

flowchart LR
    changes --> unit-test
    changes --> integration-test
    changes --> web-test
    unit-test --> e2e
    e2e --> pass[Desktop Tests Pass]
    integration-test --> pass
    web-test --> pass
    style integration-test fill:#F54E00,color:#fff
    style web-test fill:#1D4AFF,color:#fff
Loading

Related but separate: #94509 reduces macOS pool pressure from the release workflow. This PR touches only desktop-test.yml.

How did you test this code?

  • Ran the moved suite on Linux in the task sandbox, the first time it has run outside macOS: pnpm run test:e2e:web with CI=true, 2 passed. The job's build step (turbo build --filter=@posthog/web^...) and Chromium install were run the same way first.
  • Confirmed the report artifact path the upload step keeps: the run wrote products/desktop/apps/web/playwright-report/.
  • actionlint on the changed workflow, with the repo's config and the parallel-steps flattener: clean.
  • hogli lint:workflows: 9 of 9 checks pass across 133 workflows.
  • No new tests. This is CI wiring, and the suites themselves are unchanged.
  • Not verified: the queue behavior itself. That only shows up over a week of real runs.

Automatic notifications

  • Publish to changelog?

Docs update

None.

🤖 Agent context

Autonomy: Fully autonomous

  • Written by Claude Opus 5 in a PostHog cloud task, from a self-driving inbox report on the Desktop Tests duration alerts.
  • Skills invoked: /writing-simplified-technical-english, plus the repo's depot-github-runners skill for the fixed-pool and Depot Cache behavior.
  • The report proposed restoring the deleted turbo cache first. Per-step timings from the run API ruled that out, so no cache change is in this diff. Measuring instead surfaced the real driver, macOS slot contention, which the report's two structural fixes happen to target.
  • Considered and rejected: changing the workflow's concurrency group so master pushes stop serializing. That would cancel per-commit test signal on master, which is a call for the desktop team rather than a latency fix.
  • No duplicate: gh pr list --state open --search on the workflow name and on the symptom found no PR touching desktop-test.yml.
  • Public artifact: everything here comes from this repository and its public Actions run data.
  • Agent memory for the fleet was not written. The scratchpad tools were not available in this session.

Created with PostHog Desktop from this inbox report.

The p95 of Desktop Tests is queue time for a depot-macos-26 slot, not test
time: the macOS job runs in a flat 5m30s on every sampled run, while its wait
for a slot ranges from one second to sixty minutes.

Move the web E2E suite to its own Linux job. It drives a plain SPA in stock
Chromium, so it needs no macOS and no longer holds a slot in the fixed pool.

Drop integration-test from the e2e job's needs. That Linux job only requires a
green unit run, so waiting on macOS put the whole pool queue on its critical
path.

With one suite left in the macOS job, its continue-on-error handling and the
trailing verdict step are no longer needed.

Generated-By: PostHog Desktop
Task-Id: ddbc4e41-e31d-41b3-9504-4fe96d04fa9b
@trunk-io

trunk-io Bot commented Sep 7, 2026

Copy link
Copy Markdown

Merging to master in this repository is managed by Trunk.

  • To merge this pull request, check the box to the left or comment /trunk merge below.

After your PR is submitted to the merge queue, this comment will be automatically updated with its status. If the PR fails, failure details will also be posted here

@posthog

posthog Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor Author

🦔 PostHog Review reviewed this pull request

Nothing worth raising this time, so here's a calming picture instead:

A panda relaxing and waving

@github-actions github-actions Bot added the feature/desktop Feature Tag: Desktop label Sep 7, 2026
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown
Contributor

🤖 CI report

⚠️ Trunk lane — backend Python lane

This PR is assigned to the backend Python lane. It runs backend Python tests and may merge in parallel with PRs in other lanes.

@stamphog stamphog Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not approved yet — waiting on the conditions below.

Re-add the stamphog label to request another review once you have addressed this.

Gates denied: this is a CI workflow change (deny-listed infra/CI territory) classified as never-auto-approvable, and it has zero human or agent reviews providing independent assurance.

  • Modifies .github/workflows/desktop-test.yml, which is on the CI/infra deny-list and gated as T2-never — requires human review regardless of diff content.
  • No reviews or approvals present to serve as independent assurance for this risky-territory change.
Gate mechanics and policy version
Gate Result
prerequisites all clear
deny-list matches: infra_cicd
size 92L, 1F substantive — within ceiling
tier classified as T2-never: T2-never (92L, 1F, single-area, fix)
stamphog 2.0.0b4 .stamphog/policy.yml @ 651701c · reviewed head 651701c

@stamphog stamphog Bot added the reviewhog ($$$) Reviews pull requests before humans do label Sep 7, 2026
@posthog posthog Bot removed the reviewhog ($$$) Reviews pull requests before humans do label Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature/desktop Feature Tag: Desktop

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants