trunk-merge/pr-89870/30164a18-b801-4c70-822b-f27eca17d2f9

.agents/skills/authoring-ci-workflows/SKILL.md

@@ -9,6 +9,8 @@ description: >
 
 # Authoring CI workflows
 
+Before you propose a change to CI, check [things already tried](../../../docs/internal/ci-things-already-tried.md) for the idea. It records what was measured, and why some good-sounding changes were reverted or rejected.
+
 Conventions for `.github/workflows/**` and `.github/actions/**`.
 The linters own the mechanical rules (below); this skill is the **judgment calls** they can't enforce.
 

.agents/skills/debugging-ci-failures/SKILL.md

@@ -15,6 +15,8 @@ description: >
 
 # Debugging PostHog CI failures
 
+Before you propose a change to CI, check [things already tried](../../../docs/internal/ci-things-already-tried.md) for the idea. It records what was measured, and why some good-sounding changes were reverted or rejected.
+
 Find the first meaningful failure, classify it, reproduce the smallest useful
 case locally when appropriate, and report the result. Avoid public-visible or
 irreversible actions unless the user explicitly asks.

.agents/skills/depot-container-builds/SKILL.md

@@ -12,6 +12,8 @@ description: >
 
 # Depot Container Builds
 
+Before you propose a change to CI, check [things already tried](../../../docs/internal/ci-things-already-tried.md) for the idea. It records what was measured, and why some good-sounding changes were reverted or rejected.
+
 Depot runs Docker image builds on remote high-performance builders (16 CPU, 32 GB RAM, NVMe SSD cache). `depot build` is a drop-in replacement for `docker build` / `docker buildx build`. `depot bake` replaces `docker buildx bake`.
 
 ## Project Selection for Multi-Org Users

.agents/skills/django-migrations/SKILL.md

@@ -5,6 +5,8 @@ description: Django migration patterns and safety workflow for PostHog. Use when
 
 # Django migrations
 
+Before you propose a change to the migration history or to how migrations run, check [things already tried](../../../docs/internal/ci-things-already-tried.md). It records the closed attempts at squashing the history and at the Person table cutover.
+
 Read these files first, before writing or editing a migration:
 
 - `docs/published/handbook/engineering/developing-locally.md` (`## Django migrations`, `### Non-blocking migrations`, `### Resolving merge conflicts`)

.agents/skills/fixing-flaky-tests/SKILL.md

@@ -9,6 +9,8 @@ description: >
 
 # Fixing flaky tests
 
+Before you propose a change to how the suite runs in CI, check [things already tried](../../../docs/internal/ci-things-already-tried.md). It records measured verdicts on test parallelism, sharding, and coverage-based selection, so a rejected approach is not rebuilt.
+
 Three non-negotiables, in order:
 
 1. **Reproduce before you fix.** A fix for a failure you never observed is a guess.

.agents/skills/improving-drf-endpoints/SKILL.md

@@ -5,6 +5,8 @@ description: Use when editing, reviewing, or auditing DRF viewsets and serialize
 
 # Improving DRF Endpoints
 
+Before you propose a contract test against the generated OpenAPI schema, check [things already tried](../../../docs/internal/ci-things-already-tried.md). Six PRs took that idea, and none merged.
+
 ## Overview
 
 Serializer fields are the source of truth for PostHog's entire type pipeline:

.agents/skills/isolating-product-facade-contracts/SKILL.md

@@ -5,6 +5,8 @@ description: Plan and execute product isolation migrations to a facade plus cont
 
 # Isolating a product with facade and contracts
 
+Before you choose a product to isolate, check [things already tried](../../../docs/internal/ci-things-already-tried.md). It records which product was the first candidate, and why the field test moved to another one.
+
 Use this skill to migrate an existing product to the isolated architecture used by Visual review.
 Optimize for short calendar exposure, not small diffs: authoring is cheap and the verification
 chain catches mechanical breakage, while human review latency and a fast-moving master are the

.agents/skills/maintaining-python-tests/SKILL.md

@@ -6,6 +6,8 @@ description: >
 
 # Maintaining Python tests
 
+Before you propose a change to how the suite runs in CI, check [things already tried](../../../docs/internal/ci-things-already-tried.md). It records measured verdicts on test parallelism, sharding, and coverage-based selection, so a rejected approach is not rebuilt.
+
 Use this skill for an existing Python test suite. Use `/writing-tests` before adding or substantially changing coverage. Use `/fixing-flaky-tests` when intermittent failure is the main problem.
 
 The goal is not a smaller test count. The goal is a suite that catches the same realistic regressions with less compute, less waiting, and less maintenance.

.agents/skills/stacking-prs/SKILL.md

@@ -12,6 +12,8 @@ description: >
 
 # Stacked PRs with `gh stack`
 
+Before you divide a change into layers, check [things already tried](../../../docs/internal/ci-things-already-tried.md). It records when a stack cost more review effort than one PR.
+
 GitHub's native stacked PRs are enabled on this repo.
 A stack is an ordered chain of PRs where each one targets the branch of the PR below it; the bottom PR targets `master`.
 GitHub tracks the chain as a first-class object: the PR UI shows a stack map, branch protections (code owner approval, required checks) apply to **every** layer including mid-stack ones, and CI that runs on `master` PRs runs on every layer.

.depot/workflows/ci-backend-update-test-timing.yml

@@ -67,9 +67,14 @@ jobs:
                       posthog-test-durations-
 
             - name: Set the previous map aside
+              # .test_durations is not tracked, so there is nothing to restore it from
+              # and a cache miss leaves no file at all.
               run: |
-                  cp .test_durations /tmp/previous_durations
-                  git checkout -- .test_durations
+                  if [ -f .test_durations ]; then
+                      mv .test_durations /tmp/previous_durations
+                  else
+                      echo "::warning::No cached map restored; a drifting Products slice has no stand-in"
+                  fi
 
             - name: Download timing artifacts
               env:
@@ -103,12 +108,21 @@ jobs:
                   set -e
                   if [ "$products_status" -ne 0 ]; then
                       echo "::warning::Products durations refresh exited $products_status; keeping the previous products slice. Drift or an incomplete JUnit set are the expected causes, see the log above."
-                      # Only entries the fresh slices do not carry: the outlier merge would
-                      # otherwise let a stale value here beat a fresh one for the same test.
-                      jq -s 'add' $(ls /tmp/core_durations /tmp/temporal_durations 2>/dev/null) > /tmp/fresh_durations
-                      jq --slurpfile fresh /tmp/fresh_durations \
-                          'with_entries(select((.key | startswith("products/")) and ($fresh[0][.key] == null)))' \
-                          /tmp/previous_durations > /tmp/products_durations
+                      if [ -f /tmp/previous_durations ]; then
+                          # Only entries the fresh slices do not carry: the outlier merge would
+                          # otherwise let a stale value here beat a fresh one for the same test.
+                          jq -s 'add' $(ls /tmp/core_durations /tmp/temporal_durations 2>/dev/null) > /tmp/fresh_durations
+                          jq --slurpfile fresh /tmp/fresh_durations \
+                              'with_entries(select((.key | startswith("products/")) and ($fresh[0][.key] == null)))' \
+                              /tmp/previous_durations > /tmp/products_durations
+                      else
+                          # No cached slice to stand in and no fresh one. A map with no product
+                          # entries would still become the newest cache and the latest successful
+                          # artifact, so every consumer would fall back to the file-count estimate
+                          # and under-shard the fixture-heavy products. Leave the older map newest.
+                          echo "::error::Products slice failed with no previous map to stand in; refusing to publish a map without product entries"
+                          exit 1
+                      fi
                   fi
                   uv run .github/scripts/optimize_test_durations.py .test_durations \
                       --merge-files /tmp/core_durations /tmp/temporal_durations /tmp/products_durations \

.depot/workflows/ci-backend.yml

@@ -942,8 +942,15 @@ jobs:
             - name: Run product tests
               # --force: discover already decided this product needs testing, skip turbo cache
               # --log-order=stream: stream pytest output live instead of buffering until completion
-              # pytest_args: optional pytest-split flags for sharded products (e.g. "-- --splits 3 --group 1")
+              # legs: one turbo invocation per entry, each with its own pytest-split flags
+              # (e.g. "-- --splits 3 --group 1"). A job holds several legs when a split
+              # product's last shard had budget left over for whole small products, and
+              # pytest-split flags must not leak from that shard onto its job-mates.
               env:
+                  PRODUCT_LEGS: ${{ toJSON(matrix.legs) }}
+                  # Read only when an unrebased branch planned the pre-legs shape.
+                  LEGACY_FILTERS: ${{ matrix.filters }}
+                  LEGACY_PYTEST_ARGS: ${{ matrix.pytest_args }}
                   CLICKHOUSE_HOGQL_USE_NEW_EVENTS_SCHEMA: ${{ matrix.new-events-schema && 'true' || 'false' }}
                   # products/tasks/backend/temporal moved here from the Django Temporal
                   # segment, and its conftest and workflows talk to Modal. Injected only
@@ -965,17 +972,42 @@ jobs:
                   # bin-packed buckets (each product writes its own file) and split products (unioned later).
                   # sysmon: Python 3.12+'s low-overhead coverage backend, ~a few % vs ~20% for the C tracer.
                   COVERAGE_CORE: sysmon
+              shell: bash
               run: |
                   set +e
-                  pnpm turbo run backend:test ${{ matrix.filters }} --concurrency=1 --output-logs=full --force --log-order=stream ${{ matrix.pytest_args }}
-                  exit_code=$?
-                  set -e
-                  if [ $exit_code -eq 5 ]; then
-                      echo "No tests collected for this shard, this is expected when splitting tests"
-                      exit 0
-                  else
-                      exit $exit_code
+                  overall=0
+                  # A workflow edit reaches an open PR before the script edit does, so a
+                  # branch that has not rebased still plans the flat {filters, pytest_args}
+                  # shape. Normalize that into one leg rather than requiring the new key.
+                  if [ -z "$PRODUCT_LEGS" ] || [ "$PRODUCT_LEGS" = "null" ]; then
+                      PRODUCT_LEGS=$(jq -cn --arg f "$LEGACY_FILTERS" --arg a "$LEGACY_PYTEST_ARGS" \
+                          '[{filters: $f, pytest_args: $a}]')
+                      echo "Matrix entry carries no legs; running the legacy single invocation"
+                  fi
+                  leg_count=$(jq 'length' <<< "$PRODUCT_LEGS")
+                  # Neither shape present means the job would pass having run no tests, and
+                  # the rollup gate only reads this step's outcome. Fail loudly instead.
+                  if [ -z "$leg_count" ] || [ "$leg_count" -lt 1 ]; then
+                      echo "::error::No test legs in this matrix entry; refusing to pass without running tests"
+                      exit 1
                   fi
+                  for ((leg_index = 0; leg_index < leg_count; leg_index++)); do
+                      leg_filters=$(jq -r ".[$leg_index].filters" <<< "$PRODUCT_LEGS")
+                      leg_args=$(jq -r ".[$leg_index].pytest_args" <<< "$PRODUCT_LEGS")
+                      echo "::group::turbo backend:test $leg_filters $leg_args"
+                      # Word splitting is intended here: both variables carry several flags.
+                      # shellcheck disable=SC2086
+                      pnpm turbo run backend:test $leg_filters --concurrency=1 --output-logs=full --force --log-order=stream $leg_args
+                      exit_code=$?
+                      echo "::endgroup::"
+                      if [ $exit_code -eq 5 ]; then
+                          echo "No tests collected for this leg, this is expected when splitting tests"
+                      elif [ $exit_code -ne 0 ]; then
+                          overall=$exit_code
+                      fi
+                  done
+                  set -e
+                  exit $overall
     # Lightweight repo-wide checks that only need Python + uv (no Docker/DB).
     # Consolidates checks that previously each spun up their own runner.
     repo-checks:

.github/scripts/turbo-discover-sizing.test.js

@@ -7,7 +7,7 @@
 const test = require('node:test')
 const assert = require('node:assert/strict')
 
-const { pruneDeadDurations, getSegmentDuration, calculateShards, resolveProductSizing, buildMatrix, PRODUCT_JOB_OVERHEAD_SECONDS, PRODUCT_SAFETY_FACTOR, TARGET_WALL_SECONDS } = require('./turbo-discover.js')
+const { pruneDeadDurations, getSegmentDuration, calculateShards, resolveProductSizing, buildMatrix, productSplitShards, PRODUCT_JOB_OVERHEAD_SECONDS, TARGET_WALL_SECONDS } = require('./turbo-discover.js')
 
 // A path that exists in every checkout, so the existence check is deterministic.
 const LIVE_FILE = '.github/scripts/turbo-discover.js'
@@ -95,8 +95,8 @@ test('buildMatrix splits a product to the shared wall target', () => {
 
     const matrix = buildMatrix(['big-one'], union, true)
 
-    // 2000s of work, with the safety factor, over a (target - overhead) budget per shard.
-    assert.equal(matrix.length, Math.ceil((2000 * PRODUCT_SAFETY_FACTOR) / (TARGET_WALL_SECONDS - O)))
+    // 2000s of work, with the imbalance margin, over a (target - overhead) budget.
+    assert.equal(matrix.length, productSplitShards({ work: 2000, heavyCount: 0, lightWork: 2000, maxLight: 50, testCount: 40 }))
     assert.match(matrix[0].group, /^big-one \(1\/\d+\)$/)
 })
 
@@ -107,5 +107,108 @@ test('buildMatrix leaves a small product packed', () => {
 
     assert.equal(matrix.length, 1)
     assert.equal(matrix[0].group, 'small-one')
+    assert.deepEqual(matrix[0].legs, [{ filters: '--filter=@posthog/products-small-one', pytest_args: '' }])
+})
+
+test('a single-invocation entry keeps the pre-legs keys for unrebased branches', () => {
+    // A workflow edit lands on an open PR before this script does, so an old
+    // workflow reading matrix.filters must still find something to run.
+    const union = { 'products/small_one/backend/test_c.py::test_c': 100 }
+
+    const matrix = buildMatrix(['small-one'], union, true)
+
+    assert.equal(matrix[0].filters, '--filter=@posthog/products-small-one')
     assert.equal(matrix[0].pytest_args, '')
 })
+
+test('productSplitShards sizes the worst chunk, not the mean', () => {
+    const budget = TARGET_WALL_SECONDS - PRODUCT_JOB_OVERHEAD_SECONDS
+    const evenly = { work: 1000, heavyCount: 0, lightWork: 1000, maxLight: 10, testCount: 100 }
+    const coarse = { work: 1000, heavyCount: 0, lightWork: 1000, maxLight: 150, testCount: 100 }
+
+    // Same total work; the coarser grain cannot be cut as finely, so it needs more
+    // shards to keep its worst chunk inside the budget.
+    assert.ok(productSplitShards(coarse) > productSplitShards(evenly))
+    assert.ok(evenly.lightWork / productSplitShards(evenly) + evenly.maxLight <= budget)
+    assert.ok(coarse.lightWork / productSplitShards(coarse) + coarse.maxLight <= budget)
+})
+
+test('tests above half the budget each hold a shard of their own', () => {
+    const budget = TARGET_WALL_SECONDS - PRODUCT_JOB_OVERHEAD_SECONDS
+    const heavy = Math.ceil(budget * 0.8)
+
+    // Ten tests this size cannot pair, so no count below ten holds the budget,
+    // however the total work divides.
+    assert.equal(productSplitShards({ work: heavy * 10, heavyCount: 10, lightWork: 0, maxLight: 0, testCount: 10 }), 10)
+    // One heavy test and a sliver stays bounded rather than asking for a shard per
+    // second of remainder. Two tests cannot fill three shards, so the count stops
+    // there rather than planning one that collects nothing.
+    assert.equal(
+        productSplitShards({ work: budget + 1, heavyCount: 1, lightWork: 1, maxLight: 1, testCount: 2 }),
+        2
+    )
+})
+
+test('the count never exceeds the tests there are to place', () => {
+    // The fragmentation charge assumes a shape the suite may not have. Past the
+    // test count a shard collects nothing and spends a runner for it.
+    const many = { work: 10000, heavyCount: 3, lightWork: 100, maxLight: 10, testCount: 5 }
+
+    assert.equal(productSplitShards(many), 5)
+})
+
+test('a product holding one test is never split', () => {
+    const budget = TARGET_WALL_SECONDS - PRODUCT_JOB_OVERHEAD_SECONDS
+
+    // One test over the budget still gets one job: a second would collect nothing,
+    // and no split shortens the first.
+    assert.equal(
+        productSplitShards({ work: budget + 1, heavyCount: 1, lightWork: 0, maxLight: 0, testCount: 1 }),
+        1
+    )
+})
+
+test('a heavy test between light ones splits the light run', () => {
+    // Ordered [53, 301, 133] against a 320s budget: either contiguous cut leaves a
+    // 354s or 434s chunk, so two shards cannot hold the budget however it is cut.
+    assert.equal(productSplitShards({ work: 487, heavyCount: 1, lightWork: 186, maxLight: 133, testCount: 3 }), 3)
+})
+
+test('a product that fits one shard is packed, not split by its own margin', () => {
+    // 300s of work sits under the (target - overhead) budget, so the margin must
+    // not be what pushes it over into a two-way split.
+    const union = {}
+    for (let i = 0; i < 10; i++) {
+        union[`products/mid_one/backend/test_${i}.py::test_${i}`] = 30
+    }
+
+    assert.ok(300 <= TARGET_WALL_SECONDS - PRODUCT_JOB_OVERHEAD_SECONDS)
+    assert.equal(productSplitShards({ work: 300, heavyCount: 0, lightWork: 300, maxLight: 30, testCount: 10 }), 1)
+
+    const matrix = buildMatrix(['mid-one'], union, true)
+
+    assert.equal(matrix.length, 1)
+    assert.equal(matrix[0].group, 'mid-one')
+})
+
+test("a split product's last shard absorbs a small product without leaking split flags", () => {
+    const union = {}
+    for (let i = 0; i < 11; i++) {
+        union[`products/big_one/backend/test_${i}.py::test_${i}`] = 30
+    }
+    union['products/small_one/backend/test_s.py::test_s'] = 40
+
+    assert.equal(productSplitShards({ work: 330, heavyCount: 0, lightWork: 330, maxLight: 30, testCount: 11 }), 2)
+
+    const matrix = buildMatrix(['big-one', 'small-one'], union, true)
+
+    // Two jobs, not three: small-one rides along in the lighter second shard.
+    assert.equal(matrix.length, 2)
+    const shared = matrix.find((entry) => entry.group.includes('small-one'))
+    assert.equal(shared.group, 'big-one (2/2), small-one')
+    assert.equal(shared.legs.length, 2)
+    assert.match(shared.legs[0].pytest_args, /--splits 2 --group 2/)
+    // The whole product runs in its own leg, so it never sees --splits/--group.
+    assert.equal(shared.legs[1].filters, '--filter=@posthog/products-small-one')
+    assert.equal(shared.legs[1].pytest_args, '')
+})