refactor(metrics): unify the percentile convention; address #417 round-3 review

docs/early_stopping.md

@@ -17,7 +17,10 @@ estimate stays honest, which is exactly the regime edge/T2V workloads live in.
 
 For each target percentile `p` — every entry of the series' own report percentile grid at or above
 the median (`es_targets_from_grid`; the default grid yields p50/p75/p80/p90/p95/p97/p99/p99.9) —
-at confidence `c = 0.99`, over the `n` ascending-sorted latencies of a series:
+at confidence `c = 0.99`, over the `n` ascending-sorted latencies of a series
+(percentiles use the grid convention — 0-100 — at every surface; only the LoadGen-parity
+kernel keeps LoadGen's fraction domain, converted exactly once and unrounded, the same `/100`
+`np.percentile` applies internally):
 
 ```
 estimate = sorted[n - t],  t = max{ i : n >= find_min_passing(i, p, d, c) + i }
@@ -35,7 +38,7 @@ lowering `c` or raising `d` weakens the certified claim (`d > 0` certifies perce
 The pure math keeps defaulted arguments for parity tests only.
 
 **The percentile targets are not a separate list either** — they derive from the series' report
-percentile grid, filtered to `p ≥ 0.5` (`ES_MIN_PERCENTILE`): the estimate is a _tail_
+percentile grid, filtered to `≥ p50` (`ES_MIN_PERCENTILE = 50.0`, grid convention): the estimate is a _tail_
 certification (a conservative upper confidence bound), so below-median grid entries are skipped.
 One source of truth: whatever percentiles a series reports, ES covers — every scenario's gate
 percentile (p99 Server, p90 SingleStream/T2V) is always included, with nothing to tune.
@@ -44,6 +47,65 @@ Each estimate is a _marginal_ `c`-confidence statement per percentile. Reporting
 fine for diagnostics, but a joint gate across all of them holds at lower than `c` confidence
 (multiple testing) — compliance gates should use the single scenario percentile.
 
+## How the estimate is computed
+
+The statistical question: for a candidate bound `B`, can we claim "the true p-percentile is `<= B`"
+at confidence `c`? Each sample is a Bernoulli trial — over or under `B`. If the true p-percentile
+actually exceeded `B`, samples would land over `B` at a rate above `1 - p`, so observing few
+over-latency samples in a long run is evidence for the bound. LoadGen accepts `B` when
+
+```
+P( <= t over-latency among n samples | over-latency rate = 1 - p )  <  1 - c
+```
+
+— a false certification slips through with probability below `1 - c = 1%`. That binomial tail has a
+closed form as a regularized incomplete beta, `I_p(n - t, t + 1)`, evaluated with the Numerical
+Recipes `betai` continued fraction (no scipy; numerically equal to LoadGen's Gauss-hypergeometric
+form but converging in tens of iterations — see the docstrings in `metrics/early_stopping.py`).
+
+The reported estimate takes `B` to be an actual sample — the t-th highest — with `t` pushed as low
+into the tail as the test allows. Both searches exploit monotonicity (exponential bracketing +
+binary search), so the whole computation is `O(log² n)` beta evaluations on the sorted array:
+
+```
+odds(h, t)          = P(<= t over-latency among h + t trials | rate 1 - p)   # = I_p(h, t + 1)
+find_min_passing(t) = min h with odds(h, t) < 1 - c    # under-latency samples needed to absorb t
+floor               = find_min_passing(1) + 1          # smallest certifiable n (see cheat sheet)
+
+estimate(sorted, p):
+    if n < floor:  return None                         # too few samples for any claim
+    t = largest t >= 1 with find_min_passing(t) + t <= n   # the run's over-latency budget
+    return sorted[n - t]                               # t-th highest sample; t - 1 sit above it
+```
+
+Worked example (n = 10,000, p99): the floor is 662, the budget resolves to `t = 77`, and the
+estimate is the 77th-highest sample — 76 samples sit strictly above it (one budget slot is spent on
+the estimate itself, which keeps the claim strict). At the floor the budget is `t = 1` and the
+estimate is the maximum observed sample — honest but maximally conservative; as `n` grows,
+`t/n -> 1 - p` and the estimate converges onto the empirical percentile from above.
+
+## Cheat sheet: minimum samples per percentile
+
+The floor is `find_min_passing(1, p) + 1` at confidence 0.99 — the smallest run that can
+certify percentile p at all. Below it the map reports `null` for that percentile (the run
+"does not meet the standard" for that gate); at or above it the estimate is a valid
+c = 0.99 upper confidence bound.
+
+| percentile | minimum samples |
+| ---------- | --------------- |
+| p50        | 11              |
+| p75        | 24              |
+| p80        | 31              |
+| p90        | 64              |
+| p95        | 130             |
+| p97        | 219             |
+| p99        | 662             |
+| p99.9      | 6,636           |
+| p99.99     | 66,381          |
+
+Rule of thumb: floor ≈ 6.64 / (1 − p/100) — one more "9" costs 10× the samples. Contrast
+with the fixed-sample regime the feature replaces (~270k queries for Server p99).
+
 ## Layering (who owns what)
 
 ```

scripts/early_stopping_estimate_from_events.py

@@ -36,7 +36,7 @@
                                          # early_stopping_percentiles maps — exactly the
                                          # shape an ES-enabled run would have produced
       [--tokenizer <hf-model-dir-or-tokenizer.json>]   # enables TPOT
-      [--percentiles 0.5,0.9,0.95,0.99] [--confidence 0.99]  # stdout-analysis overrides
+      [--percentiles 50,90,95,99.9] [--confidence 0.99]      # stdout-analysis overrides
 """
 
 from __future__ import annotations
@@ -65,7 +65,6 @@
     CONFIDENCE,
     es_percentile_estimate,
     es_targets_from_grid,
-    grid_percentile_key,
 )
 from inference_endpoint.metrics.report import (
     SERIES_TO_SUMMARY_FIELD,
@@ -257,10 +256,11 @@ def main(argv=None):
     )
     ap.add_argument(
         "--percentiles",
-        default=",".join(
-            str(f) for f in es_targets_from_grid(DEFAULT_PERCENTILES).values()
+        default=",".join(es_targets_from_grid(DEFAULT_PERCENTILES)),
+        help=(
+            "stdout-analysis override, grid convention (0-100, e.g. 50,90,99.9); "
+            "--json always uses the summary grid"
         ),
-        help="stdout-analysis override (fractions); --json always uses the summary grid",
     )
     ap.add_argument(
         "--confidence",
@@ -274,11 +274,22 @@ def main(argv=None):
         raise SystemExit(
             "FATAL: --json requires --summary (the augmented output IS the summary)"
         )
-    fallback_fractions = [float(x) for x in args.percentiles.split(",")]
-    if not all(0.0 < f < 1.0 for f in fallback_fractions):
+    values = {float(tok) for tok in args.percentiles.split(",")}
+    if not all(0.0 < v < 100.0 for v in values):
+        raise SystemExit(
+            f"FATAL: percentiles use the grid convention and must be in (0, 100), "
+            f"got {sorted(values)}"
+        )
+    if any(v < 1.0 for v in values):
+        # a fraction-style value (pre-#423 convention) silently means a sub-1%
+        # percentile here — never a legitimate tail-certification target
+        bad = sorted(v for v in values if v < 1.0)
         raise SystemExit(
-            f"FATAL: percentiles must be in (0, 1), got {fallback_fractions}"
+            f"FATAL: percentiles now use the grid convention (0-100); {bad} would "
+            f"mean sub-1% percentiles — did you mean {[v * 100 for v in bad]}?"
         )
+    # canonical float-style keys, deduped on the parsed value ("99" == "99.0")
+    fallback = {str(v): v for v in sorted(values, reverse=True)}
     if not 0.0 < args.confidence < 1.0:
         raise SystemExit(f"FATAL: confidence must be in (0, 1), got {args.confidence}")
 
@@ -313,10 +324,7 @@ def main(argv=None):
             # the run's own grid: exact key strings, exact (descending) order
             targets = es_targets_from_grid(grid.keys())
         else:
-            targets = {
-                grid_percentile_key(f): f
-                for f in sorted(fallback_fractions, reverse=True)
-            }
+            targets = fallback
         results = {
             key: es_percentile_estimate(values, f, args.confidence)
             for key, f in targets.items()
@@ -332,11 +340,10 @@ def main(argv=None):
                 f"empirical={_fmt(r.empirical, unit, div)}  "
                 f"estimate={_fmt(r.estimate, unit, div)}"
             )
-            if r.estimate is not None and r.empirical:
+            if r.estimate is not None and r.empirical is not None:
                 gap = r.estimate - r.empirical
-                line += (
-                    f"  gap=+{_fmt(gap, unit, div)} (+{100 * gap / r.empirical:.2f}%)"
-                )
+                pct = f" (+{100 * gap / r.empirical:.2f}%)" if r.empirical else ""
+                line += f"  gap=+{_fmt(gap, unit, div)}{pct}"
             print(line)
             rows.append((field, key, r, unit, div))
         if summary is not None:
@@ -357,9 +364,10 @@ def main(argv=None):
     print("| metric | p | n | empirical | ES-adjusted | gap |")
     print("|---|---|---|---|---|---|")
     for field, key, r, unit, div in rows:
-        if r.estimate is not None and r.empirical:
+        if r.estimate is not None and r.empirical is not None:
             gap = r.estimate - r.empirical
-            g = f"+{_fmt(gap, unit, div)} (+{100 * gap / r.empirical:.2f}%)"
+            pct = f" (+{100 * gap / r.empirical:.2f}%)" if r.empirical else ""
+            g = f"+{_fmt(gap, unit, div)}{pct}"
         else:
             g = "-"
         print(

src/inference_endpoint/async_utils/services/metrics_aggregator/registry.py

@@ -45,7 +45,6 @@
     EarlyStoppingSpec,
     es_percentile_estimate,
     es_targets_from_grid,
-    grid_percentile_key,
 )
 
 from .snapshot import (
@@ -242,11 +241,22 @@ def _es_estimates(self, sorted_values) -> dict[str, float | None] | None:
         if self._es_spec is None:
             return None
         spec = self._es_spec
+        try:
+            return self._compute_es_estimates(sorted_values, spec)
+        except Exception:
+            # Best-effort by design: this runs inside publish_final's
+            # build_snapshot, where an exception would cost the ENTIRE final
+            # report (the publisher's finalized guard makes retries no-ops).
+            logger.exception(
+                "%s: early-stopping computation failed; omitting the map", self.name
+            )
+            return None
+
+    def _compute_es_estimates(
+        self, sorted_values, spec: EarlyStoppingSpec
+    ) -> dict[str, float | None]:
         if spec.percentiles is not None:  # explicit override: tests / offline analysis
-            targets = {
-                grid_percentile_key(f): f
-                for f in sorted(spec.percentiles, reverse=True)
-            }
+            targets = {str(v): float(v) for v in sorted(spec.percentiles, reverse=True)}
         else:
             targets = es_targets_from_grid(self._percentiles)
         results = {

src/inference_endpoint/async_utils/services/metrics_aggregator/snapshot.py

@@ -120,7 +120,7 @@ class SeriesStat(
     percentiles: dict[str, float]
     histogram: list[tuple[tuple[float, float], int]]
     # Early-stopping percentile estimates (COMPLETE snapshots only, when enabled):
-    # compact {grid_percentile_key: estimate-or-None} map whose keys mirror
+    # compact {grid key (str of the grid entry): estimate-or-None} map mirroring
     # ``percentiles``; None value = insufficient samples for that percentile.
     # Optional trailing field so the array_like wire format stays
     # backward-compatible. None field = not computed.