fix: return partial score data when a metrics upstream is down#729
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
There was a problem hiding this comment.
Pull request overview
This PR makes the per-orchestrator score API resilient to partial upstream outages by returning partial data instead of failing the entire request, and updates the UI to clearly distinguish “unavailable due to upstream outage” from “no data”.
Changes:
- Fetch AI score, transcoding metrics, and pricing independently and return
nullper-field when that upstream can’t be reached (502 only if all are down). - Adjust caching so degraded/partial responses use a short TTL (
s-maxage=60) while healthy responses keep the longer TTL. - Update orchestrator UI rendering to show “Unavailable” for
nullfields and keep “N/A” for legitimate empty/zero data.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| pages/api/score/[address].tsx | Adds a non-throwing JSON fetch helper and returns partial PerformanceMetrics with per-upstream null fallbacks + degraded caching. |
| lib/api/types/get-performance.ts | Updates PerformanceMetrics types to allow null per field and documents the semantic meaning of null vs empty/zero. |
| components/OrchestratingView/index.tsx | Adjusts UI access patterns and output strings to render “Unavailable” for unreachable upstream data. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
One unreachable upstream failed the whole request with a 502. Fetch each independently and fall back to null for that field, failing only when all are down. Partial responses cache for a minute so recovery is not hidden, and the UI shows "Unavailable" rather than "N/A" so our outage is not read as the orchestrator having no score. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
7069442 to
f8b58dd
Compare
|
Warning Review limit reached
Next review available in: 54 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughThe score API now tolerates failed upstream requests, propagates nullable performance data, adjusts caching for degraded responses, and renders unavailable metrics distinctly from missing or empty values. ChangesScore resilience
Estimated code review effort: 3 (Moderate) | ~20 minutes Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant ScoreAPI as GET /api/score/[address]
participant FetchJson as fetchJson
participant Upstreams as Upstream score services
participant MetricsBuilder as createMetricsObject
participant OrchestratingView
ScoreAPI->>FetchJson: Fetch score, metrics, and pricing data
FetchJson->>Upstreams: Retry upstream JSON requests
Upstreams-->>FetchJson: JSON response or failure
FetchJson-->>ScoreAPI: Nullable upstream results
ScoreAPI->>MetricsBuilder: Build regional metrics or null
MetricsBuilder-->>ScoreAPI: RegionalValues or null
ScoreAPI-->>OrchestratingView: PerformanceMetrics response
OrchestratingView-->>OrchestratingView: Render unavailable or missing states
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Problem
The AI metrics server (
leaderboard-api.livepeer.cloud) currently returns 404 for every route —/,/healthz, and/api/docsall return the bare Go404 page not found, so it is a router serving nothing rather than a URL change on our side.Because
/api/score/[address]fetched all three upstreams and failed on any non-OK response, this meant every per-orchestrator score request returned a 502, even though the transcoding metrics and pricing upstreams were healthy. One optional upstream took down the whole orchestrator page.Change
Each upstream is now fetched independently and falls back to
nullfor its own field. Anullmeans "we could not reach this upstream"; an empty or zeroed value keeps its existing meaning of "the upstream answered and has no data". Those stay distinguishable because the upstreams return{}with a 200 for an unknown orchestrator, so no extra bookkeeping field is needed.Cache-Controlheader moved below the address check because the TTL now depends on the fetch results.pricePerPixelisnullrather than0when the pricing server is unreachable, so an outage cannot advertise an orchestrator as free. A0from a healthy server still means genuinely unpriced.Also fixes network errors and timeouts skipping the per-upstream error logging entirely — those threw out of
Promise.alland surfaced as a generic 500 with no indication of which upstream failed.Verification
Against the live outage:
200 instead of the previous 502,
topAIScore: null, price and transcoding scores intact, and the one-minute TTL.tsc, ESLint, and Prettier all pass.Not verified: the healthy path (
s-maxage≈3600) and the all-upstreams-down 502, since the AI server cannot currently be brought back and the other two cannot be taken down locally.Notes for reviewers
Cache-Controlheader below the address check. Previously they were cached for an hour. Easy to restore if unwanted.pages/api/score/index.tsx:86-89has a pre-existing bug wherepricePerPixelis assigned the whole.find()result instead of.PricePerPixel. Left untouched for a separate PR — it sits on the field this PR makes nullable, so it is adjacent but unrelated.pages/api/regions/index.tsduplicates this fetch-or-null pattern without the try/catch or logging. HoistingfetchJsonintolib/would let it drop the copy, but that is out of scope here.Summary by CodeRabbit