Skip to content

fix: return partial score data when a metrics upstream is down#729

Merged
rickstaa merged 1 commit into
mainfrom
fix/score-endpoint-partial-metrics
Jul 16, 2026
Merged

fix: return partial score data when a metrics upstream is down#729
rickstaa merged 1 commit into
mainfrom
fix/score-endpoint-partial-metrics

Conversation

@rickstaa

@rickstaa rickstaa commented Jul 16, 2026

Copy link
Copy Markdown
Member

Problem

The AI metrics server (leaderboard-api.livepeer.cloud) currently returns 404 for every route — /, /healthz, and /api/docs all return the bare Go 404 page not found, so it is a router serving nothing rather than a URL change on our side.

Because /api/score/[address] fetched all three upstreams and failed on any non-OK response, this meant every per-orchestrator score request returned a 502, even though the transcoding metrics and pricing upstreams were healthy. One optional upstream took down the whole orchestrator page.

Change

Each upstream is now fetched independently and falls back to null for its own field. A null means "we could not reach this upstream"; an empty or zeroed value keeps its existing meaning of "the upstream answered and has no data". Those stay distinguishable because the upstreams return {} with a 200 for an unknown orchestrator, so no extra bookkeeping field is needed.

  • 502 only when every upstream is down, since that is never a legitimate empty state.
  • Partial responses cache for a minute instead of an hour, so upstream recovery is not hidden. The Cache-Control header moved below the address check because the TTL now depends on the fetch results.
  • The UI shows "Unavailable" rather than "N/A" for unreachable fields. An orchestrator with a good AI score rendering "N/A" during our outage reads as "this orchestrator does not do AI work" — a claim about them caused by a fault on our side.
  • pricePerPixel is null rather than 0 when the pricing server is unreachable, so an outage cannot advertise an orchestrator as free. A 0 from a healthy server still means genuinely unpriced.

Also fixes network errors and timeouts skipping the per-upstream error logging entirely — those threw out of Promise.all and surfaced as a generic 500 with no indication of which upstream failed.

Verification

Against the live outage:

$ curl -i localhost:3000/api/score/0x847791cbf03be716a7fe9dc8c9affe17bd49ae5e
HTTP/1.1 200 OK
Cache-Control: public, s-maxage=60, stale-while-revalidate=60

{"pricePerPixel":245.796,"successRates":{...,"GLOBAL":99.26},"scores":{...},"topAIScore":null}

200 instead of the previous 502, topAIScore: null, price and transcoding scores intact, and the one-minute TTL. tsc, ESLint, and Prettier all pass.

Not verified: the healthy path (s-maxage≈3600) and the all-upstreams-down 502, since the AI server cannot currently be brought back and the other two cannot be taken down locally.

Notes for reviewers

  • Invalid-address 400s are no longer CDN-cached, a side effect of moving the Cache-Control header below the address check. Previously they were cached for an hour. Easy to restore if unwanted.
  • pages/api/score/index.tsx:86-89 has a pre-existing bug where pricePerPixel is assigned the whole .find() result instead of .PricePerPixel. Left untouched for a separate PR — it sits on the field this PR makes nullable, so it is adjacent but unrelated.
  • pages/api/regions/index.ts duplicates this fetch-or-null pattern without the try/catch or logging. Hoisting fetchJson into lib/ would let it drop the copy, but that is out of scope here.

Summary by CodeRabbit

  • Bug Fixes
    • Performance metrics now clearly distinguish unavailable data from missing or zero-valued results.
    • Regional AI and transcoding scores, price-per-pixel values, and related statistics display “Unavailable” when source data cannot be provided, instead of misleading defaults.
    • Score retrieval now tolerates partial upstream failures and continues displaying available information.
    • Degraded responses use shorter-lived caching so refreshed data becomes available sooner.

@rickstaa
rickstaa requested a review from ECWireless as a code owner July 16, 2026 08:53
Copilot AI review requested due to automatic review settings July 16, 2026 08:53
@vercel

vercel Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
explorer-arbitrum-one Ready Ready Preview, Comment Jul 16, 2026 9:00am

Request Review

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR makes the per-orchestrator score API resilient to partial upstream outages by returning partial data instead of failing the entire request, and updates the UI to clearly distinguish “unavailable due to upstream outage” from “no data”.

Changes:

  • Fetch AI score, transcoding metrics, and pricing independently and return null per-field when that upstream can’t be reached (502 only if all are down).
  • Adjust caching so degraded/partial responses use a short TTL (s-maxage=60) while healthy responses keep the longer TTL.
  • Update orchestrator UI rendering to show “Unavailable” for null fields and keep “N/A” for legitimate empty/zero data.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
pages/api/score/[address].tsx Adds a non-throwing JSON fetch helper and returns partial PerformanceMetrics with per-upstream null fallbacks + degraded caching.
lib/api/types/get-performance.ts Updates PerformanceMetrics types to allow null per field and documents the semantic meaning of null vs empty/zero.
components/OrchestratingView/index.tsx Adjusts UI access patterns and output strings to render “Unavailable” for unreachable upstream data.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

One unreachable upstream failed the whole request with a 502. Fetch each
independently and fall back to null for that field, failing only when all are
down. Partial responses cache for a minute so recovery is not hidden, and the UI
shows "Unavailable" rather than "N/A" so our outage is not read as the
orchestrator having no score.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@rickstaa
rickstaa force-pushed the fix/score-endpoint-partial-metrics branch from 7069442 to f8b58dd Compare July 16, 2026 08:58
@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@rickstaa, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 54 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro

Run ID: 54d146dc-396a-41c4-80cc-a04c6ac207e2

📥 Commits

Reviewing files that changed from the base of the PR and between 7069442 and f8b58dd.

📒 Files selected for processing (3)
  • components/OrchestratingView/index.tsx
  • lib/api/types/get-performance.ts
  • pages/api/score/[address].tsx
📝 Walkthrough

Walkthrough

The score API now tolerates failed upstream requests, propagates nullable performance data, adjusts caching for degraded responses, and renders unavailable metrics distinctly from missing or empty values.

Changes

Score resilience

Layer / File(s) Summary
Nullable fetch contract
lib/api/types/get-performance.ts, pages/api/score/[address].tsx
Performance fields accept null, while fetchJson retries requests, logs failures, and returns nullable results.
Score API orchestration
pages/api/score/[address].tsx
The API fetches upstream data concurrently, marks degraded responses, shortens degraded cache duration, and propagates missing regional and pricing data as null.
Unavailable score rendering
components/OrchestratingView/index.tsx
Regional scores, AI scores, and price-per-pixel values render "Unavailable" when upstream data is explicitly null and retain "N/A" for missing data.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested reviewers: ecwireless

Sequence Diagram(s)

sequenceDiagram
  participant ScoreAPI as GET /api/score/[address]
  participant FetchJson as fetchJson
  participant Upstreams as Upstream score services
  participant MetricsBuilder as createMetricsObject
  participant OrchestratingView
  ScoreAPI->>FetchJson: Fetch score, metrics, and pricing data
  FetchJson->>Upstreams: Retry upstream JSON requests
  Upstreams-->>FetchJson: JSON response or failure
  FetchJson-->>ScoreAPI: Nullable upstream results
  ScoreAPI->>MetricsBuilder: Build regional metrics or null
  MetricsBuilder-->>ScoreAPI: RegionalValues or null
  ScoreAPI-->>OrchestratingView: PerformanceMetrics response
  OrchestratingView-->>OrchestratingView: Render unavailable or missing states
Loading
🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description is detailed, but it does not follow the required template and omits sections like Type of Change, Related Issues, Changes Made, and Impact / Risk. Reformat the PR body to match the template and add the missing sections, especially Type of Change, Related Issue(s), Changes Made, Testing, and Impact / Risk.
✅ Passed checks (4 passed)
Check name Status Explanation
Title check ✅ Passed The title is concise and accurately summarizes the main change: returning partial score data when an upstream is down.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/score-endpoint-partial-metrics

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@rickstaa
rickstaa merged commit 019650c into main Jul 16, 2026
9 checks passed
@rickstaa
rickstaa deleted the fix/score-endpoint-partial-metrics branch July 16, 2026 09:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants