Replace Swagger UI with Scalar, add a consumer-first API reference by mo4islona · Pull Request #114 · subsquid/sqd-portal

mo4islona · 2026-05-28T12:23:17Z

Summary

Swap Swagger UI for Scalar at /docs (via utoipa-scalar + a custom HTML/CSS template), serving an OpenAPI spec built with utoipa v5 (preserve_path_order, so the sidebar order matches paths(...)).
Use info.description to ship a consumer-first Introduction: where you landed → finalized vs. real-time data model → blockchain forks / parentBlockHash → first query. Operator material (architecture internals, self-hosting topologies) is intentionally kept out of the API reference.
Restructure the public surface: Datasets and Streaming are the only tag groups shown by default; Monitoring and Debug are marked internal (x-internal) and hidden unless SHOW_INTERNAL_DOCS=true.

ℹ️ Screenshot predates the consumer-first restructuring (sidebar sections have since changed); the renderer/theme is unchanged.

What changed

Renderer

utoipa-scalar 0.2 mounted at /docs with Scalar::custom_html(...); template in docs/openapi/scalar_template.html (theme fastify; hiddenClients, hideModels, defaultOpenAllTags; developer tools disabled).
utoipa-swagger-ui removed; /api-docs/openapi.json served by a small serve_openapi_spec handler (the spec is injected as an axum::Extension).
Template CSS tunes the Introduction reading column and forces a monospace font with full box-drawing coverage (for the ASCII diagrams).

API reference Introduction (`docs/openapi/`)

Consumer-first, three sections concatenated into info.description via include_str! in src/openapi.rs:

01-introduction.md — orientation (a portal is the HTTP gateway you query) + the finalized vs. real-time data model, with a simplified merge diagram.
02-blockchain-forks.md — the parentBlockHash protocol, what a 409 means, a reorg diagram, the conflict response body, and the recommended client recovery loop.
03-getting-started.md — a minimal first /stream query (+ link to the full SQD query syntax) and a one-line pointer to the README for self-hosting.

Rendered order: orientation → data model → forks → first query → self-host pointer. The 409 response on /stream links into the forks section via #description/how-parentblockhash-works.

Operator docs (component table, deployment topologies) were dropped from the reference — they remain in git history, and the README stays the self-hosting entry point.

`/stream` 409 response

Description matches the portal-api RFC (parentHash mismatch, previousBlocks guarantee, walk-back procedure).
Documented response body: a ConflictResponse schema (previousBlocks: [{ number, hash }]) with an example, so the rendered docs show the body instead of "No Body".

Internal-endpoint filtering

Internal #[utoipa::path]s carry the standard OpenAPI x-internal: true extension (replacing the old [INTERNAL] doc-comment marker).
build_openapi_spec(show_internal) drops marked operations and prunes now-empty tag groups, while preserving doc-only tags.
Toggle with SHOW_INTERNAL_DOCS=true / --show-internal-docs. Two unit tests in src/openapi.rs assert the filter behaviour on the real ApiDoc.

Other

graceful_shutdown.md moved under docs/decisions/.

Test plan

cargo build
cargo test --lib openapi:: — internal-filter tests pass (hide_internal_drops_marked_ops_and_empty_tags, show_internal_keeps_all_ops)
cargo run --bin gen_openapi produces the full unfiltered spec (with x-internal extensions visible)
Start portal, open /docs:
- Introduction reads consumer-first (orientation → data model → forks → first query)
- Datasets and Streaming tag groups present; Monitoring and Debug hidden
- /stream 409 shows the previousBlocks body schema + example (not "No Body")
- /api-docs/openapi.json returns the same filtered spec
Start portal with SHOW_INTERNAL_DOCS=true, confirm Monitoring / Debug groups appear

🤖 Generated with Claude Code

define-null

I did a quick look and left some suggestions for improvements.

define-null · 2026-05-28T15:54:28Z

+
+### Data sources
+
+There are two: **archival** (SQD Network) and **real-time** (your RPC, via HotblocksDB).


A bit confused by the naming "Your RPC". Is that the right context for the customer? Your RPC provider maybe?

This data-sources section was operator-facing and has been removed — the API-reference intro is now consumer-only and no longer mentions RPC providers or staking. That context now belongs with the self-hosting docs.

define-null · 2026-05-28T15:57:20Z

+
+Serves this API from SQD Network and, optionally, routes requests for the latest blocks to HotblocksDB.
+
+Downloads an assignment file at startup and periodically re-downloads it. Needs an Arbitrum RPC endpoint (and an Ethereum L1 RPC) to read on-chain state.


Is the assignment file information relevant here for the API ? Looks like an implementation detail for me, which the customer does not directly control.

Agreed — that's an implementation detail the consumer doesn't control. It's no longer in the reference intro (the operator/running content was removed).

define-null · 2026-05-28T16:01:53Z

+- If `parentBlockHash` matches the parent the portal sees for `fromBlock` → the stream starts normally.
+- If it does **not** match → a fork happened between the client's last seen block and `fromBlock`. The portal returns `409 Conflict`.
+
+### What 409 means


Instead of describing what 409 mean, I would reframe it to describe what the conflict is. That way we do not focus just on a single status code, but frame it on a more high-level.

In other words I would rename this title and other titles something like: "What a conflict is?" or "What a conflict means for the data consistency", or other appropriate formulation.

Done — there's now a What a conflict means section that frames it as the client's chain having diverged from the canonical chain, rather than leading with the status code; the 409 is mentioned only as the mechanism that surfaces it.

define-null · 2026-05-28T16:03:32Z

+
+If the client tries to resume at `fromBlock = N+2` with `parentBlockHash = hash(N+1)` (orphaned), the portal sees that `N+1` is no longer the parent of `N+2'` on the canonical chain → it answers 409.
+
+### 409 response body


Instead of duplicating documentation here, I would just reference to the 409 reponse body for the particular API. Otherwise those two may diverge.

Good call on the divergence risk. The /stream 409 now has a documented response body in this PR (a ConflictResponse schema with previousBlocks), so the prose "Conflict response body" section can reference that instead of duplicating the shape — worth trimming so the two can't drift.

define-null · 2026-05-28T16:04:09Z

+
+### Recommended client behaviour
+
+When you receive a 409:


Similarly - when the conflict occurs, or when the reorg occures. In terminology of the domain language, not implementation.

The forks section now leads with the concept ("What a conflict means") and uses domain terms — conflict / reorg — consistently. Let me know if a particular spot still reads implementation-first.

define-null · 2026-05-28T16:05:42Z

+Always pass `parentBlockHash` when resuming a stream. A client that does **not** pass it will silently process blocks from a forked chain after a reorg, with no signal that anything is wrong.
+
+
+## Self-hosting


I would suggest to move it out into a separate doc.

Done — the single intro was split into docs/openapi/{01-introduction,02-blockchain-forks,03-getting-started}.md, and the architecture + self-hosting sections were moved out of the API reference entirely (the README is the self-hosting entry point). Hope that covers what you had in mind here.

define-null · 2026-05-28T16:06:24Z

+
+### Real-time only
+
+Stream the chain tip from your own RPC. No SQD tokens needed. Best for fresh-data use cases on small chains.


I don't know what is meant by a chain tip. Maybe clarify that? Or is it a common term?

The rewrite introduces it as "the most recent blocks … the tip of the chain", so it's glossed on first use. Happy to add an explicit one-line definition of "chain tip" if it still reads as jargon.

define-null · 2026-05-28T16:07:58Z

+**1. Hotblocks service** (per RPC), **2. HotblocksDB** with `Api` retention, **3. Hotblocks-retain** to coordinate retention with the network, **4.** portal key, **5.** portal config:
+
+```yaml
+# mainnet.config.yml
+hostname: http://0.0.0.0:8080
+hotblocksDB: http://hotblocks-db:8081
+sqd_network:
+  datasets: https://cdn.subsquid.io/sqd-network/datasets.yml
+  metadata: https://cdn.subsquid.io/sqd-network/mainnet/metadata.yml
+  serve: "manual"
+datasets:
+  ethereum-mainnet:
+    real_time:
+      kind: evm
+```
+
+**6.** Save `mainnet.env` as `.env`. **7.** Run the portal (same command as in **Archival only**). **8.** Verify: `curl localhost:8080/datasets/ethereum-mainnet` reports `real_time: true` and `start_block: 0`.


This renders poorly, we need to have each ** on a separate line.

I believe this was in the self-hosting "Suitable when" lists — that whole section has been removed from the reference, so the rendering issue should be moot now. Let me know if it was elsewhere.

define-null · 2026-05-28T16:09:23Z

+/// #[utoipa::path(
+///     ...,
+///     extensions(("x-internal" = json!(true))),
+/// )]


Unless I read the comment on the left of the diff it wasn't clear what we are talking about here. I suggest we rephrase to something like: "In order to mark the api as internal.. "

Done — the marker is now documented in src/openapi.rs on INTERNAL_EXT_KEY, rephrased to: "To mark an API operation as internal — so it is pruned from the served OpenAPI spec unless show_internal is true — add this extension on its handler."

define-null · 2026-05-28T16:10:32Z

-const INTERNAL_MARKER: &str = "[INTERNAL]";
+const INTERNAL_EXT_KEY: &str = "x-internal";

 fn build_openapi_spec(show_internal: bool) -> utoipa::openapi::OpenApi {


I suggest we move this and some other openapi specific functions to a separate module.

Done — build_openapi_spec and serve_openapi_spec (along with the x-internal filtering and the ApiDoc definition) now live in src/openapi.rs.

Serve the OpenAPI reference with Scalar (utoipa v5 + utoipa-scalar) at /docs, replacing Swagger UI, and ship a consumer-first Introduction. - Renderer: Scalar at /docs via a custom HTML/CSS template (docs/openapi/scalar_template.html); /api-docs/openapi.json served by serve_openapi_spec; utoipa-swagger-ui removed. - Introduction (docs/openapi/, concatenated into info.description): 01-introduction (orientation + finalized vs. real-time data model), 02-blockchain-forks (parentBlockHash / 409 protocol), 03-getting-started (first query + README pointer). Architecture/self-hosting content is kept out of the reference. - Internal-endpoint filtering via the x-internal OpenAPI extension; build_openapi_spec(show_internal) drops internal ops and empty tags while preserving doc-only tags. Toggle with SHOW_INTERNAL_DOCS. Datasets/Streaming shown by default; Monitoring/Debug hidden. - /stream 409 documents a ConflictResponse body (previousBlocks). - Move graceful_shutdown.md under docs/decisions/. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Copilot

Pull request overview

Copilot reviewed 10 out of 12 changed files in this pull request and generated 1 comment.

+3. If `previousBlocks` does not contain any block you recognise (the divergence is deeper than the array provided), *
+   *re-request earlier blocks** — open a new stream that includes a `fromBlock` further in the past — and repeat the


Copilot AI review requested due to automatic review settings May 28, 2026 12:23

Copilot started reviewing on behalf of mo4islona May 28, 2026 12:23 View session

This comment was marked as low quality.

Sign in to view

mo4islona mentioned this pull request May 28, 2026

Add docs on how to self-host a portal #110

Merged

mo4islona requested a review from define-null May 28, 2026 12:46

define-null reviewed May 28, 2026

View reviewed changes

mo4islona changed the title ~~Replace Swagger UI with Scalar, add custom intro docs~~ Replace Swagger UI with Scalar, add a consumer-first API reference May 29, 2026

mo4islona force-pushed the feat/openapi branch from 85a99ff to 35ce97d Compare May 29, 2026 11:05

mo4islona requested review from Copilot and define-null May 29, 2026 11:58

Copilot started reviewing on behalf of mo4islona May 29, 2026 11:58 View session

Copilot AI reviewed May 29, 2026

View reviewed changes

define-null approved these changes Jun 2, 2026

View reviewed changes


		### Data sources

		There are two: archival (SQD Network) and real-time (your RPC, via HotblocksDB).


		Serves this API from SQD Network and, optionally, routes requests for the latest blocks to HotblocksDB.

		Downloads an assignment file at startup and periodically re-downloads it. Needs an Arbitrum RPC endpoint (and an Ethereum L1 RPC) to read on-chain state.


		If the client tries to resume at `fromBlock = N+2` with `parentBlockHash = hash(N+1)` (orphaned), the portal sees that `N+1` is no longer the parent of `N+2'` on the canonical chain → it answers 409.

		### 409 response body

		Always pass `parentBlockHash` when resuming a stream. A client that does not pass it will silently process blocks from a forked chain after a reorg, with no signal that anything is wrong.


		## Self-hosting


		### Real-time only

		Stream the chain tip from your own RPC. No SQD tokens needed. Best for fresh-data use cases on small chains.

		3. If `previousBlocks` does not contain any block you recognise (the divergence is deeper than the array provided), *
		re-request earlier blocks* — open a new stream that includes a `fromBlock` further in the past — and repeat the

Uh oh!

Conversation

mo4islona commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What changed

Renderer

API reference Introduction (docs/openapi/)

/stream 409 response

Internal-endpoint filtering

Other

Test plan

Uh oh!

This comment was marked as low quality.

Uh oh!

define-null left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mo4islona commented May 28, 2026 •

edited

Loading

API reference Introduction (`docs/openapi/`)

`/stream` 409 response