Skip to content

🐛 Fixed post card assets going missing until restart (in-memory serving)#29128

Draft
sagzy wants to merge 1 commit into
mainfrom
fix-card-assets-in-memory
Draft

🐛 Fixed post card assets going missing until restart (in-memory serving)#29128
sagzy wants to merge 1 commit into
mainfrom
fix-card-assets-in-memory

Conversation

@sagzy

@sagzy sagzy commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

ref https://linear.app/ghost/issue/ONC-1865

Note: This is one of two alternative fixes for the same incident, opened for comparative review. The other is the minimal disk-recovery approach: #29129. Both supersede #29056, whose review surfaced races in the recovery-on-ENOENT design (torn reads from non-atomic writes, response-cache poisoning, throttle 404ing concurrent requests, rebuild loops for never-produced files).

Problem

/public/cards.min.css and /public/cards.min.js are minified at runtime into content/public/ and served by reading them back from disk. If the built files go missing at runtime, or the content folder is unwritable (EACCES), every request 404s until the process restarts — ~486 production sites accumulated in this state during Jun 25–29.

Solution: serve card assets from memory

The disk round-trip buys nothing: the assets are rebuilt from theme sources on every boot anyway, and the serving middleware already held the response body in memory permanently. The disk copy was never a real cache — it was just the failure surface.

  • CardAssets.load() now builds in memory (Minifier.minifyInMemory()) and stores the outputs on the service; getContent(filename) is the source of truth for serving. A successful build is always servable — ENOENT/EACCES on the destination can no longer take the assets offline. The disk write still happens as best-effort back-compat, with failures logged and ignored.
  • The two card routes use a new createInMemoryAssetMiddleware() that serves straight from the service with the same header shape as before (md5 ETag, Content-Length, Cache-Control), memoized per build.
  • Build policy now lives in one place, AssetsMinificationBase.ensureLoaded(): concurrent callers join a single in-flight build; a 10s retry backoff stops a persistently failing build from minifying per request; invalidate() (theme activation) no longer forgets an in-flight build — a generation counter re-runs load() with the new config instead of allowing two concurrent minifier runs.
  • serveMiddleware() no longer lets a rejected load() escape as an unhandled rejection (which left requests hanging under Express 4) — this also fixes the hang for AdminAuthAssets.

Testing

  • New unit tests for ensureLoaded() (join-in-flight, backoff + expiry, invalidate resets backoff, mid-build invalidate re-runs load with no concurrency), CardAssets in-memory outputs (getContent, hasFile, ready + servable even when the disk write EACCESes), and the in-memory middleware (headers/ETag, {{blog-url}} replacement, 404 when the build legitimately produces no file, build trigger when not ready).
  • ghost/core unit suite: 7187 passed; the 3 failures are pre-existing on main (verified by baseline run).

🤖 Generated with Claude Code

ref https://linear.app/ghost/issue/ONC-1865

- cards.min.css/js were built into content/public and served by reading
  them back from disk, so a wiped or unwritable content folder meant a
  404 on every request until reboot
- the assets are rebuilt from theme sources on every boot anyway, so the
  disk copy is not a real cache; the serving middleware already held the
  body in memory permanently
- card assets are now kept in memory by the assets service and served
  from there; the disk write is best-effort back-compat only, so write
  failures and deleted files can no longer take the assets offline
- build policy (single-flight, retry backoff, rebuild-on-theme-change)
  now lives in one place on the service instead of per-route

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@coderabbitai

coderabbitai Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 5d8306db-a2ac-46e7-9511-064746775927

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix-card-assets-in-memory

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@nx-cloud

nx-cloud Bot commented Jul 6, 2026

Copy link
Copy Markdown

🤖 Nx Cloud AI Fix

Ensure the fix-ci command is configured to always run in your CI pipeline to get automatic fixes in future runs. For more information, please see https://nx.dev/ci/features/self-healing-ci


View your CI Pipeline Execution ↗ for commit b559938

Command Status Duration Result
nx run ghost:test:ci:integration ✅ Succeeded 2m 40s View ↗
nx run ghost:test:integration ✅ Succeeded 2m 54s View ↗
nx run ghost:test:legacy ✅ Succeeded 2m 51s View ↗
nx run ghost:test:e2e ✅ Succeeded 2m 26s View ↗
nx run-many --target=build --projects=tag:publi... ✅ Succeeded <1s View ↗
nx run-many -t test:unit -p ghost ✅ Succeeded 29s View ↗
nx run-many -t lint -p ghost ✅ Succeeded 33s View ↗
nx run @tryghost/admin:build ✅ Succeeded 9s View ↗
Additional runs (2) ✅ Succeeded ... View ↗

💡 Verify your cache is correct by running tasks in a sandbox. Read docs ↗


☁️ Nx Cloud last updated this comment at 2026-07-06 09:49:10 UTC

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant