Conversation
- Add "Recommended" badge next to the metadata toggle - Add a help popover next to the Metadata section heading linking to the FAQ - Expand FAQ item-11 explaining metadata production, with links to the getting started guide and Psych-DS documentation Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add a "how it works" link in Getting Started step 4 that deep-links to FAQ item-11, and point the Psych-DS link at the Psych-DS docs - Make the FAQ accordion open and scroll to an item when arriving via its hash (e.g. /faq#item-11) Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…tooling
DataPipe's vendored @jspsych/metadata was a June-2024 fork frozen at v0.0.1
that silently discarded nested object/array trial data. The fix lives on the
upstream main branch but is NOT in the published npm 0.0.3 (which has the same
data-loss bug and would silently drop all data given DataPipe's pre-parsed
input). Rather than depend on the stale npm release or live-track a moving
branch, vendor a PINNED upstream commit and rebuild from it, with tooling to
make future re-syncs a one-command, reviewable step.
- functions/scripts/sync-metadata.mjs (+ npm run sync:metadata): clone upstream
at a ref, build packages/metadata, copy the built dist + sanitized
package.json + LICENSE into functions/metadata/, and record provenance in
VENDORED_FROM.json. Strips the package's scripts (upstream's
prepare:"npm run build" would break `npm install` of the file: dep, since we
ship dist-only) while keeping the csv-parse runtime dep.
- .github/workflows/metadata-drift-check.yml: weekly non-blocking job that
opens/updates a tracking issue when upstream main moves past the pinned commit.
- functions/metadata/: now dist-only, pinned to upstream main 224d336. dist is
committed (deploys need no metadata build); .gitignore updated to un-ignore it.
- functions/package.json: dep stays file:metadata; add explicit typescript
devDep (the build had relied on it transitively via the removed fork).
- functions/src/metadata-production.ts: generate()'s 3rd arg is now a string
ext ('json'|'csv'), not the old boolean csv flag.
- metadata-production.test.js: fixture updated to real output (type -> @type,
numeric -> number); data-derived levels/min-max double as a silent-drop guard;
fixed a pre-existing aliasing bug in the options test.
Verified: metadata-production, metadata-update, metadata-process suites pass;
functions build (tsc) and npm install are clean.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
functions/metadata is now a pre-built vendored dist with no lockfile or build script, so the "npm ci && npm run build" step in that directory fails with EUSAGE. The dist is committed and installed as a file: dependency by the existing functions npm ci step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Both suites used logs/testlog and each deletes it at test start; jest runs them in parallel workers, so the base64 suite's delete could wipe the data suite's saveData counter between write and read (doc exists via the base64 increment, saveData undefined). Rename the base64 suite's doc to base64-testlog, matching its other doc IDs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The entire metadata block (OSF list/download/upload calls included) ran inside db.runTransaction. Firestore retries the transaction callback on contention, which would re-run those OSF network calls — a latent duplicate-write risk. The transaction now only reads, merges, and writes the Firestore metadata doc; OSF I/O happens before (existence check) and after (mirror upload) the transaction. When the merge needs the OSF copy as its base (Firestore empty, OSF populated), the transaction aborts via a sentinel, the copy is downloaded outside it, and the transaction re-runs. Also fixes two pre-existing bugs in the process: - The Firestore-only branch checked putFileOSF's result with `errorCode !== 210`, which threw even on success (success returns errorCode null); now checks `!response.success`. - The OSF-only branch uploaded the unmerged incoming metadata to OSF while Firestore got the merged version; both now get the merged one. - The create branch's putFileOSF result was silently ignored; it is now checked like the other branches. blockMetadata now receives the OSF token api-data already resolved via resolveToken, instead of re-deriving it with duplicated logic that could trigger a second token refresh per submission. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The re-vendored @jspsych/metadata expands nested object/array trial fields (survey responses, mouse-tracking samples, ...) into dotted sub-variables and exposes the per-row data behind them. Until now DataPipe only used the variable descriptions, so the described data was not actually retrievable in tabular form. This mirrors what the standalone metadata CLI writes per data file: one sidecar CSV per extracted array column (rows keyed by the join keys + element_index) and per object column (one row per trial), named with the library's own Psych-DS helpers (deriveFallbackBase/deriveArrayFilename), placed in the same subfolder as the data file. Because the CLI's sidecars are per source file, per-submission sidecars are the exact incremental equivalent — no cross-session accumulation or dedup state is needed. Flow: produceMetadata() now returns the extraction results alongside the metadata; blockMetadata() builds the sidecar payloads and returns them; api-data uploads them only after the participant's data file itself lands in OSF (no orphan sidecars on 409), best-effort — a sidecar failure is queued in the existing uploadQueue (the retry worker already handles arbitrary filenames and treats 409 as done) and logged, never failing the submission. When the main file is queued because OSF is down, the sidecars are queued alongside it. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…surface mainRows Replace the hand-rolled sidecar builder in metadata-sidecars.ts with a call to the library's shared buildPsychDSDataFiles (the same function the @jspsych/metadata CLI and browser flows use), filtering to the sidecar files (kind 'array'/'object') and keeping today's placement next to the data file. This deletes the duplicated deriveArrayFilename/disambiguate/objectsToCSV logic; sidecar output is byte-identical (verified: arrays lead with [...joinKeys, 'element_index'], objects with joinKeys). produceMetadata now also returns mainRows (the parsed JSON trial array, or parseCSV records for CSV) so a later change can emit the main Psych-DS data CSV via the same builder. generate() leaves nested columns intact in these rows; the main CSV serialises them as JSON-in-cell (lossless), with the dotted expansion in the sidecars. The main CSV (kind 'main') is discarded for now — it is wired up when api-data.ts adopts the data/ + data/raw/ layout. tsc clean; 25/25 metadata unit tests pass. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
putFileOSF previously handled only a single subfolder level (split on '/' and used components[0]/components[1]), so a nested path like "data/raw/abc.json" would break. Generalise it to split the filename into folder segments plus the file name and walk each level in turn. subfolder.ts's parsePath is replaced by resolveFolder(parentUrl, token, name): it lists the children of any level — the storage root or a parent folder's WaterButler link — and returns the named child folder's link, creating it if absent. The link chains, so callers walk nested paths one level at a time. Folder creation treats a 409 Conflict as success: concurrent submissions can race to create the same folder (now every metadata submission needs data/ and data/raw/), and the loser re-lists and returns the winner's folder. putFileOSF also switches from node-fetch to the Node 22 global fetch, matching subfolder.ts and making the whole path unit-testable via a global.fetch mock. New put-file-osf.test.js covers depth 0/1/2, folder creation, the 409 race, and upload failures (10 tests). tsc clean. NOTE: the WaterButler move-link recursion (listing/creating inside a subfolder via its move link) and the node-fetch->global fetch body handling still need live verification against a real OSF component + token. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… CSVs under data/)
With metadata on, the participant's raw submission is now the critical
upload and lands byte-verbatim at data/raw/<original name>; session
counting and queue-on-failure key off it. The full derived set — main
data CSV (byte-verbatim for CSV submissions via mainContent), sidecar
CSVs for nested columns, and .psychds-ignore at the component root — is
built by the library's shared buildPsychDSDataFiles and uploaded
best-effort under data/ after the raw file lands. Researcher subfolders
are flattened, matching the CLI. Metadata off is unchanged.
Also:
- dataset_description.json's OSF mirror is now best-effort: create
failures queue for retry (409 = concurrent create, done), update
failures self-heal on the next submission — they no longer reject the
participant's data.
- produceMetadata routes JSON through the library's parseJsonData, so a
nonstandard { "trials": [...] } wrapper is unwrapped exactly as the
CLI does; bare arrays are untouched.
- metadata-sidecars.ts is deleted: metadata-derived-files.ts consumes
the full buildPsychDSDataFiles set directly, and the upload module is
renamed metadata-derived-upload.ts to match.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
fix(metadata): re-vendor @jspsych/metadata from upstream main + sync tooling
feat(metadata): Psych-DS-compliant data layout on OSF + Firestore-only metadata transaction
- Point the metadata help popover's "Learn more" link at the Psych-DS docs - Add hover tooltips to the experiment-list status labels (Data, Base64, Conditions, Metadata) describing each feature and its enabled state Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Surface and document metadata production
Recovered submissions for metadata-active experiments were re-enqueued under their original filename instead of data/raw/<name>, producing no derived files and never merging into dataset_description.json. Worse, the queue doc was keyed off the original filename while api-data.ts keys off the transformed one, so a crash between queueUpload and cleanupPending could upload the same file twice under two names. Extract the metadataActive ? rawDataPath(filename) : filename decision into uploadPathFor() and use it in both api-data.ts and scheduled-pending-recovery.ts's promoteToQueue, for both the queued filename and the dedup key, eliminating the double-upload window as a side effect. Metadata/derived files are not regenerated for recovered sessions (documented as a known limitation): recovery has no metadata pipeline and the raw file is the source of truth.
Two cheap hardening fixes, plus the D2 decision (option b): keep returning 400 on a metadata-block failure, but stop deleting the pending-data copy so scheduled-pending-recovery can salvage it later instead of losing the submission outright. Graceful-degrade (accepting the data anyway) is a product decision left for a follow-up. - metadata-production.ts: variableMeasured is now length-checked (`?.length`) so an empty array throws the intended clean error instead of a TypeError on `variableMeasured[0]`. - metadata-production.ts: parseJsonData's result is checked for Array.isArray so a bare JSON object throws a clear "Data must be an array of trials" instead of flowing into generate()/mainRows and failing deeper with a confusing message.
Two submissions with the same leaf name in different subfolders collided at data/raw/<leaf> (the second got a 409 -> 400 OSF_FILE_EXISTS rejection): flattenName discarded the subfolder prefix instead of encoding it. flattenName now encodes path separators as `-` (condition-A/data.json -> condition-A-data.json) instead of dropping everything before the last `/`. The derived main CSV/sidecar stem follows the same encoded name automatically, so those stay collision-free too. Flat data/raw/ still matches the CLI's layout (the alternative — nesting subfolders under data/raw/ — was considered and rejected: it would diverge from the CLI and still need the encoded stem for derived files).
…iption updates Two related defects, one root cause: the uploadQueue was designed for immutable per-submission files, but dataset_description.json is mutable. - metadata-block.ts: when updateFileOSF throws, the catch no longer queues a create-PUT — it was guaranteed to 409 against the existing file, and the retry worker would mark that dead entry completed without ever applying the update. Firestore is the source of truth and every submission re-merges and re-mirrors, so the next submission repairs OSF instead (matches the code's own existing comment). - queue-upload.ts: while an entry is "pending", a newer submission with fresher content used to return early without ever queueing it, so the eventual retry pushed stale metadata. Now a "pending" re-queue overwrites the Cloud Storage payload (keeping the Firestore doc/status/retry schedule), so the retry uploads the freshest content. "processing" entries are left alone since the retry worker owns that payload right now.
uploadDerivedFiles previously uploaded N derived files serially, each independently walking (and possibly racing to create) the OSF data/ folder path — ~2(N+2) sequential OSF round-trips added to the response path before the participant's 201. - uploadDerivedFiles now resolves the data/ folder once up front and fans out the actual uploads with Promise.allSettled. Per-file 409 and queue-on-failure handling was already concurrency-safe; folder-create races among concurrent submissions still resolve via subfolder.ts's existing 409-re-list branch. - putFileOSF takes an optional pre-resolved startUrl to upload directly into (or walk any remaining segments from), skipping the redundant walk for files that share an already-resolved folder.
produceMetadata parsed JSON payloads twice (an isCsv() probe, then parseJsonData) and CSV payloads twice (once inside generate(), once via parseCSV for mainRows). Replace the isCsv probe with parseJsonData itself in a try/catch: success is the JSON path with the parsed array already in hand; a throw means CSV. For CSV, parseCSV runs once up front and its rows are passed into generate() as a pre-parsed array (confirmed in functions/metadata/dist/index.js: generate() short-circuits on Array.isArray(data) for both formats before any internal parsing) and reused as mainRows, instead of generate() re-parsing the same text. Golden metadata-production.test.js output (including byte-verbatim CSV mainContent) is unchanged.
…-read blockMetadata threw a module-level Error singleton out of the merge transaction and compared it by reference identity to detect the bootstrap case (Firestore empty, OSF populated) needing an OSF download before merging — discarding and re-running the whole transaction in that case. osfMetadataId is already known before the transaction even starts, so the download can be decided up front. Replaced with a non-transactional pre-read of metadata_doc_ref: if Firestore is empty and osfMetadataId exists, downloadMetadata runs first; the transaction then runs exactly once. It still re-reads Firestore inside, so a concurrent populate between the pre-read and the transaction still resolves correctly via firestoreMetadata ?? osfMetadata. Deletes NEEDS_OSF_METADATA, the inner runMergeTransaction closure, and the identity-check try/catch. The existing metadata-emulator suite's "in OSF but not in firestore" case already covers the bootstrap path.
…e hash changes
Scroll targeting relied on Chakra v3's private data-controls id format
and a 350ms timer, and only ran on mount, so in-page hash changes (e.g.
clicking another #item-N link without a full navigation) did nothing.
Each FAQItem now wraps its Accordion.Item in a <Box id={value}> — a DOM
node we own that's present regardless of expand/collapse state, so
scrollIntoView-style positioning no longer needs to wait on the
accordion's expand animation or depend on Chakra's internal attribute
naming. The scroll effect also listens for hashchange, not just mount.
Running the emulator suite surfaced a regression from the prior sentinel-abort-transaction removal: the OSF download (which can throw, e.g. on a 404) now happened before metadataMessage was ever set, so a download failure produced an empty metadataMessage instead of reporting which of the four Firestore/OSF states triggered it — unlike before, where the transaction always got to set metadataMessage on its first pass before the sentinel throw. metadataMessage is now set from the pre-read immediately, before the download attempt. The transaction still recomputes it from a fresh read (harmless, and correct if a concurrent write races the pre-read). Caught by functions/src/__tests__/metadata-emulator.test.js's "in OSF but not in Firestore" case under the full emulator suite.
The recovery test's expected raw-data path was written before the D1 (encode) commit landed and still expected the old lossy-flatten behavior (data/raw/data.json). Update it to the encoded name (data/raw/condition-A-data.json), matching flattenName's actual current behavior. Caught by running the full emulator suite.
…sion The "resolve data/ once up front" change moved resolveFolder outside the per-file try/catch, so an OSF/network failure threw out of uploadDerivedFiles — which api-data awaits un-try/caught after the raw file already landed and pending was cleaned up. That lost the derived files (never queued) and returned 500 instead of 201, breaking the best-effort "never fails the submission" contract. Wrap the up-front resolveFolder in try/catch and fall back to an undefined folder link, so each under-data/ file re-walks the path inside its own per-file catch (which queues on failure). Add an emulator test that drives an unreachable OSF and asserts every file is queued. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…letion The pending-refresh path read status once, then saved to Cloud Storage non-atomically. If the retry worker finished the doc (-> completed/failed, which deletes the storage object) in that window, the fresh payload was written to an orphaned path and silently lost. Re-read after the save: if the doc is still pending/processing we're done; otherwise fall through to a clean full re-queue so the fresh data isn't dropped. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ssage The parse-once path signalled "valid JSON but not a trial array" by throwing an Error and re-matching its message text, which is fragile if the vendored library ever throws that same string. Use a private NotATrialArrayError class instead; external behavior is unchanged. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
derivedFiles can now only exist on a success result, so the 400 path in api-data can send the failure response verbatim without risking a leak — the guarantee is compiler-enforced rather than resting on a defensive strip. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-icons) The hand-drawn generic glyphs read as 'a triangle', not 'Google Drive'. react-icons is already a dependency and ships monochrome Simple Icons for Google Drive, Dataverse and Zenodo; they inherit currentColor so they sit on the dark theme like everything else. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHZNijLwUhn26yCbYbn1UN
New experiment: use the providers' real marks
The Dataverse and Zenodo marks are not recognisable at label size and add nothing the provider name does not already say. Text-only radio labels. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHZNijLwUhn26yCbYbn1UN
New experiment: drop the provider icons
The sandbox client the test site used was registered under a personal Zenodo account. This swaps in the id from an application registered to datapipe@jspsych.org instead, so the credential outlives any one person's account. TEST_ZENODO_CLIENT_SECRET has already been rotated to the matching secret for this registration -- the two come from one application and cannot be updated independently. Also replaces the TODO above ZENODO_CLIENT_ID, which had been stale since the first registration filled the value in. Sandbox accounts connected under the old client will need to reconnect: invenio-oauth2server scopes tokens to (client_id, user_id), so their stored refresh tokens no longer belong to an application this deployment identifies as. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHZNijLwUhn26yCbYbn1UN
Zenodo test OAuth app: move to the datapipe@jspsych.org registration
The page was two `flex="1"` columns at maxW 1100 with gap 10, which gives
each column exactly 530px. That split was by category, not by measure, and
three things followed from it.
The integration snippets need ~920px including the copy button and both sets
of padding -- the longest line in CodeHints is 92 rendered characters -- so
roughly 40% of the page's primary copy-paste artefact sat behind a horizontal
scrollbar at every viewport width, including the widest.
Five sections went left and one went right, so the left column ran ~1500px
against the right's ~740px. That left 750-900px of empty rail beside the
settings stack, the metadata section and the whole danger zone, and gave the
reader no learnable reason why Metadata was on one side and Integration code
on the other.
And `alignSelf="flex-start"` on the root VStack overrode the `alignItems:
center` that pages/_app.js applies to main content. This was the only page in
the app that did that, so on a wide window the whole 1100px page sat hard
against the left edge, out of alignment with the navbar and footer.
The two columns could never both be satisfied here: 920 for the code plus
DESIGN.md §4's 560 for controls plus a gutter is more than the page has. So
the columns are gone and width is allocated per section by what the section
is -- full width for reference content, CONTROL_MEASURE (560px, the same
measure pages/admin/account.js uses) for controls. Section order is now the
order of the work: what is this -> how do I wire it up -> how do I configure
it -> how do I end it.
At full width the code block's `pre` gets ~964px against the ~770px the
longest line needs, so the horizontal scroll is gone outright rather than
reduced.
ExperimentInfo becomes a horizontal strip. Its rows were an HStack capped at
360px -- a correct fix for the wrong layout, since the section reserved 530px,
used 360px of it, and ran ~190px tall to state three short facts. Label above
value in a SimpleGrid needs no cap: the eye travels a line height instead of a
column width. Cells are built as a list so the grid sizes to however many
facts the experiment has (three normally, four for a legacy OSF experiment,
whose storage location takes two IDs to name).
The "No data yet" EmptyState is folded into the integration-code guidance
line, which now switches copy on `sessions === 0`. It stated a fact the header
already states twice ("0 completed sessions", "Accepting data"), and its body
copy told the researcher to "paste the code below" -- true in the two-column
layout, false the moment the columns wrapped and the card landed ~1500px
underneath the settings stack.
Verified: next build compiles, eslint clean, 24 frontend suites / 217 tests
pass, and the 16 pre-existing Ark UI act() warnings are unchanged in count.
The 38 emulator-backed suites under functions/src/__tests__/ were not run;
none of them touch the frontend.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHZNijLwUhn26yCbYbn1UN
Four small fixes, all copy or surface: The new-experiment page told researchers "the title you have typed here will still be waiting when you come back" next to a form with no title field -- it is inside the providerConnected branch, so nobody without a provider has a title to lose. The message is now one line, and the sessionStorage draft's comments no longer describe the round trip they were written for; what they still cover is the connected case. Provider descriptions on getting-started and /docs/providers drop the "Choose this if..." steers and the Connect with: row that /docs/providers /connecting already owns. Name, what it is, where files land, and the limit that will eventually matter -- which trade-off matters is the researcher's call, and the limits are the only part they cannot work out for themselves. Citing DataPipe moves out of About and into /docs/citation, with the reference in APA and BibTeX and a copy button on each. Both formats and the rendered prose derive from one set of fields in lib/citation.js: a volume number typed once per format is how the block you read and the block you pasted end up disagreeing. CopyButton's state machine moves to lib/use-copy-to-clipboard.js so the code specimen's icon button and the page's labelled button stay one implementation -- and the extracted version clears its reset timeout on unmount. Footer links lose the permanent underline. DESIGN.md §5 mandates it for prose, where color alone cannot mark a link; a labelled <nav> cluster is not prose, which is why the Navbar's links are bare color="fg" too. The four links are now one HStack rather than four items the parent's space-between scattered across the band. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R5dh3MB9EjCCU6ayEfgyyk
Experiment page: one ordered column, width allocated per section
Plainer provider copy, a citation page, and quieter footer links
Two merges to test landed 2.5 minutes apart, so their deploys overlapped. The second one's update of ssrdatapipetest came back 409 "unable to queue the operation" while the first was still running, and the deployed site kept serving the previous build -- the docs changes in #202 never appeared. Add a concurrency group to each deploy workflow so a second run queues instead of racing the first. cancel-in-progress is false: a deploy already updating functions should finish rather than be killed midway. Production has the same workflow shape and the same latent bug, so it gets its own group. Separate groups -- the two target different projects and have no reason to queue behind each other. Note that firebase deploy reports a failed function update as a warning and still exits 0, so a collision goes green rather than red. Serializing the deploys is what prevents it; the exit code will not. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01R5dh3MB9EjCCU6ayEfgyyk
Serialize deploys to each Firebase project
The page had one column but two measures: reference content (the details strip, the integration snippets) at the full 1100px, controls capped at DESIGN.md §4's 560px settings measure. Read top to bottom that is what the eye reports as broken -- two wide bordered panels, then four narrow ones, with the right-hand 540px going empty from the middle of the page down. Every section is now w="100%" at the page's 1100px. CONTROL_MEASURE and its wrapper are gone; nothing steps in or out. The cost is the one the 560px cap was buying off: a SettingsRow puts its label at the left edge and its switch at the right. What that cap was really protecting -- a 140-character description measure -- is already handled a level down, since GuidanceLine carries maxW="70ch", so row descriptions stay readable at any container width. If the switch travel becomes a problem the fix belongs inside SettingsRow, not in re-narrowing the section. Integration code also moves below the settings. It sat second on the argument that wiring up the snippet IS the job on an experiment that has never run; it is reference material the rest of the time, and a researcher who returns to this page returns to change a setting. Order is now the order of the work: what is this -> how do I configure it -> how do I wire it up -> how do I end it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHZNijLwUhn26yCbYbn1UN
Experiment page: one column, one width
Flipping a switch in the experiment panel mounted a "Saved" line under that row, which grew the row and pushed every row below it down for three seconds, then sprang them back. Feedback that the change landed was the whole panel jumping -- and if the researcher had already reached for the next switch, the switch they were aiming at had moved. Split the one SaveStatus component into the two things it was doing: - SavedFlag: the success chip, now inline in the row immediately left of the control it confirms. It is ALWAYS rendered at full size and only its opacity changes, so nothing reflows in either direction. Its visible text is a constant "Saved", which keeps the reserved width identical on every row; the caller's descriptive savedLabel moves to a visually hidden role="status" region so screen readers still hear which setting was saved. - SaveError: the failure alert, still a block below the row. It is a full sentence, it is not transient, and it should claim space. The dependent controls get the same pair one level down -- the two numeric fields in ExperimentActive, the validation detail block, and the title rename, whose confirmation sat under the page H1 and moved the entire page. Verified in headless Chrome: every row's top, height, and switch position is byte-identical before and after a save lands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHZNijLwUhn26yCbYbn1UN
Saved confirmation: move it out of the layout flow
Finalizing seals a dataset into one archive and deletes the loose files, and the submission guards already reject everything on `finalized` alone -- ahead of the active switch. But nothing told the dashboard. A finalized experiment kept `active: true`, so its own settings page drew a green "Accepting data" switch on a record that no longer accepted anything, and the experiment list said "Collecting data". The researcher's dashboard was the least accurate thing about their study. Backend: markFinalized writes `active: false` and `activeBase64: false` in the SAME update as `finalized: true`, so no reader ever sees a snapshot where the two disagree. `activeConditionAssignment` is left alone -- api-condition.ts has no finalized check by design, and switching it off here would claim a behaviour change this does not implement. Frontend, three surfaces: - Experiment list: a "Finalized" StatusIndicator REPLACES the collecting/not-collecting line. "Not collecting" is the weaker and more reversible-sounding fact, so leading with it buries the one that matters. - Experiment page header: same substitution in the status row under the title. - Data collection settings: "Accept new data" and "Accept base64 file uploads" render off and locked, each with a description naming finalizing as the reason -- a control that is off with no explanation reads as broken. All three read `finalized` FIRST, which is what stops experiments finalized before this change (still holding `active: true`) from advertising that they are collecting. Those documents are not migrated; they display correctly and the server was already rejecting their submissions. Docs: the "It cannot be undone" list said submissions are rejected "whether or not the experiment is still switched on", which is now misleading about what finalizing itself does to those switches. Tests: 10 new frontend tests over all three surfaces including the legacy shape, and 2 new emulator tests pinning the write (and pinning that condition assignment survives it). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHZNijLwUhn26yCbYbn1UN
Two changes to the same surfaces.
1. The Finalize section is Zenodo-only now, because finalizing always was.
Finalizing merges a study into one archive to stay under a provider's
file-count ceiling, and Zenodo's 100-files-per-record is the only such
ceiling in the codebase -- finalization.ts refuses every other provider
with `not-eligible`. But the Danger zone rendered for all of them, so a
Google Drive or Dataverse researcher was shown a red section promising
to stop their experiment "forever", walked through a confirmation
dialog about permanently deleting their loose files, and only then told
it could not be done. The refusal then read:
This experiment can't be finalized.
provider has no file-count cap
-- an internal capability name rendered to a researcher, which
DESIGN.md 6 bans outright.
The whole section is hidden rather than just the button: finalization
is its only entry, and an empty Danger zone is its own kind of
alarming. `not-eligible` also gets a real sentence for the paths that
can still reach it (a legacy OSF experiment, an unknown provider).
`supportsFinalizing` in lib/provider-config.js is the frontend's
presentational mirror of each adapter's `capabilities.maxFileCount`,
the same hand-synced arrangement `containerInputFields` already uses
and with the same caveat: the server stays authoritative. Legacy OSF
experiments are absent from that map entirely and correctly fall
through to false.
2. The experiment list names the provider each experiment is stored on.
A lab mid-migration has experiments on two or three providers at once
and nothing in a title says which. "Stored on Zenodo", not a bare
"Zenodo": every other item on that line says what it is ("42
sessions", "base64 uploads on"), and a lone proper noun among them
would be the only one the reader has to work out. Legacy OSF
experiments are named too -- those are the rows whose location matters
most while OSF is being retired.
Docs: /docs/data/finalizing said clicking Finalize on an ineligible
experiment reports an error. There is no longer a Finalize to click.
8 new tests: provider naming for all four providers plus the
unrecognised case, and the Danger zone appearing on Zenodo and on
nothing else.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHZNijLwUhn26yCbYbn1UN
Finalizing an experiment closes it, visibly
`requiredFields` is compared with `Array.includes` against JSON keys and CSV header cells. The match is exact, the server trims nothing, and a rejection is a bare 400 `INVALID_DATA` that does not name the field that failed. So a stray space, a pasted quote, or a trailing comma does not produce a warning -- it produces an experiment that silently rejects every participant. A textarea cannot show that. `trial_type , rt` and `trial_type,rt` look the same at a glance, and the empty string a trailing comma leaves behind is invisible by definition -- that one shipped, and `[""]` in a live experiment document is why both validators now carry a filter for it. Replace it with TagListInput: each name becomes a pill the moment it is committed, so the researcher sees exactly the string that will be compared. Built on Chakra's TagsInput rather than hand-rolled -- the machine already ships keyboard navigation between pills, in-place editing, a live region, and delimiter/paste splitting. Normalization runs on the way OUT, in one place. There are four routes in (Enter, comma, paste, in-place edit) and the machine treats them as four code paths; guarding them one by one is how you guard three of them. `normalizeList` trims, strips pasted quotes, drops empties and de-duplicates, so the caller only ever sees a clean list. `validate` is not where correctness lives -- it exists so a rejected keystroke can say why instead of the pill quietly failing to appear. Four deviations from Chakra's stock recipe, each a real defect: - `itemDeleteTrigger` ships `opacity: 0.4`, nowhere near the 3:1 floor, on the control that removes a rule from a live experiment. - That same trigger ships no focus ring; the recipe gives one to `clearTrigger` and forgets the per-item one (DESIGN.md §8.8). - It is sized `item-height / 1.5` = 18.7px and pulled over the pill's padding by a negative margin. Now 24x24 per WCAG 2.2 2.5.8; pills sit 8px apart, so the spacing exception does not rescue an 18px target. - `itemPreview` fills with `colorPalette.subtle` and no border -- 1.17:1 against the control on the light page. Now `bg.muted` + 1px `border`, the rule DESIGN.md §1 already sets for panels. Two existing house rules carried over: the control reserves the one-pill row height so the first pill does not grow the field, and a rejection notice replaces the helper line rather than mounting under it -- the same no-reflow principle as SavedFlag. Legacy `[""]` documents now draw no phantom pill, and the docs sentence describing the old comma-separated box is updated with it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016iV4rd2rgXxCYS6CMGRzN5
Required fields: show the parsed list, don't ask them to imagine it
logs/{experimentID} recorded four counters and an unbounded array of
errors stamped with a preformatted string. Four things were wrong with
that, and the first one is a bug nobody could see.
`owner` was never written. firestore.rules gates log reads on
`resource.data.owner == request.auth.uid`, and no revision of
write-log.ts has ever written that field -- so the rule could not pass
for any document in the collection. The dashboard's logs/{id}
subscription failed silently and ErrorPanel never rendered. Researchers
have not been able to see their own rejected submissions. It is now
written on every log write and seeded by create-experiment.ts;
functions/scripts/backfill-log-owner.mjs repairs the documents already
in production.
`errors` grew by arrayUnion with no bound inside a document with a 1 MB
ceiling, so an experiment erroring on every submission would eventually
make every write to the document fail -- taking the counters with it.
It is capped at the 50 most recent. The cap needs the current value, so
this goes through a transaction, which also collapses what used to be
two non-atomic set() calls into one: the `logError > 0` with no
`errors` field window that ErrorPanel had to defend against is gone.
`time` is a real Timestamp instead of a formatted en-GB string, so
error entries can be sorted and filtered. Timestamp.now() rather than
serverTimestamp(), because Firestore rejects sentinels inside an array.
Entries written before this change keep their string; ErrorPanel
renders both until the cap rotates them out.
Attempts were counted before the experiment was known to exist, so any
POST with a garbage ID inflated the request totals and created an
ownerless document nobody could read. Counted after the check now.
Added for querying service health, which per-experiment counters could
not answer: `storageProvider` and `errorsByCode` together turn "which
provider is failing and how" into a query rather than a scan;
`createdAt`/`lastRequestAt` separate dormant experiments from live
ones; and saveDataSucceeded/saveDataQueued make the outcome of a
submission explicit. There is deliberately no ...Failed counter --
failures are saveData minus the other two, exactly, and a counter
incremented at nineteen return points would drift the first time a
branch was added.
Counting experiments per provider is NOT part of this: experiments is
the source of truth for that and a count() aggregation answers it
directly. What experiments cannot answer is the failure rate, which is
what the two new fields above are for.
No firstRequestAt: maintaining it would cost a read of the log document
on every submission just to discover whether the field was already set.
createdAt, seeded in a batch create-experiment was already committing,
answers the same question for free.
pages/docs/privacy.js is updated to describe what the log now keeps and
the 50-entry rotation.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHZNijLwUhn26yCbYbn1UN
Experiment logs: make them readable, bounded, and queryable
Ticking "Allow CSV" or "Allow JSON" in the admin panel turned data
validation off and collapsed the block the checkbox lives in. The
researcher's click did the opposite of what it said, and the setting it
silently changed decides whether malformed submissions are rejected.
The cause was an id collision, not anything in the validation logic. Ark
derives a control's hidden-input id from the surrounding Field context --
`useSwitch` and `useCheckbox` both do `ids: { hiddenInput:
field?.ids.control, label: field?.ids.label }`. SettingsRow rendered its
`children` inside the row's own `Field.Root`, so both checkboxes were
handed the SAME DOM id as the master switch's hidden input.
`Checkbox.Root` is itself a `<label htmlFor={thatId}>`, and the browser
resolves a label to the FIRST element carrying that id -- the switch,
which renders above. So the click activated the switch.
The Field now wraps only the switch and its label; `SaveError` and
`children` move out to an enclosing Stack. Dependent controls elsewhere
(the numeric inputs in ExperimentActive, the required-fields input) each
open their own `Field.Root` already, so they were never affected and are
unaffected by this -- only controls that INHERITED the row's field were
broken by it, which is why the fix belongs in SettingsRow rather than in
any one call site.
Tests click the labels, not the inputs: clicking an input directly never
went through the broken path, so a test that does that passes either way.
Three of the new cases fail against the previous SettingsRow.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SHZNijLwUhn26yCbYbn1UN
…sion Stop a validation checkbox from flipping the switch above it
With validation on and neither format allowed, api-data.ts cannot reach either validator: `valid` starts false, both branches are skipped, and every submission takes the 400. The experiment silently stops collecting, and the error the researcher eventually finds in the log says the DATA is invalid -- pointing at the participant's submission, when the submission never mattered. Two clicks in a panel that gives no sign the second one means something categorically different from the first. There is no legitimate reading of "validate submissions, accept nothing", so the control now declines rather than saving it, the way TagListInput declines a duplicate. Declining is just not calling setValidationSettings: the group is controlled, so the tick mark never moves and the save effect never fires. Two things that were not free: The refusal needs a voice. A control that declines silently is indistinguishable from one that is broken, so the checkboxes gain the same always-one-line message slot TagListInput uses -- steady-state guidance swapped for the reason, never a block that mounts underneath and pushes the rest of the panel down on every click. The transient-notice machinery is now `useTransientNotice`, shared by both, so the two refusals in this panel say their piece for the same duration. And the refusal needs to reach the accessibility tree. Ark has already flipped the real <input> by the time onValueChange runs; refusing leaves React state untouched, so React re-renders with the same `checked` prop and never resets the node. What the researcher SEES stays correct -- the tick is drawn from the controlled group value -- but the hidden input is what a screen reader reads, and it now says unchecked: the opposite of what happened, and worse than no feedback at all. `resyncFormatInputs` puts the property back. This is a UI floor, not a guarantee. The server is unchanged, so a document that already holds the state still behaves as it did. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SHZNijLwUhn26yCbYbn1UN
Refuse to untick the last allowed submission format
AWS denied the SES production-access request, which would have left every
send restricted to verified recipient addresses -- failing for every real
researcher, and terminally, since mail-delivery.ts classifies a rejected
recipient as permanent. Resend has no equivalent gate.
Only the transport moved. mail.ts's document contract, the claim/lease
machinery, the delivery.* outcome fields, the TTL policy and
purge-user-data.ts's owner query are all untouched, which is what kept this
to one file plus its tests -- the same property that made the earlier
extension-to-SES swap cheap.
- buildSendEmailInput emits Resend's JSON body. It uses reply_to, not
replyTo: the SDK takes camelCase, the raw REST API takes snake_case and
ignores unknown keys silently, so getting it wrong would mean every
notification going out with no Reply-To and nothing saying so. Pinned by
its own test.
- classifySesError -> classifyMailError, rebuilt on Resend's error codes
with HTTP status as the fallback.
- The transport is one fetch to one endpoint. @aws-sdk/client-sesv2 is
gone, and with it the dynamic import that existed to keep the SDK off
apidata's cold-start path.
- Undici reports every network failure as `TypeError: fetch failed` with
the real diagnosis on .cause, so errorName unwraps it. Without that, a
momentary DNS blip classifies as unrecognised, which is terminal, which
silently loses a notification.
One behavioural change, deliberate: ambiguous failures (timeouts,
ECONNRESET) are now retryable. They were terminal because SESv2 has no
idempotency token, so a request that went out and never answered could not
be retried without risking a second copy of a notification whose whole value
is arriving once. Resend takes an Idempotency-Key; we send the mail
document's id on every attempt, so the retry is a no-op at Resend rather
than a second send. The key expires after 24h, which bounds any future retry
mechanism -- noted in AMBIGUOUS_ERRORS and in both test suites.
Six *_SES_* repo secrets become one *_RESEND_API_KEY per environment. The
test site is now expected to send: it is the only place delivery is
exercised before production, since the emulator short-circuits before
sending and the unit suites mock the transport. Reputation and daily quota
are shared with prod -- documented in runbook 2(d).
Also corrects three things found while verifying the deploy steps, none of
them caused by this change:
- The production project is osf-relay, not datapipe-prod (.firebaserc's
default, and what `firebase use default` selects). GCP project ids are
immutable, so datapipe-prod never existed; the runbook and three
migration scripts had it wrong.
- Runbook 4's console instructions named delivery.endTime where the gcloud
command says delivery.expireAt. Following the console path would create
a policy that reaps every mail document the moment it is delivered.
- mail.ts's header still described the extension as the transport and
claimed switching providers was a config change rather than a redeploy.
Claude-Session: https://claude.ai/code/session_01D1dNU7EnmBBMTn8MWfojDL
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
The TTL on mail/delivery.expireAt was created by hand with gcloud, as the runbook told us to. The next deploy deleted it: 14:51 gcloud ... --enable-ttl --project=datapipe-test -> ACTIVE 15:02 PR #211 merged to `test` 15:04 firestore: Deleting 1 field overrides... <- the deploy 15:10 TTL gone `firebase deploy --only firestore` reconciles field overrides against firestore.indexes.json, and that file said `"fieldOverrides": []` -- so a hand-made TTL was not merely unmanaged, it was something the deploy was actively instructed to delete. --force suppressed the prompt that would have said so, and the deploy reported success. The retention promise stops being kept, silently, while the runbook says it is. Declared in firestore.indexes.json instead, so it is version-controlled and recreated by the same deploy that would otherwise remove it. The block keeps the three default single-field indexes: a fieldOverride replaces the field's whole index configuration, so `"indexes": []` would additionally turn off single-field indexing for delivery.expireAt. What is committed here is what `firebase firestore:indexes` emits for a field with TTL on and default indexing, read off osf-relay while its policy was still active, so it round-trips. Runbook 4 rewritten around the declarative mechanism, keeping the timeline above -- following the old instructions produced a policy that survived exactly until the next deploy, and that is worth being unable to rediscover the hard way. Also corrects two comments in mail-delivery.ts that said the TTL keys on delivery.endTime. It keys on delivery.expireAt; endTime only gates whether expireAt gets written. That same confusion is what put the wrong field in runbook 4's console instructions. Claude-Session: https://claude.ai/code/session_01D1dNU7EnmBBMTn8MWfojDL Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…er is told (#213) * Treat realtime and deferrable mail differently when quota runs out Verification codes and upload-failure notifications are not the same kind of mail, and the difference only shows up at the daily quota. A code is realtime -- someone is watching a form, and it expires in 24 hours -- so a code delivered tomorrow is a failure with extra steps. "Your data stopped arriving" is just as true an hour later. Until now both were queued and neither was retried, which produced the worst of both. The verification endpoint returned 200 before delivery was even attempted, so a researcher was told to check an inbox nothing was coming to, and was given a 60-second cooldown on asking again. And a queued upload-failure mail that died on quota was retried by nobody, while upload-failure-notify.ts had already armed the episode as "told them" in the same transaction that enqueued it -- so the researcher was never notified and the experiment document asserted that they had been. THE BREAKER (mail-availability.ts, systemStatus/mail) Resend returns x-resend-daily-quota -- the quota USED today -- on SUCCESSFUL responses, not only on 429s. So the breaker is proactive: we learn we are at 94/100 while sending still works, rather than by failing. Verification stops at a ceiling of 90; upload-failure notifications keep going to the full 100. That asymmetry is the design decision. A researcher waiting on a code can come back later; a notification that data has stopped arriving is the only signal they get, so it should have the last sends of the day. The rate-limit headers are no help here -- ratelimit-reset describes the per-second window -- and Resend documents no daily reset time, so nothing depends on knowing one. unavailableUntil is next UTC midnight as a CEILING; what actually reopens sending is the sweeper landing a success. The deferrable path probes, the realtime path only reads. The header is free-plan-only, so it vanishes on a paid plan. Absent reads as "no daily cap", which means this logic turns itself off on upgrade rather than needing removal. VERIFICATION IS NOW SYNCHRONOUS The breaker is checked before a code is minted, a record is written, or a cooldown is armed -- so a researcher clicking during an outage costs zero Resend requests. The send is then awaited, and a failure clears the verification record so they can retry immediately instead of waiting out a cooldown for a code that does not exist. onMailCreated stands down for inline mail: the claim would make the race safe, but the loser gets skipped-in-flight and could not report an outcome, which is the ambiguity the inline path exists to remove. Inline failures are terminal, never retryable. Nothing will retry them, and a retryable error is never given delivery.expireAt -- so calling them retryable would park an address outside the TTL's reach indefinitely. The message is vague on purpose, and the same for every cause: quota, an unverified domain, a revoked key. None of it is actionable by a researcher. THE SWEEPER (scheduled-mail-retry.ts, every 10 minutes) Re-drives retryable mail, and refuses three things. Inline mail, ever. Ambiguous errors past 24 hours, because mail-delivery.ts only made timeouts retryable on the strength of Resend's Idempotency-Key and Resend honours it for a day -- REFUSED_ERRORS (refused, or never reached Resend) carry no such risk and sweep at any age. And anything at all while the breaker is shut, because sweeping into an exhausted quota fails every document AND spends one of its three MAX_ATTEMPTS doing it, so a day-long outage would exhaust every queued mail's budget and turn them all terminal. It also ages out mail nobody could deliver in three days, marking it terminal so delivery.expireAt is finally written. That closes the hole runbook 4 notes: a retryable ERROR otherwise holds an address forever, outside the TTL policy, with no sweeper to catch it. Alerting is documented rather than built (runbook 6): two log-based metrics, no code, routed somewhere Google delivers -- you cannot email yourself that you are out of email. Adds a composite index for the sweep query, a firestore.rules note that systemStatus stays unmatched and therefore closed, and 33 tests: 21 pure (both predicates as tables) and 12 emulator-backed. The emulator suite uses its own status document, because systemStatus/mail is a singleton and a suite that shut the real breaker would fail every other suite that sends mail, differently each run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1dNU7EnmBBMTn8MWfojDL * Stop deleting data the researcher was never told about Two ways the queue cleanup lost data nobody meant to lose. It deleted on `createdAt <= now - 7d` with no check on status and no check on whether anyone had been told -- and since there is no GCS lifecycle rule on the bucket, that sweep is the entire retention policy. 1. A STORAGE PROVIDER OUTAGE. The entry is still pending with retries left, so it would have uploaded fine on day eight. Deleting it on day seven throws away data that was never actually lost. 2. A NOTIFICATION THAT NEVER ARRIVED. The seven days count from SUBMISSION, so part of the window is spent before anything goes wrong -- and if the notification died on a quota outage, the window closes without the researcher ever learning there was one. upload-retention.ts now decides, per entry: delete past 14 days from createdAt whatever else is true; retain while pending with retries left; retain while retainUntil is in the future; otherwise delete, unchanged. Its own module and a pure function, because it decides the least reversible thing here and scheduled-upload-retry.ts pulls in the whole provider stack -- a test wanting only this predicate would have to load all of it. scheduled-mail-retry.ts writes retainUntil while a notification is undelivered, covering EVERY unresolved entry for the experiment rather than the one that tripped the episode. datapipe.queueDocId records only the trigger; extending that alone would leave the rest of the episode's files expiring on schedule, which is the same bug in miniature. The retention pass runs BEFORE the breaker check and therefore also while sending is paused. Extending is a Firestore write that costs no quota, and a quota outage is precisely when data ages towards deletion behind a notification that never arrived -- putting it after the check would disable the protection in the only case that needs it. An experiment whose owner has no contact email is never extended, and that falls out rather than being special-cased: upload-failure-notify.ts records suppressedReason "no-contact-email" and returns before enqueuing, so no mail document exists for the sweeper to find. Plain seven days, which is right when there is nobody to tell. The 14-day ceiling is load-bearing. Without it an experiment whose provider is dead and whose owner never reads their mail would hold research payloads forever, silently, at DataPipe's cost. THE WARNING BANNER Targets UNVERIFIED addresses, not missing ones. ContactEmailGate already walls off every admin route until a usable address exists, so "no contact email" is nearly extinct; what the gate does not check is whether the address works, since hasContactEmail() tests format only. A typo passes it, so does anything the 2026-08 backfill seeded from Auth, and upload-failure-notify.ts mails the address regardless of verified status -- so it bounces and is marked terminally failed somewhere nobody looks. Not dismissible and no flag to maintain: contactEmailVerified is written server-side by verify-contact-email.ts and only there, so the banner removes itself the moment it is acted on. 10 new tests, 1162 total. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1dNU7EnmBBMTn8MWfojDL * Close the ways mail could stop for a month, or stall forever The breaker and the sweeper both assumed the only way sending fails is a daily quota that resets at midnight. Neither assumption survives contact with the other failure modes. WHY SENDING STOPPED DECIDES HOW LONG IT STAYS STOPPED The free plan has two caps -- 100/day and 3,000/month -- on different clocks. Reusing the daily reset for a monthly exhaustion is not a small error: hit the monthly cap on the 20th and the breaker reopens at midnight, the sweeper probes into a cap with eleven days left to run, and spends one of each queued document's MAX_ATTEMPTS doing it. Three nights of that and every queued notification is terminal -- the exact retry-budget exhaustion the breaker exists to prevent. `PauseKind` names the three reasons and `pauseUntil` maps each to a reset: daily-quota to the next UTC midnight, monthly-quota to the next month, and systemic to a 15-minute cooldown. `pauseKindFor` is the table that turns a Resend error name into one, and both call sites go through it. SYSTEMIC FAILURES TRIP IT TOO A revoked key or an unverified domain is not exhaustion, and previously nothing shut on it: verificationAvailability answered "available" forever while every click minted a code, wrote a document, spent a real request and failed, with no server-side ceiling. Tripping bounds that to one request per SYSTEMIC_PAUSE_MS however many researchers are pressing the button. `validation_error` is deliberately NOT in SYSTEMIC_ERRORS. Resend uses it both for an unverified sending domain and for one researcher's typo, and a typo must not switch verification off for everybody. THE HEADER READING IS A GUESS, AND IT IS BOUNDED Resend documents x-resend-daily-quota's existence, not its semantics -- "used today" and "remaining today" are the same shape. If it is really the plan limit, every reading is 100, the reserve rule is true forever, and each send rewrites dailyQuotaObservedAt so the staleness escape hatch never fires either. `usableQuotaReading` refuses anything outside [0, 100] at the write and ignores it at the read, refusing loudly; refusing a verification on the reserve alone now logs at ERROR with a stable token, so the condition is alertable rather than silent. The runbook says how to check the number against the dashboard before trusting it. THE VERIFICATION RECORD IS THE RATE LIMIT, SO IT STAYS The failure branch used to delete it, reasoning that a researcher whose code never arrived should not wait out a cooldown for a code that does not exist. Right about the researcher, wrong about the endpoint: `sentAt` is the only server-side throttle on that path, and deleting it removed the throttle from exactly the case that needs one. The record now survives with `codeHash` cleared -- a cooldown, not a secret nobody received -- and verify-contact-email.ts answers "request a new one" rather than spending one of the five attempts on a code that was never sent. EVERY DOCUMENT THE SWEEP FINDS LEAVES BY A DOOR The queries are unordered limits, so anything the pass can look at without changing it will look at again next pass, forever, occupying the budget a deliverable notification needed. The decision table has no permanent skip left in it: every outcome either sends or writes a terminal state that takes the document out of both queries. The only skips are documents another invocation holds, which resolve within LEASE_MS. That matters beyond starvation. A retryable ERROR never gets delivery.expireAt, so it sits outside the TTL policy holding a researcher's address indefinitely. Ageing one out is how that address is finally deleted. A second query finds stranded claims. deliverMailDocument's claim rewrites the document to PROCESSING with `retryable: null` BEFORE the send, so an instance that dies mid-send left a document neither the retryable-ERROR query nor the TTL could see -- a preempted instance or a mid-deploy roll stranded an address permanently. claimDecision already knew how to recover an expired lease; nothing ever asked it to. The idempotency window is now measured from `startTime`, which is when the key was first used and never moves. Measuring from the last attempt slid the window forward with every retry, so a document retried at +20h and again at +40h was sent on a key that expired at +24h and Resend treated it as a new message. RETENTION MOVES NEXT TO THE PREDICATE THAT READS IT `extendRetentionForExperiment` lives in upload-retention.ts now. Splitting "may this data be destroyed, and what holds it back" across the mail sweeper, that module and the deletion sweep meant three files in two subsystems had to be read together to answer one question, and the mail subsystem had to know the uploadQueue's schema. The sweeper now says only what it knows -- this researcher has not been told -- and retention decides what that means. Inverting it completely was considered and rejected: asking "is a notification still undelivered?" at deletion time makes the least reversible operation in the codebase depend on a query that can fail, and a failing query there deletes data. `retainUntil` fails the other way. It covers documents about to be AGED OUT as well as sent, which is the more important half -- giving up is when the researcher will never be told at all, and it must not also be when their data goes back on the original clock. Once per experiment rather than once per document, and skipped entirely when the stored value is already more than half the grace window out: the sweeper runs every ten minutes, so without that floor a day-long outage rewrites the same field tens of thousands of times to no effect, against a 20,000/day free tier. THE DELETION SWEEP PAGES PAST WHAT IT MAY NOT DELETE cleanupOldEntries found entries by age, ascending, with one limit(50) -- but whether one may be destroyed is a separate question, and a retained entry is not removed from the result set by being looked at. Fifty retained entries at the head meant a pass that deleted nothing, forever: one experiment stuck behind a dead provider stopped every other experiment's payloads from being deleted until the blockers crossed the 14-day ceiling. It now scans up to 500 with a cursor to find its 50. ALSO - mailCollection() indirection, so a suite can isolate the collection the sweep queries. Isolating the status document was never enough on its own; the collection is the other shared singleton. The trigger still binds to MAIL_COLLECTION, which is deploy-time configuration a test may not move. - One users/{uid} subscription on /admin, passed down, replacing two. - leaseIsExpired, pauseKindFor and unresolvedQueueEntriesQuery each own a rule that was previously written out twice. - Age-out and retention writes run concurrently; a backlog was up to fifty serialized round-trips. Not batched -- a batch is atomic, and one document purged mid-sweep would take the rest with it. - Composite index for delivery.state + delivery.leaseExpiresAt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01D1dNU7EnmBBMTn8MWfojDL --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Syncs
mainwith everything merged totestsince the last release sync (#149, April 2026) — 187 commits across PRs #150–#187.Highlights
Multi-backend provider migration (#154–#161, #164) — core architecture for sending data to providers beyond OSF, plus:
Required contact email (#184) — schema + Firestore rules, signup seeding, verification round trip, first-failure notification state machine, backfill migration.
Payload encryption at rest (#185) and mail delivery via SES trigger, replacing the deprecated extension.
Metadata — version-drift fix, sidecar files, discoverability, follow-ups (#150–#153).
Docs — full restructure with new nav and redirects (#186), multi-backend guides (#178), 32 MB limit documentation.
Design pass — account settings redesign (#179), light mode (#181), homepage cleanup (#182), logo animation (#180), spacing pass (#183), branded 404 (#42).
CI & deps — ESLint on every PR (#109), dependency refresh clearing all advisories, firebase-admin v14, pinned firebase-tools (#187).
🤖 Generated with Claude Code