RFC 092: Keeping digitised archive material connected after Sierra

rfcs/092-digitised-archive-links/README.md

@@ -0,0 +1,101 @@
+# RFC 092: Keeping digitised archive material connected after Sierra
+
+## Purpose
+
+Digitised archive material on wellcomecollection.org is connected to its archive description through the CALM-synced Sierra bib, and those bibs are deliberately not migrated to FOLIO. This RFC records how the connection works today, shows that the catalogue pipeline can maintain it with no Sierra or FOLIO record existing provided the Axiell Collections record cites the Sierra b number, quantifies the gap (roughly 93% of digitised archive records carry no b number in Axiell), and proposes importing the b numbers into Axiell Collections, matched on each record's public reference and verified through the CALM RecordID carried in the migration. It is the archive-side companion to RFC 091, which covers the library-side (FOLIO-target) half of post-migration digitisation merging.
+
+**Last modified:** 2026-08-17T14:12:00+00:00
+
+**Related RFCs:**
+
+- [RFC 091: Preservation identifiers across the LMS migration](../091-digitisation-ingest-identifiers/README.md): the library-side half of the same problem. Its merge-behaviour section is written for FOLIO targets and states that the Sierra work is always the merge target; that holds for library material only, and this RFC records the archive-material case where a CALM or Axiell work is the target. Cross-referencing this from RFC 091 is a next step below.
+- [RFC 083: Stable identifiers following mass record migration](../083-stable_identifiers/README.md): the predecessor mechanism that keeps public work ids stable across the migrations. The Axiell records' CALM RecordID predecessor is what preserves the work ids of everything discussed here.
+- [RFC 090: CMS to LMS Sync](../090-axiell-folio-sync/README.md): the adjacent synchronisation work. The scoping decision that Axiell and FOLIO hold mutually exclusive record sets, which removes the Sierra bridge this RFC replaces, comes from that programme.
+
+## Context
+
+### How a digitised archive work is assembled today
+
+A digitised archive work on wellcomecollection.org is three records merged into one. Taking [nfnae3jc](https://wellcomecollection.org/works/nfnae3jc) as the example:
+
+```mermaid
+flowchart LR
+    METS["METS record<br/>digitised images"] -->|cites b33119740| SIERRA["Sierra bib<br/>b33119740"]
+    SIERRA -->|cites CALM RecordID| CALM["CALM record<br/>PPAMI/A/23"]
+    CALM ==> WORK(["merged work nfnae3jc:<br/>CALM description + Sierra item<br/>+ METS digitised images"])
+```
+
+Each hop uses a different identifier. The METS record knows only the b number. The Sierra bib knows the CALM RecordID (it was created by the CALM-to-Sierra sync). The CALM record is chosen as the face of the merged work and the other two redirect into it. There is no direct link between METS and CALM anywhere in the data; the Sierra bib is the only bridge.
+
+The worry is that Sierra retires, the CALM-synced bibs are deliberately not migrated to FOLIO (Axiell and FOLIO are meant to hold mutually exclusive record sets), and the bridge disappears:
+
+```mermaid
+flowchart LR
+    METS["METS record<br/>digitised images"] -->|cites b33119740| GONE["✗ no Sierra record;<br/>no FOLIO record, by design;<br/>nothing in AxC cites b33119740"]
+    AXC["AxC record<br/>PPAMI/A/23"] ==> WORK(["work keeps its archive description<br/>but loses its digitised images"])
+```
+
+About 94,430 works with digitised content are in the Archives and manuscripts format.
+
+### The pipeline needs only the identifier
+
+The matcher connects records that cite the same identifier even when no record owns it. The identifier acts as a meeting point, and the merger can merge a METS record directly into a CALM or Axiell work with no Sierra record in the graph. Both behaviours are covered by tests, and born-digital material already depends on them in production: an Archivematica METS record and a CALM record meet on the RefNo, which neither of them owns as their own identifier, and merge with no Sierra record involved.
+
+So the fix requires neither migrating the CALM-synced bibs into FOLIO nor any new pipeline feature. If the AxC record cites the b number, everything reconnects:
+
+```mermaid
+flowchart LR
+    METS["METS record"] -->|cites b33119740| ID(["b33119740<br/>an identifier both records cite;<br/>no record needs to own it"])
+    AXC["AxC record"] -->|"035 (Bibliographic Number)"| ID
+    AXC ==> WORK(["merged work: AxC description<br/>+ METS digitised images,<br/>keeping the work id inherited from CALM"])
+```
+
+The Axiell transformer already turns an `035 (Bibliographic Number)` into exactly this link; it is the mechanism behind the GC179 merge test records.
+
+### The missing b numbers
+
+CALM has a Bnumber field, but it was filled in for only a small minority of records. In a sample of 676 digitised archive records, 47 (about 7%) have it, and nfnae3jc's CALM record has the field empty. The migration into AxC carried the b number where CALM had it and nowhere else. In the round-2 AxC load harvested 17 August, 6,999 of 210,846 records (3.3%) carry an `035 (Bibliographic Number)`, and the AxC record for our example, though otherwise rich, cites nothing that connects it to b33119740. For roughly 93% of digitised archive material, the day Sierra's records stop flowing, the images detach from the work.
+
+### What we already have
+
+The CALM RecordID did make it into AxC. The current data carries it (it surfaces as MARC 907, present on 209,544 of the 210,846 records in the round-2 load, 99.4%), and in round-1 testing 201,072 works landed on the same public work ids as production with zero mismatches, so work URLs are stable and every AxC record has a reliable key to match an import against.
+
+We also hold the complete b number mapping. Production currently has 246,474 Sierra bibs merged into archive works, and each of those bibs cites the CALM RecordID it was synced from, so extracting (CALM RecordID, b number) pairs is a straightforward job against data we already have.
+
+## Proposal
+
+Generate the (CALM RecordID, b number) pairs from the current Sierra data and import the b numbers into the AxC records so they surface as `035 (Bibliographic Number)` in the harvest. The Axiell import matches records on their object_number, the public reference (AltRefNo); the CALM RecordID carried in the migration is how the tooling derives and verifies each record's reference before it becomes the match key. From there the existing pipeline does the rest, with no Sierra or FOLIO record needed.
+
+The agreed import format is `object_number,alternative_number,alternative_number.type`, for example `WT/D/1/20/1/35/95,b33174192,Bibliographic Number`, with the type column constant.
+
+Tooling for this lives in the catalogue-pipeline repo under `scripts/axiell_bnumber_import/` (wellcomecollection/catalogue-pipeline#3579). One script extracts the pairs from the Sierra source works' merge candidates; a second joins them against the 907 RecordIDs in the Axiell adapter store and emits the import CSV in the agreed format alongside conflict and unmatched reports, withholding rows where several AxC records would share the reference the import matches on. Both steps are read only and deterministic, and a delta mode emits only rows new since a previous CSV. The first cut, run 17 August against the round-2 store, extracted 246,204 pairs (exactly one bib per RecordID) and yields 197,090 importable rows, with 6,476 already present in AxC and 38 conflicts to review; every matched record carries a unique public reference, so nothing is withheld.
+
+The data should live in AxC rather than be enriched in the pipeline. A pipeline enrichment step is feasible as a fallback, but it leaves the linkage invisible to every other consumer of the Axiell data, and where the linkage lives is exactly the kind of what-goes-where decision this RFC exists to record.
+
+## Alternatives considered
+
+- **Enrich in the catalogue pipeline.** Hold the (CALM RecordID, b number) mapping in the pipeline and inject the merge candidate during the Axiell transform. Workable and entirely under our control, but the linkage becomes invisible to every other consumer of the Axiell data (the FOLIO sync, reporting, any future re-migration), and the mapping ossifies in a place collections staff cannot see or maintain. Kept as the fallback if the import route fails.
+- **Migrate the CALM-synced Sierra bibs into FOLIO after all.** This restores the bridge by keeping a successor record for every bib. It contradicts the agreed scoping (Axiell and FOLIO hold mutually exclusive record sets), creates tens of thousands of FOLIO records whose only purpose is to be merged over, and still needs the FOLIO-target merger work from RFC 091. Rejected as strictly more work for a worse data model.
+- **TEI-style replacement records.** Manuscripts absent from AxC are getting TEI records that carry both the Calm ID and the Sierra b number, which is this same meeting-point mechanism in a different source. That route fits material that is being re-catalogued anyway; it is not a general answer for 200,000+ Axiell records that already exist.
+- **Do nothing.** Roughly 93% of digitised archive material loses its images on cutover. The archive descriptions survive, so the loss is silent: no error, just works with the viewer gone.
+
+## Impact
+
+The import reconnects digitised images for the large majority of the ~94,430 digitised archive works, and does it before cutover rather than as a recovery job after. Public work ids are unaffected throughout; they are preserved by the CALM RecordID predecessor mechanism (RFC 083) independently of this change.
+
+Risks and caveats:
+
+- The mapping is a snapshot. Records digitised between extraction and cutover need a repeat pass, or the b number capture needs to become part of the digitisation workflow until Sierra actually stops. The extraction is cheap to re-run. Relatedly, about 42,600 of the extracted pairs have no AxC record yet, dominated by CALM records committed after the migration extract; those arrive with the final delta migration load and need a second import pass, which the tooling's delta mode covers.
+- The coverage figures above were measured against the round-2 load of 17 August; a further load before cutover would need the same measurement repeated.
+- Import quality depends on the CALM RecordID surviving in AxC, which round-1 testing supports (201,072 matched ids, zero mismatches), but the import job should report unmatched rows rather than dropping them.
+- Digitised AV whose Sierra bib is not CALM-linked is out of scope and worse off: a METS record can never stand alone as a work, so if those bibs are migrated nowhere the works drop out of the catalogue entirely. That needs its own decision and owner.
+- Old Sierra work ids currently redirect to the archive work; after cutover they will 404. The CALM record always came first for these, so few such links should be in the wild, but it is a real behaviour change.
+
+## Next steps
+
+1. Generate the (CALM RecordID, b number) CSV from current Sierra data: done, first cut 17 August via wellcomecollection/catalogue-pipeline#3579 (tracked in [wellcomecollection/platform#6525](https://github.com/wellcomecollection/platform/issues/6525)).
+2. Agree the import route and format with collections staff: format agreed 17 August (`object_number,alternative_number,alternative_number.type`, matching on the public reference, the AltRefNo); Victoria Webb has confirmed importing b numbers into Axiell archive records is straightforward.
+3. Finalise the import list from the measured gap (6,999 of 210,846 records carry the field in the 17 August load).
+4. Run the import, re-harvest, and verify a sample of digitised archive works merge without their Sierra records.
+5. Cross-reference this RFC from RFC 091 and soften its "the Sierra work is always the target" wording to the library-material case.
+6. Find an owner for the non-CALM-linked digitised AV decision.

rfcs/README.md

@@ -74,6 +74,7 @@ _This is generated from the RFCs in this directory using `.scripts/create_table_
 
 | RFC ID | Summary | Next Line | Last Modified |
 |--------|---------|-----------|---------------|
+| [092-digitised-archive-links](092-digitised-archive-links/README.md) | RFC 092: Keeping digitised archive material connected after Sierra | Digitised archive material on wellcomecollection.org is connected to its archive description through the CALM-synced Sierra bib, and those bibs are deliberately not migrated to FOLIO. This RFC records how the connection works today, shows that the catalogue pipeline can maintain it with no Sierra or FOLIO record existing provided the Axiell Collections record cites the Sierra b number, quantifies the gap (roughly 93% of digitised archive records carry no b number in Axiell), and proposes importing the b numbers into Axiell Collections, matched on each record's public reference and verified through the CALM RecordID carried in the migration. It is the archive-side companion to RFC 091, which covers the library-side (FOLIO-target) half of post-migration digitisation merging. | 17 Aug 2026 |
 | [085-IIIF-Identities-and-Migration](085-IIIF-Identities-and-Migration/README.md) | RFC 085: Identifiers of and within IIIF resources after the migration | To record the agreed approach to IIIF resource identity across the LMS migration: preservation identifiers remain the canonical URIs for IIIF Manifests and Collections, for old and new content alike, with Work IDs acting only as redirect sources. The RFC retains the analysis that led to this decision — chiefly the behaviour of Canvas IDs and annotation targets — and sets out the consequences for the DDS: the redirect contract, the constraints on the form of new preservation identifiers, sub-package identity, and the persistence of Canvas identity. | 07 Aug 2026 |
 | [091-digitisation-ingest-identifiers](091-digitisation-ingest-identifiers/README.md) | RFC 091: Preservation identifiers across the LMS migration | Wellcome Collection is migrating its library management system from Sierra to Folio, and its archive management system from CALM to Axiell Collections. The Sierra b-number is embedded in storage locations, METS records, IIIF manifest URIs, and the join key that merges digitised content onto the public catalogue work. This RFC names that identifier role the **preservation identifier**, records the decision that preservation identifiers remain the canonical identifiers for digital objects (nothing is minted at ingest, and IIIF Manifest URIs do not move to the catalogue Work id), and sets out cross-migration and post-migration ingest paths covering both digitised and born-digital content. For items ingested after the migration the proposed preservation identifier is the Folio instance HRID, e.g. `in00012345`. | 07 Aug 2026 |
 | [089-identifiers-api](089-identifiers-api/README.md) | RFC 089: Identifiers API | This RFC proposes a small, read-only **Identifiers API** that resolves a **canonical** catalogue identifier to its **source** identifier(s) and back, served from the catalogue ID Registry (the same store the ID Minter writes to, per [RFC 083](../083-stable_identifiers/README.md)). It provides that translation in one place, between the canonical ids the public surface uses and the source ids (Sierra numbers, FOLIO UUIDs, CALM/Axiell refs) that the underlying systems require across the Sierra/CALM → FOLIO/Axiell migration. It sets out the contract, the AWS architecture, the authentication and cost model, the caching strategy, and what a working prototype has already established. | 05 Aug 2026 |