XSD 1.1 store-time document validation + Saxon catalog resolver by duncdrum · Pull Request #6530 · eXist-db/exist

duncdrum · 2026-06-26T12:05:42Z

Summary

Configures Saxon (fn:transform() / transform:transform()) to resolve xsl:import and document() URIs via the system catalog — fixes issue Feature request: configure saxon to use webapp/WEB-INF/catalog.xml #350 for the XSLT transform path
Validates documents against their XSD 1.1 schemas at store time, routing through the XSD 1.1-capable pipeline when the stored document's root namespace resolves to one via the system catalog
Routes org.exist.validation.Validator through the same XSD 1.1 detection
Auto-generates SchemaVersion.java constants from each XSD's xs:schema/@version at build time
Removes a spurious WARNING: Failed to load catalog message when no default catalog exists

What changed

Area	Files
Saxon catalog resolver	`SaxonConfiguration.java`, `Transform.java`, `URIResolution.java`
XSD 1.1 store-time validation	`Xsd11ValidationHelper.java` (new), `Xsd11SchemaCache.java` (new), `Xsd11SchemaDetection.java` (new)
`Validator` routing	`Validator.java`, `ValidatorXsd11Test.java` (new)
Catalog warning suppression	`ResolverFactory.java`
SchemaVersion codegen	`schema/generate-schema-version.xsl`, `exist-core/pom.xml` `schema-version-codegen` execution
Tests	`Xsd11StoreTimeValidationTest.java`, `catalog.xql`

Xsd11SchemaCache (Caffeine-backed) and Xsd11ValidationHelper were split out of MutableCollection per review — that logic is validation/schema infrastructure, not Collection state.

Related issues

Closes [BUG] Jaxp does not let you validate XML Schema validity #5541 (validation:jaxp() now reports invalid XSD 1.1 schemas instead of silently misbehaving)
Closes part of Feature request: configure saxon to use webapp/WEB-INF/catalog.xml #350 (Saxon catalog resolver for xsl:import/document(); the fn:transform() + stylesheet-node case tracked separately, not fixed here)

Test plan

Xsd11StoreTimeValidationTest, ValidatorXsd11Test — XSD 1.1 documents validate correctly at store time and via Validator
catalog.xql — transform:transform() resolves xsl:import via catalog
Full validation/XSD11 suite (63 tests across 14 classes) — pass after the Xsd11SchemaCache/Xsd11ValidationHelper extraction
license:check, schema governance, SchemaVersionSyncTest — all pass
CI green (Test & documentation + XQTS)

Notes

PR-C of a 5-PR split from #6505 (superseded — needs closing separately). #6528 is merged; this branch's diff now shows only its own commits. Next in merge order: #6531 → #6550 → #6551.

🤖 Generated with Claude Code

line-o

OK - stale comment is not a blocker

@Version

…antic changes Add a mvn validate execution (xml:transform@schema-governance) that catches semantic edits to the seven native XSD schemas without a matching @Version bump; bump all seven XSDs to reflect post-2023 semantic changes. Closes eXist-db#6190

…consolidate governance CI Add schemaVersion="x.y.z" to all five canonical config instances so administrators can see at a glance which XSD version a config file targets, and so startup code can log a DEBUG or WARN when it doesn't match. Consolidate the four separate governance shell scripts into a single governance.xsl, making the version-bump check fully Maven-native. Add SchemaVersionSyncTest to guard the generated constants against drift.

Include schema/ (conf.xsd, collection.xconf.xsd, descriptor.xsd, controller-config.xsd, mime-types.xsd, users.xsd, server.xsd, security-manager.xsd, expath-pkg.xsd and its extensions) in the tarball, zip, Docker image, IzPack installer, and macOS .app bundle as $EXIST_HOME/schema/. Also fix the "abrev" -> "abbrev" typo in expath-pkg.xsd. Closes eXist-db#6189 Partially addresses eXist-db#6008

dizzzz

added some comments.

duncdrum · 2026-07-13T14:17:56Z

MutableCollection.java (Namespaces / Caffeine / separate class)
The namespace constants are fixed in f7f3463460: XML_SCHEMA_NS was dead code (dropped), XSD_1_1_NS is now Namespaces.XSD_1_1_NS. For the schema cache and XSD operations: agreed they'd sit better in a dedicated class — deferred to a follow-up PR so this chain can land first.

JettyStart.java:125
Split to standalone #6572. Only JettyStart.java was touched in that commit; it's independent of the XSD work.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

@Version

…generate SchemaVersion constants Extend the Maven validate phase to check every schema/*.xsd against the W3C XSD 1.1 meta-schema (upgrading the bundled XMLSchema.xsd/dtd to the 2009 XSD 1.1 revision); caught and removed five xsi:type="dcterms:W3CDTF" appinfo annotations with no backing schema, bumping the affected schema @Version values. Wire a generate-sources execution that reads each governed XSD's xs:schema/@Version and emits SchemaVersion.java constants so they can never drift from the schemas. Closes eXist-db#5541

…line Wire the XML Catalog API into Saxon's XsltCompiler so xsl:import, xsl:include, and document() URIs resolve via the eXist-db entity catalog (Closes eXist-db#350). Generalize MutableCollection's at-store-time validation to route through a JAXP 1.1 SchemaFactory when the schema or its catalog- resolved meta-schema requires XSD 1.1; extract the detection logic into Xsd11SchemaDetection so validation:jaxp() and org.exist.validation.Validator share the same probe-compile path. Suppress the spurious xmlresolver WARNING for the non-existent "default" catalog entry. Closes eXist-db#5541

… XML_SCHEMA_NS XML_SCHEMA_NS was never read — callers already used XMLConstants directly. XSD_1_1_NS is promoted to Namespaces.XSD_1_1_NS so it is available codebase-wide without duplication. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

dizzzz

I have some open remarks still

line-o · 2026-07-20T19:59:05Z

@duncdrum could you have a look please

…f MutableCollection MutableCollection carried a schema cache (plain ConcurrentHashMap) plus the namespace-resolution/validation logic that uses it -- neither is Collection state. Splits this into two package-private classes: - Xsd11SchemaCache: the per-namespace "needs XSD 1.1?" cache, now backed by Caffeine for consistency with the codebase's other caches (no eviction policy, since the namespace set is finite and admin-controlled -- this was never a correctness concern, just a style/testability one). - Xsd11ValidationHelper: resolveXsd11SchemaForNamespace, the dynamic-discovery schema holder, parseOrValidateXmlSource, validateWithXsd11Schema, and the Xsd11LexicalHandlerForwarder record -- store-time XSD 1.1 validation, not Collection bookkeeping. MutableCollection keeps only its public clearXsd11SchemaByNamespaceCache() facade (GrammarTooling.java's external call site) and a single delegating call from storeXmlDocument(); 9 imports that only the moved code needed are dropped. Behavior-preserving: all 63 tests across the validation/XSD11 suite pass unchanged. Addresses dizzzz's review comments on eXist-db#6530/eXist-db#6551.

@Version

…reated edit 4e949b4 dropped xsi:type="dcterms:W3CDTF" from the dcterms:created annotation in 5 sibling schema files and correctly bumped @Version on 4 of them (collection.xconf.xsd, cxan.xsd, exist.xsd, repo.xsd) -- this one was missed, leaving expath-pkg.xsd's content changed but @Version still 1.1.0. CI's schema governance check (ci-schema-checks.yml, added in PR-A) catches exactly this. Bumps to 1.1.1, a documentation/non-semantic patch per schema/README.md's policy, matching the sibling files' bumps. Verified locally against the real merge-base with upstream/develop: mvn -N xml:transform@schema-governance passes, SchemaVersionSyncTest passes unaffected (expath-pkg.xsd isn't one of the 5 SchemaVersion.java-tracked schemas).

duncdrum · 2026-07-26T18:56:20Z

@line-o @dizzzz all comments answered

dizzzz · 2026-07-30T13:14:21Z

+ */
+final class Xsd11SchemaCache {
+
+    private static final Cache<String, Optional<Schema>> CACHE = Caffeine.newBuilder().build();


Great! we might want to define a max time of caching or a max amount of cached items to prevent memory leaks.

Good instinct in general, but I don't think it applies here: the key space is the system catalog's namespace list, which is fixed for the life of the running JVM — same as any other config eXist loads at startup/broker-pool-init. A catalog change needs a restart to take effect anyway, and that restart clears this cache along with everything else. So there's no unbounded growth path, just a cache that tops out at "however many namespaces the catalog declares" and stays there. Not adding an eviction policy for a ceiling that's already fixed by config, but happy to revisit if that assumption changes.

dizzzz

LGTM

dizzzz · 2026-07-30T13:24:01Z

I am still a bit confused/worried why we need to add these XSDs/DTDs ourself, als they are explicitly available in the xmlresolver.org -data package which we already bundle. @duncdrum @line-o

`resolveXsd11SchemaForNamespace` use a plain Schema local and wrap once, at the `Xsd11SchemaCache.put()` call site where the cache's Optional-typed value (needed because Caffeine can't cache null) actually matters.

duncdrum · 2026-08-17T11:04:32Z

@dizzzz XML Schema language's own XSD/DTD have been vendored in this repo since before XSD 1.1 support existed. We switched our own files to xsd 1.1 but kept shipping xsd 1.0 schema lang files. My guess is that the vendoring was specifically so schema validation doesn't have a hard runtime dependency on an external package.

Eliminating the vendored copies entirely and resolve them from xmlresolver.org-data instead, is a bigger call about a load-bearing dependency for core validation that touches validation during compile of our own files and of validation from inside xquery in the running instance, including catalog resolving from xsl, type checking etc. We can do this in a follow-up.

For now I would suggest to stick with the old approach, while introducing the new feature, and to get rid of conflicting versions. Switching the source of the files should be a patch release. It's a lot of code without functional changes. In other words quite a bit of work, for very little gain. I m more concerned about the new functionality and features to hit develop for testing and feedback.

dizzzz · 2026-08-17T15:45:54Z

Agreed

duncdrum requested a review from a team as a code owner June 26, 2026 12:05

This was referenced Jun 26, 2026

Generate test conf.xml/controller-config.xml fixtures from canonical at build time #6531

Merged

Ship versioned native XSD schemas with governance CI #6528

Merged

Generate SchemaVersion constants from XSDs; validate XSDs against W3C meta-schema #6529

Closed

line-o reviewed Jun 29, 2026

View reviewed changes

Comment thread exist-core/src/main/java/org/exist/xquery/functions/fn/transform/Transform.java

line-o requested a review from a team June 30, 2026 18:16

This was referenced Jul 5, 2026

[Dev6] fix XSLT imports #6544

Merged

[feature] Add @enabled attribute to conf.xml elements to disable entries without removing them #6550

Merged

[feature] Add ModuleFactory and IndexFactory SPI for auto-discovery of bundled modules and indexes #6551

Open

line-o approved these changes Jul 7, 2026

View reviewed changes

line-o requested a review from a team July 7, 2026 07:15

duncdrum force-pushed the dp-xsd11-validation branch from 494496f to 798040e Compare July 7, 2026 09:25

duncdrum force-pushed the dp-xsd11-validation branch from 798040e to 1e02258 Compare July 7, 2026 13:10

duncdrum added 2 commits July 7, 2026 15:32

duncdrum force-pushed the dp-xsd11-validation branch 2 times, most recently from bb51a30 to a5fb068 Compare July 7, 2026 13:58

dizzzz requested review from dizzzz and reinhapa July 12, 2026 16:14