Skip to content

ci: test against the packages the eXist-db image bundles - #865

Open
joewiz wants to merge 1 commit into
eXist-db:developfrom
joewiz:ci/test-bundled-packages
Open

ci: test against the packages the eXist-db image bundles#865
joewiz wants to merge 1 commit into
eXist-db:developfrom
joewiz:ci/test-bundled-packages

Conversation

@joewiz

@joewiz joewiz commented Aug 19, 2026

Copy link
Copy Markdown
Member

[This PR was prompted by Joe, drafted by Claude Code, and reviewed by Joe.]

Problem

The workflow started the test container with:

docker run --volume $(pwd)/build:/exist/autodeploy:ro ... existdb/existdb:latest

A bind mount replaces the directory, so everything the eXist-db image ships in /exist/autodeploy was discarded and CI deployed exactly three packages: the freshly built eXide, plus the two the workflow downloaded — roaster-1.12.0 and existdb-openapi-0.9.5.

That is not the stack anyone runs. existdb/existdb:latest bundles:

dashboard-2.0.9.xar                      monex-5.1.0.xar
eXide-4.0.1.xar                          packageservice-1.3.14.xar
exist-documentation-7.0.0.xar            roaster-1.12.2.xar
exist-function-documentation-3.0.1.xar   semver-xq-3.0.0.xar
existdb-openapi-0.10.0.xar               templating-1.2.1.xar
functx-1.0.1.xar

eXide's own README tells contributors to docker cp ./build/*.xar exist-ci:exist/autodeploy, which layers onto that set — so the documented local procedure and CI have been testing different stacks.

The pins are now actively broken

As of today's existdb/existdb:latest, the pinned roaster-1.12.0 cannot serialize eXide's login response, so POST /api/auth/session returns:

400  Error while serializing xml: org.xml.sax.SAXException:
     err:SENR0001 Cannot serialize a map(*) with the XML or text output method

Every spec calls cy.loginXHR first, so the whole suite fails — 100% of specs, on any branch. Isolated locally against the same image, with only the Roaster version changed:

eXide Roaster existdb-openapi POST /api/auth/session
develop 1.12.0 (the pin) 0.9.5 (the pin) 400
develop 1.12.2 (bundled) 0.9.5 (the pin) 200
develop 1.12.2 (bundled) 0.10.0 (bundled) 200

develop's last green run predates this; the next push to develop goes red. This PR is the fix.

Why the divergence mattered before that

This is also what made eXide#821 unresolvable for two months. The pinned existdb-openapi-0.9.5 predates the /api/query error-envelope regression, so CI stayed green while every user on a stock eXist-db 7 hit a query error eXide could not display. Reproduced on a clean container, same eXist-db build, same query:

existdb-openapi /api/query response to 1 + "oops" query-error specs
0.9.5 (the pin) HTTP 500 + { code, line, column, description } pass
0.9.7 (shipped with 7.0.0-beta3) HTTP 200 + { error: "Invalid context-item: …" } fail
0.10.0 (current) HTTP 400 + { code, message, line, column, raw } fail without #840

The pins also block PRs: eXide#814 requires Roaster >= 1.12.1 and cannot pass on a bed that supplies 1.12.0.

Change

Assemble the autodeploy set from the image's own packages, with only eXide swapped for the build under test:

CID=$(docker create existdb/existdb:latest)
docker cp "$CID":/exist/autodeploy/. autodeploy/
rm -f autodeploy/eXide-*.xar
cp build/eXide-*.xar autodeploy/

The two gh release download steps go away with the pins. Copying the set out and mounting it (rather than docker cping the xar in) keeps the deployment deterministic: exactly one eXide, even across a version bump.

Merge ordering

CI is red right now regardless of this PR, so this is about the shortest path back to green rather than about avoiding a regression.

Merge eXide#840 first, then this. Against the bundled existdb-openapi 0.10.0, query_error_structured_spec fails on current develop — the real incompatibility this PR stops hiding, and #840 is its fix. With #840 in, this PR takes the suite to fully green on the stack users actually run.

Verification

Full 39-spec suite, clean container each run, existdb/existdb:latest (7.0.0-SNAPSHOT, build 2026-08-18):

  • develop + bundled packages (openapi 0.10.0): 1 failing — query_error_structured, per the ordering note above
  • develop + eXide#840 + bundled packages: query-error specs pass
  • develop + the old CI pins (roaster 1.12.0, openapi 0.9.5, no other apps): all 38 specs fail at login, per the table above

The workflow started the container with `--volume $(pwd)/build:/exist/autodeploy`.
A bind mount replaces the directory, so every package the image ships was
discarded and CI deployed only three: the freshly built eXide, the pinned
roaster-1.12.0, and the pinned existdb-openapi-0.9.5.

That is not the stack anyone runs. The image bundles a newer existdb-openapi and
Roaster plus dashboard, monex, functx, the documentation apps, packageservice,
semver-xq and templating; eXide's own README documents `docker cp`ing the xar
into the container, which layers onto that set. The divergence hid a real
regression for weeks (eXide#821): the pinned 0.9.5 predates existdb-openapi's
/api/query error-envelope regression, so CI stayed green while every user hit a
query error that eXide could not display.

Assemble the autodeploy set from the image's own packages with only eXide
swapped for the build under test. Dropping the two `gh release download` steps
also removes the pins, which is what forced a roaster >= 1.12.1 PR (eXide#814)
to fail on a bed that supplied 1.12.0.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@duncdrum

Copy link
Copy Markdown
Contributor

@joewiz yes this is a know problem of all our Ci configs. Luckily that means there is plenty of prior art for how to approach this.

The basics principles are:

  1. expath dependencies are declared in expath-pkg.xml CI results must show that the deps declared there are the ones the app runs correctly with. (If it doesn't they have to be adjusted)
  2. A second matrix should test with whatever ships by default with exist, as that's of most relevance to our users.

Using a volume with just the deps declared in expath-pkg ensures 1. But requires determining deps on CI either manually (what we have here) or by parsing expath-pkg.xml inside the workflow.

For 2. one can just drop the whole volume from CI, but then one has to ensure that the app under test is actually the xar being compiled, and not the one inherited from the base image.

using -slim images and xst install is another approach for small and medium apps.

None of these solutions are great, they are all hacks around the shortcomings of auto deploys and its lack of dependency resolution.

Extracting the xars from the image is not the right approach imv, it fails to satisfy 1. and it's just a long-winded way to tackle 2.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants