Publish JMH benchmark results to a gh-pages dashboard - #6608
Merged
line-o merged 2 commits intoAug 3, 2026
Conversation
exist-core-jmh and exist-indexes-jmh have no continuous tracking: numbers
only exist as one-off local runs pasted into PR descriptions. Port the
gh-pages benchmark-dashboard approach from BetaMasaheft/betmas-e2e
(benchmark-action/github-action-benchmark) so JMH results accumulate as a
time series instead.
Adds .github/workflows/ci-benchmarks.yml: builds and runs both JMH modules
weekly (and on manual dispatch) with reduced iteration counts suitable for
a shared runner (-wi 2 -i 5 -f 1 - a trend signal, not a publishable
number), then pushes each module's JSON output to gh-pages under
dev/bench/core and dev/bench/indexes via the jmh tool adapter. Report-only:
alerts on >150% regressions via commit comment, never fails the run.
Publish steps are guarded to github.repository == 'eXist-db/exist' so a
fork running this workflow doesn't attempt to push to someone else's
gh-pages. GitHub Pages itself still needs to be enabled in this repo's
Settings once the workflow has run at least once and created the gh-pages
branch.
Adds a "Continuous tracking" section to exist-indexes-jmh/README.md and
drops the roadmap item ("CI nightly job posting numbers to a tracking
issue") this supersedes.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Refs eXist-db#6387 PR eXist-db#6318 (util:expand Lucene term-rewrite cache, closes eXist-db#5738) originally included two perf-ratio assertions in surefire (UtilExpandHighlightingPerformanceTest); a later commit on that PR removed them per review - hard perf thresholds are flaky on shared CI runners, and the index-extension reactor had no benchmark module yet. eXist-db#6387 tracked migrating them to JMH. exist-core-jmh already depends on exist-index-lucene and hosts other Lucene benchmarks (LucenePhraseQueryBenchmark, ReindexDeleteStrategyBenchmark), so this adds UtilExpandHighlightingBenchmark there rather than standing up the separate extensions/indexes/ reactor eXist-db#6387 originally proposed - that gap it assumed no longer exists. Mirrors ArrowOperatorBenchmark's arrow*/direct* pairing: each shape has a *HighlightingOff / *HighlightingOn pair over an identical corpus and compiled query, so the dashboard shows both series and their trend rather than asserting a single ratio threshold. - expandSingleHit{Off,On}: single hit via subsequence($hits, 1, 1) - expandBatchWildcard{Off,On}: util:expand($hits) over ~half of a 5,000-entry corpus matched by a lemma:a* wildcard - the path the per-Query term-rewrite cache targets Each benchmark still asserts the expected hit count (a "fast but wrong" guard, not a performance threshold). Smoke-tested locally (-wi 1 -i 1 -f 1 -w 3s -r 3s): expandBatchWildcardHighlightingOff ~41ms/op vs HighlightingOn ~228ms/op, consistent with PR eXist-db#6318's measured ~4-5x. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
duncdrum
marked this pull request as ready for review
July 26, 2026 18:40
duncdrum
added
performance
dizzzz
approved these changes
Jul 27, 2026
Contributor
Author
|
In case anyone is wondering. The missing maven wrapper in core is inherited prior art. I will add one together with the reactor refactoring. But that touches upon #6530 s generation of conf.xml approach and is blocked by that landing first. |
Member
|
please could you elaborate on what is needed? |
Contributor
Author
|
Nothing is needed this can be merged as is. There is an inconsistency between the core and index benchmarks in how they are wired up in maven, and the maven reactor layout for benchmarks in general. Fixing these depends on codegen for conf xml. |
4 tasks
line-o
pushed a commit
that referenced
this pull request
Aug 17, 2026
exist-core-jmh and exist-indexes-jmh are gated behind the perf-tests Maven profile in the root pom.xml, not part of the default reactor. Every run of the JMH Benchmarks workflow (#6608) has failed since it merged, all three runs erroring identically at the first build step: [ERROR] Could not find the selected project in the reactor: exist-core-jmh Add -Pperf-tests to all three mvn invocations that select these modules via -pl (both `install -am` build steps, and the `exec:exec` step for exist-indexes-jmh). The `java -jar` step for exist-core-jmh needs no change - it invokes the already-built jar directly, no reactor lookup. Verified locally: `mvn install -Pperf-tests -pl exist-core-jmh -am` and `-pl exist-indexes-jmh -am` both build clean; `mvn exec:exec -Pperf-tests -pl exist-indexes-jmh -Dbenchmark.args="RangeEqWhereClauseBenchmark ..."` runs end to end. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
exist-core-jmh/exist-indexes-jmhJMH results to a gh-pages dashboard viabenchmark-action/github-action-benchmark, porting the approach from BetaMasaheft/betmas-e2e.util:expandhighlighting perf-ratio tests (originally in PR [optimize] util:expand: cache Lucene terms per query, skip empty-termMap scans #6318, removed from surefire per review) into a JMH benchmark, added to the existingexist-core-jmhmodule rather than the new reactor [ci/test] Add index-benchmark Maven reactor + migrate util:expand perf-ratio tests to JMH #6387 had proposed — that module already exists and already hosts other Lucene benchmarks.What changed
.github/workflows/ci-benchmarks.yml(new) — weekly + manual dispatch; builds and runs both JMH modules with reduced iterations (-wi 2 -i 5 -f 1); pushes-rf jsonoutput togh-pagesunderdev/bench/{core,indexes}. Report-only: alerts on >150% regressions, never fails the run.exist-core-jmh/README.md(new) /exist-indexes-jmh/README.md(updated) — document all benchmark classes and the new continuous-tracking workflow.UtilExpandHighlightingBenchmark.java(new) —expandSingleHit{Off,On}/expandBatchWildcard{Off,On}pairs (mirrorsArrowOperatorBenchmark's pairing style), each still guarding the expected hit count.Status
gh-pagesbranch created and GitHub Pages enabled: https://exist-db.github.io/exist/ is live (placeholder content until the workflow's first run).benchmarks/tree,exist-jmh-parent) is a separate future track with no issue filed yet — intentionally not part of this PR.Test plan
mvn install -pl exist-core-jmh -am -Pperf-tests -DskipTestsbuilds cleanUtilExpandHighlightingBenchmarklocally: batch highlighting off ~41ms/op vs on ~228ms/op (~5.6x), consistent with [optimize] util:expand: cache Lucene terms per query, skip empty-termMap scans #6318's measured rangeworkflow_dispatchrun pending (first real test of the JSON round-trip intogithub-action-benchmark)🤖 Generated with Claude Code