OPENNLP-1886: UniNE light and minimal stemmer tiers with vocabulary parity fixtures - #1166
Draft
krickert wants to merge 4 commits into
Draft
OPENNLP-1886: UniNE light and minimal stemmer tiers with vocabulary parity fixtures#1166krickert wants to merge 4 commits into
krickert wants to merge 4 commits into
Conversation
krickert
changed the title
krickert
changed the title
krickert
force-pushed
the
light-stemmers
branch
7 times, most recently
from
July 19, 2026 11:41
e31be7c to
fdbe065
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 20, 2026
… map; cite per-branch pinning tests apache#1163 (OPENNLP-1883, the thread-safe stemmer + StemmerFactory seam) merged to apache main; move it into the merged-upstream node, repoint light-stemmers (apache#1166) and hunspell onto main since their seam dependency now ships upstream, and drop the stale open-PR row. Also lands the previously uncommitted skill-pass enrichment: each branch row now names the UsageExample/ManualExample test that pins its manual listing.
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
…est head All nine open heads now sit directly on a864230 and report mergeable. Two were reporting conflicts and both cleared: apache#1167 through a plain rebase, and apache#1152 by pointing its apache-hosted sentencepiece base branch at the refreshed head it had drifted away from, which shrinks its diff back to the 30 commits it owns. apache#1166 shed the 13 OPENNLP-1883 commits it carried, since apache#1163 is upstream as a single squash, and is 3 commits now. Also correct what the draft flag on apache#1182 means: the branch is review-ready and waits on the upstream queue, not on unfinished work.
…ry parity fixtures Sixteen stemmers in the new opennlp.tools.stemmer.light package: light and minimal tiers for German, French, Spanish, Norwegian (Bokmaal and Nynorsk varieties), Swedish, plus light stemmers for Finnish, Hungarian, Italian, Portuguese, Russian and the English minimal stemmer. Adapted from Apache Lucene's analysis-common module; the UniNE algorithms carry Jacques Savoy's BSD notice in each source file and in the distribution LICENSE. These fill the tier between no stemming and the aggressive Snowball algorithms. Every stemmer is stateless and thread-safe, implements Stemmer, and is its own StemmerFactory. Input is expected lowercase, matching the algorithms' original contract; null input fails loudly. Parity is asserted against the original implementations: bundled fixtures sample the algorithms' vocabulary test data (up to 2000 word/stem pairs per stemmer, all pairs for the small Norwegian lists) regenerated by running the Lucene classes, so any behavioral drift fails the test. The Galician and Portuguese minimal stemmers are excluded because they build on the RSLP rule engine, which is its own effort.
…amples Extend docbkx/stemmer.xml with the light and minimal stemmer tiers, wire the chapter into the manual, and add StemmerFactoryUsageExampleTest and LightStemmerUsageExampleTest asserting every value the chapter prints. Also condense the StemmerFactory javadoc to the contract.
…ment with the review conventions
…ighten javadoc
- Move the varargs argument validation of the Norwegian stemmers into a new
package private NorwegianVariety.toSet(first, more) helper that null checks
first, more and every element, then returns an EnumSet of the selection.
- Have NorwegianLightStemmer and NorwegianMinimalStemmer delegate to that
helper, which drops the duplicated boolean accumulation loop from both
constructors while keeping the documented IllegalArgumentException contract.
- Replace the bare {@inheritdoc} on newStemmer() in all sixteen light and
minimal stemmers with a note that the call returns the same instance because
the stemmers are thread-safe, so the factory behaviour is not a surprise.
- Correct the FrenchMinimalStemmer class javadoc, which described it as a light
stemmer, and add the missing sentence periods on the PortugueseLightStemmer
and SpanishLightStemmer headings.
- Rewrite the SwedishMinimalStemmer class javadoc so the listed endings match
what stem() actually strips: -are in addition to -ar/arne/arna/aren, the
genitive -s, and a trailing -a, -e or -n.
- Pin the null varargs array case for NorwegianLightStemmer in
LightStemmerContractTest, which the existing null element test did not cover.
- Use assertSame instead of assertEquals for the newStemmer() identity check in
LightStemmerParityTest, since equality would pass for any two instances.
- Assert that a parity fixture line contains a tab before splitting it, so a
malformed fixture reports the offending file and line instead of failing with
a StringIndexOutOfBoundsException.
- Drop the redundant public modifier from LightStemmerUsageExampleTest to match
the package private convention of the other tests.
- Add the Jacques Savoy BSD notice to LICENSE covering the UniNE algorithms in
opennlp.tools.stemmer.light, matching the per file headers.
- Describe the light stemmer instances in stemmer.xml as immutable rather than
stateless, and rewrap the surrounding thread-safety paragraph.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
STACKED on #1163 (stemmer-factory): this branch builds on that PR's StemmerFactory contract, and since that head lives on a fork it cannot be this PR's base ref. Until #1163 merges, the diff here shows its commits too; only the last commit (fdbe065) is this PR. After #1163 lands, the branch rebases onto main and the diff collapses to this change alone.
Adds sixteen stemmers in a new opennlp.tools.stemmer.light package: light and minimal tiers for German, French, Spanish, Norwegian (Bokmaal and Nynorsk varieties), and Swedish, plus light stemmers for Finnish, Hungarian, Italian, Portuguese, and Russian, and the English minimal stemmer. These fill the tier between no stemming and the aggressive Snowball algorithms: they remove plural and inflectional endings with far less conflation. Adapted from Apache Lucene's analysis-common module; the UniNE algorithms carry Jacques Savoy's BSD notice in each source file and in the distribution LICENSE.
Every stemmer is stateless and thread-safe, implements Stemmer, and is its own StemmerFactory. Input is expected lowercase, matching the algorithms' original contract; null input fails loudly.
Parity is asserted against the original implementations: bundled fixtures sample the algorithms' vocabulary test data (up to 2000 word/stem pairs per stemmer, complete lists for Norwegian) regenerated by running the originals, so any behavioral drift fails the test. The Galician and Portuguese minimal stemmers are excluded because they build on the RSLP rule engine, which is its own effort.