OPENNLP-1887: Lexical expansion over the knowledge base seam: synonyms, hypernyms, optional hyponyms - #1167
Draft
krickert wants to merge 12 commits into
Draft
OPENNLP-1887: Lexical expansion over the knowledge base seam: synonyms, hypernyms, optional hyponyms#1167krickert wants to merge 12 commits into
krickert wants to merge 12 commits into
Conversation
krickert
changed the title
krickert
changed the title
krickert
force-pushed
the
wordnet-expansion
branch
6 times, most recently
from
July 17, 2026 20:45
5db6a8a to
917e4be
Compare
krickert
force-pushed
the
wordnet-expansion
branch
2 times, most recently
from
July 19, 2026 11:41
1b90059 to
f4cc90c
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
…ENNLP-1895 recorded Restate the map against apache main a864230, cut as 3.0.0-M5 on 2026-07-24. apache#1177 (OPENNLP-1870) merged upstream and moves into the merged box, apache#1190 and apache#1191 are marked ready for review, and OPENNLP-1895 (quantized embedding tables) joins the diagram in its own colour: filed in JIRA with the pull request deliberately held until apache#1165 and apache#1152 move. Statuses now carry the measured GitHub draft flag and how far each head sits behind main, which surfaces three things the old text did not: apache#1182 is a draft again, apache#1167 is based on main rather than on apache#1155 and carries the seam and isBlank commits as copies, and apache#1152 reports conflicts only because its apache-hosted sentencepiece base has diverged from the refreshed head.
krickert
force-pushed
the
wordnet-expansion
branch
from
July 24, 2026 19:27
ff066b0 to
b1aef66
Compare
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
…est head All nine open heads now sit directly on a864230 and report mergeable. Two were reporting conflicts and both cleared: apache#1167 through a plain rebase, and apache#1152 by pointing its apache-hosted sentencepiece base branch at the refreshed head it had drifted away from, which shrinks its diff back to the 30 commits it owns. apache#1166 shed the 13 OPENNLP-1883 commits it carried, since apache#1163 is upstream as a single squash, and is 3 commits now. Also correct what the draft flag on apache#1182 means: the branch is review-ready and waits on the upstream queue, not on unfinished work.
krickert
added a commit
to ai-pipestream/opennlp
that referenced
this pull request
Jul 24, 2026
The clean-room CFSA2/FSA5 readers and the PoliMorf lemmatizer had no node because until today they had no branch: they were written on the OPENNLP-1887 line under formats: and lemmatizer: titles and were only visible as a sentence in that pull request's notes. They now sit on their own branch off current main and get a node and a row like every other staged feature. The note on apache#1167 says what is actually true: the commits were copied out, and dropping them from that branch is a separate rebase that has not happened.
krickert
force-pushed
the
wordnet-expansion
branch
from
July 28, 2026 15:15
e8b4470 to
89a5a27
Compare
…s and a Morphy lemmatizer Adds the LexicalKnowledgeBase contract in opennlp.tools.wordnet and the opennlp-wordnet module implementing it twice: WnLmfReader for WN-LMF XML and WndbReader for the legacy WNDB database files, with reader-equivalence coverage over miniature fixtures of both formats. The WN-LMF reader skips DOCTYPE declarations unresolved with DTD support off, so Open English WordNet releases parse unmodified while entity expansion stays closed; the WNDB fixtures are pinned to LF so their embedded byte offsets survive Windows checkout. The Morphy lemmatizer resolves inflected forms through suffix rules and the format's exception lists. Null arguments fail loudly with IllegalArgumentException, malformed data raises the checked InvalidFormatException, and the public seam carries no brand name: WordNet stays in the names of the classes that actually read WordNet formats.
…nsion over the lexical knowledge base Expands a term into the synonyms sharing its synsets, the lemmas of its hypernym ancestors up to a configured depth (following both the direct and the instance relation), and optionally its direct hyponyms. Each expansion carries a deterministic heuristic weight: sense rank and every relation step multiply configurable decays, so consumers can discount looser expansions instead of treating them as the original term. Results exclude the input, deduplicate case-insensitively keeping the highest weight, and order stably by weight, kind, and term. Inflected input works through an optional Lemmatizer invoked with the WordNetPos name as the tag, which the Morphy lemmatizer understands: dogs expands through dog, mice through the exception list to mouse. Hypernym walks are visited-checked so malformed cyclic data terminates. Tests cover the graph behavior on a hand-built lexicon (sense ranking, decay arithmetic, dedup, cycle termination, validation) and run the whole stack over the miniature WN-LMF and WNDB fixtures, asserting both readers produce identical expansions.
… definition A blank check under the toolkit's whitespace definition, which unlike String.isBlank covers the no-break spaces, so annotators validating labels and identifiers share one predicate instead of each carrying a private copy. Reads whole code points; tests pin the no-break and figure spaces, the empty string, and a supplementary-plane letter.
…yping SynsetSimilarity scores noun synset pairs with the path, Wu-Palmer, and Leacock-Chodorow measures over the knowledge base seam. HypernymTyper labels a word by walking its senses' hypernym and instance-hypernym chains to the nearest caller-registered anchor concept, so a knowledge base can type names as person, organization, or location without a model. Blank checks follow the toolkit whitespace definition.
…examples Add the lexical expansion section to docbkx/wordnet.xml and LexicalExpansionUsageExampleTest asserting the expansion values the chapter prints; carry the WordNet usage example alongside it.
…ryLemmatizer Add PoliMorfDictionaryReader, adapting a surfaceForm/lemma/tag table (the layout PoliMorf publishes under BSD 2-Clause and morfologik text export produces) into the existing DictionaryLemmatizer: re-order columns, lower-case forms to match the case-insensitive lookup, and merge alternative lemmas per form and tag. Pure JVM, no carrot2 dependency; dictionary data stays caller-supplied. Manual section and mirror-tested example.
Read the morfologik CFSA2 automaton format (BSD) with no dependency on that library and enumerate the accepted byte sequences: header, flags and label-table parsing, v-coded targets, next-arc and target-next resolution, and terminating-node stop. Validated against a ground-truth automaton generated by morfologik's own serializer. This is the finite-state layer under bring-your-own morfologik-format dictionaries; decoding sequences into form/lemma/tag entries follows.
…izer Add MorfologikDictionaryReader: enumerate a CFSA2 automaton, split each sequence on the separator into form/encodedBase/tag, and decode the base form for all four morfologik encoders (NONE, SUFFIX, PREFIX, INFIX; control bytes offset by 'A'). Reads metadata from an explicit charset/encoder/separator or from a .info stream. Clean-room, no carrot2 dependency; dictionary data caller-supplied. Decode cases and the end-to-end CFSA2 fixture were generated by morfologik's own encoder and serializer.
Add FSA5Reader (the older fixed-goto morfologik automaton format) and an FsaSequenceReader interface whose read() dispatches on the version byte, so the morfologik dictionary reader now accepts both FSA5 and CFSA2. Validated against an FSA5 ground-truth automaton and an end-to-end FSA5 dictionary from morfologik's own serializer. Manual section documents reading morfologik dictionaries.
Walk the automaton with a single growable byte buffer, pushing each arc label on descent and popping on return, so only accepted sequences allocate. The prior code copied the whole prefix at every arc, internal ones included. Uses plain byte[] and int offsets, no ByteBuffer, keeping the reader allocation-light on large dictionaries.
…ment with the review conventions
…ck, and pinning tests - Document every private helper, constructor and enum constant in CFSA2Reader, FSA5Reader, GrowableByteSequence and MorfologikDictionaryReader, so no member of the readers is left undocumented. - Standardize the exception documentation on the toolkit phrasing, "@throws X Thrown if ...", across the formats readers, LexicalExpander and its builder. - Correct two javadoc statements that contradicted the code: depthFromRoot in SynsetSimilarity measures the distance to the farthest ancestor, not the shortest way up, and MorfologikDictionaryReader lower-cases surface forms on load because DictionaryLemmatizer lower-cases the queried token. - Drop the "thread safety is implementation specific" boilerplate from CFSA2Reader, FSA5Reader and PoliMorfDictionaryReader and state the actual guarantee instead: the instances hold only immutable state, so enumeration may run concurrently. - Fold the duplicated magic-header check of the two readers into a shared FsaSequenceReader.requireFsaHeader, and express MAGIC as the string "\fsa" rather than a mutable byte array on an interface. - Report a truncated header separately from a bad magic header, so a short but otherwise valid CFSA2 or FSA5 file no longer reports the wrong reason. - Rename gtl in FSA5Reader to gotoLength, name the packed header byte it is decoded from, and delete the unused FLAGS_MASK constant. - Extract FIELD_SEPARATOR and LEMMA_SEPARATOR constants in MorfologikDictionaryReader, pin the encoder lookup to Locale.ROOT, and remove the redundant head local in join3. - Adopt the toolkit argument-message style in LexicalExpander, dropping the leading article and the trailing period from every validation message. - Validate the synset identifiers in leacockChodorow, which alone among the SynsetSimilarity measures skipped the check and computed a result for a null identifier. - Replace the parallel visited set in nearestAnchor with the depths map that already tracks the same synsets, so a single map bounds the walk over cyclic data. - Trim the three-line commentary on the underflow guard in expandSense to one line stating the reason. - Share the fixture taxonomy from SynsetSimilarityTest instead of building a second copy in HypernymTyperTest, and drop the typer tests SynsetSimilarityTest duplicated now that HypernymTyperTest covers them. - Reuse the shared ExpansionAssertions.find in LexicalExpansionUsageExampleTest rather than its own private copy of the same helper. - Turn the eight-way Expansion validation test into a parameterized test, and add pinning tests for the accepting case, for the remaining null-argument paths of wuPalmer, shortestDistance and leacockChodorow, and for unknown synsets being reported as unrelated rather than fatal. - Use assertFalse instead of a negated assertTrue, import ArrayList rather than naming it fully qualified, and make the wordnet test classes public in line with the rest of the suite.
krickert
force-pushed
the
wordnet-expansion
branch
from
July 31, 2026 10:38
89a5a27 to
840d4fd
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
STACKED on #1155 (wordnet-api): this branch builds on that PR's LexicalKnowledgeBase seam, and since that head lives on a fork it cannot be this PR's base ref. Until #1155 merges, the diff here shows its commits too; only the commits from 1404a50 onward are this PR. One of them (082592d) is a carried copy of the OPENNLP-1888 StringUtil.isBlank utility this code uses; it drops out automatically once that lands. After #1155 lands, the branch rebases onto main and the diff collapses to this change alone.
Adds LexicalExpander to the opennlp-wordnet module: given a term, it produces weighted related terms from a LexicalKnowledgeBase, the synonyms sharing its synsets, the lemmas of hypernym ancestors up to a configured depth (following both the direct and the instance relation, visited-checked so cyclic data terminates), and optionally direct hyponyms. Each expansion carries a deterministic heuristic weight (sense-rank decay times depth decay, both configurable), results are deduplicated case-insensitively keeping the highest weight, the input term is never returned, and ordering is stable. An optional Lemmatizer fallback expands inflected input; the Morphy lemmatizer of this module plugs in directly, so dogs expands via dog and mice via mouse.
Tests cover the behavior over a hand-built graph with a controlled shape (sense ranking, depth decay, dedupe, cycle termination, hyponym opt-in, configuration validation) plus integration over the miniature WN-LMF and WNDB fixtures asserting both readers expand identically.