OPENNLP-1894: Add dictionary-based tokenization for Japanese, Korean, and Chinese - #1191

Common-prefix lookup walks a trie built at load time instead of probing
substrings per length, terminating on the first missing prefix and
allocating nothing per position.

(cherry picked from commit e10ce4b)

A Viterbi search maximizing summed word log-probabilities segments
Chinese and similar scripts from a plain word-count lexicon, with
unlisted characters falling back to single-character words. The user
supplies the lexicon and thereby accepts its license; nothing is
bundled.

(cherry picked from commit bff3f23)

…nigram segmenters, corrected javadoc

…s, add an EUC-JP loading example

…ng download helper

…e their category run, and validate context ids at load

…expressible dictionary values

The lattice tokenizer rescanned the same-category run from every position, so
a run of L characters cost on the order of L squared category lookups; a
16,000-character katakana run measured around half a second. One right-to-left
pass per stretch now fixes every position's category and run end, and the same
16,000-character run tokenizes in about half a millisecond at 31 million
characters per second. The character table holds Category instances instead of
names, so the per-character path compares by identity with no name-map lookup,
and a char.def mapping to an undefined category now fails at load naming the
code point. The lexicon trie's children are sorted character arrays found by
binary search, so a descent no longer boxes a Character per step. matrix.def
loading rejects connection costs outside the 16-bit range instead of silently
truncating them, and dimension products beyond the addressable array size fail
at the header. The unigram segmenter's unknown-character fallback advances one
code point, never one code unit, so an unknown supplementary character is
stepped over whole and no span can split its surrogate halves.

…recoded labels

The lexicon trie's per-node child lookup, a binary search over the node's
fan-out, paid about a dozen comparisons at the root of a real dictionary; the
classic base/check double-array makes every transition one array read and one
comparison. Characters are recoded into dense labels ordered by descending
frequency before the array is built, so the array stays compact although CJK
surfaces draw on tens of thousands of distinct characters, and a character the
lexicon never uses misses in the recode table before the array is consulted.
On the IPADIC harness the prefix walk now matches the fastest previous
implementation at 5.6M chars/s with strictly constant-time transitions, and
building the array adds about a quarter second to the 392k-entry load.

…er-position lists

The Viterbi lattice held one ArrayList per text position plus one fresh
candidate list per position, pure allocation churn on long stretches. Nodes
ending at a position now chain through their own link field behind a single
head reference per position, and candidate gathering fills one scratch list
reused across positions, so building the lattice allocates nothing besides the
nodes themselves. IPADIC throughput on the 400k-character harness rises from
5.6M to 6.5M characters per second with identical output.

…example

Add a lattice tokenizer section to the manual citing LatticeUsageExampleTest.

…lexicon accessors

…e tokenizer overrides

The frequency lexicon was trimmed with String.trim(), which strips only ASCII
control characters and the space. A line starting with an ideographic space
(U+3000), ordinary in hand-edited CJK text files, therefore kept that space as
part of the word and pushed the count field one token to the right, so the load
failed as a malformed count. The lexicon reader now trims with
StringUtil.trimUnicodeWhitespace, matching the White_Space convention the rest
of the tokenizer already scans by, and a test pins the leading U+3000 case.
The mecab reader's line and numeric-field trims move to the same call so one
class does not mix two whitespace judgments; those fields are ASCII in valid
dictionaries, so the behavior there is unchanged.

Both tokenizer views also gain {@inheritdoc} and their null contract, and the
unknown-candidate helper drops a static modifier it did not need.

…fold fixture duplication

- Document the private lattice helpers decode, relax, and candidates, and the
  installer's boundedStream, with the parameter, return, and exception contracts
  the review expects every method to carry.
- Document the WordEntry and Category record components and the double-array
  builder's findBase and ensureCapacity helpers.
- Record on analyze, tokenize, and tokenizePos that a unk.def without a DEFAULT
  template leaves the lattice disconnected and makes them throw
  IllegalStateException.
- State on readLines that it never returns an empty list, which is what lets the
  matrix.def header be read before the emptiness check.
- Rename the Tokenizer override parameter from s to text in LatticeTokenizer and
  UnigramSegmenter, so the javadoc names a parameter that exists.
- Hoist the matrix.def, char.def, and unk.def file names, the DEFAULT category
  name, the 0x code point prefix, the .. range separator, and the flag value
  into named constants in MecabDictionary, and let LatticeTokenizer reach the
  DEFAULT name through MecabDictionary instead of repeating the literal.
- Name the tar block size, header field offsets, and field lengths in the
  TarGzArchives test helper instead of writing 512, 124, and 148 inline.
- Replace the boolean[1] capture in candidates with a check that the candidate
  list is still empty, which is the same signal without the array.
- Track the best boundary total in decode instead of recomputing the incumbent's
  connection cost on every comparison.
- Drop the categories map field from MecabDictionary, which nothing read once
  the constructor resolved the DEFAULT category out of it.
- Match the char.def code point prefix once, case insensitively, rather than
  testing 0x and 0X separately, and cut the range at the separator's own length.
- Trim the matrix.def header before parsing it and report an empty first line as
  an empty matrix.def, since readLines never yields the empty list the previous
  check was looking for.
- Split the omnibus malformed-dictionary test into named cases that pin the
  messages for a missing definition file, a char.def without DEFAULT, a lexicon
  with no entries, and an empty matrix.def.
- Parameterize the malformed char.def cases and the malformed unigram lexicon
  cases, which were repeated assertThrows calls over one fixture shape.
- Add a Morpheme test pinning the null and empty argument rejections and the
  defensive copy of the feature list.
- Extend the invalid-argument tests to the entry points that were uncovered:
  MecabDictionary.load with a null directory or charset, the installer's null
  target, UnigramSegmenter's path and stream overloads, and both tokenize
  methods of each tokenizer.
- Fold the repeated Files.write fixture calls into one write helper and hoist
  the shared lexicon, matrix, char.def, and unk.def fixture text into constants.
- Correct dev/README-mecab-dictionaries.md to say that dicrc is the
  configuration file the distributions ship alongside the csv and def files a
  MecabDictionary reads, rather than implying the dictionary reads dicrc itself.