Skip to content

Speed up index builds with numeric prefix IDs - #12

Merged
platypii merged 1 commit into
masterfrom
perf/numeric-prefix-ids
Jul 31, 2026
Merged

Speed up index builds with numeric prefix IDs#12
platypii merged 1 commit into
masterfrom
perf/numeric-prefix-ids

Conversation

@platypii

Copy link
Copy Markdown
Contributor

Summary

  • encode default ASCII n-gram prefixes as dense numeric bucket IDs
  • avoid prefix substring allocations and outer string Map lookups in flushBlock
  • retain a collision-free numeric fallback for custom Unicode ngramChars
  • preserve global UTF-16 lexical ordering when draining prefix buckets

Performance

On the 25,000-row WildChat slice with hyparquet-writer 0.16.1:

  • mean index build: 28.63s -> 25.67s (10.3% faster)
  • flushBlock sampled CPU: 5.94s -> 3.05s (48.8% faster)
  • output size unchanged at 22,157,787 bytes
  • old and new indexes have identical SHA-256 hashes

Validation

  • npm test (145 tests)
  • npm run lint
  • npx tsc --noEmit

@platypii
platypii merged commit c5270ff into master Jul 31, 2026
6 checks passed
@platypii
platypii deleted the perf/numeric-prefix-ids branch July 31, 2026 19:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant