feat: compress RLE child buffers#7663
Open
Xuanwo wants to merge 2 commits into
Open
Conversation
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7329.
RLE miniblock compression currently stores run values and run lengths as raw
Flatchild buffers, so repeated or low-entropy child buffers can dominate the encoded page even after RLE chooses a better shape than raw fixed-width data.This change lets RLE child buffers choose the smallest valid representation among
Flat,OutOfLineBitpacking(Flat), andGeneral(Flat)when a general compression scheme is configured. The decoder validates the supported nested child encodings and preserves the existing flat path, while rejecting combinations where both children require the run count.The strategy automatically enables RLE child bitpacking and reuses explicit field compression settings for LZ4/Zstd child candidates, so configured general compression can reduce the values and run-length payload without adding a new public compression knob.
Performance data from a local release microbenchmark is below. Payload bytes exclude metadata;
shapeisvalues/run_lengths.flat/flat)flat/flat)flat/flat)bitpack/flat)general/general)general/general)bitpack/flat)bitpack/general)bitpack/general)flat/bitpack)flat/general)flat/general)flat/flat)flat/flat)flat/flat)The encoder falls back to
Flatwhen a child encoding does not shrink the payload. In the high-benefit cases, child bitpacking keeps encode/decode time in the same order of magnitude as plain RLE, while LZ4/Zstd trade more CPU for substantially smaller payloads. For example, short runs with small values measured about 512/237 us encode/decode for plain RLE, 517/248 us with child bitpacking, 585/265 us with LZ4, and 633/294 us with Zstd.