Skip to content

feat: compress RLE child buffers#7663

Open
Xuanwo wants to merge 2 commits into
mainfrom
xuanwo/issue-7329-rle-child-compression
Open

feat: compress RLE child buffers#7663
Xuanwo wants to merge 2 commits into
mainfrom
xuanwo/issue-7329-rle-child-compression

Conversation

@Xuanwo

@Xuanwo Xuanwo commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Fixes #7329.

RLE miniblock compression currently stores run values and run lengths as raw Flat child buffers, so repeated or low-entropy child buffers can dominate the encoded page even after RLE chooses a better shape than raw fixed-width data.

This change lets RLE child buffers choose the smallest valid representation among Flat, OutOfLineBitpacking(Flat), and General(Flat) when a general compression scheme is configured. The decoder validates the supported nested child encodings and preserves the existing flat path, while rejecting combinations where both children require the run count.

The strategy automatically enables RLE child bitpacking and reuses explicit field compression settings for LZ4/Zstd child candidates, so configured general compression can reduce the values and run-length payload without adding a new public compression knob.

Performance data from a local release microbenchmark is below. Payload bytes exclude metadata; shape is values/run_lengths.

Scenario Plain RLE Auto child bitpack LZ4 configured Zstd configured
Long constant runs 384 B 384 B (flat/flat) 384 B (flat/flat) 384 B (flat/flat)
Short runs, small values 327,680 B 90,112 B (bitpack/flat) 5,120 B (general/general) 5,440 B (general/general)
Short runs, monotonic values 327,680 B 196,608 B (bitpack/flat) 132,224 B (bitpack/general) 132,800 B (bitpack/general)
Short runs, random values 327,680 B 286,720 B (flat/bitpack) 263,296 B (flat/general) 263,872 B (flat/general)
Variable U16 run lengths 26,226 B 26,226 B (flat/flat) 26,226 B (flat/flat) 26,226 B (flat/flat)

The encoder falls back to Flat when a child encoding does not shrink the payload. In the high-benefit cases, child bitpacking keeps encode/decode time in the same order of magnitude as plain RLE, while LZ4/Zstd trade more CPU for substantially smaller payloads. For example, short runs with small values measured about 512/237 us encode/decode for plain RLE, 517/248 us with child bitpacking, 585/265 us with LZ4, and 633/294 us with Zstd.

@github-actions github-actions Bot added A-encoding Encoding, IO, file reader/writer enhancement New feature or request labels Jul 7, 2026
@Xuanwo Xuanwo marked this pull request as ready for review July 7, 2026 10:30
@codecov

codecov Bot commented Jul 7, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.68969% with 197 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
rust/lance-encoding/src/encodings/physical/rle.rs 82.17% 92 Missing and 36 partials ⚠️
rust/lance-encoding/src/compression.rs 58.18% 51 Missing and 18 partials ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

A-encoding Encoding, IO, file reader/writer enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support secondary compression for RLE child buffers

1 participant