Skip to content

perf: skip lid blocks - #407

Merged
cheb0 merged 3 commits into
mainfrom
0-lid-block-skip
Aug 24, 2026
Merged

perf: skip lid blocks#407
cheb0 merged 3 commits into
mainfrom
0-lid-block-skip

Conversation

@cheb0

@cheb0 cheb0 commented Apr 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

Description

Summary

  • new frac version: v5
  • LID block ext1 now has firstLID and lastLID encoded
  • isContinued is only used for older frac versions
  • no legacy format TID adjusting for a new frac version
  • as previously, we always locate and load the first block eagerly but calling NextGeq allows to skip some blocks

Measurements

With 16k LID blocks, there are only 4 search requests (out of 50+ I tested) where this feature improves cold query perf. All four search requests fit into "intersect dense and sparse column" type.

Query env Total cold, ms   LID blocks read cold (branch), ms   LID blocks read (branch) cold diff LID blocks read, diff Trial_1 Trial_2 Trial_3
service:E and login:L and moduleName:S and (zone:A or zone:B or zone:C) and exists:message prod 0 22.29 ±1.03 164 20.6 ±1.10 62 -7.6% -62.2% -12.6% -8.9% -7.6%
request_host:huge AND remote_addr:X (found) lb 0 18.22 ±0.97 69 17.53 ±0.82 42 -3.8% -39.1% -3.8% -9.8% -3.8%
request_host:small AND response_status:200 lb 69 33.64 ±1.92 248 30.44 ±1.60 138 -9.5% -44.4% -2% -10.9% -9.5%
response_status:500 | by request_method lb 172 52.58 ±2.14 377 51.68 ±1.54 333 -1.7% -11.7% -7.9% -20.6% -1.7%
  • I have read and followed all requirements in CONTRIBUTING.md;
  • I used LLM/AI assistance to make this pull request;

@cheb0
cheb0 marked this pull request as ready for review April 24, 2026 11:33
forshev
forshev approved these changes Apr 27, 2026
View reviewed changes
Comment thread frac/sealed/lids/table.go Outdated
@cheb0 cheb0 added the performance Features or improvements that positively affect seq-db performance label May 12, 2026
@eguguchkin eguguchkin added this to the v0.72.0 milestone Jun 3, 2026
Base automatically changed from 330-lid-block-size-config to main June 5, 2026 12:12
@eguguchkin
eguguchkin marked this pull request as draft June 8, 2026 11:49
@eguguchkin eguguchkin removed this from the v0.72.0 milestone Jun 8, 2026
@cheb0
cheb0 force-pushed the 0-lid-block-skip branch 3 times, most recently from 2ec59f6 to b7a0835 Compare July 15, 2026 13:37
@cheb0
cheb0 marked this pull request as ready for review July 15, 2026 13:37
@coderabbitai

coderabbitai Bot commented Jul 15, 2026

Copy link
Copy Markdown

Caution

Review failed

An error occurred during the review process. Please try again later.

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@cheb0
cheb0 force-pushed the 0-lid-block-skip branch from b7a0835 to b3f137c Compare July 15, 2026 13:42
@codecov-commenter

codecov-commenter commented Jul 15, 2026 •
edited
Loading

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 67.79661% with 19 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.11%. Comparing base (08b0423) to head (2b14330).

Files with missing lines Patch % Lines
frac/sealed_loader.go 42.85% 11 Missing and 1 partial ⚠️
frac/sealed/lids/table.go 74.07% 4 Missing and 3 partials ⚠️
Additional details and impacted files 
@@            Coverage Diff             @@
##             main     #407      +/-   ##
==========================================
- Coverage   71.12%   71.11%   -0.01%     
==========================================
  Files         233      233              
  Lines       19052    19092      +40     
==========================================
+ Hits        13551    13578      +27     
- Misses       4475     4483       +8     
- Partials     1026     1031       +5

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
Indexer-4 42fb1c 0f79ba
572784732.00 B/op 782569576.00 B/op 1.37 🔴

@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
Indexer-4 42fb1c 3b55ca
572784732.00 B/op 774026566.00 B/op 1.35 🔴

@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
Indexer-4 42fb1c 7aacd4
572784732.00 B/op 785417068.00 B/op 1.37 🔴

@cheb0
cheb0 force-pushed the 0-lid-block-skip branch from b3f137c to 3e3dc61 Compare July 17, 2026 11:50
@github-actions

github-actions Bot commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

🔴 Performance Degradation

Some benchmarks have degraded compared to the previous run.
Click on Show table button to see full list of degraded benchmarks.

Show table
Name Previous Current Ratio Verdict
Indexer-4 8d4014 ad004f
572462538.00 B/op 757607172.00 B/op 1.32 🔴

@eguguchkin eguguchkin modified the milestones: v0.76.0, v0.77.0 Jul 20, 2026
@cheb0
cheb0 force-pushed the 0-lid-block-skip branch from 3e3dc61 to 3b45105 Compare August 24, 2026 13:48
@cheb0
cheb0 merged commit 365d02f into main Aug 24, 2026
8 checks passed
@cheb0
cheb0 deleted the 0-lid-block-skip branch August 24, 2026 14:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@forshev forshev forshev approved these changes
@eguguchkin eguguchkin eguguchkin approved these changes

Assignees

No one assigned

Labels

performance Features or improvements that positively affect seq-db performance

Projects

None yet

Milestone

v0.77.0

Development

Successfully merging this pull request may close these issues.

4 participants

@cheb0 @codecov-commenter @forshev @eguguchkin