Add DiskANN benchmark pipeline to GitHub Actions - #857

Draft

Yuanyuan Tian (YuanyuanTian-hh) wants to merge 31 commits into

mainfrom

user/tianyuanyuan/add-benchmark-pipeline

Commits on Mar 19, 2026

Migrate DiskANN benchmark pipeline from ADO to GitHub Actions
Yuanyuan Tian (from Dev Box)
committed
c6402fb

Commits on Mar 20, 2026

Fix openai-100K distance metric and add gitignore patterns
Yuanyuan Tian (from Dev Box)
committed
2217353
Add push trigger to benchmarks workflow for pre-merge testing
Yuanyuan Tian (from Dev Box)
committed
6b7b250
Fix baseline run: use input config from current branch checkout
Yuanyuan Tian (from Dev Box)
committed
d4a6abd
Fix markdown conversion: replace broken csvtomd with inline Python
Yuanyuan Tian (from Dev Box)
committed
c6d54a9
Update benchmark configs: align build/search params to fix low recall
Yuanyuan Tian (from Dev Box)
committed
5775198

Commits on Mar 23, 2026

Fix recall_at: set to 100 to match groundtruth file K=100
Yuanyuan Tian (from Dev Box)
committed
7ac841e
Remove stale absolute contracts (qps/recall/total_time): calibrated for ADO mimir-enron, not applicable to public datasets on GitHub runners. Threshold calibration tracked in PBI.
Yuanyuan Tian (from Dev Box)
committed
7d5a72e
remove comments
Yuanyuan Tian (from Dev Box)
committed
c294c16
Fix build_ram_limit_gb: reduce 10->4 to fit GitHub runner RAM (7GB limit)
Yuanyuan Tian (from Dev Box)
committed
b8e1d6d
Fix wikipedia distance: cosine_normalized->cosine (vectors are not L2-normalized, metric is inner product)
Yuanyuan Tian (from Dev Box)
committed
137cae0
Fix wikipedia distance: cosine->inner_product (groundtruth uses raw ip, not cosine similarity)
Yuanyuan Tian (from Dev Box)
committed
96b63a8

Commits on Mar 24, 2026

Align build threads
Yuanyuan Tian (from Dev Box)
committed
1b4cfc8
Speed up benchmarks: build threads 1->4, openai pq_chunks 384->192
Yuanyuan Tian (from Dev Box)
committed
5e6a6e0
Temp: disable concurrency cancellation for A/A batch testing
Yuanyuan Tian (from Dev Box)
committed
59d25b9

Commits on Mar 26, 2026

revert A/A test settings, update OpenAI config to SQ_1_2.0
Yuanyuan Tian (from Dev Box)
committed
751e775

Commits on Mar 27, 2026

Remove micro-benchmark-iai comments
Yuanyuan Tian (from Dev Box)
committed
2c4d235
use GitHub Release assets for benchmark datasets
Yuanyuan Tian (from Dev Box)
committed
f58e084
extract csv-to-markdown into reusable script
Yuanyuan Tian (from Dev Box)
committed
0f5c277
calibrate contract thresholds from GitHub runner data
Yuanyuan Tian (from Dev Box)
committed
b8cddb5
add daily A/A benchmark stability test with failure notification
Yuanyuan Tian (from Dev Box)
committed
accdf2c
widen mean_cpus threshold to 15% for shared-runner CPU noise
Yuanyuan Tian (from Dev Box)
committed
5b0e7f7

Commits on Mar 30, 2026

move benchmark datasets to separate repo (YuanyuanTian-hh/diskann-benchmark-data)
Yuanyuan Tian (from Dev Box)
committed
639d4bb
consolidate 3 benchmark scripts into single benchmark_validate.py
Yuanyuan Tian (from Dev Box)
committed
1b52de2

Commits on Mar 31, 2026

switch benchmark jobs to self-hosted 1ES runner pool (diskann-github)
Yuanyuan Tian (from Dev Box)
committed
9779e3c
replace gh CLI with curl for dataset downloads (gh not available on 1ES runners)
Yuanyuan Tian (from Dev Box)
committed
db7e97a
fix latency_95: read p95_latency instead of p999_latency from benchmark JSON
Yuanyuan Tian (from Dev Box)
committed
ea91c62

Commits on Apr 1, 2026

revert to ubuntu-latest runners, switch dataset source to BAB v0.4.0
Yuanyuan Tian (from Dev Box)
committed
360cdc7

Commits on Apr 2, 2026

address PR review: reduce search_list to 200, remove hardcoded Rust version, fix missing-field handling, clean up orphaned thresholds, switch data source to BAB v0.4.0
Yuanyuan Tian (from Dev Box)
committed
9d635be
widen latency_95 threshold to 15% for shared-runner noise
Yuanyuan Tian (from Dev Box)
committed
4ca80d1
replace push trigger with pull_request trigger targeting main
Yuanyuan Tian (from Dev Box)
committed
d168932