Skip to content

Bound DuckDB workers for pair matching - #143

Draft
afermg wants to merge 2 commits into
mainfrom
perf/duckdb-thread-budget
Draft

Bound DuckDB workers for pair matching#143
afermg wants to merge 2 commits into
mainfrom
perf/duckdb-thread-budget

Conversation

@afermg

@afermg afermg commented Aug 28, 2026

Copy link
Copy Markdown
Collaborator

Summary

matching.find_pairs currently lets DuckDB size its worker pool from every visible CPU. On large shared hosts this can oversubscribe a small metadata self-join: the benchmark host exposes 384 logical CPUs.

For pandas-backed queries, this PR:

  • defaults DuckDB to at most 8 workers
  • respects process CPU affinity where supported
  • allows an explicit COPAIRS_DUCKDB_THREADS positive-integer override
  • executes the query through the configured local connection

DuckDB relation inputs retain their origin connection without pandas materialization or configuration changes. The primary multilabel query remains outside this PR because #137 owns that implementation seam.

Real-workload benchmark

JUMP-lite group_low: 19,521 profiles, 37,143 positive pairs, and 515,309 negative pairs. Three fresh processes per implementation, five iterations per query; table entries are medians of process-level warm medians.

Query Current Bounded default Speedup
Positive 0.0968 s 0.0206 s 4.70×
Negative 0.1238 s 0.0363 s 3.41×
Combined 0.2206 s 0.0569 s 3.88×

A separate 1/4/8/16/32-thread sweep produced combined medians of 0.0769/0.0655/0.0609/0.0681/0.0781 seconds, respectively, supporting 8 as the bounded default on this workload.

Pair counts and normalized pair sets were identical. Row order is intentionally not asserted: the SQL query has no ORDER BY, so neither the existing nor candidate implementation defines a row-order contract.

Absolute timings on this shared host are noisy, but the bounded default was consistently faster for group_low.

Validation

  • Full local suite: 81 passed on Python 3.12
  • Combined matching suites: 31 passed
  • 14 focused thread-budget, pandas/relation parity, non-materialization, fallback, environment-validation, and existing-validation tests passed
  • relations execute on their origin connection
  • invalid environment values fail with an actionable error
  • Ruff latest format/check passed
  • git diff --check passed
  • independent review performed; ordering-test blocker corrected

@afermg
afermg marked this pull request as draft September 2, 2026 17:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant