PiPNN 3/6: add core graph construction

[weiyaoluo (SeliMeli)]

PiPNN builds ANN graph candidates without searching a partially built graph for each inserted point. This PR adds the provider-independent graph core under diskann::graph::pipnn and connects the numerical kernels from #1287 to their production callers.

The core borrows a dense MatrixView<T>, DiskANN graph policy, and a caller-owned Rayon pool. It returns one adjacency list for each dataset point. Provider state, start and frozen points, quantization, serialization, and search remain outside this module.

Domain terms

• A leader is a sampled dataset point that acts as the center of one child partition.
• fanout[level] is the number of nearest leaders assigned at that recursive partition level. Later levels assign one leader.
• A leaf is a bounded child partition used for local neighbor selection.
• leaf_k is the number of local neighbors retained per point. It is not graph degree R.
• A replica is one deterministic partition pass with its own derived seed.

Code map

1. mod.rs
   • validates PiPNN policy and metric compatibility;
   • selects architecture and metric once per graph build;
   • carries concrete A and M through partition, leaf, and finalization stages;
   • removes PR2's temporary dead-code suppressions because every kernel now has a production caller.
2. partitioning.rs
   • samples leaders and gathers their vectors;
   • computes striped point-to-leader GEMM;
   • calls PiPNN 2/6: add numerical kernels #1287's partition kernel, scatters overlapping children, and recursively splits oversized clusters;
   • merges undersized leaves without exceeding c_max.
3. leaf_build.rs
   • gathers and converts leaf vectors;
   • calls PiPNN 2/6: add numerical kernels #1287's lower-triangular A · Aᵀ and leaf-selection kernels;
   • converts leaf-local neighbors to unique symmetric dataset-ID candidates.
4. finalization.rs
   • keeps candidate lists at or below R unchanged;
   • applies shared Vamana RobustPrune only to overfull lists.

End-to-end flow

Validate dataset and policy → select architecture once → match metric once → build overlapping bounded leaves → gather one leaf → compute its lower Gram matrix → select leaf-local neighbors → translate to global IDs → merge direct candidates → RobustPrune only overfull rows.

Invariants

• Input has at least one point and dimension. Point IDs fit in u32.
• 0 < c_min <= c_max; sampling is in (0, 1]; fanout values are positive; 1 <= leaf_k <= 3; replicas are nonzero.
• Every replica covers every point. Leaves can overlap, but each leaf is non-empty and has at most c_max points.
• Leader-column IDs are local child-partition IDs. Leaf-neighbor IDs are local leaf positions. The owning stage converts both to dataset IDs.
• Leaf IDs are strictly increasing and unique before local-to-global translation.
• Worker buffers retain capacity but expose only the active prefix for the current stripe or leaf.
• Partition worker failures propagate through Rayon immediately.
• The core uses the Rayon pool from PiPNNBuildContext.

Review path

1. Start with PiPNNConfig, PiPNNBuildContext, and build-wide architecture/metric selection.
2. Follow one oversized cluster through sampling, striped assignment, scatter, and recursion.
3. Follow one leaf through conversion, lower-triangle GEMM, local selection, and global candidate insertion.
4. Finish with bounded-list reuse and overfull-list RobustPrune.
5. Confirm the three PR2 dead-code suppressions disappear in this layer.

Validation

• Partition tests cover deterministic seeds, replica coverage, fanout exhaustion, striped assignment, malformed assignment rejection, ordered scatter, conversion, and recursion limits.
• Leaf and finalization tests cover ID validation, candidate symmetry, deduplication, allocation errors, buffer reuse, and exact bounded/overfull output.
• Public graph tests cover source data types, all metrics, determinism, graph IDs, and graph degree.
• Existing PiPNN 2/6: add numerical kernels #1287 differential kernel tests remain active through the same pipnn feature.

Stack relation

Stack 3/6. Depends on #1315 and #1287. #1291 adds disk-index integration; #1294 adds benchmark entry points; #1295 adds optional HashPrune candidate merging.

Stack 3/6: #1287 → #1291

[Copilot]

Pull request overview

This PR introduces the “core” PiPNN build pipeline in the diskann-pipnn crate, wiring together deterministic partitioning, leaf-local candidate construction, and final pruning into a public build_graph API with a validated build context.

Changes:

• Adds PiPNNConfig validation and a PiPNNBuildContext that binds PiPNN policy to DiskANN graph pruning policy and a caller-owned Rayon thread pool.
• Implements the three main stages: partitioning (partitioning.rs), leaf candidate construction (leaf_build.rs), and final pruning via shared Vamana robust prune (finalization.rs).
• Adds comprehensive unit/integration tests and a Criterion benchmark for core scenarios; updates dependencies, lockfile, and mutation-test exclusions.

Reviewed changes

Copilot reviewed 13 out of 14 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
diskann-pipnn/src/lib.rs	Adds public PiPNN API (PiPNNConfig, PiPNNBuildContext, build_graph) and stage orchestration.
diskann-pipnn/src/partitioning.rs	Implements deterministic overlapping partition construction and leader assignment/scatter.
diskann-pipnn/src/partitioning/tests.rs	Adds unit tests covering partition determinism, invariants, error cases, and helpers.
diskann-pipnn/src/leaf_build.rs	Builds leaf-local symmetric k-NN candidates and accumulates global candidates safely in parallel.
diskann-pipnn/src/leaf_build/tests.rs	Adds unit tests for candidate correctness, invariants, type support, and error handling.
diskann-pipnn/src/finalization.rs	Orders/prunes candidate rows using shared robust_prune and validates candidate IDs/shape.
diskann-pipnn/src/finalization/tests.rs	Adds unit tests for pruning behavior and candidate validation failures.
diskann-pipnn/src/tests.rs	Tests effective_metric behavior for integer cosine-normalized handling.
diskann-pipnn/tests/config.rs	Integration tests for config validation and graph-policy compatibility checks.
diskann-pipnn/tests/build_graph.rs	Integration tests for end-to-end graph building, invariants, determinism, and type/metric support.
diskann-pipnn/benches/core.rs	Adds a Criterion benchmark for stage-focused core build scenarios.
diskann-pipnn/Cargo.toml	Updates crate dependencies/dev-dependencies and registers the new core benchmark target.
Cargo.lock	Records dependency graph changes for the updated diskann-pipnn crate dependencies.
.cargo/mutants.toml	Adds mutation-test exclusions for key PiPNN public boundary checks and partitioning invariants.

Comments suppressed due to low confidence (1)

diskann-pipnn/src/partitioning.rs:604

• size_of::<f32>() is used without being in scope (no use std::mem::size_of; and not qualified), so this function won’t compile as written.

fn assignment_stripe_rows(leaders: usize) -> usize {
    (ASSIGNMENT_CACHE_TARGET_BYTES / (leaders.max(1) * size_of::<f32>()))
        .clamp(MIN_ASSIGNMENT_STRIPE_ROWS, MAX_ASSIGNMENT_STRIPE_ROWS)
}

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (2)

diskann-pipnn/src/partitioning.rs:637

• size_of is used without being in scope (std::mem::size_of), which will not compile. Qualify the call or import it.

fn assignment_stripe_rows(leaders: usize) -> usize {
    (ASSIGNMENT_CACHE_TARGET_BYTES / (leaders.max(1) * size_of::<f32>()))
        .clamp(MIN_ASSIGNMENT_STRIPE_ROWS, MAX_ASSIGNMENT_STRIPE_ROWS)
}

diskann-pipnn/src/partitioning.rs:19

• Norm is imported but never used in this module, which will trip unused_imports warnings (and can become CI failures under -D warnings). Remove it from the import list.

use diskann::{utils::VectorRepr, ANNError, ANNResult};
use diskann_linalg::Transpose;
use diskann_utils::views::MatrixView;
use diskann_vector::{distance::Metric, norm::FastL2NormSquared, Norm};
use rand::{prelude::IndexedRandom, SeedableRng};
use rayon::prelude::*;

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (1)

diskann-pipnn/src/partitioning.rs:390

• gather_rows uses TypeId::of::<T>(), which implicitly requires T: 'static. Making that bound explicit here avoids surprising/indirect trait-bound errors later and matches the public build_graph boundary (which already requires 'static).

fn gather_rows<T>(data: MatrixView<'_, T>, indices: &[u32], output: &mut [f32]) -> ANNResult<()>
where
    T: VectorRepr,

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

diskann-pipnn/src/partitioning.rs:641

• size_of::<f32>() is used without being imported or qualified, which will fail to compile. Qualify it with std::mem::size_of (or add an explicit import).

    let rows = ASSIGNMENT_CACHE_TARGET_BYTES / (leaders.max(1) * size_of::<f32>());

[Copilot]

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (4)

diskann-pipnn/src/leaf_build.rs:222

• build_leaf is executed from a Rayon parallel context (via build_leaf_candidates), so it should also explicitly require T: Send + Sync to reflect the actual thread-safety requirement.

where
    T: VectorRepr + 'static,
{

diskann-pipnn/src/leaf_build/tests.rs:154

• assert_source_type forwards T into the parallel leaf build path, so it should also include Send + Sync bounds to match the production requirements.

fn assert_source_type<T>(data: &[T])
where
    T: diskann::utils::VectorRepr + 'static,
{

diskann-pipnn/src/leaf_build.rs:193

• build_leaf_candidates uses Rayon parallel iteration over data, so T must be Send + Sync. Making this explicit in the signature avoids confusing trait-bound errors at call sites and documents the thread-safety requirement.

This issue also appears on line 220 of the same file.

where
    T: VectorRepr + 'static,
{

diskann-pipnn/src/leaf_build/tests.rs:35

• This test helper calls build_leaf_candidates, which (via Rayon) requires T: Send + Sync. Add the bounds here so the test continues to compile once the production signature is tightened.

This issue also appears on line 151 of the same file.

where
    T: diskann::utils::VectorRepr + 'static,
{

[weiyaoluo (SeliMeli)]

Restored the intended stack boundary: PR3 now inherits the numerical kernels from #1287, adds their production partition/leaf callers, and removes all three temporary PR2 allow(dead_code) attributes. The cumulative PR3 and top-of-stack source trees are unchanged.