perf: increase default memory pool to 150MB with 40MB sort spill reservation

[westonpace]

Raises the per-partition memory pool from 100MB to 150MB and sets the sort spill reservation to 40MB (up from the DataFusion default of 10MB) to give sort operations more headroom while spilling to disk (we should still spill at roughly the same rate).

The previous defaults were 100MB / 10MB. This usually worked but certain patterns would lead to false memory exhaustion errors:

OSError: LanceError(IO): Resources exhausted: Additional allocation failed for ExternalSorterMerge[0] with top memory consumers (across reservations) as:
  ExternalSorterMerge[0]#1(can spill: false) consumed 58.5 MB, peak 58.5 MB,
  ExternalSorter[0]#0(can spill: true) consumed 41.4 MB, peak 89.9 MB.
Error: Failed to allocate additional 345.9 KB for ExternalSorterMerge[0] with 27.6 MB already allocated for this reservation - 92.2 KB remain available for the total pool, /home/pace/lance/rust/lance-datafusion/src/chunker.rs:49:46

The problem happens as follows:

1. The sort node accumulates batches of data without modifying them until it determines a spill is needed. During this phase each batch counts double against the pool reservation. This is meant to provide overhead for the later steps. In our above example we can see spilling was triggered at 41.4MB which is about half of the 90MB pool (half, because each batch is counted double)
2. The sort node determines that spilling is needed. First, it must sort the data that has accumulated in memory. Each batch is sorted by itself. This batch sort is in-place and doesn't affect reservations much.
3. A cursor is created for each in-memory batch. The in-memory batches are then fed into a merge sort.
4. The merge sort accumulates batches of data to send to the spill. Once a batch is accumulated it is written to the spill file.

Both the cursors and the accumulation require additional space. This is the ExternalSorterMerge[0] mentioned above. It is given the overcounting described in step 1. In other words, once this starts, we have half the reservation in ExternalSorter[0] and half the reservation in ExternalSorterMerge[0]. This "additional space" should be about the same size as the input. This is why we count each batch twice.

In practice, the ExternalSorterMerge[0] reservation ends up being slightly higher (for various reasons). This is what the sort_spill_reservation_bytes is supposed to account for. Datafusion defaults this to 10MB. There is no guidance (and I can't get Claude to come up with any good guidance) as to what this value should be set to. However, 10MB seems like too little. This PR updates it to about 1/3 of the memory pool size. In theory it shouldn't grow proportionally to the memory pool but in practice it seems to. I also really don't want to expose it as yet another knob that users have to tune so I'm hoping 1/3 is slightly conservative but good enough.

[Copilot]

Pull request overview

This PR adjusts default DataFusion execution memory settings in lance-datafusion to reduce false “resources exhausted” errors during sort spilling by increasing the available per-partition pool and reserving more headroom for spill/merge overhead.

Changes:

• Increase the default per-partition memory pool from 100MB to 150MB.
• Configure DataFusion’s sort_spill_reservation_bytes to mem_pool_size / 3 (50MB with the new default).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hamersaw]

This sounds reasonable.

[westonpace]

I ended up tweaking things slightly. One set of tests was failing because 3 tests ran at once and tried to create an index. Each test grabbed 50MB of headroom from the same shared pool and then none of them could proceed.

I dropped the limit down to 40, which is a somewhat arbitrary way to make this pass. In addition, I made 40MB the max. If we keep scaling the headroom with the pool size then we have an effective cap on the max number of concurrent operations (something less than pool size / head room) which I don't think we want. Hopefully the headroom needs do not grow linearly with the pool size or else I think we will just have to fix the upstream.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!