Batch processing: fix maxBatches for partitioned data

[mzueva]

MILAB-6799
Previous version incorrectly estimate total data size for partitioned data and this lead to wrong maxBatches calculation. This PR fixes the issue

Greptile Summary

The PR fixes batch-count enforcement and row preservation for partitioned PColumn data by moving the authoritative calculation into the split template, where the true per-scope row count is available.

• Recomputes batchCount from actual records for both passContent modes, preventing partition-count estimates from dropping rows or bypassing the cap.
• Divides the global maxBatches budget among isolation scopes and deterministically inflates each scope’s batch size.
• Adds integration fixtures and coverage for inline and partitioned data across content, TSV-blob, and Parquet-blob paths.
• PColumn — a typed Platforma data column whose records may be stored inline or in partitions; partitioned PColumns now use their actual row count during splitting.
• Batch — a contiguous subset of sorted PColumn rows processed by one body invocation; its size may now be inflated to honor the configured cap.
• maxBatches — the soft upper bound on the total number of batch invocations; enforcement moves from an unreliable partition-based estimate to per-scope true counts.
• Isolation scope — a group of records processed independently according to isolation axes; each scope now receives an equal integer share of the global batch budget.
• passContent — selects whether batch data is supplied directly as content or as a file reference; both paths now recompute batch counts from authoritative data.
• actualRecords — the true number of rows in one isolation scope; newly used to derive batch count and effective batch size.
• scopeMaxBatches — the newly introduced per-isolation-scope share of maxBatches, passed from the orchestrator to the split template.

Confidence Score: 5/5

The PR appears safe to merge, with the changed batch-count paths preserving rows and enforcing the configured global cap.

The authoritative row count is now obtained inside each split scope, cap allocation remains globally bounded, and focused tests cover partitioned inputs along both direct-content and file-reference paths.

Important Files Changed

Filename	Overview
sdk/workflow-tengo/src/pframes/process-pcolumn-batch.tpl.tengo	Allocates the global batch cap across isolation scopes and passes the resulting per-scope budget to split renders.
sdk/workflow-tengo/src/pframes/process-pcolumn-batch-split.tpl.tengo	Recomputes batch counts from authoritative per-scope row counts and inflates batch size when necessary.
tests/workflow-tengo/src/pframes/proc_batch.tpl.tengo	Extends the integration harness with genuine partitioned inputs and batch-boundary-observing body modes.
tests/workflow-tengo/src/pframes/proc_batch_filter.test.ts	Adds observable cap and deterministic-boundary coverage for direct-content and Parquet blob paths.
tests/workflow-tengo/src/pframes/proc_batch_max_batches.test.ts	Adds focused regressions proving losslessness and maxBatches enforcement for partitioned TSV inputs in both passContent modes.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
  A[Partitioned or inline PColumn] --> B[Orchestrator groups isolation scopes]
  B --> C[Assign equal scopeMaxBatches budget]
  C --> D[Split template obtains actualRecords]
  D --> E{Computed batches exceed scope budget?}
  E -- Yes --> F[Inflate batchSize deterministically]
  E -- No --> G[Keep effective batchSize]
  F --> H[Slice and invoke batch body]
  G --> H
  H --> I[Merge batch outputs]

Loading

_{Reviews (1): Last reviewed commit: "Fix max batch count for partitioned data" | Re-trigger Greptile}

Context used:

• Context used - Terms is a types in codebase. Provide the list of ... (source)
• Knowledge Base — workflow-tengo: the block workflow authoring SDK

[changeset-bot]

🦋 Changeset detected

Latest commit: b309bd3

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 4 packages

Name	Type
@platforma-sdk/workflow-tengo	Minor
@milaboratories/pl-middle-layer	Patch
@platforma-sdk/pl-cli	Patch
@platforma-sdk/test	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[notion-workspace]

[sdk-processColumn] maxBatches parameter is not working with partitioned data like parquet input files

[codecov]

❌ 1 Tests Failed:

Tests completed	Failed	Passed	Skipped
2347	1	2346	24

View the top 1 failed test(s) by shortest run time

src/v3.test.ts > v3: prerunArgs fastNumbers test

Stack Traces | 10.3s run time

Error: Test timed out in 10000ms.
If this is a long-running test, pass a timeout value as the last argument or configure it globally with "testTimeout".
 ❯ src/v3.test.ts:16:1

To view more test analytics, go to the Test Analytics Dashboard
_{📋 Got 3 mins? Take this short survey to help us improve Test Analytics.}