From b59c1b4652fff0033b87031c46cbe99c00abcacf Mon Sep 17 00:00:00 2001 From: Dylan Kupsh Date: Tue, 11 Aug 2026 17:01:21 -0700 Subject: [PATCH] Specify learned search for DSE and Place and Route Add nine ML search documents split into a shared core and two concrete searches. ML Environment Core, ML Model Core, and ML Training Core own what every learned search shares: graph observation projection, action indexing over a live enumeration, episode accounting, the integer reward boundary, the disjoint-union batch and graph-transformer trunk, and the run identity and checkpoint boundary. The DSE and PnR environment, model-architecture, and training documents own their own decision spaces on top of that core, and own no legality, mappability, or objective fact. Name the one intermediate the search harness needs from the Fabric pipeline. VerifiedFabricClosure is the value after independent reverification and before ArtifactStore::put, so a harness may evaluate and discard a candidate without publishing it. It is a transient in-process value, not an Artifact family, a persistent schema, or a second identity authority, and it gives the ADG Builder no second finalization path. Add SearchPolicy.realization_move_radius to Place and Route. Bounded(r) forms a realization-binding proposal domain rather than filtering a draw taken from one, so a radius-emptied anchor is absent instead of drawn and retried. It is a search preference and never a legality authority, and movableDecisionCount now counts the proposal domain, which coincides with A(M,C,S) under Unbounded. Record Ray as the stack's one modified dependency. The Loom-owned fork at externals/ray carries a graph-space sampler and connector patch that the ML environment contract requires, an upgrade pins both the fork commit and the upstream release it was rebased onto, and the exact Gymnasium and PyTorch versions the ML documents target are pinned in the stack rather than in the documents that consume them. Documentation only; no code or test changes. Co-Authored-By: Claude Opus 5 (1M context) --- docs/rationales/README.md | 1 + docs/rationales/ml.md | 596 +++++++++++ docs/spec-adg-builder.md | 6 + docs/spec-fabric-artifact.md | 17 + docs/spec-loom-stack.md | 34 + docs/spec-ml-core-environment.md | 826 +++++++++++++++ docs/spec-ml-core-model.md | 646 ++++++++++++ docs/spec-ml-core-training.md | 902 ++++++++++++++++ docs/spec-ml-dse-environment.md | 1325 ++++++++++++++++++++++++ docs/spec-ml-dse-model-architecture.md | 316 ++++++ docs/spec-ml-dse-training.md | 597 +++++++++++ docs/spec-ml-pnr-environment.md | 1211 ++++++++++++++++++++++ docs/spec-ml-pnr-model-architecture.md | 368 +++++++ docs/spec-ml-pnr-training.md | 527 ++++++++++ docs/spec-pnr.md | 73 +- 15 files changed, 7442 insertions(+), 3 deletions(-) create mode 100644 docs/rationales/ml.md create mode 100644 docs/spec-ml-core-environment.md create mode 100644 docs/spec-ml-core-model.md create mode 100644 docs/spec-ml-core-training.md create mode 100644 docs/spec-ml-dse-environment.md create mode 100644 docs/spec-ml-dse-model-architecture.md create mode 100644 docs/spec-ml-dse-training.md create mode 100644 docs/spec-ml-pnr-environment.md create mode 100644 docs/spec-ml-pnr-model-architecture.md create mode 100644 docs/spec-ml-pnr-training.md diff --git a/docs/rationales/README.md b/docs/rationales/README.md index 655949a84..194ae94f2 100644 --- a/docs/rationales/README.md +++ b/docs/rationales/README.md @@ -25,6 +25,7 @@ are not architecture facts and are not reproduced here. | [Fabric Memory And Resources](fabric-memory-and-resources.md) | [Fabric Memory](../spec-fabric-mem.md), [Fabric Resource Contract](../spec-fabric-resource-contract.md), [Fabric Boundary](../spec-fabric-boundary.md), [Fabric FIFO](../spec-fabric-fifo.md), [Fabric Switch](../spec-fabric-switch.md), [Fabric Instantiate](../spec-fabric-instantiate.md) | | [Mapping And PnR](mapping-and-pnr.md) | [Mapping Artifact](../spec-mapping-artifact.md), [Mapping Identity](../spec-mapping-identity.md), [Mapping Memory](../spec-mapping-memory.md), [Mapping Verification](../spec-mapping-verification.md), [TechMapping Generation](../spec-tech-mapping.md), [Place And Route](../spec-pnr.md) | | [Evaluation And DSE](evaluation-and-dse.md) | [Evaluation And DSE](../spec-dse-feedback.md), [Evaluation Metrics](../spec-evaluation-metrics.md), [External Tool Invocation](../spec-external-tool-invocation.md), [FPA Evaluation](../spec-fpa-estimation.md) | +| [ML Search](ml.md) | [ML Environment Core](../spec-ml-core-environment.md), [ML Model Core](../spec-ml-core-model.md), [ML DSE Environment](../spec-ml-dse-environment.md), [ML DSE Model Architecture](../spec-ml-dse-model-architecture.md), [ML Training Core](../spec-ml-core-training.md), [ML DSE Training](../spec-ml-dse-training.md), [ML PnR Environment](../spec-ml-pnr-environment.md), [ML PnR Model Architecture](../spec-ml-pnr-model-architecture.md), [ML PnR Training](../spec-ml-pnr-training.md) | | [Simulation](simulation.md) | [Simulation Artifacts](../spec-simulation-artifacts.md), [DFG-sim](../spec-sim-dfg.md), [CGRA-sim](../spec-sim-cgra.md), [Simulation Comparison](../spec-sim-comparison.md) | | [Runtime And Deployment](runtime-and-deployment.md) | [Executable Closure](../spec-executable-closure.md), [Configuration And Deployment](../spec-configuration-deployment.md), [Runtime ABI](../spec-runtime-abi.md), [Implementation Platform](../spec-implementation-platform.md) | | [Hardware Backend](hardware-backend.md) | [RTL Lowering](../spec-rtl-lowering.md), [Hardware Implementation](../spec-hardware-implementation.md), [Implementation Platform](../spec-implementation-platform.md), [EDA Tooling](../spec-eda-tooling.md), [External Tool Invocation](../spec-external-tool-invocation.md), [FPA Evaluation](../spec-fpa-estimation.md) | diff --git a/docs/rationales/ml.md b/docs/rationales/ml.md new file mode 100644 index 000000000..251f7991b --- /dev/null +++ b/docs/rationales/ml.md @@ -0,0 +1,596 @@ +# ML Search Rationale + +Normative contracts are owned by +[ML Environment Core](../spec-ml-core-environment.md), +[ML Model Core](../spec-ml-core-model.md), +[ML DSE Environment](../spec-ml-dse-environment.md), +[ML DSE Model Architecture](../spec-ml-dse-model-architecture.md), +[ML Training Core](../spec-ml-core-training.md), +[ML DSE Training](../spec-ml-dse-training.md), +[ML PnR Environment](../spec-ml-pnr-environment.md), +[ML PnR Model Architecture](../spec-ml-pnr-model-architecture.md), and +[ML PnR Training](../spec-ml-pnr-training.md). + +Loom uses learned search in two places that look different and are not. One +explores a design space by selecting candidate-generator decisions; the other +places and routes one fixed problem by selecting Place and Route Actions. In +both, the learner is a proposal policy over decisions some other owner already +defined, and everything that decides what a decision means stays with that +owner. These documents explain why that boundary is where it is, and what was +rejected on either side of it. + +## Why A Learned Search Policy Is A Harness, Not A Plan Node + +A reinforcement-learning agent needs the two things the resolved plan +deliberately refuses: a mutable current design and a runtime loop. Adding +either to the plan would make termination, deterministic work, recovery, and +cache identity depend on a policy's sampled behavior, and the plan would stop +being readable before execution. + +The alternative that was rejected was a generic environment action language +over node names and property bags. It would have reproduced exactly the problem +typed domain generators exist to prevent, and it would have needed its own +verifier to decide which actions were legal. + +What the environment does instead is treat the agent as a search policy over +decisions that already exist. A candidate-generator kind already owns a closed +decision union, a finite decision domain, and a canonical decision order, so an +action can be an ordinal in that order rather than a new vocabulary. The +episode's state is an ordinary Builder derivation that stops before +publication, so legality is answered by the same finalizer that answers it for +a published candidate, and nothing partial or DSE-only exists. Only the +retained decision sequence crosses back: replaying it through ordinary Generate +and Promote nodes is what makes a discovered design a real candidate with real +lineage and real Evidence. The harness never selects, never promotes, and never +publishes. + +Reward needed no new concept for the same reason. Search energy is already one +selected weighted level over quantized objective codes, and a policy that needs +a reward wants its signed difference across a transition. Reusing it keeps +reward exact and integral, and keeps a single objective authority for annealing +and learning alike. + +Workload feasibility stayed a gate rather than a penalty because the two answer +different questions. A design that cannot run the workload set is not a poor +point in the space; it is outside it. Scoring it would teach a policy to trade +mappability against area, and there is no exchange rate for that trade. Keeping +proof separate from budget exhaustion follows the same discipline the rest of +the stack uses: only exact admission or a sound bound proves impossibility, and +an exhausted search budget proves nothing about the design. + +Scope became configuration for the same reason the plan uses typed generators. +The episode may explore one Module, a complete multi-core System, and the +software expression of its workloads, and the obvious way to offer that is a +flag per capability. Flags do not compose: three of them are eight +configurations, most of which are meaningless, and each new family doubles the +count again. Binding a set of exploration domains instead makes permission and +extension the same act, and requiring each domain to state the same small set +of facts before it can be explored is a more useful admission test than a +boolean: a family nobody can describe that precisely is a family nobody is +ready to search. + +Some of those facts exist because a generator's child is not always the +episode's subject. A Spatial rewrite inside a System episode yields a Module, +and a Module is an intermediate design input rather than a System candidate. A +software rewrite yields a program whose TechMapping no longer binds. In both +cases the step is not over when the generator returns, and the follow-up is +mechanically determined by the decision's own target. A completion that would +need a second free choice was rejected as a design, because it would make the +environment an author of decisions the agent did not take. + +Targeting an existing trainer's environment definition rather than inventing +one was the cheaper correctness decision, and the one place it was allowed to +dictate the data model is the action space, which a fixed-size space genuinely +requires. The observation was not allowed to follow. Padding the graph to a +fixed extent would have made every batch mostly inert rows at a bound large +enough to be safe, and would have refused legal designs at any smaller bound — +a trainer's preference for rectangular arrays deciding which hardware the +search may occupy. The design space is variable-sized, so the observation is a +variable-size graph. + +That choice is what forced the trainer to be a fork, and the trade was made +deliberately in that direction. The alternatives were to bend the data to the +tool, which is the padding just rejected, or to write a trainer from scratch, +which buys a large surface of well-understood RL machinery for the sake of one +missing capability. Patching graph-space support into the sampler is the +smaller of the three, but it is not free: it is the only modified dependency in +the stack, and every upgrade becomes a rebase. It is affordable only because +the fork is reachable from nothing but the search harness, so its blast radius +stops at training. A fork that could reach a compiler, a Mapping, or an +Artifact schema would not have been worth this. + +A parallel sampler does impose one real cost. It creates many environment +copies, so seed independence cannot come from a single counter; deriving it +from each copy's own coordinates buys independence without coordination, at the +cost of tying exact reproduction to the sampling topology. That cost is visible +in the contract instead of being discovered later as nondeterminism. + +The trainer dependency is confined to its own layer for the same reason +generators are typed, so a second trainer, an offline replay tool, or a +scripted search can use the environment without acquiring it, and its version +movement cannot reach the C++ side. + +Masking is where a search policy can quietly corrupt its own training, which is +why the contract constrains an advisory mask so tightly. It is a policy +commitment rather than a proof, so it cannot be allowed to decide what is +legal. The subtler exposure is that a mask which moves across a run differs +between the moment an action was sampled and the moment that sample is learned +from, and the importance ratio is then wrong in a way no loss curve reveals. +Carrying the mask in the batch costs a little memory and removes the whole +class of failure. + +Software and hardware are symmetric under the feasibility gate, and that +symmetry is the argument for exploring them together. A rewrite that the +current fabric cannot map fails exactly as a fabric edit that drops a resource +the workload needs. Neither is a rewrite-legality question: every catalog +rewrite is externally equivalence-preserving by its owner's contract, so what a +rejection reports is a mismatch between this hardware and this expression of +the workload, which is precisely the joint decision the search exists to make. + +## Why The Node And Action Space Is Combined + +The observation unifies three things a conventional design keeps apart: the +entities of the state, the relations between them, and the actions currently +available. Two separate merges produce that, and only the first is +unconditional. + +The first merge draws links as nodes, because the decisions do. Removing an +occurrence, replacing a point connection, changing a transport link, and +refactoring a graph definition are all ordinary decisions, and a place and +route Action names a physical traversal as readily as an occurrence. An +observation that represented only entities as nodes would leave a large +fraction of the action space pointing at something the policy cannot see. +Promoting every targetable entity to a node makes the graph closed under +decision targets, and that closure is what lets an action stay a single ordinal +while still being scored per node and per link. + +The second merge is the one that matters and it is easy to state wrongly. What +is unified is the *addressing*: an action names graph nodes, and a policy +scores it from the embeddings of the nodes it names. An action set is normally +a side channel — a vector of logits produced from a state embedding, indexed by +a scheme the model reconstructs from nothing — and the whole point is that here +it is not. Scoring an action is reading embeddings the encoder already +produced, and the action index is an ordinal in a canonical order rather than a +decoded coordinate. + +What is *not* unified is the carrier, and the choice between the two forms was +made a consequence of arity rather than a house style. Variable arity needs +out-degree to express it at all; fixed arity does not, and paying a node and +two arcs per entry is ruinous exactly where enumerations are largest — place +and route, where an agent choosing both which realization to place and where +would otherwise spend most of its encoder on its own action set. Deriving the +carrier from arity is what keeps one contract serving both. + +Three alternatives to the addressing were rejected, and each fails on a +different case. + +A flat product of decision kind against entity index cannot express a decision +that names a connection, and cannot express one that names a set of actors at +all. It also has to declare a static per-type stride and decode with the +dynamic one, so it either wastes the majority of its declared space or lets the +two strides disagree. + +A separate action-embedding tower avoids the stride but has to build its own +representation of the entity a decision targets, duplicating the encoder and +letting the two representations drift. When they drift, the head is scoring an +action against a picture of the state that the value head does not share. + +Scoring an action from its target alone — attention over entities with no +per-action term — collapses every alternative on one target into one +indistinguishable action. Choosing where to place a realization and choosing +which of two prototypes to substitute are then the same logit, which is the +entire decision in both environments. + +What the shared addressing buys is a head that factorizes cleanly. An action's +score is an anchor term over what it acts on plus a second term over what it +selects, both read from the graph, so an action carrying no alternatives is +scored directly on its node and one carrying alternatives is scored among them +while sharing its anchor term. That sharing is what lets experience with one +choice inform another on the same target, and the prior architecture's +per-instruction embedding exists for the same reason. + +Place and route is where this stops being a convenience. There an action's +value is a graph node — an occurrence, an endpoint, a traversal — and not an +ordinal in a catalog, so there is no enumeration of values a term could read +from. Only an addressing scheme in which the choice is a node has anything for +the second term to read, whichever carrier names it. + +The cost is real and is stated rather than hidden. Under either carrier the +observation grows with the live enumeration, and the enumeration bound is a +declared capacity that refuses an over-large state rather than truncating one: +truncating would silently hide whichever actions sort last, and the policy +would never learn that they existed. + +## Why Learned Place And Route Selects Existing Actions + +Place and route already had everything a learned search needs except a learner. +It owns a closed Action algebra, a deterministic dynamic domain over one exact +candidate, a transactional mutation mechanism that commits or rolls back +Mapping and Evaluation state together, and an objective closure that already +reduces a candidate to one integer. What it does not own is a good way to +choose which Action to propose next; its annealer draws one from weighted kinds +and uniform canonical domains, which is a deliberately unbiased choice and +therefore a deliberately uninformed one. + +So the learned environment replaces the selector and nothing else. The +alternative that was rejected was a second placer with its own occupancy model, +its own route cost, and its own notion of a partial mapping. It would have +needed a verifier to decide whether its states were legal, and the moment two +things can answer that question they eventually disagree — which is the same +argument that keeps the DSE environment out of the candidate-generator +business. + +Two rules invert relative to the DSE environment, and both inversions are +forced by what is fixed. There, the hardware moves and the workload's +mappability is a gate: a design that cannot run the program is outside the +space. Here the hardware is fixed and mappability is the objective, so a +candidate with unrouted obligations is an ordinary interior point rather than +an excluded one, and it would be perverse to reject the states the search +exists to pass through. And there, an unmasked action is not a promise that the +step advances, because feasibility is discovered downstream; here every +enumerated Action is a member of a domain the owner derived, so the only thing +a mask expresses is which phase the episode is in. + +Incremental placement was expressed as a sweep of rebinds over a complete +candidate rather than as a genuinely partial one, and that is a concession +worth recording as such. A partial candidate would route only settled +dependencies and do strictly less work; the reason it lost is that +`CandidateState` is complete by construction and four owners — the objective, +the handshake topological index, the movable-decision count, and the base +verifier — would each have needed a defined meaning for an unbound decision +that the ordinary product search has no use for. Teaching the product path a +state that exists only for a harness is a worse trade than paying redundant +routing inside the harness. + +The two arms exist because there are two honest answers to who fixes an +imperfect placement, and they measure different things. Handing the result to a +bounded annealer and charging for how far it had to move scores the learned +construction directly: the bound is what makes the recovered energy a statement +about the construction rather than about the annealer. Keeping the agent in +control and charging per repair scores something else — whether it can +recognize that a Mapping is good enough — and gives it routing and resource +Actions that a placement-only neighborhood cannot express. Collapsing them into +one configurable environment would have produced a record where most fields are +inert under most settings. + +That the cleanup bound became a search-policy field rather than a constraint +set was the one place performance decided a contract. The radius is expressible +today as a constraint set restricting each realization's placement domain to a +ball, at zero cost to any owner. But a constraint set is an input to freeze, +and freeze is the dominant cost in this environment; a per-episode one would +give every episode a distinct cache key and turn a warm cache into no cache. A +bound on which proposals a run may make is in any case a property of the search +rather than of the problem, so the field landed where that property belongs. + +Reward needed nothing new for the second time. A per-step signed energy +difference is potential-based shaping with the objective as the potential, so +an episode's return telescopes to the improvement it achieved while every step +still carries signal. The prior system reached the same place by hand, with a +five-tier weighted potential whose constants were spaced by powers of a +hundred; reusing the resolved closure gets the same shaping with the weights +owned by configuration and the arithmetic exact. + +## Why Some Design-Space Actions Read The Schedule + +Most design-space edits are structural. A prototype is swapped, a count is +adjusted, an inventory changes, and nothing about where the software currently +runs has any bearing on what the edit means. A few are not, and the exception +is worth stating because it looks at first like a layering violation: the +hardware search reading the software mapping to decide which hardware edit to +offer. + +Deleting an occurrence is the clarifying case. Structurally, removing a switch +or a function unit is trivial and almost always useless — every net whose route +went through it is now uncarryable, so the child fails to map and the search +learns only that deletion is bad. To be worth offering at all, the deletion has +to come with reconnection, and that is where the topology stops being able to +help. It admits a combinatorial number of possible reconnections and has no +opinion about which matter; adding all of them replaces a deleted node with a +worse connectivity problem than the one it removed. + +The current mapping does have an opinion, and it is the right one. The nets +that actually traverse the deleted occurrence are exactly the ones that need +somewhere else to go, and their upstream and downstream endpoints are exactly +the links worth adding. Reading the schedule turns an edit with a large useless +neighborhood into one with a small purposeful one, which is the difference +between a decision kind a policy can learn and a decision kind it learns to +avoid. + +The deeper reason it is worth the coupling is that the resulting child keeps +the parent's placement carryable. A structurally reconnected child usually has +to be re-mapped from scratch, which throws away both the probe cost and the +evidence that the parent's placement was good; a schedule-preserved one can be +warm-started from the mapping it was derived from. The edit and the schedule +stop being adversaries. That is also why the name is worth having: these are +not "mapping-aware" edits in general, they are edits whose whole point is that +an existing schedule survives them. + +What made this safe to admit was finding the seam that keeps it cheap. Whether +an occurrence *can* be removed is structural, so the size of the action set +still costs nothing to compute, and the environment's whole cheapest-first step +ordering — reject a revisit, reject an over-capacity state, and only then pay +for a probe — survives untouched. Only the contents of one action wait for the +mapping. Had the mapping decided how many actions exist, every capacity test +would have moved behind a probe and the ordering that makes the environment +affordable would have inverted. + +The environment still authors nothing. It selects references and the owners +decide what they mean and whether the result is legal, so a reconnection the +mapping suggested and the Fabric owner rejects comes back as an ordinary +rejection. Reading the schedule is evidence-gathering about which decisions are +worth offering, and it never becomes authority over which are valid. + +## Why The Shared Environment Contract Was Extracted + +The first ML environment defined its observation container, its action surface, +its Python package, its trainer conformance, and its benchmark obligations +inline, which was correct while it was the only one. The second needed all five +unchanged and none of the surrounding episode rules, which is the shape that +makes a shared owner worth the churn. + +The alternative was for the second environment to cite the first as the owner +of those five things. It was rejected because it makes an ordering claim that +is not true: neither environment is upstream of the other, and a reader of the +place and route contract would have to consult a design-space-exploration +document to learn what a graph observation is. Ownership in this stack is +supposed to name where a fact lives, not which document happened to need it +first. + +What stayed behind is the part that is genuinely per-environment, and the +dividing line is worth stating because it is not obvious. The container, the +roles, the index correspondence, and the buffer rules are shared; the column +catalogs are not, because the facts worth exposing about a fabric being edited +and about a mapping being built barely overlap and a shared catalog would be +their union, carrying an inapplicable majority in either. The same reasoning +splits the benchmark contract: what every harness owes is shared, and the +stages it decomposes into are not. + +The models split on exactly the same line, and for a reason that is worth +recording separately: almost everything between an observation and a logit is +the same problem twice. Batching ragged graphs, refusing to feed a catalog +ordinal to a linear layer, a pre-normalized edge-aware trunk, multi-scale +pooling, a bounded value head, and — most of all — the masking rules are +independent of what is being searched, and the masking rules are the ones it +would be most expensive to get subtly different in two places, since every one +of them protects a training-time invariant that no loss curve reveals when it +breaks. What differs is the policy head, which is where the two searches +actually differ: one scores a decision against a catalog of values, the other +scores a pair of nodes. + +The parallel is not quite exact, and the difference says something. The two +environments share a data contract; the two models share a data contract *and* +a set of correctness rules. That is why the model core carries prose about +importance ratios and precision that reads more like a hazard list than a +schema: those are the parts where an implementation can be fast, plausible, and +silently wrong. + +## Why The Placement Choice Is Scored Against Its Anchor + +The prior architecture scored hardware slots from the slot's own embedding and +a pooled graph context, and conditioned on the software node being placed only +through the candidate mask. That is a defensible factoring and it is cheap: the +slot head runs once per slot, not once per pair, so its cost is the hardware +graph rather than the enumeration. + +It is also unable to represent most of the problem. Scored that way, a policy +learns which slots are good in general — central, uncongested, well connected — +which is a real signal and the wrong one. Placement quality is a property of a +pair: a producer wants to be near its consumers, and whether an occurrence +satisfies that depends entirely on which realization is being placed and where +its neighbours already are. A head that cannot see the pair can rank slots but +cannot match them. + +So the choice term reads the anchor's embedding too, and the cost is that it is +evaluated once per available action rather than once per node. That cost was +worth paying, and the two responses to it are worth distinguishing, because the +distinction was nearly lost. + +One is exact and is simply adopted: an algebraic rearrangement that avoids +materializing a concatenation, which is a memory saving and not an arithmetic +one. Promising it as an arithmetic reduction would have claimed something the +algebra does not contain. + +The other is not exact. A genuine per-pair reduction exists, and it computes a +different and strictly less expressive function. That makes it a modelling +decision wearing a performance costume, so it is admitted as a configured +variant measured against the reference rather than substituted for it. The +honest place for an optimization that changes what the model computes is the +configuration, where someone has to choose it and a checkpoint records which +was chosen. + +## Why Pretraining Imitates The Destination, Not The Path + +Place and route already has a search that works. Simulated annealing produces +good placements, slowly, and the obvious way to bootstrap a learned policy is +to show it what the annealer did. The prior system did precisely that: it +recorded a greedy fill followed by every accepted Metropolis move, and trained +on the sequence. + +That trains the wrong thing. An annealing run is a walk that spends most of its +time in states worse than where it started — that is what an acceptance kernel +is *for*, and a run that never occupied a worse state was not annealing. A +policy fitted to that sequence learns to place badly and then shuffle, because +that is what the demonstration shows. The behaviour is faithfully reproduced +and useless. + +What is worth copying is where the walk ended. So the generator keeps the final +placement, throws the path away, and synthesizes a clean construction sweep +that reaches that placement directly — each realization bound once, nothing +undone. The demonstration is then a claim about good placements rather than a +recording of a search, which is the thing a construction phase can actually +imitate. + +One consequence has to be stated rather than smoothed over: the synthesized +sweep does not reproduce the annealer's energy. Placing in a different order +routes the nets differently, so the same final placement carries a different +number. The replayed value is the honest one because it is the state the policy +will actually occupy, and recording the annealer's would describe a candidate +the demonstration never reaches. + +This also explains why demonstrations cover construction and stop. The repair +phase has no destination to imitate — repair *is* the walk — so it is learned +online, from reward, where a walk is the right thing to learn. + +## Why Pretraining And Online Training Are One Run + +They could have been two runs joined by a checkpoint path, which is what the +prior system did. Making them stages of one run buys three things that +arrangement cannot. + +The handoff becomes checkable. A checkpoint path passed between two invocations +is validated by neither, and the prior system documented two ways it silently +failed: a restore aimed at a component path that does not exist loads nothing +and reports success, and an evaluation that reads weights from a sampler rather +than the learner reports the parameters from before the load. Both produce a +run that looks warm-started and is not. As a stage boundary inside one run, +loading is an obligation with a reported parameter count and tensor digest. + +The reference for improvement becomes well defined. The question the two stages +exist to answer is whether online training improved on what the demonstrations +delivered, and that is a comparison against the parameters at the boundary. Two +separate runs have two separate histories and no shared reference; one run has +the boundary reading built in. + +And the things that must agree are forced to agree. `gamma` is the sharp case: +the reward is an exact energy difference, so a discount is a statement about +which future improvements count, and a pretraining stage that discounted +differently would fit a value head to a different return than the online stage +optimizes. Two runs can disagree about that silently. One run's adoption +rejects it. + +## Why Evaluation Compares Against A Run's Own History + +The tempting baseline is the annealer — it is what the demonstrations came +from, and beating it is the point. It was rejected for evaluation *during* +training on cost: an annealing invocation per case per evaluation is the +dominant cost in this environment, and a test protocol that ran one would cost +more than the training it measures. That comparison is a question about a +finished checkpoint, and it runs offline against recorded results. + +What a run needs while it is running is whether it is getting better, and the +honest reference for that is its own earlier weights on the same cases. Hence a +per-case series against a fixed test set, with each evaluation reporting its +difference from the previous one and from the most recent stage boundary. + +Reporting per case rather than per aggregate is the part that matters. An +aggregate hides the case that regressed, and a policy that improves its mean +while losing its hardest problems is exactly the failure mode a placement +policy falls into — the easy instances are numerous and the hard ones are the +reason anyone wanted a learned placer. A mean over them would have called that +progress. + +## Why The Mask Is Packed + +The observation carries its arrays in the narrowest form that represents them, +and the action mask goes further and is packed to one bit per outcome. Both are +unusual enough in a specification to need a reason, because storage width is +ordinarily an implementation's business. + +It is here because of what an observation is used for. An observation is not +produced once and read once. It is produced tens of millions of times per +experiment, it is copied into a sample buffer, it is carried across a process +boundary to a learner, and then — because the mask an action was sampled under +has to be the mask it is learned under — it is held for as many epochs as the +algorithm takes. Every byte in an observation is multiplied by the batch size, +by the epoch count, and by the number of environment copies a sampler runs. +That multiplier is what turns a storage question into a throughput question: +past some batch size the run stops being limited by arithmetic and starts being +limited by how much of the batch fits in memory at once, and the only lever on +that is how wide the batch is. + +The widths themselves were free to take. An `ArcRole` has three values and was +occupying eight bytes on the most numerous extent in the observation; a role, a +kind, and a boolean placement flag were doing the same on the second most +numerous. Nothing was gained by that uniformity, and no catalog had to grow to +give it up, since the width a value needs is already fixed by the owner that +bounds the value. + +The mask is the sharp case and is worth separating from the rest. It is the one +array whose size is a capacity rather than a state. Everything else in the +observation costs what the state actually needs — the graph is ragged and the +decision instance is exactly the live enumeration — but the mask spans +`enumeration_bound + 1` outcomes at every step of every episode regardless of +how few are live. And a capacity is chosen with headroom, deliberately, so most +of the mask is usually inert. Narrowing it to one byte per outcome would still +leave it the largest fixed term in an observation at the bounds these searches +use. Packing to one bit is what makes a capacity chosen generously stop costing +what it was chosen at, which in turn means a bound can be set for the worst +state a run might reach rather than for the memory it would cost to allow it. + +Packing is also, unusually, the more honest representation rather than a +compression of a nicer one. A mask entry is a bit — there is no state it can +carry that a bit cannot — and the byte-per-entry form was the encoding that +added information the mask does not have. Several of the masking rules read +as consequences of that representation rather than as constraints imposed on +top of it. The one obligation packing genuinely adds is zeroing the pad bits, +and that is the price of the observation having a canonical byte form at all. + +What is deliberately not claimed is that any of this is free. Unpacking and +widening are real work and they happen on every forward pass. The bet is that a +shift and a cast on a device that was about to touch the value anyway costs +less than moving eight times the bytes to reach it. What makes that a +measurement rather than an assertion is that the model harnesses report the +bytes and a reference run that widens at the boundary instead — a stage that +only existed when the saving was discarded would have answered a different +question. + +## Why The ML Environments And Models Are Benchmarked + +Everywhere else in Loom, speed is an engineering concern that the contracts +mention only where it changes admission. Here it is in the specifications, with +named harnesses, named stages, and required breakdowns. That difference needs a +reason, and the reason is that the environment and the model sit on the inner +loop of every experiment. + +A compile runs once per design. An environment step runs tens of millions of +times per training run, and the model runs once per step on top of it. A factor +of two in either is a factor of two in the wall clock of every experiment +anyone runs afterwards, which is not a performance detail but the difference +between a curriculum that can reach large designs and one that cannot. Leaving +that to be discovered informally means discovering it after the schedule has +already been built around the slow version. + +The stronger reason is that both contracts deliberately offer options whose +cost is invisible without measurement. Warm-starting a probe from the parent +trades a cache and path-independent energy for throughput. Interleaving +per-workload regeneration with probing pays off only if early abort is common. +Retaining frozen models across episodes is what makes place and route resets +affordable at all. Emitting route arcs makes the observation much larger for a +policy that may not need them. Each is an option the specification states, and +an option nobody measures is an option nobody can actually choose; writing the +measurement into the contract is what keeps those from becoming defaults that +nobody revisits. + +One aggregate number would answer none of it. A step is a pipeline of +completely unrelated owners — a draft finalization, a Mapping probe, an +inference call, a marshalling boundary — and a single latency moves for reasons +it cannot attribute. Decomposing into named stages is what makes a regression +locatable, and naming the stages in the specification rather than in a tool is +what keeps two implementations comparable. + +The two mandatory breakdowns are mandatory because both hide real regressions +when blended, and a training run moves exactly the mixes they separate: a +policy's failure mix moves constantly, so a genuine probe slowdown can hide +behind a rising early-rejection rate and read as an improvement. + +Instrumentation is off outside the harness because measuring it would change +it, which is also why the contract refuses to let harness stage times be +compared against real throughput. + +The model is measured in two regimes because inference and training have +opposite profiles and an optimization that helps one routinely harms the other, +so one averaged number reliably optimizes the wrong half. The split against the +environment step comes first for a related reason: without it, the obvious work +is to speed up whichever half is already small. + +Everything these harnesses produce is nonsemantic and removable, and that is +load-bearing rather than administrative. A benchmark result must never become +something a selection, promotion, or conformance decision can consume, because +a timing is a property of a machine on a day and admitting one as a durable +fact would let hardware variation reach a semantic outcome. That is also why +budgets are ratios against a recorded baseline tuple of exact identities and +configuration digests rather than absolute durations — an absolute duration +encodes a machine — and why deterministic work-unit counts are reported +alongside the timings and labelled as the cross-machine measure. The wall time +is for the engineer; the work units are the number that means the same thing +twice. diff --git a/docs/spec-adg-builder.md b/docs/spec-adg-builder.md index d73b44980..c5064da10 100644 --- a/docs/spec-adg-builder.md +++ b/docs/spec-adg-builder.md @@ -1483,6 +1483,12 @@ deterministic elaboration only. There is no generic hardware action language, mutable candidate graph, caller-authored property bag, or DSE-only construction path. +A caller that requires a verified candidate without an Artifact takes the +`VerifiedFabricClosure` owned by +[Finalization And Publication](spec-fabric-artifact.md#finalization-and-publication) +and does not publish it. The Builder gains no second finalization path from +that choice. + ## Conformance Anchors The stable Builder anchors are deliberately small: diff --git a/docs/spec-fabric-artifact.md b/docs/spec-fabric-artifact.md index 7a84aec99..ade4223c6 100644 --- a/docs/spec-fabric-artifact.md +++ b/docs/spec-fabric-artifact.md @@ -328,6 +328,23 @@ reimported the result cannot return `FinalizedFabricRoot` or claim artifact success. It must return the typed unavailable, invalid, incomplete, or store failure owned by the first unsatisfied stage. +The pipeline has exactly one named intermediate. A `VerifiedFabricClosure` is +the value that exists after independent reverification and before +`ArtifactStore::put`: canonical bytes, the computed candidate +ArtifactIdentity, and the validated dependency closure, with no store object. +It is not a reduced finalization mode, because every semantic stage above has +already run and none may be skipped to reach it; it is the same derivation +observed one step before its terminal. Publication consumes a `VerifiedFabricClosure` and +returns the published `ArtifactRootReference`. + +A caller that requires an Artifact publishes. A caller that only requires a +verified candidate and its exact identity, such as a search harness evaluating +a candidate it may discard, may hold the closure and publish later or never. +A `VerifiedFabricClosure` is a transient in-process value, not an Artifact +family, a persistent schema, or a second identity authority: until publication +no other owner may reference it, nothing may depend on it, and any value +derived from it is a removable projection. + Fabric failure atomicity means one root object is complete or absent; it does not mean that the root and its dependency graph become visible in one transaction. Dependencies are independently valid, immutable, shareable diff --git a/docs/spec-loom-stack.md b/docs/spec-loom-stack.md index 508e1728f..74b2259e9 100644 --- a/docs/spec-loom-stack.md +++ b/docs/spec-loom-stack.md @@ -751,6 +751,40 @@ identity remain Loom-owned; Loom does not maintain a gem5 patch stack or edit the pinned submodule source. A gem5 upgrade is a separate exact dependency change with Runtime ABI and System simulation conformance. +Ray is the one modified dependency. It is a Loom-owned fork at +`externals/ray`, and Loom does maintain its patch stack. An upgrade is a +separate exact dependency change: it selects an exact upstream Ray release, +rebases the patch stack onto it, reruns the ML environment's conformance +anchors, and atomically pins both the resulting fork commit and the upstream +release tag it was built from. Recording only the fork commit would be +insufficient, since the patch stack is the difference between the two and a +fork commit alone does not say what it is a patch stack against. + +The patch adds graph-space support to the sampler and connector pipeline, which +[ML Environment Core](spec-ml-core-environment.md#rllib-environment-definition) +requires because every ML search environment's observation is a variable-size +graph. That capability is the only reason the fork exists; a patch that is not +required by a Loom-owned contract belongs upstream instead. Carrying a patch +stack is a standing cost recorded here rather than a precedent for modifying +any other pinned dependency. + +The fork is admitted on a narrower footing than the others: no Artifact, +Evidence, or semantic configuration schema depends on it, so a rebase can break +training without invalidating a semantic artifact. It does reach the search +harness's own training views, which are digest-covered like any other, so a +rebase that changed their vocabulary would change a training run's identity. +That is the intended blast radius, and it is the whole of it. + +Every Python runtime a search-harness document states conformance against is +pinned here to an exact version, and the pinned Ray fork is required to be +compatible with all of them. Currently that is the Gymnasium release the +[ML Environment Core](spec-ml-core-environment.md#rllib-environment-definition) +targets and the PyTorch release +[ML Model Core](spec-ml-core-model.md#module-boundary) targets. They are pinned +here rather than in the documents that consume them, so that a second ML +document adopting either does not give the stack two places to disagree about +which runtime it runs on. + ## Verification Boundary Tests protect stable semantic anchors: canonical schema and identity, diff --git a/docs/spec-ml-core-environment.md b/docs/spec-ml-core-environment.md new file mode 100644 index 000000000..376919685 --- /dev/null +++ b/docs/spec-ml-core-environment.md @@ -0,0 +1,826 @@ +# ML Environment Core + +This document defines the contract every Loom reinforcement-learning search +environment shares: how a search state is projected into one observation, how +an action index addresses one member of a live enumeration, how an episode's +transitions and endings are accounted for, how reward crosses the integer +boundary, how a parallel sampler stays reproducible, and how the whole thing is +presented to Python and measured. + +It defines no design space of its own. What is being searched, which decisions +exist, what makes one legal, and what an episode is trying to achieve are owned +by the environment documents that build on this one: + +- [ML DSE Environment](spec-ml-dse-environment.md) searches Loom's design space + by selecting candidate-generator decisions over a Fabric subject; and +- [ML PnR Environment](spec-ml-pnr-environment.md) searches one fixed Mapping + problem by selecting typed Place and Route Actions. + +Two environments already share more machinery than either owns alone, which is +why the machinery is here rather than in whichever document happened to need it +first. A third environment appends a bullet above and states the small set of +obligations each section below names; it does not reopen this contract. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [Evaluation and DSE](spec-dse-feedback.md#objectives-and-quality-gates) owns + `ObjectiveDimension`, `ExactAffineQuantization`, `ObjectiveVector`, + `WeightedLevel`, `TotalOrdering`, `SearchEnergyRef`, and the statement that a + reward is the signed difference of a selected search energy; +- [Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism) + owns the seeded PRNG protocol, its `nextBounded` projection over canonically + sorted domains, and its prohibition on host entropy and library + distributions; +- [Evaluation Metrics](spec-evaluation-metrics.md) owns `ExactRatio`; +- [Resolved Configuration](spec-config-ssot.md#component-views) owns + component-view framing, canonical view bytes, and `component_view_digest`; +- [Operational Observations](spec-dse-feedback.md#operational-observations) + owns the nonsemantic status of wall time, the prohibition on summing + concurrent times, and deterministic work summaries as the cross-machine cost + measure; and +- [Full-Stack Architecture](spec-loom-stack.md#external-dependency-pinning) + owns the exact revisions of every search-harness dependency, including the + Ray fork the RLlib section targets. + +The RLlib environment definition, its env context, its registration mechanism, +and its sampling topology are owned by that external dependency. This document +states which of its obligations bind an environment and does not restate, +extend, or version them; the training document that configures an environment +configures it. + +This document owns only the observation container, its combined node-and-link +space, the two enumeration encodings, and the storage form of every observation +array including the packed mask layout; the action surface and masking algebra; +the step accounting identity and the termination-versus-truncation rule; the +reward boundary; the determinism record, the PRNG preimage shape, and the +episode-start override contract; the trajectory record's status; the `loomml` +package layering and its interaction contract; and the benchmarking obligations +every harness satisfies. + +## Nonsemantic Boundary + +An ML environment is a nonsemantic search harness. It is not a Candidate +Generator, Mapping, Evaluation, objective, configuration, or Artifact +authority. It defines no action language of its own and no second design space. +Every decision it applies is an existing owner-typed decision, every legality +answer comes from that decision's ordinary owner, and every reward is a +projection of the objective algebra owned by +[Objectives and Quality Gates](spec-dse-feedback.md#objectives-and-quality-gates). + +No environment operation publishes an `EvaluationRequest`, +`EvaluationEvidence`, an `InvocationManifest` record, or a lineage edge, and no +environment value enters a Fabric, Mapping, or Evaluation Artifact. An +observation column, a reward, a mask bit, a step record, and every output of +every harness in the ML stack are removable projections; regenerating them may +change presentation but must preserve every referenced semantic fact, and no +consumer may treat one as a semantic result. Nothing on that list is an +Artifact, Evidence, or a report schema, and no selection, promotion, or +conformance decision consumes one. + +An environment introduces no persistent schema family beyond its own resolved +configuration, which is an ordinary component view. Its episode state is +process-local and its trajectory record is a transient value: a trajectory is +not an Artifact, not Evidence, and not a plan, and it carries no outcome, no +reward, and no selection claim. Only the steps that advanced are retained, so a +non-advancing attempt stays an episode-local observation and never becomes +fake history. + +An environment run is therefore never an authority for selection. A result an +environment discovers becomes real only when its environment document states an +ordinary owner-owned path that reconstructs it. + +## Combined Node And Link Space + +Decisions target entities and relations alike. A decision may name an +occurrence, a connection, a transport link, an actor, a graph, or a route, so a +projection that represented only entities as graph nodes would leave every +relation-valued decision pointing at nothing. The observation is therefore a +combined node-and-link space: an entity that another representation would draw +as an arc is instead a node, and the arc it replaces becomes two arcs through +it. + +Nodes are laid out in role blocks in this order, each block dense and in +canonical owner order: + +```text +GraphNodeRole = + FabricOccurrence // 0 + | FabricConnection // 1 + | DataflowOperation // 2 + | DataflowValue // 3 + | Decision // 4 +``` + +Ordinals are stable. A new role appends; reordering, deleting, or repurposing +one is an incompatible change to this document's schema version. An environment +whose state contains no member of a block contributes no rows for it and needs +no flag to say so, because its own contract already decides which blocks exist. + +A `FabricConnection` node replaces the direct arc it stands for: an original +connection from occurrence `u` to occurrence `v` contributes node `l` and the +two arcs `u -> l` and `l -> v`. `DataflowValue` nodes are promoted from values +under the same rule. A parallel connection is one node carrying its current +count as a feature, not one node per parallel lane, so a decision that changes +a count has exactly one target and never changes the node inventory. + +"Decision" is this document's generic name for one member of the live +enumeration; an environment document names what its members actually are. The +`Decision` block is present exactly when the environment uses the +`DecisionNodes` encoding defined below. + +The space is closed under decision targets. For every member of the live +enumeration, every owner-local reference its normalized payload carries +resolves to exactly one node ordinal in this graph. A reference that resolves +to no node, to more than one node, or to a node outside the current state is a +projection defect rather than an unaddressable action. This closure is what +lets the enumeration and the graph agree: a policy scores a decision by +attending over the nodes it names, so an action is per-node and per-link even +though the action index itself is an ordinal in a canonical order. + +## Enumeration Encoding + +The closure rule says every live entry names graph nodes. How it names them is +one of two forms: + +```text +EnumerationEncoding = + DecisionNodes // 0 + | DecisionColumns // 1 +``` + +Under `DecisionNodes`, each live entry is a node in the `Decision` block and +each reference it carries is an arc from that node to the node it names. An +entry's target count is its out-degree, its target roles are arc features, and +its own categorical and numeric properties are columns of the node matrix. A +decision's ordinal within the `Decision` block is its `ActionIndex`. + +Under `DecisionColumns`, the `Decision` block is absent and the enumeration is +carried alongside the graph as its own dense matrix, one row per live entry. +The references an entry carries are columns holding node ordinals into the same +instance's graph, and its own properties are further columns of the same row. +An entry's `ActionIndex` is its row ordinal. + +The choice between them is not free. **An enumeration whose entries have +variable target arity uses `DecisionNodes`; one whose entries have fixed arity +uses `DecisionColumns`.** A decision that names one entity plus a reference set +of unbounded size needs out-degree to express its targets, and columns would +force either a maximum arity or an offset table beside the matrix. An action +that names exactly an anchor and a choice needs two columns, and a node with +two arcs says the same thing while adding a row and two arcs to the graph for +every member of an enumeration that is routinely far larger than the state it +ranges over. Neither form is a preference: an environment whose arity is fixed +and that chose nodes anyway would spend most of its encoder on its own action +set. + +What is common to both is the part a policy actually consumes. An entry names +graph nodes, those nodes carry the encoder's representation of what the entry +acts on, and the entry's index is an ordinal in the canonical order. A model +scores an entry from the embeddings of the nodes it names under either +encoding; what differs is where it reads the naming from. + +## The Graph Instance + +The observation is one variable-size graph, carried by the Gymnasium `Graph` +space and produced as one `GraphInstance` per state. It is not padded, not +bounded, and not reshaped to a fixed extent: the node count, arc count, and +their feature matrices are exactly the size the state requires, and a state +that needs more room simply produces a larger instance. + +```text +Observation { + graph: GraphInstance + decisions: GraphInstance + action_mask: packed bitmask over enumeration_bound + 1 outcomes + objective_codes: dense array + scalar_features: dense array +} + +GraphInstance { + nodes: per-column array + edges: per-column array + edge_links: dense array<(arc ordinal, 2), uint32> +} +``` + +`decisions` carries the enumeration under `DecisionColumns` and is empty under +`DecisionNodes`, where the enumeration is already in `graph`. It is a +`GraphInstance` with rows and no arcs rather than a bare matrix, because the +observation already needs one ragged per-instance carrier and this is it: the +same space, the same batching, and no second capability asked of a consumer +that can already carry one. Its `nodes` matrix is indexed by live entry ordinal +over the environment's own `DecisionColumn` catalog, and its `edges` and +`edge_links` are empty. + +Under `DecisionNodes`, an entry's targets are arcs from its decision node, so +its target count is its out-degree and nothing needs an offset array, a slot +table, or a per-decision count column. This is the same move the combined space +makes for connections: an entity with a variable number of relations becomes a +node, and the relations become arcs. + +`objective_codes` holds the current directed code of each dimension in the +environment's selected closure, in ascending `ObjectiveDimensionRef` order, +exactly as produced by the quantization owned by Evaluation and DSE. It is the +state's objective position, not a reward. + +`NodeFeatureColumn`, `ArcFeatureColumn`, `ScalarFeatureColumn`, and +`DecisionColumn` are closed catalogs each environment document owns, because +the facts worth exposing about a Fabric being edited and about a Mapping being +built are not the same facts. A shared catalog would have to be the union of +both and would carry an inapplicable majority in either environment. Six +obligations bind every such catalog: + +1. `NodeFeatureColumn` ordinal zero is `Role`, carrying the node's + `GraphNodeRole`, so a consumer can interpret every other column without + knowing which environment produced the instance; +2. a column that does not apply to a node's role is encoded as negative one, so + one node matrix with role-conditioned columns keeps every node addressable + by one ordinal, which is what lets a policy attend over entity and decision + nodes together; +3. `ArcFeatureColumn` ordinal zero is `ArcRole`, whose ordinal zero is + `Structural`, whose ordinal one is `Placement` where the environment + projects a placement relation, and whose ordinal two is `Route` where it + projects a routing relation; an environment appends its own roles after + those and omits the ones it does not project. A target arc's role is carried + by `ArcRole` itself rather than by a second column that would be + inapplicable on every structural arc. An environment with no second arc + column declares its `ArcRole` catalog and no `ArcFeatureColumn` wrapper, + since a one-member catalog is a name for its member; +4. a categorical column holds an owner catalog's ordinal rather than an + environment-local enum wherever such a catalog exists, so a new owner kind + extends the observation without a new column and without a new role; and +5. a `DecisionColumn` that names a target holds a node ordinal into the same + instance's `graph`, and the target-closure rule binds it exactly as it binds + a target arc. A `DecisionColumn` catalog is empty under `DecisionNodes`; and +6. a node catalog is extended by one capacity column and one usage column per + member of the state's fixed canonical `FabricResourceStateRef` catalog, in + that catalog's order, exactly when the environment sets + `include_resource_states`. The appended group is fixed for the episode, so + the catalog keeps a fixed column count even though its row count varies. + This group is stated + here rather than per environment because every environment appends the same + one from the same owner catalog; an environment needing a second, different + appended group states that one itself. + +Every array is C-contiguous with the declared element type; its lifetime and +the prohibition on writing it are owned by +[Interaction Contract](#interaction-contract). Column order within each catalog +is part of this contract; row counts are not, and no consumer may infer a state +fact from an extent beyond the count that extent literally is. + +### Storage Widths + +Every observation array is stored in the narrowest form that represents its +declared value domain exactly. Nothing is stored wider to make two arrays the +same type. + +The domain fixes the form, so no catalog declares a width. A boolean domain is +one bit per entry, packed. A bounded-ordinal or bounded-count domain is the +narrowest integer holding it: a categorical column's from its owner catalog's +cardinality, a counting column's from the bound its owner already places on +what it counts, a magnitude's from the range its quantization declares. A +column that uses the negative-one absence encoding is signed and its width +accounts for that value; a column that is never absent is unsigned. Requiring a +declaration as well would put a second source beside a fact the owner already +fixes, and the two would eventually disagree about a column neither had +changed. + +`scalar_features` is the one named exemption and stays `int64`: it is one fixed +short array per observation, so narrowing it would buy nothing measurable +against a per-column rule where a single type reads more clearly. + +Because widths differ per column, `nodes` and `edges` are column-major: one +contiguous array per column, in catalog order. They are not one +two-dimensional matrix. This costs nothing a consumer wanted — the model +embeds and projects each column separately, so it indexes columns individually +in either layout — and it is what lets a one-byte column occupy one byte per +row instead of being widened to its matrix's element type. + +A column spans only the role blocks it applies to, and is stored over exactly +those blocks' contiguous ordinal ranges. Blocks are already dense and in a +fixed order, so a column that is inapplicable to a role has no entries for it +rather than a run of absence values. Addressability is unaffected, since a +block is a contiguous range of node ordinals. The absence encoding remains for +a column that is inapplicable to some member of a block it does span. + +A consumer that needs a wider or unpacked form produces it after transfer, on +the device that consumes the value, and never in the environment or in a layer +between them. Widening at the boundary would restore exactly the bytes this +rule exists to avoid, in the hottest place and for the whole journey through +the connectors and the sample buffer; widening on device is a cast against +arithmetic that was going to touch the value anyway. This is the only rule +about the two forms: nothing else in this document distinguishes a value by +which one it is in. + +Why the observation is worth this much attention, and why the mask is the +sharpest case, is +[Why The Mask Is Packed](rationales/ml.md#why-the-mask-is-packed). + +#### The Packed Mask Layout + +`action_mask` is the one boolean-domain array, so it is the one array the rule +above packs. Bit `i` is bit `i & 7` of byte `i >> 3`, least significant first, +making the array `ceil((enumeration_bound + 1) / 8)` bytes of `uint8`, and +every bit above `enumeration_bound` in the final byte is zero. + +Zeroing the pad bits is what keeps two states with the same admissible set +byte-identical. Leaving them undefined would make the observation's bytes +depend on whatever the producer's buffer last held, and would do it only for +states whose live count is not a multiple of eight — breaking the +byte-identical-enumeration property +[Action Surface And Masking](#action-surface-and-masking) states, +intermittently. + +The bit order is fixed here rather than left to a producer because both sides +index it. A mask is the one observation member where a layout disagreement is +silent: a wrong feature column trains a worse policy, while a wrong bit order +masks a different action than the one the environment refused, and the first +symptom is an importance ratio that is wrong for reasons no loss curve +explains. + +`graph` and `decisions` are the only parts of the observation whose extent +varies, and neither is padded for the same reason: a bound large enough for the +largest reachable state wastes most of every batch on inert rows, and this is +most acute for `decisions`, whose row count is the live enumeration length and +whose capacity would have to be `enumeration_bound`. `objective_codes` and +`scalar_features` are fixed-length over closed catalogs, and the action mask is +fixed-length because a Gymnasium action space is fixed at construction and +`Discrete(enumeration_bound)` cannot vary per state. What distinguishes the +mask is that its extent is a declared capacity plus the stop entry rather than +a real count: decision entries at or beyond the live enumeration length are +clear, and the observation carries no entry for them under either encoding. + +## Action Surface And Masking + +The interaction surface is: + +```text +action = { + decision: Discrete(enumeration_bound) + stop: Discrete(2) +} +``` + +`enumeration_bound` is a static capacity so the space has a fixed declared +shape, while the live enumeration length varies with the state. A policy +therefore scores real decisions rather than a fixed factorization of decision +kind against node index. This replaces a flat product space over node slots and +decision types: such a space must declare a static per-type stride and decode +with the dynamic one, which either wastes the majority of the declared space or +lets the two strides disagree. + +An environment's enumeration is a pure function of its current state and its +resolved configuration, so two runs with equal state and configuration produce +byte-identical enumerations in the same order. + +Masking is the primary invalid-action mechanism. An index at or beyond the live +count, an index whose mask bit an environment cleared, and an out-of-range +index are each errors: the environment refuses the call, leaves the state +unchanged, and reports the violated precondition. A masked index is not +silently coerced to a no-op, to the nearest legal index, or to a terminal step, +because each of those makes the training signal indistinguishable from a real +decision. + +`stop` set to one ends the episode electively at the current state and is +mutually exclusive with advancing; the `decision` field is ignored in that +call. An elective stop is the only outcome the policy itself controls. + +`action_mask` covers one more outcome than `enumeration_bound` because it +covers the stop outcome as well as the decision slots, and it is indexed +exactly as the action distribution's outcomes are: bit `i` below +`enumeration_bound` masks the decision at that index, and bit +`enumeration_bound` masks stopping. One mask indexed one way is what lets a +model apply it additively to its logits without a second layout to keep in +step. + +An environment may therefore clear stop for a state in which electing to stop +has no meaning. What it may not produce is a *non-terminal* state with no +admissible outcome at all: such a state has no defined action distribution, and +reaching one while the episode continues is a contract violation rather than an +ending. Packed, that test is whether every byte of `action_mask` is zero, which +the pad-bit rule makes exact. + +The mask of an observation returned alongside an ending is never a distribution +input. No action is sampled from it, the value head does not read it, and no +consumer composes it or counts it as a fully-masked fallback. It may therefore +be all-clear, and an environment with no admissible outcome at a terminal state +returns one rather than manufacturing an outcome that does not exist. + +A state whose enumeration exceeds `enumeration_bound` is refused rather than +truncated, because truncation permanently hides whichever part of the +enumeration sorts last. The refusal names both the capacity and the required +length, so the configuration can be corrected rather than guessed at. +`enumeration_bound` is the only capacity in this contract, because the +observation graph has none. + +## Step Accounting And Episode Endings + +Every step carries at most one transition, and carries none exactly when it is +an elective stop, so an episode record satisfies + +```text +steps == advanced + non_advancing + elective_stops +``` + +by construction with no unaccounted remainder. `non_advancing` is the class of +steps that carried a transition which did not take effect. Each environment +names that class in its own vocabulary — one search calls it a rejection and +another a failure — and reports its count and its reasons under that name. What +is fixed here is the accounting, not the word. + +A transition and an episode ending are separate facts because they co-occur: a +non-advancing step may both fail and end the episode, an advance may both +advance and +land in a state with no admissible decision, and an elective stop ends the +episode with no transition at all. A single three-way union could express none +of those without discarding half of what happened, so a step result carries +both, and its ending member is present exactly when the episode is over. An +episode records exactly one ending, on its final step, whichever transition +that step did or did not carry. + +An ending an environment reaches by advancing into a state with no admissible +decision is reported on the step that produced that state, not on a later call, +so a consumer is never handed a state it has no legal index to act on. + +An ending is either a termination or a truncation, and the two are reported +distinctly. A termination means the episode reached a state its contract +defines as final, and a value estimate must not bootstrap past it. A truncation +means the episode was cut off by a harness limit while the state remained +ordinary, and a value estimate must bootstrap. Reporting the two as one flag is +invalid. + +An ending an environment defines as a truncation is itself priced at zero. +Charging a penalty for a limit the policy did not choose teaches it to avoid +states that merely take longer to leave, so among endings only the one the +policy elects carries a price. + +That rule prices the *ending*, not the state the episode ended in. An +environment may charge for a property of its final candidate on whichever step +ends the episode, truncating or not, because such a charge is about something +the policy produced rather than about a limit it did not choose. The two are +told apart by what they read: an ending price reads the terminal reason and +nothing else, and a terminal state charge reads the state and never the reason. +An environment that charged more for the same final state under one ending than +under another would be pricing the limit through a state charge, which this +rule forbids in either spelling. + +## Reward Contract + +Reward is the signed difference of a selected search energy across a +transition. Energy is the value of the selected `WeightedLevel`, its checked +`uint128` arithmetic, and the sign-plus-magnitude form of its difference are +all owned by +[Objectives and Quality Gates](spec-dse-feedback.md#objectives-and-quality-gates); +an environment document states only which two states the difference is taken +between and what its non-transition outcomes cost. + +With energy as a potential, a per-step signed energy difference is +potential-based shaping, so an episode's undiscounted return telescopes to the +total improvement between its first and last state while every step still +carries signal. An environment states only what breaks that property for it. + +The energy is a directed code, so a decrease is an improvement. The reported +sign is positive when the child energy is below the parent energy and negative +when it is above. Every conversion, subtraction, product, and sum is checked; +an overflow is a resolved-policy failure, never a clamp, a saturation, or a +candidate penalty. A parent's energy is retained rather than recomputed, so a +step evaluates one energy, not two. + +The native environment emits the exact `(sign, magnitude)` pair and nothing +else. It performs no scaling, normalization, clipping, discounting, or shaping. +The `float64` the RLlib environment definition requires is produced in +`loomml.env`, which applies one `ExactRatio` scale, and every other reward +transform belongs to the training document that configures that environment. +Keeping the conversion there is what lets the core stay integer-only and keeps +a mantissa width out of a canonical, digest-covered configuration; the adapter +is also the only layer that can state the exactness condition its own float +type imposes. + +That scale is `reward_adapter.scale`, a training-view field. `loomml.env` +binds only an environment view, so the layer that reads a training +configuration passes the resolved ratio down at construction and `loomml.env` +applies the ratio it was given. Putting the scale in the environment view +instead would make every scale change a new environment digest and discard that +environment's caches for a number the environment does not use. + +Reward codes an environment charges for non-transition outcomes are stated as +`uint64` magnitudes in the energy domain and applied with the sign the outcome +fixes, so no configuration view carries a signed or floating-point number. Such +a code is a declared constant and is never derived from the state, so a step +that carried no evaluated candidate can never be scored as though it had one. + +An unavailable objective source has no numeric value at all. A step whose +selected closure cannot be evaluated fails rather than being scored from a +substitute, a default, or a neighbouring dimension; each environment names the +outcome it reports, and none of them produces a number. + +## Determinism And Copy Coordinates + +The seeded PRNG protocol, its `nextBounded` projection over canonically sorted +domains, and its prohibition on host entropy and library distributions are +owned by +[Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism). +Every environment view carries the same two-field record, because both fields +are inputs to that protocol rather than facts about a search: + +```text +EnvironmentDeterminismPolicy { + master_seed: u64 + prng_protocol: Sha256SeededXoshiro256StarStar_1_0 +} +``` + +Each environment adds only its own domain separator and its own stream +purposes, in this preimage shape: + +```text +ASCII(environment domain separator) + || u64be(effective_seed) + || u32be(env_runner_index) + || u32be(vector_index) + || u64be(local_episode_index) + || u32be(stream_purpose_ordinal) +``` + +`effective_seed` is the episode's seed input, which is the configuration's +declared master seed unless the consumer supplied an override at `reset`. +`env_runner_index` and `vector_index` identify one environment copy within a +parallel sampling topology and are supplied by the consumer; a single +non-parallel consumer supplies zero for both. `local_episode_index` counts +episodes on that copy from zero. + +Including the copy coordinates in the preimage is what guarantees two copies +never draw the same sequence, and it is why a copy needs no coordination with +any other copy to stay independent. A copy coordinate is not host entropy: it +is a declared input the consumer supplies and replay reproduces. + +A supplied `reset` seed becomes that episode's effective seed, and the local +episode counter restarts. It is an input the caller supplies rather than an +override of a digest-covered configuration field, so the configured master seed +remains the default the view declares and the trajectory records which seed +actually ran. Silently ignoring the argument is invalid: it makes an +environment that appears seeded but is not, which is a failure mode that +survives every test that only checks a run against itself. + +Every environment accepts an exact episode start through the Gymnasium +`options` argument instead of drawing one, and names the payload that carries +it. This is an obligation rather than a permission because a peer contract +consumes it: [Test Protocol](spec-ml-core-training.md#test-protocol) runs every +test case by passing its instance as that override, so an environment without +one cannot be evaluated at all. + +Four rules bind every override, whatever payload it carries. Its members must +already appear in the inventory the resolved configuration declares, so an +override selects from that inventory and cannot introduce anything the +configuration does not. When present it replaces the drawing steps entirely, so +no selection stream is consulted and the copy coordinates do not reach the +choice. The same override on any copy therefore produces the same episode, +which is what lets a fixed evaluation set run identically across runners and is +the only way an episode start is not a draw. + +And a start failure under an override is returned as it stands. Retrying +requires a draw to retry and an override supplies none, so no redraw is +attempted and no retry budget an environment may otherwise declare is +consulted. Retrying would be wrong even where it were possible: a case that +silently ran a different instance than the one it names is not that case. + +Policy sampling is not an environment stream: the learned policy owns its own +randomness and supplies an action index, so an environment replay is +reproducible given the same configuration, coordinates, seed, and action +sequence. + +## Python Boundary + +### Layering + +A learned policy interacts through a Python package `loomml` with three layers, +each depending only on the one below: + +```text +loomml.rllib RLlib environment definitions, registration, and env context +loomml.env the Gymnasium Env surfaces +loomml._core native extension over the episodes, enumerations, and arrays +``` + +`loomml._core` exposes each episode lifecycle, its enumeration, its observation +arrays, and its closed outcome unions. `loomml.env` adds the Gymnasium `Env` +surface and holds no state the core does not own. `loomml.rllib` adds the +RLlib-specific obligations below. One environment occupies one module in each +layer, named for itself, so `loomml.env` and `loomml.rllib` are packages rather +than modules and adding an environment adds modules rather than editing shared +ones. + +The layering is required, not stylistic. `loomml._core` must not import Ray, +and `loomml.env` must not import Ray, so the native boundary and the episode +semantics stay independent of a training framework. A second trainer, an +offline replay tool, or a plain scripted search therefore uses an environment +without acquiring RLlib, and RLlib version movement cannot reach the C++ side. + +### RLlib Environment Definition + +An environment is consumed as an RLlib environment, and the single-agent +environment definition of the Loom RLlib fork is the normative conformance +target. That definition is `gymnasium.Env` with graph observation support, so +conformance is stated against the fork and satisfied through Gymnasium rather +than through a parallel adapter. `loomml.rllib` supplies, per environment: + +- a creator registered through `register_env` that accepts the env-context + configuration and returns one `gymnasium.Env` instance; +- static `observation_space` and `action_space` attributes fixed for the + instance's lifetime; and +- `reset(seed, options)` and `step(action)` returning the five-tuple with + `terminated` and `truncated` reported distinctly. + +Three obligations follow from that target. + +The observation uses the Gymnasium `Graph` space, so a batch of observations is +a batch of `GraphInstance` values with differing node and arc counts rather +than a stackable rectangular array. Upstream RLlib's connectors flatten +fixed-shape spaces and cannot carry that batch; the fork's graph-space support +is what makes this space consumable, and it is the specific capability the fork +exists to add. This document therefore depends on a forked trainer rather than +on an unmodified upstream release, which +[External Dependency Pinning](spec-loom-stack.md#external-dependency-pinning) +records with the pinning and patch-stack terms that choice implies. + +Padding to a fixed extent was rejected for the reason +[The Graph Instance](#the-graph-instance) states, and the fork's graph-space +support is what makes the unpadded space consumable. A ragged batch is the +honest shape of the data. + +Masked actions use the parametric-action convention. The mask and the decision +nodes are members of the observation, not a side channel, an environment +method, or a wrapper attribute, so the connector pipeline carries them to the +model unchanged. The decision nodes and their target arcs are the +available-action embeddings that convention expects. + +The mask enters that space as a fixed-shape `Box` of `uint8` and not as +`MultiBinary`, because `MultiBinary` is one entry per outcome and packing it is +the point. Its shape is a capacity and therefore constant, so it stacks +rectangularly and needs nothing the fork's graph support adds; the mask is the +one observation member the unpadded argument does not apply to, since its +extent never varied with the state to begin with. + +The copy coordinates in the determinism preimage are RLlib's `worker_index` and +`vector_index`, which it passes in the env context across `num_env_runners` +actors and `num_envs_per_env_runner` vector slots; the adapter forwards both. A +run reproduces exactly when the seed and both counts are unchanged. Changing +either count changes which seeds are drawn, which is a property of the sampling +topology rather than a defect, and it is reported rather than hidden. + +### Interaction Contract + +The array contract is zero-copy. Each observation array is exposed as a +buffer-protocol view over environment-owned memory with the element type and +layout declared above, valid until the next call that advances or resets that +environment, and never written by the consumer. A consumer that needs an array +past that point copies it. The environment does not defensively copy on every +step, because observation marshalling is otherwise the dominant per-step Python +cost. RLlib's own connectors copy what they retain, so the buffer lifetime is +compatible with sampling without a defensive copy per step. + +A non-advancing step neither advances nor resets, so it does not invalidate the +buffers: the state being observed is the one already observed, and only the +step and failure scalars change in place. Rebuilding the observation there is +permitted but pointless, and an environment names only which of its scalars +move. + +No layer between the environment and the model unpacks the mask or widens a +narrow column, on the terms [Storage Widths](#storage-widths) sets. Both stay +in the environment's own form until the model consumes them, which +[Masking](spec-ml-core-model.md#masking) and +[Observation Batching](spec-ml-core-model.md#observation-batching) place on the +device. + +Closed native outcome unions map to Python by kind rather than by message. An +ordinary non-advancing or terminal outcome is data returned from `step`, not an +exception, because both are expected transitions. A violated precondition, a +masked or out-of-range action index, a malformed configuration view, and an +owner internal failure raise distinct typed exceptions carrying the exact +reason discriminant. Diagnostic text is presentation and is never parsed. + +One environment instance is owned by one thread. Several vector copies may live +in one process; they share no state, no arrays, and no PRNG stream. An instance +is not inherited across `fork`; an env-runner process constructs its own. + +Every dependency named here is a search-harness dependency, and its exact +revision — including the Ray fork whose environment definition this section +targets, and the Python runtimes this document states conformance against — is +pinned by +[External Dependency Pinning](spec-loom-stack.md#external-dependency-pinning), +which also owns the containment argument that makes a forked trainer tolerable. +This document does not pin them separately. + +## Benchmarking Harness Contract + +Step and reset latency are first-class engineering concerns because their cost +compounds across a search, so every environment ships a harness that measures +them directly, on the removable-projection terms +[Nonsemantic Boundary](#nonsemantic-boundary) sets for every harness here. + +Each environment document names its own harness binary and its own closed list +of step and reset stages, because the stages of a search over drafts and the +stages of a search over one frozen problem have nothing in common. What every +harness owes is the same. + +Reported measures are per-stage wall time at p50, p90, and p99; steps per +second per environment copy; throughput against both `num_env_runners` and +`num_envs_per_env_runner`, since those scale differently and an aggregate +worker count hides which one is saturating; cache hit rate for every cache the +environment declares; the non-advancing rate by the environment's own reason +catalog; +and allocation counts per stage. + +Two breakdowns are required of every harness because an aggregate over either +is misleading. Every measure is reported separately for advancing and for +non-advancing steps: a step that fails early contributes no sample to the +stages after its failure point, so a blended percentile moves whenever the +policy's failure mix moves, and a real regression can hide behind a rising +early-failure rate. And every measure is reported per the environment's own +action or decision partition, because the members of one enumeration do not +cost the same. An environment document may require further breakdowns; it may +not drop these. + +Stage instrumentation is inactive outside the harness. A stage boundary implies +enough clock reads and allocator interception to be a measurable fraction of +the shortest stages, so an environment stepped by a trainer carries none of it, +and harness-reported per-stage times are not comparable to uninstrumented steps +per second. + +The nonsemantic status of wall time, the prohibition on summing concurrent +times, and deterministic work summaries as the cross-machine cost measure are +owned by +[Operational Observations](spec-dse-feedback.md#operational-observations). A +harness reports the deterministic work-unit counts alongside its timings and +states which is which. A regression budget is a ratio against a recorded +baseline tuple of exact identities and configuration digests, never an absolute +duration. + +## Anchor Verification + +Stable tests cover every reference of every live enumeration member resolving +to exactly one node ordinal in the combined space, including a relation-valued +target, under either enumeration encoding; a multi-reference decision exposing +every member of its set or tuple in canonical payload order rather than only +its first; a parallel connection appearing as one node whose count is a feature +and a count-adjusting decision leaving the node inventory unchanged; under +`DecisionNodes`, a decision's target arcs carrying its complete reference set +with the correct `ArcRole` per arc, its out-degree equalling that set's size, +its ordinal within the `Decision` block equalling its `ActionIndex`, and +`decisions` being empty; under `DecisionColumns`, the `Decision` block being +absent, `decisions` carrying exactly one row per live entry and no arcs, a +row's ordinal equalling its `ActionIndex`, and every target-naming column +holding a node ordinal that resolves in the same instance's `graph`; an +environment whose entries have variable target arity being refused the +`DecisionColumns` encoding; two states with different node counts producing +`GraphInstance` values of different extents, with no inert node, no inert arc, +and no extent that is not a real count; every environment's `NodeFeatureColumn` +catalog beginning with `Role` and its `ArcRole` catalog beginning with +`Structural`; a column stored over only the role blocks it applies to, and the +negative-one absence value appearing only within a block the column does span; +every observation array being stored in the narrowest exact form of its value +domain with no catalog declaring a width, and `scalar_features` being the one +exemption; a column-major `nodes` and `edges` rather than a two-dimensional +matrix; the action mask being the only +fixed-length array whose extent is a capacity, covering `enumeration_bound + 1` +outcomes with stopping at the last, packed one bit per outcome at bit `i & 7` +of byte `i >> 3` with every bit above `enumeration_bound` in the final byte +zero, its decision bits at or beyond the live count being clear, and its bit +indices agreeing with the action distribution's outcome indices; two states +with the same admissible set producing byte-identical masks at every live count +including one that is not a multiple of eight; the mask crossing the Python +boundary packed and being unpacked by no layer below the model; an environment +clearing the stop bit for a state in which stopping has no meaning, and a state +with every byte zero being a contract violation rather than an ending, and the +mask of an observation returned alongside an ending being neither composed, +counted as a fallback, nor read by the value head; a state +whose enumeration exceeds +`enumeration_bound` being refused with both the capacity and the required +length reported rather than being shortened; a masked, stale, or out-of-range +index being refused with the state unchanged and no coercion to a legal index +or to a terminal step; every environment defining an episode start override, an +override selecting only from declared inventory, consulting no selection +stream, producing the same episode on any copy, and a start failure under one +being returned as it stands with no redraw and no retry budget consulted; a +non-advancing step invalidating no buffer; an elective stop ignoring the +decision field; the +canonical enumeration order being identical across runs for equal state and +configuration; the step accounting identity holding over a complete episode; +termination and truncation being reported distinctly, a truncation's ending +price being zero, and a terminal state charge applying identically under a +truncation and a termination that leave the same state; a buffer remaining +valid across a call that neither advances nor resets and being invalidated by +one that does; an ordinary non-advancing or terminal outcome being returned as +data +while a precondition violation, a masked index, a malformed view, and an +internal failure each raise a distinct typed exception carrying its reason +discriminant; `loomml._core` and `loomml.env` importing without Ray present; +two environment copies with distinct coordinates drawing disjoint sequences +from one seed; a `reset` seed argument becoming the effective seed and being +recorded as such rather than ignored; and an instrumented harness run producing +the same formal results as an uninstrumented one. + +Tests do not pin the live node, arc, or decision counts of any particular +state, the bound values a profile selects, wall-time numbers, per-decision cost +ratios, allocation counts, diagnostic text, or Python formatting. diff --git a/docs/spec-ml-core-model.md b/docs/spec-ml-core-model.md new file mode 100644 index 000000000..59e9fca70 --- /dev/null +++ b/docs/spec-ml-core-model.md @@ -0,0 +1,646 @@ +# ML Model Core + +This document defines the contract every Loom learned search policy shares: how +a batch of observations becomes one disjoint-union graph, how raw observation +columns become encoder inputs, the graph-transformer trunk over them, the graph +context both heads read, the value head, the masking discipline, the action +distribution, the checkpoint boundary, and what every forward-pass benchmark +owes. + +It defines no policy head. Which actions exist, how they are anchored, and what +scores them are owned by the model documents that build on this one: + +- [ML DSE Model Architecture](spec-ml-dse-model-architecture.md) scores + candidate-generator decisions for + [ML DSE Environment](spec-ml-dse-environment.md); and +- [ML PnR Model Architecture](spec-ml-pnr-model-architecture.md) scores typed + Place and Route Actions for [ML PnR Environment](spec-ml-pnr-environment.md). + +A model is a search policy, not a Loom semantic authority. It proposes actions; +the environment decides what those actions mean and whether they are +admissible, and every legality, mappability, and objective fact remains owned +where [ML Environment Core](spec-ml-core-environment.md) and the environment +documents place it. A model checkpoint is not an Artifact, is not +`EvaluationEvidence`, and never becomes a registered prediction contract. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [ML Environment Core](spec-ml-core-environment.md) owns the observation + container, the combined node-and-link space, the enumeration encodings, the + action surface, the action mask, and the reward this model's value head + regresses toward; +- each environment document owns its own node, arc, scalar, and decision column + catalogs, and the enumeration whose entries a policy head scores; +- [Evaluation and DSE](spec-dse-feedback.md#objectives-and-quality-gates) owns + `ExactAffineQuantization` and the objective algebra whose codes enter as + features; +- [Evaluation and DSE](spec-dse-feedback.md#model-parameters-and-training) owns + `ModelParameterContractRef` and the registered prediction contracts, which a + policy is not; +- [Resolved Configuration](spec-config-ssot.md#component-views) owns + component-view framing, canonical view bytes, and `component_view_digest`; +- [Operational Observations](spec-dse-feedback.md#operational-observations) + owns the nonsemantic status of wall time and deterministic work summaries as + the cross-machine cost measure; and +- [Full-Stack Architecture](spec-loom-stack.md#external-dependency-pinning) + owns the exact revisions of every search-harness dependency, including the + Ray fork whose module and distribution contracts this document targets. + +The `RLModule` interface, its column names, and its distribution protocol are +owned by that external dependency. This document states which of its +obligations bind a model and does not restate, extend, or version them; the +training documents configure it. + +This document owns only the parts that are the same in every model: what +happens between an observation and the values `Columns.ACTION_DIST_INPUTS` and +`Columns.VF_PREDS` carry, the framing every model configuration view follows, +and the obligations every forward-pass harness satisfies. + +## Module Boundary + +A model is one `TorchRLModule` implementing `ValueFunctionAPI`, in the new +RLlib API stack: + +```text +LoomRLModule(TorchRLModule, ValueFunctionAPI) { + setup() + _forward_inference(batch) -> {ACTION_DIST_INPUTS} + _forward_exploration(batch) -> {ACTION_DIST_INPUTS} + _forward_train(batch) -> {ACTION_DIST_INPUTS, VF_PREDS, ...} + compute_values(batch, embeddings) -> VF_PREDS + get_train_action_dist_cls() / get_exploration_action_dist_cls() + -> the model's action distribution +} +``` + +The three forward entry points share one encoder pass and differ only in what +they return and in whether exploration-time logit shaping is applied. A +`_forward_train` call must produce logits identical to the +`_forward_exploration` call that collected the same batch, given the same +parameters and the same recorded masks; any divergence makes the policy ratio +wrong for every on-policy update built on it. + +`compute_values` accepts precomputed embeddings so that a training step +evaluates the encoder once. Recomputing the encoder for the value head is a +correctness-neutral waste of the dominant cost in the forward pass, and it is +the largest single saving available in the training path. + +## Observation Batching + +The environment emits a variable-size `GraphInstance` per state, so a batch is +ragged. The model consumes it as one disjoint-union graph in the standard +message-passing form: + +```text +BatchedGraph { + node_columns: per-column array, each at its declared width + arc_columns: per-column array, each at its declared width + edge_index: int64 [2, total_arcs] + batch: int64 [total_nodes] + ptr: int64 [batch_size + 1] +} +``` + +Per-instance node ordinals are offset into the union, and `batch` records each +node's originating instance so pooling and per-instance normalization stay +correct. `ActionIndex` is a per-instance ordinal, so the model maps it through +`ptr` and the instance's own offset rather than treating it as a union-wide +index; conflating the two silently scores another episode's actions. + +An environment using the `DecisionColumns` encoding supplies a second ragged +instance, and it batches the same way into its own union with its own `batch` +and `ptr` vectors. A column of that union that names a node holds a +per-instance node ordinal and is offset into the node union before it is used +as an index, for the same reason `ActionIndex` is. + +Nothing in the batch is padded. The observation's fixed-length arrays batch as +ordinary rectangular tensors. + +The batch carries columns, not feature matrices. Arrays arrive in the forms +[Storage Widths](spec-ml-core-environment.md#storage-widths) declares and stay +in them: each column is concatenated across instances at its own declared +width, and no dense `[rows, feature_dim]` tensor of widened columns is ever +built. Building one would restore exactly the bytes that section exists to +remove, on the two most numerous extents, and would do it on the device where +the encoder then has to read around it. There is no node-feature matrix in this +record because the first thing the model does to every column is embed or +project it separately, and the vector those produce is the encoder's input. + +`edge_index` is the one widened tensor. The scatter operations require `int64`, +so it is produced on device from the observation's narrower `edge_links` — a +tensor the model builds, not bytes it received. + +A large union may be processed in chunks when device memory requires it. A +chunked pass must produce logits and values identical to an unchunked one; a +chunking scheme that changes results is a defect, not a memory strategy, and +per-instance pooling makes the identity achievable because no quantity is +reduced across instances. + +## Feature Embedding + +Raw observation columns are not fed to the encoder as numbers. Every +categorical column is an ordinal in a closed owner catalog, and an ordinal +consumed as a scalar asserts an order and a distance the catalog does not have: +it would let the encoder interpolate between two unrelated entity kinds. Each +is therefore embedded through its own table, sized from its owner catalog's +cardinality. + +Numeric columns are projected rather than embedded, but never consumed raw. A +column the environment marks absent with negative one contributes a zero +magnitude and a set absence indicator, because feeding negative one as a +magnitude makes absence a large negative quantity the encoder cannot +distinguish from a real extreme value. Magnitudes that span orders of magnitude +are projected through a signed logarithmic transform before their linear layer. + +A capacity and a usage of the same resource are supplied with their occupancy +ratio alongside the two magnitudes, since headroom is the quantity a placement +or a hardware decision actually trades and the encoder should not have to learn +division. + +A capacity and a usage column appended per `FabricResourceStateRef` member are +projected per resource state and reduced to one resource embedding per node, so +the node vector width does not grow with the resource catalog. The environment +core appends that group identically for every environment, so the reduction is +stated once here rather than in each model document. + +`objective_codes` are exact bounded integers whose bounds are known: each +dimension's `ExactAffineQuantization` declares its index range, so a code maps +to its exact position in `[0, 1]` with no estimated scale and no running +normalizer. Objective position enters as a graph-level feature so the encoder +can condition a local decision on the current global position. + +Scalar features enter as graph-level features. A pair of a counter and its +bound is supplied as its ratio as well as its magnitudes, so remaining budget +is directly available rather than inferred. + +All embeddings and projections are concatenated per node and passed through one +input projection to the encoder width, so the encoder sees one uniform node +vector regardless of role. A graph-level feature may instead be projected once +per instance and added to that result, which is exact because one linear layer +over a concatenation is the sum of its per-block projections; materializing a +duplicated row per node and multiplying through it costs with node count where +the equivalent form costs with batch size. Arc embeddings are concatenated to +the arc feature width and supplied to the encoder as edge features. + +## Encoder + +The encoder is a stack of graph-transformer layers with edge-aware attention. +Attention is over graph neighborhoods rather than over all node pairs, which is +what makes it tractable on states whose node count varies by an order of +magnitude across a curriculum, and what makes the combined node-and-link space +meaningful: a connection node attends to exactly the occurrences it joins. + +```text +GraphTransformerStack { + layers: positive integer + width: positive integer, divisible by head count + heads: positive integer + feedforward: positive integer + dropout: ratio in [0, 1) + edge_dim: arc embedding width + jumping_knowledge: None | Concat | Max +} +``` + +Each layer is pre-normalized: the layer normalizes its input, applies +multi-head edge-conditioned attention, applies dropout, and adds the unmodified +residual, then does the same around a position-wise feed-forward sublayer. +Pre-normalization is required rather than optional, because post-normalization +destabilizes gradient flow at the depths and graph sizes this search reaches. A +final normalization follows the stack so the heads receive bounded features. + +The feed-forward sublayer is a departure from the prior architecture and is +stated as one. That stack was attention and residual only, so its single +nonlinearity was the attention softmax and depth bought message-passing hops +rather than capacity. Separating the two is what lets a configuration reach +further across a graph without also being forced to add parameters, and add +parameters without also being forced to reach further. + +Arc features are supplied to every layer. Environments distinguish structural +relations from placement, routing, and target relations, and an encoder that +ignored arc features would treat a placement relation, a hardware connection, +and a decision's reference to its target as the same edge. This is also a +departure from the prior architecture, and a sharper one than it appears: that +system had an edge projection and an edge-dimension-aware convolution, but no +arc features ever reached the convolution, so the projection was untrained +weight in the optimizer. Arc features here are load-bearing, not optional. + +Jumping-knowledge aggregation over per-layer outputs is supported so the model +can select an effective neighborhood radius per node instead of committing to +the stack depth. `Max` aggregation preserves the encoder width; `Concat` +multiplies it by the layer count, and every head width sized from the encoder +width must be sized from the aggregated width instead. A new aggregation +appends and must state its output width. A configuration whose head widths and +aggregated encoder width disagree is invalid at adoption rather than at the +first forward pass. + +One shared encoder produces the graph embedding, and every head reads it. Any +per-head refinement is a small stack applied after the shared trunk, never a +second trunk from raw features: separate trunks discard the representation +sharing that makes the value head's signal useful to the policy, and double the +dominant cost of the forward pass. + +## Graph Context + +Every head needs a whole-graph summary, and a single mean over nodes is a poor +one because graph size varies across the curriculum. The context is a +multi-scale pooling of the node embeddings, per batch instance: + +```text +context = project(concat(mean_pool(h), sum_pool(h), max_pool(h))) +``` + +`mean` is size-invariant, `sum` carries extent, and `max` carries the single +most salient node; together they let a head distinguish a small state from a +uniformly-scaled large one. The sum branch is scaled down by the square root of +the instance's node count and the projection is followed by normalization, +because an unscaled sum grows with node count and would otherwise make context +magnitude a proxy for graph size and drive logit scale with it. + +Pooling is per instance using the `batch` vector. Pooling across the union is a +defect that leaks one episode's state into another's decisions. + +## Value Head + +The value head reads the shared graph embedding through the same context +pooling and emits one scalar per instance. It is an ordinary multilayer +perceptron over the context, whose dropout comes from the encoder record and +whose initialization gain is one of the constants a model view declares. + +The value head predicts the return of the environment's reward, which is a +signed exact-integer energy difference scaled once at the adapter. Its output +is bounded to a configured symmetric range. That bound is a stability +mechanism, not a semantic claim: a value estimate that escapes the reachable +return range destabilizes the advantage estimate long before it becomes visible +as a policy failure. Its cost is stated rather than hidden — a saturated +estimate has no gradient — which is why the bound is checked against the +reachable return range a training document declares, and not chosen. What that +range is, and how a `value_bound` below it is refused, are owned by +[The Reachable Return Range](spec-ml-core-training.md#the-reachable-return-range). + +The value head never reads the action mask, any per-action feature, or anything +else that distinguishes one available action from another. It estimates the +state's worth, and giving it action-set information invites it to model the +enumeration rather than the state. + +## Masking + +Masking is applied additively to logits, using a large finite negative +constant. Negative infinity is not used: it produces undefined entropy and NaN +gradients when a whole group is masked, and every masked-everything case must +degrade to a defined distribution rather than to a NaN. + +Two mask sources compose, in this order: + +1. the environment's `action_mask`, which is authoritative and already covers + slots beyond the live count and any entry the environment itself cleared; + and +2. advisory masks, which remove actions the configuration declares obviously + unproductive. + +The first arrives packed, in the layout +[Action Surface And Masking](spec-ml-core-environment.md#action-surface-and-masking) +fixes. It is unpacked on the device that will consume it, after the batch has +been transferred and as part of the forward pass, so the packed form is what +crosses the sampler, the sample buffer, and the transfer, and only the expanded +form the logits actually need is ever materialized. Unpacking is a shift and a +test over a tensor whose width is the action capacity — trivial against the +encoder it sits behind, and cheaper than the memory traffic it removes. + +Composition is bitwise. The environment's mask and each advisory mask are +combined by conjunction, which is not an implementation note but the exact +statement of the first rule below: an AND can only clear bits, so a +model-side mask that tried to add an action cannot express itself in the +composition at all. + +The first is always applied and needs no model-side recomputation: a slot the +environment cleared is a slot with no live entry, and rederiving that bit per +forward pass would scatter ordinals into a capacity-wide tensor in the hot path +to reproduce a value that arrived correct. The second is where liberal masking +is available and also where it is bounded, because an advisory mask that +removes a legal action is a policy commitment rather than a correctness +statement, and must never be presented as proof that an action is bad. + +Five rules keep masking from corrupting training. + +Masks never add actions. A mask may only clear a bit the environment set; a +model-side mask that sets a bit the environment cleared proposes an action the +environment will refuse as masked. + +A fully masked state falls back. If composing the masks leaves no admissible +outcome, the composition is discarded and the environment's mask alone is used, +and the fallback is counted. Silently emitting a uniform distribution over +impossible actions produces a step the environment refuses for reasons no +diagnostic explains. + +The mask used at collection is recorded and reused at update. Advisory masks +may depend on configuration that changes across a run, so recomputing them at +training time can mask an action that was available when it was sampled. The +resulting log-probability is not the behavior policy's, and the importance +ratio is silently wrong. The mask travels in the batch, packed, in the +environment's own layout: a batch that carried the expanded form would spend +eight times the memory on the one column it retains for the whole update, and +the recorded bits are the same bits either way. + +Mask composition and distribution construction run in full precision, outside +any reduced-precision region, on an explicitly cast copy of the logits. The +additive constant is far outside the range of common half-precision formats, so +a masked logit computed in half precision saturates to infinity and +reintroduces exactly the undefined entropy and NaN gradients that choosing a +finite constant avoided. The related hazard is worth naming: under mixed +precision a trunk may remain in one precision while a head's linear layers emit +another, so a head that scatters into a buffer typed from the trunk fails on a +dtype it did not expect. + +A logit vector materialized at the action capacity is filled at its unreal +slots with the masking constant, never with zero. A trainer rebuilds the +distribution from `ACTION_DIST_INPUTS` to compute the update log-probability, +so a zero fill gives those slots real probability mass and normalizes the +rebuilt distribution differently from the one that was actually sampled. The +two log-probabilities then disagree for reasons no loss curve reveals. + +That is a rule about a vector inside a forward pass, not about what the batch +retains. `ACTION_DIST_INPUTS` travel at the live count, ragged, exactly as +`decisions` does. Retaining them at the capacity would put four bytes per +outcome per sample beside a mask that +[Storage Widths](spec-ml-core-environment.md#storage-widths) just reduced to +one bit, held for as many epochs as the algorithm takes, and the slots so +retained are at the masking constant and contribute nothing to the ratio or to +the KL term. The rebuilt distribution is identical either way, because a slot +at the masking constant and an absent slot carry the same probability. + +Logits are not clipped. Clipping bounds them by zeroing the gradient on exactly +the actions the policy is most confident about, and the masked distribution is +already bounded without it. A model that needs clipping to stay finite has a +scale problem in its heads that clipping would hide. + +Hierarchical masking is consistent by construction: an anchor group is masked +exactly when all its members are masked, so a group never retains probability +its members cannot receive, and a member never receives probability from a +masked group. + +## Action Distribution + +The distribution is a custom RLlib Torch distribution over the environment's +two-component action space, exposing sampling, log-probability, entropy, and +KL. Its inputs are the masked joint logits, and it samples one joint outcome +and emits the `{decision, stop}` pair that outcome encodes. + +The head emits one joint distribution over the `enumeration_bound + 1` outcomes +those two components can jointly take: + +```text +outcome i < enumeration_bound -> { decision: i, stop: 0 } +outcome enumeration_bound -> { decision: 0, stop: 1 } +``` + +`decision` is ignored by the environment when `stop` is one, so the emitted +value is fixed at zero to keep the encoding one-to-one; a sampler that emitted +a live index alongside a set stop flag would produce two encodings of one +action and split its probability across them. Log-probability and entropy are +computed over the joint outcome, never per component, because the components +are not independent: exactly one of the two carries the action. + +The single joint softmax is what makes acting and stopping mutually exclusive. +Two independent per-component distributions would let a policy simultaneously +raise an action's probability and the stop probability, which the environment +resolves by discarding the action, so the gradient on that action would be +attributed to something that never occurred. + +Entropy is computed over admissible outcomes only. Including masked slots at +the mask constant contributes a vanishing but nonzero term that scales with +`enumeration_bound` rather than with the live count, which makes the entropy +bonus depend on the action-space capacity instead of on the choice actually +available. + +Sampling is deterministic given a seed and the logits. The model owns its own +sampling randomness; an environment's determinism policy covers episode +construction only. + +## Policy Head Factorization + +A policy head scores each live entry from the embeddings of the nodes that +entry names. Each model owns what its own terms read; what every model shares +is the shape those terms compose in. + +A head factors an entry's logit into an anchor term over what the entry acts on +and a selection term over what it selects, and entries sharing an anchor form +one anchor group. `s_anchor` is constant within a group by construction, +because the anchor includes every input the anchor term reads, which is what +makes the factorization a hierarchy rather than a reparameterization: + +```text +P(e) = P(anchor(e)) * P(e | anchor(e)) + +P(anchor) = softmax over anchor groups of + s_anchor(g) + logsumexp over members of s_select +P(e | anchor) = softmax over that group's members of s_select(e) +``` + +The two-level form and the flat softmax over `logit(e)` induce the same +distribution, so an implementation may compute either; the hierarchy is +normative for how the scores compose, not for the order of arithmetic. Because +`s_anchor` is constant within a group it may be evaluated once per group and +broadcast, and a harness reports the mean anchor-group size alongside its head +stage, since the saving is exactly proportional to that mean and is largest in +the configurations where the hierarchy is doing the most work. + +What is not permitted is a head that scores an entry from its anchor alone with +no selection term, which collapses every alternative on one anchor into one +indistinguishable action. + +Stop is one additional logit computed from the graph context alone, since +electing to stop is a property of the state rather than of any entry. + +## Parameters And Checkpoints + +A checkpoint is the parameter tensors plus the exact model configuration that +shapes them. Loading requires an exact match of every dimension the +configuration fixes, including the embedding table sizes, which are derived +from owner catalog cardinalities. A catalog that gains a member invalidates the +tables sized from it, and the load fails rather than silently truncating or +zero-extending a table whose ordinals have shifted. + +A model checkpoint is not a `ModelParameterBundle` and its reference is not a +`ModelParameterContractRef`. Those name registered prediction contracts whose +outputs become metric predictions inside `EvaluationEvidence`, subject to +support-region and leakage rules. A policy network predicts which action to try +next; it produces no metric, enters no Evidence, and is never consulted for a +value a quality gate or an objective dimension reads. Confusing the two would +put an unvalidated search heuristic inside the evaluation path. + +A checkpoint carries parameters, and a consumer that loads one for a purpose +other than resuming its own run takes the parameters and nothing else. +Optimizer moments, a learning-rate position, and any algorithm-owned running +statistic belong to the run that produced them and are meaningless under a +different objective; a training document that sequences one algorithm after +another states that boundary, and this document fixes only that a checkpoint is +able to be read that way — its parameter tensors are addressable without its +training state. + +## Model Configuration Framing + +Each model's shape is one immutable component view of its own, following the +framing owned by [Component Views](spec-config-ssot.md#component-views). This +document fixes only three properties of any such view. + +Every embedding table's size is derived from its owner catalog's cardinality +rather than declared, so a view carries widths and the catalogs fix heights. + +A view sizes the parameter tensors, which is why no other view may carry one of +its fields. What a checkpoint pins is the tensor-shaping projection of the +view: the embedding widths, the encoder record, the head widths, and any field +that selects between scorers of different parameter shapes. The rest of the +view — the value bound and the advisory mask set — is recorded on the +checkpoint and checked for reporting, not required to match. + +The split is the same one +[Configuration Split](spec-ml-core-training.md#configuration-split) draws for +the training view, and for the same reason: a value bound is a clamp scalar and +an advisory set is a masking policy, and neither changes the shape of a single +tensor. Pinning them would make a sweep over either discard every checkpoint +the run had produced and retrain from scratch, which confuses a parameter-shape +constraint with a policy preference. A run that raises its `step_bound` mid-run +may have to raise its value bound with it, and that must not invalidate the +parameters it has already learned. + +A view declares the encoder record, the head widths, the value bound, the +advisory mask set, and the initialization constants. `HeadWidths` is one +positive hidden width per head the model declares, named for that head, so a +model with two policy heads and a value head declares three; the record is not +fixed here because the heads are not. + +`AdvisoryMaskRef` names one member of the closed catalog of advisory masks a +model document declares, and `InitializationConstants` is that document's +per-tensor initialization gains. Both are per-model for the same reason the +heads are. Two models never share one +view, because their catalogs differ and a shared view would size a table for a +catalog the other model does not have. + +## Forward Benchmarking Obligations + +A model runs once per environment step during rollout collection, so its +forward latency multiplies by every step of every episode of every worker. +Every model therefore ships a harness that measures the forward pass directly, +on the removable-projection terms +[Nonsemantic Boundary](spec-ml-core-environment.md#nonsemantic-boundary) sets +for every harness in this stack. + +Each model document names its own harness binary and its own closed list of +forward stages, because the heads differ and a shared stage list would either +be too coarse to locate a regression or carry stages one model does not have. +Stages partition the pass: none contains another, so the stage sum is the pass. +What every harness owes is the same. + +Unpacking the mask and widening a narrow column are not stages. A cast is +required to happen immediately before the operation that consumes it and only +for the columns that operation reads, so it is inside `embed` for a feature +column and inside the masking stage for the mask; timing it separately would +either double-count those stages or force an unfused materialized copy, and the +copy is the capacity-width tensor +[Storage Widths](spec-ml-core-environment.md#storage-widths) exists to prevent. +An instrumented run that materialized it would also stop measuring the +uninstrumented one. + +What every harness reports instead is the quantity that makes the layout +falsifiable: bytes unpacked and bytes cast, attributed to the stage that did +it, together with a reference run that widens at the boundary rather than on +device. The difference between the two is the layout's actual value, and it is +a number rather than an argument. A layout claimed to save bandwidth and +measured by a stage that only exists when the saving is discarded would answer +the wrong question. + +Two regimes are measured separately and never averaged. Rollout inference is +latency-bound: batch size is one instance per environment copy, graphs are +small, and fixed per-call overhead dominates. Training is throughput-bound: +batches are large, graphs are batched into one union, and arithmetic dominates. +An optimization that helps one routinely harms the other, so one number for +both hides which is binding. + +The three forward entry points are measured separately, and `compute_values` is +measured both with and without precomputed embeddings, because the encoder-once +rule is the largest single saving in the training path and a harness that never +measures it cannot show that it holds. + +Cost is reported as a function of state size, never as a single number. A +curriculum moves states across an order of magnitude, so a latency measured on +a small one predicts nothing about a large one. Each stage is reported against +the size quantities that drive it, and a stage whose measured growth exceeds +its expected order is a defect worth finding. + +The split against the environment step is reported before anything else. +Sampling throughput is set by the sum of the environment step and this model's +forward pass, and a forward pass that is a small fraction of a step can be made +twice as fast for no throughput gain at all. Reporting the split first is what +prevents optimizing the smaller half. + +Every optimization is validated against an unoptimized reference on the same +inputs and parameters, and a configuration whose logits or values differ beyond +a declared tolerance is reported as a failed configuration rather than as a +faster one. Inference-time and training-time numerics must agree within that +same tolerance: a model whose rollout path is optimized differently from its +learner path produces log-probabilities the learner did not produce, and the +resulting importance ratio is wrong for the same reason a recomputed mask makes +it wrong. Speed obtained by letting the two paths diverge is not speed; it is a +silent change to the objective being optimized. + +Two optimizations carry traps general enough to state here. A compiled or +graph-captured execution path that treats node and arc counts as static will +recompile continuously and run slower than the eager path, and the +recompilation is invisible in a steady-state average, so recompilation counts +are reported alongside latency and any bucketing that reduces them is reported +with the bucket boundaries it introduces. And reduced precision excludes mask +application, for the reason Masking states, so a harness verifies that masked +logits remain finite in every precision configuration it measures. + +A regression budget is a ratio against a recorded baseline of exact parameter +and configuration identities at a fixed state size, never an absolute duration. +The primary reported figure is the model's share of rollout wall time, because +that is the quantity an improvement has to move to matter. + +## Conformance Anchors + +Stable tests cover a categorical column reaching the encoder through an +embedding table rather than as a scalar, and an absent numeric column being +distinguishable from a real extreme value; a capacity and usage pair reaching +the encoder with their occupancy ratio; an objective code mapping to its exact +position in `[0, 1]` from its declared quantization bounds; a graph-level +feature added once per instance producing the same result as a per-node +broadcast; per-instance pooling never mixing nodes across a batched union; a +node-naming column of a batched decision union being offset into the node union +before use; a chunked pass reproducing an unchunked pass exactly; an +`ActionIndex` resolving to its own instance's entries; a configuration whose +head widths disagree with a `Concat` jumping-knowledge width being refused at +adoption; the value head producing identical predictions whether or not it is +given precomputed embeddings, and never varying with the action mask; a masked +slot receiving no probability and contributing no entropy; a fully masked +composition falling back to the environment mask and counting the fallback; a +model-side mask that sets a bit the environment cleared being refused, and mask +composition being a conjunction that cannot express such a mask at all; the +environment's mask being unpacked on the device that consumes it and reaching +the batch and the sample buffer packed; an unpacked mask agreeing bit for bit +with the environment's packed one, including across the pad bits of a final +byte; a mask recorded at collection being the mask applied at update, and a +batch whose recomputed mask differs being rejected rather than used; unreal +slots of a +materialized logit vector carrying the masking constant at its unreal slots, +`ACTION_DIST_INPUTS` being retained at the live count rather than at the +capacity, and a distribution rebuilt from those inputs reproducing the sampled +distribution's log-probability; masked logits remaining finite in every +supported precision +configuration and mask composition running outside any reduced-precision +region; logits reaching the distribution unclipped; a sampled stop outcome +decoding to a zero `decision` component, and the joint distribution never +assigning probability to a set stop flag beside a live index; a stop logit +competing in the same normalization as the action logits; `_forward_train` and +`_forward_exploration` producing identical logits for equal parameters, +observations, and masks; `_forward_inference` computing no value; a checkpoint +load failing when a catalog cardinality no longer matches the embedding table +it sized; and every optimized configuration reproducing the unoptimized +reference's logits and values within the declared tolerance. + +Tests do not pin layer counts, widths, head counts, initialization constants, +advisory mask sets, learned parameter values, wall-time numbers, device +placement, precision configurations, or tolerance values. diff --git a/docs/spec-ml-core-training.md b/docs/spec-ml-core-training.md new file mode 100644 index 000000000..bffe5605e --- /dev/null +++ b/docs/spec-ml-core-training.md @@ -0,0 +1,902 @@ +# ML Training Core + +This document defines the contract every Loom reinforcement-learning training +run shares: how a run's settings split across immutable component views, how an +algorithm is bound and how a run made of several algorithm stages hands off +between them, how hyperparameters and schedules are expressed exactly, what an +episode statistic may be computed from, how a fixed test set is executed and +aggregated, and what a checkpoint and a run identity are. + +It fits no particular policy to no particular environment. Which search is +being trained, what a corpus of training instances is, and which statistics are +worth reporting about that search are owned by the training documents that +build on this one: + +- [ML DSE Training](spec-ml-dse-training.md) fits the design-space policy; and +- [ML PnR Training](spec-ml-pnr-training.md) fits the place-and-route policy. + +A training run is a search-harness activity. It publishes no Artifact, acquires +no Evidence, and produces no candidate. Its outputs are a policy checkpoint, +which +[Parameters And Checkpoints](spec-ml-core-model.md#parameters-and-checkpoints) +already establishes is not a registered prediction contract, and a stream of +removable statistics. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [ML Environment Core](spec-ml-core-environment.md) owns the episode + protocol's shape, the action surface, the reward boundary, the step + accounting identity, the termination-versus-truncation rule, and the + determinism and copy-coordinate contract every seed stream derives from; +- [ML Model Core](spec-ml-core-model.md) owns the module boundary, the masking + discipline, the action distribution, and the checkpoint's parameter and + configuration match; +- each environment document owns its own resolved configuration view, its + priced outcomes, its terminal reasons, and its trajectory record; +- each model document owns its own configuration view and the tensors it sizes; +- [Objectives and Quality Gates](spec-dse-feedback.md#objectives-and-quality-gates) + owns `ObjectiveDimension`, `ExactAffineQuantization`, directed codes, + `ObjectiveVector`, `WeightedLevel`, and `SearchEnergyRef`; +- [Model Parameters and Training](spec-dse-feedback.md#model-parameters-and-training) + owns the training of registered prediction contracts, which is a different + activity from this one and shares no record with it; +- [Evaluation Metrics](spec-evaluation-metrics.md#metric-registry) owns + `ExactRatio`; +- [Component Views](spec-config-ssot.md#component-views) owns view framing, + canonical bytes, and `component_view_digest`; +- [Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism) + owns the seeded PRNG protocol; +- [Operational Observations](spec-dse-feedback.md#operational-observations) + owns the nonsemantic status of wall time; and +- [External Dependency Pinning](spec-loom-stack.md#external-dependency-pinning) + owns the Ray fork revision and its patch stack. + +This document owns only the configuration split and its composition rule, the +nonsemantic boundary, the algorithm binding surface and the stage sequence, the +hyperparameter and schedule expression, the reachable-return-range obligation, +the reward adapter, the rollout and learner topology, the stage invariance +rule, the statistics production rule and the environment-independent +statistics, the test protocol, checkpoint and run identity, the reproduction +claims, and the harness surface. + +## Nonsemantic Boundary + +A training run is not an evaluation, a selection, a promotion, or a candidate +generation. It publishes no Artifact, no `EvaluationRequest`, no +`EvaluationEvidence`, no `InvocationManifest` record, and no lineage edge. A +statistic, a test score, a checkpoint, and a log stream are removable +projections: regenerating them may change presentation but must preserve every +referenced semantic fact, and no consumer may treat one as a semantic result. + +A result a training run discovers becomes real only through the replay path its +environment document defines. A test score never gates promotion, never enters +a quality gate, and never selects a candidate; it selects a checkpoint, which +is a search heuristic. + +## Configuration Split + +A run's settings are several separate immutable component views plus one that +binds them, each following the framing, canonical byte representation, and +digest contract owned by +[Component Views](spec-config-ssot.md#component-views). Every training document +declares its own descriptor bytes, qualified by the search it fits, because two +runs over different searches share no view and an unqualified descriptor would +let one be adopted as the other. + +The run view binds one digest of each participating view and nothing else. +Every training document declares its own, and each carries at least a training +view, a model view, and a test-set view; what else it binds depends on where +its training instances come from. + +Composition is by digest rather than by inlining, and three rules follow from +it. + +No field appears in two views. A quantity is owned by exactly one view, and a +view that repeats another's field creates two sources for one fact that a +future edit will silently disagree about. In particular a training view carries +no episode bound and no observation policy; those are the environment's, and a +run that wants them different binds a different environment view. + +Changing a hyperparameter does not disturb the environment digest. Every +environment declares some cached asset whose key names nothing a training view +carries, so a hyperparameter sweep leaves that cache valid. Had the two been +one view, every sweep would have discarded a cache that takes real work to +warm, and the sweep's first iterations would have measured a cold start rather +than learning. + +A checkpoint pins the model digest alone. Loading requires an exact match of +the model view, because that is what sizes the parameter tensors; it requires +nothing of the training view, because fine-tuning a checkpoint under different +hyperparameters is an ordinary operation and refusing it would confuse a +parameter-shape constraint with a policy preference. The environment digest is +recorded rather than required, since a checkpoint remains loadable against a +different instance pool but not against a different action-space shape. + +Adoption validates each view independently, then validates the run view's +cross-view conditions. A run whose views are individually valid and jointly +inconsistent fails at adoption, not at the first step that notices. + +### Inventory Identity + +Every inventory a view names — instances, corpora, demonstrations, test cases — +is enumerated by exact identity. A run never names a directory, and no +inventory is ever resolved by globbing a path. A globbed set changes when a +file lands in it, so two runs that recorded the same configuration would have +trained on different data with nothing in either record to distinguish them, +and the digest that is supposed to identify a configuration would identify only +its spelling. + +Where an inventory is generated rather than named, generating it again from the +recorded seed, protocol, and view digests reproduces it exactly. No host +entropy, no wall clock, and no container iteration order participates. A +generator that cannot be re-run to the same set makes its own view's digest +meaningless, since the digest would then name a recipe with more than one +result. + +Each training document names the inventory types its run carries and the +identity by which each is compared; the rule that they are enumerated and +regenerable is this one. + +## Training Configuration + +```text +ResolvedTrainingConfigView { + stages: ordered nonempty sequence + topology: RolloutTopology + reward_adapter: RewardAdapter + logging: LoggingPolicy + evaluation: EvaluationSchedule + checkpointing: CheckpointPolicy +} + +RolloutTopology { + num_env_runners: uint32 + num_envs_per_env_runner: positive uint32 + num_learners: uint32 + rollout_fragment_length: positive uint64 | AutoFromBatch + total_env_steps: positive uint64 +} + +RewardAdapter { + scale: ExactRatio + advantage_standardization: PerMinibatch | None +} +``` + +A training document instantiates this record under its own descriptor and adds +no field to it. + +The view carries no seed. Every stream a run draws from is seeded elsewhere and +by something narrower: an episode's randomness by the environment view's +[Determinism And Copy Coordinates](spec-ml-core-environment.md#determinism-and-copy-coordinates), +and a test case's +residual randomness by its own `case_seed`. A seed here would seed nothing that +is not already seeded, and would be ambiguous with the environment view's +wherever a reader met the two together. + +### Algorithm Binding + +Training uses the algorithm implementations of the pinned Ray fork. Loom +authors no policy-gradient loss, no advantage estimator, no clipping rule, no +KL control, and no imitation weighting. The value of grounding a run in an +existing implementation comes entirely from not modifying it: a divergence +between a Loom-authored update and the published algorithm is invisible in a +loss curve and is the first thing a failed run would otherwise have to rule +out. + +```text +AlgorithmBinding = + Ppo(PpoBinding) // 0 + | Marwil(MarwilBinding) // 1 + +PpoBinding { + gamma: Schedule + lambda: Schedule + clip_param: Schedule + vf_clip_param: ExactRatio + vf_loss_coeff: Schedule + entropy_coeff: Schedule + kl_coeff: Schedule + kl_target: ExactRatio + learning_rate: Schedule + grad_clip: ExactRatio + grad_clip_by: GlobalNorm | Value + train_batch_size: positive uint64 + minibatch_size: positive uint64 + num_epochs: positive uint64 +} + +MarwilBinding { + beta: ExactRatio + gamma: Schedule + vf_loss_coeff: Schedule + learning_rate: Schedule + grad_clip: ExactRatio + grad_clip_by: GlobalNorm | Value + train_batch_size: positive uint64 + minibatch_size: positive uint64 + num_epochs: positive uint64 + advantage_norm_update_rate: ExactRatio + advantage_exponent_clamp: ExactRatio +} +``` + +Ordinals are stable; a new algorithm appends. The union exists because an +offline stage and an online stage are the same run over the same policy, and a +document whose algorithm field admitted only one of them could not express that +at all. + +An offline algorithm reads demonstrations rather than sampling the environment. +Which view supplies them is not a field of the binding: the run view already +binds one digest of each participating view, so a binding that named the corpus +again would be a second source for a fact already fixed. A training document +whose run binds an offline stage names the demonstration view there, and states +what a valid one contains. + +`MarwilBinding.beta` at zero is behaviour cloning and at one is full advantage +weighting. That distinction is a property of the demonstrations rather than a +preference: on demonstrations whose returns barely vary, the exponential +weighting collapses toward one and the run is cloning whatever `beta` says, so +a training document states which regime its demonstration source supports +rather than leaving `beta` to be tuned into meaninglessness. + +`advantage_exponent_clamp` is normative rather than a tuning knob. The +advantage weight is an exponential, and its argument grows with the advantage +estimate an untrained value head produces; past a modest bound the exponential +leaves the range of a single-precision float and every weight becomes infinite +in one step. Clamping the exponent bounds the weight without changing its +ordering. + +`vf_loss_coeff` is stated against the policy loss's magnitude rather than +defaulted. A masked per-node policy loss and a scalar value loss differ by +orders of magnitude, and a coefficient chosen without regard to that difference +gives the value head effectively no gradient, whereupon it regresses to the +mean and every advantage the imitation weight reads is noise. + +Loom supplies exactly four things to whichever algorithm a stage binds, each +owned elsewhere and none of them the update itself: the environment through +`loomml.rllib`, as +[RLlib Environment Definition](spec-ml-core-environment.md#rllib-environment-definition) +specifies; the `RLModule`, as +[Module Boundary](spec-ml-core-model.md#module-boundary) specifies, together +with the custom action distribution; callbacks that observe episodes and emit +the statistics below; and the custom evaluation function that runs the fixed +test set. + +A `Learner` subclass is permitted for instrumentation only. It may read +gradients, activations, and parameter norms and emit statistics; it may not +change a loss term, an optimizer step, a gradient, or an update order. An +instrumented run and an uninstrumented run must produce identical parameters +from identical inputs, which is the same invariance +[Forward Benchmarking Obligations](spec-ml-core-model.md#forward-benchmarking-obligations) +requires of the model's optimizations. + +Three obligations the environment and model documents impose land on the +trainer's configuration rather than on its code. Termination and truncation +stay distinct, and the value target must bootstrap past a truncation and must +not bootstrap past a termination; a connector or wrapper that collapses the two +flags is invalid. The mask travels in the batch unchanged from collection to +update, as [Masking](spec-ml-core-model.md#masking) requires, because a +pipeline that lets the learner recompute it produces an importance ratio that +is wrong in a way no diagnostic reveals. And the observation stays a ragged +graph batch, so a configuration that inserts a flattening connector ahead of +the module defeats the only reason the fork exists. + +### Stages And Handoff + +A run is an ordered sequence of stages, each binding one algorithm and one +environment view: + +```text +TrainingStage { + algorithm: AlgorithmBinding + environment_config_digest: ComponentViewDigest + advance: optional +} + +StageAdvance = + AtEnvSteps(uint64) + | AtPlateau { + metric: LearnerStatisticRef + window: positive uint32 + tolerance: ExactRatio + grace_env_steps: uint64 + } +``` + +A single-algorithm, single-environment run is the one-stage case and needs no +special form. The final stage carries no advance and every earlier stage +carries one; the run ends when the final stage does. There is no `AtRunEnd` +member, because being last is a fact the ordered sequence already fixes and a +member spelling it would be a second source that adoption then has to check +against the position. + +`AtEnvSteps` counts sampled steps from the run's start rather than from the +stage's, so the values across a run strictly increase and the reading is +unambiguous at any point. + +A stage binds a whole environment view rather than a delta against its +predecessor, so the configuration that produced any point of a run is +recoverable exactly, and a stage boundary is one atomic change rather than a +set of independently applied overrides. + +A stage boundary always transfers **model parameters**. It transfers +**algorithm state** — optimizer moments, the advantage normalizer, the +iteration counter — exactly when the two stages bind the same arm of +`AlgorithmBinding`, and never when the arm changes. Carrying moments +accumulated under one objective into gradients of another produces first +updates that are neither stage's; discarding them across a boundary that only +moved the environment would make every advance a cold restart of the +optimizer, which is the cost that would make a many-stage run unaffordable. + +Two failure modes are contract obligations rather than implementation notes, +because both present as a successful run. + +A parameter load that does not reach the module is a cold start wearing a warm +start's name. A checkpoint's component tree is addressed by name, and a restore +aimed at a component path that does not exist succeeds silently and loads +nothing. The boundary is therefore observable: the run reports the loaded +parameter count and a digest of the loaded tensors, and a stage that reports +neither has not demonstrated that it loaded anything. + +An evaluation at a stage boundary must observe the loaded parameters. Sampling +workers hold their own copies, and an evaluation that draws weights from a +worker rather than from the learner reports the parameters that worker last +held, which at a boundary is the previous stage's or the initial ones. The +boundary evaluation is the reading everything downstream compares against, so +reporting the wrong parameters there mislabels every later comparison. + +`AtPlateau` names a learner statistic and stops the stage when its moving +window varies by less than `tolerance`, after `grace_env_steps`. It exists +because a stage's useful length is not always known in advance: a stage that +ends when its objective stops improving is measuring a property of the run +rather than asserting a step count. The metric must be one the stage's own +algorithm emits. + +### Exact Rational Hyperparameters + +No binary floating-point number appears in a training view. Every real-valued +hyperparameter is an `ExactRatio`, and the `float64` a trainer requires is +produced once where the configuration crosses into Python, exactly as +[Reward Contract](spec-ml-core-environment.md#reward-contract) requires of the +reward scale. + +The reason is identity. A view is digest-covered, and a digest over a decimal +literal rounded to a mantissa is a digest over a rounding: two runs that +recorded the same configuration could differ in the last bit of a learning +rate, and no record would distinguish them. Keeping the exact ratio in the view +keeps the configuration a value and moves the conversion to the one layer that +can state the exactness condition its own float type imposes. + +The model's `value_bound` is not here. It sizes a clamp inside the model and +belongs to the model view, which is what a checkpoint pins. + +### Schedules + +A hyperparameter that varies over a run is a schedule: + +```text +Schedule = + Constant(ExactRatio) + | PiecewiseLinear(ordered nonempty sequence<{ + at_env_steps: uint64 + value: ExactRatio + }>) +``` + +The first breakpoint is at zero, breakpoints strictly increase, and the value +past the last breakpoint is that breakpoint's value. + +Breakpoints are keyed on environment steps sampled, never on training +iterations. An iteration count depends on batch size, on how many steps a +sampler happened to return, and on how a stage was resumed; a sampled step +count does not. A schedule is therefore a pure function of the step count, so a +resumed run continues the anneal from where it was rather than restarting it, +and two runs that differ only in topology still follow the same schedule +against the same abscissa. + +### The Reachable Return Range + +The model view's `value_bound` is checked rather than chosen, and what it is +checked against is an obligation on each training document rather than a +formula here. + +A training document declares the per-episode return bound of the environment +its run binds, in that environment's own reward vocabulary. It is the document +that can: the arithmetic reads the environment's codes and step bound on one +side and this document's `reward_adapter.scale` and stage sequence on the +other, so neither the environment nor this core holds both halves. The bound is +derived from two things the environment already owns: the quantization width of +its selected search energy, since each +dimension's `ExactAffineQuantization` fixes the interval and a single +transition's signed difference cannot exceed its width; and the codes it +charges for outcomes that are not transitions, together with the bound on how +many of each an episode may carry. Environments charge structurally different +sets of codes, so a formula fixed here would be one search's arithmetic imposed +on another's. + +Two rules bind every such declaration. The bound accounts for every step, not +one of each kind: an episode that spends its whole step budget on the most +expensive outcome is not a corner case but the first thing an untrained policy +produces. And where a term's magnitude depends on the instance rather than the +configuration, the declared bound is the maximum over the instance pool, taken +at adoption. + +A declared bound is in the environment's own integer reward units, and +`value_bound` is in the scaled units the value head emits, so the comparison +multiplies the declared bound by `reward_adapter.scale` before making it. A +scale above one would otherwise let through a bound that clips every return it +was checked against. + +A run's requirement is that scaled maximum across every stage it binds, because +a later stage may raise a bound, a code, or the scale. Adoption requires +`value_bound` to be at least that maximum. A bound below it clips a return the +environment can actually produce, which biases the advantage estimate exactly +where the search is doing best; a bound far above it is permitted and merely +weakens the stability mechanism. + +That the range is computable at all is a consequence of the environment +refusing to normalize. Reward is exact, integral, and drawn from a declared +bounded quantization, so nothing here needs an estimated scale or a running +normalizer. + +### Reward Transforms + +The adapter applies exactly one transform: the declared `ExactRatio` scale that +turns an integer reward into the float the trainer requires. + +Running normalization is rejected for three reasons. It makes the reward a +function of the batch rather than of the transition, so the same transition is +worth different amounts at different points of a run. It destroys the exactness +the environment paid for. And it hides exactly the failure it appears to fix: a +reward whose scale is wrong is a configuration error, and normalizing it away +means never seeing it. + +Clipping, discounting inside the adapter, potential-based shaping terms, and +per-dimension reweighting are likewise excluded. A run that wants a different +weighting binds a different objective closure, which is a semantic change with +a digest, rather than a transform with none. + +`advantage_standardization` is not a reward transform. It standardizes the +advantage estimate inside the loss, which is the algorithm owner's own +mechanism, and it is exposed here only because it is a configuration field a +run sets. + +## Rollout And Learner Topology + +`num_env_runners` and `num_envs_per_env_runner` are part of the reproduction +tuple, not free scaling knobs. Every environment derives each copy's PRNG +streams from its own coordinates, so changing either count changes which seeds +are drawn and which instances an episode sees. That is a property of the +sampling topology rather than a defect, and the run records both counts so a +run that does not reproduce can be told apart from one that was never +configured the same way. + +Sizing follows the measured split rather than a default. Each environment +document reports what dominates its step and each model document reports what +fraction of rollout time inference takes, so a topology is chosen by moving +runner count against that measured split rather than by matching learner count +to available accelerators. + +The environment's own interaction contract governs instance lifetime, and this +document adds no second lifecycle. + +## Curriculum + +A curriculum is not a separate record. It is what the stage sequence already +expresses: consecutive stages whose environment views differ. No nested second +sequence is needed, and admitting one would give a run two advance mechanisms +for the same thing. + +Consecutive stages may differ in their environment views only in fields that +fix neither the action-space shape, the observation shape, nor the reward +semantics. A policy cannot be trained across a change in any of the three: the +action space would change extent mid-run, the embedding tables would be sized +for a catalog the observation no longer produces, and the value head would be +regressing toward a different quantity than before. + +The rule is stated by what a field determines rather than by naming a record, +because environments partition their views differently and the same field name +sits inside an episode policy in one and beside it in another. Each environment +document names which of its own fields fall on each side, and adoption checks +the partition that document declares. A rule written as one record name would +forbid, in some other environment, exactly the curriculum that environment most +obviously wants. + +An environment's cached assets survive a stage advance whenever the advance +changes no field their key names. A key that names an individual instance +rather than the set the view declares therefore survives any advance that only +grows that set, which is what makes a many-stage run affordable rather than a +repeated cold start. Each environment document says which of its caches have +that property. + +## Episode Statistics + +### The Production Rule + +Every episode statistic is a function of values the environment already emits: +observation columns, objective codes, scalar features, the step result, the +action taken, and the retained trajectory. A callback never re-derives a fact +from an Artifact, never calls a semantic owner, and never computes a quantity +an owner would compute differently. A statistic that cannot be produced this +way is not a statistic this document defines; it is a request for the +environment to emit something it does not. + +Every statistic is keyed by a member of a closed catalog. No key is built by +formatting a file name, a path, or an identity digest into a metric name. +String-keyed series multiply with the instance pool and cannot be enumerated in +advance, so a consumer cannot tell a missing series from a series that was +never produced. + +### Objective Dimension Statistics + +One record per dimension of the environment's selected closure, per episode: + +```text +DimensionEpisodeStatistic { + entry_code: uint64 // at reset + final_code: uint64 // at the episode's last state + best_code: uint64 // minimum over the episode's states + step_of_best: uint64 +} +``` + +A consumer reads improvement as `entry_code - final_code` and best improvement +as `entry_code - best_code`; neither is a stored field, because both operands +are already here and a stored difference is a second source for one fact. + +Improvement is uniformly oriented without a per-dimension rule, because a +directed code is already direction-normalized by its `ExactAffineQuantization`: +a lower code is better for every dimension regardless of which direction the +dimension optimizes. `entry_code` is retained because a final value alone is +not a result — episodes start from different instances, and a final code says +nothing about whether the policy improved anything. + +`retain_step_series` extends these to a per-step series and is off by default, +because it is one array per dimension per episode and is the largest logging +term by a wide margin. + +### Outcome And Action Statistics + +Every episode reports the accounting the environment core's step identity +already guarantees: + +```text +episode_return scaled, matching the trainer's own return +episode_length steps, spanning sampler chunks +advanced_steps +non_advancing_steps +elective_stops zero or one +unaccounted_steps episode_length - the three above +terminal_reason one member of the environment's catalog +mean_live_decision_count +max_live_decision_count +mask_fallback_count +``` + +The names here are the neutral ones +[Step Accounting And Episode Endings](spec-ml-core-environment.md#step-accounting-and-episode-endings) +fixes. Each environment reports the same quantities under its own word for its +non-advancing class and its own word for one member of the live enumeration; +what is fixed is the accounting, not the vocabulary. + +There is no headroom statistic. The distance from `max_live_decision_count` to +the environment's `enumeration_bound` is a difference of two values already +recorded, and a stored difference is a second source for one fact on the same +terms the objective statistics above refuse one. + +`unaccounted_steps` exists to be zero. The environment guarantees the identity +by construction, so a nonzero mean is a defect in the callback rather than in +the environment, and the usual cause is an episode length read off a single +sampler chunk. + +`terminal_reason` is reported as a rate per member of the environment's +terminal catalog rather than as a single mode, because the members mean +different things: one is the ending the policy elected and the rest are limits +the harness imposed, and a policy that never elects its own ending is a +different failure from one that elects it immediately. + +A run reports the non-advancing rate per member of the environment's own +failure catalog, never as one aggregate. A single rate conflates causes that +call for opposite responses, and the environment already separated them. + +`max_live_decision_count` against the environment's `enumeration_bound` is what +predicts a capacity refusal before it happens. +`mask_fallback_count` counts the model's fully-masked fallback, reported here +because this is the layer that observes it rather than because this layer owns +the advisory set. + +Action frequencies are keyed by the environment's own action partition and each +is paired with that partition member's success rate — advances over attempts. +The frequency says what the policy tries and the pair says whether trying +works, and a policy collapsed onto one member of the partition is visible here +and nowhere else. + +### Cost Statistics + +```text +mean_step_duration +mean_reset_duration +episodes_per_hour_per_copy +model_inference_share_of_rollout +``` + +Wall time is nonsemantic in the sense +[Operational Observations](spec-dse-feedback.md#operational-observations) +defines, and these four are labelled as such wherever they are reported. None +is summed across concurrent copies. Deterministic work summaries remain the +cross-machine cost measure, and `model_inference_share_of_rollout` is the +figure a model optimization has to move. + +Per-stage step decomposition is not reported here. It belongs to the benchmark +harnesses, whose instrumentation is inactive elsewhere; a trainer carrying +stage boundaries per step would pay a measurable fraction to produce numbers +not comparable to its own uninstrumented throughput. + +### Breakdown Axes + +```text +LoggingPolicy { + enabled_axes: canonical set + retain_step_series: bool + per_episode_sample_rate: ExactRatio +} +``` + +`BreakdownAxis` is a closed catalog each training document owns, because the +axes worth splitting a statistic along are the ones its own search has. Two +members are mandatory and take ordinals zero and one in every such catalog: +`Stage`, over the run's stage sequence, and `TerminalReason`, over the +environment's terminal catalog. Fixing their ordinals is what lets a consumer +reading two searches' statistics key on the two axes both are guaranteed to +have; everything after ordinal one is per-document. + +Axes are opt-in because their cost is multiplicative: an axis over an instance +pool turns every statistic into one series per pool member, and two such axes +multiply. The default is no axis, and a diagnosis enables the one axis it +needs. + +An axis's members are exact references, keyed by identity rather than by a file +name, so a series survives a reorganization of where anything is stored. + +The sink is presentation. No sink is normative, and a run that writes to no +sink computes the same policy. + +### Chunk Invariance + +An episode statistic must not depend on where the sampler cuts an episode. A +sampler returns fragments, and an episode may span several; a statistic +computed per fragment and averaged is a different quantity from the same +statistic computed per episode. + +Two forms are permitted: a statistic accumulated across fragments until the +episode ends and emitted once at its end, and a statistic that is a pure +function of the episode's final state. Two are forbidden: a statistic emitted +per fragment as though the fragment were an episode, and a statistic whose +value depends on the fragment length the topology happened to produce. + +## Learner Statistics + +Loss terms, entropy, KL, explained variance, gradient norms, and timing are the +algorithm owner's statistics, reported unchanged. This document neither +redefines nor renames them, and three of them are worth reading against +contracts stated elsewhere. + +Entropy is over admissible outcomes only, per +[Action Distribution](spec-ml-core-model.md#action-distribution), so it is +comparable across states with different live counts. An entropy that tracks +`enumeration_bound` rather than the live count indicates entropy computed over +masked slots. + +Explained variance is against a return whose range is declared, so a +persistently negative value is a value-head failure rather than a scaling +problem. + +A KL spike at a curriculum boundary is the expected consequence of the +environment view changing under a fixed action space, and a KL spike at a stage +boundary is the expected consequence of the objective changing under fixed +parameters. Neither is a defect; both are worth distinguishing from a spike +with no boundary near it. + +## Test Protocol + +### The Test Set + +```text +ResolvedTestSetConfigView { + environment_config_digest: ComponentViewDigest + cases: ordered nonempty sequence + case_timeout: optional +} + +TestCase { + instance: + case_seed: u64 +} +``` + +A case names its instance exactly. Nothing is drawn. A case runs by passing its +instance as the episode start override its environment defines, so no selection +stream is consulted and the copy coordinates cannot reach the episode. That is +what makes a case's result independent of which evaluation runner executed it, +and why the protocol needs no coordination between runners. + +A test set's inventory is disjoint from that of every stage the run binds, +compared by exact identity. A case the policy trained on measures how well the +run memorized it, which is the one thing a test score is not for. Each training +document names only the granularity at which identity is compared, since what +counts as one instance differs by search. + +A test set binds its own environment view, which is what makes disjointness +expressible at all: an override may only name inventory the view it runs +against declares, so a test set sharing a training view would have to name +inventory that view declares and could never be adopted. Adoption checks that +the case inventory is exactly what the test set's own view declares. + +The two view kinds must agree on everything a checkpoint depends on. A test-set +environment view differs from the training views only in fields that fix +neither the action-space shape, the observation shape, nor the reward semantics +— the same partition consecutive stages obey — because a score produced +against a different action space is not a score of the same policy. + +The set is ordered and complete. Every case runs on every evaluation, in case +order, exactly once. A sampled test set is rejected: a score that moves because +a different subset was drawn is indistinguishable from a score that moves +because the policy changed, which defeats the only purpose the set has. + +### Determinism + +A test episode is deterministic in both halves. The policy acts greedily — +exploration off, the distribution's mode rather than a sample — so the model +contributes no randomness, and `case_seed` seeds whatever residual randomness +an implementation retains. The environment is reproducible given its +configuration, the copy coordinates, the seed, and the action sequence, and a +case consults no selection stream at all. + +Both halves being deterministic is what discharges the exact-reproduction claim +[Reproduction](#reproduction) makes for a test run, and that claim in turn is +the only reason two checkpoints are comparable at all. + +### Execution And Aggregation + +Cases are distributed across evaluation runners by case ordinal. Distribution +is a throughput decision with no semantic content. + +```text +TestCaseOutcome = + Completed { the episode statistics catalog above } + | StartFailed { the environment's episode start outcome } + | TimedOut { steps completed } + +EvaluationSchedule { + every_env_steps: uint64 + at_run_end: bool +} +``` + +A failed or timed-out case is reported, counted, and excluded from the +completed aggregate — never dropped. Dropping biases toward the cases that ran, +so a degrading policy would show an improving mean as its hardest cases quietly +left the set. A result therefore carries the completed count alongside every +aggregate, and an aggregate over a changed number of cases is not comparable to +its predecessor. + +`case_timeout` is nonsemantic wall time. A timed-out case retains its partial +statistics, tagged as partial, and they are never averaged into a completed +aggregate. + +Evaluation runs on a schedule keyed on sampled environment steps, for the same +reason schedules are. + +### What A Test Score Is Not + +A test episode falls on the search-heuristic side of +[Nonsemantic Boundary](#nonsemantic-boundary), on the same terms as a training +episode. The boundary is worth restating here because a test episode is where +it is easiest to lose: these episodes run the same machinery a real search +would and read the same objective dimensions, so the numbers look like results, +and treating them as results would put an unvalidated search heuristic where a +verified one belongs. + +## Checkpoints And Run Identity + +```text +CheckpointPolicy { + every_env_steps: uint64 + retain_last: positive uint32 + retain_best_by_test_score: bool +} +``` + +A run is identified by its run view's digest and by nothing beside it. There is +no run-key record, because a record with one field is a name for that field and +a second place to change it. Everything a key would want is already inside the +digested view — the topology counts, the stage sequence, every participating +view — so a run at a different runner count already has a different identity, +and carrying any of it again would be two sources for one fact. + +A checkpoint records that digest, the sampled step count, the stage ordinal and +stage it was taken in, and the parameters. Loading requires an exact +model-view match per +[Parameters And Checkpoints](spec-ml-core-model.md#parameters-and-checkpoints), +and a load whose catalog cardinality no longer matches an embedding table fails +rather than truncating it. + +A checkpoint selected by test score records which test-set digest selected it, +because a score is only meaningful against the set that produced it. + +## Reproduction + +This document makes exactly two reproducibility claims and declines a third. + +A test run reproduces exactly: one checkpoint and one test-set digest give +identical statistics on any machine at any runner count. + +A trajectory reproduces exactly: the recorded configuration, coordinates, seed, +and action sequence reconstruct the episode its environment's replay path +defines. + +A learning curve does not reproduce bit-for-bit, and claiming otherwise would +be false. Concurrent learner reductions, accelerator kernel nondeterminism, and +asynchronous sample collection whose completion order depends on wall time all +perturb the update sequence. What a run records is enough to reconstruct its +data, its configuration, and every episode it ran — which is what reproduction +has to mean here. + +## Harnesses + +Every training document names its own harnesses, qualified by the search they +fit, on the removable-projection terms +[Nonsemantic Boundary](spec-ml-core-environment.md#nonsemantic-boundary) sets +for every harness in this stack. + +Two obligations bind them wherever they appear. A harness that runs a test set +against a checkpoint runs the same protocol the training-time evaluation runs, +so a score produced during a run and a score produced afterward are the same +quantity. And a harness that prepares training data verifies an existing +preparation without regenerating it, so a corpus can be checked without being +rebuilt. + +## Conformance Anchors + +Stable tests cover a hyperparameter edit changing the training digest and +leaving the environment digest and every cached asset valid; a checkpoint +loading under a changed training view and failing under a changed model view; a +schedule evaluated at a given sampled step count producing the same value on a +resumed run as on an uninterrupted one; a stage boundary transferring +parameters always, and transferring optimizer moments, the advantage +normalizer, and the iteration counter across a same-arm boundary while +transferring none of them across an arm change; a stage boundary reporting the +loaded parameter count and +tensor digest, and a restore aimed at a path that does not exist failing rather +than loading nothing silently; a boundary evaluation observing the loaded +parameters rather than a sampler's stale copy; an `AtPlateau` advance naming a +statistic its own stage's algorithm emits; a single-stage run behaving +identically to the same configuration expressed without stages; an instrumented +learner producing parameters identical to an uninstrumented one from identical +inputs; a truncation bootstrapping and a termination not; a batch whose mask +was recomputed rather than carried being rejected; every episode statistic +being a function of environment-emitted values alone; `unaccounted_steps` being +zero over a complete episode; an episode statistic computed across a sampler +chunk boundary equalling the same statistic computed from an unchunked episode; +consecutive stages differing only in fields the environment declares neutral +being accepted and ones differing in a shape-fixing field being rejected; +algorithm state carrying across a same-arm stage boundary and resetting across +an arm change; a +`value_bound` below the environment's declared return range being rejected, +including where a term's magnitude is instance-dependent and the maximum over +the pool is what binds, and the comparison being made against the declared +bound scaled by `reward_adapter.scale`; a test set running every case exactly +once in case order; a test case consulting no environment selection stream and +producing an identical trajectory at two runner counts and two copy +coordinates; one checkpoint and one test-set digest reproducing identical +statistics across machines; a failed or timed-out case being counted and +excluded from the completed aggregate rather than dropped; and a run's identity +being the run view digest alone. + +Tests do not pin hyperparameter values, schedule breakpoints, stage counts, +plateau tolerances, topology counts, the contents of any particular test set, +wall-time numbers, throughput, learning curves, achieved test scores, sink +formats, or diagnostic text. diff --git a/docs/spec-ml-dse-environment.md b/docs/spec-ml-dse-environment.md new file mode 100644 index 000000000..66bae91cc --- /dev/null +++ b/docs/spec-ml-dse-environment.md @@ -0,0 +1,1325 @@ +# ML DSE Environment + +This document defines the interaction boundary through which a learned search +policy explores Loom's design space. The environment presents one episode as a +sequence of typed design decisions over one subject, either a single +`fabric.module` or a complete multi-core `fabric.system`, optionally together +with the software expression of the workloads that subject must run. It scores +each transition against a resolved objective and retains the decision sequence +for later replay. + +Which decisions an episode may take is configuration rather than schema. The +environment binds a set of exploration domains, each of which is one existing +candidate generator, and a domain the configuration omits contributes nothing. +A parameter family over an existing parent kind, existing reference kinds, and +an existing completion form is one catalog member and one obligation-table row. +A family that needs a new reference kind, a new value form, or a new completion +form extends the observation projection and the completion algebra as well; +what appending never requires is widening a decision union or adding a +permission flag. + +## Ownership + +Every fact the environment exposes resolves to one exact owner: + +- [Evaluation and DSE](spec-dse-feedback.md#candidate-generators) owns + candidate-generator kinds, their closed decision unions, decision domains, + canonical decision order, and lineage contributions; +- [Evaluation and DSE](spec-dse-feedback.md#objectives-and-quality-gates) owns + `ObjectiveDimension`, `ExactAffineQuantization`, `ObjectiveVector`, + `WeightedLevel`, `TotalOrdering`, `SearchEnergyRef`, and quality gates; +- [ADG Builder](spec-adg-builder.md#failure-atomic-finalization) owns draft + construction and authoring-boundary checks; +- [Fabric Artifact](spec-fabric-artifact.md#finalization-and-publication) owns + the finalization pipeline, canonical bytes, root identity, the verified + pre-publication closure, and publication; +- [Fabric Identity](spec-fabric-identity.md#owner-local-reference-kind-catalog) + owns the local-reference kind catalog and, at + [Mapping-Visible Entity Catalog](spec-fabric-identity.md#mapping-visible-entity-catalog), + `EntityId`; +- the `fabric.*` specifications own every typed hardware parameter domain; +- [SCF To DFG](spec-compiler-part-3-dfg.md#canonical-dataflow-rewrite-catalog) + owns the closed Canonical Dataflow rewrite catalog, its decision wire, its + canonical decision order, and the equivalence obligations every rewrite + preserves; +- [Place And Route](spec-pnr.md#objective-projection) owns the Mapping + violation catalog `V` and measure catalog `G`; + [Invocation Contracts](spec-pnr.md#invocation-contracts) owns invocation + outcomes; + [Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism) + owns search policy and the seeded PRNG protocol; and + [Evaluation Transaction](spec-pnr.md#evaluation-transaction) owns the + ephemeral probe adapter and its `rebuild`, `probe`, `commit`, and `discard` + operations; +- [TechMapping Generation](spec-tech-mapping.md) owns TechMapping construction + and its exact `D` and `F` binding; +- [Evaluation Metrics](spec-evaluation-metrics.md) owns metric kinds, units, + observation forms, and `ExactRatio`; +- [Resolved Configuration](spec-config-ssot.md#component-views) owns + component-view framing, canonical view bytes, and `component_view_digest`, + and at [Cache Dependencies](spec-config-ssot.md#cache-dependencies) owns the + cache-family contract; +- [ML Environment Core](spec-ml-core-environment.md) owns the observation + container and its combined node-and-link space, the action surface and + masking algebra, the step accounting identity, the + termination-versus-truncation rule, the reward boundary, the PRNG preimage + shape, the `loomml` package layering and its interaction contract, and the + benchmarking obligations every harness satisfies; and +- [Full-Stack Architecture](spec-loom-stack.md) owns external dependency + pinning and the fast-evaluation cascade. + +The environment owns only the episode protocol, the exploration-domain catalog, +the enumeration rule, its action index contract, and the schedule-derived +reference rule, the DSE observation +catalogs and their curriculum-neutral partition, the workload-feasibility +invariant, the step and rejection outcome algebra, the visited set and the +per-state rejection mask, the trajectory retention record, and its own +benchmarking stages. + +## Nonsemantic Boundary + +The environment is a nonsemantic search harness, on the terms +[Nonsemantic Boundary](spec-ml-core-environment.md#nonsemantic-boundary) states +for every ML environment. It defines no hardware action language, no mutable +candidate intermediate representation, and no second design space. Every +decision it applies is an existing owner-typed candidate-generator decision, +every legality answer comes from the ordinary Builder, Fabric finalizer, and +Dataflow rewrite owners, and every mappability answer comes from the ordinary +Place and Route owner. + +No episode subject is published while the episode holds it. The single +exception is the component a `Reattach` completion publishes so the subject can +depend on it, defined under Step. That component is published by its own +generator through the ordinary path, with the `CandidateDecision` lineage +contribution that generator's descriptor owns, so it is an ordinary +intermediate candidate rather than a publication mode this document invents; +what the environment withholds is selection, not lineage. + +The environment adds nothing to the subjects deliberately absent from the first +version by +[Full-Stack Architecture](spec-loom-stack.md#deliberate-first-version-boundaries). + +An environment run is therefore never an authority for candidate selection. A +design the environment discovers becomes a real candidate only by replaying its +trajectory through ordinary `Generate` and `Promote` plan nodes, as defined by +Trajectory Retention And Replay below. + +## Exploration Domains + +An exploration domain is one existing candidate generator whose decisions the +environment may enumerate. Domains are the unit of permission and the unit of +extension: a domain is permitted exactly when the resolved configuration binds +it, and a new parameter family is added by appending one member to the closed +catalog below and one row to the obligation table, to the extent the intro +states, and never by widening a decision union or adding a parallel switch. + +```text +ExplorationDomainKind = + SpatialTopology // 0 + | SpatialMicroarchitecture // 1 + | SystemComposition // 2 + | DataflowRewrite // 3 +``` + +Ordinals are stable. A new kind appends; reordering, deleting, or repurposing +one is an incompatible change to this document's schema version. Each kind +binds its generator's own resolved configuration and defines no decision, match +rule, or value domain of its own. + +One table is the complete per-domain contract. Every rule elsewhere in this +document quantifies over its rows rather than naming a domain, so appending a +domain is appending a row: + +| Domain | Generator | Operates on | Required subject | Completion | +| --- | --- | --- | --- | --- | +| `SpatialTopology` | kind 13 `spatial_topology_rewrite` | one `fabric.module` | either | `Reattach` via `SystemComposition` under `System` | +| `SpatialMicroarchitecture` | kind 14 `spatial_microarchitecture_rewrite` | one `fabric.module` | either | `Reattach` via `SystemComposition` under `System` | +| `SystemComposition` | kind 15 `system_composition_rewrite` | one `fabric.system` | `System` | none | +| `DataflowRewrite` | the [Dataflow rewrite generator](spec-dse-feedback.md#deterministic-work-candidate-sets-and-cache) | one Canonical Dataflow Program | either | `ReplaceParent` | + +"Operates on" is both the exact parent a decision derives from and the kind of +child it produces; every registered generator preserves its parent's kind. +"Completion" is the mechanical follow-up defined under Step, and a `Reattach` +names the domain that performs it. + +Every episode's probe requires the Spatial PnR and TechMapping views +unconditionally, and a `System` episode's probe requires the System PnR view +its subject arm carries. None of those is per-domain, so no row names a view +and no rule derives one from the table. + +A domain is admissible only when its row states every column and its decisions +are representable, meaning every owner-local reference they can carry resolves +to exactly one node under the target-closure rule of Combined Node And Link +Space. Representability is a property of the domain and the projection +together, so a future domain is admitted by extending both. + +Adoption is then one rule quantified over the table rather than a list of +per-domain checks: for every bound row, the subject it requires matches +`episode_subject`, and the domain any `Reattach` completion names is itself +bound; and no two bindings share a domain kind. Binding a Spatial row under a +`System` subject without also binding `SystemComposition` is therefore invalid +at adoption, because its completion could not run. + +Each row's decision union, its typed value domains, its canonical decision +order, and the rules that make one of its domain members well formed belong +entirely to that generator. The hardware unions of kinds 13, 14, and 15 are +defined by [Candidate Generators](spec-dse-feedback.md#candidate-generators); +the closed Canonical Dataflow rewrite catalog and its member ordinals are +defined by +[Canonical Dataflow Rewrite Catalog](spec-compiler-part-3-dfg.md#canonical-dataflow-rewrite-catalog), +which states that no other document may add, remove, rename, or reorder a +rewrite kind. This document reproduces none of them, so a catalog revision does +not strand a second copy here. + +What the environment adds is only the enumeration rule: for a bound row, its +generator's decisions in that generator's own canonical order, over the exact +parent the row's "Operates on" column names. Every owner-local enumeration rule +survives unchanged, including the ones that make a decision's availability +depend on the current Fabric, and every rewrite's equivalence obligation is its +catalog owner's. A software decision therefore changes how a workload is +expressed and never what it computes. + +`DataflowRewrite` explores the complete catalog over the workloads the episode +already holds; there is no environment-local catalog-kind selector, because a +subset selector would make the enumeration a filtered view of the generator's +canonical order rather than that order itself. Its binding carries the +generator's own immutable component view, +`loom.dataflow_rewrite_generator.config.1.1`, whose `scope_expansion_limit` and +semantic value domain are owned by +[Resolved Configuration](spec-config-ssot.md#schema-ownership). The other three +rows bind `SpatialTopologyRewriteConfig`, +`SpatialMicroarchitectureRewriteConfig`, and `SystemCompositionRewriteConfig` +respectively, the exact descriptor-owned records defined by +[Candidate Generators](spec-dse-feedback.md#candidate-generators). No row +introduces an environment-authored configuration record. + +Two families remain outside this contract rather than represented by inert +domains. `FabricTemplateConfig` of kind 12 is absent because a template +expansion selects an episode seed rather than advancing an episode. +`ImplementationFlowConfig` of kind 16 is absent because it produces a +`HardwareImplementation` rather than a design the feasibility gate can close +against a workload. Either is reopened by appending a domain kind that can +state all five obligations above. + +## Resolved Environment Configuration + +The environment consumes one immutable component view with schema descriptor +bytes `loom.ml_dse_environment.config.1.0`, following the framing, canonical +byte representation, and digest contract owned by +[Resolved Configuration](spec-config-ssot.md#component-views): + +```text +ResolvedMlDseEnvironmentConfigView { + episode_subject: EpisodeSubject + enabled_domains: canonical nonempty set + episode_policy: EpisodePolicy + probe_policy: ProbePolicy + selected_objective_closure: SelectedObjectiveClosure + observation_policy: ObservationPolicy + determinism_policy: EnvironmentDeterminismPolicy +} + +EpisodeSubject = + SpatialModule + | System { + admissible_modules: canonical nonempty set + system_pnr_config_view: ResolvedPnrConfigView + } + +EpisodePolicy { + seed_roots: canonical nonempty set + workload_pool: canonical nonempty set + workloads_per_episode: positive uint64 + start_retry_bound: positive uint64 + step_bound: positive uint64 + consecutive_rejection_bound: positive uint64 + visited_state_retention: uint64 + rejection_reward_code: uint64 + stop_reward_code: uint64 +} + +WorkloadBinding { + dataflow: ArtifactRootReference + tech_mapping: ArtifactRootReference +} + +ProbePolicy { + spatial_pnr_config_view: ResolvedPnrConfigView + tech_mapping_config_view: ResolvedTechMappingConfigView + warm_start_from_parent: bool +} + +ObservationPolicy { + enumeration_bound: positive uint32 + include_placement_edges: bool + include_resource_states: bool +} +``` + +`EpisodeSubject` carries what its arm needs rather than leaving subject-keyed +fields elsewhere to be required-or-forbidden by prose. The `System` arm holds +the finalized Module candidates kind 15 selects from and the System PnR view +that arm's probe requires, so neither can be present without the subject that +uses them or absent under the subject that needs them. + +`EpisodePolicy` owns every remaining input `reset` consumes, so an episode +start is a function of this view and the copy coordinates alone. Every +`seed_roots` member's root kind must match the subject arm, and +`workloads_per_episode` must not exceed the pool size. + +`tech_mapping_config_view` is unconditional: a TechMapping binds exact `D` and +`F`, so every advancing step regenerates at least one, whatever the bound +domains are. + +Membership in `enabled_domains` is the permission. A domain the set binds is +explored; a domain the set omits contributes nothing to the enumeration, no +masked entries, and no observation blocks. There is no separate permission flag +beside the binding, because a permitted domain with no configuration and a +configured domain that is forbidden are both states this contract has no use +for. + +`SelectedObjectiveClosure` and `ResolvedPnrConfigView` are the exact records +owned by +[Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism), +materialized as the selected transitive closure with view-local ordinals under +the projection rule owned by +[Component Views](spec-config-ssot.md#component-views). + +This view therefore contains two objective closures, and their roles are +disjoint. The closure inside `probe_policy.spatial_pnr_config_view` governs +PnR's internal search only: its total ordering, search energy, and focused +closure steer the probe toward a closed Mapping and never leave it. The view's +own `selected_objective_closure` is the sole reward authority; its search +energy is the value the reward differences, and it may carry dimensions the +probe has no notion of, such as a parameter-backed predicted metric. Neither +closure derives, overrides, or defaults the other. + +Adoption requires them to be consistent where they overlap. Every +`MappingViolationSource` and `MappingMeasureSource` dimension in the reward +closure must reference a descriptor that the probe view's temporary-violation +policy and measure catalog admit, so a reward dimension cannot read a fact the +probe never produces. A dimension referencing an obligation tagged `HeldOut` is +invalid here for the reason it is invalid everywhere. A reward closure whose +Evaluation-sourced dimensions no registered parameter contract satisfies is +invalid at adoption. A closure whose dimensions are unsatisfied by any selected +closure is invalid at adoption rather than at the first step that needs the +missing value. The set of parameter contracts an episode invokes needs no field +of its own: it is exactly the set the closure's Evaluation-sourced dimensions +reference, and a field beside it could only disagree. + +The canonical view encoder writes fields in the schema order above, under the +encoding and adoption rules +[Resolved Configuration](spec-config-ssot.md#component-views) owns. This view +adds only that its optionals use a `u32be` absent or present discriminant +followed by the payload when present. + +The PRNG preimage shape, the meaning of `effective_seed` and the copy +coordinates, the `reset` seed override, and the exclusion of policy sampling +from every environment stream are owned by +[Determinism And Copy Coordinates](spec-ml-core-environment.md#determinism-and-copy-coordinates). +This document adds only its own domain separator, which is + +```text +ASCII("loom.ml_dse_environment.prng.sha256_seeded_xoshiro256starstar.1.0") +``` + +and its stream purpose ordinals, which are `SeedSelection = 0` and +`WorkloadSelection = 1`. + +### Curriculum-Neutral Fields + +[Curriculum](spec-ml-core-training.md#curriculum) requires each environment +document to partition its own view by what a field determines. Here the +shape-fixing side is `episode_subject`, `enabled_domains`, +`selected_objective_closure`, `probe_policy`, `observation_policy`, and +`determinism_policy`: the first two fix the action space's extent, the closure +and the reward codes fix what a return means, the observation policy fixes the +column catalog the embedding tables are sized for, and the determinism policy +fixes the stream layout. + +Everything inside `EpisodePolicy` except the two reward codes is neutral, which +is what a curriculum over this environment actually wants to move: the seed +roots, the workload pool, the workloads per episode, the step and rejection +bounds, and the visited-set retention. Moving from small architectures to large +ones is a change of `seed_roots`, and lengthening episodes is a change of +`step_bound`. The Mapping cache survives such an advance, because its key names +one seed root and one workload rather than the sets this view declares: +extending `seed_roots` or `workload_pool` leaves every entry the earlier stages +warmed valid, which is what makes growing a pool the affordable curriculum it +is meant to be. + +## Workload Feasibility Invariant + +Every state the environment occupies is workload-feasible. For each Canonical +Dataflow Program in the state's current workload set, the probe must produce a +Mapping that could be finalized: full owner recomputation succeeds and all five +Mapping violation magnitudes are zero. This is the same closure condition +[Place And Route](spec-pnr.md#final-closure-and-verification) requires of a +selected candidate, evaluated over the ephemeral candidate rather than a +published Artifact. + +The closure level follows the episode subject. A `SpatialModule` episode +requires one closed Spatial Mapping per workload against the current Module. A +`System` episode requires one closed System Mapping per workload against the +current System, together with the Spatial Mappings that System Mapping imports; +the environment does not accept a System whose constituent Modules close +individually while the System does not, because service continuity, transport +capacity, and progress closure are System-level facts. + +The invariant is a gate on state occupancy, not a term in the energy. Removing +the last function unit that supplies an operation the workload needs, deleting +the last route between two required endpoints, shrinking a memory or FIFO below +what the workload's claims demand, narrowing a boundary inventory the workload +binds, removing the AccCore a workload's launch requires, or cutting the last +transport connection carrying a service leg all end the step. The agent never +occupies such a state and never receives a graded score for approaching one. + +When `DataflowRewrite` is bound the workload set is itself part of the state. A +software decision replaces one workload with its rewritten child, and the +invariant is evaluated over the resulting set: a rewrite that produces a +program the current hardware cannot map is rejected exactly as a hardware +decision that removes a resource the workload needs. The two directions are +symmetric, which is the point of exploring them jointly. Rewrite legality +itself is never at issue here, because every catalog rewrite is externally +equivalence-preserving by its owner's contract; what a rejection reports is +that this hardware cannot map that expression of the workload. + +Three consequences follow. + +The enumeration mask is advisory and feasibility is authoritative. Decision +domains constrain which decisions exist, and the Builder rejects a draft that +violates an authoring-boundary rule, but neither knows in advance which legal +decision removes the resource some workload happens to require. An unmasked +action is therefore not a guarantee that the step advances. The environment +treats a legal-but-infeasible decision as an ordinary first-class outcome; it +is not an assertion failure, not an exception, and not a silent no-op that +leaves the agent unable to distinguish action from inaction. + +Proof and budget are different answers and are never merged. A PnR +`ProvenInfeasible` result is a sound proof that no Mapping exists for that +workload on that candidate. Exhausting the probe's configured search work with +residual violations is inconclusive: it establishes nothing about the design. +Only exact admission or a sound bound may prove infeasibility, so an exhausted +budget is never reported as a proof. Both reject the step, and both are +recorded under distinct reasons so that a rising inconclusive rate is legible +as a probe-budget problem rather than as a shrinking design space. + +Failure is attributed per workload. The step record names the exact Canonical +Dataflow that failed, its reason, and its residual violation magnitudes by +descriptor ordinal. A step that fails several workloads records the first +failure in canonical probe order; the environment does not continue probing +after the invariant is already broken. + +## Action Space + +The action space is the concatenation of each enabled domain's canonical +decision order. For the current state the environment enumerates every bound +domain in ascending `ExplorationDomainKind` ordinal, and within a domain +enumerates that generator's decisions in the generator's own exact canonical +order, truncated by its `max_children_per_parent` or `scope_expansion_limit`. +Those are the generators' own semantic work policies and are the only +truncations applied. `enumeration_bound` is an observation capacity, not a +further truncation: a state whose enumeration exceeds it is refused as +described under Observation, never silently shortened. Ascending domain ordinal +is the only cross-domain rule, so enabling or disabling a domain shifts a +suffix of the enumeration but never permutes another domain's internal order. +An action is one ordinal in the concatenation: + +```text +EnumeratedDecision { + domain: ExplorationDomainKind + decision_kind: u32be ordinal in that domain's closed decision union + parent: ParentSelector + targets: ordered nonempty sequence + value: ReplacementPrototype(prototype ordinal) + | BoundedIntegerDelta(delta) + | NoValue +} + +DecisionTarget { + reference: owner-local reference + role: Primary | GroupMember +} + +ParentSelector = + EpisodeSubject + | SubjectComponent(FabricLocalReference) + | CollectionOrdinal(uint32) + +ActionIndex = uint32 +``` + +`parent` qualifies the whole decision because every reference in one decision's +normalized payload is local to the same root; a decision does not straddle two +parents. `targets` carries every reference that payload names, in its canonical +order, with at most one `Primary`. A decision naming one entity has a single +target, and one naming an entity plus a reference set has a `Primary` followed +by its `GroupMember` members, which is the shape the observation's target arcs +and the policy's anchor summary both consume. + +An owner-local reference is the reference its owner already defines: a +`FabricLocalReference` from +[Fabric Identity](spec-fabric-identity.md#owner-local-reference-kind-catalog), +or the `ActorRef`, `GraphRef`, or `StaticGraphLaunchRef` the rewrite catalog's +normalized decision carries. The environment transports them unchanged and +assigns no parallel identifier. + +Every such reference is local to one root, so a target is meaningless without +naming which root, and `ParentSelector` is that qualifier in one form for every +domain. `EpisodeSubject` selects the subject itself. `SubjectComponent` selects +a component of the subject by an ordinary Fabric reference, which is how a +decision whose row operates on a `fabric.module` addresses one Module of a +`fabric.system`. `CollectionOrdinal` selects a member of an ordinal-indexed +collection the state holds, which is how a decision addresses one workload. The +legal selector for a decision is derived from its row's "Operates on" column +against the episode subject rather than written out per domain: a row that +operates on the subject's own kind uses `EpisodeSubject`, and any other +selector for it is invalid. + +`NoValue` covers a decision fully determined by its target. + +Collection ordinals are assigned at `reset` in the canonical order of the +selected collection and are stable for the whole episode. A `ReplaceParent` +completion substitutes the child at exactly its parent's ordinal and leaves +every other ordinal untouched, so a retained target, a retained mask bit, and a +node block never silently repoint at a different member because an identity +sorted differently after a decision. + +The interaction surface, the `enumeration_bound` capacity, the masking algebra, +the elective stop, and the refusal of an over-capacity enumeration are owned by +[Action Surface And Masking](spec-ml-core-environment.md#action-surface-and-masking). +This document adds only which mask bits an environment clears: an index whose +decision was already rejected at this state, under the per-state rejection mask +Backtracking defines. + +Every enumerated decision's references resolve into the combined node-and-link +observation space, so a decision that names an occurrence, a connection, a +transport link, an actor, or a graph is scored the same way: by attending over +the graph nodes it targets. Per-node and per-link actions are therefore the +observation's shape rather than a second indexing scheme, and the action index +stays a single ordinal even for a decision that carries several references. + +Indexing the canonical decision order is what removes the per-type stride a +flat product space over node slots and decision types would need, and it +inherits determinism directly from the generator descriptor. + +### Schedule-Preserving Transformations + +Some decisions are worth taking only in a form the current schedule determines; +why that is worth the coupling is +[Why Some Design-Space Actions Read The Schedule](rationales/ml.md#why-some-design-space-actions-read-the-schedule). +A schedule-preserving transformation is one whose `GroupMember` references are +chosen from the state's own Mapping rather than from the Fabric's structure +alone: the decision kind, its `Primary` target, and the legality of the child +are exactly what they would be otherwise, and what the Mapping supplies is +which further references the decision names. + +`RemoveOccurrence` is the case that motivates the class. Deleting an occurrence +the Mapping currently routes through breaks every net whose route traverses it, +so a useful removal carries reconnection: links joining the upstream and +downstream endpoints of the traversals that occurrence was carrying. Which +links those are is not a question the topology can answer. The topology admits +an enormous number of reconnections, nearly all of them useless, and the +complete set would replace one occurrence with a connectivity explosion. The +Mapping names the few that the workload actually uses, because they are the +ones its routes already run through. + +That is what makes the class schedule-*preserving*. A child reconnected from +the parent's routes leaves the parent's placement carryable, so warm-starting +that child's probe begins from a mapping that still closes rather than one the +edit invalidated. A structurally reconnected child usually has to be re-mapped +from nothing, which costs the probe the option exists to save and discards the +evidence that the parent's placement was good. + +The reference set is Mapping-derived; the enumeration's extent is not. A state +offers one entry per removable occurrence whether or not a Mapping exists, so +the enumeration's length remains a function of the state and the resolved +configuration alone. This is load-bearing rather than incidental: it is what +keeps step 7 ahead of step 8, keeps the reset capacity test cheap enough to +precede workload feasibility, and keeps that test's retained per-seed verdict +valid. Only the references inside an entry wait for the state's probe, and they +are filled when the observation that reports them is built. + +The environment authors no reconnection semantics. Every reference it names is +an ordinary owner-local reference, every decision it emits is an existing +generator decision, and stage 3 runs the owner's ordinary acceptance path over +the result — so a reconnection the Mapping suggested and the owner refuses is +an ordinary rejection rather than a special case. The Mapping is being read as +evidence about which references are worth naming, never as authority for +whether the child is legal. + +Path dependence follows the probe. With `warm_start_from_parent` clear, a +state's Mapping is a pure function of that state, so the reference set is too +and the core's enumeration-purity rule holds unchanged. With it set, the +Mapping is path-dependent by construction, so a schedule-preserving decision's +references are path-dependent with it. That is the same cost the option already +carries for the reward, extended to the content of an action, and it is +recorded here rather than discovered when two runs over one state disagree +about what they were offered. + +Replay needs no Mapping. A `Trajectory` records each step's full +`EnumeratedDecision`, references included, and replay runs the recorded +decision rather than re-deriving it, so a replayed step reproduces the +reconnection the episode actually took without reconstructing the probe that +suggested it. + +## Observation + +### Combined Node And Link Space + +The combined node-and-link space, its `GraphNodeRole` catalog, the +connection-as-node rule, the two enumeration encodings, and the target-closure +rule are owned by +[Combined Node And Link Space](spec-ml-core-environment.md#combined-node-and-link-space) +and [Enumeration Encoding](spec-ml-core-environment.md#enumeration-encoding). +This document states only what that space contains for a DSE episode. + +This environment uses the `DecisionNodes` encoding, because its decisions have +variable target arity: one names a single entity, and another names a +distinguished subject plus a reference set whose size the generator decides. +Only out-degree expresses that, so the `Decision` block is present and +`decisions` is empty. + +Every decision kind this environment enumerates depends on the closure. +Removing an occurrence names an occurrence, replacing a point connection names +a connection, changing a transport connection names a System link, and +refactoring a graph definition names graphs and launches, so an exploration +domain whose decisions can carry a reference the space cannot represent is not +admissible. That admissibility test is the reason the Exploration Domains table +requires representability of every row. + +`DataflowOperation` and `DataflowValue` blocks are present exactly when a bound +domain operates on a Canonical Dataflow Program, which needs no configuration +flag because the binding already decides it. `AdjustParallelConnectionCount` +has exactly one target because a parallel connection is one node carrying its +count as a feature. + +When `include_placement_edges` is set, the probe's current Mapping contributes +one `Placement` arc from each Dataflow node to the Fabric node that realizes it +and one `Route` arc from each Dataflow value node to each Fabric node its route +traverses, which is how the policy sees the present mapping. The two roles are +distinct because the two relations are: realization is one-to-one and is what a +capacity decision moves, while traversal is one-to-many and is what a +connectivity decision moves, and an encoder given one role for both would have +to separate them from context it does not have. Those arcs carry no Mapping +semantics of their own and are derived from the probe result of the state being +observed. + +### The Graph Instance + +The `Observation` and `GraphInstance` container shapes, the no-padding rule, +the negative-one absent sentinel, the buffer lifetime, and the obligations +every column catalog satisfies are owned by +[The Graph Instance](spec-ml-core-environment.md#the-graph-instance). The +closed column catalogs this environment owns are: + +```text +DseNodeFeatureColumn = + Role // 0 + | EntityKind // 1 + | CapabilityCount // 2 + | CapacityMagnitude // 3 + | BufferDepth // 4 + | ResidualViolationCount // 5 + | PlacedDegree // 6 + | DecisionDomain // 7 + | DecisionKind // 8 + | DecisionValueForm // 9 + | DecisionValueOrdinal // 10 + | DecisionDeltaMagnitude // 11 + | DecisionDeltaSign // 12 + +DseArcRole = + Structural // 0 + | Placement // 1 + | Route // 2 + | DecisionTargetPrimary // 3 + | DecisionTargetMember // 4 + +DseScalarFeatureColumn = + StepOrdinal // 0 + | StepBound // 1 + | WorkloadCount // 2 + | ConsecutiveRejections // 3 +``` + +The six decision columns span the `Decision` block and the entity columns span +the entity blocks, on the core's role-block span rule. + +There is no node-ordinal column: a node's ordinal is its position in the array, +and the core's own rule forbids consuming an ordinal as a projected scalar. + +`EntityKind` is the owner-local reference kind ordinal from +[Fabric Identity](spec-fabric-identity.md) for a Fabric node and the Dataflow +owner's node or value kind ordinal for a Dataflow node. Every System entity +therefore appears as an ordinary `FabricOccurrence` node carrying its own +Fabric Identity kind ordinal, and transport and service connections appear as +ordinary `FabricConnection` nodes. + +There is no `LiveDecisionCount` scalar. The live count is the `Decision` +block's size, which the instance already carries, and a scalar beside it would +be a second source for one extent. + +A decision node carries at most one `Primary` target arc, its distinguished +subject when its owner defines one, and any number of `GroupMember` target +arcs, which are the members of a reference set or tuple the decision carries. A +decision whose owner defines no distinguished subject has only `GroupMember` +arcs. That is the shape a decision naming one entity plus a complete adapter +set requires; a projection that kept only the first target would silently +discard what the decision actually rewrites. + +When a bound domain operates on a Canonical Dataflow Program, the +`DataflowOperation` and `DataflowValue` blocks span every workload in the +state's current workload set rather than one workload, in collection-ordinal +order, so a rewrite decision's target node is addressable exactly as a Fabric +decision's target is. Advancing on a software decision changes how many nodes +those blocks contribute, which changes the instance's extents; that is ordinary +for this space and needs no accommodation. + +`CapabilityCount`, `CapacityMagnitude`, and `BufferDepth` are mechanical +projections of the exact Fabric owner facts for the referenced entity. +`ResidualViolationCount` and `PlacedDegree` are projections of the probe +result. The typed atoms of the `FabricResourceStateRef` catalog the core +appends under `include_resource_states` are owned by +[Fabric Resource Contract](spec-fabric-resource-contract.md). + +The scalars a rejected step updates in place, under the core's buffer-lifetime +rule, are the cleared mask bit and the step and rejection counters. + +A state whose enumeration exceeds `enumeration_bound` is rejected with +`EnumerationBoundExceeded`, which is the core's refusal rule reported under +this environment's own reason name; truncation would permanently hide the +decisions of whichever domains sort last. + +## Episode Start + +An episode is created from an exact seed set and an exact workload set. Each +seed is a finalized Fabric root whose kind matches `episode_subject`: a +`fabric.module` for a `SpatialModule` episode and a `fabric.system` for a +`System` episode. A seed is a builtin target, a user-supplied Fabric, or one +output of the `fabric_template` generator; exploration never begins from an +empty mutable graph, and a seed of the wrong kind is invalid rather than +adapted. Each workload is an exact Canonical Dataflow Program Artifact together +with the TechMapping the probe requires; both are ordinary static inputs and +acquire no environment-local identity. + +The `System` subject arm carries the admissible finalized Module candidate set +that kind 15 consumes, since `AddAccCore` and `ReplaceSpatialAttachment` select +from an explicit finite set rather than synthesizing a Module. That set is an +ordinary generator input the configuration supplies; the environment does not +author or extend it. A Module a Spatial domain derives mid-episode is reachable +by a later completion because it is the child of the decision being completed, +not because the environment added it to a candidate set it does not own. + +`reset` performs this ordered protocol: + +1. derive the episode's PRNG streams from the effective seed and the copy + coordinates; +2. select one seed root from `seed_roots` through `SeedSelection`, or adopt the + one the start override supplies; +3. select the episode's initial workload set through `WorkloadSelection`, or + adopt the one the start override supplies; +4. enumerate the seed and reject it when the enumeration is empty or exceeds + `enumeration_bound`; +5. insert the initial state's identity into the visited set; and +6. establish the workload-feasibility invariant by obtaining one closed Mapping + at the episode subject's closure level for every selected workload, then + build the first observation. + +Step 5 is what makes the seed state visited from the episode's first step. +Omitting it would make the one state every episode is guaranteed to occupy the +one state a decision may return to for free, and a pair of inverse decisions +would cycle through it indefinitely without ever being rejected as +`AlreadyVisited` — precisely the circling the visited set exists to catch. + +Step 4 precedes step 6 for the reason the step protocol orders its own tests: +the enumeration is a pure function of the seed and the configuration, while a +closure-level Mapping is the dominant start cost. Enumerating first means a +seed that could never offer a legal action is rejected for a fraction of one +Mapping, rather than after `workloads_per_episode` of them, and the retry +budget multiplies that saving by every redraw. + +Being a pure function of the seed and the enumeration-relevant fields of the +view, step 4's verdict is retained per seed root against those fields, and a +redraw that lands on a seed already judged consults that verdict instead of +re-enumerating. Keying it on the enumeration-relevant fields rather than on the +whole view is what lets a stage advance that only grows the workload pool keep +every verdict, on the same terms the Mapping cache survives one. A run of +`10^5` episodes over a few hundred seed roots would otherwise recompute a few +hundred fixed answers thousands of times each, and an unusable seed would be +re-enumerated and re-rejected on every episode that happened to draw it. This +is an ordinary cache: a hit and a miss produce the same formal result, and the +retry budget still bounds the redraws. + +The per-workload work in step 5 has no cross-item dependence and may run +concurrently, on the same terms stage 8 of a step states. + +The start override is the Gymnasium `options` argument, carrying an exact +episode start rather than a drawn one: + +```text +EpisodeStartOverride { + seed_root: ArtifactRootReference + workloads: ordered nonempty sequence +} +``` + +The core's override rules bind it: its members must appear in `seed_roots` and +`workload_pool`, and when present it replaces steps 2 and 3 entirely, so +neither `SeedSelection` nor `WorkloadSelection` is consulted. + +Step 5 is a precondition, not a best effort. A seed that cannot map every +selected workload is not a valid episode start: + +```text +EpisodeStartOutcome = + Started + | SeedInfeasible { seed_root, workload } + | SeedMappingNotClosed { seed_root, workload } + | SeedEnumerationEmpty { seed_root } + | SeedEnumerationBoundExceeded { seed_root, bound, required } + | RetryBudgetExhausted + | Invalid { violated precondition } +``` + +`SeedInfeasible` and `SeedMappingNotClosed` carry the proof-versus-budget +distinction the feasibility invariant defines. Either causes a retry with the +next draw from the same deterministic streams, bounded by `start_retry_bound`; +exhausting it is `RetryBudgetExhausted` and never a silently degraded episode +over a subset of the workloads. + +Under an episode start override there is no next draw, so `start_retry_bound` +is what the core's no-retry rule leaves unconsulted, and the outcome is +returned rather than converted to `RetryBudgetExhausted`. `Invalid` covers a +seed whose root kind does +not match the subject arm and a workload whose TechMapping does not bind the +selected seed. + +The two enumeration failures are both start failures rather than valid +episodes, and both are retried. A drawn seed whose enumeration is empty is +reported as `SeedEnumerationEmpty` because `Started` must guarantee at least +one legal action index; a drawn seed whose enumeration exceeds +`enumeration_bound` is reported as `SeedEnumerationBoundExceeded`, carrying the +capacity and the required length. Neither is `Invalid`, since another seed in +the same set may enumerate within capacity, and `reset` step 4 already tests +them together. Only a configuration whose every seed fails one of the two is an +invalid configuration, and that is reported when the retry budget is exhausted +over it. + +Mapping and TechMapping results may come from an ordinary cache family under +[Cache Dependencies](spec-config-ssot.md#cache-dependencies), which already +owns the versioned canonical dependency key, the removability rule, and the +requirement that a hit and a miss produce the same formal result. This document +adds only the granularity: the cached unit is one workload, not the episode's +workload set. A Mapping depends on the seed root, that one workload with its +TechMapping, and every probe view that produced it, and on nothing about which +other workloads the episode happened to draw. Every probe view means both the +Spatial PnR view and, for a `System` episode, the System PnR view its subject +arm carries: a System-episode entry produced under one System PnR contract must +not be returned to an episode running another, which is exactly the case where +a hit would return a Mapping a miss would not have produced. Keying on the set +would instead make the key space the combinations of the pool rather than the +pool itself, so changing one workload would discard the mappings of every other +workload in the draw even though they are byte-identical and reusable. +Per-workload keying makes the same cache reach a useful hit rate after a few +dozen episodes instead of effectively never. + +The same grant covers stage 8, and it matters more there than at `reset`. A +TechMapping is a pure function of the child identity, one workload, and the +TechMapping view, unconditionally. A probe result is a pure function of the +child identity, one workload with its TechMapping, and the probe views whenever +`warm_start_from_parent` is false, which is exactly the configuration whose +energy is path-independent. Both recur constantly once a policy converges on a +decision kind, and they recur across every environment copy in the run, so a +step-time cache reaches a useful hit rate far sooner than the start-time one it +extends. With warm starting enabled the probe is path-dependent by construction +and is not cacheable; that is a further cost of the option, and it is measured +rather than assumed. + +## Step + +One step applies exactly one enumerated decision. The protocol is ordered, and +every stage may reject: + +1. validate the action against the live mask and the retained per-state + rejection mask; +2. derive a fresh draft from the exact parent the decision's domain names and + apply the one owner-typed decision to it; +3. run the domain owner's ordinary acceptance path for that child, stopping + before durable publication; +4. publish the accepted child through its own generator when and only when the + decision's row completes by `Reattach`, skipping a component whose identity + the store already resolves; +5. derive the completion its domain's row names, so the state is again a + root-complete subject of the episode's kind; +6. reject when the resulting state identity is already in the visited set; +7. reject when the resulting state's enumeration exceeds `enumeration_bound`; +8. for each workload in canonical order, regenerate its TechMapping if the + decision invalidated it and then probe it against the candidate, + warm-started from the parent state's mapping when `warm_start_from_parent` + is set, requiring a closed Mapping with all five violation magnitudes zero; +9. read the `G` measures from the probe and every predicted metric from the + parameter-bundle inference kernels the selected closure's dimensions + reference; and +10. quantize each objective source, form the new `ObjectiveVector`, and + evaluate the selected search-energy `WeightedLevel`. + +The order is cheapest-rejecting-first as far as the completion form allows, and +where it holds it is load-bearing rather than incidental. Stages 6 and 7 both +answer from the completed state alone: a state identity and the enumeration +length are functions of that state and the resolved configuration, with no +Mapping, TechMapping, or objective value involved. Placing them ahead of stage +8 means a revisit or an over-capacity candidate never costs a probe, and both +are expected outcomes rather than corner cases. This is the property +[Schedule-Preserving Transformations](#schedule-preserving-transformations) +preserves by keeping a Mapping-derived reference set inside an entry whose +existence the Mapping does not decide. + +For a `None` or `ReplaceParent` completion the ordering is unqualified, because +stage 4 publishes nothing and those two rejections therefore also precede every +durable write. A `Reattach` is the exception, and it is the reason stage 4 sits +where it does. An Artifact owner requires every direct dependency of a root to +be durably published before that root's closure is derived, and a subject that +names a component depends on it, so a component held only as an unpublished +verified closure cannot be reattached at all. The completion therefore cannot +be derived before its component is published, and the state identity stages 6 +and 7 test does not exist until the completion has run. Deriving the completion +first and publishing afterwards is not a cheaper ordering but an invalid one: +it would name a subject closure over a dependency the store does not hold. + +A `Reattach` step consequently pays its component's store insertion before the +visited and capacity tests, which is the case where the memo below is worth the +most. + +Stage 4 deduplicates. The component's canonical identity is already exact from +stage 3, and the store is content-addressed, so a component whose identity the +store already resolves is not re-serialized and not re-inserted. A converging +policy revisits the same component constantly, and without this rule every +revisit would rewrite bytes the store already holds. An implementation may +answer stage 6 earlier still, from a memo of +`(parent state identity, EnumeratedDecision)` to child state identity, and skip +stages 2 through 5 entirely on a hit; the child identity is a deterministic +function of that pair, so the memo is an optimization and not a second identity +authority. On a `Reattach` row that memo is what keeps a revisit from paying a +publication for a state the episode is about to reject. + +Stage 8 interleaves regeneration with probing per workload rather than +regenerating every TechMapping first. The two orders are observationally +identical, because probe order is already the order in which a failure is +reported, but the interleaved one stops at the first failing workload instead +of paying regeneration for workloads it never probes. That matters because a +Fabric decision invalidates every workload's TechMapping while the probe aborts +on the first failure, so the barrier form would waste all but one regeneration +on precisely the rejections early abort exists to make cheap. + +Independent work within a stage may run concurrently. Per-workload regeneration +and probing in stage 8 and per-contract inference in stage 9 have no cross-item +dependence, and a concurrent implementation satisfies this contract exactly +when it reports the failure lowest in the configured probe order among those it +observed. Cancelling the remaining probes once a failure is known is permitted +and expected; what is fixed is the reported result, not the execution order. + +Stage 3 is what makes the candidate real without making it persistent. For a +Fabric domain it produces the `VerifiedFabricClosure` owned by +[Finalization And Publication](spec-fabric-artifact.md#finalization-and-publication) +and does not publish it, so the child has passed the same finalizer and +verifier as a published Fabric and carries the same canonical identity while +remaining un-referenceable by any other owner. For `DataflowRewrite` the child +passes the same rewrite match, normalization, and equivalence obligations its +catalog owner requires. In every case the candidate is fully verified rather +than provisionally accepted, and its identity is exact enough to serve as a +visited-set key. + +What stage 4 publishes is the component alone. It is published by its own +generator as an ordinary intermediate candidate, which is the status its owner +already gives it, while the subject closure that consumes it remains +unpublished. Deferred publication is a property of the episode's subject, not +of every value a step touches. + +This is a real cost and it is stated rather than hidden: a `Reattach` step pays +canonical publication and store insertion for its component, so a `System` +episode with a Spatial domain bound is more expensive per step than a +`SpatialModule` episode over the same decisions, and it accumulates +content-addressed objects for components no selected design may ever reference. +An episode that does not want that cost does not bind a Spatial domain under a +`System` subject; the alternating batches of the ordinary joint search reach +the same designs without it. + +Stage 5 exists because a generator's child is not always the episode's subject, +and it is driven by the "Completion" column rather than by named domain pairs: + +```text +Completion = + None + | Reattach { via: ExplorationDomainKind } + | ReplaceParent +``` + +`None` means the child is already the subject. `Reattach` means the child is a +component of the subject, so stage 4 publishes that child through its own +generator's ordinary path and the named domain then applies the one decision +that rebinds exactly the parent selector the original decision targeted, +leaving every other relation untouched. The reattach invocation receives the +published child as its admissible component set, which is an ordinary generator +input the caller supplies per invocation, not a configuration field the +environment extends. `ReplaceParent` means the child substitutes for its parent +in an ordinal-indexed collection the state holds, at exactly that parent's +ordinal, and publishes nothing. + +A row's completion is derived rather than chosen: `None` when the row's +"Operates on" kind is the episode subject's kind, `ReplaceParent` when it +operates on a member of a collection the state holds, and otherwise `Reattach` +via the unique bound row whose "Operates on" kind is the subject's. That is the +same derivation the `ParentSelector` rule already uses, so the table's +Completion column records the outcome rather than introducing a free parameter. + +No completion invents a decision the agent did not take. Both forms are +functions of the decision's own parent selector, which is why they need no +choice of their own; a domain whose completion would require a second free +choice cannot state its row and is therefore not admissible. A step may expand +to more than one generator invocation on replay, which Trajectory Retention And +Replay records exactly. + +Stage 8's regeneration runs on every advancing step, because a TechMapping is +bound to exact `D` and `F` and every decision changes one of them. What differs +is how much it invalidates: a software decision invalidates the TechMapping of +exactly the rewritten workload, while a Fabric decision invalidates every +workload's, so the stage is most expensive for the domain that changes the +fewest programs. Regeneration uses the +[root-complete TechMapping path](spec-tech-mapping.md#root-complete-central-adapter) +with `probe_policy.tech_mapping_config_view`. A workload whose TechMapping +cannot be regenerated has no probe input at all, which is a distinct rejection +from a workload that maps poorly. + +Stage 8 is also the feasibility gate defined above. Stage 9 obtains predicted +metrics by calling a registered `ModelParameterContract` 's `project_features` +and `infer` in process; this is the parameter-backed tier of the +fast-evaluation cascade owned by +[Full-Stack Architecture](spec-loom-stack.md#joint-design-exploration). A +predicted value may rank and may reward, but it can never reject a candidate as +impossible, and `infer` returning `Unsupported` outside its exact training +support region is never replaced by an extrapolation, a bound, a midpoint, or a +default. + +Only the contracts the selected closure's dimensions reference are invoked, and +that set is the definition rather than a configured list. Inference is the +expensive tier of the cascade, so a contract whose prediction no objective +dimension reads is never run at all. + +The outcome algebra is closed: + +```text +StepResult { + transition: optional + episode_end: optional +} + +StepTransition = + Advanced { energy_delta_code, energy_delta_sign } + | Rejected { reason: RejectionReason } + +RejectionReason = + DraftRejected { domain, authoring diagnostic } + | FinalizationRejected { domain, finalizer diagnostic } + | ClosureCompletionRejected { domain, completion diagnostic } + | TechMappingUnavailable { workload } + | AlreadyVisited { identity } + | WorkloadProvenInfeasible { workload, violation magnitudes } + | WorkloadMappingNotClosed { workload, residual violation magnitudes } + | ObjectiveUnavailable { dimension } + | EnumerationBoundExceeded { bound, required } + +TerminalReason = + ElectiveStop + | StepBoundReached + | ConsecutiveRejectionBoundReached + | EnumerationEmpty +``` + +`WorkloadProvenInfeasible` and `WorkloadMappingNotClosed` stay separate for the +reason the feasibility invariant states: one is a proof and the other is a +budget artifact. `TechMappingUnavailable` is separate from both because the +workload never reached the probe, so nothing was established about mappability +at all. `ClosureCompletionRejected` reports that a child was accepted by its +own owner but could not be reattached as a root-complete subject, which is a +System-level or workload-set-level failure rather than a defect in the child. +`ObjectiveUnavailable` is this environment's name for the core's +unavailable-objective outcome, and `EnumerationBoundExceeded` is its name for +the core's over-capacity refusal. Every reason names the domain, workload, or +bound it belongs to, so a rejection profile attributes cost and failure to the +domain that produced it. + +A transition and an episode ending are separate facts because they co-occur, on +the terms +[Step Accounting And Episode Endings](spec-ml-core-environment.md#step-accounting-and-episode-endings) +states. Here the rejection that reaches `consecutive_rejection_bound` both +rejects and ends the episode; the advance that lands in a state with no +admissible decisions both advances and ends it; and an elective stop ends the +episode with no transition at all. + +An absent transition means the policy ended the episode instead of acting, +which happens exactly for `ElectiveStop`; a third union member for that case +would let a consumer write two checks that can disagree. `EnumerationEmpty` is +reported on the step that advances into the empty state, not on a later call, +so the consumer is never handed a state with no legal action index. + +`Advanced` commits the candidate as the new current state, inserts its identity +into the visited set, and clears the per-state rejection mask for the new +state. Every `Rejected` transition restores the parent state exactly and +advances no episode position other than the step counter. The state identity +used for the visited set is the episode subject's identity together with the +ordinal-indexed identity sequence of the current workload set, so a software +decision that returns the design to a previously visited pair is caught exactly +as a hardware decision that does. + +`episode_end` distinguishes termination from truncation on the terms +[Step Accounting And Episode Endings](spec-ml-core-environment.md#step-accounting-and-episode-endings) +states. `ElectiveStop` and `EnumerationEmpty` are terminations; +`StepBoundReached` and `ConsecutiveRejectionBoundReached` are truncations. + +## Backtracking + +A step never mutates the parent. Stage 2 derives a fresh draft from the exact +parent, so a rejection has nothing to roll back: it discards the child and the +parent is the value it already was. The parent's canonical identity after a +rejection therefore equals its identity before the attempt as a consequence of +the protocol, not as a promise about anyone's restore path. The only +transactional state in a step is internal to the probe, where +[Evaluation Transaction](spec-pnr.md#evaluation-transaction) owns `commit` and +`discard`. + +The environment then clears the rejected decision's bit in a per-state +rejection mask so the same decision is not re-proposed from the same state. The +mask belongs to the state, not to the episode: advancing to a new state starts +a new empty mask, and returning to a previously visited identity is already +rejected by the visited set. The visited set retains up to +`visited_state_retention` identities in insertion order, which is what prevents +an episode from spending its step budget alternating between two states. + +The step accounting identity +[Step Accounting And Episode Endings](spec-ml-core-environment.md#step-accounting-and-episode-endings) +states holds over an episode of this environment with no unaccounted remainder. +Consecutive rejections are counted, exposed through `scalar_features`, and end +the episode at the configured bound. + +## Reward + +Reward is the signed difference of the selected search energy across the +transition, on the terms +[Reward Contract](spec-ml-core-environment.md#reward-contract) states. This +document says only which two states the difference is taken between and what +this environment's non-transition outcomes cost. + +The parent's energy the core retains is the one produced when the parent was +entered, by `reset` or by the stage 10 that advanced into it. + +A `Rejected` transition yields `rejection_reward_code` with a negative sign, +independent of the rejection reason. + +`stop_reward_code` applies with a negative sign to `ElectiveStop` alone. +`StepBoundReached` and `ConsecutiveRejectionBoundReached` yield zero because +they are truncations, which the core prices at zero. `EnumerationEmpty` also +yields zero, since the step that reports it already carries its own `Advanced` +reward. Only the ending the policy elects is priced. + +Path dependence is a consequence of `warm_start_from_parent` and is stated +rather than hidden. Warm starting seeds each probe from the parent's mappings, +so the mapping a probe returns, and therefore the `G` measures and the energy, +can depend on how a state was reached. State identity does not depend on the +path, so two paths to the same design may report different energies and a cycle +of deltas need not sum to zero. A configuration that requires path-independent +energy, such as one whose analysis assumes telescoping deltas, sets +`warm_start_from_parent` false and pays the cold-probe cost; a configuration +optimizing throughput sets it true and accepts a reward that is exact per +transition but not a potential function. + +## Trajectory Retention And Replay + +An episode retains one transient record: + +```text +Trajectory { + config_view_digest: ComponentViewDigest + episode_subject: SpatialModule | System + seed_root: ArtifactRootReference + initial_workloads: ordered sequence + effective_seed: u64 + env_runner_index: uint32 + vector_index: uint32 + local_episode_index: uint64 + advanced_steps: ordered sequence +} + +TrajectoryStep { + decision: EnumeratedDecision + completion: optional + child_identity: ArtifactIdentity +} +``` + +A step records the decision the agent chose, the mechanical completion stage 5 +derived, and the exact identity of the child that step produced, which stage 3 +computed whether or not it published. + +Replay is a finite sequence of resolved plans, one per recorded step, each +resolved after its predecessor completed. A step's plan binds its parent as an +exact static artifact set, which is available because the preceding step's plan +published it, runs the `Generate` node for the recorded decision, and selects +the output whose identity equals `child_identity`. Selection by recorded +identity is an ordinary property of the retained trajectory, not a new plan +mechanism: the next step's plan binds that exact artifact as a static input. +There is no `Promote`, no best-child rule, and no Artifact Store scan. + +How narrowly a step's node can be aimed depends on what its generator's +configuration exposes, and the two cases differ. A hardware rewrite generator +owns a decision-domain set whose member fixes one exact target selector plus a +finite value set or bounded range, so replay may bind the singleton domain +admitting exactly the recorded decision; the node then produces that one child +and publishes no siblings. The Dataflow rewrite generator exposes no such +narrowing. Its whole resolved configuration is one expansion limit, and it +enumerates the catalog over a frontier in canonical order, so the smallest node +that reaches the recorded decision also publishes every decision preceding it. +Those siblings are ordinary valid Artifacts that the plan does not select, and +accepting them is the cost of replaying a software step. Claiming a singleton +domain for that generator would be claiming a configuration field it does not +have. + +Replay is therefore one `Generate` node per recorded decision, including a +completion when the step carries one, with no special arithmetic per domain: a +`Reattach` completion is itself an enumerated decision of the domain its row +names, and a `ReplaceParent` completion records none because the generator's +own child is already the new collection member. Replay publishes real Artifacts +at every step, with the ordinary `CandidateDecision` lineage contributions +those descriptors own, which is exactly the cost the episode deferred. A replay +that selects any output other than the recorded identity is a different plan +that explores a neighborhood; it is not this trajectory. + +A `Trajectory` is a compact way to name a sequence of generator decisions so +that an authoritative run can reconstruct them; a candidate it names becomes +real only when a real `Generate` node publishes it and a real `Promote` node +acquires Evidence and applies quality gates. + +## Python Boundary + +The `loomml` package layering, the RLlib conformance target and its +obligations, the zero-copy array contract, the outcome-versus-exception rule, +and the threading and `fork` rules are owned by +[Python Boundary](spec-ml-core-environment.md#python-boundary). This +environment occupies `loomml.env.dse` and `loomml.rllib.dse` and adds no rule +of its own. + +Two of the core's obligations are worth naming against this environment's own +values, because they are what the obligations are for here. The decision nodes +and target arcs the parametric-action convention carries are the ones The Graph +Instance defines, and the `reset` seed the caller supplies replaces +`determinism_policy.master_seed` as the episode's effective seed and is +recorded in the `Trajectory`. + +## Benchmarking + +The harness obligations, the required breakdowns, the instrumentation rule, and +the ratio-based regression budget are owned by +[Benchmarking Harness Contract](spec-ml-core-environment.md#benchmarking-harness-contract). +This environment ships `loom-dse-env-bench`, whose rejection-reason breakdown +is keyed by `RejectionReason` and whose decision partition is +`ExplorationDomainKind`, because the domains do not cost the same. + +The harness decomposes one step into these stages and reports each separately: + +```text +enumeration build the canonical decision order and mask +draft_apply derive the child draft and apply one decision +accept the domain owner's acceptance path for the child +identity canonical bytes and root identity +completion the completion its domain's row names +techmap regenerate an invalidated TechMapping +probe per workload, closed Mapping or typed failure +inference project_features and infer per referenced contract +observation build the observation arrays +marshal expose buffers across the Python boundary +``` + +and one reset into `seed_load`, `cache_lookup`, `cold_mapping` per workload, +and `first_observation`. + +Two further breakdowns are required beyond the core's two. The probe is +reported per workload, together with the mean workloads probed and TechMappings +regenerated per rejected step, which is what shows whether the interleaving of +stage 8 is earning its place. And `warm_start_from_parent` is measured the same +way, with and without the parent's mappings, split by whether the decision +changed the Fabric or a workload. + +## Anchor Verification + +Stable tests cover adoption rejecting a reward closure whose Mapping-sourced +dimension the probe view does not admit, whose Evaluation-sourced dimension no +configured parameter contract satisfies, or which references a held-out +obligation; the set of parameter contracts an episode invokes being exactly the +set the reward closure's Evaluation-sourced dimensions reference, with no +configuration field beside it; adoption rejecting a duplicate domain binding, a +bound row whose required subject does not match the subject arm, and a +`Reattach` completion whose named domain is not itself bound; an omitted domain +contributing no enumerated entry, no mask bit, and no observation block; an +exploration domain whose decisions carry a reference the combined space cannot +represent being refused at adoption; a decision whose row operates on the +subject's own kind rejecting any selector but `EpisodeSubject`, and a decision +whose row operates on a component requiring a `SubjectComponent` or +`CollectionOrdinal` selector; a `ReplaceParent` completion leaving every other +collection ordinal unchanged; a state whose enumeration exceeds +`enumeration_bound` reporting `EnumerationBoundExceeded` rather than being +shortened; a revisited or over-capacity candidate being rejected without any +probe or TechMapping regeneration running; a schedule-preserving decision's +`GroupMember` references being those the state's Mapping routes through its +`Primary` target, the enumeration's length being unchanged by whether a Mapping +exists, a reconnection the domain owner refuses being an ordinary rejection, +and a replayed step reproducing the recorded references without reconstructing +a probe; a decision returning the design to +the seed state being rejected as `AlreadyVisited` on the episode's first +opportunity; a start failure under an `EpisodeStartOverride` being returned as +it stands rather than retried or converted to `RetryBudgetExhausted`; a +rejected step regenerating no +TechMapping for a workload the probe never reached; a rejection that reaches +the consecutive-rejection bound reporting both its `RejectionReason` and its +`TerminalReason`; an advance into an empty enumeration reporting both on the +same step; enabling a domain shifting only the enumeration suffix at and after +its ordinal; enumeration and mask agreement, including a cleared bit after a +rejection at that state; a draft, acceptance, or completion rejection producing +no child, no identity, and no lineage; a decision that removes a capability, +route, capacity, AccCore, or transport connection a workload requires being +rejected with the correct per-workload reason and leaving the parent +byte-identical; a rewrite whose child the current hardware cannot map being +rejected with the same per-workload reasons as a hardware decision; a workload +whose TechMapping cannot be regenerated reporting `TechMappingUnavailable` +rather than a mapping outcome; a probe budget too small to close a reachable +Mapping reporting `WorkloadMappingNotClosed` and never +`WorkloadProvenInfeasible`; a concurrent probe implementation reporting the +same failure as a sequential one; every state reachable within an episode +satisfying the feasibility invariant at the episode subject's closure level; a +`Reattach` completion publishing its component and leaving a root-complete +subject whose only changed relation is the targeted one; `reset` refusing a +seed whose root kind does not match the subject arm and refusing a seed that +cannot map every selected workload; a drawn seed with an empty enumeration +reporting `SeedEnumerationEmpty` and one whose enumeration exceeds +`enumeration_bound` reporting `SeedEnumerationBoundExceeded`, both retrying +rather than starting and neither being `Invalid`; a `Reattach` step publishing +its component before the visited and capacity tests while a `None` or +`ReplaceParent` step publishes nothing before them; a realization arc carrying +`Placement` and a route-traversal arc carrying `Route`; an +`EpisodeStartOverride` consulting neither selection stream and producing the +same episode on every copy, and one naming inventory outside the configured +sets being rejected; a per-workload cache entry surviving a change to another +workload in the draw, and a `System`-episode entry not being returned under a +different System PnR view; `ObjectiveUnavailable` never producing a numeric +reward; a truncation yielding zero reward while an elective stop yields the +configured stop code; equal configuration, coordinates, seed, and action +sequence reproducing an identical trajectory; and trajectory replay selecting +the output whose identity equals `child_identity` at every step and reproducing +identical finalized identities in order, including the node a `Reattach` +completion contributes, with a hardware step narrowed to a singleton domain +publishing no sibling and a software step publishing the catalog prefix it +must. + +Tests do not pin the live node, arc, or decision counts of any particular +state, the bound values a profile selects, which reconnection a particular +Mapping implies, wall-time numbers, per-domain cost ratios, probe-ordering +heuristics, corpus contents, diagnostic text, or Python formatting. diff --git a/docs/spec-ml-dse-model-architecture.md b/docs/spec-ml-dse-model-architecture.md new file mode 100644 index 000000000..027ebe2ed --- /dev/null +++ b/docs/spec-ml-dse-model-architecture.md @@ -0,0 +1,316 @@ +# ML DSE Model Architecture + +This document defines the learned policy that consumes the ML DSE Environment's +observations and emits its actions. It owns the embedding of the environment's +graph observation, the graph-transformer encoder over that embedding, the +hierarchical policy head that scores enumerated decisions, the value head, the +masking contract, and the parameter and checkpoint boundary. + +The model is a search policy on the terms +[ML Model Core](spec-ml-core-model.md) states, and every legality, mappability, +and objective fact remains owned where +[ML DSE Environment](spec-ml-dse-environment.md) places it. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [ML Model Core](spec-ml-core-model.md) owns the module boundary, observation + batching, the feature-embedding discipline, the encoder, the graph context, + the value head, masking, the action distribution, the checkpoint boundary, + and the obligations every forward-pass harness satisfies; +- [The Graph Instance](spec-ml-core-environment.md#the-graph-instance) owns the + observation container, and + [The Graph Instance](spec-ml-dse-environment.md#the-graph-instance) owns this + environment's node and arc feature catalogs, action mask, objective codes, + and scalar features; +- [Action Space](spec-ml-dse-environment.md#action-space) owns the enumerated + decisions, their canonical order, their targets, and `ActionIndex`; +- [Combined Node And Link Space](spec-ml-core-environment.md#combined-node-and-link-space) + owns the node roles, the arc-role obligation, and the target-closure rule + this document's policy head relies on; +- [Reward](spec-ml-dse-environment.md#reward) owns the reward the value head + regresses toward; +- [RLlib Environment Definition](spec-ml-core-environment.md#rllib-environment-definition) + owns the trainer-facing environment contract and the forked trainer this + model plugs into; +- [Evaluation and DSE](spec-dse-feedback.md#model-parameters-and-training) owns + `ModelParameterContractRef` and the registered prediction contracts, which + this model is not; and +- [ML Training Core](spec-ml-core-training.md) owns the algorithm binding and + the loss it brings, the rollout and learner topology, the stage sequence, and + every hyperparameter schedule, and + [ML DSE Training](spec-ml-dse-training.md) owns this environment's reachable + return range. + +This document owns only the DSE feature mapping, the hierarchical policy head, +the model configuration view, and its own benchmarking stages. It restates no +environment fact and defines no new observation, action, or reward. + +## Module Boundary And Batching + +The module boundary, the three forward entry points and their shared encoder +pass, `compute_values` accepting precomputed embeddings, and the disjoint-union +batching of ragged graph observations are owned by +[Module Boundary](spec-ml-core-model.md#module-boundary) and +[Observation Batching](spec-ml-core-model.md#observation-batching). This model +is `LoomDseRLModule` and adds no entry point and changes no signature. + +This environment uses the `DecisionNodes` enumeration encoding, so its +enumerated decisions are nodes of the observation graph and it supplies no +separate decision instance. `ActionIndex` is a per-instance decision-block +ordinal and is mapped through `ptr` and that block's offset. + +## Feature Embedding + +The embedded categorical columns are: + +```text +column table size source catalog +Role GraphNodeRole cardinality environment +EntityKind Fabric local-reference kinds + plus Dataflow node/value kinds Fabric Identity, Dataflow +DecisionDomain ExplorationDomainKind cardinality environment +DecisionKind sum over domains of that + domain's decision-union size candidate generators +DecisionValueForm value-form cardinality environment +DecisionValueOrdinal per-domain prototype cardinality candidate generators +DecisionDeltaSign sign cardinality environment +ArcRole DseArcRole cardinality environment +``` + +`DecisionKind` is embedded on the `(DecisionDomain, DecisionKind)` pair rather +than on the kind ordinal alone, because kind ordinals restart per domain and a +shared table would tie unrelated decisions together. `DecisionValueOrdinal` is +the direct analogue of the old architecture's per-instruction embedding table: +it is what lets the policy reason about a specific replacement prototype rather +than about an opaque index, and it is the axis along which the value factor of +the hierarchical head operates. + +Numeric projection, the absent-column rule, the signed logarithmic transform, +the capacity-usage-occupancy triple, the mapping of `objective_codes` into +`[0, 1]`, and the concatenate-then-project input path are owned by +[Feature Embedding](spec-ml-core-model.md#feature-embedding). Two of this +environment's columns need the treatment named rather than derived. + +`ResidualViolationCount` and `PlacedDegree` are projections of the probe result +rather than of the design, so they change when the same design is probed +differently; they are projected as ordinary magnitudes and carry no special +status, but a configuration comparing runs across a changed probe view is +comparing two different features under one name. + +`StepOrdinal` and `StepBound` are supplied as their ratio as well as their +magnitudes, so remaining budget is directly available; `ConsecutiveRejections` +is supplied as a magnitude, as is the `Decision` block's size. + +## Encoder And Context + +The `GraphTransformerStack` record, pre-normalization, the feed-forward +sublayer, edge-aware attention, jumping knowledge, the one-shared-trunk rule, +and the multi-scale per-instance graph context are owned by +[Encoder](spec-ml-core-model.md#encoder) and +[Graph Context](spec-ml-core-model.md#graph-context). + +This environment's `DseArcRole` catalog supplies the edge features. `Placement` +and `Route` are the probe's realization and traversal relations, which differ +in arity and in which decisions move them, and `DecisionTargetPrimary` against +`DecisionTargetMember` is what lets a decision node's attention distinguish the +entity it rewrites from the set it rewrites it against. + +`DecisionDeltaSign` is embedded rather than projected because a sign is +categorical, which is what the value term's delta form requires: a magnitude +carries the size of a change and the sign carries its direction, and projecting +the sign as a number would make a negative delta a small positive one. + +## Hierarchical Policy Head + +The environment's action is one `ActionIndex` plus a stop flag, and its +decision nodes carry both their target arcs and their value columns. The head +therefore scores decisions where they live, on the graph, and factors the score +into a part that depends on where the decision acts and a part that depends on +what value it selects. + +Every decision node `d` has an anchor and a value: + +```text +anchor(d) = ( primary target node of d, + pooled GroupMember target set of d, + DecisionDomain of d, + DecisionKind of d ) + +value(d) = ValuePrototype(DecisionValueOrdinal) if form is ReplacementPrototype + | ValueDelta(DecisionDeltaMagnitude, DecisionDeltaSign) + if form is BoundedIntegerDelta + | Absent if form is NoValue +``` + +The anchor includes the whole target set, not only the primary, so a decision +naming an entity plus a reference set is scored on the set it actually +rewrites. Decisions sharing an anchor form one anchor group; a group is a +singleton exactly when its decisions carry no value. + +`value(d)` reads whichever columns its form populates. A `BoundedIntegerDelta` +decision has no prototype ordinal, and its ordinal column is the absent +sentinel, so a value term reading only `DecisionValueOrdinal` would give every +delta on one target an identical score and make the magnitudes +indistinguishable. A delta is embedded from its magnitude and sign instead: the +sign is categorical, and the magnitude is projected, so adjacent magnitudes +score similarly and the head generalizes across a range rather than memorizing +each step of it. + +The head produces two scores per decision: + +```text +s_anchor(d) = MlpAnchor([ h_primary(d), h_members(d), e_kind(d), context ]) +s_value(d) = 0 if value(d) is Absent + = MlpValue([ h_primary(d), e_value(d), e_kind(d), context ]) + otherwise +logit(d) = s_anchor(d) + s_value(d) +``` + +`s_anchor` is constant within an anchor group by construction, because the +anchor includes every input `MlpAnchor` reads: the primary target, the pooled +member set, and the kind, evaluated against one shared context. This is what +makes the factorization a genuine hierarchy rather than a reparameterization: + +A boolean decision, one carrying no value, is a singleton group whose +conditional factor is one, so it is scored directly on its node or link exactly +as asked. A value-carrying decision is scored on its node or link by the anchor +factor and among its alternatives by the value factor, which is the operation +selection of the prior architecture generalized to every value-carrying +decision kind rather than to instructions alone. + +The factorization, the group-constancy rule, the flat-versus-two-level +equivalence, the broadcast saving, and the stop logit are owned by +[Policy Head Factorization](spec-ml-core-model.md#policy-head-factorization); +the joint distribution these logits feed and its stop encoding by +[Action Distribution](spec-ml-core-model.md#action-distribution). + +Slots beyond the instance's `Decision` block size have no decision node and +receive a masked logit; the stop outcome is masked only when the environment's +mask clears it. + +## Value Head, Masking, And Distribution + +The value head and its bound, the masking discipline and its five rules, and +the joint action distribution over `enumeration_bound + 1` outcomes are owned +by [Value Head](spec-ml-core-model.md#value-head), +[Masking](spec-ml-core-model.md#masking), and +[Action Distribution](spec-ml-core-model.md#action-distribution). The +checkpoint boundary is owned by +[Parameters And Checkpoints](spec-ml-core-model.md#parameters-and-checkpoints). + +One property of this environment is worth naming against the masking contract. +The environment's own mask already carries a per-state rejection record, so a +bit it clears may encode that a decision was tried at this state and failed. +That is a fact about the episode rather than about the decision, and it is the +reason the recorded mask must travel in the batch: recomputing it at update +time against a different episode position would mask a decision that was +available when it was sampled. + +## Model Configuration + +The model's shape is one immutable component view with schema descriptor bytes +`loom.ml_dse_model.config.1.0`, following the framing owned by +[Component Views](spec-config-ssot.md#component-views) and the three properties +[Model Configuration Framing](spec-ml-core-model.md#model-configuration-framing) +fixes: + +```text +ResolvedMlDseModelConfigView { + embedding_widths: total table + encoder: GraphTransformerStack + head_widths: DseHeadWidths + value_bound: positive uint64 + advisory_mask_set: canonical set + initialization: InitializationConstants +} +``` + +```text +DseHeadWidths { + anchor_hidden: positive uint32 + value_head_hidden: positive uint32 + value_hidden: positive uint32 +} +``` + +`value_bound` is the symmetric range the value head clamps to, in the scaled +reward's units. `advisory_mask_set` names the optional masks the masking +contract admits. + +## Inference Benchmarking + +The harness obligations, the two regimes, the entry-point split, the scaling +rule, the rollout split, invariance against an unoptimized reference, and +ratio-based budgets are owned by +[Forward Benchmarking Obligations](spec-ml-core-model.md#forward-benchmarking-obligations). +This model ships `loom-dse-model-bench`. + +The harness decomposes one forward pass into these stages: + +```text +gather assemble the batched graph and move it to the device +embed embedding lookups, numeric projection, input projection +encode the graph-transformer stack +pool per-instance multi-scale context +policy anchor and value scoring over decision nodes +mask mask composition and additive application +distribute distribution construction and sampling +value the value head, when the regime computes it +``` + +`encode` is the expected dominant term, since attention cost scales with arc +count, and `gather` is the expected dominant term for a small graph on an +accelerator, where the transfer costs more than the arithmetic. Both +expectations are worth measuring rather than assuming. + +Each stage is reported against node count, arc count, and live decision count, +with the fitted growth of `encode` against arc count and of `policy` against +live decision count. + +Three optimizations specific to this head are measured against an unoptimized +baseline, and each carries a failure mode a naive measurement will miss. + +Anchor-group evaluation, on the terms +[Policy Head Factorization](spec-ml-core-model.md#policy-head-factorization) +states. The mean group size is reported alongside the `policy` stage. + +Live decisions only. The head emits logits over `enumeration_bound` slots, but +only the live ones have decision nodes. Scoring runs over live decisions and +scatters into the emitted logit vector; running the scoring +multilayer perceptrons over dead slots wastes work proportional to the gap +between capacity and live count, which is largest exactly when the capacity is +set conservatively. + +Batched inference across copies. Vectorized environment copies on one runner +present several instances at once, and batching them into one union graph +amortizes the per-call overhead that dominates latency-bound inference. The +harness reports latency against the copy count to show where that amortization +saturates. + +Device placement is measured across the size range and the crossover is +reported rather than assumed: for small graphs, host inference can beat +accelerator inference because the transfer in `gather` exceeds the arithmetic +it feeds. + +## Conformance Anchors + +Stable tests cover an `ActionIndex` resolving to its own instance's decision +block; a decision carrying a reference set contributing every member to its +anchor summary rather than only its primary; a value-free decision receiving a +zero value term and forming a singleton anchor group; two decisions sharing an +anchor receiving an identical anchor score; two `BoundedIntegerDelta` decisions +on one target with different magnitudes receiving different scores; the +two-level and flat formulations producing identical distributions to numerical +tolerance; a stop logit competing in the same normalization as the decision +logits; a per-resource capacity and usage pair reducing to one resource +embedding rather than widening the node vector with the resource catalog; +scoring restricted to live decisions producing logits identical to scoring +every slot; the mean anchor-group size being reported alongside the `policy` +stage; and every optimized configuration reproducing the unoptimized +reference's logits and values within the declared tolerance. + +Tests do not pin layer counts, widths, head counts, initialization constants, +the advisory mask set, learned parameter values, wall-time numbers, device +crossover points, or throughput. diff --git a/docs/spec-ml-dse-training.md b/docs/spec-ml-dse-training.md new file mode 100644 index 000000000..951c40619 --- /dev/null +++ b/docs/spec-ml-dse-training.md @@ -0,0 +1,597 @@ +# ML DSE Training + +This document defines how the search policy of +[ML DSE Model Architecture](spec-ml-dse-model-architecture.md) is fitted +against the environment of [ML DSE Environment](spec-ml-dse-environment.md). It +owns the split of a run's settings into separate resolved configurations, the +binding to an existing PPO implementation, the per-episode statistics a run +reports, the fixed test set and the deterministic protocol that evaluates it, +and the offline corpus of training architectures and synthetic training +workloads. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [ML DSE Environment](spec-ml-dse-environment.md) owns the episode protocol, + the action space, the observation, the reward's exact `(sign, magnitude)` + pair, the step and rejection outcome algebra, the terminal reasons, the + `Trajectory`, the Python layering, and `ResolvedMlDseEnvironmentConfigView`; +- [ML DSE Model Architecture](spec-ml-dse-model-architecture.md) owns the + module boundary, the encoder, the heads, the action distribution, the mask + composition and its fallback, and the checkpoint's parameter and + configuration match; +- [ML Training Core](spec-ml-core-training.md) owns the nonsemantic boundary, + the configuration split and inventory identity, the algorithm binding + surface, the stage sequence and its handoff, exact-rational hyperparameters, + schedules, the reward adapter, topology, the stage invariance rule, the + statistics production rule, the test protocol, checkpoints and run identity, + and reproduction; +- [Objectives and Quality Gates](spec-dse-feedback.md#objectives-and-quality-gates) + owns `ObjectiveDimension`, `ExactAffineQuantization`, directed codes, + `ObjectiveVector`, `WeightedLevel`, and `SearchEnergyRef`; +- [Model Parameters and Training](spec-dse-feedback.md#model-parameters-and-training) + owns the training of registered prediction contracts, which is a different + activity from this one and shares no record with it; +- [Candidate Generators](spec-dse-feedback.md#candidate-generators) owns the + `fabric_template` generator this document's architecture corpus is produced + by; +- [Evaluation Metrics](spec-evaluation-metrics.md#metric-registry) owns every + `MetricKind` and `ExactRatio`, and + [FPA Evaluation](spec-fpa-estimation.md#metrics) owns which physical metrics + are registered and what may produce them; +- [Component Views](spec-config-ssot.md#component-views) owns view framing, + canonical bytes, and `component_view_digest`, and + [Cache Dependencies](spec-config-ssot.md#cache-dependencies) owns the cache + family the corpus prepopulates; +- [Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism) + owns the seeded PRNG protocol; +- [Source Integration](spec-compiler-part-1-source.md) and the compiler + pipeline own how any program, synthetic or real, becomes a Canonical Dataflow + Program; +- [Corpus](spec-loom-stack.md#corpus), [LoomBench](spec-loombench.md), and + [Real Application Portfolio](spec-application-portfolio.md) own the real + program inventories, which this document's synthetic corpus is never part of; + and +- [External Dependency Pinning](spec-loom-stack.md#external-dependency-pinning) + owns the Ray fork revision and its patch stack. + +This document owns only this environment's return-range arithmetic and its +discounting consequence, the DSE statistics catalog and breakdown axes, the +test case payload and the granularity its isolation is compared at, the corpus +view and its offline generator, and its harnesses. + +## Nonsemantic Boundary + +Under +[Nonsemantic Boundary](spec-ml-core-training.md#nonsemantic-boundary), the +owner-owned path by which a design this run discovers becomes a real candidate +is the one +[Trajectory Retention And Replay](spec-ml-dse-environment.md#trajectory-retention-and-replay) +defines. + +The corpus this document generates is the one place training touches durable +state, and it touches it only through existing owners. A training architecture +is an ordinary finalized Fabric Artifact and a training workload is an ordinary +Canonical Dataflow Program Artifact, each produced by the owner that already +produces them. The corpus view names those roots; it does not define a second +Artifact kind, a second program authority, or a store of its own. + +## Configuration Split + +This run's views, under the framing +[Configuration Split](spec-ml-core-training.md#configuration-split) owns: + +```text +loom.ml_dse_environment.config.1.0 ResolvedMlDseEnvironmentConfigView +loom.ml_dse_model.config.1.0 ResolvedMlDseModelConfigView +loom.ml_dse_training.config.1.0 ResolvedMlDseTrainingConfigView +loom.ml_dse_corpus.config.1.0 ResolvedMlDseCorpusConfigView +loom.ml_dse_test_set.config.1.0 ResolvedMlDseTestSetConfigView +loom.ml_dse_run.config.1.0 ResolvedMlDseRunConfigView +``` + +```text +ResolvedMlDseRunConfigView { + training_config_digest: ComponentViewDigest + model_config_digest: ComponentViewDigest + corpus_config_digest: ComponentViewDigest + test_set_config_digest: ComponentViewDigest +} +``` + +`ResolvedMlDseTrainingConfigView` is the core's `ResolvedTrainingConfigView` +under this descriptor, with no field added. Every stage binds `Ppo`; a run that +wants a curriculum spends more than one stage on it, and one that does not is +the single-stage case. + +The cached asset the core's digest-independence rule names is, here, the +Mapping cache family; it takes a few dozen episodes to become useful. + +Adoption validates each view independently, then validates the run view's +cross-view conditions: every `seed_roots` and `workload_pool` member of every +bound training environment view appears in the corpus view; the test set's own +environment view and its cases are admitted under +[Test Protocol](#test-protocol) and are disjoint from the corpus under +[Isolation](#isolation); the +model view's embedding table sizes match the owner catalog cardinalities the +environment view's bound domains reach; and the model view's value bound covers +the reachable return range below, maximized over every bound environment view. + +## Training Configuration + +The training view, the algorithm bindings, the stage sequence and its handoff, +exact-rational hyperparameters, schedules, and the reward adapter's single +scale are owned by +[Training Configuration](spec-ml-core-training.md#training-configuration). This +environment supplies two things of its own. + +### The Reachable Return Range + +This environment's declaration under the core's obligation. A transition's +reward magnitude is bounded by the quantization width of the search energy's +dimensions, and every step contributes one term: an advancing step at most that +width, a rejected step `rejection_reward_code`, and the one step that may be an +elective stop `stop_reward_code`. An episode carries at most `step_bound` +steps, so the undiscounted return is bounded by + +```text +step_bound * max(energy_width, rejection_reward_code) + stop_reward_code +``` + +all times the adapter's scale. The core's every-step rule is what makes the +rejection code multiply `step_bound` here rather than appear once, and it binds +because a policy early in training rejects far more often than it advances. + +### Path Dependence And Discounting + +Discounting at one has an exact meaning here worth stating. When +`warm_start_from_parent` is clear, the environment's energy is +path-independent, so an undiscounted return telescopes to the total improvement +from the seed to the final design and every intermediate reward cancels. A run +that wants that property sets both, and a run that discounts is deliberately +preferring earlier improvement to later. With warm starting set the telescoping +does not hold, because the energy a state reports depends on how it was +reached, and this document states that rather than letting a configuration +assume otherwise. + +## Algorithm Binding And Topology + +The algorithm binding surface, what Loom supplies to the trainer, the +instrumentation-only rule for a `Learner` subclass, rollout and learner +topology, and the stage sequence with its invariance rule are owned by +[ML Training Core](spec-ml-core-training.md); the fields a stage may vary are +declared by +[Curriculum-Neutral Fields](spec-ml-dse-environment.md#curriculum-neutral-fields). + +One of those lands on this environment's own values. Its step is dominated by +per-workload Mapping probes, as +[Benchmarking](spec-ml-dse-environment.md#benchmarking) reports, so sampling is +normally the binding cost and the topology is chosen by moving runner count +against that split. + +## Episode Statistics + +The production rule, `DimensionEpisodeStatistic`, the outcome accounting, the +cost statistics, `LoggingPolicy`, and chunk invariance are owned by +[Episode Statistics](spec-ml-core-training.md#episode-statistics). This +environment names its own vocabulary and adds what only it has. + +Its non-advancing step class is `rejected`, so the core's accounting identity +reads `steps == advanced + rejected + elective_stops`, and the rejection rate +is reported per member of `RejectionReason` and per `ExplorationDomainKind`. A +single aggregate rate conflates causes that call for opposite responses: +`AlreadyVisited` is search behavior, `WorkloadMappingNotClosed` is a +probe-budget problem, `WorkloadProvenInfeasible` is a real proof about the +design, and `EnumerationBoundExceeded` is a configuration error. Its action +partition is the `(ExplorationDomainKind, decision kind)` pair, because a +decision kind ordinal restarts per domain and a shared key would tie unrelated +decisions together. + +Two statistics are this environment's own. `visited_revisit_rate` is the share +of rejections that were `AlreadyVisited`, which is what says whether the policy +is circling. And a dimension of the selected closure that sits outside the +search-energy level appears in these statistics and contributes nothing to the +reward, which is how a run watches a metric it does not optimize; a dimension +absent from the closure produces an absent statistic rather than a zero. + +```text +DseBreakdownAxis = + Stage // 0, mandatory + | TerminalReason // 1, mandatory + | SeedRoot // 2, per architecture + | Workload // 3, per Canonical Dataflow Program + | Domain // 4, per ExplorationDomainKind + | RejectionReason // 5 +``` + +`SeedRoot` and `Workload` are the expensive members: a corpus with a few +hundred architectures and a few thousand workloads turns every statistic into +hundreds of thousands of series, which is why axes are opt-in. + +### Occupancy Statistics + +Utilization is reported when the environment's `include_resource_states` is +set, and is absent otherwise. Per episode, at entry and at the final state: + +```text +resource_occupancy[FabricResourceStateRef] usage / capacity, as ExactRatio +placed_occurrence_fraction nodes with PlacedDegree > 0 +mean_residual_violation_count +``` + +Occupancy is the ratio of the usage and capacity columns the observation +already carries per resource state, aggregated over the episode's Fabric nodes. +It is the quantity a hardware decision actually trades — headroom — and it is +the direct check on whether the search is removing resources the workload was +not using or resources it was. A run whose area improves while occupancy stays +flat is removing slack; one whose area improves while occupancy climbs toward +one is approaching the feasibility gate, and its rejection rate will confirm +it. + +`placed_occurrence_fraction` is the Mapping's own view of the same question, +from the probe result's `PlacedDegree`, and it is reported alongside because +the two disagree in an informative way: a fabric can be resource-saturated +while most occurrences go unplaced, and that is a routing problem rather than a +capacity one. + +## Test Protocol + +The test-set container, its determinism, its execution and aggregation, and +what a test score is not are owned by +[Test Protocol](spec-ml-core-training.md#test-protocol). This environment +supplies its case payload and its isolation rule. + +```text +DseTestCase { + instance: EpisodeStartOverride + case_seed: u64 +} +``` + +The instance is the environment's own override payload rather than a re-spelled +copy of its fields, so a field added there reaches a test case without an edit +here. + +### Isolation + +The disjointness the core requires is compared here by exact Artifact root +identity: adoption rejects a run whose test-set seed roots intersect the +corpus's architectures, or whose test-set Canonical Dataflow roots intersect +the corpus's Canonical Dataflow roots. + +The workload comparison is on the Dataflow root alone, not on the +`WorkloadBinding` pair. A binding is a Dataflow root together with a +TechMapping, and the same program bound to a different TechMapping is a +different pair but the same program; comparing pairs would let a training +workload reappear in the test set under a regenerated TechMapping and pass as +disjoint, which is precisely the leakage this rule exists to prevent. The +TechMapping is a derived artifact of the architecture it was built against, so +it carries no independent identity worth isolating. + +Root identity is the right granularity for the rest because it is what the +environment consumes. A test workload generated from the same synthetic +parameters as a training workload but with a different seed is a different +program and a different root, and is admissible; the same program under a +different file name is the same root, and is not. Checking names instead of +roots would admit exactly the duplicate the rule exists to exclude. + +A test set may also draw its cases from the real corpora — LoomBench cases or +portfolio operators — and this is the more informative choice, because the +synthetic corpus is calibrated to resemble them and a test on real programs +measures whether that calibration transferred. Those corpora are owned by +[LoomBench](spec-loombench.md) and +[Real Application Portfolio](spec-application-portfolio.md); the test set +references their Artifacts and defines no membership rule of its own. + +## Training Corpus + +### Two Halves, One Pairing + +```text +ResolvedMlDseCorpusConfigView { + architectures: canonical nonempty set + workloads: canonical nonempty set + pairings: canonical nonempty set + generation: CorpusGenerationRecord + calibration: CalibrationRecord +} + +FeasibilityPairing { + architecture: ArtifactRootReference + workload: WorkloadBinding +} +``` + +The corpus is admitted as a bipartite pairing rather than as two independent +lists. A pairing asserts that the offline generator obtained a closed Mapping +for that workload on that architecture at the episode subject's closure level, +under the probe view the run will use. Adoption requires every architecture to +appear in at least one pairing and every workload to appear in at least one. + +Without that requirement a corpus can be individually valid and jointly +useless. The environment's `reset` is a precondition, not a best effort: a seed +that cannot map every selected workload is retried and, past +`start_retry_bound`, reports `RetryBudgetExhausted`. A corpus whose +architectures and workloads were generated independently produces exactly that +failure at a rate nobody predicted, and it produces it after paying cold PnR +for each attempt. Establishing the pairing offline, once, converts a per-reset +gamble into an adoption check. + +A pairing is not a promise that every episode start succeeds. The environment +selects several workloads per episode and a pairing is per workload, so a draw +can still fail on a combination. The pairing bounds that failure to +combinations rather than admitting workloads no architecture can run at all. + +### Architecture Corpus + +An architecture is an ordinary finalized `fabric.module` or `fabric.system` +Artifact, matching the episode subject. It is produced by the `fabric_template` +generator of kind 12 through the ordinary ADG Builder and finalization path, +and it is published. The environment's episode start already admits a +`fabric_template` output as a seed, so this introduces nothing. + +Kind 12 is where the architecture corpus lives precisely because the +environment excludes it as an exploration domain: a template expansion selects +an episode seed rather than advancing an episode. The same generator therefore +serves both roles without either one reaching into the other. + +The corpus declares its coverage over the axes the template generator +parameterizes — array extent, processing-element composition and operation mix, +memory capacity and count, boundary inventory width, switch topology and +connectivity, FIFO depth, and for a `System` subject the core count and +transport topology. Coverage is reported per axis bucket rather than as a +total, because a corpus of a thousand architectures that all differ in FIFO +depth teaches a policy one axis. + +The extent range is not an aesthetic choice. The encoder is a graph transformer +whose cost scales with arc count and whose inference latency +[Forward Benchmarking Obligations](spec-ml-core-model.md#forward-benchmarking-obligations) +reports against node count, and the curriculum is expected to move designs +across an order of magnitude in node count. A corpus spanning one decade of +node count is what makes that curriculum expressible; one spanning a factor of +two makes a stage sequence meaningless. + +### Synthetic Workload Corpus + +A training workload is a synthetic program. Its operations compute nothing +meaningful: it has no reference output, no oracle, and no correctness claim, +and it exists to have the structure of a program rather than the behavior of +one. + +A synthetic program is nonetheless an ordinary program. The generator emits +ordinary compiler input, and the ordinary frontend compiles it into a Canonical +Dataflow Program Artifact with its TechMapping, through exactly the path +[Source Integration](spec-compiler-part-1-source.md) and the compiler pipeline +define. There is no synthetic-only path into the environment, no hand-authored +Dataflow, and no generator-owned Artifact kind. A synthetic program the +compiler rejects is a generator defect rather than a special case to +accommodate — and the compiler rejecting it is the cheapest possible discovery +that the generator is producing something no real program could be. + +The corresponding exclusion is exact. A synthetic workload never enters +`SourceTranslationUnitInventory` or `OperatorWorkloadInventory`, never becomes +a LoomBench case, and never joins the application portfolio. Those inventories +require a reference oracle and a semantic claim, which a program of meaningless +operations cannot supply. A synthetic workload is admissible for training and +for nothing else; it may not produce Evidence, satisfy a conformance gate, or +appear in a correctness result. + +### Structural Axes And Calibration + +The corpus is meant to resemble programs found in the wild, and what it +resembles is their structure, not their computation. The generator is +parameterized over a closed set of structural axes: + +```text +StructuralAxis = + OperationMix // distribution over the target's operation catalog + | GraphDepth // longest dependence chain + | GraphWidth // available parallelism at a level + | FanoutDistribution // out-degree of a produced value + | ReconvergenceRate // values consumed by more than one path + | RecurrenceDistance // loop-carried dependence distance + | LoopNestDepth + | TripCountMagnitude + | StreamCount // concurrent memory streams + | StrideRegularity // affine versus irregular access + | ReuseDistance + | PredicationRate + | DataDependentBranchRate + | VectorWidthDistribution + | LiveValuePressure // concurrent live values at a program point +``` + +Each axis has a declared target distribution, and a generated program is a draw +from their joint parameterization. The axes are the ones that change what a +Mapping and a fabric decision must contend with: depth against width sets what +parallelism a fabric can exploit, recurrence distance sets what a temporal +element must hold, stream count and stride regularity set what the memory +frontier must supply, and predication rate sets how much control the fabric +must absorb. An axis that does not change what the search decides does not +belong here, however faithfully it describes a real program. + +Calibration is a comparison, not a claim: + +```text +CalibrationRecord { + reference_corpus: canonical nonempty set + per_axis_tolerance: total table +} +``` + +The reference distribution is measured from the real corpora — the LoomBench +cases and portfolio operators the reference set names — by the same axis +extractor that measures a generated program, and the corpus is admitted when +every axis matches within its declared tolerance. The reference is derived from +the Artifacts `reference_corpus` names rather than written down, so it moves +when the real corpus moves and cannot become a stale table describing programs +nobody runs anymore. Deriving it is not the same as re-measuring it: a root is +immutable and content-addressed, so one root's axis vector may be retained +against that root indefinitely, and what each adoption recomputes is the +aggregate over whichever roots the set now names. + +A corpus that has never been compared is not calibrated, and this is worth +stating because the failure is invisible. A generator with plausible parameters +produces programs that look reasonable to a reader and can still be +systematically shallower, narrower, or more regular than anything real, and a +policy trained on them learns a design space that does not exist. The +comparison is what converts "we chose sensible parameters" into a checkable +statement. + +Matching every axis marginally is not matching the joint distribution, and this +document does not claim otherwise. Marginal agreement per axis is the admission +test because it is checkable and because it catches the large errors; a +generated corpus remains a model of real programs, and the test set drawn from +real programs is what measures whether the model transferred. + +### Offline Generation + +Corpus generation is a batch job that runs before training and produces three +things: the published architecture and workload Artifacts, the pairings, and a +prepopulated Mapping cache. + +```text +CorpusGenerationRecord { + generator_seed: u64 + prng_protocol: Sha256SeededXoshiro256StarStar_1_0 + architecture_axis_targets + workload_axis_targets + target_counts_per_bucket + probe_binding: CorpusProbeBinding +} + +CorpusProbeBinding { + episode_subject: + SpatialModule + | System { system_pnr_config_view_digest: ComponentViewDigest } + spatial_pnr_config_view_digest: ComponentViewDigest + tech_mapping_config_view_digest: ComponentViewDigest +} +``` + +The unit of work is one `(architecture, workload)` pair, which is also the unit +of the Mapping cache. The job establishes a pairing by obtaining a closed +Mapping, and that Mapping is written into the ordinary cache family +[Cache Dependencies](spec-config-ssot.md#cache-dependencies) owns, under the +key the environment's episode start already uses: the seed root, that one +workload with its TechMapping, and the probe view. No second store, no second +key, and no corpus-local mapping format. + +This is where the cost of training moves offline. The environment's own +analysis is that step 5 of `reset` dominates start cost and that per-workload +keying is what lets the cache reach a useful hit rate; running that step for +every pairing ahead of time means a training run's resets hit a warm cache from +the first episode rather than after a few dozen. It also means the pairing and +the cache entry are produced by the same work, so a corpus cannot claim a +pairing it never computed. + +The job is parallel over pairs, resumable, and content-addressed. A pair whose +Mapping is already cached and whose pairing is already recorded is skipped, so +an interrupted run continues rather than restarting, and two jobs over the same +corpus converge on the same result. Generation order is by bucket so that an +interrupted job leaves coverage spread across the axis buckets rather than +complete on the first bucket and empty on the rest. + +A pair whose Mapping does not close is recorded as a non-pairing and is not +retried indefinitely; a workload that closes on no architecture in the corpus +is reported and excluded, which is the signal that the workload generator has +drifted outside what the architecture generator produces. + +`CorpusProbeBinding` binds the cache entries to the complete set of views that +produced them, because that is what the cache key is. The environment's key +covers the seed root, the workload with its TechMapping, and *every* probe view +the episode ran under: the Spatial PnR view and the TechMapping view always, +and the System PnR view its subject arm carries for a `System` episode. A +record naming only the Spatial view could not distinguish two corpora generated +under different System contracts, so a `System` run would take a hit on an +entry a miss would not have produced — which is exactly the case per-view +keying exists to prevent. + +Adoption compares the complete binding against every bound environment view's +`probe_policy` and subject arm, and reports a mismatch rather than letting a +run silently pay cold PnR for every reset while a full cache sits unused. + +### Corpus Identity + +The inventory this run carries is the corpus, and its identity is the exact +`ArtifactRootReference`, on the terms +[Inventory Identity](spec-ml-core-training.md#inventory-identity) sets. An +enumerated corpus is therefore a value: extending it produces a new digest, and +every run that used the old one remains exactly reproducible. + +What discharges the regeneration half here is that the same `generator_seed`, +axis targets, and compiler configuration produce byte-identical Artifacts and +therefore identical roots, so a corpus can be reconstructed rather than +archived. The seeded PRNG protocol and its prohibition on host entropy are the +PnR owner's; this generator adds only its own domain separator and stream +purposes. + +## Checkpoints, Reproduction, And Harnesses + +Checkpoints and run identity, the two reproduction claims and the one this +document declines, and the obligations every harness satisfies are owned by +[ML Training Core](spec-ml-core-training.md). This run ships: + +```text +loom-dse-corpus generate, extend, verify, and calibrate a corpus view +loom-dse-train run training against a run view +loom-dse-test run a test set against a checkpoint +``` + +`loom-dse-corpus` verifies an existing corpus without regenerating it: every +named root resolves, every pairing's Mapping is cached under the recorded probe +binding, coverage per bucket meets the declared targets, and every calibration +axis is within tolerance. + +## Conformance Anchors + +Stable tests cover the run view rejecting a corpus that omits a bound training +environment view's seed root or pool workload, a test-set environment view +whose inventory intersects the corpus by root identity, a test case naming +inventory outside its own environment view, a test-set environment view +differing from the training views in any field +[Curriculum-Neutral Fields](spec-ml-dse-environment.md#curriculum-neutral-fields) +declares shape-fixing, including a reward code inside `EpisodePolicy`, a model +view +whose embedding table sizes do not match the reachable catalog cardinalities, +and a `value_bound` below the return range computed from the declared +quantization bounds, `step_bound`, `rejection_reward_code`, and +`stop_reward_code`; a run whose test set is admissible being adopted at all, so +that the corpus, isolation, and override rules are jointly satisfiable rather +than only individually stated; a `value_bound` sized for one rejection being +rejected against an episode of `step_bound` rejections; a hyperparameter edit +changing the training digest and leaving every cached Mapping valid; a stage +advance differing only in fields +[Curriculum-Neutral Fields](spec-ml-dse-environment.md#curriculum-neutral-fields) +declares neutral being accepted and one differing in `enabled_domains`, +`selected_objective_closure`, or `enumeration_bound` being rejected; a stage +advance retaining the Mapping cache entries earlier stages produced; +`improvement` being positive for a `Minimize` and for a `Maximize` dimension +that each improved; a closure dimension outside the +search-energy level appearing in the statistics and contributing nothing to the +reward; a dimension absent from the closure producing an absent statistic +rather than zero; a step rejected with `ObjectiveUnavailable` producing no +numeric estimate; rejection rates being reported per `RejectionReason` and per +`ExplorationDomainKind`; action frequencies being keyed by +`(domain, decision kind)` pairs and no statistic key being built from a file +name; occupancy statistics being absent when `include_resource_states` is +clear; corpus adoption rejecting an architecture or workload with no pairing; a +corpus pairing being backed by a cached Mapping under the recorded probe +binding; a training run whose Spatial PnR, TechMapping, or System PnR view +differs from the corpus's reporting the mismatch at adoption, and a `System` +-subject run being refused a corpus whose binding carries no System PnR digest; +a synthetic workload reaching the environment only as a compiled Canonical +Dataflow Program Artifact and being refused admission to every real-program +inventory; a corpus regenerated from the same seed and axis targets producing +identical Artifact roots; a corpus resolved by enumeration rather than by +directory contents, so that adding a file changes no existing run's data; a +calibration comparison failing when a generated axis distribution exceeds its +declared tolerance against the measured reference; and an interrupted corpus +job resuming without recomputing a cached pairing. + +Tests do not pin hyperparameter values, schedule breakpoints, layer or width +choices, topology counts, corpus size, axis target distributions, tolerance +values, the contents of any particular corpus or test set, wall-time numbers, +throughput, learning curves, achieved test scores, sink formats, or diagnostic +text. diff --git a/docs/spec-ml-pnr-environment.md b/docs/spec-ml-pnr-environment.md new file mode 100644 index 000000000..e83ac5479 --- /dev/null +++ b/docs/spec-ml-pnr-environment.md @@ -0,0 +1,1211 @@ +# ML PnR Environment + +This document defines the interaction boundary through which a learned search +policy places and routes one Mapping problem. An episode holds one exact +Spatial PnR invocation — a Canonical Dataflow Program, its TechMapping, a fully +elaborated Fabric, a resolved PnR configuration, and a MappingConstraintSet — +and presents its search as a sequence of typed Place and Route Actions. The +policy binds each realization to a hardware occurrence until every realization +is placed, and then either hands the result to a distance-bounded annealing +cleanup or keeps repairing it itself. + +The environment is a selector, not a placer. Every Action it applies is a +member of the closed `SpatialMappingAction` algebra +[Place And Route](spec-pnr.md#actions-and-movetransaction) already owns, every +transition goes through that owner's `MoveTransaction`, and every score is a +projection of that invocation's own objective closure. What the environment +replaces is the deterministic proposal selector, and nothing else. + +The relationship to [ML DSE Environment](spec-ml-dse-environment.md) is +symmetric and the difference is worth stating at the top, because it inverts +one rule. There, a closed Mapping is a gate: a design that cannot run the +workload is outside the space, and a step that reaches one is rejected. Here, a +closed Mapping is the goal: residual Mapping violations are ordinary candidate +state that the objective prices, and an episode spends most of its life in +states that are not yet closed. The two documents share their observation +container, action surface, and Python boundary through +[ML Environment Core](spec-ml-core-environment.md); they share no episode rule. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [Spatial PnR](spec-pnr.md#spatial-pnr) owns the exact five-input invocation + `(D, T, F, C, K)`, its aggregate freeze entry, and its exact binding + requirements; +- [Native State](spec-pnr.md#native-state) owns `FrozenModel`, + `CandidateState`, `SearchScratch`, the factorized candidate domains, and the + freeze cache key; +- [Actions And MoveTransaction](spec-pnr.md#actions-and-movetransaction) owns + the closed `SpatialMappingAction` algebra, the dynamic domain `A(M,C,S)`, the + transition `Apply(M,C,S,a)`, the dependency closure a binding change carries, + and the sole mutation mechanism; +- [Deterministic Initialization And Action Proposal](spec-pnr.md#deterministic-initialization-and-action-proposal) + owns the canonical typed decision-key order, the canonical choice orders, the + transport-routing scopes, and the frozen-topology hop distance; +- [Annealing And Replay](spec-pnr.md#annealing-and-replay) owns the annealing + policy, `realization_move_radius`, the acceptance protocol, and the seeded + PRNG protocol; +- [Evaluation Transaction](spec-pnr.md#evaluation-transaction) owns the online + probe protocol and the ephemeral Evaluation adapter; +- [Objective Projection](spec-pnr.md#objective-projection) owns the Mapping + violation catalog `V` and measure catalog `G`, and + [Final Closure And Verification](spec-pnr.md#final-closure-and-verification) + owns what makes a candidate closed; +- [Resolved View](spec-pnr.md#resolved-view) owns `ResolvedPnrConfigView` and + `SelectedObjectiveClosure`; +- [Spatial MappingConstraintSet Contract](spec-pnr.md#spatial-mappingconstraintset-contract) + owns `K` and its projection catalog; +- [Evaluation and DSE](spec-dse-feedback.md#objectives-and-quality-gates) owns + the objective algebra, quantization, and the statement that a reward is the + signed difference of a selected search energy; +- [TechMapping Generation](spec-tech-mapping.md) owns `T` and its exact `D` and + `F` binding; +- [Fabric Identity](spec-fabric-identity.md#owner-local-reference-kind-catalog) + owns the local-reference kind catalog the observation's `EntityKind` column + carries; +- [ML Environment Core](spec-ml-core-environment.md) owns the observation + container and its combined node-and-link space, the action surface and + masking algebra, the step accounting identity, the + termination-versus-truncation rule, the reward boundary, the PRNG preimage + shape, the `loomml` package layering and its interaction contract, and the + benchmarking obligations every harness satisfies; and +- [Resolved Configuration](spec-config-ssot.md#component-views) owns + component-view framing, canonical view bytes, and `component_view_digest`, + and at [Cache Dependencies](spec-config-ssot.md#cache-dependencies) owns the + cache-family contract. + +This document owns only the episode arms, its resolved configuration view and +its curriculum-neutral partition, the frozen-model and seed-state retention +policy, the construction, repair, and cleanup protocols, the PnR action index +contract, the PnR observation catalogs, the step and failure outcome algebra, +the trajectory retention record, and its own benchmarking stages. + +## Nonsemantic Boundary + +The environment is a nonsemantic search harness, on the terms +[Nonsemantic Boundary](spec-ml-core-environment.md#nonsemantic-boundary) states +for every ML environment. It is not a placer, a router, a resource allocator, +an objective, or a Mapping authority. It owns no occupancy model, no route +cost, no distance heuristic of its own, and no second candidate representation. + +It publishes no `SpatialMapping`. An episode's final candidate is mutable +`CandidateState`, which +[Canonical Search Sequence](spec-pnr.md#canonical-search-sequence) already says +never enters a central candidate set. Nothing in an episode runs independent +final verification, assigns a Mapping identity, or emits a lineage edge. + +Three consequences follow. + +Legality is never the environment's answer. A proposed Action either produces a +committed transition or is rolled back by `MoveTransaction`, and the +environment reports which. It does not pre-screen a choice the owner would +reject, does not retry a rolled-back Action under a different anchor, and does +not repair a candidate the owner refused to change. + +Cost is never the environment's answer either. The objective the reward +differences is the one `pnr_config_view.selected_objective_closure` already +declares, evaluated by the ordinary online protocol. A policy that wants a +different trade-off binds a different closure; it does not get a reward term +this document invents. + +Infeasibility is never the environment's answer. `CandidateDomain(u)` is +derived at freeze from `F` and `K`, so a choice the environment never +enumerates is a choice the owner never offered. The environment adds mask bits, +and a mask bit is a phase rule, not a proof. + +## Episode Arms + +An episode is one of two arms, each carrying exactly the fields its own +protocol needs: + +```text +PnrEpisodeArm = + ConstructThenAnneal { // 0 + cleanup_annealing_policy: SearchPolicy.annealing + cleanup_displacement_reward_code: uint64 + } + | ConstructThenRepair { // 1 + repair_step_bound: positive uint64 + repair_step_reward_code: uint64 + } +``` + +Ordinals are stable. A new arm appends; reordering, deleting, or repurposing +one is an incompatible change to this document's schema version. + +Both arms run the same construction phase and differ only in what happens once +every realization has been placed. `ConstructThenAnneal` hands the candidate to +one bounded annealing run and ends; the agent's whole contribution is the +construction, and it is scored on the quality of what construction produced +*minus how much cleanup that construction turned out to need*. +`ConstructThenRepair` keeps the agent in control, letting it rebind whatever it +placed badly, and prices each repair it takes. + +The arms are two answers to one question — who fixes an imperfect placement — +and they are kept as arms rather than as a flag pair because the fields are not +shared. A radius and an annealing policy mean nothing to an agent-driven +repair, and a repair bound and a per-repair price mean nothing to a single +annealing transition. A record carrying all five would have a +required-or-forbidden rule per field and no way to state it except prose. + +`SearchPolicy.annealing` is the exact annealing record +[Annealing And Replay](spec-pnr.md#annealing-and-replay) owns, including its +`realization_move_radius` field, carried here as a complete value rather than +as a reference into `pnr_config_view.search_policy`. The two govern different +runs and have no reason to match: the view's policy is what an ordinary Spatial +PnR invocation of this problem would use, while the cleanup is a deliberately +short, deliberately local run. Neither is an environment-authored policy, and +adoption validates this one exactly as its owner requires. + +The distance bound the arm needs is that record's own +`realization_move_radius`, and the arm carries no second radius field beside +it. A field naming the same bound twice would be two things a future edit could +make disagree, and the one the annealing run actually reads would win silently. +What this arm requires instead is the `Bounded` arm of that field. An +unbounded cleanup makes `cleanup_displacement` a measure of the annealer rather +than of the construction, and it would leave the return range's displacement +term with no bound to read; a `ConstructThenAnneal` arm carrying `Unbounded` is +rejected at adoption. + +## Resolved PnR Environment Configuration + +The environment consumes one immutable component view with schema descriptor +bytes `loom.ml_pnr_environment.config.1.0`, following the framing, canonical +byte representation, and digest contract owned by +[Resolved Configuration](spec-config-ssot.md#component-views): + +```text +ResolvedMlPnrEnvironmentConfigView { + episode_arm: PnrEpisodeArm + problem_pool: canonical nonempty set + pnr_config_view: ResolvedPnrConfigView + episode_policy: PnrEpisodePolicy + observation_policy: PnrObservationPolicy + determinism_policy: EnvironmentDeterminismPolicy +} + +SpatialPnrProblemBinding { + dataflow: ArtifactRootReference // D + tech_mapping: ArtifactRootReference // T + fabric: ArtifactRootReference // F + constraints: ArtifactRootReference // K +} + +PnrEpisodePolicy { + anchor_selection: AgentSelected | CanonicalNext + step_bound: positive uint64 + consecutive_failure_bound: positive uint64 + failed_transition_reward_code: uint64 + incomplete_closure_reward_code: uint64 +} + +PnrObservationPolicy { + enumeration_bound: positive uint32 + include_route_edges: bool + include_resource_states: bool +} + +``` + +`SpatialPnrProblemBinding` is exactly the persistent part of the five-input +invocation. `C` is absent from it and carried once by the view, because every +episode of one configuration searches under the same resolved policy and a +per-problem `C` would let two members of one pool disagree about what the +reward means. Adoption requires each member to satisfy the exact bindings +[Spatial PnR](spec-pnr.md#spatial-pnr) states — `T.D == D.id`, `T.F == F.id`, +and `K.D/T/F == D.id/T.id/F.id` — because a binding that does not is not an +invocation at all and failing at the first `reset` that draws it would report a +configuration error as a runtime one. + +There is exactly one objective closure here, and that is deliberate. The DSE +environment carries two because its probe is an inner search whose own +objective must not be the reward; this environment *is* the search, so the +closure that steers it and the closure that scores it are necessarily the same +one. A second would have to disagree with the first about what a good Mapping +is, and the policy would learn the disagreement rather than the problem. The +reward authority is therefore `pnr_config_view.selected_objective_closure` and +the view carries no closure of its own. + +`PnrEpisodePolicy` holds what applies to the whole episode rather than to one +phase. `step_bound` counts every step an episode takes, construction and repair +alike, and `consecutive_failure_bound` and the two reward codes apply in both +phases; only `anchor_selection` is construction-specific. Splitting a +per-construction record from a per-repair one was rejected because +`ConstructThenAnneal` has no repair phase and would leave the second record +entirely inert. + +`anchor_selection` chooses what the policy decides each construction step. +`AgentSelected` enumerates every unplaced realization crossed with its legal +occurrence domain, so the policy chooses both which realization to place and +where. `CanonicalNext` fixes the anchor as the first unplaced realization in +the canonical typed decision-key order and enumerates only its choices. The +trade-off is stated rather than defaulted away: `AgentSelected` makes placement +order part of what is learned, which is the more interesting half of the +problem, and it makes the enumeration the sum of every unplaced realization's +domain rather than one realization's, so `enumeration_bound` has to be sized +for it. `CanonicalNext` is the cheaper regime and the honest baseline against +which learned ordering is measured. + +The canonical view encoder writes fields in the schema order above, under the +encoding and adoption rules +[Resolved Configuration](spec-config-ssot.md#component-views) owns. This view +adds only that an `ArtifactRootReference` and an embedded owner record use +their owner's canonical encoding. + +The PRNG preimage shape, the meaning of `effective_seed` and the copy +coordinates, and the `reset` seed override are owned by +[Determinism And Copy Coordinates](spec-ml-core-environment.md#determinism-and-copy-coordinates). +This document adds only its own domain separator, which is + +```text +ASCII("loom.ml_pnr_environment.prng.sha256_seeded_xoshiro256starstar.1.0") +``` + +and its stream purpose ordinals, which are `ProblemSelection = 0` and +`CleanupSeeding = 1`. + +`CleanupSeeding` exists because the cleanup run is an ordinary annealing run +and therefore consumes PnR's own `Calibration`, `ActionProposal`, and +`Acceptance` streams, which that owner derives from a master seed and a seed +index. The environment draws that seed index once per episode from +`CleanupSeeding` rather than reusing a counter, so the cleanup is a function of +the episode's effective seed and copy coordinates exactly as every other +episode-local draw is. Reusing `local_episode_index` directly was rejected: a +`reset` seed override would then change which problem an episode ran but not +which cleanup it got. + +### Curriculum-Neutral Fields + +[Curriculum](spec-ml-core-training.md#curriculum) requires each environment +document to partition its own view by what a field determines. Here the +shape-fixing side is `episode_arm`, `pnr_config_view`, `observation_policy`, +`determinism_policy`, and, inside `PnrEpisodePolicy`, `anchor_selection` and +the two reward codes. The arm decides which phases exist and therefore which +Action kinds are ever live; the Place and Route view carries the objective +closure the reward is a difference of; `anchor_selection` decides whether the +policy chooses anchors at all, which changes the enumeration's extent rather +than its size. + +`problem_pool` is neutral, and that it is neutral is the point of stating the +rule by what a field determines rather than by naming a record. The pool sits +outside `PnrEpisodePolicy` here while the design-space environment's +corresponding field sits inside its episode policy, so a rule phrased as one +record name would forbid growing the problem pool — the one curriculum this +environment most obviously wants. `step_bound` and `consecutive_failure_bound` +are neutral for the same reason they are there: they bound an episode's length +and price nothing. + +Growing the pool costs nothing already warm. A stage that adds problems keeps +every retained frozen model and canonical seed state for the problems it +already had, because those are keyed on one `SpatialPnrProblemBinding` rather +than on `problem_pool`: the key names a member, not the set, so extending the +set invalidates nothing in it. + +## Problem Instance And Frozen Model Reuse + +Freeze is the dominant cost in this environment and it is the reason `reset` is +worth a section of its own. `freezeSpatialPnrProblem(D, T, F, C, K)` validates, +resolves, indexes, and precomputes the complete Fabric projection; a step, by +contrast, touches only changed incidence. An implementation that froze per +episode would spend most of a training run in `reset`. + +It does not have to. The Spatial PnR cache key +[Native State](spec-pnr.md#native-state) defines already hashes exact `D.id`, +`T.id`, `F.id`, `K.id`, the fields of `C` that freeze reads, freeze and +importer semantics, the native-layout ABI, and the actual `PnrIndex` width. +Every one of those is constant across the episodes of one configuration that +draw the same `SpatialPnrProblemBinding`, so the environment retains published +`FrozenModel` values under that key and reuses them across episodes and across +the vector copies living in one process. + +The key names the freeze-relevant projection of `C` rather than +`component_view_digest(C)`, and the difference is the one that decides whether +a study is affordable. Freeze precomputes a Fabric projection; a reward closure +and a search policy are not inputs to it. Keying on the whole digest would make +a sweep over `selected_objective_closure` — the obvious study over this +environment — re-freeze every problem in the pool for a result byte-identical +to the one it discarded, which is the same trap +[Configuration Split](spec-ml-core-training.md#configuration-split) draws the +training and environment views apart to avoid. The full digest is still +compared on a hit, as the revalidation below requires; it is a check, not the +key. A `FrozenModel` is immutable and explicitly +shared across workers, so sharing it across environment copies introduces no +mutability the owner does not already permit. + +A cache hit revalidates the descriptor, canonical view bytes, digest framing, +and exact artifact inputs before reuse, exactly as the owner requires. +Retention across episodes is an ordinary cache under +[Cache Dependencies](spec-config-ssot.md#cache-dependencies): a hit and a miss +produce the same formal result, and evicting the whole cache changes cost and +nothing else. + +This is also why the cleanup radius is a search-policy field rather than a +per-episode `K`: `K` is an input to freeze, so a per-episode one would give +every episode a distinct cache key and destroy exactly the reuse this section +depends on. The field's home is +[Annealing And Replay](spec-pnr.md#annealing-and-replay). + +## Construction Phase + +Every episode begins in the construction phase and leaves it when every Compute +and Memory Realization has been placed exactly once. + +`reset` builds the candidate through `createCanonicalSpatialCandidate` +-equivalent construction: the canonical assignment the owner's attempt-zero +initializer produces, with every RouteTree left visibly unrouted. The +environment then marks every realization *unplaced* in an episode-local bitset. +That bitset is the only construction state the environment owns. + +Each construction step applies exactly one `RealizationBindingAction` whose +anchor is an unplaced realization, through the ordinary probe-and-commit path. +On commit the anchor becomes placed. Routing of that realization's dependencies +is not something this document adds: a binding change already invalidates old +attachments and route claims, rebuilds every incident route dependency, and +updates resource-time, buffer, tag, memory, and handshake state inside the same +transaction, which is precisely "route the dependencies of the node just +placed". The environment issues no routing Action of its own during +construction. + +A realization becomes placed whether or not its incident nets closed. A net +whose other endpoint is still unplaced routes against that endpoint's canonical +seed binding and is re-closed when that endpoint is later bound, and a net that +cannot close at all leaves an `UnroutedObligation` violation in the candidate. +Both are ordinary candidate state that the objective prices and the reward +reports, which is what makes construction a dense-reward problem rather than +one that pays only at the end. + +An unplaced realization whose dynamic Action domain offers no alternative to +its current binding becomes placed without a step. The owner's domain contains +only anchors with at least one legal alternative, so a realization whose +`CandidateDomain` is the singleton its canonical assignment already selected +contributes no entry and the agent has nothing to choose; treating it as +unplaced would let construction stall with realizations outstanding and no live +entry to advance on, which the action surface has no defined behavior for. +Marking it placed is exact rather than a concession: the choice the agent would +have made is the only choice there is, and the candidate already holds it. + +Settling is therefore re-evaluated in both directions after `reset` and after +every commit, because a commit can shrink another anchor's domain to a +singleton and can equally restore an alternative to one already settled. A +settled realization whose domain regains an alternative returns to unplaced and +re-enters the enumeration. One-way settling would leave the exactness claim +false: a realization settled while its only choice was forced would stay +settled after the choice came back, and the agent would never be offered a +decision that had become real. + +The rebind-sweep formulation has a real cost and it is stated rather than +hidden. Because `CandidateState` is always a complete assignment, an early +construction step routes nets whose far ends are still arbitrary, and a later +step redoes that work. A genuinely partial candidate with an unbound sentinel +would do strictly less work and was rejected as a semantic change to +[Native State](spec-pnr.md#native-state); why that trade went the way it did is +[Why Learned Place And Route Selects Existing Actions](rationales/ml.md#why-learned-place-and-route-selects-existing-actions). + +Construction is not an initializer replacement in the owner's sense. The +owner's initializer propagates singleton domains and hard relation consequences +to a fixed point and backtracks on contradiction; the environment does neither, +because every state it occupies is already a complete legal assignment and +there is nothing to backtrack from. What the policy replaces is the choice of +which anchor to move next and which choice to take, which is the only part of +that protocol that is a heuristic rather than a proof. + +## Cleanup Phase + +When the last realization is placed, the arm decides what happens next. The +step that places it reports its ordinary transition either way; what differs is +whether that step also ends the episode. + +### Bounded-Radius Annealing + +Under `ConstructThenAnneal`, the step that completes construction also runs one +annealing run over the same candidate and then ends the episode with +`CleanupComplete`. The agent takes no action during the cleanup; it is a single +transition from the agent's point of view, and the observation that step +returns is the post-cleanup state. + +The run is an ordinary one. It uses `cleanup_annealing_policy` unchanged, PnR's +own acceptance kernel and cooling schedule, and PnR's own streams seeded by the +index drawn from `CleanupSeeding`. The run-start occupancy its radius anchors +on is exactly the placement construction produced. + +That anchoring is the whole point of the bound this arm requires: a bounded +cleanup can only tidy locally, so what it recovers is a measure of what +construction left on the table nearby. + +`cleanup_displacement` is that measure, and it is a mechanical projection of +two candidates rather than a statistic the run reports: + +```text +cleanup_displacement = + sum over realizations of + directed frozen-topology hop distance from its + construction-final occurrence to its cleanup-final occurrence + + count of realizations whose occurrence changed +``` + +The distance is the same one +[Deterministic Initialization And Action Proposal](spec-pnr.md#deterministic-initialization-and-action-proposal) +defines and the radius itself measures over, so no second distance model +appears. The two terms answer different questions and both are needed: the hop +sum says how far the cleanup had to reach, and the moved count says how broadly +it had to intervene. A single far move and a hundred adjacent ones are not the +same failure, and a sum alone cannot tell them apart. + +`cleanup_displacement` is zero exactly when the cleanup accepted no move that +changed an occurrence, which is the case the arm exists to reward. + +### Agent Repair + +Under `ConstructThenRepair`, completing construction advances the episode into +the repair phase and the episode continues. The live enumeration opens to the +complete dynamic Action domain: the agent may rebind an already-placed +realization, issue a `WholeNet`, `SingleSink`, `RootedSubtree`, or +`WitnessRegion` routing Action, or take a resource-allocation Action. Nothing +in the domain is masked by phase any more. + +Each committed step in this phase is a repair step and is charged +`repair_step_reward_code`. The episode ends when the agent elects to stop or +when it has taken `repair_step_bound` repair steps. + +Two properties make this a different problem from the annealing arm rather than +a slower version of it. The agent decides *when* it is done, so it is scored on +recognizing a good enough Mapping and not only on producing one; and it may +take routing and resource Actions, which construction never offers, so it can +repair a congestion problem without moving anything. Neither is available to a +bounded annealing run whose whole neighborhood is realization rebinding. + +The repair phase deliberately has no radius. The agent's every move is scored, +so an unhelpful long move is already paid for by its energy delta and its step +charge, and bounding it as well would price the same mistake twice while +removing the one capability — reaching across the fabric when that is genuinely +right — that distinguishes a learned repair from a local search. + +That holds because a radius bounds an annealing run's proposal domain and not +the candidate's Action domain, per +[Annealing And Replay](spec-pnr.md#annealing-and-replay). The only radius that +governs anything in an episode is the one the arm's `cleanup_annealing_policy` +carries, and it governs the cleanup run alone. + +## Action Space + +The action surface, the `enumeration_bound` capacity, the masking algebra, and +the elective stop are owned by +[Action Surface And Masking](spec-ml-core-environment.md#action-surface-and-masking). + +The enumeration is the candidate's own dynamic Action domain, in the owner's +canonical order. For the current candidate, the environment enumerates the +realization-binding choices, then the transport-routing choices, then the +resource-allocation choices, each anchor in canonical typed decision-key order +and each anchor's choices in that anchor's canonical choice order. An action is +one ordinal in that concatenation: + +```text +EnumeratedPnrAction { + kind: RealizationBinding | TransportRouting | ResourceAllocation + anchor: ordinal in that kind's canonical anchor domain + choice: ordinal within that anchor's contiguous choice range +} + +ActionIndex = uint32 +``` + +Nothing here is a new enumeration. `A(M,C,S)` is already deterministic, already +partitioned by kind, and already grouped into contiguous per-anchor choice +ranges; this document only fixes that kinds concatenate in the order shown and +that an `ActionIndex` is the resulting ordinal. Two runs with equal candidate, +frozen model, and resolved configuration therefore produce byte-identical +enumerations in the same order, and that follows from the owner's determinism +rather than from a rule here. + +Phase decides which of those entries are live, and this is the only place the +environment clears a mask bit: + +- in the construction phase, every entry whose kind is not `RealizationBinding` + is cleared, and every `RealizationBinding` entry whose anchor is already + placed is cleared; +- under `anchor_selection` of `CanonicalNext`, every `RealizationBinding` entry + whose anchor is not the first unplaced realization in canonical order is also + cleared; +- in the repair phase, nothing is cleared; and +- the stop outcome is cleared for the whole construction phase, and under + `ConstructThenAnneal` for the whole episode. + +Stop is cleared during construction because an episode that stops before every +realization is placed has produced nothing a Mapping could be made from, and an +always-available exit from a phase whose early rewards are negative is an exit +a policy learns to take. Under `ConstructThenAnneal` the agent never reaches a +phase where stopping is meaningful at all, so the outcome is cleared for the +whole episode; the episode ends when construction does. + +A cleared mask bit is a phase rule and never a legality claim. Every entry the +mask clears is a legal Action of the current candidate, and the same entry +becomes live again when the phase changes. This is the opposite of the DSE +environment's per-state rejection mask, which records that a decision was tried +and failed; nothing here is masked because it failed. + +The policy scores an action from the two nodes it names — the anchor it acts on +and the choice it selects — both of which are ordinary nodes of the observation +graph, named by the entry's own columns. That is what keeps an action a single +ordinal even though it is a pair. + +## Observation + +### The PnR Node And Link Space + +The combined node-and-link space, its `GraphNodeRole` catalog, the +connection-as-node rule, the two enumeration encodings, and the target-closure +rule are owned by +[Combined Node And Link Space](spec-ml-core-environment.md#combined-node-and-link-space) +and [Enumeration Encoding](spec-ml-core-environment.md#enumeration-encoding). +This section states only what that space contains for a PnR episode. + +Four role blocks are present in every episode, and this is the environment that +needs all four at once: + +- `FabricOccurrence` spans every occurrence in the frozen model's Fabric + projection, in canonical owner order — the hardware graph; +- `FabricConnection` spans every physical traversal between them, each + replacing the direct arc it stands for — the routing resources, which must be + nodes here rather than arcs because a routing Action names them and a route's + occupancy is a per-traversal fact; +- `DataflowOperation` spans the Realizations of `T` — the software graph's + nodes, which are what construction places; and +- `DataflowValue` spans the residual logical nets — the dependencies that + routing closes. + +The `Decision` block is absent, because this environment uses the +`DecisionColumns` encoding. + +Placement arcs are unconditional here, unlike in the DSE environment where they +are a configuration option. The current mapping is not extra context in this +environment; it is the state. Each placed Realization node carries one +`Placement` arc to the occurrence bound to it, so "where is this node placed" +is one hop for the encoder rather than a numeric column it has to learn to +dereference. + +When `include_route_edges` is set, each logical-net node additionally carries +one `Route` arc to every `FabricConnection` node its current RouteTree +traverses. This is the expensive part of the observation and it is optional for +that reason: a fully routed candidate has far more route incidences than +placements, and a policy that only places may not need them. A policy that +takes routing Actions in the repair phase does. + +Every live action names exactly two nodes — its anchor and its choice — and it +names them by column rather than by arc. Both resolve under the target-closure +rule, which for this environment means every anchor and every choice of every +live entry is addressable: a realization binding names a Realization node and +an occurrence node, a routing Action names a logical-net node and, for a scoped +variant, the endpoint or traversal node its scope anchors on, and a +resource-allocation Action names its demand and its selected endpoint. + +A PnR action names two nodes and never more, so its arity is fixed and the core +selects `DecisionColumns` for it. The overhead the core's rule warns about is +not marginal here: under `AgentSelected` the enumeration is the sum over +unplaced realizations of each one's legal occurrence domain, so an ordinary +problem enumerates far more actions than the design has entities. + +What the two encodings share is the part that matters to a policy, and it is +the part this environment depends on: an action is scored from the embeddings +of the nodes it names. A DSE decision's value is a prototype ordinal or a +bounded delta, so a value term can read it from a column directly. A PnR +action's value is a graph node — an occurrence, an endpoint, a traversal — so +the column holds that node's ordinal and the value term reads its embedding. A +policy head that scored a PnR action from its anchor alone would make every +occurrence on one realization indistinguishable, which is the entire decision. + +### Column Catalogs + +The `Observation` and `GraphInstance` container shapes, the no-padding rule, +the negative-one absent sentinel, the buffer lifetime, and the obligations +every column catalog satisfies are owned by +[The Graph Instance](spec-ml-core-environment.md#the-graph-instance). The +closed column catalogs this environment owns are: + +```text +PnrNodeFeatureColumn = + Role // 0 + | EntityKind // 1 + | CapabilityCount // 2 + | CapacityMagnitude // 3 + | CapacityUsage // 4 + | CapacityOveruse // 5 + | BufferDepth // 6 + | InDegree // 7 + | OutDegree // 8 + | SelfCycle // 9 + | Placed // 10 + | UnroutedObligationCount // 11 + | RouteClaimCount // 12 + | TagUnassignedCount // 13 + | TagConflictCount // 14 + +PnrArcRole = + Structural // 0 + | Placement // 1 + | Route // 2 + +PnrDecisionColumn = + ActionKind // 0 + | AnchorNode // 1 + | ChoiceNode // 2 + | ChoiceDistance // 3 + +PnrScalarFeatureColumn = + StepOrdinal // 0 + | StepBound // 1 + | Phase // 2 + | PlacedCount // 3 + | RealizationCount // 4 + | UnmaskedActionCount // 5 + | RepairStepCount // 6 + | RepairStepBound // 7 + | ConsecutiveFailures // 8 + +EpisodePhase = + Construction // 0 + | Repair // 1 +``` + +`EntityKind` is the owner-local reference kind ordinal from +[Fabric Identity](spec-fabric-identity.md) for a Fabric node and the Dataflow +or TechMapping owner's kind ordinal for a Realization or net node, so a new +Fabric occurrence kind extends the observation without a new column and without +a new role. + +`CapacityMagnitude`, `CapacityUsage`, and `CapacityOveruse` are the raw +declared capacity, the raw current usage, and the raw current overuse of the +referenced entity, projected from the candidate's own occupancy caches. Overuse +is carried separately rather than left to be derived from the other two because +it is the quantity the objective prices and a policy should read it directly +rather than reconstruct it. The typed atoms of the `FabricResourceStateRef` +catalog the core appends under `include_resource_states` are owned by +[Fabric Resource Contract](spec-fabric-resource-contract.md). + +`Placed` carries the construction bitset on a Realization node. Which +occurrence a Realization is bound to, and which Realization occupies an +occurrence, are not columns: the `Placement` arc carries that relation in both +directions, so a policy reads it in one message-passing hop rather than by +dereferencing an ordinal, which is a thing an embedding space cannot do. + +`UnroutedObligationCount`, `RouteClaimCount`, `TagUnassignedCount`, and +`TagConflictCount` are per-node projections of four of the five Mapping +violation magnitudes and of the traversal claim, attributed to the entity that +carries them. The fifth violation, hard progress, is a whole-candidate fact +with no per-node attribution and appears only through `objective_codes`. + +`PnrDecisionColumn` is the row catalog of `decisions`, one row per live entry +in the enumeration order Action Space defines, so a row's ordinal is its +`ActionIndex`. + +`AnchorNode` and `ChoiceNode` hold node ordinals into the same observation's +`graph`, under the core's rule for a target-naming column. They are the entry's +complete reference set, and the target-closure rule binds them exactly as it +binds a target arc. + +`ActionKind` carries the entry's `SpatialMappingAction` kind. + +`ChoiceDistance` is the directed frozen-topology hop distance from the anchor's +current occurrence to the choice's occurrence for a realization binding, and +the absent sentinel otherwise. It is the one derived column in any of this +environment's catalogs, and it is derived because distance is the single most +load-bearing quantity in placement while being the one a graph encoder is worst +at recovering: reading it off the embeddings would mean propagating information +across as many message-passing layers as the fabric has hops, so a policy on a +large fabric could not represent it at all at any practical depth. Its two +endpoints are Fabric occurrences and the metric is the frozen topology's, so +the value for one occurrence pair is constant for the whole problem and an +entry's distance changes only when its anchor rebinds. + +The selected closure whose codes `objective_codes` carries is +`pnr_config_view.selected_objective_closure`. + +A failed step neither advances nor resets, so under the core's buffer-lifetime +rule it does not invalidate the buffers; the candidate was rolled back to the +value it already had, and only the step and failure scalars change in place. + +## Episode Start + +An episode is created from one problem binding. There is no workload set and no +seed design: the problem is fixed for the whole episode, and what varies is the +candidate. + +`reset` performs this ordered protocol: + +1. derive the episode's PRNG streams from the effective seed and the copy + coordinates; +2. select one `SpatialPnrProblemBinding` from `problem_pool` through + `ProblemSelection`, or adopt the one the start override supplies; +3. under `ConstructThenAnneal`, adopt the effective seed as the cleanup seed + index when a start override is present, and otherwise draw it from + `CleanupSeeding`, in either case on every episode of that arm whether or not + construction ever completes; +4. acquire the `FrozenModel` and the canonical seed state for that binding and + `pnr_config_view` from the retained cache, or publish them by freezing and + building; +5. adopt the canonical candidate, its objective vector, its search energy, and + its settled bitset, and mark every realization unplaced; +6. adopt the retained canonical enumeration and its refusal verdict; and +7. build the first observation. + +Step 3 precedes step 4 so that the stream position after `reset` does not +depend on whether a cache hit occurred, which is what keeps a run with a warm +cache and a run with a cold one formally identical. + +The canonical seed state is retained beside the `FrozenModel` and under the +same key, because it is constant under the same key. The canonical candidate is +the owner's attempt-zero initializer output, and its objective vector, search +energy, settled bitset, first enumeration, and that enumeration's empty-or- +over-capacity verdict are pure functions of it — none reads the episode's seed, +its streams, or anything an episode has yet done. The enumeration is retained +with the rest, which matters most of all: under `AgentSelected` it spans every +unplaced realization's occurrence domain and is the largest of the episode. +Every episode drawing one problem would otherwise rebuild and re-evaluate an +identical value, once per episode for the whole run. Steps 5 and 6 are +therefore copies, and the energy is not recomputed at all but carried. This is +retention on the same terms as the frozen model: a hit and a miss produce the +same formal result. + +Step 6 precedes step 7 because refusing is cheaper than projecting a graph the +episode will not use, and a configuration that could never offer a legal action +should be rejected without paying for an observation. + +An override adopts the effective seed rather than drawing, because +`CleanupSeeding` is derived from a preimage that includes the copy coordinates +and an overridden episode must not depend on them. Drawing there would make a +`ConstructThenAnneal` case anneal differently on each runner, which is +precisely the reproducibility the override exists to provide and which the test +protocol depends on. The effective seed is the caller's own input, so a case is +determined by its problem and its seed alone. + +The episode start override this environment defines is one +`SpatialPnrProblemBinding`, carried by the Gymnasium `options` argument. It is +the payload itself rather than a record wrapping it, since a record with one +field is a name for that field. + +The core's override rules bind it: it must appear in `problem_pool`, and when +present it replaces step 2 entirely, so `ProblemSelection` is not consulted. + +```text +PnrEpisodeStartOutcome = + Started + | ProblemProvenInfeasible { problem, diagnostic } + | FreezeCapacityExceeded { problem, required_index_width } + | EnumerationBoundExceeded { problem, bound, required } + | Invalid { violated precondition } +``` + +`ProblemProvenInfeasible` reports that freeze proved an empty well-formed +domain, which is a sound proof about the problem and not about this episode; it +is reported rather than retried, because every retry would draw from the same +pool and a pool member that is infeasible is infeasible on every draw. +`FreezeCapacityExceeded` is separate because it is a Loom build-capacity error +rather than Mapping infeasibility, exactly as +[Native State](spec-pnr.md#native-state) requires, and it names the required +`LOOM_PNR_INDEX_BITS` width so the remedy is stated rather than guessed. + +`EnumerationBoundExceeded` reports a canonical candidate with more admissible +Actions than the action space can index, naming both the capacity and the +required length so the configuration can be corrected rather than guessed at. +It is separate from `Invalid` for that reason: the core's refusal rule requires +both numbers, and folding it into a violated-precondition report would drop +them. + +`Invalid` covers a canonical candidate with no live entry, which cannot happen +for a well-formed problem with at least one movable realization and is +therefore a projection defect rather than a state to retry from. It also covers +a binding whose exact `D`/`T`/`F`/`K` coupling does not hold and a problem +outside `problem_pool` named by an override. + +There is no retry budget, and its absence is the point. A DSE episode redraws +because a seed may be unusable; a PnR problem that adoption accepted is usable +by construction, so a failure here is a configuration or owner defect and +retrying would hide it. + +## Step + +One step applies exactly one enumerated Action. The protocol is ordered, and +stages 1 through 5 may fail: + +1. validate the action against the live mask; +2. decode the index to a typed `SpatialMappingAction` against the current + enumeration; +3. probe the transition, which computes its complete dependency closure, + applies it in a shadow candidate, closes every affected route, and evaluates + the resulting `V/G` and the selected search energy; +4. rebuild the dynamic Action domain against the probed shadow candidate, and + fail when its length exceeds `enumeration_bound`; +5. commit the probe when stages 3 and 4 both succeeded, and roll it back + otherwise; +6. mark the anchor placed when stage 5 committed a construction binding, and + settle every realization the commit left with no alternative; +7. run the cleanup and end the episode when stage 6 placed the last realization + under `ConstructThenAnneal`, or advance the phase to `Repair` under + `ConstructThenRepair`; +8. build the observation over the candidate the step left behind, applying the + phase mask to that candidate's domain. + +Stage 8 names the resulting candidate rather than stage 4's, because stage 4 +builds against a shadow that a failed step discards; masking that domain after +a rollback would describe a candidate the episode does not hold. A failed step +leaves the candidate, its domain and its mask exactly as they were, so it +recomputes none of them — the buffers are still the ones the core's +buffer-lifetime rule keeps valid. + +A step that ends the episode returns an observation of the state it ended in, +carrying an empty `decisions` and an all-clear mask, which +[Action Surface And Masking](spec-ml-core-environment.md#action-surface-and-masking) +exempts from the admissible-outcome rule precisely because no action is ever +sampled from a terminal observation. That is what keeps the cleanup outside the +capacity contract: stage 7 rebinds realizations and can leave a candidate whose +domain exceeds `enumeration_bound`, and there is nothing to roll back to by +then, but the observation that reports it enumerates nothing and so has nothing +to exceed. + +Stage 3 is the whole of the owner's online protocol and this document adds +nothing to it. In particular there is no acceptance test: the annealing policy +resolves a probe against a temperature and may reject an +improving-in-expectation move, whereas the policy here has already chosen, so +every probe that the owner does not fail and stage 4 admits is committed. +Reintroducing an acceptance kernel on top of a learned proposal would put two +selectors in series and make the reward describe neither. + +Stage 4 precedes the commit because that is the only place the capacity test +can protect the candidate. The enumeration is a function of the candidate, so +it does not exist until the transition has been applied; testing it after the +commit would leave the episode holding a state the observation cannot +represent, with nothing to roll back to. Rebuilding against the probe's shadow +candidate tests the state the step would produce while the step can still be +undone, which is what lets `EnumerationBoundExceeded` be an ordinary transition +failure that leaves the candidate unchanged. + +The enumeration's length does not depend on the phase, which is why the phase +transition at stage 7 needs no capacity test of its own. The domain is the +candidate's complete dynamic Action domain under every phase; what the phase +changes is the mask, and clearing a mask bit removes no entry. Opening the +repair phase therefore raises `UnmaskedActionCount` and leaves the enumeration +exactly the length stage 4 already admitted. The two are different quantities +and are named differently for that reason: the enumeration's length is how many +entries `decisions` carries, which the mask never changes, while +`UnmaskedActionCount` is how many of them the policy may currently select. + +The outcome algebra is closed: + +```text +PnrStepResult { + transition: optional + episode_end: optional +} + +PnrStepTransition = + Advanced { energy_delta_code, energy_delta_sign } + | Failed { reason: PnrTransitionFailureReason } + +PnrTransitionFailureReason = + IntrinsicInvalid { diagnostic } + | WorkLimit { diagnostic } + | ObjectiveUnavailable { dimension } + | EnumerationBoundExceeded { bound, required } + +PnrTerminalReason = + ElectiveStop + | CleanupComplete + | RepairBoundReached + | StepBoundReached + | ConsecutiveFailureBoundReached + | EnumerationEmpty +``` + +`IntrinsicInvalid` and `WorkLimit` are exactly the two members of the owner's +transition-failure taxonomy, and keeping them apart is the same +proof-versus-budget discipline the rest of the stack uses. `IntrinsicInvalid` +means the Action cannot produce a legal candidate — a newly closed directed +handshake cycle is the canonical case — and is a fact about the Action. +`WorkLimit` means the router exhausted its configured budget for this move and +establishes nothing at all about whether the Action was good. A blended reason +would let a rising router-budget problem read as a policy that proposes illegal +moves. + +`ObjectiveUnavailable` is this environment's name for the core's +unavailable-objective outcome, and its transaction is rolled back with the +rest. + +A transition and an episode ending are separate facts because they co-occur, on +the terms +[Step Accounting And Episode Endings](spec-ml-core-environment.md#step-accounting-and-episode-endings) +states. The step that places the last realization under `ConstructThenAnneal` +both advances and ends the episode; the failure that reaches +`consecutive_failure_bound` both fails and ends it; an elective stop ends it +with no transition at all. The core's `non_advancing` class is `Failed` here. + +`ElectiveStop`, `CleanupComplete`, and `EnumerationEmpty` are terminations; +`RepairBoundReached`, `StepBoundReached`, and `ConsecutiveFailureBoundReached` +are truncations, each cutting the episode off by a configured limit while the +candidate remained ordinary. + +## Reward + +The per-step reward is the signed difference of the selected search energy +across the transition, on the terms +[Reward Contract](spec-ml-core-environment.md#reward-contract) states. The +parent energy is retained from when the parent was entered and is not +recomputed, so a step evaluates one energy. + +This is the same reward shape the DSE environment uses and it is worth naming +what it amounts to here: with energy as a potential, a per-step signed energy +difference is potential-based shaping, so the return of an episode telescopes +to the total improvement from the canonical candidate to the final one, and the +dense per-step signal changes what is learnable without changing what is +optimal. That the weighting inside it — violations against traversal claim — +comes from the resolved closure rather than from constants in this document is +what lets one environment serve a search that cares about congestion and one +that cares about latency. + +Three terminal terms are charged on top, and each answers one requirement of +its arm. + +`ConstructThenAnneal` charges the product of `cleanup_displacement_reward_code` +and `cleanup_displacement` with a negative sign on the step that ends the +episode, alongside that step's ordinary energy delta, which is the energy the +cleanup recovered. The episode's return is therefore the final Mapping's +quality minus the amount of cleanup that quality required. Both halves are +necessary: without the energy term the policy is not scored on the design at +all, and without the displacement term it learns that a mediocre construction +is free because the annealer will fix it. + +`ConstructThenRepair` charges `repair_step_reward_code` with a negative sign on +each committed repair step. The episode's return is the final Mapping's quality +minus how long repair took, which is the same trade the other arm makes with +the annealer's work in place of the agent's. + +Both arms charge `incomplete_closure_reward_code` with a negative sign on the +final step of any episode whose final candidate is not closed, meaning its five +Mapping violation magnitudes are not all zero. Those magnitudes are the ones +the probe already maintains incrementally and evaluated for this very +candidate, so the charge reads a value the step produced rather than triggering +a recomputation — which is also what keeps the rule consistent with this +environment never running independent final verification. + +That last charge is deliberately not an ending price, and the distinction +matters because the core prices a truncation at zero. What the core refuses to +charge for is the *limit*, which the policy did not choose. Closure is a +property of the candidate, which the policy did choose, and a truncated episode +that leaves an unclosed candidate has failed at the task regardless of why it +stopped. Pricing the limit and pricing the state are different charges and only +the first is forbidden. + +A `Failed` transition yields `failed_transition_reward_code` with a negative +sign, independent of the reason. It does not distinguish `IntrinsicInvalid` +from `WorkLimit`, because charging more for one would make the reward depend on +a router budget. + +`cleanup_displacement_reward_code` multiplied by a displacement over a large +design is the one product here with real magnitude, and it is checked like +every other. + +## Trajectory Retention And Replay + +An episode retains one transient record: + +```text +PnrTrajectory { + config_view_digest: ComponentViewDigest + problem: SpatialPnrProblemBinding + effective_seed: u64 + env_runner_index: uint32 + vector_index: uint32 + local_episode_index: uint64 + cleanup_seed_index: optional + advanced_actions: ordered sequence +} +``` + +`cleanup_seed_index` is present exactly under `ConstructThenAnneal`. + +Replay is deterministic re-execution. Given the same configuration digest, the +same problem binding, and the recorded action sequence, replaying against a +`FrozenModel` for that binding reproduces an identical final `CandidateState`, +an identical objective vector, and an identical energy. Nothing about the copy +coordinates or the seed is needed for that, because the recorded actions +already name every choice the streams would have made; the seed fields are +retained so that the episode itself can be regenerated rather than only its +outcome. + +Turning a replayed candidate into a published `SpatialMapping` is outside this +document. A candidate becomes a Mapping only through the canonical search +sequence's own path — final global negotiated closure, full owner +recomputation, independent verification, and finalization — and reaching that +path from a learned policy means invoking the policy as a search-policy +selector inside an ordinary Spatial PnR invocation, which is a change +[Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism) owns +and this document does not make. Until that exists, this environment produces +trained policies and measurements, not Mappings. Stating the boundary is +preferable to a replay rule that publishes from a harness, which would put a +training artifact on the product path. + +## Python Boundary + +The `loomml` package layering, the RLlib conformance target and its +obligations, the zero-copy array contract, the outcome-versus-exception rule, +and the threading and `fork` rules are owned by +[Python Boundary](spec-ml-core-environment.md#python-boundary). This +environment occupies `loomml.env.pnr` and `loomml.rllib.pnr` and adds no rule +of its own. + +One consequence is worth naming because it is this environment's dominant +implementation constraint. The native layer holds a `FrozenModel`, a +`CandidateState`, and the owner's reusable scratch, none of which crosses into +Python; what crosses is the observation buffers and one action index. A step is +therefore a native transaction plus one marshalling boundary, and the shared +`FrozenModel` is what makes several vector copies in one process affordable. +Under the core's rule that an instance is owned by one thread and shares no +mutable state, the copies share only that immutable model. + +## Benchmarking + +The harness obligations, the required breakdowns, the instrumentation rule, and +the ratio-based regression budget are owned by +[Benchmarking Harness Contract](spec-ml-core-environment.md#benchmarking-harness-contract). +This environment ships `loom-pnr-env-bench`, whose failure-reason breakdown is +keyed by `PnrTransitionFailureReason` and whose action partition is the +`SpatialMappingAction` kind, because a routing Action and a binding Action do +not cost remotely the same. + +The harness decomposes one step into these stages and reports each separately: + +```text +enumeration rebuild the dynamic Action domain and the phase mask +decode map the action index to a typed SpatialMappingAction +probe_closure the shadow transition and its incident route closure +probe_objective incremental V/G recomputation and search-energy evaluation +resolve commit or roll back the transaction +cleanup the bounded annealing run and its displacement projection +observation build the graph and the decision columns +marshal expose buffers across the Python boundary +``` + +`probe_closure` and `probe_objective` are the two halves of the single probe +the step protocol performs, split here because they are the two costs worth +telling apart and reported as one probe nowhere. Stages partition the step; +none contains another, so the stage sum is the step. + +A reset decomposes into `problem_lookup`, `freeze`, `seed_state_build`, +`enumeration`, and `first_observation`. The seed state's build and evaluation +are one stage because retention makes them one event: on a hit neither runs. +`seed_state_build` covers the canonical enumeration as well, since retention +makes the whole seed state one event: on a hit none of it runs. It is reported +separately from `freeze` because the two differ by orders of magnitude and a +blended `reset` percentile over a mixed hit-and-miss population describes +neither. + +Two further breakdowns are required beyond the core's two. + +Every measure is reported per phase the arm actually runs. A construction step +and a repair step run the same code over candidates with completely different +route densities, and a blended percentile moves whenever an episode's phase mix +moves, which it does throughout training as the policy learns to finish +construction faster. Under an arm with one phase the breakdown is that phase, +reported without a second empty column, and the core's action partition +collapses the same way: it is over the kinds the bound arm can actually make +live, which under `ConstructThenAnneal` is `RealizationBinding` alone. + +And `reset` is reported with the frozen-model cache hit rate alongside it, +separated into hit and miss populations rather than blended, because a single +`reset` percentile over a mixed population describes neither and moves with the +pool size rather than with any code change. The hit rate is the number that +decides whether the environment is affordable at all, and it is reported as a +first-class measure and not as a footnote to a latency. + +The harness reports the deterministic PnR work-unit counts — assignment +attempts, endpoint expansions, negotiation iterations — alongside its timings +and states which is which, because those are the cross-machine cost measure and +the timings are not. + +## Anchor Verification + +Stable tests cover adoption rejecting a `problem_pool` member whose `T.D`, +`T.F`, or `K.D/T/F` binding is not exact; adoption rejecting a +`ConstructThenAnneal` arm whose `cleanup_annealing_policy` carries an +`Unbounded` `realization_move_radius`, and a `ConstructThenRepair` arm whose +`repair_step_bound` is zero; the enumeration equalling the owner's dynamic +Action domain in canonical kind, anchor, and choice order, and being +byte-identical across two runs with equal candidate and configuration; the +construction mask clearing every non-`RealizationBinding` entry and every entry +whose anchor is already placed, and `CanonicalNext` additionally clearing every +anchor but the canonically first unplaced one; the repair phase clearing +nothing; the stop outcome being clear for the whole construction phase and for +the whole of a `ConstructThenAnneal` episode; a cleared entry becoming live +again when the phase changes, and no cleared entry ever being reported as +illegal; the `Decision` block being absent and `decisions` carrying one row per +enumerated entry and no arcs, in the enumeration order, with a row's +ordinal equalling its `ActionIndex`; every live entry's `AnchorNode` and +`ChoiceNode` resolving to exactly one node ordinal in the same observation's +`graph`, including an occurrence-valued, a traversal-valued, an +endpoint-valued, and a net-valued reference; `ChoiceDistance` equalling the +owner's directed frozen-topology hop distance for a realization binding and the +absent sentinel for every other kind; a `Placement` arc joining a placed +Realization node to its occurrence node in both directions and none joining an +unplaced one; route arcs being absent when `include_route_edges` is clear and +spanning exactly the current RouteTree traversals when it is set; +resource-state columns being absent when `include_resource_states` is clear; a +committed binding closing every incident net through the owner's own dependency +closure with no environment-issued routing Action; a net to an unplaced +realization remaining an ordinary candidate violation rather than a step +failure; construction advancing to the repair phase or to the cleanup exactly +when the last realization is placed and never before; a `ConstructThenAnneal` +episode running exactly one cleanup, reporting `CleanupComplete` on the same +step that placed the last realization, and returning the post-cleanup +observation; `cleanup_displacement` being zero exactly when no occurrence +changed, and counting both the hop sum and the moved count; a +`ConstructThenRepair` episode charging exactly one `repair_step_reward_code` +per committed repair step and none during construction; an unplaced realization +with no alternative to its current binding being settled without a step at +`reset` and again after a commit that shrinks its domain to a singleton, and +construction never reaching a state with realizations outstanding and no live +entry; an `IntrinsicInvalid` rollback leaving the candidate equal to its +pre-probe value in every selected decision and every rebuildable cache; +`IntrinsicInvalid` and `WorkLimit` never being collapsed into one reason; +`ObjectiveUnavailable` producing no numeric reward and rolling back its +transaction; a transition whose resulting enumeration exceeds +`enumeration_bound` failing with `EnumerationBoundExceeded` before the commit +and leaving the candidate unchanged; the enumeration length being identical +before and after the construction-to-repair phase advance while +`UnmaskedActionCount` rises while the enumeration's length does not; a masked, +stale, or out-of-range index being refused +with the candidate unchanged; the step accounting identity holding over a +complete episode with `Failed` as its non-advancing class; termination and +truncation being reported distinctly for their respective terminal reasons; a +truncation yielding zero for the ending itself while still charging +`incomplete_closure_reward_code` when its final candidate is unclosed, and +charging nothing when it is closed; the frozen model being reused across two +episodes drawing the same problem binding and not being reused under a changed +the freeze-relevant projection of `C` or a changed `K` identity, and being +reused across a changed `selected_objective_closure`; a warm-cache run and a +cold-cache run producing identical stream positions and identical trajectories; +`ProblemProvenInfeasible` and `FreezeCapacityExceeded` being reported +distinctly and neither being retried; a canonical candidate whose enumeration +exceeds `enumeration_bound` reporting `EnumerationBoundExceeded` with both the +capacity and the required length rather than `Invalid`; a `ConstructThenAnneal` +case under a start override annealing identically at two runner counts and two +copy coordinates; a settled realization whose domain regains an alternative +returning to unplaced and re-entering the enumeration; the observation returned +on a step that ends the episode carrying an empty `decisions` and an all-clear +mask, and no non-terminal state ever presenting one; a failed step leaving the +enumeration and the mask it had before the step rather than those of the +discarded shadow candidate; a `pnr_config_view` whose +`realization_move_radius` being adopted at either arm and changing no +enumeration this environment produces; an episode start override consulting no +`ProblemSelection` draw and +producing the same problem on every copy, and one naming a problem outside +`problem_pool` being rejected; two environment copies with distinct coordinates +drawing disjoint problem sequences from one seed; a `reset` seed argument +becoming the effective seed and being recorded as such; and equal +configuration, coordinates, seed, and action sequence reproducing an identical +trajectory, an identical final `CandidateState`, and an identical search +energy. + +Tests do not pin the live node, arc, or action counts of any particular +candidate, the bound or radius values a configuration selects, which occurrence +a policy chooses, achieved energies or closure rates, cleanup displacement +values, wall-time numbers, per-kind cost ratios, cache hit rates, corpus +contents, diagnostic text, or Python formatting. diff --git a/docs/spec-ml-pnr-model-architecture.md b/docs/spec-ml-pnr-model-architecture.md new file mode 100644 index 000000000..7cfca3f68 --- /dev/null +++ b/docs/spec-ml-pnr-model-architecture.md @@ -0,0 +1,368 @@ +# ML PnR Model Architecture + +This document defines the learned policy that consumes the ML PnR Environment's +observations and emits its actions. It owns the embedding of that environment's +column catalogs, the two-factor policy head that scores an action from the +nodes it names, the model's own configuration view, and the harness that +measures its forward pass. + +The model is a search policy on the terms +[ML Model Core](spec-ml-core-model.md) states. It proposes Actions; the +environment applies them through Place and Route's own `MoveTransaction`, and +every legality, closure, and objective fact remains owned where +[ML PnR Environment](spec-ml-pnr-environment.md) places it. + +Everything a Loom policy does that is not specific to what it is searching — +the batching, the embedding discipline, the graph-transformer trunk, the +context pooling, the value head, masking, the action distribution, and the +checkpoint boundary — is owned by [ML Model Core](spec-ml-core-model.md). This +document is the difference between that and a placer. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [ML Model Core](spec-ml-core-model.md) owns the module boundary, observation + batching, the feature-embedding discipline, the encoder, the graph context, + the value head, masking, the action distribution, the checkpoint boundary, + and the obligations every forward-pass harness satisfies; +- [Column Catalogs](spec-ml-pnr-environment.md#column-catalogs) owns + `PnrNodeFeatureColumn`, `PnrArcRole`, `PnrScalarFeatureColumn`, + `PnrDecisionColumn`, and `EpisodePhase`; +- [Action Space](spec-ml-pnr-environment.md#action-space) owns the enumeration, + its canonical order, `ActionIndex`, and which entries the phase mask clears; +- [The PnR Node And Link Space](spec-ml-pnr-environment.md#the-pnr-node-and-link-space) + owns the role blocks, the placement and route arcs, and the rule that an + action names its anchor and choice by column; +- [Reward](spec-ml-pnr-environment.md#reward) owns the reward the value head + regresses toward; +- [Enumeration Encoding](spec-ml-core-environment.md#enumeration-encoding) owns + the `DecisionColumns` form this model consumes; +- [Actions And MoveTransaction](spec-pnr.md#actions-and-movetransaction) owns + the Action kinds an entry's `ActionKind` column names; and +- [Resolved Configuration](spec-config-ssot.md#component-views) owns + component-view framing. + +This document owns only the PnR feature mapping, the policy head, the model +configuration view, and the forward benchmarking stages. + +## Module Boundary + +The model is `LoomPnrRLModule`, one `TorchRLModule` implementing +`ValueFunctionAPI` on the terms +[Module Boundary](spec-ml-core-model.md#module-boundary) states. It adds no +entry point and changes no signature. + +## Feature Embedding + +The core fixes how a column class is consumed; this section fixes which class +each PnR column belongs to and why, for the cases where the answer is not +mechanical. + +Embedded categorical columns: + +```text +column table size source catalog +Role GraphNodeRole cardinality environment core +EntityKind Fabric local-reference kinds plus + Dataflow and TechMapping kinds Fabric Identity, Dataflow +ActionKind SpatialMappingAction kind cardinality Place And Route +Phase EpisodePhase cardinality PnR environment +``` + +`Phase` is a graph-level categorical rather than a node column, and it is +embedded rather than projected for the ordinary reason: `Construction` and +`Repair` are two regimes, not two points on a scale, and the model's behaviour +in one should not be constrained to lie on a line through the other. + +Every `PnrNodeFeatureColumn` not named below is a projected numeric column with +the absence indicator the core requires. `CapacityMagnitude`, `CapacityUsage`, +and `CapacityOveruse` are projected with the occupancy ratio the core requires +alongside them, and the same treatment extends to each capacity and usage pair +the environment appends when `include_resource_states` is set. + +Two columns need a stated decision rather than a classification. + +`AnchorNode` and `ChoiceNode` are not features at all. They are indices, and +the head consumes them by gathering `h` at those ordinals; projecting an +ordinal would assert a distance between two unrelated nodes that happen to sort +near each other. Whether an entry's anchor is already placed is likewise not a +column: `Placed` on the anchor node is that bit, and the head already has the +anchor's embedding in hand. + +`ChoiceDistance` is the load-bearing feature of this model and gets its own +treatment. Hop counts near zero decide most placements and hop counts far from +zero are nearly interchangeable, so a single linear projection would spend its +resolution in the wrong place. Small distances are embedded from a table +indexed by exact hop count up to a configured bucket count, and distances +beyond it are projected through the core's signed logarithmic transform into a +shared remainder. The near range therefore keeps exact resolution while the far +range still generalizes across magnitudes, and the absent sentinel — every +action kind but a realization binding — takes the core's zero-plus-indicator +form rather than a distance of zero, which would otherwise read as "adjacent". + +Scalar features are graph-level, with the ratios the core requires supplied +alongside the magnitudes: `StepOrdinal` against `StepBound`, `PlacedCount` +against `RealizationCount`, and `RepairStepCount` against `RepairStepBound`. +`UnmaskedActionCount` and `ConsecutiveFailures` are supplied as magnitudes. +`PlacedCount` over `RealizationCount` is how far construction has progressed, +which is the quantity that most changes what a good action looks like, so +requiring the model to divide two magnitudes to recover it would be a +gratuitous obstacle. + +One absence is a decision rather than an oversight. The model receives +`CapabilityCount` for an occurrence but not the identity of the operations that +occurrence supports. The prior architecture embedded a per-node instruction +bitmask because its candidate mask was computed separately and the network had +to learn which unit could host which operation. Here the enumeration is derived +from `CandidateDomain`, so an occurrence that cannot host a realization is +never offered as a choice and compatibility is never a question the policy has +to answer. What remains useful is scarcity — whether an occurrence is a +specialist worth saving for something else — and a count carries that. A +capability-set feature would be paying for a distinction the mask already made. + +## Encoder + +The encoder is the core's `GraphTransformerStack`, unchanged, with +`PnrArcRole` supplying edge features. Its three roles are the three +relations a placement policy has to tell apart: `Structural` is the fabric's +own connectivity and the software graph's dependencies, `Placement` is the +current mapping, and `Route` is where the nets currently run. + +`Route` arcs are what make the trunk expensive: the optional projection +[Observation](spec-ml-pnr-environment.md#observation) governs can dominate arc +count, and therefore attention cost, when it is enabled. + +## Graph Context And Value Head + +Both are the core's, unchanged. The value head reads the pooled context and +emits one bounded scalar per instance, and its bound is checked against the +range [Online Training](spec-ml-pnr-training.md#online-training) declares +rather than chosen. + +## Policy Head + +An action names two nodes and carries its own kind and distance, so the head +scores it from those directly. Nothing is reconstructed from an index and no +action has an embedding of its own. + +```text +anchor(a) = ( AnchorNode of a, ActionKind of a ) + +s_anchor(a) = MlpAnchor([ h[AnchorNode(a)], e_kind(a), context ]) +s_choice(a) = MlpChoice([ h[AnchorNode(a)], h[ChoiceNode(a)], + e_distance(a), e_kind(a), context ]) +logit(a) = s_anchor(a) + s_choice(a) +``` + +`h` is the shared trunk's node embedding, indexed by the ordinal the entry's +column holds. This is the whole of "act on the predicted node": the score of +placing a realization on an occurrence is a function of that realization's +embedding and that occurrence's embedding, both produced by one pass over the +state. + +`s_choice` is this model's selection term, so actions sharing an anchor form +one group and compose on the terms +[Policy Head Factorization](spec-ml-core-model.md#policy-head-factorization) +states. + +The hierarchy is what this environment is shaped for. During construction every +entry of one unplaced realization is one group, so the anchor factor is "which +realization to place next" and the choice factor is "where" — exactly the +decomposition the search actually makes. Under `CanonicalNext` anchor selection +the mask leaves one group live and the anchor factor degenerates to a constant, +which is the correct behaviour rather than a special case: the environment +already made that choice, so the policy has nothing to contribute to it. + +### Conditioning The Choice On The Anchor + +`s_choice` reads the anchor's embedding as well as the choice's, and that is a +deliberate departure from the architecture this model translates. The prior +`SwapSchedulerModel` scored hardware slots from the slot embedding and the +graph context alone, conditioning on the selected software node only through +the mask. It could therefore learn which slots are good in general — central, +uncongested, well connected — but not which slot suits *this* node, which is +most of the placement problem: a producer wants to be near its consumers, and +which occurrence satisfies that is a property of the pair. + +The cost is that `s_choice` is evaluated once per live action rather than once +per node, and this document is precise about which part of that cost is +recoverable. + +The prior architecture's `_SplitScorer` factoring is admitted and extended. A +scorer of the form `out(activation(W_1 x_1 + ... + W_k x_k))` is exactly the +scorer `out(activation(W · concat(x_1, ..., x_k)))` with `W` partitioned, so +`MlpAnchor` and `MlpChoice` may be evaluated in the factored form. What that +buys is memory: the concatenation is never materialized and never saved for the +backward pass, which matters at live-action counts in the hundreds of +thousands. What it does not buy is arithmetic, because the activation and the +output projection are still evaluated per action. A specification that claimed +otherwise would be promising a saving the algebra does not contain. + +A genuine per-action reduction requires a scorer whose pair term is bilinear — +an inner product between one projection of the anchor and one of the choice, so +each side is projected once per node and the pair costs a dot product. That +computes a different function, with strictly less expressive pair interaction +than an activation over the sum. It is permitted as a configured variant and is +validated against the reference form on the same inputs and parameters like any +other optimization; it is not presented as an equivalent implementation of the +same head. + +### Stop + +The environment masks the core's stop logit for the whole construction phase +and for the whole of a `ConstructThenAnneal` episode, and the head applies that +mask like any other: it never derives its own stop admissibility from the phase +feature, because the mask is the authority and two sources for one fact +eventually disagree. + +The joint distribution over `enumeration_bound + 1` outcomes and its stop +encoding are the core's. + +## Masking + +Masking is the core's contract. Two properties of this environment are worth +stating because they make it easier here than in the design-space environment. + +The environment's mask is a phase rule rather than a rejection record. Every +entry it clears is a legal Action of the current candidate, cleared because the +phase does not admit it, and the same entry becomes live again when the phase +changes. Nothing in a PnR episode is masked because it was tried and failed, so +no mask bit depends on episode history. + +Anchor-group consistency is automatic. Construction clears every entry of an +already-placed realization, which is precisely a whole anchor group, and clears +whole kinds rather than individual choices. The core's rule that a group is +masked exactly when all its members are masked therefore holds without the head +arranging for it, and a partially masked group only arises from an advisory +mask. + +## Model Configuration + +The model's shape is one immutable component view with schema descriptor bytes +`loom.ml_pnr_model.config.1.0`, following the framing owned by +[Component Views](spec-config-ssot.md#component-views) and the three properties +[Model Configuration Framing](spec-ml-core-model.md#model-configuration-framing) +fixes: + +```text +ResolvedMlPnrModelConfigView { + embedding_widths: total table + distance_bucket_count: positive uint32 + encoder: GraphTransformerStack + head_widths: PnrHeadWidths + choice_scorer: Factored | Bilinear + value_bound: positive uint64 + advisory_mask_set: canonical set + initialization: InitializationConstants +} + +PnrHeadWidths { + anchor_hidden: positive uint32 + choice_hidden: positive uint32 + value_hidden: positive uint32 +} +``` + +`distance_bucket_count` is the exact-hop table height Feature Embedding +describes; it sizes a parameter tensor, which is why it belongs in this view +and not in a training one. `choice_scorer` selects between the two forms +Conditioning The Choice On The Anchor defines, and it is a view field rather +than an implementation switch because `Bilinear` changes the function and +therefore the parameter shapes, so a checkpoint is correctly incompatible +across it. + +## Forward Benchmarking + +This model ships `loom-pnr-model-bench`, on the obligations +[Forward Benchmarking Obligations](spec-ml-core-model.md#forward-benchmarking-obligations) +states. + +The harness decomposes one forward pass into these stages: + +```text +gather assemble the batched graph and decision columns, move to device +embed embedding lookups, numeric projection, input projection +encode the graph-transformer stack +pool per-instance multi-scale context +anchor anchor scoring, once per anchor group +choice choice scoring, once per live action +mask mask composition and additive application +distribute distribution construction and sampling +value the value head, when the entry point computes it +``` + +`anchor` and `choice` are separate stages, where a model whose value is a +catalog ordinal can report one policy stage. They scale with different +quantities — anchor-group count and live-action count — and the whole argument +of Conditioning The Choice On The Anchor is that the second is the price of the +first being worth having. A blended policy stage would make it impossible to +say whether that price is what it was expected to be, or whether the factored +form is doing anything. + +`gather` is reported with the decision columns separated from the graph, since +their row count is the live-action count and the graph's is the state size, and +those move independently: a repair-phase state has a stable graph and a much +larger enumeration than a late-construction one. + +Beyond the core's required measures, three breakdowns are required here. + +Every measure is reported per phase, on the terms +[Benchmarking](spec-ml-pnr-environment.md#benchmarking) states for the +environment harness. What differs here is that a construction and a repair +forward pass differ in live-action count as well as in route density, so the +scaling below separates on both. + +Scaling is reported against node count, arc count, live-action count, +anchor-group count, and the mean anchor-group size. The last is what makes the +anchor-group saving +[Policy Head Factorization](spec-ml-core-model.md#policy-head-factorization) +describes legible. + +The `choice_scorer` variants are measured against each other and against the +reference form. `Bilinear` is admitted for speed and computes a different +function, so its measurement reports both its stage cost and its divergence +from `Factored` on the same inputs, rather than reporting a latency alone and +leaving the quality question to a separate exercise. + +The rollout split against the environment step is reported first, as the core +requires, and it matters more here than for a design-space policy: a PnR +environment step contains a Mapping probe with incident route closure, which is +plausibly an order of magnitude more expensive than this forward pass. A model +optimization that halves an already-small share buys nothing, and the split is +what says so before the work is done rather than after. + +## Conformance Anchors + +Stable tests cover a `Placement` arc reaching the encoder as an edge feature +and no node column restating that relation; `ChoiceDistance` at a hop count +inside the bucket range reaching an exact table row and one beyond it reaching +the projected remainder, and the absent sentinel producing the core's +zero-plus-indicator rather than the encoding of distance zero; two actions +sharing an anchor node and kind receiving an identical `s_anchor`; two choices +on one anchor receiving different scores; an action whose anchor and choice +embeddings are swapped receiving a different score, so the pair term is not +symmetric; the factored scorer reproducing the concatenated reference exactly; +the `Bilinear` variant being reported with its divergence from `Factored` +rather than as an equivalent; scoring restricted to live actions producing +logits identical to scoring every slot; a `CanonicalNext` configuration leaving +exactly one live anchor group and its anchor factor contributing no gradient to +the choice among realizations; a masked anchor group receiving no probability +and contributing no entropy; construction clearing whole anchor groups so that +no partially masked group arises without an advisory mask; the stop logit being +masked throughout construction and throughout a `ConstructThenAnneal` episode, +and the head deriving stop admissibility from the mask rather than from the +phase feature; the two-level and flat formulations producing identical +distributions to numerical tolerance; `_forward_train` and +`_forward_exploration` producing identical logits for equal parameters, +observations, and masks; `_forward_inference` computing no value; +`compute_values` producing identical predictions with and without precomputed +embeddings; a decision-column node ordinal being offset into the batched node +union before use, so an action never scores another instance's nodes; and a +checkpoint load failing across a changed `distance_bucket_count` or a changed +`choice_scorer`. + +Tests do not pin layer counts, widths, head counts, the bucket count, the +scorer variant, initialization constants, the advisory mask set, learned +parameter values, wall-time numbers, per-phase cost ratios, device placement, +precision configurations, or tolerance values. diff --git a/docs/spec-ml-pnr-training.md b/docs/spec-ml-pnr-training.md new file mode 100644 index 000000000..d1723ba54 --- /dev/null +++ b/docs/spec-ml-pnr-training.md @@ -0,0 +1,527 @@ +# ML PnR Training + +This document defines how the place-and-route policy of +[ML PnR Model Architecture](spec-ml-pnr-model-architecture.md) is fitted +against the environment of [ML PnR Environment](spec-ml-pnr-environment.md). It +owns the demonstration corpus that pretraining imitates, the two-stage run that +turns those demonstrations into an online policy, this environment's +return-range arithmetic and statistics, and the test protocol that says whether +the run improved anything. + +A run has two stages and they answer different questions. The first learns +where a good placement puts things, from demonstrations a simulated annealer +produced. The second learns to do better than that, from its own experience. +The whole point of their being one run rather than two is that the second +begins from the first's parameters. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [ML Training Core](spec-ml-core-training.md) owns everything a Loom training + run does that is not specific to what is being searched: the nonsemantic + boundary, the configuration split and inventory identity, the algorithm + binding surface, the stage sequence and its handoff, exact-rational + hyperparameters, schedules, the reward adapter, topology, the stage + invariance rule, the statistics production rule, the test protocol, + checkpoints and run identity, reproduction, and the harness surface; +- [ML PnR Environment](spec-ml-pnr-environment.md) owns the episode arms, the + action space and its phase mask, the observation, the reward and every code + it charges, `PnrTerminalReason`, `PnrTransitionFailureReason`, + its episode start override payload, `PnrTrajectory`, and + `ResolvedMlPnrEnvironmentConfigView`; +- [Model Configuration](spec-ml-pnr-model-architecture.md#model-configuration) + owns `ResolvedMlPnrModelConfigView` and the tensors it sizes; +- [Annealing And Replay](spec-pnr.md#annealing-and-replay) owns the annealing + policy, its acceptance kernel, and the seeded PRNG protocol the demonstration + generator draws from; +- [Deterministic Initialization And Action Proposal](spec-pnr.md#deterministic-initialization-and-action-proposal) + owns the Action proposal selector and the frozen-topology hop distance + `PlacementLocality` orders by; +- [Objectives and Quality Gates](spec-dse-feedback.md#objectives-and-quality-gates) + owns `ExactAffineQuantization` and the search energy the return range is + derived from; and +- [Component Views](spec-config-ssot.md#component-views) owns view framing, + canonical bytes, and `component_view_digest`. + +This document owns only the demonstration corpus and its generator, the PnR +stage configuration, this environment's return-range arithmetic, its statistics +catalog and breakdown axes, its test set and comparison rules, and its +harnesses. + +## Configuration Split + +This run's views, under the framing +[Configuration Split](spec-ml-core-training.md#configuration-split) owns: + +```text +loom.ml_pnr_environment.config.1.0 ResolvedMlPnrEnvironmentConfigView +loom.ml_pnr_model.config.1.0 ResolvedMlPnrModelConfigView +loom.ml_pnr_training.config.1.0 ResolvedMlPnrTrainingConfigView +loom.ml_pnr_demonstrations.config.1.0 ResolvedMlPnrDemonstrationConfigView +loom.ml_pnr_test_set.config.1.0 ResolvedMlPnrTestSetConfigView +loom.ml_pnr_run.config.1.0 ResolvedMlPnrRunConfigView +``` + +```text +ResolvedMlPnrRunConfigView { + training_config_digest: ComponentViewDigest + model_config_digest: ComponentViewDigest + demonstration_config_digest: ComponentViewDigest + test_set_config_digest: ComponentViewDigest +} +``` + +`ResolvedMlPnrTrainingConfigView` is the core's `ResolvedTrainingConfigView` +under this descriptor, with no field added. + +There is no corpus view, and its absence is the substantive difference from the +design-space run. That run generates its training instances, so it needs an +architecture corpus, a synthetic workload corpus, structural axes, and a +calibration record to argue that what it generated resembles what it will meet. +A PnR run does not generate anything: its instances are exact +`SpatialPnrProblemBinding` values naming Artifacts that already exist, and a +problem is a problem whether or not anyone calibrated it. What the +demonstration view carries is not a corpus of instances but a corpus of +*placements* for instances the configuration already names. + +Adoption validates each view, then three cross-view conditions: every problem +the demonstration view names appears in the training stages' bound environment +views; the test set's problems appear in its own environment view and in no +training stage's; and the model view's `value_bound` covers the return range +below, maximized over every bound stage. + +## Demonstration Corpus + +Pretraining imitates simulated annealing. What it imitates is where the +annealer *arrived*, not how it got there, and that distinction is the whole +design of this section. + +### What Is Imitated + +An annealing run is a Metropolis walk. Its intermediate states are bad by +construction — that is what the acceptance kernel is for, and a run that never +occupied a worse state was not annealing. A demonstration recording that walk +teaches a policy to reproduce it: to place badly, then move, then move again. +The prior system recorded exactly that, a greedy fill followed by every +accepted swap, and it was imitating a search rather than a solution. + +So the generator keeps the annealer's final placement and discards its path. +The demonstration is a construction sequence that reaches that placement +directly: each realization bound once, to the occurrence the annealer left it +on, in one sweep with no move ever undone. + +### The Generator + +For each problem in the view, under streams derived from `generator_seed` by +the protocol +[Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism) +owns: + +```text +1. reset the environment on that problem +2. step it with a scripted annealer, which selects each Action through the + proposal selector and acceptance kernel Place and Route already owns, + under `annealing_policy`, until that policy's schedule completes +3. read the final placement: each Compute and Memory Realization, and the + occurrence it is bound to +4. discard the action sequence entirely +5. reset again, and emit one `RealizationBindingAction` per realization, in + `demonstration_order`, binding each directly to its final occurrence +6. record the placement of step 3, the sweep's length, and the energy and + closure the sweep ended at +``` + +The scripted annealer is not a second search authority. It selects Actions with +the owner's own proposal and acceptance protocols, and every Action it takes is +an ordinary environment step, so every recorded index is by construction a +legal `ActionIndex` under the phase mask that produced it. Nothing here needs a +recording hook inside Place and Route, and nothing depends on a diagnostic +stream. + +`demonstration_order` is a real parameter rather than a detail. Under +`AgentSelected` anchor selection the policy learns which realization to place +next as well as where, so the order a demonstration uses is part of what it +teaches. `CanonicalDecisionKey` uses the owner's canonical typed decision-key +order and is the honest default. `PlacementLocality` orders each realization +after the already-placed neighbours it shares a net with, nearest first by the +frozen-topology hop distance, which is the order a person would place by hand +and the one worth measuring the default against. + +### What The Record Carries + +```text +ResolvedMlPnrDemonstrationConfigView { + problems: canonical nonempty set + generator: DemonstrationGenerationRecord + admitted: canonical nonempty set +} + +DemonstrationGenerationRecord { + generator_seed: u64 + prng_protocol: Sha256SeededXoshiro256StarStar_1_0 + annealing_policy: SearchPolicy.annealing + demonstration_order: CanonicalDecisionKey | PlacementLocality + pnr_config_view_digest: ComponentViewDigest +} + +DemonstrationRecord { + problem: SpatialPnrProblemBinding + final_placement: ordered sequence + action_count: uint64 + replayed_final_energy_code: uint64 + replayed_closed: bool +} +``` + +`final_placement` is the annealer's result and the only thing about the +annealing run this document keeps: the occurrence each Compute and Memory +Realization ended bound to, in the owner's canonical decision-key order, so its +length is the realization count and its meaning needs no second field to say +which entry is which. A `PlacedOccurrence` is whichever occurrence reference +that realization's binding relation targets — a `FabricFuOccurrenceRef` for a +Compute Realization and a `FabricMemoryOccurrenceRef` for a Memory one — so an +entry is exactly what the `RealizationBindingAction` the sweep emits carries, +and no new reference kind appears here. + +It is a field rather than something re-derived because it is the one input to +a demonstration that nothing else determines. Given it, the recorded sweep is a +pure function of the problem, the order, and the environment view, so replay +runs no annealer at all. Without it, the only path back to the same placement +is re-running the annealing pass, which this document elsewhere prices as the +dominant cost in this environment — and adoption, every read of the offline +stage, and every verification would each pay it again for a result already +computed once. + +`replayed_final_energy_code` is the energy the second pass produced, and it is +the authority. It is not the energy the annealing run reported, and the two +differ: placing realizations in a different order routes their nets +differently, so the same final placement carries a different routing and a +different energy. Recording the annealer's number would describe a state the +demonstration never occupies. + +A record carries no observation and no action sequence. Both are regenerated by +sweeping `final_placement` in `demonstration_order` against the environment +view, which is deterministic and involves no search — so the view stays a +configuration rather than becoming a dataset, and a demonstration set is a +value that can be compared by digest. A placement vector is bounded by the +realization count and does not make it one. + +### Admission + +A demonstration is admitted only if it is something the online stage could have +produced. Three conditions, each a check on the record rather than a run of +it: + +Its `action_count` is within the `step_bound` every training stage's +environment view declares. The prior system did not check this, and its offline +episodes ran to lengths the online policy was never permitted to reach, so +pretraining taught a behaviour the environment then forbade. + +Its `final_placement` binds every realization exactly once: its length is the +problem's realization count and no anchor repeats. That is what the +"every action is unmasked" condition reduces to here — the construction mask +admits realization bindings on unplaced anchors, so a sweep that never places +one twice is unmasked at every step by construction, and a repeated anchor is +the only way a configured `demonstration_order` could break it. The same length +check establishes that no realization is left unplaced. + +Adoption therefore performs no sweep. Both structural conditions are decidable +from the placement vector directly, and running a construction pass per +demonstration to rediscover them would make every adoption — and every restart, +and every sweep iteration that re-adopts — pay one probe with route closure per +realization for facts already in the record. Checking that the recorded energy +and closure are the ones a sweep actually produces is +[Harnesses](#harnesses) work, done once against a corpus rather than at every +run that binds it. + +A demonstration covers construction and nothing else. Under +`ConstructThenAnneal` that is the whole episode, so pretraining sees complete +episodes. Under `ConstructThenRepair` the repair phase has no demonstration at +all, and learning it is the online stage's job — which is the division of +labour the two stages exist for, not a gap in the corpus. + +### Determinism + +The demonstration set is the inventory this run carries, enumerated and +regenerable on the terms +[Inventory Identity](spec-ml-core-training.md#inventory-identity) sets. What +discharges the regeneration half here is that the same `generator_seed`, +`annealing_policy`, `demonstration_order`, and `pnr_config_view_digest` +reproduce the same admitted set exactly: every draw the generator makes comes +from a stream the protocol derives. The prior generator failed all of this at +once — it +shuffled with an unseeded generator, it passed its seed to a flag the scheduler +did not read for scheduling so every retry re-ran an identical stream, and its +output layout varied with task completion order — and the resulting corpus +could not be rebuilt or compared. + +## Stages + +The default run is two stages: + +```text +stage 0 Marwil advancing AtPlateau +stage 1 Ppo final, carrying no advance +``` + +Both are the core's `TrainingStage`, and the handoff is the core's: stage 1 +begins from stage 0's parameters and, the arm having changed, from no algorithm +state at all. The demonstrations stage 0 reads are the ones +`ResolvedMlPnrRunConfigView.demonstration_config_digest` binds; the binding +names no corpus of its own, so there is exactly one place a run says which +placements it pretrains on. + +`gamma` must agree across the two stages, and adoption checks it. The reward is +an exact energy difference, so a discount is not a tuning knob but a statement +about which future improvements count; a pretraining stage that discounted +differently would fit a value head to a different return than the one the +online stage optimizes, and the transferred head would be biased in a direction +no diagnostic attributes to the handoff. + +A single-stage run is legal and is how an ablation isolates either half: stage +0 alone measures what the demonstrations contain, and stage 1 alone measures +what online training reaches without them. The pair is the claim; the singles +are the evidence for it. + +### Pretraining + +The `Marwil` stage reads its demonstrations by sweeping each admitted record's +`final_placement` through the environment, which yields observation, action, +reward, and termination per step at the cost of one construction pass and no +search. A demonstration sample carries **no behaviour-policy +log-probability**, and none is required: the algorithm applies no importance +correction, and a synthesized placement sweep has no behaviour policy that +could report one. A pipeline that fabricates a log-probability to satisfy a +schema is reporting a number about a policy that never acted. + +`beta` is stated against what the demonstration set contains. When every +demonstration is a completed annealing placement, returns vary little across +the set, the advantage weighting flattens toward one, and the stage is +behaviour cloning whatever `beta` nominally says. A configuration that wants +weighting to do work needs demonstrations of visibly different quality — a set +generated under several annealing policies, or one that admits partial-quality +placements — and this document requires the choice to be deliberate rather than +inherited. + +### Online Training + +The `Ppo` stage is the core's binding, over the environment views the stages +declare. Its return range is this environment's: + +```text +step_bound * max(energy_width + repair_step_reward_code, + failed_transition_reward_code) + + cleanup_displacement_reward_code * max_displacement + + incomplete_closure_reward_code +``` + +`energy_width` is the quantization width of the selected search energy. +`repair_step_reward_code` participates only under `ConstructThenRepair` and +`cleanup_displacement_reward_code` only under `ConstructThenAnneal`, and the +maximum is taken over whichever arm each bound stage declares. + +The repair charge is added to the energy width rather than maximized against +it, and that is the difference from the design-space arithmetic worth naming. A +rejected step there carries a code and no energy delta, because nothing was +committed, so the two are alternatives and a maximum is exact. A committed +repair step here carries both: the transition's own energy delta *and* the +per-repair charge. Maximizing them would understate every repair step by +whichever term is smaller, and a `value_bound` checked against that +understatement clips the returns of exactly the arm it was computed for. + +`max_displacement` is the one term that depends on the problem rather than the +configuration. A cleanup moves each realization at most `r` hops, where +`Bounded(r)` is the arm's `cleanup_annealing_policy.realization_move_radius`, +and [Annealing And Replay](spec-pnr.md#annealing-and-replay) anchors +displacement at the run-start occurrence, so displacement is bounded by the +realization count times `r` plus the moved count, itself bounded by the +realization count. Only a `ConstructThenAnneal` arm has a cleanup, and that arm +is required to carry `Bounded`, so the term always has a radius to read and +contributes nothing under the other arm — the same condition +`cleanup_displacement_reward_code` already carries. + +Both quantities are available without a frozen model: the realization count is +how many Compute and Memory Realizations the TechMapping the problem binds +declares, and `r` is a configuration field. That matters because freezing every +pool member to obtain an integer already present in `T` would make adoption pay +this environment's dominant cost for nothing. The bound is computed per problem +and the run's requirement is the +maximum over the pool. That a term is instance-dependent is exactly why +[The Reachable Return Range](spec-ml-core-training.md#the-reachable-return-range) +leaves the arithmetic to a training document rather than fixing a formula. + +## Statistics + +The production rule, the objective-dimension record, the outcome accounting, +the cost statistics, and chunk invariance are the core's. This environment +names its own vocabulary and adds what only it has. + +Its non-advancing step class is `failed`, so the core's accounting identity +reads `steps == advanced + failed + elective_stops`, and the failure rate is +reported per member of `PnrTransitionFailureReason`. Keeping `IntrinsicInvalid` +apart from `WorkLimit` matters here more than a blended rate would suggest: one +says the policy proposed something that cannot produce a legal candidate and +the other says the router ran out of budget, and a rising second with a flat +first is a configuration problem that a blended rate would present as a policy +problem. + +Its action partition is the `SpatialMappingAction` kind, so action frequencies +and their success rates are keyed by kind. Under construction only one kind is +live, so the partition is informative exactly in the repair phase — which is +where the question "is the policy doing anything but rebinding" is worth +asking. + +Four statistics are this environment's own: + +```text +placed_fraction_at_end placed realizations over realization count +closure_rate episodes whose final candidate closed +cleanup_displacement ConstructThenAnneal only +repair_steps_taken ConstructThenRepair only +``` + +`cleanup_displacement` is the arm-A quantity the reward already prices, +reported directly because it is the measure of what construction left undone. A +run whose energy improves while its displacement also rises is not improving +construction; it is leaning harder on the annealer, and only the pair +distinguishes the two. + +```text +PnrBreakdownAxis = + Stage // 0, mandatory + | TerminalReason // 1, mandatory + | Problem // 2 + | EpisodeArm // 3 + | ActionKind // 4 +``` + +`Problem` is the axis that answers whether a run improved everywhere or only on +the instances it saw most, and it is the expensive one: it multiplies every +statistic by the pool size, so it is opt-in like every other axis. + +## Test Protocol + +`ResolvedMlPnrTestSetConfigView` is the core's `ResolvedTestSetConfigView` +under this descriptor, and this document supplies only its case payload: + +```text +PnrTestCase { + instance: SpatialPnrProblemBinding + case_seed: u64 +} +``` + +A `SpatialPnrProblemBinding` names a Canonical Dataflow Program, a TechMapping, +a Fabric, and a MappingConstraintSet, so a test set is a grid over exactly the +two things a run wants to hold fixed and vary: the workloads, which vary `D`, +and the hardware versions, which vary `F`. Both are named rather than drawn, so +the set is the same set at every evaluation. + +### Results And Comparison + +```text +PnrTestCaseResult { + outcome: TestCaseOutcome + final_energy_code: uint64 +} +``` + +`final_energy_code` is the only quantity a case adds. Everything else worth +reading about the episode — its length, its terminal reason, its closure, its +displacement — is already inside the `Completed` arm of `outcome`, which +carries this environment's episode statistics catalog. Repeating any of it here +would be two sources for one fact, multiplied by every case and every +evaluation the run retains. + +The run retains one series per case, appending a result at each evaluation. +What makes a series a measurement rather than a log is what it is compared +against, and here that is the run's own history: an evaluation reports each +case's result together with its difference from the same case at the previous +evaluation and at the most recent stage boundary. + +The stage-boundary reading is the one that carries the argument. It is what the +pretrained policy achieved before online training touched it, so the difference +against it is exactly the question the two-stage design asks — whether PPO +improved on what the demonstrations delivered. The previous-evaluation +difference is the ordinary progress signal. + +Improvement over iterations is therefore a trend in a per-case series against a +fixed reference, not a movement in an aggregate. An aggregate over cases hides +the case that regressed, and a policy that improves its mean while losing its +hardest problems is the failure this reporting exists to catch. + +Two comparability rules, because a series only means something if its terms +measure the same thing. Results compare only across evaluations sharing both a +test-set digest and an environment view digest. And a stage advance that +changes either starts a new series rather than extending the old one, with the +new series' first reading becoming its own reference. + +There is deliberately no annealer baseline in these numbers. Comparing a policy +to the annealer that produced its demonstrations is a real question, but it is +a question about a finished checkpoint rather than about a run in progress, and +answering it during training would require a full annealing invocation per case +per evaluation — which is the dominant cost in this environment and would make +evaluation cost more than training. A finished comparison runs offline against +the results `loom-pnr-train` recorded. + +## Harnesses + +```text +loom-pnr-demos generate, extend, and verify a demonstration view +loom-pnr-train run training against a run view +loom-pnr-test run a test set against a checkpoint +``` + +Their outputs are removable projections, on the terms +[Harnesses](spec-ml-core-training.md#harnesses) states. + +`loom-pnr-demos` verifies an existing set without regenerating it, which +`final_placement` is what makes possible: every named problem resolves, every +admitted record's sweep reproduces its recorded energy and closure, and every +record satisfies the three admission conditions against the environment views +the run binds. No annealing runs. That last check is what catches a +demonstration set that was valid for the run that produced it and is not valid +for the run about to use it. + +## Conformance Anchors + +Stable tests cover a demonstration whose recorded energy is the replay's rather +than the annealing run's, and the two differing when `demonstration_order` +changes; a demonstration containing exactly one binding per realization, in +`demonstration_order`, with no realization bound twice and none left unplaced; +every demonstration action being unmasked at the step it is taken; a +demonstration whose `action_count` exceeds a bound stage's `step_bound` being +refused at adoption; a demonstration set regenerated from the same seed, +policy, order, and Place and Route view digest reproducing the same admitted +set; a record's sweep and its admission checks running without any annealing +invocation; the scripted +annealer selecting every Action through the owner's proposal and acceptance +protocols and consuming no host entropy; a `Marwil` stage requiring no +behaviour-policy log-probability and rejecting a batch that fabricates one; the +two stages being rejected at adoption when their `gamma` differs; a stage-1 +evaluation at the boundary reporting the parameters stage 0 produced; a +single-stage run of either algorithm being legal; the return range accounting +for `step_bound` occurrences of the most expensive per-step charge rather than +one, including its arm-conditional terms, a committed repair step counting its +energy delta and its repair charge together rather than the larger of the two, +and its `max_displacement` term being the maximum over the problem pool rather +than a configured constant and being computed without freezing any problem; a +`value_bound` below that maximum being refused; +the failure rate being reported per `PnrTransitionFailureReason` with +`IntrinsicInvalid` and `WorkLimit` never blended; action frequencies keyed by +`SpatialMappingAction` kind; `cleanup_displacement` reported only under +`ConstructThenAnneal` and `repair_steps_taken` only under +`ConstructThenRepair`; a test case naming a problem that appears in no training +stage's environment view; a result series comparing only across evaluations +sharing a test-set digest and an environment view digest, and a stage advance +that changes either starting a new series; each evaluation reporting a per-case +difference against both the previous evaluation and the most recent stage +boundary; a case result carrying no quantity its `outcome` already carries; and +`loom-pnr-demos` refusing a demonstration set that is valid for its generating +run and invalid for the run binding it. + +Tests do not pin `beta`, annealing policy values, demonstration counts, the +chosen `demonstration_order`, plateau tolerances, stage lengths, the contents +of any particular demonstration set or test set, achieved energies, closure +rates, displacement values, wall-time numbers, or diagnostic text. diff --git a/docs/spec-pnr.md b/docs/spec-pnr.md index 1371d6733..44fde0ce8 100644 --- a/docs/spec-pnr.md +++ b/docs/spec-pnr.md @@ -2155,6 +2155,7 @@ SearchPolicy { cooling_ratio: ExactRatio proposals_per_level_base: uint64 proposals_per_movable_decision: uint64 + realization_move_radius: RealizationMoveRadius } focused_closure { proposal_limit: positive uint64 @@ -2210,7 +2211,8 @@ PnR-kernel defaults. All initial builtin profiles use Action weights `1:3:2` for realization, routing, and resource Actions; PathFinder `Multiplicative` with initial pressure `1`, reduced growth ratio `3/2`, and history increment `1`; quantile `3/4`; target acceptance `4/5`; fallback -temperature `1024`; minimum temperature `1`; cooling ratio `19/20`; no route +temperature `1024`; minimum temperature `1`; cooling ratio `19/20`; +`realization_move_radius` `Unbounded`; no route guidance; and deterministic master seed `0` with the protocols named above. ```text @@ -2537,9 +2539,61 @@ SearchPolicy.annealing { cooling_ratio proposals_per_level_base proposals_per_movable_decision + realization_move_radius } + +RealizationMoveRadius = + Unbounded // 0 + | Bounded(positive uint64) ``` +`realization_move_radius` bounds how far one `RealizationBindingAction` may +move an occurrence from where that realization was bound when the annealing run +started. `Unbounded` is the value every builtin profile selects. It is a union +rather than an integer with a reserved value because zero is not a bound of +zero hops, and spelling it as one would let arithmetic over the field compute a +displacement bound of zero for a run with no bound at all. + +`Bounded(r)` removes from that run's proposal domain every realization-binding +choice whose occurrence exceeds `r` directed frozen-topology hops from that +realization's run-start occurrence, using the same immutable endpoint, arc, +payload-width, and attachment domains the initializer's preference refinement +already measures distance over, so no second topology or router model appears. + +The bound is a search preference and not a legality authority. It removes +proposals, never candidate legality: a choice it excludes remains a legal +Mapping decision, and an empty proposal neighborhood is not `ProvenInfeasible`. +It applies only to realization-binding proposals; routing and resource Actions +are untouched. + +The bound forms a proposal domain rather than filtering a draw taken from one. +An annealing run's proposal domain is the projection of `A(M,C,S)` this policy +selects from; `A(M,C,S)` itself is what the candidate admits and the radius +does not narrow it. Within that projection a realization anchor whose every +alternative the radius excludes is not present, the realization-binding kind is +not live when no such anchor remains, and every `nextBounded` call still ranges +over a nonempty canonical domain. Filtering after the draw was rejected because +it would leave empty anchors in the projection, which the Action proposal +contract forbids and which `nextBounded` cannot be called over. + +A selector that is not this policy — one that enumerates `A(M,C,S)` directly +rather than drawing a proposal from it — is unaffected by the radius under +either arm, because the projection is formed by the run that reads the field +and not by the candidate. + +A `Bounded` radius therefore lowers `movableDecisionCount` and shortens a +proposal level. That is a real consequence of the bound and not a defect: a +level's length is proportional to the decisions the level can actually move, +and a run that may move fewer of them has less to propose. The level length is +still fixed once at level start from the domain rebuilt there, so no +proposal-local rebuild changes it mid-level. + +The run-start occurrence is fixed once, at the start of the annealing run that +reads the policy, and is not re-anchored per temperature level or per accepted +move. Re-anchoring would let a sequence of within-radius moves drift +arbitrarily far, which is exactly the property a caller selecting a radius is +buying against. + Fractions use canonical integer or fixed-point values. Quantile is in `[0,1]`, target acceptance is in `(0,1)`, temperatures are positive, and the reduced cooling ratio is strictly between zero and one. Per-level proposal count is: @@ -2551,8 +2605,11 @@ proposals_per_level_base ``` `movableDecisionCount` counts each canonical typed selected-decision anchor -whose current dynamic Action domain contains at least one legal alternative, -plus one routing decision for each residual logical net or service leg. +whose current proposal domain contains at least one legal alternative, plus one +routing decision for each residual logical net or service leg. It is the +proposal domain and not `A(M,C,S)` because the count exists to size a proposal +level, and a level that counted anchors this run may not propose would draw +proposals it must then discard. Under `Unbounded` the two coincide. `SingleSink`, `RootedSubtree`, `WitnessRegion`, and `Global` are neighborhoods over those routing decisions and do not add decisions. Choice cardinality does not multiply the count. The domain is rebuilt once at level start for this @@ -3035,6 +3092,16 @@ Tests protect semantic anchors rather than implementation shape: central `Promote` separation; replay-stable annealing; focused closure; and exact-repair taxonomy, proof-bearing statuses, solver-call budget, and lexicographically canonical extraction; +* `realization_move_radius` `Unbounded` admitting the complete + realization-binding proposal domain, a `Bounded` radius excluding exactly the + choices beyond it while leaving `A(M,C,S)`, `CandidateDomain`, routing and + resource proposals, final verification, and `K` admission unchanged, a + radius-emptied anchor being + absent from the dynamic domain rather than drawn and retried, the + realization-binding kind becoming not live when every such anchor is empty, + an empty neighborhood not being reported as `ProvenInfeasible`, and the + run-start occurrence not being re-anchored by an accepted move or a + temperature level; * shared objective dimensions, three-valued CNF truth, independent full `V/G` and `Q`, TotalOrdering versus SearchEnergy separation, base verification, and exact admission;