diff --git a/docs/rationales/README.md b/docs/rationales/README.md index 655949a84..194ae94f2 100644 --- a/docs/rationales/README.md +++ b/docs/rationales/README.md @@ -25,6 +25,7 @@ are not architecture facts and are not reproduced here. | [Fabric Memory And Resources](fabric-memory-and-resources.md) | [Fabric Memory](../spec-fabric-mem.md), [Fabric Resource Contract](../spec-fabric-resource-contract.md), [Fabric Boundary](../spec-fabric-boundary.md), [Fabric FIFO](../spec-fabric-fifo.md), [Fabric Switch](../spec-fabric-switch.md), [Fabric Instantiate](../spec-fabric-instantiate.md) | | [Mapping And PnR](mapping-and-pnr.md) | [Mapping Artifact](../spec-mapping-artifact.md), [Mapping Identity](../spec-mapping-identity.md), [Mapping Memory](../spec-mapping-memory.md), [Mapping Verification](../spec-mapping-verification.md), [TechMapping Generation](../spec-tech-mapping.md), [Place And Route](../spec-pnr.md) | | [Evaluation And DSE](evaluation-and-dse.md) | [Evaluation And DSE](../spec-dse-feedback.md), [Evaluation Metrics](../spec-evaluation-metrics.md), [External Tool Invocation](../spec-external-tool-invocation.md), [FPA Evaluation](../spec-fpa-estimation.md) | +| [ML Search](ml.md) | [ML Environment Core](../spec-ml-core-environment.md), [ML Model Core](../spec-ml-core-model.md), [ML DSE Environment](../spec-ml-dse-environment.md), [ML DSE Model Architecture](../spec-ml-dse-model-architecture.md), [ML Training Core](../spec-ml-core-training.md), [ML DSE Training](../spec-ml-dse-training.md), [ML PnR Environment](../spec-ml-pnr-environment.md), [ML PnR Model Architecture](../spec-ml-pnr-model-architecture.md), [ML PnR Training](../spec-ml-pnr-training.md) | | [Simulation](simulation.md) | [Simulation Artifacts](../spec-simulation-artifacts.md), [DFG-sim](../spec-sim-dfg.md), [CGRA-sim](../spec-sim-cgra.md), [Simulation Comparison](../spec-sim-comparison.md) | | [Runtime And Deployment](runtime-and-deployment.md) | [Executable Closure](../spec-executable-closure.md), [Configuration And Deployment](../spec-configuration-deployment.md), [Runtime ABI](../spec-runtime-abi.md), [Implementation Platform](../spec-implementation-platform.md) | | [Hardware Backend](hardware-backend.md) | [RTL Lowering](../spec-rtl-lowering.md), [Hardware Implementation](../spec-hardware-implementation.md), [Implementation Platform](../spec-implementation-platform.md), [EDA Tooling](../spec-eda-tooling.md), [External Tool Invocation](../spec-external-tool-invocation.md), [FPA Evaluation](../spec-fpa-estimation.md) | diff --git a/docs/rationales/ml.md b/docs/rationales/ml.md new file mode 100644 index 000000000..251f7991b --- /dev/null +++ b/docs/rationales/ml.md @@ -0,0 +1,596 @@ +# ML Search Rationale + +Normative contracts are owned by +[ML Environment Core](../spec-ml-core-environment.md), +[ML Model Core](../spec-ml-core-model.md), +[ML DSE Environment](../spec-ml-dse-environment.md), +[ML DSE Model Architecture](../spec-ml-dse-model-architecture.md), +[ML Training Core](../spec-ml-core-training.md), +[ML DSE Training](../spec-ml-dse-training.md), +[ML PnR Environment](../spec-ml-pnr-environment.md), +[ML PnR Model Architecture](../spec-ml-pnr-model-architecture.md), and +[ML PnR Training](../spec-ml-pnr-training.md). + +Loom uses learned search in two places that look different and are not. One +explores a design space by selecting candidate-generator decisions; the other +places and routes one fixed problem by selecting Place and Route Actions. In +both, the learner is a proposal policy over decisions some other owner already +defined, and everything that decides what a decision means stays with that +owner. These documents explain why that boundary is where it is, and what was +rejected on either side of it. + +## Why A Learned Search Policy Is A Harness, Not A Plan Node + +A reinforcement-learning agent needs the two things the resolved plan +deliberately refuses: a mutable current design and a runtime loop. Adding +either to the plan would make termination, deterministic work, recovery, and +cache identity depend on a policy's sampled behavior, and the plan would stop +being readable before execution. + +The alternative that was rejected was a generic environment action language +over node names and property bags. It would have reproduced exactly the problem +typed domain generators exist to prevent, and it would have needed its own +verifier to decide which actions were legal. + +What the environment does instead is treat the agent as a search policy over +decisions that already exist. A candidate-generator kind already owns a closed +decision union, a finite decision domain, and a canonical decision order, so an +action can be an ordinal in that order rather than a new vocabulary. The +episode's state is an ordinary Builder derivation that stops before +publication, so legality is answered by the same finalizer that answers it for +a published candidate, and nothing partial or DSE-only exists. Only the +retained decision sequence crosses back: replaying it through ordinary Generate +and Promote nodes is what makes a discovered design a real candidate with real +lineage and real Evidence. The harness never selects, never promotes, and never +publishes. + +Reward needed no new concept for the same reason. Search energy is already one +selected weighted level over quantized objective codes, and a policy that needs +a reward wants its signed difference across a transition. Reusing it keeps +reward exact and integral, and keeps a single objective authority for annealing +and learning alike. + +Workload feasibility stayed a gate rather than a penalty because the two answer +different questions. A design that cannot run the workload set is not a poor +point in the space; it is outside it. Scoring it would teach a policy to trade +mappability against area, and there is no exchange rate for that trade. Keeping +proof separate from budget exhaustion follows the same discipline the rest of +the stack uses: only exact admission or a sound bound proves impossibility, and +an exhausted search budget proves nothing about the design. + +Scope became configuration for the same reason the plan uses typed generators. +The episode may explore one Module, a complete multi-core System, and the +software expression of its workloads, and the obvious way to offer that is a +flag per capability. Flags do not compose: three of them are eight +configurations, most of which are meaningless, and each new family doubles the +count again. Binding a set of exploration domains instead makes permission and +extension the same act, and requiring each domain to state the same small set +of facts before it can be explored is a more useful admission test than a +boolean: a family nobody can describe that precisely is a family nobody is +ready to search. + +Some of those facts exist because a generator's child is not always the +episode's subject. A Spatial rewrite inside a System episode yields a Module, +and a Module is an intermediate design input rather than a System candidate. A +software rewrite yields a program whose TechMapping no longer binds. In both +cases the step is not over when the generator returns, and the follow-up is +mechanically determined by the decision's own target. A completion that would +need a second free choice was rejected as a design, because it would make the +environment an author of decisions the agent did not take. + +Targeting an existing trainer's environment definition rather than inventing +one was the cheaper correctness decision, and the one place it was allowed to +dictate the data model is the action space, which a fixed-size space genuinely +requires. The observation was not allowed to follow. Padding the graph to a +fixed extent would have made every batch mostly inert rows at a bound large +enough to be safe, and would have refused legal designs at any smaller bound — +a trainer's preference for rectangular arrays deciding which hardware the +search may occupy. The design space is variable-sized, so the observation is a +variable-size graph. + +That choice is what forced the trainer to be a fork, and the trade was made +deliberately in that direction. The alternatives were to bend the data to the +tool, which is the padding just rejected, or to write a trainer from scratch, +which buys a large surface of well-understood RL machinery for the sake of one +missing capability. Patching graph-space support into the sampler is the +smaller of the three, but it is not free: it is the only modified dependency in +the stack, and every upgrade becomes a rebase. It is affordable only because +the fork is reachable from nothing but the search harness, so its blast radius +stops at training. A fork that could reach a compiler, a Mapping, or an +Artifact schema would not have been worth this. + +A parallel sampler does impose one real cost. It creates many environment +copies, so seed independence cannot come from a single counter; deriving it +from each copy's own coordinates buys independence without coordination, at the +cost of tying exact reproduction to the sampling topology. That cost is visible +in the contract instead of being discovered later as nondeterminism. + +The trainer dependency is confined to its own layer for the same reason +generators are typed, so a second trainer, an offline replay tool, or a +scripted search can use the environment without acquiring it, and its version +movement cannot reach the C++ side. + +Masking is where a search policy can quietly corrupt its own training, which is +why the contract constrains an advisory mask so tightly. It is a policy +commitment rather than a proof, so it cannot be allowed to decide what is +legal. The subtler exposure is that a mask which moves across a run differs +between the moment an action was sampled and the moment that sample is learned +from, and the importance ratio is then wrong in a way no loss curve reveals. +Carrying the mask in the batch costs a little memory and removes the whole +class of failure. + +Software and hardware are symmetric under the feasibility gate, and that +symmetry is the argument for exploring them together. A rewrite that the +current fabric cannot map fails exactly as a fabric edit that drops a resource +the workload needs. Neither is a rewrite-legality question: every catalog +rewrite is externally equivalence-preserving by its owner's contract, so what a +rejection reports is a mismatch between this hardware and this expression of +the workload, which is precisely the joint decision the search exists to make. + +## Why The Node And Action Space Is Combined + +The observation unifies three things a conventional design keeps apart: the +entities of the state, the relations between them, and the actions currently +available. Two separate merges produce that, and only the first is +unconditional. + +The first merge draws links as nodes, because the decisions do. Removing an +occurrence, replacing a point connection, changing a transport link, and +refactoring a graph definition are all ordinary decisions, and a place and +route Action names a physical traversal as readily as an occurrence. An +observation that represented only entities as nodes would leave a large +fraction of the action space pointing at something the policy cannot see. +Promoting every targetable entity to a node makes the graph closed under +decision targets, and that closure is what lets an action stay a single ordinal +while still being scored per node and per link. + +The second merge is the one that matters and it is easy to state wrongly. What +is unified is the *addressing*: an action names graph nodes, and a policy +scores it from the embeddings of the nodes it names. An action set is normally +a side channel — a vector of logits produced from a state embedding, indexed by +a scheme the model reconstructs from nothing — and the whole point is that here +it is not. Scoring an action is reading embeddings the encoder already +produced, and the action index is an ordinal in a canonical order rather than a +decoded coordinate. + +What is *not* unified is the carrier, and the choice between the two forms was +made a consequence of arity rather than a house style. Variable arity needs +out-degree to express it at all; fixed arity does not, and paying a node and +two arcs per entry is ruinous exactly where enumerations are largest — place +and route, where an agent choosing both which realization to place and where +would otherwise spend most of its encoder on its own action set. Deriving the +carrier from arity is what keeps one contract serving both. + +Three alternatives to the addressing were rejected, and each fails on a +different case. + +A flat product of decision kind against entity index cannot express a decision +that names a connection, and cannot express one that names a set of actors at +all. It also has to declare a static per-type stride and decode with the +dynamic one, so it either wastes the majority of its declared space or lets the +two strides disagree. + +A separate action-embedding tower avoids the stride but has to build its own +representation of the entity a decision targets, duplicating the encoder and +letting the two representations drift. When they drift, the head is scoring an +action against a picture of the state that the value head does not share. + +Scoring an action from its target alone — attention over entities with no +per-action term — collapses every alternative on one target into one +indistinguishable action. Choosing where to place a realization and choosing +which of two prototypes to substitute are then the same logit, which is the +entire decision in both environments. + +What the shared addressing buys is a head that factorizes cleanly. An action's +score is an anchor term over what it acts on plus a second term over what it +selects, both read from the graph, so an action carrying no alternatives is +scored directly on its node and one carrying alternatives is scored among them +while sharing its anchor term. That sharing is what lets experience with one +choice inform another on the same target, and the prior architecture's +per-instruction embedding exists for the same reason. + +Place and route is where this stops being a convenience. There an action's +value is a graph node — an occurrence, an endpoint, a traversal — and not an +ordinal in a catalog, so there is no enumeration of values a term could read +from. Only an addressing scheme in which the choice is a node has anything for +the second term to read, whichever carrier names it. + +The cost is real and is stated rather than hidden. Under either carrier the +observation grows with the live enumeration, and the enumeration bound is a +declared capacity that refuses an over-large state rather than truncating one: +truncating would silently hide whichever actions sort last, and the policy +would never learn that they existed. + +## Why Learned Place And Route Selects Existing Actions + +Place and route already had everything a learned search needs except a learner. +It owns a closed Action algebra, a deterministic dynamic domain over one exact +candidate, a transactional mutation mechanism that commits or rolls back +Mapping and Evaluation state together, and an objective closure that already +reduces a candidate to one integer. What it does not own is a good way to +choose which Action to propose next; its annealer draws one from weighted kinds +and uniform canonical domains, which is a deliberately unbiased choice and +therefore a deliberately uninformed one. + +So the learned environment replaces the selector and nothing else. The +alternative that was rejected was a second placer with its own occupancy model, +its own route cost, and its own notion of a partial mapping. It would have +needed a verifier to decide whether its states were legal, and the moment two +things can answer that question they eventually disagree — which is the same +argument that keeps the DSE environment out of the candidate-generator +business. + +Two rules invert relative to the DSE environment, and both inversions are +forced by what is fixed. There, the hardware moves and the workload's +mappability is a gate: a design that cannot run the program is outside the +space. Here the hardware is fixed and mappability is the objective, so a +candidate with unrouted obligations is an ordinary interior point rather than +an excluded one, and it would be perverse to reject the states the search +exists to pass through. And there, an unmasked action is not a promise that the +step advances, because feasibility is discovered downstream; here every +enumerated Action is a member of a domain the owner derived, so the only thing +a mask expresses is which phase the episode is in. + +Incremental placement was expressed as a sweep of rebinds over a complete +candidate rather than as a genuinely partial one, and that is a concession +worth recording as such. A partial candidate would route only settled +dependencies and do strictly less work; the reason it lost is that +`CandidateState` is complete by construction and four owners — the objective, +the handshake topological index, the movable-decision count, and the base +verifier — would each have needed a defined meaning for an unbound decision +that the ordinary product search has no use for. Teaching the product path a +state that exists only for a harness is a worse trade than paying redundant +routing inside the harness. + +The two arms exist because there are two honest answers to who fixes an +imperfect placement, and they measure different things. Handing the result to a +bounded annealer and charging for how far it had to move scores the learned +construction directly: the bound is what makes the recovered energy a statement +about the construction rather than about the annealer. Keeping the agent in +control and charging per repair scores something else — whether it can +recognize that a Mapping is good enough — and gives it routing and resource +Actions that a placement-only neighborhood cannot express. Collapsing them into +one configurable environment would have produced a record where most fields are +inert under most settings. + +That the cleanup bound became a search-policy field rather than a constraint +set was the one place performance decided a contract. The radius is expressible +today as a constraint set restricting each realization's placement domain to a +ball, at zero cost to any owner. But a constraint set is an input to freeze, +and freeze is the dominant cost in this environment; a per-episode one would +give every episode a distinct cache key and turn a warm cache into no cache. A +bound on which proposals a run may make is in any case a property of the search +rather than of the problem, so the field landed where that property belongs. + +Reward needed nothing new for the second time. A per-step signed energy +difference is potential-based shaping with the objective as the potential, so +an episode's return telescopes to the improvement it achieved while every step +still carries signal. The prior system reached the same place by hand, with a +five-tier weighted potential whose constants were spaced by powers of a +hundred; reusing the resolved closure gets the same shaping with the weights +owned by configuration and the arithmetic exact. + +## Why Some Design-Space Actions Read The Schedule + +Most design-space edits are structural. A prototype is swapped, a count is +adjusted, an inventory changes, and nothing about where the software currently +runs has any bearing on what the edit means. A few are not, and the exception +is worth stating because it looks at first like a layering violation: the +hardware search reading the software mapping to decide which hardware edit to +offer. + +Deleting an occurrence is the clarifying case. Structurally, removing a switch +or a function unit is trivial and almost always useless — every net whose route +went through it is now uncarryable, so the child fails to map and the search +learns only that deletion is bad. To be worth offering at all, the deletion has +to come with reconnection, and that is where the topology stops being able to +help. It admits a combinatorial number of possible reconnections and has no +opinion about which matter; adding all of them replaces a deleted node with a +worse connectivity problem than the one it removed. + +The current mapping does have an opinion, and it is the right one. The nets +that actually traverse the deleted occurrence are exactly the ones that need +somewhere else to go, and their upstream and downstream endpoints are exactly +the links worth adding. Reading the schedule turns an edit with a large useless +neighborhood into one with a small purposeful one, which is the difference +between a decision kind a policy can learn and a decision kind it learns to +avoid. + +The deeper reason it is worth the coupling is that the resulting child keeps +the parent's placement carryable. A structurally reconnected child usually has +to be re-mapped from scratch, which throws away both the probe cost and the +evidence that the parent's placement was good; a schedule-preserved one can be +warm-started from the mapping it was derived from. The edit and the schedule +stop being adversaries. That is also why the name is worth having: these are +not "mapping-aware" edits in general, they are edits whose whole point is that +an existing schedule survives them. + +What made this safe to admit was finding the seam that keeps it cheap. Whether +an occurrence *can* be removed is structural, so the size of the action set +still costs nothing to compute, and the environment's whole cheapest-first step +ordering — reject a revisit, reject an over-capacity state, and only then pay +for a probe — survives untouched. Only the contents of one action wait for the +mapping. Had the mapping decided how many actions exist, every capacity test +would have moved behind a probe and the ordering that makes the environment +affordable would have inverted. + +The environment still authors nothing. It selects references and the owners +decide what they mean and whether the result is legal, so a reconnection the +mapping suggested and the Fabric owner rejects comes back as an ordinary +rejection. Reading the schedule is evidence-gathering about which decisions are +worth offering, and it never becomes authority over which are valid. + +## Why The Shared Environment Contract Was Extracted + +The first ML environment defined its observation container, its action surface, +its Python package, its trainer conformance, and its benchmark obligations +inline, which was correct while it was the only one. The second needed all five +unchanged and none of the surrounding episode rules, which is the shape that +makes a shared owner worth the churn. + +The alternative was for the second environment to cite the first as the owner +of those five things. It was rejected because it makes an ordering claim that +is not true: neither environment is upstream of the other, and a reader of the +place and route contract would have to consult a design-space-exploration +document to learn what a graph observation is. Ownership in this stack is +supposed to name where a fact lives, not which document happened to need it +first. + +What stayed behind is the part that is genuinely per-environment, and the +dividing line is worth stating because it is not obvious. The container, the +roles, the index correspondence, and the buffer rules are shared; the column +catalogs are not, because the facts worth exposing about a fabric being edited +and about a mapping being built barely overlap and a shared catalog would be +their union, carrying an inapplicable majority in either. The same reasoning +splits the benchmark contract: what every harness owes is shared, and the +stages it decomposes into are not. + +The models split on exactly the same line, and for a reason that is worth +recording separately: almost everything between an observation and a logit is +the same problem twice. Batching ragged graphs, refusing to feed a catalog +ordinal to a linear layer, a pre-normalized edge-aware trunk, multi-scale +pooling, a bounded value head, and — most of all — the masking rules are +independent of what is being searched, and the masking rules are the ones it +would be most expensive to get subtly different in two places, since every one +of them protects a training-time invariant that no loss curve reveals when it +breaks. What differs is the policy head, which is where the two searches +actually differ: one scores a decision against a catalog of values, the other +scores a pair of nodes. + +The parallel is not quite exact, and the difference says something. The two +environments share a data contract; the two models share a data contract *and* +a set of correctness rules. That is why the model core carries prose about +importance ratios and precision that reads more like a hazard list than a +schema: those are the parts where an implementation can be fast, plausible, and +silently wrong. + +## Why The Placement Choice Is Scored Against Its Anchor + +The prior architecture scored hardware slots from the slot's own embedding and +a pooled graph context, and conditioned on the software node being placed only +through the candidate mask. That is a defensible factoring and it is cheap: the +slot head runs once per slot, not once per pair, so its cost is the hardware +graph rather than the enumeration. + +It is also unable to represent most of the problem. Scored that way, a policy +learns which slots are good in general — central, uncongested, well connected — +which is a real signal and the wrong one. Placement quality is a property of a +pair: a producer wants to be near its consumers, and whether an occurrence +satisfies that depends entirely on which realization is being placed and where +its neighbours already are. A head that cannot see the pair can rank slots but +cannot match them. + +So the choice term reads the anchor's embedding too, and the cost is that it is +evaluated once per available action rather than once per node. That cost was +worth paying, and the two responses to it are worth distinguishing, because the +distinction was nearly lost. + +One is exact and is simply adopted: an algebraic rearrangement that avoids +materializing a concatenation, which is a memory saving and not an arithmetic +one. Promising it as an arithmetic reduction would have claimed something the +algebra does not contain. + +The other is not exact. A genuine per-pair reduction exists, and it computes a +different and strictly less expressive function. That makes it a modelling +decision wearing a performance costume, so it is admitted as a configured +variant measured against the reference rather than substituted for it. The +honest place for an optimization that changes what the model computes is the +configuration, where someone has to choose it and a checkpoint records which +was chosen. + +## Why Pretraining Imitates The Destination, Not The Path + +Place and route already has a search that works. Simulated annealing produces +good placements, slowly, and the obvious way to bootstrap a learned policy is +to show it what the annealer did. The prior system did precisely that: it +recorded a greedy fill followed by every accepted Metropolis move, and trained +on the sequence. + +That trains the wrong thing. An annealing run is a walk that spends most of its +time in states worse than where it started — that is what an acceptance kernel +is *for*, and a run that never occupied a worse state was not annealing. A +policy fitted to that sequence learns to place badly and then shuffle, because +that is what the demonstration shows. The behaviour is faithfully reproduced +and useless. + +What is worth copying is where the walk ended. So the generator keeps the final +placement, throws the path away, and synthesizes a clean construction sweep +that reaches that placement directly — each realization bound once, nothing +undone. The demonstration is then a claim about good placements rather than a +recording of a search, which is the thing a construction phase can actually +imitate. + +One consequence has to be stated rather than smoothed over: the synthesized +sweep does not reproduce the annealer's energy. Placing in a different order +routes the nets differently, so the same final placement carries a different +number. The replayed value is the honest one because it is the state the policy +will actually occupy, and recording the annealer's would describe a candidate +the demonstration never reaches. + +This also explains why demonstrations cover construction and stop. The repair +phase has no destination to imitate — repair *is* the walk — so it is learned +online, from reward, where a walk is the right thing to learn. + +## Why Pretraining And Online Training Are One Run + +They could have been two runs joined by a checkpoint path, which is what the +prior system did. Making them stages of one run buys three things that +arrangement cannot. + +The handoff becomes checkable. A checkpoint path passed between two invocations +is validated by neither, and the prior system documented two ways it silently +failed: a restore aimed at a component path that does not exist loads nothing +and reports success, and an evaluation that reads weights from a sampler rather +than the learner reports the parameters from before the load. Both produce a +run that looks warm-started and is not. As a stage boundary inside one run, +loading is an obligation with a reported parameter count and tensor digest. + +The reference for improvement becomes well defined. The question the two stages +exist to answer is whether online training improved on what the demonstrations +delivered, and that is a comparison against the parameters at the boundary. Two +separate runs have two separate histories and no shared reference; one run has +the boundary reading built in. + +And the things that must agree are forced to agree. `gamma` is the sharp case: +the reward is an exact energy difference, so a discount is a statement about +which future improvements count, and a pretraining stage that discounted +differently would fit a value head to a different return than the online stage +optimizes. Two runs can disagree about that silently. One run's adoption +rejects it. + +## Why Evaluation Compares Against A Run's Own History + +The tempting baseline is the annealer — it is what the demonstrations came +from, and beating it is the point. It was rejected for evaluation *during* +training on cost: an annealing invocation per case per evaluation is the +dominant cost in this environment, and a test protocol that ran one would cost +more than the training it measures. That comparison is a question about a +finished checkpoint, and it runs offline against recorded results. + +What a run needs while it is running is whether it is getting better, and the +honest reference for that is its own earlier weights on the same cases. Hence a +per-case series against a fixed test set, with each evaluation reporting its +difference from the previous one and from the most recent stage boundary. + +Reporting per case rather than per aggregate is the part that matters. An +aggregate hides the case that regressed, and a policy that improves its mean +while losing its hardest problems is exactly the failure mode a placement +policy falls into — the easy instances are numerous and the hard ones are the +reason anyone wanted a learned placer. A mean over them would have called that +progress. + +## Why The Mask Is Packed + +The observation carries its arrays in the narrowest form that represents them, +and the action mask goes further and is packed to one bit per outcome. Both are +unusual enough in a specification to need a reason, because storage width is +ordinarily an implementation's business. + +It is here because of what an observation is used for. An observation is not +produced once and read once. It is produced tens of millions of times per +experiment, it is copied into a sample buffer, it is carried across a process +boundary to a learner, and then — because the mask an action was sampled under +has to be the mask it is learned under — it is held for as many epochs as the +algorithm takes. Every byte in an observation is multiplied by the batch size, +by the epoch count, and by the number of environment copies a sampler runs. +That multiplier is what turns a storage question into a throughput question: +past some batch size the run stops being limited by arithmetic and starts being +limited by how much of the batch fits in memory at once, and the only lever on +that is how wide the batch is. + +The widths themselves were free to take. An `ArcRole` has three values and was +occupying eight bytes on the most numerous extent in the observation; a role, a +kind, and a boolean placement flag were doing the same on the second most +numerous. Nothing was gained by that uniformity, and no catalog had to grow to +give it up, since the width a value needs is already fixed by the owner that +bounds the value. + +The mask is the sharp case and is worth separating from the rest. It is the one +array whose size is a capacity rather than a state. Everything else in the +observation costs what the state actually needs — the graph is ragged and the +decision instance is exactly the live enumeration — but the mask spans +`enumeration_bound + 1` outcomes at every step of every episode regardless of +how few are live. And a capacity is chosen with headroom, deliberately, so most +of the mask is usually inert. Narrowing it to one byte per outcome would still +leave it the largest fixed term in an observation at the bounds these searches +use. Packing to one bit is what makes a capacity chosen generously stop costing +what it was chosen at, which in turn means a bound can be set for the worst +state a run might reach rather than for the memory it would cost to allow it. + +Packing is also, unusually, the more honest representation rather than a +compression of a nicer one. A mask entry is a bit — there is no state it can +carry that a bit cannot — and the byte-per-entry form was the encoding that +added information the mask does not have. Several of the masking rules read +as consequences of that representation rather than as constraints imposed on +top of it. The one obligation packing genuinely adds is zeroing the pad bits, +and that is the price of the observation having a canonical byte form at all. + +What is deliberately not claimed is that any of this is free. Unpacking and +widening are real work and they happen on every forward pass. The bet is that a +shift and a cast on a device that was about to touch the value anyway costs +less than moving eight times the bytes to reach it. What makes that a +measurement rather than an assertion is that the model harnesses report the +bytes and a reference run that widens at the boundary instead — a stage that +only existed when the saving was discarded would have answered a different +question. + +## Why The ML Environments And Models Are Benchmarked + +Everywhere else in Loom, speed is an engineering concern that the contracts +mention only where it changes admission. Here it is in the specifications, with +named harnesses, named stages, and required breakdowns. That difference needs a +reason, and the reason is that the environment and the model sit on the inner +loop of every experiment. + +A compile runs once per design. An environment step runs tens of millions of +times per training run, and the model runs once per step on top of it. A factor +of two in either is a factor of two in the wall clock of every experiment +anyone runs afterwards, which is not a performance detail but the difference +between a curriculum that can reach large designs and one that cannot. Leaving +that to be discovered informally means discovering it after the schedule has +already been built around the slow version. + +The stronger reason is that both contracts deliberately offer options whose +cost is invisible without measurement. Warm-starting a probe from the parent +trades a cache and path-independent energy for throughput. Interleaving +per-workload regeneration with probing pays off only if early abort is common. +Retaining frozen models across episodes is what makes place and route resets +affordable at all. Emitting route arcs makes the observation much larger for a +policy that may not need them. Each is an option the specification states, and +an option nobody measures is an option nobody can actually choose; writing the +measurement into the contract is what keeps those from becoming defaults that +nobody revisits. + +One aggregate number would answer none of it. A step is a pipeline of +completely unrelated owners — a draft finalization, a Mapping probe, an +inference call, a marshalling boundary — and a single latency moves for reasons +it cannot attribute. Decomposing into named stages is what makes a regression +locatable, and naming the stages in the specification rather than in a tool is +what keeps two implementations comparable. + +The two mandatory breakdowns are mandatory because both hide real regressions +when blended, and a training run moves exactly the mixes they separate: a +policy's failure mix moves constantly, so a genuine probe slowdown can hide +behind a rising early-rejection rate and read as an improvement. + +Instrumentation is off outside the harness because measuring it would change +it, which is also why the contract refuses to let harness stage times be +compared against real throughput. + +The model is measured in two regimes because inference and training have +opposite profiles and an optimization that helps one routinely harms the other, +so one averaged number reliably optimizes the wrong half. The split against the +environment step comes first for a related reason: without it, the obvious work +is to speed up whichever half is already small. + +Everything these harnesses produce is nonsemantic and removable, and that is +load-bearing rather than administrative. A benchmark result must never become +something a selection, promotion, or conformance decision can consume, because +a timing is a property of a machine on a day and admitting one as a durable +fact would let hardware variation reach a semantic outcome. That is also why +budgets are ratios against a recorded baseline tuple of exact identities and +configuration digests rather than absolute durations — an absolute duration +encodes a machine — and why deterministic work-unit counts are reported +alongside the timings and labelled as the cross-machine measure. The wall time +is for the engineer; the work units are the number that means the same thing +twice. diff --git a/docs/spec-adg-builder.md b/docs/spec-adg-builder.md index d73b44980..c5064da10 100644 --- a/docs/spec-adg-builder.md +++ b/docs/spec-adg-builder.md @@ -1483,6 +1483,12 @@ deterministic elaboration only. There is no generic hardware action language, mutable candidate graph, caller-authored property bag, or DSE-only construction path. +A caller that requires a verified candidate without an Artifact takes the +`VerifiedFabricClosure` owned by +[Finalization And Publication](spec-fabric-artifact.md#finalization-and-publication) +and does not publish it. The Builder gains no second finalization path from +that choice. + ## Conformance Anchors The stable Builder anchors are deliberately small: diff --git a/docs/spec-fabric-artifact.md b/docs/spec-fabric-artifact.md index 7a84aec99..ade4223c6 100644 --- a/docs/spec-fabric-artifact.md +++ b/docs/spec-fabric-artifact.md @@ -328,6 +328,23 @@ reimported the result cannot return `FinalizedFabricRoot` or claim artifact success. It must return the typed unavailable, invalid, incomplete, or store failure owned by the first unsatisfied stage. +The pipeline has exactly one named intermediate. A `VerifiedFabricClosure` is +the value that exists after independent reverification and before +`ArtifactStore::put`: canonical bytes, the computed candidate +ArtifactIdentity, and the validated dependency closure, with no store object. +It is not a reduced finalization mode, because every semantic stage above has +already run and none may be skipped to reach it; it is the same derivation +observed one step before its terminal. Publication consumes a `VerifiedFabricClosure` and +returns the published `ArtifactRootReference`. + +A caller that requires an Artifact publishes. A caller that only requires a +verified candidate and its exact identity, such as a search harness evaluating +a candidate it may discard, may hold the closure and publish later or never. +A `VerifiedFabricClosure` is a transient in-process value, not an Artifact +family, a persistent schema, or a second identity authority: until publication +no other owner may reference it, nothing may depend on it, and any value +derived from it is a removable projection. + Fabric failure atomicity means one root object is complete or absent; it does not mean that the root and its dependency graph become visible in one transaction. Dependencies are independently valid, immutable, shareable diff --git a/docs/spec-loom-stack.md b/docs/spec-loom-stack.md index 508e1728f..74b2259e9 100644 --- a/docs/spec-loom-stack.md +++ b/docs/spec-loom-stack.md @@ -751,6 +751,40 @@ identity remain Loom-owned; Loom does not maintain a gem5 patch stack or edit the pinned submodule source. A gem5 upgrade is a separate exact dependency change with Runtime ABI and System simulation conformance. +Ray is the one modified dependency. It is a Loom-owned fork at +`externals/ray`, and Loom does maintain its patch stack. An upgrade is a +separate exact dependency change: it selects an exact upstream Ray release, +rebases the patch stack onto it, reruns the ML environment's conformance +anchors, and atomically pins both the resulting fork commit and the upstream +release tag it was built from. Recording only the fork commit would be +insufficient, since the patch stack is the difference between the two and a +fork commit alone does not say what it is a patch stack against. + +The patch adds graph-space support to the sampler and connector pipeline, which +[ML Environment Core](spec-ml-core-environment.md#rllib-environment-definition) +requires because every ML search environment's observation is a variable-size +graph. That capability is the only reason the fork exists; a patch that is not +required by a Loom-owned contract belongs upstream instead. Carrying a patch +stack is a standing cost recorded here rather than a precedent for modifying +any other pinned dependency. + +The fork is admitted on a narrower footing than the others: no Artifact, +Evidence, or semantic configuration schema depends on it, so a rebase can break +training without invalidating a semantic artifact. It does reach the search +harness's own training views, which are digest-covered like any other, so a +rebase that changed their vocabulary would change a training run's identity. +That is the intended blast radius, and it is the whole of it. + +Every Python runtime a search-harness document states conformance against is +pinned here to an exact version, and the pinned Ray fork is required to be +compatible with all of them. Currently that is the Gymnasium release the +[ML Environment Core](spec-ml-core-environment.md#rllib-environment-definition) +targets and the PyTorch release +[ML Model Core](spec-ml-core-model.md#module-boundary) targets. They are pinned +here rather than in the documents that consume them, so that a second ML +document adopting either does not give the stack two places to disagree about +which runtime it runs on. + ## Verification Boundary Tests protect stable semantic anchors: canonical schema and identity, diff --git a/docs/spec-ml-core-environment.md b/docs/spec-ml-core-environment.md new file mode 100644 index 000000000..376919685 --- /dev/null +++ b/docs/spec-ml-core-environment.md @@ -0,0 +1,826 @@ +# ML Environment Core + +This document defines the contract every Loom reinforcement-learning search +environment shares: how a search state is projected into one observation, how +an action index addresses one member of a live enumeration, how an episode's +transitions and endings are accounted for, how reward crosses the integer +boundary, how a parallel sampler stays reproducible, and how the whole thing is +presented to Python and measured. + +It defines no design space of its own. What is being searched, which decisions +exist, what makes one legal, and what an episode is trying to achieve are owned +by the environment documents that build on this one: + +- [ML DSE Environment](spec-ml-dse-environment.md) searches Loom's design space + by selecting candidate-generator decisions over a Fabric subject; and +- [ML PnR Environment](spec-ml-pnr-environment.md) searches one fixed Mapping + problem by selecting typed Place and Route Actions. + +Two environments already share more machinery than either owns alone, which is +why the machinery is here rather than in whichever document happened to need it +first. A third environment appends a bullet above and states the small set of +obligations each section below names; it does not reopen this contract. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [Evaluation and DSE](spec-dse-feedback.md#objectives-and-quality-gates) owns + `ObjectiveDimension`, `ExactAffineQuantization`, `ObjectiveVector`, + `WeightedLevel`, `TotalOrdering`, `SearchEnergyRef`, and the statement that a + reward is the signed difference of a selected search energy; +- [Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism) + owns the seeded PRNG protocol, its `nextBounded` projection over canonically + sorted domains, and its prohibition on host entropy and library + distributions; +- [Evaluation Metrics](spec-evaluation-metrics.md) owns `ExactRatio`; +- [Resolved Configuration](spec-config-ssot.md#component-views) owns + component-view framing, canonical view bytes, and `component_view_digest`; +- [Operational Observations](spec-dse-feedback.md#operational-observations) + owns the nonsemantic status of wall time, the prohibition on summing + concurrent times, and deterministic work summaries as the cross-machine cost + measure; and +- [Full-Stack Architecture](spec-loom-stack.md#external-dependency-pinning) + owns the exact revisions of every search-harness dependency, including the + Ray fork the RLlib section targets. + +The RLlib environment definition, its env context, its registration mechanism, +and its sampling topology are owned by that external dependency. This document +states which of its obligations bind an environment and does not restate, +extend, or version them; the training document that configures an environment +configures it. + +This document owns only the observation container, its combined node-and-link +space, the two enumeration encodings, and the storage form of every observation +array including the packed mask layout; the action surface and masking algebra; +the step accounting identity and the termination-versus-truncation rule; the +reward boundary; the determinism record, the PRNG preimage shape, and the +episode-start override contract; the trajectory record's status; the `loomml` +package layering and its interaction contract; and the benchmarking obligations +every harness satisfies. + +## Nonsemantic Boundary + +An ML environment is a nonsemantic search harness. It is not a Candidate +Generator, Mapping, Evaluation, objective, configuration, or Artifact +authority. It defines no action language of its own and no second design space. +Every decision it applies is an existing owner-typed decision, every legality +answer comes from that decision's ordinary owner, and every reward is a +projection of the objective algebra owned by +[Objectives and Quality Gates](spec-dse-feedback.md#objectives-and-quality-gates). + +No environment operation publishes an `EvaluationRequest`, +`EvaluationEvidence`, an `InvocationManifest` record, or a lineage edge, and no +environment value enters a Fabric, Mapping, or Evaluation Artifact. An +observation column, a reward, a mask bit, a step record, and every output of +every harness in the ML stack are removable projections; regenerating them may +change presentation but must preserve every referenced semantic fact, and no +consumer may treat one as a semantic result. Nothing on that list is an +Artifact, Evidence, or a report schema, and no selection, promotion, or +conformance decision consumes one. + +An environment introduces no persistent schema family beyond its own resolved +configuration, which is an ordinary component view. Its episode state is +process-local and its trajectory record is a transient value: a trajectory is +not an Artifact, not Evidence, and not a plan, and it carries no outcome, no +reward, and no selection claim. Only the steps that advanced are retained, so a +non-advancing attempt stays an episode-local observation and never becomes +fake history. + +An environment run is therefore never an authority for selection. A result an +environment discovers becomes real only when its environment document states an +ordinary owner-owned path that reconstructs it. + +## Combined Node And Link Space + +Decisions target entities and relations alike. A decision may name an +occurrence, a connection, a transport link, an actor, a graph, or a route, so a +projection that represented only entities as graph nodes would leave every +relation-valued decision pointing at nothing. The observation is therefore a +combined node-and-link space: an entity that another representation would draw +as an arc is instead a node, and the arc it replaces becomes two arcs through +it. + +Nodes are laid out in role blocks in this order, each block dense and in +canonical owner order: + +```text +GraphNodeRole = + FabricOccurrence // 0 + | FabricConnection // 1 + | DataflowOperation // 2 + | DataflowValue // 3 + | Decision // 4 +``` + +Ordinals are stable. A new role appends; reordering, deleting, or repurposing +one is an incompatible change to this document's schema version. An environment +whose state contains no member of a block contributes no rows for it and needs +no flag to say so, because its own contract already decides which blocks exist. + +A `FabricConnection` node replaces the direct arc it stands for: an original +connection from occurrence `u` to occurrence `v` contributes node `l` and the +two arcs `u -> l` and `l -> v`. `DataflowValue` nodes are promoted from values +under the same rule. A parallel connection is one node carrying its current +count as a feature, not one node per parallel lane, so a decision that changes +a count has exactly one target and never changes the node inventory. + +"Decision" is this document's generic name for one member of the live +enumeration; an environment document names what its members actually are. The +`Decision` block is present exactly when the environment uses the +`DecisionNodes` encoding defined below. + +The space is closed under decision targets. For every member of the live +enumeration, every owner-local reference its normalized payload carries +resolves to exactly one node ordinal in this graph. A reference that resolves +to no node, to more than one node, or to a node outside the current state is a +projection defect rather than an unaddressable action. This closure is what +lets the enumeration and the graph agree: a policy scores a decision by +attending over the nodes it names, so an action is per-node and per-link even +though the action index itself is an ordinal in a canonical order. + +## Enumeration Encoding + +The closure rule says every live entry names graph nodes. How it names them is +one of two forms: + +```text +EnumerationEncoding = + DecisionNodes // 0 + | DecisionColumns // 1 +``` + +Under `DecisionNodes`, each live entry is a node in the `Decision` block and +each reference it carries is an arc from that node to the node it names. An +entry's target count is its out-degree, its target roles are arc features, and +its own categorical and numeric properties are columns of the node matrix. A +decision's ordinal within the `Decision` block is its `ActionIndex`. + +Under `DecisionColumns`, the `Decision` block is absent and the enumeration is +carried alongside the graph as its own dense matrix, one row per live entry. +The references an entry carries are columns holding node ordinals into the same +instance's graph, and its own properties are further columns of the same row. +An entry's `ActionIndex` is its row ordinal. + +The choice between them is not free. **An enumeration whose entries have +variable target arity uses `DecisionNodes`; one whose entries have fixed arity +uses `DecisionColumns`.** A decision that names one entity plus a reference set +of unbounded size needs out-degree to express its targets, and columns would +force either a maximum arity or an offset table beside the matrix. An action +that names exactly an anchor and a choice needs two columns, and a node with +two arcs says the same thing while adding a row and two arcs to the graph for +every member of an enumeration that is routinely far larger than the state it +ranges over. Neither form is a preference: an environment whose arity is fixed +and that chose nodes anyway would spend most of its encoder on its own action +set. + +What is common to both is the part a policy actually consumes. An entry names +graph nodes, those nodes carry the encoder's representation of what the entry +acts on, and the entry's index is an ordinal in the canonical order. A model +scores an entry from the embeddings of the nodes it names under either +encoding; what differs is where it reads the naming from. + +## The Graph Instance + +The observation is one variable-size graph, carried by the Gymnasium `Graph` +space and produced as one `GraphInstance` per state. It is not padded, not +bounded, and not reshaped to a fixed extent: the node count, arc count, and +their feature matrices are exactly the size the state requires, and a state +that needs more room simply produces a larger instance. + +```text +Observation { + graph: GraphInstance + decisions: GraphInstance + action_mask: packed bitmask over enumeration_bound + 1 outcomes + objective_codes: dense array + scalar_features: dense array +} + +GraphInstance { + nodes: per-column array + edges: per-column array + edge_links: dense array<(arc ordinal, 2), uint32> +} +``` + +`decisions` carries the enumeration under `DecisionColumns` and is empty under +`DecisionNodes`, where the enumeration is already in `graph`. It is a +`GraphInstance` with rows and no arcs rather than a bare matrix, because the +observation already needs one ragged per-instance carrier and this is it: the +same space, the same batching, and no second capability asked of a consumer +that can already carry one. Its `nodes` matrix is indexed by live entry ordinal +over the environment's own `DecisionColumn` catalog, and its `edges` and +`edge_links` are empty. + +Under `DecisionNodes`, an entry's targets are arcs from its decision node, so +its target count is its out-degree and nothing needs an offset array, a slot +table, or a per-decision count column. This is the same move the combined space +makes for connections: an entity with a variable number of relations becomes a +node, and the relations become arcs. + +`objective_codes` holds the current directed code of each dimension in the +environment's selected closure, in ascending `ObjectiveDimensionRef` order, +exactly as produced by the quantization owned by Evaluation and DSE. It is the +state's objective position, not a reward. + +`NodeFeatureColumn`, `ArcFeatureColumn`, `ScalarFeatureColumn`, and +`DecisionColumn` are closed catalogs each environment document owns, because +the facts worth exposing about a Fabric being edited and about a Mapping being +built are not the same facts. A shared catalog would have to be the union of +both and would carry an inapplicable majority in either environment. Six +obligations bind every such catalog: + +1. `NodeFeatureColumn` ordinal zero is `Role`, carrying the node's + `GraphNodeRole`, so a consumer can interpret every other column without + knowing which environment produced the instance; +2. a column that does not apply to a node's role is encoded as negative one, so + one node matrix with role-conditioned columns keeps every node addressable + by one ordinal, which is what lets a policy attend over entity and decision + nodes together; +3. `ArcFeatureColumn` ordinal zero is `ArcRole`, whose ordinal zero is + `Structural`, whose ordinal one is `Placement` where the environment + projects a placement relation, and whose ordinal two is `Route` where it + projects a routing relation; an environment appends its own roles after + those and omits the ones it does not project. A target arc's role is carried + by `ArcRole` itself rather than by a second column that would be + inapplicable on every structural arc. An environment with no second arc + column declares its `ArcRole` catalog and no `ArcFeatureColumn` wrapper, + since a one-member catalog is a name for its member; +4. a categorical column holds an owner catalog's ordinal rather than an + environment-local enum wherever such a catalog exists, so a new owner kind + extends the observation without a new column and without a new role; and +5. a `DecisionColumn` that names a target holds a node ordinal into the same + instance's `graph`, and the target-closure rule binds it exactly as it binds + a target arc. A `DecisionColumn` catalog is empty under `DecisionNodes`; and +6. a node catalog is extended by one capacity column and one usage column per + member of the state's fixed canonical `FabricResourceStateRef` catalog, in + that catalog's order, exactly when the environment sets + `include_resource_states`. The appended group is fixed for the episode, so + the catalog keeps a fixed column count even though its row count varies. + This group is stated + here rather than per environment because every environment appends the same + one from the same owner catalog; an environment needing a second, different + appended group states that one itself. + +Every array is C-contiguous with the declared element type; its lifetime and +the prohibition on writing it are owned by +[Interaction Contract](#interaction-contract). Column order within each catalog +is part of this contract; row counts are not, and no consumer may infer a state +fact from an extent beyond the count that extent literally is. + +### Storage Widths + +Every observation array is stored in the narrowest form that represents its +declared value domain exactly. Nothing is stored wider to make two arrays the +same type. + +The domain fixes the form, so no catalog declares a width. A boolean domain is +one bit per entry, packed. A bounded-ordinal or bounded-count domain is the +narrowest integer holding it: a categorical column's from its owner catalog's +cardinality, a counting column's from the bound its owner already places on +what it counts, a magnitude's from the range its quantization declares. A +column that uses the negative-one absence encoding is signed and its width +accounts for that value; a column that is never absent is unsigned. Requiring a +declaration as well would put a second source beside a fact the owner already +fixes, and the two would eventually disagree about a column neither had +changed. + +`scalar_features` is the one named exemption and stays `int64`: it is one fixed +short array per observation, so narrowing it would buy nothing measurable +against a per-column rule where a single type reads more clearly. + +Because widths differ per column, `nodes` and `edges` are column-major: one +contiguous array per column, in catalog order. They are not one +two-dimensional matrix. This costs nothing a consumer wanted — the model +embeds and projects each column separately, so it indexes columns individually +in either layout — and it is what lets a one-byte column occupy one byte per +row instead of being widened to its matrix's element type. + +A column spans only the role blocks it applies to, and is stored over exactly +those blocks' contiguous ordinal ranges. Blocks are already dense and in a +fixed order, so a column that is inapplicable to a role has no entries for it +rather than a run of absence values. Addressability is unaffected, since a +block is a contiguous range of node ordinals. The absence encoding remains for +a column that is inapplicable to some member of a block it does span. + +A consumer that needs a wider or unpacked form produces it after transfer, on +the device that consumes the value, and never in the environment or in a layer +between them. Widening at the boundary would restore exactly the bytes this +rule exists to avoid, in the hottest place and for the whole journey through +the connectors and the sample buffer; widening on device is a cast against +arithmetic that was going to touch the value anyway. This is the only rule +about the two forms: nothing else in this document distinguishes a value by +which one it is in. + +Why the observation is worth this much attention, and why the mask is the +sharpest case, is +[Why The Mask Is Packed](rationales/ml.md#why-the-mask-is-packed). + +#### The Packed Mask Layout + +`action_mask` is the one boolean-domain array, so it is the one array the rule +above packs. Bit `i` is bit `i & 7` of byte `i >> 3`, least significant first, +making the array `ceil((enumeration_bound + 1) / 8)` bytes of `uint8`, and +every bit above `enumeration_bound` in the final byte is zero. + +Zeroing the pad bits is what keeps two states with the same admissible set +byte-identical. Leaving them undefined would make the observation's bytes +depend on whatever the producer's buffer last held, and would do it only for +states whose live count is not a multiple of eight — breaking the +byte-identical-enumeration property +[Action Surface And Masking](#action-surface-and-masking) states, +intermittently. + +The bit order is fixed here rather than left to a producer because both sides +index it. A mask is the one observation member where a layout disagreement is +silent: a wrong feature column trains a worse policy, while a wrong bit order +masks a different action than the one the environment refused, and the first +symptom is an importance ratio that is wrong for reasons no loss curve +explains. + +`graph` and `decisions` are the only parts of the observation whose extent +varies, and neither is padded for the same reason: a bound large enough for the +largest reachable state wastes most of every batch on inert rows, and this is +most acute for `decisions`, whose row count is the live enumeration length and +whose capacity would have to be `enumeration_bound`. `objective_codes` and +`scalar_features` are fixed-length over closed catalogs, and the action mask is +fixed-length because a Gymnasium action space is fixed at construction and +`Discrete(enumeration_bound)` cannot vary per state. What distinguishes the +mask is that its extent is a declared capacity plus the stop entry rather than +a real count: decision entries at or beyond the live enumeration length are +clear, and the observation carries no entry for them under either encoding. + +## Action Surface And Masking + +The interaction surface is: + +```text +action = { + decision: Discrete(enumeration_bound) + stop: Discrete(2) +} +``` + +`enumeration_bound` is a static capacity so the space has a fixed declared +shape, while the live enumeration length varies with the state. A policy +therefore scores real decisions rather than a fixed factorization of decision +kind against node index. This replaces a flat product space over node slots and +decision types: such a space must declare a static per-type stride and decode +with the dynamic one, which either wastes the majority of the declared space or +lets the two strides disagree. + +An environment's enumeration is a pure function of its current state and its +resolved configuration, so two runs with equal state and configuration produce +byte-identical enumerations in the same order. + +Masking is the primary invalid-action mechanism. An index at or beyond the live +count, an index whose mask bit an environment cleared, and an out-of-range +index are each errors: the environment refuses the call, leaves the state +unchanged, and reports the violated precondition. A masked index is not +silently coerced to a no-op, to the nearest legal index, or to a terminal step, +because each of those makes the training signal indistinguishable from a real +decision. + +`stop` set to one ends the episode electively at the current state and is +mutually exclusive with advancing; the `decision` field is ignored in that +call. An elective stop is the only outcome the policy itself controls. + +`action_mask` covers one more outcome than `enumeration_bound` because it +covers the stop outcome as well as the decision slots, and it is indexed +exactly as the action distribution's outcomes are: bit `i` below +`enumeration_bound` masks the decision at that index, and bit +`enumeration_bound` masks stopping. One mask indexed one way is what lets a +model apply it additively to its logits without a second layout to keep in +step. + +An environment may therefore clear stop for a state in which electing to stop +has no meaning. What it may not produce is a *non-terminal* state with no +admissible outcome at all: such a state has no defined action distribution, and +reaching one while the episode continues is a contract violation rather than an +ending. Packed, that test is whether every byte of `action_mask` is zero, which +the pad-bit rule makes exact. + +The mask of an observation returned alongside an ending is never a distribution +input. No action is sampled from it, the value head does not read it, and no +consumer composes it or counts it as a fully-masked fallback. It may therefore +be all-clear, and an environment with no admissible outcome at a terminal state +returns one rather than manufacturing an outcome that does not exist. + +A state whose enumeration exceeds `enumeration_bound` is refused rather than +truncated, because truncation permanently hides whichever part of the +enumeration sorts last. The refusal names both the capacity and the required +length, so the configuration can be corrected rather than guessed at. +`enumeration_bound` is the only capacity in this contract, because the +observation graph has none. + +## Step Accounting And Episode Endings + +Every step carries at most one transition, and carries none exactly when it is +an elective stop, so an episode record satisfies + +```text +steps == advanced + non_advancing + elective_stops +``` + +by construction with no unaccounted remainder. `non_advancing` is the class of +steps that carried a transition which did not take effect. Each environment +names that class in its own vocabulary — one search calls it a rejection and +another a failure — and reports its count and its reasons under that name. What +is fixed here is the accounting, not the word. + +A transition and an episode ending are separate facts because they co-occur: a +non-advancing step may both fail and end the episode, an advance may both +advance and +land in a state with no admissible decision, and an elective stop ends the +episode with no transition at all. A single three-way union could express none +of those without discarding half of what happened, so a step result carries +both, and its ending member is present exactly when the episode is over. An +episode records exactly one ending, on its final step, whichever transition +that step did or did not carry. + +An ending an environment reaches by advancing into a state with no admissible +decision is reported on the step that produced that state, not on a later call, +so a consumer is never handed a state it has no legal index to act on. + +An ending is either a termination or a truncation, and the two are reported +distinctly. A termination means the episode reached a state its contract +defines as final, and a value estimate must not bootstrap past it. A truncation +means the episode was cut off by a harness limit while the state remained +ordinary, and a value estimate must bootstrap. Reporting the two as one flag is +invalid. + +An ending an environment defines as a truncation is itself priced at zero. +Charging a penalty for a limit the policy did not choose teaches it to avoid +states that merely take longer to leave, so among endings only the one the +policy elects carries a price. + +That rule prices the *ending*, not the state the episode ended in. An +environment may charge for a property of its final candidate on whichever step +ends the episode, truncating or not, because such a charge is about something +the policy produced rather than about a limit it did not choose. The two are +told apart by what they read: an ending price reads the terminal reason and +nothing else, and a terminal state charge reads the state and never the reason. +An environment that charged more for the same final state under one ending than +under another would be pricing the limit through a state charge, which this +rule forbids in either spelling. + +## Reward Contract + +Reward is the signed difference of a selected search energy across a +transition. Energy is the value of the selected `WeightedLevel`, its checked +`uint128` arithmetic, and the sign-plus-magnitude form of its difference are +all owned by +[Objectives and Quality Gates](spec-dse-feedback.md#objectives-and-quality-gates); +an environment document states only which two states the difference is taken +between and what its non-transition outcomes cost. + +With energy as a potential, a per-step signed energy difference is +potential-based shaping, so an episode's undiscounted return telescopes to the +total improvement between its first and last state while every step still +carries signal. An environment states only what breaks that property for it. + +The energy is a directed code, so a decrease is an improvement. The reported +sign is positive when the child energy is below the parent energy and negative +when it is above. Every conversion, subtraction, product, and sum is checked; +an overflow is a resolved-policy failure, never a clamp, a saturation, or a +candidate penalty. A parent's energy is retained rather than recomputed, so a +step evaluates one energy, not two. + +The native environment emits the exact `(sign, magnitude)` pair and nothing +else. It performs no scaling, normalization, clipping, discounting, or shaping. +The `float64` the RLlib environment definition requires is produced in +`loomml.env`, which applies one `ExactRatio` scale, and every other reward +transform belongs to the training document that configures that environment. +Keeping the conversion there is what lets the core stay integer-only and keeps +a mantissa width out of a canonical, digest-covered configuration; the adapter +is also the only layer that can state the exactness condition its own float +type imposes. + +That scale is `reward_adapter.scale`, a training-view field. `loomml.env` +binds only an environment view, so the layer that reads a training +configuration passes the resolved ratio down at construction and `loomml.env` +applies the ratio it was given. Putting the scale in the environment view +instead would make every scale change a new environment digest and discard that +environment's caches for a number the environment does not use. + +Reward codes an environment charges for non-transition outcomes are stated as +`uint64` magnitudes in the energy domain and applied with the sign the outcome +fixes, so no configuration view carries a signed or floating-point number. Such +a code is a declared constant and is never derived from the state, so a step +that carried no evaluated candidate can never be scored as though it had one. + +An unavailable objective source has no numeric value at all. A step whose +selected closure cannot be evaluated fails rather than being scored from a +substitute, a default, or a neighbouring dimension; each environment names the +outcome it reports, and none of them produces a number. + +## Determinism And Copy Coordinates + +The seeded PRNG protocol, its `nextBounded` projection over canonically sorted +domains, and its prohibition on host entropy and library distributions are +owned by +[Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism). +Every environment view carries the same two-field record, because both fields +are inputs to that protocol rather than facts about a search: + +```text +EnvironmentDeterminismPolicy { + master_seed: u64 + prng_protocol: Sha256SeededXoshiro256StarStar_1_0 +} +``` + +Each environment adds only its own domain separator and its own stream +purposes, in this preimage shape: + +```text +ASCII(environment domain separator) + || u64be(effective_seed) + || u32be(env_runner_index) + || u32be(vector_index) + || u64be(local_episode_index) + || u32be(stream_purpose_ordinal) +``` + +`effective_seed` is the episode's seed input, which is the configuration's +declared master seed unless the consumer supplied an override at `reset`. +`env_runner_index` and `vector_index` identify one environment copy within a +parallel sampling topology and are supplied by the consumer; a single +non-parallel consumer supplies zero for both. `local_episode_index` counts +episodes on that copy from zero. + +Including the copy coordinates in the preimage is what guarantees two copies +never draw the same sequence, and it is why a copy needs no coordination with +any other copy to stay independent. A copy coordinate is not host entropy: it +is a declared input the consumer supplies and replay reproduces. + +A supplied `reset` seed becomes that episode's effective seed, and the local +episode counter restarts. It is an input the caller supplies rather than an +override of a digest-covered configuration field, so the configured master seed +remains the default the view declares and the trajectory records which seed +actually ran. Silently ignoring the argument is invalid: it makes an +environment that appears seeded but is not, which is a failure mode that +survives every test that only checks a run against itself. + +Every environment accepts an exact episode start through the Gymnasium +`options` argument instead of drawing one, and names the payload that carries +it. This is an obligation rather than a permission because a peer contract +consumes it: [Test Protocol](spec-ml-core-training.md#test-protocol) runs every +test case by passing its instance as that override, so an environment without +one cannot be evaluated at all. + +Four rules bind every override, whatever payload it carries. Its members must +already appear in the inventory the resolved configuration declares, so an +override selects from that inventory and cannot introduce anything the +configuration does not. When present it replaces the drawing steps entirely, so +no selection stream is consulted and the copy coordinates do not reach the +choice. The same override on any copy therefore produces the same episode, +which is what lets a fixed evaluation set run identically across runners and is +the only way an episode start is not a draw. + +And a start failure under an override is returned as it stands. Retrying +requires a draw to retry and an override supplies none, so no redraw is +attempted and no retry budget an environment may otherwise declare is +consulted. Retrying would be wrong even where it were possible: a case that +silently ran a different instance than the one it names is not that case. + +Policy sampling is not an environment stream: the learned policy owns its own +randomness and supplies an action index, so an environment replay is +reproducible given the same configuration, coordinates, seed, and action +sequence. + +## Python Boundary + +### Layering + +A learned policy interacts through a Python package `loomml` with three layers, +each depending only on the one below: + +```text +loomml.rllib RLlib environment definitions, registration, and env context +loomml.env the Gymnasium Env surfaces +loomml._core native extension over the episodes, enumerations, and arrays +``` + +`loomml._core` exposes each episode lifecycle, its enumeration, its observation +arrays, and its closed outcome unions. `loomml.env` adds the Gymnasium `Env` +surface and holds no state the core does not own. `loomml.rllib` adds the +RLlib-specific obligations below. One environment occupies one module in each +layer, named for itself, so `loomml.env` and `loomml.rllib` are packages rather +than modules and adding an environment adds modules rather than editing shared +ones. + +The layering is required, not stylistic. `loomml._core` must not import Ray, +and `loomml.env` must not import Ray, so the native boundary and the episode +semantics stay independent of a training framework. A second trainer, an +offline replay tool, or a plain scripted search therefore uses an environment +without acquiring RLlib, and RLlib version movement cannot reach the C++ side. + +### RLlib Environment Definition + +An environment is consumed as an RLlib environment, and the single-agent +environment definition of the Loom RLlib fork is the normative conformance +target. That definition is `gymnasium.Env` with graph observation support, so +conformance is stated against the fork and satisfied through Gymnasium rather +than through a parallel adapter. `loomml.rllib` supplies, per environment: + +- a creator registered through `register_env` that accepts the env-context + configuration and returns one `gymnasium.Env` instance; +- static `observation_space` and `action_space` attributes fixed for the + instance's lifetime; and +- `reset(seed, options)` and `step(action)` returning the five-tuple with + `terminated` and `truncated` reported distinctly. + +Three obligations follow from that target. + +The observation uses the Gymnasium `Graph` space, so a batch of observations is +a batch of `GraphInstance` values with differing node and arc counts rather +than a stackable rectangular array. Upstream RLlib's connectors flatten +fixed-shape spaces and cannot carry that batch; the fork's graph-space support +is what makes this space consumable, and it is the specific capability the fork +exists to add. This document therefore depends on a forked trainer rather than +on an unmodified upstream release, which +[External Dependency Pinning](spec-loom-stack.md#external-dependency-pinning) +records with the pinning and patch-stack terms that choice implies. + +Padding to a fixed extent was rejected for the reason +[The Graph Instance](#the-graph-instance) states, and the fork's graph-space +support is what makes the unpadded space consumable. A ragged batch is the +honest shape of the data. + +Masked actions use the parametric-action convention. The mask and the decision +nodes are members of the observation, not a side channel, an environment +method, or a wrapper attribute, so the connector pipeline carries them to the +model unchanged. The decision nodes and their target arcs are the +available-action embeddings that convention expects. + +The mask enters that space as a fixed-shape `Box` of `uint8` and not as +`MultiBinary`, because `MultiBinary` is one entry per outcome and packing it is +the point. Its shape is a capacity and therefore constant, so it stacks +rectangularly and needs nothing the fork's graph support adds; the mask is the +one observation member the unpadded argument does not apply to, since its +extent never varied with the state to begin with. + +The copy coordinates in the determinism preimage are RLlib's `worker_index` and +`vector_index`, which it passes in the env context across `num_env_runners` +actors and `num_envs_per_env_runner` vector slots; the adapter forwards both. A +run reproduces exactly when the seed and both counts are unchanged. Changing +either count changes which seeds are drawn, which is a property of the sampling +topology rather than a defect, and it is reported rather than hidden. + +### Interaction Contract + +The array contract is zero-copy. Each observation array is exposed as a +buffer-protocol view over environment-owned memory with the element type and +layout declared above, valid until the next call that advances or resets that +environment, and never written by the consumer. A consumer that needs an array +past that point copies it. The environment does not defensively copy on every +step, because observation marshalling is otherwise the dominant per-step Python +cost. RLlib's own connectors copy what they retain, so the buffer lifetime is +compatible with sampling without a defensive copy per step. + +A non-advancing step neither advances nor resets, so it does not invalidate the +buffers: the state being observed is the one already observed, and only the +step and failure scalars change in place. Rebuilding the observation there is +permitted but pointless, and an environment names only which of its scalars +move. + +No layer between the environment and the model unpacks the mask or widens a +narrow column, on the terms [Storage Widths](#storage-widths) sets. Both stay +in the environment's own form until the model consumes them, which +[Masking](spec-ml-core-model.md#masking) and +[Observation Batching](spec-ml-core-model.md#observation-batching) place on the +device. + +Closed native outcome unions map to Python by kind rather than by message. An +ordinary non-advancing or terminal outcome is data returned from `step`, not an +exception, because both are expected transitions. A violated precondition, a +masked or out-of-range action index, a malformed configuration view, and an +owner internal failure raise distinct typed exceptions carrying the exact +reason discriminant. Diagnostic text is presentation and is never parsed. + +One environment instance is owned by one thread. Several vector copies may live +in one process; they share no state, no arrays, and no PRNG stream. An instance +is not inherited across `fork`; an env-runner process constructs its own. + +Every dependency named here is a search-harness dependency, and its exact +revision — including the Ray fork whose environment definition this section +targets, and the Python runtimes this document states conformance against — is +pinned by +[External Dependency Pinning](spec-loom-stack.md#external-dependency-pinning), +which also owns the containment argument that makes a forked trainer tolerable. +This document does not pin them separately. + +## Benchmarking Harness Contract + +Step and reset latency are first-class engineering concerns because their cost +compounds across a search, so every environment ships a harness that measures +them directly, on the removable-projection terms +[Nonsemantic Boundary](#nonsemantic-boundary) sets for every harness here. + +Each environment document names its own harness binary and its own closed list +of step and reset stages, because the stages of a search over drafts and the +stages of a search over one frozen problem have nothing in common. What every +harness owes is the same. + +Reported measures are per-stage wall time at p50, p90, and p99; steps per +second per environment copy; throughput against both `num_env_runners` and +`num_envs_per_env_runner`, since those scale differently and an aggregate +worker count hides which one is saturating; cache hit rate for every cache the +environment declares; the non-advancing rate by the environment's own reason +catalog; +and allocation counts per stage. + +Two breakdowns are required of every harness because an aggregate over either +is misleading. Every measure is reported separately for advancing and for +non-advancing steps: a step that fails early contributes no sample to the +stages after its failure point, so a blended percentile moves whenever the +policy's failure mix moves, and a real regression can hide behind a rising +early-failure rate. And every measure is reported per the environment's own +action or decision partition, because the members of one enumeration do not +cost the same. An environment document may require further breakdowns; it may +not drop these. + +Stage instrumentation is inactive outside the harness. A stage boundary implies +enough clock reads and allocator interception to be a measurable fraction of +the shortest stages, so an environment stepped by a trainer carries none of it, +and harness-reported per-stage times are not comparable to uninstrumented steps +per second. + +The nonsemantic status of wall time, the prohibition on summing concurrent +times, and deterministic work summaries as the cross-machine cost measure are +owned by +[Operational Observations](spec-dse-feedback.md#operational-observations). A +harness reports the deterministic work-unit counts alongside its timings and +states which is which. A regression budget is a ratio against a recorded +baseline tuple of exact identities and configuration digests, never an absolute +duration. + +## Anchor Verification + +Stable tests cover every reference of every live enumeration member resolving +to exactly one node ordinal in the combined space, including a relation-valued +target, under either enumeration encoding; a multi-reference decision exposing +every member of its set or tuple in canonical payload order rather than only +its first; a parallel connection appearing as one node whose count is a feature +and a count-adjusting decision leaving the node inventory unchanged; under +`DecisionNodes`, a decision's target arcs carrying its complete reference set +with the correct `ArcRole` per arc, its out-degree equalling that set's size, +its ordinal within the `Decision` block equalling its `ActionIndex`, and +`decisions` being empty; under `DecisionColumns`, the `Decision` block being +absent, `decisions` carrying exactly one row per live entry and no arcs, a +row's ordinal equalling its `ActionIndex`, and every target-naming column +holding a node ordinal that resolves in the same instance's `graph`; an +environment whose entries have variable target arity being refused the +`DecisionColumns` encoding; two states with different node counts producing +`GraphInstance` values of different extents, with no inert node, no inert arc, +and no extent that is not a real count; every environment's `NodeFeatureColumn` +catalog beginning with `Role` and its `ArcRole` catalog beginning with +`Structural`; a column stored over only the role blocks it applies to, and the +negative-one absence value appearing only within a block the column does span; +every observation array being stored in the narrowest exact form of its value +domain with no catalog declaring a width, and `scalar_features` being the one +exemption; a column-major `nodes` and `edges` rather than a two-dimensional +matrix; the action mask being the only +fixed-length array whose extent is a capacity, covering `enumeration_bound + 1` +outcomes with stopping at the last, packed one bit per outcome at bit `i & 7` +of byte `i >> 3` with every bit above `enumeration_bound` in the final byte +zero, its decision bits at or beyond the live count being clear, and its bit +indices agreeing with the action distribution's outcome indices; two states +with the same admissible set producing byte-identical masks at every live count +including one that is not a multiple of eight; the mask crossing the Python +boundary packed and being unpacked by no layer below the model; an environment +clearing the stop bit for a state in which stopping has no meaning, and a state +with every byte zero being a contract violation rather than an ending, and the +mask of an observation returned alongside an ending being neither composed, +counted as a fallback, nor read by the value head; a state +whose enumeration exceeds +`enumeration_bound` being refused with both the capacity and the required +length reported rather than being shortened; a masked, stale, or out-of-range +index being refused with the state unchanged and no coercion to a legal index +or to a terminal step; every environment defining an episode start override, an +override selecting only from declared inventory, consulting no selection +stream, producing the same episode on any copy, and a start failure under one +being returned as it stands with no redraw and no retry budget consulted; a +non-advancing step invalidating no buffer; an elective stop ignoring the +decision field; the +canonical enumeration order being identical across runs for equal state and +configuration; the step accounting identity holding over a complete episode; +termination and truncation being reported distinctly, a truncation's ending +price being zero, and a terminal state charge applying identically under a +truncation and a termination that leave the same state; a buffer remaining +valid across a call that neither advances nor resets and being invalidated by +one that does; an ordinary non-advancing or terminal outcome being returned as +data +while a precondition violation, a masked index, a malformed view, and an +internal failure each raise a distinct typed exception carrying its reason +discriminant; `loomml._core` and `loomml.env` importing without Ray present; +two environment copies with distinct coordinates drawing disjoint sequences +from one seed; a `reset` seed argument becoming the effective seed and being +recorded as such rather than ignored; and an instrumented harness run producing +the same formal results as an uninstrumented one. + +Tests do not pin the live node, arc, or decision counts of any particular +state, the bound values a profile selects, wall-time numbers, per-decision cost +ratios, allocation counts, diagnostic text, or Python formatting. diff --git a/docs/spec-ml-core-model.md b/docs/spec-ml-core-model.md new file mode 100644 index 000000000..59e9fca70 --- /dev/null +++ b/docs/spec-ml-core-model.md @@ -0,0 +1,646 @@ +# ML Model Core + +This document defines the contract every Loom learned search policy shares: how +a batch of observations becomes one disjoint-union graph, how raw observation +columns become encoder inputs, the graph-transformer trunk over them, the graph +context both heads read, the value head, the masking discipline, the action +distribution, the checkpoint boundary, and what every forward-pass benchmark +owes. + +It defines no policy head. Which actions exist, how they are anchored, and what +scores them are owned by the model documents that build on this one: + +- [ML DSE Model Architecture](spec-ml-dse-model-architecture.md) scores + candidate-generator decisions for + [ML DSE Environment](spec-ml-dse-environment.md); and +- [ML PnR Model Architecture](spec-ml-pnr-model-architecture.md) scores typed + Place and Route Actions for [ML PnR Environment](spec-ml-pnr-environment.md). + +A model is a search policy, not a Loom semantic authority. It proposes actions; +the environment decides what those actions mean and whether they are +admissible, and every legality, mappability, and objective fact remains owned +where [ML Environment Core](spec-ml-core-environment.md) and the environment +documents place it. A model checkpoint is not an Artifact, is not +`EvaluationEvidence`, and never becomes a registered prediction contract. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [ML Environment Core](spec-ml-core-environment.md) owns the observation + container, the combined node-and-link space, the enumeration encodings, the + action surface, the action mask, and the reward this model's value head + regresses toward; +- each environment document owns its own node, arc, scalar, and decision column + catalogs, and the enumeration whose entries a policy head scores; +- [Evaluation and DSE](spec-dse-feedback.md#objectives-and-quality-gates) owns + `ExactAffineQuantization` and the objective algebra whose codes enter as + features; +- [Evaluation and DSE](spec-dse-feedback.md#model-parameters-and-training) owns + `ModelParameterContractRef` and the registered prediction contracts, which a + policy is not; +- [Resolved Configuration](spec-config-ssot.md#component-views) owns + component-view framing, canonical view bytes, and `component_view_digest`; +- [Operational Observations](spec-dse-feedback.md#operational-observations) + owns the nonsemantic status of wall time and deterministic work summaries as + the cross-machine cost measure; and +- [Full-Stack Architecture](spec-loom-stack.md#external-dependency-pinning) + owns the exact revisions of every search-harness dependency, including the + Ray fork whose module and distribution contracts this document targets. + +The `RLModule` interface, its column names, and its distribution protocol are +owned by that external dependency. This document states which of its +obligations bind a model and does not restate, extend, or version them; the +training documents configure it. + +This document owns only the parts that are the same in every model: what +happens between an observation and the values `Columns.ACTION_DIST_INPUTS` and +`Columns.VF_PREDS` carry, the framing every model configuration view follows, +and the obligations every forward-pass harness satisfies. + +## Module Boundary + +A model is one `TorchRLModule` implementing `ValueFunctionAPI`, in the new +RLlib API stack: + +```text +LoomRLModule(TorchRLModule, ValueFunctionAPI) { + setup() + _forward_inference(batch) -> {ACTION_DIST_INPUTS} + _forward_exploration(batch) -> {ACTION_DIST_INPUTS} + _forward_train(batch) -> {ACTION_DIST_INPUTS, VF_PREDS, ...} + compute_values(batch, embeddings) -> VF_PREDS + get_train_action_dist_cls() / get_exploration_action_dist_cls() + -> the model's action distribution +} +``` + +The three forward entry points share one encoder pass and differ only in what +they return and in whether exploration-time logit shaping is applied. A +`_forward_train` call must produce logits identical to the +`_forward_exploration` call that collected the same batch, given the same +parameters and the same recorded masks; any divergence makes the policy ratio +wrong for every on-policy update built on it. + +`compute_values` accepts precomputed embeddings so that a training step +evaluates the encoder once. Recomputing the encoder for the value head is a +correctness-neutral waste of the dominant cost in the forward pass, and it is +the largest single saving available in the training path. + +## Observation Batching + +The environment emits a variable-size `GraphInstance` per state, so a batch is +ragged. The model consumes it as one disjoint-union graph in the standard +message-passing form: + +```text +BatchedGraph { + node_columns: per-column array, each at its declared width + arc_columns: per-column array, each at its declared width + edge_index: int64 [2, total_arcs] + batch: int64 [total_nodes] + ptr: int64 [batch_size + 1] +} +``` + +Per-instance node ordinals are offset into the union, and `batch` records each +node's originating instance so pooling and per-instance normalization stay +correct. `ActionIndex` is a per-instance ordinal, so the model maps it through +`ptr` and the instance's own offset rather than treating it as a union-wide +index; conflating the two silently scores another episode's actions. + +An environment using the `DecisionColumns` encoding supplies a second ragged +instance, and it batches the same way into its own union with its own `batch` +and `ptr` vectors. A column of that union that names a node holds a +per-instance node ordinal and is offset into the node union before it is used +as an index, for the same reason `ActionIndex` is. + +Nothing in the batch is padded. The observation's fixed-length arrays batch as +ordinary rectangular tensors. + +The batch carries columns, not feature matrices. Arrays arrive in the forms +[Storage Widths](spec-ml-core-environment.md#storage-widths) declares and stay +in them: each column is concatenated across instances at its own declared +width, and no dense `[rows, feature_dim]` tensor of widened columns is ever +built. Building one would restore exactly the bytes that section exists to +remove, on the two most numerous extents, and would do it on the device where +the encoder then has to read around it. There is no node-feature matrix in this +record because the first thing the model does to every column is embed or +project it separately, and the vector those produce is the encoder's input. + +`edge_index` is the one widened tensor. The scatter operations require `int64`, +so it is produced on device from the observation's narrower `edge_links` — a +tensor the model builds, not bytes it received. + +A large union may be processed in chunks when device memory requires it. A +chunked pass must produce logits and values identical to an unchunked one; a +chunking scheme that changes results is a defect, not a memory strategy, and +per-instance pooling makes the identity achievable because no quantity is +reduced across instances. + +## Feature Embedding + +Raw observation columns are not fed to the encoder as numbers. Every +categorical column is an ordinal in a closed owner catalog, and an ordinal +consumed as a scalar asserts an order and a distance the catalog does not have: +it would let the encoder interpolate between two unrelated entity kinds. Each +is therefore embedded through its own table, sized from its owner catalog's +cardinality. + +Numeric columns are projected rather than embedded, but never consumed raw. A +column the environment marks absent with negative one contributes a zero +magnitude and a set absence indicator, because feeding negative one as a +magnitude makes absence a large negative quantity the encoder cannot +distinguish from a real extreme value. Magnitudes that span orders of magnitude +are projected through a signed logarithmic transform before their linear layer. + +A capacity and a usage of the same resource are supplied with their occupancy +ratio alongside the two magnitudes, since headroom is the quantity a placement +or a hardware decision actually trades and the encoder should not have to learn +division. + +A capacity and a usage column appended per `FabricResourceStateRef` member are +projected per resource state and reduced to one resource embedding per node, so +the node vector width does not grow with the resource catalog. The environment +core appends that group identically for every environment, so the reduction is +stated once here rather than in each model document. + +`objective_codes` are exact bounded integers whose bounds are known: each +dimension's `ExactAffineQuantization` declares its index range, so a code maps +to its exact position in `[0, 1]` with no estimated scale and no running +normalizer. Objective position enters as a graph-level feature so the encoder +can condition a local decision on the current global position. + +Scalar features enter as graph-level features. A pair of a counter and its +bound is supplied as its ratio as well as its magnitudes, so remaining budget +is directly available rather than inferred. + +All embeddings and projections are concatenated per node and passed through one +input projection to the encoder width, so the encoder sees one uniform node +vector regardless of role. A graph-level feature may instead be projected once +per instance and added to that result, which is exact because one linear layer +over a concatenation is the sum of its per-block projections; materializing a +duplicated row per node and multiplying through it costs with node count where +the equivalent form costs with batch size. Arc embeddings are concatenated to +the arc feature width and supplied to the encoder as edge features. + +## Encoder + +The encoder is a stack of graph-transformer layers with edge-aware attention. +Attention is over graph neighborhoods rather than over all node pairs, which is +what makes it tractable on states whose node count varies by an order of +magnitude across a curriculum, and what makes the combined node-and-link space +meaningful: a connection node attends to exactly the occurrences it joins. + +```text +GraphTransformerStack { + layers: positive integer + width: positive integer, divisible by head count + heads: positive integer + feedforward: positive integer + dropout: ratio in [0, 1) + edge_dim: arc embedding width + jumping_knowledge: None | Concat | Max +} +``` + +Each layer is pre-normalized: the layer normalizes its input, applies +multi-head edge-conditioned attention, applies dropout, and adds the unmodified +residual, then does the same around a position-wise feed-forward sublayer. +Pre-normalization is required rather than optional, because post-normalization +destabilizes gradient flow at the depths and graph sizes this search reaches. A +final normalization follows the stack so the heads receive bounded features. + +The feed-forward sublayer is a departure from the prior architecture and is +stated as one. That stack was attention and residual only, so its single +nonlinearity was the attention softmax and depth bought message-passing hops +rather than capacity. Separating the two is what lets a configuration reach +further across a graph without also being forced to add parameters, and add +parameters without also being forced to reach further. + +Arc features are supplied to every layer. Environments distinguish structural +relations from placement, routing, and target relations, and an encoder that +ignored arc features would treat a placement relation, a hardware connection, +and a decision's reference to its target as the same edge. This is also a +departure from the prior architecture, and a sharper one than it appears: that +system had an edge projection and an edge-dimension-aware convolution, but no +arc features ever reached the convolution, so the projection was untrained +weight in the optimizer. Arc features here are load-bearing, not optional. + +Jumping-knowledge aggregation over per-layer outputs is supported so the model +can select an effective neighborhood radius per node instead of committing to +the stack depth. `Max` aggregation preserves the encoder width; `Concat` +multiplies it by the layer count, and every head width sized from the encoder +width must be sized from the aggregated width instead. A new aggregation +appends and must state its output width. A configuration whose head widths and +aggregated encoder width disagree is invalid at adoption rather than at the +first forward pass. + +One shared encoder produces the graph embedding, and every head reads it. Any +per-head refinement is a small stack applied after the shared trunk, never a +second trunk from raw features: separate trunks discard the representation +sharing that makes the value head's signal useful to the policy, and double the +dominant cost of the forward pass. + +## Graph Context + +Every head needs a whole-graph summary, and a single mean over nodes is a poor +one because graph size varies across the curriculum. The context is a +multi-scale pooling of the node embeddings, per batch instance: + +```text +context = project(concat(mean_pool(h), sum_pool(h), max_pool(h))) +``` + +`mean` is size-invariant, `sum` carries extent, and `max` carries the single +most salient node; together they let a head distinguish a small state from a +uniformly-scaled large one. The sum branch is scaled down by the square root of +the instance's node count and the projection is followed by normalization, +because an unscaled sum grows with node count and would otherwise make context +magnitude a proxy for graph size and drive logit scale with it. + +Pooling is per instance using the `batch` vector. Pooling across the union is a +defect that leaks one episode's state into another's decisions. + +## Value Head + +The value head reads the shared graph embedding through the same context +pooling and emits one scalar per instance. It is an ordinary multilayer +perceptron over the context, whose dropout comes from the encoder record and +whose initialization gain is one of the constants a model view declares. + +The value head predicts the return of the environment's reward, which is a +signed exact-integer energy difference scaled once at the adapter. Its output +is bounded to a configured symmetric range. That bound is a stability +mechanism, not a semantic claim: a value estimate that escapes the reachable +return range destabilizes the advantage estimate long before it becomes visible +as a policy failure. Its cost is stated rather than hidden — a saturated +estimate has no gradient — which is why the bound is checked against the +reachable return range a training document declares, and not chosen. What that +range is, and how a `value_bound` below it is refused, are owned by +[The Reachable Return Range](spec-ml-core-training.md#the-reachable-return-range). + +The value head never reads the action mask, any per-action feature, or anything +else that distinguishes one available action from another. It estimates the +state's worth, and giving it action-set information invites it to model the +enumeration rather than the state. + +## Masking + +Masking is applied additively to logits, using a large finite negative +constant. Negative infinity is not used: it produces undefined entropy and NaN +gradients when a whole group is masked, and every masked-everything case must +degrade to a defined distribution rather than to a NaN. + +Two mask sources compose, in this order: + +1. the environment's `action_mask`, which is authoritative and already covers + slots beyond the live count and any entry the environment itself cleared; + and +2. advisory masks, which remove actions the configuration declares obviously + unproductive. + +The first arrives packed, in the layout +[Action Surface And Masking](spec-ml-core-environment.md#action-surface-and-masking) +fixes. It is unpacked on the device that will consume it, after the batch has +been transferred and as part of the forward pass, so the packed form is what +crosses the sampler, the sample buffer, and the transfer, and only the expanded +form the logits actually need is ever materialized. Unpacking is a shift and a +test over a tensor whose width is the action capacity — trivial against the +encoder it sits behind, and cheaper than the memory traffic it removes. + +Composition is bitwise. The environment's mask and each advisory mask are +combined by conjunction, which is not an implementation note but the exact +statement of the first rule below: an AND can only clear bits, so a +model-side mask that tried to add an action cannot express itself in the +composition at all. + +The first is always applied and needs no model-side recomputation: a slot the +environment cleared is a slot with no live entry, and rederiving that bit per +forward pass would scatter ordinals into a capacity-wide tensor in the hot path +to reproduce a value that arrived correct. The second is where liberal masking +is available and also where it is bounded, because an advisory mask that +removes a legal action is a policy commitment rather than a correctness +statement, and must never be presented as proof that an action is bad. + +Five rules keep masking from corrupting training. + +Masks never add actions. A mask may only clear a bit the environment set; a +model-side mask that sets a bit the environment cleared proposes an action the +environment will refuse as masked. + +A fully masked state falls back. If composing the masks leaves no admissible +outcome, the composition is discarded and the environment's mask alone is used, +and the fallback is counted. Silently emitting a uniform distribution over +impossible actions produces a step the environment refuses for reasons no +diagnostic explains. + +The mask used at collection is recorded and reused at update. Advisory masks +may depend on configuration that changes across a run, so recomputing them at +training time can mask an action that was available when it was sampled. The +resulting log-probability is not the behavior policy's, and the importance +ratio is silently wrong. The mask travels in the batch, packed, in the +environment's own layout: a batch that carried the expanded form would spend +eight times the memory on the one column it retains for the whole update, and +the recorded bits are the same bits either way. + +Mask composition and distribution construction run in full precision, outside +any reduced-precision region, on an explicitly cast copy of the logits. The +additive constant is far outside the range of common half-precision formats, so +a masked logit computed in half precision saturates to infinity and +reintroduces exactly the undefined entropy and NaN gradients that choosing a +finite constant avoided. The related hazard is worth naming: under mixed +precision a trunk may remain in one precision while a head's linear layers emit +another, so a head that scatters into a buffer typed from the trunk fails on a +dtype it did not expect. + +A logit vector materialized at the action capacity is filled at its unreal +slots with the masking constant, never with zero. A trainer rebuilds the +distribution from `ACTION_DIST_INPUTS` to compute the update log-probability, +so a zero fill gives those slots real probability mass and normalizes the +rebuilt distribution differently from the one that was actually sampled. The +two log-probabilities then disagree for reasons no loss curve reveals. + +That is a rule about a vector inside a forward pass, not about what the batch +retains. `ACTION_DIST_INPUTS` travel at the live count, ragged, exactly as +`decisions` does. Retaining them at the capacity would put four bytes per +outcome per sample beside a mask that +[Storage Widths](spec-ml-core-environment.md#storage-widths) just reduced to +one bit, held for as many epochs as the algorithm takes, and the slots so +retained are at the masking constant and contribute nothing to the ratio or to +the KL term. The rebuilt distribution is identical either way, because a slot +at the masking constant and an absent slot carry the same probability. + +Logits are not clipped. Clipping bounds them by zeroing the gradient on exactly +the actions the policy is most confident about, and the masked distribution is +already bounded without it. A model that needs clipping to stay finite has a +scale problem in its heads that clipping would hide. + +Hierarchical masking is consistent by construction: an anchor group is masked +exactly when all its members are masked, so a group never retains probability +its members cannot receive, and a member never receives probability from a +masked group. + +## Action Distribution + +The distribution is a custom RLlib Torch distribution over the environment's +two-component action space, exposing sampling, log-probability, entropy, and +KL. Its inputs are the masked joint logits, and it samples one joint outcome +and emits the `{decision, stop}` pair that outcome encodes. + +The head emits one joint distribution over the `enumeration_bound + 1` outcomes +those two components can jointly take: + +```text +outcome i < enumeration_bound -> { decision: i, stop: 0 } +outcome enumeration_bound -> { decision: 0, stop: 1 } +``` + +`decision` is ignored by the environment when `stop` is one, so the emitted +value is fixed at zero to keep the encoding one-to-one; a sampler that emitted +a live index alongside a set stop flag would produce two encodings of one +action and split its probability across them. Log-probability and entropy are +computed over the joint outcome, never per component, because the components +are not independent: exactly one of the two carries the action. + +The single joint softmax is what makes acting and stopping mutually exclusive. +Two independent per-component distributions would let a policy simultaneously +raise an action's probability and the stop probability, which the environment +resolves by discarding the action, so the gradient on that action would be +attributed to something that never occurred. + +Entropy is computed over admissible outcomes only. Including masked slots at +the mask constant contributes a vanishing but nonzero term that scales with +`enumeration_bound` rather than with the live count, which makes the entropy +bonus depend on the action-space capacity instead of on the choice actually +available. + +Sampling is deterministic given a seed and the logits. The model owns its own +sampling randomness; an environment's determinism policy covers episode +construction only. + +## Policy Head Factorization + +A policy head scores each live entry from the embeddings of the nodes that +entry names. Each model owns what its own terms read; what every model shares +is the shape those terms compose in. + +A head factors an entry's logit into an anchor term over what the entry acts on +and a selection term over what it selects, and entries sharing an anchor form +one anchor group. `s_anchor` is constant within a group by construction, +because the anchor includes every input the anchor term reads, which is what +makes the factorization a hierarchy rather than a reparameterization: + +```text +P(e) = P(anchor(e)) * P(e | anchor(e)) + +P(anchor) = softmax over anchor groups of + s_anchor(g) + logsumexp over members of s_select +P(e | anchor) = softmax over that group's members of s_select(e) +``` + +The two-level form and the flat softmax over `logit(e)` induce the same +distribution, so an implementation may compute either; the hierarchy is +normative for how the scores compose, not for the order of arithmetic. Because +`s_anchor` is constant within a group it may be evaluated once per group and +broadcast, and a harness reports the mean anchor-group size alongside its head +stage, since the saving is exactly proportional to that mean and is largest in +the configurations where the hierarchy is doing the most work. + +What is not permitted is a head that scores an entry from its anchor alone with +no selection term, which collapses every alternative on one anchor into one +indistinguishable action. + +Stop is one additional logit computed from the graph context alone, since +electing to stop is a property of the state rather than of any entry. + +## Parameters And Checkpoints + +A checkpoint is the parameter tensors plus the exact model configuration that +shapes them. Loading requires an exact match of every dimension the +configuration fixes, including the embedding table sizes, which are derived +from owner catalog cardinalities. A catalog that gains a member invalidates the +tables sized from it, and the load fails rather than silently truncating or +zero-extending a table whose ordinals have shifted. + +A model checkpoint is not a `ModelParameterBundle` and its reference is not a +`ModelParameterContractRef`. Those name registered prediction contracts whose +outputs become metric predictions inside `EvaluationEvidence`, subject to +support-region and leakage rules. A policy network predicts which action to try +next; it produces no metric, enters no Evidence, and is never consulted for a +value a quality gate or an objective dimension reads. Confusing the two would +put an unvalidated search heuristic inside the evaluation path. + +A checkpoint carries parameters, and a consumer that loads one for a purpose +other than resuming its own run takes the parameters and nothing else. +Optimizer moments, a learning-rate position, and any algorithm-owned running +statistic belong to the run that produced them and are meaningless under a +different objective; a training document that sequences one algorithm after +another states that boundary, and this document fixes only that a checkpoint is +able to be read that way — its parameter tensors are addressable without its +training state. + +## Model Configuration Framing + +Each model's shape is one immutable component view of its own, following the +framing owned by [Component Views](spec-config-ssot.md#component-views). This +document fixes only three properties of any such view. + +Every embedding table's size is derived from its owner catalog's cardinality +rather than declared, so a view carries widths and the catalogs fix heights. + +A view sizes the parameter tensors, which is why no other view may carry one of +its fields. What a checkpoint pins is the tensor-shaping projection of the +view: the embedding widths, the encoder record, the head widths, and any field +that selects between scorers of different parameter shapes. The rest of the +view — the value bound and the advisory mask set — is recorded on the +checkpoint and checked for reporting, not required to match. + +The split is the same one +[Configuration Split](spec-ml-core-training.md#configuration-split) draws for +the training view, and for the same reason: a value bound is a clamp scalar and +an advisory set is a masking policy, and neither changes the shape of a single +tensor. Pinning them would make a sweep over either discard every checkpoint +the run had produced and retrain from scratch, which confuses a parameter-shape +constraint with a policy preference. A run that raises its `step_bound` mid-run +may have to raise its value bound with it, and that must not invalidate the +parameters it has already learned. + +A view declares the encoder record, the head widths, the value bound, the +advisory mask set, and the initialization constants. `HeadWidths` is one +positive hidden width per head the model declares, named for that head, so a +model with two policy heads and a value head declares three; the record is not +fixed here because the heads are not. + +`AdvisoryMaskRef` names one member of the closed catalog of advisory masks a +model document declares, and `InitializationConstants` is that document's +per-tensor initialization gains. Both are per-model for the same reason the +heads are. Two models never share one +view, because their catalogs differ and a shared view would size a table for a +catalog the other model does not have. + +## Forward Benchmarking Obligations + +A model runs once per environment step during rollout collection, so its +forward latency multiplies by every step of every episode of every worker. +Every model therefore ships a harness that measures the forward pass directly, +on the removable-projection terms +[Nonsemantic Boundary](spec-ml-core-environment.md#nonsemantic-boundary) sets +for every harness in this stack. + +Each model document names its own harness binary and its own closed list of +forward stages, because the heads differ and a shared stage list would either +be too coarse to locate a regression or carry stages one model does not have. +Stages partition the pass: none contains another, so the stage sum is the pass. +What every harness owes is the same. + +Unpacking the mask and widening a narrow column are not stages. A cast is +required to happen immediately before the operation that consumes it and only +for the columns that operation reads, so it is inside `embed` for a feature +column and inside the masking stage for the mask; timing it separately would +either double-count those stages or force an unfused materialized copy, and the +copy is the capacity-width tensor +[Storage Widths](spec-ml-core-environment.md#storage-widths) exists to prevent. +An instrumented run that materialized it would also stop measuring the +uninstrumented one. + +What every harness reports instead is the quantity that makes the layout +falsifiable: bytes unpacked and bytes cast, attributed to the stage that did +it, together with a reference run that widens at the boundary rather than on +device. The difference between the two is the layout's actual value, and it is +a number rather than an argument. A layout claimed to save bandwidth and +measured by a stage that only exists when the saving is discarded would answer +the wrong question. + +Two regimes are measured separately and never averaged. Rollout inference is +latency-bound: batch size is one instance per environment copy, graphs are +small, and fixed per-call overhead dominates. Training is throughput-bound: +batches are large, graphs are batched into one union, and arithmetic dominates. +An optimization that helps one routinely harms the other, so one number for +both hides which is binding. + +The three forward entry points are measured separately, and `compute_values` is +measured both with and without precomputed embeddings, because the encoder-once +rule is the largest single saving in the training path and a harness that never +measures it cannot show that it holds. + +Cost is reported as a function of state size, never as a single number. A +curriculum moves states across an order of magnitude, so a latency measured on +a small one predicts nothing about a large one. Each stage is reported against +the size quantities that drive it, and a stage whose measured growth exceeds +its expected order is a defect worth finding. + +The split against the environment step is reported before anything else. +Sampling throughput is set by the sum of the environment step and this model's +forward pass, and a forward pass that is a small fraction of a step can be made +twice as fast for no throughput gain at all. Reporting the split first is what +prevents optimizing the smaller half. + +Every optimization is validated against an unoptimized reference on the same +inputs and parameters, and a configuration whose logits or values differ beyond +a declared tolerance is reported as a failed configuration rather than as a +faster one. Inference-time and training-time numerics must agree within that +same tolerance: a model whose rollout path is optimized differently from its +learner path produces log-probabilities the learner did not produce, and the +resulting importance ratio is wrong for the same reason a recomputed mask makes +it wrong. Speed obtained by letting the two paths diverge is not speed; it is a +silent change to the objective being optimized. + +Two optimizations carry traps general enough to state here. A compiled or +graph-captured execution path that treats node and arc counts as static will +recompile continuously and run slower than the eager path, and the +recompilation is invisible in a steady-state average, so recompilation counts +are reported alongside latency and any bucketing that reduces them is reported +with the bucket boundaries it introduces. And reduced precision excludes mask +application, for the reason Masking states, so a harness verifies that masked +logits remain finite in every precision configuration it measures. + +A regression budget is a ratio against a recorded baseline of exact parameter +and configuration identities at a fixed state size, never an absolute duration. +The primary reported figure is the model's share of rollout wall time, because +that is the quantity an improvement has to move to matter. + +## Conformance Anchors + +Stable tests cover a categorical column reaching the encoder through an +embedding table rather than as a scalar, and an absent numeric column being +distinguishable from a real extreme value; a capacity and usage pair reaching +the encoder with their occupancy ratio; an objective code mapping to its exact +position in `[0, 1]` from its declared quantization bounds; a graph-level +feature added once per instance producing the same result as a per-node +broadcast; per-instance pooling never mixing nodes across a batched union; a +node-naming column of a batched decision union being offset into the node union +before use; a chunked pass reproducing an unchunked pass exactly; an +`ActionIndex` resolving to its own instance's entries; a configuration whose +head widths disagree with a `Concat` jumping-knowledge width being refused at +adoption; the value head producing identical predictions whether or not it is +given precomputed embeddings, and never varying with the action mask; a masked +slot receiving no probability and contributing no entropy; a fully masked +composition falling back to the environment mask and counting the fallback; a +model-side mask that sets a bit the environment cleared being refused, and mask +composition being a conjunction that cannot express such a mask at all; the +environment's mask being unpacked on the device that consumes it and reaching +the batch and the sample buffer packed; an unpacked mask agreeing bit for bit +with the environment's packed one, including across the pad bits of a final +byte; a mask recorded at collection being the mask applied at update, and a +batch whose recomputed mask differs being rejected rather than used; unreal +slots of a +materialized logit vector carrying the masking constant at its unreal slots, +`ACTION_DIST_INPUTS` being retained at the live count rather than at the +capacity, and a distribution rebuilt from those inputs reproducing the sampled +distribution's log-probability; masked logits remaining finite in every +supported precision +configuration and mask composition running outside any reduced-precision +region; logits reaching the distribution unclipped; a sampled stop outcome +decoding to a zero `decision` component, and the joint distribution never +assigning probability to a set stop flag beside a live index; a stop logit +competing in the same normalization as the action logits; `_forward_train` and +`_forward_exploration` producing identical logits for equal parameters, +observations, and masks; `_forward_inference` computing no value; a checkpoint +load failing when a catalog cardinality no longer matches the embedding table +it sized; and every optimized configuration reproducing the unoptimized +reference's logits and values within the declared tolerance. + +Tests do not pin layer counts, widths, head counts, initialization constants, +advisory mask sets, learned parameter values, wall-time numbers, device +placement, precision configurations, or tolerance values. diff --git a/docs/spec-ml-core-training.md b/docs/spec-ml-core-training.md new file mode 100644 index 000000000..bffe5605e --- /dev/null +++ b/docs/spec-ml-core-training.md @@ -0,0 +1,902 @@ +# ML Training Core + +This document defines the contract every Loom reinforcement-learning training +run shares: how a run's settings split across immutable component views, how an +algorithm is bound and how a run made of several algorithm stages hands off +between them, how hyperparameters and schedules are expressed exactly, what an +episode statistic may be computed from, how a fixed test set is executed and +aggregated, and what a checkpoint and a run identity are. + +It fits no particular policy to no particular environment. Which search is +being trained, what a corpus of training instances is, and which statistics are +worth reporting about that search are owned by the training documents that +build on this one: + +- [ML DSE Training](spec-ml-dse-training.md) fits the design-space policy; and +- [ML PnR Training](spec-ml-pnr-training.md) fits the place-and-route policy. + +A training run is a search-harness activity. It publishes no Artifact, acquires +no Evidence, and produces no candidate. Its outputs are a policy checkpoint, +which +[Parameters And Checkpoints](spec-ml-core-model.md#parameters-and-checkpoints) +already establishes is not a registered prediction contract, and a stream of +removable statistics. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [ML Environment Core](spec-ml-core-environment.md) owns the episode + protocol's shape, the action surface, the reward boundary, the step + accounting identity, the termination-versus-truncation rule, and the + determinism and copy-coordinate contract every seed stream derives from; +- [ML Model Core](spec-ml-core-model.md) owns the module boundary, the masking + discipline, the action distribution, and the checkpoint's parameter and + configuration match; +- each environment document owns its own resolved configuration view, its + priced outcomes, its terminal reasons, and its trajectory record; +- each model document owns its own configuration view and the tensors it sizes; +- [Objectives and Quality Gates](spec-dse-feedback.md#objectives-and-quality-gates) + owns `ObjectiveDimension`, `ExactAffineQuantization`, directed codes, + `ObjectiveVector`, `WeightedLevel`, and `SearchEnergyRef`; +- [Model Parameters and Training](spec-dse-feedback.md#model-parameters-and-training) + owns the training of registered prediction contracts, which is a different + activity from this one and shares no record with it; +- [Evaluation Metrics](spec-evaluation-metrics.md#metric-registry) owns + `ExactRatio`; +- [Component Views](spec-config-ssot.md#component-views) owns view framing, + canonical bytes, and `component_view_digest`; +- [Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism) + owns the seeded PRNG protocol; +- [Operational Observations](spec-dse-feedback.md#operational-observations) + owns the nonsemantic status of wall time; and +- [External Dependency Pinning](spec-loom-stack.md#external-dependency-pinning) + owns the Ray fork revision and its patch stack. + +This document owns only the configuration split and its composition rule, the +nonsemantic boundary, the algorithm binding surface and the stage sequence, the +hyperparameter and schedule expression, the reachable-return-range obligation, +the reward adapter, the rollout and learner topology, the stage invariance +rule, the statistics production rule and the environment-independent +statistics, the test protocol, checkpoint and run identity, the reproduction +claims, and the harness surface. + +## Nonsemantic Boundary + +A training run is not an evaluation, a selection, a promotion, or a candidate +generation. It publishes no Artifact, no `EvaluationRequest`, no +`EvaluationEvidence`, no `InvocationManifest` record, and no lineage edge. A +statistic, a test score, a checkpoint, and a log stream are removable +projections: regenerating them may change presentation but must preserve every +referenced semantic fact, and no consumer may treat one as a semantic result. + +A result a training run discovers becomes real only through the replay path its +environment document defines. A test score never gates promotion, never enters +a quality gate, and never selects a candidate; it selects a checkpoint, which +is a search heuristic. + +## Configuration Split + +A run's settings are several separate immutable component views plus one that +binds them, each following the framing, canonical byte representation, and +digest contract owned by +[Component Views](spec-config-ssot.md#component-views). Every training document +declares its own descriptor bytes, qualified by the search it fits, because two +runs over different searches share no view and an unqualified descriptor would +let one be adopted as the other. + +The run view binds one digest of each participating view and nothing else. +Every training document declares its own, and each carries at least a training +view, a model view, and a test-set view; what else it binds depends on where +its training instances come from. + +Composition is by digest rather than by inlining, and three rules follow from +it. + +No field appears in two views. A quantity is owned by exactly one view, and a +view that repeats another's field creates two sources for one fact that a +future edit will silently disagree about. In particular a training view carries +no episode bound and no observation policy; those are the environment's, and a +run that wants them different binds a different environment view. + +Changing a hyperparameter does not disturb the environment digest. Every +environment declares some cached asset whose key names nothing a training view +carries, so a hyperparameter sweep leaves that cache valid. Had the two been +one view, every sweep would have discarded a cache that takes real work to +warm, and the sweep's first iterations would have measured a cold start rather +than learning. + +A checkpoint pins the model digest alone. Loading requires an exact match of +the model view, because that is what sizes the parameter tensors; it requires +nothing of the training view, because fine-tuning a checkpoint under different +hyperparameters is an ordinary operation and refusing it would confuse a +parameter-shape constraint with a policy preference. The environment digest is +recorded rather than required, since a checkpoint remains loadable against a +different instance pool but not against a different action-space shape. + +Adoption validates each view independently, then validates the run view's +cross-view conditions. A run whose views are individually valid and jointly +inconsistent fails at adoption, not at the first step that notices. + +### Inventory Identity + +Every inventory a view names — instances, corpora, demonstrations, test cases — +is enumerated by exact identity. A run never names a directory, and no +inventory is ever resolved by globbing a path. A globbed set changes when a +file lands in it, so two runs that recorded the same configuration would have +trained on different data with nothing in either record to distinguish them, +and the digest that is supposed to identify a configuration would identify only +its spelling. + +Where an inventory is generated rather than named, generating it again from the +recorded seed, protocol, and view digests reproduces it exactly. No host +entropy, no wall clock, and no container iteration order participates. A +generator that cannot be re-run to the same set makes its own view's digest +meaningless, since the digest would then name a recipe with more than one +result. + +Each training document names the inventory types its run carries and the +identity by which each is compared; the rule that they are enumerated and +regenerable is this one. + +## Training Configuration + +```text +ResolvedTrainingConfigView { + stages: ordered nonempty sequence + topology: RolloutTopology + reward_adapter: RewardAdapter + logging: LoggingPolicy + evaluation: EvaluationSchedule + checkpointing: CheckpointPolicy +} + +RolloutTopology { + num_env_runners: uint32 + num_envs_per_env_runner: positive uint32 + num_learners: uint32 + rollout_fragment_length: positive uint64 | AutoFromBatch + total_env_steps: positive uint64 +} + +RewardAdapter { + scale: ExactRatio + advantage_standardization: PerMinibatch | None +} +``` + +A training document instantiates this record under its own descriptor and adds +no field to it. + +The view carries no seed. Every stream a run draws from is seeded elsewhere and +by something narrower: an episode's randomness by the environment view's +[Determinism And Copy Coordinates](spec-ml-core-environment.md#determinism-and-copy-coordinates), +and a test case's +residual randomness by its own `case_seed`. A seed here would seed nothing that +is not already seeded, and would be ambiguous with the environment view's +wherever a reader met the two together. + +### Algorithm Binding + +Training uses the algorithm implementations of the pinned Ray fork. Loom +authors no policy-gradient loss, no advantage estimator, no clipping rule, no +KL control, and no imitation weighting. The value of grounding a run in an +existing implementation comes entirely from not modifying it: a divergence +between a Loom-authored update and the published algorithm is invisible in a +loss curve and is the first thing a failed run would otherwise have to rule +out. + +```text +AlgorithmBinding = + Ppo(PpoBinding) // 0 + | Marwil(MarwilBinding) // 1 + +PpoBinding { + gamma: Schedule + lambda: Schedule + clip_param: Schedule + vf_clip_param: ExactRatio + vf_loss_coeff: Schedule + entropy_coeff: Schedule + kl_coeff: Schedule + kl_target: ExactRatio + learning_rate: Schedule + grad_clip: ExactRatio + grad_clip_by: GlobalNorm | Value + train_batch_size: positive uint64 + minibatch_size: positive uint64 + num_epochs: positive uint64 +} + +MarwilBinding { + beta: ExactRatio + gamma: Schedule + vf_loss_coeff: Schedule + learning_rate: Schedule + grad_clip: ExactRatio + grad_clip_by: GlobalNorm | Value + train_batch_size: positive uint64 + minibatch_size: positive uint64 + num_epochs: positive uint64 + advantage_norm_update_rate: ExactRatio + advantage_exponent_clamp: ExactRatio +} +``` + +Ordinals are stable; a new algorithm appends. The union exists because an +offline stage and an online stage are the same run over the same policy, and a +document whose algorithm field admitted only one of them could not express that +at all. + +An offline algorithm reads demonstrations rather than sampling the environment. +Which view supplies them is not a field of the binding: the run view already +binds one digest of each participating view, so a binding that named the corpus +again would be a second source for a fact already fixed. A training document +whose run binds an offline stage names the demonstration view there, and states +what a valid one contains. + +`MarwilBinding.beta` at zero is behaviour cloning and at one is full advantage +weighting. That distinction is a property of the demonstrations rather than a +preference: on demonstrations whose returns barely vary, the exponential +weighting collapses toward one and the run is cloning whatever `beta` says, so +a training document states which regime its demonstration source supports +rather than leaving `beta` to be tuned into meaninglessness. + +`advantage_exponent_clamp` is normative rather than a tuning knob. The +advantage weight is an exponential, and its argument grows with the advantage +estimate an untrained value head produces; past a modest bound the exponential +leaves the range of a single-precision float and every weight becomes infinite +in one step. Clamping the exponent bounds the weight without changing its +ordering. + +`vf_loss_coeff` is stated against the policy loss's magnitude rather than +defaulted. A masked per-node policy loss and a scalar value loss differ by +orders of magnitude, and a coefficient chosen without regard to that difference +gives the value head effectively no gradient, whereupon it regresses to the +mean and every advantage the imitation weight reads is noise. + +Loom supplies exactly four things to whichever algorithm a stage binds, each +owned elsewhere and none of them the update itself: the environment through +`loomml.rllib`, as +[RLlib Environment Definition](spec-ml-core-environment.md#rllib-environment-definition) +specifies; the `RLModule`, as +[Module Boundary](spec-ml-core-model.md#module-boundary) specifies, together +with the custom action distribution; callbacks that observe episodes and emit +the statistics below; and the custom evaluation function that runs the fixed +test set. + +A `Learner` subclass is permitted for instrumentation only. It may read +gradients, activations, and parameter norms and emit statistics; it may not +change a loss term, an optimizer step, a gradient, or an update order. An +instrumented run and an uninstrumented run must produce identical parameters +from identical inputs, which is the same invariance +[Forward Benchmarking Obligations](spec-ml-core-model.md#forward-benchmarking-obligations) +requires of the model's optimizations. + +Three obligations the environment and model documents impose land on the +trainer's configuration rather than on its code. Termination and truncation +stay distinct, and the value target must bootstrap past a truncation and must +not bootstrap past a termination; a connector or wrapper that collapses the two +flags is invalid. The mask travels in the batch unchanged from collection to +update, as [Masking](spec-ml-core-model.md#masking) requires, because a +pipeline that lets the learner recompute it produces an importance ratio that +is wrong in a way no diagnostic reveals. And the observation stays a ragged +graph batch, so a configuration that inserts a flattening connector ahead of +the module defeats the only reason the fork exists. + +### Stages And Handoff + +A run is an ordered sequence of stages, each binding one algorithm and one +environment view: + +```text +TrainingStage { + algorithm: AlgorithmBinding + environment_config_digest: ComponentViewDigest + advance: optional +} + +StageAdvance = + AtEnvSteps(uint64) + | AtPlateau { + metric: LearnerStatisticRef + window: positive uint32 + tolerance: ExactRatio + grace_env_steps: uint64 + } +``` + +A single-algorithm, single-environment run is the one-stage case and needs no +special form. The final stage carries no advance and every earlier stage +carries one; the run ends when the final stage does. There is no `AtRunEnd` +member, because being last is a fact the ordered sequence already fixes and a +member spelling it would be a second source that adoption then has to check +against the position. + +`AtEnvSteps` counts sampled steps from the run's start rather than from the +stage's, so the values across a run strictly increase and the reading is +unambiguous at any point. + +A stage binds a whole environment view rather than a delta against its +predecessor, so the configuration that produced any point of a run is +recoverable exactly, and a stage boundary is one atomic change rather than a +set of independently applied overrides. + +A stage boundary always transfers **model parameters**. It transfers +**algorithm state** — optimizer moments, the advantage normalizer, the +iteration counter — exactly when the two stages bind the same arm of +`AlgorithmBinding`, and never when the arm changes. Carrying moments +accumulated under one objective into gradients of another produces first +updates that are neither stage's; discarding them across a boundary that only +moved the environment would make every advance a cold restart of the +optimizer, which is the cost that would make a many-stage run unaffordable. + +Two failure modes are contract obligations rather than implementation notes, +because both present as a successful run. + +A parameter load that does not reach the module is a cold start wearing a warm +start's name. A checkpoint's component tree is addressed by name, and a restore +aimed at a component path that does not exist succeeds silently and loads +nothing. The boundary is therefore observable: the run reports the loaded +parameter count and a digest of the loaded tensors, and a stage that reports +neither has not demonstrated that it loaded anything. + +An evaluation at a stage boundary must observe the loaded parameters. Sampling +workers hold their own copies, and an evaluation that draws weights from a +worker rather than from the learner reports the parameters that worker last +held, which at a boundary is the previous stage's or the initial ones. The +boundary evaluation is the reading everything downstream compares against, so +reporting the wrong parameters there mislabels every later comparison. + +`AtPlateau` names a learner statistic and stops the stage when its moving +window varies by less than `tolerance`, after `grace_env_steps`. It exists +because a stage's useful length is not always known in advance: a stage that +ends when its objective stops improving is measuring a property of the run +rather than asserting a step count. The metric must be one the stage's own +algorithm emits. + +### Exact Rational Hyperparameters + +No binary floating-point number appears in a training view. Every real-valued +hyperparameter is an `ExactRatio`, and the `float64` a trainer requires is +produced once where the configuration crosses into Python, exactly as +[Reward Contract](spec-ml-core-environment.md#reward-contract) requires of the +reward scale. + +The reason is identity. A view is digest-covered, and a digest over a decimal +literal rounded to a mantissa is a digest over a rounding: two runs that +recorded the same configuration could differ in the last bit of a learning +rate, and no record would distinguish them. Keeping the exact ratio in the view +keeps the configuration a value and moves the conversion to the one layer that +can state the exactness condition its own float type imposes. + +The model's `value_bound` is not here. It sizes a clamp inside the model and +belongs to the model view, which is what a checkpoint pins. + +### Schedules + +A hyperparameter that varies over a run is a schedule: + +```text +Schedule = + Constant(ExactRatio) + | PiecewiseLinear(ordered nonempty sequence<{ + at_env_steps: uint64 + value: ExactRatio + }>) +``` + +The first breakpoint is at zero, breakpoints strictly increase, and the value +past the last breakpoint is that breakpoint's value. + +Breakpoints are keyed on environment steps sampled, never on training +iterations. An iteration count depends on batch size, on how many steps a +sampler happened to return, and on how a stage was resumed; a sampled step +count does not. A schedule is therefore a pure function of the step count, so a +resumed run continues the anneal from where it was rather than restarting it, +and two runs that differ only in topology still follow the same schedule +against the same abscissa. + +### The Reachable Return Range + +The model view's `value_bound` is checked rather than chosen, and what it is +checked against is an obligation on each training document rather than a +formula here. + +A training document declares the per-episode return bound of the environment +its run binds, in that environment's own reward vocabulary. It is the document +that can: the arithmetic reads the environment's codes and step bound on one +side and this document's `reward_adapter.scale` and stage sequence on the +other, so neither the environment nor this core holds both halves. The bound is +derived from two things the environment already owns: the quantization width of +its selected search energy, since each +dimension's `ExactAffineQuantization` fixes the interval and a single +transition's signed difference cannot exceed its width; and the codes it +charges for outcomes that are not transitions, together with the bound on how +many of each an episode may carry. Environments charge structurally different +sets of codes, so a formula fixed here would be one search's arithmetic imposed +on another's. + +Two rules bind every such declaration. The bound accounts for every step, not +one of each kind: an episode that spends its whole step budget on the most +expensive outcome is not a corner case but the first thing an untrained policy +produces. And where a term's magnitude depends on the instance rather than the +configuration, the declared bound is the maximum over the instance pool, taken +at adoption. + +A declared bound is in the environment's own integer reward units, and +`value_bound` is in the scaled units the value head emits, so the comparison +multiplies the declared bound by `reward_adapter.scale` before making it. A +scale above one would otherwise let through a bound that clips every return it +was checked against. + +A run's requirement is that scaled maximum across every stage it binds, because +a later stage may raise a bound, a code, or the scale. Adoption requires +`value_bound` to be at least that maximum. A bound below it clips a return the +environment can actually produce, which biases the advantage estimate exactly +where the search is doing best; a bound far above it is permitted and merely +weakens the stability mechanism. + +That the range is computable at all is a consequence of the environment +refusing to normalize. Reward is exact, integral, and drawn from a declared +bounded quantization, so nothing here needs an estimated scale or a running +normalizer. + +### Reward Transforms + +The adapter applies exactly one transform: the declared `ExactRatio` scale that +turns an integer reward into the float the trainer requires. + +Running normalization is rejected for three reasons. It makes the reward a +function of the batch rather than of the transition, so the same transition is +worth different amounts at different points of a run. It destroys the exactness +the environment paid for. And it hides exactly the failure it appears to fix: a +reward whose scale is wrong is a configuration error, and normalizing it away +means never seeing it. + +Clipping, discounting inside the adapter, potential-based shaping terms, and +per-dimension reweighting are likewise excluded. A run that wants a different +weighting binds a different objective closure, which is a semantic change with +a digest, rather than a transform with none. + +`advantage_standardization` is not a reward transform. It standardizes the +advantage estimate inside the loss, which is the algorithm owner's own +mechanism, and it is exposed here only because it is a configuration field a +run sets. + +## Rollout And Learner Topology + +`num_env_runners` and `num_envs_per_env_runner` are part of the reproduction +tuple, not free scaling knobs. Every environment derives each copy's PRNG +streams from its own coordinates, so changing either count changes which seeds +are drawn and which instances an episode sees. That is a property of the +sampling topology rather than a defect, and the run records both counts so a +run that does not reproduce can be told apart from one that was never +configured the same way. + +Sizing follows the measured split rather than a default. Each environment +document reports what dominates its step and each model document reports what +fraction of rollout time inference takes, so a topology is chosen by moving +runner count against that measured split rather than by matching learner count +to available accelerators. + +The environment's own interaction contract governs instance lifetime, and this +document adds no second lifecycle. + +## Curriculum + +A curriculum is not a separate record. It is what the stage sequence already +expresses: consecutive stages whose environment views differ. No nested second +sequence is needed, and admitting one would give a run two advance mechanisms +for the same thing. + +Consecutive stages may differ in their environment views only in fields that +fix neither the action-space shape, the observation shape, nor the reward +semantics. A policy cannot be trained across a change in any of the three: the +action space would change extent mid-run, the embedding tables would be sized +for a catalog the observation no longer produces, and the value head would be +regressing toward a different quantity than before. + +The rule is stated by what a field determines rather than by naming a record, +because environments partition their views differently and the same field name +sits inside an episode policy in one and beside it in another. Each environment +document names which of its own fields fall on each side, and adoption checks +the partition that document declares. A rule written as one record name would +forbid, in some other environment, exactly the curriculum that environment most +obviously wants. + +An environment's cached assets survive a stage advance whenever the advance +changes no field their key names. A key that names an individual instance +rather than the set the view declares therefore survives any advance that only +grows that set, which is what makes a many-stage run affordable rather than a +repeated cold start. Each environment document says which of its caches have +that property. + +## Episode Statistics + +### The Production Rule + +Every episode statistic is a function of values the environment already emits: +observation columns, objective codes, scalar features, the step result, the +action taken, and the retained trajectory. A callback never re-derives a fact +from an Artifact, never calls a semantic owner, and never computes a quantity +an owner would compute differently. A statistic that cannot be produced this +way is not a statistic this document defines; it is a request for the +environment to emit something it does not. + +Every statistic is keyed by a member of a closed catalog. No key is built by +formatting a file name, a path, or an identity digest into a metric name. +String-keyed series multiply with the instance pool and cannot be enumerated in +advance, so a consumer cannot tell a missing series from a series that was +never produced. + +### Objective Dimension Statistics + +One record per dimension of the environment's selected closure, per episode: + +```text +DimensionEpisodeStatistic { + entry_code: uint64 // at reset + final_code: uint64 // at the episode's last state + best_code: uint64 // minimum over the episode's states + step_of_best: uint64 +} +``` + +A consumer reads improvement as `entry_code - final_code` and best improvement +as `entry_code - best_code`; neither is a stored field, because both operands +are already here and a stored difference is a second source for one fact. + +Improvement is uniformly oriented without a per-dimension rule, because a +directed code is already direction-normalized by its `ExactAffineQuantization`: +a lower code is better for every dimension regardless of which direction the +dimension optimizes. `entry_code` is retained because a final value alone is +not a result — episodes start from different instances, and a final code says +nothing about whether the policy improved anything. + +`retain_step_series` extends these to a per-step series and is off by default, +because it is one array per dimension per episode and is the largest logging +term by a wide margin. + +### Outcome And Action Statistics + +Every episode reports the accounting the environment core's step identity +already guarantees: + +```text +episode_return scaled, matching the trainer's own return +episode_length steps, spanning sampler chunks +advanced_steps +non_advancing_steps +elective_stops zero or one +unaccounted_steps episode_length - the three above +terminal_reason one member of the environment's catalog +mean_live_decision_count +max_live_decision_count +mask_fallback_count +``` + +The names here are the neutral ones +[Step Accounting And Episode Endings](spec-ml-core-environment.md#step-accounting-and-episode-endings) +fixes. Each environment reports the same quantities under its own word for its +non-advancing class and its own word for one member of the live enumeration; +what is fixed is the accounting, not the vocabulary. + +There is no headroom statistic. The distance from `max_live_decision_count` to +the environment's `enumeration_bound` is a difference of two values already +recorded, and a stored difference is a second source for one fact on the same +terms the objective statistics above refuse one. + +`unaccounted_steps` exists to be zero. The environment guarantees the identity +by construction, so a nonzero mean is a defect in the callback rather than in +the environment, and the usual cause is an episode length read off a single +sampler chunk. + +`terminal_reason` is reported as a rate per member of the environment's +terminal catalog rather than as a single mode, because the members mean +different things: one is the ending the policy elected and the rest are limits +the harness imposed, and a policy that never elects its own ending is a +different failure from one that elects it immediately. + +A run reports the non-advancing rate per member of the environment's own +failure catalog, never as one aggregate. A single rate conflates causes that +call for opposite responses, and the environment already separated them. + +`max_live_decision_count` against the environment's `enumeration_bound` is what +predicts a capacity refusal before it happens. +`mask_fallback_count` counts the model's fully-masked fallback, reported here +because this is the layer that observes it rather than because this layer owns +the advisory set. + +Action frequencies are keyed by the environment's own action partition and each +is paired with that partition member's success rate — advances over attempts. +The frequency says what the policy tries and the pair says whether trying +works, and a policy collapsed onto one member of the partition is visible here +and nowhere else. + +### Cost Statistics + +```text +mean_step_duration +mean_reset_duration +episodes_per_hour_per_copy +model_inference_share_of_rollout +``` + +Wall time is nonsemantic in the sense +[Operational Observations](spec-dse-feedback.md#operational-observations) +defines, and these four are labelled as such wherever they are reported. None +is summed across concurrent copies. Deterministic work summaries remain the +cross-machine cost measure, and `model_inference_share_of_rollout` is the +figure a model optimization has to move. + +Per-stage step decomposition is not reported here. It belongs to the benchmark +harnesses, whose instrumentation is inactive elsewhere; a trainer carrying +stage boundaries per step would pay a measurable fraction to produce numbers +not comparable to its own uninstrumented throughput. + +### Breakdown Axes + +```text +LoggingPolicy { + enabled_axes: canonical set + retain_step_series: bool + per_episode_sample_rate: ExactRatio +} +``` + +`BreakdownAxis` is a closed catalog each training document owns, because the +axes worth splitting a statistic along are the ones its own search has. Two +members are mandatory and take ordinals zero and one in every such catalog: +`Stage`, over the run's stage sequence, and `TerminalReason`, over the +environment's terminal catalog. Fixing their ordinals is what lets a consumer +reading two searches' statistics key on the two axes both are guaranteed to +have; everything after ordinal one is per-document. + +Axes are opt-in because their cost is multiplicative: an axis over an instance +pool turns every statistic into one series per pool member, and two such axes +multiply. The default is no axis, and a diagnosis enables the one axis it +needs. + +An axis's members are exact references, keyed by identity rather than by a file +name, so a series survives a reorganization of where anything is stored. + +The sink is presentation. No sink is normative, and a run that writes to no +sink computes the same policy. + +### Chunk Invariance + +An episode statistic must not depend on where the sampler cuts an episode. A +sampler returns fragments, and an episode may span several; a statistic +computed per fragment and averaged is a different quantity from the same +statistic computed per episode. + +Two forms are permitted: a statistic accumulated across fragments until the +episode ends and emitted once at its end, and a statistic that is a pure +function of the episode's final state. Two are forbidden: a statistic emitted +per fragment as though the fragment were an episode, and a statistic whose +value depends on the fragment length the topology happened to produce. + +## Learner Statistics + +Loss terms, entropy, KL, explained variance, gradient norms, and timing are the +algorithm owner's statistics, reported unchanged. This document neither +redefines nor renames them, and three of them are worth reading against +contracts stated elsewhere. + +Entropy is over admissible outcomes only, per +[Action Distribution](spec-ml-core-model.md#action-distribution), so it is +comparable across states with different live counts. An entropy that tracks +`enumeration_bound` rather than the live count indicates entropy computed over +masked slots. + +Explained variance is against a return whose range is declared, so a +persistently negative value is a value-head failure rather than a scaling +problem. + +A KL spike at a curriculum boundary is the expected consequence of the +environment view changing under a fixed action space, and a KL spike at a stage +boundary is the expected consequence of the objective changing under fixed +parameters. Neither is a defect; both are worth distinguishing from a spike +with no boundary near it. + +## Test Protocol + +### The Test Set + +```text +ResolvedTestSetConfigView { + environment_config_digest: ComponentViewDigest + cases: ordered nonempty sequence + case_timeout: optional +} + +TestCase { + instance: + case_seed: u64 +} +``` + +A case names its instance exactly. Nothing is drawn. A case runs by passing its +instance as the episode start override its environment defines, so no selection +stream is consulted and the copy coordinates cannot reach the episode. That is +what makes a case's result independent of which evaluation runner executed it, +and why the protocol needs no coordination between runners. + +A test set's inventory is disjoint from that of every stage the run binds, +compared by exact identity. A case the policy trained on measures how well the +run memorized it, which is the one thing a test score is not for. Each training +document names only the granularity at which identity is compared, since what +counts as one instance differs by search. + +A test set binds its own environment view, which is what makes disjointness +expressible at all: an override may only name inventory the view it runs +against declares, so a test set sharing a training view would have to name +inventory that view declares and could never be adopted. Adoption checks that +the case inventory is exactly what the test set's own view declares. + +The two view kinds must agree on everything a checkpoint depends on. A test-set +environment view differs from the training views only in fields that fix +neither the action-space shape, the observation shape, nor the reward semantics +— the same partition consecutive stages obey — because a score produced +against a different action space is not a score of the same policy. + +The set is ordered and complete. Every case runs on every evaluation, in case +order, exactly once. A sampled test set is rejected: a score that moves because +a different subset was drawn is indistinguishable from a score that moves +because the policy changed, which defeats the only purpose the set has. + +### Determinism + +A test episode is deterministic in both halves. The policy acts greedily — +exploration off, the distribution's mode rather than a sample — so the model +contributes no randomness, and `case_seed` seeds whatever residual randomness +an implementation retains. The environment is reproducible given its +configuration, the copy coordinates, the seed, and the action sequence, and a +case consults no selection stream at all. + +Both halves being deterministic is what discharges the exact-reproduction claim +[Reproduction](#reproduction) makes for a test run, and that claim in turn is +the only reason two checkpoints are comparable at all. + +### Execution And Aggregation + +Cases are distributed across evaluation runners by case ordinal. Distribution +is a throughput decision with no semantic content. + +```text +TestCaseOutcome = + Completed { the episode statistics catalog above } + | StartFailed { the environment's episode start outcome } + | TimedOut { steps completed } + +EvaluationSchedule { + every_env_steps: uint64 + at_run_end: bool +} +``` + +A failed or timed-out case is reported, counted, and excluded from the +completed aggregate — never dropped. Dropping biases toward the cases that ran, +so a degrading policy would show an improving mean as its hardest cases quietly +left the set. A result therefore carries the completed count alongside every +aggregate, and an aggregate over a changed number of cases is not comparable to +its predecessor. + +`case_timeout` is nonsemantic wall time. A timed-out case retains its partial +statistics, tagged as partial, and they are never averaged into a completed +aggregate. + +Evaluation runs on a schedule keyed on sampled environment steps, for the same +reason schedules are. + +### What A Test Score Is Not + +A test episode falls on the search-heuristic side of +[Nonsemantic Boundary](#nonsemantic-boundary), on the same terms as a training +episode. The boundary is worth restating here because a test episode is where +it is easiest to lose: these episodes run the same machinery a real search +would and read the same objective dimensions, so the numbers look like results, +and treating them as results would put an unvalidated search heuristic where a +verified one belongs. + +## Checkpoints And Run Identity + +```text +CheckpointPolicy { + every_env_steps: uint64 + retain_last: positive uint32 + retain_best_by_test_score: bool +} +``` + +A run is identified by its run view's digest and by nothing beside it. There is +no run-key record, because a record with one field is a name for that field and +a second place to change it. Everything a key would want is already inside the +digested view — the topology counts, the stage sequence, every participating +view — so a run at a different runner count already has a different identity, +and carrying any of it again would be two sources for one fact. + +A checkpoint records that digest, the sampled step count, the stage ordinal and +stage it was taken in, and the parameters. Loading requires an exact +model-view match per +[Parameters And Checkpoints](spec-ml-core-model.md#parameters-and-checkpoints), +and a load whose catalog cardinality no longer matches an embedding table fails +rather than truncating it. + +A checkpoint selected by test score records which test-set digest selected it, +because a score is only meaningful against the set that produced it. + +## Reproduction + +This document makes exactly two reproducibility claims and declines a third. + +A test run reproduces exactly: one checkpoint and one test-set digest give +identical statistics on any machine at any runner count. + +A trajectory reproduces exactly: the recorded configuration, coordinates, seed, +and action sequence reconstruct the episode its environment's replay path +defines. + +A learning curve does not reproduce bit-for-bit, and claiming otherwise would +be false. Concurrent learner reductions, accelerator kernel nondeterminism, and +asynchronous sample collection whose completion order depends on wall time all +perturb the update sequence. What a run records is enough to reconstruct its +data, its configuration, and every episode it ran — which is what reproduction +has to mean here. + +## Harnesses + +Every training document names its own harnesses, qualified by the search they +fit, on the removable-projection terms +[Nonsemantic Boundary](spec-ml-core-environment.md#nonsemantic-boundary) sets +for every harness in this stack. + +Two obligations bind them wherever they appear. A harness that runs a test set +against a checkpoint runs the same protocol the training-time evaluation runs, +so a score produced during a run and a score produced afterward are the same +quantity. And a harness that prepares training data verifies an existing +preparation without regenerating it, so a corpus can be checked without being +rebuilt. + +## Conformance Anchors + +Stable tests cover a hyperparameter edit changing the training digest and +leaving the environment digest and every cached asset valid; a checkpoint +loading under a changed training view and failing under a changed model view; a +schedule evaluated at a given sampled step count producing the same value on a +resumed run as on an uninterrupted one; a stage boundary transferring +parameters always, and transferring optimizer moments, the advantage +normalizer, and the iteration counter across a same-arm boundary while +transferring none of them across an arm change; a stage boundary reporting the +loaded parameter count and +tensor digest, and a restore aimed at a path that does not exist failing rather +than loading nothing silently; a boundary evaluation observing the loaded +parameters rather than a sampler's stale copy; an `AtPlateau` advance naming a +statistic its own stage's algorithm emits; a single-stage run behaving +identically to the same configuration expressed without stages; an instrumented +learner producing parameters identical to an uninstrumented one from identical +inputs; a truncation bootstrapping and a termination not; a batch whose mask +was recomputed rather than carried being rejected; every episode statistic +being a function of environment-emitted values alone; `unaccounted_steps` being +zero over a complete episode; an episode statistic computed across a sampler +chunk boundary equalling the same statistic computed from an unchunked episode; +consecutive stages differing only in fields the environment declares neutral +being accepted and ones differing in a shape-fixing field being rejected; +algorithm state carrying across a same-arm stage boundary and resetting across +an arm change; a +`value_bound` below the environment's declared return range being rejected, +including where a term's magnitude is instance-dependent and the maximum over +the pool is what binds, and the comparison being made against the declared +bound scaled by `reward_adapter.scale`; a test set running every case exactly +once in case order; a test case consulting no environment selection stream and +producing an identical trajectory at two runner counts and two copy +coordinates; one checkpoint and one test-set digest reproducing identical +statistics across machines; a failed or timed-out case being counted and +excluded from the completed aggregate rather than dropped; and a run's identity +being the run view digest alone. + +Tests do not pin hyperparameter values, schedule breakpoints, stage counts, +plateau tolerances, topology counts, the contents of any particular test set, +wall-time numbers, throughput, learning curves, achieved test scores, sink +formats, or diagnostic text. diff --git a/docs/spec-ml-dse-environment.md b/docs/spec-ml-dse-environment.md new file mode 100644 index 000000000..66bae91cc --- /dev/null +++ b/docs/spec-ml-dse-environment.md @@ -0,0 +1,1325 @@ +# ML DSE Environment + +This document defines the interaction boundary through which a learned search +policy explores Loom's design space. The environment presents one episode as a +sequence of typed design decisions over one subject, either a single +`fabric.module` or a complete multi-core `fabric.system`, optionally together +with the software expression of the workloads that subject must run. It scores +each transition against a resolved objective and retains the decision sequence +for later replay. + +Which decisions an episode may take is configuration rather than schema. The +environment binds a set of exploration domains, each of which is one existing +candidate generator, and a domain the configuration omits contributes nothing. +A parameter family over an existing parent kind, existing reference kinds, and +an existing completion form is one catalog member and one obligation-table row. +A family that needs a new reference kind, a new value form, or a new completion +form extends the observation projection and the completion algebra as well; +what appending never requires is widening a decision union or adding a +permission flag. + +## Ownership + +Every fact the environment exposes resolves to one exact owner: + +- [Evaluation and DSE](spec-dse-feedback.md#candidate-generators) owns + candidate-generator kinds, their closed decision unions, decision domains, + canonical decision order, and lineage contributions; +- [Evaluation and DSE](spec-dse-feedback.md#objectives-and-quality-gates) owns + `ObjectiveDimension`, `ExactAffineQuantization`, `ObjectiveVector`, + `WeightedLevel`, `TotalOrdering`, `SearchEnergyRef`, and quality gates; +- [ADG Builder](spec-adg-builder.md#failure-atomic-finalization) owns draft + construction and authoring-boundary checks; +- [Fabric Artifact](spec-fabric-artifact.md#finalization-and-publication) owns + the finalization pipeline, canonical bytes, root identity, the verified + pre-publication closure, and publication; +- [Fabric Identity](spec-fabric-identity.md#owner-local-reference-kind-catalog) + owns the local-reference kind catalog and, at + [Mapping-Visible Entity Catalog](spec-fabric-identity.md#mapping-visible-entity-catalog), + `EntityId`; +- the `fabric.*` specifications own every typed hardware parameter domain; +- [SCF To DFG](spec-compiler-part-3-dfg.md#canonical-dataflow-rewrite-catalog) + owns the closed Canonical Dataflow rewrite catalog, its decision wire, its + canonical decision order, and the equivalence obligations every rewrite + preserves; +- [Place And Route](spec-pnr.md#objective-projection) owns the Mapping + violation catalog `V` and measure catalog `G`; + [Invocation Contracts](spec-pnr.md#invocation-contracts) owns invocation + outcomes; + [Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism) + owns search policy and the seeded PRNG protocol; and + [Evaluation Transaction](spec-pnr.md#evaluation-transaction) owns the + ephemeral probe adapter and its `rebuild`, `probe`, `commit`, and `discard` + operations; +- [TechMapping Generation](spec-tech-mapping.md) owns TechMapping construction + and its exact `D` and `F` binding; +- [Evaluation Metrics](spec-evaluation-metrics.md) owns metric kinds, units, + observation forms, and `ExactRatio`; +- [Resolved Configuration](spec-config-ssot.md#component-views) owns + component-view framing, canonical view bytes, and `component_view_digest`, + and at [Cache Dependencies](spec-config-ssot.md#cache-dependencies) owns the + cache-family contract; +- [ML Environment Core](spec-ml-core-environment.md) owns the observation + container and its combined node-and-link space, the action surface and + masking algebra, the step accounting identity, the + termination-versus-truncation rule, the reward boundary, the PRNG preimage + shape, the `loomml` package layering and its interaction contract, and the + benchmarking obligations every harness satisfies; and +- [Full-Stack Architecture](spec-loom-stack.md) owns external dependency + pinning and the fast-evaluation cascade. + +The environment owns only the episode protocol, the exploration-domain catalog, +the enumeration rule, its action index contract, and the schedule-derived +reference rule, the DSE observation +catalogs and their curriculum-neutral partition, the workload-feasibility +invariant, the step and rejection outcome algebra, the visited set and the +per-state rejection mask, the trajectory retention record, and its own +benchmarking stages. + +## Nonsemantic Boundary + +The environment is a nonsemantic search harness, on the terms +[Nonsemantic Boundary](spec-ml-core-environment.md#nonsemantic-boundary) states +for every ML environment. It defines no hardware action language, no mutable +candidate intermediate representation, and no second design space. Every +decision it applies is an existing owner-typed candidate-generator decision, +every legality answer comes from the ordinary Builder, Fabric finalizer, and +Dataflow rewrite owners, and every mappability answer comes from the ordinary +Place and Route owner. + +No episode subject is published while the episode holds it. The single +exception is the component a `Reattach` completion publishes so the subject can +depend on it, defined under Step. That component is published by its own +generator through the ordinary path, with the `CandidateDecision` lineage +contribution that generator's descriptor owns, so it is an ordinary +intermediate candidate rather than a publication mode this document invents; +what the environment withholds is selection, not lineage. + +The environment adds nothing to the subjects deliberately absent from the first +version by +[Full-Stack Architecture](spec-loom-stack.md#deliberate-first-version-boundaries). + +An environment run is therefore never an authority for candidate selection. A +design the environment discovers becomes a real candidate only by replaying its +trajectory through ordinary `Generate` and `Promote` plan nodes, as defined by +Trajectory Retention And Replay below. + +## Exploration Domains + +An exploration domain is one existing candidate generator whose decisions the +environment may enumerate. Domains are the unit of permission and the unit of +extension: a domain is permitted exactly when the resolved configuration binds +it, and a new parameter family is added by appending one member to the closed +catalog below and one row to the obligation table, to the extent the intro +states, and never by widening a decision union or adding a parallel switch. + +```text +ExplorationDomainKind = + SpatialTopology // 0 + | SpatialMicroarchitecture // 1 + | SystemComposition // 2 + | DataflowRewrite // 3 +``` + +Ordinals are stable. A new kind appends; reordering, deleting, or repurposing +one is an incompatible change to this document's schema version. Each kind +binds its generator's own resolved configuration and defines no decision, match +rule, or value domain of its own. + +One table is the complete per-domain contract. Every rule elsewhere in this +document quantifies over its rows rather than naming a domain, so appending a +domain is appending a row: + +| Domain | Generator | Operates on | Required subject | Completion | +| --- | --- | --- | --- | --- | +| `SpatialTopology` | kind 13 `spatial_topology_rewrite` | one `fabric.module` | either | `Reattach` via `SystemComposition` under `System` | +| `SpatialMicroarchitecture` | kind 14 `spatial_microarchitecture_rewrite` | one `fabric.module` | either | `Reattach` via `SystemComposition` under `System` | +| `SystemComposition` | kind 15 `system_composition_rewrite` | one `fabric.system` | `System` | none | +| `DataflowRewrite` | the [Dataflow rewrite generator](spec-dse-feedback.md#deterministic-work-candidate-sets-and-cache) | one Canonical Dataflow Program | either | `ReplaceParent` | + +"Operates on" is both the exact parent a decision derives from and the kind of +child it produces; every registered generator preserves its parent's kind. +"Completion" is the mechanical follow-up defined under Step, and a `Reattach` +names the domain that performs it. + +Every episode's probe requires the Spatial PnR and TechMapping views +unconditionally, and a `System` episode's probe requires the System PnR view +its subject arm carries. None of those is per-domain, so no row names a view +and no rule derives one from the table. + +A domain is admissible only when its row states every column and its decisions +are representable, meaning every owner-local reference they can carry resolves +to exactly one node under the target-closure rule of Combined Node And Link +Space. Representability is a property of the domain and the projection +together, so a future domain is admitted by extending both. + +Adoption is then one rule quantified over the table rather than a list of +per-domain checks: for every bound row, the subject it requires matches +`episode_subject`, and the domain any `Reattach` completion names is itself +bound; and no two bindings share a domain kind. Binding a Spatial row under a +`System` subject without also binding `SystemComposition` is therefore invalid +at adoption, because its completion could not run. + +Each row's decision union, its typed value domains, its canonical decision +order, and the rules that make one of its domain members well formed belong +entirely to that generator. The hardware unions of kinds 13, 14, and 15 are +defined by [Candidate Generators](spec-dse-feedback.md#candidate-generators); +the closed Canonical Dataflow rewrite catalog and its member ordinals are +defined by +[Canonical Dataflow Rewrite Catalog](spec-compiler-part-3-dfg.md#canonical-dataflow-rewrite-catalog), +which states that no other document may add, remove, rename, or reorder a +rewrite kind. This document reproduces none of them, so a catalog revision does +not strand a second copy here. + +What the environment adds is only the enumeration rule: for a bound row, its +generator's decisions in that generator's own canonical order, over the exact +parent the row's "Operates on" column names. Every owner-local enumeration rule +survives unchanged, including the ones that make a decision's availability +depend on the current Fabric, and every rewrite's equivalence obligation is its +catalog owner's. A software decision therefore changes how a workload is +expressed and never what it computes. + +`DataflowRewrite` explores the complete catalog over the workloads the episode +already holds; there is no environment-local catalog-kind selector, because a +subset selector would make the enumeration a filtered view of the generator's +canonical order rather than that order itself. Its binding carries the +generator's own immutable component view, +`loom.dataflow_rewrite_generator.config.1.1`, whose `scope_expansion_limit` and +semantic value domain are owned by +[Resolved Configuration](spec-config-ssot.md#schema-ownership). The other three +rows bind `SpatialTopologyRewriteConfig`, +`SpatialMicroarchitectureRewriteConfig`, and `SystemCompositionRewriteConfig` +respectively, the exact descriptor-owned records defined by +[Candidate Generators](spec-dse-feedback.md#candidate-generators). No row +introduces an environment-authored configuration record. + +Two families remain outside this contract rather than represented by inert +domains. `FabricTemplateConfig` of kind 12 is absent because a template +expansion selects an episode seed rather than advancing an episode. +`ImplementationFlowConfig` of kind 16 is absent because it produces a +`HardwareImplementation` rather than a design the feasibility gate can close +against a workload. Either is reopened by appending a domain kind that can +state all five obligations above. + +## Resolved Environment Configuration + +The environment consumes one immutable component view with schema descriptor +bytes `loom.ml_dse_environment.config.1.0`, following the framing, canonical +byte representation, and digest contract owned by +[Resolved Configuration](spec-config-ssot.md#component-views): + +```text +ResolvedMlDseEnvironmentConfigView { + episode_subject: EpisodeSubject + enabled_domains: canonical nonempty set + episode_policy: EpisodePolicy + probe_policy: ProbePolicy + selected_objective_closure: SelectedObjectiveClosure + observation_policy: ObservationPolicy + determinism_policy: EnvironmentDeterminismPolicy +} + +EpisodeSubject = + SpatialModule + | System { + admissible_modules: canonical nonempty set + system_pnr_config_view: ResolvedPnrConfigView + } + +EpisodePolicy { + seed_roots: canonical nonempty set + workload_pool: canonical nonempty set + workloads_per_episode: positive uint64 + start_retry_bound: positive uint64 + step_bound: positive uint64 + consecutive_rejection_bound: positive uint64 + visited_state_retention: uint64 + rejection_reward_code: uint64 + stop_reward_code: uint64 +} + +WorkloadBinding { + dataflow: ArtifactRootReference + tech_mapping: ArtifactRootReference +} + +ProbePolicy { + spatial_pnr_config_view: ResolvedPnrConfigView + tech_mapping_config_view: ResolvedTechMappingConfigView + warm_start_from_parent: bool +} + +ObservationPolicy { + enumeration_bound: positive uint32 + include_placement_edges: bool + include_resource_states: bool +} +``` + +`EpisodeSubject` carries what its arm needs rather than leaving subject-keyed +fields elsewhere to be required-or-forbidden by prose. The `System` arm holds +the finalized Module candidates kind 15 selects from and the System PnR view +that arm's probe requires, so neither can be present without the subject that +uses them or absent under the subject that needs them. + +`EpisodePolicy` owns every remaining input `reset` consumes, so an episode +start is a function of this view and the copy coordinates alone. Every +`seed_roots` member's root kind must match the subject arm, and +`workloads_per_episode` must not exceed the pool size. + +`tech_mapping_config_view` is unconditional: a TechMapping binds exact `D` and +`F`, so every advancing step regenerates at least one, whatever the bound +domains are. + +Membership in `enabled_domains` is the permission. A domain the set binds is +explored; a domain the set omits contributes nothing to the enumeration, no +masked entries, and no observation blocks. There is no separate permission flag +beside the binding, because a permitted domain with no configuration and a +configured domain that is forbidden are both states this contract has no use +for. + +`SelectedObjectiveClosure` and `ResolvedPnrConfigView` are the exact records +owned by +[Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism), +materialized as the selected transitive closure with view-local ordinals under +the projection rule owned by +[Component Views](spec-config-ssot.md#component-views). + +This view therefore contains two objective closures, and their roles are +disjoint. The closure inside `probe_policy.spatial_pnr_config_view` governs +PnR's internal search only: its total ordering, search energy, and focused +closure steer the probe toward a closed Mapping and never leave it. The view's +own `selected_objective_closure` is the sole reward authority; its search +energy is the value the reward differences, and it may carry dimensions the +probe has no notion of, such as a parameter-backed predicted metric. Neither +closure derives, overrides, or defaults the other. + +Adoption requires them to be consistent where they overlap. Every +`MappingViolationSource` and `MappingMeasureSource` dimension in the reward +closure must reference a descriptor that the probe view's temporary-violation +policy and measure catalog admit, so a reward dimension cannot read a fact the +probe never produces. A dimension referencing an obligation tagged `HeldOut` is +invalid here for the reason it is invalid everywhere. A reward closure whose +Evaluation-sourced dimensions no registered parameter contract satisfies is +invalid at adoption. A closure whose dimensions are unsatisfied by any selected +closure is invalid at adoption rather than at the first step that needs the +missing value. The set of parameter contracts an episode invokes needs no field +of its own: it is exactly the set the closure's Evaluation-sourced dimensions +reference, and a field beside it could only disagree. + +The canonical view encoder writes fields in the schema order above, under the +encoding and adoption rules +[Resolved Configuration](spec-config-ssot.md#component-views) owns. This view +adds only that its optionals use a `u32be` absent or present discriminant +followed by the payload when present. + +The PRNG preimage shape, the meaning of `effective_seed` and the copy +coordinates, the `reset` seed override, and the exclusion of policy sampling +from every environment stream are owned by +[Determinism And Copy Coordinates](spec-ml-core-environment.md#determinism-and-copy-coordinates). +This document adds only its own domain separator, which is + +```text +ASCII("loom.ml_dse_environment.prng.sha256_seeded_xoshiro256starstar.1.0") +``` + +and its stream purpose ordinals, which are `SeedSelection = 0` and +`WorkloadSelection = 1`. + +### Curriculum-Neutral Fields + +[Curriculum](spec-ml-core-training.md#curriculum) requires each environment +document to partition its own view by what a field determines. Here the +shape-fixing side is `episode_subject`, `enabled_domains`, +`selected_objective_closure`, `probe_policy`, `observation_policy`, and +`determinism_policy`: the first two fix the action space's extent, the closure +and the reward codes fix what a return means, the observation policy fixes the +column catalog the embedding tables are sized for, and the determinism policy +fixes the stream layout. + +Everything inside `EpisodePolicy` except the two reward codes is neutral, which +is what a curriculum over this environment actually wants to move: the seed +roots, the workload pool, the workloads per episode, the step and rejection +bounds, and the visited-set retention. Moving from small architectures to large +ones is a change of `seed_roots`, and lengthening episodes is a change of +`step_bound`. The Mapping cache survives such an advance, because its key names +one seed root and one workload rather than the sets this view declares: +extending `seed_roots` or `workload_pool` leaves every entry the earlier stages +warmed valid, which is what makes growing a pool the affordable curriculum it +is meant to be. + +## Workload Feasibility Invariant + +Every state the environment occupies is workload-feasible. For each Canonical +Dataflow Program in the state's current workload set, the probe must produce a +Mapping that could be finalized: full owner recomputation succeeds and all five +Mapping violation magnitudes are zero. This is the same closure condition +[Place And Route](spec-pnr.md#final-closure-and-verification) requires of a +selected candidate, evaluated over the ephemeral candidate rather than a +published Artifact. + +The closure level follows the episode subject. A `SpatialModule` episode +requires one closed Spatial Mapping per workload against the current Module. A +`System` episode requires one closed System Mapping per workload against the +current System, together with the Spatial Mappings that System Mapping imports; +the environment does not accept a System whose constituent Modules close +individually while the System does not, because service continuity, transport +capacity, and progress closure are System-level facts. + +The invariant is a gate on state occupancy, not a term in the energy. Removing +the last function unit that supplies an operation the workload needs, deleting +the last route between two required endpoints, shrinking a memory or FIFO below +what the workload's claims demand, narrowing a boundary inventory the workload +binds, removing the AccCore a workload's launch requires, or cutting the last +transport connection carrying a service leg all end the step. The agent never +occupies such a state and never receives a graded score for approaching one. + +When `DataflowRewrite` is bound the workload set is itself part of the state. A +software decision replaces one workload with its rewritten child, and the +invariant is evaluated over the resulting set: a rewrite that produces a +program the current hardware cannot map is rejected exactly as a hardware +decision that removes a resource the workload needs. The two directions are +symmetric, which is the point of exploring them jointly. Rewrite legality +itself is never at issue here, because every catalog rewrite is externally +equivalence-preserving by its owner's contract; what a rejection reports is +that this hardware cannot map that expression of the workload. + +Three consequences follow. + +The enumeration mask is advisory and feasibility is authoritative. Decision +domains constrain which decisions exist, and the Builder rejects a draft that +violates an authoring-boundary rule, but neither knows in advance which legal +decision removes the resource some workload happens to require. An unmasked +action is therefore not a guarantee that the step advances. The environment +treats a legal-but-infeasible decision as an ordinary first-class outcome; it +is not an assertion failure, not an exception, and not a silent no-op that +leaves the agent unable to distinguish action from inaction. + +Proof and budget are different answers and are never merged. A PnR +`ProvenInfeasible` result is a sound proof that no Mapping exists for that +workload on that candidate. Exhausting the probe's configured search work with +residual violations is inconclusive: it establishes nothing about the design. +Only exact admission or a sound bound may prove infeasibility, so an exhausted +budget is never reported as a proof. Both reject the step, and both are +recorded under distinct reasons so that a rising inconclusive rate is legible +as a probe-budget problem rather than as a shrinking design space. + +Failure is attributed per workload. The step record names the exact Canonical +Dataflow that failed, its reason, and its residual violation magnitudes by +descriptor ordinal. A step that fails several workloads records the first +failure in canonical probe order; the environment does not continue probing +after the invariant is already broken. + +## Action Space + +The action space is the concatenation of each enabled domain's canonical +decision order. For the current state the environment enumerates every bound +domain in ascending `ExplorationDomainKind` ordinal, and within a domain +enumerates that generator's decisions in the generator's own exact canonical +order, truncated by its `max_children_per_parent` or `scope_expansion_limit`. +Those are the generators' own semantic work policies and are the only +truncations applied. `enumeration_bound` is an observation capacity, not a +further truncation: a state whose enumeration exceeds it is refused as +described under Observation, never silently shortened. Ascending domain ordinal +is the only cross-domain rule, so enabling or disabling a domain shifts a +suffix of the enumeration but never permutes another domain's internal order. +An action is one ordinal in the concatenation: + +```text +EnumeratedDecision { + domain: ExplorationDomainKind + decision_kind: u32be ordinal in that domain's closed decision union + parent: ParentSelector + targets: ordered nonempty sequence + value: ReplacementPrototype(prototype ordinal) + | BoundedIntegerDelta(delta) + | NoValue +} + +DecisionTarget { + reference: owner-local reference + role: Primary | GroupMember +} + +ParentSelector = + EpisodeSubject + | SubjectComponent(FabricLocalReference) + | CollectionOrdinal(uint32) + +ActionIndex = uint32 +``` + +`parent` qualifies the whole decision because every reference in one decision's +normalized payload is local to the same root; a decision does not straddle two +parents. `targets` carries every reference that payload names, in its canonical +order, with at most one `Primary`. A decision naming one entity has a single +target, and one naming an entity plus a reference set has a `Primary` followed +by its `GroupMember` members, which is the shape the observation's target arcs +and the policy's anchor summary both consume. + +An owner-local reference is the reference its owner already defines: a +`FabricLocalReference` from +[Fabric Identity](spec-fabric-identity.md#owner-local-reference-kind-catalog), +or the `ActorRef`, `GraphRef`, or `StaticGraphLaunchRef` the rewrite catalog's +normalized decision carries. The environment transports them unchanged and +assigns no parallel identifier. + +Every such reference is local to one root, so a target is meaningless without +naming which root, and `ParentSelector` is that qualifier in one form for every +domain. `EpisodeSubject` selects the subject itself. `SubjectComponent` selects +a component of the subject by an ordinary Fabric reference, which is how a +decision whose row operates on a `fabric.module` addresses one Module of a +`fabric.system`. `CollectionOrdinal` selects a member of an ordinal-indexed +collection the state holds, which is how a decision addresses one workload. The +legal selector for a decision is derived from its row's "Operates on" column +against the episode subject rather than written out per domain: a row that +operates on the subject's own kind uses `EpisodeSubject`, and any other +selector for it is invalid. + +`NoValue` covers a decision fully determined by its target. + +Collection ordinals are assigned at `reset` in the canonical order of the +selected collection and are stable for the whole episode. A `ReplaceParent` +completion substitutes the child at exactly its parent's ordinal and leaves +every other ordinal untouched, so a retained target, a retained mask bit, and a +node block never silently repoint at a different member because an identity +sorted differently after a decision. + +The interaction surface, the `enumeration_bound` capacity, the masking algebra, +the elective stop, and the refusal of an over-capacity enumeration are owned by +[Action Surface And Masking](spec-ml-core-environment.md#action-surface-and-masking). +This document adds only which mask bits an environment clears: an index whose +decision was already rejected at this state, under the per-state rejection mask +Backtracking defines. + +Every enumerated decision's references resolve into the combined node-and-link +observation space, so a decision that names an occurrence, a connection, a +transport link, an actor, or a graph is scored the same way: by attending over +the graph nodes it targets. Per-node and per-link actions are therefore the +observation's shape rather than a second indexing scheme, and the action index +stays a single ordinal even for a decision that carries several references. + +Indexing the canonical decision order is what removes the per-type stride a +flat product space over node slots and decision types would need, and it +inherits determinism directly from the generator descriptor. + +### Schedule-Preserving Transformations + +Some decisions are worth taking only in a form the current schedule determines; +why that is worth the coupling is +[Why Some Design-Space Actions Read The Schedule](rationales/ml.md#why-some-design-space-actions-read-the-schedule). +A schedule-preserving transformation is one whose `GroupMember` references are +chosen from the state's own Mapping rather than from the Fabric's structure +alone: the decision kind, its `Primary` target, and the legality of the child +are exactly what they would be otherwise, and what the Mapping supplies is +which further references the decision names. + +`RemoveOccurrence` is the case that motivates the class. Deleting an occurrence +the Mapping currently routes through breaks every net whose route traverses it, +so a useful removal carries reconnection: links joining the upstream and +downstream endpoints of the traversals that occurrence was carrying. Which +links those are is not a question the topology can answer. The topology admits +an enormous number of reconnections, nearly all of them useless, and the +complete set would replace one occurrence with a connectivity explosion. The +Mapping names the few that the workload actually uses, because they are the +ones its routes already run through. + +That is what makes the class schedule-*preserving*. A child reconnected from +the parent's routes leaves the parent's placement carryable, so warm-starting +that child's probe begins from a mapping that still closes rather than one the +edit invalidated. A structurally reconnected child usually has to be re-mapped +from nothing, which costs the probe the option exists to save and discards the +evidence that the parent's placement was good. + +The reference set is Mapping-derived; the enumeration's extent is not. A state +offers one entry per removable occurrence whether or not a Mapping exists, so +the enumeration's length remains a function of the state and the resolved +configuration alone. This is load-bearing rather than incidental: it is what +keeps step 7 ahead of step 8, keeps the reset capacity test cheap enough to +precede workload feasibility, and keeps that test's retained per-seed verdict +valid. Only the references inside an entry wait for the state's probe, and they +are filled when the observation that reports them is built. + +The environment authors no reconnection semantics. Every reference it names is +an ordinary owner-local reference, every decision it emits is an existing +generator decision, and stage 3 runs the owner's ordinary acceptance path over +the result — so a reconnection the Mapping suggested and the owner refuses is +an ordinary rejection rather than a special case. The Mapping is being read as +evidence about which references are worth naming, never as authority for +whether the child is legal. + +Path dependence follows the probe. With `warm_start_from_parent` clear, a +state's Mapping is a pure function of that state, so the reference set is too +and the core's enumeration-purity rule holds unchanged. With it set, the +Mapping is path-dependent by construction, so a schedule-preserving decision's +references are path-dependent with it. That is the same cost the option already +carries for the reward, extended to the content of an action, and it is +recorded here rather than discovered when two runs over one state disagree +about what they were offered. + +Replay needs no Mapping. A `Trajectory` records each step's full +`EnumeratedDecision`, references included, and replay runs the recorded +decision rather than re-deriving it, so a replayed step reproduces the +reconnection the episode actually took without reconstructing the probe that +suggested it. + +## Observation + +### Combined Node And Link Space + +The combined node-and-link space, its `GraphNodeRole` catalog, the +connection-as-node rule, the two enumeration encodings, and the target-closure +rule are owned by +[Combined Node And Link Space](spec-ml-core-environment.md#combined-node-and-link-space) +and [Enumeration Encoding](spec-ml-core-environment.md#enumeration-encoding). +This document states only what that space contains for a DSE episode. + +This environment uses the `DecisionNodes` encoding, because its decisions have +variable target arity: one names a single entity, and another names a +distinguished subject plus a reference set whose size the generator decides. +Only out-degree expresses that, so the `Decision` block is present and +`decisions` is empty. + +Every decision kind this environment enumerates depends on the closure. +Removing an occurrence names an occurrence, replacing a point connection names +a connection, changing a transport connection names a System link, and +refactoring a graph definition names graphs and launches, so an exploration +domain whose decisions can carry a reference the space cannot represent is not +admissible. That admissibility test is the reason the Exploration Domains table +requires representability of every row. + +`DataflowOperation` and `DataflowValue` blocks are present exactly when a bound +domain operates on a Canonical Dataflow Program, which needs no configuration +flag because the binding already decides it. `AdjustParallelConnectionCount` +has exactly one target because a parallel connection is one node carrying its +count as a feature. + +When `include_placement_edges` is set, the probe's current Mapping contributes +one `Placement` arc from each Dataflow node to the Fabric node that realizes it +and one `Route` arc from each Dataflow value node to each Fabric node its route +traverses, which is how the policy sees the present mapping. The two roles are +distinct because the two relations are: realization is one-to-one and is what a +capacity decision moves, while traversal is one-to-many and is what a +connectivity decision moves, and an encoder given one role for both would have +to separate them from context it does not have. Those arcs carry no Mapping +semantics of their own and are derived from the probe result of the state being +observed. + +### The Graph Instance + +The `Observation` and `GraphInstance` container shapes, the no-padding rule, +the negative-one absent sentinel, the buffer lifetime, and the obligations +every column catalog satisfies are owned by +[The Graph Instance](spec-ml-core-environment.md#the-graph-instance). The +closed column catalogs this environment owns are: + +```text +DseNodeFeatureColumn = + Role // 0 + | EntityKind // 1 + | CapabilityCount // 2 + | CapacityMagnitude // 3 + | BufferDepth // 4 + | ResidualViolationCount // 5 + | PlacedDegree // 6 + | DecisionDomain // 7 + | DecisionKind // 8 + | DecisionValueForm // 9 + | DecisionValueOrdinal // 10 + | DecisionDeltaMagnitude // 11 + | DecisionDeltaSign // 12 + +DseArcRole = + Structural // 0 + | Placement // 1 + | Route // 2 + | DecisionTargetPrimary // 3 + | DecisionTargetMember // 4 + +DseScalarFeatureColumn = + StepOrdinal // 0 + | StepBound // 1 + | WorkloadCount // 2 + | ConsecutiveRejections // 3 +``` + +The six decision columns span the `Decision` block and the entity columns span +the entity blocks, on the core's role-block span rule. + +There is no node-ordinal column: a node's ordinal is its position in the array, +and the core's own rule forbids consuming an ordinal as a projected scalar. + +`EntityKind` is the owner-local reference kind ordinal from +[Fabric Identity](spec-fabric-identity.md) for a Fabric node and the Dataflow +owner's node or value kind ordinal for a Dataflow node. Every System entity +therefore appears as an ordinary `FabricOccurrence` node carrying its own +Fabric Identity kind ordinal, and transport and service connections appear as +ordinary `FabricConnection` nodes. + +There is no `LiveDecisionCount` scalar. The live count is the `Decision` +block's size, which the instance already carries, and a scalar beside it would +be a second source for one extent. + +A decision node carries at most one `Primary` target arc, its distinguished +subject when its owner defines one, and any number of `GroupMember` target +arcs, which are the members of a reference set or tuple the decision carries. A +decision whose owner defines no distinguished subject has only `GroupMember` +arcs. That is the shape a decision naming one entity plus a complete adapter +set requires; a projection that kept only the first target would silently +discard what the decision actually rewrites. + +When a bound domain operates on a Canonical Dataflow Program, the +`DataflowOperation` and `DataflowValue` blocks span every workload in the +state's current workload set rather than one workload, in collection-ordinal +order, so a rewrite decision's target node is addressable exactly as a Fabric +decision's target is. Advancing on a software decision changes how many nodes +those blocks contribute, which changes the instance's extents; that is ordinary +for this space and needs no accommodation. + +`CapabilityCount`, `CapacityMagnitude`, and `BufferDepth` are mechanical +projections of the exact Fabric owner facts for the referenced entity. +`ResidualViolationCount` and `PlacedDegree` are projections of the probe +result. The typed atoms of the `FabricResourceStateRef` catalog the core +appends under `include_resource_states` are owned by +[Fabric Resource Contract](spec-fabric-resource-contract.md). + +The scalars a rejected step updates in place, under the core's buffer-lifetime +rule, are the cleared mask bit and the step and rejection counters. + +A state whose enumeration exceeds `enumeration_bound` is rejected with +`EnumerationBoundExceeded`, which is the core's refusal rule reported under +this environment's own reason name; truncation would permanently hide the +decisions of whichever domains sort last. + +## Episode Start + +An episode is created from an exact seed set and an exact workload set. Each +seed is a finalized Fabric root whose kind matches `episode_subject`: a +`fabric.module` for a `SpatialModule` episode and a `fabric.system` for a +`System` episode. A seed is a builtin target, a user-supplied Fabric, or one +output of the `fabric_template` generator; exploration never begins from an +empty mutable graph, and a seed of the wrong kind is invalid rather than +adapted. Each workload is an exact Canonical Dataflow Program Artifact together +with the TechMapping the probe requires; both are ordinary static inputs and +acquire no environment-local identity. + +The `System` subject arm carries the admissible finalized Module candidate set +that kind 15 consumes, since `AddAccCore` and `ReplaceSpatialAttachment` select +from an explicit finite set rather than synthesizing a Module. That set is an +ordinary generator input the configuration supplies; the environment does not +author or extend it. A Module a Spatial domain derives mid-episode is reachable +by a later completion because it is the child of the decision being completed, +not because the environment added it to a candidate set it does not own. + +`reset` performs this ordered protocol: + +1. derive the episode's PRNG streams from the effective seed and the copy + coordinates; +2. select one seed root from `seed_roots` through `SeedSelection`, or adopt the + one the start override supplies; +3. select the episode's initial workload set through `WorkloadSelection`, or + adopt the one the start override supplies; +4. enumerate the seed and reject it when the enumeration is empty or exceeds + `enumeration_bound`; +5. insert the initial state's identity into the visited set; and +6. establish the workload-feasibility invariant by obtaining one closed Mapping + at the episode subject's closure level for every selected workload, then + build the first observation. + +Step 5 is what makes the seed state visited from the episode's first step. +Omitting it would make the one state every episode is guaranteed to occupy the +one state a decision may return to for free, and a pair of inverse decisions +would cycle through it indefinitely without ever being rejected as +`AlreadyVisited` — precisely the circling the visited set exists to catch. + +Step 4 precedes step 6 for the reason the step protocol orders its own tests: +the enumeration is a pure function of the seed and the configuration, while a +closure-level Mapping is the dominant start cost. Enumerating first means a +seed that could never offer a legal action is rejected for a fraction of one +Mapping, rather than after `workloads_per_episode` of them, and the retry +budget multiplies that saving by every redraw. + +Being a pure function of the seed and the enumeration-relevant fields of the +view, step 4's verdict is retained per seed root against those fields, and a +redraw that lands on a seed already judged consults that verdict instead of +re-enumerating. Keying it on the enumeration-relevant fields rather than on the +whole view is what lets a stage advance that only grows the workload pool keep +every verdict, on the same terms the Mapping cache survives one. A run of +`10^5` episodes over a few hundred seed roots would otherwise recompute a few +hundred fixed answers thousands of times each, and an unusable seed would be +re-enumerated and re-rejected on every episode that happened to draw it. This +is an ordinary cache: a hit and a miss produce the same formal result, and the +retry budget still bounds the redraws. + +The per-workload work in step 5 has no cross-item dependence and may run +concurrently, on the same terms stage 8 of a step states. + +The start override is the Gymnasium `options` argument, carrying an exact +episode start rather than a drawn one: + +```text +EpisodeStartOverride { + seed_root: ArtifactRootReference + workloads: ordered nonempty sequence +} +``` + +The core's override rules bind it: its members must appear in `seed_roots` and +`workload_pool`, and when present it replaces steps 2 and 3 entirely, so +neither `SeedSelection` nor `WorkloadSelection` is consulted. + +Step 5 is a precondition, not a best effort. A seed that cannot map every +selected workload is not a valid episode start: + +```text +EpisodeStartOutcome = + Started + | SeedInfeasible { seed_root, workload } + | SeedMappingNotClosed { seed_root, workload } + | SeedEnumerationEmpty { seed_root } + | SeedEnumerationBoundExceeded { seed_root, bound, required } + | RetryBudgetExhausted + | Invalid { violated precondition } +``` + +`SeedInfeasible` and `SeedMappingNotClosed` carry the proof-versus-budget +distinction the feasibility invariant defines. Either causes a retry with the +next draw from the same deterministic streams, bounded by `start_retry_bound`; +exhausting it is `RetryBudgetExhausted` and never a silently degraded episode +over a subset of the workloads. + +Under an episode start override there is no next draw, so `start_retry_bound` +is what the core's no-retry rule leaves unconsulted, and the outcome is +returned rather than converted to `RetryBudgetExhausted`. `Invalid` covers a +seed whose root kind does +not match the subject arm and a workload whose TechMapping does not bind the +selected seed. + +The two enumeration failures are both start failures rather than valid +episodes, and both are retried. A drawn seed whose enumeration is empty is +reported as `SeedEnumerationEmpty` because `Started` must guarantee at least +one legal action index; a drawn seed whose enumeration exceeds +`enumeration_bound` is reported as `SeedEnumerationBoundExceeded`, carrying the +capacity and the required length. Neither is `Invalid`, since another seed in +the same set may enumerate within capacity, and `reset` step 4 already tests +them together. Only a configuration whose every seed fails one of the two is an +invalid configuration, and that is reported when the retry budget is exhausted +over it. + +Mapping and TechMapping results may come from an ordinary cache family under +[Cache Dependencies](spec-config-ssot.md#cache-dependencies), which already +owns the versioned canonical dependency key, the removability rule, and the +requirement that a hit and a miss produce the same formal result. This document +adds only the granularity: the cached unit is one workload, not the episode's +workload set. A Mapping depends on the seed root, that one workload with its +TechMapping, and every probe view that produced it, and on nothing about which +other workloads the episode happened to draw. Every probe view means both the +Spatial PnR view and, for a `System` episode, the System PnR view its subject +arm carries: a System-episode entry produced under one System PnR contract must +not be returned to an episode running another, which is exactly the case where +a hit would return a Mapping a miss would not have produced. Keying on the set +would instead make the key space the combinations of the pool rather than the +pool itself, so changing one workload would discard the mappings of every other +workload in the draw even though they are byte-identical and reusable. +Per-workload keying makes the same cache reach a useful hit rate after a few +dozen episodes instead of effectively never. + +The same grant covers stage 8, and it matters more there than at `reset`. A +TechMapping is a pure function of the child identity, one workload, and the +TechMapping view, unconditionally. A probe result is a pure function of the +child identity, one workload with its TechMapping, and the probe views whenever +`warm_start_from_parent` is false, which is exactly the configuration whose +energy is path-independent. Both recur constantly once a policy converges on a +decision kind, and they recur across every environment copy in the run, so a +step-time cache reaches a useful hit rate far sooner than the start-time one it +extends. With warm starting enabled the probe is path-dependent by construction +and is not cacheable; that is a further cost of the option, and it is measured +rather than assumed. + +## Step + +One step applies exactly one enumerated decision. The protocol is ordered, and +every stage may reject: + +1. validate the action against the live mask and the retained per-state + rejection mask; +2. derive a fresh draft from the exact parent the decision's domain names and + apply the one owner-typed decision to it; +3. run the domain owner's ordinary acceptance path for that child, stopping + before durable publication; +4. publish the accepted child through its own generator when and only when the + decision's row completes by `Reattach`, skipping a component whose identity + the store already resolves; +5. derive the completion its domain's row names, so the state is again a + root-complete subject of the episode's kind; +6. reject when the resulting state identity is already in the visited set; +7. reject when the resulting state's enumeration exceeds `enumeration_bound`; +8. for each workload in canonical order, regenerate its TechMapping if the + decision invalidated it and then probe it against the candidate, + warm-started from the parent state's mapping when `warm_start_from_parent` + is set, requiring a closed Mapping with all five violation magnitudes zero; +9. read the `G` measures from the probe and every predicted metric from the + parameter-bundle inference kernels the selected closure's dimensions + reference; and +10. quantize each objective source, form the new `ObjectiveVector`, and + evaluate the selected search-energy `WeightedLevel`. + +The order is cheapest-rejecting-first as far as the completion form allows, and +where it holds it is load-bearing rather than incidental. Stages 6 and 7 both +answer from the completed state alone: a state identity and the enumeration +length are functions of that state and the resolved configuration, with no +Mapping, TechMapping, or objective value involved. Placing them ahead of stage +8 means a revisit or an over-capacity candidate never costs a probe, and both +are expected outcomes rather than corner cases. This is the property +[Schedule-Preserving Transformations](#schedule-preserving-transformations) +preserves by keeping a Mapping-derived reference set inside an entry whose +existence the Mapping does not decide. + +For a `None` or `ReplaceParent` completion the ordering is unqualified, because +stage 4 publishes nothing and those two rejections therefore also precede every +durable write. A `Reattach` is the exception, and it is the reason stage 4 sits +where it does. An Artifact owner requires every direct dependency of a root to +be durably published before that root's closure is derived, and a subject that +names a component depends on it, so a component held only as an unpublished +verified closure cannot be reattached at all. The completion therefore cannot +be derived before its component is published, and the state identity stages 6 +and 7 test does not exist until the completion has run. Deriving the completion +first and publishing afterwards is not a cheaper ordering but an invalid one: +it would name a subject closure over a dependency the store does not hold. + +A `Reattach` step consequently pays its component's store insertion before the +visited and capacity tests, which is the case where the memo below is worth the +most. + +Stage 4 deduplicates. The component's canonical identity is already exact from +stage 3, and the store is content-addressed, so a component whose identity the +store already resolves is not re-serialized and not re-inserted. A converging +policy revisits the same component constantly, and without this rule every +revisit would rewrite bytes the store already holds. An implementation may +answer stage 6 earlier still, from a memo of +`(parent state identity, EnumeratedDecision)` to child state identity, and skip +stages 2 through 5 entirely on a hit; the child identity is a deterministic +function of that pair, so the memo is an optimization and not a second identity +authority. On a `Reattach` row that memo is what keeps a revisit from paying a +publication for a state the episode is about to reject. + +Stage 8 interleaves regeneration with probing per workload rather than +regenerating every TechMapping first. The two orders are observationally +identical, because probe order is already the order in which a failure is +reported, but the interleaved one stops at the first failing workload instead +of paying regeneration for workloads it never probes. That matters because a +Fabric decision invalidates every workload's TechMapping while the probe aborts +on the first failure, so the barrier form would waste all but one regeneration +on precisely the rejections early abort exists to make cheap. + +Independent work within a stage may run concurrently. Per-workload regeneration +and probing in stage 8 and per-contract inference in stage 9 have no cross-item +dependence, and a concurrent implementation satisfies this contract exactly +when it reports the failure lowest in the configured probe order among those it +observed. Cancelling the remaining probes once a failure is known is permitted +and expected; what is fixed is the reported result, not the execution order. + +Stage 3 is what makes the candidate real without making it persistent. For a +Fabric domain it produces the `VerifiedFabricClosure` owned by +[Finalization And Publication](spec-fabric-artifact.md#finalization-and-publication) +and does not publish it, so the child has passed the same finalizer and +verifier as a published Fabric and carries the same canonical identity while +remaining un-referenceable by any other owner. For `DataflowRewrite` the child +passes the same rewrite match, normalization, and equivalence obligations its +catalog owner requires. In every case the candidate is fully verified rather +than provisionally accepted, and its identity is exact enough to serve as a +visited-set key. + +What stage 4 publishes is the component alone. It is published by its own +generator as an ordinary intermediate candidate, which is the status its owner +already gives it, while the subject closure that consumes it remains +unpublished. Deferred publication is a property of the episode's subject, not +of every value a step touches. + +This is a real cost and it is stated rather than hidden: a `Reattach` step pays +canonical publication and store insertion for its component, so a `System` +episode with a Spatial domain bound is more expensive per step than a +`SpatialModule` episode over the same decisions, and it accumulates +content-addressed objects for components no selected design may ever reference. +An episode that does not want that cost does not bind a Spatial domain under a +`System` subject; the alternating batches of the ordinary joint search reach +the same designs without it. + +Stage 5 exists because a generator's child is not always the episode's subject, +and it is driven by the "Completion" column rather than by named domain pairs: + +```text +Completion = + None + | Reattach { via: ExplorationDomainKind } + | ReplaceParent +``` + +`None` means the child is already the subject. `Reattach` means the child is a +component of the subject, so stage 4 publishes that child through its own +generator's ordinary path and the named domain then applies the one decision +that rebinds exactly the parent selector the original decision targeted, +leaving every other relation untouched. The reattach invocation receives the +published child as its admissible component set, which is an ordinary generator +input the caller supplies per invocation, not a configuration field the +environment extends. `ReplaceParent` means the child substitutes for its parent +in an ordinal-indexed collection the state holds, at exactly that parent's +ordinal, and publishes nothing. + +A row's completion is derived rather than chosen: `None` when the row's +"Operates on" kind is the episode subject's kind, `ReplaceParent` when it +operates on a member of a collection the state holds, and otherwise `Reattach` +via the unique bound row whose "Operates on" kind is the subject's. That is the +same derivation the `ParentSelector` rule already uses, so the table's +Completion column records the outcome rather than introducing a free parameter. + +No completion invents a decision the agent did not take. Both forms are +functions of the decision's own parent selector, which is why they need no +choice of their own; a domain whose completion would require a second free +choice cannot state its row and is therefore not admissible. A step may expand +to more than one generator invocation on replay, which Trajectory Retention And +Replay records exactly. + +Stage 8's regeneration runs on every advancing step, because a TechMapping is +bound to exact `D` and `F` and every decision changes one of them. What differs +is how much it invalidates: a software decision invalidates the TechMapping of +exactly the rewritten workload, while a Fabric decision invalidates every +workload's, so the stage is most expensive for the domain that changes the +fewest programs. Regeneration uses the +[root-complete TechMapping path](spec-tech-mapping.md#root-complete-central-adapter) +with `probe_policy.tech_mapping_config_view`. A workload whose TechMapping +cannot be regenerated has no probe input at all, which is a distinct rejection +from a workload that maps poorly. + +Stage 8 is also the feasibility gate defined above. Stage 9 obtains predicted +metrics by calling a registered `ModelParameterContract` 's `project_features` +and `infer` in process; this is the parameter-backed tier of the +fast-evaluation cascade owned by +[Full-Stack Architecture](spec-loom-stack.md#joint-design-exploration). A +predicted value may rank and may reward, but it can never reject a candidate as +impossible, and `infer` returning `Unsupported` outside its exact training +support region is never replaced by an extrapolation, a bound, a midpoint, or a +default. + +Only the contracts the selected closure's dimensions reference are invoked, and +that set is the definition rather than a configured list. Inference is the +expensive tier of the cascade, so a contract whose prediction no objective +dimension reads is never run at all. + +The outcome algebra is closed: + +```text +StepResult { + transition: optional + episode_end: optional +} + +StepTransition = + Advanced { energy_delta_code, energy_delta_sign } + | Rejected { reason: RejectionReason } + +RejectionReason = + DraftRejected { domain, authoring diagnostic } + | FinalizationRejected { domain, finalizer diagnostic } + | ClosureCompletionRejected { domain, completion diagnostic } + | TechMappingUnavailable { workload } + | AlreadyVisited { identity } + | WorkloadProvenInfeasible { workload, violation magnitudes } + | WorkloadMappingNotClosed { workload, residual violation magnitudes } + | ObjectiveUnavailable { dimension } + | EnumerationBoundExceeded { bound, required } + +TerminalReason = + ElectiveStop + | StepBoundReached + | ConsecutiveRejectionBoundReached + | EnumerationEmpty +``` + +`WorkloadProvenInfeasible` and `WorkloadMappingNotClosed` stay separate for the +reason the feasibility invariant states: one is a proof and the other is a +budget artifact. `TechMappingUnavailable` is separate from both because the +workload never reached the probe, so nothing was established about mappability +at all. `ClosureCompletionRejected` reports that a child was accepted by its +own owner but could not be reattached as a root-complete subject, which is a +System-level or workload-set-level failure rather than a defect in the child. +`ObjectiveUnavailable` is this environment's name for the core's +unavailable-objective outcome, and `EnumerationBoundExceeded` is its name for +the core's over-capacity refusal. Every reason names the domain, workload, or +bound it belongs to, so a rejection profile attributes cost and failure to the +domain that produced it. + +A transition and an episode ending are separate facts because they co-occur, on +the terms +[Step Accounting And Episode Endings](spec-ml-core-environment.md#step-accounting-and-episode-endings) +states. Here the rejection that reaches `consecutive_rejection_bound` both +rejects and ends the episode; the advance that lands in a state with no +admissible decisions both advances and ends it; and an elective stop ends the +episode with no transition at all. + +An absent transition means the policy ended the episode instead of acting, +which happens exactly for `ElectiveStop`; a third union member for that case +would let a consumer write two checks that can disagree. `EnumerationEmpty` is +reported on the step that advances into the empty state, not on a later call, +so the consumer is never handed a state with no legal action index. + +`Advanced` commits the candidate as the new current state, inserts its identity +into the visited set, and clears the per-state rejection mask for the new +state. Every `Rejected` transition restores the parent state exactly and +advances no episode position other than the step counter. The state identity +used for the visited set is the episode subject's identity together with the +ordinal-indexed identity sequence of the current workload set, so a software +decision that returns the design to a previously visited pair is caught exactly +as a hardware decision that does. + +`episode_end` distinguishes termination from truncation on the terms +[Step Accounting And Episode Endings](spec-ml-core-environment.md#step-accounting-and-episode-endings) +states. `ElectiveStop` and `EnumerationEmpty` are terminations; +`StepBoundReached` and `ConsecutiveRejectionBoundReached` are truncations. + +## Backtracking + +A step never mutates the parent. Stage 2 derives a fresh draft from the exact +parent, so a rejection has nothing to roll back: it discards the child and the +parent is the value it already was. The parent's canonical identity after a +rejection therefore equals its identity before the attempt as a consequence of +the protocol, not as a promise about anyone's restore path. The only +transactional state in a step is internal to the probe, where +[Evaluation Transaction](spec-pnr.md#evaluation-transaction) owns `commit` and +`discard`. + +The environment then clears the rejected decision's bit in a per-state +rejection mask so the same decision is not re-proposed from the same state. The +mask belongs to the state, not to the episode: advancing to a new state starts +a new empty mask, and returning to a previously visited identity is already +rejected by the visited set. The visited set retains up to +`visited_state_retention` identities in insertion order, which is what prevents +an episode from spending its step budget alternating between two states. + +The step accounting identity +[Step Accounting And Episode Endings](spec-ml-core-environment.md#step-accounting-and-episode-endings) +states holds over an episode of this environment with no unaccounted remainder. +Consecutive rejections are counted, exposed through `scalar_features`, and end +the episode at the configured bound. + +## Reward + +Reward is the signed difference of the selected search energy across the +transition, on the terms +[Reward Contract](spec-ml-core-environment.md#reward-contract) states. This +document says only which two states the difference is taken between and what +this environment's non-transition outcomes cost. + +The parent's energy the core retains is the one produced when the parent was +entered, by `reset` or by the stage 10 that advanced into it. + +A `Rejected` transition yields `rejection_reward_code` with a negative sign, +independent of the rejection reason. + +`stop_reward_code` applies with a negative sign to `ElectiveStop` alone. +`StepBoundReached` and `ConsecutiveRejectionBoundReached` yield zero because +they are truncations, which the core prices at zero. `EnumerationEmpty` also +yields zero, since the step that reports it already carries its own `Advanced` +reward. Only the ending the policy elects is priced. + +Path dependence is a consequence of `warm_start_from_parent` and is stated +rather than hidden. Warm starting seeds each probe from the parent's mappings, +so the mapping a probe returns, and therefore the `G` measures and the energy, +can depend on how a state was reached. State identity does not depend on the +path, so two paths to the same design may report different energies and a cycle +of deltas need not sum to zero. A configuration that requires path-independent +energy, such as one whose analysis assumes telescoping deltas, sets +`warm_start_from_parent` false and pays the cold-probe cost; a configuration +optimizing throughput sets it true and accepts a reward that is exact per +transition but not a potential function. + +## Trajectory Retention And Replay + +An episode retains one transient record: + +```text +Trajectory { + config_view_digest: ComponentViewDigest + episode_subject: SpatialModule | System + seed_root: ArtifactRootReference + initial_workloads: ordered sequence + effective_seed: u64 + env_runner_index: uint32 + vector_index: uint32 + local_episode_index: uint64 + advanced_steps: ordered sequence +} + +TrajectoryStep { + decision: EnumeratedDecision + completion: optional + child_identity: ArtifactIdentity +} +``` + +A step records the decision the agent chose, the mechanical completion stage 5 +derived, and the exact identity of the child that step produced, which stage 3 +computed whether or not it published. + +Replay is a finite sequence of resolved plans, one per recorded step, each +resolved after its predecessor completed. A step's plan binds its parent as an +exact static artifact set, which is available because the preceding step's plan +published it, runs the `Generate` node for the recorded decision, and selects +the output whose identity equals `child_identity`. Selection by recorded +identity is an ordinary property of the retained trajectory, not a new plan +mechanism: the next step's plan binds that exact artifact as a static input. +There is no `Promote`, no best-child rule, and no Artifact Store scan. + +How narrowly a step's node can be aimed depends on what its generator's +configuration exposes, and the two cases differ. A hardware rewrite generator +owns a decision-domain set whose member fixes one exact target selector plus a +finite value set or bounded range, so replay may bind the singleton domain +admitting exactly the recorded decision; the node then produces that one child +and publishes no siblings. The Dataflow rewrite generator exposes no such +narrowing. Its whole resolved configuration is one expansion limit, and it +enumerates the catalog over a frontier in canonical order, so the smallest node +that reaches the recorded decision also publishes every decision preceding it. +Those siblings are ordinary valid Artifacts that the plan does not select, and +accepting them is the cost of replaying a software step. Claiming a singleton +domain for that generator would be claiming a configuration field it does not +have. + +Replay is therefore one `Generate` node per recorded decision, including a +completion when the step carries one, with no special arithmetic per domain: a +`Reattach` completion is itself an enumerated decision of the domain its row +names, and a `ReplaceParent` completion records none because the generator's +own child is already the new collection member. Replay publishes real Artifacts +at every step, with the ordinary `CandidateDecision` lineage contributions +those descriptors own, which is exactly the cost the episode deferred. A replay +that selects any output other than the recorded identity is a different plan +that explores a neighborhood; it is not this trajectory. + +A `Trajectory` is a compact way to name a sequence of generator decisions so +that an authoritative run can reconstruct them; a candidate it names becomes +real only when a real `Generate` node publishes it and a real `Promote` node +acquires Evidence and applies quality gates. + +## Python Boundary + +The `loomml` package layering, the RLlib conformance target and its +obligations, the zero-copy array contract, the outcome-versus-exception rule, +and the threading and `fork` rules are owned by +[Python Boundary](spec-ml-core-environment.md#python-boundary). This +environment occupies `loomml.env.dse` and `loomml.rllib.dse` and adds no rule +of its own. + +Two of the core's obligations are worth naming against this environment's own +values, because they are what the obligations are for here. The decision nodes +and target arcs the parametric-action convention carries are the ones The Graph +Instance defines, and the `reset` seed the caller supplies replaces +`determinism_policy.master_seed` as the episode's effective seed and is +recorded in the `Trajectory`. + +## Benchmarking + +The harness obligations, the required breakdowns, the instrumentation rule, and +the ratio-based regression budget are owned by +[Benchmarking Harness Contract](spec-ml-core-environment.md#benchmarking-harness-contract). +This environment ships `loom-dse-env-bench`, whose rejection-reason breakdown +is keyed by `RejectionReason` and whose decision partition is +`ExplorationDomainKind`, because the domains do not cost the same. + +The harness decomposes one step into these stages and reports each separately: + +```text +enumeration build the canonical decision order and mask +draft_apply derive the child draft and apply one decision +accept the domain owner's acceptance path for the child +identity canonical bytes and root identity +completion the completion its domain's row names +techmap regenerate an invalidated TechMapping +probe per workload, closed Mapping or typed failure +inference project_features and infer per referenced contract +observation build the observation arrays +marshal expose buffers across the Python boundary +``` + +and one reset into `seed_load`, `cache_lookup`, `cold_mapping` per workload, +and `first_observation`. + +Two further breakdowns are required beyond the core's two. The probe is +reported per workload, together with the mean workloads probed and TechMappings +regenerated per rejected step, which is what shows whether the interleaving of +stage 8 is earning its place. And `warm_start_from_parent` is measured the same +way, with and without the parent's mappings, split by whether the decision +changed the Fabric or a workload. + +## Anchor Verification + +Stable tests cover adoption rejecting a reward closure whose Mapping-sourced +dimension the probe view does not admit, whose Evaluation-sourced dimension no +configured parameter contract satisfies, or which references a held-out +obligation; the set of parameter contracts an episode invokes being exactly the +set the reward closure's Evaluation-sourced dimensions reference, with no +configuration field beside it; adoption rejecting a duplicate domain binding, a +bound row whose required subject does not match the subject arm, and a +`Reattach` completion whose named domain is not itself bound; an omitted domain +contributing no enumerated entry, no mask bit, and no observation block; an +exploration domain whose decisions carry a reference the combined space cannot +represent being refused at adoption; a decision whose row operates on the +subject's own kind rejecting any selector but `EpisodeSubject`, and a decision +whose row operates on a component requiring a `SubjectComponent` or +`CollectionOrdinal` selector; a `ReplaceParent` completion leaving every other +collection ordinal unchanged; a state whose enumeration exceeds +`enumeration_bound` reporting `EnumerationBoundExceeded` rather than being +shortened; a revisited or over-capacity candidate being rejected without any +probe or TechMapping regeneration running; a schedule-preserving decision's +`GroupMember` references being those the state's Mapping routes through its +`Primary` target, the enumeration's length being unchanged by whether a Mapping +exists, a reconnection the domain owner refuses being an ordinary rejection, +and a replayed step reproducing the recorded references without reconstructing +a probe; a decision returning the design to +the seed state being rejected as `AlreadyVisited` on the episode's first +opportunity; a start failure under an `EpisodeStartOverride` being returned as +it stands rather than retried or converted to `RetryBudgetExhausted`; a +rejected step regenerating no +TechMapping for a workload the probe never reached; a rejection that reaches +the consecutive-rejection bound reporting both its `RejectionReason` and its +`TerminalReason`; an advance into an empty enumeration reporting both on the +same step; enabling a domain shifting only the enumeration suffix at and after +its ordinal; enumeration and mask agreement, including a cleared bit after a +rejection at that state; a draft, acceptance, or completion rejection producing +no child, no identity, and no lineage; a decision that removes a capability, +route, capacity, AccCore, or transport connection a workload requires being +rejected with the correct per-workload reason and leaving the parent +byte-identical; a rewrite whose child the current hardware cannot map being +rejected with the same per-workload reasons as a hardware decision; a workload +whose TechMapping cannot be regenerated reporting `TechMappingUnavailable` +rather than a mapping outcome; a probe budget too small to close a reachable +Mapping reporting `WorkloadMappingNotClosed` and never +`WorkloadProvenInfeasible`; a concurrent probe implementation reporting the +same failure as a sequential one; every state reachable within an episode +satisfying the feasibility invariant at the episode subject's closure level; a +`Reattach` completion publishing its component and leaving a root-complete +subject whose only changed relation is the targeted one; `reset` refusing a +seed whose root kind does not match the subject arm and refusing a seed that +cannot map every selected workload; a drawn seed with an empty enumeration +reporting `SeedEnumerationEmpty` and one whose enumeration exceeds +`enumeration_bound` reporting `SeedEnumerationBoundExceeded`, both retrying +rather than starting and neither being `Invalid`; a `Reattach` step publishing +its component before the visited and capacity tests while a `None` or +`ReplaceParent` step publishes nothing before them; a realization arc carrying +`Placement` and a route-traversal arc carrying `Route`; an +`EpisodeStartOverride` consulting neither selection stream and producing the +same episode on every copy, and one naming inventory outside the configured +sets being rejected; a per-workload cache entry surviving a change to another +workload in the draw, and a `System`-episode entry not being returned under a +different System PnR view; `ObjectiveUnavailable` never producing a numeric +reward; a truncation yielding zero reward while an elective stop yields the +configured stop code; equal configuration, coordinates, seed, and action +sequence reproducing an identical trajectory; and trajectory replay selecting +the output whose identity equals `child_identity` at every step and reproducing +identical finalized identities in order, including the node a `Reattach` +completion contributes, with a hardware step narrowed to a singleton domain +publishing no sibling and a software step publishing the catalog prefix it +must. + +Tests do not pin the live node, arc, or decision counts of any particular +state, the bound values a profile selects, which reconnection a particular +Mapping implies, wall-time numbers, per-domain cost ratios, probe-ordering +heuristics, corpus contents, diagnostic text, or Python formatting. diff --git a/docs/spec-ml-dse-model-architecture.md b/docs/spec-ml-dse-model-architecture.md new file mode 100644 index 000000000..027ebe2ed --- /dev/null +++ b/docs/spec-ml-dse-model-architecture.md @@ -0,0 +1,316 @@ +# ML DSE Model Architecture + +This document defines the learned policy that consumes the ML DSE Environment's +observations and emits its actions. It owns the embedding of the environment's +graph observation, the graph-transformer encoder over that embedding, the +hierarchical policy head that scores enumerated decisions, the value head, the +masking contract, and the parameter and checkpoint boundary. + +The model is a search policy on the terms +[ML Model Core](spec-ml-core-model.md) states, and every legality, mappability, +and objective fact remains owned where +[ML DSE Environment](spec-ml-dse-environment.md) places it. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [ML Model Core](spec-ml-core-model.md) owns the module boundary, observation + batching, the feature-embedding discipline, the encoder, the graph context, + the value head, masking, the action distribution, the checkpoint boundary, + and the obligations every forward-pass harness satisfies; +- [The Graph Instance](spec-ml-core-environment.md#the-graph-instance) owns the + observation container, and + [The Graph Instance](spec-ml-dse-environment.md#the-graph-instance) owns this + environment's node and arc feature catalogs, action mask, objective codes, + and scalar features; +- [Action Space](spec-ml-dse-environment.md#action-space) owns the enumerated + decisions, their canonical order, their targets, and `ActionIndex`; +- [Combined Node And Link Space](spec-ml-core-environment.md#combined-node-and-link-space) + owns the node roles, the arc-role obligation, and the target-closure rule + this document's policy head relies on; +- [Reward](spec-ml-dse-environment.md#reward) owns the reward the value head + regresses toward; +- [RLlib Environment Definition](spec-ml-core-environment.md#rllib-environment-definition) + owns the trainer-facing environment contract and the forked trainer this + model plugs into; +- [Evaluation and DSE](spec-dse-feedback.md#model-parameters-and-training) owns + `ModelParameterContractRef` and the registered prediction contracts, which + this model is not; and +- [ML Training Core](spec-ml-core-training.md) owns the algorithm binding and + the loss it brings, the rollout and learner topology, the stage sequence, and + every hyperparameter schedule, and + [ML DSE Training](spec-ml-dse-training.md) owns this environment's reachable + return range. + +This document owns only the DSE feature mapping, the hierarchical policy head, +the model configuration view, and its own benchmarking stages. It restates no +environment fact and defines no new observation, action, or reward. + +## Module Boundary And Batching + +The module boundary, the three forward entry points and their shared encoder +pass, `compute_values` accepting precomputed embeddings, and the disjoint-union +batching of ragged graph observations are owned by +[Module Boundary](spec-ml-core-model.md#module-boundary) and +[Observation Batching](spec-ml-core-model.md#observation-batching). This model +is `LoomDseRLModule` and adds no entry point and changes no signature. + +This environment uses the `DecisionNodes` enumeration encoding, so its +enumerated decisions are nodes of the observation graph and it supplies no +separate decision instance. `ActionIndex` is a per-instance decision-block +ordinal and is mapped through `ptr` and that block's offset. + +## Feature Embedding + +The embedded categorical columns are: + +```text +column table size source catalog +Role GraphNodeRole cardinality environment +EntityKind Fabric local-reference kinds + plus Dataflow node/value kinds Fabric Identity, Dataflow +DecisionDomain ExplorationDomainKind cardinality environment +DecisionKind sum over domains of that + domain's decision-union size candidate generators +DecisionValueForm value-form cardinality environment +DecisionValueOrdinal per-domain prototype cardinality candidate generators +DecisionDeltaSign sign cardinality environment +ArcRole DseArcRole cardinality environment +``` + +`DecisionKind` is embedded on the `(DecisionDomain, DecisionKind)` pair rather +than on the kind ordinal alone, because kind ordinals restart per domain and a +shared table would tie unrelated decisions together. `DecisionValueOrdinal` is +the direct analogue of the old architecture's per-instruction embedding table: +it is what lets the policy reason about a specific replacement prototype rather +than about an opaque index, and it is the axis along which the value factor of +the hierarchical head operates. + +Numeric projection, the absent-column rule, the signed logarithmic transform, +the capacity-usage-occupancy triple, the mapping of `objective_codes` into +`[0, 1]`, and the concatenate-then-project input path are owned by +[Feature Embedding](spec-ml-core-model.md#feature-embedding). Two of this +environment's columns need the treatment named rather than derived. + +`ResidualViolationCount` and `PlacedDegree` are projections of the probe result +rather than of the design, so they change when the same design is probed +differently; they are projected as ordinary magnitudes and carry no special +status, but a configuration comparing runs across a changed probe view is +comparing two different features under one name. + +`StepOrdinal` and `StepBound` are supplied as their ratio as well as their +magnitudes, so remaining budget is directly available; `ConsecutiveRejections` +is supplied as a magnitude, as is the `Decision` block's size. + +## Encoder And Context + +The `GraphTransformerStack` record, pre-normalization, the feed-forward +sublayer, edge-aware attention, jumping knowledge, the one-shared-trunk rule, +and the multi-scale per-instance graph context are owned by +[Encoder](spec-ml-core-model.md#encoder) and +[Graph Context](spec-ml-core-model.md#graph-context). + +This environment's `DseArcRole` catalog supplies the edge features. `Placement` +and `Route` are the probe's realization and traversal relations, which differ +in arity and in which decisions move them, and `DecisionTargetPrimary` against +`DecisionTargetMember` is what lets a decision node's attention distinguish the +entity it rewrites from the set it rewrites it against. + +`DecisionDeltaSign` is embedded rather than projected because a sign is +categorical, which is what the value term's delta form requires: a magnitude +carries the size of a change and the sign carries its direction, and projecting +the sign as a number would make a negative delta a small positive one. + +## Hierarchical Policy Head + +The environment's action is one `ActionIndex` plus a stop flag, and its +decision nodes carry both their target arcs and their value columns. The head +therefore scores decisions where they live, on the graph, and factors the score +into a part that depends on where the decision acts and a part that depends on +what value it selects. + +Every decision node `d` has an anchor and a value: + +```text +anchor(d) = ( primary target node of d, + pooled GroupMember target set of d, + DecisionDomain of d, + DecisionKind of d ) + +value(d) = ValuePrototype(DecisionValueOrdinal) if form is ReplacementPrototype + | ValueDelta(DecisionDeltaMagnitude, DecisionDeltaSign) + if form is BoundedIntegerDelta + | Absent if form is NoValue +``` + +The anchor includes the whole target set, not only the primary, so a decision +naming an entity plus a reference set is scored on the set it actually +rewrites. Decisions sharing an anchor form one anchor group; a group is a +singleton exactly when its decisions carry no value. + +`value(d)` reads whichever columns its form populates. A `BoundedIntegerDelta` +decision has no prototype ordinal, and its ordinal column is the absent +sentinel, so a value term reading only `DecisionValueOrdinal` would give every +delta on one target an identical score and make the magnitudes +indistinguishable. A delta is embedded from its magnitude and sign instead: the +sign is categorical, and the magnitude is projected, so adjacent magnitudes +score similarly and the head generalizes across a range rather than memorizing +each step of it. + +The head produces two scores per decision: + +```text +s_anchor(d) = MlpAnchor([ h_primary(d), h_members(d), e_kind(d), context ]) +s_value(d) = 0 if value(d) is Absent + = MlpValue([ h_primary(d), e_value(d), e_kind(d), context ]) + otherwise +logit(d) = s_anchor(d) + s_value(d) +``` + +`s_anchor` is constant within an anchor group by construction, because the +anchor includes every input `MlpAnchor` reads: the primary target, the pooled +member set, and the kind, evaluated against one shared context. This is what +makes the factorization a genuine hierarchy rather than a reparameterization: + +A boolean decision, one carrying no value, is a singleton group whose +conditional factor is one, so it is scored directly on its node or link exactly +as asked. A value-carrying decision is scored on its node or link by the anchor +factor and among its alternatives by the value factor, which is the operation +selection of the prior architecture generalized to every value-carrying +decision kind rather than to instructions alone. + +The factorization, the group-constancy rule, the flat-versus-two-level +equivalence, the broadcast saving, and the stop logit are owned by +[Policy Head Factorization](spec-ml-core-model.md#policy-head-factorization); +the joint distribution these logits feed and its stop encoding by +[Action Distribution](spec-ml-core-model.md#action-distribution). + +Slots beyond the instance's `Decision` block size have no decision node and +receive a masked logit; the stop outcome is masked only when the environment's +mask clears it. + +## Value Head, Masking, And Distribution + +The value head and its bound, the masking discipline and its five rules, and +the joint action distribution over `enumeration_bound + 1` outcomes are owned +by [Value Head](spec-ml-core-model.md#value-head), +[Masking](spec-ml-core-model.md#masking), and +[Action Distribution](spec-ml-core-model.md#action-distribution). The +checkpoint boundary is owned by +[Parameters And Checkpoints](spec-ml-core-model.md#parameters-and-checkpoints). + +One property of this environment is worth naming against the masking contract. +The environment's own mask already carries a per-state rejection record, so a +bit it clears may encode that a decision was tried at this state and failed. +That is a fact about the episode rather than about the decision, and it is the +reason the recorded mask must travel in the batch: recomputing it at update +time against a different episode position would mask a decision that was +available when it was sampled. + +## Model Configuration + +The model's shape is one immutable component view with schema descriptor bytes +`loom.ml_dse_model.config.1.0`, following the framing owned by +[Component Views](spec-config-ssot.md#component-views) and the three properties +[Model Configuration Framing](spec-ml-core-model.md#model-configuration-framing) +fixes: + +```text +ResolvedMlDseModelConfigView { + embedding_widths: total table + encoder: GraphTransformerStack + head_widths: DseHeadWidths + value_bound: positive uint64 + advisory_mask_set: canonical set + initialization: InitializationConstants +} +``` + +```text +DseHeadWidths { + anchor_hidden: positive uint32 + value_head_hidden: positive uint32 + value_hidden: positive uint32 +} +``` + +`value_bound` is the symmetric range the value head clamps to, in the scaled +reward's units. `advisory_mask_set` names the optional masks the masking +contract admits. + +## Inference Benchmarking + +The harness obligations, the two regimes, the entry-point split, the scaling +rule, the rollout split, invariance against an unoptimized reference, and +ratio-based budgets are owned by +[Forward Benchmarking Obligations](spec-ml-core-model.md#forward-benchmarking-obligations). +This model ships `loom-dse-model-bench`. + +The harness decomposes one forward pass into these stages: + +```text +gather assemble the batched graph and move it to the device +embed embedding lookups, numeric projection, input projection +encode the graph-transformer stack +pool per-instance multi-scale context +policy anchor and value scoring over decision nodes +mask mask composition and additive application +distribute distribution construction and sampling +value the value head, when the regime computes it +``` + +`encode` is the expected dominant term, since attention cost scales with arc +count, and `gather` is the expected dominant term for a small graph on an +accelerator, where the transfer costs more than the arithmetic. Both +expectations are worth measuring rather than assuming. + +Each stage is reported against node count, arc count, and live decision count, +with the fitted growth of `encode` against arc count and of `policy` against +live decision count. + +Three optimizations specific to this head are measured against an unoptimized +baseline, and each carries a failure mode a naive measurement will miss. + +Anchor-group evaluation, on the terms +[Policy Head Factorization](spec-ml-core-model.md#policy-head-factorization) +states. The mean group size is reported alongside the `policy` stage. + +Live decisions only. The head emits logits over `enumeration_bound` slots, but +only the live ones have decision nodes. Scoring runs over live decisions and +scatters into the emitted logit vector; running the scoring +multilayer perceptrons over dead slots wastes work proportional to the gap +between capacity and live count, which is largest exactly when the capacity is +set conservatively. + +Batched inference across copies. Vectorized environment copies on one runner +present several instances at once, and batching them into one union graph +amortizes the per-call overhead that dominates latency-bound inference. The +harness reports latency against the copy count to show where that amortization +saturates. + +Device placement is measured across the size range and the crossover is +reported rather than assumed: for small graphs, host inference can beat +accelerator inference because the transfer in `gather` exceeds the arithmetic +it feeds. + +## Conformance Anchors + +Stable tests cover an `ActionIndex` resolving to its own instance's decision +block; a decision carrying a reference set contributing every member to its +anchor summary rather than only its primary; a value-free decision receiving a +zero value term and forming a singleton anchor group; two decisions sharing an +anchor receiving an identical anchor score; two `BoundedIntegerDelta` decisions +on one target with different magnitudes receiving different scores; the +two-level and flat formulations producing identical distributions to numerical +tolerance; a stop logit competing in the same normalization as the decision +logits; a per-resource capacity and usage pair reducing to one resource +embedding rather than widening the node vector with the resource catalog; +scoring restricted to live decisions producing logits identical to scoring +every slot; the mean anchor-group size being reported alongside the `policy` +stage; and every optimized configuration reproducing the unoptimized +reference's logits and values within the declared tolerance. + +Tests do not pin layer counts, widths, head counts, initialization constants, +the advisory mask set, learned parameter values, wall-time numbers, device +crossover points, or throughput. diff --git a/docs/spec-ml-dse-training.md b/docs/spec-ml-dse-training.md new file mode 100644 index 000000000..951c40619 --- /dev/null +++ b/docs/spec-ml-dse-training.md @@ -0,0 +1,597 @@ +# ML DSE Training + +This document defines how the search policy of +[ML DSE Model Architecture](spec-ml-dse-model-architecture.md) is fitted +against the environment of [ML DSE Environment](spec-ml-dse-environment.md). It +owns the split of a run's settings into separate resolved configurations, the +binding to an existing PPO implementation, the per-episode statistics a run +reports, the fixed test set and the deterministic protocol that evaluates it, +and the offline corpus of training architectures and synthetic training +workloads. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [ML DSE Environment](spec-ml-dse-environment.md) owns the episode protocol, + the action space, the observation, the reward's exact `(sign, magnitude)` + pair, the step and rejection outcome algebra, the terminal reasons, the + `Trajectory`, the Python layering, and `ResolvedMlDseEnvironmentConfigView`; +- [ML DSE Model Architecture](spec-ml-dse-model-architecture.md) owns the + module boundary, the encoder, the heads, the action distribution, the mask + composition and its fallback, and the checkpoint's parameter and + configuration match; +- [ML Training Core](spec-ml-core-training.md) owns the nonsemantic boundary, + the configuration split and inventory identity, the algorithm binding + surface, the stage sequence and its handoff, exact-rational hyperparameters, + schedules, the reward adapter, topology, the stage invariance rule, the + statistics production rule, the test protocol, checkpoints and run identity, + and reproduction; +- [Objectives and Quality Gates](spec-dse-feedback.md#objectives-and-quality-gates) + owns `ObjectiveDimension`, `ExactAffineQuantization`, directed codes, + `ObjectiveVector`, `WeightedLevel`, and `SearchEnergyRef`; +- [Model Parameters and Training](spec-dse-feedback.md#model-parameters-and-training) + owns the training of registered prediction contracts, which is a different + activity from this one and shares no record with it; +- [Candidate Generators](spec-dse-feedback.md#candidate-generators) owns the + `fabric_template` generator this document's architecture corpus is produced + by; +- [Evaluation Metrics](spec-evaluation-metrics.md#metric-registry) owns every + `MetricKind` and `ExactRatio`, and + [FPA Evaluation](spec-fpa-estimation.md#metrics) owns which physical metrics + are registered and what may produce them; +- [Component Views](spec-config-ssot.md#component-views) owns view framing, + canonical bytes, and `component_view_digest`, and + [Cache Dependencies](spec-config-ssot.md#cache-dependencies) owns the cache + family the corpus prepopulates; +- [Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism) + owns the seeded PRNG protocol; +- [Source Integration](spec-compiler-part-1-source.md) and the compiler + pipeline own how any program, synthetic or real, becomes a Canonical Dataflow + Program; +- [Corpus](spec-loom-stack.md#corpus), [LoomBench](spec-loombench.md), and + [Real Application Portfolio](spec-application-portfolio.md) own the real + program inventories, which this document's synthetic corpus is never part of; + and +- [External Dependency Pinning](spec-loom-stack.md#external-dependency-pinning) + owns the Ray fork revision and its patch stack. + +This document owns only this environment's return-range arithmetic and its +discounting consequence, the DSE statistics catalog and breakdown axes, the +test case payload and the granularity its isolation is compared at, the corpus +view and its offline generator, and its harnesses. + +## Nonsemantic Boundary + +Under +[Nonsemantic Boundary](spec-ml-core-training.md#nonsemantic-boundary), the +owner-owned path by which a design this run discovers becomes a real candidate +is the one +[Trajectory Retention And Replay](spec-ml-dse-environment.md#trajectory-retention-and-replay) +defines. + +The corpus this document generates is the one place training touches durable +state, and it touches it only through existing owners. A training architecture +is an ordinary finalized Fabric Artifact and a training workload is an ordinary +Canonical Dataflow Program Artifact, each produced by the owner that already +produces them. The corpus view names those roots; it does not define a second +Artifact kind, a second program authority, or a store of its own. + +## Configuration Split + +This run's views, under the framing +[Configuration Split](spec-ml-core-training.md#configuration-split) owns: + +```text +loom.ml_dse_environment.config.1.0 ResolvedMlDseEnvironmentConfigView +loom.ml_dse_model.config.1.0 ResolvedMlDseModelConfigView +loom.ml_dse_training.config.1.0 ResolvedMlDseTrainingConfigView +loom.ml_dse_corpus.config.1.0 ResolvedMlDseCorpusConfigView +loom.ml_dse_test_set.config.1.0 ResolvedMlDseTestSetConfigView +loom.ml_dse_run.config.1.0 ResolvedMlDseRunConfigView +``` + +```text +ResolvedMlDseRunConfigView { + training_config_digest: ComponentViewDigest + model_config_digest: ComponentViewDigest + corpus_config_digest: ComponentViewDigest + test_set_config_digest: ComponentViewDigest +} +``` + +`ResolvedMlDseTrainingConfigView` is the core's `ResolvedTrainingConfigView` +under this descriptor, with no field added. Every stage binds `Ppo`; a run that +wants a curriculum spends more than one stage on it, and one that does not is +the single-stage case. + +The cached asset the core's digest-independence rule names is, here, the +Mapping cache family; it takes a few dozen episodes to become useful. + +Adoption validates each view independently, then validates the run view's +cross-view conditions: every `seed_roots` and `workload_pool` member of every +bound training environment view appears in the corpus view; the test set's own +environment view and its cases are admitted under +[Test Protocol](#test-protocol) and are disjoint from the corpus under +[Isolation](#isolation); the +model view's embedding table sizes match the owner catalog cardinalities the +environment view's bound domains reach; and the model view's value bound covers +the reachable return range below, maximized over every bound environment view. + +## Training Configuration + +The training view, the algorithm bindings, the stage sequence and its handoff, +exact-rational hyperparameters, schedules, and the reward adapter's single +scale are owned by +[Training Configuration](spec-ml-core-training.md#training-configuration). This +environment supplies two things of its own. + +### The Reachable Return Range + +This environment's declaration under the core's obligation. A transition's +reward magnitude is bounded by the quantization width of the search energy's +dimensions, and every step contributes one term: an advancing step at most that +width, a rejected step `rejection_reward_code`, and the one step that may be an +elective stop `stop_reward_code`. An episode carries at most `step_bound` +steps, so the undiscounted return is bounded by + +```text +step_bound * max(energy_width, rejection_reward_code) + stop_reward_code +``` + +all times the adapter's scale. The core's every-step rule is what makes the +rejection code multiply `step_bound` here rather than appear once, and it binds +because a policy early in training rejects far more often than it advances. + +### Path Dependence And Discounting + +Discounting at one has an exact meaning here worth stating. When +`warm_start_from_parent` is clear, the environment's energy is +path-independent, so an undiscounted return telescopes to the total improvement +from the seed to the final design and every intermediate reward cancels. A run +that wants that property sets both, and a run that discounts is deliberately +preferring earlier improvement to later. With warm starting set the telescoping +does not hold, because the energy a state reports depends on how it was +reached, and this document states that rather than letting a configuration +assume otherwise. + +## Algorithm Binding And Topology + +The algorithm binding surface, what Loom supplies to the trainer, the +instrumentation-only rule for a `Learner` subclass, rollout and learner +topology, and the stage sequence with its invariance rule are owned by +[ML Training Core](spec-ml-core-training.md); the fields a stage may vary are +declared by +[Curriculum-Neutral Fields](spec-ml-dse-environment.md#curriculum-neutral-fields). + +One of those lands on this environment's own values. Its step is dominated by +per-workload Mapping probes, as +[Benchmarking](spec-ml-dse-environment.md#benchmarking) reports, so sampling is +normally the binding cost and the topology is chosen by moving runner count +against that split. + +## Episode Statistics + +The production rule, `DimensionEpisodeStatistic`, the outcome accounting, the +cost statistics, `LoggingPolicy`, and chunk invariance are owned by +[Episode Statistics](spec-ml-core-training.md#episode-statistics). This +environment names its own vocabulary and adds what only it has. + +Its non-advancing step class is `rejected`, so the core's accounting identity +reads `steps == advanced + rejected + elective_stops`, and the rejection rate +is reported per member of `RejectionReason` and per `ExplorationDomainKind`. A +single aggregate rate conflates causes that call for opposite responses: +`AlreadyVisited` is search behavior, `WorkloadMappingNotClosed` is a +probe-budget problem, `WorkloadProvenInfeasible` is a real proof about the +design, and `EnumerationBoundExceeded` is a configuration error. Its action +partition is the `(ExplorationDomainKind, decision kind)` pair, because a +decision kind ordinal restarts per domain and a shared key would tie unrelated +decisions together. + +Two statistics are this environment's own. `visited_revisit_rate` is the share +of rejections that were `AlreadyVisited`, which is what says whether the policy +is circling. And a dimension of the selected closure that sits outside the +search-energy level appears in these statistics and contributes nothing to the +reward, which is how a run watches a metric it does not optimize; a dimension +absent from the closure produces an absent statistic rather than a zero. + +```text +DseBreakdownAxis = + Stage // 0, mandatory + | TerminalReason // 1, mandatory + | SeedRoot // 2, per architecture + | Workload // 3, per Canonical Dataflow Program + | Domain // 4, per ExplorationDomainKind + | RejectionReason // 5 +``` + +`SeedRoot` and `Workload` are the expensive members: a corpus with a few +hundred architectures and a few thousand workloads turns every statistic into +hundreds of thousands of series, which is why axes are opt-in. + +### Occupancy Statistics + +Utilization is reported when the environment's `include_resource_states` is +set, and is absent otherwise. Per episode, at entry and at the final state: + +```text +resource_occupancy[FabricResourceStateRef] usage / capacity, as ExactRatio +placed_occurrence_fraction nodes with PlacedDegree > 0 +mean_residual_violation_count +``` + +Occupancy is the ratio of the usage and capacity columns the observation +already carries per resource state, aggregated over the episode's Fabric nodes. +It is the quantity a hardware decision actually trades — headroom — and it is +the direct check on whether the search is removing resources the workload was +not using or resources it was. A run whose area improves while occupancy stays +flat is removing slack; one whose area improves while occupancy climbs toward +one is approaching the feasibility gate, and its rejection rate will confirm +it. + +`placed_occurrence_fraction` is the Mapping's own view of the same question, +from the probe result's `PlacedDegree`, and it is reported alongside because +the two disagree in an informative way: a fabric can be resource-saturated +while most occurrences go unplaced, and that is a routing problem rather than a +capacity one. + +## Test Protocol + +The test-set container, its determinism, its execution and aggregation, and +what a test score is not are owned by +[Test Protocol](spec-ml-core-training.md#test-protocol). This environment +supplies its case payload and its isolation rule. + +```text +DseTestCase { + instance: EpisodeStartOverride + case_seed: u64 +} +``` + +The instance is the environment's own override payload rather than a re-spelled +copy of its fields, so a field added there reaches a test case without an edit +here. + +### Isolation + +The disjointness the core requires is compared here by exact Artifact root +identity: adoption rejects a run whose test-set seed roots intersect the +corpus's architectures, or whose test-set Canonical Dataflow roots intersect +the corpus's Canonical Dataflow roots. + +The workload comparison is on the Dataflow root alone, not on the +`WorkloadBinding` pair. A binding is a Dataflow root together with a +TechMapping, and the same program bound to a different TechMapping is a +different pair but the same program; comparing pairs would let a training +workload reappear in the test set under a regenerated TechMapping and pass as +disjoint, which is precisely the leakage this rule exists to prevent. The +TechMapping is a derived artifact of the architecture it was built against, so +it carries no independent identity worth isolating. + +Root identity is the right granularity for the rest because it is what the +environment consumes. A test workload generated from the same synthetic +parameters as a training workload but with a different seed is a different +program and a different root, and is admissible; the same program under a +different file name is the same root, and is not. Checking names instead of +roots would admit exactly the duplicate the rule exists to exclude. + +A test set may also draw its cases from the real corpora — LoomBench cases or +portfolio operators — and this is the more informative choice, because the +synthetic corpus is calibrated to resemble them and a test on real programs +measures whether that calibration transferred. Those corpora are owned by +[LoomBench](spec-loombench.md) and +[Real Application Portfolio](spec-application-portfolio.md); the test set +references their Artifacts and defines no membership rule of its own. + +## Training Corpus + +### Two Halves, One Pairing + +```text +ResolvedMlDseCorpusConfigView { + architectures: canonical nonempty set + workloads: canonical nonempty set + pairings: canonical nonempty set + generation: CorpusGenerationRecord + calibration: CalibrationRecord +} + +FeasibilityPairing { + architecture: ArtifactRootReference + workload: WorkloadBinding +} +``` + +The corpus is admitted as a bipartite pairing rather than as two independent +lists. A pairing asserts that the offline generator obtained a closed Mapping +for that workload on that architecture at the episode subject's closure level, +under the probe view the run will use. Adoption requires every architecture to +appear in at least one pairing and every workload to appear in at least one. + +Without that requirement a corpus can be individually valid and jointly +useless. The environment's `reset` is a precondition, not a best effort: a seed +that cannot map every selected workload is retried and, past +`start_retry_bound`, reports `RetryBudgetExhausted`. A corpus whose +architectures and workloads were generated independently produces exactly that +failure at a rate nobody predicted, and it produces it after paying cold PnR +for each attempt. Establishing the pairing offline, once, converts a per-reset +gamble into an adoption check. + +A pairing is not a promise that every episode start succeeds. The environment +selects several workloads per episode and a pairing is per workload, so a draw +can still fail on a combination. The pairing bounds that failure to +combinations rather than admitting workloads no architecture can run at all. + +### Architecture Corpus + +An architecture is an ordinary finalized `fabric.module` or `fabric.system` +Artifact, matching the episode subject. It is produced by the `fabric_template` +generator of kind 12 through the ordinary ADG Builder and finalization path, +and it is published. The environment's episode start already admits a +`fabric_template` output as a seed, so this introduces nothing. + +Kind 12 is where the architecture corpus lives precisely because the +environment excludes it as an exploration domain: a template expansion selects +an episode seed rather than advancing an episode. The same generator therefore +serves both roles without either one reaching into the other. + +The corpus declares its coverage over the axes the template generator +parameterizes — array extent, processing-element composition and operation mix, +memory capacity and count, boundary inventory width, switch topology and +connectivity, FIFO depth, and for a `System` subject the core count and +transport topology. Coverage is reported per axis bucket rather than as a +total, because a corpus of a thousand architectures that all differ in FIFO +depth teaches a policy one axis. + +The extent range is not an aesthetic choice. The encoder is a graph transformer +whose cost scales with arc count and whose inference latency +[Forward Benchmarking Obligations](spec-ml-core-model.md#forward-benchmarking-obligations) +reports against node count, and the curriculum is expected to move designs +across an order of magnitude in node count. A corpus spanning one decade of +node count is what makes that curriculum expressible; one spanning a factor of +two makes a stage sequence meaningless. + +### Synthetic Workload Corpus + +A training workload is a synthetic program. Its operations compute nothing +meaningful: it has no reference output, no oracle, and no correctness claim, +and it exists to have the structure of a program rather than the behavior of +one. + +A synthetic program is nonetheless an ordinary program. The generator emits +ordinary compiler input, and the ordinary frontend compiles it into a Canonical +Dataflow Program Artifact with its TechMapping, through exactly the path +[Source Integration](spec-compiler-part-1-source.md) and the compiler pipeline +define. There is no synthetic-only path into the environment, no hand-authored +Dataflow, and no generator-owned Artifact kind. A synthetic program the +compiler rejects is a generator defect rather than a special case to +accommodate — and the compiler rejecting it is the cheapest possible discovery +that the generator is producing something no real program could be. + +The corresponding exclusion is exact. A synthetic workload never enters +`SourceTranslationUnitInventory` or `OperatorWorkloadInventory`, never becomes +a LoomBench case, and never joins the application portfolio. Those inventories +require a reference oracle and a semantic claim, which a program of meaningless +operations cannot supply. A synthetic workload is admissible for training and +for nothing else; it may not produce Evidence, satisfy a conformance gate, or +appear in a correctness result. + +### Structural Axes And Calibration + +The corpus is meant to resemble programs found in the wild, and what it +resembles is their structure, not their computation. The generator is +parameterized over a closed set of structural axes: + +```text +StructuralAxis = + OperationMix // distribution over the target's operation catalog + | GraphDepth // longest dependence chain + | GraphWidth // available parallelism at a level + | FanoutDistribution // out-degree of a produced value + | ReconvergenceRate // values consumed by more than one path + | RecurrenceDistance // loop-carried dependence distance + | LoopNestDepth + | TripCountMagnitude + | StreamCount // concurrent memory streams + | StrideRegularity // affine versus irregular access + | ReuseDistance + | PredicationRate + | DataDependentBranchRate + | VectorWidthDistribution + | LiveValuePressure // concurrent live values at a program point +``` + +Each axis has a declared target distribution, and a generated program is a draw +from their joint parameterization. The axes are the ones that change what a +Mapping and a fabric decision must contend with: depth against width sets what +parallelism a fabric can exploit, recurrence distance sets what a temporal +element must hold, stream count and stride regularity set what the memory +frontier must supply, and predication rate sets how much control the fabric +must absorb. An axis that does not change what the search decides does not +belong here, however faithfully it describes a real program. + +Calibration is a comparison, not a claim: + +```text +CalibrationRecord { + reference_corpus: canonical nonempty set + per_axis_tolerance: total table +} +``` + +The reference distribution is measured from the real corpora — the LoomBench +cases and portfolio operators the reference set names — by the same axis +extractor that measures a generated program, and the corpus is admitted when +every axis matches within its declared tolerance. The reference is derived from +the Artifacts `reference_corpus` names rather than written down, so it moves +when the real corpus moves and cannot become a stale table describing programs +nobody runs anymore. Deriving it is not the same as re-measuring it: a root is +immutable and content-addressed, so one root's axis vector may be retained +against that root indefinitely, and what each adoption recomputes is the +aggregate over whichever roots the set now names. + +A corpus that has never been compared is not calibrated, and this is worth +stating because the failure is invisible. A generator with plausible parameters +produces programs that look reasonable to a reader and can still be +systematically shallower, narrower, or more regular than anything real, and a +policy trained on them learns a design space that does not exist. The +comparison is what converts "we chose sensible parameters" into a checkable +statement. + +Matching every axis marginally is not matching the joint distribution, and this +document does not claim otherwise. Marginal agreement per axis is the admission +test because it is checkable and because it catches the large errors; a +generated corpus remains a model of real programs, and the test set drawn from +real programs is what measures whether the model transferred. + +### Offline Generation + +Corpus generation is a batch job that runs before training and produces three +things: the published architecture and workload Artifacts, the pairings, and a +prepopulated Mapping cache. + +```text +CorpusGenerationRecord { + generator_seed: u64 + prng_protocol: Sha256SeededXoshiro256StarStar_1_0 + architecture_axis_targets + workload_axis_targets + target_counts_per_bucket + probe_binding: CorpusProbeBinding +} + +CorpusProbeBinding { + episode_subject: + SpatialModule + | System { system_pnr_config_view_digest: ComponentViewDigest } + spatial_pnr_config_view_digest: ComponentViewDigest + tech_mapping_config_view_digest: ComponentViewDigest +} +``` + +The unit of work is one `(architecture, workload)` pair, which is also the unit +of the Mapping cache. The job establishes a pairing by obtaining a closed +Mapping, and that Mapping is written into the ordinary cache family +[Cache Dependencies](spec-config-ssot.md#cache-dependencies) owns, under the +key the environment's episode start already uses: the seed root, that one +workload with its TechMapping, and the probe view. No second store, no second +key, and no corpus-local mapping format. + +This is where the cost of training moves offline. The environment's own +analysis is that step 5 of `reset` dominates start cost and that per-workload +keying is what lets the cache reach a useful hit rate; running that step for +every pairing ahead of time means a training run's resets hit a warm cache from +the first episode rather than after a few dozen. It also means the pairing and +the cache entry are produced by the same work, so a corpus cannot claim a +pairing it never computed. + +The job is parallel over pairs, resumable, and content-addressed. A pair whose +Mapping is already cached and whose pairing is already recorded is skipped, so +an interrupted run continues rather than restarting, and two jobs over the same +corpus converge on the same result. Generation order is by bucket so that an +interrupted job leaves coverage spread across the axis buckets rather than +complete on the first bucket and empty on the rest. + +A pair whose Mapping does not close is recorded as a non-pairing and is not +retried indefinitely; a workload that closes on no architecture in the corpus +is reported and excluded, which is the signal that the workload generator has +drifted outside what the architecture generator produces. + +`CorpusProbeBinding` binds the cache entries to the complete set of views that +produced them, because that is what the cache key is. The environment's key +covers the seed root, the workload with its TechMapping, and *every* probe view +the episode ran under: the Spatial PnR view and the TechMapping view always, +and the System PnR view its subject arm carries for a `System` episode. A +record naming only the Spatial view could not distinguish two corpora generated +under different System contracts, so a `System` run would take a hit on an +entry a miss would not have produced — which is exactly the case per-view +keying exists to prevent. + +Adoption compares the complete binding against every bound environment view's +`probe_policy` and subject arm, and reports a mismatch rather than letting a +run silently pay cold PnR for every reset while a full cache sits unused. + +### Corpus Identity + +The inventory this run carries is the corpus, and its identity is the exact +`ArtifactRootReference`, on the terms +[Inventory Identity](spec-ml-core-training.md#inventory-identity) sets. An +enumerated corpus is therefore a value: extending it produces a new digest, and +every run that used the old one remains exactly reproducible. + +What discharges the regeneration half here is that the same `generator_seed`, +axis targets, and compiler configuration produce byte-identical Artifacts and +therefore identical roots, so a corpus can be reconstructed rather than +archived. The seeded PRNG protocol and its prohibition on host entropy are the +PnR owner's; this generator adds only its own domain separator and stream +purposes. + +## Checkpoints, Reproduction, And Harnesses + +Checkpoints and run identity, the two reproduction claims and the one this +document declines, and the obligations every harness satisfies are owned by +[ML Training Core](spec-ml-core-training.md). This run ships: + +```text +loom-dse-corpus generate, extend, verify, and calibrate a corpus view +loom-dse-train run training against a run view +loom-dse-test run a test set against a checkpoint +``` + +`loom-dse-corpus` verifies an existing corpus without regenerating it: every +named root resolves, every pairing's Mapping is cached under the recorded probe +binding, coverage per bucket meets the declared targets, and every calibration +axis is within tolerance. + +## Conformance Anchors + +Stable tests cover the run view rejecting a corpus that omits a bound training +environment view's seed root or pool workload, a test-set environment view +whose inventory intersects the corpus by root identity, a test case naming +inventory outside its own environment view, a test-set environment view +differing from the training views in any field +[Curriculum-Neutral Fields](spec-ml-dse-environment.md#curriculum-neutral-fields) +declares shape-fixing, including a reward code inside `EpisodePolicy`, a model +view +whose embedding table sizes do not match the reachable catalog cardinalities, +and a `value_bound` below the return range computed from the declared +quantization bounds, `step_bound`, `rejection_reward_code`, and +`stop_reward_code`; a run whose test set is admissible being adopted at all, so +that the corpus, isolation, and override rules are jointly satisfiable rather +than only individually stated; a `value_bound` sized for one rejection being +rejected against an episode of `step_bound` rejections; a hyperparameter edit +changing the training digest and leaving every cached Mapping valid; a stage +advance differing only in fields +[Curriculum-Neutral Fields](spec-ml-dse-environment.md#curriculum-neutral-fields) +declares neutral being accepted and one differing in `enabled_domains`, +`selected_objective_closure`, or `enumeration_bound` being rejected; a stage +advance retaining the Mapping cache entries earlier stages produced; +`improvement` being positive for a `Minimize` and for a `Maximize` dimension +that each improved; a closure dimension outside the +search-energy level appearing in the statistics and contributing nothing to the +reward; a dimension absent from the closure producing an absent statistic +rather than zero; a step rejected with `ObjectiveUnavailable` producing no +numeric estimate; rejection rates being reported per `RejectionReason` and per +`ExplorationDomainKind`; action frequencies being keyed by +`(domain, decision kind)` pairs and no statistic key being built from a file +name; occupancy statistics being absent when `include_resource_states` is +clear; corpus adoption rejecting an architecture or workload with no pairing; a +corpus pairing being backed by a cached Mapping under the recorded probe +binding; a training run whose Spatial PnR, TechMapping, or System PnR view +differs from the corpus's reporting the mismatch at adoption, and a `System` +-subject run being refused a corpus whose binding carries no System PnR digest; +a synthetic workload reaching the environment only as a compiled Canonical +Dataflow Program Artifact and being refused admission to every real-program +inventory; a corpus regenerated from the same seed and axis targets producing +identical Artifact roots; a corpus resolved by enumeration rather than by +directory contents, so that adding a file changes no existing run's data; a +calibration comparison failing when a generated axis distribution exceeds its +declared tolerance against the measured reference; and an interrupted corpus +job resuming without recomputing a cached pairing. + +Tests do not pin hyperparameter values, schedule breakpoints, layer or width +choices, topology counts, corpus size, axis target distributions, tolerance +values, the contents of any particular corpus or test set, wall-time numbers, +throughput, learning curves, achieved test scores, sink formats, or diagnostic +text. diff --git a/docs/spec-ml-pnr-environment.md b/docs/spec-ml-pnr-environment.md new file mode 100644 index 000000000..e83ac5479 --- /dev/null +++ b/docs/spec-ml-pnr-environment.md @@ -0,0 +1,1211 @@ +# ML PnR Environment + +This document defines the interaction boundary through which a learned search +policy places and routes one Mapping problem. An episode holds one exact +Spatial PnR invocation — a Canonical Dataflow Program, its TechMapping, a fully +elaborated Fabric, a resolved PnR configuration, and a MappingConstraintSet — +and presents its search as a sequence of typed Place and Route Actions. The +policy binds each realization to a hardware occurrence until every realization +is placed, and then either hands the result to a distance-bounded annealing +cleanup or keeps repairing it itself. + +The environment is a selector, not a placer. Every Action it applies is a +member of the closed `SpatialMappingAction` algebra +[Place And Route](spec-pnr.md#actions-and-movetransaction) already owns, every +transition goes through that owner's `MoveTransaction`, and every score is a +projection of that invocation's own objective closure. What the environment +replaces is the deterministic proposal selector, and nothing else. + +The relationship to [ML DSE Environment](spec-ml-dse-environment.md) is +symmetric and the difference is worth stating at the top, because it inverts +one rule. There, a closed Mapping is a gate: a design that cannot run the +workload is outside the space, and a step that reaches one is rejected. Here, a +closed Mapping is the goal: residual Mapping violations are ordinary candidate +state that the objective prices, and an episode spends most of its life in +states that are not yet closed. The two documents share their observation +container, action surface, and Python boundary through +[ML Environment Core](spec-ml-core-environment.md); they share no episode rule. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [Spatial PnR](spec-pnr.md#spatial-pnr) owns the exact five-input invocation + `(D, T, F, C, K)`, its aggregate freeze entry, and its exact binding + requirements; +- [Native State](spec-pnr.md#native-state) owns `FrozenModel`, + `CandidateState`, `SearchScratch`, the factorized candidate domains, and the + freeze cache key; +- [Actions And MoveTransaction](spec-pnr.md#actions-and-movetransaction) owns + the closed `SpatialMappingAction` algebra, the dynamic domain `A(M,C,S)`, the + transition `Apply(M,C,S,a)`, the dependency closure a binding change carries, + and the sole mutation mechanism; +- [Deterministic Initialization And Action Proposal](spec-pnr.md#deterministic-initialization-and-action-proposal) + owns the canonical typed decision-key order, the canonical choice orders, the + transport-routing scopes, and the frozen-topology hop distance; +- [Annealing And Replay](spec-pnr.md#annealing-and-replay) owns the annealing + policy, `realization_move_radius`, the acceptance protocol, and the seeded + PRNG protocol; +- [Evaluation Transaction](spec-pnr.md#evaluation-transaction) owns the online + probe protocol and the ephemeral Evaluation adapter; +- [Objective Projection](spec-pnr.md#objective-projection) owns the Mapping + violation catalog `V` and measure catalog `G`, and + [Final Closure And Verification](spec-pnr.md#final-closure-and-verification) + owns what makes a candidate closed; +- [Resolved View](spec-pnr.md#resolved-view) owns `ResolvedPnrConfigView` and + `SelectedObjectiveClosure`; +- [Spatial MappingConstraintSet Contract](spec-pnr.md#spatial-mappingconstraintset-contract) + owns `K` and its projection catalog; +- [Evaluation and DSE](spec-dse-feedback.md#objectives-and-quality-gates) owns + the objective algebra, quantization, and the statement that a reward is the + signed difference of a selected search energy; +- [TechMapping Generation](spec-tech-mapping.md) owns `T` and its exact `D` and + `F` binding; +- [Fabric Identity](spec-fabric-identity.md#owner-local-reference-kind-catalog) + owns the local-reference kind catalog the observation's `EntityKind` column + carries; +- [ML Environment Core](spec-ml-core-environment.md) owns the observation + container and its combined node-and-link space, the action surface and + masking algebra, the step accounting identity, the + termination-versus-truncation rule, the reward boundary, the PRNG preimage + shape, the `loomml` package layering and its interaction contract, and the + benchmarking obligations every harness satisfies; and +- [Resolved Configuration](spec-config-ssot.md#component-views) owns + component-view framing, canonical view bytes, and `component_view_digest`, + and at [Cache Dependencies](spec-config-ssot.md#cache-dependencies) owns the + cache-family contract. + +This document owns only the episode arms, its resolved configuration view and +its curriculum-neutral partition, the frozen-model and seed-state retention +policy, the construction, repair, and cleanup protocols, the PnR action index +contract, the PnR observation catalogs, the step and failure outcome algebra, +the trajectory retention record, and its own benchmarking stages. + +## Nonsemantic Boundary + +The environment is a nonsemantic search harness, on the terms +[Nonsemantic Boundary](spec-ml-core-environment.md#nonsemantic-boundary) states +for every ML environment. It is not a placer, a router, a resource allocator, +an objective, or a Mapping authority. It owns no occupancy model, no route +cost, no distance heuristic of its own, and no second candidate representation. + +It publishes no `SpatialMapping`. An episode's final candidate is mutable +`CandidateState`, which +[Canonical Search Sequence](spec-pnr.md#canonical-search-sequence) already says +never enters a central candidate set. Nothing in an episode runs independent +final verification, assigns a Mapping identity, or emits a lineage edge. + +Three consequences follow. + +Legality is never the environment's answer. A proposed Action either produces a +committed transition or is rolled back by `MoveTransaction`, and the +environment reports which. It does not pre-screen a choice the owner would +reject, does not retry a rolled-back Action under a different anchor, and does +not repair a candidate the owner refused to change. + +Cost is never the environment's answer either. The objective the reward +differences is the one `pnr_config_view.selected_objective_closure` already +declares, evaluated by the ordinary online protocol. A policy that wants a +different trade-off binds a different closure; it does not get a reward term +this document invents. + +Infeasibility is never the environment's answer. `CandidateDomain(u)` is +derived at freeze from `F` and `K`, so a choice the environment never +enumerates is a choice the owner never offered. The environment adds mask bits, +and a mask bit is a phase rule, not a proof. + +## Episode Arms + +An episode is one of two arms, each carrying exactly the fields its own +protocol needs: + +```text +PnrEpisodeArm = + ConstructThenAnneal { // 0 + cleanup_annealing_policy: SearchPolicy.annealing + cleanup_displacement_reward_code: uint64 + } + | ConstructThenRepair { // 1 + repair_step_bound: positive uint64 + repair_step_reward_code: uint64 + } +``` + +Ordinals are stable. A new arm appends; reordering, deleting, or repurposing +one is an incompatible change to this document's schema version. + +Both arms run the same construction phase and differ only in what happens once +every realization has been placed. `ConstructThenAnneal` hands the candidate to +one bounded annealing run and ends; the agent's whole contribution is the +construction, and it is scored on the quality of what construction produced +*minus how much cleanup that construction turned out to need*. +`ConstructThenRepair` keeps the agent in control, letting it rebind whatever it +placed badly, and prices each repair it takes. + +The arms are two answers to one question — who fixes an imperfect placement — +and they are kept as arms rather than as a flag pair because the fields are not +shared. A radius and an annealing policy mean nothing to an agent-driven +repair, and a repair bound and a per-repair price mean nothing to a single +annealing transition. A record carrying all five would have a +required-or-forbidden rule per field and no way to state it except prose. + +`SearchPolicy.annealing` is the exact annealing record +[Annealing And Replay](spec-pnr.md#annealing-and-replay) owns, including its +`realization_move_radius` field, carried here as a complete value rather than +as a reference into `pnr_config_view.search_policy`. The two govern different +runs and have no reason to match: the view's policy is what an ordinary Spatial +PnR invocation of this problem would use, while the cleanup is a deliberately +short, deliberately local run. Neither is an environment-authored policy, and +adoption validates this one exactly as its owner requires. + +The distance bound the arm needs is that record's own +`realization_move_radius`, and the arm carries no second radius field beside +it. A field naming the same bound twice would be two things a future edit could +make disagree, and the one the annealing run actually reads would win silently. +What this arm requires instead is the `Bounded` arm of that field. An +unbounded cleanup makes `cleanup_displacement` a measure of the annealer rather +than of the construction, and it would leave the return range's displacement +term with no bound to read; a `ConstructThenAnneal` arm carrying `Unbounded` is +rejected at adoption. + +## Resolved PnR Environment Configuration + +The environment consumes one immutable component view with schema descriptor +bytes `loom.ml_pnr_environment.config.1.0`, following the framing, canonical +byte representation, and digest contract owned by +[Resolved Configuration](spec-config-ssot.md#component-views): + +```text +ResolvedMlPnrEnvironmentConfigView { + episode_arm: PnrEpisodeArm + problem_pool: canonical nonempty set + pnr_config_view: ResolvedPnrConfigView + episode_policy: PnrEpisodePolicy + observation_policy: PnrObservationPolicy + determinism_policy: EnvironmentDeterminismPolicy +} + +SpatialPnrProblemBinding { + dataflow: ArtifactRootReference // D + tech_mapping: ArtifactRootReference // T + fabric: ArtifactRootReference // F + constraints: ArtifactRootReference // K +} + +PnrEpisodePolicy { + anchor_selection: AgentSelected | CanonicalNext + step_bound: positive uint64 + consecutive_failure_bound: positive uint64 + failed_transition_reward_code: uint64 + incomplete_closure_reward_code: uint64 +} + +PnrObservationPolicy { + enumeration_bound: positive uint32 + include_route_edges: bool + include_resource_states: bool +} + +``` + +`SpatialPnrProblemBinding` is exactly the persistent part of the five-input +invocation. `C` is absent from it and carried once by the view, because every +episode of one configuration searches under the same resolved policy and a +per-problem `C` would let two members of one pool disagree about what the +reward means. Adoption requires each member to satisfy the exact bindings +[Spatial PnR](spec-pnr.md#spatial-pnr) states — `T.D == D.id`, `T.F == F.id`, +and `K.D/T/F == D.id/T.id/F.id` — because a binding that does not is not an +invocation at all and failing at the first `reset` that draws it would report a +configuration error as a runtime one. + +There is exactly one objective closure here, and that is deliberate. The DSE +environment carries two because its probe is an inner search whose own +objective must not be the reward; this environment *is* the search, so the +closure that steers it and the closure that scores it are necessarily the same +one. A second would have to disagree with the first about what a good Mapping +is, and the policy would learn the disagreement rather than the problem. The +reward authority is therefore `pnr_config_view.selected_objective_closure` and +the view carries no closure of its own. + +`PnrEpisodePolicy` holds what applies to the whole episode rather than to one +phase. `step_bound` counts every step an episode takes, construction and repair +alike, and `consecutive_failure_bound` and the two reward codes apply in both +phases; only `anchor_selection` is construction-specific. Splitting a +per-construction record from a per-repair one was rejected because +`ConstructThenAnneal` has no repair phase and would leave the second record +entirely inert. + +`anchor_selection` chooses what the policy decides each construction step. +`AgentSelected` enumerates every unplaced realization crossed with its legal +occurrence domain, so the policy chooses both which realization to place and +where. `CanonicalNext` fixes the anchor as the first unplaced realization in +the canonical typed decision-key order and enumerates only its choices. The +trade-off is stated rather than defaulted away: `AgentSelected` makes placement +order part of what is learned, which is the more interesting half of the +problem, and it makes the enumeration the sum of every unplaced realization's +domain rather than one realization's, so `enumeration_bound` has to be sized +for it. `CanonicalNext` is the cheaper regime and the honest baseline against +which learned ordering is measured. + +The canonical view encoder writes fields in the schema order above, under the +encoding and adoption rules +[Resolved Configuration](spec-config-ssot.md#component-views) owns. This view +adds only that an `ArtifactRootReference` and an embedded owner record use +their owner's canonical encoding. + +The PRNG preimage shape, the meaning of `effective_seed` and the copy +coordinates, and the `reset` seed override are owned by +[Determinism And Copy Coordinates](spec-ml-core-environment.md#determinism-and-copy-coordinates). +This document adds only its own domain separator, which is + +```text +ASCII("loom.ml_pnr_environment.prng.sha256_seeded_xoshiro256starstar.1.0") +``` + +and its stream purpose ordinals, which are `ProblemSelection = 0` and +`CleanupSeeding = 1`. + +`CleanupSeeding` exists because the cleanup run is an ordinary annealing run +and therefore consumes PnR's own `Calibration`, `ActionProposal`, and +`Acceptance` streams, which that owner derives from a master seed and a seed +index. The environment draws that seed index once per episode from +`CleanupSeeding` rather than reusing a counter, so the cleanup is a function of +the episode's effective seed and copy coordinates exactly as every other +episode-local draw is. Reusing `local_episode_index` directly was rejected: a +`reset` seed override would then change which problem an episode ran but not +which cleanup it got. + +### Curriculum-Neutral Fields + +[Curriculum](spec-ml-core-training.md#curriculum) requires each environment +document to partition its own view by what a field determines. Here the +shape-fixing side is `episode_arm`, `pnr_config_view`, `observation_policy`, +`determinism_policy`, and, inside `PnrEpisodePolicy`, `anchor_selection` and +the two reward codes. The arm decides which phases exist and therefore which +Action kinds are ever live; the Place and Route view carries the objective +closure the reward is a difference of; `anchor_selection` decides whether the +policy chooses anchors at all, which changes the enumeration's extent rather +than its size. + +`problem_pool` is neutral, and that it is neutral is the point of stating the +rule by what a field determines rather than by naming a record. The pool sits +outside `PnrEpisodePolicy` here while the design-space environment's +corresponding field sits inside its episode policy, so a rule phrased as one +record name would forbid growing the problem pool — the one curriculum this +environment most obviously wants. `step_bound` and `consecutive_failure_bound` +are neutral for the same reason they are there: they bound an episode's length +and price nothing. + +Growing the pool costs nothing already warm. A stage that adds problems keeps +every retained frozen model and canonical seed state for the problems it +already had, because those are keyed on one `SpatialPnrProblemBinding` rather +than on `problem_pool`: the key names a member, not the set, so extending the +set invalidates nothing in it. + +## Problem Instance And Frozen Model Reuse + +Freeze is the dominant cost in this environment and it is the reason `reset` is +worth a section of its own. `freezeSpatialPnrProblem(D, T, F, C, K)` validates, +resolves, indexes, and precomputes the complete Fabric projection; a step, by +contrast, touches only changed incidence. An implementation that froze per +episode would spend most of a training run in `reset`. + +It does not have to. The Spatial PnR cache key +[Native State](spec-pnr.md#native-state) defines already hashes exact `D.id`, +`T.id`, `F.id`, `K.id`, the fields of `C` that freeze reads, freeze and +importer semantics, the native-layout ABI, and the actual `PnrIndex` width. +Every one of those is constant across the episodes of one configuration that +draw the same `SpatialPnrProblemBinding`, so the environment retains published +`FrozenModel` values under that key and reuses them across episodes and across +the vector copies living in one process. + +The key names the freeze-relevant projection of `C` rather than +`component_view_digest(C)`, and the difference is the one that decides whether +a study is affordable. Freeze precomputes a Fabric projection; a reward closure +and a search policy are not inputs to it. Keying on the whole digest would make +a sweep over `selected_objective_closure` — the obvious study over this +environment — re-freeze every problem in the pool for a result byte-identical +to the one it discarded, which is the same trap +[Configuration Split](spec-ml-core-training.md#configuration-split) draws the +training and environment views apart to avoid. The full digest is still +compared on a hit, as the revalidation below requires; it is a check, not the +key. A `FrozenModel` is immutable and explicitly +shared across workers, so sharing it across environment copies introduces no +mutability the owner does not already permit. + +A cache hit revalidates the descriptor, canonical view bytes, digest framing, +and exact artifact inputs before reuse, exactly as the owner requires. +Retention across episodes is an ordinary cache under +[Cache Dependencies](spec-config-ssot.md#cache-dependencies): a hit and a miss +produce the same formal result, and evicting the whole cache changes cost and +nothing else. + +This is also why the cleanup radius is a search-policy field rather than a +per-episode `K`: `K` is an input to freeze, so a per-episode one would give +every episode a distinct cache key and destroy exactly the reuse this section +depends on. The field's home is +[Annealing And Replay](spec-pnr.md#annealing-and-replay). + +## Construction Phase + +Every episode begins in the construction phase and leaves it when every Compute +and Memory Realization has been placed exactly once. + +`reset` builds the candidate through `createCanonicalSpatialCandidate` +-equivalent construction: the canonical assignment the owner's attempt-zero +initializer produces, with every RouteTree left visibly unrouted. The +environment then marks every realization *unplaced* in an episode-local bitset. +That bitset is the only construction state the environment owns. + +Each construction step applies exactly one `RealizationBindingAction` whose +anchor is an unplaced realization, through the ordinary probe-and-commit path. +On commit the anchor becomes placed. Routing of that realization's dependencies +is not something this document adds: a binding change already invalidates old +attachments and route claims, rebuilds every incident route dependency, and +updates resource-time, buffer, tag, memory, and handshake state inside the same +transaction, which is precisely "route the dependencies of the node just +placed". The environment issues no routing Action of its own during +construction. + +A realization becomes placed whether or not its incident nets closed. A net +whose other endpoint is still unplaced routes against that endpoint's canonical +seed binding and is re-closed when that endpoint is later bound, and a net that +cannot close at all leaves an `UnroutedObligation` violation in the candidate. +Both are ordinary candidate state that the objective prices and the reward +reports, which is what makes construction a dense-reward problem rather than +one that pays only at the end. + +An unplaced realization whose dynamic Action domain offers no alternative to +its current binding becomes placed without a step. The owner's domain contains +only anchors with at least one legal alternative, so a realization whose +`CandidateDomain` is the singleton its canonical assignment already selected +contributes no entry and the agent has nothing to choose; treating it as +unplaced would let construction stall with realizations outstanding and no live +entry to advance on, which the action surface has no defined behavior for. +Marking it placed is exact rather than a concession: the choice the agent would +have made is the only choice there is, and the candidate already holds it. + +Settling is therefore re-evaluated in both directions after `reset` and after +every commit, because a commit can shrink another anchor's domain to a +singleton and can equally restore an alternative to one already settled. A +settled realization whose domain regains an alternative returns to unplaced and +re-enters the enumeration. One-way settling would leave the exactness claim +false: a realization settled while its only choice was forced would stay +settled after the choice came back, and the agent would never be offered a +decision that had become real. + +The rebind-sweep formulation has a real cost and it is stated rather than +hidden. Because `CandidateState` is always a complete assignment, an early +construction step routes nets whose far ends are still arbitrary, and a later +step redoes that work. A genuinely partial candidate with an unbound sentinel +would do strictly less work and was rejected as a semantic change to +[Native State](spec-pnr.md#native-state); why that trade went the way it did is +[Why Learned Place And Route Selects Existing Actions](rationales/ml.md#why-learned-place-and-route-selects-existing-actions). + +Construction is not an initializer replacement in the owner's sense. The +owner's initializer propagates singleton domains and hard relation consequences +to a fixed point and backtracks on contradiction; the environment does neither, +because every state it occupies is already a complete legal assignment and +there is nothing to backtrack from. What the policy replaces is the choice of +which anchor to move next and which choice to take, which is the only part of +that protocol that is a heuristic rather than a proof. + +## Cleanup Phase + +When the last realization is placed, the arm decides what happens next. The +step that places it reports its ordinary transition either way; what differs is +whether that step also ends the episode. + +### Bounded-Radius Annealing + +Under `ConstructThenAnneal`, the step that completes construction also runs one +annealing run over the same candidate and then ends the episode with +`CleanupComplete`. The agent takes no action during the cleanup; it is a single +transition from the agent's point of view, and the observation that step +returns is the post-cleanup state. + +The run is an ordinary one. It uses `cleanup_annealing_policy` unchanged, PnR's +own acceptance kernel and cooling schedule, and PnR's own streams seeded by the +index drawn from `CleanupSeeding`. The run-start occupancy its radius anchors +on is exactly the placement construction produced. + +That anchoring is the whole point of the bound this arm requires: a bounded +cleanup can only tidy locally, so what it recovers is a measure of what +construction left on the table nearby. + +`cleanup_displacement` is that measure, and it is a mechanical projection of +two candidates rather than a statistic the run reports: + +```text +cleanup_displacement = + sum over realizations of + directed frozen-topology hop distance from its + construction-final occurrence to its cleanup-final occurrence + + count of realizations whose occurrence changed +``` + +The distance is the same one +[Deterministic Initialization And Action Proposal](spec-pnr.md#deterministic-initialization-and-action-proposal) +defines and the radius itself measures over, so no second distance model +appears. The two terms answer different questions and both are needed: the hop +sum says how far the cleanup had to reach, and the moved count says how broadly +it had to intervene. A single far move and a hundred adjacent ones are not the +same failure, and a sum alone cannot tell them apart. + +`cleanup_displacement` is zero exactly when the cleanup accepted no move that +changed an occurrence, which is the case the arm exists to reward. + +### Agent Repair + +Under `ConstructThenRepair`, completing construction advances the episode into +the repair phase and the episode continues. The live enumeration opens to the +complete dynamic Action domain: the agent may rebind an already-placed +realization, issue a `WholeNet`, `SingleSink`, `RootedSubtree`, or +`WitnessRegion` routing Action, or take a resource-allocation Action. Nothing +in the domain is masked by phase any more. + +Each committed step in this phase is a repair step and is charged +`repair_step_reward_code`. The episode ends when the agent elects to stop or +when it has taken `repair_step_bound` repair steps. + +Two properties make this a different problem from the annealing arm rather than +a slower version of it. The agent decides *when* it is done, so it is scored on +recognizing a good enough Mapping and not only on producing one; and it may +take routing and resource Actions, which construction never offers, so it can +repair a congestion problem without moving anything. Neither is available to a +bounded annealing run whose whole neighborhood is realization rebinding. + +The repair phase deliberately has no radius. The agent's every move is scored, +so an unhelpful long move is already paid for by its energy delta and its step +charge, and bounding it as well would price the same mistake twice while +removing the one capability — reaching across the fabric when that is genuinely +right — that distinguishes a learned repair from a local search. + +That holds because a radius bounds an annealing run's proposal domain and not +the candidate's Action domain, per +[Annealing And Replay](spec-pnr.md#annealing-and-replay). The only radius that +governs anything in an episode is the one the arm's `cleanup_annealing_policy` +carries, and it governs the cleanup run alone. + +## Action Space + +The action surface, the `enumeration_bound` capacity, the masking algebra, and +the elective stop are owned by +[Action Surface And Masking](spec-ml-core-environment.md#action-surface-and-masking). + +The enumeration is the candidate's own dynamic Action domain, in the owner's +canonical order. For the current candidate, the environment enumerates the +realization-binding choices, then the transport-routing choices, then the +resource-allocation choices, each anchor in canonical typed decision-key order +and each anchor's choices in that anchor's canonical choice order. An action is +one ordinal in that concatenation: + +```text +EnumeratedPnrAction { + kind: RealizationBinding | TransportRouting | ResourceAllocation + anchor: ordinal in that kind's canonical anchor domain + choice: ordinal within that anchor's contiguous choice range +} + +ActionIndex = uint32 +``` + +Nothing here is a new enumeration. `A(M,C,S)` is already deterministic, already +partitioned by kind, and already grouped into contiguous per-anchor choice +ranges; this document only fixes that kinds concatenate in the order shown and +that an `ActionIndex` is the resulting ordinal. Two runs with equal candidate, +frozen model, and resolved configuration therefore produce byte-identical +enumerations in the same order, and that follows from the owner's determinism +rather than from a rule here. + +Phase decides which of those entries are live, and this is the only place the +environment clears a mask bit: + +- in the construction phase, every entry whose kind is not `RealizationBinding` + is cleared, and every `RealizationBinding` entry whose anchor is already + placed is cleared; +- under `anchor_selection` of `CanonicalNext`, every `RealizationBinding` entry + whose anchor is not the first unplaced realization in canonical order is also + cleared; +- in the repair phase, nothing is cleared; and +- the stop outcome is cleared for the whole construction phase, and under + `ConstructThenAnneal` for the whole episode. + +Stop is cleared during construction because an episode that stops before every +realization is placed has produced nothing a Mapping could be made from, and an +always-available exit from a phase whose early rewards are negative is an exit +a policy learns to take. Under `ConstructThenAnneal` the agent never reaches a +phase where stopping is meaningful at all, so the outcome is cleared for the +whole episode; the episode ends when construction does. + +A cleared mask bit is a phase rule and never a legality claim. Every entry the +mask clears is a legal Action of the current candidate, and the same entry +becomes live again when the phase changes. This is the opposite of the DSE +environment's per-state rejection mask, which records that a decision was tried +and failed; nothing here is masked because it failed. + +The policy scores an action from the two nodes it names — the anchor it acts on +and the choice it selects — both of which are ordinary nodes of the observation +graph, named by the entry's own columns. That is what keeps an action a single +ordinal even though it is a pair. + +## Observation + +### The PnR Node And Link Space + +The combined node-and-link space, its `GraphNodeRole` catalog, the +connection-as-node rule, the two enumeration encodings, and the target-closure +rule are owned by +[Combined Node And Link Space](spec-ml-core-environment.md#combined-node-and-link-space) +and [Enumeration Encoding](spec-ml-core-environment.md#enumeration-encoding). +This section states only what that space contains for a PnR episode. + +Four role blocks are present in every episode, and this is the environment that +needs all four at once: + +- `FabricOccurrence` spans every occurrence in the frozen model's Fabric + projection, in canonical owner order — the hardware graph; +- `FabricConnection` spans every physical traversal between them, each + replacing the direct arc it stands for — the routing resources, which must be + nodes here rather than arcs because a routing Action names them and a route's + occupancy is a per-traversal fact; +- `DataflowOperation` spans the Realizations of `T` — the software graph's + nodes, which are what construction places; and +- `DataflowValue` spans the residual logical nets — the dependencies that + routing closes. + +The `Decision` block is absent, because this environment uses the +`DecisionColumns` encoding. + +Placement arcs are unconditional here, unlike in the DSE environment where they +are a configuration option. The current mapping is not extra context in this +environment; it is the state. Each placed Realization node carries one +`Placement` arc to the occurrence bound to it, so "where is this node placed" +is one hop for the encoder rather than a numeric column it has to learn to +dereference. + +When `include_route_edges` is set, each logical-net node additionally carries +one `Route` arc to every `FabricConnection` node its current RouteTree +traverses. This is the expensive part of the observation and it is optional for +that reason: a fully routed candidate has far more route incidences than +placements, and a policy that only places may not need them. A policy that +takes routing Actions in the repair phase does. + +Every live action names exactly two nodes — its anchor and its choice — and it +names them by column rather than by arc. Both resolve under the target-closure +rule, which for this environment means every anchor and every choice of every +live entry is addressable: a realization binding names a Realization node and +an occurrence node, a routing Action names a logical-net node and, for a scoped +variant, the endpoint or traversal node its scope anchors on, and a +resource-allocation Action names its demand and its selected endpoint. + +A PnR action names two nodes and never more, so its arity is fixed and the core +selects `DecisionColumns` for it. The overhead the core's rule warns about is +not marginal here: under `AgentSelected` the enumeration is the sum over +unplaced realizations of each one's legal occurrence domain, so an ordinary +problem enumerates far more actions than the design has entities. + +What the two encodings share is the part that matters to a policy, and it is +the part this environment depends on: an action is scored from the embeddings +of the nodes it names. A DSE decision's value is a prototype ordinal or a +bounded delta, so a value term can read it from a column directly. A PnR +action's value is a graph node — an occurrence, an endpoint, a traversal — so +the column holds that node's ordinal and the value term reads its embedding. A +policy head that scored a PnR action from its anchor alone would make every +occurrence on one realization indistinguishable, which is the entire decision. + +### Column Catalogs + +The `Observation` and `GraphInstance` container shapes, the no-padding rule, +the negative-one absent sentinel, the buffer lifetime, and the obligations +every column catalog satisfies are owned by +[The Graph Instance](spec-ml-core-environment.md#the-graph-instance). The +closed column catalogs this environment owns are: + +```text +PnrNodeFeatureColumn = + Role // 0 + | EntityKind // 1 + | CapabilityCount // 2 + | CapacityMagnitude // 3 + | CapacityUsage // 4 + | CapacityOveruse // 5 + | BufferDepth // 6 + | InDegree // 7 + | OutDegree // 8 + | SelfCycle // 9 + | Placed // 10 + | UnroutedObligationCount // 11 + | RouteClaimCount // 12 + | TagUnassignedCount // 13 + | TagConflictCount // 14 + +PnrArcRole = + Structural // 0 + | Placement // 1 + | Route // 2 + +PnrDecisionColumn = + ActionKind // 0 + | AnchorNode // 1 + | ChoiceNode // 2 + | ChoiceDistance // 3 + +PnrScalarFeatureColumn = + StepOrdinal // 0 + | StepBound // 1 + | Phase // 2 + | PlacedCount // 3 + | RealizationCount // 4 + | UnmaskedActionCount // 5 + | RepairStepCount // 6 + | RepairStepBound // 7 + | ConsecutiveFailures // 8 + +EpisodePhase = + Construction // 0 + | Repair // 1 +``` + +`EntityKind` is the owner-local reference kind ordinal from +[Fabric Identity](spec-fabric-identity.md) for a Fabric node and the Dataflow +or TechMapping owner's kind ordinal for a Realization or net node, so a new +Fabric occurrence kind extends the observation without a new column and without +a new role. + +`CapacityMagnitude`, `CapacityUsage`, and `CapacityOveruse` are the raw +declared capacity, the raw current usage, and the raw current overuse of the +referenced entity, projected from the candidate's own occupancy caches. Overuse +is carried separately rather than left to be derived from the other two because +it is the quantity the objective prices and a policy should read it directly +rather than reconstruct it. The typed atoms of the `FabricResourceStateRef` +catalog the core appends under `include_resource_states` are owned by +[Fabric Resource Contract](spec-fabric-resource-contract.md). + +`Placed` carries the construction bitset on a Realization node. Which +occurrence a Realization is bound to, and which Realization occupies an +occurrence, are not columns: the `Placement` arc carries that relation in both +directions, so a policy reads it in one message-passing hop rather than by +dereferencing an ordinal, which is a thing an embedding space cannot do. + +`UnroutedObligationCount`, `RouteClaimCount`, `TagUnassignedCount`, and +`TagConflictCount` are per-node projections of four of the five Mapping +violation magnitudes and of the traversal claim, attributed to the entity that +carries them. The fifth violation, hard progress, is a whole-candidate fact +with no per-node attribution and appears only through `objective_codes`. + +`PnrDecisionColumn` is the row catalog of `decisions`, one row per live entry +in the enumeration order Action Space defines, so a row's ordinal is its +`ActionIndex`. + +`AnchorNode` and `ChoiceNode` hold node ordinals into the same observation's +`graph`, under the core's rule for a target-naming column. They are the entry's +complete reference set, and the target-closure rule binds them exactly as it +binds a target arc. + +`ActionKind` carries the entry's `SpatialMappingAction` kind. + +`ChoiceDistance` is the directed frozen-topology hop distance from the anchor's +current occurrence to the choice's occurrence for a realization binding, and +the absent sentinel otherwise. It is the one derived column in any of this +environment's catalogs, and it is derived because distance is the single most +load-bearing quantity in placement while being the one a graph encoder is worst +at recovering: reading it off the embeddings would mean propagating information +across as many message-passing layers as the fabric has hops, so a policy on a +large fabric could not represent it at all at any practical depth. Its two +endpoints are Fabric occurrences and the metric is the frozen topology's, so +the value for one occurrence pair is constant for the whole problem and an +entry's distance changes only when its anchor rebinds. + +The selected closure whose codes `objective_codes` carries is +`pnr_config_view.selected_objective_closure`. + +A failed step neither advances nor resets, so under the core's buffer-lifetime +rule it does not invalidate the buffers; the candidate was rolled back to the +value it already had, and only the step and failure scalars change in place. + +## Episode Start + +An episode is created from one problem binding. There is no workload set and no +seed design: the problem is fixed for the whole episode, and what varies is the +candidate. + +`reset` performs this ordered protocol: + +1. derive the episode's PRNG streams from the effective seed and the copy + coordinates; +2. select one `SpatialPnrProblemBinding` from `problem_pool` through + `ProblemSelection`, or adopt the one the start override supplies; +3. under `ConstructThenAnneal`, adopt the effective seed as the cleanup seed + index when a start override is present, and otherwise draw it from + `CleanupSeeding`, in either case on every episode of that arm whether or not + construction ever completes; +4. acquire the `FrozenModel` and the canonical seed state for that binding and + `pnr_config_view` from the retained cache, or publish them by freezing and + building; +5. adopt the canonical candidate, its objective vector, its search energy, and + its settled bitset, and mark every realization unplaced; +6. adopt the retained canonical enumeration and its refusal verdict; and +7. build the first observation. + +Step 3 precedes step 4 so that the stream position after `reset` does not +depend on whether a cache hit occurred, which is what keeps a run with a warm +cache and a run with a cold one formally identical. + +The canonical seed state is retained beside the `FrozenModel` and under the +same key, because it is constant under the same key. The canonical candidate is +the owner's attempt-zero initializer output, and its objective vector, search +energy, settled bitset, first enumeration, and that enumeration's empty-or- +over-capacity verdict are pure functions of it — none reads the episode's seed, +its streams, or anything an episode has yet done. The enumeration is retained +with the rest, which matters most of all: under `AgentSelected` it spans every +unplaced realization's occurrence domain and is the largest of the episode. +Every episode drawing one problem would otherwise rebuild and re-evaluate an +identical value, once per episode for the whole run. Steps 5 and 6 are +therefore copies, and the energy is not recomputed at all but carried. This is +retention on the same terms as the frozen model: a hit and a miss produce the +same formal result. + +Step 6 precedes step 7 because refusing is cheaper than projecting a graph the +episode will not use, and a configuration that could never offer a legal action +should be rejected without paying for an observation. + +An override adopts the effective seed rather than drawing, because +`CleanupSeeding` is derived from a preimage that includes the copy coordinates +and an overridden episode must not depend on them. Drawing there would make a +`ConstructThenAnneal` case anneal differently on each runner, which is +precisely the reproducibility the override exists to provide and which the test +protocol depends on. The effective seed is the caller's own input, so a case is +determined by its problem and its seed alone. + +The episode start override this environment defines is one +`SpatialPnrProblemBinding`, carried by the Gymnasium `options` argument. It is +the payload itself rather than a record wrapping it, since a record with one +field is a name for that field. + +The core's override rules bind it: it must appear in `problem_pool`, and when +present it replaces step 2 entirely, so `ProblemSelection` is not consulted. + +```text +PnrEpisodeStartOutcome = + Started + | ProblemProvenInfeasible { problem, diagnostic } + | FreezeCapacityExceeded { problem, required_index_width } + | EnumerationBoundExceeded { problem, bound, required } + | Invalid { violated precondition } +``` + +`ProblemProvenInfeasible` reports that freeze proved an empty well-formed +domain, which is a sound proof about the problem and not about this episode; it +is reported rather than retried, because every retry would draw from the same +pool and a pool member that is infeasible is infeasible on every draw. +`FreezeCapacityExceeded` is separate because it is a Loom build-capacity error +rather than Mapping infeasibility, exactly as +[Native State](spec-pnr.md#native-state) requires, and it names the required +`LOOM_PNR_INDEX_BITS` width so the remedy is stated rather than guessed. + +`EnumerationBoundExceeded` reports a canonical candidate with more admissible +Actions than the action space can index, naming both the capacity and the +required length so the configuration can be corrected rather than guessed at. +It is separate from `Invalid` for that reason: the core's refusal rule requires +both numbers, and folding it into a violated-precondition report would drop +them. + +`Invalid` covers a canonical candidate with no live entry, which cannot happen +for a well-formed problem with at least one movable realization and is +therefore a projection defect rather than a state to retry from. It also covers +a binding whose exact `D`/`T`/`F`/`K` coupling does not hold and a problem +outside `problem_pool` named by an override. + +There is no retry budget, and its absence is the point. A DSE episode redraws +because a seed may be unusable; a PnR problem that adoption accepted is usable +by construction, so a failure here is a configuration or owner defect and +retrying would hide it. + +## Step + +One step applies exactly one enumerated Action. The protocol is ordered, and +stages 1 through 5 may fail: + +1. validate the action against the live mask; +2. decode the index to a typed `SpatialMappingAction` against the current + enumeration; +3. probe the transition, which computes its complete dependency closure, + applies it in a shadow candidate, closes every affected route, and evaluates + the resulting `V/G` and the selected search energy; +4. rebuild the dynamic Action domain against the probed shadow candidate, and + fail when its length exceeds `enumeration_bound`; +5. commit the probe when stages 3 and 4 both succeeded, and roll it back + otherwise; +6. mark the anchor placed when stage 5 committed a construction binding, and + settle every realization the commit left with no alternative; +7. run the cleanup and end the episode when stage 6 placed the last realization + under `ConstructThenAnneal`, or advance the phase to `Repair` under + `ConstructThenRepair`; +8. build the observation over the candidate the step left behind, applying the + phase mask to that candidate's domain. + +Stage 8 names the resulting candidate rather than stage 4's, because stage 4 +builds against a shadow that a failed step discards; masking that domain after +a rollback would describe a candidate the episode does not hold. A failed step +leaves the candidate, its domain and its mask exactly as they were, so it +recomputes none of them — the buffers are still the ones the core's +buffer-lifetime rule keeps valid. + +A step that ends the episode returns an observation of the state it ended in, +carrying an empty `decisions` and an all-clear mask, which +[Action Surface And Masking](spec-ml-core-environment.md#action-surface-and-masking) +exempts from the admissible-outcome rule precisely because no action is ever +sampled from a terminal observation. That is what keeps the cleanup outside the +capacity contract: stage 7 rebinds realizations and can leave a candidate whose +domain exceeds `enumeration_bound`, and there is nothing to roll back to by +then, but the observation that reports it enumerates nothing and so has nothing +to exceed. + +Stage 3 is the whole of the owner's online protocol and this document adds +nothing to it. In particular there is no acceptance test: the annealing policy +resolves a probe against a temperature and may reject an +improving-in-expectation move, whereas the policy here has already chosen, so +every probe that the owner does not fail and stage 4 admits is committed. +Reintroducing an acceptance kernel on top of a learned proposal would put two +selectors in series and make the reward describe neither. + +Stage 4 precedes the commit because that is the only place the capacity test +can protect the candidate. The enumeration is a function of the candidate, so +it does not exist until the transition has been applied; testing it after the +commit would leave the episode holding a state the observation cannot +represent, with nothing to roll back to. Rebuilding against the probe's shadow +candidate tests the state the step would produce while the step can still be +undone, which is what lets `EnumerationBoundExceeded` be an ordinary transition +failure that leaves the candidate unchanged. + +The enumeration's length does not depend on the phase, which is why the phase +transition at stage 7 needs no capacity test of its own. The domain is the +candidate's complete dynamic Action domain under every phase; what the phase +changes is the mask, and clearing a mask bit removes no entry. Opening the +repair phase therefore raises `UnmaskedActionCount` and leaves the enumeration +exactly the length stage 4 already admitted. The two are different quantities +and are named differently for that reason: the enumeration's length is how many +entries `decisions` carries, which the mask never changes, while +`UnmaskedActionCount` is how many of them the policy may currently select. + +The outcome algebra is closed: + +```text +PnrStepResult { + transition: optional + episode_end: optional +} + +PnrStepTransition = + Advanced { energy_delta_code, energy_delta_sign } + | Failed { reason: PnrTransitionFailureReason } + +PnrTransitionFailureReason = + IntrinsicInvalid { diagnostic } + | WorkLimit { diagnostic } + | ObjectiveUnavailable { dimension } + | EnumerationBoundExceeded { bound, required } + +PnrTerminalReason = + ElectiveStop + | CleanupComplete + | RepairBoundReached + | StepBoundReached + | ConsecutiveFailureBoundReached + | EnumerationEmpty +``` + +`IntrinsicInvalid` and `WorkLimit` are exactly the two members of the owner's +transition-failure taxonomy, and keeping them apart is the same +proof-versus-budget discipline the rest of the stack uses. `IntrinsicInvalid` +means the Action cannot produce a legal candidate — a newly closed directed +handshake cycle is the canonical case — and is a fact about the Action. +`WorkLimit` means the router exhausted its configured budget for this move and +establishes nothing at all about whether the Action was good. A blended reason +would let a rising router-budget problem read as a policy that proposes illegal +moves. + +`ObjectiveUnavailable` is this environment's name for the core's +unavailable-objective outcome, and its transaction is rolled back with the +rest. + +A transition and an episode ending are separate facts because they co-occur, on +the terms +[Step Accounting And Episode Endings](spec-ml-core-environment.md#step-accounting-and-episode-endings) +states. The step that places the last realization under `ConstructThenAnneal` +both advances and ends the episode; the failure that reaches +`consecutive_failure_bound` both fails and ends it; an elective stop ends it +with no transition at all. The core's `non_advancing` class is `Failed` here. + +`ElectiveStop`, `CleanupComplete`, and `EnumerationEmpty` are terminations; +`RepairBoundReached`, `StepBoundReached`, and `ConsecutiveFailureBoundReached` +are truncations, each cutting the episode off by a configured limit while the +candidate remained ordinary. + +## Reward + +The per-step reward is the signed difference of the selected search energy +across the transition, on the terms +[Reward Contract](spec-ml-core-environment.md#reward-contract) states. The +parent energy is retained from when the parent was entered and is not +recomputed, so a step evaluates one energy. + +This is the same reward shape the DSE environment uses and it is worth naming +what it amounts to here: with energy as a potential, a per-step signed energy +difference is potential-based shaping, so the return of an episode telescopes +to the total improvement from the canonical candidate to the final one, and the +dense per-step signal changes what is learnable without changing what is +optimal. That the weighting inside it — violations against traversal claim — +comes from the resolved closure rather than from constants in this document is +what lets one environment serve a search that cares about congestion and one +that cares about latency. + +Three terminal terms are charged on top, and each answers one requirement of +its arm. + +`ConstructThenAnneal` charges the product of `cleanup_displacement_reward_code` +and `cleanup_displacement` with a negative sign on the step that ends the +episode, alongside that step's ordinary energy delta, which is the energy the +cleanup recovered. The episode's return is therefore the final Mapping's +quality minus the amount of cleanup that quality required. Both halves are +necessary: without the energy term the policy is not scored on the design at +all, and without the displacement term it learns that a mediocre construction +is free because the annealer will fix it. + +`ConstructThenRepair` charges `repair_step_reward_code` with a negative sign on +each committed repair step. The episode's return is the final Mapping's quality +minus how long repair took, which is the same trade the other arm makes with +the annealer's work in place of the agent's. + +Both arms charge `incomplete_closure_reward_code` with a negative sign on the +final step of any episode whose final candidate is not closed, meaning its five +Mapping violation magnitudes are not all zero. Those magnitudes are the ones +the probe already maintains incrementally and evaluated for this very +candidate, so the charge reads a value the step produced rather than triggering +a recomputation — which is also what keeps the rule consistent with this +environment never running independent final verification. + +That last charge is deliberately not an ending price, and the distinction +matters because the core prices a truncation at zero. What the core refuses to +charge for is the *limit*, which the policy did not choose. Closure is a +property of the candidate, which the policy did choose, and a truncated episode +that leaves an unclosed candidate has failed at the task regardless of why it +stopped. Pricing the limit and pricing the state are different charges and only +the first is forbidden. + +A `Failed` transition yields `failed_transition_reward_code` with a negative +sign, independent of the reason. It does not distinguish `IntrinsicInvalid` +from `WorkLimit`, because charging more for one would make the reward depend on +a router budget. + +`cleanup_displacement_reward_code` multiplied by a displacement over a large +design is the one product here with real magnitude, and it is checked like +every other. + +## Trajectory Retention And Replay + +An episode retains one transient record: + +```text +PnrTrajectory { + config_view_digest: ComponentViewDigest + problem: SpatialPnrProblemBinding + effective_seed: u64 + env_runner_index: uint32 + vector_index: uint32 + local_episode_index: uint64 + cleanup_seed_index: optional + advanced_actions: ordered sequence +} +``` + +`cleanup_seed_index` is present exactly under `ConstructThenAnneal`. + +Replay is deterministic re-execution. Given the same configuration digest, the +same problem binding, and the recorded action sequence, replaying against a +`FrozenModel` for that binding reproduces an identical final `CandidateState`, +an identical objective vector, and an identical energy. Nothing about the copy +coordinates or the seed is needed for that, because the recorded actions +already name every choice the streams would have made; the seed fields are +retained so that the episode itself can be regenerated rather than only its +outcome. + +Turning a replayed candidate into a published `SpatialMapping` is outside this +document. A candidate becomes a Mapping only through the canonical search +sequence's own path — final global negotiated closure, full owner +recomputation, independent verification, and finalization — and reaching that +path from a learned policy means invoking the policy as a search-policy +selector inside an ordinary Spatial PnR invocation, which is a change +[Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism) owns +and this document does not make. Until that exists, this environment produces +trained policies and measurements, not Mappings. Stating the boundary is +preferable to a replay rule that publishes from a harness, which would put a +training artifact on the product path. + +## Python Boundary + +The `loomml` package layering, the RLlib conformance target and its +obligations, the zero-copy array contract, the outcome-versus-exception rule, +and the threading and `fork` rules are owned by +[Python Boundary](spec-ml-core-environment.md#python-boundary). This +environment occupies `loomml.env.pnr` and `loomml.rllib.pnr` and adds no rule +of its own. + +One consequence is worth naming because it is this environment's dominant +implementation constraint. The native layer holds a `FrozenModel`, a +`CandidateState`, and the owner's reusable scratch, none of which crosses into +Python; what crosses is the observation buffers and one action index. A step is +therefore a native transaction plus one marshalling boundary, and the shared +`FrozenModel` is what makes several vector copies in one process affordable. +Under the core's rule that an instance is owned by one thread and shares no +mutable state, the copies share only that immutable model. + +## Benchmarking + +The harness obligations, the required breakdowns, the instrumentation rule, and +the ratio-based regression budget are owned by +[Benchmarking Harness Contract](spec-ml-core-environment.md#benchmarking-harness-contract). +This environment ships `loom-pnr-env-bench`, whose failure-reason breakdown is +keyed by `PnrTransitionFailureReason` and whose action partition is the +`SpatialMappingAction` kind, because a routing Action and a binding Action do +not cost remotely the same. + +The harness decomposes one step into these stages and reports each separately: + +```text +enumeration rebuild the dynamic Action domain and the phase mask +decode map the action index to a typed SpatialMappingAction +probe_closure the shadow transition and its incident route closure +probe_objective incremental V/G recomputation and search-energy evaluation +resolve commit or roll back the transaction +cleanup the bounded annealing run and its displacement projection +observation build the graph and the decision columns +marshal expose buffers across the Python boundary +``` + +`probe_closure` and `probe_objective` are the two halves of the single probe +the step protocol performs, split here because they are the two costs worth +telling apart and reported as one probe nowhere. Stages partition the step; +none contains another, so the stage sum is the step. + +A reset decomposes into `problem_lookup`, `freeze`, `seed_state_build`, +`enumeration`, and `first_observation`. The seed state's build and evaluation +are one stage because retention makes them one event: on a hit neither runs. +`seed_state_build` covers the canonical enumeration as well, since retention +makes the whole seed state one event: on a hit none of it runs. It is reported +separately from `freeze` because the two differ by orders of magnitude and a +blended `reset` percentile over a mixed hit-and-miss population describes +neither. + +Two further breakdowns are required beyond the core's two. + +Every measure is reported per phase the arm actually runs. A construction step +and a repair step run the same code over candidates with completely different +route densities, and a blended percentile moves whenever an episode's phase mix +moves, which it does throughout training as the policy learns to finish +construction faster. Under an arm with one phase the breakdown is that phase, +reported without a second empty column, and the core's action partition +collapses the same way: it is over the kinds the bound arm can actually make +live, which under `ConstructThenAnneal` is `RealizationBinding` alone. + +And `reset` is reported with the frozen-model cache hit rate alongside it, +separated into hit and miss populations rather than blended, because a single +`reset` percentile over a mixed population describes neither and moves with the +pool size rather than with any code change. The hit rate is the number that +decides whether the environment is affordable at all, and it is reported as a +first-class measure and not as a footnote to a latency. + +The harness reports the deterministic PnR work-unit counts — assignment +attempts, endpoint expansions, negotiation iterations — alongside its timings +and states which is which, because those are the cross-machine cost measure and +the timings are not. + +## Anchor Verification + +Stable tests cover adoption rejecting a `problem_pool` member whose `T.D`, +`T.F`, or `K.D/T/F` binding is not exact; adoption rejecting a +`ConstructThenAnneal` arm whose `cleanup_annealing_policy` carries an +`Unbounded` `realization_move_radius`, and a `ConstructThenRepair` arm whose +`repair_step_bound` is zero; the enumeration equalling the owner's dynamic +Action domain in canonical kind, anchor, and choice order, and being +byte-identical across two runs with equal candidate and configuration; the +construction mask clearing every non-`RealizationBinding` entry and every entry +whose anchor is already placed, and `CanonicalNext` additionally clearing every +anchor but the canonically first unplaced one; the repair phase clearing +nothing; the stop outcome being clear for the whole construction phase and for +the whole of a `ConstructThenAnneal` episode; a cleared entry becoming live +again when the phase changes, and no cleared entry ever being reported as +illegal; the `Decision` block being absent and `decisions` carrying one row per +enumerated entry and no arcs, in the enumeration order, with a row's +ordinal equalling its `ActionIndex`; every live entry's `AnchorNode` and +`ChoiceNode` resolving to exactly one node ordinal in the same observation's +`graph`, including an occurrence-valued, a traversal-valued, an +endpoint-valued, and a net-valued reference; `ChoiceDistance` equalling the +owner's directed frozen-topology hop distance for a realization binding and the +absent sentinel for every other kind; a `Placement` arc joining a placed +Realization node to its occurrence node in both directions and none joining an +unplaced one; route arcs being absent when `include_route_edges` is clear and +spanning exactly the current RouteTree traversals when it is set; +resource-state columns being absent when `include_resource_states` is clear; a +committed binding closing every incident net through the owner's own dependency +closure with no environment-issued routing Action; a net to an unplaced +realization remaining an ordinary candidate violation rather than a step +failure; construction advancing to the repair phase or to the cleanup exactly +when the last realization is placed and never before; a `ConstructThenAnneal` +episode running exactly one cleanup, reporting `CleanupComplete` on the same +step that placed the last realization, and returning the post-cleanup +observation; `cleanup_displacement` being zero exactly when no occurrence +changed, and counting both the hop sum and the moved count; a +`ConstructThenRepair` episode charging exactly one `repair_step_reward_code` +per committed repair step and none during construction; an unplaced realization +with no alternative to its current binding being settled without a step at +`reset` and again after a commit that shrinks its domain to a singleton, and +construction never reaching a state with realizations outstanding and no live +entry; an `IntrinsicInvalid` rollback leaving the candidate equal to its +pre-probe value in every selected decision and every rebuildable cache; +`IntrinsicInvalid` and `WorkLimit` never being collapsed into one reason; +`ObjectiveUnavailable` producing no numeric reward and rolling back its +transaction; a transition whose resulting enumeration exceeds +`enumeration_bound` failing with `EnumerationBoundExceeded` before the commit +and leaving the candidate unchanged; the enumeration length being identical +before and after the construction-to-repair phase advance while +`UnmaskedActionCount` rises while the enumeration's length does not; a masked, +stale, or out-of-range index being refused +with the candidate unchanged; the step accounting identity holding over a +complete episode with `Failed` as its non-advancing class; termination and +truncation being reported distinctly for their respective terminal reasons; a +truncation yielding zero for the ending itself while still charging +`incomplete_closure_reward_code` when its final candidate is unclosed, and +charging nothing when it is closed; the frozen model being reused across two +episodes drawing the same problem binding and not being reused under a changed +the freeze-relevant projection of `C` or a changed `K` identity, and being +reused across a changed `selected_objective_closure`; a warm-cache run and a +cold-cache run producing identical stream positions and identical trajectories; +`ProblemProvenInfeasible` and `FreezeCapacityExceeded` being reported +distinctly and neither being retried; a canonical candidate whose enumeration +exceeds `enumeration_bound` reporting `EnumerationBoundExceeded` with both the +capacity and the required length rather than `Invalid`; a `ConstructThenAnneal` +case under a start override annealing identically at two runner counts and two +copy coordinates; a settled realization whose domain regains an alternative +returning to unplaced and re-entering the enumeration; the observation returned +on a step that ends the episode carrying an empty `decisions` and an all-clear +mask, and no non-terminal state ever presenting one; a failed step leaving the +enumeration and the mask it had before the step rather than those of the +discarded shadow candidate; a `pnr_config_view` whose +`realization_move_radius` being adopted at either arm and changing no +enumeration this environment produces; an episode start override consulting no +`ProblemSelection` draw and +producing the same problem on every copy, and one naming a problem outside +`problem_pool` being rejected; two environment copies with distinct coordinates +drawing disjoint problem sequences from one seed; a `reset` seed argument +becoming the effective seed and being recorded as such; and equal +configuration, coordinates, seed, and action sequence reproducing an identical +trajectory, an identical final `CandidateState`, and an identical search +energy. + +Tests do not pin the live node, arc, or action counts of any particular +candidate, the bound or radius values a configuration selects, which occurrence +a policy chooses, achieved energies or closure rates, cleanup displacement +values, wall-time numbers, per-kind cost ratios, cache hit rates, corpus +contents, diagnostic text, or Python formatting. diff --git a/docs/spec-ml-pnr-model-architecture.md b/docs/spec-ml-pnr-model-architecture.md new file mode 100644 index 000000000..7cfca3f68 --- /dev/null +++ b/docs/spec-ml-pnr-model-architecture.md @@ -0,0 +1,368 @@ +# ML PnR Model Architecture + +This document defines the learned policy that consumes the ML PnR Environment's +observations and emits its actions. It owns the embedding of that environment's +column catalogs, the two-factor policy head that scores an action from the +nodes it names, the model's own configuration view, and the harness that +measures its forward pass. + +The model is a search policy on the terms +[ML Model Core](spec-ml-core-model.md) states. It proposes Actions; the +environment applies them through Place and Route's own `MoveTransaction`, and +every legality, closure, and objective fact remains owned where +[ML PnR Environment](spec-ml-pnr-environment.md) places it. + +Everything a Loom policy does that is not specific to what it is searching — +the batching, the embedding discipline, the graph-transformer trunk, the +context pooling, the value head, masking, the action distribution, and the +checkpoint boundary — is owned by [ML Model Core](spec-ml-core-model.md). This +document is the difference between that and a placer. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [ML Model Core](spec-ml-core-model.md) owns the module boundary, observation + batching, the feature-embedding discipline, the encoder, the graph context, + the value head, masking, the action distribution, the checkpoint boundary, + and the obligations every forward-pass harness satisfies; +- [Column Catalogs](spec-ml-pnr-environment.md#column-catalogs) owns + `PnrNodeFeatureColumn`, `PnrArcRole`, `PnrScalarFeatureColumn`, + `PnrDecisionColumn`, and `EpisodePhase`; +- [Action Space](spec-ml-pnr-environment.md#action-space) owns the enumeration, + its canonical order, `ActionIndex`, and which entries the phase mask clears; +- [The PnR Node And Link Space](spec-ml-pnr-environment.md#the-pnr-node-and-link-space) + owns the role blocks, the placement and route arcs, and the rule that an + action names its anchor and choice by column; +- [Reward](spec-ml-pnr-environment.md#reward) owns the reward the value head + regresses toward; +- [Enumeration Encoding](spec-ml-core-environment.md#enumeration-encoding) owns + the `DecisionColumns` form this model consumes; +- [Actions And MoveTransaction](spec-pnr.md#actions-and-movetransaction) owns + the Action kinds an entry's `ActionKind` column names; and +- [Resolved Configuration](spec-config-ssot.md#component-views) owns + component-view framing. + +This document owns only the PnR feature mapping, the policy head, the model +configuration view, and the forward benchmarking stages. + +## Module Boundary + +The model is `LoomPnrRLModule`, one `TorchRLModule` implementing +`ValueFunctionAPI` on the terms +[Module Boundary](spec-ml-core-model.md#module-boundary) states. It adds no +entry point and changes no signature. + +## Feature Embedding + +The core fixes how a column class is consumed; this section fixes which class +each PnR column belongs to and why, for the cases where the answer is not +mechanical. + +Embedded categorical columns: + +```text +column table size source catalog +Role GraphNodeRole cardinality environment core +EntityKind Fabric local-reference kinds plus + Dataflow and TechMapping kinds Fabric Identity, Dataflow +ActionKind SpatialMappingAction kind cardinality Place And Route +Phase EpisodePhase cardinality PnR environment +``` + +`Phase` is a graph-level categorical rather than a node column, and it is +embedded rather than projected for the ordinary reason: `Construction` and +`Repair` are two regimes, not two points on a scale, and the model's behaviour +in one should not be constrained to lie on a line through the other. + +Every `PnrNodeFeatureColumn` not named below is a projected numeric column with +the absence indicator the core requires. `CapacityMagnitude`, `CapacityUsage`, +and `CapacityOveruse` are projected with the occupancy ratio the core requires +alongside them, and the same treatment extends to each capacity and usage pair +the environment appends when `include_resource_states` is set. + +Two columns need a stated decision rather than a classification. + +`AnchorNode` and `ChoiceNode` are not features at all. They are indices, and +the head consumes them by gathering `h` at those ordinals; projecting an +ordinal would assert a distance between two unrelated nodes that happen to sort +near each other. Whether an entry's anchor is already placed is likewise not a +column: `Placed` on the anchor node is that bit, and the head already has the +anchor's embedding in hand. + +`ChoiceDistance` is the load-bearing feature of this model and gets its own +treatment. Hop counts near zero decide most placements and hop counts far from +zero are nearly interchangeable, so a single linear projection would spend its +resolution in the wrong place. Small distances are embedded from a table +indexed by exact hop count up to a configured bucket count, and distances +beyond it are projected through the core's signed logarithmic transform into a +shared remainder. The near range therefore keeps exact resolution while the far +range still generalizes across magnitudes, and the absent sentinel — every +action kind but a realization binding — takes the core's zero-plus-indicator +form rather than a distance of zero, which would otherwise read as "adjacent". + +Scalar features are graph-level, with the ratios the core requires supplied +alongside the magnitudes: `StepOrdinal` against `StepBound`, `PlacedCount` +against `RealizationCount`, and `RepairStepCount` against `RepairStepBound`. +`UnmaskedActionCount` and `ConsecutiveFailures` are supplied as magnitudes. +`PlacedCount` over `RealizationCount` is how far construction has progressed, +which is the quantity that most changes what a good action looks like, so +requiring the model to divide two magnitudes to recover it would be a +gratuitous obstacle. + +One absence is a decision rather than an oversight. The model receives +`CapabilityCount` for an occurrence but not the identity of the operations that +occurrence supports. The prior architecture embedded a per-node instruction +bitmask because its candidate mask was computed separately and the network had +to learn which unit could host which operation. Here the enumeration is derived +from `CandidateDomain`, so an occurrence that cannot host a realization is +never offered as a choice and compatibility is never a question the policy has +to answer. What remains useful is scarcity — whether an occurrence is a +specialist worth saving for something else — and a count carries that. A +capability-set feature would be paying for a distinction the mask already made. + +## Encoder + +The encoder is the core's `GraphTransformerStack`, unchanged, with +`PnrArcRole` supplying edge features. Its three roles are the three +relations a placement policy has to tell apart: `Structural` is the fabric's +own connectivity and the software graph's dependencies, `Placement` is the +current mapping, and `Route` is where the nets currently run. + +`Route` arcs are what make the trunk expensive: the optional projection +[Observation](spec-ml-pnr-environment.md#observation) governs can dominate arc +count, and therefore attention cost, when it is enabled. + +## Graph Context And Value Head + +Both are the core's, unchanged. The value head reads the pooled context and +emits one bounded scalar per instance, and its bound is checked against the +range [Online Training](spec-ml-pnr-training.md#online-training) declares +rather than chosen. + +## Policy Head + +An action names two nodes and carries its own kind and distance, so the head +scores it from those directly. Nothing is reconstructed from an index and no +action has an embedding of its own. + +```text +anchor(a) = ( AnchorNode of a, ActionKind of a ) + +s_anchor(a) = MlpAnchor([ h[AnchorNode(a)], e_kind(a), context ]) +s_choice(a) = MlpChoice([ h[AnchorNode(a)], h[ChoiceNode(a)], + e_distance(a), e_kind(a), context ]) +logit(a) = s_anchor(a) + s_choice(a) +``` + +`h` is the shared trunk's node embedding, indexed by the ordinal the entry's +column holds. This is the whole of "act on the predicted node": the score of +placing a realization on an occurrence is a function of that realization's +embedding and that occurrence's embedding, both produced by one pass over the +state. + +`s_choice` is this model's selection term, so actions sharing an anchor form +one group and compose on the terms +[Policy Head Factorization](spec-ml-core-model.md#policy-head-factorization) +states. + +The hierarchy is what this environment is shaped for. During construction every +entry of one unplaced realization is one group, so the anchor factor is "which +realization to place next" and the choice factor is "where" — exactly the +decomposition the search actually makes. Under `CanonicalNext` anchor selection +the mask leaves one group live and the anchor factor degenerates to a constant, +which is the correct behaviour rather than a special case: the environment +already made that choice, so the policy has nothing to contribute to it. + +### Conditioning The Choice On The Anchor + +`s_choice` reads the anchor's embedding as well as the choice's, and that is a +deliberate departure from the architecture this model translates. The prior +`SwapSchedulerModel` scored hardware slots from the slot embedding and the +graph context alone, conditioning on the selected software node only through +the mask. It could therefore learn which slots are good in general — central, +uncongested, well connected — but not which slot suits *this* node, which is +most of the placement problem: a producer wants to be near its consumers, and +which occurrence satisfies that is a property of the pair. + +The cost is that `s_choice` is evaluated once per live action rather than once +per node, and this document is precise about which part of that cost is +recoverable. + +The prior architecture's `_SplitScorer` factoring is admitted and extended. A +scorer of the form `out(activation(W_1 x_1 + ... + W_k x_k))` is exactly the +scorer `out(activation(W · concat(x_1, ..., x_k)))` with `W` partitioned, so +`MlpAnchor` and `MlpChoice` may be evaluated in the factored form. What that +buys is memory: the concatenation is never materialized and never saved for the +backward pass, which matters at live-action counts in the hundreds of +thousands. What it does not buy is arithmetic, because the activation and the +output projection are still evaluated per action. A specification that claimed +otherwise would be promising a saving the algebra does not contain. + +A genuine per-action reduction requires a scorer whose pair term is bilinear — +an inner product between one projection of the anchor and one of the choice, so +each side is projected once per node and the pair costs a dot product. That +computes a different function, with strictly less expressive pair interaction +than an activation over the sum. It is permitted as a configured variant and is +validated against the reference form on the same inputs and parameters like any +other optimization; it is not presented as an equivalent implementation of the +same head. + +### Stop + +The environment masks the core's stop logit for the whole construction phase +and for the whole of a `ConstructThenAnneal` episode, and the head applies that +mask like any other: it never derives its own stop admissibility from the phase +feature, because the mask is the authority and two sources for one fact +eventually disagree. + +The joint distribution over `enumeration_bound + 1` outcomes and its stop +encoding are the core's. + +## Masking + +Masking is the core's contract. Two properties of this environment are worth +stating because they make it easier here than in the design-space environment. + +The environment's mask is a phase rule rather than a rejection record. Every +entry it clears is a legal Action of the current candidate, cleared because the +phase does not admit it, and the same entry becomes live again when the phase +changes. Nothing in a PnR episode is masked because it was tried and failed, so +no mask bit depends on episode history. + +Anchor-group consistency is automatic. Construction clears every entry of an +already-placed realization, which is precisely a whole anchor group, and clears +whole kinds rather than individual choices. The core's rule that a group is +masked exactly when all its members are masked therefore holds without the head +arranging for it, and a partially masked group only arises from an advisory +mask. + +## Model Configuration + +The model's shape is one immutable component view with schema descriptor bytes +`loom.ml_pnr_model.config.1.0`, following the framing owned by +[Component Views](spec-config-ssot.md#component-views) and the three properties +[Model Configuration Framing](spec-ml-core-model.md#model-configuration-framing) +fixes: + +```text +ResolvedMlPnrModelConfigView { + embedding_widths: total table + distance_bucket_count: positive uint32 + encoder: GraphTransformerStack + head_widths: PnrHeadWidths + choice_scorer: Factored | Bilinear + value_bound: positive uint64 + advisory_mask_set: canonical set + initialization: InitializationConstants +} + +PnrHeadWidths { + anchor_hidden: positive uint32 + choice_hidden: positive uint32 + value_hidden: positive uint32 +} +``` + +`distance_bucket_count` is the exact-hop table height Feature Embedding +describes; it sizes a parameter tensor, which is why it belongs in this view +and not in a training one. `choice_scorer` selects between the two forms +Conditioning The Choice On The Anchor defines, and it is a view field rather +than an implementation switch because `Bilinear` changes the function and +therefore the parameter shapes, so a checkpoint is correctly incompatible +across it. + +## Forward Benchmarking + +This model ships `loom-pnr-model-bench`, on the obligations +[Forward Benchmarking Obligations](spec-ml-core-model.md#forward-benchmarking-obligations) +states. + +The harness decomposes one forward pass into these stages: + +```text +gather assemble the batched graph and decision columns, move to device +embed embedding lookups, numeric projection, input projection +encode the graph-transformer stack +pool per-instance multi-scale context +anchor anchor scoring, once per anchor group +choice choice scoring, once per live action +mask mask composition and additive application +distribute distribution construction and sampling +value the value head, when the entry point computes it +``` + +`anchor` and `choice` are separate stages, where a model whose value is a +catalog ordinal can report one policy stage. They scale with different +quantities — anchor-group count and live-action count — and the whole argument +of Conditioning The Choice On The Anchor is that the second is the price of the +first being worth having. A blended policy stage would make it impossible to +say whether that price is what it was expected to be, or whether the factored +form is doing anything. + +`gather` is reported with the decision columns separated from the graph, since +their row count is the live-action count and the graph's is the state size, and +those move independently: a repair-phase state has a stable graph and a much +larger enumeration than a late-construction one. + +Beyond the core's required measures, three breakdowns are required here. + +Every measure is reported per phase, on the terms +[Benchmarking](spec-ml-pnr-environment.md#benchmarking) states for the +environment harness. What differs here is that a construction and a repair +forward pass differ in live-action count as well as in route density, so the +scaling below separates on both. + +Scaling is reported against node count, arc count, live-action count, +anchor-group count, and the mean anchor-group size. The last is what makes the +anchor-group saving +[Policy Head Factorization](spec-ml-core-model.md#policy-head-factorization) +describes legible. + +The `choice_scorer` variants are measured against each other and against the +reference form. `Bilinear` is admitted for speed and computes a different +function, so its measurement reports both its stage cost and its divergence +from `Factored` on the same inputs, rather than reporting a latency alone and +leaving the quality question to a separate exercise. + +The rollout split against the environment step is reported first, as the core +requires, and it matters more here than for a design-space policy: a PnR +environment step contains a Mapping probe with incident route closure, which is +plausibly an order of magnitude more expensive than this forward pass. A model +optimization that halves an already-small share buys nothing, and the split is +what says so before the work is done rather than after. + +## Conformance Anchors + +Stable tests cover a `Placement` arc reaching the encoder as an edge feature +and no node column restating that relation; `ChoiceDistance` at a hop count +inside the bucket range reaching an exact table row and one beyond it reaching +the projected remainder, and the absent sentinel producing the core's +zero-plus-indicator rather than the encoding of distance zero; two actions +sharing an anchor node and kind receiving an identical `s_anchor`; two choices +on one anchor receiving different scores; an action whose anchor and choice +embeddings are swapped receiving a different score, so the pair term is not +symmetric; the factored scorer reproducing the concatenated reference exactly; +the `Bilinear` variant being reported with its divergence from `Factored` +rather than as an equivalent; scoring restricted to live actions producing +logits identical to scoring every slot; a `CanonicalNext` configuration leaving +exactly one live anchor group and its anchor factor contributing no gradient to +the choice among realizations; a masked anchor group receiving no probability +and contributing no entropy; construction clearing whole anchor groups so that +no partially masked group arises without an advisory mask; the stop logit being +masked throughout construction and throughout a `ConstructThenAnneal` episode, +and the head deriving stop admissibility from the mask rather than from the +phase feature; the two-level and flat formulations producing identical +distributions to numerical tolerance; `_forward_train` and +`_forward_exploration` producing identical logits for equal parameters, +observations, and masks; `_forward_inference` computing no value; +`compute_values` producing identical predictions with and without precomputed +embeddings; a decision-column node ordinal being offset into the batched node +union before use, so an action never scores another instance's nodes; and a +checkpoint load failing across a changed `distance_bucket_count` or a changed +`choice_scorer`. + +Tests do not pin layer counts, widths, head counts, the bucket count, the +scorer variant, initialization constants, the advisory mask set, learned +parameter values, wall-time numbers, per-phase cost ratios, device placement, +precision configurations, or tolerance values. diff --git a/docs/spec-ml-pnr-training.md b/docs/spec-ml-pnr-training.md new file mode 100644 index 000000000..d1723ba54 --- /dev/null +++ b/docs/spec-ml-pnr-training.md @@ -0,0 +1,527 @@ +# ML PnR Training + +This document defines how the place-and-route policy of +[ML PnR Model Architecture](spec-ml-pnr-model-architecture.md) is fitted +against the environment of [ML PnR Environment](spec-ml-pnr-environment.md). It +owns the demonstration corpus that pretraining imitates, the two-stage run that +turns those demonstrations into an online policy, this environment's +return-range arithmetic and statistics, and the test protocol that says whether +the run improved anything. + +A run has two stages and they answer different questions. The first learns +where a good placement puts things, from demonstrations a simulated annealer +produced. The second learns to do better than that, from its own experience. +The whole point of their being one run rather than two is that the second +begins from the first's parameters. + +## Ownership + +Every fact this document depends on resolves to one exact owner: + +- [ML Training Core](spec-ml-core-training.md) owns everything a Loom training + run does that is not specific to what is being searched: the nonsemantic + boundary, the configuration split and inventory identity, the algorithm + binding surface, the stage sequence and its handoff, exact-rational + hyperparameters, schedules, the reward adapter, topology, the stage + invariance rule, the statistics production rule, the test protocol, + checkpoints and run identity, reproduction, and the harness surface; +- [ML PnR Environment](spec-ml-pnr-environment.md) owns the episode arms, the + action space and its phase mask, the observation, the reward and every code + it charges, `PnrTerminalReason`, `PnrTransitionFailureReason`, + its episode start override payload, `PnrTrajectory`, and + `ResolvedMlPnrEnvironmentConfigView`; +- [Model Configuration](spec-ml-pnr-model-architecture.md#model-configuration) + owns `ResolvedMlPnrModelConfigView` and the tensors it sizes; +- [Annealing And Replay](spec-pnr.md#annealing-and-replay) owns the annealing + policy, its acceptance kernel, and the seeded PRNG protocol the demonstration + generator draws from; +- [Deterministic Initialization And Action Proposal](spec-pnr.md#deterministic-initialization-and-action-proposal) + owns the Action proposal selector and the frozen-topology hop distance + `PlacementLocality` orders by; +- [Objectives and Quality Gates](spec-dse-feedback.md#objectives-and-quality-gates) + owns `ExactAffineQuantization` and the search energy the return range is + derived from; and +- [Component Views](spec-config-ssot.md#component-views) owns view framing, + canonical bytes, and `component_view_digest`. + +This document owns only the demonstration corpus and its generator, the PnR +stage configuration, this environment's return-range arithmetic, its statistics +catalog and breakdown axes, its test set and comparison rules, and its +harnesses. + +## Configuration Split + +This run's views, under the framing +[Configuration Split](spec-ml-core-training.md#configuration-split) owns: + +```text +loom.ml_pnr_environment.config.1.0 ResolvedMlPnrEnvironmentConfigView +loom.ml_pnr_model.config.1.0 ResolvedMlPnrModelConfigView +loom.ml_pnr_training.config.1.0 ResolvedMlPnrTrainingConfigView +loom.ml_pnr_demonstrations.config.1.0 ResolvedMlPnrDemonstrationConfigView +loom.ml_pnr_test_set.config.1.0 ResolvedMlPnrTestSetConfigView +loom.ml_pnr_run.config.1.0 ResolvedMlPnrRunConfigView +``` + +```text +ResolvedMlPnrRunConfigView { + training_config_digest: ComponentViewDigest + model_config_digest: ComponentViewDigest + demonstration_config_digest: ComponentViewDigest + test_set_config_digest: ComponentViewDigest +} +``` + +`ResolvedMlPnrTrainingConfigView` is the core's `ResolvedTrainingConfigView` +under this descriptor, with no field added. + +There is no corpus view, and its absence is the substantive difference from the +design-space run. That run generates its training instances, so it needs an +architecture corpus, a synthetic workload corpus, structural axes, and a +calibration record to argue that what it generated resembles what it will meet. +A PnR run does not generate anything: its instances are exact +`SpatialPnrProblemBinding` values naming Artifacts that already exist, and a +problem is a problem whether or not anyone calibrated it. What the +demonstration view carries is not a corpus of instances but a corpus of +*placements* for instances the configuration already names. + +Adoption validates each view, then three cross-view conditions: every problem +the demonstration view names appears in the training stages' bound environment +views; the test set's problems appear in its own environment view and in no +training stage's; and the model view's `value_bound` covers the return range +below, maximized over every bound stage. + +## Demonstration Corpus + +Pretraining imitates simulated annealing. What it imitates is where the +annealer *arrived*, not how it got there, and that distinction is the whole +design of this section. + +### What Is Imitated + +An annealing run is a Metropolis walk. Its intermediate states are bad by +construction — that is what the acceptance kernel is for, and a run that never +occupied a worse state was not annealing. A demonstration recording that walk +teaches a policy to reproduce it: to place badly, then move, then move again. +The prior system recorded exactly that, a greedy fill followed by every +accepted swap, and it was imitating a search rather than a solution. + +So the generator keeps the annealer's final placement and discards its path. +The demonstration is a construction sequence that reaches that placement +directly: each realization bound once, to the occurrence the annealer left it +on, in one sweep with no move ever undone. + +### The Generator + +For each problem in the view, under streams derived from `generator_seed` by +the protocol +[Search Policy And Determinism](spec-pnr.md#search-policy-and-determinism) +owns: + +```text +1. reset the environment on that problem +2. step it with a scripted annealer, which selects each Action through the + proposal selector and acceptance kernel Place and Route already owns, + under `annealing_policy`, until that policy's schedule completes +3. read the final placement: each Compute and Memory Realization, and the + occurrence it is bound to +4. discard the action sequence entirely +5. reset again, and emit one `RealizationBindingAction` per realization, in + `demonstration_order`, binding each directly to its final occurrence +6. record the placement of step 3, the sweep's length, and the energy and + closure the sweep ended at +``` + +The scripted annealer is not a second search authority. It selects Actions with +the owner's own proposal and acceptance protocols, and every Action it takes is +an ordinary environment step, so every recorded index is by construction a +legal `ActionIndex` under the phase mask that produced it. Nothing here needs a +recording hook inside Place and Route, and nothing depends on a diagnostic +stream. + +`demonstration_order` is a real parameter rather than a detail. Under +`AgentSelected` anchor selection the policy learns which realization to place +next as well as where, so the order a demonstration uses is part of what it +teaches. `CanonicalDecisionKey` uses the owner's canonical typed decision-key +order and is the honest default. `PlacementLocality` orders each realization +after the already-placed neighbours it shares a net with, nearest first by the +frozen-topology hop distance, which is the order a person would place by hand +and the one worth measuring the default against. + +### What The Record Carries + +```text +ResolvedMlPnrDemonstrationConfigView { + problems: canonical nonempty set + generator: DemonstrationGenerationRecord + admitted: canonical nonempty set +} + +DemonstrationGenerationRecord { + generator_seed: u64 + prng_protocol: Sha256SeededXoshiro256StarStar_1_0 + annealing_policy: SearchPolicy.annealing + demonstration_order: CanonicalDecisionKey | PlacementLocality + pnr_config_view_digest: ComponentViewDigest +} + +DemonstrationRecord { + problem: SpatialPnrProblemBinding + final_placement: ordered sequence + action_count: uint64 + replayed_final_energy_code: uint64 + replayed_closed: bool +} +``` + +`final_placement` is the annealer's result and the only thing about the +annealing run this document keeps: the occurrence each Compute and Memory +Realization ended bound to, in the owner's canonical decision-key order, so its +length is the realization count and its meaning needs no second field to say +which entry is which. A `PlacedOccurrence` is whichever occurrence reference +that realization's binding relation targets — a `FabricFuOccurrenceRef` for a +Compute Realization and a `FabricMemoryOccurrenceRef` for a Memory one — so an +entry is exactly what the `RealizationBindingAction` the sweep emits carries, +and no new reference kind appears here. + +It is a field rather than something re-derived because it is the one input to +a demonstration that nothing else determines. Given it, the recorded sweep is a +pure function of the problem, the order, and the environment view, so replay +runs no annealer at all. Without it, the only path back to the same placement +is re-running the annealing pass, which this document elsewhere prices as the +dominant cost in this environment — and adoption, every read of the offline +stage, and every verification would each pay it again for a result already +computed once. + +`replayed_final_energy_code` is the energy the second pass produced, and it is +the authority. It is not the energy the annealing run reported, and the two +differ: placing realizations in a different order routes their nets +differently, so the same final placement carries a different routing and a +different energy. Recording the annealer's number would describe a state the +demonstration never occupies. + +A record carries no observation and no action sequence. Both are regenerated by +sweeping `final_placement` in `demonstration_order` against the environment +view, which is deterministic and involves no search — so the view stays a +configuration rather than becoming a dataset, and a demonstration set is a +value that can be compared by digest. A placement vector is bounded by the +realization count and does not make it one. + +### Admission + +A demonstration is admitted only if it is something the online stage could have +produced. Three conditions, each a check on the record rather than a run of +it: + +Its `action_count` is within the `step_bound` every training stage's +environment view declares. The prior system did not check this, and its offline +episodes ran to lengths the online policy was never permitted to reach, so +pretraining taught a behaviour the environment then forbade. + +Its `final_placement` binds every realization exactly once: its length is the +problem's realization count and no anchor repeats. That is what the +"every action is unmasked" condition reduces to here — the construction mask +admits realization bindings on unplaced anchors, so a sweep that never places +one twice is unmasked at every step by construction, and a repeated anchor is +the only way a configured `demonstration_order` could break it. The same length +check establishes that no realization is left unplaced. + +Adoption therefore performs no sweep. Both structural conditions are decidable +from the placement vector directly, and running a construction pass per +demonstration to rediscover them would make every adoption — and every restart, +and every sweep iteration that re-adopts — pay one probe with route closure per +realization for facts already in the record. Checking that the recorded energy +and closure are the ones a sweep actually produces is +[Harnesses](#harnesses) work, done once against a corpus rather than at every +run that binds it. + +A demonstration covers construction and nothing else. Under +`ConstructThenAnneal` that is the whole episode, so pretraining sees complete +episodes. Under `ConstructThenRepair` the repair phase has no demonstration at +all, and learning it is the online stage's job — which is the division of +labour the two stages exist for, not a gap in the corpus. + +### Determinism + +The demonstration set is the inventory this run carries, enumerated and +regenerable on the terms +[Inventory Identity](spec-ml-core-training.md#inventory-identity) sets. What +discharges the regeneration half here is that the same `generator_seed`, +`annealing_policy`, `demonstration_order`, and `pnr_config_view_digest` +reproduce the same admitted set exactly: every draw the generator makes comes +from a stream the protocol derives. The prior generator failed all of this at +once — it +shuffled with an unseeded generator, it passed its seed to a flag the scheduler +did not read for scheduling so every retry re-ran an identical stream, and its +output layout varied with task completion order — and the resulting corpus +could not be rebuilt or compared. + +## Stages + +The default run is two stages: + +```text +stage 0 Marwil advancing AtPlateau +stage 1 Ppo final, carrying no advance +``` + +Both are the core's `TrainingStage`, and the handoff is the core's: stage 1 +begins from stage 0's parameters and, the arm having changed, from no algorithm +state at all. The demonstrations stage 0 reads are the ones +`ResolvedMlPnrRunConfigView.demonstration_config_digest` binds; the binding +names no corpus of its own, so there is exactly one place a run says which +placements it pretrains on. + +`gamma` must agree across the two stages, and adoption checks it. The reward is +an exact energy difference, so a discount is not a tuning knob but a statement +about which future improvements count; a pretraining stage that discounted +differently would fit a value head to a different return than the one the +online stage optimizes, and the transferred head would be biased in a direction +no diagnostic attributes to the handoff. + +A single-stage run is legal and is how an ablation isolates either half: stage +0 alone measures what the demonstrations contain, and stage 1 alone measures +what online training reaches without them. The pair is the claim; the singles +are the evidence for it. + +### Pretraining + +The `Marwil` stage reads its demonstrations by sweeping each admitted record's +`final_placement` through the environment, which yields observation, action, +reward, and termination per step at the cost of one construction pass and no +search. A demonstration sample carries **no behaviour-policy +log-probability**, and none is required: the algorithm applies no importance +correction, and a synthesized placement sweep has no behaviour policy that +could report one. A pipeline that fabricates a log-probability to satisfy a +schema is reporting a number about a policy that never acted. + +`beta` is stated against what the demonstration set contains. When every +demonstration is a completed annealing placement, returns vary little across +the set, the advantage weighting flattens toward one, and the stage is +behaviour cloning whatever `beta` nominally says. A configuration that wants +weighting to do work needs demonstrations of visibly different quality — a set +generated under several annealing policies, or one that admits partial-quality +placements — and this document requires the choice to be deliberate rather than +inherited. + +### Online Training + +The `Ppo` stage is the core's binding, over the environment views the stages +declare. Its return range is this environment's: + +```text +step_bound * max(energy_width + repair_step_reward_code, + failed_transition_reward_code) + + cleanup_displacement_reward_code * max_displacement + + incomplete_closure_reward_code +``` + +`energy_width` is the quantization width of the selected search energy. +`repair_step_reward_code` participates only under `ConstructThenRepair` and +`cleanup_displacement_reward_code` only under `ConstructThenAnneal`, and the +maximum is taken over whichever arm each bound stage declares. + +The repair charge is added to the energy width rather than maximized against +it, and that is the difference from the design-space arithmetic worth naming. A +rejected step there carries a code and no energy delta, because nothing was +committed, so the two are alternatives and a maximum is exact. A committed +repair step here carries both: the transition's own energy delta *and* the +per-repair charge. Maximizing them would understate every repair step by +whichever term is smaller, and a `value_bound` checked against that +understatement clips the returns of exactly the arm it was computed for. + +`max_displacement` is the one term that depends on the problem rather than the +configuration. A cleanup moves each realization at most `r` hops, where +`Bounded(r)` is the arm's `cleanup_annealing_policy.realization_move_radius`, +and [Annealing And Replay](spec-pnr.md#annealing-and-replay) anchors +displacement at the run-start occurrence, so displacement is bounded by the +realization count times `r` plus the moved count, itself bounded by the +realization count. Only a `ConstructThenAnneal` arm has a cleanup, and that arm +is required to carry `Bounded`, so the term always has a radius to read and +contributes nothing under the other arm — the same condition +`cleanup_displacement_reward_code` already carries. + +Both quantities are available without a frozen model: the realization count is +how many Compute and Memory Realizations the TechMapping the problem binds +declares, and `r` is a configuration field. That matters because freezing every +pool member to obtain an integer already present in `T` would make adoption pay +this environment's dominant cost for nothing. The bound is computed per problem +and the run's requirement is the +maximum over the pool. That a term is instance-dependent is exactly why +[The Reachable Return Range](spec-ml-core-training.md#the-reachable-return-range) +leaves the arithmetic to a training document rather than fixing a formula. + +## Statistics + +The production rule, the objective-dimension record, the outcome accounting, +the cost statistics, and chunk invariance are the core's. This environment +names its own vocabulary and adds what only it has. + +Its non-advancing step class is `failed`, so the core's accounting identity +reads `steps == advanced + failed + elective_stops`, and the failure rate is +reported per member of `PnrTransitionFailureReason`. Keeping `IntrinsicInvalid` +apart from `WorkLimit` matters here more than a blended rate would suggest: one +says the policy proposed something that cannot produce a legal candidate and +the other says the router ran out of budget, and a rising second with a flat +first is a configuration problem that a blended rate would present as a policy +problem. + +Its action partition is the `SpatialMappingAction` kind, so action frequencies +and their success rates are keyed by kind. Under construction only one kind is +live, so the partition is informative exactly in the repair phase — which is +where the question "is the policy doing anything but rebinding" is worth +asking. + +Four statistics are this environment's own: + +```text +placed_fraction_at_end placed realizations over realization count +closure_rate episodes whose final candidate closed +cleanup_displacement ConstructThenAnneal only +repair_steps_taken ConstructThenRepair only +``` + +`cleanup_displacement` is the arm-A quantity the reward already prices, +reported directly because it is the measure of what construction left undone. A +run whose energy improves while its displacement also rises is not improving +construction; it is leaning harder on the annealer, and only the pair +distinguishes the two. + +```text +PnrBreakdownAxis = + Stage // 0, mandatory + | TerminalReason // 1, mandatory + | Problem // 2 + | EpisodeArm // 3 + | ActionKind // 4 +``` + +`Problem` is the axis that answers whether a run improved everywhere or only on +the instances it saw most, and it is the expensive one: it multiplies every +statistic by the pool size, so it is opt-in like every other axis. + +## Test Protocol + +`ResolvedMlPnrTestSetConfigView` is the core's `ResolvedTestSetConfigView` +under this descriptor, and this document supplies only its case payload: + +```text +PnrTestCase { + instance: SpatialPnrProblemBinding + case_seed: u64 +} +``` + +A `SpatialPnrProblemBinding` names a Canonical Dataflow Program, a TechMapping, +a Fabric, and a MappingConstraintSet, so a test set is a grid over exactly the +two things a run wants to hold fixed and vary: the workloads, which vary `D`, +and the hardware versions, which vary `F`. Both are named rather than drawn, so +the set is the same set at every evaluation. + +### Results And Comparison + +```text +PnrTestCaseResult { + outcome: TestCaseOutcome + final_energy_code: uint64 +} +``` + +`final_energy_code` is the only quantity a case adds. Everything else worth +reading about the episode — its length, its terminal reason, its closure, its +displacement — is already inside the `Completed` arm of `outcome`, which +carries this environment's episode statistics catalog. Repeating any of it here +would be two sources for one fact, multiplied by every case and every +evaluation the run retains. + +The run retains one series per case, appending a result at each evaluation. +What makes a series a measurement rather than a log is what it is compared +against, and here that is the run's own history: an evaluation reports each +case's result together with its difference from the same case at the previous +evaluation and at the most recent stage boundary. + +The stage-boundary reading is the one that carries the argument. It is what the +pretrained policy achieved before online training touched it, so the difference +against it is exactly the question the two-stage design asks — whether PPO +improved on what the demonstrations delivered. The previous-evaluation +difference is the ordinary progress signal. + +Improvement over iterations is therefore a trend in a per-case series against a +fixed reference, not a movement in an aggregate. An aggregate over cases hides +the case that regressed, and a policy that improves its mean while losing its +hardest problems is the failure this reporting exists to catch. + +Two comparability rules, because a series only means something if its terms +measure the same thing. Results compare only across evaluations sharing both a +test-set digest and an environment view digest. And a stage advance that +changes either starts a new series rather than extending the old one, with the +new series' first reading becoming its own reference. + +There is deliberately no annealer baseline in these numbers. Comparing a policy +to the annealer that produced its demonstrations is a real question, but it is +a question about a finished checkpoint rather than about a run in progress, and +answering it during training would require a full annealing invocation per case +per evaluation — which is the dominant cost in this environment and would make +evaluation cost more than training. A finished comparison runs offline against +the results `loom-pnr-train` recorded. + +## Harnesses + +```text +loom-pnr-demos generate, extend, and verify a demonstration view +loom-pnr-train run training against a run view +loom-pnr-test run a test set against a checkpoint +``` + +Their outputs are removable projections, on the terms +[Harnesses](spec-ml-core-training.md#harnesses) states. + +`loom-pnr-demos` verifies an existing set without regenerating it, which +`final_placement` is what makes possible: every named problem resolves, every +admitted record's sweep reproduces its recorded energy and closure, and every +record satisfies the three admission conditions against the environment views +the run binds. No annealing runs. That last check is what catches a +demonstration set that was valid for the run that produced it and is not valid +for the run about to use it. + +## Conformance Anchors + +Stable tests cover a demonstration whose recorded energy is the replay's rather +than the annealing run's, and the two differing when `demonstration_order` +changes; a demonstration containing exactly one binding per realization, in +`demonstration_order`, with no realization bound twice and none left unplaced; +every demonstration action being unmasked at the step it is taken; a +demonstration whose `action_count` exceeds a bound stage's `step_bound` being +refused at adoption; a demonstration set regenerated from the same seed, +policy, order, and Place and Route view digest reproducing the same admitted +set; a record's sweep and its admission checks running without any annealing +invocation; the scripted +annealer selecting every Action through the owner's proposal and acceptance +protocols and consuming no host entropy; a `Marwil` stage requiring no +behaviour-policy log-probability and rejecting a batch that fabricates one; the +two stages being rejected at adoption when their `gamma` differs; a stage-1 +evaluation at the boundary reporting the parameters stage 0 produced; a +single-stage run of either algorithm being legal; the return range accounting +for `step_bound` occurrences of the most expensive per-step charge rather than +one, including its arm-conditional terms, a committed repair step counting its +energy delta and its repair charge together rather than the larger of the two, +and its `max_displacement` term being the maximum over the problem pool rather +than a configured constant and being computed without freezing any problem; a +`value_bound` below that maximum being refused; +the failure rate being reported per `PnrTransitionFailureReason` with +`IntrinsicInvalid` and `WorkLimit` never blended; action frequencies keyed by +`SpatialMappingAction` kind; `cleanup_displacement` reported only under +`ConstructThenAnneal` and `repair_steps_taken` only under +`ConstructThenRepair`; a test case naming a problem that appears in no training +stage's environment view; a result series comparing only across evaluations +sharing a test-set digest and an environment view digest, and a stage advance +that changes either starting a new series; each evaluation reporting a per-case +difference against both the previous evaluation and the most recent stage +boundary; a case result carrying no quantity its `outcome` already carries; and +`loom-pnr-demos` refusing a demonstration set that is valid for its generating +run and invalid for the run binding it. + +Tests do not pin `beta`, annealing policy values, demonstration counts, the +chosen `demonstration_order`, plateau tolerances, stage lengths, the contents +of any particular demonstration set or test set, achieved energies, closure +rates, displacement values, wall-time numbers, or diagnostic text. diff --git a/docs/spec-pnr.md b/docs/spec-pnr.md index 1371d6733..44fde0ce8 100644 --- a/docs/spec-pnr.md +++ b/docs/spec-pnr.md @@ -2155,6 +2155,7 @@ SearchPolicy { cooling_ratio: ExactRatio proposals_per_level_base: uint64 proposals_per_movable_decision: uint64 + realization_move_radius: RealizationMoveRadius } focused_closure { proposal_limit: positive uint64 @@ -2210,7 +2211,8 @@ PnR-kernel defaults. All initial builtin profiles use Action weights `1:3:2` for realization, routing, and resource Actions; PathFinder `Multiplicative` with initial pressure `1`, reduced growth ratio `3/2`, and history increment `1`; quantile `3/4`; target acceptance `4/5`; fallback -temperature `1024`; minimum temperature `1`; cooling ratio `19/20`; no route +temperature `1024`; minimum temperature `1`; cooling ratio `19/20`; +`realization_move_radius` `Unbounded`; no route guidance; and deterministic master seed `0` with the protocols named above. ```text @@ -2537,9 +2539,61 @@ SearchPolicy.annealing { cooling_ratio proposals_per_level_base proposals_per_movable_decision + realization_move_radius } + +RealizationMoveRadius = + Unbounded // 0 + | Bounded(positive uint64) ``` +`realization_move_radius` bounds how far one `RealizationBindingAction` may +move an occurrence from where that realization was bound when the annealing run +started. `Unbounded` is the value every builtin profile selects. It is a union +rather than an integer with a reserved value because zero is not a bound of +zero hops, and spelling it as one would let arithmetic over the field compute a +displacement bound of zero for a run with no bound at all. + +`Bounded(r)` removes from that run's proposal domain every realization-binding +choice whose occurrence exceeds `r` directed frozen-topology hops from that +realization's run-start occurrence, using the same immutable endpoint, arc, +payload-width, and attachment domains the initializer's preference refinement +already measures distance over, so no second topology or router model appears. + +The bound is a search preference and not a legality authority. It removes +proposals, never candidate legality: a choice it excludes remains a legal +Mapping decision, and an empty proposal neighborhood is not `ProvenInfeasible`. +It applies only to realization-binding proposals; routing and resource Actions +are untouched. + +The bound forms a proposal domain rather than filtering a draw taken from one. +An annealing run's proposal domain is the projection of `A(M,C,S)` this policy +selects from; `A(M,C,S)` itself is what the candidate admits and the radius +does not narrow it. Within that projection a realization anchor whose every +alternative the radius excludes is not present, the realization-binding kind is +not live when no such anchor remains, and every `nextBounded` call still ranges +over a nonempty canonical domain. Filtering after the draw was rejected because +it would leave empty anchors in the projection, which the Action proposal +contract forbids and which `nextBounded` cannot be called over. + +A selector that is not this policy — one that enumerates `A(M,C,S)` directly +rather than drawing a proposal from it — is unaffected by the radius under +either arm, because the projection is formed by the run that reads the field +and not by the candidate. + +A `Bounded` radius therefore lowers `movableDecisionCount` and shortens a +proposal level. That is a real consequence of the bound and not a defect: a +level's length is proportional to the decisions the level can actually move, +and a run that may move fewer of them has less to propose. The level length is +still fixed once at level start from the domain rebuilt there, so no +proposal-local rebuild changes it mid-level. + +The run-start occurrence is fixed once, at the start of the annealing run that +reads the policy, and is not re-anchored per temperature level or per accepted +move. Re-anchoring would let a sequence of within-radius moves drift +arbitrarily far, which is exactly the property a caller selecting a radius is +buying against. + Fractions use canonical integer or fixed-point values. Quantile is in `[0,1]`, target acceptance is in `(0,1)`, temperatures are positive, and the reduced cooling ratio is strictly between zero and one. Per-level proposal count is: @@ -2551,8 +2605,11 @@ proposals_per_level_base ``` `movableDecisionCount` counts each canonical typed selected-decision anchor -whose current dynamic Action domain contains at least one legal alternative, -plus one routing decision for each residual logical net or service leg. +whose current proposal domain contains at least one legal alternative, plus one +routing decision for each residual logical net or service leg. It is the +proposal domain and not `A(M,C,S)` because the count exists to size a proposal +level, and a level that counted anchors this run may not propose would draw +proposals it must then discard. Under `Unbounded` the two coincide. `SingleSink`, `RootedSubtree`, `WitnessRegion`, and `Global` are neighborhoods over those routing decisions and do not add decisions. Choice cardinality does not multiply the count. The domain is rebuilt once at level start for this @@ -3035,6 +3092,16 @@ Tests protect semantic anchors rather than implementation shape: central `Promote` separation; replay-stable annealing; focused closure; and exact-repair taxonomy, proof-bearing statuses, solver-call budget, and lexicographically canonical extraction; +* `realization_move_radius` `Unbounded` admitting the complete + realization-binding proposal domain, a `Bounded` radius excluding exactly the + choices beyond it while leaving `A(M,C,S)`, `CandidateDomain`, routing and + resource proposals, final verification, and `K` admission unchanged, a + radius-emptied anchor being + absent from the dynamic domain rather than drawn and retried, the + realization-binding kind becoming not live when every such anchor is empty, + an empty neighborhood not being reported as `ProvenInfeasible`, and the + run-start occurrence not being re-anchored by an accepted move or a + temperature level; * shared objective dimensions, three-valued CNF truth, independent full `V/G` and `Q`, TotalOrdering versus SearchEnergy separation, base verification, and exact admission;