jit: make trace readers and terminal cleanup nonwaiting

Author: Frityet

notes/gc2-jit-tactical-smr-marker-2026-07-13.md

@@ -70,30 +70,30 @@ active node.
 The same ordinary collision existed one level later in trace compilation.
 `lj_trace_free_unpublished()` retires a compact trace which has no public trace
 number, epoch, or retire-list link. Its recorder token/local construction owner
-is the sole body authority. Before publishing the epoch, the old path called
-the generic raw marker for the trace and non-mcode exit table; an IDLE writer
-loss could therefore pin `NO_RECLAIM` even though no semantic edge was missing.
-
-Only that explicitly unpublished pre-claim step now uses the publication
-marker. It first tries one outer SMR admission covering every trace-body
-validation and exit-table load. On admission loss it publishes only T's root
-retry request and returns without dereferencing T or the exit table. On
-success, the nested T/exit-table publication marks are reentrant. The epoch
-claim is immediately followed by the existing mandatory
-`trace_preservebody()` pass and retire-list publication before the recorder
-token is released. That pass still marks KGC/prototype/snapshot/link children
-and retains its fail-closed behavior. Public trace retirement, aged retired-root
-scans, and exclusive-reclaimer requeue continue using their mandatory or
-reclaim-held marker variants.
-
-The trace-retire fixture holds the IDLE writer while the recorder token owns a
-synthetic unpublished body and exit table. It proves both marks remain clear,
-SMR remains writer-owned with zero readers, and the activation snapshot is
-bit-for-bit unchanged. After release, it starts MARK and the exact runtime
-retirement path marks both allocations plus a real starting-prototype child,
-then claims/lists the trace for normal later grace/terminal drain. This
-distinguishes the pre-claim raw publication from the mandatory post-claim
-semantic traversal.
+is the sole body authority. The assembler copy is not a partially published
+semantic trace: it has copied IR bytes, but no trace slot, prototype/root-spine
+edge, snapshots, exit table, debugger descriptor or executable mcode. Its KGC
+operands remain rooted by token-private `J->cur` until the failed recording is
+aborted. Treating the copy as a semantic body required a looping post-claim
+traversal and manufactured ownership for fields the allocation never acquired.
+
+Unpublished retirement now sets the immutable
+`TRACE_RETIRED_UNPUBLISHED` kind, claims an epoch, publishes the exact body on
+the token-owned retire list, and performs only one-shot pre/post raw marks.
+Admission loss requests a root retry and returns without reading the compact
+payload. There is deliberately no `trace_preservebody()` pass. Root scans mark
+the exact listed allocation without decoding children; mature reclaim skips
+public-slot, inbound-link, debugger, root-spine, exit-table and mcode teardown,
+validates the strict scratch shape, and frees that allocation exactly. Public
+trace retirement keeps the existing semantic preservation policy.
+
+The trace-retire fixture holds the IDLE metadata writer while the recorder
+token calls the real `lj_trace_free_unpublished()` path. The scratch includes a
+deliberately unmarked KGC operand, while snapshots, start prototype, exit table
+and mcode remain NULL. Both tactical marks lose without waiting; the exact body
+and KGC marks remain clear, SMR stays writer-owned with zero readers, and the
+activation snapshot is unchanged. The tagged/listed body then survives young
+grace and is physically reclaimed at the mature epoch.
 
 ## Adjacent audit item
 

notes/jit-read-smr-nonwaiting-2026-07-19.md

@@ -0,0 +1,117 @@
+# Nonwaiting JIT trace-reader SMR tranche
+
+Date: 2026-07-19
+
+This b1.2.1 checkpoint removes the looping GC2 SMR admission API from the JIT
+reflection, recorder lookup, assembler lookup and optional GDB-JIT metadata
+files. It is a bounded tranche, not a claim that trace publication, flush,
+retirement or the repository-wide wait inventory is complete.
+
+## Runtime policy
+
+The `jit.util` trace-reflection functions are observational under concurrent
+mutation. They already used a one-shot recorder-token acquisition and returned
+no result when a peer owned it. They now apply the same rule to SMR: failure to
+enter the current trace-body generation is a transient no-result observation.
+If the function acquired the recorder token, it releases that token before
+returning. This prevents an exclusive reclaimer that also needs the token from
+forming a token/SMR dependency cycle.
+
+Reflection does not allocate while retaining trace-body SMR. `tracesnap`
+copies its at-most-255 immutable map entries to bounded C-stack storage before
+building the result table. `tracemc` first sizes the current observation,
+grows the owner TG's managed scratch buffer outside SMR, and then performs a
+second one-shot lookup which copies only a body that fits. String interning is
+therefore outside both trace SMR and the temporary recorder-token claim; an
+allocation error cannot leak a trace pin or strand a reclaimer. The second
+admitted interval also converts the mcode address to an integer scalar before
+release; no expired trace or mcode pointer value is evaluated afterward.
+`traceexitstub` applies the same rule to its borrowed exit-stub address.
+`tracek` reduces scalar constants to copied IR words; for `IR_KGC`, whose word
+still names a reclaimable child, it additionally acquires an exact GC2
+allocation lease under trace SMR, publishes the copied TValue as a Lua stack
+root after leaving trace SMR, and only then releases the child lease. A
+transient or stale child admission is another no-result observation.
+
+Recorder and assembler lookups have a stronger semantic obligation, but both
+are speculative. A failed one-shot SMR admission aborts that recording turn
+with the admission-specific `LJ_TRERR_SMRRETRY`, allowing execution to continue
+in the interpreter and a later hot turn to record again. The new error was
+appended to the catalog so every pre-existing numeric trace-error value remains
+unchanged. It gets the same immediate-hotcount policy as `LJ_TRERR_RETRY`, but
+only `SMRRETRY` suppresses the TRACE-abort event: ordinary optimizer, stale-body
+and table retries remain observable through the existing API. The assembler's
+target-return-PC shortcut is optional and simply retains the snapshot PC when
+admission loses.
+
+The hot-side probe likewise remains interpreted when admission loses. After a
+snapshot has already been restored, a `JLOOP` target revalidation collision
+returns to bytecode redispatch. Neither optimization can make an exiting
+mutator wait for a trace reclaimer.
+
+GDB-JIT metadata is optional. If a side trace's parent cannot be admitted, its
+debug entry is omitted instead of waiting or publishing an object with a
+guessed parent stack adjustment. Normal trace execution and retirement are
+unchanged.
+
+Debug trace-PC recovery similarly reports no bytecode position on contention.
+The x86/x64 final link assembler retries the recording turn. Bytecode dumping
+recovers patched root instructions from the prototype's immutable sidecar;
+only a compatibility fallback tries trace-body SMR, and that admission is
+one-shot with an already bounded live-bytecode resample.
+
+There are now no `lj_gc2_smr_read_enter()` call sites in `lib_jit.c`,
+`lj_record.c`, `lj_asm.c`, `lj_asm_x86.h`, `lj_bcwrite.c`, `lj_debug.c` or
+`lj_gdbjit.c`. A monotonic source gate protects this completed domain. Blocking
+admissions remain in `lj_trace.c` and GC/safepoint lifecycle files and must be
+converted with their matching flush/retirement descriptors; this tranche does
+not hide that debt behind an allowlist.
+
+## Terminal cleanup
+
+Failed assembler copies now retire as raw unpublished scratch descriptors: they
+carry no semantic snapshot/prototype/exit-table graph and use one-shot raw marks
+around token-owned list publication. Successful assembly, ordinary cancel and
+owner-abort paths publish INTERP+IDLE and release the recorder token before
+dispatch repair. First-area `MCODEAL` and max-trace exhaustion likewise defer a
+full flush until after that terminal release. An asynchronously aborted
+`MCODELM` restart falls through full slot cleanup instead of leaking `J->cur`,
+and down-recursion retries only when its restarted recorder remains non-IDLE.
+
+The normal active `TRACE_START` path still installs its recording dispatch
+overlay while retaining the token. Any wait/dependency hidden in that active
+overlay update is explicit remaining debt; silent-IDLE start exits no longer
+share it.
+
+## Evidence
+
+The recorder-token fixture closes SMR in `LJ_GC2_SMR_META_EXCLUSIVE` and calls
+every trace-numbered `jit.util` reader. Each call returns without a result,
+does not increment the reader count and leaves the recorder token unowned. A
+regression to the looping admission API hangs this deliberately closed scope
+and is terminated by the focused suite timeout.
+
+The immutable-start-instruction fixture also publishes a real patched `JLOOP`,
+closes SMR exclusively, dumps the owning function, reloads the bytecode and
+executes it. This proves that bytecode serialization uses the prototype
+sidecar rather than waiting for or dereferencing the trace vector. Serialization
+now acquire-loads every mutable `BCIns` independently and always runs a captured
+JIT opcode through sidecar/unpatch recovery. It no longer races a bulk `memcpy`
+or a stale `PROTO_ILOOP`/prototype-trace fast-path decision against publication
+and flush.
+
+The same fixture exposed a stale XPOLL assertion after mutable-global tracing
+became conservative. A safe runnable side trace may now end at the interpreter
+instead of using `LJ_TRLINK_ROOT`; the test now checks the actual invariant, an
+exact snapshot on a non-loop XPOLL in any runnable side-trace topology.
+
+## Remaining event debt
+
+The exact closed-gate `SMRRETRY` path cannot safely call arbitrary TRACE-abort
+handlers while it still owns the recorder token, because a handler may enter a
+reader needed by the already-admitted writer. Other abort events retain their
+historical visibility. They still run token-held and therefore retain a more
+general close-after-check dependency race if a metadata-exclusive owner closes
+after event preparation begins. Moving those callbacks requires preserving
+`jit.dump` and user abort-observation ordering and is intentionally recorded as
+follow-up debt rather than hidden by suppressing ordinary `RETRY` events.

notes/jit-unpublished-scratch-retire-2026-07-19.md

@@ -0,0 +1,79 @@
+# Raw unpublished JIT scratch retirement
+
+Date: 2026-07-19
+
+## Problem and object kind
+
+Assembler retry/abort can leave `J->curfinal`, a compact allocation made by
+`lj_trace_alloc()`. It contains a copied IR span, but it has never acquired the
+semantic ownership of a published `GCtrace`: there is no trace number or slot,
+prototype/root-spine edge, snapshot/snapmap publication, exit table, debugger
+entry, executable mcode or native-entry pin. The live recorder's `J->cur` is the
+semantic root for KGC operands until abort cleanup ends.
+
+The old unpublished path routed this allocation through ordinary retired-trace
+preservation. That could loop in `trace_preservebody()` while the recorder token
+was held and an exclusive SMR owner needed the same token. It also interpreted
+uninitialized/non-owned fields as if this were a published trace.
+
+## Retirement protocol
+
+`TRACE_RETIRED_UNPUBLISHED` is an immutable atomic kind bit. The recorder owner
+clears `J->curfinal` first, sets the kind, performs a one-shot raw pre-mark,
+claims the retirement epoch, closes native-pin admission, publishes the exact
+allocation on the token-owned retire list, and performs a one-shot raw post-mark.
+Either mark may lose SMR admission; the local owner before list publication and
+the list owner afterward are the lifetime authority, while the miss reopens the
+active root certificate. Neither mark decodes the IR or any child field.
+
+All three runtime retirement sites clear the token-private pointer before this
+handoff: assembler IR-buffer replacement, synchronous `trace_abort()` and
+`lj_trace_abort_owner()`. `lj_trace_alloc()` initializes `snap`, `snapmap`, GC
+links, optional pointer-auth/GDB fields and every other scratch discriminator to
+NULL/zero so validation never reads indeterminate storage.
+
+## Retire-list consumer audit
+
+Every production traversal recognizes the immutable kind:
+
+- GC root marking preserves only the exact allocation.
+- Mature reclaim bypasses inbound-link scanning, slot release, debugger removal
+  and root-spine unlink, then validates and destroys the exact allocation.
+- Mcode-area reference scans report no reference for scratch bodies.
+- Stale-start-instruction recovery skips them.
+- Terminal pin preflight validates their shape; close skips debugger teardown.
+- Runtime/terminal destruction never frees an exit table for this kind.
+- Requeue and payload-preservation decisions remain raw and nonsemantic.
+
+The strict validator requires zero trace/link/root identities, NULL semantic
+side pointers, zero mcode size and pin count, CLOSED pin admission, a nonzero
+retirement epoch, no debugger entry, and exactly the unpublished kind bit.
+Runtime corruption requeues fail-closed; terminal corruption aborts rather than
+guessing ownership.
+
+## Adjacent recorder terminal ordering
+
+Successful assembly and terminal error/owner-abort cleanup now publish INTERP
+and IDLE, release the recorder token, and only then run ordinary dispatch repair.
+First-area `MCODEAL` and trace-number exhaustion defer full flush until after
+that release. `trace_start()` returns ACTIVE, IDLE or FLUSH_ALL so invalid
+bytecode, GC gating, `PROTO_NOJIT` and max-trace exits cannot refresh dispatch
+while token-held. Down-recursion retries only when the restarted recorder
+actually remains non-IDLE. An asynchronously aborted `MCODELM` restart falls
+through ordinary slot cleanup instead of publishing IDLE with a leaked pending
+trace. A stale hot dispatch also checks `JIT_F_ON` before attempting the token.
+
+## Evidence and remaining debt
+
+The focused retire fixture invokes the real unpublished retirement path while
+holding both the recorder token and `LJ_GC2_SMR_META_EXCLUSIVE`. A deliberately
+unmarked KGC operand proves there is no semantic traversal. The call returns
+with zero readers, unchanged activation, an unmarked but tagged/listed exact
+body, and then reclaims it only after full trace grace.
+
+`LJ_TRERR_SMRRETRY` separately identifies failed recorder/assembler SMR
+admission. Only that known closed-gate path suppresses TRACE-abort callbacks;
+ordinary `LJ_TRERR_RETRY` remains observable. Generic abort callbacks still run
+while the recorder token is held and retain a close-after-check dependency race.
+Relocating them must preserve `jit.dump` and user abort-event semantics and is
+explicit follow-up work.