Add hrr-replay-findings skill for read-only HRR replay triage

skills/hrr-replay-findings/SKILL.md

@@ -0,0 +1,103 @@
+---
+name: hrr-replay-findings
+description: >-
+  Replay and analyze HIP Runtime Replay (HRR) capture archives. Classifies GPU
+  faults and replay divergence (read-only/OOB write, illegal access, NaN/Inf,
+  hang, OOM) and extracts fault address, event index, and kernel. Use when the
+  user provides an HRR archive, asks to replay a recording, triage a GPU crash,
+  or mentions hrr-playback, capture.hrr, or Memory access fault during replay.
+---
+
+# HRR Replay Findings
+
+Replay an HRR archive on the host GPU, then produce a structured finding.
+
+## What the user should say
+
+The user only needs to point at the recording. Examples:
+
+- *"Replay and analyze this HRR archive: `/data/capture.hrr/pid-1842`"*
+- *"What's in this capture? `capture.hrr/pid-1842`"*
+- *"Analyze this replay log from an HRR run"* (log-only path)
+
+The user should **not** need to name scripts, set env vars, pick a GPU, or know where ROCm is installed.
+
+## What to ask the user (only if missing)
+
+| Missing | Ask once |
+|---------|----------|
+| Archive path | *"Which `capture.hrr/pid-*` directory should I use?"* |
+| `hrr-playback` not found after discovery | *"Where is `hrr-playback` installed on this machine?"* |
+
+Do **not** ask for: GPU index, Docker, source trees, HIP library paths, ROCm install path (assume `/opt/rocm`).
+
+## Agent workflow
+
+```
+1. Resolve archive — user path, or largest events.bin under capture.hrr/pid-*
+2. Discover hrr-playback (see below); ask user only if not found
+3. Run skill scripts/run_hrr_replay.sh --archive <dir> --analyze
+4. Read .finding.md and explain in plain language
+```
+
+**Execute in the same turn** — do not narrate planning steps.
+
+### Discover `hrr-playback` (in order)
+
+1. `command -v hrr-playback`
+2. `$ROCM_PATH/bin/hrr-playback` (default `ROCM_PATH=/opt/rocm`)
+3. `/opt/rocm/bin/hrr-playback`
+4. User-provided path → set `HRR_PLAYBACK` for that run only
+
+`run_hrr_replay.sh` adds `/opt/rocm/lib` and a sibling `lib/` next to the playback binary to `LD_LIBRARY_PATH` automatically.
+
+### Discover archive
+
+If the user gives `capture.hrr/` without a `pid-*` child, pick the `pid-*` directory whose `events.bin` is largest.
+
+### Commands (agent runs these — not the user)
+
+```bash
+SKILL=<path-to>/hrr-replay-findings   # from .cursor/skills or installed skills dir
+
+# Metadata only (~seconds, no full replay):
+"$SKILL/scripts/run_hrr_replay.sh" --archive <pid-dir> --info
+
+# Full replay + structured finding:
+"$SKILL/scripts/run_hrr_replay.sh" --archive <pid-dir> --analyze
+```
+
+Log-only (no replay):
+
+```bash
+python3 "$SKILL/scripts/analyze_replay_finding.py" \
+  --log <replay.log> --archive <pid-dir> --format markdown -o finding.md
+```
+
+## System assumptions
+
+| Assumption | Default |
+|------------|---------|
+| ROCm install | `/opt/rocm` (`$ROCM_PATH` override) |
+| GPU | Auto-pick device with most free VRAM |
+| Replay mode | Native host (`/dev/kfd`); no Docker |
+
+## Fault taxonomy
+
+| `fault_class` | Meaning |
+|---------------|---------|
+| `replay_pass` | Clean replay |
+| `read_only_page_fault` | Write to read-only page |
+| `illegal_memory_access` | Other GPU memory fault |
+| `nan_inf_divergence` | D2H numerical mismatch |
+| `hang` | Device/queue hang |
+| `replay_oom` | Out of VRAM — report environment issue |
+| `replay_fatal_api` | HIP API error stopped replay |
+
+## Capture explainer (short)
+
+- **events.bin** — recorded HIP API sequence
+- **blobs/** — code objects and payloads
+- **Trailer** — absent when the original run crashed; reader recovers complete events
+
+See [reference.md](reference.md) and [examples.md](examples.md).

skills/hrr-replay-findings/examples.md

@@ -0,0 +1,32 @@
+# Examples
+
+## What the user says
+
+**Full replay + analysis**
+
+> Replay and analyze my HRR archive at `/data/crash/capture.hrr/pid-1842`
+
+**Archive summary only**
+
+> Summarize this HRR capture: `capture.hrr/pid-1842`
+
+**Existing log**
+
+> Analyze this HRR replay log: `replay.log` (archive is `capture.hrr/pid-1842`)
+
+The user does not mention scripts, `HRR_PLAYBACK`, or GPU numbers.
+
+## What the agent does
+
+1. Finds `hrr-playback` on `PATH` or `/opt/rocm/bin/hrr-playback`
+2. If missing, asks: *"Where is hrr-playback installed?"*
+3. Runs `run_hrr_replay.sh --archive ... --analyze`
+4. Presents the finding
+
+## If hrr-playback is not in a standard location
+
+User answers: *"It's in `/opt/amd-hrr/bin/hrr-playback`"*
+
+Agent sets `HRR_PLAYBACK=/opt/amd-hrr/bin/hrr-playback` for that session only and re-runs.
+
+If a `lib/` directory sits beside `bin/`, the runner picks it up automatically.

skills/hrr-replay-findings/reference.md

@@ -0,0 +1,123 @@
+# HRR capture reference
+
+## Directory layout
+
+```
+capture.hrr/
+  pid-<capture_pid>/
+    events.bin          # primary event stream (may be GB-scale)
+    blobs/              # code objects, graphs, memcpy payloads
+    (optional metadata files)
+```
+
+Pick the `pid-*` directory with the **largest `events.bin`** for the faulting process.
+
+## events.bin record model (conceptual)
+
+Each event has:
+
+| Field | Role |
+|-------|------|
+| Thread id | Capturing host thread |
+| Sequence / event index | Monotonic call index in replay |
+| API id | HIP API (malloc, launch, memcpy, sync, …) |
+| Payload | API-specific bytes (variable-length for kernel launches) |
+
+**Kernel launch payload** includes: stream, kernel name, code-object hash, grid, block, shared memory, **kernarg blob** (pointer table + struct args), optional D2H snapshot descriptors.
+
+## Completeness markers
+
+| Signal | Meaning |
+|--------|---------|
+| `Complete: YES` (`--info`) | Clean shutdown trailer present |
+| `recovered N events` | Crash capture; trailer missing; reader kept all complete records |
+| `Torn trailing record` | Last record partial; preceding events valid |
+
+On recent ROCm builds with crash-resilient HRR capture, crash captures are **expected** to lack a trailer and still be replayable (reader reports `recovered N events`).
+
+## Replay log lines
+
+### Progress
+
+```
+[HRR progress] elapsed_s=... seq=13118764 kernels=797227 d2h_pass=4303 d2h_fail=0 ...
+```
+
+- `seq` — last replayed event sequence number (use as **failing_call_index** proxy when fault follows)
+- `kernels` — kernel launch count so far
+- `d2h_*` — device-to-host validation counters
+
+### GPU memory fault (ROCr)
+
+```
+Memory access fault by GPU node-N (Agent handle: 0x...) on address 0xADDR. Reason: ...
+:0:rocdevice.cpp:NNNN: Memory Fault Error [..., faulting addr: 0xADDR, kernel: Cijk_...]
+```
+
+Extract: **fault_address**, **kernel_name**, **gpu_node**, **fault_reason**.
+
+### Hang analysis block
+
+```
+Dispatch Header = 0x..., grid=[...], workgroup=[...], kernarg_address=0x..., kernel_obj=0x...
+```
+
+Extract: **kernarg_address**, **grid**, **workgroup** — ties fault to launch packet.
+
+### Fatal API abort
+
+```
+[HRR] Fatal: T146 Event 9268 (hipMalloc) returned 2 (out of memory) — aborting replay
+```
+
+Extract: **failing_thread**, **failing_call_index**, **failing_api**.
+
+### Suballoc fidelity (optional playback feature)
+
+```
+[HRR] SUBALLOC OOB: kernel arg[10] rec 0x... resolves inside a captured segment but in no active tensor block
+```
+
+High count on `arg[10]` with `d2h_fail=0` and later MAF → likely **stale/OOB device pointer** in kernarg, not host numerics.
+
+## Tensile / hipBLASLt kernel name cheat sheet
+
+Example:
+
+```
+Cijk_Alik_Bljk_BBS_BH_Bias_HA_S_SAV_UserArgs_MT128x192x128_..._SK3_..._WS64_WG16_16_1
+```
+
+| Token | Meaning |
+|-------|---------|
+| `Cijk_*` | Contraction GEMM family |
+| `MT128x192x128` | Macro-tile dimensions |
+| `SK3` | StreamK variant |
+| `WS64` | Workspace-related sizing hint |
+| `Bias_HA` | Bias + HPA layout flags |
+
+**read_only_page_fault** on StreamK GEMM → investigate edge tile / workspace (`AddressWS`) / output (`AddressD`) stores.
+
+## Playback build fidelity
+
+The same archive can **PASS** on one `hrr-playback` build and fault on another. Record which playback binary produced each result.
+
+## ROCm layout (assumed)
+
+| Path | Role |
+|------|------|
+| `/opt/rocm/bin/hrr-playback` | Default playback tool location |
+| `/opt/rocm/lib` | HIP/HSA and ROCm runtime libraries |
+| `$ROCM_PATH` | Override prefix if ROCm is not under `/opt/rocm` |
+
+`run_hrr_replay.sh` prepends these to `LD_LIBRARY_PATH`. If `hrr-playback` lives in `<prefix>/bin/`, `<prefix>/lib` is added automatically.
+
+## Parser script
+
+```bash
+python3 skills/hrr-replay-findings/scripts/analyze_replay_finding.py --help
+```
+
+Outputs JSON or Markdown `Finding` with fields:
+
+`outcome`, `fault_class`, `fault_address`, `failing_event_seq`, `failing_call_index`, `kernel_name`, `kernarg_address`, `d2h_fail`, `archive_events`, …