corpus-research skill + literature-CLI enrichment (snowball, provenance, rejected-stats sidecar)

README.md

@@ -35,6 +35,13 @@ npx skills add allenai/asta-plugins --all -g
 > /plugin install asta-preview
 ```
 
+- **Corpus Research** (this branch only, preview) - Build and interrogate a curated,
+  evidence-linked literature corpus over multiple rounds (acquisition, judging, extraction,
+  reports). Dev setup: `make install`, then launch Claude Code with the plugin loaded
+  (see DEVELOPER.md for the recommended alias). Before a session, run the environment
+  check from your working folder:
+  `bash <repo>/plugins/asta-preview/skills/corpus-research/scripts/preflight.sh feat/corpus-research`
+  (validates the asta CLI, your `S2_API_KEY`, auth token, and the paper-finder endpoint).
 - **Find Literature** - General paper searching and citation finding
 - **Literature Report Generation** - Comprehensive report writing with synthesis
 - **Run Experiment** - Computational experiments with automated report generation

plugins/asta-preview/skills/corpus-research/references/co-inquiry.md

@@ -0,0 +1,64 @@
+# Learn-mode / co-work conventions (v0)
+
+Everything here is OFFERED, never required (P3): conventions a session composes when the user
+is learning through the corpus, not just interrogating it. The frame: co-work between two
+agents, each holding (1) their own understanding and (2) a model of the other's — and both can
+be stale. Neither side is an oracle. The mode's defining behavior is healthy constructive
+doubt in BOTH directions, with repair when the models diverge.
+
+## 1. Learner ledger — `vault/learner.md`
+The system's model of the user, made INSPECTABLE AND NEGOTIABLE. Record the user's stated
+beliefs, assumptions, and compass anchors VERBATIM (their words, quoted — never paraphrased),
+each with date + status `{active | tested | revised | retired}`. Append when the user states a
+belief or intrigue; the USER owns edits — a user disagreeing with an entry is itself signal,
+not an error. Sections:
+- **Compass** — what the user says the work is ultimately FOR; check load-bearing decisions
+  against it.
+- **Expertise map** — user-DECLARED expert/novice areas. Scaffolding that helps novices hurts
+  experts (expertise-reversal): explanation depth and pushback framing key off this field.
+  Never infer entries; ask or wait for declaration.
+- **Beliefs** — dated verbatim entries with status. A belief gets TESTED only when a question
+  touches it (no automatic misconception hunting); a status flip cites the evidence.
+
+## 2. Turn-stance read + calibrated pushback
+Before executing a user turn, read it: **RULING** (execute) / **HYPOTHESIS** (test it) /
+**REACTION** (probe what triggered it). Silent by default — surface the read ONLY when it
+changes the action. Push back only when expected value exceeds the interruption cost (the
+mixed-initiative when-rule): one sentence, naming the consequence the user may not see
+("replacing loses the comparison — still replace?"). Proportionality is the zeroth row: doubt
+scales with stakes; never interrogate 1+1. **Misfire log**: record every pushback + outcome
+`{upheld | overruled}` in the session notes — wrong pushbacks are data, not embarrassments.
+
+## 3. Premise-stating on action
+At load-bearing moves, state the premise: "acting on X because I believe Y" — the
+instruction-side twin of the how-performed note. It hands the user a hook to catch the
+session's stale model before damage.
+
+## 4. Trajectory links
+QUESTIONS.log entries gain one optional field: `spawned_from: <id of parent question/turn>`.
+Morphing questions stay one traceable arc instead of disconnected asks. One field, no new file.
+
+## 5. Two-way uncertainty (seed)
+State the session's own confidence where the probe is licensed — membership, synthesis,
+disagreement shapes — and NEVER absence-like shapes (measured: confident-but-wrong on every
+absence flip). Ask for the user's uncertainty SPARINGLY, at genuine junctures — uncertainty is
+a stage signal to steer by, not a form to fill.
+
+## 6. Gricean norm block (adopted from measured results: +27% task accuracy, +8% appropriate
+## clarification when stated explicitly)
+Include in learn-mode session guidance:
+> Say as much as the exchange needs and no more. Say only what you have evidence for — and
+> mark what you don't. Make it relevant to what the user is actually pursuing (their words,
+> their compass). Be orderly and plain. When an instruction is ambiguous and the ambiguity
+> changes the outcome, ask ONE targeted clarifying question instead of guessing.
+
+## 7. SRL-lite session frame
+Open with a 2-line goal in the user's words ("what do you want to understand by the end?").
+Close with a 2-line reflection (what moved, what's still open) — appended to the ledger arc,
+not a ceremony.
+
+## 8. The erosion warning (design principle, not a primitive)
+Well-powered RCTs: assistance can raise in-task performance while ERODING persistence,
+unassisted skill, and metacognitive calibration. At learning junctures prefer prompting the
+user's own reasoning (a self-explanation one-liner: "what would X predict here?") over handing
+the answer. Corrective friction is a feature.

plugins/asta-preview/skills/corpus-research/references/codebook.md

@@ -0,0 +1,69 @@
+# Codebook — grounded aspect vocabulary, DERIVED from this corpus
+
+When a question needs content families (which abilities/methods/phenomena/techniques a corpus
+covers), you need a shared tag vocabulary. DERIVE it from THIS corpus — never import a codebook
+from another thread (that's a train-test leak and it won't fit).
+
+## Derivation (grounded coding)
+1. **Open-code** a sample (titles+abstracts of the relevant set): note the phenomenon/method each
+   analyzes, in the papers' own terms.
+2. **Cluster** the open codes into ~10–20 families; each family = {name, one-line definition,
+   a few exemplar corpusIds}.
+3. **Name + freeze** as `codebook@v1`; write `codebook_version` into thread.json.
+4. **Apply by exact-match** to every relevant paper as an ordered tag batch
+   (`<run>/tag-batches/<NN>-<name>.jsonl`: {corpusId, primary_family, secondary_families, confidence}).
+5. **Monitor "Other"** — genuine "other/unclassifiable" (excluding empty-abstract) growing past
+   ~10% of the relevant set triggers re-derivation: re-open-code the "Other" pile → codebook@v(N+1)
+   → re-tag → rebuild. Bump the version; the substrate stamps it on every record.
+
+## Two failure modes the gates catch (both real)
+- **Untagged ≠ Other.** When the relevant set EXPANDS (new acquisition rounds), the new papers
+  must be tagged before any distribution analysis, or an inflated "Other" bucket silently distorts
+  the family distribution. `substrate.py`'s TAG_GATE fails <90% real-family — do not quote a
+  distribution until it passes.
+- **Over-conservative tagging.** A batch that dumps classifiable papers into "Other" fails the
+  same gate. Re-tag them (they usually fit existing families) rather than inventing families.
+
+## Parametric family anchor — REQUIRED gate after codebook@v1 (§anchor)
+A corpus-derived codebook is circular by construction: a whole missing family produces no
+"Other" growth, no tag-gate failure, nothing. The anchor is the validated check for that class
+(caught 4 real families hidden in a real run's "Other"; correct null on a good codebook;
+charter-conditioning is the false-positive control — 15/20 spurious naive → 0 conditioned):
+> **Protocol.** Reading ONLY the thread charter (question, deliverables, out_of_scope_families)
+> — NOT the codebook or corpus — enumerate 12–25 content families you'd expect this literature
+> to span, with one-line definitions; WRITE THEM DOWN before opening the codebook. Then align
+> each to the codebook: PRESENT / SUBSUMED / ABSENT. For every ABSENT family, title-scan the
+> run's candidates and core: (a) papers in-core → carve out the family or add it as a NAMED
+> residual (never silently in Other); (b) papers in candidates but not core → audit the
+> curation/scope decision; (c) papers nowhere → thin-literature note or an acquisition probe.
+> For SUBSUMED families with substantial in-core mass, consider promotion at codebook@v2.
+> Record the table in the run's coverage evidence; the gate FAILS if any ABSENT-case-(a) family
+> is left uncarved and unnamed.
+The anchor is one-directional (it misses families you didn't think to enumerate) — it
+complements grounded open-coding, never replaces it.
+
+## A codebook is also a scope audit
+Deriving families exposes clusters that don't belong to the thread's phenomenon (an orthogonal
+subfield, a methodology-only cluster). Those become `scope.out_of_scope_families` in thread.json
+(when scope.axis == "separate") — the family-based half of the scope test.
+
+## Methods/entities get their own codebook
+The same recipe derives a METHODS codebook (normalize free-text method strings into countable
+families) or an ENTITY vocabulary — whatever the question aggregates over. Derive per axis.
+
+## Canonicalization maps are DATA — gate them like data
+Entity/name canonicalization (model names, method names, dataset names) uses a **2-level scheme**:
+raw string → `{canonical_name, family}` (spelling variants → one canonical; versions → one
+family). Rules, each learned from a real failure:
+- **Frozen artifact, never inline.** The map is a versioned FILE (`canon-map.json`) applied
+  deterministically [T]. Inline normalization ships its errors silently; a file is a 30-second
+  audit.
+- **Attested names only.** `canonical_name` must appear (modulo punctuation/case) somewhere in
+  THIS corpus — a title or an extraction. NEVER invent a name; when none is attested, keep the
+  raw string as the canonical and assign only `family`. Real failure: parameter sizes became
+  phantom versions ("FooCoder 33B" → "Foo 33", "a 1.1B Bar-architecture model" → "Bar 1.1").
+- **Machine self-check → review queue.** Scan the map for unattested canonical names (normalized
+  substring check against the corpus vocabulary); fix or revert each hit; surface UNCERTAIN
+  mappings to the user at a beat — a review queue, not silent auto-accept.
+- **General rule: pair every "produce X" with "gate X".** Any derived mapping/table ships only
+  after an acceptance check on the artifact itself — same discipline as merges and substrates.