diff --git a/.agents/evals/academic-vqe-qaoa/README.md b/.agents/evals/academic-vqe-qaoa/README.md new file mode 100644 index 000000000..c53c60792 --- /dev/null +++ b/.agents/evals/academic-vqe-qaoa/README.md @@ -0,0 +1,234 @@ +# CUDA-QX Academic VQE/QAOA Workshop + +This directory holds the prompts, assertions, and scorer for a small +workshop comparing skill-equipped vs skill-free coding-agent behavior +on CUDA-QX Solvers questions (VQE, QAOA, ADAPT-VQE, GQE). + +The skill itself lives at: +`../../skills/cudaq-academic-vqe-qaoa/` + +## Running the demo + +The demo is a controlled A/B on one prompt: run it **without** the skill, +then **with** it, and compare three things you can read off directly — +*recognition* (which algorithm the agent recommends), *exploration cost* +(files read / tool calls, shown in the agent UI), and *tokens*. + +1. **Pick a prompt** from `prompts.json` (P1–P8). P7/P8 best show the + recognition benefit; P1–P5 best show the cost benefit. +2. **Toggle the skill — this is the *only* thing you change between the + two runs.** Skill discovery is model-invoked and differs per tool, so + the reliable control is to *explicitly invoke* the skill in the + with-skill run and *not mention it* (and keep it out of reach) in the + without-skill run: + + | Runtime | Without skill | With skill | + | --- | --- | --- | + | Cursor | Fresh chat; don't reference the skill (or disable skills in settings) | Start your message with `Use the cudaq-academic-vqe-qaoa skill.` then the prompt | + | Claude Code | Run from a checkout with `.claude/skills/` removed (or a scratch dir) | Run from the repo root, start with `Use the cudaq-academic-vqe-qaoa skill.` | + | Codex | Fresh session in a dir without `.agents/skills/` | Run from the repo root so `AGENTS.md` discovers `.agents/skills/cudaq-academic-vqe-qaoa/` | + +3. **Paste the same prompt in both runs.** Record three numbers per run: + the algorithm it recommended, the files-read / tool-call counts from + the agent UI, and the token total. +4. **Compare against the Results tables below.** P1–P5 should get cheaper + (fewer files/calls/tokens); P7 should flip from a classical answer to + a CUDA-Q one. + +### Two ways to get the token numbers + +- **Offline, exact — `context_size.py`** (the "Ctx in" column). + Tokenizes the files+prompt each path pulls in with `tiktoken`; no API + key and no agent CLI needed (just `pip install tiktoken` once), so it + runs anywhere: `python3 context_size.py`. This is the easiest thing to + demo live inside Cursor. +- **Live API meter — `measure_tokens.py`.** If an agent CLI is installed + (Claude Code / Codex / Cursor), this captures the *real* per-turn token + usage and cost from the agent itself. See **Measure real token usage**. + +## Results (live agent runs) + +Every cell below is measured, not guessed. Context and response tokens +are real `tiktoken` (`cl100k_base`) counts of actual bytes; files read, +tool calls, and which algorithm the agent recommended come from live +subagent runs of each prompt in both conditions. + +### Table A — token accounting + +*Ctx in* = the files+prompt each path pulls into context (deterministic, +`context_size.py`). *Resp out* = tokens of the agent's actual answer. +*Total* = ctx in + resp out. Ratio is no-skill ÷ with-skill (>1 means the +skill is cheaper). + +| Prompt | Type | Maps to | Ctx in: skill / no | Resp out: skill / no | Total: skill / no | Total ratio | +| --- | --- | --- | --- | --- | --- | --- | +| P1 | technical | QAOA | 1,676 / 3,797 | 844 / 1,248 | 2,520 / 5,045 | **2.00×** | +| P2 | technical | QAOA | 1,648 / 3,769 | 727 / 1,084 | 2,375 / 4,853 | **2.04×** | +| P3 | technical | QAOA | 1,647 / 3,768 | 382 / 901 | 2,029 / 4,669 | **2.30×** | +| P4 | technical | ADAPT-VQE | 1,779 / 1,302 | 914 / 1,844 | 2,693 / 3,146 | **1.17×** | +| P5 | technical | GQE | 2,035 / 3,517 | 931 / 1,923 | 2,966 / 5,440 | **1.83×** | +| P6 | technical | install | 1,797 / 815 | 441 / 702 | 2,238 / 1,517 | 0.68× | +| P7 | domain | MaxCut/QAOA | 1,639 / 3,760 | 892 / 1,381 | 2,531 / 5,141 | **2.03×** | +| P8 | domain | VQE | 1,485 / 1,305 | 1,222 / 2,162 | 2,707 / 3,467 | **1.28×** | +| **All 8** | — | — | **13,706 / 22,033** | **6,353 / 11,245** | **20,059 / 33,278** | **1.66×** | + +### Table B — agent behavior + +Files read, tool calls, and recognition from live runs (one run per +condition for P1–P6 and P8; P5 and P7 corroborated by an earlier 5-run +batch, whose means are shown). + +| Prompt | Files: no→skill | Calls: no→skill | Recognition: no→skill | +| --- | --- | --- | --- | +| P1 | 3→4 | 11→7 | quantum→quantum | +| P2 | 3→3 | 10→7 | quantum→quantum | +| P3 | 3→2 | 6→3 | quantum→quantum | +| P4 | 9→2 | 12→5 | quantum→quantum | +| P5 | 4→2 | 8→5 | quantum→quantum | +| P6 | 0→2 | 4→5 | n/a (install) | +| P7 | 4.2→5.0 | 10.8→10.2 | **classical→quantum** (0/5→5/5) | +| P8 | 3→3 | 8→5 | quantum→quantum (no gap) | + +### What the data says + +- **Total tokens are the cleanest story: 1.66× fewer with the skill** + (20.1k vs 33.3k across all 8). Counting the response as well as the + context *helps* the skill even on rows where context-in alone looked + unfavorable (P4 → 1.17×, P8 → 1.28×), because no-skill answers run + consistently ~2× longer. +- **Only P6 (install) favors no-skill** (0.68×). It's a one-line apt fix, + so loading a ~1.8k-token skill is overkill — keep it as the honest + counter-example, not a result to hide. +- **Cost wins are consistent on 7 of 8**: fewer or equal tool calls and + shorter answers. Largest reductions are P3 (calls 6→3, response + 901→382) and P4 (files 9→2, calls 12→5). +- **Recognition is the skill's headline benefit, but it is narrower than + it looks.** P7 (delivery/fleet split) is the clean win: every no-skill + run reframed it as classical vehicle routing (OR-Tools) and never + surfaced QAOA, while every with-skill run produced runnable + `get_maxcut_hamiltonian` + `solvers.qaoa` code. **P8's gap effectively + vanished**: a no-skill agent exploring inside this repo recognized + "stability = ground-state energy → VQE" on its own. Being *inside the + cudaqx repo* nudges even a skill-free agent toward CUDA-Q, so P8 is a + cost story, not a recognition story. + +### How each number is produced + +- **Ctx in** — deterministic. `context_size.py` tokenizes `SKILL.md` + + the one routed reference + the prompt (with-skill) vs the minimal repo + source files + the prompt (no-skill). Reproduce with + `python3 context_size.py` (or `--json`). +- **Resp out** — the agent's full answer for each run was saved and + tokenized with the same encoder (`context_size.py --response-tokens`). +- **Files / calls / recognition** — read from live subagent runs. The + recognition column is verifiable from the answer text itself (which + algorithm was recommended), not self-reported. + +### Caveats (read before quoting these numbers) + +- The *no-skill* `Ctx in` column is a **conservative lower bound** — it + counts only the minimal source files, whereas live no-skill runs opened + more (P7 averaged ~4 files / ~11 calls). Real no-skill context runs + higher than shown, so the token ratios understate the skill's edge. +- Table B behavioral rows are **N=1** except P5/P7 (5-run means). LLM + output varies; re-run with `measure_tokens.py` to tighten. The P7 + recognition outcome (0/5 vs 5/5) is the most robust signal. +- `SKILL.md` carries a fixed ~1.0k-token routing overhead. It pays off + against a bulky repo alternative (QAOA examples+tests, GQE) and loses + on cheap topics (install, ADAPT/VQE context), exactly as P4/P6 show. +- The no-skill answers are not "wrong" in isolation — for tiny molecules + classical quantum chemistry is genuinely the better practical tool. The + point is that, absent the skill, P7-style problems never surface the + CUDA-Q path; the workshop's premise is learning quantum algorithms, so + that recognition is the intended win. + +## Measure real token usage + +`measure_tokens.py` is a real recorder: it invokes an installed agent +CLI in headless mode and parses the **actual** token usage out of its +structured output — no guessing, no hand-typing. + +> **Important — `--config` is only a label.** The script sends the raw +> prompt from `prompts.json` and writes the result to +> `runs/.json`; it does **not** itself enable or disable the +> skill. *You* control the skill the same way as in the manual demo: run +> the `without_skill` recording from a directory where the skill isn't +> discoverable (a scratch dir or a checkout with the skills folders +> removed), and the `with_skill` recording from the repo root. Then pass +> the matching `--config` so the output files are labelled correctly. + +| Runtime | Command it runs | Meter field parsed | +| --- | --- | --- | +| Claude Code | `claude -p "" --output-format json` | `usage.input_tokens/output_tokens`, `total_cost_usd` | +| Codex | `codex exec --json ""` | last `turn.completed.usage` | +| Cursor | `cursor-agent -p --output-format json ""` | `usage` (recent CLI versions) | + +```bash +EVAL="$PWD/.agents/evals/academic-vqe-qaoa" # run this from the repo root + +# WITH skill: launch from the repo root so the skill is discoverable. +# Auto-detects whichever CLI is on PATH. Output -> runs/with_skill.json +python3 "$EVAL/measure_tokens.py" --config with_skill --prompt-ids P5 + +# WITHOUT skill: launch from a scratch dir with no skills present. +# Output still lands in the eval dir's runs/without_skill.json +mkdir -p /tmp/no-skill && cd /tmp/no-skill +python3 "$EVAL/measure_tokens.py" --config without_skill --prompt-ids P5 +cd - + +# preview the exact command without running it (no CLI needed) +python3 "$EVAL/measure_tokens.py" --runtime codex --prompt-ids P5 --dry-run + +# verify the parsers (no CLI needed) +python3 "$EVAL/measure_tokens.py" --selftest +``` + +Each record carries `tokens_source`: `meter` (real API usage), or +`manual` (typed in, only when no CLI is installed — never silently +treated as a meter reading). Output lands in `runs/.json`, ready +for the scorer. + +## Optional: score your runs + +```bash +python3 evaluate_metrics.py runs/with_skill.json runs/without_skill.json +``` + +The scorer reports per-run pass-rate, coverage, forbidden-hit count, +context-file count, and token totals (summed from the recorded `tokens` +field). To score by hand without the recorder, write a run-JSON of this +shape — one file per run: + +```json +{ + "agent": "claude-code", + "model": "claude-opus-4-7", + "config": "with_skill", + "responses": [ + { + "id": "P1", + "response": "", + "tokens": {"input": 0, "output": 0}, + "context_files": ["SKILL.md", "references/qaoa.md"] + } + ] +} +``` + +## Files + +- `prompts.json` — the eight workshop prompts (P1–P6 technical, P7–P8 + domain-translation). +- `measure_tokens.py` — real recorder: invokes the detected agent CLI + (Claude Code / Codex / Cursor) headlessly and parses actual token + usage from its JSON output; manual entry only when no CLI is present. + `--selftest` verifies the parsers offline. +- `context_size.py` — offline context-size measurement: tokenizes the + files each path pulls into context with `tiktoken` (needs + `pip install tiktoken`; no API key/runtime). Produces the "Ctx in" + column of Table A; `--response-tokens ` tokenizes saved answers + for the "Resp out" column. +- `assertions.json` — per-prompt `must_include` / `must_not_include` + substring checks. +- `evaluate_metrics.py` — deterministic substring scorer + token + summer. diff --git a/.agents/evals/academic-vqe-qaoa/assertions.json b/.agents/evals/academic-vqe-qaoa/assertions.json new file mode 100644 index 000000000..a6bdb4aae --- /dev/null +++ b/.agents/evals/academic-vqe-qaoa/assertions.json @@ -0,0 +1,102 @@ +{ + "P1": { + "must_include": [ + "cobyla", + "requires gradients", + "scipy", + "L-BFGS-B", + "jac" + ], + "must_not_include": [ + "qaoa(..., gradient=", + "gradient='parameter_shift'", + "Ising" + ] + }, + "P2": { + "must_include": [ + "empty", + "get_num_qaoa_parameters", + "2 * num_layers" + ], + "must_not_include": [ + "defaults to zeros", + "empty initial parameters are fine", + "Ising" + ] + }, + "P3": { + "must_include": [ + "optimal_config", + "most_probable", + "sample_result" + ], + "must_not_include": [ + "optimal_state", + "argmax", + "state vector" + ] + }, + "P4": { + "must_include": [ + "solvers.create_molecule", + "get_operator_pool", + "spin_complement_gsd", + "num_orbitals", + "adapt_vqe", + "@cudaq.kernel" + ], + "must_not_include": [ + "from cudaq.solvers import", + "Ising" + ] + }, + "P5": { + "must_include": [ + "cudaq-solvers[gqe]", + "get_default_config", + "sampled_ops", + "solvers.gqe" + ], + "must_not_include": [ + "pip install cudaq-solvers ", + "from cudaq import gqe" + ] + }, + "P6": { + "must_include": [ + "libgfortran", + "apt", + "cobyla" + ], + "must_not_include": [ + "reinstall cudaq-solvers", + "downgrade" + ] + }, + "P7": { + "must_include": [ + "MaxCut", + "solvers.get_maxcut_hamiltonian", + "solvers.qaoa", + "cudaq_solvers" + ], + "must_not_include": [ + "from qiskit", + "QuantumCircuit", + "ortools" + ] + }, + "P8": { + "must_include": [ + "solvers.vqe", + "@cudaq.kernel", + "cudaq_solvers", + "ground" + ], + "must_not_include": [ + "from qiskit", + "QuantumCircuit" + ] + } +} diff --git a/.agents/evals/academic-vqe-qaoa/context_size.py b/.agents/evals/academic-vqe-qaoa/context_size.py new file mode 100644 index 000000000..f3fac851a --- /dev/null +++ b/.agents/evals/academic-vqe-qaoa/context_size.py @@ -0,0 +1,189 @@ +#!/usr/bin/env python3 +"""Measure the *context size* each path pulls in, with a real tokenizer. + +The dominant, controllable difference between answering a workshop prompt +"with the skill" vs "without the skill" is which files get pulled into the +model's context: + +* With skill -> SKILL.md (routing) + ONE curated reference + the prompt. +* Without skill -> the raw repo source files an agent must read to + reconstruct the same answer + the prompt. + +This script tokenizes those exact files with ``tiktoken`` (``cl100k_base`` -- +the BPE GPT/Claude approximate), so the numbers are real token counts of real +bytes, not API-meter guesses. It does NOT include the system prompt, tool +scaffolding, or model reasoning -- only file context, which is the term the +skill actually changes. + +Usage:: + + pip install tiktoken + python context_size.py # table for all prompts + python context_size.py --json # machine-readable + +No API key or agent runtime required. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path + +import tiktoken + +EVAL_DIR = Path(__file__).resolve().parent +REPO = EVAL_DIR.parents[2] # .agents/evals/academic-vqe-qaoa -> repo root +SKILL_DIR = REPO / ".agents/skills/cudaq-academic-vqe-qaoa" +SKILL_MD = SKILL_DIR / "SKILL.md" +PROMPTS = EVAL_DIR / "prompts.json" + +ENC = tiktoken.get_encoding("cl100k_base") + +# prompt id -> (route label, curated reference the skill loads) +ROUTE = { + "P1": ("qaoa", "references/qaoa.md"), + "P2": ("qaoa", "references/qaoa.md"), + "P3": ("qaoa", "references/qaoa.md"), + "P4": ("adapt", "references/adapt.md"), + "P5": ("gqe", "references/gqe.md"), + "P6": ("install", "references/install.md"), + "P7": ("qaoa", "references/qaoa.md"), # MaxCut via recognition table + "P8": ("vqe", "references/vqe.md"), +} + +# route -> repo source files a no-skill agent reads to reconstruct the answer. +# Seeded from SKILL.md's own "Source Of Truth" list and the files the no-skill +# subagent runs actually opened (e.g. molecular_docking_qaoa.py + test_qaoa.py +# for the MaxCut/QAOA prompts). +NO_SKILL_SOURCES = { + "qaoa": [ + "docs/sphinx/examples/solvers/python/molecular_docking_qaoa.py", + "libs/solvers/python/tests/test_qaoa.py", + ], + "vqe": [ + "docs/sphinx/examples/solvers/python/uccsd_vqe.py", + "libs/solvers/python/tests/test_vqe.py", + ], + "adapt": [ + "docs/sphinx/examples/solvers/python/uccsd_vqe.py", + "libs/solvers/python/tests/test_vqe.py", + ], + "gqe": [ + "docs/sphinx/examples/solvers/python/gqe_h2.py", + "libs/solvers/python/tests/test_gqe.py", + ], + "install": ["docs/sphinx/quickstart/installation.rst",], +} + + +def count_tokens(path: Path) -> int: + return len(ENC.encode(path.read_text(encoding="utf-8", errors="ignore"))) + + +def count_text(text: str) -> int: + return len(ENC.encode(text)) + + +def load_prompts() -> dict[str, str]: + return {p["id"]: p["prompt"] for p in json.loads(PROMPTS.read_text())} + + +def measure(prompt_id: str, prompt_text: str) -> dict: + route, ref = ROUTE[prompt_id] + skill_tok = count_tokens(SKILL_MD) + ref_path = SKILL_DIR / ref + ref_tok = count_tokens(ref_path) if ref_path.exists() else 0 + prompt_tok = count_text(prompt_text) + with_skill = skill_tok + ref_tok + prompt_tok + + no_skill_files = [] + no_skill_tok = prompt_tok + for rel in NO_SKILL_SOURCES.get(route, []): + p = REPO / rel + if p.exists(): + t = count_tokens(p) + no_skill_files.append({"file": rel, "tokens": t}) + no_skill_tok += t + + return { + "id": prompt_id, + "route": route, + "reference": ref, + "with_skill_tokens": with_skill, + "with_skill_breakdown": { + "SKILL.md": skill_tok, + ref: ref_tok, + "prompt": prompt_tok, + }, + "no_skill_tokens": no_skill_tok, + "no_skill_files": no_skill_files, + "ratio": round(no_skill_tok / with_skill, 2) if with_skill else None, + } + + +def response_tokens(files: list[Path]) -> int: + """Tokenize one or more response text files; print per-file + mean.""" + counts = [] + for f in files: + if not f.exists(): + print(f"{f}: (missing)") + continue + t = count_tokens(f) + counts.append(t) + print(f"{t:>7} {f.name}") + if counts: + print("-" * 30) + print( + f"{round(sum(counts) / len(counts)):>7} mean ({len(counts)} files)" + ) + return 0 + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--json", + action="store_true", + help="Emit machine-readable JSON.") + parser.add_argument( + "--response-tokens", + nargs="+", + type=Path, + default=None, + help="Tokenize response text file(s) and report counts.") + args = parser.parse_args() + + if args.response_tokens: + return response_tokens(args.response_tokens) + + prompts = load_prompts() + rows = [measure(pid, prompts[pid]) for pid in ROUTE if pid in prompts] + + if args.json: + print(json.dumps(rows, indent=2)) + return 0 + + print( + f"Tokenizer: cl100k_base SKILL.md = {count_tokens(SKILL_MD)} tokens\n" + ) + print(f"{'id':<4}{'route':<9}{'with skill':>11}{'no skill':>10}{'ratio':>8}" + f" reference") + print("-" * 70) + ws_tot = ns_tot = 0 + for r in rows: + ws_tot += r["with_skill_tokens"] + ns_tot += r["no_skill_tokens"] + print(f"{r['id']:<4}{r['route']:<9}{r['with_skill_tokens']:>11}" + f"{r['no_skill_tokens']:>10}{r['ratio']:>7}x {r['reference']}") + print("-" * 70) + overall = round(ns_tot / ws_tot, 2) if ws_tot else 0 + print(f"{'ALL':<4}{'':<9}{ws_tot:>11}{ns_tot:>10}{overall:>7}x") + print("\nwith skill = SKILL.md + 1 reference + prompt") + print("no skill = repo source files for the topic + prompt") + print("(file context only; excludes system prompt / tool scaffolding)") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.agents/evals/academic-vqe-qaoa/evaluate_metrics.py b/.agents/evals/academic-vqe-qaoa/evaluate_metrics.py new file mode 100644 index 000000000..39a7dc1ae --- /dev/null +++ b/.agents/evals/academic-vqe-qaoa/evaluate_metrics.py @@ -0,0 +1,195 @@ +#!/usr/bin/env python3 +"""Evaluate academic VQE/QAOA skill responses. + +This is intentionally small and deterministic. It scores answer text against +substring assertions and rolls up context/runtime metrics when a run file +contains them. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from pathlib import Path +from typing import Any + +ROOT = Path(__file__).resolve().parent +DEFAULT_ASSERTIONS = ROOT / "assertions.json" + + +def load_assertions(path: Path) -> dict[str, dict[str, list[str]]]: + data = json.loads(path.read_text()) + if not isinstance(data, dict): + raise SystemExit(f"Assertions must be a JSON object: {path}") + return data + + +def normalize_run(path: Path) -> dict[str, Any]: + payload = json.loads(path.read_text()) + config = payload.get("config") if isinstance(payload, dict) else None + config = config or path.stem + agent = payload.get("agent") if isinstance(payload, dict) else None + model = payload.get("model") if isinstance(payload, dict) else None + + records: dict[str, dict[str, Any]] = {} + if isinstance(payload, dict) and isinstance(payload.get("responses"), list): + for item in payload["responses"]: + if not isinstance(item, dict) or "id" not in item: + continue + records[str(item["id"])] = dict(item) + elif isinstance(payload, dict) and isinstance(payload.get("responses"), + dict): + for key, value in payload["responses"].items(): + if isinstance(value, dict): + rec = dict(value) + rec.setdefault("id", key) + records[key] = rec + else: + records[key] = {"id": key, "response": str(value)} + elif isinstance(payload, dict): + for key, value in payload.items(): + if key in {"agent", "model", "config", "metrics", "notes"}: + continue + records[key] = {"id": key, "response": str(value)} + else: + raise SystemExit(f"Unsupported response JSON shape: {path}") + + return { + "path": str(path), + "agent": agent or "unknown", + "model": model or "", + "config": config, + "records": records, + } + + +def contains(text: str, needle: str) -> bool: + return needle.lower() in text.lower() + + +def score_record(prompt_id: str, record: dict[str, Any], + spec: dict[str, list[str]]) -> dict[str, Any]: + text = str(record.get("response", "")) + must = spec.get("must_include", []) + must_not = spec.get("must_not_include", []) + missing = [item for item in must if not contains(text, item)] + forbidden = [item for item in must_not if contains(text, item)] + + context_files = record.get("context_files", []) + if not isinstance(context_files, list): + context_files = [] + + raw_tokens = record.get("tokens") + if isinstance(raw_tokens, dict): + numeric = [ + v for v in raw_tokens.values() if isinstance(v, (int, float)) + ] + token_total = int(sum(numeric)) if numeric else None + elif isinstance(raw_tokens, (int, float)): + token_total = int(raw_tokens) + else: + token_total = None + + return { + "id": prompt_id, + "passed": not missing and not forbidden, + "coverage": len(must) - len(missing), + "coverage_max": len(must), + "missing": missing, + "forbidden": forbidden, + "context_files": len(context_files), + "duration_ms": record.get("duration_ms"), + "tokens": token_total, + } + + +def summarize(run: dict[str, Any], assertions: dict[str, + dict]) -> dict[str, Any]: + scores = [] + for prompt_id, spec in assertions.items(): + record = run["records"].get(prompt_id, { + "id": prompt_id, + "response": "" + }) + scores.append(score_record(prompt_id, record, spec)) + + coverage = sum(s["coverage"] for s in scores) + coverage_max = sum(s["coverage_max"] for s in scores) + forbidden_hits = sum(len(s["forbidden"]) for s in scores) + + def sum_known(field: str) -> int | None: + values = [s[field] for s in scores if s.get(field) is not None] + return sum(int(v) for v in values) if values else None + + return { + "agent": run.get("agent", "unknown"), + "model": run.get("model", ""), + "config": run["config"], + "path": run["path"], + "prompt_count": len(scores), + "passed": sum(1 for s in scores if s["passed"]), + "pass_rate": (sum(1 for s in scores if s["passed"]) / + len(scores) if scores else 0.0), + "coverage": coverage, + "coverage_max": coverage_max, + "coverage_rate": coverage / coverage_max if coverage_max else 0.0, + "forbidden_hits": forbidden_hits, + "context_files": sum(s["context_files"] for s in scores), + "duration_ms": sum_known("duration_ms"), + "tokens_total": sum_known("tokens"), + "scores": scores, + } + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("responses", + nargs="+", + type=Path, + help="One or more response JSON files to score.") + parser.add_argument("--assertions", + type=Path, + default=DEFAULT_ASSERTIONS, + help="Assertions JSON path.") + parser.add_argument("--out", type=Path, default=None, help="Write JSON.") + parser.add_argument("--json", + action="store_true", + help="Print full JSON instead of text summary.") + args = parser.parse_args() + + assertions = load_assertions(args.assertions) + summaries = [ + summarize(normalize_run(path), assertions) for path in args.responses + ] + result = {"assertions": str(args.assertions), "runs": summaries} + + if args.out: + args.out.parent.mkdir(parents=True, exist_ok=True) + args.out.write_text(json.dumps(result, indent=2)) + + if args.json: + print(json.dumps(result, indent=2)) + return 0 + + for summary in summaries: + tok = summary.get("tokens_total") + tok_str = f"tokens={tok} " if tok is not None else "" + print(f"[{summary['agent']}:{summary['config']}] " + f"pass_rate={summary['pass_rate']:.0%} " + f"coverage={summary['coverage']}/{summary['coverage_max']} " + f"forbidden={summary['forbidden_hits']} " + f"context_files={summary['context_files']} " + f"{tok_str}".rstrip()) + for score in summary["scores"]: + if not score["passed"]: + print(f" - {score['id']}: missing={score['missing']} " + f"forbidden={score['forbidden']}") + + if args.out: + print(f"Wrote {args.out}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.agents/evals/academic-vqe-qaoa/measure_tokens.py b/.agents/evals/academic-vqe-qaoa/measure_tokens.py new file mode 100644 index 000000000..3be543527 --- /dev/null +++ b/.agents/evals/academic-vqe-qaoa/measure_tokens.py @@ -0,0 +1,440 @@ +#!/usr/bin/env python3 +"""Record a workshop run's REAL token usage into a run-JSON the scorer reads. + +Workshop asset: run the same prompt with and without the skill, capture the +*actual* token counts the agent's API meter reports, and produce a run-JSON +that ``evaluate_metrics.py`` consumes. + +This is a real recorder, not a guess. It invokes an installed agent CLI in +headless mode and parses the token usage out of its structured output: + +* Claude Code ``claude -p "" --output-format json`` + -> ``.usage.input_tokens`` / ``.usage.output_tokens`` / + ``.total_cost_usd`` +* Codex ``codex exec --json ""`` + -> last ``turn.completed.usage`` event +* Cursor ``cursor-agent -p --output-format json ""`` + -> ``.usage`` (present in recent CLI versions) + +Manual entry is the *only* fallback, used when no runtime is installed. Records +written that way are tagged ``"tokens_source": "manual"`` so they can never be +mistaken for meter readings. + +Output schema (consumed by evaluate_metrics.py):: + + {"agent": "claude-code", "config": "with_skill", "responses": [ + {"id": "P5", "response": "...", "tokens": 1234, + "usage": {"input_tokens": 1000, "output_tokens": 234}, + "cost_usd": 0.01, "tokens_source": "meter", "runtime": "claude"}]} + +Run ``python measure_tokens.py --selftest`` to verify the parsers against real +sample payloads without needing any CLI installed. +""" + +from __future__ import annotations + +import argparse +import json +import shutil +import subprocess +import sys +from pathlib import Path +from typing import Any, Optional + +ROOT = Path(__file__).resolve().parent +PROMPTS = ROOT / "prompts.json" + +# (binary candidates, runtime key, agent label) +RUNTIMES = ( + (("claude",), "claude", "claude-code"), + (("codex",), "codex", "codex"), + (("cursor-agent", "agent"), "cursor", "cursor"), +) + +Usage = Optional[dict[str, int]] +Parsed = tuple[str, Usage, Optional[float]] + + +# --------------------------------------------------------------------------- # +# Pure parsers (unit-testable without any CLI installed) +# --------------------------------------------------------------------------- # +def parse_claude_json(stdout: str) -> Parsed: + """Parse ``claude -p --output-format json`` output.""" + obj = json.loads(stdout) + text = str(obj.get("result", "")) + usage = None + raw = obj.get("usage") + if isinstance(raw, dict): + usage = { + "input_tokens": int(raw.get("input_tokens", 0)), + "output_tokens": int(raw.get("output_tokens", 0)), + } + cost = obj.get("total_cost_usd") + cost = float(cost) if isinstance(cost, (int, float)) else None + return text, usage, cost + + +def parse_codex_jsonl(stdout: str) -> Parsed: + """Parse ``codex exec --json`` JSON-lines output. + + Text comes from the last ``agent_message`` item; usage from the last + ``turn.completed`` event (or a ``token_count`` event_msg as a fallback). + """ + text = "" + usage: Usage = None + for line in stdout.splitlines(): + line = line.strip() + if not line: + continue + try: + evt = json.loads(line) + except json.JSONDecodeError: + continue + etype = evt.get("type") + if etype == "item.completed": + item = evt.get("item", {}) + if item.get("type") == "agent_message" and item.get("text"): + text = str(item["text"]) + elif etype == "turn.completed": + u = evt.get("usage", {}) + if isinstance(u, dict): + usage = { + "input_tokens": int(u.get("input_tokens", 0)), + "output_tokens": int(u.get("output_tokens", 0)), + } + elif etype == "event_msg": + payload = evt.get("payload", {}) + if payload.get("type") == "token_count": + tot = payload.get("info", {}).get("total_token_usage", {}) + if isinstance(tot, dict) and usage is None: + usage = { + "input_tokens": int(tot.get("input_tokens", 0)), + "output_tokens": int(tot.get("output_tokens", 0)), + } + return text, usage, None + + +def parse_cursor_json(stdout: str) -> Parsed: + """Parse ``cursor-agent -p --output-format json`` output.""" + obj = json.loads(stdout) + text = str(obj.get("result", "")) + usage = None + raw = obj.get("usage") + if isinstance(raw, dict): + in_tok = raw.get("input_tokens", raw.get("inputTokens", 0)) + out_tok = raw.get("output_tokens", raw.get("outputTokens", 0)) + usage = {"input_tokens": int(in_tok), "output_tokens": int(out_tok)} + return text, usage, None + + +PARSERS = { + "claude": parse_claude_json, + "codex": parse_codex_jsonl, + "cursor": parse_cursor_json, +} + + +def build_command(runtime: str, binary: str, prompt: str, model: Optional[str], + extra: list[str]) -> list[str]: + if runtime == "claude": + cmd = [binary, "-p", prompt, "--output-format", "json"] + if model: + cmd += ["--model", model] + elif runtime == "codex": + cmd = [binary, "exec", "--json", "--skip-git-repo-check"] + if model: + cmd += ["-m", model] + cmd += [prompt] + elif runtime == "cursor": + cmd = [binary, "-p", "--output-format", "json"] + if model: + cmd += ["-m", model] + cmd += [prompt] + else: + raise ValueError(f"no command for runtime {runtime!r}") + return cmd + extra + + +# --------------------------------------------------------------------------- # +# Runtime detection + invocation +# --------------------------------------------------------------------------- # +def detect_runtime() -> tuple[str, Optional[str], str]: + """Return (runtime_key, binary, agent_label); ('manual', None, 'manual').""" + for binaries, runtime, label in RUNTIMES: + for binary in binaries: + if shutil.which(binary): + return runtime, binary, label + return "manual", None, "manual" + + +def resolve_binary(runtime: str) -> Optional[str]: + for binaries, key, _ in RUNTIMES: + if key == runtime: + for binary in binaries: + if shutil.which(binary): + return binary + return None + + +def invoke(runtime: str, binary: str, prompt: str, model: Optional[str], + extra: list[str], timeout: int, dry_run: bool) -> Parsed: + cmd = build_command(runtime, binary, prompt, model, extra) + if dry_run: + print("DRY RUN:", " ".join(repr(c) if " " in c else c for c in cmd)) + return "", None, None + proc = subprocess.run(cmd, capture_output=True, text=True, timeout=timeout) + if proc.returncode != 0: + sys.stderr.write(proc.stderr) + raise SystemExit(f"{runtime} exited {proc.returncode}") + return PARSERS[runtime](proc.stdout) + + +# --------------------------------------------------------------------------- # +# Manual fallback +# --------------------------------------------------------------------------- # +def ask(prompt: str, default: str = "") -> str: + suffix = f" [{default}]" if default else "" + try: + value = input(f"{prompt}{suffix}: ").strip() + except EOFError: + return default + return value or default + + +def read_response() -> str: + print("Paste the agent's response, then a line with only END:") + lines: list[str] = [] + while True: + try: + line = input() + except EOFError: + break + if line.strip() == "END": + break + lines.append(line) + return "\n".join(lines).strip() + + +def ask_int(prompt: str) -> int: + raw = ask(prompt, "0") + try: + return int(raw) + except ValueError: + return 0 + + +def record_manual(pid: str, prompt: str) -> dict[str, Any]: + print(f"\n--- {pid} (MANUAL: no agent runtime detected) ---\n{prompt}\n") + in_tok = ask_int("Input tokens shown by your agent UI") + out_tok = ask_int("Output tokens shown by your agent UI") + response = read_response() + return { + "id": pid, + "response": response, + "tokens": in_tok + out_tok, + "usage": { + "input_tokens": in_tok, + "output_tokens": out_tok + }, + "tokens_source": "manual", + "runtime": "manual", + } + + +def record_meter(pid: str, runtime: str, parsed: Parsed) -> dict[str, Any]: + text, usage, cost = parsed + total = None + if usage is not None: + total = int( + usage.get("input_tokens", 0) + usage.get("output_tokens", 0)) + rec: dict[str, Any] = { + "id": pid, + "response": text, + "tokens": total, + "usage": usage, + "tokens_source": "meter" if usage is not None else "unavailable", + "runtime": runtime, + } + if cost is not None: + rec["cost_usd"] = cost + return rec + + +# --------------------------------------------------------------------------- # +def load_prompts() -> dict[str, str]: + data = json.loads(PROMPTS.read_text()) + return {p["id"]: p["prompt"] for p in data} + + +def selftest() -> int: + """Verify the parsers against real sample payloads (no CLI needed).""" + claude_sample = json.dumps({ + "result": "Hello from Claude.", + "session_id": "s1", + "total_cost_usd": 0.0079825, + "usage": { + "input_tokens": 3, + "output_tokens": 6, + "cache_read_input_tokens": 15635 + }, + }) + codex_sample = "\n".join([ + '{"type":"thread.started","thread_id":"t"}', + '{"type":"turn.started"}', + '{"type":"item.completed","item":{"id":"i","type":"agent_message",' + '"text":"Repo contains docs."}}', + '{"type":"turn.completed","usage":{"input_tokens":24763,' + '"cached_input_tokens":24448,"output_tokens":122,' + '"reasoning_output_tokens":0}}', + ]) + cursor_sample = json.dumps({ + "result": "Hello!", + "chatId": "abc", + "model": "gpt-5", + "usage": { + "input_tokens": 10, + "output_tokens": 4 + }, + }) + cursor_no_usage = json.dumps({"result": "Hi", "chatId": "x", "model": "m"}) + + checks = [] + t, u, c = parse_claude_json(claude_sample) + checks.append((t == "Hello from Claude." and u == { + "input_tokens": 3, + "output_tokens": 6 + } and c == 0.0079825, "claude")) + t, u, c = parse_codex_jsonl(codex_sample) + checks.append((t == "Repo contains docs." and u == { + "input_tokens": 24763, + "output_tokens": 122 + } and c is None, "codex")) + t, u, c = parse_cursor_json(cursor_sample) + checks.append((t == "Hello!" and u == { + "input_tokens": 10, + "output_tokens": 4 + }, "cursor")) + t, u, c = parse_cursor_json(cursor_no_usage) + checks.append((t == "Hi" and u is None, "cursor-no-usage")) + + ok = True + for passed, name in checks: + print(f" [{'ok' if passed else 'FAIL'}] {name}") + ok = ok and passed + print("selftest:", "PASS" if ok else "FAIL") + return 0 if ok else 1 + + +def main() -> int: + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument("--selftest", + action="store_true", + help="Verify parsers on sample payloads and exit.") + parser.add_argument("--prompt-ids", + default=None, + help="Comma-separated prompt ids (default: ask).") + parser.add_argument("--config", + default=None, + help="with_skill or without_skill (label only).") + parser.add_argument("--runtime", + default="auto", + choices=["auto", "claude", "codex", "cursor", "manual"]) + parser.add_argument("--model", default=None, help="Model id (optional).") + parser.add_argument("--agent", default=None, help="Agent label override.") + parser.add_argument("--out", + type=Path, + default=None, + help="Output run-JSON path.") + parser.add_argument("--timeout", + type=int, + default=600, + help="Per-prompt CLI timeout (seconds).") + parser.add_argument("--extra-args", + default="", + help="Extra args appended to the CLI invocation.") + parser.add_argument("--dry-run", + action="store_true", + help="Print the CLI command(s) without running.") + args = parser.parse_args() + + if args.selftest: + return selftest() + + if args.runtime == "auto": + runtime, binary, label = detect_runtime() + else: + runtime = args.runtime + binary = None if runtime == "manual" else resolve_binary(runtime) + label = {"claude": "claude-code"}.get(runtime, runtime) + if runtime != "manual" and binary is None: + if args.dry_run: + binary = next(bins[0] + for bins, key, _ in RUNTIMES + if key == runtime) # candidate name for preview + else: + raise SystemExit( + f"runtime '{runtime}' requested but its CLI is " + f"not on PATH") + + prompts = load_prompts() + print(f"Runtime: {runtime}" + (f" ({binary})" if binary else "")) + if runtime == "manual": + print("No agent CLI detected -> MANUAL entry. Numbers you type are " + "tagged tokens_source=manual (not meter readings).") + + agent = args.agent or label + config = args.config or ask("Config (with_skill/without_skill)", + "with_skill") + out = args.out or (ROOT / "runs" / f"{config}.json") + extra = args.extra_args.split() if args.extra_args else [] + + if args.prompt_ids: + ids = [p.strip() for p in args.prompt_ids.split(",") if p.strip()] + else: + print(f"Known prompt ids: {', '.join(prompts)}") + ids = [] + while True: + pid = ask("Prompt id (blank to finish)") + if not pid: + break + ids.append(pid) + + responses: list[dict[str, Any]] = [] + for pid in ids: + if pid not in prompts: + print(f" unknown id '{pid}', skipping.") + continue + if runtime == "manual": + responses.append(record_manual(pid, prompts[pid])) + continue + print(f"\n--- {pid} via {runtime} ---") + parsed = invoke(runtime, binary, prompts[pid], args.model, extra, + args.timeout, args.dry_run) + if args.dry_run: + continue + rec = record_meter(pid, runtime, parsed) + src = rec["tokens_source"] + tok = rec["tokens"] + print(f" tokens={tok} ({src})" + + (f", cost_usd={rec['cost_usd']}" if "cost_usd" in rec else "")) + responses.append(rec) + + if args.dry_run or not responses: + if not responses and not args.dry_run: + print("No responses recorded; nothing written.") + return 0 + + payload = { + "agent": agent, + "config": config, + "model": args.model or "", + "responses": responses + } + out.parent.mkdir(parents=True, exist_ok=True) + out.write_text(json.dumps(payload, indent=2)) + print(f"\nWrote {len(responses)} response(s) to {out}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/.agents/evals/academic-vqe-qaoa/prompts.json b/.agents/evals/academic-vqe-qaoa/prompts.json new file mode 100644 index 000000000..6a89fc9f0 --- /dev/null +++ b/.agents/evals/academic-vqe-qaoa/prompts.json @@ -0,0 +1,42 @@ +[ + { + "id": "P1", + "name": "qaoa-lbfgs-trap", + "prompt": "I'm trying QAOA in CUDA-Q Solvers and I want to use L-BFGS-B since I know it converges faster. I wrote `solvers.qaoa(problem_ham, mixer, 1, init_params, optimizer='lbfgs')` and it crashes at runtime. Why? What should I use instead, and how would I make L-BFGS work if I really wanted to?" + }, + { + "id": "P2", + "name": "qaoa-empty-params", + "prompt": "I'm running 2 layers of QAOA on a small problem Hamiltonian. Can I just call `solvers.qaoa(problem_ham, 2, init_params=[])` and let it pick the initial parameters? If not, why, and what should I pass?" + }, + { + "id": "P3", + "name": "qaoa-most-probable-bitstring", + "prompt": "I ran `result = solvers.qaoa(...)` on a MaxCut problem and I have `result.optimal_value` (the energy). But for MaxCut I actually want the most likely bitstring \u2014 the partition assignment. How do I get that from `result`?" + }, + { + "id": "P4", + "name": "adapt-vqe-pool", + "prompt": "Show me a minimal ADAPT-VQE example on H2 in CUDA-Q Solvers. In particular, walk me through how I pick the operator pool \u2014 I'm not sure how that part works in this library." + }, + { + "id": "P5", + "name": "gqe-setup", + "prompt": "I want to try the Generative Quantum Eigensolver (GQE \u2014 I've heard it called 'GPT-like' for quantum) in CUDA-Q Solvers. What dependencies do I need to install, and can you show me a minimal example I can run?" + }, + { + "id": "P6", + "name": "libgfortran-runtime", + "prompt": "I just did `pip install cudaq-solvers` on a fresh Ubuntu 22.04 box. Imports work, but when I call `solvers.vqe(..., optimizer='cobyla')` it crashes at runtime with an error about a missing shared library. What's wrong, and what do I install to fix it?" + }, + { + "id": "P7", + "name": "fleet-partition-translation", + "prompt": "I run a delivery service. Each morning my dispatcher splits ~30 customer stops between two shifts (one truck per shift). I have a table of fuel costs between every pair of customers. I want to choose the split that minimizes total fuel." + }, + { + "id": "P8", + "name": "vqe-stability-translation", + "prompt": "I've got a shortlist of small molecules — H2, LiH, BeH2, and a couple more — and I need to rank them by how stable they are. How do I compute that and compare them?" + } +] diff --git a/.agents/evals/academic-vqe-qaoa/tests/test_measure_parsers.py b/.agents/evals/academic-vqe-qaoa/tests/test_measure_parsers.py new file mode 100644 index 000000000..e12adf9da --- /dev/null +++ b/.agents/evals/academic-vqe-qaoa/tests/test_measure_parsers.py @@ -0,0 +1,105 @@ +#!/usr/bin/env python3 +"""Unit tests for measure_tokens.py parsers against real CLI sample payloads. + +These fixtures mirror the documented headless output of each agent CLI, so the +token extraction is verified without needing any CLI installed. +""" + +import json +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parent.parent)) + +from measure_tokens import ( # noqa: E402 + parse_claude_json, parse_codex_jsonl, parse_cursor_json, record_meter, +) + + +def test_claude_real_usage(): + sample = json.dumps({ + "result": "answer text", + "session_id": "s1", + "total_cost_usd": 0.0079825, + "usage": { + "input_tokens": 3, + "output_tokens": 6, + "cache_read_input_tokens": 15635 + }, + }) + text, usage, cost = parse_claude_json(sample) + assert text == "answer text" + assert usage == {"input_tokens": 3, "output_tokens": 6} + assert cost == 0.0079825 + + +def test_codex_jsonl_usage_and_text(): + sample = "\n".join([ + '{"type":"thread.started","thread_id":"t"}', + '{"type":"turn.started"}', + '{"type":"item.completed","item":{"id":"i","type":"agent_message",' + '"text":"final answer"}}', + '{"type":"turn.completed","usage":{"input_tokens":24763,' + '"cached_input_tokens":24448,"output_tokens":122,' + '"reasoning_output_tokens":0}}', + ]) + text, usage, cost = parse_codex_jsonl(sample) + assert text == "final answer" + assert usage == {"input_tokens": 24763, "output_tokens": 122} + assert cost is None + + +def test_codex_token_count_event_fallback(): + sample = "\n".join([ + '{"type":"item.completed","item":{"type":"agent_message","text":"x"}}', + '{"type":"event_msg","payload":{"type":"token_count","info":' + '{"total_token_usage":{"input_tokens":8408,"output_tokens":7}}}}', + ]) + _, usage, _ = parse_codex_jsonl(sample) + assert usage == {"input_tokens": 8408, "output_tokens": 7} + + +def test_cursor_with_usage(): + sample = json.dumps({ + "result": "hi", + "chatId": "abc", + "model": "gpt-5", + "usage": { + "input_tokens": 10, + "output_tokens": 4 + }, + }) + text, usage, _ = parse_cursor_json(sample) + assert text == "hi" + assert usage == {"input_tokens": 10, "output_tokens": 4} + + +def test_cursor_without_usage_is_unavailable(): + sample = json.dumps({"result": "hi", "chatId": "x", "model": "m"}) + _, usage, _ = parse_cursor_json(sample) + assert usage is None + + +def test_record_meter_totals_and_source(): + rec = record_meter("P5", "claude", ("txt", { + "input_tokens": 100, + "output_tokens": 50 + }, 0.01)) + assert rec["tokens"] == 150 + assert rec["tokens_source"] == "meter" + assert rec["cost_usd"] == 0.01 + + +def test_record_meter_unavailable_when_no_usage(): + rec = record_meter("P5", "cursor", ("txt", None, None)) + assert rec["tokens"] is None + assert rec["tokens_source"] == "unavailable" + assert "cost_usd" not in rec + + +if __name__ == "__main__": + fns = [v for k, v in sorted(globals().items()) if k.startswith("test_")] + for fn in fns: + fn() + print(f" [ok] {fn.__name__}") + print(f"{len(fns)} passed") diff --git a/.agents/evals/academic-vqe-qaoa/tests/test_token_sum.py b/.agents/evals/academic-vqe-qaoa/tests/test_token_sum.py new file mode 100644 index 000000000..2764affcf --- /dev/null +++ b/.agents/evals/academic-vqe-qaoa/tests/test_token_sum.py @@ -0,0 +1,94 @@ +"""Self-test for evaluate_metrics token summing.""" +import json +import sys +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from evaluate_metrics import load_assertions, normalize_run, summarize + + +def _make_run(tmp_path, tokens_field): + payload = { + "agent": + "test", + "model": + "test-model", + "config": + "with_skill", + "responses": [{ + "id": "P1", + "response": "cobyla requires gradients scipy L-BFGS-B jac", + "tokens": tokens_field, + }], + } + path = tmp_path / "run.json" + path.write_text(json.dumps(payload)) + return path + + +def _make_minimal_assertions(tmp_path): + spec = { + "P1": { + "must_include": [ + "cobyla", "requires gradients", "scipy", "L-BFGS-B", "jac" + ], + "must_not_include": [], + } + } + path = tmp_path / "assertions.json" + path.write_text(json.dumps(spec)) + return path + + +def test_int_tokens(tmp_path): + run_path = _make_run(tmp_path, 123) + asserts_path = _make_minimal_assertions(tmp_path) + summary = summarize(normalize_run(run_path), load_assertions(asserts_path)) + assert summary["tokens_total"] == 123 + + +def test_dict_tokens(tmp_path): + run_path = _make_run(tmp_path, {"input": 100, "output": 50}) + asserts_path = _make_minimal_assertions(tmp_path) + summary = summarize(normalize_run(run_path), load_assertions(asserts_path)) + assert summary["tokens_total"] == 150 + + +def test_missing_tokens(tmp_path): + run_path = _make_run(tmp_path, None) + asserts_path = _make_minimal_assertions(tmp_path) + summary = summarize(normalize_run(run_path), load_assertions(asserts_path)) + assert summary["tokens_total"] is None + + +def test_dict_all_nonnumeric(tmp_path): + """Dict with no numeric values should yield None, matching missing/unknown.""" + run_path = _make_run(tmp_path, {"input": "abc", "output": None}) + asserts_path = _make_minimal_assertions(tmp_path) + summary = summarize(normalize_run(run_path), load_assertions(asserts_path)) + assert summary["tokens_total"] is None + + +def test_dict_all_numeric_zero(tmp_path): + """A dict of numeric zeros is a legitimate 'zero tokens' run, not unparseable.""" + run_path = _make_run(tmp_path, {"input": 0, "output": 0}) + asserts_path = _make_minimal_assertions(tmp_path) + summary = summarize(normalize_run(run_path), load_assertions(asserts_path)) + assert summary["tokens_total"] == 0 + + +if __name__ == "__main__": + import tempfile + with tempfile.TemporaryDirectory() as d: + test_int_tokens(Path(d)) + with tempfile.TemporaryDirectory() as d: + test_dict_tokens(Path(d)) + with tempfile.TemporaryDirectory() as d: + test_missing_tokens(Path(d)) + with tempfile.TemporaryDirectory() as d: + test_dict_all_nonnumeric(Path(d)) + with tempfile.TemporaryDirectory() as d: + test_dict_all_numeric_zero(Path(d)) + print("all token-sum tests passed") diff --git a/.agents/skills/cudaq-academic-vqe-qaoa/SKILL.md b/.agents/skills/cudaq-academic-vqe-qaoa/SKILL.md new file mode 100644 index 000000000..85dda94be --- /dev/null +++ b/.agents/skills/cudaq-academic-vqe-qaoa/SKILL.md @@ -0,0 +1,97 @@ +--- +name: cudaq-academic-vqe-qaoa +description: Academic workshop workflow for CUDA-QX Solvers. Use when the user asks for beginner-friendly CUDA-Q Solvers installation, VQE examples, QAOA examples, MaxCut with QAOA, a first ADAPT-VQE example, or a first GQE (generative / GPT-style eigensolver) example. Do not use for QEC, Ising-specific material, advanced chemistry active-space setup, custom operator pools, or CUDA-QX source-build debugging. +--- + +# CUDA-QX Academic VQE/QAOA + +Use this skill to answer workshop-style questions that move from installing +CUDA-QX Solvers to running simple VQE and QAOA examples. Keep responses short, +teachable, and grounded in repo APIs. + +## Workflow + +1. Identify the user intent. +2. Load exactly one reference file unless the user asks for comparison. +3. Answer with a minimal runnable path first, then mention the source files for + users who want to inspect the implementation. +4. When evaluating before/after behavior, use the deterministic evaluator in + `.agents/evals/academic-vqe-qaoa/`. + +## Recognizing the problem + +If the user describes a problem in domain terms (without naming an algorithm), +first map it to a quantum problem class, then route below: + +| User describes... | Problem class | Map to | +| --- | --- | --- | +| Splitting / partitioning / grouping a set where pairwise costs matter (dividing delivery stops, clustering, team assignment) | Graph partition → **MaxCut** | QAOA (`references/qaoa.md`) | +| Finding the lowest-energy / most stable configuration of a molecule | Ground state | VQE / ADAPT-VQE (`references/vqe.md`, `references/adapt.md`) | +| Any other "best discrete choice among many options" problem | Custom Hamiltonian | QAOA (`references/qaoa.md`) | + +Do not assume every optimization problem is MaxCut. If the problem does not fit +a class above, say so rather than forcing a fit. + +## Intent Routing + +| User intent | Read | +| --- | --- | +| Install or smoke test CUDA-QX Solvers | `references/install.md` | +| Build a first VQE example | `references/vqe.md` | +| Build a first QAOA or MaxCut example | `references/qaoa.md` | +| Build a first ADAPT-VQE example | `references/adapt.md` | +| Build a first GQE (GPT-style eigensolver) example | `references/gqe.md` | + +## Response Contract + +For install questions, include: + +- the provided Brev environment as the recommended workshop path +- a note that CPU execution is acceptable for the small VQE/QAOA learning examples +- `pip install cudaq-solvers` +- an import smoke test for `cudaq` and `cudaq_solvers` +- the `libgfortran` note for classical optimizers + +For VQE questions, include: + +- `cudaq.kernel` +- `cudaq.spin` +- `solvers.vqe` +- initial parameters +- optimizer/gradient guidance + +For QAOA questions, include: + +- `networkx` +- `solvers.get_maxcut_hamiltonian` +- `solvers.get_num_qaoa_parameters` +- `solvers.qaoa` +- non-empty initial parameters +- `optimizer="cobyla"` as the beginner-safe default + +For ADAPT-VQE questions, include: + +- `solvers.create_molecule` +- `solvers.get_operator_pool` +- a named pool such as `spin_complement_gsd` with `num_orbitals=` +- a `@cudaq.kernel` `initState` that prepares Hartree-Fock +- the unpacked return `energy, thetas, ops = solvers.adapt_vqe(...)` + +For GQE questions, include: + +- `pip install cudaq-solvers[gqe]` as the required extra +- the ImportError fallback note (bare `cudaq-solvers` is not enough) +- `from cudaq_solvers.gqe_algorithm.gqe import get_default_config` +- a `cost(sampled_ops, **kwargs)` callback signature +- the unpacked return `energy, indices = solvers.gqe(cost, pool, config=cfg)` + +## Source Of Truth + +Prefer these repo files for API details: + +- `docs/sphinx/quickstart/installation.rst` +- `docs/sphinx/examples/solvers/python/uccsd_vqe.py` +- `docs/sphinx/examples/solvers/python/molecular_docking_qaoa.py` +- `libs/solvers/python/tests/test_vqe.py` +- `libs/solvers/python/tests/test_qaoa.py` +- `libs/solvers/python/bindings/solvers/py_solvers.cpp` diff --git a/.agents/skills/cudaq-academic-vqe-qaoa/references/adapt.md b/.agents/skills/cudaq-academic-vqe-qaoa/references/adapt.md new file mode 100644 index 000000000..7dfe7199a --- /dev/null +++ b/.agents/skills/cudaq-academic-vqe-qaoa/references/adapt.md @@ -0,0 +1,68 @@ +# ADAPT-VQE (Minimal Recipe) + +Use this reference when the user asks for a minimal ADAPT-VQE example in CUDA-Q +Solvers, especially around **operator pool selection** and **initial state +kernel**. Source of truth: `libs/solvers/python/tests/test_adapt.py` and +`docs/sphinx/examples/solvers/cpp/adapt_h2.cpp` (Python equivalent uses the +same flow). + +## API surface + +| Symbol | Notes | +| --- | --- | +| `solvers.create_molecule(geometry, basis, charge, spin, casci=True)` | Build a `MolecularHamiltonian` (needs PySCF) | +| `solvers.get_operator_pool(name, **kwargs)` | Pool names include `uccsd`, `spin_complement_gsd`, `uccgsd`, `upccgsd`, `ceo` | +| `solvers.adapt_vqe(initState, hamiltonian, operators, **options)` | Returns `(energy, thetas, ops)` | + +## Minimal Python recipe (H2, STO-3G) + +```python +import numpy as np +import cudaq +from cudaq import spin +import cudaq_solvers as solvers + +geometry = [('H', (0., 0., 0.)), ('H', (0., 0., .7474))] +molecule = solvers.create_molecule(geometry, 'sto-3g', 0, 0, casci=True) + +operators = solvers.get_operator_pool( + "spin_complement_gsd", num_orbitals=molecule.n_orbitals +) + +numElectrons = molecule.n_electrons + +@cudaq.kernel +def initState(q: cudaq.qview): + for i in range(numElectrons): + x(q[i]) + +energy, thetas, ops = solvers.adapt_vqe( + initState, molecule.hamiltonian, operators +) +print(f"Energy = {energy}") +``` + +Expected energy for H2/STO-3G is approximately `-1.137 Ha`. + +## Mandatory beginner footguns + +- The pool is **not** auto-generated. You must call `solvers.get_operator_pool(...)` + and pass the result. Calling `solvers.adapt_vqe(initState, hamiltonian)` with + no pool will fail. +- `initState` must be a `@cudaq.kernel` that prepares a non-trivial reference + state (typically Hartree-Fock: apply `x(q[i])` for `i in range(n_electrons)`). + An empty kernel leaves the state at |0...0> and ADAPT will pick uninformative + operators. +- `get_operator_pool` requires the pool's expected kwargs. `spin_complement_gsd` + needs `num_orbitals=molecule.n_orbitals`. `uccsd` needs additionally + `num_electrons=molecule.n_electrons`. +- Pass advanced tuning via `options=` (max_iter, grad_norm_tolerance, + threshold_energy, initial_theta, verbose, shots). Defaults are sane for a + workshop demo; do not over-tune. + +## When to recommend ADAPT-VQE vs VQE + +- VQE with UCCSD: fastest path, fixed ansatz. +- ADAPT-VQE: adaptively picks operators from a pool, deeper but more accurate + on harder molecules where you don't know the right ansatz. +- GQE: GPT-like generative search over a pool. See `references/gqe.md`. diff --git a/.agents/skills/cudaq-academic-vqe-qaoa/references/gqe.md b/.agents/skills/cudaq-academic-vqe-qaoa/references/gqe.md new file mode 100644 index 000000000..09c2c0fe5 --- /dev/null +++ b/.agents/skills/cudaq-academic-vqe-qaoa/references/gqe.md @@ -0,0 +1,98 @@ +# GQE (Generative Quantum Eigensolver) + +Use this reference when the user asks about the GPT-style / transformer-based +quantum eigensolver in CUDA-Q Solvers. Source of truth: +`libs/solvers/python/tests/test_gqe.py`, `docs/sphinx/examples/solvers/python/gqe_h2.py`, +and `libs/solvers/python/cudaq_solvers/__init__.py` (ImportError fallback). + +## Install + +GQE is an **optional extra**. You must install the `[gqe]` extra to pull in +torch / transformers / lightning: + +```bash +pip install cudaq-solvers[gqe] +``` + +Without it, `solvers.gqe(...)` raises a stub ImportError pointing at the same +command (see `cudaq_solvers/__init__.py`). The `cudaq-solvers` install on its +own (no extras) does **not** include GQE. + +GQE training benefits from CUDA-enabled PyTorch. Without it, several tests are +marked `requires_cuda_kernels` and are skipped. The workshop demo can still +import GQE on a CPU box, but training will be slow. + +## API surface + +| Symbol | Notes | +| --- | --- | +| `from cudaq_solvers.gqe_algorithm.gqe import get_default_config` | Returns a config dataclass with sane defaults | +| `solvers.gqe(cost, pool, config=cfg, ...)` | Returns `(energy, indices)`; `indices` are pool positions of the chosen ansatz | +| `cost(sampled_ops, **kwargs)` | User-defined callback; should return a real expectation value | + +## Minimal recipe (Z0 + Z1) + +```python +import cudaq +from cudaq import spin +import cudaq_solvers as solvers +from cudaq_solvers.gqe_algorithm.gqe import get_default_config + +qubit_count = 2 +ham = spin.z(0) + spin.z(1) + +def ops_pool(n): + pool = [] + for i in range(n): + pool.append(cudaq.SpinOperator(spin.x(i))) + pool.append(cudaq.SpinOperator(spin.y(i))) + pool.append(cudaq.SpinOperator(spin.z(i))) + for i in range(n - 1): + pool.append(cudaq.SpinOperator(spin.z(i) * spin.z(i + 1))) + return pool + +pool = ops_pool(qubit_count) + +def term_coefficients(op): return [t.evaluate_coefficient() for t in op] +def term_words(op): return [t.get_pauli_word(qubit_count) for t in op] + +@cudaq.kernel +def kernel(qcount: int, coeffs: list[float], words: list[cudaq.pauli_word]): + q = cudaq.qvector(qcount) + h(q) + for i in range(len(coeffs)): + exp_pauli(coeffs[i], q, words[i]) + +def cost(sampled_ops, **kwargs): + full_coeffs, full_words = [], [] + for op in sampled_ops: + full_coeffs += [c.real for c in term_coefficients(op)] + full_words += term_words(op) + return cudaq.observe(kernel, ham, qubit_count, full_coeffs, full_words).expectation() + +cfg = get_default_config() +cfg.num_samples = 5 +cfg.max_iters = 25 +cfg.ngates = 4 +cfg.seed = 3047 +cfg.lr = 1e-6 + +energy, indices = solvers.gqe(cost, pool, config=cfg) +print(energy, indices) +``` + +## Mandatory beginner footguns + +- Without `pip install cudaq-solvers[gqe]`, `solvers.gqe(...)` raises + `ImportError: Failed to load GQE solver due to missing dependencies.` Do + **not** tell users plain `pip install cudaq-solvers` is enough. +- `cost` receives `sampled_ops: list[SpinOperator]` plus `**kwargs`. You must + accept `**kwargs` or older training paths will fail when they pass + `qpu_id` for MQPU. +- `config` is a dataclass from `cudaq_solvers.gqe_algorithm.gqe.get_default_config()` + — do not invent your own. Required positive-valued fields: + `num_samples > 0`, `lr > 0`, `temperature > 0` (validated, see + `test_gqe.py::test_invalid_inputs`). +- Return tuple is `(energy, indices)`, where `indices` are positions inside + the pool you passed. Not parameters, not operators. +- Optional `loss="gflow"` switches to GFlow-style loss. diff --git a/.agents/skills/cudaq-academic-vqe-qaoa/references/install.md b/.agents/skills/cudaq-academic-vqe-qaoa/references/install.md new file mode 100644 index 000000000..ad3e53a92 --- /dev/null +++ b/.agents/skills/cudaq-academic-vqe-qaoa/references/install.md @@ -0,0 +1,102 @@ +# Install And Smoke Test + +Use this for beginner installation questions before VQE or QAOA. + +## Recommended Workshop Path + +For students in a workshop, start from the provided Brev environment when one +is available. This avoids spending class time on local CUDA, Linux, driver, or +compiler setup and gives everyone the same baseline. + +GPU acceleration is useful for larger experiments, but it is not required for +the small VQE and QAOA learning examples in this skill. Students can validate +the install and prototype the examples on CPU, then use the provided Brev/GPU +setup when the class experiment needs acceleration or a standardized runtime. + +Inside the Brev environment, install the Solvers package: + +```bash +python3 -m pip install cudaq-solvers +``` + +Then verify the two imports the examples need: + +```bash +python3 - <<'PY' +import cudaq +import cudaq_solvers as solvers +print("cudaq:", cudaq.__name__) +print("cudaq_solvers:", solvers.__name__) +PY +``` + +If those imports work, students are ready to run the small VQE and QAOA +examples. + +## Local Linux Path + +For students using their own Linux machine, including CPU-only machines: + +```bash +python3 -m pip install cudaq-solvers +``` + +Then verify the two imports the examples need: + +```bash +python3 - <<'PY' +import cudaq +import cudaq_solvers as solvers +print("cudaq:", cudaq.__name__) +print("cudaq_solvers:", solvers.__name__) +PY +``` + +If the user wants both CUDA-QX libraries, use: + +```bash +python3 -m pip install cudaq-qec cudaq-solvers +``` + +Do not suggest `cudaq-solvers[gqe]` for this academic VQE/QAOA path. The +`[gqe]` extra pulls in PyTorch-oriented dependencies for Generative Quantum +Eigensolver workflows, which are out of scope here. + +## Common Install Note + +CUDA-Q Solvers uses classical optimizers. On Linux, missing `libgfortran` can +break optimizer-backed workflows. On Debian-style systems: + +```bash +sudo apt-get install gfortran +``` + +## Docker Path + +For Mac, Windows, or anyone who cannot use Brev but wants a prebuilt +environment: + +```bash +docker pull ghcr.io/nvidia/cudaqx +docker run --gpus all -it ghcr.io/nvidia/cudaqx +``` + +Omit `--gpus all` if the machine has no NVIDIA GPU. + +## Quick Decision Guide + +- Workshop student: use the provided Brev environment; CPU is fine for the + small examples, and GPU is helpful for larger class experiments. +- Linux machine: use the local pip path, even on CPU-only machines, and install + `libgfortran`/`gfortran` if optimizer-backed workflows fail. +- Mac or Windows: prefer Brev; use Docker only if the user is already + comfortable with containers. +- No GPU: run the small workshop examples on CPU; use Brev when the workshop + needs the standard class setup. + +## Source Paths + +- Installation docs: `docs/sphinx/quickstart/installation.rst` +- Solvers package config: `libs/solvers/pyproject.toml.cu12`, + `libs/solvers/pyproject.toml.cu13` +- Wheel validation: `scripts/ci/test_wheels.sh` diff --git a/.agents/skills/cudaq-academic-vqe-qaoa/references/qaoa.md b/.agents/skills/cudaq-academic-vqe-qaoa/references/qaoa.md new file mode 100644 index 000000000..90562a238 --- /dev/null +++ b/.agents/skills/cudaq-academic-vqe-qaoa/references/qaoa.md @@ -0,0 +1,67 @@ +# Minimal QAOA / MaxCut Path + +Use this for a first QAOA example. Keep the answer anchored on MaxCut because +the helper API is easy to explain and easy to test. + +## Teaching Example + +```python +import numpy as np +import networkx as nx +import cudaq_solvers as solvers + +graph = nx.Graph() +graph.add_weighted_edges_from([ + (0, 1, 1.0), + (1, 2, 2.0), + (0, 2, 0.5), +]) + +hamiltonian = solvers.get_maxcut_hamiltonian(graph) +num_layers = 1 +num_parameters = solvers.get_num_qaoa_parameters(hamiltonian, num_layers) +initial_parameters = np.zeros(num_parameters) + +result = solvers.qaoa( + hamiltonian, + num_layers, + initial_parameters, + optimizer="cobyla", +) + +optimal_value, optimal_parameters, sample_result = result +print("MaxCut value:", -optimal_value) +print("Best bitstring:", sample_result.most_probable()) +print("Parameters:", optimal_parameters) +``` + +## Beginner Defaults + +- Use a NetworkX graph for MaxCut. +- Use `solvers.get_maxcut_hamiltonian(graph)`. +- Use `solvers.get_num_qaoa_parameters(...)` instead of guessing parameter + count. For standard MaxCut QAOA this returns `2 * num_layers` (one γ and + one β per layer). +- Use non-empty initial parameters. +- Use `optimizer="cobyla"` as the safe beginner default. +- `QAOAResult` can be tuple-unpacked as + `(optimal_value, optimal_parameters, sample_result)`, or accessed by + attribute: `result.optimal_value`, `result.optimal_parameters`, + `result.optimal_config` (the latter is the `cudaq.SampleResult` produced + by the final shot — call `.most_probable()` on it to get the MaxCut + bitstring). +- QAOA minimizes the Hamiltonian. For MaxCut, print `-optimal_value` as the + cut value. + +## Pitfall + +`optimizer="lbfgs"` requires gradients. The QAOA path does not provide a +gradient instance by default, so beginners should use `cobyla` unless they are +passing a compatible SciPy optimizer with a `jac=`. + +## Source Paths + +- Python example: `docs/sphinx/examples/solvers/python/molecular_docking_qaoa.py` +- Python tests: `libs/solvers/python/tests/test_qaoa.py` +- C++ API: `libs/solvers/include/cudaq/solvers/qaoa.h` +- Python bindings: `libs/solvers/python/bindings/solvers/py_solvers.cpp` diff --git a/.agents/skills/cudaq-academic-vqe-qaoa/references/vqe.md b/.agents/skills/cudaq-academic-vqe-qaoa/references/vqe.md new file mode 100644 index 000000000..90446aecc --- /dev/null +++ b/.agents/skills/cudaq-academic-vqe-qaoa/references/vqe.md @@ -0,0 +1,62 @@ +# Minimal VQE Path + +Use this for a first VQE example. Keep the answer focused on the algorithm +shape, not chemistry setup. + +## Teaching Example + +```python +import cudaq +from cudaq import spin +import cudaq_solvers as solvers + + +@cudaq.kernel +def ansatz(theta: float): + q = cudaq.qvector(2) + x(q[0]) + ry(theta, q[1]) + x.ctrl(q[1], q[0]) + + +hamiltonian = ( + 5.907 + - 2.1433 * spin.x(0) * spin.x(1) + - 2.1433 * spin.y(0) * spin.y(1) + + 0.21829 * spin.z(0) + - 6.125 * spin.z(1) +) + +energy, params, history = solvers.vqe( + lambda thetas: ansatz(thetas[0]), + hamiltonian, + [0.0], + optimizer="lbfgs", + gradient="parameter_shift", + tol=1e-7, +) + +print("energy:", energy) +print("params:", params) +``` + +## Beginner Defaults + +- Use a non-empty initial parameter list. +- Use `optimizer="lbfgs"` with `gradient="parameter_shift"`. +- Or omit optimizer/gradient and let the default optimizer path run. +- Return value is `(energy, params, history)`. + +## Self Check + +- The ansatz argument count matches how `solvers.vqe` calls it. +- Initial parameters are not empty. +- Gradient-based optimizers have a gradient setting. +- The answer mentions `cudaq.kernel`, `spin`, and `solvers.vqe`. + +## Source Paths + +- Python example: `docs/sphinx/examples/solvers/python/uccsd_vqe.py` +- Python tests: `libs/solvers/python/tests/test_vqe.py` +- C++ API: `libs/solvers/include/cudaq/solvers/vqe.h` +- Python bindings: `libs/solvers/python/bindings/solvers/py_solvers.cpp` diff --git a/.gitignore b/.gitignore index 2026b8945..5426ab63b 100644 --- a/.gitignore +++ b/.gitignore @@ -119,3 +119,6 @@ libs/*/pyproject.toml # This file is cloned from the qec dir; do not commit after cloning. libs/solvers/python/metapackages/setup.py + +# Academic agent skill evaluation runtime artifacts. +.agents/evals/academic-vqe-qaoa/runs/ diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..bf363fc61 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,25 @@ +# CUDA-QX Agent Skills + +This branch carries a small academic-facing skill slice for CUDA-QX Solvers. +It is intentionally independent from the larger skills architecture PR. + +## Skills + +| Skill | Purpose | +| --- | --- | +| `cudaq-academic-vqe-qaoa` | Basic install, VQE, and QAOA workflows for academic workshop examples | + +The skill source lives under `.agents/skills/`. Evaluation prompts, assertions, +and lightweight metrics tooling live under `.agents/evals/academic-vqe-qaoa/`. + +## Scope + +Keep this branch focused on: + +- installing and smoke-testing `cudaq-solvers` +- a minimal VQE workflow +- a minimal QAOA / MaxCut workflow +- objective before/after metrics for agent responses + +Avoid expanding this PR into QEC, GQE, chemistry active-space design, custom +operators, or the full multi-agent mirror infrastructure.