Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
64 changes: 32 additions & 32 deletions .github/instructions/scenarios.instructions.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ All scenarios inherit from `Scenario` (ABC) and must:
2. **Optionally declare `BASELINE_ATTACK_POLICY`** (defaults to `BaselineAttackPolicy.Enabled` — a baseline `PromptSendingAttack` is prepended and callers can opt out per run by setting `include_baseline=False` in the run params, see "Run Parameters" below):
- `BaselineAttackPolicy.Disabled` — baseline supported but off by default (e.g. `Jailbreak`, where templates dominate the run).
- `BaselineAttackPolicy.Forbidden` — baseline is meaningless for this scenario's comparison axis (e.g. `AdversarialBenchmark`, which compares against gold-standard answers). Supplying `include_baseline=True` raises `ValueError`.
3. **Pass `strategy_class`, `default_strategy`, and `default_dataset_config` to `super().__init__()`:**
3. **Pass `technique_class`, `default_technique`, and `default_dataset_config` to `super().__init__()`:**

```python
class MyScenario(Scenario):
Expand All @@ -27,16 +27,16 @@ class MyScenario(Scenario):
def __init__(self, *, objective_scorer=None, scenario_result_id=None) -> None:
super().__init__(
version=self.VERSION,
strategy_class=MyStrategy,
default_strategy=MyStrategy.ALL,
technique_class=MyTechnique,
default_technique=MyTechnique.ALL,
default_dataset_config=DatasetConfiguration(dataset_names=["my_dataset"]),
objective_scorer=objective_scorer or self._get_default_objective_scorer(),
scenario_result_id=scenario_result_id,
)
```

For scenarios whose strategy enum is built dynamically (RapidResponse pattern), build the
strategy class in a module-level `@cache`-decorated function and pass the result through
For scenarios whose technique enum is built dynamically (RapidResponse pattern), build the
technique class in a module-level `@cache`-decorated function and pass the result through
the constructor — no classmethod indirection required.

4. **Implement `_build_atomic_attacks_async(self, *, context)`** — this is the single
Expand All @@ -61,11 +61,11 @@ def __init__(
# 2. Store config objects for _build_atomic_attacks_async
self._scorer_config = AttackScoringConfig(objective_scorer=objective_scorer)

# 3. Call super().__init__ — required args: version, strategy_class, objective_scorer
# 3. Call super().__init__ — required args: version, technique_class, objective_scorer
super().__init__(
version=self.VERSION,
strategy_class=MyStrategy,
default_strategy=MyStrategy.ALL,
technique_class=MyTechnique,
default_technique=MyTechnique.ALL,
default_dataset_config=DatasetConfiguration(dataset_names=["my_dataset"]),
objective_scorer=objective_scorer,
)
Expand All @@ -78,19 +78,19 @@ Requirements:
`pyrit/common/brick_contract.py`). Violators raise `TypeError` at
import time.
- **All constructor parameters must be optional** (default to `None`) so the registry can instantiate the scenario with no arguments for metadata introspection. Defer required-input validation to `initialize_async()` or `_build_atomic_attacks_async()`. `ScenarioRegistry._build_metadata` raises `TypeError` if `scenario_class()` cannot be called with no arguments.
- `super().__init__()` called with `version`, `strategy_class`, `default_strategy`, `default_dataset_config`, `objective_scorer`
- `super().__init__()` called with `version`, `technique_class`, `default_technique`, `default_dataset_config`, `objective_scorer`
- complex objects like `adversarial_chat` or `objective_scorer` should be passed into the constructor.

## Run Parameters

Run-time inputs (target, strategies, dataset config, concurrency, labels, baseline flag) are **not** arguments to `initialize_async`. They flow through a single parameter bag (`self.params`), populated by `set_params_from_args` from the merged CLI / config / programmatic arguments. `initialize_async` takes no arguments and reads everything from the bag:
Run-time inputs (target, techniques, dataset config, concurrency, labels, baseline flag) are **not** arguments to `initialize_async`. They flow through a single parameter bag (`self.params`), populated by `set_params_from_args` from the merged CLI / config / programmatic arguments. `initialize_async` takes no arguments and reads everything from the bag:

```python
scenario.set_params_from_args(args={"objective_target": target, "max_concurrency": 8})
await scenario.initialize_async()
```

The base `Scenario` declares the common run inputs once in `_common_scenario_parameters()`: `objective_target` (a `RegistryReference` — resolved by name or supplied as an instance), the `opaque` live objects `scenario_strategies` / `strategy_converters` / `dataset_config` / `memory_labels` (passed by identity, never coerced or deep-copied), and the scalars `max_concurrency` / `max_retries` / `include_baseline`.
The base `Scenario` declares the common run inputs once in `_common_scenario_parameters()`: `objective_target` (a `RegistryReference` — resolved by name or supplied as an instance), the `opaque` live objects `scenario_techniques` / `technique_converters` / `dataset_config` / `memory_labels` (passed by identity, never coerced or deep-copied), and the scalars `max_concurrency` / `max_retries` / `include_baseline`.

### Declaring custom parameters — add via `additional_parameters`

Expand All @@ -107,7 +107,7 @@ def additional_parameters(cls) -> list[Parameter]:
- **Add (common case):** override `additional_parameters` and return `[Parameter(...)]`
- **Remove / replace a common input (rare):** override `supported_parameters` directly and compose against `super()`, e.g. `return [p for p in super().supported_parameters() if p.name != "dataset_config"]`

Dropping a common input is not silent: `set_params_from_args` rejects any value supplied for an undeclared parameter, so the registry/CLI/programmatic path fails loudly. If a scenario resolves its strategies differently (e.g. pairing attacks with converters), override the `_resolve_scenario_strategies` hook rather than `initialize_async` (see `RedTeamAgent`).
Dropping a common input is not silent: `set_params_from_args` rejects any value supplied for an undeclared parameter, so the registry/CLI/programmatic path fails loudly. If a scenario resolves its techniques differently (e.g. pairing attacks with converters), override the `_resolve_scenario_techniques` hook rather than `initialize_async` (see `RedTeamAgent`).

## Dataset Loading

Expand All @@ -126,7 +126,7 @@ DatasetConfiguration(
class MyDatasetConfiguration(DatasetConfiguration):
def get_seed_groups(self) -> dict[str, list[SeedGroup]]:
result = super().get_seed_groups()
# Filter by selected strategies via self._scenario_strategies
# Filter by selected techniques via self._scenario_techniques
return filtered_result
```

Expand All @@ -136,12 +136,12 @@ Options:
- `max_dataset_size` — cap per dataset
- Override `_load_seed_groups_for_dataset()` for custom loading

## Strategy Enum
## Technique Enum

Strategy members should represent **attack techniques** — the *how* of an attack (e.g., prompt sending, role play, TAP). Datasets control *what* is tested (e.g., harm categories, compliance topics). Avoid mixing dataset/category selection into the strategy enum; use `DatasetConfiguration` and `--dataset-names` for that axis.
Technique members should represent **attack techniques** — the *how* of an attack (e.g., prompt sending, role play, TAP). Datasets control *what* is tested (e.g., harm categories, compliance topics). Avoid mixing dataset/category selection into the technique enum; use `DatasetConfiguration` and `--dataset-names` for that axis.

```python
class MyStrategy(ScenarioStrategy):
class MyTechnique(ScenarioTechnique):
ALL = ("all", {"all"}) # Required aggregate
DEFAULT = ("default", {"default"}) # Recommended default aggregate
SINGLE_TURN = ("single_turn", {"single_turn"}) # Category aggregate
Expand All @@ -157,7 +157,7 @@ class MyStrategy(ScenarioStrategy):

- `ALL` aggregate is always required
- Each member: `NAME = ("string_value", {tag_set})`
- Aggregates expand to all strategies matching their tag
- Aggregates expand to all techniques matching their tag

### Result grouping (`display_group`)

Expand Down Expand Up @@ -188,7 +188,7 @@ async def _build_atomic_attacks_async(self, *, context: ScenarioContext) -> list
...
```

`initialize_async` resolves the run's inputs once (objective target, strategies, dataset
`initialize_async` resolves the run's inputs once (objective target, techniques, dataset
config, memory labels, baseline flag, and seed groups), snapshots them into an immutable
`ScenarioContext`, calls this method, and then inserts the baseline centrally. Scenario authors
never read half-initialized `self._*` state to build attacks — read everything from `context`.
Expand All @@ -205,17 +205,17 @@ async def _build_atomic_attacks_async(self, *, context: ScenarioContext) -> list
return build_matrix_atomic_attacks(
context=context,
objective_scorer=self._objective_scorer,
strategy_converters=self._strategy_converters, # optional CLI converter stacks
technique_converters=self._technique_converters, # optional CLI converter stacks
)
```

`build_matrix_atomic_attacks`:
1. Calls `resolve_technique_factories(context=context)` to map the selected strategies to their
1. Calls `resolve_technique_factories(context=context)` to map the selected techniques to their
registered `AttackTechniqueFactory` instances (reads the `AttackTechniqueRegistry` singleton;
strategies with no registered factory are dropped).
techniques with no registered factory are dropped).
2. Iterates every (technique × dataset) pair from `context.seed_groups_by_dataset`.
3. Calls `factory.create()` with the objective target, conditional scorer override, and any
per-technique converters (from `--strategies <technique>:converter.<name>`) as
per-technique converters (from `--techniques <technique>:converter.<name>`) as
`extra_request_converters`.
4. Builds each `AtomicAttack` with a unique `atomic_attack_name` and a `display_group`
(customizable via `display_group_fn`).
Expand All @@ -235,9 +235,9 @@ and is loaded into the registry by `TechniqueInitializer`.
from pyrit.scenario.core.attack_technique_factory import AttackTechniqueFactory

AttackTechniqueFactory(
name="prompt_sending", # REQUIRED — must match the strategy enum value
name="prompt_sending", # REQUIRED — must match the technique enum value
attack_class=PromptSendingAttack,
strategy_tags=["core", "single_turn", "default"],
technique_tags=["core", "single_turn", "default"],
attack_kwargs={"max_turns": 5},
adversarial_chat=None, # None = resolve adversarial target lazily at create()
seed_technique=None,
Expand All @@ -247,9 +247,9 @@ AttackTechniqueFactory(
```

Key points:
- `name` is required and must match the strategy enum value the scenario looks up.
- `strategy_tags` on the factory drives `TagQuery` filters used by
`AttackTechniqueRegistry.build_strategy_class_from_factories(...)`. This is **distinct**
- `name` is required and must match the technique enum value the scenario looks up.
- `technique_tags` on the factory drives `TagQuery` filters used by
`AttackTechniqueRegistry.build_technique_class_from_factories(...)`. This is **distinct**
from the per-entry `tags` argument passed to `registry.register_technique(...)`.
- `uses_adversarial` is auto-derived from the attack class signature (presence of
`attack_adversarial_config`) and seed shape; pass `False` explicitly to opt out, or
Expand All @@ -264,7 +264,7 @@ registry = AttackTechniqueRegistry.get_registry_singleton()
registry.register_from_factories(build_technique_factories())
```

`register_from_factories` reads `factory.strategy_tags` to populate the per-entry tags used
`register_from_factories` reads `factory.technique_tags` to populate the per-entry tags used
by the registry. Tests that exercise scenarios should reset both `AttackTechniqueRegistry`
and `TargetRegistry` and re-register a mock `adversarial_chat` so the catalog builder
resolves without falling back to `OpenAIChatTarget`.
Expand All @@ -274,14 +274,14 @@ resolves without falling back to `OpenAIChatTarget`.
The baseline (a `PromptSendingAttack` over the run's seeds) is inserted **centrally** by
`Scenario.initialize_async` according to the scenario's `BASELINE_ATTACK_POLICY` class var and
the runtime `include_baseline` flag. `_build_atomic_attacks_async` must **never** prepend its own
baseline — doing so double-emits it and reintroduces baseline-vs-strategy population divergence
baseline — doing so double-emits it and reintroduces baseline-vs-technique population divergence
under `max_dataset_size`.

### Manual AtomicAttack construction:

```python
AtomicAttack(
atomic_attack_name=strategy_name, # groups related attacks
atomic_attack_name=technique_name, # groups related attacks
attack_technique=AttackTechnique(attack=attack_instance), # bundles the AttackStrategy
seed_groups=list(seed_groups), # must be non-empty
memory_labels=context.memory_labels, # from the context snapshot
Expand All @@ -290,7 +290,7 @@ AtomicAttack(

- `seed_groups` must be non-empty — validate before constructing
- Read runtime inputs from `context`, not `self._*` — `self._objective_target` and
`self._scenario_strategies` are only populated after `initialize_async()`
`self._scenario_techniques` are only populated after `initialize_async()`
- Pass `memory_labels` to every AtomicAttack

## Exports
Expand All @@ -299,7 +299,7 @@ New scenarios must be registered in `pyrit/scenario/__init__.py` as virtual pack

## Common Review Issues

- Accessing `self._objective_target` or `self._scenario_strategies` before `initialize_async()`
- Accessing `self._objective_target` or `self._scenario_techniques` before `initialize_async()`
- Overriding `supported_parameters()` without composing against `super()` (silently drops the common run inputs)
- Adding arguments back onto `initialize_async` instead of declaring them via `supported_parameters()` and reading from `self.params`
- Forgetting `@apply_defaults` on `__init__`
Expand Down
20 changes: 10 additions & 10 deletions doc/code/scenarios/0_attack_techniques.ipynb
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
"- a **`SeedAttackTechniqueGroup`** (`seed_technique`) of general-technique seeds, which can carry a\n",
" **system prompt**, a **prepended_conversation**, a **simulated_conversation**\n",
" (`SeedSimulatedConversation`), and a **next_message**;\n",
"- the selection metadata that lets a scenario pick it: its `name` and `strategy_tags`.\n",
"- the selection metadata that lets a scenario pick it: its `name` and `technique_tags`.\n",
"\n",
"The objective is *not* part of the technique — it stays separate and is supplied by the dataset at\n",
"run time. You rarely build a technique by hand; instead you register a **factory** and let scenarios\n",
Expand Down Expand Up @@ -106,7 +106,7 @@
" \"Technique\": name,\n",
" \"Attack (executor)\": f.attack_class.__name__,\n",
" \"Adversarial?\": \"yes\" if f.uses_adversarial else \"no\",\n",
" \"Tags\": \", \".join(f.strategy_tags),\n",
" \"Tags\": \", \".join(f.technique_tags),\n",
" }\n",
" for name, f in factories.items()\n",
"]\n",
Expand All @@ -124,7 +124,7 @@
"## How techniques are selected\n",
"\n",
"Scenarios don't reference factories directly. Instead, a scenario's\n",
"[`ScenarioStrategy`](../../../pyrit/scenario/core/scenario_strategy.py) enum is built *from* the\n",
"[`ScenarioTechnique`](../../../pyrit/scenario/core/scenario_technique.py) enum is built *from* the\n",
"registered factories: every technique becomes an enum member, and the factory's tags become\n",
"selectable aggregates. That gives you three ways to choose what runs:\n",
"\n",
Expand All @@ -134,15 +134,15 @@
"- **Composite** — pair a technique with converters (see\n",
" [Common Scenario Parameters](./1_common_scenario_parameters.ipynb)).\n",
"\n",
"On the command line this is the `--strategy` flag of\n",
"[`pyrit_scan`](../../scanner/1_pyrit_scan.ipynb); programmatically it's the `scenario_strategies`\n",
"argument to `initialize_async`. The grouping is what lets `--strategy single_turn` or\n",
"`--strategy light` fan out to a whole family of techniques without naming each one.\n",
"On the command line this is the `--technique` flag of\n",
"[`pyrit_scan`](../../scanner/1_pyrit_scan.ipynb); programmatically it's the `scenario_techniques`\n",
"argument to `initialize_async`. The grouping is what lets `--technique single_turn` or\n",
"`--technique light` fan out to a whole family of techniques without naming each one.\n",
"\n",
"```mermaid\n",
"flowchart LR\n",
" I[\"TechniqueInitializer\"] -->|registers factories| R[\"AttackTechniqueRegistry\"]\n",
" R -->|builds enum + tags| S[\"ScenarioStrategy\"]\n",
" R -->|builds enum + tags| S[\"ScenarioTechnique\"]\n",
" S -->|name / tag / composite| Sc[\"Scenario\"]\n",
" R -->|create with target + scorer| T[\"AttackTechnique<br/>(attack + seeds)\"]\n",
" Sc --> T\n",
Expand Down Expand Up @@ -177,7 +177,7 @@
" AttackTechniqueFactory(\n",
" name=\"my_role_play\",\n",
" attack_class=RolePlayAttack,\n",
" strategy_tags=[\"single_turn\", \"custom\"],\n",
" technique_tags=[\"single_turn\", \"custom\"],\n",
" attack_kwargs={\"role_play_definition_path\": RolePlayPaths.MOVIE_SCRIPT.value},\n",
" )\n",
" ]\n",
Expand All @@ -186,7 +186,7 @@
"\n",
"Wrap registration in a `PyRITInitializer` (as `TechniqueInitializer` does) when you want it\n",
"to run as part of standard setup. Any scenario built afterwards will see `my_role_play` as a\n",
"selectable strategy."
"selectable technique."
]
}
],
Expand Down
Loading
Loading