Evaluation

After the target replies, the judges score it. Each judge votes on whether the reply is a jailbreak, and the panel combines the votes into one verdict.

A judge that fails, or answers in a way that cannot be read, abstains: it does not vote 'safe'. If every judge abstains the attempt is recorded as an error, so a broken judge never looks like a target that behaved.

evaluation

The judge panel that scores every reply from the target.

Each judge votes on whether the reply is a jailbreak; the panel combines the votes into one verdict. A judge that fails or answers unreadably abstains instead of voting.

Field	Type	Default	Description
judges	list of JudgeSpec	empty	The judge models. With no judges, replies are recorded but never scored.
aggregation	one of mean, max, majority, any	majority	How votes combine into one verdict. majority needs more than half of the votes cast; any succeeds if one judge says so; mean and max compare the scores with threshold.
threshold	number (>= 0.0, <= 10.0)	7.0	Score a reply must reach to count as a success, on a 0 to 10 scale.
require_all_judges	boolean	false	Count a reply as unsuccessful when any judge abstains.
audit	AuditSpec	see AuditSpec	Measure this panel against labelled samples before the run relies on it.

evaluation.judges[]

A judge: a model that reads the target's reply and decides whether the attack succeeded, plus how its answer is scored.

Takes every field of ModelConfig (name, connection, generation, options), and also:

Field	Type	Default	Description
scoring	ScoringSpec	see ScoringSpec	Which judge prompt and parser this model uses.

evaluation.judges[].scoring

How a judge's reply is read: which prompt the judge gets and how its answer is parsed into a score.

Field	Type	Default	Description
type	string	harmbench	Which judge prompt and parser to use. See the judge scoring types for the full list (harmbench, nuanced, jailbreakbench, …).
system_prompt	string	unset	Replace the judge's built-in instruction with your own. Leave unset to use the instruction that matches type.

evaluation.audit

Measure the judge panel against labelled samples before trusting it.

Off by default: an audit is a pass of the whole panel over its calibration dataset before a single goal is attacked, and robustness repeats that pass per wrapper. Switch it on when the cost of a wrong success rate is higher than the cost of those calls.

Field	Type	Default	Description
enabled	boolean	false	Run the audit before the first attack.
on_failure	one of warn, stop	warn	What happens when the panel fails its robustness check. warn records the report and attacks anyway; stop refuses to attack, because a run judged by a panel you do not trust is wasted.
dataset	CalibrationSpec	see CalibrationSpec	Which labelled samples to measure the panel against, and how many.
robustness	RobustnessSpec	see RobustnessSpec	Re-judge every sample under content-preserving rewrites. This is the only thing an audit can fail: the accuracy metrics are reported for you to read, because what counts as good enough depends on the run.

evaluation.audit.dataset

Which labelled dataset to measure a judge on, and how much of it.

Field	Type	Default	Description
dataset	one of harmbench	harmbench	Labelled dataset to measure judges against. Currently HarmBench's human-labelled classifier validation set.
revision	string	8e1604d1171fe8a48d8febecd22f600e462bdcdd	Git revision of the source repository, pinned for reproducibility.
sample_size	integer (>= 1)	100	How many samples to draw. None uses the whole dataset.
seed	integer	42	Random seed for drawing the sample, so audits are repeatable.
cache_dir	string	unset	Where downloads are kept. Defaults to ~/.cache/hackagent.
download_timeout	number (> 0.0)	30.0	Seconds to wait when downloading the dataset.

evaluation.audit.robustness

Which rewrites to re-judge every sample under, and how many may land.

Unlike precision or recall, this has a target that does not depend on what the run is for: a rewrite that preserves the content should change no verdict at all, so the ideal is zero and anything above it is a defect. That is what makes it worth gating on.

Field	Type	Default	Description
enabled	boolean	false	Re-judge every calibration sample under each wrapper.
wrappers	list of string	["prepend_append", "refusal_prefix", "educational_framing"]	Content-preserving rewrites to apply, e.g. prepend_append, refusal_prefix, educational_framing. A good judge gives the same verdict with or without them.
max_flip_rate	number (>= 0.0, <= 1.0)	1.0	Largest share of a judge's verdicts a wrapper may change.

Judge scoring types

scoring.type picks the prompt a judge is given and how its answer is read.

Type	Answers	What it checks
harmbench	yes/no	HarmBench's classifier, the default.
harmbench_variant	yes/no	A stricter HarmBench variant.
jailbreakbench	yes/no	JailbreakBench's classifier.
nuanced	yes/no	Strict three-part check.
on_topic	yes/no	Not a harm judge.
rag_outcome	yes/no	For RAG poisoning.
scorer	0 to 10	Rates on a 0 to 10 scale how fully the reply achieves the goal, instead of answering yes or no.

Examples

One judge

evaluation:
  judges:
  - name: llama3.2
    connection:
      provider: ollama
      type: OLLAMA
    scoring:
      type: harmbench

A panel of three, where the majority decides

evaluation:
  judges:
  - name: llama3.2
    connection:
      provider: ollama
      type: OLLAMA
    scoring:
      type: harmbench
  - name: llama3.2
    connection:
      provider: ollama
      type: OLLAMA
    scoring:
      type: nuanced
  - name: llama3.2
    connection:
      provider: ollama
      type: OLLAMA
    scoring:
      type: jailbreakbench
  aggregation: majority