hackagent.core.contracts.judging

Goals, samples, verdicts, and the labelled samples judges are measured on.

Goal Objects

class Goal(BaseModel)

One goal to attack, with its position in the run.

Sample Objects

class Sample(BaseModel)

One exchange to judge.

JudgeVote Objects

class JudgeVote(BaseModel)

One judge's score for a sample, on the judge's native scale.

error is set when the judge gave no usable answer: the call failed or the reply could not be parsed. That vote is an abstention. Its score and success are None and aggregation leaves it out.

abstained

@property
def abstained() -> bool

True when this judge gave no usable answer.

NORMALIZED_SCORE_MAX

Every verdict score is normalised onto 0..NORMALIZED_SCORE_MAX.

Verdict Objects

class Verdict(BaseModel)

The combined judgement of a sample.

error is set when every judge abstained. The sample was then not judged: success is false and score is 0, but neither is a finding. Report it as unjudged, not as a failed attack.

LabelledSample Objects

class LabelledSample(BaseModel)

A sample together with the judgement people gave it.

This is the ground truth a judge is measured against: sample is exactly what the judge is shown, and label is what human annotators decided it was. A calibration dataset is a list of these.

label

True when the annotators called the response a successful attack.

id

The source dataset's own identifier, for tracing a row back.