Skip to main content

hackagent.reporting.goals

Goal verdicts and the summaries built from them.

A goal's verdict is jailbroken if any attempt is a jailbreak; else mitigated if its scope is complete and every attempt is mitigated; else unresolved. A campaign scope is complete when the campaign run and every attack run in it are COMPLETED; one attack run's own summary always counts as complete, because the runner records a goal's attempts after that attack has finished the goal.

VERDICT_CODES​

How /matrix codes a per-attack verdict.

UNKNOWN_CATEGORY​

Category of a goal no attempt labelled, and subcategory of one with none.

goal_verdict​

def goal_verdict(*, attempts: int, jailbreak: int, mitigated: int,
                 complete: bool) -> GoalVerdict

The verdict of a goal from its attempt counts.

attack_verdict​

def attack_verdict(outcome: GoalOutcome) -> AttackVerdict

One attack run's verdict on a goal: error when every attempt errored.

is_complete​

def is_complete(status: str, attack_runs: Iterable[AttackRun]) -> bool

A campaign scope is complete when it and all its attack runs completed.

run_order​

def run_order(runs: Iterable[AttackRun]) -> list[AttackRun]

Attack runs in report order: by created_at, then id.

summarize_goals​

def summarize_goals(
    outcomes: Iterable[GoalOutcome],
    complete: bool,
    runs: Sequence[AttackRun] = ()) -> list[GoalSummary]

One summary per goal index, ordered by index.

A goal's text comes from the most recently created attack run of runs that recorded it, and its labels from the most recently created one that recorded a category (created_at, then id; runs not in runs come first).

outcome_counts​

def outcome_counts(goals: Iterable[GoalSummary]) -> OutcomeCounts

How many of goals were recorded, and how many have each verdict.

subcategory_name​

def subcategory_name(goal: GoalSummary) -> str

The subcategory a goal is grouped under; empty is Unspecified.

category_summaries​

def category_summaries(goals: Iterable[GoalSummary]) -> list[CategorySummary]

Counts per category (sorted), and per subcategory in order of first goal.

attack_outcomes​

def attack_outcomes(goals: list[GoalSummary], outcomes: Iterable[GoalOutcome],
                    runs: Sequence[AttackRun]) -> list[GoalSummary]

goals with each attack run's own verdict on them, in run order.

GoalFilters Objects​

@dataclass(frozen=True)
class GoalFilters()

The filters of /goals; see :func:~hackagent.reporting.http.goal_filters.

filter_goals​

def filter_goals(goals: Iterable[GoalSummary],
                 filters: GoalFilters) -> list[GoalSummary]

The goals that pass every filter, in order.