Skip to main content

hackagent.reporting.tallies

Stored per-goal tallies and the functions that keep them.

Reports never recount results or scan traces. A backend keeps two tables that mirror :class:GoalOutcome and :class:GoalUsage column for column, and updates them in the same transaction as every write that affects them:

  • result created or updated: recompute the goal's :class:GoalOutcome with :func:goal_outcome from the light :class:ResultRow of every result of that (attack run, goal index), keeping the stored trace coverage;
  • trace created: apply :func:add_trace to the goal's stored rows, given the :class:TraceFlags of the result's earlier traces;
  • result or trace deleted: recompute both rows of the goal from what is left, with :func:goal_outcome, :func:trace_coverage and :func:goal_usage (or delete them when no result is left);
  • attack run deleted: delete its rows.

Timestamps are timezone-aware datetimes. Ids are lowercase canonical UUID strings; ordering "by created_at then id" compares those strings, which is also how Postgres orders uuid and SQLite orders TEXT.

INTERACTION​

Label of the usage row that counts a goal's interaction steps.

INTERACTION_STEP​

step_type of the step the runner records for each exchange with the target. Both stores keep it as written (StepKind.INTERACTION; the platform's Trace.StepType.INTERACTION) and every rule compares it exactly.

ResultRow Objects​

@dataclass(frozen=True)
class ResultRow()

The light columns of one result, which its goal's outcome is computed from.

Read for every result of one (attack run, goal index)::

SELECT id, goal, evaluation_status, evaluation_notes,
       agent_specific_data->>'category'                AS category,
       agent_specific_data->>'subcategory'             AS subcategory,
       (agent_specific_data->'attack_metadata'
          ->>'goal_active_target_elapsed_s')::float     AS active_elapsed_s,
       created_at, updated_at
FROM result WHERE run_id = %s AND goal_index = %s

category and subcategory are None when missing or empty. active_elapsed_s is None unless it is a JSON number. :meth:from_data builds the row from a result already in memory.

TraceRow Objects​

@dataclass(frozen=True)
class TraceRow()

The light columns of one trace step, which usage and coverage count.

Read with::

SELECT result_id, step_type,
       content->>'node'  AS node,
       content->>'label' AS label,
       content->>'scope' AS scope,
       COALESCE((content->'data'->'usage'->>'prompt_tokens')::bigint,
                (content->'response'->'usage'->>'prompt_tokens')::bigint)
           AS prompt_tokens,
       COALESCE((content->'data'->'usage'->>'completion_tokens')::bigint,
                (content->'response'->'usage'->>'completion_tokens')::bigint)
           AS completion_tokens
FROM trace WHERE result_id IN (...)

:meth:from_content builds the row from a trace already in memory, which is what a backend does when it creates one.

TraceCoverage Objects​

@dataclass(frozen=True)
class TraceCoverage()

How many attempts of a goal recorded each kind of trace step.

labelled_call_attempts​

Attempts with a call node that carries a non-empty label.

target_call_attempts​

Attempts with a call node labelled target.

interaction_attempts​

Attempts with an interaction step.

TraceFlags Objects​

@dataclass(frozen=True)
class TraceFlags()

Which kinds of trace step one attempt has recorded so far.

GoalOutcome Objects​

@dataclass(frozen=True)
class GoalOutcome()

Table goal_outcome: one row per (attack run, goal index).

Primary key (run_id, goal_index); index (campaign_run_id, run_id, goal_index).

========================= ============ =================================== column type value ========================= ============ =================================== campaign_run_id uuid, null the attack run's campaign run run_id uuid the attack run goal_index integer goal text goal text of the latest attempt category text of the latest attempt that has one, '' when none has subcategory text of that same attempt, or '' attempts integer results jailbreak integer results per bucket mitigated integer error integer pending integer first_jailbreak_attempt integer, null ordinal (1-based, created_at then id) of the first jailbreak min_jailbreak_elapsed_s float, null smallest active goal clock among the jailbreaks that carry one timed_jailbreaks integer jailbreaks that carry the clock first_created_at timestamp MIN(created_at) of the results last_updated_at timestamp MAX(updated_at) of the results labelled_call_attempts integer see :class:TraceCoverage target_call_attempts integer interaction_attempts integer ========================= ============ ===================================

"Latest" is the greatest (created_at, id).

GoalUsage Objects​

@dataclass(frozen=True)
class GoalUsage()

Table goal_usage: one row per (attack run, goal index, label).

Primary key (run_id, goal_index, label); index (campaign_run_id, run_id, goal_index). label is a call node's label, or '' for interaction steps (see :func:usage_label).

================= ========== ========================================== column type value ================= ========== ========================================== campaign_run_id uuid, null the attack run's campaign run run_id uuid the attack run goal_index integer label text calls integer steps counted under this label goal_scoped_calls integer of which content.scope = 'goal' prompt_tokens bigint sum over the steps that report it completion_tokens bigint sum over the steps that report it usage_calls integer steps that report both token counts ================= ========== ==========================================

usage_label​

def usage_label(trace: TraceRow) -> Optional[str]

The usage row a trace step counts under, or None when it counts nowhere.

An interaction step counts under ''; any other step whose node is call counts under its label. A call node without a label names no role and is not counted.

trace_flags​

def trace_flags(traces: Iterable[TraceRow]) -> TraceFlags

What traces (the steps of one attempt) have recorded.

trace_coverage​

def trace_coverage(traces: Iterable[TraceRow]) -> TraceCoverage

Coverage of a goal from every step of every one of its attempts.

goal_outcome​

def goal_outcome(
    *,
    campaign_run_id: Optional[str],
    run_id: str,
    goal_index: int,
    results: Sequence[ResultRow],
    coverage: TraceCoverage = TraceCoverage()
) -> Optional[GoalOutcome]

The outcome row of a goal from every one of its results, or None when it has none. coverage is kept as given: results do not change it.

goal_usage​

def goal_usage(*, campaign_run_id: Optional[str], run_id: str, goal_index: int,
               traces: Iterable[TraceRow]) -> list[GoalUsage]

The usage rows of a goal from every step of every one of its attempts, ordered by label.

add_trace​

def add_trace(outcome: GoalOutcome, usage: Optional[GoalUsage],
              trace: TraceRow,
              before: TraceFlags) -> tuple[GoalOutcome, Optional[GoalUsage]]

Count one new trace step of an attempt of outcome's goal.

usage is the stored row under :func:usage_label of trace (None when there is none yet), and before the :func:trace_flags of the attempt's earlier steps. Returns the outcome to store and the usage row to upsert (None when the step counts under no label).