Skip to main content

hackagent.attacks.evaluator.metrics

Metrics and evaluation analysis utilities.

This module provides reusable metric calculation and analysis functions for attack evaluation.

Functions: calculate_success_rate: Overall success rate from results calculate_confidence_score: Average confidence from results group_by_goal: Group results by goal calculate_per_goal_metrics: Per-goal metric breakdown generate_summary_report: Comprehensive summary report

Usage: from hackagent.attacks.evaluator.metrics import ( calculate_success_rate, generate_summary_report, )

is_successful_result​

def is_successful_result(row: Dict[str, Any]) -> bool

Best-effort success derivation from common result representations.

A result row is considered "successful" when the attack achieved its objective against the victim (e.g. a jailbreak judge voted harmful). The inverse — a row that is not successful — means the goal was mitigated by the victim for that particular attempt. This function is the single source of truth for that determination, reused by summary metrics as well as multi-attack chaining (see HackAgent.hack_chain).

calculate_success_rate​

def calculate_success_rate(results: List[Dict[str, Any]]) -> float

Calculate overall success rate from results.

Arguments:

  • results - List of attack result dictionaries

Returns:

Success rate as float between 0 and 1

group_by_goal​

def group_by_goal(
        results: List[Dict[str, Any]]) -> Dict[str, List[Dict[str, Any]]]

Group results by goal.

Arguments:

  • results - List of attack result dictionaries

Returns:

Dictionary mapping goals to their results

extract_judge_scores​

def extract_judge_scores(
        results: List[Dict[str, Any]]) -> Dict[str, List[int]]

Extract attack judge scores from present eval_* columns as 0/1 lists.

calculate_majority_vote_asr​

def calculate_majority_vote_asr(results: list[dict]) -> float

Calculate Attack Success Rate using majority vote across judges.

For a single judge, return the average of that judge's votes.

calculate_fleiss_kappa​

def calculate_fleiss_kappa(results: list[dict]) -> float

Calculate Fleiss' Kappa for agreement among judges.

For a single judge, return 1.0 (perfect agreement trivially).

calculate_per_judge_strictness​

def calculate_per_judge_strictness(results: list[dict]) -> dict

Calculate Per-Judge Strictness (Bias Gap) from attack judges only.

Returns:

Dict with each judge's strictness (safe-rate = 1 - average jailbreak vote)

  • overall bias gap. Judge keys are only the eval_* columns present in results, plus "bias_gap".

calculate_per_judge_asr​

def calculate_per_judge_asr(results: list[dict]) -> dict

Calculate per-judge ASR for every present eval_* vote column.

calculate_per_goal_metrics​

def calculate_per_goal_metrics(
        results: List[Dict[str, Any]]) -> Dict[str, Dict[str, Any]]

Calculate metrics for each goal separately.

Arguments:

  • results - List of attack result dictionaries

Returns:

Dictionary mapping goals to their metrics

generate_summary_report​

def generate_summary_report(results: List[Dict[str, Any]]) -> Dict[str, Any]

Generate comprehensive summary report from results.

Arguments:

  • results - List of attack result dictionaries

Returns:

Summary report dictionary