Skip to main content

hackagent.evaluation.metrics

Metrics over verdicts.

A verdict whose judges all abstained (Verdict.error) is not a judgement. Rates and means leave it out of the denominator; summary counts it under abstained. An abstaining or missing vote is missing, not a safe vote.

success_rate​

def success_rate(verdicts: Sequence[Verdict]) -> float

Fraction of judged verdicts marked successful. Empty input is 0.

mean_score​

def mean_score(verdicts: Sequence[Verdict]) -> float

Mean judged verdict score on the 0..10 scale. Empty input is 0.

majority_vote_rate​

def majority_vote_rate(verdicts: Sequence[Verdict]) -> float

Share of judged verdicts where more than half the votes cast say success.

A tie is not a success. With at most one judge, this is :func:success_rate.

fleiss_kappa​

def fleiss_kappa(verdicts: Sequence[Verdict]) -> float

Fleiss' kappa across judge votes.

Only verdicts every judge voted on are used, since kappa assumes a fixed number of raters per item. One judge, or no such verdicts, is 1.

per_judge_strictness​

def per_judge_strictness(verdicts: Sequence[Verdict]) -> Dict[str, float]

Safe-rate (1 - positive rate) per judge over its cast votes, plus bias_gap.

summary​

def summary(verdicts: Sequence[Verdict]) -> Dict[str, Any]

Compact report for a list of verdicts.