Summary

Metric Lookup Table¶

Metric	Formula	What it rewards	When to use
Precision	$\frac{TP}{TP+FP}$	Clean results (few false positives)	Any binary decision; especially when FP is costly
Recall	$\frac{TP}{TP+FN}$	Completeness (few misses)	Exhaustive search; patent, legal, medical
$F_\beta$	$\frac{(\beta^2+1) \cdot p \cdot r}{\beta^2 \cdot p + r}$	Balances precision and recall with tuneable weight	Single-number summary; $\beta=1$ default
$P@k$	Precision in top $k$	Clean top-of-list	Web search, short result pages
MRR	Mean of $1/\text{rank}_{\text{first relevant}}$	First relevant result high	QA, known-item search
R-Precision	$p@R$	Balance at a natural cutoff	Self-adjusting single-point summary
AP	Mean of $p_k$ at relevant positions, divided by $	\text{Rel}	$
MAP	Mean of per-query AP	System-level ranking quality	Benchmark comparison (TREC standard)
$nDCG@k$	$DCG_k / IDCG_k$	High-grade documents placed early	Web search, graded relevance, user experience
Accuracy	$\frac{TP+TN}{P+N}$	Overall correct decisions	Balanced classification; equals micro-precision and micro-recall
Specificity (TNR)	$\frac{TN}{TN+FP}$	Few false alarms	Medical tests, biometric security
AUC	Area under ROC curve	Score separation across all thresholds	Comparing classifiers without committing to a threshold

Evaluation requires a benchmark: a collection, information needs, relevance judgments, and a performance goal. Without any component, a number is not interpretable.
Precision and recall are in tension. Improving one typically costs the other, and the $F_\beta$ -measure makes the trade-off explicit.
Ranked evaluation adds position: AP and nDCG reward systems that place relevant (or high-grade) material where users will see it.
Graded relevance distinguishes “somewhat useful” from “essential”. nDCG captures what binary metrics cannot: the quality of the relevant documents, not only their presence.
Classification evaluation reuses precision and recall but adds accuracy, specificity, and the confusion matrix. Accuracy equals micro-precision equals micro-recall in single-label settings.
Prevalence distorts accuracy: when one class dominates, a trivial baseline can score higher than a real classifier. Always check class balance before trusting accuracy.
A continuous score becomes a binary decision only after choosing a threshold. The ROC curve shows all possible trade-offs; AUC summarizes overall score separation.
No benchmark score is a claim about generalization. Incomplete judgments, data contamination, prevalence mismatch, and overfitting to static test sets all limit what a number can mean.

p = \frac{TP}{TP+FP} \qquad r = \frac{TP}{TP+FN}

Precision and recall: the fundamental pair from which all other metrics derive.

AP = \frac{1}{|\text{Rel}|} \sum_{k: d_k \in \text{Rel}} p_k

Average precision: area under the precision-recall curve, the standard per-query measure.

nDCG_k = \frac{\sum_{i=1}^k \frac{rel_i}{\log_2(i+1)}}{\text{IDCG}_k}

Normalized discounted cumulative gain: the standard for graded relevance evaluation.

AUC = P(s(x_+) > s(x_-))

Area under the ROC curve: the probability that a random positive scores higher than a random negative.

(Understand) A system achieves 95% accuracy on a dataset where 95% of items belong to class A. Is this system useful? What additional metric would reveal the problem?
(Analyze) System X has MAP = 0.60 and System Y has MAP = 0.55, both measured on 50 queries. Under what circumstances might Y actually be the better system?
(Evaluate) A face-unlock system reports AUC = 0.99 and a spam filter reports AUC = 0.99. Are they equally good? What additional information do you need to judge each for its intended use?
(Apply) Given a ranked list of 10 documents where relevant documents appear at positions 1, 4, and 7 (out of 8 relevant in the collection), compute $P@5$ , AP, and explain why AP is less than the average of the three precision values.
(Analyze) Explain why macro-averaged recall across 4 information needs can disagree with micro-averaged recall, and give a concrete scenario where the disagreement is large.