A model learns by lowering a loss and is judged by metrics. Both are formulas over pixels or voxels, and each answers a different question: how much of the object did the model find, how much did it invent, how far off is its boundary. This page explains the ones you will meet in the project and in papers, starting from counting pixels. The code that computes them for the project’s benchmark is osv/eval/metrics.py; for an overview of the models, see the comparison of volumes and video.
Notation
- P is the set of pixels the model predicts for a class, G the ground truth, the pixels an annotator labelled. For a volume read “voxels” for “pixels”.
- p is the model’s predicted probability for a pixel and g its true label, 1 or 0. An index i runs over pixels, c over classes, n over images.
- |P| is the number of pixels in P, ∂P its boundary.
Counting pixels: the confusion matrix
Every pixel falls into one of four groups for a given class: predicted and true (true positive, TP), predicted but not true (false positive, FP), true but missed (false negative, FN), and neither (true negative, TN).
- TP · true positive
- FP · false positive
- FN · false negative
- TN · true negative
Read: TP for class c is the number of pixels i whose prediction and truth are both c. With more than two classes the counts form a C × C table, rows for the truth and columns for the prediction; the project accumulates this table over every pixel of a split and derives all the metrics below from it. Pixels labelled 255, “ignore”, are left out of the table.
Pixel accuracy, and why it misleads
The share of pixels that got the right class; N is the number of pixels. In the example it is (12 + 60) / 96 = 0.75. It sounds like the natural score, but background dominates surgical images. If instruments cover 2 % of a frame, a model that predicts no instrument at all scores 0.98. Report it, if at all, next to the metrics below.
Precision and recall
- Precision is the share of the prediction that is right: how much the model invents. Example: 12 / 24 = 0.5.
- Recall, also called sensitivity, is the share of the truth that was found: how much the model misses. Example: 12 / 24 = 0.5.
- Specificity is the share of the background that stayed background. It is close to 1 for almost any model when the object is small, for the same reason as accuracy.
Precision and recall pull against each other. Lowering the probability threshold at which a pixel counts as “instrument” raises recall and lowers precision. Their harmonic mean, F1, is high only when both are:
IoU and Dice
The two headline metrics of segmentation compare the predicted and the true region directly.
- IoU, intersection over union or the Jaccard index: the overlap divided by everything either mask covers. Example: 12 / 36 = 0.33.
- Dice, the Dice–Sørensen coefficient: twice the overlap divided by the two sizes added. Example: 24 / 48 = 0.5. For a single class Dice is exactly F1 from the section above.
- Both run from 0, no overlap, to 1, a perfect match, and both ignore true negatives, so a large background does not inflate them.
Dice and IoU always rank two predictions of the same image the same way, because each is a function of the other. Dice is the larger number for any imperfect mask; never compare a Dice from one paper with an IoU from another.
Several classes: mean IoU
Mean IoU averages the per-class IoU over C′, the classes that actually occur in the evaluated split. That is the project’s rule: a class with no true pixels in the split has no score, neither 0 nor 1, and the report says how many classes were averaged. In CholecSeg8k three of the thirteen classes are missing from a whole-video split, and scoring them as 0 would drag the mean down for no reason.
Pooled or averaged over images
The pooled IoU adds the counts of all M images first and divides once; the mean IoU computes one IoU per image and averages them. Pooled weights large objects more; the mean gives every image the same weight, so a few frames with a tiny, missed instrument pull it down hard. The project’s M1 experiment reports the pooled binary IoU over all test pixels. Say which one you report.
Try it
Move the prediction against a fixed true instrument and watch the metrics. Make the object small and press “Predict nothing”: accuracy stays high, IoU and Dice drop to zero.
- TP
- FP
- FN
- TN
- Pixel accuracy
- –
- Precision
- –
- Recall
- –
- IoU
- –
- Dice
- –
- HD95, pixels
- –
Boundaries: Hausdorff distance and surface Dice
Overlap metrics hardly notice a thin wrong spike or a missing tip, which matter for instruments and for organs at risk. Boundary metrics measure distances between the two contours, in pixels or, using the voxel spacing, in millimetres.
- d(a, B) is the distance from a point a to the nearest point of the boundary ∂B.
- Hausdorff distance (HD) is the largest such distance in either direction: the worst point of the prediction measured against the truth, or of the truth against the prediction. One stray pixel far away sets it.
- HD95 replaces the maximum by the 95th percentile, P95, of each direction and takes the larger of the two, which is how MONAI computes it. It ignores the worst 5 % of points and is far more stable. Lower is better for both; 0 is a perfect match.
Normalised surface Dice (NSD) at a tolerance τ is the share of both boundaries that lies within τ of the other (Nikolov et al., 2021). It asks “how much of the contour would a clinician have to correct?” when deviations below τ do not matter. τ is chosen per structure, for example from the variation between annotators, and must be reported with the score. In 2D the same idea gives boundary F1 and Boundary IoU, which score thin structures such as instrument shafts fairly.
In 3D
The same distances on surfaces: the true structure is a sphere of radius 20 mm, the prediction leaks into a neighbour on one side and falls short on another. Points of the predicted surface within τ of the truth are green; farther points turn orange, darker with distance. Move τ and watch HD stay put while NSD changes.
- τ
- –
- HD
- –
- HD95
- –
- NSD at τ
- –
Losses: what the model actually optimises
A metric is computed after thresholding, on hard masks, and is not differentiable. Training needs a smooth function of the predicted probabilities: the loss. Lower is better, and it is computed on every batch.
Cross-entropy punishes a low probability for the true class of each pixel: the penalty −log p grows without bound as p approaches 0. It treats every pixel alike, so on images dominated by background the model can do well on the loss while ignoring small structures. Class weights in the sum are the usual remedy. The project’s M1 baseline trains with plain cross-entropy, plus 0.4 times the same loss on DeepLabV3’s auxiliary head.
Soft Dice loss puts the Dice formula on probabilities instead of hard masks; ε, a small constant, keeps it defined when both are empty. It looks at each class as a whole and is insensitive to how much background there is (Milletari et al., 2016). nnU-Net trains with the sum of the two.
Focal loss (Lin et al., 2017) multiplies cross-entropy by (1 − p)^γ, where p is the probability of the true class: easy pixels that the model already gets right count for little, hard ones for a lot. γ = 2 is the common choice.
Reading training curves
Training loss keeps falling as long as the model can memorise the training images. Validation loss, on images the model never trains on, falls while the model learns things that generalise and rises once it starts memorising: overfitting. Keep the checkpoint that is best on validation; the project selects M1 checkpoints by validation IoU. The test set plays no part in this choice and is scored once, at the end.
Images, not masks: PSNR and SSIM
Some models output an image, for example the project’s M5 desmoking models, which remove smoke from a frame. Their output is compared with a clean reference image.
PSNR, peak signal-to-noise ratio, is the mean squared error on a logarithmic scale, in decibels, for images scaled to 0–1. Higher is better; every 10 dB is a tenfold smaller error. Identical images would score infinity, so the project caps it at 100 dB.
SSIM, structural similarity (Wang et al., 2004), compares local means μ, contrasts σ and the correlation σab of two images a and b in a small window, and averages over the image. It follows what people see better than PSNR does: blur lowers it, a uniform change of brightness much less. The project computes it on luminance with an 11-pixel Gaussian window, σ = 1.5. It runs up to 1 for identical images.
One number is not enough
A score on a test set is an estimate. Report how uncertain it is.
- Mean and 95 % interval. For n scores, for example one per frame or per seed, the interval is the mean plus or minus 1.96 standard errors, s / √n. It assumes the scores are independent, which frames of one video are not.
- Bootstrap by video. Resample whole videos with replacement many times, recompute the metric each time, and take the 2.5th and 97.5th percentiles. The project’s M1 decision used this, resampling test videos and seeds, paired between the two arms.
- Seeds. The same training run with another random seed gives another score. The project’s rule is at least three seeds per arm.
Latency percentiles follow the same idea for speed: sort n measured times and take the value at rank ⌈0.95 n⌉. The p95 says how slow the slowest one in twenty frames is; a mean hides those frames. See the latency budget.
At a glance
| Metric | Range | Better | Use it for | Watch out for |
|---|---|---|---|---|
| Pixel accuracy | 0–1 | higher | a sanity check | inflated by background |
| Precision, recall | 0–1 | higher | what is invented, what is missed | each alone is easy to max out |
| IoU, Dice | 0–1 | higher | overlap of masks, the headline score | small objects, absent classes, pooled or per image |
| HD, HD95 | 0–∞ | lower | worst boundary errors | HD follows one outlier; empty masks give infinity |
| NSD at τ | 0–1 | higher | share of the boundary that needs no correction | meaningless without τ |
| Cross-entropy, Dice loss | 0–∞, 0–1 | lower | training | not a test metric |
| PSNR, SSIM | dB, up to 1 | higher | image restoration | PSNR misses blur |
References
- Maier-Hein et al. Metrics reloaded: recommendations for image analysis validation. Nature Methods, 2024.
- Taha and Hanbury. Metrics for evaluating 3D medical image segmentation: analysis, selection, and tool. BMC Medical Imaging, 2015.
- Nikolov et al. Clinically applicable segmentation of head and neck anatomy for radiotherapy. JMIR, 2021 (surface Dice).
- Csurka et al. What is a good evaluation measure for semantic segmentation? BMVC 2013 (boundary F1); Cheng et al. Boundary IoU. CVPR 2021.
- Wang et al. Image quality assessment: from error visibility to structural similarity. IEEE TIP, 2004.
- Lin et al. Focal Loss for Dense Object Detection. ICCV 2017.
Back to the segmentation guide: 3D volumes and surgical video.