LECTURE 3

Lecture 3. Assisted annotation and agreement between annotators

How surgical video gets labelled faster without getting worse: annotation platforms, pre-annotation with SAM 2, active learning with MONAI Label, safeguards against automation bias, and measuring agreement with Cohen's and Fleiss' kappa and Dice. With diagrams, formulas, interactive examples and 3D animations.

This page is still being reviewed by our scientific, clinical and legal experts. It describes plans, not finished work.

An hour of laparoscopic video at 25 frames per second is 90,000 frames. Drawing every instrument and organ by hand is out of the question: CholecSeg8k, one of the field’s main datasets, has pixel labels for 8,080 frames from 17 operations. Manual labelling is also subjective, because where the gallbladder ends and fat begins is a judgement. This lecture covers the tools that speed annotation up, the safeguards that keep a model’s suggestions from quietly becoming the truth, and how to measure whether two people agree. It is lecture 3 of the learning programme, after lecture 2 on de-identification.

Every frame on this page is synthetic.

The loop

video frames1pre-annotationSAM 2 or a model2annotatorcorrects with clicks3reviewagreement checks4datasettrainingbetter modelcontrol frames, 10 %no pre-annotation5
Assisted annotation is a loop: every round of labels trains a model that pre-annotates the next round.
  1. Frames come from de-identified video.
  2. Pre-annotation: a model, SAM 2 or one trained on earlier rounds, proposes masks.
  3. The annotator accepts, corrects with a few clicks or redraws.
  4. Review checks the labels and the agreement between annotators.
  5. Control frames, one in ten, are annotated from scratch, without pre-annotation, to measure how much the suggestions sway people.

Annotation platforms

  • CVAT is the project’s platform, in a closed workspace for invited annotators. Shapes include polygons, masks drawn with a brush, rectangles, points and skeletons; tags label a whole frame, for example its phase or “smoke present”; tracks follow an object through a video and interpolate between keyframes. AI tools plug in as interactors, which turn clicks into a mask, detectors and trackers; the community edition can run the original Segment Anything as an interactor, and CVAT’s SAM 2 tracker, which carries polygons and masks forward through frames, comes in the Enterprise edition or as an AI agent for CVAT Online and Enterprise.
  • Label Studio builds its interface from an XML configuration and connects models through an ML backend. Its SAM 2 examples cover images, with masks, and video, where the backend currently tracks one object with boxes.
  • osv.annotate, the project’s connector to both, is planned. Today it holds only the configuration of the pre-annotation pipeline: SAM 2 as the pre-annotator, 10 % control frames and a delta-contour threshold of 0.90 (see the safeguards below).

Pre-annotation with SAM 2

box+−positive clicknegative clickprompts on one framemaskcarried to the next frames
One positive click, one negative click and a box on a single frame give a mask that SAM 2 then carries forward.

SAM 2 segments an object from a prompt: a positive click on the object, a negative click on something that is not the object, a box, or a rough mask. In a video it keeps a memory of earlier frames and prompts and carries the mask forward. When it drifts, onto the liver behind a grasper for example, the annotator adds a click on the frame where it went wrong, and the correction propagates from there. How the memory works is explained on the video page.

To feel the clicking, try the toy below. It is not SAM 2: it grows a region of similar colour from each click, a technique decades older and far weaker. A positive click adds a region, a negative click removes one.

Automation bias

A suggestion on screen is persuasive. Tired annotators accept masks they would have drawn differently, and a dataset slowly takes on the model’s errors; a model trained on it then repeats them with more confidence. The project’s annotation guideline and rule R-20 set the safeguards:

  • Control frames. 10 % of the queue is shown without pre-annotation, unmarked, so annotators cannot tell which frames are controls. This share is never switched off.
  • Delta-contour Dice compares masks of the same frames drawn with and without pre-annotation; it should be at least 0.90.
  • Acceptance rate, the share of SAM 2 masks accepted without a single edit, should stay below 60 %.
  • Time per frame far below the median, under 40 % of it, is a warning sign, and a systematic shift in area above 5 % for a class takes that class out of pre-annotation.
  • Pre-annotation is switched off per class, never globally: instruments have crisp edges and SAM 2 handles them well; ducts and dissection planes are where it drifts.

Active learning with MONAI Label

3D SlicerOHIFCVATREST APIMONAI Label serverinfer: pre-annotatetrain on new labelsstrategy: next samplescoring: dropout entropydatastoreimages and labels
A viewer asks the server for the next image, gets a pre-annotation, and sends the corrected label back; the server retrains in between.

Not every frame is worth the same effort. Ten thousand frames of a steady dissection teach a model less than fifty frames of the situations it gets wrong. Active learning lets the model choose what to label next.

MONAI Label is a server that sits between a viewer and a model: 3D Slicer or OHIF for CT and MRI, CVAT for video. Its endoscopy app ships a tool-tracking model, an interactive DeepEdit model and an in-body classifier, and it can use CVAT as its datastore. The server pre-annotates, trains on new labels and, through a strategy, picks the next image to label:

  1. random or first: a baseline.
  2. epistemic: the image the model is least sure about. The model runs T times with dropout switched on; where the runs disagree, the averaged prediction is spread over classes and its entropy is high.
p‾i,c=1T∑tpi,c(t) H=∑i(−∑cp‾i,clogp‾i,c)

Here p̄ is the prediction averaged over T dropout passes for pixel i and class c, and H the entropy summed over pixels: zero where every pass agrees on one class, large where they scatter. Images with the largest H come first.

Uncertainty is not everything. It favours outliers and near-duplicates of the same hard frame, and it is unreliable while the model has seen almost nothing. Keep a share of random picks, start from a small random sample, and keep the validation set random and fixed.

Agreement between annotators

Labels have no ground truth to compare with. What can be measured is whether two or more competent people, given the same frames and the same guideline, produce the same labels. Low agreement means the class definition, the guideline or the training needs work, and it caps how good any model can look.

Cohen’s kappa, two annotators

annotator Bannotator Asmokeno smoketotalsmoke20525no smoke106575total3070100Po = (20 + 65) / 100 = 0.85Pe = 0.25 · 0.30 + 0.75 · 0.70 = 0.60κ = (0.85 − 0.60) / (1 − 0.60) = 0.625
Two annotators, 100 frames, one question: is there smoke?
κ=Po−Pe1−Pe Po=∑cnccN,Pe=∑cnc·N·n·cN

Po is the observed agreement, the share of frames on which both chose the same class. Pe is the agreement expected by chance: for each class, the share of frames annotator A put in it times the share annotator B did, summed over classes. Kappa (Cohen, 1960) is how far the observed agreement is above chance, as a share of the way from chance to perfect: 1 is perfect agreement, 0 is no better than chance, below 0 worse.

Why not just Po? Because a rare class makes agreement cheap. If 95 % of frames show no smoke, two annotators who almost never mark smoke agree on most frames without agreeing on any smoke. Try the presets:

Landis and Koch (1977) named bands that are still quoted: below 0 poor, up to 0.20 slight, up to 0.40 fair, up to 0.60 moderate, up to 0.80 substantial, above that almost perfect. They are a convention, not a law; what is good enough depends on the task.

Fleiss’ kappa, many annotators

Pi=1r(r−1)∑cnic(nic−1),pc=1Nr∑inic κ=P‾−P‾e1−P‾e,P‾=1N∑iPi,P‾e=∑cpc2

Fleiss’ kappa (1971) extends the idea to r annotators per item. For each of N items, nic annotators chose class c; Pi is the share of annotator pairs that agree on item i, P̄ its mean over items, and P̄e the chance agreement from the overall share pc of each class. The project’s guideline asks for a Fleiss kappa of at least 0.70 on categorical labels.

Masks: Dice between annotators

For masks the question is how much two drawings of the same structure overlap. Dice and IoU work exactly as for a model against ground truth (see the metrics page); with more than two annotators, average over all pairs:

Dicepairs=2R(R−1)∑a<bDice(Ma,Mb)

The guideline asks for a mean pairwise Dice of at least 0.80. Where annotators keep disagreeing, for example on the boundaries of an inflamed gallbladder, the frame is marked needs-clinician and settled by the clinical lead. Several masks can also be merged into one consensus estimate that weighs each annotator by how reliable they appear, as STAPLE does (Warfield et al., 2004).

Checklist

  • Annotate in CVAT, inside the closed workspace; never export frames to a personal tool.
  • Use pre-annotation as a draft, and correct it as if you had drawn it.
  • Leave control frames alone and never hide the acceptance-rate telemetry.
  • Prefer labelling what the model is unsure about, with a share of random picks.
  • Report agreement: Cohen’s or Fleiss’ kappa for categories, pairwise Dice for masks, with the number of items.

Further reading