An hour of laparoscopic video at 25 frames per second is 90,000 frames. Drawing every instrument and organ by hand is out of the question: CholecSeg8k, one of the field’s main datasets, has pixel labels for 8,080 frames from 17 operations. Manual labelling is also subjective, because where the gallbladder ends and fat begins is a judgement. This lecture covers the tools that speed annotation up, the safeguards that keep a model’s suggestions from quietly becoming the truth, and how to measure whether two people agree. It is lecture 3 of the learning programme, after lecture 2 on de-identification.
Every frame on this page is synthetic.
The loop
- Frames come from de-identified video.
- Pre-annotation: a model, SAM 2 or one trained on earlier rounds, proposes masks.
- The annotator accepts, corrects with a few clicks or redraws.
- Review checks the labels and the agreement between annotators.
- Control frames, one in ten, are annotated from scratch, without pre-annotation, to measure how much the suggestions sway people.
Annotation platforms
- CVAT is the project’s platform, in a closed workspace for invited annotators. Shapes include polygons, masks drawn with a brush, rectangles, points and skeletons; tags label a whole frame, for example its phase or “smoke present”; tracks follow an object through a video and interpolate between keyframes. AI tools plug in as interactors, which turn clicks into a mask, detectors and trackers; the community edition can run the original Segment Anything as an interactor, and CVAT’s SAM 2 tracker, which carries polygons and masks forward through frames, comes in the Enterprise edition or as an AI agent for CVAT Online and Enterprise.
- Label Studio builds its interface from an XML configuration and connects models through an ML backend. Its SAM 2 examples cover images, with masks, and video, where the backend currently tracks one object with boxes.
osv.annotate, the project’s connector to both, is planned. Today it holds only the configuration of the pre-annotation pipeline: SAM 2 as the pre-annotator, 10 % control frames and a delta-contour threshold of 0.90 (see the safeguards below).
Pre-annotation with SAM 2
SAM 2 segments an object from a prompt: a positive click on the object, a negative click on something that is not the object, a box, or a rough mask. In a video it keeps a memory of earlier frames and prompts and carries the mask forward. When it drifts, onto the liver behind a grasper for example, the annotator adds a click on the frame where it went wrong, and the correction propagates from there. How the memory works is explained on the video page.
To feel the clicking, try the toy below. It is not SAM 2: it grows a region of similar colour from each click, a technique decades older and far weaker. A positive click adds a region, a negative click removes one.
Automation bias
A suggestion on screen is persuasive. Tired annotators accept masks they would have drawn differently, and a dataset slowly takes on the model’s errors; a model trained on it then repeats them with more confidence. The project’s annotation guideline and rule R-20 set the safeguards:
- Control frames. 10 % of the queue is shown without pre-annotation, unmarked, so annotators cannot tell which frames are controls. This share is never switched off.
- Delta-contour Dice compares masks of the same frames drawn with and without pre-annotation; it should be at least 0.90.
- Acceptance rate, the share of SAM 2 masks accepted without a single edit, should stay below 60 %.
- Time per frame far below the median, under 40 % of it, is a warning sign, and a systematic shift in area above 5 % for a class takes that class out of pre-annotation.
- Pre-annotation is switched off per class, never globally: instruments have crisp edges and SAM 2 handles them well; ducts and dissection planes are where it drifts.
Active learning with MONAI Label
Not every frame is worth the same effort. Ten thousand frames of a steady dissection teach a model less than fifty frames of the situations it gets wrong. Active learning lets the model choose what to label next.
MONAI Label is a server that sits between a viewer and a model: 3D Slicer or OHIF for CT and MRI, CVAT for video. Its endoscopy app ships a tool-tracking model, an interactive DeepEdit model and an in-body classifier, and it can use CVAT as its datastore. The server pre-annotates, trains on new labels and, through a strategy, picks the next image to label:
- random or first: a baseline.
- epistemic: the image the model is least sure about. The model runs T times with dropout switched on; where the runs disagree, the averaged prediction is spread over classes and its entropy is high.
Here p̄ is the prediction averaged over T dropout passes for pixel i and class c, and H the entropy summed over pixels: zero where every pass agrees on one class, large where they scatter. Images with the largest H come first.
- labelled points
- –
- accuracy on all points
- –
Uncertainty is not everything. It favours outliers and near-duplicates of the same hard frame, and it is unreliable while the model has seen almost nothing. Keep a share of random picks, start from a small random sample, and keep the validation set random and fixed.
Agreement between annotators
Labels have no ground truth to compare with. What can be measured is whether two or more competent people, given the same frames and the same guideline, produce the same labels. Low agreement means the class definition, the guideline or the training needs work, and it caps how good any model can look.
Cohen’s kappa, two annotators
Po is the observed agreement, the share of frames on which both chose the same class. Pe is the agreement expected by chance: for each class, the share of frames annotator A put in it times the share annotator B did, summed over classes. Kappa (Cohen, 1960) is how far the observed agreement is above chance, as a share of the way from chance to perfect: 1 is perfect agreement, 0 is no better than chance, below 0 worse.
Why not just Po? Because a rare class makes agreement cheap. If 95 % of frames show no smoke, two annotators who almost never mark smoke agree on most frames without agreeing on any smoke. Try the presets:
- observed agreement Po
- –
- chance agreement Pe
- –
- Cohen's kappa
- –
- Landis and Koch
- –
Landis and Koch (1977) named bands that are still quoted: below 0 poor, up to 0.20 slight, up to 0.40 fair, up to 0.60 moderate, up to 0.80 substantial, above that almost perfect. They are a convention, not a law; what is good enough depends on the task.
Fleiss’ kappa, many annotators
Fleiss’ kappa (1971) extends the idea to r annotators per item. For each of N items, nic annotators chose class c; Pi is the share of annotator pairs that agree on item i, P̄ its mean over items, and P̄e the chance agreement from the overall share pc of each class. The project’s guideline asks for a Fleiss kappa of at least 0.70 on categorical labels.
Masks: Dice between annotators
For masks the question is how much two drawings of the same structure overlap. Dice and IoU work exactly as for a model against ground truth (see the metrics page); with more than two annotators, average over all pairs:
The guideline asks for a mean pairwise Dice of at least 0.80. Where annotators keep disagreeing, for example on the boundaries of an inflamed gallbladder, the frame is marked needs-clinician and settled by the clinical lead. Several masks can also be merged into one consensus estimate that weighs each annotator by how reliable they appear, as STAPLE does (Warfield et al., 2004).
Checklist
- Annotate in CVAT, inside the closed workspace; never export frames to a personal tool.
- Use pre-annotation as a draft, and correct it as if you had drawn it.
- Leave control frames alone and never hide the acceptance-rate telemetry.
- Prefer labelling what the model is unsure about, with a share of random picks.
- Report agreement: Cohen’s or Fleiss’ kappa for categories, pairwise Dice for masks, with the number of items.
Further reading
- The project’s annotation guideline.
- CVAT documentation, Label Studio ML backends, MONAI Label.
- Ravi et al. SAM 2: Segment Anything in Images and Videos. 2024.
- Cohen, J. A Coefficient of Agreement for Nominal Scales. Educational and Psychological Measurement, 1960.
- Fleiss, J. L. Measuring nominal scale agreement among many raters. Psychological Bulletin, 1971.
- Landis and Koch. The Measurement of Observer Agreement for Categorical Data. Biometrics, 1977.
- Warfield et al. STAPLE. IEEE TMI, 2004.