Each stage is a research question
The illustrations below show the sequence: a 3D model before surgery, masks on video during surgery, a timeline afterwards. Each stage is a separate research problem with its own data, permissions and evaluation. It is not a working clinical system.
Tools:
- video is annotated in CVAT, with suggestions from the SAM 2 model;
- volumetric scans (CT, MRI) in 3D Slicer and MONAI Label.
Quality first, volume second
The clinical lead approves class definitions, rules for unclear cases and decisions on disputes. We check how well experts agree with each other:
- 10% of frames are annotated independently by several people;
- agreement is measured with Cohen’s or Fleiss’ κ, mask overlap with Dice and IoU;
- target thresholds: κ ≥ 0.70 and mean Dice ≥ 0.80.
When experts disagree, we clarify the guideline or mark the uncertainty explicitly instead of quietly averaging. Recurring disputes trigger a new version of the guideline.
Guarding against blind trust in suggestions
An automatic suggestion can quietly shift an annotator’s judgement. So we:
- give 10% of frames without suggestions, mixed in with the rest;
- compare the same person’s work with and without suggestions, at least two weeks apart;
- watch the share of masks accepted without a single edit, and unusually fast sessions.
If a problem shows up, suggestions for that class are switched off until it is reviewed.
Speed: a target, not a result
The target is at most 30–50 ms per frame and at least 30 frames per second. It is a goal, not a measurement. Future speed reports will state the hardware, input size, numeric precision and p50/p95/p99 latency.