The project’s models are numbered M1 to M5. Each answers one question about an operation: where the instruments are, where the anatomy is, which phase the operation is in, what the organs look like in 3D before it starts, and what lies behind the smoke. This lecture takes them one at a time: input, output, data, metric, a starting architecture, and an honest status. It is lecture 7 of the learning programme, after lecture 6 on preprocessing.
Every image on this page is synthetic or drawn by code. All five are research tasks: RESEARCH USE ONLY – NOT FOR CLINICAL USE, and no model has been released.
| Task | Output | Public data | Main metric | Starting architecture | Status |
|---|---|---|---|---|---|
| M1 instruments | a mask per instrument part or type | CholecSeg8k, EndoVis 2017 | IoU, Dice, boundary F1 | nnU-Net 2D or Mask2Former; DeepLabV3 in RFC 0001 | experiment finished |
| M2 anatomy | a mask per structure | CholecSeg8k, Endoscapes | IoU and Dice per class | SAM 2, fine-tuned | planned |
| M3 phases | a phase for every frame | Cholec80, CholecT50 | accuracy, F1 per phase | TeCNO, a temporal convolutional network | planned |
| M4 3D organs | a 3D mask per organ and vessel | TotalSegmentator | Dice, surface distance | nnU-Net 3D, TotalSegmentator | planned |
| M5 smoke | a clean frame | synthetic smoke over CholecT50 | PSNR, SSIM | residual U-Net | first baselines |
The architectures are the starting points named in osv/models, not decisions. Metrics are explained on the metrics page, architectures on the engineering pages.
M1. Instrument segmentation
M1 finds every pixel that belongs to an instrument and says what it is. Labels come at three levels of detail, the ones the EndoVis 2017 challenge used (Allan et al., 2019):
- binary: instrument or not;
- type: grasper, hook, scissors, clip applier;
- parts: shaft, joint and jaws.
The levels answer different questions. A binary mask is enough to know that something is in the way. The type tells you what the surgeon is doing. The jaws are what touches tissue, so they are what a proximity warning has to watch, while the shaft only passes through the field.
EndoVis 2017 labels parts on robotic da Vinci instruments in porcine procedures. CholecSeg8k labels two instrument types, grasper and L-hook, with no parts. A part-level M1 for laparoscopic instruments therefore needs labels the public data does not have yet.
Where it stands. The first M1 experiment, RFC 0001, asked whether rendered instruments could replace real labels for a binary grasper-and-hook mask. On seven real test videos, a DeepLabV3-ResNet50 trained on real labels reached an IoU of 0.772, and the same model trained only on synthetic instruments reached 0.116. The hypothesis was rejected, and the result is published with its negative half.
M2. Anatomy segmentation
M2 outlines the anatomy of a cholecystectomy: the gallbladder, the liver, the hepatoduodenal ligament and, above all, the two critical structures, the cystic duct and the cystic artery. Cutting the wrong duct injures the common bile duct, one of the most serious complications of the operation. The surgeon’s safeguard is the critical view of safety (CVS), described on the cholecystectomy page. Before anything is clipped, exactly two structures, the cystic duct and the cystic artery, must be seen entering the gallbladder. A model that finds those two structures reliably could one day help check that view.
The data is thin exactly where it matters:
- CholecSeg8k has the liver, gallbladder and cystic duct among its 13 classes. The cystic duct appears in 241 of 8,080 frames, and the cystic artery is missing.
- Endoscapes (Murali et al., 2023) labels the gallbladder, cystic duct, cystic artery, cystic plate and the dissected hepatocystic triangle. It has segmentation masks for 493 frames from 50 of its 201 videos and CVS ratings by three experts for 11,090 frames.
- The hepatoduodenal ligament is labelled in neither.
Which structures M2 segments, and where one ends and the next begins, is the clinical lead’s decision (rule R-14), not an engineer’s. The registry names a fine-tuned SAM 2 as the starting point, the model whose prompts and memory were covered in lecture 3.
M3. Phase recognition
M3 names the phase of the operation at every moment. Cholec80 defines seven phases, each walked through on the cholecystectomy page: preparation, Calot triangle dissection, clipping and cutting, gallbladder dissection, gallbladder packaging, cleaning and coagulation, and gallbladder retraction. Clipping and cutting are one phase there, and the gallbladder comes out in the last.
A single frame is often ambiguous: dissection looks the same in phase 2 and in phase 4. The order of events resolves it, so M3 is a sequence model. In the style of TeCNO (Czempiel et al., 2020), a 2D network turns each frame into a feature vector, and a temporal convolutional network reads the sequence.
Causal means the prediction at time t uses only frames up to t, so the model can run during the operation. Dilated means the gaps between the frames a layer combines double from layer to layer, so the view into the past grows exponentially with depth:
With kernels of 3 and 10 layers, one output sees 2,047 frames, over half an hour at one frame a second.
Why time matters, in a toy: a ten-minute operation with made-up phase lengths, and a per-second classifier that is right three times out of four.
- per second: accuracy · phase changes
- –
- smoothed: accuracy · phase changes
- –
The true operation changes phase 6 times. The raw classifier flickers about 240 times. A majority vote over a window removes the flicker; online, it also arrives late at every true change, and a window that is too long erases short phases. A trained temporal network learns this trade-off instead of fixing one window. Accuracy alone hides flicker, so phase recognition is also reported with F1 or Jaccard per phase.
M4. Preoperative 3D segmentation
M4 works before the operation, on CT or MRI: a 3D mask of every organ and vessel, from which a surgeon can see the liver, the gallbladder and the vessels of a particular patient. The starting points are nnU-Net (Isensee et al.), which configures a 3D U-Net from the dataset itself, and TotalSegmentator, an nnU-Net trained on more than a thousand CTs. The preparation is lecture 6; sliding-window inference and the 3D architectures are on the volumes page. Dice says how much of an organ is right, and the surface distances say how far its edge is off, which matters more for a thin vessel.
M4 is the only task on scans rather than video, and it brings the 3D concerns of the earlier lectures with it: de-identification of the head and the DICOM header (lecture 2), splitting by patient rather than by scan (lecture 5), and memory.
M5. Smoke removal and image restoration
M5 takes a smoky frame and returns the clean frame behind it. The training pairs are synthetic, Blender smoke composited over clean CholecT50 frames by osv.augment, as described in lecture 4. The metrics are PSNR and SSIM against the clean frame, always next to the score of the smoky input left as it is.
Where it stands. On 518 synthetic validation pairs, a residual U-Net reached 33.08 dB PSNR against 28.99 dB for the input; Pix2Pix scored 27.85 dB and damages lightly smoked frames; SRGAN did not converge in three attempts. This is one seed on the validation split, with the test untouched and no model released, and it has not been measured on real smoke (experiment log). A restored frame must not invent anatomy: a model that paints plausible tissue where the smoke was would be worse than no model.
What the tasks share
- The same rules. Patient-level splits, a test set touched once, at least three seeds, and a model card before any release.
- Shared data. M1, M2 and M3 can learn from the same videos, and M5 could clean frames before the others see them.
- Clinical authority. What counts as a class or a phase is the clinical lead’s decision, for every task.
Checklist
- Know which task your work serves, and what its input and output are.
- Check whether the public data has the labels your task needs before you plan a model.
- Report the task’s main metric next to its baseline: the smoky input for M5, a per-frame model for M3.
- Leave class and phase definitions to the clinical lead.
Further reading
- The project’s experiment log and its model cards for M1 and M5.
- Allan et al. 2017 Robotic Instrument Segmentation Challenge. 2019.
- Murali et al. The Endoscapes Dataset for Surgical Scene Segmentation, Object Detection, and Critical View of Safety Assessment. 2023.
- Czempiel et al. TeCNO: Surgical Phase Recognition with Multi-Stage Temporal Convolutional Networks. MICCAI 2020.
- Isensee et al. nnU-Net: Self-adapting Framework for U-Net-Based Medical Image Segmentation. 2018.