Computer-assisted intervention works with two very different kinds of image data. Before and after an operation there are 3D volumes: CT and MRI scans stored as DICOM series or NIfTI files. During the operation there is a video stream from the laparoscope. Both are “segmentation”, but the constraint that shapes the model is different: GPU memory for volumes, time per frame for video. This guide compares the two and helps you pick a starting point. Two pages go deeper: 3D volumes and surgical video.
Read this first
- Research use only. Nothing here is a clinical tool or a recommendation for patient care.
- Starting points, not verdicts. The recommendations below are reasonable first baselines for a new task. Which model wins depends on your data, labels and budget; measure on your own validation split.
- Figures come from their sources. Class counts, patch sizes and speeds are quoted from the papers and model documentation linked on each page, on the hardware they name. The project has not measured them.
- The project has no released model yet. The M1 model card explains why; no checkpoint is published. Where this guide points to OpenSurgicalVision, it points to code, data loaders and evaluation, not to weights.
Two kinds of data
3D volumes: CT and MRI
A volume is a grid of voxels, often hundreds of millions of them, with a spacing that differs between axes: thin pixels within a slice, thicker steps between slices. A whole volume rarely fits on a GPU at full resolution together with the network’s activations, so models train on patches and predict with a sliding window. Time is rarely the limit: a few seconds to a minute per scan is usually acceptable.
- Frameworks: MONAI (3D transforms, cached datasets,
SlidingWindowInferer, model bundles), nnU-Net (a self-configuring pipeline that picks patch size, spacing and network from the dataset), PyTorch with DDP for multi-GPU training. - Architectures: 3D U-Net and its residual variants, SegResNet, UNETR and Swin UNETR; whole-body models such as TotalSegmentator and VISTA-3D; MAISI for generating volumes with their masks.
- Read more: segmentation of 3D volumes.
Surgical video
A laparoscopic video is a sequence of 2D colour frames, 25 to 60 per second. Each frame is small next to a CT volume, but the model has one frame interval to finish if it is to keep up with the camera, and its masks must not flicker from frame to frame. Smoke, blood, specular highlights, motion blur and instruments that leave and enter the view make the data hard.
- Frameworks: PyTorch and torchvision, the project’s own OpenSurgicalVision code for surgical datasets, augmentations and evaluation, and NVIDIA TensorRT and Holoscan for low-latency deployment.
- Architectures: per-frame encoder-decoders (DeepLabV3+, HRNet, SegFormer, EfficientNet U-Nets), recurrent and video-transformer models for temporal context, and SAM 2 for promptable segmentation with a memory of past frames.
- Read more: segmentation of surgical video.
Building blocks
Most architectures on these pages are assembled from the same few parts. The colours are the same in every diagram on the volumes and video pages.
- convolution
- attention
- change of resolution
- memory, state
- input, output
- Convolution. A small kernel of weights, 3×3 here and 3×3×3 in a volume, slides over the input and computes one output value per position. The same weights are used everywhere, so a pattern is recognised wherever it appears.
- Dilated (atrous) convolution. The kernel’s taps are spread apart. With a rate of 2 the nine weights cover a 5×5 area: a wider view without more weights and without lowering the resolution. DeepLab is built on it.
- Downsampling. A stride-2 convolution or pooling halves each side: four cells become one, eight voxels in a volume. Each level of an encoder sees a larger area on a coarser grid.
- Upsampling. A transposed convolution or interpolation doubles each side and brings coarse features back towards the input resolution in the decoder.
- Skip connection. The encoder’s feature map at a given resolution is copied to the decoder and concatenated with the upsampled features. It returns the fine detail, edges and thin structures, that downsampling lost.
- Self-attention. Every token, a patch of the image or volume, computes how much each other token matters to it and takes a weighted sum of their values. The view is global within one layer; the cost grows with the square of the number of tokens.
- Window attention. Attention only within fixed windows of tokens keeps the cost proportional to the number of tokens. Swin shifts the windows from one layer to the next so that information still crosses their borders.
- Residual block. The block learns a correction F that is added to its input. The identity path lets gradients pass through deep networks; nnU-Net’s residual encoder and SegResNet are built from such blocks.
Side by side
| 3D volumes (DICOM / NIfTI) | Surgical video (2D+t) | |
|---|---|---|
| Input tensor | C × D × H × W, one volume, trained on patches |
B × T × C × H × W, a sequence of frames |
| Limit that shapes the model | GPU memory: patch size × batch × network width | Time: one frame interval, 33 ms at 30 fps |
| Base frameworks | MONAI, nnU-Net | PyTorch, torchvision, TensorRT, Holoscan |
| Representative architectures | nnU-Net (residual U-Net), SegResNet, Swin UNETR, VISTA-3D | DeepLabV3+, SegFormer, SAM 2, video transformers |
| Hard parts of the data | Anisotropic spacing, scanner and protocol shifts, partial labels | Smoke, blood, glare, motion blur, occlusion |
| Usual metrics | Dice, 95th-percentile Hausdorff distance, surface Dice | IoU and mean IoU, boundary F1, frame-to-frame stability, latency p50 / p95 |
| Typical public data | TotalSegmentator, BTCV, KiTS, BraTS | CholecSeg8k, CholecT50, EndoVis 2017 |
A note on transformers for volumes: a careful benchmark by the nnU-Net authors, nnU-Net Revisited (MICCAI 2024), found that well-configured convolutional U-Nets, scaled to the available GPU, match or beat the transformer and Mamba models they compared. Start from a strong CNN baseline before reaching for a newer architecture.
Choose a starting point
Pick a task and the hardware the model has to run on. Without JavaScript all six recommendations are listed below.
Multi-organ CT · workstation GPU
Start with nnU-Net trained on your labels; the ResEnc L preset is planned for a 24 GB GPU. If the structures you need are already covered, try a ready model first: TotalSegmentator (117 classes in its default CT task), MONAI's wholeBody_ct_segmentation bundle (SegResNet, 104 structures) or vista3d (127 classes plus interactive correction with clicks).
- Preprocessing: de-identify, convert DICOM to NIfTI, reorient to one convention, resample to the model's spacing (1.5 mm for the wholeBody bundle), clip and normalise Hounsfield units the way the model was trained.
- Inference: sliding window with 96³ to 128³ patches, an overlap of 0.25 to 0.5 and Gaussian blending.
Multi-organ CT · edge GPU
Start with a lower-resolution model: TotalSegmentator with --fast, or the 3.0 mm model of the wholeBody_ct_segmentation bundle, which ships a TensorRT inference config. On an edge device memory, not time, is usually the limit for CT.
- Preprocessing: as on a workstation, resampled to 3 mm; expect small structures to suffer.
- Inference: half precision, the largest patch that fits, and fewer windows (a lower overlap, or a crop to the body before the sliding window).
Laparoscopic instruments · workstation GPU
Start with a supervised per-frame model, DeepLabV3 or DeepLabV3+ with a ResNet-50 backbone, trained on CholecSeg8k. This is the project's own M1 baseline: the training code is in osv/train/m1.py, with no released checkpoint. For tracking through occlusions, add SAM 2.1 prompted with the per-frame model's mask.
- Preprocessing: full 854 × 480 frames, splits by whole video so no operation appears in two splits, and the CC BY-NC-SA licence of the source data carried to the model.
- Evaluation: IoU and boundary F1 per class, plus how often masks flicker between neighbouring frames.
Laparoscopic instruments · edge GPU
Start with a light encoder-decoder: MONAI's endoscopic_tool_segmentation (a U-Net with an EfficientNet-B2 encoder, 736 × 480 input, binary mask, TensorRT config) or SegFormer-B0 / B1. The MONAI bundle was trained on private data, so fine-tune it on yours. SAM 2.1 tiny is the lightest SAM 2 model if you need propagation; measure it on the target device.
- Deployment: export to TensorRT in FP16, run the pipeline in Holoscan or a similar zero-copy framework, and measure glass-to-glass latency, p50 and p95, on the real device.
- Budget: try the latency budget to see where the milliseconds go.
Tumour in MRI · workstation GPU
Start with nnU-Net 3d_fullres on your labels. Ready models for brain tumours: MONAI's brats_mri_segmentation (SegResNet on four sequences, three tumour sub-regions) and Swin UNETR from the BraTS 2021 challenge.
- Preprocessing: co-register the sequences (T1, T1c, T2, FLAIR), strip the skull, resample to 1 mm isotropic, normalise each channel on brain voxels only.
- Evaluation: Dice and HD95 per sub-region; report cases where the model finds no tumour separately.
Tumour in MRI · edge GPU
Start with SegResNet from brats_mri_segmentation, which ships an ONNX-TensorRT inference config, rather than a transformer.
- Inference: crop to the brain's bounding box before the sliding window, use half precision, and keep the preprocessing identical to training: an edge device does not excuse a different normalisation.
Where to go next
- Metrics: how segmentation models are measured, from the confusion matrix to Hausdorff distance, with formulas and an interactive example.
- Model zoos and code: MONAI Model Zoo, nnU-Net, torchvision models, SAM 2, OpenSurgicalVision.
- Volumes: TotalSegmentator, BTCV, KiTS, BraTS.
- Video: CholecSeg8k, EndoVis 2017, and the project’s pages on datasets and experiments.
- Contribute: the engineering track and research topics.