FOR ENGINEERS

Segmentation architectures: from CT volumes to surgical video

A comparative guide for engineers to the frameworks and models used to segment 3D medical volumes (CT, MRI) and laparoscopic video in real time, and to choosing a starting point.

Computer-assisted intervention works with two very different kinds of image data. Before and after an operation there are 3D volumes: CT and MRI scans stored as DICOM series or NIfTI files. During the operation there is a video stream from the laparoscope. Both are “segmentation”, but the constraint that shapes the model is different: GPU memory for volumes, time per frame for video. This guide compares the two and helps you pick a starting point. Two pages go deeper: 3D volumes and surgical video.

Read this first

  • Research use only. Nothing here is a clinical tool or a recommendation for patient care.
  • Starting points, not verdicts. The recommendations below are reasonable first baselines for a new task. Which model wins depends on your data, labels and budget; measure on your own validation split.
  • Figures come from their sources. Class counts, patch sizes and speeds are quoted from the papers and model documentation linked on each page, on the hardware they name. The project has not measured them.
  • The project has no released model yet. The M1 model card explains why; no checkpoint is published. Where this guide points to OpenSurgicalVision, it points to code, data loaders and evaluation, not to weights.

Two kinds of data

3D volumes: CT and MRI

A volume is a grid of voxels, often hundreds of millions of them, with a spacing that differs between axes: thin pixels within a slice, thicker steps between slices. A whole volume rarely fits on a GPU at full resolution together with the network’s activations, so models train on patches and predict with a sliding window. Time is rarely the limit: a few seconds to a minute per scan is usually acceptable.

  • Frameworks: MONAI (3D transforms, cached datasets, SlidingWindowInferer, model bundles), nnU-Net (a self-configuring pipeline that picks patch size, spacing and network from the dataset), PyTorch with DDP for multi-GPU training.
  • Architectures: 3D U-Net and its residual variants, SegResNet, UNETR and Swin UNETR; whole-body models such as TotalSegmentator and VISTA-3D; MAISI for generating volumes with their masks.
  • Read more: segmentation of 3D volumes.

Surgical video

A laparoscopic video is a sequence of 2D colour frames, 25 to 60 per second. Each frame is small next to a CT volume, but the model has one frame interval to finish if it is to keep up with the camera, and its masks must not flicker from frame to frame. Smoke, blood, specular highlights, motion blur and instruments that leave and enter the view make the data hard.

  • Frameworks: PyTorch and torchvision, the project’s own OpenSurgicalVision code for surgical datasets, augmentations and evaluation, and NVIDIA TensorRT and Holoscan for low-latency deployment.
  • Architectures: per-frame encoder-decoders (DeepLabV3+, HRNet, SegFormer, EfficientNet U-Nets), recurrent and video-transformer models for temporal context, and SAM 2 for promptable segmentation with a memory of past frames.
  • Read more: segmentation of surgical video.

Building blocks

Most architectures on these pages are assembled from the same few parts. The colours are the same in every diagram on the volumes and video pages.

  • convolution
  • attention
  • change of resolution
  • memory, state
  • input, output
3×3 convolutionsame weights everywhere1Dilated convolutionrate 2: wider view2Downsampling ×2stride 2 or pooling3Upsampling ×2transposed conv4Skip connectionencoder detail to decoder5Self-attentioneach token weighs all6Window attentiononly inside a window7Residual blockoutput = input + F(input)8concatconvnormact+identity
Each block drawn on a small 2D grid; in a volume the same operations work on 3D grids of voxels.
  1. Convolution. A small kernel of weights, 3×3 here and 3×3×3 in a volume, slides over the input and computes one output value per position. The same weights are used everywhere, so a pattern is recognised wherever it appears.
  2. Dilated (atrous) convolution. The kernel’s taps are spread apart. With a rate of 2 the nine weights cover a 5×5 area: a wider view without more weights and without lowering the resolution. DeepLab is built on it.
  3. Downsampling. A stride-2 convolution or pooling halves each side: four cells become one, eight voxels in a volume. Each level of an encoder sees a larger area on a coarser grid.
  4. Upsampling. A transposed convolution or interpolation doubles each side and brings coarse features back towards the input resolution in the decoder.
  5. Skip connection. The encoder’s feature map at a given resolution is copied to the decoder and concatenated with the upsampled features. It returns the fine detail, edges and thin structures, that downsampling lost.
  6. Self-attention. Every token, a patch of the image or volume, computes how much each other token matters to it and takes a weighted sum of their values. The view is global within one layer; the cost grows with the square of the number of tokens.
  7. Window attention. Attention only within fixed windows of tokens keeps the cost proportional to the number of tokens. Swin shifts the windows from one layer to the next so that information still crosses their borders.
  8. Residual block. The block learns a correction F that is added to its input. The identity path lets gradients pass through deep networks; nnU-Net’s residual encoder and SegResNet are built from such blocks.

Side by side

3D volumes (DICOM / NIfTI) Surgical video (2D+t)
Input tensor C × D × H × W, one volume, trained on patches B × T × C × H × W, a sequence of frames
Limit that shapes the model GPU memory: patch size × batch × network width Time: one frame interval, 33 ms at 30 fps
Base frameworks MONAI, nnU-Net PyTorch, torchvision, TensorRT, Holoscan
Representative architectures nnU-Net (residual U-Net), SegResNet, Swin UNETR, VISTA-3D DeepLabV3+, SegFormer, SAM 2, video transformers
Hard parts of the data Anisotropic spacing, scanner and protocol shifts, partial labels Smoke, blood, glare, motion blur, occlusion
Usual metrics Dice, 95th-percentile Hausdorff distance, surface Dice IoU and mean IoU, boundary F1, frame-to-frame stability, latency p50 / p95
Typical public data TotalSegmentator, BTCV, KiTS, BraTS CholecSeg8k, CholecT50, EndoVis 2017

A note on transformers for volumes: a careful benchmark by the nnU-Net authors, nnU-Net Revisited (MICCAI 2024), found that well-configured convolutional U-Nets, scaled to the available GPU, match or beat the transformer and Mamba models they compared. Start from a strong CNN baseline before reaching for a newer architecture.

Choose a starting point

Pick a task and the hardware the model has to run on. Without JavaScript all six recommendations are listed below.

Multi-organ CT · workstation GPU

Start with nnU-Net trained on your labels; the ResEnc L preset is planned for a 24 GB GPU. If the structures you need are already covered, try a ready model first: TotalSegmentator (117 classes in its default CT task), MONAI's wholeBody_ct_segmentation bundle (SegResNet, 104 structures) or vista3d (127 classes plus interactive correction with clicks).

  • Preprocessing: de-identify, convert DICOM to NIfTI, reorient to one convention, resample to the model's spacing (1.5 mm for the wholeBody bundle), clip and normalise Hounsfield units the way the model was trained.
  • Inference: sliding window with 96³ to 128³ patches, an overlap of 0.25 to 0.5 and Gaussian blending.

Multi-organ CT · edge GPU

Start with a lower-resolution model: TotalSegmentator with --fast, or the 3.0 mm model of the wholeBody_ct_segmentation bundle, which ships a TensorRT inference config. On an edge device memory, not time, is usually the limit for CT.

  • Preprocessing: as on a workstation, resampled to 3 mm; expect small structures to suffer.
  • Inference: half precision, the largest patch that fits, and fewer windows (a lower overlap, or a crop to the body before the sliding window).

Laparoscopic instruments · workstation GPU

Start with a supervised per-frame model, DeepLabV3 or DeepLabV3+ with a ResNet-50 backbone, trained on CholecSeg8k. This is the project's own M1 baseline: the training code is in osv/train/m1.py, with no released checkpoint. For tracking through occlusions, add SAM 2.1 prompted with the per-frame model's mask.

  • Preprocessing: full 854 × 480 frames, splits by whole video so no operation appears in two splits, and the CC BY-NC-SA licence of the source data carried to the model.
  • Evaluation: IoU and boundary F1 per class, plus how often masks flicker between neighbouring frames.

Laparoscopic instruments · edge GPU

Start with a light encoder-decoder: MONAI's endoscopic_tool_segmentation (a U-Net with an EfficientNet-B2 encoder, 736 × 480 input, binary mask, TensorRT config) or SegFormer-B0 / B1. The MONAI bundle was trained on private data, so fine-tune it on yours. SAM 2.1 tiny is the lightest SAM 2 model if you need propagation; measure it on the target device.

  • Deployment: export to TensorRT in FP16, run the pipeline in Holoscan or a similar zero-copy framework, and measure glass-to-glass latency, p50 and p95, on the real device.
  • Budget: try the latency budget to see where the milliseconds go.

Tumour in MRI · workstation GPU

Start with nnU-Net 3d_fullres on your labels. Ready models for brain tumours: MONAI's brats_mri_segmentation (SegResNet on four sequences, three tumour sub-regions) and Swin UNETR from the BraTS 2021 challenge.

  • Preprocessing: co-register the sequences (T1, T1c, T2, FLAIR), strip the skull, resample to 1 mm isotropic, normalise each channel on brain voxels only.
  • Evaluation: Dice and HD95 per sub-region; report cases where the model finds no tumour separately.

Tumour in MRI · edge GPU

Start with SegResNet from brats_mri_segmentation, which ships an ONNX-TensorRT inference config, rather than a transformer.

  • Inference: crop to the brain's bounding box before the sliding window, use half precision, and keep the preprocessing identical to training: an edge device does not excuse a different normalisation.

Where to go next