A CT or MRI volume is a 3D grid of voxels. The tools that work on it look different from 2D computer vision for three reasons: the volume is large, its spacing is anisotropic, and anatomy needs context across tens of centimetres. This page covers the data, the frameworks, the main architectures and how patch size, GPU memory and sliding-window inference trade against each other. For video, see segmentation of surgical video; for an overview, the comparison of the two.
From DICOM to a tensor
- DICOM stores a scan as a series of 2D slices, each with a header: patient and study data, the position and orientation of the slice, pixel spacing, rescale slope and intercept. Headers carry personal data, and some images carry burned-in text, so de-identification comes before anything else; the project’s pipeline is in osv/deid.
- NIfTI stores the whole volume in one file with an affine matrix that maps voxel indices to millimetres. Most training code reads NIfTI; dcm2niix is the usual converter.
- Orientation and spacing. Pick one axis convention (RAS, for example) and one target spacing, and apply them the same way in training and inference. A typical abdominal CT has 0.6 to 0.9 mm pixels within a slice and 1 to 5 mm between slices.
- Intensities. CT values are Hounsfield units and comparable between scanners, so models clip them to a window and scale them. MRI intensities have no fixed scale, so models normalise each scan, usually with a z-score over the body or brain.
Frameworks
- MONAI Core is PyTorch for medical imaging: dictionary-based 3D transforms that keep image and label aligned,
CacheDatasetandPersistentDatasetthat keep the results of the deterministic transforms in memory or on disk,sliding_window_inference, losses and metrics such as Dice, Hausdorff distance and surface Dice, and bundles: packaged models with configs for training, evaluation and inference. MONAI Label serves such models to annotation tools (3D Slicer, OHIF) for interactive labelling. - nnU-Net (Isensee et al., Nature Methods 2021) reads a dataset’s spacing, size and intensity statistics and configures everything itself: target spacing, patch size, network depth, batch size, augmentation. It trains up to four configurations:
2d,3d_fullres,3d_lowresand3d_cascade_fullres, which refines the low-resolution result. Its residual-encoder presets are planned for GPU budgets: ResEnc M for 9 to 11 GB, L for 24 GB, XL for 40 GB. - PyTorch DDP spreads training over several GPUs. In 3D the batch per GPU is small, often two patches, so data parallelism mostly buys a larger effective batch and shorter wall time; sharded training (FSDP) matters only for models whose weights do not fit on one GPU, which is rare for segmentation networks.
Architectures
| Model | Idea | Reference |
|---|---|---|
| 3D U-Net | Encoder-decoder of 3D convolutions with skip connections between levels of the same resolution | Çiçek et al., MICCAI 2016 |
| V-Net | 3D U-Net with residual stages and a Dice loss | Milletari et al., 3DV 2016 |
| SegResNet | Residual encoder, light decoder; MONAI’s default for many bundles | Myronenko, BrainLes 2018 |
| UNETR | A plain vision transformer as the encoder, on 16³ patches of the volume, with a convolutional decoder | Hatamizadeh et al., WACV 2022 |
| Swin UNETR | A hierarchical Swin transformer with shifted-window attention as the encoder | Hatamizadeh et al., 2022; pre-training: Tang et al., CVPR 2022 |
| nnU-Net ResEnc | A residual-encoder U-Net inside nnU-Net, scaled to the GPU | Isensee et al., MICCAI 2024 |
Transformers promised global context, and on some benchmarks they report gains. The nnU-Net authors’ rigorous re-evaluation found that most such claims did not hold against a well-tuned convolutional baseline with the same compute: a residual U-Net in nnU-Net, scaled to the hardware, was the strongest recipe. Report your baseline’s configuration and budget next to any new model’s result.
3D U-Net, level by level
- convolution
- attention
- change of resolution
- memory, state
- input, output
- Convolution block. Two 3×3×3 convolutions, each followed by normalisation (instance norm in nnU-Net) and a nonlinearity. “32 · 96³” means 32 feature channels on a 96³ grid.
- Downsampling. A stride-2 convolution halves every side, so the next level holds an eighth of the voxels and can afford twice the channels.
- Bottleneck. The coarsest level, 6³ here: each voxel’s features summarise a large part of the patch.
- Upsampling. A transposed convolution doubles every side again.
- Skip connection. The encoder’s features of the same size are concatenated with the upsampled ones and restore boundaries the bottleneck could not keep.
- Output head. A 1×1×1 convolution maps the features to one score per class for every voxel; softmax turns the scores into probabilities.
- Deep supervision. nnU-Net also computes the loss on downsampled masks at lower decoder levels, which helps the deeper layers learn.
UNETR: a transformer as the encoder
- Patches as tokens. The 96³ volume is cut into 16³ patches: 6 × 6 × 6 = 216 tokens, each a vector of 4,096 voxel values.
- Embedding. A linear layer maps each patch to a vector of 768 values and adds a learned position embedding, because attention itself does not know where a token came from.
- Transformer layers. Twelve layers of multi-head self-attention and an MLP. Every token attends to every other from the first layer on.
- Taps. The outputs of layers 3, 6, 9 and 12 are reshaped back into 6³ grids and upsampled by transposed convolutions to 48³, 24³ and 12³; layer 12 stays at 6³.
- Decoder. As in a U-Net: upsample, concatenate with the tap of the same size, convolve.
- Output. A convolution on the input volume keeps full-resolution detail; the last decoder block joins it, and a 1×1×1 convolution gives the mask.
Swin UNETR: windows that shift
- Windows. Tokens attend only to tokens in the same window, so the cost grows linearly with the volume, not with its square.
- Shifted windows. The next layer moves the window grid by half a window: tokens that sat at a border now share a window with their neighbours across it.
- Stages. Patch merging joins 2 × 2 × 2 neighbouring tokens and halves the grid at each stage, like downsampling in a U-Net; the stage outputs feed a convolutional decoder through skip connections.
Whole-body and foundation models
- TotalSegmentator (Wasserthal et al., Radiology: AI 2023) is a set of nnU-Net models: 104 structures in the paper, 117 in the current default CT task, and an MRI task with 50. It is the quickest way to whole-body masks, and its dataset is a common source for training other models.
- VISTA-3D (He et al., 2024) combines automatic segmentation of 127 classes with interactive 3D segmentation from clicks and zero-shot segmentation of structures it was not trained on. It is available as the vista3d MONAI bundle, with 128³ patches.
Generating volumes with their masks
MAISI (Guo et al., WACV 2025) generates CT volumes with latent diffusion: an autoencoder compresses the volume, a diffusion model generates in the compressed space, and a ControlNet conditions the result on a segmentation mask of up to 127 structures. Because the mask comes first, every generated volume has its ground truth. The current maisi_ct_generative bundle adds Rectified Flow for faster sampling. The bundle’s weights carry a non-commercial licence; check it before you train on its output.
- Mask. A label volume with up to 127 anatomical structures, drawn by hand or taken from a real segmentation.
- ControlNet. A trainable copy of the diffusion network’s encoder that reads the mask and steers every denoising step.
- Latent diffusion. The model starts from noise and denoises step by step in the autoencoder’s compressed space, which makes 3D generation at high resolution affordable; Rectified Flow needs fewer steps.
- Decoder. The autoencoder’s decoder turns the latent into a full-resolution CT volume.
- The pair. The volume is generated to match the mask, so together they are ready for training a segmentation model.
Patch size, memory and the sliding window
A network trains on patches cut from the volume and predicts a whole volume by sliding a window of the same size across it. Larger patches give more context and fewer windows but need more memory; activation memory of a convolutional network grows roughly in proportion to the voxels in the patch. The overlap between windows trades time for smoother seams: predictions near a window’s edge are weaker, so overlapping windows are blended with a Gaussian weight that favours each window’s centre (MONAI’s mode="gaussian" with sigma_scale=0.125; nnU-Net uses the same σ of one eighth of the patch and a step of half a patch).
- Windows. Overlapping windows of the patch size cover the volume; the network predicts each one separately.
- Gaussian weight. Inside a window, predictions near the centre count more than those at the edge, where the network saw less context.
- Stitching. Every voxel’s prediction is the weighted average of all the windows that covered it.
Reference points from the model documentation:
| Model | Patch | Memory |
|---|---|---|
| MONAI wholeBody_ct_segmentation, Swin UNETR BTCV | 96³ | bundle defaults |
| MONAI vista3d | 128³ | bundle defaults |
| nnU-Net ResEnc M / L / XL | planned per dataset | 9–11 / 24 / 40 GB for training |
The calculator below counts the windows MONAI’s sliding_window_inference places over a volume, with its rule: a step of the patch size times one minus the overlap, windows until one reaches the far edge, and an axis shorter than the patch padded up to it.
- Windows
- 48
- Voxels per window
- 884,736
- Predictions per voxel, on average
- 3.24
- Activation memory per window, relative to 96³
- 1.00×
Worked example. An abdominal CT resampled to 1.5 mm is about 200 × 256 × 256 voxels. With 96³ windows and an overlap of 0.25 the step is 72 voxels: 3 windows along the depth and 4 along each in-plane axis, 48 windows in all, and every voxel is predicted 3.24 times on average. Doubling the patch edge to 192³ cuts the windows to 8 but needs about eight times the activation memory per window.
Ways to fit a larger model or patch: mixed precision, gradient checkpointing, a smaller batch with more accumulation steps, cropping to the body before patching, and running the sliding window with the volume on the CPU and only the windows on the GPU (MONAI’s sw_device and device arguments).
Measuring
The formulas behind these metrics, with pictures and an interactive example, are on the metrics page.
- Dice per class, never only the mean over classes: a large organ hides a missed small one.
- 95th-percentile Hausdorff distance (HD95) and normalised surface Dice for boundaries, in millimetres, using the voxel spacing.
- Report empty predictions and empty labels explicitly; Dice is undefined when both are empty. Metrics Reloaded (Maier-Hein et al., Nature Methods 2024) is a guide to choosing metrics for the task.
- Split by patient, never by slice or patch, and keep the test set untouched until the final run.
References
- Cardoso et al. MONAI: An open-source framework for deep learning in healthcare. 2022.
- Isensee et al. nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature Methods, 2021; nnU-Net Revisited. MICCAI 2024.
- Wasserthal et al. TotalSegmentator: Robust Segmentation of 104 Anatomic Structures in CT Images. Radiology: Artificial Intelligence, 2023.
- He et al. VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imaging. 2024.
- Guo et al. MAISI: Medical AI for Synthetic Imaging. WACV 2025.
- Datasets: BTCV (13 abdominal organs), KiTS19 and KiTS23 (kidney tumours), BraTS (brain tumours).
Back to the comparison of volumes and video.