Lecture 7 named the project’s tasks. This lecture is about the models that solve them, and about training them when one GPU is not enough. It is lecture 8 of the curriculum, after lecture 7 on the tasks M1 to M5. Three topics:
- Transformers for volumes: UNETR and SwinUNETR, and what changes when a U-Net’s encoder is replaced by attention.
- VISTA-3D: one model for automatic segmentation and for corrections by clicks.
- Scaling: DDP, FSDP and
bf16-mixedprecision in Lightning.
The code is checked against MONAI 1.6.1 and Lightning 2.6.6. The basics of the 3D U-Net, patch size and sliding-window inference are on the volumes page and are taken as known here. Every image on this page is drawn by code.
Why attention, and what it costs
A 3 × 3 × 3 convolution sees 27 neighbouring voxels. Its field of view grows only with the depth of the network, and linking the liver to the spleen on the other side of the abdomen takes many layers. Self-attention links any two tokens in the first layer: every token weighs every other.
The price is the number of pairs. A volume of side D cut into patches of side p gives N tokens, and global attention scores N² pairs in every layer and every head. Attention in windows of M³ tokens scores only N · M³ pairs. The numbers below are for a 96³ patch, D = 96:
Hence two answers and two architectures. UNETR takes large 16³ patches, and there are so few tokens that global attention is affordable. SwinUNETR takes small 2³ patches and pays with windows: on 110,592 tokens, one head’s global attention matrix in bf16 would weigh about 24 GB.
UNETR: a ViT as the encoder
UNETR (Hatamizadeh et al., WACV 2022) takes a plain vision transformer, ViT-B, and puts it in place of a U-Net’s encoder. The decoder stays convolutional.
In MONAI:
from monai.networks.nets import UNETR
model = UNETR(
in_channels=1, out_channels=14, # BTCV: 13 organs and background
img_size=(96, 96, 96), # a multiple of 16 on every axis
feature_size=16, hidden_size=768, mlp_dim=3072, num_heads=12,
proj_type="conv", norm_name="instance", res_block=True,
)
What the code shows, and what follows from it:
- The patch is fixed. In
unetr.pythe patch size is the constant 16, andimg_sizeis required because the position embeddings are learned for exactly 6 × 6 × 6 positions. The model is tied to its input size: inference runs a sliding window of the same size, 96³. - Taps. The decoder takes
hidden_states_out[3],[6]and[9], the outputs of the 4th, 7th and 10th blocks counting from one, and the output of the last layer. The paper calls them z3, z6, z9 and z12; it is easy to mix them up when comparing with the code. - Detail comes from convolutions. One token covers 16 mm at 1 mm spacing. The decoder takes fine edges from a convolutional branch that reads the original volume at full resolution.
- Size. 92.8 million parameters (counted in MONAI 1.6.1), five times a SegResNet with 32 initial filters (18.8 million).
SwinUNETR: small patches, windows and a shift
SwinUNETR (Hatamizadeh et al., 2022) takes the encoder of the Swin Transformer (Liu et al., ICCV 2021). It is built like a convolutional encoder: four stages, each halving the grid.
- 2³ patches. There are many tokens, and each is small: 2 mm at 1 mm spacing.
- 7³ windows. Attention is computed only inside a window of 343 tokens, so the cost grows linearly with the volume.
- The shift. In every second layer the grid of windows moves by half a window,
7 // 2 = 3tokens on each axis, and tokens at a window’s border share a window with their neighbours across it. - Patch merging. Between stages, 2 × 2 × 2 neighbouring tokens merge into one and the channels double: a hierarchy of features, as in a U-Net, whose outputs reach a convolutional decoder through skip connections.
The shift is done cleverly. Simply moving the windows would leave partial windows at the edges of the volume, and more of them. MONAI instead rolls the tokens themselves, torch.roll by −3 on each axis, cuts the same number of windows as before and applies an attention mask: tokens that wrapped round from the opposite side do not see each other inside a window. After attention, the tokens are rolled back.
Why the shift matters shows in a toy. A 16 × 16 grid of tokens with 4 × 4 windows, as the volumes page draws Swin in 2D. Click a token and see how many tokens its information reaches in L layers.
Without the shift, information stays in its window of 16 tokens forever, however many layers are added. With it, every two layers widen the reach, and patch merging between stages speeds the growth further.
In MONAI 1.6.1:
from monai.networks.nets import SwinUNETR
model = SwinUNETR(
in_channels=1, out_channels=14,
feature_size=48, # 24 by default; 48 matches the pretrained weights
use_checkpoint=True, # recompute activations in the backward pass
)
# patch_size=2, depths=(2, 2, 2, 2), num_heads=(3, 6, 12, 24), window_size=7 are the defaults
- No
img_size. Older examples passimg_size=(96, 96, 96); current MONAI has no such parameter. Instead,forwardchecks that every side of the input divides bypatch_size ** 5, that is by 32. 96 and 128 work. A video frame of 854 × 480 for the 2D variant (spatial_dims=2) needs padding to 864 in width: 480 divides by 32, 854 does not. - Size. 62.2 million parameters with
feature_size=48, fewer than UNETR, but far more activations in memory: 512 times more tokens at the first stage. The numbers are in the memory section below. use_v2=Trueadds a residual convolution block at the start of every stage (SwinUNETR-V2).- Pretraining. Tang et al. (CVPR 2022) pretrained the encoder self-supervised on 5,050 public CT scans, then fine-tuned on BTCV and the Medical Segmentation Decathlon. Weights and an example are in the MONAI tutorial SwinUNETR on BTCV.
Should you take a transformer. The nnU-Net authors’ re-evaluation (Isensee et al., MICCAI 2024) showed that at equal budget a well-tuned convolutional U-Net does not lose to UNETR or SwinUNETR, and more often wins. The starting point for M4 is still nnU-Net. A transformer pays off when there are pretrained weights for your modality, or so much data that pretraining earns its cost. A comparison is published together with a baseline of the same budget.
VISTA-3D: automatic and clicks in one model
VISTA-3D (He et al., 2024) is a foundation model for CT. It is trained on 11,454 volumes and segments 127 classes of anatomy and lesions automatically. It also takes clicks in 3D and corrects its answer by them, and with clicks it can outline a structure that was not in its training.
What is inside, by MONAI’s code (monai/networks/nets/vista3d.py):
- The encoder is convolutional. It is
SegResNetDS2, not a transformer: “foundation” here means trained on a large and varied set, not built on attention. 218.0 million parameters in all, nearly all of them in the encoder; the heads weigh less than a million. - The class head. Every class number has a learned embedding. A class’s mask is the dot product of that embedding with every voxel’s features, then a sigmoid. Classes do not compete through a softmax, so a new class in fine-tuning is a new embedding, not a new output layer.
- The point head. Built like SAM’s decoder: the clicks’ coordinates and signs (+ or −) become tokens that read the volume’s features and give a binary mask of the object under the clicks.
- The merge. Given both a class and clicks,
gaussian_combinereplaces the automatic answer near each click with the point head’s answer, with a weight that falls off with distance as a Gaussian.
So the model closes the working loop of lecture 3: the automatic branch labels a cohort, and a person corrects its mistakes with a few clicks instead of redrawing everything. Try it on a slice of the lecture 1 phantom. The automatic answer here has two mistakes: the liver spills over its edge into fat and stomach, and the back of the spleen is missed. Pick a class and a kind of click, and click where the mistake is.
- Dice of the automatic answer for the class
- –
- Dice after the clicks
- –
A minus on the fat at the liver’s edge removes the excess, a plus on the back of the spleen adds what was missed. Everything beyond the reach stays as it was. The toy simplifies two things: its point head is perfect and returns exactly the organ that was clicked, and the border of a click’s reach is sharp, where VISTA-3D blends the two answers smoothly.
How to run it. The model ships as the MONAI bundle vista3d with a 128³ patch; when memory runs short, the bundle’s README suggests reducing patch_size to 96³. Commands from the README:
python -m monai.bundle download "vista3d" --bundle_dir "bundles/"
cd bundles/vista3d
# automatic, one class by its number
python -m monai.bundle run --config_file configs/inference.json --input_dict "{'image':'spleen_03.nii.gz','label_prompt':[3]}"
# by clicks: points in voxels, 1 for the object, 0 for background
python -m monai.bundle run --config_file configs/inference.json --input_dict "{'image':'spleen_03.nii.gz','points':[[128,128,16],[100,100,16]],'point_labels':[1,0]}"
Class numbers are in the bundle’s configs/metadata.json.
Where the “hundreds of classes” are. VISTA-3D itself has 127. Hundreds belong to its successor NV-Segment-CTMR: it starts from the VISTA-3D checkpoint, is fine-tuned on more than 30,000 CT and MRI scans and segments over 300 classes, but only automatically, without clicks.
Licences. The code is Apache 2.0. For the weights the sources disagree: the VISTA repository’s README names the NVIDIA Open Model License for VISTA-3D, which allows commercial use, while the model zoo bundle’s README names the NCLS v1 licence. The NV-Segment-CTMR weights are non-commercial. Before building anything but research on the weights, check the licence of the exact file you downloaded.
For the project. VISTA-3D and TotalSegmentator are starting points for M4, preoperative 3D segmentation. Both work on CT, not on laparoscopic video. Patient scans they run on go through de-identification and stay where they are stored.
Training on several GPUs
Lightning in brief
Lightning separates what a model computes from where and how it computes. The model is a LightningModule with training_step, validation_step and configure_optimizers. Devices, the distribution strategy and precision are arguments of the Trainer. Lightning starts the processes, wraps the model, sets a DistributedSampler and gathers metrics from every GPU.
import lightning as L
import torch
from monai.inferers import sliding_window_inference
from monai.losses import DiceCELoss
from monai.networks.nets import SwinUNETR
class Segmenter(L.LightningModule):
def __init__(self):
super().__init__()
self.net = SwinUNETR(in_channels=1, out_channels=14, feature_size=48, use_checkpoint=True)
self.loss = DiceCELoss(to_onehot_y=True, softmax=True)
def training_step(self, batch, batch_idx):
loss = self.loss(self.net(batch["image"]), batch["label"])
self.log("train/loss", loss, sync_dist=True)
return loss
def validation_step(self, batch, batch_idx):
logits = sliding_window_inference(batch["image"], roi_size=(96, 96, 96), sw_batch_size=4,
predictor=self.net, overlap=0.5)
self.log("val/loss", self.loss(logits, batch["label"]), sync_dist=True)
def configure_optimizers(self):
return torch.optim.AdamW(self.parameters(), lr=1e-4, weight_decay=1e-5)
trainer = L.Trainer(accelerator="gpu", devices=4, strategy="ddp", precision="bf16-mixed", max_epochs=300)
trainer.fit(Segmenter(), train_loader, val_loader)
The loaders come from lecture 6. Going from one GPU to four, or from DDP to FSDP, changes one line of the Trainer, not the model.
DDP: a copy of the model on every GPU
Distributed Data Parallel starts one process per GPU. Each process holds a full copy of the model and its own part of the batch. In the backward pass the gradients are averaged over all GPUs by an all-reduce, in buckets, overlapping with computation. Then every copy takes the same step and the copies stay identical.
Three consequences:
- The effective batch is N · b: four GPUs with two patches each give a batch of 8. The learning rate is tuned to the effective batch.
- BatchNorm on each GPU computes statistics over its own two patches. MONAI’s 3D networks use instance norm by default, which does not mind. If a model has BatchNorm, the
Trainerhas async_batchnorm=Trueflag. - Logs from all GPUs are averaged only with
sync_dist=True, as in the example above.
FSDP: each GPU holds a share of everything
Fully Sharded Data Parallel divides not only the data between GPUs but the model itself. Each GPU stores 1/N of the weights, gradients and optimiser state. Before each layer its weights are gathered in full (all-gather), used and dropped, and the gradients go to their owners (reduce-scatter). It is the idea of ZeRO (Rajbhandari et al., 2020), implemented in PyTorch (Zhao et al., 2023).
How much it saves, counting Adam, Ψ parameters, N GPUs, b patches per GPU and A bytes of activations per patch:
In mixed precision the weights and gradients stay in float32, 4 bytes each, and Adam’s two moments add 8 bytes. The last line is how many bytes each GPU sends per step in DDP with a ring all-reduce.
In Lightning:
from lightning.pytorch.strategies import FSDPStrategy
from monai.networks.nets.swin_unetr import SwinTransformerBlock
strategy = FSDPStrategy(
auto_wrap_policy={SwinTransformerBlock}, # the unit gathered and dropped
activation_checkpointing_policy={SwinTransformerBlock},
sharding_strategy="FULL_SHARD", # or SHARD_GRAD_OP, HYBRID_SHARD
)
trainer = L.Trainer(accelerator="gpu", devices=4, strategy=strategy, precision="bf16-mixed")
FULL_SHARD shards everything. SHARD_GRAD_OP keeps the weights whole and shards gradients and the optimiser. HYBRID_SHARD shards within a node and replicates across nodes. The string strategy="fsdp" gives FULL_SHARD with default settings.
The main point for 3D: activations, not weights, fill the memory. We measured it in MONAI 1.6.1, for one 96³ patch, counting the tensors autograd keeps for the backward pass, without the weights themselves.
| Model | Parameters | Weights, gradients and Adam, 16Ψ | Activations per patch, float32 | Activations per patch, bf16-mixed |
|---|---|---|---|---|
| UNETR | 92.8 M | 1.5 GB | 1.06 GB | 0.73 GB |
SwinUNETR, feature_size=48 |
62.2 M | 1.0 GB | 4.84 GB | 3.98 GB |
the same with use_checkpoint=True |
62.2 M | 1.0 GB | 2.75 GB | 1.59 GB |
| VISTA-3D encoder (automatic branch) | 218.0 M | 3.5 GB | 2.51 GB | 1.65 GB |
The measurement ran on a CPU, so it lacks cuDNN workspaces and memory fragmentation: real use on a GPU is higher. The proportion does not change. In SwinUNETR the activations of one patch weigh three to five times the weights, gradients and Adam together. In UNETR, with its 216 tokens, they are comparable, and overtake them from two patches per GPU. FSDP shards the “16Ψ” column, and for all three models it fits on one GPU. FSDP does not shard activations: each GPU computes its own patches. So DDP is usually enough for segmentation networks, and memory is saved elsewhere: a smaller patch, recomputed activations (use_checkpoint), bf16-mixed and one or two patches per GPU. FSDP is needed when 16Ψ does not fit, which means models from a billion parameters up, not the segmentation networks of this lecture.
Check for yourself. The bar is one GPU’s memory, the dashes are typical card sizes.
- memory of one GPU
- –
- of which weights, gradients and Adam
- –
- effective batch
- –
- sent by each GPU per step
- –
Switching from DDP to FULL_SHARD shrinks the blue, green and purple parts but not the orange one. Traffic per step follows Lightning’s settings: DDP averages gradients in float32, while FSDP with bf16-mixed gathers the weights in bfloat16 twice (for the forward and the backward pass) and scatters the gradients in bfloat16.
bf16-mixed: float32’s range in 16 bits
A floating-point number is a sign s, an exponent e of a few bits with a bias b, and a mantissa m of k bits:
Exponent bits set the range, mantissa bits the precision. float16 and bfloat16 both take 16 bits but split them differently:
| Format | Exponent bits | Mantissa bits | Largest number | Smallest normal | Step after 1 |
|---|---|---|---|---|---|
| float32 | 8 | 23 | 3.4 · 10³⁸ | 1.2 · 10⁻³⁸ | 1.2 · 10⁻⁷ |
| float16 | 5 | 10 | 65,504 | 6.1 · 10⁻⁵ | 9.8 · 10⁻⁴ |
| bfloat16 | 8 | 7 | 3.4 · 10³⁸ | 1.2 · 10⁻³⁸ | 7.8 · 10⁻³ |
float16 also has subnormal numbers down to 6.0 · 10⁻⁸, and anything smaller becomes zero. bfloat16 has float32’s exponent, hence the same range, and a precision of about two to three significant digits.
What “mixed” means. With bf16-mixed, Lightning turns on torch.autocast: convolutions and matrix multiplications run in bfloat16 on tensor cores, activations take less memory, and the weights, their update, normalisations, softmax and loss functions stay in float32. With bf16-true, everything is stored in bfloat16, the weights included.
Why it is steadier than float16. A deep network on a large volume yields tiny gradients, especially from the Dice of small structures, and in float16 they become zero. Large values, such as attention logits, turn into infinity above 65,504, and then into NaN. That is why Lightning’s 16-mixed creates a GradScaler: it multiplies the loss by a large factor so that gradients do not vanish, and skips a step when an infinity appears. bfloat16 has float32’s range, with neither underflow nor overflow, and bf16-mixed runs without scaling. Lightning even raises an error if given a GradScaler together with bf16-mixed.
Try it: a number is stored in all three formats, rounded to the nearest.
- float32
- –
- float16
- –
- bfloat16
- –
float32
float16
bfloat16
The gradient 2 · 10⁻⁸ becomes zero in float16 and survives with the 2¹⁶ factor. The logit 70,000 overflows in float16. The weight 1 + 10⁻³ survives in float16 but becomes exactly 1 in bfloat16: a small weight update is lost. That is why the weights and their update stay in float32, and why bf16-true is risky for training.
Hardware. bfloat16 tensor cores arrived with the Ampere architecture: A100, RTX 30xx, L4, H100 and newer. T4 and V100 lack them, and the T4 is Colab’s free GPU: there, use precision="16-mixed" with a GradScaler. torch.cuda.is_bf16_supported() returns True by default even under software emulation, which is slow. Hardware support is what torch.cuda.is_bf16_supported(including_emulation=False) reports.
Where the project is now
osv has no Lightning yet. M1 and M5 train in plain PyTorch on one GPU: torch.amp.autocast, which on CUDA defaults to float16, together with a GradScaler (osv/train/m1.py, osv/train/desmoke.py). The function get_default_trainer_config() declares bf16 precision and activation recomputation as defaults, but no training loop uses it yet. Lightning, bf16-mixed and several GPUs are the goal of the curriculum’s fourth week, not finished work. UNETR, SwinUNETR and VISTA-3D have not yet been trained in the project.
Checklist
- Compare a transformer with nnU-Net or SegResNet at the same budget, and publish both configurations.
- For UNETR the input is fixed and a multiple of 16; for SwinUNETR, a multiple of 32 on every axis.
- Before turning on FSDP, measure what fills the memory. In 3D it is usually activations: start with
use_checkpoint, a smaller patch andbf16-mixed. - Use DDP for segmentation networks; FSDP when 16Ψ does not fit on one GPU.
bf16-mixedon Ampere and newer;16-mixedwith aGradScaleron T4 and V100.bf16-truefor training only by deliberate choice.- Tune the learning rate to the effective batch N · b, and log with
sync_dist=True. - For VISTA-3D, check the licence of the weights you downloaded, and remember that the model works on CT only.
Further reading
- Hatamizadeh et al. UNETR: Transformers for 3D Medical Image Segmentation. WACV 2022.
- Hatamizadeh et al. Swin UNETR: Swin Transformers for Semantic Segmentation of Brain Tumors in MRI Images. 2022.
- Tang et al. Self-Supervised Pre-Training of Swin Transformers for 3D Medical Image Analysis. CVPR 2022.
- Liu et al. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows. ICCV 2021.
- He et al. VISTA3D: A Unified Segmentation Foundation Model For 3D Medical Imaging. 2024.
- Isensee et al. nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentation. MICCAI 2024.
- Rajbhandari et al. ZeRO: Memory Optimizations Toward Training Trillion Parameter Models. SC 2020.
- Zhao et al. PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel. VLDB 2023.
- Micikevicius et al. Mixed Precision Training. ICLR 2018.
- Kalamkar et al. A Study of BFLOAT16 for Deep Learning Training. 2019.
- Lightning documentation on precision and FSDP.