LECTURE 8

Lecture 8. Architectures in MONAI Core and Lightning

From U-Net to transformers: UNETR and SwinUNETR in MONAI, the VISTA-3D foundation model with automatic segmentation and click corrections, training on several GPUs with DDP and FSDP, and bf16-mixed precision in Lightning. With diagrams, formulas, interactive examples and 3D animations.

This page is still being reviewed by our scientific, clinical and legal experts. It describes plans, not finished work.

Lecture 7 named the project’s tasks. This lecture is about the models that solve them, and about training them when one GPU is not enough. It is lecture 8 of the curriculum, after lecture 7 on the tasks M1 to M5. Three topics:

  • Transformers for volumes: UNETR and SwinUNETR, and what changes when a U-Net’s encoder is replaced by attention.
  • VISTA-3D: one model for automatic segmentation and for corrections by clicks.
  • Scaling: DDP, FSDP and bf16-mixed precision in Lightning.

The code is checked against MONAI 1.6.1 and Lightning 2.6.6. The basics of the 3D U-Net, patch size and sliding-window inference are on the volumes page and are taken as known here. Every image on this page is drawn by code.

Why attention, and what it costs

A 3 × 3 × 3 convolution sees 27 neighbouring voxels. Its field of view grows only with the depth of the network, and linking the liver to the spleen on the other side of the abdomen takes many layers. Self-attention links any two tokens in the first layer: every token weighs every other.

The price is the number of pairs. A volume of side D cut into patches of side p gives N tokens, and global attention scores N² pairs in every layer and every head. Attention in windows of M³ tokens scores only N · M³ pairs. The numbers below are for a 96³ patch, D = 96:

N=(Dp)3 N2pairs,window:N·M3 UNETR,p=16:N=216,N2=46 656 Swin,p=2:N=110 592 N2≈1.2·1010,N·73≈3.8·107

Hence two answers and two architectures. UNETR takes large 16³ patches, and there are so few tokens that global attention is affordable. SwinUNETR takes small 2³ patches and pays with windows: on 110,592 tokens, one head’s global attention matrix in bf16 would weigh about 24 GB.

UNETR: a ViT as the encoder

UNETR (Hatamizadeh et al., WACV 2022) takes a plain vision transformer, ViT-B, and puts it in place of a U-Net’s encoder. The decoder stays convolutional.

volume1 × 96³16³ patches216 tokens1linear embedding+ position21234567891011123transformer layersz3deconv → 48³448³z6deconv → 24³24³z9deconv → 12³12³z126³6³5conv96³96³1×1×1 conv6maskK × 96³
UNETR as in the paper: 16³ patches, 12 transformer layers, taps from intermediate layers to the decoder.

In MONAI:

from monai.networks.nets import UNETR

model = UNETR(
    in_channels=1, out_channels=14,  # BTCV: 13 organs and background
    img_size=(96, 96, 96),           # a multiple of 16 on every axis
    feature_size=16, hidden_size=768, mlp_dim=3072, num_heads=12,
    proj_type="conv", norm_name="instance", res_block=True,
)

What the code shows, and what follows from it:

  • The patch is fixed. In unetr.py the patch size is the constant 16, and img_size is required because the position embeddings are learned for exactly 6 × 6 × 6 positions. The model is tied to its input size: inference runs a sliding window of the same size, 96³.
  • Taps. The decoder takes hidden_states_out[3], [6] and [9], the outputs of the 4th, 7th and 10th blocks counting from one, and the output of the last layer. The paper calls them z3, z6, z9 and z12; it is easy to mix them up when comparing with the code.
  • Detail comes from convolutions. One token covers 16 mm at 1 mm spacing. The decoder takes fine edges from a convolutional branch that reads the original volume at full resolution.
  • Size. 92.8 million parameters (counted in MONAI 1.6.1), five times a SegResNet with 32 initial filters (18.8 million).

SwinUNETR: small patches, windows and a shift

SwinUNETR (Hatamizadeh et al., 2022) takes the encoder of the Swin Transformer (Liu et al., ICCV 2021). It is built like a convolutional encoder: four stages, each halving the grid.

  1. 2³ patches. There are many tokens, and each is small: 2 mm at 1 mm spacing.
  2. 7³ windows. Attention is computed only inside a window of 343 tokens, so the cost grows linearly with the volume.
  3. The shift. In every second layer the grid of windows moves by half a window, 7 // 2 = 3 tokens on each axis, and tokens at a window’s border share a window with their neighbours across it.
  4. Patch merging. Between stages, 2 × 2 × 2 neighbouring tokens merge into one and the channels double: a hierarchy of features, as in a U-Net, whose outputs reach a convolutional decoder through skip connections.

The shift is done cleverly. Simply moving the windows would leave partial windows at the edges of the volume, and more of them. MONAI instead rolls the tokens themselves, torch.roll by −3 on each axis, cuts the same number of windows as before and applies an attention mask: tokens that wrapped round from the opposite side do not see each other inside a window. After attention, the tokens are rolled back.

Why the shift matters shows in a toy. A 16 × 16 grid of tokens with 4 × 4 windows, as the volumes page draws Swin in 2D. Click a token and see how many tokens its information reaches in L layers.

Without the shift, information stays in its window of 16 tokens forever, however many layers are added. With it, every two layers widen the reach, and patch merging between stages speeds the growth further.

In MONAI 1.6.1:

from monai.networks.nets import SwinUNETR

model = SwinUNETR(
    in_channels=1, out_channels=14,
    feature_size=48,       # 24 by default; 48 matches the pretrained weights
    use_checkpoint=True,   # recompute activations in the backward pass
)
# patch_size=2, depths=(2, 2, 2, 2), num_heads=(3, 6, 12, 24), window_size=7 are the defaults
  • No img_size. Older examples pass img_size=(96, 96, 96); current MONAI has no such parameter. Instead, forward checks that every side of the input divides by patch_size ** 5, that is by 32. 96 and 128 work. A video frame of 854 × 480 for the 2D variant (spatial_dims=2) needs padding to 864 in width: 480 divides by 32, 854 does not.
  • Size. 62.2 million parameters with feature_size=48, fewer than UNETR, but far more activations in memory: 512 times more tokens at the first stage. The numbers are in the memory section below.
  • use_v2=True adds a residual convolution block at the start of every stage (SwinUNETR-V2).
  • Pretraining. Tang et al. (CVPR 2022) pretrained the encoder self-supervised on 5,050 public CT scans, then fine-tuned on BTCV and the Medical Segmentation Decathlon. Weights and an example are in the MONAI tutorial SwinUNETR on BTCV.

Should you take a transformer. The nnU-Net authors’ re-evaluation (Isensee et al., MICCAI 2024) showed that at equal budget a well-tuned convolutional U-Net does not lose to UNETR or SwinUNETR, and more often wins. The starting point for M4 is still nnU-Net. A transformer pays off when there are pretrained weights for your modality, or so much data that pretraining earns its cost. A comparison is published together with a baseline of the same budget.

VISTA-3D: automatic and clicks in one model

VISTA-3D (He et al., 2024) is a foundation model for CT. It is trained on 11,454 volumes and segments 127 classes of anatomy and lesions automatically. It also takes clicks in 3D and corrects its answer by them, and with clicks it can outline a structure that was not in its training.

CT volumepatch 128³SegResNet encoderconvolutional, sharedfeaturesfor every voxelclass numberone of 127class headclass embedding · featurespoint headlike SAM's decoderclicks+ and − in 3Dmergeclicks win near clicks
One encoder, two heads. The automatic branch takes a class number, the interactive one takes clicks; near the clicks the interactive one wins.

What is inside, by MONAI’s code (monai/networks/nets/vista3d.py):

  • The encoder is convolutional. It is SegResNetDS2, not a transformer: “foundation” here means trained on a large and varied set, not built on attention. 218.0 million parameters in all, nearly all of them in the encoder; the heads weigh less than a million.
  • The class head. Every class number has a learned embedding. A class’s mask is the dot product of that embedding with every voxel’s features, then a sigmoid. Classes do not compete through a softmax, so a new class in fine-tuning is a new embedding, not a new output layer.
  • The point head. Built like SAM’s decoder: the clicks’ coordinates and signs (+ or −) become tokens that read the volume’s features and give a binary mask of the object under the clicks.
  • The merge. Given both a class and clicks, gaussian_combine replaces the automatic answer near each click with the point head’s answer, with a weight that falls off with distance as a Gaussian.

So the model closes the working loop of lecture 3: the automatic branch labels a cohort, and a person corrects its mistakes with a few clicks instead of redrawing everything. Try it on a slice of the lecture 1 phantom. The automatic answer here has two mistakes: the liver spills over its edge into fat and stomach, and the back of the spleen is missed. Pick a class and a kind of click, and click where the mistake is.

A minus on the fat at the liver’s edge removes the excess, a plus on the back of the spleen adds what was missed. Everything beyond the reach stays as it was. The toy simplifies two things: its point head is perfect and returns exactly the organ that was clicked, and the border of a click’s reach is sharp, where VISTA-3D blends the two answers smoothly.

How to run it. The model ships as the MONAI bundle vista3d with a 128³ patch; when memory runs short, the bundle’s README suggests reducing patch_size to 96³. Commands from the README:

python -m monai.bundle download "vista3d" --bundle_dir "bundles/"
cd bundles/vista3d
# automatic, one class by its number
python -m monai.bundle run --config_file configs/inference.json --input_dict "{'image':'spleen_03.nii.gz','label_prompt':[3]}"
# by clicks: points in voxels, 1 for the object, 0 for background
python -m monai.bundle run --config_file configs/inference.json --input_dict "{'image':'spleen_03.nii.gz','points':[[128,128,16],[100,100,16]],'point_labels':[1,0]}"

Class numbers are in the bundle’s configs/metadata.json.

Where the “hundreds of classes” are. VISTA-3D itself has 127. Hundreds belong to its successor NV-Segment-CTMR: it starts from the VISTA-3D checkpoint, is fine-tuned on more than 30,000 CT and MRI scans and segments over 300 classes, but only automatically, without clicks.

Licences. The code is Apache 2.0. For the weights the sources disagree: the VISTA repository’s README names the NVIDIA Open Model License for VISTA-3D, which allows commercial use, while the model zoo bundle’s README names the NCLS v1 licence. The NV-Segment-CTMR weights are non-commercial. Before building anything but research on the weights, check the licence of the exact file you downloaded.

For the project. VISTA-3D and TotalSegmentator are starting points for M4, preoperative 3D segmentation. Both work on CT, not on laparoscopic video. Patient scans they run on go through de-identification and stay where they are stored.

Training on several GPUs

Lightning in brief

Lightning separates what a model computes from where and how it computes. The model is a LightningModule with training_step, validation_step and configure_optimizers. Devices, the distribution strategy and precision are arguments of the Trainer. Lightning starts the processes, wraps the model, sets a DistributedSampler and gathers metrics from every GPU.

import lightning as L
import torch
from monai.inferers import sliding_window_inference
from monai.losses import DiceCELoss
from monai.networks.nets import SwinUNETR


class Segmenter(L.LightningModule):
    def __init__(self):
        super().__init__()
        self.net = SwinUNETR(in_channels=1, out_channels=14, feature_size=48, use_checkpoint=True)
        self.loss = DiceCELoss(to_onehot_y=True, softmax=True)

    def training_step(self, batch, batch_idx):
        loss = self.loss(self.net(batch["image"]), batch["label"])
        self.log("train/loss", loss, sync_dist=True)
        return loss

    def validation_step(self, batch, batch_idx):
        logits = sliding_window_inference(batch["image"], roi_size=(96, 96, 96), sw_batch_size=4,
                                          predictor=self.net, overlap=0.5)
        self.log("val/loss", self.loss(logits, batch["label"]), sync_dist=True)

    def configure_optimizers(self):
        return torch.optim.AdamW(self.parameters(), lr=1e-4, weight_decay=1e-5)


trainer = L.Trainer(accelerator="gpu", devices=4, strategy="ddp", precision="bf16-mixed", max_epochs=300)
trainer.fit(Segmenter(), train_loader, val_loader)

The loaders come from lecture 6. Going from one GPU to four, or from DDP to FSDP, changes one line of the Trainer, not the model.

DDP: a copy of the model on every GPU

Distributed Data Parallel starts one process per GPU. Each process holds a full copy of the model and its own part of the batch. In the backward pass the gradients are averaged over all GPUs by an all-reduce, in buckets, overlapping with computation. Then every copy takes the same step and the copies stay identical.

DDPFSDPthe whole model on every GPUweights, gradients, Adam: 16 bytes per parameterforward and backward passon this GPU's part of the batchall-reducegradients averaged over all GPUsthe same Adam stepon every GPU1/N of everything on each GPUweights, gradients, Adam: 16/N bytes per parameterall-gather before each layerthe layer's weights in full, for a momentforward and backward passthen the full weights are droppedreduce-scattereach GPU keeps its 1/N of the gradientsAdam stepon this GPU's shard only
One training step. DDP on the left, FSDP on the right; the dashed arrow: the gather and the pass repeat for every layer.

Three consequences:

  • The effective batch is N · b: four GPUs with two patches each give a batch of 8. The learning rate is tuned to the effective batch.
  • BatchNorm on each GPU computes statistics over its own two patches. MONAI’s 3D networks use instance norm by default, which does not mind. If a model has BatchNorm, the Trainer has a sync_batchnorm=True flag.
  • Logs from all GPUs are averaged only with sync_dist=True, as in the example above.

FSDP: each GPU holds a share of everything

Fully Sharded Data Parallel divides not only the data between GPUs but the model itself. Each GPU stores 1/N of the weights, gradients and optimiser state. Before each layer its weights are gathered in full (all-gather), used and dropped, and the gradients go to their owners (reduce-scatter). It is the idea of ZeRO (Rajbhandari et al., 2020), implemented in PyTorch (Zhao et al., 2023).

How much it saves, counting Adam, Ψ parameters, N GPUs, b patches per GPU and A bytes of activations per patch:

MDDP=16Ψ+b·A MFSDP≈16ΨN+b·A 16=4weights+4gradients+8Adam VDDP=2·N−1N·4Ψ

In mixed precision the weights and gradients stay in float32, 4 bytes each, and Adam’s two moments add 8 bytes. The last line is how many bytes each GPU sends per step in DDP with a ring all-reduce.

In Lightning:

from lightning.pytorch.strategies import FSDPStrategy
from monai.networks.nets.swin_unetr import SwinTransformerBlock

strategy = FSDPStrategy(
    auto_wrap_policy={SwinTransformerBlock},               # the unit gathered and dropped
    activation_checkpointing_policy={SwinTransformerBlock},
    sharding_strategy="FULL_SHARD",                        # or SHARD_GRAD_OP, HYBRID_SHARD
)
trainer = L.Trainer(accelerator="gpu", devices=4, strategy=strategy, precision="bf16-mixed")

FULL_SHARD shards everything. SHARD_GRAD_OP keeps the weights whole and shards gradients and the optimiser. HYBRID_SHARD shards within a node and replicates across nodes. The string strategy="fsdp" gives FULL_SHARD with default settings.

The main point for 3D: activations, not weights, fill the memory. We measured it in MONAI 1.6.1, for one 96³ patch, counting the tensors autograd keeps for the backward pass, without the weights themselves.

Model Parameters Weights, gradients and Adam, 16Ψ Activations per patch, float32 Activations per patch, bf16-mixed
UNETR 92.8 M 1.5 GB 1.06 GB 0.73 GB
SwinUNETR, feature_size=48 62.2 M 1.0 GB 4.84 GB 3.98 GB
the same with use_checkpoint=True 62.2 M 1.0 GB 2.75 GB 1.59 GB
VISTA-3D encoder (automatic branch) 218.0 M 3.5 GB 2.51 GB 1.65 GB

The measurement ran on a CPU, so it lacks cuDNN workspaces and memory fragmentation: real use on a GPU is higher. The proportion does not change. In SwinUNETR the activations of one patch weigh three to five times the weights, gradients and Adam together. In UNETR, with its 216 tokens, they are comparable, and overtake them from two patches per GPU. FSDP shards the “16Ψ” column, and for all three models it fits on one GPU. FSDP does not shard activations: each GPU computes its own patches. So DDP is usually enough for segmentation networks, and memory is saved elsewhere: a smaller patch, recomputed activations (use_checkpoint), bf16-mixed and one or two patches per GPU. FSDP is needed when 16Ψ does not fit, which means models from a billion parameters up, not the segmentation networks of this lecture.

Check for yourself. The bar is one GPU’s memory, the dashes are typical card sizes.

Switching from DDP to FULL_SHARD shrinks the blue, green and purple parts but not the orange one. Traffic per step follows Lightning’s settings: DDP averages gradients in float32, while FSDP with bf16-mixed gathers the weights in bfloat16 twice (for the forward and the backward pass) and scatters the gradients in bfloat16.

bf16-mixed: float32’s range in 16 bits

A floating-point number is a sign s, an exponent e of a few bits with a bias b, and a mantissa m of k bits:

x=(−1)s·2e−b·(1+m2k)

Exponent bits set the range, mantissa bits the precision. float16 and bfloat16 both take 16 bits but split them differently:

Format Exponent bits Mantissa bits Largest number Smallest normal Step after 1
float32 8 23 3.4 · 10³⁸ 1.2 · 10⁻³⁸ 1.2 · 10⁻⁷
float16 5 10 65,504 6.1 · 10⁻⁵ 9.8 · 10⁻⁴
bfloat16 8 7 3.4 · 10³⁸ 1.2 · 10⁻³⁸ 7.8 · 10⁻³

float16 also has subnormal numbers down to 6.0 · 10⁻⁸, and anything smaller becomes zero. bfloat16 has float32’s exponent, hence the same range, and a precision of about two to three significant digits.

What “mixed” means. With bf16-mixed, Lightning turns on torch.autocast: convolutions and matrix multiplications run in bfloat16 on tensor cores, activations take less memory, and the weights, their update, normalisations, softmax and loss functions stay in float32. With bf16-true, everything is stored in bfloat16, the weights included.

Why it is steadier than float16. A deep network on a large volume yields tiny gradients, especially from the Dice of small structures, and in float16 they become zero. Large values, such as attention logits, turn into infinity above 65,504, and then into NaN. That is why Lightning’s 16-mixed creates a GradScaler: it multiplies the loss by a large factor so that gradients do not vanish, and skips a step when an infinity appears. bfloat16 has float32’s range, with neither underflow nor overflow, and bf16-mixed runs without scaling. Lightning even raises an error if given a GradScaler together with bf16-mixed.

Try it: a number is stored in all three formats, rounded to the nearest.

The gradient 2 · 10⁻⁸ becomes zero in float16 and survives with the 2¹⁶ factor. The logit 70,000 overflows in float16. The weight 1 + 10⁻³ survives in float16 but becomes exactly 1 in bfloat16: a small weight update is lost. That is why the weights and their update stay in float32, and why bf16-true is risky for training.

Hardware. bfloat16 tensor cores arrived with the Ampere architecture: A100, RTX 30xx, L4, H100 and newer. T4 and V100 lack them, and the T4 is Colab’s free GPU: there, use precision="16-mixed" with a GradScaler. torch.cuda.is_bf16_supported() returns True by default even under software emulation, which is slow. Hardware support is what torch.cuda.is_bf16_supported(including_emulation=False) reports.

Where the project is now

osv has no Lightning yet. M1 and M5 train in plain PyTorch on one GPU: torch.amp.autocast, which on CUDA defaults to float16, together with a GradScaler (osv/train/m1.py, osv/train/desmoke.py). The function get_default_trainer_config() declares bf16 precision and activation recomputation as defaults, but no training loop uses it yet. Lightning, bf16-mixed and several GPUs are the goal of the curriculum’s fourth week, not finished work. UNETR, SwinUNETR and VISTA-3D have not yet been trained in the project.

Checklist

  • Compare a transformer with nnU-Net or SegResNet at the same budget, and publish both configurations.
  • For UNETR the input is fixed and a multiple of 16; for SwinUNETR, a multiple of 32 on every axis.
  • Before turning on FSDP, measure what fills the memory. In 3D it is usually activations: start with use_checkpoint, a smaller patch and bf16-mixed.
  • Use DDP for segmentation networks; FSDP when 16Ψ does not fit on one GPU.
  • bf16-mixed on Ampere and newer; 16-mixed with a GradScaler on T4 and V100. bf16-true for training only by deliberate choice.
  • Tune the learning rate to the effective batch N · b, and log with sync_dist=True.
  • For VISTA-3D, check the licence of the weights you downloaded, and remember that the model works on CT only.

Further reading