LECTURE 4

Lecture 4. Synthesis of medical volumes, masks and surgical artefacts

How generative models make up for rare conditions and anatomical variants: 3D latent diffusion in MAISI, rectified flow in MAISI v2, body geometry set by 19 conditions in NV-Generate-Body, CT from your own masks through a ControlNet, and the physical smoke model in osv.augment. With diagrams, formulas, interactive examples and 3D animations.

This page is still being reviewed by our scientific, clinical and legal experts. It describes plans, not finished work.

Rare conditions and anatomical variants are scarce in labelled data. A model that has seen three livers with a large tumour has not learned what such a liver looks like, and a dataset cannot simply wait for more patients. Generative models can make more examples, and when the mask is generated first, every synthetic image arrives with an exact label. This lecture covers how 3D latent diffusion works, why rectified flow cuts sampling from 1000 steps to about 30, how to set the size of an organ or a tumour, how to turn your own masks into CT, and what the project simulates on surgical video today. It is lecture 4 of the learning programme, after lecture 3 on annotation.

Every image on this page is synthetic or drawn by code. MAISI is not part of the OSV pipeline today; this lecture is background for anyone who wants to bring it in.

What synthesis can and cannot do

Synthetic data helps where real data is thin: rare findings, unusual anatomy, a tumour of a given size, a patient group that is under-represented. Generated together with its mask, a synthetic volume costs no annotation time, and no patient stands behind it, although a generator can still memorise its training data (see checking synthetic data).

It cannot invent what its training data never showed, and a model trained on synthetic data alone may not work on real data. The project has measured this. In RFC 0001, a segmentation model trained on synthetic instruments composited over real backgrounds reached a binary IoU of 0.116 on real test video; the same model trained on real labels reached 0.772. Synthetic data is a supplement to real data, and it is judged on real data only.

Latent diffusion in 3D: MAISI

traininggenerationCT volume512 × 512 × 768VAE encoder÷ 4 along each axislatent z₀4 × 128 × 128 × 192diffusion U-Netlearns to remove noisenoisethe latent’s shapethe same U-NetN steps: 1000 or 30VAE decoder× 4 along each axissynthetic CT512 × 512 × 768trained weights
Training at the top, generation at the bottom. The diffusion model never sees voxels, only the latent.

A CT volume of 512 × 512 × 768 voxels holds 201 million values. A diffusion network has to run on its input hundreds of times, and at that size it does not fit in GPU memory. MAISI (Guo et al., WACV 2025) therefore works in two parts:

  1. A variational autoencoder (VAE) is trained once to compress a volume four times along each axis into four channels: 4 × 128 × 128 × 192, or 12.6 million numbers, 16 times fewer. Its decoder turns such a latent back into voxels.
  2. A diffusion U-Net learns to generate latents. It also receives the voxel spacing, so one model makes volumes of different sizes and resolutions.

The first MAISI models used DDPM (Ho et al., 2020). Training takes a clean latent x0, mixes it with Gaussian noise ε to the level of a random step t out of 1000, and asks the network to tell which noise was added:

xt=α‾tx0+1−α‾tε,ε∼𝒩(0,I) L=‖ε−εθ(xt,t)‖2,t=1,…,1000

Generation starts from pure noise and removes a little of the predicted noise at each of the 1000 steps. Running the diffusion in an autoencoder’s latent rather than on pixels is the idea of latent diffusion (Rombach et al., 2022), which also powers Stable Diffusion; MAISI carries it into three dimensions.

The size of the volume matters more than any other setting. A model has only seen certain fields of view, the volume’s size in voxels times the voxel spacing in millimetres. Asked for a field of view outside that range, a 128 mm cube of whole-body CT for example, it returns volumes that are useless even though the request is technically valid. Start from the recommended sizes and spacings in the model repository.

Rectified flow: MAISI v2

DDPM’s path from data to noise is curved, and following a curve takes many small steps. Rectified flow (Liu et al., 2022) joins a data point and a noise sample with a straight line and trains the network to predict the velocity along it, which is the same at every point of the line:

xt=(1−t)x0+tε,t∈[0,1] L=‖(ε−x0)−vθ(xt,t)‖2

Here t = 0 is the data and t = 1 the noise; papers and code differ in this convention, so check it before reading a scheduler. Generation goes backwards from noise in N steps of the simplest solver, Euler’s method: move against the predicted velocity for a time 1/N.

xt−Δt=xt−Δt·vθ(xt,t),Δt=1N,N≈30steps

MAISI v2 (Zhao et al., AAAI 2026) switched to rectified flow and added a region-specific contrastive loss, which makes the result follow its conditions more closely. Its authors report about 30 sampling steps instead of DDPM’s 1000, 33 times faster for the latent diffusion. On the autoPET 2023 benchmark, which none of the models was trained on, the repository reports an FID of 5.124 for rflow-ct against 6.083 for ddpm-ct; lower is better.

Why 30 steps and not one? Try it: points are noise samples on their way to 64 data points, a ring and a cluster.

  • Curved, each pair on its own. Each noise point knows its data point and follows a quarter circle to it. Every Euler step leaves along the tangent and overshoots the bend, so a few steps land off target.
  • Straight, each pair on its own. On a straight line the velocity never changes, and a single step lands exactly. This is the hope behind rectified flow.
  • What a model learns. A network does not know which data point a noise sample was paired with. Lines from different pairs cross, and at a crossing the best it can predict is the average of their velocities. The tool computes that average exactly for these 64 points. The paths bend where they have to split between the ring and the cluster. With one step every point lands on the mean of the data, a blur; at 30 steps they land on the data. Liu et al. straighten the paths further by retraining on the model’s own pairs, which they call reflow.

The NV-Generate-CTMR repository publishes five sets of weights:

Weights Modality Output Steps Largest volume Weights licence
rflow-ct CT image, or image and mask 30 512 × 512 × 768 NVIDIA Open Model
ddpm-ct CT image, or image and mask 1000 512 × 512 × 768 NVIDIA Open Model
rflow-mr MRI: brain, abdomen, breast, prostate image only 30 512 × 512 × 128 NVIDIA non-commercial
rflow-mr-brain brain MRI: T1w, T2w, FLAIR, SWI image only 30 512 × 512 × 256 NVIDIA Open Model
rflow-mask CT masks mask of 132 labels 100 256 × 256 × 256 NVIDIA Open Model

Only CT comes with masks. The MRI weights make images without labels, and rflow-mr may not be used commercially. The code is Apache 2.0; the weights carry their own licences.

Controlled body geometry: NV-Generate-Body

1 · the mask19 conditionsone size, demographics1rflow-mask100 steps, guidance s = 22mask, 256³132 labels with the body3or your own maskremapped, plus label 2002 · the imageControlNetreads the mask4rflow-ct U-Netfrom noise, 30 steps5VAE decoderCT volumethe mask is its truth
Image and mask pairs come in two stages: first the mask, then a CT drawn to it. The mask can be generated or brought from outside.

To get a liver with a 4 cm tumour, the mask has to have one. NV-Generate-Body (Zhao et al., 2026), the rflow-mask weights, generates a whole-body mask of 256 × 256 × 256 voxels with 132 labels from a vector of 19 numbers:

  • slots 0–8: the sizes of 9 organs: liver, spleen, stomach, pancreas, colon, left kidney, right kidney, lung and gallbladder;
  • slots 9–13: the sizes of 5 kinds of tumour: lung tumour, pancreatic tumour, hepatic tumour, colon cancer primaries and bone lesion;
  • slots 14–18: age, sex, weight, BMI and height.

A slot without a value holds −1. Sizes are relative, from 0 to 1. Age is divided by 120, weight by 200 kg, BMI by 75 and height by 200 cm; sex is 1 for male and 0 for female. The model was trained on 8,571 masks from 26 cohorts. Its authors report a Spearman correlation of 0.70 to 0.92 between the requested and the generated organ size, a tumour recall of 85 to 100 % for lung, pancreatic and colon tumours and bone lesions, and a correlation of 0.78 for BMI.

The repository’s code adds rules that the paper’s summary does not mention:

  • One size at a time. A request may set the size of one organ or one tumour, not several.
  • Weight or BMI, not both. They come from different training datasets and appear together in fewer than 0.4 % of cases.
  • Height is unreliable. It is present in about 0.4 % of the training data.
  • Avoid the hepatic tumour slot. The documentation warns that its recall is low and the results are unpredictable, which matters for a project centred on the liver and gallbladder.

How does the vector reach the network? Through adaptive group normalisation (AdaGN). Every residual block normalises its features and then scales and shifts them by amounts predicted from the conditions together with the time step, an idea known as FiLM (Perez et al., 2018). When a pretrained network is extended this way, γ starts at zero, so the network begins unchanged. In NV-Generate-Body the same vector also enters through cross-attention.

h′=GroupNorm(h)·(1+γ)+β (γ,β)=MLP(et+E(y)),y∈ℝ19

At every step the model runs twice: once with the conditions and once with all 19 slots set to −1. Classifier-free guidance (Ho and Salimans, 2022) then pushes the result past the conditioned answer, away from the unconditioned one. NV-Generate-Body uses s = 2.

v^=vθ(xt,t,∅)+s·(vθ(xt,t,y)−vθ(xt,t,∅)),s=2

Build a request below. The vector and the configuration are built the way the repository’s scripts/sample.py builds them; the 3D body is a cartoon that shows what the numbers ask for.

CT from your own masks: ControlNet

The second stage draws a CT to fit a mask. ControlNet (Zhang et al., 2023) does it without retraining the diffusion model. A trainable copy of the U-Net’s encoder reads the mask and adds its features into the frozen U-Net through convolutions whose weights start at zero, so training starts from the unchanged model and the mask’s influence grows only as it helps.

MAISI’s ControlNet expects a mask in its own vocabulary of 132 labels, listed in configs/label_dict.json, plus the body envelope, label 200, which tells it where the body ends. A few of the numbers:

Structure Label Structure Label
liver 1 gallbladder 10
spleen 3 stomach 12
pancreas 4 hepatic vessel 25
right kidney 5 hepatic tumour 26
aorta 6 colon 62
body envelope 200

There are two ways to make such a mask from a real CT:

  1. Segment it with NVIDIA’s nv-segment-ct, whose labels already follow the MAISI vocabulary, and add the body envelope with scripts.utils.add_body_envelope; the segmenter never outputs label 200.
  2. Segment it with another model, TotalSegmentator for example, remap its labels to the MAISI numbers, and add the body envelope.

Then run:

python -m scripts.infer_image_from_mask \
  -t ./configs/config_network_rflow.json \
  -i ./configs/config_infer.json \
  -e ./configs/environment_rflow-ct.json \
  --mask /path/to/your_mask.nii.gz

A real patient’s mask is derived from patient data. It goes through the same de-identification and stays in the same place as the CT it came from.

Checking synthetic data

A synthetic set is a dataset like any other, and it needs evidence before anyone trains on it:

  • Realism. FID compares the statistics of synthetic and real images; for CT the repository computes it on 2.5D slices. A low FID does not prove correct anatomy.
  • Agreement with the mask. Segment the synthetic volume with an independent model and compare it with the mask it was drawn from, with Dice from the metrics page. Drop volumes that disagree, and report how many were dropped.
  • Clinical plausibility. A clinician reviews a sample. Class definitions and the catalogue of anatomical variants belong to the clinical lead (rule R-14).
  • Memorisation. 3D latent diffusion models can reproduce their training volumes almost exactly (Dar et al., 2023). Compare each synthetic volume with its nearest training volume before calling a synthetic set private.
  • Usefulness. Train with and without the synthetic data and test both on the same real test set. Synthetic data never enters validation or test.
  • Provenance. Record the weights, their version, the configuration, the seed and the licence, as for any dataset in the project (rules R-22 and R-23).

Surgical artefacts in osv.augment

On surgical video the project does not use a generative model. It models one artefact, smoke, physically and composites it over real frames. Here is what the code holds today:

Effect In the osv repository
Surgical smoke implemented in osv/augment/smoke.py; training pairs for the M5 de-smoking model
Synthetic instruments over real backgrounds implemented in osv/augment/instruments.py; the experiment of RFC 0001
Blood on the lens listed in osv.robustness as a stress test; not implemented, not evaluated
Glare on metal jaws listed in osv.robustness as a stress test; not implemented, not evaluated
Blender renderRGBA, 919 frameswarpflip, scale, shiftthickness, lightgain g, lamp L, tint τcompositeJ′(1 − a) + C·atraining pairsmoky in, clean outclean frameCholecT50field mask mno smoke on vignette
A rendered smoke layer goes over a clean frame; the clean frame stays as the target, so the pair teaches a model to remove smoke.

The smoke layers are the project’s own renders: 36 sequences, 919 frames at 1280 × 720, simulated with Blender’s Mantaflow and rendered as straight-alpha RGBA under CC BY 4.0. The composite follows the single-scattering haze model that the de-smoking literature trains on:

a=(1−(1−α)g)·m I=J′(1−a)+Ca,C=csmoke·b·L·τ
  • α is the render’s opacity at a pixel. The gain g raises the transmittance 1 − α to a power. Doubling g doubles the smoke’s optical thickness instead of clipping the opacity at 1.
  • m is the field mask. The scope’s black vignette is found as dark pixels connected to the frame’s border, and smoke never sits there: it is in the abdomen, not on the lens housing.
  • C is the smoke’s colour scaled by a brightness b, by the frame’s own lamp level L (the 99th percentile of brightness) and by a small tint τ. A white render pasted at full brightness over a dim frame is the first thing a model learns to spot.
  • J′ is the clean frame, blurred where the smoke is dense: forward scattering softens edges behind smoke rather than only fading them.

Every parameter is drawn from a generator seeded per pair: scale 0.8 to 1.4, shift up to a quarter of the frame, gain 0.6 to 2.5, brightness 0.75 to 1.05, tint ±4 %, blur up to 2.5 pixels at 854 pixels of width. The validation and test pairs therefore regenerate bit for bit. Smoke changes no label, because the tissue is still there behind it. The pairs inherit the licence of their clean frames, CholecT50’s CC BY-NC-SA 4.0. On 518 synthetic validation pairs, the residual U-Net trained on them raised PSNR from 28.99 dB for the smoky input to 33.08 dB. It has not been evaluated on real smoke yet (M5 model card).

Try the same model on a synthetic frame. The smoke layer here is noise shaped into a plume by code, not a Blender render; everything after it follows composite().

The two missing effects have their own physics, which an implementation would have to respect:

  • Glare. The lamp sits on the scope’s axis, so polished metal jaws reflect it almost straight back into the camera. The highlight saturates the sensor, clips to white, blooms into its surroundings and moves with the instrument’s angle. A physical version would render instruments with a metallic surface, as the RFC 0001 renders do, rather than paint bright spots.
  • Blood on the lens. A drop sits on the front lens, far outside the depth of field, so it shows as a soft red disc that magnifies and tints what is behind it. It stays from frame to frame until the lens is cleaned.

Whether the pixels hidden by blood or glare keep their label is a decision for the annotation guideline and the clinical lead. The sketches in the tool above draw the look and nothing more.

Checklist

  • Generate the mask first and keep it as the label; check the image against it.
  • Keep the field of view in the range the model was trained on.
  • Keep synthetic data out of validation and test, and report results on real data.
  • Record the weights, version, configuration, seed and licence of every synthetic set; mind the non-commercial terms.
  • Check for memorisation before calling synthetic data private.
  • In osv.augment, only smoke is implemented; do not claim blood or glare augmentation.

Further reading