Rare conditions and anatomical variants are scarce in labelled data. A model that has seen three livers with a large tumour has not learned what such a liver looks like, and a dataset cannot simply wait for more patients. Generative models can make more examples, and when the mask is generated first, every synthetic image arrives with an exact label. This lecture covers how 3D latent diffusion works, why rectified flow cuts sampling from 1000 steps to about 30, how to set the size of an organ or a tumour, how to turn your own masks into CT, and what the project simulates on surgical video today. It is lecture 4 of the learning programme, after lecture 3 on annotation.
Every image on this page is synthetic or drawn by code. MAISI is not part of the OSV pipeline today; this lecture is background for anyone who wants to bring it in.
What synthesis can and cannot do
Synthetic data helps where real data is thin: rare findings, unusual anatomy, a tumour of a given size, a patient group that is under-represented. Generated together with its mask, a synthetic volume costs no annotation time, and no patient stands behind it, although a generator can still memorise its training data (see checking synthetic data).
It cannot invent what its training data never showed, and a model trained on synthetic data alone may not work on real data. The project has measured this. In RFC 0001, a segmentation model trained on synthetic instruments composited over real backgrounds reached a binary IoU of 0.116 on real test video; the same model trained on real labels reached 0.772. Synthetic data is a supplement to real data, and it is judged on real data only.
Latent diffusion in 3D: MAISI
A CT volume of 512 × 512 × 768 voxels holds 201 million values. A diffusion network has to run on its input hundreds of times, and at that size it does not fit in GPU memory. MAISI (Guo et al., WACV 2025) therefore works in two parts:
- A variational autoencoder (VAE) is trained once to compress a volume four times along each axis into four channels: 4 × 128 × 128 × 192, or 12.6 million numbers, 16 times fewer. Its decoder turns such a latent back into voxels.
- A diffusion U-Net learns to generate latents. It also receives the voxel spacing, so one model makes volumes of different sizes and resolutions.
The first MAISI models used DDPM (Ho et al., 2020). Training takes a clean latent x0, mixes it with Gaussian noise ε to the level of a random step t out of 1000, and asks the network to tell which noise was added:
Generation starts from pure noise and removes a little of the predicted noise at each of the 1000 steps. Running the diffusion in an autoencoder’s latent rather than on pixels is the idea of latent diffusion (Rombach et al., 2022), which also powers Stable Diffusion; MAISI carries it into three dimensions.
The size of the volume matters more than any other setting. A model has only seen certain fields of view, the volume’s size in voxels times the voxel spacing in millimetres. Asked for a field of view outside that range, a 128 mm cube of whole-body CT for example, it returns volumes that are useless even though the request is technically valid. Start from the recommended sizes and spacings in the model repository.
Rectified flow: MAISI v2
DDPM’s path from data to noise is curved, and following a curve takes many small steps. Rectified flow (Liu et al., 2022) joins a data point and a noise sample with a straight line and trains the network to predict the velocity along it, which is the same at every point of the line:
Here t = 0 is the data and t = 1 the noise; papers and code differ in this convention, so check it before reading a scheduler. Generation goes backwards from noise in N steps of the simplest solver, Euler’s method: move against the predicted velocity for a time 1/N.
MAISI v2 (Zhao et al., AAAI 2026) switched to rectified flow and added a region-specific contrastive loss, which makes the result follow its conditions more closely. Its authors report about 30 sampling steps instead of DDPM’s 1000, 33 times faster for the latent diffusion. On the autoPET 2023 benchmark, which none of the models was trained on, the repository reports an FID of 5.124 for rflow-ct against 6.083 for ddpm-ct; lower is better.
Why 30 steps and not one? Try it: points are noise samples on their way to 64 data points, a ring and a cluster.
- Curved, each pair on its own. Each noise point knows its data point and follows a quarter circle to it. Every Euler step leaves along the tangent and overshoots the bend, so a few steps land off target.
- Straight, each pair on its own. On a straight line the velocity never changes, and a single step lands exactly. This is the hope behind rectified flow.
- What a model learns. A network does not know which data point a noise sample was paired with. Lines from different pairs cross, and at a crossing the best it can predict is the average of their velocities. The tool computes that average exactly for these 64 points. The paths bend where they have to split between the ring and the cluster. With one step every point lands on the mean of the data, a blur; at 30 steps they land on the data. Liu et al. straighten the paths further by retraining on the model’s own pairs, which they call reflow.
The NV-Generate-CTMR repository publishes five sets of weights:
| Weights | Modality | Output | Steps | Largest volume | Weights licence |
|---|---|---|---|---|---|
rflow-ct |
CT | image, or image and mask | 30 | 512 × 512 × 768 | NVIDIA Open Model |
ddpm-ct |
CT | image, or image and mask | 1000 | 512 × 512 × 768 | NVIDIA Open Model |
rflow-mr |
MRI: brain, abdomen, breast, prostate | image only | 30 | 512 × 512 × 128 | NVIDIA non-commercial |
rflow-mr-brain |
brain MRI: T1w, T2w, FLAIR, SWI | image only | 30 | 512 × 512 × 256 | NVIDIA Open Model |
rflow-mask |
CT masks | mask of 132 labels | 100 | 256 × 256 × 256 | NVIDIA Open Model |
Only CT comes with masks. The MRI weights make images without labels, and rflow-mr may not be used commercially. The code is Apache 2.0; the weights carry their own licences.
Controlled body geometry: NV-Generate-Body
To get a liver with a 4 cm tumour, the mask has to have one. NV-Generate-Body (Zhao et al., 2026), the rflow-mask weights, generates a whole-body mask of 256 × 256 × 256 voxels with 132 labels from a vector of 19 numbers:
- slots 0–8: the sizes of 9 organs: liver, spleen, stomach, pancreas, colon, left kidney, right kidney, lung and gallbladder;
- slots 9–13: the sizes of 5 kinds of tumour: lung tumour, pancreatic tumour, hepatic tumour, colon cancer primaries and bone lesion;
- slots 14–18: age, sex, weight, BMI and height.
A slot without a value holds −1. Sizes are relative, from 0 to 1. Age is divided by 120, weight by 200 kg, BMI by 75 and height by 200 cm; sex is 1 for male and 0 for female. The model was trained on 8,571 masks from 26 cohorts. Its authors report a Spearman correlation of 0.70 to 0.92 between the requested and the generated organ size, a tumour recall of 85 to 100 % for lung, pancreatic and colon tumours and bone lesions, and a correlation of 0.78 for BMI.
The repository’s code adds rules that the paper’s summary does not mention:
- One size at a time. A request may set the size of one organ or one tumour, not several.
- Weight or BMI, not both. They come from different training datasets and appear together in fewer than 0.4 % of cases.
- Height is unreliable. It is present in about 0.4 % of the training data.
- Avoid the hepatic tumour slot. The documentation warns that its recall is low and the results are unpredictable, which matters for a project centred on the liver and gallbladder.
How does the vector reach the network? Through adaptive group normalisation (AdaGN). Every residual block normalises its features and then scales and shifts them by amounts predicted from the conditions together with the time step, an idea known as FiLM (Perez et al., 2018). When a pretrained network is extended this way, γ starts at zero, so the network begins unchanged. In NV-Generate-Body the same vector also enters through cross-attention.
At every step the model runs twice: once with the conditions and once with all 19 slots set to −1. Classifier-free guidance (Ho and Salimans, 2022) then pushes the result past the conditioned answer, away from the unconditioned one. NV-Generate-Body uses s = 2.
Build a request below. The vector and the configuration are built the way the repository’s scripts/sample.py builds them; the 3D body is a cartoon that shows what the numbers ask for.
NVIDIA warns that conditioning on the hepatic tumour has low recall and unpredictable results.
The 19 values the model receives:
The same request in the configuration file:
CT from your own masks: ControlNet
The second stage draws a CT to fit a mask. ControlNet (Zhang et al., 2023) does it without retraining the diffusion model. A trainable copy of the U-Net’s encoder reads the mask and adds its features into the frozen U-Net through convolutions whose weights start at zero, so training starts from the unchanged model and the mask’s influence grows only as it helps.
MAISI’s ControlNet expects a mask in its own vocabulary of 132 labels, listed in configs/label_dict.json, plus the body envelope, label 200, which tells it where the body ends. A few of the numbers:
| Structure | Label | Structure | Label |
|---|---|---|---|
| liver | 1 | gallbladder | 10 |
| spleen | 3 | stomach | 12 |
| pancreas | 4 | hepatic vessel | 25 |
| right kidney | 5 | hepatic tumour | 26 |
| aorta | 6 | colon | 62 |
| body envelope | 200 |
There are two ways to make such a mask from a real CT:
- Segment it with NVIDIA’s
nv-segment-ct, whose labels already follow the MAISI vocabulary, and add the body envelope withscripts.utils.add_body_envelope; the segmenter never outputs label 200. - Segment it with another model, TotalSegmentator for example, remap its labels to the MAISI numbers, and add the body envelope.
Then run:
python -m scripts.infer_image_from_mask \
-t ./configs/config_network_rflow.json \
-i ./configs/config_infer.json \
-e ./configs/environment_rflow-ct.json \
--mask /path/to/your_mask.nii.gz
A real patient’s mask is derived from patient data. It goes through the same de-identification and stays in the same place as the CT it came from.
Checking synthetic data
A synthetic set is a dataset like any other, and it needs evidence before anyone trains on it:
- Realism. FID compares the statistics of synthetic and real images; for CT the repository computes it on 2.5D slices. A low FID does not prove correct anatomy.
- Agreement with the mask. Segment the synthetic volume with an independent model and compare it with the mask it was drawn from, with Dice from the metrics page. Drop volumes that disagree, and report how many were dropped.
- Clinical plausibility. A clinician reviews a sample. Class definitions and the catalogue of anatomical variants belong to the clinical lead (rule R-14).
- Memorisation. 3D latent diffusion models can reproduce their training volumes almost exactly (Dar et al., 2023). Compare each synthetic volume with its nearest training volume before calling a synthetic set private.
- Usefulness. Train with and without the synthetic data and test both on the same real test set. Synthetic data never enters validation or test.
- Provenance. Record the weights, their version, the configuration, the seed and the licence, as for any dataset in the project (rules R-22 and R-23).
Surgical artefacts in osv.augment
On surgical video the project does not use a generative model. It models one artefact, smoke, physically and composites it over real frames. Here is what the code holds today:
| Effect | In the osv repository |
|---|---|
| Surgical smoke | implemented in osv/augment/smoke.py; training pairs for the M5 de-smoking model |
| Synthetic instruments over real backgrounds | implemented in osv/augment/instruments.py; the experiment of RFC 0001 |
| Blood on the lens | listed in osv.robustness as a stress test; not implemented, not evaluated |
| Glare on metal jaws | listed in osv.robustness as a stress test; not implemented, not evaluated |
The smoke layers are the project’s own renders: 36 sequences, 919 frames at 1280 × 720, simulated with Blender’s Mantaflow and rendered as straight-alpha RGBA under CC BY 4.0. The composite follows the single-scattering haze model that the de-smoking literature trains on:
- α is the render’s opacity at a pixel. The gain g raises the transmittance 1 − α to a power. Doubling g doubles the smoke’s optical thickness instead of clipping the opacity at 1.
- m is the field mask. The scope’s black vignette is found as dark pixels connected to the frame’s border, and smoke never sits there: it is in the abdomen, not on the lens housing.
- C is the smoke’s colour scaled by a brightness b, by the frame’s own lamp level L (the 99th percentile of brightness) and by a small tint τ. A white render pasted at full brightness over a dim frame is the first thing a model learns to spot.
- J′ is the clean frame, blurred where the smoke is dense: forward scattering softens edges behind smoke rather than only fading them.
Every parameter is drawn from a generator seeded per pair: scale 0.8 to 1.4, shift up to a quarter of the frame, gain 0.6 to 2.5, brightness 0.75 to 1.05, tint ±4 %, blur up to 2.5 pixels at 854 pixels of width. The validation and test pairs therefore regenerate bit for bit. Smoke changes no label, because the tissue is still there behind it. The pairs inherit the licence of their clean frames, CholecT50’s CC BY-NC-SA 4.0. On 518 synthetic validation pairs, the residual U-Net trained on them raised PSNR from 28.99 dB for the smoky input to 33.08 dB. It has not been evaluated on real smoke yet (M5 model card).
Try the same model on a synthetic frame. The smoke layer here is noise shaped into a plume by code, not a Blender render; everything after it follows composite().
The two missing effects have their own physics, which an implementation would have to respect:
- Glare. The lamp sits on the scope’s axis, so polished metal jaws reflect it almost straight back into the camera. The highlight saturates the sensor, clips to white, blooms into its surroundings and moves with the instrument’s angle. A physical version would render instruments with a metallic surface, as the RFC 0001 renders do, rather than paint bright spots.
- Blood on the lens. A drop sits on the front lens, far outside the depth of field, so it shows as a soft red disc that magnifies and tints what is behind it. It stays from frame to frame until the lens is cleaned.
Whether the pixels hidden by blood or glare keep their label is a decision for the annotation guideline and the clinical lead. The sketches in the tool above draw the look and nothing more.
Checklist
- Generate the mask first and keep it as the label; check the image against it.
- Keep the field of view in the range the model was trained on.
- Keep synthetic data out of validation and test, and report results on real data.
- Record the weights, version, configuration, seed and licence of every synthetic set; mind the non-commercial terms.
- Check for memorisation before calling synthetic data private.
- In
osv.augment, only smoke is implemented; do not claim blood or glare augmentation.
Further reading
- Guo et al. MAISI: Medical AI for Synthetic Imaging. WACV 2025.
- Zhao et al. MAISI-v2: Accelerated 3D High-Resolution Medical Image Synthesis with Rectified Flow and Region-specific Contrastive Loss. AAAI 2026.
- Zhao et al. NV-Generate-Body: 3D Latent Diffusion Model for Controllable Anatomy, Tumor, and Demographic Body Mask Synthesis. 2026.
- The NV-Generate-CTMR repository, its weights and its licences.
- Ho et al. Denoising Diffusion Probabilistic Models. 2020. Rombach et al. High-Resolution Image Synthesis with Latent Diffusion Models. 2022.
- Liu et al. Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow. 2022.
- Perez et al. FiLM: Visual Reasoning with a General Conditioning Layer. 2018. Ho and Salimans. Classifier-Free Diffusion Guidance. 2022. Zhang et al. Adding Conditional Control to Text-to-Image Diffusion Models. 2023.
- Dar et al. Investigating Data Memorization in 3D Latent Diffusion Models for Medical Image Synthesis. 2023.
- In the osv repository:
osv/augment/smoke.py, the M5 model card and RFC 0001.