What this programme is
A course outline for contributors, machine learning engineers and researchers who join OpenSurgicalVision. It runs from the structure of medical data files to fast inference in surgical robotics, in nine modules and eighteen lectures. Two ideas run through it: every result must be reproducible end to end, and people and AI coding agents work on the code together.
RESEARCH USE ONLY — NOT FOR CLINICAL USE. The software, models, weights, augmentations and datasets of OpenSurgicalVision are for research, benchmarking and teaching. The project is not a medical device, has no FDA 510(k)/PMA clearance or EU MDR 2017/745 certification, and must not be used for diagnosis, decisions during surgery or patient care.
How to read it
This is a draft, waiting for review.
- Not everything here is built yet.
osv.deid,osv.augment,osv.datasets,osv.evalandosv.trainexist and work.osv.annotate,osv.serve,osv.robustnessandosv.deployso far hold only their interface and intent; the lectures that use them describe the planned design. - Some tools come from outside the project. MAISI, VISTA-3D, the NVIDIA medical AI skills catalogue, MONAI Deploy, Mercure, Triton and ClearML are tools of the field that the course teaches. OpenSurgicalVision does not use them yet.
- The pass criteria are targets for exercises, not acceptance thresholds of the project.
To practise right away, start with the Colab experiments: ten notebooks on synthetic data cover patient-level splits, segmentation models and metrics from modules 3, 4 and 8.
Module 1. Medical data formats, volume geometry and de-identification
Lecture 1. Surgical and diagnostic data formats
The operating room produces 2D frames (PNG, JPG), laparoscopic video (MP4 with H.264/HEVC) and 3D preoperative scans (DICOM, NIfTI). What each format holds and how it is built:
- Endoscopic video: inter-frame compression, variable bit rate, 25–60 frames per second, YUV420p against sRGB and what electrocautery flashes do to colour.
- DICOM: information object definitions (IODs), the patient → study → series → instance hierarchy, reading metadata with
pydicom. - NIfTI (
.nii,.nii.gz): from separate DICOM slices to one continuous 3D/4D voxel grid with a fixed spacing. - Volume geometry: the 4×4 affine matrix that maps a voxel’s index in the grid to millimetre coordinates in the patient’s space: a 3×3 rotation and scaling block plus a translation.
- Orientation and Hounsfield units: RAS against LPS coordinate systems; converting stored CT values to Hounsfield units with the rescale slope and intercept,
HU = SV × slope + intercept.
Read the lecture, with diagrams, formulas, interactive examples and 3D animations.
Lecture 2. De-identification in four layers
Public repositories and benchmarks must never leak protected health information (PHI). osv.deid defends in four layers:
- DICOM metadata (PS3.15): removing or pseudonymising direct and indirect identifiers under the Basic Application Level Confidentiality Profile: patient tags, institution addresses, procedure dates, scanner serial numbers.
- Private tags: removing vendor private blocks, where undocumented patient data often hides.
- Burned-in text: OCR over frames and ultrasound images for printed names, record numbers or hospital names, then masking them permanently.
- Defacing: altering the facial surface in preoperative CT and MRI so a face cannot be rebuilt in a 3D renderer.
Today osv.deid is the gate that checks these four layers and fails a pull request that still contains PHI; the removal itself is done beforehand with dedicated tools. Read the lecture, with diagrams, formulas, interactive examples and a 3D animation.
Module 2. Annotation, active learning and generative synthesis
Lecture 3. Assisted annotation and agreement between annotators
Labelling dense video by hand is slow, costly and subjective. This lecture covers modern annotation set-ups:
- Annotation platforms: connectors from
osv.annotate(planned) to CVAT and Label Studio, with polygons, binary masks and semantic tags. - Pre-annotation with foundation models: Segment Anything 2 (SAM 2) for tracking and segmenting instruments over time from positive and negative clicks or boxes.
- Active learning with MONAI Label: a client-server loop between 3D Slicer or OHIF and a training server that picks the unlabelled cases the model is least certain about.
- Inter-rater reliability: Cohen’s and Fleiss’ kappa for categorical labels,
κ = (Po − Pe) / (1 − Pe), where Po is the observed agreement and Pe the agreement expected by chance; Dice and IoU for comparing masks pixel by pixel.
Read the lecture, with diagrams, formulas, interactive examples and 3D animations.
Lecture 4. Synthesis of medical volumes, masks and surgical artefacts
Rare conditions and anatomical variants are scarce in labelled data. 3D generative models can fill the gap:
- MAISI and 3D latent diffusion: a 3D VAE compresses CT or MRI volumes into a compact latent space, and a diffusion model learns there.
- Rectified flow (MAISI v2): the
rflow-ct,rflow-mrandrflow-mr-brainweights. Onlyrflow-ctmakes pairs of a synthetic volume and a 132-class segmentation mask; the MRI weights make images alone. Its authors report about 30 sampling steps instead of DDPM’s 1000, roughly 33 times faster, with a better FID. - Controlled body geometry (
rflow-mask, NV-Generate-Body): a 19-value conditioning vector, applied through adaptive group normalisation, sets the size of one of 9 organs or 5 kinds of tumour, and demographics: age, sex, weight or BMI, and height. - CT from your own masks with ControlNet: conditioning on MAISI’s 132-class vocabulary, always including the body envelope (label 200).
- Surgical artefacts in
osv.augment: a physical model of surgical smoke composited over CholecT50 frames. Blood on the lens and glare on metal jaws are listed as robustness tests but not yet implemented.
Read the lecture, with diagrams, formulas, interactive examples and 3D animations.
Module 3. Datasets and preprocessing pipelines
Lecture 5. Surgical datasets and patient-level splits
- Datasets: Cholec80 (80 cholecystectomy videos with phase and instrument labels), CholecSeg8k (frame-level segmentation of 13 classes), EndoVis (MICCAI instrument challenges) and the TotalSegmentator CT dataset.
- Leakage: neighbouring frames of one operation are nearly identical. Shuffling frames of one patient into both train and test inflates the score, because the model memorises that patient’s anatomy.
- Patient-level splits in
osv.datasets:train,valandtestare split by patient, so no anatomy appears in two splits.
Read the lecture, with diagrams, formulas, interactive examples and 3D animations.
Lecture 6. Preprocessing, normalisation and fast data loading with MONAI
- Spatial transforms: resampling to isotropic 1.0 × 1.0 × 1.0 mm voxels (
Spacingd), orienting to RAS (Orientationd), cropping to the informative region (CropForegroundd). - Intensity normalisation: Hounsfield windowing for soft tissue (
ScaleIntensityRanged, −175 to +250 HU) and standardising MRI intensities to zero mean and unit variance. - Faster input:
CacheDatasetkeeps deterministic transforms in memory,PersistentDatasetstores precomputed tensors on a fast local disk, so the GPU does not wait for data.
Read the lecture, with diagrams, formulas, interactive examples and 3D animations.
Module 4. Neural network architectures for classification and segmentation
Lecture 7. The project’s tasks, M1–M5
- M1, instrument segmentation: jaws, shafts and joints of graspers, dissectors and scissors.
- M2, anatomy segmentation: gallbladder, liver, hepatoduodenal ligament and critical structures such as the cystic artery and cystic duct.
- M3, phase recognition: the stages of an operation (dissection, clipping, cutting, extraction).
- M4, preoperative 3D segmentation: organs and vessels in CT and MRI.
- M5, desmoking and restoration: restoring the view through dense smoke.
Read the lecture, with diagrams, a formula, interactive examples and 3D animations.
Lecture 8. Architectures in MONAI Core and Lightning
- From U-Net to transformers: UNETR (a ViT encoder with a deconvolution decoder) and SwinUNETR (hierarchical shifted windows with local attention).
- VISTA-3D: a foundation model trained on large medical cohorts that combines automatic segmentation of hundreds of classes with click-based interactive segmentation.
- Scaling training: Distributed Data Parallel and Fully Sharded Data Parallel on multi-GPU machines;
bf16-mixedprecision to keep gradients stable on high-resolution 3D volumes.
Read the lecture, with diagrams, formulas, interactive examples and 3D animations.
Module 5. Robotic systems and computer-assisted intervention
Lecture 9. Computer-assisted intervention and instrument tracking
- Computer-assisted intervention (CAI): computer vision as the sensing layer for intelligent manipulators and surgical robots.
- Tracking in space and time: estimating the 6-DoF pose of instruments relative to anatomical landmarks.
- Continuous video: suppressing flicker and smoothing masks over time with Kalman filters and recurrent layers.
Read the lecture, with diagrams, formulas, interactive examples and 3D animations.
Lecture 10. Streaming inference under a hard latency budget
- Streaming with
osv.serve(planned): an asynchronous FastAPI server over WebSockets or gRPC that overlays masks on live video without extra memory copies. - The latency budget: p95 latency ≤ 50 ms at ≥ 30 frames per second, as declared in
osv.deploy. - Latency profiling: splitting the pipeline into capture, colour conversion, transfer to the GPU, the forward pass, mask post-processing and display.
Read the lecture, with diagrams, formulas, interactive examples and 3D animations.
Module 6. AI agents, the Model Context Protocol and medical AI skills
Lecture 11. AI-first development and MCP
- AI-first contributors: the project’s shared context lives in
.ai/context.md, with prompt templates in.ai/prompts/. - One set of rules for every agent:
make setup-aigenerates the configuration for Claude Code, Cursor, GitHub Copilot and Aider;make check-aichecks they agree. - Model Context Protocol: the project configures 13 MCP servers in
.ai/mcp/servers.json. They give agents controlled access to datasets, the MLflow experiment server, CVAT annotations and the literature without touching the source data.
Lecture 12. A catalogue of verified skills
NVIDIA’s medical AI skills catalogue (external, not used by the project yet) shows how agent work can be standardised:
- A skill’s specification (
SKILL.md) and manifest (skill_manifest.yaml): inputs, environment limits, dependencies and side effects. - Typical tasks: DICOM preprocessing (
dicom-series-to-volume), general segmentation (nv-segment-ct), CT and MRI synthesis (nv-generate-ct-rflow,nv-generate-mr-brain). - Evidence packs and paired verifiers: an automatic audit trail (hashes of inputs and outputs, GPU memory logs, no PHI in the output) and checkers that test whether a result is physically plausible (
ct_synthesis_quality_v1,dicom_volume_quality_v1).
Module 7. Fine-tuning and experiment management
Lecture 13. Fine-tuning and continual learning
- MONAI bundles: self-contained model packages from the MONAI Model Zoo, changed through
config.jsonwithout touching the architecture code. - Fine-tuning generation and segmentation models: for example VISTA-3D and latent diffusion models on local data (
nv-segment-ct-finetune,nv-generate-mr-brain-finetune). - Continual learning: avoiding catastrophic forgetting when instruments or light sources change, with contrastive losses and a frozen encoder.
Lecture 14. Reproducibility, data versions and experiment tracking
- Versioning large data: DVC ties Git commits to data checksums in
dvc.lock. - Tracking experiments: MLflow (the project’s server), Aim or ClearML for metrics, loss curves, attention maps and hyperparameters; exporting models with their metadata for the Hugging Face Hub.
Module 8. Evaluation metrics and robustness testing
Lecture 15. Metrics for medical computer vision
- Overlap: the Dice similarity coefficient, twice the overlap of prediction and ground truth divided by the sum of their areas, and the Jaccard index (IoU), the overlap divided by their union.
- Boundary distances: the 95th percentile of the Hausdorff distance (HD95) and the average surface distance (ASD), which catch outliers and local boundary errors.
- Quality of synthetic data: realism by the Fréchet Inception Distance (FID), diversity by SSIM and multi-scale SSIM.
The formulas, with pictures and an interactive example, are on the metrics page.
Lecture 16. Stress-testing robustness
- Optical anomalies: heavy bleeding, sudden defocus and halos from back-scattered light, with the perturbation catalogue in
osv.robustness(the full suite is planned). - Automation bias: finding hidden systematic errors when patients’ demographic or anatomical characteristics shift.
Module 9. Model export, inference optimisation and deployment
Lecture 17. Optimising for hardware: ONNX and TensorRT
- ONNX export: freezing the PyTorch graph and checking operator support for 3D operations such as
GridSampleand 3D convolutions. - TensorRT: fusing layers (convolution, bias and activation in one CUDA kernel) and calibrating FP16 and INT8 quantisation so Dice drops by less than 0.005.
Lecture 18. Containers and orchestration in hospital IT
- MONAI Application Package (MAP): packaging a pipeline with the MONAI Deploy App SDK, from reading DICOM to writing DICOM SEG and DICOM SR.
- Serving at scale: Triton Inference Server with dynamic batching, and BentoML.
- Routing clinical data: the open-source Mercure DICOM orchestrator receives series from scanners, runs MAP containers and returns masks to PACS or 3D Slicer.
Skills matrix
Each module ends with a practical task. The pass criteria are targets for the exercise, not project thresholds.
| Module | Main tools | Input | Output | Pass criterion |
|---|---|---|---|---|
| 1 (lectures 1–2) | pydicom, nibabel, osv.deid |
Raw CT/MRI DICOM series, MP4 video | NIfTI volumes, de-identified frames | All files pass the DICOM preflight check; no PHI tags left |
| 2 (lectures 3–4) | SAM 2, MONAI Label, rflow-ct, ControlNet | Unlabelled surgical video, CT masks | Polygon masks, synthetic CT and mask pairs | Fleiss’ κ ≥ 0.75; synthetic CT with FID ≤ 6.0 on the test split |
| 3 (lectures 5–6) | osv.datasets, MONAI transforms |
Cholec80, CholecSeg8k, TotalSegmentator | Patient-level splits, tensor cache | No patient in two splits; data loading at least 3× faster |
| 4 (lectures 7–8) | PyTorch, MONAI Core, Lightning | Normalised frames and 3D volumes | M1–M5 checkpoints | Stable training in bf16-mixed; Dice ≥ 0.82 on validation |
| 5 (lectures 9–10) | osv.serve, FastAPI, WebSockets |
A live stream of endoscopic frames | Streamed instrument masks | p95 ≤ 50 ms at ≥ 30 frames per second |
| 6 (lectures 11–12) | MCP servers, Claude Code, medical AI skills | SKILL.md specifications, manifests |
Evidence packs | Passes the paired verifier |
| 7 (lectures 13–14) | MONAI bundles, DVC, MLflow, ClearML | VISTA-3D checkpoints, target datasets | Fine-tuned weights pinned in dvc.lock |
Pipeline reproduces from the commit hash; no drop on neighbouring classes |
| 8 (lectures 15–16) | osv.eval, osv.robustness, SciPy |
Model predictions, stress tests | Confusion matrices, degradation profiles | HD95 reported; Dice drops by no more than 15% under smoke |
| 9 (lectures 17–18) | TensorRT, ONNX, MONAI Deploy SDK | PyTorch .pt checkpoints, DICOM folders |
Optimised .engine, MAP container |
Inference at least 2.5× faster than PyTorch; valid DICOM SEG |
Where to go next
- Colab experiments: practice for modules 3, 4 and 8 on synthetic data, no medical data needed.
- For computer science students: what you get from the project and how to start.
- Research topics: term papers, theses and talks.
- How contributing works: community norms and your first pull request.