LEARNING PROGRAMME

Learning programme: computer vision and AI in surgery

Nine modules and eighteen lectures for OpenSurgicalVision contributors: from DICOM and de-identification to datasets, models, real-time inference, AI agents, evaluation and deployment.

This page is still being reviewed by our scientific, clinical and legal experts. It describes plans, not finished work.

What this programme is

A course outline for contributors, machine learning engineers and researchers who join OpenSurgicalVision. It runs from the structure of medical data files to fast inference in surgical robotics, in nine modules and eighteen lectures. Two ideas run through it: every result must be reproducible end to end, and people and AI coding agents work on the code together.

RESEARCH USE ONLY — NOT FOR CLINICAL USE. The software, models, weights, augmentations and datasets of OpenSurgicalVision are for research, benchmarking and teaching. The project is not a medical device, has no FDA 510(k)/PMA clearance or EU MDR 2017/745 certification, and must not be used for diagnosis, decisions during surgery or patient care.

How to read it

This is a draft, waiting for review.

  • Not everything here is built yet. osv.deid, osv.augment, osv.datasets, osv.eval and osv.train exist and work. osv.annotate, osv.serve, osv.robustness and osv.deploy so far hold only their interface and intent; the lectures that use them describe the planned design.
  • Some tools come from outside the project. MAISI, VISTA-3D, the NVIDIA medical AI skills catalogue, MONAI Deploy, Mercure, Triton and ClearML are tools of the field that the course teaches. OpenSurgicalVision does not use them yet.
  • The pass criteria are targets for exercises, not acceptance thresholds of the project.

To practise right away, start with the Colab experiments: ten notebooks on synthetic data cover patient-level splits, segmentation models and metrics from modules 3, 4 and 8.

Module 1. Medical data formats, volume geometry and de-identification

Lecture 1. Surgical and diagnostic data formats

The operating room produces 2D frames (PNG, JPG), laparoscopic video (MP4 with H.264/HEVC) and 3D preoperative scans (DICOM, NIfTI). What each format holds and how it is built:

  • Endoscopic video: inter-frame compression, variable bit rate, 25–60 frames per second, YUV420p against sRGB and what electrocautery flashes do to colour.
  • DICOM: information object definitions (IODs), the patient → study → series → instance hierarchy, reading metadata with pydicom.
  • NIfTI (.nii, .nii.gz): from separate DICOM slices to one continuous 3D/4D voxel grid with a fixed spacing.
  • Volume geometry: the 4×4 affine matrix that maps a voxel’s index in the grid to millimetre coordinates in the patient’s space: a 3×3 rotation and scaling block plus a translation.
  • Orientation and Hounsfield units: RAS against LPS coordinate systems; converting stored CT values to Hounsfield units with the rescale slope and intercept, HU = SV × slope + intercept.

Read the lecture, with diagrams, formulas, interactive examples and 3D animations.

Lecture 2. De-identification in four layers

Public repositories and benchmarks must never leak protected health information (PHI). osv.deid defends in four layers:

  1. DICOM metadata (PS3.15): removing or pseudonymising direct and indirect identifiers under the Basic Application Level Confidentiality Profile: patient tags, institution addresses, procedure dates, scanner serial numbers.
  2. Private tags: removing vendor private blocks, where undocumented patient data often hides.
  3. Burned-in text: OCR over frames and ultrasound images for printed names, record numbers or hospital names, then masking them permanently.
  4. Defacing: altering the facial surface in preoperative CT and MRI so a face cannot be rebuilt in a 3D renderer.

Today osv.deid is the gate that checks these four layers and fails a pull request that still contains PHI; the removal itself is done beforehand with dedicated tools. Read the lecture, with diagrams, formulas, interactive examples and a 3D animation.

Module 2. Annotation, active learning and generative synthesis

Lecture 3. Assisted annotation and agreement between annotators

Labelling dense video by hand is slow, costly and subjective. This lecture covers modern annotation set-ups:

  • Annotation platforms: connectors from osv.annotate (planned) to CVAT and Label Studio, with polygons, binary masks and semantic tags.
  • Pre-annotation with foundation models: Segment Anything 2 (SAM 2) for tracking and segmenting instruments over time from positive and negative clicks or boxes.
  • Active learning with MONAI Label: a client-server loop between 3D Slicer or OHIF and a training server that picks the unlabelled cases the model is least certain about.
  • Inter-rater reliability: Cohen’s and Fleiss’ kappa for categorical labels, κ = (Po − Pe) / (1 − Pe), where Po is the observed agreement and Pe the agreement expected by chance; Dice and IoU for comparing masks pixel by pixel.

Read the lecture, with diagrams, formulas, interactive examples and 3D animations.

Lecture 4. Synthesis of medical volumes, masks and surgical artefacts

Rare conditions and anatomical variants are scarce in labelled data. 3D generative models can fill the gap:

  • MAISI and 3D latent diffusion: a 3D VAE compresses CT or MRI volumes into a compact latent space, and a diffusion model learns there.
  • Rectified flow (MAISI v2): the rflow-ct, rflow-mr and rflow-mr-brain weights. Only rflow-ct makes pairs of a synthetic volume and a 132-class segmentation mask; the MRI weights make images alone. Its authors report about 30 sampling steps instead of DDPM’s 1000, roughly 33 times faster, with a better FID.
  • Controlled body geometry (rflow-mask, NV-Generate-Body): a 19-value conditioning vector, applied through adaptive group normalisation, sets the size of one of 9 organs or 5 kinds of tumour, and demographics: age, sex, weight or BMI, and height.
  • CT from your own masks with ControlNet: conditioning on MAISI’s 132-class vocabulary, always including the body envelope (label 200).
  • Surgical artefacts in osv.augment: a physical model of surgical smoke composited over CholecT50 frames. Blood on the lens and glare on metal jaws are listed as robustness tests but not yet implemented.

Read the lecture, with diagrams, formulas, interactive examples and 3D animations.

Module 3. Datasets and preprocessing pipelines

Lecture 5. Surgical datasets and patient-level splits

  • Datasets: Cholec80 (80 cholecystectomy videos with phase and instrument labels), CholecSeg8k (frame-level segmentation of 13 classes), EndoVis (MICCAI instrument challenges) and the TotalSegmentator CT dataset.
  • Leakage: neighbouring frames of one operation are nearly identical. Shuffling frames of one patient into both train and test inflates the score, because the model memorises that patient’s anatomy.
  • Patient-level splits in osv.datasets: train, val and test are split by patient, so no anatomy appears in two splits.

Read the lecture, with diagrams, formulas, interactive examples and 3D animations.

Lecture 6. Preprocessing, normalisation and fast data loading with MONAI

  • Spatial transforms: resampling to isotropic 1.0 × 1.0 × 1.0 mm voxels (Spacingd), orienting to RAS (Orientationd), cropping to the informative region (CropForegroundd).
  • Intensity normalisation: Hounsfield windowing for soft tissue (ScaleIntensityRanged, −175 to +250 HU) and standardising MRI intensities to zero mean and unit variance.
  • Faster input: CacheDataset keeps deterministic transforms in memory, PersistentDataset stores precomputed tensors on a fast local disk, so the GPU does not wait for data.

Read the lecture, with diagrams, formulas, interactive examples and 3D animations.

Module 4. Neural network architectures for classification and segmentation

Lecture 7. The project’s tasks, M1–M5

  • M1, instrument segmentation: jaws, shafts and joints of graspers, dissectors and scissors.
  • M2, anatomy segmentation: gallbladder, liver, hepatoduodenal ligament and critical structures such as the cystic artery and cystic duct.
  • M3, phase recognition: the stages of an operation (dissection, clipping, cutting, extraction).
  • M4, preoperative 3D segmentation: organs and vessels in CT and MRI.
  • M5, desmoking and restoration: restoring the view through dense smoke.

Read the lecture, with diagrams, a formula, interactive examples and 3D animations.

Lecture 8. Architectures in MONAI Core and Lightning

  • From U-Net to transformers: UNETR (a ViT encoder with a deconvolution decoder) and SwinUNETR (hierarchical shifted windows with local attention).
  • VISTA-3D: a foundation model trained on large medical cohorts that combines automatic segmentation of hundreds of classes with click-based interactive segmentation.
  • Scaling training: Distributed Data Parallel and Fully Sharded Data Parallel on multi-GPU machines; bf16-mixed precision to keep gradients stable on high-resolution 3D volumes.

Read the lecture, with diagrams, formulas, interactive examples and 3D animations.

Module 5. Robotic systems and computer-assisted intervention

Lecture 9. Computer-assisted intervention and instrument tracking

  • Computer-assisted intervention (CAI): computer vision as the sensing layer for intelligent manipulators and surgical robots.
  • Tracking in space and time: estimating the 6-DoF pose of instruments relative to anatomical landmarks.
  • Continuous video: suppressing flicker and smoothing masks over time with Kalman filters and recurrent layers.

Read the lecture, with diagrams, formulas, interactive examples and 3D animations.

Lecture 10. Streaming inference under a hard latency budget

  • Streaming with osv.serve (planned): an asynchronous FastAPI server over WebSockets or gRPC that overlays masks on live video without extra memory copies.
  • The latency budget: p95 latency ≤ 50 ms at ≥ 30 frames per second, as declared in osv.deploy.
  • Latency profiling: splitting the pipeline into capture, colour conversion, transfer to the GPU, the forward pass, mask post-processing and display.

Read the lecture, with diagrams, formulas, interactive examples and 3D animations.

Module 6. AI agents, the Model Context Protocol and medical AI skills

Lecture 11. AI-first development and MCP

  • AI-first contributors: the project’s shared context lives in .ai/context.md, with prompt templates in .ai/prompts/.
  • One set of rules for every agent: make setup-ai generates the configuration for Claude Code, Cursor, GitHub Copilot and Aider; make check-ai checks they agree.
  • Model Context Protocol: the project configures 13 MCP servers in .ai/mcp/servers.json. They give agents controlled access to datasets, the MLflow experiment server, CVAT annotations and the literature without touching the source data.

Lecture 12. A catalogue of verified skills

NVIDIA’s medical AI skills catalogue (external, not used by the project yet) shows how agent work can be standardised:

  • A skill’s specification (SKILL.md) and manifest (skill_manifest.yaml): inputs, environment limits, dependencies and side effects.
  • Typical tasks: DICOM preprocessing (dicom-series-to-volume), general segmentation (nv-segment-ct), CT and MRI synthesis (nv-generate-ct-rflow, nv-generate-mr-brain).
  • Evidence packs and paired verifiers: an automatic audit trail (hashes of inputs and outputs, GPU memory logs, no PHI in the output) and checkers that test whether a result is physically plausible (ct_synthesis_quality_v1, dicom_volume_quality_v1).

Module 7. Fine-tuning and experiment management

Lecture 13. Fine-tuning and continual learning

  • MONAI bundles: self-contained model packages from the MONAI Model Zoo, changed through config.json without touching the architecture code.
  • Fine-tuning generation and segmentation models: for example VISTA-3D and latent diffusion models on local data (nv-segment-ct-finetune, nv-generate-mr-brain-finetune).
  • Continual learning: avoiding catastrophic forgetting when instruments or light sources change, with contrastive losses and a frozen encoder.

Lecture 14. Reproducibility, data versions and experiment tracking

  • Versioning large data: DVC ties Git commits to data checksums in dvc.lock.
  • Tracking experiments: MLflow (the project’s server), Aim or ClearML for metrics, loss curves, attention maps and hyperparameters; exporting models with their metadata for the Hugging Face Hub.

Module 8. Evaluation metrics and robustness testing

Lecture 15. Metrics for medical computer vision

  • Overlap: the Dice similarity coefficient, twice the overlap of prediction and ground truth divided by the sum of their areas, and the Jaccard index (IoU), the overlap divided by their union.
  • Boundary distances: the 95th percentile of the Hausdorff distance (HD95) and the average surface distance (ASD), which catch outliers and local boundary errors.
  • Quality of synthetic data: realism by the Fréchet Inception Distance (FID), diversity by SSIM and multi-scale SSIM.

The formulas, with pictures and an interactive example, are on the metrics page.

Lecture 16. Stress-testing robustness

  • Optical anomalies: heavy bleeding, sudden defocus and halos from back-scattered light, with the perturbation catalogue in osv.robustness (the full suite is planned).
  • Automation bias: finding hidden systematic errors when patients’ demographic or anatomical characteristics shift.

Module 9. Model export, inference optimisation and deployment

Lecture 17. Optimising for hardware: ONNX and TensorRT

  • ONNX export: freezing the PyTorch graph and checking operator support for 3D operations such as GridSample and 3D convolutions.
  • TensorRT: fusing layers (convolution, bias and activation in one CUDA kernel) and calibrating FP16 and INT8 quantisation so Dice drops by less than 0.005.

Lecture 18. Containers and orchestration in hospital IT

  • MONAI Application Package (MAP): packaging a pipeline with the MONAI Deploy App SDK, from reading DICOM to writing DICOM SEG and DICOM SR.
  • Serving at scale: Triton Inference Server with dynamic batching, and BentoML.
  • Routing clinical data: the open-source Mercure DICOM orchestrator receives series from scanners, runs MAP containers and returns masks to PACS or 3D Slicer.

Skills matrix

Each module ends with a practical task. The pass criteria are targets for the exercise, not project thresholds.

Module Main tools Input Output Pass criterion
1 (lectures 1–2) pydicom, nibabel, osv.deid Raw CT/MRI DICOM series, MP4 video NIfTI volumes, de-identified frames All files pass the DICOM preflight check; no PHI tags left
2 (lectures 3–4) SAM 2, MONAI Label, rflow-ct, ControlNet Unlabelled surgical video, CT masks Polygon masks, synthetic CT and mask pairs Fleiss’ κ ≥ 0.75; synthetic CT with FID ≤ 6.0 on the test split
3 (lectures 5–6) osv.datasets, MONAI transforms Cholec80, CholecSeg8k, TotalSegmentator Patient-level splits, tensor cache No patient in two splits; data loading at least 3× faster
4 (lectures 7–8) PyTorch, MONAI Core, Lightning Normalised frames and 3D volumes M1–M5 checkpoints Stable training in bf16-mixed; Dice ≥ 0.82 on validation
5 (lectures 9–10) osv.serve, FastAPI, WebSockets A live stream of endoscopic frames Streamed instrument masks p95 ≤ 50 ms at ≥ 30 frames per second
6 (lectures 11–12) MCP servers, Claude Code, medical AI skills SKILL.md specifications, manifests Evidence packs Passes the paired verifier
7 (lectures 13–14) MONAI bundles, DVC, MLflow, ClearML VISTA-3D checkpoints, target datasets Fine-tuned weights pinned in dvc.lock Pipeline reproduces from the commit hash; no drop on neighbouring classes
8 (lectures 15–16) osv.eval, osv.robustness, SciPy Model predictions, stress tests Confusion matrices, degradation profiles HD95 reported; Dice drops by no more than 15% under smoke
9 (lectures 17–18) TensorRT, ONNX, MONAI Deploy SDK PyTorch .pt checkpoints, DICOM folders Optimised .engine, MAP container Inference at least 2.5× faster than PyTorch; valid DICOM SEG

Where to go next

Explore all contribution tracks