SURGICAL VIDEO

Segmenting surgical video in real time

Frameworks, architectures and latency budgets for segmenting instruments and tissue in laparoscopic video: per-frame models, temporal models, SAM 2 and deployment in the operating room.

A laparoscope delivers 25 to 60 colour frames a second. A model that runs during the operation has to segment each frame before the next one arrives, keep its masks stable from frame to frame, and cope with smoke, blood, glare, motion blur and instruments that come and go. This page covers the data, the frameworks, the main architectures and the latency budget. For CT and MRI, see segmentation of 3D volumes; for an overview, the comparison of the two.

The data

  • Frames. Cholec80 videos, the source of most laparoscopic datasets, are 25 fps; CholecSeg8k frames are 854 × 480. Endoscope images have a dark border outside the circular or octagonal field of view, and a text overlay on some systems, both of which a model must not learn as features.
  • Labels. CholecSeg8k has 8,080 frames from 17 Cholec80 videos with 13 classes of tissue and instruments; CholecT50 labels instrument–verb–target triplets; EndoVis 2017 has robotic instrument parts and types. Labels exist for a fraction of frames, so models learn per frame and the temporal part comes later.
  • Splits by video. Neighbouring frames are almost identical; a frame-level split leaks the test set into training. Split by whole operation.
  • Licences. The Cholec datasets are CC BY-NC-SA; a model trained on them inherits the non-commercial and share-alike terms. See the project’s datasets.

Frameworks

  • OpenSurgicalVision. The project’s code: loaders for CholecSeg8k and CholecT50 with official whole-video splits (osv.datasets), training augmentations, synthetic smoke and instrument compositing (osv.augment), an architecture registry that experiments must use (osv.arch), and segmentation metrics (osv.eval). It has no EndoVis loader and no released checkpoint yet; the M1 model card records a DeepLabV3-ResNet50 baseline and its results.
  • torchvision has pre-trained segmentation models (DeepLabV3, LR-ASPP with MobileNetV3) and video backbones (R(2+1)D, MViT, Video Swin as swin3d). PyTorchVideo adds video data loading and more video models.
  • TensorRT compiles a trained network for a specific GPU, fusing layers and running in FP16 or INT8.
  • Holoscan SDK is NVIDIA’s framework for low-latency sensor pipelines, endoscopy included: it moves frames from capture card to GPU to display without copies through the CPU. Triton Inference Server serves models over the network to many clients; that hop adds latency, so it suits offline analysis better than the loop in the operating room.
  • MONAI has 2D bundles too: endoscopic_tool_segmentation is a U-Net with an EfficientNet-B2 encoder for binary instrument masks at 736 × 480.

Architectures

One frame at a time

Model Idea Reference
DeepLabV3+ Atrous convolutions keep resolution; a spatial pyramid of dilations sees context at several scales; a light decoder sharpens the boundaries Chen et al., ECCV 2018
HRNet Keeps a high-resolution branch through the whole network and exchanges information with lower-resolution branches Wang et al., TPAMI 2020
SegFormer Hierarchical transformer encoder without positional encodings and an all-MLP decoder; B0 to B5 sizes Xie et al., NeurIPS 2021
EfficientNet U-Net A U-Net decoder on an EfficientNet encoder; a common choice for edge devices MONAI FlexibleUNet

Per-frame models are the baseline: they are fast and simple to train on sparse labels, but each frame is judged alone, so masks flicker and an instrument hidden by smoke vanishes.

DeepLabV3+, block by block

  • convolution
  • attention
  • change of resolution
  • memory, state
  • input, output
frame3 × 480 × 854ResNet-50atrousoutput stride 1611×1 conv3×3, rate 623×3, rate 123×3, rate 18image pooling3concat1×1 conv↑4concat,3×3 convs,↑4 → mask51×1 conv · features at 1/44
DeepLabV3+ with a ResNet-50 backbone at an output stride of 16, on a CholecSeg8k frame size.
  1. Backbone. ResNet-50 whose last stages use dilated convolutions instead of striding, so features stay at 1/16 of the frame instead of 1/32.
  2. Atrous spatial pyramid pooling (ASPP). Parallel 3×3 convolutions with dilation rates 6, 12 and 18, plus a 1×1 convolution, look at the same features with different fields of view: thin instrument tips and whole organs.
  3. Image pooling. Features averaged over the whole frame add global context, such as how much smoke or blood there is.
  4. Low-level features. Features from early in the backbone, at 1/4 of the frame, carry sharp edges; a 1×1 convolution reduces their channels.
  5. Decoder. The pyramid’s output is upsampled four times, concatenated with the low-level features, refined by 3×3 convolutions and upsampled four times to the mask. This decoder is the “+” over DeepLabV3; the project’s M1 baseline is DeepLabV3 without it.

SegFormer, block by block

framestage 11/41MLPalready 1/44stage 21/82MLP↑ ×2 to 1/4stage 31/16MLP↑ ×4 to 1/4stage 41/32MLP↑ ×8 to 1/4attention + Mix-FFN3concatMLP fuse5mask1/4 → ↑4
The four encoder stages and the all-MLP decoder of SegFormer.
  1. Overlapping patch embedding. A strided convolution turns the frame into tokens at 1/4 of its size; later stages merge them to 1/8, 1/16 and 1/32.
  2. Efficient self-attention. Keys and values come from a shrunk version of the token grid, which makes global attention affordable at high resolution.
  3. Mix-FFN. A 3×3 depthwise convolution inside the feed-forward layer gives tokens a sense of position, so the model needs no positional encoding and copes with other frame sizes.
  4. All-MLP decoder. An MLP projects each stage’s output to one width, and it is upsampled to 1/4.
  5. Fuse and predict. The four maps are concatenated, fused by another MLP and turned into a mask at 1/4 of the frame, then upsampled to full size.

Using time

  • ConvLSTM and ConvGRU (Shi et al., NeurIPS 2015) put a recurrent cell with convolutional gates between encoder and decoder. The cell carries a feature map from frame to frame, which steadies masks at the cost of one hidden state in memory.
  • Video transformers, TimeSformer and Video Swin, attend across space and time over a clip. They were built for clip classification and are used in surgery mostly for phase and action recognition; for dense masks at the camera’s rate they are heavy.
  • Post-hoc smoothing of per-frame outputs, for example a moving average of probabilities or warping the previous mask with optical flow, is cheap and often enough to remove flicker.
frame t − 1encoder1ConvLSTM2decoder4mask t − 1h, cframe tencoderConvLSTMdecodermask th, cframe t + 1encoderConvLSTMdecodermask t + 13
A recurrent U-Net unrolled over three frames; the same weights are used at every step.
  1. Encoder per frame. Every frame passes through the same encoder.
  2. ConvLSTM cell. Its input, forget and output gates are convolutions, so its state is a spatial feature map rather than a vector.
  3. State. The hidden state h and the cell state c carry from frame to frame what the cell chose to keep, such as where an instrument was before smoke covered it.
  4. Decoder. Draws the current frame’s mask from the cell’s output.

Promptable segmentation with memory: SAM 2

SAM 2 (Ravi et al., 2024) segments an object in every frame of a video once it is given a prompt: a click, a box or a mask on one frame. A memory bank stores features and predicted masks of recent frames and of the prompted ones; memory attention conditions the current frame on them, which carries the object through occlusions and changes of view. Its authors report 39.5 fps for SAM 2.1 large and 91.2 fps for tiny on an A100 with the model compiled, for one object.

frame timage encoderHiera1memoryattention2maskdecoder4mask tprompt encoderclick · box · mask3memoryencoder5memory bankrecent frames,prompted frames6
The SAM 2 video pipeline for one object.
  1. Image encoder. A hierarchical transformer, Hiera, embeds each frame once.
  2. Memory attention. Transformer blocks in which the current frame’s features attend to the memory bank.
  3. Prompt encoder. Turns clicks, boxes or a mask on any frame into tokens for the decoder.
  4. Mask decoder. Produces the mask, a score for its quality and whether the object is visible in this frame at all.
  5. Memory encoder. Combines the predicted mask with the frame’s features into a compact memory.
  6. Memory bank. Keeps the memories of the most recent frames, six by default, and of the prompted frames, plus object pointers; the next frame attends to them.

In surgery SAM 2 is a tracker, not a detector: something has to give the first prompt and catch new instruments as they enter, such as a per-frame model or a person in an annotation tool. It was not trained on surgical video, so test it on yours before relying on it. In annotation work it pays off at once: one corrected mask propagates to the following frames.

Latency budget in the operating room

What matters is glass-to-glass latency: the time from light reaching the camera to the overlay appearing on the monitor. It is the sum of every stage. Throughput is another matter: if the stages run in parallel on consecutive frames, the pipeline keeps up with the camera as long as its slowest stage is shorter than one frame interval, even when the sum is longer. A latency target is a project decision, written for example as “p95 under 50 ms, measured glass to glass”; it is not a standard, and the right value depends on what the overlay is for.

The default values below are an example, not measurements. Move the sliders to see which limit you hit.

Worked example. At 30 fps a frame arrives every 33.3 ms. Stages of 8, 3, 15, 2 and 10 ms add up to 38 ms: processed one frame at a time, the pipeline manages 26 fps and drops frames. Run in parallel, its slowest stage, inference at 15 ms, allows about 67 fps, so it keeps up, but every frame still reaches the screen 38 ms after it was captured. The red line on the bar marks one frame interval.

How to measure: time each stage with CUDA events rather than CPU clocks, report p50 and p95 over minutes of video rather than a mean over a few frames, and measure glass to glass with a camera filming a millisecond timer or an LED and the monitor in the same shot. The project’s model cards ask for p50 and p95 on an RTX 4090 and a Jetson AGX Orin; none has been measured yet.

Measuring quality

The formulas behind these metrics, with pictures and an interactive example, are on the metrics page.

  • IoU and Dice per class and their mean over classes; for instruments also the binary mask (any instrument against background).
  • Boundary F1 or boundary IoU, because thin instrument shafts and jaws are mostly boundary.
  • Temporal stability: how much a mask changes between neighbouring frames when the scene barely does.
  • Robustness: scores on frames with smoke, blood or motion blur reported separately. The project’s M5 desmoking experiment and synthetic smoke in osv.augment are a place to start.

References

Back to the comparison of volumes and video.