LECTURE 10

Lecture 10. Streaming inference under a hard latency budget

How to put masks on live laparoscopic video with p95 latency of at most 50 ms at 30 frames per second: the tail of the distribution and queues, the six stages of the pipeline and how to profile them, transfer to the GPU without extra copies, and the asynchronous osv.serve server on FastAPI. With diagrams, formulas, measurements, interactive examples and 3D animations.

This page is still being reviewed by our scientific, clinical and legal experts. It describes plans, not finished work.

A model that segments a frame well in 200 ms is useless for an overlay during surgery: the surgeon moves the instrument, and the mask arrives where the instrument was a fifth of a second ago. This is lecture 10 of the curriculum, after lecture 9 on instrument tracking. Three topics:

  • The latency budget: what “p95 ≤ 50 ms at ≥ 30 frames per second” means and why the tail of the distribution matters, not the mean.
  • Profiling: the six stages from frame capture to the monitor, where the time goes and how to measure it.
  • The osv.serve streaming server: transfer without extra copies and an asynchronous server on FastAPI.

osv.serve has not been written yet: osv only has its interface and description. So this lecture describes the planned design. Every measurement on this page was taken on a laptop with an Intel Core Ultra 7 155H and no NVIDIA GPU; numbers for a GPU are calculations, and each time that is said. The project’s models are for research (RESEARCH USE ONLY – NOT FOR CLINICAL USE).

The budget: two conditions, not one

The budget is written in osv.deploy:

from osv.deploy import get_latency_budget

get_latency_budget()
# {'target_latency_p95_ms': 50.0, 'target_fps': 30.0,
#  'supported_engines': ['PyTorch', 'ONNXRuntime', 'TensorRT']}

It holds two different conditions.

  • Throughput: ≥ 30 frames per second. A new frame arrives every 33.3 ms. If the pipeline does not finish it in time, frames pile up or are lost. Stages can work in parallel on consecutive frames, and then the slowest stage has to keep up with the camera, not their sum. This is covered on the page about the latency budget for video.
  • Latency: p95 ≤ 50 ms. A frame’s latency is the time from its capture to the overlay appearing on the monitor, “glass to glass”. p95 is the 95th percentile: 95 % of frames must appear no later than 50 ms after capture.
p95=min{t:|{k:Lk≤t}|≥0.95·N} N=1800:N−0.95·N=90frames

Why a percentile and not the mean. At 30 frames per second a minute holds 1800 frames, and 5 % of them is 90 frames a minute that are allowed to be late. The mean does not see that: latency is not distributed symmetrically, it has a long right tail. Garbage collection, another process on the same GPU, the OS scheduler and thermal throttling all add to it. Even plain downscaling of a 1920×1080 frame to 854×480 with OpenCV took 1.41 ms at the median and 2.82 ms at p95 on our laptop – twice as long. When a stage runs many times per frame, or a frame passes through many stages, rare delays in the parts add up to frequent delays of the whole. Dean and Barroso cover this in The Tail at Scale.

The queue: where latency grows by itself

Frames arrive at equal intervals T = 33.3 ms, and processing takes a varying time S. A frame that arrives while the model is busy with the previous one waits. The wait of the next frame follows Lindley’s recursion:

Wk+1=max(0,Wk+Sk−T) Lk=Wk+Sk+C+Vk

A frame’s latency adds up the wait, the processing, capture and display C, and the wait for the screen refresh V.

While S is on average shorter than T, the wait returns to zero after every spike. But a 100 ms spike leaves three frames queued behind it, and they take several frames to drain. If S is on average longer than T, a FIFO queue grows without limit: every frame waits longer than the one before.

A streaming overlay does not need every frame, it needs the newest. So instead of a queue it keeps one slot: a new frame replaces the one the model has not yet taken. Latency stops building up, and the price is dropped frames.

The example’s model: processing time follows a log-normal distribution with the given median and spread σ, and in rare frames a 100 ms pause is added to it. Latency also includes capture and display and the wait for the next refresh of a 60 Hz screen – anywhere from 0 to 16.7 ms, because the camera’s and the monitor’s clocks are not synchronised. This is a model, not a measurement; it is there to show the shape of the distribution.

At the defaults (median 15 ms, a pause in 0.5 % of frames) p95 is 44.8 ms, inside the budget. But p99 is 94 ms: each pause drags several queued frames behind it, and 39 frames a minute take longer than 50 ms. With a slot for the newest frame p95 is almost the same, 44.1 ms, p99 drops to 48.5 ms, 14 frames are late and 14 frames are dropped.

Without pauses and with a median of 20 ms the mean is 40.4 ms and looks comfortable, while p95 is 49.5 ms, right at the limit. With a median of 35 ms, again without pauses, the model is slower than the camera: with a FIFO queue latency grows to almost 4 seconds within a minute, and with the slot p95 stays at 90 ms but 110 frames are dropped – the 30 frames per second condition is already broken.

The six stages of the pipeline

To stay inside the budget, it has to be split between the stages. Glass-to-glass latency is the sum of six stages:

capturecapture cardthe whole frame1colourNV12 → RGB,size, float2to GPUcopy overPCIe3networkforward passTensorRT, fp164maskargmax, colours,encoding5displayoverlay andscreen refresh6hardwarehardwareour codeglass-to-glass latency = sum of the six stages; p95 ≤ 50 ms
A frame's path from the camera to the monitor. The hardware sets the first and the last stage; the four in between are our code.
  1. Frame capture. The camera sends video over SDI or HDMI, and the capture card receives it line by line. The frame is ready for processing once it has been received in full, and with a 60 Hz input sending one frame takes up to 16.7 ms. The driver then puts the frame into a buffer in memory.
  2. Colour conversion. Cameras and capture cards deliver YUV – for instance NV12, with brightness at full resolution and colour at half along each axis. The network needs RGB of a given size in floating point, normalised the same way as in training.
  3. Transfer to the GPU. The frame is copied over the PCIe bus from the processor’s memory to the graphics card’s memory.
  4. The forward pass. Usually the longest stage and the main candidate for speeding up: TensorRT or ONNX Runtime, half precision, INT8. These are exactly the engines osv.deploy lists.
  5. Mask post-processing. Argmax over classes, smoothing over time from lecture 9, scaling back to the screen size, class colours, encoding for sending.
  6. Display. The overlay is put on the video, the frame waits for the next screen refresh – up to 16.7 ms at 60 Hz – and the monitor adds its own delay.

Measurements on a processor

To have real numbers, we measured the middle stages on synthetic frames on a laptop with a Core Ultra 7 155H: OpenCV 5.0, PyTorch 2.14 on 8 threads, 300–600 repeats after a warm-up. The network is LR-ASPP with MobileNetV3-Large from torchvision with 13 CholecSeg8k classes and random weights: weights do not matter for timing.

Stage Operation p50, ms p95, ms
Colour NV12 → RGB, 1920×1080 0.18 0.21
Colour downscale to 854×480, INTER_AREA 1.41 2.82
Colour to float, CHW, normalisation, 854×480 0.53 0.58
Copy copy of an RGB frame 1920×1080, 6.2 MB 0.23 0.23
Copy torch.from_numpy: a tensor over the same bytes 0.001 0.001
Network LR-ASPP MobileNetV3, 854×480, on the processor 60.0 65.0
Mask argmax over 13 channels, NCHW tensor 43.9 45.6
Mask argmax over 13 channels, channels last 3.0 3.6
Mask class colours by NumPy indexing, 1920×1080 19.5 20.1
Mask colours with cv2.LUT and blending with the frame, 1920×1080 2.5 3.3
Sending PNG of an 854×480 mask, 3709 bytes 0.64 0.66
Sending JPEG of the 1920×1080 frame with overlay, quality 80 10.2 10.4

What follows from this:

  • The network on a processor eats the budget by itself. 60 ms per frame is fewer than 17 frames per second even without the other stages. A GPU is needed, and the project has not yet measured its time on one: model cards must report p95 latency, but both existing cards, M1 and M5, record that p50 and p95 on an RTX 4090 or Jetson AGX Orin have not been measured.
  • Post-processing can cost more than the network. A naive argmax and colouring of the mask took 63 ms together – more than the forward pass.
  • Memory layout decides. The same argmax over the same numbers is 15 times faster when channels come last and neighbouring values sit next to each other in memory. Colouring with cv2.LUT together with blending into the frame is 8 times faster than colouring alone by array indexing.
  • Even simple operations have a tail. Downscaling a frame takes 1.41 ms at the median and 2.82 ms at p95.

How to measure on a GPU

CUDA operations are asynchronous: calling model(x) queues kernels and returns at once. The processor’s clock around such a call measures the queueing, not the computation. Calling torch.cuda.synchronize() after every stage makes the times right, but the stages stop overlapping and the pipeline slows down. So time on a GPU is measured with CUDA events, recorded into the same command stream:

import torch

start = torch.cuda.Event(enable_timing=True)
end = torch.cuda.Event(enable_timing=True)

start.record()
logits = model(x)
end.record()
end.synchronize()                 # wait until the GPU reaches end
forward_ms = start.elapsed_time(end)

Three more rules. Drop the first few dozen frames: they include initialisation, cuDNN’s choice of algorithms and filling caches. Compute p50 and p95 over minutes of video, not over ten frames. Measure glass-to-glass latency from outside, with a camera that films a millisecond timer in front of the laparoscope and the monitor with the overlay in the same frame.

Stages working in parallel rely on the GPU having its own command queues – CUDA streams – so that copying can run at the same time as computation. While the network computes frame n, frame n + 1 is already being copied to the GPU, and the mask of frame n − 1 is coming back.

The numbers here are made up for illustration; they are not a measurement. One after another, the next frame starts only once the previous one has passed all five stages: at the defaults that is 35 ms, more than 33.3, and every frame is later than the one before – the sixth already waits 8.3 ms on top of its 35. With overlapping, each stage works on its own frame and the slowest one sets the pace – the network, 26 ms, or 38 frames per second. But one frame’s latency is still the sum of the stages: overlapping helps throughput, not latency. The example assumes preparation and post-processing have their own processor threads, and copying in each direction has its own queue.

Without extra copies

Every copy of a frame costs processor time and memory bandwidth, and a copy over PCIe also occupies the bus. The transfer time is estimated like this:

t≈BR,R≈25GB/s 854·480·3=1.23MB→t≈0.05ms 13·854·480·4=21.3MB→t≈0.85ms

Here B is the bytes and R the bus bandwidth. 25 GB/s is about what PCIe 4.0 x16 gives when copying from pinned memory; the theoretical limit is about 32 GB/s in each direction. The PCIe transfer itself is small. What matters more is how many copies the code around it makes and what exactly is sent. The naive path and the path without extra copies:

naive: 26.2 MB over PCIe, 58.7 ms of CPU per frameNV12 framein pageable memoryCPU: preparecolour, size, float2.1 msdriver copyinto a temporarypinned bufferPCIe: 4.92 MBnetwork on GPU21.3 MB backCPU: maskargmax, overlay,JPEG: 56.6 msto the network176 KB per frame42 Mbit/swithout extra copies: 3.52 MB over PCIe, 0.6 ms of CPU per frameNV12 framein a pinned bufferfrom the cardPCIe: 3.11 MBstraight fromthe bufferGPU: preparecolour, size, floatnetwork on GPUargmax on GPUPCIe: 0.41 MBuint8 maskPNG of the mask3.7 KB, 0.6 ms CPU0.9 Mbit/s
The naive path sends 7 times more bytes over the bus and 48 times more over the network, and spends almost 60 ms of processor time per frame.
  • Pinned memory. Ordinary processor memory is pageable: the OS can swap a page out. CUDA cannot copy from such memory directly and first copies the data into a temporary pinned (page-locked) buffer itself – the PyTorch guide explains this in detail. If the frame is put straight into a pinned buffer, the extra copy is gone, and tensor.to("cuda", non_blocking=True) returns at once while the copy runs in parallel with computation.
  • Send uint8 and convert on the GPU. An 854×480 frame is 1.23 MB in uint8 and 4.92 MB in float32. The processor need not touch the frame at all: send the original NV12 1920×1080, which is 3.11 MB, and do colour, size and normalisation on the GPU.
  • Return the mask, not the logits. Logits for 13 classes in float32 are 21.3 MB per frame; a uint8 mask is 0.41 MB. Argmax on the GPU, and only the mask comes back.
  • Do not route the video through the server. The overlay can be applied where the video is displayed: in a browser or in an application at the monitor. Then only the mask goes over the network: 3.7 KB as PNG against 176 KB for the frame as JPEG on our synthetic frame (on a real one the size depends on the content).
  • Do not copy inside the process without need. torch.from_numpy gives a tensor over the same bytes in a microsecond, and .copy() of a 1920×1080 frame takes 0.23 ms. A trifle, until there are ten such copies per frame.

The limit of this idea is a frame that never passes through the processor’s memory at all: the capture card writes it straight into GPU memory. Pipelines built on NVIDIA Holoscan SDK, which are also built for endoscopy, work this way.

Processor time here is the sum of the p50 measurements from the table above, the bytes are exact, and the PCIe time is the formula’s estimate at 25 GB/s. The naive path is selected by default: 26.2 MB over the bus, 58.7 ms of processor time and 42 Mbit/s over the network. The path without extra copies – NV12 to the GPU, the mask back, the overlay at the client – sends 3.52 MB, spends 0.6 ms on the processor and sends 0.9 Mbit/s over the network. Bus time is around a millisecond or less in both cases – the gain comes from what the code no longer does.

The asynchronous osv.serve server

The osv.serve documentation describes it as a FastAPI server streaming over WebSocket or WebRTC for overlays on surgical video. There is no code yet, so what follows is the planned design and a tested sketch.

clientbrowser or appat the monitorasyncio event loopreceive()ws.receive_bytes()slot for 1 frameQueue(maxsize=1)a new frame replaces an old oneinfer_and_send()newest frame → maskthread: segment()the model on the GPUasyncio.to_threadframesmask + frame numberonly receiving and sending on the event loop; everything long in a thread or a process
Two tasks on the event loop and the model in a separate thread. The one-frame slot keeps latency from building up.

The server is asynchronous: one asyncio event loop serves the connections as long as no task keeps it busy. Hence the main rule: nothing long inside async def. A 20 ms forward pass called directly in the handler stops receiving and sending on every connection for those 20 ms. The model runs in a separate thread through asyncio.to_thread or in a separate process, and the event loop only receives and sends.

import asyncio
from fastapi import FastAPI, WebSocket, WebSocketDisconnect

app = FastAPI()

@app.websocket("/stream")
async def stream(ws: WebSocket):
    await ws.accept()
    latest: asyncio.Queue[bytes] = asyncio.Queue(maxsize=1)

    async def receive():
        while True:
            frame = await ws.receive_bytes()
            if latest.full():
                latest.get_nowait()       # drop the frame the model did not get to
            latest.put_nowait(frame)

    async def infer_and_send():
        while True:
            frame = await latest.get()
            mask = await asyncio.to_thread(segment, frame)  # does not block the event loop
            await ws.send_bytes(mask)

    try:
        async with asyncio.TaskGroup() as tasks:
            tasks.create_task(receive())
            tasks.create_task(infer_and_send())
    except* WebSocketDisconnect:
        pass

We ran this sketch with FastAPI 0.146 and uvicorn 0.54, with a 40 ms stand-in for segment – a model slower than the camera. The client sent 150 frames at 30 per second. Replies came for 125 frames, latency was 58 ms at p50 and 73 ms at p95, and it did not grow. With an ordinary unbounded queue replies came for all 150 frames, but within 5 seconds latency grew to 1.1 s, with a p50 of 570 ms.

What a real server needs as well:

  • The frame’s number and time in the reply. A mask should carry the frame’s number and capture time: the client puts it on the right frame, or at least knows how stale it is.
  • One owner of the GPU. Several connections should not run the model in parallel and get in each other’s way: one task owns the model, and the others receive its results.
  • Backpressure on the way out. A slow client must not build up a send queue; the “newest only” rule applies on the way out as well.
  • Capture on the server. In the operating room frames do not come over the network from a browser but from a capture card on the same machine; the receiving task in the sketch is replaced by reading the card, and only masks go out over the network.

WebSocket, gRPC or WebRTC

Property WebSocket gRPC WebRTC
Transport one TCP connection after an HTTP handshake (RFC 6455) HTTP/2 over TCP mostly UDP
Messages arbitrary bytes or text Protocol Buffers by a schema media streams and data channels
In a browser built in only through gRPC-Web: no client-side or bidirectional streaming built in
Good at a simple two-way stream, available in FastAPI a strict schema and generated clients, services talking to each other low-latency video; a lost packet does not hold up the next ones
Limitation a lost TCP packet holds up all the following ones the same for TCP harder to set up: signalling, NAT

For a mask going to a monitor on the same network WebSocket is the simplest choice, and FastAPI supports it out of the box. gRPC is convenient when the client is not a browser but another service, for instance a robotics stack. WebRTC is needed if the video itself goes over the network.

Message size matters more than the protocol. We measured round-trip time through FastAPI, uvicorn and websockets 15 on the same laptop, without a real network, with a mask-sized reply:

Message from the client p50, ms p95, ms
3.7 KB (size of a PNG mask) 0.12 0.23
176 KB (JPEG of a 1920×1080 frame) 16.4 16.8
6.2 MB (uncompressed RGB frame) 503 512

This is a property of this particular Python stack, not a limit of WebSocket: other servers will move the same bytes faster. The conclusion stays the same: masks over the network – yes; uncompressed frames through a Python server – no; and any choice has to be measured on your own stack.

Where the project is now

osv.serve and osv.deploy hold only an interface so far: get_server_metadata() returns a description of the server with the mandatory non-clinical notice, and get_latency_budget() returns the budget from this lecture. There is no export to ONNX or TensorRT yet, no server and no latency measurement on a GPU. The model cards for M1 (instrument segmentation) and M5 (desmoking) both state that inference latency on an RTX 4090 or Jetson AGX Orin has not been measured. The first task here is to measure, not to speed up: p50 and p95 of each of the six stages on the target GPU.

Checklist

  • Report p50, p95 and p99 over minutes of video, not a mean over a few frames; drop the first frames after start-up.
  • Measure GPU time with CUDA events and glass-to-glass latency with an external camera.
  • Check throughput (the slowest stage shorter than 33.3 ms) and latency (the sum of the stages, p95 ≤ 50 ms) separately.
  • Keep one slot for the newest frame, not a queue; count dropped frames.
  • Nothing long in async def: the model goes in a separate thread or process.
  • Put frames in pinned memory, send uint8, convert on the GPU, return the mask, not the logits.
  • Watch memory layout: argmax over channels and colouring the mask can cost more than the network.
  • Send the mask with the frame’s number and time; do not route video through a Python server.

Reading list