LECTURE 1

Lecture 1. Surgical and diagnostic data formats

What PNG and JPEG frames, laparoscopic video, DICOM and NIfTI store and how they are built: codecs and YUV colour, the DICOM hierarchy, voxel grids, the affine matrix, RAS and LPS, and Hounsfield units. With formulas, diagrams, interactive examples and 3D animations.

This page is still being reviewed by our scientific, clinical and legal experts. It describes plans, not finished work.

Three kinds of image data come out of an operating room and the radiology department around it: two-dimensional frames (PNG, JPEG), laparoscopic video (MP4 with H.264 or HEVC) and three-dimensional scans made before the operation (DICOM, NIfTI). They differ in what they store, how they compress it and how a pixel or voxel maps to a place in the patient. Get one of these wrong and a model learns from mirrored, discoloured or mis-scaled data without any error message. This is the first lecture of the learning programme, module 1.

All images on this page are synthetic: the frames come from the project’s Blender animation, the CT slices from a phantom drawn by code. No patient data is used.

The three kinds at a glance

Format Dimensions Typical source What one file holds Read it with
PNG, JPEG 2D, colour frames extracted from video, screenshots, masks one image, 8 bits per channel Pillow, OpenCV
MP4 (H.264, HEVC) 2D + time laparoscope tower, recorder a compressed stream of frames, often with sound PyAV, OpenCV, ffmpeg
DICOM 2D slice, series make 3D CT, MRI, ultrasound, X-ray one image and its metadata: patient, study, geometry, scanner pydicom
NIfTI 3D or 4D grid conversion from DICOM, research pipelines a whole volume and its geometry nibabel, SimpleITK

2D frames: PNG and JPEG

  • PNG is lossless: every pixel comes back exactly as saved. It stores 8 or 16 bits per channel, with or without transparency. Use it for masks: a mask’s pixel values are class numbers, and they must survive saving untouched.
  • JPEG is lossy. It converts the image to luma and colour difference, usually halves the colour resolution, cuts the image into 8 × 8 blocks, and throws away fine detail the eye notices least. Blocks and ringing appear at sharp edges, such as instrument borders. Never save a mask as JPEG: the class number 3 next to 0 becomes 1, 2 and 3 along the edge.
  • sRGB. The 0–255 values of a normal PNG or JPEG are not proportional to light. They are encoded with the sRGB curve, which spends more codes on dark tones; doubling a value does not double the brightness. Most models never notice, but blending, blurring or simulating light physically should happen on linear values.

Endoscopic video

Container and codec

An .mp4 file is a container: it holds one or more streams, timestamps and metadata. The pictures inside are compressed by a codec, usually H.264 (AVC) or H.265 (HEVC). HEVC needs about half the bitrate for the same quality but more computation to decode. Check what a file really contains before you trust it: ffprobe lists the codec, frame rate, pix_fmt, color_range and color_space of each stream.

Inter-frame compression

IBBPBBPBBPBBIone group of pictures (GOP)↑ bytes per frametime →
A typical group of pictures; the arrows show which frames a frame is predicted from. Bar heights are illustrative.

Neighbouring frames of a video are nearly identical, so the codec stores most frames as differences.

  • I frames (intra) are compressed on their own, like a JPEG. Decoding can start only at an I frame, so seeking jumps to the nearest one before the target.
  • P frames (predicted) store motion vectors and the remaining difference from an earlier frame.
  • B frames (bi-directional) refer to frames before and after them. They are the smallest, and they are why the order frames are stored in differs from the order they are shown in.
  • Variable bitrate. The encoder spends bits where the picture is hard to predict: fast camera motion, smoke, a flash, fluid. A calm minute of dissection may take a tenth of the bytes of a minute with heavy smoke. File size says little about duration or quality.

Frame rate and timestamps

Laparoscopic video runs at 25 to 60 frames per second: 25 or 50 in former PAL countries, 30 or 60 elsewhere; Cholec80 is 25. Some recorders write variable frame rate, where the gap between frames changes. A frame’s number then is not its time: use each frame’s presentation timestamp (pts) to align annotations, phases and sound.

A raw frame of 854 × 480 pixels in YUV 4:2:0 takes 1.5 bytes per pixel. Uncompressed, one second at 25 fps would be:

bytes per second=w×h×1.5×f 854×480×1.5×25≈15.4MB/s

That is about 120 megabits per second. A typical H.264 recording of that size uses one to a few megabits per second: the codec compresses it some 30 to 100 times.

YUV 4:2:0 against sRGB

RGB16 px × 3 = 48 bytesY (luma)16 bytesCb4 bytesCr4 bytes++4:2:0 = 24 bytes, 1.5 per pixel
Y keeps every pixel; Cb and Cr are averaged over 2 × 2 pixels and drawn here around mid-grey.

Video codecs do not store red, green and blue. They store luma Y, the brightness, and two colour differences, Cb (blue minus luma) and Cr (red minus luma). The eye resolves brightness much better than colour, so yuv420p, the usual pixel format, keeps Cb and Cr at half the width and half the height: 4:2:0. Converting to RGB for a model uses the standard’s matrix, here BT.709, the HD standard, with all values on a 0–1 scale:

Y=0.2126R+0.7152G+0.0722B R=Y+1.5748·Cr G=Y−0.1873·Cb−0.4681·Cr B=Y+1.8556·Cb

Two details cause most colour bugs:

  • Matrix. Standard-definition video uses BT.601 with different coefficients. Decoding HD video with the wrong matrix tints tissue slightly green or magenta.
  • Range. In “limited” or TV range, luma runs from 16 to 235 and colour from 16 to 240, not 0–255. Decoded as full range, black turns grey and the image looks washed out; the reverse crushes shadows.

Flashes from the coagulator

Electrosurgery makes sparks and bright smoke, and wet tissue reflects the light source directly. Three things happen in the video. Pixels hit the top of the range and clip: detail there is gone, and because the red channel of tissue clips first, the colour of a clipped area shifts towards yellow and white. The camera’s automatic exposure and white balance react over the following frames, so the whole picture darkens and changes tint for a moment. And the encoder suddenly needs many more bits, so quality drops in exactly those frames. Models trained on calm frames fail on them; the project’s desmoking experiment works on a related problem.

Try it on a synthetic frame: look at the planes one by one, see what 4:2:0 loses, and fire the coagulator.

DICOM

DICOM (Digital Imaging and Communications in Medicine) is both a file format and a network protocol; every CT, MRI and X-ray machine speaks it. A DICOM file holds one image, a CT slice for example, together with everything known about it.

Information objects

The standard describes each kind of image as an Information Object Definition (IOD): CT Image, MR Image, Ultrasound Image, Segmentation and so on. An IOD is a list of modules, groups of attributes that belong together: Patient, General Study, General Series, Image Plane (position, orientation, pixel spacing), Image Pixel (rows, columns, bits, the pixels themselves). Which modules are mandatory depends on the IOD.

Patient, study, series, instance

PatientPatientID (0010,0020)1Study: CT abdomenStudyInstanceUID (0020,000D)2Study: MRI liverStudyInstanceUIDSeries: axial 2.5 mm(0020,000E)3Series: axial 0.6 mm(0020,000E)Series: T2(0020,000E)Series: T1 + contrast(0020,000E)4instance = one image, one file · SOPInstanceUID (0008,0018)
One patient, two studies, four series; each small rectangle is one file.
  1. Patient: the person, identified by PatientID.
  2. Study: one examination, for example an abdominal CT on one day, with a globally unique StudyInstanceUID.
  3. Series: one acquisition or reconstruction within the study, such as axial slices of 2.5 mm, the same scan rebuilt at 0.6 mm, or one MRI sequence; SeriesInstanceUID. A model needs exactly one series: mixing two gives slices of different thickness or contrast in one volume.
  4. Instance: one image, usually one file; SOPInstanceUID. A CT series of 300 slices is 300 files.

Inside a file

128-byte preambleDICMfile meta (0002,xxxx)data set: data elements in tag ordertagnameVRlengthvalue(0010,0020)PatientIDLO10ANON-0001(0020,0032)ImagePositionPatientDS16-175\-175\100(0028,0030)PixelSpacingDS80.7\0.7(0028,1052)RescaleInterceptDS6-1024(0028,1053)RescaleSlopeDS21(7FE0,0010)PixelDataOW524288stored values
Values are from a synthetic slice. Backslashes separate the parts of a multi-valued element.

After a 128-byte preamble and the letters DICM comes a list of data elements. Each has a tag, two hexadecimal numbers (group, element); a value representation (VR) that says what kind of value it is, such as DS for a decimal string, UI for a unique identifier or OW for 16-bit words; a length; and the value. Part 6 of the standard lists every tag, part 5 the value representations, part 10 the file format.

Tag Name Why you need it
(0020,000E) SeriesInstanceUID group files into one volume
(0020,0032) ImagePositionPatient position of the slice’s first pixel, mm
(0020,0037) ImageOrientationPatient directions of the slice’s rows and columns
(0028,0030) PixelSpacing spacing between rows, then between columns, mm
(0028,1052), (0028,1053) RescaleIntercept, RescaleSlope stored values to Hounsfield units
(7FE0,0010) PixelData the stored values themselves

Reading metadata with pydicom

import pydicom
from pydicom.pixels import apply_modality_lut

ds = pydicom.dcmread("slice_0001.dcm", stop_before_pixels=True)  # headers only: fast
print(ds.Modality, ds.SeriesInstanceUID)
print(ds.ImagePositionPatient, ds.ImageOrientationPatient, ds.PixelSpacing)
print(ds.RescaleSlope, ds.RescaleIntercept)

ds = pydicom.dcmread("slice_0001.dcm")         # with pixels
hu = apply_modality_lut(ds.pixel_array, ds)     # stored value × slope + intercept

The code is for pydicom 3; in pydicom 2 the function lives in pydicom.pixel_data_handlers.util. Two rules when you build a volume from a series:

  • Sort slices by position, not by file name or InstanceNumber. Project each slice’s ImagePositionPatient onto the slice normal, the cross product of the two directions in ImageOrientationPatient, and sort by that number.
  • Take the slice spacing from the positions. SliceThickness is how thick the reconstructed slice is, not how far apart slices are; overlapping reconstructions have a smaller spacing than thickness.

Headers carry names, birth dates, hospital names and dates of examinations. Nothing leaves a clinical environment before de-identification, the subject of lecture 2; the project checks the result with its osv.deid gate.

NIfTI: one file, one grid

NIfTI (.nii, or .nii.gz compressed) stores a whole volume as one continuous grid of voxels with a fixed step. A NIfTI-1 file is a 348-byte header followed by the voxel values.

  • Dimensions: up to seven, in practice 3D (a CT) or 4D (an MRI time series, or several values per voxel).
  • pixdim: the voxel size along each axis, in millimetres, and for 4D the time between volumes.
  • qform and sform: two ways to store the voxel-to-world mapping, a rotation as a quaternion plus scaling, or a full affine matrix. Readers such as nibabel prefer the sform when it is set.
  • scl_slope and scl_inter: like DICOM’s rescale, applied to stored values on reading.

Converting DICOM to NIfTI, usually with dcm2niix, means doing everything from the previous section once: pick a series, sort slices, check the spacing, compute the geometry, write one file. After that, the volume is one array.

import nibabel as nib

img = nib.load("ct.nii.gz")
print(img.shape, img.header.get_zooms())   # (512, 512, 120) and voxel sizes in mm
print(img.affine)                          # 4 × 4, see below
print(nib.aff2axcodes(img.affine))         # e.g. ('L', 'P', 'S')
volume = img.get_fdata()                   # float array, scl_slope and scl_inter applied

Volume geometry: the affine matrix

A voxel has an index, three whole numbers (i, j, k) that say where it sits in the array. The patient has millimetres. The affine matrix connects them: a 4 × 4 matrix that turns the index into a position in the scanner’s patient coordinate system.

0.700-17500.70-175002.51000001×ijk1=xyz1voxel indexposition, mm1rotation × spacing (3 × 3)2translation: where voxel (0, 0, 0) lies, mm3always 0 0 0 1
A volume with 0.7 mm pixels and 2.5 mm slices whose first voxel lies at (−175, −175, 100) mm, without rotation.
[xyz1]=A[ijk1],A=[R·Dt0001] D=diag(Δi,Δj,Δk)
  1. R · D, the 3 × 3 block: R is a rotation, the directions the voxel axes point in; D holds the voxel sizes. Each column of the block is the step, in millimetres, of one voxel along one index axis.
  2. t, the translation: the position of voxel (0, 0, 0).
  3. The last row is always 0 0 0 1; it lets one matrix product both rotate and shift.

In the example, in DICOM’s LPS coordinates, voxel (100, 120, 40) lies at (−175 + 100 × 0.7, −175 + 120 × 0.7, 100 + 40 × 2.5) = (−105, −91, 200) mm. From a DICOM series the same matrix follows from the headers, with S the ImagePositionPatient of the first slice, X and Y the two directions of ImageOrientationPatient, Δr and Δc the two values of PixelSpacing and Δs the spacing between slices:

P(i,j,k)=S+iΔcX+jΔrY+kΔsn n=X×Y

Move a voxel through the grid below and watch the matrix carry it from index space to millimetres. The grid is small and its voxels are large so that they can be seen; it is also tilted, to show what the rotation block does.

Orientation: RAS and LPS

anteriorposteriorpatient's rightpatient's leftxyRAS+ · NIfTI, nibabel, 3D Sliceranteriorposteriorpatient's rightpatient's leftxyLPS+ · DICOM, ITK, SimpleITKz: towards the head
An axial slice as radiologists view it, from the feet: the patient's right is on the left of the image.

Millimetres need axes, and two conventions are in use. Both name the direction each axis points to:

  • RAS+: the first axis grows towards the patient’s Right, the second towards Anterior (the front), the third towards Superior (the head). NIfTI’s world coordinates, nibabel, FreeSurfer and 3D Slicer use it.
  • LPS+: towards the Left, Posterior (the back) and Superior. DICOM’s patient coordinate system, ITK and SimpleITK use it.

The two differ by the sign of the first two axes:

(x,y,z)RAS=(−x,−y,z)LPS ARAS=diag(−1,−1,1,1)·ALPS

A second, separate question is which way the voxel axes point. nib.aff2axcodes answers it: for a volume kept in DICOM’s voxel order it returns ('L', 'P', 'S'), meaning that increasing i moves to the left, j to the back and k to the head, while the world coordinates are still RAS. Mixing the conventions mirrors left and right without any error, so a liver segmentation ends up on the spleen’s side. Check orientation codes whenever you combine files from different tools.

Hounsfield units

A CT scanner measures how strongly each voxel attenuates X-rays, μ, and reports it on the Hounsfield scale, fixed by water at 0 and air at −1000:

HU=SV·slope+intercept HU=1000·μ−μwaterμwater−μair

DICOM does not store HU directly. It stores whole numbers, the stored values SV, usually 12 to 16 bits without a sign, and the header says how to convert them: HU = SV × RescaleSlope + RescaleIntercept. A slope of 1 and an intercept of −1024 are the most common pair, so a stored 1024 is water, but read both from every file, because they may differ between scanners and even between slices of one series. MRI intensities have no such scale; they must be normalised per scan.

-1000-500050010001500airlungfatwatersoft tissuebonelung window W 1500 / L −600soft tissue W 400 / L 40bone W 1800 / L 400dense bone and metal: up to 3000 and moreHU
Approximate values; the exact numbers depend on the tissue, the scanner and contrast agent. Window settings differ between departments.

A screen shows about 256 shades of grey, and the eye tells apart fewer; the HU range is several thousand. A window, a width W and a level L in HU, picks the range that matters and spreads it over black to white; everything below is black, everything above white:

g=min(1,max(0,HU−(L−W2)W))

Models do the same: a CT model is trained on HU clipped to a range and scaled, and inference must clip and scale identically. Try the windows on a synthetic slice; move the pointer over it to read stored values and HU.

Checklist

  • Masks in PNG, never JPEG; frames as PNG or straight from the video, not re-encoded.
  • For video: codec, pix_fmt, color_range, color_space and frame rate checked with ffprobe; frames aligned by timestamp.
  • For DICOM: one series per volume, slices sorted by position, spacing from positions, rescale applied, headers de-identified.
  • For NIfTI: affine and aff2axcodes inspected; never mix RAS and LPS without converting.
  • For CT: HU clipped and scaled the same way in training and inference.

Further reading