A model that has seen frames of an operation will recognise the rest of that operation, much as you would recognise the rest of a film from a few stills. That is memory, not skill. Tested on frames from the same patients it trained on, a model scores high and then fails on the next patient. This lecture covers the public datasets the project works with, how data leaks from training into test, and how osv.datasets splits by patient so that it cannot. It is lecture 5 of the learning programme, after lecture 4 on synthesis.
Every image on this page is synthetic or drawn by code; no frame from the datasets below is shown.
Five datasets
| Dataset | What it holds | Labels | Split by | Licence |
|---|---|---|---|---|
| Cholec80 | 80 cholecystectomy videos by 13 surgeons, Strasbourg, 25 frames per second | 7 phases for every frame; presence of 7 instruments once a second | operation | CC BY-NC-SA 4.0 |
| CholecSeg8k | 8,080 frames from 17 Cholec80 videos | pixel masks of 13 classes | Cholec80 video | CC BY-NC-SA 4.0 |
| CholecT50 | 50 videos at 1 frame per second; 45 carry Cholec80 numbers | 100 classes of ⟨instrument, verb, target⟩ triplets, phases | video; official split 35 / 5 / 10 | CC BY-NC-SA 4.0 |
| EndoVis 2017 | 10 robotic sequences of 300 frames, porcine, da Vinci Xi | instrument masks: binary, parts, type | sequence | challenge terms |
| TotalSegmentator | CT: 1,939 images in version 3.0.0 of the open dataset | 3D masks of 117 structures | patient | CC BY 4.0 |
Cholec80 (Twinanda et al., IEEE TMI 2017) is the field’s reference for cholecystectomy video. Every frame carries one of seven phases, the ones walked through on the cholecystectomy page: preparation, Calot triangle dissection, clipping and cutting, gallbladder dissection, gallbladder packaging, cleaning and coagulation, and gallbladder retraction. Once a second it records which of seven instruments is present: grasper, bipolar, hook, scissors, clipper, irrigator and specimen bag. An instrument counts as present when at least half of its tip is visible. All 80 operations come from one hospital.
CholecSeg8k (Hong et al., 2020) adds pixel masks to 8,080 frames of 17 Cholec80 videos, in 13 classes: black background, abdominal wall, liver, gastrointestinal tract, fat, grasper, connective tissue, blood, cystic duct, L-hook, gallbladder, hepatic vein and liver ligament. The classes are very unequal. The liver is in all 8,080 frames, the cystic duct in 241 and the liver ligament in 240, all of them from a single video. The project’s data card records how the class colours were mapped and who signed the mapping off.
CholecT50 (Nwoye et al., Medical Image Analysis 2022) labels 50 videos once a second with triplets such as ⟨hook, dissect, gallbladder⟩. Its authors publish an official split by video (Nwoye and Padoy, 2022), and the project uses it as published. CholecSeg8k and CholecT50 share their source:
Look at the bold cells. Two of the three videos in the CholecSeg8k test split, 18 and 52, are in CholecT50’s training split. A model pre-trained on CholecT50 and tested on CholecSeg8k has seen those two operations before. This is why RFC 0001, which drew backgrounds from CholecT50, tested only on the seven videos that are not in it, 2,960 frames.
EndoVis is the series of endoscopic vision challenges held at MICCAI; its robotic instrument sub-challenge began in 2015. The 2017 robotic instrument segmentation challenge (Allan et al., 2019) used 10 sequences from porcine procedures, sampled at one frame a second. Its organisers gave the first 225 frames of 8 sequences for training and kept the last 75 frames of the same sequences for testing, plus 2 whole sequences. Training and test frames came from the same procedures, so teams had to evaluate each of the 8 test pieces with a model that had not seen that sequence, at least 9 models per team. The organisers wrote that future challenges would release entirely separate training and test sets. The 2018 challenge (Allan et al., 2020) added the anatomy around the instruments.
TotalSegmentator (Wasserthal et al., Radiology: AI 2023) is a CT dataset and a model trained on it. The paper used 1,204 CT examinations with 104 structures; the open dataset on Zenodo, now at version 3.0.0, holds 1,939 CT images with 117 structures under CC BY 4.0. One patient can have several examinations, and they belong in the same split. It is not yet onboarded in osv.datasets.
Why frames leak
At 25 frames per second, neighbouring frames are 40 ms apart. Even at one frame a second they show the same liver, the same fat, the same light and the same instruments. A model asked about a test frame can find its near twin among the training frames and copy the answer. It does well on that test without learning anything that carries over to a new patient.
How likely is a twin? Split frames at random with a share p in training, and look k frames to either side of a test frame. The chance that at least one of those 2k neighbours is a training frame is:
With the usual 70 % in training, 91 % of test frames have a training frame right next to them. How much this inflates the score has been measured in MRI. Yagis et al. (2021) split brain scans by slice instead of by subject, and test accuracy rose by 29 to 55 %, depending on the dataset. With labels assigned at random, the slice-level split still reached about 96 % accuracy, while the subject-level split gave the 50 % that chance predicts.
Try it on a toy: 12 videos of 60 frames, three phases of 20 frames each. A frame’s features are mostly its patient’s own signature, with a weak signal of the phase and a slow drift. A nearest-neighbour classifier names the phase of each test frame from the closest training frame.
- test frames with a training frame a second away
- –
- videos in both training and test
- –
- nearest-neighbour accuracy on test
- –
Split by frame, the classifier is right on almost every test frame, because the nearest training frame is the next second of the same operation. Split by video, it has to rely on the weak phase signal and drops to between a half and three quarters, depending on the draw, with a third being chance. Only the second number says anything about a new patient.
The same thing seen in the model’s feature space: frames of one operation lie along a short path, in a place that belongs to that patient.
- mean distance to the nearest training frame
- –
The patient is not the only thing that can leak:
- The same operation in two datasets. CholecSeg8k and CholecT50 share ten videos, so pre-training on one and testing on the other can put the same operation on both sides.
- The same surgeon, hospital or device. All of Cholec80 comes from one hospital and 13 surgeons. A test on Cholec80 says little about another hospital’s cameras and habits. An external test needs data from elsewhere.
- The same procedure, cut in time. EndoVis 2017 split sequences into a beginning for training and an end for testing, and had to work around it.
- The same patient, several scans. In CT, two examinations of one patient are one patient.
- The test set itself. Choosing a model, a threshold or a normalisation by looking at the test results leaks it too. That is what validation is for.
Patient-level splits in osv.datasets
The project’s rule R-08: dataset splits are partitioned by patient, operation or session; frame-level random splits are a critical bug. In CholecSeg8k the key is the Cholec80 video, videoNN, one operation of one patient. osv.datasets.cholecseg8k.build_patient_split assigns whole videos and never cuts one:
- Each split’s target is its share of all frames: 70, 15 and 15 %.
- Videos are taken from the largest to the smallest; equal sizes go in order of their number, so the result is always the same.
- Each video goes to the split that is furthest below its target, as a share of that target.
Here nv is the number of frames in video v, rs the target share and Ls the frames already given to split s. The published split, osv/datasets/splits/cholecseg8k.json:
Run the same algorithm with other shares. The frame counts are the published ones; the shares must add up to 100.
Because no video is ever cut, frame-level leakage is impossible by construction, not merely tested for. It is tested anyway. assert_no_patient_leakage fails if a video appears in two splits, tests/test_cholecseg8k_split.py::test_no_patient_leakage runs it in CI, and the /split-check command answers SPLIT: CLEAN or LEAKAGE (n) after any change to a split. CholecT50 uses its official split; official_split checks that every video is in exactly one part, and a video the table does not place is an error, not a frame to drop.
A whole-video split has a price. Rare classes live in a few videos, so they end up where those videos go:
| Class | Training | Validation | Test | Consequence |
|---|---|---|---|---|
| Cystic duct | 242 | 0 | 2 | a test score from 2 objects means nothing |
| Hepatic vein | 385 | 0 | 0 | cannot be evaluated |
| Liver ligament | 0 | 248 | 0 | never seen in training; cannot be learned or tested |
The numbers count objects. The liver ligament is in one video only, 09, so no split by video can put it in training and test at once. These three classes are reported as N/A, never as 0 and never inside a macro average. If they matter, the answer is more data or a documented leave-one-video-out protocol, not a quiet change to the split. A published split is frozen; changing it later makes earlier numbers incomparable.
Finally, the test set is kept out of reach. Rule R-25 keeps the held-out test set in a separate DVC track and bucket, and the CI token that runs experiments cannot read it. An improvement counts only on at least three finished seeds with confidence intervals that do not overlap.
Checklist
- Split by patient, operation or session, never by frame; write down the key you split by.
- Check whether two datasets share patients before pre-training on one and testing on the other.
- Never change a published split; report classes the split cannot evaluate as
N/A. - Tune on validation, never on test.
- Run
/split-checkafter any change to a split.
Further reading
- Twinanda et al. EndoNet: A Deep Architecture for Recognition Tasks on Laparoscopic Videos. IEEE TMI, 2017. The Cholec80 dataset.
- Hong et al. CholecSeg8k: A Semantic Segmentation Dataset for Laparoscopic Cholecystectomy Based on Cholec80. 2020.
- Nwoye et al. Rendezvous: Attention Mechanisms for the Recognition of Surgical Action Triplets in Endoscopic Videos. Medical Image Analysis, 2022. Nwoye and Padoy. Data Splits and Metrics for Method Benchmarking on Surgical Action Triplet Datasets. 2022.
- Allan et al. 2017 Robotic Instrument Segmentation Challenge. 2019. Allan et al. 2018 Robotic Scene Segmentation Challenge. 2020.
- Wasserthal et al. TotalSegmentator: Robust Segmentation of 104 Anatomic Structures in CT Images. Radiology: Artificial Intelligence, 2023.
- Yagis et al. Effect of data leakage in brain MRI classification using 2D convolutional neural networks. Scientific Reports, 2021.
- In the osv repository: the CholecSeg8k data card,
osv/datasets/cholecseg8k.pyand the published split.