The question
Pixel labels for surgical video are slow to make, and the instrument boundaries are the least consistent part of them. A renderer gives unlimited instruments with exact masks. We tested the strong version of the idea:
H1. A model trained only on synthetic instruments is not worse than the same model trained on real labels, on real instruments.
The hypothesis, the margin and the decision rule were fixed in a design document (RFC 0001) before any training run.
How we tested it
- Synthetic data. Grasper and L-hook models were built in Blender and rendered 20,000 times with exact masks. The renders were laid over 717 frames of the public CholecT50 dataset that contain no instrument, each checked by a person. The augmentations covered light, blur, tissue over the tip, the dark edge of the endoscope image and simulated smoke.
- Real data. CholecSeg8k: 4,000 labelled frames from 8 videos for training and 1,120 frames from 2 videos for validation.
- Test. The 7 CholecSeg8k videos that never served as a synthetic background, 2,960 frames in all. Before the first test run, a person checked every test clip for instruments that have no class of their own. Nothing had to be excluded.
- Model. The same DeepLabV3-ResNet50, settings and number of training steps in every arm. Each arm ran with 3 seeds.
| Arm | Trained on |
|---|---|
| A | synthetic instruments only |
| B | real labels only |
| C10 / C25 / C50 / C100 | arm A’s model, then 10 / 25 / 50 / 100 % of the real frames |
- Model selection. Arm A chose its checkpoint on synthetic validation data only, so no real label reached the synthetic arm.
- Test runs. Every chosen checkpoint was run on the test videos exactly once, and each run was recorded.
Results
The metric is the overlap (IoU) of the instrument mask over all test pixels. A value of 1 is perfect. The range is a 95 % interval over test videos and seeds.
| Arm | Real labels | IoU | 95 % interval |
|---|---|---|---|
| A, synthetic only | 0 % | 0.116 | 0.04 – 0.19 |
| C10 | 10 % | 0.775 | 0.66 – 0.83 |
| C25 | 25 % | 0.767 | 0.63 – 0.83 |
| C50 | 50 % | 0.778 | 0.65 – 0.84 |
| C100 | 100 % | 0.764 | 0.62 – 0.84 |
| B, real only | 100 % | 0.772 | 0.65 – 0.84 |
H1 is rejected. Arm A is 65.5 points behind arm B, with a 95 % interval of −75.9 to −53.5. The design allowed a gap of 2 points at most.
The synthetic-only model learned the synthetic task: it reached 0.96 on synthetic validation data. On real validation frames it scored about 0.45, and on the test videos 0.12. We checked one explanation, that it had memorised the tissue of the validation operations from the backgrounds. Only 29 of the 717 backgrounds come from those operations, so this does not explain the gap. Why the model fails on real instruments is not yet known.
Data as a table
| Epoch | Synthetic val, mean | Real val, mean |
|---|---|---|
| 1 | 0.912 | 0.431 |
| 2 | 0.925 | 0.399 |
| 3 | 0.938 | 0.386 |
| 4 | 0.941 | 0.399 |
| 5 | 0.944 | 0.363 |
| 6 | 0.949 | 0.467 |
| 7 | 0.949 | 0.412 |
| 8 | 0.953 | 0.442 |
| 9 | 0.953 | 0.346 |
| 10 | 0.956 | 0.383 |
| 11 | 0.957 | 0.334 |
| 12 | 0.958 | 0.409 |
| 13 | 0.959 | 0.383 |
| 14 | 0.960 | 0.403 |
| 15 | 0.961 | 0.394 |
| 16 | 0.962 | 0.364 |
| 17 | 0.963 | 0.398 |
| 18 | 0.963 | 0.438 |
| 19 | 0.963 | 0.429 |
| 20 | 0.963 | 0.442 |
Six validation frames
The figure shows what the seed-1 models of arms A, B and C10 predict on six real validation frames. The frames were spaced evenly and fixed before any prediction was seen. Test frames are not shown: running the models on them again would be another use of the test set.
- Where a grasper stands out against the tissue, arm A finds part of it (IoU 0.39 to 0.76).
- Where the instrument sits at the edge of the view or has a dark shaft (rows 2 and 6), arm A finds almost nothing (0.00 and 0.02). Arms B and C10 find these instruments (0.73 to 0.90).
- In the last frame arm B marks the L-hook as a grasper, and C10 labels it correctly. The IoU of the instrument mask does not count the class.
Six frames illustrate the numbers above. They are not evidence on their own.
What these numbers are not
- Not proof that synthetic pre-training saves labels. Arms C reach arm B with 10 % of the real frames, about 400 frames, and gain nothing beyond that. But the design had no arm trained on those 10 % without the synthetic step. Without it we cannot tell whether the synthetic step helped, or whether 400 real frames are enough on their own.
- Not a verdict on synthetic data in general. It holds for this renderer, these instruments and 717 backgrounds, which is fewer than the roughly 1,000 we had planned.
- Not a clinical result. Speed, glare and blood were not measured separately, and no model is released.
RESEARCH USE ONLY – NOT FOR CLINICAL USE.
What went wrong along the way
- The first renders were three times slower than they had to be. The denoiser ran on the CPU while the GPU waited. After the fix, 20,000 renders took about 3 hours.
- The synthetic arm crashed after a few epochs. Some smoke images had been damaged when they were copied to the GPU server. The runs were resumed from their last checkpoint, and every image is now checked before training starts.
- Six fine-tuning runs started from unfinished models because of that crash. They were marked invalid, never run on the test, and the training code now refuses to start them.
Possible next steps
- A control arm trained on 10 % and 25 % of the real frames without the synthetic step. This is the cheapest way to learn what the synthetic step adds.
- Domain randomisation: deliberately unrealistic textures and colours, to test whether the gap is one of appearance.
The full model card with run identifiers is in the project repository. See also the experiment log and why negative results matter.