The task
Electrocautery fills the laparoscopic view with smoke. The task is to restore the clean frame from a smoky one. The same moment cannot be filmed both with and without smoke, so we train on synthetic pairs: smoke simulated in Blender is laid over clean frames of the public CholecT50 dataset of laparoscopic cholecystectomies.
- Training: 5,882 clean frames from 35 videos, with new smoke drawn every epoch.
- Validation: 518 fixed pairs from 5 other videos.
- Split: by video, so no operation is on both sides.
Every model is compared with the no-op baseline: the smoky frame left as it is. A model that cannot beat doing nothing is of no use.
Results
All three models were trained in one session on a rented A40 GPU, with 15 epochs each on full 854×480 frames and one seed. The table gives PSNR on the validation pairs in dB, with the difference from the no-op baseline in brackets.
| Model | All frames | Light smoke | Medium smoke | Dense smoke |
|---|---|---|---|---|
| Residual U-Net | 33.08 (+4.09) | 41.05 (+2.54) | 34.47 (+5.42) | 28.27 (+3.89) |
| Pix2Pix | 27.85 (−1.14) | 32.97 (−5.54) | 29.00 (−0.05) | 24.57 (+0.19) |
| SRGAN | did not converge | – | – | – |
| No-op baseline | 28.99 | 38.51 | 29.05 | 24.38 |
The residual U-Net predicts a correction to the input rather than a new image. It improves on the input at every smoke density, most of all on medium smoke.
Pix2Pix, a GAN, barely changes medium and dense smoke, and it damages frames with light smoke: its generator paints plausible texture even where the frame was almost clean. This is why the output of a GAN must never be read as what the smoke was hiding.
What did not work, and why
- Crops. The U-Net was first trained on 256×256 crops, a common default.
- After 11 epochs it was still below the input: 28.75 against 28.99 dB.
- Crops dilute the smoke and never show the dark edge of the endoscope image.
- On full frames the same network was 2.4 dB ahead of the input after 5 epochs.
- SRGAN, three attempts.
- During training the output was clipped to the valid range, so pixels that left it got no gradient to bring them back. Within an epoch every pixel turned pure red, and PSNR was 6.3 dB.
- After the fix, one run accidentally used the old code and repeated the failure.
- In the third attempt the generator first trains on pixel error alone, as in the original paper, and still made frames worse than the input: 25.4 dB. The most likely cause is the optimiser settings borrowed from Pix2Pix; the paper used different ones. We stopped there rather than spend more GPU hours.
- Operations. A forgotten process trained next to the main run for 2.5 hours and halved its speed. Checks for this are now part of the training runbook.
What these numbers are not
- Not a finished series. These are results for one seed on the validation split. We claim an improvement only on at least three seeds with non-overlapping confidence intervals, so this series is unfinished. The held-out test split has not been touched.
- Not real smoke. The smoke is synthetic, and no clinician has validated it against real smoke. Real smoky frames are not labelled yet.
- Not a full evaluation. Speed, glare, blood and defocus have not been measured.
- Not a release. No model is published.
RESEARCH USE ONLY – NOT FOR CLINICAL USE. Restored frames must never be shown to a surgeon during an operation.
Possible next steps
- Two more seeds for the U-Net, then a single, recorded check on the test split.
- Labelling real smoky frames, so that models can also be checked on real smoke.
- Another SRGAN attempt with the optimiser settings from the paper.
The full model card with run identifiers is in the project repository. See also the experiment log and why negative results matter.