A number is only the start of a question
A metric is easy to quote and hard to check. Which patients were in the test set? Was the training budget the same? Is this a typical run or the best of several? Without answers, another researcher cannot tell what the number actually shows.
That is why, in OpenSurgicalVision, the experiment record matters more than the headline. The unit of progress is a hypothesis linked to a data version, a fixed split, a configuration, complete run records and a conclusion with its limits.
What it takes to repeat an experiment
A good experiment card answers:
- what changed and what stayed the same;
- which data and permissions were used;
- how much compute it took;
- which runs finished and which did not;
- how to repeat the comparison without asking the author.
For the same reason, validation and the held-out test do different jobs. If you keep tuning a model to the test set, it stops being an independent check. So in the benchmark access to the test is limited and visible to everyone.
The description of the environment is part of the result too. An unstated library version or a manual preparation step can matter more than a small difference in a metric.
Where to start
You can help right now. Read a setup guide as a newcomer, find one unstated assumption and propose a clarification. Start with the documentation track or look at the experiment log.