- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies when incomplete evaluation references still support a reliable model comparison. It introduces a benchmark and a protocol that considers all references compatible with the available evidence, certifies comparisons when they agree, and otherwise seeks more evidence or returns UNKNOWN. Experiments across structured tasks, code evaluation, and speech recognition show that changing the reference can preserve rankings, change pairwise comparisons, or reverse the winner.
Critical strengths
-
The paper distinguishes estimated answers from evidence-supported conclusions. I think this is a meaningful research perspective in ML/DL research. Besides, allowing UNKNOWN and reporting coverage alongside certified error provides a useful discipline for evaluation under incomplete information.
-
The fixed-output design isolates the effect of the evaluation reference. This makes the ranking changes easier to interpret. The speech-recognition winner reversal is particularly consequential because it shows that reference quality can affect practical model selection.
Critical weaknesses
- The contribution beyond established ideas is not sufficiently clear. The benchmark and synthesis could still be valuable, but many experiments illustrate expected consequences of these principles rather than establish substantially new insights.
- The strongest practical examples and the certification results remain somewhat disconnected. The modern ML panels show what happens after stronger references become available. The certification demonstrations mainly use controlled or explicitly structured settings. This leaves unclear how much the proposed protocol helps with realistic model selection when the reference is still incomplete.
Questions for authors
- What is the main contribution beyond applying existing certain-answer and decision-region ideas to evaluation? Which reported finding or capability best establishes that additional contribution?
Thank you for your response. The rebuttal partially addresses my concerns. I am maintaining my original rating of 2.