- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This work claims that there is a gap in current benchmarks in that they do not account for human review, which would be present in an actual deployed setting. The effectiveness of human review is very dependent on the model's understanding of its own uncertainty, so models that know when they are likely wrong are cheaper to fix with human review (as you have to review fewer cases, only the ones the model flags). Our benchmarks should be adjusted accordingly, giving higher scores to models that are calibrated. And this paper proposes a simulation setup to do that.
Critical strengths
- Measuring how well model's understand their own certainty is important for deployed settings and should be evaluated in benchmarks
Critical weaknesses
-
This is one of the worst-written papers I have reviewed this year. It is so bad that it feels fully AI written. I could barely understand it. It needs to be rewritten from scratch, by a human, with a focus on clarity. As a simple but clear example of the lack of care put into this work, look at the following in the reproducability statement: " To support independent reproduction, an anonymized version of the reference testbed together with the evaluation configurations, split seeds, and analysis scripts used to produce the reported tables and figures is provided as supplementary material." There is no attached supplement of course. The lack of code release, combined with the poor quality of writing, means that I can't properly review or understand the experimental setup.
-
I think the overall proposal is not good. I agree that benchmarks should also measure internal model uncertainty estimates, as it is useful to know if the model accurately captures when it is right/wrong, but I think the more complex simulation proposed by this work is bad since it adds additional assumptions/complexity. It would be better to simply add brier score or something of that nature to standard benchmark metrics.
Questions for authors
None
Response to Reviews
Dear Area Chair and Reviewers,
Thank you for your time and careful feedback on our submission. We agree that the main concerns need more work than the discussion period allows.
In a future version, we will let human-review accuracy vary with case difficulty and base it on real human-review data. We will also compare READY directly with risk-controlled selective prediction and learning-to-defer methods on the same data, including Pareto Testing and Calibrate-Then-Delegate, and explain more clearly how READY differs. Finally, we will rewrite the paper for clarity and add the related work the reviewers suggested.
We will release the code and evaluation configurations with the revised version.
Thank you again for your helpful comments.
Best regards, The Authors