- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes a calibration-based approach to predict LLM judges’ resolution and demonstrates how reasoning budgets can reverse their preferences.
Critical strengths
The research question of resolution is meaningful and the main claim is clear, also the paper is well organized.
Critical weaknesses
- The writing of abstract needs to abbreviate, and also make it more intuitive. For example, when introducing the resolution example, authors can name the phenom and then measure its impact. Also, what’s the relationship of the 3 ‘R’s? Are they parallel? Current writing needs big improvement.
- Authors should clarify what qwen models they use in table 1, to consist with the notion of gemma 3-4B.
- After adapting the now existing measurement problem solution in LLM as a judge area, do authors post any new methods? Seems current method can be directly get from a very simple statistical process. Thus, there will be necessary to add these baselines: (1)Empirical channel (to see what GET actually added) (2)IRT/GRM (3) Simple regression
- The theoretical proof on GET provides a framework, but the current version hasn’t proven it’s necessary and can not be substitute.
Questions for authors
See above