- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces GRIP, a procedurally generated benchmark for geometric, spatial, and physical reasoning. It evaluates both initial answers and responses to user challenges or peer answers, measuring when models correct mistakes and when they abandon correct answers.
Critical strengths
The motivation is relevant, as single-turn accuracy does not fully capture reliability during interaction. Procedural generation provides control over scenes and reference answers. The no-challenge re-asking control is also useful for distinguishing challenge-induced changes from ordinary answer variability.
Critical weaknesses
-
The contribution over prior work is unclear. PENDULUM already distinguishes incorrect-to-correct and correct-to-incorrect transitions through progressive and regressive sycophancy. Although the protocols differ, the central distinction is established. Procedural generation is useful, but the paper does not clearly show what additional diagnostic insight GRIP provides beyond its construction and task coverage.
-
The scoring protocol weakens the main comparisons. Appendix D.3 and Table 10 state that open-weight and hosted models use different comparators and that their scores are not directly comparable. However, Table 3 and Figure 3 present these groups together, and the paper discusses their performance separation. A shared scoring procedure is needed to support these conclusions. Appendix B.6 also lacks sufficient detail about the semantic validator and its human verification, including audit size and agreement rates.
Questions for authors
What does GRIP reveal that cannot be captured by the corrective and regressive behaviors already studied in PENDULUM?
Can the authors report results using a shared scoring procedure and provide quantitative details of the human audit?