- Rating
- 4
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper introduces Grounded Information Measure (GIM), a novel 820 question benchmark which distinguishes itself by requiring the coordination of multiple cognitive operations over broadly accessible knowledge. The dataset consists of a 615 public and 205 private instances including 229 multimodal items and a rubric specific scoring of 528 items. The paper analyses the scores using a continious-response 2PL IRT scoring and reports an extensive leaderboard across model and thinking-effort configurations to analyse test-time compute scaling.
Critical strengths
- I believe the idea of the benchmark is valuable in the context of the popularity of benchmarks that become increasingly distant from real world tasks.
- The rubric-decomposed scoring is very interesting in highlighting the distinctive strengths of the different LLMs enabling their comparison
- The dataset construction, LLM coverage, and the scoring is sophisticated and well done. The "centaur" ceiling (human + LLM) is an interesting addition and I think its more meaningful than a human alone measurement.
- The empirical analysis is broad and the statistical rigor behind the reported results is convincing.
- The manuscript is straight forward to read, well written, and easy to understand
Critical weaknesses
- Given that each benchmark item is a combination of multiple rubrics, calculating/estimating some interaction effects between rubrics may be an interesting addition if the sample size allows for it.
Questions for authors
I have no questions.
Thank you for the positive assessment and for the suggestion about interactions between rubric criteria. Some of this interaction is by design: rubrics are also how GIM awards partial credit, and many items ask the model to explain its reasoning or impose progressively harder requirements, so reaching the correct final answer requires passing through specific intermediate steps. Across 470 rubric items, criteria within an item correlate at a median of 0.36, and still at 0.19 after removing each model's average success on each criterion, so the coupling is not explained by model ability alone; since each criterion is graded in a separate judge call, it is not an artifact of a single judgment either. All criteria are met together 2.8× more often than independence predicts, and more tightly coupled items discriminate better between models (Spearman 0.60).