acceptodds
Under review as a conference paper at ICLR 2027

GIM: Evaluating models via tasks that integrate multiple cognitive domains

Abstract

As LLM benchmarks saturate, the evaluation community has pursued two strategies to increase difficulty: escalating knowledge demands (GPQA, HLE) or removing knowledge entirely in favor of abstract reasoning (ARC-AGI). The first conflates memorization with capability; the second divorces reasoning from the practical contexts in which it matters. We take a different approach. The Grounded Integration Measure (GIM) is a benchmark of 820 original problems (615 public, 205 private) where difficulty comes from integration; individual problems require coordinating multiple cognitive operations (constraint satisfaction, state tracking, epistemic vigilance, audience calibration) over broadly accessible knowledge, so that reasoning stays grounded in realistic tasks without being gated on specialized expertise. Each problem is an original expert-authored composition, majority with rubric-decomposed scoring (median 6 independently judged criteria). A balanced public–private split provides a built-in contamination diagnostic. We calibrate a judge-aware continuous response 2-parameter logistic (2PL) IRT model across 53 test-configurations (unique model × thinking-level pairs) and five calibrated judges, using 203,800 epoch-averaged prompt–judge cells derived from >1M raw judge-scored observations, producing robust ability estimates that correctly order test-configurations even when raw accuracy is distorted by errors, missing data, or judge leniency differences. Using this framework, we present a comprehensive leaderboard spanning 22 models and 47 reporting test-configurations, and conduct what is to our knowledge the most extensive published study of how test-time compute trades off against model capability on a fixed benchmark: 11 models swept across 35 test-configurations. We observe that within-family configuration choices, such as thinking budget and quantization, matter as much as model selection, and increasing thinking tokens has diminishing marginal returns. We release the evaluation framework, calibrated IRT parameters, and all public problems.

Primary area:
datasets and benchmarks
Keywords:
large language models, benchmark, item response theory, rubric-based evaluation, test-time compute

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rc68OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces Grounded Information Measure (GIM), a novel 820 question benchmark which distinguishes itself by requiring the coordination of multiple cognitive operations over broadly accessible knowledge. The dataset consists of a 615 public and 205 private instances including 229 multimodal items and a rubric specific scoring of 528 items. The paper analyses the scores using a continious-response 2PL IRT scoring and reports an extensive leaderboard across model and thinking-effort configurations to analyse test-time compute scaling.

Critical strengths
  • I believe the idea of the benchmark is valuable in the context of the popularity of benchmarks that become increasingly distant from real world tasks.
  • The rubric-decomposed scoring is very interesting in highlighting the distinctive strengths of the different LLMs enabling their comparison
  • The dataset construction, LLM coverage, and the scoring is sophisticated and well done. The "centaur" ceiling (human + LLM) is an interesting addition and I think its more meaningful than a human alone measurement.
  • The empirical analysis is broad and the statistical rigor behind the reported results is convincing.
  • The manuscript is straight forward to read, well written, and easy to understand
Critical weaknesses
  • Given that each benchmark item is a combination of multiple rubrics, calculating/estimating some interaction effects between rubrics may be an interesting addition if the sample size allows for it.
Questions for authors

I have no questions.

1 comment

Thank you for the positive assessment and for the suggestion about interactions between rubric criteria. Some of this interaction is by design: rubrics are also how GIM awards partial credit, and many items ask the model to explain its reasoning or impose progressively harder requirements, so reaching the correct final answer requires passing through specific intermediate steps. Across 470 rubric items, criteria within an item correlate at a median of 0.36, and still at 0.19 after removing each model's average success on each criterion, so the coupling is not explained by model ability alone; since each criterion is graded in a separate judge call, it is not an artifact of a single judgment either. All criteria are met together 2.8× more often than independence predicts, and more tightly coupled items discriminate better between models (Spearman 0.60).

Reviewer_Xi8mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces GIM, a benchmark that evaluates LLM capabilities from the perspective of integration: answering a question requires not only common knowledge but also the coordination of multiple cognitive operations. GIM consists of 820 original questions, including 615 public and 205 private items. The authors employ a judge-aware two-parameter logistic item response theory (2PL IRT) model with judge fixed effects to estimate item difficulty and model ability. They provide a leaderboard covering 22 models and 47 configurations and investigate the effects of test-time compute.

Critical strengths
  1. The main contribution is to conceptualize integration as a distinct dimension of LLM capability. Compared with HLE and GPQA, which emphasize challenging or specialized knowledge, and ARC-AGI, which focuses on abstract reasoning, GIM aims to capture capabilities that are relevant to practical professional and general-purpose scenarios, where answering a question often requires combining knowledge with multiple cognitive operations.

  2. The benchmark contains original questions and is supported by a reasonably extensive evaluation of models and configurations. The release of the data and item bank is also valuable to the research community and may facilitate further investigation of this capability dimension.

Critical weaknesses
  1. The paper would benefit from substantial improvements in presentation and consistency. IRT first appears in the abstract without being expanded. Although its full name is given in the contributions, the abbreviation is not formally defined until Section 5. The Introduction is also very brief, while much of the motivation is relegated to Appendix A. More importantly, several numerical inconsistencies need clarification. Section 6 states that the strongest model achieves full credit on approximately 39% of the questions, whereas Figure 22 appears to show values below 1% for all configurations. The reported (θ\theta) values in Figure 15 also do not appear to match those in Table 4. In addition, the number of evaluated configurations varies across the paper, with 47, 48, and 53 all being reported. These inconsistencies make it difficult to interpret the results and assess the reliability of the reported findings.

  2. The paper's central claim is that GIM captures a new viewpoint of LLM capability that is not adequately measured by existing benchmarks. However, this claim is not sufficiently supported by the current experiments. First, the paper does not report correlations with existing benchmarks or sufficiently analyze the relationship between item difficulty and the degree of integration required. Without these analyses, it remains unclear whether GIM measures a genuinely distinct capability dimension or primarily reflects existing capabilities such as knowledge and reasoning. Second, the World Knowledge category appears to be among the most difficult categories. This raises an important question: are the observed difficulties primarily attributable to integration demands, or are they driven by the knowledge required to answer the questions? Similarly, the claim that the questions rely on "broadly accessible knowledge" lacks direct validation. A baseline measuring unaided human performance would help establish whether the questions are indeed broadly accessible and whether the benchmark reflects integration rather than primarily specialized knowledge.

Questions for authors
  1. Could the authors report correlations between GIM and existing benchmarks and analyze the relationship between item difficulty and the degree of integration required? How do these analyses establish that GIM measures a distinct capability dimension rather than primarily reflecting knowledge or general reasoning? Could the authors provide an unaided human baseline to substantiate the claim that the benchmark relies on broadly accessible knowledge?

  2. Why does the main text describe selection of the Top Human+AI configuration using IRT ability, while Appendix G refers to raw scores? How do the authors account for potential batch-difficulty effects and selection bias when each participant answers only one batch? Why does the Average Human+AI configuration perform approximately one logit worse than Gemini 3.1 Pro alone, and what interpretation do the authors give to this result?

  3. Could the authors clarify the scoring formula, particularly the use of (n) rather than (∑c\sum{c}) in the denominator, and explain how judge uncertainty affects the resulting scores? Could they re-examine the confidence intervals and the public-private contamination analysis using a statistically justified uncertainty-estimation procedure?

  4. Would the authors provide quantitative agreement results between LLM judges and human judges on a representative subset of the benchmark? Could they also disclose the three frontier models and six confidential configurations used for question filtering, or provide sufficient information to assess their potential influence on the benchmark?

  5. Could the authors clarify the IRB status of the Centaur study and explain the rationale for the applicable review procedure? Can they also discuss the absence of some recent models from the evaluation and whether including them might materially affect the reported leaderboard?

I find the proposed GIM benchmark promising and potentially valuable to the community. I also agree with the authors that GIM's setting is more realistic than the other two categories discussed in the paper. However, the current version leaves important questions about construct validity, methodological reliability, statistical uncertainty, and experimental transparency unresolved.

I would be more positive toward acceptance if the authors could convincingly address the central concerns, particularly by validating the relationship between GIM and existing benchmarks, reporting human-judge agreement, revisiting the confidence intervals, and clarifying the selection procedures and models used during benchmark construction.

3 comments

Thank you for the thorough review. Given the 5,000-character limit on replies, we focus on the four central concerns you list.

Correlations with existing benchmarks and degree of integration. Against published scores, GIM's model ranking correlates with HLE at 0.99, ARC-AGI-2 at 0.98 and GPQA Diamond at 0.94 (Spearman), and less with factual recall (SimpleQA Verified 0.75) and instruction following (IFBench 0.70). Item difficulty rises monotonically with the degree of integration, measured as the number of distinct cognitive categories an item requires (median b from −0.67 for one category to +0.12 for four or more, about +0.15 SD per category with controls), while the number of rubric criteria shows no association. Correlation across current models cannot establish whether two benchmarks measure the same dimension: HLE, which is knowledge-heavy, and ARC-AGI-2, which is abstract, correlate at 0.97 despite being designed to measure different abilities. What distinguishes GIM from knowledge-heavy and abstract benchmarks is its design, which the difficulty gradient supports.

Human–judge agreement. There was no time to have humans grade a representative subset, which we leave to a future revision. Instead, we measured agreement with ground truth computed in code. Every judge agrees with a rule-based checker on 95–96% of explicitly stated numeric answers; on word and character limits three judges reach 92–95% accuracy and two only 72%, but removing every item with a counting criterion leaves the leaderboard unchanged (Spearman 0.9998); and no judge consistently favors its own model family.

Confidence intervals and the public/private analysis. We re-examined both with a prompt-cluster bootstrap with full refits, which accounts for what the published SE missed: it treated the five judges' scores of one response as independent, although their residuals correlate strongly (ICC 0.64), and ignored item-parameter uncertainty. The median SE rises from 0.024 to 0.078, so 95% intervals are about ±0.15. The top of the leaderboard is stable (GPT 5.4 Pro X-High ranks 1–2, Gemini 3.1 Pro High 3–4), but only 5 of 45 adjacent pairs are separable and the Gemma 4 quantization differences are not significant, so we withdraw that claim; we will report bootstrap and rank intervals throughout. With these intervals, the public/private gaps show no contamination signal: the median gap is 0.13 against a median SE of 0.15, and the mean difference is +0.001.

Centaur selection procedure. The main text is correct: Top Human + AI is the 15 retained participants with the highest IRT ability, and Appendix G's "highest average raw score" is an error we will correct. Raw scores enter only the exclusion step, which targeted off-task submissions from participants who judged the LLM's answer instead of answering the prompt with its help (Appendix G). We implemented it as a raw-score cutoff (0.30, removing 51 of 246 participants), and a blind check confirms it: of one random answer per participant, 28 of 40 from removed participants but 1 of 40 from retained ones evaluated an AI response instead of answering. IRT ability accounts for batch difficulty: unlike raw scores (ρ = −0.17), it is uncorrelated with the difficulty of a participant's batch (ρ = 0.005). On selection bias: the Top group is an intentional selection, presented as a pilot and a suggestive existence proof (Section 5 and Limitations), and its score is optimistic because it was chosen and scored on the same answers; chosen on part of each participant's answers and scored on the rest, it reaches 0.62–0.64 rather than 1.73. We will report this estimate and soften the current claim that the group exceeds the autonomous frontier.

Models used for question filtering. Vendor-sourced items had to stump two of three frontier models available at sourcing: the 2025 batches were filtered against the frontier models of the time, and the 2026 batches against Gemini 3 Pro, GPT-5.2 and Claude Opus 4.5, none of which is on our leaderboard. Among reported models, only Gemini 2.5 Pro and Claude 4 Opus can have been filters, and only for 2025 batches; Claude 4 Opus scores 0.73 logits below what its GPQA Diamond score predicts, so we will flag it as possibly affected, while Gemini 2.5 Pro shows no consistent shift. None of the six confidential configurations was used for filtering. They are withheld from reporting for confidentiality (see the introduction), and naming them could reveal the authors' identity; refitting without them shifts every score by the same constant.

The centaur study was reviewed as part of the research project as a whole; the Ethics statement describes its IRB status and the rationale for this procedure. We will also fix the presentation inconsistencies you found, which come from moving the paper from one judge to five, and address your remaining questions in a future revision.

Reviewer_Xi8mOpenReview ↗

Thanks; central concerns remain

Thanks for the detailed reply. Several issues are resolved: the CI re-estimation, the Top Human+AI correction, and the disclosure of the filtering models.

My main concern stands. The claim that integration is a distinct dimension still lacks sufficient empirical support. I agree rank correlations cannot settle dimensionality, but that argument works against GIM's own leaderboard too. What remains is a modest +0.15 SD per category gradient with undescribed controls, and World Knowledge being among the hardest categories is still unexplained. Design rationale is not construct validity.

The corrections also weaken the paper as written: only 5 of 45 adjacent pairs are separable, and the corrected Top Human+AI estimate (0.62 to 0.64, not 1.73) no longer clearly supports exceeding the frontier. And judge validation covers only mechanically checkable criteria. What share of the rubric do they represent? The open-ended part, where integration is actually judged, has no human ground truth.

Thank you for the follow-up. We agree with your main point and want to set out what the paper contributes independently of it.

Integration. The current evidence does not establish integration as a distinct dimension, and the rank-correlation argument applies to GIM's own leaderboard too. The category gradient (controls: primary category, modality, prompt length, knowledge proxies) is modest, and we do not rest the claim on it. We revised to present integration as the design principle behind the items, not as a validated construct. On World Knowledge: two LLM coders, without access to difficulty data, agree (κ 0.70) that WK items mostly hinge on one specific fact, and with that controlled WK items are no harder than others. GIM reduces knowledge gating; it does not eliminate it.

What the paper contributes without that claim. First, the item bank: 820 original, expert-authored problems, 205 held out, on which the strongest configuration scores 0.9 or higher on only 29% of prompts. Second, the IRT layer, which the paper did not make clear enough. The method is not new; the contribution is a released, calibrated scale: item parameters plus judge offsets, so anyone can score a new model with a single judge, on any subset of the bank, with missing responses handled consistently. It is also why your questions could be answered from existing data:

  • Removing the 84 items with questioned or counting rubrics, or all WK items, leaves the leaderboard unchanged (Spearman 0.9998 and 0.998).
  • Refitting without the six withheld configurations shifts every score by the same constant (item difficulties correlate at 0.995).
  • Centaur participants who answered different batches are comparable: IRT ability is uncorrelated with batch difficulty (ρ = 0.005), unlike raw scores (ρ = −0.17).

Separability and the centaur estimate. "5 of 45 adjacent pairs" is the resolution of the scale, and we revised to report it as such. Many adjacent entries are thinking levels of one model, and the coarser findings survive: the top ranks are stable and Low→Medium thinking gains are significant for 8 of 10 models. We revised to report the held-out estimate (0.62–0.64) for Top Human + AI and present the centaur study as a pilot, without the comparison to the autonomous frontier.

Judges. The code-checked criteria are about 2% of graded units. On the open-ended 98%, the five judges, from different model families, agree closely (Fleiss κ 0.90 on final-answer criteria, 0.83 on intermediate steps). Agreement is not accuracy, and we revised to state the lack of human ground truth as a limitation.

With these changes, the paper's claim is the one you described as valuable: an original, partly held-out item bank, released as a calibrated scale that others can score new models on.

Reviewer_dSpEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces GIM, a benchmark of original, expert-authored problems with a public split and a held-out private split. The central premise is that difficulty should come from the need to integrate several cognitive operations (constraint satisfaction, state tracking, epistemic vigilance, audience calibration, procedural fidelity) over broadly accessible knowledge, rather than from specialized knowledge or from abstract, ungrounded puzzles. Most problems are scored by LLM judges against decomposed rubrics, and the remainder against a reference answer. Scores are aggregated with a continuous-response 2PL IRT model that includes a per-judge leniency term, so that a new model can be placed on the released scale using any one of the calibrated judges. The authors evaluate a broad range of proprietary and open-weight models across thinking budgets and quantization levels, and add a human-with-LLM baseline. Validation consists of inter-judge agreement, single-judge recovery of the joint ability scale, leave-one-model-out refits, per-epoch variability, and public/private consistency as a contamination diagnostic. The paper also analyzes test-time compute and difficulty by category.

Critical strengths

The main reason I lean towards acceptance is the problem set itself. The items are written by humans, follow detailed authoring and rubric guidelines, and are partly held out. They also sit in a region that existing benchmarks cover poorly, namely tasks that do not require obscure knowledge but still defeat frontier models. Several of the worked examples, such as spotting an anachronism in a period document, recognizing that an itinerary across time zones is impossible, or pitching an explanation to a particular reader, expose failure modes that knowledge-heavy and puzzle-style benchmarks miss. For a benchmark paper, the quality and longevity of the items matter more than anything else.

I also find the scoring framework useful in practice. Because judge leniency is modeled explicitly and the item parameters are released together with the judge offsets, other groups can place a new model on the leaderboard with one calibrated judge rather than rerunning the full panel. The IRT layer also handles missing responses consistently and gives item-level parameters that can be used to retire saturated items later. Combined with the sweep over thinking budgets and quantization levels, this makes the release more than a one-off ranking.

Critical weaknesses

My first concern is that the paper never directly tests its central claim that difficulty comes from integration. The evidence offered is mostly illustrative examples and category-level summaries. I could not find an ablation that splits problems into single-operation sub-tasks, an analysis relating item difficulty to the number of interacting operations or constraints, or a comparison with existing benchmarks showing that GIM measures something other than general reasoning ability. The per-category ability profiles are also fairly flat across models, which a single general factor would explain at least as well as the integration hypothesis. Since this framing is what sets GIM apart from GAIA, GPQA or HLE, the paper at present reads as a carefully built general reasoning benchmark rather than as evidence for a new axis of difficulty.

My second concern is whether the scores and their stated uncertainty can be trusted. The LLM judges are compared only with one another and never with human graders, and agreement among LLM judges says little about validity, especially in the categories the authors themselves call more subjective. Some of the showcased rubrics are also problematic:

  • In C.11, the word positions checked by the rubric do not match the most literal reading of "every second word".
  • In C.1, the rubric does not appear to accept a valid alternative opening, namely ferrying the goat together with the dove.
  • C.5 has only "avoids ..." criteria, so a near-empty answer could receive full credit.
  • C.9 asks an LLM judge to verify exact word counts. If problems like these appear among hand-picked examples, the rest of the bank should be audited. On the statistical side, the reported confidence intervals look narrow compared with the public/private gaps shown in the appendix. Scores from different judges on the same response seem to be treated as independent observations, and uncertainty in the item parameters is not carried into the ability estimates. In addition, the confidence-weighted per-sample score is not normalized, so judge uncertainty leaks into model ability. All of this matters because the fine-grained leaderboard and the thinking-level comparisons make up much of the paper's contribution.
Questions for authors

Has any subset of responses been graded by humans, and how well do the judges agree with them? Relatedly, are the rubric problems noted above isolated, or would a re-audit of a random sample of items find similar issues?

Do the reported standard errors account for the correlation among judges scoring the same response and for uncertainty in the item parameters? I would also like to understand how the stated intervals square with the public/private differences in the appendix.

Is there any direct evidence that integration drives difficulty, for example how item difficulty varies with the number of interacting operations, or how the integrated problems compare with their single-operation components?

1 comment

Thank you for the detailed review. We answer your three questions, which cover the central concerns.

Human grading and judge agreement. No subset of responses has been graded by humans; we leave a human–judge agreement study to a future revision. Instead, we checked the judges against ground truth computed in code: every judge agrees with a rule-based checker on 95–96% of explicitly stated numeric answers; on word and character limits three judges reach 92–95% accuracy and two only 72%; and no judge consistently favors its own model family.

Would a re-audit find similar issues? Every item went through several human checks before release: two expert reviews at the vendor before delivery, a final review by us at acceptance and, after acceptance, a re-examination of every item no model solved, in which flawed items were fixed or removed. Automated checks are of limited help here: an automated review in February 2026 flagged 97 items as critical, and human review of all 97 removed 3 and corrected 9 answers and 3 prompts, but upheld 64 of the 73 answers it called incorrect; on a benchmark built to defeat models, model-based audits often flag correct items. We do not expect the dataset to be error-free, but given how many checks each item went through, we expect errors to be the exception. Their effect is also bounded: removing the four items you cite plus all 81 items with counting criteria leaves the leaderboard unchanged (Spearman 0.9998, at most one rank place).

Standard errors and the public/private differences. The reported SEs account for neither: they treat the five judges' scores of one response as independent, although their residuals correlate strongly (ICC 0.64), and they ignore item-parameter uncertainty. A prompt-cluster bootstrap with full refits, which accounts for both, gives a median SE of 0.078 instead of 0.024 (95% intervals of about ±0.15). The stated intervals are too narrow to square with the public/private differences. With the bootstrap intervals the differences are consistent with noise: the median gap is 0.13 against a median SE of 0.15, and the mean difference is +0.001. The diagnostic can only detect gaps above about 0.3 logits. The top of the leaderboard is stable (GPT 5.4 Pro X-High ranks 1–2, Gemini 3.1 Pro High 3–4) and Low→Medium thinking gains remain significant for 8 of 10 models, but only 5 of 45 adjacent pairs are separable and the quantization differences are not significant, so we withdraw that claim and will report bootstrap and rank intervals throughout. Finally, refitting with normalized or unweighted confidence scores leaves the leaderboard essentially unchanged (r ≥ 0.9996), because the judge fixed effect absorbs the judges' confidence habits.

Direct evidence that integration drives difficulty. Comparing integrated problems with their single-operation components needs new authoring and inference, so we leave it to a future revision. For the number of interacting operations, we use the number of distinct cognitive categories an item requires: item difficulty rises monotonically with it (median b from −0.67 for one category to +0.12 for four or more; about +0.15 SD per category, controlling for primary category, modality, prompt length and knowledge proxies), while the number of rubric criteria, which reflects how finely an answer is decomposed rather than how many operations it combines, shows no association. We will add this analysis. Against existing benchmarks, GIM's model ranking correlates with HLE at 0.99, ARC-AGI-2 at 0.98 and GPQA Diamond at 0.94 (Spearman), but high correlation across current models does not establish a shared construct: HLE and ARC-AGI-2, designed to measure knowledge and abstraction respectively, correlate at 0.97 with each other. Flat category profiles are likewise expected when current models improve on everything together: a general factor explains 95% of the between-model variance in category abilities, yet the category-specific deviations are partly reliable (split-half 0.46).

Comments on the paper

1 comment

We thank the reviewers for careful, specific reviews. Each individual reply addresses that reviewer's central concerns; all new analyses use the existing model responses and judge outputs. The main changes:

  • Uncertainty. A prompt-cluster bootstrap with full refits gives standard errors about three times the published ones (median 0.078 vs. 0.024). The top of the leaderboard is stable and the public/private gaps are consistent with noise, but only 5 of 45 adjacent pairs are separable, and we withdraw the quantization claim. We will report bootstrap and rank intervals.
  • Relation to existing benchmarks. GIM's model ranking correlates with HLE (0.99), ARC-AGI-2 (0.98) and GPQA Diamond (0.94) about as strongly as HLE and ARC-AGI-2 correlate with each other (0.97) despite their different designs, and item difficulty rises with the number of distinct cognitive categories an item requires.
  • Judges. Against ground truth computed in code, judges agree on 95–96% of explicit numeric answers; two judges count words poorly, but removing every counting item leaves the leaderboard unchanged. A human–judge study is left to a future revision.
  • Presentation. We will fix the inconsistencies left over from moving the paper from one judge to five.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.