acceptodds
Under review as a conference paper at ICLR 2027

GAMUT: A Benchmark for Factual Completeness with Two-Level Meta-Rubrics

Abstract

We introduce GAMUT (Grounded Assessment of Multimodal Factuality), an everyday deep research benchmark for factual completeness, built from naturally phrased questions concerning everyday subjects, yet requiring multi-step research and multi-paragraph answers. Evaluating these answers exposes a fundamental tension in rubric-based evaluation. A faithful rubric must express the structure of the space of good answers: open-ended sets of acceptable options, ordered processes, and relative importance. Consistent automatic grading, however, favors narrow, independently checkable criteria. GAMUT addresses this tension with a two-level meta-rubric design: a structured meta-rubric captures answer requirements at authoring time, which are compiled by fixed mechanical rules into a flat binary checklist for evaluation. The benchmark comprises 1,813 questions grounded in real-world images across 10 domains, each paired with an evidence-backed rubric verified by expert human annotators. Evaluating 14 frontier and open-weight models, we find GAMUT challenging and highly discriminative, with scores ranging from 5.1% to 58.7%, and broadly consistent rankings across three LLM judges.

Primary area:
datasets and benchmarks
Keywords:
factual completeness, long-form factuality, rubric-based evaluation, LLM-as-a-judge, open-ended generation, multimodal benchmarks

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uHRTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GAMUT, a benchmark of 1,813 image-grounded "everyday deep research" questions (images from CRAG-MM, 10 domains) targeting factual completeness (recall) of long-form answers rather than precision. Its central idea is a two-level meta-rubric: an expressive structured rubric authored at construction time (types: Simple Knowledge, Strict/Flexible List, Process, Relationship; plus meta-insights and three importance tiers — Answer-Critical, Valuable, Context) that is deterministically compiled into a flat binary checklist for LLM-judge scoring. Scoring uses four verdicts (meets / partially meets / missing / contradicts), penalizing a contradiction more heavily than an omission, and combines the three tiers with fixed global weights (0.6 / 0.3 / 0.1). Fourteen models are evaluated with three judges; scores span 5.1%–58.7%. A text-only variant (1,806 questions) isolates the cost of visual identification.

Critical strengths

1.Well-posed conceptual problem. The precision-vs-recall framing and the "expressiveness vs. grading-reliability" tension are genuine and clearly articulated; a flat one-fact-per-check rubric truly cannot represent open sets, ordering, or relative importance. 2.Elegant two-level design. Authoring in the structured space and grading in the compiled binary space is a clean resolution, and the compilation rules are concrete and auditable. 3.Careful construction and grounding: cited evidence snippets (97%/94% coverage), multi-round human revision, and entity/question corrections. 4.Broad, discriminative evaluation: 14 models, 3 judges, judge-agreement analysis, per-tier and verdict breakdowns, and a text-only control.

Critical weaknesses

1.The central claim is asserted, not tested. The thesis is that the two-level design inherits low grading variance while retaining expressiveness, yet there is no ablation comparing grading against the structured meta-rubric holistically vs. grading the compiled binary checklist (e.g., judge variance, inter-judge agreement, correlation with human completeness judgments). Cross-judge rank agreement is not the same evidence. This is the most important missing experiment.

2.Multimodality is thin. The visual channel acts almost entirely as an entity-identification gate: the text-only variant raises every model's score by 7–22 points. The benchmark is thus closer to a factual-recall test with a perception front-end than to deep multimodal reasoning; the title/framing overstate the multimodal contribution.

3."Partially meets" reintroduces the subjectivity the design avoids. The motivation is that judges are more reliable on binary items, yet the four-way verdict asks for a graded judgment again, with no reliability analysis specific to the partial/contradicts verdicts.

4.Arbitrary scoring constants with no sensitivity analysis. The partial-credit weight, the contradiction penalty, the tier weights, and the decision to also penalize contradictions in the denominator are all justified only by intuition; "roughly a passing score" is undefined. Rank stability under perturbation should be shown.

5.Rubric non-exhaustiveness penalizes valid alternatives. For Strict Lists and Processes, a correct fact/step outside the enumerated set is scored as missing or (worse) contradicting; the asymmetric contradiction penalty amplifies this risk.

Questions for authors

1.Can you provide a direct variance/agreement comparison between structured-rubric grading and compiled-binary grading on a subset? 2.What is the inter-annotator agreement for rubric revision (Appendix D is referenced but absent from the provided text)? 3.How sensitive are the rankings to the contradiction penalty and the tier weights?

4 comments

We thank the reviewer for a detailed and constructive review, and for recognizing both the problem and the two-level design. We agree that directly comparing grading the meta-rubric with grading the compiled rubric was the most important missing experiment. It has been added in the revision, along with the hyperparameter sensitivity analyses.

W1 & Q1: Evaluating meta-rubrics directly

We thank the reviewer for this suggestion. We added this experiment to the new Appendix G. Grading the compiled binary rubric is more stable than grading the meta-rubric directly. Even frontier judges are significantly more consistent across repeated runs with compiled rubrics, and the advantage widens with a less capable judge, both across runs and across judges (Table 8 in Appendix G).

W2: multimodality

We agree that GAMUT does not target deep multimodal reasoning, and we do not claim it does: the title and framing are about factual completeness, and the paper states that GAMUT "is multimodal-first, [but] its evaluation framework is modality-agnostic." The visual signal plays the role it plays in real use. Users point a camera at something and ask about it, so recognizing the subject is the first step of answering, and our results show it is a substantial bottleneck: the subject-identification check is contradicted for 11–38% of questions across models. The text-only variant isolates this cost, so the benchmark measures knowledge completeness rather than perception alone. Our contribution is therefore twofold: a multimodal benchmark for everyday deep-research questions, where the image is how the question is grounded, and a modality-agnostic evaluation framework, which we demonstrate on the released text-only variant. We will tighten any wording that suggests multimodal reasoning beyond grounding.

W3 & W4 & Q3: Hyperparameter sensitivity

We thank the reviewer for raising this question. It is an experiment that could have strengthened our paper. We re-scored the existing verdicts of all three judges under alternative hyper-parameters (new Appendix F), and show that model rankings are highly stable under all of these choices. Tier weights: Across all 884 weightings on a 0.01 grid that keep the tier order (w_AC ≥ w_V ≥ w_C), from Answer-Critical only (1, 0, 0) to equal weights (1/3, 1/3, 1/3), at most 3 of the 91 model pairs change order under any judge (Kendall τ ≥ 0.93). No model moves by more than one rank, and Gemini 3.1 Pro ranks first under every weighting. (Figure 6 in Appendix F) Contradiction penalty: Under the primary judge, the ranking is unchanged for −3.2 ≤ μ ≤ −1.75. Beyond this band, every change in the ranking goes in the direction the penalty is designed for: harsher penalties favor models with a lower contradiction rate. Partial credit: Varying λ over its full range, 0 ≤ λ ≤ 1 in steps of 0.05, changes the order of at most 2 of the 91 model pairs under any judge (Kendall’s τ ≥ 0.96). The ranking is identical to that under the paper’s λ = 0.5 for every λ ≥ 0.3 under the Gemini judge, every λ ≥ 0.45 under Claude, and every 0.15 ≤ λ ≤ 0.65 under Qwen3-VL 235B.

W5: rubric exhaustiveness

While we acknowledge that it is impossible to make the rubric exhaustive, covering every single aspect that a good answer may contain, we would like to distinguish two cases: annotation errors and limitations of the framework.

  • Strict Lists and Processes: The annotator is expected to enumerate every required item or step. Missing one is inevitable in practice, but it is an annotation error rather than an inherent limitation of our method, and our multi-round revision is designed to catch such omissions.
  • Flexible Lists: We concede that exhaustive coverage is much harder, often impossible. We do not claim that GAMUT fully solves open-ended answer spaces, but that it handles them substantially better than prior work. Existing rubric methods represent each reference fact as an independent required item, so an open set is either written as individually required items or collapsed into a single vague item. A Flexible List instead credits any sufficient subset through a coverage threshold, with graded partial credit. Framing the content explicitly as a list also changes how annotators work: they set out to enumerate the pool rather than record whichever items the draft happened to mention, and we observed noticeably more accurate and more complete pools as a result. In our first annotation round, which used flat rubrics, only 26.7% of rubrics graded coverage of an enumerated set; with meta-rubrics, 86.4% do. Extending credit to valid items outside the pool, e.g., by having the judge verify them against the web, is a natural next step.

Q2: rubric creation inter-annotator agreement

For Rubric Creation, IAA was not feasible given the complexity and length of the task. Each rubric item involves several open-ended decisions (which of the three tiers it belongs to, how it is worded, which evidence supports it), and the same fact can be validly placed in different tiers or phrased in different ways, so agreement between two annotators is hard to define and expensive to obtain. We instead rely on multi-round annotation, where each round reviews and revises the previous one (two human rounds interleaved with LLM refinement), together with the 10% audits.

Dear Reviewer uHRT,

Thank you again for your detailed review. With the discussion period now about halfway through, we wanted to gently follow up on our response. We added the experiment you identified as most important, a direct comparison of grading the meta-rubric versus the compiled checklist (App. G), along with sensitivity analyses for the tier weights, contradiction penalty, partial credit, and judge prompt (App. F). We would be grateful for your thoughts on whether these address your concerns, and we are happy to answer any remaining questions.

Dear Reviewer uHRT,

As the discussion period ends tomorrow, we wanted to follow up once more on our response. In brief: the experiment you identified as most important is in the new Appendix G, where compiled rubrics are graded more consistently than the meta-rubric graded directly, even by frontier judges; and the ranking is stable under alternative tier weights, contradiction penalties, and partial credit (App. F).

We also ran one new analysis related to your W5 that our earlier reply did not include. To measure how much valid content outside a rubric's candidate pool affects the evaluation, we took every coverage-threshold check a response did not fully meet (100 questions × 14 models) and used web-verified claims to find valid items the response mentions outside the rubric, judged independently by two frontier models (GPT-6 Astra and Claude Fable 5.1). Crediting these items raises GAMUT scores by 0.9 points on average (at most 1.3 for any model) and leaves the model ranking unchanged. The limitation is real, but its impact on the current evaluation is small.

We hope our response and the additional experiments addressed your concerns and you would consider changing your recommendation. If any question remains, we would be glad to address it in the time left. Thank you again for your time and detailed review.

Reviewer_pLKxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces a benchmark for factual completeness in long-form answers in the context of everyday deep research. The benchmark comprises 1,813 questions grounded in real-world images across 10 domains. It adopts a two-level evaluation rubric consisting of a meta-rubric layer and a checklist layer, with items assigned to three importance tiers. The authors evaluate 14 models on the benchmark.

Critical strengths
  1. The paper categorizes checklist items into three importance tiers, enabling a more fine-grained evaluation of long-form answers.
  2. The two-level scoring framework separates the structured description of answer requirements from the actual scoring procedure, providing a clear approach to evaluation with a sound design rationale.
Critical weaknesses
  1. Although the two-level scoring framework has a sound design rationale, the additional benefits of first constructing meta-rubrics and then compiling them into checklists, compared with directly authoring checklist items with equivalent expressive power, have not been directly validated.
  2. Human review in the paper primarily focuses on the construction of questions and rubrics, but agreement between model-based and human evaluations is not reported.
  3. A Flexible List allows responses to cover different subsets of the candidate pool, but content outside that pool receives no coverage credit. Therefore, its support for open-ended answer spaces remains constrained by the predefined candidate pool.
  4. During human revision, annotators consult the meta-rubric while directly modifying the compiled checklist. However, the paper does not adequately explain whether these changes are also reflected in the corresponding higher-level meta-rubric or whether the final checklist items can be regenerated from it. This weakens the verifiability of the claimed “deterministic compilation.”
Questions for authors
  1. What additional benefits does the framework of first constructing meta-rubrics and then compiling them into checklists offer over directly authoring checklist items with equivalent expressive power?
  2. After annotators consult the meta-rubric and directly modify the compiled checklist, is the corresponding meta-rubric updated to reflect those changes?
4 comments

We thank the reviewer for a clear and constructive review, and for recognizing the sound design rationale of the two-level framework. The reviewer raises valid points about validating its benefits and about agreement with humans, and we added new experiments for both.

W1: Evaluating meta-rubrics directly

We thank the reviewer for this suggestion. We added this experiment to the new Appendix G. Grading the compiled binary rubric is more stable than grading the meta-rubric directly. Even frontier judges are significantly more consistent across repeated runs with compiled rubrics, and the advantage widens with a less capable judge, both across runs and across judges (Table 8 in Appendix G).

W2: human-model agreement

During the rebuttal, we asked two annotators to independently grade 100 (response, rubric check) pairs with the same four-way verdicts, without seeing any judge's verdict. Frontier judges agree with each other significantly more than the two annotators do (85% vs. 66%, p = 0.003), and agree with each annotator slightly more often than the annotators agree with each other (70% vs. 66%; App. E). This matches our experience during development: frontier models perform this mechanical checking more consistently than human graders. Human expertise in GAMUT is instead spent where it matters most: every question and rubric was verified by expert annotators against web evidence (Section 4).

W3: Open-ended lists

We agree with the reviewer and view this as valuable future work. We do not claim that GAMUT fully solves open-ended answer spaces, but that it handles them substantially better than prior work. Existing rubric methods represent each reference fact as an independent required item, so an open set is either written as individually required items or collapsed into a single vague item. A Flexible List instead credits any sufficient subset through a coverage threshold, with graded partial credit. Framing the content explicitly as a list also changes how annotators work: they set out to enumerate the pool rather than record whichever items the draft happened to mention, and we observed noticeably more accurate and more complete pools as a result. In our first annotation round, which used flat rubrics, only 26.7% of rubrics graded coverage of an enumerated set; with meta-rubrics, 86.4% do. Extending credit to valid items outside the pool, e.g., by having the judge verify them against the web, is a natural next step.

W4 & Q2: Annotating the compiled rubrics

We thank the reviewer for raising this; it is a design choice we debated ourselves. During revision, annotators reason in the meta-rubric space (checking a flexible list for missing items, a process for misordered steps, a fact for its tier) but record their changes directly on the binary rubric, effectively performing the compilation step themselves. We chose this because the binary rubric is what is used for scoring, and we wanted every final check to be human-verified without a second annotation task to re-verify a recompiled checklist. The meta-rubrics are not updated during annotation. We concede that this is not ideal, but unfortunately we did not have enough time or resources to have the annotators revise the meta-rubrics first, and separately verify the resulting binary rubrics.

Q1: directly authoring checklist items

We agree that, technically, one can directly author exactly the same set of binary checks. Our meta-rubric mainly provides a language, or a thinking framework, to describe the complex structures that open-ended answers require, such as open-ended lists with tiered coverage thresholds and processes whose order matters. We arrived at this design empirically. We initially annotated with flat rubrics and found many issues, especially around open-ended lists and processes. In that first round, only 27% of questions had any coverage check and none had an order check; with meta-rubrics, 86% and 17% do. To the best of our knowledge, no existing dataset systematically models elements such as tiered coverage threshold rubrics or sequence order rubrics, even though flat checklists could in principle express them.

Dear Reviewer pLKx,

Thank you again for your constructive review. With the discussion period now about halfway through, we wanted to gently follow up on our response. We added a direct comparison of grading the meta-rubric versus the compiled checklist (App. G), a human-judge agreement study (App. E), and clarifications on open-ended lists and on how annotators revise the compiled rubrics. We would appreciate knowing whether these address your concerns, and we are happy to provide any further details.

The authors’ additional experiments demonstrate that using binary checklists can improve scoring stability. The human-AIagreement analysis and the explanation of the role of Meta-rubric authoring have also alleviated some of my concerns. However, I still have some reservations. Although Flexible List allows different valid subsets within the candidate pool to serve as references, valid content outside this pool still cannot receive coverage credit. Identifying this issue as future work clarifies the method’s boundaries, but does not eliminate the impact of this limitation on the current evaluation.

Regarding the manual revision process, I understand the practical rationale for prioritizing verification of the final scoring items. However, because these manual revisions were not propagated back to the Meta-rubric, they may increase the maintenance burden for future benchmark updates and recompilation and may also weaken the structural advantages of the two-level framework. I would appreciate further clarification from the authors on these concerns.

AuthorsOpenReview ↗

Follow-up response to Reviewer pLKx

We thank the reviewer for the follow-up, and are glad the new experiments addressed several concerns. We ran two further analyses on the remaining two.

Flexible Lists.

We measured the impact directly. On 100 questions × 14 models, we took every coverage-threshold check a response did not fully meet (1,914 of 3,090) and used the web-verified claims from our fact-checking study (App. H) to find valid items the response named outside the pool, judged independently by two frontier models (GPT-6 Astra and Claude Fable 5.1). Responses name a valid out-of-pool item in 13% of threshold checks, and crediting these items would change 6% of threshold checks to meets. The effect on the evaluation is small and uniform: GAMUT scores rise by 0.9 points on average (at most 1.3 for any model), and the model ranking is unchanged. In a more generous variant, where an item counts if either model accepts it and any valid out-of-pool item earns at least partial credit, scores rise by 1.9 points on average and only one pair of models 0.6 points apart swaps (τ = 0.98). The limitation is real, but its impact on the current evaluation is limited.

Revisions and the meta-rubric.

We thank the reviewer for pressing on this point. It led us to audit the annotators' revisions, and we found a minor yet systematic annotation issue rather than a limitation of the framework (new App. I, Table 10).

We use two frontier models (GPT-6 Astra and Claude Fable 5.1) to check every revision made in the final round of human annotation, each judging independently whether the revision can be carried back to the meta-rubric by a direct edit. The two agree on 98% of revisions. Across the benchmark, about 6% of all checks are affected by revisions that both models judge "uncompilable": no direct edit of the meta-rubric compiles to the revised check. We report the breakdown on the 546 questions (of the 721 revised) where the two models agree on every revision, including the category of each uncompilable tier movement. In these questions, most (86%) uncompilable revisions are of a single kind: moving an individual check to a different importance tier, and a further 7% add a check in a tier the conversion rules do not produce. Together, 93% of uncompilable revisions concern only the importance tier of a check, so the structure the meta-rubric provides (lists, coverage thresholds, processes) is intact.

We then use the same two frontier models to analyze every tier movement. Of the 872 uncompilable tier movements, 67% are unjustified: the tier compiled from the meta-rubric was correct, and the annotator’s revision was wrong. The other 33% are reasonable changes but fail to follow the annotation guideline: 31% call for splitting or restructuring a list (e.g., splitting a Flexible List into a Strict List and a Flexible List when there are both essential and optional items), and 2% moved individual checks where the whole meta-rubric should have moved. We found no movement that the framework cannot express.

To quantify the impact of these uncompilable revisions, we rebuilt the rubrics for these 546 questions by compiling a corrected revision of the meta-rubric, and re-scored all 14 models. GAMUT scores rise by 0.6 points on average, per-response scores correlate at r = 0.99 with the released ones, and the model ranking is unchanged.

We will release the meta-rubrics for all questions together with this audit, including corrected meta-rubrics for the questions analyzed above, so that future updates can be made on the meta-rubric and recompiled.

Reviewer_jEKsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GAMUT, a multimodal benchmark for factual completeness in everyday deep-research questions. Its central design is a two-level rubric: a structured meta-rubric represents open-ended sets, ordered processes, relationships, and importance tiers, and deterministic compilation converts it into flat binary checks for LLM-based grading. The benchmark contains 1,813 image-grounded questions across 10 domains, with evidence-backed rubrics reviewed by human annotators. The authors evaluate 14 frontier and open-weight models and report that the benchmark is challenging and produces broadly stable model rankings across three LLM judges. A text-only variant is also provided.

Critical strengths
  1. The paper targets a real gap in long-form factuality evaluation: existing approaches often measure precision or coverage of flat atomic facts, while open-ended answers also involve alternative options, ordering, relationships, and importance. The proposed meta-rubric representation addresses this gap in a concrete and interpretable way.

  2. The benchmark construction and evaluation are substantial. The dataset covers 1,813 questions, 10 domains, 14 models, multiple judges, and a text-only control. The reported ranking stability across judges and the detailed breakdown by importance tier make the benchmark useful for diagnosing model behavior rather than only producing one aggregate number.

Critical weaknesses
  1. GAMUT measures rubric coverage and contradiction but does not evaluate claims outside the rubric. A model could add unsupported or hallucinated content without being penalized, so the score is not a complete factuality measure. This limitation directly affects the interpretation of the benchmark as an evaluation of factual completeness in free-form answers.

  2. There is substantial evaluator circularity: Gemini 3.1 Pro is used in question and rubric construction and is also the primary judge. Agreement with Claude and Qwen3-VL is useful, but it is still LLM-to-LLM agreement and does not establish independent human validity. The paper needs stronger evidence that the scores measure human-perceived factual completeness rather than compatibility with the construction/judge model.

Questions for authors
  1. On an independently human-scored subset of model answers, what are the correlations and rank-order agreement between GAMUT and human factual-completeness judgments? Please separate agreement on rubric coverage from agreement on unsupported claims outside the rubric.

  2. How sensitive are model rankings to the global weights (0.6/0.3/0.1), contradiction penalty, and judge prompt? In particular, do the main conclusions remain under reasonable alternative weightings?

  3. How were the annotators selected and what was their independent inter-annotator agreement during rubric construction? The current evidence mainly establishes agreement in constructing the rubric, not validity of the final score on generated answers.

4 comments

We thank the reviewer for a thoughtful and constructive review, and for recognizing that GAMUT addresses a real gap in long-form factuality evaluation. The concerns raised are valid, and we ran new experiments on agreement with humans, sensitivity to scoring choices, and errors outside the rubric.

W1: Off-rubric hallucinations

We agree that GAMUT is a completeness (recall) metric and does not score every claim in a response, which we acknowledge in the Limitations section.

GAMUT penalizes contradictions, and it's less likely for more important claims to be off-rubric. To quantify what falls outside the rubric, we ran a search-grounded fact-check using frontier models following the GPT-5 system card protocol on 180 responses (30 questions × 6 models): GPT-6 Astra extracts claims, Claude Fable 5.1 verifies each one with web search, and GPT-6 Astra maps false claims to the rubric. Excluding judge errors, about 71% of material false claims are penalized. Among the remaining 29%, nearly half (14%) are “off-topic”: content too peripheral to belong in a complete answer. GAMUT neither rewards such statements when they are true nor penalizes them when false. Only 2.5% of material false claims are unpenalized Answer-Critical claims.

Overall, the gap is real but limited, and GAMUT can be paired with a precision metric to cover it. We demonstrate this in the new Appendix H, combining GAMUT with the a search-grounded precision metric on 100 questions for all 14 models. The two metrics capture different aspects of factuality. Weaker models score low on both, but among proprietary models the two are uncorrelated (τ = 0.00): some models are more complete, others more precise; GPT-5.4, for example, is the most precise model but ranks sixth on GAMUT. They also weigh hallucinations differently. GAMUT penalizes a false claim by how central it is to the answer, while precision counts every claim equally: an answer that misidentifies the pictured plant still has 83% of its claims verified as true, because most are true of the wrong plant, but scores −63% on GAMUT. Combining the two as F1 leaves the ranking nearly unchanged (τ = 0.98 with GAMUT), consistent with a limited gap, while precision refines the comparison among frontier models. Which metric matters more depends on the use case.

W2 & Q1: Human model agreement

We agree that validating LLM judges against humans is important, but respectfully disagree that our evaluation is circular. Our conclusions do not depend on the judge. Claude Opus 4.8 produces the same model ranking as the primary judge (Gemini 3.1 Pro), and Qwen3-VL 235B also preserves the same broad ordering (Table 2). Neither frontier judge favors its own model: each scores its own model within 0.3 points of the other judge's score (Section 6.1). For this rebuttal, we further add a fourth judge, GPT-6 Astra, which was never involved in GAMUT's construction (App. E). Its ranking matches Gemini's and Claude's except for one adjacent pair (τ = 0.98). Gemini 3.1 Pro ranks first under all four judges.

During the rebuttal, we asked two annotators to independently grade 100 (response, rubric check) pairs with the same four-way verdicts, without seeing any judge's verdict. Frontier judges agree with each other significantly more than the two annotators do (85% vs. 66% of checks, p = 0.003), and agree with each annotator slightly more often than the annotators agree with each other (70% vs. 66%; App. E). This matches our experience during development: frontier models perform this mechanical checking more consistently than human graders. Human expertise in GAMUT is instead spent where it matters most: every question and rubric was verified by expert annotators against web evidence (Section 4).

Q2: Hyper-parameter sensitivity

We thank the reviewer for raising this question. It is an experiment that could have strengthened our paper. We re-scored the existing verdicts of all three judges under alternative hyper-parameters (new Appendix F), and show that model rankings are highly stable under all of these choices.

  • Tier weights: Across all 884 weightings on a 0.01 grid that keep the tier order (w_AC ≥ w_V ≥ w_C), from Answer-Critical only (1, 0, 0) to equal weights (1/3, 1/3, 1/3), at most 3 of the 91 model pairs change order under any judge (Kendall τ ≥ 0.93). No model moves by more than one rank, and Gemini 3.1 Pro ranks first under every weighting. (Figure 6 in Appendix F)
  • Contradiction penalty: Under the primary judge, the ranking is unchanged for −3.2 ≤ μ ≤ −1.75. Beyond this band, every change in the ranking goes in the direction the penalty is designed for: harsher penalties favor models with a lower contradiction rate.
  • Partial credit: Varying λ over its full range, 0 ≤ λ ≤ 1 in steps of 0.05, changes the order of at most 2 of the 91 model pairs under any judge (Kendall’s τ ≥ 0.96). The ranking is identical to that under the paper’s λ = 0.5 for every λ ≥ 0.3 under the Gemini judge, every λ ≥ 0.45 under Claude, and every 0.15 ≤ λ ≤ 0.65 under Qwen3-VL 235B.
  • Judge prompt: We re-judged a random 100-question subset (all 14 models) with Gemini 3.1 Pro using a paraphrased prompt and a minimal prompt without guidance or examples (new App. F.4). The paraphrased prompt reproduces the original verdicts on 94% of checks (κ = 0.91), as closely as re-running the original prompt (95%, κ = 0.92), with an identical ranking. Even the minimal prompt agrees on 91% (κ = 0.85, τ = 0.93), a smaller change than switching the judge from Gemini to Claude (87%, κ = 0.80).

Q3:

  • Annotator selection: Annotators qualified through a screening test for each task: scored gold jobs with a 95% pass bar for Question Creation, and a guidelines quiz with a 90% pass bar for Rubric Creation, followed by two training sessions and practice examples (Appendix D). In both tasks, 10% of completed jobs were audited and returned with feedback.
  • Agreement: For Question Creation, each question was reviewed independently by two annotators, with a third adjudicating disagreements. For Rubric Creation, IAA was not feasible given the complexity and length of the task. Each rubric item involves several open-ended decisions (which of the three tiers it belongs to, how it is worded, which evidence supports it), and the same fact can be validly placed in different tiers or phrased in different ways, so agreement between two annotators is hard to define and expensive to obtain. We instead rely on multi-round annotation, where each round reviews and revises the previous one (two human rounds interleaved with LLM refinement), together with the 10% audits.

Dear Reviewer jEKs,

Thank you again for your thoughtful review. With the discussion period now about halfway through, we wanted to gently follow up on our response. To address your main concerns, we added a human-judge agreement study and a fourth, independent judge (App. E), sensitivity analyses for the scoring constants and judge prompt (App. F), and a search-grounded analysis of hallucinations outside the rubric, including a combination of GAMUT with a precision metric (App. H). We would be grateful to hear whether these address your concerns, and we are happy to clarify anything further.

Dear Reviewer jEKs,

As the discussion period ends tomorrow, we wanted to follow up once more on our response. In brief: frontier judges agree with each human annotator as often as the annotators agree with each other (App. E); a fourth judge never involved in building GAMUT reproduces the ranking (τ = 0.98); the ranking is stable under alternative tier weights, contradiction penalties, and judge prompts (App. F); and GAMUT can be combined with a search-grounded precision metric to cover claims outside the rubric (App. H).

We hope our response and the additional experiments addressed your concerns. If any question remains, we would be glad to address it in the time left. Thank you again for your time and thoughtful review.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions

We thank all reviewers for their constructive feedback, and for recognizing the strengths of this work: it targets a real gap in long-form factuality evaluation (jEKs, uHRT); the two-level meta-rubric resolves the tension between expressiveness and grading reliability with a sound and elegant design (pLKx, uHRT); importance tiers enable fine-grained, diagnostic evaluation (jEKs, pLKx); the benchmark construction is substantial and carefully grounded (jEKs, uHRT); and the evaluation is broad and discriminative, with 14 models, multiple judges, and a text-only control (jEKs, uHRT).

We’ve made the following additions and revisions to address commonly raised questions. We will polish the writing and integrate some of these results into the main text in the final version.

Agreement between LLM judges and human annotators (App. E, Table 5).

Two annotators independently graded 100 (response, rubric check) pairs without seeing any judge's verdict. Frontier judges agree with each other significantly more than the two annotators do (85% vs. 66%, p = 0.003), and agree with each annotator slightly more often than the annotators agree with each other (70% vs. 66%). (jEKs W2, Q1, Q3; pLKx W2; uHRT W1)

A fourth, independent judge (App. E, Tables 2 and 4).

We re-grade all 14 models with GPT-6 Astra, a frontier model never involved in GAMUT's construction. It reproduces the Gemini and Claude ranking except for one adjacent pair (τ = 0.98) and agrees with the primary judge on 87% of checks, the same as Gemini - Claude agreement. Gemini 3.1 Pro ranks first under all four judges. (jEKs W2)

Sensitivity to the scoring constants (App. F, Figures 6 and 7, Table 7).

We conduct an ablation on each score constant: tier weights, contradiction penalty μ, partial credit λ, and judge prompt. Our evaluation and the overall ranking are highly stable across all these choices. (jEKs Q2; uHRT W3, W4, Q3)

  • A grid search across 884 tier weights sees at most 3 of 91 model pairs change order (Kendall τ ≥ 0.93), and Gemini 3.1 Pro ranks first under every weighting.
  • Varying partial credit over 0 ≤ λ ≤ 1 changes at most 2 pairs.
  • The ranking is unchanged for −3.2 ≤ μ ≤ −1.75, and beyond that range the contradiction penalty reorders models only in the direction it is designed to: models with a higher contradiction rate rank lower under a harsher μ.
  • We re-grade a 100-question subset with a paraphrased judge prompt, which reproduces the original verdicts on 94% of checks, as closely as re-running the original prompt (95%), with an identical ranking.

Grading the meta-rubric directly (App. G, Table 8).

We grade 2,800 responses (200 questions × 14 models) against both the meta-rubric and its compiled binary rubric, three times with each of three judges. The compiled rubrics are graded more consistently: even frontier judges are significantly more consistent across repeated runs, and the advantage widens for a less capable judge. (pLKx W1; uHRT W1, Q1) </br>

Combining GAMUT with a precision metric (App. H, Table 9).

We pair GAMUT with a search-grounded precision metric (GPT-6 Astra extracts claims; Claude Fable 5.1 verifies each one with web search) on 100 questions for all 14 models. The two capture different aspects of factuality: among proprietary models they are uncorrelated, with some models more complete and others more precise, and they weigh hallucinations differently, as GAMUT penalizes a false claim by how central it is to the answer while precision counts every claim equally. Using a combined F1 score as the final metric leaves the ranking nearly unchanged compared to using GAMUT alone (τ = 0.98 with GAMUT). (jEKs W1, Q1)

(Update Nov 16) Annotator revisions and the meta-rubric (App. I, Table 10).

We audit every revision annotators made to the compiled checklists with two frontier models (GPT-6 Astra and Claude Fable 5.1), each judging independently whether a direct edit of the meta-rubric reproduces it. Revisions that cannot be reproduced ("uncompilable") affect about 6% of all checks, and 93% of them concern only the importance tier of a check. These uncompilable revisions reflect annotator errors, and none is beyond what the framework can express. Rebuilding the rubrics of 546 audited questions from corrected meta-rubrics changes GAMUT scores by 0.6 points on average and leaves the model ranking unchanged. (pLKx W4, Q2)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.