- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces GAMUT, a benchmark of 1,813 image-grounded "everyday deep research" questions (images from CRAG-MM, 10 domains) targeting factual completeness (recall) of long-form answers rather than precision. Its central idea is a two-level meta-rubric: an expressive structured rubric authored at construction time (types: Simple Knowledge, Strict/Flexible List, Process, Relationship; plus meta-insights and three importance tiers — Answer-Critical, Valuable, Context) that is deterministically compiled into a flat binary checklist for LLM-judge scoring. Scoring uses four verdicts (meets / partially meets / missing / contradicts), penalizing a contradiction more heavily than an omission, and combines the three tiers with fixed global weights (0.6 / 0.3 / 0.1). Fourteen models are evaluated with three judges; scores span 5.1%–58.7%. A text-only variant (1,806 questions) isolates the cost of visual identification.
Critical strengths
1.Well-posed conceptual problem. The precision-vs-recall framing and the "expressiveness vs. grading-reliability" tension are genuine and clearly articulated; a flat one-fact-per-check rubric truly cannot represent open sets, ordering, or relative importance. 2.Elegant two-level design. Authoring in the structured space and grading in the compiled binary space is a clean resolution, and the compilation rules are concrete and auditable. 3.Careful construction and grounding: cited evidence snippets (97%/94% coverage), multi-round human revision, and entity/question corrections. 4.Broad, discriminative evaluation: 14 models, 3 judges, judge-agreement analysis, per-tier and verdict breakdowns, and a text-only control.
Critical weaknesses
1.The central claim is asserted, not tested. The thesis is that the two-level design inherits low grading variance while retaining expressiveness, yet there is no ablation comparing grading against the structured meta-rubric holistically vs. grading the compiled binary checklist (e.g., judge variance, inter-judge agreement, correlation with human completeness judgments). Cross-judge rank agreement is not the same evidence. This is the most important missing experiment.
2.Multimodality is thin. The visual channel acts almost entirely as an entity-identification gate: the text-only variant raises every model's score by 7–22 points. The benchmark is thus closer to a factual-recall test with a perception front-end than to deep multimodal reasoning; the title/framing overstate the multimodal contribution.
3."Partially meets" reintroduces the subjectivity the design avoids. The motivation is that judges are more reliable on binary items, yet the four-way verdict asks for a graded judgment again, with no reliability analysis specific to the partial/contradicts verdicts.
4.Arbitrary scoring constants with no sensitivity analysis. The partial-credit weight, the contradiction penalty, the tier weights, and the decision to also penalize contradictions in the denominator are all justified only by intuition; "roughly a passing score" is undefined. Rank stability under perturbation should be shown.
5.Rubric non-exhaustiveness penalizes valid alternatives. For Strict Lists and Processes, a correct fact/step outside the enumerated set is scored as missing or (worse) contradicting; the asymmetric contradiction penalty amplifies this risk.
Questions for authors
1.Can you provide a direct variance/agreement comparison between structured-rubric grading and compiled-binary grading on a subset? 2.What is the inter-annotator agreement for rubric revision (Appendix D is referenced but absent from the provided text)? 3.How sensitive are the rankings to the contradiction penalty and the tier weights?
4 comments
W5: rubric exhaustiveness
While we acknowledge that it is impossible to make the rubric exhaustive, covering every single aspect that a good answer may contain, we would like to distinguish two cases: annotation errors and limitations of the framework.
- Strict Lists and Processes: The annotator is expected to enumerate every required item or step. Missing one is inevitable in practice, but it is an annotation error rather than an inherent limitation of our method, and our multi-round revision is designed to catch such omissions.
- Flexible Lists: We concede that exhaustive coverage is much harder, often impossible. We do not claim that GAMUT fully solves open-ended answer spaces, but that it handles them substantially better than prior work. Existing rubric methods represent each reference fact as an independent required item, so an open set is either written as individually required items or collapsed into a single vague item. A Flexible List instead credits any sufficient subset through a coverage threshold, with graded partial credit. Framing the content explicitly as a list also changes how annotators work: they set out to enumerate the pool rather than record whichever items the draft happened to mention, and we observed noticeably more accurate and more complete pools as a result. In our first annotation round, which used flat rubrics, only 26.7% of rubrics graded coverage of an enumerated set; with meta-rubrics, 86.4% do. Extending credit to valid items outside the pool, e.g., by having the judge verify them against the web, is a natural next step.
Q2: rubric creation inter-annotator agreement
For Rubric Creation, IAA was not feasible given the complexity and length of the task. Each rubric item involves several open-ended decisions (which of the three tiers it belongs to, how it is worded, which evidence supports it), and the same fact can be validly placed in different tiers or phrased in different ways, so agreement between two annotators is hard to define and expensive to obtain. We instead rely on multi-round annotation, where each round reviews and revises the previous one (two human rounds interleaved with LLM refinement), together with the 10% audits.
Dear Reviewer uHRT,
Thank you again for your detailed review. With the discussion period now about halfway through, we wanted to gently follow up on our response. We added the experiment you identified as most important, a direct comparison of grading the meta-rubric versus the compiled checklist (App. G), along with sensitivity analyses for the tier weights, contradiction penalty, partial credit, and judge prompt (App. F). We would be grateful for your thoughts on whether these address your concerns, and we are happy to answer any remaining questions.
Dear Reviewer uHRT,
As the discussion period ends tomorrow, we wanted to follow up once more on our response. In brief: the experiment you identified as most important is in the new Appendix G, where compiled rubrics are graded more consistently than the meta-rubric graded directly, even by frontier judges; and the ranking is stable under alternative tier weights, contradiction penalties, and partial credit (App. F).
We also ran one new analysis related to your W5 that our earlier reply did not include. To measure how much valid content outside a rubric's candidate pool affects the evaluation, we took every coverage-threshold check a response did not fully meet (100 questions × 14 models) and used web-verified claims to find valid items the response mentions outside the rubric, judged independently by two frontier models (GPT-6 Astra and Claude Fable 5.1). Crediting these items raises GAMUT scores by 0.9 points on average (at most 1.3 for any model) and leaves the model ranking unchanged. The limitation is real, but its impact on the current evaluation is small.
We hope our response and the additional experiments addressed your concerns and you would consider changing your recommendation. If any question remains, we would be glad to address it in the time left. Thank you again for your time and detailed review.
We thank the reviewer for a detailed and constructive review, and for recognizing both the problem and the two-level design. We agree that directly comparing grading the meta-rubric with grading the compiled rubric was the most important missing experiment. It has been added in the revision, along with the hyperparameter sensitivity analyses.
W1 & Q1: Evaluating meta-rubrics directly
We thank the reviewer for this suggestion. We added this experiment to the new Appendix G. Grading the compiled binary rubric is more stable than grading the meta-rubric directly. Even frontier judges are significantly more consistent across repeated runs with compiled rubrics, and the advantage widens with a less capable judge, both across runs and across judges (Table 8 in Appendix G).
W2: multimodality
We agree that GAMUT does not target deep multimodal reasoning, and we do not claim it does: the title and framing are about factual completeness, and the paper states that GAMUT "is multimodal-first, [but] its evaluation framework is modality-agnostic." The visual signal plays the role it plays in real use. Users point a camera at something and ask about it, so recognizing the subject is the first step of answering, and our results show it is a substantial bottleneck: the subject-identification check is contradicted for 11–38% of questions across models. The text-only variant isolates this cost, so the benchmark measures knowledge completeness rather than perception alone. Our contribution is therefore twofold: a multimodal benchmark for everyday deep-research questions, where the image is how the question is grounded, and a modality-agnostic evaluation framework, which we demonstrate on the released text-only variant. We will tighten any wording that suggests multimodal reasoning beyond grounding.
W3 & W4 & Q3: Hyperparameter sensitivity
We thank the reviewer for raising this question. It is an experiment that could have strengthened our paper. We re-scored the existing verdicts of all three judges under alternative hyper-parameters (new Appendix F), and show that model rankings are highly stable under all of these choices. Tier weights: Across all 884 weightings on a 0.01 grid that keep the tier order (w_AC ≥ w_V ≥ w_C), from Answer-Critical only (1, 0, 0) to equal weights (1/3, 1/3, 1/3), at most 3 of the 91 model pairs change order under any judge (Kendall τ ≥ 0.93). No model moves by more than one rank, and Gemini 3.1 Pro ranks first under every weighting. (Figure 6 in Appendix F) Contradiction penalty: Under the primary judge, the ranking is unchanged for −3.2 ≤ μ ≤ −1.75. Beyond this band, every change in the ranking goes in the direction the penalty is designed for: harsher penalties favor models with a lower contradiction rate. Partial credit: Varying λ over its full range, 0 ≤ λ ≤ 1 in steps of 0.05, changes the order of at most 2 of the 91 model pairs under any judge (Kendall’s τ ≥ 0.96). The ranking is identical to that under the paper’s λ = 0.5 for every λ ≥ 0.3 under the Gemini judge, every λ ≥ 0.45 under Claude, and every 0.15 ≤ λ ≤ 0.65 under Qwen3-VL 235B.