acceptodds
Withdrawn from ICLR 2027

The Cost Of Knowing: A Resource-Aware Protocol For Benchmarking Hallucination Beyond Static Leaderboards

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

On standard factuality tasks, frontier models now cluster near the top of the scale. The question is therefore shifting from how factual a system is toward how much compute that factuality costs. Static leaderboards score factuality in isolation and treat compute as free, so they cannot tell a genuinely better system apart from one that simply spends more. Consider a ranking reversal. A brute-force Best-of-4 agent posts the higher raw factuality score (H-Score 0.9169 vs. 0.9103) and would top a static leaderboard, but once cost is counted it is the worse system, losing on Q-Score (0.5169 vs. 0.5217) at roughly four times the tokens and latency, under a reported cost weight whose sensitivity we sweep. So the system that tops a static leaderboard can be the worse one to deploy. To make this trade-off visible, we introduce MAS-HQ (Multi-Agent System Hallucination Quest), a resource- aware evaluation protocol. It wraps any factuality detector and normalizes for cost, and it pits systems against each other rather than scoring them in isolation. The Q-Score measures factuality minus normalized cost under a competitive match. Across summarization and open-domain QA, single-agent baselines drift into resource-heavy over-optimization, while competition elicits more resource-efficient policies. These gains are small but consistent, and stable across 100 trials. The axis stays discriminative for frontier systems (Gemini-2.5-Pro, and GPT-5 in simulated preview) whose raw factuality scores are already bunched near the ceiling. MAS- HQ provides a reproducible way to measure how much a factual answer costs.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
hallucination, benchmark, agent

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_sWBNOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper's research problem is the cost of LLMs achieving factuality, and argues that as LLMs clusters at the top of the factuality leaderboards, the cost separates them apart. Thus, the paper introduces MAS-HQ and Q-score to measure the quality-efficiency ability of frontier LLMs. They also introduces Q-Agent, an agent that decides when to revise its worst summary. Most of the experiments run on news summarization and Table.8 for SimpleQA. Results are mostly from A/B matches between two copies of Q-Agent: A processes passages in ascending order and B in reverse, and each observes a snapshot of the other’s stats whenever the other performs a review.

Critical strengths
  1. Study of the cost of factuality is an important and interesting research problem.
  2. Full prompts and pseudocode are provided in the appendix, which helps transparency.
  3. The figures, tables and layout are clean and professionally presented.
Critical weaknesses
  1. Differences between competing agents are really tiny on both H-score and Q-score: H-Score gaps are ≈0.003 and Q-Score gaps are mostly <0.01, driven almost entirely by small cost differences between near-identical runs.
  2. P is never clearly defined, and the cost weight β is applied inconsistently. Although β = 0.01 is stated, the tables' results imply β 1 1 (e.g., Table 4: Q = H − P, 0.9103 − 0.3886 = 0.5217), while SimpleQA (Table 8) is consistent with β = 0.01 (Q ≈ H − 0.004). Even the β = 0 rows in Table 4 subtract the full penalty.
  3. In most experiments, A and B are identical systems differing only in passage order, and outcomes appear driven by processing position, not by agent or model behavior.
  4. Simple baselines like single-pass summarization without review are missing. The paper compares the Q-agent's A/B setup in most experiments, without doing other baseline experiments that might be more interesting to know.
Questions for authors
  1. Please give the exact definition of P (how the four costs are normalized, combined and weighted, and whether it is per passage)
  2. Most experiments compare two copies of Q-Agent that differ only in processing order. What does a win indicate about a system?
  3. What value of β was used for each table? The summarization results appear consistent with β = 1, SimpleQA with β = 0.01, and the β = 0 rows in Table 4 still subtract the penalty.”
  4. Please provide more baselines, e.g., single-pass summarization without review, and always-review baselines, scored under the same protocol.
Reviewer_6YctOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that conventional factuality benchmarks overlook the computational cost required to achieve high factuality scores. To address this limitation, the authors propose MAS-HQ, a resource-aware evaluation protocol in which two competing LLM-based agents generate and iteratively refine summaries of approximately 810 news passages.

The framework introduces a Q-Score that combines factuality, measured by HHEM-2.1-Open, with a penalty based on token usage, API calls, review iterations, and execution time. These costs are normalized relative to the competing agents within each match.

The central finding is a ranking reversal: although a Best-of-4 baseline achieves the highest factuality score (0.9169), the proposed Q-Agent A reportedly achieves a slightly higher Q-Score (0.5217 vs. 0.5169) at substantially lower computational cost. Additional experiments examine different LLM backbones, cost-penalty coefficients, alternative agent configurations, and evaluation datasets.

Critical strengths

The paper addresses a meaningful limitation of conventional factuality benchmarks: models should be compared not only by their factual accuracy but also by the computational resources required to achieve it. This is particularly relevant as performance differences among frontier models become increasingly small. The proposed cost-aware evaluation perspective is valuable, and the ranking-reversal condition in Eq. (1) offers a simple way to illustrate the trade-off between factuality and efficiency.

Critical weaknesses

The most significant concern is that the reported Q-Scores appear inconsistent with the scoring formula and hyperparameters specified in the paper. Since the claimed ranking reversal is the primary empirical contribution, these inconsistencies directly affect the validity of the main conclusion.

Section 4.1 and Table 3 specify α = 1 and β = 0.01. However, each of the four normalized resource-usage terms is bounded by 1. Consequently, even if the terms are summed, the total cost penalty cannot exceed 0.04 under the stated setting. Nevertheless, the main tables report penalties of approximately 0.4, and every Best-of-N result exhibits an identical penalty of 0.4000.

Several additional inconsistencies reinforce this concern:

  • Incorrect behavior at β = 0: According to the proposed formula, Q-Score should equal H-Score when β = 0. However, Table 4 continues to show penalties of approximately 0.39–0.40, with the lower-factuality agent retaining its advantage. This contradicts both the mathematical definition and the accompanying discussion.
  • Disagreement between tables: Table 13 reports a Q-Score of 0.8715 for a run that appears as 0.5217 in Table 1.
  • Failure to reproduce the claimed ranking: Recomputing the scores using the raw statistics in Table 13 and the stated β = 0.01 yields Q-Agent B outperforming Q-Agent A (0.8732 vs. 0.8714). Furthermore, Best-of-4, when normalized against itself as Table 2 appears to imply, achieves 0.8769, outperforming both Q-Agents.

These discrepancies suggest that the main results may have been calculated using different cost coefficients or normalization procedures from those described in the paper. Importantly, the headline ranking reversal does not hold when the stated configuration is applied to the reported statistics.

Even under joint normalization, the differences among the Q-Agents and Best-of-1 are approximately 0.003 or smaller. These differences are comparable to, or smaller than, the variation introduced by passage ordering alone, which reaches 0.005 at β = 0.01 and 0.027 at β = 0.1 for same-model agents.

Overall, the paper needs to establish that its central finding is reproducible under a clearly defined and consistently applied scoring protocol.


The motivation for cost-aware factuality evaluation is reasonable, but the paper does not adequately position its contribution relative to existing research. Several peer-reviewed works have already explored evaluation beyond task accuracy, including Dynaboard (NeurIPS 2021), HELM (TMLR 2023), AI Agents That Matter (TMLR 2025), and Cost-of-Pass (ICLR 2026). These works are neither cited nor meaningfully discussed.

Against this background, the principal distinction appears to be the combination of match-relative resource normalization and competitive multi-agent evaluation. However, the paper does not clearly establish why competition is necessary for evaluating the factuality–cost trade-off, or what additional insights the proposed framework provides over simpler cost-aware evaluation methods.

Several aspects of the experimental design also require clarification:

  • Cross-match comparability: Because resource costs are normalized relative to an opponent, Q-Scores from different matches are not necessarily directly comparable. This raises concerns about the system rankings presented in Table 2.
  • Unclear model identification: Two of the three frontier-model comparisons in Table 3 involve “GPT-5 (simulated preview),” but the paper does not explain what this model represents or whether the reported results come from actual model executions.
  • Insufficient experimental details: The implementation of Best-of-N, ReAct, SimpleQA, and the three-agent setting is not described clearly enough to assess whether the comparisons use consistent scoring and normalization procedures. The SimpleQA extension also lacks a single-agent baseline.
  • Ambiguous protocol specification: The paper does not explicitly define how the four resource terms are aggregated into the cost penalty. Furthermore, Algorithm 1 describes sequential agent execution, leaving unclear how the optional opponent-observation mechanism operates.

These issues make it difficult to determine which conclusions arise from the proposed competitive framework rather than from implementation choices, normalization effects, or differences in computational budgets.

Questions for authors
  1. Which values of β and which normalization procedures were used to produce each reported table? The released code appears to use β = 0.1, but it is unclear which implementation and configuration generated the reported results.

  2. Could the authors provide precise implementation details for Best-of-N, ReAct, SimpleQA, and the three-agent experiments, including how resource costs are normalized across different configurations?

  3. The review mechanism selects the passage with the lowest HHEM score, and the revising agent is given that score. Do the reported factuality improvements persist when evaluated by an independent factuality judge? How does summary length change after revision?

  4. Does the proposed multi-agent framework outperform a simpler single-agent baseline that receives the same cost-aware instructions and access to its own resource-usage statistics?

Reviewer_HRNbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that static factuality leaderboards treat compute as free and proposes MAS-HQ, a resource-aware evaluation protocol. Two LLM agents (Q-Agents, each made of Policy/Summary/Review/Evaluation sub-agents) summarize the same ~810 Vectara-leaderboard news passages, one in forward and one in reverse order. Factuality (H-Score) is scored by HHEM-2.1-Open. The Q-Score is mean(α·H − β·P), where P aggregates API calls, tokens, review count and wall-clock time, each max-normalized within the match; the main setting is α=1, β=0.01. An optional 'vision mechanism' reveals a snapshot of an agent's state to its opponent whenever it reviews.

The headline result is a ranking reversal: single-agent Best-of-4 has a higher H-Score than Q-Agent A (0.9169 vs 0.9103) but a lower Q-Score (0.5169 vs 0.5217) at ~4x the cost (Table 2). Further experiments cover:

  • same-model matches for five LLMs and cross-model matches;
  • frontier models (Gemini-2.5-Pro and 'GPT-5 (simulated preview)');
  • a β sweep, ReAct substitution, a 3-agent match and SimpleQA;
  • order/vision ablations and a 100-trial repeat of one match. The appendix gives a POSG/PBE formalization.
Critical strengths
  1. A timely question with a simple, explicit scoring rule. Saturating factuality scores do hide the compute spent to reach them, and Eq. (1) (p.4) states the reversal condition β/α > ΔH/ΔP explicitly. The authors are clear about scope: Q is a match-specific payoff, not a standalone model score (p.4), and the POSG is used only as descriptive vocabulary (p.5).

  2. The raw data are reported in enough detail to recompute the scores. Tables 13-14 (pp.18-19) give per-agent API calls, input/output tokens, reviews and runtimes; all prompts (pp.17-18) and Algorithm 1 (p.19) are included; and the experiments span seven LLMs and several protocol variants. Most of the issues below could only be checked because these numbers are available.

Critical weaknesses
  1. Penalty scale. P is built from max-normalized components (p.4) and β = 0.01 (p.6; Table 3 header), so β·P should be at most about 0.04. The reported penalties H − Q are 0.36-0.40, and the most expensive agent always has exactly 0.4000. Using the raw counts in Table 13, the 27 rows of Tables 1, 5, 6, 10, 11 and Fig. 4 are reproduced to within about ±0.0002 by Q = H − 0.1·Σ_k(c_k / max c_k) over the four cost components, i.e., an effective weight 10x the stated one. The paper does not say how P is aggregated, so the sum is my inference, but it fits all 27 rows. For example, for GPT-4o-mini A, Σ = 0.991 + 0.938 + 0.974 + 0.983 = 3.887 and H − Q = 0.3886. Fig. 3 (right, p.7) agrees: the main-table values (A/B mean ≈ 0.52) sit at β = 0.10, while β = 0.01 gives ≈ 0.87. That matches the 0.8715 reported for agent A in Table 13 (p.18), which in turn contradicts the 0.5217 given for the same run in Table 1.

  2. The β sweep does not change the score. In Table 4 (p.6) the penalty is 0.39-0.40 at β = 0, 0.01 and 0.02. At β = 0, Q ≠ H, and the lower-H agent A still wins (0.5267 vs 0.5189), although the text says that at β = 0 "the reversal disappears" (p.7).

  3. Re-scoring at the stated β = 0.01 flips the winner in the GPT-4o-mini match of Table 1 (B 0.8732 vs A 0.8714), in both same-model frontier matches of Table 3 (A wins 0.8912 vs 0.8888 and 0.8864 vs 0.8830), in Table 5, in the first match of Table 6, and in the A-wins of Tables 10-11. The reversals that survive (Gemini-2.0-Flash, Grok-3, Fig. 4) have margins of 0.0008-0.002, about the size of the 0.0009 single-run SD in Table 12. Taking the effective β ≈ 0.1 at face value does not help either: Fig. 3 then shows review lowering Q, from about 0.64 before review to about 0.52 after.

  4. Best-of-N is scored in a different normalization pool. In Table 2 (p.5), H − Q is exactly 0.4000 for Best-of-1, -2 and -4, so each baseline was normalized against itself and its ~4x spend never enters its Q, whereas Q-Agent A's 0.5217 comes from the A-vs-B match. The headline comparison (0.5169 vs 0.5217; Abstract, p.6) therefore compares payoffs from different matches, which p.4 itself says is not meaningful, and it makes Q rise monotonically with N among the baselines. When A and Best-of-4 are normalized together, A wins clearly at either scale (0.9008 vs 0.8769 at β = 0.01; 0.8150 vs 0.5169 at β = 0.1), so the qualitative point probably holds, but it becomes close to true by construction. The informative comparison is against cost-matched Best-of-1, and that one depends on the weight: pairwise, Best-of-1 wins 0.8744 vs 0.8732 at β = 0.01, and A wins at β = 0.1.

Questions for authors
  1. What is the exact formula for P_i (how the four components are combined, and any scaling), and which β was used for each table? Please re-score all tables from the raw logs at the stated β and over a real β range, and explain Table 13's 0.8715, Fig. 3 (right), and the non-zero penalties at β = 0 in Table 4. Do the reversals in Tables 1, 3, 5 and 6 survive?
  2. How is Best-of-N implemented: how are candidates generated and selected (e.g., by HHEM), what does its Review column count, and which agents is it normalized against? Please re-score Table 2 with Best-of-1/2/4 and Q-Agents A/B in one normalization pool.
  3. What does "GPT-5 (simulated preview)" mean? Were these real API calls?
  4. Why are Table 7's Q-Agent rows identical to Table 4's β = 0 rows, why do the ReAct and Q-Agent H-Scores coincide to four decimals, and how do the exact ties in Table 9 arise? Were outputs cached or decoding deterministic?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.