- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper's research problem is the cost of LLMs achieving factuality, and argues that as LLMs clusters at the top of the factuality leaderboards, the cost separates them apart. Thus, the paper introduces MAS-HQ and Q-score to measure the quality-efficiency ability of frontier LLMs. They also introduces Q-Agent, an agent that decides when to revise its worst summary. Most of the experiments run on news summarization and Table.8 for SimpleQA. Results are mostly from A/B matches between two copies of Q-Agent: A processes passages in ascending order and B in reverse, and each observes a snapshot of the other’s stats whenever the other performs a review.
Critical strengths
- Study of the cost of factuality is an important and interesting research problem.
- Full prompts and pseudocode are provided in the appendix, which helps transparency.
- The figures, tables and layout are clean and professionally presented.
Critical weaknesses
- Differences between competing agents are really tiny on both H-score and Q-score: H-Score gaps are ≈0.003 and Q-Score gaps are mostly <0.01, driven almost entirely by small cost differences between near-identical runs.
- P is never clearly defined, and the cost weight β is applied inconsistently. Although β = 0.01 is stated, the tables' results imply β 1 1 (e.g., Table 4: Q = H − P, 0.9103 − 0.3886 = 0.5217), while SimpleQA (Table 8) is consistent with β = 0.01 (Q ≈ H − 0.004). Even the β = 0 rows in Table 4 subtract the full penalty.
- In most experiments, A and B are identical systems differing only in passage order, and outcomes appear driven by processing position, not by agent or model behavior.
- Simple baselines like single-pass summarization without review are missing. The paper compares the Q-agent's A/B setup in most experiments, without doing other baseline experiments that might be more interesting to know.
Questions for authors
- Please give the exact definition of P (how the four costs are normalized, combined and weighted, and whether it is per passage)
- Most experiments compare two copies of Q-Agent that differ only in processing order. What does a win indicate about a system?
- What value of β was used for each table? The summarization results appear consistent with β = 1, SimpleQA with β = 0.01, and the β = 0 rows in Table 4 still subtract the penalty.”
- Please provide more baselines, e.g., single-pass summarization without review, and always-review baselines, scored under the same protocol.