- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper says "correct" is the wrong yardstick for answering from a document. A model can say something that is true in the real world but that the document never supported. That is overclaiming. Or it can hedge and stay vaguer than the document allows. That is underclaiming. Exact match scoring rewards lucky guesses and punishes neither. The authors define a better target: the Maximal Supported Answer. It is the most specific answer that is still true in every world the evidence leaves open. If the document only says "the 1990s," then "the 1990s" is right. The true year 1994 is an overclaim. "Twentieth century" is an underclaim. They prove this target is unique. It sharpens as you add evidence and backs off as you remove it. They build a synthetic benchmark that adds, deletes, and fakes evidence to test whether a model tracks the target. They also propose a selection policy that picks how specific to be, under a calibrated risk budget, with no access to the answer.
Critical strengths
1/The core reframing is genuinely clarifying. Pulling apart world true, evidence supported, and maximally supported is a clean idea. The paired ladders (additions, deletions, placebos) test whether a model really responds to evidence, not just recalls the answer. 2/The theory is tidy and honest. Uniqueness rests on support completeness. The paper openly shows a small case where an incomparable candidate space breaks it. I appreciated that. 3/Good model breadth. The frozen generator plus calibrated scorer design is deployable without fine tuning. The risk control carries through to the multi claim setting. 3/On novelty: the distinctive move is conditioning the target on external evidence, not on model confidence. The denotational definition of the optimum is a real step beyond the closest relatives. Those relatives include conformal factuality back off, hierarchical selective classification, and conformal linguistic calibration. The contribution holds up. It just overlaps more than the paper admits (see weaknesses).
Critical weaknesses
1/The evaluation is fully synthetic, and this is disqualifying for the paper's central claim. Every headline result rests on clean, prebuilt hierarchies the authors constructed. There is no natural data at all. So we cannot tell whether the method solves a real problem or a problem the benchmark was built to let it solve. For a paper whose contribution is a practical, deployable policy, a single self made dataset is not enough evidence to accept. 2/The target is only well defined when the document is simple, and real documents are not. The method assumes evidence lands on a tidy rung of a known ladder. Real evidence is scattered across sentences, needs multi hop combination, is ambiguous, or does not correspond to any node ("shortly after the merger"). In those cases the Maximal Supported Answer is undefined, and the whole framework has nothing to select. The benchmark quietly assumes this problem away. The authors' own weakest space, source attribution, is exactly where the clean hierarchy breaks, and the numbers drop there, which suggests the issue is structural, not incidental. 3/The method leans entirely on a trained support scorer, and its reliability is never stressed. The oracle gap is mostly a coverage gap, meaning the correct answer is often never even proposed. That points to the generation and scoring pipeline as the real bottleneck, yet the paper does not show how performance degrades as the scorer weakens or as documents get harder. The guarantee is only as sound as that one learned component, and we are given no evidence it holds up.
Questions for authors
1/Can the central result be reproduced on even one natural dataset? Without a real benchmark (HotpotQA, LongFact, or similar), the claim that this resolves the accuracy granularity dilemma in deployment is unsupported. This is the single thing that would most change my assessment. 2/What happens to the target when the evidence does not map to a clean hierarchy node, as in scattered, multi hop, or ambiguous documents? If the Maximal Supported Answer is undefined in those cases, how large is that fraction on realistic data? 3/How do supported accuracy and overclaiming move as the support scorer degrades? Since most of the gap to the oracle is coverage, the paper needs to show the pipeline is robust, not just the selection rule. Source attribution underperforms and is also where uniqueness bends. Is that a scorer failure or a target definition failure? The answer decides whether the framework generalizes beyond clean ladders.