acceptodds
Under review as a conference paper at ICLR 2027

Beyond Correctness: The Maximal Supported Answer

Abstract

Answer correctness is usually evaluated against a single target, even when the available evidence licenses several answers at different resolutions. A model that says only "in the twentieth century" despite decisive evidence for 1994 is correct but uninformative; a model that says 1994 from decade-level evidence is informative but unsupported. We formalize this tension through the maximal supported answer: the most specific answer whose denotation contains every world compatible with the supplied evidence. In support-complete answer lattices, this target is unique and moves monotonically toward specificity as evidence is refined. We instantiate the framework in MaxSupport, a paired evidence-ladder benchmark spanning time, geography, numerical intervals, entity taxonomies, and source attribution. We study LatticeCal as a calibrated non-oracle instantiation that estimates candidate-level support risk and chooses the deepest feasible node without accessing the denotational oracle at test time. In an author-constructed synthetic evaluation across five semantic spaces, LatticeCal reaches 64.5 exact-MSA with a 4.2% support-violation rate. CSS remains slightly stronger on geography and attribution, while HSC leads on taxonomy, showing that answer-space geometry determines which form of calibrated backoff works best. Paired additions and removals further reveal whether resolution tracks evidence rather than parametric recall, and claimwise risk allocation extends the framework to long responses.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, evidence grounding, answer specificity, selective prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_BqCvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper says "correct" is the wrong yardstick for answering from a document. A model can say something that is true in the real world but that the document never supported. That is overclaiming. Or it can hedge and stay vaguer than the document allows. That is underclaiming. Exact match scoring rewards lucky guesses and punishes neither. The authors define a better target: the Maximal Supported Answer. It is the most specific answer that is still true in every world the evidence leaves open. If the document only says "the 1990s," then "the 1990s" is right. The true year 1994 is an overclaim. "Twentieth century" is an underclaim. They prove this target is unique. It sharpens as you add evidence and backs off as you remove it. They build a synthetic benchmark that adds, deletes, and fakes evidence to test whether a model tracks the target. They also propose a selection policy that picks how specific to be, under a calibrated risk budget, with no access to the answer.

Critical strengths

1/The core reframing is genuinely clarifying. Pulling apart world true, evidence supported, and maximally supported is a clean idea. The paired ladders (additions, deletions, placebos) test whether a model really responds to evidence, not just recalls the answer. 2/The theory is tidy and honest. Uniqueness rests on support completeness. The paper openly shows a small case where an incomparable candidate space breaks it. I appreciated that. 3/Good model breadth. The frozen generator plus calibrated scorer design is deployable without fine tuning. The risk control carries through to the multi claim setting. 3/On novelty: the distinctive move is conditioning the target on external evidence, not on model confidence. The denotational definition of the optimum is a real step beyond the closest relatives. Those relatives include conformal factuality back off, hierarchical selective classification, and conformal linguistic calibration. The contribution holds up. It just overlaps more than the paper admits (see weaknesses).

Critical weaknesses

1/The evaluation is fully synthetic, and this is disqualifying for the paper's central claim. Every headline result rests on clean, prebuilt hierarchies the authors constructed. There is no natural data at all. So we cannot tell whether the method solves a real problem or a problem the benchmark was built to let it solve. For a paper whose contribution is a practical, deployable policy, a single self made dataset is not enough evidence to accept. 2/The target is only well defined when the document is simple, and real documents are not. The method assumes evidence lands on a tidy rung of a known ladder. Real evidence is scattered across sentences, needs multi hop combination, is ambiguous, or does not correspond to any node ("shortly after the merger"). In those cases the Maximal Supported Answer is undefined, and the whole framework has nothing to select. The benchmark quietly assumes this problem away. The authors' own weakest space, source attribution, is exactly where the clean hierarchy breaks, and the numbers drop there, which suggests the issue is structural, not incidental. 3/The method leans entirely on a trained support scorer, and its reliability is never stressed. The oracle gap is mostly a coverage gap, meaning the correct answer is often never even proposed. That points to the generation and scoring pipeline as the real bottleneck, yet the paper does not show how performance degrades as the scorer weakens or as documents get harder. The guarantee is only as sound as that one learned component, and we are given no evidence it holds up.

Questions for authors

1/Can the central result be reproduced on even one natural dataset? Without a real benchmark (HotpotQA, LongFact, or similar), the claim that this resolves the accuracy granularity dilemma in deployment is unsupported. This is the single thing that would most change my assessment. 2/What happens to the target when the evidence does not map to a clean hierarchy node, as in scattered, multi hop, or ambiguous documents? If the Maximal Supported Answer is undefined in those cases, how large is that fraction on realistic data? 3/How do supported accuracy and overclaiming move as the support scorer degrades? Since most of the gap to the oracle is coverage, the paper needs to show the pipeline is robust, not just the selection rule. Source attribution underperforms and is also where uniqueness bends. Is that a scorer failure or a target definition failure? The answer decides whether the framework generalizes beyond clean ladders.

Reviewer_H8EvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to address unsupported specificity and excessive vagueness in evidence-grounded LLM answers. It formalizes the Maximal Supported Answer (MSA) as “the most informative candidate that holds across all possible worlds compatible with the given evidence.” (page 1)

Critical strengths
  1. The formulation separates evidential support from both factual truth and answer informativeness. The example that “answering “1994” is world-true but unsupported” (page 17) makes the limitation of exact-answer correctness concrete and motivates a useful evaluation target.

  2. The paired-intervention design tests responsiveness to evidence rather than only static accuracy. The authors explain, “we hold the query fixed while programmatically injecting evidence additions, evidence deletions, and length-matched placebo texts.” (page 2) This provides controlled tests of whether answer specificity changes when the evidence warrants it.

Critical weaknesses
  1. The experiments do not establish performance when the answer hierarchy must be inferred from real evidence. Appendix G states, “Open-domain systems must induce or retrieve a meaningful answer order.” (page 17), whereas the benchmark supplies structured semantic spaces. I recommend a real-document evaluation with automatically induced candidate hierarchies and human support labels, to measure errors hidden by the supplied structure.

  2. The reported advantage over the closest selection baselines needs uncertainty estimates. The paper describes LATTICECAL as “outperforming HSC (63.9% MSA) and CSS (63.1%).” (page 7), but its 64.5% MSA is only 0.6 and 1.4 percentage points higher, respectively. I recommend family-level paired confidence intervals and risk-matched comparisons across models, to establish whether the improvement is stable and meaningful.

  3. The risk guarantee does not by itself cover the final wording of a deployed answer. Appendix G notes, “Selecting a safe node does not ensure that a later free-form realization stays within its denotation.” (page 17), so supported node selection can still be followed by unsupported generated details. I recommend evaluating the complete selection-to-generation pipeline with constrained realization or a final support check, to establish end-to-end reliability.

Questions for authors

Please see the Weaknesses section.

Reviewer_dDRHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper defines the Maximal Supported Answer (MSA): the most specific candidate answer that is true in every world consistent with the retrieved evidence. When the candidate set is join-closed, the MSA is unique and monotone under evidence refinement. MaxSupport is a synthetic benchmark over five hierarchies with paired evidence additions, deletions and placebos, labelled by a symbolic oracle. LatticeCal closes LLM proposals under hierarchy ancestors and scores them with a trained DeBERTa support scorer. It returns the most specific candidate that passes calibrated depth-specific thresholds. On seven LLMs it reports 64.5% exact MSA and 4.2% support violations (SV), against 28.8% SV for native generation.

Critical strengths
  1. The target is well posed. The paper separates world-true, supported and maximally supported answers, and reports SV together with underclaims and root rate. So neither lucky parametric guesses nor always-abstain policies look good.

  2. The paired add/delete/placebo design with exact oracle labels is a useful test of evidence responsiveness. I do not know another benchmark that scores both overclaiming and underclaiming against an evidence-licensed target.

Critical weaknesses
  1. The aggregation of the main results is unclear and leaves a potentially large inconsistency unresolved. Table 6 gives LatticeCal a mean depth of 2.6 at E2E_2, where level 2 is licensed and the maximum is 3, so at least 60% of E2E_2 outputs violate support. Equal averaging over the four regular rungs would imply about 28% SV. This conflicts with Table 1 only if it uses the same conditions and weights; its description as endpoint results does not establish this. Calibration is also underspecified: Appendix D.2 does not define the weighted upper bound or clearly identify which conditions and negative variants enter each family failure. The reported test SV is compatible with a 5% population-risk guarantee, but the information provided is insufficient to verify its application to the evaluated outcomes.

  2. The empirical support for LatticeCal is limited. Baseline implementation details are missing, so it is unclear whether HSC-style and CSS-style share the proposals, scorer and calibration data. Endpoint MSA gains over these baselines are small; paired gains are larger, but uncertainty estimates and ablations are absent. The ingredients are also known: calibrated back-off to weaker claims [1], possible-world specificity with calibrated nested answer sets [2] (cited in the paper), and Bonferroni selection over a finite grid as in Learn-then-Test [3]. Theorem 1 and Proposition 1 follow in a few lines from join-closure and set inclusion. These antecedents make the empirical benefit of the integration important to establish. The benchmark has useful teaching examples, but no complete generated family with its evidence ladder, interventions and oracle labels, nor templates or code. This limits reproducibility of a central contribution.

[1] C. Mohri, T. Hashimoto. "Language Models with Conformal Factuality Guarantees." ICML 2024. https://arxiv.org/abs/2402.10978

[2] Z. Jiang, A. Liu, B. Van Durme. "Conformal Linguistic Calibration: Trading-off between Factuality and Specificity." NeurIPS 2025. https://arxiv.org/abs/2502.19110

[3] A. N. Angelopoulos, S. Bates, E. J. Candès, M. I. Jordan, L. Lei. "Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control." Annals of Applied Statistics 2025. https://arxiv.org/abs/2110.01052

Questions for authors
  1. Which rungs, empty conditions and negative variants enter Table 1, with what weights, and how does this aggregation relate to Table 6? Does the Native policy receive evidence?
  2. What is the calibrated family failure event and the weighted upper bound? Please report selected calibration failure counts and root-fallback use by space.
  3. Do HSC-style and CSS-style use the same proposals, scorer and calibration data as LatticeCal? What data and labels train the scorer?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.