acceptodds
Under review as a conference paper at ICLR 2027

Verdicts Without Evidence: Directional Critique Rewrites Abstention

Abstract

In fact verification and multi-agent review, a language model increasingly acts as the final adjudicator. It reads a claim, the retrieved evidence, and a critique written by another party, then returns SUPPORTS, REFUTES, or NOINFO. The evidence is meant to supply the grounds for a verdict; the critique is meant only to flag what the evidence may have missed. Nothing in the task makes that division binding. To see what a critique does when the evidence cannot settle the claim, we replace the evidence with registry metadata and keep NOINFO available. Across 200 claims, three commercial models, and five conditions of 2,400 runs each, we vary only a single critique sentence that recommends a direction. That sentence alone raises unwarranted verdicts from 6/2400 to 380/2400, a rise that holds on each of the three models. The new verdicts do not simply follow the recommendation, since 90 of the 380 contradict it. When we stratify the claims by material, every verdict on synthetic claims is REFUTES whatever direction the critique asks for, while on external claims every verdict follows the recommendation. Because frequency and direction move apart, we report two quantities: the commitment rate, how often the model leaves abstention, and the direction adoption, how often a verdict follows the recommendation. Under that decomposition, a careful-reading prompt reduces the frequency of commitment without changing the direction of the remaining verdicts. A rule stating that critique text is not evidence lowers commitment to 3.04% and reverses the composition of the verdicts that remain. Neither that rule nor a stronger evidence-only convention returns the model to its no-critique baseline. We propose Closed-Evidence Abstention Invariance as the target for abstention-enabled review: a critique mechanism has to be judged on commitment, direction adoption, and accuracy under evidence at once.

Primary area:
foundation or frontier models, including LLMs
Keywords:
fact verification, abstention, LLM-as-a-judge, multi-agent critique, evidence grounding, unwarranted verdicts, evaluation methodology

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_pZMbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper asks whether a single directional critique can push a model out of abstention when the evidence is missing and NOINFO is an allowed answer. The authors replace the evidence with archival metadata that says nothing about whether the claim is true, and then add one sentence that recommends a direction without offering any evidence. The main experiment uses 200 claims, some taken from FEVER and SciFact and some generated from templates, and tests three commercial models under five conditions. The five conditions are no critique, the critique alone, and the critique plus one of three instructions (a careful-reading prompt, a rule that critique text is not evidence, or a stricter rule that only the evidence can decide the verdict). Each condition is run 2,400 times. Without a critique, only 6 of the 2,400 runs return SUPPORTS or REFUTES. With the critique this number rises to 380, and 90 of these verdicts go against the recommended direction. The authors split these verdicts into a commitment rate (how often the model leaves abstention) and direction adoption (how often a verdict follows the recommendation). Comparing the careful-reading prompt with the rule that critique text is not evidence, they find that the prompt lowers commitment but leaves the direction of the remaining verdicts unchanged. The rule lowers commitment from 15.83% to 3.04%, and most of the verdicts that survive oppose the recommendation. The paper also reports a panel that separates the two claim types, a replication of the rule, and an experiment that counts how many errors a critique corrects when the evidence is present. The authors also propose Closed-Evidence Abstention Invariance (CEAI), which requires the answer to stay NOINFO when a critique without evidence is added to an input that lacks evidence.

Critical strengths
  1. The experimental protocol is described clearly. The counts are reported in full and are internally consistent, and every test I recomputed from them matched the paper. This is why I trust the data even though I disagree with how it is interpreted.
  2. All three models show the behavior the paper describes, so the main effect does not depend on a single model.
Critical weaknesses
  1. Two of the paper’s main explanations are contradicted by its own tables. The first is the material explanation. Section 4.3 attributes the 90 opposite-direction verdicts of the main experiment (Table 4) to synthetic claims always getting REFUTES. But in the stratified panel (Appendix C.1, Table 13), a SUPPORTS recommendation on synthetic claims yields 52/580 and 38/578 REFUTES verdicts, which is not significantly above the 35/567 with no critique (p = 0.076 and 0.810 by the authors’ own tests in Appendix C.1). So the critique adds no reliable opposite-direction verdicts there, and the panel cannot explain the ones in the main experiment. The main experiment itself is never split by material (Section 3.3), even though the material type is obvious from the claim text. The effect of the rules is read backwards. Each claim is run equally often under both recommendations, so any verdict unrelated to the recommendation should give a DA of 50% on average. Under the two rules DA is 31.5% and 24.2% (Table 4), and a SUPPORTS recommendation yields more REFUTES verdicts than a REFUTES recommendation does (37 vs. 16 and 31 vs. 8, summed from Table 8). The rules make the model push against the critique, Section 5 and Appendix E.1 say the rules leave only material-driven verdicts, but the numbers say the opposite. This matters because these are the two points the paper relies on to set itself apart from work on sycophancy.

2.The headline effect is measured under an instruction that allows the critique to be used, and it gets small and unstable once the contract is made explicit. The base instruction (Section 3.1 and Appendix A.1) lists the critique channel among the materials to use, yet Section 3.1 says only the evidence channel has authority, and Appendix A.1 concedes the gap. So I don’t think the 15.83% can be called “unwarranted” when the task itself never says the critique doesn’t count. With a rule saying critique text is not evidence, the main batch drops to 3.04% (Table 4), and in Appendix C.5, where the rule appears in every input, the same critique gives 0/2400. This contradicts the abstract’s claim that neither rule restores the baseline. It also means CEAI (Section 3.1, Eq. 1) assumes a rule that the main experiment never actually gave the model.

Overall, I think the effect itself is real, but the explanations the paper offers for it are not convincing, and the paper never gets past a behavioral account of why it happens.

Questions for authors
  1. How many of the 200 claims in the main experiment are synthetic and how many are external? Could you report Table 8 split by material? If most of the 90 opposite-direction verdicts turn out to be on external claims, what is left of the explanation that the material decides the direction?
  2. In the panel, the verdict rate on synthetic claims under a SUPPORTS recommendation is not higher than the no-critique baseline. Given that, what evidence supports the idea that the critique induces verdicts against its own recommendation?
  3. Under both rules DA is significantly below 50%. In a fully crossed design, do you agree that this means the model is moving against the recommendation? In C.5 the critique plus the rule gives 0/2400, while the main experiment gives 73/2400. What exactly differs between the two?
Reviewer_xgtpOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether an LLM acting as a final adjudicator (e.g., in fact verification or multi-agent review) keeps abstaining when the supplied evidence cannot decide a claim, but a critique recommending a direction without evidence is added

Critical strengths
  1. Timely and well-controlled study. Analyzing LLM behavior in fact verification and multi-agent review is a valuable problem. The paper isolates the effect cleanly: it varies a single critique sentence and keeps the evidence fixed. Controls with a bare label (0/990) and a neutral note (4/989) show that the trigger is a directional critique.
  2. Useful measurement framing. Splitting the failure rate into commitment rate and direction adoption reveals differences that a single error rate would hide. For example, the reading prompt and the channel rule both lower failures, but the channel rule changes the direction of the remaining verdicts (DA ~77% vs ~32%).
Critical weaknesses
  1. Clarity. The paper is hard to follow. It introduces many conditions, batches, and notations (E, R, g, DA, ACR, CEAI), and key results are spread across the main text and several appendix experiments. It is often hard to tell which experiment supports which claim.
  2. Overly generic problem formulation. The core question, what a critique does when the evidence cannot settle the claim, is posed in general terms. The paper’s own results show that the behavior depends heavily on the corpus. External claims always follow the recommendation, while synthetic claims are always REFUTES, possibly because of how the placeholder is worded. The question should be formulated and analyzed in the context of specific corpora and their properties, not as a general property of the model.
  3. Lack of coherent findings. The results read as a collection of observations rather than a unified conclusion. Commitment rates vary widely across batches (24.1%, 15.83%, 6.17%, 0%). The main experiment is not stratified by material, so the 90 opposing verdicts remain unexplained. The threshold account is post hoc. As a result, it is unclear what the main insight is or what practitioners should take away from the paper.
Questions for authors
  1. How would you justify that under "NOINFO" setting, the model follows the recommended direction, is not a problem of prompting template, but a generalizable behavior?
  2. For synthetic claims, the absent-evidence placeholder states that “the visible record gives no marker value.” For a claim asserting that a specimen carries a marker, this sentence can reasonably be read as refuting evidence. The consistent REFUTES verdicts on synthetic materials may therefore reflect normal reasoning over the placeholder rather than a material-dependent tendency. How do you rule out this explanation for the “joint decision” finding in Section 4.3?
Reviewer_z59mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper examines a fact-verification setting in which an LLM receives a claim, an evidence field, and an optional critique. The LLM must output one of three labels: SUPPORTS, REFUTES, or NOINFO. To isolate the effect of critique, the authors replace claim-relevant evidence with non-decisive registry or archival metadata. The correct response is therefore NOINFO, regardless of whether the critique recommends SUPPORTS or REFUTES.

Critical strengths

the paper addresses a meaningful failure mode in LLM based verification, where critiques and intermediate reasoning outputs may be incorrectly treated as evidence. this issue is relevant to multi-agent debate and fact-checking

Separating commitment rate from direction adoption is a valuable analytical choice.

Critical weaknesses

the authors should formulate stronger theoretical properties, such as invariance under a defined family of non-evidentiary transformations, or develop a causal model separating critique-induced commitment from directional influence.

the synthetic evidence placeholder may itself contain directional information. This is one of my most significant concerns. The synthetic evidence placeholder contains statements such as: The visible record gives no marker value. Although the authors intend this statement to represent missing evidence, its wording may be interpreted as negative evidence against a claim asserting the existence of a marker. This is especially problematic because the paper reports that all 1,011 unwarranted verdicts across the synthetic conditions of the stratified panel are REFUTES, regardless of the critique recommendation.

Questions for authors

can the authors repeat the synthetic experiments using a neutral metadata placeholder that does not mention the attribute queried by the claim? what are the commitment and direction-adoption rates for external versus synthetic claims in the main 12,000-run experiment?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.