- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper asks whether a single directional critique can push a model out of abstention when the evidence is missing and NOINFO is an allowed answer. The authors replace the evidence with archival metadata that says nothing about whether the claim is true, and then add one sentence that recommends a direction without offering any evidence. The main experiment uses 200 claims, some taken from FEVER and SciFact and some generated from templates, and tests three commercial models under five conditions. The five conditions are no critique, the critique alone, and the critique plus one of three instructions (a careful-reading prompt, a rule that critique text is not evidence, or a stricter rule that only the evidence can decide the verdict). Each condition is run 2,400 times. Without a critique, only 6 of the 2,400 runs return SUPPORTS or REFUTES. With the critique this number rises to 380, and 90 of these verdicts go against the recommended direction. The authors split these verdicts into a commitment rate (how often the model leaves abstention) and direction adoption (how often a verdict follows the recommendation). Comparing the careful-reading prompt with the rule that critique text is not evidence, they find that the prompt lowers commitment but leaves the direction of the remaining verdicts unchanged. The rule lowers commitment from 15.83% to 3.04%, and most of the verdicts that survive oppose the recommendation. The paper also reports a panel that separates the two claim types, a replication of the rule, and an experiment that counts how many errors a critique corrects when the evidence is present. The authors also propose Closed-Evidence Abstention Invariance (CEAI), which requires the answer to stay NOINFO when a critique without evidence is added to an input that lacks evidence.
Critical strengths
- The experimental protocol is described clearly. The counts are reported in full and are internally consistent, and every test I recomputed from them matched the paper. This is why I trust the data even though I disagree with how it is interpreted.
- All three models show the behavior the paper describes, so the main effect does not depend on a single model.
Critical weaknesses
- Two of the paper’s main explanations are contradicted by its own tables. The first is the material explanation. Section 4.3 attributes the 90 opposite-direction verdicts of the main experiment (Table 4) to synthetic claims always getting REFUTES. But in the stratified panel (Appendix C.1, Table 13), a SUPPORTS recommendation on synthetic claims yields 52/580 and 38/578 REFUTES verdicts, which is not significantly above the 35/567 with no critique (p = 0.076 and 0.810 by the authors’ own tests in Appendix C.1). So the critique adds no reliable opposite-direction verdicts there, and the panel cannot explain the ones in the main experiment. The main experiment itself is never split by material (Section 3.3), even though the material type is obvious from the claim text. The effect of the rules is read backwards. Each claim is run equally often under both recommendations, so any verdict unrelated to the recommendation should give a DA of 50% on average. Under the two rules DA is 31.5% and 24.2% (Table 4), and a SUPPORTS recommendation yields more REFUTES verdicts than a REFUTES recommendation does (37 vs. 16 and 31 vs. 8, summed from Table 8). The rules make the model push against the critique, Section 5 and Appendix E.1 say the rules leave only material-driven verdicts, but the numbers say the opposite. This matters because these are the two points the paper relies on to set itself apart from work on sycophancy.
2.The headline effect is measured under an instruction that allows the critique to be used, and it gets small and unstable once the contract is made explicit. The base instruction (Section 3.1 and Appendix A.1) lists the critique channel among the materials to use, yet Section 3.1 says only the evidence channel has authority, and Appendix A.1 concedes the gap. So I don’t think the 15.83% can be called “unwarranted” when the task itself never says the critique doesn’t count. With a rule saying critique text is not evidence, the main batch drops to 3.04% (Table 4), and in Appendix C.5, where the rule appears in every input, the same critique gives 0/2400. This contradicts the abstract’s claim that neither rule restores the baseline. It also means CEAI (Section 3.1, Eq. 1) assumes a rule that the main experiment never actually gave the model.
Overall, I think the effect itself is real, but the explanations the paper offers for it are not convincing, and the paper never gets past a behavioral account of why it happens.
Questions for authors
- How many of the 200 claims in the main experiment are synthetic and how many are external? Could you report Table 8 split by material? If most of the 90 opposite-direction verdicts turn out to be on external claims, what is left of the explanation that the material decides the direction?
- In the panel, the verdict rate on synthetic claims under a SUPPORTS recommendation is not higher than the no-critique baseline. Given that, what evidence supports the idea that the critique induces verdicts against its own recommendation?
- Under both rules DA is significantly below 50%. In a fully crossed design, do you agree that this means the model is moving against the recommendation? In C.5 the critique plus the rule gives 0/2400, while the main experiment gives 73/2400. What exactly differs between the two?