acceptodds
Under review as a conference paper at ICLR 2027

Knowing What to Reconsider: Claim-Level Metacognition for Language Models

Abstract

Humans know that a claim can be false, outdated or unsupported, and they reconsider the ones that matter. Language models lack a good rule for which claims to re-check. On synthetic requests with a false premise among six claims, four open-weight models from 12B to 70B answer only 9–28% correctly. Yet two of their own signals usually flag that premise, a probe on hidden states and the model's probability that each claim is true. Naming that premise in a short note gets it re-checked; naming another claim does not. We propose to re-check a claim when its chance of being wrong, times how much the answer depends on it, times the odds a check catches it, exceeds the cost. We estimate that chance with either signal and hold the other terms fixed. If the most suspect claim passes a threshold, the note names it. With the false premise hidden among 24 claims, this policy lets the models answer 58–87% correctly, against 28–50% when asked to verify every claim the answer uses, while adding under half as many generated tokens. Catching a false claim takes not universal scepticism, but knowing what to reconsider.

Primary area:
foundation or frontier models, including LLMs
Keywords:
metacognition, selective reconsideration, value of information, linear probes, false premises, agentic reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_4EoZOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates whether language models can identify which claims in their input context deserve reconsideration before producing an answer. The central observation is that language models often rely on false premises even when their own internal representations or explicit truth-probability estimates can distinguish those premises from correct claims. To address this discrepancy, the authors propose a claim-level re-checking framework that separates monitoring from control. Monitoring is implemented through either a learned probe over hidden representations or the model's P(True) estimate for individual claims. Control is implemented by appending a targeted note that explicitly identifies a suspicious claim and instructs the model to verify it before answering. A threshold-based policy determines whether to trigger this intervention. The authors evaluate the approach on synthetic false-premise tasks, position-balanced and context-size-controlled datasets, outdated-figure tickets with tool access, and several public benchmarks involving retrieval conflicts and sycophancy. The results show substantial improvements in correcting false-premise answers in controlled synthetic settings. In particular, on the 24-claim benchmark, the proposed gates achieve 58–87% repair rates, compared with 28–50% for the single-prompt verify-all baseline, while requiring fewer additional generated tokens. However, the improvements are considerably less pronounced on public benchmarks, particularly when misleading claims have already been adopted by the model.

Critical strengths

Experimental rigor and validation: Several controls support the naming effect. The paper balances error positions, compares against random and neighboring true claims, and holds gate firing fixed while randomizing the selected knowledge claim. The latter control loses 17–20 percentage points on two models (p. 49). These results provide evidence that choosing the claim matters. The no-type-filter and factored CoVe experiments are also valuable.

Technical novelty and innovation: Connecting input-claim scores to a targeted prompt is a useful extension of work on error detection and correction. The paper tests whether the signal can change the final answer, rather than stopping at probe accuracy.

Critical weaknesses

W1. The methodological contribution is limited. The method combines existing truth scores with a threshold and a targeted prompt. Equation (1) fixes most decision terms, while estimating them performs worse (p. 43). A standard logistic probe also achieves higher localization accuracy among all six claims on two models, but is not tested in the complete pipeline (p. 36), leaving the value of the proposed probe unclear.

W2. Selective checking does not show a consistent accuracy–cost advantage. With 10% false-premise inputs, Qwen's two gates achieve 74%/70% overall accuracy, versus 88%/91% for the unnamed note/verify-all (Table 24). At 24 claims, the gates also add 585/1,055 total tokens versus 344 for verify-all, despite generating fewer tokens (Table 33). These results establish neither consistently better overall accuracy nor a measured runtime advantage.

W3. Incorrect factual labels weaken the evaluation of unnecessary checking. Appendix C.3 acknowledges that some supposedly true claims are false or contestable. Removing nine answer-dependent pairs preserves much of the repair gain, but leaves disputed distractors unaudited. Consequently, gate firing on these nominally correct contexts cannot reliably measure unnecessary checking; the factual labels need independent validation.

W4. The main policy's transfer remains unverified. Algorithm 1 is tested only on the synthetic context-size set; public experiments change the gating and selection rules and show limited gains over the unnamed note with more false alarms (p. 8, Table 26). The headline tool-use comparison also supplies the outdated claim's location manually (p. 5). These results do not establish the same automatic policy beyond the synthetic setting.

Questions for authors
  1. Does unchanged Algorithm 1, calibrated on a separate development set, outperform the unnamed note on public text at comparable false-alarm rates?

  2. Under identical calculation instructions, how does gating compare with always naming the highest-scoring claim in overall accuracy and latency, including scoring costs?

  3. Do the results hold after independently validating and correcting all factual labels, including distractors?

  4. How does a standard logistic probe perform in the complete checking pipeline under the same calibration procedure?

  5. Can you release the model-running and probe-training code during review and independently audit the current public-benchmark false-alarm labels?

Reviewer_tkk2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript aims at studying the behaviour of LLMs in relation to false claims entering their input context, in particular considering cases in which LLMs rely on such claims even if their internal signals suggest that they are incorrect. The authors propose two selective verification methods, each employing the model’s own signals (in one case, a hidden state probe, and in the other the model estimated probability that a claim is true) to evaluate how suspicious are claims, combined with a verification threshold, used to decide when to intervene, and prompt the model to re-check the most suspicious one, that is named specifically in a verification note.

The study evaluates the behaviour of four LLMs using synthetic datasets tailored for the goal, and public benchmarks involving misleading retrieved information and sycophancy. The proposed methods are compared with a no verification baseline, always-on verifications (verify-all prompt, unnamed generic claims verification, name-every-claim verification), and factored chain-of-verification.

Critical strengths
  • The paper is clear. The distinction between a model’s ability to detect false information and its ability to act on it is important, and deserves to be studied.

  • Experiments are well designed, isolating well different effects. The authors use paired correct and incorrect contexts, position-balanced claims, and several control conditions.

  • The analysis of the results is carried on in depth, both in the main text and in the Appendix. The reporting is extremely transparent, including cases in which the proposed methods do not outperform the alternatives and highlighting all the main limitations.

Critical weaknesses
  • The strongest results come from synthetic datasets containing false premises that contradict common knowledge. Performance is considerably weaker on public benchmarks. This raises questions about how effectively the proposed method would generalise to more realistic contexts.

  • One of the main contributions of the paper is to propose the use of the model's internal signals to decide what claim should be verified. This assumes that those signals can indicate when a claim is unreliable. Section 4.5, though, shows that this assumption is only partially true. According to the results included in that section, if the model is convinced by false information or lacks the correct knowledge, those same signals become less reliable. The method therefore risks failing to intervene precisely in some of the situations where it would be most valuable.

Questions for authors
  • How do the findings of this study help guide future research directions aimed at overcoming the mentioned weaknesses? I would appreciate it if the authors could expand the Discussion section to address this point more explicitly.

  • The authors select the verification threshold to achieve approximately a 10% firing rate on correct contexts. How should the threshold be selected in real-world applications, where the frequency of false claims and the costs of unnecessary verification may vary?

  • In the final part of the Discussion, it is suggested that estimating the additional terms of Equation (1) could extend the proposed methods to different claim types. However, earlier in the paper it was mentioned that tests using the full-rule gate perform worse on knowledge claims than the simpler probe-based approach (with results included in Appendix). Could the authors clarify if they overall envision improvements from the addition of these estimates or not, and why, given the negative experimental results on knowledge claims?

Reviewer_B8f7OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The main contribution of this paper is a study of whether an LM can identify which claim in a user’s input to re-check. I.e. if the input says an hour has 70 minutes, a model can use that number even though it answers correctly when asked about the hour by itself. The authors test prompt pairs with a correct or false claim and add a case with a newer file correcting a figure in a work ticket. They then score claims with a hidden-state probe or P(True), and append a note about the most suspicious claim if its score passes a threshold. On synthetic inputs with 24 claims, the gates answer 58-87% of false-premise examples correctly across four models, compared with 28-50% for a verify-all prompt. The useful finding is that a model can signal that a claim is false even though it still relies on its answer which can be used to make it reconsider. However, I think this is a borderline reject a.) this is a known phenomena as they disclose in Extended Related Works in Appendix H, 2.) the strongest results are in a constrained synthetic setting, 3.) there are simpler options that score better in the appendix i.e. logistic regression probe ranks the false claim first among all six claims. Also, the prose was a bit difficult to read (LLM-esque?)

Critical strengths
  1. The paired correct and false versions let us see both fixes and false positives, so there's value in locating a problem.
Critical weaknesses
  1. They use dataset annotations to select checkable knowledge claims + exclude scenario details which often score as suspicious even though they can’t be checked. They do test unfiltered gates on Qwen and Gemma but an actual system has to find and classify claims in free text. Also, on the public benchmark the unnamed note comes about five points of the gated ones.
  2. I doubt the practical cost and accuracy tradeoff is as clear as the “generated-token” claim seems, both gates use more total tokens than the verify-all prompt. So at 10% FPR, the best always-on check is more accurate than the probe gate in most comparisons.
  3. A logistic probe put the false claim first on more Qwen and Gemma examples but the authors didn’t test the gate with that probe.
Questions for authors
  1. How well does the gate work when model predicts boundaries of the claim and the knowledge vs. scenario distinction?
  2. Does the plain logistic probe improve repair or cost when used in the same gate?
Reviewer_wTb5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies claim-level monitoring and correction in language models. It uses a hidden-state probe or P(True) to rank input claims and, when the most suspicious knowledge claim crosses a threshold, appends a note asking the model to re-check it. Experiments on synthetic false-premise requests and outdated-record tasks compare targeted naming with unnamed notes and broader verification. Additional evaluations examine probe transfer, correction and false alarms on public retrieval and sycophancy benchmarks. The paper also analyzes claim-count-dependent thresholds, token costs and failures on claims that models adopt or do not know.

Critical strengths
  1. The paper documents a useful gap between detecting a false premise and correcting an answer that relies on it. Position-balanced examples, generic-only probes, lexical and surprisal controls, and evaluations across four model families support the diagnostic finding. The reporting also includes clean-input results, confidence intervals, token accounting and negative transfer results, which helps delimit the conclusions.
  2. The targeted-naming comparisons are informative because the same note names the false premise, a random claim, a neighboring true claim, or no claim. At 24 claims, choosing the probe-ranked claim also improves repair over a random knowledge claim at identical firing times on Qwen3-14B and Gemma-3-12B. The synthetic repair gains and the factored CoVe comparison provide useful evidence about the role of claim selection.
Critical weaknesses
  1. My main worry is that the final policy has not been tested end to end on natural inputs. Algorithm 1 is evaluated only on the synthetic context-size set with supplied claim-type annotations; the public experiments instead name every above-threshold sentence, leaving the calibrated top-one policy and its P(True) variant untested there. On TriviaQA-RGB, gains over the unnamed note vary across four models, with additional false alarms in every model (Table 26); the simulated one-sentence gates also add false alarms. The claim-count-dependent threshold in Appendix C.3 reduces firing on the synthetic correct versions but assumes independent claim scores from a common distribution. Since identifying knowledge claims in free text remains untested, its practical calibration and robustness to correlated claims would benefit from end-to-end validation.
  2. It is also unclear whether selective triggering is where the value lies. Truth scoring, probe-guided control, and localized correction all have precedents in related work, and the tested rule holds dependence, checking effectiveness, and cost fixed; the fuller rule localizes the false premise less reliably. On the paper’s own numbers, an always-on check is often competitive or better, particularly when false premises are rare. Fewer generated tokens does not necessarily mean lower cost either, since the gates add detection and prefill work, factored CoVe matches their repair rates on the two models tested at 24 claims albeit at higher token cost, and no end-to-end latency comparison is given.
  3. Some supporting results remain sensitive to data labels and prompt wording. Appendix C.3 identifies false or contestable statements labeled true, including the thirty-two-teeth and six-string-guitar examples. Excluding affected pairs preserves most repair comparisons, but these labels complicate interpretation of firing on supposedly clean inputs. The ticket results are also sensitive to wording: asking the model to open the file first substantially improves the every-sentence condition, and the plural naming note reduces the apparent penalty for naming both errors on the multi-error tickets. These controls suggest that the advantage of selecting a single claim may depend partly on the verification instruction.
Questions for authors
  1. Can both final gates be tested unchanged on held-out natural contexts, using automatically identified claims and separately calibrated thresholds? Please compare against the same unnamed note and report accuracy together with false alarms.
  2. Can the value of selective triggering be isolated by giving all conditions the same basic reasoning instruction and comparing accuracy-cost tradeoffs across false-premise prevalences? What part of the expected-utility formulation contributes beyond thresholding a conventional score?
  3. Would the simpler logistic probe, which localizes better over all six claims in Table 16, preserve or improve downstream repair?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.