acceptodds
Under review as a conference paper at ICLR 2027

When Reasoning Contradicts the Table: Diagnosing Error Recovery in Language Models

Abstract

Language models often solve TableQA by generating reasoning traces that cite intermediate evidence. However, an incorrect table reference can make an otherwise coherent reasoning trace inconsistent with the evidence in the table. When such a conflict arises, can models return to the evidence and recover on their own? We investigate this with the *Evidence–Trace Conflict* (ETC) protocol: we hold the table, question, and continuation point fixed while varying whether the supplied reasoning prefix is correct or contains data referencing errors (DREs). Across model families and three TableQA benchmarks, conflicting prefixes cause accuracy drops of 49.4 to 78.7 percentage points. On WikiTableQuestions (WTQ), an explicit reconsideration cue placed after the conflict improves accuracy by 67.3 percentage points, from 18.0% to 85.3%, whereas a reminder in the original user prompt, before the conflicting prefix, provides little help. This contrast reveals a *self-initiation gap* under evidence–trace conflict: models often recover when reconsideration is triggered after the conflict, but rarely initiate it on their own. Guided by this diagnosis, we fine-tune models on self-generated ordinary and recovery continuations to internalize the externally elicited behavior. This training raises single-pass recovery from 18.0% to 88.0% on WTQ and achieves over 85% recovery on two unseen TableQA benchmarks. Together, these results show that when reasoning contradicts the table, models often follow erroneous traces despite having access to the correct evidence, but can learn to recover without additional inference-time intervention.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, table question answering, chain-of-thought reasoning, evidence grounding, error recovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_QALpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper investigates whether large language models can recover from reasoning errors when intermediate Chain-of-Thought steps contradict the original table evidence. The authors introduce the Evidence–Trace Conflict protocol, which constructs matched correct and conflicting reasoning prefixes to evaluate models' error recovery abilities and the effects of reconsideration prompts. Based on the observed gap between spontaneous and externally prompted recovery, they propose Recovery-pair SFT, which fine-tunes models on ordinary reasoning continuations and self-generated error-recovery continuations. Experiments across three TableQA benchmarks show substantial improvements in recovery accuracy under controlled evidence–trace conflicts, including transfer to unseen table datasets without additional inference-time interventions.

Critical strengths
  1. The analysis of different error types and intervention timings might be useful.

  2. The proposed Recovery-pair SFT achieves significant improvements on synthetically constructed evidence–trace conflicts, including transfer from WTQ training data to two unseen TableQA benchmarks.

Critical weaknesses
  1. The biggest weakness is the evaluation framework itself. Though proposed evaluation framework does control the situation very well, and the pipeline seems reasonable, it might not evaluate the ability of LLMs correctly. In detail, the prefix for the evaluation is provided by the evaluation framework instead of the model itself, which causes a distribution mismatch. The model itself might never encounter the prefix on its own. Therefore, all observations in this paper are only made under this circumstance instead of a real-world situation, which weakens the paper's conclusion. For example, the model might actually verify with its own prefix, but this evaluation cannot tell.

  2. Following the previous issue, the second weakness is about the actual utility of the proposed SFT method. Despite the large improvements reported on synthetic conflicts, Recovery-pair SFT achieves only 15.25% recovery accuracy on naturally occurring data referencing errors, substantially below its performance on the controlled evaluation. Moreover, the method does not improve standard TableQA accuracy without supplied reasoning prefixes, leaving its practical benefit for normal model inference unclear.

Questions for authors
  1. Could the author solve my concerns above?

  2. Could the authors test whether the thinking model has the same issue?

6 comments
AuthorsOpenReview ↗

To Reviewer QALp(1)

Thank you for your thoughtful feedback and for highlighting the strengths of our work. We address the weaknesses you identified and answer your questions below. We have marked all changes in blue in the revised manuscript.

W1. Distribution mismatch from constructed prefixes

Thank you for raising the concern that constructed prefixes may differ from the reasoning traces models generate on their own.

Why we use constructed prefixes. We select relatively easy questions that the model can generally answer correctly without a supplied prefix. This reduces the risk that low accuracy under conflicting prefixes reflects the difficulty of the questions themselves. On this selected set, the matched ETC design holds the table, question, and continuation point fixed while varying whether the prefix contains a data referencing error (DRE). Constructed prefixes also let us vary error type and position systematically and verify that following the DRE leads to an incorrect answer. Naturally generated DREs can co-occur with calculation or logical errors, and few natural traces meet all of our criteria. We therefore use constructed examples for the controlled evaluation and to obtain sufficient training pairs. This design trades some realism for experimental control, and we agree that the distribution difference matters when interpreting the results. To address your concern about whether our findings extend beyond constructed prefixes, we report two complementary evaluations below.

Recovery from the model's own prefixes. In Appendix A.6 (Table 14), we evaluate 59 naturally occurring DREs from traces generated by Qwen3-8B itself. We truncate each trace and supply the retained tokens verbatim, without editing the prefix or injecting an error. The base model recovers on only 2/59 cases (3.39%), compared with 9/59 (15.25%) after Recovery-pair SFT. This experiment addresses the source of the prefix while retaining the continuation setup.

End-to-end self-recovery. Following your suggestion, we also evaluate Qwen3-8B on 4,344 WTQ questions with no supplied prefix or external intervention (Table 4). The incidence of self-generated DREs is similar before and after SFT (23.09% and 22.15%). Among traces containing a self-generated DRE, explicit self-detection rises from 29/1,003 (2.89%) to 112/962 (11.64%), and successful self-recovery rises from 9/1,003 (0.90%) to 39/962 (4.05%). Thus, unaided recovery is rare even when the base model generates the entire trace itself. Recovery-pair SFT improves this behavior.

AuthorsOpenReview ↗

To Reviewer QALp(2)

W2. Recovery on naturally occurring DREs and in end-to-end reasoning

Thank you for your suggestions. Our primary contribution is the evidence–trace conflict perspective, the ETC protocol, and the empirical findings it enables. Recovery-pair SFT plays a supporting role as a mitigation motivated by the diagnosis.

Why is recovery lower on naturally occurring DREs? Two factors help explain the gap. The controlled synthetic ETC set and the natural-DRE set differ in distribution. In addition, initiating a check is only the first step in recovering from a natural DRE.

For the controlled ETC set, we select questions the model can generally answer on its own. This reduces the influence of question difficulty when we measure the effect of an answer-changing DRE. For the natural-DRE set, we select traces in which the model produced an eligible DRE during ordinary generation, without an additional difficulty filter. To illustrate the difference between the resulting distributions, we evaluate base Qwen3-8B on both sets without a supplied prefix. It answers 132/150 (88.00%) of the controlled questions and 17/59 (28.81%) of the natural-set questions correctly (Appendix A.6, Table 13). Recovery-pair SFT also uses synthetic training pairs, and we leave closing the distribution gap between these training pairs and naturally occurring DREs to future work.

Recovery from a natural DRE involves initiating a check, locating the erroneous step, retrieving the correct table evidence, and using it to repair the reasoning. In our staged-assistance experiment (Table 15), flagging the presence of an incorrect citation increases the number of correct answers from 2/59 to 16/59. Adding the error location increases it to 19/59, and supplying the correct table evidence as well increases it to 44/59. Perfect repair would yield 59/59, which serves as an ideal upper bound. These results show that substantial barriers to recovery remain after the model is prompted to initiate a check. Recovery-pair SFT improves unaided continuation from 3.39% to 15.25% on the natural prefixes (Table 14). Improving the subsequent steps of error localization, evidence retrieval, and repair is a direction for future work.

Does the benefit extend to end-to-end reasoning? The original Condition A evaluation covered only 150 questions from the controlled subset. Because this subset was selected for questions the model can generally answer on its own, base Qwen3-8B already reached 88.0% without a supplied prefix. This small, relatively easy subset may offer too few opportunities to observe a benefit from self-recovery. We therefore evaluated all 4,344 WTQ test questions without a supplied prefix or external intervention (Table 4).

On the full test set, overall answer accuracy rises from 54.51% to 62.57% (+8.06+8.06 percentage points). Among traces containing a self-generated DRE, explicit self-detection rises from 2.89% to 11.64%, and successful self-recovery rises from 0.90% to 4.05%. The trajectory analysis shows that after Recovery-pair SFT, the model more often detects and repairs errors it makes during ordinary generation.

Q1. Evaluation and practical utility

Thank you for raising these concerns. We address them in our responses to W1 and W2, where we discuss the difference between constructed and self-generated prefixes and report the new end-to-end results for Recovery-pair SFT.

Q2. Recovery in thinking mode

Thank you for suggesting that we test whether the same issue arises in thinking mode. We evaluated Qwen3-4B and Qwen3-8B in thinking mode on WTQ under conflicting prefixes (Appendix A.8, Table 18).

Without intervention, Qwen3-8B recovered more often than Qwen3-4B (73.33% vs. 46.67%), but both models still failed to recover on a substantial fraction of cases. A post-conflict cue raised recovery accuracy to 76.00% for Qwen3-4B and 86.67% for Qwen3-8B. A separate review turn raised it to 91.33% and 90.67%, respectively.

Thus, the gap between unaided and externally prompted recovery persists in thinking mode for both models, even though Qwen3-8B recovers more often on its own.

AuthorsOpenReview ↗

Look forward to further discussions

Dear reviewer,

We hope this message finds you well. We greatly appreciate your time and effort in reviewing our paper. We have carefully answered all the questions you raised about our paper and look forward to further discussions with you.

Reviewer_2f3WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces the Evidence–Trace Conflict (ETC) protocol to study whether language models can recover when a reasoning trace conflicts with visible table evidence. The authors find that models often follow the incorrect trace, while post-conflict reconsideration cues can largely restore performance. They further propose recovery-pair SFT to improve single-pass recovery without additional inference-time intervention.

Critical strengths
  1. The paper is clearly written and easy to follow.
  2. The matched correct/conflicting prefix design is well controlled and helps isolate the effect of evidence–trace conflict.
  3. The diagnosis and mitigation are well connected, and recovery-pair SFT achieves strong results across three TableQA datasets.
Critical weaknesses
  1. Most experiments use synthetically constructed numeric or entity errors. On naturally occurring DREs, recovery-pair SFT improves accuracy only from 3.39% to 15.25%, which is much lower than the controlled ETC results.
  2. Recovery-pair SFT is mainly standard supervised fine-tuning on generated recovery continuations. The main novelty appears to come from the ETC protocol rather than the training method.
  3. Most baselines are prompting or multi-call methods. It is unclear whether the gains come specifically from the recovery-pair design or simply from training on conflicting prefixes.
  4. The main training experiments cover only two models and TableQA tasks. More complex errors, such as missing evidence, calculation errors, and naturally generated reasoning errors, are not sufficiently evaluated.
Questions for authors
  1. Why does recovery-pair SFT perform much worse on naturally occurring DREs than on synthetically constructed conflicts?
  2. Could the authors compare with simpler SFT baselines trained on the same conflicting-prefix data and under the same compute budget?
  3. Can the authors provide an end-to-end evaluation where models generate their own reasoning traces and must recover from naturally occurring errors?
  4. How sensitive are the results to the stronger model used to construct and validate the conflicting prefixes, and can the authors provide more extensive human validation and multi-seed results?
4 comments
AuthorsOpenReview ↗

To Reviewer 2f3W(1)

Thank you for your thoughtful feedback and for highlighting the strengths of our work. We address the weaknesses you identified and answer your questions below. We have marked all changes in blue in the revised manuscript.

W1. Recovery on naturally occurring DREs

Thank you for your suggestions. Our primary contribution is the evidence–trace conflict perspective, the ETC protocol, and the empirical findings it enables. Recovery-pair SFT plays a supporting role as a mitigation motivated by the diagnosis.

Why is recovery lower on naturally occurring DREs? Two factors help explain the gap. The controlled synthetic ETC set and the natural-DRE set differ in distribution. In addition, initiating a check is only the first step in recovering from a natural DRE.

For the controlled ETC set, we select questions the model can generally answer on its own. This reduces the influence of question difficulty when we measure the effect of an answer-changing DRE. For the natural-DRE set, we select traces in which the model produced an eligible DRE during ordinary generation, without an additional difficulty filter. To illustrate the difference between the resulting distributions, we evaluate base Qwen3-8B on both sets without a supplied prefix. It answers 132/150 (88.00%) of the controlled questions and 17/59 (28.81%) of the natural-set questions correctly (Appendix A.6, Table 13). Recovery-pair SFT also uses synthetic training pairs, and we leave closing the distribution gap between these training pairs and naturally occurring DREs to future work.

Recovery from a natural DRE involves initiating a check, locating the erroneous step, retrieving the correct table evidence, and using it to repair the reasoning. In our staged-assistance experiment (Table 15), flagging the presence of an incorrect citation increases the number of correct answers from 2/59 to 16/59. Adding the error location increases it to 19/59, and supplying the correct table evidence as well increases it to 44/59. Perfect repair would yield 59/59, which serves as an ideal upper bound. These results show that substantial barriers to recovery remain after the model is prompted to initiate a check. Recovery-pair SFT improves unaided continuation from 3.39% to 15.25% on the natural prefixes (Table 14). Improving the subsequent steps of error localization, evidence retrieval, and repair is a direction for future work.

W2. Contribution of Recovery-pair SFT

Thank you for your feedback. We agree that our primary contribution is the evidence–trace conflict perspective, the ETC protocol, and the empirical findings it enables. Recovery-pair SFT is a lightweight mitigation motivated by this diagnosis.

The training objective is standard supervised fine-tuning. The distinctive element is how we construct the training pairs. For the same table and question, we pair a correct prefix with an ordinary continuation and a conflicting prefix with a recovery continuation. This provides supervision for continuing a sound reasoning trace and for returning to the table evidence when the trace contains a conflict.

W3. Comparison with alternative SFT baselines

Thank you for your suggestion. Section 5.6 compares Recovery-pair SFT with Correct-only and Wrong-only SFT. Following your suggestion, we added Always-check SFT, which uses self-checking continuations for both correct and conflicting prefixes. Always-check and Recovery-pair SFT use the same 1:1 mix of correct and conflicting prefixes and the same budget of 2,400 training example presentations (Table 5).

Wrong-only and Always-check achieve recovery accuracy comparable to Recovery-pair SFT on conflicting prefixes. This suggests that supervision on conflicting prefixes accounts for much of the recovery gain. The correct-prefix half of the recovery-pair design serves a separate purpose: it teaches the model to continue normally when the prefix agrees with the table. Together, the two halves supervise when to continue and when to initiate recovery.

To test whether this distinction is reflected in model behavior, we audited how often each method re-examines an already correct prefix. Recovery-pair SFT shows a lower self-reflection rate than both Wrong-only and Always-check on correct WTQ prefixes (Appendix A.4, Table 10). Thus, the paired design maintains comparable recovery accuracy on conflicting prefixes and reduces unnecessary rechecking on correct ones.

W4. Scope of models, tasks, and error types

Thank you for your suggestions. Given the time and experimental effort required, we were unable to extend the training experiments to additional models or tasks during this revision. We leave these extensions to future work.

Our study primarily examines conflicts between table evidence and reasoning traces, with a specific focus on data referencing errors (DREs). We leave the study of other error types, including evidence omissions and calculation errors, to future work.

AuthorsOpenReview ↗

To Reviewer 2f3W(2)

Q1. Recovery gap between constructed and natural DREs

Thank you for your suggestion. Please see our response to W1 for details.

Q2. Matched-budget SFT baselines

Thank you for your suggestion. Please see our response to W3 for the matched-budget comparison with simpler SFT baselines.

Q3. End-to-end recovery from self-generated DREs

Thank you for your suggestion. We added an end-to-end evaluation of Qwen3-8B before and after Recovery-pair SFT on all 4,344 WTQ test questions (Table 4). The model generates its full reasoning trace from scratch, without a supplied prefix or external intervention. We then measure how often it detects and corrects its own data referencing errors (DREs).

After fine-tuning, overall answer accuracy rises from 54.51% to 62.57% (+8.06+8.06 percentage points). Among traces containing a self-generated DRE, explicit self-detection rises from 2.89% to 11.64%, and successful self-recovery rises from 0.90% to 4.05%.

These results show that Recovery-pair SFT improves both detection of and recovery from self-generated DREs during end-to-end reasoning. The gain in overall answer accuracy also demonstrates a practical benefit for end-to-end table question answering.

Q4. Construction-model sensitivity, manual audit, and training seeds

Thank you for your suggestions.

Sensitivity to the construction model. We reconstructed conflicting prefixes for the same 50-question WTQ subset using Qwen3.8-Max in place of DeepSeek-V4 to inject errors and rewrite the affected reasoning for internal consistency. Accuracy under conflicting prefixes remained low with both construction models: 18.00% (9/50) with DeepSeek-V4 and 14.00% (7/50) with Qwen3.8-Max (Appendix A.7, Table 16). These results suggest that our main finding about the impact of conflicting prefixes generalizes across construction models.

Human validation. We manually audited all WTQ continuations from the base and fine-tuned Qwen3-8B under both correct and conflicting prefixes. On conflicting prefixes, the fine-tuned model explicitly identified the error in 38/150 cases and implicitly corrected it by replacing the incorrect evidence in 99/150 cases. Counting both behaviors as evidence of detection, the rate rose from 11.3% to 91.3% after SFT. On correct prefixes, the false-positive rate was 0.7% before SFT and 1.3% after SFT (Appendix A.5, Table 11). This pattern indicates that Recovery-pair SFT improves detection of errors in conflicting prefixes while keeping false detections rare on correct prefixes.

Multi-seed results. We repeated Recovery-pair SFT with two additional training seeds, 44 and 46, alongside the original seed 42. Across the three seeds, recovery accuracy under conflicting prefixes ranged from 82.67% to 88.00% on WTQ, 83.93% to 86.61% on TableBench, and 80.54% to 85.52% on HiTab (Appendix A.7, Table 17). Recovery-pair SFT maintained strong recovery on conflicting prefixes and high accuracy on correct prefixes across all three datasets, indicating robustness to the choice of training seed within the tested settings.

AuthorsOpenReview ↗

Look forward to further discussions

Dear reviewer,

We hope this message finds you well. We greatly appreciate your time and effort in reviewing our paper. We have carefully answered all the questions you raised about our paper and look forward to further discussions with you.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer,

Thank you again for your time and valuable feedback. We have posted our point-by-point responses and added experiments and analyses addressing your concerns in the revised manuscript.

As the discussion deadline approaches, we would greatly appreciate any follow-up feedback on whether our responses address your concerns. We would be happy to clarify any remaining questions.

Best regards,

The Authors

Reviewer_vNsvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates why large language models (LLMs) often fail to correct erroneous reasoning steps, even when the correct evidence remains available in the original table.

To study this phenomenon, the authors introduce the Evidence–Trace Conflict (ETC) protocol, which constructs matched reasoning prefixes containing either correct evidence references or artificially injected errors. By comparing model continuations under these conditions, the authors examine how conflicting reasoning traces affect model performance. Their experiments show that erroneous prefixes substantially reduce answer accuracy, while explicit reconsideration prompts can restore much of the lost performance. The authors interpret this discrepancy as a self-initiation gap.

The paper further proposes Recovery-pair Supervised Fine-Tuning (SFT), which trains models on ordinary continuations following correct prefixes and recovery continuations following erroneous prefixes. For Qwen3-8B, the proposed method improves recovery accuracy from 18.0% to 88.0% on WikiTableQuestions (WTQ), with substantial improvements on TableBench and HiTab.

The paper addresses an important problem in reliable LLM reasoning and presents promising results under controlled experimental conditions. However, the evidence remains insufficient to establish whether the proposed method enables models to spontaneously detect and correct errors arising during their own reasoning processes.

Critical strengths

S1. Important research problem and a systematic diagnostic protocol

The tendency of LLMs to follow erroneous reasoning traces despite having access to correct evidence is an important challenge for developing reliable reasoning systems. The ETC protocol provides a relatively controlled framework for investigating this issue by holding the table and question fixed while modifying the correctness of the reasoning prefix.

S2. Interesting observations regarding externally triggered reconsideration

On WTQ, accuracy under conflicting prefixes is only 18.0%, but increases to 85.3% when an explicit table-check instruction is introduced after the conflict. This result provides interesting evidence that models may possess substantial error-recovery capabilities that are not reliably activated during ordinary continuation.

S3. Simple training approach with substantial performance improvements

Recovery-pair SFT is relatively straightforward to implement and does not require reinforcement learning or additional inference-time verification modules. The substantial improvements on TableBench and HiTab after training only on WTQ suggest that the learned recovery behavior can transfer across different table-question-answering benchmarks. The authors also provide useful ablation studies comparing correct-only, wrong-only, and mixed training configurations.

Critical weaknesses

W1. Lack of evidence for spontaneous error recovery during end-to-end reasoning

My primary concern is the discrepancy between the proposed training and evaluation setup and the conditions encountered during actual LLM reasoning.

Under the ETC protocol, erroneous prefixes are constructed by first generating correct reasoning traces and then artificially modifying numerical values or entity references. The model is subsequently evaluated on its ability to continue from these externally supplied erroneous prefixes.

However, in realistic applications, a model generates its own reasoning steps and may introduce errors during generation. It must then recognize and correct these errors without external intervention.

This distinction is important for assessing the practical significance of Recovery-pair SFT. Demonstrating that a model can recover from an externally supplied erroneous prefix does not necessarily establish that it can spontaneously identify and correct errors arising during its own reasoning process.

Appendix A.4 reports an evaluation under Condition A, where the model generates its reasoning without a supplied prefix. Interestingly, accuracy on WTQ slightly decreases from 88.0% before fine-tuning to 86.7% afterward. Thus, the paper does not demonstrate an improvement in overall task performance under ordinary, prefix-free reasoning conditions. Although final-answer accuracy alone cannot establish whether spontaneous recovery has improved, this result highlights the need for more detailed trajectory-level analysis.

I encourage the authors to evaluate the proposed method in an end-to-end reasoning setting, separately measuring error occurrence, spontaneous error detection, and successful error correction. Such an evaluation would provide more direct evidence of whether the learned recovery behavior transfers to realistic reasoning scenarios.

W2. Limited generalization to naturally occurring reasoning errors

The paper includes an exploratory evaluation of naturally occurring Data Referencing Errors (DREs) in Section 6 and Appendix A.5. While this additional experiment is valuable, its results differ substantially from those obtained with synthetically constructed conflicts.

Recovery-pair SFT achieves 88.0% recovery accuracy on the synthetic WTQ evaluation set, whereas recovery accuracy on naturally occurring DREs is only 15.25%. Although this represents an improvement over the base model's 3.39%, it remains considerably below the performance achieved with an explicit Review Turn (27.12%) or Self-check intervention (35.59%).

These results raise concerns about whether the recovery behavior learned from synthetically generated errors generalizes effectively to errors produced during natural reasoning.

Furthermore, the paper provides limited details about the size, construction, filtering criteria, and error-type distribution of the naturally occurring DRE evaluation set. The authors attribute the lower performance partly to the additional difficulties of error localization and repair, but this interpretation is not supported by sufficiently detailed empirical analysis.

A more comprehensive evaluation of naturally occurring errors, including performance breakdowns by error type, position, and complexity, would substantially strengthen the paper. In particular, understanding why recovery performance differs so significantly between synthetic and naturally occurring errors is essential for assessing the scope and limitations of the proposed method.

W3. Unclear whether Recovery-pair SFT explicitly improves error detection

The paper argues that Recovery-pair SFT enables models to recover from erroneous reasoning without externally supplied reconsideration instructions.

However, the proposed training procedure relies on conventional next-token SFT, where correct prefixes are paired with ordinary continuations and erroneous prefixes are paired with recovery continuations. The training loss is computed only over the continuation tokens.

Although this setup may encourage the model to learn different behaviors depending on the consistency of the prefix, it does not directly establish that the model has acquired the ability to detect inconsistencies between the original table and its previous reasoning.

Interestingly, Table 4 shows that wrong-only training achieves 92.7% recovery accuracy on WTQ, outperforming the 88.0% achieved by the mixed correct-and-wrong training configuration. While the mixed configuration reduces unnecessary reconsideration on correct prefixes, this finding does not by itself demonstrate that the model has learned a reliable, selective error-detection mechanism.

An alternative explanation is that the model learns to generate recovery-style continuations from patterns encountered during training, rather than explicitly identifying contradictions in the available evidence.

I therefore suggest separating error detection from error correction in the evaluation. For example, the authors could directly measure whether the model correctly identifies inconsistencies in both correct and erroneous prefixes before assessing its ability to repair them.

Additionally, comparisons with alternative SFT strategies, such as standard correct-reasoning SFT or SFT that encourages table re-verification regardless of prefix correctness, would help clarify the specific contribution of the proposed recovery-pair construction.

W4. The causal interpretation of the self-initiation gap requires further validation

The authors interpret the substantial performance difference between pre-conflict reminders and post-conflict reconsideration instructions as evidence of a self-initiation gap.

However, these experimental conditions differ not only in the timing of the instructions but also in their placement and relationship to the ongoing generation. Consequently, the observed differences may reflect several interacting factors, including recency effects, sensitivity to conversational structure, or learned associations between particular discourse markers and corrective reasoning.

The paper provides useful controls comparing expressions such as "Now" versus "Wait" and "Continue" versus "Rethink." Nevertheless, these controls do not fully isolate the proposed self-initiation mechanism from alternative explanations.

Additional experiments that more carefully control instruction content, placement, and conversational structure would strengthen the interpretation that the primary bottleneck is the model's inability to initiate recovery independently.

Questions for authors

Q1. End-to-end spontaneous error correction

Does Recovery-pair SFT improve the ability of models to spontaneously detect and correct their own errors during end-to-end reasoning, without externally supplied erroneous prefixes?

Could the authors provide an evaluation in which models generate complete reasoning trajectories from scratch, with separate measurements of error occurrence, spontaneous error detection, and successful recovery?

Q2. Generalization to naturally occurring errors

How was the naturally occurring DRE evaluation set in Appendix A.5 constructed and filtered? Could the authors clarify the models used to generate the erroneous traces, the dataset size, and the distribution of error types and complexities?

In particular, could the authors investigate why Recovery-pair SFT achieves strong performance on synthetically constructed conflicts but only 15.25% recovery accuracy on naturally occurring errors?

Q3. Distinguishing error detection from error correction

How can the authors verify that Recovery-pair SFT genuinely improves error-detection capabilities rather than simply teaching the model to produce recovery-style continuations?

Could the authors evaluate the model's ability to distinguish correct from erroneous prefixes independently of its final answer accuracy?

Q4. Interpretation of the self-initiation gap

Can the authors further disentangle the proposed self-initiation gap from possible recency effects, instruction-placement effects, and sensitivity to reconsideration-related discourse markers?

Additional controlled experiments could help establish whether the observed improvements are primarily attributable to activating an existing recovery capability.

7 comments
AuthorsOpenReview ↗

To Reviewer vNsv(1)

Thank you for your thoughtful feedback and for highlighting the strengths of our work. We address the weaknesses you identified and answer your questions below. We have marked all changes in blue in the revised manuscript.

W1. Spontaneous error recovery in end-to-end reasoning

Thank you for your suggestion.

End-to-end evaluation of spontaneous error recovery. The original Condition A evaluation covered only 150 questions from the controlled subset. This subset was selected for questions the model can generally answer on its own. This small, relatively easy subset may offer too few opportunities to observe a benefit from self-recovery. We therefore evaluated Qwen3-8B before and after Recovery-pair SFT on all 4,344 WTQ test questions (Table 4). The model generates its full reasoning trace from scratch, without a supplied prefix or external intervention.

On the full test set, overall answer accuracy rises from 54.51% to 62.57% (+8.06+8.06 percentage points). The incidence of self-generated DREs is similar before and after SFT (23.09% vs. 22.15%). Among traces containing a self-generated DRE, explicit self-detection rises from 2.89% to 11.64%, and successful self-recovery rises from 0.90% to 4.05%. The trajectory analysis shows that after Recovery-pair SFT, the model more often detects and repairs its own DREs during ordinary generation.

W2. Generalization to naturally occurring DREs

Thank you for your suggestions. We expanded Appendix A.6 to describe how we collected and filtered the 59 natural-DRE examples from reasoning traces generated by Qwen3-8B. Table 12 reports the distribution by error type, position of the first DRE, and source of the incorrect reference. Below, we discuss two factors that help explain why recovery accuracy is lower on the natural-DRE set.

Distribution differences. The controlled synthetic ETC set and the natural-DRE set differ in distribution. For the controlled set, we select questions the model can generally answer on its own. This reduces the influence of question difficulty when we measure the effect of an answer-changing DRE. For the natural set, we select traces in which the model produced an eligible DRE during ordinary generation, without an additional difficulty filter. To illustrate this difference, we evaluate base Qwen3-8B on both sets without a supplied prefix. It answers 132/150 (88.00%) of the controlled questions and 17/59 (28.81%) of the natural-set questions correctly (Appendix A.6, Table 13). Recovery-pair SFT also uses synthetic training pairs, and we leave closing the distribution gap between these training pairs and naturally occurring DREs to future work.

Barriers after recovery initiation. Recovering from a natural DRE involves initiating a check, locating the erroneous step, retrieving the correct table evidence, and using it to repair the reasoning. In our staged-assistance experiment (Appendix A.6, Table 15), flagging the presence of an incorrect citation increases the number of correct answers from 2/59 to 16/59. Adding the error location increases it to 19/59, and supplying the correct table evidence as well increases it to 44/59. Perfect repair would yield 59/59, which serves as an ideal upper bound. These results show that substantial barriers to recovery remain after the model is prompted to initiate a check.

By improving the model’s ability to detect DREs and initiate recovery on its own, Recovery-pair SFT increases unaided continuation accuracy on natural prefixes from 3.39% to 15.25% (Appendix A.6, Table 14). We leave further improvements in the subsequent steps of error localization, evidence retrieval, and repair to future work.

AuthorsOpenReview ↗

To Reviewer vNsv(2)

W3. Behavioral evidence of error detection

Thank you for your suggestions.

Measuring spontaneous error detection. Directly asking whether a prefix contains a reference that conflicts with the table would itself cue the model to inspect the prefix. To evaluate detection during unaided continuation, we manually audited the outputs of base and fine-tuned Qwen3-8B on 150 correct and 150 conflicting WTQ prefixes per model. Generic rechecking did not count as detection. To explicitly identify the specific incorrect reference or implicitly correct it by substituting the corresponding value from the table, the model must recognize the discrepancy. We therefore treat both behaviors as evidence of error detection. We scored these behaviors independently of final-answer correctness.

After Recovery-pair SFT, detection evidence on conflicting prefixes rose from 17/150 (11.3%) to 137/150 (91.3%), comprising 38 explicit identifications and 99 implicit corrections after fine-tuning. On correct prefixes, false detections remained rare (1/150 before SFT and 2/150 after SFT, Appendix A.5, Table 11). This pattern indicates that Recovery-pair SFT improves error detection on conflicting prefixes while keeping false detections rare on correct ones.

Alternative SFT baselines. Section 5.6 compares Recovery-pair SFT with Correct-only and Wrong-only SFT. Following your suggestion, we added Always-check SFT, which pairs both correct and conflicting prefixes with review-derived continuations. All settings use 2,400 training-example presentations. On conflicting WTQ prefixes, Correct-only reaches 18.0%, Wrong-only 92.7%, Always-check 88.7%, and Recovery-pair SFT 88.0% (Table 5).

These results indicate that supervision on conflicting prefixes accounts for much of the recovery gain. The correct-prefix half of Recovery-pair SFT teaches the model when to continue normally. On correct WTQ prefixes, it rechecks less often than both Wrong-only and Always-check SFT (Appendix A.4, Table 10). The benefit of the paired design is therefore selective recovery: it retains strong recovery on conflicting prefixes and reduces unnecessary rechecking on correct ones.

W4. Interpreting the self-initiation gap

Thank you for your suggestions.

The self-initiation gap is a behavioral finding. When we insert a DRE into the reasoning trace, the model often continues along the erroneous trajectory despite having access to the correct table evidence. A short self-check cue placed after the conflict can prompt the model to reconsider the trace and recover. We use self-initiation gap to describe this difference between unaided and externally prompted recovery. The following controls help clarify which features of the cue matter for this behavior.

Cue placement and recency. We placed the same cue, Wait, let me re-check the table., at three positions within a single assistant turn: at the start of the turn, immediately before the step containing the DRE, and after the conflicting prefix (Appendix A.10, Table 19). Recovery remained limited at both pre-conflict positions, even when the cue immediately preceded the DRE. Placing the same cue after the conflicting prefix raised accuracy to 80.7%. Thus, proximity to the DRE alone was insufficient to elicit substantial recovery. Since all three placements used the same speaker role, differences in speaker role cannot account for the observed accuracy gap. These results indicate that the cue is particularly effective once the conflict is already present in the trace.

Sensitivity to discourse markers. Appendix A.10 (Table 21) reports a 2×22 \times 2 comparison crossing Now versus Wait with Continue versus Rethink. Changing Continue to Rethink can improve recovery even with the same opening marker. We also compare a generic rethink cue with an explicit table-check cue (Section 3.2 and Appendix A.10, Table 20). Together, these comparisons show that wording affects the size of the recovery gain and that recovery can be elicited by several formulations. We leave a broader study of cue wording to future work.

Together, these results support the behavioral interpretation that the model often fails to initiate a check unaided but can recover when prompted to reconsider after the conflict. The specific internal mechanism remains an open question.

AuthorsOpenReview ↗

To Reviewer vNsv(3)

Q1. End-to-end evaluation of spontaneous error recovery

Thank you for your suggestion. Please see our response to W1 for the end-to-end evaluation and separate results on error occurrence, spontaneous detection, and successful recovery.

Q2. Natural-DRE dataset construction and the recovery gap

Thank you for your suggestion. Please see our response to W2 for details on how we constructed and filtered the natural-DRE evaluation set, the distribution of errors, and our analysis of why recovery accuracy is lower on naturally occurring DREs.

Q3. Distinguishing error detection from correction

Thank you for your suggestion. Please see our response to W3 for the manual audit of behavioral evidence of error detection on correct and conflicting prefixes, scored independently of final-answer accuracy.

Q4. Controls for cue placement and wording

Thank you for your suggestion. Please see our response to W4 for the cue placement and wording controls, and for how these results inform our behavioral interpretation of the self-initiation gap.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary to Reviewers, AC and SACs

Dear Reviewers, AC, and SACs,

Thank you for taking the time to review our paper. We greatly appreciate your valuable comments, which have helped us strengthen the evaluation and clarify our contributions. We reiterate the core contributions and summarize the main additions in our rebuttal below to facilitate further review. Detailed responses are available in the individual review threads. Changes to the manuscript are marked in blue.

Core Contributions:

  • A controlled protocol for studying evidence–trace conflicts. We introduce Evidence–Trace Conflict (ETC), which holds the table, question, and continuation point fixed while varying whether the reasoning prefix contains a data referencing error (DRE). This matched design enables systematic analysis of how incorrect references affect subsequent reasoning.
  • Empirical findings on trace-following behavior and the self-initiation gap. Using ETC, we find that conflicting prefixes reduce accuracy by 49.4–78.7 percentage points across model families and three TableQA benchmarks, showing that models often follow erroneous traces despite having access to the correct table evidence. On WTQ, a post-conflict table-check cue raises accuracy from 18.0% to 85.3%, whereas a reminder before the conflicting prefix provides little help. Together, these findings reveal both the disruptive effect of evidence–trace conflicts and a gap between recovery that models can perform when prompted and recovery they initiate on their own.
  • A lightweight method for learning selective recovery. Motivated by this diagnosis, Recovery-pair SFT pairs correct prefixes with ordinary continuations and conflicting prefixes with recovery continuations. It raises Qwen3-8B recovery accuracy from 18.0% to 88.0% on WTQ and achieves over 85% on two unseen TableQA benchmarks, without additional inference-time intervention. These results demonstrate that recovery behavior can be learned and transferred across benchmarks.

Rebuttal Summary:

  • Demonstrating practical benefits in end-to-end reasoning. On all 4,344 WTQ test questions, without supplied prefixes or external intervention, Recovery-pair SFT improves overall accuracy from 54.51% to 62.57%. Among traces containing self-generated DREs, explicit detection rises from 2.89% to 11.64%, and successful recovery from 0.90% to 4.05% (Table 4). On a matched 3,000-question subset, Recovery-pair SFT achieves 66.20%, versus 64.10% for Correct-only SFT and 58.47% for the base model. This comparison demonstrates an additional end-to-end benefit beyond training on correct continuations alone.
  • Establishing transfer to naturally occurring errors and explaining the recovery gap. On 59 natural-DRE prefixes, Recovery-pair SFT raises unaided recovery from 3.39% to 15.25%, demonstrating transfer beyond constructed conflicts. We document the set's construction and distribution, and identify two factors that help explain the smaller gain: distribution differences between synthetic and natural errors, and challenges in localization, evidence retrieval, and repair after a check begins. Supplying correct evidence after flagging and locating the error increases successful recoveries from 19/59 to 44/59, highlighting evidence retrieval's importance. Narrowing the distribution gap and strengthening subsequent recovery steps are directions for future work (Appendix A.6, Tables 12–15).
  • Clarifying paired training and providing behavioral evidence of detection. We added Always-check SFT under the same budget of 2,400 training-example presentations. The baseline comparisons show that Recovery-pair SFT balances strong recovery with reduced unnecessary rechecking on correct prefixes. A manual audit further finds that explicit error identification or implicit correction of cited evidence increases from 11.3% to 91.3% on conflicting prefixes after SFT, while false detections remain rare on correct prefixes. Together, these results support effective and selective recovery behavior (Table 5 and Appendix A.4–A.5, Tables 10–11).
  • Strengthening cue controls and robustness evaluations. Holding cue wording and speaker role fixed, accuracy is 18.0% at the assistant-turn start, 22.0% immediately before the DRE, and 80.7% after the conflicting prefix. Thus, proximity alone is insufficient, and role differences cannot explain this comparison. Additional experiments support the corresponding findings across prefix-construction models, training seeds, and models in thinking mode (Appendix A.7–A.10, Tables 16–21).

We would like to thank the reviewers, AC, and SACs again for your hard work and valuable suggestions for this paper. Please reconsider our contributions to the understanding and improvement of evidence-grounded reasoning in light of the additional results and revised manuscript.

Best wishes,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.