- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
This paper investigates whether large language models can recover from reasoning errors when intermediate Chain-of-Thought steps contradict the original table evidence. The authors introduce the Evidence–Trace Conflict protocol, which constructs matched correct and conflicting reasoning prefixes to evaluate models' error recovery abilities and the effects of reconsideration prompts. Based on the observed gap between spontaneous and externally prompted recovery, they propose Recovery-pair SFT, which fine-tunes models on ordinary reasoning continuations and self-generated error-recovery continuations. Experiments across three TableQA benchmarks show substantial improvements in recovery accuracy under controlled evidence–trace conflicts, including transfer to unseen table datasets without additional inference-time interventions.
Critical strengths
-
The analysis of different error types and intervention timings might be useful.
-
The proposed Recovery-pair SFT achieves significant improvements on synthetically constructed evidence–trace conflicts, including transfer from WTQ training data to two unseen TableQA benchmarks.
Critical weaknesses
-
The biggest weakness is the evaluation framework itself. Though proposed evaluation framework does control the situation very well, and the pipeline seems reasonable, it might not evaluate the ability of LLMs correctly. In detail, the prefix for the evaluation is provided by the evaluation framework instead of the model itself, which causes a distribution mismatch. The model itself might never encounter the prefix on its own. Therefore, all observations in this paper are only made under this circumstance instead of a real-world situation, which weakens the paper's conclusion. For example, the model might actually verify with its own prefix, but this evaluation cannot tell.
-
Following the previous issue, the second weakness is about the actual utility of the proposed SFT method. Despite the large improvements reported on synthetic conflicts, Recovery-pair SFT achieves only 15.25% recovery accuracy on naturally occurring data referencing errors, substantially below its performance on the controlled evaluation. Moreover, the method does not improve standard TableQA accuracy without supplied reasoning prefixes, leaving its practical benefit for normal model inference unclear.
Questions for authors
-
Could the author solve my concerns above?
-
Could the authors test whether the thinking model has the same issue?
6 comments
To Reviewer QALp(2)
W2. Recovery on naturally occurring DREs and in end-to-end reasoning
Thank you for your suggestions. Our primary contribution is the evidence–trace conflict perspective, the ETC protocol, and the empirical findings it enables. Recovery-pair SFT plays a supporting role as a mitigation motivated by the diagnosis.
Why is recovery lower on naturally occurring DREs? Two factors help explain the gap. The controlled synthetic ETC set and the natural-DRE set differ in distribution. In addition, initiating a check is only the first step in recovering from a natural DRE.
For the controlled ETC set, we select questions the model can generally answer on its own. This reduces the influence of question difficulty when we measure the effect of an answer-changing DRE. For the natural-DRE set, we select traces in which the model produced an eligible DRE during ordinary generation, without an additional difficulty filter. To illustrate the difference between the resulting distributions, we evaluate base Qwen3-8B on both sets without a supplied prefix. It answers 132/150 (88.00%) of the controlled questions and 17/59 (28.81%) of the natural-set questions correctly (Appendix A.6, Table 13). Recovery-pair SFT also uses synthetic training pairs, and we leave closing the distribution gap between these training pairs and naturally occurring DREs to future work.
Recovery from a natural DRE involves initiating a check, locating the erroneous step, retrieving the correct table evidence, and using it to repair the reasoning. In our staged-assistance experiment (Table 15), flagging the presence of an incorrect citation increases the number of correct answers from 2/59 to 16/59. Adding the error location increases it to 19/59, and supplying the correct table evidence as well increases it to 44/59. Perfect repair would yield 59/59, which serves as an ideal upper bound. These results show that substantial barriers to recovery remain after the model is prompted to initiate a check. Recovery-pair SFT improves unaided continuation from 3.39% to 15.25% on the natural prefixes (Table 14). Improving the subsequent steps of error localization, evidence retrieval, and repair is a direction for future work.
Does the benefit extend to end-to-end reasoning? The original Condition A evaluation covered only 150 questions from the controlled subset. Because this subset was selected for questions the model can generally answer on its own, base Qwen3-8B already reached 88.0% without a supplied prefix. This small, relatively easy subset may offer too few opportunities to observe a benefit from self-recovery. We therefore evaluated all 4,344 WTQ test questions without a supplied prefix or external intervention (Table 4).
On the full test set, overall answer accuracy rises from 54.51% to 62.57% ( percentage points). Among traces containing a self-generated DRE, explicit self-detection rises from 2.89% to 11.64%, and successful self-recovery rises from 0.90% to 4.05%. The trajectory analysis shows that after Recovery-pair SFT, the model more often detects and repairs errors it makes during ordinary generation.
Q1. Evaluation and practical utility
Thank you for raising these concerns. We address them in our responses to W1 and W2, where we discuss the difference between constructed and self-generated prefixes and report the new end-to-end results for Recovery-pair SFT.
Q2. Recovery in thinking mode
Thank you for suggesting that we test whether the same issue arises in thinking mode. We evaluated Qwen3-4B and Qwen3-8B in thinking mode on WTQ under conflicting prefixes (Appendix A.8, Table 18).
Without intervention, Qwen3-8B recovered more often than Qwen3-4B (73.33% vs. 46.67%), but both models still failed to recover on a substantial fraction of cases. A post-conflict cue raised recovery accuracy to 76.00% for Qwen3-4B and 86.67% for Qwen3-8B. A separate review turn raised it to 91.33% and 90.67%, respectively.
Thus, the gap between unaided and externally prompted recovery persists in thinking mode for both models, even though Qwen3-8B recovers more often on its own.
Look forward to further discussions
Dear reviewer,
We hope this message finds you well. We greatly appreciate your time and effort in reviewing our paper. We have carefully answered all the questions you raised about our paper and look forward to further discussions with you.
To Reviewer QALp(1)
Thank you for your thoughtful feedback and for highlighting the strengths of our work. We address the weaknesses you identified and answer your questions below. We have marked all changes in blue in the revised manuscript.
W1. Distribution mismatch from constructed prefixes
Thank you for raising the concern that constructed prefixes may differ from the reasoning traces models generate on their own.
Why we use constructed prefixes. We select relatively easy questions that the model can generally answer correctly without a supplied prefix. This reduces the risk that low accuracy under conflicting prefixes reflects the difficulty of the questions themselves. On this selected set, the matched ETC design holds the table, question, and continuation point fixed while varying whether the prefix contains a data referencing error (DRE). Constructed prefixes also let us vary error type and position systematically and verify that following the DRE leads to an incorrect answer. Naturally generated DREs can co-occur with calculation or logical errors, and few natural traces meet all of our criteria. We therefore use constructed examples for the controlled evaluation and to obtain sufficient training pairs. This design trades some realism for experimental control, and we agree that the distribution difference matters when interpreting the results. To address your concern about whether our findings extend beyond constructed prefixes, we report two complementary evaluations below.
Recovery from the model's own prefixes. In Appendix A.6 (Table 14), we evaluate 59 naturally occurring DREs from traces generated by Qwen3-8B itself. We truncate each trace and supply the retained tokens verbatim, without editing the prefix or injecting an error. The base model recovers on only 2/59 cases (3.39%), compared with 9/59 (15.25%) after Recovery-pair SFT. This experiment addresses the source of the prefix while retaining the continuation setup.
End-to-end self-recovery. Following your suggestion, we also evaluate Qwen3-8B on 4,344 WTQ questions with no supplied prefix or external intervention (Table 4). The incidence of self-generated DREs is similar before and after SFT (23.09% and 22.15%). Among traces containing a self-generated DRE, explicit self-detection rises from 29/1,003 (2.89%) to 112/962 (11.64%), and successful self-recovery rises from 9/1,003 (0.90%) to 39/962 (4.05%). Thus, unaided recovery is rare even when the base model generates the entire trace itself. Recovery-pair SFT improves this behavior.