- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposed Counterfactual Answer Prediction Training (CAPT), a fine-tuning objective that trains LLMs to predict answer changes under localized perturbations rather than re-solving modified problems. The authors tested this tool on 4817 counterfactual triples and compared four training strategies. They argued that counterfactual reasoning is different from forward problem-solving and combing forwarding with CAPT can have a better performance with a threshold.
Critical strengths
- CAPT provides a straightforward way to supervise how localized input perturbations affect mathematical answers. The objective is easy to implement and evaluate.
- The paper includes forward-only, CAPT-only, mixed-training, and CF augmentation baselines, ablations and statistical tests. These experiments provide a useful starting point for studying counterfactual answer prediction.
Critical weaknesses
- The paper argues that delta prediction without fully resolving inflects a better causal understanding ability. But it fails to prove this statement. In this paper, we cannot clearly see why fully resolving a delta problem has less causal understanding ability than prediction delta outcome. The CF-augmentation baseline also differs in both inputs and supervision, making it difficult to isolate the benefit of delta prediction.
- In experiment, Forward+CAPT does not significantly outperform Forward-only on CF accuracy (31.9% vs. 27.6%, p=0.4576) or MATH-500 (42.0% vs. 41.5%, p=1.0). Also, the claimed capacity threshold is confounded by comparing Qwen2.5 models with Qwen3-8B. Additionally, some conclusions rely on only 9–16 test examples, limiting their reliability.
Questions for authors
- Why is predicting answer deltas without resolving the problem considered stronger evidence of causal understanding? To me, resolving a delta problem may require more understanding of the causality structure.
- Would CAPT still outperform a baseline model that receives the same inputs (P, A, π) but predicts the modified answer A' instead of the delta under the same budget constraint?
- The Forward-only methods achieve nearly identical accuracy on MATH-500. How do the authors determine that the improvement in OOD performance is attributable to CAPT rather than additional mathematical fine-tuning or exposure to the training data?
- How can the claimed capability thresholds be distinguished from the differences between the Qwen2.5 and Qwen3 model series?