acceptodds
Under review as a conference paper at ICLR 2027

Counterfactual Answer Prediction Training: Improving LLMs' Causal Understanding via Input-Perturbation Difference Prediction

Abstract

Large language models can solve math problems but may lack true causal understanding: models that correctly solve a problem often cannot predict how localized perturbations to it affect the answer. We introduce Counterfactual Answer Prediction Training (CAPT), a fine-tuning method that trains LLMs to predict answer deltas () under localized input perturbations, without re-solving the perturbed problem. A diagnostic study reveals a strong forward–counterfactual dissociation: Qwen3-8B solves 87.0% of GSM8K problems but predicts correct causal deltas on only 15.5% of relevant perturbations (dissociation score ), with the gap most severe for entity swap (10.0%) and condition negation (11.1%). We then construct a counterfactual training dataset of 4,817 triples from GSM8K and fine-tune with LoRA across four arms. Forward+CAPT (50/50 mix) substantially improves counterfactual competence (relevant CF accuracy: 15.5% 31.9%, , ), improves localization of causally irrelevant edits to 100% (from 97.8%), and improves out-of-distribution forward reasoning (MATH-500: 33.5% 42.0%, ). Critically, standard data augmentation fails to improve causal competence (18.1% vs. 31.9%, for the difference), indicating that the delta prediction objective—not mere exposure to perturbations—drives the gain. Scaling experiments across four model sizes (0.5B–8B) reveal a sharp capacity threshold: counterfactual competence remains flat at 9% for models up to 3B before jumping to 31.9% at 8B, while forward accuracy scales smoothly—providing additional evidence that forward and counterfactual reasoning are distinct capabilities.

Primary area:
causal reasoning
Keywords:
causal understanding, counterfactual reasoning, mathematical reasoning, instruction fine-tuning, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_si3rOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposed Counterfactual Answer Prediction Training (CAPT), a fine-tuning objective that trains LLMs to predict answer changes under localized perturbations rather than re-solving modified problems. The authors tested this tool on 4817 counterfactual triples and compared four training strategies. They argued that counterfactual reasoning is different from forward problem-solving and combing forwarding with CAPT can have a better performance with a threshold.

Critical strengths
  1. CAPT provides a straightforward way to supervise how localized input perturbations affect mathematical answers. The objective is easy to implement and evaluate.
  2. The paper includes forward-only, CAPT-only, mixed-training, and CF augmentation baselines, ablations and statistical tests. These experiments provide a useful starting point for studying counterfactual answer prediction.
Critical weaknesses
  1. The paper argues that delta prediction without fully resolving inflects a better causal understanding ability. But it fails to prove this statement. In this paper, we cannot clearly see why fully resolving a delta problem has less causal understanding ability than prediction delta outcome. The CF-augmentation baseline also differs in both inputs and supervision, making it difficult to isolate the benefit of delta prediction.
  2. In experiment, Forward+CAPT does not significantly outperform Forward-only on CF accuracy (31.9% vs. 27.6%, p=0.4576) or MATH-500 (42.0% vs. 41.5%, p=1.0). Also, the claimed capacity threshold is confounded by comparing Qwen2.5 models with Qwen3-8B. Additionally, some conclusions rely on only 9–16 test examples, limiting their reliability.
Questions for authors
  1. Why is predicting answer deltas without resolving the problem considered stronger evidence of causal understanding? To me, resolving a delta problem may require more understanding of the causality structure.
  2. Would CAPT still outperform a baseline model that receives the same inputs (P, A, π) but predicts the modified answer A' instead of the delta under the same budget constraint?
  3. The Forward-only methods achieve nearly identical accuracy on MATH-500. How do the authors determine that the improvement in OOD performance is attributable to CAPT rather than additional mathematical fine-tuning or exposure to the training data?
  4. How can the claimed capability thresholds be distinguished from the differences between the Qwen2.5 and Qwen3 model series?
Reviewer_dhyxOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Counterfactual Answer Prediction Training (CAPT), which trains language models to directly predict the answer delta caused by localized perturbations without re-solving the perturbed problem. Experiments show that Qwen3-8B achieves an 87.0% forward-solving accuracy on GSM8K, but only a 15.5% counterfactual delta prediction accuracy on relevant perturbations. Based on a counterfactual triple dataset constructed from GSM8K, combining forward and CAPT mixed fine-tuning improves relevant counterfactual accuracy and achieves forward accuracy gains on the out-of-distribution benchmark MATH-500.

Critical strengths
  1. The paper formalizes localized text modifications in math word problems as causal interventions, quantifying the performance gap between forward pattern matching and causal effect anticipation in current models.
  2. This work designs control groups including forward fine-tuning, CAPT fine-tuning, mixed fine-tuning, and standard data augmentation that re-solves perturbed problems, verifying that directly learning the delta prediction objective.
Critical weaknesses
  1. The contribution of this work is standard prompt-based fine-tuning using problem modifications and answer deltas as supervised data, with its claimed contributions primarily concentrated on empirical evaluations and comparative analyses. Moreover, the highlighted insight has already been widely explored and verified in robustness testing, prompt perturbation, and causal decoupled reasoning, e.g., RobustQA: Benchmarking the robustness of domain adaptation for open-domain question answering; Can foundation models talk causality?. In symbolic and mathematical reasoning, existing works have also investigated verifying intermediate dependency graphs or answer sensitivity via perturbation operators, e.g., Measuring faithfulness in chain-of-thought reasoning; Towards generalizable reasoning: group causal counterfactual policy optimization for LLM reasoning.
  2. Changing an initial quantity can cause complex ripple effects through non-linear logical branches. The paper restricts the model via the prompt and forces it to output only a single numerical value. Does this contradict the mechanism of solving complex compositional problems by unrolling computation chains autoregressively? Forcing such jump-step outputs is very likely to induce the model to fit shallow statistical correlations of numerical deltas.
  3. All formalizations and experiments in this paper are tied entirely to math arithmetic problems with single numerical answers. In tasks such as algebraic proofs, logical reasoning, code generation, or long-form planning, where simple scalar delta concepts generally do not exist, this method cannot be directly transferred.
  4. Many critical details remain unclear. For example, five perturbation types are listed, but the text fails to specify whether they are generated via hard-coded regex rules or with the assistance of an external model. For condition negation and relational swaps, how is the mathematical self-consistency of the modified problem's new answer A′A' guaranteed? For instance, the authors note in Appendix B that condition negation may render the problem ill-posed, how is ΔA\Delta A defined under such circumstances?
  5. In the evaluation test subsets, Condition negation has only n=9n=9 samples, Entity swap has only n=10n=10, and Relational has only n=16n=16. Reporting percentage accuracies on such small subsets with single-digit or low double-digit sample sizes is highly susceptible to individual sample variance. Meanwhile, the experiments lack comparisons against recent mainstream reasoning enhancement baselines, such as self-consistency, CoT with verification, etc.
Questions for authors

Please refer to the above comments.

Reviewer_WFWdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that LLMs that solve math problems often cannot predict how a small edit to a problem changes its answer. It calls this “counterfactual competence” and measures it as exact-match accuracy on the answer delta ΔA = A′ − A. On 200 GSM8K-derived evaluation items, Qwen3-8B solves 87.0% of problems but gets only 15.5% of “relevant” deltas right (18/116), which the authors report as a dissociation score of 0.822.

They propose CAPT, a LoRA fine-tuning objective on (problem, answer, perturbation) → ΔA triples. The dataset has 4,817 triples across five perturbation types. They compare four arms against the base model: forward-only SFT, CAPT-only, a 50/50 Forward+CAPT mix, and a “CF-augmentation” control that trains on re-solving the perturbed problems. Forward+CAPT raises relevant-delta accuracy from 15.5% to 31.9% and MATH-500 accuracy from 33.5% to 42.0%. The CF-augmentation arm reaches only 18.1%, which the authors read as evidence that the objective, not the data, drives the gain. A scaling study (Qwen2.5 0.5B/1.5B/3B vs. Qwen3-8B) shows relevant-delta accuracy flat near 9% up to 3B and 31.9% at 8B. The authors interpret this as a sharp capacity threshold and as evidence that forward and counterfactual reasoning are distinct capabilities.

Critical strengths
  1. The question is well motivated and cheap to verify. Because deltas have exact answers, “can the model predict the effect of an edit” becomes a clean metric. The diagnostic includes useful controls: irrelevant perturbations, a re-derivation rate of 7.1%, and a re-solving robustness measure. The observation that training on perturbed problems (CF-augmentation) does not improve delta prediction (18.1% vs. 15.5%, p = 0.61) is an interesting and honestly reported negative result.

  2. The paper is transparent about its design and limits. It reports paired significance tests (Table 9), including non-significant ones. It lists single training runs, small subsamples and exact-match scoring as limitations, and describes how deltas were recomputed. Prompts and ablations over type coverage, mixing ratio and LoRA rank are given.

  3. The method is simple, cheap and adds no inference cost.

Critical weaknesses
  1. The central “dissociation” is confounded by the output format. The CAPT prompt requires a single line, DELTA: <number>, with no reasoning and no re-solving. Forward accuracy (87%) is measured with step-by-step chain of thought. The 0.822 dissociation therefore compares a CoT task with a no-CoT task, and the paper never measures forward accuracy without CoT or delta prediction with CoT. The paper’s own robustness numbers show how large the gap is. The base model re-solves perturbed problems at 81% (Table 3), so “re-solve, then subtract A” would give roughly 80% delta accuracy, far above the 31.9% reached by the trained model. The data are equally consistent with the model lacking the ability to do multi-step arithmetic in one token sequence, which is not the same as lacking causal understanding. The framing in the abstract and the Pearl/do-calculus language (Sec. 3.1) is not supported at this strength.

  2. The claim that the objective, not the data, drives the gains is not established. (a) Forward-only SFT already reaches 27.6% relevant-CF accuracy, and the difference from Forward+CAPT (31.9%) is not significant (p = 0.46, Table 9). The abstract and Finding 2 compare only against CF-augmentation. (b) The CF-augmentation control is mismatched in format. It never sees the delta-prediction prompt or output format at training time, so a failure to improve there is expected. It is not a clean test of “mere exposure to perturbations.” A fairer control would train on the same triples with a delta-format target but different content, or evaluate CF-augmentation by solving both problems and subtracting. (c) The control’s own numbers look odd. CF-augmentation robustness (re-solving perturbed problems, the task it was trained on) drops from 81.0% to 42.0%, identical to CAPT-only. This suggests a training or evaluation issue that should be explained.

  3. The out-of-distribution “causal transfer” claim is not supported. Forward-only reaches 41.5% on MATH-500 versus 42.0% for Forward+CAPT (p = 1.0), so the 33.5% → 42.0% gain is attributable to generic fine-tuning. The ablations point the same way. Condition-negation-only training gives the best MATH-500 score (46.0%) while its CF accuracy is 16.4%, close to base. “No numeric” training also gets 45.0% with 16.4% CF. MATH-500 gains are thus decoupled from counterfactual competence. A base MATH-500 accuracy of 33.5% for Qwen3-8B is also unusually low, so the gain may mostly reflect output-format or decoding effects fixed by any SFT on the model’s own solutions. In-distribution GSM8K accuracy also drops (87.0% → 81.0%). The paper marks this as “ns” and does not discuss it.

  4. Statistical support is thinner than the p-values suggest, and the per-type story rests on a handful of items. Every arm is a single training run, and the paired tests capture only variance over about 116 evaluation items, not training variance. About a dozen comparisons are made with no correction, and non-significance of the ablations is read as proof of robustness (e.g. “not sensitive to adapter capacity”), while Figure 4’s title calls rank 32 a “sweet spot.” The headline claim that entity swap and condition negation are the hardest types rests on 1/10 and 1/9 correct. CAPT’s gain on condition negation is 1 → 2 items, and entity swap does not improve at all. The gain is concentrated in numeric (+13 items) and relational (+5 items), the types the paper calls easiest, which contradicts the narrative that CAPT instils deep structural understanding. Single-type ablations (entity-only 12.9%, condition-only 16.4%, relational-only 19.0%) are near base, suggesting type-specific pattern learning and not general causal competence. The localization improvement (97.8% → 100%) is near ceiling and partly trivial. Zero-delta examples make up about 41% of the training data, and the “remove irrelevant examples” ablation shows only that a model never shown a zero delta stops predicting zero.

  5. The scaling and “emergence” claim is confounded and underdetermined. The “scale” axis mixes model generations and variants: Qwen2.5-0.5B-/1.5B-Instruct, Qwen2.5-3B (apparently the base model), and Qwen3-8B, which is also the teacher for the training solutions. With four points and one run each, a “sharp capacity threshold” cannot be distinguished from a family or generation effect. The small-model behavior looks like a floor and not a smooth capability: 0.5B and 1.5B have identical relevant (8.6%) and overall (47.0%) accuracy, and 3B has degraded localization (0.775). Only CAPT-trained models are reported, with no untrained baselines at each size, so the figure cannot support the claim that forward and counterfactual reasoning scale differently. Both were trained, and only CF has a floor. The p = 2×10⁻⁶ compares two different model families and is not evidence of a threshold.

Questions for authors
  1. Several numbers do not reconcile. Table 1 gives a 482-triple test split with 175 numeric and 214 irrelevant items, but evaluation uses 200 items (81 numeric, 84 irrelevant). Figure 3 shows n = 74 failures, whereas 116 × (1 − 0.319) ≈ 79. The Figure 3 caption mentions “representative examples” that are not shown.

  2. CF irrelevant (100% for all arms, including base) and Localization (97.8% for base) are not clearly distinguished.

  3. Citations look mismatched. Huang et al. 2023 is cited for GSM8K-Hard and robustness studies but the bibliography lists “Large language models can self-improve,” and Qwen2.5-Math is cited for the Qwen2.5 base models. Closely related work on perturbed or what-if math evaluation is missing (e.g. GSM-Symbolic, GSM-Plus, CLadder).

Please also look at the weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.