- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper presents a well-motivated study of counterfactual question answering, separating the extraction of a causal model from the computation of an answer. Its main contribution is the controlled comparison in which a prompted language model and an exact executor receive the same frozen causal representation. The results provide evidence that exact execution can substantially improve accuracy when answering requires inferring latent causes, while offering little statistically resolved advantage when the relevant values are already known. The authors argue that language models can make mistakes during either stage, and that separating the two makes it possible to identify and address failures in the reasoning process.
Critical strengths
-
The decomposition of the problem is useful. The matched experiment is particularly valuable where the prompted answerer and executor receive the same frozen representation. This makes it possible to isolate differences in the answering stage rather than conflating them with differences in interpretation.
-
The experimental methodology is described clearly and the evaluation protocol is refreshingly explicit. The benchmark audit is a useful contribution.
-
The limitations have been well considered and are described clearly.
Critical weaknesses
-
While there is novelty, it is relatively narrow. My concern is whether its incremental methodological contribution is sufficiently substantial for ICLR. The strongest novelty lies in the experimental isolation of the answering stage and the finding about latent-state inference, rather than in the executor itself.
-
The empirical evidence is based on a narrow slice of causal reasoning - the main datasets are generated using deterministic Boolean rules composed of AND, OR, single causes and negation. This reduces the generalizability of the results. It would be good to see a more extensive evaluation on realistic or independently constructed narratives, or a clearer demonstration of how the method could handle causal structures outside its current formalism.
-
The results do not establish that reasoning-trained language models generally need an external executor, or that the executor is always preferable. Better prompts, more efficient reasoning models, or alternative inference budgets could change the comparison.
-
The executor is extremely fast once the causal model has been extracted. However, the full pipeline uses multiple model calls, and the paper reports increased latency and token use relative to a one-pass prompt. The comparison with the reasoning model also measures answering cost on an already extracted problem. As a result, the paper establishes a strong answering-stage efficiency advantage, but the practical end-to-end advantage is more dependent on the application and the cost of reading the story. Enumeration also becomes impractical as the number of free root variables grows, as the authors acknowledge.
-
The authors report single runs and note that inference may vary across hardware and serving configurations. Their paired statistical analyses and item-level artefacts are helpful, but repeated runs would provide additional evidence about the stability of the model-based comparisons.
Questions for authors
Please address points raised in Weaknesses.
2 comments
Dear Reviewer G4w6, thank you again for your review. Today's revision adds one analysis to the point your review raises first. Page numbers are those of the revised PDF.
The strongest novelty lies in the experimental isolation of the answering stage and the finding about latent-state inference, rather than in the executor itself.
We agree. In the matched comparison the executor is the exact reference, and the findings are about the language model. A comparison of a translate-then-solve pipeline with direct prompting gives one number per test set. With the extraction frozen and an exact answer for every frozen problem, that number separates into reading, computing, and doing both in one pass. Three findings follow.
- No latent state. The model computes about as well as the executor on chains of up to 11 rules (93.1% against 95.6% on E1, Section 5.2). We had expected long chains to be where exact computation helps.
- Latent state, non-reasoning models. Every non-reasoning model we tested is 7 to 33 points behind the executor on the same problems. The gap is in the computing. It stays at 23.1 points when the model is given the generator's own correct problem, and it falls from 16.2 to 4.7 points when one consistent assignment of the latent values is disclosed (Section 5.3).
- Latent state, a reasoning model. It computes the frozen E2 problems as well as the executor (95.7% against 96.0%, Appendix E) and still loses stories when it answers from the text. Today's analysis locates them. On the 267 E2 stories where Qwen3-30B-A3B-Thinking's own extraction is exactly the stored problem, it answers 15 wrongly from the story and 3 once it is handed the extracted problem (exact McNemar 14 stories against 2, p = 0.004, and 15 against 3 on the 266 stories of a second run, p = 0.008). For 14 of the 15, the model's extraction is exact and its answer to the extracted problem is correct.
The third finding covers 14 of the 18 E2 stories this model loses when it answers directly (Table 4). Table 5 (page 13) lists the nearest prior work by whether it gives the same extracted problem to the answerers it compares, and this paper is the only row marked yes.
Changed in the paper. Appendix E has a new passage and table with this analysis ("Reading, solving, and both in one pass", page 27). Its result is now stated in the contribution list (page 3), at the end of the third paragraph of Section 5.7 (page 9) and in the Conclusion (page 10).
Thank you for the review. We added the checks you requested in Section 5.7 and Appendix E (new text in blue), and repeated the main comparisons.
Novelty. You identify the frozen-extraction control and latent-state result as the central contributions. They suggest a further prediction: even a strong reasoning model should struggle when observations jointly constrain many hidden roots. We tested that in E5, 160 new problems with 4–16 free roots, under a protocol and two predictions committed before testing.
The structured-input arm removes reading errors: the model gets the exact problem and still scores below the pipeline, which has to read it. At high effort it generates a median 37,721 tokens per story, versus 3,745 for the pipeline, with 110/160 answers unfinished at the service limit. This measures cost and completion difficulty, not completed high-effort accuracy. E5 stays within Boolean causal rules, so the result concerns harder latent inference within that formalism.
Reasoning models, prompts and budgets. We ran Qwen3-30B-A3B-Thinking both as a one-pass reader for the executor and as a direct answerer:
These are from the first of two runs. The pipeline is significantly ahead on E2, rewording and alpha in both (second run: 99.7 vs 93.8, 100 vs 95.4, 94.2 vs 92.3). On backdoor and CLadder the differences are not significant. Across both CounterBench splits it scores 92.8% and 93.1%, versus the previously published best of 91.8% from a 671B model.
Under a matched 4,096-token generation cap, gpt-oss-120b as reader plus executor beats direct answering on all five sets: 100 vs 98.5 on E2, 99.4 vs 98.2 on rewording, 93.5 vs 89.0 on alpha, 87.5 vs 75.5 on backdoor and 90.0 vs 84.5 on CLadder (alpha and CLadder use 200-story random samples). A 2,048-token cap widens the gap on E2 and rewording (Appendix E). The direct answerer used our strongest prompt; another written for this inference did not improve it (Appendix A.4). Five-sample voting with Gemma-4-31B costs four times the generated tokens of one pass and remains 20.6 points below the executor (Section 5.5).
Independent wording. CounterBench and CLadder were built by other groups; CLadder uses everyday language, while the reworded set puts E2 into a different grammar. The Qwen3 comparison above uses a single read with no fallback. With non-reasoning Llama-3.3-70B reading, the pipeline scores 99.7% on E2 and 97.2% on CLadder, against 81.8% and 94.0% when Llama answers directly.
End-to-end cost and scaling. Including the reader, the Qwen3 pipeline uses 25–39% fewer input plus generated tokens on the five earlier sets and 41–66% less median time per story across the six sets of Section 5.7. The executor takes 0.20–23 ms per problem up to ten free roots, and under ten seconds at the median with 16 roots (2^17 worlds including the intervened root). Enumeration is exponential, so this runtime does not extend to arbitrarily many roots.
Repeated runs. On frozen E2 problems, Qwen2.5-72B scores 88.9%, 88.0% and 87.7% across three runs. As reader, Llama-3.3-70B scores 93.1%, 93.1% and 92.9% on CounterBench alpha and 86.5%, 87.0% and 87.0% on backdoor. The Qwen3 end-to-end comparison was also run twice, as reported above.