acceptodds
Under review as a conference paper at ICLR 2027

Separating Reading from Execution in Counterfactual Question Answering

Abstract

A counterfactual question over an explicit causal story asks what would have happened had one variable been different. A language model answering in one pass does two jobs at once: it reads the story into a causal model and it computes the answer. We separate them a schema constrained call extracts a boolean causal model, a deterministic executor enumerates every world that model allows and use the separation to ask when exact execution is worth its cost. Crossing the two stages answers that. Handed perfectly parsed rules, bare in-text answering still collapses, so the computation rather than the read is the weak link. But handed the same extracted model the executor receives, a prompt tuned for the task is not distinguishable from it where the question already fixes the story’s hidden causes: the paired difference is small and its interval spans zero, resolving neither an advantage nor equivalence. The advantage appears where answering requires abduction, inferring un- observed causes before intervening. On 325 pre-registered abduction items the executor scores 95.7% against 74.8% for the strongest separated prompt, 21 percentage points, and 97.8% against 79.9% on the 229 items whose an- swer the story determines. A prompt we built specifically for abduction did worse, not better, because it declines on half the determined items; we report that as a diagnostic, and no finite prompt search can show that no prompt would close the gap. The same executor audits the benchmark it is scored on: it certifies that 60 of CounterBench’s official answers contradict their own stories and flags 20 more it cannot evaluate, each with a hand-checkable certificate. As supporting evidence the pipeline beats the strongest prompting baseline at equal scale on CounterBench’s official splits, 92.4% against 89.4%, while on real-language CLadder a heavily scaffolded prompt edges two points ahead of it.

Primary area:
causal reasoning
Keywords:
counterfactual reasoning, causal reasoning, llm, neurosymbolic, abduction, structural causal models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_G4w6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a well-motivated study of counterfactual question answering, separating the extraction of a causal model from the computation of an answer. Its main contribution is the controlled comparison in which a prompted language model and an exact executor receive the same frozen causal representation. The results provide evidence that exact execution can substantially improve accuracy when answering requires inferring latent causes, while offering little statistically resolved advantage when the relevant values are already known. The authors argue that language models can make mistakes during either stage, and that separating the two makes it possible to identify and address failures in the reasoning process.

Critical strengths
  1. The decomposition of the problem is useful. The matched experiment is particularly valuable where the prompted answerer and executor receive the same frozen representation. This makes it possible to isolate differences in the answering stage rather than conflating them with differences in interpretation.

  2. The experimental methodology is described clearly and the evaluation protocol is refreshingly explicit. The benchmark audit is a useful contribution.

  3. The limitations have been well considered and are described clearly.

Critical weaknesses
  1. While there is novelty, it is relatively narrow. My concern is whether its incremental methodological contribution is sufficiently substantial for ICLR. The strongest novelty lies in the experimental isolation of the answering stage and the finding about latent-state inference, rather than in the executor itself.

  2. The empirical evidence is based on a narrow slice of causal reasoning - the main datasets are generated using deterministic Boolean rules composed of AND, OR, single causes and negation. This reduces the generalizability of the results. It would be good to see a more extensive evaluation on realistic or independently constructed narratives, or a clearer demonstration of how the method could handle causal structures outside its current formalism.

  3. The results do not establish that reasoning-trained language models generally need an external executor, or that the executor is always preferable. Better prompts, more efficient reasoning models, or alternative inference budgets could change the comparison.

  4. The executor is extremely fast once the causal model has been extracted. However, the full pipeline uses multiple model calls, and the paper reports increased latency and token use relative to a one-pass prompt. The comparison with the reasoning model also measures answering cost on an already extracted problem. As a result, the paper establishes a strong answering-stage efficiency advantage, but the practical end-to-end advantage is more dependent on the application and the cost of reading the story. Enumeration also becomes impractical as the number of free root variables grows, as the authors acknowledge.

  5. The authors report single runs and note that inference may vary across hardware and serving configurations. Their paired statistical analyses and item-level artefacts are helpful, but repeated runs would provide additional evidence about the stability of the model-based comparisons.

Questions for authors

Please address points raised in Weaknesses.

2 comments

Thank you for the review. We added the checks you requested in Section 5.7 and Appendix E (new text in blue), and repeated the main comparisons.

Novelty. You identify the frozen-extraction control and latent-state result as the central contributions. They suggest a further prediction: even a strong reasoning model should struggle when observations jointly constrain many hidden roots. We tested that in E5, 160 new problems with 4–16 free roots, under a protocol and two predictions committed before testing.

E5, gpt-oss-120bAccuracy
Reads once + executor78.1
Answers raw story, low effort55.6 (p = 3e-5)
Given exact structured problem65.6
Always "unsure"50.0

The structured-input arm removes reading errors: the model gets the exact problem and still scores below the pipeline, which has to read it. At high effort it generates a median 37,721 tokens per story, versus 3,745 for the pipeline, with 110/160 answers unfinished at the service limit. This measures cost and completion difficulty, not completed high-effort accuracy. E5 stays within Boolean causal rules, so the result concerns harder latent inference within that formalism.

Reasoning models, prompts and budgets. We ran Qwen3-30B-A3B-Thinking both as a one-pass reader for the executor and as a direct answerer:

Qwen3-30BE2RewordedCB alphaCB backdoorCLadder
Reads once + executor99.710093.887.596.3
Answers directly94.595.791.488.596.7
Exact McNemar p8e-51e-4.005.5.23
Median generated tokens (k), pipeline / direct3.4 / 6.34.1 / 6.82.0 / 4.42.0 / 3.51.8 / 3.8

These are from the first of two runs. The pipeline is significantly ahead on E2, rewording and alpha in both (second run: 99.7 vs 93.8, 100 vs 95.4, 94.2 vs 92.3). On backdoor and CLadder the differences are not significant. Across both CounterBench splits it scores 92.8% and 93.1%, versus the previously published best of 91.8% from a 671B model.

Under a matched 4,096-token generation cap, gpt-oss-120b as reader plus executor beats direct answering on all five sets: 100 vs 98.5 on E2, 99.4 vs 98.2 on rewording, 93.5 vs 89.0 on alpha, 87.5 vs 75.5 on backdoor and 90.0 vs 84.5 on CLadder (alpha and CLadder use 200-story random samples). A 2,048-token cap widens the gap on E2 and rewording (Appendix E). The direct answerer used our strongest prompt; another written for this inference did not improve it (Appendix A.4). Five-sample voting with Gemma-4-31B costs four times the generated tokens of one pass and remains 20.6 points below the executor (Section 5.5).

Independent wording. CounterBench and CLadder were built by other groups; CLadder uses everyday language, while the reworded set puts E2 into a different grammar. The Qwen3 comparison above uses a single read with no fallback. With non-reasoning Llama-3.3-70B reading, the pipeline scores 99.7% on E2 and 97.2% on CLadder, against 81.8% and 94.0% when Llama answers directly.

End-to-end cost and scaling. Including the reader, the Qwen3 pipeline uses 25–39% fewer input plus generated tokens on the five earlier sets and 41–66% less median time per story across the six sets of Section 5.7. The executor takes 0.20–23 ms per problem up to ten free roots, and under ten seconds at the median with 16 roots (2^17 worlds including the intervened root). Enumeration is exponential, so this runtime does not extend to arbitrarily many roots.

Repeated runs. On frozen E2 problems, Qwen2.5-72B scores 88.9%, 88.0% and 87.7% across three runs. As reader, Llama-3.3-70B scores 93.1%, 93.1% and 92.9% on CounterBench alpha and 86.5%, 87.0% and 87.0% on backdoor. The Qwen3 end-to-end comparison was also run twice, as reported above.

Dear Reviewer G4w6, thank you again for your review. Today's revision adds one analysis to the point your review raises first. Page numbers are those of the revised PDF.

The strongest novelty lies in the experimental isolation of the answering stage and the finding about latent-state inference, rather than in the executor itself.

We agree. In the matched comparison the executor is the exact reference, and the findings are about the language model. A comparison of a translate-then-solve pipeline with direct prompting gives one number per test set. With the extraction frozen and an exact answer for every frozen problem, that number separates into reading, computing, and doing both in one pass. Three findings follow.

  1. No latent state. The model computes about as well as the executor on chains of up to 11 rules (93.1% against 95.6% on E1, Section 5.2). We had expected long chains to be where exact computation helps.
  2. Latent state, non-reasoning models. Every non-reasoning model we tested is 7 to 33 points behind the executor on the same problems. The gap is in the computing. It stays at 23.1 points when the model is given the generator's own correct problem, and it falls from 16.2 to 4.7 points when one consistent assignment of the latent values is disclosed (Section 5.3).
  3. Latent state, a reasoning model. It computes the frozen E2 problems as well as the executor (95.7% against 96.0%, Appendix E) and still loses stories when it answers from the text. Today's analysis locates them. On the 267 E2 stories where Qwen3-30B-A3B-Thinking's own extraction is exactly the stored problem, it answers 15 wrongly from the story and 3 once it is handed the extracted problem (exact McNemar 14 stories against 2, p = 0.004, and 15 against 3 on the 266 stories of a second run, p = 0.008). For 14 of the 15, the model's extraction is exact and its answer to the extracted problem is correct.

The third finding covers 14 of the 18 E2 stories this model loses when it answers directly (Table 4). Table 5 (page 13) lists the nearest prior work by whether it gives the same extracted problem to the answerers it compares, and this paper is the only row marked yes.

Changed in the paper. Appendix E has a new passage and table with this analysis ("Reading, solving, and both in one pass", page 27). Its result is now stated in the contribution list (page 3), at the end of the third paragraph of Section 5.7 (page 9) and in the Conclusion (page 10).

Reviewer_dEPYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies whether language models really need to work out the answers to what-if (counterfactual) questions. The authors propose having a model first turn a story into a structured description, then letting a program calculate the answer. The results suggest this works especially well when some facts are missing and need to be inferred. The approach performs strongly on CounterBench, and the authors use additional controlled tests to understand where the improvement comes from.

Critical strengths

[S1] The comparison is well designed. Both answerers receive the same extracted information, which makes it easier to tell where the improvement comes from.

Critical weaknesses

[W1] The approach relies heavily on reading the story correctly, and its robustness is still questionable. Simply rewording the same problems reduces exact rule extraction from 82.8% to 44.9%. On CLadder, the proposed pipeline also falls behind the one-pass baseline (86.7% vs. 89.0%), with most examples requiring a fallback. This raises concerns about how well the approach would work beyond the specific writing patterns used in the experiments. The supported problem types are also quite limited, covering only simple Boolean rules rather than more general causal relationships.

[W2] The practical benefit of the full system is less clear than the reported execution speed suggests. The near-zero execution time excludes the cost of extracting information from the story. On CounterBench-alpha, the complete pipeline takes roughly twice as long as one-pass prompting and uses substantially more input tokens. A stronger reasoning model also nearly closes the accuracy gap. The paper would be more convincing with an end-to-end comparison against stronger reasoning models, including total time, token usage, and accuracy. The current results establish the value of exact execution more clearly than the efficiency of the complete approach.

Questions for authors

[Q1] How much of the performance loss on reworded stories and CLadder could be recovered with a more robust extraction method?

[Q2] How does the full pipeline compare with a strong reasoning model answering directly from the original stories, considering both accuracy and total inference cost?

[Q3] Since the reported CounterBench results use an official answer key containing identified errors, how sensitive are the main benchmark comparisons to correcting those labels?

4 comments

Thank you for the review. The original 14B reader had the weakness you identified: rewording reduced extraction accuracy and the repeated reading calls made the full pipeline expensive. We have now tested stronger readers and compared the complete pipeline with the same reasoning model answering directly. The results are in the revised abstract, Section 5.7 and Appendix E (new text in blue).

W2 / Q2 (strong reasoning model and end-to-end cost). Qwen3-30B-A3B-Thinking either reads each story once for the executor, with no fallback, or answers the raw story:

Qwen3-30BE2RewordedCB alphaCB backdoorCLadder
Reads once + executor99.710093.887.596.3
Answers directly94.595.791.488.596.7
Exact McNemar p8e-51e-4.005.5.23
Median generated tokens (k), pipeline / direct3.4 / 6.34.1 / 6.82.0 / 4.42.0 / 3.51.8 / 3.8

This is the first of two runs. Counting the reader, total input plus generated tokens are 25–39% lower than direct answering. Median seconds per story, pipeline versus direct, are 60 vs 103 on E2, 59 vs 119 on the rewording, 30 vs 60 on alpha, 26 vs 60 on backdoor and 26 vs 67 on CLadder. The executor adds under a millisecond per story on these sets. The pipeline is significantly more accurate on E2, rewording and alpha in both runs (second: 99.7 vs 93.8, 100 vs 95.4, 94.2 vs 92.3); backdoor and CLadder are not significantly different. Across both CounterBench splits it scores 92.8% and 93.1% in two runs, compared with a published best of 91.8% from a 671B model.

E5 checks whether the gap disappears as reasoning models become stronger. We built 160 problems in which observations jointly constrain 4–16 free roots, with protocol and predictions committed before testing. With gpt-oss-120b reading, the pipeline scores 78.1%. The same model gets 55.6% answering the raw story at low effort (p = 3e-5) and 65.6% when handed the exact structured problem. The latter removes reading errors, but not the computation gap. At high effort it generates a median 37,721 tokens versus 3,745 for the pipeline, with 110/160 answers unfinished at the service limit. This is not a completed high-effort accuracy comparison.

W1 / Q1 (robustness to phrasing). With the original 14B reader, exact extraction fell from 82.8% to 44.9% under rewording; on CLadder the pipeline scored 86.7% against 89.0% for one pass, with most items sent to fallback. With gpt-oss-120b, exact extraction reaches 99.7% on E2 and 100% on the rewording. The 14B pipeline also recovers to 91.4% on the rewording by re-reading when extraction is unusable, compared with 67.1% for its one-pass prompt.

The stronger readers need no fallback. One Qwen3 read gives 100% on reworded E2 and 96.3% on CLadder; Llama-3.3-70B gives 97.5% and 97.2% (Llama direct on CLadder: 94.0%). CLadder was independently written in everyday wording. The causal rules remain deterministic and Boolean; we have not tested general causal models.

Q3 (CounterBench audit). We flagged 80 answers from the stories alone, before seeing model predictions. The flags recur when the problems are rebuilt from separately released model files. Scores below use official key / 80 flagged items excluded / 60 contradictions relabelled and 20 unevaluable items excluded:

SystemCB alphaCB backdoor
Qwen3 reader + executor93.8 / 99.3 / 99.387.5 / 100 / 100
Qwen3 direct91.4 / 94.9 / 94.788.5 / 100 / 100
Qwen2.5-14B pipeline92.4 / 97.8 / 97.987.5 / 100 / 100
Qwen2.5-14B one-pass89.4 / 92.4 / 92.079.0 / 86.3 / 85.6

On the 1,120 unflagged items, Qwen3 has 46 vs 5 discordant items in favor of the pipeline (p = 2e-9), and 35 vs 2 in the second run. The 14B pipeline remains ahead of its one-pass control under all three keys. The key changes absolute scores, not the main matched-model result.

Dear Reviewer dEPY, thank you again for your review. Page numbers are those of the revised PDF.

The paper would be more convincing with an end-to-end comparison against stronger reasoning models, including total time, token usage, and accuracy.

This comparison is Table 4 (page 10). Qwen3-30B-A3B-Thinking either reads each story once for the executor, with no re-reading, or answers the story itself, and the reading is counted in the cost. The complete pipeline scores 99.7% against 94.5% on E2, 100.0% against 95.7% on the reworded stories and 93.8% against 91.4% on CounterBench alpha. It is never significantly less accurate, and it uses about half the generated tokens and 41–66% less time (Section 5.7).

Both answerers receive the same extracted information, which makes it easier to tell where the improvement comes from.

Today's revision adds one analysis that uses this design to locate the improvement for the reasoning model. On the 267 E2 stories where its own extraction is exactly the stored problem, it answers 15 wrongly from the story and 3 once it is handed the extracted problem (Appendix E, page 27). For 14 of the 15, its extraction is exact and its answer to the extracted problem is correct. These are 14 of the 18 E2 stories the model loses when it answers directly (Table 4). The answers are lost when it reads and computes in one pass, and separating the two steps recovers them.

Changed in the paper. Appendix E has a new passage and table with this analysis ("Reading, solving, and both in one pass", page 27). Its result is now stated in the contribution list (page 3), at the end of the third paragraph of Section 5.7 (page 9) and in the Conclusion (page 10).

Thanks for the author's response. Overall, W1's responses simply repeat my questions, and W2's response didn't address efficiency? I would like to learn from other reviewers regarding their options of the robustness of proposed framework and evaluations.

Moreover, did the author team change the ICLR template? The font looks different with the default style.

Thank you for checking. We did not change the template. The paper uses the official iclr2027_conference.sty, unmodified, with the template's own \usepackage{iclr2027_conference,times}. Our TeX engine did not apply the template's Times font and fell back to Latin Modern, which is wider, so the earlier PDFs had less room on each page, not more. We have re-uploaded the PDF with the template's Times font. The content is unchanged. With the new font, Table 4 is on page 9, Table 5 on page 12 and the passage "Reading, solving, and both in one pass" (Appendix E) on page 25.

On efficiency (W2): Table 4 measures it end to end. Reading each story once for the executor uses about half the generated tokens of the same reasoning model answering directly (3,411 against 6,264 per story on E2) and 41–66% less time. On the extracted problems the executor needs under a millisecond, against a median of 64 seconds for gpt-oss-120b.

Reviewer_Jke1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper separates counterfactual QA over explicit causal stories into reading, where an LLM extracts a boolean causal model, and exact execution by a deterministic program. Its main tool is a matched comparison: each extraction is frozen and given both to the executor and to a prompted LLM answerer, so any difference comes from the computation step alone. The paper finds that exact execution helps mainly when the answer requires inferring latent causes, that a reasoning model closes this gap at much higher cost, and that end-to-end accuracy is ultimately bounded by reading.

Critical strengths
  1. The experimental design is clean and the analysis is careful. Freezing the extraction and giving the same problem to both answerers isolates the computation step, and the controls single out latent-cause inference as where exact execution helps. The paper also states the limits of its own result: a reasoning model closes the gap, and reading caps end-to-end accuracy.

  2. Using the executor to audit CounterBench is a useful by-product: it identifies official answers that contradict their own stories, with certificates that can be checked by hand.

Critical weaknesses
  1. Limited novelty. The method combines established translate-then-solve pipelines with the textbook counterfactual recipe, and the central finding, that LLMs fail when latent causes must be inferred, has already been reported. The main addition is the frozen-extraction control.

  2. The main finding holds only in a regime that current models are leaving. The advantage of exact execution is shown only against small, quantised, non-reasoning answerers; the one reasoning model tested matches the executor, so what remains is a cost advantage. The pipeline itself is never tested with a stronger reader, and end to end it is limited by reading, which breaks once the stories move away from the benchmark's phrasing. The paper therefore does not show that separating reading from execution matters for current strong models, which limits its impact.

Questions for authors
  1. Why does the reasoning model close the gap? Do its reasoning traces carry out something like the abduction-action-prediction structure (enumerating or constraining the latent roots), or a different and possibly better procedure? An analysis of the traces on E2 would show what the executor adds beyond what reasoning already does.

  2. How large must a reasoning model be to close the gap? A size sweep within one reasoning family (and, as a control, a non-quantised 30-70B non-reasoning model with its chat template) would show whether the ~20-point gap is a property of non-reasoning computation or of the small quantised models used.

  3. Can the gap reverse with stronger reasoning models? If a frontier reasoning model (e.g. Kimi-K3) answers directly from the raw story, it may beat the pipeline, because the extraction schema and reader errors cap what the executor can do (as in Sec. 5.6). Did you compare a reasoning model reading the raw story against the pipeline, or use a reasoning model as the reader?

  4. Counting the reader's own LLM calls (about four per item, Sec. 5.1), how much of the executor's cost advantage over a reasoning model remains end to end?

3 comments

Thank you for the detailed review and concrete questions. The revised abstract, Section 5.7 and Appendix E contain the new experiments (text in blue).

Q3 (a reasoning model as reader, versus answering the story). We ran Qwen3-30B-A3B-Thinking both ways. It reads each story once for the executor, without re-reading or fallback, or answers the raw story itself:

Qwen3-30BE2RewordedCB alphaCB backdoorCLadder
Reads once + executor99.710093.887.596.3
Answers directly94.595.791.488.596.7
Exact McNemar p8e-51e-4.005.5.23
Median generated tokens (k), pipeline / direct3.4 / 6.34.1 / 6.82.0 / 4.42.0 / 3.51.8 / 3.8

This is the first of two runs. The pipeline is significantly more accurate on E2, rewording and alpha in both (second run: 99.7 vs 93.8, 100 vs 95.4, 94.2 vs 92.3). Backdoor and CLadder are not significantly different. With Llama-3.3-70B, a non-reasoning reader, the pipeline reaches 99.7% on E2 and 97.2% on CLadder, versus 81.8% and 94.0% for Llama answering directly. On the reworded stories one read gives 100% with Qwen3 and 97.5% with Llama.

The CounterBench audit also affects this comparison. We flagged 80 answers that contradict their stories or are unevaluable as written, using the stories alone before inspecting model predictions. On the other 1,120 items Qwen3 scores 99.4% with the pipeline and 95.7% directly (46 vs 5 discordant items, p = 2e-9). Under the official key the pipeline scores 92.8% and 93.1% across both splits in two runs, versus the previously published best of 91.8% from a 671B model.

W2 (a regime current models are leaving). We could not run Kimi-K3, but E5 tests whether the gap returns for a stronger model when observations constrain many hidden causes jointly. It contains 160 problems with 4–16 free roots, under a protocol and two predictions committed before testing. With gpt-oss-120b reading, the pipeline scores 78.1%, while the same model answering the raw story at low effort scores 55.6% (p = 3e-5; always answering "unsure" gives 50%). Given the exact structured problem, removing reading errors, the model reaches 65.6%. At high effort it generates a median 37,721 tokens per story, versus 3,745 for the pipeline, and 110/160 answers remain unfinished at the service limit. We do not treat these as completed high-effort errors. E5 stays within Boolean rules.

W1 (novelty). Logic-LM, SatLM and LINC show that translating text into programs can help, but their comparisons do not hold reading fixed across answerers. Ours gives both the LLM and executor the same frozen extraction, isolating computation. E5 tests the resulting latent-cause finding under harder joint constraints: even an exact structured input leaves gpt-oss-120b below the read-and-execute pipeline. Executable Counterfactuals trains models to improve this inference; here we test what exact computation adds without training another reasoner. The executor itself is not a new algorithm.

Q1 (why reasoning closes E2). The gpt-oss traces often carry out abduction: every trace lists roots, about two thirds constrain them from observations, and 68% of "unsure" problems are split into cases. These are labels from the first of two LLM judges (agreement κ = .81–.95). gpt-oss handles E3 with up to ten free roots (Section 5.4) but falls behind on E5; the executor performs the enumeration exactly.

Q2 (model size). On frozen E2 problems the executor scores 96.0%. Full-precision non-reasoning Phi-4 14B, Llama-3.3-70B and Qwen2.5-72B score 87.4%, 85.8% and 88.9% (p ≤ 2e-4 each). In the Qwen3 reasoning family, 8B is below the executor (89.2%, p = 3e-5), while 30B reaches 95.7% (Section 5.5, Appendix E).

Q4 (end-to-end cost). With the reader's call counted, the Qwen3 pipeline uses 25–39% fewer input plus generated tokens than direct answering on the five earlier sets and 41–66% less median time per story across all six Section 5.7 sets.

Thank you for the thorough response. The new experiments address most of my questions: the stronger readers settle my concerns about robustness and end-to-end cost, and the trace analysis and size sweep clearly explain why reasoning models close the gap. I appreciate the effort and care that went into this work.

My remaining reservation is about the size of the contribution. With a reasoning model that completes its answers (Qwen3-30B-Thinking), the advantage on E2 and CounterBench comes to a few points on sets that are near ceiling, and the main benefit shifts to cost. The accuracy gap is more noticeable on E5, which is designed to require enumeration over many latent roots; there, the margin over Qwen3-30B is 5-7 points and not significant in the second run. The gpt-oss comparison is run at low effort, since the high-effort arm mostly fails to finish.

As I read it, the main finding of the paper is that when a problem reduces to combinatorial search, an exact solver is both more accurate and cheaper. The paper supports this well, although I regard the increment over translate-then-solve and tool-offloading work as modest. I am keeping my score, and I want to recognize the careful experimental work.

Thank you for the follow-up. Page and section numbers are those of the revised PDF.

As I read it, the main finding of the paper is that when a problem reduces to combinatorial search, an exact solver is both more accurate and cheaper. [...] I regard the increment over translate-then-solve and tool-offloading work as modest.

We agree that this is what E5 shows. E2 leaves at most three roots free, so there is no long search in it, and there the gain has another source. The same model gets more answers right when its reasoning is broken into two steps, reading the story into a problem and then computing that problem.

The frozen extraction lets us count this story by story, on the stories where the model's own extraction is exactly the stored problem. Wrong answers:

ModelStoriesStory, in one passSame model, handed the extracted problemExecutor
Qwen3-30B-A3B-Thinking2671530
Llama-3.3-70B25244270

The 267 stories contain 15 of the 18 E2 stories the reasoning model loses when it answers directly (the complete pipeline loses 1, Table 4). It computes 14 of those 15 correctly once the extracted problem is in front of it (exact McNemar 14 stories against 2, p = 0.004, and 15 against 3 on the 266 stories of a second run, p = 0.008).

A comparison of a pipeline with direct prompting gives one number, and that number cannot say whether the model misread the story, miscomputed the problem, or lost the answer doing both at once. The stored extraction and the exact answer to it tell the three apart for each story, and for the reasoning model 14 of the 15 wrong answers are of the third kind.

Changed in the paper. Appendix E has a new passage and table with this analysis ("Reading, solving, and both in one pass", page 27). Its result is now stated in the contribution list (page 3), at the end of the third paragraph of Section 5.7 (page 9) and in the Conclusion (page 10).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revision update

We thank the three reviewers. The reviews raise three common questions, and we have added experiments to address each. The new results are in Section 5.7 (Table 4) and Appendix E. Added text is in blue.

  1. Does exact execution still help with stronger reasoning models? We ran Qwen3-30B-A3B-Thinking in both roles: it either reads each story once for the executor, with no fallback, or answers the raw story directly. In two runs the pipeline is significantly more accurate on E2 (99.7 vs 94.5 in the first run), its rewording (100 vs 95.7), and CounterBench alpha (93.8 vs 91.4). It is not significantly worse on backdoor or CLadder. Across both CounterBench splits it scores 92.8% and 93.1% in the two runs, above the best published result of 91.8%. We also built E5, 160 new problems whose observations jointly constrain 4–16 hidden roots. With gpt-oss-120b reading, the pipeline scores 78.1%, against 55.6% when the same model answers directly at low effort (p = 3e-5). Even when handed the exact structured problem, it reaches only 65.6%. We committed the E5 protocol and two predictions before any test-set run.

  2. What does the complete pipeline cost? Counting the reader, the Qwen3 pipeline uses 25–39% fewer input plus generated tokens than the same model answering directly on the five earlier test sets. Median time per story is 41–66% lower across all six sets in Section 5.7. The executor itself contributes very little to that time.

  3. Does reading hold up away from our phrasing? A single read with no fallback is enough. On the reworded stories the pipeline scores 100% with Qwen3-30B reading and 97.5% with Llama-3.3-70B. On CLadder, written independently in everyday language, it scores 96.3% and 97.2%. The tested causal rules remain deterministic and Boolean, we do not claim that these results extend to arbitrary causal models.

We also repeated the main model comparisons and reported CounterBench scores under the official key and two audit-based alternatives (Appendix E).

The revised PDF adds one analysis, "Reading, solving, and both in one pass" (Appendix E, page 27), and states its result in the contribution list (page 3), in Section 5.7 (page 9) and in the Conclusion (page 10). With the extraction frozen and an exact answer for every frozen problem, a wrong answer to a story can be traced to the reading, to the computing, or to doing both in one pass. On the 267 E2 stories where Qwen3-30B-A3B-Thinking's own extraction is exactly the stored problem, it answers 15 wrongly from the story and 3 once it is handed the extracted problem (exact McNemar 14 stories against 2, p = 0.004, and 15 against 3 on the 266 stories of a second run, p = 0.008). For 14 of the 15, the model's extraction is exact and its answer to the extracted problem is correct.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.