Separating Reading from Execution in Counterfactual Question Answering

A counterfactual question over an explicit causal story asks what would have happened had one variable been different. A language model answering in one pass does two jobs at once: it reads the story into a causal model and it computes the answer. We separate them a schema constrained call extracts a boolean causal model, a deterministic executor enumerates every world that model allows and use the separation to ask when exact execution is worth its cost. Crossing the two stages answers that. Handed perfectly parsed rules, bare in-text answering still collapses, so the computation rather than the read is the weak link. But handed the same extracted model the executor receives, a prompt tuned for the task is not distinguishable from it where the question already fixes the story’s hidden causes: the paired difference is small and its interval spans zero, resolving neither an advantage nor equivalence. The advantage appears where answering requires abduction, inferring un- observed causes before intervening. On 325 pre-registered abduction items the executor scores 95.7% against 74.8% for the strongest separated prompt, 21 percentage points, and 97.8% against 79.9% on the 229 items whose an- swer the story determines. A prompt we built specifically for abduction did worse, not better, because it declines on half the determined items; we report that as a diagnostic, and no finite prompt search can show that no prompt would close the gap. The same executor audits the benchmark it is scored on: it certifies that 60 of CounterBench’s official answers contradict their own stories and flags 20 more it cannot evaluate, each with a hand-checkable certificate. As supporting evidence the pipeline beats the strongest prompting baseline at equal scale on CounterBench’s official splits, 92.4% against 89.4%, while on real-language CLadder a heavily scaffolded prompt edges two points ahead of it.