- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper introduces EICR, a framework for counterfactual reasoning that keeps all compatible causal explanations. An oracle generates structured certificates containing factual checks, counterfactual outcomes, answer sets or bounds, and identification labels, which are used for supervised fine-tuning. The authors also introduce EIBench, a synthetic benchmark with code, structural causal model, and natural-language representations. Experiments report improvements in answer recovery and reductions in false certainty over several baselines.
Critical strengths
- The distinction between hidden-state ambiguity and target ambiguity is useful: several compatible explanations can still imply the same counterfactual answer.
- The Answer-Set SFT baseline is relevant because it helps separate supervision of the final answer and identification label from supervision of the complete supporting trace.
Critical weaknesses
- The conceptual and methodological contribution of this paper appears to be very limited. Once a complete candidate mechanism family and admissible completion space are supplied, retaining the explanations consistent with the observation and comparing their intervention outcomes is a direct solution to the stated problem. The training objective is standard supervised fine-tuning on oracle traces, and the theoretical results largely restate the consequences of exhaustive enumeration under exact execution. The coupon example illustrates familiar underdetermination, but does not establish why the proposed certificate framework is needed or what substantial new capability it provides beyond existing executable counterfactual reasoning.
- The benchmark provides insufficient evidence for the broader value of the approach. A synthetic benchmark is appropriate for controlled evaluation, but here the tasks, oracle, certificate format, and supervision are all built around the proposed procedure. Improvements on this benchmark only proves performance within that construction. The parallel code, SCM, and natural-language versions share the same underlying tasks, so they do not provide independent evidence of broader task generalization. Evaluation on an independently constructed task set or clearly specified structural extrapolation would make the contribution more convincing.
- The oracle and data-generation protocol are not sufficiently justified or specified. The authors introduce sampling mechanisms and executing completions, but leave the mechanism grammar, variable and domain sizes, completion-space distributions, template counts, and split sizes unclear. For bounded tasks, the paper refers to symbolic interval procedures without specifying the function restrictions or solver guarantees needed to establish sharp bounds. The soundness and completeness propositions assume a complete candidate space and correct execution, and they do not validate how these assumptions are ensured in the benchmark. The construction of unresolved cases also needs clarification, including whether incompleteness is explicitly signaled in the input.
- The certificate-validity results appear inconsistent with the stated definition. Table 4 reports 84.7 certificate validity but only 79.6 identification accuracy for the full model. A valid certificate is described as including the correct identification label, so certificate validity should not exceed identification accuracy under the same evaluation population and averaging.
- The training comparisons and ablations are difficult to assess. Concrete optimization settings, training steps, decoding budgets, self-consistency sample counts, and exact checkpoint identifiers are missing. It is also unclear how removing certificate components changes the training targets and evaluation: scoring an intentionally omitted field against the full certificate could make part of the ablation degradation automatic.
Questions for authors
See weakness.