- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper introduces AbductionBench, a unified framework that integrates 27 existing datasets and task adaptations for evaluating abductive reasoning. It covers 10 domains, hypothesis generation and selection, and 3 evidence-delivery modes: static, passive, and interactive. The authors evaluate 8 LLMs and Jev. The experiments compare task formats, models, and prompting methods, and examine which process features are associated with reasoning success or failure.
Critical strengths
-
The evaluation framework has engineering value. The authors unify input/output interfaces across many existing datasets. This could reduce the effort required to adapt datasets and build evaluation pipelines.
-
The observations in §5.4 and §5.5 are interesting.
Critical weaknesses
-
The presentation are poor. Many results are presented in a highly compressed format. For example, Figure 1 would benefit from a vector format. Table 1 places too much information on one page, with numerical entries that are too small to read comfortably. Besides, the format of the whole paper does not meet ICLR’s template.
-
The main text provides insufficient detail on how the experiments are conducted. This make the results difficult to read and verify.
-
The paper does not sufficiently explain what new insights the unified framework enables. For example, the task-dependent benefits of CoT have already been systematically studied in [1]. The association between longer reasoning chains and lower accuracy has also been examined in [2]. This paper does not offer deeper explanations.
-
Key statistics and citations have not been adequately checked.
- Table 1 reports for Qwen-3.5-27B on UniADILR-HGc under CoT. If this represents the equally weighted mean and standard deviation of 3 nonnegative accuracy scores (as suggested by the caption), the reported values are mathematically incompatible. See Question 1 below.
- Table 2 on page 16 cites CLIMATE-FEVER as Diggelmann et al. (2026b), while the corresponding reference lists 2021.
- Sections D.1 and D.2 contain identical content.
The paper’s main contribution is the integration of existing dataset interfaces. It does not introduce a new constructed dataset. Although they conduct many experiments, most main conclusions have limited originality. Given these limitations and the presentation problems, I consider the current work more suitable as a technical report than as an ICLR paper.
Questions for authors
-
How was the result in Table 1 calculated? For three accuracy scores , the sample standard deviation must satisfy . Therefore implies . Is this a transcription error or a calculation error? Have the other reported statistics been checked for the same issue?
-
Could the temporal patterns in §5.4 help guide early error detection or correction? Since unresolved contradictions are identified using the full trajectory, whether a related signal could be detected from partial traces and used to trigger early verification or correction? This would help clarify the potential applications of the observed patterns.
-
How much of the decline in relevance in §5.5 is explained by changes in the composition of the remaining episodes? The authors acknowledge that later turns include only episodes that have not ended. The downward trend in Figure 4a may therefore mainly reflect easier cases ending earlier, rather than question quality declining within the same case. Can the authors decouple the effects of these two components through experiments?
[1] Sprague et al. To CoT or not to CoT?, arxiv.org/abs/2409.12183
[2] Feng et al. What Characterizes Effective Reasoning?, arxiv.org/abs/2509.19284