acceptodds
Withdrawn from ICLR 2027

AbductionBench: A Unified Framework for Evaluating Abductive Reasoning in Large Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Abductive reasoning is the inference of plausible explanations for observed evidence, particularly when information is incomplete or supports multiple competing hypotheses. Evaluating this capability requires more than a single task: a system may need to generate an explanation, discriminate among competing hypotheses, revise its hypotheses as evidence arrives, or decide what evidence to seek next. Yet these capabilities are typically evaluated through isolated benchmarks with different task formulations and scoring procedures, so claims that a model is “good at abduction” may reflect only a narrow aspect of this capability. We introduce AbductionBench, a unified framework that brings this fragmented landscape into a common evaluation space. Its taxonomy characterizes abductive tasks by hypothesis format, application domain, and evidence-delivery mode, covering generation and selection tasks across static, passive, and interactive settings while preserving their task-specific execution structure. We instantiate this framework on 27 datasets spanning ten domains and evaluate eight LLMs from four model families—GPT, Gemini, Gemma, and Qwen—alongside Jev, a non-LLM System One decision system evaluated on single-choice selection tasks. Beyond harmonizing heterogeneous tasks, AbductionBench separates final-answer evaluation from analysis of the observable reasoning traces and interaction histories produced alongside those answers. We analyze written traces as observable model outputs rather than faithful representations of internal computation, characterizing evidence use, hypothesis comparison, uncertainty, contradiction handling, and answer commitment. Chain-of-thought is not uniformly beneficial across models and tasks, and within datasets, longer traces are associated with lower accuracy. Jev matches or exceeds small LLMs when selecting among explanations over compact evidence but falls substantially behind on longer cases, suggesting that candidate discrimination and extended evidence integration place different demands on a system. In interactive tasks, the proportion of actions judged relevant declines at later steps, and a relevant first question is associated with a correct final answer; in the passive task, models rarely revise their initial hypothesis as new evidence arrives. Together, these findings highlight the value of evaluating abductive reasoning through both final answers and observable evidence use, hypothesis revision, and information gathering.

Primary area:
datasets and benchmarks
Keywords:
abductive reasoning, evaluation framework, benchmark, large language models, hypothesis generation, hypothesis selection, inference to best explanation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_9pdKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces AbductionBench, a unified framework that integrates 27 existing datasets and task adaptations for evaluating abductive reasoning. It covers 10 domains, hypothesis generation and selection, and 3 evidence-delivery modes: static, passive, and interactive. The authors evaluate 8 LLMs and Jev. The experiments compare task formats, models, and prompting methods, and examine which process features are associated with reasoning success or failure.

Critical strengths
  1. The evaluation framework has engineering value. The authors unify input/output interfaces across many existing datasets. This could reduce the effort required to adapt datasets and build evaluation pipelines.

  2. The observations in §5.4 and §5.5 are interesting.

Critical weaknesses
  1. The presentation are poor. Many results are presented in a highly compressed format. For example, Figure 1 would benefit from a vector format. Table 1 places too much information on one page, with numerical entries that are too small to read comfortably. Besides, the format of the whole paper does not meet ICLR’s template.

  2. The main text provides insufficient detail on how the experiments are conducted. This make the results difficult to read and verify.

  3. The paper does not sufficiently explain what new insights the unified framework enables. For example, the task-dependent benefits of CoT have already been systematically studied in [1]. The association between longer reasoning chains and lower accuracy has also been examined in [2]. This paper does not offer deeper explanations.

  4. Key statistics and citations have not been adequately checked.

    • Table 1 reports 14.3±28.914.3\pm28.9 for Qwen-3.5-27B on UniADILR-HGc under CoT. If this represents the equally weighted mean and standard deviation of 3 nonnegative accuracy scores (as suggested by the caption), the reported values are mathematically incompatible. See Question 1 below.
    • Table 2 on page 16 cites CLIMATE-FEVER as Diggelmann et al. (2026b), while the corresponding reference lists 2021.
    • Sections D.1 and D.2 contain identical content.

The paper’s main contribution is the integration of existing dataset interfaces. It does not introduce a new constructed dataset. Although they conduct many experiments, most main conclusions have limited originality. Given these limitations and the presentation problems, I consider the current work more suitable as a technical report than as an ICLR paper.

Questions for authors
  1. How was the result 14.3±28.914.3\pm28.9 in Table 1 calculated? For three accuracy scores x1,x2,x3≥0x_1,x_2,x_3\geq 0, the sample standard deviation must satisfy s≤3xˉs\leq\sqrt{3}\bar{x}. Therefore xˉ=14.3\bar{x} = 14.3 implies s≤24.77<28.9s \le 24.77 < 28.9. Is this a transcription error or a calculation error? Have the other reported statistics been checked for the same issue?

  2. Could the temporal patterns in §5.4 help guide early error detection or correction? Since unresolved contradictions are identified using the full trajectory, whether a related signal could be detected from partial traces and used to trigger early verification or correction? This would help clarify the potential applications of the observed patterns.

  3. How much of the decline in relevance in §5.5 is explained by changes in the composition of the remaining episodes? The authors acknowledge that later turns include only episodes that have not ended. The downward trend in Figure 4a may therefore mainly reflect easier cases ending earlier, rather than question quality declining within the same case. Can the authors decouple the effects of these two components through experiments?

[1] Sprague et al. To CoT or not to CoT?, arxiv.org/abs/2409.12183

[2] Feng et al. What Characterizes Effective Reasoning?, arxiv.org/abs/2509.19284

Reviewer_2n6yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AbductionBench, integrating 27 existing datasets across ten domains to evaluate LLM abductive reasoning. It assesses eight LLMs on hypothesis generation and selection under static, sequential, and interactive settings, using outcome and reasoning-trace metrics. Results highlight difficulties in generating explanations, gathering relevant evidence, and revising hypotheses.

Critical strengths
  1. This work integrates 27 datasets across ten domains, this broad coverage enables evaluation across diverse abductive reasoning tasks. It also includes multiple evaluation settings, covering hypothesis generation and selection under static, passive, and interactive evidence delivery.
  2. The analysis goes beyond final accuracy. It examines evidence use, hypothesis comparison, uncertainty, and revision to characterize model behavior, and investigates whether models revise predictions as new evidence arrives.
Critical weaknesses
  1. My major concern is that the novelty and contribution of this work are limited. The benchmark primarily integrates existing datasets, while several evaluation dimensions overlap with prior work. In particular, CEDAR-GRPO already examines evidence coverage, directionality, branching, backtracking, and uncertainty.

  2. The paper provides no systematic evaluation of LLM judge quality or complete agreement statistics. This undermines confidence in the subjective trace metrics that underpin many conclusions.

  3. Evaluation uses only 50 records per dataset. Reporting some CoT results only on completed outputs can also bias comparisons when completion rates vary across models or task difficulty.

  4. Minor issues for the presentation. The text in table one is way too small and almost impossible to read.

Questions for authors

N/A

Reviewer_m5nLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The author introduces AbductionBench, a benchmark that evaluates abductive reasoning with a carefully designed taxonomy. The author categorizes a task by hypothesis format, application domain, and evidence-delivery mode. The author also analyzes written traces on hypothesis comparison, uncertainty, contradiction handling, and answer commitment. The author also reports interesting findings, such as longer traces having lower accuracy and Jev matching small LLMs on compact evidence. Overall, this paper covers many aspects of the abduction reasoning landscape and many interesting results and insights.

Critical strengths
  1. I appreciate the comprehensiveness of the paper in terms of the task, the taxonomy, and the model. The inclusion of the latest system-one model, Jev, and its comparison with other LLMs is very exciting. The taxonomy is carefully designed and well executed.
  2. The author makes much effort to combine and convert existing datasets for abduction reasoning use.
Critical weaknesses

My biggest concern with this paper is that its contribution is thin within ICLR's scope. I agree that the taxonomy of abduction reasoning and the repurposing of existing datasets as a research resource are valuable to the research community overall, but they may fit better in a linguistics or logic conference or journal. On the other hand, I'd like to see more insights on the machine-learning side of abduction reasoning. For example, what deep insights could help develop abduction reasoning methods for LLMs? Or can this dataset also be used for post-training to improve abduction reasoning in LLMs?

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.