acceptodds
Under review as a conference paper at ICLR 2027

Characterizing Abductive Reasoning Failures in Large Language Models with Elenchos

Abstract

Large language models (LLMs) are highly capable of reasoning from known rules to their consequences, but their ability to reason in the opposite direction – from observed behavior to the rules that produced it – remains poorly understood. Recent work has characterized this transition from observations to hypotheses about underlying rules as an “abductive jump” and argued that it remains challenging for current reasoning systems. To investigate this challenge, we introduce Elenchos, a framework for evaluating abductive inference in controlled executable formal environments, instantiated here in a dependently typed -calculus kernel. Given a pair of kernels, exactly one of which is sound and the other intentionally mutated, agents interact with both through a black-box interface to determine which of the two kernels is sound and identify the mutations responsible for observed behavioral differences. Evaluating frontier and mid-tier LLMs on Elenchos reveals a striking dissociation between detecting an alteration and identifying its cause: models frequently determine which kernel is sound and gather probes that uniquely distinguish competing mutations, yet often fail to identify the mutations responsible for the observed behavioral differences. To isolate the mechanisms driving this breakdown, we conduct diagnostic ablation studies on GPT-5.4 (high reasoning). These show that forward simulation of mutated systems is substantially harder than simulation of the sound system, but that simulation accuracy does not predict attribution accuracy during interactive reasoning. Remarkably, when the predicted consequences of competing hypotheses are provided explicitly, the model identifies the responsible mutation with near-perfect accuracy. Together, these results point to two interacting bottlenecks in abductive reasoning: deriving the behavioral consequences of competing hypotheses and integrating those consequences with accumulated evidence during interactive attribution.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM, reasoning, abductive reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_H7RgOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Elenchos, a framework for evaluating abductive reasoning in LLMs through interactive diagnosis of mutated formal systems.

Critical strengths

From what I understand, the research problem seems quite interesting.

The experiments are comprehensive, covering 13 models with different difficulty levels and mutation configurations.

Although I am not familiar with this research area, I can see that the authors have put considerable effort into this work, especially given the detailed Supplementary Material and the overall presentation. I would encourage the authors to explain their work in a more intuitive and accessible way, which I believe would help readers from different backgrounds better understand its contributions.

Critical weaknesses

My main concern is the clarity of the presentation. The paper focuses on a rather specialized research problem, making it difficult for readers unfamiliar with the area to understand the motivation and contributions from the title and abstract. They choose 'other topics in machine learning (i.e., none of the above)'. I suggest explaining the problem more intuitively in the abstract and including a simple illustrative example or figure to help readers understand the task.

I am not very familiar with the formal methods background of this paper, so I find it difficult to assess the technical details of the proposed framework. From the experimental perspective, one potential limitation is that the detailed ablation studies are conducted only on GPT-5.4, which raises questions about generalizability. However, my main concern remains the accessibility of the presentation, as I believe the paper would benefit from a clearer and more intuitive explanation of the research problem and its significance.

Questions for authors

Could you explain abductive reasoning more directly? Or provide some practical applications.

2 comments

We thank the reviewer for the candid assessment and for pointing out that the motivation and conceptual background may not be sufficiently clear to readers outside the field. Below, we explain abductive reasoning more directly and illustrate our findings with a practical example. We have revised the abstract and Introduction accordingly.

Concretely, Section 2 now describes the task in plain words before the formal definitions: two copies of a program behave differently, and the agent, given a list of the ways one copy may have been altered, must find out by experiment which alteration explains the difference. The Introduction now distinguishes selective from creative abduction, Section 2 states that Elenchos evaluates selective abduction, and the supplementary material contains a worked session with real kernel outputs (Table S12), referenced from Section 2.

What is abductive reasoning?

Abductive reasoning infers the most plausible explanation for an observation. It complements deduction, which derives consequences from given rules, and induction, which generalizes patterns from observations. Two forms of abduction are commonly distinguished (Magnani, 2000, 2009; Stanford Encyclopedia of Philosophy, "Scientific Discovery"): selective abduction chooses an explanation from a set of known hypotheses, whereas creative abduction generates new explanatory hypotheses, as in scientific research. Elenchos evaluates selective abduction. Abductive reasoning is central to medical diagnosis, software debugging, cybersecurity, scientific discovery, and fault diagnosis in engineering.

A practical example: medical diagnosis.

Consider an LLM acting as a diagnostician. A patient presents with symptoms; the LLM considers a list of candidate diseases, orders tests (blood pressure, blood work, and so on), and must name the disease that explains all results. In Elenchos, the patient corresponds to the corrupted kernel, the candidate diseases to the catalog of possible mutations, and the tests to the probes.

Translated to this setting, our findings suggest that the LLM would reliably recognize that the patient is ill (detection) but would often fail to name the exact disease (attribution), particularly when several diseases interact. Strikingly, it often orders tests whose results already single out the correct disease and still names a wrong one. Our ablations locate this failure: when GPT-5.4 (High) is told what each candidate disease would produce on the tests it ordered, it resolves 0.99 of these cases; when it must derive this itself, its derivations single out the correct disease in only 0.07. The bottleneck is therefore not comparing test results with expectations, but reliably deriving what each candidate disease implies for the tests performed.

These results are not specific to GPT-5.4. The detection-attribution gap appears in all 13 models of our main evaluation, which span four model families (OpenAI, Google, Anthropic, Mistral). We also repeated the ablations on Claude Haiku 4.5, Mistral Large 3 and Gemini 2.5 Flash: when told what each candidate would produce, they resolve 0.54-0.67 of these cases, and when they must derive it themselves, their derivations single out the correct one in only 0.00-0.01. Whether the findings carry over from our formal setting to domains such as medicine remains open.

Thanks for the additional explanations from the authors. I now understand the scenario. I think this problem is interesting and novel. However, I still couldn't evaluate the depth of the experiment. I think the AC could consider my score to be 3.5.

Reviewer_zs5QOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Elenchos, a framework for evaluating abductive reasoning in large language models through interactive black-box diagnosis of executable formal systems. Using a dependently typed lambda-calculus kernel, the framework requires agents to identify which of two kernels has been modified and infer the underlying mutation configuration from observed behavioral differences.

The authors evaluate 13 LLM configurations across different mutation complexities and prompt information levels. The results reveal a substantial gap between identifying a corrupted kernel and correctly attributing the responsible mutations. Further trace-based analyses distinguish evidence acquisition failures from interpretation failures. Diagnostic ablations on GPT-5.4 (high reasoning) investigate the roles of forward simulation, accumulated evidence, and explicit candidate predictions.

The findings suggest that LLMs frequently struggle to derive and use the behavioral consequences of competing hypotheses, even when sufficient diagnostic evidence has already been collected.

Critical strengths
  1. A well-motivated and controlled framework for evaluating abductive reasoning.

The paper addresses an important but relatively underexplored aspect of LLM reasoning: inferring underlying rules from observed consequences. Elenchos provides a controlled executable environment with known mutation configurations and verifiable outcomes. Unlike conventional static reasoning benchmarks, the interactive probing protocol enables a more fine-grained investigation of evidence acquisition and hypothesis attribution. This is a meaningful contribution to understanding the limitations of current reasoning models.

  1. Strong diagnostic analysis that provides insights beyond benchmark performance.

The paper goes beyond reporting accuracy by investigating why models fail. The detection-attribution gap is particularly interesting, and the trace-based analysis demonstrates that models can fail even when their own probes uniquely identify the correct mutation. The diagnostic ablations provide further insights: supplying explicit behavioral predictions for candidate mutations substantially improves attribution performance. These findings offer a useful empirical characterization of abductive reasoning failures and may inform future research on hypothesis evaluation and evidence integration.

Critical weaknesses
  1. Limited generalizability beyond the selected formal environment and mutation ontology.

Although the framework is described as substrate-agnostic, the experiments are conducted using only one dependently typed lambda-calculus kernel and a predefined ontology of 13 mutations. Furthermore, the main evaluation includes only 10 selected mutation pairs from a larger set of distinguishable configurations. It remains unclear whether the observed detection-attribution gap and the identified failure mechanisms generalize to other formal systems, programming environments, or less constrained abductive reasoning tasks. This limits the scope of the paper's broader conclusions.

  1. Limited evidence for the generality and causal interpretation of the diagnostic findings.

The detailed ablation experiments are conducted primarily on GPT-5.4 with high reasoning effort. While the detection-attribution gap is observed across multiple models, the proposed explanation of the underlying failure mechanisms is not independently validated across model families.

Moreover, the informed-reader ablation provides the model with executable predictions for candidate configurations, substantially simplifying the original task. Although the resulting performance improvement is informative, it does not fully isolate the contributions of forward simulation and hypothesis-evidence integration during interactive reasoning. Additional controlled experiments would strengthen the mechanistic conclusions.

Questions for authors
  1. Generalizability: Can the authors provide evidence that the detection-attribution gap persists in another executable formal environment or with a different mutation ontology?

  2. Mutation selection: What criteria were used to select the 10 mutation pairs for the main evaluation from the larger set of distinguishable pairs? How sensitive are the reported findings to this selection?

  3. Mechanistic interpretation: In the informed-reader ablation, candidate predictions are explicitly provided. How can the authors distinguish improvements due to reduced forward-simulation difficulty from those arising from simplified hypothesis comparison?

  4. Cross-model validation: Do the diagnostic findings obtained with GPT-5.4 (high reasoning), particularly the improvements from explicit candidate predictions, hold for other model families such as Gemini or Claude?

6 comments
AuthorsOpenReview ↗

Rebuttal 1/4

The reviewer's questions prompted new experiments whose results we find highly interesting and valuable, and which have substantially strengthened the paper: a new ablation that separates deriving the effects of candidate configurations from matching them against the evidence (Q3), and the replication of all ablations on models from three further families (Q4). We thank the reviewer for the careful and constructive assessment of our work. Below, we address each point and describe these experiments.

Q1 Generalizability: Can the authors provide evidence that the detection-attribution gap persists in another executable formal environment or with a different mutation ontology?

To date, we have evaluated Elenchos exclusively on the dependently typed λ\lambda-calculus described in the manuscript, using a fixed set of 13 singleton mutations.

In addition, we report a second, independent run on the same kernel and the same 13 mutations under a harder protocol, which was omitted from the submission for space (Table R1 below). It uses a broader range of mutation combinations and a less informative system prompt. The prompt states only that at most one kernel is corrupted and that a corrupted kernel carries one, two or three mutations, without giving their number, so that 378 configurations are admissible; the probe budget is 65. The run covers 6 singletons, 4 pairs (two of which, {ID_02, ID_12} and {ID_07, ID_13}, are not in the main evaluation) and 3 triples, with five sessions each, for 30 models from the same four families, including the 13 models of the main evaluation. In the additional clean-pair sessions, in which neither kernel is corrupted, the agent correctly reported no corruption in 0.890.89 of 150 cases; these sessions are not included below.

Importantly, the detection–attribution gap persists under this protocol. Over the 1,950 sessions with at least one mutation, kernel detection is 0.700.70 and exact attribution 0.200.20 (a gap of 0.500.50, against 0.370.37 in the main evaluation). The gap is present in every model and every family, and for the 13 models shared with the main evaluation, attribution falls from 0.280.28 to 0.110.11 while detection stays at 0.600.60 versus 0.650.65. Of the 240 misattributions at k=3k=3 in which the corrupted kernel was correctly identified, all miss at least one active mutation, and 215 report fewer than three mutations while none reports more, the same under-reporting pattern as in the main evaluation. Absolute values are not directly comparable to the main evaluation, and with five sessions per configuration and model, the individual rows of Table R1 are noisy. We emphasize, however, that all mutations considered in these experiments are derived from the original set of 13 singleton mutations and therefore do not represent an independently developed mutation ontology.

We agree that evaluating Elenchos with different mutation ontologies, as well as porting the framework to other formal systems, will be necessary to investigate its generalizability. Both represent important directions for future work.

We have also revised the manuscript to reflect this limitation more accurately. Specifically, we have replaced the term "substrate-agnostic" with "designed to be portable", since the applicability of Elenchos to other formal systems has not yet been empirically established.

AuthorsOpenReview ↗

Table R1 for Rebuttal 1/4

Table R1. Run under the harder protocol: per-model results by mutation count kk. kk: number of active mutations; nn: sessions; VV: exact correctness; VKV_K: kernel detection; VAV_A: exact attribution. Five sessions per configuration (6 singletons, 4 pairs, 3 triples); the clean-pair sessions are not included; the last rows pool all 30 models. †: model also in the main evaluation. Excluded as incomplete: claude-opus-4-7 (25 sessions), gemini-3-flash-preview (21 sessions).

ModelkknnVVVKV_KVAV_A
magistral-medium-25091300.3000.9000.300
2200.1000.8000.100
3150.0000.9330.000
magistral-small-25091300.0670.4330.100
2200.0000.8500.000
3150.0000.4670.000
ministral-3b-2512 †1300.0000.4000.000
2200.0000.3500.000
3150.0000.4670.000
ministral-8b-2512 †1300.0000.1330.033
2200.0000.0000.000
3150.0000.0000.000
ministral-14b-2512 †1300.0000.1670.000
2200.0000.0000.000
3150.0000.3330.000
mistral-large-2512 †1300.2670.6330.367
2200.0000.5500.000
3150.0000.5330.000
mistral-medium-25081300.0000.1330.000
2200.0000.1000.000
3150.0000.0670.000
mistral-medium-3-51300.1330.6670.133
2200.0500.8000.050
3150.0670.7330.067
mistral-small-26031300.0670.4670.100
2200.0000.6000.000
3150.0000.6000.000
claude-haiku-4-5 †1300.3000.9670.300
2200.0500.6000.050
3150.0000.8670.000
claude-sonnet-4-61300.3331.0000.333
2200.3500.9000.350
3150.0000.8000.000
claude-sonnet-4-6_low1300.5671.0000.567
2200.2500.7500.250
3150.2000.8670.200
claude-sonnet-4-6_high1300.6671.0000.667
2200.2500.7500.250
3150.2670.8670.267
claude-opus-4-7_high1300.8331.0000.833
2200.2000.9500.200
3150.1331.0000.133
gemini-2.5-flash †1300.2330.9330.233
2200.0500.7500.050
3150.0000.7330.000
gemini-2.5-pro †1300.1000.9670.100
2200.0500.9000.050
3150.0000.9330.000
gemini-3.1-pro-preview1300.8001.0000.800
2200.2501.0000.250
3150.4001.0000.400
gemini-3.5-flash1300.9671.0000.967
2200.3000.7500.300
3150.3330.7330.333
gpt-5.4 †1300.3001.0000.300
2200.1000.7500.100
3150.0000.6670.000
gpt-5.4-mini †1300.1000.4000.100
2200.0000.3500.000
3150.0000.3330.000
gpt-5.4-nano †1300.0670.3000.067
2200.0000.2500.000
3150.0000.0000.000
gpt-5.4_low †1300.2331.0000.233
2200.1000.7500.100
3150.3330.7330.333
gpt-5.4_medium †1300.4001.0000.400
2200.0000.7500.000
3150.3330.7330.333
gpt-5.4_high †1300.2670.9670.267
2200.0000.8000.000
3150.3330.7330.333
gpt-5.4_xhigh1300.1000.5670.100
2200.0000.4000.000
3150.2670.5330.267
gpt-5.51300.3671.0000.367
2200.1000.8000.100
3150.0001.0000.000
gpt-5.5_low1300.3001.0000.300
2200.2500.7500.250
3150.1330.7330.133
gpt-5.5_medium1300.4331.0000.433
2200.0000.8000.000
3150.3330.8670.333
gpt-5.5_high1300.4001.0000.400
2200.2500.8000.250
3150.3330.8670.333
gpt-5.5_xhigh1300.4330.9000.433
2200.3000.7500.300
3150.3330.6670.333
All 30 models19000.3010.7640.308
26000.1000.6450.100
34500.1270.6600.127
AuthorsOpenReview ↗

Rebuttal 2/4

Q2 Mutation selection: What criteria were used to select the 10 mutation pairs for the main evaluation from the larger set of distinguishable pairs? How sensitive are the reported findings to this selection?

Pair selection was not random but structured by the following criteria:

  • Representation of the central phenomenon. A key claim of the paper is that the detection–attribution gap widens under interacting mutations. A pair sample dominated by additive configurations would underrepresent exactly this regime. We therefore selected 3 of the 9 interacting mutation pairs, deliberately including a particularly difficult one (ID_04 + ID_08).
  • Class-level selection. We prioritized mutation classes that are central to the substrate influencing a large fraction of the dependently typed λ\lambda-calculus (binding discipline, computational rule, type equality) while being non-obvious to distinguish. Conversely, we excluded free_name, which has the highest subsumption rate in the ontology, because pairs involving it would mostly re-test the identifiability filter rather than attribution ability; and universe, which exhibits no observable relationship to any other class, so that pairs containing it are trivially separable or degenerate.
  • Discrimination under confusability. We selected additive pairs whose members induce similar type-level effects and must therefore be actively distinguished rather than merely detected.

The final set of 10 pairs was selected to best satisfy these criteria.

The absolute metric values/gap width necessarily depend on this selection: choosing the ten easiest or hardest pairs based on singleton performance (Figure S1), or including all interacting pairs, would shift the concrete numbers. We argue, however, that the qualitative finding – the gap itself – is robust, for three reasons:

  • The detection–attribution gap is present in every one of the 13 evaluated models, so it does not depend on the specific behavior of selected pairs.
  • Table S5 shows the gap and the degradation under k=2k=2 and under interaction in essentially every individual model, not only on average.
  • The per-mutation difficulty analysis (Figure S1) shows substantial heterogeneity across the ontology, and our selection spans this range rather than clustering at one extreme.

We acknowledge two limitations, which we will state explicitly in the revision. First, the class-level criteria involve researcher judgment. Second, the selection is deliberately skewed toward confusable configurations that are representative of the abductive challenge, including the hardest interacting pair; it nevertheless spans the difficulty range of admissible pairs.

Reviewer_6ZVPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Introduces an testbed for abductive reasoning. The setup uses black-boxes of two lambda-calculus kernels, with one that has been corrupted by 1-2 of total 13 pre-defined mutations. The models/agents are then asked to identify the corrupted kernel within fixed probe budget. There are three prompt levels. The authors evaluate 13 model configs (4,485 sessions), with no code execution or web search allowed.

Main findings:

  • Authors find a large detection - attribution gap on average.
  • Model's own probes uniquely identified the answer in 44.6% of sessions, but attribution still failed in 53.7% of those.
  • Ablations on GPT high with trace rereading recovers only ~19% of interpretation failures. Supplying candidate's kernel-computed predictions recovers ~99%.

Based on their ablations, the authors conclude "deriving the behavioral consequences of competing hypotheses and integrating those consequences with accumulated evidence during interactive attribution.

Critical strengths

The setup is a clever way to test abductive understanding in models. Because the kernel is executable and the hypothesis space is finite, the authors can replay every probe under every candidate and say objectively whether the evidence the agent collected was sufficient.

The ablation chain is well-designed & reported results are rigorous.

Critical weaknesses

My main issues with the paper are that the scale and diversity of the setup / task limit how general the claims. The claims / framing used by the authors of an "abductive jump" is much stronger than the setup they have used (which is very restricted). Its a clever idea but claims are overblown. The hypothesis space is a closed, explicitly enumerated list of 13 described mutations, and the task is to choose among at most 91 configurations by active experimentation. The authors acknowledge the predefined hypothesis space as a limitation, but the discussion is still presented as abductive jump.

Questions for authors
  • Have you run a condition in which models may execute code but are not given the kernel source?
  • Can you say more on 99% recovery? is this a consequence of the task getting trivialized?
  • Have you tried with non-OpenAI models? Do the results hold? What is the gap across models?
3 comments
AuthorsOpenReview ↗

Rebuttal 1/3

We first want to thank the reviewer for providing insightful comments regarding the "abductive jump". The philosophical literature distinguishes between creative and selective abduction (Magnani, 2000, 2009; Stanford Encyclopedia of Philosophy, "Scientific Discovery"). Zahavy's notion of the "abductive jump" primarily concerns creative abduction, involving the generation of novel explanatory principles. Our experimental setup, by contrast, addresses selective abduction: identifying the underlying explanation from a predefined set of competing hypotheses. We agree that this represents a more restricted form of abductive reasoning and that our original framing did not sufficiently emphasize this distinction.

The restriction is deliberate, however, as it allows us to investigate specific components of abductive inference in a controlled experimental setting, with explicitly defined hypotheses and known ground truth. Importantly, our results show that models frequently fail to identify the mutations responsible for observed behavioral differences, despite the relatively small and explicitly defined hypothesis space. Thus, substantial difficulties arise even in this constrained setting.

We have revised the introduction and conclusion to distinguish selective from creative abduction more clearly and to delimit the scope of our claims accordingly. In particular, we have moderated the use of "abductive jump" to avoid implying that our experiments investigate the generation of novel explanatory principles.

Have you run a condition in which models may execute code but are not given the kernel source?

We thank the reviewer for this suggestion. In our experiments, we deliberately restricted tool use to the task-specific interface, which allows models to submit probes, inspect the experimental context, and monitor their remaining probe budget. General-purpose code execution was disabled to ensure that the evaluation focused on abductive attribution through interaction with the experimental environment, rather than on approaches involving external computational scaffolding.

We have not evaluated the proposed condition. Allowing code execution without access to the kernel source would certainly be an interesting extension, as it would help determine whether external computational assistance improves hypothesis exploration, probe generation, and ultimately abductive attribution. We consider this an important direction for future work.

AuthorsOpenReview ↗

Rebuttal 2/3

Can you say more on 99% recovery? is this a consequence of the task getting trivialized?

We thank the reviewer for raising this point. The 0.99 recovery in Ablation 2 ("informed reader") is an upper bound by design, rather than a measure of overall abductive performance. Attributing the observed behavior to a configuration requires two steps: deriving the effect of each candidate configuration on each probe, i.e., the output a kernel carrying exactly these mutations would produce on that probe, and matching these derived effects against the outputs actually observed. Ablation 2 removes the derivation step. We obtain the true effects by running the session's probes on a kernel carrying each candidate configuration, and supply them to the model: for each probe, the reference kernel's output, followed by the groups of configurations whose output deviates from it in the same way. The model still sees the observed outputs of both kernels, but receives no eliminations, verdicts, or indication of which kernel is corrupted. It must still identify the corrupted kernel and the single configuration consistent with all probes. Thus, the 0.99 measures how well the model performs the matching step when the true effects are supplied.

We agree that this ablation simplifies the task, since it supplies not only the effects but also groups the configurations by shared effect, which simplifies the matching as well. To separate deriving the effects from matching them, we have therefore introduced a new Ablation 4 ("derive-then-match"), using the same 189 sessions. Here, the model itself must derive the effect of every candidate configuration on each discriminating probe, from the mutation catalog and the reference kernel's outputs alone; the outputs the two kernels actually produced are withheld. A program then matches these derived effects against the observed outputs, using the same uniqueness test as our failure analysis (Section 4.2): a configuration survives only if its derived effects agree with the observed outputs on every discriminating probe. For GPT-5.4 (High), on its 120 interpretation-failure sessions, the model's own derived effects single out the correct configuration in only 0.07 [0.03, 0.11] of draws, compared with 0.19 under fresh rereading and 0.99 when the true effects are supplied. In 0.91 of draws, at least one effect the model derives for the true configuration contradicts the observed output, although 0.60 of the effects it derives for that configuration are correct.

These results suggest that deriving the effects, rather than matching them to the observed outputs, is the principal bottleneck. Importantly, however, they do not necessarily imply that the models' derivations are generally inaccurate. Rather, even relatively limited errors in the derived effects may be sufficient to make the correct candidate configuration appear incompatible with one or more observed outputs. Such errors may therefore propagate through the subsequent evaluation of competing hypotheses and have a disproportionate impact on final attribution. In other words, error amplification may be a plausible explanation, although the present experiments do not directly establish this mechanism.

The same qualitative pattern occurs with Claude Haiku 4.5, Mistral Large 3, and Gemini 2.5 Flash, although their informed-reader upper bounds are lower (0.54, 0.64, and 0.67, respectively). We have revised Section 4.3 and Supplementary Section C.2 to clarify the interpretation of Ablation 2 as an upper bound on the matching step and added Ablation 4 to isolate the derivation of effects. The full protocol, controls, and results are provided in Supplementary Section C (Tables S8-S10 and Figures S9-S10).

AuthorsOpenReview ↗

Rebuttal 3/3

Have you tried with non-OpenAI models? Do the results hold? What is the gap across models?

We performed additional experiments on non-OpenAI models for the ablations (see below). The main evaluation already includes four model families: OpenAI, Google Gemini, Anthropic Claude, and Mistral (Table 1). The detection-attribution gap occurs consistently across all four. For example, kernel detection versus exact attribution is 0.93 versus 0.35 for Gemini 2.5 Pro, 0.89 versus 0.36 for Gemini 2.5 Flash, 0.78 versus 0.34 for Claude Haiku 4.5, and 0.65 versus 0.31 for Mistral Large 3. For comparison, GPT-5.4 (High) achieves 0.89 versus 0.46. Thus, the gap is not specific to OpenAI models. Moreover, conditional on the model's probes uniquely identifying the correct configuration, attribution still fails in 0.54 of cases across all 13 models (Section 4.2), indicating that the detection--attribution gap persists even when sufficient evidence has been collected.

We have also extended the ablation studies to three non-OpenAI models: Claude Haiku 4.5, Mistral Large 3, and Gemini 2.5 Flash. All four ablations were replicated using the same protocols and scoring procedures. The main findings replicate across all three models: fresh rereading recovers only 0.08-0.14 of interpretation failures, compared with 0.19 for GPT-5.4 (High). Supplying the true effects of each candidate configuration (Ablation 2) increases recovery to 0.54-0.67 (versus 0.99 for GPT-5.4 (High)), whereas the models' own derived effects (Ablation 4) yield only 0.00-0.01 recovery (versus 0.07 for GPT-5.4 (High)). Although the precise performance levels and patterns of forward-simulation errors differ across models, the central finding is consistent: models can identify the underlying configuration when given the true effects, but struggle to derive these effects accurately enough to support attribution.

Finally, in our response to Reviewer zs5Q (Q1, Table R1) we report a second, larger run, omitted from the submission for space, covering 30 models from the same four families under a harder protocol, including configurations with up to three mutations. The detection-attribution gap again appears across all model families. For example, detection versus attribution is 0.98 versus 0.48 for Claude Opus 4.7 (High), 1.00 versus 0.54 for Gemini 3.1 Pro, and 0.91 versus 0.34 for GPT-5.5 (High). Across 1,950 sessions involving at least one mutation, detection is 0.70, whereas exact attribution is only 0.20.

Together, these results indicate that the detection--attribution gap is robust across model families, model versions, and evaluation protocols, although its magnitude varies between models.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank all reviewers for their careful reading and stimulating comments. They led to new experiments, a sharper statement of what the paper shows, and a clearer presentation; we believe the paper has gained substantially in value. The individual responses address each point in detail. A revised PDF has been uploaded; the changes are:

Scope and framing

  • Introduction and Discussion distinguish creative from selective abduction; Section 2 states that Elenchos evaluates selective abduction in a predefined hypothesis space, and "abductive jump" is used accordingly.
  • Section 2 states the task in plain language and places it as consistency-based diagnosis with active probing, with references.
  • "Substrate-agnostic" replaced by "designed to be portable"; the evaluated instances are described as a curated set of configurations.
  • Abstract, Discussion and Limitations rewritten to the revised findings: the limiting step is the reliable derivation of each hypothesis's consequences, not their comparison with the evidence.
  • Limitations name the single substrate, the predefined hypothesis space, the prompt dependence, and the one-model-per-family ablations with the quantities that are model-specific.

New experiments and analyses

  • All ablations replicated on one model from each of the other families (Claude, Mistral, Gemini), each rereading its own sessions in Ablations 1, 2 and 4; summarized in Section 4.3, in full in the supplement.
  • New Ablation 4 (derive-then-match) determines whether interpretation failures arise from difficulties in predicting consequences of competing hypotheses or from difficulties in systematically comparing these predictions across the hypothesis space, which Ablation 2 alone could not.
  • New figures: an overview of the four ablations in the main text; in the supplement, which mutations the models fail to derive (forward simulation and derive-then-match) and whether the evidence still singles out the true configuration under the models' own prediction tables. The supplement also reports the failure decomposition for each model family.

We hope the revision addresses the reviewers' concerns and are glad to clarify any remaining point.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.