- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Elenchos, a framework for evaluating abductive reasoning in LLMs through interactive diagnosis of mutated formal systems.
Critical strengths
From what I understand, the research problem seems quite interesting.
The experiments are comprehensive, covering 13 models with different difficulty levels and mutation configurations.
Although I am not familiar with this research area, I can see that the authors have put considerable effort into this work, especially given the detailed Supplementary Material and the overall presentation. I would encourage the authors to explain their work in a more intuitive and accessible way, which I believe would help readers from different backgrounds better understand its contributions.
Critical weaknesses
My main concern is the clarity of the presentation. The paper focuses on a rather specialized research problem, making it difficult for readers unfamiliar with the area to understand the motivation and contributions from the title and abstract. They choose 'other topics in machine learning (i.e., none of the above)'. I suggest explaining the problem more intuitively in the abstract and including a simple illustrative example or figure to help readers understand the task.
I am not very familiar with the formal methods background of this paper, so I find it difficult to assess the technical details of the proposed framework. From the experimental perspective, one potential limitation is that the detailed ablation studies are conducted only on GPT-5.4, which raises questions about generalizability. However, my main concern remains the accessibility of the presentation, as I believe the paper would benefit from a clearer and more intuitive explanation of the research problem and its significance.
Questions for authors
Could you explain abductive reasoning more directly? Or provide some practical applications.
2 comments
Thanks for the additional explanations from the authors. I now understand the scenario. I think this problem is interesting and novel. However, I still couldn't evaluate the depth of the experiment. I think the AC could consider my score to be 3.5.
Rebuttal
We thank the reviewer for the candid assessment and for pointing out that the motivation and conceptual background may not be sufficiently clear to readers outside the field. Below, we explain abductive reasoning more directly and illustrate our findings with a practical example. We have revised the abstract and Introduction accordingly.
Concretely, Section 2 now describes the task in plain words before the formal definitions: two copies of a program behave differently, and the agent, given a list of the ways one copy may have been altered, must find out by experiment which alteration explains the difference. The Introduction now distinguishes selective from creative abduction, Section 2 states that Elenchos evaluates selective abduction, and the supplementary material contains a worked session with real kernel outputs (Table S12), referenced from Section 2.
What is abductive reasoning?
Abductive reasoning infers the most plausible explanation for an observation. It complements deduction, which derives consequences from given rules, and induction, which generalizes patterns from observations. Two forms of abduction are commonly distinguished (Magnani, 2000, 2009; Stanford Encyclopedia of Philosophy, "Scientific Discovery"): selective abduction chooses an explanation from a set of known hypotheses, whereas creative abduction generates new explanatory hypotheses, as in scientific research. Elenchos evaluates selective abduction. Abductive reasoning is central to medical diagnosis, software debugging, cybersecurity, scientific discovery, and fault diagnosis in engineering.
A practical example: medical diagnosis.
Consider an LLM acting as a diagnostician. A patient presents with symptoms; the LLM considers a list of candidate diseases, orders tests (blood pressure, blood work, and so on), and must name the disease that explains all results. In Elenchos, the patient corresponds to the corrupted kernel, the candidate diseases to the catalog of possible mutations, and the tests to the probes.
Translated to this setting, our findings suggest that the LLM would reliably recognize that the patient is ill (detection) but would often fail to name the exact disease (attribution), particularly when several diseases interact. Strikingly, it often orders tests whose results already single out the correct disease and still names a wrong one. Our ablations locate this failure: when GPT-5.4 (High) is told what each candidate disease would produce on the tests it ordered, it resolves 0.99 of these cases; when it must derive this itself, its derivations single out the correct disease in only 0.07. The bottleneck is therefore not comparing test results with expectations, but reliably deriving what each candidate disease implies for the tests performed.
These results are not specific to GPT-5.4. The detection-attribution gap appears in all 13 models of our main evaluation, which span four model families (OpenAI, Google, Anthropic, Mistral). We also repeated the ablations on Claude Haiku 4.5, Mistral Large 3 and Gemini 2.5 Flash: when told what each candidate would produce, they resolve 0.54-0.67 of these cases, and when they must derive it themselves, their derivations single out the correct one in only 0.00-0.01. Whether the findings carry over from our formal setting to domains such as medicine remains open.