acceptodds
Under review as a conference paper at ICLR 2027

Can LLMs take the plunge? First, map the stairs

Abstract

Can large language models generate genuinely new hypotheses that may lead to scientific discoveries, or just combine the training data in a sophisticated way? We make this broad question testable by decomposing abduction (generation of explanatory hypotheses from observations) into a staircase of diagnostic steps. One path tests whether LLMs can detect and act on their own errors; the other climbs from selecting among supplied explanations, to recognizing structural correspondences, to constructing unsupplied hypotheses with testable consequences. For routine abduction, the tested LLMs underweight base rates relative to sample evidence, a pattern Grether (1980) documented in humans; across increasing model sizes, both fitted weights rise but their imbalance persists. Internal error signals become most informative where outputs are confidently wrong. In a toy model tracked across training, the internal signal becomes more informative than output uncertainty after a grokking-like transition. Across two frontier-model families, structural-recognition accuracy remains near chance in a single pass (0.50-0.54) but rises to 0.98-1.00 when hidden serial computation is permitted. In radical abduction without supplied candidates, when contradictory evidence signals a failure, models reach the correct hypothesis class in one world family, and occasionally produce correct final programs, but do not complete the full evidence-bound chain. Without such a trigger, two frontier-model families do not distinguish rule-bearing worlds from matched rule-free ones unless the hypothesis family is supplied. When we supply the method, a second pass repairs 110 of 114 first-pass errors. An interior reader can allocate retries but without a conclusive advantage over a trained output reader. Computation and supplied structure improve performance substantially, while constructing and testing explanations without those supports remains unreliable.

Primary area:
foundation or frontier models, including LLMs
Keywords:
abduction, hypothesis generation, large language models, reasoning, interpretability, scientific discovery

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_Zy1tOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper asks whether LLMs can generate new hypotheses, which is called abduction. The authors split abduction into several steps and test each step separately. These steps include choosing among given causes, recognizing structures that match, noticing their own errors, and building new explanations. Each step is tested with and without extra support. The paper finds that the models do well with support, but remain unreliable without it.

However, the paper is hard to follow, and the many results do not form a clear insight.

Critical strengths

The paper studies a very timely question: whether LLMs can generate new hypotheses or only recombine their training data.

Critical weaknesses
  1. The abstract piles up results but leaves out many details, so it is hard to follow. Several terms are used without explanation, such as "routine abduction", "both fitted weights", and "internal error signals". The results are also listed one after another without links between them. For example, L18–20 is about cue weighting, while L20–21 suddenly turns to error detection. In addition, L20–21 and L21–23 seem to repeat the same finding.
  2. The introduction is too long, and its main point is hard to find. Much of it covers philosophical background and details of prior work. I suggest moving this content into a related work section or the appendix. In addition, the introduction does not clearly state the main contribution of the paper.
  3. Section 2 is very hard to read. It describes the experiments with many metaphors, stories from the history of science, and new terms. The actual tasks are simple, such as judging whether two small graphs are isomorphic. I suggest that the authors first state each task directly in one or two sentences, and then explain the motivation.
  4. The evidence in Section 3 does not show that the model knows its own errors. The error signal is read by an external probe trained with correctness labels. This only shows that a good classifier can be found at some layer for some tasks. It does not show that the model itself notices the error. The signal also depends on the model and the task, and it is no longer significant against a stronger output measure. In addition, the probe needs labels to train, which are not available for truly new problems. It is therefore unclear how this signal can serve as the first step of abduction.
  5. The paper lacks a clear insight. Most results are negative, or succeed only when support is given. The negative results are either explained by the need for more reasoning, which is expected, or described by the authors as a limit of the measurement. It is therefore hard to see what new insight the staircase gives about whether LLMs can perform abduction.
  6. The paper runs too many experiments, but none of them is studied in depth. The manuscript reads like a pile of preliminary results. I suggest that the authors redesign the experiments. The main argument should also be better organized and made clear, instead of being scattered across the paper.
Questions for authors

Please refer to weaknesses.

Reviewer_RitaOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper constructs a hierarchy of tasks that are necessary for LLMs to be able to solve before they can perform abduction. Different datasets are constructed for each task. Evaluating several current LLMs, the paper finds that the base LLM doesn’t perform any of these tasks well without additional help, such as constraining the hypothesis space to a few possible hypotheses or providing evidence that contradicts the LLM’s current hypothesis.

Critical strengths
  1. The paper breaks down an important but abstract skill, abduction, into multiple concrete tasks that are easier to test on LLMs.
Critical weaknesses
  1. LLMs being able to perform the tasks in this paper well may be necessary conditions but not sufficient conditions for LLMs to be able to perform abduction. In addition, it is extra difficult to assess how well the LLMs perform on the proposed task hierarchy, as benchmark datasets are constructed in entirely unrelated manners across each step of the abduction hierarchy.
  2. It is not clear what the formal definition of abduction is, and what concrete properties distinguish abduction from sophisticated combination. Two significant differences seem to be that abduction has a very large hypothesis space that cannot be exhaustively searched, and an abduction is only credible when, to quote the paper, “deduction derives their consequences, and induction tests them”. If these are the only two differences between abduction and sophisticated combinations, neither of them is tested in this paper.
Questions for authors
  1. Could the authors specify what concrete properties distinguish abduction from sophisticated combinations, and how the experiments in this paper test these properties?
  2. Could the authors justify why we would want models to inherently notice when errors happen? Specifically, humans don’t seem to have this ability. Instead, we rely on objective experiments and observations to inform us on whether these results contradict our hypotheses or not.
  3. In Experiment 11, can the authors justify why we expect LLMs to propose a new hypothesis without contradicting observations? Specifically, when scientists propose hypotheses, such as multiple interpretations of quantum mechanics, all hypotheses that have no contradictory evidence are equally true, or empirically indistinguishable. So why would we expect LLMs to change their hypotheses when there is no contradicting evidence?
  4. Could the authors propose a single throughline across all dataset constructions in this paper, so that models’ performances on different tasks reflect their ability on different abduction steps, not on the differences in dataset construction?
  5. Could the authors justify why certain models are chosen to be tested on specific abduction tasks while being omitted in other experiments? Alternatively, could the authors provide a more comprehensive table of experiment results testing all models on all tasks, and explain whether the main findings of the paper stay the same?
Reviewer_cnJEOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

As far as I understand, this paper investigates whether LLMs are truly capable of reasoning that generates novel ideas, as opposed to amalgamation of existing ones. The authors draw on historical references in inductive, deductive, and abductive reasoning, as well as recent work exploring hypothesis generation in frontier models, to present multiple sides of the argument.

Overall, I found this paper extremely difficult to follow in its current form. The section titles and table of contents do not provide a coherent outline, and the paper does not clearly state its central research question or main contributions. The purpose of the experiments, the reasoning behind their design, and their relationship to the paper's claims are also unclear. Large portions of the prose feel formulaic and disconnected, giving the impression of LLM-generated text that has not been sufficiently revised or polished. This makes it difficult to understand what conclusions or claims the paper is trying to establish, what the experiments actually show, and what is novel about the work. The submission needs substantial revisions and rewriting before I can meaningfully assess its scientific contribution.

Critical strengths

Understanding how well LLMs can generate creative ideas and novel hypotheses, and where their limitations lie, is an important research direction for the ICLR community. A careful investigation grounded in well-designed experiments could make a valuable contribution to the field. However, the paper in its current form does not meet this potential.

Critical weaknesses

Significant presentation, clarity, and logical flow issues

The abstract is unclear - I did not fully understand the main problem the paper focuses on investigating, and the significance of the authors’ findings were not clear. It reads very crowded with technical terminology that’s undefined (eg, "routine abduction", "radical abduction", "grokking-like transition", and results on undefined metrics right off the bat, and drowns the reader in un-contextualized information. The terminology ends up clarified later in the introduction, supported with references, but if the authors absolutely need to add it to the abstract, then they should define the pivotal terms in-place there too.

The writing is convoluted, and the paper's organization makes its arguments difficult to follow. It's unclear which section serves as the introduction or where the main claims and contributions are summarized. The discussion of prior work is not clearly identified as a related work section, and reads as a loosely-connected collection of facts, without thematically organized subsections, or a clear explanation of how the cited research relates to the paper. Section titles like "CHOOSING AND RECOGNIZING", "THE STAIRCASE", and "AN ALARM IS NOT AN ADDRESS" are confusing and give the reader very little indication of their content or purpose. A more recognizable structure, informative headings, and an explicit summary of the contributions would help understand and assess the work.

The first person singular language in the AI Use Statement section is acceptable, but suggests that the paper has a single author. In future submissions, I suggest adhering to the more neutral “we”: (L489): “Under my supervision,”, (L503): “However, the genesis of ideas, questions and final claims are mine, and I solely take full responsibility.”.

A lot of generated-sounding prose throughout the paper makes it difficult to follow the authors' arguments:

  • (L97-100): “Farmer reports a motivating episode in which a human supplied the decision to search and the broad strategy of searching for visually analogous images, though not the target correspondence. Farmer’s paper specifies DAB-30 but does not report benchmark results; its arXiv record states that execution is reported in a companion paper, which we could not locate.” - we typically do not include references we can’t verify or read in a submission.

  • (L157-159): “The introduction named the steps. Table 1 turns them into experiments, showing what is given, what must be found, and which support, if any, we hand back to measure what it buys.” - *what does this mean?

  • (L231-234): “Five design rules keep every experiment honest (Appendix A): paired contrasts, ground truth fixed independently of the model and checked mechanically, a positive control behind every null, predictions and thresholds frozen before each run, and access that ranges from a toy model’s training trajectory through open-weight internal states to frontier behavior alone.” - results that are important enough to be referenced as essential in “keeping every experiment honest” should not be reported in the Appendix, they should be in the main text.

  • (L240): “A search for a better explanation starts with noticing. So our first question is narrow.”

  • (L251): “We first watched the wrong thing (Experiment 1).”

  • (L366-367): “An initial Gemma positive did not replicate. Later relational readouts also had uneven control sensitivity (Appendix D)” - we typically don’t report initial vs. later results; your paper is meant to be the final snapshot of results, conclusions, and analysis.

Questions for authors

What is an "interior reader" vs. a "trained output reader"? (L31-32)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.