- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper asks whether LLMs can generate new hypotheses, which is called abduction. The authors split abduction into several steps and test each step separately. These steps include choosing among given causes, recognizing structures that match, noticing their own errors, and building new explanations. Each step is tested with and without extra support. The paper finds that the models do well with support, but remain unreliable without it.
However, the paper is hard to follow, and the many results do not form a clear insight.
Critical strengths
The paper studies a very timely question: whether LLMs can generate new hypotheses or only recombine their training data.
Critical weaknesses
- The abstract piles up results but leaves out many details, so it is hard to follow. Several terms are used without explanation, such as "routine abduction", "both fitted weights", and "internal error signals". The results are also listed one after another without links between them. For example, L18–20 is about cue weighting, while L20–21 suddenly turns to error detection. In addition, L20–21 and L21–23 seem to repeat the same finding.
- The introduction is too long, and its main point is hard to find. Much of it covers philosophical background and details of prior work. I suggest moving this content into a related work section or the appendix. In addition, the introduction does not clearly state the main contribution of the paper.
- Section 2 is very hard to read. It describes the experiments with many metaphors, stories from the history of science, and new terms. The actual tasks are simple, such as judging whether two small graphs are isomorphic. I suggest that the authors first state each task directly in one or two sentences, and then explain the motivation.
- The evidence in Section 3 does not show that the model knows its own errors. The error signal is read by an external probe trained with correctness labels. This only shows that a good classifier can be found at some layer for some tasks. It does not show that the model itself notices the error. The signal also depends on the model and the task, and it is no longer significant against a stronger output measure. In addition, the probe needs labels to train, which are not available for truly new problems. It is therefore unclear how this signal can serve as the first step of abduction.
- The paper lacks a clear insight. Most results are negative, or succeed only when support is given. The negative results are either explained by the need for more reasoning, which is expected, or described by the authors as a limit of the measurement. It is therefore hard to see what new insight the staircase gives about whether LLMs can perform abduction.
- The paper runs too many experiments, but none of them is studied in depth. The manuscript reads like a pile of preliminary results. I suggest that the authors redesign the experiments. The main argument should also be better organized and made clear, instead of being scattered across the paper.
Questions for authors
Please refer to weaknesses.