- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The authors propose a benchmark for abductive reasoning. They create the benchmark synthetically but grounded in realistic scientific domains so there's a shorter bridge to imagining real applications in the sciences. They benchmark a few GPT and Qwen3 models and find that models do better in complete information settings than hidden information settings.
Critical strengths
- The related work seems comprehensive and clearly lays out the authors' intended differences for their paper over and beyond previous work. The framework they're laying out also seems to be well-rooted in pre-ML science (e.g., referring to Mitchell).
- I think the approach to the benchmark is good. A synthetic generative process, but grounding it in real scientific domains.
Critical weaknesses
- The writing style is not up to ICLR standards and is what most makes this a "Clear rejection" rather than a borderline paper for me. In particular:
- I find the writing of the introduction to be overly verbose and hard to parse. Many sentence also follow a "sentence 'colon or semi-colon' next sentence" structure, which comes off as a little formulaic and could use an editing pass to clean up and make sure it reflects the authors' intended meaning without introducing unnecessary verbosity. (I queried an LLM to find the number of times this "clause :/; clause" structure appeared and got 52 times, which is a slight overestimate because of legitimate colons such as those before equations, but indicates a larger formulaic writing style.)
- I also have a preference against the way that the entire main text is structured, with section 3 devoted to "Evaluation", section 4 to "Benchmark construction", section 5 to "Experiments", but section 4.3 seems to contain evaluation and experimental information, and section 3.2 is a broader theoretical claim about forecast invariance that doesn't seem to have much relevance to the authors' main contributions.
- Consider this paragraph at the beginning of section 3.2: """ A central challenge in evaluating abductive reasoning is hypothesis underdetermination: the observed evidence may be consistent with multiple admissible hypotheses. If such hypotheses imply different outcomes for a novel experiment, the evidence alone does not uniquely determine which forecast should be considered correct. Consequently, evaluating a single intended latent theory as the unique answer may penalize alternative hypotheses that are equally consistent with the available evidence. """ The second clause of the first sentence is identical in meaning to the second sentence and to the third sentence. I don't feel like this ICLR-level writing.
- The authors allude to a theoretical framework, but I don't feel like it is sufficiently justified either in the paper or from the conclusions that they're providing from that theory.
- Re: Section 3.2. Is this level of math needed for the main contributions the authors laid out? Is Proposition A.2 needed for the claim that, to paraphraase li 199, there is a unique hypothesis when available queries distinguish the hypothesis from every alternative?
- I'm not sure why so much space in the main text is devoted to section 3.2 when sections 3.3 and 3.4 seem to be definable independent of it and 3.2 is not related to the authors' contributions.
- Insufficient space devoted to key details and findings:
- Re: Section 4.2 noting that physics examples are simultaneous, chemistry is sequential, and biology has asynchronous state transitions. This seems important, I think it would be worth spending more time on breaking down this distinction and its ramifications over some of the other content of the main text.
- Not sure why the authors don't discuss the Qwen and GPT-oss results in section 5.2.
- I think more space should have been devoted in the main paper to the study of GPT-oss to identify latent rules. This seems like a major part of the narrative that merits more setup and examination rather than a short paragraph.
- Miscellaneous:
- Why are "Coverage" and "Tokens" shown in Table 3 when they're not discussed in the main text? Also, Table 3 names the 2nd super column "(c) Tokens" instead of "(b) Tokens".
- Interested in bootstrapped confidence intervals if the experiments are too costly to run over multiple sees, because while 1200 examples isn't tiny it's also not large enough for readers to understand whether, for example, a delta of 0.016 is meaningful for GPT-5 (high)'s Overall CR-HR gap.
- If this is a proposed benchmark, I think a wider range of models should have been used, rather than focusing on two model families (GPT and Qwen) with relatively narrow ranges of sizes. I understand that cost may be an issue, and there is merit in developing the benchmark for others to do the more expensive runs on, but it limits the significance of the results and whether the benchmark is sufficienty challenging or interesting for subsequent researchers.
Questions for authors
Why does the CR prompt have a "SUPPLIED ADDITIONAL RULES" section that the HR prompt does not have?
2 comments
Re: W1. Given the relatively late revision, I do not have the time to reread the entire paper to make sure that all my concerns about writing style are addressed. While I appreciate the use of blue highlighting to distinguish where revisions have been made, I have some doubt about the recall of these blue highlights. E.g., Table 3 of the new paper has been completely reworked yet has no blue highlighting in the new uploaded version.
Note: I'm not sure why 2-1 and 2-2 are labelled in that way, wouldn't it make more sense to label them W2 and W3? Regardless, I'll treat this as a good faith error on the authors' part.
Re: W2-1. I appreciate the cut to exclude mathematical propositions from the main paper that I consider extraneous.
Re: W2-2.
- I appreciate the authors including an extra sentence or so for each of the domains to explain more about the domains. However, I don't feel like this fully addresses the heart of my review, specifically about the "ramifications" of such distinctions. As near as I can tell, the nearest thing to such a discussion in the main text is section 5.2's phrase "The domains also differ in difficulty. For most models, the gap is smallest in physics." which I consider to be surface-level and not sufficient for a results section.
- Thanks for including some open models here. This is another place where the blue highlighting is letting us reviewers down a little. For example, Section 5.2 now starts with "Table 2 reports CR and HR accuracy..." where the submitted version started with "Table 2 reports a positive CR-HR gap for eve...", which just makes it a little hard to quickly turn around a revised review. I hope the authors will keep this in mind and endeavor to get their rebuttal in a bit earlier and a bit easier for the reviewers to see differences between the revised and submitted versions.
- I appreciate this inclusion of this additional analysis. I don't feel like it's currently contextualized within the larger paper at a conference-level yet. It does feel like a quick addition, rather than a natural part of the larger main text at this point.
Re: W3. Thank you for running the confidence intervals. Another failure of blue highlighting if the new Table 6 is the old Table 3, because it has been edited and shows up as un-highlighted in the revised version of the paper.
Re: Q1. I think this was just confusion on my end. I thought there was some other reasonable difference between the prompts and so wasn't sure about why the SUPPLIED ADDITIONAL RULES difference was necessary, too. But, now that I look back on it, I think it must have been a mistake on my side. Thanks for explaining.
After reading the author's response to me (and skimming the responses to other authors), I am willing to revise my score upwards to a 2, but do not feel that some of my concerns have been addressed sufficiently to move the paper further upwards. Namely, continuing concerns about writing style and insufficient discussion and contextualization of results.
Thank you for your detailed comments on our work. Our responses to each comment follow.
(W1) Writing style and paper organization
We have revised the introduction for concision. Section 4.3 was confusing because “experiment” referred both to interventions within a world and to our model evaluation. We now use “experiment” only for the former and have renamed Section 5 to “Results”.
(W2-1) Excessive theoretical content in Section 3.2
We agree and have shortened Section 3.2. We moved Proposition A.2 to Appendix A.1 and kept the focus on forecast invariance and its role in the paired evaluation. Section 3.2 matters because our design allows multiple evidence-consistent hypotheses. Theorem 3.1 shows that forecast invariance is necessary and sufficient for scoring not to depend on which of them serves as the reference, and Section 4.4 checks this condition exactly. CR and HR therefore share the same evidence-determined targets, so the gap is not an artifact of an arbitrarily chosen reference hypothesis.
(W2-2) Insufficient discussion of key details and findings
We revised the main text on all three points.
(1) Domain composition
Section 4.2 now explains how each domain composes rules (simultaneous, sequential, asynchronous). Section 5.2 compares results across domains (Table 2). Among near-ceiling models (GPT-OSS-120B, GPT-5 family), chemistry has the lowest HR accuracy. Their CR accuracy is about 0.99 in all domains, so the difference lies in inference. The domains also differ in state representation and rule grammar, so we do not attribute this difference to composition alone.
(2) Discussion of open-weight models
Section 5.2 now covers all open-weight models. Their gap ranges from 0.079 (GPT-OSS-120B) to 0.308 (Mistral-Small-4-119B). For the Qwen3 models, CR accuracy itself is well below ceiling (0.68–0.78), so their errors arise under both conditions.
(3) Rule-stating study
We extended this study to three models (Table A; Section 5.3). GPT-OSS-120B states an evidence-consistent hypothesis for 81.4% of problems and follows its own hypothesis nearly as reliably as the given rules (0.967 vs. 0.990). Its errors thus come from its hypotheses, not from execution. For weaker models, faithfulness falls well below CR.
Table A. Rule-stating analysis (360 instances). HR-RS is HR with rule stating, Consistent is the share of stated hypotheses that fit all observations, and Faithfulness the share of forecasts that follow the model’s own hypothesis.
(W3) Miscellaneous: Table 3, confidence intervals, and model coverage
We have fixed the errors in the former Table 3 (now Table 6), and explained coverage and token usage in Appendix B. We have also added Nemotron-3-Super-120B-A12B, Mistral-Small-4-119B, and Gemma-4-26B-A4B, extending the evaluation beyond the Qwen and GPT families.
We now report 95% paired bootstrap confidence intervals for all models in Appendix C.1 (Table I). For the 0.016 gap you asked about, the gap of GPT-5 (high) is now +0.032 [+0.026, +0.037], and even its smallest domain gap (physics) is +0.010 [+0.005, +0.016]. Both exclude zero.
Table I. Main results with paired bootstrap 95% CIs (1,800 instances), as in our response to Reviewer kpE4.
(Q1) Role of the “SUPPLIED ADDITIONAL RULES” section in the CR prompt
We should have explained this more clearly. This section is the only difference between the two prompts and is the manipulation on which the benchmark rests. In CR, it supplies the hidden rules, so the model only needs to execute them. In HR, it is removed, so the model must infer the hidden rules from the evidence. The world description, public rules, evidence, and forecast queries are identical, so the CR–HR gap measures the additional cost of hidden rules.