acceptodds
Under review as a conference paper at ICLR 2027

When Rules Are Hidden: A Paired Evaluation of Abductive Reasoning

Abstract

Large language models (LLMs) are increasingly used for scientific discovery, where they must not only reason from known rules but also infer hypotheses and latent mechanisms from observations. Abductive reasoning is therefore a central capability, yet its evaluation faces two challenges. First, abduction is confounded by deduction: an incorrect prediction may arise from inferring an inadequate hypothesis or from reasoning incorrectly from it. Second, finite observations can admit multiple plausible hypotheses, making evaluation against a single ground-truth explanation problematic. We introduce RuleCast, a paired benchmark that addresses both challenges. Each problem is posed under Complete-Rule and Hidden-Rule conditions that differ only in whether the latent rules required for prediction are provided or must be inferred from evidence, so their gap characterizes the additional difficulty of inferring missing premises. Problems are built on controlled transition-system worlds in three scientific domains, where multiple evidence-consistent hypotheses may exist but agree on every evaluated forecast, removing the need for a unique ground-truth hypothesis. Across open-weight and proprietary reasoning models, forecast accuracy consistently drops when rules must be inferred, even for models that execute the given rules nearly perfectly. An error analysis over the enumerable hypothesis space locates the bottleneck in verifying candidate hypotheses against the evidence rather than in executing a committed hypothesis.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Abductive Reasoning, Reasoning Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FsHFOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose a benchmark for abductive reasoning. They create the benchmark synthetically but grounded in realistic scientific domains so there's a shorter bridge to imagining real applications in the sciences. They benchmark a few GPT and Qwen3 models and find that models do better in complete information settings than hidden information settings.

Critical strengths
  • The related work seems comprehensive and clearly lays out the authors' intended differences for their paper over and beyond previous work. The framework they're laying out also seems to be well-rooted in pre-ML science (e.g., referring to Mitchell).
  • I think the approach to the benchmark is good. A synthetic generative process, but grounding it in real scientific domains.
Critical weaknesses
  • The writing style is not up to ICLR standards and is what most makes this a "Clear rejection" rather than a borderline paper for me. In particular:
    • I find the writing of the introduction to be overly verbose and hard to parse. Many sentence also follow a "sentence 'colon or semi-colon' next sentence" structure, which comes off as a little formulaic and could use an editing pass to clean up and make sure it reflects the authors' intended meaning without introducing unnecessary verbosity. (I queried an LLM to find the number of times this "clause :/; clause" structure appeared and got 52 times, which is a slight overestimate because of legitimate colons such as those before equations, but indicates a larger formulaic writing style.)
    • I also have a preference against the way that the entire main text is structured, with section 3 devoted to "Evaluation", section 4 to "Benchmark construction", section 5 to "Experiments", but section 4.3 seems to contain evaluation and experimental information, and section 3.2 is a broader theoretical claim about forecast invariance that doesn't seem to have much relevance to the authors' main contributions.
    • Consider this paragraph at the beginning of section 3.2: """ A central challenge in evaluating abductive reasoning is hypothesis underdetermination: the observed evidence may be consistent with multiple admissible hypotheses. If such hypotheses imply different outcomes for a novel experiment, the evidence alone does not uniquely determine which forecast should be considered correct. Consequently, evaluating a single intended latent theory as the unique answer may penalize alternative hypotheses that are equally consistent with the available evidence. """ The second clause of the first sentence is identical in meaning to the second sentence and to the third sentence. I don't feel like this ICLR-level writing.
  • The authors allude to a theoretical framework, but I don't feel like it is sufficiently justified either in the paper or from the conclusions that they're providing from that theory.
    • Re: Section 3.2. Is this level of math needed for the main contributions the authors laid out? Is Proposition A.2 needed for the claim that, to paraphraase li 199, there is a unique hypothesis when available queries distinguish the hypothesis from every alternative?
    • I'm not sure why so much space in the main text is devoted to section 3.2 when sections 3.3 and 3.4 seem to be definable independent of it and 3.2 is not related to the authors' contributions.
  • Insufficient space devoted to key details and findings:
    • Re: Section 4.2 noting that physics examples are simultaneous, chemistry is sequential, and biology has asynchronous state transitions. This seems important, I think it would be worth spending more time on breaking down this distinction and its ramifications over some of the other content of the main text.
    • Not sure why the authors don't discuss the Qwen and GPT-oss results in section 5.2.
    • I think more space should have been devoted in the main paper to the study of GPT-oss to identify latent rules. This seems like a major part of the narrative that merits more setup and examination rather than a short paragraph.
  • Miscellaneous:
    • Why are "Coverage" and "Tokens" shown in Table 3 when they're not discussed in the main text? Also, Table 3 names the 2nd super column "(c) Tokens" instead of "(b) Tokens".
    • Interested in bootstrapped confidence intervals if the experiments are too costly to run over multiple sees, because while 1200 examples isn't tiny it's also not large enough for readers to understand whether, for example, a delta of 0.016 is meaningful for GPT-5 (high)'s Overall CR-HR gap.
    • If this is a proposed benchmark, I think a wider range of models should have been used, rather than focusing on two model families (GPT and Qwen) with relatively narrow ranges of sizes. I understand that cost may be an issue, and there is merit in developing the benchmark for others to do the more expensive runs on, but it limits the significance of the results and whether the benchmark is sufficienty challenging or interesting for subsequent researchers.
Questions for authors

Why does the CR prompt have a "SUPPLIED ADDITIONAL RULES" section that the HR prompt does not have?

2 comments

Thank you for your detailed comments on our work. Our responses to each comment follow.

(W1) Writing style and paper organization

We have revised the introduction for concision. Section 4.3 was confusing because “experiment” referred both to interventions within a world and to our model evaluation. We now use “experiment” only for the former and have renamed Section 5 to “Results”.

(W2-1) Excessive theoretical content in Section 3.2

We agree and have shortened Section 3.2. We moved Proposition A.2 to Appendix A.1 and kept the focus on forecast invariance and its role in the paired evaluation. Section 3.2 matters because our design allows multiple evidence-consistent hypotheses. Theorem 3.1 shows that forecast invariance is necessary and sufficient for scoring not to depend on which of them serves as the reference, and Section 4.4 checks this condition exactly. CR and HR therefore share the same evidence-determined targets, so the gap is not an artifact of an arbitrarily chosen reference hypothesis.

(W2-2) Insufficient discussion of key details and findings

We revised the main text on all three points.

(1) Domain composition

Section 4.2 now explains how each domain composes rules (simultaneous, sequential, asynchronous). Section 5.2 compares results across domains (Table 2). Among near-ceiling models (GPT-OSS-120B, GPT-5 family), chemistry has the lowest HR accuracy. Their CR accuracy is about 0.99 in all domains, so the difference lies in inference. The domains also differ in state representation and rule grammar, so we do not attribute this difference to composition alone.

(2) Discussion of open-weight models

Section 5.2 now covers all open-weight models. Their gap ranges from 0.079 (GPT-OSS-120B) to 0.308 (Mistral-Small-4-119B). For the Qwen3 models, CR accuracy itself is well below ceiling (0.68–0.78), so their errors arise under both conditions.

(3) Rule-stating study

We extended this study to three models (Table A; Section 5.3). GPT-OSS-120B states an evidence-consistent hypothesis for 81.4% of problems and follows its own hypothesis nearly as reliably as the given rules (0.967 vs. 0.990). Its errors thus come from its hypotheses, not from execution. For weaker models, faithfulness falls well below CR.

Table A. Rule-stating analysis (360 instances). HR-RS is HR with rule stating, Consistent is the share of stated hypotheses that fit all observations, and Faithfulness the share of forecasts that follow the model’s own hypothesis.

ModelCRHRHR-RSConsistentFaithfulness
Qwen3-30B-A3B0.7510.4850.4630.0940.456
Gemma-4-26B-A4B0.8490.7060.6620.3500.748
GPT-OSS-120B0.9900.9120.9370.8140.967

(W3) Miscellaneous: Table 3, confidence intervals, and model coverage

We have fixed the errors in the former Table 3 (now Table 6), and explained coverage and token usage in Appendix B. We have also added Nemotron-3-Super-120B-A12B, Mistral-Small-4-119B, and Gemma-4-26B-A4B, extending the evaluation beyond the Qwen and GPT families.

We now report 95% paired bootstrap confidence intervals for all models in Appendix C.1 (Table I). For the 0.016 gap you asked about, the gap of GPT-5 (high) is now +0.032 [+0.026, +0.037], and even its smallest domain gap (physics) is +0.010 [+0.005, +0.016]. Both exclude zero.

Table I. Main results with paired bootstrap 95% CIs (1,800 instances), as in our response to Reviewer kpE4.

ModelCR Acc. [95% CI]HR Acc. [95% CI]Δ [95% CI]
Qwen3-8B0.682 [0.674,0.691]0.445 [0.436,0.454]+0.237 [+0.227,+0.247]
Qwen3-30B-A3B0.748 [0.739,0.757]0.468 [0.459,0.478]+0.280 [+0.268,+0.291]
Qwen3-32B0.782 [0.774,0.789]0.556 [0.545,0.566]+0.226 [+0.215,+0.238]
Gemma-4-26B-A4B0.843 [0.835,0.851]0.676 [0.665,0.687]+0.167 [+0.156,+0.178]
Mistral-Small-4-119B0.891 [0.885,0.897]0.583 [0.572,0.595]+0.308 [+0.296,+0.321]
Nemotron-3-Super-120B-A12B0.917 [0.912,0.922]0.793 [0.781,0.805]+0.124 [+0.112,+0.136]
GPT-OSS-120B0.991 [0.989,0.992]0.912 [0.903,0.920]+0.079 [+0.070,+0.088]
GPT-5-nano0.738 [0.730,0.746]0.407 [0.398,0.418]+0.331 [+0.319,+0.342]
GPT-5-mini0.992 [0.990,0.993]0.869 [0.858,0.881]+0.123 [+0.111,+0.134]
GPT-5 (low)0.993 [0.992,0.995]0.859 [0.847,0.870]+0.135 [+0.124,+0.146]
GPT-5 (high)0.999 [0.999,1.000]0.968 [0.962,0.973]+0.032 [+0.026,+0.037]

(Q1) Role of the “SUPPLIED ADDITIONAL RULES” section in the CR prompt

We should have explained this more clearly. This section is the only difference between the two prompts and is the manipulation on which the benchmark rests. In CR, it supplies the hidden rules, so the model only needs to execute them. In HR, it is removed, so the model must infer the hidden rules from the evidence. The world description, public rules, evidence, and forecast queries are identical, so the CR–HR gap measures the additional cost of hidden rules.

Re: W1. Given the relatively late revision, I do not have the time to reread the entire paper to make sure that all my concerns about writing style are addressed. While I appreciate the use of blue highlighting to distinguish where revisions have been made, I have some doubt about the recall of these blue highlights. E.g., Table 3 of the new paper has been completely reworked yet has no blue highlighting in the new uploaded version.

Note: I'm not sure why 2-1 and 2-2 are labelled in that way, wouldn't it make more sense to label them W2 and W3? Regardless, I'll treat this as a good faith error on the authors' part.

Re: W2-1. I appreciate the cut to exclude mathematical propositions from the main paper that I consider extraneous.

Re: W2-2.

  1. I appreciate the authors including an extra sentence or so for each of the domains to explain more about the domains. However, I don't feel like this fully addresses the heart of my review, specifically about the "ramifications" of such distinctions. As near as I can tell, the nearest thing to such a discussion in the main text is section 5.2's phrase "The domains also differ in difficulty. For most models, the gap is smallest in physics." which I consider to be surface-level and not sufficient for a results section.
  2. Thanks for including some open models here. This is another place where the blue highlighting is letting us reviewers down a little. For example, Section 5.2 now starts with "Table 2 reports CR and HR accuracy..." where the submitted version started with "Table 2 reports a positive CR-HR gap for eve...", which just makes it a little hard to quickly turn around a revised review. I hope the authors will keep this in mind and endeavor to get their rebuttal in a bit earlier and a bit easier for the reviewers to see differences between the revised and submitted versions.
  3. I appreciate this inclusion of this additional analysis. I don't feel like it's currently contextualized within the larger paper at a conference-level yet. It does feel like a quick addition, rather than a natural part of the larger main text at this point.

Re: W3. Thank you for running the confidence intervals. Another failure of blue highlighting if the new Table 6 is the old Table 3, because it has been edited and shows up as un-highlighted in the revised version of the paper.

Re: Q1. I think this was just confusion on my end. I thought there was some other reasonable difference between the prompts and so wasn't sure about why the SUPPLIED ADDITIONAL RULES difference was necessary, too. But, now that I look back on it, I think it must have been a mistake on my side. Thanks for explaining.

After reading the author's response to me (and skimming the responses to other authors), I am willing to revise my score upwards to a 2, but do not feel that some of my concerns have been addressed sufficiently to move the paper further upwards. Namely, continuing concerns about writing style and insufficient discussion and contextualization of results.

Reviewer_3dKROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper first observes two challenges of evaluating abduction reasoning: first, abduction is confounded by deduction; second, one problem can admit many hypotheses. This paper introduces a benchmark called RuleCast that attempts to address these two challenges. Each problem has two settings: one where the latent rules are provided and one where they are not. The benchmark draws on three scientific domains and allows multiple hypotheses. The evaluation result shows that accuracy drops when rules are latent. The error analysis shows that the bottleneck is verifying the proposed hypothesis against the evidence.

Critical strengths
  1. The paper is very theory-minded. I appreciate the author's effort to give as many concrete mathematical definitions as possible.
  2. The paper applies theories to practice to construct a real-world benchmark with three domains. To precisely control the hypothesis space, the author constructs the state-transition system and puts it into the domains, making the benchmark both sound and practical.
Critical weaknesses
  1. I think the paper is written with too many technical details without high-level intuitive insights, especially in section 3. It would be better if this section had more illustrative examples and conveyed high-level ideas before zooming in on the mathematical details.
  2. I question the setup of the E->H->Y pipeline. This pipeline is natural in scientific discovery: we form a hypothesis H, and if it's correct, we can apply it to get other useful Y. However, for evaluating abductive reasoning, why do we need this pipeline in the first place? Abductive reasoning aims to generate hypotheses that explain evidence; applying it is not part of it. In other words, why not just set E = Y and see if the model can verify the hypothesis itself?
  3. I don't think abductive reasoning and deductive reasoning can be totally separated from each other. Also, the author seems self-contradictory. The author suggests that the bottleneck is verifying candidate hypotheses against the evidence, rather than executing a committed hypothesis. Why can we view verification as executing an uncommitted hypothesis, and, in that case, is verification itself deductive reasoning?
Questions for authors

See Critical Weaknesses #2 and #3. Also, why do we have the complete rule setting? I understand that the author wants to isolate some effects. But if we present all the latent rules, why do we still call the benchmark abductive reasoning? Isn't the sole purpose of abductive reasoning to infer latent rules? Or do you not consider the complete rule setting part of the abductive reasoning benchmark? Please be specific.

1 comment

Thank you for your thoughtful questions on the conceptual basis of our work. Our responses to each comment follow.

(W1) Section 3 lacks high-level intuition and examples

Following the suggestion, Section 3 now begins with an overview based on Figure 2 (previously Figure 1), before the formal definitions. We also shortened Section 3.2 and kept its supporting results in Appendix A.

(W2/Q1) Why evaluate via forecasts, and what the CR condition is for

If we understand correctly, you suggest evaluating hypotheses directly against the evidence (setting E = Y). We also agree that applying a hypothesis is not itself abduction. We nevertheless use forecasts as the main metric for two reasons. First, we study abduction as one of the forms of reasoning required for scientific discovery, and in science a hypothesis is evaluated by the new predictions it makes. Pauli's neutrino hypothesis [1], for instance, was accepted on the strength of the predictions it made. Following this practice, we evaluate hypotheses through forecasts. Second, forecasts make paired evaluation possible. Forecast accuracy reflects not only abductive reasoning but also deductive ability, and paired evaluation is designed to partly separate the two. This requires a task that produces the same kind of output under both conditions. Hypotheses cannot be scored under CR because they are already given there, whereas forecasting can be performed in both conditions. For these reasons, CR is not an abduction task but a deductive control. HR requires inferring the hidden rules from the evidence, and CR removes this demand by supplying them while keeping everything else identical. The CR–HR gap therefore measures the cost of hidden rules, separated from the deductive ability both conditions share. We call RuleCast an abduction benchmark because this gap is its target. However, we do not claim that inferring the hidden rules is pure abduction, since checking candidate hypotheses against the evidence is itself deduction. We also evaluated stated hypotheses directly (see our response to Reviewer kpE4, W1/Q2). The share of evidence-consistent hypotheses falls as hidden rules increase (GPT-OSS-120B: 0.975, 0.792, 0.675), in the same direction as the forecast gap.

[1] Laurie M. Brown. The idea of the neutrino. Physics Today, 31(9):23–28, 1978.

(W3) Abduction vs. deduction, verification as deduction

We agree that abduction and deduction cannot be fully separated, and the paired design does not assume that they can. As you suggest, verifying a candidate hypothesis executes an uncommitted hypothesis and is deductive in content. This does not conflict with the paired design. CR controls for executing a given hypothesis once. HR additionally requires applying the same operation to many self-generated candidates and deciding which to discard. This demand exists only when rules are hidden, so deductive errors made during search belong in the CR–HR gap by design. For capable models, deduction works well once a hypothesis is fixed. GPT-OSS-120B follows its own stated hypothesis at a rate close to its CR accuracy (0.967 vs. 0.990). Failures instead come from not reaching a consistent hypothesis, and most of its inconsistent hypotheses are near misses, with 48% violating only one observation and 84% at most two. The revised manuscript describes the difficulty of HR in these terms.

Reviewer_kpE4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces RuleCast, a benchmark for evaluating reasoning when some governing rules must be inferred from experimental evidence. Each instance is evaluated under paired Complete-Rule and Hidden-Rule conditions. To avoid requiring recovery of a unique explanation, the benchmark retains only forecast queries on which all evidence-consistent hypotheses agree. The dataset contains 1,200 instances across three synthetic transition-system domains. Experiments demonstrate a consistent accuracy gap between the two conditions, and an additional analysis attributes hidden-rule failures primarily to candidate verification rather than hypothesis execution. 5736_When_Rules_Are_Hidden_A_P

Critical strengths

The paired Complete-Rule/Hidden-Rule design provides a useful control for evaluating the additional difficulty of inferring missing rules. Forecast-invariant scoring is particularly appealing: it accommodates multiple evidence-consistent explanations while preserving an unambiguous evaluation target.

Critical weaknesses
  1. The diagnostic analysis covers only GPT-OSS-120B, and detailed examples of incorrectly rejected valid hypotheses come from just three fully inspected failure traces; requesting explicit hypotheses changes model behavior. Oracle-verification intervention and replication across models are needed.
  2. Section 4.2 treats behaviorally equivalent rule sets as distinct hypotheses, whereas Appendix E.1 merges them when computing statistics, which can change the version-space fraction and its interpretation. The counting convention should be clarified and whether trends depend on it assessed.
  3. The CR–HR gap does not separately identify inference and execution errors, and GPT-5 (high) already achieves 98.3% accuracy. Evidence-removal/shuffling controls, budgeted search baselines, and harder configurations are needed.
  4. Small version-space fractions occur predominantly in two-hidden-rule problems, and evidence-depth groups contain separately generated instances; matched or stratified comparisons are needed. Main results lack paired confidence intervals, and repeated decoding would help quantify variability.
  5. Tables 3 and 4 list different token limits for GPT-5-family models (400K vs 128K), token usage is described as both median and mean; Section 5.3 overstates the smallest-bin gap for GPT-5 (high) and retains an unresolved editorial note. These should be reconciled against evaluation logs.
Questions for authors

Q1. Are the version-space fractions calculated over syntactically distinct rule sets or distinct transition dynamics? Can the authors report both and show whether the difficulty trends persist? Are all structural constraints used to prune admissible hypotheses explicitly available to the evaluated models?

Q2. Can the authors provide an oracle-verification intervention that holds candidate generation approximately fixed, and quantify how much of the Hidden-Rule error it removes? Does the verification-bottleneck conclusion replicate beyond GPT-OSS-120B?

Q3. How much does Hidden-Rule accuracy decrease when the evidence is removed or shuffled? What happens with more than two hidden rules or experiments longer than three steps?

Q4. Can the authors provide paired confidence intervals and repeat the depth/version-space analyses within matched structural configurations? How sensitive are the results to stochastic decoding and benchmark-generation seeds?

Q5. Which token budgets were actually used, and are the reported token statistics means or medians?

4 comments

Thank you for your detailed and constructive review. Our responses to each comment follow.

(W1/Q2) Diagnostic analysis beyond GPT-OSS-120B

We repeated the rule-stating analysis for three models on a stratified 360-instance subset (Table A). In this variant (HR-RS), the model states the hidden rules before forecasting. Since this changes accuracy by up to 0.045, we treat it as a diagnostic of HR rather than a substitute.

Table A. Rule-stating analysis (360 instances). HR-RS is HR with rule stating, Consistent is the share of stated hypotheses that fit all observations, and Faithfulness the share of forecasts that follow the model’s own hypothesis.

ModelCRHRHR-RSConsistentFaithfulness
Qwen3-30B-A3B0.7510.4850.4630.0940.456
Gemma-4-26B-A4B0.8490.7060.6620.3500.748
GPT-OSS-120B0.9900.9120.9370.8140.967

Faithfulness measures how well a model executes its own hypothesis, as CR does for the given rules. For GPT-OSS-120B, the two are close (0.967 vs. 0.990), so its errors come from hypotheses inconsistent with the evidence rather than from execution. Most of these are near misses, with 48% violating one observation and 84% at most two (n = 44). For Gemma-4-26B-A4B (0.748 vs. 0.849) and Qwen3-30B-A3B (0.456 vs. 0.751), faithfulness falls below CR, so their gap also includes difficulty in using a hypothesis once found. Both costs arise only when rules are hidden, and the gap captures them together. This analysis cannot, however, separate failures to generate a consistent candidate from failures to check candidates against the evidence. We have therefore revised the paper to attribute HR errors to not reaching an evidence-consistent hypothesis, rather than to verification. A pilot tool-based oracle-verification intervention was inconclusive because tool use varied widely across models, so we leave it to future work.

(W2/Q1-1) Counting convention for the version space

Thank you for pointing out this inconsistency. The main text defines a hypothesis as a rule-level completion, but for efficiency the implementation merged completions whose one-step transitions are identical from every state. The original appendix statistics and version-space figure used this merged level. Since the forecast labels depend only on this transition (through ReachR(x)\mathrm{Reach}_R(x)), merged completions assign the same label to every evidence item and forecast. Instance validation, scoring, and all accuracies are therefore unaffected, and only ∣HD∣|\mathcal H_D|, ∣V(E)∣|\mathcal V(E)|, and their ratio differ. The revision reports these at the rule level (Appendix F.1).

Exhaustive enumeration over all 1,800 problems shows that the counts can differ substantially. For biology with three hidden rules, the median ∣V(E)∣|\mathcal V(E)| is 94 at the merged level but 660 at the rule level (Table B). The numerator and denominator grow together, however, so the fraction is nearly unchanged (Spearman 0.97), and its rank correlation with the gap changes by at most 0.02 for every model in Table C.

Table B. Hypothesis-space and version-space sizes per panel (median over all problems in the panel). Class: completions with identical behavior merged. Rule: rule-level completions.

DomainRobsR_{obs}RhidR_{hid}∥HD∥\|\mathcal H_D\| Class∥HD∥\|\mathcal H_D\| Rule∥V(E)∥\|\mathcal V(E)\| Class∥V(E)∥\|\mathcal V(E)\| RuleFraction ClassFraction Rule
Physics21219318481.83%2.52%
Physics2222,50750,403831970.36%0.39%
Physics231.45M5.31M3,79914,6040.25%0.28%
Physics31218317481.91%2.52%
Physics3222,22750,086972230.43%0.44%
Physics331.42M5.26M3,66713,6310.25%0.26%
Chemistry214040112.60%2.50%
Chemistry221,1391,560470.39%0.42%
Chemistry2328,23959,2801022020.37%0.34%
Chemistry313839225.13%5.13%
Chemistry321,0151,482480.40%0.54%
Chemistry3323,89354,8341011920.42%0.35%
Biology2164106243.42%3.77%
Biology221,5715,5659240.56%0.42%
Biology2318,789192,920946600.48%0.34%
Biology3159105233.57%2.86%
Biology321,3385,46011300.77%0.55%
Biology3315,103187,4601111,0660.82%0.57%

Table C. CR–HR gap Δ in the smallest and largest version-space-fraction bins, and Spearman correlation ρ between fraction and per-problem Δ (Class / Rule).

ModelΔ <0.5%Δ >5%ρ
Qwen3-8B+0.259/+0.255+0.206/+0.207−0.07/−0.07
Qwen3-30B-A3B+0.290/+0.286+0.254/+0.258−0.03/−0.02
Qwen3-32B+0.237/+0.232+0.220/+0.204−0.02/−0.04
Gemma-4-26B-A4B+0.200/+0.203+0.131/+0.134−0.12/−0.14
Mistral-Small-4-119B+0.355/+0.353+0.257/+0.242−0.15/−0.15
GPT-OSS-120B+0.151/+0.147+0.025/+0.026−0.27/−0.27
GPT-5 (high)+0.067/+0.067+0.006/+0.008−0.23/−0.23

(W2/Q1-2) Are all structural constraints available to the models?

Yes. Domain-specific constraints only define each domain’s grammar (e.g., biology rules act along the cycle), which the prompt states. The only additional filter applies to the public rules, which the prompt shows. We exclude rules that never change the state, so that every given rule matters for the problem. Hidden-rule candidates range over the full grammar, and forecast invariance is checked over this full space.

(W3/Q3) Ceiling effects and separating inference from execution

We agree that the CR–HR gap alone does not tell inference errors from execution errors. We first address the analyses requested in W3/Q3 and then discuss what they suggest about the two.

(1) More hidden rules

Following the suggestion, we added configurations with three hidden rules and experiments of up to four steps. On the extended benchmark, the HR accuracy of GPT-5 (high) falls from 0.983 to 0.968, and its gap doubles from 0.016 to 0.032 [0.026, 0.037]. For models that solve one-hidden-rule problems, the gap grows sharply with the number of hidden rules (Table D). For GPT-5 (high), it rises from 0.006 at h=1 to 0.048 at h=3, and its HR error rate increases about sevenfold. Weaker models already perform poorly at h=1, so their gaps stay flat.

Table D. HR accuracy and CR–HR gap Δ by the number of hidden rules h (1,800 instances).

ModelHR h=1HR h=2HR h=3Δ h=1Δ h=2Δ h=3
Qwen3-8B0.4170.4360.482+0.224+0.239+0.248
Qwen3-30B-A3B0.4520.4510.503+0.290+0.290+0.259
Qwen3-32B0.5330.5340.600+0.239+0.233+0.207
Gemma-4-26B-A4B0.6740.6520.702+0.167+0.188+0.146
Mistral-Small-4-119B0.5980.5490.603+0.294+0.339+0.292
Nemotron-3-Super-120B-A12B0.8920.7580.729+0.030+0.157+0.186
GPT-OSS-120B0.9710.8740.890+0.020+0.117+0.100
GPT-5-nano0.4490.3880.385+0.282+0.361+0.350
GPT-5-mini0.9630.8310.814+0.030+0.161+0.177
GPT-5 (low)0.9300.8220.824+0.062+0.171+0.171
GPT-5 (high)0.9930.9590.952+0.006+0.040+0.048

(2) Longer experiments

With evidence experiments of up to four steps, the gap grows with the number of steps for most models (Table E). This growth comes from HR: CR accuracy changes little, while HR accuracy falls. For GPT-5 (low), for example, HR falls from 0.968 to 0.774 while CR stays at 0.99. Longer experiments thus make the hidden rules harder to infer without making them harder to execute.

Table E. CR and HR accuracy by the number of evidence steps s (1,800 instances; Table 10 in the revision).

ModelCR s=1CR s=2CR s=3CR s=4HR s=1HR s=2HR s=3HR s=4
Qwen3-8B0.6720.6770.6940.6860.4480.4280.4520.452
Qwen3-30B-A3B0.7510.7410.7540.7470.4770.4540.4730.470
Qwen3-32B0.7890.7660.7860.7860.6240.5220.5300.545
Gemma-4-26B-A4B0.8650.8350.8410.8320.7640.6620.6360.642
Mistral-Small-4-119B0.9050.8850.8990.8770.6470.5800.5590.547
Nemotron-3-Super-120B-A12B0.9340.9250.9040.9070.8970.8140.7460.715
GPT-OSS-120B0.9930.9920.9880.9890.9520.9120.8940.888
GPT-5-nano0.7620.7430.7300.7160.4890.3950.3750.370
GPT-5-mini0.9960.9930.9900.9880.9690.8670.8390.802
GPT-5 (low)0.9950.9920.9930.9930.9680.8730.8190.774
GPT-5 (high)1.0000.9981.0000.9990.9960.9750.9580.942

(3) Removing or shuffling the evidence

Removing the evidence lowers HR accuracy for every model, and shuffling the outcomes has a similar effect (Table F). As a reference, the version-space reasoner forecasts with a hypothesis drawn uniformly from V(E)\mathcal V(E), and we report its expected accuracy. For GPT-OSS-120B, accuracy falls from 0.912 to 0.373 without evidence and to 0.364 with shuffled outcomes, close to the 0.335 of the version-space reasoner without evidence. With only three experiments, GPT-OSS-120B falls to 0.803, while the version-space reasoner still reaches 0.911.

Table F. HR accuracy under evidence controls (360 instances).

ConditionQwen3-30B-A3BGemma-4-26B-A4BGPT-OSS-120BVersion-space reasoner
5 experiments (default)0.4850.7060.9121.000
3 experiments0.4540.6400.8030.911
No evidence0.3170.4100.3730.335
Shuffled outcomes0.3750.4080.364–

(4) Budgeted search

We add a solver that samples k random candidate hypotheses and forecasts with the one that best fits the evidence (Table G). Its accuracy reaches 0.902 at k = 100 and 0.975 at k = 1,000. The chance of sampling an evidence-consistent hypothesis drops sharply with the number of hidden rules (0.939 for h=1 vs. 0.371 for h=3 at k = 100). The benchmark thus requires substantial search, and this demand grows with the number of hidden rules.

Table G. Budgeted random search (1,800 instances). Hit: an evidence-consistent hypothesis is among the k candidates. Best-fit: forecast accuracy of the best-fitting candidate.

k=1k=10k=100k=1,000k=10,000
Hit, h=10.0340.2990.9391.0001.000
Hit, h=20.0140.1160.4630.8930.991
Hit, h=30.0080.0740.3710.7720.961
Best-fit, all0.3330.6510.9020.9750.997

(5) Inference vs. execution

Together, these results point to inference as the main source of HR errors for capable models. Longer experiments lower HR but not CR accuracy (2), HR accuracy depends on the evidence (3), and GPT-OSS-120B executes its own stated hypotheses nearly as reliably as the given rules (faithfulness 0.967 vs. CR 0.990, W1/Q2). For weaker models, the gap also includes difficulty in executing their own hypotheses.

(W4/Q4) Confounding, paired CIs, and variability

We address the confounding of the version-space and depth analyses, the confidence intervals, and the sensitivity to decoding and generation seeds in turn.

(1) Version-space fraction and the number of hidden rules.

We agree that the version-space fraction is confounded with the number of hidden rules. To separate the two, we vary the fraction while keeping the world and the number of hidden rules fixed (Section 5.3 and Table 3 in the revision). On 360 instances, we extend the evidence from 5 to 7 and 10, nesting the sets so that the queries stay identical. CR accuracy is nearly unchanged, while the gap grows for Gemma-4-26B-A4B (0.142 → 0.187) and for GPT-OSS-120B at ten items (0.079 → 0.121). Qwen3-30B-A3B, near its floor, shows no increase. For capable models, a smaller version space thus increases difficulty even with the number of hidden rules fixed.

Table H. Nested evidence sets (360 instances).

ModelEvidenceCRHRΔ [95% CI]
Qwen3-30B-A3B50.7510.485+0.266 [+0.240,+0.292]
70.7540.495+0.259 [+0.233,+0.283]
100.7450.501+0.244 [+0.219,+0.267]
Gemma-4-26B-A4B50.8490.706+0.142 [+0.116,+0.168]
70.8460.673+0.173 [+0.150,+0.196]
100.8290.642+0.187 [+0.162,+0.214]
GPT-OSS-120B50.9900.912+0.079 [+0.060,+0.099]
70.9890.921+0.068 [+0.049,+0.088]
100.9890.868+0.121 [+0.098,+0.144]

(2) Evidence steps

The revised benchmark balances evidence steps within every structural configuration, with 25 instances for each number of steps from one to four in every panel. Evidence steps are therefore no longer confounded with the number of observed or hidden rules. Within this balanced design, CR accuracy is flat across evidence steps, whereas HR accuracy declines for capable models, so the gap grows with evidence steps. For GPT-OSS-120B, for example, CR stays at 0.99 while the gap increases from 0.041 to 0.100 (Table E in our response to W3/Q3). This indicates that multi-step evidence makes inference harder without affecting execution.

(3) Paired confidence intervals

We now report instance-level paired bootstrap 95% CIs for all models in the appendix (Table I). All CR–HR gaps are positive with CIs excluding zero, including GPT-5 (high), whose gap is 0.032 [0.026, 0.037].

Table I. Main results with paired bootstrap 95% CIs (1,800 instances).

ModelCR Acc. [95% CI]HR Acc. [95% CI]Δ [95% CI]
Qwen3-8B0.682 [0.674, 0.691]0.445 [0.436, 0.454]+0.237 [+0.227, +0.247]
Qwen3-30B-A3B0.748 [0.739, 0.757]0.468 [0.459, 0.478]+0.280 [+0.268, +0.291]
Qwen3-32B0.782 [0.774, 0.789]0.556 [0.545, 0.566]+0.226 [+0.215, +0.238]
Gemma-4-26B-A4B0.843 [0.835, 0.851]0.676 [0.665, 0.687]+0.167 [+0.156, +0.178]
Mistral-Small-4-119B0.891 [0.885, 0.897]0.583 [0.572, 0.595]+0.308 [+0.296, +0.321]
Nemotron-3-Super-120B-A12B0.917 [0.912, 0.922]0.793 [0.781, 0.805]+0.124 [+0.112, +0.136]
GPT-OSS-120B0.991 [0.989, 0.992]0.912 [0.903, 0.920]+0.079 [+0.070, +0.088]
GPT-5-nano0.738 [0.730, 0.746]0.407 [0.398, 0.418]+0.331 [+0.319, +0.342]
GPT-5-mini0.992 [0.990, 0.993]0.869 [0.858, 0.881]+0.123 [+0.111, +0.134]
GPT-5 (low)0.993 [0.992, 0.995]0.859 [0.847, 0.870]+0.135 [+0.124, +0.146]
GPT-5 (high)0.999 [0.999, 1.000]0.968 [0.962, 0.973]+0.032 [+0.026, +0.037]

(4) Decoding seeds

We rerun three models on the 360-instance subset with three decoding seeds, using each model’s recommended sampling settings. The standard deviation of the gap across seeds is 0.019 for Qwen3-30B-A3B, 0.015 for Gemma-4-26B-A4B, and 0.004 for GPT-OSS-120B, an order of magnitude smaller than the gaps themselves.

Table J. Decoding seeds (360 instances).

ModelSeedCRHRΔ
Qwen3-30B-A3B00.7510.485+0.266
10.7580.472+0.286
20.7370.488+0.248
Mean ± SD0.749 ± 0.0110.482 ± 0.009+0.267 ± 0.019
Gemma-4-26B-A4B00.8490.706+0.142
10.8380.688+0.149
20.8540.684+0.170
Mean ± SD0.847 ± 0.0080.693 ± 0.012+0.154 ± 0.015
GPT-OSS-120B00.9900.912+0.079
10.9930.907+0.086
20.9880.901+0.087
Mean ± SD0.990 ± 0.0030.907 ± 0.005+0.084 ± 0.004

(5) Generation seeds

We regenerate the full benchmark of 1,800 instances with three generation seeds. The standard deviation of the gap across seeds is at most 0.010, the CIs overlap, and the ranking of the models is unchanged.

Table K. Generation seeds (1,800 instances each).

ModelData seedCRHRΔ [95% CI]
Qwen3-30B-A3B00.7480.468+0.280 [+0.268,+0.291]
10.7560.475+0.282 [+0.270,+0.293]
20.7420.461+0.281 [+0.270,+0.292]
Mean±SD0.749±0.0070.468±0.007+0.281±0.001
Gemma-4-26B-A4B00.8430.676+0.167 [+0.156,+0.178]
10.8500.672+0.179 [+0.168,+0.190]
20.8490.662+0.187 [+0.176,+0.198]
Mean±SD0.847±0.0040.670±0.007+0.178±0.010
GPT-OSS-120B00.9910.912+0.079 [+0.070,+0.088]
10.9910.924+0.067 [+0.057,+0.077]
20.9910.920+0.070 [+0.060,+0.081]
Mean±SD0.991±0.0000.919±0.006+0.072±0.006

(W5/Q5) Miscellaneous: Token budgets, token statistics, and Section 5.3

Thank you for pointing out these inconsistencies. We have corrected them accordingly.

  • Token Limits: The 400K figure in Table 3 (now Table 6) was the provider’s total context window for the GPT-5 family, not the generation limit used in our evaluation. The token table, now in the appendix with the generation settings, reports the per-request output-token cap of 128K for all GPT-5 models (GPT-5-nano, GPT-5-mini, and GPT-5 low/high), consistent with the logs.
  • Median vs. Mean: The main text now reports the median number of generated tokens consistently, and Appendix B reports both the median and the mean for every model and condition.
  • Section 5.3: We agree that the smallest-bin gap was overstated for GPT-5 (high). We have revised the statement to reflect the smaller gap for this model while preserving the overall decreasing trend, and removed the leftover editorial note.

Comments on the paper

1 comment

We thank all reviewers for their constructive feedback. Below we summarize the main changes. All changes in the revised PDF are highlighted in blue.

  • Extended benchmark. We add configurations with three hidden rules and experiments of up to four steps. We report paired bootstrap 95% CIs for all models in the appendix.
  • Broader model coverage. We add Nemotron-3-Super-120B-A12B, Mistral-Small-4-119B, and Gemma-4-26B-A4B. We decode open-weight models with their recommended sampling settings.
  • Analyses of the gap. Section 5.3 now shows how the gap varies with the number of hidden rules, evidence steps, forecast steps, and the version space, including a controlled test that shrinks the version space with the number of hidden rules fixed. The appendix adds evidence controls, a budgeted random-search baseline, seed variability, and a rule-stating analysis on three models.
  • Clarity. Section 3 now begins with an overview based on Figure 2, Section 3.2 is shortened, and Section 4 describes how each domain composes rules. "Experiment" now refers only to interventions within a world, and Section 5 is renamed "Results".

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.