acceptodds
Under review as a conference paper at ICLR 2027

Obs2Rule: Benchmarking the Rule Discovery Ability of LLM Agents

Abstract

For agents aiming to conduct scientific discovery autonomously, uncovering a new law requires designing experiments, seeing through complex experimental observations, and finally extracting the regularity that produced them. We formalize this process as rule discovery, comprising two stages: experiment design, deciding which observations to collect and how to collect them systematically, and rule induction, compressing the accumulated evidence into a general statement that also holds for cases never observed. To evaluate both stages jointly, we propose Obs2Rule, a text-based simulation benchmark of 32 manually designed environments spanning 12 scenarios. Each environment offers no goal to achieve and no reward; the agent must autonomously design and carry out experiments to uncover the latent rule that governs the environment, and express the induced rule as executable Python code, which is then verified against manually crafted test cases. This design decouples rule discovery from the task completion measured by most existing scientific discovery benchmarks, and provides an objective, generalization-sensitive criterion for grading induced rules. Evaluating four LLM-driven agents, we find that the best agent passes only 66% of environments, and just 53% of those requiring active intervention. Further analysis shows that current agents can articulate rules once given adequate evidence, but are weak at designing and executing the systematic experiments needed to obtain it.

Primary area:
datasets and benchmarks
Keywords:
text-based simulation, scientific discovery agents, rule induction

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_36WcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

Obs2Rule evaluates agents that design experiments, collect observations, and express discovered rules as executable Python functions. It contains 32 environments across 12 scenarios, including active, passive, and stochastic settings. Four LLM-driven agents are evaluated against human-designed test suites. An oracle-observation comparison is used to argue that experiment design is the main bottleneck.

Critical strengths
  • Requiring an executable Python rule makes understanding directly testable and generalization-sensitive, unlike benchmarks that score only task completion.
  • Removing the in-environment goal and reward, and separating active, passive, and stochastic scenarios, gives a clean setting for studying experiment design as its own skill.
  • The scenario variations, including counter-intuitive ones, reduce the chance of passing by recalling commonsense priors. The paper is also honest that counter-intuitive variations are simpler, so this comparison is confounded with complexity.
Critical weaknesses
  • The oracle condition reuses the final test cases as observations (Section 5). The active-scenario gain from 0.33 to 0.85 therefore shows that access to evaluation cases helps; it does not isolate experiment design from induction. Manually checking that code does not hard-code outputs does not rule out a lookup table or overfitting to the supplied cases. A held-out observation set covering the same rule branches is needed.
  • The "experiment design is the bottleneck" claim is overgeneralized. The oracle condition gives complete evidence, which is an upper bound on evidence quality, not a measure of design skill. Passive scenarios barely improve and stochastic scenarios stay at 0.5; for multi-factor stochastic rules, induction remains a bottleneck. The defensible claim is narrower: evidence acquisition is a bottleneck in active scenarios.
  • The benchmark conflates hypothesis selection with experiment design. The task description names the target action but not the relevant state variables; an agent can spend its budget on grid size or object color and fail the test cases without ever testing the relevant condition (door open/closed). Because the paper does not analyze the experiments agents actually propose, it cannot distinguish "did not identify the relevant variables" from "did not cover the relevant conditions." The oracle condition supplies observations covering the relevant conditions, so the gain may reflect being handed the hypothesis space.
  • The comparison lacks a matched exploration budget and repeated-run uncertainty. The agent may propose unlimited experiments within three rounds, with up to 50 steps per experiment; the paper does not report total experiments, steps, model calls, or tokens per model. With temperature 1, a single run, and 32 environments, the model rankings and the bottleneck conclusion may be noise.
  • The paper lacks the baselines needed to interpret its results: no prior-only/no-experiment condition, no random or fixed-coverage exploration, no human performance, and no scaffold ablation or stronger scientific-agent baseline. It also discusses related benchmarks only qualitatively and never runs them under a matched protocol, so the claim that task completion and rule discovery diverge is not empirically situated. Active vs passive and intuitive vs counter-intuitive comparisons are further confounded by rule complexity, and the 32 text-game environments support only a limited "scientific discovery" claim.
Questions for authors
  1. Can you separate oracle observations from the final test cases, using held-out observations that cover the same rule branches? Does the 0.33→0.85 active-scenario gain survive that separation?

  2. Please report total experiments, environment steps, model calls, and tokens per model, enforce a common total budget, and report repeated-run confidence intervals. Add prior-only, random/fixed-coverage, and human baselines so the contribution of agent-designed experiments is measurable.

  3. Can you analyze the experiments agents actually propose—which variables they manipulate, whether they identify the relevant variables, and what branch coverage they achieve? A condition that provides a candidate variable list or state schema would separate relevance identification from experiment coverage.

  4. Can you match active/passive and intuitive/counter-intuitive variations for rule complexity, ablate the agent scaffold (proposal rounds, revision, stopping criterion, prompting), and compare against a stronger scientific-discovery agent?

  5. Can you run the same models on related benchmarks (e.g., ScienceWorld, ScienceAgentBench, DiscoveryWorld) under a matched protocol, and report a comparison table? Does task-completion performance actually diverge from rule-discovery performance?

Reviewer_vRirOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formalizes scientific discovery from the perspective of rule discovery comprising of experiment design and rule induction steps. The paper proposes a benchmark with 32 environments and 12 scenarios that test agents on discovering latent rules that govern the environment.

Critical strengths

Formulation of scientific discovery is novel and offers a different perspective than existing scientific benchmarks.

The paper contributes insightful findings from experiments and analysis. In particular, I found interesting that the agents mainly fail by missing edge cases rather than fundamentally not being able to induce the rule. It is also interesting that agents perform better on counter-intuitive tasks.

Critical weaknesses

Although the benchmark tests numerous scenarios (active, passive, stochastic), the task seems pretty narrow in scope in that it takes place in a 7 by 5 grid with fixed actions. The task feels a bit toy-ish. The paper could benefit greatly from either a more diverse set of tasks or a more realistic one.

The analysis on a counter-intuitive scenario is interesting, but one task is not enough to conclude that agents do not rely on pre-existing knowledge. The paper claims that what is more important is the complexity of the problems and that more complex tasks in the benchmark tend to be more intuitive ones. This section could benefit a lot from more concrete evidence that this is the case.

The paper proposes an agent architecture from scratch, and not from an existing architecture. As a result, the design decisions are not very well motivated. The paper could benefit from an explanation of the agent harness and ablations on it if possible.

The findings in the first discussion point suggests a key weakness in the benchmark. Models can rely on prior knowledge to solve tasks, so the benchmark often does not actually measure their ability to induce rules from observations only.

Questions for authors

Are there any insights to why observation accuracy is higher for Kimi than the pass rate while for DeepSeek and GLM, it is the opposite?

Reviewer_yuhYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Obs2Rule evaluates agents that explore text-based environments and then submit executable rules predicting state transitions or probabilities. It contains 32 environments across twelve active, passive, and stochastic scenarios, including counter-intuitive variants. Four agents are evaluated using environment-level and case-level correctness. An additional experiment supplies human-authored observations to separate evidence collection from rule construction. The main conclusion is that systematic experimental design is a larger bottleneck than rule induction.

Critical strengths
  1. The paper provides a useful way to evaluate what an agent has learned from interaction. Requiring an executable rule tests whether its understanding extends beyond completing one particular task, and state-transition tests avoid reliance on an LLM judge.

  2. The environment variations probe different requirements for discovery. Active interventions, passive observations, and stochastic sampling offer complementary tests, while counter-intuitive rules reduce the reliability of simply applying commonsense assumptions.

  3. Reporting both environment-level and case-level success is informative. The large gap between these measures shows that many agents capture most conditions but miss specific cases, which is useful for understanding their failure modes.

Critical weaknesses
  1. The oracle experiment does not independently test rule generalization. It supplies the same manually designed cases later used for scoring. Checking that outputs are not literally hard-coded does not resolve this issue; demonstrations disjoint from the scoring cases are needed to support the conclusion about induction ability.

  2. The diagnosis of poor experiment design is not sufficiently isolated. Missing observations can also result from execution failures or early stopping. Moreover, incorrect observation wording does not necessarily mean that a condition was never explored. Direct coverage measurements would be more convincing than exact string accuracy.

  3. The experimental comparison needs better budget and uncertainty controls. Agents can propose unlimited experiments despite the fifty-step limit per experiment, and temperature-one results are reported without repeated-run uncertainty. A systematic exploration baseline under a common total budget would make the comparison more informative.

  4. The stochastic results are difficult to interpret without sample counts. The fixed 0.01 tolerance may require substantially different sampling effort across conditions. All models also remain at 50% with oracle observations, suggesting an induction limitation that should be distinguished from the active-scenario findings.

Questions for authors

See weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.