- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
Obs2Rule evaluates agents that design experiments, collect observations, and express discovered rules as executable Python functions. It contains 32 environments across 12 scenarios, including active, passive, and stochastic settings. Four LLM-driven agents are evaluated against human-designed test suites. An oracle-observation comparison is used to argue that experiment design is the main bottleneck.
Critical strengths
- Requiring an executable Python rule makes understanding directly testable and generalization-sensitive, unlike benchmarks that score only task completion.
- Removing the in-environment goal and reward, and separating active, passive, and stochastic scenarios, gives a clean setting for studying experiment design as its own skill.
- The scenario variations, including counter-intuitive ones, reduce the chance of passing by recalling commonsense priors. The paper is also honest that counter-intuitive variations are simpler, so this comparison is confounded with complexity.
Critical weaknesses
- The oracle condition reuses the final test cases as observations (Section 5). The active-scenario gain from 0.33 to 0.85 therefore shows that access to evaluation cases helps; it does not isolate experiment design from induction. Manually checking that code does not hard-code outputs does not rule out a lookup table or overfitting to the supplied cases. A held-out observation set covering the same rule branches is needed.
- The "experiment design is the bottleneck" claim is overgeneralized. The oracle condition gives complete evidence, which is an upper bound on evidence quality, not a measure of design skill. Passive scenarios barely improve and stochastic scenarios stay at 0.5; for multi-factor stochastic rules, induction remains a bottleneck. The defensible claim is narrower: evidence acquisition is a bottleneck in active scenarios.
- The benchmark conflates hypothesis selection with experiment design. The task description names the target action but not the relevant state variables; an agent can spend its budget on grid size or object color and fail the test cases without ever testing the relevant condition (door open/closed). Because the paper does not analyze the experiments agents actually propose, it cannot distinguish "did not identify the relevant variables" from "did not cover the relevant conditions." The oracle condition supplies observations covering the relevant conditions, so the gain may reflect being handed the hypothesis space.
- The comparison lacks a matched exploration budget and repeated-run uncertainty. The agent may propose unlimited experiments within three rounds, with up to 50 steps per experiment; the paper does not report total experiments, steps, model calls, or tokens per model. With temperature 1, a single run, and 32 environments, the model rankings and the bottleneck conclusion may be noise.
- The paper lacks the baselines needed to interpret its results: no prior-only/no-experiment condition, no random or fixed-coverage exploration, no human performance, and no scaffold ablation or stronger scientific-agent baseline. It also discusses related benchmarks only qualitatively and never runs them under a matched protocol, so the claim that task completion and rule discovery diverge is not empirically situated. Active vs passive and intuitive vs counter-intuitive comparisons are further confounded by rule complexity, and the 32 text-game environments support only a limited "scientific discovery" claim.
Questions for authors
-
Can you separate oracle observations from the final test cases, using held-out observations that cover the same rule branches? Does the 0.33→0.85 active-scenario gain survive that separation?
-
Please report total experiments, environment steps, model calls, and tokens per model, enforce a common total budget, and report repeated-run confidence intervals. Add prior-only, random/fixed-coverage, and human baselines so the contribution of agent-designed experiments is measurable.
-
Can you analyze the experiments agents actually propose—which variables they manipulate, whether they identify the relevant variables, and what branch coverage they achieve? A condition that provides a candidate variable list or state schema would separate relevance identification from experiment coverage.
-
Can you match active/passive and intuitive/counter-intuitive variations for rule complexity, ablate the agent scaffold (proposal rounds, revision, stopping criterion, prompting), and compare against a stronger scientific-discovery agent?
-
Can you run the same models on related benchmarks (e.g., ScienceWorld, ScienceAgentBench, DiscoveryWorld) under a matched protocol, and report a comparison table? Does task-completion performance actually diverge from rule-discovery performance?