acceptodds
Under review as a conference paper at ICLR 2027

ExplorationBench: Measuring AI Systems' Exploration in Verifiable Alien Worlds

Abstract

Scientific discovery begins where known problems end. There, AI systems must engage in exploration: framing hypotheses, designing experiments, and iterating on the results. However, evaluating this ability is difficult: (1) how to verify whether a genuinely new hypothesis holds, and (2) how to determine whether a system has discovered it through exploration or merely recalled related knowledge from pre-training data. To this end, we introduce ExplorationBench, which turns the wicked problem of evaluating scientific exploration into a concrete and tractable framework built on verifiable Alien Worlds: their rules are executable, so every answer can be checked exactly, and they conflict with familiar knowledge, so recall alone cannot solve the tasks. The benchmark contains two sandboxes, AlienCode (31 discovery targets, 70 tasks) and AlienLogic (24 discovery targets, 70 tasks). Each sandbox provides a flawed manual, task-specific environmental feedback, and a dedicated tool-call schema. Systems use these resources to explore the sandbox, then solve held-out tasks. We evaluate 10 AI systems and find that the strongest systems can acquire and apply unfamiliar rules, while performance varies substantially across trajectories and continued exploration can stall or reverse earlier gains. ExplorationBench represents a step towards AI systems that can acquire and apply genuinely new knowledge through exploration in unknown environments.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, Exploration, Scientific Discovery, LLM Agents, Tool Use, Verifiable Environments

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_9jG5OpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a benchmark dataset, called ExplorationBench, to evaluate if AI systems can get real new knowledge by actively exploring, rather than only relying on recalling and reasoning over existing knowledge. Specifically, it introduces AlienCode and AlienLogic, which are two executable sandboxes with deliberately conflicting rules against those of familiar priors. The system learns by exploring and interacting with the environment for 4 rounds, and after that it is asked to solve questions in the held-out set without the help of tools. Three distinct milestones, including probe design, rule discovery, and rule application, are reported and evaluated verifiably without the help of an LLM judge.

Critical strengths
  1. All the questions in this benchmark are rigorously verifiable via an interpreter or proof checker, and no LLM judge is involved. The benchmark is designed in such a way that the pretraining priors actively conflict with the hidden rules, so the model is also not gonna be able to hack it by recalling. All this makes the work a trustworthy benchmark to probe LLM systems' ability to explore new knowledge and is resistant to data contamination.

  2. The work provides two different sandboxes that comprehensively measure LLMs' capability across different cognitive operations, including both program synthesis and formal proof. The detailed case study provided by the authors also provides insight into the underlying mechanisms of LLMs for exploring new knowledge.

Critical weaknesses
  1. The benchmark datasets are still of a synthetic nature. Without the presence of other natural science domains, it remains unclear how the capabilities benchmarked against this work would translate into performance on real-world scientific exploration.

  2. Only 3 trajectories per system per sandbox are sampled, which are not sufficient for estimating the trajectory distribution at any level of precision. The 73.3 points variance shown in finding 7 is very extreme, confirming this lack of precision. Also, the Best@3 metric itself can be misleading, since this metric only answers "if a system can ever succeed" rather than "if this system can reliably succeed", which may, in turn, favor unreliable systems with just one lucky trajectory.

Questions for authors
  1. Your Finding 2 (designing probes beats receiving identical evidence) is one of the most intriguing results in the paper. Can you provide a more detailed analysis of the mechanism behind this finding, or do a case study of the high-performing trajectories to characterize their behavior?

  2. Can you report rankings at higher ns, say n=5-10? You claimed in the paper that the choice of aggregate changes the ranking, but I am still not fully convinced, since more trajectories might, in turn, stabilize the ranking.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 9jG5 (1/3)

We sincerely thank you for your recognition of our work and your valuable suggestions! To answer your questions, we ran 140 new autonomous trajectories so that every system now has ten per sandbox, replayed the probes of each system's first five trajectories to study the mechanism behind Finding 2, and traced two high-scoring trajectories probe by probe. All new experiments and changes are included in the revised paper.

Weakness 1: The benchmark datasets are still of a synthetic nature. Without the presence of other natural science domains, it remains unclear how the capabilities benchmarked against this work would translate into performance on real-world scientific exploration.

Response: We agree that this is a boundary of our work. The design choice rests on three considerations:

(1) Artificial worlds make exploration exactly measurable. Measuring exploration requires two things at once. First, answers must be exactly verifiable, whereas testing a new hypothesis in real science is often costly and slow, and observations are noisy. Second, answers must not be recallable, whereas the laws of known domains are present in pretraining corpora. The rules of the Alien Worlds are executable and conflict with what models know, so the complete loop of proposing hypotheses, designing experiments, reading feedback and applying new knowledge can be measured in isolation.

(2) The two worlds cover different cognitive operations. AlienCode tests program semantics and AlienLogic formal reasoning. The rankings of the ten systems in the two worlds have a Spearman correlation of only 0.35: a system that explores well in one world does not necessarily do so in the other, so the benchmark does not measure a single narrow skill.

(3) The framework extends. The procedure for constructing a world is general, and the same method can generate new worlds; the supplementary material already provides two demo worlds and the complete evaluation code.

Appendix D states this boundary: our results concern executable rules and deterministic feedback, not real research with physical experiments and noisy observations. The revision adds there that noisy feedback, experiment costs and open-ended hypothesis spaces are our next step.

Weakness 2 & Question 2: Only 3 trajectories per system per sandbox are sampled, which are not sufficient for estimating the trajectory distribution at any level of precision. [...] Also, the Best@3 metric itself can be misleading [...]. Can you report rankings at higher ns, say n=5-10?

Response: You are right that more trajectories estimate the distribution more precisely. We therefore extended all ten systems to ten autonomous trajectories in both sandboxes, all under the paper's protocol.

Agreement with the ranking at ten trajectories (Spearman correlation):

SandboxComparisonBestMeanWorst
AlienCode3 vs. 10 trajectories0.910.990.95
AlienCode5 vs. 10 trajectories0.910.980.97
AlienLogic3 vs. 10 trajectories0.700.830.82
AlienLogic5 vs. 10 trajectories0.900.900.87

Results with ten trajectories (M4, %):

SystemAlienCode Best@3Best@10Mean@10AlienLogic Best@3Best@10Mean@10
Claude Opus 589.093.871.379.583.875.3
GPT-5.6 Sol88.190.581.976.797.175.8
Gemini 3.8 Flash79.089.554.859.576.766.2
Kimi K379.079.027.974.379.567.4
Grok 4.669.084.356.181.081.977.0
Qwen3.8-Max65.782.966.377.677.669.7
Hy4 preview63.363.326.476.280.567.4
Seed2.1 Pro40.067.125.269.070.060.4
DeepSeek-V4-Pro12.933.316.175.275.263.8
DeepSeek-Flash 1017*68.676.254.668.176.265.6

*DeepSeek-V4.1-Flash has been retired; DeepSeek-Flash 1017 stands in for it. Best@3 uses the first three trajectories, which for the other nine systems are the three in the paper.

We draw the following conclusions:

(1) Rankings are largely stable with more trajectories (first table). In AlienLogic, where scores lie closer together, the correlations are lower at three trajectories but already about 0.9 at five.

(2) Systems with close scores need more trajectories or confidence intervals to separate. In AlienLogic, all ten systems lie between 60.4 and 77.0 by Mean@10. If each system keeps three of its ten trajectories at random, Best@3 orders 84% of system pairs the same way as Best@10 in AlienCode, but only 77% in AlienLogic. The revision accordingly limits our statement about the choice of aggregate to systems with close scores and adds confidence intervals.

(3) The Best@n bias you point out is real, and it grows with n. From three to ten trajectories, Best@n rises by up to 27.1 points in AlienCode and 20.5 in AlienLogic. For example, Seed2.1 Pro's Best@n in AlienCode rises from 40.0 at n = 3 to 67.1 at n = 10, while its Mean@10 is only 25.2. The next comment explains why the main ranking still uses three attempts.

AuthorsOpenReview ↗

Response to Reviewer 9jG5 (2/3)

Why the main ranking uses three attempts (W2, continued). This is part of the evaluation design, not only a matter of cost. We measure how well a system explores with limited opportunities: at most 12 tool calls in each of four rounds, and likewise only three independent attempts. Best@3 therefore asks not whether a system can succeed at all, but whether it succeeds within three attempts. A system that needs many attempts to produce one good trajectory explores less efficiently, and once attempts are unlimited, the score increasingly rewards retrying, as the Seed2.1 Pro example shows. Keeping n small is precisely what limits the effect of "one lucky trajectory" that you are concerned about.

For reliability, the revision reports Mean@n and Worst@n beside Best@3, with bootstrap confidence intervals for Mean@n (Table 6); where the Mean@n intervals of two systems overlap, we do not claim that their reliability differs. The paper's main conclusions rest on controlled comparisons between conditions, not on fine-grained rankings between systems. These results are in Appendix G.3 (Table 6), and the discussion of Best@n is in Appendix C.1.

Question 1: Your Finding 2 (designing probes beats receiving identical evidence) is one of the most intriguing results in the paper. Can you provide a more detailed analysis of the mechanism behind this finding, or do a case study of the high-performing trajectories to characterize their behavior?

Response: To analyze the mechanism, we ran a paired hindsight experiment: each of the first five autonomous trajectories of every system is replayed with its own probes, and each trajectory is compared with its own replay, so that neither side is selected by its score. The results led us to revise the argument for this finding, and revealed a new one.

(1) The original gap came mainly from selecting the best trajectory. Given identical evidence, designing the experiments autonomously brings no detectable extra benefit. Over 49 pairs in AlienCode, the mean difference is −2.2 points (95% CI −9.8 to +5.3, p = 0.40). Over 50 pairs in AlienLogic, it is 0.0 points (95% CI −2.1 to +1.9), statistically equivalent within ±2 points (see our response to Reviewer VuUJ).

(2) Yet designing one's own experiments still matters. At three repetitions each, fixed-probe exploration falls below autonomous exploration for 10/10 systems in both sandboxes, by a median of 31.7 points in AlienCode and 16.0 in AlienLogic (p = 0.002). The revised Finding 2 is therefore: the gain of exploration comes from the evidence carried by the experiments a system designs itself; handing the same evidence back to the same system recovers the gain, whereas generic probes do not.

(3) Mechanism: the bottleneck lies at a different step in each sandbox.

SandboxMean absolute difference, trajectory vs. its replayCorrelationShare of between-trajectory spread explained by the probes
AlienCode14.5 points0.7343%
AlienLogic3.4 points0.8889%
  • AlienLogic: the evidence almost determines the outcome; which proofs were tried sets the accuracy reached.
  • AlienCode: more than half of the spread remains once the evidence is fixed. For example, Claude Opus 5's second trajectory reaches 49.0% autonomously but 83.3% when its probes are replayed; Gemini 3.8 Flash's fourth trajectory reaches 76.2% autonomously but only 8.1% on replay.

This gives controlled support for Finding 4 of the paper: AlienLogic is limited by discovering the rules, and AlienCode by using them.

(4) What high-scoring trajectories do: the key evidence comes early and quickly becomes a rule. We compare the 25 autonomous AlienCode trajectories that end at or above 75% (mean 83.1%) with the 27 that end below 25% (mean 12.7%), focusing on the two keystone rules R14 (the PLUCK index shifts left by one) and R15 (both CARVE bounds shift left by one):

  • High-scoring trajectories: all 25 ran an experiment exposing R14 within the first two rounds and state R14 correctly at M4; 23 stated it in the round of exposure or the next, 15 in the same round. R15 is similar: 24 exposed it within two rounds, all 25 state it at M4, and 24 stated it within one round of exposure.
  • Low-scoring trajectories meet such experiments later. Of the 24 that exposed R14, only 1 did so in round 1, most in rounds 2 and 3; none of the 22 that exposed R15 did so in round 1. Most also never state the rule: 20 of the 24 for R14 and 19 of the 22 for R15.

The next comment traces two such trajectories.

AuthorsOpenReview ↗

Response to Reviewer 9jG5 (3/3)

Two high-scoring trajectories (Q1, continued). The rules involved are all introduced in the paper's case studies; lists and indices below are actual values after the rewrite.

  • Integer literals are rewritten (R01, R01+). At the top level an integer is XOR-ed with 27, so 0 is written 27 and 1 is written 26. Inside a function (CRAFT) body it is XOR-ed with 53 instead, so 0 is written 53 and 1 is written 52.
  • R14: PLUCK(seq, i) actually takes element i−1, so index 0 returns the last element.
  • R15: both bounds of the slice CARVE shift left by one.
  • R16: STRAND builds a list in reversed order.

Case 1: misread at first, then corrected by an experiment that tells two readings apart (Claude Opus 5, trajectory 1, M4 82.4%).

  • Round 1: the experiment cannot tell two readings apart. The system takes index 0 of the list [24, 25, 26] and gets 26. Two readings explain this: under the true rule the index shifts left, 0 becomes −1, and the last element, 26, is returned; under "reverse the list, then index", element 0 of [26, 25, 24] is also 26. Since STRAND already reverses, the system chose reversal: its M1 report states R14 as (get (reverse seq) i) and R15 with swapped bounds as (slice seq (+ hi 1) (+ lo 1)), and accuracy is 16.2%.
  • Round 2: an experiment that tells them apart. The system takes index 1 of the four-element list [31, 24, 25, 26]: shifting left predicts 31, reversing predicts 25, and the sandbox returns 31. The system then writes "PLUCK is 1-based". In the same round, a slice on four elements returns [31, 24], where the swapped bounds predict an empty list.
  • Result: the M2 report states R14 and R15 correctly, and accuracy rises from 16.2% to 83.8%, and from 12.4% to 81.0% on the 51 tasks that use R14 or R15. When all of this trajectory's probes are replayed to the same model, the jump occurs at the same milestone (M1 16.7%, M2 82.9%).

Case 2: rules stated correctly but not used, until the system chased an anomaly it had recorded (Claude Opus 5, trajectory 5, M4 90.0%).

  • Stated, but not used. From M0 on, this trajectory states R01+ correctly from the worked examples (69 of the 70 tasks need it). At M3 its report states 26 rules correctly, including R14 and R15, yet accuracy is only 14.3%, because its answers still encode numbers inside function bodies with the top-level key 27. For example, task A46 asks for a function that returns the last element of a list. The answer DELIVER PLUCK(lst, 27) means index 0, but inside a function body 27 becomes 46, the index is out of range, and every test fails. Of the 70 answers at M3 (the first per task), 59 write top-level values such as 27 and 26 inside function bodies, and only 3 write 53 or 52.
  • An anomaly it recorded. The last experiment of round 3 is a recursive factorial. The system meant "return 1 when n ≤ 1" and wrote UPON BENEATH(n, 26): DELIVER 26, but inside the body 26 becomes 47, so fact(5) returns 47. Its round summary says: "Unsolved: the recursive factorial fact(5) returns 47 (should be 120). … the next round must locate the true semantics of recursion and nested calls."
  • Round 4 chases it. The system reproduces the effect with several functions and writes "Breakthrough! Literals inside function bodies seem to use a different key." It then writes 0, 1 and 100 inside a function body, gets 53, 52 and 81, and concludes "Key inside function bodies = 53 (top level = 27), fully verified." Rewritten with key 53, the factorial returns the correct 120.
  • Result: the report states the same 26 rules at M4 as at M3, yet accuracy rises from 14.3% to 90.0%, and from 9.8% to 88.2% on the 51 tasks that use R14 or R15. 66 of the 70 answers now use the function-body key, and A46 passes every test.

The two cases show three things:

  • High-scoring trajectories run targeted experiments. Each one either separates two of the system's own hypotheses (Case 1) or chases an anomaly it met (Case 2), designed from its hypotheses at the time. Fixed generic probes cannot do this, which can explain why fixed-probe exploration falls far below autonomous exploration.
  • Once the experiment is run, the evidence suffices. Replayed with the same probes, the same model learns at the same point (Case 1). This is the revised Finding 2.
  • Stating a rule is not using it. In Case 2 the rule report does not change at all, yet accuracy rises by 75.7 points, consistent with Finding 6.

In summary, the revision rewrites Finding 2 according to these results. The paired experiment is in Appendix G.1 (Table 4), the statistics on high- and low-scoring trajectories in Appendix G.4, and the two cases are Cases 5 and 6 in Appendix H.

Thank you again for your careful review and valuable suggestions. All new experiments and changes are in the revised paper; please let us know if you have any further questions.

Reviewer_VuUJOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ExplorationBench, a benchmark for evaluating whether AI systems can discover unfamiliar rules through autonomous experimentation and apply what they have learned to unseen tasks. The authors construct two verifiable environments, AlienCode and AlienLogic, where the underlying rules differ from familiar programming and logical semantics. The benchmark tracks performance throughout exploration and includes several control conditions to study the roles of experiment selection, rule discovery, and rule application. Experiments across ten AI systems reveal substantial differences in exploration ability, a gap between knowing and applying rules, and considerable variability across exploration trajectories.

Critical strengths

1. The paper addresses an important and well-motivated research question. I consider the problem formulation to be the strongest contribution of this paper. Being able to solve problems using existing knowledge is different from knowing what experiments to conduct in an unfamiliar environment and learning from their outcomes. Evaluating the latter is challenging: the rules need to be sufficiently unfamiliar to prevent reliance on pretrained knowledge, while the evaluation still requires reliable ground truth. The proposed Alien Worlds provide an effective way to address this tension. Importantly, the benchmark evaluates whether the acquired knowledge can be used on unseen tasks, rather than relying solely on the model's reported discoveries. I find this a meaningful evaluation setting with clear relevance to future research on autonomous learning and scientific discovery.

2. The experiments provide interesting insights beyond benchmark rankings. I particularly appreciate the open-book experiments in Figure 4. In AlienLogic, providing the complete rules leads to substantially better performance than autonomous exploration, whereas in AlienCode, having an exploration history can remain beneficial even when the rules are explicitly provided. This suggests that acquiring rules and applying them can present different challenges. The analysis of rule reports and the large performance differences across trajectories further reveal limitations that would be difficult to identify from final accuracy alone. These findings make the paper more than a benchmark release and offer useful directions for subsequent research.

Critical weaknesses

1. The hindsight comparison does not fully isolate the benefit of autonomous experiment selection. In Finding 2, the authors compare the Best@3 autonomous trajectory with a hindsight run that replays its probes. While this is a useful control, the autonomous trajectory is selected based on its final accuracy, which reflects both the quality of the selected experiments and how successfully the model interprets and applies their results. The hindsight run receives the same experimental feedback but must independently interpret it and update its hypotheses. Therefore, the observed gap may partly reflect the selection of a particularly successful reasoning trajectory, rather than the benefit of choosing experiments autonomously. This distinction matters because Finding 2 is one of the paper's central conclusions. A comparison without this post-hoc selection asymmetry would strengthen the claim. Otherwise, the authors should interpret the results more cautiously, particularly the conclusion that autonomous experiment selection adds no benefit in AlienLogic.

Questions for authors

1. On the interpretation of Finding 2. Is there a comparison that does not select the autonomous reference trajectory based on its final accuracy, but still supports the advantage of autonomous exploration over hindsight replay? This would help clarify how much of the observed difference comes from experiment selection rather than the post-hoc selection of a successful trajectory. A related question is whether the benefit comes primarily from planning informative experiments or from adapting subsequent experiments to the feedback received.

2. On the sources of failure in unsuccessful exploration trajectories. The case studies in Appendix H provide useful examples of how exploration can succeed or fail. Could the authors further examine whether low-performing trajectories failed to conduct experiments that would reveal important rules, or whether they received relevant evidence but failed to interpret or retain it correctly? For example, an analysis focusing on the key rules identified in AlienCode could help distinguish failures in experiment selection from failures in evidence interpretation. I believe this distinction would make the benchmark's diagnostic findings more informative.

3 comments
AuthorsOpenReview ↗

Response to Reviewer VuUJ (1/2)

We sincerely thank you for your recognition of our work, especially of the problem formulation and the open-book experiments! For your questions, we ran 82 new paired replays (99 pairs in total) and a failure attribution that re-executes every probe of 103 AlienCode trajectories. We address each point below; all new experiments and changes are included in the revised paper.

Weakness 1 & Question 1: The hindsight comparison does not fully isolate the benefit of autonomous experiment selection. In Finding 2, the authors compare the Best@3 autonomous trajectory with a hindsight run that replays its probes. While this is a useful control, the autonomous trajectory is selected based on its final accuracy, which reflects both the quality of the selected experiments and how successfully the model interprets and applies their results. The hindsight run receives the same experimental feedback but must independently interpret it and update its hypotheses. Therefore, the observed gap may partly reflect the selection of a particularly successful reasoning trajectory, rather than the benefit of choosing experiments autonomously. This distinction matters because Finding 2 is one of the paper's central conclusions. A comparison without this post-hoc selection asymmetry would strengthen the claim. Otherwise, the authors should interpret the results more cautiously, particularly the conclusion that autonomous experiment selection adds no benefit in AlienLogic. Is there a comparison that does not select the autonomous reference trajectory based on its final accuracy, but still supports the advantage of autonomous exploration over hindsight replay? This would help clarify how much of the observed difference comes from experiment selection rather than the post-hoc selection of a successful trajectory. A related question is whether the benefit comes primarily from planning informative experiments or from adapting subsequent experiments to the feedback received.

Response: Thank you for your thorough and precise analysis! You are completely right: the Best@3 trajectory is selected by its final accuracy, so its gap to hindsight may include a selection component. Following your suggestion, we ran a paired hindsight experiment without post-hoc selection:

  • Pairing: for each of the first five autonomous trajectories of every system (both sandboxes), the same model replays that trajectory's own probes, and the two are compared pair by pair, with neither side selected by outcome.
  • Other settings: identical to the paper.
SandboxPairsPairs with autonomous higherMean difference (autonomous − replay)95% CIWilcoxon p
AlienCode4922−2.2−9.8 to +5.30.40
AlienLogic50270.0−2.1 to +1.90.39

Note: one replay could not be completed because the provider repeatedly refused the requests, so AlienCode has 49 pairs.

We draw the following conclusions:

(1) Without selection, autonomous exploration and hindsight do not differ detectably. This holds at every stage: the mean difference at each milestone from M1 to M4 is within ±4 points, and in AlienCode a trajectory states 21.6 of the 31 rules correctly at M4 on average, against 22.0 for its replay.

(2) The AlienCode gap in the submitted paper (autonomous ahead for 9/10 systems by a median of 17.4 points) came mainly from post-hoc selection. The revision removes this claim.

(3) The AlienLogic conclusion that autonomous experiment selection adds no benefit holds under the paired comparison, and is now backed by statistical equivalence within ±2 points.

(4) Self-designed experiments remain clearly better than generic probes: at equal repetitions, for 10/10 systems in both sandboxes, p = 0.002 (see our response to Reviewer 136r).

On planning versus adapting: a replay keeps the content and order of the experiments and removes only the step of deciding what to test next after seeing the feedback, and it ties with autonomous exploration. Once the same experiments are run, adapting them to feedback on the fly adds no detectable value; what matters is the information the experiments carry. Fixed probes are as many but unrelated to the system's hypotheses, and are clearly worse. We cannot further separate planning good experiments in advance from reaching a good set by adapting along the way, because both end in the same sequence of probes.

In summary, the revised Finding 2 keeps the original thread: the gain of exploration comes from the evidence carried by the experiments the system designs itself, not from whether it decides each one on the fly. The revision rewrites the main text accordingly, and the full paired experiment is in Appendix G.1 (Table 4).

AuthorsOpenReview ↗

Response to Reviewer VuUJ (2/2)

Question 2: On the sources of failure in unsuccessful exploration trajectories. The case studies in Appendix H provide useful examples of how exploration can succeed or fail. Could the authors further examine whether low-performing trajectories failed to conduct experiments that would reveal important rules, or whether they received relevant evidence but failed to interpret or retain it correctly? For example, an analysis focusing on the key rules identified in AlienCode could help distinguish failures in experiment selection from failures in evidence interpretation. I believe this distinction would make the benchmark's diagnostic findings more informative.

Response: Thank you for this very valuable suggestion! Following your idea, we attributed failures for the two keystone rules of AlienCode: R14 (the PLUCK index shifts left by one) and R15 (both CARVE bounds shift left by one), which enter 51 of the 70 tasks.

  • Exposure: every probe of every trajectory is executed twice, with the complete rule set and with one rule switched off. If the outputs differ, the probe exposes that rule, since the feedback the system read depended on it. The worked examples expose neither rule.
  • Three outcomes: comparing the rule report at M4 with whether the rule was ever exposed, each trajectory either states the rule correctly, was exposed to it but does not state it (an interpretation failure), or was never exposed to it (a selection failure).
RuleStated correctlyExposed, not statedNever exposed
R1461 (mean M4 64.4%)39 (25.8%)3 (3.8%)
R1564 (64.3%)34 (23.6%)5 (5.0%)

The table covers 103 autonomous AlienCode trajectories: ten for each of the ten systems, plus the three of DeepSeek-V4.1-Flash in the paper. We draw the following conclusions:

(1) Failures lie mainly in interpreting the evidence. Of the 42 trajectories that do not state R14 correctly, 39 had already executed a probe that exposes it; for R15, 34 of 39.

(2) Selection failures are rare but the most severe: only 3 and 5 trajectories, with mean M4 of 3.8% and 5.0%.

(3) The key evidence mostly comes early. Of the trajectories that exposed R14, 84% first did so within the first two rounds; for R15, 66%.

(4) Whether the same evidence is read as a rule is itself highly variable. A trajectory and its replay execute the same probes, so their exposure is identical. Yet in 24% of the pairs for R14 and 22% for R15, one states the rule and the other does not, about equally often in each direction (6 vs. 6 for R14, 6 vs. 5 for R15).

This agrees with a variance decomposition of the paired replays (Appendix G.1): in AlienCode, the probes explain only about 43% of the spread between trajectories. The analysis is in Appendix G.4 (Table 7) of the revision, and Finding 5 now adds the conclusion: in AlienCode, the main bottleneck of exploration is reading rules out of feedback already obtained, not thinking of which experiments to run.

Thank you again for your careful review and constructive comments, which have substantially strengthened our analysis. We have included all new experiments and changes in the revised paper. If you have any further questions, please let us know and we will do our best to respond.

Dear Reviewer VuUJ,

Thank you again for your thoughtful review. As the discussion period closes on November 18 (AoE), we would like to check whether our responses address your two questions: the paired comparison without post-hoc selection (99 pairs), on which we revised Finding 2, and the failure attribution for the two keystone rules, which shows that most misses come from reading the evidence rather than collecting it. We would be glad to clarify anything further.

Reviewer_136rOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether models can proactively conduct experiments to discover unfamiliar rules and subsequently apply them to new tasks. The authors construct two environments—a "code world" and a "formal logic world"—where the underlying rules have been modified. These environments contain 31 and 24 discovery targets, respectively, along with 70 tasks for each. Starting with an error-prone manual and a set of fixed examples, the model undergoes four rounds of exploration. After each round, it must report the discovered rules and solve held-out tasks within an isolated, tool-free environment.

Critical strengths
  • By requiring both the collection of evidence and the completion of held-out tasks, the framework provides a more robust assessment of whether the discovered knowledge is truly actionable, rather than merely restated.
  • The paper goes beyond reporting final performance metrics; it illustrates the progression across rounds, highlights differences between various exploration trajectories, and details how the discovered rules are applied.
Critical weaknesses
  • The primary baseline for autonomous exploration is derived from the best-performing trajectory among three attempts, whereas the "hindsight," "fixed-probe," and "no-tool" baselines are typically evaluated based on a single run. Simply re-running the same probes used in the best trajectory—while potentially aligning with the environmental evidence—fails to account for the selection bias inherent in choosing the best trajectory based on final performance (which may have conflated successful reasoning, effective knowledge integration, and accurate answer generation).
  • In the formal logic world, the "unprovable" label is determined by a search constrained by a finite set of formulas, proof length, depth, and the number of nodes. The paper does not demonstrate that these constraints encompass the entire space of valid, permissible proofs.
Questions for authors
  1. If all experimental conditions were evaluated using the same number of repetitions and a consistent aggregation method, would autonomous exploration still outperform the "fixed-probe" baseline?
  2. What is the scope of the search used to identify "unprovable" problems, and how is it guaranteed that all valid proofs are covered? Furthermore, did any instances of UNPROVABLE_PASSED_VERIFIER occur during the actual experiments?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 136r (1/2)

We sincerely thank you for your valuable comments and for your recognition of our evaluation design and process analysis! For your questions, we ran 43 new fixed-probe runs and 82 paired replays so that every condition has the same number of repetitions, certified the 23 unprovable theorems (9 by a complete argument, 14 by exhaustive search), and re-checked all 10,124 proofs submitted for them. We address each point below; all new experiments and changes are included in the revised paper.

Weakness 1 & Question 1: The primary baseline for autonomous exploration is derived from the best-performing trajectory among three attempts, whereas the "hindsight," "fixed-probe," and "no-tool" baselines are typically evaluated based on a single run. Simply re-running the same probes used in the best trajectory—while potentially aligning with the environmental evidence—fails to account for the selection bias inherent in choosing the best trajectory based on final performance (which may have conflated successful reasoning, effective knowledge integration, and accurate answer generation). If all experimental conditions were evaluated using the same number of repetitions and a consistent aggregation method, would autonomous exploration still outperform the "fixed-probe" baseline?

Response: Thank you for this valuable comment! You are right that comparing only the best trajectory with single control runs introduces selection bias. We therefore ran two sets of experiments in which every condition has the same number of repetitions and the same aggregation.

(1) Fixed-probe and without-tool controls at equal repetitions and aggregation. We extended fixed-probe exploration to three runs per system and compared the mean of these three runs with the mean of the three autonomous trajectories:

SandboxSystems with autonomous higherMedian differenceWilcoxon pWorst autonomous trajectory above best fixed-probe run
AlienCode10/10+31.7 points0.0025/10
AlienLogic10/10+16.0 points0.0029/10

The without-tool (no-tool) control shows the same:

  • Means: the mean of the three autonomous trajectories is higher for 10/10 systems in both sandboxes, by a median of 40.5 and 23.1 points.
  • Worst trajectory: the worst autonomous trajectory beats without-tool answering for 7/10 systems in AlienCode and 10/10 in AlienLogic.

So the answer is yes: with equal repetitions and the same aggregation, autonomous exploration still clearly outperforms both fixed probes and without-tool answering, and this conclusion of the paper is unchanged.

(2) Hindsight control: now a paired comparison. Each of the first five autonomous trajectories of every system is replayed once with its own probes and compared pair by pair, with neither side selected. The two do not differ detectably:

  • AlienCode: 49 pairs, mean difference −2.2 points (95% CI −9.8 to +5.3).
  • AlienLogic: 50 pairs, mean difference 0.0 points (95% CI −2.1 to +1.9).

Accordingly, we revised the hindsight claim of Finding 2: the gain of exploration comes from the evidence carried by the experiments the system designs itself, not from whether it decides each one on the fly (see our response to Reviewer VuUJ). These experiments are in Appendices G.1 and G.2 (Tables 4 and 5) of the revision.

AuthorsOpenReview ↗

Response to Reviewer 136r (2/2)

Weakness 2 & Question 2: In the formal logic world, the "unprovable" label is determined by a search constrained by a finite set of formulas, proof length, depth, and the number of nodes. The paper does not demonstrate that these constraints encompass the entire space of valid, permissible proofs. What is the scope of the search used to identify "unprovable" problems, and how is it guaranteed that all valid proofs are covered? Furthermore, did any instances of UNPROVABLE_PASSED_VERIFIER occur during the actual experiments?

Response: Thank you for this rigorous question! You are right that a bounded search alone cannot prove that no proof exists. We justify the 23 unprovable theorems at three levels, detailed in Appendix F.7 of the revision.

(1) Nine theorems have a complete argument that needs no search bound. For these nine, a patched rule makes the goal itself an illegal derived line, so no proof of any length can end in it.

(2) The other 14 have bounded exhaustive certificates.

  • Search space: the subformula closure of the premises and goal plus ⊥, with assumption depth at most 3; every step is checked by the actual verifier.
  • Result: within a budget of 60 million nodes, the search exhausts every proof up to length 8 or 9 for each of the 14 theorems and finds none.
  • Limits: the search does not generate OR_E steps. On the 41 provable theorems that need no OR_E, it recovers a proof of the reference length for 32 within the same budget; the other 9 need longer proofs than the budget reaches. These 14 certificates therefore hold within the stated bounds, not for proofs of any length.

(3) Adversarial evidence from the experiments. Across all our experiments, the ten systems answered these 23 theorems 20,797 times, and 10,124 of the answers submit a proof instead of declining. Re-checked one by one with the current verifier, none is accepted: most violate a patched side condition (9,008) or use a disabled axiom (534).

On UNPROVABLE_PASSED_VERIFIER: Thank you for asking. Yes, it occurred in the submitted version. Two theorems, H01 and H13, were labeled unprovable, but the verifier accepted valid proofs submitted by models, 1,348 times in total.

  • How it was caught: by design, the harness records an accepted proof of an unprovable theorem as DESIGN_BUG and never scores it as correct. This is how these two theorems were found.
  • How we corrected it: we relabeled the two theorems as provable and regraded all archived answers to them (3,281) with the same scoring rules, without calling any model again.
  • Effect on results: the number of unprovable theorems falls from 25 to 23. The AlienLogic Best@3 ranking is unchanged: Grok 4.6 drops from 83.8 to 81.0, and the other systems rise by 1.4 to 1.9 points. After the regrade, no such instance remains in the scored data.

The revised PDF uses the corrected numbers, and the certification and the audit are in Appendix F.7.

Thank you again for your rigorous review; your questions have helped us substantially strengthen the reliability of our experimental design. We have included all new experiments and changes in the revised paper. If you have any further questions, please let us know and we will do our best to respond.

Dear Reviewer 136r,

Thank you again for your review. As the discussion period closes on November 18 (AoE), we would like to check whether our responses address your two concerns:

  • Equal repetitions: with every condition run three times and averaged the same way, autonomous exploration still beats fixed probes for 10/10 systems in both sandboxes (median +31.7 and +16.0 points, p = 0.002).
  • Unprovable theorems: 9 of the 23 are certified by a complete argument and 14 by bounded exhaustive search, and none of the 10,124 proofs that systems submitted for them is accepted by the verifier.

If these address your concerns, we would be grateful if you would consider updating your evaluation. Otherwise, we would be happy to provide any further details before the discussion closes.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers for their careful reading and constructive suggestions. During the discussion period we ran 269 new evaluation runs under exactly the paper's protocol, covering all ten systems in both sandboxes. Each run is scored on 1,050 answers (70 held-out tasks, five milestones, three answers each), so the new runs add 282,450 graded answers. With them, every system now has ten autonomous trajectories per sandbox; each of the first five trajectories of every system is replayed with its own probes, so that every trajectory, not only the best one, is compared with a hindsight run on the same probes (99 pairs); and fixed-probe exploration is repeated three times per system, so that all conditions are compared at the same number of repetitions. We also conducted additional analyses to address the reviewers' questions: a failure attribution for the two keystone rules of AlienCode, a certification of the 23 unprovable theorems of AlienLogic, and a re-check of all 10,124 proofs that systems submitted for them.

The revised paper incorporates these results, with every change marked in blue. Finding 2 and Figure 4 now rest on the matched and paired comparisons, and Findings 5 and 7 add the failure attribution and the stability of rankings over ten trajectories. The appendix adds the new analyses (Appendices G.1–G.4, Tables 4–7), the certification of the unprovable theorems (Appendix F.7), a discussion of why the main ranking uses three attempts (Appendix C.1), and two case studies of high-scoring trajectories (Cases 5 and 6). The AlienLogic numbers are updated after two theorems, H01 and H13, were relabeled as provable, as we explain in our response to Reviewer 136r.

We reply to each reviewer's questions in detail under the corresponding review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.