acceptodds
Under review as a conference paper at ICLR 2027

Witness: Discovery, Deciphering and Epiphany in Interactive Puzzle Environments

Abstract

Automated science needs agents that can work out the rules of an unfamiliar environment by interacting with it. Interactive rule-discovery puzzles offer a controlled setting for studying this ability: an agent infers hidden rules through experimentation and uses what it has inferred to reach a stated goal. We ask what limits current language models on these puzzles and whether reinforcement learning (RL) improves performance on rules held out from training. To study both, we introduce WITNESS, a 2D grid-based puzzle environment with ground-truth ASCII observations and controlled access to rules. An agentic pipeline generates games for WitnessGym, the RL training suite, and WitnessBench, comprising public validation and private test games. The validation set separately tests new compositions of trained rule primitives and primitives absent from training. Under a shared harness, the best of 18 frontier proprietary and open-weight models solves only 24% of private test level slots, with scores sensitive to the observation interface and agent configuration. Providing ground-truth rules raises Opus-5's validation RHAE-L5 (relative human action efficiency over the first five levels) from 59.9 to 97.8, whereas a 27B open-weight model gains only 2.1 points and remains limited even with the rules provided. RL on WitnessGym raises the 27B model's private test RHAE-L5 from 2.1 to 5.4 and yields a mean gain of 4.1 points on four external discovery benchmarks. Together, these results point to rule acquisition as a major difficulty for frontier models like Opus-5 while smaller models further struggle on rule-based execution, and indicate that RL on hidden-rule puzzles transfers to broader rules and real-world tasks beyond training. Benchmark, training split, and scripts are provided in supplementary materials.

Primary area:
datasets and benchmarks
Keywords:
benchmark, rule discovery, RL environments, interactive rule discovery, RL training gym, game environment

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2esROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WITNESS, an interactive 2D grid-based environment for studying hidden-rule discovery in language models. The framework contains WitnessBench for evaluation, WitnessGym for RL training, and WitnessForge for generating and validating new games. The authors evaluate 18 proprietary and open-weight models under a shared interaction setting, and also use rule-access and observation-interface experiments to study whether the main difficulty comes from rule discovery or rule execution. The paper further trains Qwen3.8-27B with RL and reports improvements on unseen rules, private test games, and several external reasoning benchmarks.

Critical strengths
  1. The benchmark provides a clean and controlled setting for studying interactive rule discovery.
  2. The split between held-out compositions and held-out rule primitives is useful for evaluating different types of generalization.
  3. The paper evaluates many frontier and open-weight models under a shared interaction framework.
  4. It is also interesting that the paper goes beyond benchmarking and studies whether RL training can transfer to unseen rules and external tasks.
Critical weaknesses
  1. WITNESS is still a quite abstract 2D ASCII environment, so its connection to real-world scientific discovery is somewhat indirect.
  2. The positive RL results are mainly demonstrated on Qwen3.8-27B, while Qwen3.5-9B does not improve under the same setting, so the generality of the RL conclusion is still unclear.
  3. The number of independent games and rules is relatively limited, although each game contains multiple levels and seeds.
  4. The benchmark score is quite sensitive to the observation format and agent configuration, so it may partly measure the whole agent system rather than only the reasoning ability of the base model.
  5. The average transfer gain on external benchmarks is positive, but the improvement is not equally convincing on every individual benchmark.
Questions for authors

See Weakness

5 comments
AuthorsOpenReview ↗

Response to Reviewer 2esR (1/2)

Thank you for the detailed review. Our answers are below; all changes in the revised PDF are highlighted in blue.

W1. The environment is abstract and its connection to real-world scientific discovery is indirect.

We see the connection to scientific discovery in the interaction loop that WITNESS exercises rather than in the domain itself: an agent proposes a hypothesis about hidden rules, picks an action to test it, and revises the hypothesis from the outcome. Whether what is learned in this loop carries over to other domains is what the external transfer experiment tests. The model trained on WITNESS improves on PhysGym (discovering a physical law through self-designed experiments, +3.4), WILT (choosing tests that eliminate competing hypotheses, +2.3), Failing-to-Falsify (seeking evidence against one's own hypothesis, +6.1), and CipherBank (inferring an unseen encoding, +4.8), and the gain holds for a second training seed (+3.8, 95% CI [+1.7, +5.8]). Following Reviewer Yngb's suggestion, the abstract now describes this as transfer to external benchmarks of rule inference and hypothesis testing. To summarize, the connection to scientific discovery runs through the hypothesis-testing loop, and the external transfer results are the evidence for it.

W2. The positive RL results are mainly on Qwen3.8-27B; Qwen3.5-9B does not improve.

In RL, whether a task is learnable depends on the current policy: tasks that a model solves never or always provide no learning signal, and which tasks fall in between shifts as the same model trains [1, 2, 3, 4, 5]. The 9B result is one we chose to include rather than leave out, and a new experiment now helps explain it. Qwen3.8 has no 9B release, so we took the fine-tuned Qwen3.5-9B (on data distilled from Qwen3.8) and trained it with the same RL recipe. This model (Qwen3.8-9B-Distill, new Figure 9) is more learnable from the same RL task recipe than Qwen3.5-9B: maximum gains of +5.29, +0.85 for validation RHAE-L5 and test RHAE-L5, respectively. Whether a model learns from these games therefore depends on its base training recipe as well as its size, and the caption of Figure 6 now describes the comparison in these terms. How far the RL result extends across model families is a question we flag in the revised Conclusion.

[1] Minqi Jiang, Edward Grefenstette, Tim Rocktäschel. Prioritized Level Replay. ICML 2021. arXiv:2010.03934.

[2] Tambet Matiisen, Avital Oliver, Taco Cohen, John Schulman. Teacher-Student Curriculum Learning. IEEE Transactions on Neural Networks and Learning Systems, 2020. arXiv:1707.00183.

[3] Thomas Foster, Anya Sims, Johannes Forkel, Mattie Fellows, Jakob Foerster. Learning to Reason at the Frontier of Learnability. arXiv:2502.12272, 2025.

[4] Qiying Yu et al. DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476, 2025.

[5] Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, Jieyu Zhao. Efficient Reinforcement Finetuning via Adaptive Curriculum Learning. arXiv:2504.05520, 2025.

AuthorsOpenReview ↗

Response to Reviewer 2esR (2/2)

W3. The number of independent games and rules is relatively limited.

At its current size the benchmark already spreads the 18 models over a wide range of scores, and since submission we have added more private games to check that this holds as the set grows. The submitted version has 47 games (22 training, 8 validation, 17 private test) with 760 verified levels, and the generation pipeline (Section 2.3) is built to extend it. This is within the range of comparable game benchmarks and training suites, which report 70 games for DiG-Bench (21 public) [6], 6 games for lmgame-Bench [7], and 3 training games for SPIRAL [8], and each WITNESS game adds a multi-level curriculum, giving 760 verified levels and 1,700 level slots per model.

Since submission we added 9 private test games, enlarging the private set from 17 to 26, and evaluated 11 models (listed in the new Table 14) on all 26 games in a single set of runs with an updated version of the canonical harness (differences listed in the caption of Table 14; model settings in Table 16). Both rankings below come from these same runs, so the comparison is between the 17-game and 26-game sets rather than between harness versions. The ranking on the 26-game set matches the ranking on the 17-game set (Spearman ρ\rho = 1.00 for OAE-L20 and 0.99 for RHAE-L5); per-model scores on the new games are in the new Table 14. On rules versus levels, Table 6 shows that gains on trained layouts carry over to unseen layouts of the same rules, which is why the benchmark invests in distinct rules rather than in more levels per rule. The upshot is that the benchmark already discriminates between models, and the enlarged private set confirms the rankings.

[6] Ruairidh M. Battleday et al. DiG-Bench: Discovery in Games. arXiv:2608.12593, 2026.

[7] Lanxiang Hu et al. lmgame-Bench: How Good Are LLMs at Playing Games? ICLR 2026. arXiv:2505.15146.

[8] Bo Liu et al. SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning. ICLR 2026. arXiv:2506.24119.

W4. Scores are sensitive to observation format and agent configuration.

This is a point we share, and Section 3.2 is where the paper measures it: a benchmark score reflects the model together with its harness, and we wanted to know by how much. Replacing the ASCII board with the raw grid changes scores by different amounts across models and reverses the order of Opus-4.8 and Kimi-K3, and adding code execution (Prime-Agent) changes scores further. That is why every model comparison in the paper is made under one canonical harness, so the harness is held fixed across the 18 models, and why Section 3.2 quantifies the dependence. Section 3.2 now states this reading explicitly: the benchmark score is a score of the model under the canonical harness, not a harness-free measure of the base model. In short, the benchmark gives a controlled comparison across models under one harness, and the harness dependence is measured and reported.

W5. The external transfer gain is uneven across benchmarks.

We expected the gain to vary across benchmarks, which is why the paper reports a confidence interval for each one rather than only the mean. The four benchmarks were chosen to probe different aspects of discovery: inferring an unseen encoding from examples (CipherBank), discovering a physical law through self-designed experiments (PhysGym), choosing tests that eliminate competing hypotheses (WILT), and seeking tests that could falsify one's own hypothesis (Failing-to-Falsify). At step 20, all four point estimates are positive for both training seeds, and the mean gain is +4.1 (95% CI [+2.1, +6.2]) for the first seed and +3.8 (95% CI [+1.7, +5.8]) for the second. The intervals differ in width because the benchmarks differ in size and variance (786 items for CipherBank against 80 instances for Failing-to-Falsify), and we read them as showing which aspects transfer more clearly rather than as a uniform effect. Overall, the direction of transfer is consistent across benchmarks and seeds; its size differs by benchmark, and the paper reports it that way.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 2esR,

Thank you again for your review. We would be grateful to know whether our response and the revised PDF address your concerns, in particular, the 9B-model result in Figure 9 and the external-benchmark transfer in Appendix B.4. We are happy to clarify anything before the discussion closes.

Reviewer_YngbOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WITNESS provides interactive grid puzzles with hidden rules, controlled observations and rule access, and rule-based training splits to study model failures and transfer from game RL.

Critical strengths
  1. The environment supports diagnosis beyond a leaderboard score. ASCII observations and ground-truth rule cards help separate state interpretation from rule acquisition. The generation pipeline checks rule necessity, implementation agreement, and solution replay. In Table 3, the matched generic-text control makes Opus-5's increase from 63.6 to 97.8 with rule cards persuasive evidence that rule acquisition is a major difficulty on these games.

  2. The training study reports both gains and limitations. The 27B model's private-test RHAE-L5 rises from 2.1 to 5.4, and the mean improvement across four external evaluations is 4.1 points, with a paired interval of [2.1, 6.2]. The paper also reports no improvement for 9B, no advantage of credit-C over broadcast, and substantial sensitivity to the agent harness.

Critical weaknesses

The environment is a useful contribution. Two claims need clearer support.

  1. The rule-level split is difficult to audit from the paper. It distinguishes primitives from compositions but does not provide the full relationships among training, validation, and private-test rules. A new predicate can still combine familiar primitives; Medallion Mirage, for example, includes reusable color-region structure. A rule inventory should identify what is newly composed and what introduces an untrained primitive. This can be made available to reviewers without exposing the private set publicly.

  2. The RL results do not isolate improvement in rule discovery. Comparing the base model with game RL leaves open gains in execution, output discipline, or planning. Table 5 shows that RL also greatly improves performance when the true rules are supplied. External transfer supports the usefulness of training, but does not establish that hidden-rule training develops discovery skills more effectively than training with known rules. The abstract's extension to real-world tasks should be correspondingly cautious.

Questions for authors

Can you provide a primitive-overlap matrix for the training and evaluation games? A matched training condition with ground-truth rules would help distinguish discovery-specific gains from better execution. Does external transfer hold for a second independent training seed? Please also explain why the external evaluation uses step 20 while the suite's selected broadcast checkpoint is step 10.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Yngb (1/3)

Thank you for the positive assessment and for the specific requests. We added the primitive-overlap analysis, evaluated the second training seed on the external benchmarks, and revised how the external-transfer checkpoint is reported. We also conducted the proposed experiment (matched training condition with ground-truth rules) to address discovery-specific gains. All changes in the revised PDF are highlighted in blue.

W1 and Q1. Can you provide a primitive-overlap matrix for the training and evaluation games?

Yes, and the tables are now in the paper: we added game-by-primitive tables (new Tables 10, 12, 13) in the new Appendix B.5, and the new Table 10 shows that every private test game introduces at least one primitive, predicate or mechanism that does not appear in any training game. For each game the table lists the number of win-condition and perception primitives, how many of them appear in the training set, and how many are new. For training and validation games, their rules are given in supplementary materials, and the full matrix is given with primitive names in Table 13 (primitive definitions in Table 12); for the private test games the same counts are given under anonymized game IDs, because the private set is refreshed periodically and its rules are withheld to prevent contamination.

The Medallion Mirage example touches on two different senses of "composition", and Appendix B.5 now separates the two senses (Section 2.2 points to it). The paper's "held-out compositions" are conjunctions of independent predicates: a valid path has to satisfy every predicate at once. A new predicate built from familiar primitives is a different case. Medallion Mirage reuses the region and color structures that the engine provides to every game, but its semantics are new (a medallion cell reflects every colored square through itself onto an effective cell, and regions are judged at the effective cells), so a path that is valid under it can violate the classic color-region rule. We count such a game as introducing a new predicate. Table 12 lists the primitives, the components that rules are built from, each reading one aspect of the board (regions, colors, nodes, or edges); they are distinct from the rule predicates themselves, whose full statements for the training and validation games are in the supplementary material. Table 13 shows game by game which primitives each rule uses. To summarize, the split is now documented game by game, and the composition/primitive distinction is defined at the level of rule predicates rather than board structures.

If we look at unseen primitives instead of unseen predicates, the new Table 11 shows that RL models also beat more levels than the base model in test games that introduce untrained primitives.

AuthorsOpenReview ↗

Response to Reviewer Yngb (2/3)

W2. The RL results do not isolate improvement in rule discovery.

What Section 4 shows is transfer: after RL, the model does better on held-out compositions, held-out primitives, the private test set, and four external benchmarks. We do not attribute this gain to rule discovery alone, and Table 5 supports the same reading: RL also raises performance when the rules are supplied, so part of the gain is in execution, output discipline, and planning. Section 4 now says this directly: "Table 5 in Appendix B.1 shows that the RL checkpoints also score higher when the ground-truth rules are supplied, so the gains include better execution under known rules and are not specific to rule discovery." The closing sentence of the abstract now reads: "Together, these results point to rule acquisition as a major difficulty for frontier models like Opus-5 while smaller models further struggle on rule-based execution, and indicate that RL on hidden-rule puzzles improves performance on rules held out from training and on four external benchmarks of rule inference and hypothesis testing."

We ran the matched control the review proposed. We trained Qwen3.8-27B with the same recipe and seeds, adding each training game’s ground-truth rule card to its system prompt; validation and test are run without cards. We added a new Figure 10 to compare the training runs and eval results. We also checked the training trajectories rather than assuming what the cards remove. With cards, the policy states far fewer hypotheses about the win condition (about 1 per episode vs. 7-11 without cards), but it still runs the rest of the interactive loop: it probes actions, writes about 40 and retracts about 23 rules per episode while grounding the given rules in each level, and tests and discards candidate solutions.

This control retains most of the held-out gain: validation RHAE-L5 is within noise of standard training at every matched iteration, and test RHAE-L5 at iteration 10 is 4.9 vs. 5.4 (base 2.1). Evaluated without cards, it also proposes win-condition hypotheses almost as often as the standard policy (8.6 vs. 8.9-11.5 per test episode; base 5.3).

We draw three conclusions. (1) Most of the held-out gain does not depend on practicing rule inference on the training games; the part that does is at most the remaining 0.6 RHAE-L5, within one standard error. (2) What transfers is the interactive procedure that both conditions practice: probing, stating and testing hypotheses about the current level, and executing plans precisely. We see this as part of discovery in an interactive setting rather than mere execution: a test is informative only if the plan is executed correctly, and the habit of hypothesizing carries over to unseen games. (3) The inductive step itself is where the open problem lies. The rule-card probe (Appendix B.1) shows that the RL policy still gains far more from being given the rules (15.9 to 47.7) than the base model (8.0 to 10.1): for a 27B model, acting on rules is itself a bottleneck, which frontier models have largely removed (about 98 with cards), so it is natural that RL addresses it first. Rule induction remains the main capability that the benchmark measures, and whether RL can improve it is the question that WITNESS is built to keep asking. We have added the control and this distinction to Figure 10 and Appendix B.1.

AuthorsOpenReview ↗

Response to Reviewer Yngb (3/3)

Q2. Does external transfer hold for a second independent training seed?

Yes. We evaluated the step-20 checkpoint of the second broadcast training seed on the four external benchmarks with the same protocol as the first (thinking on, 32k cap, same number of repetitions). The mean gain over the base model is +3.8 points (95% CI [+1.7, +5.8]), against +4.1 (95% CI [+2.1, +6.2]) for the first seed, and all four per-benchmark point estimates are positive for both seeds. The second seed is added to the new Table 8. In short, the external transfer is reproduced on an independent training run.

Q3. Why step 20 for external evaluation while the suite selects step 10?

The two criteria happen to favor different checkpoints, and the revised text now reports both so that readers can see each. On the WITNESS validation set, the best broadcast checkpoint is step 10 and the best credit-C checkpoint is step 20. On the external benchmarks, the gain grows with training step for both methods (Table 7): for broadcast, +1.6 (95% CI [-0.3, +3.6]) at step 10 and +4.1 (95% CI [+2.1, +6.2]) at step 20; for credit-C, +1.6 (95% CI [-0.6, +3.8]) at step 10 and +2.7 (95% CI [+0.6, +4.9]) at step 20. The submission reported step 20 in Section 4 as the stronger external result, and the revision adds the step-10 result alongside it.

Appendix B.4 and Tables 7 and 8 now give both numbers: at the validation-selected step 10 the external gain is +1.6 (95% CI [-0.3, +3.6]), and at step 20 it is +4.1 (95% CI [+2.1, +6.2]), so we describe the external gain as increasing with training step rather than as a single selected checkpoint. On the WITNESS private test set, the broadcast step-20 checkpoint scores 6.0, compared with 5.4 at step 10; the in-suite performance holds. Overall, the external gain grows with training step, and both checkpoints are now in Appendix B.4 and Tables 7 and 8.

Reviewer_LBXzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WITNESS is an interactive grid-puzzle environment where models must discover hidden rules through experimentation and feedback, then plan paths to solve levels. The paper provides a game generator, training and evaluation sets, and experiments that reveal the true rules to separate rule discovery from solving difficulties. RL training on 22 games improves Qwen3.8-27B’s private-test action-efficiency score from 2.1 to 5.4 and completed levels from 67/1700 to 122/1700. It also yields an average gain of 4.1 percentage points across four external benchmarks.

Critical strengths
  1. The evaluation tests generalization beyond training games. It includes new combinations of familiar rules and previously unseen rules, allowing the authors to assess whether learned capabilities transfer to new tasks. The generator also checks solvability and filters levels that simple strategies can solve.
  2. The ground-truth-rule comparison provides a useful diagnosis. Giving Opus-5 the rules raises its validation score from 59.9 to 97.8, whereas the base 27B model improves only from 8.0 to 10.1. This reveals different bottlenecks: some models mainly struggle to discover the rules, while others still struggle to solve puzzles after receiving them.
Critical weaknesses
  1. The benefit of RL beyond simply enabling thinking remains unclear. Training and the main RL comparisons disable thinking, with the trained model scoring 5.4. However, the base model already scores 6.4 under its highest reasoning setting. Different inference budgets prevent a direct comparison, but the current results establish improvement with thinking disabled without showing whether RL further improves performance with thinking enabled.
  2. Model size is confounded with model version. The 9B model is Qwen3.5, while the 27B model is Qwen3.8. Their different responses to RL could therefore reflect model version rather than parameter count. This comparison does not establish that complex discovery capabilities emerge through model scaling.
Questions for authors

With thinking enabled under the same inference settings, does the existing RL-trained checkpoint still outperform the base model on WITNESS? This would clarify whether RL improves performance beyond the gains already available from enabling thinking.

4 comments
AuthorsOpenReview ↗

Response to Reviewer LBXz

Thank you for the review and for the precise question about thinking. We ran the requested thinking-on experiment and added a second experiment on the model-version question. All changes in the revised PDF are highlighted in blue.

Q1 and W1. Does RL help beyond enabling thinking?

Yes. When both models run with thinking on under the same inference settings, the RL-trained model still comes out ahead: 7.6 versus 6.1 RHAE-L5 on the private test set, so RL adds to what thinking provides instead of replacing it.

The experiment is as follows. We trained Qwen3.8-27B with the same broadcast RL recipe as in the paper, but with thinking on (effort = low), and evaluated both the base model and the trained model on the 17 private test games with 5 seeds, thinking on, effort = low. Training with thinking on takes 10.3 hours per step, against 4 hours with thinking off, so within the discussion period we trained to step 10, the same step we selected for the thinking-off run. The base model scores 6.1 and the thinking-on checkpoint 7.6. For reference, the thinking-off checkpoint scores 5.4 with thinking off (the number in the paper) and 4.3 when thinking is switched on at inference without retraining. The two numbers come from the same checkpoint and differ only in the inference mode: with thinking on it still beats a few more levels than with thinking off (119 versus 113 of 1,700), but it takes more actions to do so, and RHAE-L5, which rewards action efficiency, drops from 5.4 to 4.3. The model trained with thinking off learned to put its reasoning in the <meta> field; turning thinking on at inference moves part of that reasoning elsewhere and lengthens the action sequence, which is why the fair comparison for thinking-on inference is a checkpoint trained with thinking on. So thinking alone (6.1) and RL alone (5.4) land at a similar level, and RL on top of thinking (7.6) goes higher.

One clarification: the 6.1 differs from the 6.4 in Table 1 only in effort (low versus maximum); both models in this comparison use effort = low.

In short, RL on WitnessGym improves the model beyond what enabling thinking already gives. The new result is in the new Table 8, and Section 4 now notes that thinking-on training was run at effort = low.

W2. Model size is confounded with model version.

Figure 6 compares the two models available to us, which differ in version as well as in size; to look at the two factors separately, we added an experiment that varies the recipe while holding the size fixed. Qwen3.8 has no official 9B release, so we found a public 9B model (called Qwen3.8-9B-Distill) with the Qwen3.8 recipe: it is a fine-tuned Qwen3.5-9B on 70k traces of SFT data distilled from a Qwen3.8 teacher. We trained it with exactly the same RL recipe. This model (Qwen3.8-9B-Distill, new Figure 9) is more learnable from the same RL task recipe than Qwen3.5-9B: maximum gains of +5.29, +0.85 for validation RHAE-L5 and test RHAE-L5, respectively. What separates Qwen3.5-9B from Qwen3.8-27B is how learnable each base model is, and size is only one of the factors behind that.

The revised caption of Figure 6 now states: "The two models differ in both size and version, so the difference in learnability is not attributed to scale; see Figure 9 for a 9B model with a different data recipe." Overall, whether a model learns from these games depends on its base recipe as well as its size, and Figures 6 and 9 now present the comparison in these terms.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer LBXz,

Thank you again for your review. We would appreciate your confirmation on whether our response and the revised PDF address your concerns; in particular, whether the thinking-on comparison in Table 8 (RL with thinking on improves private test RHAE-L5 from 6.1 to 7.6 over the thinking-on base model) resolves your question regarding whether RL provides gains beyond simply enabling thinking, and whether our discussion of model size versus version (new Figure 9) addresses the confound you raised. We are happy to clarify anything before the discussion closes.

AuthorsOpenReview ↗

Thank-you note

Dear Reviewer LBXz,

As the discussion closes today, we would like to thank you again for your review, especially for raising the questions about thinking and about model size versus model version, which made our comparisons more careful.

We know this is a busy time. Should any point in our response still seem unclear or insufficient, we would be happy to clarify it before the discussion ends.

I appreciate the authors’ detailed response and the newly provided experimental results. These additions satisfactorily resolve my primary concerns.

Comments on the paper

3 comments
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful reading. We are glad that Reviewer Yngb finds the environment "a useful contribution" and the rule-card result "persuasive evidence that rule acquisition is a major difficulty", that Reviewer LBXz finds the ground-truth-rule comparison "a useful diagnosis" that "reveals different bottlenecks", and that Reviewer 2esR finds the benchmark "a clean and controlled setting" and the composition/primitive split "useful". During the discussion period we added six pieces of evidence addressing the main concerns and revised the text accordingly; all changes in the revised PDF are highlighted in blue.

  1. For Reviewer LBXz, we trained the 27B model with thinking on and evaluated it with thinking on under the same inference settings as the base model. Private test RHAE-L5 rises from 6.1 (base) to 7.6 (RL step 10). RL on WitnessGym therefore improves the model beyond what enabling thinking already provides. We added Table 8 in Appendix B.4.
  2. For Reviewers LBXz and 2esR, we separated model version from model size as far as the available models allow: Qwen3.8 has no official 9B release, so we found a public 9B model (called Qwen3.8-9B-Distill) with the Qwen3.8 recipe: it is a fine-tuned Qwen3.5-9B on 70k traces of SFT data distilled from a Qwen3.8 teacher. We trained Qwen3.8-9B-Distill with the same RL recipe. This model shows learnability while Qwen3.5-9B did not (new Figure 9), so the RL gains are not confined to a single 27B model, and whether a model learns from these games depends on its base recipe rather than on its size alone. The caption of Figure 6 now describes the 9B versus 27B comparison in these terms, with Figure 9 as the companion result.
  3. For Reviewer Yngb, game-by-primitive tables (new Tables 12, 13) for all training and validation games, and anonymized per-game counts for the private test games (new Table 10), are given in a new Appendix section B.5. The new Table 9 confirms that every private test game introduces at least one new element (an untrained primitive, a new predicate, or an untrained feedback mechanism) absent from training, so the transfer results in Section 4 are measured on rules held out from RL training. The new Table 11 shows that RL models also beat more levels than the base model in test games that introduce untrained primitives.
  4. For Reviewer Yngb, the external transfer gain holds for the second independent training seed (+3.8, 95% CI [+1.7, +5.8], against +4.1 for the first), with all four per-benchmark point estimates positive for both seeds, so the transfer is reproducible across training runs. The new Table 8 and revised text report external transfer at both the validation-selected checkpoint (step 10) and step 20.
  5. For Reviewer 2esR, we enlarged the private test set from 17 to 26 games; model rankings on the enlarged set are reported in new Table 14 and match those on the 17-game set, so the benchmark's separation of models is stable under new games and the generation pipeline extends the private set as intended.
  6. For Reviewer Yngb, we ran the proposed matched control: RL with ground-truth rule cards in the training prompt, evaluated without cards (new Figure 10). It retains most of the held-out gain (test RHAE-L5 4.9 versus 5.4, base 2.1), which shows that what transfers is the interactive procedure practiced under both conditions, and the revised Appendix B.1 now states this.
AuthorsOpenReview ↗

Update: revised PDF uploaded (Figure 10 completed, with corrections)

We have uploaded a new revision. The only change since our first response is Figure 10, marked in blue. The ground-truth rule-card control suggested by Reviewer Yngb now extends to step 20, as the run continued (previously through step 10, with only one seed after step 6).

The completed runs require corrections to the following statements in our responses: (1) On validation, the two conditions are now within noise at most matched steps rather than at every one (mean difference 0.7 over steps 2 to 20). Half of the validation games (4 of 8) compose rules from training games. (2) Test RHAE-L5 at step 10 is 4.7 vs. 5.4, not 4.9 vs. 5.4 (base 2.1). The earlier 4.9 was seed 43 alone, compared against a two-seed mean. At step 20 it is 5.3 vs. 6.3. (3) The part of the test gain that depends on practicing rule induction is about 0.9 RHAE-L5, not “at most 0.6, within one standard error”: the gap is 0.77 at step 10 and 0.92 at step 20, ~1.2 eval standard errors in each case.

We also dense-tested every two steps to compare, and the GT-card-trained policy is lower at every matched test step from 2 to 20 (mean difference 0.9), so the gap is consistent (see new Figure 10). We interpret it as an upper bound on the contribution of rule-induction practice.

The direction of our conclusions is unchanged: a large proportion of the held-out gain comes from the interactive procedure that both conditions practice; a smaller yet consistent part of the gain depends on practicing rule induction on the training games, which we identify as the main open problem. All other numbers in our responses are unchanged.

AuthorsOpenReview ↗

Summary of concerns and revisions

We thank the reviewers and the area chair for their careful reading. To make the response easier to navigate for the area chair and all reviwers, we organize it and summarize each main concern together with the changes we made. All changes are marked in blue in the revised PDF. In the general response, we also complete Figure 10 and correct three numbers from our first response.

Reviewer LBXz

  • Does RL help beyond enabling thinking? (W1, Q1) We trained and evaluated with thinking on. Private test RHAE-L5 rises from 6.1 (base) to 7.6 (RL step 10) (new Table 8, Appendix B.4).
  • Model size is confounded with model version. (W2) A public 9B model built with the Qwen3.8 recipe (Qwen3.8-9B-Distill) learns under the same RL recipe, whereas Qwen3.5-9B does not (new Figure 9). Whether a model learns therefore depends on its base recipe, not on its size alone.

Reviewer Yngb

  • Primitive overlap between training and evaluation. (W1, Q1) New Appendix B.5 gives game-by-primitive tables for all training and validation games and per-game counts for the private test games (Tables 9 to 13). Every private test game introduces at least one element absent from training.
  • The RL results do not isolate rule discovery. (W2) We ran the proposed matched-control experiment: RL with ground-truth rule cards in the training prompt, evaluated without cards (new Figure 10, both seeds to step 20). It retains most of the test gain (4.7 vs. 5.4 at step 10; 5.3 vs. 6.3 at step 20; base 2.1). The remaining gap of ~0.9 is consistent across steps, and we read it as an upper bound on the contribution of practicing rule induction.
  • Second training seed. (Q2) External transfer holds for the second seed (+3.8, 95% CI [+1.7, +5.8], against +4.1 for the first), with all four per-benchmark point estimates positive for both seeds (Table 8).
  • Step 20 vs. step 10. (Q3) The revised text reports external transfer at both checkpoints (Table 7 and 8).

Reviewer 2esR

  • Its connection to real-world scientific discovery is indirect. (W1) The connection to scientific discovery runs through the hypothesis-testing loop, and the external transfer results (Table 7 and 8) are the evidence for it.
  • The 9B model does not improve. (W2) See new Figure 9: a 9B model with the newer recipe does improve.
  • Few games and rules. (W3) We enlarged the private test set from 17 to 26 games. Model rankings on the enlarged set match those on the 17-game set (new Table 14).
  • Sensitivity to harness. (W4) Section 3.2 measures this effect directly.
  • Uneven transfer across benchmarks. (W5) The paper reports a confidence interval for each benchmark, and all four point evals (new Table 8) are positive for both training seeds.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.