- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ELEMENTARY is a generator-based interactive escape-room benchmark with four evidence regimes: association, observation, intervention, and counterfactual reasoning. Its central methodological feature is structural validation intended to ensure that the permitted evidence determines each question’s solution. Across 120 fixed questions, no evaluated model is consistently strong; the I3 diagnostic particularly shows high initial-control success but near-zero transfer success when correct action ordering must generalize.
Critical strengths
The benchmark design is strong and interesting - it validates solvability and attempts to exclude shortcuts under the formal mechanism spaces. The intervention-transfer setup is also interesting. The large gap between first-target and second-target success highlights a potentially important distinction between local control and reusable mechanism knowledge.
Critical weaknesses
- Each model/family result uses only 10 fixed instances and allows (if the reviewer understood correctly) one deterministic attempt per instance and there are no repeated generation seeds, confidence intervals, or statistical comparisons. The reported differences should therefore be treated as descriptive rather than robust capability estimates.
- There are no non-agentic/search, oracle-information, budget-relaxed, or simple rule-induction controls. Consequently, low transfer success could arise from exploration difficulty, hidden budget/interface constraints, or generator conventions rather than specifically from failure to recover causal structure.
Questions for authors
- Do conclusions persist across independently generated evaluation sets and repeated model runs, with uncertainty intervals?
- Can the paper add controls for search/planning ability, oracle evidence, and the Lock-2 budget/distractor design to isolate the proposed causal interpretation?
6 comments
W1.2 Repeated-run Evaluation
We sampled two original questions per family and independently ran each model–question pair three times, using a fresh environment and interaction history for each run. We evaluated three representative models spanning different performance levels in the original evaluation: Gemini-3.8-Flash (44.2%, highest), GPT-5.6-Sol (34.2%, intermediate), and DeepSeek-V4.1-Flash (10.8% for the September checkpoint, lowest). The repeated runs used its October 17 update, DeepSeek-V4.1-Flash-1017. This yielded 216 episodes.
Table 3 reports the repeated-run results and uncertainty estimates.
Table 3. Repeated-run Stability
24 stratified questions · 3 models · 3 independent runs · 216 episodes.
| Model | Run 1 | Run 2 | Run 3 | Mean SR ± SD (pp) | 95% CI | OIR |
|---|---|---|---|---|---|---|
| Gemini-3.8-Flash | 4/24 | 10/24 | 11/24 | 34.72 ± 15.77 | [29.17, 40.28] | 41.67% |
| GPT-5.6-Sol | 5/24 | 7/24 | 8/24 | 27.78 ± 6.36 | [22.22, 33.33] | 16.67% |
| DeepSeek-V4.1-Flash-1017 | 10/24 | 10/24 | 7/24 | 37.50 ± 7.22 | [29.17, 45.83] | 20.83% |
SD is computed across the three run-level success rates, in percentage points. The 95% CIs use 20,000 family-stratified question-cluster bootstrap resamples, retaining all three runs for each sampled question. Outcome Inconsistency Rate (OIR) is the percentage of questions with both successful and failed outcomes across the three runs.
On these questions, O2 succeeded in all 18 episodes, while O3 failed in all 18. This pattern held across all nine model–run combinations. All 18 I3 episodes reached Lock 1, but only one reached Lock 2.
Together, these evaluations show that the main family-level contrasts persist across independent questions and repeated runs, although overall success rates vary across runs.
Manuscript Changes. Section 5.2 now summarizes the O2–O3 results on independent questions and repeated runs. Section 5.1 points to Appendix K (Tables 17–18) for the additional evaluations and Appendix N (Table 22) for the complete pooled family-level results.
W2 / Q2. Causes of I3 Transfer Failure
We added controls to distinguish task feasibility, exploration difficulty, mechanism information, action ordering, and budget sensitivity. A generator-aware active-exploration agent solved all ten original I3 questions within the standard budgets, while additional LLM controls showed substantial effects from mechanism information and adjustment limits.
W2.1 Active Exploration and Ordering Baselines
Generator-aware active exploration. We evaluated a rule-based agent with access to the generator's structural rules, but not the instance-specific hidden graph or channel settings. It enumerated candidate Relay graphs and channel settings, read the operation log, and performed diagnostic interventions by testing pairs of Relay levels with the same remainder modulo four. After each intervention, it eliminated mechanisms inconsistent with the observed changes.
On all ten original I3 questions, it identified the hidden mechanism and completed Lock 1 within 31–43 environment steps (under the 50-step horizon). It then used the inferred graph to set Relays in upstream-to-downstream order, reaching the Lock 2 target in exactly five adjustments on every question (10/10).
Search, ordering rules, and distractors. In I3, Lock 2 specifies a target value for each of the five Relays. All five are relevant, so the challenge is not to select relevant Relays from a larger set of distractors, but to determine an effective action order. Setting an upstream Relay can overwrite values already established downstream.
We evaluated random search and simple ordering controls, summarized in Table 4.
Table 4. Random and Rule-based I3 Controls
| Strategy | Evaluation Setting | Success |
|---|---|---|
| Random legal Relay values and random action order | 10,000 sampled plans per question across 10 questions; no Lock 2 targets or hidden mechanism | 0/100,000 |
| All 120 target-setting action orders | 120 permutations per question across 10 questions; five target values fixed | 29/1,200 (2.42%) |
| Ascending Relay ID order | 70 original Lock 2 starting states | 0/70 |
| Descending Relay ID order | 70 original Lock 2 starting states | 0/70 |
| Observed-ancestor ordering | 70 original Lock 2 starting states; dependencies inferred from visible Lock 1 propagation | 2/70 |
The three ordering rules were evaluated on the 70 original Lock 2 starting states, preserved from the end of Lock 1. Observed-ancestor ordering placed Relay i before Relay j whenever changing i visibly changed j during Lock 1, breaking ties by Relay ID.
Each of the ten questions had between one and four successful orders among its 120 permutations (Appendix C.5). Thus, every original question admits a solution within the five-adjustment budget, although successful orders are sparse.
We thank the reviewer for the constructive feedback. We have expanded the evaluation with independently generated questions and repeated runs, and added controls for active exploration, mechanism information, action ordering, and budget sensitivity to better characterize I3 transfer failures.
W1 / Q1. Evaluation Size and Variability
We evaluated whether the main findings persist beyond the original fixed questions and single runs. Both the independent-question evaluation and repeated runs reproduced the O2–O3 performance gap and low I3 transfer success.
W1.1 Independent New-question Evaluation
The original evaluation used 120 validated questions (ten per family), shared across all models. We generated and validated another 120 questions, again with ten per family, and evaluated all seven models, yielding 810 episodes.
All seven models solved more O2 than O3 questions. A1 also matched or outperformed A3 for all seven models (five higher, two tied). In I3, all 70 episodes reached Lock 1, but only one reached Lock 2 (1/70). Table 1 reports the per-model results.
Table 1. Independent New-question Evaluation
120 new questions · 7 models · 810 episodes. Overall SR reports successes over evaluated episodes; family-level results are success counts out of 10.
Claude-Opus-5 was evaluated on the nine Association, Observation, and Intervention families (90 questions), following the original evaluation scope. I3 Lock 2 success is conditioned on Lock 1 completion.
DeepSeek model update. The original benchmark comparison used the September checkpoint of DeepSeek-V4.1-Flash. For the additional evaluations, we used its October 17 update, DeepSeek-V4.1-Flash-1017, to rerun both the original and new question sets. It achieved 43/120 (35.83%) on the original questions and 39/120 (32.50%) on the new questions. We retain the original September results in the main benchmark comparison.
Pooled family-level results. Combining the original and independent sets provides 20 questions per family for model–family combinations with complete coverage, reducing success-rate granularity from 10 to 5 percentage points. Table 2 reports the pooled results for the A1/A3 and O2/O3 comparisons. Both contrasts persist for all seven models.
Table 2. Pooled Results on Key Family-level Contrasts
Each cell counts successes out of 20. DeepSeek uses its October 17 update on both question sets.
The complete twelve-family pooled results are provided in Appendix N (Table 22).