acceptodds
Under review as a conference paper at ICLR 2027

ELEMENTARY: SHERLOCKED by Causality — From Clues to Causes in Interactive LLM Agents

Abstract

Interactive LLM agents may succeed on the current task without recovering causal structure that remains useful when conditions or objectives change. We introduce ELEMENTARY, a generator-based escape-room benchmark that separates evidence sufficiency from agent reasoning failure by controlling how solution-relevant evidence becomes available within a shared interactive environment. The benchmark defines four evidence-conditioned reasoning regimes: Association exposes explicit relations; Observation provides sufficient passive evidence; Intervention requires active evidence acquisition; and Counterfactual requires reasoning from factual traces through historical revision and replay. Across twelve generator families, every generated question is validated so that the permitted evidence determines the required task-level solution. We evaluate seven proprietary and open-weight LLMs on 120 validated questions. No model is consistently strong across evidence conditions, and the highest average success rate is only %. More importantly, success on the current task can hide a failure to recover reusable causal structure. In I3 Causal Propagation, all model–question episodes reach the first control target, but only reaches the second target when successful control requires propagation-aware action ordering. Other diagnostics reveal additional failures in evidence search, passive inference, and historical revision. Together, these results show that current agents can often act on available clues without recovering the causal structure needed to succeed when the objective changes.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, causal reasoning, interactive reasoning, benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kcZHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ELEMENTARY is a generator-based interactive escape-room benchmark with four evidence regimes: association, observation, intervention, and counterfactual reasoning. Its central methodological feature is structural validation intended to ensure that the permitted evidence determines each question’s solution. Across 120 fixed questions, no evaluated model is consistently strong; the I3 diagnostic particularly shows high initial-control success but near-zero transfer success when correct action ordering must generalize.

Critical strengths

The benchmark design is strong and interesting - it validates solvability and attempts to exclude shortcuts under the formal mechanism spaces. The intervention-transfer setup is also interesting. The large gap between first-target and second-target success highlights a potentially important distinction between local control and reusable mechanism knowledge.

Critical weaknesses
  • Each model/family result uses only 10 fixed instances and allows (if the reviewer understood correctly) one deterministic attempt per instance and there are no repeated generation seeds, confidence intervals, or statistical comparisons. The reported differences should therefore be treated as descriptive rather than robust capability estimates.
  • There are no non-agentic/search, oracle-information, budget-relaxed, or simple rule-induction controls. Consequently, low transfer success could arise from exploration difficulty, hidden budget/interface constraints, or generator conventions rather than specifically from failure to recover causal structure.
Questions for authors
  1. Do conclusions persist across independently generated evaluation sets and repeated model runs, with uncertainty intervals?
  2. Can the paper add controls for search/planning ability, oracle evidence, and the Lock-2 budget/distractor design to isolate the proposed causal interpretation?
6 comments

We thank the reviewer for the constructive feedback. We have expanded the evaluation with independently generated questions and repeated runs, and added controls for active exploration, mechanism information, action ordering, and budget sensitivity to better characterize I3 transfer failures.

W1 / Q1. Evaluation Size and Variability

We evaluated whether the main findings persist beyond the original fixed questions and single runs. Both the independent-question evaluation and repeated runs reproduced the O2–O3 performance gap and low I3 transfer success.

W1.1 Independent New-question Evaluation

The original evaluation used 120 validated questions (ten per family), shared across all models. We generated and validated another 120 questions, again with ten per family, and evaluated all seven models, yielding 810 episodes.

All seven models solved more O2 than O3 questions. A1 also matched or outperformed A3 for all seven models (five higher, two tied). In I3, all 70 episodes reached Lock 1, but only one reached Lock 2 (1/70). Table 1 reports the per-model results.

Table 1. Independent New-question Evaluation

120 new questions · 7 models · 810 episodes. Overall SR reports successes over evaluated episodes; family-level results are success counts out of 10.

ModelOverall SRA1 / A3O2 / O3I3 Lock 1 → Lock 2
Gemini-3.8-Flash46/120 (38.33%)9 / 15 / 410 → 0
Grok-4.629/120 (24.17%)0 / 04 / 110 → 0
GPT-5.6-Sol54/120 (45.00%)9 / 37 / 210 → 0
DeepSeek-V4.1-Flash-101739/120 (32.50%)8 / 07 / 210 → 1
Kimi-K315/120 (12.50%)5 / 02 / 110 → 0
Claude-Opus-5*9/90 (10.00%)4 / 04 / 010 → 0
Qwen3.8-2.4T-A95B16/120 (13.33%)0 / 08 / 010 → 0
Overall—5 higher, 2 ties7/7 higher on O270/70 → 1/70

Claude-Opus-5 was evaluated on the nine Association, Observation, and Intervention families (90 questions), following the original evaluation scope. I3 Lock 2 success is conditioned on Lock 1 completion.

DeepSeek model update. The original benchmark comparison used the September checkpoint of DeepSeek-V4.1-Flash. For the additional evaluations, we used its October 17 update, DeepSeek-V4.1-Flash-1017, to rerun both the original and new question sets. It achieved 43/120 (35.83%) on the original questions and 39/120 (32.50%) on the new questions. We retain the original September results in the main benchmark comparison.

Pooled family-level results. Combining the original and independent sets provides 20 questions per family for model–family combinations with complete coverage, reducing success-rate granularity from 10 to 5 percentage points. Table 2 reports the pooled results for the A1/A3 and O2/O3 comparisons. Both contrasts persist for all seven models.

Table 2. Pooled Results on Key Family-level Contrasts

Each cell counts successes out of 20. DeepSeek uses its October 17 update on both question sets.

ModelA1A3O2O3
Gemini-3.8-Flash19/202/2014/206/20
Grok-4.62/200/2012/201/20
GPT-5.6-Sol18/203/2016/203/20
DeepSeek-V4.1-Flash-101718/200/2015/205/20
Kimi-K315/200/2010/202/20
Claude-Opus-58/200/206/202/20
Qwen3.8-2.4T-A95B5/200/2016/200/20

The complete twelve-family pooled results are provided in Appendix N (Table 22).

W1.2 Repeated-run Evaluation

We sampled two original questions per family and independently ran each model–question pair three times, using a fresh environment and interaction history for each run. We evaluated three representative models spanning different performance levels in the original evaluation: Gemini-3.8-Flash (44.2%, highest), GPT-5.6-Sol (34.2%, intermediate), and DeepSeek-V4.1-Flash (10.8% for the September checkpoint, lowest). The repeated runs used its October 17 update, DeepSeek-V4.1-Flash-1017. This yielded 216 episodes.

Table 3 reports the repeated-run results and uncertainty estimates.

Table 3. Repeated-run Stability

24 stratified questions · 3 models · 3 independent runs · 216 episodes.

ModelRun 1Run 2Run 3Mean SR ± SD (pp)95% CIOIR
Gemini-3.8-Flash4/2410/2411/2434.72 ± 15.77[29.17, 40.28]41.67%
GPT-5.6-Sol5/247/248/2427.78 ± 6.36[22.22, 33.33]16.67%
DeepSeek-V4.1-Flash-101710/2410/247/2437.50 ± 7.22[29.17, 45.83]20.83%

SD is computed across the three run-level success rates, in percentage points. The 95% CIs use 20,000 family-stratified question-cluster bootstrap resamples, retaining all three runs for each sampled question. Outcome Inconsistency Rate (OIR) is the percentage of questions with both successful and failed outcomes across the three runs.

On these questions, O2 succeeded in all 18 episodes, while O3 failed in all 18. This pattern held across all nine model–run combinations. All 18 I3 episodes reached Lock 1, but only one reached Lock 2.

Together, these evaluations show that the main family-level contrasts persist across independent questions and repeated runs, although overall success rates vary across runs.

Manuscript Changes. Section 5.2 now summarizes the O2–O3 results on independent questions and repeated runs. Section 5.1 points to Appendix K (Tables 17–18) for the additional evaluations and Appendix N (Table 22) for the complete pooled family-level results.

W2 / Q2. Causes of I3 Transfer Failure

We added controls to distinguish task feasibility, exploration difficulty, mechanism information, action ordering, and budget sensitivity. A generator-aware active-exploration agent solved all ten original I3 questions within the standard budgets, while additional LLM controls showed substantial effects from mechanism information and adjustment limits.

W2.1 Active Exploration and Ordering Baselines

Generator-aware active exploration. We evaluated a rule-based agent with access to the generator's structural rules, but not the instance-specific hidden graph or channel settings. It enumerated candidate Relay graphs and channel settings, read the operation log, and performed diagnostic interventions by testing pairs of Relay levels with the same remainder modulo four. After each intervention, it eliminated mechanisms inconsistent with the observed changes.

On all ten original I3 questions, it identified the hidden mechanism and completed Lock 1 within 31–43 environment steps (under the 50-step horizon). It then used the inferred graph to set Relays in upstream-to-downstream order, reaching the Lock 2 target in exactly five adjustments on every question (10/10).

Search, ordering rules, and distractors. In I3, Lock 2 specifies a target value for each of the five Relays. All five are relevant, so the challenge is not to select relevant Relays from a larger set of distractors, but to determine an effective action order. Setting an upstream Relay can overwrite values already established downstream.

We evaluated random search and simple ordering controls, summarized in Table 4.

Table 4. Random and Rule-based I3 Controls

StrategyEvaluation SettingSuccess
Random legal Relay values and random action order10,000 sampled plans per question across 10 questions; no Lock 2 targets or hidden mechanism0/100,000
All 120 target-setting action orders120 permutations per question across 10 questions; five target values fixed29/1,200 (2.42%)
Ascending Relay ID order70 original Lock 2 starting states0/70
Descending Relay ID order70 original Lock 2 starting states0/70
Observed-ancestor ordering70 original Lock 2 starting states; dependencies inferred from visible Lock 1 propagation2/70

The three ordering rules were evaluated on the 70 original Lock 2 starting states, preserved from the end of Lock 1. Observed-ancestor ordering placed Relay i before Relay j whenever changing i visibly changed j during Lock 1, breaking ties by Relay ID.

Each of the ten questions had between one and four successful orders among its 120 permutations (Appendix C.5). Thus, every original question admits a solution within the five-adjustment budget, although successful orders are sparse.

Reviewer_Gbo9OpenReview ↗
Rating
3(+2)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ELEMENTARY, a generator-based text game environment/benchmark for interactive LLM agents. Tasks are organized from four evidence regimes defined by how solution-relevant evidence becomes available: Association (relations explicit), Observation (sufficient passive evidence), Intervention (active manipulation), and Counterfactual (historical revision and replay). All generated question is validated so that the permitted evidence uniquely determines the required solution. The best average success is 44.2%, no model is consistently strong across regimes, and the regimes do not form a difficulty ladder.

Critical strengths
  1. Informative Diagnoses The deisgn in the Intervention regime is helpful to check whether knowledge acquired in one stage transfers to another or not when the goal changes. The result show that aggregate success could be not revealing enough: could be the case that a model performs unevenly, and the paper show case this result very well.

  2. Sufficient task Validity authors use a version space of mechanisms consistent with the evidence and validate each question means a unique solution while shortcut instances where rejected.

Critical weaknesses
  1. Limited Evaluation and possible over-claim: the headline figure 1 shows 10^7 to 10^23 approximate generator capacity, while the evaluation is on 120 cases (10 fixed questions per family, seed 42) with one run each. We do not have within-family variance shown as ablation and cells move in steps of 10 points.

  2. The central causal claim is not traditional causal and not shown to be actionable. I find the identifiability of an unique solution is only guaranteed within the generator's own rule/mechanism while success is scored on reaching targets. We do not have human baseline shown or discussion of how this text game can be helpful for LLM in real-world application on which direction, and how we can improve from this benchmark.

Questions for authors
  1. How expensive (in token count, time, or other dimensions) is one run?
6 comments

We thank the reviewer for the constructive feedback. ELEMENTARY is a generator-based benchmark covering four evidence-conditioned regimes and twelve task families. Its generators support task spaces ranging from approximately 10710^7 to 102310^{23} valid instances, while regime-specific validators check reachability, task-level identifiability, and shortcut avoidance. The benchmark now includes 240 validated questions (120 original and 120 independently generated). We will release the benchmark and an interface for generating and validating additional questions, supporting the development of more capable and reliable agents through targeted evaluation.

W1. Evaluation Scale and Variability

The generator capacities in Figure 1 represent the valid task spaces supported by our generators, not the number of questions evaluated. We expanded the evaluation with independently generated questions and repeated runs. Both evaluations reproduced the O2–O3 performance gap and I3 transfer failure (Tables 1 and 2).

W1.1 Independent New-question Evaluation

The original evaluation used 120 validated questions (ten per family). We generated and validated another 120 questions, again with ten per family. The seven models completed 810 episodes on this new set.

All seven models solved more O2 than O3 questions, while A1 matched or outperformed A3 (five higher, two tied). In I3, all 70 episodes reached Lock 1, but only one reached Lock 2 (1/70). Table 1 reports the results.

Table 1. Independent New-question Evaluation

120 new questions · 7 models · 810 episodes. Family-level results are success counts out of 10.

ModelOverall SRA1 / A3O2 / O3I3 Lock 1 → Lock 2
Gemini-3.8-Flash38.33%9 / 15 / 410 → 0
Grok-4.624.17%0 / 04 / 110 → 0
GPT-5.6-Sol45.00%9 / 37 / 210 → 0
DeepSeek-V4.1-Flash-101732.50%8 / 07 / 210 → 1
Kimi-K312.50%5 / 02 / 110 → 0
Claude-Opus-5*10.00%4 / 04 / 010 → 0
Qwen3.8-2.4T-A95B13.33%0 / 08 / 010 → 0
Overall—5 higher, 2 ties7/7 higher on O270/70 → 1/70

Claude-Opus-5 was evaluated on 90 questions, following the original evaluation scope. I3 Lock 2 success is conditioned on Lock 1 completion.

DeepSeek model update. The original benchmark comparison used the September checkpoint of DeepSeek-V4.1-Flash. For the additional evaluations, we used its October 17 update, DeepSeek-V4.1-Flash-1017, to rerun both the original and new question sets. It achieved 43/120 (35.83%) on the original questions and 39/120 (32.50%) on the new questions. The original September results are retained in the main benchmark comparison.

Pooled family-level results. Combining the original and new sets provides 20 questions per family, reducing success-rate granularity from 10 to 5 percentage points for families with complete coverage. Appendix N (Table 22) reports these pooled results, using the October DeepSeek update on both sets.

W1.2 Repeated-run Evaluation

We selected three models spanning different performance levels in the original evaluation: Gemini-3.8-Flash (44.2%, highest), GPT-5.6-Sol (34.2%, intermediate), and DeepSeek-V4.1-Flash (10.8% for the September checkpoint, lowest). The repeated runs used the October 17 update, DeepSeek-V4.1-Flash-1017.

We randomly sampled two questions per family from the original set and independently ran each model–question pair three times, with a fresh environment and interaction history for each run, yielding 216 episodes. Table 2 reports run-level success rates, mean, standard deviation, 95% confidence intervals, and Outcome Inconsistency Rate (OIR).

Table 2. Repeated-run Stability

24 stratified questions · 3 models · 3 independent runs · 216 episodes.

ModelRun 1Run 2Run 3Mean SR ± SD (pp)95% CIOIR
Gemini-3.8-Flash4/2410/2411/2434.72 ± 15.77[29.17, 40.28]41.67%
GPT-5.6-Sol5/247/248/2427.78 ± 6.36[22.22, 33.33]16.67%
DeepSeek-V4.1-Flash-101710/2410/247/2437.50 ± 7.22[29.17, 45.83]20.83%

Mean SR and SD are computed across the three run-level success rates; SD is in percentage points. The 95% CIs use 20,000 family-stratified question-cluster bootstrap resamples, sampling questions within each family while retaining all three run outcomes per sampled question. OIR is the percentage of the 24 questions on which a model both succeeded and failed across the three runs.

On this subset, all 18 O2 episodes succeeded and all 18 O3 episodes failed. All 18 I3 episodes reached Lock 1, but only one reached the new Lock 2 target within five adjustments.

Together, these results show that the O2–O3 contrast and I3 transfer failure persist across independent questions and repeated model runs, even as individual run-level success rates vary.

Manuscript Changes. We have clarified generator capacity and evaluation-set size in Figure 1, specified the original 120-question evaluation scope in the Abstract, Introduction, Figure 2, Table 2, and Conclusion, and clarified task-level identifiability in Section 3.2. Section 5.1 points to the additional evaluations. Appendix K (Tables 17–18) reports the independent-question and repeated-run results, while Appendix N (Table 22) provides the pooled results across 240 questions. These revisions are highlighted in yellow, while the shared robustness revisions in Appendix K retain their red highlighting.

W2. Causal Scope, Actionability, Human Evaluation, and Practical Relevance

This section clarifies the scope of ELEMENTARY's causal claims, examines whether causal information supports executable control in I3, and reports a matched human baseline and the benchmark's practical implications for agent development.

W2.1 Causal Scope and Task-level Identifiability

Pearl's framework provides the formal semantics for associational, interventional, and counterfactual reasoning. As stated in Section 3.1:

"ELEMENTARY builds on these causal semantics but studies a different dimension of reasoning: how an interactive agent obtains enough evidence to determine a task-relevant answer."

ELEMENTARY focuses on how interactive agents acquire and use evidence under four conditions: Association, Observation, Intervention, and Counterfactual. We do not claim to evaluate unrestricted causal discovery or require recovery of a complete causal graph.

Section 3.2 formally defines task-level identifiability (Eq. 2): every mechanism consistent with the available evidence must imply the same task-level solution. As stated in the paper:

"Evidence can therefore be sufficient for the task even when the complete mechanism is not uniquely recovered."

Each generator specifies its candidate mechanism space, and the validators check identifiability within that space. Thus, our guarantee is explicitly scoped to the generator-defined mechanism space: the remaining uncertainty need not be eliminated if it does not change the required task-level solution.

W2.2 From Causal Knowledge to Action

Section 3.2 explicitly connects task-level solutions to executable control:

"For control tasks, s(M,g)s(M, g) denotes the control solution required to reach the target under the hidden mechanism, rather than the target state itself."

I3 operationalizes this definition. During Lock 1, agents intervene to observe causal propagation. Lock 2 preserves the resulting environment state but introduces a new target. Agents must select and execute an effective action sequence to reach it within five adjustments.

True-mechanism control. We tested whether providing causal information improves executable control. We selected the same three model families used in the repeated-run evaluation, spanning high, intermediate, and lower performance in the original benchmark: Gemini-3.8-Flash (44.2%), GPT-5.6-Sol (34.2%), and DeepSeek-V4.1-Flash (10.8% for the September checkpoint).

At Lock 2, we provided the models with the true directed graph, channel conditions, and propagation rules, but no action plan or execution order. Models still selected and executed their own actions under the original five-adjustment limit. Table 3 reports the results.

Table 3. LLM Lock-2 Success With and Without True Mechanism Information

Three models · 10 original I3 questions per model · 30 episodes per condition.

ModelMatched baseline (no mechanism supplied)True Mechanism Information
Gemini-3.8-Flash0/1010/10
GPT-5.6-Sol0/105/10
DeepSeek-V4.1-Flash2/1010/10
Total2/3025/30

The matched three-model baseline is 2/30. The original seven-model I3 evaluation reported 1/70 in a separate evaluation batch.

Providing the true mechanism increased success from 2/30 to 25/30. GPT-5.6-Sol nevertheless failed on five of ten questions, showing that access to causal information does not automatically ensure successful action selection and execution.

We also performed an offline replay of the 69 original I3 failures. For each episode, we preserved its Lock-2 starting state and all five original Relay–value assignments, changing only their execution order under the true mechanism. The reordered actions reached the target in 36/69 episodes. This demonstrates that action ordering can recover many failures, although the reordering was performed offline rather than discovered by the agents during their original runs.

Reviewer_4mpvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce ELEMENTARY, which is a benchmark for evaluating how interactive agents obtain and use relevant evidence. It defines four evidence-conditioned regimes: explicit relation composition, passive inference, active intervention, and counterfactual historical revision, comprising twelve task families. Its central methodological contribution is task-level identifiability: regime-specific validators check that permitted evidence determines the required solution within the defined candidate mechanism space, alongside reachability and anti-shortcut constraints. Experiments show that all 70 I3 episodes achieve an initial control target, but only one achieves a subsequent target requiring propagation-aware action ordering under a constrained budget. Additional diagnostics distinguish evidence-search inefficiency, prediction errors, and unsuccessful historical revisions. These results highlight a gap between immediate task completion and reliable reuse of causal knowledge when objectives change.

Critical strengths
  1. Evaluates the causality reasoning ability of interactive agents. The benchmark distinguishes explicit relations, sufficient passive evidence, intervention-dependent evidence, and historical revision within a shared interactive framework.

  2. A principled treatment of evidence sufficiency in interactive evaluation. ELEMENTARY explicitly distinguishes task-level identifiability from agent success and implements regime-specific validation of reachability, evidence sufficiency, and shortcut avoidance.

Critical weaknesses
  1. Limited evaluation size: Each family has only ten questions, with one run per model–question pair and no uncertainty estimates. More task seeds and repeated runs would help show whether the findings hold beyond this small, fixed set.

  2. Sensitivity to budgets: Success depends heavily on search and adjustment limits, but the paper does not vary these limits or test whether revealing the exact budgets changes performance. This makes it difficult to separate reasoning limitations from sensitivity to the evaluation protocol.

  3. Missing diagnostic baselines: Random or simple heuristic strategies would put the low success rates in context. Providing the true mechanism would also help distinguish difficulty learning the system from difficulty planning actions.

Questions for authors

What explains the I3 transfer failure? Since Lock 1 can end before the mechanism is identified, did agents actually collect enough evidence to solve Lock 2? Could you distinguish insufficient exploration, incorrect mechanism inference, and planning errors through trajectory analysis or controlled experiments?

6 comments

We thank the reviewer for the constructive feedback. ELEMENTARY is a generator-based benchmark that separates evidence sufficiency from agent reasoning failure across four evidence-conditioned regimes and twelve task families. We have extended the evaluation with an independent question set and repeated runs, quantified sensitivity to both interaction budgets, and added random, heuristic, true-mechanism, and trajectory-level analyses. Together, these results show that I3 transfer failure is not explained by task infeasibility or budget disclosure alone: it involves both whether Lock 1 evidence supports a viable Lock 2 strategy and whether the agent can execute that strategy in the correct order.

W1. Evaluation Robustness

The O2–O3 performance gap and I3 transfer failure persisted on independently generated questions and across repeated model runs (Tables 1 and 2).

W1.1 Independent New-question Evaluation

The original evaluation used ten validated questions per family, shared across all models (Section 5.1). We generated 120 additional questions, with ten per family, and evaluated all seven models, yielding 810 episodes.

On the new questions, all seven models solved more O2 than O3 questions, while A1 matched or outperformed A3 (five higher, two tied). Most importantly, all 70 I3 episodes reached Lock 1, but only one reached Lock 2 (1/70). Table 1 reports the results.

Table 1. Independent New-question Evaluation

120 new questions · 7 models · 810 episodes.
Family-level results are success counts out of 10.

ModelOverall SRA1 / A3O2 / O3I3 Lock 1 → Lock 2
Gemini-3.8-Flash38.33%9 / 15 / 410 → 0
Grok-4.624.17%0 / 04 / 110 → 0
GPT-5.6-Sol45.00%9 / 37 / 210 → 0
DeepSeek-V4.1-Flash-101732.50%8 / 07 / 210 → 1
Kimi-K312.50%5 / 02 / 110 → 0
Claude-Opus-5*10.00%4 / 04 / 010 → 0
Qwen3.8-2.4T-A95B13.33%0 / 08 / 010 → 0
Overall—5 higher, 2 ties7/7 higher on O270/70 → 1/70

Claude-Opus-5 was evaluated on 90 questions, following the original evaluation scope. I3 Lock 2 success is conditioned on Lock 1 completion.

DeepSeek model update. The original benchmark comparison used the September checkpoint of DeepSeek-V4.1-Flash. For the additional evaluations, we used its October 17 update, DeepSeek-V4.1-Flash-1017, to rerun both the original and new question sets. It achieved 43/120 (35.83%) on the original questions and 39/120 (32.50%) on the new questions. We retain the original September results in the main benchmark comparison.

W1.2 Repeated-run Evaluation

We selected three models spanning different performance levels in the original evaluation: Gemini-3.8-Flash (44.2%, highest), GPT-5.6-Sol (34.2%, intermediate), and DeepSeek-V4.1-Flash, which had the lowest original success rate (10.8% for the September checkpoint). The repeated runs used the October 17 update, DeepSeek-V4.1-Flash-1017.

We randomly sampled two questions per family from the original set and independently ran each model–question pair three times, with a fresh environment and interaction history for each run, yielding 216 episodes. Table 2 reports run-level success rates, mean, standard deviation, 95% confidence intervals, and Outcome Inconsistency Rate (OIR).

Table 2. Repeated-run Stability

24 stratified questions · 3 models · 3 independent runs · 216 episodes.

ModelRun 1Run 2Run 3Mean SR ± SD (pp)95% CIOIR
Gemini-3.8-Flash4/2410/2411/2434.72 ± 15.77[29.17, 40.28]41.67%
GPT-5.6-Sol5/247/248/2427.78 ± 6.36[22.22, 33.33]16.67%
DeepSeek-V4.1-Flash-101710/2410/247/2437.50 ± 7.22[29.17, 45.83]20.83%

Mean SR and SD are computed across the three run-level success rates; SD is in percentage points. 95% CIs use 20,000 family-stratified question-cluster bootstrap resamples, sampling questions within each family while retaining all three run outcomes per sampled question. OIR is the percentage of the 24 questions on which a model both succeeded and failed across the three runs.

Across the repeated-run subset, all 18 O2 episodes succeeded and all 18 O3 episodes failed. This pattern held across all nine model–run combinations. For I3, all 18 episodes reached Lock 1, but only one reached the new Lock 2 target within five adjustments.

Together, these results show that the O2–O3 contrast and I3 transfer failure persist across independent questions and repeated model runs, even as individual run-level success rates vary.

Manuscript Changes. We have added the independent-question and repeated-run evaluation protocols in Section 5.1 and their results in Appendix K (Tables 17–18). The corresponding revisions are highlighted in red.

W2. Budget Sensitivity

The two budgets serve distinct evaluation objectives rather than imposing arbitrary interaction limits. In Association, the Puzzle Interaction Budget (PIB) measures evidence-acquisition efficiency by distinguishing targeted collection from broad search. In Intervention, the adjustment budget tests whether agents can use evidence acquired during exploration to control the system under a new target. We conducted sensitivity analyses to quantify how these budgets affect measured performance.

W2.1 Association: Puzzle Interaction Budget (PIB)

PIB design. As described in Appendix A.4, PIB defines a question-specific allowance on puzzle-evidence selections, independently of the environment-step horizon. It is designed to "distinguish targeted evidence collection from broad search." The allowance is computed before play by following the reference-guided solution process: the initial Reference selection counts once, and each subsequent stage accounts for the worst-case number of currently accessible, previously uncounted candidates compatible with the relation observed so far. Candidate sets are updated as new evidence becomes available. The reference-guided path determines the allowance but is not exposed to the agent; the agent must still discover the relevant relations through interaction.

PIB is a soft budget: exceeding it does not terminate an episode, and agents can continue exploring and may still recover the correct code. Official success, however, requires solving within PIB; correct solutions reached afterward are recorded as over-budget recovery.

Budget sensitivity. To quantify the effect of the PIB threshold independently of agent behavior, we conducted an offline re-scoring analysis of 210 recorded Association episodes, keeping the original trajectories and final answers unchanged. We progressively relaxed the threshold from B to B+8 and also report results without a PIB gate, counting all eventual correct solutions.

Table 3 compares the original cohort, using the September DeepSeek checkpoint, with an updated cohort that retains the original results for the other six models and replaces only the 30 September DeepSeek episodes with October 17 DeepSeek-V4.1-Flash-1017 reruns on the same questions.

Table 3. Association Success under Different PIB Thresholds

210 episodes per cohort. Success counts are reported at each threshold; the no-PIB-gate condition counts all eventual correct solutions.

PIB thresholdOriginal (September DeepSeek)Updated (October DeepSeek-1017)
B53/21064/210
B+269/21080/210
B+482/21093/210
B+684/21099/210
B+891/210111/210
No PIB gate112/210138/210

In the updated cohort, 74 episodes eventually recovered the correct code after exceeding the original PIB. Allowing two additional selections brought 16 of these episodes within the revised threshold, while 27 correct episodes remained over budget even at B+8. Since the trajectories and final answers were held fixed, these results isolate how the PIB threshold affects official success, rather than reflecting changes in agent behavior.

W2.2 Intervention: Adjustment Budget

Budget design. The Intervention adjustment budget serves a different purpose from PIB. Lock 1 provides a 50-step environment horizon for exploration, without a separate adjustment budget. Once its target is reached, the environment state is preserved and Lock 2 introduces a new target, requiring the agent to use previously acquired information for further control. The adjustment budgets are two for I1, two for I2, and five for I3.

These limits are tied to the family-specific control tasks. Targets are generated from the hidden mechanism and validated for reachability before a question is admitted. In I3, the Lock 2 target is constructed so that successful control requires actions consistent with the directed propagation structure (Appendix C). Thus, the adjustment budget measures state-changing control during transfer, rather than simply limiting the number of observations an agent can obtain.

Budget disclosure. To cover different performance levels, we selected the same three representative model families as in the repeated-run evaluation: Gemini-3.8-Flash (44.2%, highest), GPT-5.6-Sol (34.2%, intermediate), and DeepSeek-V4.1-Flash (10.8% for its September checkpoint, lowest). We evaluated them on the ten original I3 questions, explicitly revealing the five-adjustment Lock 2 limit from the beginning of Lock 1 while keeping other interaction conditions unchanged.

Budget relaxation. We then increased the Lock 2 limit from five to ten effective adjustments. The target was checked after each adjustment, and the episode terminated at the first successful target hit. Of the 27 successful episodes, 26 first reached the target on adjustments 6–10; only one succeeded within the original five-adjustment limit.

Table 4. I3 Budget Control Experiments

Three models · 10 original I3 questions per model · 30 episodes per condition.

ModelMatched baselineBudget RevealedBudget Relaxed
Gemini-3.8-Flash0/100/1010/10
GPT-5.6-Sol0/100/108/10
DeepSeek-V4.1-Flash2/100/109/10
Total2/300/3027/30

The matched baseline (2/30) is shared by the three-model budget and true-mechanism controls (Tables 4 and 6). The original seven-model evaluation reported in the paper yielded 1/70.

Adjustment accounting. An environment step records a legal interaction with the environment, whereas an effective adjustment is a legal control action that changes a selected control from its current value. Each state-changing action consumes one adjustment, regardless of how many downstream Relays are affected by propagation. Inspection and reading consume no adjustments. Setting a control to its current value is a no-op: it consumes an environment step but no adjustment and triggers no downstream propagation (Appendix C.3).

Together, these analyses establish the distinct roles of the two budgets. PIB quantifies evidence-acquisition efficiency under a question-specific allowance, while the Intervention adjustment budget evaluates the ability to transfer acquired information into state-changing control. Relaxing the PIB threshold brings more correct solutions within the official allowance; in I3, disclosing the budget alone does not improve success, whereas additional adjustment opportunities enable most tested episodes to reach the new target.

Manuscript Changes. We have clarified the budget designs in Section 4.3, added the PIB computation in Appendix A.4 and the standard target-check rules in Appendix C.3, and reported the budget sensitivity analyses in Appendix L (Tables 19–20). The corresponding revisions are highlighted in red.

Comments on the paper

2 comments

Dear ICLR 2027 Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

Thank you very much for your time and thoughtful feedback.

We have carefully revised the manuscript in response to your constructive comments and suggestions. To help locate the revisions, the color-coded highlights in the updated manuscript correspond to the following reviewers:

  • Red: Revisions addressing Reviewer 4mpv, including the shared robustness revisions in Appendix K.
  • Yellow: Revisions addressing Reviewer Gbo9. The shared robustness revisions in Appendix K retain their red highlighting.
  • Blue: Revisions addressing Reviewer kcZH.

We provide detailed point-by-point responses below and indicate the corresponding manuscript changes in each response.

We sincerely appreciate your constructive feedback. We believe these revisions have strengthened the empirical support, clarity, and overall presentation of ELEMENTARY, and we hope they adequately address your concerns.

Thank you again for your time and consideration.

AuthorsOpenReview ↗

ELEMENTARY: Summary of Revisions and Key Evidence

Dear ICLR 2027 Program Chairs, Senior Area Chairs, Area Chairs, and Reviewers,

Thank you for your thoughtful feedback and constructive suggestions.

ELEMENTARY is a generator-based interactive benchmark evaluating how LLM agents acquire and use task-relevant causal evidence across four evidence-conditioned regimes and twelve generator families. Its validators check reachability, task-level identifiability, and shortcut avoidance. We will release the benchmark and an interface for generating and validating additional questions, supporting the development of more capable and reliable agents through targeted evaluation.

We have revised the manuscript and conducted additional experiments to address the main concerns:

  • Evaluation robustness (4mpv, Gbo9, kcZH). We added 120 independently generated questions (240 validated questions in total), evaluated all seven models on the new set (810 episodes), and conducted 216 repeated-run episodes with 95% confidence intervals. The O2–O3 performance gap and I3 transfer failure persisted (Appendices K and N).

  • Budget sensitivity and diagnostic baselines (4mpv, kcZH). We analyzed PIB sensitivity and I3 budget controls, including a relaxed budget of up to ten adjustments with first-hit termination (27/30). We added random, heuristic, and active-exploration baselines; a generator-aware agent solved all ten original I3 questions within the standard budgets (Appendix L).

  • Causal scope and I3 failure analysis (4mpv, Gbo9, kcZH). We clarified task-level identifiability within the generator-defined mechanism space and analyzed all 70 original I3 trajectories. Providing the true mechanism increased matched Lock-2 success from 2/30 to 25/30, while offline action reordering recovered 36/69 original failures (Section 3.2; Appendices L–M).

  • Human evaluation, practical relevance, and efficiency (Gbo9). We added a matched human baseline (12 participants, 72 episodes), discussed directions for targeted agent improvement, and reported per-model evaluation costs. Across 810 episodes, average usage was 218.9K tokens and 663.4 seconds of cumulative API-response latency per episode (Appendices O–P).

Revision highlights:

  • 🟥 Red — Reviewer 4mpv, including shared robustness revisions in Appendix K.
  • 🟨 Yellow — Reviewer Gbo9.
  • 🟦 Blue — Reviewer kcZH.

Detailed point-by-point responses and corresponding manuscript changes are provided under each review. We believe these revisions have strengthened the empirical support, clarity, and practical value of ELEMENTARY, and we welcome further questions or suggestions.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.