acceptodds
Under review as a conference paper at ICLR 2027

SCOPE: Scaling Counterfactual, Observation-Preserving Environment Families for Terminal Agents

Abstract

Reinforcement learning on executable environments has become the main route to terminal agents, and scaling such environments is its precondition. Whatever their source, most existing efforts produce the same unit of data, a task with one solution verified in one initial state. Yet a successful run shows only that one plan worked in one state, not that the agent explored the state first. When several legal states fit its observations and it acts on one unchecked, the reward credits the run as fully as one that checked first. We term this shortfall exploration debt, and scaling by successful runs accumulates it rather than collects it. We present SCOPE, which scales environments around what a successful run leaves unresolved rather than around the runs a source yields. SCOPE locates the expansion point at which a frozen reference rests on facts its history has not fixed, grows a counterfactual family of initial states that reproduce the history yet break the continuation, certifies one blind check and one program that complete every member, and delivers each member from its root and from a frontier after the prefix, all by real execution and independent of any learner. The resulting SCOPE-4K holds 4,000 executable terminal training environments spanning 11 domains, with at least 1,024 counterfactual families at its core and at least 80% long-horizon tasks. Direct RL on this collection, with the verifier's reward alone, then collects the debt without any reward shaping. Extensive experiments on Terminal-Bench 2.1 and four held-out suites demonstrate that SCOPE is (1) effective, lifting Qwen3.5-9B from 22.2% to 39.4% on Terminal-Bench 2.1 and surpassing the strongest constructed baseline by up to 6.3 points, (2) scale-consistent, holding its gains from Qwen3.5-4B to 27B on one fixed collection, and (3) cost-efficient, since neither a reminder, a fourfold inference budget, nor a matched construction budget closes the gap. Our code is available here, and SCOPE-4K is provided as supplementary material.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Environment Scaling, Terminal Agents, Agentic Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_cibNOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

In this work, the authors study the problem of single-state terminal environments containing examples where the model passes them by lucky guesses. Training on them doesn't punish those lucky guesses, so the generalization is poor. In this work, SCOPE builds some sibling environments to existing ones with some filters so that the constructed siblings have the same environment state, and in those, the guess fails. By training on these with GRPO on Terminal Bench 2.1, the 9B model sees an increase from 22.2 to 39.4, 6.3 points above the comparable best baseline. They also release the constructed environments and the full pipeline.

Critical strengths
  • The authors identify a clear problem in some of the environments used for terminal benchmark RL training.
  • And the approach being taken to not fix the reward but construct diverse environments, and the results shown by real execution, seem like the right bet.
  • The matching baselines also isolate the value that their construction checks provide. For example, for the 9B model, 33.1 versus 39.4, so the checks they constructed are also important.
  • We also show that the gain actually comes from learning to check, not just thinking more. Gives confidence in the approach
  • Data and pipeline being released is a big plus.
Critical weaknesses
  • Most of the evaluation is in-house, with Terminal Bench 2.1 being the only external benchmark. The benchmark "unseen" is built with the same checks that their method trains on, so it's unclear whether these gains would generalize in out-of-distribution benchmarks.
  • A tiny SFT of 262 demos already gives an 8.6 boost, and no larger SFT baseline or no-privileged-teacher baseline is shown. It's unclear whether an RL stage is needed here or this is just something which can be fixed on an SFT stage.
  • The paper is a bit hard to read, and the text is very dense and full of invented terms, with the math formalizing simple checks. Figure 2 is unreadable.
  • The scaling story is unclear. If a run sees only 63 of 1,024 families, then how is the 1,024-family pool being credited with a jump of 10.6 points?
  • It also isn't clear how common the problem is. The paper says that 68% of tasks are affected, but the number came from searching for this problem by constructing a pipeline to break tasks
Questions for authors
  • It would be interesting to see how larger safety and privileged compute baselines compare with the RL approach here, with an ISO compute setting.
  • It would be interesting to see if strong state-of-the-art open models or frontier models still fail on these sibling states. Could be that with stronger RL training or a more rubric-based one, this problem naturally goes away.
  • Which terminal bench tasks actually improve, and are the gains on tasks that need this kind of verification, or are they spread more evenly?
  • How much of this depends on the stronger teacher ie GLM5.3 here? If you're using such a strong teacher, why can't we just now do direct distillation from that teacher on some tasks rather than constructing these complex environments?
  • Did you measure regression on any coding or general benchmarks with this?
Reviewer_48ULOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces an issue and defines it as exploration debt, then proposes SCOPE to solve this issue. SCOPE addresses this issue by starting from a successful reference trajectory, selecting a point where the continuation depends on unresolved information, and constructing counterfactual environments. The experimental results show solid gains over baselines.

Critical strengths
  1. The paper defines an interesting and meaningful failure of Reinforcement Learning: exploration debt. Exploration debt is useful for defining the gap between successful trajectories and robust behavior
  2. The experiment is thorough, and the ablation study strongly supports the main method.
Critical weaknesses
  1. The method depends on a successful reference trajectory, and this trajectory could already solve the task, but may solve the task in a state-dependent manner.
  2. The format and the presentation have some problems. In Figure 1, the bottom-right figure lacks a caption; the axes are not defined; the top figure has text overlapping the image. In Figure 2, the icon and text also overlap. For sections 2.2, 2.3, 2.4, I believe the icons in the subsection headings are not included in the template.
Questions for authors

How sensitive is the framework to the ability of the LLMs used for expansion-point selection and counterfactual review? Have the authors tested weaker models?

Reviewer_L9A5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that an outcome reward can’t tell an agent that checked a hidden fact before acting from one that guessed and got lucky, unless the training distribution contains states where the guess fails. The authors call this unresolved dependence on the state “exploration debt.” The proposed method, SCOPE, builds such states from successful reference runs:

  • An LLM teacher’s first verified solution is frozen as the reference. LLM proposers and judges pick one step where later decisions rest on facts the history hasn’t revealed.
  • LLM-proposed edits to the initial state are kept only if execution confirms three things:
    • (H) the prefix history is reproduced byte for byte;
    • (F) the frozen continuation fails;
    • (I) one blind check plus one common program completes every member of the family, verified on fresh rebuilds.
  • Each member is delivered from its root and from a post-prefix “frontier,” and families are sampled with balanced weights.
  • Training uses GRPO on the verifier reward alone, with no shaping and no SFT.

On multiple sizes of Qwen, SCOPE raises Terminal-Bench success rate. The paper also reports four internal held-out suites, matched-cost and inference-budget comparisons, gate ablations, information interventions, and comparisons with six released terminal datasets.

Critical strengths
  1. Clear and principled framing, grounded on implicit partial observability. The argument that outcome reward can’t separate checking from guessing without states that tell them apart is crisp
  2. Certification by execution. History Preservation, Continuation Failure and Family witness, are all checked by running code, not by LLM judgment. The witness gate is an especially thoughtful design choice: it ensures each family can be solved by one behavior the learner could acquire, so it doesn’t just penalize the original plan
  3. Carefully designed empirical study
  4. Candid limitations section and dataset release
Critical weaknesses
  1. The problem isn't new. A policy that succeeds without checking is still optimal for its training data. This is underspecification leading to an OOD failure which there is plenty of literature for.
  2. Missing comparisons with environment design. The method is close to regret-based environment design (PAIRED, ACCEL, PLR) and reverse curriculum, but all baselines are variants of the authors' own pipeline.
  3. The pipeline does the exploration, not the policy. LLMs find what to check and write the solution. It's unclear whether the policy transfers to new kinds of hidden facts, and the models used for construction aren't clearly specified.
  4. Open-loop design. Families target the teacher's blind spots, not the learner's.
  5. Limited external evidence. The internal suites come from the same pipeline, so Terminal-Bench 2.1 is the only independent result. The margins there are tight, the intervals ignore seed variance, and only one model family is tested.
  6. Narrow, under-described families. The paper doesn't show which edit types make it into the dataset, and the main text has no full example.
  7. Confusing setup. It's unclear what data the main table uses. The headline gain also includes what any extra RL would give.
  8. Hard to read. Dense, jargon-heavy, and key details are in the appendix.
Questions for authors
  1. Which data does the main table use: the matched number of variants per parent or the full SCOPE-4K? The same 9B result appears in both settings.
  2. Which models build the data (reference solutions, point selection, edits, check and program)? How sensitive are the results to their strength?
  3. How does SCOPE compare with simple randomization over known hidden factors, or with a UED-style method such as PLR?
  4. Does the learned checking transfer to kinds of hidden facts that aren't in the edit library?
  5. Did you try SCOPE on other external benchmarks?
  6. What kinds of edits make it into SCOPE-4K? A couple of complete example families would help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.