- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 3
Read review
Paper summary
In this work, the authors study the problem of single-state terminal environments containing examples where the model passes them by lucky guesses. Training on them doesn't punish those lucky guesses, so the generalization is poor. In this work, SCOPE builds some sibling environments to existing ones with some filters so that the constructed siblings have the same environment state, and in those, the guess fails. By training on these with GRPO on Terminal Bench 2.1, the 9B model sees an increase from 22.2 to 39.4, 6.3 points above the comparable best baseline. They also release the constructed environments and the full pipeline.
Critical strengths
- The authors identify a clear problem in some of the environments used for terminal benchmark RL training.
- And the approach being taken to not fix the reward but construct diverse environments, and the results shown by real execution, seem like the right bet.
- The matching baselines also isolate the value that their construction checks provide. For example, for the 9B model, 33.1 versus 39.4, so the checks they constructed are also important.
- We also show that the gain actually comes from learning to check, not just thinking more. Gives confidence in the approach
- Data and pipeline being released is a big plus.
Critical weaknesses
- Most of the evaluation is in-house, with Terminal Bench 2.1 being the only external benchmark. The benchmark "unseen" is built with the same checks that their method trains on, so it's unclear whether these gains would generalize in out-of-distribution benchmarks.
- A tiny SFT of 262 demos already gives an 8.6 boost, and no larger SFT baseline or no-privileged-teacher baseline is shown. It's unclear whether an RL stage is needed here or this is just something which can be fixed on an SFT stage.
- The paper is a bit hard to read, and the text is very dense and full of invented terms, with the math formalizing simple checks. Figure 2 is unreadable.
- The scaling story is unclear. If a run sees only 63 of 1,024 families, then how is the 1,024-family pool being credited with a jump of 10.6 points?
- It also isn't clear how common the problem is. The paper says that 68% of tasks are affected, but the number came from searching for this problem by constructing a pipeline to break tasks
Questions for authors
- It would be interesting to see how larger safety and privileged compute baselines compare with the RL approach here, with an ISO compute setting.
- It would be interesting to see if strong state-of-the-art open models or frontier models still fail on these sibling states. Could be that with stronger RL training or a more rubric-based one, this problem naturally goes away.
- Which terminal bench tasks actually improve, and are the gains on tasks that need this kind of verification, or are they spread more evenly?
- How much of this depends on the stronger teacher ie GLM5.3 here? If you're using such a strong teacher, why can't we just now do direct distillation from that teacher on some tasks rather than constructing these complex environments?
- Did you measure regression on any coding or general benchmarks with this?