- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to build long-horizon agentic environments and make them useful for RL training. The authors introduce Tev-Gym, with 1,170 environments across computing and mathematics. The main idea is to start from a latent world and use it to construct the task inputs, verifier, and progress scorer together. A workflow DAG describes the intermediate checkpoints and their dependencies. The paper also proposes progress-potential reward shaping, so failed rollouts can still provide training signals when they make different amounts of progress. The authors train 9B and 27B models and report improvements on ALE-CM and several Terminal-Bench evaluations. They further compare different choices in environment construction, reward design, and advantage normalization through ablation experiments.
Critical strengths
-
The environment construction is the strongest part for me. Building the task inputs, verifier, and progress scorer from the same latent world helps keep them consistent. The independent reference solvers and exploit checks also give more confidence on the reward quality. This is quite important when using these environments for RL training.
-
The progress reward addresses a practical problem in training. For difficult tasks, a group of failed rollouts can easily give no useful learning signal. Scoring how much progress they made allows these samples to still contribute to training. The scorer comes from the environment specification, so there is no need of training another reward model.
-
The evaluation covers both 9B and 27B models, with improvements on ALE-CM and several Terminal-Bench evaluations. I find the Terminal-Bench 2.1 results useful, since they give some evidence that the benefit extends beyond ALE-CM. The ablations also help understand better how environment construction and the different reward choices affect the results.
Critical weaknesses
-
The reward ablations leave some uncertainty about the additional benefit of turn-level credit assignment. As explicitly shown in Eq. (4), the undiscounted return equals terminal success plus a final-state progress bonus. A useful additional baseline would use same final reward with ordinary rollout-level advantages. The sparse baseline removes partial credit entirely, while the pooled-normalization variant still uses turn-level returns. This comparison would help understand better how much benefit comes from partial credit and how much comes from the turn-level training design.
-
The stability of the ALE-CM improvements is difficult to assess from the reported results. The evaluation contains 18 tasks with five rollouts per task, and the 27B Pass@5 improvement corresponds to increasing the solved task count from three to five. Confidence intervals and results from repeated training seeds are not reported. These would give more confidence on the stability of the improvement and the observed Pass@5 parity with the specified frontier models.
-
A smaller reporting issue concerns the training budget of the ablations. Table 5 already provides useful evidence through GPU-hours per step and evaluation scores. However, the total optimization steps and cumulative GPU-hours for each ablation are not explicitly reported. It would be useful to report these budgets more clearly, together with learning curves against cumulative GPU-hours. This would make the efficiency comparison easier to judge under the same compute budget.
Questions for authors
-
Have you tried a baseline using same final reward, terminal success plus the final-state progress bonus, but with ordinary rollout-level advantages? Since Eq. (4) telescopes under the undiscounted formulation, this comparison would help understand better the additional benefit of turn-level credit assignment beyond giving partial credit to failed rollouts.
-
Could you provide confidence intervals for the ALE-CM results, and results from training with different random seeds? The 27B Pass@5 improvement corresponds to increasing the solved task count from three to five out of 18 tasks. Additional measurements would give more confidence on how stable this improvement and the reported Pass@5 parity are.
-
For the ablations in Table 5, how many optimization steps were used for each run, and what was the total GPU budget? Could you also provide learning curves against cumulative GPU-hours? GPU-hours per step is already useful, but these additional results would make it easier to compare the convergence with the same amount of compute.
4 comments
Rebuttal Response (2/n)
W2 and Q2: Could you provide confidence intervals for the ALE-CM results, and results from training with different random seeds? The 27B Pass@5 improvement corresponds to increasing the solved task count from three to five out of 18 tasks.
We agree, and we will report task counts and confidence intervals alongside the rates.
The gain in task counts. TEV-27B's P@5 gain is 3 → 5 of 18 tasks. Both new tasks are in Math & Science (2/8 → 4/8), while ML (0/5) and Coding & System (1/5) are unchanged. S@1, which averages the graded score over 90 rollouts per model, is the less noisy signal, and it rises in all three categories: Math & Science 0.15 → 0.23, ML 0.21 → 0.26, Coding & System 0.74 → 0.82.
Confidence intervals. Paired bootstrap 95% CIs (10,000 resamples of tasks; for S@1 and P@1, the 5 trials within each task are resampled too) give TEV-27B's ALE-CM gains as:
| ALE-CM, 27B (18 tasks) | Qwen3.8-27B | TEV-27B | Δ [95% CI] |
|---|---|---|---|
| S@1 | 0.33 | 0.40 | +0.067 [+0.001, +0.140] |
| S@5 | 0.43 | 0.54 | +0.114 [−0.008, +0.281] |
| P@1 | 0.12 | 0.16 | +0.033 [−0.022, +0.100] |
| P@5 | 0.17 | 0.28 | +0.111 [+0.000, +0.278] |
The S@1 gain excludes zero and the P@5 gain has a lower bound of 0.000. The S@5 and P@1 CIs dip below zero by at most 0.022, as expected with 18 tasks, so no CI points to a regression. We will add these CIs to every Δ row; the table for all benchmarks is in our response to Reviewer Nci5 (W1.1), where 6 of the 16 gains exclude zero.
Training seeds. Every run in the paper uses seed 42. A 27B run takes about four days on 32 B200 GPUs, so we cannot repeat it with new seeds within the discussion period.
Rebuttal Response (3/3)
W3 and Q3: For the ablations in Table 5, how many optimization steps were used for each run, and what was the total GPU budget? Could you also provide learning curves against cumulative GPU-hours?
Thank you for this suggestion. We will add the number of optimizer steps and the cumulative GPU-hours of every run to Table 5 caption, together with learning curves of training reward and ALE-CM score against cumulative GPU-hours in our appendix.
All 9B models are trained for 42 steps in this ablation study. Therefore the total GPU budget is the GPU-h / step column in our paper's Table 5 times 42 steps.
Learning curves. We have added these curves to the revised paper, with all new text in red. Figure 6 in Appendix B (page 33, lines 1758–1773) plots the training score, the mean score of the 192 rollouts of each step, against the training step for PPRS and the reward ablations. The paragraph Training curves of the reward ablations (Appendix B, lines 1749–1756) discusses the curves, and the Table 5 caption now points to the figure. Over these 42 steps:
- PPRS has the highest mean training score: 0.47, against 0.19 for Sparse, 0.24 for a terminal-only reward with partial credit, and 0.37 for Direct reward (additive).
- PPRS is still improving at step 42, where its smoothed score reaches 0.65. The two terminal-only arms stay below 0.4 throughout.
- Direct reward catches up with PPRS between steps 17 and 22, but then stays between 0.34 and 0.50 while PPRS keeps rising.
Because each run's GPU-h/step is constant, the step axis converts to cumulative GPU-hours by multiplying by the GPU-h/step column of Table 5. The 42 steps cost 378 H100 GPU-hours for PPRS, against 701 for Sparse and 407 for Direct reward, so PPRS reaches a higher training score at a lower GPU budget.
Additional Results for TB 4.0 generalization
We further report results of TEV-320B on the full TB 4.0 (66 tasks). Our main contribution is the latent-world-first data synthesis pipeline. To further showcase the capability of our curation method, we extended TEV-GYM to new domains such as Software and Security and enlarged existing domains such as Science and ML, yielding TEV-GYM-improved. To train a model that can learn from this high-quality, complex, long-horizon data, we chose GLM-5.3-Flash (a 320B model) and RL-trained it with the same recipe as in our paper. The results of TEV-320B, with confidence intervals, are listed below.
| TB 4.0, full (66 tasks) | P@1 | P@5 |
|---|---|---|
| GLM-5.3-Flash (base) | 0.358 ± 0.049 | 0.561 ± 0.061 |
| TEV-320B (ours) | 0.448 ± 0.054 | 0.667 ± 0.059 |
| Δ [95% CI] | +0.090 [+0.010, +0.165] | +0.106 [0.000, +0.210] |
| Other models | ||
| Fable 5 | 0.445 ± 0.050 | 0.682 ± 0.058 |
| GPT-5.6 Sol | 0.373 ± 0.049 | 0.606 ± 0.061 |
As the table shows, TEV-320B significantly surpasses its base model on TB 4.0 in both P@1 and P@5. Compared with proprietary models, TEV-320B matches Fable 5 at 30× lower cost (Fable 5 is priced at USD 10 / 50 per 1M input / output tokens, while GLM-5.3-Flash, the base model of TEV-320B, is priced at USD 0.15 / 0.50). These results demonstrate the significance of our data synthesis pipeline and PPRS training method.
We sincerely appreciate your thoughtful comments and hope our responses have addressed your concerns. Please let us know if you have any further questions.
Best, Authors
I think those feedbacks really address my concerns, and I will main my score. Thanks for your hard work.
Rebuttal Response (1/n)
Thank you for your careful reading and constructive feedback. We are glad you found that building the task inputs, verifier, and progress scorer from one latent world keeps them consistent, and that the independent reference solvers and exploit checks give confidence in the reward quality. We are also glad you found that the progress reward addresses a practical problem in training without another reward model, and that the TB 2.1 results show the benefit extends beyond ALE-CM.
We address each point below and will update the paper accordingly. We are also happy to engage further during the discussion period to clarify any remaining questions.
Line numbers. All line numbers below refer to the revised PDF, where new text is in red. Pages 1–9 are unchanged, so main-text line numbers are the same as in the submitted version.
We thank the reviewer for this precise question. We trained this baseline at 9B with the same recipe (Final partial-reward, rollout-level below) and evaluated it on ALE-CM:
This baseline barely improves on Sparse and is clearly worse than PPRS. It stays at the level of Sparse and the base model (S@1 0.26 vs. 0.25; P@1 and P@5 unchanged at 0.06), while TEV-9B reaches 0.29 S@1, 0.40 S@5, and 0.11 P@5. The gain of PPRS therefore comes from turn-level credit assignment, not from partial credit alone. With the same final reward, crediting each turn by its return G_t already raises S@1 and S@5 (PPRS, pooled norm.: 0.28 and 0.35), and normalizing advantages per turn (TEV-9B) further raises S@5 to 0.40 and P@5 from 0.06 to 0.11.
The training curve of this arm, shown in our response to W3 below, gives the same finding: over the 42 training steps its mean training score is 0.24, close to Sparse (0.19) and far below PPRS (0.47). This also matches our intuition. A terminal partial reward gives partial credit only at the end of the rollout. Although failed rollouts no longer all receive 0, many rollouts in a group can still end with the same reward, and such a group gives no learning signal. PPRS avoids this by crediting each turn with the progress it makes, so rollouts that end with the same reward still receive different per-turn advantages.