acceptodds
Under review as a conference paper at ICLR 2027

Tev: Scaling Long-Horizon Agentic Environments with Dense Progress Rewards

Abstract

Long-horizon agents have drawn growing attention for their ability to work autonomously—from hours to days—on economically valuable tasks, a capability essential to automated discovery and recursive self-improvement (RSI). Yet open-source models still struggle on long-horizon tasks, largely for lack of both training data and a working recipe, and scaling both is hard: long, complex tasks are difficult to curate and verify, and their rewards are sparse. We address both. First, we introduce Tev-Gym, the first long-horizon agent training dataset with verifiable, dense, progressive rewards—1,170 verified task environments across 7 domains. It is built on a latent-world-first approach that first constructs a latent world: a complete and correct hidden specification of the task domain, workflow, and expected state after each step. Each task is then progressively rendered from this specification into an agent-facing environment, enabling controllable complexity and a natural progress metric. Second, we turn this metric into a training recipe: via progress-potential reward shaping (PPRS), we convert progress into a dense per-step reward that improves training efficiency and makes otherwise-intractable tasks trainable. Training with this recipe yields Tev-27B, which achieves substantial gains on the Agents' Last Exam computing and math sciences domains (ALE-CM), matching GPT-6 Astra and Claude Opus 5 at Pass@5. Despite training only on ALE-related data, it generalizes to the Terminal-Bench (TB) 3.0 and 4.0 subsets, surpassing GLM 5.2 and Gemini 3.7 Flash, and reaches Claude Opus 4.8-level performance on middle-horizon tasks such as TB 2.1.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM, Agents, RL, Long-Horizon Tasks, Environment Synthesis, Dense Reward, Synthetic Data

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_RBSHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to build long-horizon agentic environments and make them useful for RL training. The authors introduce Tev-Gym, with 1,170 environments across computing and mathematics. The main idea is to start from a latent world and use it to construct the task inputs, verifier, and progress scorer together. A workflow DAG describes the intermediate checkpoints and their dependencies. The paper also proposes progress-potential reward shaping, so failed rollouts can still provide training signals when they make different amounts of progress. The authors train 9B and 27B models and report improvements on ALE-CM and several Terminal-Bench evaluations. They further compare different choices in environment construction, reward design, and advantage normalization through ablation experiments.

Critical strengths
  1. The environment construction is the strongest part for me. Building the task inputs, verifier, and progress scorer from the same latent world helps keep them consistent. The independent reference solvers and exploit checks also give more confidence on the reward quality. This is quite important when using these environments for RL training.

  2. The progress reward addresses a practical problem in training. For difficult tasks, a group of failed rollouts can easily give no useful learning signal. Scoring how much progress they made allows these samples to still contribute to training. The scorer comes from the environment specification, so there is no need of training another reward model.

  3. The evaluation covers both 9B and 27B models, with improvements on ALE-CM and several Terminal-Bench evaluations. I find the Terminal-Bench 2.1 results useful, since they give some evidence that the benefit extends beyond ALE-CM. The ablations also help understand better how environment construction and the different reward choices affect the results.

Critical weaknesses
  1. The reward ablations leave some uncertainty about the additional benefit of turn-level credit assignment. As explicitly shown in Eq. (4), the undiscounted return equals terminal success plus a final-state progress bonus. A useful additional baseline would use same final reward with ordinary rollout-level advantages. The sparse baseline removes partial credit entirely, while the pooled-normalization variant still uses turn-level returns. This comparison would help understand better how much benefit comes from partial credit and how much comes from the turn-level training design.

  2. The stability of the ALE-CM improvements is difficult to assess from the reported results. The evaluation contains 18 tasks with five rollouts per task, and the 27B Pass@5 improvement corresponds to increasing the solved task count from three to five. Confidence intervals and results from repeated training seeds are not reported. These would give more confidence on the stability of the improvement and the observed Pass@5 parity with the specified frontier models.

  3. A smaller reporting issue concerns the training budget of the ablations. Table 5 already provides useful evidence through GPU-hours per step and evaluation scores. However, the total optimization steps and cumulative GPU-hours for each ablation are not explicitly reported. It would be useful to report these budgets more clearly, together with learning curves against cumulative GPU-hours. This would make the efficiency comparison easier to judge under the same compute budget.

Questions for authors
  1. Have you tried a baseline using same final reward, terminal success plus the final-state progress bonus, but with ordinary rollout-level advantages? Since Eq. (4) telescopes under the undiscounted formulation, this comparison would help understand better the additional benefit of turn-level credit assignment beyond giving partial credit to failed rollouts.

  2. Could you provide confidence intervals for the ALE-CM results, and results from training with different random seeds? The 27B Pass@5 improvement corresponds to increasing the solved task count from three to five out of 18 tasks. Additional measurements would give more confidence on how stable this improvement and the reported Pass@5 parity are.

  3. For the ablations in Table 5, how many optimization steps were used for each run, and what was the total GPU budget? Could you also provide learning curves against cumulative GPU-hours? GPU-hours per step is already useful, but these additional results would make it easier to compare the convergence with the same amount of compute.

4 comments
AuthorsOpenReview ↗

Rebuttal Response (1/n)

Thank you for your careful reading and constructive feedback. We are glad you found that building the task inputs, verifier, and progress scorer from one latent world keeps them consistent, and that the independent reference solvers and exploit checks give confidence in the reward quality. We are also glad you found that the progress reward addresses a practical problem in training without another reward model, and that the TB 2.1 results show the benefit extends beyond ALE-CM.

We address each point below and will update the paper accordingly. We are also happy to engage further during the discussion period to clarify any remaining questions.

Line numbers. All line numbers below refer to the revised PDF, where new text is in red. Pages 1–9 are unchanged, so main-text line numbers are the same as in the submitted version.

W1 and Q1: Have you tried a baseline using the same final reward, terminal success plus the final-state progress bonus, but with ordinary rollout-level advantages? Since Eq. (4) telescopes under the undiscounted formulation, this comparison would help separate the benefit of turn-level credit assignment from that of giving partial credit to failed rollouts.

We thank the reviewer for this precise question. We trained this baseline at 9B with the same recipe (Final partial-reward, rollout-level below) and evaluated it on ALE-CM:

ALE-CM (9B)Partial creditCreditAdv. norm.S@1S@5P@1P@5
Qwen3.5-9B (base)–––0.250.330.060.06
Sparsenowhole rolloutper rollout0.250.320.060.06
Final partial-reward, rollout-level (requested)yes, λΦ(s_T)whole rolloutper rollout0.260.330.060.06
PPRS, pooled norm.yesG_t, γ = 1per rollout0.280.350.060.06
TEV-9B (ours)yesG_t, γ = 1per turn0.290.400.070.11

This baseline barely improves on Sparse and is clearly worse than PPRS. It stays at the level of Sparse and the base model (S@1 0.26 vs. 0.25; P@1 and P@5 unchanged at 0.06), while TEV-9B reaches 0.29 S@1, 0.40 S@5, and 0.11 P@5. The gain of PPRS therefore comes from turn-level credit assignment, not from partial credit alone. With the same final reward, crediting each turn by its return G_t already raises S@1 and S@5 (PPRS, pooled norm.: 0.28 and 0.35), and normalizing advantages per turn (TEV-9B) further raises S@5 to 0.40 and P@5 from 0.06 to 0.11.

The training curve of this arm, shown in our response to W3 below, gives the same finding: over the 42 training steps its mean training score is 0.24, close to Sparse (0.19) and far below PPRS (0.47). This also matches our intuition. A terminal partial reward gives partial credit only at the end of the rollout. Although failed rollouts no longer all receive 0, many rollouts in a group can still end with the same reward, and such a group gives no learning signal. PPRS avoids this by crediting each turn with the progress it makes, so rollouts that end with the same reward still receive different per-turn advantages.

AuthorsOpenReview ↗

Rebuttal Response (2/n)

W2 and Q2: Could you provide confidence intervals for the ALE-CM results, and results from training with different random seeds? The 27B Pass@5 improvement corresponds to increasing the solved task count from three to five out of 18 tasks.

We agree, and we will report task counts and confidence intervals alongside the rates.

The gain in task counts. TEV-27B's P@5 gain is 3 → 5 of 18 tasks. Both new tasks are in Math & Science (2/8 → 4/8), while ML (0/5) and Coding & System (1/5) are unchanged. S@1, which averages the graded score over 90 rollouts per model, is the less noisy signal, and it rises in all three categories: Math & Science 0.15 → 0.23, ML 0.21 → 0.26, Coding & System 0.74 → 0.82.

Confidence intervals. Paired bootstrap 95% CIs (10,000 resamples of tasks; for S@1 and P@1, the 5 trials within each task are resampled too) give TEV-27B's ALE-CM gains as:

ALE-CM, 27B (18 tasks)Qwen3.8-27BTEV-27BΔ [95% CI]
S@10.330.40+0.067 [+0.001, +0.140]
S@50.430.54+0.114 [−0.008, +0.281]
P@10.120.16+0.033 [−0.022, +0.100]
P@50.170.28+0.111 [+0.000, +0.278]

The S@1 gain excludes zero and the P@5 gain has a lower bound of 0.000. The S@5 and P@1 CIs dip below zero by at most 0.022, as expected with 18 tasks, so no CI points to a regression. We will add these CIs to every Δ row; the table for all benchmarks is in our response to Reviewer Nci5 (W1.1), where 6 of the 16 gains exclude zero.

Training seeds. Every run in the paper uses seed 42. A 27B run takes about four days on 32 B200 GPUs, so we cannot repeat it with new seeds within the discussion period.

AuthorsOpenReview ↗

Rebuttal Response (3/3)

W3 and Q3: For the ablations in Table 5, how many optimization steps were used for each run, and what was the total GPU budget? Could you also provide learning curves against cumulative GPU-hours?

Thank you for this suggestion. We will add the number of optimizer steps and the cumulative GPU-hours of every run to Table 5 caption, together with learning curves of training reward and ALE-CM score against cumulative GPU-hours in our appendix.

All 9B models are trained for 42 steps in this ablation study. Therefore the total GPU budget is the GPU-h / step column in our paper's Table 5 times 42 steps.

Learning curves. We have added these curves to the revised paper, with all new text in red. Figure 6 in Appendix B (page 33, lines 1758–1773) plots the training score, the mean score of the 192 rollouts of each step, against the training step for PPRS and the reward ablations. The paragraph Training curves of the reward ablations (Appendix B, lines 1749–1756) discusses the curves, and the Table 5 caption now points to the figure. Over these 42 steps:

  • PPRS has the highest mean training score: 0.47, against 0.19 for Sparse, 0.24 for a terminal-only reward with partial credit, and 0.37 for Direct reward (additive).
  • PPRS is still improving at step 42, where its smoothed score reaches 0.65. The two terminal-only arms stay below 0.4 throughout.
  • Direct reward catches up with PPRS between steps 17 and 22, but then stays between 0.34 and 0.50 while PPRS keeps rising.

Because each run's GPU-h/step is constant, the step axis converts to cumulative GPU-hours by multiplying by the GPU-h/step column of Table 5. The 42 steps cost 378 H100 GPU-hours for PPRS, against 701 for Sparse and 407 for Direct reward, so PPRS reaches a higher training score at a lower GPU budget.

Additional Results for TB 4.0 generalization

We further report results of TEV-320B on the full TB 4.0 (66 tasks). Our main contribution is the latent-world-first data synthesis pipeline. To further showcase the capability of our curation method, we extended TEV-GYM to new domains such as Software and Security and enlarged existing domains such as Science and ML, yielding TEV-GYM-improved. To train a model that can learn from this high-quality, complex, long-horizon data, we chose GLM-5.3-Flash (a 320B model) and RL-trained it with the same recipe as in our paper. The results of TEV-320B, with confidence intervals, are listed below.

TB 4.0, full (66 tasks)P@1P@5
GLM-5.3-Flash (base)0.358 ± 0.0490.561 ± 0.061
TEV-320B (ours)0.448 ± 0.0540.667 ± 0.059
Δ [95% CI]+0.090 [+0.010, +0.165]+0.106 [0.000, +0.210]
Other models
Fable 50.445 ± 0.0500.682 ± 0.058
GPT-5.6 Sol0.373 ± 0.0490.606 ± 0.061

As the table shows, TEV-320B significantly surpasses its base model on TB 4.0 in both P@1 and P@5. Compared with proprietary models, TEV-320B matches Fable 5 at 30× lower cost (Fable 5 is priced at USD 10 / 50 per 1M input / output tokens, while GLM-5.3-Flash, the base model of TEV-320B, is priced at USD 0.15 / 0.50). These results demonstrate the significance of our data synthesis pipeline and PPRS training method.

We sincerely appreciate your thoughtful comments and hope our responses have addressed your concerns. Please let us know if you have any further questions.

Best, Authors

I think those feedbacks really address my concerns, and I will main my score. Thanks for your hard work.

Reviewer_Nci5OpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the scarcity of long-horizon agent training data and sparse rewards. It builds tasks backward from a hidden ground-truth world: render inputs, inject invertible corruptions, and derive the prompt, verifier, and a per-step progress score Φ from the same world. This yields TEV-Gym (1,170 environments, 7 domains). Φ is used as a dense RL reward (PPRS). Trained Qwen 9B/27B improve on ALE-CM and Terminal-Bench subsets.

Critical strengths
  1. Single-anchor task construction. Prompt, inputs, verifier, and Φ are all derived from one latent world, so they cannot drift apart, and the 8 generation-time gates (exploit probes, solver agreement, role independence) make verification exact. This is a cleaner design than LLM-authored tasks filtered post hoc.

  2. Dense reward comes for free. Because every intermediate state is known, the progress signal needs no reward model or LLM judge, and it handles regressions. It is also why 27B training runs at all: under terminal-only reward it fails to complete a single step.

  3. Cheap and reproducible. Environments are model-free functions of generator and seed (~30 s CPU each), so the corpus can be regenerated exactly and extended without further model calls.

Critical weaknesses
  1. Empirical gains do not support the claims.
  • Evaluation uses 18 (ALE-CM) and 22/18 (TB subsets) tasks with no confidence intervals; P@1 improves by only 0.04 (27B) and 0.01 (9B) on ALE-CM, and by 0.03 on TB 2.1. "Matching GPT-6 Astra and Opus 5" holds only at P@5 (S@1: 0.40 vs. 0.62).
  • Moreover, the generators explicitly target ALE-CM, with training tasks mirroring specific eval tasks (retail ETL, AWS cost optimization, checkpoint consolidation), and the "generalization" TB subsets are selected by the same domain; n-gram decontamination cannot rule out this template-level overlap.
  1. Limited long-horizon scope and missing comparisons.
  • Tasks take a mid-tier frontier model a median of 12 minutes with the workflow given in the prompt, far from the hours-to-days setting motivated; the design restricts tasks to recovering a unique end state, so horizon grows with workload rather than planning depth.
  • The claim that no long-horizon training corpus exists contradicts the paper's own citations: RST [1] synthesizes 37k long-horizon terminal tasks and trains with RL, and T1 [2] uses a dense verifier-count reward. These are recent preprints, so a full comparison is not required, but the "first" claim should be revised and the differences discussed. Evaluating on Long-Horizon-Terminal-Bench [3], which the paper cites, would also better support the long-horizon claim than the hand-selected TB subsets.

  • [1] Li et al., Recursive Synthesis for Long-Horizon Terminal Tasks, arXiv:2608.05466, 2026.
  • [2] Yang et al., T1: Terminal Agent Reinforcement Learning for Long-Horizon Tasks, arXiv:2609.11042, 2026.
  • [3] Li et al., Long-Horizon-Terminal-Bench, arXiv:2607.08964, 2026.
Questions for authors

shown as weakness

8 comments
AuthorsOpenReview ↗

Rebuttal Response (1/n)

Thank you for your careful reading and constructive feedback. We are glad you found that our single-anchor task construction is a cleaner design than LLM-authored tasks filtered post hoc, that dense reward comes for free without a reward model or LLM judge and is why 27B training runs at all, and that our environments are cheap and exactly reproducible.

We address each point below and will update the paper accordingly. We are also happy to engage further during the discussion period to clarify any remaining questions.

Line numbers. All line numbers below refer to the revised PDF, where new text is in red. Pages 1–9 are unchanged, so main-text line numbers are the same as in the submitted version.

W1.1: Evaluation uses 18 (ALE-CM) and 22/18 (TB subsets) tasks with no confidence intervals; P@1 improves is limited. "Matching GPT-6 Astra and Opus 5" holds only at P@5 (S@1: 0.40 vs. 0.62).

Long-horizon benchmarks are small number by nature. Each task takes hours of expert work to author and up to eight hours to evaluate: ALE-CM has 18 tasks, Long-Horizon-Terminal-Bench, which the reviewer suggests, has 46, and the full TB 4.0 has 66. This is why we evaluate on four benchmarks rather than one: ALE-CM (18 tasks), the TB 3.0 and TB 4.0 CM subsets (22 and 18), and the full TB 2.1 (89), close to 150 tasks in total.

We add the Confidence intervals in the below table as requested. To clarify, S@1 and P@1 already average over the trials of each task, 5 for most tasks (Appendix C.2). We now also report a paired bootstrap 95% CI for every gain in the table below (10,000 resamples of tasks; for S@1 and P@1, the trials within each task are resampled too).

Absolute P@1 deltas look small because P@1 is low for every model on these tasks. On ALE-CM the best frontier agent, Claude Fable 5, reaches only 0.29 P@1. The 27B gain of +0.04 is +33% relative, and it moves TEV-27B above Qwen3.8-2.4T-A95B (0.14), a model with 2.4T parameters run in the same harness. On TB 2.1 the 27B base already scores 0.764, so little headroom is left; +0.03 takes it to 0.794, above Claude Opus 4.8 (0.789). The 9B gain on TB 2.1 is +28% relative.

Matching GPT-6 Astra and Opus 5 holds only at P@5 The abstract and the Figure 1 caption both say "matching GPT-6 Astra and Claude Opus 5 at Pass@5". We agree that a clear gap remains at S@1 (0.40 vs. 0.62), and we will state it next to the P@5 claim. But we believe this already a strong enough results and we did not overclaim anything.

The gains are consistent across benchmarks and scales. Every aggregate metric improves, with no regression. We relist base model vs ours models results from our paper's Table 2-4. Each cell gives base ± SE → ours ± SE, then the gain with its paired bootstrap 95% CI. SE is the bootstrap standard error over tasks, from the same 10,000 resamples. The per-model SEs are wide because tasks differ greatly in difficulty, but the paired CIs cancel this shared variation: 6 of the 16 gains exclude zero (ALE-CM 27B S@1, TB 3.0 and TB 4.0 CM P@5, TB 2.1 9B P@1, and TB 2.1 27B P@1 and P@5), 7 more have a lower bound of 0.000, and the remaining 3 dip below zero by at most 0.022, so no CI points to a regression.

Benchmark (tasks)ModelS@1S@5P@1P@5
ALE-CM (18)9B0.25 ± 0.07 → 0.29 ± 0.06, +0.045 [+0.000, +0.134]0.33 ± 0.08 → 0.40 ± 0.09, +0.075 [+0.000, +0.204]0.06 ± 0.05 → 0.07 ± 0.06, +0.014 [+0.000, +0.056]0.06 ± 0.05 → 0.11 ± 0.07, +0.056 [+0.000, +0.167]
ALE-CM (18)27B0.33 ± 0.09 → 0.40 ± 0.09, +0.067 [+0.001, +0.140]0.43 ± 0.10 → 0.54 ± 0.10, +0.114 [−0.008, +0.281]0.12 ± 0.07 → 0.16 ± 0.08, +0.033 [−0.022, +0.100]0.17 ± 0.09 → 0.28 ± 0.10, +0.111 [+0.000, +0.278]
TB 3.0 CM (22)27B= P@1= P@50.09 ± 0.05 → 0.14 ± 0.06, +0.056 [+0.000, +0.136]0.24 ± 0.07 → 0.27 ± 0.08, +0.033 [+0.004, +0.124]
TB 4.0 CM (18)27B= P@1= P@50.04 ± 0.05 → 0.10 ± 0.06, +0.065 [+0.000, +0.184]0.19 ± 0.07 → 0.25 ± 0.08, +0.062 [+0.001, +0.226]
TB 2.1 (89)9B= P@1= P@50.18 ± 0.04 → 0.23 ± 0.04, +0.049 [+0.004, +0.099]0.29 ± 0.05 → 0.35 ± 0.05, +0.056 [−0.011, +0.124]
TB 2.1 (89)27B= P@1= P@50.76 ± 0.03 → 0.79 ± 0.03, +0.031 [+0.003, +0.056]0.86 ± 0.04 → 0.88 ± 0.04, +0.020 [+0.001, +0.054]
AuthorsOpenReview ↗

Rebuttal Response (2/n)

W1.2: The generators explicitly target ALE-CM, with training tasks mirroring specific eval tasks, and the "generalization" TB subsets are selected by the same domain; n-gram decontamination cannot rule out this template-level overlap.

ALE-CM is an in-domain evaluation by design, and the paper says so. §2 (Stage 0) states that "our training data targets ALE-CM". Building training data for the domain of the target benchmark is standard practice: SWE-Smith targets SWE-bench and TMAX targets Terminal-Bench. What our decontamination rules out is instance-level copying. Every generator is our own procedural design that never reproduces a source task's prompt, inputs, answer, or grader text, and no environment shares a 13-gram or an 8-gram with any evaluation task. We agree that n-gram statistics cannot rule out family-level overlap.

TB 2.1 is the full benchmark, with no selection. Both scales improve on all 89 tasks: TEV-9B raises P@1 from 0.178 to 0.227 (+28%), and TEV-27B from 0.764 to 0.794, above Claude Opus 4.8. TB 2.1 is also the evaluation set lexically furthest from our data (median nearest-neighbour prompt Jaccard 0.10).

The TB 3.0/4.0 subsets are selected by domain because TEV-GYM covers one general domain, computing and math. The subsets are not hand-picked: an LLM judged every TB 3.0 task against one fixed criterion (Appendix C), and the subsets track the full benchmarks, with subset and full-set P@1 correlating at Pearson r = 0.96 (TB 3.0) and 0.97 (TB 4.0) across leaderboard models. Lexically, TB tasks are further from TEV-GYM (median nearest-neighbour Jaccard 0.13) than from the other tasks of their own benchmark (0.24–0.25).

We further add results on the full TB 4.0 (66 tasks), with no domain selection. TB 3.0 is now marked as deprecated by the official Terminal-Bench team, and the full TB 4.0 keeps 18 of our 22 TB 3.0 tasks.

TB 4.0, full (P@1 / P@5)Software (18)ML (11)Science (14)Operations (9)Security, Hardware, Media (14)All (66)
Qwen3.8-27B (base)0.06 / 0.140.07 / 0.180.02 / 0.070.01 / 0.060.00 / 0.000.03 / 0.09
TEV-27B (ours)0.09 / 0.200.06 / 0.260.04 / 0.070.00 / 0.000.00 / 0.000.04 / 0.11

TEV-27B improves on the full benchmark at both P@1 and P@5, although 48 of its 66 tasks lie outside our CM subset. The gains concentrate in Software and ML, the categories closest to our computing-and-math corpus. Both models score 0 on Security, Hardware, and Media.

We further report results of TEV-320B on the full TB 4.0 (66 tasks). Our main contribution is the latent-world-first data synthesis pipeline. To further showcase the capability of our curation method, we extended TEV-GYM to new domains such as Software and Security and enlarged existing domains such as Science and ML, yielding TEV-GYM-improved. To train a model that can learn from this high-quality, complex, long-horizon data, we chose GLM-5.3-Flash (a 320B model) and RL-trained it with the same recipe as in our paper. The results of TEV-320B, with confidence intervals, are listed below.

TB 4.0, full (66 tasks)P@1P@5
GLM-5.3-Flash (base)0.358 ± 0.0490.561 ± 0.061
TEV-320B (ours)0.448 ± 0.0540.667 ± 0.059
Δ [95% CI]+0.090 [+0.010, +0.165]+0.106 [0.000, +0.210]
Other models
Fable 50.445 ± 0.0500.682 ± 0.058
GPT-5.6 Sol0.373 ± 0.0490.606 ± 0.061

As the table shows, TEV-320B significantly surpasses its base model on TB 4.0 in both P@1 and P@5. Compared with proprietary models, TEV-320B matches Fable 5 at 30× lower cost (Fable 5 is priced at USD 10 / 50 per 1M input / output tokens, while GLM-5.3-Flash, the base model of TEV-320B, is priced at USD 0.15 / 0.50). These results demonstrate the significance of our data synthesis pipeline and PPRS training method.

N-gram decontamination and template-level overlap. We agree that n-gram statistics cannot rule out template-level overlap with ALE-CM, which is intended because ALE-CM is our in-domain target, while instance-level copying is ruled out because every environment is rendered from a freshly seeded world and a hash check rejects any input tree shared with an evaluation instance. Please refer to our response to Reviewer K2ns (W1 and W2) for details, including the Jaccard similarity of each nearest same-family environment.

AuthorsOpenReview ↗

Rebuttal Response (3/n)

W2.1: Tasks take a mid-tier frontier model a median of 12 minutes with the workflow given in the prompt, far from the hours-to-days setting motivated; the design restricts tasks to recovering a unique end state, so horizon grows with workload rather than planning depth.

The 12.4 minutes compares corpora; it does not measure task length. We report it to compare training corpora under one evaluator (Table 1), and the appendix states that it "is not an estimate of human effort". Wall-clock time depends on the agent: Gemini 3.6 Flash is a fast model, and it fails most of these tasks. It solves 29% of rollouts, nothing at all on 26 of the 92 sampled environments, and nothing on 52 further environments at the larger scale tiers. Under the same evaluator, every prior training corpus takes 0.5–2.0 minutes, so TEV-GYM tasks run 6× longer than the longest prior corpus. For the model we train, they are long: the untrained Qwen3.8-27B exhausts its 65k-token context before submitting in 52% of rollouts.

"Hours to days" describes where the evaluation task is happening, not our training tasks. Our claim is that TEV-GYM is the first training corpus in the long-horizon regime with a dense, verifiable reward, and that training on it improves performance on tasks that do take hours: each TB 3.0/4.0 task requires about eight hours of expert work and model evaluation.

We respectfully disagree that horizon grows with workload rather than planning depth. In TEV-GYM it is the opposite: horizon is defined by dependency depth, and workload is a separate control that we keep apart from it.

  • Horizon is the critical path of the workflow DAG. Each environment's workflow is compiled into a DAG of dependent stages, and "its critical path, the longest chain of dependent stages, defines the task's horizon" (§2.1, L164–165). We quote this critical path, "not the observed rollout length T", because "it does not inflate with an agent's detours" (Appendix A.5, L1139–1140).
  • Depth and workload are two separate controls. The scale tier "grows W ... and lengthens the operational horizon: how much must be read, joined, and written", whereas the workflow DAG, together with the number of corruption classes, "fixes the dependency depth: how many stages must be completed, and in which order, before any output can be right" (§2.2, L203–207). Raising the scale tier grows the median input tree from 17 files (51 KB) to 28 files (586 KB) "while the workflow DAG is unchanged" (Appendix A.4, L1133–1135). More workload therefore does not make a task deeper, and depth does not come from workload.
  • The depth is substantial. A TEV-GYM environment has a median of 6 workflow stages (range 3–12) and 2 injected corruptions (range 1–7) (Appendix A.7, L1185–1186), and one wrong intermediate result invalidates every downstream artifact.

The prompt states the contract, not the solution. Like ALE and TB prompts, ours gives the required outputs, their schemas, and the requirements. It does not say which records, shards, or fields are corrupted. The agent must find the injected corruptions in the inputs and invert them across the dependent stages, so it has to plan which stages to complete, and in which order, before any output can be right.

A unique end state does not fix the path. Exact verification requires a known correct answer, as in any exactly verified benchmark. The path to it is open: Φ depends only on the workspace, so "two rollouts that reach the same workspace by different routes, in a different order of steps or with a different implementation, receive the same reward" (§3, L308–310). We agree that open-ended tasks, where the goal itself must be discovered, are outside this design, and we will state this as a limitation.

Reviewer_K2nsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces procedurally generated agentic environments for computing and math domains. The environment generation pipeline uses a seeded generator to construct the ground-truth and workflows. The paper proposes eight automated gates: reward causality, exploit probes, role independence, solver agreement, progress validity, metamorphic invariance, six-seed diversity, and public-contract hygiene to ensure environment quality. Training uses DPPO. The authors train report gains on an 18-task (computing-and-math) split of Agents’ Last Exam (ALE-CM), on 22/18-task “CM” subsets of Terminal-Bench 3.0/4.0, and on TB 2.1.

Critical strengths
  • Using one state/world to anchor the task, verifier, and reward signal seems reasonable. It also gives state supervision without a PRM or an LLM as judge.
  • Environment gating: especially solver agreement are rigorous ways to gate synthetic environments.
  • Clear motivation for dense reward: made clear in appendix B
Critical weaknesses
  • Decontamination against eval: the overlap with ALE-CM appears to unaccounted for. Training examples from the paper are a retail warehouse ETL (Fig. 3), AWS cost optimization (Example 1, explicitly a training environment), and checkpoint consolidation (A.4). These match very closely to ALE-CM tasks data_pipeline_etl_instance_1 (“Retail ETL Warehouse Cleanup”), cost_optimization_1 (“AWS Cost Optimization Analysis”), and mp_checkpoint_consolidation_v2, as well as TB’s mp-checkpoint-consolidation.

  • The decontamination methodology mentioned in the paper seems mostly lexical so it likely can't detect a generator that just re-implements the eval tasks but with slightly modified seeds or phrasing. Even so, the maximum nearest-neighbour Jaccard from ALE-CM to TEV-GYM (0.38) exceeds the maximum between ALE-CM tasks themselves (0.34).

  • Small effect sizes: ALE-CM has 18 tasks and the reported p@5 gain from 0.17 to 0.28 corresponds to about two additional tasks. Using evidence seems a bit overstated since it's unclear if it's just noise.

  • No comparative baseline: the environment’s graded terminal score as the rollout reward was not reported. Also, no baselines reported with the same harness. Trained Qwen models are evaluated in Vanillux2, while frontier models use other harnesses. TB 2.1 mixes harnesses (Terminus-2 for 27B, Vanillux2 for 9B). Some gains may reflect harness adaptation rather than capability. For example, 52% of base rollouts exhaust context before submitting.

Questions for authors
  • Can you give per-task ALE-CM results for base and trained models? Which fourth Math & Science task does TEV-27B solve at P@5 that no frontier model solves?
  • How many trials were used for your own TB rows (Qwen, TEV, GLM-5.3-Flash)?
  • How sensitive are the results to λ, and how do they compare against graded-terminal-score reward?
  • How do results look on the full TB 3.0/4.0 benchmarks, and on held-out TEV-GYM generators?
7 comments
AuthorsOpenReview ↗

Rebuttal Response (1/n)

Thank you for your careful reading and detailed feedback. We are glad you found that anchoring the task, verifier, and reward signal in one latent world is reasonable and gives state supervision without a PRM or an LLM as judge, that our environment gates, especially solver agreement, are rigorous ways to gate synthetic environments, and that the motivation for dense reward is clear.

We provide detailed explanations and address each point below and will update the paper accordingly. We are also happy to engage further during the discussion period to clarify any remaining questions.

Line numbers. All line numbers below refer to the revised PDF, where new text is in red. Pages 1–9 are unchanged, so main-text line numbers are the same as in the submitted version.

W1: Decontamination against eval: the overlap with ALE-CM appears unaccounted for.

We thank the reviewer for this careful check. We do not have instance-level contamination. Our latent-world-first pipeline generates freshly new training data with controlled difficulty: every environment is rendered from a newly seeded latent world with its own records, injected defects, and answer. Each generator draws on a domain's specifications "for structure and difficulty only, never reproducing its prompt, input, answer, or grader text" (Appendix A.1, L1044–1047), and the diversity gate enforces by hash that no environment shares a public tree with any evaluation instance (Appendix A.9).

ALE-CM is our in-domain evaluation target, and the paper says so: "Our training data targets ALE-CM" (§2.1, L152) and "Our in-domain suite, ALE-CM" (§4.1, L349). Covering the same task families is therefore on purpose in building TEV-GYM. The three environments the reviewer lists (retail warehouse ETL, AWS cost optimization, checkpoint consolidation) are same-family environments of data_pipeline_etl_instance_1, cost_optimization_1, and mp_checkpoint_consolidation_v2. They are not copies: none shares an 8-gram with its ALE-CM task, and their prompt Jaccard similarities to it (0.31, 0.20, and 0.20) are below the 0.34 that two different ALE-CM tasks reach with each other.

Beyond the in-domain suite, we already demonstrate generalization on the TB 3.0 and 4.0 CM subsets (Table 3) and on the full TB 2.1, for which no training data was targeted (Table 4). On the TB side, the overlap the reviewer points to is the checkpoint-consolidation task, which happens to appear in both benchmarks (ALE-CM's mp_checkpoint_consolidation_v2 and TB's mp-checkpoint-consolidation). It is only 1 of our 22 TB tasks, and its nearest TEV-GYM environment reaches a Jaccard of only 0.15.

We will state the in-domain design more prominently in §4.1 and report the nearest same-family environment of every ALE-CM task in Appendix A.9.

AuthorsOpenReview ↗

Rebuttal Response (2/n)

W2: The decontamination methodology seems mostly lexical, and the maximum nearest-neighbour Jaccard from ALE-CM to TEV-GYM (0.38) exceeds the maximum between ALE-CM tasks themselves (0.34).

We agree. N-gram and Jaccard statistics measure text reuse. They cannot detect a re-implementation with new seeds and new wording, and a same-family generator is exactly that. We followed the standard decontamination protocol for training corpora: 13-gram overlap as in GPT-3 (Brown et al., 2020), and 8-gram overlap as in PaLM (Chowdhery et al., 2022) and Tülu 3 (Lambert et al., 2024). Our decontamination is also structural, not only lexical: since every environment is rendered from a freshly seeded world, its input files, injected defects, and answer differ from those of any evaluation instance, and the diversity gate checks this by hash (Appendix A.9). What remains is family-level overlap, which is intended for ALE-CM (W1). Detecting overlap at the level of tasks would need a semantic judge. LLM-based decontamination exists for text corpora: the LLM Decontaminator of Yang et al. (2023) retrieves the top-k training samples closest to each test item by embedding similarity and asks GPT-4 whether each pair is a rephrasing. For agentic environments, a faithful check would have to compare every training environment, including its input files, injected defects, and grader, against every evaluation task, which is far more costly, and to our knowledge no prior agentic training corpus has done so. Such a judge would also not change the conclusion here: it would flag the same-family pairs that we already report and intend for ALE-CM (W1), while the hash check above already rules out shared input trees.

The 0.38 vs. 0.34 gap is expected from this in-domain design. The 18 ALE-CM tasks are all different problems, so their maximum comes from benchmark phrasing shared by unrelated tasks (the 0.34 pair is cfr_game_theory_equilibrium and particle_filter_nonlinear_tracking). TEV-GYM was built to cover the same task families, so an ALE-CM task's nearest TEV-GYM neighbour is a same-family environment: the 0.38 pair is os_log_permission_guard_v1 and our log-permission generator. A margin of 0.04 over two unrelated tasks is far from a copy; two environments of the same generator score 1.00 under the same metric. This confirms family-level overlap, which we state in W1, not instance-level copying.

W3: Small effect sizes: ALE-CM has 18 tasks and the reported p@5 gain from 0.17 to 0.28 corresponds to about two additional tasks.

We agree that ALE-CM is small, and the reviewer's count is right: TEV-27B's P@5 gain is 3 → 5 of 18 tasks, both new tasks in Math & Science (2/8 → 4/8). We will report task counts next to every rate.

ALE-CM is small because long-horizon tasks are scarce, not by our choice. Long-horizon tasks exist only in human-authored benchmarks (§1, L76–78). Each task takes hours of expert work to author and up to eight hours to evaluate (§4.1, L348–349), so every long-horizon benchmark is small: the whole Agents' Last Exam has 152 tasks, only 23 of them in computing and math (18 run in a Linux container, Appendix C.2), Long-Horizon-Terminal-Bench has 46, and the full TB 4.0 has 66. This is why we do not rely on ALE-CM alone: we also evaluate on the TB 3.0 and 4.0 CM subsets (22 and 18 tasks) and on the full TB 2.1 (89 tasks), 147 tasks across four benchmarks.

Further evidence that the gains are not noise:

  • Consistent direction. Across both scales and all four benchmarks, every one of the 16 aggregate metrics improves (Tables 2–4), with no regression.
  • S@1, the less noisy signal. S@1 averages the graded score over 90 rollouts per model instead of counting solved tasks. For TEV-27B it rises in all three categories: Math & Science 0.15 → 0.23, ML 0.21 → 0.26, Coding & System 0.74 → 0.82.
  • Every 27B checkpoint improves. Across the eight 27B checkpoints we evaluated on ALE-CM, S@5 is above the base model at every one (0.46–0.54 vs. 0.43), and P@5 at seven of the eight (0.22–0.28 vs. 0.17).
  • Same-data controls do not move. Training the 9B model on the same TEV-GYM tasks with terminal-only reward leaves ALE-CM at the base level (S@1 0.25 → 0.25, P@5 0.06 → 0.06; Table 5), whereas PPRS reaches 0.29 and 0.11.
  • Confidence intervals. We computed paired bootstrap 95% CIs for every gain (10,000 resamples of tasks; for S@1 and P@1, the trials within each task are resampled too). 6 of the 16 gains exclude zero, including TEV-27B's S@1 gain on ALE-CM, +0.067 [+0.001, +0.140], its P@5 gains on the TB 3.0 and 4.0 CM subsets, and TEV-9B's P@1 gain on the full TB 2.1, +0.049 [+0.004, +0.099]. Seven more have a lower bound of 0.000, and the remaining three dip below zero by at most 0.022, so no CI points to a regression. The full table is in our response to Reviewer Nci5 (W1.1), and we will add these CIs to every Δ row.
AuthorsOpenReview ↗

Rebuttal Response (3/n)

W4.1: No comparative baseline: the environment's graded terminal score as the rollout reward was not reported.

We understand this baseline as using the graded score of the submitted workspace as a single reward per rollout, i.e., partial credit at the end but no PPRS. We argue this is a weak baseline that does not test what our method contributes, and we provide the new baseline experiment results in the table below:

  • A terminal grade exists only for rollouts that submit. A rollout that exhausts its turn or token budget is never graded terminally (§3). The untrained Qwen3.8-27B fails to submit in 52% of its rollouts on our environments (Appendix A, L1212–1214), so a graded terminal reward still gives most rollouts of a hard task the same zero, and their groups still carry no gradient. PPRS grades the workspace after every turn, submitted or not; this starvation is exactly the problem dense reward addresses (§3.1).
  • Empirically, it barely improves on sparse reward. We trained this arm (Partial, terminal only) at 9B with the same recipe. Over the 42 training steps its mean training score is 0.24, close to Sparse (0.19) and far below PPRS (0.47); its smoothed score stays below 0.4 throughout, while PPRS is still rising at the end of the window, where its smoothed score reaches 0.65. The curves are in the new Figure 6 in Appendix B of the revised paper (page 33, L1758–1773), discussed in the paragraph Training curves of the reward ablations (Appendix B, L1749–1756); all new text is in red.
  • It also barely improves evaluation. We evaluated this arm on ALE-CM. It stays at the level of Sparse (S@1 0.26 vs. 0.25; P@1 and P@5 unchanged at 0.06), while PPRS reaches 0.29 S@1 and 0.11 P@5:
ALE-CM (9B)Partial creditCreditAdv. norm.S@1S@5P@1P@5
Qwen3.5-9B (base)–––0.250.330.060.06
Sparsenowhole rolloutper rollout0.250.320.060.06
Final partial-reward, rollout-level (requested)yes, λΦ(s_T)whole rolloutper rollout0.260.330.060.06
PPRS, pooled norm.yesG_t, γ = 1per rollout0.280.350.060.06
TEV-9B (ours)yesG_t, γ = 1per turn0.290.400.070.11

These evaluation results agree with the training curve above, and they match our intuition. A terminal partial reward gives partial credit only at the end of the rollout. Although failed rollouts no longer all receive 0, many rollouts in a group can still end with the same reward, and such a group gives no learning signal. PPRS avoids this by crediting each turn with the progress it makes, so rollouts that end with the same reward still receive different per-turn advantages.

W4.2: Also, no baselines reported with the same harness.

We do report baselines in the same harness. Table 2 evaluates the two strongest open models in Vanillux2, the harness of our Qwen models. TEV-27B outperforms both Qwen3.8-2.4T-A95B, a model with 2.4T parameters, and DeepSeek-V4-pro at S@1, S@5, and P@1 (P@1 0.16 vs. 0.14 and 0.11), and ties Qwen3.8-2.4T-A95B at P@5 (0.28).

Frontier models use their default and best harnesses. They run in Codex, Claude Code, and Gemini CLI, "the setting most favourable to them" (Appendix C.2, L1810–1811). These harnesses are built for their models and are stronger than our harness, so the comparison is conservative for TEV. Our Qwen models run in Vanillux2 because we also trained in it, following TMAX (Ivison et al., 2026). We believe these comparisons demonstrate the significance of our results.

On "TB 2.1 mixes harnesses (Terminus-2 for 27B, Vanillux2 for 9B)", we agree this is a small caveat that we should have stated more clearly, and we will. However, while the harness differs between model scales, it never differs within a comparison. Each Δ row of Table 4 compares a base model with its own trained checkpoint in the same harness:

  • 9B: Qwen3.5-9B and TEV-9B both run in Vanillux2 (P@1 0.178 → 0.227, P@5 0.292 → 0.348).
  • 27B: Qwen3.8-27B and TEV-27B both run in Terminus-2 (P@1 0.764 → 0.794, P@5 0.864 → 0.884).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all three reviewers for their constructive feedback and are encouraged that they find:

  • Our environment construction is the strongest part of the paper (RBSH): deriving task inputs, verifier, and progress scorer from one latent world keeps them consistent, gives state supervision without a PRM or LLM judge, and is cleaner than post-hoc filtered LLM tasks (RBSH, K2ns, Nci5).
  • Our gates, especially solver agreement, are rigorous and make verification exact; reference solvers and exploit checks give confidence in reward quality; and the corpus is cheap and exactly reproducible (K2ns, Nci5, RBSH).
  • Our progress reward addresses a practical training problem with clear motivation: it comes for free (no reward model or LLM judge) and is why 27B training runs at all (RBSH, K2ns, Nci5).
  • Our evaluation spans 9B and 27B models, TB 2.1 gives evidence that the benefit extends beyond ALE-CM, and ablations clarify how environment and reward choices affect results (RBSH).
  • The paper is clear and easy to follow (K2ns, Nci5).

Below we summarize the main shared concerns (details in our individual responses):

1. Benchmark size (K2ns, Nci5, RBSH). Long-horizon tasks take hours of expert work to author and up to 8h to evaluate, so benchmarks are naturally small (ALE-CM: 18 tasks; full TB 4.0: 66). We therefore evaluate on four benchmarks (147 tasks), where all 16 aggregate metrics improve with no regression. As requested, we added two evaluations on full TB 4.0 (66 tasks, no domain selection):

  • TEV-27B improves P@1 (0.03→0.04) and P@5 (0.09→0.11), despite 73% of tasks lying outside CM.
  • TEV-320B (GLM-5.3-Flash trained with our recipe on TEV-GYM extended to new domains) raises P@1 from 0.358 to 0.448, on par with Claude Fable 5 (0.445) and above GPT-5.6 Sol (0.373).

2. Confidence intervals (K2ns, Nci5, RBSH). We computed paired bootstrap 95% CIs for all 16 gains (10,000 task resamples, plus within-task trial resampling for S@1/P@1). Pairing each trained model with its base on identical resampled tasks cancels task-difficulty variance, showing consistent gains:

  • 6/16 gains are statistically significant (CIs exclude 0): TEV-27B's ALE-CM S@1 (+0.067 [+0.001, +0.140]), P@5 on both TB CM subsets, and 3/4 gains on full TB 2.1.
  • 7 more have a 0.000 lower bound (trained ≥ base in ≥97.5% of resamples).
  • The remaining 3 dip below 0 by at most 0.022—no CI points to a regression.

3. Decontamination (K2ns, Nci5). There is no instance-level contamination: each environment comes from a freshly seeded latent world (own records, injected defects, answer); hash checks reject shared input trees; and none share an 8- or 13-gram with any eval task. Family-level overlap with ALE-CM is by design ("Our training data targets ALE-CM", §2.1), like SWE-Smith for SWE-bench and TMAX for Terminal-Bench. Still, nearest same-family environments for the three named ALE-CM tasks have prompt Jaccard similarities of only 0.31, 0.20, and 0.20 (vs. 0.34 between two distinct ALE-CM tasks). Generalization is demonstrated on untargeted benchmarks: TB 3.0-CM, TB 4.0-CM, and full TB 2.1.

Discussion

As discussion nearly closes, only two of three reviewers have replied.

Reviewer RBSH. We are glad Reviewer RBSH confirmed that "those feedbacks really address my concern" and thank them for acknowledging our hard work during the rebuttal.

Reviewer Nci5. We are glad our responses fully clarified the long-horizon scope and addressed their three remaining points:

  1. Gains beyond CM. Our paper and TEV-GYM target computing and math (CM). Full TB 4.0 (added by request) goes beyond this scope (48/66 tasks non-CM), yet TEV-27B still improves P@1 (0.03→0.04) and P@5 (0.09→0.11). Our paper already demonstrates significant improvement across all 4 benchmarks.
  2. ALE-CM P@1 significance. P@1 is coarse on a small benchmark: on 14/18 tasks, base and trained 27B match across all rollouts, so the gain rests on 3 more fully solved rollouts out of 90. S@1 shows a significant ALE-CM gain (+0.067 [+0.001, +0.140]), alongside significant gains on TB 3.0-CM, TB 4.0-CM, and full TB 2.1.
  3. TEV-320B. TEV-GYM-improved is our own dataset (same latent-world-first pipeline, extended to new domains like Software/Security), and TEV-320B uses the same recipe. We use a 320B base (GLM-5.3-Flash) only because full TB 4.0's complex long-horizon tasks need a strong base; trained on our data, it raises P@1 from 0.358 to 0.448, on par with Claude Fable 5 (0.445) and above GPT-5.6 Sol (0.373). A separate appendix will report TEV-320B's dataset and base model.

We welcome further discussion or requests for clarification, and thank the reviewers and AC for their time.

Kind regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.