- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Long-Horizon-Terminal-Bench 2.0, a benchmark of 77 terminal tasks across 13 domains. It evaluates 15 models with a shared agent framework and a three-hour budget per task. Its core design is subtask-based grading: deterministic verifiers award partial credit for intermediate objectives, so unsuccessful runs can be told apart beyond pass/fail. The paper uses these scores to analyze rankings, progress over time, and failure patterns.
The problem is relevant and the benchmark reflects substantial effort. However, the partial-credit scores are not validated as measures of functional progress. Published reference solutions also conflict with the stated full-score guarantee, several statistics are inconsistent, and the long-horizon conclusions rest on a single time budget.
Critical strengths
- Targets a real gap in outcome-only evaluation. Failed long-running agents can leave artifacts of very different quality, and binary success rates hide that.
- Covers diverse technical domains with containerized environments, a shared agent framework, and programmatic verifiers, which gives a sound basis for reproducible comparison.
- The checkpoint and termination analyses go beyond a leaderboard. With validated scoring, they could show where extended execution helps, stalls, or loses earlier gains.
Critical weaknesses
1. Partial credit is not validated as functional progress. Intermediate rewards and subgoal-based scoring are established ideas, as the related work acknowledges. The contribution therefore rests on how well subtasks and weights are designed and validated. The paper does not explain how they were chosen. It also does not show that they track functional completion better than simple alternatives such as test pass rates or native task metrics (e.g., levels completed or simulator reward in game tasks). Finer scores and fewer ties do not by themselves make a ranking more valid.
The shared reconstruction-task dense_reward.py illustrates the risk. Its six-stage ladder awards 0.5 for three checks: an editable artifact exists, reconstruct.py exists without copying the public manifest, and the private oracle is available. None of these requires the candidate to run or pass any semantic assertion. A program that raises an exception immediately can therefore score 0.5, and oracle availability is a property of the evaluation setup, not of the agent. This undercuts the paper's reading of high-scoring failures as near-completions.
2. Reference solutions contradict the full-score guarantee. Section 2.3 states that the official gold solution must achieve 1.0 on the full hidden suite. However:
- The DuckDB solution describes itself as a no-op that performs no optimization, yields a ~1.0× baseline, and expects partial credit. Its only change adds a comment to
CMakeLists.txt. - The OpenROAD solution states that it "does NOT yet reach DRC=0." The grading spec assigns a weight of 0.15 to zero DRC violations.
This does not show that full credit is unattainable, but the published solutions do not support the guarantee. Please clarify whether they are outdated, and provide per-task reference scores and verification logs for the evaluated version.
3. Reported statistics are inconsistent.
- Overall: 1,145 scored runs with 192 passes implies 953 unresolved runs. Appendix B.1 reports 745 timeouts plus 209 other unresolved runs, which is 954. Also, 15 models × 77 tasks = 1,155, so 10 runs are unaccounted for.
- Kimi K3: 17 passes (Fig. 3) plus 61 unresolved (Fig. 8) is 78, exceeding the 77 tasks.
- GPT-5.6-sol: 32 + 23 + 20 = 75. How the remaining two tasks enter denominators and mean reward is unclear.
- GPT-6-astra: The figures report 37 passes, 33 timeouts (mean 0.27), and 7 other failures (mean 0.51). The overall mean is then at most (37 + 33×0.27 + 7×0.51)/77 ≈ 0.643, or below 0.646 allowing for rounding, not the reported 0.68.
These may stem from different scoring versions, exclusions, or run-selection rules. A unified per-model, per-task results table is needed.
4. Long-horizon conclusions depend on a single, cost-driven budget. The three-hour limit was chosen for cost reasons (Section 3.1), yet the prevalence of timeouts is used to argue that sustained execution is the main bottleneck. One budget cannot tell whether more time would bring completion, continued incremental progress, or redundant activity.
A matched 3h vs. 6h comparison would test this claim directly. It should cover at least one leading and one mid-tier model, with everything else held fixed. It should report strict and soft success, functional partial credit, timeout rates, and ranking changes.1. How were subtasks and weights chosen, and were they validated against expert judgments of functional completion? 2. What do nonfunctional baselines score? Do the rankings and conclusions hold once nonfunctional and infrastructure credit is removed? 3. Can you provide per-task logs showing that the reference solutions reach 1.0 on the hidden suites? How do the DuckDB and OpenROAD cases fit with Section 2.3? 4. Can you reconcile the run counts and the GPT-6-astra mean? Please specify the scoring version, denominator, and run-selection policy behind each figure. 5. How do completion rates, partial progress, and rankings change under a six-hour budget?
Questions for authors
- How were subtasks and weights chosen, and were they validated against expert judgments of functional completion?
- What do nonfunctional baselines score? Do the rankings and conclusions hold once nonfunctional and infrastructure credit is removed?
- Can you provide per-task logs showing that the reference solutions reach 1.0 on the hidden suites? How do the DuckDB and OpenROAD cases fit with Section 2.3?
- Can you reconcile the run counts and the GPT-6-astra mean? Please specify the scoring version, denominator, and run-selection policy behind each figure.
- How do completion rates, partial progress, and rankings change under a six-hour budget?
9 comments
Response to Reference solutions contradict the full-score guarantee
The supplied references include functional baselines and approximate solutions that do not necessarily satisfy every performance or engineering target. Section 2.3 overclaimed generalized the full-score requirement to all references; we will revise it to distinguish these roles explicitly.
Our audit executed all 47 supplied workflow references, of which 31 achieve exactly 1.0. The DuckDB example illustrates the distinction between functional correctness and optimization: its reference passes all 22 correctness queries, but achieves 0.993× speedup against the 1.3× target, yielding 0.763955. It therefore supplies a correctness baseline without demonstrating the requested performance improvement. Similarly, the OpenROAD reference scores 0.84, satisfying timing, area, and DRC requirements while leaving LVS and power requirements unmet.
These references remain useful for demonstrating achieved requirements and identifying the remaining work, but they do not certify full-task completion. We will report each reference’s measured score, passed and failed requirements, and verification logs, and distinguish partial solutions from execution or packaging failures. This preserves the benchmark’s substantive task requirements. We will clarify that supplied references may demonstrate functional correctness or partial completion without meeting every engineering or optimization target. Their failure to achieve full credit does not establish that those targets are unattainable.
Response to Reconciling Run Counts, Denominators, and Mean Scores
We thank the reviewer for identifying these inconsistencies. We reconstructed a unified per-model, per-task table from a single frozen evaluation snapshot, recording the selected attempt, native score, terminal status, inclusion decision, and denominator. These results reconcile the original three-hour evaluation; they are separate from the corrected reconstruction-score analysis.
Across the historical 15-model evaluation, 1,155 task slots comprise 1,093 included outcomes and 62 exclusions. Included outcomes partition into 189 scores ≥0.95, 733 unresolved budget timeouts, and 171 other unresolved outcomes. Outcomes scoring ≥0.95 are classified first; only the remaining included outcomes are divided into timeout and other unresolved categories, preventing double counting.
Our reconciliation uses an explicit selection policy: one indexed attempt per model/task, with no best-retry selection. Valid zero scores and expected budget timeouts remain included; missing scores, unexpected execution errors, and failed verifier prerequisites are excluded with documented reasons rather than assigned artificial zeros. The audit identified 32 model–task score differences between archived tables, demonstrating the need for a single source for all reported figures. The unified table now accounts for all 1,155 task slots and makes every denominator explicit. We will use it to regenerate the affected figures and provide anonymized task-level records so reviewers can verify the reported counts and means directly.
| Model | Included | Excluded | Score ≥0.95 | Unresolved timeout | Other unresolved | Mean native score |
|---|---|---|---|---|---|---|
| DeepSeek-V4-Flash | 76 | 1 | 9 | 52 | 15 | 0.5127 |
| DeepSeek-V4-Pro | 75 | 2 | 9 | 49 | 17 | 0.5297 |
| Doubao-Seed-2.1-Pro | 47 | 3 | 8 | 32 | 7 | 0.4789 |
| Gemini-3.1-Pro | 66 | 1 | 5 | 56 | 5 | 0.4565 |
| Gemini-3.8-Flash | 71 | 6 | 15 | 56 | 0 | 0.5142 |
| GLM-5.2 | 75 | 2 | 8 | 44 | 23 | 0.5034 |
| GLM-5.3 | 75 | 2 | 11 | 59 | 5 | 0.5326 |
| GPT-5.6-Sol | 74 | 3 | 33 | 21 | 20 | 0.6949 |
| GPT-6-astra | 76 | 1 | 36 | 33 | 7 | 0.6962 |
| Grok-4.6 | 77 | 0 | 10 | 58 | 9 | 0.4198 |
| Hunyuan HY4 | 75 | 2 | 15 | 58 | 2 | 0.5332 |
| Kimi K2.6 | 77 | 0 | 2 | 61 | 14 | 0.4202 |
| Kimi K3 | 76 | 1 | 16 | 44 | 16 | 0.5628 |
| MiniMax-M3 | 76 | 1 | 5 | 47 | 24 | 0.4664 |
| Qwen-3.8-Max | 77 | 0 | 7 | 63 | 7 | 0.4200 |
| Total | 1,093 | 62 | 189 | 733 | 171 | — |
Response to Partial credit is not validated as functional progress
We agree that finer scores and fewer ties do not establish functional validity. The reconstruction ladder’s credit for artifact creation and oracle availability was a concrete implementation problem, and we have replaced it across all 30 reconstruction tasks.
The revised primary reward is the equally weighted fraction of fixed hidden scenarios completed successfully. Each scenario requires successful candidate execution and every expected native semantic assertion to pass. Artifact creation, entrypoint presence, and oracle availability earn no credit; oracle failures invalidate an evaluation. Evaluation continues through later scenarios after candidate failures. Native assertion-pass fractions are reported separately, distinguishing partial satisfaction of checks from completing a scenario.
For construction, the 47 workflow tasks were reviewed by domain experts and the authors, while the 30 reconstruction tasks used DeepSeek validation of a construction process based on reference artifacts and held-out behavioral evaluation. We will state this split explicitly. This task-review process is distinct from validating numerical scores against independent expert judgments of model outputs.
The weights therefore have an explicit interpretation: equal contribution from each specified scenario, without infrastructure-stage credit. This is a transparent native completion measure, rather than a claim of expert-calibrated quality. We will document the actual task-specific rubrics and their construction more explicitly. The 47 workflow graders remain outside this reconstruction correction.
Additionally, we removedthe infrastructure-credit across all 30 reconstruction graders and reevaluated recovered submissions. The revised score measures the fraction of hidden scenarios completed successfully, with no credit for artifact creation, entrypoint presence, or oracle availability. We confirmed that 120 nonfunctional controls score 0, 30 privileged oracle controls score 1, and 60 controlled-coverage cases receive exactly their known scenario-completion fractions. Among 332 accepted trajectory-derived replays across 13 models, 149 candidates previously scoring at least 0.5 complete no hidden scenario and now score 0. Each accepted replay first reproduced the archived outcomes under the original grader.
We recomputed rankings on 44 tasks valid for all 13 models, comprising 15 reconstruction tasks and 29 unchanged workflow tasks. GPT-5.6-Sol and GPT-6-astra remain first and second compared with the original scoring on this same subset, even when unresolved reconstruction outcomes retain conservative score bounds. Some other rank intervals overlap. Thus, the leading positions persist after removing infrastructure credit on this subset.