acceptodds
Under review as a conference paper at ICLR 2027

Long-horizon-terminal-bench 2.0: Testing the limits of agents on long-horizon terminal tasks with dense reward-based grading

Abstract

AI agents have become increasingly capable of autonomously completing short and well-specified terminal tasks. However, existing terminal benchmarks either focus on short, simple problems that finish within a few minutes or on hard tasks evaluated only by their final outcome. This setup overlooks intermediate progress and partial solutions, leading to sparse reward signals and an incomplete picture of agent capability. We curate Long-Horizon-Terminal-Bench 2.0 (LHTB 2.0), a diverse set of 77 long-horizon tasks grouped into 13 domains. Each task follows a Terminal-Bench-style Harbor setup with a reference solution or simulation engine and is further decomposed into fine-grained graded subtasks. We design dense intermediate rewards and partial credit, allowing evaluation to capture not only whether an agent reaches the final goal, but also how far it progresses on long-horizon workflows. We evaluate 15 frontier models on LHTB 2.0. The strongest current system, GPT-6-astra, reaches the soft-pass threshold (R ≥ 0.95) on 37/77 tasks (48.1%) and achieves a dense mean reward of 0.68, followed by GPT-5.6-sol at 32/77 tasks (41.6%) and a mean reward of 0.67. Across 1,145 scored task runs, only 16.8% pass the soft-pass threshold, while 15.4% make no meaningful progress (R < 0.05) and 67.9% achieve partial progress. These results reveal substantial headroom for improvement. We further analyze common failure modes and error patterns of models and agents on long-horizon terminal tasks.

Primary area:
datasets and benchmarks
Keywords:
Terminal Tasks, Frontier Models, Long-horizon tasks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HsarOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Long-Horizon-Terminal-Bench 2.0, a benchmark of 77 terminal tasks across 13 domains. It evaluates 15 models with a shared agent framework and a three-hour budget per task. Its core design is subtask-based grading: deterministic verifiers award partial credit for intermediate objectives, so unsuccessful runs can be told apart beyond pass/fail. The paper uses these scores to analyze rankings, progress over time, and failure patterns.

The problem is relevant and the benchmark reflects substantial effort. However, the partial-credit scores are not validated as measures of functional progress. Published reference solutions also conflict with the stated full-score guarantee, several statistics are inconsistent, and the long-horizon conclusions rest on a single time budget.

Critical strengths
  • Targets a real gap in outcome-only evaluation. Failed long-running agents can leave artifacts of very different quality, and binary success rates hide that.
  • Covers diverse technical domains with containerized environments, a shared agent framework, and programmatic verifiers, which gives a sound basis for reproducible comparison.
  • The checkpoint and termination analyses go beyond a leaderboard. With validated scoring, they could show where extended execution helps, stalls, or loses earlier gains.
Critical weaknesses

1. Partial credit is not validated as functional progress. Intermediate rewards and subgoal-based scoring are established ideas, as the related work acknowledges. The contribution therefore rests on how well subtasks and weights are designed and validated. The paper does not explain how they were chosen. It also does not show that they track functional completion better than simple alternatives such as test pass rates or native task metrics (e.g., levels completed or simulator reward in game tasks). Finer scores and fewer ties do not by themselves make a ranking more valid.

The shared reconstruction-task dense_reward.py illustrates the risk. Its six-stage ladder awards 0.5 for three checks: an editable artifact exists, reconstruct.py exists without copying the public manifest, and the private oracle is available. None of these requires the candidate to run or pass any semantic assertion. A program that raises an exception immediately can therefore score 0.5, and oracle availability is a property of the evaluation setup, not of the agent. This undercuts the paper's reading of high-scoring failures as near-completions.

2. Reference solutions contradict the full-score guarantee. Section 2.3 states that the official gold solution must achieve 1.0 on the full hidden suite. However:

  • The DuckDB solution describes itself as a no-op that performs no optimization, yields a ~1.0× baseline, and expects partial credit. Its only change adds a comment to CMakeLists.txt.
  • The OpenROAD solution states that it "does NOT yet reach DRC=0." The grading spec assigns a weight of 0.15 to zero DRC violations.

This does not show that full credit is unattainable, but the published solutions do not support the guarantee. Please clarify whether they are outdated, and provide per-task reference scores and verification logs for the evaluated version.

3. Reported statistics are inconsistent.

  • Overall: 1,145 scored runs with 192 passes implies 953 unresolved runs. Appendix B.1 reports 745 timeouts plus 209 other unresolved runs, which is 954. Also, 15 models × 77 tasks = 1,155, so 10 runs are unaccounted for.
  • Kimi K3: 17 passes (Fig. 3) plus 61 unresolved (Fig. 8) is 78, exceeding the 77 tasks.
  • GPT-5.6-sol: 32 + 23 + 20 = 75. How the remaining two tasks enter denominators and mean reward is unclear.
  • GPT-6-astra: The figures report 37 passes, 33 timeouts (mean 0.27), and 7 other failures (mean 0.51). The overall mean is then at most (37 + 33×0.27 + 7×0.51)/77 ≈ 0.643, or below 0.646 allowing for rounding, not the reported 0.68.

These may stem from different scoring versions, exclusions, or run-selection rules. A unified per-model, per-task results table is needed.

4. Long-horizon conclusions depend on a single, cost-driven budget. The three-hour limit was chosen for cost reasons (Section 3.1), yet the prevalence of timeouts is used to argue that sustained execution is the main bottleneck. One budget cannot tell whether more time would bring completion, continued incremental progress, or redundant activity.

A matched 3h vs. 6h comparison would test this claim directly. It should cover at least one leading and one mid-tier model, with everything else held fixed. It should report strict and soft success, functional partial credit, timeout rates, and ranking changes.1. How were subtasks and weights chosen, and were they validated against expert judgments of functional completion? 2. What do nonfunctional baselines score? Do the rankings and conclusions hold once nonfunctional and infrastructure credit is removed? 3. Can you provide per-task logs showing that the reference solutions reach 1.0 on the hidden suites? How do the DuckDB and OpenROAD cases fit with Section 2.3? 4. Can you reconcile the run counts and the GPT-6-astra mean? Please specify the scoring version, denominator, and run-selection policy behind each figure. 5. How do completion rates, partial progress, and rankings change under a six-hour budget?

Questions for authors
  1. How were subtasks and weights chosen, and were they validated against expert judgments of functional completion?
  2. What do nonfunctional baselines score? Do the rankings and conclusions hold once nonfunctional and infrastructure credit is removed?
  3. Can you provide per-task logs showing that the reference solutions reach 1.0 on the hidden suites? How do the DuckDB and OpenROAD cases fit with Section 2.3?
  4. Can you reconcile the run counts and the GPT-6-astra mean? Please specify the scoring version, denominator, and run-selection policy behind each figure.
  5. How do completion rates, partial progress, and rankings change under a six-hour budget?
9 comments
AuthorsOpenReview ↗

Response to Partial credit is not validated as functional progress

We agree that finer scores and fewer ties do not establish functional validity. The reconstruction ladder’s credit for artifact creation and oracle availability was a concrete implementation problem, and we have replaced it across all 30 reconstruction tasks.

The revised primary reward is the equally weighted fraction of fixed hidden scenarios completed successfully. Each scenario requires successful candidate execution and every expected native semantic assertion to pass. Artifact creation, entrypoint presence, and oracle availability earn no credit; oracle failures invalidate an evaluation. Evaluation continues through later scenarios after candidate failures. Native assertion-pass fractions are reported separately, distinguishing partial satisfaction of checks from completing a scenario.

For construction, the 47 workflow tasks were reviewed by domain experts and the authors, while the 30 reconstruction tasks used DeepSeek validation of a construction process based on reference artifacts and held-out behavioral evaluation. We will state this split explicitly. This task-review process is distinct from validating numerical scores against independent expert judgments of model outputs.

The weights therefore have an explicit interpretation: equal contribution from each specified scenario, without infrastructure-stage credit. This is a transparent native completion measure, rather than a claim of expert-calibrated quality. We will document the actual task-specific rubrics and their construction more explicitly. The 47 workflow graders remain outside this reconstruction correction.

Additionally, we removedthe infrastructure-credit across all 30 reconstruction graders and reevaluated recovered submissions. The revised score measures the fraction of hidden scenarios completed successfully, with no credit for artifact creation, entrypoint presence, or oracle availability. We confirmed that 120 nonfunctional controls score 0, 30 privileged oracle controls score 1, and 60 controlled-coverage cases receive exactly their known scenario-completion fractions. Among 332 accepted trajectory-derived replays across 13 models, 149 candidates previously scoring at least 0.5 complete no hidden scenario and now score 0. Each accepted replay first reproduced the archived outcomes under the original grader.

We recomputed rankings on 44 tasks valid for all 13 models, comprising 15 reconstruction tasks and 29 unchanged workflow tasks. GPT-5.6-Sol and GPT-6-astra remain first and second compared with the original scoring on this same subset, even when unresolved reconstruction outcomes retain conservative score bounds. Some other rank intervals overlap. Thus, the leading positions persist after removing infrastructure credit on this subset.

New RankModelNew mean score
1GPT-5.6-Sol0.7087
2GPT-6-astra0.6562
3Kimi K30.5053–0.5318
4Hunyuan HY40.4962
5–6GLM-5.30.4601–0.4866
5–6DeepSeek-V4-Flash0.4523–0.4637
7–8DeepSeek-V4-Pro0.4194–0.4422
7–11Doubao-Seed-2.1-Pro0.3690–0.4296
8–9GLM-5.20.4097
9–11Grok-4.60.3701
9–11MiniMax-M30.3468–0.3733
12Qwen-3.8-Max0.3451
13Kimi K2.60.2525–0.2639
AuthorsOpenReview ↗

Response to Reference solutions contradict the full-score guarantee

The supplied references include functional baselines and approximate solutions that do not necessarily satisfy every performance or engineering target. Section 2.3 overclaimed generalized the full-score requirement to all references; we will revise it to distinguish these roles explicitly.

Our audit executed all 47 supplied workflow references, of which 31 achieve exactly 1.0. The DuckDB example illustrates the distinction between functional correctness and optimization: its reference passes all 22 correctness queries, but achieves 0.993× speedup against the 1.3× target, yielding 0.763955. It therefore supplies a correctness baseline without demonstrating the requested performance improvement. Similarly, the OpenROAD reference scores 0.84, satisfying timing, area, and DRC requirements while leaving LVS and power requirements unmet.

These references remain useful for demonstrating achieved requirements and identifying the remaining work, but they do not certify full-task completion. We will report each reference’s measured score, passed and failed requirements, and verification logs, and distinguish partial solutions from execution or packaging failures. This preserves the benchmark’s substantive task requirements. We will clarify that supplied references may demonstrate functional correctness or partial completion without meeting every engineering or optimization target. Their failure to achieve full credit does not establish that those targets are unattainable.

AuthorsOpenReview ↗

Response to Reconciling Run Counts, Denominators, and Mean Scores

We thank the reviewer for identifying these inconsistencies. We reconstructed a unified per-model, per-task table from a single frozen evaluation snapshot, recording the selected attempt, native score, terminal status, inclusion decision, and denominator. These results reconcile the original three-hour evaluation; they are separate from the corrected reconstruction-score analysis.

Across the historical 15-model evaluation, 1,155 task slots comprise 1,093 included outcomes and 62 exclusions. Included outcomes partition into 189 scores ≥0.95, 733 unresolved budget timeouts, and 171 other unresolved outcomes. Outcomes scoring ≥0.95 are classified first; only the remaining included outcomes are divided into timeout and other unresolved categories, preventing double counting.

Our reconciliation uses an explicit selection policy: one indexed attempt per model/task, with no best-retry selection. Valid zero scores and expected budget timeouts remain included; missing scores, unexpected execution errors, and failed verifier prerequisites are excluded with documented reasons rather than assigned artificial zeros. The audit identified 32 model–task score differences between archived tables, demonstrating the need for a single source for all reported figures. The unified table now accounts for all 1,155 task slots and makes every denominator explicit. We will use it to regenerate the affected figures and provide anonymized task-level records so reviewers can verify the reported counts and means directly.

ModelIncludedExcludedScore ≥0.95Unresolved timeoutOther unresolvedMean native score
DeepSeek-V4-Flash761952150.5127
DeepSeek-V4-Pro752949170.5297
Doubao-Seed-2.1-Pro47383270.4789
Gemini-3.1-Pro66155650.4565
Gemini-3.8-Flash716155600.5142
GLM-5.2752844230.5034
GLM-5.3752115950.5326
GPT-5.6-Sol7433321200.6949
GPT-6-astra761363370.6962
Grok-4.6770105890.4198
Hunyuan HY4752155820.5332
Kimi K2.6770261140.4202
Kimi K37611644160.5628
MiniMax-M3761547240.4664
Qwen-3.8-Max77076370.4200
Total1,09362189733171—
Reviewer_DqRdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper presents a long-horizon benchmark with 77 tasks and 13 domains. The deterministic subtasks are set to provide a dense resward that beasure both task completion and partial progress. The benchmark is tested against 15 frontier models and analyzed on the hardness of each task in the appendix.

Critical strengths
  1. Technical contribution: A long-horizon task benchmark for Language-Models that contain problems that where current state of the art models are still limited. The paper further presents a dense reward formulation for the long-term tasks.
  2. Empirical strength: Testing with multiple models, the analysis of the cost, along with utilization of the SOTA models to determine which problems are still hard. The benchmarking is across diverse tasks with well-validated metrics and comprehensive evaluations. Addition of dense reward also provides a means of evaluation partial success, which could be significant in training future models to improve long term planning capabilities through RL.
  3. Relevance and impact: A long-horizon benchmark for LLMs that highlights the limitations of existing SOTA models in long term planning tasks is relevant contribution to the field for accurate performance analysis of Language Models.
Critical weaknesses
  • Qualitative evaluation of the benchmark is mentioned but the paper seems to gloss over it. How were the qualitative measures determined, is it based on the performance of the existing models or was there a human-based ranking? A clarification of how the qualitative evaluation is conducted would improve the quality of the paper.
Questions for authors
  • Could the authors clarify if the provided the long-term planning problems presented can be solved by existing heuristics? Current SOTA models have the capability to write code for dedicated solvers for problems, often offsetting their capabilities for long-term planning. What are the measures taken to prevent such approaches?
2 comments
AuthorsOpenReview ↗

Response to Clarifying Qualitative Evaluation and Task Validation

We assess task quality through expert and author review for the 47 workflow tasks and DeepSeek-assisted validation for the 30 reconstruction tasks. We agree that the manuscript should distinguish these procedures from model-performance analysis and numerical scoring.

For reconstruction tasks, construction starts from working domain artifacts. Reference executions across parameter settings generate input–output observations; some observations are public, while unseen scenarios are reserved for evaluation. We repeat DeepSeek-V4-Pro runs check instruction–verifier alignment and assess difficulty under the specified harness and three-hour budget. These runs provide model-dependent calibration, rather than an independent human ranking of task difficulty.

Benchmark scores themselves come from executable verifiers using task-specific assertions, exact comparisons, or numerical tolerances. Expert review of task construction should not be interpreted as evidence that the numerical reward scale was calibrated against independent human judgments of agent outputs. We will clarify the reviewers’ roles, review criteria, and how review findings informed task revisions, distinguishing these from model-based difficulty calibration and automated scoring.

AuthorsOpenReview ↗

Response to Existing Heuristics, Dedicated Solvers, and Planning Claims

Yes, some constituent problems can be solved using existing heuristics or dedicated solvers. LHTB does not impose a benchmark-wide ban on these methods: whether they are permitted depends on the task’s stated interaction policy. For programming and engineering workflows, selecting, implementing, and checking a suitable algorithm can be part of the intended agent behavior.

For tasks that explicitly target manual planning, restrictions are stated in the instructions and checked during evaluation. Rush Hour Campaign, for example, prohibits writing code, invoking solvers, and using search algorithms. Its verifier simulates submitted routes and checks available submission and process records for prohibited computational assistance, applying a score cap when violations are detected. This provides a concrete enforcement mechanism, although log-based checks cannot guarantee detection of every prohibited strategy.

Hidden scenarios test whether submitted implementations generalize beyond public examples; by themselves, they do not prevent solver use. We will therefore distinguish tasks with solver restrictions from workflows that permit computational tools. Success should be interpreted as completing the task under its stated constraints, rather than as proof of unaided internal planning.

Reviewer_eyskOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Long-Horizon-Terminal-Bench 2.0 (LHTB 2.0), a benchmark of 77 containerized terminal tasks spanning 13 domains. Each task is decomposed into subtasks, and a deterministic grader inside the container scores the final state after a rollout. The final reward is a weighted average of subtask scores, which allows partial credit rather than binary pass/fail. The authors evaluate 15 frontier models using the Harbor framework and the Terminus-2 agent harness, reporting pass rates at R≥0.9R ≥ 0.9, R≥0.95R ≥ 0.95, and R=1.0R=1.0, mean normalized reward, LLM calls per task, runtime, and estimated cost. The main results show that the strongest model, GPT-6-astra, resolves 48.1%48.1\% (37/77) of tasks at R≥0.95R ≥ 0.95; many unsuccessful runs still make partial progress; and about 78%78\% of unresolved runs end at the 3-hour timeout. The paper also analyzes reward changes at 30/60-minute checkpoints, process reward on unresolved runs, per-domain difficulty, and cost-reward frontiers.

Critical strengths
  • The task suite covers a wide range of realistic professional workflows, and the construction effort is substantial.
  • Subtask-level dense grading is a reasonable design choice for long-horizon tasks, and Figure~4 and the near-miss analysis show that binary grading would discard meaningful progress signals.
  • The cost, time, and token analyses are useful for practitioners who actually run this benchmark.
Critical weaknesses

Limited novelty

Dense subtask grading, process reward, and rubric-based evaluation are not new ideas. The core recipe---decompose realistic workflows into intermediate checks, hide harder stress cases, and report partial credit---is reasonable but familiar. Compared with long-horizon agent benchmarks such as Terminal-Bench 2.0, Frontier-SWE, and SWE-Marathon, LHTB 2.0 looks more like an extension in scale and domain coverage than a new evaluation principle or a new scientific question. The main finding is descriptive: models make partial progress but often time out. This is useful, but it is hard to argue that it changes our understanding of long-horizon capability.

Limited practical value and actionability

Although hidden stress cases and low-weight public checks help prevent overfitting, they also make failure diagnosis difficult: the main leaderboard mostly says "the model did not finish'' or "it timed out,'' rather than telling developers where the trajectory went wrong or which capability is missing. If the goal is to guide agent development, this score has limited actionability. In addition, tasks are tied to specific libraries, file formats, and tool versions, which implies high long-term maintenance cost and a risk of becoming outdated quickly. The paper also lacks an argument that the dense reward is semantically aligned with the capabilities practitioners actually care about.

Insufficient principled justification for benchmark construction

Why 77 tasks, these 13 domains, and this particular balance? How were subtask weights chosen, and how sensitive are the main conclusions to changes in those weights? Difficulty was calibrated with DeepSeek-V4-Pro, but it is unclear how much of the difficulty comes from the tasks themselves versus the time budget, harness, or hidden stress cases. Overall, the suite reads more like a well-executed collection of hard tasks than an instrument with a clearly defined measurement target.

Questions for authors
  • What unique capability does LHTB 2.0 measure that Terminal-Bench 2.0, Frontier-SWE, or SWE-Marathon do not already measure? Can the authors give a concrete example where the rankings or conclusions would differ in an informative way?
  • How were subtasks and weights determined? Are the main leaderboard conclusions stable under plausible alternative weightings or subtask granularities?
  • Hidden stress cases make failure analysis harder. How should practitioners use this benchmark to improve agents, rather than only to rank models?
  • Many tasks depend on specific library versions and toolchains. What is the maintenance plan, and how will the benchmark avoid becoming a snapshot of a particular period's toolchain?
6 comments
AuthorsOpenReview ↗

Response to Novelty and the measurement target

We thank the reviewer for encouraging us to sharpen the novelty claim. LHTB’s contribution is a benchmark for studying whether agents can turn extended execution into verified task completion and preserve progress along the way. It brings together 77 tasks across 13 domains, combining expert-reviewed workflows with reconstruction tasks evaluated on hidden scenarios. The contribution lies in this task design and empirical characterization, rather than in introducing dense rewards as a scoring principle.

The distinction is between performing more actions and satisfying the intended task requirements. Our audit provides concrete examples: the RISC-V starter passes 114/125 native tests but fails all 11 bug-sensitive tests, while the DuckDB reference passes all correctness queries but misses its performance target. These outcomes distinguish general test coverage from successful repair, and functional correctness from performance completion. They demonstrate why evaluating these workflows requires examining which requirements an agent satisfies, not only its aggregate score.

Our additional experiments examine this distinction over time. With six-hour active-time budgets and three-hour checkpoints within the same attempts, mean native scores increase from 0.5189 to 0.5297 across 57 GPT-5.6-Sol pairs and from 0.4838 to 0.5157 across 61 Kimi K3 pairs. Every included pair records additional agent turns, yet NINE TASKS REGRESS for each model. Thus, the study reveals both continued progress and failures to retain earlier measured gains, a behavior that previous endpoint score works alone cannot expose.

Together, the task suite, requirement-specific diagnostics, and within-attempt comparisons provide a concrete contribution to evaluating sustained agent performance. We will make this contribution explicit in the revision and position it as complementary to existing benchmarks. The duration comparisons use native graders and include verified early-success endpoints; original and recovery cohorts are reported separately because execution and verifier handling changed.

AuthorsOpenReview ↗

Response to Insufficient principled justification for benchmark construction

We thank the reviewer for identifying the need to clarify the benchmark’s measurement target and construction rationale. LHTB evaluates agents’ ability to complete and verify terminal workflows under a specified tool interface and resource budget, including whether they preserve useful progress during extended execution.

The release combines 47 workflow tasks reviewed by domain experts and the authors with 30 reconstruction tasks validated using DeepSeek. We will document these distinct construction and validation procedures separately. The 77 tasks and 13 domains constitute the current curated release; we do not claim that this composition is optimal or statistically representative of terminal work.

For the reconstruction tasks, repeated DeepSeek-V4-Pro runs assess difficulty under the specified harness and three-hour budget and check instruction–verifier alignment. This calibration is conditional on the evaluation setup: a failed run alone cannot isolate model capability from task complexity, resource constraints, or evaluation errors.

We also revised reconstruction scoring to award equal credit for completed hidden scenarios, requiring successful execution and all expected semantic assertions to pass. Our alternative-weighting analysis supports the leader’s stability on the tested subset while revealing changes elsewhere in the ranking. We provide the detailed sensitivity results in our response to Question 2 in the following thread.

AuthorsOpenReview ↗

Response on LHTB’s Distinct Contribution and Comparison with Existing Benchmarks

LHTB’s contribution is a benchmark and empirical study of whether agents can turn extended execution into verified completion and preserve progress along the way. Its 77 tasks across 13 domains enable examination of targeted functionality, correctness, and performance across heterogeneous terminal workflows.

Our audit provides concrete examples where separating these requirements changes the conclusion. The RISC-V starter passes 114/125 native tests but fails all 11 bug-sensitive tests: high aggregate test coverage does not imply successful repair. The DuckDB reference passes all correctness queries but misses its performance target: correct outputs do not imply completion of the optimization task. Our within-attempt duration comparisons additionally reveal nine score regressions per model despite additional agent turns, showing behavior that an endpoint score alone cannot expose.

These examples establish diagnostic value, rather than an exclusive capability or a demonstrated ranking difference against the named benchmarks. We will position LHTB as complementary to them and make this distinction explicit.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.