acceptodds
Under review as a conference paper at ICLR 2027

AgentHorizon: Evaluating Agentic Judges for Long-Horizon Computer-Use Tasks

Abstract

Computer-use agents are capable of completing complex tasks, which is increasingly using automatic judges to determine the success, either for training or for evaluation without human involvement. Despite their flexibility, their reliability on long tasks spanning multiple applications remains unclear. A trajectory, composed of long sequences of screenshots and actions, may appear complete, but in reality violates constraints from the instruction or introducing an unwanted side effect. To identifying these errors, a judge needs to carefully examine the trajectory with respect to the user's instruction. To this end, we introduce AgentHorizon, a benchmark of 1,373 computer-use tasks (instructions and trajectories pairs) drawn from 166 hours of human-recorded trajectories spanning three operating systems. By recording trajectories for closely related instructions, we can construct negative tasks by simply swapping the instructions. This paired design evaluates judges in their ability to distinguish a truly successful trajectory from one that completed a similar (but incompatible) request. We release the benchmark under three splits: a frontier split, AgentHorizon (AH), simplified split, AgentHorizon-Simple (AH-S), and a development split, AgentHorizon-Development (AH-D). We further evaluate eleven judges by (1) directly passing the full trajectory (with up to 300 screenshots and actions), and (2) by using them as coding agent across five agent harnesses. We find that our best agentic judge, GPT-5.5, achieves 80.9% balanced accuracy on the AH subset. We find that tool-use improves certain models but results in worse performance for open-weight models, and that judges differ drastically in their ability to accept a valid trajectory reject failed ones. Our findings highlight the need for judges that are capable of locating and verifying often hidden evidence that a task was properly completed inside long interaction histories. We release our benchmark and reuslts publicly.

Primary area:
datasets and benchmarks
Keywords:
agents, computer-use, computer-use agents, llm judge, agentic judge

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_UPTtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AgentHorizon, a benchmark for evaluating LLM-based judges on long-horizon computer-use tasks. The benchmark consists of 1,373 instruction–trajectory pairs derived from 166 hours of human-recorded computer interactions. Its primary contribution is a paired construction strategy in which annotators execute two closely related instructions and the corresponding trajectories are swapped to generate negative examples. The authors evaluate both direct LLM judges and agentic judges operating through different coding-agent harnesses.

Critical strengths
  1. The authors put significant effort into constructing this benchmark, involving over 6,000 hours of human annotation and review. I appreciate the substantial effort involved in collecting and validating the dataset.

  2. Reliable evaluation of computer-use agents is an increasingly important problem, particularly as agents execute longer workflows across multiple applications. Evaluating the reliability of automated judges is a worthwhile research direction.

Critical weaknesses
  1. Model selection. The paper evaluates several strong models, but it is unclear whether the most recent frontier models are adequately covered. The authors should better justify their model selection.

  2. Unclear evaluation objective. I am not sure whether this benchmark aims to evaluate the final outcome or the intermediate process. If the goal is to judge the final outcome, it is unclear why such long trajectories are necessary. If the goal is to evaluate the process, the evaluation objective should be better defined.

  3. Unclear task distribution. The paper does not clearly explain the task distribution or how the tasks were selected. It is also unclear how representative these tasks are of real-world computer-use agent tasks.

  4. Questionable negative sample construction. The negative samples are mainly constructed by swapping instructions between similar tasks. Such mismatches may be relatively easy to identify by checking a few relevant steps, without requiring long-horizon reasoning. I am not convinced that these negative samples adequately reflect real agent failures.

  5. Insufficient evidence for long-horizon reasoning. The paper does not clearly demonstrate that judging these tasks requires understanding the full trajectory. It would be useful to compare against a stronger baseline using final artifacts or a few key observations rather than the entire trajectory.

Questions for authors
  1. What is the intended evaluation target of AgentHorizon: final outcome correctness, process-level instruction compliance, or long-horizon temporal reasoning? How are these capabilities distinguished in the benchmark?

  2. Can the authors provide a detailed task taxonomy, including domain frequencies, application combinations, task complexity, and cross-step dependency structures?

  3. What fraction of negative examples can be resolved by inspecting only one or a few task-relevant screenshots or actions? How many genuinely require integrating evidence across distant trajectory steps?

  4. Have the authors considered comparing full-trajectory judging against a stronger final-state baseline that provides actual output artifacts or application states rather than only the final screenshot?

  5. How well does the distribution of instruction-swapped negative examples reflect the failure modes encountered in real computer-use agent executions?

Reviewer_5wVKOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper investigate and compare normal LLM judge versus agentic judge on long horizon tasks involving computer use. Long-horizon tasks are more challenging as the judge would need to determine which part of the interaction history matters. In order to evaluate whether a judge can correctly determine the success of a long-horizon computer-use trajectory with respect to the task instruction, the authors introduce AGENTHORIZON, a benchmark derived from human-recorded trajectories consisting of paired tasks. The paired tasks consist of difficult negative samples that is based on swapping closely related instructions to create hard negative samples. The benchmark consists of three subsets: i) AgentHorizon (AH) containing most challenging tasks, ii) AgentHorizon-Simple (AH-S) contains simpler tasks, and iii) AgentHorizon-Development (AH-D) that is created mainly for development.

Critical strengths
  1. The paper addresses a timely problem of creating benchmark to evaluate agentic judges for long-horizon computer tasks which is very relevant today.
  2. The paper also provides a scalable approach to create paired positive and negative data by deriving B_instruct from A_instruct, and later swapping the two instructions to create the negative sample.
Critical weaknesses
  1. The list of contributions is not properly articulated. Please clearly clarify and enumerate how the benchmark differs from existing works in terms of novelty.
  2. Missing code and dataset which prevented me from assessing the quality of the work. This is especially important when this is under the benchmark track.
Questions for authors
  1. The approach of generating negative samples by modifying or perturbing the positive sample is a known method in the literature. A related work [1] adopted a similar approach where they modify/perturb the agent trajectory and pair it with the original instruction to create the negative sample. Please cite this work and discuss the advantages of your method as compared to [1].

[1] Chen, Z., Chen, D., Sun, R., Liu, W., & Gan, C. (2025, May). Scaling autonomous agents via automatic reward modeling and planning. In International Conference on Learning Representations (Vol. 2025, pp. 29237-29268).

  1. Section 3.2 explains that annotators derive B_instruct from A_instruct (Fig 3). How do annotators ensure that the two instructions remain closely related, and is the degree of modification (e.g., number of requirements) controlled to maintain comparable execution trajectories? Otherwise, swapping the instructions may produce trivial negative examples rather than the intended hard negative examples.
  2. How does the AH-neg score changes as the pair of B_instruct and A_instruct instructions widens in differences?
  3. In Table 2, why are the metrics for AH-S pos and AH-S neg missing?
2 comments
AuthorsOpenReview ↗

10 days since our response was posted; we are looking forward for your reply

Dear reviewer 5wVK,

We haven't heard back since we posted our response 10 days ago, where we address both of your concerns and answered your question. We are looking forward your reply, as it is the last day of discussions.

Reviewer_5wVKOpenReview ↗

Official comment from Reviewer 5wVK

Thank for your rebuttal. However my concern regarding the meaningful construction of negative samples Q2 still remains (this is also raised by reviewer Reviewer UPTt). Therefore, I maintain my score.

Reviewer_m4yMOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes AgentHorizon (see note below on the naming though), a new benchmark for evaluating LLM judges on long-horizon agentic tasks. The benchmark consists of positive and negative pairs of task instructions and trajectories, and judges are evaluated on whether they pass or fail the given trajectory on the given task. This is different from some other efforts that try also to generate the trajectory; here both the task and the trajectory are provided as input, and the goal is simple binary classification.

The work appears well motivated and addresses an important problem as the community is starting to rely heavily on LLM judges. It's somewhat surprising, though probably reassuring, that a strong agentic judge, based on GPT-5.5, achieves nearly 81% balanced accuracy (I like that the authors have been careful about choices such as computing balanced accuracy, not simply accuracy).

Critical strengths
  • The authors follow a simple but very neat idea, namely using closely related (but semantically different) task instructions and corresponding agent trajectories to create positive and negative examples for evaluating LLM judges. While similar in spirit to earlier concepts like contrast sets, grouped evaluations, etc., the specific use here is novel to my knowledge, and simple yet effective.

  • Empirical evaluation reported in the paper provides both a positive message -- that the best agentic LLM judges like GPT-5.5 and Gemini 3.1 pro are ~80% accurate -- and an opportunity for improvement in LLM judge design. E.g., "direct" judges for frontier LLMs clearly lag behind, while open models currently do not appear to serve reliably as judges, even in the agentic mode. I expect these findings, as well as the benchmark, will be helpful in pushing research forward on LLM judges.

Critical weaknesses
  • The only hesitation I have about this work is whether the pairs of tasks are actually different enough to constitute clearly identifiable positive-negative instructions for a trajectory. The authors mention (section 3.2) removing pairs with semantically equivalent instructions. This is presumably done with an LLM and is inherently a noisy process where false negatives can be problematic. The authors also mention all "negative candidates" being reviewed by a vendor. This probably suffices, but any further clarification the authors can provide on this process would be valuable. For instance, it appears to suggest that a human baseline on the construction positive and negative pairs would be close to 100%, but I am not sure if this is the right interpretation or if there is a different human eval the authors have conducted and can comment on.

  • This isn't really a weakness, but the authors might want to reconsider the name AgentHorizon. As such, it sounds like a benchmark (similar to many with this style of names) is about evaluating agents rather than evaluating judges. It can thus get lost among the many agent evaluation benchmark, even though it's somewhat unique in that it targets a different ability.

Questions for authors

If you have any further clarifications or comments on the two points in the weaknesses section, please share.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.