- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AgentHorizon, a benchmark for evaluating LLM-based judges on long-horizon computer-use tasks. The benchmark consists of 1,373 instruction–trajectory pairs derived from 166 hours of human-recorded computer interactions. Its primary contribution is a paired construction strategy in which annotators execute two closely related instructions and the corresponding trajectories are swapped to generate negative examples. The authors evaluate both direct LLM judges and agentic judges operating through different coding-agent harnesses.
Critical strengths
-
The authors put significant effort into constructing this benchmark, involving over 6,000 hours of human annotation and review. I appreciate the substantial effort involved in collecting and validating the dataset.
-
Reliable evaluation of computer-use agents is an increasingly important problem, particularly as agents execute longer workflows across multiple applications. Evaluating the reliability of automated judges is a worthwhile research direction.
Critical weaknesses
-
Model selection. The paper evaluates several strong models, but it is unclear whether the most recent frontier models are adequately covered. The authors should better justify their model selection.
-
Unclear evaluation objective. I am not sure whether this benchmark aims to evaluate the final outcome or the intermediate process. If the goal is to judge the final outcome, it is unclear why such long trajectories are necessary. If the goal is to evaluate the process, the evaluation objective should be better defined.
-
Unclear task distribution. The paper does not clearly explain the task distribution or how the tasks were selected. It is also unclear how representative these tasks are of real-world computer-use agent tasks.
-
Questionable negative sample construction. The negative samples are mainly constructed by swapping instructions between similar tasks. Such mismatches may be relatively easy to identify by checking a few relevant steps, without requiring long-horizon reasoning. I am not convinced that these negative samples adequately reflect real agent failures.
-
Insufficient evidence for long-horizon reasoning. The paper does not clearly demonstrate that judging these tasks requires understanding the full trajectory. It would be useful to compare against a stronger baseline using final artifacts or a few key observations rather than the entire trajectory.
Questions for authors
-
What is the intended evaluation target of AgentHorizon: final outcome correctness, process-level instruction compliance, or long-horizon temporal reasoning? How are these capabilities distinguished in the benchmark?
-
Can the authors provide a detailed task taxonomy, including domain frequencies, application combinations, task complexity, and cross-step dependency structures?
-
What fraction of negative examples can be resolved by inspecting only one or a few task-relevant screenshots or actions? How many genuinely require integrating evidence across distant trajectory steps?
-
Have the authors considered comparing full-trajectory judging against a stronger final-state baseline that provides actual output artifacts or application states rather than only the final screenshot?
-
How well does the distribution of instruction-swapped negative examples reflect the failure modes encountered in real computer-use agent executions?
10 days since our response was posted; we are looking forward for your reply
Dear reviewer 5wVK,
We haven't heard back since we posted our response 10 days ago, where we address both of your concerns and answered your question. We are looking forward your reply, as it is the last day of discussions.