acceptodds
Under review as a conference paper at ICLR 2027

Beyond Outcomes: Dual-View Relational Learning for Efficient Agent Benchmarking

Abstract

Agent benchmarks are substantially more costly to evaluate than conventional LLM benchmarks. Benchmark compression is therefore a natural solution, yet existing methods primarily model redundancy in task–model final-score distributions, which is important in agentic evaluation. To address this limitation, we analyze large-scale trajectories and identify six complementary process signals that are systematically associated with final agent performance. To disentangle agent performance redundancy from a complete perspective, we propose **DualViewEval**, an agent benchmark compression method that jointly exploits outcome and process relations to learn an exact-size miniset and predict the full-benchmark scores. Across five agent benchmarks and five representative baselines, DualViewEval achieves the best results in all datasets. With only 20 tasks, it achieves – compression on APEX-Agents and BFCL, reducing mean absolute error (MAE) by – over the strongest competitors while improving Kendall's by up to relative to EssenceBench on SWE-bench Verified. The selected minisets further reveal capability differences among different agents, providing compact and diagnostic feedback for efficient agentic model development.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Agent Benchmark Compression, Miniset Selection, Trajectory Analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_T3VHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DualViewEval to reduce the high cost of agent benchmarking. The method extracts six execution-process metrics, constructs outcome and process relation matrices, and jointly optimizes task subset selection and full-benchmark score prediction through weighted fusion, straight-through hard Top-K selection, and Kernel Ridge Regression. Experiments on five agent benchmarks demonstrate promising performance in score prediction and ranking preservation.

Critical strengths

1.The paper introduces a dual-view relational approach that combines outcome and process information for task subset selection. This is well-motivated for agent evaluation, where similar final outcomes may arise from substantially different execution behaviors.

2.The paper develops a compact, six-dimensional process representation intended to be applicable across different agent benchmarks and incorporates it into the selection-and-prediction framework through a learned fusion coefficient γ.

3.The proposed straight-through hard Top-K mechanism guarantees an exact-size task subset while enabling joint optimization of task selection and score prediction through surrogate gradients, providing a practical solution for fixed-budget evaluation.

Critical weaknesses

1.The core method primarily combines conventional techniques, including inner-product relation matrices, weighted fusion, hard Top-K selection, and Kernel Ridge Regression, resulting in relatively limited methodological novelty. Although the paper provides ablation studies validating individual components, it does not sufficiently compare alternative process-relation constructions or structurally different fusion strategies. Consequently, the distinctive advantages and necessity of the proposed dual-view relational modeling mechanism remain insufficiently established.

2.The process representation relies on benchmark-specific trajectory parsers, while Table 8 shows substantial variations in process metrics for the same underlying model across different agent harnesses. The current experiments do not sufficiently distinguish model-specific behavior from harness-induced execution differences, raising concerns about the reliability of the learned process relations.

3.Although Table 2 reports means and standard deviations over ten splits, explicit paired significance tests or confidence intervals are not provided. Given the relatively small number of agent configurations in some benchmarks and the small performance differences in certain settings, the statistical reliability of the reported improvements remains insufficiently established.

Questions for authors

1.Could the authors further clarify the methodological novelty of the proposed dual-view relational learning mechanism and compare it with alternative process-relation constructions or fusion strategies to demonstrate its necessity and distinctive advantages?

2.Could the authors evaluate the proposed process representation under a fixed agent harness and clarify how model-specific behavioral differences are distinguished from harness-induced effects?

3.Could the authors provide paired statistical significance tests or confidence intervals across the same evaluation splits to establish the reliability of the reported performance improvements?

Reviewer_4qN7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper seeks to tackle the expensive cost of agentic benchmarks by introducing DualViewEval, a benchmark compression method. The goal of the paper is to find subsets of existing agentic evaluation benchmarks that still reflect the performance and ranking of models on the full benchmarks. The authors analyze and characterize agentic trajectories with six measurements, such as agent steps and tool entropy. These are used to construct task-wise relation matrices. Then, these features are used to train a score predictor from the smaller set to the full benchmark numbers. The experiments are conducted over a large number of benchmarks and agentic configurations, and the comparisons are comprehensive against other approaches. The authors demonstrate that DualViewEval overall achieves lower MSE compared to these baselines.

Critical strengths
  • The core idea of the paper in reducing the cost of agentic benchmarks is an extremely useful one to the community, since the expensive cost can limit access to many researchers. The paper seems to achieve this goal well and show good results on how evaluating on a few tasks may be representative of other tasks in five different benchmarks.
Critical weaknesses
  • The paper focuses on coding agentic benchmarks, and the relation features seem handcrafted, but it’s unclear how they may transfer to other types of agentic tasks, such as computer use or search agents.
  • The presentation feels a bit unclear to me — it seems like the predictor is trained using all outcome pairs in each dataset and also evaluated on the same pairs, though I’m not familiar with other benchmarks compression methods. It would be a lot more useful to the community if there are deeper analysis on if a new model or framework were to be introduced, and the predictor did not include them in training, how that new model/framework would do.
Questions for authors
  • how did you choose the six measurements? they seem reasonable but it also feels like there could be a number of other things to measure, such as length of accumulated context or number of compactions in many agentic frameworks or relations between the tool calls.
  • how much of the task characterization is dependent on the model vs. the task?
Reviewer_nCJvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DualViewEval, a benchmark compression framework designed to reduce the cost of evaluating LLM agents while preserving full-benchmark performance estimates and model rankings. The central observation is that final task outcomes alone may not adequately capture differences in agent execution behavior. To address this limitation, the authors identify six trajectory-derived process metrics and construct task-specific relational matrices from both outcome and process information. These matrices are combined into a learned agent kernel, while a straight-through hard Top-K selector and a Kernel Ridge predictor are jointly optimized to select an exact-size task subset and estimate full-benchmark scores for unseen agents.

The method is evaluated on five agent benchmarks—BFCL, τ²-Bench, Terminal-Bench 2, SWE-bench Verified, and APEX-Agents—against five benchmark compression baselines. Experiments across multiple subset budgets show consistently competitive results, particularly in mean absolute error (MAE), with generally strong ranking preservation. Ablation studies examine the contributions of the two relational views, subset selection, and prediction architecture. An additional model-family holdout experiment provides evidence of transfer to previously unseen model configurations.

Critical strengths

1. A well-motivated process-aware formulation of agent benchmark compression.

The paper addresses a practical and increasingly important problem: evaluating modern agents is considerably more expensive than evaluating conventional language models. Its central insight—that agents with similar final outcomes may exhibit meaningfully different execution behaviors—is compelling. The proposed six-dimensional process representation makes this insight operational, while the dual-view relational formulation provides a coherent mechanism for incorporating process information into benchmark compression.

This is consequential because it extends compression beyond outcome-based redundancy and provides a potentially useful direction for evaluation methods designed specifically for agentic systems.

2. Broad empirical evaluation with consistent improvements and useful ablations.

The experiments cover five benchmarks spanning function calling, interactive environments, terminal operations, software engineering, and professional workflows. DualViewEval achieves the lowest reported MAE across all five benchmarks at the three principal subset budgets in Table 2, often with favorable Kendall's τ.

Importantly, the ablations in Table 3 show that the process view contributes improvements beyond the outcome-only variant, while the model-family holdout experiment in Appendix C.1 provides additional evidence of generalization. These results make the empirical contribution more convincing than a performance comparison based on a single benchmark or evaluation setting.

Critical weaknesses

1. The independence and generalizability of the process representation are insufficiently established.

The six process measurements are selected from twelve candidate statistics through an analysis of their associations with agent performance across the same five benchmarks used in the main evaluation. The manuscript does not clearly establish whether this feature-design procedure was independent of held-out agents' full-benchmark outcomes. If the final metrics were selected using performance information from the evaluation populations, the reported gains may partly reflect benchmark-specific feature selection.

Furthermore, several metrics depend on benchmark-specific trajectory parsing and tool-call classification. Their reliability under different agent harnesses, logging conventions, or incomplete execution traces is not systematically evaluated. These issues matter because the claimed generality of the process representation is central to the paper's contribution.

2. Generalization and practical efficiency require stronger validation.

The main experiments randomly partition agent configurations, but some benchmarks contain relatively few configurations, making validation and test sets small. For example, τ²-Bench contains only 18 configurations, and APEX-Agents contains 26. This may make ranking estimates and validation-based checkpoint selection sensitive to individual configurations. Additionally, SWE-bench Verified contains multiple harnesses applied to the same underlying models, raising questions about whether configuration-level splits adequately measure transfer to genuinely unseen models and execution environments. The Qwen-family holdout experiment on APEX-Agents partially addresses this concern, but analogous evidence across other benchmarks is missing.

The efficiency analysis also focuses primarily on subset evaluation and model training time, without fully accounting for the initial expense of collecting and parsing complete trajectories from reference agents. Since the method's primary motivation is evaluation efficiency, a more complete amortized cost analysis would strengthen its practical claims.

Questions for authors

1. Independence of process-feature selection. Were the six process metrics finalized independently of the held-out agents' full-benchmark scores, or did the correlation analysis used to select them include all agent configurations? Could the authors provide a leakage-free feature-selection analysis, for example using training-only statistics or leave-one-benchmark-out feature selection? This clarification is important for assessing whether the reported gains generalize beyond the datasets used to develop the representation.

2. End-to-end cost and process-extraction robustness. Could the authors report the total cost of building the calibration data, including full-benchmark reference-agent executions, trajectory parsing, feature extraction, and optimization, along with the number of subsequent evaluations required to amortize this cost? Additionally, how sensitive are the results to missing trajectory fields or errors in tool-call categorization? These measurements would help determine whether the method retains its advantages under practical deployment conditions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.