- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes DualViewEval to reduce the high cost of agent benchmarking. The method extracts six execution-process metrics, constructs outcome and process relation matrices, and jointly optimizes task subset selection and full-benchmark score prediction through weighted fusion, straight-through hard Top-K selection, and Kernel Ridge Regression. Experiments on five agent benchmarks demonstrate promising performance in score prediction and ranking preservation.
Critical strengths
1.The paper introduces a dual-view relational approach that combines outcome and process information for task subset selection. This is well-motivated for agent evaluation, where similar final outcomes may arise from substantially different execution behaviors.
2.The paper develops a compact, six-dimensional process representation intended to be applicable across different agent benchmarks and incorporates it into the selection-and-prediction framework through a learned fusion coefficient γ.
3.The proposed straight-through hard Top-K mechanism guarantees an exact-size task subset while enabling joint optimization of task selection and score prediction through surrogate gradients, providing a practical solution for fixed-budget evaluation.
Critical weaknesses
1.The core method primarily combines conventional techniques, including inner-product relation matrices, weighted fusion, hard Top-K selection, and Kernel Ridge Regression, resulting in relatively limited methodological novelty. Although the paper provides ablation studies validating individual components, it does not sufficiently compare alternative process-relation constructions or structurally different fusion strategies. Consequently, the distinctive advantages and necessity of the proposed dual-view relational modeling mechanism remain insufficiently established.
2.The process representation relies on benchmark-specific trajectory parsers, while Table 8 shows substantial variations in process metrics for the same underlying model across different agent harnesses. The current experiments do not sufficiently distinguish model-specific behavior from harness-induced execution differences, raising concerns about the reliability of the learned process relations.
3.Although Table 2 reports means and standard deviations over ten splits, explicit paired significance tests or confidence intervals are not provided. Given the relatively small number of agent configurations in some benchmarks and the small performance differences in certain settings, the statistical reliability of the reported improvements remains insufficiently established.
Questions for authors
1.Could the authors further clarify the methodological novelty of the proposed dual-view relational learning mechanism and compare it with alternative process-relation constructions or fusion strategies to demonstrate its necessity and distinctive advantages?
2.Could the authors evaluate the proposed process representation under a fixed agent harness and clarify how model-specific behavioral differences are distinguished from harness-induced effects?
3.Could the authors provide paired statistical significance tests or confidence intervals across the same evaluation splits to establish the reliability of the reported performance improvements?