- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how changes to agent benchmark implementations can alter reported model performance. It shows that the verdicts and scores assigned to the same recorded agent executions can change depending on which benchmark version grades them. To address this, the authors propose BenchPulse, a continuous-evaluation framework that models benchmark versions as five-layer contracts (task, verifier, environment, execution policy, aggregation). BenchPulse consists of ContractIR, which analyzes repository diffs to identify which parts of the evaluation contract changed, and PulseSet, which uses these changes and cached execution dependencies to select which runs to regrade, prioritizing runs whose verdict changes could reverse close model comparisons. A census of 4,119 commits across 15 benchmarks estimates that 68.72% of commits affect the evaluation contract. Re-evaluating cached executions under selected historical transitions reveals changes in verdicts, scores, and rankings (e.g., −5.66 pp on τ-bench and 893 pairwise ranking reversals on BFCL). Across τ-bench and τ²-bench, PulseSet regrades 20.48% of cached executions and detects 95.45% of the verdict changes that would be found by regrading all executions.
Critical strengths
A relevant problem, with concrete fixed-execution evidence. Prior work on evolving benchmarks primarily studies task refreshes to limit contamination (LiveBench, LiveCodeBench), environmental variations that affect agent behavior (TimeWarp, OpenApps), or efficient evaluation under a fixed benchmark (DISCO, micro-benchmarking). In contrast, this paper isolates the effects of benchmark implementation changes by re-grading identical recorded executions under different evaluation contracts. The observed score shifts and ranking reversals provide concrete evidence that benchmark maintenance can affect reported results, highlighting an important concern for reproducibility and longitudinal model comparisons.
Critical weaknesses
Limited empirical evidence. The paper examines only a few selected benchmark updates and provides limited information about the models evaluated. The 4,119-commit analysis measures how often benchmarks change, not how often those changes affect scores. Moreover, none of the four τ²-bench ranking reversals is statistically significant under the authors' bootstrap analysis, making their practical importance unclear. Insufficient validation of BenchPulse. ContractIR performs worse than simple file-path detection on F1 (80.30% vs. 83.47%), and its layer-attribution accuracy is not evaluated. PulseSet's total runtime is not compared against regrading every cached run, which the paper implies is efficient (~10s), so its practical savings remain unclear. The method also relies on update-specific rules, with limited evidence that it generalizes to complicated, unseen benchmark updates. Moreover, the headline 95.45% recall combines an exact-dependency case (τ-bench) with the more challenging τ²-bench selection experiment, which achieves 93.12% recall.
Questions for authors
Experimental coverage. Which models were evaluated, and how were they selected? Can the authors test a broader set of benchmark updates, provide per-model before-and-after scores, and assess how often updates lead to substantial score changes or statistically meaningful ranking reversals? How do these changes compare with normal run-to-run variation?
Method details and generality. How does ContractIR identify affected contract layers, and how are PulseSet's dependency rules and risk scores constructed? How much benchmark-specific engineering is required? Can the authors demonstrate the complete workflow on previously unseen updates without designing rules specifically for those updates?