acceptodds
Under review as a conference paper at ICLR 2027

BenchPulse: Monitoring Verdict and Ranking Drift in Evolving Agent Benchmarks

Abstract

Agent benchmarks are increasingly maintained as software: tasks are repaired, tools evolve, verifiers are corrected, and leaderboard formulas change. Yet model scores from different benchmark versions are routinely compared as if the measurement contract were fixed. We introduce BenchPulse, a framework that models benchmark repository transitions as interventions on a five-layer evaluation contract—task, verifier, environment, execution policy, and aggregation. A 4,119-commit census across 15 agent benchmarks maps how these contracts evolve. ContractIR represents the affected contract surface of a code change, while PulseSet uses cached execution dependencies and conclusion sensitivity to select a compact replay panel before observing new outcomes. Across 8,495 frozen development executions, real updates change 330 verdicts; a pre-specified held-out SWE-bench transition changes 17/642 additional frozen verdicts, and a separate 17-system audit changes 34 approximate 95% interval conclusions. A single verifier commit moves -bench by -5.66 percentage points, a -bench release moves individual model panels by up to 9.02 points and reverses 4 point-estimate orderings, and a BFCL aggregation change creates 893 strict point-order reversals among 5,886 model pairs, including 658 with at least a one-point gap under both contracts. Layer-aware monitoring re-evaluates 20.48% of cached executions and recovers 95.45% of changed verdicts; it recovers all observed full-panel point-order reversals with zero model calls. Two negative controls spanning 2,370 frozen executions produce no flips, showing that semantic-looking maintenance is not automatically consequential. These results establish benchmark evolution as a measurable source of reported-result drift and provide a practical continuous-evaluation protocol for versioned agent benchmarks.

Primary area:
datasets and benchmarks
Keywords:
agent benchmarks, benchmark evolution, evaluation drift, regression testing, leaderboard reliability, software agents, measurement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_c2s4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how changes to agent benchmark implementations can alter reported model performance. It shows that the verdicts and scores assigned to the same recorded agent executions can change depending on which benchmark version grades them. To address this, the authors propose BenchPulse, a continuous-evaluation framework that models benchmark versions as five-layer contracts (task, verifier, environment, execution policy, aggregation). BenchPulse consists of ContractIR, which analyzes repository diffs to identify which parts of the evaluation contract changed, and PulseSet, which uses these changes and cached execution dependencies to select which runs to regrade, prioritizing runs whose verdict changes could reverse close model comparisons. A census of 4,119 commits across 15 benchmarks estimates that 68.72% of commits affect the evaluation contract. Re-evaluating cached executions under selected historical transitions reveals changes in verdicts, scores, and rankings (e.g., −5.66 pp on τ-bench and 893 pairwise ranking reversals on BFCL). Across τ-bench and τ²-bench, PulseSet regrades 20.48% of cached executions and detects 95.45% of the verdict changes that would be found by regrading all executions.

Critical strengths

A relevant problem, with concrete fixed-execution evidence. Prior work on evolving benchmarks primarily studies task refreshes to limit contamination (LiveBench, LiveCodeBench), environmental variations that affect agent behavior (TimeWarp, OpenApps), or efficient evaluation under a fixed benchmark (DISCO, micro-benchmarking). In contrast, this paper isolates the effects of benchmark implementation changes by re-grading identical recorded executions under different evaluation contracts. The observed score shifts and ranking reversals provide concrete evidence that benchmark maintenance can affect reported results, highlighting an important concern for reproducibility and longitudinal model comparisons.

Critical weaknesses

Limited empirical evidence. The paper examines only a few selected benchmark updates and provides limited information about the models evaluated. The 4,119-commit analysis measures how often benchmarks change, not how often those changes affect scores. Moreover, none of the four τ²-bench ranking reversals is statistically significant under the authors' bootstrap analysis, making their practical importance unclear. Insufficient validation of BenchPulse. ContractIR performs worse than simple file-path detection on F1 (80.30% vs. 83.47%), and its layer-attribution accuracy is not evaluated. PulseSet's total runtime is not compared against regrading every cached run, which the paper implies is efficient (~10s), so its practical savings remain unclear. The method also relies on update-specific rules, with limited evidence that it generalizes to complicated, unseen benchmark updates. Moreover, the headline 95.45% recall combines an exact-dependency case (τ-bench) with the more challenging τ²-bench selection experiment, which achieves 93.12% recall.

Questions for authors

Experimental coverage. Which models were evaluated, and how were they selected? Can the authors test a broader set of benchmark updates, provide per-model before-and-after scores, and assess how often updates lead to substantial score changes or statistically meaningful ranking reversals? How do these changes compare with normal run-to-run variation?

Method details and generality. How does ContractIR identify affected contract layers, and how are PulseSet's dependency rules and risk scores constructed? How much benchmark-specific engineering is required? Can the authors demonstrate the complete workflow on previously unseen updates without designing rules specifically for those updates?

Reviewer_4xBDOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

BenchPulse argues that agent benchmark scores drift across versions even when agent behavior is fixed. It splits a benchmark into 5 layers (task, verifier, environment, execution policy, aggregation), tags commits by layer (ContractIR), and re-scores a small subset of saved runs to catch changes (PulseSet). On tau-bench, tau2/tau3, BFCL and SWE-bench, real updates flip verdicts and rankings; re-scoring 20% of runs recovers 95% of flips with no model calls.

Critical strengths
  • The problem is real, people compare scores across benchmark versions as if nothing changed, and benchmarks get patched all the time.
  • Reusing saved agent traces to avoid re-running models is neat and cheap: no model calls, a few seconds on one CPU, and the artifacts are released. Quite useful tool for benchmark maintainers.
Critical weaknesses
  • The problem is real but not surprising. As errors in benchmarks get discovered and their tasks get fixed, the scores models obtain are expected to change to reflect those fixes (e.g., Terminal-Bench 4.0, https://www.tbench.ai/news/terminal-bench-4-0, and tau2-Bench-Verified, https://arxiv.org/abs/2512.07850). Maintainers already handle this: Terminal-Bench 4.0 explicitly says its changes "require re-running trials", and Sierra's tau-bench v1.0.1 release notes already publish the same re-grade the paper reports (same +9.02 shift, all flips upward).

  • Too unreliable to trust. If tasks are wrong, the scores need to be regenerated by re-running the agent, not re-scored from old traces. And if there is an implicit change, like a timeout from a dependency default (e.g., an HTTP client), it would look like a simple "dependency bump" in the commit but would still affect scores. Task, environment and execution-policy changes are supposed to be in scope but are never measured (every result uses zero model calls), so I am not sure how this method works under those changes. Trusting this method could add noise, because it can report "no change" when the change actually needs a re-run, and it can't tell a fix from a regression.

  • No reliable way to decide when a re-run is needed. The paper never explains or evaluates how it decides that a change can be checked from saved logs. Each replay uses a hand-built adapter, and the held-out SWE-bench case was chosen because it was replayable (selection bias). Even the tau2 replay includes environment changes that alter what the agent sees (tool output order #403, KB doc fix #388), so the frozen replay measures the new grader on old behavior, not the new benchmark.

  • Version labeling is unclear: the "tau2-bench release" is actually tau-bench v1.0.1 (banking_knowledge only). Please state the exact tag and domain.

  • Small evidence. Only a few transitions are replayed, the held-out SWE-bench case is 17 flips on 2 tasks, and for tau-bench and SWE-bench finding affected runs is basically a lookup. The 4 tau2 ranking reversals have CIs that overlap zero.

Questions for authors
  • How do you decide when a change needs a re-run vs. can be re-scored from logs? Can you report how accurate that decision is?
  • How would BenchPulse handle implicit changes from dependencies (e.g., a default timeout in an HTTP client)?
Reviewer_rAKxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

BenchPulse studies how changes to agent-benchmark software affect evaluation results while keeping archived agent executions fixed. It represents benchmark changes via five evaluation layers and proposes selecting cached executions using dependencies on changed components and sensitivity of model comparisons. The paper combines a repository-history audit with several historical replay studies, reporting 95.45% verdict-change recall while re-evaluating 20.48% of cached executions across two development transitions.

Critical strengths

The empirical question is useful and cleanly isolated. Keeping execution fixed separates changes in measurement from changes in agent behavior. The historical verifier and aggregation updates produce substantial effects, as shown in the papers experiments.

The evaluation includes meaningful controls and qualifications. The SWE-bench transition, zero-effect controls, random baselines, and explicit distinction between point-estimate reversals and uncertainty-qualified conclusions strengthen the evidence. The paper also transparently reports missed flips and a false ranking alert.

Critical weaknesses

The proposed monitoring algorithm is insufficiently specified. 3.1 and 3.2 do not explain how contract atoms and execution dependencies are extracted, how the transition-specific risk score and risk mass are computed. Consequently, it is difficult to assess how much of the method is reusable versus manually designed for individual transitions.

While the experiments convincingly show that benchmark updates can change evaluation results, the effectiveness and efficiency of the proposed monitoring method are less clear. The heuristic selection method is primarily tested on a single benchmark update, while the other experiments use exact rules based on cached data. Although the method reduces the number of executions that need to be reevaluated, the paper does not compare its total computational cost against simply reevaluating all cached executions.

Questions for authors

Could you specify the dependency/risk rules and final selection procedure, including which components require manual, benchmark-specific, or transition-specific implementation? Which rules were fixed before evaluating the held-out transition?

What is the total cost of dependency extraction, selection, and replay compared with rescoring every cached execution? In the exact predicate cases, how much of the changed verifier computation is already performed during selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.