BenchPulse: Monitoring Verdict and Ranking Drift in Evolving Agent Benchmarks

Agent benchmarks are increasingly maintained as software: tasks are repaired, tools evolve, verifiers are corrected, and leaderboard formulas change. Yet model scores from different benchmark versions are routinely compared as if the measurement contract were fixed. We introduce BenchPulse, a framework that models benchmark repository transitions as interventions on a five-layer evaluation contract—task, verifier, environment, execution policy, and aggregation. A 4,119-commit census across 15 agent benchmarks maps how these contracts evolve. ContractIR represents the affected contract surface of a code change, while PulseSet uses cached execution dependencies and conclusion sensitivity to select a compact replay panel before observing new outcomes. Across 8,495 frozen development executions, real updates change 330 verdicts; a pre-specified held-out SWE-bench transition changes 17/642 additional frozen verdicts, and a separate 17-system audit changes 34 approximate 95% interval conclusions. A single verifier commit moves $\tau$-bench by -5.66 percentage points, a ${\tau }^2$-bench release moves individual model panels by up to 9.02 points and reverses 4 point-estimate orderings, and a BFCL aggregation change creates 893 strict point-order reversals among 5,886 model pairs, including 658 with at least a one-point gap under both contracts. Layer-aware monitoring re-evaluates 20.48% of cached executions and recovers 95.45% of changed verdicts; it recovers all observed full-panel point-order reversals with zero model calls. Two negative controls spanning 2,370 frozen executions produce no flips, showing that semantic-looking maintenance is not automatically consequential. These results establish benchmark evolution as a measurable source of reported-result drift and provide a practical continuous-evaluation protocol for versioned agent benchmarks.