Agent Evaluation Reliability: More Tasks Won't (Always) Fix An Agent Leaderboard

Agent evaluations are increasingly used to compare models, assess capabilities, and inform deployment decisions, yet observed scores can reflect not only the model but also the effects of the evaluation conditions such as the scaffolds or tasks. This makes reliability claim-dependent: an evaluation that reliably ranks deployed systems may not reliably rank underlying models or produce reliable absolute scores. We ask which conclusions current agent evaluations reliably support and what additional evaluation would actually improve them. Using Generalizability theory, we develop a Bayesian variance-decomposition framework for sparse, imbalanced agent leaderboards and apply it to 22 benchmarks from the Holistic Agent Leaderboard and Harbor Index. The framework separates signal, performance differences relevant to the intended claim, from noise, irrelevant variation that can still change scores or rankings. We find four practical results: (1) Reliability depends on the measurement goal. Fixed model–scaffold systems are ranked reliably ($E{\rho }^2=0.935$–$0.994$), while underlying-model reliability is substantially lower ($0.148$–$0.841$). (2) Scaffold choice can change evaluation conclusions. We introduce inter-scaffold reliability, measuring whether scaffolds preserve model rankings, and show that scaffold effects vary substantially across evaluations. (3) More tasks cannot resolve all uncertainty. Even infinitely many similarly constructed tasks improve model-ranking reliability by at most $0.097$ when uncertainty is dominated by limited scaffold coverage. (4) Pooling diverse benchmarks can improve cross-task rankings at lower cost. For rankings across diverse agentic tasks, pooling benchmarks raises projected reliability from $0.44$ to $0.75$ at the same task budget and can reduce projected cost by up to $83\%$. Evaluation design should therefore follow the intended claim: practitioners should identify what a score or ranking should mean, diagnose what limits its reliability, and spend evaluation budget on the sources of uncertainty that matter.