acceptodds
Under review as a conference paper at ICLR 2027

TITRATE : Evaluating LLM agents with fewer executions

Abstract

Agentic benchmarks run an agent through multi-step tasks in real environments. Each run costs money and compute, and its outcome is random. Existing efficientevaluation methods fix a budget, typically 10% of the tasks, and report whatever accuracy results. An evaluator usually needs the opposite: given a target accuracy, how many runs are required? With 10% of the tasks, the share of agents scored within four points of their full-benchmark score ranges from 39% to 83% across five benchmarks, so the same budget buys very different reliability. We propose T ITRATE, which derives the number of runs from the tolerance and adapts item response theory to agentic benchmarks in three ways. Task parameters are fitted on the few dozen agents a benchmark has already evaluated, so T ITRATE shrinks the estimated discriminations to keep tasks that look informative by chance from being selected first. An agent–task random effect separates an agent’s lasting advantage on a task from run-to-run randomness, which tells T ITRATE which tasks are worth rerunning. Because the model’s own error estimate is optimistic, T ITRATE scales it by a factor learned by replaying the procedure on those agents, which turns a tolerance into a task count. Asked to place 95% of agents within four points of their full-benchmark score, T ITRATE places 91% to 97% on all five benchmarks under random splits. With 10% of the tasks, it estimates the full-benchmark score at least as accurately as six existing methods on every benchmark, and choosing which tasks to rerun matches uniform rerunning with about 20% fewer runs.

Primary area:
datasets and benchmarks
Keywords:
agent evaluation, efficient benchmarking, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fPndOpenReview ↗
Rating
3(+2)
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces TITRATE, an IRT-based approach to more efficiently estimate an agent's ability while evaluating it on as few tasks as possible. TITRATE first trains a standard IRT model on the not-held-out evaluation data, with a modification where they regularize the discrimination parameters in IRT to be closer to 1 (less confident); this is motivated by the classical psychometrics literature showing that when there are few test-takers, IRT's learned discrimination parameters are often too large. Then, for new test agents, they do the standard trick of continuously choosing the next task that will optimize Fischer information, but they use a novel stopping criterion based on the uncertainty in the prediction of the agent's score on the full benchmark. They test their method against other adaptive testing baselines like ATLAS and Fluid Benchmarking across 5 agentic benchmarks. Their results show that on most benchmarks, TITRATE is better than Atlas at predicting the performance of agents on the full benchmark while requiring fewer tasks to evaluate on. In addition to this main method, they also discuss ways of predicting the ability rankings of agents within a group, as well as obtaining more precise estimates of benchmark accuracy rate by running on certain tasks multiple times.

Critical strengths
  1. novel stopping criterion The authors correctly address that a weakness of prior adaptive testing approaches is that their stopping criterion is in terms of the IRT latent ability scores, whose scale is relative and thus uninterpretable. In contrast, TITRATE’s stopping criterion is based on error in the agent’s predicted performance on the full benchmark, a more interpretable metric.
  2. good efficient evaluation motivation The authors provide a bit of motivation for efficient evaluation methods that I haven't seen before: quick iteration for autoresearch agents.
Critical weaknesses
  1. Poor (AI-generated) writing quality, and lack of a cohesive story. The paper exhibits all the signs of AI-generated writing which severely impede understanding of what the authors actually tested, why they tested it, and what the main takeaways are. At a high level, the main setting of the paper is very unclear: at first, it seems like the goal is to do efficient evaluation of a single new agent using a subset of the data. But then in the middle of the introduction and methods section (section 3.5), the paper starts talking about rerunning on tasks, i.e. the evaluator has already run the agent on every task in the benchmark and still has more compute to spare. The paper also elusively talks about “cohorts” and “certificate” for ranking these cohorts; it’s not clear until methods section 3.6 that instead of the primary goal of trying to predict the agents’ full-benchmark performance, the paper is also interested in ranking the relative abilities of a group of agents. At a low level, the paper uses self-defined technical jargon without first explaining what it means, such as “"validation-selected shrinkage," “runs the bank once," "we include it as a panel." The fourth paragraph of the introduction is a microcosm for the whole paper: its main point is unclear; it’s very repetitive; there are numbers provided as evidence without proper explanation of what they mean or why the numbers are important. This paper was a miserable experience to read. Clear presentation is a very important factor in a paper besides just the experiments conducted. This paper would have to be rewritten and restructured by a human in order for me to consider accepting it.
  2. Quantitative results that don’t support the main claims of the paper. The main paper contains four tables. Table 1 is about experimental details, not results. Table 2 shows that for the smaller benchmarks — CORE, tau-bench, and Terminal-Bench 2.0 — getting within 2-4% error on full-benchmark accuracy takes evaluating over 80% of the tasks in the benchmark, which means not saving a very large factor of evaluation compute. Table 3's comparison to picking tasks uniformly at random would be more convincing if it used 1/5 of the dataset, as with 1/10 of the dataset it’s always using fewer tasks than TITRATE. Table 4 is an unfair comparison because of the metric used: for FLUID and Atlas, as the number of tasks approaches the entire benchmark, their rankings of the different agents approaches the IRT ability rankings, while TITRATE’s rankings approach the rankings of by the agents’ full-benchmark score; these rankings could be very different, hence why there are so many n.r.’s in the table. Neither set of rankings is objectively better than the other, but the Kendall's correlation metric used is based on full-benchmark score, making TITRATE look artificially good compared to FLUID and Atlas.
Questions for authors

Clarification requests:

  1. What is the main setting of the paper? Is TITRATE meant for a compute-bound regime where the evaluator wants to use a subset of the dataset or a compute-plentiful regime where they can run on the entire benchmark anymore? dependence assumption
  2. In Section 3.5, why do the authors assume that different runs of the same agent on the same task are dependent? Why are they not independent draws from a Bernoulli(pijp_{ij})? Being dependent implies that the outcome of the first attempt affects the outcome of the second attempt, which seems unlikely if the environment and the agent are reset between attempts.

Suggestions for future work:

  • The authors' introduction proposes an interesting application of efficient evaluation methods to auto-research. It would be interesting to see if these efficient evaluation metrics are more useful than simply evaluating on a random subset in autoresearch.
5 comments
AuthorsOpenReview ↗

Response to Reviewer fPnd

We thank the reviewer for the detailed feedback. We will submit a revised manuscript before the end of the rebuttal period, with the main changes highlighted in blue. The changes described below will appear in that version.

Q1. What is the main setting? Is TITRATE intended for a compute-bound or a compute-plentiful regime?

TITRATE targets compute-bound evaluation. It estimates a new agent’s full-benchmark score from a subset of tasks, stopping when its calibrated error estimate meets the requested tolerance.

Agent outcomes can vary across attempts, so some benchmarks require reruns. Terminal-Bench 2.0 averages five attempts on each of 89 tasks. Running every task once therefore uses only 89 of the 445 runs in the official evaluation. Section 3.5 asks which tasks to rerun to reach the same score accuracy with fewer total runs than the official protocol, so this setting is also compute-bound.

Q2. Why model dependence between attempts if the agent and environment are reset?

We agree that one attempt does not affect the next after a reset. Our model also assumes that each attempt is an independent Bernoulli draw with the same success probability for that agent on that task.

What we do not know is that probability. An agent may perform well overall but consistently struggle with a particular task, for example because it cannot use a required tool. The standard 2PL model may miss this difficulty, so we add an agent–task effect shared by all attempts on that task. A failure gives us evidence that this agent finds the task harder than we predicted, and we therefore lower our prediction for the next attempt. The failure changes what we know about the agent, not how the next attempt runs. This dependence in our predictions is what we mean by correlation across attempts. We will clarify this distinction in Section 3.5.

The experiments support including this effect. On Terminal-Bench 2.0, it improves predictions on held-out data, with higher log likelihood on all 30 splits. Using it to select reruns also reduces score error relative to the binomial 2PL at all five tested budgets.

W1. Writing quality and lack of a cohesive story

We are revising the writing to clarify the main setting, explain the terminology, and improve the flow of the paper. The revised manuscript will be submitted before the end of the rebuttal period.

W2. Table 2: The savings on small benchmarks are limited

We agree that the savings on small benchmarks are modest. To assess how much room for savings remains, we consider a hypothetical evaluator given an advantage unavailable in practice: before selecting any tasks, it receives an estimate of the test agent’s ability obtained from that agent’s full benchmark results, at no evaluation cost. We call this evaluator a model-based oracle.

The oracle evaluates the tasks whose outcomes are hardest to predict (pass probabilities closest to 50%) and estimates the remaining outcomes using the known probabilities.

Under this model, reaching an expected 95% of estimates within four percentage points requires 25 of 45 tasks on CORE-bench Hard and 36 of 50 on τ-bench airline, compared with 30 and 41 for TITRATE on average. Even with this additional information, the oracle still needs a substantial fraction of the tasks.

W2. Table 3: How does TITRATE compare with random sampling at 20% or at a matched task count?

TITRATE outperforms uniform random sampling on all four benchmarks in both comparisons. This holds when both methods run 20% of the tasks and when random sampling receives the same average task budget as TITRATE’s stopping rule.

Each entry is the percentage of test agents whose estimated score is within four percentage points of their full-benchmark score. All methods use the same 30 splits and test agents.

BenchmarkRandom, 20%TITRATE, 20%Random, matched countTITRATE, stopping rule
SWE-bench Verified67%91%71%94%
SWE-bench Lite53%95%55%94%
CORE-bench Hard26%54%63%95%
τ-bench airline27%41%81%97%
  • Random, 20%: Uniformly sample 20% of the tasks and use the observed pass rate as the score estimate.
  • TITRATE, 20%: Let TITRATE select 20% of the tasks and estimate the full score from their outcomes.
  • Random, matched count: Use the same random-sampling procedure, with the task count fixed at TITRATE’s mean stopping count on that benchmark.
  • TITRATE, stopping rule: Let TITRATE select tasks and decide when to stop for each agent at the four-point tolerance.

W2. Table 4: The ranking target unfairly favors TITRATE

We agree and we thank the reviewer for pointing this out. We recomputed the table using each method’s own predicted pass rate. This correction required no retraining of the baselines and used the same test agents, selected tasks, and observed outcomes.

For Fluid, we take the ability estimate obtained from the selected tasks and use its fitted 2PL model to predict the agent’s pass probability on each remaining task. We add these probabilities to the number of observed passes and divide by the total number of benchmark tasks. For ATLAS, we use the full-benchmark pass rate returned by its released p-IRT estimator. We then rank agents by these predicted scores and compare each ranking with the official pass-rate ranking.

At each task count, we average Kendall’s τ over the 30 splits. The table reports the first task count at which this average reaches the target, using linear interpolation between evaluated counts.

BenchmarkTarget τFluidATLASTITRATE
SWE-bench Verified0.905763†45
SWE-bench Verified0.95208†198†188
SWE-bench Lite0.9079†76†48
SWE-bench Lite0.95156†162†107
CORE-bench Hard0.90252623
CORE-bench Hard0.9534†35†32
τ-bench airline0.9044†40†34
τ-bench airline0.9547†45†43
Terminal-Bench 2.00.9019†1716
Terminal-Bench 2.00.9551n.r.67

Entries are tasks per agent. † indicates that TITRATE needed significantly fewer tasks than the marked baseline in a paired bootstrap comparison. n.r. means the target was not reached.

Under the corrected comparison, most n.r. entries disappear. TITRATE still requires the fewest tasks in 9 of the 10 settings shown.

Thank you to the authors for the rebuttal.

I'm satisfied with the answers to the two clarification questions. Please make sure to change line 251 "Attempts of one agent on one task are not independent" in the rewrite of the paper to be more clear that it just means there's a model-scaffold interaction term that gets calibrated on repeated attempts.

I'm glad to see the technical improvements for Table 3 and Table 4. I'm now convinced that TITRATE's stopping condition makes a narrow improvement to previous IRT efficient evaluation works, though the work is still mostly incremental. For Table 2, I think it's reasonable to state in the limitations that one cannot hope for large efficiency savings on small benchmarks, as you showed in the rebuttal.

Until the revised manuscript is given, I will maintain my score, but depending on the new presentation, I am open to raising my score to a weak accept.

Reviewer_UJRVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors propose a method to compress evaluations on a benchmark by identifying the number of runs conditioned on the tolerance provided to estimate the performance on the benchmark. This is a well motivated paper & is a need in the current setting, where several benchmarks may take a long time to run or may be cost intensive. Overall, the algorithm identifies right tasks by pruning tasks that are spurious, scales the error estimate by replaying procedures on identified agents.

Critical strengths

I see the following strengths in the paper: 1/ It is is a well motivated paper and addresses a clear problem. Today, several benchmarks take a long time to run and are also expensive. 2/ The design seems to be well crafted and experimental choices are well justified. This provides rigor & provides clear direction to the practitioner. 3/ The results seem to justify the approach well & are presented in a manner that justify the strengths of the algorithm.

Critical weaknesses

I see the following weaknesses: 1/ Several evaluations in the wild are not benchmarks (e.g., live feedback on platforms such as WedDevArena or beta feedback from a selected customer pool) & benchmarks themselves have flaws. Therefore, while applying this method to existing benchmarks helps in running them efficiently, it may not solve fundamental evaluation problems.

2/ I am unsure of generalization of this method to unseen benchmarks. The algorithm seems to work well when applied per benchmark & calibrated within. However, if a new benchmark comes & it does not satisfy the requirements that experimented benchmarks in the paper satisfy, it remains unclear if the method will generalize.

Questions for authors

N/A

1 comment

We thank the reviewer for the thoughtful feedback and the positive assessment of our motivation and experimental design.

W1. Applicability outside benchmarks

We agree that live feedback and customer evaluations can reveal aspects of agent performance that benchmarks miss. TITRATE addresses a narrower question: once a benchmark has been chosen, how many runs are needed to estimate a new agent's full-benchmark score to a specified accuracy? It reduces the cost of that evaluation but does not establish whether the benchmark reflects real-world use or correct flaws in its design. We will make this scope explicit in the revised introduction.

W2. Generalization to new benchmarks

The default TITRATE procedure requires previous agents' results on the same benchmark. These results are used to learn task pass probabilities and calibrate when evaluation can stop. On a new benchmark with no previous results, this procedure cannot yet be calibrated. The IRT baselines also require previous results to fit their task models.

To address this case, we added a randomized variant that reduces to uniform task sampling when no historical results are available. It uses the new agent's sampled outcomes to estimate the score and its uncertainty and decide when to stop. As historical results become available, the model can guide sampling toward informative tasks and reduce the number of runs needed.

We tested the absence of history by hiding all previous agents' results on SWE-bench Verified and Lite. This simulates the lack of historical data. Each entry below gives the percentage of evaluations whose estimated score is within four percentage points of the full-benchmark score, followed by the average number of tasks evaluated. The target percentage is 95%.

ProcedureVerified (500 tasks)Lite (300 tasks)
Randomized TITRATE, no history95% (250)96% (184)
Randomized TITRATE, full history97% (138)97% (115)
Default TITRATE, full history94% (112)94% (63)

The randomized procedure also works without historical results: 95% of evaluations on Verified and 96% on Lite produce score estimates within four percentage points of the full-benchmark score.

We will upload a revised PDF before the end of the rebuttal period, including these explanations and results, with the main changes highlighted in blue.

Reviewer_m7aLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces TITRATE, an adaptive framework for evaluating LLM agents with fewer benchmark executions. It fits a IRT model to historical agent-task outcomes, selects tasks based on their expected information, and stops when a calibrated score-error target is reached. It also models correlated repeated attempts and supports adaptive reruns. Experiments on SWE-bench, CORE-bench, τ-bench, and Terminal-Bench show TITRATE can estimate full-benchmark scores with substantially fewer runs than fixed-budget and prior adaptive methods. It also reduces repeated-attempt evaluations by about 20%.

Critical strengths
  1. The paper tries to address an important practical problem: agent evaluations are expensive, while fixed-size task subsets provide inconsistent score accuracy. TITRATE directly controls the desired score error and adaptively determines how many tasks to run.

  2. The method is evaluated across several agent benchmarks and explicitly models correlations between repeated attempts on the same task. Its consistent reduction in the number of executions, including roughly 20% fewer repeated runs, demonstrates practical value beyond a purely methodological proposal.

  3. The paper considers several practical evaluation settings, including per-agent score estimation, repeated attempts, and cohort ranking, instead of treating them as a single evaluation problem.

Critical weaknesses
  1. The method’s reliability depends on the evaluated agents being drawn from the historical calibration distribution. Under time-based splits with later and stronger agents, the per-agent rule achieves only 0.85–0.88 of the stated 0.95 coverage, and the cohort certificate frequently misses its target. This weakens the main reliability claim for rapidly improving frontier models.

  2. The efficiency claims are validated only in a limited scope: adaptive reruns are tested on releases from one benchmark family, while monetary savings are measured for only 35 SWE-bench submissions. It remains unclear whether the reported savings generalize across benchmarks, agent configurations, and different cost structures.

  3. TITRATE also requires a sufficiently large and representative historical response matrix. With only a small number of prior agents, task-discrimination estimates may be driven by sampling noise, causing maximum-information selection to prioritize spurious tasks. This limits the method’s applicability to new benchmarks or rapidly changing task pools.

Questions for authors
  1. How does TITRATE handle a new agent whose ability is outside the historical calibration distribution?

  2. How many historical agents are needed for reliable calibration on a new benchmark?

  3. Do the reported rerun and cost savings generalize beyond Terminal-Bench and the evaluated SWE-bench submissions?

3 comments
AuthorsOpenReview ↗

Response to Reviewer m7aL

We thank the reviewer for the thoughtful feedback. We address each question together with the corresponding weakness below. We will upload a revised PDF before the end of the rebuttal period, including the additional results reported here and highlighting the main changes in blue.

Q1 / W1. How does TITRATE handle agents outside the historical calibration distribution?

We extend TITRATE naturally to this setting with a randomized variant. It randomly selects the next task, giving higher selection probabilities to tasks whose outcomes the model is less certain about. Every unevaluated task retains a chance of being selected. The new agent's observed prediction errors correct the full-benchmark score, weighted to account for unequal sampling probabilities. Stopping uses uncertainty in these corrected estimates instead of transferring an error-calibration factor from historical agents. The interval remains approximate.

Time shifts and stronger agents. We first used the same chronological splits discussed in the review: six cutoffs, training on the earliest 50–75% of submissions and testing on the next 25%. Separately, we held out the strongest 25% of agents and trained on random 90% subsets of the remaining agents over 30 repetitions.

The table reports results for the randomized variant. Coverage is the percentage of estimates within four percentage points of the full score; the target is 95%. Task counts are means.

Benchmark (tasks)Test splitCoverageMean tasks evaluated
Verified (500)Later submissions95.0%129
Lite (300)Later submissions96.0%122
Verified (500)Strongest quarter96.5%108
Lite (300)Strongest quarter95.1%129
CORE (45)Strongest quarter90.6%42
τ-bench (50)Strongest quarter100.0%50

On both SWE-bench benchmarks, the randomized variant reaches approximately 95% coverage under time shifts and when testing stronger agents, while evaluating substantially fewer tasks than the full benchmark. On Terminal-Bench 2.0, it uses only 148–157 of 445 runs, although coverage remains below the target at 92.5–93.4%.

Savings are more limited on small benchmarks because each task has a larger effect on the full score: one outcome changes the score by 2.22 percentage points on CORE and 2 points on τ-bench. At a four-point tolerance, only a few incorrectly predicted outcomes can therefore exceed the error budget. This helps explain why many tasks still need to be evaluated. CORE's below-target coverage also shows that the approximate stopping rule remains imperfect in this setting.

Note that the 138-task Verified average in our response to UJRV uses random agent splits; the 108-task average here uses only the strongest quarter. They concern different test populations and training pools, so they are not a comparison of the effect of distribution shift on the same agents.

Q2 / W3. How many historical agents are needed for reliable calibration on a new benchmark?

We have not established a minimum historical-pool size that ensures 95% stopping coverage on a new benchmark. Our implementation needs at least 12 historical agents to perform its internal calibration, but this is an implementation minimum, not an empirical reliability threshold.

Our additional experiment measures the data needed for score prediction, not stopping coverage. Over 30 random 75/25 agent splits, we trained the current TITRATE procedure on progressively larger subsets of the historical pool, keeping the test agents fixed. All fitting and hyperparameter selection used only those historical agents. Entries below are mean absolute score errors in percentage points at a fixed budget of approximately 10% of the tasks.

BenchmarkTasks evaluated5 historical agents152030Full historical pool
SWE-bench Verified504.762.542.462.342.34 (130 agents)
SWE-bench Lite304.532.792.592.432.38 (63 agents)
CORE-bench Hard410.167.977.427.586.91 (49 agents)
τ-bench airline59.248.077.357.897.58 (35 agents)
Terminal-Bench 2.097.904.484.384.214.43 (95 agents)

On Verified, Lite, and Terminal-Bench, 15–20 historical agents bring prediction error close to the full-pool result: the upper end of the paired 95% interval for the additional error is below 0.5 points. This does not establish reliable stopping with 15–20 agents; we have not measured stopping coverage across these pool sizes.

CORE and τ-bench are small, so each task carries a large weight in the full score. The 10% budget reveals only 4–5 outcomes, making accurate full-score prediction difficult. ATLAS and Fluid also show substantial prediction errors at this budget.

Q3 / W2. Do the rerun and cost savings generalize beyond Terminal-Bench and the evaluated SWE-bench submissions?

Rerun savings on an additional benchmark. On the newly added Toolathlon benchmark (Li et al., 2025), TITRATE uses 20% fewer total executions than uniform rerunning at similar observed score error (216 versus 270 executions).

Toolathlon contains 22 models, 108 tasks, and three recorded attempts per task. Across 30 random 75/25 agent splits, both methods first run every task once, then allocate additional attempts either uniformly or adaptively. Both use the same repeat-aware score estimator, so only the rerun allocation differs. The table reports score MAE in percentage points (lower is better); execution counts include the initial 108 runs.

Total executionsUniform rerun allocationTITRATE allocation
1621.350.93
2160.880.55
2590.640.22
2700.560.13

At each equal budget shown, TITRATE's error is significantly lower in paired 95% bootstrap intervals over splits.

Estimated dollar savings on Toolathlon. Only 14 Toolathlon models have usable public cost records covering every task, so we restrict the cost analysis to these models. We estimate evaluation cost by multiplying each task's mean recorded cost per attempt by the number of attempts allocated to it, including the initial run. Across the same 30 splits, TITRATE at 216 runs has 25.4% lower estimated dollar cost than uniform rerunning at 270 runs, averaging relative savings across models. Their MAEs on these 14 models are 0.55 and 0.57 points, respectively. Compared with full evaluation (324 runs), TITRATE reduces the estimated dollar cost by 38.1%.

I still have concerns about stopping reliability and calibration with few historical agents. The stronger-agent and Toolathlon results do strengthen the paper, though, so I’ll raise my overall rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.