- Rating
- 3(+2)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces TITRATE, an IRT-based approach to more efficiently estimate an agent's ability while evaluating it on as few tasks as possible. TITRATE first trains a standard IRT model on the not-held-out evaluation data, with a modification where they regularize the discrimination parameters in IRT to be closer to 1 (less confident); this is motivated by the classical psychometrics literature showing that when there are few test-takers, IRT's learned discrimination parameters are often too large. Then, for new test agents, they do the standard trick of continuously choosing the next task that will optimize Fischer information, but they use a novel stopping criterion based on the uncertainty in the prediction of the agent's score on the full benchmark. They test their method against other adaptive testing baselines like ATLAS and Fluid Benchmarking across 5 agentic benchmarks. Their results show that on most benchmarks, TITRATE is better than Atlas at predicting the performance of agents on the full benchmark while requiring fewer tasks to evaluate on. In addition to this main method, they also discuss ways of predicting the ability rankings of agents within a group, as well as obtaining more precise estimates of benchmark accuracy rate by running on certain tasks multiple times.
Critical strengths
- novel stopping criterion The authors correctly address that a weakness of prior adaptive testing approaches is that their stopping criterion is in terms of the IRT latent ability scores, whose scale is relative and thus uninterpretable. In contrast, TITRATE’s stopping criterion is based on error in the agent’s predicted performance on the full benchmark, a more interpretable metric.
- good efficient evaluation motivation The authors provide a bit of motivation for efficient evaluation methods that I haven't seen before: quick iteration for autoresearch agents.
Critical weaknesses
- Poor (AI-generated) writing quality, and lack of a cohesive story. The paper exhibits all the signs of AI-generated writing which severely impede understanding of what the authors actually tested, why they tested it, and what the main takeaways are. At a high level, the main setting of the paper is very unclear: at first, it seems like the goal is to do efficient evaluation of a single new agent using a subset of the data. But then in the middle of the introduction and methods section (section 3.5), the paper starts talking about rerunning on tasks, i.e. the evaluator has already run the agent on every task in the benchmark and still has more compute to spare. The paper also elusively talks about “cohorts” and “certificate” for ranking these cohorts; it’s not clear until methods section 3.6 that instead of the primary goal of trying to predict the agents’ full-benchmark performance, the paper is also interested in ranking the relative abilities of a group of agents. At a low level, the paper uses self-defined technical jargon without first explaining what it means, such as “"validation-selected shrinkage," “runs the bank once," "we include it as a panel." The fourth paragraph of the introduction is a microcosm for the whole paper: its main point is unclear; it’s very repetitive; there are numbers provided as evidence without proper explanation of what they mean or why the numbers are important. This paper was a miserable experience to read. Clear presentation is a very important factor in a paper besides just the experiments conducted. This paper would have to be rewritten and restructured by a human in order for me to consider accepting it.
- Quantitative results that don’t support the main claims of the paper. The main paper contains four tables. Table 1 is about experimental details, not results. Table 2 shows that for the smaller benchmarks — CORE, tau-bench, and Terminal-Bench 2.0 — getting within 2-4% error on full-benchmark accuracy takes evaluating over 80% of the tasks in the benchmark, which means not saving a very large factor of evaluation compute. Table 3's comparison to picking tasks uniformly at random would be more convincing if it used 1/5 of the dataset, as with 1/10 of the dataset it’s always using fewer tasks than TITRATE. Table 4 is an unfair comparison because of the metric used: for FLUID and Atlas, as the number of tasks approaches the entire benchmark, their rankings of the different agents approaches the IRT ability rankings, while TITRATE’s rankings approach the rankings of by the agents’ full-benchmark score; these rankings could be very different, hence why there are so many n.r.’s in the table. Neither set of rankings is objectively better than the other, but the Kendall's correlation metric used is based on full-benchmark score, making TITRATE look artificially good compared to FLUID and Atlas.
Questions for authors
Clarification requests:
- What is the main setting of the paper? Is TITRATE meant for a compute-bound regime where the evaluator wants to use a subset of the dataset or a compute-plentiful regime where they can run on the entire benchmark anymore? dependence assumption
- In Section 3.5, why do the authors assume that different runs of the same agent on the same task are dependent? Why are they not independent draws from a Bernoulli()? Being dependent implies that the outcome of the first attempt affects the outcome of the second attempt, which seems unlikely if the environment and the agent are reset between attempts.
Suggestions for future work:
- The authors' introduction proposes an interesting application of efficient evaluation methods to auto-research. It would be interesting to see if these efficient evaluation metrics are more useful than simply evaluating on a random subset in autoresearch.
5 comments
W1. Writing quality and lack of a cohesive story
We are revising the writing to clarify the main setting, explain the terminology, and improve the flow of the paper. The revised manuscript will be submitted before the end of the rebuttal period.
W2. Table 2: The savings on small benchmarks are limited
We agree that the savings on small benchmarks are modest. To assess how much room for savings remains, we consider a hypothetical evaluator given an advantage unavailable in practice: before selecting any tasks, it receives an estimate of the test agent’s ability obtained from that agent’s full benchmark results, at no evaluation cost. We call this evaluator a model-based oracle.
The oracle evaluates the tasks whose outcomes are hardest to predict (pass probabilities closest to 50%) and estimates the remaining outcomes using the known probabilities.
Under this model, reaching an expected 95% of estimates within four percentage points requires 25 of 45 tasks on CORE-bench Hard and 36 of 50 on τ-bench airline, compared with 30 and 41 for TITRATE on average. Even with this additional information, the oracle still needs a substantial fraction of the tasks.
W2. Table 3: How does TITRATE compare with random sampling at 20% or at a matched task count?
TITRATE outperforms uniform random sampling on all four benchmarks in both comparisons. This holds when both methods run 20% of the tasks and when random sampling receives the same average task budget as TITRATE’s stopping rule.
Each entry is the percentage of test agents whose estimated score is within four percentage points of their full-benchmark score. All methods use the same 30 splits and test agents.
| Benchmark | Random, 20% | TITRATE, 20% | Random, matched count | TITRATE, stopping rule |
|---|---|---|---|---|
| SWE-bench Verified | 67% | 91% | 71% | 94% |
| SWE-bench Lite | 53% | 95% | 55% | 94% |
| CORE-bench Hard | 26% | 54% | 63% | 95% |
| τ-bench airline | 27% | 41% | 81% | 97% |
- Random, 20%: Uniformly sample 20% of the tasks and use the observed pass rate as the score estimate.
- TITRATE, 20%: Let TITRATE select 20% of the tasks and estimate the full score from their outcomes.
- Random, matched count: Use the same random-sampling procedure, with the task count fixed at TITRATE’s mean stopping count on that benchmark.
- TITRATE, stopping rule: Let TITRATE select tasks and decide when to stop for each agent at the four-point tolerance.
W2. Table 4: The ranking target unfairly favors TITRATE
We agree and we thank the reviewer for pointing this out. We recomputed the table using each method’s own predicted pass rate. This correction required no retraining of the baselines and used the same test agents, selected tasks, and observed outcomes.
For Fluid, we take the ability estimate obtained from the selected tasks and use its fitted 2PL model to predict the agent’s pass probability on each remaining task. We add these probabilities to the number of observed passes and divide by the total number of benchmark tasks. For ATLAS, we use the full-benchmark pass rate returned by its released p-IRT estimator. We then rank agents by these predicted scores and compare each ranking with the official pass-rate ranking.
At each task count, we average Kendall’s τ over the 30 splits. The table reports the first task count at which this average reaches the target, using linear interpolation between evaluated counts.
| Benchmark | Target τ | Fluid | ATLAS | TITRATE |
|---|---|---|---|---|
| SWE-bench Verified | 0.90 | 57 | 63† | 45 |
| SWE-bench Verified | 0.95 | 208† | 198† | 188 |
| SWE-bench Lite | 0.90 | 79† | 76† | 48 |
| SWE-bench Lite | 0.95 | 156† | 162† | 107 |
| CORE-bench Hard | 0.90 | 25 | 26 | 23 |
| CORE-bench Hard | 0.95 | 34† | 35† | 32 |
| τ-bench airline | 0.90 | 44† | 40† | 34 |
| τ-bench airline | 0.95 | 47† | 45† | 43 |
| Terminal-Bench 2.0 | 0.90 | 19† | 17 | 16 |
| Terminal-Bench 2.0 | 0.95 | 51 | n.r. | 67 |
Entries are tasks per agent. † indicates that TITRATE needed significantly fewer tasks than the marked baseline in a paired bootstrap comparison. n.r. means the target was not reached.
Under the corrected comparison, most n.r. entries disappear. TITRATE still requires the fewest tasks in 9 of the 10 settings shown.
Thank you to the authors for the rebuttal.
I'm satisfied with the answers to the two clarification questions. Please make sure to change line 251 "Attempts of one agent on one task are not independent" in the rewrite of the paper to be more clear that it just means there's a model-scaffold interaction term that gets calibrated on repeated attempts.
I'm glad to see the technical improvements for Table 3 and Table 4. I'm now convinced that TITRATE's stopping condition makes a narrow improvement to previous IRT efficient evaluation works, though the work is still mostly incremental. For Table 2, I think it's reasonable to state in the limitations that one cannot hope for large efficiency savings on small benchmarks, as you showed in the rebuttal.
Until the revised manuscript is given, I will maintain my score, but depending on the new presentation, I am open to raising my score to a weak accept.
Response to Reviewer fPnd
We thank the reviewer for the detailed feedback. We will submit a revised manuscript before the end of the rebuttal period, with the main changes highlighted in blue. The changes described below will appear in that version.
Q1. What is the main setting? Is TITRATE intended for a compute-bound or a compute-plentiful regime?
TITRATE targets compute-bound evaluation. It estimates a new agent’s full-benchmark score from a subset of tasks, stopping when its calibrated error estimate meets the requested tolerance.
Agent outcomes can vary across attempts, so some benchmarks require reruns. Terminal-Bench 2.0 averages five attempts on each of 89 tasks. Running every task once therefore uses only 89 of the 445 runs in the official evaluation. Section 3.5 asks which tasks to rerun to reach the same score accuracy with fewer total runs than the official protocol, so this setting is also compute-bound.
Q2. Why model dependence between attempts if the agent and environment are reset?
We agree that one attempt does not affect the next after a reset. Our model also assumes that each attempt is an independent Bernoulli draw with the same success probability for that agent on that task.
What we do not know is that probability. An agent may perform well overall but consistently struggle with a particular task, for example because it cannot use a required tool. The standard 2PL model may miss this difficulty, so we add an agent–task effect shared by all attempts on that task. A failure gives us evidence that this agent finds the task harder than we predicted, and we therefore lower our prediction for the next attempt. The failure changes what we know about the agent, not how the next attempt runs. This dependence in our predictions is what we mean by correlation across attempts. We will clarify this distinction in Section 3.5.
The experiments support including this effect. On Terminal-Bench 2.0, it improves predictions on held-out data, with higher log likelihood on all 30 splits. Using it to select reruns also reduces score error relative to the binomial 2PL at all five tested budgets.