acceptodds
Under review as a conference paper at ICLR 2027

Task-efficient agent evaluation using cached traces

Abstract

Efficient benchmarking estimates a system's full benchmark performance from a small number of task executions. Standard approaches rely solely on the binary correctness of executions from a collection of already-evaluated reference systems. An agentic system, however, produces a rich trace for each execution that can reveal substantially more about its capabilities. Leveraging this information requires useful trace representations. We propose representing a trace by extracting a structured summary of it with an LLM and embedding the summary with a standard text encoder. These representations preserve correctness- and task-related signal while minimizing the presence of system identity-, model family-, and harness-related signals that dominate raw trace embeddings. Given this representation for each task execution from all reference systems, we construct low-dimensional representations of the systems, and predict a new system's score from its position among the reference systems. On SWE-bench Verified and Terminal-Bench 2.0, augmenting an item-response model with this geometry-based prediction reduces mean absolute error from 0.13 to 0.08 (Wilcoxon ) and from 0.14 to 0.12 (Wilcoxon ), respectively, when provided a single random executed task. Further, our approach dominates sample average and IRT-only methods for any given number of executed tasks for both datasets.

Primary area:
generative models
Keywords:
evaluation, agents, trace embeddings

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_fjbcOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The contributions of this paper lie along two axes:

  1. Trace Representation: The authors suggest the trubric methodology, which compacts long traces using specified rubrics whose fields are designed for task-wise verification. This methodology helps retain task and correctness signals while reducing system identity, model family, and harness signals.

  2. Efficient Benchmarking: The authors combine IRT and the Product Kernel Perspective Space (PKPS) to estimate the full-benchmark accuracy of a new agent without running all benchmark tasks. It shows better performance than other previous setups, but I'd suggest including better baselines that were previously used.

Critical strengths

S1. The paper combines two research directions, trace representation and efficient benchmarking, and shows a performance improvement in agent evaluation in a task-efficient way. The authors demonstrate this empirically with two different benchmarks and various setups.

S2. The motivation of this paper is clear, and the methodology aligns soundly with it. Using the information in long trajectories for agent evaluation is important, and they suggest a new method, trubric, to summarize it. They used the embedded trubric results and IRT for efficient benchmarking.

Critical weaknesses

W1. The alternative methods considered in this paper seem weak, and other alternatives could be discussed. For example, Table 1 only includes raw-trace and generic results, not IRT+raw or IRT+generic. Also, this paper uses PKPS (a variation of DKPS) instead of DKPS, while Helm et al. (2026a) already showed that IRT+DKPS is powerful for efficient benchmarking (although they showed this only for responses). A comparison with IRT+DKPS is required to show the novelty of this paper.

W2. The model and harness tags of the systems are unclear. Some systems are unclassified, and some are misclassified. For example, the lingma-* systems are misclassified as gpt, and forgecode/forge-code and logos/logosagent are classified as different harnesses.

Questions for authors

Q1. Can you update Table 1 with IRT + raw or IRT + generic, optimizing α\alpha by cross-validation? Since IRT + trubric may include an ensemble effect, IRT + raw or IRT + generic comparisons would be useful to better understand the performance of trubric.

Q2. Unlike DKPS (Helm et al., 2026a), PKPS uses the reference systems’ traces on all MM tasks via the task kernel in Eq. (1). Does this actually improve accuracy? Could the authors compare PKPS, at the same mm, against a DKPS baseline restricted to the target’s tasks T0T_0? If the gain is small, the geometry component reduces to DKPS on trubric embeddings.

Q3. According to the appendix, the model tags used in the experiments include many '–' entries. How were these systems handled in the leave-one-family-out experiment? Almost 50% of the systems in SWE-bench fall into this category. Also, the lingma-agent lingma-swe-gpt-72b / 7b systems are misclassified as gpt. Can you show the results either after reclassifying systems or after removing same-family systems in a conservative way?

Q4. L345-348 state that this method is applicable in settings where grading target tasks is infeasible, but the prompts in Appendix 3 (L810-817, L833-834) specify verification tests in the rubrics. The claim in the main text may be true when soft (or partial) evaluation or verification is possible. (I think the correctness results of trubric in Figure 1 also show that the rubric can partially include correctness verification.) Can you report how y^geom\hat{y}_{geom} performs when the verification and final-state fields are removed from the rubric, i.e., when the summaries no longer include the agents' observed test outcomes?

Q5. Can you provide a brief summary of the PKPS paper (Anonymous, 2026), e.g., its abstract and main results, so that the contribution of the present submission can be assessed relative to that work? If there is a theoretical guarantee for PKPS (analogous to the task-efficiency result of Helm et al., 2026a for DKPS), please also summarize it.

Additional Comments

A1. Other methodologies that generate task-specific rubrics from task descriptions are not cited, e.g., TICK (Cook et al., arXiv 2410.03608, NeurIPS 2024 workshop) and BiGGen Bench (Kim et al., NAACL 2025).

4 comments
AuthorsOpenReview ↗

Initial response (1/2)

Thank you for taking the time to review our paper. We think your summary of our work is fair. We appreciate you noting that we empirically demonstrate a performance improvement in a task-efficient way and that our motivation is clear. We attempt to address your weaknesses / questions inline below:

Additional baselines

We have run the suggested baselines -- IRT ++ raw and IRT ++ generic rubric and also included a DKPS-based trubric with and without IRT -- through the exact pipeline of Table 1. The trubric (DKPS) rows use the same trubric embeddings as the trubric (PKPS) rows, so the only difference between them is the geometry construction (see "PKPS" below). MAE at mm tasks:

SWE-bench Verified:

random m=1m=1m=5m=5m=20m=20adaptive m=1m=1m=5m=5m=20m=20
IRT0.1270.0880.0480.0800.0620.034
raw trace0.0990.0930.0890.0960.0830.081
++ IRT0.0970.0770.0470.0700.0560.035
generic0.0790.0710.0680.0740.0700.069
++ IRT0.0790.0650.0450.0680.0530.034
trubric (DKPS)0.0980.0730.0620.0760.0710.057
++ IRT0.0960.0660.0440.0690.0530.034
trubric (PKPS)0.0810.0690.0600.0680.0560.060
++ IRT0.0810.0630.0430.0620.0480.034

Terminal-Bench 2.0:

random m=1m=1m=5m=5m=20m=20adaptive m=1m=1m=5m=5m=20m=20
IRT0.1370.0970.0520.1120.0670.039
raw trace0.1350.1260.1200.1280.1250.113
++ IRT0.1270.0950.0510.1070.0660.039
generic0.1390.1270.1150.1440.1300.117
++ IRT0.1300.0950.0510.1120.0670.040
trubric (DKPS)0.1480.1250.0980.1300.0950.082
++ IRT0.1340.0950.0510.1100.0650.038
trubric (PKPS)0.1270.1070.0930.1100.0920.084
++ IRT0.1220.0900.0500.0990.0590.037

You are right that part of the trubric (PKPS) ++ IRT improvement is an ensemble effect. With that said, the trace representation (raw vs generic vs trubric) and the geometry construction (PKPS vs DKPS) still matter: trubric (PKPS) ++ IRT is the best predictor in 10 of the 12 (benchmark, selection, budget) settings and within 0.002 of the best in the remaining two. We will add these rows to Table 1 (or, if space does not permit, to an appendix table referenced from Table 1).

Audited / corrected model and harness tags

Thank you for catching these. We have since audited the original table and found three main kinds of issues: substring-matching errors (e.g., one of the lingma agents matched the "gpt" keyword), fine-tuned models improperly assigned (FrogBoss is fine-tuned from Qwen3 and so is now assigned to Qwen), and provider normalization (e.g., merging the "o1" and the "o3" families).

We re-ran our analysis after correcting these issues. The results on SWE-bench are essentially unchanged for m=1/5/20m=1/5/20: IRT 0.127/0.088/0.048 →\to 0.126/0.086/0.047, IRT ++ trubric 0.081/0.063/0.043 →\to 0.079/0.063/0.042. On Terminal-Bench 2.0 the corrections were more consequential: IRT 0.137/0.097/0.052 →\to 0.145/0.102/0.053 and IRT ++ trubric 0.122/0.090/0.050 →\to 0.134/0.098/0.053, but the ordering of the methods is unchanged. The harness merges change MAE by at most 0.001 on either benchmark. We will update our analysis using the corrected tags and do not expect any of our conclusions to change.

PKPS

The PKPS paper decomposes the definition of the (dis)similarity between models that DKPS uses into the product of a query kernel and a response kernel: the query kernel κ(t,t′)\kappa(t, t') compares the tasks, and the response kernel ⟨ϕ(xat),ϕ(xbt′)⟩\langle \phi(x_{at}), \phi(x_{bt'}) \rangle compares the embedded traces produced on those tasks. From this perspective, DKPS uses the indicator query kernel κ(t,t′)=1{t=t′}\kappa(t, t') = \mathbb{1}\{t = t'\} -- so it can only include information from tasks that both systems have executed and cannot use information from cached reference traces for tasks the target agent has not executed. Changing the query kernel to a strictly positive definite kernel, such as the RBF kernel on task-description embeddings, allows us to use information from all existing reference traces, weighted by task similarity. Eq. (1) is this construction applied to the trubric embeddings.

The main theoretical result of the PKPS paper is that PKPS is query-efficient relative to DKPS when the queries / tasks that the reference systems have responded to are highly unpaired. The main empirical result is that PKPS dominates DKPS on HELM Lite [Liang et al., 2023] and Every Eval Ever [Batzner et al., 2026]. With the DKPS vs PKPS results above, the results in this paper extend those observations to the agent evaluation setting via trubric.

AuthorsOpenReview ↗

Initial response (2/2)

Description of how we handled untagged models in the leave-one-family-out experiment

A system with no model tag is treated as its own (singleton) family. So under leave-one-family-out, only the system itself is excluded from its reference pool. Most "--" entries are scaffold submissions that do not disclose the underlying model (e.g., tags of "Undisclosed" or "Multiple", which we also map to untagged). Where the underlying model is publicly documented we have now filled the tag in as part of the audit described above.

We acknowledge that the remaining undisclosed systems could in principle share a family with their references, which would make the protocol slightly optimistic for those targets. As mentioned above, the difference in the average MAE between the original and corrected tags is small on SWE-bench Verified (at most 0.003 at any budget) and does not change any ordering on Terminal-Bench 2.0, so we do not believe residual mistagging would change our qualitative conclusions.

Removing verification and end state sections from trubric

We ran this ablation in response to Reviewer zsGC (please see that response for the full table). Summarizing: with the verification and final-state fields removed (so the summaries no longer include the agents' observed test outcomes), trubric degrades modestly but our main conclusions hold.

Other methods for generating task-specific rubrics

Thank you for the pointers -- we will cite and discuss both TICK (Cook et al.) and BiGGen Bench (Kim et al., NAACL 2025) in the related work.


Thank you, again, for taking the time to review our paper. Please let us know if we were unable to address your concerns!

References

  • Anonymous. Benchmark Prediction with Sparse and Unpaired Cached Responses. Under review, 2026. (The PKPS paper; cited in the submission as Anonymous, 2026.)
  • Batzner, J., Nelaturu, S. H., Stachura, D., et al. Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results. arXiv:2606.14516, 2026.
  • Cook, J., Rocktäschel, T., Foerster, J., Aumiller, D., and Wang, A. TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation. arXiv:2410.03608, 2024.
  • Kim, S., et al. The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models. NAACL, 2025.
  • Liang, P., Bommasani, R., Lee, T., et al. Holistic Evaluation of Language Models. Transactions on Machine Learning Research, 2023. (HELM; the HELM Lite scenarios.)

Thank you for the detailed response, the additional experiments, and the tag audit. The new results are informative: for example, PKPS clearly improves over DKPS at m = 1, and the ensemble reduces tail errors in the one-task regime. After reading the response, however, I still have some questions about the distinct contribution of this paper, which I summarize below.

1. Ensemble effect. The new baselines suggest that most of the improvement over IRT comes from ensembling. With IRT added, trubric differs from generic by at most 0.006 on SWE-bench Verified (e.g., generic + IRT 0.079/0.065/0.045 vs. trubric + IRT 0.081/0.063/0.043 at random m = 1/5/20, with generic + IRT better at m = 1), and from raw by at most 0.008 on Terminal-Bench 2.0. No significance tests are reported for these comparisons. Please report paired tests of trubric + IRT against raw + IRT and generic + IRT.

2. Tags. The tag issue remains. Untagged systems are treated as singleton families, but their underlying models are unknown. So whenever untagged systems are involved, either as the target or as references, the protocol cannot ensure that systems sharing the target's model family are excluded from the reference pool (and likewise for the agent/scaffold under leave-one-harness-out). In the original table, roughly half of the SWE-bench Verified systems are untagged, so similar systems may remain in many reference pools. The tag-correction results address only the re-tagged systems, not the remaining untagged ones. Unless untagged systems are excluded, the reported results cannot be regarded as a strict leave-one-family-out evaluation. Please rerun the main comparison with all untagged systems excluded from both target and reference pools, and confirm whether the new tables use the corrected tags.

3. Relation to the PKPS paper (Q5). Thank you for the summary. In my review, I listed the efficient-benchmarking component (combining IRT with PKPS-based prediction) as one of the two main contributions and a strength (S1). Whether this holds depends on its relation to the PKPS paper (Anonymous, 2026). Please state precisely which methodological components and experimental settings are shared with the PKPS paper, and which are new in this submission. If the efficient-benchmarking pipeline is largely shared, the distinct contribution would rest mainly on the trubric representation, which makes points 1 and 2 above more important.

Thank you for taking the time to respond / acknowledge our rebuttal!

Ensemble effect

Here are the p-values from Wilcoxon's paired signed-rank test for IRT + trubric versus IRT + raw and IRT + generic:

random:

p-valuem=1m=1m=3m=3m=5m=5m=10m=10m=20m=20
SWE-bench: vs. IRT + raw2.5×10−42.5 \times 10^{-4}8.4×10−68.4 \times 10^{-6}5.7×10−85.7 \times 10^{-8}1.3×10−81.3 \times 10^{-8}1.0×10−61.0 \times 10^{-6}
SWE-bench: vs. IRT + generic0.130.980.520.0851.2×10−31.2 \times 10^{-3}
Terminal-Bench: vs. IRT + raw0.490.0950.0140.0100.013
Terminal-Bench: vs. IRT + generic0.230.0580.0380.0145.8×10−35.8 \times 10^{-3}

adaptive queries:

p-valuem=1m=1m=3m=3m=5m=5m=10m=10m=20m=20
SWE-bench: vs. IRT + raw0.0150.0447.2×10−47.2 \times 10^{-4}0.250.51
SWE-bench: vs. IRT + generic0.0220.610.0260.620.50
Terminal-Bench: vs. IRT + raw0.179.6×10−39.6 \times 10^{-3}0.0320.250.31
Terminal-Bench: vs. IRT + generic0.0310.260.0190.420.19

IRT + trubric is significantly better (per a 0.05 threshold) than IRT + generic on 8 of 20 and IRT + raw on 13 of 20 of the (benchmark, number of queries, query type) tuples. It is never significantly worse than either.

The meta p-values for these comparisons (per Fisher's method) are 6.8×10−76.8 \times 10^{-7} and 3.9×10−283.9 \times 10^{-28} for trubric versus generic and trubric versus raw, respectively. Lastly, we note that we are the first (that we are aware of) to apply either trubric or generic rubrics to construct embeddings of the agents based on cached traces and to use them for efficient benchmarking.

Tags

We have removed all untagged models -- which the original protocol treated as singleton families -- from the leave-one-family-out analysis. We note that this approach is pretty conservative: it removes 42 of the 107 SWE-bench Verified systems (largely commercial scaffold submissions whose underlying model is undisclosed, e.g., solver, nfactorial, emergent, devlo, warp, cortexa) and 1 of the 65 Terminal-Bench 2.0 systems (JJAgent, which reports "Multiple"). The leave-one-family-out exclusions for the remaining systems are built from the corrected tags.

In any case, here are the results for IRT, trubric, and IRT + trubric (lowest number in a column is bolded if it is significantly better than the second best):

SWE-bench Verified:

MAErandom m=1m=1m=5m=5m=20m=20adaptive m=1m=1m=5m=5m=20m=20
IRT0.1450.0960.0530.1140.0620.039
trubric0.1110.0970.0930.0960.0910.098
IRT + trubric0.1110.0860.0560.0960.0650.051

Terminal-Bench 2.0:

MAErandom m=1m=1m=5m=5m=20m=20adaptive m=1m=1m=5m=5m=20m=20
IRT0.1490.1060.0560.1270.0720.039
trubric0.1440.1240.1140.1350.1090.107
IRT + trubric0.1380.1030.0560.1240.0670.047

The IRT + trubric method still wins at small mm, though it is actually outperformed at "large" mm. This is, again, likely due to geometry-based methods requiring a decent number of models to induce a good bias (see the left panel of Figure 4 to understand the effect of the number of models on trubric-based prediction).

We will adopt this more conservative approach to the leave-one-family-out evaluation throughout the updated manuscript.

PKPS delineation

The paper that proposes PKPS analyzes it in the context of both query efficient evaluation and (task, score) matrix completion. For query efficient evaluation, the results are in a different setting where observed cached responses are randomly sampled for each (model, task) pair. It does not include the traditional query/task efficient setting that we consider here (i.e., where the reference systems have fully observed tasks). It includes IRT as a baseline for query efficient evaluation. It does not explore IRT + PKPS as a viable method for embedding agentic traces.

We view the main contribution here as using embeddings of structured rubrics to improve the task efficiency. As mentioned above, we are the first to use both generic rubrics and task-specific rubrics for geometry construction. We hope this clears things up!


Thank you for taking the time to respond to our rebuttal! We think your lines of questions have improved the sharpness of our results.

Reviewer_YuQVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes trubric, a method for predicting the performance of an agentic system on a given benchmark using prior evaluations of related agentic systems on a subset of prior tasks. The core idea is to generate embeddings of agentic traces based on a structured summary of the agent’s results on given tasks. Then, use the geometric similarity between prior evaluations and the preliminary evaluations for a current system to predict how the system will perform on a whole benchmark. The authors evaluate trubric on two benchmarks and show it can predict benchmark performance with a small subset of task evaluations.

Critical strengths

The problem is interesting and relevant.

The idea to summarize traces according to a task-specific rubric is interesting and seems to work well.

The empirical results suggest that trubric is outperforming the other baselines compared in this work.

Critical weaknesses

The description of the trubric algorithm was not clear to me. Several components, like the use of IRT and the multidimensional scaling, were not described in sufficient detail. I wanted to see more precisely what is computed using what data. Sections 3.1 and 3.2 were not detailed enough to precisely convey the whole algorithm, and there is no pseudocode for the approach.

I am concerned that the premise of the paper is misleading or could lead to improper use of these tools. The goal of benchmarking is often to compare different methods (or in this case, agentic systems). However, trubric does not help a user determine whether one method is better than another with any confidence; it simply gives a point estimate of the benchmark score. I think this work could be significantly more powerful if combined with confidence estimates (e.g. using conformal prediction). In other words, help users know how many real evaluations they need to do before concluding that they can early-stop the benchmark eval.

The evaluation does not compare to any of the methods listed as prior work for task-efficient evaluation. Why is that? Given that the field has prior work, it seems natural to do this comparison.

The paper states, “For efficient benchmarking with traces, an ideal representation encodes (i) and (ii) but not (iii)-(v).” I don’t understand or agree with this statement. If this were true, you could just embed the task description and a binary correctness bit. So what are the exact requirements for a “good” representation? It would be nice to see a bit more thought put into this question.

In Figure 1, it’s not clear to me how you got these numbers, or what they mean. Why does trubric have the highest scores for the categories you claim are useless, (iii)-(v)? Is there a precise description of what you are plotting in Figure 1? (Apologies if I missed it).

Questions for authors

See above

4 comments
AuthorsOpenReview ↗

Initial response (1/2)

Thank you for taking the time to review our paper. We think your summary of our paper is fair. We appreciate you recognizing the relevance of the problem and that trubric works well relative to the provided baselines. We address your weaknesses / questions inline below:

Descriptions of method

trubric algorithm

trubric maps a trace to a vector in two stages, using only the task description and the trace text:

  1. Rubric generation (once per task). An LLM is prompted with the task description tjt_j and produces a task-specific rubric. The rubric is composed of KK fields, each a short description of an aspect of execution behavior to look for in the trace (e.g., how the relevant code was located, how the fix was verified). The rubric is shared by all traces on that task.
  2. Summarization and embedding (once per trace). For each trace xijx_{ij}, an LLM is prompted with the trace and the rubric and produces a short free-text summary zij(k)z_{ij}^{(k)} for each field. Each summary is embedded separately with a standard text encoder gg, and the KK embeddings are concatenated: ϕ(xij)=(g(zij(1)),…,g(zij(K)))∈RK⋅p\phi(x_{ij}) = (g(z_{ij}^{(1)}), \ldots, g(z_{ij}^{(K)})) \in \mathbb{R}^{K \cdot p}.

The representations are centered per (task, field) by subtracting the mean over systems, so that what remains is how a system's behavior on a task differs from other systems' behavior on the same task. This process does not use any correctness labels or scores.

Use of multi-dimensional scaling

trubric produces a representation of an individual trace. Following Helm et al. (2026), we use the representations of individual traces to get a representation of the agent itself. We use multi-dimensional scaling to get the representation of the agent that we can then use as feature vectors for standard regression.

Concretely: Eq. (1) defines a similarity between two systems as a kernel-weighted average (over pairs of tasks) of the inner products between their centered trace embeddings. From these similarities we form the (n+1)×(n+1)(n+1) \times (n+1) matrix of squared distances and apply classical MDS to obtain dd-dimensional coordinates ψ0,ψ1,…,ψn\psi_0, \psi_1, \ldots, \psi_n (with dd chosen by cross-validation). These representations are then used as feature vectors for standard regression trained on the reference models' full benchmark scores to produce y^geom\hat{y}_{geom}.

Use of IRT

The trubric-based representations of the agents (and y^geom\hat{y}_{geom}) do not use trace-level correctness, which is what standard efficient benchmarking methods use. To include this information in our final prediction, we also fit a model using this information. We chose IRT because it is the standard baseline here [Maia Polo et al., 2024; Kipnis et al., 2025]. Concretely, we fit a two-parameter logistic model to the reference systems' item-level scores and estimate a target's benchmark score from the posterior over its ability given its mm observed trace-level scores (we discuss this a little in Appendix B but point you to our references for more details).

Since the trubric-based prediction and the IRT prediction use different types of information, we consider the convex combination αy^IRT+(1−α)y^geom\alpha \hat{y}_{IRT} + (1 - \alpha) \hat{y}_{geom} as the final predictor.

If these descriptions clear things up for you, we will add a version of them to the main text. In any case, we will add pseudocode for the entire process to the appendix.

AuthorsOpenReview ↗

Initial response (2/2)

Conformal prediction / confidence estimates

We agree that estimates with calibrated uncertainty make the tool substantially more useful -- thank you for the suggestion.

We have added split-conformal prediction intervals [Angelopoulos and Bates, 2023] around the predicted scores. Under the leave-one-system-out protocol with adaptively selected tasks, the calibration scores for a given target are the reference systems' absolute prediction errors on the target's task set. We report the mean absolute error of the point estimates ±\pm the mean interval half-width, with the empirical coverage -- the fraction of target systems whose true full-benchmark score falls inside the interval -- in parentheses.

Conformal prediction intervals for IRT + trubric at α=0.9\alpha = 0.9, SWE-bench Verified:

nn (refs)m=1m = 1m=3m = 3m=5m = 5m=10m = 10m=20m = 20
106 (all)0.057 ±\pm 0.130 (0.897)0.046 ±\pm 0.109 (0.935)0.041 ±\pm 0.099 (0.935)0.037 ±\pm 0.095 (0.925)0.031 ±\pm 0.080 (0.925)
400.065 ±\pm 0.139 (0.906)0.052 ±\pm 0.113 (0.906)0.042 ±\pm 0.107 (0.916)0.041 ±\pm 0.101 (0.906)0.034 ±\pm 0.087 (0.906)
200.067 ±\pm 0.150 (0.888)0.052 ±\pm 0.123 (0.916)0.046 ±\pm 0.116 (0.944)0.045 ±\pm 0.106 (0.888)0.038 ±\pm 0.095 (0.944)
100.085 ±\pm 0.177 (0.878)0.070 ±\pm 0.143 (0.869)0.060 ±\pm 0.128 (0.869)0.056 ±\pm 0.123 (0.878)0.049 ±\pm 0.113 (0.897)

Terminal-Bench 2.0:

nn (refs)m=1m = 1m=3m = 3m=5m = 5m=10m = 10m=20m = 20
64 (all)0.092 ±\pm 0.205 (0.892)0.068 ±\pm 0.144 (0.923)0.054 ±\pm 0.131 (0.892)0.045 ±\pm 0.129 (0.954)0.035 ±\pm 0.111 (0.969)
400.098 ±\pm 0.207 (0.892)0.070 ±\pm 0.145 (0.877)0.053 ±\pm 0.133 (0.954)0.047 ±\pm 0.124 (0.939)0.034 ±\pm 0.112 (0.985)
200.108 ±\pm 0.221 (0.877)0.075 ±\pm 0.163 (0.877)0.059 ±\pm 0.147 (0.923)0.052 ±\pm 0.136 (0.939)0.044 ±\pm 0.120 (0.939)
100.120 ±\pm 0.265 (0.862)0.092 ±\pm 0.177 (0.862)0.079 ±\pm 0.155 (0.862)0.072 ±\pm 0.146 (0.923)0.060 ±\pm 0.142 (0.908)

In general, the intervals attain near-nominal empirical coverage across the entire grid -- down to m=1m = 1 task and n=10n = 10 reference systems -- while both the error of the point estimates and the widths of the intervals shrink as either budget grows. We see similar behavior at α=0.8\alpha = 0.8 and α=0.95\alpha = 0.95 (full-pool coverage 0.77--0.86 and 0.94--1.00 across the two benchmarks), with the caveat that α=0.95\alpha = 0.95 requires n≥19n \ge 19 since the split-conformal guarantee cannot exceed n/(n+1)n / (n+1).

On the specific use case of deciding whether one system is better than another, Figure 5 (right panels) reports head-to-head comparisons: with a single executed task per system, IRT ++ trubric identifies the better of two systems (true scores differing by ≥0.05\ge 0.05) in 88% of pairs on SWE-bench Verified. For a complementary analysis of the tails of the point-estimate error distribution (error percentiles and worst cases over task draws and systems), please see our response to Reviewer zsGC. We will add these results to the appendix and discuss them briefly in the main text.

Baselines and related work

The two sets of baselines that we discuss in the related work are the IRT family and the DKPS family. We already include the main IRT baseline (two-parameter logistic model [Maia Polo et al., 2024; Kipnis et al., 2025]) and adopt the related query-selection method when evaluating adaptive probes (Table 1 and Figure 4, center). We have also included a direct comparison to the main DKPS baseline in our response to Reviewer fjbc.

We discuss the relationship between our work and other (somewhat related) work in our response to Reviewer fjbc.

If we misunderstood your ask please let us know!

AuthorsOpenReview ↗

Initial response (3/2)

[apologies -- we miscounted the number of comments required to respond!]

The ideal trace representation and Figure 1

For efficient benchmarking we think a trace representation should encode behavior and task, and not necessarily the system (or components thereof) that generated the trace.

Representations that encode behavior and task without encoding authorship will have an advantage in practice when evaluating an entirely new component of a system (base model family, harness, etc.): the target's similarity to references is then driven by what it does on the tasks rather than by which family or harness produced it, so the geometry transfers to systems whose components are absent from the reference pool. We believe this is one of the reasons trubric (and the generic rubric) outperform the raw trace for efficient benchmarking under the leave-one-family-out and leave-one-harness-out protocols. We will add this comment to the discussion.

On what Figure 1 plots: for each of the five attributes we define a classification task on the trace embeddings. For example, for (i) task, we let the label of a trace be which of the 20 panel tasks it was executed on, and the classifier must predict the task from the trace embedding alone. We then train a linear classifier on (representation, label) pairs and evaluate held-out balanced accuracy. For (i) and (ii) a higher accuracy is "better". For (iii)--(v) a near-chance accuracy is "better". The spokes of the radar plot for (i) and (ii) are at the edge for higher accuracies. The spokes of the radar plot for (iii)--(v) are at the edge for closer-to-chance accuracies -- so trubric has the largest values on (iii)--(v) because the probes recover the least system, family, and harness signal from its embeddings. The precise definitions and the raw balanced accuracies underlying Figure 1 are in Appendix A.2 (Table 2).

If this description was helpful to you then we will update these results in the main text and appendix accordingly!


Thank you again for taking the time to review our paper. Please let us know if we were able to properly address your questions / concerns!

References

  • Angelopoulos, A. N., and Bates, S. Conformal Prediction: A Gentle Introduction. Foundations and Trends in Machine Learning, 16(4):494--591, 2023.
  • Helm, H., Johnson, B., and Priebe, C. Query-Efficient Model Evaluation Using Cached Responses. ICML, 2026.
  • Kipnis, A., Voudouris, K., Schulze Buschoff, L. M., and Schulz, E. metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models. ICLR, 2025.
  • Maia Polo, F., Weber, L., Choshen, L., Sun, Y., Xu, G., and Yurochkin, M. tinyBenchmarks: Evaluating LLMs with Fewer Examples. ICML, 2024.
AuthorsOpenReview ↗

Summary of initial response

Unfortunately we did not get the chance to engage with you during the reviewer-author discussion period. We briefly summarize our response to each of your points above:

Descriptions of the method.

We rewrote the descriptions of the trubric construction, the MDS-based geometry, and the IRT baseline (full text in our response) and will add pseudocode for the end-to-end estimator to the appendix.

Uncertainty quantification.

We ran split-conformal prediction intervals under a leave-one-system-out protocol across task budgets and reference-pool sizes. Empirical coverage is close to nominal throughout (0.86--0.99 at the 90% level across both benchmarks and all pool sizes), and the tables above report the point estimate ±\pm interval half-width alongside the achieved coverage.

Additional baselines.

We added IRT-blended raw-trace, generic-rubric, and DKPS-restricted baselines (full tables in our response to reviewer fjbc). trubric (PKPS) + IRT is the best or statistically tied in every (benchmark, budget, selection) setting.

The ideal trace representation and Figure 1.

The radar spokes are chance-normalized balanced accuracies of linear probes, with attributes (iii)--(v) plotting the complement so that outward is always better-for-evaluation. trubric scoring highest on those spokes means the probes are nearest to chance-level, i.e., the identity-related information has been removed. The raw accuracies are in Table 2 and we will clarify the caption.


We believe we have adequately addressed your initial weaknesses and questions & we hope these summaries make it easier for you to assess our original response.

Reviewer_zsGCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes trubric, a task-efficient method for estimating an AI agent’s full benchmark score from only a few executions. An LLM converts each execution trace into a task-specific structured summary, a text encoder embeds that summary, and the resulting geometry positions the target agent relative to previously evaluated reference agents. Combining this trace-based estimate with item response theory reduces single-task mean absolute error from 0.13 to 0.08 on SWE-bench Verified and from 0.14 to 0.12 on Terminal-Bench 2.0. The combined estimator generally outperforms sample averages, raw-trace embeddings, and IRT alone across task budgets. Under the paper’s stated cost model, which treats rubric generation, summarization, and embedding costs as negligible, this could substantially reduce evaluation cost. The evidence, however, is limited to two curated benchmarks, one rubric-generating and summarizing LLM, unoptimized rubric fields, and reference systems with nearly complete task coverage.

Critical strengths

Strong predictive accuracy under extremely small evaluation budgets. Under leave-one-family-out evaluation with only one randomly selected target task, IRT + trubric reduces MAE from 0.127 to 0.081 on SWE-bench Verified and from 0.137 to 0.122 on Terminal-Bench 2.0 relative to IRT alone; the paper reports statistical significance at $$p<10^{-4} and \p<10^{-3}$$, respectively. The combined estimator also remains best or near-best across larger task budgets, random and adaptive task selection, and system-, family-, and harness-exclusion protocols. This is consequential because the paper targets the practically important regime in which agent executions are expensive and only one or a few tasks can be afforded, and the results demonstrate meaningful average error reductions on both evaluated benchmarks.

Critical weaknesses

While the paper presents promising empirical results, I highlight the following concerns, which I hope will help clarify its contribution.

  1. The experiments do not fully isolate whether the trace geometry captures behavioral information beyond outcome-related cues. The canonical trubric fields include verification and final state, and, in the SWE-bench probe experiment, correctness is recoverable from trubric embeddings with 0.82–0.85 balanced accuracy across seven encoders. Process evaluators such as Agent-as-a-Judge (Zhuge et al., ICML 2025) and TRACE (Kim et al., ICML 2026) motivate a matched control that directly predicts correctness or execution quality from the same traces. An adapted score-only comparison based on the system–task outcome matrix, such as SparseEval (Zhang et al., ICLR 2026), would provide another important control. Without these comparisons or an ablation removing outcome-related fields, it remains unclear whether the gains arise from richer behavioral geometry or primarily from extracting success cues.

  2. Reliability in the emphasized one-task regime is not fully characterized. The paper reports mean error over 50 random task draws but omits upper-tail and worst-case errors, while its pairwise evaluation excludes systems whose true scores differ by less than 0.05. Although they study standard LM rather than agent benchmarks, Yauney et al. (ICLR 2026) show that very small benchmark subsets often fail to preserve rankings between similarly performing models, highlighting the same statistical risk. Reporting percentile and maximum errors, close-system comparisons, and robustness to held-out task clusters is therefore needed to assess reliability. Without such evidence, favorable average MAE does not establish dependable low-budget evaluation.

Zhuge, M., et al. “Agent-as-a-Judge: Evaluate Agents with Agents.” International Conference on Machine Learning (ICML), 2025.

Kim, W., et al. “Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents.” International Conference on Machine Learning (ICML), 2026.

Zhang, T., et al. “SparseEval: Efficient Evaluation of Large Language Models by Sparse Optimization.” International Conference on Learning Representations (ICLR), 2026.

Yauney, G., Warraich, S., and Swayamdipta, S. “How Reliable is Language Model Micro-Benchmarking?.” International Conference on Learning Representations (ICLR), 2026.

Questions for authors
  1. Can the authors clarify how much of trubric’s improvement comes from behavioral information beyond outcome-related cues? In particular, can they report an ablation removing the verification and final-state fields, or a matched baseline in which the same LLM directly predicts trace correctness or execution quality and those predictions are mapped to the full benchmark score using the same reference systems? Evidence that trubric retains its advantage under these controls would materially strengthen my assessment of the methodological contribution.

  2. Can the authors characterize the reliability of the m=1m=1 regime beyond average MAE? Specifically, can they report upper-tail and maximum errors across task draws, performance on system pairs separated by less than 0.05, and results when semantic task clusters are held out? My assessment would improve substantially if the method remains reliable in these difficult cases rather than performing well only on average.

5 comments
AuthorsOpenReview ↗

Initial Response (1/3)

Thank you for taking the time to review our paper. We think your summary of our work is fair. We also appreciate your noting that our empirical results are consequential and targeting an important regime. We address your weaknesses and questions inline below:

Isolating behavioral information and outcome-related cues.

We re-ran our analysis after removing the two outcome-related rubric sections (verification and final state) under the leave-one-family-out protocol:

MAEIRTtrubric+IRT (full rubric)trubric+IRT (behavior fields)trubric+IRT (outcome fields)
SWE-bench, m=1m=10.1270.0810.0870.084
SWE-bench, m=3m=30.1050.0680.0700.073
SWE-bench, m=5m=50.0880.0630.0640.067
Terminal-Bench, m=1m=10.1370.1220.1280.122
Terminal-Bench, m=3m=30.1120.1040.1080.104
Terminal-Bench, m=5m=50.0970.0900.0940.091

While there is a small decrease in performance on both benchmarks at $ m = 1 $ when the outcome-related rubric sections are removed, both ablated variants remain significantly better than IRT alone (Wilcoxon on paired per-system errors, $ p < 10^{-3} $ for both benchmarks). We do note that the behavior-only variant is better than the outcome-only variant at $ m \ge 3 $ on SWE-bench and comparable at $ m \ge 3 $ on Terminal-Bench. We plan to add this ablation to the appendix and comment on it in the main text.

Agent-as-a-Judge, TRACE, and SparseEval

Thank you for making us aware of these three papers. We have reviewed each. From our understanding, both Agent-as-a-Judge and TRACE were proposed in the context of different problem settings and are not easily adapted to ours without taking some creative liberties.

In particular, Agent-as-a-Judge evaluates whether a single observed run satisfies a human-authored requirement specification. Adapting it to our benchmarks would require authoring requirement graphs for every task (they introduce a new, smaller benchmark with human-authored requirement graphs for their analysis) and, more fundamentally, the outcomes of the $ m $ executed tasks are already provided in our setting. We view the output of an Agent-as-a-Judge type system as an alternative to the correctness label that we use to train the benchmark predictors and not necessarily an efficient-benchmarking method.

Similarly, TRACE scores a tool-use trajectory along efficiency, hallucination, and adaptivity dimensions. Its efficiency score takes ground-truth correctness as an input and its adaptivity score requires injecting tool failures into a live environment, which is not possible for the cached traces of the closed leaderboard systems motivating our analysis.

SparseEval, however, was proposed in the context of efficient benchmark evaluation. They study a much larger suite (~5,000 reference models on six classic LLM benchmarks) with query budgets of 20-100 and train a 4-layer MLP aggregator on the reference models to select a small subset of high information tasks. They note that the linear special case of their proposed objective recovers existing score-only subset selection methods (such as tinyBenchmarks [Maia Polo et al., 2024] and TailoredBench [Yuan et al., 2025]) that we discuss and use in the paper.

In a setting with ~100 reference systems rather than thousands, fitting the nonlinear aggregator is likely not feasible. We already include an analysis composing IRT-based anchor selection methods with trubric-based PKPS (Table 1 and Figure 3).

We plan to discuss SparseEval as part of an updated related works section.

AuthorsOpenReview ↗

Initial response (2/3)

Yauney et al. and rank preservation

For models that differ less than 0.05 in performance, practitioners will often consider additional information (cost, inference speed, etc.) when selecting a model to use. So we do not necessarily agree with Yauney et al.'s premise.

With that said, we report results in the context of their framework (minimum detectable ability difference (MDAD)). In particular, we computed, for each estimator and budget, the smallest true-score gap at which the estimated ordering of a pair of systems agrees with the full-benchmark ordering at least 80% of the time.

Under random task selection:

MDAD, random selectionm=1m=1m=3m=3m=5m=5m=10m=10m=20m=20
SWE-bench: sample score0.600.300.200.120.08
SWE-bench: IRT$ > 0.69 $*0.400.220.120.08
SWE-bench: trubric0.160.120.120.120.10
SWE-bench: IRT + trubric0.160.120.100.080.06
Terminal-Bench: sample score0.620.280.200.140.08
Terminal-Bench: IRT$ > 0.73 $*0.400.280.140.08
Terminal-Bench: trubric0.420.320.220.160.12
Terminal-Bench: IRT + trubric0.320.220.160.120.08

*does not reach 80% agreement at any gap observed in the panel (the largest true-score gap is 0.69 / 0.73).

And under the adaptive task-selection protocol described in the paper:

MDAD, adaptive selectionm=1m=1m=3m=3m=5m=5m=10m=10m=20m=20
SWE-bench: sample score0.180.120.100.120.12
SWE-bench: IRT0.240.140.080.060.06
SWE-bench: trubric0.120.100.080.080.10
SWE-bench: IRT + trubric0.100.080.060.060.06
Terminal-Bench: sample score0.260.260.140.100.10
Terminal-Bench: IRT0.420.200.100.080.04
Terminal-Bench: trubric0.220.120.120.120.12
Terminal-Bench: IRT + trubric0.160.120.100.080.04

The IRT + trubric ensemble is best or tied in every cell of both tables. With adaptive selection, a single task suffices for the ensemble to correctly order systems whose true scores differ by 0.10 at 80% agreement on SWE-bench -- a resolution IRT alone reaches only around $ m = 5 $.

These results partly confirm Yauney et al.'s observation that score-only micro-benchmarks extract little rank information beyond random sampling at small budgets: under random selection, IRT's MDAD is no better than the sample score's at any budget. At the same time, trubric meaningfully increases rank preservation at small $ m $. We plan to add both MDAD tables and the full agreement-versus-gap curves to the appendix.

AuthorsOpenReview ↗

Initial response (3/3)

Reliability of the one task regime.

This is a fair point. We computed the full error distribution (over the 50 task draws ×\times all systems) of using a random $ m = 1 $ task for IRT, trubric, and the IRT + trubric ensemble:

SWE-bench Verified, random $ m = 1 $:

MAEmedianp90p95p99maxfrac $ > 0.10 $
IRT0.1270.1180.2330.2930.3810.4950.58
trubric0.0810.0640.1750.2140.3380.5290.30
IRT + trubric0.0810.0630.1740.2100.3380.4710.30

Terminal-Bench 2.0, random $ m = 1 $:

MAEmedianp90p95p99maxfrac $ > 0.10 $
IRT0.1370.1190.2750.3430.4580.5330.58
trubric0.1270.1030.2740.3390.4480.6040.51
IRT + trubric0.1220.1030.2540.3120.4170.5760.51

While trubric-based prediction does not completely avoid bad tail behavior, we note the ensemble improves over IRT more in the tail than at the mean. We also computed the error distribution at $ m = 1 $ under the adaptive task-selection protocol described in the paper:

SWE-bench Verified, adaptive $ m = 1 $:

MAEmedianp90p95maxfrac $ > 0.10 $
IRT0.0800.0660.1710.2380.3920.25
trubric0.0680.0550.1510.1740.2950.22
IRT + trubric0.0620.0460.1350.1770.3350.20

Terminal-Bench 2.0, adaptive $ m = 1 $:

MAEmedianp90p95maxfrac $ > 0.10 $
IRT0.1120.0920.2570.3160.3660.46
trubric0.1100.0820.2260.3130.4130.43
IRT + trubric0.0990.0740.2160.2530.3680.37

The ensemble again has the lightest tail.

Along a similar vein of what you suggested, we also calculated the tail of the absolute error for predicting a given system's score under adaptive selection as the budget grows:

SWE-bench Verified, adaptive, IRT / IRT + trubric:

$ m $p90p95max
10.171 / 0.1350.238 / 0.1770.392 / 0.335
30.134 / 0.1100.185 / 0.1530.326 / 0.323
50.145 / 0.1090.156 / 0.1420.289 / 0.272
100.106 / 0.0910.134 / 0.1160.171 / 0.200
200.065 / 0.0690.086 / 0.1000.131 / 0.174

Terminal-Bench 2.0, adaptive, IRT / IRT + trubric:

$ m $p90p95max
10.257 / 0.2160.316 / 0.2530.366 / 0.368
30.157 / 0.1490.182 / 0.1870.210 / 0.206
50.132 / 0.1340.137 / 0.1430.196 / 0.209
100.102 / 0.0970.124 / 0.1290.159 / 0.149
200.069 / 0.0690.084 / 0.0840.127 / 0.114

The trace information is most valuable for tail control in the lowest-budget regime but improves over IRT alone everywhere.

We plan to add these tables and related discussion to the appendix.

Holding out tasks from semantic clusters.

This is a nice suggestion. We clustered the task-description embeddings of each benchmark with a Gaussian mixture model, selecting the number of clusters and the covariance structure automatically by BIC (Fraley & Raftery, 2002), fit on the leading 8 principal components of the embedded task descriptions. BIC selects $ k = 7 $ clusters on SWE-bench (the clusters align with repository structure (adjusted Rand index 0.45 against repository labels)) and $ k = 9 $ clusters on Terminal-Bench. We then evaluated the benchmark predictors using only semantically concentrated probes: all $ m $ tasks drawn from a single cluster (10 draws per cluster with at least $ m $ tasks), compared to the random draws used in the paper:

SWE-bench Verified, MAE (random / within-cluster):

$ m $IRTtrubricIRT + trubric
10.127 / 0.1260.081 / 0.0790.081 / 0.078
30.105 / 0.1020.071 / 0.0710.068 / 0.068
50.088 / 0.0890.069 / 0.0700.063 / 0.064
100.067 / 0.0650.065 / 0.0630.054 / 0.053

Terminal-Bench 2.0, MAE (random / within-cluster):

$ m $IRTtrubricIRT + trubric
10.137 / 0.1340.127 / 0.1260.122 / 0.120
30.112 / 0.1120.117 / 0.1110.104 / 0.099
50.097 / 0.0970.107 / 0.1060.090 / 0.090
100.073 / 0.0750.098 / 0.1000.070 / 0.072

Restricting all $ m $ tasks to a single semantic cluster leaves every estimator essentially unchanged. The spread across individual clusters is also relatively modest (the per-cluster ensemble MAE at $ m = 1 $ ranges over 0.074-0.082 on SWE-bench and 0.110-0.131 on Terminal-Bench. We plan to add this analysis to the appendix).


Thank you again for taking the time to review our work. Please let us know if we were able to address your concerns!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.