- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The contributions of this paper lie along two axes:
-
Trace Representation: The authors suggest the trubric methodology, which compacts long traces using specified rubrics whose fields are designed for task-wise verification. This methodology helps retain task and correctness signals while reducing system identity, model family, and harness signals.
-
Efficient Benchmarking: The authors combine IRT and the Product Kernel Perspective Space (PKPS) to estimate the full-benchmark accuracy of a new agent without running all benchmark tasks. It shows better performance than other previous setups, but I'd suggest including better baselines that were previously used.
Critical strengths
S1. The paper combines two research directions, trace representation and efficient benchmarking, and shows a performance improvement in agent evaluation in a task-efficient way. The authors demonstrate this empirically with two different benchmarks and various setups.
S2. The motivation of this paper is clear, and the methodology aligns soundly with it. Using the information in long trajectories for agent evaluation is important, and they suggest a new method, trubric, to summarize it. They used the embedded trubric results and IRT for efficient benchmarking.
Critical weaknesses
W1. The alternative methods considered in this paper seem weak, and other alternatives could be discussed. For example, Table 1 only includes raw-trace and generic results, not IRT+raw or IRT+generic. Also, this paper uses PKPS (a variation of DKPS) instead of DKPS, while Helm et al. (2026a) already showed that IRT+DKPS is powerful for efficient benchmarking (although they showed this only for responses). A comparison with IRT+DKPS is required to show the novelty of this paper.
W2. The model and harness tags of the systems are unclear. Some systems are unclassified, and some are misclassified. For example, the lingma-* systems are misclassified as gpt, and forgecode/forge-code and logos/logosagent are classified as different harnesses.
Questions for authors
Q1. Can you update Table 1 with IRT + raw or IRT + generic, optimizing by cross-validation? Since IRT + trubric may include an ensemble effect, IRT + raw or IRT + generic comparisons would be useful to better understand the performance of trubric.
Q2. Unlike DKPS (Helm et al., 2026a), PKPS uses the reference systems’ traces on all tasks via the task kernel in Eq. (1). Does this actually improve accuracy? Could the authors compare PKPS, at the same , against a DKPS baseline restricted to the target’s tasks ? If the gain is small, the geometry component reduces to DKPS on trubric embeddings.
Q3. According to the appendix, the model tags used in the experiments include many '–' entries. How were these systems handled in the leave-one-family-out experiment? Almost 50% of the systems in SWE-bench fall into this category. Also, the lingma-agent lingma-swe-gpt-72b / 7b systems are misclassified as gpt. Can you show the results either after reclassifying systems or after removing same-family systems in a conservative way?
Q4. L345-348 state that this method is applicable in settings where grading target tasks is infeasible, but the prompts in Appendix 3 (L810-817, L833-834) specify verification tests in the rubrics. The claim in the main text may be true when soft (or partial) evaluation or verification is possible. (I think the correctness results of trubric in Figure 1 also show that the rubric can partially include correctness verification.) Can you report how performs when the verification and final-state fields are removed from the rubric, i.e., when the summaries no longer include the agents' observed test outcomes?
Q5. Can you provide a brief summary of the PKPS paper (Anonymous, 2026), e.g., its abstract and main results, so that the contribution of the present submission can be assessed relative to that work? If there is a theoretical guarantee for PKPS (analogous to the task-efficiency result of Helm et al., 2026a for DKPS), please also summarize it.
Additional Comments
A1. Other methodologies that generate task-specific rubrics from task descriptions are not cited, e.g., TICK (Cook et al., arXiv 2410.03608, NeurIPS 2024 workshop) and BiGGen Bench (Kim et al., NAACL 2025).
4 comments
Initial response (2/2)
Description of how we handled untagged models in the leave-one-family-out experiment
A system with no model tag is treated as its own (singleton) family. So under leave-one-family-out, only the system itself is excluded from its reference pool. Most "--" entries are scaffold submissions that do not disclose the underlying model (e.g., tags of "Undisclosed" or "Multiple", which we also map to untagged). Where the underlying model is publicly documented we have now filled the tag in as part of the audit described above.
We acknowledge that the remaining undisclosed systems could in principle share a family with their references, which would make the protocol slightly optimistic for those targets. As mentioned above, the difference in the average MAE between the original and corrected tags is small on SWE-bench Verified (at most 0.003 at any budget) and does not change any ordering on Terminal-Bench 2.0, so we do not believe residual mistagging would change our qualitative conclusions.
Removing verification and end state sections from trubric
We ran this ablation in response to Reviewer zsGC (please see that response for the full table). Summarizing: with the verification and final-state fields removed (so the summaries no longer include the agents' observed test outcomes), trubric degrades modestly but our main conclusions hold.
Other methods for generating task-specific rubrics
Thank you for the pointers -- we will cite and discuss both TICK (Cook et al.) and BiGGen Bench (Kim et al., NAACL 2025) in the related work.
Thank you, again, for taking the time to review our paper. Please let us know if we were unable to address your concerns!
References
- Anonymous. Benchmark Prediction with Sparse and Unpaired Cached Responses. Under review, 2026. (The PKPS paper; cited in the submission as Anonymous, 2026.)
- Batzner, J., Nelaturu, S. H., Stachura, D., et al. Every Eval Ever: A Unifying Schema and Community Repository for AI Evaluation Results. arXiv:2606.14516, 2026.
- Cook, J., Rocktäschel, T., Foerster, J., Aumiller, D., and Wang, A. TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation. arXiv:2410.03608, 2024.
- Kim, S., et al. The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models. NAACL, 2025.
- Liang, P., Bommasani, R., Lee, T., et al. Holistic Evaluation of Language Models. Transactions on Machine Learning Research, 2023. (HELM; the HELM Lite scenarios.)
Thank you for the detailed response, the additional experiments, and the tag audit. The new results are informative: for example, PKPS clearly improves over DKPS at m = 1, and the ensemble reduces tail errors in the one-task regime. After reading the response, however, I still have some questions about the distinct contribution of this paper, which I summarize below.
1. Ensemble effect. The new baselines suggest that most of the improvement over IRT comes from ensembling. With IRT added, trubric differs from generic by at most 0.006 on SWE-bench Verified (e.g., generic + IRT 0.079/0.065/0.045 vs. trubric + IRT 0.081/0.063/0.043 at random m = 1/5/20, with generic + IRT better at m = 1), and from raw by at most 0.008 on Terminal-Bench 2.0. No significance tests are reported for these comparisons. Please report paired tests of trubric + IRT against raw + IRT and generic + IRT.
2. Tags. The tag issue remains. Untagged systems are treated as singleton families, but their underlying models are unknown. So whenever untagged systems are involved, either as the target or as references, the protocol cannot ensure that systems sharing the target's model family are excluded from the reference pool (and likewise for the agent/scaffold under leave-one-harness-out). In the original table, roughly half of the SWE-bench Verified systems are untagged, so similar systems may remain in many reference pools. The tag-correction results address only the re-tagged systems, not the remaining untagged ones. Unless untagged systems are excluded, the reported results cannot be regarded as a strict leave-one-family-out evaluation. Please rerun the main comparison with all untagged systems excluded from both target and reference pools, and confirm whether the new tables use the corrected tags.
3. Relation to the PKPS paper (Q5). Thank you for the summary. In my review, I listed the efficient-benchmarking component (combining IRT with PKPS-based prediction) as one of the two main contributions and a strength (S1). Whether this holds depends on its relation to the PKPS paper (Anonymous, 2026). Please state precisely which methodological components and experimental settings are shared with the PKPS paper, and which are new in this submission. If the efficient-benchmarking pipeline is largely shared, the distinct contribution would rest mainly on the trubric representation, which makes points 1 and 2 above more important.
Thank you for taking the time to respond / acknowledge our rebuttal!
Ensemble effect
Here are the p-values from Wilcoxon's paired signed-rank test for IRT + trubric versus IRT + raw and IRT + generic:
random:
| p-value | |||||
|---|---|---|---|---|---|
| SWE-bench: vs. IRT + raw | |||||
| SWE-bench: vs. IRT + generic | 0.13 | 0.98 | 0.52 | 0.085 | |
| Terminal-Bench: vs. IRT + raw | 0.49 | 0.095 | 0.014 | 0.010 | 0.013 |
| Terminal-Bench: vs. IRT + generic | 0.23 | 0.058 | 0.038 | 0.014 |
adaptive queries:
| p-value | |||||
|---|---|---|---|---|---|
| SWE-bench: vs. IRT + raw | 0.015 | 0.044 | 0.25 | 0.51 | |
| SWE-bench: vs. IRT + generic | 0.022 | 0.61 | 0.026 | 0.62 | 0.50 |
| Terminal-Bench: vs. IRT + raw | 0.17 | 0.032 | 0.25 | 0.31 | |
| Terminal-Bench: vs. IRT + generic | 0.031 | 0.26 | 0.019 | 0.42 | 0.19 |
IRT + trubric is significantly better (per a 0.05 threshold) than IRT + generic on 8 of 20 and IRT + raw on 13 of 20 of the (benchmark, number of queries, query type) tuples. It is never significantly worse than either.
The meta p-values for these comparisons (per Fisher's method) are and for trubric versus generic and trubric versus raw, respectively. Lastly, we note that we are the first (that we are aware of) to apply either trubric or generic rubrics to construct embeddings of the agents based on cached traces and to use them for efficient benchmarking.
Tags
We have removed all untagged models -- which the original protocol treated as singleton families -- from the leave-one-family-out analysis. We note that this approach is pretty conservative: it removes 42 of the 107 SWE-bench Verified systems (largely commercial scaffold submissions whose underlying model is undisclosed, e.g., solver, nfactorial, emergent, devlo, warp, cortexa) and 1 of the 65 Terminal-Bench 2.0 systems (JJAgent, which reports "Multiple"). The leave-one-family-out exclusions for the remaining systems are built from the corrected tags.
In any case, here are the results for IRT, trubric, and IRT + trubric (lowest number in a column is bolded if it is significantly better than the second best):
SWE-bench Verified:
| MAE | random | adaptive | ||||
|---|---|---|---|---|---|---|
| IRT | 0.145 | 0.096 | 0.053 | 0.114 | 0.062 | 0.039 |
| trubric | 0.111 | 0.097 | 0.093 | 0.096 | 0.091 | 0.098 |
| IRT + trubric | 0.111 | 0.086 | 0.056 | 0.096 | 0.065 | 0.051 |
Terminal-Bench 2.0:
| MAE | random | adaptive | ||||
|---|---|---|---|---|---|---|
| IRT | 0.149 | 0.106 | 0.056 | 0.127 | 0.072 | 0.039 |
| trubric | 0.144 | 0.124 | 0.114 | 0.135 | 0.109 | 0.107 |
| IRT + trubric | 0.138 | 0.103 | 0.056 | 0.124 | 0.067 | 0.047 |
The IRT + trubric method still wins at small , though it is actually outperformed at "large" . This is, again, likely due to geometry-based methods requiring a decent number of models to induce a good bias (see the left panel of Figure 4 to understand the effect of the number of models on trubric-based prediction).
We will adopt this more conservative approach to the leave-one-family-out evaluation throughout the updated manuscript.
PKPS delineation
The paper that proposes PKPS analyzes it in the context of both query efficient evaluation and (task, score) matrix completion. For query efficient evaluation, the results are in a different setting where observed cached responses are randomly sampled for each (model, task) pair. It does not include the traditional query/task efficient setting that we consider here (i.e., where the reference systems have fully observed tasks). It includes IRT as a baseline for query efficient evaluation. It does not explore IRT + PKPS as a viable method for embedding agentic traces.
We view the main contribution here as using embeddings of structured rubrics to improve the task efficiency. As mentioned above, we are the first to use both generic rubrics and task-specific rubrics for geometry construction. We hope this clears things up!
Thank you for taking the time to respond to our rebuttal! We think your lines of questions have improved the sharpness of our results.
Initial response (1/2)
Thank you for taking the time to review our paper. We think your summary of our work is fair. We appreciate you noting that we empirically demonstrate a performance improvement in a task-efficient way and that our motivation is clear. We attempt to address your weaknesses / questions inline below:
We have run the suggested baselines -- IRT + raw and IRT + generic rubric and also included a DKPS-based trubric with and without IRT -- through the exact pipeline of Table 1. The trubric (DKPS) rows use the same trubric embeddings as the trubric (PKPS) rows, so the only difference between them is the geometry construction (see "PKPS" below). MAE at m tasks:
SWE-bench Verified:
Terminal-Bench 2.0:
You are right that part of the trubric (PKPS) + IRT improvement is an ensemble effect. With that said, the trace representation (raw vs generic vs trubric) and the geometry construction (PKPS vs DKPS) still matter: trubric (PKPS) + IRT is the best predictor in 10 of the 12 (benchmark, selection, budget) settings and within 0.002 of the best in the remaining two. We will add these rows to Table 1 (or, if space does not permit, to an appendix table referenced from Table 1).
Thank you for catching these. We have since audited the original table and found three main kinds of issues: substring-matching errors (e.g., one of the lingma agents matched the "gpt" keyword), fine-tuned models improperly assigned (FrogBoss is fine-tuned from Qwen3 and so is now assigned to Qwen), and provider normalization (e.g., merging the "o1" and the "o3" families).
We re-ran our analysis after correcting these issues. The results on SWE-bench are essentially unchanged for m=1/5/20: IRT 0.127/0.088/0.048 → 0.126/0.086/0.047, IRT + trubric 0.081/0.063/0.043 → 0.079/0.063/0.042. On Terminal-Bench 2.0 the corrections were more consequential: IRT 0.137/0.097/0.052 → 0.145/0.102/0.053 and IRT + trubric 0.122/0.090/0.050 → 0.134/0.098/0.053, but the ordering of the methods is unchanged. The harness merges change MAE by at most 0.001 on either benchmark. We will update our analysis using the corrected tags and do not expect any of our conclusions to change.
The PKPS paper decomposes the definition of the (dis)similarity between models that DKPS uses into the product of a query kernel and a response kernel: the query kernel κ(t,t′) compares the tasks, and the response kernel ⟨ϕ(xat),ϕ(xbt′)⟩ compares the embedded traces produced on those tasks. From this perspective, DKPS uses the indicator query kernel κ(t,t′)=1{t=t′} -- so it can only include information from tasks that both systems have executed and cannot use information from cached reference traces for tasks the target agent has not executed. Changing the query kernel to a strictly positive definite kernel, such as the RBF kernel on task-description embeddings, allows us to use information from all existing reference traces, weighted by task similarity. Eq. (1) is this construction applied to the trubric embeddings.
The main theoretical result of the PKPS paper is that PKPS is query-efficient relative to DKPS when the queries / tasks that the reference systems have responded to are highly unpaired. The main empirical result is that PKPS dominates DKPS on HELM Lite [Liang et al., 2023] and Every Eval Ever [Batzner et al., 2026]. With the DKPS vs PKPS results above, the results in this paper extend those observations to the agent evaluation setting via trubric.