acceptodds
Under review as a conference paper at ICLR 2027

How Many Parameters Is a Trace Worth? Measuring Borrowed Capability in LLM Agents

Abstract

Language-model agents rarely operate in a vacuum, as they typically have access to previous requests and execution traces. These traces are especially valuable for systems that rely on small local models, where traces produced by a larger model can guide a smaller, faster one. In this paper, we study borrowed capability: the degree to which a weaker model, given a stronger model's verified execution trace on a related task, can perform as if it had the stronger model's capability. We derive 1,200 targets from 300 source tasks across six agentic benchmarks; each target changes the source's arguments, context, procedure, or goal, so that replaying the source trace fails. Across 20 open checkpoints from four families (270M-34B), each attempting every target with and without the trace, we find that a trace is worth roughly doubling the executor's size, provided that the executor is competent in the environment, i.e., solves at least one of its original tasks on its own. For competent executors, trace access raises exact success by 12 points, about as much as replacing the executor with a same-family model twice its size. Within an environment, larger executors close a larger share of the gap to the trace's author, although the smallest ones gain the most points. Executors below this competence threshold gain only 1.5 points, unless what they fail is the required answer format, which the trace demonstrates. Traces help least when the interface has changed, and there they can mislead. A trace, in short, can make a small agent act like a larger one, but it cannot teach an agent to act.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agents, adaptation, traces

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xcDzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper quantifies how traces from stronger models improve LLM agents across 20 models and six benchmarks. Its paired evaluation suggests gains comparable to doubling model size for competent executors under a fitted performance model.

Critical strengths
  1. Useful research question: Quantifies trace utility through performance gains, gap recovery, and parameter equivalence.
  2. Broad evaluation: Covers 20 models, six environments, and four task transformations using paired comparisons.
  3. Informative analysis: Examines scale, formatting, and interface effects, with transparent reporting and useful sensitivity checks.
Critical weaknesses
  1. Insufficient independent validation: Task generation, checker construction, and review rely on the same model, without systematic human auditing.
  2. Appendices have a strong AI-like, formulaic style, with repetitive positioning that reduces information density and comparison depth.
Questions for authors

See weaknesses.

1 comment

Dear Reviewer xcDz,

Thank you for your review! We added more experiments, including a human audit, and clarified several parts of the paper, especially the appendices. We hope these changes can address your concerns and that you will consider increasing your score accordingly. Going over each one:

Insufficient independent validation: Task generation, checker construction, and review rely on the same model, without systematic human auditing.

We performed a human audit on 120 targets (five targets for each of 24 environment-type groups) sampled from 108 sources. We evaluated whether the task is similar in difficulty to the source task, whether the task type label is correct, and whether the task is clear and plausible. Final human type labels agreed with the assigned labels in 115/120 cases (95.8%) and all tasks were marked clear and plausible, with 116 tasks (96.7%) being labelled of similar difficulty and four Slightly Harder tasks. We summarize all five disagreements in Appendix Q.1. If we exclude all five targets, the improvement changes by only 0.06 points (0.03 if we further exclude the four Slightly Harder tasks), and the improvement multiplier remains the same to two decimal places. We believe this audit gives more evidence that label or difficulty discrepancies have limited influence on the aggregated results. We have also included a summary in Section 3.

We also generate traces using Claude Opus 5.5 for 107 sources underlying 118 targets and evaluate them using the same checkers at eight executor checkpoints. Using these Claude Opus 5.5 traces improves executor success rate by 8.58 points [4.17, 12.82] over not providing any traces, with a difference of -1.59 [-3.73, 0.32] points compared with using Sol traces. For the 752 executor-task pairs where the executor solved at least one source task unaided in that environment, using Claude Opus 5.5 traces improves performance over no trace by 4.65 points [-0.55, 9.72]. Details are in Appendix Q.

Appendices have a strong AI-like, formulaic style, with repetitive positioning that reduces information density and comparison depth.

We rephrased some parts in the appendices that were too information-dense (e.g., source task selection, how generated tasks were authored and reviewed, how we provide traces to the executor model, handling executor failures, estimating uncertainty intervals) to hopefully make them less formulaic. We used shorter sentences, were more explicit about some terms (e.g., what we mean by reference solutions and context checks), and separated the description of an experiment from the interpretation of its results as much as possible. We hope these clarifications can better highlight our contributions (some clarifications are in Section 5).

We also added the trace-control table and summaries of the presentation, decoding, and second-author studies to the main paper, reporting sample sizes and uncertainty.

Please let us know if you have other questions and we will do our best to clarify before the discussion period ends. Thanks again for your review!

If you have any further questions, we’ll try to answer them before the end of the discussion period!

Best,

The Authors

Reviewer_hvoqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how much capability smaller language-model agents can gain from verified execution traces generated by stronger models. Using 1,200 adapted tasks across six benchmarks and 20 open-source checkpoints, the authors evaluate trace-assisted execution under four types of task changes. Results show that trace access improves success by 11.9 percentage points for competent executors, corresponding to approximately twice the model size under a fitted performance-equivalence measure. The study also identifies competence-dependent benefits and negative transfer when interfaces change.

Critical strengths
  1. The paper presents a well-designed experimental framework with paired comparisons across 20 checkpoints and six environments, providing substantial empirical evidence for measuring trace transfer.
  2. The findings are informative and practically relevant. The proposed parameter-equivalence measure quantifies trace benefits, while the analyses of competence thresholds and harmful transfer provide useful insights into when smaller agents can effectively reuse experience.
Critical weaknesses
  1. Limited main-text experimental presentation. The main paper contains only one results table, while important per-model, per-environment, and sensitivity analyses are deferred to the appendix (Tables 8–11). The standalone Research Question and Key Observation boxes also interrupt the presentation flow.
  2. Sensitivity to experimental choices. The headline results depend partly on a revised APIBank presentation adopted after observing earlier results. Additionally, the same model generated and reviewed the adapted tasks without systematic human validation, raising concerns about evaluation independence.
Questions for authors
  1. How robust are the main conclusions under the original trace presentation and alternative parameter-equivalence assumptions?
  2. Could the authors provide independent validation of the generated tasks and evaluation checkers?
1 comment

Dear Reviewer hvoq,

Thanks for your review and suggestions! We have included experiments to address your concerns. Going one by one:

Limited main-text experimental presentation.

We have included more results in the main text. Specifically, in Table 2, we have included results with unrelated and related trace controls, including separate results for APIBank, BFCL, and Mind2Web. In Section 4.3, we describe our decoding strategy, our second-author check and our presentation strategy results. We have also summarized our audit results in Section 3. We have also clarified some of the more dense parts of the methods sections (although most of the details are in the appendix to avoid overwhelming the reader). Detailed sensitivity analyses and model lists are still in the appendices.

The headline results depend partly on a revised APIBank presentation adopted after observing earlier results.

We present APIBank results using a particular presentation strategy (i.e., we place the current, modified interface before the original task description and before the source trace). In the main results (Section 4.3), we use a fixed presentation strategy across the six environments, and present results using a different presentation strategy in APIBank and BFCL separately. Using our main presentation strategy, we find that performance on APIBank context changes degrades substantially with traces, where success drops from 59.1% to 6.4% averaged across 20 checkpoints. In 604 of the 936 failures with traces (64.5%), models reference an outdated field instead of the updated one (see Appendix P).

For 50 context-change targets from each of BFCL and APIBank (100 total), we run three different presentation strategies (no history, history before task, history after task) across 19 model checkpoints (5,700 total runs). Here, we label the current task identically in all three settings and label the history in both traced settings (i.e. the task is held constant across settings, and only the presence and order of history changes). We find that including history before the task increases performance by 5.74 points [3.05, 8.63] compared to excluding it (71.58% versus 65.84% success), while including history after the task changes performance by −7.47 points [−10.58, −4.32] compared to excluding it (58.37% success). We also keep other aspects of the serving setup constant. Details are in Appendix O.

How robust are the main conclusions under the original trace presentation and alternative parameter-equivalence assumptions?

We include more robustness checks around our parallel curves fitting strategy for estimating the size multiplier:

  • If we remove one model checkpoint at a time, our estimated multiplier ranges from 1.48 to 1.58
  • If we fit a model that allows the effect of traces to scale as a function of log model size, we estimate a multiplier of 1.70 for 1.65B models and 1.38 for 12.5B models.
  • We also fit nondecreasing unaided curves within each family, using common tasks across sizes and no extrapolation.

We also include results that compare using related versus unrelated traces.

Could the authors provide independent validation of the generated tasks and evaluation checkers?

We conduct an audit where one human reviewer labels 120 generated tasks (five for each environment modification type combination, from a total of 108 source tasks; we did not condition our sampling on model performance). We find that our automatically generated labels match human labels 115/120 (95.8%) of the time (this includes all 30 context changes). The reviewer judged all tasks plausible and clear, and rated 116 (96.7%) as Similar in difficulty and four as Slightly Harder than the original task. Removing the five disputed targets changes the gain by 0.06 points; removing those five plus the four Slightly Harder targets changes it by 0.03 points in total. Our estimate of the multiplier effect of using traces does not change (to two decimal places) in either case. We provide more details about this audit in Appendix Q.1.

We also checked that all 600 reference solutions for the static tasks (APIBank, BFCL, and Mind2Web) pass their exact checkers, and that all 600 blank-response controls score zero (Appendices C–D).

We hope that our new results address your concerns, and if they do, you will see fit to raise your score. We once again thank you for your review and look forward to the rest of this discussion period! (which to be honest is quite short, since we spent most of it collecting new results)

Best,

The Authors

Reviewer_o31GOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper measures "borrowed capability": how much a small open model improves when it is given a stronger model's verified execution trace of a related task.

Tasks. There are 300 source tasks from six benchmarks: AppWorld, AgentBench OS, InterCode SQL, BFCL, APIBank, and Mind2Web. GPT-5.6 Sol solves each source once. For every source, coding agents running the same model author four targets that change the arguments (D1), the interface or context (D2), the procedure (D3), or the goal (D4). Each target is built so that replaying the source trace fails.

Executors and conditions. Twenty frozen checkpoints from four families (Qwen3, Qwen3.5, Gemma 3, Falcon-H1; 270M to 34B) attempt every target once at temperature 0, with and without the trace.

Findings.

  1. For executors that solve at least one source unaided ("competent"), the trace raises success from 48.6% to 60.5%. This is worth about doubling the model size within a family (pooled 2.05×).
  2. Below this threshold the gain is 1.5 points, except where the only failure is the answer format.
  3. Within an environment, larger executors close a larger share of the gap to the author.
Critical strengths

S1. A clean, large, and transparent paired design. In this study, retrieval and representation are held fixed. Every executor attempts every target with and without the trace, and replaying the trace is checked to fail. The study covers 48,000 cells across six environments and 20 checkpoints, and includes complete coverage, frozen artifacts, bootstrap by source, sensitivity tables, and released code. This makes the measurements trustworthy as reported and reusable.

S2. Actionable findings beyond the average. First, a trace helps only executors that can already operate the environment. Second, pooling across environments hides the within-environment scale trend. Third, an obsolete interface in the trace can turn 52% success into 6%. These results are directly relevant to routing in local/remote systems and to the evaluation of memory-based agents.

Critical weaknesses
  1. A key control is missing. TRACE adds a full worked episode in the same environment, while NOTRACE adds nothing. The gain therefore mixes related-task information with generic in-environment demonstration: tool conventions, interaction protocol, and output format. Excluding malformed answers addresses only format. A control that gives an unrelated verified trace from the same environment, is needed before the gain can be attributed to borrowing from a related task.

  2. The multiplier is not a stable unit.

    1. It comes from a parallel log-linear fit that the authors say does not always hold, with many implied sizes outside the observed range.
    2. Its value varies with the analysis: 1.5× to 1.8× for three families, 5.4× for Falcon-H1 (14.3× under the checkpoint-mean fit), and 2.6× pooled without environment intercepts.

    It is therefore unclear whether "a trace is worth doubling the executor" generalizes beyond this setup.

  3. No human validation of the targets. All 1,200 targets, their type labels, the difficulty matching, and their checkers were authored and reviewed by agents running GPT-5.6 Sol, the trace author. The authors "did not review items systematically by hand" and acknowledge that the phrasing may align with the traces. The per-type conclusions (for example, that context changes mislead) depend on these labels, but there is no human audit of type correctness, difficulty parity, or naturalness.

Questions for authors
  1. What is the gain when the executor receives a verified trace of an unrelated source from the same environment? How much of the 11.9 points remains attributable to relatedness?
  2. Did humans audit random samples of targets for type correctness, difficulty parity, and naturalness? What was the agreement?
  3. How stable are the results with sampled decoding and several attempts per cell, especially for Qwen in thinking mode?
  4. How do the gain and multiplier change with a second trace author?
1 comment

Dear Reviewer o31G,

Thank you so much for the helpful suggestions! We’ve run many new analyses and a human audit to address your concerns. Below are our responses to your specific comments.

What is the gain when the executor receives a verified trace of an unrelated source from the same environment?

On APIBank, BFCL, and Mind2Web across 17 checkpoints (200 targets in each env; 30,600 total runs), using an unrelated verified trace from the same env gives a 60.91% success rate, compared to 53.20% with no trace at all and 62.33% with the related source trace. The difference between related and unrelated traces is +1.42 points (95% CI from -0.39 to 3.27; +5.68 on Mind2Web, +5.35 on BFCL, -6.76 on APIBank where reusing the related source’s obsolete interface can be misleading).

Similarly, on AppWorld and AgentBench OS (5 checkpoints for a total of 6,000 runs), using an unrelated verified trace from the same env gives a 6.85% success rate, compared to 5.75% with no trace at all and 12.50% with the related source trace (+5.65 points [4.35, 7.00] related vs. unrelated). We’ve added these results to Table 2.

The multiplier is not a stable unit.

We agree that the multiplier shouldn’t be interpreted as some universal exchange rate. At the same time, the multiplier is fairly stable when recalculated excluding one checkpoint at a time (from 1.48 to 1.58; overall 1.53 [1.42, 1.66]). It does vary with executor model size when we allow the benefit of traces to vary with log model size (1.70 for executors around 1.65B params; 1.38 for executors around 12.5B params). And it’s 2.11 [1.93, 2.31] when we exclude APIBank.

We also fit nondecreasing unaided curves within each model family, using common targets across sizes and no extrapolation. For 13 of the 20 model checkpoints we test traces on, performance with traces falls within the fitted unaided score range; the remaining seven exceed that range, so we report bounds (see Appendix Q).

Did humans audit random samples of targets for type correctness, difficulty parity, and naturalness? What was the agreement?

We randomly sampled five source-target pairs per environment–task-type group without conditioning on executor model performance (108 sources, 120 targets) and had one reviewer label each example. Of the 120 targets, 115 (95.8%) received the same final label as we had assigned (including all 30 context changes); all targets were judged to be clear and plausible requests, and 116 (96.7%) were judged to be similarly difficult to the source task (four were judged slightly harder). Removing all five disputed targets, including two borderline cases where the assigned label was also judged plausible, changes the gain by 0.06 points. Removing the four Slightly Harder targets as well changes the gain by 0.03 points in total; the multiplier is the same to two decimal places in either case (Appendix Q.1).

How stable are the results with sampled decoding and several attempts per cell, especially for Qwen in thinking mode?

We re-ran evaluations with Qwen3.5-27B and Qwen3-4B with and without traces using three samples per prompt (temperature 0.6, top-p 0.95) on the same 120 targets as above (1,440 total runs). Traces increase single-sample success rate by 6.67 points (95% CI [1.67, 11.83]) and success on at least one of three samples by 3.33 points (95% CI from -1.28 to 8.12). Benefit is larger for Qwen3.5-27B (+11.94 points per sample; +9.17 given three samples) than Qwen3-4B (+1.39 points per sample; -2.50 given three samples, although CIs include zero in both cases). We don’t see clear evidence of a trace advantage in success on at least one of three samples across both models. This does not test whether an agent can identify which sample succeeds.

How do the gain and multiplier change with a second trace author?

We compared GPT-5.6 Sol traces, Claude Opus 5.5 traces, and no trace on 118 tasks (2,832 runs across eight executor checkpoints). Claude Opus 5.5 traces improve executor success rate by 8.58 points (95% CI [4.17, 12.82]) over no trace, with a difference of -1.59 points (95% CI from -3.73 to 0.32) relative to GPT-5.6 Sol traces overall. The gain is +4.65 points (95% CI from -0.55 to 9.72) over no trace on the 752 executor-target pairs where the executor model is competent in that environment. With two model sizes per family, we report size bounds rather than fit an author-specific multiplier, but e.g. aided Falcon-H1-7B using Opus traces outperforms unaided Falcon-H1-34B.

Thank you again for your review, and we hope that you’ll support our acceptance! We were busy collecting this data until the last day, but if you have any questions between now and the end of the discussion period, we’ll do our best to address them.

Best,

The Authors

Comments on the paper

1 comment

Dear reviewers,

Thanks for your feedback, which has helped us significantly improve the paper. Over the course of the rebuttal period, we collected as many results as possible (hence our last-minute response), which we hope address all of your concerns. We’ve added individual responses to each reviewer, but as a summary:

  • To address o31G’s question about what a related trace adds beyond a generic demonstration, we now compare the performance using a) no trace, b) an unrelated trace, and c) a related source task trace on AgentBench OS and AppWorld, as well as APIBank, BFCL, and Mind2Web. Related traces score higher than unrelated traces by 5.65 percentage points (95% CI 4.35 to 7.00) on AgentBench OS and AppWorld and by 1.42 points [−0.39, 3.27] on APIBank, BFCL, and Mind2Web. The difference on the three static environments is uncertain. We include these results in Table 2 and discuss the benefits of trace access and relatedness separately.
  • For all three reviewers’ concerns about target quality, we performed a human audit of 120 targets (5 per environment-type group) and found that the review agreed with the assigned type labels in 115/120 cases (95.8%); all targets were rated as plausible and clear, and 116/120 as Similar in difficulty. Removing the five targets for which our reviewer disagreed with the assigned labels changes the gain by 0.06 points and does not change the multiplier (to two decimal places). We include a discussion of these results in Appendix Q.1.
  • For hvoq’s concern about presentation, we’ve compared three prompts with the same labeled current task on 100 APIBank and BFCL targets at 19 checkpoints. We found that including the history at the beginning of the prompt outperforms no trace by 5.74 points, while including it at the end underperforms no trace by 7.47 points (100 targets from APIBank and BFCL, 19 checkpoints).
  • For o31G and hvoq, we’ve checked how the size comparison depends on the fitted curve and observed size range. We additionally tested sampled decoding at two checkpoints and traces from Claude Opus 5.5 at eight checkpoints. We’ve added details about our fitting procedure and the number of data points underlying each comparison in Appendices H and Q.
  • For hvoq and xcDz, we moved our trace-related ablations and a summary of our additional experiments to the main text, cut down on repeated context-setting (which was making the text quite hard to read), and rewrote some of the densest sections of the appendix for clarity. We also made our discussion of related work (e.g., trajectory transfer on AppWorld, PACE-Bench) more concrete.

Overall, we’re very thankful for the reviewers’ feedback, and we’re looking forward to further productive discussion!

Best,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.