- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper quantifies how traces from stronger models improve LLM agents across 20 models and six benchmarks. Its paired evaluation suggests gains comparable to doubling model size for competent executors under a fitted performance model.
Critical strengths
- Useful research question: Quantifies trace utility through performance gains, gap recovery, and parameter equivalence.
- Broad evaluation: Covers 20 models, six environments, and four task transformations using paired comparisons.
- Informative analysis: Examines scale, formatting, and interface effects, with transparent reporting and useful sensitivity checks.
Critical weaknesses
- Insufficient independent validation: Task generation, checker construction, and review rely on the same model, without systematic human auditing.
- Appendices have a strong AI-like, formulaic style, with repetitive positioning that reduces information density and comparison depth.
Questions for authors
See weaknesses.
Dear Reviewer xcDz,
Thank you for your review! We added more experiments, including a human audit, and clarified several parts of the paper, especially the appendices. We hope these changes can address your concerns and that you will consider increasing your score accordingly. Going over each one:
We performed a human audit on 120 targets (five targets for each of 24 environment-type groups) sampled from 108 sources. We evaluated whether the task is similar in difficulty to the source task, whether the task type label is correct, and whether the task is clear and plausible. Final human type labels agreed with the assigned labels in 115/120 cases (95.8%) and all tasks were marked clear and plausible, with 116 tasks (96.7%) being labelled of similar difficulty and four Slightly Harder tasks. We summarize all five disagreements in Appendix Q.1. If we exclude all five targets, the improvement changes by only 0.06 points (0.03 if we further exclude the four Slightly Harder tasks), and the improvement multiplier remains the same to two decimal places. We believe this audit gives more evidence that label or difficulty discrepancies have limited influence on the aggregated results. We have also included a summary in Section 3.
We also generate traces using Claude Opus 5.5 for 107 sources underlying 118 targets and evaluate them using the same checkers at eight executor checkpoints. Using these Claude Opus 5.5 traces improves executor success rate by 8.58 points [4.17, 12.82] over not providing any traces, with a difference of -1.59 [-3.73, 0.32] points compared with using Sol traces. For the 752 executor-task pairs where the executor solved at least one source task unaided in that environment, using Claude Opus 5.5 traces improves performance over no trace by 4.65 points [-0.55, 9.72]. Details are in Appendix Q.
We rephrased some parts in the appendices that were too information-dense (e.g., source task selection, how generated tasks were authored and reviewed, how we provide traces to the executor model, handling executor failures, estimating uncertainty intervals) to hopefully make them less formulaic. We used shorter sentences, were more explicit about some terms (e.g., what we mean by reference solutions and context checks), and separated the description of an experiment from the interpretation of its results as much as possible. We hope these clarifications can better highlight our contributions (some clarifications are in Section 5).
We also added the trace-control table and summaries of the presentation, decoding, and second-author studies to the main paper, reporting sample sizes and uncertainty.
Please let us know if you have other questions and we will do our best to clarify before the discussion period ends. Thanks again for your review!
If you have any further questions, we’ll try to answer them before the end of the discussion period!
Best,
The Authors