- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a five-dimensional evaluation framework that separates Solution Framing (C1), Execution (C2), Feedback Control (C3), and experience-driven gains within and across tasks. Across seven frontier models and 36 primary long-horizon AI R&D tasks, the study shows that similar final scores can hide different process bottlenecks and that trajectory-derived experience can help or hurt depending on the model and task.
Critical strengths
- The framework addresses an important limitation of terminal-score evaluation by separating direction finding, reliable implementation, and retention/recovery of progress.
- The intra-task erasure and inter-task transfer protocols use matched conditions and isolated workspaces, making the experience intervention more interpretable than uncontrolled before/after comparisons.
- The study covers multiple models, task categories, repeated rollouts, trajectory diagnostics, and explicit metric definitions rather than reporting only aggregate scores.
Critical weaknesses
- The paper relies on the classification of C1 through C3, but it doesn't include the corresponding sensitivity analysis or human validation. We can't confirm whether this classification into C1 through C3 is robust.
- An intertask experiment simultaneously altered the source task selection and source task domain, and the quality chosen was High Performance Quality. This might cause the resulting transfer quality to stem from bias introduced by high-quality tasks.
- Lack of confidence intervals
I will improve my score if all my concerns are addressed.
Questions for authors
see above
4 comments
Rebuttal by Authors (Part 2 of 2)
W3: Confidence intervals and statistical significance.
To account for run-to-run variance in long-horizon research tasks, our evaluation already conducts three independent rollouts per model-task pair (756 baseline runs across 36 tasks) and reports mean performance, following standard practice in AutoLab (Xu et al., 2026) and MLE-bench (Chan et al., 2025). To further address this concern, we computed 95% confidence intervals across all seven models and six evaluation dimensions using a percentile bootstrap with 100,000 task-level resamples (weighting tasks equally and computing paired differences on shared tasks). The results confirm that incorporating confidence intervals largely preserves our rankings and main conclusions:
- Overall rankings: Claude-Opus-4.7’s first-place lead over all six other models is statistically significant (all six paired 95% CIs lie above zero after Bonferroni adjustment), and 17 of the 21 model pairs have paired 95% CIs excluding zero in the exact direction reported in Fig. 2.
- Process capabilities: Even for models with nearly identical final scores like GPT-5.5 (0.6633) and Gemini-3.1-Pro (0.6523), paired 95% CIs exclude zero in opposite directions, favoring GPT-5.5 on Execution (C2: ) and Gemini-3.1-Pro on Feedback Control (C3).
- Experience reuse: Overall, seven-model mean gains are significantly positive for both intra-task ( ) and inter-task reuse ( ). At the model level, paired CIs also confirm divergent responses to experience: for example, the lowest-baseline model DeepSeek-V4-Pro achieves a significant transfer gain ( ) that exceeds that of the top-baseline Claude-Opus-4.7 by (paired 95% CI ).
| Model | Final Score | C1 (Framing) | C2 (Execution) | C3 (Feedback) | Intra-task Gain | Inter-task Gain |
|---|---|---|---|---|---|---|
| Claude-Opus-4.7 | 0.7393 [0.6642, 0.8111] | 0.6115 [0.5467, 0.6762] | 0.9666 [0.9489, 0.9825] | 0.9201 [0.8780, 0.9561] | +0.0362 [−0.0049, 0.0852] | +0.0011 [−0.0398, 0.0444] |
| GLM-5.2 | 0.6815 [0.5995, 0.7626] | 0.5389 [0.4648, 0.6136] | 0.9365 [0.9020, 0.9643] | 0.9108 [0.8615, 0.9526] | +0.0793 [−0.0107, 0.1785] | +0.0400 [−0.0200, 0.0964] |
| GPT-5.5 | 0.6633 [0.5747, 0.7500] | 0.5553 [0.4777, 0.6339] | 0.9584 [0.9344, 0.9788] | 0.8577 [0.8035, 0.9035] | +0.0730 [0.0011, 0.1479] | +0.0634 [0.0011, 0.1260] |
| Gemini-3.1-Pro | 0.6523 [0.5879, 0.7187] | 0.5554 [0.4810, 0.6306] | 0.8891 [0.8391, 0.9345] | 0.9205 [0.8751, 0.9602] | +0.1278 [0.0559, 0.2145] | −0.0172 [−0.0572, 0.0264] |
| Kimi-K2.7-Code | 0.5865 [0.5125, 0.6629] | 0.4733 [0.3949, 0.5521] | 0.8804 [0.8372, 0.9210] | 0.8745 [0.7968, 0.9380] | −0.0127 [−0.0925, 0.0576] | +0.0209 [−0.0597, 0.0960] |
| LongCat-2.0 | 0.5724 [0.4878, 0.6596] | 0.4776 [0.3990, 0.5593] | 0.8880 [0.8418, 0.9294] | 0.9280 [0.8941, 0.9582] | +0.1454 [0.0702, 0.2342] | −0.0210 [−0.0603, 0.0129] |
| DeepSeek-V4-Pro | 0.5024 [0.4180, 0.5886] | 0.5188 [0.4478, 0.5917] | 0.8877 [0.8274, 0.9359] | 0.7716 [0.6894, 0.8482] | +0.0885 [−0.0126, 0.2005] | +0.0933 [0.0283, 0.1642] |
We will incorporate these confidence intervals and significance analyses into the revised manuscript.
We hope these additional analyses and clarifications have fully addressed all your concerns, and we would be grateful if you would consider updating your evaluation accordingly. Please let us know if any further questions remain.
Thank you for the author's detailed response. My concerns have been mostly addressed and I will raise my score to 3.
Dear Reviewer ivKS,
Thank you very much for your supportive response and for raising your score. We are glad to hear that our additional experiments and clarifications have addressed your concerns. We will incorporate all of these analyses and discussions into the final version of the manuscript. Thank you again for your time and constructive feedback throughout the review process.
Rebuttal by Authors (Part 1 of 2)
Dear Reviewer ivKS,
We sincerely appreciate your constructive review, and are grateful for your recognition of how our framework addresses key limitations of terminal-score evaluation, our strictly controlled experience-reuse protocols, and our broad empirical coverage.
W1: Sensitivity analysis and human validation of C1–C3.
To directly address your concerns regarding the robustness and validity of C1–C3, we conducted both a systematic sensitivity analysis and a blinded human validation:
Sensitivity analysis: On the same evaluated trajectories and under the same task-level aggregation, we varied one metric setting at a time across C1 (checkpoint horizon, stage weights, sampling granularity), C2 (repair penalty strength), and C3 (retention–recovery weight, regression threshold, sampling granularity, intermediate-attempt discount, and recovery criterion). Across all nine tested factors, model rankings remain highly consistent with the default configuration, yielding Spearman rank correlations (ρ) between 0.786 and 1.000:
Human validation: To examine whether C1–C3 reflect human judgment of the target research behaviors, we conducted a blinded evaluation on 56 randomly selected, anonymized same-task trajectory pairs (covering all 7 models and 25 tasks) with the automated scores withheld. For each pair, human evaluators independently judged whether trajectory A demonstrated stronger, weaker, or comparable behavior to trajectory B in C1, C2, and C3 (assigning a win, loss, or tie) across two separate sessions with reversed A/B presentation order. Comparing these human judgments with the preferences induced by our metric scores on all non-tied pairs (with 95% CIs estimated via 5,000 task-block bootstrap resamples) shows high directional agreement across all three dimensions:
Together, the high ranking stability across hyperparameter variations (ρ≥0.786) and strong directional agreement with human evaluators (87.8%–93.8%) confirm that our C1–C3 classifications are both credible and robust.
W2: Source task selection and domain control in inter-task experience transfer.
We understand your concern that the criteria for selecting source and target tasks may influence the observed transfer gains. However, we would like to clarify that our goal is not to compare transfer gains across domains or tasks, but to evaluate how effectively different models reuse their own experience under identical conditions. Considering that high-quality source tasks may yield more informative lessons for related tasks, we select one task with high-scoring solutions and a substantial record of experimentation as the source in each domain, with other tasks in the same domain serving as targets. Crucially, all seven models are evaluated on the exact same source–target pairs, with each model distilling lessons solely from its own source trajectory and measured against its own lesson-free baseline, ensuring a fair cross-model comparison on equal terms.