acceptodds
Under review as a conference paper at ICLR 2027

Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development

Abstract

LLM agents are increasingly capable of improving models, systems, and other technical artifacts through autonomous, long-horizon experimentation. Yet existing benchmarks primarily evaluate these agents by final task scores, which capture outcome quality but provide limited insight into the research process. Evaluating runs in isolation also leaves unclear whether accumulated experience improves subsequent performance. To address these gaps, we introduce a unified evaluation framework that follows the research loop to assess how agents formulate directions (Solution Framing), implement solutions (Execution), and preserve or recover progress through feedback (Feedback Control). We further use controlled comparisons within and across tasks to examine whether experience from this loop improves subsequent research. Evaluating seven frontier models across a range of long-horizon AI R&D tasks, we find that similar final scores can conceal distinct process bottlenecks and divergent responses to experience. Some agents struggle to identify effective directions, while others implement strong solutions but fail to retain or restore them after subsequent experiments fail. Likewise, experience reuse can improve one agent's performance while degrading another's, despite comparable baseline performance without experience. These findings provide a more detailed account of autonomous research capabilities and limitations, informing model training, inference-time strategies, and experience management toward more reliable research and self-improvement.

Primary area:
datasets and benchmarks
Keywords:
Autonomous Research Agents, Process-Level Evaluation, Experience Reuse, Long-Horizon Reasoning, AI for AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ivKSOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a five-dimensional evaluation framework that separates Solution Framing (C1), Execution (C2), Feedback Control (C3), and experience-driven gains within and across tasks. Across seven frontier models and 36 primary long-horizon AI R&D tasks, the study shows that similar final scores can hide different process bottlenecks and that trajectory-derived experience can help or hurt depending on the model and task.

Critical strengths
  • The framework addresses an important limitation of terminal-score evaluation by separating direction finding, reliable implementation, and retention/recovery of progress.
  • The intra-task erasure and inter-task transfer protocols use matched conditions and isolated workspaces, making the experience intervention more interpretable than uncontrolled before/after comparisons.
  • The study covers multiple models, task categories, repeated rollouts, trajectory diagnostics, and explicit metric definitions rather than reporting only aggregate scores.
Critical weaknesses
  • The paper relies on the classification of C1 through C3, but it doesn't include the corresponding sensitivity analysis or human validation. We can't confirm whether this classification into C1 through C3 is robust.
  • An intertask experiment simultaneously altered the source task selection and source task domain, and the quality chosen was High Performance Quality. This might cause the resulting transfer quality to stem from bias introduced by high-quality tasks.
  • Lack of confidence intervals

I will improve my score if all my concerns are addressed.

Questions for authors

see above

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1 of 2)

Dear Reviewer ivKS,

We sincerely appreciate your constructive review, and are grateful for your recognition of how our framework addresses key limitations of terminal-score evaluation, our strictly controlled experience-reuse protocols, and our broad empirical coverage.

W1: Sensitivity analysis and human validation of C1–C3.

To directly address your concerns regarding the robustness and validity of C1–C3, we conducted both a systematic sensitivity analysis and a blinded human validation:

  • Sensitivity analysis: On the same evaluated trajectories and under the same task-level aggregation, we varied one metric setting at a time across C1 (checkpoint horizon, stage weights, sampling granularity), C2 (repair penalty strength), and C3 (retention–recovery weight, regression threshold, sampling granularity, intermediate-attempt discount, and recovery criterion). Across all nine tested factors, model rankings remain highly consistent with the default configuration, yielding Spearman rank correlations (ρ\rho) between 0.786 and 1.000:

    MetricSensitivity factorSettings testedSpearman ρ vs. default (min–max)
    C1Checkpoint horizonH = 10, 15, 20, 30, 400.857–1.000
    C1Stage weightsDefault; uniform over checkpoints; early-heavy; late-heavy0.893–1.000
    C1Checkpoint granularityRetain every 2nd or 3rd checkpoint; preserve endpoints; enumerate sampling offsets0.821–0.893
    C2Pre-commit repair penalty0.5×, 1× (default), and 1.5× the default penalty strength0.786–1.000
    C3Retention–recovery weightRetention weight α = 0.25, 0.50, 0.750.857–1.000
    C3Regression thresholdε = 0.005, 0.01, 0.02, 0.050.964–1.000
    C3Checkpoint granularityRetain every 2nd or 3rd checkpoint; preserve endpoints; enumerate sampling offsets0.929–1.000
    C3Intermediate-attempt discountMilder; default; stronger; no discount0.964–1.000
    C3Recovery criterionFull recovery only versus default partial-recovery scoring0.964
  • Human validation: To examine whether C1–C3 reflect human judgment of the target research behaviors, we conducted a blinded evaluation on 56 randomly selected, anonymized same-task trajectory pairs (covering all 7 models and 25 tasks) with the automated scores withheld. For each pair, human evaluators independently judged whether trajectory A demonstrated stronger, weaker, or comparable behavior to trajectory B in C1, C2, and C3 (assigning a win, loss, or tie) across two separate sessions with reversed A/B presentation order. Comparing these human judgments with the preferences induced by our metric scores on all non-tied pairs (with 95% CIs estimated via 5,000 task-block bootstrap resamples) shows high directional agreement across all three dimensions:

    MetricConcordant / non-tied judgmentsDirectional agreement95% CI
    C190/9693.8%83.7%–100.0%
    C236/4187.8%69.8%–100.0%
    C347/5388.7%75.6%–98.0%

Together, the high ranking stability across hyperparameter variations (ρ≥0.786\rho \ge 0.786) and strong directional agreement with human evaluators (87.8%–93.8%87.8\%\text{--}93.8\%) confirm that our C1–C3 classifications are both credible and robust.

W2: Source task selection and domain control in inter-task experience transfer.

We understand your concern that the criteria for selecting source and target tasks may influence the observed transfer gains. However, we would like to clarify that our goal is not to compare transfer gains across domains or tasks, but to evaluate how effectively different models reuse their own experience under identical conditions. Considering that high-quality source tasks may yield more informative lessons for related tasks, we select one task with high-scoring solutions and a substantial record of experimentation as the source in each domain, with other tasks in the same domain serving as targets. Crucially, all seven models are evaluated on the exact same source–target pairs, with each model distilling lessons solely from its own source trajectory and measured against its own lesson-free baseline, ensuring a fair cross-model comparison on equal terms.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2 of 2)

W3: Confidence intervals and statistical significance.

To account for run-to-run variance in long-horizon research tasks, our evaluation already conducts three independent rollouts per model-task pair (756 baseline runs across 36 tasks) and reports mean performance, following standard practice in AutoLab (Xu et al., 2026) and MLE-bench (Chan et al., 2025). To further address this concern, we computed 95% confidence intervals across all seven models and six evaluation dimensions using a percentile bootstrap with 100,000 task-level resamples (weighting tasks equally and computing paired differences on shared tasks). The results confirm that incorporating confidence intervals largely preserves our rankings and main conclusions:

  • Overall rankings: Claude-Opus-4.7’s first-place lead over all six other models is statistically significant (all six paired 95% CIs lie above zero after Bonferroni adjustment), and 17 of the 21 model pairs have paired 95% CIs excluding zero in the exact direction reported in Fig. 2.
  • Process capabilities: Even for models with nearly identical final scores like GPT-5.5 (0.6633) and Gemini-3.1-Pro (0.6523), paired 95% CIs exclude zero in opposite directions, favoring GPT-5.5 on Execution (C2: +0.0682+0.0682 [0.0146,0.1236][0.0146, 0.1236]) and Gemini-3.1-Pro on Feedback Control (C3).
  • Experience reuse: Overall, seven-model mean gains are significantly positive for both intra-task (+0.0768+0.0768 [0.0489,0.1059][0.0489, 0.1059]) and inter-task reuse (+0.0258+0.0258 [0.0129,0.0387][0.0129, 0.0387]). At the model level, paired CIs also confirm divergent responses to experience: for example, the lowest-baseline model DeepSeek-V4-Pro achieves a significant transfer gain (+0.0933+0.0933 [0.0283,0.1642][0.0283, 0.1642]) that exceeds that of the top-baseline Claude-Opus-4.7 by +0.092+0.092 (paired 95% CI [0.007,0.178][0.007, 0.178]).
ModelFinal ScoreC1 (Framing)C2 (Execution)C3 (Feedback)Intra-task GainInter-task Gain
Claude-Opus-4.70.7393 [0.6642, 0.8111]0.6115 [0.5467, 0.6762]0.9666 [0.9489, 0.9825]0.9201 [0.8780, 0.9561]+0.0362 [−0.0049, 0.0852]+0.0011 [−0.0398, 0.0444]
GLM-5.20.6815 [0.5995, 0.7626]0.5389 [0.4648, 0.6136]0.9365 [0.9020, 0.9643]0.9108 [0.8615, 0.9526]+0.0793 [−0.0107, 0.1785]+0.0400 [−0.0200, 0.0964]
GPT-5.50.6633 [0.5747, 0.7500]0.5553 [0.4777, 0.6339]0.9584 [0.9344, 0.9788]0.8577 [0.8035, 0.9035]+0.0730 [0.0011, 0.1479]+0.0634 [0.0011, 0.1260]
Gemini-3.1-Pro0.6523 [0.5879, 0.7187]0.5554 [0.4810, 0.6306]0.8891 [0.8391, 0.9345]0.9205 [0.8751, 0.9602]+0.1278 [0.0559, 0.2145]−0.0172 [−0.0572, 0.0264]
Kimi-K2.7-Code0.5865 [0.5125, 0.6629]0.4733 [0.3949, 0.5521]0.8804 [0.8372, 0.9210]0.8745 [0.7968, 0.9380]−0.0127 [−0.0925, 0.0576]+0.0209 [−0.0597, 0.0960]
LongCat-2.00.5724 [0.4878, 0.6596]0.4776 [0.3990, 0.5593]0.8880 [0.8418, 0.9294]0.9280 [0.8941, 0.9582]+0.1454 [0.0702, 0.2342]−0.0210 [−0.0603, 0.0129]
DeepSeek-V4-Pro0.5024 [0.4180, 0.5886]0.5188 [0.4478, 0.5917]0.8877 [0.8274, 0.9359]0.7716 [0.6894, 0.8482]+0.0885 [−0.0126, 0.2005]+0.0933 [0.0283, 0.1642]

We will incorporate these confidence intervals and significance analyses into the revised manuscript.


We hope these additional analyses and clarifications have fully addressed all your concerns, and we would be grateful if you would consider updating your evaluation accordingly. Please let us know if any further questions remain.

Thank you for the author's detailed response. My concerns have been mostly addressed and I will raise my score to 3.

Dear Reviewer ivKS,

Thank you very much for your supportive response and for raising your score. We are glad to hear that our additional experiments and clarifications have addressed your concerns. We will incorporate all of these analyses and discussions into the final version of the manuscript. Thank you again for your time and constructive feedback throughout the review process.

Reviewer_Y9fqOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper came up with an evaluation framework for long-horizon AI research agents. Beyond final scores, it also evaluates qualities of the process: based on verifier results and execution signals, it computes problem framing, execution and feedback control; and through counterfactual comparisons, it evaluates in-task and cross-task experience use. Under the same Claude Code harness, the authors used 36 AutoLab tasks to evaluate 7 frontier models, and made 3 observations. Firstly, models with similar final scores could have different process profiles. In addition, the preserved experience can be used for the next step, but the transfer effect among experience differs among tasks. Finally, the most common type in the best solutions is stacking known techniques, and these solutions exploit evaluation shortcuts more often than they introduce novel ideas.

Critical strengths
  1. Evaluation infrastructure itself is a solid contribution, It uses fixed harness and tests the influence of harness. The metrics are definite and definitions are specific. And it also reports the cost. This makes the framework a reasonable foundation for future research agent studies.
Critical weaknesses
  1. Most discoveries are expected or previously reported. The related work of the paper mentions that experience reuse is helpful but not stable, while research agents mostly recombine known techniques. The paper also fails to clarify what additional information its comprehensive metric provided compared with simple metrics suck as best score, execution success rate and peak retention. The actually informative and specific conclusions, such as reversal of model rankings and weaker models are more reliant on experience, provided no uncertainty estimations when reported. Also, the final conclusion didn’t consider weaker models have larger headrooms.

  2. Although agents are asked to commit after every iteration, but the workload of every iteration is determined by the agents themselves, so the commit numbers different models are asked to evaluate differs significantly. Therefore, even when using a uniform time span of 20 checkpoints, the checkpoints used for comparing Solution Framing scores cover different amounts of work; the Feedback Control score, on the other hand, depends on exactly how many instances of regression a single trajectory contains. While the paper acknowledges this limitation of the Feedback Control metric, the main text still interprets LongCat’s higher Feedback Control score as an advantage in the process; furthermore, no sensitivity analysis was conducted on the manually assigned weights underlying the metrics.

Questions for authors
  1. Compared to simple trajectory metrics such as best score, execution success rate, and peak retention, what additional diagnostic value do composite process metrics provide?

  2. Do the process metrics align with experts’ assessments of trace behavior? Do the key conclusions from model comparisons still hold after adjusting the checkpoint granularity or metric weights?

  3. Do the conclusions that were originally novel—such as the reversal in model rankings under experience reuse and the greater reliance on experience by weaker models—still hold after providing confidence intervals across tasks and rollouts, and after controlling for each model’s headroom at branch points?

The evaluation setup is useful, but the main findings are largely expected or already reported, and the current evidence does not support the more specific claims that would make the paper informative. Addressing this would require new findings rather than revisions, so I recommend rejection.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1 of 2)

Dear Reviewer Y9fq,

We sincerely appreciate your thorough review and constructive feedback. We are grateful that you acknowledge the strengths of our work, particularly that our evaluation infrastructure provides a solid contribution and a reasonable basis for studying research agents.

W1.1: New empirical insights and contributions beyond prior work.

As discussed in the Related Work section, we evaluate agents on long-horizon autonomous AI R&D tasks, where they tackle open-ended optimization problems with budgets of 2–12 hours per task. In contrast, prior studies of experience reuse and process evaluation have typically focused on shorter-horizon tasks. For example, EvoAgentBench gives agents 30 minutes per run to complete a clearly defined task. Whether earlier evaluation methods and findings generalize to this setting requires direct empirical study. Regarding your related point on solution novelty, our observation that agents often combine established techniques is consistent with contemporaneous work and plays a supporting role in our analysis of Solution Framing (C1). Our core contribution is a systematic evaluation of seven frontier models across five dimensions of long-horizon AI R&D. It characterizes differences in their research behavior, identifies model-specific bottlenecks, and guides targeted improvements, providing a joint assessment that prior studies have not offered.

W1.2 & Q1: Diagnostic value of the composite process metrics (C1–C3) beyond simple trajectory statistics.

The composite metrics capture process information omitted by the corresponding simple statistics: C1 jointly measures how high a score is reached and how early it is reached; C2 accounts for both successful delivery and code-related build failures incurred before delivery; and C3 combines peak retention with recovery efficiency following regressions. Consequently, identical best scores, execution success rates, or peak retention values can correspond to different research processes.

Using the full dataset of 756 runs, we computed run-level Spearman correlations between each composite metric and its corresponding simple statistic, retaining all ties and excluding only missing values. The results are shown below.

ComparisonSpearman ρ
Best score vs. C10.928
Execution success rate vs. C20.594
Peak retention vs. C30.692

C1's high correlation is consistent with its incorporation of solution quality, while meaningful distinctions remain. For example, on concurrent_kv_wal, two runs both started at zero and achieved a best score of 1.0, but first reached it at checkpoints 5 and 11, respectively, yielding C1 scores of 0.9112 and 0.5960 (checkpoint counts include the initial state). This illustrates how C1 distinguishes score-attainment trajectories that best score alone cannot differentiate. C2 and C3 additionally characterize implementation rework and recovery behavior, respectively.

W1.3 & Q3: Statistical uncertainty and headroom-controlled analysis of the experience-reuse findings.

To address your concern about the robustness of our experience-reuse findings (model orderings reversing under experience reuse and the weaker model still relying more on experience), we computed confidence intervals over tasks and rollouts and controlled for each model’s headroom at the branch point, finding that both conclusions still hold.

(1) Under comparable headroom and confidence intervals, the weaker model still relies more on experience. We selected cases where LongCat and Claude had very similar performance at the branch point (difference ≤0.05). Their mean scores were 0.4271 and 0.4196, respectively, indicating comparable headroom. On this subset, LongCat-2.0 still achieves a significant experience gain (+0.1593+0.1593, 95% CI [0.0239,0.3543][0.0239, 0.3543]) whereas Claude-Opus-4.7 does not (−0.0134-0.0134, 95% CI [−0.0708,0.0412][-0.0708, 0.0412]), yielding a statistically significant paired difference of +0.1728+0.1728 (95% CI [0.0442,0.3331][0.0442, 0.3331]). Moreover, expanding intra-task branching across all three seeds (611 paired continuations) remains significantly positive overall (+0.0408+0.0408, 95% CI [0.0141,0.0675][0.0141, 0.0675]) and preserves the exact top-3 beneficiary ordering in Fig. 5 (LongCat-2.0, Gemini-3.1-Pro, DeepSeek-V4-Pro).

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2 of 2)

(2) Reversed model orderings under cross-task lesson transfer still hold after accounting for uncertainty across tasks and rollouts. We estimated 95% confidence intervals from 100,000 joint bootstrap resamples. Under joint task-and-rollout resampling on the 19 target tasks, the lowest-baseline model DeepSeek-V4-Pro achieves a significant positive transfer gain (+0.0933+0.0933, 95% CI [0.011,0.187][0.011, 0.187], reducing zero-score runs from 13/57 to 0/57), which significantly exceeds LongCat-2.0's change (−0.0210-0.0210; paired gain difference +0.1143+0.1143, 95% CI [0.0174,0.2194][0.0174, 0.2194]) and reverses their average ordering (from 0.35 vs. 0.43 without lessons to 0.44 vs. 0.41 with lessons). Likewise, GLM-5.2 overtakes Gemini-3.1-Pro on the 19 targets (from 0.51 vs. 0.53 to 0.55 vs. 0.52) in 18 of 19 leave-one-target-out checks, backed by a statistically significant best@3 gain (+0.0674+0.0674, 95% CI [0.0011,0.1519][0.0011, 0.1519]) that lifts GLM-5.2 from 4th to 2nd place (Fig. 17).

W2 & Q2: Human validation, checkpoint granularity, and hyperparameter sensitivity of C1–C3.

To address your concern directly, we conducted sensitivity analyses and human validation, and both support the reliability of our metrics. We will discuss these findings in the revised manuscript.

(1) Sensitivity analyses show that model rankings remain consistent across changes in checkpoint granularity and metric weights. Holding the evaluated trajectories and task-balanced aggregation fixed, we varied one setting at a time across C1 (checkpoint horizon, stage weights, sampling granularity), C2 (repair penalty), and C3 (retention–recovery weight, regression threshold, sampling granularity, intermediate-attempt discount, and recovery criterion). Across all nine tested factors, including subsampling every 2nd or 3rd checkpoint to vary commit granularity, model rankings remain highly consistent with the default configuration (Spearman ρ∈[0.786,1.000]\rho \in [0.786, 1.000]):

MetricSensitivity factorSettings testedSpearman ρ vs. default (min–max)
C1Checkpoint horizonH = 10, 15, 20, 30, 400.857–1.000
C1Stage weightsDefault; uniform over checkpoints; early-heavy; late-heavy0.893–1.000
C1Checkpoint granularityRetain every 2nd or 3rd checkpoint; preserve endpoints; enumerate sampling offsets0.821–0.893
C2Pre-commit repair penalty0.5×, 1× (default), and 1.5× the default penalty strength0.786–1.000
C3Retention–recovery weightRetention weight α = 0.25, 0.50, 0.750.857–1.000
C3Regression thresholdε = 0.005, 0.01, 0.02, 0.050.964–1.000
C3Checkpoint granularityRetain every 2nd or 3rd checkpoint; preserve endpoints; enumerate sampling offsets0.929–1.000
C3Intermediate-attempt discountMilder; default; stronger; no discount0.964–1.000
C3Recovery criterionFull recovery only versus default partial-recovery scoring0.964

(2) We added an expert evaluation and found high agreement between expert judgments and our process metrics (87.8%–93.8%). Specifically, we conducted a blinded evaluation on 56 randomly selected, anonymized same-task trajectory pairs (covering all 7 models and 25 tasks) with automated scores withheld. Human evaluators independently made win/loss/tie judgments for each pair across two separate sessions with reversed A/B order (112 total judgments per dimension). Computing directional agreement on non-tied judgments (with 95% CIs from 5,000 task-block bootstrap resamples preserving within-pair dependence) confirms strong alignment between C1–C3 and human assessments:

MetricConcordant / non-tied judgmentsDirectional agreement95% CI
C190/9693.8%83.7%–100.0%
C236/4187.8%69.8%–100.0%
C347/5388.7%75.6%–98.0%

(3) Interpreting LongCat-2.0’s C3 score. As shown in Appendix E.2 (Fig. 14), LongCat-2.0 ranks first on aggregate C3 (0.9280.928), third on peak retention (0.9620.962), and sixth on conditional recovery (0.5200.520). We will report these results separately in §4.1 and describe LongCat-2.0’s observed strength specifically as retaining peak performance over the evaluated commits.


We hope the additional analyses and clarifications address the concerns you raised. We would appreciate it if you could revisit your assessment in light of our response, and we would be happy to answer any follow-up questions.

AuthorsOpenReview ↗

Follow-up as the Discussion Period Approaches Its End

Dear Reviewer Y9fq,

We hope you are doing well. As the rebuttal and discussion period will conclude tomorrow, we wanted to follow up on our response and additional results.

We would greatly appreciate it if you could let us know whether our response has addressed your concerns or whether there are any remaining questions that we may clarify before the discussion period ends.

Thank you again for your time and thoughtful consideration.

Best regards,

The Authors

Reviewer_9BpROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Summary: The authors propose a systematic evaluation framework for long-horizon AI research agents and argue that final task scores are not enough to evaluate the research capability of these agents. The proposed framework includes three process-level dimensions: Solution Framing (C1) – the agent's ability to identify promising solutions; Execution (C2) – the agent's ability to create executable and correct changes; and Feedback Control (C3) – the agent's ability to maintain improvements and recover from regressions. The authors also evaluate experience-based self-improvement through intra-task experience retention and inter-task knowledge transfer. The authors evaluated seven frontier language models on 36 AutoLab tasks using 756 independent trajectories, as well as conducting additional experiments on other benchmarks, harness configurations, and solution novelty. The results demonstrate that models with similar final scores may have very different process bottlenecks and that experience accumulated over time can either improve or degrade subsequent performance. The authors emphasize the importance of process-aware evaluations and experience management for developing more reliable autonomous research agents.

Contributions: 1.The authors developed three process-level metrics (C1, C2, and C3) to evaluate solution framing, execution reliability, and feedback control based on observable trajectory data. These metrics supplement traditional final score evaluations and provide more in-depth analysis of agent behavior. 2. The authors conducted controlled intra-task and inter-task experiments to determine how accumulated experience affects future performance. In the intra-task experiment, they compared agents with and without previous trajectory history. In the inter-task experiment, they evaluated the transfer of lessons learned from one task to another related task. 3. The authors evaluated seven frontier models on 36 long-horizon AI research tasks and identified model-specific and task-dependent differences that were not apparent when looking at aggregate final scores. 4. The authors also analyzed solution novelty, trajectory dynamics, harness sensitivity, and different forms of experience representation. These analyses provided additional insights into the limitations of existing research agents and possible areas for improvement.

Validation: The authors have provided extensive empirical evidence to support their claims by conducting 36 AutoLab tasks across model development, system optimization, puzzle/challenge tasks, and CUDA optimization. They tested seven of the most advanced models available today. Each model was run three times for each task, producing a total of 756 trajectories. Based upon the results of these tests, the authors demonstrated that even though two agents may produce similar final scores, they can be very different in terms of how reliable they are at executing commands and how well they respond to feedback. For example, both GPT-5.5 and Gemini-3.1-Pro achieved similar final performance and C1 scores; however, GPT-5.5 performed better on C2 and Gemini-3.1-Pro performed better on C3.

The authors also conducted experience reuse experiments to demonstrate why it is important to evaluate agents beyond single runs. When evaluating agents within the same task (intra-task), six out of seven agents showed positive average improvements when allowed to retain experience. LongCat-2.0 had the largest average improvement of approximately 0.145. When evaluating agents across different tasks (inter-task), DeepSeek-V4-Pro had an average improvement of 0.093, while some agents experienced negative transfer. In addition to experience reuse, the authors also investigated harness configurations, solution novelty, and complementary optimization benchmarks. All of these experiments provided evidence that agent performance is affected by many different parts of the research process. However, although the validation primarily shows descriptive differences in observable behaviors, it is unclear to what degree the proposed metrics measure independent underlying capabilities.

Critical strengths
  1. The paper identifies a significant gap in existing autonomous research benchmarks, which have focused on final task performance. Evaluating intermediate research processes is particularly important for understanding long term agent reliability and identifying failure modes.
  2. Interpretable and reproducible evaluation metrics. The proposed process metrics are computed from verifier outcomes and execution records rather than relying on subjective LLM based evaluations. The detailed definitions and implementation procedures make the framework relatively transparent and reproducible.
  3. Controlled investigation of experience reuse. The intra-task experience-erasure and inter-task lesson-transfer experiments provide a good approach to studying how previous research experience impacts subsequent agent performance. These controlled comparisons provide more evidence than purely observational trajectory analysis.
  4. Extensive experimental coverage and insightful findings. The evaluation was conducted using multiple frontier models and a wide variety of optimization tasks. The additional analyses of harnesses, solution novelty, and experience-transfer failures provide useful insights into the behaviors and limitations of current research agents.
Critical weaknesses
  1. Weak construct validity of the proposed process measures. While C1, C2 and C3 are understandable, they may not accurately represent the underlying abilities that they are designed to assess. For example, C1 measures both the quality of the solution and when the improvement occurred; C2 is largely based upon whether an artifact was successfully delivered; and C3 is based upon both peak retention and recovery behavior. Therefore, variations in the scores of the measures may not directly reflect variations in the intended research abilities.
  2. Sensitivity to trajectory length and checkpoint frequency. C1 has a fixed horizon of 20 checkpoints and will pad shorter trajectories and truncate longer ones. C3 can give high scores to trajectories with few or no regressions. Since models vary greatly in the number of evaluated commits, these decisions could lead to systematic bias when comparing models. The authors should conduct sensitivity analysis using different checkpoint horizons and time-normalized evaluations.
  3. Lack of evidence for long-term self-improvement. The intra-task experiments only evaluate performance at the next commit after experience retention or removal. The inter-task experiments only evaluate a single transfer step. While these experiments show short-term effects of experience reuse, they do not demonstrate whether agents can build up knowledge and continually improve over multiple research tasks.
  4. Lack of statistical uncertainty and robustness analysis. Several differences in process scores and experience gains are reported in the paper, some of which are very small. However, the authors do not provide confidence intervals or significance tests for most comparisons. Due to the stochastic nature of long-horizon agent behavior and the small number of rollouts per task, further uncertainty analysis is needed to determine if the conclusions are reliable.
  5. Limited generalizability beyond the chosen benchmark and harness. Most evaluations are conducted using AutoLab tasks within a common Claude Code harness. Although some supplemental experiments use different harnesses and benchmarks, these evaluations are still limited. Therefore, it is unknown if the proposed metrics and observed patterns will apply to larger research environments, such as open-ended scientific discovery.
  6. Confounding variables in experience-reuse interventions. In the intra-task environment, removing experience also removes code comments and other context that may be important for implementation. Similarly, transferring lessons between tasks provides additional structured guidance that may affect performance independently of actual experiential learning. More carefully controlled studies would help to establish a causal relationship between the results.
Questions for authors
  1. Are the process-level results and model rankings robust to the fixed 20-checkpoint horizon used in C1? Have the authors considered normalizing trajectories by wall-clock time, computation budget, or number of experiments to account for differences in iteration frequency?

  2. How much of the observed improvement in the intra-task evaluation can be attributed specifically to accumulated experimental experience as opposed to the additional contextual information retained by the agent? Would preserving implementation-related code comments while removing only episodic history change the results?

  3. Can the authors demonstrate that the proposed metrics can guide interventions that improve agent performance? For example, would selecting targeted harness modifications based on an agent's weakest process dimension lead to better performance on held-out tasks compared with interventions selected using final scores alone?

  4. Since C3 can assign high scores to agents with few or no regressions, how can the evaluation distinguish genuine recovery ability from simply avoiding situations that require recovery? Would separately reporting retention and recovery scores, conditioned on sufficient regression events, change the conclusions?

6 comments
AuthorsOpenReview ↗

Rebuttal by Authors (Part 1 of 4)

Dear Reviewer 9BpR,

We sincerely appreciate your detailed and constructive review, and are grateful for your supportive rating and recognition of our interpretable and reproducible process metrics, our controlled investigation of experience reuse, and the breadth of our empirical evaluation.

W1: Construct validity between the process metrics (C1–C3) and target research capabilities.

Since open-ended research capabilities cannot yet be reliably measured directly, we design C1–C3 as explicit, reproducible proxies grounded in observable trajectories. To directly address your concern about whether C1–C3 reflect the intended research capabilities, we conducted a blinded human validation on 56 randomly selected, anonymized trajectory pairs. For each pair, human evaluators independently assessed C1, C2, and C3 and assigned a win, loss, or tie (judging whether trajectory A demonstrated stronger, weaker, or comparable behavior to trajectory B in the corresponding dimension). Each pair was evaluated in two separate sessions with reversed A/B presentation order. We then computed the directional agreement between human judgments and the preferences induced by our metric scores on all pairs where both expressed a strict preference (with 95% CIs estimated via 5,000 task-block bootstrap resamples):

DimensionDirectional agreement95% CI
C193.8%83.7%–100.0%
C287.8%69.8%–100.0%
C388.7%75.6%–98.0%

Across all three dimensions, our rule-based scores show strong consistency with human evaluators, reaching 93.8% agreement on C1, 87.8% on C2, and 88.7% on C3, with all 95% CIs substantially above chance. As an initial step toward process-level evaluation of long-horizon autonomous research, this high alignment indicates that C1–C3 serve as effective, interpretable proxies for the intended research capabilities.

W2 & Q1: Sensitivity to C1 checkpoint horizon, iteration frequency, and metric hyperparameters.

Because agents autonomously determine the scope of each iteration and ground-truth verifier scores are only produced at committed checkpoints, trajectories cannot be continuously interpolated along wall-clock time without introducing artificial assumptions; instead, we report wall-clock time, token cost, and commit counts alongside our metrics (Appendices B and E) and directly test whether our process metrics are sensitive to checkpoint horizon, commit frequency, and hyperparameter choices.

Holding the trajectories and task-balanced aggregation fixed, we varied one factor at a time across C1–C3, including varying C1’s horizon across H∈{10,15,20,30,40}H \in \{10, 15, 20, 30, 40\} and subsampling every 2nd or 3rd checkpoint to simulate lower commit frequencies. Across all tested settings, model rankings remain highly consistent with the default configuration (Spearman ρ∈[0.786,1.000]\rho \in [0.786, 1.000]):

MetricSensitivity FactorSettings TestedSpearman ρ vs. Default
C1Checkpoint horizon (HH)H=10,15,20,30,40H = 10, 15, 20, 30, 40 (stage boundaries scaled proportionally)0.857–1.000 (H=10H=10: 0.964; H=15,20H=15,20: 1.000; H=30H=30: 0.893; H=40H=40: 0.857)
C1Stage weightsDefault; uniform; early-heavy; late-heavy0.893–1.000
C1Checkpoint granularityRetain every 2nd or 3rd checkpoint (preserving endpoints across offsets)0.821–0.893
C2Pre-commit repair penalty0.5×0.5\times, 1.0×1.0\times (default), and 1.5×1.5\times default penalty strength0.786–1.000
C3Retention–recovery weightRetention weight α∈{0.25,0.50,0.75}\alpha \in \{0.25, 0.50, 0.75\}0.857–1.000
C3Regression threshold (ϵ\epsilon)ϵ∈{0.005,0.01,0.02,0.05}\epsilon \in \{0.005, 0.01, 0.02, 0.05\}0.964–1.000
C3Checkpoint granularityRetain every 2nd or 3rd checkpoint (preserving endpoints across offsets)0.929–1.000
C3Intermediate-attempt discountMilder; default; stronger; no discount0.964–1.000
C3Recovery criterionFull recovery only vs. default partial-recovery scoring0.964

Specifically for C1, Claude-Opus-4.7 ranks first across all horizons H∈[10,40]H \in [10, 40]. Minor rank shifts occur only between closely matched models. For example, Gemini-3.1-Pro and GPT-5.5 (tied at 0.555 under the default H=20H=20) swap positions between H=10H=10 (where Gemini leads due to higher early capture) and H=40H=40 (where GPT leads due to stronger later refinement). Overall, our process rankings are robust to both horizon choice and iteration frequency.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 2 of 4)

W2 & Q4: Disentangling peak retention from conditional recovery in C3.

We agree that an aggregate C3 score alone does not distinguish whether a model avoids regressions or actively recovers from them, a boundary we explicitly note in §2.2.1 and address in Appendix E.2 (Fig. 14) by separately reporting Peak Retention and Conditional Recovery Credit (evaluated strictly on trajectories with ≥1\ge 1 regression episode) alongside regression exposure:

ModelAggregate C3 (Rank)Peak Retention (Rank)Conditional Recovery (Rank)Runs / Episodes with Regressions
LongCat-2.00.9280 (1)0.9615 (3)0.5195 (6)23 / 40
Gemini-3.1-Pro0.9205 (2)0.9882 (1)0.3231 (7)15 / 20
Claude-Opus-4.70.9201 (3)0.9811 (2)0.7110 (1)46 / 135
GLM-5.20.9108 (4)0.9575 (5)0.7034 (2)45 / 131
Kimi-K2.7-Code0.8745 (5)0.9321 (6)0.5287 (5)31 / 54
GPT-5.50.8577 (6)0.9594 (4)0.6136 (3)70 / 134
DeepSeek-V4-Pro0.7716 (7)0.8042 (7)0.5422 (4)32 / 56

Separating these components clarifies the underlying behavior without altering our framework's diagnostic conclusions: LongCat-2.0 and Gemini-3.1-Pro achieve the top aggregate C3 scores primarily through strong peak retention and fewer evaluated commit rounds/regressions (40 and 20 dip episodes), whereas Claude-Opus-4.7 and GLM-5.2 demonstrate the strongest actual recovery ability (0.7110 and 0.7034, ranking 1st and 2nd) despite being exposed to over three times as many regression episodes (135 and 131). In the revised manuscript, we will highlight this disaggregated breakdown directly in §4.1 to avoid over-interpreting aggregate C3 as standalone recovery ability.

W3: Evaluation of long-term self-improvement.

We agree that long-term self-improvement is an important research goal. However, because long-horizon autonomous research involves high task complexity, numerous interacting variables, and substantial computational cost, as an initial investigation we intentionally focused on single-step evaluations to cleanly isolate the effect of prior experience and avoid confounding variables:

  • Intra-task evaluation: In our early exploration, we also examined multi-commit continuations after the branch point. However, in longer continuations, we observed that agents could rediscover cleared experiences through several rounds of experimentation and feedback. Scoring the first commit limits the reset branch’s opportunity to relearn those findings, reducing the influence of newly acquired experience on the score comparison.
  • Inter-task evaluation: Extending this evaluation to a sequence of research tasks poses additional challenges. Since individual AutoLab tasks have wall-clock budgets of up to 12 hours, repeating long task sequences across multiple models would require substantial computational resources, and a reliable comparison would also need to account for differences in task difficulty and the order in which agents encounter the tasks. Moreover, even a single transfer step already uncovers substantial instability and negative transfer, indicating that understanding and stabilizing single-step reuse is an essential prerequisite before studying multi-step accumulation.

Evaluating longer-term self-improvement across sequences of autonomous research tasks therefore warrants a dedicated study, which we leave to future work.

AuthorsOpenReview ↗

Rebuttal by Authors (Part 3 of 4)

W4: Statistical uncertainty, significance tests, and rollout robustness.

We appreciate your emphasis on statistical uncertainty and robustness. To account for run-to-run stochasticity in long-horizon research tasks, our evaluation conducts three independent rollouts per model-task pair (108 rollouts per model and 756 baseline runs in total across the 36 tasks) and reports mean performance, following standard practice in AutoLab (Xu et al., 2026) and MLE-bench (Chan et al., 2025).

To provide formal uncertainty estimates and significance tests, we computed 95% confidence intervals and conducted paired bootstrap tests (100,000 task-level resamples, weighting tasks equally) across all seven models and six evaluation dimensions. The results confirm that incorporating confidence intervals largely preserves our rankings and main conclusions:

  • Overall rankings: Claude-Opus-4.7’s first-place lead over all six other models is statistically significant (all six paired 95% CIs lie above zero after Bonferroni adjustment), and 17 of the 21 model pairs have paired 95% CIs excluding zero in the exact direction reported in Fig. 2.
  • Process capabilities: Even for models with nearly identical final scores like GPT-5.5 (0.6633) and Gemini-3.1-Pro (0.6523), paired 95% CIs exclude zero in opposite directions, favoring GPT-5.5 on Execution (C2: +0.0682+0.0682 [0.0146,0.1236][0.0146, 0.1236]) and Gemini-3.1-Pro on Feedback Control (C3).
  • Experience reuse: Seven-model mean gains are significantly positive for both intra-task (+0.0768+0.0768 [0.0489,0.1059][0.0489, 0.1059]) and inter-task reuse (+0.0258+0.0258 [0.0129,0.0387][0.0129, 0.0387]), and the lowest-baseline model DeepSeek-V4-Pro gains significantly more from transfer than the top-baseline Claude-Opus-4.7 (+0.092+0.092, paired 95% CI [0.007,0.178][0.007, 0.178]).
ModelFinal ScoreC1 (Framing)C2 (Execution)C3 (Feedback)Intra-task GainInter-task Gain
Claude-Opus-4.70.7393 [0.6642, 0.8111]0.6115 [0.5467, 0.6762]0.9666 [0.9489, 0.9825]0.9201 [0.8780, 0.9561]+0.0362 [−0.0049, 0.0852]+0.0011 [−0.0398, 0.0444]
GLM-5.20.6815 [0.5995, 0.7626]0.5389 [0.4648, 0.6136]0.9365 [0.9020, 0.9643]0.9108 [0.8615, 0.9526]+0.0793 [−0.0107, 0.1785]+0.0400 [−0.0200, 0.0964]
GPT-5.50.6633 [0.5747, 0.7500]0.5553 [0.4777, 0.6339]0.9584 [0.9344, 0.9788]0.8577 [0.8035, 0.9035]+0.0730 [0.0011, 0.1479]+0.0634 [0.0011, 0.1260]
Gemini-3.1-Pro0.6523 [0.5879, 0.7187]0.5554 [0.4810, 0.6306]0.8891 [0.8391, 0.9345]0.9205 [0.8751, 0.9602]+0.1278 [0.0559, 0.2145]−0.0172 [−0.0572, 0.0264]
Kimi-K2.7-Code0.5865 [0.5125, 0.6629]0.4733 [0.3949, 0.5521]0.8804 [0.8372, 0.9210]0.8745 [0.7968, 0.9380]−0.0127 [−0.0925, 0.0576]+0.0209 [−0.0597, 0.0960]
LongCat-2.00.5724 [0.4878, 0.6596]0.4776 [0.3990, 0.5593]0.8880 [0.8418, 0.9294]0.9280 [0.8941, 0.9582]+0.1454 [0.0702, 0.2342]−0.0210 [−0.0603, 0.0129]
DeepSeek-V4-Pro0.5024 [0.4180, 0.5886]0.5188 [0.4478, 0.5917]0.8877 [0.8274, 0.9359]0.7716 [0.6894, 0.8482]+0.0885 [−0.0126, 0.2005]+0.0933 [0.0283, 0.1642]

W5: Generalizability across benchmarks, harnesses, and broader research environments.

To ensure a controlled cross-model comparison and enable in-depth trajectory analysis, our main evaluation uses a shared harness across the 36 AutoLab tasks, which already span four diverse AI R&D categories. Moreover, as you note, we further validate generalizability in the appendix across additional benchmarks (MLS-Bench and Auto NanoGPT; Appendix C) and four coding harnesses (Claude Code, Codex CLI, Kimi Code CLI, and OpenCode; Appendix D.1), where our core observations and analyses remain consistent.

Regarding open-ended scientific discovery, we agree that it is an important direction, though it falls outside the scope of the present study, which focuses specifically on verifiable AI R&D tasks. While our framework could also be applied to scientific discovery settings that provide verifiable evaluation signals, extending evaluation to broader open-ended discovery is a promising direction for future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.