acceptodds
Under review as a conference paper at ICLR 2027

What is Missing from AI Post-Training AI: An Empirical Analysis

Abstract

Large language model (LLM) agents can now post-train an LLM end-to-end, raising the prospect of recursive self-improvement (RSI). Yet this progress is measured by aggregate benchmark scores, which cannot tell whether an agent executes a fixed plan well or strategically revises the plan when it fails. We separate these two capabilities: execution-level capability, iterating within an established training strategy, and strategy-level capability, revising that strategy as experimental evidence accumulates. Analyzing 1,338 post-training trajectories of frontier agents, we find that agents reliably execute post-training but lock into a default strategy, which follows the agent rather than the task, and only 2.1% of transitions between adjacent training runs ever change strategy. We then test whether the agent lacks experience, reasoning, or the decision to switch. (1) Experience improves execution but not the strategy. (2) Additional reasoning compute yields front-loaded gains on easier tasks but refines, rather than revises, the committed strategy. (3) Human review before training changes which strategy the agent locks into, not whether it locks in, whereas a single mid-run instruction outperforms the agent's own continuation by up to 17.44 points under the same budget. In conclusion, what the agent lacks is the decision to reopen a committed strategy and try another one. Realizing RSI therefore calls for interaction protocols and training signals that make strategy revision an explicit, rewarded decision. We release our code at https://anonymous.4open.science/r/AI-PostTraining-AI.

Primary area:
foundation or frontier models, including LLMs
Keywords:
AI-for-AI, LLM Post-Training, Recursive Self-Improvement

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_t1yoOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper examines how effectively LLM agents can post-train other LLMs, distinguishing between two capabilities: execution, or implementing a specified strategy, running experiments, and producing artifacts; and strategy exploration, or exploring alternative approaches to post-training. An analysis of 1,338 post-training trajectories from PostTrainBench suggests that current agents are capable executors but struggle to explore alternatives once they have committed to a strategy. Additional analyses suggest that providing more context, increasing the reasoning budget, and introducing human intervention do not reliably produce meaningful strategic diversity.

Critical strengths
  1. The paper addresses an important question about LLM agents’ ability to explore. Success on complex, long-horizon tasks depends on exploring the space of possible approaches, beyond generating or executing a single plausible solution. Distinguishing execution from strategy exploration provides a useful lens for assessing agent capabilities.

  2. The analysis goes beyond final benchmark scores. Examining agents’ execution processes and strategic choices helps explain why they succeed or fail at post-training. This process-level perspective provides information that aggregate performance metrics alone cannot capture.

Critical weaknesses

The main concern is the novelty of the findings and their specificity to post-training. Although the question is important, the paper does not clearly establish what studying it in this setting reveals beyond existing observations that LLMs produce limited diversity and become anchored to earlier assumptions or responses in multi-turn and long-horizon tasks. Relevant prior work includes:

  • LLMs Get Lost in Multi-Turn Conversation (https://arxiv.org/abs/2505.06120), which analyzes how early assumptions and mistakes persist across conversational turns, causing models to become increasingly unreliable.

  • Old Habits Die Hard: How Conversational History Geometrically Traps LLMs (https://arxiv.org/abs/2603.03308), which studies how conversational history biases subsequent generations.

  • Escaping the Echo Trap: On Credit Assignment Failure in Multi-turn LLM Self-Reflection (https://aclanthology.org/2026.acl-long.1636/), how multi-turn self-reflection can reinforce repeated strategies and suppress exploration, and proposes a modified optimization method to address the issue.

Without a clearer distinction from these findings, the contribution appears primarily to be a case study of known limitations in the post-training setting. This limits my assessment of its conceptual novelty, despite the value of the empirical analysis.

Questions for authors

Follow-up on the weakness, how do simple exploration baselines, such as independent resampling or restarting without prior trajectory context, affect strategic diversity and final performance? Do agent harnesses that explicitly encourage alternative strategies alleviate the observed exploration failure? These comparisons would help establish whether the limitation is intrinsic to the models or substantially shaped by the interaction setup.

Reviewer_ThZSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work explores the strategies and decision making of frontier LLM agents tasked with post-training small base LLMs. The work explores common errors in the agents’ post-training strategies and implements methods to reduce those errors. Most notably, agents tend to adopt a single training method through the whole post-training process, likely resulting in suboptimal outcomes. Providing guidance (from an LLM, human, or skill library) improves outcomes, but does not lead to more varied training strategies.

Critical strengths

The topic is interesting, the results point towards meaningful failure modes of automated post-training, and the paper is overall well-written. The difference in performance between the baseline performance and experience-driven performance is substantial, suggesting that the agents make meaningful strategy errors without this guidance. A large number of trajectories are analyzed, and the authors are fairly thoroughly in exploring the failure modes and how they can be remedied.

Critical weaknesses
  • The mathematical definitions of “execution-level gap” and “strategy-level gap” seem to not be used to analyze the empirical results (and they seem highly-nontrivial to compute). Yet, the authors make claims about these quantities that appear unsupported by data. For example, “The execution-level gap in equation 1 leaves limited headroom” seems not supported by data. If these definitions aren’t used, perhaps they should be removed.
  • The authors seem to assume that a low rate of switching strategies is suboptimal, rather than showing that it is suboptimal. Sentences like “Among the 3,557 adjacent training pairs, only 74 (2.1%) ever probe an alternative” imply that a 2.1% switch rate is irrational or suboptimal, but this needs to be justified more directly.
  • Providing some key pieces of information about posttrainbench in the main text could make the paper clearer. For example, where does the training data come from? Do agents generate their own SFT data? Do they generate their own RL environments?
  • I think section 2.1 should describe how checkpoints are handled, and should state whether decisions about retaining checkpoints are considered execution-level or strategy-level.
  • What the authors call “experience” is, in my view, broader than the word “experience” suggests. For example, having a second agent providing feedback based on a separate prompt that encourages actionable suggestions seems to be providing more than just experience. This isn’t necessarily a problem with the content, but perhaps choose a different term.
  • Some claims seem overstated. For example:
    • “Scaling agent context and reasoning compute therefore has a clear ceiling”. The data doesn’t seem to support there being a clear ceiling.
    • “Measuring RSI Progress Requires Strategy-Level Metrics” is too strong. Perhaps “Strategy-level metrics give insight into RSI progress” would be more accurate.
  • Some writing is unclear. I can’t parse the meaning of “Scaling deepens the loop that already closes,” which is written as a headline conclusion. The word “reopen” in the context of “reopening the committed strategy” seems to be used in an odd way.
  • Several of the figures are insufficiently specified. For example, the meaning of “RFT -> GRPO” in Figure 6b is unspecified. Figure 5 appears to show averages of the datapoints, yet the averages seem to not track the datapoints. Was some kind of extreme smoothing or simple functional fit applied? Also, what is the shading showing? The diagrams in Figure 4 are unexplained.
Questions for authors

Most of my questions are listed in the Weaknesses section above. Two other questions are:

  • Is the evaluator agent prompted to suggest alternative strategies? I think more detail about the prompting would be helpful
  • Does the skill library contain training recipes that simply instruct the agent how to optimally perform post-training?
Reviewer_WAGiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper analyzes 1338 LLM-agent post-training trajectories and conducts controlled interventions, finding that agents execute training effectively but rarely revise their initial strategies, while explicit mid-run human guidance can overcome this rigidity and improve performance.

Critical strengths
  • The question is timely and well posed. Separating execution-level from strategy-level capability gives a concrete lens on what aggregate post-training benchmark scores hide (Section 1).

  • The trajectory analysis is large-scale: 1,338 PostTrainBench trajectories across seven benchmarks, four base models and three agent frameworks. The headline is striking and clearly stated: only 74 of 3,557 adjacent pairs change strategy, and the initial strategy follows the agent rather than the task (Section 3; Table 1).

  • The paper tests three explanations (experience, reasoning compute, decision) with concrete interventions. These include the evaluator analysis (all 22 execution-level suggestions adopted vs. 0 of 21 strategy-level ones) and same-checkpoint forks under a matched budget. Limitations, including retrospective branch selection, are candidly acknowledged, and code is released (Section 4; Appendix D; Limitations).

Critical weaknesses
  • The robustness of the pooled 2.1% switch rate (74/3,557 adjacent pairs) is unclear. 733 of 5,111 experiments lack algorithm labels and are skipped over when pairs are formed, which can hide out-and-back switches (15 of 35 algorithm transitions return to SFT; Appendix A.4). Data provenance is identifiable for only 1,801 of 4,344 supervised experiments (1,401 pairs), yet Table 5 normalizes data-source changes by all 3,557 pairs. The three-way data taxonomy (curated / self-generated / mixed) also does not register a switch between curated datasets as a change (Appendix A.3).

  • The forks show that better outcomes were reachable, but they do not separate a missing decision from missing knowledge of which alternative to try. The reconsider fork (Appendix D.2.3) removes both the cue and the named alternative. The evaluator's ignored suggestions (Table 7) come from different trajectories and are suggestions, not directives. There is one fork per benchmark, and branch points and alternatives were selected after the runs completed, as the authors acknowledge (Appendix D.2; Limitations). In addition, the recorded GSM8K continuation (47.76%) is far below the experience-driven mean in Table 2 (77.30 ± 3.8), so the gain of up to 17.44 points is measured against an unusually weak baseline.

  • Section 4.2 never varies reasoning compute. Fig. 5 plots observed cumulative tokens within runs and across scaffolds (1.8–9.2× apart), so token use is confounded with elapsed time, training progress and scaffold. This supports diminishing observed returns, not the claimed "clear ceiling" on reasoning compute.

Questions for authors
  1. What are the switch rates on pairs where both algorithm and data source are identified, without skipping over unlabeled runs, and under finer categories such as dataset identity? Please also report the per-trajectory distribution of ρ(τ).

  2. Do the gains hold with prespecified branch points across multiple trajectories and seeds? Please include an arm that is told the same alternative exists but is not instructed to switch.

  3. What happens when the main agent's reasoning or token budget is varied within the same scaffold at a fixed wall-clock and GPU budget? Please report switch rates as well as scores.

1 comment

Dear Reviewer WAGi, thanks again for your detailed comments. We have addressed your concerns as fully as possible by clarifying the annotation and pair-construction protocol, the intervention controls, and the interpretation of token usage, and by revising the corresponding parts of the manuscript. As the rebuttal deadline is approaching, we would be grateful if you could review our response and let us know whether any concern remains. We would very much appreciate the opportunity to discuss your feedback further.

Reviewer_JbxJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies the limitations of LLM agents in autonomous post-training, distinguishing execution-level optimization from strategy-level revision. Analyzing 1,338 PostTrainBench trajectories, the authors find that agents effectively refine training pipelines but rarely change strategies (2.1% switch rate). Controlled experiments suggest that additional experience and reasoning compute mainly improve execution, while human guidance can trigger strategy changes and improve performance.

Critical strengths

I think the paper is working on a timely and important problem in autonomous AI research and RSI. I appreciate the detailed trajectory-level analysis and controlled experiments, particularly the experience-driven framework and its component ablations. I also find the paper is well-written and very easy to follow.

Critical weaknesses
  • Limited novel insights in the trajectory analysis

I find the incremental contribution of Section 3 somewhat limited. PostTrainBench [1] (Section 5.2), whose trajectories this paper reanalyzes, has already observed that agents predominantly rely on SFT, exhibit agent-specific preferences for training methods, and repeatedly refine data and hyperparameters rather than exploring alternative training approaches. The main findings in Section 3 largely reinforce these existing observations. While the proposed execution-level versus strategy-level distinction and the 2.1% switch rate provide a more quantitative characterization, they offer limited additional insight into the underlying behavior. In particular, the analysis does not go much beyond documenting how rarely agents change strategies to explain when or why this behavior becomes problematic.

  • Insufficient consideration of the exploration–exploitation trade-off under limited budgets

The paper interprets the low strategy-switch rate (2.1%) as evidence of a major bottleneck in current research agents. However, I am not convinced this conclusion necessarily follows. Under a fixed compute budget, exploring alternative strategies comes at the cost of further optimizing the current one. A good research agent should balance exploration and exploitation, rather than simply switch strategies more often. The paper does not systematically examine this trade-off or establish whether allocating more resources to strategy exploration would actually improve overall performance. I think this is a fundamental limitation, as the analysis primarily shows that agents rarely switch strategies, without demonstrating that their current allocation of effort between strategy exploration and execution-level refinement is suboptimal.

  • The underlying cause of strategy lock-in remains unclear

Even if we accept strategy lock-in as a limitation, I am not convinced the paper identifies its underlying cause. The authors conclude that agents lack the decision to switch, but the human-guided interventions also specify which alternative strategy to pursue. The observed improvements could therefore result from the human's ability to identify a better strategy, rather than simply triggering a switching decision. These experiments show that agents can execute alternative strategies when guided, but do not establish that the decision to switch itself is the primary bottleneck.

[1] Rank, B., Bhatnagar, H., Prabhu, A., Eisenberg, S., Nguyen, K., Bethge, M., & Andriushchenko, M. (2026). PostTrainBench: Can LLM Agents Automate LLM Post-Training? Proceedings of the 43rd International Conference on Machine Learning (ICML). arXiv:2603.08640.

Questions for authors

See weakness above.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal and Revisions

We thank the reviewers and the AC for their time and thoughtful feedback. Reviewers generally recognized the importance of our study, while raising a central question: Do agents struggle to initiate a strategy change, or to effectively execute an alternative strategy once a change is initiated?

To directly address this concern, we conducted additional controlled experiments comparing optional suggestions with explicit instructions, using the same starting checkpoint and remaining budget. Optional suggestions prompted agents to explore alternatives, but they spent substantial effort on tuning and validation with limited gains. In contrast, when explicitly instructed on which strategy to switch to, agents implemented the alternative efficiently and achieved better final performance. These results demonstrate that strategy execution is not the bottleneck; rather, the key challenge lies in recognizing when and how to switch strategies. We also clarified the study scope, annotation protocol, intervention controls, random seed settings, and interpretation of token usage in the revised manuscript.

Overall, we believe this work provides a self-contained empirical study of strategy revision in AI post-training AI, within its clearly defined RSI scope. Our central finding is that agents can effectively execute a chosen post-training strategy, yet struggle to revise it even when evidence and viable alternatives are available. Through trajectory analysis and controlled interventions, we identify and characterize this gap, moving beyond execution-level capability to examine agents' ability to make and revise research decisions. We believe this is an important and underexplored aspect of RSI that current execution-focused harness improvements cannot address. Our findings establish a concrete empirical basis for studying strategy revision and motivate future work on mechanisms such as policy-level learning signals.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.