acceptodds
Under review as a conference paper at ICLR 2027

Beyond Data Quantity: From Bootstrap Emergence to Composition-Sensitive Self-Evolution

Abstract

Self-evolving agents aim to reduce reliance on expert supervision by learning from their own interaction experience, yet it remains unclear how self-evolution emerges from an initially unreliable agent and what limits further improvement once executable self-generation becomes possible. We study this process in a controlled tool-augmented VideoQA setting with nested, task-balanced expert supervision and controlled within-task solution logic. Our experiments reveal two empirically distinct regimes. Before bootstrap, performance is strongly sensitive to expert supervision. A small increase in structured trajectories can transform an unreliable planner into one capable of executable self-generation. After bootstrap, further increases in expert trajectory quantity yield diminishing and strongly task-dependent gains, and this pattern persists after matching the optimization budget. We further characterize this post-bootstrap regime. Compared with continued expert-only training, self-generated experience produces a substantially broader set of successful tool-use patterns. At the action level, residual failures exhibit heterogeneous bottlenecks across generation, execution, and correct completion. Finally, controlled interventions show that repeatedly exposing the model to existing trajectories or applying coarse rebalancing provides limited benefit, whereas changing trajectory composition or fine-grained action-level supervision can alter the affected capability. Together, these results identify a transition from quantity-sensitive bootstrap emergence to composition-sensitive post-bootstrap learning, providing an empirical characterization of how tool-augmented self-evolution changes after it becomes executable.

Primary area:
learning theory
Keywords:
Self-evolution, Vision-Language Models, Data Scaling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Wrb9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the role played by a limited number of expert trajectories for a planner in the process of beginning generation of executable trajectories in a VideoQA environment when the tools and the multimodal worker are held constant. Also, how helpful is the addition of more and more expert data after this point? Three independent runs were performed and their results analyzed from the perspective of behaviors. The authors found out that performance is data-dependent prior to bootstrap, whereas adding additional data afterwards yields decreasing returns depending on the task. Behavior-based weighting and trajectory modifications were also explored in order to examine these phenomena.

Critical strengths
  1. The experimental control conditions and trajectory counting are well described. Expert sets are nested and balanced for tasks, and the main experiment results are provided for three independent experiments. Moreover, the appendix gives the number of generated, correct, accepted and really added training trajectories for the first run of the experiment. It allows me to understand the actual data collected by self-training (Tables 1-4, 13-15).

  2. The comparison of different kinds of experience that could be added is informative. SVA-Random and SVA-Enhanced add equal numbers of trajectories, but the latter shows better results. Comparing to the basic setup, it increases the overall accuracy from 39.1% to 43.7%, and SVA accuracy from 11.5% to 19.8%. I see this as one of the paper's most useful findings, as it suggests that the content of the added experience deserves closer study (Tables 10 and 18).

Critical weaknesses
  • There are some inconsistencies in experimental details. First, Section 4.2 says that small data cases are paired with 1,415 post-bootstrap retraining updates, while Appendix A.7.2 mentions expert-initialization updates. The difference is important because these controls serve different purposes, which is crucial for my interpretation of diminishing returns. Second, the frequencies of the 13 rows in Table 16 total 1,828, which is 35 less than the number of action spans provided in the caption (1,863). I need the full frequency numbers and the set of behaviors used for normalization to verify the BAW weights.

  • The trajectory-composition experiments are not controlled enough to determine what makes the difference. It is stated in Appendix A.7.3 that expert initialization dataset is changed at first, then comes the rest of the pipeline. Therefore, the experiments prove that expert dataset composition influences the final self-evolution, yet they do not prove its effect on initialization, trajectory creation, and retraining separately. BAW is applied to the combined post-bootstrap training set, and I appreciate the evidence it gives about supervision allocation.To support the further claim that the learning mechanism changes after bootstrap, I would like to see a comparison starting from the same initialized checkpoint and changing only the subsequent data composition.

Questions for authors
  1. Are the 1,415 updates in Table 9 related to expert initialization or retraining after bootstrapping? Can you provide the number of updates, training units, and data size at each step, and explain why the main text and Appendix mention different steps?

  2. Why is there a difference of 35 action spans between the mentioned frequencies and the total in Table 16? Are there any additional unmentioned behaviors that participate in the normalization in Eq. (10)? Please specify the whole list of frequencies and weights because they influence my understanding of the BAW experiment.

  3. Would it be possible to start from the same expert-initialized checkpoint, change only the composition of the post-bootstrap training data, and match updates and training tokens? In the absence of such control, I suggest constraining your conclusions in Table 10 to the effect of expert data composition on the final results of self-evolution.

Reviewer_2LL3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how tool-augmented agents bootstrap self-evolution from limited expert supervision and what limits further improvement after successful self-generation becomes possible. Using VideoQA as a controlled testbed, the authors identify two distinct learning regimes: a quantity-sensitive bootstrap stage and a composition-sensitive post-bootstrap stage. The paper further introduces Stage-wise Behavior Analysis to characterize behavioral bottlenecks and Behavior-Aware Weighting (BAW) to investigate the effects of fine-grained supervision. Extensive experiments and controlled interventions suggest that the composition of training trajectories, rather than merely their quantity, plays an important role in post-bootstrap learning.

Critical strengths
  1. The proposed Stage-wise Behavior Analysis is intuitive and well-designed. Decomposing agent failures into behavior generation, valid execution, and correct completion provides a more informative characterization of agent capabilities than trajectory-level accuracy alone. The Behavior-Aware Weighting (BAW) approach is conceptually sound and easy to implement.
  2. The paper conducts experiments across multiple supervision scales, two VideoQA benchmarks, and different model configurations. The comparisons between continued training and self-evolution, matched optimization budgets, and different trajectory-composition interventions help disentangle the effects of data quantity, optimization, and experience composition.
  3. The observation that supervision quantity plays different roles before and after bootstrap is interesting and potentially valuable. In particular, the finding that self-generated trajectories introduce more diverse successful tool-use patterns provides useful evidence for understanding the benefits of agent self-evolution beyond additional training.
Critical weaknesses
  1. While the authors position tool-augmented VideoQA as a controlled environment for studying agent self-evolution, the rationale for selecting this domain is not sufficiently developed. Why is VideoQA particularly suitable for investigating bootstrap emergence and post-bootstrap learning compared with more widely studied agent settings, such as coding agents and computer-use agents (CUA)? What unique challenges does VideoQA introduce, and which aspects of the proposed findings are expected to generalize to other agent environments? A clearer discussion of these questions would strengthen the paper's motivation and broader relevance.
  2. The self-evolution framework relies on ground-truth answers to filter self-generated trajectories. Although this design helps isolate learning dynamics, it assumes access to a reliable verifier, which may not be available in realistic autonomous-agent settings. It would be useful to investigate whether the conclusions remain valid under imperfect verification or weaker supervision.
Questions for authors

See the weakness part.

Reviewer_L7LtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the amount and composition of expert supervision affect self-training in tool-augmented VideoQA. The authors vary the size of the expert dataset and examine behavioral reweighting and targeted trajectory additions. They report that limited expert supervision can enable executable self-generation, while further gains become task-dependent and sensitive to the composition of the training data.

Critical strengths

The nested, task-balanced expert subsets and fixed tool environment make the comparisons across supervision sizes easier to interpret. This is a useful setup for studying when expert supervision enables self-training and how its benefits change with scale.

Critical weaknesses

Expert diversity. My concern is how much the observed diminishing returns depend on the construction of the expert data. Appendix A.3 explains that demonstrations follow a controlled solution structure within each task, so adding examples may introduce relatively little strategic diversity. Tables 5 and 8 analyze generated trajectories, but do not show how behavioral coverage in the expert data changes with supervision size. This makes it difficult to judge whether the same trend would hold with more varied expert strategies.

Source of gains. Table 10 does not show whether SVA-Enhanced improves the model before self-training, increases the gain from self-training, or both. Appendix A.7.3 states that the interventions modify the expert data before initialization and rerun the full pipeline. The final improvements are useful, but their interpretation as evidence about post-bootstrap learning remains unclear.

Training budget. Section 4.2 describes the matched 1,415 updates as post-bootstrap retraining, whereas Appendix A.7.2 describes them as expert initialization. These are different controls, and the distinction matters for interpreting Table 9. The authors should clarify which stage was actually budget-matched.

Questions for authors

As the expert dataset grows, how does its coverage of behaviors and solution strategies change? Does it mainly add examples of existing patterns, or introduce additional patterns?

Could the authors report accuracy after expert initialization and after self-training for the Table 10 interventions?

Were the 1,415 matched updates in Table 9 applied during expert initialization or during post-bootstrap retraining?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.