- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explores the role played by a limited number of expert trajectories for a planner in the process of beginning generation of executable trajectories in a VideoQA environment when the tools and the multimodal worker are held constant. Also, how helpful is the addition of more and more expert data after this point? Three independent runs were performed and their results analyzed from the perspective of behaviors. The authors found out that performance is data-dependent prior to bootstrap, whereas adding additional data afterwards yields decreasing returns depending on the task. Behavior-based weighting and trajectory modifications were also explored in order to examine these phenomena.
Critical strengths
-
The experimental control conditions and trajectory counting are well described. Expert sets are nested and balanced for tasks, and the main experiment results are provided for three independent experiments. Moreover, the appendix gives the number of generated, correct, accepted and really added training trajectories for the first run of the experiment. It allows me to understand the actual data collected by self-training (Tables 1-4, 13-15).
-
The comparison of different kinds of experience that could be added is informative. SVA-Random and SVA-Enhanced add equal numbers of trajectories, but the latter shows better results. Comparing to the basic setup, it increases the overall accuracy from 39.1% to 43.7%, and SVA accuracy from 11.5% to 19.8%. I see this as one of the paper's most useful findings, as it suggests that the content of the added experience deserves closer study (Tables 10 and 18).
Critical weaknesses
-
There are some inconsistencies in experimental details. First, Section 4.2 says that small data cases are paired with 1,415 post-bootstrap retraining updates, while Appendix A.7.2 mentions expert-initialization updates. The difference is important because these controls serve different purposes, which is crucial for my interpretation of diminishing returns. Second, the frequencies of the 13 rows in Table 16 total 1,828, which is 35 less than the number of action spans provided in the caption (1,863). I need the full frequency numbers and the set of behaviors used for normalization to verify the BAW weights.
-
The trajectory-composition experiments are not controlled enough to determine what makes the difference. It is stated in Appendix A.7.3 that expert initialization dataset is changed at first, then comes the rest of the pipeline. Therefore, the experiments prove that expert dataset composition influences the final self-evolution, yet they do not prove its effect on initialization, trajectory creation, and retraining separately. BAW is applied to the combined post-bootstrap training set, and I appreciate the evidence it gives about supervision allocation.To support the further claim that the learning mechanism changes after bootstrap, I would like to see a comparison starting from the same initialized checkpoint and changing only the subsequent data composition.
Questions for authors
-
Are the 1,415 updates in Table 9 related to expert initialization or retraining after bootstrapping? Can you provide the number of updates, training units, and data size at each step, and explain why the main text and Appendix mention different steps?
-
Why is there a difference of 35 action spans between the mentioned frequencies and the total in Table 16? Are there any additional unmentioned behaviors that participate in the normalization in Eq. (10)? Please specify the whole list of frequencies and weights because they influence my understanding of the BAW experiment.
-
Would it be possible to start from the same expert-initialized checkpoint, change only the composition of the post-bootstrap training data, and match updates and training tokens? In the absence of such control, I suggest constraining your conclusions in Table 10 to the effect of expert data composition on the final results of self-evolution.