- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies post-training of LLM agents for text-based household planning. A demonstration provides only one trajectory, so it does not show the outcomes of alternative actions or how to recover from mistakes. The paper builds on Ego2World, which compiles egocentric video annotations into deterministic, partially observed symbolic environments whose states can be saved and restored. EgoVine uses these environments to execute choices that are not in the recorded trajectory, and converts the resulting alternatives, failures, and repairs into supervised targets (Union) and branch-relative policy updates.
Critical strengths
-
The problem is interesting. Expanding experience beyond the demonstrated trajectory by executing alternatives, failures, and repairs in restorable environments is a compelling approach.
-
Adding recovery experience to supervised training substantially improves performance (+15.0 pp for the 0.5B model, Appendix F.1), and the trained planners outperform Qwen Flash and DeepSeek Flash on the Main panel.
Critical weaknesses
-
Evaluation limited to a self-constructed benchmark that overlaps with the training distribution. All results are reported on panels constructed by the authors, and the evaluated worlds include sources used during training and development (Section 6, Appendix F). Generalization to unseen videos or tasks is therefore not measured.
-
Different training recipes for the two models. The 0.5B model is trained with Union, recovery-expanded supervision, and a policy update, whereas the 1.5B model is trained with reference-action supervision followed by branch-relative RL, without Union (Appendix C.3). The paper does not explain this choice. Since model size and training recipe differ at the same time, the results do not show which components contribute to the performance of each model.
-
Limited evidence for the main components. The reported results do not show clear improvements from several components of the framework.
- Union: there is no direct comparison with reference-only SFT at the same model size, and the auxiliary outputs (Predict, Select, Rejoin) are not evaluated (Appendix H.3). The 1.5B model, which performs best on Main and Long, does not use Union.
- Branch-relative RL: the improvements over the SFT parents are +1.37 pp for 0.5B, and +0.59 pp and +5.10 pp for 1.5B, and all confidence intervals include zero (Appendix F).
- Iterative collection: refreshing the collector yields a difference of 0.00 pp (Appendix H.6).
- Readability. The paper is difficult to follow.
- It introduces many new terms (Union, BunchFusion, Bunch TreeRL, rejoin, connector, certificate).
- The main text consists mostly of abstract statements, while concrete definitions are scattered across Appendix B.
- Provenance details such as adapter hashes, seed numbers, and file paths take up considerable space in the main text and appendix.
- Section 3 reads more like a method overview than a problem formulation.
Questions for authors
Please read the weakness above.
- How does EgoVine perform on held-out videos that were not used during training or development?
- Why are the 0.5B and 1.5B models trained with different recipes? Could the authors report results for both sizes under the same recipe?
- Could the authors provide ablations at the same model size that isolate the effects of Union (compared with reference-only SFT), branch-relative RL, and iterative collection?
- Figures 1 and 2 appear to be embedded as raster images, while the result plots (Figures 3–9) are vector graphics.
Reminder: Please Review Authors’ Rebuttal and Provide Feedback
Dear Reviewers,
Thank you for your reviews. If you have not already done so, please read the authors' rebuttal and share your feedback in the discussion thread at your earliest convenience. Your timely responses would be greatly appreciated.
Best,
AC