acceptodds
Under review as a conference paper at ICLR 2027

EgoVine: Post-Training with Branching Experience Compiled from Real-World Videos

Abstract

Real-world egocentric videos capture multi-step household procedures, but their recorded trajectories leave alternative actions and recovery from the learner's own mistakes largely unexplored. We introduce EgoVine, a post-training framework that turns a fixed demonstration set into interactive experience for long-horizon planning. EgoVine operates in executable symbolic environments compiled from real-world video annotations, where an agent must coordinate object manipulations under partial observation and account for delayed consequences. Restoring a decision state allows the learner to explore alternative actions and compare their eventual outcomes. Union jointly trains action generation and consequence prediction in one model, using the same executions to supervise candidate comparison and recovery. Verified repairs add recovery targets from failed attempts, while comparisons among sampled continuations guide policy updates. The updated planner then returns to the environments, with its own decisions and mistakes shaping the next round of experience. On a shared benchmark, EgoVine-1.5B achieves 81.8% success on Main planning tasks and 66.7% on longer task compositions, outperforming all evaluated Flash LLM baselines on both. EgoVine connects real demonstrations to an iterative learning process that trains compact planners on choices and futures beyond the recorded trajectories.

Primary area:
applications to robotics, autonomy, planning
Keywords:
egocentric video, interactive learning, executable environments, language model planning, post-training, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_VqBhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies post-training of LLM agents for text-based household planning. A demonstration provides only one trajectory, so it does not show the outcomes of alternative actions or how to recover from mistakes. The paper builds on Ego2World, which compiles egocentric video annotations into deterministic, partially observed symbolic environments whose states can be saved and restored. EgoVine uses these environments to execute choices that are not in the recorded trajectory, and converts the resulting alternatives, failures, and repairs into supervised targets (Union) and branch-relative policy updates.

Critical strengths
  1. The problem is interesting. Expanding experience beyond the demonstrated trajectory by executing alternatives, failures, and repairs in restorable environments is a compelling approach.

  2. Adding recovery experience to supervised training substantially improves performance (+15.0 pp for the 0.5B model, Appendix F.1), and the trained planners outperform Qwen Flash and DeepSeek Flash on the Main panel.

Critical weaknesses
  1. Evaluation limited to a self-constructed benchmark that overlaps with the training distribution. All results are reported on panels constructed by the authors, and the evaluated worlds include sources used during training and development (Section 6, Appendix F). Generalization to unseen videos or tasks is therefore not measured.

  2. Different training recipes for the two models. The 0.5B model is trained with Union, recovery-expanded supervision, and a policy update, whereas the 1.5B model is trained with reference-action supervision followed by branch-relative RL, without Union (Appendix C.3). The paper does not explain this choice. Since model size and training recipe differ at the same time, the results do not show which components contribute to the performance of each model.

  3. Limited evidence for the main components. The reported results do not show clear improvements from several components of the framework.

  • Union: there is no direct comparison with reference-only SFT at the same model size, and the auxiliary outputs (Predict, Select, Rejoin) are not evaluated (Appendix H.3). The 1.5B model, which performs best on Main and Long, does not use Union.
  • Branch-relative RL: the improvements over the SFT parents are +1.37 pp for 0.5B, and +0.59 pp and +5.10 pp for 1.5B, and all confidence intervals include zero (Appendix F).
  • Iterative collection: refreshing the collector yields a difference of 0.00 pp (Appendix H.6).
  1. Readability. The paper is difficult to follow.
  • It introduces many new terms (Union, BunchFusion, Bunch TreeRL, rejoin, connector, certificate).
  • The main text consists mostly of abstract statements, while concrete definitions are scattered across Appendix B.
  • Provenance details such as adapter hashes, seed numbers, and file paths take up considerable space in the main text and appendix.
  • Section 3 reads more like a method overview than a problem formulation.
Questions for authors

Please read the weakness above.

  1. How does EgoVine perform on held-out videos that were not used during training or development?
  2. Why are the 0.5B and 1.5B models trained with different recipes? Could the authors report results for both sizes under the same recipe?
  3. Could the authors provide ablations at the same model size that isolate the effects of Union (compared with reference-only SFT), branch-relative RL, and iterative collection?
  4. Figures 1 and 2 appear to be embedded as raster images, while the result plots (Figures 3–9) are vector graphics.
Reviewer_xWxnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EgoVine, a post-training framework that turns a fixed demonstration set into interactive experience for long-horizon planning. EgoVine operates in executable symbolic environments compiled from real-world video annotations. Each execution provides corresponding targets for action choice, consequences, and continuation, with the checked outcome determining its supervision. On a shared benchmark, EgoVine-1.5B achieves 81.8% success on Main planning tasks and 66.7% on longer task compositions, outperforming all evaluated Flash LLM baselines on both.

Critical strengths
  1. Interesting Motivation: Turns a fixed demonstration set into interactive experience for long-horizon planning. EgoVine operates in executable symbolic environments compiled from real-world video annotations.
  2. Reasonable Framework: EgoVine introduces Union to jointly learn action generation, consequence prediction, candidate selection, and recovery.
  3. Results: On a shared benchmark, EgoVine-1.5B achieves 81.8% success on Main planning tasks and 66.7% on longer task compositions, outperforming all evaluated Flash LLM baselines on both。
Critical weaknesses
  1. The evaluation is limited to text-based planning in symbolic environments. It remains unclear whether the demonstrated improvements can generalize to unseen environments or real-world embodied execution.
  2. Insufficient ablation of RL training components. The paper introduces Bunch TreeRL, but not sufficiently demonstrated. According to Table 11, most of the performance improvement comes from recovery-expanded SFT (215/338 to 269/338), the RL policy update provides only a marginal gain (269/338 to 274/338).
  3. Limited evaluation of generalization to unseen scenes. The evaluations are conducted in symbolic environments from real-world videos, and the evaluation includes video sources used during training. It is unclear whether the method can generalize to new environments.
Questions for authors
  1. Provide more results to demonstrate whether the method can generalize beyond symbolic environments, such as to visual observations or real-world embodied tasks.
  2. Provide further ablation studies to clarify the contribution of Bunch TreeRL.
  3. Provide evaluation results on completely unseen video sources.

See Weaknesses.

Reviewer_EQYfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a post-training framework designed to improve performance on long-horizon planning by turning demonstrations into “environments” for interactive experience.

Critical strengths
  1. The paper demonstrates a principled way in which one can use on-policy search and policy gradient to optimize a policy when one has access to an executable world. The paper shows how a single demonstration can be used as a reliable starting point for this search tree in an effective manner.
  2. The paper shows that this principled objective can be hillclimbed using the executable world in a reliable manner and shows promising results against Qwen Flash and DeepSeek Flash.
Critical weaknesses
  1. The proposed method relies on the existence of Ego2World in the first place. Without such executable worlds, the proposed method cannot execute choices and cover counterfactuals and get supervision for state distributions of various actors. It is unclear to me whether it is realistic to assume that such worlds can be created for tasks in general, since the construction of Ego2World itself is highly expensive (for example, HD-Epic has 41 hours of videos capturing 69 recipes in over 400 kitchens averaging over 250 annotations per minute, which is quite a lot for a task that is of a single type really). Furthermore, the mapping of verbs into executable action types is done manually and seems quite expensive/impractical in a more general setting. Ego2World paper itself shows us that one cannot simply automate the construction of such worlds with LLMs as it leads to extremely high hallucination rates and low coverage of state key changes, etc.
  2. The limitations of the worlds carry over to Ego2Vine as well. For example, when encountering novel action combinations beyond the manually curated framework, the system fails during precondition verification despite the action sequence being perfectly physically feasible.
  3. Continuing from point (2): this is crucial because the main strength in EgoVine is that you are allowing a model to explore counterfactuals from a demonstration to get supervision on a much broader state visitation distribution. However, if the executable world only works on state transitions that were compiled from the demonstration recordings, then it seems like the distribution of counterfactuals you can cover is also limited.
  4. The choice of models being tested against are somewhat limited. Even if we simply consider small and fast models, it is worth considering Gemma or Gemini Flash for example. In fact, since you are allowing your own model to be trained in the executable world while the baselines are frozen, it would be wise to consider comparing against more models to see how far your RL fine-tuning can take EgoVine compared to larger frozen models as well.
  5. There is a lack of sufficient baselines and ablations for the method proposed. These include a standard GRPO baseline, DAgger baseline, and other important ablations pertaining to the search algorithm (other search strategies such as UCB/PUCT action selection could be informative) for instance.
Questions for authors

Please see the discussion on weaknesses.

Comments on the paper

1 comment
Area_Chair_ptABOpenReview ↗

Reminder: Please Review Authors’ Rebuttal and Provide Feedback

Dear Reviewers,

Thank you for your reviews. If you have not already done so, please read the authors' rebuttal and share your feedback in the discussion thread at your earliest convenience. Your timely responses would be greatly appreciated.

Best,
AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.