acceptodds
Under review as a conference paper at ICLR 2027

BranchPO: Demonstration-Anchored Branching Preference Optimization for VLA Post-Training

Abstract

Pretrained vision–language–action policies can remain unreliable on multi-stage deployment tasks, while collecting expert corrections at learner-visited states is expensive and from-start autonomous rollouts can undersample later phases. We present BranchPO, a post-training framework that reuses one successful demonstration as a phase-indexed interaction scaffold. Semantic roots provide access to distinct task phases; autonomous sibling continuations are collected from each root and compared with cached expert segments. For flow-matching policies, BranchPO optimizes a root-balanced, reference-relative energy preference objective with explicit expert attraction, requiring neither a learned critic nor additional expert action labels. An idealized analysis characterizes root-local outcome contrast, heterogeneous phase corrections, and the comparison weights retained by the implemented loss. Across RoboCasa and real-robot manipulation tasks, BranchPO improves the SFT initialization on every evaluated task and generally outperforms AWR and trajectory-level DPO. Scaling from-start whole-trajectory preferences does not recover the benefit of rooted continuations, while a positive-only adaptation control explains only a small part of the gain. These results show that demonstration structure can guide both autonomous data collection and local credit assignment for effective VLA post-training.

Primary area:
applications to robotics, autonomy, planning
Keywords:
VLA Models, Robot Learning, Preference Optimization, Policy Post-Training, Demonstration-Guided Exploration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CHqgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes BranchPO, a post-training framework for flow-based vision-language-action policies. A single successful demonstration defines starting states for selected task phases, called roots, and provides expert reference segments. The initial policy collects multiple autonomous continuations from each root. Training compares these continuations with the corresponding expert segments using a reference-relative flow-energy preference objective and an additional expert-fitting term. Sampling is balanced across roots. Experiments on five simulated and three real-robot tasks report improvements over the initial policies, with comparisons against AWR and Trajectory-DPO. A separate experiment includes a one-demonstration SFT control.

Critical strengths
  1. The main strength is the data-collection pipeline. Starting rollouts from selected demonstration phases allows targeted collection at later stages that may rarely be reached through full-task rollouts. Multiple continuations from each root can provide more diverse local experience and better coverage of these task-relevant states.

  2. The reported results show substantial success-rate improvements over the initial policies in both simulation and real-robot experiments. Evaluation starts from the full-task initial state, providing evidence that the pipeline improves end-to-end task performance.

Critical weaknesses
  1. Training budgets and collection costs are not fully reported. The paper does not provide a complete per-task breakdown of training data and optimization budgets for the methods in Table 1. Table 2 includes useful action-frame-matched and scaling comparisons on PastryDisplay, but budget comparability remains unclear across the full benchmark, particularly for AWR.The practical collection cost is also unclear. Root annotation, state preparation, phase confirmation, and unsuccessful resets require operator time that action-frame counts do not capture. Since SFT-only is the policy before experiment-specific adaptation, the current comparisons do not establish whether the same human effort would be better spent collecting additional demonstrations and continuing SFT.

  2. The experiments do not sufficiently separate the benefit of local preference learning from that of targeted collection. The comparison with Trajectory-DPO changes both the states accessed during collection and the granularity of preference comparisons. For AWR, it is unclear whether the training data, phase coverage, sampling, and optimization budgets match those of BranchPO. The one-demo SFT control is useful, but covers only one task and is described as a proxy rather than a controlled loss ablation. The results support the overall pipeline, while leaving unclear how much additional benefit the preference objective provides when data coverage and training budgets are held fixed.

Questions for authors
  1. Could the authors provide a per-task, per-method budget table for Table 1, including expert demonstrations, autonomous trajectories or segments, action frames, and optimization updates? Please distinguish shared initialization data from additional post-training data and identify which budgets are matched.

  2. Does AWR use the same rooted autonomous data as BranchPO, and how are its return weights computed? On a representative task, could the authors compare BranchPO with AWR on the same autonomous continuations and positive-only fine-tuning on the same expert windows, while matching root weighting and optimization budgets?

  3. How much active operator time does collection require per task, including root annotation, preparation, confirmation, and unsuccessful resets? How does this compare with collecting additional demonstrations for SFT? A comparison under matched human effort would help assess practical efficiency beyond the paper’s fixed-demonstration setting.

4 comments

We sincerely thank the reviewer for the detailed questions and constructive feedback. We hope our response can address the concerns.

  1. Training Data and Optimization Budgets (W1 / Q1)

We apologize for lack of details. Below we provide the training data and optimization budgets for each task. We will include these details in the revised manuscript.

For comparison experiments in Table 1, all methods start from the same SFT checkpoint for a given task. In simulation, we use the RoboCasa π₀.₅ Human300 checkpoint at step 75,000. For each real-robot task, we fine-tune π₀.₅ base on 50 teleoperated demonstrations for 20,000 steps and use the resulting checkpoint.

On each simulation task, AWR trains its actor and critic on the 100 existing Human300 demonstrations and 64 autonomous rollouts. Trajectory-DPO pairs the same 64 rollouts with one demonstration sampled from Human300. BranchPO also uses one demonstration, but collects 64 local segments from 8 roots, with 8 rollouts per root. These segments are compared with the corresponding expert segments to form the preference pairs.

For each real-robot task, AWR trains its actor and critic on the 50 SFT demonstrations and 50 retained failure rollouts. Trajectory-DPO pairs the same 50 failure rollouts with one demonstration sampled from the SFT demonstration data. BranchPO also uses the same one demonstration and collects the rooted segments shown below. And no additional demonstrations were collected for BranchPO, only auto rollouts were collected.

The table reports the autonomous data used for post-training. Frame counts include only retained rollouts and exclude demonstrations and filtered-out rollouts. T-DPO denotes Trajectory-DPO. For BranchPO, the values in parentheses give the number of roots and segments per root.

\hline \text{Task} & \text{AWR/T-DPO traj.} & \text{AWR/T-DPO frames} & \text{BranchPO seg.} & \text{BranchPO frames} \\\\ \hline \text{OvenBroilFish} & 64 & 38{,}331 & 64\\;(8\times 8) & 8{,}768 \\\\ \text{PastryDisplay} & 64 & 21{,}376 & 64\\;(8\times 8) & 4{,}080 \\\\ \text{FoodCleanup} & 64 & 29{,}189 & 64\\;(8\times 8) & 6{,}728 \\\\ \text{RemoveBroiledFish} & 64 & 52{,}748 & 64\\;(8\times 8) & 9{,}488 \\\\ \text{LoadDishwasher} & 64 & 48{,}052 & 64\\;(8\times 8) & 7{,}800 \\\\ \text{Banana placement} & 50 & 18{,}260 & 20\\;(2\times 10) & 4{,}138 \\\\ \text{Tissue retrieval} & 50 & 33{,}017 & 40\\;(4\times 10) & 7{,}869 \\\\ \text{Drawer storage} & 50 & 59{,}176 & 70\\;(7\times 10) & 12{,}957 \\\\ \hline \end{array}$$ All three methods use 10,000 policy updates per task, and we report the final checkpoint. AWR additionally trains its critic for 5,000 steps at the beginning. Thus, the policy initialization and number of policy updates are matched, while the expert data and autonomous frame budgets differ across methods.
  1. Targeted Collection, Local Preferences, and AWR (W2 / Q2):

We agree with the reviewer that the Trajectory-DPO comparison changes both the rollout starting states and preference granularity, making it difficult to separate their contributions, as also noted by Reviewer 1n3A-W2.

Therefore for the simulated PastryDisplay task, we update several new comparisons to examine these choices more directly: one changes the collection strategy while retaining local preferences, and the other changes preference granularity while retaining rooted collection. We also test the expert-fitting term and, following the reviewer's suggestion, compare BranchPO with AWR and positive-only fine-tuning using the same rooted continuations and expert windows, respectively:

From-start local. We split trajectories collected from the task start into local segments and pair them with demonstration segments from matching phases. The policy and demonstration segments may start from different states, and phases not reached by the policy are ignored.

Rooted trajectory. We start rollouts from the same demonstration roots as BranchPO and continue to task termination or a predefined horizon. Each rollout is paired with the corresponding demonstration suffix using a trajectory-level preference loss.

BranchPO w/o regularization. We keep the rooted expert–policy preference pairs and the preference loss, but remove the expert-fitting regularization by setting λ to 0.

Positive-only. We fine-tune the policy on the same expert windows as BranchPO using only the expert-fitting objective, while matching root weighting and the number of policy updates.

AWR (rooted). We train an AWR policy on the same 64 rooted continuations as BranchPO, keeping the original AWR critic fixed. The critic was trained separately on existing demonstrations and task-start rollouts. We match the root weighting and the number of policy updates to BranchPO.

\hline \text{Method} & \text{Policy training data} & \text{Policy objective} & \text{Task success} & \text{Gained} & \text{Lost} \\\\ \hline \text{Base} & \text{--} & \text{--} & 19/50\\;(38\\%) & - & - \\\\ \text{Trajectory-DPO (K64)} & \text{64 from-start rollouts + expert demo} & \text{Trajectory-level preference} & 17/50\\;(34\\%) & 11 & 13 \\\\ \text{From-start local} & \text{From-start segments + expert windows} & \text{Local preference} & 11/50\\;(22\\%) & 4 & 12 \\\\ \text{Rooted trajectory} & \text{Rooted continuations + expert suffixes} & \text{Trajectory-level preference} & 27/50\\;(54\\%) & 14 & 6 \\\\ \text{BranchPO w/o regularization} & \text{Rooted segments + expert windows} & \text{Local preference }(\lambda = 0) & 28/50\\;(56\\%) & 19 & 10 \\\\ \text{Positive-only} & \text{Expert windows only} & \text{Expert-only SFT} & 22/50\\;(44\\%) & 15 & 12 \\\\ \text{AWR (rooted)} & \text{64 rooted segments} & \text{Advantage-weighted regression} & 20/50\\;(40\\%) & 14 & 13 \\\\ \text{BranchPO} & \text{Rooted segments + expert windows} & \text{Local preference }(\lambda = 0.1) & 48/50\\;(96\\%) & 29 & 0 \\\\ \hline \end{array}$$ All methods are evaluated on the same 50 cases. Gained and Lost count cases newly solved and lost relative to Base, respectively. On PastryDisplay, the from-start local and rooted-trajectory comparisons suggest that combining rooted collection with local preferences is beneficial. Neither AWR trained on the same rooted continuations nor positive-only fine-tuning on the same expert windows matches BranchPO. “Does AWR use the same rooted autonomous data as BranchPO, and how are its return weights computed? ” We apologize for lack of details. The AWR baseline in Table 1 uses autonomous trajectories collected from the task start, which cover only the phases reached by the policy; BranchPO instead collects continuations from intermediate roots. AWR also uses existing demonstrations. Each trajectory is labeled with a sparse binary outcome: 1 for success and 0 for failure. We train the critic on both existing demonstrations and autonomous rollouts, learning an action-value function $Q(o,a)$ and a state-value function $V(o)$. The value function is fitted using expectile regression with $\tau=0.75$. For each training sample, we compute the advantage $A(o,a)=Q(o,a)-V(o)$ and weight the policy loss based on $\exp(A/T_{\mathrm{AWR}})$, where $T_{\mathrm{AWR}}$ is a temperature.
  1. Human Effort and Additional Demonstrations (W1 / Q3)

We apologize for lack of details about the actual human effort. We agree that autonomous-frame counts only do not capture the effort needed to establish roots, as also noted by Reviewer 9koU-W2/Q2.

For the root selection on both sim and real-robot task, we manually mark seven roots at key stages in one existing demonstration (e.g. Drawer storage, open the drawer, put in the cube, close the drawer, …); this one-time annotation takes approximately 2–3 minutes for each task. Then we collect K continuations per root, requiring approximately roots \times segments resets (as shown in the data detail Table). For each reset, the robot automatically returns to its recorded root position using a safe force control, while the human operator manually restores the environment; restoring both takes approximately 5–10 seconds. Collection takes about 20 minutes under operator supervision, including all resets and autonomous execution.

We also agree that human effort could be used to collect additional demonstrations for SFT or to provide expert labels for policy-visited states like Dagger to improve policy performance efficiently. BranchPO explores whether a policy can refine itself from its own rollout exploration without additional expert action labels. We have not completed the matched-human-effort SFT comparison due to time reason. We may conduct the experiments under a matched human-effort budget in our future work.

Dear reviewer CHqg,

Thank you for your constructive feedback. We have carefully responsed to your concerns in our rebuttal.

As the discussion period is coming to an end, we would greatly appreciate your thoughts on our responses. We would be very happy to clarify any remaining concerns.

Thank you again for your time and consideration!

Reviewer_1n3AOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes BranchPO to addresses insufficient late-stage exploration and coarse credit assignment in multi-stage manipulation tasks. It collects autonomous continuations from semantic roots along a successful demonstration and trains the policy. Experiments report substantial improvements over SFT and trajectory-level preference baselines in simulation and real-robot manipulation.

Critical strengths
  1. The paper targets at a meaningful limitation of autonomous robot post-training. Using demonstration states to access later task phases avoids requiring the policy to succeed in all earlier stages before collecting useful experience.
  2. The empirical improvements are substantial. BranchPO consistently improves across the evaluated tasks, with particularly strong gains in real-robot manipulation and a useful comparison against increased whole-trajectory preference data.
Critical weaknesses
  1. The novelty relative to closely related prior work needs further clarification. Demonstration-based intermediate-state training and flow-matching preference optimization have both been explored previously, and it is not sufficiently clear what new methodological insight arises from their combination.
  2. The experimental evidence does not sufficiently isolate the contributions of the proposed components. The main comparisons change both the rollout starting-state distribution and the preference granularity, making it difficult to attribute the substantial gains to either component; the evaluation is also limited to a single VLA architecture, leaving generality across architectures and model scales uncertain. I would be open to revising my rating if the authors can clarify these concerns and provide convincing supporting evidence.
Questions for authors

See above

4 comments

We sincerely thank the reviewer for positive assessment of our work and insightful comments. We hope our response below can address the reviewer's concerns.

  1. Methodological Novelty and Related Work (W1)

We appreciate the reviewer’s recognition that roots collection provide access to laters task phases. And we also agree with the reviewer that demonstration-guided intermediate-state collection [1–3] and flow-matching preference optimization [4, 5] have precedents on diverse tasks. BranchPO goes beyond just applying these ideas side by side. Our insight is to connect phase-local preference learning to an outcome-conditioned decomposition of the reference policy. In our idealized analysis, the success- and failure-conditioned distributions correspond to the same reference policy at the same root. Proposition 1 relates their policy-score contrast to a local improvement direction. This motivates using a demonstration to select phase roots and collecting multiple autonomous continuations from each root.

This view also suggests a simple data-collection procedure: once a root is established, we compare its autonomous continuations with the corresponding recorded demonstration segment. This provides a preference signal without additional expert corrections, dense reward annotations, or a learned critic. On real robots, an operator assists with root setup and confirmation.

References:

[1] C. Resnick, R. Raileanu, S. Kapoor, A. Peysakhovich, K. Cho, and J. Bruna. Backplay: Man muss immer umkehren. arXiv:1807.06919, 2018.

[2] N. Di Palo and E. Johns. Learning Multi-Stage Tasks with One Demonstration via Self-Replay. Proceedings of the 5th Conference on Robot Learning, PMLR 164:1180–1189, 2022.

[3] G. Papagiannis and E. Johns. MILES: Making Imitation Learning Easy with Self-Supervision. Proceedings of the 8th Conference on Robot Learning, PMLR 270:810–829, 2025.

[4] J. Liu et al. Improving Video Generation with Human Feedback. arXiv:2501.13918, 2025.

[5] Y. Wu, H. Zhang, J. Tan, X. Wang, and Z. Zhang. FlowPRO: Reward-Free Reinforced Fine-Tuning of Flow-Matching VLAs via Proximalized Preference Optimization. arXiv:2606.05468, 2026.

  1. Contributions of Rooted Collection and Local Preferences (W2)

We agree with the reviewer that this comparison alone does not reveal how much each factor contributes to the observed performance difference. Reviewer comments CHqg-W2 also raised this. In our original data construction, the rollout starting state and available preference comparison are coupled: BranchPO collects autonomous segments over local windows starting from intermediate demonstration roots. These segments do not contain the complete autonomous trajectories from the task start needed to directly construct full-task trajectory preference pairs. We therefore used separately collected from-start trajectories for the Trajectory-DPO comparison.

To examine the individual design choices more directly, we conducted the following new comparisons on PastryDisplay task in simulation:

From-start local: We split trajectories collected from the task start into local segments and paired them with demonstration segments from matching phases. Unlike BranchPO, each policy segment starts from a state reached by the policy, which may differ from the demonstration root even when the phase matches. This retains the local preference objective while changing the data-collection strategy; phases not reached by the policy provide no local pairs.

Rooted trajectory: Starting from the same eight demonstration roots as BranchPO, we collected autonomous continuations to task termination or a predefined horizon. Each continuation was compared with the corresponding expert suffix using a trajectory-level preference loss. This retains rooted collection while changing the comparison scope from a local window to a full root-to-horizon continuation.

BranchPO w/o regularization: We used the same 64 rooted expert–policy segment pairs as BranchPO and retained its preference loss, but removed the expert-fitting regularization term by setting λ to 0. The expert segment remained the preferred side of each pair. This examines the role of the regularization term.

For reference, Base is the pretrained policy, Trajectory-DPO (K64) compares the demonstration with 64 full autonomous trajectories collected from the task start, and BranchPO uses 64 rooted segment pairs from eight roots with eight autonomous continuations per root.

The table also includes positive-only fine-tuning on the same expert windows and rooted AWR on the same autonomous continuations based on Reviewer CHqg’s suggestion Q2. All methods are evaluated on the same 50 cases. Gained and Lost are measured relative to Base.

\hline \text{Method} & \text{Policy training data} & \text{Policy objective} & \text{Task success} & \text{Gained} & \text{Lost} \\\\ \hline \text{Base} & \text{--} & \text{--} & 19/50\\;(38\\%) & - & - \\\\ \text{Trajectory-DPO (K64)} & \text{64 from-start rollouts + expert demo} & \text{Trajectory-level preference} & 17/50\\;(34\\%) & 11 & 13 \\\\ \text{From-start local} & \text{From-start segments + expert windows} & \text{Local preference} & 11/50\\;(22\\%) & 4 & 12 \\\\ \text{Rooted trajectory} & \text{Rooted continuations + expert suffixes} & \text{Trajectory-level preference} & 27/50\\;(54\\%) & 14 & 6 \\\\ \text{BranchPO w/o regularization} & \text{Rooted segments + expert windows} & \text{Local preference }(\lambda = 0) & 28/50\\;(56\\%) & 19 & 10 \\\\ \text{Positive-only} & \text{Expert windows only} & \text{Expert-only SFT} & 22/50\\;(44\\%) & 15 & 12 \\\\ \text{AWR (rooted)} & \text{64 rooted segments} & \text{Advantage-weighted regression} & 20/50\\;(40\\%) & 14 & 13 \\\\ \text{BranchPO} & \text{Rooted segments + expert windows} & \text{Local preference }(\lambda = 0.1) & 48/50\\;(96\\%) & 29 & 0 \\\\ \hline \end{array}$$ The results show that proposed BranchPO performs better than other ablations. It suggest that the three design choices are useful in this setting.

Multi-architecture Test:

To assess a second VLA architecture, we also evaluated the BranchPO method on GR00T N1.5 in simulation. π₀.₅ uses a PaliGemma vision-language backbone and a Gemma-based action expert, whereas GR00T N1.5 combines an Eagle vision-language backbone with a diffusion-transformer action head; both are flow-based policies.

\hline \text{Policy} & \text{OvenBroilFish} & \text{PastryDisplay} & \text{FoodCleanup} & \text{RemoveBroiledFish} & \text{LoadDishwasher} \\\\ \hline \pi_{0.5}\\;\text{Base} & 2\\% & 38\\% & 28\\% & 6\\% & 24\\% \\\\ \boldsymbol{\pi_{0.5}}\\;\mathbf{BranchPO} & \mathbf{72\\%} & \mathbf{96\\%} & \mathbf{64\\%} & \mathbf{28\\%} & \mathbf{42\\%} \\\\ \text{GR00T N1.5 Base} & 16\\% & 76\\% & 44\\% & 32\\% & 4\\% \\\\ \mathbf{GR00T}\\;\mathbf{N1.5}\\;\mathbf{BranchPO} & \mathbf{52\\%} & \mathbf{92\\%} & \mathbf{90\\%} & \mathbf{34\\%} & \mathbf{10\\%} \\\\ \hline \end{array}$$ Across all five simulation tasks, BranchPO improves upon the corresponding base policy for both π₀.₅ and GR00T N1.5. These results suggest that BranchPO can benefit both VLA architectures, although the gains vary across tasks.

Dear reviewer 1n3A,

Thank you for your constructive feedback. We have carefully responsed to your concerns in our rebuttal.

As the discussion period is coming to an end, we would greatly appreciate your thoughts on our responses. We would be very happy to clarify any remaining concerns.

Thank you again for your time and consideration!

Reviewer_9koUOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

BranchPO uses a successful demonstration to define phase-specific starting states and expert reference segments for VLA post-training. Autonomous continuations from each starting state are compared with the corresponding expert segment using a reference-relative flow-energy preference loss and an explicit expert-attraction term. The method is evaluated on five RoboCasa tasks and three real-robot tasks, with additional comparisons against larger whole-trajectory datasets and positive-only adaptation.

Critical strengths
  1. The use of demonstration states to reach later task phases addresses a concrete limitation of from-start collection. The method is straightforward, and the paper carefully distinguishes local segment preferences from expert corrections at every learner-visited state. It also states the limitations of the flow-energy surrogate and idealized analysis clearly.

  2. The empirical improvements are substantial. BranchPO improves over SFT-only on all eight evaluated tasks and beats AWR on seven. On PastryDisplay, it reaches 48/50 successes while retaining all 19 successes of the initial policy. The whole-trajectory data-scaling experiment and one-demo SFT control provide useful evidence that merely collecting more full episodes or repeating the demonstration is insufficient under the tested recipes.

Critical weaknesses
  1. Appendix C reports several shared optimization settings, but I could not find the number of training updates or checkpoint-selection rule for the Table 2 variants. Trajectory-DPO deteriorates from 38% to 0% as its dataset grows, so the training schedule matters to interpreting this comparison. Clarifying how the schedule and tuning were handled would help distinguish the effect of comparison scope from that of optimization.

  2. The autonomous-frame counts do not include the effort required to establish roots. The paper explains saved-state restoration in simulation and operator-assisted resets on the real robot, but does not quantify the latter. This is a limitation of the practical collection-cost comparison; it does not undermine the measured success-rate gains.

Questions for authors
  1. What training-update budgets and checkpoint-selection rules were used for the Table 2 variants? Were these settings shared across the scaling runs, and was any validation-based tuning performed?

  2. For a representative real-robot task, can you estimate the operator time and reset attempts needed to collect the rooted dataset, in addition to autonomous execution? An estimate from the existing collection would be sufficient.

3 comments

We sincerely thank the reviewer for the positive assessment and constructive feedback. We hope our response can address the concerns.

  1. Training Budgets and Checkpoint Selection for Table 2 (W1 / Q1)

We apologize for the lack of details about the training in Table 2. All methods start from the same RoboCasa π₀.₅ Human300 checkpoint. BranchPO and each Trajectory-DPO variant are trained for 10,000 policy updates, and we report the checkpoint at final step for each run. The Trajectory-DPO variants differ in the number of autonomous trajectories collected from the task start, while BranchPO uses 64 rooted segments. We will clarify these details in the revised manuscript.

To examine the effect of training duration, we additionally evaluated Trajectory-DPO (K64) at different checkpoints on PastryDisplay:

\hline \text{Method} & \text{5,000 steps} & \text{10,000 steps} & \text{20,000 steps} & \text{30,000 steps} \\\\ \hline \text{Trajectory-DPO (K64)} & 18/50\\;(36\\%) & 17/50\\;(34\\%) & 13/50\\;(26\\%) & 15/50\\;(30\\%) \\\\ \hline \end{array}$$ The 5,000-step result is close to the reported 10,000-step result, while training to 20,000 or 30,000 steps does not improve upon it or close the gap to BranchPO. The results suggest that simply training for longer does not improve performance in this setting. Another possible factor is that preference optimization may shift the policy away from behaviors that were already successful: the expert-fitting term anchors the demonstration, but does not directly preserve the base policy on other states. This is consistent with the loss of SFT-only successes observed in Table 2. 2. Human Effort for Rooted Data Collection (W2 / Q2) We agree that the reported autonomous-frame counts only do not capture the human effort required for post-training. We provide details of our data collection procedure and experimental setup as following and also revised the manuscript. For each task, there will be only one-time human annotation. We identify roots along one existing successful demonstration based on the task’s stages. For example, in the Drawer Storage task, we select roots around key stages such as opening the drawer, grasping and placing the cbue inside, and closing the drawer, which takes approximately 2-3 minutes. In this task, we collect 10 autonomous rollouts from each root, for 70 rollouts in total. Each rollout requires a reset, resulting in approximately 70 resets. The robot automatically returns to its recorded root position using a safe force-controll, while the human operator manually restores the environment. Restoring both takes approximately 5–10 seconds per reset. Based on our recollection, collecting the full dataset takes about 20 minutes under operator supervision, including both resets and autonomous execution. Manual intervention and autonomous execution were not timed separately. We will include these details in the revision.

The response answers most of my questions, and the extra detail gives me a clearer picture of the work. Taking that into account, I would still rate the paper a 3.

Dear Reviewer 9koU,

Thank you for your follow-up and evaluation. We appreciate your feedback and time, which have helped us a lot to improve the manuscript.

Best regards, Authors

Comments on the paper

1 comment

Dear reviewers,

As the reminded earlier, please check the author's response and provide your feedback before the end of the discuss phase.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.