- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes BranchPO, a post-training framework for flow-based vision-language-action policies. A single successful demonstration defines starting states for selected task phases, called roots, and provides expert reference segments. The initial policy collects multiple autonomous continuations from each root. Training compares these continuations with the corresponding expert segments using a reference-relative flow-energy preference objective and an additional expert-fitting term. Sampling is balanced across roots. Experiments on five simulated and three real-robot tasks report improvements over the initial policies, with comparisons against AWR and Trajectory-DPO. A separate experiment includes a one-demonstration SFT control.
Critical strengths
-
The main strength is the data-collection pipeline. Starting rollouts from selected demonstration phases allows targeted collection at later stages that may rarely be reached through full-task rollouts. Multiple continuations from each root can provide more diverse local experience and better coverage of these task-relevant states.
-
The reported results show substantial success-rate improvements over the initial policies in both simulation and real-robot experiments. Evaluation starts from the full-task initial state, providing evidence that the pipeline improves end-to-end task performance.
Critical weaknesses
-
Training budgets and collection costs are not fully reported. The paper does not provide a complete per-task breakdown of training data and optimization budgets for the methods in Table 1. Table 2 includes useful action-frame-matched and scaling comparisons on PastryDisplay, but budget comparability remains unclear across the full benchmark, particularly for AWR.The practical collection cost is also unclear. Root annotation, state preparation, phase confirmation, and unsuccessful resets require operator time that action-frame counts do not capture. Since SFT-only is the policy before experiment-specific adaptation, the current comparisons do not establish whether the same human effort would be better spent collecting additional demonstrations and continuing SFT.
-
The experiments do not sufficiently separate the benefit of local preference learning from that of targeted collection. The comparison with Trajectory-DPO changes both the states accessed during collection and the granularity of preference comparisons. For AWR, it is unclear whether the training data, phase coverage, sampling, and optimization budgets match those of BranchPO. The one-demo SFT control is useful, but covers only one task and is described as a proxy rather than a controlled loss ablation. The results support the overall pipeline, while leaving unclear how much additional benefit the preference objective provides when data coverage and training budgets are held fixed.
Questions for authors
-
Could the authors provide a per-task, per-method budget table for Table 1, including expert demonstrations, autonomous trajectories or segments, action frames, and optimization updates? Please distinguish shared initialization data from additional post-training data and identify which budgets are matched.
-
Does AWR use the same rooted autonomous data as BranchPO, and how are its return weights computed? On a representative task, could the authors compare BranchPO with AWR on the same autonomous continuations and positive-only fine-tuning on the same expert windows, while matching root weighting and optimization budgets?
-
How much active operator time does collection require per task, including root annotation, preparation, confirmation, and unsuccessful resets? How does this compare with collecting additional demonstrations for SFT? A comparison under matched human effort would help assess practical efficiency beyond the paper’s fixed-demonstration setting.
4 comments
- Targeted Collection, Local Preferences, and AWR (W2 / Q2):
We agree with the reviewer that the Trajectory-DPO comparison changes both the rollout starting states and preference granularity, making it difficult to separate their contributions, as also noted by Reviewer 1n3A-W2.
Therefore for the simulated PastryDisplay task, we update several new comparisons to examine these choices more directly: one changes the collection strategy while retaining local preferences, and the other changes preference granularity while retaining rooted collection. We also test the expert-fitting term and, following the reviewer's suggestion, compare BranchPO with AWR and positive-only fine-tuning using the same rooted continuations and expert windows, respectively:
From-start local. We split trajectories collected from the task start into local segments and pair them with demonstration segments from matching phases. The policy and demonstration segments may start from different states, and phases not reached by the policy are ignored.
Rooted trajectory. We start rollouts from the same demonstration roots as BranchPO and continue to task termination or a predefined horizon. Each rollout is paired with the corresponding demonstration suffix using a trajectory-level preference loss.
BranchPO w/o regularization. We keep the rooted expert–policy preference pairs and the preference loss, but remove the expert-fitting regularization by setting λ to 0.
Positive-only. We fine-tune the policy on the same expert windows as BranchPO using only the expert-fitting objective, while matching root weighting and the number of policy updates.
AWR (rooted). We train an AWR policy on the same 64 rooted continuations as BranchPO, keeping the original AWR critic fixed. The critic was trained separately on existing demonstrations and task-start rollouts. We match the root weighting and the number of policy updates to BranchPO.
\hline \text{Method} & \text{Policy training data} & \text{Policy objective} & \text{Task success} & \text{Gained} & \text{Lost} \\\\ \hline \text{Base} & \text{--} & \text{--} & 19/50\\;(38\\%) & - & - \\\\ \text{Trajectory-DPO (K64)} & \text{64 from-start rollouts + expert demo} & \text{Trajectory-level preference} & 17/50\\;(34\\%) & 11 & 13 \\\\ \text{From-start local} & \text{From-start segments + expert windows} & \text{Local preference} & 11/50\\;(22\\%) & 4 & 12 \\\\ \text{Rooted trajectory} & \text{Rooted continuations + expert suffixes} & \text{Trajectory-level preference} & 27/50\\;(54\\%) & 14 & 6 \\\\ \text{BranchPO w/o regularization} & \text{Rooted segments + expert windows} & \text{Local preference }(\lambda = 0) & 28/50\\;(56\\%) & 19 & 10 \\\\ \text{Positive-only} & \text{Expert windows only} & \text{Expert-only SFT} & 22/50\\;(44\\%) & 15 & 12 \\\\ \text{AWR (rooted)} & \text{64 rooted segments} & \text{Advantage-weighted regression} & 20/50\\;(40\\%) & 14 & 13 \\\\ \text{BranchPO} & \text{Rooted segments + expert windows} & \text{Local preference }(\lambda = 0.1) & 48/50\\;(96\\%) & 29 & 0 \\\\ \hline \end{array}$$ All methods are evaluated on the same 50 cases. Gained and Lost count cases newly solved and lost relative to Base, respectively. On PastryDisplay, the from-start local and rooted-trajectory comparisons suggest that combining rooted collection with local preferences is beneficial. Neither AWR trained on the same rooted continuations nor positive-only fine-tuning on the same expert windows matches BranchPO. “Does AWR use the same rooted autonomous data as BranchPO, and how are its return weights computed? ” We apologize for lack of details. The AWR baseline in Table 1 uses autonomous trajectories collected from the task start, which cover only the phases reached by the policy; BranchPO instead collects continuations from intermediate roots. AWR also uses existing demonstrations. Each trajectory is labeled with a sparse binary outcome: 1 for success and 0 for failure. We train the critic on both existing demonstrations and autonomous rollouts, learning an action-value function $Q(o,a)$ and a state-value function $V(o)$. The value function is fitted using expectile regression with $\tau=0.75$. For each training sample, we compute the advantage $A(o,a)=Q(o,a)-V(o)$ and weight the policy loss based on $\exp(A/T_{\mathrm{AWR}})$, where $T_{\mathrm{AWR}}$ is a temperature.- Human Effort and Additional Demonstrations (W1 / Q3)
We apologize for lack of details about the actual human effort. We agree that autonomous-frame counts only do not capture the effort needed to establish roots, as also noted by Reviewer 9koU-W2/Q2.
For the root selection on both sim and real-robot task, we manually mark seven roots at key stages in one existing demonstration (e.g. Drawer storage, open the drawer, put in the cube, close the drawer, …); this one-time annotation takes approximately 2–3 minutes for each task. Then we collect K continuations per root, requiring approximately roots \times segments resets (as shown in the data detail Table). For each reset, the robot automatically returns to its recorded root position using a safe force control, while the human operator manually restores the environment; restoring both takes approximately 5–10 seconds. Collection takes about 20 minutes under operator supervision, including all resets and autonomous execution.
We also agree that human effort could be used to collect additional demonstrations for SFT or to provide expert labels for policy-visited states like Dagger to improve policy performance efficiently. BranchPO explores whether a policy can refine itself from its own rollout exploration without additional expert action labels. We have not completed the matched-human-effort SFT comparison due to time reason. We may conduct the experiments under a matched human-effort budget in our future work.
Dear reviewer CHqg,
Thank you for your constructive feedback. We have carefully responsed to your concerns in our rebuttal.
As the discussion period is coming to an end, we would greatly appreciate your thoughts on our responses. We would be very happy to clarify any remaining concerns.
Thank you again for your time and consideration!
We sincerely thank the reviewer for the detailed questions and constructive feedback. We hope our response can address the concerns.
We apologize for lack of details. Below we provide the training data and optimization budgets for each task. We will include these details in the revised manuscript.
For comparison experiments in Table 1, all methods start from the same SFT checkpoint for a given task. In simulation, we use the RoboCasa π₀.₅ Human300 checkpoint at step 75,000. For each real-robot task, we fine-tune π₀.₅ base on 50 teleoperated demonstrations for 20,000 steps and use the resulting checkpoint.
On each simulation task, AWR trains its actor and critic on the 100 existing Human300 demonstrations and 64 autonomous rollouts. Trajectory-DPO pairs the same 64 rollouts with one demonstration sampled from Human300. BranchPO also uses one demonstration, but collects 64 local segments from 8 roots, with 8 rollouts per root. These segments are compared with the corresponding expert segments to form the preference pairs.
For each real-robot task, AWR trains its actor and critic on the 50 SFT demonstrations and 50 retained failure rollouts. Trajectory-DPO pairs the same 50 failure rollouts with one demonstration sampled from the SFT demonstration data. BranchPO also uses the same one demonstration and collects the rooted segments shown below. And no additional demonstrations were collected for BranchPO, only auto rollouts were collected.
The table reports the autonomous data used for post-training. Frame counts include only retained rollouts and exclude demonstrations and filtered-out rollouts. T-DPO denotes Trajectory-DPO. For BranchPO, the values in parentheses give the number of roots and segments per root.
\hline \text{Task} & \text{AWR/T-DPO traj.} & \text{AWR/T-DPO frames} & \text{BranchPO seg.} & \text{BranchPO frames} \\\\ \hline \text{OvenBroilFish} & 64 & 38{,}331 & 64\\;(8\times 8) & 8{,}768 \\\\ \text{PastryDisplay} & 64 & 21{,}376 & 64\\;(8\times 8) & 4{,}080 \\\\ \text{FoodCleanup} & 64 & 29{,}189 & 64\\;(8\times 8) & 6{,}728 \\\\ \text{RemoveBroiledFish} & 64 & 52{,}748 & 64\\;(8\times 8) & 9{,}488 \\\\ \text{LoadDishwasher} & 64 & 48{,}052 & 64\\;(8\times 8) & 7{,}800 \\\\ \text{Banana placement} & 50 & 18{,}260 & 20\\;(2\times 10) & 4{,}138 \\\\ \text{Tissue retrieval} & 50 & 33{,}017 & 40\\;(4\times 10) & 7{,}869 \\\\ \text{Drawer storage} & 50 & 59{,}176 & 70\\;(7\times 10) & 12{,}957 \\\\ \hline \end{array}$$ All three methods use 10,000 policy updates per task, and we report the final checkpoint. AWR additionally trains its critic for 5,000 steps at the beginning. Thus, the policy initialization and number of policy updates are matched, while the expert data and autonomous frame budgets differ across methods.