- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The manuscript tackles a limitation of robot foundation models. Standard Behavior Cloning treats all successful demonstrations as equally good, so the resulting policies average over diverse execution qualities such as speed and smoothness and offer no inference-time steerability.
To address this, the authors propose Quality-Controllable Policy. It measures trajectory quality along the desired axes using rule-based scoring functions, discretizes the scores into quantile bins such as 4 bins, and conditions a flow-matching policy on these discrete quality tokens, either via learned embeddings or prompt injection for VLAs such as π0.5. To push execution quality beyond the limits of the initial human demonstrations, the authors further introduce an Iterative Hindsight Relabeling loop. Rollouts are sampled requesting the highest quality bin, successful rollouts are retroactively relabeled by their actual achieved physical metric, added to the dataset, and the policy is retrained on updated quantile boundaries.
The method is evaluated on standard simulation benchmarks including Robomimic and RoboEval, and on physical Franka arm tasks covering wine serving for stability and conveyor picking for speed. It shows consistent joint improvements in task success and execution quality, outperforming quality-filtered BC and offline and online RL baselines.
Critical strengths
Rather than resorting to complex and brittle online reinforcement learning, such as tuning multi-objective reward trade-offs with critics, QCP frames quality control as a supervised conditioning problem. The iterative hindsight relabeling loop elegantly expands the policy's quality boundary beyond human demonstration limits by leveraging policy stochasticity and adaptive quantiles. The paper provides exhaustive empirical evidence, including controlling up to six attributes simultaneously, clear comparisons against competitive RL baselines such as DSRL, EXPO and SCIQL, and crucial ablation studies in Fig. 8 and Fig. 9. The finding that discrete quantile bins outperform continuous targets because of density support in the training distribution is an important takeaway for the robot learning community. Demonstrating QCP on a physical Franka robot via $\pi$0.5 fine-tuning for two distinct physical requirements, namely reducing tilt in wine serving by 19.7 percent and boosting conveyor throughput by 23 percent beyond human teleoperator speeds, validates the practical utility in real-world deployment.
Critical weaknesses
The method's premise assumes access to an analytical scoring function mτ, such as jerk, execution steps or tilt angle. While suitable for kinematic and geometric attributes, it cannot trivially scale to perceptual, semantic or non-analytical qualities such as naturalness, safety near humans or delicateness of contact without introducing human-in-the-loop annotations or learned reward models. In Algorithm 1, Line 9, the policy πθk+1 is retrained from scratch, or reset to the pretrained checkpoint, at each iteration to avoid stale label-to-behavior mappings. As iterations K progress and accumulated data grows, the cumulative training cost becomes substantial, which may hinder rapid online deployment or continuous lifelong learning. In Footnote 3, the authors mention that incorporating failed rollouts as low-quality data degraded performance. In standard Hindsight Experience Replay and offline RL, failed trajectories often provide valuable negative signals or boundary information. Completely discarding them represents a missed sample-efficiency opportunity that warrants further theoretical and empirical clarification.
Questions for authors
Could the authors elaborate on whether warm-starting or partial fine-tuning, for example only updating the quality embedding layer and final projection layers, was attempted instead of full reinitialization from scratch. What specifically causes policy collapse if the previous checkpoint is continually trained on the updated quantile boundaries. How does the policy behave if an out-of-distribution quality request is provided, for example extrapolating beyond q equals B minus 1, or requesting contradictory multi-attribute combinations such as maximum speed combined with near-zero jerk. Does the policy fail gracefully or exhibit erratic kinematics. Could you provide more intuition on why including failed rollouts as q equals 0 degraded performance. Was it due to the policy learning to replicate task-failure modes when lower quality bins were conditioned, or did it degrade the feature representation overall.
5 comments
Author Response to Reviewer Cb5T (2/3)
3. Learning from failed rollouts
Thank you for this thoughtful question. We agree that failed trajectories can carry useful information. We would like to share our intuition on why simply including them as q = 0 degraded performance in QCP.
Conflating task outcome with execution quality. Assigning every failure to q = 0 mixes two different notions: whether the task succeeded, and how well the motion was executed. For example, a rollout may fail the task while moving very fast. In terms of speed, this trajectory should belong to a high-quality bin, but labeling it q = 0 tells the model that it represents the lowest quality. Such inconsistencies may make it harder for the model to learn what each quality level represents, which could weaken the conditioning signal. Excluding failures may therefore provide cleaner quality supervision.
Negative transfer from imitation. As the reviewer suggests, the policy may also learn to replicate failure modes. Because QCP is built on behavioral cloning, actions from failed rollouts remain imitation targets even under the q = 0 label. The label gives no explicit signal to avoid those actions; it may only increase their likelihood under a particular condition. This differs from HER or offline RL, where failed trajectories are used through rewards or value estimates that can explicitly assign them low credit.
Experiments. Consistent with this intuition, on Square-MH, including failed rollouts as q = 0 reduced task success from 81.7% to 75.0% in the speed experiment and from 63.3% to 61.3% in the smoothness experiment, compared with training on successful rollouts alone. To address the first issue, we also tested conditioning separately on success and quality, which separates task outcome from execution quality. This did not consistently improve task success over training on successful rollouts alone. This suggests that the second issue, imitating failed actions under BC, remains even when the labels are disentangled.
We therefore view the effective use of failed rollouts as a promising direction for future work, for example through outcome-based weighting or value-based credit assignment that can explicitly down-weight failed actions instead of imitating them. We discuss these results in Section 6 (Discussion) of the revised draft.
Author Response to Reviewer Cb5T (3/3)
4. Out-of-distribution quality requests
Thank you for raising this interesting question. In QCP’s discrete formulation, any quality score above the highest threshold maps to the top bin (), so higher scores all correspond to the same quality level. To explore how the policy responds to requests beyond the training range, we ran an experiment with continuous quality conditioning.
On Robomimic Square (MH), we evaluated the continuous variant using three training seeds and 300 episodes per request.
| Metric | q=1.2 | q=1.1 | q=1 | q=0 | q=-0.1 | q=-0.2 |
| Speed SR (%) | 0.0 | 0.0 | 11.3 ± 1.4 | 18.3 ± 2.0 | 2.7 ± 0.0 | 0.0 |
| Speed execution steps ↓ | – | – | 208.6 ± 8.1 | 298.5 ± 10.2 | 307.0 ± 8.7 | – |
| Smoothness SR (%) | 0.0 | 0.0 | 49.0 ± 3.5 | 20.7 ± 1.4 | 0.0 | 0.0 |
| Smoothness action jerk ↓ | – | – | 0.02754 ± 0.00392 | 0.47062 ± 0.02103 | – | – |
Extrapolation on either side substantially reduced SR and did not reliably produce the requested behavior. These results support QCP’s iterative approach, which can reliably target progressively higher-quality behavior.
For multiple attributes, individually valid requests can form an unsupported or infeasible combination. We have not explicitly addressed such requests. Future work could use attribute weights to select target combinations supported by the data, as in multi-objective Decision Transformers [3].
References
[1] Lee et al. PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training. ICML 2021.
[2] Wang et al. RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback. ICML 2024.
[3] Zhu et al. Scaling Pareto-Efficient Decision Making via Offline Multi-Objective RL. ICLR 2023.
Author Response to Reviewer Cb5T (1/3)
We deeply appreciate your insightful comments and efforts in reviewing our manuscript. We respond to each of your comments one-by-one in what follows.
1. Extending QCP Beyond Analytical Metrics
We thank the reviewer for raising this important point. While our experiments use analytical metrics such as jerk and execution steps, we expect that QCP could also be used with non-analytical metrics. The framework only requires a scalar quality score for each trajectory, so any scoring function can be used, including learned ones. For qualities that are difficult to measure analytically, such as naturalness, safety near humans, or delicacy of contact, the score can come from a reward model learned from human preferences, as in PEBBLE [1], or from VLM feedback, as in RL-VLM-F [2].
We have not yet evaluated this extension, but we believe it makes QCP applicable to a much broader range of qualities. We added a brief discussion of using learned reward models to score trajectories in Section 6 (Discussion).
2. Warm-starting and training cost
Following your valuable suggestion, we compared short warm-start updates with longer runs from a fresh initialization. Warm starts may be a practical option when training cost is a concern, while fresh initialization may help push the performance frontier for specific quality metrics (e.g., execution speed).
On Robomimic Square (MH), we compared fresh initialization at each iteration (100K updates) with warm-starting from the previous checkpoint (5K, 10K, or 100K updates). For each setting, we trained separate policies for speed and smoothness over six iterations with three seeds. Appendix J of the revised draft now includes the results, checkpoint-wise learning curves, and diagnostics.
At a matched training budget, the warm-started policy approached the original QCP in smoothness but failed to match its execution speed. Future work could explore more efficient adaptation methods for QCP.