- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper learns from expert data plus suboptimal data. The authors argue that scalar weights cannot express “where the error is and how to fix it,” so they generate structured language labels for each (s, a) pair, including task progress <T>, action quality <A>, and movement guidance <M>.
The labels come from a template generator g, based on task features: object distance, gripper state, and displacement relative to a reference target. g is distilled into a differentiable LLM-Captioner (SmolLM2-135M plus an MLP projection). During training, an LC-loss is added: the cross-entropy of the “expert label” under the policy’s action, minus the cross-entropy under the expert action, with a hinge applied. It is added to both BC and Diffusion Policy (DP; DP uses one-step reconstruction with reweighting). Theoretically, under assumptions such as language sufficiency, the LC loss upper-bounds performance degradation.
Results: Averaged over 7 tasks, the method improves BC by 8.2% and DP by 4.3%.
Critical strengths
The idea is novel: it uses language as a differentiable critic signal rather than compressing feedback into a scalar.
The control design is thorough: scalar, classifier, shuffled, verbose, opaque/named token, captioner trained from scratch, VLM-generated labels, and human labels. The authors also honestly report diagnostics showing that their theoretical assumptions are violated. The derivation of the DP reweighting is clear.
Critical weaknesses
-
The role of “language” is not confirmed. The real source of information is hand-written heuristics that use privileged state. µg uses object distance, displacement relative to a reference target, “dist_delta,” and so on, so <M> is essentially directional action supervision, equivalent to a hand-crafted dense reward or correction signal. The baselines do not have access to this information, so the comparison is unfair. The authors’ own ablations also suggest that most of the gain comes from the features rather than the language.
-
The improvements are small, the variance is large, and the cost is high. The average is 71.3 vs. 65.4, and many tasks have overlapping standard deviations. The average gain for LC-DP comes mainly from RELOCATE (DP’s std there is 24.2). Training time also increases substantially over BC.
Questions for authors
Please refer to the weaknesses above.
If these are addressed, I would be happy to raise my score.
8 comments
Q2. The reliability of performance gain and training cost
We thank the reviewer for raising this concern. We added further statistical analysis to see how reliable the improvements are. We also explain why extra training can be worthwhile when expert data are limited or imperfect.
Table R2. Improvements and their uncertainty. Brackets indicate 95% bootstrap confidence intervals. Mean and IQM gains are absolute success-rate differences in percentage points; probability of improvement is reported as a percentage.
| Comparison | Mean gain [95% CI] | IQM gain [95% CI] | Probability of improvement [95% CI] |
|---|---|---|---|
| LC-BC vs. BC | 6.23 [3.49, 8.97] | 5.58 [1.26, 10.00] | 71.14 [59.43, 82.57] |
| LC-BC vs. ILID | 5.89 [3.14, 8.63] | 3.58 [-0.42, 7.37] | 67.14 [57.14, 76.86] |
| LC-DP vs. DP | 2.97 [0.46, 6.34] | 3.05 [0.21, 6.42] | 64.29 [51.71, 76.29] |
-
How reliable are the improvements? Following Agarwal et al. [1], we analyze all training seeds across 7 tasks. We resample training runs independently within each task and method 50000 times to estimate 95% confidence intervals for (1) the average success rate gain, (2) IQM (interquartile mean) gain, and (3) probability of improvement. IQM reduces the influence of unusually high or low scores. Probability of improvement measures how often our method beats the baseline.
As shown in Table R2, LC-BC improves average success over BC by 6.23% and ILID by 5.89%, with both 95% confidence intervals above zero. Its IQM gain over BC is also positive at 5.58% [1.26, 10.00]. Against ILID, the IQM gain is 3.58%, although its interval includes zero, so this comparison remains less certain.
For LC-DP, the IQM gain of 3.05% [0.21, 6.42] is slightly larger than the mean gain of 2.97%. Thus, the improvement remains when unusually high or low scores have less influence on the comparison. Removing Relocate entirely also leaves a positive average gain of 1.73% [0.27, 3.27] across the remaining six tasks, showing that the overall benefit extends beyond Relocate.
The probability-of-improvement results provide further support: LC-BC beats BC with an estimated probability of 71.14% and ILID with 67.14%, while LC-DP beats DP with 64.29%, when comparing independently sampled runs on the same task. All three 95% confidence intervals lie above 50%. Together, these analyses support an overall improvement across the evaluated tasks.
-
When is the extra training worthwhile? The LC-loss and LLM-Captioner are used only during training, so there is no additional test time cost. Training does take a lot more time to finish. However, the extra training brings a large benefit when demonstrations are imperfect or scarce (Section 5.3):
- Learning from imperfect data. With mixed-quality data at N=3, LC-BC achieves 78.0 3.5%, compared with 55.3 11.0% for BC and 64.7 5.8% for the strongest alternative. At N=2, LC-DP achieves 70.0 2.0%, compared with 56.0 4.0% for DP and 58.7 3.1% for the strongest alternative.
- Needing fewer expert demonstrations. LC-BC reaches 93.2% using only 10% of the expert data, close to BC’s 94.0% using all expert data. LC-DP reaches 92.4% using only 5%, close to DP’s 94.4% using all expert data.
- These results show that extra computation can reduce the need for large amounts of clean expert data. This can be useful when collecting more or better demonstrations is expensive, although we have not measured total cost savings.
We sincerely thank the reviewer for encouraging this closer evaluation. We will add these statistical results and clarify the practical benefit of the extra training: better use of limited or imperfect expert data, with no additional deployment cost.
References
[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G. Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS, 2021.
Thank you for the response and the additional analysis. I appreciate the work and will raise my score to 2. However, I have two remaining questions that I consider central:
-
Is privileged information fully removed in the human/VLM experiments? In Table R1, only the labels on the expert data are replaced. Are the critiques on the suboptimal data still generated by g? If so, the privileged movement guidance <M> is still present during training, and Table R1 does not show that LC-IL works without g. The o4-mini experiment appears to be the only setting where g is fully replaced. Could you specify which tasks it covers, its standard deviations, and what o4-mini receives as input (images, or the same state features used by g)?
-
Is the language-specific gain statistically reliable? Under the same underlying features, LC-BC/LC-DP improve over the scalar variant by only 2.7 / 3.9 points on average. That puts roughly half of the total gain over BC on the features rather than the language form. Could you report 95% bootstrap confidence intervals for LC vs. the feature-derived scalar variant, as you did in Table R2?
We sincerely thank the reviewer for the thoughtful feedback. Below, we address your concerns about additional supervision, performance gains, and training cost, supported by new experiments and statistical analyses.
Q1. The role of language and additional behavioral knowledge.
We thank the reviewer for raising this distinction. Enabling policies to learn from additional behavioral knowledge is one of the central motivations of LC-IL. Humans and VLMs can assess action quality and describe corrections without supplying exact privileged simulation state and actions. Language provides an interface for expressing this feedback, and our contribution is a differentiable objective that turns it into supervision for policy models.
Table R1. Human and VLM-generated labels with expert data in data efficiency setting. Values are mean ± standard deviation of success rates (%) on Box-Close.
Learning from human and VLM labels. We evaluate alternative annotation sources by replacing critiques on the selected expert data with human or GPT-6 Astra annotations, using 2% and 5% expert data. As shown in Table R1, both sources improve LC-BC’s mean success over BC at both expert-data fractions.
Human critiques are particularly effective for LC-DP, achieving 74.4% and 89.6% mean success with 2% and 5% expert data, respectively, compared with 71.2% and 85.6% for DP. Meanwhile, GPT-6 Astra matches μg’s mean LC-BC success with 2% expert data (71.6%) and achieves a higher mean with 5% (88.0% versus 86.4%).
These results support human and VLM labels as useful sources of behavioral supervision across both policy families. We will state this more clearly in the paper.
The role of μg. Our scripted generator provides reproducible annotations for studying the framework; it is one source of critiques, not a required component of LC-IL. We acknowledge that its rules encode task-specific knowledge. For instance, hand-object distances in Box-Close are computable from the observed state. Consequently, comparisons with demonstration-only baselines include the benefit of additional supervision and do not isolate the contribution of language.
However, this limitation of μg does not imply that language critique requires a scripted controller. Beyond the partial replacements in Table R1, our existing o4-mini experiment replaces μg along with a 360M LLM-Captioner; it achieves 74.0%/74.8% for LC-BC/LC-DP, compared with 66.4%/72.4% for BC/DP. Thus, the framework can make behavioral knowledge from alternative sources usable for policy learning.
Benefits of language as a learning signal representation. Our feature-derived scalar and categorical variants use supervision from the same underlying generator features.
With the same underlying annotation knowledge, LC-BC and LC-DP achieve higher mean success than categorical supervision on 7/7 and 6/7 tasks, averaging 71.3% vs. 63.1% and 72.6% vs. 68.6%. The largest gain is 23.6% on Hammer (LC-BC: 75.6% → 99.2%). Both outperform feature-derived scalar supervision on 5/7 tasks, with average gains of 2.7% and 3.9%, reaching 12.4% on Relocate (LC-DP: 65.2% → 77.6%).
Language also accommodates human and VLM labels that do not come with the predefined numerical features used by μg. Converting such feedback into feature targets requires additional effort. LC-BC and LC-DP use them directly as supervision, incorporating behavioral knowledge from different sources without first converting it into features or scalar signals.
We thank the reviewer for prompting this clarification. We will clarify the role and limitations of μg and emphasize the evidence for language as both an effective learning signal and an interface for human and VLM behavioral knowledge in the revised paper.