acceptodds
Under review as a conference paper at ICLR 2027

Language-Critique Imitation Learning from Suboptimal Demonstrations

Abstract

Prior work on imitation learning from suboptimal demonstrations typically extracts supervision as a scalar per state or trajectory (e.g., a confidence estimate, discriminator score, or importance weight), which ranks behavior but cannot express task stage, what went wrong, or how to correct it. We propose language-critique imitation learning, an offline algorithm that learns policies with language labels based on expert and suboptimal demonstrations. Our method constructs language labels describing task progress, action optimality, and movement guidance, produced either by a lightweight feature-based generator or an off-the-shelf VLM. We then introduce a language-critique loss to train the policy using language signals without compressing them to scalars. We instantiate this loss for behavior cloning and diffusion policies, yielding LC-BC and LC-DP. We theoretically show that the proposed objective upper-bounds the expert performance gap under standard assumptions. Across seven continuous-control tasks in navigation, driving, and manipulation, LC-BC and LC-DP achieve the highest average success rates in their policy families, improving average success by 8.2% (BC) and 4.3% (DP). Further experiments demonstrate robustness to noisy expert selection and, on one task, comparable performance to full-data BC and diffusion policies using only 10% and 5% of the expert data, respectively. These results show that language critiques provide a practical and reliable learning signal for suboptimal demonstrations.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Imitation learning from suboptimal demonstrations, Learning from language

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sh5iOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper learns from expert data plus suboptimal data. The authors argue that scalar weights cannot express “where the error is and how to fix it,” so they generate structured language labels for each (s, a) pair, including task progress <T>, action quality <A>, and movement guidance <M>.

The labels come from a template generator μ\mug, based on task features: object distance, gripper state, and displacement relative to a reference target. μ\mug is distilled into a differentiable LLM-Captioner (SmolLM2-135M plus an MLP projection). During training, an LC-loss is added: the cross-entropy of the “expert label” under the policy’s action, minus the cross-entropy under the expert action, with a hinge applied. It is added to both BC and Diffusion Policy (DP; DP uses one-step reconstruction with reweighting). Theoretically, under assumptions such as language sufficiency, the LC loss upper-bounds performance degradation.

Results: Averaged over 7 tasks, the method improves BC by 8.2% and DP by 4.3%.

Critical strengths

The idea is novel: it uses language as a differentiable critic signal rather than compressing feedback into a scalar.

The control design is thorough: scalar, classifier, shuffled, verbose, opaque/named token, captioner trained from scratch, VLM-generated labels, and human labels. The authors also honestly report diagnostics showing that their theoretical assumptions are violated. The derivation of the DP reweighting is clear.

Critical weaknesses
  1. The role of “language” is not confirmed. The real source of information is hand-written heuristics that use privileged state. µg uses object distance, displacement relative to a reference target, “dist_delta,” and so on, so <M> is essentially directional action supervision, equivalent to a hand-crafted dense reward or correction signal. The baselines do not have access to this information, so the comparison is unfair. The authors’ own ablations also suggest that most of the gain comes from the features rather than the language.

  2. The improvements are small, the variance is large, and the cost is high. The average is 71.3 vs. 65.4, and many tasks have overlapping standard deviations. The average gain for LC-DP comes mainly from RELOCATE (DP’s std there is 24.2). Training time also increases substantially over BC.

Questions for authors

Please refer to the weaknesses above.

If these are addressed, I would be happy to raise my score.

8 comments

We sincerely thank the reviewer for the thoughtful feedback. Below, we address your concerns about additional supervision, performance gains, and training cost, supported by new experiments and statistical analyses.

Q1. The role of language and additional behavioral knowledge.

We thank the reviewer for raising this distinction. Enabling policies to learn from additional behavioral knowledge is one of the central motivations of LC-IL. Humans and VLMs can assess action quality and describe corrections without supplying exact privileged simulation state and actions. Language provides an interface for expressing this feedback, and our contribution is a differentiable objective that turns it into supervision for policy models.

Table R1. Human and VLM-generated labels with expert data in data efficiency setting. Values are mean ±\pm standard deviation of success rates (%) on Box-Close.

PolicyExpert-data label source2% expert data5% expert data
BCNone48.8 ±\pm 11.977.6 ±\pm 11.9
LC-BCμg\mu_g71.6 ±\pm 5.786.4±5.0‾\underline{86.4 \pm 5.0}
LC-BCHuman68.4±6.9‾\underline{68.4 \pm 6.9}84.8 ±\pm 2.7
LC-BCGPT-6 Astra71.6 ±\pm 6.788.0 ±\pm 6.8
DPNone71.2 ±\pm 1.885.6 ±\pm 3.3
LC-DPμg\mu_g77.6 ±\pm 7.192.4 ±\pm 1.7
LC-DPHuman74.4±2.6‾\underline{74.4 \pm 2.6}89.6±3.3‾\underline{89.6 \pm 3.3}
LC-DPGPT-6 Astra71.2 ±\pm 1.185.6 ±\pm 1.7
  1. Learning from human and VLM labels. We evaluate alternative annotation sources by replacing critiques on the selected expert data with human or GPT-6 Astra annotations, using 2% and 5% expert data. As shown in Table R1, both sources improve LC-BC’s mean success over BC at both expert-data fractions.

    Human critiques are particularly effective for LC-DP, achieving 74.4% and 89.6% mean success with 2% and 5% expert data, respectively, compared with 71.2% and 85.6% for DP. Meanwhile, GPT-6 Astra matches μg\mu_g’s mean LC-BC success with 2% expert data (71.6%) and achieves a higher mean with 5% (88.0% versus 86.4%).

    These results support human and VLM labels as useful sources of behavioral supervision across both policy families. We will state this more clearly in the paper.

  2. The role of μg\mu_g. Our scripted generator provides reproducible annotations for studying the framework; it is one source of critiques, not a required component of LC-IL. We acknowledge that its rules encode task-specific knowledge. For instance, hand-object distances in Box-Close are computable from the observed state. Consequently, comparisons with demonstration-only baselines include the benefit of additional supervision and do not isolate the contribution of language.

    However, this limitation of μg\mu_g does not imply that language critique requires a scripted controller. Beyond the partial replacements in Table R1, our existing o4-mini experiment replaces μg\mu_g along with a 360M LLM-Captioner; it achieves 74.0%/74.8% for LC-BC/LC-DP, compared with 66.4%/72.4% for BC/DP. Thus, the framework can make behavioral knowledge from alternative sources usable for policy learning.

  3. Benefits of language as a learning signal representation. Our feature-derived scalar and categorical variants use supervision from the same underlying generator features.

    With the same underlying annotation knowledge, LC-BC and LC-DP achieve higher mean success than categorical supervision on 7/7 and 6/7 tasks, averaging 71.3% vs. 63.1% and 72.6% vs. 68.6%. The largest gain is 23.6% on Hammer (LC-BC: 75.6% → 99.2%). Both outperform feature-derived scalar supervision on 5/7 tasks, with average gains of 2.7% and 3.9%, reaching 12.4% on Relocate (LC-DP: 65.2% → 77.6%).

  4. Language also accommodates human and VLM labels that do not come with the predefined numerical features used by μg\mu_g. Converting such feedback into feature targets requires additional effort. LC-BC and LC-DP use them directly as supervision, incorporating behavioral knowledge from different sources without first converting it into features or scalar signals.

We thank the reviewer for prompting this clarification. We will clarify the role and limitations of μg\mu_g and emphasize the evidence for language as both an effective learning signal and an interface for human and VLM behavioral knowledge in the revised paper.

Q2. The reliability of performance gain and training cost

We thank the reviewer for raising this concern. We added further statistical analysis to see how reliable the improvements are. We also explain why extra training can be worthwhile when expert data are limited or imperfect.

Table R2. Improvements and their uncertainty. Brackets indicate 95% bootstrap confidence intervals. Mean and IQM gains are absolute success-rate differences in percentage points; probability of improvement is reported as a percentage.

ComparisonMean gain [95% CI]IQM gain [95% CI]Probability of improvement [95% CI]
LC-BC vs. BC6.23 [3.49, 8.97]5.58 [1.26, 10.00]71.14 [59.43, 82.57]
LC-BC vs. ILID5.89 [3.14, 8.63]3.58 [-0.42, 7.37]67.14 [57.14, 76.86]
LC-DP vs. DP2.97 [0.46, 6.34]3.05 [0.21, 6.42]64.29 [51.71, 76.29]
  1. How reliable are the improvements? Following Agarwal et al. [1], we analyze all training seeds across 7 tasks. We resample training runs independently within each task and method 50000 times to estimate 95% confidence intervals for (1) the average success rate gain, (2) IQM (interquartile mean) gain, and (3) probability of improvement. IQM reduces the influence of unusually high or low scores. Probability of improvement measures how often our method beats the baseline.

    As shown in Table R2, LC-BC improves average success over BC by 6.23% and ILID by 5.89%, with both 95% confidence intervals above zero. Its IQM gain over BC is also positive at 5.58% [1.26, 10.00]. Against ILID, the IQM gain is 3.58%, although its interval includes zero, so this comparison remains less certain.

    For LC-DP, the IQM gain of 3.05% [0.21, 6.42] is slightly larger than the mean gain of 2.97%. Thus, the improvement remains when unusually high or low scores have less influence on the comparison. Removing Relocate entirely also leaves a positive average gain of 1.73% [0.27, 3.27] across the remaining six tasks, showing that the overall benefit extends beyond Relocate.

    The probability-of-improvement results provide further support: LC-BC beats BC with an estimated probability of 71.14% and ILID with 67.14%, while LC-DP beats DP with 64.29%, when comparing independently sampled runs on the same task. All three 95% confidence intervals lie above 50%. Together, these analyses support an overall improvement across the evaluated tasks.

  2. When is the extra training worthwhile? The LC-loss and LLM-Captioner are used only during training, so there is no additional test time cost. Training does take a lot more time to finish. However, the extra training brings a large benefit when demonstrations are imperfect or scarce (Section 5.3):

    1. Learning from imperfect data. With mixed-quality data at N=3, LC-BC achieves 78.0 ±\pm 3.5%, compared with 55.3 ±\pm 11.0% for BC and 64.7 ±\pm 5.8% for the strongest alternative. At N=2, LC-DP achieves 70.0 ±\pm 2.0%, compared with 56.0 ±\pm 4.0% for DP and 58.7 ±\pm 3.1% for the strongest alternative.
    2. Needing fewer expert demonstrations. LC-BC reaches 93.2% using only 10% of the expert data, close to BC’s 94.0% using all expert data. LC-DP reaches 92.4% using only 5%, close to DP’s 94.4% using all expert data.
    3. These results show that extra computation can reduce the need for large amounts of clean expert data. This can be useful when collecting more or better demonstrations is expensive, although we have not measured total cost savings.

We sincerely thank the reviewer for encouraging this closer evaluation. We will add these statistical results and clarify the practical benefit of the extra training: better use of limited or imperfect expert data, with no additional deployment cost.

References

[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G. Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS, 2021.

Thank you for the response and the additional analysis. I appreciate the work and will raise my score to 2. However, I have two remaining questions that I consider central:

  1. Is privileged information fully removed in the human/VLM experiments? In Table R1, only the labels on the expert data are replaced. Are the critiques on the suboptimal data still generated by g? If so, the privileged movement guidance <M> is still present during training, and Table R1 does not show that LC-IL works without g. The o4-mini experiment appears to be the only setting where g is fully replaced. Could you specify which tasks it covers, its standard deviations, and what o4-mini receives as input (images, or the same state features used by g)?

  2. Is the language-specific gain statistically reliable? Under the same underlying features, LC-BC/LC-DP improve over the scalar variant by only 2.7 / 3.9 points on average. That puts roughly half of the total gain over BC on the features rather than the language form. Could you report 95% bootstrap confidence intervals for LC vs. the feature-derived scalar variant, as you did in Table R2?

Reviewer_S12nOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes using a language critique loss in addition to behavior cloning and diffusion policy losses for imitation learning. A structured generator (either rule based or an off the shelf VLM) labels a large mixed quality dataset with task progress, action optimality, and movement guidance. An LLM captioner is finetuned on the labeled mixed quality dataset embeds (st,at,lt)(s_t, a_t, l_t) into a single token. The imitation policy is then trained on the expert dataset with the frozen LLM captioner with either a BC loss or a diffusion loss, plus the cross entropy loss of the expert label under the policy action. On several different tasks, LC-BC and LC-DP give higher average scores than other baselines.

Critical strengths

The paper's main strength is the variety of different tasks and baselines used for evaluation, ranging from navigation to peg insertion and methods that can use subpotimal demonstrations. The authors also perform a variety of experiments on mixed quality demonstrations, fewer demonstrations, and language label ablation studies.

Critical weaknesses

The biggest weakness seems to be that the language labels contain some privileged information that doesn't clearly show that language labels are superior to other approaches. For example, in Table 4, μg\mu_g contains distance between the hand and target, reference minus executed gripper command etc. On BoxClose Table 4c, categorical heads (87.2) and a from scratch captioner (88.8) seems to match μg\mu_g (87.6) for LC-BC. Further more, a scalar computed directly from the privileged features (Table 20) ties LC-BC on BoxClose and beats it on PegInsert, albeit with a lower overall average.

Another concern is that the standard deviations are quite high and it's difficult to discern whether the results are valid or are just seed noise (Table 1).

Questions for authors
  1. Does μg\mu_g use the expert controller for the mixed quality dataset, i.e. what is the reference target and reference gripper command? If they come from an expert controller, how does LC-BC fare against say a baseline with the same access?
6 comments

We sincerely thank the reviewer for the careful review and helpful feedback. Below, we clarify the contribution of language and present additional experiments and statistical analyses to address the reliability of our results.

Q1. Controller-derived information and the contribution of language.

We thank the reviewer for raising the concern. We acknowledge that μg\mu_g supplies additional behavioral knowledge. However, one of the central motivations of LC-IL is to enable policy learning from additional knowledge, which can be provided by humans or advanced VLMs. We will clarify the role of μg\mu_g, explain what information it uses, and discuss alternative sources of language critiques and the advantages of using language as a learning signal representation.

  1. The role of μg\mu_g and the information it uses. Our scripted generator μg\mu_g provides reproducible language labels for studying how behavioral language critiques can help policy learning. Nevertheless, we acknowledge that μg\mu_g still contains some privileged information, but in a reasonable way. For example, in Box-Close, the observed state contains the position of the hand, object, and goal, so the distances can be inferred directly.

    But the privileged part might be that we explicitly coded the distance between the hand and the goal. As well as the gripper target, the rule is coded as: if it’s close to the object, then close it; if it’s far from the object, then keep it open. These rules therefore encode knowledge about desirable behavior and critiques.

    However, μg\mu_g is only one of the possible annotation sources, allowing us to quickly design and evaluate our LC algorithms. The framework actually allows other sources, including humans and strong VLMs, to provide language labels.

Table R1. Human and VLM-generated labels with expert data in data efficiency setting. Values are mean ±\pm standard deviation of success rates (%) on Box-Close.

PolicyExpert-data label source2% expert data5% expert data
BCNone48.8 ±\pm 11.977.6 ±\pm 11.9
LC-BCμg\mu_g71.6 ±\pm 5.786.4±5.0‾\underline{86.4 \pm 5.0}
LC-BCHuman68.4±6.9‾\underline{68.4 \pm 6.9}84.8 ±\pm 2.7
LC-BCGPT-6 Astra71.6 ±\pm 6.788.0 ±\pm 6.8
DPNone71.2 ±\pm 1.885.6 ±\pm 3.3
LC-DPμg\mu_g77.6 ±\pm 7.192.4 ±\pm 1.7
LC-DPHuman74.4±2.6‾\underline{74.4 \pm 2.6}89.6±3.3‾\underline{89.6 \pm 3.3}
LC-DPGPT-6 Astra71.2 ±\pm 1.185.6 ±\pm 1.7
  1. Learning from human and VLM critiques. The scripted generator provides reproducible annotations, but it is only one possible source of critiques. Humans and VLMs can assess behavior and describe corrections without supplying exact action targets. Our differentiable LC-loss makes this language usable for policy training. We evaluate human and GPT-6 Astra (which does not require privileged information) labels on the 2% and 5% expert-data.

    As shown in Table R1, human and GPT-6 Astra labels improve LC-BC’s mean success at both expert-data fractions. With 2% expert data, they achieve 68.4% and 71.6%, respectively, compared with 48.8% for BC. With 5%, they achieve 84.8% versus 77.6%% and 88.0%, compared with 77.6% for BC. Notably, GPT-6 Astra matches μg\mu_g’s mean LC-BC success with 2% expert data (71.6%) and achieves a higher mean with 5% (88.0% versus 86.4%). Human labels are particularly effective for LC-DP, achieving 74.4% and 89.6% mean success with 2% and 5% expert data, respectively, compared with 71.2% and 85.6% for DP.

    Our existing o4-mini experiment replaces μg\mu_g throughout annotation. With a 360M captioner, LC-BC and LC-DP achieve 74.0% and 74.8%, compared with 66.4% and 72.4% for BC and DP. Together, these results show that LC-IL can use behavioral supervision from sources beyond the scripted generator.

  1. Benefits of language as a learning signal representation. We thank the reviewer for asking what language adds beyond categorical and feature-derived scalar supervision. We highlight two benefits: (1) better policy performance across tasks and settings, and (2) the ability to use feedback without predefined numerical features or class targets.
    1. Our categorical variants use the same critique components, while feature-derived variants in Table 20 compute supervision directly from the numerical features underlying μg\mu_g. Across seven tasks, LC-BC and LC-DP achieve higher mean success than categorical supervision on 7/7 and 6/7 tasks, averaging 71.3% vs. 63.1% and 72.6% vs. 68.6%, with a gain of 23.6% on Hammer for LC-BC. Against the direct feature-derived variant in Table 20, both improve on 5/7 tasks, averaging 71.3% vs. 68.6% for LC-BC and 72.6% vs. 68.7% for LC-DP. The largest gains are 11.2% on Hammer and 12.4% on Relocate, respectively. We agree that on Box-Close, categorical variants and LLM-Captioners trained from scratch match LC-BC; feature-derived variants also tie LC-BC. For LC-DP, however, language pretraining improves from 63.2% to 74.0%. The mixed-quality results provide further support: at N=3, LC-BC achieves 78.0%, versus 64.7% for categorical and 56.7% for language-derived scalar supervision, using identical filtered partitions and the same annotation source.
    2. Language labels without predefined features or class targets. Human and VLM labels do not necessarily provide the numerical features used by μg\mu_g or fit into a predefined categories. Applying feature-based or categorical supervision would require defining suitable targets and extracting them from the feedback. LC-BC and LC-DP use the language labels directly, allowing different sources of behavioral knowledge to guide policy learning without first converting them into task-specific features, categorical targets, or scalar scores.

In summary, these results support language as an effective policy learning signal and a way to incorporate human and VLM knowledge into training. We thank the reviewer for encouraging us to clarify these benefits. In the revised paper, we will clarify the role and limitations of μg\mu_g and highlight both the matched comparisons and the results from alternative annotation sources.

Q2. Reliability of performance improvements.

We thank the reviewer for raising the concern about seed variability. We added statistical analyses to check how reliable the improvements in Table 1 are. We also highlight the larger gains observed in the degraded data settings.

Table R2. Improvements and their uncertainty. Brackets indicate 95% bootstrap confidence intervals. Mean and IQM gains are absolute success-rate differences in percentage points; probability of improvement is reported as a percentage.

ComparisonMean gain [95% CI]IQM gain [95% CI]Probability of improvement [95% CI]
LC-BC vs. BC6.23 [3.49, 8.97]5.58 [1.26, 10.00]71.14 [59.43, 82.57]
LC-BC vs. ILID5.89 [3.14, 8.63]3.58 [-0.42, 7.37]67.14 [57.14, 76.86]
LC-DP vs. DP2.97 [0.46, 6.34]3.05 [0.21, 6.42]64.29 [51.71, 76.29]
  1. Statistical analysis across tasks and training seeds. Following Agarwal et al. [1], we analyze all training seeds across 7 tasks. We resample training runs independently within each task and method 50000 times to estimate 95% confidence intervals for (1) the average success rate gain, (2) IQM (interquartile mean) gain, and (3) probability of improvement. IQM reduces the influence of unusually high or low scores. Probability of improvement measures how often our method beats the baseline.

    As shown in Table R2, LC-BC improves average success over BC by 6.23% and ILID (which is our strongest baseline) by 5.89%, while LC-DP improves over DP by 2.97%. All three confidence intervals for these average gains lie above zero. The IQM gains over BC and DP also remain positive, suggesting that improvements still hold when extreme scores are excluded.

    The estimated probabilities of improvement are 71.14%, 67.14%, and 64.29%, respectively, with all three confidence intervals above 50%. Even after excluding Relocate, LC-DP retains an average gain of 1.73% [0.27, 3.27] across the other six tasks. These checks support an overall improvement for our methods.

  2. Larger gains with degraded data settings. Section 5.3 provides further evidence across different data conditions.

    1. Mixed-quality data. At N=3, LC-BC achieves 78.0 ±\pm 3.5%, compared with 55.3 ±\pm 11.0% for BC and 64.7 ±\pm 5.8% for the strongest alternative. At N=2, LC-DP achieves 70.0 ±\pm 2.0%, compared with 56.0 ±\pm 4.0% for DP and 58.7 ±\pm 3.1% for the strongest alternative. Both methods achieve the highest mean success within their policy families at all four tested quality levels.
    2. Less expert data. With 2% expert data, LC-BC achieves 71.6 ±\pm 5.7%, compared with 48.8 ±\pm 11.9% for BC. With 5%, LC-DP achieves 92.4 ±\pm 1.7%, compared with 85.6 ±\pm 3.3% for DP.

In these examples, the gains are substantially larger. They provide additional evidence of useful improvements beyond the averages in Table 1.

We thank the reviewer for encouraging this closer evaluation. We will include these statistical analyses and highlight the results under degraded data conditions to make the evidence of improvements clearer.

References

[1] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron Courville, and Marc G. Bellemare. Deep Reinforcement Learning at the Edge of the Statistical Precipice. NeurIPS, 2021.

Reviewer_8ohyOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses imitation learning from a small expert dataset supplemented with suboptimal demonstrations. Rather than training the policy directly on the suboptimal data, the authors label every state-action pair with a short templated critique describing task progress, action quality, and movement guidance. They then train a small language model (the LLM-Captioner) to predict these critiques from the state and action. While training the policy on expert data, they add an auxiliary loss that measures whether the captioner would describe the policy's predicted action the same way it describes the expert action. This provides a dense, task-aware learning signal while leaving inference unchanged, since the captioner is used only during training. They implement this method for both a behavior cloning policy (LC-BC) and diffusion policy (LC-DP). Across seven tasks spanning navigation, driving, and manipulation, both variants improve average success over their base policies and prior methods for learning from suboptimal demonstrations. Detailed ablations on the Box-Close task support many of the design choices, though some results are mixed and reveal sensitivity to hyper-parameters. Further experiments on Box-Close show that the method also provides gains when expert data is scarce or noisy.

Critical strengths
  1. The core idea is an interesting way to make use of suboptimal demonstrations. A critic trained on annotated suboptimal data provides dense supervision, and the policy is trained so that the critic describes its actions the same way it describes the expert's. While scripted annotation works better for this paper, as VLMs improve I imagine this form of annotation will become even more scalable. I also like that rather than using these language annotations for chain-of-thought, the method uses them as a training-only auxiliary signal, which keeps inference cheap.

  2. The evaluation suite is reasonably designed and the ablations are extensive, with consistent improvements over baselines and an honest reporting of the shortcomings. The experiments covers seven tasks with both feedforward and diffusion policies, along with a range of ablations over label components, label sources, and critic design. The results with scarce or noisy expert data are particularly encouraging: on Box-Close, LC-BC with 10% of the expert data performs comparably to BC trained on all of it (93.2% vs. 94.0%).

Critical weaknesses
  1. The method may be difficult to use in practice. It requires task-specific label generation (a hand-written script using privileged simulator state, since VLM labels are much less accurate), training a separate language model on those labels (~25 hours on Box-Close), and then back-propagating through that model at every policy training step (which increases training time from ~20 minutes to ~8 hours for BC).

  2. Several ablations behave differently than the proposed mechanism would predict, which raises questions about why the method works. If the language critiques provide richer supervision as claimed, a more capable captioner should help or at least not hurt, and labels with fewer components should still provide some benefit. Instead, scaling the captioner from 135M to 360M parameters drops LC-BC on Box-Close from 87.6% to 70.4%, and every partial-label variant of LC-DP performs at or below plain diffusion policy. LC-DP also improves over diffusion policy at only one of the five tested loss weights. Together, these results suggest the gains depend on a fairly narrow configuration, and it is unclear how much tuning would be needed to apply the method to a new task.

Questions for authors
  1. The loss weight λ\lambda varies substantially across tasks (Table 14). How did you tune it for each task, and why do you think the method is so sensitive to this parameter?

  2. Why doesn't additional captioner capacity help? Are the larger captioners in Appendix I.4 less accurate critics? Could you report their accuracy on a held-out portion of the dataset, as done for the 135M model in Appendix A.6? Since the smallest model tested currently works best, would an even smaller language model perform better still?

  3. Appendix I.5 shows that training the critic without any suboptimal data significantly hurts performance. How much suboptimal data is actually needed, and what type of suboptimal demonstrations are used (e.g., random noise injection, play data, earlier RL checkpoints)?

  4. Where in the rollouts do the improvements from your method materialize? Is it in fine-grained movements, in subgoal selection, or is there no clear pattern across tasks in where the language loss helps?

6 comments

We sincerely thank the reviewer for the careful reading and thoughtful, constructive feedback. We appreciate the suggestions. We have done additional experiments on LLM-Captioner behavior, loss-weight sensitivity, and suboptimal-data efficiency. We address each concern below and will incorporate the corresponding analyses and clarifications into the revised paper.

Q1. Practicality: task-specific annotations, privileged simulator state, and training overhead.

Our framework provides a platform to incorporate a more expressive learning representation, i.e., language, into policy learning from suboptimal demonstrations. Our language critique (LC) methods can leverage language labels from various sources, beyond the language generator μg\mu_g, including VLMs and human supervisors. We have experimented with the following language sources.

  1. Human labels. With human labels on 2% of the expert data, LC-BC achieves 68.4% success versus 48.8% for BC, and LC-DP achieves 74.4% versus 71.2% for DP. With 5% data, LC-BC achieves 84.8% versus 77.6% for BC, and LC-DP achieves 89.6% versus 85.6% for DP. Human labels improve mean success over BC and DP at both expert-data fractions.
  2. o4-mini labels. With o4-mini annotations on all data and a 360M LLM-Captioner, LC-BC and LC-DP achieve 74.0% and 74.8%, compared with 66.4% and 72.4% for BC and DP.
  3. GPT-6 Astra labels. We additionally test a better VLM, GPT-6 Astra. With GPT-6 Astra labels on 2% of the expert data, LC-BC and LC-DP achieve 71.6 ±\pm 6.7% and 71.2 ±\pm 1.1%, respectively. With 5% data, they achieve 88.0 ±\pm 6.8% and 85.6 ±\pm 1.7%. LC-BC exceeds BC’s mean success at both fractions, while LC-DP matches DP’s mean. Notably, GPT-6 Astra also matches μg\mu_g’s mean LC-BC success with 2% expert data (71.6%) and achieves a higher mean with 5% (88.0% versus 86.4%).
  4. These results demonstrate that annotation sources that do not rely on privileged simulator state can provide useful supervision.

We acknowledge that our language generator is based on privileged simulator state and task-specific rules. For geometric inputs, perception methods such as FoundationPose [1] and ReKep [2] estimate object poses or identify keypoints from RGB-D observations. Meanwhile, Code as Policies [7] and CaP-X [8] demonstrate that language models can generate executable programs that compose perception and control primitives. Building on these findings, a programmed language generator for LC-IL could query perception outputs, compute geometric relations, and return structured language critiques. This offers a potential approach toward reducing reliance on both the privileged simulator state and manually written rules. Evaluating this integration within LC-IL is a promising direction for future work.

We also acknowledge the substantial training overhead. We want to emphasize first that the LC-loss and LLM-Captioner are used only during training, with no additional deployment cost. Second, the degraded-data experiments in Section 5.3 demonstrate a useful trade-off between additional training computation and demonstration quantity and quality, since lower-quality data and less expert data mean cheaper data collection costs:

  1. Fewer expert demonstrations. LC-BC with only 10% expert data reaches 93.2% success, compared with 94.0% for full-data BC; LC-DP with only 5% reaches 92.4%, compared with 94.4% for full-data DP.
  2. Mixed-quality demonstrations. With imperfect pseudo-expert partitions, LC-BC reaches 78.0% versus 55.3% for BC at N=3, while LC-DP reaches 70.0% versus 56.0% for DP at N=2. Both methods achieve the highest mean success rates within their policy families at every tested filtering threshold, with all methods using identical filtered partitions.

Together, they show that additional offline computation supports learning from both fewer and less reliable expert demonstrations, potentially reducing overall data collection effort. This makes the overhead worthwhile when obtaining abundant and high-quality data is expensive.

Q2. LLM-Captioner scaling issue and partial labels.

In our use case, a larger LLM-Captioner is not necessarily a more capable or suitable model for providing policy supervision. Its usefulness depends on 2 things: (1) the language labels it predicts, and (2) the gradients it supplies during policy optimization. Following the reviewer’s suggestion, we examine these 2 aspects and additionally evaluate a smaller captioner.

Table R1. Captioner diagnostics and downstream policy performance on Box-Close.

CaptionerJoint accuracy (%)Median rrMedian ccConflict (%)LC-BC success (%)LC-DP success (%)
Pythia-70M20.11.010.15936.666.4 ±\pm 11.968.8 ±\pm 5.2
SmolLM2-135M51.94.810.4586.287.6 ±\pm 6.574.0 ±\pm 5.7
SmolLM2-360M41.43.180.3459.970.4 ±\pm 9.270.8 ±\pm 3.6
Qwen2-0.5B56.81.940.5907.979.2 ±\pm 11.170.8 ±\pm 7.7
Mistral-7B53.21.470.2578.964.8 ±\pm 7.665.2 ±\pm 4.8
  1. Label prediction accuracy. Table R1 shows that SmolLM2-135M achieves higher joint accuracy than SmolLM2-360M (51.9% versus 41.4%), with higher action-quality and movement guidance <M> accuracies (90.5%/58.9% versus 85.2%/48.0%). However, Qwen2-0.5B achieves the highest accuracy (56.8%) while yielding lower policy success than the 135M captioner. Thus, larger captioners are not uniformly less accurate, and better critique prediction alone does not guarantee better policy supervision.
  2. Gradient behavior. Using identical state-action data and 20 shared suboptimal policy checkpoints, with frozen LLM-captioners and λ=0.01\lambda=0.01, we measure the weighted gradient-norm ratio rr to quantify the relative strength of LC supervision compared with the BC loss, motivated by gradient-based loss balancing [5]. We also report the gradient cosine cc and the fraction of comparisons with c<0c<0 to see how the proposed LC-loss gradient agrees with the BC loss, following the gradient-conflict criterion in PCGrad [6]:
\qquad c=\frac{\langle g_{\mathrm{LC}},g_{\mathrm{BC}}\rangle} {\lVert g_{\mathrm{LC}}\rVert_2\lVert g_{\mathrm{BC}}\rVert_2},$$ where $g_{\mathrm{LC}} = \nabla_{\theta}\mathcal{L}_{\mathrm{LC}}$ and $g_{\mathrm{BC}} = \nabla_{\theta}\mathcal{L}_{\mathrm{BC}}$. Comparisons with zero-norm gradients are excluded from the cosine and conflict statistics. Table R1 shows that the 135M captioner provides the strongest relative LC gradients (median r=4.81), positive alignment with BC (median c=0.458), and the lowest conflict fraction (6.2%). Together, these observations suggest that its critique supervision supplies a substantial training signal that generally agrees with expert imitation. However, stronger gradients are not necessarily better, and alignment measures agreement with BC rather than task success. For example, Qwen2-0.5B has higher accuracy (56.8% versus 51.9%) and stronger alignment (c=0.59 versus 0.458), yet supplies a weaker relative LC signal (r=1.94 versus 4.81) and yields lower LC-BC success (79.2% versus 87.6%). Thus, neither prediction accuracy nor gradient alignment alone explains the performance differences. These findings suggest considering critique accuracy, gradient strength, and compatibility with imitation jointly when assessing a captioner’s suitability. This interpretation is consistent with prior work on classifier-guided diffusion, where predictive accuracy alone does not ensure useful guidance gradients [3, 4]. 3. **Smaller captioners and annotation sources.** Table R1 also shows that the newly evaluated Pythia-70M achieves **66.4%/68.8%** for LC-BC/LC-DP, below the 135M model’s **87.6%/74.0%**. Its diagnostics also show lower joint critique accuracy (20.1% versus 51.9%), weaker gradient alignment (median $c=0.159$ versus 0.458), and more frequent conflicts with BC (36.6% versus 6.2%). These observations indicate that Pythia both predicts critiques less accurately and provides supervision that less consistently agrees with expert imitation. Further reducing model size therefore does not improve performance in this setting. 4. Together, these findings indicate that captioner suitability does not depend on model size alone. It depends on label accuracy and the gradient for policy learning. We will add these diagnostics to Appendix I.4. **Partial labels.** Finally, we agree that none of the partial language label baselines improves LC-DP over DP in the reported Box-Close ablation. However, partial critiques can substantially improve LC-BC: `<T>+<M>` achieves **92.4%**, versus **66.4%** for BC. These results suggest that the effectiveness of a critique subset depends on the policy formulation, with the full combination performing best for LC-DP in this experiment. We will clarify this interpretation in Section 5.4.

Q3. Sensitivity to the LC-loss weight and tuning across tasks.

We acknowledge LC-DP’s sensitivity in the original Box-Close configuration, where it improves over DP at only one of the five tested weights (Figure 14). Our additional sweeps on Box-Close with reduced expert data show improvements at multiple weights, indicating that the range of effective weights depends on the data setting. We clarify our tuning procedure and these results below.

  1. Tuning procedure. In most cases, we searched λ∈{0.1,0.01,0.001,0.0001}\lambda\in\lbrace0.1,0.01,0.001,0.0001\rbrace while keeping other hyperparameters fixed, including learning rate, batch size, training epochs, and learning-rate scheduling. We first evaluated each candidate using one or two training seeds, selected the best-performing weight, and expanded the selected configuration to five seeds. λ\lambda balances objectives whose gradient magnitudes can differ across tasks and captioners. Different selected weights therefore do not necessarily imply narrow sensitivity within each task. Table R1 illustrates this: the same λ=0.01\lambda=0.01 produces different relative LC gradient strengths across captioners.
  2. Additional sweeps. Although LC-DP improves at only one tested weight in the Box-Close sweep, the reduced-expert-data experiments show gains at multiple weights. As shown in Table R2, with 5% expert data, LC-DP exceeds DP’s mean at all three tested weights from 10−110^{-1} to 10−310^{-3}; with 10%, it does so from 10−210^{-2} to 10−410^{-4}. These results show gains at multiple weights spanning two orders of magnitude in several settings. We will document the tuning procedure in Appendix G.2 and expand the sensitivity analysis in Appendix I.7.

Table R2. LC-DP success rates (%), mean ±\pm standard deviation across five seeds. Bold indicates means above DP, not statistical significance. These runs follow the Section 5.3 expert-data-efficiency configuration, retaining the remaining general data and language labels.

LC-loss weight λ\lambda2% expert data5% expert data10% expert data
DP baseline71.2 ±\pm 1.885.6 ±\pm 3.392.8 ±\pm 1.1
10−110^{-1}77.6 ±\pm 7.192.4 ±\pm 1.790.0 ±\pm 1.4
10−210^{-2}71.6 ±\pm 4.389.2 ±\pm 1.893.6 ±\pm 1.7
10−310^{-3}72.0 ±\pm 2.086.8 ±\pm 2.394.4 ±\pm 1.7
10−410^{-4}70.8 ±\pm 4.184.0 ±\pm 2.894.4 ±\pm 1.7

Comments on the paper

1 comment

Dear Area Chair and Reviewers,

We truly thank the Area Chair for handling our submission and the reviewers for their helpful feedback. We are very encouraged that our responses and additional evidence helped address the reviewers’ concerns, and that two reviewers raised their scores following the discussion. We sincerely appreciate their careful consideration of the new results and clarifications. Below, we summarize the main concerns and how we addressed them.

  • [All reviewers] Main contribution and clarification on scripted labels: We now state more clearly that our main contribution is the differentiable LC-loss. It allows policies to learn from humans, VLMs, or the task-specific scripted language generator μg\mu_g. We added experiments that fully replace μg\mu_g-generated labels with human or GPT-6 Astra expert labels and o4-mini suboptimal labels. With 2% expert data on Box-Close, LC-BC achieves 66.0% and 69.2%, respectively, versus 48.8% for BC. These results show that the method can work without the scripted generator. (Section 5.4, Table 3; Section 6)

  • [Reviewers S12n and sh5i] Benefits beyond task features: The reviewers asked whether the gains come only from the extra information in the labels. We added statistical comparisons against scalar supervision built from the same task features. LC-BC and LC-DP improve mean success by 2.69% [0.11%, 5.54%] and 3.89% [1.26%, 7.14%], respectively. Both 95% confidence intervals exclude zero, supporting a benefit over the tested feature-based alternatives. (Appendices I.5 and I.7, Tables 21 and 24)

  • [Reviewers S12n and sh5i] Reliability of the improvements: To check whether the gains reflect seed variability, we added bootstrap confidence intervals using seven tasks, five seeds, and 50,000 resamples. The mean-gain intervals are positive for LC-BC over BC and ILID, and LC-DP over DP. IQM results also support improvements over BC and DP. After removing Relocate, LC-DP still improves mean success by 1.73% [0.27%, 3.27%], showing that the gain is not limited to this task. (Appendix I.7, Table 23)

  • [Reviewer 8ohy] Why larger LLM-Captioners do not always help: We added label-prediction and gradient analyses, and tested a smaller Pythia-70M model. SmolLM2-135M combines accurate label prediction with stronger relative LC gradients and fewer conflicts with BC gradients. It achieves the best policy performance among the tested LLM-Captioner backbones. These results suggest that LLM-Captioner accuracy alone is insufficient; the gradients it provides also matter for policy learning. (Appendix I.3, Tables 18–19)

  • [Reviewer 8ohy] Loss-weight sensitivity: We clarified how we select the loss weight and added sweeps with 2%, 5%, and 10% expert data. In each setting, LC-DP exceeds DP’s mean success at three tested weights spanning two orders of magnitude. Thus, improvements are not always restricted to a narrow weight range, although tuning remains necessary. (Appendices G.2 and I.6, Table 22)

  • [Reviewer 8ohy] How much suboptimal data is needed: We added experiments using 50% of the suboptimal data to train the LLM-Captioner. LC-BC and LC-DP achieve 76.4% and 70.8%, versus 68.4% and 66.8% when the LLM-Captioner uses only expert data. This shows that even half of the suboptimal data still provides useful supervision, while the full dataset gives the best results. (Appendix I.4, Table 20)

  • [Reviewers 8ohy and sh5i] Training cost and practical value: We acknowledge the extra training cost and clarify that the method adds no inference cost. On Box-Close, LC-BC uses 10% expert data to reach 93.2%, compared with 94.0% for full-data BC. LC-DP uses 5% to reach 92.4%, compared with 94.4% for full-data DP. These results show that extra training can be useful when high-quality expert demonstrations are expensive to collect. (Section 5.3, Table 2; Appendix G.4, Table 17)

We appreciate the reviewers’ suggestions, which helped us clarify the paper and strengthen the evidence. We have revised the paper based on this valuable feedback and marked the modified passages in blue for ease of review. We sincerely thank the Area Chair and reviewers for their time, effort, and care in evaluating our work. We would be grateful if you could consider the revised paper and additional evidence in your final assessment. Thank you again for your constructive feedback and thoughtful consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.