acceptodds
Withdrawn from ICLR 2027

Compositional Language-Instructed Reinforcement Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A compositional natural-language instruction decomposes a sparse-reward task into an ordered sequence of subgoal instructions, without providing a criterion for when to advance. Most prior approaches learn a completion signal and deliver it to the agent as auxiliary reward. We present GLEAN (Grounded Language-Effect ANchoring), which learns a completion detector from segmented offline demonstrations and uses it to advance the policy through the subgoals. The detector summarizes the egocentric observations so far into a filter state, which is also an input to the policy. It scores the change in that state since the previous completed subgoal against the active subgoal instruction and compares the score with a threshold calibrated on held-out demonstrations. The policy is initialized by behavior cloning and, throughout online reinforcement learning, continues imitating the demonstrations' subgoal segments while each detection advances it to the next subgoal instruction; we call this grounded subgoal anchoring. On six BabyAI levels with a matched interaction budget, GLEAN achieves higher final success than every baseline that receives a completion signal as reward on all five compositional levels, including baselines that start from a policy behavior-cloned on the same demonstrations. Providing the environment's exact completion predicates as reward still yields lower final success than GLEAN with its learned detector, so the advantage of advancement with continued imitation over reward does not come from detector errors

Primary area:
reinforcement learning
Keywords:
reinforcement learning, instruction following, language grounding, subgoal completion detection, learning from demonstrations, behavior cloning, sparse reward, partial observability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_xJiTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper looks at the problem of language-conditioned RL. It introduces GLEAN, a method of incorporating trained subsequences with sub-instructions into a method by training a policy to follow each subgoal, a detector to see when subgoals are completed and online RL. The paper shows results on BabyAI against several baselines.

Critical strengths

Despite there being a lot of work in language-conditioned RL, the method does seem to be a new contribution and the related work section mostly does a good job of positioning the work (although as a minor nit, I think some older works, especially Policy Sketches may have been missed).

The method is evaluated on a variety of strong baselines which are introduced and explained well in the related work including HRLLI, CAREL and HIGhER and Ella, covering a pretty wide variety of language conditioned RL methods.

The methodology seems sound. While intro might have described it better, S4 and figure 1 make the method fairly clear and explain how the method works clearly.

Critical weaknesses

The biggest weakness to me are the experimental validations:

  • First, the method is only validated on BabyAI which is a bit old and a bit toy. This is the only environment the method is run or compared on, so it's a bit difficult to tell how well this method really generalizes. For instance, would the completion detector work nearly as well on environments that are not just 2D grids (e.g. 3D environments, video or even real world settings). This is a big part of the method and I am not sure BabyAI is the best test for it.
  • While the number and recency of baselines is impressive, the results themselves make the usefulness of the method slightly mixed. It seems that HIGhER+BC is more effective on half of these environments. The statement about where the paper's method is better: "achieves higher final success than every baseline that receives a completion signal as reward" is not clear to me. It would be useful for authors to explain further what they mean and say where they think their method has advantages over others, especially HIGhER+BC.

The data-budget experiment is appreciated, but if it's important it really ought to be in the main text of the paper.

Questions for authors

See above: Can you explain further what you mean and say where they think their method has advantages over others, especially HIGhER+BC? I am also having trouble understanding what you mean in your abstract when you describe where you method is better. Can you explain this?

Reviewer_u4nuOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies reinforcement learning from language instructions, using demonstrations split into segments with subgoal labels. Its completion model is learned offline: a causal encoder processes the observation history, and changes in the encoded state are compared against the language embedding for the current subgoal. After this training, the model is kept fixed. Thresholds fitted on held-out demonstrations decide when to move on to the following instruction, while the encoded history is also supplied to the policy. Policy learning starts with behavior cloning on individual subgoals, followed by recurrent PPO with demonstration-segment imitation updates throughout training. The study uses six BabyAI task families.

Critical strengths
  1. Advancing too early can cause an instruction to be skipped permanently. The proposed use of history differences, together with thresholds calibrated over windows, gives this problem a concrete treatment. How the detector is supervised offline and then held fixed is described clearly enough to examine and reproduce. For me, this is the main methodological contribution.

  2. The main BabyAI results use a common policy backbone and environment-interaction budget, cover multiple seeds, and provide both demonstration-based baselines and oracle variants. In Table 2, GoToSeq and MoveTwoAcross suffer considerably when continued imitation is removed. Excluding the filter-state input also lowers performance on these tasks. These ablations help explain the role of continued imitation in stable training.

Critical weaknesses
  1. The new contribution is fairly small, and its effect is not separated well enough. Earlier work already covers several parts of the approach: Policy Sketches uses subtask sequences, CAREL tracks finished instructions, and a lot of papers combines initialization from demonstrations with demonstration-augmented RL. What is new here seems mainly to be the history-difference completion model and the way it is integrated through calibration. The experiments do not give a clear measure of what these choices add. Appendix A.4 compares detector variants under an older setup, with no ablation of the calibration horizon. Table 2 gives stronger support for keeping imitation during training than for needing this particular completion rule.

  2. The test setting is limited, and some comparisons use different training conditions. Every task comes from BabyAI, with symbolic observations, generated language, scripted segment boundaries, and no more than four subgoals. There is no evaluation in a 3D embodied environment or robotics setting to check how the approach behaves when these assumptions no longer hold. Some baselines do use demonstrations, but CAREL does not, and the reward-based variants in Table 1 do not continue imitation during RL. With continued imitation added, the reward-based variant in Table 2 scores above GLEAN on MoveTwoAcross, 0.784 versus 0.469, and reaches nearly the same result on GoToSeq, 0.894 versus 0.891. That comparison also changes conditioning and policy inputs, so it does not isolate the effect of signal use.

Questions for authors
  1. How do the authors interpret the reward-based results with continued imitation in Table 2 in relation to the claim favoring advancement over rewards? Is there a comparison that changes only how the completion signal is used, keeping imitation updates, conditioning, and access to the detector state the same?

  2. The setup in Appendix A.4 includes self-imitation and completion rewards, leaves out the filter-state input, and does not vary the calibration horizon. Could the authors provide readout and calibration-window comparisons for the final GLEAN setup? I would also like to see which results connect these choices to online task success, rather than only to offline detection scores.

  3. Could CAREL be trained with the same demonstrations, behavior-cloning initialization, continued-imitation budget, and subgoal annotations? I would find this comparison helpful in deciding whether the improvement comes from the completion method or from demonstration supervision, and it could change my assessment.

  4. Are there results outside BabyAI, or for longer subgoal sequences and imperfect segmentation? I am concerned that the conclusions may depend on structured instructions and accurate scripted boundaries. Evidence under less controlled conditions could change my assessment. Evaluation on a separate 3D embodied benchmark would also be useful in future work.

Reviewer_ZwAgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors propose GLEAN, which is an approach for compositional language-instructed RL, where the agent must complete a sequence of sub-tasks under sparse rewards. In this approach, the authors first train a subgoal completion detector using segmented expert demonstrations and use it to determine when the agent must move on to the next subgoal (instead of providing additional rewards). The policy is initialized using behavior cloning objective, and then online PPO training is used. The authors provide experimental results on six BabyAI environments and show that the approach outperforms several language-guided RL and imitation learning baselines.

Critical strengths
  1. The paper uses a clear motivation for using subgoal completion detection to advance through language instructions, rather than using it only as an auxiliary reward. This addresses an important problem in sparse-reward RL.

  2. The experimental evaluation includes several relevant baselines and detailed ablations. The results show improvements on many BabyAI tasks.

Critical weaknesses
  1. My main concern is the limited technical novelty of the proposed completion detector. The method assumes access to expert demonstrations with known subgoal divisions and uses these to train a supervised completion predictor. The overall approach combines supervised completion detection and online RL with continued imitation, and it is unclear whether this combination provides sufficient novelty. Furthermore, the reliance on segmented expert demonstrations limits the applicability of the proposed approach.

  2. Another concern is whether the experiments clearly show that subgoal advancement is better than using completion-based rewards. In Table 2, the reward-based variant with continued imitation performs better than GLEAN on MoveTwoAcross. Therefore, I am not sure whether subgoal advancement provides an advantage over completion-based rewards. Furthermore, the writing is hard to follow and can be improved, especially, the experiment section can be significantly improved by adding a more detailed discussion of the observed results.

Questions for authors
  1. Can the authors include an ablation where an auxiliary classification head is added to the policy to predict subgoal completion, instead of training a separate completion detector? This would help establish whether a separate detector is necessary.

  2. In Table 2, the variant without subgoal advancement and filter-state input, but with completion rewards, outperforms GLEAN on MoveTwoAcross. Can the authors explain why this variant performs better?

  3. Appendix A.6 presents a formal analysis that is not discussed in the main paper. Furthermore, the analysis refers to an InfoNCE objective and hindsight relabeling, which are not used in the proposed method. Am I missing something?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.