acceptodds
Under review as a conference paper at ICLR 2027

Learning to Follow Instructions from Imperfect Guidance: Reference-Guided Reinforcement Learning with Demonstration Anchoring

Abstract

Planners, experts, and language models can guide instruction-following reinforcement learning (RL), but may receive inaccurate state descriptions or be impractical to query at deployment. To address this problem, we initialize policies by imitation and fine-tune them with rewards, temporary action-probability targets from a frozen reference on learner-visited states and continuing demonstration supervision (anchoring). We train policies that learn from imperfect guidance and act without it. In BabyAI, with left/right-swapped inputs to an online language guide, anchoring raises mean final success from 18.3% to 27.7% (five seeds, one initializer). Separately, across five independent BabyAI datasets and model fits, the framework improves time-averaged success by 6.5 percentage points over anchoring alone and 8.1 over the unchanged initial policy. Exchanging the same targets among states sharing an instruction lowers time-averaged success by 4.6 points relative to own-state guidance. Another BabyAI study restricts exchanges to targets sharing the most probable action, reducing this gap from 5.1 to 0.7 points; further benefit from exact own-state probabilities remains unresolved. In CrafText, own-state guidance beats reassignment, uniform targets and anchoring alone, but not the initial policy on main-study time-averaged success, despite higher final success.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, instruction following, learning from demonstrations, behavior cloning, policy regularization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_4QngOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies instruction-following RL with training-time guidance that may be imperfect. It proposes initializing the policy with bc, then fine-tuning it with PPO using a decaying KL penalty toward a frozen reference and continued supervision on demonstrations. Experiments in BabyAI and CrafText evaluate corrupted guidance, reference choices, and interventions that reassign reference targets across states.

Critical strengths
  1. The reference-property interventions are carefully designed and provide a useful empirical decomposition of what makes reference guidance effective.
Critical weaknesses
  1. The paper does not formulate a general or established imperfect-guidance setting before introducing its method. Although the left/right corruption experiment is operationally specified, it represents a particular synthetic perturbation rather than a general definition of imperfect guidance. The proposed PPO, reference-KL, and demonstration-anchoring objective is not derived from explicit assumptions about which signals are reliable or how guidance may be imperfect. Instead, the paper introduces a generic training recipe and subsequently shows that it retains some effectiveness under one constructed corruption. Consequently, “learning from imperfect guidance” is not well support.

  2. Behavior cloning is an integral component of the proposed pipeline rather than merely an implementation detail. However, the transition from offline BC to online PPO is non-trivial: the critic-specific branch receives no supervision during BC, and the paper reports early performance declines after PPO fine-tuning. Despite this, the core experiments do not include a from-scratch PPO ablation. It is therefore unclear whether BC improves online learning over direct PPO.

  3. The motivation for demonstration anchoring is to prevent the PPO policy from drifting away from demonstrated behavior. However, the paper does not adequately distinguish this mechanism from the more conventional alternative of kl regularizing PPO toward the fixed initial BC policy. Consequently, the experiments do not establish whether anchoring is necessary or whether a standard initial-policy KL regularizer would provide the same benefit.

Questions for authors
  1. In the corrupted-Qwen setting, Online-Qwen + Anchor appears to outperform No-reference + Anchor (.277 vs. .199), but this contrast is not reported as a primary paired comparison and uses only one BC initializer. Could the authors provide a direct statistical test across independently constructed pipelines to establish whether imperfect guidance itself improves over removing the reference entirely?
2 comments
AuthorsOpenReview ↗

Design rationale, initialization and alternatives to anchoring

Thank you for the review and the questions about claim scope, initialization and anchoring.

We contribute a BC-to-PPO training scheme with replicated gains and controlled tests of guidance (p. 2). We address weaknesses 1-3, then Q1. We explain the rationale and report scratch-PPO, anchor-replacement and corrupted-guide tests.

Citations refer to the submission; New marks completed rebuttal experiments. AUC is time-averaged success (fraction); gaps are percentage points and brackets confidence intervals.

1. Imperfect-guidance setting and objective rationale (weakness)

The tested intervention changes Qwen's descriptions, not the reward function, learner observations or demonstration labels (p. 6). Table 1 (p. 7) tests corrupted online guidance; Table 2 tests fitted-reference learning without corruption.

Reference KL supplies early learner-state advice. Its weight decays to zero to stop constraining reward learning; demonstration supervision continues to limit drift on demonstrated inputs (Introduction, p. 1; Eqs. (3)-(4), p. 4; schedules, p. 5). This is an empirical design, not a derivation under a general guide-error model. We will revise the title/introduction and replace the abstract's "learn from imperfect guidance" with "learn with training-time guidance and deploy without it" in the final manuscript.

2. Does BC improve online learning over direct PPO? (weakness)

To answer your second weakness, we completed a new BC-versus-scratch test: 40 plain-PPO policies, no reference or anchor, five independent data/BC pipelines, four paired seeds, two initializations. Architecture, 1.5M-step online budgets and evaluation match. Scratch randomizes actor/shared features; both arms copy the same BC-untrained critic-specific layers and reset optimizer/memory. Shared features can still change initial values.

BC/scratch AUC is .08113/.02871: +5.24 [3.55,6.93] (primary 95%, five seed-averaged pipeline means); final success is .12638/.06140. BC helps at equal online budgets, not necessarily equal total cost. Plain BC-initialized PPO remains 7.07 [5.36,8.77] AUC points below retained BC (.15180; exploratory 95%). The same five BC checkpoints were re-evaluated on new episodes; Table 2's .15110 uses the original episodes (p. 7). BC alone does not prevent decline; its cause is not isolated.

3. Demonstration anchoring versus initial-policy KL (weakness)

Anchoring fits demonstrator labels on demonstration states; persistent BC-KL matches the initializer's probabilities on learner-visited states. Both losses persist. Original Frozen-BC retains anchoring and anneals reference KL (Section 3.4, p. 4), so it does not test replacing the anchor.

New replacement test: with/without fitted guidance, replace anchoring by policy-to-frozen-BC KL. Constant weight .05 was selected before testing on excluded development data, separately with/without guidance. Five existing pipelines/four new paired seeds give 20 policies per condition; initialization, PPO, 1.5M-step budgets and evaluation match. Inputs are uncorrupted.

Continuing lossNo fitted reference AUCWith fitted reference AUC
Demonstration anchor.17209.22962
Persistent BC-KL.20001.20776

For your replacement question, the fixed-guidance AUC contrast below is inconclusive. Holding guidance fixed, anchor minus BC-KL is +2.19 [-.11,4.48] (primary 97.5%, two-comparison adjustment over five seed-averaged pipeline means). Final-success gain is +4.47 [2.66,6.27] (secondary 95%). Without guidance, BC-KL beats Anchor only by +2.79 [1.26,4.32] (secondary 95%).

The full scheme beats BC-KL alone by +2.96 [.78,5.14] (planned secondary, unadjusted 95%). Thus whole-scheme utility does not establish unique anchor necessity.

Q1: Does corrupted guidance improve over removing it?

Your Q1 asks for a paired test across independent pipelines; Table 1's .277/.199 comparison used one initializer. Those are final-success fractions. New matched test: 25 policies, five independent data/BC pipelines, one new paired seed each: clean/swapped Qwen crossed with anchor on/off, plus no-guide anchoring. Initialization, demonstrations, PPO, KL schedules, 1.5M-step budgets and evaluation match within pipeline.

Corrupted-guide-plus-anchor/no-guide-plus-anchor AUC is .22694/.17180: +5.51 [-2.09,13.12] (primary 97.5%, five pipeline differences). Final success is .31100/.22810, +8.29 [-.67,17.25] (secondary 95%). Added corrupted-guide benefit remains unresolved. The other primary contrast, corrupted-minus-clean AUC with anchoring minus that without, is +.27 [-6.92,7.46] (97.5%); protection is also unresolved. One seed per pipeline limits precision.

Anchoring improves corrupted-Qwen AUC from .14720 to .22694, +7.97 [5.06,10.89] (secondary 95%). This extends the anchoring observation across initializers; it does not establish protection. We will report these tests and distinctions in the final manuscript.

AuthorsOpenReview ↗

Follow-up on initialization, regularization and claim scope

Thank you again for your review. As the discussion deadline approaches, we would appreciate your assessment of the completed BC-versus-scratch comparison, demonstration-anchoring versus persistent BC-KL comparison, and five-pipeline corrupted-guide versus no-guide test reported in our response.

These experiments establish benefits of BC initialization at the tested online budget and provide additional evidence for the complete training scheme.

Do these additions address your concerns about initialization and the relevant alternatives? Please let us know which remaining issue is most important to your assessment so that we can respond directly.

Reviewer_RHS5OpenReview ↗
Rating
2(+1)
Confidence
1
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper starts from a policy initialized by behavior cloning (BC) and adds a KL penalty toward a frozen reference's action distributions to PPO training, while continuing supervised learning on demonstration actions. The KL weight gradually decays to zero, while demonstration supervision continues throughout training. The authors evaluate this training scheme by corrupting guide descriptions, exchanging reference outputs, and comparing different reference sources. Their experiments show improved learning in BabyAI. Reassigning reference outputs to other states reduces performance, but the effect becomes much smaller when the top recommendation is preserved; the additional value of exact own-state probabilities remains unclear. In the main CrafText study, the scheme slightly outperforms the PPO controls.

Critical strengths

n BabyAI, exchanging reference outputs across states reduces AUC by 5.13 percentage points, whereas preserving the top recommendation reduces the gap to 0.72 percentage points. I think this provides some empirical insight into the role of the top recommendation in guidance. During PPO training following BC initialization, combining own-state reference guidance with continuing demonstration supervision produces clear learning benefits in BabyAI. Learning AUC improves over both PPO with continuing demonstration supervision alone (Anchor only) and retaining the initial BC policy without further training (retained BC). The gains are consistently in the same direction across multiple demonstration datasets with separately trained initial policies and references.

Critical weaknesses

Overall, I find the paper very difficult to follow. Concepts are often used before they are defined. The experimental setup is where "inaccurate state descriptions" are first explained in concrete terms: the text descriptions provided to Qwen are altered, while the learner's observations remain unchanged. Understanding the relationship between these inputs also requires consulting the appendix. There are many such issues, which seriously hinder understanding of the paper. As for the experimental analysis, the benefits of matching guidance to states are largely expected and intuitive. However, the same-top-action reassignment experiments seem only to show that the top recommendation is important. Does the rest of the probability distribution provide any additional benefit? When is it worth constructing an additional reference, and when is retaining or copying an existing BC policy sufficient?

Questions for authors
  1. What impact does swapping left and right in the descriptions actually have on guidance quality? In the paper, 0.9 is the probability of applying left/right swapping to a description string, not an action-error rate. Could the authors report how Qwen's top recommendation, action distribution, or agreement with the planner changes between clean and corrupted descriptions of the same set of states (please also make sure to define these terms)?
  2. During same-top-action reassignment, which preserves the top recommendation, how much do the individual action probabilities actually change? I could not find an experiment addressing this.
  3. Are there any existing comparisons between a full-position expert reference and a Frozen-BC reference with matched offline fitting coverage and training settings?
3 comments
AuthorsOpenReview ↗

Inputs, guidance diagnostics and matched reference comparisons

Thank you for the careful review.

We clarify inputs, measure advice changes, and test top-only guidance and matched fitting.

New marks completed rebuttal experiments; citations refer to the submission. AUC is time-averaged success (fraction); gaps are percentage points and brackets confidence intervals.

1. Concepts used before definition (weakness)

The BabyAI learner uses a symbolic partial view, instruction and recurrent memory. Qwen receives text descriptions/history and hints about feasible actions and targets. Before constructing Qwen's context, each description string has probability .9 of a left/right-word swap. Learner observations, environment, action meanings and demonstration labels stay unchanged (Section 4, p. 6). This is not an action-error rate.

Behavior cloning (BC) fits demonstration actions. PPO learns from rewards with temporary KL matching to guide probabilities on learner states and continuing demonstration supervision (anchoring). The guide is online Qwen or a frozen reference fitted to action labels or probabilities. Guide KL ends; anchoring continues; only the policy is deployed (Sections 3.1-3.3, pp. 2-4; schedules, p. 5).

We will introduce guide inputs, left/right corruption, Aligned/Reassigned and anchoring in Section 1 of the final manuscript, and make Section 4 self-contained.

Q1: How swapping changes guidance quality

New diagnostic: 3,000 paired states, five policies, three training stages, five task families; 6,000 valid queries, none used for training. Physical states/action histories, model and probability reconstruction match; text histories evolve separately.

The top recommendation is the most probable action; planner agreement is its match rate with the planner's action. Total variation, TV(p,q)=12∑a∣p(a)−q(a)∣\mathrm{TV}(p,q)=\frac12\sum_a|p(a)-q(a)|, measures change between six-action distributions p,qp,q (here clean/corrupted); zero means identical.

Planner agreement falls 56.83% to 39.43%, or -17.40 [-20.18,-14.34] points. The top action changes on 44.17% [41.84,46.43] of states; mean TV is .42795 [.40601,.44928]. Descriptive 95% intervals resample paired episodes. Planner disagreement need not be an error, and changed advice alone does not establish worse learning.

Q2: How much do individual probabilities change?

New diagnostic: 640 rollouts, 20 Aligned policies, five pipelines/four seeds. Aligned uses each state's own target. We evaluate both exchanges on the same visits, each having another state with the same instruction and top action. Same-top donors share both; unrestricted donors need only share the instruction.

Mean TV is .10967 [.09951,.11984] same-top versus .49840 [.46998,.52682] unrestricted. Same-top absolute changes average .09024 at the top action and .12911 summed over others; 24.76% of eligible visits have TV below 10−1210^{-12}. Descriptive 95% intervals use five pipeline means. The weaker perturbation limits conclusions about other probabilities from Table 4's small gap (p. 8).

Learning benefit beyond the top recommendation (weakness)

New direct learning test: five reused pipelines/four new paired seeds; matched initialization, PPO, anchoring and 1.5M-step budgets. Each arm queries its own learner states. Top-only assigns 5/6 to the top action and 1/30 to each other, then uses the shared scheduled power and normalization (p. 5). Smoothing was selected on excluded development data.

Full/top-only AUC is .22962/.22042: +.92 [-.97,2.81] (primary 97.5%, two-comparison adjustment over five seed-averaged pipeline means). Full-vector advantage remains unresolved.

Q3 and reference-value concern: matched fitting coverage

Table 6 (p. 9) did not match coverage. New test: 15 full-position expert-reference policies versus 15 existing Frozen-BC (fixed initializer copy) policies, five CrafText pipelines/three seeds. Both fits use all steps with matched budgets/selection, PPO initialization, critic treatment, anchoring and KL schedule. BC fits sampled expert actions; the reference fits expert probabilities.

Expert/Frozen-BC AUC is .31902/.32432: -.53 [-1.55,.49] (primary 95%, five seed-averaged pipeline means). Extra-fit benefit remains unresolved. Frozen-BC guides PPO rather than deploying BC. Original main-study Aligned AUC .2956 is below retained BC .3118 (Section 5.2, p. 7).

We will add these tests. Table 2 (p. 7; uncorrupted inputs, fitted references) gives Aligned/Anchor-only/retained-BC AUC .23167/.16638/.15110: +6.53 [3.71,9.35] and +8.06 [5.87,10.25] (secondary 95%, five independent pipelines, all positive). Anchor only omits guidance; retained BC omits PPO. Exchange preserves each instruction's target collection within a rollout but changes state assignments (Section 3.4, p. 4), reducing AUC to .18544: +4.62 [2.65,6.60] for own-state targets (prespecified 95%). These support training benefit and state-specific guidance beyond instruction-level action tendencies.

Reviewer_RHS5OpenReview ↗

Response to authors

Thank you for the detailed response and targeted additional experiments. These substantially clarify my questions about guidance corruption, same-top-action reassignment, and matched reference fitting. However, the additional benefits of full-distribution guidance and separately fitted references remain unestablished. I therefore raise my rating from 1 to 2, while retaining concerns about the strength of the contribution.

AuthorsOpenReview ↗

Follow-up on the contribution assessment

Thank you for reconsidering the submission and for confirming that the additional experiments substantially clarify your questions. We understand that your remaining concern is the strength of the contribution.

The positive contribution we emphasize is the combined training scheme’s replicated learning benefit: gains of 6.53 and 8.06 percentage points over anchoring alone and retained BC, respectively, across five independent pipelines (original Table 2, p. 7), together with controlled evidence about state-specific guidance.

Is there a specific remaining concern about these supported contributions that we could address during the remaining discussion?

Reviewer_tHk4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies instruction-following RL with a frozen training-time reference and continued demonstration supervision. A BC-initialized policy is trained with PPO, an annealed reference KL loss and a demonstration anchor. Experiments in BabyAI and CrafText compare own-state guidance with reassigned targets, different reference sources and retaining the initial policy.

Critical strengths
  1. Reassigning targets while preserving their collection, followed by the same-top-action control, provides a useful way to study what the guidance contributes.
  2. I appreciate the transparent reporting of negative results and the use of independent data/model pipelines for uncertainty estimates. The distinction between final success and time-averaged success is also important here.
Critical weaknesses
  1. My main concern is the added value of the full reference distribution beyond its top-action recommendation. In BabyAI, Aligned exceeds unrestricted Reassigned by 5.13 points, but exceeds same-top-action reassignment by only 0.72 points, with an interval spanning zero (Table 4). This leaves the extra distributional benefit unresolved; it does not prove that soft probabilities are useless. Different learner state distributions also prevent treating these gaps as a clean additive decomposition.
  2. Frozen-BC performs best in the CrafText mixture-study cohort, but the expert and mixture references use fewer fitting positions (Table 6). This is not a matched comparison of reference types. In the separate main cohort, Aligned also has lower time-averaged success than retained BC. These results make the practical incremental benefit of a separately fitted reference unclear.
Questions for authors
  1. Can you compare Aligned with top-action-only guidance on the same learner-visited states, holding PPO, anchoring, and training budget fixed?
  2. Can you compare Frozen-BC with a full-position expert reference in the same CrafText cohort, with matched initialization, critic treatment, and KL schedule?
  3. Can a matched clean-versus-corrupted experiment across multiple initializers identify whether anchoring specifically reduces harm from incorrect guidance?
2 comments
AuthorsOpenReview ↗

Direct tests of target probabilities, reference fitting and corruption

Thank you for the careful review and for recognizing the controlled comparisons.

The review raises three issues: full probabilities beyond the top recommendation, the value of another reference fit, and corruption-specific protection. We ran a direct top-only test for W1/Q1, a coverage-matched comparison for W2/Q2, and a five-pipeline clean/corrupted test for Q3.

Citations refer to the original submission; New marks completed rebuttal experiments. AUC is time-averaged success (fraction); gaps are percentage points and brackets confidence intervals.

1. Full reference distribution beyond its top recommendation (W1/Q1)

Your first weakness concern and Q1 ask what full probabilities add beyond the top action. The new top-only test below addresses this.

New direct test: five existing data/BC/reference pipelines, four new paired seeds, 20 policies per condition. Initialization, PPO, anchoring and 1.5M-step budgets match. Each arm queries its own learner states. Top-only assigns 5/6 to the most probable action and 1/30 to each other, then applies the shared scheduled power and renormalization (p. 5). Smoothing was selected on excluded development data to keep policy-to-target KL finite.

Full/top-only AUC is .22962/.22042: +.92 [-.97,2.81] (primary 97.5% Student-t interval, adjusted for two comparisons over five seed-averaged pipeline means). Full-vector superiority remains uncertain. As you note, Table 4's 5.13/.72-point unrestricted/same-top gaps (p. 8) are not additive: later learner trajectories differ, and same-top exchange retains whole vectors.

Exchange preserves instruction-level target vectors but changes their state assignments (Section 3.4, p. 4). This preservation holds within each rollout. CrafText favors own-state over exchanged targets: .29232/.27478 AUC, +1.75 [.62,2.89] (Table 3, p. 8; primary 95%, five pipelines), with exchange KL weight rescaled using mean losses (p. 5).

2. Separately fitted reference versus Frozen-BC (W2/Q2)

Your second weakness concern and Q2 ask for matched fitting coverage. Table 6 (p. 9) did not provide it; the new test below does. Frozen-BC is a fixed copy of the initializer.

New matched test: expert probabilities supervise all 64 steps. Fifteen new expert-reference policies share five pipelines/three seeds with 15 existing Frozen-BC and 15 Anchor-only policies. Data, architecture, fitting budget/selection, PPO initialization, zeroed final critic layer, anchoring, KL schedule and evaluation match. BC fits sampled expert actions; the reference fits expert probabilities.

Expert/Frozen-BC/Anchor-only AUC is .31902/.32432/.30429. Expert minus Frozen-BC is -.53 [-1.55,.49] (primary 95%); expert minus Anchor only is +1.47 [.44,2.51] (exploratory 95%), over five seed-averaged pipeline means. An extra-fit advantage is not established, nor is equivalence.

Frozen-BC guides continued PPO with anchoring; retained BC omits training (.29123 in this cohort, p. 9). The separate main cohort's Aligned AUC .2956 remains below retained BC .3118 despite final success .3241 (Section 5.2, p. 7).

3. Does anchoring specifically reduce corruption-related harm? (Q3)

New matched test: 25 policies, five independent data/BC pipelines, one new paired PPO seed each. Clean/swapped Qwen is crossed with anchor on/off, plus no-guide anchoring. Initialization, demonstrations, PPO, KL schedules, 1.5M-step budgets and evaluation match within pipeline. Only Qwen's text is corrupted; Table 1's clean/corrupted comparison was unmatched (p. 7).

Guide inputNo anchor AUCAnchor AUC
Clean.15143.22850
Swapped.14720.22694
No guide--.17180

The interaction directly tests Q3: corrupted-minus-clean AUC with anchoring minus that without, +.27 [-6.92,7.46] (primary 97.5%, five pipeline differences). Corruption-specific protection remains inconclusive.

Anchoring gains are +7.71 [2.34,13.08] clean and +7.97 [5.06,10.89] swapped (secondary 95%): benefits within each condition, not evidence of corruption-specific protection. Unanchored corruption changes AUC by -.42 [-4.40,3.55] (secondary 95%). Separately, corrupted-guide versus no-guide anchoring gives +5.51 [-2.09,13.12] (the other primary 97.5% contrast). Both primary intervals use Student-t with two-comparison adjustment. One seed per pipeline limits precision; "--" means not run.

We have now run the requested top-only, matched-coverage and multi-initializer comparisons and will add them in the final manuscript. They leave the stronger claims open. The supported training benefit remains: original Table 2 (p. 7) gives Aligned/Anchor-only/retained-BC AUC .23167/.16638/.15110, gains +6.53 [3.71,9.35] and +8.06 [5.87,10.25] (secondary 95%, five independent pipelines). Together with the assignment controls, this supports the training scheme and evidence about how guidance is transferred.

AuthorsOpenReview ↗

Follow-up on the additional comparisons

Thank you again for your review. With the discussion deadline approaching, we would appreciate your assessment of the additional experiments in our response: the direct top-action-only test, the coverage-matched expert versus Frozen-BC comparison, and the five-pipeline clean/corrupted-guidance study.

Our response connects these comparisons to the paper’s positive contribution: replicated BabyAI learning gains and controlled evidence about assigning guidance to learner states. Could you let us know which concerns remain after these additions, particularly regarding the value of these findings? We would be glad to provide further clarification before the discussion closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.