- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies instruction-following RL with training-time guidance that may be imperfect. It proposes initializing the policy with bc, then fine-tuning it with PPO using a decaying KL penalty toward a frozen reference and continued supervision on demonstrations. Experiments in BabyAI and CrafText evaluate corrupted guidance, reference choices, and interventions that reassign reference targets across states.
Critical strengths
- The reference-property interventions are carefully designed and provide a useful empirical decomposition of what makes reference guidance effective.
Critical weaknesses
-
The paper does not formulate a general or established imperfect-guidance setting before introducing its method. Although the left/right corruption experiment is operationally specified, it represents a particular synthetic perturbation rather than a general definition of imperfect guidance. The proposed PPO, reference-KL, and demonstration-anchoring objective is not derived from explicit assumptions about which signals are reliable or how guidance may be imperfect. Instead, the paper introduces a generic training recipe and subsequently shows that it retains some effectiveness under one constructed corruption. Consequently, “learning from imperfect guidance” is not well support.
-
Behavior cloning is an integral component of the proposed pipeline rather than merely an implementation detail. However, the transition from offline BC to online PPO is non-trivial: the critic-specific branch receives no supervision during BC, and the paper reports early performance declines after PPO fine-tuning. Despite this, the core experiments do not include a from-scratch PPO ablation. It is therefore unclear whether BC improves online learning over direct PPO.
-
The motivation for demonstration anchoring is to prevent the PPO policy from drifting away from demonstrated behavior. However, the paper does not adequately distinguish this mechanism from the more conventional alternative of kl regularizing PPO toward the fixed initial BC policy. Consequently, the experiments do not establish whether anchoring is necessary or whether a standard initial-policy KL regularizer would provide the same benefit.
Questions for authors
- In the corrupted-Qwen setting, Online-Qwen + Anchor appears to outperform No-reference + Anchor (.277 vs. .199), but this contrast is not reported as a primary paired comparison and uses only one BC initializer. Could the authors provide a direct statistical test across independently constructed pipelines to establish whether imperfect guidance itself improves over removing the reference entirely?
2 comments
Follow-up on initialization, regularization and claim scope
Thank you again for your review. As the discussion deadline approaches, we would appreciate your assessment of the completed BC-versus-scratch comparison, demonstration-anchoring versus persistent BC-KL comparison, and five-pipeline corrupted-guide versus no-guide test reported in our response.
These experiments establish benefits of BC initialization at the tested online budget and provide additional evidence for the complete training scheme.
Do these additions address your concerns about initialization and the relevant alternatives? Please let us know which remaining issue is most important to your assessment so that we can respond directly.
Design rationale, initialization and alternatives to anchoring
Thank you for the review and the questions about claim scope, initialization and anchoring.
We contribute a BC-to-PPO training scheme with replicated gains and controlled tests of guidance (p. 2). We address weaknesses 1-3, then Q1. We explain the rationale and report scratch-PPO, anchor-replacement and corrupted-guide tests.
Citations refer to the submission; New marks completed rebuttal experiments. AUC is time-averaged success (fraction); gaps are percentage points and brackets confidence intervals.
1. Imperfect-guidance setting and objective rationale (weakness)
The tested intervention changes Qwen's descriptions, not the reward function, learner observations or demonstration labels (p. 6). Table 1 (p. 7) tests corrupted online guidance; Table 2 tests fitted-reference learning without corruption.
Reference KL supplies early learner-state advice. Its weight decays to zero to stop constraining reward learning; demonstration supervision continues to limit drift on demonstrated inputs (Introduction, p. 1; Eqs. (3)-(4), p. 4; schedules, p. 5). This is an empirical design, not a derivation under a general guide-error model. We will revise the title/introduction and replace the abstract's "learn from imperfect guidance" with "learn with training-time guidance and deploy without it" in the final manuscript.
2. Does BC improve online learning over direct PPO? (weakness)
To answer your second weakness, we completed a new BC-versus-scratch test: 40 plain-PPO policies, no reference or anchor, five independent data/BC pipelines, four paired seeds, two initializations. Architecture, 1.5M-step online budgets and evaluation match. Scratch randomizes actor/shared features; both arms copy the same BC-untrained critic-specific layers and reset optimizer/memory. Shared features can still change initial values.
BC/scratch AUC is .08113/.02871: +5.24 [3.55,6.93] (primary 95%, five seed-averaged pipeline means); final success is .12638/.06140. BC helps at equal online budgets, not necessarily equal total cost. Plain BC-initialized PPO remains 7.07 [5.36,8.77] AUC points below retained BC (.15180; exploratory 95%). The same five BC checkpoints were re-evaluated on new episodes; Table 2's .15110 uses the original episodes (p. 7). BC alone does not prevent decline; its cause is not isolated.
3. Demonstration anchoring versus initial-policy KL (weakness)
Anchoring fits demonstrator labels on demonstration states; persistent BC-KL matches the initializer's probabilities on learner-visited states. Both losses persist. Original Frozen-BC retains anchoring and anneals reference KL (Section 3.4, p. 4), so it does not test replacing the anchor.
New replacement test: with/without fitted guidance, replace anchoring by policy-to-frozen-BC KL. Constant weight .05 was selected before testing on excluded development data, separately with/without guidance. Five existing pipelines/four new paired seeds give 20 policies per condition; initialization, PPO, 1.5M-step budgets and evaluation match. Inputs are uncorrupted.
For your replacement question, the fixed-guidance AUC contrast below is inconclusive. Holding guidance fixed, anchor minus BC-KL is +2.19 [-.11,4.48] (primary 97.5%, two-comparison adjustment over five seed-averaged pipeline means). Final-success gain is +4.47 [2.66,6.27] (secondary 95%). Without guidance, BC-KL beats Anchor only by +2.79 [1.26,4.32] (secondary 95%).
The full scheme beats BC-KL alone by +2.96 [.78,5.14] (planned secondary, unadjusted 95%). Thus whole-scheme utility does not establish unique anchor necessity.
Q1: Does corrupted guidance improve over removing it?
Your Q1 asks for a paired test across independent pipelines; Table 1's .277/.199 comparison used one initializer. Those are final-success fractions. New matched test: 25 policies, five independent data/BC pipelines, one new paired seed each: clean/swapped Qwen crossed with anchor on/off, plus no-guide anchoring. Initialization, demonstrations, PPO, KL schedules, 1.5M-step budgets and evaluation match within pipeline.
Corrupted-guide-plus-anchor/no-guide-plus-anchor AUC is .22694/.17180: +5.51 [-2.09,13.12] (primary 97.5%, five pipeline differences). Final success is .31100/.22810, +8.29 [-.67,17.25] (secondary 95%). Added corrupted-guide benefit remains unresolved. The other primary contrast, corrupted-minus-clean AUC with anchoring minus that without, is +.27 [-6.92,7.46] (97.5%); protection is also unresolved. One seed per pipeline limits precision.
Anchoring improves corrupted-Qwen AUC from .14720 to .22694, +7.97 [5.06,10.89] (secondary 95%). This extends the anchoring observation across initializers; it does not establish protection. We will report these tests and distinctions in the final manuscript.