acceptodds
Under review as a conference paper at ICLR 2027

Feedback Manipulation Regularization: Enabling Offline Agent Alignment for Imitation Learning

Abstract

Reinforcement learning (RL) research has begun to shift focus toward alignment, ensuring agents learn behaviors that adhere to human values. While human demonstrations and feedback have proven crucial for alignment, existing approaches that combine these signals are largely designed for the contextual bandit framing of language generation. Within this framing, recent work has started moving beyond multi-stage pipelines toward fully offline, single-stage pipelines that can reduce computational overhead and remove the need for environment access, broadening alignment's applicability. Yet little work extends such pipelines to sequential decision-making settings, where these complementary inputs can serve as a richer, interconnected signal. We propose Feedback Manipulation Regularization (FMR), an algorithm-agnostic method that harnesses evaluative feedback as a corrective signal to improve the alignment of imitation learning policies. We adapt Safety Gymnasium environments as a principled testbed for alignment evaluation, on which FMR reduces misalignment by up to 98% across a range of imitation learning algorithms while often improving aptitude. FMR remains robust in limited-data regimes, even when demonstrations become predominantly noisy and uninformative.

Primary area:
reinforcement learning
Keywords:
imitation learning, alignment, learning from feedback, learning from noisy demonstrations, interactive reinforcement learning, reinforcement learning, machine learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_zJwAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to study the alignment problem between imitation learning polices and human feedback and preference. The authors propose Feedback Manipulation Regularization, which as a corrective term to improve the alignment quality. Evaluations on Safety Gym demonstrates the superior performance of the proposed method, FMR

Critical strengths
  1. This paper studies the alignment problem between imitation learning robot policies and human preference, which is an important and timely topic;

  2. The authors provide theoretical analysis for their proposed method FMR.

Critical weaknesses
  1. I think the authors need to compare with very simple data selection methods (keep "good" data, discard "bad" data, only imitate from "good" data)

  2. The authors need to discuss why they select the evaluation tasks. It seems that the authors add massive adaptations from Safe Gym. The authors need to justify why they select these evaluation tasks and whether their massive adaptations create the settings that do not resemble real settings and favor the proposed method.

  3. The authors need to illustrate whether the proposed method works with modern imitation learning methods such as diffusion based or flow matching based.

Questions for authors

See the weakness points I list.

2 comments

We would like to thank the reviewer for their time and feedback. Below we address both your questions and weaknesses. Any changes made have been highlighted in the paper.

[W1 Data Filtering] We agree, and we explored this idea early on. We used feedback to filter the original DID^I data, while keeping all original DED^E data in DED^E. For ReCOIL, data receiving positive feedback is moved to DED^E, while data receiving negative or no feedback remains in DID^I. For BC, which cannot learn from DID^I, data receiving positive or no feedback is moved to DED^E and data receiving negative feedback is removed, since also removing data without feedback would greatly reduce the dataset size. In both cases, performance falls short of FMR (see tables below). BC with filtering generally improves success and misalignment over baseline BC, but fails to reduce misalignment in low-overlap settings with scarce aligned data (e.g., PathBB at the 10-50 ratio). We hypothesize that this could be because filtering removes useful information, including general state-space coverage and information about undesirable behaviors. ReCOIL with filtering performs worse than its baseline in all settings, likely because misaligned data that incorrectly receives positive feedback is assigned to DED^E, where its influence is amplified. In contrast, FMR learns not only what the agent should do but, equally importantly, what it should avoid doing, which is exactly the information that filtering discards or dilutes. We have added these results to Appendix C.

PathM with demonstration filtering

AlgorithmRatioSuc.Mis.
BC50-500.446 ± 0.4970.370 ± 0.308
10-500.308 ± 0.4620.465 ± 0.288
BC Filter50-500.700 ± 0.4590.228 ± 0.317
10-500.496 ± 0.5000.386 ± 0.338
BC+FMR50-500.866 ± 0.3400.104 ± 0.225
10-500.808 ± 0.3940.148 ± 0.252
ReCOIL50-500.951 ± 0.2150.113 ± 0.208
10-500.855 ± 0.3520.150 ± 0.239
ReCOIL Filter50-500.885 ± 0.3190.203 ± 0.241
10-500.787 ± 0.4090.289 ± 0.258
ReCOIL+FMR50-500.988 ± 0.1070.015 ± 0.084
10-500.947 ± 0.2240.020 ± 0.092

PathBB with demonstration filtering

AlgorithmRatioSuc.Mis.
BC50-500.463 ± 0.4990.308 ± 0.325
10-500.248 ± 0.4320.440 ± 0.300
BC Filter50-500.652 ± 0.4760.266 ± 0.402
10-500.354 ± 0.4780.476 ± 0.418
BC+FMR50-500.713 ± 0.4520.162 ± 0.270
10-500.474 ± 0.4990.333 ± 0.344
ReCOIL50-500.780 ± 0.4150.153 ± 0.305
10-500.592 ± 0.4920.237 ± 0.334
ReCOIL Filter50-500.598 ± 0.4900.225 ± 0.336
10-500.458 ± 0.4980.339 ± 0.375
ReCOIL+FMR50-500.919 ± 0.2730.048 ± 0.167
10-500.708 ± 0.4550.097 ± 0.211

[W2 Environment Selection] Our task selection and modifications are motivated in Section 5.1 and Appendix B: Safety Gymnasium’s cost signals provide a principled measure of misalignment, which we use only for evaluation. Since the original navigation tasks do not define a specific intended behavior beyond avoiding hazards, we modified them only to create clear alignment scenarios in which a single designated path to the goal is aligned. For the velocity tasks, we slightly adjusted the velocity thresholds for certain tasks to allow room for distinct gait patterns, as we found some of the preset thresholds too restrictive. Both modifications were made to provide clearer and more accurate alignment evaluation, and they do not favor FMR or any other method, as hazards are unobservable to the agent and all methods are trained on the same demonstrations and feedback.

[W3 Algorithm Selection] Our selected baselines span classical imitation learning methods (BC, IQL) to prominent methods designed specifically for learning from noisy demonstrations (DemoDICE, ReCOIL), which reflects the setting of this work. Adapting FMR to diffusion and flow-matching policies is likely less straightforward. We believe this is a interesting direction that warrants its own investigation in future work.

Thanks for the additional results. My concerns have been solved. As I already indicate positive scores, I will keep it.

Reviewer_Zx3SOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Feedback Manipulation Regularization (FMR), which adds a regularization term to offline imitation learning from trusted and imperfect demonstrations. It uses local evaluative feedback to encourage acceptable behavior and discourage undesirable actions. The authors evaluate FMR with several imitation learners on discrete-action navigation and locomotion tasks, measuring both task performance and misalignment. The experiments examine whether FMR improves the underlying learners and how it compares with alternative ways of using the feedback.

Critical strengths
  • The paper addresses the worthwhile question of whether local evaluative feedback can help recover useful behavior from imperfect demonstrations. These demonstrations can contain useful examples alongside unwanted behavior, and the paper proposes a concrete way to use that information selectively.

  • The experimental coverage is substantial, spanning several learning algorithms, task settings, and demonstration ratios. The authors also examine feedback coverage, credit-assignment windows, and hyperparameter sensitivity. Including expert-only baselines provides a useful reference for assessing what the additional imperfect demonstrations and feedback contribute.

  • Reporting task performance and misalignment separately makes the evaluation more informative. It distinguishes completing the task from following the desired behavior and makes trade-offs between these objectives visible.

Critical weaknesses
  • The practical benefit of adding imperfect demonstrations and evaluative feedback is not consistently established relative to using trusted demonstrations alone. BC trained on only 10 trusted demonstrations outperforms ReCOIL+FMR trained on 10 trusted and 50 imperfect demonstrations with feedback on both task performance and misalignment in PathBB and SlowWalk, based on the reported means. BC also has lower misalignment in SlowHop. The SlowSwim difference is tiny, while PathM shows a clear benefit from FMR. The limitation is acknowledged on page 9, but the numerical comparisons appear only in Tables 13–17 on pages 31–32. These results are central to assessing what the additional data and feedback actually provide.

  • The experimental presentation is a major weakness. I had to reconstruct the setup and results across the main text and appendices to assess the claims. PathM and PathBB use the same navigation task with different hazard layouts, but the illustrations needed to understand their routes and hazards are deferred to Figures 4–5 on page 16. SlowSwim, SlowHop, and SlowWalk receive brief descriptions of their gaits and velocity limits, followed by substantial discussion of their results in Sections 5.2 and 5.3. Yet Figures 2 and 3 show only navigation. The corresponding locomotion comparisons are in Figures 13 and 17 on pages 22 and 25. These are three of the four environment types evaluated, so leaving their illustrations and comparative plots outside the main text makes a substantial part of the experimental argument difficult to follow.

  • Figure 1 is introduced before the information needed to interpret it. Its caption refers to PathM, a 10-50 data ratio, and misalignment before explaining the route, what the demonstration counts represent, or how misalignment is measured. It is referenced in the introduction, but the experimental section never explicitly returns to it and connects it to the setup. I had to work backwards from later definitions and appendix illustrations to understand a figure presented at the start as evidence for the paper’s contribution.

  • The navigation comparisons favor FMR, but the locomotion results do not support the same unqualified conclusion. In Table 9 on page 26, CPL has lower mean SlowHop misalignment at both 50-50 and 25-50. At 50-50, it also has the same rounded mean return as ReCOIL+FMR. At 10-50, FMR achieves lower misalignment at the cost of lower return. Section 5.3 acknowledges that CPL is competitive, but still concludes that FMR consistently achieves the best trade-off. The clearer navigation wins appear in the main figures, while these qualifications receive less attention and require consulting the appendix.

  • The headline reductions, including "up to 98%" need clearer numerical context. This claim appears in both the abstract and Section 5.2, but the main text does not show the absolute baseline and final misalignment values for that comparison. A large relative reduction does not by itself show the size of the original problem. The relevant baseline and data ratio should be identifiable alongside the associated task performance.

  • The claim that evaluative feedback is less ambiguous than preferences conflates the specificity of the feedback with the evaluator’s ability to provide it reliably. An evaluator may know which behavior is better without knowing whether either is acceptable, or which actions deserve credit or blame. Asking for absolute judgments about local behavior can therefore demand more knowledge from the evaluator. FMR assumes access to these judgments alongside an identified trusted demonstration set, but the experiments do not establish that evaluators can provide these judgments with less ambiguity or greater reliability.

  • The KL derivation does not justify the final regularizer as written. The normalized KL contains a policy-dependent log-normalizer, while the standard generalized KL for unnormalized measures contains a policy-dependent mass correction. Neither can generally be absorbed into a constant. The final penalty remains usable as a surrogate, but Corollary 4.7 needs a corrected derivation or a precise alternative definition. The statement in Remark 4.8 that negative feedback increases entropy and positive feedback decreases it is also not generally valid.

  • Appendix A.2 acknowledges that critic-value scale can weaken feedback modulation, but the paper does not test this sensitivity or explain how it is controlled across algorithms. Multiplying the values by a positive constant preserves their ranking but can make the softmax distribution overly concentrated, potentially reducing the influence of feedback during training. A rescaling experiment and an explanation of any value normalization would help establish how robust FMR is to this issue.

  • All experiments use discrete actions, including the locomotion tasks. The proposed modification for continuous actions is evaluated only in discrete settings. Calling the extension straightforward goes beyond the evidence provided.

Questions for authors
  • How do you interpret the practical benefit of adding imperfect demonstrations and feedback when expert-only BC has better reported means on both metrics in PathBB and SlowWalk? Are expert-only comparisons available with 25 and 50 trusted demonstrations, beyond the comparison using 10?

  • What supports the claim that ReCOIL+FMR consistently achieves the best trade-off in Section 5.3, given the SlowHop results in Table 9? In particular, CPL has lower mean misalignment at 50-50 with the same rounded mean return.

  • Which definition of generalized KL is used in Corollary 4.7, and how are the policy-dependent normalization or mass terms accounted for? Under what conditions does the entropy claim in Remark 4.8 hold?

  • How is critic-value scale controlled before applying softmax? Have you tested whether rescaling values while preserving their ranking changes FMR’s effectiveness?

2 comments

We would like to thank the reviewer for their time and feedback. Below we address both your questions and weaknesses. Any changes made have been highlighted in the paper.

[W2 Velocity Plots] Due to the page limit, the velocity result plots were relegated to the appendix. With the additional page now available, we have moved them back into the main paper. If space permits, we will also move other figures, such as the path illustrations, into the main text; otherwise, the main paper will reference them in the appendix.

[W3 Figure 1] We included Figure 1 as a teaser to give readers an immediate visual summary of the misalignment of existing methods and the improvement from FMR. The first half of the caption conveys the high-level takeaway, while the latter half provides detail for readers who return to the figure. Additionally, the full PathM results, including these curves, appear in Appendix C.1.

[W5 Reduction Percentages] The 98% figure corresponds to the largest reduction in misalignment, obtained by DemoDICE+FMR on SlowSwim at the 50-50 data ratio, where misalignment drops from 0.04 to 0.001, giving (0.04−0.001)/0.04=0.975≈98%(0.04 - 0.001)/0.04 = 0.975 \approx 98\%. While the baseline misalignment is small, reducing an already-low rate to near zero is nontrivial, as the remaining violations tend to be rare and subtle. All other reductions can be computed from Tables 1–5 in the appendix.

[W6 Preferences] Our claim refers to the relational ambiguity of preferences, where a preferred behavior is not necessarily an aligned one. Evaluative feedback removes this by asking whether a behavior is aligned rather than which of two is better. There is no guarantee that evaluators provide evaluative feedback more reliably, and how much knowledge the evaluator needs varies by task (i.e., complexity of aligned policy). However, we see these as general challenges of human evaluation that are shared by learning from preferences. FMR also does not require knowledge of the correct action or feedback at every instance. If an evaluator is unsure, they can simply provide no feedback. Finally, the assumption of an expert demonstration set follows standard learning-from-noisy-demonstrations settings (e.g., ReCOIL, DemoDICE), and we show this set can be small (e.g., the 10-50 ratio).

[W9 Continuous Actions] By "straightforward" we refer to the adaptation of the method itself. Extending FMR to continuous actions requires no structural changes. The generalized temperature τgen\tau^\text{gen} adjusts only the selected action so positive feedback no longer requires adjusting non-selected actions. This removes the need to enumerate the action space. Appendix~C shows that this generalized form performs on par with the original formulation, indicating that the modification required for continuous actions does not degrade FMR’s original effectiveness. Evaluating FMR in continuous-control benchmarks is a natural next step.

[Q1/W1 Imperfect Demonstrations] We agree that in low-overlap settings like PathBB and SlowWalk, training on expert data alone is stronger, as expected. As Section 6 discusses, when imperfect demonstrations rarely contain the aligned behavior, most of that data is irrelevant to learning it. We do not argue that imperfect data should always be added. Rather, it is often unavoidable, so the goal is to learn effectively when it is present, motivating the idea of learning from noisy demonstrations. Since overlap is rarely known in advance, FMR makes including imperfect data a safer choice. With high overlap, ReCOIL+FMR outperforming expert-only BC/IQL on PathM suggests that feedback turns aligned sub-trajectories into useful signal. With low overlap, FMR still greatly reduces base ReCOIL’s misalignment, even if it does not match expert-only performance. For discussions and results on why filtering trusted (i.e., aligned) demonstrations only is insufficient, see our response to Reviewer zJwA’s W1.

[Q2/W4 CPL Performance] We agree that CPL achieves slightly lower misalignment than ReCOIL+FMR on SlowHop at the 50-50 and 25-50 ratios, and the two are comparable on SlowSwim. Our claim refers to consistency across all tasks. CPL is competitive only on these two velocity tasks, while ReCOIL+FMR outperforms it on PathM, PathBB, and SlowWalk at every ratio, with the gap generally growing as aligned data becomes scarcer (e.g., 10-50).

[Q3/W7 FMR Theory] We agree that the original presentation was imprecise. Our intent was to motivate FMR from the reverse-KL decomposition in Proposition 4.6 rather than to claim an exact KL-equivalent objective. As the reviewer notes, both the normalized reverse-KL and the generalized KL contain policy-dependent normalization or mass-correction terms that cannot be absorbed into a constant. We have revised Corollary 4.7 (now Definition 4.7) to remove the generalized KL formulation and instead define FMR explicitly as a surrogate regularizer motivated by Proposition 4.6. FMR retains the feedback-dependent term Ea∼πθ(⋅∣s)[log⁡τ(s,a)]\mathbb{E}_{a\sim\pi_\theta(\cdot\mid s)}[\log\tau(s,a)] and omits the policy-dependent normalization term log⁡Zθ(s)\log Z_\theta(s).

We also agree that the entropy statement in Remark 4.8 was generalization. The revised remark states that negative feedback increases entropy and positive feedback decreases it when the selected action is the most probable action under the policy, which is typical as the policy fits the demonstrations. Sufficiently large adjustments, or adjustments to less probable actions, may affect entropy in either direction. Empirically, this feedback-driven modulation of entropy is visible in the state-space entropy heatmaps in Appendix C.

[Q4/W8 Critic Scaling] Aside from FMR’s own feedback-adaptive temperature scaling, critic values are not rescaled before the softmax. The appendix note is a warning about how temperature scaling interacts with large logits. Because FMR’s reweighting amounts to a fixed logit shift of log⁡τ(s,a)\log \tau(s,a), large critic values produce near-deterministic distributions on which feedback has little effect, so reducing the critic’s scale could make the probabilities easier to manipulate through feedback. We have not tested ranking-preserving rescaling directly, but FMR’s consistent gains across critic-based methods suggest critic magnitudes in our tasks remained within an effective range.

Reviewer_mcwyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Feedback Manipulation Regularization (FMR), a method for incorporating evaluative feedback into offline imitation learning. FMR uses positive and negative feedback on demonstrated actions to construct a probability-based regularizer that encourages desirable actions and suppresses undesirable ones. The method is evaluated with BC, IQ-Learn, DemoDICE, and ReCOIL on navigation and locomotion tasks with mixed-quality demonstrations. The results show reduced constraint violations while often maintaining or improving task performance.

Critical strengths
  1. The paper addresses a practical problem: learning from imperfect demonstrations when feedback can identify undesirable behavior without specifying the correct alternative action.

  2. FMR is simple to incorporate into existing imitation learning objectives, and its compatibility is evaluated across several algorithmic families.

  3. The experiments assess both task performance and misalignment, with additional analyses of demonstration quality, feedback coverage, and hyperparameter sensitivity.

Critical weaknesses
  1. My main concern is the comparison with the reward-based baseline. How is the feedback converted into rewards for DVL? Is it rescaled, normalized, or clipped after credit assignment? Since repeated feedback can accumulate, its scale may affect training. The paper studies FMR’s hyperparameters, but it is unclear whether the reward scale for DVL was similarly tuned. Could the authors provide a reward-scale sensitivity analysis to clarify whether DVL’s weaker performance comes from treating feedback as reward or from the chosen configuration?

  2. The alignment settings seem relatively limited. The desired behaviors are explicitly defined, such as following a specified path or staying below a velocity threshold, and feedback for the locomotion tasks is generated from environment costs. How would FMR perform when feedback is noisy, inconsistent, or conflicts with the long-term task objective? Could the authors provide additional experiments or discuss how these controlled settings support the broader alignment claims?

  3. I am confused about the relationship between the KL formulation and the final surrogate. Proposition 4.6 includes a log⁡Zθ(s)\log Z_\theta(s) term, but this term is omitted from the FMR regularizer. Since Zθ(s)Z_\theta(s) depends on the policy parameters, why can it be removed?

Questions for authors

see weakness

2 comments

We would like to thank the reviewer for their time and feedback. Below we address both your questions and weaknesses. Any changes made have been highlighted in the paper.

[W1 DVL Baseline] For DVL, feedback-derived rewards were used directly without rescaling, normalization, or clipping, and are bounded by the number of steps in the credit assignment window (9, 18, and 30 for the evaluated window sizes). Even so, the feedback scale for tasks does not vary greatly within those bounds. Following the original implementation, DVL already includes safeguards against large values, namely a Huber critic loss and clipped advantage weights in the actor update, so additional reward clipping was not required for stable training. Regardless, we did attempt naive feedback clipping for the navigation tasks which clipped accumulated feedback (after credit assignment) to either -1 or 1. This effectively removes magnitude and lead to a slight decrease in performance for DVL (see the below table).

PathM comparison with clipping DVL feedback (i.e., reward)

AlgorithmRatioSuc.Mis.
DVL50-500.773 ± 0.4190.148 ± 0.260
10-500.520 ± 0.5000.308 ± 0.302
DVL Clip50-500.711 ± 0.4530.185 ± 0.278
10-500.432 ± 0.4960.371 ± 0.302
ReCOIL+FMR50-500.988 ± 0.1070.015 ± 0.084
10-500.947 ± 0.2240.020 ± 0.092

PathBB comparison with clipping DVL feedback (i.e., reward)

AlgorithmRatioSuc.Mis.
DVL50-500.771 ± 0.4200.108 ± 0.222
10-500.400 ± 0.4900.287 ± 0.286
DVL Clip50-500.730 ± 0.4440.130 ± 0.252
10-500.370 ± 0.4830.324 ± 0.301
ReCOIL+FMR50-500.919 ± 0.2730.048 ± 0.167
10-500.708 ± 0.4550.097 ± 0.211

For DVL, we use the hyperparameters recommended in the original paper. Because DVL and ReCOIL share the same underlying dual-RL formulation, differing mainly in that DVL learns from reward while ReCOIL imitates expert demonstrations, we maintained parity between their hyperparameters to ensure a fair comparison. As one example, to test the sensitivity of this choice, we varied the action temperature and found that raising it above 0.1 improves DVL’s performance. However, it improves ReCOIL’s performance as well, so the gap between the two algorithms remains, and the relative ranking is unchanged. See the results for PathBB using 10-50 data ratio below.

PathBB comparison of action temperatures (AT)

AlgorithmRatioSuc.Mis.
ReCOIL+FMR AT=0.110-500.708 ± 0.4550.097 ± 0.211
ReCOIL+FMR AT=0.510-500.758 ± 0.4280.072 ± 0.217
DVL AT=0.110-500.400 ± 0.4900.287 ± 0.286
DVL AT=0.310-500.449 ± 0.4980.239 ± 0.288
DVL AT=0.510-500.466 ± 0.4990.189 ± 0.259
DVl AT=0.7510-500.422 ± 0.4940.167 ± 0.248

These results are not currently included in the appendix, but we would be happy to add them if the reviewer finds them helpful.

[W2 Alignment Setting and Noisy Feedback] Alignment is challenging to evaluate, especially quantitatively in sequential decision-making environments; how to properly quantify alignment remains largely underexplored in the literature (Section 2) and is often task- and environment-dependent. As such, we adapted Safe RL environments and designed tasks that clearly distinguish aligned from misaligned behavior, while still introducing ambiguity through imperfect demonstrations that contain both aligned and misaligned sub-trajectories. For example, as the reviewer notes, aligned behavior in navigation centers on following a specific path while avoiding distractor behaviors such as circling the goal. Likewise, the velocity tasks use velocity constraints as a proxy to elicit specific gaits (i.e., a specific velocity threshold maps to a specific gait).

Regarding noisy feedback, the navigation feedback comes from a real human evaluator and is assigned to states through an approximate credit assignment window, so it is already inherently imprecise. Additionally, the feedback scalability results in Section 6, where FMR degrades gracefully as feedback coverage is reduced, may serve as a potential proxy for robustness to incomplete or inconsistent feedback. We agree that systematically studying inconsistent or conflicting feedback is an important direction for future work; however, it is essentially its own line of work, as it raises the question of how noisy feedback can be reasonably modeled and assessed. For instance, simply injecting uniform noise into feedback provides limited insight, as real human feedback errors are structured rather than random (e.g., concentrated in ambiguous states or biased toward certain behaviors), and uniform noise across overlapping states may largely average out. A more informative, though costly, approach would be to evaluate how FMR generalizes across feedback from a variety of human evaluators.

[W3 Dropping Normalization Term] We thank the reviewer for identifying this point. The KL decomposition in Proposition 4.6 indeed contains the policy-dependent normalization term log⁡Zθ(s)\log Z_\theta(s) which has a non-zero gradient ∇log⁡Zθ(s)\nabla \log Z_\theta(s). Our intent was not to claim that the FMR regularizer exactly minimizes the reverse-KL objective. Rather, FMR uses the first term Ea∼πθ(⋅∣s)[log⁡τ(s,a)]\mathbb{E}_{a\sim \pi_\theta(\cdot\mid s)}\big[\log \tau(s,a)\big] as a computationally simple surrogate that preserves the directional influence of feedback on the policy. We revised Section 4 to explicitly state that FMR is motivated by the reverse-KL decomposition but optimizes only the feedback-weighted expectation term as a surrogate regularizer.

Reviewer_uDGaOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes to regularize imitation learning algorithm (like behavior cloning) via preference alignment. This is achieved by scaling the action probability via temperature scaling according to human annotation. Specifically, a high temperature will be added if the action is discouraged, a low temperature will be added if the action is encouraged; and temperature=1 for unlabeled actions.

The proposed method was evaluated on Safety Gymnasium environments; and the outperformed its baselines in both success rate and alignment.

Critical strengths

1, The proposed method is a general regularization and can be theoretically combined with different imitation learning algorithms.

2, The method is well-motivated and described, especially Section 4.

3, In experiments, the proposed method significantly outperformed its baselines.

Critical weaknesses

1, My major concern is with the action space. Specifically, current agent actions are usually infinite and each action might contain several tokens. However, the experiments is limited to tasks with finite action space. Even the continuous space experiments are limited to the discrete-action environments. I am concerned that the proposed method cannot readily be generalized to common agent tasks.

2, Second, action-level expert annotation is very expensive, while this work only evaluates on synthetic "expert" feedbacks. That said, I am not sure how much the proposed alignment can be generalized to practical agent settings.

Questions for authors

1, In Figure 1, only the misalignment was reported between original imitation learning and their regularized counter part. Could you report the success rate as well?

2, Can the method be applied to LLM agent scenarios where the action space is natural language or structured tool calls?

2 comments

We would like to thank the reviewer for their time and feedback. Below we address both your questions and weaknesses. Any changes made have been highlighted in the paper.

[W1 Action Space] We agree this is an important consideration. The original framing of this work is centered on compatibility with humans either demonstrating or providing feedback, and humans typically operate in discrete action spaces when demonstrating, which motivated our use of discretized environments (Section 5.1). However, FMR is designed to extend beyond finite action spaces: the generalized feedback-adaptive temperature adjusts only the selected action and does not require enumerating the action space, with preliminary experiments showing negligible performance differences from the original form (Section 6, Appendix C). This suggests an extension to continuous spaces, such as traditional continuous-control RL environments, which we leave to future work.

[W2 Scalability] We would like to clarify that only the velocity tasks use synthetic feedback, while all navigation tasks use real human feedback (Section 5.1). We discuss this issue of feedback scalability in Section 6, where ReCOIL+FMR shows only a slight drop in performance at 50% feedback coverage and remains marginally above baseline ReCOIL at 20% coverage. As the volume of demonstrations increases, state-action pairs naturally overlap, rendering feedback on every individual trajectory redundant. This redundancy can be seen in the task feedback heatmap visualizations in Appendix B, where adjacent states in similar trajectories generally receive similar feedback. Consequently, we expect scaling FMR to large-data regimes to benefit from feedback generalization across similar state-action pairs, and we cite promising directions of research that could help achieve this. An alternative approach could look at supplementing real human feedback with more synthetic feedback. We believe this work shows FMR is an effective method, with feedback scaling to reduce human workload being a separate but important issue to solve in future work.

[Q1 Success in Figure 1] Figure 1 is actually a snippet from Figure 11 in the appendix which does report the success rate. The main point of this figure was to focus on misalignment, but we understand success rate is important to help put the misalignment scores in perspective. At the very least, we now mention the full results, including success rate, are located in Appendix C.

[Q2 LLM and Tool Application] In principle, FMR could be applied to LLM agents, although the effectiveness may depend heavily on how the problem is framed. FMR is specifically designed to work better in sequential decision making environments as you are trying to provide feedback to a relevant series actions. In contrast, the traditional LLM framing is a contextual bandit, where feedback is given for the complete response. A naive approach applies the signal uniformly to every token, pushing them all up or down together, which may be too coarse. Because FMR operates at the state-action level, a modified labeling interface could support finer-grained feedback, such as on individual response segments, reasoning steps, or intermediate decisions.

For tool selection, FMR can be directly applied. In a myopic setup where an agent selects one tool at a time and receives feedback on whether that selection was a good or bad, the problem maps to the discrete action space formulation used by FMR. In a series of tool selections, FMR should be still suitable, since credit assignment can propagate feedback backward to the prior selections, similar to how we credit feedback to prior states in our experiments.

Applying FMR to LLM agents poses an interesting direction for future work with challenges. A key challenge is determining how to represent the agent's state for effective feedback manipulation. Given such a representation, it is also important to understand which components of the agent's decision making process such as reasoning, planning, or tool selection, can benefit most from FMR in complex agentic settings.

I thank the authors for their detailed responses. They make sense to me. However, my concern with the continuous/natural language action space has not been fully addressed. As a result, I will keep our current positive score, but do not raise further.

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your contributions to the review process. As the author–reviewer discussion period ends on November 18, I would like to kindly remind everyone to review the authors' rebuttals and engage in the discussion.

In particular, please indicate whether your main concerns have been adequately addressed and clarify any remaining issues that may affect your final assessment.

Your feedback will be very helpful for the subsequent reviewer–AC discussion and meta-review process.

Thank you for your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.