acceptodds
Under review as a conference paper at ICLR 2027

Inverse RL Helps Align AI by Imitating Humans

Abstract

Language model alignment aims to make model behavior reflect desirable properties such as helpfulness, safety, and instruction following. Existing methods draw supervision from expert demonstrations, preference judgments, or verifiable outcomes. Demonstrations typically guide supervised fine-tuning (SFT), while preference judgments and verifiers provide rewards for reinforcement learning (RL). This raises a question: can the demonstrations alone be used to derive an explicit, interpretable reward useful in a post-training pipeline? Motivated by inverse reinforcement learning, we introduce Projected Alignment Reward Estimated from Demonstrations (PARED). PARED learns an explicit reward from expert demonstrations over a small set of response-level features, using a lightweight discriminator that separates demonstrations from the policy's own samples in this feature space. Experiments with inference-time reranking and adversarial on-policy RL show improvements both when starting from a base model without an SFT stage and when further optimizing an SFT-trained model. Additionally, we demonstrate that PARED can be used for contextual alignment, in which a single policy learns to adapt its responses to the preferences of different audiences.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
AI alignment, large language models, reinforcement learning, imitation learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_HugVOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper provides a carefully engineered and interpretable instantiation of classical feature-based adversarial imitation learning for language-model alignment called Projected Alignment Reward Estimated from Demonstrations (PARED). PARED maps expert and policy responses into a low-dimensional feature space capturing helpfulness, harmlessness, topics, and length, and trains a logistic discriminator whose expert-likeness score serves as the reward. The reward is used for both best-of-NN reranking and KL-regularized on-policy training, with online refitting as the policy evolves. Experiments on HH-RLHF and UltraFeedback with Qwen2.5 and Gemma suggest that PARED can improve response quality, including after SFT, and support audience-conditioned alignment. The paper also shows that the interpretable feature space exposes length-based reward exploitation and that online discriminator refitting can mitigate this failure mode.

Critical strengths

This paper has two clear strengths:

  1. Offfer a clear and well-motivated research objective. The proposed approach is simple and practical, yet achieves encouraging empirical improvements, including when applied after supervised fine-tuning.

  2. The explicit feature-based reward facilitates inspection and diagnosis of reward exploitation. The length-exploitation study provides a concrete example of this benefit and demonstrates how online discriminator refitting can mitigate the observed failure mode.

Critical weaknesses

This paper has two major weaknesses which needs to clarify:

  1. Audience-specific adaptation is not directly tested. Table 3 shows gains over Instruct for both audiences. However, a policy that is simply better or more cautious overall could produce the same result without actually adapting to the audience. Also, the audience is given only as “Population: A/B” (App. F.1). If the Instruct baseline sees the same tokens, it has no way to know the user is a child, which may inflate the child-audience gains.

  2. The source of the gains is unclear. Two of the eight features are helpfulness and harmlessness scores from Gemma-3-27B, which are already strong quality signals. Compared with the direct-reward baseline, PARED is better on Qwen from SFT (85.8% vs. 73.8%), similar from Instruct (72.1% vs. 72.4%), and worse on Gemma (Table 10). Since that baseline uses fixed equal weights, it remains unclear how much of the gain comes from PARED itself rather than from the evaluator scores.

Questions for authors
  1. What system prompt does the Instruct baseline use? It would be helpful to report results when the trained policy is given the swapped audience label, and to add a baseline where Instruct is told the audience explicitly in its prompt.
  2. Could you add a discriminator that uses only the h and s features, and a direct-reward baseline with audience-specific weights tuned on validation data? Also, why does direct reward outperform PARED on Gemma?
  3. Since removing length does not hurt performance, why is it part of the default feature set? How should practitioners choose features for a new task?
3 comments

Thank you for the careful review. We’re glad you found the objective well motivated and the method practical. Your questions led us to run new experiments, reported below.

W1/Q1: Audience adaptation and the Instruct baseline. The Instruct baseline received the same opaque labels (“Population: A/B”) as the trained policies. This was deliberate, so that audience behavior would be learned from demonstrations rather than supplied by a description/suggested by the name of the audience, but we agree it left the baseline unable to infer the audience. We ran both evaluations you suggested.

Against Instruct prompted with “The user is a general adult” or “The user is a child aged 9 to 13,” results are nearly unchanged: SFT+PARED wins 85.7% (paper: 85.8%), with 91.7% on child prompts (paper: 91.5%). The opaque labels did not inflate the main gains.

The policy knows its audience only from the system-prompt tag (“Population: A” for adults, “Population: B” for children). We gave each policy every held-out prompt twice, once with the correct tag and once with the other, and judged the two responses under the true audience. A policy that ignored the tag would sit near 50%. SFT+PARED’s correctly tagged responses win 83.9% [80.5, 86.8], and under the child tag its responses are more cautious and shorter, so it adapts to the audience rather than being uniformly better. This adaptation is maintained through RL: the SFT initialization scores 78.2% [74.5, 81.5], and PARED improves quality for both audiences without eroding it (adults 73.6% -> 79.5%, children 87.8% -> 91.7% against the explicit-audience baseline).

W2/Q2: Source of the gains. We agree the helpfulness and harmlessness scores are strong signals. Our direct-reward baseline already isolates them: it optimizes exactly these two scores with fixed equal weights. With length removed, PARED uses the same two scores plus topic features and reaches 84.6% from both initializations, versus 72.4% and 73.8% for direct reward (Table 5; separate evaluation batch, Appendix E.4.3). Due to GPU capacity constraints and queues, we will not be able add the h,s-only discriminator and a tuned audience-weighted direct reward before the discussion period ends. Our hypothesis is that the key difference is not the weighting itself but the objective. A weighted direct reward, however tuned, still pushes the scores as high as possible, while the discriminator pushes the policy toward the levels found in the demonstrations. The paper shows signs of this: after SFT, direct-reward RL falls below SFT alone, while PARED improves on it (Section 4.3). We would therefore expect an h,s-only discriminator to retain part of PARED’s advantage over direct reward, with topic features accounting for the rest.

On Gemma, we don’t have a conclusive explanation. One observation is that PARED’s feature matching is much weaker there: from Instruct, feature discrepancy falls by 10% versus 69% for Qwen, and discriminator AUC stays near 0.8 (Section 4.5), suggesting a less informative reward signal. We will note this as an open question.

Q3: Length in the default feature set. One use of PARED is to match chosen aspects of the demonstrations, not only quality-related ones. Length is such an aspect: we included it so the policy would match the experts’ response length, and with online refitting its lengths stay much closer to the expert distribution than under a frozen reward (Section 4.6). Matching length is largely independent of quality, so we would not expect it to change judged results, and from SFT it does not (85.8% with length, 84.6% without). This also strengthens the main result: had including length raised the win-rate, part of the gain could reflect the judge’s known preference for longer responses [1], but since it does not, PARED’s gains are not driven by response length. For a new task, practitioners choose features for two purposes: properties that reflect the desired behavior, and properties they want the policy to match, such as length or format. Inspecting the fitted coefficients during training then helps catch features the policy exploits.

References

  • [1] Dubois, Galambosi, Liang, and Hashimoto. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators. COLM 2024

Thank you for your suggestions on testing audience adaptation, which led to new experiments. We hope the explicit-audience baseline and swapped-label test address your concerns, as they show that our main results hold and that the SFT-initialized policy adapts to the audience. We’re happy to clarify anything further before the discussion closes.

Thank you for the additional experiments. Most of my concerns has been addressed. I have raised my soundness score.

Reviewer_Yj1QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces Projected Alignment Reward Estimated from Demonstrations (PARED). PARED learns an explicit reward from expert demonstrations over a small set of response-level features, using a lightweight descriminator that separates demonstrations from the policy's own samples in this feature space. The paper conducts experiments on HH-RLHF with Qwen2.5-3B-instruct and gemma-3-4b-it; and UltraFeedback with Qwen2.5-3B-Instruct. Results show that PARED outperforms SFT, DPO, and Direct Reward.

Critical strengths
  1. The paper proposes a new approach PARED to learn from demonstrations. PARED outperforms SFT, DPO, and Direct Reward in the paper's experimental setup, demonstrating its effectiveness in adapting the model with expert examples. This could be useful when there is a handful of expert demonstration availables.
  2. The proposed method leverages the idea of using a discriminator to distinguish expert demonstration from policy samples. This method can be applied in both test-time training and RL.
Critical weaknesses
  1. The experiments are not very comprehensive. First, the main experiments are only done on HH-RLHF. It's unclear whether the proposed method still work on other scenarios (e.g., personalization and other type of alignment tasks). Second, the paper only compares with SFT, DPO, and Direct Reward. There has a lot of methods for preference learning, including DITTO, KTO, SimPO, etc, which work better than DPO on preference learning. If the contribution of the work is a new preference learning method, it shall compare with more advanced methods.
  2. The paper does not come with qualitative analysis of results or detailed analysis of why the proposed method work. Thus, it's uclear to me on what situations would the proposed method work the best. The feature space analysis is interesting, but it also makes me wonder how we shall pick the feature space if we apply the method beyond training the model to be helpful and harmless.

[1] Aligning Language Models with Demonstrated Feedback, Shaikh et al., ICLR 2025

[2] KTO: Model Alignment as Prospect Theoretic Optimization, Ethayarajh et al., PMLR 2024

[3] SimPO: Simple Preference Optimization with a Reference-Free Reward, Xia et al., NeurIPS 2024

Questions for authors
  1. The paper also compares PARED with Direct Reward. I am unsure whether the results would be a bit misleading since you only compare them on HH-RLHF dataset. Suppose the demonstrations come from more verifiable tasks (e.g., the expert demonstrates how to do a work), I am wondering if the comparison outcome still holds.
3 comments

Thank you for the review. We’re glad you see PARED as useful when only a handful of expert demonstrations exist, and that you found the feature space analysis interesting. That analysis is the core of the paper, and we think it answers several of the concerns below.

W1: Scope of experiments and baselines. Beyond HH-RLHF, the paper evaluates PARED on UltraFeedback, a different prompt distribution with demonstrations from a different model, where both PARED initializations outperform SFT (Section 4.4, Table 4), and replicates the HH-RLHF results with a second policy family, Gemma 3-4B. Regarding personalization, our main experiments are a form of group-level personalization: separate rewards are learned for adult and child audiences from their demonstrations and guide one shared, audience-conditioned policy (Section 4.3, Appendix A). We do not claim personalization to individual users, which we note as a boundary of the study.

On baselines, we would like to clarify the framing. PARED is not a preference-learning method: it learns from demonstrations alone, without preference annotations. KTO [2] and SimPO [3] are designed for preference data. Our DPO baseline uses expert demonstrations as chosen responses and policy samples as rejected ones, which is close in spirit to DITTO [1], and KTO or SimPO could be substituted into the same construction. All of these methods, like SFT, train on the full demonstration text, so adding them would refine the end-to-end quality comparison but would not test the paper’s claim: that a compact, inspectable reward learned from demonstrations can support reranking and policy optimization at competitive quality while making failures like reward exploitation diagnosable and fixable. We will add DITTO, KTO, and SimPO to the Related Work discussion.

W2: Analysis of why and when the method works. The paper includes a qualitative example (Figure 2) and two analyses of the method’s behavior: Section 4.5 shows that PARED moves the policy’s feature distribution toward the demonstrations, and Section 4.6 shows how inspecting the reward revealed a length shortcut that online refitting suppressed. Appendix B characterizes what the discriminator tests: matching of the selected features, their squares, and pairwise products. We agree the paper should state more directly when PARED is most suitable. PARED targets alignment settings where the desired behavior can be described through measurable response properties but cannot be checked by a verifier, such as how cautious an answer should be for a given audience. When a reliable verifier exists, methods that combine it with demonstrations, such as VARL [4], are a better fit. In the settings PARED targets, the goal is not only response quality but also a reward whose incentives can be inspected, so that failures like the length shortcut can be caught and corrected. We will add a short paragraph making this explicit. On choosing features for other tasks: practitioners choose features for two purposes, properties that reflect the desired behavior and properties they want the policy to match, such as length or format. Only the evaluator scores are task-specific, since the topic features are learned from the text itself, and inspecting the fitted coefficients during training helps catch features the policy exploits. Automating feature selection, for example with LLM-based or coding agents that propose and test candidate features, is a natural direction for future work.

Q1: Comparison with direct reward on verifiable tasks. We agree that the outcome would likely differ on verifiable tasks. When a reliable verifier exists, optimizing it directly is a natural choice, and PARED is not designed to replace it. Concurrent work such as VARL [4] combines verifiable rewards with a demonstration discriminator (Appendix A). PARED targets the complementary setting where no verifier exists. We also note that direct-reward RL is a strong baseline in our experiments: it matches PARED from Instruct initialization on Qwen and is stronger after SFT on Gemma (Table 10). PARED’s advantage is clearest after SFT on Qwen, where direct-reward RL falls below SFT alone, suggesting that learning how the features characterize demonstrations helps when refining an already-supervised policy.

References

  • [1] Shaikh et al. Aligning Language Models with Demonstrated Feedback. ICLR 2025.
  • [2] Ethayarajh et al. Model Alignment as Prospect Theoretic Optimization. ICML 2024.
  • [3] Meng et al. SimPO: Simple Preference Optimization with a Reference-Free Reward. NeurIPS 2024.
  • [4] Damani, Puri, Shenfeld, and Andreas. Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations. arXiv:2607.01181, 2026.

I thank the authors for the response!

On baselines, we would like to clarify the framing. PARED is not a preference-learning method: it learns from demonstrations alone, without preference annotations. KTO [2] and SimPO [3] are designed for preference data. Our DPO baseline uses expert demonstrations as chosen responses and policy samples as rejected ones, which is close in spirit to DITTO [1], and KTO or SimPO could be substituted into the same construction. All of these methods, like SFT, train on the full demonstration text, so adding them would refine the end-to-end quality comparison but would not test the paper’s claim: that a compact, inspectable reward learned from demonstrations can support reranking and policy optimization at competitive quality while making failures like reward exploitation diagnosable and fixable. We will add DITTO, KTO, and SimPO to the Related Work discussion.

I didn't fully get the rationale here. Given this, why don't you think PARED is better than DITTO or using expert demonstrations as chosen responses and policy samples as rejected ones in KTO and SimPO?

Thank you for following up. We think “better” can mean two different things here, so let us separate them.

If “better” means higher response quality, we make no claim relative to DITTO, KTO, or SimPO, since we did not compare against them. All three build on likelihood-based training over the full response text, whose standard instances are SFT and DPO, and we compare against those, with DPO trained on the same expert-versus-policy pairs. We ask whether learning the reward over a compact projection of the responses (evaluator scores, topics, length) gives up quality relative to training on the full text. We observed that PARED is competitive with both standard baselines (Table 3).

If “better” means competitive quality together with something these methods cannot provide, PARED is better because it is also inspectable. These methods yield a policy whose reward is implicit in its log-probabilities; PARED yields an explicit reward over named properties that can be inspected, reused for reranking, and corrected when the policy exploits it, as in the length study. This combination is the advantage we claim, and it does not depend on where PARED would rank against each method in response quality alone.

Reviewer_5n9LOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Projected Alignment Reward Estimated from Demonstrations (PARED) that learns an explicit reward from expert demonstration over a small set of hand-crafted response-level features. It trains a discriminator to distinguish expert demonstrations from policy-generated responses in this feature space, using the resulting classification score as a reward. Experiments on HH-RLHF and UltraFeedback report improvements over SFT and DPO baselines.

Critical strengths
  • Learning from demonstration is an important and practical research problem as collecting pairwise data is difficult.
  • The paper is easy to follow
Critical weaknesses
  • The novelty of this paper is limited. Constructing expert–policy response pairs to learn a reward function is not new and has been explored in prior work.

  • The reward model relies on a small set of manually designed features, primarily helpfulness, harmlessness, and topic information. This limits its applicability across different tasks, domains, and datasets, particularly given the diverse and subjective nature of human preferences. -The experiments mainly compare against SFT, DPO, and a simple direct-reward baseline while omitting previous baselines on learning from demonstration [1,2]

  • [1] Xiao et al. 2024. How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective. In EMNLP 2024.

  • [2] Li et al. 2025. Beyond Imitation: Recovering Dense Rewards from Demonstrations. Arxiv: 2510.02493

Questions for authors

See weakness

2 comments

We thank the reviewer for recognizing the importance of learning from demonstrations and for finding the paper easy to follow. We address each concern below.

W1: Novelty. We agree that contrasting expert and policy responses to learn a reward has clear precedents, which we discuss in Section 2 (Related Work) and Appendix A (Detailed Positioning). As Section 2 states, adversarial learning from demonstrations or use of inverse reinforcement learning is not a claimed novelty of our work. Our contribution is to constrain the learned reward to a small set of named, response-level properties. One benefit of this setup is that it makes the recovered reward inspectable, which allows practitioners to see what the reward incentivizes, detect when the policy exploits it, and correct this. The clearest example is the length study (Section 4.6): the discriminator’s coefficients revealed that it rewarded longer completions, the policy exploited this, and online refitting suppressed it. Because the reward is expressed in named features, the coefficients pointed directly to the property being exploited; with an opaque reward model, diagnosing this failure would require separate probing.

W2: Reliance on hand-designed features. We agree that a compact feature map limits what the reward can express, and we discuss this trade-off in Section 5. The restriction is deliberate: the feature map defines the scope of the learned reward, which is what makes its incentives readable and correctable during optimization, as in the length diagnosis mentioned above. The feature set itself is practitioner-chosen, and automating its selection, for example with coding agents that propose and test candidate features, is a natural direction for future work.

W3: Missing baselines. We thank the reviewer for these references. GSIL [1] learns from demonstrations through an offline self-imitation objective, and Li et al. [2] recover a dense token-level reward from the SFT policy. Both are relevant comparisons for end-to-end alignment quality, but neither produces a reward whose components can be inspected. We chose SFT and DPO as baselines because they are the most widely adopted methods for this setting and train on the full demonstration text, whereas PARED sees only the projected features. Even so, SFT-initialized PARED performs best among them on HH-RLHF (Table 3). Like SFT and DPO, [1] and [2] operate on the full demonstration text, as do most inverse-RL and adversarial imitation methods for language models [3, 4, 5, 6], which learn rewards or critics over raw text or high-dimensional model representations. Adding them would refine the end-to-end quality comparison but would not test the paper’s claim, which is that a compact, demonstration-learned reward can support both reranking and policy optimization at a quality competitive with standard baselines, while making failures like length exploitation diagnosable and fixable. We will add discussion on both of these methods to the Related Work discussion.

We hope these clarifications address the reviewer’s concerns, and we would be glad to answer any further questions.

References

  • [1] Xiao et al. How to Leverage Demonstration Data in Alignment for Large Language Model? A Self-Imitation Learning Perspective. EMNLP 2024.
  • [2] Li et al. Beyond Imitation: Recovering Dense Rewards from Demonstrations. arXiv:2510.02493, 2025.
  • [3] Wulfmeier et al. Imitating Language via Scalable Inverse Reinforcement Learning. NeurIPS 2024.
  • [4] Sun and van der Schaar. Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning. arXiv:2405.15624, 2024.
  • [5] Joselowitz et al. Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning. COLM 2025.
  • [6] Cai, Ryabinin, and Provilkov. Escaping the Verifier: Learning to Reason via Demonstrations. ICML 2026.

Thank you again for your feedback. We hope that our response clarifies that the contribution is restricting the learned reward to named, inspectable properties, not learning from expert–policy pairs as such. We’re happy to clarify anything further before the discussion closes.

Comments on the paper

1 comment

We thank all reviewers for their time and comments. We were glad that reviewers recognized the paper’s central idea. HugV writes that “the explicit feature-based reward facilitates inspection and diagnosis of reward exploitation,” with the length-exploitation study as “a concrete example of this benefit,” and Yj1Q found “the feature space analysis interesting.” Reviewers also described the paper as having “a clear and well-motivated research objective” with an approach that is “simple and practical, yet achieves encouraging empirical improvements, including when applied after supervised fine-tuning” (HugV), as “easy to follow” and addressing “an important and practical research problem” (5n9L), and as “useful when there is a handful of expert demonstrations available” (Yj1Q).

To clarify the paper’s scope: PARED is not proposed as a new preference-learning objective or as a more accurate imitation method. It learns from demonstrations alone, without preference annotations, and its contribution is the representation: a reward restricted to named response properties, which makes the reward’s incentives inspectable and correctable during optimization, and which supports separate rewards for different audiences guiding one shared policy. Our baselines and analyses are chosen to test this claim, and we discuss this choice in the individual threads.

Following HugV’s suggestions, we ran two new evaluations of audience adaptation. Our main results hold against an Instruct baseline that is told the audience explicitly, and a swapped-label test shows that SFT-initialized PARED adapts its responses to the audience rather than being uniformly better. Details are in the HugV thread, and we respond to each reviewer individually.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.