- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
This paper provides a carefully engineered and interpretable instantiation of classical feature-based adversarial imitation learning for language-model alignment called Projected Alignment Reward Estimated from Demonstrations (PARED). PARED maps expert and policy responses into a low-dimensional feature space capturing helpfulness, harmlessness, topics, and length, and trains a logistic discriminator whose expert-likeness score serves as the reward. The reward is used for both best-of- reranking and KL-regularized on-policy training, with online refitting as the policy evolves. Experiments on HH-RLHF and UltraFeedback with Qwen2.5 and Gemma suggest that PARED can improve response quality, including after SFT, and support audience-conditioned alignment. The paper also shows that the interpretable feature space exposes length-based reward exploitation and that online discriminator refitting can mitigate this failure mode.
Critical strengths
This paper has two clear strengths:
-
Offfer a clear and well-motivated research objective. The proposed approach is simple and practical, yet achieves encouraging empirical improvements, including when applied after supervised fine-tuning.
-
The explicit feature-based reward facilitates inspection and diagnosis of reward exploitation. The length-exploitation study provides a concrete example of this benefit and demonstrates how online discriminator refitting can mitigate the observed failure mode.
Critical weaknesses
This paper has two major weaknesses which needs to clarify:
-
Audience-specific adaptation is not directly tested. Table 3 shows gains over Instruct for both audiences. However, a policy that is simply better or more cautious overall could produce the same result without actually adapting to the audience. Also, the audience is given only as “Population: A/B” (App. F.1). If the Instruct baseline sees the same tokens, it has no way to know the user is a child, which may inflate the child-audience gains.
-
The source of the gains is unclear. Two of the eight features are helpfulness and harmlessness scores from Gemma-3-27B, which are already strong quality signals. Compared with the direct-reward baseline, PARED is better on Qwen from SFT (85.8% vs. 73.8%), similar from Instruct (72.1% vs. 72.4%), and worse on Gemma (Table 10). Since that baseline uses fixed equal weights, it remains unclear how much of the gain comes from PARED itself rather than from the evaluator scores.
Questions for authors
- What system prompt does the Instruct baseline use? It would be helpful to report results when the trained policy is given the swapped audience label, and to add a baseline where Instruct is told the audience explicitly in its prompt.
- Could you add a discriminator that uses only the h and s features, and a direct-reward baseline with audience-specific weights tuned on validation data? Also, why does direct reward outperform PARED on Gemma?
- Since removing length does not hurt performance, why is it part of the default feature set? How should practitioners choose features for a new task?
3 comments
Thank you for your suggestions on testing audience adaptation, which led to new experiments. We hope the explicit-audience baseline and swapped-label test address your concerns, as they show that our main results hold and that the SFT-initialized policy adapts to the audience. We’re happy to clarify anything further before the discussion closes.
Thank you for the additional experiments. Most of my concerns has been addressed. I have raised my soundness score.
Thank you for the careful review. We’re glad you found the objective well motivated and the method practical. Your questions led us to run new experiments, reported below.
W1/Q1: Audience adaptation and the Instruct baseline. The Instruct baseline received the same opaque labels (“Population: A/B”) as the trained policies. This was deliberate, so that audience behavior would be learned from demonstrations rather than supplied by a description/suggested by the name of the audience, but we agree it left the baseline unable to infer the audience. We ran both evaluations you suggested.
Against Instruct prompted with “The user is a general adult” or “The user is a child aged 9 to 13,” results are nearly unchanged: SFT+PARED wins 85.7% (paper: 85.8%), with 91.7% on child prompts (paper: 91.5%). The opaque labels did not inflate the main gains.
The policy knows its audience only from the system-prompt tag (“Population: A” for adults, “Population: B” for children). We gave each policy every held-out prompt twice, once with the correct tag and once with the other, and judged the two responses under the true audience. A policy that ignored the tag would sit near 50%. SFT+PARED’s correctly tagged responses win 83.9% [80.5, 86.8], and under the child tag its responses are more cautious and shorter, so it adapts to the audience rather than being uniformly better. This adaptation is maintained through RL: the SFT initialization scores 78.2% [74.5, 81.5], and PARED improves quality for both audiences without eroding it (adults 73.6% -> 79.5%, children 87.8% -> 91.7% against the explicit-audience baseline).
W2/Q2: Source of the gains. We agree the helpfulness and harmlessness scores are strong signals. Our direct-reward baseline already isolates them: it optimizes exactly these two scores with fixed equal weights. With length removed, PARED uses the same two scores plus topic features and reaches 84.6% from both initializations, versus 72.4% and 73.8% for direct reward (Table 5; separate evaluation batch, Appendix E.4.3). Due to GPU capacity constraints and queues, we will not be able add the h,s-only discriminator and a tuned audience-weighted direct reward before the discussion period ends. Our hypothesis is that the key difference is not the weighting itself but the objective. A weighted direct reward, however tuned, still pushes the scores as high as possible, while the discriminator pushes the policy toward the levels found in the demonstrations. The paper shows signs of this: after SFT, direct-reward RL falls below SFT alone, while PARED improves on it (Section 4.3). We would therefore expect an h,s-only discriminator to retain part of PARED’s advantage over direct reward, with topic features accounting for the rest.
On Gemma, we don’t have a conclusive explanation. One observation is that PARED’s feature matching is much weaker there: from Instruct, feature discrepancy falls by 10% versus 69% for Qwen, and discriminator AUC stays near 0.8 (Section 4.5), suggesting a less informative reward signal. We will note this as an open question.
Q3: Length in the default feature set. One use of PARED is to match chosen aspects of the demonstrations, not only quality-related ones. Length is such an aspect: we included it so the policy would match the experts’ response length, and with online refitting its lengths stay much closer to the expert distribution than under a frozen reward (Section 4.6). Matching length is largely independent of quality, so we would not expect it to change judged results, and from SFT it does not (85.8% with length, 84.6% without). This also strengthens the main result: had including length raised the win-rate, part of the gain could reflect the judge’s known preference for longer responses [1], but since it does not, PARED’s gains are not driven by response length. For a new task, practitioners choose features for two purposes: properties that reflect the desired behavior, and properties they want the policy to match, such as length or format. Inspecting the fitted coefficients during training then helps catch features the policy exploits.
References