- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the negative impact of RL, especially in persuasion scenarios. Experiments reveal that RL trains LLMs to produce misaligned fabrications as training progresses. The authors then suggest several remedies to fix this misalignment in persuasion optimization, but also show that there is a trade-off.
Critical strengths
- RL has become a common post-training method for LLM and this paper reveals RL’s negative impact for social use of LLMs. This seems to be a timely issue for researchers.
- The analysis is interesting and shows the RL’s negative impact.
- RL makes LLMs more deceptive and more likely to make fabricated claims. (Figure 2)
- Whether receiver is stubborn or not, RL increases the model’s usage of misaligned strategies. (Figure 1)
- Figure 4 and 5 show that as RL progresses models generate more obvious fabrications (model don’t know whether their claims are fabricated or not) and this becomes more strong when they get deep into the sender’s role.
Critical weaknesses
- Settings are quite far from the real world setting.
- Sender can see the receiver’s inner thoughts, which cannot happen in reality.
- Receiver’s prompt contains a phrase “the evidence they do reveal is always true.” Here “they” refers to senders, so receivers cannot doubt the fabrication. This is unrealistic in a real-world setting.
- The same model serves as both the penalty auditor during training and the evaluation auditor.
- This can train model to evade this specific auditor (judge) rather tan genuinely becoming more aligned.
Questions for authors
- Can you make the sender's and receiver's prompt without unrealistic phrase? and test the tendency is still same? (weakness 1)
- Can you set two LLM different (weakness 2) and see what happens?
3 comments
Responses to Questions
Q1. Removing the unrealistic phrase from the prompt and testing whether the tendency persists
We removed "however, the evidence they do reveal is always true" from the receiver prompt (allowing the juror to doubt uncorroborated claims) and re-trained Qwen3-4B under outcome-only RL. Outcome-only RL still drives substantial misalignment: misaligned strategies per game increase from 2.8 to 3.4, and fabricated claims per game increase from 1.24 to 6.13, consistent with the one that contains the phrase. This indicates that the phrase "however, the evidence they do reveal is always true" does not play a critical role in our experiment.
Q2. Setting the training auditor and evaluation auditor to two different LLMs
We ran another round of strategy audit using GPT-5.4-mini as the auditor for both checkpoints before and after RL. Compared with checkpoints trained without penalty, training with penalty reduced misaligned strategies per game from 3.35 to 1.24 on 4B and from 3.14 to 1.15 on 8B, closely matching Qwen3.5-35B-A3B. See table below:
| Sender | Belief Shift | Misaligned per Game (Qwen Judge) | Misaligned per Game (GPT Judge) |
|---|---|---|---|
| 4B, Full strategy prompt, no penalty, step 0 | 0.41 | 2.72 | 3.44 |
| 8B, Full strategy prompt, no penalty, step 0 | 0.34 | 1.58 | 2.30 |
| 4B, Full strategy prompt, no penalty, step 100 | 0.62 | 2.94 | 3.35 |
| 8B, Full strategy prompt, no penalty, step 100 | 0.84 | 2.67 | 3.14 |
| 4B, Full strategy prompt + GPT-5.4-mini Distill + penalty, step 200 | 0.55 | 0.68 | 1.24 |
| 8B, Full strategy prompt + GPT-5.4-mini Distill + penalty, step 200 | 0.50 | 0.49 | 1.15 |
Response to authors' Rebuttal
Thank you for the clear and focused response. I appreciate the clarification on the receiver's outputs, the new ablation removing "the evidence they do reveal is always true," and the evaluation with an independent GPT-5.4-mini judge.
Although these responses have addressed my main concerns to a reasonable extent, I believe my current score remains appropriate when considering the overall results presented in the paper. Therefore, I will maintain my current score.
We thank the reviewer's positive, insightful comments and for recognizing our findings in training more aligned LLMs for persuasion and acknowledging the project is timely. We address the feedback and questions below, and have updated the paper accordingly.
Responses to Weaknesses
W1. Realism of the persuasion setting
W2. Using the same model for training penalties and evaluation auditing
We agree that sharing Qwen3.5-35B-A3B between training-time penalties and test-time evaluation could in principle allow the sender to evade that specific judge. Here, we additionally evaluate with GPT-5.4-mini as an independent auditor. As shown in Response to Q2 below, the alignment gains hold under the independent judge.