- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies a preference trap in persuasion RL, where an agent gains reward by exploiting the training simulator's response patterns and then performs worse on other user models. It proposes SimPatch, which finds repeated dialogue patterns with high rewards and adds targeted rules and rewritten examples to the simulator's prompt during training.
Critical strengths
-
The paper studies how transfer can decline while training reward continues to increase. Tracking this change during training gives a useful account of how agents become dependent on a simulator's response patterns.
-
SimPatch targets specific simulator responses through prompt rules and examples, without training the simulator's weights or changing the RL algorithm.
-
The evaluation uses three user models that are separate from the training simulator and patch judge.
Critical weaknesses
-
The paper does not independently validate the judge's decisions about which simulator responses should be patched. The filtering ablation shows that this step matters, but does not show how often it rejects a valid response or misses a real shortcut.
-
The strategy analysis mainly tracks repeated promises to donate the full $2 while ignoring the user's replies. Classifying other standard persuasion strategies as robust does not independently show that they transfer better.
-
Adding all twelve rules at the start changes both patch timing and the initial difficulty of the task. Table 2 therefore does not isolate the benefit of detecting traps online from the benefit of gradually making training harder.
-
Appendix A.3 reports that 54 of the 172 test persuadee speakers also appear in training. The current split therefore does not isolate generalization to unseen user profiles, even though the test language models are separate.
-
The algorithm has no explicit check of a rule's effects or rollback step after the rule is added. Section 4.3 shows that an overly broad rule can remove further patching opportunities, so recovery from an incorrect patch remains unclear.
-
All experiments use one charity donation task with at most five exchanges. The results provide limited evidence about whether the method handles preference traps in other tasks or with another training simulator.
Questions for authors
-
How does SimPatch compare with adding the same rules gradually on a fixed schedule?
-
Could you report results separately for speaker profiles that appear in training and those that do not?
2 comments
Thank you for the response. My concerns about the judge validation, the strategy analysis, the rollback step and the seen versus unseen speakers are now resolved, and the matched schedule control answers my first question. As the new experiments also show, however, the effect does not carry over to other settings, since the gain disappears on hy3 and hy4 and turns negative on CraigslistBargain, so the scope of the claim remains my main concern.
I also noticed the large number of confidence intervals in the response. I understand that this is rigorous and I do not question the numbers, but I have to admit that it set off a reflex in me, which is probably my own and not a fair reading of your work. This may simply be the authors' style, or a necessary strategy given how some reviews are written, but at least for this paper I did not see any other reviewer explicitly asking for confidence intervals.
After taking the rebuttal and the other reviews into account, I have decided to keep my original rating.
W1. Independent validation of the judge. Thank you for pushing on this. Two independent models (hy3, hy4), blind to the judge's verdict, rule, reward and step, labelled each of the 85 audited groups with a rubric fixed in advance: does the group show a simulator concession that should be patched? Both said no for all 85, while the judge wrote a rule for 79, so the judge's precision against either labeller is 0/79 (Wilson upper bound 0.046). A blind audit of the 13 stored rules found 2 that target a real concession (one over-broad), 3 that over-restrict and 8 that are off-target. The labellers do see the repeated pattern that the reward gate selects (yes for 85/85 audited groups, and also for 21-24 of 24 gate-failing controls); what they do not see is a concession a rule should remove. These are LLM labels from one vendor, not human labels. Together with the replay control (PowU, Q1), this is why the revision attributes the gain to saturation-triggered rule insertion rather than to the judge's diagnosis, and describes the judge as a rule writer whose decisions are not independently validated.
W2. Strategy analysis. We coded every policy utterance of the 4B GRPO and SimPatch transcripts (3,630 utterances, 11 categories, coder and taxonomy fixed in advance). SimPatch uses more distinct strategies per dialogue (+0.28 [+0.20, +0.35]) and shifts content toward emotional and logical appeals, but its strategy distribution is less diverse across a task's dialogues (entropy -0.175 [-0.247, -0.102] bits), and repeated full-amount asks are more frequent (7.5% for GRPO vs 39.1% for SimPatch). We replace the submission's "diversifies strategies" description with these results.
W3 / Q1. Matched schedule. See the reply to PowU, Q1 (replay_own, replay_generic, three seeds).
W4 / Q2. Seen vs. unseen speakers. Transcript-backed batch, macro over the three original persuadees, SimPatch - GRPO:
The gain holds on unseen speakers at both sizes. It is smaller there; unseen minus seen is -0.059 [-0.147, +0.029] (4B) and -0.064 [-0.150, +0.024] (8B), so the data neither show nor rule out memorisation of this size. The 8B unseen result does not hold on the newest evaluation batch of the same checkpoints (general response, corrections).
W5. Rollback. The optional rollback retires the rules written at step t if the share of full-amount dialogues was at least 0.2 at step t and falls below 0.02 at step t+1. Its thresholds were set from the 8B run's step 149 -> 150 collapse, and it was not used in the submission. We have now run it (seed 2001, same protocol). It works as designed: it fired once, when the first rule (step 154) dropped the full-amount share from 0.42 to 0.008, and retired that rule at step 155; the run ended with 3 rules. It does not improve transfer, though: on the primary metric rollback is below default SimPatch (-0.007 [-0.014, -0.000]) and below GRPO (-0.010 [-0.017, -0.003]).
W6. Other tasks and training simulators. See the reply to PowU, Q2 (CraigslistBargain; Llama-3-8B simulator) and the reply to fSq1 (SOTOPIA).