acceptodds
Under review as a conference paper at ICLR 2027

Mitigating the Preference Trap of LLM User Simulators in Persuasion Dialogue RL

Abstract

LLM agents, despite strong performance on logical-reasoning tasks, still underperform in persuasion, a non-collaborative social interaction that demands adaptive strategies across a wide range of people over long-horizon conversations. Multi-turn reinforcement learning with LLM user simulators offers a promising paradigm to endow LLM agents with persuasion capabilities through open-ended interaction. However, we reveal that this paradigm suffers from a subtle training dynamic, which we term the preference trap: as training progresses, agents can be misled into abandoning broadly effective strategies in favor of brittle ones that exploit the training simulator's narrow preferences, earning high training reward but failing to generalize to unseen counterparts. Such exploits cannot be foreseen before training, because they only appear as the policy evolves. We therefore propose SimPatch, which detects exploited preferences during training and patches them through targeted instructions added to the simulator's prompt. SimPatch first identifies suspicious high-reward rollouts that share similar interaction patterns, then diagnoses the simulator-specific preference that makes the underlying strategy unusually effective. It then converts this preference into targeted instructions and representative examples, which are added to the simulator's prompt for subsequent rollouts. Without modifying the simulator weights, agent architecture, or RL algorithm, SimPatch provides a lightweight way to prevent repeated exploitation of simulator-specific preferences. On three held-out LLM persuadees, SimPatch raises GRPO's reward from 0.257 to 0.493 (Qwen3-4B) and from 0.283 to 0.391 (Qwen3-8B), outperforming all baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
persuasion task, agentic training, social agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_RNtWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a preference trap in persuasion RL, where an agent gains reward by exploiting the training simulator's response patterns and then performs worse on other user models. It proposes SimPatch, which finds repeated dialogue patterns with high rewards and adds targeted rules and rewritten examples to the simulator's prompt during training.

Critical strengths
  1. The paper studies how transfer can decline while training reward continues to increase. Tracking this change during training gives a useful account of how agents become dependent on a simulator's response patterns.

  2. SimPatch targets specific simulator responses through prompt rules and examples, without training the simulator's weights or changing the RL algorithm.

  3. The evaluation uses three user models that are separate from the training simulator and patch judge.

Critical weaknesses
  1. The paper does not independently validate the judge's decisions about which simulator responses should be patched. The filtering ablation shows that this step matters, but does not show how often it rejects a valid response or misses a real shortcut.

  2. The strategy analysis mainly tracks repeated promises to donate the full $2 while ignoring the user's replies. Classifying other standard persuasion strategies as robust does not independently show that they transfer better.

  3. Adding all twelve rules at the start changes both patch timing and the initial difficulty of the task. Table 2 therefore does not isolate the benefit of detecting traps online from the benefit of gradually making training harder.

  4. Appendix A.3 reports that 54 of the 172 test persuadee speakers also appear in training. The current split therefore does not isolate generalization to unseen user profiles, even though the test language models are separate.

  5. The algorithm has no explicit check of a rule's effects or rollback step after the rule is added. Section 4.3 shows that an overly broad rule can remove further patching opportunities, so recovery from an incorrect patch remains unclear.

  6. All experiments use one charity donation task with at most five exchanges. The results provide limited evidence about whether the method handles preference traps in other tasks or with another training simulator.

Questions for authors
  1. How does SimPatch compare with adding the same rules gradually on a fixed schedule?

  2. Could you report results separately for speaker profiles that appear in training and those that do not?

2 comments

W1. Independent validation of the judge. Thank you for pushing on this. Two independent models (hy3, hy4), blind to the judge's verdict, rule, reward and step, labelled each of the 85 audited groups with a rubric fixed in advance: does the group show a simulator concession that should be patched? Both said no for all 85, while the judge wrote a rule for 79, so the judge's precision against either labeller is 0/79 (Wilson upper bound 0.046). A blind audit of the 13 stored rules found 2 that target a real concession (one over-broad), 3 that over-restrict and 8 that are off-target. The labellers do see the repeated pattern that the reward gate selects (yes for 85/85 audited groups, and also for 21-24 of 24 gate-failing controls); what they do not see is a concession a rule should remove. These are LLM labels from one vendor, not human labels. Together with the replay control (PowU, Q1), this is why the revision attributes the gain to saturation-triggered rule insertion rather than to the judge's diagnosis, and describes the judge as a rule writer whose decisions are not independently validated.

W2. Strategy analysis. We coded every policy utterance of the 4B GRPO and SimPatch transcripts (3,630 utterances, 11 categories, coder and taxonomy fixed in advance). SimPatch uses more distinct strategies per dialogue (+0.28 [+0.20, +0.35]) and shifts content toward emotional and logical appeals, but its strategy distribution is less diverse across a task's dialogues (entropy -0.175 [-0.247, -0.102] bits), and repeated full-amount asks are more frequent (7.5% for GRPO vs 39.1% for SimPatch). We replace the submission's "diversifies strategies" description with these results.

W3 / Q1. Matched schedule. See the reply to PowU, Q1 (replay_own, replay_generic, three seeds).

W4 / Q2. Seen vs. unseen speakers. Transcript-backed batch, macro over the three original persuadees, SimPatch - GRPO:

Seen (79 tasks, 54 speakers)Unseen (121 tasks)All (200)
4B+0.161 [+0.091, +0.229]+0.102 [+0.046, +0.159]+0.125 [+0.081, +0.169]
8B+0.146 [+0.078, +0.213]+0.082 [+0.025, +0.139]+0.107 [+0.064, +0.151]

The gain holds on unseen speakers at both sizes. It is smaller there; unseen minus seen is -0.059 [-0.147, +0.029] (4B) and -0.064 [-0.150, +0.024] (8B), so the data neither show nor rule out memorisation of this size. The 8B unseen result does not hold on the newest evaluation batch of the same checkpoints (general response, corrections).

W5. Rollback. The optional rollback retires the rules written at step t if the share of full-amount dialogues was at least 0.2 at step t and falls below 0.02 at step t+1. Its thresholds were set from the 8B run's step 149 -> 150 collapse, and it was not used in the submission. We have now run it (seed 2001, same protocol). It works as designed: it fired once, when the first rule (step 154) dropped the full-amount share from 0.42 to 0.008, and retired that rule at step 155; the run ended with 3 rules. It does not improve transfer, though: on the primary metric rollback is below default SimPatch (-0.007 [-0.014, -0.000]) and below GRPO (-0.010 [-0.017, -0.003]).

W6. Other tasks and training simulators. See the reply to PowU, Q2 (CraigslistBargain; Llama-3-8B simulator) and the reply to fSq1 (SOTOPIA).

Thank you for the response. My concerns about the judge validation, the strategy analysis, the rollback step and the seen versus unseen speakers are now resolved, and the matched schedule control answers my first question. As the new experiments also show, however, the effect does not carry over to other settings, since the gain disappears on hy3 and hy4 and turns negative on CraigslistBargain, so the scope of the claim remains my main concern.

I also noticed the large number of confidence intervals in the response. I understand that this is rigorous and I do not question the numbers, but I have to admit that it set off a reflex in me, which is probably my own and not a fair reading of your work. This may simply be the authors' style, or a necessary strategy given how some reviews are written, but at least for this paper I did not see any other reviewer explicitly asking for confidence intervals.

After taking the rebuttal and the other reviews into account, I have decided to keep my original rating.

Reviewer_PowUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies overfitting when persuasion agents are trained with reinforcement learning against an LLM user simulator. It proposes SimPatch, which identifies repeated successful strategies that appear to exploit the simulator, then adds targeted instructions and examples to the simulator’s prompt during training. Experiments on a donation task show improved rewards against three held-out user models, using two sizes of Qwen agents.

Critical strengths

The paper provides a practical way to update the training environment in response to weaknesses exposed by the agent. The improvements are measured against held-out user models that do not receive the patches, providing meaningful evidence of transfer. The paper also compares against several relevant alternatives, including simulator mixing, generic hardening, and adversarial training.

Critical weaknesses

The experiments do not sufficiently establish the distinctive contribution of targeted preference diagnosis. The underlying problem is overfitting to the training simulator. The potential novelty lies in identifying and correcting specific weaknesses during training, but the ablations do not clearly separate this benefit from gradually adjusting the environment’s difficulty. Adding all rules at the start changes the training schedule, while inserting raw successful dialogues gives the simulator examples of the behavior it should avoid. The failure of these alternatives does not establish that targeted diagnosis is necessary or better than a well-designed curriculum.

The evidence supports a narrower contribution than the proposed method suggests. All main experiments use one donation task, one primary training simulator, and two agents from the same model family. Testing against three other user models demonstrates transfer across models within this task, but does not establish that the method works across different incentives, simulator weaknesses, or training environments.

The paper shows a useful local improvement, but does not yet establish when this approach should work or what its targeted diagnosis adds beyond simpler adaptive training methods.

Questions for authors

What evidence separates the benefit of targeted preference diagnosis from gradually making the simulator harder? Is there a comparison with predefined rules introduced on a matched schedule?

Does the method remain effective when the training simulator or the type of exploitable behavior changes? Please distinguish existing results from proposed future work.

How stable is the improvement over simulator mixing across independent training runs?

2 comments

Q1. Diagnosis vs. gradually hardening the simulator; predefined rules on a matched schedule. Thank you, this is the most informative control we added. All arms resume the same GRPO checkpoint (step 99) and train to step 249 with identical flags and seed; only the simulator prompt differs.

  • replay_own: the rules SimPatch wrote in its own run, inserted at the steps where they were written, with no audit and no judge.
  • replay_generic: 12 generic hardening rules written in advance by another model, inserted at the same steps.

Held-out reward at step 249, mean over seeds 2001-2003 (hy3 n=200, hy4 the first 100 test tasks, DeepSeek n=200):

Armhy3hy4DeepSeek (secondary)
Zero-shot0.0090.0010.241
GRPO0.0300.0030.328
SimPatch0.0260.0040.280
replay_own0.0340.0040.276
replay_generic0.0170.0030.452
DAPO0.0190.0040.341
SimPatch - control, primary macroSeed 2001Seed 2002Seed 2003Mean of seeds
GRPO-0.003-0.003+0.001-0.001 [-0.006, +0.003]
replay_own+0.000+0.002-0.015 [-0.023, -0.007]-0.004 [-0.009, +0.000]
replay_generic+0.003+0.005+0.007 [+0.002, +0.013]+0.005 [+0.001, +0.009]
DAPO+0.006+0.007-0.002+0.003 [-0.001, +0.008]

Two findings follow. First, SimPatch's own rules are as effective when replayed on the same schedule as when written online, and SimPatch is above generic rules on the same schedule on the primary metric (+0.005 [+0.001, +0.009]); what carries the effect is which rules are inserted and when, i.e. hardening triggered by saturated groups, rather than the per-group diagnosis. The revision says so explicitly. Second, the ranking of trained arms depends on the held-out persuadee: on DeepSeek, SimPatch - GRPO is -0.048 [-0.079, -0.018], - replay_own +0.003 [-0.027, +0.033] and - replay_generic -0.173 [-0.208, -0.137]. The generic rules were written by hy4, so hy4 is not a clean persuadee for that arm. replay_generic is also the matched counterpart of the removed Sim-Hard baseline.

Limits: all arms share their first 100 steps; hy4 closes 1-2% of dialogues for every arm and barely separates them; one dialogue per task at temperature 0.7. Dialogues with an empty persuadee reply were rerun once and are left out if still empty (at most 1 per cell).

Q2. Other training simulators or exploit types. Both new settings confirm the preference trap.

CraigslistBargain (buyer side). A GRPO buyer trained against Qwen2.5-72B closes 90.2% of its deals below the price the dataset's human seller accepted, by 27.3% of the listing on average, and held-out LLM sellers concede the same way, a clear instance of the trap in a second task. We therefore evaluate against sellers that hold the human deal price as a floor. We trained SimPatch on Craigslist from scratch with default settings (250 steps, 63 rules) and evaluated on two held-out sellers, 200 paired dialogues per cell, reward 0 without a deal:

SellerFloorZero-shotGRPOSimPatchGRPO trained vs. floor (reference)
hy3on0.2890.2580.2270.440
hy3off0.3630.6070.5810.681
hy4on0.2150.2060.1760.419

Training GRPO against a seller that holds the floor raises this to 0.440 (hy3) and 0.419 (hy4), so the drop comes from the training simulator's concessions, as the paper argues. SimPatch with default settings does not close it here: with the floor on (primary, macro over hy3 and hy4), SimPatch - GRPO is -0.031 [-0.056, -0.006] and SimPatch - zero-shot -0.051 [-0.085, -0.016]; its rules do not stop the buyer from relying on a seller who concedes, and we report this as a limitation. Disclosure: the floor was introduced after we saw the below-human-price deals, and held-out sellers often state their floor outright (about 40% of hy3 and 20% of hy4 turns, similar across arms).

A different training simulator. We retrained both arms on P4G from the base 4B policy with Llama-3-8B-Instruct as the training simulator (250 steps, seed 2001, all else unchanged):

Zero-shotGRPOSimPatch
hy30.0090.0030.013
DeepSeek (secondary)0.2410.1580.176

GRPO ends below the untrained model (primary -0.004 [-0.008, +0.000]; DeepSeek -0.082 [-0.131, -0.036]), so the trap appears with this simulator too. SimPatch is above GRPO on the primary metric (+0.005 [+0.002, +0.009]; hy3 +0.009 [+0.002, +0.017]; DeepSeek +0.018 [-0.015, +0.051]), though not above the untrained model (+0.002 [-0.003, +0.007]; on DeepSeek both arms are below it, SimPatch -0.064 [-0.114, -0.016]). It wrote a single rule and rewards are near the floor (hy3 deal rates 2.5-8.5%), so we read this as supportive but weak evidence. SOTOPIA: see the reply to fSq1.

Q3. Stability over independent runs. Three seeds of each arm, same protocol (table above):

SimPatch - GRPO, same seedPrimary macrohy3DeepSeekRules
2001-0.003 [-0.010, +0.005]-0.008 [-0.022, +0.007]-0.112 [-0.158, -0.065]4
2002-0.003 [-0.011, +0.005]-0.007 [-0.021, +0.008]-0.085 [-0.131, -0.041]9
2003+0.001 [-0.006, +0.008]+0.002 [-0.010, +0.014]+0.053 [-0.001, +0.107]1

The primary result is stable across seeds (no seed separates the arms). On DeepSeek the sign changes between seeds, and GRPO itself moves by 0.05 between seeds there. The seeds share the first 100 steps, so they understate between-run variance. Simulator-mixing checkpoints were not kept, so we could not re-train that baseline with seeds; in the submitted setting SimPatch is above it at both sizes under the uniform re-scoring (general response).

Reviewer_fSq1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SimPatch, an online framework that dynamically detects and corrects exploitable simulator behaviors during training to mitigate an issue where agents learn to exploit simulator-specific behavioral preferences rather than generalizable interaction strategies. SimPatch first identifies rollout groups with consistently high rewards, then uses an LLM judge (DeepSeek-V4-Flash) to determine whether the trajectories within each group exhibit recurring interaction patterns involving unjustified concessions by the simulator. The judge then generates corrective instructions and rewritten response examples, which are incorporated into the simulator's prompt for subsequent training iterations. Experiments on PersuasionForGood show that SimPatch improves the persuader's performance against three unseen LLM simulators compared with GRPO and several simulator-modification baselines.

Critical strengths
  • The paper presents an interesting idea and targets an important problem where agents exploit simulator-specific behavioral preferences rather than learning generalizable interaction strategies. SimPatch aims to mitigate this issue by dynamically modifying the simulator's prompt based on an LLM judge's feedback.
  • The proposed method is intuitive and only requires prompt modifications to the simulator without updating its model parameters.
Critical weaknesses
  • All experiments are conducted on the same type of persuasion task, where the agent asks users to donate part of their $2 task payment. Although the authors evaluate on different user simulators and personas, the underlying persuasion objective remains the same and is relatively simple. It is unclear whether the proposed method generalizes to more diverse persuasion settings or other social tasks, such as negotiation and conflict resolution. Why didn't the authors consider evaluating on other social interaction benchmarks, such as SOTOPIA?

  • This paper compares SimPatch against standard GRPO and several simulator-modification approaches. However, it is unclear why the authors do not compare against methods that explicitly address reward saturation or zero-variance rollout groups during RL, such as DAPO-style dynamic sampling, especially considering that one of the paper's main goals is to maintain informative reward variance during training.

  • It wasn't clear how SimPatch diversifies strategies compared to standard GRPO. Some qualitative analyses would be useful to understand the effect of SimPatch.

  • The paper does not provide analyses of hyperparameter sensitivity. For example, how would the method perform with different thresholds for the mean reward and standard deviation, which are currently set to 0.9 and 0.1, respectively? Also, how would the performance change when using a different LLM judge?

  • [Minor] The paper includes several simulator-modification baselines (Sim-Hard, Sim-VS, Sim-Ensemble, and Sim-AdvTrain), but the rationale for selecting these approaches is not clearly explained in the experimental setup section.

  • [Minor] Missing related work in 'Social intelligence of LLM agents' Section: Infusing Theory of Mind into Socially Intelligent LLM Agents (Hwang et al., Findings of ACL 2026). It seems relevant to the discussion of adapting strategies to different dialogue partners in various social situations, such as persuasion, negotiation, and conflict resolution.

Questions for authors

n/a

1 comment

Other tasks. We added CraigslistBargain (reply to PowU, Q2), where the trap clearly appears but SimPatch with default settings is below GRPO, and SOTOPIA. Setup: 90 environments split by environment into 63 train and 27 test, with the 14 environments of the released benchmark_agents.json (which we take as SOTOPIA-hard) all in test; the policy (Qwen3-4B-Instruct) plays one character, the training partner is Qwen2.5-72B, and the reward is the SOTOPIA-Eval goal score / 10 from DeepSeek-V4-Flash (Pearson 0.70 with the released GPT-4 goal scores on 20 episodes). GRPO and SimPatch from step 0, 150 steps, seed 2001; 270 test dialogues:

PartnerZero-shotGRPOSimPatchSimPatch - GRPO
hy3 (held out, primary)0.6000.5950.604+0.009 [-0.018, +0.036]
DeepSeek-V4-Flash0.6570.6950.706+0.011 [-0.013, +0.034]
Qwen2.5-72B (training partner)0.7120.7190.723+0.004 [-0.020, +0.029]

SimPatch is numerically highest against every partner, but no difference is distinguishable from zero, and neither arm improves on the untrained model against the held-out partner; the hard subset gives the same picture. SOTOPIA is a largely cooperative setting in which the base policy already scores at least 9/10 on many groups, so SimPatch's gate fires from the first steps and it wrote 256 rules, several of them scenario-specific; we discuss this as a boundary of the method. Limits: free-text turns with a [LEAVE] action instead of SOTOPIA's JSON action space, the goal dimension only, and DeepSeek instead of GPT-4 as evaluator.

DAPO-style dynamic sampling. Your question is fair: in the final 25 steps GRPO had 53.7% zero-variance groups vs 19.3% for SimPatch. We added a DAPO arm under the same protocol (resume step 99, zero-variance groups redrawn up to 4 times), three seeds. SimPatch - DAPO on the primary metric is +0.006, +0.007 and -0.002 by seed, mean +0.003 [-0.001, +0.008]; on DeepSeek it is -0.061 [-0.092, -0.031]. Keeping reward variance by resampling alone does not help transfer on the primary persuadees, and neither method is above GRPO there.

How SimPatch changes strategies. See the reply to RNtW, W2; the revision replaces "diversifies strategies" with the coded results.

Hyperparameter and judge sensitivity. Four SimPatch variants under the P4G protocol (seed 2001, resume step 99), each changing one thing. Primary metric, difference to default SimPatch and to GRPO of the same seed:

VariantRulesvs SimPatchvs GRPO
Gate loose (mean >= 0.8, std <= 0.2)9-0.003 [-0.010, +0.003]-0.006 [-0.012, +0.001]
Gate strict (mean >= 0.95, std <= 0.05)6-0.006 [-0.013, +0.001]-0.008 [-0.015, -0.001]
Judge = Qwen2.5-72B (the training simulator)3-0.006 [-0.013, +0.001]-0.008 [-0.015, -0.001]
Rollback on (see RNtW, W5)3-0.007 [-0.014, -0.000]-0.010 [-0.017, -0.003]

The default setting is numerically the best of the five, and no variant moves SimPatch above GRPO under this protocol, so our conclusions do not depend on these choices. One seed per variant.

Baseline rationale. Sim-Hard tests a fixed, hand-written hardening; Sim-VS tests response diversity without targeting; Sim-Ensemble tests simulator mixing; Sim-AdvTrain tests updating the simulator's weights. We add this to the setup. Sim-Hard has no surviving run and is removed; the matched-schedule replay_generic arm (reply to PowU, Q1) takes its place.

Related work. We will cite Hwang et al. (Findings of ACL 2026) in the social-intelligence section.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank the reviewers for their careful reading and constructive questions. All three asked (i) whether online diagnosis adds anything beyond gradually hardening the simulator and (ii) whether the findings hold beyond one task, one training simulator and one run. We ran new experiments for both, and they substantially strengthen the paper. Arms, held-out models, metric and reporting rule were fixed in a protocol file before any rebuttal run was evaluated, and we report every arm we ran. Differences are paired over test tasks with 95% bootstrap CIs.

Main takeaways.

  1. The preference trap generalizes. It reappears with a second training simulator (Llama-3-8B-Instruct: GRPO ends below the untrained model, DeepSeek -0.082 [-0.131, -0.036]) and on a second task (CraigslistBargain: the GRPO buyer closes 90.2% of deals below the human price, and its reward on a held-out seller drops from 0.607 to 0.258 once the seller holds that price).
  2. SimPatch's gain in the submitted setting survives a uniform, transcript-backed re-scoring. With every cell scored by one rule (the explicit donation marker), SimPatch is above GRPO at both sizes (4B +0.125 [+0.082, +0.169], 8B +0.107 [+0.064, +0.151]), above simulator mixing at both sizes (+0.109 [+0.067, +0.150], +0.174 [+0.134, +0.215]) and above GRPO on persuadee speakers never seen in training (4B +0.102 [+0.046, +0.159], 8B +0.082 [+0.025, +0.139]).
  3. The new controls identify the active ingredient. Inserting SimPatch's own rules at the same steps without diagnosis matches SimPatch (-0.004 [-0.009, +0.000]), and two independent LLM labellers do not confirm the judge's per-group diagnoses (RNtW, W1). The benefit therefore comes from saturation-triggered rule insertion, and the revision attributes it there.
  4. We delineate the scope. When all arms resume one GRPO checkpoint at step 99 and are tested on two further held-out persuadees (hy3, hy4; primary metric fixed in advance), GRPO and SimPatch are about +0.01 above the untrained model in every seed, and SimPatch, GRPO, DAPO and rule replay are within 0.01 of each other (SimPatch - GRPO -0.001 [-0.006, +0.003], three seeds). SimPatch is above GRPO with the Llama-3-8B simulator (+0.005 [+0.002, +0.009]), though not above the untrained model there, and it does not transfer to CraigslistBargain (-0.031 [-0.056, -0.006]) or SOTOPIA (+0.009 [-0.018, +0.036]). The revision states these as limitations.

Corrections we found while re-scoring, all fixed in the revision.

  • The 4B GRPO x Doubao cell (0.324) has no surviving transcript; two re-evaluations of the same checkpoint give 0.658 [0.595, 0.719] and 0.677 [0.616, 0.738]. The transcript-backed 4B gain is therefore +0.125 rather than +0.236, and on Doubao alone SimPatch is on par with GRPO (-0.015 [-0.099, +0.067]).
  • Three further cells (4B Sim-Ensemble x GPT-4o-mini, 8B Sim-VS and Sim-AdvTrain x Ministral) come from re-runs whose transcripts were not kept, and like the 0.324 cell all lower a baseline. We replace all four with the locally recorded evaluation of the same checkpoint. Sim-Hard has no surviving run and is removed; the matched-schedule generic-rule arm takes its place.
  • At 8B the GPT-4o-mini contrast changes sign between two evaluation batches of the same checkpoints (+0.069 vs -0.072; macro +0.107 vs +0.040 [-0.004, +0.085]), and the 8B run wrote a single rule. We report both batches.

Revised paper (changes in blue). Transcript-backed, uniformly re-scored main table and abstract (4B 0.362 -> 0.487, 8B 0.283 -> 0.391); a new section with the matched-schedule controls, three seeds, DAPO, the Llama-3-8B simulator, CraigslistBargain, SOTOPIA, judge validation, strategy coding, seen/unseen speakers and sensitivity; "consistently improves" and "diversifies strategies" removed; failed-trajectory rates and re-runs per arm in the appendix. hy3 and hy4 are named in the revision (withheld here for anonymity; the submission's persuadee endpoints were no longer available to us).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.