acceptodds
Under review as a conference paper at ICLR 2027

Reinforcing Human Behavior Simulation via Verbal Feedback

Abstract

Humans learn social norms and behaviors from verbal feedback (e.g., a parent saying “that was rude” or a friend explaining “here's why that hurt”). Yet, learning from feedback for LLMs has largely focused on domains like code and math where RL rewards are directly verifiable and condensed into scalar values. As LLMs are increasingly also used for simulating human behavior, e.g., standing in for users, patients, students, and other personas, there is a pressing need to make them more human-like, which requires embracing a fundamentally different kind of signal: feedback that is verbal, subjective, and multi-faceted. We present Ditto, a model trained by treating verbal feedback as a first-class signal in reinforcement learning. After each rollout, Ditto receives verbal feedback and generates a feedback-conditioned improved rollout; both outputs are jointly optimized with GRPO, distilling verbal guidance into the base policy without requiring feedback at test time. We also introduce Soul (Simulation gym Of hUman-Like behavior), a unified benchmark and training data suite spanning 10 tasks across six categories: Theory of Mind, character role play, social skill, learner simulation, user simulation, and persona simulation. Ditto achieves an average 36% improvement over the base model and exceeds GPT-5.4 on 6 of 10 Soul benchmarks, demonstrating that RL with verbal feedback is a promising direction for training LLMs to simulate human behavior.

Primary area:
foundation or frontier models, including LLMs
Keywords:
User Simulation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pD8FOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The authors introduce DITTO, an RL approach to incorporate text feedback into the learning of LLM simulators of human behavior. They also present SOUL, a benchmark of human behavior simulation that captures 10 tasks across several behavioral categories.

Critical strengths
  • The paper addresses a very important gap: Learning from rich feedback beyond reward scalars, namely text feedback. And they explore this on the increasingly relevant but understudied in this context field of social simulation.
  • I like simplicity of idea: Add text feedback to prompt to guide behavior of new rollouts and add these to GRPO.
  • DITTO achieves strong empirical improvements. DITTO even outperforms its own judge (GPT-5-nano) suggesting that judge's performance need not completely limit the downstream task performance.
Critical weaknesses
  • I was surprised by how well GRPO without any text feedback performed. This raised the question: Is text feedback even worth it, or should we use instead the extra computation power to train GRPO for longer (more training steps)? Another tradeoff to analyze: DITTO requires double the rollouts per prompt w.r.t. GRPO. What happens if we instead trained GRPO on double the rollouts?
  • I found it hard to understand your exact contribution w.r.t RLTF. In Section 3.4. you provide 3 reasons, but could you clarify them? Namely:
    1. Is reason 1 just that you substitute one optimizer (AWR) for another (GRPO)? If so, since both are established RL training methods, it would benefit from some more explanation of why GRPO is better suited for this task, if not, could you please clarify what you mean by (1)?
    2. RLTF also uses judge generated critics. Could you clarify what you mean by "judge-generated reflection hh rather than raw feedback"? What is the difference?
    3. I agree that the application to social simulation is important (I highlight this as one of your strengths). However, it seems like RLTF can also be applied to these environments (you do this in Section 6.2), so I am not sure this is a limitation of the RLTF approach itself.
  • I could only find std values for Sotopia with 4 seeds. It would be nice to see std or IQR values on Figure 3 or Table 1.
  • Minor presentation issues:
    • Mention AWR stands for advantage-weighted regression when you introduce it in Section 3.4., not all the way down in Appendix E.
    • Hard to interpret results when no clear definition of what the "primary metric" is provided in the main text.
Questions for authors
  • Do the gains persist compared to GRPO under the same total training budget?
  • Could you please clarify the distinctions between DITTO and RLTF approaches?
3 comments

Thank you for the detailed review and for recognizing that learning from text feedback beyond scalar rewards addresses an important gap in social simulation, that the idea is simple, and that Ditto achieves strong improvements, even outperforming its own judge. We answer your questions as follows.


Is text feedback even worth it, or should we use instead the extra computation power to train GRPO for longer (more training steps)? … What happens if we instead trained GRPO on double the rollouts?

Do the gains persist compared to GRPO under the same total training budget?

Yes. We gave GRPO Ditto's rollout budget in two ways, doubling the batch size or the rollouts per prompt, evaluated on Sotopia as in Table 2:

MethodConfigurationSotopia score
GRPObatch size 64, group size 84.23
GRPObatch size 128, group size 84.38
GRPObatch size 64, group size 164.36
Dittobatch size 64, group size 8 + teacher rollout4.69 ± 0.02

Both budgets improve GRPO (4.38 and 4.36 vs. 4.23), but it still falls behind Ditto (4.69). We will add this comparison to the paper.

Text feedback also helps on aspects the scalar reward does not directly optimize: on Sotopia, GRPO's secret-keeping score degrades during training, while Ditto preserves it (Section 6.2, Figure 3).

In summary, Ditto outperforms GRPO under the same rollout budget, so its gains come from verbal feedback, not just extra computation.


I found it hard to understand your exact contribution w.r.t RLTF. In Section 3.4. you provide 3 reasons, but could you clarify them?

Could you please clarify the distinctions between DITTO and RLTF approaches?

We clarify each reason below and will revise Section 3.4 accordingly.


Is reason 1 just that you substitute one optimizer (AWR) for another (GRPO)?

Not only. The main difference is the teacher rollout's advantage: Ditto normalizes it jointly with the student rollouts by the group mean and std, whereas RLTF-SD subtracts the mean student reward:

Ditto:

Lgroup=−∑i=1G∑j∈0,1∑tmin⁡(ρi,j,tAi,j, clip(ρi,j,t,1−ϵ,1+ϵ)Ai,j),Ai,j=ri,j−mean(rk,0,rk,1k=1G)std(rk,0,rk,1k=1G),ρi,j,t=πθ(yi,j,t∣x,yi,j,<t)πold(yi,j,t∣x,yi,j,<t)\mathcal{L}_{\text{group}}=-\sum_{i=1}^{G}\sum_{j\in\\{0,1\\}}\sum_{t}\min(\rho_{i,j,t}A_{i,j},\ \mathrm{clip}(\rho_{i,j,t},1-\epsilon,1+\epsilon)A_{i,j}),\quad A_{i,j}=\frac{r_{i,j}-\mathrm{mean}(\\{r_{k,0},r_{k,1}\\}_{k=1}^{G})}{\mathrm{std}(\\{r_{k,0},r_{k,1}\\}_{k=1}^{G})},\quad \rho_{i,j,t}=\frac{\pi_\theta(y_{i,j,t}\mid x,y_{i,j,<t})}{\pi_{\text{old}}(y_{i,j,t}\mid x,y_{i,j,<t})}

RLTF-SD:

LSD=−λ∑i=1GAilog⁡πθ(yi,1∣x),Ai=ri,1−1G∑k=1Grk,0,λ=0.1\mathcal{L}_{\text{SD}}=-\lambda\sum_{i=1}^{G}A_{i}\log\pi_\theta(y_{i,1}\mid x),\quad A_{i}=r_{i,1}-\frac{1}{G}\sum_{k=1}^{G}r_{k,0},\quad \lambda=0.1

TermDitto (Lgroup\mathcal{L}_{\text{group}})RLTF-SD (LSD\mathcal{L}_{\text{SD}})
Rollouts trained under xxstudent yi,0y_{i,0} and teacher yi,1y_{i,1}teacher yi,1y_{i,1}
Advantage(ri,j−μ)/σ(r_{i,j}-\mu)/\sigma, with μ,σ\mu,\sigma over all 2G2G rewardsri,1−rˉ0r_{i,1}-\bar{r}_0, with rˉ0\bar{r}_0 the mean student reward; no std
Weight11λ=0.1\lambda=0.1
Ratio and clippingclipped ρ=πθ/πold\rho=\pi_\theta/\pi_{\text{old}}none

Joint normalization puts student and teacher rollouts on one scale, suiting our continuous judge rewards whose scale varies across tasks; in our ablation, Ditto reaches 4.69 on Sotopia vs. 3.59 for RLTF-SD (Table 2).


RLTF also uses judge generated critics. Could you clarify what you mean by "judge-generated reflection rather than raw feedback"? What is the difference?

Both methods use an LLM judge; the difference is what the teacher sees. In RLTF, the teacher sees its previous attempt y0y_0 and the judge's critique of it, and revises y0y_0. In Ditto, the teacher sees only the task prompt xx and the judge's reflection hh on how to improve, and attempts the task from scratch.


However, it seems like RLTF can also be applied to these environments (you do this in Section 6.2), so I am not sure this is a limitation of the RLTF approach itself.

We agree that RLTF can be applied to social simulation. Section 3.4 lists this as a difference in the target setting rather than a limitation of RLTF, and we will clarify this in the paper.


I could only find std values for Sotopia with 4 seeds. It would be nice to see std or IQR values on Figure 3 or Table 1.

Thank you for the suggestion. We will add std over seeds to Figure 3 and Table 1. On Sotopia, Ditto obtains 4.69 ± 0.02 over four seeds, and all four (4.65 to 4.72) exceed every GRPO configuration (at most 4.38), implying that Ditto robustly outperforms baselines.


Mention AWR stands for advantage-weighted regression when you introduce it in Section 3.4., not all the way down in Appendix E.

Hard to interpret results when no clear definition of what the "primary metric" is provided in the main text.

Thank you for pointing these out. We will spell out advantage-weighted regression (AWR) in Section 3.4 and define the primary metric of each task in the main text (currently in Appendix D).

Thank you for addressing my questions.

  1. The comparison with larger batch sizes and group size are very helpful to alleviate my concern that DITTO's gains were simply due to more rollouts. Ideally, the comparison would match the total computation cost (number of tokens, training time), but I found matching the number of rollouts already informative.
  2. I believe the paper would really benefit from incorporating your clarification of the differences between DITTO and RLTF-SD. Specially, the answer to 1. explaining how DITTO changes the advantage estimation. A question that comes up again is attribution, i.e., isolationg which changes drive the improvement. I recognize that the paper includes a revised-output-only ablation, which partially addresses this question. A more targeted experiment would compute DITTO's joint-group advantage (with same initial and revised outputs and loss fixed) but use AWR style method (-\sum A_{DITTO} \log \pi). I view this as a useful additional analysis, but not needed for my updated assessment.

Given these additional experiments and clarifications I am increasing my score to a 3.

Thank you for your thoughtful follow-up and for raising your score! We appreciate your constructive feedback and are glad our clarifications addressed your concerns. In the final version, we will include the rollout-matched and compute-matched GRPO comparison, incorporate the clarification of the differences between Ditto and RLTF-SD into Section 3.4 (especially how Ditto changes the advantage estimation), and add the attribution analysis you proposed, which applies Ditto’s joint-group advantage in an AWR-style objective with the same student and teacher outputs. Thank you again for your engagement throughout the discussion!

Reviewer_6jeEOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed a reinforcement learning method to incorporate human-like subjective feedback in LLM training. Based on a benchmark consisting of ten tasks introduced by the authors, their approach achieves 36% improvement over the baseline model that DITTO was based upon. The authors also show that their approach is better than GRPO.

Critical strengths

As LLMs are used more popularly, it's important to let LLMs behave/respond like humans, at least in human-facing tasks. The training approach proposed in this paper provides a novel way to achieve this goal. In addition, the benchmark datasets that the authors created will also be useful for future studies to test and improve.

Critical weaknesses

After training this way, did the model degrade in other tasks? For example, coding and solving math problems. I wish authors to benchmark your model against others using some common public benchmarks instead of only SOUL. This is because DITTO models might perform well on the SOUL tasks, but worse on general-purpose tasks that the original LLM was good at. I think it's important to have comprehensive evaluation on those general-purpose non-SOUL tasks and make sure the performance did not degrade.

Questions for authors
  1. I don't think the authors provide enough information about the limitation of the method. For example, it requires a stronger teacher model to learn from, and that real evaluation involving human annotators may not be applicable when the benchmark is very large and complicated. I encourage the authors to discuss some of the limitations that the audience of this paper should be aware of.
  2. Are you planning to release the code and datasets so future researchers can reproduce your results and improve upon your idea?
1 comment

Thank you for the positive review and for recognizing that making LLMs behave like humans in human-facing tasks is important, that our training approach offers a novel way to achieve this, and that SOUL will be useful for future studies. We answer your questions as follows.

After training this way, did the model degrade in other tasks? For example, coding and solving math problems.

No. We evaluated Ditto and its base model, Qwen3-VL-8B-Instruct, on six public benchmarks, using the same prompts and sampling settings for both models:

BenchmarkQwen3-VL-8B-InstructDitto
MATH-50092.094.0
AIME 202547.552.5
HumanEval+86.086.0
LiveCodeBench v651.052.0
MMLU-Pro69.073.0
IFEval88.086.0

Ditto performs comparably to the base model on non-SOUL benchmarks, and no difference is statistically significant (paired bootstrap, 95% CI). We find this check useful, although retaining these abilities may not always be the goal in human behavior simulation, where some applications call for simulating non-expert users. We will add these results to the paper.


I don't think the authors provide enough information about the limitation of the method. For example, it requires a stronger teacher model to learn from, and that real evaluation involving human annotators may not be applicable when the benchmark is very large and complicated.

Thank you for the suggestion; we will expand the limitations discussion in Appendix A. Regarding the teacher, we will clarify that the teacher is the policy itself conditioned on the verbal feedback, so no stronger model is needed; the only external model is the LLM judge, which GRPO also uses for rewards, and Ditto outperforms GPT-5-nano, our main judge model, on 9 of 10 tasks (Table 1).

Regarding human evaluation, we agree that it is costly at scale, and we will state this cost as a limitation. At the same time, some human evaluation is likely necessary to validate any human behavior simulator, so we use it to validate the automatic results on the four judge-scored tasks (Section 6.3). Scaling human evaluation to large benchmarks and complex, long-horizon behaviors is an open research question, which we will note as future work. We will also consider expanding the limitations section to other possible limitations, such as reward hacking and setup-dependent training stability.


Are you planning to release the code and datasets so future researchers can reproduce your results and improve upon your idea?

Yes! We will release the code, the SOUL datasets and the trained Ditto models.

Reviewer_1evmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces DITTO, which uses verbal feedback to train LLMs for human behavior simulation. The model first generates a response, receives feedback from an LLM judge, and then generates another response using that feedback. Both responses are used to train the model, with an additional objective for learning to follow feedback. At test time, the model generates responses without feedback. The paper also introduces SOUL, a training and evaluation suite covering ten tasks, including theory of mind, role play, social interaction, and user simulation. DITTO improves over standard GRPO on eight tasks, and the paper includes ablations and a small human evaluation.

Critical strengths
  1. The evaluation covers a useful range of tasks. Rather than focusing on one type of social interaction, the paper tests ten tasks with different formats. SOUL could be a useful resource for further work on human simulation.

  2. The paper includes several checks beyond the main results. The ablations, seed checks, and human evaluation help support the findings. Improvements on exact-match tasks also suggest that the gains are not simply due to the preferences of an LLM judge.

Critical weaknesses
  1. The distillation objective could be better justified. Section 3.3 explicitly explains that the (y1,x)(y_1,x) term distills feedback-conditioned responses into the base policy. However, the reason for using the clipped ratio in Equation (8) for these samples remains unclear. Since they were generated with feedback but evaluated without it, a clearer justification would help establish why this particular surrogate is appropriate.

  2. The evidence for the added benefit of verbal feedback is promising but incomplete. Table 2 includes a useful higher-compute GRPO baseline, which improves the Sotopia score from 4.23 to 4.38 but remains below DITTO’s 4.69. However, the actual token and feedback costs are not directly compared, and this control is not reported across the other tasks. The human evaluation also omits GRPO, so it does not establish whether verbal feedback improves human-rated simulation quality beyond standard RL. Uncertainty estimates would also help interpret the small reported differences against GPT-5.4.

Questions for authors
  1. Section 3.3 explains that the (y1,x)(y_1,x) term performs distillation. Could you clarify the motivation for using the clipped ratio in Equation (8) for these feedback-conditioned samples, and how this choice affects optimization stability?

  2. Table 2 includes a useful higher-compute GRPO baseline. How does its actual training cost compare with DITTO, including generated tokens and feedback calls? Including GRPO in the human evaluation would also help establish whether the feedback improves perceived simulation quality.

2 comments

Thank you for the thoughtful review and for recognizing that SOUL covers a useful range of tasks and can serve as a resource for future work on human simulation, and that the ablations, seed checks, human evaluation and gains on exact-match tasks support our findings. We answer your questions as follows.


Could you clarify the motivation for using the clipped ratio in Equation (8) for these feedback-conditioned samples, and how this choice affects optimization stability?

Thank you for the question. We apply the same clipped objective to (y1,x)(y_1, x) as to (y0,x)(y_0, x) so that both are optimized within one group: y1y_1 is reinforced only when it outperforms the group, under the same step-size control. Since both πθ\pi_\theta and πold\pi_{\text{old}} in Eq. (8) condition on xx, the ratio measures how the policy's likelihood of y1y_1 without feedback changes over the mini-batch updates of a step. Clipping matters most for these samples: their tokens often have low probability under xx, so their ratios move quickly, and clipping prevents the policy from over-fitting to them within a step. Training is stable across seeds (4.69 ± 0.02 over four seeds on Sotopia, Table 2), and we will clarify this in Section 3.3.


However, the actual token and feedback costs are not directly compared, and this control is not reported across the other tasks.

How does its actual training cost compare with DITTO, including generated tokens and feedback calls?

Thank you for the question. We compare the actual training cost over the 120 training steps on Sotopia (Table 2):

MethodTokens processedJudge and feedback calls / stepSotopia
GRPO (g), batch size 64, group size 80.27B5124.23
GRPO (h), batch size 128, group size 80.53B1,0244.38
GRPO, batch size 64, group size 160.56B1,0244.36
Ditto, configuration (g)0.39B~9504.69

Ditto processes about 30% fewer tokens and makes fewer judge and feedback calls than both rollout-matched GRPO baselines while achieving a higher score. We will add this cost-controlled comparison to the paper.


Including GRPO in the human evaluation would also help establish whether the feedback improves perceived simulation quality.

Uncertainty estimates would also help interpret the small reported differences against GPT-5.4.

Thank you for the suggestion. We computed 95% paired bootstrap confidence intervals for the human evaluation. Human annotators rate Ditto above its base model on all four benchmarks, and the differences against GPT-5.4 lie within the 95% confidence intervals, showing that Ditto-8B performs on par with GPT-5.4 under human evaluation. The gains over GRPO also hold on the six exact-match tasks scored without an LLM judge, where Ditto outperforms GRPO on five (Table 1). We agree that a larger human evaluation including GRPO would further reduce the uncertainty; as it could not be completed within the rebuttal period, we will add it (together with the confidence intervals) to the final version.

Thank you for the clarifications, which address my concern about rollout resources on Sotopia.

My remaining concern is the significance of the methodological contribution beyond related feedback-distillation approaches such as RLTF and SDPO. DITTO offers a distinct objective, but comparisons with these methods are concentrated on Sotopia. I appreciate the training curves, four-seed results, and smaller-model experiment. These provide useful evidence, although sensitivity to feedback sources, training-data selection, and relative loss weights remains less explored.

The human evaluation also excludes GRPO, so it does not establish whether DITTO improves human-rated simulation quality over this baseline. Exact-match gains are encouraging but do not directly answer this question.

I acknowledge the positive results and the value of SOUL, but remain unconvinced that the methodological increment and current validation together are sufficient for acceptance. I therefore maintain the original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.