- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The authors introduce DITTO, an RL approach to incorporate text feedback into the learning of LLM simulators of human behavior. They also present SOUL, a benchmark of human behavior simulation that captures 10 tasks across several behavioral categories.
Critical strengths
- The paper addresses a very important gap: Learning from rich feedback beyond reward scalars, namely text feedback. And they explore this on the increasingly relevant but understudied in this context field of social simulation.
- I like simplicity of idea: Add text feedback to prompt to guide behavior of new rollouts and add these to GRPO.
- DITTO achieves strong empirical improvements. DITTO even outperforms its own judge (GPT-5-nano) suggesting that judge's performance need not completely limit the downstream task performance.
Critical weaknesses
- I was surprised by how well GRPO without any text feedback performed. This raised the question: Is text feedback even worth it, or should we use instead the extra computation power to train GRPO for longer (more training steps)? Another tradeoff to analyze: DITTO requires double the rollouts per prompt w.r.t. GRPO. What happens if we instead trained GRPO on double the rollouts?
- I found it hard to understand your exact contribution w.r.t RLTF. In Section 3.4. you provide 3 reasons, but could you clarify them? Namely:
- Is reason 1 just that you substitute one optimizer (AWR) for another (GRPO)? If so, since both are established RL training methods, it would benefit from some more explanation of why GRPO is better suited for this task, if not, could you please clarify what you mean by (1)?
- RLTF also uses judge generated critics. Could you clarify what you mean by "judge-generated reflection rather than raw feedback"? What is the difference?
- I agree that the application to social simulation is important (I highlight this as one of your strengths). However, it seems like RLTF can also be applied to these environments (you do this in Section 6.2), so I am not sure this is a limitation of the RLTF approach itself.
- I could only find std values for Sotopia with 4 seeds. It would be nice to see std or IQR values on Figure 3 or Table 1.
- Minor presentation issues:
- Mention AWR stands for advantage-weighted regression when you introduce it in Section 3.4., not all the way down in Appendix E.
- Hard to interpret results when no clear definition of what the "primary metric" is provided in the main text.
Questions for authors
- Do the gains persist compared to GRPO under the same total training budget?
- Could you please clarify the distinctions between DITTO and RLTF approaches?
3 comments
Thank you for addressing my questions.
- The comparison with larger batch sizes and group size are very helpful to alleviate my concern that DITTO's gains were simply due to more rollouts. Ideally, the comparison would match the total computation cost (number of tokens, training time), but I found matching the number of rollouts already informative.
- I believe the paper would really benefit from incorporating your clarification of the differences between DITTO and RLTF-SD. Specially, the answer to 1. explaining how DITTO changes the advantage estimation. A question that comes up again is attribution, i.e., isolationg which changes drive the improvement. I recognize that the paper includes a revised-output-only ablation, which partially addresses this question. A more targeted experiment would compute DITTO's joint-group advantage (with same initial and revised outputs and loss fixed) but use AWR style method (-\sum A_{DITTO} \log \pi). I view this as a useful additional analysis, but not needed for my updated assessment.
Given these additional experiments and clarifications I am increasing my score to a 3.
Thank you for your thoughtful follow-up and for raising your score! We appreciate your constructive feedback and are glad our clarifications addressed your concerns. In the final version, we will include the rollout-matched and compute-matched GRPO comparison, incorporate the clarification of the differences between Ditto and RLTF-SD into Section 3.4 (especially how Ditto changes the advantage estimation), and add the attribution analysis you proposed, which applies Ditto’s joint-group advantage in an AWR-style objective with the same student and teacher outputs. Thank you again for your engagement throughout the discussion!
Thank you for the detailed review and for recognizing that learning from text feedback beyond scalar rewards addresses an important gap in social simulation, that the idea is simple, and that Ditto achieves strong improvements, even outperforming its own judge. We answer your questions as follows.
Yes. We gave GRPO Ditto's rollout budget in two ways, doubling the batch size or the rollouts per prompt, evaluated on Sotopia as in Table 2:
Both budgets improve GRPO (4.38 and 4.36 vs. 4.23), but it still falls behind Ditto (4.69). We will add this comparison to the paper.
Text feedback also helps on aspects the scalar reward does not directly optimize: on Sotopia, GRPO's secret-keeping score degrades during training, while Ditto preserves it (Section 6.2, Figure 3).
In summary, Ditto outperforms GRPO under the same rollout budget, so its gains come from verbal feedback, not just extra computation.
We clarify each reason below and will revise Section 3.4 accordingly.
Not only. The main difference is the teacher rollout's advantage: Ditto normalizes it jointly with the student rollouts by the group mean and std, whereas RLTF-SD subtracts the mean student reward:
Ditto:
Lgroup=−∑i=1G∑j∈0,1∑tmin(ρi,j,tAi,j, clip(ρi,j,t,1−ϵ,1+ϵ)Ai,j),Ai,j=std(rk,0,rk,1k=1G)ri,j−mean(rk,0,rk,1k=1G),ρi,j,t=πold(yi,j,t∣x,yi,j,<t)πθ(yi,j,t∣x,yi,j,<t)
RLTF-SD:
LSD=−λ∑i=1GAilogπθ(yi,1∣x),Ai=ri,1−G1∑k=1Grk,0,λ=0.1
Joint normalization puts student and teacher rollouts on one scale, suiting our continuous judge rewards whose scale varies across tasks; in our ablation, Ditto reaches 4.69 on Sotopia vs. 3.59 for RLTF-SD (Table 2).
Both methods use an LLM judge; the difference is what the teacher sees. In RLTF, the teacher sees its previous attempt y0 and the judge's critique of it, and revises y0. In Ditto, the teacher sees only the task prompt x and the judge's reflection h on how to improve, and attempts the task from scratch.
We agree that RLTF can be applied to social simulation. Section 3.4 lists this as a difference in the target setting rather than a limitation of RLTF, and we will clarify this in the paper.
Thank you for the suggestion. We will add std over seeds to Figure 3 and Table 1. On Sotopia, Ditto obtains 4.69 ± 0.02 over four seeds, and all four (4.65 to 4.72) exceed every GRPO configuration (at most 4.38), implying that Ditto robustly outperforms baselines.
Thank you for pointing these out. We will spell out advantage-weighted regression (AWR) in Section 3.4 and define the primary metric of each task in the main text (currently in Appendix D).