- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents PPD-RL, a multi-turn RL framework for LLM agents that interact with simulated users across four environments through a shared action interface. The agent keeps an explicit user profile and is rewarded for correcting it when the user implicitly reveals a preference, alongside task rewards and an LLM-judged preference score. Turn credit mixes per-turn reward with a hand-designed progress signal.
Critical strengths
Training agents that both complete tasks and adapt to user preferences is a relevant goal, and making profile correction an explicit, rule-verifiable action is a clean way to make part of it measurable.
Critical weaknesses
-
The paper builds heavily on UserRL but presents much of it as new. UserRL is cited only in related work and as a baseline, yet TravelGym and TauGym, the “interact with env(choice, content)” interface, the turn-credit options the paper refers to (including the “Equalized” baseline mentioned in the limitations), reward-to-go trajectory scoring, and the SFT-then-multi-turn-GRPO recipe on Qwen3-4B all appear in UserRL. None of this is acknowledged where these components are introduced, and the standardized action space and environment are listed as a main contribution. What is new is a memory operation, two additional environments, a fairly simple profile layer, and a modified turn-credit rule, and the paper should be explicit about this.
-
The credit assignment does not match its description and is narrow in scope. The introduction describes measuring how strongly behaviors converge across successful trajectories, but Eq. 4 is only a weighted mix of normalized per-turn rewards and a hand-designed progress count, with no cross-trajectory statistic. It is also active only on successful TravelGym trajectories; the other three environments use uniform credit, so the reported differences there can only arise indirectly. The appeal to potential-based shaping does not apply, since progress is clipped at zero and used as a weight on the advantage rather than added as a reward. The ablation compares only against uniform credit; the reward-to-go and EM variants mentioned in the text are not reported.
-
The evaluation may not support the main claims. The same LLM judge serves as both a training reward and the preference metric, and the same simulator is used for training and testing, so the preference gains over baselines not trained with this judge may be expected and say little about real user experience. Most of the improvement may come from SFT, and it is not stated which model generated the SFT trajectories. It is also unclear whether the baselines were retrained on the same gyms and data, and the tables report no variance across seeds. Finally, memory is reset every episode, so the cross-session personalization claimed in the abstract and introduction may be never tested.
Questions for authors
How were the SFT trajectories generated, and how do UserRL and InfoPO perform when trained on the same gyms and data?
How do the results change with a different judge and user simulator at evaluation time, and how do the reward-to-go and EM credit variants compare?
Dear Authors,
we have not yet received a response and would appreciate an update during the rebuttal period.
Thank you.