acceptodds
Withdrawn from ICLR 2027

Interact, Remember, and Act: Preference and Progress-Driven Reinforcement Learning for LLM Agents

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Effective collaboration between users and large language model (LLM) agents depends on both successful task execution and interactions that accommodate user preferences. However, existing agentic reinforcement learning (RL) approaches incorporating user simulator provide limited modeling of agents' capabilities and user experience, with no unified environment across heterogeneous tasks, and efforts to improve user experience and long-horizon credit assignment remain largely separate. In order to address these issues, we introduce PPD-RL, a multi-turn agentic RL framework jointly driven by user preferences and task progress. Specifically, PPD-RL defines a standardized action space for agents encompassing clarification queries, tool use, memory revisions, and final answers, together with unified interaction environments for training and inference. Moreover, to enable personalized assistance across sessions, the agent maintains an explicit user profile, updates it when certain attributes in the profile are missing or drifting, and conditions subsequent interactions on the corrected memory. Concurrently, to improve these behaviors, PPD-RL combines the task goal, rule-based memory feedback, and an LLM-judged preference score as RL rewards. It then distributes credit across multi-turns using positive rewards and measurable task progress, linking the learning signal to actions that advance task completion. We train and evaluate PPD-RL across various gym families spanning travel planning, customer service, and collaborative programming, measuring both task performance and user experience. Experimental results demonstrate that PPD-RL outperforms existing methods in this area and the larger models included in our evaluation on both dimensions. Code is available at https://anonymous.4open.science/r/PPD-RL.

Primary area:
reinforcement learning
Keywords:
LLM agents, Reinforcement learning, User-centric, Multi-turn interaction, Credit assignment

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_hS37OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents PPD-RL, a multi-turn RL framework for LLM agents that interact with simulated users across four environments through a shared action interface. The agent keeps an explicit user profile and is rewarded for correcting it when the user implicitly reveals a preference, alongside task rewards and an LLM-judged preference score. Turn credit mixes per-turn reward with a hand-designed progress signal.

Critical strengths

Training agents that both complete tasks and adapt to user preferences is a relevant goal, and making profile correction an explicit, rule-verifiable action is a clean way to make part of it measurable.

Critical weaknesses
  1. The paper builds heavily on UserRL but presents much of it as new. UserRL is cited only in related work and as a baseline, yet TravelGym and TauGym, the “interact with env(choice, content)” interface, the turn-credit options the paper refers to (including the “Equalized” baseline mentioned in the limitations), reward-to-go trajectory scoring, and the SFT-then-multi-turn-GRPO recipe on Qwen3-4B all appear in UserRL. None of this is acknowledged where these components are introduced, and the standardized action space and environment are listed as a main contribution. What is new is a memory operation, two additional environments, a fairly simple profile layer, and a modified turn-credit rule, and the paper should be explicit about this.

  2. The credit assignment does not match its description and is narrow in scope. The introduction describes measuring how strongly behaviors converge across successful trajectories, but Eq. 4 is only a weighted mix of normalized per-turn rewards and a hand-designed progress count, with no cross-trajectory statistic. It is also active only on successful TravelGym trajectories; the other three environments use uniform credit, so the reported differences there can only arise indirectly. The appeal to potential-based shaping does not apply, since progress is clipped at zero and used as a weight on the advantage rather than added as a reward. The ablation compares only against uniform credit; the reward-to-go and EM variants mentioned in the text are not reported.

  3. The evaluation may not support the main claims. The same LLM judge serves as both a training reward and the preference metric, and the same simulator is used for training and testing, so the preference gains over baselines not trained with this judge may be expected and say little about real user experience. Most of the improvement may come from SFT, and it is not stated which model generated the SFT trajectories. It is also unclear whether the baselines were retrained on the same gyms and data, and the tables report no variance across seeds. Finally, memory is reset every episode, so the cross-session personalization claimed in the abstract and introduction may be never tested.

Questions for authors

How were the SFT trajectories generated, and how do UserRL and InfoPO perform when trained on the same gyms and data?

How do the results change with a different judge and user simulator at evaluation time, and how do the reward-to-go and EM credit variants compare?

Reviewer_rsd5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PPD-RL trains a Qwen3-4B agent across four interactive environments using a common interface for communication, tool use, memory updates, and task actions. A synthetic user profile specifies four communication preferences; the agent may need to correct one missing or incorrect attribute after a user disclosure. Training combines task rewards, rule-based memory feedback, and an LLM-judged preference score. The implemented credit rule described in Section 4.3 combines positive reward and progress in TravelGym and uses uniform credit where progress is unavailable. Experiments compare the full SFT-plus-RL recipe against general-purpose models and agentic RL baselines, with ablations of training stages, the profile layer, and credit assignment. The full system improves both aggregate task performance and judged preference adherence.

Critical strengths
  1. The profile-update task makes a useful behavior explicit and verifiable. Correcting a missing or outdated preference during task execution is more informative than simply supplying a persona in the initial prompt. The rule-based memory action separates correct updates from fluent but incorrect acknowledgments, and the common interface makes it possible to train this behavior across several task backends.
  2. The complete training recipe produces meaningful gains and includes relevant controls. Relative to SFT alone, adding RL improves every task column and preference adherence in Table 3. Table 4 uses the same SFT start and hyperparameters for component ablations, and the paper reports the ColGym trade-off when the profile layer is removed rather than presenting personalization as uniformly beneficial.
Critical weaknesses
  1. Two central contribution claims do not match the specified implementation. The introduction and contribution list describe credit derived from action-space convergence across successful trajectories. Section 4.3 instead gives a weighted positive-reward/progress rule and explicitly falls back to uniform credit in TauGym, Tau2Gym, and ColGym; I could not identify the claimed convergence statistic. Separately, the abstract and introduction emphasize memory across sessions, while Appendix A.3 states that memory resets from init_memory and that durable cross-session storage is not assumed in the experiments. These discrepancies affect what method and capability have actually been demonstrated. Clarifying the implemented method and narrowing the memory claim would be preferable to adding an unrelated new evaluation suite.
  2. The profile ablation does not isolate the claimed benefit of explicit memory. Table 4 removes the initial memory, memory action, and preference reward together. The preference-score drop from 0.743 to 0.627 therefore supports the profile-and-reward package, but cannot by itself support the statement that it “directly supports explicit memory.” A policy receiving the same initial profile information, preference reward, and user disclosures, but relying on conversation context rather than writable memory, is the relevant missing comparison. This matters because explicit memory is a main contribution, not a peripheral design choice.
Questions for authors
  1. Which credit-assignment implementation produced Tables 2–4, and where does the action-space convergence statistic enter it? Can you also confirm that the reported personalization experiments are confined to individual episodes, as Appendix A.3 states?
  2. With initial profile information, preference reward, and user disclosures held fixed, does writable memory improve subsequent preference adherence over keeping the same information in conversation context? Existing results for this comparison, if available, would materially strengthen my assessment.
Reviewer_hewEOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how a multi-turn LLM agent can simultaneously improve task completion and adherence to user preferences. The authors propose Preference and Progress-Driven Reinforcement Learning (PPD-RL), employing four action categories—interact, tool, memory, and answer. Explicit user profiles record tone, response detail, address style, and terminology usage preferences. The memory action is used to complete missing attributes or correct changed attributes. Training combines SFT cold start with multi-turn GRPO. Rewards are derived from task results, rule-checked memory updates, and preference scores evaluated by the LLM.

Critical strengths

The paper uses a fixed user profile schema and defines scoring rules for missing or changing attributes, repeated modifications, and erroneous modifications. This allows the evaluation to not only determine the style of the final conversation but also record and verify whether the agent took action based on the user's corrections. Four gyms covering travel planning, customer service, and programming employ different forms of task rewards, including intensive, sparse, continuous, and binary.

Critical weaknesses
  • The w/o user profile ablation experiments in Table 4 simultaneously removed the initial memory, memory actions, and preference rewards. Please retain preference rewards when testing settings with no memory and frozen memory to separately evaluate the value of explicit memory. Training and evaluation used the same score judged by LLM, requiring the introduction of an independent judge or human review.

  • Section 4.3, Eq. (4), only weights the normalized positive reward and progress within a single trajectory. Furthermore, progress-based contribution allocation is only used for TravelGym; the other three gyms use uniform allocation. Please provide a consistent method definition and distinguish between the direct effects of this mechanism and its potential multi-task transfer effects.

Questions for authors

Please test other user simulators. Using the same simulator for training and testing controls one factor, but does not show that the agent handles different user behavior well.

1 comment

Dear Authors,

we have not yet received a response and would appreciate an update during the rebuttal period.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.