acceptodds
Under review as a conference paper at ICLR 2027

Learning What to Practice Next: Experience-Adaptive Self-Evolution for Multi-Turn Empathetic Dialogue

Abstract

Reinforcement learning with verifiable feedback typically updates model parameters while leaving the distribution of subsequent experience fixed. This separation is especially limiting in multi-turn empathetic dialogue, where interaction conditions shape entire trajectories and their training value changes as the policy improves. We introduce experience-adaptive self-evolution, which turns verified emotion outcomes into an online signal for deciding what the policy should practice next. Continuous outcomes train the dialogue policy, while thresholded group outcomes estimate the training value of reusable intent-state units and reallocate future interactions. Hierarchical evidence sharing stabilizes sparse estimates, and uncertainty-guided exploration with uniform rehearsal preserves coverage. With the simulator and verifier frozen, the framework adapts experience without changing the role-playing target or increasing the rollout budget. On SAGE, it raises Qwen3-8B Overall from 53.87 to 79.24, outperforming protocol-matched uniform emotion-reward RL by 7.23 points; the advantage also holds at Qwen3-4B by 4.14 points. Cross-benchmark and human evaluations further show improvement beyond the training-aligned metric.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Reinforcement Learning, Empathetic Dialogue, Multi-Turn Conversation, Experience-Adaptive Learning, Verifiable Feedback, Curriculum Learning, Self-Evolution

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_M6zdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes an experience-adaptive self-evolution framework for multi-turn empathetic dialogue reinforcement learning. The work addresses the policy–experience gap, where policy competence evolves during training while the distribution of subsequent training interactions remains fixed. The proposed framework organizes training environments using predefined support intents and controllable interaction states. Continuous terminal emotion rewards are used for GRPO policy optimization, while thresholded group outcomes are accumulated to estimate the training value of intent–state units. A controller then adjusts the sampling distribution over interaction states using historical pass rates, hierarchical evidence sharing, and uncertainty-guided exploration. Experiments on Qwen3-8B and Qwen3-4B show improvements over uniform emotion-reward RL under matched rollout budgets. Additional ablations, cross-benchmark evaluations, and human assessments provide evidence for the effectiveness of the proposed framework.

Critical strengths
  1. Clear motivation and practical framework. The paper identifies a meaningful limitation of fixed experience distributions in multi-turn reinforcement learning. The proposed framework adapts the conditions under which future interactions are generated while keeping the simulator, underlying scenario semantics, and verifier fixed. This provides a practical way to improve experience allocation without modifying the underlying GRPO algorithm or increasing the rollout budget.
  2. Consistent empirical improvements and targeted ablations. The proposed method improves SAGE Overall from 72.01 to 79.24 on Qwen3-8B and from 44.73 to 48.87 on Qwen3-4B compared with uniform emotion-reward RL. The main comparisons include three independent training runs. Furthermore, the shuffled-feedback and frozen-distribution ablations provide useful evidence that condition-aligned feedback and online allocation contribute to the performance gains.
Critical weaknesses
  1. Limited methodological novelty relative to existing adaptive curriculum learning methods. The central idea of allocating more training resources to conditions near the policy's competence boundary is closely related to existing adaptive curriculum learning approaches, including AdaRFT and VCRL. These methods already explore training-resource allocation based on success rates, reward variability, or estimated learning utility. The proposed framework extends this principle to multi-turn empathetic dialogue by constructing reusable intent–state units and adapting their sampling probabilities. While this is a reasonable domain-specific extension, the additional methodological contribution remains relatively limited. Moreover, stronger comparisons with existing adaptive curriculum strategies adapted to the same interaction-state space would help clarify whether the improvements primarily arise from the proposed controller or from introducing an appropriate controllable environment representation.
  2. Dependence on a manually constructed interaction-state representation. The framework relies on eight predefined support intents and 24 interaction states obtained from three manually selected behavioral dimensions. While the interaction-state design is intuitive and the authors provide an adherence audit, its sensitivity and generalizability remain unclear. In particular, the audit reports that all three behavioral dimensions are simultaneously recovered correctly in 68.3% of trajectories by a held-out judge. This result suggests that the intended interaction conditions are not always unambiguously reflected in the generated dialogues, potentially introducing noise into the controller's condition-level statistics. Furthermore, the paper does not sufficiently investigate alternative behavioral dimensions, different discretization granularities, or settings in which users' support needs evolve during a conversation. A more systematic analysis would help establish whether the proposed factorization is broadly effective or primarily tailored to the current simulator and dataset.
Questions for authors
  1. Comparison with prior curriculum-learning approaches: Could the authors provide further comparisons with adapted success-rate-based curriculum methods, such as AdaRFT, under the same intent–state representation and rollout budget? Would additional independent training runs for VCRL be feasible?
  2. Validity of the training-value estimator: Is there empirical evidence that interaction conditions with pass rates near 0.5 produce larger subsequent policy improvements? How does the proposed accumulated pass-rate estimator compare with estimators based on recent success rates, measured learning progress, or continuous reward statistics?
  3. Robustness of interaction-state construction: How sensitive is the framework to the selected behavioral dimensions and their discretization granularity? How does imperfect realization of the intended interaction states affect the reliability of the controller?
  4. Interpretation of allocation dynamics: The reported controller distributions remain relatively close to uniform, despite substantial improvements on SAGE. Could the authors provide a more detailed analysis of intent-specific allocation changes and their relationship to training gains?
Reviewer_S9QDOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the mismatch between evolving policy capabilities and fixed training experience distributions during reinforcement learning for large language models. It proposes an adaptive experience allocation framework guided by historical feedback. Its core mechanism is supported by targeted ablation studies, and the paper provides comprehensive experimental evaluations and reproducibility materials, making the results convincing. Although the adaptability of cumulative historical statistics under non-stationary policies warrants further investigation, this limitation does not undermine the contributions demonstrated under the current experimental settings. Overall, I believe this work makes a valuable research contribution supported by solid empirical evidence, and I recommend acceptance.

Critical strengths

1.Comprehensive Experimental Validation and Strong Reproducibility. The method uses continuous emotion rewards to update the GRPO policy, while thresholded feedback from the same trajectory groups is used to update the experience allocation controller. The DualLoopRolloutAdapter ensures that trajectories within the same GRPO group share the same scenario and interaction state, allowing experience allocation across groups to be updated based on historical outcomes without modifying the simulator or reward function. In particular, the Shuffled Feedback and Frozen Final Distribution experiments directly demonstrate the importance of condition-aligned feedback and online experience allocation. Furthermore, the authors provide source code, training configurations, and experimental data, strengthening the credibility and reproducibility of the results.

2.The implementation employs Leave-One-Intent statistical estimation to prevent observations from the target intent from being double-counted in the prior. It also combines a success–failure boundary score with an uncertainty bonus to dynamically adjust the sampling probabilities of training states, balancing the reliability of sparse experience estimates with the need for exploration.

Critical weaknesses

The paper employs cumulative historical statistics to estimate the training value of different interaction states. However, as the reinforcement learning policy continuously evolves, earlier experiences may gradually become less representative of the policy's current capabilities. The current implementation does not incorporate explicit temporal decay or selective forgetting mechanisms, which may cause the controller to adapt slowly to rapid changes in policy competence. Although the applicability of the proposed experience allocation mechanism beyond the evaluated settings could be explored further, I do not identify a critical weakness that substantially undermines the paper's central contributions.

Questions for authors

While the proposed online updates and exploration mechanisms help maintain responsiveness to policy changes, earlier experiences may become less representative of its current capabilities. Have the authors examined whether temporal decay or selective forgetting would improve adaptation, particularly during later stages of training? The question is offered as a suggestion for future research rather than a requirement for acceptance.

Reviewer_ZPUQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper proposes an Experience-Adaptive Self-Evolution reinforcement learning framework designed for multi-turn empathetic support dialogues. The core idea is to leverage emotional feedback from historical conversations to dynamically adjust the sampling probabilities of various interaction states during subsequent training, thereby ensuring that the model focuses its practice on interaction conditions that currently offer the greatest learning value.

Critical strengths
  1. The methodological design is comprehensive, balancing exploration with stability. 2.Good experimental performance. On Qwen3-8B, the SAGE Overall score improved from 72.01 (achieved by RLVER) to 79.24; a gain of 4.14 points was also observed on Qwen3-4B.
Critical weaknesses
  1. Missing competitive baselines. Given that EIBench introduces CTC-GRPO, why is this method not compared or discussed when evaluating on EIBench? 2.The proposed framework largely combines established adaptive curriculum learning principles, success-rate-based prioritization, uncertainty-driven exploration, and uniform rehearsal. Its methodological distinction from recent adaptive RL methods is insufficiently demonstrated.
Questions for authors

see above

Comments on the paper

1 comment
AuthorsOpenReview ↗

Follow-up on our author responses

Dear reviewers,

Thank you for your feedback. We wanted to briefly follow up on our author responses, which include an EIBench comparison with CTC-GRPO, comparisons with AdaRFT and VCRL under the same interaction-state space and rollout budget, and additional results on allocation estimators and temporal decay.

If any of your concerns remain open, we would be happy to clarify. Thank you for taking the time to consider our responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.