- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes an experience-adaptive self-evolution framework for multi-turn empathetic dialogue reinforcement learning. The work addresses the policy–experience gap, where policy competence evolves during training while the distribution of subsequent training interactions remains fixed. The proposed framework organizes training environments using predefined support intents and controllable interaction states. Continuous terminal emotion rewards are used for GRPO policy optimization, while thresholded group outcomes are accumulated to estimate the training value of intent–state units. A controller then adjusts the sampling distribution over interaction states using historical pass rates, hierarchical evidence sharing, and uncertainty-guided exploration. Experiments on Qwen3-8B and Qwen3-4B show improvements over uniform emotion-reward RL under matched rollout budgets. Additional ablations, cross-benchmark evaluations, and human assessments provide evidence for the effectiveness of the proposed framework.
Critical strengths
- Clear motivation and practical framework. The paper identifies a meaningful limitation of fixed experience distributions in multi-turn reinforcement learning. The proposed framework adapts the conditions under which future interactions are generated while keeping the simulator, underlying scenario semantics, and verifier fixed. This provides a practical way to improve experience allocation without modifying the underlying GRPO algorithm or increasing the rollout budget.
- Consistent empirical improvements and targeted ablations. The proposed method improves SAGE Overall from 72.01 to 79.24 on Qwen3-8B and from 44.73 to 48.87 on Qwen3-4B compared with uniform emotion-reward RL. The main comparisons include three independent training runs. Furthermore, the shuffled-feedback and frozen-distribution ablations provide useful evidence that condition-aligned feedback and online allocation contribute to the performance gains.
Critical weaknesses
- Limited methodological novelty relative to existing adaptive curriculum learning methods. The central idea of allocating more training resources to conditions near the policy's competence boundary is closely related to existing adaptive curriculum learning approaches, including AdaRFT and VCRL. These methods already explore training-resource allocation based on success rates, reward variability, or estimated learning utility. The proposed framework extends this principle to multi-turn empathetic dialogue by constructing reusable intent–state units and adapting their sampling probabilities. While this is a reasonable domain-specific extension, the additional methodological contribution remains relatively limited. Moreover, stronger comparisons with existing adaptive curriculum strategies adapted to the same interaction-state space would help clarify whether the improvements primarily arise from the proposed controller or from introducing an appropriate controllable environment representation.
- Dependence on a manually constructed interaction-state representation. The framework relies on eight predefined support intents and 24 interaction states obtained from three manually selected behavioral dimensions. While the interaction-state design is intuitive and the authors provide an adherence audit, its sensitivity and generalizability remain unclear. In particular, the audit reports that all three behavioral dimensions are simultaneously recovered correctly in 68.3% of trajectories by a held-out judge. This result suggests that the intended interaction conditions are not always unambiguously reflected in the generated dialogues, potentially introducing noise into the controller's condition-level statistics. Furthermore, the paper does not sufficiently investigate alternative behavioral dimensions, different discretization granularities, or settings in which users' support needs evolve during a conversation. A more systematic analysis would help establish whether the proposed factorization is broadly effective or primarily tailored to the current simulator and dataset.
Questions for authors
- Comparison with prior curriculum-learning approaches: Could the authors provide further comparisons with adapted success-rate-based curriculum methods, such as AdaRFT, under the same intent–state representation and rollout budget? Would additional independent training runs for VCRL be feasible?
- Validity of the training-value estimator: Is there empirical evidence that interaction conditions with pass rates near 0.5 produce larger subsequent policy improvements? How does the proposed accumulated pass-rate estimator compare with estimators based on recent success rates, measured learning progress, or continuous reward statistics?
- Robustness of interaction-state construction: How sensitive is the framework to the selected behavioral dimensions and their discretization granularity? How does imperfect realization of the intended interaction states affect the reliability of the controller?
- Interpretation of allocation dynamics: The reported controller distributions remain relatively close to uniform, despite substantial improvements on SAGE. Could the authors provide a more detailed analysis of intent-specific allocation changes and their relationship to training gains?
Follow-up on our author responses
Dear reviewers,
Thank you for your feedback. We wanted to briefly follow up on our author responses, which include an EIBench comparison with CTC-GRPO, comparisons with AdaRFT and VCRL under the same interaction-state space and rollout budget, and additional results on allocation estimators and temporal decay.
If any of your concerns remain open, we would be happy to clarify. Thank you for taking the time to consider our responses.