acceptodds
Under review as a conference paper at ICLR 2027

Memory-Use Policy Collapse in Role-Playing LLMs: Mitigation with Diversity-Controlled Post-Training

Abstract

Memory use in role-playing dialogue is inherently one-to-many, as the same scenario allows multiple valid memory-use policies. However, models may concentrate on only a few of these policies, producing repetitive responses that may weaken user immersion. We identify this phenomenon as Memory-Use Policy Collapse, which occurs in memory selection and memory use. We find this problem in strong API models, showing that valid memory use does not guarantee diverse memory use. To mitigate it, we introduce a plug-in memory planner between retrieval and response generation. The planner produces a memory-use plan that specifies which memories to use and how they should shape the next response. We use instance-specific rubrics to construct multiple valid plans with distinct plan-intent types for each scenario, and then train the planner with Diversified Supervised Fine-Tuning (D-SFT) followed by Diversity-Controlled GRPO (DC-GRPO). This training aims to improve plan validity while maintaining diversity. On the test set, DC-GRPO achieves a plan validity rate of 75.34%, compared with 60.16% for the vanilla SFT baseline. DC-GRPO increases Effective Plan-Intent Types@10 from 1.649 with the vanilla GRPO baseline to 4.978, achieving greater diversity among valid plans. Our planner also improves the validity and diversity of generated responses. Human evaluation further shows an overall preference for responses generated with DC-GRPO.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Role-playing LLMs, Long-term memory, Memory-use policy collapse, Memory planning, Reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_4uMNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies Memory-Use Policy Collapse in role-playing dialogue, where repeated generations favor a few memory choices and response intents despite several reasonable alternatives. The authors train a separate memory planner using D-SFT with multiple valid plans per state, followed by DC-GRPO. DC-GRPO combines validity rewards, rewards based on the number of memories used, and reweighting based on how often memory sets appear. The experiments assess the validity and diversity of plans and responses, with comparisons across six response models, human evaluation, and external benchmarks.

Critical strengths
  1. The paper usefully distinguishes individual response quality from the range of reasonable behaviors covered across repeated generations. Its separate treatment of memory selection, memory count, and plan intent helps explain different forms of repetition. The separate planner makes memory choices easier to inspect and can guide different response models, with results across six models supporting its practical usefulness.

  2. The experiments include useful controls and human evidence. Vanilla SFT and D-SFT use the same training states, number of examples, and total loss weight per state; component ablations and human judgments provide further support. The authors interpret the results carefully, acknowledging that the multi-memory reward mainly changes memory counts and that broader type coverage can come with a less even distribution.

Critical weaknesses

1. The consistency and reliability of the evaluation evidence need clarification.

The vanilla GRPO results in Table 2 appear inconsistent with Section 4.2. For a state with Ns>0N_s > 0 valid plans, let FsF_s count valid plan-intent types and DsD_s denote the most common type’s share. Each other type requires at least one sample. With K=10K = 10:

Fs≤1+Ns(1−Ds)≤11−10Ds.F_s \leq 1 + N_s(1 - D_s) \leq 11 - 10D_s.

Section 4.2 averages FsF_s over all states and DsD_s over states with valid plans. If qq is the fraction of states with valid plans:

Fˉ≤q(11−10Dˉ)≤11−10Dˉ.\bar{F} \leq q(11 - 10\bar{D}) \leq 11 - 10\bar{D}.

Table 2 reports Fˉ=2.103\bar{F} = 2.103 and Dˉ=0.892\bar{D} = 0.892, implying an upper bound of 2.080. Rounding does not resolve this discrepancy. Please check the values and whether both metrics use the same samples and stated counting and averaging rules. A short worked example would help; the effect on the conclusions depends on the explanation or correction.

GPT-5.5 is also used for data construction, RL validity judgments, and final automated evaluation, so shared judgment preferences may affect the reported gains. Human evaluation and the data audit offer useful independent evidence but cover the automated metrics only partially. The audit examines constructed plans and their responses, while the preference study assesses response quality; neither directly checks validity labels and type counts for each method’s test outputs in Tables 2 and 6.

In Table 7, 65.14 percent of audited plan pairs met the acceptance rule for distinct intents. Observed agreement was 68.14 percent, with Fleiss’ κ=0.006\kappa = 0.006; the uneven label distribution matters when interpreting kappa. Please provide evaluation prompts, intent-distinction and type-grouping rules, and difficult examples. A small check of existing test outputs by another model family or human annotators, with method identities hidden, could assess whether the main rankings and size of the gains remain similar. The size of any judge effect remains uncertain.

2. Clearer comparison settings and configuration choices would help establish the value of DC-GRPO.

Section 4.1 states that DC-GRPO starts from D-SFT, but vanilla GRPO’s starting checkpoint is unclear. Appendix A covers API planners and response models, while the Qwen planner’s sampling parameters and thinking mode during RL rollouts and test generation remain unclear. These details matter for comparisons across repeated samples.

The D-SFT and vanilla GRPO comparisons provide useful evidence. In Table 3, removing rarity reweighting raises Valid Rate from 75.34 to 77.80 percent and lowers Effective Types@10 from 4.978 to 4.732, supporting a tradeoff between validity and diversity. The benefit over other frequency-based methods remains less clear. GAPO (Anschel et al., 2025), already cited in the paper, adjusts rewards using valid-output frequencies within a rollout group; DC-GRPO adjusts positive advantages using valid plans’ memory-set frequencies. This distinction may matter, but GAPO assumes a predefined set of valid responses, so any adaptation would need explanation.

A comparison with an adapted GAPO or a simpler frequency-based reward would help assess the proposed advantage reweighting, keeping the D-SFT checkpoint, validity judge, rewards for memory counts, training budget, and decoding settings as similar as possible.

The choice of the full configuration also needs clarification. Removing the multi-memory reward raises Valid Rate from 75.34 to 77.18 percent and Effective Types@10 from 4.978 to 5.048, while the full method increases the share of plans using at least two memories from 10.14 to 31.44 percent (Table 3). Please explain the practical benefit of this shift and its tradeoff with validity and semantic coverage, using any existing response-level evidence. Please also report the number of independent training runs and how results were combined for the main methods and key ablations; uncertainty estimates would help interpret small differences.

Questions for authors
  1. For the vanilla GRPO row in Table 2, can the authors confirm the samples and averaging rules used for both metrics and explain the apparent inconsistency?

  2. Did vanilla GRPO and DC-GRPO start from the same D-SFT checkpoint and use the same training budget and sampling settings?

  3. What practical benefit motivates the higher use of multiple memories in the full configuration, and is there existing response-level evidence that supports this choice?

1 comment

Thank you for the detailed response and additional experiments. The correction to Table 2 and clarification of the training settings resolve my earlier questions. The cross-judge evaluation and human validation also strengthen my confidence in the results.

The GAPO-style comparison shows higher validity for DC-GRPO and greater semantic diversity for the baseline. I recognize the value of this trade-off, but remain unsure whether the demonstrated advance is substantial enough to support acceptance. The new multi-memory reward ablation is limited to one fixed response model. Whether this component provides similar benefits with other response models remains unclear.

I appreciate the stronger evidence and acknowledge that several of my concerns have been addressed. On balance, I maintain my rating of 2/4.

Reviewer_fry3OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates Memory-Use Policy Collapse in role-playing LLMs, where models repeatedly favor a limited subset of valid memory-use strategies despite the availability of multiple reasonable alternatives. The authors characterize this phenomenon along three dimensions: memory selection, memory cardinality, and plan-intent diversity.

To address this problem, the paper introduces a modular memory planner trained through Diversified Supervised Fine-Tuning (D-SFT) and Diversity-Controlled GRPO (DC-GRPO). D-SFT uses multiple rubric-guided valid plans for each dialogue scenario, while DC-GRPO combines validity-gated memory-cardinality shaping and rarity-based advantage reweighting to encourage diverse memory selections.

Experiments on CoSER-derived scenarios demonstrate improvements in plan validity and diversity. The proposed approach also improves response-level diversity across six speaker models, with additional evidence from human preference evaluations and external memory benchmarks.

Critical strengths

S1. Meaningful problem formulation and a well-motivated approach.

The paper identifies an interesting limitation of existing memory-augmented role-playing systems: valid memory use does not necessarily imply diverse memory-use policies. The distinction between memory-selection collapse, memory-cardinality collapse, and plan-intent collapse provides a useful framework for analyzing repetitive memory behavior. Separating memory planning from response generation is also a practical design choice that enables the method to be applied to different speaker models without retraining them.

S2. Comprehensive experimental evaluation with encouraging results.

The experiments provide consistent evidence that diversity-aware post-training improves memory-use behavior. DC-GRPO increases plan validity from 63.74% with D-SFT to 75.34%, while improving Effective Plan-Intent Types@10 from 4.567 to 4.978. Compared with vanilla GRPO, the proposed method substantially improves diversity, although at the cost of lower validity.

The evaluation extends beyond planner-level metrics to six downstream speaker models, human preference comparisons, and external benchmarks. DC-GRPO achieves a 63.1% human preference win rate against direct generation and 59.5% against D-SFT. These evaluations strengthen the practical relevance of the proposed framework.

Critical weaknesses

W1. Limited reliability of the evaluation framework, particularly for semantic diversity.

A central concern is the reliability of the automatically evaluated validity and plan-intent diversity metrics.

The same instance-specific rubrics are used to guide data construction, provide rewards during reinforcement learning, and evaluate generated plans. Although the evaluation uses a held-out test set, this shared evaluation framework may introduce systematic biases and potentially favor behaviors aligned with the rubric-generation process.

More importantly, Appendix Table 7 reports extremely low inter-annotator agreement measured by Fleiss' kappa: 0.026 for canonical-plan validity, 0.006 for plan-intent distinctness, and -0.027 for response realization validity. These values raise substantial concerns about the reliability of the underlying annotation criteria, especially because plan-intent distinctness is central to the paper's main contribution.

The authors should provide additional validation of the automatic judge against independent human annotations, including agreement statistics and error analysis.

W2. The relationship between reward design, semantic diversity, and response quality requires stronger justification.

DC-GRPO explicitly rewards memory-cardinality patterns and reweights advantages according to the rarity of selected-memory sets. However, the primary objective is to improve semantic plan-intent diversity rather than merely diversify memory selections.

The current experiments do not fully establish whether the observed improvements arise from genuinely richer memory-use policies or from the reward mechanism encouraging broader memory-selection patterns.

Furthermore, Table 2 shows a substantial validity-diversity trade-off: vanilla GRPO achieves 91.73% plan validity, whereas DC-GRPO achieves 75.34%. While the latter provides significantly greater diversity, the paper would benefit from a more systematic analysis of this trade-off.

The component ablations also suggest that not every design choice improves the primary diversity metric. In particular, removing the multi-memory reward slightly increases Effective Types@10 from 4.978 to 5.048 and improves validity from 75.34% to 77.18%. This raises questions about the necessity of this component for the central objective.

Questions for authors

Q1. Evaluation reliability. Appendix Table 7 reports Fleiss' kappa values close to zero for all three human-audit targets, including 0.006 for plan-intent distinctness. Could the authors explain this low agreement and provide additional evidence supporting the reliability of the automatic plan-intent classification and validity judgments?

Q2. Independent evaluation. Since instance-specific rubrics are used during both training and evaluation, could the authors evaluate the trained planners using an independently constructed rubric set or a different LLM judge? This would help determine whether the reported improvements generalize beyond the original evaluation framework.

Q3. Reward design. DC-GRPO encourages diversity through memory cardinality and selected-set rarity rather than directly optimizing plan-intent diversity. Could the authors provide evidence that these memory-selection changes lead to genuinely distinct semantic policies rather than superficial variations in selected memories?

Q4. Component necessity. According to Table 3, removing the multi-memory reward improves both validity (77.18% vs. 75.34%) and Effective Types@10 (5.048 vs. 4.978). What additional benefit justifies retaining this component? Could the authors clarify whether memory-cardinality diversity is independently important for downstream response quality?

Q5. Validity-diversity trade-off. Vanilla GRPO achieves substantially higher plan validity (91.73%) than DC-GRPO (75.34%). Could the authors provide a validity-diversity Pareto analysis, for example by varying reward coefficients, to better characterize whether the proposed method achieves a favorable balance?

Q6. Statistical robustness. Could the authors provide confidence intervals or statistical significance tests for human preference results, particularly the 59.5% win rate of DC-GRPO over D-SFT? Additional details regarding the number of independent judgments and annotator agreement would help establish the robustness of these findings.

Reviewer_chAwOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies the Memory-Use Policy Collapse problem in the LLM role playing task, where the planner model lacks diversity in generating the plans for the response generation module. The paper proposes a post-training pipeline to improve the diversity of the planner. The pipeline includes three main stages: (1) generating the rubric-guided dataset, which contains multiple plans per state, (2) D-SFT: fine-tune the planner on this multi-reference dataset, (3)DC-GRPO: a GRPO variant with customized reward and advantage functions to prevent the collapse. Results show that DC-GRPO planner produces more diverse plan and memory selection than SFT and planners based on proprietary models (e.g., GPT-5.6-Terra). Finally, human evaluation shows a trend of preferring the responses generated based on the output of DC-GRPO planner, over those from D-SFT.

Critical strengths
  • DC-GRPO algorithm is sound and well-motivated
  • The proposed model outperforms the baselines in diversity on both the plan generation and response generation
Critical weaknesses
  • The paper use the same judge (GPT-5.5) for evaluating the validity in both training and testing the model, which raises the concern of reward hacking
  • The paper does not share the prompts used for model training, LLM-judge (e.g., validity evaluated, intent type), rubric-generation, which makes those parts unclear, and cannot be assessed.
  • The inter-annotator agreement is not reported for the human preference experiment (Figure 3). The paper does not report the number of annotators for Table 7. The inter-annotator agreement in the data audit (Table 7) is weak, Fleiss’ κ\kappa ranging from -0.027 to 0.026.
  • The paper uses LLM-as-a-judge in several places: judging the plan validity, response validity, intent type, but the reliability of the LLM is never evaluated against human judgement. This raises the concern that the evaluation metrics do not correctly reflect the model performance.
Questions for authors

Please check the Weaknesses

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.