- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies Memory-Use Policy Collapse in role-playing dialogue, where repeated generations favor a few memory choices and response intents despite several reasonable alternatives. The authors train a separate memory planner using D-SFT with multiple valid plans per state, followed by DC-GRPO. DC-GRPO combines validity rewards, rewards based on the number of memories used, and reweighting based on how often memory sets appear. The experiments assess the validity and diversity of plans and responses, with comparisons across six response models, human evaluation, and external benchmarks.
Critical strengths
-
The paper usefully distinguishes individual response quality from the range of reasonable behaviors covered across repeated generations. Its separate treatment of memory selection, memory count, and plan intent helps explain different forms of repetition. The separate planner makes memory choices easier to inspect and can guide different response models, with results across six models supporting its practical usefulness.
-
The experiments include useful controls and human evidence. Vanilla SFT and D-SFT use the same training states, number of examples, and total loss weight per state; component ablations and human judgments provide further support. The authors interpret the results carefully, acknowledging that the multi-memory reward mainly changes memory counts and that broader type coverage can come with a less even distribution.
Critical weaknesses
1. The consistency and reliability of the evaluation evidence need clarification.
The vanilla GRPO results in Table 2 appear inconsistent with Section 4.2. For a state with valid plans, let count valid plan-intent types and denote the most common type’s share. Each other type requires at least one sample. With :
Section 4.2 averages over all states and over states with valid plans. If is the fraction of states with valid plans:
Table 2 reports and , implying an upper bound of 2.080. Rounding does not resolve this discrepancy. Please check the values and whether both metrics use the same samples and stated counting and averaging rules. A short worked example would help; the effect on the conclusions depends on the explanation or correction.
GPT-5.5 is also used for data construction, RL validity judgments, and final automated evaluation, so shared judgment preferences may affect the reported gains. Human evaluation and the data audit offer useful independent evidence but cover the automated metrics only partially. The audit examines constructed plans and their responses, while the preference study assesses response quality; neither directly checks validity labels and type counts for each method’s test outputs in Tables 2 and 6.
In Table 7, 65.14 percent of audited plan pairs met the acceptance rule for distinct intents. Observed agreement was 68.14 percent, with Fleiss’ ; the uneven label distribution matters when interpreting kappa. Please provide evaluation prompts, intent-distinction and type-grouping rules, and difficult examples. A small check of existing test outputs by another model family or human annotators, with method identities hidden, could assess whether the main rankings and size of the gains remain similar. The size of any judge effect remains uncertain.
2. Clearer comparison settings and configuration choices would help establish the value of DC-GRPO.
Section 4.1 states that DC-GRPO starts from D-SFT, but vanilla GRPO’s starting checkpoint is unclear. Appendix A covers API planners and response models, while the Qwen planner’s sampling parameters and thinking mode during RL rollouts and test generation remain unclear. These details matter for comparisons across repeated samples.
The D-SFT and vanilla GRPO comparisons provide useful evidence. In Table 3, removing rarity reweighting raises Valid Rate from 75.34 to 77.80 percent and lowers Effective Types@10 from 4.978 to 4.732, supporting a tradeoff between validity and diversity. The benefit over other frequency-based methods remains less clear. GAPO (Anschel et al., 2025), already cited in the paper, adjusts rewards using valid-output frequencies within a rollout group; DC-GRPO adjusts positive advantages using valid plans’ memory-set frequencies. This distinction may matter, but GAPO assumes a predefined set of valid responses, so any adaptation would need explanation.
A comparison with an adapted GAPO or a simpler frequency-based reward would help assess the proposed advantage reweighting, keeping the D-SFT checkpoint, validity judge, rewards for memory counts, training budget, and decoding settings as similar as possible.
The choice of the full configuration also needs clarification. Removing the multi-memory reward raises Valid Rate from 75.34 to 77.18 percent and Effective Types@10 from 4.978 to 5.048, while the full method increases the share of plans using at least two memories from 10.14 to 31.44 percent (Table 3). Please explain the practical benefit of this shift and its tradeoff with validity and semantic coverage, using any existing response-level evidence. Please also report the number of independent training runs and how results were combined for the main methods and key ablations; uncertainty estimates would help interpret small differences.
Questions for authors
-
For the vanilla GRPO row in Table 2, can the authors confirm the samples and averaging rules used for both metrics and explain the apparent inconsistency?
-
Did vanilla GRPO and DC-GRPO start from the same D-SFT checkpoint and use the same training budget and sampling settings?
-
What practical benefit motivates the higher use of multiple memories in the full configuration, and is there existing response-level evidence that supports this choice?
Thank you for the detailed response and additional experiments. The correction to Table 2 and clarification of the training settings resolve my earlier questions. The cross-judge evaluation and human validation also strengthen my confidence in the results.
The GAPO-style comparison shows higher validity for DC-GRPO and greater semantic diversity for the baseline. I recognize the value of this trade-off, but remain unsure whether the demonstrated advance is substantial enough to support acceptance. The new multi-memory reward ablation is limited to one fixed response model. Whether this component provides similar benefits with other response models remains unclear.
I appreciate the stronger evidence and acknowledge that several of my concerns have been addressed. On balance, I maintain my rating of 2/4.