- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper formulates history-conditioned conversational recommendation as a two-stage process using a shared-parameter LLM: generate a request-specific natural-language user state, then rerank candidates from that state. NCCA samples several states per request and several rankings per state. Mean sibling reward estimates state utility, while within-state relative advantages train the ranking stage. Conditional Amplification multiplies positive ranking advantages by , retaining negative corrective updates.
The authors analyze credit noise caused by mixing state quality with ranking stochasticity, deriving nested-estimation reliability and a gradient decomposition for asymmetric amplification. On fixed 20-candidate reranking versions of TG-ReDial and LLM-REDIAL, NDCG@10 improves over Traj-GRPO from 0.4015 to 0.4543 and from 0.3984 to 0.4585, approximately 13.2% and 15.1%. Diagnostics include frozen-policy reward pools, history replacement, component ablations, sampling allocations, and validation curves.
Critical strengths
-
The mechanistic analysis is supported by several complementary diagnostics. Pages 7–8 use disjoint rollout subsets to form a finite-sample oracle and credit estimates, reducing direct reuse of the same rewards for validation. In Appendix C.5, Table 6, credit MSE on the frozen Traj-GRPO pool falls from 0.0134 at n=1 to 0.0040 at n=4; on the NCCA pool it falls from 0.0146 to 0.0043. History replacement demonstrates user-specific state content. Appendix C.8 also provides validation-curve evidence that NCCA exceeds Traj-GRPO's best validation checkpoint, so checkpoint selection has not simply been left unexamined.
-
The improvement over the same-backbone baseline is substantial, with matched terminal-evaluation budgets. All RL methods start from Base, use 16 ranking-reward evaluations per request, and share data, reward, and optimization steps. Table 1 reports absolute NDCG@10 gains of 0.0528 and 0.0601. The full combination has higher means than either component alone in Table 2, supporting an interaction hypothesis. Generating four states instead of sixteen offers potential sampling savings, although token counts and wall-clock measurements are needed to establish total computational savings.
Critical weaknesses
-
The evidence does not yet fully separate the effects of credit rules, stage weights, and post-training history use. Equation (19) uses at n=4, whereas the complete token/sequence/stage normalization of Traj-GRPO is less explicit. With shared parameters, reallocating the two stage losses may affect performance. Switching NC in Table 2 also changes the allocation from (16,1) to (4,4), and Appendix B already acknowledges weight changes in allocation experiments. The most informative control would fix nested rollouts and stage weights while changing only the credit rule, alongside one suitable conditional-baseline or branched-value reference. Reproducing every cited method is unnecessary.
The history interventions also have a specific limitation. Appendix C.7, Table 8 compares NCCA with Base but does not include corresponding Traj-GRPO interventions. Full−No-history is 0.0288 for NCCA and 0.0193 for Base. The total Full difference between NCCA and Base is 0.0681, of which the No-history difference is already 0.0586. This supports an increased history effect after training, but does not attribute the entire gain to better memory credit assignment, nor establish that this mechanism explains the improvement over Traj-GRPO. These are arithmetic contrasts within the same intervention table, not a causal decomposition of training gains. The inference on page 6 that similar performance with and without a state means the interface is unused is also too strong: absence of a net gain is not absence of state use.
-
Data construction and uncertainty reporting are insufficient for reproducible, general conclusions. Page 6 and Appendix C.1 specify the last eligible session, earlier history, target-title leakage filtering, and shared frozen candidate sets. These are meaningful controls. Still missing are sufficiently detailed post-filter instance/user counts, complete train/validation/test partitions, domain composition, negative-sampling proportions, and executable construction rules. History-sensitive negatives are a legitimate stress test, but define the scope of the conclusions; their use is not evidence of leakage. The authors should also explain how potentially relevant but unobserved metadata-matched candidates are treated, rather than equating non-interaction with genuine negative preference.
Each trained LLM has only two training seeds, followed by eight stochastic decodes per instance. Decodes do not replace independently trained runs. The aggregation of training-level and decoding-level variation into Table 1's standard deviations is unclear; a gap measured in pooled standard deviations describes effect size, not a significance test. Tables 2/5 lack corresponding uncertainty, particularly important on LLM-REDIAL, where NC alone changes 0.3984 to 0.3997 but the full method reaches 0.4585. Validation curves and frozen pools do not replace a training-stability check. Sequential baselines lack text input and assign unseen candidates a median score, making them contextual references; the central algorithmic comparison should rely on LLM controls with matched information. Results on 20 candidates do not directly establish full-catalog retrieval or online multiround utility.
Questions for authors
- Primary; could affect the recommendation: Please specify Traj-GRPO's complete loss normalization. With (4,4) rollouts, stage weights, and optimization budget fixed, how do shared credit and NCCA compare? Can one appropriate conditional-baseline/branched-estimation control explain the remaining gain?
- Primary; could affect the recommendation: Please provide post-filter data sizes, partitioning and user-deduplication rules, negative-construction proportions/rules, and the aggregation protocol for two training seeds and eight decodes. Prioritize per-training-seed results and paired intervals, clarify metric averaging versus ranking fusion, and report uncertainty for the principal ablations.
- Mechanistic clarification: Can the same Full/No-history/No-state interventions be reported for Traj-GRPO to test whether NCCA specifically improves history use? Random or independently retrieved negatives would further examine generalization, but a complete new online system is not requested.
- Theoretical clarification: Please distinguish the standardized update from the original unbiased estimator. The cross term underlying Eq. (10) is generally ; explaining actual gradient-noise reduction through this equation requires either its magnitude or a suitably qualified conclusion. Appendix B already identifies as arising from a particular surrogate criterion; retain that qualification rather than claiming universal optimality.
5 comments
Response to Reviewer aFAi
Q2. Questions on Data Construction and Evaluation
(1) Dataset construction pipeline and statistics. Following the sequential conversational recommendation setting of MemoCRS, we group sessions by user ID. The last eligible session is the target session; the request is the last user utterance before the target recommendation, and earlier sessions and interactions form history . Both datasets are split by user into training, validation, and test sets, with no user overlap. We then retain instances with valid history and target sessions, a target mapped to the item catalog, and explicit acceptance feedback. Instances whose target title already appears in the history, dialogue context, or request are filtered out. The resulting counts are below.
| Dataset | Split | Instances | Users |
|---|---|---|---|
| TG-ReDial | Train | 1170 | 1170 |
| TG-ReDial | Validation | 127 | 127 |
| TG-ReDial | Test | 129 | 129 |
| LLM-REDIAL | Train | 3892 | 3892 |
| LLM-REDIAL | Validation | 488 | 488 |
| LLM-REDIAL | Test | 485 | 485 |
LLM-REDIAL covers four domains, Books, Movies, Electronics, and Sports. The domain composition (number of instances) across the training, validation, and test splits is shown below.
| Split | Books | Movies | Electronics | Sports |
|---|---|---|---|---|
| Train | 2386 | 801 | 465 | 240 |
| Validation | 299 | 101 | 58 | 30 |
| Test | 274 | 114 | 66 | 31 |
(2) Negative candidate sampling. Each instance contains 1 positive target and 19 negatives drawn from three sources.
- Rejected items are items explicitly rejected in the user's history.
- Metadata-matched items have attributes matching the user's historical rejection reasons.
- History-sensitive hard negatives are metadata-matched items that also share attributes with the target. They resemble the target in some respects while carrying attributes associated with historical rejection.
Here, ''attributes'' come from the dataset's item metadata. We first take the union of rejected items and history-sensitive hard negatives. If it contains at least 19 items, we randomly sample 19; otherwise, we fill the remaining positions with other metadata-matched items. The target and negatives are shuffled to form the final candidate set. This sampling process ensures that negative candidate construction uses expressed rejection feedback and its reasons rather than equating non-interaction with genuine negative preference.
(3) Training seeds and decoding aggregation. Each trained LLM has two independently trained checkpoints, one per training seed. Each checkpoint generates eight responses per test instance, giving 16 decoding runs. For each run, we compute each metric over the test set, without ranking fusion, excluding outputs that cannot be parsed into a valid candidate permutation. Table 1 reports the mean and sample standard deviation of these 16 run-level scores.
(4) Per-training-seed results and ablation uncertainty. Tables S1–S6 in https://anonymous.4open.science/r/Supp_Materials-3006/supplementary_results.md provide per-training-seed results and mean ± std for Tables 1, 2, and 5 in the paper. From the results we notice that
- For Table 1, NCCA achieves the highest NDCG@10 among the trained LLMs reported in Table S1, in both training seeds on both datasets;
- For Table 2: NCCA consistently outperforms NC-only and CA-only in both training seeds;
- For Table 5: The default , achieves the highest NDCG@10 among the tested configurations in both seeds on both datasets.
(5) Alternative negatives and candidate sizes. To examine generalization beyond the negative construction applied in the paper, we evaluated random and BM25-retrieved negatives on both datasets with 10, 20, and 50 candidates, retaining one positive item per instance. TG-ReDial NDCG@10 is shown below.
| Method | 10 random | 10 BM25 | 20 random | 20 BM25 | 50 random | 50 BM25 |
|---|---|---|---|---|---|---|
| Base | 0.6388 | 0.5291 | 0.4865 | 0.3625 | 0.2827 | 0.2034 |
| Traj-GRPO | 0.6557 | 0.5230 | 0.4951 | 0.3569 | 0.3057 | 0.2054 |
| NCCA | 0.6616 | 0.5401 | 0.5068 | 0.3658 | 0.3288 | 0.2119 |
NCCA outperforms Base and Traj-GRPO in all six settings, showing that its advantage is not confined to history-sensitive hard negatives or 20 candidates. The NDCG@10 advantage also holds on LLM-REDIAL. Full results and other metrics are in Part I of the link: https://anonymous.4open.science/r/Supp_Materials-3006/supplementary_results.md.
Response to Reviewer aFAi
Q3. Questions on History Interventions
We added the requested Full / No-history / No-state interventions for Traj-GRPO to the Table 8 comparison. The results below are TG-ReDial NDCG@10:
| Model | Full | No-history | No-state | Full − No-history | Full − No-state |
|---|---|---|---|---|---|
| Base | 0.3818 | 0.3625 | 0.3587 | 0.0193 | 0.0231 |
| Traj-GRPO | 0.4027 | 0.3833 | 0.3673 | 0.0194 | 0.0354 |
| NCCA | 0.4499 | 0.4211 | 0.3919 | 0.0288 | 0.0580 |
NCCA's Full−No-history gain is 0.0288, compared with 0.0194 for Traj-GRPO, showing a larger recommendation benefit from user history. We agree that these intervention differences characterize the trained models' use of history and state inputs. Training attribution is supported separately by Table 2 and the matched comparisons in Q1. We will clarify this distinction in §3.3 and the Figure 2 caption.
We also agree that “absence of a net gain is not absence of state use.” We will replace “the interface goes unused” with the supported comparison that NCCA obtains a larger Full − No-state gain, 0.0580 versus 0.0354, indicating a larger recommendation benefit from the state input.
Q4. Theoretical Clarifications
(1) Unbiased estimators and standardized updates. Thank you for reminding us to make this distinction explicit. In theory, the unstandardized gradient estimator with a leave-one-out baseline is unbiased. When the group mean includes the sample itself, the estimator differs from the unbiased estimator only by a fixed scaling factor (Appendix A.3). In practice, we use GRPO, a widely used RL algorithm for LLMs, consistent with related work [1,2]. The additional standardization in GRPO means that its gradient estimator generally no longer has this unbiasedness guarantee. Nevertheless, standardization preserves cancellation of the state term in the ranking-credit numerator, as well as the signs and ordering of within-state advantages. Analyzing unstandardized gradient estimators in theory while using standardized estimators in training is also common in related work [3]. We will revise the wording in §2.3 and Appendix A.3 to clearly distinguish the properties of the unstandardized estimators from the standardized training updates.
(2) The cross term underlying Eq. (10). The full second-moment decomposition is
+2\mathbb E[(Q-b)(R-Q)\|s_R\|^2].$$ We denote the three terms on the right-hand side by $A$, $B$, and $C$, respectively. The paper assumes that $R-Q$ and $\|s_R\|^2$ are uncorrelated conditional on $(x,z)$, which gives $C=0$ because $\mathbb E[R-Q\mid x,z]=0$. We examined the first five NCCA training steps on each dataset using nested reward estimates and a last-decoder-block gradient-norm proxy that retains token normalization. We divided squared loss-gradient norms by the squared recorded advantages. For zero-advantage cases, we squared the same-step mean proxy norm from nonzero-advantage cases. The estimated ratios are below. | Step | TG-ReDial $C/B$ | TG-ReDial $C/(A+B)$ | LLM-REDIAL $C/B$ | LLM-REDIAL $C/(A+B)$ | |---|---:|---:|---:|---:| | 1 | +0.45% | +0.13% | +0.28% | +0.10% | | 2 | +2.62% | +0.64% | −0.39% | −0.14% | | 3 | +1.10% | +0.29% | +0.04% | +0.02% | | 4 | +0.86% | +0.24% | +2.26% | +0.86% | | 5 | −0.50% | −0.13% | +0.08% | +0.03% | Across the measured steps, the maximum $|C|/B$ is **2.62%**, and the maximum $|C|/(A+B)$ is **0.86%**. The cross term is therefore a small numerical correction in this last-block diagnostic, supporting the approximation while retaining the conditional assumption for the exact variance relation. **(3) Scope of the optimal sampling depth.** $n^\star=1+\rho^{-1/2}$ is the continuous optimum of the particular surrogate $U(n)$ in Appendix B (Eqs. 30–31). This criterion balances the normalized upstream and downstream information through their geometric mean. The measured $\rho$ gives $n^*\in[2.6,2.9]$, suggesting the neighboring feasible depths 2 and 4 when $n$ must divide $B=16$. Both outperform $n=8$ in Table 5, with $(4,4)$ achieving the highest NDCG@10. The surrogate guides candidate configurations, while downstream performance determines the practical choice. We will explicitly retain “optimal under this surrogate” in §3.6 and the Figure 4 caption. # References [1] Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning [2] Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning [3] Group-in-Group Policy Optimization for LLM Agent Training
Response to Reviewer aFAi
We sincerely thank the reviewer for recognizing the complementary mechanistic diagnostics and the substantial improvement over the same-backbone baseline.
Q1. Questions on Baselines
(1) Traj-GRPO loss normalization. Traj-GRPO and NCCA use the same token and sequence aggregation. We average over valid generated tokens within each sequence, then over sequences within each stage. Both methods follow the weighting rule in Eq. (19), with stage weights λP=1/(1+n) and λR=n/(1+n). Traj-GRPO samples 16 states per request and one ranking per state, so its stage weights are λP=1/2 and λR=1/2. For Traj-GRPO, both stages of trajectory s share the same advantage Astraj. Substituting this shared advantage, (m,n)=(16,1), and the stage weights into Eq. (19) gives
JTraj=21⋅161∑s=116Lclip(zs,Astraj)+21⋅161∑s=116Lclip(ys,Astraj).
(2) Comparisons with shared credit and a branched-estimation reference.
For both comparisons, we evaluate variants with and without CA. The CA variants use η=0.3 on the reranking side, matching NCCA. The results below are NDCG@10.
Both with and without CA, our two-stage advantage construction outperforms the shared-credit and TreeRL controls on both datasets, supporting the additional training value of our method.