acceptodds
Under review as a conference paper at ICLR 2027

NCCA: Nested Credit with Conditional Amplification for LLM-Based Conversational Recommendation

Abstract

Conversational recommender systems often use past dialogues to recover user preferences missing from a current request. We study conversational recommendation as a two-stage pipeline using large language models (LLMs) that first distills the dialogue history into a request-specific user state in natural language and then reranks item candidates from that state. Because downstream ranking quality can be computed automatically from observed feedback, the pipeline can be trained end-to-end with reinforcement learning with verifiable rewards (RLVR). A straightforward implementation assigns the same outcome advantage to both stages. In conversational recommendation, however, this signal mixes two sources of variation: state quality and stochastic ranking execution, with the latter especially strong when only a few candidate positions yield high reward. This coupling makes the feedback for each stage noisy: state updates inherit variation from stochastic ranking execution, while reranking updates vary with state quality. To solve this problem, we propose NCCA (Nested Credit with Conditional Amplification), a credit-assignment strategy for joint two-stage training. In NCCA, Nested Credit samples multiple states per request and multiple rankings per state, averages sibling rewards for state credit, and performs within-state relative credit for reranking. Thus, it suppresses execution noise upstream and isolates state-conditioned ranking variation downstream. Furthermore, Conditional Amplification upweights positive within-state advantages while retaining negative corrective updates. This strengthens the learning signal from the relatively few successful rankings without discarding corrections from lower-reward rankings. Our experiments show that NCCA improves NDCG@10 by 13.2% and 15.1% over the shared-advantage RLVR baseline on TG-ReDial and LLM-REDIAL, and enables a 4B model to match strong general-purpose LLMs.

Primary area:
reinforcement learning
Keywords:
conversational recommendation, large language models, reinforcement learning, credit assignment, long-term memory

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_aFAiOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper formulates history-conditioned conversational recommendation as a two-stage process using a shared-parameter LLM: generate a request-specific natural-language user state, then rerank candidates from that state. NCCA samples several states per request and several rankings per state. Mean sibling reward estimates state utility, while within-state relative advantages train the ranking stage. Conditional Amplification multiplies positive ranking advantages by 1+η1+\eta, retaining negative corrective updates.

The authors analyze credit noise caused by mixing state quality with ranking stochasticity, deriving nested-estimation reliability and a gradient decomposition for asymmetric amplification. On fixed 20-candidate reranking versions of TG-ReDial and LLM-REDIAL, NDCG@10 improves over Traj-GRPO from 0.4015 to 0.4543 and from 0.3984 to 0.4585, approximately 13.2% and 15.1%. Diagnostics include frozen-policy reward pools, history replacement, component ablations, sampling allocations, and validation curves.

Critical strengths
  1. The mechanistic analysis is supported by several complementary diagnostics. Pages 7–8 use disjoint rollout subsets to form a finite-sample oracle and credit estimates, reducing direct reuse of the same rewards for validation. In Appendix C.5, Table 6, credit MSE on the frozen Traj-GRPO pool falls from 0.0134 at n=1 to 0.0040 at n=4; on the NCCA pool it falls from 0.0146 to 0.0043. History replacement demonstrates user-specific state content. Appendix C.8 also provides validation-curve evidence that NCCA exceeds Traj-GRPO's best validation checkpoint, so checkpoint selection has not simply been left unexamined.

  2. The improvement over the same-backbone baseline is substantial, with matched terminal-evaluation budgets. All RL methods start from Base, use 16 ranking-reward evaluations per request, and share data, reward, and optimization steps. Table 1 reports absolute NDCG@10 gains of 0.0528 and 0.0601. The full combination has higher means than either component alone in Table 2, supporting an interaction hypothesis. Generating four states instead of sixteen offers potential sampling savings, although token counts and wall-clock measurements are needed to establish total computational savings.

Critical weaknesses
  1. The evidence does not yet fully separate the effects of credit rules, stage weights, and post-training history use. Equation (19) uses λP=1/5,λR=4/5\lambda_P=1/5,\lambda_R=4/5 at n=4, whereas the complete token/sequence/stage normalization of Traj-GRPO is less explicit. With shared parameters, reallocating the two stage losses may affect performance. Switching NC in Table 2 also changes the allocation from (16,1) to (4,4), and Appendix B already acknowledges weight changes in allocation experiments. The most informative control would fix nested rollouts and stage weights while changing only the credit rule, alongside one suitable conditional-baseline or branched-value reference. Reproducing every cited method is unnecessary.

    The history interventions also have a specific limitation. Appendix C.7, Table 8 compares NCCA with Base but does not include corresponding Traj-GRPO interventions. Full−No-history is 0.0288 for NCCA and 0.0193 for Base. The total Full difference between NCCA and Base is 0.0681, of which the No-history difference is already 0.0586. This supports an increased history effect after training, but does not attribute the entire gain to better memory credit assignment, nor establish that this mechanism explains the improvement over Traj-GRPO. These are arithmetic contrasts within the same intervention table, not a causal decomposition of training gains. The inference on page 6 that similar performance with and without a state means the interface is unused is also too strong: absence of a net gain is not absence of state use.

  2. Data construction and uncertainty reporting are insufficient for reproducible, general conclusions. Page 6 and Appendix C.1 specify the last eligible session, earlier history, target-title leakage filtering, and shared frozen candidate sets. These are meaningful controls. Still missing are sufficiently detailed post-filter instance/user counts, complete train/validation/test partitions, domain composition, negative-sampling proportions, and executable construction rules. History-sensitive negatives are a legitimate stress test, but define the scope of the conclusions; their use is not evidence of leakage. The authors should also explain how potentially relevant but unobserved metadata-matched candidates are treated, rather than equating non-interaction with genuine negative preference.

    Each trained LLM has only two training seeds, followed by eight stochastic decodes per instance. Decodes do not replace independently trained runs. The aggregation of training-level and decoding-level variation into Table 1's standard deviations is unclear; a gap measured in pooled standard deviations describes effect size, not a significance test. Tables 2/5 lack corresponding uncertainty, particularly important on LLM-REDIAL, where NC alone changes 0.3984 to 0.3997 but the full method reaches 0.4585. Validation curves and frozen pools do not replace a training-stability check. Sequential baselines lack text input and assign unseen candidates a median score, making them contextual references; the central algorithmic comparison should rely on LLM controls with matched information. Results on 20 candidates do not directly establish full-catalog retrieval or online multiround utility.

Questions for authors
  1. Primary; could affect the recommendation: Please specify Traj-GRPO's complete loss normalization. With (4,4) rollouts, stage weights, and optimization budget fixed, how do shared credit and NCCA compare? Can one appropriate conditional-baseline/branched-estimation control explain the remaining gain?
  2. Primary; could affect the recommendation: Please provide post-filter data sizes, partitioning and user-deduplication rules, negative-construction proportions/rules, and the aggregation protocol for two training seeds and eight decodes. Prioritize per-training-seed results and paired intervals, clarify metric averaging versus ranking fusion, and report uncertainty for the principal ablations.
  3. Mechanistic clarification: Can the same Full/No-history/No-state interventions be reported for Traj-GRPO to test whether NCCA specifically improves history use? Random or independently retrieved negatives would further examine generalization, but a complete new online system is not requested.
  4. Theoretical clarification: Please distinguish the standardized update from the original unbiased estimator. The cross term underlying Eq. (10) is generally 2E[(Q−b)(R−Q)∥sR∥2]2\mathbb E[(Q-b)(R-Q)\|s_R\|^2]; explaining actual gradient-noise reduction through this equation requires either its magnitude or a suitably qualified conclusion. Appendix B already identifies n∗=1+ρ−1/2n^*=1+\rho^{-1/2} as arising from a particular surrogate criterion; retain that qualification rather than claiming universal optimality.
5 comments
AuthorsOpenReview ↗

Response to Reviewer aFAi

We sincerely thank the reviewer for recognizing the complementary mechanistic diagnostics and the substantial improvement over the same-backbone baseline.


Q1. Questions on Baselines

(1) Traj-GRPO loss normalization. Traj-GRPO and NCCA use the same token and sequence aggregation. We average over valid generated tokens within each sequence, then over sequences within each stage. Both methods follow the weighting rule in Eq. (19), with stage weights λP=1/(1+n)\lambda_P=1/(1+n) and λR=n/(1+n)\lambda_R=n/(1+n). Traj-GRPO samples 16 states per request and one ranking per state, so its stage weights are λP=1/2\lambda_P=1/2 and λR=1/2\lambda_R=1/2. For Traj-GRPO, both stages of trajectory ss share the same advantage AstrajA_s^{\mathrm{traj}}. Substituting this shared advantage, (m,n)=(16,1)(m,n)=(16,1), and the stage weights into Eq. (19) gives

JTraj=12⋅116∑s=116Lclip(zs,Astraj)+12⋅116∑s=116Lclip(ys,Astraj).J_{\mathrm{Traj}}=\frac{1}{2}\cdot\frac{1}{16}\sum_{s=1}^{16}L_{\mathrm{clip}}(z_s,A_s^{\mathrm{traj}})+\frac{1}{2}\cdot\frac{1}{16}\sum_{s=1}^{16}L_{\mathrm{clip}}(y_s,A_s^{\mathrm{traj}}).

(2) Comparisons with shared credit and a branched-estimation reference.

  • Shared-credit control. We added Shared-(4,4) to test the contribution of two-stage advantage construction under matched training settings. It matches NC's (4,4)(4,4) sampling structure, stage weights (λP,λR)=(0.2,0.8)(\lambda_P,\lambda_R)=(0.2,0.8), and optimization budget, replacing the two-stage advantage computation with Traj-GRPO-style shared advantages.
  • Branched-estimation reference. Following Reviewer DcJg's suggestion, we adapt TreeRL, a tree-based credit assignment method combining global and local advantages, to this task under the same computational budget. This tests whether an existing branched-estimation method can explain the remaining performance gain.

For both comparisons, we evaluate variants with and without CA. The CA variants use η=0.3\eta=0.3 on the reranking side, matching NCCA. The results below are NDCG@10.

MethodTG-ReDialLLM-REDIAL
Base0.37180.3624
Traj-GRPO0.40150.3984
Shared-(4,4)0.42480.3797
TreeRL-(4,4)0.41960.3938
NC0.43020.3997
Shared-(4,4) + CA0.43320.3836
TreeRL-(4,4) + CA0.43420.4198
NCCA0.45430.4585

Both with and without CA, our two-stage advantage construction outperforms the shared-credit and TreeRL controls on both datasets, supporting the additional training value of our method.


AuthorsOpenReview ↗

Response to Reviewer aFAi

Q2. Questions on Data Construction and Evaluation

(1) Dataset construction pipeline and statistics. Following the sequential conversational recommendation setting of MemoCRS, we group sessions by user ID. The last eligible session is the target session; the request is the last user utterance before the target recommendation, and earlier sessions and interactions form history HH. Both datasets are split by user into training, validation, and test sets, with no user overlap. We then retain instances with valid history and target sessions, a target mapped to the item catalog, and explicit acceptance feedback. Instances whose target title already appears in the history, dialogue context, or request are filtered out. The resulting counts are below.

DatasetSplitInstancesUsers
TG-ReDialTrain11701170
TG-ReDialValidation127127
TG-ReDialTest129129
LLM-REDIALTrain38923892
LLM-REDIALValidation488488
LLM-REDIALTest485485

LLM-REDIAL covers four domains, Books, Movies, Electronics, and Sports. The domain composition (number of instances) across the training, validation, and test splits is shown below.

SplitBooksMoviesElectronicsSports
Train2386801465240
Validation2991015830
Test2741146631

(2) Negative candidate sampling. Each instance contains 1 positive target and 19 negatives drawn from three sources.

  • Rejected items are items explicitly rejected in the user's history.
  • Metadata-matched items have attributes matching the user's historical rejection reasons.
  • History-sensitive hard negatives are metadata-matched items that also share attributes with the target. They resemble the target in some respects while carrying attributes associated with historical rejection.

Here, ''attributes'' come from the dataset's item metadata. We first take the union of rejected items and history-sensitive hard negatives. If it contains at least 19 items, we randomly sample 19; otherwise, we fill the remaining positions with other metadata-matched items. The target and negatives are shuffled to form the final candidate set. This sampling process ensures that negative candidate construction uses expressed rejection feedback and its reasons rather than equating non-interaction with genuine negative preference.

(3) Training seeds and decoding aggregation. Each trained LLM has two independently trained checkpoints, one per training seed. Each checkpoint generates eight responses per test instance, giving 16 decoding runs. For each run, we compute each metric over the test set, without ranking fusion, excluding outputs that cannot be parsed into a valid candidate permutation. Table 1 reports the mean and sample standard deviation of these 16 run-level scores.

(4) Per-training-seed results and ablation uncertainty. Tables S1–S6 in https://anonymous.4open.science/r/Supp_Materials-3006/supplementary_results.md provide per-training-seed results and mean ± std for Tables 1, 2, and 5 in the paper. From the results we notice that

  • For Table 1, NCCA achieves the highest NDCG@10 among the trained LLMs reported in Table S1, in both training seeds on both datasets;
  • For Table 2: NCCA consistently outperforms NC-only and CA-only in both training seeds;
  • For Table 5: The default η=0.3\eta=0.3, (m,n)=(4,4)(m,n)=(4,4) achieves the highest NDCG@10 among the tested configurations in both seeds on both datasets.

(5) Alternative negatives and candidate sizes. To examine generalization beyond the negative construction applied in the paper, we evaluated random and BM25-retrieved negatives on both datasets with 10, 20, and 50 candidates, retaining one positive item per instance. TG-ReDial NDCG@10 is shown below.

Method10 random10 BM2520 random20 BM2550 random50 BM25
Base0.63880.52910.48650.36250.28270.2034
Traj-GRPO0.65570.52300.49510.35690.30570.2054
NCCA0.66160.54010.50680.36580.32880.2119

NCCA outperforms Base and Traj-GRPO in all six settings, showing that its advantage is not confined to history-sensitive hard negatives or 20 candidates. The NDCG@10 advantage also holds on LLM-REDIAL. Full results and other metrics are in Part I of the link: https://anonymous.4open.science/r/Supp_Materials-3006/supplementary_results.md.


AuthorsOpenReview ↗

Response to Reviewer aFAi

Q3. Questions on History Interventions

We added the requested Full / No-history / No-state interventions for Traj-GRPO to the Table 8 comparison. The results below are TG-ReDial NDCG@10:

ModelFullNo-historyNo-stateFull − No-historyFull − No-state
Base0.38180.36250.35870.01930.0231
Traj-GRPO0.40270.38330.36730.01940.0354
NCCA0.44990.42110.39190.02880.0580

NCCA's Full−No-history gain is 0.0288, compared with 0.0194 for Traj-GRPO, showing a larger recommendation benefit from user history. We agree that these intervention differences characterize the trained models' use of history and state inputs. Training attribution is supported separately by Table 2 and the matched comparisons in Q1. We will clarify this distinction in §3.3 and the Figure 2 caption.

We also agree that “absence of a net gain is not absence of state use.” We will replace “the interface goes unused” with the supported comparison that NCCA obtains a larger Full − No-state gain, 0.0580 versus 0.0354, indicating a larger recommendation benefit from the state input.


Q4. Theoretical Clarifications

(1) Unbiased estimators and standardized updates. Thank you for reminding us to make this distinction explicit. In theory, the unstandardized gradient estimator with a leave-one-out baseline is unbiased. When the group mean includes the sample itself, the estimator differs from the unbiased estimator only by a fixed scaling factor (Appendix A.3). In practice, we use GRPO, a widely used RL algorithm for LLMs, consistent with related work [1,2]. The additional standardization in GRPO means that its gradient estimator generally no longer has this unbiasedness guarantee. Nevertheless, standardization preserves cancellation of the state term in the ranking-credit numerator, as well as the signs and ordering of within-state advantages. Analyzing unstandardized gradient estimators in theory while using standardized estimators in training is also common in related work [3]. We will revise the wording in §2.3 and Appendix A.3 to clearly distinguish the properties of the unstandardized estimators from the standardized training updates.

(2) The cross term underlying Eq. (10). The full second-moment decomposition is

+2\mathbb E[(Q-b)(R-Q)\|s_R\|^2].$$ We denote the three terms on the right-hand side by $A$, $B$, and $C$, respectively. The paper assumes that $R-Q$ and $\|s_R\|^2$ are uncorrelated conditional on $(x,z)$, which gives $C=0$ because $\mathbb E[R-Q\mid x,z]=0$. We examined the first five NCCA training steps on each dataset using nested reward estimates and a last-decoder-block gradient-norm proxy that retains token normalization. We divided squared loss-gradient norms by the squared recorded advantages. For zero-advantage cases, we squared the same-step mean proxy norm from nonzero-advantage cases. The estimated ratios are below. | Step | TG-ReDial $C/B$ | TG-ReDial $C/(A+B)$ | LLM-REDIAL $C/B$ | LLM-REDIAL $C/(A+B)$ | |---|---:|---:|---:|---:| | 1 | +0.45% | +0.13% | +0.28% | +0.10% | | 2 | +2.62% | +0.64% | −0.39% | −0.14% | | 3 | +1.10% | +0.29% | +0.04% | +0.02% | | 4 | +0.86% | +0.24% | +2.26% | +0.86% | | 5 | −0.50% | −0.13% | +0.08% | +0.03% | Across the measured steps, the maximum $|C|/B$ is **2.62%**, and the maximum $|C|/(A+B)$ is **0.86%**. The cross term is therefore a small numerical correction in this last-block diagnostic, supporting the approximation while retaining the conditional assumption for the exact variance relation. **(3) Scope of the optimal sampling depth.** $n^\star=1+\rho^{-1/2}$ is the continuous optimum of the particular surrogate $U(n)$ in Appendix B (Eqs. 30–31). This criterion balances the normalized upstream and downstream information through their geometric mean. The measured $\rho$ gives $n^*\in[2.6,2.9]$, suggesting the neighboring feasible depths 2 and 4 when $n$ must divide $B=16$. Both outperform $n=8$ in Table 5, with $(4,4)$ achieving the highest NDCG@10. The surrogate guides candidate configurations, while downstream performance determines the practical choice. We will explicitly retain “optimal under this surrogate” in §3.6 and the Figure 4 caption. # References [1] Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning [2] Rank-GRPO: Training LLM-based Conversational Recommender Systems with Reinforcement Learning [3] Group-in-Group Policy Optimization for LLM Agent Training
Reviewer_YXYbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies credit assignment in a two-stage conversational recommender, where an LLM first summarizes user history into a request-specific state and then ranks candidate items. The proposed NCCA method samples multiple rankings per state, uses their average reward to evaluate the state, and computes ranking advantages within each state. It also upweights positive ranking advantages while retaining negative updates. Experiments on TG-ReDial and LLM-REDIAL show improvements over shared-advantage GRPO, supported by component ablations, history interventions, and analyses of credit-estimation reliability.

Critical strengths

S1. Clear motivation and technical-sound design. The paper identifies a practical problem in jointly training state generation and ranking from a shared reward. Averaging downstream outcomes and comparing rankings within each state are reasonable ways to construct stage-specific feedback.

S2. Promising empirical results. NCCA improves over shared-advantage GRPO on both datasets. The component ablations, history interventions, and frozen-policy analyses provide useful evidence beyond aggregate performance.

Critical weaknesses

W1. Limited methodological novelty. Nested Credit closely follows existing Monte Carlo and tree-based credit assignment. VinePPO [1] estimates intermediate-state values through downstream rollouts, while TreeRL [2] averages descendant rewards and computes local advantages relative to parent nodes. Although the paper discusses these methods, it does not clearly establish what its recommendation-specific adaptation adds over directly applying them.

W2. User-state quality is not directly evaluated. The state generator is trained and assessed through downstream ranking rewards, which measure usefulness to the current reranker rather than fidelity to the user’s history. The history interventions show that the reranker uses the state, but do not establish whether training produces more accurate, grounded preference summaries. Direct assessments of state quality would strengthen the interpretation of the upstream learning gains.

W3. Performance depends substantially on amplification strength. On LLM-REDIAL, Nested Credit alone barely improves over Traj-GRPO, while the full method gains substantially at (\eta=0.3). The sparse sweep, without uncertainty estimates or a clear selection protocol, leaves the robustness of this interaction uncertain.

[1] Kazemnejad, Amirhossein, et al. "Vineppo: Refining credit assignment in rl training of llms." arXiv preprint arXiv:2410.01679 (2024).

[2] Hou, Zhenyu, et al. "Treerl: Llm reinforcement learning with on-policy tree search." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025.

Questions for authors
  1. What features of this task make NCCA particularly useful compared with a direct application of existing Monte Carlo or tree-based credit assignment? A concrete comparison would help clarify the contribution beyond adapting these ideas to recommendation.
  2. How does NCCA change the content of user states? A few paired examples before and after training, including failures, would help show whether it improves preference extraction, factual grounding, or communication with the reranker.
  3. What explains the much stronger interaction between Nested Credit and amplification on LLM-REDIAL? Is this pattern consistent across training seeds, and can validation performance reliably identify a useful range of amplification strengths?
5 comments
AuthorsOpenReview ↗

Response to Reviewer YXYb

We sincerely thank the reviewer for recognizing the clear motivation, the design of stage-specific feedback, and the value of the diagnostics beyond aggregate performance.


W1. Recommendation-specific Contribution

Our key novelty is to analyze the difficulties of training user-state generation and candidate reranking from a shared terminal reward, and to design credit assignment and updates that address these difficulties together. We identify cross-stage credit mixing and the difficulty of learning from relatively few high-reward rankings in sparse-positive candidate reranking. Existing conditional value estimation provides the foundations for our design.

(1) NC and conditional value estimation. To separate state quality from ranking stochasticity, NC follows the same principle as VinePPO and TreeRL by estimating intermediate-state value through downstream sampling.

  • VinePPO draws additional continuations at reasoning-step boundaries, using these auxiliary samples for value estimation.
  • TreeRL branches at uncertain tokens and combines global and local advantages.

Both methods organize credit around intermediate points within reasoning trajectories. However, our task already provides a fixed state–ranking boundary, where the state conveys user history to the reranker. NC directly uses this boundary, separating the advantage computation of the two stages. This construction cancels the state component in the ranking-credit numerator, directly addressing the cross-stage mixing identified in our task.

(2) CA and asymmetric advantage weighting. In addition to conditional value estimation, we introduce Conditional Amplification to strengthen learning from the relatively few successful rankings in sparse-positive candidate reranking. It upweights positive within-state advantages while retaining negative corrective updates.

In our response to Reviewer DcJg, the matched-budget TreeRL comparison shows that NCCA retains NDCG@10 gains of 0.0201 and 0.0387 on TG-ReDial and LLM-REDIAL even after adding the same CA to TreeRL, further supporting the value of this task-specific design. We will make this contribution and the distinctions from the related methods more explicit in the revision.


W2. Direct Evaluation of User-State Quality

We added two assessments on TG-ReDial. Base, Traj-GRPO, and NCCA each generate one user state for every test instance. The first assessment directly examines the state content; the second tests its usefulness to a fixed reranker.

(1) Pairwise assessment of state content. We use gpt-5.6-sol to compare pairs of states generated by different models against the full user context. Each pair is evaluated in both presentation orders, and the results are averaged to reduce position bias. The five criteria cover the current request's goals and constraints, faithful representation of historical preferences, avoidance of unsupported content, usefulness to the reranker, and overall reliability. NCCA's win rates are shown below.

CriterionAssessment questionNCCA vs. BaseNCCA vs. Traj-GRPO
Current-request matchingDoes the state capture the current request's goals, constraints, and priorities?68.4%62.9%
Fidelity to user historyDoes the state accurately reflect the preferences expressed in the user's history?78.9%65.5%
Control of unsupported contentDoes the state avoid preferences or facts unsupported by the input?78.9%64.1%
Usefulness to the rerankerDoes the state give the reranker clear criteria for selecting or excluding candidates?71.1%63.2%
Overall reliabilityDoes the state reliably support recommendation for the current request?73.7%63.5%

NCCA is preferred over Traj-GRPO on all five criteria, including 65.5% for fidelity to user history and 64.1% for control of unsupported content. These assessments support more faithful representation of historical evidence and clearer organization of information relevant to the current request. They directly examine the grounding of the state.

(2) State substitution with a fixed reranker. We fix Qwen3-4B-Thinking-2507 as the reranker and provide states generated by the three methods.

State sourceFixed-reranker NDCG@10
Base0.3718
Traj-GRPO0.3991
NCCA0.4364

Using NCCA states improves the same reranker's NDCG@10 by 0.0373 over using Traj-GRPO states. This shows that the generated states help a fixed reranker make better recommendation decisions, complementing the direct content assessment. Together, the two evaluations provide evidence for both factual grounding and usefulness to the reranker.


AuthorsOpenReview ↗

Response to Reviewer YXYb

W3. Questions on Amplification Strengths

3.1 Why is the interaction between NC and CA stronger on LLM-REDIAL?

NC addresses the mixing of state quality and ranking execution, while CA strengthens learning from relatively scarce better rankings. Our input statistics show that TG-ReDial provides an average of 13.4 historical recommendation records per instance, compared with 7.7 on LLM-REDIAL. So less historical feedback may make both state inference and ranking more difficult, complicating credit assignment and making better rankings harder to obtain.

To examine whether the two problems separately addressed by NC and CA are more pronounced on LLM-REDIAL, we use Qwen3-4B-Thinking-2507 to perform (4,16) nested sampling on both test sets. The results are shown below.

DiagnosticTG-ReDialLLM-REDIAL
Sign mismatch between shared and within-state credit13.4%25.3%
Within-state reward skewness0.2120.355
Proportion of rankings with positive within-state advantage45.8%28.2%

(1) Credit mixing is more pronounced on LLM-REDIAL. The higher sign-mismatch rate means that shared trajectory credit and within-state credit more often assign opposite update directions to the same ranking. Amplifying shared credit can therefore further reinforce a relatively poor ranking because it benefited from a good state. NC evaluates the state through its average downstream reward and the ranking through its relative performance within that state, giving CA the appropriate comparison reference.

(2) Better within-state rankings are also scarcer on LLM-REDIAL. The more right-skewed reward distribution and lower positive-advantage proportion indicate that fewer sampled rankings exceed their state's mean reward. NC alone still weights positive and negative advantages symmetrically. CA increases the positive-side weight while retaining correction of poorer rankings, strengthening learning from these better outcomes.

These observations agree with Table 2. On LLM-REDIAL, NC alone and CA alone improve NDCG@10 by 0.3% and 0.9%, respectively, whereas their combination improves it by 15.1%. On TG-ReDial, where both diagnostic problems are less pronounced, NC alone and CA alone already yield 7.1% and 4.1% gains. NC identifies which rankings are relatively better, and CA strengthens learning from them. The greater presence of both problems on LLM-REDIAL provides a mechanism-based explanation for the stronger interaction.

3.2 Is the pattern consistent across training seeds?

Yes, the advantage of the full combination appears in both independent training runs. The LLM-REDIAL ablations are shown below in NDCG@10.

NCAmp.Seed 1Seed 2Mean
NoNo0.39600.40070.3984
NoYes0.40630.39760.4019
YesNo0.40330.39620.3997
YesYes0.45950.45760.4585

NCCA achieves the highest score among all four configurations in each run. Its gains over NC alone are 0.0562 and 0.0614, and it also outperforms CA alone in both runs. Thus, the large benefit from combining NC and amplification is reproduced across the two training seeds.

3.3 Can validation performance identify useful amplification strengths?

We select η\eta from {0,0.1,0.3,0.5}\{0,0.1,0.3,0.5\}. During training, we save a checkpoint every five steps and record validation NDCG@10. The amplification strength and its corresponding checkpoint are selected according to validation performance.

(1) Consistency between validation and test performance. To examine this selection procedure, we additionally evaluated all saved checkpoints from the Table 5 training runs, including the different training seeds, on the test set. Across these runs and checkpoints, validation and test NDCG@10 have a Pearson correlation of r=0.93r=0.93, supporting the validation-based procedure. Here we further provide the following validation-based selection example on TG-ReDial with (m,n)=(4,4)(m,n)=(4,4). Validation selects η=0.3\eta=0.3, which also achieves the highest test NDCG@10 in Table 5.

η\etaValidation NDCG@10Test NDCG@10
00.47910.4302
0.10.49260.4469
0.30.50760.4543
0.50.46080.4278

(2) Uncertainty estimates. We also report per-training-seed NDCG@10 and mean ± standard deviation results of amplification sweep experiment (Table 5) in https://anonymous.4open.science/r/Supp_Materials-3006/supplementary_results.md. These supplementary results further address the uncertainty of the amplification sweep.

I thank the authors for their thoughtful response and for running the additional experiments. The matched Shared-(4,4) controls show that the gains are not explained by sampling or stage weights alone, and the adapted TreeRL comparison supports the practical value of the design. The new user-state evaluations also address my concern about whether training improves the states themselves.

My main remaining concern is the methodological contribution. The core estimator is a straightforward adaptation of existing tree-based credit assignment. Centering rewards within each state removes the state mean, but does not by itself establish lower gradient variance. Amplifying positive advantages works well in the reported experiments, but the analysis offers limited guidance on when the extra weighting should help.

The rebuttal strengthens the empirical case for NCCA, but my concern about the methodological contribution remains. On balance, I will keep my original rating.

Reviewer_DcJgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for improving credit assignment in two-stage LLM-based conversational recommendation. The main issue is that trajectory-level GRPO assigns the same reward to user state generation and candidate reranking, even though the reward is affected by state quality and randomness in ranking. NCCA addresses this by sampling multiple rankings for each user state and computing separate advantages for the two stages.

Critical strengths
  • The experiments provide good support for the paper's central hypothesis about the limitations of trajectory-level GRPO. The analyses of reward variance and credit reliability help explain the motivation behind the proposed method.

  • The paper provides clear experimental evidence for the effectiveness of the proposed approach. The ablation studies show how the framework contribute to the overall performance and highlight the benefit of combining the two components.

Critical weaknesses
  • While the frozen-policy diagnostics support improved credit estimation, the training experiments do not fully isolate its contribution to downstream performance from changes in advantage normalization, sampling structure, and stage-loss weights. On LLM-REDIAL, Nested Credit alone improves NDCG@10 by only 0.3% compared to 15.1% for the full method(table 2). Controlled comparisons with matched sampling structures and loss weights would better establish the source of the gains.

  • Nested Credit closely relates to conditional-rollout value estimation in methods such as VinePPO and TreeRL, while Conditional Amplification builds on asymmetric advantage weighting. The novelty therefore appears primarily in their integration for two-stage conversational recommendation. Clearer methodological distinctions and comparisons with adapted baselines under matched computational budgets would strengthen the contribution.

Questions for authors

The evaluation uses fixed candidate sets with one positive item and 19 negatives (in section3.1), closely matching the sparse-positive setting motivating Conditional Amplification. Could you evaluate NCCA under different candidate sizes, positive densities, and reward functions, and report the corresponding intra-state correlation and reward skewness?

4 comments
AuthorsOpenReview ↗

Response to Reviewer DcJg

We sincerely thank the reviewer for recognizing the experimental support for our analysis of trajectory-level GRPO and the benefit of combining the two components.


W1. Isolating the Contribution of Credit Assignment

1.1 Controlled comparisons with matched sampling structures and loss weights

Following your suggestion, we added Shared-(4,4), matching NC's (4,4)(4,4) sampling structure, stage weights (λP,λR)=(0.2,0.8)(\lambda_P,\lambda_R)=(0.2,0.8), and optimization budget. We replace the two-stage advantage computation with Traj-GRPO-style shared advantages. We also train Shared-(4,4) + CA, applying the same amplification strength η=0.3\eta=0.3 as NCCA on the reranking side. These controls compare the advantage constructions with other factors matched. The results below are NDCG@10.

MethodTG-ReDialLLM-REDIAL
Base0.37180.3624
Traj-GRPO0.40150.3984
Shared-(4,4)0.42480.3797
NC0.43020.3997
Shared-(4,4) + CA0.43320.3836
NCCA0.45430.4585

Without CA, NC improves over Shared-(4,4) by 0.0054 and 0.0200 on TG-ReDial and LLM-REDIAL. With the same CA setting, NCCA improves over Shared-(4,4) + CA by 0.0211 and 0.0749. Both paired comparisons show additional recommendation gains from the two-stage advantage construction, with larger gains when combined with CA. Thus, changes in sampling structure, stage-loss weights, or the addition of CA alone do not explain the full improvement.

1.2 Why does NC alone yield a small gain on LLM-REDIAL?

NC addresses the mixing of state quality and ranking execution, while CA strengthens learning from relatively scarce better rankings. Our input statistics show that TG-ReDial provides an average of 13.4 historical recommendation records per instance, compared with 7.7 on LLM-REDIAL. So less historical feedback may make both state inference and ranking more difficult, complicating credit assignment and making better rankings harder to obtain.

To examine whether the two problems separately addressed by NC and CA are more pronounced on LLM-REDIAL, we use Qwen3-4B-Thinking-2507 to perform (4,16) nested sampling on both test sets. The results are shown below.

DiagnosticTG-ReDialLLM-REDIAL
Sign mismatch between shared and within-state credit13.4%25.3%
Within-state reward skewness0.2120.355
Proportion of rankings with positive within-state advantage45.8%28.2%

(1) Credit mixing is more pronounced on LLM-REDIAL. The higher sign-mismatch rate means that shared trajectory credit and within-state credit more often assign opposite update directions to the same ranking. Amplifying shared credit can therefore further reinforce a relatively poor ranking because it benefited from a good state. NC evaluates the state through its average downstream reward and the ranking through its relative performance within that state, giving CA the appropriate comparison reference.

(2) Better within-state rankings are also scarcer on LLM-REDIAL. The more right-skewed reward distribution and lower positive-advantage proportion indicate that fewer sampled rankings exceed their state's mean reward. NC alone still weights positive and negative advantages symmetrically. CA increases the positive-side weight while retaining correction of poorer rankings, strengthening learning from these better outcomes.

These observations agree with Table 2. On LLM-REDIAL, NC alone and CA alone improve NDCG@10 by 0.3% and 0.9%, respectively, whereas their combination improves it by 15.1%. On TG-ReDial, where both diagnostic problems are less pronounced, NC alone and CA alone already yield 7.1% and 4.1% gains. NC identifies which rankings are relatively better, and CA strengthens learning from them. The greater presence of both problems on LLM-REDIAL provides a mechanism-based explanation for the stronger interaction.


AuthorsOpenReview ↗

Response to Reviewer DcJg

W2. Comparisons with Baselines

Our key novelty is to analyze the difficulties of training user-state generation and candidate reranking from a shared terminal reward, and to design credit assignment and updates that address these difficulties together. We identify cross-stage credit mixing and the difficulty of learning from relatively few high-reward rankings in sparse-positive candidate reranking. Existing conditional value estimation and asymmetric weighting provide the foundations for our design.

(1) NC and conditional value estimation. To separate state quality from ranking stochasticity, NC follows the same principle as VinePPO and TreeRL by estimating intermediate-state value through downstream sampling.

  • VinePPO draws additional continuations at reasoning-step boundaries, using these auxiliary samples for value estimation.
  • TreeRL branches at uncertain tokens and combines global and local advantages.

Both methods organize credit around intermediate points within reasoning trajectories. However, our task already provides a fixed state–ranking boundary, where the state conveys user history to the reranker. NC directly uses this boundary, separating the advantage computation of the two stages. This construction cancels the state component in the ranking-credit numerator, directly addressing the cross-stage mixing identified in our task.

(2) CA and asymmetric advantage weighting. In sparse-positive candidate reranking, our analysis shows that positive within-state advantages are concentrated in relatively few successful rankings. We therefore introduce CA to strengthen learning from these rankings after NC has separated the credit, while retaining negative corrective updates. Applying asymmetric weighting directly to the shared trajectory advantage would instead amplify a signal that still mixes state quality with ranking execution, potentially reinforcing a ranking that performs poorly relative to others from the same state.

(3) Comparison with adapted baselines under matched computational budgets. We adapt TreeRL to this task and compare it with NCCA under the same computational budget. We also train TreeRL-(4,4) + CA, applying the same amplification strength η=0.3\eta=0.3 on the reranking side. The results below are NDCG@10.

MethodTG-ReDialLLM-REDIAL
Base0.37180.3624
Traj-GRPO0.40150.3984
TreeRL-(4,4)0.41960.3938
NC0.43020.3997
TreeRL-(4,4) + CA0.43420.4198
NCCA0.45430.4585

Without CA, NC improves over TreeRL-(4,4) by 0.0106 and 0.0059 on the two datasets. With the same CA setting, NCCA still improves over TreeRL-(4,4) + CA by 0.0201 and 0.0387. These results support the training value of the advantage construction designed for the two-stage recommendation task and its combination with conditional amplification. The adapted branch-based credit method, even with the same amplification, does not fully account for NCCA's performance gains. We will clarify the methodological distinctions and include the adaptation details and comparisons in the revision.


Q1. Candidate Sets and Evaluation Metrics

We added evaluations on both datasets with 10, 20, and 50 candidates. Each instance retains one positive item, giving positive densities of 10%, 5%, and 2%, respectively. Following Reviewer aFAi's suggestion, we use random negatives and BM25-retrieved negatives in this experiment to examine candidate construction settings different from those in the main paper. We compare the methods using multiple recommendation metrics. The table below shows the TG-ReDial results with random negatives.

CandidatesPositive densityMethodNDCG@1NDCG@5NDCG@10Recall@5Recall@10
1010%Base0.33190.56980.63880.78441.0000
1010%Traj-GRPO0.36330.58840.65570.78891.0000
1010%NCCA0.36560.60110.66160.81131.0000
205%Base0.23520.42220.48650.60210.8007
205%Traj-GRPO0.23960.43660.49510.62940.8109
205%NCCA0.24790.44780.50680.63310.8150
502%Base0.11340.23570.28270.35100.4979
502%Traj-GRPO0.11640.25130.30570.38290.5510
502%NCCA0.12570.27670.32880.41870.5803

Across all three candidate sizes, NCCA achieves higher NDCG than Base and Traj-GRPO, with Recall gains where the metric is not saturated. Its recommendation gains therefore persist as the candidate size and corresponding positive density change. We also observe consistent performance advantages on LLM-REDIAL and with BM25 negatives. Complete results are available in Part I of https://anonymous.4open.science/r/Supp_Materials-3006/supplementary_results.md.

AuthorsOpenReview ↗

Sincere Thanks and Looking Forward to Your Comments

Dear Reviewer DcJg,

Thank you very much for your generous recognition of the experimental support for our analysis of trajectory-level GRPO and the benefit of combining the two components. Your questions on isolating the contribution of credit assignment and on the comparison with existing baselines pushed us to strengthen the paper considerably, and we have responded to each of them carefully in the rebuttal.

With the discussion period ending soon, we would deeply appreciate it if you could take a look at our responses, and we are more than happy to discuss any remaining point at your convenience.

Thank you once again for your careful review and your patience.

Thank to the authors for the answers to my questions. However, I still believe my initial overall rating reflects my assessment, and I am inclined to keep it.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.