acceptodds
Under review as a conference paper at ICLR 2027

DG-HCA: Dual-Granularity Hierarchical Credit Assignment for Generative Recommendation

Abstract

Generative recommendation with explicit reasoning typically relies on sequence-level reinforcement learning, where a single outcome-level advantage is broadcast to all tokens in a rollout. However, for hierarchical Semantic IDs (SIDs), such coarse-grained supervision can induce conflicting credit signals: incorrect suffix decisions may be reinforced by positive advantages, while correct prefix decisions may be suppressed by negative ones. To address this issue, we propose DG-HCA, a Dual-Granularity Hierarchical Credit Assignment framework for reasoning-based generative recommendation. DG-HCA grounds policy optimization in retrieval feedback from Constrained Beam Search and decomposes credit at two complementary granularities. At the reasoning-trajectory level, Ranking-Driven Reasoning Credit evaluates trajectories according to their retrieval quality. At the SID-action level, Target-Path SID Credit performs relative credit assignment over hierarchical decisions along the ground-truth path, while Target-Path Recovery provides additional supervision when Beam Search deviates from the target SID. Extensive experiments on three public Amazon datasets demonstrate that DG-HCA achieves competitive recommendation performance and obtains the best NDCG@10 on all three datasets. Ablation studies further verify the effectiveness of SID-level credit assignment and Target-Path Recovery. These results highlight the importance of aligning reinforcement-learning credit with the structured generation process in reasoning-based generative recommendation.

Primary area:
generative models
Keywords:
Generative Recommendation, Hierarchical Credit Assignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_B5tqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DG-HCA separates reinforcement-learning supervision for recommendation reasoning from supervision for hierarchical SID decisions. Reasoning trajectories receive beam-ranking rewards; target-path SID actions receive positive credit and competing actions receive negative credit. When beam search misses the target, training adds a recovery suffix without changing the retrieved list. Experiments use Qwen3-1.7B on three Amazon categories, with component ablations and thinking/non-thinking comparisons.

Critical strengths
  1. The SID branch supplies supervision even when retrieval misses. Recovery inserts the target suffix only into the training action group, not into the beam or evaluation results (§3.4.2). This is a concrete way to train on beam failures without inflating measured retrieval.
  2. The ablations support a contribution from both branches. On Games, NDCG@10 falls from 0.0572 to 0.0506 without SID credit and to 0.0526 without recovery (Table 2). Removing the CoT loss also lowers all four Industrial metrics (Table 8), although those differences are small.
Critical weaknesses
  1. The benefit over simpler target-supervised training is unresolved. SID credit has a prescribed positive sign for the target and a negative sign for competitors. Appendix F.4 explicitly derives a local contrastive form. A1–A3 remove SID supervision or recovery, but do not compare against cross-entropy or a contrastive loss using the same beam candidates and recovered targets. These results therefore do not isolate the value of the clipped relative-credit formulation from the additional target supervision.
  2. The paper's own non-thinking results weaken the case for its chosen inference mode. Stage 3 without thinking has higher NDCG@5 and NDCG@10 on many cases (Table 3). On Games, Stage 2 without thinking already reaches NDCG@10 of 0.0641, above Stage 3 with thinking (0.0572) and without thinking (0.0600). Thinking does improve Stage 3 Recall@10, so this is a trade-off; the paper has not established why the added reasoning is the preferable operating point.
Questions for authors
  1. Attribution check: How does the SID objective compare with target cross-entropy or contrastive training when beam candidates, recovered targets, initialization, and training budget are held fixed?
  2. Clarification: Given Table 3, which accuracy/latency trade-off motivates using thinking-mode Stage 3 over the non-thinking checkpoints? Can the recommendation of this operating point be supported quantitatively?
3 comments

Dear Reviewer,

Thank you again for your time and thoughtful review. As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal. We would greatly appreciate it if you could share any further thoughts or let us know whether our responses have addressed your concerns.

We are also happy to provide any additional clarifications if needed. Thank you very much for your time and consideration!

The added objective comparisons substantially address my main concern. Holding the supervision, recovery mechanism, initialization, and training budget fixed, DG-HCA outperforms the CE and contrastive alternatives on the reported results. The Ratio-InfoNCE comparison and the removal of clipping provide useful evidence beyond the original component ablations that the optimization formulation contributes to the gains.

The latency and long-tail analyses also clarify the intended role of thinking. These responses resolve enough of my original concerns to support acceptance. I will raise my rating from 2 to 3.

We would like to express our heartfelt gratitude for your thoughtful feedback, encouraging comments, and willingness to reconsider your assessment in light of our rebuttal. We are truly grateful for the time and effort you have devoted to reviewing our work, as well as your constructive suggestions and positive recognition of our efforts. Your support and encouragement mean a great deal to us, and we sincerely appreciate your careful consideration and valuable insights.

Reviewer_F6TXOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DG-HCA (Dual-Granularity Hierarchical Credit Assignment), a framework designed to address credit assignment conflicts in Semantic ID (SID)-based generative recommendation. The authors identify a limitation of conventional Group Relative Policy Optimization (GRPO), where a single sequence-level advantage is uniformly assigned to all generated tokens, potentially reinforcing incorrect SID suffixes while suppressing correct prefixes.

To address this issue, DG-HCA introduces three complementary mechanisms: (1) Ranking-Driven Reasoning Credit, which evaluates reasoning trajectories based on retrieval quality; (2) Target-Path SID Credit, which performs fine-grained relative optimization over hierarchical SID decisions along the ground-truth path; and (3) Target-Path Recovery, which provides additional supervision when Beam Search fails to recover the complete target SID.

Experiments on three Amazon recommendation datasets demonstrate competitive performance, with DG-HCA achieving the best NDCG@10 results across all three datasets.

Critical strengths

1. Well-motivated problem formulation and fine-grained credit assignment.

The paper clearly illustrates two types of credit assignment conflicts in Figure 1: Wrong Suffix Promoted and Correct Prefix Suppressed. These issues are particularly relevant to hierarchical SID generation, where different token positions contribute differently to the correctness of the final recommendation.

Compared with standard GRPO, which assigns the same advantage to all tokens in a sequence, the proposed target-path-based credit assignment provides more localized optimization signals. This is a reasonable and well-motivated approach to addressing the mismatch between sequence-level rewards and hierarchical SID decisions.

2. Competitive experimental performance.

The experimental results demonstrate the effectiveness of DG-HCA across three datasets. As reported in Table 1, DG-HCA achieves the highest NDCG@10 on Industrial (0.1026), Games (0.0572), and Office (0.1257), compared with the strongest baseline results of 0.1016, 0.0529, and 0.1245, respectively.

Furthermore, the ablation studies in Table 2 demonstrate the benefits of fine-grained credit assignment. For example, on Games, NDCG@10 improves from 0.0453 in Stage 2 to 0.0572 in Stage 3. These results provide empirical support for the proposed framework.

Critical weaknesses

1. Limited methodological novelty relative to existing fine-grained credit assignment approaches.

The main contribution of DG-HCA lies in decomposing credit assignment into reasoning-level and SID-level optimization, together with local relative optimization along the ground-truth SID path. However, several closely related ideas have already been explored in prior work:

  • SAPO jointly optimizes reasoning blocks and their corresponding SID tokens, introducing a finer-grained optimization structure.
  • Sibling-GRPO performs relative optimization among sibling actions sharing the same parent node, which is conceptually similar to the local action groups constructed in DG-HCA.
  • HCGRec employs target-prefix prompting to alleviate the zero-advantage problem in challenging recommendation samples.
  • OneRec-Think incorporates Beam Search-based feedback to better align reinforcement learning with recommendation performance.

In particular, the central SID credit assignment mechanism in Equation (8) assigns positive credit to the ground-truth action and distributes an equal total amount of negative credit among competing actions。This formulation can be interpreted as a form of target-guided local contrastive policy optimization, rather than a fundamentally new credit assignment principle.

Although DG-HCA integrates these ideas into a coherent framework with Beam-evidence-based credit scaling and Target-Path Recovery, the individual components are closely related to existing approaches. Consequently, the methodological novelty appears somewhat incremental. The authors should more clearly articulate the conceptual distinctions between DG-HCA and these closely related methods, particularly SAPO and Sibling-GRPO, beyond their integration within a unified framework.

2. The effectiveness of Target-Path Recovery requires further clarification.

The ablation results in Table 2 reveal an interesting observation: removing Target-Path Recovery (A3) actually improves NDCG@5 on Industrial from 0.0899 to 0.0922. Meanwhile, the complete DG-HCA framework achieves better Recall@10 (0.1524 vs. 0.1458) and slightly better NDCG@10 (0.1026 vs. 0.1023).

These results suggest that Target-Path Recovery may introduce a trade-off between top-ranked recommendation accuracy and broader retrieval coverage.

It would be helpful to clarify under which conditions Target-Path Recovery is most effective. Specifically, is Recovery primarily beneficial for difficult samples where Beam Search fails to retrieve the target, or does its effectiveness depend on the depth of the longest target prefix covered by Beam Search?

A more detailed explanation of these results would help clarify the practical contribution of Target-Path Recovery and the circumstances under which it provides the greatest benefit.

Questions for authors

see weaknesses

2 comments

Dear Reviewer,

Thank you again for your time and thoughtful review. As the discussion period is approaching its end, we would like to kindly follow up on our rebuttal. We would greatly appreciate it if you could share any further thoughts or let us know whether our responses have addressed your concerns.

We are also happy to provide any additional clarifications if needed. Thank you very much for your time and consideration!

Most of my concerns are solved. Therefore, I maintain my positive score.

Reviewer_fFhCOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DG-HCA, a dual-granularity hierarchical credit assignment framework designed to improve reasoning-based generative recommendation with Semantic IDs (SIDs). The authors identify that standard sequence-level reinforcement learning methods like GRPO broadcast outcome-level advantages uniformly across all tokens, which causes conflicting credit signals when generating hierarchical SIDs. To address this, DG-HCA decomposes credit into two levels: a ranking-driven reasoning credit at the trajectory level based on constrained beam search feedback, and a target-path SID credit at the action level that performs relative optimization among sibling decisions along the ground-truth path. A target-path recovery mechanism is also introduced to provide supervision when beam search deviates from the target. Extensive experiments on three Amazon datasets show that the method achieves the best NDCG@10 scores, supported by comprehensive ablation studies and theoretical analysis.

Critical strengths

Strengths

  1. The identification of credit assignment conflicts in hierarchical SID generation is highly insightful, and the motivating example clearly illustrates why uniform advantage broadcasting is problematic for structured outputs.

  2. The proposed dual-granularity decomposition elegantly addresses both trajectory-level quality and token-level structural correctness without altering the inference-time beam search procedure.

  3. The paper provides unusually strong theoretical backing for a recommendation systems paper, including formal proofs regarding reward boundedness, hierarchical informativeness, and the stability of length-normalized policy ratios.

  4. The experimental evaluation is thorough, featuring well-designed ablations, bucket-wise analysis for different item popularity and visibility scenarios, and an interesting comparison between thinking and non-thinking modes.

Critical weaknesses

Weaknesses

  1. The absolute performance improvements over the strongest baselines are relatively modest, raising questions about whether the added complexity of the three-stage pipeline and dual-granularity optimization justifies the marginal gains.

  2. Running constrained beam search during training for every sampled trajectory introduces significant computational overhead, yet the paper lacks a detailed wall-clock time or FLOPs comparison against simpler baselines like standard GRPO.

  3. The method relies on a substantial number of hyperparameters such as the saturation rate, evidence weights, and hierarchy depth weights, but the sensitivity analysis provided in the appendix covers only a small subset of them.

  4. The evaluation is restricted to three Amazon datasets, making it unclear how well the proposed credit assignment strategy generalizes to other domains, larger item catalogs, or deeper SID hierarchies beyond D=3.

Questions for authors

1.How does the computational cost and training time of Stage 3 compare quantitatively to standard GRPO training, given the requirement to perform constrained beam search for each rollout?

  1. Could the authors clarify how sensitive the overall performance is to the choice of SID depth D, especially since the current experiments only use D=3?

3.Is there any plan to evaluate the framework on datasets outside the Amazon ecosystem to demonstrate its broader applicability to different recommendation scenarios?

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Rebuttal Updates & Reviewer Discussion for the Area Chair

Dear Area Chair,

We sincerely thank you and all reviewers for your constructive feedback and recognition of DG-HCA's motivation, methodological design, theoretical foundations, and experimental effectiveness. During the rebuttal period, we conducted substantial additional experiments and provided detailed clarifications to address the reviewers' concerns. We are particularly encouraged that all three reviewers have acknowledged our responses and maintained or strengthened their positive assessments.

To facilitate your meta-review, we summarize our key updates and reviewer feedback below:

  • Methodological Novelty & Objective Validation (B5tq, F6TX): Clarified the distinctions from SAPO, Sibling-GRPO, and related methods. Added rigorous controlled comparisons against CE, InfoNCE, Ratio-InfoNCE, and a no-clipping variant, confirming DG-HCA's superior performance under matched experimental settings. Reviewer B5tq explicitly acknowledged that these experiments substantially addressed their main concern.

  • Reasoning Effectiveness & Inference Efficiency (B5tq): Added quantitative latency comparisons, long-tail recommendation analyses, and an adaptive Thinking/Non-thinking strategy, reducing average inference latency from 2.09s to 0.87s. Reviewer B5tq confirmed that these analyses resolved their concerns sufficiently to support acceptance.

  • Computational Cost & Generalization (fFhC): Provided detailed training-cost comparisons (1.46–1.64× GRPO training time), additional hierarchy-weight ablations, and cross-domain evaluation on Steam, while retaining zero additional inference overhead. Reviewer fFhC explicitly confirmed that their concerns were resolved.

  • Target-Path Recovery & Technical Clarity (F6TX): Further explained the mechanism and effectiveness of Target-Path Recovery, including its relationship to target-prefix coverage and hierarchical credit assignment. Reviewer F6TX acknowledged that most of their concerns had been resolved.

Overview of Reviewer Consensus: We are grateful that Reviewer B5tq increased their score from 2 to 3 and explicitly expressed support for acceptance. Reviewers fFhC and F6TX both maintained their positive scores of 3, acknowledging that our rebuttal had addressed their concerns.

We respectfully hope that these additional experimental results, technical clarifications, and positive reviewer acknowledgments will help inform your final assessment.

Thank you very much for your time and consideration.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.