acceptodds
Desk rejected at ICLR 2027

SAPO: Step-Aligned Policy Optimization for Reasoning-Based Generative Recommendation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Generative recommendation treats next-item prediction as autoregressive item-identifier generation. Specifically, items are encoded as semantic identifiers (SIDs), which are short coarse-to-fine token sequences whose early tokens capture broad semantics and later tokens refine them. Recent work augments this paradigm with reasoning traces and optimizes them via reinforcement learning with verifiable rewards, typically an outcome-reward algorithm scored on the generated SID. However, in realistic recommendation, such reward design becomes a single scalar per rollout; when a generated SID mismatches, it cannot identify which SID-token prediction caused the mismatch and may penalize matched SID-token positions together with the mismatched position. We identify that the natural unit of credit assignment in this setting is a single reasoning step (one thinking block paired with one SID token). We instantiate this idea in SAPO (Step-Aligned Policy Optimization): rather than broadcasting one advantage to the whole response, SAPO computes a separate group-relative advantage for each reasoning step and applies it only to the corresponding thinking block and SID token. Across three real-world recommendation datasets, SAPO stabilizes RL training and achieves competitive or improved performance, with the largest gains where fine-grained credit assignment matters most. Our results suggest that reinforcement-learning objectives for structured generation should mirror the decoder's own decomposition of the output. Our code is available at https://anonymous.4open.science/r/SAPO.

Primary area:
reinforcement learning
Keywords:
reinforcement learning with verifiable rewards, credit assignment, group relative policy optimization, LLM post-training, reasoning, generative recommendation, semantic identifiers, sequential recommendation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kB31OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SAPO pairs each thinking block with one SID-token position and assigns that pair a level-specific match reward and group-relative advantage. It averages token contributions within each pair while retaining a blocked generation layout: all reasoning precedes all SID tokens. Experiments on three Amazon categories compare ranking quality, training dynamics, reward/advantage ablations, block-credit permutations, and larger backbones.

Critical strengths
  1. The evidence goes beyond a single outcome-GRPO comparison. SAPO leads 11 of 12 entries in Table 1. Table 2 separates dense match rewards from localized advantages, and Appendix N reports three Stage-3 seeds on Industrial from a common Stage-2 checkpoint.
  2. The block-to-level pairing is tested directly. Appendix P keeps SID-token credit fixed and changes only the advantages assigned to thinking blocks. Both reversed and random assignments reduce all four Office metrics. This supports the usefulness of the ordered pairing under the actual blocked decoding layout.
Critical weaknesses
  1. The stability comparison also changes the policy-loss scale. Equation (9) divides each step's advantage by its length while retaining the total-response-token denominator used in Equation (2). This changes both credit placement and the weighting/scale of the policy update. Appendix K uses the same nominal learning rate and KL coefficient for SAPO and outcome baselines. Table 2 does not separately isolate step normalization, so reduced gradient norms and KL drift cannot by themselves be attributed to better credit assignment.
  2. Table 4 does not reconcile with the stated sample construction. Appendix H describes leave-one-out splitting with sliding-window next-item pairs. For Office, 4,866 users with an average full sequence length of 5.97 imply roughly 29,050 interactions, yet the table lists 38,924 training pairs before validation and test are counted. The same issue appears in the other categories. This needs clarification about augmentation, repeated windows, or what the reported average measures; it prevents reconstructing the stated dataset from the paper alone.
Questions for authors
  1. Implementation clarification: Is Equation (9)'s double reduction exactly what is implemented, and how is the KL term reduced relative to it? Can a normalization-only or scale-matched outcome baseline separate the stability effect from the credit-placement effect?
  2. Data clarification: What counting or augmentation rule reconciles Table 4's training-pair counts with its user counts and average sequence lengths? Please specify which samples enter Stage 3.
5 comments

We thank the reviewer for the detailed feedback and constructive reviews. We have uploaded a revised manuscript (all changes are highlighted in blue).

W1 & Q1

Weakness 1: The stability comparison also changes the policy-loss scale. Equation (9) divides each step's advantage by its length while retaining the total-response-token denominator used in Equation (2). This changes both credit placement and the weighting/scale of the policy update. Appendix K uses the same nominal learning rate and KL coefficient for SAPO and outcome baselines. Table 2 does not separately isolate step normalization, so reduced gradient norms and KL drift cannot by themselves be attributed to better credit assignment.

Question 1: Implementation clarification: Is Equation (9)'s double reduction exactly what is implemented, and how is the KL term reduced relative to it? Can a normalization-only or scale-matched outcome baseline separate the stability effect from the credit-placement effect?

Response:

We thank the reviewer for this careful reading. Eq. (9) changes both credit placement and update scale. Prior work has shown that how GRPO normalizes by response length and aggregates the token-level loss changes how each response is weighted in the policy update [1], [2]. We therefore separate the two effects with a 2×2 design on Office:

OfficeCreditAggregationR@5N@5R@10N@10
Stage 2 (reference)––0.11430.09330.14100.1020
Pure GRPO (Table 2 "w/o both")outcomeGRPO token-mean0.12350.10010.14900.1083
Normalization-only (Table 2, formerly "w/o per-step match reward")outcome (shared by all steps)Eq. (9)0.12680.10290.15350.1115
Per-step credit, no step normalization (new)per-stepGRPO token-mean0.11460.09250.14000.1013
Full SAPOper-stepEq. (9)0.13710.11450.16490.1241

In Table 2's "w/o per-step match reward", every step receives the same final exact-match reward, so all steps carry the same exact-match signal and the change from GRPO is the Eq. (9) normalization. This row is the scale-matched outcome baseline, and we have renamed it "normalization-only" in the revised manuscript to make this explicit.

Stability. With per-step credit but without step normalization, the model stays at the Stage-2 level (N@10 0.1013 vs. 0.1020), and individual reasoning-step lengths change sharply during training. Figure 4 (Industrial) shows the same split: the single-component ablations, including normalization-only, avoid pure GRPO's gradient-norm explosion, and SAPO reaches the highest reward curve (Section 5.4).

Ranking gains at matched scale. SAPO outperforms normalization-only on all 12 dataset × metric entries in Table 2: +0.0103 / +0.0116 / +0.0114 / +0.0126 on Office, and +0.0022 to +0.0055 on Video Games and Industrial. The dense-scalar baseline ("w/o per-step advantage") also uses Eq. (9), and SAPO outperforms it on 11 of 12 entries.

Implementation. For the policy-gradient term, each step's advantage is divided by its token count ∣τ(k)∣+1|\tau^{(k)}|+1 as in Eq. (9), and the result is aggregated with a token-mean over all response tokens (the ∑ℓ∣yℓ∣\sum_\ell|y_\ell| denominator of Eq. (2)). The KL term uses the low-variance (k3) estimator to the Stage-2 reference with coefficient 10−310^{-3}; it is added to the loss with the same token-mean over all response tokens and no step-length division.

SAPO and normalization-only share the same Eq. (9) scaling of the PG term and the same KL term, so their PG-to-KL weight is identical and the comparison isolates credit placement. Relative to pure GRPO, dividing the PG term by ∣y(k)∣|y^{(k)}| while leaving KL undivided raises the effective KL strength relative to PG and lowers the effective step size. We have stated these details in Section 4.3 and Appendices J–K, added the 2×2 comparison as Appendix X, and revised Sections 4.3, 5.3 and 5.4 to attribute stability to step normalization and the matched-scale ranking gain to per-step credit placement.

W2 & Q2

Weakness 2: Table 4 does not reconcile with the stated sample construction. Appendix H describes leave-one-out splitting with sliding-window next-item pairs. For Office, 4,866 users with an average full sequence length of 5.97 imply roughly 29,050 interactions, yet the table lists 38,924 training pairs before validation and test are counted. The same issue appears in the other categories. This needs clarification about augmentation, repeated windows, or what the reported average measures; it prevents reconstructing the stated dataset from the paper alone.

Question 2: Data clarification: What counting or augmentation rule reconciles Table 4's training-pair counts with its user counts and average sequence lengths? Please specify which samples enter Stage 3.

Response:

We thank the reviewer for checking this. The description in the original manuscript was incorrect, and the "#Users" column listed the number of test samples rather than distinct users. Each user's chronologically sorted sequence yields sliding-window samples (history of at most 10 items, next-item target). We sort the samples from all users by time and split them 8:1:1 into training, validation and test sets. There is therefore no information leakage between the three sets. The corrected statistics are:

DatasetTrain / Valid / Test samplesDistinct usersDistinct users in testAvg. seq. len.
Office38,924 / 4,866 / 4,8668,3282,0245.97
Video Games49,133 / 6,142 / 6,1429,0662,1226.45
Industrial36,259 / 4,532 / 4,5337,6941,9875.96

"Avg. seq. len." is the mean of (history length + 1) over test samples. With this correction the counts are consistent: the training, validation and test samples form an 8:1:1 split (38,924 / 4,866 / 4,866 for Office), and 5.97 is averaged over the 4,866 test samples rather than over users.

Stage 3 uses all training samples (38,924 / 49,133 / 36,259), which are also the instances used to build the Stage-2 traces. The Stage-3 checkpoint is selected on the validation split. All methods are evaluated on the same test samples, so every comparison in the paper is on identical data.

We have corrected Section 5.1, Appendix H and Table 4 (columns and caption) in the revised manuscript, and added the distinct-user counts.

References

[1] Liu, Zichen, et al. "Understanding R1-Zero-Like Training: A Critical Perspective." COLM 2025.

[2] Yu, Qiying, et al. "DAPO: An Open-Source LLM Reinforcement Learning System at Scale." NeurIPS 2025.

Thank you again for your careful review. If anything in our response would benefit from further clarification, we would be happy to elaborate and discuss any remaining concerns.

Reviewer_x8ueOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAPO, a step-aligned policy optimization method for reasoning-based generative recommendation. It pairs each reasoning block with a corresponding semantic-ID (SID) token and introduces step-level rewards and advantages to improve credit assignment during reinforcement learning. Experiments on three Amazon Reviews categories show improvements over several recommendation baselines.

Critical strengths
  1. The paper identifies a reasonable limitation of rollout-level rewards in hierarchical SID generation. The motivation is clear and the proposed solution is intuitive.
  2. The method is relatively simple to implement and introduces little additional training overhead.
  3. The experiments include comparisons with existing methods, component ablations, and training-dynamics analysis. The additional results in the appendix provide useful evidence for the effectiveness of SAPO.
Critical weaknesses
  1. Limited methodological novelty. The main contribution is to apply per-step rewards and advantages to the existing hierarchical SID structure. These components are closely related to established reward shaping and advantage estimation techniques. While the combination is reasonable, the paper does not clearly demonstrate a substantial methodological advance beyond adapting existing ideas to this particular setting.

  2. Insufficient comparison with simpler alternatives. Although the paper includes component ablations, it remains unclear how much of the improvement comes specifically from assigning advantages to reasoning blocks. A stronger comparison with SID-token-level reward assignment, without explicit reasoning-step alignment, would help establish the necessity of the proposed design.

  3. Limited theoretical justification. The objective-consistency result relies on the existence of an exact-match policy. This assumption may not hold in practical recommendation settings. Moreover, rewarding individual SID-token matches may not always reflect meaningful progress when earlier tokens in the hierarchy are incorrect. The implications of these issues deserve further discussion.

  4. Limited experimental generalization. The evaluation focuses on three relatively small Amazon categories with a fixed three-level SID hierarchy. While the appendix includes additional scaling and stability experiments, the effectiveness of SAPO under deeper hierarchies, different data distributions, and realistic implicit feedback remains unclear. Some improvements over the strongest baseline are also relatively small.

Questions for authors
  1. How much improvement does SAPO provide over a simpler approach that assigns rewards and advantages directly to SID-token positions without aligning them with reasoning blocks?

  2. Since all reasoning blocks are generated before the SID tokens, what evidence supports the claimed correspondence between an individual reasoning block and its assigned SID level?

  3. Would SAPO remain effective with deeper or unbalanced SID hierarchies, or when rewards are derived from noisy user interactions rather than ground-truth SID matches?

6 comments

We thank the reviewer for the detailed feedback and constructive reviews. We have uploaded a revised manuscript (all changes are highlighted in blue).

Weakness 1

Weakness 1: Limited methodological novelty. The main contribution is to apply per-step rewards and advantages to the existing hierarchical SID structure. These components are closely related to established reward shaping and advantage estimation techniques. While the combination is reasonable, the paper does not clearly demonstrate a substantial methodological advance beyond adapting existing ideas to this particular setting.

Response:

We thank the reviewer for this comment. Process reward models [2] provide step-level supervision, which requires human step labels or a learned verifier. Reward shaping [1] can make the reward denser. Under GRPO [3], however, the reward of a rollout is still turned into a single advantage shared by all of its tokens (Section 3), so the model cannot tell which SID level was wrong. Semantic IDs from a hierarchical codebook [4] can be checked level by level (Appendix B.2), so SAPO places the advantage on each level and aligns the reward, the advantage, and the token aggregation with SID levels.

Table 2 tests whether a denser reward contributes to the gain. The dense-scalar baseline ("w/o per-step advantage") uses the summed per-level match reward and the same step-normalized aggregation as SAPO, and it uses one advantage per rollout instead of one per level. SAPO outperforms it on 11 of 12 entries. The contribution of aligning thinking-block credit with SID levels is measured by the shared think-credit variant in our response to Weakness 2 and Question 1. We have rewritten the contribution statement in Section 1 accordingly.


W2 & Q1

Weakness 2: Insufficient comparison with simpler alternatives. Although the paper includes component ablations, it remains unclear how much of the improvement comes specifically from assigning advantages to reasoning blocks. A stronger comparison with SID-token-level reward assignment, without explicit reasoning-step alignment, would help establish the necessity of the proposed design.

Question 1: How much improvement does SAPO provide over a simpler approach that assigns rewards and advantages directly to SID-token positions without aligning them with reasoning blocks?

Response:

We thank the reviewer for suggesting this comparison. SAPO gives each SID level credit in two places: the SID token itself and the thinking block paired with it. To measure the contribution of block credit, we vary block credit while keeping SID-token credit fixed.

In Appendix P (Office), the advantages applied to thinking blocks are reversed or randomly permuted per example. Both variants reduce all four metrics: NDCG by 11.0–12.6% and Recall by 11.7–15.4%. Because the reversed mapping is deterministic, the drop is due to the placement of block credit. A new Industrial run with a random block-to-level mapping gives the same result (table in our response to Question 2).

We also ran a "shared think-credit" variant on Office and Industrial, which is short for the "SID-token-level assignment without reasoning-step alignment" approach the reviewer describes. SID token kk keeps its own per-level group-relative advantage, and all thinking-block tokens share one rollout-level advantage (the group z-score of the summed per-level match reward).

DatasetMethodR@5N@5R@10N@10
OfficePure GRPO0.12350.10010.14900.1083
Dense scalar: summed match reward, rollout adv. (Table 2 "w/o per-step adv.")0.12290.10080.14940.1093
Shared think-credit: per-level adv. on SID tokens, rollout adv. on thinking0.12470.09950.15190.1083
Full SAPO0.13710.11450.16490.1241
IndustrialPure GRPO0.10320.08460.12070.0903
Dense scalar (Table 2 "w/o per-step adv.")0.11710.09160.14010.0990
Shared think-credit0.09930.08120.12550.0898
Full SAPO0.11810.09350.14440.1018

The gap between shared think-credit and SAPO reflects the additional benefit of aligning thinking-block credit with SID levels. Across the four metrics, SAPO exceeds shared think-credit by 0.0124–0.0158 on Office and 0.0120–0.0189 on Industrial, while shared think-credit stays within 0.0025 of the dense-scalar baseline on Office and falls below it on Industrial, so per-level advantages on the SID tokens alone do not explain SAPO's gain. We have added the shared think-credit results as Appendix S and the Industrial random-mapping run to Appendix P of the revised manuscript.

W3

Weakness 3: Limited theoretical justification. The objective-consistency result relies on the existence of an exact-match policy. This assumption may not hold in practical recommendation settings. Moreover, rewarding individual SID-token matches may not always reflect meaningful progress when earlier tokens in the hierarchy are incorrect. The implications of these issues deserve further discussion.

Response:

We thank the reviewer for raising these two issues.

Non-realizable case. In the original manuscript, Proposition 1 stated the realizable case, and Proposition 2 (Appendix E.1) also treated the non-realizable case. Part 2 (optimum preservation) assumes realizability. Part 3 holds in general: Jmatch(π)=α∑kRk(π)J_{\mathrm{match}}(\pi)=\alpha\sum_k R_k(\pi), where RkR_k is the level-kk marginal match probability, while Jout(π)J_{\mathrm{out}}(\pi) is the joint probability that all levels match, so Jout≤min⁡kRkJ_{\mathrm{out}}\le\min_k R_k. JmatchJ_{\mathrm{match}} therefore rewards level-wise marginals and separates policies that have equal exact-match rates but different marginals. We have added Part 3 to the main text as part (ii) of Proposition 1 in Section 4.2.

Later levels under a wrong prefix. SAPO rewards each level independently, so later levels still receive a signal when an earlier level is wrong. We measured how often this happens and tested a prefix-gated reward.

On Industrial, we drew 16 trie-constrained SID samples at temperature 1 per test prompt, after each model's own thinking blocks. An earlier SID level is wrong but a later level matches in 1.3% of SAPO's samples, and these matches make up 3.7% of all its per-level matches. Under a wrong level-1 code, level 2 matches 0.8% of the time, against 39.7% under a correct one. The Stage-2 initialization gives nearly the same numbers (1.3%; 3.8%). Such matches are rare, and SAPO does not make them more frequent than the Stage-2 model does.

SIDReasoner already uses a prefix-gated RL reward (0.25 if level 1 matches, 0.5 if levels 1–2 match, 1.0 if all three match) with a rollout-level advantage. SAPO outperforms it in Table 1 and in the multi-seed results in our response to Weakness 4. Section 5.2 and Appendices B and G of the original manuscript loosely called this an outcome reward; we have corrected the description in the revised manuscript. To isolate the reward form, we trained a prefix-gated variant of SAPO on Industrial, which keeps SAPO's per-step advantages and rewards step kk only if levels 1..k1..k all match:

IndustrialReward formAdvantageR@5N@5R@10N@10
SIDReasonerprefix-gated (0.25/0.5/1.0)rollout-level0.11090.09050.14380.1010
SAPOindependent per-levelper-step0.11810.09350.14440.1018
SAPO + prefix-gated per-step rewardprefix-gated per-stepper-step0.09930.08080.12570.0893

Prefix gating lowers accuracy on all four metrics, while the case it targets covers ~1.3% of sampled SIDs. We therefore keep independent per-level rewards.

We have added the prefix-gated variant and the diagnostic to Appendix P of the revised manuscript.


W4 & Q3

Weakness 4: Limited experimental generalization. The evaluation focuses on three relatively small Amazon categories with a fixed three-level SID hierarchy. While the appendix includes additional scaling and stability experiments, the effectiveness of SAPO under deeper hierarchies, different data distributions, and realistic implicit feedback remains unclear. Some improvements over the strongest baseline are also relatively small.

Question 3: Would SAPO remain effective with deeper or unbalanced SID hierarchies, or when rewards are derived from noisy user interactions rather than ground-truth SID matches?

Response:

We thank the reviewer for this suggestion. Below we report multi-seed results on the margins over the strongest baseline, followed by each setting the reviewer raised.

Multi-seed results. We ran three Stage-3 seeds from a common Stage-2 checkpoint for each method (mean ± std; outcome-GRPO is the pure GRPO variant of Table 2):

DatasetMethodR@5N@5R@10N@10
Officeoutcome-GRPO0.1227±0.00420.0993±0.00270.1482±0.00500.1075±0.0031
SIDReasoner0.1370±0.00240.1108±0.00160.1619±0.00310.1188±0.0019
SAPO0.1373±0.00260.1145±0.00090.1648±0.00210.1250±0.0012
Video Gamesoutcome-GRPO0.0310±0.01400.0225±0.01050.0433±0.01510.0264±0.0114
SIDReasoner0.0569±0.00190.0379±0.00120.0887±0.00200.0480±0.0013
SAPO0.0624±0.00170.0416±0.00100.0991±0.00180.0533±0.0015

On Office and Video Games, SAPO has the highest mean on every metric, and its gains are most consistent on NDCG and R@10. On Industrial, Figure 10 in Appendix N shows that SAPO outperforms outcome-GRPO on all four metrics with a smaller standard deviation, and also has a higher mean than SIDReasoner on every metric.

Larger catalogs. We trained single runs on two larger Amazon catalogs: Toys (19,412 users / 11,924 items) and Sports (35,598 users / 18,357 items). Both use a per-user leave-one-out split: each user's last item is the test target and the second-to-last item the validation target.

DatasetMethodR@5N@5R@10N@10
ToysSIDReasoner0.03420.02120.05890.0291
SAPO0.03590.02200.06180.0306
SportsSIDReasoner0.01530.00900.02970.0135
SAPO0.01500.01020.03040.0139

Deeper hierarchy. With a four-level SID hierarchy on Office (K=4), SAPO is higher than SIDReasoner on all four metrics.

Office, K=4R@5N@5R@10N@10
SIDReasoner0.09130.06430.11560.0866
SAPO0.09520.07080.12220.0901

Unbalanced hierarchies. The hierarchies in the main experiments are unbalanced: level 1 uses only 19–34% of its codes (Table 5).

Multiple positives. Under a two-positive setting on Office, built with a per-user leave-two-out split, each rollout's step rewards are computed against the positive whose SID has the greatest hierarchical overlap with the generated SID. SAPO is higher on all four metrics.

Office, two positivesR@5N@5R@10N@10
SIDReasoner0.08630.05600.11370.0699
SAPO0.08840.05860.11540.0727

Noisy targets. On Industrial, each Stage-3 training target is replaced by a uniformly random catalog item with probability ε∈{0.2,0.4}\varepsilon\in\{0.2, 0.4\}, while validation and test stay clean. SAPO stays within 2% of clean training on NDCG@10 (−1.9% at ε=0.2\varepsilon=0.2 and −0.5% at ε=0.4\varepsilon=0.4; neither significant under a paired sign-flip test, p=0.35p=0.35 and 0.790.79), so its denser per-level reward does not amplify target noise in this range.

Backbone scale. In Appendix O (Industrial), SAPO outperforms SIDReasoner in all 12 backbone-size × metric comparisons. At 8B the gains are +0.0066 / +0.0074 / +0.0012 / +0.0055 on R@5 / N@5 / R@10 / N@10. The values plotted in Appendix O, Figure 11, are:

IndustrialBackboneR@5N@5R@10N@10
SIDReasoner1.7B0.11090.09050.14380.1010
SAPO1.7B0.11810.09350.14440.1018
SIDReasoner4B0.12010.09800.14660.1071
SAPO4B0.12880.10080.14700.1082
SIDReasoner8B0.12370.10040.14900.1095
SAPO8B0.13030.10780.15020.1150

We have added the multi-seed table to Appendix N, and the Toys/Sports, K=4, two-positive and noisy-target results as Appendices T, U, V and W of the revised manuscript.

Reviewer_ECyhOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors propose SAPO, which shifts the reward attribution in generative recommendation from assigning a single advantage value to the entire sequence to assigning an independent advantage value to each inference step. This allows for more precise rewarding or penalizing of specific SID tokens, thereby enhancing both training stability and recommendation performance.

Critical strengths
  1. This paper provides fine-grained credit assignment by aligning rewards with individual SID tokens, enhancing learning precision.

  2. This paper stabilizes reinforcement learning training dynamics, reducing volatility and improving convergence rates. They demonstrates improved recommendation performance across multiple datasets, indicating its robust applicability.

Critical weaknesses
  1. SAPO increases implementation complexity due to the need for step-aligned reward and advantage computation.

  2. SAPO relies on high-quality dataset tokenization to be effective, which may limit its applicability in noisy data environments.

  3. The steps for reward alignment and advantage computation proposed in this paper are already quite standard in the field of LLM research; extending them to recommendation systems seems like a rather straightforward conclusion.

Questions for authors

See weaknesses part.

6 comments

We thank the reviewer for the detailed feedback and constructive reviews. We have uploaded a revised manuscript (all changes are highlighted in blue).

Weakness 1

Weakness 1: SAPO increases implementation complexity due to the need for step-aligned reward and advantage computation.

Response:

We thank the reviewer for raising this point. As defined in Section 4, SAPO changes three parts of standard GRPO: the reward (per-level matches at SID positions), the advantage (group z-score per level instead of per rollout), and the token aggregation (division by step length). It does not require a learned reward model, a critic, or extra rollouts. Appendix L (Table 7, Office, 4 GPUs) reports the cost:

MethodWall-clock / step (s)Peak GPU memory (GB)Reward + advantage / step (s)
Outcome85.1574.830.59
SAPO85.0374.891.13

The extra reward/advantage computation (0.54 s) is about 0.63% of a full update step, and peak memory is essentially the same. We have added a one-sentence summary of this overhead to the main text of the revised manuscript.


Weakness 2

Weakness 2: SAPO relies on high-quality dataset tokenization to be effective, which may limit its applicability in noisy data environments.

Response:

We thank the reviewer for raising this concern. SAPO uses the same kind of RQ-VAE semantic-ID tokenizer as other SID-based recommenders such as TIGER [1] and SIDReasoner, and its per-level reward requires a hierarchical SID, which residual quantizers such as the RQ-VAE of [1] provide. We tested SAPO with a different tokenization, with different upstream reasoning traces, and with noisy training targets.

Deeper hierarchy. With a four-level RQ-VAE on Office (K=4), SAPO is ahead of SIDReasoner on all four metrics.

Office, K=4R@5N@5R@10N@10
SIDReasoner0.09130.06430.11560.0866
SAPO0.09520.07080.12220.0901

Upstream trace quality. SAPO is less sensitive to the Stage-2 reasoning traces than Stage 2 itself: when the trace generator changes, the spread in relative R@10 / N@10 shrinks from 5.0% / 4.4% after Stage 2 to 0.9% / 0.6% after SAPO (Appendix Q).

Noisy targets. On Industrial, each Stage-3 training target is replaced by a uniformly random catalog item with probability ε∈{0.2,0.4}\varepsilon\in\{0.2, 0.4\}, while validation and test stay clean. SAPO stays within 2% of clean training on NDCG@10 (−1.9% at ε=0.2\varepsilon=0.2 and −0.5% at ε=0.4\varepsilon=0.4; neither significant under a paired sign-flip test, p=0.35p=0.35 and 0.790.79), so its denser per-level reward does not amplify target noise in this range.

We have added the K=4 and noisy-target results as Appendices U and W of the revised manuscript.

Weakness 3

Weakness 3: The steps for reward alignment and advantage computation proposed in this paper are already quite standard in the field of LLM research; extending them to recommendation systems seems like a rather straightforward conclusion.

Response:

We thank the reviewer for this comment. Step-level rewards [2], [3] and group-relative advantages [4] are standard tools in LLM reinforcement learning. Applying them to reasoning-based SID decoding raises three problems that standard LLM reasoning tasks do not have, and SAPO is designed around them.

First, the verifiable signal is not on the reasoning steps. In LLM process supervision, each step is a span of reasoning scored by human labels [2] or a learned process reward model [4]. Here nothing scores the thinking text. Only the KK SID tokens can be verified, and all of them come after the last thinking block, so token order does not say which block should receive level-kk credit. That pairing has to be specified. Reversing or randomly permuting it lowers NDCG by 11.0–12.6% on Office (Appendix P).

Second, one rollout carries KK advantages, and the steps differ widely in length. The reward checks only the SID token and does not depend on step length, but under GRPO's token-mean aggregation the total weight of step kk grows with its token count. With one advantage per rollout, all steps are pushed in the same direction and step length only rescales the update; with KK advantages, step length decides which level's signal dominates. Per-step advantages with token-mean aggregation, close to the process-supervision variant of GRPO [4], do not train on Office: N@10 ends at 0.1013, against 0.1020 for the Stage-2 initialization. Step normalization (Eq. (9)) removes the length factor (full SAPO: 0.1241).

Third, outcome rewards are far sparser than in typical LLM reasoning tasks. From the Stage-2 checkpoint, all 16 rollouts of a group miss the target item in 72–99% of groups across five Amazon catalogs (the three main datasets plus Toys and Sports), so the outcome hit advantage is zero there. In 41–54% of these groups on Office, Industrial and Video Games, at least one SID level's match still varies and gives SAPO a signal. A denser scalar reward captures these differences but, under GRPO, still broadcasts one advantage per rollout, so it cannot tell which level was right: the dense-scalar baseline in Table 2 is below SAPO on 11 of 12 entries.

We have rewritten the contribution statement in Section 1 accordingly and added the group-signal diagnostic to Appendix E.1 of the revised manuscript.

References

[1] Rajput, Shashank, et al. "Recommender Systems with Generative Retrieval." NeurIPS 2023.

[2] Lightman, Hunter, et al. "Let's Verify Step by Step." ICLR 2024.

[3] Ng, Andrew Y., Daishi Harada, and Stuart Russell. "Policy invariance under reward transformations: Theory and application to reward shaping." ICML 1999.

[4] Shao, Zhihong, et al. "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models." arXiv:2402.03300, 2024.

Reviewer_ECyhOpenReview ↗

Official Comment by Reviewer ECyh

Thank you for the response and the new experiments. The concern about implementation complexity is resolved, but two questions remain. First, none of the added experiments varies the quality of the tokenizer itself: how does SAPO behave with a lower-quality SID tokenizer, and how do the baselines behave under the same noisy targets? Second, on novelty, the per-step advantage is the part that goes beyond standard reward shaping, yet in Table 2 removing it makes a clear difference only on Office. Is the gain from the per-step advantage statistically significant on each dataset? I would consider revising my score if you could provide further clarification of reliability across different datasets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.