- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SAPO pairs each thinking block with one SID-token position and assigns that pair a level-specific match reward and group-relative advantage. It averages token contributions within each pair while retaining a blocked generation layout: all reasoning precedes all SID tokens. Experiments on three Amazon categories compare ranking quality, training dynamics, reward/advantage ablations, block-credit permutations, and larger backbones.
Critical strengths
- The evidence goes beyond a single outcome-GRPO comparison. SAPO leads 11 of 12 entries in Table 1. Table 2 separates dense match rewards from localized advantages, and Appendix N reports three Stage-3 seeds on Industrial from a common Stage-2 checkpoint.
- The block-to-level pairing is tested directly. Appendix P keeps SID-token credit fixed and changes only the advantages assigned to thinking blocks. Both reversed and random assignments reduce all four Office metrics. This supports the usefulness of the ordered pairing under the actual blocked decoding layout.
Critical weaknesses
- The stability comparison also changes the policy-loss scale. Equation (9) divides each step's advantage by its length while retaining the total-response-token denominator used in Equation (2). This changes both credit placement and the weighting/scale of the policy update. Appendix K uses the same nominal learning rate and KL coefficient for SAPO and outcome baselines. Table 2 does not separately isolate step normalization, so reduced gradient norms and KL drift cannot by themselves be attributed to better credit assignment.
- Table 4 does not reconcile with the stated sample construction. Appendix H describes leave-one-out splitting with sliding-window next-item pairs. For Office, 4,866 users with an average full sequence length of 5.97 imply roughly 29,050 interactions, yet the table lists 38,924 training pairs before validation and test are counted. The same issue appears in the other categories. This needs clarification about augmentation, repeated windows, or what the reported average measures; it prevents reconstructing the stated dataset from the paper alone.
Questions for authors
- Implementation clarification: Is Equation (9)'s double reduction exactly what is implemented, and how is the KL term reduced relative to it? Can a normalization-only or scale-matched outcome baseline separate the stability effect from the credit-placement effect?
- Data clarification: What counting or augmentation rule reconciles Table 4's training-pair counts with its user counts and average sequence lengths? Please specify which samples enter Stage 3.
5 comments
W2 & Q2
Weakness 2: Table 4 does not reconcile with the stated sample construction. Appendix H describes leave-one-out splitting with sliding-window next-item pairs. For Office, 4,866 users with an average full sequence length of 5.97 imply roughly 29,050 interactions, yet the table lists 38,924 training pairs before validation and test are counted. The same issue appears in the other categories. This needs clarification about augmentation, repeated windows, or what the reported average measures; it prevents reconstructing the stated dataset from the paper alone.
Question 2: Data clarification: What counting or augmentation rule reconciles Table 4's training-pair counts with its user counts and average sequence lengths? Please specify which samples enter Stage 3.
Response:
We thank the reviewer for checking this. The description in the original manuscript was incorrect, and the "#Users" column listed the number of test samples rather than distinct users. Each user's chronologically sorted sequence yields sliding-window samples (history of at most 10 items, next-item target). We sort the samples from all users by time and split them 8:1:1 into training, validation and test sets. There is therefore no information leakage between the three sets. The corrected statistics are:
| Dataset | Train / Valid / Test samples | Distinct users | Distinct users in test | Avg. seq. len. |
|---|---|---|---|---|
| Office | 38,924 / 4,866 / 4,866 | 8,328 | 2,024 | 5.97 |
| Video Games | 49,133 / 6,142 / 6,142 | 9,066 | 2,122 | 6.45 |
| Industrial | 36,259 / 4,532 / 4,533 | 7,694 | 1,987 | 5.96 |
"Avg. seq. len." is the mean of (history length + 1) over test samples. With this correction the counts are consistent: the training, validation and test samples form an 8:1:1 split (38,924 / 4,866 / 4,866 for Office), and 5.97 is averaged over the 4,866 test samples rather than over users.
Stage 3 uses all training samples (38,924 / 49,133 / 36,259), which are also the instances used to build the Stage-2 traces. The Stage-3 checkpoint is selected on the validation split. All methods are evaluated on the same test samples, so every comparison in the paper is on identical data.
We have corrected Section 5.1, Appendix H and Table 4 (columns and caption) in the revised manuscript, and added the distinct-user counts.
References
[1] Liu, Zichen, et al. "Understanding R1-Zero-Like Training: A Critical Perspective." COLM 2025.
[2] Yu, Qiying, et al. "DAPO: An Open-Source LLM Reinforcement Learning System at Scale." NeurIPS 2025.
Thank you again for your careful review. If anything in our response would benefit from further clarification, we would be happy to elaborate and discuss any remaining concerns.
We thank the reviewer for the detailed feedback and constructive reviews. We have uploaded a revised manuscript (all changes are highlighted in blue).
W1 & Q1
Response:
We thank the reviewer for this careful reading. Eq. (9) changes both credit placement and update scale. Prior work has shown that how GRPO normalizes by response length and aggregates the token-level loss changes how each response is weighted in the policy update [1], [2]. We therefore separate the two effects with a 2×2 design on Office:
In Table 2's "w/o per-step match reward", every step receives the same final exact-match reward, so all steps carry the same exact-match signal and the change from GRPO is the Eq. (9) normalization. This row is the scale-matched outcome baseline, and we have renamed it "normalization-only" in the revised manuscript to make this explicit.
Stability. With per-step credit but without step normalization, the model stays at the Stage-2 level (N@10 0.1013 vs. 0.1020), and individual reasoning-step lengths change sharply during training. Figure 4 (Industrial) shows the same split: the single-component ablations, including normalization-only, avoid pure GRPO's gradient-norm explosion, and SAPO reaches the highest reward curve (Section 5.4).
Ranking gains at matched scale. SAPO outperforms normalization-only on all 12 dataset × metric entries in Table 2: +0.0103 / +0.0116 / +0.0114 / +0.0126 on Office, and +0.0022 to +0.0055 on Video Games and Industrial. The dense-scalar baseline ("w/o per-step advantage") also uses Eq. (9), and SAPO outperforms it on 11 of 12 entries.
Implementation. For the policy-gradient term, each step's advantage is divided by its token count ∣τ(k)∣+1 as in Eq. (9), and the result is aggregated with a token-mean over all response tokens (the ∑ℓ∣yℓ∣ denominator of Eq. (2)). The KL term uses the low-variance (k3) estimator to the Stage-2 reference with coefficient 10−3; it is added to the loss with the same token-mean over all response tokens and no step-length division.
SAPO and normalization-only share the same Eq. (9) scaling of the PG term and the same KL term, so their PG-to-KL weight is identical and the comparison isolates credit placement. Relative to pure GRPO, dividing the PG term by ∣y(k)∣ while leaving KL undivided raises the effective KL strength relative to PG and lowers the effective step size. We have stated these details in Section 4.3 and Appendices J–K, added the 2×2 comparison as Appendix X, and revised Sections 4.3, 5.3 and 5.4 to attribute stability to step normalization and the matched-scale ranking gain to per-step credit placement.