- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies chain-of-thought distillation for LLM-based candidate reranking (Amazon Reviews 2023; 10 candidates with 3 positives and 7 uniformly sampled negatives). It observes that Gemma-4-26B teacher traces are dominated by verification and intermediate-ranking loops that rarely change the answer (§6.1, Tables 3–4). It proposes SCOReD, which works in three steps. (i) An LLM segments each trace into six typed stages. (ii) Segments are bucketed by attention from the student's </think> token. (iii) Each segment is edited with KEEP/REWRITE/FUSE/PRUNE, choosing the edit by a reward that combines the student's log-probability of the teacher's answer with length and perplexity penalties (Eq. 1–2). Qwen3 students (0.6B/1.7B/4B) are then fine-tuned on the edited traces. On Beauty, NDCG rises over raw-trace SFT (0.773→0.801 for 0.6B, 0.782→0.802 for 1.7B). Traces are about 42–44% shorter and parse errors drop (Table 1). Gains over raw SFT hold on three other domains (Table 2) and shrink with student size (Table 8). DAPO and OPSD after SFT bring no further gains (§6.3).
Critical strengths
- A well-motivated idea: edit away what the student does not need before distilling. §6.1 shows why raw teacher traces are poor distillation targets in recommendation. Verification and intermediate-ranking segments make up 63.3% of segments, and 92.89% of refinement loops leave the ranking unchanged (Tables 3–4). Turning this diagnosis into segment-level editing (KEEP/REWRITE/FUSE/PRUNE) rather than token pruning or whole-trace summarization is a sensible and reusable idea for CoT distillation.
- Consistent practical gains over raw-trace SFT. SCOReD improves NDCG over raw-trace SFT in all eight domain × size settings (Table 2) and on 4B (Table 8). It also shortens generated traces by 42–50% and lowers invalid-output rates (e.g., 3.51% → 0.40% for 0.6B, Table 1). For latency-constrained recommenders, a shorter and more format-reliable student is a meaningful outcome.
Critical weaknesses
- The contribution of the two student-aware components, attention scoring and the reward, is not established. The only ablation sweeps τ_low (Tables 5/7), which leaves both components in place throughout.
- Attention scoring. Its role is unclear. Attention from the pre-SFT student's </think> token only sorts segments into buckets. It is justified by an observation borrowed from Lan et al. (2026), with no analysis of what it captures, and the layers and heads used are not specified. With τ_high = 0.05, only the top 5% of segments can be kept, and MEDIUM segments are always rewritten. No variant replaces it with random or teacher-based bucketing.
- Reward. α = 0.005 and β = 0.1 are fixed. No result removes the length penalty, the perplexity penalty or the log P term, so their individual contributions are unknown. At τ_low = 0, the reward only chooses KEEP vs REWRITE for the top 5%. Yet this setting already moves NDCG from 0.773 to 0.793 (Table 5). Much of the gain may therefore come from having Gemma rewrite nearly every segment, not from student-aware selection.
- Baselines. A*-Thought matches SCOReD's NDCG (0.799 vs 0.801; 0.802 vs 0.802), has higher Recall@3 and produces shorter traces (0.85K vs 1.33K tokens; Table 1). If invalid rankings score zero NDCG, the NDCG on valid outputs only is nearly identical for raw-trace SFT and SCOReD (about 0.801 vs 0.804 for 0.6B, and 0.805 vs 0.805 for 1.7B, from Table 1). Much of the gain may then reflect output validity rather than better ranking.
- The evidence for better reasoning is weak, and several reported numbers do not reconcile.
- LAS. The claim that SCOReD keeps a clearer link between reasoning and ranking (§5) rests mainly on LAS. But over 90% of student CoTs already state the final ranking (App. G, Table 10), so recovering the ranking from those traces is close to trivial. LAS corrects for this only by weighting the small non-leaking subset at 50%. That subset differs in size by method (45 for Baseline SFT, 204 for SCOReD, 248 for A*-Thought), so the methods are scored on different examples, and no uncertainty is reported. More fundamentally, LAS measures consistency between the explanation and the decision, not whether the reasoning caused the decision. Even so, SCOReD's LAS is not higher than raw-trace SFT for 0.6B (0.656 vs 0.660), and Compress-Distill has the highest LAS for 1.7B (0.690).
- Reporting. The motivating statistics in §1 (9.35 rechecks per trace; 79% unchanged) differ from §6.1 (8.65 verification segments; 92.89% of loops unchanged; 47.15% of traces changed). The abstract's "2.88% NDCG / 2.96% Recall@3" cannot be derived from Tables 1, 2 or 8. Table 1's 3.51% baseline error rate for 0.6B conflicts with Table 6 (126/4,474 = 2.82%). The post-SFT experiments start from the τ_low = 0.00 student, not the selected 0.50 model (App. E–F).
Questions for authors
- With τ_low = 0.5 fixed on Beauty/0.6B, could you report ablations that (a) replace </think>-attention bucketing with random or teacher-attention bucketing, (b) rewrite every segment without reward-based selection, and (c) set α = 0 and β = 0 separately, and drop the log P term? Which layers and heads are averaged for the attention score? These would show what each component contributes.
- How are invalid rankings scored in Table 1? Please report NDCG on examples that are valid for both methods, using your existing outputs. Please also reconcile the 3.51% vs 2.82% baseline error rates (Tables 1 and 6).
- Does SCOReD's LAS advantage over A*-Thought persist when the explicit final-ranking statement is masked from the student CoTs, or when LAS is computed on a common set of examples? Please also give bootstrap confidence intervals for the LAS differences.
- How do the §1 statistics (9.35 rechecks; 79%) relate to those in §6.1? How is the abstract's 2.88% / 2.96% computed? Could you report means and standard deviations over a few SFT seeds for Table 1?
- Why were DAPO and OPSD started from the τ_low = 0.00 student, and does the negative result hold from the τ_low = 0.50 model?
Response to Reviewer K6dr
We thank the reviewer for the constructive feedback and for recognizing SCOReD's motivation, practical improvements, and consistent gains across recommendation domains and student sizes. We address each question below.
Q1. Component Ablations and Attention Scoring
We conduct a new ablation without student-attention importance scores or reward-based action selection: redundant intermediate-ranking stages are pruned using only the semantic stage labels, reducing teacher CoT length by 44.1%. This stage-based variant improves over Baseline SFT (NDCG 0.7818 vs. 0.773; error rate 3.51% → 2.46%), and adding student-aware selection raises NDCG further to 0.801. Compared with A*-Thought, which is also student-aware but removes sentences independently, SCOReD achieves comparable ranking metrics with higher LAS and a lower error rate. Please see Common Response 4 for details. We acknowledge that this experiment removes multiple components simultaneously. We will report additional component-level ablations and results across additional SFT seeds in the final manuscript.
Importance scoring with </think> token attention. The importance of a CoT token is computed from the attention of the </think> token to that token, averaged across all attention heads and layers. Segment importance is then computed by averaging the importance scores of its constituent tokens.
Q2. Invalid Ranking Scoring and Error Rate Discrepancy
Erroneous rankings receive a score of zero for NDCG, MAP, and Recall@3. As detailed in Common Response 1, the overwhelming majority of these errors originate in the reasoning process rather than in final-answer formatting.
We report ranking metrics on the common error-free subsets (3,926 examples for 0.6B and 3,966 for 1.7B) in Common Response 2. We find no statistically significant differences in ranking metrics between methods, including the Gemma-4-26B teacher.
Regarding the discrepancy between Tables 1 and 6, we mistakenly reported the 1.7B Baseline SFT error rate in Table 6. The 3.51% error rate in Table 1 is the correct value for the 0.6B Baseline SFT model. We will correct this in the final manuscript.
Q3. LAS and Explicit Ranking Leakage
We conduct an additional evaluation where we remove explicit complete rankings from the CoTs before LAS evaluation. SCOReD continues to outperform A*-Thought on both student sizes (0.2222 vs. 0.1987 for 0.6B; 0.2239 vs. 0.2046 for 1.7B), indicating that its relative advantage persists after removing explicit ranking statements. Please see Common Response 3 for details, including bootstrap CIs for the LAS differences. We additionally report LAS with 95% CIs on the common error-free subsets in Common Response 2, where SCOReD also exceeds A*-Thought (0.661 vs. 0.593 for 0.6B; 0.636 vs. 0.613 for 1.7B).
Q4. Clarification Regarding Reported Metrics
The difference comes from how refinement loops are counted. The average of 9.35 includes all R → ... → R/F patterns of all 8 rollouts of the Gemma teacher, while 8.65 counts only verification patterns (R → V → R and R → V → C → R) of the highest-NDCG rollout. Similarly, 79% of all refinement loops leave the ranking unchanged, compared to 92.89% of verification loops. The highest-NDCG rollouts usually start out with a better intermediate ranking and do not change their ranking often. We will clarify this distinction in the manuscript.
Based on Table 1, the 0.6B student's relative improvements are +3.62% in NDCG and +3.39% in Recall@3. The abstract's 2.88% NDCG and 2.96% Recall@3 improvements were computed under an earlier evaluation protocol in which incomplete rankings were scored as-is, while other erroneous rankings received zero. However, incomplete rankings can artificially inflate ranking metrics by omitting negative candidates ranked above a ground-truth item. We therefore adopt the consistent scoring convention described in Common Response 1, assigning zero to all erroneous rankings. We will update the abstract accordingly.
Q5. DAPO and OPSD Initialization
We selected the $\tau_{\mathrm{low}}=0.00$ checkpoint for the reported post-SFT experiments because its initial NDCG of 0.793 leaves greater room for potential improvement through DAPO or OPSD. In comparison, the $\tau_{\mathrm{low}}=0.50$ student already achieves 0.801 NDCG, approaching the teacher's 0.803.
We additionally conduct pilot experiments initialized from the $\tau_{\mathrm{low}}=0.50$ checkpoint:
Both pilot experiments show a slight performance regression relative to the SCOReD initialization, consistent with the trends observed in our reported $\tau_{\mathrm{low}}=0.00$ experiments.