- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes ResTD, which uses residual-derived codeword distributions from a frozen indexer to supervise a generative retriever’s decoder states. It supervises current and future SID levels without changing the index or inference procedure. Experiments show gains in multilingual retrieval, NQ320K document retrieval, and generative recommendation.
Critical strengths
- The method uses information from the indexer without adding inference cost. ResTD derives soft targets from residual geometry and supervises future SID levels at earlier decoder states. Reconstructing residuals along stored SID paths and correcting teacher–SID disagreements handles a practical compatibility issue. The auxiliary heads are removed at inference, so the method requires no changes to the deployed index or decoding procedure.
- The experiments include thorough controls. The main comparisons use the same initialization, minibatch order, optimization budget, and three paired seeds. ResTD improves over continued fine-tuning, current-level distillation, and matched future hard-target supervision. The code-conditioned, permuted, and shape-matched teacher controls provide further evidence that residual-derived preferences are useful. The NQ320K and recommendation experiments extend the evaluation beyond multilingual product search, and the measured training overhead is modest.
Critical weaknesses
-
The advantage over simpler representation supervision and related distributional methods remains unclear. Soft codebook supervision has been used in RQ-Transformer and related recommendation methods, including ETEGRec and SODA. ResTD’s main distinction is the combination of a frozen indexing teacher, SID-consistent targets, and multi-horizon decoder supervision.
The controls support the value of residual-derived targets, but there is no matched comparison with direct residual-vector regression or codebook-expectation matching at the same decoder states and horizons. This would help determine whether the distributional objective adds value beyond simpler continuous supervision using the same indexing information. A more explicit comparison with existing codebook-distribution alignment methods would also clarify the novelty.
-
The broad baseline comparison is less controlled than the CaLIR experiments. Table 1 combines published baseline scores with reproduced CaLIR and ResTD results. Appendix A.3 evaluates distinct SIDs using binary relevance for Exact, Substitute, and Complement items, taking the maximum relevance across products that share a SID. It is unclear whether all published baselines use the same evaluation unit and relevance definition.
The matched CaLIR comparisons remain convincing. However, the claim that ResTD outperforms all sparse, dense, and generative baselines needs either qualification or a reevaluation of selected strong baselines under the same protocol.
Questions for authors
- How does ResTD compare with direct residual-vector regression or codebook-expectation matching using the same heads, horizons, training budget, and auxiliary-loss schedule? Please also explain how it differs from SODA’s distribution-level supervision.
- Do all Table 1 scores use the same SID-level deduplication and binary relevance definitions? If not, could selected strong baselines be reevaluated under the shared protocol, or could the table distinguish directly comparable results from published scores?
- In the recommendation experiments, were TIGER and the other baselines trained with matched data, tokenization, and optimization settings? Please report variability across seeds, particularly for the smaller gains on Instruments.
5 comments
Official Response to Reviewer g3Ki (2/3)
Q2: (From Weakness 2 and Question 2) Do all Table 1 scores use the same SID-level deduplication and binary relevance definitions? If not, could selected strong baselines be reevaluated under the shared protocol, or could the table distinguish directly comparable results from published scores?
A2: Thank you for pointing this out. We apologize for the lack of clarity about the sources and evaluation protocols for the scores in Table 1. We will clarify these details in the experimental setup appendix.
(2.1) For the published baselines that also appear in the CaLIR paper’s main table, Table 1 uses the ESCI scores reported in that paper. The scores for FORGE, HGRec, LaSER, and CaLIR in Table 1 come from reproductions conducted for this study. The table below lists the published CaLIR scores, the reproduced baseline scores, and the ResTD results.
Published and locally evaluated results on ESCI (%)
| Method | US R@5 | US R@10 | US R@100 | US N@10 | US N@100 | ES R@5 | ES R@10 | ES R@100 | ES N@10 | ES N@100 | JP R@5 | JP R@10 | JP R@100 | JP N@10 | JP N@100 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| FORGE (reproduced) | 5.59 | 9.26 | 35.11 | 8.50 | 16.21 | 4.87 | 8.29 | 28.42 | 9.99 | 16.66 | 3.88 | 6.58 | 24.06 | 8.27 | 14.22 |
| HGRec (reproduced) | 6.70 | 11.41 | 34.61 | 10.19 | 17.32 | 5.76 | 9.60 | 31.56 | 11.37 | 18.32 | 3.78 | 6.18 | 24.37 | 8.40 | 13.83 |
| LaSER (reproduced) | 6.52 | 10.23 | 35.37 | 8.19 | 15.74 | 5.28 | 8.40 | 32.22 | 8.69 | 16.16 | 5.88 | 8.78 | 28.66 | 9.28 | 15.53 |
| CaLIR (published) | 7.22 | 11.75 | 36.15 | 10.75 | 18.14 | 6.06 | 10.43 | 34.83 | 12.15 | 20.03 | 5.57 | 9.64 | 31.71 | 11.74 | 18.85 |
| CaLIR (reproduced) | 7.50 | 12.69 | 37.40 | 11.12 | 18.79 | 6.03 | 10.48 | 34.15 | 12.43 | 19.88 | 6.05 | 9.77 | 30.89 | 11.77 | 18.42 |
| ResTD | 9.28 | 14.34 | 39.58 | 12.99 | 20.79 | 6.86 | 11.76 | 35.38 | 14.22 | 21.54 | 7.61 | 11.63 | 32.86 | 14.18 | 20.79 |
(2.2) For each query, we treat Exact, Substitute, and Complement labels as relevant. When multiple products share a complete SID, we assign that SID the highest relevance label among the products mapped to it. Before computing the metrics, we deduplicate each ranking by complete SID and retain its highest-ranked occurrence. The published CaLIR scores follow the product-level ranking protocol described in the CaLIR paper and are listed separately in the table.
For a fair comparison, we follow the training protocol described in the CaLIR paper when reproducing CaLIR. We use exactly the same complete-SID deduplication rule and binary relevance definition to evaluate the reproduced FORGE, HGRec, LaSER, and CaLIR models, as well as ResTD.
Official Response to Reviewer g3Ki (3/3)
Q3: (From Question 3) In the recommendation experiments, were TIGER and the other baselines trained with matched data, tokenization, and optimization settings? Please report variability across seeds, particularly for the smaller gains on Instruments.
A3: Thank you for asking us to clarify the recommendation setup and report results across seeds.
(3.1) All baselines and ResTD use the same data splits, item tokenizer, and SID mapping. We evaluate them against the same full item catalog. For each random seed, we use the same order of training examples, optimization procedure, and maximum number of training epochs. Each method uses its default hyperparameters.
(3.2) The table reports the results for each of the three seeds, along with the mean and standard deviation. The last row reports ResTD’s relative improvement over the strongest baseline’s three-seed mean for each metric.
Recommendation results across three training seeds (%)
| Method | Seed | B-Shop R@10 | B-Shop N@10 | I-Shop R@10 | I-Shop N@10 | Yelp R@10 | Yelp N@10 |
|---|---|---|---|---|---|---|---|
| SASRec | 42 | 5.88 | 3.13 | 9.47 | 6.90 | 2.96 | 1.52 |
| SASRec | 2027 | 5.83 | 3.05 | 9.49 | 6.86 | 3.04 | 1.63 |
| SASRec | 2028 | 6.03 | 3.23 | 9.36 | 6.97 | 2.93 | 1.60 |
| SASRec | Mean ± SD | 5.91 ± 0.10 | 3.14 ± 0.09 | 9.44 ± 0.07 | 6.91 ± 0.06 | 2.98 ± 0.06 | 1.58 ± 0.06 |
| BigRec | 42 | 2.99 | 1.98 | 5.76 | 4.91 | 1.69 | 1.42 |
| BigRec | 2027 | 3.13 | 1.84 | 5.66 | 5.00 | 1.84 | 1.47 |
| BigRec | 2028 | 2.99 | 1.85 | 5.83 | 5.06 | 1.66 | 1.34 |
| BigRec | Mean ± SD | 3.04 ± 0.08 | 1.89 ± 0.08 | 5.75 ± 0.09 | 4.99 ± 0.08 | 1.73 ± 0.10 | 1.41 ± 0.07 |
| P5-SID | 42 | 5.84 | 3.35 | 9.64 | 7.30 | 3.24 | 1.70 |
| P5-SID | 2027 | 5.73 | 3.20 | 9.53 | 7.21 | 3.34 | 1.64 |
| P5-SID | 2028 | 5.94 | 3.22 | 9.77 | 7.38 | 3.24 | 1.59 |
| P5-SID | Mean ± SD | 5.84 ± 0.11 | 3.26 ± 0.08 | 9.65 ± 0.12 | 7.30 ± 0.09 | 3.27 ± 0.06 | 1.64 ± 0.06 |
| P5-CID | 42 | 5.97 | 3.47 | 9.87 | 7.51 | 3.47 | 1.81 |
| P5-CID | 2027 | 5.88 | 3.39 | 9.84 | 7.41 | 3.54 | 1.72 |
| P5-CID | 2028 | 6.06 | 3.56 | 9.99 | 7.61 | 3.46 | 1.80 |
| P5-CID | Mean ± SD | 5.97 ± 0.09 | 3.47 ± 0.09 | 9.90 ± 0.08 | 7.51 ± 0.10 | 3.49 ± 0.04 | 1.78 ± 0.05 |
| LETTER | 42 | 6.72 | 3.64 | 11.22 | 8.31 | 4.26 | 2.31 |
| LETTER | 2027 | 6.81 | 3.68 | 11.22 | 8.42 | 4.24 | 2.20 |
| LETTER | 2028 | 6.75 | 3.58 | 11.29 | 8.34 | 4.41 | 2.40 |
| LETTER | Mean ± SD | 6.76 ± 0.05 | 3.63 ± 0.05 | 11.24 ± 0.04 | 8.36 ± 0.06 | 4.30 ± 0.09 | 2.30 ± 0.10 |
| TIGER | 42 | 6.10 | 3.31 | 10.58 | 7.97 | 4.07 | 2.13 |
| TIGER | 2027 | 5.99 | 3.44 | 10.64 | 7.89 | 4.22 | 2.16 |
| TIGER | 2028 | 5.97 | 3.42 | 10.43 | 8.12 | 4.08 | 2.02 |
| TIGER | Mean ± SD | 6.02 ± 0.07 | 3.39 ± 0.07 | 10.55 ± 0.11 | 7.99 ± 0.12 | 4.12 ± 0.08 | 2.10 ± 0.07 |
| ETEGRec | 42 | 6.15 | 3.35 | 11.06 | 8.10 | 4.15 | 2.14 |
| ETEGRec | 2027 | 6.26 | 3.34 | 10.92 | 8.20 | 4.24 | 2.06 |
| ETEGRec | 2028 | 6.02 | 3.22 | 10.94 | 8.16 | 4.21 | 2.21 |
| ETEGRec | Mean ± SD | 6.14 ± 0.12 | 3.30 ± 0.07 | 10.97 ± 0.08 | 8.15 ± 0.05 | 4.20 ± 0.05 | 2.14 ± 0.08 |
| ResTD | 42 | 7.28 | 4.00 | 11.42 | 8.45 | 4.87 | 2.66 |
| ResTD | 2027 | 7.37 | 4.02 | 11.53 | 8.56 | 5.02 | 2.76 |
| ResTD | 2028 | 7.35 | 4.08 | 11.47 | 8.58 | 4.88 | 2.54 |
| ResTD | Mean ± SD | 7.33 ± 0.05 | 4.03 ± 0.04 | 11.47 ± 0.06 | 8.53 ± 0.07 | 4.92 ± 0.08 | 2.65 ± 0.11 |
| Improv. (%) | Mean | +8.48 | +11.01 | +2.05 | +2.07 | +14.41 | +15.20 |
On Instruments, ResTD improves both recall and ranking over TIGER and LETTER, the strongest baseline, at every seed. The paired gains over LETTER are 0.23 ± 0.07 percentage points in R@10 and 0.17 ± 0.06 percentage points in N@10. The smaller gains on Instruments are consistent across all three seeds.
Thank you again for taking the time to review our paper. We hope our responses and additional results address your concerns. We would be happy to answer any further questions.
Official Response to Reviewer g3Ki (1/3)
We sincerely appreciate your careful review and thoughtful comments. We provide detailed responses to your questions below and have highlighted the corresponding changes in the revised manuscript.
A1: Thank you for suggesting these controls. We added direct residual-vector regression and codebook-expectation matching to test whether matching codeword distributions improves retrieval when all three methods use the same indexing information.
(1.1) All three variants use the same supervised decoder positions, projection heads, prediction horizons, training budget, and auxiliary-loss schedule. Direct residual-vector regression trains the projected decoder state to predict the residual vector reconstructed along the final stored SID. Codebook-expectation matching trains the same state to predict the mean codeword vector weighted by the corrected teacher probabilities. ResTD uses the same corrected teacher distribution. It computes a student distribution from the distances between the projected state and the frozen codewords, then matches it to the teacher distribution using KL divergence.
Comparison of distributional supervision and continuous regression objectives on ESCI (%)
ResTD scores higher than both baselines on every metric across all three ESCI locales. The comparison with codebook-expectation matching directly tests how to use the same residual teacher. Matching probability distributions over the actual codewords achieves better retrieval than regressing the teacher’s expected codeword vector.
(1.2) ResTD’s contribution is to reconstruct codeword preferences at each level along the final SID stored in a fixed index and use those preferences to supervise earlier decoder states across multiple horizons. We reconstruct the residuals along the collision-resolved SID and adjust the teacher distributions so that each stored codeword has the highest probability while preserving the probability ratios among the alternatives. During training, the same decoder state, conditioned on the query and the current SID prefix, matches teacher distributions from both the current and later quantization levels. This lets the decoder learn preferences among the actual alternative codewords at later levels before generating the corresponding codes.
RQ-Transformer applies residual soft labels to autoregressive next-code prediction at the corresponding level. ETEGRec aligns the codeword distributions of the user history and the target item, and aligns the first decoder state with the reconstructed item representation. SODA constructs distributions from pooled history encoder states and uses their similarity to the item distribution in a contrastive ranking objective.
To evaluate the effect of soft targets at later levels, we compare future hard SID targets with future residual soft targets while keeping current-level supervision and the prediction horizon fixed.
Retrieval performance with matched horizons and different future targets
With the same prediction horizon, future residual soft targets improve retrieval ranking in all three locales. The comparison shows that relative preferences among alternative codewords at later levels provide a useful training signal for earlier decoder states.