- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper shows that RQ-VAE semantic-ID tokenization is not invariant to a rescaling of the input embeddings that leaves the upstream recommender's rankings unchanged.
- Counterexample. Centered item scaling x → μ + c(x − μ) with inverse history scaling preserves every pairwise teacher-score difference (Eq. 2; maximum discrepancy 5.25×10⁻⁷). With native SASRec-512 embeddings the same RQ-VAE recipe collapses the catalog to one base SID in all nine runs; with c = 8 it uses the codebooks broadly and improves Hit@10/NDCG@10 in all nine paired runs (Table 1).
- Capacity vs. optimization. An exact reparameterization of the first encoder and last decoder layers shows the native-scale tokenizer can represent the same non-collapsed IDs (Prop. 1, verified in float64), so the failure is one of optimization rather than capacity.
- Proposed fix. At mapped states the reconstruction loss and its latent gradient scale by γ² while the quantization terms are unchanged (Eqs. 6–7), yielding a reconstruction-weight calibration f_t = (c_s/c_t)² f_s = 64 (Eq. 8).
- Validation. From a shared native initialization, calibration restores code use and reaches Hit@10 close to direct rescaling (Table 1); mapped-continuation experiments trace the effect to encoder update ratios and MinPPL trajectories (Fig. 2); Sec. 6 examines other input representations, catalog-wide normalization and residual FSQ, and reports that broader code use does not uniformly help.
Critical strengths
A clean causal chain from a controlled counterexample to a minimal intervention, within its chosen setting.
- The upstream recommendation function is held exactly fixed, so the comparison isolates tokenizer training rather than representation quality.
- Prop. 1 proves that the native-scale architecture can represent the good solution, and native-input controls stay collapsed when codebook size varies from 64 to 512 or latent width from 32 to 128 (Sec. 4.1), ruling out capacity.
- The difference is pinned to a specific term-balance change in the objective and in the encoder gradient (Eqs. 6–7), then verified at matched states: first-step encoder update-norm ratios of 1.740/1.149/1.026 with identical codebook updates, and MinPPL trajectories diverging from the same start (Fig. 2).
- The fix is a single scalar, and it reproduces the gains of rescaling (calibrated minus Scale-8 Hit@10: +0.00034, +0.00072, −0.00163).
- The paper is also honest about negative results: endpoint loss does not track code use (Sec. 5.3), broader code use does not consistently help (Sec. 6.1), and catalog-wide normalization still shows seed-dependent collapse (Sec. 6.2).
- Why it matters: the mechanism → intervention → re-measurement chain is rare in the SID literature, and the lesson that representation comparisons must control for numerical scale is worth stating.
Critical weaknesses
- The claim is about SID pipelines in general, but the downstream validation covers only one generator family, so the generality of both the finding and the remedy is unestablished.
- (a) A single downstream architecture. Every downstream number in the paper comes from one TIGER-style T5 encoder–decoder Transformer: four encoder and four decoder layers of width 128, randomly initialized, with trie-constrained beam search of width 20 (Sec. 3, App. B.3). Current generative recommenders span rather different consumers of SIDs — decoder-only generative recommenders, HSTU/OneRec-style generators, and retrieval-style consumers that use SIDs as targets or as features. The paper's central downstream effect is explicitly entangled with the generator's structure: under collapse all items share one four-token prefix so the suffix carries all discrimination, and the exported vocabulary shrinks from 42,543–49,360 tokens to 33,152–33,229 after calibration, changing the generator's tied embedding size and parameter count (Sec. 5.2, Table 5). The paper itself notes that "the experiment does not separately identify the contribution of semantic grouping." With one generator family, it cannot be determined how much of the Hit@10 gain is a property of SID quality versus of this particular encoder–decoder with this vocabulary and tying scheme.
- (b) A single tokenizer recipe, and a scale that is also narrow. The main study uses one RQ-VAE configuration (four 256-entry codebooks, 32-dim latent, residual STE with no reconstruction path to codebooks, losses mean-reduced over d_x = 512 versus latent coordinates, greedy levels 1–3 plus Sinkhorn at level 4). Residual FSQ appears only as a tokenizer-level control with no downstream evaluation (Sec. 6.3), and all three domains are small Amazon categories (Beauty, Instruments, Games). Since the collapse plausibly depends on the d_x/d_z reduction convention, it is unclear how much of the phenomenon is specific to this recipe.
- (c) Comparisons are also narrow. The remedy is not compared, under the main matched protocol, against the standard collapse remedies the paper cites: GRID-style input whitening, catalog-wide normalization (Sec. 6.2 states it "does not establish a matched comparison with calibration"), dead-code re-initialization, or SimVQ-style reparameterization. The analytic coefficient 64 also derives from a pilot-selected reference scale c = 8, and f_R ∈ {16, 32, 128, 256} all recover full codebook use (Sec. 5.2), so the derived value is not what matters.
- Why it matters: the paper's value is as a general lesson about the representation–tokenizer interface. Demonstrated on one generator, one tokenizer recipe and three small catalogs, and without matched comparisons against existing remedies, that lesson is not yet shown to generalize.
- The headline downstream gains are measured against a fully degenerate baseline, and away from collapse the effects are small and mixed.
- Native raw maps the entire catalog to a single base SID with NMSE ≈ 1.0 (Table 1). Complete collapse of this kind would be caught by routine utilization monitoring, so the +0.021–0.044 Hit@10 improvements largely compare a working tokenizer against a broken one.
- Where tokenizers are not collapsed, scale changes have small and inconsistent downstream effects: between RMS .008 and .064, Hit@10 ranges from −0.0018 to +0.0150 across nine representation–domain pairs, with non-monotonicity at intermediate scales (Table 2, App. F.2).
- Why it matters: the practical significance beyond "avoid a broken tokenizer" is not demonstrated, which compounds the generality concern above.
Questions for authors
- More downstream families. Please evaluate calibration with at least one downstream generator that is not a TIGER-style T5 encoder–decoder: for example a decoder-only generative recommender, an HSTU/OneRec-style generator, or a retrieval-style consumer that uses SIDs as features. Do the Hit@10/NDCG@10 gains persist?
- Disentangle SID quality from generator structure. Can you hold the generator vocabulary and tied-embedding size fixed across arms (e.g. by equalizing effective vocabulary, or by re-indexing the collapsed arm), so that the downstream gain is attributable to the learned partitions rather than to the change in prediction structure?
- Matched comparison with existing remedies. Under the main matched protocol, how does calibration compare downstream with GRID-style whitening, catalog-wide normalization, and dead-code reset / k-means re-initialization?
- Dependence on the loss-reduction convention. R is mean-reduced over the 512 input dimensions while Q is averaged over latent coordinates (Eq. 3). Is the collapse largely a consequence of this convention? Would a dimension-matched or sum-reduced reconstruction loss avoid it without any calibration?
- Do the conclusions hold on substantially larger catalogs (≥10⁵–10⁶ items) and under a non-RQ-VAE quantizer with downstream evaluation?
5 comments
Response to Reviewer bKXe — Part 2/3: Existing Remedies and Loss Reductions
Q3 / Weakness 1(c): Matched comparison with existing collapse remedies
Both calibration and catalog normalization recover broad code use with comparable downstream accuracy. ZCA whitening also prevents complete collapse but performs less well downstream, while the tested dead-code reset does not recover code use.
We compared Native raw, Native calibrated, catalog-wide normalization, ZCA whitening, ZCA whitening rescaled to native RMS, dead-code reset, and Scale-8 starting from the same frozen SASRec-512 catalog, with condition-specific preprocessing where applicable, and a common RQ-VAE architecture, tokenizer training budget, and TIGER evaluation protocol (seed 42). Table R7 reports code utilization and downstream accuracy.
Table R7. Matched collapse remedies: MinPPL and test Hit@10 (Beauty / Instruments / Games; seed 42).
| Condition | MinPPL (B / I / G) | Hit@10 (B / I / G) |
|---|---|---|
| Native raw | .004 / .004 / .004 | .04469 / .09646 / .06534 |
| Dead-code reset | .004 / .004 / .004 | .04469 / .09646 / .06534 |
| ZCA whitening + native RMS | .004 / .004 / .004 | .04469 / .09646 / .06534 |
| ZCA whitening | .933 / .748 / .816 | .05741 / .10409 / .10969 |
| Catalog normalization | .870 / .804 / .816 | .06560 / .11387 / .11897 |
| Native calibrated | .698 / .822 / .806 | .06453 / .11526 / .12050 |
| Scale-8 | .922 / .892 / .870 | .06865 / .11267 / .12057 |
Rescaling whitened embeddings back to native RMS leads to complete collapse again, identifying numerical scale as an important factor in the whitening result. Under the tested reset rule and residual STE, dead-code reset achieves the same test accuracy as Native raw.
These comparisons show that different remedies affect the input distribution or training objective in different ways, and broad code utilization does not necessarily yield the highest recommendation accuracy. Calibration provides an objective-level intervention derived from the scale mapping while leaving the original item embeddings unchanged, making it particularly useful for isolating the optimization mechanism studied here.
Q4 / Weakness 1(b): Is collapse driven by the loss-reduction convention?
The loss-reduction convention is an important factor because it changes the effective balance between reconstruction and quantization. Both coordinate-sum variants avoid complete collapse on native inputs, consistent with our optimization analysis.
We directly compared coordinate-mean and coordinate-sum reductions. The original objective is
where averages over input coordinates and over latent coordinates. The alternative reductions are
\text{Sum both:}\quad &512R+32Q=32(16R+Q),\\\\ \text{Sum reconstruction only:}\quad &512R+Q. \end{aligned}$$ The first has the same relative reconstruction-to-quantization weighting as $16R+Q$; the second corresponds to a relative reconstruction coefficient of 512. The common factor of 32 need not produce identical AdamW trajectories. The outcomes are compared in Table R8. **Table R8. Loss-reduction conventions: MinPPL and test Hit@10 (Beauty / Instruments / Games; seed 42).** | Objective | MinPPL (B / I / G) | Hit@10 (B / I / G) | |---|---|---| | $R+Q$ | .004 / .004 / .004 | .04469 / .09646 / .06534 | | $16R+Q$ | .821 / .811 / .855 | .06444 / .11051 / .12161 | | $512R+32Q$ | .854 / .775 / .807 | .06623 / .11300 / .11915 | | $64R+Q$ (calibrated) | .698 / .822 / .806 | .06453 / .11526 / .12050 | | $512R+Q$ | .851 / .861 / .838 | .06533 / .11347 / .11944 | Both coordinate-sum conventions also achieve downstream accuracy close to calibration. **The observed collapse is not an intrinsic inability to quantize native embeddings. This is the mechanism analyzed in the paper: at exactly mapped tokenizer states, input rescaling changes the reconstruction term and its latent gradient while leaving the corresponding quantization terms unchanged. The loss-reduction convention is another determinant of that balance.**Response to Reviewer bKXe — Part 3/3: Quantizer Generality and Practical Significance
Q5 / Weakness 1(b): Non-RQ-VAE tokenizers and larger catalogs
Scale sensitivity depends on the quantizer. Residual-FSQ shows smaller, mixed downstream changes without complete collapse, whereas RQ-KMeans produces identical IDs under Native and Scale-8. Our analytical mapping is independent of catalog size, but larger-catalog outcomes remain untested.
Residual-FSQ downstream evaluation. We added full TIGER evaluation of frozen Residual-FSQ tokenizers on Native, Scale-8, and catalog-normalized inputs (seed 42). Unlike Native raw RQ-VAE, Residual-FSQ does not collapse at native scale: base-SID collision is 0.36%–0.41% across the three datasets. Table R9 summarizes its downstream results.
Table R9. Residual-FSQ test results (Hit@10 / NDCG@10; seed 42).
| Dataset | Native | Scale-8 | Normalized |
|---|---|---|---|
| Beauty | .05830 / .03160 | .05902 / .03222 | .06256 / .03366 |
| Instruments | .11201 / .08755 | .11048 / .08514 | .10993 / .08508 |
| Games | .12185 / .08223 | .11895 / .08056 | .12277 / .08267 |
Scale-8 gives a small gain on Beauty but decreases both metrics on Instruments and Games. These mixed directions distinguish fixed-grid quantization from the RQ-VAE collapse regime; the RQ-VAE-specific calibration rule does not directly transfer to Residual-FSQ.
RQ-KMeans. As a non-neural, distance-based control, we compared Native and Scale-8 with four residual K-means levels and 256 codes per level, across three datasets and seeds 42–44. All nine paired runs have layer-wise assignment ARI = 1, full-SID agreement = 1, and identical downstream metrics, as detailed in Table R10.
Table R10. RQ-KMeans Native versus Scale-8 (test Hit@10 / NDCG@10; mean over seeds 42–44).
| Dataset | Native Hit@10 / NDCG@10 | Scale-8 Hit@10 / NDCG@10 | ΔHit@10 / ΔNDCG@10 |
|---|---|---|---|
| Beauty | .0633 / .0331 | .0633 / .0331 | .0000 / .0000 |
| Instruments | .1126 / .0868 | .1126 / .0868 | .0000 / .0000 |
| Games | .1222 / .0813 | .1222 / .0813 | .0000 / .0000 |
This agrees with the scale equivariance of Euclidean distances and K-means centroid updates under consistently transformed initialization. It also shows that native SASRec embeddings support useful discrete representations under another tokenizer. This contrast isolates tokenizer training mechanism as a key determinant of the observed non-invariance.
Catalog size. Our current empirical catalogs contain 10,439–17,256 items. Evaluation at – items remains to be completed. Proposition 1's exact reparameterization and the mapped-state loss/gradient identities in Section 4.2 do not depend on catalog cardinality, while collapse frequency and downstream impact at larger scales require separate empirical testing.
Weakness 2: Practical significance beyond the collapsed baseline
Our findings extend beyond recovering collapsed tokenizers: numerical scale affects downstream recommendation even without complete collapse, and our analysis identifies an optimization confound relevant to SID construction and evaluation.
We conducted an additional fixed-vocabulary evaluation using non-collapsed SASRec-32 SID catalogs at RMS 0.016 and 0.064 across three datasets. Within each dataset, generator vocabulary, parameter count, global-item-ID suffixes, and training protocols were matched.
Table R11. Non-collapsed SASRec-32 scale comparison with fixed vocabulary (test Hit@10 / NDCG@10; seed 42).
| Dataset | RMS 0.016 | RMS 0.064 | ΔHit@10 | ΔNDCG@10 |
|---|---|---|---|---|
| Beauty | .05682 / .03082 | .05866 / .03211 | +.00184 | +.00129 |
| Instruments | .11161 / .08554 | .11438 / .08821 | +.00277 | +.00267 |
| Games | .11699 / .07937 | .11783 / .08004 | +.00084 | +.00067 |
Both Hit@10 and NDCG@10 improve across all three datasets, demonstrating that scale-dependent downstream effects persist beyond complete collapse, with generator capacity and suffix identity controlled.
The practical significance extends beyond performance gains from rescaling. Our central finding is that representations preserving exactly the same upstream recommendation function can yield different learned SIDs and downstream outcomes under an identical tokenizer training recipe. Consequently, comparisons of item representations may conflate differences in recommendation information with scale-dependent tokenizer optimization.
Our exact reparameterization, mapped-state loss/gradient analysis, and matched interventions explain this mechanism and provide a principled basis for controlling numerical scale and objective weighting. This establishes both a mechanistic explanation and a methodological consideration for reliable SID construction and evaluation.
We appreciate the reviewer's specific suggestions, which helped clarify both the mechanism and its scope across tokenizers and downstream consumers.
Response to Reviewer bKXe — Part 1/3: Downstream Architecture and Vocabulary Controls
We sincerely thank Reviewer bKXe for the careful assessment of our controlled counterexample, exact reparameterization, and optimization analysis, as well as for the thoughtful and constructive suggestions that have helped us strengthen our experimental evaluation and clarify the scope of our findings. In response, we conducted controlled experiments on downstream architecture, vocabulary and suffix effects, existing collapse remedies, loss reductions, and alternative quantizers. Unless stated otherwise, results use tokenizer and generator seed 42 on the same test split; RQ-KMeans is evaluated over seeds 42–44. Our responses follow below.
Q1 / Weakness 1(a): Does the downstream improvement extend beyond TIGER/T5?
Yes. With a decoder-only GPT-2-style recommender, calibration improves NDCG@10 on all three datasets and Hit@10 on Beauty and Instruments; the Hit@10 change on Games is negligible. The downstream effect is therefore not limited to TIGER's encoder–decoder architecture.
We evaluated a 12-layer decoder-only GPT-2-style generative recommender (hidden size 128, four attention heads, tied embeddings), trained from scratch. It autoregressively predicts the target item's five SID tokens from the history SIDs. Evaluation uses full-catalog trie-constrained beam search (beam size 20). We reuse the frozen SID catalogs for Native raw, Native calibrated (fR=64), and Scale-8. Table R4 reports the resulting test accuracy.
Table R4. Decoder-only GPT-2 test results (Hit@10 / NDCG@10; seed 42).
Calibration increases Hit@10 by .01997 on Beauty and .01340 on Instruments. On Games, Hit@10 is nearly unchanged (+.00011), while NDCG@10 rises by .00349. NDCG@10 improves on all three datasets, and Scale-8 exceeds Native raw on both metrics throughout. The size of the gain varies substantially by dataset and consumer, extending the empirical scope beyond the original generator.
Q2 / Weakness 1(a): Do the gains persist when vocabulary size, tied-embedding parameter count, and suffix identity are held fixed?
Yes. Calibration retains its advantage over Native raw when vocabulary size, tied-embedding parameter count, and the fifth-token item-ID mapping are held fixed.
We conducted a fixed-vocabulary and shared-suffix control. Within each dataset, Native raw, Calibrated, and Scale-8 use identical vocabulary sizes and generator parameter counts. We compare the original group-local suffix with a global-item-ID suffix, where the fifth SID token uses the same unique item-ID mapping across conditions. Table R5 reports the controlled model sizes, and Table R6 reports the global-suffix test results.
Table R5. Fixed generator vocabulary and parameter count within each dataset.
Table R6. Fixed vocabulary and global-item-ID suffix: test Hit@10 / NDCG@10 (seed 42).
With vocabulary capacity and suffix identity controlled, calibration still improves Hit@10 over Native raw by approximately .0205 / .0314 / .0387 and NDCG@10 by .0119 / .0247 / .0285 (Beauty / Instruments / Games). The gains persist under the matched vocabulary and global-suffix controls, and the group-local suffix control shows the same direction of improvement.