acceptodds
Under review as a conference paper at ICLR 2027

Same Rankings, Different Semantic IDs: Optimization Non-Invariance in Tokenization

Abstract

Semantic-ID-based generative recommendation relies on tokenizers such as RQ-VAE to convert continuous item representations into discrete codes. Although scale and distribution are known to affect codebook collapse, their effect on the full recommendation pipeline is harder to isolate when the upstream representation also changes. We study input rescaling that preserves every pairwise recommendation-score difference. In our main study with 512-dimensional SASRec embeddings, the same training procedure collapses to a single base SID on native inputs, whereas rescaling produces broad codebook use and substantially better downstream recommendation. An exact tokenizer reparameterization shows that the same non-collapsed code assignments remain representable at native scale, separating representational capacity from training outcome. At these mapped states, reconstruction loss and its encoder-output gradient scale quadratically, whereas latent quantization terms remain unchanged. This analysis yields a simple reconstruction-loss calibration that keeps the native embeddings fixed. Controlled experiments from matched starting points show that the calibration changes encoder optimization and recovers broad code use. Across three Amazon datasets, calibration improves Hit@10 and NDCG@10 in all nine paired runs, reaching accuracy close to direct input rescaling. These findings show why representation comparisons should account for numerical scale and tokenizer optimization.

Primary area:
optimization
Keywords:
Recommender Systems, Generative Recommendation, Semantic IDs, Vector Quantization, Representation Learning, Optimization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bKXeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper shows that RQ-VAE semantic-ID tokenization is not invariant to a rescaling of the input embeddings that leaves the upstream recommender's rankings unchanged.

  • Counterexample. Centered item scaling x → μ + c(x − μ) with inverse history scaling preserves every pairwise teacher-score difference (Eq. 2; maximum discrepancy 5.25×10⁻⁷). With native SASRec-512 embeddings the same RQ-VAE recipe collapses the catalog to one base SID in all nine runs; with c = 8 it uses the codebooks broadly and improves Hit@10/NDCG@10 in all nine paired runs (Table 1).
  • Capacity vs. optimization. An exact reparameterization of the first encoder and last decoder layers shows the native-scale tokenizer can represent the same non-collapsed IDs (Prop. 1, verified in float64), so the failure is one of optimization rather than capacity.
  • Proposed fix. At mapped states the reconstruction loss and its latent gradient scale by γ² while the quantization terms are unchanged (Eqs. 6–7), yielding a reconstruction-weight calibration f_t = (c_s/c_t)² f_s = 64 (Eq. 8).
  • Validation. From a shared native initialization, calibration restores code use and reaches Hit@10 close to direct rescaling (Table 1); mapped-continuation experiments trace the effect to encoder update ratios and MinPPL trajectories (Fig. 2); Sec. 6 examines other input representations, catalog-wide normalization and residual FSQ, and reports that broader code use does not uniformly help.
Critical strengths

A clean causal chain from a controlled counterexample to a minimal intervention, within its chosen setting.

  • The upstream recommendation function is held exactly fixed, so the comparison isolates tokenizer training rather than representation quality.
  • Prop. 1 proves that the native-scale architecture can represent the good solution, and native-input controls stay collapsed when codebook size varies from 64 to 512 or latent width from 32 to 128 (Sec. 4.1), ruling out capacity.
  • The difference is pinned to a specific term-balance change in the objective and in the encoder gradient (Eqs. 6–7), then verified at matched states: first-step encoder update-norm ratios of 1.740/1.149/1.026 with identical codebook updates, and MinPPL trajectories diverging from the same start (Fig. 2).
  • The fix is a single scalar, and it reproduces the gains of rescaling (calibrated minus Scale-8 Hit@10: +0.00034, +0.00072, −0.00163).
  • The paper is also honest about negative results: endpoint loss does not track code use (Sec. 5.3), broader code use does not consistently help (Sec. 6.1), and catalog-wide normalization still shows seed-dependent collapse (Sec. 6.2).
  • Why it matters: the mechanism → intervention → re-measurement chain is rare in the SID literature, and the lesson that representation comparisons must control for numerical scale is worth stating.
Critical weaknesses
  1. The claim is about SID pipelines in general, but the downstream validation covers only one generator family, so the generality of both the finding and the remedy is unestablished.
  • (a) A single downstream architecture. Every downstream number in the paper comes from one TIGER-style T5 encoder–decoder Transformer: four encoder and four decoder layers of width 128, randomly initialized, with trie-constrained beam search of width 20 (Sec. 3, App. B.3). Current generative recommenders span rather different consumers of SIDs — decoder-only generative recommenders, HSTU/OneRec-style generators, and retrieval-style consumers that use SIDs as targets or as features. The paper's central downstream effect is explicitly entangled with the generator's structure: under collapse all items share one four-token prefix so the suffix carries all discrimination, and the exported vocabulary shrinks from 42,543–49,360 tokens to 33,152–33,229 after calibration, changing the generator's tied embedding size and parameter count (Sec. 5.2, Table 5). The paper itself notes that "the experiment does not separately identify the contribution of semantic grouping." With one generator family, it cannot be determined how much of the Hit@10 gain is a property of SID quality versus of this particular encoder–decoder with this vocabulary and tying scheme.
  • (b) A single tokenizer recipe, and a scale that is also narrow. The main study uses one RQ-VAE configuration (four 256-entry codebooks, 32-dim latent, residual STE with no reconstruction path to codebooks, losses mean-reduced over d_x = 512 versus latent coordinates, greedy levels 1–3 plus Sinkhorn at level 4). Residual FSQ appears only as a tokenizer-level control with no downstream evaluation (Sec. 6.3), and all three domains are small Amazon categories (Beauty, Instruments, Games). Since the collapse plausibly depends on the d_x/d_z reduction convention, it is unclear how much of the phenomenon is specific to this recipe.
  • (c) Comparisons are also narrow. The remedy is not compared, under the main matched protocol, against the standard collapse remedies the paper cites: GRID-style input whitening, catalog-wide normalization (Sec. 6.2 states it "does not establish a matched comparison with calibration"), dead-code re-initialization, or SimVQ-style reparameterization. The analytic coefficient 64 also derives from a pilot-selected reference scale c = 8, and f_R ∈ {16, 32, 128, 256} all recover full codebook use (Sec. 5.2), so the derived value is not what matters.
  • Why it matters: the paper's value is as a general lesson about the representation–tokenizer interface. Demonstrated on one generator, one tokenizer recipe and three small catalogs, and without matched comparisons against existing remedies, that lesson is not yet shown to generalize.
  1. The headline downstream gains are measured against a fully degenerate baseline, and away from collapse the effects are small and mixed.
  • Native raw maps the entire catalog to a single base SID with NMSE ≈ 1.0 (Table 1). Complete collapse of this kind would be caught by routine utilization monitoring, so the +0.021–0.044 Hit@10 improvements largely compare a working tokenizer against a broken one.
  • Where tokenizers are not collapsed, scale changes have small and inconsistent downstream effects: between RMS .008 and .064, Hit@10 ranges from −0.0018 to +0.0150 across nine representation–domain pairs, with non-monotonicity at intermediate scales (Table 2, App. F.2).
  • Why it matters: the practical significance beyond "avoid a broken tokenizer" is not demonstrated, which compounds the generality concern above.
Questions for authors
  1. More downstream families. Please evaluate calibration with at least one downstream generator that is not a TIGER-style T5 encoder–decoder: for example a decoder-only generative recommender, an HSTU/OneRec-style generator, or a retrieval-style consumer that uses SIDs as features. Do the Hit@10/NDCG@10 gains persist?
  2. Disentangle SID quality from generator structure. Can you hold the generator vocabulary and tied-embedding size fixed across arms (e.g. by equalizing effective vocabulary, or by re-indexing the collapsed arm), so that the downstream gain is attributable to the learned partitions rather than to the change in prediction structure?
  3. Matched comparison with existing remedies. Under the main matched protocol, how does calibration compare downstream with GRID-style whitening, catalog-wide normalization, and dead-code reset / k-means re-initialization?
  4. Dependence on the loss-reduction convention. R is mean-reduced over the 512 input dimensions while Q is averaged over latent coordinates (Eq. 3). Is the collapse largely a consequence of this convention? Would a dimension-matched or sum-reduced reconstruction loss avoid it without any calibration?
  5. Do the conclusions hold on substantially larger catalogs (≥10⁵–10⁶ items) and under a non-RQ-VAE quantizer with downstream evaluation?
5 comments

Response to Reviewer bKXe — Part 1/3: Downstream Architecture and Vocabulary Controls


We sincerely thank Reviewer bKXe for the careful assessment of our controlled counterexample, exact reparameterization, and optimization analysis, as well as for the thoughtful and constructive suggestions that have helped us strengthen our experimental evaluation and clarify the scope of our findings. In response, we conducted controlled experiments on downstream architecture, vocabulary and suffix effects, existing collapse remedies, loss reductions, and alternative quantizers. Unless stated otherwise, results use tokenizer and generator seed 42 on the same test split; RQ-KMeans is evaluated over seeds 42–44. Our responses follow below.


Q1 / Weakness 1(a): Does the downstream improvement extend beyond TIGER/T5?

Yes. With a decoder-only GPT-2-style recommender, calibration improves NDCG@10 on all three datasets and Hit@10 on Beauty and Instruments; the Hit@10 change on Games is negligible. The downstream effect is therefore not limited to TIGER's encoder–decoder architecture.

We evaluated a 12-layer decoder-only GPT-2-style generative recommender (hidden size 128, four attention heads, tied embeddings), trained from scratch. It autoregressively predicts the target item's five SID tokens from the history SIDs. Evaluation uses full-catalog trie-constrained beam search (beam size 20). We reuse the frozen SID catalogs for Native raw, Native calibrated (fR=64f_R=64), and Scale-8. Table R4 reports the resulting test accuracy.

Table R4. Decoder-only GPT-2 test results (Hit@10 / NDCG@10; seed 42).

DatasetNative rawCalibratedScale-8
Beauty.03923 / .02019.05920 / .03094.06188 / .03248
Instruments.08919 / .06571.10259 / .07610.10285 / .07640
Games.10163 / .06206.10174 / .06555.10569 / .06793

Calibration increases Hit@10 by .01997 on Beauty and .01340 on Instruments. On Games, Hit@10 is nearly unchanged (+.00011), while NDCG@10 rises by .00349. NDCG@10 improves on all three datasets, and Scale-8 exceeds Native raw on both metrics throughout. The size of the gain varies substantially by dataset and consumer, extending the empirical scope beyond the original generator.

Q2 / Weakness 1(a): Do the gains persist when vocabulary size, tied-embedding parameter count, and suffix identity are held fixed?

Yes. Calibration retains its advantage over Native raw when vocabulary size, tied-embedding parameter count, and the fifth-token item-ID mapping are held fixed.

We conducted a fixed-vocabulary and shared-suffix control. Within each dataset, Native raw, Calibrated, and Scale-8 use identical vocabulary sizes and generator parameter counts. We compare the original group-local suffix with a global-item-ID suffix, where the fifth SID token uses the same unique item-ID mapping across conditions. Table R5 reports the controlled model sizes, and Table R6 reports the global-suffix test results.

Table R5. Fixed generator vocabulary and parameter count within each dataset.

DatasetFixed vocabulary VVGenerator parameters
Beauty45,21010,246,528
Instruments43,56310,035,712
Games50,38010,908,288

Table R6. Fixed vocabulary and global-item-ID suffix: test Hit@10 / NDCG@10 (seed 42).

DatasetNative rawCalibratedScale-8
Beauty.04402 / .02309.06448 / .03501.06833 / .03632
Instruments.08229 / .06315.11373 / .08787.11610 / .08896
Games.07737 / .04998.11602 / .07848.11897 / .08055

With vocabulary capacity and suffix identity controlled, calibration still improves Hit@10 over Native raw by approximately .0205 / .0314 / .0387 and NDCG@10 by .0119 / .0247 / .0285 (Beauty / Instruments / Games). The gains persist under the matched vocabulary and global-suffix controls, and the group-local suffix control shows the same direction of improvement.

Response to Reviewer bKXe — Part 2/3: Existing Remedies and Loss Reductions


Q3 / Weakness 1(c): Matched comparison with existing collapse remedies

Both calibration and catalog normalization recover broad code use with comparable downstream accuracy. ZCA whitening also prevents complete collapse but performs less well downstream, while the tested dead-code reset does not recover code use.

We compared Native raw, Native calibrated, catalog-wide normalization, ZCA whitening, ZCA whitening rescaled to native RMS, dead-code reset, and Scale-8 starting from the same frozen SASRec-512 catalog, with condition-specific preprocessing where applicable, and a common RQ-VAE architecture, tokenizer training budget, and TIGER evaluation protocol (seed 42). Table R7 reports code utilization and downstream accuracy.

Table R7. Matched collapse remedies: MinPPL and test Hit@10 (Beauty / Instruments / Games; seed 42).

ConditionMinPPL (B / I / G)Hit@10 (B / I / G)
Native raw.004 / .004 / .004.04469 / .09646 / .06534
Dead-code reset.004 / .004 / .004.04469 / .09646 / .06534
ZCA whitening + native RMS.004 / .004 / .004.04469 / .09646 / .06534
ZCA whitening.933 / .748 / .816.05741 / .10409 / .10969
Catalog normalization.870 / .804 / .816.06560 / .11387 / .11897
Native calibrated.698 / .822 / .806.06453 / .11526 / .12050
Scale-8.922 / .892 / .870.06865 / .11267 / .12057

Rescaling whitened embeddings back to native RMS leads to complete collapse again, identifying numerical scale as an important factor in the whitening result. Under the tested reset rule and residual STE, dead-code reset achieves the same test accuracy as Native raw.

These comparisons show that different remedies affect the input distribution or training objective in different ways, and broad code utilization does not necessarily yield the highest recommendation accuracy. Calibration provides an objective-level intervention derived from the scale mapping while leaving the original item embeddings unchanged, making it particularly useful for isolating the optimization mechanism studied here.

Q4 / Weakness 1(b): Is collapse driven by the loss-reduction convention?

The loss-reduction convention is an important factor because it changes the effective balance between reconstruction and quantization. Both coordinate-sum variants avoid complete collapse on native inputs, consistent with our optimization analysis.

We directly compared coordinate-mean and coordinate-sum reductions. The original objective is

L=R+Q,\mathcal{L}=R+Q,

where RR averages over dx=512d_x=512 input coordinates and QQ over dz=32d_z=32 latent coordinates. The alternative reductions are

\text{Sum both:}\quad &512R+32Q=32(16R+Q),\\\\ \text{Sum reconstruction only:}\quad &512R+Q. \end{aligned}$$ The first has the same relative reconstruction-to-quantization weighting as $16R+Q$; the second corresponds to a relative reconstruction coefficient of 512. The common factor of 32 need not produce identical AdamW trajectories. The outcomes are compared in Table R8. **Table R8. Loss-reduction conventions: MinPPL and test Hit@10 (Beauty / Instruments / Games; seed 42).** | Objective | MinPPL (B / I / G) | Hit@10 (B / I / G) | |---|---|---| | $R+Q$ | .004 / .004 / .004 | .04469 / .09646 / .06534 | | $16R+Q$ | .821 / .811 / .855 | .06444 / .11051 / .12161 | | $512R+32Q$ | .854 / .775 / .807 | .06623 / .11300 / .11915 | | $64R+Q$ (calibrated) | .698 / .822 / .806 | .06453 / .11526 / .12050 | | $512R+Q$ | .851 / .861 / .838 | .06533 / .11347 / .11944 | Both coordinate-sum conventions also achieve downstream accuracy close to calibration. **The observed collapse is not an intrinsic inability to quantize native embeddings. This is the mechanism analyzed in the paper: at exactly mapped tokenizer states, input rescaling changes the reconstruction term and its latent gradient while leaving the corresponding quantization terms unchanged. The loss-reduction convention is another determinant of that balance.**

Response to Reviewer bKXe — Part 3/3: Quantizer Generality and Practical Significance


Q5 / Weakness 1(b): Non-RQ-VAE tokenizers and larger catalogs

Scale sensitivity depends on the quantizer. Residual-FSQ shows smaller, mixed downstream changes without complete collapse, whereas RQ-KMeans produces identical IDs under Native and Scale-8. Our analytical mapping is independent of catalog size, but larger-catalog outcomes remain untested.

Residual-FSQ downstream evaluation. We added full TIGER evaluation of frozen Residual-FSQ tokenizers on Native, Scale-8, and catalog-normalized inputs (seed 42). Unlike Native raw RQ-VAE, Residual-FSQ does not collapse at native scale: base-SID collision is 0.36%–0.41% across the three datasets. Table R9 summarizes its downstream results.

Table R9. Residual-FSQ test results (Hit@10 / NDCG@10; seed 42).

DatasetNativeScale-8Normalized
Beauty.05830 / .03160.05902 / .03222.06256 / .03366
Instruments.11201 / .08755.11048 / .08514.10993 / .08508
Games.12185 / .08223.11895 / .08056.12277 / .08267

Scale-8 gives a small gain on Beauty but decreases both metrics on Instruments and Games. These mixed directions distinguish fixed-grid quantization from the RQ-VAE collapse regime; the RQ-VAE-specific 64R+Q64R+Q calibration rule does not directly transfer to Residual-FSQ.

RQ-KMeans. As a non-neural, distance-based control, we compared Native and Scale-8 with four residual K-means levels and 256 codes per level, across three datasets and seeds 42–44. All nine paired runs have layer-wise assignment ARI = 1, full-SID agreement = 1, and identical downstream metrics, as detailed in Table R10.

Table R10. RQ-KMeans Native versus Scale-8 (test Hit@10 / NDCG@10; mean over seeds 42–44).

DatasetNative Hit@10 / NDCG@10Scale-8 Hit@10 / NDCG@10ΔHit@10 / ΔNDCG@10
Beauty.0633 / .0331.0633 / .0331.0000 / .0000
Instruments.1126 / .0868.1126 / .0868.0000 / .0000
Games.1222 / .0813.1222 / .0813.0000 / .0000

This agrees with the scale equivariance of Euclidean distances and K-means centroid updates under consistently transformed initialization. It also shows that native SASRec embeddings support useful discrete representations under another tokenizer. This contrast isolates tokenizer training mechanism as a key determinant of the observed non-invariance.

Catalog size. Our current empirical catalogs contain 10,439–17,256 items. Evaluation at 10510^5–10610^6 items remains to be completed. Proposition 1's exact reparameterization and the mapped-state loss/gradient identities in Section 4.2 do not depend on catalog cardinality, while collapse frequency and downstream impact at larger scales require separate empirical testing.

Weakness 2: Practical significance beyond the collapsed baseline

Our findings extend beyond recovering collapsed tokenizers: numerical scale affects downstream recommendation even without complete collapse, and our analysis identifies an optimization confound relevant to SID construction and evaluation.

We conducted an additional fixed-vocabulary evaluation using non-collapsed SASRec-32 SID catalogs at RMS 0.016 and 0.064 across three datasets. Within each dataset, generator vocabulary, parameter count, global-item-ID suffixes, and training protocols were matched.

Table R11. Non-collapsed SASRec-32 scale comparison with fixed vocabulary (test Hit@10 / NDCG@10; seed 42).

DatasetRMS 0.016RMS 0.064ΔHit@10ΔNDCG@10
Beauty.05682 / .03082.05866 / .03211+.00184+.00129
Instruments.11161 / .08554.11438 / .08821+.00277+.00267
Games.11699 / .07937.11783 / .08004+.00084+.00067

Both Hit@10 and NDCG@10 improve across all three datasets, demonstrating that scale-dependent downstream effects persist beyond complete collapse, with generator capacity and suffix identity controlled.

The practical significance extends beyond performance gains from rescaling. Our central finding is that representations preserving exactly the same upstream recommendation function can yield different learned SIDs and downstream outcomes under an identical tokenizer training recipe. Consequently, comparisons of item representations may conflate differences in recommendation information with scale-dependent tokenizer optimization.

Our exact reparameterization, mapped-state loss/gradient analysis, and matched interventions explain this mechanism and provide a principled basis for controlling numerical scale and objective weighting. This establishes both a mechanistic explanation and a methodological consideration for reliable SID construction and evaluation.


We appreciate the reviewer's specific suggestions, which helped clarify both the mechanism and its scope across tokenizers and downstream consumers.

Reviewer_RRVjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This work investigates the codebook collapse problem in Semantic-ID-based generative recommendation, where native input embeddings collapse into a single base SID. The authors propose rescaling the input embeddings while correspondingly transforming user-history representations to preserve recommendation equivalence, thereby achieving broader codebook utilization. The authors further conduct experiments showing that reconstruction-loss calibration improves Hit@10 and NDCG@10 in all nine paired runs.

Critical strengths

1.The observed phenomenon and the analysis of its underlying cause are insightful.

2.The experiments are sufficient and easy to follow.

Critical weaknesses

1.The abstract is difficult to understand and could be written more clearly.

2.The core methodological novelty is relatively limited, as the proposed approach mainly relies on reparameterization and reconstruction-loss reweighting.

Questions for authors

In Appendix E.2, the author show that several reconstruction-loss weights (16, 32, 128, and 256) can recover broad codebook utilization. However, the experiment does not include downstream recommendation results or a direct comparison with the proposed coefficient of 64 under the same initialization. Could the author provide a more comprehensive comparison using Hit@10 and NDCG@10? Without this comparison, it is unclear whether the proposed calibration provide a specific advantage over conventional reconstruction-loss reweighting.

5 comments

Response to Reviewer RRVj — Part 1/2

We sincerely thank the reviewer RRVj for recognizing the significance of our findings and the analysis of the underlying optimization mechanism, as well as for the thoughtful and constructive suggestions. In response, we expanded the matched reconstruction-weight sweep to include downstream results for all coefficients, clarified the analytical role of 64, and improved the manuscript presentation. Our responses follow below.


1. Reconstruction-Weight Sweep and Downstream Performance (Question for Authors)

To directly address the requested reconstruction-weight comparison, we expanded our original six-weight study to 9 reconstruction weights × 3 datasets × 3 random seeds (42–44), totaling 81 complete tokenizer–generator experiments.

We evaluate fR∈1,2,4,8,16,32,64,128,256f_R \in \\{1,2,4,8,16,32,64,128,256\\}. The additional coefficients 2, 4, and 8 allow us to examine the transition from codebook collapse to recovery in greater detail.

Within each dataset–seed block, all coefficients share the same completed K-means initialization, random state, and minibatch sequence, with identical downstream training and evaluation protocols. The tokenizer diagnostics and test results are reported in Tables R1 and R2, respectively.

Table R1. MinPPL across reconstruction weights (mean over seeds 42–44).

fRf_RBeautyInstrumentsGames
1.0039.0039.0039
2.0039.0039.0039
4.0039.2328.0039
8.6588.7744.5013
16.7600.8026.8337
32.8786.8184.8083
64.7396.8483.8105
128.8619.8202.8011
256.8368.8477.8169

Table R2. Test Hit@10 / NDCG@10 across reconstruction weights (mean over seeds 42–44).

fRf_RBeautyInstrumentsGames
1.0425 / .0223.0921 / .0713.0745 / .0489
2.0425 / .0223.0921 / .0713.0745 / .0489
4.0425 / .0223.1006 / .0773.0745 / .0489
8.0635 / .0342.1116 / .0852.1018 / .0682
16.0643 / .0344.1127 / .0865.1200 / .0811
32.0653 / .0353.1113 / .0851.1189 / .0800
64.0650 / .0347.1113 / .0848.1216 / .0818
128.0659 / .0352.1114 / .0853.1211 / .0815
256.0671 / .0363.1130 / .0862.1217 / .0819

Tables R1 and R2 show two main patterns.

(1) A clear transitional interval from complete collapse to stable recovery.

For fR∈1,2f_R \in \\{1,2\\}, all nine runs across the three datasets exhibit complete collapse (MinPPL = 0.0039). As the reconstruction weight increases, codebook utilization begins to recover:

  • fR=4f_R=4: Only one seed on Instruments recovers, while the remaining eight runs still exhibit complete collapse.
  • fR=8f_R=8: Eight of the nine runs recover, with only one seed on Games remaining fully collapsed.
  • fR∈[16,256]f_R \in [16,256]: All 45 runs avoid complete collapse and recover broad codebook utilization.

Within this stable recovery range, mean MinPPL ranges from 0.7396 to 0.8786 on Beauty, 0.8026 to 0.8483 on Instruments, and 0.8011 to 0.8337 on Games.

These results demonstrate that reconstruction weighting substantially affects tokenizer optimization, and that the transition from collapse to recovery is not determined by a single unique coefficient.

(2) Downstream performance becomes relatively stable after recovery.

For fR≥16f_R \geq 16, both Hit@10 and NDCG@10 are substantially higher than those of the fully collapsed Native raw baseline across all three datasets.

When selecting the reconstruction weight based on validation NDCG@10, the selected coefficients are 256, 128, and 256 for Beauty, Instruments, and Games, respectively. Compared with the fixed coefficient fR=64f_R=64, these validation-selected weights yield the following improvements on the test set:

  • Beauty: Hit@10 +0.00212, NDCG@10 +0.00159.
  • Instruments: Hit@10 +0.00010, NDCG@10 +0.00055.
  • Games: Hit@10 +0.00010, NDCG@10 +0.00004.

The reference-matched coefficient 64 lies within the stable recovery region and achieves test performance close to validation-selected weights, without downstream reconstruction-weight search.

Response to Reviewer RRVj — Part 2/2


2. Why 64? How Does This Differ from Conventional Loss Reweighting?

The central contribution of our work is not simply adjusting the reconstruction-loss weight. Rather, we investigate why representations that are equivalent with respect to the upstream recommendation function can lead to different tokenizer optimization outcomes. We provide a mechanistic explanation, validate it experimentally, and derive a corresponding loss-calibration rule from this analysis.

First, in Section 4.1 (Proposition 1), we construct an exact tokenizer reparameterization between different input scales. By transforming the first encoder layer and the final decoder layer while keeping the remaining parameters and codebooks unchanged, the tokenizers at both scales can produce identical latent representations and code assignments. This establishes that the native-scale tokenizer has sufficient representational capacity to express the non-collapsed SIDs. The observed collapse therefore cannot be attributed simply to insufficient model capacity.

Next, in Section 4.2, we show that at these exactly mapped tokenizer states, the reconstruction and quantization terms satisfy

Rt=γ2Rs,Qt=Qs,R_t=\gamma^2 R_s,\qquad Q_t=Q_s,

where γ=ct/cs\gamma=c_t/c_s.

To match the reference-scale objective balance and latent-gradient contributions at the mapped states, we can therefore choose

=\left(\frac{c_s}{c_t}\right)^2 f_s.$$ For the reference scale $c_s=8$, target scale $c_t=1$, and reference weight $f_s=1$, this directly gives $$f_t=64.$$ **Therefore, 64 is an analytically derived objective-matching coefficient for a specified reference scale, rather than an optimal hyperparameter selected through downstream performance tuning.** The expanded 81-run reconstruction-weight experiment further supports this analysis. As the coefficient increases from 1 to 16, the tokenizer transitions from complete collapse to broad codebook recovery. Across the range 16–256, all 45 runs avoid complete collapse and maintain relatively stable downstream performance. The analytically derived coefficient 64 lies within this effective range and achieves strong performance without requiring a downstream grid search over reconstruction weights. **The main contribution of our work is establishing a coherent chain of evidence, from a controlled counterexample with recommendation-equivalent representations, through exact reparameterization and loss/gradient analysis, to matched experimental interventions.** This demonstrates that numerical scale and tokenizer optimization can act as important confounding factors when comparing Semantic IDs constructed from different upstream representations, and should therefore be controlled in experimental design and interpretation. ## **3. Novelty and Contribution of the Paper (Weakness 2)** We have further clarified the positioning of our contributions. Our central question is: **Why can a downstream tokenizer learn substantially different discrete representations when the upstream recommendation function remains unchanged?** This distinction has practical implications. When comparing Semantic IDs constructed from different embedding sources, observed performance differences may arise from the information encoded in the representations, their numerical scales, or the optimization behavior of the tokenizer. Without controlling for the latter two factors, it becomes difficult to determine whether performance differences reflect representation quality or the tokenization process itself. Numerical scale and the relative weighting of tokenizer objectives should therefore be treated as explicit control factors in the design and evaluation of Semantic-ID systems. Through a controlled counterexample, exact reparameterization, optimization analysis, and matched experiments, our work separates these factors. This provides a basis for more reliable comparisons of representation methods, a better understanding of tokenizer training behavior, and more systematic diagnosis of optimization issues in discrete representation learning. ## **4. Writing and Presentation (Weakness 1)** We have rewritten the Abstract and streamlined the Introduction and the overall narrative of the main sections to improve clarity and readability. We thank the reviewer again for the constructive feedback.

Dear Reviewer RRVj,

I hope this message finds you well. As the discussion period draws to a close, we would like to check whether our responses and additional experiments have adequately addressed your concerns. If you have any further questions or feedback, please feel free to let us know. We would be happy to provide additional clarification or address any remaining points.

Thank you again for your time and thoughtful consideration of our work.

Reviewer_crYFOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies that in the item tokenization process (item representations -> semantic IDs) of semantic ID-based generative recommendation, how the numerical scale of the input item representations could affect the training of an RQ-VAE like item tokenizer. The finding is really interesting that even the input item representations may preserve identical dot-wise recommendation performance, the numerical scale itself affects the item tokenizer (e.g., whether collapse happens) and the downstream generative recommendation performance. Extensive experiments across public datasets, and different item representations, are conducted to convey this insight.

Critical strengths
  1. The finding itself is really interesting. TL;DR is even the input item representations may preserve identical dot-wise recommendation performance, the numerical scale itself affects the item tokenizer (e.g., whether collapse happens) and the downstream generative recommendation performance. This finding may encourage people pay more attention on representation numerical scale, of which people may not care much before, to further improve their generative recommendation model.
  2. To verify finding, the authors conducted several sets of controllable experiments across multiple public datasets and item representations.
Critical weaknesses
  1. The findings may be restricted to certain tokenization methods (or even certain implementations). The study mostly focuses on RQ-VAE-like item tokenizer. Specifically, the numerical scale -> codebook collapse issue mostly stems from the optimization implementation (e.g., as analyzed in Section 2.2, on residual STE). This raises a natural question if people use other tokenizers would the findings still hold? The authors explored a bit on FSQ in the appendix. A discussion on some widely used other item tokenization methods would make the paper more comprehensive.
  2. Writing can be improved.
    1. It would be better to include a small paragraph at the beginning of each section to briefly summarize the motivation/intuition of conducting the experiments that will be introduced in this section. The current version directly introduces experimental settings, making it a bit difficult to follow for the first-time read.
    2. It's a good convention to not list numbers that are non-comparable at the same column in the tables. For example, in Table 2, ranking metrics on different datasets are not numerically comparable, but they are all in the same column.
  3. Code is not available during the reviewing phase.
Questions for authors

Does the finding still hold for other popular item tokenization methods like RQ KMeans?

4 comments

We sincerely thank the reviewer crYF for your careful reading, encouraging assessment, and constructive suggestions. In response, we evaluated RQ-KMeans across three datasets and seeds, improved the manuscript organization and tables, and prepared an anonymous code repository. Our responses follow below.

1. RQ-KMeans Scale Sensitivity (Weakness 1 / Question for Authors)

We conducted additional experiments with RQ-KMeans. The results show that under controlled initialization and identical optimization protocols, RQ-KMeans is invariant to the centered scaling considered in our study, providing a clear contrast to the behavior observed with RQ-VAE.

Experimental Setup

We use four-level Residual K-Means with 256 centroids per level and 10 restarts per level. Experiments are conducted on Beauty, Instruments, and Games, with three random seeds (42–44) per dataset. We apply RQ-KMeans to both Native and Scale-8 SASRec-512 embeddings.

Experimental Results

Across all nine paired runs:

  • SID assignments agree at every level and for complete SIDs (100% agreement between scales).
  • MinPPL, base collision, and downstream Hit@10/NDCG@10 are unchanged across scales; reconstruction NMSE differs only at approximately 10−1110^{-11}.

Table R3 compares Native and Scale-8 RQ-KMeans results directly, alongside the corresponding RQ-VAE outcomes (mean test Hit@10 / NDCG@10 over seeds 42–44).

Table R3. RQ-KMeans scale comparison and RQ-VAE reference results (test Hit@10 / NDCG@10; seeds 42–44).

DatasetRQ-KMeans NativeRQ-KMeans Scale-8RQ-VAE Native rawRQ-VAE calibrated
Beauty.0633 / .0331.0633 / .0331.0452 / .0230.0666 / .0358
Instruments.1126 / .0868.1126 / .0868.0898 / .0687.1130 / .0865
Games.1222 / .0813.1222 / .0813.0767 / .0500.1202 / .0814

Analysis

These results are consistent with the scale equivariance of RQ-KMeans. For Euclidean-distance-based Residual K-Means, uniformly scaling the inputs and centroids preserves nearest-centroid assignments, allowing the residual assignments to remain unchanged. In contrast, tokenizers such as RQ-VAE jointly learn an encoder, decoder, and quantization objective, where reconstruction and latent quantization terms transform differently under input scaling.

Our central finding is therefore not that all tokenizers respond similarly to input scaling, but that equivalence of the upstream recommendation function does not necessarily imply equivalent optimization outcomes for learned tokenizers. RQ-KMeans provides an informative control, showing that the pronounced differences observed with RQ-VAE depend on its specific optimization mechanism.

Importantly, these results also reinforce our conclusion that the Native embeddings do contain useful item-discriminative information for recommendation. Their collapse under the studied RQ-VAE training procedure is not evidence that this information is insufficient.

2. Writing and Presentation

We appreciate the reviewer's suggestions regarding the clarity and organization of the manuscript. We revised the presentation to make the research question and experimental rationale easier to follow.

First, we rewrote the Abstract and Introduction to present the central research question more directly: Why can input representations that preserve exactly the same recommendation preferences still produce substantially different Semantic IDs?

Second, we added motivating explanations at the beginning of the main sections and experimental subsections, allowing the experiments to follow naturally from the research questions. The revised narrative follows a clearer progression:

Controlled counterexample → Capacity versus optimization → Loss and gradient analysis → Controlled intervention → Scope and limitations.

For Table 2, we reorganized the previously mixed cross-dataset metrics into dataset-specific columns, making the comparisons more intuitive and easier to follow.

3. Code Availability

We have prepared the key code for this work in an anonymous repository, allowing reviewers to inspect the implementation and reproduction configurations:

https://anonymous.4open.science/status/ICLR27-19911

We thank the reviewer again for the constructive feedback.

Dear Reviewer crYF,

I hope this message finds you well. As the discussion period draws to a close, we would like to check whether our responses and additional experiments have adequately addressed your concerns. If you have any further questions or feedback, please feel free to let us know. We would be happy to provide additional clarification or address any remaining points.

Thank you again for your time and thoughtful consideration of our work.

Thank you to the authors for taking time in addressing my concerns. I'll keep my positive rating.

Dear Reviewer crYF,

Thank you for your positive assessment and continued support. We sincerely appreciate your thoughtful feedback. The RQ-KMeans experiments you suggested provide an informative scale-equivariant contrast to RQ-VAE, further clarifying the underlying optimization mechanism and its scope. Your suggestions have also helped us improve the clarity, completeness, and presentation of our work.

We deeply appreciate your time, expertise, and supportive evaluation throughout the review process.

Comments on the paper

2 comments

We sincerely thank all reviewers and the Area Chairs for their time, careful evaluations, and constructive feedback. Their comments have helped us strengthen the experimental evidence, clarify our contributions, and improve the presentation of the paper.

In response, we have revised the manuscript and conducted additional controlled experiments. These results further support our findings while clarifying the scope and practical implications.

The key revisions and additions are highlighted in blue in the updated PDF for ease of reference. Detailed responses to individual comments are provided in the corresponding reviewer discussions.

We appreciate the opportunity to address these questions and hope the revised manuscript and additional evidence help inform the final assessment.

AuthorsOpenReview ↗

General Response

We sincerely thank all reviewers and the Area Chairs for their careful evaluations and constructive feedback. We appreciate the recognition of our findings and optimization analysis. We have carefully addressed the reviewers' questions through additional controlled experiments, theoretical clarifications, and substantial improvements to the manuscript.

Below, we summarize the core contributions of our work, the additional experimental evidence, and the manuscript revisions made during the rebuttal.

1. Core Contributions of Our Work

1. Optimization Non-Invariance under Recommendation Equivalence. We establish that representations preserving exactly the same upstream recommendation function can produce substantially different learned Semantic IDs and downstream recommendation outcomes under an identical tokenizer training procedure. This identifies numerical scale as an important factor at the representation–tokenizer interface.

2. Mechanistic Explanation and Principled Calibration. Through exact tokenizer reparameterization (Proposition 1), we demonstrate that the non-collapsed solution remains representable at native scale, distinguishing optimization behavior from architectural capacity. Our loss and gradient analysis (Section 4.2) identifies how reconstruction and quantization objectives respond differently to input scaling, motivating an analytically derived calibration rule validated through matched interventions.

3. Implications for Reliable SID Construction and Evaluation. Our findings establish that differences in downstream performance can arise from tokenizer optimization even when upstream recommendation information is preserved. This provides a mechanistic and methodological basis for more reliable comparisons of item representations and Semantic-ID tokenizers.

2. Additional Experimental Results During Rebuttal

We conducted several targeted experiments in response to the reviewers' main concerns:

  • Reconstruction-weight analysis (RRVj): Expanded the weight sweep to 81 matched tokenizer–generator experiments (9 weights × 3 datasets × 3 seeds), including downstream Hit@10 and NDCG@10 for every coefficient. The results validate the importance of objective balance and clarify the analytical significance of the calibration coefficient. (Tables R1–R2).

  • Downstream generality and SID attribution (bKXe): Evaluated a decoder-only generator (Table R4), introduced fixed-vocabulary and global-suffix controls (Tables R5–R6), and conducted a frequency-matched shuffled-SID comparison (Table R12). These experiments strengthen the evidence for downstream effects across generator configurations and the predictive value of learned SID assignments.

  • Alternative tokenizers and optimization controls (crYF, bKXe): Evaluated RQ-KMeans (Tables R3, R10), Residual-FSQ (Table R9), existing collapse remedies (Table R7), and alternative loss-reduction conventions (Table R8). These comparisons clarify the mechanism underlying RQ-VAE scale sensitivity and its dependence on tokenizer design and optimization.

  • Practical significance beyond collapse (bKXe): Added fixed-vocabulary comparisons across three non-collapsed SASRec-32 settings (Table R11), demonstrating scale-dependent downstream differences without complete codebook collapse.

3. Manuscript Revisions and Reproducibility

In response to crYF and RRVj, we rewrote the Abstract, improved the Introduction, added motivating explanations to the main sections, and reorganized tables for clearer comparisons. We also provided an anonymous code repository to support reproducibility.

The manuscript revisions are highlighted in blue, and detailed results and explanations are available in the corresponding reviewer responses.

4. Summary

Together, the theoretical analysis and expanded experimental evidence reinforce our central conclusion: Recommendation equivalence at the upstream teacher level does not guarantee equivalent Semantic-ID tokenizer optimization.

Our work identifies and explains this optimization non-invariance, validates a principled intervention, and establishes numerical scale and tokenizer objective balance as important considerations for reliable SID construction and evaluation.

We thank the reviewers again for their valuable suggestions and hope these additional results and revisions will be considered in the final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.