acceptodds
Under review as a conference paper at ICLR 2027

Shortcut or Semantics? Understanding Conditioning Interfaces for Steerable Generative Retrieval

Abstract

Natural-language steering enables recommender systems to adapt to explicit user requests that go beyond historical interaction data. In generative retrieval, this requires conditioning a semantic ID (SID) decoder on textual preferences, but how best to interface language representations with the SID decoder remains unresolved. To address this, we propose a decoupled dual-branch interface that injects a frozen language model's final-layer representations via pooled global modulation and token-level cross-attention. We evaluate this design against end-to-end joint conditioning across three e-commerce domains. Joint conditioning reaches the highest retrieval accuracy, but suffers from catastrophic forgetting of general reasoning; meanwhile, the dual-branch interface trails by only 0.34 to 0.79 percentage points while preserving pretrained capabilities. Across meaning-preserving rephrasings, joint conditioning degrades sharply under synonym substitution, revealing that the model memorizes literal word matches instead of learning actual semantic intent. The dual-branch interface, in contrast, remains anchored in the frozen model's semantic space, exhibiting 55-74% lower degradation under lexical substitution across the three domains. These results establish our proposed dual-branch interface as a robust, semantically grounded foundation for steerable generative retrieval.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
generative retrieval, semantic IDs, recommender systems, natural language steering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ghQjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates how natural-language preferences should condition semantic-ID generation for steerable recommendation. It proposes a decoupled dual-branch interface that connects a frozen language tower to a trainable SID generator through pooled global modulation and token-level cross-attention, using the language tower’s final-layer representations. Under a shared Gemma 3 backbone, the paper compares this interface with end-to-end joint conditioning, cross-attention-only conditioning, and representation-depth controls. Experiments on three Amazon domains evaluate preference recommendation and find/avoid steering, alongside auxiliary steering tasks, meaning-preserving rephrasings, and general-language capability tests. Joint conditioning achieves the highest original-test retrieval accuracy, while the proposed interface trails by 0.34–0.79 percentage points in preference Recall@10 and improves over cross-attention alone. The paper reports smaller degradation under synonym substitution and preservation of the frozen language tower’s pretrained capabilities, whereas joint fine-tuning substantially degrades the tested reasoning capabilities. Component and depth ablations further examine the roles of global modulation, cross-attention, and final-layer conditioning.

Critical strengths
  1. The paper evaluates lexical robustness and capability retention in addition to retrieval accuracy, revealing meaningful failure modes of joint conditioning.

  2. The dual-branch design is technically coherent, and the ablations support the complementary roles of global modulation and token-level cross-attention.

Critical weaknesses
  1. The benefit of using a causal LLM as the frozen text encoder remains unclear. Comparisons with a bidirectional encoder of comparable size and smaller or distilled alternatives would help establish whether the additional model capacity and serving cost are justified.

  2. Freezing the language tower preserves its pretrained capabilities but prevents retrieval supervision from adapting its representations. Could this constraint limit the conditioning interface's flexibility when handling domain-specific terminology or new preference types?

  3. It is unclear whether the robustness gains come from freezing the language tower or from the proposed dual-branch interface. Please evaluate the frozen cross-attention-only baseline under the same rephrasing tests to isolate the additional benefit of global modulation.

Questions for authors
  1. Rephrasing sensitivity and a single attention example do not establish that joint conditioning relies on keyword memorization rather than semantic understanding. Could the authors support this interpretation with controlled keyword interventions and analysis across more examples, or moderate the claims accordingly?

  2. Will more comparisons with public datasets be provided to validate the generalizability?

6 comments
AuthorsOpenReview ↗

Reply to Reviewer ghQj

We thank ghQj for their feedback!

Some naming / terminology clarification in our replies below: Dual-Branch Injection corresponds to Final-State Injection in the initial submission. Manuscript references use revised numbering unless marked “submitted” (which means the original submitted version); page and line references point to the revised, highlighted PDF.

CW1. Why use a causal LLM rather than a bidirectional or smaller encoder?

Our goal is retrieval as an add-on to a resident causal LM: its hidden states condition a trainable SID generator without updating the host. Reusing the host avoids loading separate text-encoder weights into accelerator HBM in this deployment. The SID generator adds its own memory and compute costs; we make no measured latency claim. The deployment motivation is in §1 (p. 1, lines 51–53; highlighted).

Using the same pretrained Gemma-3-1B-IT checkpoint, tokenizer, and hidden dimension controls the language backbone across interfaces. A bidirectional or smaller encoder comparison would address standalone retriever efficiency, which we have not evaluated. The backbone control is described in §§1 and 4, and the encoder-comparison scope in §6.

The revised manuscript clarifies this motivation in §1 and corrects the initialization description in §4.2 (p. 6, lines 279–281 and 302–303; highlighted), Figure 2, and Appendix C.1: only the language tower loads pretrained weights; the SID generator uses the Gemma-3-1B architecture with random initialization.

CW2. Does freezing limit adaptation to domain-specific terminology or new preference types?

Retrieval adaptation occurs in the trainable modulation projections, cross-attention, and SID generator, while the host representations remain fixed. This supports new request formats already representable by the host. The trainable pathways are specified in §4.2 (p. 6, lines 286–304) and Appendix C.1.

The submitted Intent Following task presents up to five candidate preferences with distractors, a format absent from training. Dual-Branch vs. Full-FT test R@10 is 5.55% vs. 6.38% on Beauty, 2.72% vs. 2.84% on Sports, and 4.84% vs. 4.60% on Toys (Train = 0; validation Intent Following is used in model selection; §§3.2 and 5.1, Appendices D.2 and E.4, Table 11). Under syntactic and conversational rewrites, Dual-Branch exceeds LoRA in all domains and Full-FT on Sports and Toys (Table G1).

The new Steam experiments extend the domain evaluation to games (Q2), and attribute flips test responses to changed requests without further training (Q1). Unfamiliar terminology still depends on the host's representations; these experiments do not test learning new concepts or zero-shot domain transfer. This scope is stated in §6 (p. 10, lines 513–523; highlighted).

CW3. Does rephrasing robustness come from freezing or from the dual-branch interface?

The requested comparison shows that lexical robustness is shared by both frozen interfaces, while the dual-branch design adds retrieval and steering gains. With the same frozen backbone and generator architecture, Dual-Branch improves test Preference R@10 by 0.15–1.24 pp and Sentiment@10 by 0.81–1.71 pp across all four datasets (Tables G1 and G3), and improves attribute selectivity across Amazon (Table G2). These gains go beyond keeping the language tower frozen.

Table G1. Original-request retrieval and paired rephrasing on Amazon (%, three-seed mean ±\pm sample SD). Test columns use held-out original requests; the four validation columns compare the original wording with lexical, syntactic, and conversational rewrites. Bold marks the highest mean within each domain and metric. Manuscript: §5.1, Table 2 (p. 7, lines 333–348), §5.4, and Appendix E.6, Table 13 (p. 30, lines 1592–1605).

Table G1(a). Beauty

InterfaceTest Preference R@10 ↑\uparrowTest Sentiment@10 ↑\uparrowVal Original R@10 ↑\uparrowVal Lexical R@10 ↑\uparrowVal Syntactic R@10 ↑\uparrowVal Conversational R@10 ↑\uparrow
Joint Conditioning (Full-FT)9.43±0.09\mathbf{9.43 \pm 0.09}7.01±0.16\mathbf{7.01 \pm 0.16}11.60±0.10\mathbf{11.60 \pm 0.10}7.62±0.097.62 \pm 0.0911.09±0.09\mathbf{11.09\pm0.09}11.51±0.06\mathbf{11.51\pm0.06}
Joint Conditioning (LoRA)8.22±0.118.22 \pm 0.115.72±0.315.72 \pm 0.3110.23±0.2210.23 \pm 0.225.47±0.295.47 \pm 0.298.57±0.078.57 \pm 0.078.44±0.408.44 \pm 0.40
Frozen Cross-Attention-only8.34±0.078.34 \pm 0.074.63±0.274.63 \pm 0.2710.89±0.2310.89 \pm 0.239.94±0.02\mathbf{9.94 \pm 0.02}10.75±0.1510.75\pm0.1510.75±0.1410.75\pm0.14
Dual-Branch Injection (ours)8.73±0.188.73 \pm 0.185.59±0.125.59 \pm 0.1211.21±0.1111.21 \pm 0.119.49±0.309.49 \pm 0.3011.08±0.1411.08\pm0.1411.18±0.0511.18\pm0.05

Table G1(b). Sports

InterfaceTest Preference R@10 ↑\uparrowTest Sentiment@10 ↑\uparrowVal Original R@10 ↑\uparrowVal Lexical R@10 ↑\uparrowVal Syntactic R@10 ↑\uparrowVal Conversational R@10 ↑\uparrow
Joint Conditioning (Full-FT)5.40±0.08\mathbf{5.40 \pm 0.08}6.15±0.27\mathbf{6.15 \pm 0.27}6.53±0.07\mathbf{6.53 \pm 0.07}5.61±0.075.61 \pm 0.076.29±0.116.29\pm0.116.27±0.106.27\pm0.10
Joint Conditioning (LoRA)4.52±0.014.52 \pm 0.015.45±0.305.45 \pm 0.305.88±0.135.88 \pm 0.134.71±0.254.71 \pm 0.255.34±0.165.34 \pm 0.165.15±0.275.15 \pm 0.27
Frozen Cross-Attention-only4.91±0.024.91 \pm 0.024.45±0.154.45 \pm 0.156.24±0.146.24 \pm 0.146.03±0.166.03 \pm 0.166.18±0.166.18\pm0.166.13±0.156.13\pm0.15
Dual-Branch Injection (ours)5.06±0.165.06 \pm 0.165.65±0.065.65 \pm 0.066.45±0.066.45 \pm 0.066.04±0.10\mathbf{6.04 \pm 0.10}6.39±0.05\mathbf{6.39\pm0.05}6.40±0.11\mathbf{6.40\pm0.11}

Table G1(c). Toys

InterfaceTest Preference R@10 ↑\uparrowTest Sentiment@10 ↑\uparrowVal Original R@10 ↑\uparrowVal Lexical R@10 ↑\uparrowVal Syntactic R@10 ↑\uparrowVal Conversational R@10 ↑\uparrow
Joint Conditioning (Full-FT)7.79±0.37\mathbf{7.79 \pm 0.37}5.33±0.18\mathbf{5.33 \pm 0.18}9.50±0.37\mathbf{9.50 \pm 0.37}8.02±0.508.02 \pm 0.508.72±0.428.72\pm0.428.51±0.358.51\pm0.35
Joint Conditioning (LoRA)5.92±0.285.92 \pm 0.283.46±0.183.46 \pm 0.188.07±0.298.07 \pm 0.296.47±0.316.47 \pm 0.317.05±0.247.05 \pm 0.246.86±0.156.86 \pm 0.15
Frozen Cross-Attention-only6.34±0.146.34 \pm 0.143.13±0.093.13 \pm 0.098.75±0.118.75 \pm 0.118.54±0.078.54 \pm 0.078.74±0.128.74\pm0.128.75±0.138.75\pm0.13
Dual-Branch Injection (ours)7.00±0.197.00 \pm 0.193.94±0.103.94 \pm 0.109.02±0.089.02 \pm 0.088.65±0.05\mathbf{8.65 \pm 0.05}8.89±0.09\mathbf{8.89\pm0.09}8.89±0.07\mathbf{8.89\pm0.07}

For lexical rephrasing, cross-attention-only is higher on Beauty, the frozen interfaces are close on Sports and Toys, and Dual-Branch is higher on Steam. Dual-Branch also has higher syntactic and conversational R@10 than cross-attention-only in all four datasets. Configurations are independently validation-selected; each rephrasing pair holds history and target fixed. Figure 3 now includes all four interfaces (§5.4, pp. 9–10, lines 444–496; highlighted; Appendix E.6, Table 13).

The LoRA comparison uses 375 configurations per Amazon domain: ranks {4, 8, 16, 32, 64}, learning rates {0.0005, 0.001, 0.005, 0.01, 0.02}, batch sizes {8, 12, 16, 32, 48}, and training steps {30k, 50k, 70k}, followed by three-seed retraining. Dual-Branch exceeds selected LoRA in original test Preference R@10 and all nine Amazon rephrasing comparisons. Sweep and selection details are in Appendices D.1 (p. 22, lines 1163–1170; highlighted) and D.2 and Table 8 (p. 23).

Q1. Do keyword interventions and additional examples support the semantics interpretation?

We have narrowed the keyword-memorization claim to observed lexical sensitivity and added a 2,612-pair attribute-flip evaluation. This directly tests whether changing the requested attribute changes retrieval while history and most wording remain fixed.

The evaluation contains 2,612 request pairs across Beauty, Sports, and Toys, tested on existing checkpoints without further training. For a concrete example, the same Toys history is paired with “Select a Melissa & Doug sound puzzle that depicts modes of transportation” and “Select a Melissa & Doug sound puzzle that depicts melodic orchestral tools.” Let MAM_A contain the catalog's vehicle sound puzzles and MBM_B its musical-instrument sound puzzles. For the first request, MAM_A is the desired group and MBM_B the opposing group; for the second, these roles reverse. The groups are disjoint. The attribute phrases avoid the original attribute-value words and word stems of at least four characters, while retaining shared brand and product context.

For the two top-10 prediction sets TAT_A and TBT_B, selectivity measures the desired-group hit rate minus the opposing-group hit rate, averaged over the two request directions:

\begin{aligned} &\underbrace{\Pr(T_A \cap M_A \neq \varnothing) - \Pr(T_A \cap M_B \neq \varnothing)}_{\text{Request A}}\\\\[0.5em] &+ \underbrace{\Pr(T_B \cap M_B \neq \varnothing) - \Pr(T_B \cap M_A \neq \varnothing)}_{\text{Request B}} \end{aligned} \right].$$ Pr is the empirical fraction of paired records; selectivity is reported in percentage points. Identical predictions for the two requests give zero selectivity. **Table G2. Attribute selectivity on paired paraphrased-attribute requests (percentage points; higher is better).** Domain scores are three-seed mean $\pm$ sample SD. The final column is the equal-weight mean across domains. Bold marks the highest mean in each column. Manuscript: §5.1 (p. 8, lines 386–389; highlighted) and Appendix E.7 (pp. 31–32, lines 1652–1703; highlighted), Table 16. | Interface | Beauty $100\Delta \uparrow$ | Sports $100\Delta \uparrow$ | Toys $100\Delta \uparrow$ | Mean across domains: $100\Delta$ (pp) $\uparrow$ | | --- | ---: | ---: | ---: | ---: | | Joint Conditioning (Full-FT) | **$\mathbf{9.99 \pm 0.70}$** | **$\mathbf{5.59 \pm 0.38}$** | **$\mathbf{8.00 \pm 2.64}$** | **7.86** | | Joint Conditioning (LoRA) | $4.86 \pm 0.59$ | $1.09 \pm 0.78$ | $3.01 \pm 0.67$ | 2.99 | | Frozen Global-only | $-0.05 \pm 0.27$ | $0.00 \pm 0.28$ | $0.17 \pm 0.42$ | 0.04 | | Frozen Cross-Attention-only | $1.42 \pm 0.19$ | $1.03 \pm 0.53$ | $1.94 \pm 0.50$ | 1.46 | | Dual-Branch Injection (ours) | $3.80 \pm 2.20$ | $2.68 \pm 1.14$ | $7.27 \pm 0.71$ | 4.58 | Dual-Branch exceeds both frozen single-branch interfaces in every domain, and LoRA on Sports, Toys, and the domain average. LoRA is higher on Beauty in selectivity and Sentiment@10. Full-FT has the highest selectivity despite lower lexical robustness, supporting separate evaluation of these behaviors. Selectivity covers the labeled catalog groups. The evaluation is in §5.1 and Appendix E.7; the abstract (p. 1, lines 22–32; highlighted) and §§1, 5.4, and 6 state the narrower conclusions. Appendix E.1 (p. 24, lines 1270–1272; highlighted) and Figure 5 treat the attention example as qualitative, not proof of keyword-only behavior.
Reviewer_y96oOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates conditioning interfaces for steerable generative retrieval and proposes a dual-branch architecture combining global modulation and token-level cross-attention with a frozen language model. Experiments across three Amazon domains show competitive retrieval performance, improved robustness to preference rephrasing, and preservation of pretrained language capabilities compared with joint conditioning.

Critical strengths
  1. The proposed dual-branch architecture combines global feature modulation with token-level cross-attention to enable effective interaction between language representations and the SID decoder.

2.The experimental evaluation is comprehensive, covering three recommendation domains, with ablation studies, preference rephrasing robustness tests, and general language capability assessments supporting the effectiveness of the proposed method.

Critical weaknesses
  1. The proposed dual-branch architecture mainly combines existing global feature modulation and cross-attention mechanisms.

  2. The comparison between joint training and a frozen language encoder does not adequately control for differences in training strategies, making it difficult to determine whether catastrophic forgetting is caused by the conditioning interface itself.

  3. The preference rephrasing robustness experiments and single-case attention analysis are insufficient to demonstrate that the model genuinely learns semantic information rather than relying on surface lexical features.

Questions for authors

please refer to Weaknesses

4 comments
AuthorsOpenReview ↗

Reply to Reviewer y96o

We thank y96o for their feedback!

Some naming / terminology clarification in our replies below: Dual-Branch Injection corresponds to Final-State Injection in the initial submission. Manuscript references use revised numbering unless marked “submitted” (which means the original submitted version); page and line references point to the revised, highlighted PDF.

W1. Contribution beyond existing modulation and cross-attention mechanisms

Our contribution is a language-to-SID interface that improves retrieval and steering while reusing a frozen causal LM. Against cross-attention-only with the same frozen backbone and generator architecture, Dual-Branch raises test Preference R@10 by 0.15–1.24 pp and Sentiment@10 by 0.81–1.71 pp across all four datasets (Table Y1), and improves attribute selectivity in all three Amazon domains (Table Y4). These gains establish the value of the combined interface beyond freezing the host.

Mender uses language-to-SID cross-attention, and pooled text modulation plus token cross-attention has precedents in text-to-music generation. We build on these mechanisms to study history-conditioned autoregressive SID retrieval. This distinction is explicit in §2 (p. 3, lines 126–133; highlighted).

The pooled states drive per-block AdaRMS modulation and residual gates, while token states supply cross-attention keys and values. This provides a trainable retrieval add-on to a resident causal LM. See §1 (p. 1, lines 51–53; highlighted) and §4.2.

Table Y1. Held-out retrieval and Find/Avoid steering with frozen-language interfaces (%, three-seed mean ±\pm sample SD). Sentiment@10 measures paired anchor retrieval under Find and exclusion under Avoid, with browsing history fixed. Amazon results are from the submitted component/main comparisons; Steam is an additional dataset evaluated during rebuttal. Manuscript: §§5.1 and 5.2 (p. 8, lines 395–400), Table 2, Appendix E.3, Table 10 (p. 26), and Appendix E.6, Table 14 (p. 31).

DomainInterfacePreference R@10 ↑\uparrowSentiment@10 ↑\uparrow
BeautyFrozen Global-only6.40±0.126.40 \pm 0.121.53±0.291.53 \pm 0.29
BeautyFrozen Cross-Attention-only8.34±0.078.34 \pm 0.074.63±0.274.63 \pm 0.27
BeautyDual-Branch Injection8.73±0.18\mathbf{8.73 \pm 0.18}5.59±0.12\mathbf{5.59 \pm 0.12}
SportsFrozen Cross-Attention-only4.91±0.024.91 \pm 0.024.45±0.154.45 \pm 0.15
SportsDual-Branch Injection5.06±0.16\mathbf{5.06 \pm 0.16}5.65±0.06\mathbf{5.65 \pm 0.06}
ToysFrozen Cross-Attention-only6.34±0.146.34 \pm 0.143.13±0.093.13 \pm 0.09
ToysDual-Branch Injection7.00±0.19\mathbf{7.00 \pm 0.19}3.94±0.10\mathbf{3.94 \pm 0.10}
SteamFrozen Cross-Attention-only25.01±0.2825.01 \pm 0.288.15±0.218.15 \pm 0.21
SteamDual-Branch Injection26.25±0.12\mathbf{26.25 \pm 0.12}9.86±0.43\mathbf{9.86 \pm 0.43}

Sentiment@10 improves by 20.7–27.0% relative to cross-attention-only across the four datasets. On Beauty, Dual-Branch also exceeds global-only. These component comparisons use independently validation-selected configurations with different training budgets.

W2. Training strategy and the attribution of capability loss

We do not attribute catastrophic forgetting to the conditioning interface alone: the original comparison changes both the interface and the language-model update strategy. Host capability preservation follows from freezing, while the adaptation sweeps below examine retrieval–retention trade-offs within joint conditioning.

On Beauty, we evaluated 75 Full-FT and 375 LoRA checkpoints. Within the joint-conditioning architecture, the retrieval-selected LoRA checkpoint scores zero on BBH, GSM8K, and TriviaQA, whereas lower-learning-rate configurations retain substantially more capability at lower retrieval accuracy (Table Y2). These results support a retrieval–retention trade-off across adaptation settings rather than an interface-only explanation of forgetting.

Table Y2. Beauty retrieval-capability trade-off at original sweep checkpoints (%). NLP scores evaluate the adapted models. These are individual sweep checkpoints, not three-seed capability estimates. Manuscript: §5.3 (pp. 8–9, lines 422–440; highlighted), Table 3, and Appendix E.5. The 8-Task NLP Mean is the unweighted mean of the eight task accuracies in manuscript Table 3.

Beauty Sweep CheckpointSweep Val Pref R@10 ↑\uparrowBBH ↑\uparrowGSM8K ↑\uparrowTriviaQA ↑\uparrowMMLU ↑\uparrow8-Task NLP Mean ↑\uparrow
Pretrained Gemma-3-1B-IT (unmodified)--29.7820.0719.6340.0236.46
Joint Conditioning (Full-FT sweep, selected)11.710.000.000.0225.2821.39
LoRA sweep (r=64, lr=0.02, selected)9.200.000.000.0025.3618.60
LoRA sweep (r=64, lr=0.005)7.7817.8412.734.5037.0030.89
LoRA sweep (r=64, lr=0.001)3.1225.6520.3915.8638.4335.30

The 375 LoRA configurations per Amazon domain cover ranks {4, 8, 16, 32, 64}, peak learning rates {0.0005, 0.001, 0.005, 0.01, 0.02}, batch sizes {8, 12, 16, 32, 48}, and training steps {30k, 50k, 70k}. All attention and MLP projections and the tied token embeddings are adapted. The selected configuration in each domain was separately retrained over seeds 42–44 for retrieval and rephrasing (Table Y3). See Appendices D.1 (p. 22, lines 1163–1170; highlighted) and D.2 and Table 8 (p. 23) for the sweep and selected settings.

Table Y2 uses individual original-sweep checkpoints with SIDs-only history for LoRA. Table Y3 uses separate three-seed retrainings with history titles. Checkpoint correspondence and retention scope are documented in Appendix E.5 (pp. 29–30, lines 1536–1546 and 1575–1577).

The interface contribution is evaluated separately under a shared frozen-host regime: Dual-Branch improves retrieval and steering over cross-attention-only (W1, Table Y1). We have made the distinction between host freezing and interface benefits explicit in §5.3 (p. 9, lines 438–440; highlighted), §6, and Appendix E.5.

We have corrected the initialization description in §4.2 (p. 6, lines 279–281 and 302–303; highlighted), Figure 2, and Appendix C.1: only the language tower loads pretrained Gemma-3-1B-IT weights; the SID generator uses the same architecture with random initialization. Retained language capabilities refer to the host.

W3. Lexical robustness and response to attribute changes

We added controlled attribute-flip tests and narrowed the mechanism claims: lexical robustness alone does not establish semantic understanding.

Both frozen interfaces outperform Full-FT and LoRA on lexical substitution (Table Y3), so robustness is not unique to global modulation. Updated Figure 3 includes all four (§5.4, pp. 9–10, lines 444–496; highlighted).

Table Y3. Amazon retrieval and rephrasing (%, three-seed mean ±\pm sample SD). Pref = Preference R@10; validation columns report R@10 with fixed history/target. Full-FT/LoRA use Joint Conditioning; the other interfaces freeze the host. Full results: Appendix E.6, Table 13 (p. 30, lines 1592–1605).

Table Y3(a). Beauty

InterfaceTest Pref ↑\uparrowTest Sentiment@10 ↑\uparrowVal Original ↑\uparrowVal Lexical ↑\uparrowVal Syntactic ↑\uparrowVal Conversational ↑\uparrow
Full-FT9.43±0.09\mathbf{9.43\pm0.09}7.01±0.16\mathbf{7.01\pm0.16}11.60±0.10\mathbf{11.60\pm0.10}7.62±0.097.62\pm0.0911.09±0.09\mathbf{11.09\pm0.09}11.51±0.06\mathbf{11.51\pm0.06}
LoRA8.22±0.118.22\pm0.115.72±0.315.72\pm0.3110.23±0.2210.23\pm0.225.47±0.295.47\pm0.298.57±0.078.57\pm0.078.44±0.408.44\pm0.40
Cross-attention-only8.34±0.078.34\pm0.074.63±0.274.63\pm0.2710.89±0.2310.89\pm0.239.94±0.02\mathbf{9.94\pm0.02}10.75±0.1510.75\pm0.1510.75±0.1410.75\pm0.14
Dual-Branch8.73±0.188.73\pm0.185.59±0.125.59\pm0.1211.21±0.1111.21\pm0.119.49±0.309.49\pm0.3011.08±0.1411.08\pm0.1411.18±0.0511.18\pm0.05

Table Y3(b). Sports

InterfaceTest Pref ↑\uparrowTest Sentiment@10 ↑\uparrowVal Original ↑\uparrowVal Lexical ↑\uparrowVal Syntactic ↑\uparrowVal Conversational ↑\uparrow
Full-FT5.40±0.08\mathbf{5.40\pm0.08}6.15±0.27\mathbf{6.15\pm0.27}6.53±0.07\mathbf{6.53\pm0.07}5.61±0.075.61\pm0.076.29±0.116.29\pm0.116.27±0.106.27\pm0.10
LoRA4.52±0.014.52\pm0.015.45±0.305.45\pm0.305.88±0.135.88\pm0.134.71±0.254.71\pm0.255.34±0.165.34\pm0.165.15±0.275.15\pm0.27
Cross-attention-only4.91±0.024.91\pm0.024.45±0.154.45\pm0.156.24±0.146.24\pm0.146.03±0.166.03\pm0.166.18±0.166.18\pm0.166.13±0.156.13\pm0.15
Dual-Branch5.06±0.165.06\pm0.165.65±0.065.65\pm0.066.45±0.066.45\pm0.066.04±0.10\mathbf{6.04\pm0.10}6.39±0.05\mathbf{6.39\pm0.05}6.40±0.11\mathbf{6.40\pm0.11}

Table Y3(c). Toys

InterfaceTest Pref ↑\uparrowTest Sentiment@10 ↑\uparrowVal Original ↑\uparrowVal Lexical ↑\uparrowVal Syntactic ↑\uparrowVal Conversational ↑\uparrow
Full-FT7.79±0.37\mathbf{7.79\pm0.37}5.33±0.18\mathbf{5.33\pm0.18}9.50±0.37\mathbf{9.50\pm0.37}8.02±0.508.02\pm0.508.72±0.428.72\pm0.428.51±0.358.51\pm0.35
LoRA5.92±0.285.92\pm0.283.46±0.183.46\pm0.188.07±0.298.07\pm0.296.47±0.316.47\pm0.317.05±0.247.05\pm0.246.86±0.156.86\pm0.15
Cross-attention-only6.34±0.146.34\pm0.143.13±0.093.13\pm0.098.75±0.118.75\pm0.118.54±0.078.54\pm0.078.74±0.128.74\pm0.128.75±0.138.75\pm0.13
Dual-Branch7.00±0.197.00\pm0.193.94±0.103.94\pm0.109.02±0.089.02\pm0.088.65±0.05\mathbf{8.65\pm0.05}8.89±0.09\mathbf{8.89\pm0.09}8.89±0.07\mathbf{8.89\pm0.07}

The new test uses 2,612 attribute-flip pairs across Amazon, without retraining. With history fixed, “Select a Melissa & Doug sound puzzle that depicts ...” switches between “modes of transportation” and “melodic orchestral tools.” Disjoint vehicle (MAM_A) and musical-instrument (MBM_B) groups swap desired/opposing roles. Attribute phrases exclude original attribute-value words and stems of at least four characters; shared context remains.

For top-10 sets TA,TBT_A,T_B, selectivity measures desired-group hits minus opposing-group hits:

\begin{aligned} &\underbrace{\Pr(T_A \cap M_A \neq \varnothing) - \Pr(T_A \cap M_B \neq \varnothing)}_{\text{Request A}}\\\\[0.5em] &+ \underbrace{\Pr(T_B \cap M_B \neq \varnothing) - \Pr(T_B \cap M_A \neq \varnothing)}_{\text{Request B}} \end{aligned} \right].$$ $\Pr$ is the fraction of pairs; unchanged predictions yield zero selectivity. **Table Y4. Attribute selectivity $100\Delta$ (pp; three-seed mean $\pm$ sample SD).** The final column averages domains. See §5.1 (p. 8, lines 386–389) and Appendix E.7, Table 16 (pp. 31–32, lines 1652–1703; highlighted). |Interface|Beauty $\uparrow$|Sports $\uparrow$|Toys $\uparrow$|Domain mean $\uparrow$| |---|---:|---:|---:|---:| |Full-FT|$\mathbf{9.99\pm0.70}$|$\mathbf{5.59\pm0.38}$|$\mathbf{8.00\pm2.64}$|**7.86**| |LoRA|$4.86\pm0.59$|$1.09\pm0.78$|$3.01\pm0.67$|2.99| |Global-only|$-0.05\pm0.27$|$0.00\pm0.28$|$0.17\pm0.42$|0.04| |Cross-attention-only|$1.42\pm0.19$|$1.03\pm0.53$|$1.94\pm0.50$|1.46| |Dual-Branch|$3.80\pm2.20$|$2.68\pm1.14$|$7.27\pm0.71$|4.58| Dual-Branch exceeds both frozen single-branch interfaces in all domains. Full-FT leads on selectivity despite lower lexical robustness: the two tests measure distinct behaviors. Selectivity covers labeled catalog groups. We narrowed the claims in the abstract (p. 1, lines 22–32), §§1, 5.4, 6, and treat attention as qualitative (Appendix E.1, p. 24, lines 1270–1272; Figure 5; highlighted).
AuthorsOpenReview ↗

Follow-up to reviewer y96o

Dear Reviewer y96o,

We sincerely thank your feedback. We believe we have fully addressed your concerns. We here summarize our rebuttal context and look forward to your follow-up comments.

  • W1: It's true that both global feature modulation and cross-attention are not new, but combining them in steerable generative retrieval in the recsys domain and showing the steerability robustness tradeoff is still new to the community.
  • W2: We've updated in the manuscript why we focused on a frozen pretrained LM setup and added another controlled study (LoRA) which shares the same interface with single tower. This should address the control concern you raised.
  • W3: We've added the controlled attribute-flip pairs (eval-only) to demonstrate that how each models can follows the modified instructions and scoped our semantic-understanding claims.
Reviewer_pDr2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies natural-language conditioning for semantic-ID generative retrieval. It connects a frozen language tower to a trainable SID generator through global modulation and cross-attention. Across three Amazon domains, the method improves over cross-attention alone and is more robust to preference rephrasing than joint conditioning.

Critical strengths
  • The paired rephrasing evaluation is useful, with consistent improvements under synonym substitution across three domains.
  • The authors carefully audit synthetic shortcuts in the benchmark. The proposed interface also improves preference retrieval and directional steering over cross-attention alone.
Critical weaknesses
  • The central shortcut-versus-semantics conclusion goes beyond the evidence. Rephrasing sensitivity shows lexical brittleness, but does not establish that joint conditioning only memorizes keywords or that the proposed model understands intent.
  • The contribution beyond existing frozen-model and adapter approaches remains unclear. Preserving an unchanged language tower is expected, and the key robustness comparison omits alternatives that could offer the same benefit without the proposed dual-branch interface.
  • The claimed advantage of final-layer semantic abstraction is not established by the depth experiment. It compares two routing strategies, but the paper interprets the result as evidence that intermediate representations contain harmful lexical noise.
Questions for authors
  1. The synonym results are interesting, but I don't think they establish the shortcut-versus-semantics distinction. A model can use semantic information and still overfit to familiar wording. Conversely, lower sensitivity to wording does not necessarily mean better intent following. Does the proposed model respond correctly when the requested attribute changes but most words stay the same? The find/avoid test helps, but it only checks one anchor item's inclusion or exclusion, not whether the remaining recommendations satisfy the constraint. The single attention example doesn't settle this either. The robustness result is convincing, but the proposed explanation is much less so.

  2. What does the new interface add beyond existing ways to preserve pretrained representations? Prior work already uses language-to-SID cross-attention [1]. Your frozen cross-attention baseline is particularly relevant, yet it is missing from the rephrasing experiment. An adapter baseline such as LoRA [2] is also missing. Keeping an untouched language tower preserves that tower's capabilities by construction. It doesn't show that global modulation solves forgetting or that the SID generator retains those capabilities. The retrieval gains are useful, but the main claimed advantage needs to be established against these alternatives, not only full fine-tuning.

  3. The source-depth conclusion is stronger than the experiment supports. Broadcasting the final layer to every generator block beats matching language and generator layers under global-only modulation. That does not show that intermediate layers are inherently noisy or less semantic. The difference could come from giving every block the same information rather than changing the source across depth. Appendix E.2 acknowledges this limitation, but the main text still presents lexical-noise filtering as an established explanation. How does a fixed intermediate layer broadcast to all blocks compare? Without that distinction, final-layer broadcasting is an empirical design choice, not an established principle about SID decoding.

[1] Preference Discerning with LLM-Enhanced Generative Retrieval [2] LoRA: Low-Rank Adaptation of Large Language Models

9 comments
AuthorsOpenReview ↗

Reply to Reviewer pDr2

We thank pDr2 for their feedback!

Some naming / terminology clarification in our replies below: Dual-Branch Injection corresponds to Final-State Injection in the initial submission; Joint Conditioning (Full-FT) denotes full-parameter single-tower fine-tuning. Manuscript references use revised numbering unless marked “submitted” (which means the original submitted version); page and line references point to the revised, highlighted PDF.

Q1. Wording robustness, counterfactual attribute changes, and Avoid compliance

Q1(a). Does synonym robustness establish "shortcuts versus semantics"?

Synonym robustness establishes wording stability, not whether the model follows changed requests. The new attribute-flip and Find/Avoid analyses below test request responsiveness directly.

  1. Joint Conditioning (Full-FT) has the highest attribute selectivity despite its lower lexical-rephrasing R@10 (Tables R1 and R3).
  2. Dual-Branch responds to changed attributes and polarity; Frozen Global-only has near-zero attribute selectivity (Q1(b)–(c)).

We have narrowed the claims in the abstract (p. 1, lines 22–32; highlighted) and §§1, 5.4, and 6 to these measured behaviors; Appendix E.1 (p. 24, lines 1270–1272; highlighted) and Figure 5 treat the attention plot as a qualitative illustration.

Q1(b). Does the model respond when the requested attribute changes while most words stay fixed?

Dual-Branch shifts retrieval toward the requested attribute group in all three domains (Table R1). We tested 2,612 request pairs on existing checkpoints without further training, holding history and the request frame fixed and changing only the attribute phrase. The paraphrases exclude the original attribute-value words and word stems of at least four characters. For example:

  • Request A: "Select a Melissa & Doug sound puzzle that depicts modes of transportation" (desired catalog group MAM_A: vehicle sound puzzles; contrasting group MBM_B: musical-instrument sound puzzles, with MA∩MB=∅M_A \cap M_B = \varnothing).
  • Request B: "Select a Melissa & Doug sound puzzle that depicts melodic orchestral tools" (desired group MBM_B; contrasting group MAM_A).

Let TAT_A and TBT_B denote the top-10 predicted sets for Requests A and B. Selectivity Δ\Delta measures whether the top-10 favors the requested attribute group over the contrasting group across both directions:

\begin{aligned} &\underbrace{\Pr(T_A \cap M_A \neq \varnothing) - \Pr(T_A \cap M_B \neq \varnothing)}_{\text{Request A}}\\\\[0.5em] &+ \underbrace{\Pr(T_B \cap M_B \neq \varnothing) - \Pr(T_B \cap M_A \neq \varnothing)}_{\text{Request B}} \end{aligned} \right].$$ Pr is the empirical fraction of paired records; selectivity is reported in percentage points. Identical predictions for both requests give zero selectivity. PairedFlip@10 additionally requires both directions to succeed without any labeled opposing item (definition in S1). **Table R1. Counterfactual attribute selectivity ($100\Delta$, pp) and strict PairedFlip@10 (%) on the 2,612-pair paraphrased-attribute slice across three Amazon domains.** Manuscript: §5.1 (p. 8, lines 386–389; highlighted) and Appendix E.7 (pp. 31–32, lines 1652–1714; highlighted), Tables 16–17. | Interface | Beauty $100\Delta \uparrow$ | Sports $100\Delta \uparrow$ | Toys $100\Delta \uparrow$ | Mean across domains: $100\Delta$ (pp) $\uparrow$ | Mean across domains: PairedFlip@10 (%) $\uparrow$ | |---|---:|---:|---:|---:|---:| | Joint Conditioning (Full-FT) | **$\mathbf{9.99 \pm 0.70}$** | **$\mathbf{5.59 \pm 0.38}$** | **$\mathbf{8.00 \pm 2.64}$** | **7.86** | **2.32** | | Joint Conditioning (LoRA) | $4.86 \pm 0.59$ | $1.09 \pm 0.78$ | $3.01 \pm 0.67$ | 2.99 | 0.89 | | Frozen Global-only | $-0.05 \pm 0.27$ | $0.00 \pm 0.28$ | $0.17 \pm 0.42$ | 0.04 | 0.00 | | Frozen Cross-Attention-only | $1.42 \pm 0.19$ | $1.03 \pm 0.53$ | $1.94 \pm 0.50$ | 1.46 | 0.27 | | Dual-Branch Injection (ours) | $3.80 \pm 2.20$ | $2.68 \pm 1.14$ | $7.27 \pm 0.71$ | 4.58 | 1.03 | Domain columns are three-seed means ± sample SD (seeds 42–44); the last two columns average the three domains equally. Dual-Branch exceeds both frozen single-branch interfaces in every domain. It also exceeds LoRA on Sports and Toys and on the three-domain mean; LoRA is higher on Beauty. Full-FT has the highest mean selectivity. PairedFlip@10 gives the same ordering of domain averages.

Q1(c). In Find/Avoid, do the remaining top-10 recommendations satisfy the Avoid constraint?

Beyond the anchor-based Sentiment@10 metric, we now inspect complete top-10 outputs: a paired Find/Avoid example (Table R2) and a 60-request audit (Table R2b).

Avoid training uses standard positive SFT on replacement items screened for constraint compliance and history compatibility, retaining candidates with distance_score ≥ 7 (§3.4, p. 5, lines 254–258; highlighted; Appendices A.4 and B.2). This teaches the generator to produce a history-compatible alternative under Avoid.

With the same checkpoint and history, “Find harsh beads in exfoliating scrubs” retrieves ten facial-care items. Switching to “Avoid” produces nine hair-care or styling items and one facial cleansing cloth, with no exfoliating scrub in the top ten. The hair-care items relate to John Frieda Root Booster in the history. Table R2 shows the complete illustrative Avoid list.

Table R2. Illustrative complete Avoid top-10 on Beauty (Dual-Branch Injection, seed 43). Manuscript: §5.1 and Appendix E.8 (pp. 32–33, lines 1717–1750; highlighted), Table 18. History (abridged): (1) Sunflowers fragrance; (2) Dermalogica Gentle Cream Exfoliant; (3) Body Toolz facial skin-care tool; (4) John Frieda Root Booster. Request: "Avoid harsh beads in exfoliating scrubs." (Recommended item titles are verbatim catalog strings with HTML entities decoded.)

RankRecommended itemEvidence concerning AvoidHistory connection
1Alterna Caviar Replenishing Moisture Shampoo & Conditioner Duo (8.5 oz each)Hair shampoo/conditioner; outside exfoliating-scrub categoryHair care (#4)
2WEN by Chaz Dean Lavender Cleansing Conditioner 16 ozHair conditioner; outside exfoliating-scrub categoryHair care (#4)
3TRESemme Thermal Creations Heat Tamer Protective Spray 8 fl oz (236 ml)Hair protection spray; outside exfoliating-scrub categoryHair styling (#4)
4Remington B90V Pearl Ceramic Vent Hair Brush with Real Crushed PearlsHair brush; outside exfoliating-scrub categoryHair styling (#4)
5OGX Shampoo, Thick & Full Biotin & Collagen, 13ozHair shampoo; outside exfoliating-scrub categoryHair care (#4)
6Remington CI95AC/2 Tstudio Salon Collection Pearl Digital Ceramic Curling Wand, 1/2 Inch - 1 InchHair styling tool; outside exfoliating-scrub categoryHair styling (#4)
7Enjoy Luxury Duo (Shampoo and Conditioner) - 33.8 OzHair shampoo/conditioner; outside exfoliating-scrub categoryHair care (#4)
8Wen By Chaz Dean Fall Apple Spice Mist 2 OzHair mist; outside exfoliating-scrub categoryHair care (#4)
9WEN by Chaz Dean Six Thirteen Replenishing Treatment Mist 2 fl ozHair treatment mist; outside exfoliating-scrub categoryHair care (#4)
10Olay 4-in-1 Normal Daily Facial Cloths, 66-CountFacial cleansing cloths; outside exfoliating-scrub categoryFacial skin care (#2–3)

We also audited 60 randomly sampled Avoid requests, twenty per domain, using the same requests and assigned seed across methods. A single-LLM audit inspected titles, available descriptions, and review excerpts after SID-to-item mapping. We counted a violation only when this evidence explicitly indicated the excluded attribute.

Table R2b. Detected Avoid violations across complete top-10 lists on the same 60 sampled requests. The metric counts recommendation slots with a detected violation out of 600 slots per method (60 requests × 10 recommendations), based on the available product evidence. Repeated items across requests count separately. Manuscript: §5.1 (p. 8, lines 389–392; highlighted) and Appendix E.8 (p. 33, lines 1753–1777; highlighted), Table 19.

MethodRecommendations with detected violations ↓\downarrow
Joint Conditioning (Full-FT)51/600 (8.5%)
Joint Conditioning (LoRA)48/600 (8.0%)
Frozen Cross-Attention-only44/600 (7.3%)
Dual-Branch Injection (ours)37/600 (6.2%)

Dual-Branch has the lowest detected-violation rate in this sample; unflagged items mean no violation was detected in the available evidence.

Q2. Contribution beyond frozen cross-attention and LoRA adapters

Q2(a). What does Dual-Branch Injection add beyond Frozen Cross-Attention, including under rephrasing?

Our contribution is a language-to-SID interface that improves retrieval and steering while reusing a frozen causal LM. Against cross-attention-only with the same frozen backbone and generator architecture, Dual-Branch raises test Preference R@10 by 0.15–1.24 pp and Sentiment@10 by 0.81–1.71 pp across all four datasets (Tables R3–R4 below).

The gains establish the value of the combined interface beyond freezing the pretrain LM; The contribution and relation to prior work are emphasized in §§1–2 (p. 3, lines 126–133; highlighted); the interface is specified in §4.2.

Table R3. Three-domain Amazon test steering and paired validation rephrasing. Manuscript: §5.1, Table 2 (p. 7, lines 333–348), §5.4, and Appendix E.6, Table 13 (p. 30, lines 1592–1605).

Table R3(a). Beauty

InterfaceTest Preference R@10 ↑\uparrowTest Sentiment@10 ↑\uparrowVal Original R@10 ↑\uparrowVal Lexical R@10 ↑\uparrowVal Syntactic R@10 ↑\uparrowVal Conversational R@10 ↑\uparrow
Joint Conditioning (Full-FT)9.43±0.09\mathbf{9.43 \pm 0.09}7.01±0.16\mathbf{7.01 \pm 0.16}11.60±0.10\mathbf{11.60 \pm 0.10}7.62±0.097.62 \pm 0.0911.09±0.09\mathbf{11.09\pm0.09}11.51±0.06\mathbf{11.51\pm0.06}
Joint Conditioning (LoRA)8.22±0.118.22 \pm 0.115.72±0.315.72 \pm 0.3110.23±0.2210.23 \pm 0.225.47±0.295.47 \pm 0.298.57±0.078.57 \pm 0.078.44±0.408.44 \pm 0.40
Frozen Cross-Attention-only8.34±0.078.34 \pm 0.074.63±0.274.63 \pm 0.2710.89±0.2310.89 \pm 0.239.94±0.02\mathbf{9.94 \pm 0.02}10.75±0.1510.75\pm0.1510.75±0.1410.75\pm0.14
Dual-Branch Injection (ours)8.73±0.188.73 \pm 0.185.59±0.125.59 \pm 0.1211.21±0.1111.21 \pm 0.119.49±0.309.49 \pm 0.3011.08±0.1411.08\pm0.1411.18±0.0511.18\pm0.05

Table R3(b). Sports

InterfaceTest Preference R@10 ↑\uparrowTest Sentiment@10 ↑\uparrowVal Original R@10 ↑\uparrowVal Lexical R@10 ↑\uparrowVal Syntactic R@10 ↑\uparrowVal Conversational R@10 ↑\uparrow
Joint Conditioning (Full-FT)5.40±0.08\mathbf{5.40 \pm 0.08}6.15±0.27\mathbf{6.15 \pm 0.27}6.53±0.07\mathbf{6.53 \pm 0.07}5.61±0.075.61 \pm 0.076.29±0.116.29\pm0.116.27±0.106.27\pm0.10
Joint Conditioning (LoRA)4.52±0.014.52 \pm 0.015.45±0.305.45 \pm 0.305.88±0.135.88 \pm 0.134.71±0.254.71 \pm 0.255.34±0.165.34 \pm 0.165.15±0.275.15 \pm 0.27
Frozen Cross-Attention-only4.91±0.024.91 \pm 0.024.45±0.154.45 \pm 0.156.24±0.146.24 \pm 0.146.03±0.166.03 \pm 0.166.18±0.166.18\pm0.166.13±0.156.13\pm0.15
Dual-Branch Injection (ours)5.06±0.165.06 \pm 0.165.65±0.065.65 \pm 0.066.45±0.066.45 \pm 0.066.04±0.10\mathbf{6.04 \pm 0.10}6.39±0.05\mathbf{6.39\pm0.05}6.40±0.11\mathbf{6.40\pm0.11}

Table R3(c). Toys

InterfaceTest Preference R@10 ↑\uparrowTest Sentiment@10 ↑\uparrowVal Original R@10 ↑\uparrowVal Lexical R@10 ↑\uparrowVal Syntactic R@10 ↑\uparrowVal Conversational R@10 ↑\uparrow
Joint Conditioning (Full-FT)7.79±0.37\mathbf{7.79 \pm 0.37}5.33±0.18\mathbf{5.33 \pm 0.18}9.50±0.37\mathbf{9.50 \pm 0.37}8.02±0.508.02 \pm 0.508.72±0.428.72\pm0.428.51±0.358.51\pm0.35
Joint Conditioning (LoRA)5.92±0.285.92 \pm 0.283.46±0.183.46 \pm 0.188.07±0.298.07 \pm 0.296.47±0.316.47 \pm 0.317.05±0.247.05 \pm 0.246.86±0.156.86 \pm 0.15
Frozen Cross-Attention-only6.34±0.146.34 \pm 0.143.13±0.093.13 \pm 0.098.75±0.118.75 \pm 0.118.54±0.078.54 \pm 0.078.74±0.128.74\pm0.128.75±0.138.75\pm0.13
Dual-Branch Injection (ours)7.00±0.197.00 \pm 0.193.94±0.103.94 \pm 0.109.02±0.089.02 \pm 0.088.65±0.05\mathbf{8.65 \pm 0.05}8.89±0.09\mathbf{8.89\pm0.09}8.89±0.07\mathbf{8.89\pm0.07}

Table R4. Steam test steering and paired validation lexical rephrasing. Manuscript: §5.1 (p. 8, lines 381–386; highlighted) and Appendix E.6, Table 14 (p. 31, lines 1621–1628; also reports syntactic and conversational rewrites).

InterfaceTest Preference R@10 ↑\uparrowTest Sentiment@10 ↑\uparrowVal Original R@10 ↑\uparrowVal Lexical R@10 ↑\uparrow
Joint Conditioning (Full-FT)27.74±0.25\mathbf{27.74 \pm 0.25}14.18±0.33\mathbf{14.18 \pm 0.33}29.63±0.25\mathbf{29.63 \pm 0.25}18.09±1.7718.09 \pm 1.77
Frozen Cross-Attention-only25.01±0.2825.01 \pm 0.288.15±0.218.15 \pm 0.2127.11±0.2627.11 \pm 0.2618.77±2.0218.77 \pm 2.02
Dual-Branch Injection (ours)26.25±0.1226.25 \pm 0.129.86±0.439.86 \pm 0.4328.44±0.2428.44 \pm 0.2421.78±0.37\mathbf{21.78 \pm 0.37}

The Amazon cross-attention baseline was already reported in the submission; Tables R3–R4 add the requested rephrasing comparisons, LoRA, and Steam. Both frozen interfaces outperform Full-FT under lexical substitution across all four datasets, so this robustness is not unique to global modulation. Figure 3 now includes all four interfaces (§5.4, pp. 9–10, lines 444–496; highlighted; Appendix E.6, Table 13).

Comments on the paper

2 comments
AuthorsOpenReview ↗

General comment to the reviewers and Area Chair

We thank all reviewers for their insightful reviews and feedback. We have revised the manuscript and added experiments addressing the shared questions about the conditioning interface, alternative adaptation strategies, and request following. Manuscript references below use revised numbering; relevant passages are highlighted in yellow. CW and Q refer to Critical Weaknesses and Questions for Authors in the original reviews.

  • Interface contribution (pDr2 Q2; y96o CW1; ghQj CW1/CW3). With the host frozen in both models, Dual-Branch improves test Preference R@10 by 0.15–1.24 percentage points and Sentiment@10 by 0.81–1.71 points over cross-attention-only across Beauty, Sports, Toys, and Steam. These gains support our language-to-SID interface as a retrieval add-on to a resident causal LM (§§1, 4.2, and 5.1; Table 2; Appendix E.6, Table 14).

  • LoRA comparisons (pDr2 Q2; y96o CW2). We added sweeps of 375 configurations per Amazon domain (Appendices D.1–D.2), three-seed retrieval and rephrasing comparisons (§§5.1 and 5.4; Table 2; Appendix E.6, Table 13), and an eight-task Beauty capability sweep (§5.3; Table 3; Appendix E.5). Dual-Branch exceeds validation-selected LoRA in original test Preference R@10 and all nine Amazon rephrasing comparisons.

  • Additional evaluation. Steam extends the evaluation to games, addressing ghQj Q2 (§5.1; Appendix E.6, Tables 14–15). We also added 2,612 attribute-flip pairs to test responses to changed requests, addressing pDr2 Q1, y96o CW3, and ghQj Q1 (§5.1; Appendix E.7, Tables 16–17). A 60-request audit of complete Avoid top-10 lists addresses pDr2 Q1 (§5.1; Appendix E.8, Table 19).

  • Source-depth controls (pDr2 Q3). A 90-run layer sweep compares fixed-source broadcasting across ten layers. We retain final-layer conditioning as an empirical design choice (§5.2; Appendix E.2, Figure 6 and Table 9).

  • Clarified claims and implementation. Full-FT remains strongest on original-request retrieval and attribute selectivity, while both frozen interfaces are stronger under lexical substitution. We narrowed the mechanism claims to distinguish wording stability from request responsiveness, addressing pDr2 Q1, y96o CW3, and ghQj Q1 (abstract; §§1, 5.4, and 6). We also corrected the SID generator's initialization description to random initialization (§4.2; Figure 2; Appendix C.1).

The individual replies address each reviewer's questions with the relevant results.

AuthorsOpenReview ↗

Summary of Rebuttal Experiments and Manuscript Revisions

Dear Area Chair and Reviewers,

As the discussion period nears its close, we sincerely thank the reviewer who has already engaged with our rebuttal and raised their score. We believe our new experiments, calibrated claims, and scope clarifications (yellow for Round-1, green for Round-2 in the updated 34-page PDF) address the core empirical questions across the reviews, and we warmly invite the remaining reviewers to let us know if any questions remain and kindly consider raising their scores as well. Below is a concise cross-cutting summary of all updates:

  • Dual-branch gains over frozen single-branch baselines: Under the same frozen Gemma-3-1B-IT host and randomly initialized SID generator (§4.2 and Figure 2), Dual-Branch consistently outperforms Frozen Cross-Attention-only across all four datasets (Table 2, Table 15) and lifts three-domain counterfactual attribute selectivity.
  • Single-tower LoRA sweeps and capability trade-offs: Across 375 LoRA configurations per Amazon domain and an 8-task Beauty capability evaluation (Table 3, Figure 8), retrieval-selected LoRA (lr = 0.02) still drops to 0% on BBH, GSM8K, and TriviaQA, while lower learning rates preserve reasoning only at much lower retrieval. Keeping the tied embedding/output matrix trainable matches Full-FT and maximizes single-stream LoRA's retrieval capacity, yet Dual-Branch still beats selected LoRA on test Preference R@10 across all Amazon domains, all paired rephrasings, and mean attribute selectivity.
  • Phrasing stability vs. counterfactual request following and Avoid compliance: We narrowed main-text claims to distinguish phrasing stability from request responsiveness (framing Figure 5 as qualitative). Both frozen interfaces remain far more stable than Full-FT and LoRA under synonym substitution (including on the shared non-fallback rewritten subsets of Sports and Toys in new Table 14, where Full-FT and LoRA drop a lot with Cross-Attention-only highest on Beauty and subset lexical R@10), while 2,612 paraphrased attribute-flip pairs (Tables 17–18) and a 60-request complete top-10 Avoid audit (Table 20) test request responsiveness directly.
  • Cross-dataset validation on Steam and source-depth controls: Independent RQ-VAE and retriever training on the public Steam game dataset (Tables 15–16) reproduces the same retrieval, steering, and rephrasing trends beyond Amazon, and a 90-run fixed-source broadcast sweep across 10 layers (Figure 6, Table 9) supports final-layer conditioning as an empirical design choice without claiming intermediate lexical noise.
  • Deployment scope and explicit boundaries: We clarified in §§1 and 6 that Dual-Branch targets adding steerable SID retrieval as an extension to an already-resident causal LLM (matching backbone, tokenizer, and width against Joint Conditioning rather than benchmarking standalone bidirectional/compact encoders or serving latency), noted unseen-format generalization on Intent Following (Train = 0) and colloquial/syntactic rewrites, and explicitly stated in §6 that unfamiliar terminology absent from pretraining remains bounded by the frozen host.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.