acceptodds
Under review as a conference paper at ICLR 2027

Learning to Navigate Fixed Semantic IDs with Joint-Prefix Behavioral Distillation

Abstract

Semantic identifiers (SIDs) turn generative recommendation into an irreversible path-search problem: once an early prefix is pruned, the target item becomes unreachable. We identify a training–search mismatch in SID retrieval: behavioral L2 supervision is normalized separately within each L1 prefix (parent branch), whereas beam search compares cumulative scores across L1L2 prefixes rooted at different L1 codes. We propose Joint-Prefix Behavioral-Neighbor Distillation (JP-BND), which projects training-partition behavioral evidence onto joint L1L2 prefixes and distills it over the joint prefixes that compete under the current student, without changing the SID mapping or inference procedure. A joint-KL decomposition separates within-parent conditional alignment from cross-parent marginal alignment. On Delayed development, Full-SID Hit@500 increases from 3.82% for the Hard-example BND (H-BND) baseline to 8.60% with a matched multi-parent conditional control and 10.66% with core JP-BND. Fixed-candidate KL diagnostics show that conditional training primarily reduces within-parent mismatch, whereas joint normalization additionally reduces cross-parent marginal mismatch. A 5,000-instance constrained beam-search audit further reveals selective cross-parent reordering rather than a uniform lift in target-prefix margins. The gains remain stable across three training seeds and a separate Delayed held-out evaluation, persist at every tested inference beam width from 20 to 500, and transfer at the SID-path level to Amazon Books and Home public datasets. Together, these results show that aligning behavioral supervision with the prefixes that actually compete during beam search improves generative retrieval over fixed SID spaces.

Primary area:
generative models
Keywords:
generative recommendation, semantic identifiers, autoregressive decoding, beam search, knowledge distillation, structured prediction

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_6GcuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

JP-BND distills behavioral evidence over joint L1L2 prefixes that compete under the current student's beam frontier. It keeps the SID mapping and inference procedure fixed. A matched conditional objective and the KL chain rule separate within-parent supervision from cross-parent marginal alignment. Evaluation includes a private temporal benchmark, three training seeds, beam-search diagnostics, and public Amazon Books/Home experiments.

Critical strengths
  1. The central mechanism has a well-matched comparison. On Delayed development, Full-SID Hit@500 rises from 3.82% for H-BND to 8.60% for the conditional control and 10.66% for joint distillation. The joint-versus-conditional gain is 2.06 percentage points, with a paired 95% interval of [1.56, 2.56] (Tables 4 and 14). This separates joint normalization from merely adding multi-parent supervision.
  2. The diagnostics support a specific, appropriately limited explanation. On identical frozen candidates, joint training reduces parent-marginal KL while leaving conditional KL almost unchanged (Table 15). Search traces show selective rescues and regressions rather than a uniform margin increase. Three-seed held-out results and actual beam-width reruns strengthen the evidence beyond one development-set endpoint.
Critical weaknesses
  1. The training cost is substantial. Core JP-BND takes 46.84 hours versus 2.35 hours for H-BND on the same hardware, excluding preparation (§4.2; Appendix A.5). Matching update counts does not match computation. The paper establishes an accuracy gain at much greater training cost, but not an advantage over spending comparable compute on the baseline.
  2. Better path coverage has clear limits as a recommendation improvement. Dynamics Full-SID falls from 1.98% to 1.25% (Table 4). Within the fixed width-500 output, SID NDCG@10 falls from 0.007633 to 0.001539 (Table 11), despite gains when search is rerun with narrower beams. On Home, core JP-BND improves SID coverage but reduces Expected Item by 0.052 percentage points (Table 19). These results limit the demonstrated benefit to particular retrieval objectives and populations.
Questions for authors
  1. Compute comparison: Does JP-BND retain an advantage when H-BND receives a comparable wall-clock training budget, rather than the same number of updates?
  2. Scope clarification: Is the additional cross-parent benefit also present on Books/Home against the matched conditional control? Appendix F.3 explicitly leaves this effect unidentified, which limits how directly the temporal mechanism transfers to the public results.
2 comments
AuthorsOpenReview ↗

Equal-Time Comparison and Joint Normalization on Public Datasets

Q1. Does the Delayed advantage persist when H-BND receives the same training time?

Thank you for recognizing the matched conditional comparison and mechanism audits. Yes. The Delayed gain persists after giving H-BND the same student-training budget. We improved Core JP-BND's computational efficiency without changing the training objective, data, Stage 2 initialization, candidate selection rule, or inference. Batched prefix scoring, shared prompt computation, and GPU processing of candidate expansion and teacher alignment reduced measured student training time from 46.84 to 7.81 hours (6.0×) for 3,519 updates on eight H20 GPUs. On eight fixed examples, relative differences against the original serial scorer were 1.97 × 10⁻⁴ for joint loss and 7.27 × 10⁻⁵ for global gradient ℓ₂ norm.

We independently trained H-BND from the same Stage 2 checkpoint and training view for 7.81 hours on eight H20 GPUs. It completed 10,653 updates with repeated data passes and a time-budgeted warmup/cosine schedule. We evaluated the final checkpoint with beam width B=500 and an item budget of 500.

MethodTraining (h)UpdatesFull-SID@500 (%)Expected Item@500 (%)
H-BND, original budget2.353,5193.822.14
H-BND, equal time7.8110,6534.122.58
Core JP-BND (efficient implementation)7.813,51910.643.80

More training helps H-BND, but does not close the gap. At equal training time, Core improves Delayed Full-SID by +6.52 percentage points (pp), 95% paired CI [5.68, 7.36], and Expected Item by +1.22 pp [0.64, 1.80]. Dynamics favors H-BND (Core minus H-BND: Full-SID −0.80 pp [−1.04, −0.56], Expected Item −0.691 pp [−0.882, −0.503]). On Retention, the differences are +0.505 pp [0.000, 1.011] and +0.012 pp [−0.459, 0.487]. Intervals use 10,000 paired bootstrap draws over evaluation instances for one training seed. Section 4.2 reports this comparison. Appendix A.5 provides the full results (Tables 8–9) and separately documents reference and teacher preparation costs.

Q2. Does joint normalization improve retrieval on Books and Home over the matched conditional control?

Yes. Core JP-BND improves Full-SID coverage over Conditional on both public datasets. Under the TIGER protocol in Section 4.4 (Table 5), both arms use dense A2 as their base (without the industrial hard-example mask or optional boundary/anchor losses) and share initialization, training instances, update budget, behavioral teacher support, candidate construction, and inference. Both use the same 3,295-update budget and differ in whether the additional behavioral supervision is normalized within each parent or jointly over competing L1L2 prefixes.

DomainConditional Full-SID (%)Core Full-SID (%)Core−Conditional Full-SID (pp, 95% CI)Core−Conditional Expected Item (pp, 95% CI)
Books2.85022.8914+0.0412 [0.0198, 0.0629]−0.0118 [−0.0263, 0.0023]
Home4.21004.2357+0.0256 [0.0032, 0.0478]−0.0734 [−0.0918, −0.0541]

Each domain has 1,046,262 paired evaluation instances. The additional joint-normalization gain in SID-path coverage is positive in both domains, extending the industrial matched-control result to public data.

Expected Item is a different endpoint: it depends on the fixed SID-to-item mapping, collision groups, returned group order, and 500-item expansion budget (Appendix C.7). The generator scores SIDs, not individual items within a shared SID. Thus, more recovered target SIDs need not yield more item credit. The Books Expected Item interval includes zero. On Home, Conditional has higher Expected Item. The submitted manuscript reports the same path–item distinction for Core versus A2 on Home: +0.220 pp Full-SID and −0.052 pp Expected Item (Appendix F.3, Table 26).

The matched controls are reported in Section 4.4 and Table 5, with paired intervals in Appendix F.3 (Table 26).

Together, these results show that the Delayed gain survives the tested equal-time budget and that joint normalization contributes additional SID-path coverage on both public datasets.

Thanks for your reply. My concern is partially resolved. I will maintain my positive score.

Reviewer_3KhvOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper addresses the "train-search mismatch" problem in generative recommendation, where semantic ID prefixes may be erroneously pruned during beam search. To mitigate this, we propose a method called "Joint-Prefix Behavioral-Neighbor Distillation" (JP-BND), which leverages behavioral information to improve the prediction of joint prefixes across different parent branches. Experiments conducted on a proprietary video recommendation dataset and two public Amazon datasets demonstrate that this approach enhances the retrieval performance of semantic IDs.

Critical strengths

This paper identifies a significant "training-search mismatch" issue in generative recommendation and proposes a targeted solution. The experimental section includes controlled ablation studies, training with multiple random seeds, and evaluations on both private and public datasets.

Critical weaknesses
  1. The issue of a "training-search mismatch" has been addressed in previous research, such as RIPOR, BEAR, and SmartGR. The primary contribution of this paper appears to be merely an incremental extension of existing "prefix-aware" training methods.
  2. Although JP-BND improves the Hit@500 metric in the Full-SID scenario, it underperforms compared to H-BND across metrics such as Hit@10, Hit@20, Hit@50, MRR, and nDCG. This raises questions regarding its practical utility in real-world recommendation scenarios.
  3. While the method yields performance gains in "delayed" scenarios, it shows a significant decline in performance within "dynamic" scenarios. Furthermore, the performance improvements observed on public Amazon datasets are relatively limited.
  4. The training time for JP-BND is approximately 20 times that of H-BND. The paper fails to provide a sufficient justification for this computational overhead, nor does it offer a comparison based on equivalent computational resources.
  5. The comparisons with SmartGR and BEAR in the paper utilize modified versions rather than the original implementations. Consequently, a more rigorous evaluation against existing methods under fair experimental conditions is necessary.
Questions for authors
  1. Why does JP-BND show improvement on the Hit@500 metric, yet underperform compared to H-BND on the Hit@10, Hit@20, MRR, and nDCG metrics?
  2. Why does the proposed method exhibit a significant drop in performance on the Dynamics dataset?
  3. How do the authors account for the fact that their training cost is nearly 20 times higher than that of H-BND?
  4. Why is the performance gain observed on the public Amazon dataset substantially smaller than the gains achieved on the private dataset?
  5. Can the authors provide a more compelling comparison, specifically, one against the original implementations of existing methods such as SmartGR and BEAR?
4 comments
AuthorsOpenReview ↗

Methodological Contribution, Ranking Trade-offs, and Dynamics

Thank you for recognizing the controlled evaluation. We have added a comparison using the same training time, public matched conditional controls, and a transfer of the released BEAR loss to Books and Home. We address the contribution and retrieval behavior here, followed by the new comparisons in Response 2. References are to the revised manuscript.

W1. What does JP-BND add beyond existing prefix-aware training?

JP-BND studies how to distribute behavioral supervision across the parent branches that compete during decoding. RIPOR ranks positive and negative prefixes using teacher margins. BEAR targets local token survival along teacher-forced paths. SmartGR distills hierarchical predictions and prefix rankings from a larger teacher. JP-BND constructs a multi-target distribution from training-partition behavioral relations and distills it over the current student's legal L1L2 candidates across parents, retaining the original SID mapping and decoder.

The key distinction is visible in a matched comparison. The conditional control already supervises multiple parents, but normalizes separately within each one. Adding the same offset to all scores under one parent leaves its conditional distribution unchanged, although it changes that parent's position relative to other branches. Joint normalization directly supervises this allocation across parents. AR remains unchanged in both arms.

The additional joint effect is measured, rather than inferred from the full improvement over H-BND. Delayed Full-SID is 3.82% for H-BND, 8.60% for Conditional, and 10.66% for Core (Section 5.1, Table 6). The joint increment is +2.06 pp [1.56, 2.56] (Appendix E.1, Table 19), with held-out increments of +2.30, +2.44, and +2.47 pp across three seeds, each with a positive paired interval (Appendix D.1). On the same 107 active instances and identical frozen candidates, parent-marginal KL decreases from 2.6324 to 2.2848 nats, while conditional KL stays near 1.933 (Section 5.2, Figure 2(a); Appendix E.2, Table 20). These results connect the added joint objective to both retrieval gains and the intended distributional change.

W2 / Q1. Why does Hit@500 improve while head ranking worsens?

The improvement is broader candidate coverage, rather than concentration of hits at the top of the same list. Of 411 Full-SID rescues over H-BND, 314 appear at ranks 101–500 (Appendix D.2). The behavioral teacher distributes supervision across several plausible continuations, and the results show this coverage-versus-concentration trade-off.

Fixed B500 outputH-BNDConditionalCore JP-BND
Hit@10 (%)1.360.160.34
Hit@20 (%)1.620.360.64
Hit@500 (%)3.828.6010.66
SID NDCG@100.0076330.0006960.001539

The head-ranking decline is already present with multi-parent conditional supervision. Adding joint normalization improves every reported fixed-list cutoff, MRR, and SID NDCG over Conditional, although H-BND retains the stronger head ranking (Appendix D.2, Tables 14–15).

Our intended use is candidate retrieval under a fixed budget of 500 SID paths. Full-SID Hit@500 measures target-path coverage, and Expected Item separately measures its realization under a 500-item budget. For a smaller search budget, we also reran decoding: at B=20, Core reaches 1.94% versus 1.28% for H-BND, a gain of +0.66 pp [0.24, 1.06] (Section 4.3, Table 4). This changes pruning at every depth, whereas fixed-list Hit@20 only truncates an existing B500 output. We retain both evaluations because they answer different retrieval questions.

W3 / Q2. Why does performance decrease on Dynamics?

The support audit points to a mismatch between the behavioral teacher's evidence and the Dynamics targets. Using the same training graph and support definition, target L1L2 support covers 31.38% of Delayed, 31.16% of Delayed held-out, but only 0.93% of Dynamics. The Dynamics losses concentrate among the 9,907 instances whose target prefixes lack graph support (Section 5.2; Appendix E.5).

Our interpretation is that redistributing probability toward graph-supported alternatives helps when those alternatives match the evaluation targets, but can move probability away from unsupported targets. The observed progression is consistent with this explanation: Dynamics Full-SID changes from 1.98% to 1.41% to 1.25% for H-BND, Conditional, and Core (Section 5.1, Table 6). The larger decrease occurs when multi-parent behavioral supervision is introduced, followed by a smaller decrease with joint normalization. This is a descriptive association, not a causal attribution. The revised manuscript reports the support analysis in Section 5.2 and Appendix E.5, with all three population results retained.

AuthorsOpenReview ↗

Training Efficiency, Public Transfer, and BEAR Comparison

W4 / Q3. Is the training cost justified under a comparable budget?

We reduced Core JP-BND's training cost and confirmed that its Delayed advantage persists under an equal-time comparison. Batched prefix scoring, shared prompt computation, and GPU candidate processing reduced training time from 46.84 to 7.81 hours (6.0×) for 3,519 updates on eight H20 GPUs. The data, Stage 2 initialization, training objective, candidate selection rule, and inference remained unchanged. On eight fixed examples, relative differences from the original serial scorer were 1.97 × 10⁻⁴ in joint loss and 7.27 × 10⁻⁵ in gradient norm, both below the prespecified 10⁻³ tolerance (Appendix A.5).

We independently trained H-BND from the same Stage 2 checkpoint and training data for 7.81 hours, completing 10,653 updates with repeated data passes and a time-budgeted warmup/cosine schedule. Only the final checkpoint was evaluated.

MethodHoursUpdatesDelayed Full-SID (%)Expected Item (%)
H-BND, original budget2.353,5193.822.14
H-BND, equal time7.8110,6534.122.58
Core JP-BND (efficient implementation)7.813,51910.643.80

At equal training time, Core retains +6.52 percentage points (pp) Full-SID [5.68, 7.36] and +1.22 pp Expected Item [0.64, 1.80] on Delayed. Dynamics favors H-BND (Core−H-BND: −0.80 pp [−1.04, −0.56] Full-SID and −0.691 pp [−0.882, −0.503] Expected Item). Retention differences have intervals including zero. This comparison concerns student training time, not total pipeline cost. Section 4.2 and Appendix A.5 (Tables 8–9) report the results and separately document available reference/teacher preparation costs.

Q4. Why are the public-domain gains smaller?

The public experiments test transfer to a different architecture and training pipeline, not a replication of the industrial setting. The industrial Qwen3-0.6B model undergoes SID-token alignment and content-to-SID training, whereas public TIGER uses a T5 encoder–decoder with codebook-initialized SID embeddings and neither stage. The industrial base is hard-gated H-BND, while public models use dense A2. Nonempty joint teacher support covers 84.43% of industrial training instances versus 62.40% on Books. These differences may contribute to the smaller gains, but are not individually controlled explanations (Appendices A.3 and F.2–F.3).

Within each public domain, Conditional and Core share initialization, training data, the 3,295-update budget, behavioral support, candidate selection procedure, and inference. They differ in conditional versus joint normalization.

DomainConditional Full-SID (%)Core Full-SID (%)Δ Full-SID (pp, 95% CI)Δ Expected Item (pp, 95% CI)
Books2.85022.8914+0.0412 [0.0198, 0.0629]−0.0118 [−0.0263, 0.0023]
Home4.21004.2357+0.0256 [0.0032, 0.0478]−0.0734 [−0.0918, −0.0541]

Joint normalization adds Full-SID coverage in both domains, although the gains are modest (Section 4.4, Table 5; Appendix F.3, Table 26). Expected Item also depends on SID collisions, returned group order, and the fixed item budget.

W5 / Q5. Comparison with the released BEAR loss

We transferred BEAR's released CE-reweighting objective to TIGER on both Books and Home. The implementation preserves legal-child normalization, detached thresholds and weights, and the fallback rule. Within each domain, BEAR uses A0's architecture, initialization, training data, and 3,295-update budget, without adding BPA, BND, or JP supervision. We set the training cutoff to K=500 to match the beam (released default: 10), with other hyperparameters fixed without tuning.

DomainMethodFull-SID Hit@500 (%)Expected Item@500 (%)
BooksA0: AR2.36010.9690
BooksA2: AR+BPA+BND2.73851.1358
BooksBEAR loss on TIGER1.98610.9558
BooksCore JP-BND2.89141.2126
HomeA0: AR3.9972.778
HomeA2: AR+BPA+BND4.0162.861
HomeBEAR loss on TIGER3.1291.890
HomeCore JP-BND4.2362.809

On Books and Home, Core exceeds the released BEAR loss in Full-SID by +0.9053 and +1.1070 pp, respectively. Expected Item improves by +0.2568 and +0.9190 pp, respectively (Section 4.4, Table 5; Appendix F.3, Table 27).

This evaluates BEAR's released training loss, not its original end-to-end system. The industrial BEAR-style and SmartGR-style comparisons are separate adaptations (Appendix D.5.2; Section 4.2, Table 3). Each public domain has 1,046,262 paired evaluation instances; intervals use 10,000 paired bootstrap draws for one training seed. The revised manuscript reports these comparisons in Section 4.4 and Appendix F.3.

Thank you for the additional experiments. However, my concerns remain unresolved for three reasons:

  1. Although JP-BND improves Hit@500, Hit@10 drops from 1.36% to 0.34%, and nDCG@10 drops from 0.007633 to 0.001539 compared with H-BND. These substantial declines raise concerns about its practical value for recommendation.
  2. The authors still do not compare against the original implementations of SmartGR and BEAR. The transferred BEAR loss even performs worse than the basic AR baseline on both public datasets.
  3. JP-BND performs worse on Dynamics (1.25% vs. 1.98%). Its improvements on Amazon Books and Home are also minimal, with decreases in Expected Item. Therefore, I am not convinced that the proposed method offers a meaningful advantage over existing approaches. I maintain my rating of 1
AuthorsOpenReview ↗

Follow-up: retrieval objectives, BEAR control, and Dynamics

Thank you for the follow-up. We clarify the retrieval objective and report a new control isolating the effect of BEAR's reweighting. Manuscript references below refer to the revised PDF.

1. Hit@500 evaluates candidate retrieval, not a final top-10 display list

Published industrial architectures retrieve hundreds of candidates before ranking, as in YouTube [1]. Our candidate retrieval objective and B500 budget are specified in Section 4.1. The reported Hit@10 and nDCG@10 declines show weaker head ranking within the same B500 list. They do not negate the improvement in coverage of that candidate pool.

On Delayed, Core improves Full-SID@500 from 3.82% to 10.66% over H-BND. Expected Item@500 also rises from 2.14% to 3.90%, +1.76 pp [1.18, 2.34], under the same SID mapping, decoder, and 500-item budget (Section 4.2, Table 3). Thus, the benefit extends beyond SID coverage to expected item credit under the stated budget.

The benefit is not confined to B500. With decoding rerun at B=20, H-BND, Conditional, and Core achieve 1.28%, 1.60%, and 1.94% Full-SID, respectively. Core gains +0.66 pp [0.24, 1.06] over H-BND and +0.34 pp [0.10, 0.60] over Conditional (Section 4.3, Table 4 and Appendix D.2, Tables 12–13). This is actual B20 decoding, not truncation of a B500 list. We report both because candidate coverage and ranking within a fixed list are distinct objectives.

2. Comparison with BEAR

We verified our implementation against BEAR's official released code and followed its loss computation, including legal-child softmax, top-K thresholds and fallback, weight normalization, and stop-gradient treatment.

The original BEAR system uses a Llama-based model to generate item-title tokens, whereas our experiments use TIGER to generate three-level SIDs. We therefore transferred the released loss to our fixed-SID setting, using K=500 to match our decoding beam rather than the official default of 10. We used λ=0.25, following the official GitHub example.

To examine the transferred loss, we additionally tested λ=0, which disables BEAR's reweighting while retaining the other loss settings.

Home TIGERFull-SID@500 (%)Expected Item@500 (%)
A0: AR3.99692.7783
BEAR transfer (λ=0)2.98981.6354
BEAR transfer (λ=0.25)3.12871.8899

BEAR's reweighting improves both metrics over the λ=0 control, although the transferred objective remains below A0. The remaining gap also involves differences in normalization and loss reduction, rather than reweighting alone.

Our comparison follows BEAR's released loss implementation under a shared SID protocol, rather than reproducing its original end-to-end system.

3. Dynamics tests sensitivity to behavioral change

Dynamics uses a broader behavioral window and users disjoint from training (Section 4.1 and Appendix C.5). Under the same graph and support definition, target L1L2 support is 31.38% on Delayed, 31.16% on Delayed held-out, and 0.93% on Dynamics (Section 5.2 and Appendix E.5).

This supports the interpretation that training-graph supervision is less useful when evaluation targets lack behavioral support: probability can move toward supported alternatives instead. Dynamics Full-SID changes from 1.98% to 1.41% to 1.25% for H-BND, Conditional, and Core. The decrease from H-BND to Conditional is larger than the additional decrease from Conditional to Core. The support audit provides an explanation consistent with these results, not an isolated causal test.

Public transfer likewise separates the endpoints. Against Conditional, Full-SID gains are +0.0412 pp on Books and +0.0256 pp on Home. Expected Item has no resolved difference on Books and decreases on Home. Against A2, Books improves on both endpoints, whereas Home gains Full-SID but loses Expected Item (Section 4.4 and Appendix F.3, Table 26). These results support transfer of the normalization effect, not uniform improvement across all objectives.

Our central result is an additional retrieval benefit from joint behavioral supervision: +2.30, +2.44, and +2.47 pp Full-SID over Conditional across three Delayed held-out training seeds (Appendix D.1). These results establish a repeatable benefit for the stated retrieval task, alongside the reported ranking and Dynamics trade-offs.

Reference

[1] Covington et al. Deep Neural Networks for YouTube Recommendations. RecSys 2016, Section 2.

Reviewer_ihXTOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper's motivation is that SID-based generative recommenders generally use beam search at inference, which keeps only B live prefixes and ranks them by their joint (cumulative) score. The authors argue that during training, however, the (behavioral) supervision of each level is trained separately within each parent prefix, which creates a training–search mismatch. They propose JP-BND, a behavioral supervision that distills a teacher derived from an item-to-item behavioral graph over the joint L1L2 prefixes that compete under the current student. They show experiments on an industrial dataset with temporal splits and on Amazon Books/Home.

Critical strengths

S1. Several experiments show that the method improves performance over baselines, and the evaluation is carefully controlled. The experiments look well-designed and credible.

Critical weaknesses

W1. Clarity and complexity.

  • The paper is very difficult to understand, not well organized.
  • The abstract's description of the method is hard to parse.
  • The notation is scattered through section 3 making it really hard to follow.
  • It introduces BPA but only covers it in the appendix.
  • It is unclear if all BND, BPA, and H-BND are prior work or contributions.
  • The method looks overly engineered. Core JP-BND combines four losses (AR, BPA, BND, L_J) and the safety-augmented variant six. All coefficients are fixed, and the paper gives no intuition, sensitivity analysis or study of how the losses complement or interact.
  • It is also very costly to train. It requires several models trained in sequence. On top of that comes a 19.92× training overhead over H-BND, and that figure excludes the reference models.
  • This matters because a practitioner cannot tell which parts are needed, or whether the cost is justified.

W2. It is unclear to what to attribute the gains.

  • For the target path, AR already maximizes log p(a) + log p(b|a) = log p(a,b), the score beam search ranks. As I understand it, the mismatch is not that L2 "can be locally preferred within its parent and still be pruned because the resulting joint prefix has insufficient cumulative probability", since this cumulative (joint) probability is being directly optimized during training, not the "locally preference". The mismatch occurs due to beam search filtering on partial scores: at depth 1 it keeps the top-B parents by log p(a) alone, while AR optimizes p(a,b).
  • JP-BND doesn’t act on that step. Its candidate set comes from the student’s own top-B L1 frontier, and the gold parent is never inserted, so a target pruned at depth 1 gets no direct signal from L_J.
  • Most of the gain over H-BND (+4.78 of +6.84 points) already comes from the conditional control, i.e., from more behavioral supervision without joint normalization.
  • Prior methods appear only as “-style” adaptations, and the only public dataset (Amazon) experiments include neither them nor the conditional control.
  • The gains fall where they matter least. Head ranking gets worse than H-BND with Tables 10–11 showing this at smaller K.
  • The evidence does not isolate the paper’s central claim, that aligning supervision with beam-search competition and filtering is what improves retrieval.
Questions for authors
  1. Since AR already maximizes log p(a, b), is the mismatch the depth-1 filtering on log p(a) alone rather than the within-parent normalization of L2? If so, how does JP-BND address it, given that the gold parent is never inserted into Ω^θ?
  2. How often is the target L1 pruned at depth 1 versus depth 2, for H-BND and for JP-BND?
  3. How does the conditional control perform with the BND L1 term applied to all instances rather than only the 21.55% hard examples?
  4. How does the conditional control perform with the L2 softmax restricted to legal children?
  5. What is the expected use of the model? Why is Hit@500 the primary metric instead of the usual K=10/20?
  6. Which of BND, BPA and H-BND are prior work, and which are contributions of this paper?
  7. How does this method compare with prior-work and baselines on public datasets?
5 comments
AuthorsOpenReview ↗

Training–Search Alignment and New Ablation Results

Thank you for recognizing the controlled evaluation. We have completed the requested Dense-L1 control and address each question below. Equation and table numbers refer to the revised manuscript.

Q1. What mismatch does JP-BND address when AR already optimizes joint likelihood?

AR already optimizes the observed target's joint likelihood. JP-BND changes the additional behavioral soft supervision. The conditional control aligns child distributions separately within each parent. Joint supervision also aligns how much probability mass is assigned to each competing parent. For the same candidate set, Eqs. 17–18 give LJ=Lcond+LmargL_J = L_{\mathrm{cond}} + L_{\mathrm{marg}}.

A common shift to the joint scores under one parent cancels in its conditional softmax but changes its mass relative to other parents. The joint objective supervises this difference. AR is present in both arms.

At depth 1, a target parent outside the student frontier receives no direct target-prefix term from LJ\mathcal{L}_J. AR still supervises that target. JP-BND operates on the surviving parents and their legal L2 children, where beam search compares cumulative scores across parents. This is the competition domain addressed by the added supervision.

Replacing Lcond\mathcal{L}_{\mathrm{cond}} with LJ\mathcal{L}_J raises Delayed Full-SID from 8.60% to 10.66%, +2.06 pp [1.56, 2.56] (Section 5.1, Table 6; Appendix E.1, Table 19). The three Delayed held-out gains are +2.30, +2.44, and +2.47 pp, each with a positive paired interval (Appendix D.1). On the same 107 active instances and identical frozen candidates, parent-marginal KL falls from 2.6324 to 2.2848 nats, while conditional KL stays at approximately 1.933 (Section 5.2, Figure 2(a); Appendix E.2, Table 20). Thus the additional joint objective improves retrieval alongside better alignment across parents.

Q2. How often is the target first pruned at L1 or L2?

The native traces give the following counts on all 5,000 Delayed instances (Section 5.2; Appendix E.3, Table 21).

ModelFirst pruned at L1First pruned at L2L2 pruning among L1 survivorsFull-SID hits
H-BND1,492 (29.84%)2,826 (56.52%)80.56%191
Conditional1,496 (29.92%)1,781 (35.62%)50.83%430
Core JP-BND1,431 (28.62%)1,784 (35.68%)49.99%533

The first two percentage columns use all 5,000 instances. The third uses each model's own L1 survivors: 3,508, 3,504, and 3,569, respectively.

The largest immediate L2 improvement occurs from H-BND to Conditional. Core's further gains also occur later in decoding: 111 of its 131 Full-SID rescues over Conditional already survive immediate L2 pruning under both models (Section 5.2; Appendix E.4). Cumulative prefix scores continue to contribute to competition among complete paths.

Q3. What happens when base L1 BND is applied to all eligible instances?

To evaluate your suggestion, we introduced a new control, Dense-L1 Conditional, which applies the base L1 BND term to all instances with a nonempty L1 teacher, rather than only the 21.55% hard subset. The original L2 eligibility mask, multi-parent conditional objective, and loss averaging rule remain unchanged (Section 5.1; Appendix D.6, Table 18).

On Delayed, Dense-L1 Conditional improves Full-SID Hit@500 from 8.60% to 9.56% (+0.96 pp, 95% paired CI [0.56, 1.38]) and Expected Item@500 from 2.06% to 2.72% (+0.66 pp, [0.34, 0.98]). Core JP-BND achieves 10.66% and 3.90%, outperforming Dense-L1 Conditional by +1.10 pp [0.68, 1.52] and +1.18 pp [0.86, 1.50], respectively.

On Dynamics, Dense-L1 achieves 1.28% Full-SID versus 1.41% for the original Conditional (−0.13 pp [−0.23, −0.03]). On Retention, neither Full-SID nor Expected Item shows a statistically resolved difference from the original Conditional or Core.

Q4. What happens when the conditional L2 softmax is restricted to legal children?

Our Conditional control already uses a softmax restricted to legal L2 children under each parent. As defined in Eq. 16 (Section 3.2.3), for a parent aa with legal children C(a)\mathcal{C}(a),

= \frac{p_\theta(b \mid H_i,a)} {\sum_{b' \in \mathcal{C}i(a)} p\theta(b' \mid H_i,a)} = \frac{e^{u_b}} {\sum_{b' \in \mathcal{C}i(a)} e^{u{b'}}}.$$ The full-vocabulary normalization cancels when probabilities are renormalized over the legal children. Therefore, **the legal-child softmax you suggest is exactly what we use in our evaluated Conditional control**, which achieves **8.60% Full-SID Hit@500**, compared with **10.66% for Core JP-BND**. Both models retain the same base H-BND objective (Section 3.2.1, Eq. 6). The distinction is that Conditional normalizes L2 separately within each parent, whereas Core JP-BND jointly normalizes over legal L1L2 candidates across competing parents.
AuthorsOpenReview ↗

Retrieval Evaluation and Method Components

Q5. What is the intended use, and why is Hit@500 primary?

The model is a candidate retriever, with a fixed budget of 500 decoded SID paths. Full-SID Hit@500 measures whether the target's complete identifier enters that set. Expected Item measures the corresponding credit under the separate 500-item expansion budget and the fixed SID-to-item mapping (Section 4.1).

Fixed B500 outputH-BNDConditionalCore JP-BND
Hit@10 (%)1.360.160.34
Hit@20 (%)1.620.360.64
Hit@500 (%)3.828.6010.66

Core expands candidate coverage, while H-BND ranks the head of its B500 list better. Of the 411 Full-SID rescues over H-BND, 314 occur at ranks 101–500. Against the matched Conditional model, Core improves every reported fixed-list cutoff, MRR, and SID NDCG (Section 4.3; Appendix D.2, Tables 14–15).

Changing the search budget is a separate evaluation. With decoding rerun at B=20, H-BND, Conditional, and Core reach 1.28%, 1.60%, and 1.94% Full-SID, respectively. Core exceeds H-BND by 0.66 pp [0.24, 1.06] and Conditional by 0.34 pp [0.10, 0.60] (Section 4.3, Table 4; Appendix D.2, Tables 12–13). We report both evaluations: B500 coverage for the main retrieval task, and fixed-list ranking and actual beam-width sensitivity for smaller output settings.

Q6. Which of BND, BPA, and H-BND are prior work, and which are contributions of this paper?

BPA, BND, and H-BND are all defined and evaluated in this paper. They serve as our base training components and internal controls, while joint-prefix distillation is the central methodological contribution. AR is the standard autoregressive objective.

As summarized in Section 3 (Table 2), the training objectives have the following structure:

  • A0 = AR: Standard autoregressive training.
  • A1 = AR + BPA: Behavioral Prefix Alignment (BPA) aligns L1/L2 prediction states with target SID-token representations through contrastive supervision.
  • A2 = AR + BPA + BND: Behavioral-Neighbor Distillation (BND) introduces graph-derived soft supervision at L1 and target-parent L2.
  • H-BND = AR + BPA + hard-gated BND: BND is applied only to hard examples selected using a frozen A0 reference.
  • Conditional = H-BND + Lcond\mathcal{L}_{\mathrm{cond}}: Extends behavioral supervision to multiple parents, with separate L2 normalization within each parent.
  • Core JP-BND = H-BND + LJ\mathcal{L}_J: Replaces the conditional objective with joint normalization over competing L1L2 prefixes, additionally aligning probability mass across parents.

Conditional and Core JP-BND are parallel, matched training variants, not sequential stages. All student models are trained independently from the same Stage 2 checkpoint. Frozen A0 and H-BND reference models provide the hard-example mask and offline teacher support, respectively.

The core contribution is joint behavioral supervision over competing prefixes and its evaluation against the matched multi-parent conditional control, rather than contrastive alignment or graph distillation alone. The optional Boundary and Anchor losses are not required for the core method: Delayed Full-SID Hit@500 is 10.66% for Core and 10.62% with both auxiliaries (Section 4.2, Table 3; Appendix D.3, Table 16).

The revised manuscript defines BPA at its first mention and summarizes the training objectives and their roles in Section 3 (Table 2). Optional auxiliaries are detailed in Appendix D.3.

AuthorsOpenReview ↗

Public Baselines

Q7. How does JP-BND compare with prior work and baselines on public datasets?

We have completed two additional public-domain comparisons: a transfer of the released BEAR loss to Amazon Books and Home, and matched conditional controls on Amazon Books and Home. These experiments evaluate performance against a prior method and isolate the additional contribution of joint-prefix normalization, respectively.

1. Comparison with the released BEAR loss. We transferred BEAR's released CE-reweighting objective to TIGER on Books and Home, preserving its legal-child normalization, detached thresholds and weights, and fallback rule. Within each domain, BEAR shares the architecture, initialization, training data, and 3,295-update budget with A0, without adding BPA, BND, or JP supervision. We set the training cutoff to K=500 to match the decoding beam (released default: 10), keeping other hyperparameters fixed without tuning. All methods use the same domain-specific SID mapping, trie, and inference budgets (Section 4.4, Table 5; Appendix F.3).

DomainMethodFull-SID Hit@500 (%)Expected Item@500 (%)
BooksA0: AR2.36010.9690
BooksA2: AR+BPA+BND2.73851.1358
BooksBEAR loss on TIGER1.98610.9558
BooksCore JP-BND2.89141.2126
HomeA0: AR3.9972.778
HomeA2: AR+BPA+BND4.0162.861
HomeBEAR loss on TIGER3.1291.890
HomeCore JP-BND4.2362.809

On Books and Home, Core JP-BND outperforms the released BEAR loss in Full-SID Hit@500 by +0.9053 and +1.1070 percentage points (pp), respectively, with positive paired 95% confidence intervals in both domains. Expected Item also improves by +0.2568 and +0.9190 pp, respectively (Section 4.4, Table 5; Appendix F.3, Table 27).

This evaluates BEAR's released training loss, not its original end-to-end system. The industrial BEAR-style and SmartGR-style comparisons are separate adaptations (Appendix D.5.2; Table 3).

2. Matched conditional controls on Books and Home. As in our Delayed experiments (Section 5.1, Table 6), we trained matched multi-parent conditional controls on both public datasets to isolate the benefit of joint normalization. The distinction between conditional and joint supervision is formalized by the KL decomposition in Section 3.2.3 (Eqs. 17–18). Conditional and Core JP-BND share the same initialization, training instances, update budget, offline behavioral support, candidate construction rule, and inference protocol. Both use dense A2 as their base, without the industrial hard-example mask.

DomainConditional Full-SID (%)Core Full-SID (%)Core−Conditional Full-SID (pp, 95% CI)Core−Conditional Expected Item (pp, 95% CI)
Books2.85022.8914+0.0412 [0.0198, 0.0629]−0.0118 [−0.0263, 0.0023]
Home4.21004.2357+0.0256 [0.0032, 0.0478]−0.0734 [−0.0918, −0.0541]

Joint normalization consistently improves Full-SID coverage beyond multi-parent conditional supervision in both public domains. The additional gains are modest but have positive paired confidence intervals (Section 4.4, Table 5; Appendix F.3, Table 26).

Expected Item also depends on the fixed SID mapping, collision-group sizes, returned SID order, and item expansion budget (Appendix C.7). Since the generator does not rank individual items within a shared SID, improved SID-path coverage does not necessarily translate into higher Expected Item. The Books difference includes zero, while Home shows a lower Expected Item for Core.

Each domain uses 1,046,262 paired evaluation instances. All confidence intervals use 10,000 paired bootstrap draws over instances from one training seed. These results are reported in the revised manuscript (Section 4.4 and Appendix F.3).

Taken together, these experiments provide two complementary pieces of evidence: the released BEAR objective transfer establishes a direct public-domain comparison with a prior training method, while the matched conditional controls demonstrate the additional SID-path retrieval benefit of joint-prefix normalization beyond multi-parent behavioral supervision.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Manuscript Revision Update

Dear Reviewers,

Thank you for your constructive feedback. We have uploaded a revised manuscript incorporating the additional experiments and analyses discussed in our responses.

We revised the Abstract, Introduction, and Related Work to clarify the motivation for additional behavioral supervision, the distinction between conditional and joint normalization, and the contribution of JP-BND. We also refined the methodology to make the training objectives and their relationships clearer.

The revised manuscript incorporates equal-time training comparisons, matched conditional controls on Amazon Books and Home, additional ablations, and further mechanism analyses. We also clarify the observed trade-offs between SID-path coverage and other retrieval outcomes.

All section, equation, table, and appendix references in our responses have been updated to match the revised manuscript.

We appreciate your feedback and hope these revisions address the main concerns raised during the review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.