Learning to Navigate Fixed Semantic IDs with Joint-Prefix Behavioral Distillation

Semantic identifiers (SIDs) turn generative recommendation into an irreversible path-search problem: once an early prefix is pruned, the target item becomes unreachable. We identify a training–search mismatch in SID retrieval: behavioral L2 supervision is normalized separately within each L1 prefix (parent branch), whereas beam search compares cumulative scores across L1L2 prefixes rooted at different L1 codes. We propose Joint-Prefix Behavioral-Neighbor Distillation (JP-BND), which projects training-partition behavioral evidence onto joint L1L2 prefixes and distills it over the joint prefixes that compete under the current student, without changing the SID mapping or inference procedure. A joint-KL decomposition separates within-parent conditional alignment from cross-parent marginal alignment. On Delayed development, Full-SID Hit@500 increases from 3.82% for the Hard-example BND (H-BND) baseline to 8.60% with a matched multi-parent conditional control and 10.66% with core JP-BND. Fixed-candidate KL diagnostics show that conditional training primarily reduces within-parent mismatch, whereas joint normalization additionally reduces cross-parent marginal mismatch. A 5,000-instance constrained beam-search audit further reveals selective cross-parent reordering rather than a uniform lift in target-prefix margins. The gains remain stable across three training seeds and a separate Delayed held-out evaluation, persist at every tested inference beam width from 20 to 500, and transfer at the SID-path level to Amazon Books and Home public datasets. Together, these results show that aligning behavioral supervision with the prefixes that actually compete during beam search improves generative retrieval over fixed SID spaces.