acceptodds
Under review as a conference paper at ICLR 2027

Learning Multiresolution Relevance for Hierarchical Generative Retrieval

29 views

Abstract

Generative retrieval with semantic identifiers (SIDs) makes successive decisions over a document hierarchy. Relevant documents for the same query may share coarse prefixes and diverge at finer depths, with branching patterns varying across queries. These paths reveal how relevance is distributed across successive refinements, yet standard full-SID supervision treats them as separate training targets. To make this allocation explicit, we formulate multiresolution relevance as consistent conditional distributions induced by a single document-level relevance measure across the SID hierarchy. We introduce RARS, Resolution-Aligned Relevance Supervision, which uses the resulting refinement-level distributions to supervise a shared query representation. RARS aggregates document relevance over prefixes and trains a prefix-conditioned predictor to allocate relevance among sibling branches. All relevance-bearing children participate in local competition, and each local loss is weighted by the relevance mass reaching its parent. This objective trains the query encoder to capture both the coarse structure shared by relevant documents and their finer branch allocations. The predictor is discarded after training, preserving standard autoregressive retrieval at inference. Experiments on three multilingual ESCI locales show consistent improvements over matched full-SID training under autoregressive decoding. RARS also outperforms grouped soft-target, decoder soft-target, and sampled-tree supervision under a common retrieval rule. The gains persist across alternative identifier structures and relevance definitions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Generative Retrieval, Semantic Identifiers, Hierarchical Relevance Modeling, Multi-Resolution Retrieval, E-Commerce Search

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_r29xOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RARS, a multiresolution relevance supervision framework for hierarchical generative retrieval (GR). It projects document-level relevance into conditional distributions over ID branches and introduces an auxiliary predictor to supervise a shared query representation. The predictor is removed during inference, preserving standard autoregressive decoding. Experiments on three multilingual ESCI datasets demonstrate consistent improvements over matched baselines and alternative supervision objectives.

Critical strengths
  1. The paper addresses a limitation of full-SID supervision in GR. Modeling how relevance is distributed across hierarchical branches is well motivated, particularly for queries with multiple relevant documents.

  2. The proposed formulation is mathematically coherent and provides a natural way to represent relevance at different SID resolutions. The auxiliary prediction head also makes the approach practical, as it introduces no additional prediction head at inference.

  3. The experimental evaluation includes several useful controlled comparisons. The results under matched autoregressive decoding and alternative supervision objectives help isolate the contribution of RARS. The improvements are consistent across three locales, and the additional sensitivity analyses provide useful insights into the method’s behavior.

Critical weaknesses
  1. The proposed hierarchical objective is closely related to relevance-weighted full-SID supervision. The main difference seems to come from the auxiliary prediction head and how it affects the query representation. The additional novelty beyond existing multi-positive supervision methods is not entirely clear to me.
  2. All three main datasets come from the ESCI product-search benchmark. I wonder why the authors did not consider traditional document retrieval benchmarks with graded relevance judgments, such as Robust04 [1], GOV2 [2], or ClueWeb09-B[3]. The current evaluation makes it difficult to tell whether the proposed training method generalizes beyond product search.

References:

[1] Ellen M. Voorhees. 2004. Overview of the TREC 2004 Robust Retrieval Track. In TREC. 69–77.

[2] Charles L.A. Clarke, Nick Craswell, and Ian Soboroff. 2004. Overview of the TREC 2004 Terabyte Track. In TREC 2004.

[3] Charles L.A. Clarke, Nick Craswell, and Ian Soboroff. 2010. Overview of the TREC 2009 Web Track. In TREC 2009. 20–29.

Questions for authors
  1. How much additional training time and GPU memory does RARS require compared with the matched baseline? This would help assess whether the performance improvements justify the additional training complexity.

  2. Could the authors clarify why the NQ320K variant in Table 6 is labeled RARS, given that it shares the same trained parameters as CaLIR and the comparison mainly evaluates the effect of retrieval scoring?

7 comments
AuthorsOpenReview ↗

Official Response to Reviewer r29x (1/3)

Thank you for your recognition of our work and for offering constructive feedback. We respond to your comments below. We have marked the corresponding changes in blue in the revised manuscript.

Q1: The proposed hierarchical objective is closely related to relevance-weighted full-SID supervision. The main difference seems to come from the auxiliary prediction head and how it affects the query representation. The additional novelty beyond existing multi-positive supervision methods is not entirely clear to me.

A1: Thank you for pointing out the connection between our hierarchical objective and relevance-weighted full-SID supervision.

(1.1) The core contribution of RARS is to train a shared query representation to capture where relevant documents share a path in the SID tree, where their paths diverge, and how relevance is distributed among the child branches at each split. Our Base model aligns the query’s latent representation with multiple positive category prototypes at each level of the category hierarchy. RARS retains this multi-positive supervision and adds supervision for SID branch selection. At each parent prefix with relevant descendants, the auxiliary head uses the query representation and the current prefix to predict the relative relevance of its child branches. It compares the relevance-bearing branches with high-scoring valid sibling negatives in the same softmax.

Consider a query with two relevant documents whose relevance weights are 3 and 1. Their SIDs share the first two tokens and diverge at the third. At each of the first two steps, the target assigns all relevance mass to the shared continuation. At the third step, the target allocates that mass to the two child branches in a 3:1 ratio. RARS uses these prefix-conditioned distributions as soft targets to train the query encoder.

(1.2) To assess the contribution of the full branch distribution, we compared sampling a single target child from the relevance distribution with learning the full distribution directly, using the same auxiliary prediction head and retrieval rule. As shown in Table 1, learning the full distribution improves all five retrieval metrics across US, ES, and JP.

Table 1. Comparison of branch sampling and full branch distributions with the same auxiliary prediction head

DatasetTraining ObjectiveRecall@5Recall@10Recall@100NDCG@10NDCG@100
ESCI-USRelevance-weighted branch sampling8.8914.0139.1812.5520.29
Explicit branch distributions (RARS)9.2914.3639.6213.0020.82
Gain (pp)+0.40+0.35+0.44+0.45+0.53
ESCI-ESRelevance-weighted branch sampling6.6311.5135.2813.6921.19
Explicit branch distributions (RARS)6.8711.8035.5514.1921.53
Gain (pp)+0.24+0.29+0.27+0.50+0.34
ESCI-JPRelevance-weighted branch sampling6.5210.5531.8112.7919.55
Explicit branch distributions (RARS)6.7110.8932.0213.2319.80
Gain (pp)+0.19+0.34+0.21+0.44+0.25
AuthorsOpenReview ↗

Official Response to Reviewer r29x (2/3)

Q2: All three main datasets come from the ESCI product-search benchmark. I wonder why the authors did not consider traditional document retrieval benchmarks with graded relevance judgments, such as Robust04 [1], GOV2 [2], or ClueWeb09-B[3]. The current evaluation makes it difficult to tell whether the proposed training method generalizes beyond product search.

A2: Thank you for suggesting that we evaluate RARS on traditional document retrieval benchmarks. We initially chose ESCI because its US, ES, and JP locales share a graded relevance annotation scheme and include queries with multiple positive documents at different relevance grades. This setting allows us to evaluate SID branch supervision across languages under a common annotation protocol.

To evaluate RARS beyond product search, we conducted experiments on a 200K-document subset of Robust04, GOV2–WebAP, and ClueWeb09-B–ICAT. Obtaining the full GOV2 and ClueWeb09-B collections requires completing the research licensing and distribution procedures of the University of Glasgow and Carnegie Mellon University, respectively. For these experiments, we constructed two subsets from publicly released document text. For GOV2–WebAP, we extracted text from the WebAP archive released by UMass and deduplicated the records by their original GOV2 document IDs, yielding 6,377 documents. For ClueWeb09-B–ICAT, we used the publicly released TREC-ICAT corpus and removed entries with empty text, yielding 26,917 documents. For both subsets, we matched the original document IDs to the corresponding graded TREC document-level relevance judgments.

Within each collection, Base, Grouped soft-target, and RARS use the same model backbone, SID mapping, training budget, and query splits. All three learned methods use AR-only retrieval. Grouped soft-target applies graded relevance at the full-SID level, providing a comparison for assessing the additional benefit of branch-level supervision for ranking and recall.

Table 2. Retrieval performance on graded document retrieval collections (%)

Collection / MethodR@5R@10R@100N@10N@100
Robust04 (200K)
BM2515.7825.4461.8436.2052.31
Base18.3729.1867.9541.5256.47
Grouped soft-target19.5130.8369.1342.8458.25
RARS21.2633.3572.4445.8661.37
GOV2–WebAP (6,377 docs)
BM2517.0631.8093.5956.5674.07
Base19.6834.3795.1260.2376.38
Grouped soft-target20.8435.9196.0761.8777.64
RARS22.6938.1797.3564.1479.03
ClueWeb09-B–ICAT (26,917 docs)
BM2511.9020.4980.6534.5056.08
Base14.3223.7184.4938.2759.48
Grouped soft-target15.5625.8385.9239.9460.85
RARS17.3828.2488.4642.8363.17

Relative to Grouped soft-target, RARS improves NDCG@10 by 3.02, 2.27, and 2.89 percentage points and Recall@100 by 3.31, 1.28, and 2.54 percentage points on Robust04, GOV2–WebAP, and ClueWeb09-B–ICAT, respectively. These results show that SID branch supervision can improve ranking and recall on the graded document retrieval subsets we evaluated.

AuthorsOpenReview ↗

Official Response to Reviewer r29x (3/3)

Q3: How much additional training time and GPU memory does RARS require compared with the matched baseline? This would help assess whether the performance improvements justify the additional training complexity.

A3: Thank you for asking us to quantify the training cost of RARS. RARS reuses the retriever’s query representation, predicts SID child branches with a low-dimensional auxiliary head, and limits the number of negatives at each prefix. We compared Base and RARS across the three ESCI locales, using one A100 80 GB GPU per run. Table 3 reports training time for 300 epochs. We measured peak allocated GPU memory over 100 optimizer updates after 20 warm-up steps, with the same effective batch size, batch order, numerical precision, and optimizer settings for both methods.

Table 3. Training cost and retrieval gains relative to the matched baseline

ESCI LocaleTraining Time (h)Time Change (%)Peak GPU Memory (GiB)Memory Increase (MiB)Δ N@10 (pp)Δ R@100 (pp)
US92.66 → 93.51+0.927.140 → 7.150+9.9+1.24+1.58
ES33.03 → 34.04+3.0614.985 → 14.994+9.1+1.22+0.71
JP40.77 → 41.45+1.6716.145 → 16.157+12.2+0.57+0.70

Arrows indicate Base → RARS. pp denotes percentage points.

RARS increases training time by less than 3.1% in all three locales and adds only about 9–12 MiB of peak allocated memory per GPU. Over 300 epochs, the additional training time was 0.85 hours on US, 1.01 hours on ES, and 0.68 hours on JP. Under the same AR-only retrieval rule, RARS improves both NDCG@10 and Recall@100 in all three locales.

Q4: Could the authors clarify why the NQ320K variant in Table 6 is labeled RARS, given that it shares the same trained parameters as CaLIR and the comparison mainly evaluates the effect of retrieval scoring?

A4: Thank you for pointing this out. We apologize for the ambiguity in our original description of the NQ320K comparison. CaLIR and RARS share the model parameters learned during SID tokenization and the resulting SID mapping. We train the two retrieval models separately using their respective objectives and evaluate both with the same SID constraints and beam width. The row labeled “RARS” in Table 6 reports the results of the retriever trained with the RARS objective.

Thank you again for the time and care you devoted to reviewing our manuscript. Your constructive feedback has helped us strengthen the paper. We hope our responses address your concerns, and we would be happy to clarify any remaining points.

Reviewer_54NbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. RARS adds an additional training objective to generative retrieval to explicitly optimize prediction at each node to produce the distribution of relevant items over valid children, as opposed to simply optimizing for a sampled one-hot target at each node and training on individual relevant query-item pairs.
  2. RARS designs an approach to add this training objective in addition to the standard autoregressive objective, adding an extra prediction head during training that applies updates to the query encoder and is then discarded after training.
Critical strengths
  1. Generative retrieval is usually trained using one-hot labels even though there is a relevance distribution over valid next tokens. RARS offers a clean approach to directly optimize for the relevance distribution at each node.
  2. RARS performs multiple ablations against each stage of training and each training objective, showing that each of them contributes to its final performance.
Critical weaknesses
  1. Writing/presentation can be improved. The manuscript unnecessarily invents jargon to describe its approach and experiments, making it harder to read. Figures can use polishing and would benefit from self-contained notation and descriptions.
  2. The main source of novelty seems to be the approach to train on soft-targets. But normalized multi-label targets themselves have been well explored in literature (Multilabel reductions: What is my loss optimizing [Menon et al]). Token-level supervision exists within generative retrieval as well, even if not in the exact normalized form (Listwise Generative Retrieval Models via a Sequential Learning Process [Tang et al]). The approach to use an auxiliary prediction head with hard negatives is the greater source of novelty.
Questions for authors
  1. Why does the soft-prediction objective benefit from being treated as an auxiliary objective during training, with an additional head and updates applied only to the query encoder? How did you arrive at this arrangement? I’m also curious as to why this head isn’t used for decoding / beam search.
  2. In table 2, AR-decoder soft target and Grouped soft-target seem to be optimizing mathematically equivalent objectives, also indicated in Eq 18. Why is there is a difference in performance between the two? Is it just implementation differences?
4 comments
AuthorsOpenReview ↗

Official Response to Reviewer 54Nb (1/3)

Thank you for your careful review and thoughtful suggestions! Your feedback has helped us strengthen the manuscript and guided our revisions. Below, we respond to your questions in detail.

Q1: About simplifying the terminology in the method and experiments and making the figures self-contained with clear notation and captions.

A1: Thank you for your constructive feedback.

(1.1) We revised the wording in the Method and Experiments sections and simplified some of the terminology to make the training procedure and experimental comparisons easier to follow.

(1.2) The updated Figures 1 and 2 each include a Notation box that defines the symbols used in the figure. We marked the relevant changes in blue so they are easy to locate in the revised manuscript.

Q2: About the novelty of RARS’s soft targets and auxiliary head with hard negatives.

A2: Thank you for pointing out these two precedents.

(2.1) We agree that normalized multilabel objectives and token-level relevance supervision are both well established. Menon et al. analyze multilabel objectives normalized by the number of positive labels. ListGR calibrates the tokens of document identifiers generated by beam search by weighting them according to the corresponding documents’ relevance grades.

RARS differs from these approaches in its unit of supervision. It supervises each branch decision conditioned on the query and the current SID prefix. Once beam search prunes a prefix, any relevant products under that prefix become unreachable in that search. We therefore aggregate the relevance weights of all labeled relevant products under the prefix by child branch to construct a target distribution for the next decision. A separate auxiliary head predicts this distribution from the shared query representation. Its softmax compares the relevant children with high-scoring valid siblings under the same prefix that have zero target weight. This trains the query encoder to support multiple relevant branches and distinguish them from high-scoring candidates that could displace them during beam search.

MethodSupervision unitTarget constructionTraining
Menon et al. (PAL-N)Label set per exampleGive each positive label equal weight, with weights summing to oneTrain a label classifier
ListGR (Tang et al.)Ranked document IDs and beam-search candidatesWeight tokens by relevance grade and calibrate ID rankingsListwise training followed by token- and sequence-level calibration
RARSNext branch given a query and SID prefixSum relevance weights by child and normalize within the prefixTrain the query encoder through an auxiliary head that scores relevant children and high-scoring valid siblings in one softmax

(2.2) All ESCI-US training comparisons and sibling-negative ablations use the same all-level retrieval rule.

Table 2. Training-objective comparisons and sibling-negative ablations on ESCI-US (%)

Training settingRecall@5Recall@10Recall@100NDCG@10NDCG@100
Soft targets over complete SIDs (Grouped soft-target)8.6113.6938.8812.2820.03
Branch-level soft targets on the decoder (AR-decoder soft target)8.7413.9339.0412.4620.36
Auxiliary head with one sampled child as the target (Sampled tree)8.8914.0139.1812.5520.29
Auxiliary head with the full branch-target distribution, using only relevant children as candidates8.4813.7738.6512.1819.89
Full branch-target distribution with up to 32 randomly selected valid sibling negatives8.7913.9339.0712.5720.28
RARS9.2914.3639.6213.0020.82

With the same auxiliary head, full branch targets raise NDCG@10 from 12.55% with a sampled child to 13.00%. With the head, targets, and negative budget matched, high-scoring negatives improve Recall@100 and NDCG@10 by 0.55 and 0.43 percentage points, respectively, over random selection.

AuthorsOpenReview ↗

Official Response to Reviewer 54Nb (2/3)

Q3: Why does the soft-prediction objective benefit from being treated as an auxiliary objective during training, with an additional head and updates applied only to the query encoder? How did you arrive at this arrangement? I’m also curious as to why this head isn’t used for decoding / beam search.

A3: Thank you for this perceptive question, which pinpoints a central design choice in RARS.

(3.1) Auxiliary Head Design and Gradient Routing

Our design follows from the structure of the SID tree. Multiple relevant documents for a query can share a coarse prefix and diverge into different child branches at deeper levels. To supervise the query encoder directly on this structure, we introduce an auxiliary prediction head that estimates how relevance is distributed among the children of a given SID prefix from the query representation. The branch loss updates the auxiliary head and the query encoder, while the full-SID generation objective continues to train the main decoder. At inference, the decoder uses the query representation learned under this branch supervision.

In Table 3, both Decoder-KL variants use the main autoregressive decoder to predict the branch soft targets. In E-only, the branch loss updates only the query encoder. In E+D, it updates both the query encoder and the decoder. Both variants train the decoder with the full-SID generation loss. E-only achieves higher NDCG@10 than E+D in all three locales.

The separate auxiliary head achieves higher NDCG@10 than Decoder-KL (E-only) when both settings use the same branch soft targets and valid children, and the query encoder is the only component of the retrieval backbone updated by the branch loss. When we retain the auxiliary head and stop the branch gradient from reaching the query encoder, the retrieval scores match those of Base at the reported precision. This comparison supports our rationale for applying branch supervision to the query representation used in autoregressive retrieval.

Table 3. Ablation of the auxiliary prediction head and gradient routing (AR-only retrieval, %)

Training SetupUpdatesUS N@10US R@100ES N@10ES R@100JP N@10JP R@100
Base—11.1237.4012.4334.1511.7730.89
Decoder-KL (E+D)E+D11.5137.7212.7934.3611.9531.06
Decoder-KL (E-only)E11.8638.1212.9934.4712.1631.26
RARS (all)E+H12.1938.7113.4334.7012.2131.43
RARS (all, stop-grad)H11.1237.4012.4334.1511.7730.89
RARSE+H12.3638.9813.6534.8612.3431.59

E, D, and H denote the query encoder, autoregressive decoder, and auxiliary head, respectively. Decoder-KL uses D to predict branch soft targets, and RARS uses H. “All” includes all valid children, and “stop-grad” detaches the query representation at the head input.

(3.2) Auxiliary Head Scoring During Beam Search

The main decoder can use the query representation learned through auxiliary branch supervision directly at inference. To test whether the auxiliary head can further improve beam search, we add its weighted conditional log probability to the autoregressive score before each pruning step. Table 4 shows that joint scoring yields limited additional retrieval gains and increases per-query latency. We therefore omit the auxiliary head from beam search by default, saving the cost of computing its scores at each step.

Table 4. Retrieval effectiveness and latency with auxiliary head scoring during beam search

ESCI LocaleN@10 (%)R@100 (%)AR Latency (ms)AR+H Latency (ms)Latency Increase (%)
US12.36 → 12.3838.98 → 39.00158.5186.0+17.4
ES13.65 → 13.6434.86 → 34.89182.6222.1+21.6
JP12.34 → 12.4831.59 → 31.65175.8197.0+12.1

Arrows indicate AR-only scoring (AR) → AR scoring with the auxiliary head (AR+H).

AuthorsOpenReview ↗

Official Response to Reviewer 54Nb (3/3)

Q4: In table 2, AR-decoder soft target and Grouped soft-target seem to be optimizing mathematically equivalent objectives, also indicated in Eq 18. Why is there is a difference in performance between the two? Is it just implementation differences?

A4: Thank you for pointing out the connection between Eq. (18) and Table 2. As you note, when both formulations use the same conditional predictions, normalization support, and loss aggregation, the relevance-weighted full-SID cross-entropy and the sum of conditional KL terms weighted by parent relevance mass differ only by a target entropy term that is independent of the model parameters. The two formulations therefore yield the same parameter gradients under these conditions.

The two controls in Table 2 optimize different total losses. Grouped soft-target weights the negative log-likelihood of each complete SID by its relevance. AR-decoder soft target retains the reference loss and adds a parent-weighted conditional KL term, with the decoder’s predictions normalized over all valid children at each prefix. These choices change the training gradients and can therefore lead to different retrieval results, consistent with the equivalence in Eq. (18).

In the submitted manuscript, Appendix B.5 (Full-path consistency and grouped supervision) derives this equivalence. The two training objectives are defined in the paragraph titled “Local targets on the autoregressive decoder” in Appendix C.4 (Training objectives and retrieval controls).

Thank you again for volunteering your time and expertise to review our manuscript. We sincerely appreciate the care you put into your review. We hope our responses address your concerns and that you will reconsider our manuscript in light of them. We would be happy to answer any further questions.

AuthorsOpenReview ↗

Kindly Reminder: Discussion Phase Closing Soon

Dear Reviewer 54Nb,

Thank you again for your thoughtful feedback, which helped us improve our paper. As the discussion period draws to a close, we hope you have a chance to review our response and additional results. If you feel they address your concerns, we would be grateful for your reconsideration of the paper. We would be happy to clarify any remaining points.

Best regards,

Authors

Reviewer_TMpFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RARS, a training objective for generative retrieval over semantic identifier (SID) hierarchies. It converts document-level relevance into consistent relevance targets at each hierarchy level and trains a shared query representation to allocate relevance among child branches. The auxiliary predictor is discarded at inference. Experiments cover three ESCI locales and a controlled NQ320K comparison.

Critical strengths

1, The method aligns supervision with the retrieval process. It makes relevance allocation across SID refinements explicit instead of treating each relevant document as an independent full path. 2, The paper tests several sources of robustness. Results are examined across SID constructions, relevance definitions, and retrieval-scoring variants.

Critical weaknesses

The method’s benefit may depend on the quality and semantics of the SID hierarchy. The paper tests alternative identifier structures, but it remains unclear when a hierarchy provides useful relevance structure rather than arbitrary partitions.

Questions for authors

What properties must an SID hierarchy have for resolution-aligned supervision to help, and when would arbitrary partitions undermine it?

2 comments
AuthorsOpenReview ↗

Official Response to Reviewer TMpF

We sincerely appreciate your constructive and insightful comments. We have highlighted the corresponding changes in blue in the revised manuscript.

Q1: What properties must an SID hierarchy have for resolution-aligned supervision to help, and when would arbitrary partitions undermine it?

A1: Thank you very much for your insightful question!

(1.1) RARS requires a hierarchy of nested SID prefixes. At each prefix, RARS aggregates the relevance of documents within each child branch to form a supervision target for the next branch decision. This supervision improves retrieval when query cues help the model identify the relevant branches.

For example, in the query “65 W USB-C charger,” “USB-C charger” can identify a coarse prefix shared by several relevant products, and “65 W” can help identify the child branches containing chargers with the requested wattage. An arbitrary partition can scatter documents that match the same query cue across branches near the top of the tree. If the query gives little guidance on which branches to follow, the supervision becomes less useful.

On ESCI-ES, we held the RQ-VAE prefix tree fixed and shuffled complete SID paths. Each shuffle starts from the original assignment. Within-L2 shuffles paths among documents sharing their first two codes, Within-L1 among those sharing their first code, and Global across all documents.

Illustrative example of SID reassignment

DocumentOriginal SIDWithin-L2 shuffleWithin-L1 shuffleGlobal shuffle
A(12, 7, 3, 9)(12, 7, 8, 4)(12, 9, 2, 6)(21, 5, 1, 8)
B(12, 7, 8, 4)(12, 7, 3, 9)(12, 7, 8, 4)(12, 9, 2, 6)
C(12, 9, 2, 6)(12, 9, 2, 6)(12, 7, 3, 9)(12, 7, 8, 4)
D(21, 5, 1, 8)(21, 5, 1, 8)(21, 5, 1, 8)(12, 7, 3, 9)

Under Within-L2, document A receives B’s original SID and retains the two-code prefix (12, 7). Under Within-L1, A receives C’s original SID and retains only its first code, 12. Under the global shuffle, A receives D’s original SID and its first code changes from 12 to 21.

Table 1. Retrieval performance under controlled SID reassignment

SID assignmentMethodR@5R@10R@100N@10N@100
Original RQ-VAEBase6.0310.4834.1512.4319.88
RARS6.4011.3834.8613.6520.84
Δ+0.37+0.90+0.71+1.22+0.96
Within-L2 shuffleBase5.8210.1832.6012.0519.20
RARS5.9510.4732.8612.2619.51
Δ+0.13+0.29+0.26+0.21+0.31
Within-L1 shuffleBase5.339.5230.8311.4618.23
RARS5.369.6030.8611.5518.30
Δ+0.03+0.08+0.03+0.09+0.07
Global shuffleBase5.068.8229.7610.9017.60
RARS4.908.6129.5710.7517.37
Δ−0.16−0.21−0.19−0.15−0.23

RARS shows the largest gains over Base with the original RQ-VAE SIDs. It improves all five metrics under both local shuffles, with larger gains when two prefix levels are preserved. Under the global shuffle, RARS scores below Base on all five metrics. The gains after local shuffling suggest that the remaining coarse prefixes still help the model identify relevant branches from query cues.

(1.2) To test whether the gains persist across SIDs of varying quality, we evaluated six additional SID construction methods on ESCI-ES. The resulting SIDs differ in retrieval quality, as reflected in the performance of Base under the same retrieval protocol.

Table 2. Retrieval performance across SID construction methods

SID constructionMethodR@5R@10R@100N@10N@100
RQ-KMeansBase5.679.8332.5511.6418.90
RARS5.8210.0332.8111.8219.13
Δ+0.15+0.20+0.26+0.18+0.23
PQBase6.2210.8235.0112.7820.30
RARS6.5311.2635.5813.1820.78
Δ+0.31+0.44+0.57+0.40+0.48
OPQBase5.8810.1633.3012.0819.31
RARS6.0710.4233.5812.3119.54
Δ+0.19+0.26+0.28+0.23+0.23
SimHashBase6.1210.6134.0812.4819.78
RARS6.3810.9634.5012.8120.15
Δ+0.26+0.35+0.42+0.33+0.37
VQBase6.4311.1235.5213.0420.61
RARS6.8411.6636.1813.5521.18
Δ+0.41+0.54+0.66+0.51+0.57
VQ-VAEBase5.559.6831.9811.4218.62
RARS5.669.8332.1611.5518.79
Δ+0.11+0.15+0.18+0.13+0.17

Base and RARS use the same SID mapping within each comparison. RARS improves all five metrics across all six constructions.

Thank you again for the time and care you devoted to reviewing our manuscript. We sincerely appreciate your thoughtful suggestions and hope our response addresses your concerns.

AuthorsOpenReview ↗

Kindly Reminder: Discussion Phase Closing Soon

Dear Reviewer TMpF,

As the discussion phase is coming to a close, we kindly ask if our responses have fully addressed your questions and concerns. We truly appreciate your time and effort in reviewing our work and look forward to receiving your valuable feedback.

Best regards,

Authors

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response

Dear all reviewers:

We sincerely thank all reviewers for their careful reading of our paper and for their constructive suggestions. We are encouraged by their recognition of the following strengths.

  • Motivation: Reviewer r29x found RARS well motivated, particularly for queries with multiple relevant documents.
  • Methodology: Reviewer 54Nb described RARS as a clean approach to learning relevance distributions. Reviewer r29x found the formulation mathematically coherent and highlighted the practical design of the training-only auxiliary head.
  • Experiments: All three reviewers highlighted strengths of the evaluation. Reviewer TMpF noted the breadth of the robustness checks, Reviewer 54Nb highlighted the ablations of the training objectives, and Reviewer r29x noted the useful controlled comparisons and consistent gains across all three ESCI locales.
  • Presentation: Reviewers TMpF and r29x found the paper clear and easy to follow.

We summarize the main additions in response to each reviewer’s comments.

  • Reviewer TMpF. We added controlled SID shuffles and comparisons across six alternative SID constructions to assess when the hierarchy supports effective supervision.
  • Reviewer 54Nb. We simplified the terminology, added Notation boxes to Figures 1 and 2, and clarified the relationship to prior objectives and Eq. (18). We also compared auxiliary-head designs, gradient routing, and the effect of using the head during beam search.
  • Reviewer r29x. We added graded document retrieval experiments on subsets of Robust04, GOV2, and ClueWeb09-B, reported training time and peak GPU memory, and clarified the NQ320K comparison.

All revisions in the paper are marked in blue font. We sincerely appreciate the reviewers' constructive suggestions and remain committed to further improving our work.

Below, we address each reviewer's specific concerns in detail. We extend our heartfelt thanks to all reviewers for their recognition and valuable feedback. We welcome further discussion and look forward to continued engagement. Thank you!

Sincerely yours,

Authors

AuthorsOpenReview ↗

Official Response to Reviewer 54Nb (1/2)

Dear Reviewer 54Nb,

Thank you for your follow-up and constructive suggestions. We have submitted a revised manuscript. We revised the introduction to make the motivation and contributions of RARS easier to understand. We also added the ablation experiments reported below and our analysis of their results to the RQ2 section of the main paper.

Q1: If beam search only expands valid next tokens, why does RARS still use Lleaf\mathcal L_{\mathrm{leaf}} during training?

A1.1 In our trie-constrained beam search, the trie restricts which tokens can follow each prefix. The AR decoder still scores those tokens using log probabilities computed over its full vocabulary. The score of a valid token therefore depends on both its relative probability among valid tokens and the total probability assigned to all valid tokens. That total can vary across prefixes. Renormalizing over the valid tokens at each prefix can therefore change the ranking of paths from different prefixes.

A1.2 In RARS, Lleaf\mathcal L_{\mathrm{leaf}} directly trains the AR decoder to assign likelihoods to complete SIDs, which are the scores used in default retrieval. The branch loss trains the query encoder through the auxiliary head to learn how relevance is distributed among the children of each prefix. We agree that a single prediction head can also learn the branch targets directly and score a complete SID by adding the conditional log probabilities along its path. We tested the following settings to examine this possibility.

AuthorsOpenReview ↗

Official Response to Reviewer 54Nb (2/2)

Q2: How well do retrieval models perform when the AR decoder is trained only on branch targets, or when beam search uses only the auxiliary head?

A2.1 In setting (a), we remove Lleaf\mathcal L_{\mathrm{leaf}} and train the AR decoder to match the soft branch targets, normalizing its predictions over all valid children of each prefix. Retrieval uses the original AR-only scoring rule. This setting outperforms Base on all five metrics in all three locales, showing that the branch loss can serve as the sole SID training loss. Retaining Lleaf\mathcal L_{\mathrm{leaf}} in Decoder-KL (E+D) improves the ranking of top results in all three locales. Overall, the combined objective performs better in this comparison, and the single-objective variant remains competitive.

A2.2 In setting (b), we reuse the RARS checkpoint and score valid child branches using only the auxiliary head at each step of beam search. The lightweight head can score complete SID paths on its own. With the same checkpoint, AR-only scoring outperforms H-only scoring on all five metrics in all three locales. Combining the two scores (AR+H) gives results close to AR-only. These results support keeping standard AR beam search as the default.

A2.3 We also train the query encoder and auxiliary head using the branch loss as the sole SID training loss, then evaluate with H-only scoring. This setting gives substantially lower retrieval scores across all three locales. The auxiliary head has only 0.312M parameters, compared with 113.275M in the AR Transformer decoder, excluding token embeddings and the vocabulary output layer. Its limited capacity may partly explain the weaker performance when it is used for both SID training and retrieval. Under the same H-only scoring rule, joint training with the full-SID and branch losses gives better results, showing that the full-SID objective also helps the auxiliary head learn to retrieve.

Table R1. Training objectives and beam-search scoring on ESCI (%)

SettingSID training lossesBeam-search scoringUS R@5US R@10US R@100US N@10US N@100ES R@5ES R@10ES R@100ES N@10ES N@100JP R@5JP R@10JP R@100JP N@10JP N@100
BaseLleaf\mathcal L_{\mathrm{leaf}}AR-only7.5012.6937.4011.1218.796.0310.4834.1512.4319.886.059.7730.8911.7718.42
Decoder-KL (E+D)Lleaf+LtreeAR\mathcal L_{\mathrm{leaf}}+\mathcal L_{\mathrm{tree}}^{\mathrm{AR}}AR-only8.1313.0837.7211.5119.666.2911.1734.3612.7920.496.3710.1131.0611.9519.07
(a) AR decoder trained on branch targets aloneLtreeAR\mathcal L_{\mathrm{tree}}^{\mathrm{AR}}AR-only7.8612.9337.6111.4819.316.1810.9134.2312.5620.166.3510.0831.0111.8218.79
RARS, default retrievalLleaf+Ltree\mathcal L_{\mathrm{leaf}}+\mathcal L_{\mathrm{tree}}AR-only8.5713.3938.9812.3620.116.4011.3834.8613.6520.846.5110.2931.5912.3419.44
Auxiliary head only for SID training and retrievalLtree\mathcal L_{\mathrm{tree}}H-only7.0410.8230.089.1615.044.136.7023.387.8613.063.675.5320.266.2110.96
(b) RARS with auxiliary-head-only retrievalLleaf+Ltree\mathcal L_{\mathrm{leaf}}+\mathcal L_{\mathrm{tree}}H-only8.0212.8537.6411.7919.575.9610.8233.9113.1220.186.089.9331.0211.9118.88
RARS with auxiliary-head scoring in beam searchLleaf+Ltree\mathcal L_{\mathrm{leaf}}+\mathcal L_{\mathrm{tree}}AR+H8.6113.4439.0012.3820.156.4311.4134.8913.6420.826.5510.3431.6512.4819.51

The table lists the SID training losses and omits their weights. LtreeAR\mathcal L_{\mathrm{tree}}^{\mathrm{AR}} and Ltree\mathcal L_{\mathrm{tree}} denote the branch losses computed using the AR decoder and the auxiliary head, respectively. H denotes the auxiliary prediction head.

Your proposal to use a single AR decoder to learn the branch targets directly, without Lleaf\mathcal L_{\mathrm{leaf}}, is a simple and elegant design. Our additional experiments show that this approach achieves competitive retrieval results, although it still falls short of full RARS. We see direct branch supervision of the AR decoder as a promising direction that builds on RARS. This single-head approach has substantial potential for further improvement and could simplify training and retrieval.

Thank you again for your feedback. We hope you’re happy with our response. If you have any other questions, we’d be glad to discuss them.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.