- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RARS, a multiresolution relevance supervision framework for hierarchical generative retrieval (GR). It projects document-level relevance into conditional distributions over ID branches and introduces an auxiliary predictor to supervise a shared query representation. The predictor is removed during inference, preserving standard autoregressive decoding. Experiments on three multilingual ESCI datasets demonstrate consistent improvements over matched baselines and alternative supervision objectives.
Critical strengths
-
The paper addresses a limitation of full-SID supervision in GR. Modeling how relevance is distributed across hierarchical branches is well motivated, particularly for queries with multiple relevant documents.
-
The proposed formulation is mathematically coherent and provides a natural way to represent relevance at different SID resolutions. The auxiliary prediction head also makes the approach practical, as it introduces no additional prediction head at inference.
-
The experimental evaluation includes several useful controlled comparisons. The results under matched autoregressive decoding and alternative supervision objectives help isolate the contribution of RARS. The improvements are consistent across three locales, and the additional sensitivity analyses provide useful insights into the method’s behavior.
Critical weaknesses
- The proposed hierarchical objective is closely related to relevance-weighted full-SID supervision. The main difference seems to come from the auxiliary prediction head and how it affects the query representation. The additional novelty beyond existing multi-positive supervision methods is not entirely clear to me.
- All three main datasets come from the ESCI product-search benchmark. I wonder why the authors did not consider traditional document retrieval benchmarks with graded relevance judgments, such as Robust04 [1], GOV2 [2], or ClueWeb09-B[3]. The current evaluation makes it difficult to tell whether the proposed training method generalizes beyond product search.
References:
[1] Ellen M. Voorhees. 2004. Overview of the TREC 2004 Robust Retrieval Track. In TREC. 69–77.
[2] Charles L.A. Clarke, Nick Craswell, and Ian Soboroff. 2004. Overview of the TREC 2004 Terabyte Track. In TREC 2004.
[3] Charles L.A. Clarke, Nick Craswell, and Ian Soboroff. 2010. Overview of the TREC 2009 Web Track. In TREC 2009. 20–29.
Questions for authors
-
How much additional training time and GPU memory does RARS require compared with the matched baseline? This would help assess whether the performance improvements justify the additional training complexity.
-
Could the authors clarify why the NQ320K variant in Table 6 is labeled RARS, given that it shares the same trained parameters as CaLIR and the comparison mainly evaluates the effect of retrieval scoring?
7 comments
Official Response to Reviewer r29x (2/3)
Q2: All three main datasets come from the ESCI product-search benchmark. I wonder why the authors did not consider traditional document retrieval benchmarks with graded relevance judgments, such as Robust04 [1], GOV2 [2], or ClueWeb09-B[3]. The current evaluation makes it difficult to tell whether the proposed training method generalizes beyond product search.
A2: Thank you for suggesting that we evaluate RARS on traditional document retrieval benchmarks. We initially chose ESCI because its US, ES, and JP locales share a graded relevance annotation scheme and include queries with multiple positive documents at different relevance grades. This setting allows us to evaluate SID branch supervision across languages under a common annotation protocol.
To evaluate RARS beyond product search, we conducted experiments on a 200K-document subset of Robust04, GOV2–WebAP, and ClueWeb09-B–ICAT. Obtaining the full GOV2 and ClueWeb09-B collections requires completing the research licensing and distribution procedures of the University of Glasgow and Carnegie Mellon University, respectively. For these experiments, we constructed two subsets from publicly released document text. For GOV2–WebAP, we extracted text from the WebAP archive released by UMass and deduplicated the records by their original GOV2 document IDs, yielding 6,377 documents. For ClueWeb09-B–ICAT, we used the publicly released TREC-ICAT corpus and removed entries with empty text, yielding 26,917 documents. For both subsets, we matched the original document IDs to the corresponding graded TREC document-level relevance judgments.
Within each collection, Base, Grouped soft-target, and RARS use the same model backbone, SID mapping, training budget, and query splits. All three learned methods use AR-only retrieval. Grouped soft-target applies graded relevance at the full-SID level, providing a comparison for assessing the additional benefit of branch-level supervision for ranking and recall.
Table 2. Retrieval performance on graded document retrieval collections (%)
| Collection / Method | R@5 | R@10 | R@100 | N@10 | N@100 |
|---|---|---|---|---|---|
| Robust04 (200K) | |||||
| BM25 | 15.78 | 25.44 | 61.84 | 36.20 | 52.31 |
| Base | 18.37 | 29.18 | 67.95 | 41.52 | 56.47 |
| Grouped soft-target | 19.51 | 30.83 | 69.13 | 42.84 | 58.25 |
| RARS | 21.26 | 33.35 | 72.44 | 45.86 | 61.37 |
| GOV2–WebAP (6,377 docs) | |||||
| BM25 | 17.06 | 31.80 | 93.59 | 56.56 | 74.07 |
| Base | 19.68 | 34.37 | 95.12 | 60.23 | 76.38 |
| Grouped soft-target | 20.84 | 35.91 | 96.07 | 61.87 | 77.64 |
| RARS | 22.69 | 38.17 | 97.35 | 64.14 | 79.03 |
| ClueWeb09-B–ICAT (26,917 docs) | |||||
| BM25 | 11.90 | 20.49 | 80.65 | 34.50 | 56.08 |
| Base | 14.32 | 23.71 | 84.49 | 38.27 | 59.48 |
| Grouped soft-target | 15.56 | 25.83 | 85.92 | 39.94 | 60.85 |
| RARS | 17.38 | 28.24 | 88.46 | 42.83 | 63.17 |
Relative to Grouped soft-target, RARS improves NDCG@10 by 3.02, 2.27, and 2.89 percentage points and Recall@100 by 3.31, 1.28, and 2.54 percentage points on Robust04, GOV2–WebAP, and ClueWeb09-B–ICAT, respectively. These results show that SID branch supervision can improve ranking and recall on the graded document retrieval subsets we evaluated.
Official Response to Reviewer r29x (3/3)
Q3: How much additional training time and GPU memory does RARS require compared with the matched baseline? This would help assess whether the performance improvements justify the additional training complexity.
A3: Thank you for asking us to quantify the training cost of RARS. RARS reuses the retriever’s query representation, predicts SID child branches with a low-dimensional auxiliary head, and limits the number of negatives at each prefix. We compared Base and RARS across the three ESCI locales, using one A100 80 GB GPU per run. Table 3 reports training time for 300 epochs. We measured peak allocated GPU memory over 100 optimizer updates after 20 warm-up steps, with the same effective batch size, batch order, numerical precision, and optimizer settings for both methods.
Table 3. Training cost and retrieval gains relative to the matched baseline
| ESCI Locale | Training Time (h) | Time Change (%) | Peak GPU Memory (GiB) | Memory Increase (MiB) | Δ N@10 (pp) | Δ R@100 (pp) |
|---|---|---|---|---|---|---|
| US | 92.66 → 93.51 | +0.92 | 7.140 → 7.150 | +9.9 | +1.24 | +1.58 |
| ES | 33.03 → 34.04 | +3.06 | 14.985 → 14.994 | +9.1 | +1.22 | +0.71 |
| JP | 40.77 → 41.45 | +1.67 | 16.145 → 16.157 | +12.2 | +0.57 | +0.70 |
Arrows indicate Base → RARS. pp denotes percentage points.
RARS increases training time by less than 3.1% in all three locales and adds only about 9–12 MiB of peak allocated memory per GPU. Over 300 epochs, the additional training time was 0.85 hours on US, 1.01 hours on ES, and 0.68 hours on JP. Under the same AR-only retrieval rule, RARS improves both NDCG@10 and Recall@100 in all three locales.
Q4: Could the authors clarify why the NQ320K variant in Table 6 is labeled RARS, given that it shares the same trained parameters as CaLIR and the comparison mainly evaluates the effect of retrieval scoring?
A4: Thank you for pointing this out. We apologize for the ambiguity in our original description of the NQ320K comparison. CaLIR and RARS share the model parameters learned during SID tokenization and the resulting SID mapping. We train the two retrieval models separately using their respective objectives and evaluate both with the same SID constraints and beam width. The row labeled “RARS” in Table 6 reports the results of the retriever trained with the RARS objective.
Thank you again for the time and care you devoted to reviewing our manuscript. Your constructive feedback has helped us strengthen the paper. We hope our responses address your concerns, and we would be happy to clarify any remaining points.
Official Response to Reviewer r29x (1/3)
Thank you for your recognition of our work and for offering constructive feedback. We respond to your comments below. We have marked the corresponding changes in blue in the revised manuscript.
A1: Thank you for pointing out the connection between our hierarchical objective and relevance-weighted full-SID supervision.
(1.1) The core contribution of RARS is to train a shared query representation to capture where relevant documents share a path in the SID tree, where their paths diverge, and how relevance is distributed among the child branches at each split. Our Base model aligns the query’s latent representation with multiple positive category prototypes at each level of the category hierarchy. RARS retains this multi-positive supervision and adds supervision for SID branch selection. At each parent prefix with relevant descendants, the auxiliary head uses the query representation and the current prefix to predict the relative relevance of its child branches. It compares the relevance-bearing branches with high-scoring valid sibling negatives in the same softmax.
Consider a query with two relevant documents whose relevance weights are 3 and 1. Their SIDs share the first two tokens and diverge at the third. At each of the first two steps, the target assigns all relevance mass to the shared continuation. At the third step, the target allocates that mass to the two child branches in a 3:1 ratio. RARS uses these prefix-conditioned distributions as soft targets to train the query encoder.
(1.2) To assess the contribution of the full branch distribution, we compared sampling a single target child from the relevance distribution with learning the full distribution directly, using the same auxiliary prediction head and retrieval rule. As shown in Table 1, learning the full distribution improves all five retrieval metrics across US, ES, and JP.
Table 1. Comparison of branch sampling and full branch distributions with the same auxiliary prediction head