- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper aligns a re-ranker to multiple downstream generators, rather than the more common single downstream generator setting. The authors sample multiple re-rankings per query, and constrain the multiple generators on each ranking when producing an answer. Rather than considering each generator equally for reward aggregation, the authors upweight the generators that are more sensitive to ranking changes via variance weighting over the generator’s answers with different rankings. Models are trained on standard QA datasets, evaluated in-domain and out-of-domain, as well as evaluated under different generators than those used during training.
Critical strengths
- The argument of having the ranker generalize to multiple downstream generators is well motivated.
- Method was tested under models of different sizes and different backbone families.
- Includes an SFT baseline with matched data which motivates the RL approach.
- Generalization experiments on both unseen datasets and unseen generators.
Critical weaknesses
W1- From S3.3: Uniform averaging “wastes reward budget on generators that are insensitive to the ranker’s output on a given query.”.
I think this does not fully hold under GRPO, since a generator with sigma=0 will add the same constant to every rollout, which will not affect Eq 5 advantage computation due to the normalization (e.g., won’t the “effective signal” increase from 0.323 to 0.484 you report in 5.1 as reward spread be removed by advantage computation?).
W2 - The delta in reported values between baselines and the proposed method is small to the point that significance testing and multiple RL training seeds are necessary for confidence in the reported improvements.
W3- Table 1 seems to mix all datasets together in an average, making it hard to assess individual performance on the in-domain datasets (hotpotqa, nq, tqa). Furthermore, Appendix A states that the best checkpoint is selected by in-distribution SubEM/avg, which seems to be the same evaluation set reported in Table 1. Model selection should be done on a separate validation split, otherwise the reported values are biased.
W4 - The strength/soundness of the proposed method is questionable based on the reported results. The authors report with SubEM, which is more lenient than full EM. On PopQA, VW-MG-GRPO reaches 7.3% with Qwen3-4B, below the untrained ranker (8.8%). Search-R1, whose evaluation protocol the authors follow, reports 38.7% EM for plain retrieve-then-generate with an off-the-shelf Qwen2.5-3B-Ins, with no agent or retriever training. Part of this gap may come from the retriever choice (see W5). I understand the work is not a search agent, and serving multiple frozen generators is a reasonable motivation. Still, the paper should show that ranker alignment gives practically meaningful gains, e.g., against a stronge reranker, or against tuning the generator, so readers can judge the trade-off.
W5- The authors use BM25 over a Wikipedia corpus as the sole retriever whereas most recent work reporting on these datasets resorts to a dense retriever over the same corpus, and the method’s improvements in this scenario are not studied.
W6 - There are also inconsistencies in the reported values. Table 3 reports an average over 3 models which is then detailed in Table 8. I paste here the 2WIKI multi-hop comparison between untrained baseline and their method:
| Qwen3-4B | Qwen3-8B | LLaMA-8B | Avg | |
|---|---|---|---|---|
| No-train (point-wise) | 0.395 | 0.413 | 0.4269 | 0.4056 |
| VW-MG-GRPO | 0.395 | 0.414 | 0.429 | 0.4126 |
The mean of the given untrained ranker’s per-generator values is 0.4116, not the 0.4056 which is also reported in Table 3. With the per-generator values, VW-MG-GRPO improves over the untrained ranker by only 0.001, without significance testing or multiple seeds. The 2Wiki average for Uniform-MG-GRPO is also inconsistent (0.4073 reported vs 0.4100 computed from the per-generator values), which leaves VW-MG-GRPO only 0.003 ahead of uniform aggregation; per generator, it improves over Uniform only on Llama-3.1-8B and is worse on both Qwen models.
Questions for authors
Q1: Can the authors clarify exactly in which conditions the advantages differ between Uniform and VW, and then report empirically for what % of the training data there was a difference between Uniform and VW advantages?
4 comments
Part II
RW1. Mechanism interpretation
We agree that the §3.3 wordings and the effective-signal interpretation are incorrect, and the answer to the reviewer's parenthetical question about that example is yes. We remove "wastes reward budget", "VW-exclusive signal annealing" and "2.2× amplification". With and negligible, a generator with adds only a constant and has no effect. The step-54 example has with equal nonzero , so and the advantages are strictly identical ($ | \Delta A | _{\max}\approx7\times10^{-16}$) regardless of covariance. The ratio is exactly the rescaling normalization removes. Our experiments confirm that the spread proxy is not the effective learning signal, and our analysis rules out amplitude amplification as the explanation for VW's benefit.
Kept are the per-query-router motivation (mean of , weights , dominance, cross-query Pearson ), the power-mean inequality and as an assumption-free upper bound on . We withdraw only that causal bridge and do not replace it with a new one. Where the two rules can give different advantages, they may differ in direction rather than scale. A possible mechanism behind the gain is this directional change, but we do not establish it here and will study the mechanism in dedicated follow-up work.
RW3 / W4 / W5. Experimental setup
Our protocol follows prior work and is not introduced by VW. We use BM25 retrieval over the Wikipedia corpus, SubEM as the metric, three frozen heterogeneous generators, and align only the ranker. Table 1 is a macro-average over NQ, TriviaQA and HotpotQA. The original checkpoint was selected on the same in-distribution SubEM/avg reported in Table 1, which is a biased protocol, and Table r1 corrects it with a separate validation split. The paper does not report per-dataset results.
The 7.3% / 8.8% are per-generator SubEM on PopQA for Qwen3-4B (a held-out dataset) in Table 8, not full EM. On that cell no-train = 0.088, Uniform = 0.078 and VW = 0.073, and we do not dispute that VW is below both. This is one cell on one held-out dataset, while the avg column (the main metric) shows VW Uniform no-train . Search-R1's 38.7% EM is not comparable (single Qwen2.5-3B-Instruct, dense retriever, full EM, no ranker, no multi-generator serving). Tuning the generator is unavailable by assumption, and both trained methods beat no-train in every seed of Table r1, so alignment itself is effective.
The method is downstream of retrieval and not tied to retriever type, but we have not verified transfer beyond BM25. For the camera-ready we will add per-dataset results and state the averaging definitions explicitly, and we will attempt full EM, a stronger off-the-shelf reranker, a generator-tuning baseline under matched compute, and a dense retriever over the same Wikipedia corpus (same and settings).
RW6. Consistency of reported numbers
We thank the reviewer and confirm the inconsistency in the 2Wiki Avg cells. The No-train and Uniform Avg cells are wrong and will be corrected. We will recompute every cell and add full per-generator tables. As for the reviewer's per-generator observation, we will conduct further experiments to verify this, such as experiments using different random seeds or other held-out validation sets.
RQ1. Conditions and fraction of training data
With (negligible, so "ignoring the numerical stabilizer" holds exactly), a generator with has no effect. One nonzero , or all nonzero equal, gives and strictly identical advantages. Only ( nonzero , not all equal) can give different advantages. In two independent runs the fraction is 0.330 / 0.329, so about 67% of groups give strictly identical advantages and 33% is an upper bound. Within these groups the advantages can differ in direction rather than amplitude. Whether such differences drive the gain is a possible mechanism to be studied specifically.
Answer
Thank you for the response. I maintain my score, as the motivation and some of the analysis no longer hold under GRPO normalization (as confirmed by the authors), the reported numerical inconsistencies are acknowledged but remain uncorrected, the transfer results were not rerun under the corrected protocol, and key experiments (per-dataset results, EM, ...) are deferred.
Thank you for the response. We would like to further clarify that the experimental protocol we adopted is commonly used in previous work, and using sub_em as a QA evaluation metric for generative models is not unique to our work but is a widely used metric[1]. If strict EM is used as a downstream task metric, for the models we used, over 90% of the EM values are 0, making it an ineffective reward signal. In this case, EM can only serve as a valid metric after fine-tuning the downstream generator. As for different types of retrievers and generators, our work primarily focuses on rankers, with the goal of building a general-purpose, unified framework. In summary, although you believe our experimental setup is not the most refined, we believe it still effectively demonstrates the validity of our method. Our evaluations before and after training are consistent and the performance improvement is driven by our method and is not attributable to the factors you mentioned. Due to time constraints for this rebuttal, we are unable to conduct large-scale experiments with modified protocols. However, we believe the effectiveness of our method is not limited by these factors, and we will strive to refine our experimental protocol in the future when sufficient time and hardware resources are available. If you are interested in further discussion, please feel free to reach out at any time.
References:
[1] Enabling Large Language Models to Generate Text with Citations
Part I
We thank the reviewer for the thorough audit. We group the weaknesses into four topics and answer the question separately.
What we establish. With a separate validation split and 3 matched seeds, VW significantly outperforms Uniform in all 3 seeds. The paired gain is VW−Uniform=+0.0070±0.0024 (paired t=5.13, p≈0.036<0.05, 95% CI [0.0011,0.0128] excluding 0). Both methods beat the untrained ranker (0.4254) in every seed. We keep the algebraic inequalities and measured statistics and withdraw only the causal bridge from the spread proxy to the effective learning signal. GRPO normalization cancels reward scale (δadv≈10−8, negligible), and our experiments confirm that VW does not win by amplifying the learning signal. Where generators respond differently the two advantages may differ in direction rather than scale. A possible mechanism is this directional change, which we will study in dedicated follow-up work. The contribution is a parameter-free variance-weighting rule aligning one ranker for several frozen, heterogeneous generators (about 1.05× wall-clock), and both it and its empirical benefit are unchanged.
Table r1. SubEM/avg, separate validation split, 3 matched seeds
Table r2. Per-generator SubEM, same runs.
avgis the arithmetic mean over the three generatorsRW2. VW significantly outperforms Uniform across seeds
Table r1 uses a separate held-out validation split and 3 matched seeds. VW significantly outperforms Uniform in all 3 seeds (paired t=5.13, p≈0.036<0.05, 95% CI [0.0011,0.0128] excluding 0). The gain is about 3× the paired std and larger than either method's seed noise (0.0029 / 0.0048), and both methods beat no-train (0.4254) in every seed. The relative gap grows from +0.0053 to +0.0070. The earlier "VW worse on the weak generator" was a selection artifact and disappears (3/3 positive).