acceptodds
Under review as a conference paper at ICLR 2027

Online Alignment of Ranking Model via Variance-Weighted Multi-Generator Feedback

Abstract

We study online alignment of a pointwise ranker to the feedback of multiple downstream generators in retrieval-augmented generation (RAG). Existing work usually aligns the ranker to a single generator, which does not reflect the common deployment reality that one ranker serves several downstream models of different capabilities. When extending to scenarios involving multiple generators, how to aggregate the rewards from multiple generators becomes a key issue. A naive fix, averaging the utilities of all generators uniformly, dilutes the learning signal, because it gives equal weight to generators that are largely insensitive to document order. We propose Variance-Weighted Multi-Generator GRPO (VW-MG-GRPO). For each query we draw G top-K permutations with a Plackett–Luce policy, send them in parallel to a heterogeneous pool of generators (weak / mid / strong), and aggregate their utilities using each generator's within-group standard deviation as its weight. On Natural Questions, TriviaQA, and HotpotQA we validate the method and, more importantly, give a systematic analysis of the variance dynamics during training. We report three transferable findings: (i) variance weighting acts as an implicit per-query router rather than a smoother; (ii) the total variance signal decays over training, so variance weighting behaves as a self-annealing curriculum; and (iii) the effective per-query signal is that of uniform aggregation.

Primary area:
reinforcement learning
Keywords:
online alignment, multi-generator

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_zqoTOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aligns a re-ranker to multiple downstream generators, rather than the more common single downstream generator setting. The authors sample multiple re-rankings per query, and constrain the multiple generators on each ranking when producing an answer. Rather than considering each generator equally for reward aggregation, the authors upweight the generators that are more sensitive to ranking changes via variance weighting over the generator’s answers with different rankings. Models are trained on standard QA datasets, evaluated in-domain and out-of-domain, as well as evaluated under different generators than those used during training.

Critical strengths
  • The argument of having the ranker generalize to multiple downstream generators is well motivated.
  • Method was tested under models of different sizes and different backbone families.
  • Includes an SFT baseline with matched data which motivates the RL approach.
  • Generalization experiments on both unseen datasets and unseen generators.
Critical weaknesses

W1- From S3.3: Uniform averaging “wastes reward budget on generators that are insensitive to the ranker’s output on a given query.”.

I think this does not fully hold under GRPO, since a generator with sigma=0 will add the same constant to every rollout, which will not affect Eq 5 advantage computation due to the normalization (e.g., won’t the “effective signal” increase from 0.323 to 0.484 you report in 5.1 as reward spread be removed by advantage computation?).

W2 - The delta in reported values between baselines and the proposed method is small to the point that significance testing and multiple RL training seeds are necessary for confidence in the reported improvements.


W3- Table 1 seems to mix all datasets together in an average, making it hard to assess individual performance on the in-domain datasets (hotpotqa, nq, tqa). Furthermore, Appendix A states that the best checkpoint is selected by in-distribution SubEM/avg, which seems to be the same evaluation set reported in Table 1. Model selection should be done on a separate validation split, otherwise the reported values are biased.


W4 - The strength/soundness of the proposed method is questionable based on the reported results. The authors report with SubEM, which is more lenient than full EM. On PopQA, VW-MG-GRPO reaches 7.3% with Qwen3-4B, below the untrained ranker (8.8%). Search-R1, whose evaluation protocol the authors follow, reports 38.7% EM for plain retrieve-then-generate with an off-the-shelf Qwen2.5-3B-Ins, with no agent or retriever training. Part of this gap may come from the retriever choice (see W5). I understand the work is not a search agent, and serving multiple frozen generators is a reasonable motivation. Still, the paper should show that ranker alignment gives practically meaningful gains, e.g., against a stronge reranker, or against tuning the generator, so readers can judge the trade-off.

W5- The authors use BM25 over a Wikipedia corpus as the sole retriever whereas most recent work reporting on these datasets resorts to a dense retriever over the same corpus, and the method’s improvements in this scenario are not studied.

W6 - There are also inconsistencies in the reported values. Table 3 reports an average over 3 models which is then detailed in Table 8. I paste here the 2WIKI multi-hop comparison between untrained baseline and their method:

Qwen3-4BQwen3-8BLLaMA-8BAvg
No-train (point-wise)0.3950.4130.42690.4056
VW-MG-GRPO0.3950.4140.4290.4126

The mean of the given untrained ranker’s per-generator values is 0.4116, not the 0.4056 which is also reported in Table 3. With the per-generator values, VW-MG-GRPO improves over the untrained ranker by only 0.001, without significance testing or multiple seeds. The 2Wiki average for Uniform-MG-GRPO is also inconsistent (0.4073 reported vs 0.4100 computed from the per-generator values), which leaves VW-MG-GRPO only 0.003 ahead of uniform aggregation; per generator, it improves over Uniform only on Llama-3.1-8B and is worse on both Qwen models.

Questions for authors

Q1: Can the authors clarify exactly in which conditions the advantages differ between Uniform and VW, and then report empirically for what % of the training data there was a difference between Uniform and VW advantages?

4 comments

We thank the reviewer for the thorough audit. We group the weaknesses into four topics and answer the question separately.

What we establish. With a separate validation split and 3 matched seeds, VW significantly outperforms Uniform in all 3 seeds. The paired gain is VW−Uniform=+0.0070±0.0024\mathrm{VW}-\mathrm{Uniform}=+0.0070\pm0.0024 (paired t=5.13t=5.13, p≈0.036<0.05p\approx0.036<0.05, 95% CI [0.0011,0.0128][0.0011,0.0128] excluding 0). Both methods beat the untrained ranker (0.4254) in every seed. We keep the algebraic inequalities and measured statistics and withdraw only the causal bridge from the spread proxy to the effective learning signal. GRPO normalization cancels reward scale (δadv≈10−8\delta_{adv}\approx10^{-8}, negligible), and our experiments confirm that VW does not win by amplifying the learning signal. Where generators respond differently the two advantages may differ in direction rather than scale. A possible mechanism is this directional change, which we will study in dedicated follow-up work. The contribution is a parameter-free variance-weighting rule aligning one ranker for several frozen, heterogeneous generators (about 1.05×1.05\times wall-clock), and both it and its empirical benefit are unchanged.

Table r1. SubEM/avg, separate validation split, 3 matched seeds

SeedUniformVWVW − Uniform
10.43640.4434+0.0070
20.43810.4427+0.0046
30.44210.4514+0.0093
mean ± std0.4389 ± 0.00290.4458 ± 0.0048+0.0070 ± 0.0024

Table r2. Per-generator SubEM, same runs. avg is the arithmetic mean over the three generators

SeedMethodQwen3-4BQwen3-8BLlama-3.1-8B
1Uniform0.33030.45050.5285
1VW0.35740.44540.5275
2Uniform0.33730.45050.5265
2VW0.34930.45750.5215
3Uniform0.35040.43840.5375
3VW0.35330.46250.5385
mean ± stdUniform0.3393 ± 0.01020.4465 ± 0.00700.5308 ± 0.0059
mean ± stdVW0.3533 ± 0.00410.4551 ± 0.00880.5292 ± 0.0086

RW2. VW significantly outperforms Uniform across seeds

Table r1 uses a separate held-out validation split and 3 matched seeds. VW significantly outperforms Uniform in all 3 seeds (paired t=5.13t=5.13, p≈0.036<0.05p\approx0.036<0.05, 95% CI [0.0011,0.0128][0.0011,0.0128] excluding 0). The gain is about 3×3\times the paired std and larger than either method's seed noise (0.0029 / 0.0048), and both methods beat no-train (0.4254) in every seed. The relative gap grows from +0.0053+0.0053 to +0.0070+0.0070. The earlier "VW worse on the weak generator" was a selection artifact and disappears (3/3 positive).

RW1. Mechanism interpretation

We agree that the §3.3 wordings and the effective-signal interpretation are incorrect, and the answer to the reviewer's parenthetical question about that example is yes. We remove "wastes reward budget", "VW-exclusive signal annealing" and "2.2× amplification". With Ag=(Rg−Rˉ)/(std(R)+δadv)A_g=(R_g-\bar R)/(\mathrm{std}(R)+\delta_{adv}) and δadv≈10−8\delta_{adv}\approx10^{-8} negligible, a generator with σm=0\sigma_m=0 adds only a constant and has no effect. The step-54 example has σ=(0.484,0,0.484)\sigma=(0.484,0,0.484) with k=2k=2 equal nonzero σ\sigma, so RV=(M/k)RU+const=1.5 RU+constR^V=(M/k)R^U+\mathrm{const}=1.5\,R^U+\mathrm{const} and the advantages are strictly identical ($ | \Delta A | _{\max}\approx7\times10^{-16}$) regardless of covariance. The ratio 0.484/0.323=1.5=M/k0.484/0.323=1.5=M/k is exactly the rescaling normalization removes. Our experiments confirm that the spread proxy is not the effective learning signal, and our analysis rules out amplitude amplification as the explanation for VW's benefit.

Kept are the per-query-router motivation (mean σ\sigma of 0.0746/0.0864/0.11960.0746/0.0864/0.1196, weights 0.28/0.31/0.40≠0.330.28/0.31/0.40\ne0.33, 57.0/28.4/14.6%57.0/28.4/14.6\% dominance, cross-query Pearson [−0.16,0.24][-0.16,0.24]), the power-mean inequality σvw≥σˉ\sigma_{vw}\ge\bar\sigma and ∑mcmσm\sum_m c_m\sigma_m as an assumption-free upper bound on std(Rc)\mathrm{std}(R^c). We withdraw only that causal bridge and do not replace it with a new one. Where the two rules can give different advantages, they may differ in direction rather than scale. A possible mechanism behind the gain is this directional change, but we do not establish it here and will study the mechanism in dedicated follow-up work.

RW3 / W4 / W5. Experimental setup

Our protocol follows prior work and is not introduced by VW. We use BM25 retrieval over the Wikipedia corpus, SubEM as the metric, three frozen heterogeneous generators, and align only the ranker. Table 1 is a macro-average over NQ, TriviaQA and HotpotQA. The original checkpoint was selected on the same in-distribution SubEM/avg reported in Table 1, which is a biased protocol, and Table r1 corrects it with a separate validation split. The paper does not report per-dataset results.

The 7.3% / 8.8% are per-generator SubEM on PopQA for Qwen3-4B (a held-out dataset) in Table 8, not full EM. On that cell no-train = 0.088, Uniform = 0.078 and VW = 0.073, and we do not dispute that VW is below both. This is one cell on one held-out dataset, while the avg column (the main metric) shows VW 0.1643>0.1643> Uniform 0.1613>0.1613> no-train 0.15700.1570. Search-R1's 38.7% EM is not comparable (single Qwen2.5-3B-Instruct, dense retriever, full EM, no ranker, no multi-generator serving). Tuning the generator is unavailable by assumption, and both trained methods beat no-train in every seed of Table r1, so alignment itself is effective.

The method is downstream of retrieval and not tied to retriever type, but we have not verified transfer beyond BM25. For the camera-ready we will add per-dataset results and state the averaging definitions explicitly, and we will attempt full EM, a stronger off-the-shelf reranker, a generator-tuning baseline under matched compute, and a dense retriever over the same Wikipedia corpus (same NN and settings).

RW6. Consistency of reported numbers

We thank the reviewer and confirm the inconsistency in the 2Wiki Avg cells. The No-train and Uniform Avg cells are wrong and will be corrected. We will recompute every cell and add full per-generator tables. As for the reviewer's per-generator observation, we will conduct further experiments to verify this, such as experiments using different random seeds or other held-out validation sets.

RQ1. Conditions and fraction of training data

With δadv≈10−8\delta_{adv}\approx10^{-8} (negligible, so "ignoring the numerical stabilizer" holds exactly), a generator with σm=0\sigma_m=0 has no effect. One nonzero σ\sigma, or all nonzero σ\sigma equal, gives RV=(M/k)RU+constR^V=(M/k)R^U+\mathrm{const} and strictly identical advantages. Only candiffer\mathrm{can_differ} (k≥2k\ge2 nonzero σ\sigma, not all equal) can give different advantages. In two independent runs the candiffer\mathrm{can_differ} fraction is 0.330 / 0.329, so about 67% of groups give strictly identical advantages and 33% is an upper bound. Within these groups the advantages can differ in direction rather than amplitude. Whether such differences drive the gain is a possible mechanism to be studied specifically.

Reviewer_zqoTOpenReview ↗

Answer

Thank you for the response. I maintain my score, as the motivation and some of the analysis no longer hold under GRPO normalization (as confirmed by the authors), the reported numerical inconsistencies are acknowledged but remain uncorrected, the transfer results were not rerun under the corrected protocol, and key experiments (per-dataset results, EM, ...) are deferred.

Thank you for the response. We would like to further clarify that the experimental protocol we adopted is commonly used in previous work, and using sub_em as a QA evaluation metric for generative models is not unique to our work but is a widely used metric[1]. If strict EM is used as a downstream task metric, for the models we used, over 90% of the EM values are 0, making it an ineffective reward signal. In this case, EM can only serve as a valid metric after fine-tuning the downstream generator. As for different types of retrievers and generators, our work primarily focuses on rankers, with the goal of building a general-purpose, unified framework. In summary, although you believe our experimental setup is not the most refined, we believe it still effectively demonstrates the validity of our method. Our evaluations before and after training are consistent and the performance improvement is driven by our method and is not attributable to the factors you mentioned. Due to time constraints for this rebuttal, we are unable to conduct large-scale experiments with modified protocols. However, we believe the effectiveness of our method is not limited by these factors, and we will strive to refine our experimental protocol in the future when sufficient time and hardware resources are available. If you are interested in further discussion, please feel free to reach out at any time.

References:

[1] Enabling Large Language Models to Generate Text with Citations

Reviewer_SCFtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes VW-MG-GRPO to align a RAG ranker with multiple frozen generators using variance-weighted feedback. QA and transfer experiments show gains over uniform weighting, with analyses of weighting behavior and training dynamics.

Critical strengths
  1. The method addresses practical multi-generator alignment with a simple weighting rule and no extra trainable parameters. Tests on unseen generators and datasets provide evidence of generalization.
  2. Tests on unseen generators and datasets provide evidence of generalization.
Critical weaknesses
  1. The mechanism analysis overlooks GRPO normalization and cross-generator covariance, leaving the signal-amplification and exclusive self-annealing claims unsupported.
  2. Gains over uniform weighting are small (0.4458 → 0.4511), with only one training seed and no uncertainty estimates to establish reliability.
Questions for authors
  1. How do the mechanism claims hold after GRPO normalization, accounting for covariance and uniform aggregation’s zero-variance behavior?
  2. Could you report multiple-seed results and paired confidence intervals against Uniform-MG-GRPO?
2 comments

We thank the reviewer. Both points are addressed with a revised analysis of the learning signal (GRPO normalization, covariance, uniform's zero-variance behavior) and new 3-seed experiments with paired CIs.

What we establish. VW significantly outperforms Uniform in all 3 seeds. The paired gain is VW−Uniform=+0.0070±0.0024\mathrm{VW}-\mathrm{Uniform}=+0.0070\pm0.0024 (paired t=5.13t=5.13, p≈0.036<0.05p\approx0.036<0.05, 95% CI [0.0011,0.0128][0.0011,0.0128] excluding 0). Both methods beat the untrained ranker in every seed. We keep the algebraic inequalities and measured statistics and withdraw only the causal link from the spread proxy to the effective learning signal. GRPO cancels scale (δadv≈10−8\delta_{adv}\approx10^{-8}), and our experiments confirm VW does not win by amplifying the learning signal. Where generators respond differently the two advantages may differ in direction rather than scale. A possible mechanism is this directional change, which we will study in dedicated follow-up work rather than asserting here. The contribution is an online critic-free GRPO formulation plus a parameter-free rule wm=σm/(∑σm′+δ)w_m=\sigma_m/(\sum\sigma_{m'}+\delta) for one ranker serving several frozen heterogeneous generators, and it is unchanged. Crucially, the rule is defined purely from dispersion and makes no covariance assumption. Covariance enters only the amplitude argument that our analysis rules out.

Table r1. SubEM/avg, separate validation split, 3 matched seeds

SeedUniformVWVW − Uniform
10.43640.4434+0.0070
20.43810.4427+0.0046
30.44210.4514+0.0093
mean ± std0.4389 ± 0.00290.4458 ± 0.0048+0.0070 ± 0.0024

Table r2. Per-generator SubEM, same runs. avg is the arithmetic mean over the three generators

SeedMethodQwen3-4BQwen3-8BLlama-3.1-8B
1Uniform0.33030.45050.5285
1VW0.35740.44540.5275
2Uniform0.33730.45050.5265
2VW0.34930.45750.5215
3Uniform0.35040.43840.5375
3VW0.35330.46250.5385
mean ± stdUniform0.3393 ± 0.01020.4465 ± 0.00700.5308 ± 0.0059
mean ± stdVW0.3533 ± 0.00410.4551 ± 0.00880.5292 ± 0.0086

RW2 / Q2. Small gains, multiple seeds, paired CIs

Table r1 uses a separate held-out validation split and 3 matched seeds (paired). VW significantly outperforms Uniform in all 3 seeds (paired t=5.13t=5.13, p≈0.036<0.05p\approx0.036<0.05, 95% CI [0.0011,0.0128][0.0011,0.0128] excluding 0). The gap (0.0070) is about 3×3\times the paired std and exceeds either method's seed noise (0.0029 / 0.0048). Both beat no-train (0.4254) in every seed. The relative gap grows (+0.0053→+0.0070+0.0053\to+0.0070). The original "VW worse on the weak generator" was a selection artifact and disappears (3/3 positive).

RW1 / Q1. Mechanism under normalization, covariance, zero-variance behavior

(a) What survives. With um,g=μm+σmzm,gu_{m,g}=\mu_m+\sigma_m z_{m,g}, Rc=const+∑mcmσmzm,gR^c=\mathrm{const}+\sum_m c_m\sigma_m z_{m,g} and A=(R−Rˉ)/(std(R)+δadv)A=(R-\bar R)/(\mathrm{std}(R)+\delta_{adv}), where δadv≈10−8\delta_{adv}\approx10^{-8} is negligible against std(R)∼10−1\mathrm{std}(R)\sim10^{-1}, AA is invariant to additive constants and positive rescaling. A generator with σm=0\sigma_m=0 has no effect. One nonzero σ\sigma, or all nonzero σ\sigma equal, gives RV∝RUR^V\propto R^U and identical advantages. Only candiffer\mathrm{can_differ} (k≥2k\ge2 nonzero σ\sigma, not all equal) can give different advantages, and there they may differ in direction rather than scale. Whether such differences cause the gain is left to a dedicated follow-up study.

(b) Uniform's zero-variance behavior. For c=1/Mc=1/M,

Var(RU)=∑m(σmM)2+2∑m<m′σmσm′M2ρmm′.\mathrm{Var}(R^U)=\sum_m\left(\frac{\sigma_m}{M}\right)^2+2\sum_{m<m'}\frac{\sigma_m\sigma_{m'}}{M^2}\rho_{mm'}.

A generator with σm=0\sigma_m=0 contributes neither variance nor covariance, only a constant cancelled by R−RˉR-\bar R. A flat generator therefore does not dilute the normalized advantage, and we remove the §3.3 "wastes reward budget" wording. When all generators are flat, both rules give zero advantages, so the all-zero fallback creates no difference.

(c) The reward-spread proxy. σvw≥σˉ\sigma_{vw}\ge\bar\sigma holds identically (power means and Cauchy-Schwarz, no covariance assumption), and std(Rc)≤∑mcmσm\mathrm{std}(R^c)\le\sum_m c_m\sigma_m is an assumption-free upper bound attained at ρ=1\rho=1. Table 7 (0.158→0.3070.158\to0.307) is correct as a statement about that proxy. What we drop is that it is a stronger effective learning signal. At ρ=1\rho=1, RV=(σvw/σˉ)RU+constR^V=(\sigma_{vw}/\bar\sigma)R^U+\mathrm{const} and the advantages are strictly identical, so the realized spread reaches its upper bound in a case where VW is inert. Measured ηvar\eta_{var}, the ratio of realized Var(R)\mathrm{Var}(R) to its ρ=1\rho=1 ceiling, is 0.348 / 0.428, confirming generators are not fully collinear. This is evidence against amplitude amplification, not an identification of the mechanism behind the gain.

(d) Self-annealing. Normalized spread is flat (half-change +0.03%/−0.19%+0.03\%/-0.19\%, slope R2≈0.001R^2\approx0.001). The decline of raw reward spread (13.0% / 7.2%, R2=0.050/0.012R^2=0.050/0.012) is a σ\sigma-magnitude decay present in both runs and cancelled by normalization. We drop "VW-exclusive signal annealing". Retained as a measured fact is the weight direction adapting (HHI 0.689→0.7530.689\to0.753, uniform weights fixed at 1/M1/M), which reallocates weight as generators stop responding. We present it only as a possible mechanism and leave the causal study to dedicated follow-up work.

(e) Covariance hard bounds. Write Var(Rc)=Ac+Scov\mathrm{Var}(R^c)=A_c+S_{cov} with Ac=∑mcm2σm2A_c=\sum_m c_m^2\sigma_m^2,

Scov=2∑m<m′cmcm′Covmm′S_{cov}=2\sum_{m<m'}c_m c_{m'}\mathrm{Cov}_{mm'}

and Bc=2∑m<m′cmcm′σmσm′B_c=2\sum_{m<m'}c_m c_{m'}\sigma_m \sigma_{m'}.

Cauchy-Schwarz gives ∣Covmm′∣≤σmσm′|\mathrm{Cov}_{mm'}|\le\sigma_m\sigma_{m'},

so Scov∈[−Bc,Bc]S_{cov}\in[-B_c,B_c].

This is a loose pairwise bound because the three ρmm′\rho_{mm'}cannot vary independently. When all σm>0\sigma_m>0, PSD of the 3×33\times3 correlation matrix is 1+2ρ12ρ13ρ23−(ρ122+ρ132+ρ232)≥01+2\rho_{12}\rho_{13}\rho_{23}-(\rho_{12}^2+\rho_{13}^2+\rho_{23}^2)\ge0, so under equicorrelation ρ∈[−0.5,1]\rho\in[-0.5,1] is a strictly tighter joint feasible set.

Covariance enters only the single scalar ScovS_{cov} in Var(Rc)\mathrm{Var}(R^c), and Ac=(Rc−Rˉc)/(std(Rc)+δadv)A^c=(R^c-\bar R^c)/(\mathrm{std}(R^c)+\delta_{adv}) divides out the resulting std(Rc)\mathrm{std}(R^c) with negligible δadv\delta_{adv}. No feasible covariance can therefore convert a difference in reward spread into a difference in advantage scale. Whether AUA^U and AVA^V differ in a given group is governed by the coefficient vector amc=cmσma^c_m=c_m\sigma_m in Rc−Rˉc=∑mamczm,gR^c-\bar R^c=\sum_m a^c_m z_{m,g}. These bounds fully account for covariance in the variance analysis and, together with GRPO removing the overall scale, they rule out amplitude amplification as the source of the gain, but they do not identify the mechanism behind it, which we leave for a dedicated follow-up study.

(f) Frequency. The candiffer\mathrm{can_differ} fraction is 0.330 / 0.329 in two independent runs. About 67% of groups give strictly identical advantages, an assumption-free bound delimiting the subset where the two rules can differ.

Reviewer_fFZ6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work aligns a reranker online to serveral frozen generators with different levels of capability. For each query, a Plackett-Luce policy samples a group of top-10 orderings and every generator answers from each one. The utilities are combined into one GRPO reward, weighting each generator by the standard deviation of its utility within the group. On a mix of NQ, TriviaQA and HotpotQA, the paper reports higher average scores than uniform averaging, single generator alignment, and SFT. It is also tested on held-out generators and datasets. An analysis presents weights as a router, a self-annealing curriculum and 2.2x signal.

Critical strengths
  1. Even with plain averaging, multi-generator GRPO beats every single generator run and matched-data SFT on the in-distribution avg. It also beats every single generator run on both held-out generators. This matches what prior work like uRAG found offline and since one ranking service usually feeds several generators, it's the main reason I see value in this work.
Critical weaknesses
  1. The submission claims every existing method assumes a single and fixed downstream generator. However, prior work [1] trains one reranker on feedback from multiple RAG systems, and [2] retrains such a ranker on feedback to its rankings. The contribution should be positioned around the specific GRPO formulation and weighting rule. In distrbution, uniform averaging reaches most of the method's gain, and variance weighting adds around half a SubEM point from one seed, reversing on the weak generator. So a robust incremental benefit over uniform averaging is not yet established.

  2. GRPO already removes constant offsets and positive rescaling within each group, ignoring the numerical stabilizer. A flat genereator therefore does not dilute normalized advantages. In the easy-query scenario, if only weak generator still varies, both rules give the same advantages. Therefore I do not see why self-annealing should be specific to variance weighting. Likewise the claimed 2.2x increase the spread proxy does not show that the learning signal is stronger after normalization. Weighting can change the advantages when multiple generators have different nonzero standard deviations, but the paper does not examine those changes to explain the gains.

[1] Salemi, Alireza, and Hamed Zamani. "Towards a search engine for machines: Unified ranking for multiple retrieval-augmented large language models." Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval. 2024.

[2] Salemi, Alireza, and Hamed Zamani. "Learning to rank for multiple retrieval-augmented models through iterative utility maximization." Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR). 2025.

Questions for authors
  1. Could you rerun variance weighting and uniform averaging with at least three matched seeds and report the mean and std? Please clarify whether checkpoint selection uses a validation split separate from final evaluation; if not, please use one. A gap clearly above seed noise would raise my score.

  2. In what fraction of training groups do two or more generators have different nonzero std? Ignoring the numerical stabilizer, only these groups can give different normalized advantages. Within these groups, how often and by how much do the advantages actually differ?

3 comments

We thank the reviewer. All points are addressed with new 3-seed experiments and an updated analysis of the learning signal.

What we establish. With a separate validation split and 3 matched seeds, VW significantly outperforms Uniform in all 3 seeds. The paired gain is VW−Uniform=+0.0070±0.0024\mathrm{VW}-\mathrm{Uniform}=+0.0070\pm0.0024 (paired t=5.13t=5.13, p≈0.036<0.05p\approx0.036<0.05, 95% CI [0.0011,0.0128][0.0011,0.0128] excluding 0). Both methods beat the untrained ranker (0.4254) in every seed. We keep the paper's algebraic inequalities and measured statistics and withdraw only the causal bridge from the spread proxy to the effective learning signal. GRPO normalization cancels reward scale (δadv≈10−8\delta_{adv}\approx10^{-8}, negligible), and our experiments confirm that VW does not win by amplifying the learning signal. A possible mechanism behind the gain is this directional change, but we do not establish it here and will study the mechanism in dedicated follow-up work. The contribution is unchanged. It is an online, critic-free GRPO formulation plus a parameter-free variance-weighting rule for one ranker serving several frozen, heterogeneous generators (about 1.05×1.05\times wall-clock), and its empirical benefit is unchanged too.

Table r1. SubEM/avg, separate validation split, 3 matched seeds

SeedUniformVWVW − Uniform
10.43640.4434+0.0070
20.43810.4427+0.0046
30.44210.4514+0.0093
mean ± std0.4389 ± 0.00290.4458 ± 0.0048+0.0070 ± 0.0024

Table r2. Per-generator SubEM, same runs. avg is the arithmetic mean over the three generators

SeedMethodQwen3-4BQwen3-8BLlama-3.1-8B
1Uniform0.33030.45050.5285
1VW0.35740.44540.5275
2Uniform0.33730.45050.5265
2VW0.34930.45750.5215
3Uniform0.35040.43840.5375
3VW0.35330.46250.5385
mean ± stdUniform0.3393 ± 0.01020.4465 ± 0.00700.5308 ± 0.0059
mean ± stdVW0.3533 ± 0.00410.4551 ± 0.00880.5292 ± 0.0086

RW1. Positioning relative to [1][2]

We agree that "every existing method assumes a single fixed generator" is an overstatement and remove it. [1][2] already train a ranker on multi-RAG feedback and we claim no priority there. The contribution is repositioned exactly as suggested. It is (a) online critic-free GRPO pooling heterogeneous utilities into one scalar reward, (b) the variance-weighting rule wm=σm/(∑m′σm′+δ)w_m=\sigma_m/(\sum_{m'}\sigma_{m'}+\delta) with an all-zero fallback, and (c) the corrected empirical gains. We will cite [1][2] and state the distinction as training objective (group-relative RL vs. iterative utility maximization) and object of study (the aggregation rule). This question is specific to online RL and is not treated in [1][2].

RW1 (second part) / Q1. Robust gain, seeds, validation split

Both original concerns are confirmed (single seed 42, checkpoint selected on the evaluation set). Table r1 uses a separate held-out validation split and 3 matched seeds. VW significantly outperforms Uniform in all 3 seeds (paired t=5.13t=5.13, p≈0.036<0.05p\approx0.036<0.05, 95% CI [0.0011,0.0128][0.0011,0.0128] excluding 0). The gap is about 3×3\times the paired std and larger than either method's seed noise. The relative gap grows from +0.0053+0.0053 to +0.0070+0.0070. The "weak-generator reversal" was a selection artifact, with per-seed differences +0.0271+0.0271, +0.0120+0.0120 and +0.0029+0.0029 (mean +0.0140+0.0140, 3/3 positive).

RW2. GRPO normalization

We agree with the mathematics. With um,g=μm+σmzm,gu_{m,g}=\mu_m+\sigma_m z_{m,g} and Ag=(Rg−Rˉ)/(std(R)+δadv)A_g=(R_g-\bar R)/(\mathrm{std}(R)+\delta_{adv}),

RgU=const+1M∑mσmzm,g,RgV=const+1∑mσm∑mσm2zm,g.R^U_g=\mathrm{const}+\frac{1}{M}\sum_m\sigma_m z_{m,g},\qquad R^V_g=\mathrm{const}+\frac{1}{\sum_m\sigma_m}\sum_m\sigma_m^2 z_{m,g}.

Here δadv≈10−8\delta_{adv}\approx10^{-8} is negligible against std(R)∼10−1\mathrm{std}(R)\sim10^{-1}, so the reviewer's "ignoring the numerical stabilizer" holds exactly. AA is invariant to additive constants and positive rescaling. A generator with σm=0\sigma_m=0 contributes only a constant and has no effect, so uniform averaging does not dilute the normalized advantage. If exactly one generator has nonzero σ\sigma, or if all nonzero σ\sigma are equal, then RV∝RUR^V\propto R^U and the advantages are identical. Advantages can differ only when k≥2k\ge2 generators have nonzero σ\sigma that are not all equal. We denote this condition candiffer\mathrm{can_differ}. It is exactly the reviewer's closing clause and is quantified in Q2.

Kept. σˉ=1M∑mσm\bar\sigma=\frac{1}{M}\sum_m\sigma_m and σvw=∑mσm2/∑mσm\sigma_{vw}=\sum_m\sigma_m^2/\sum_m\sigma_m are first- and second-order power means with σvw≥σˉ\sigma_{vw}\ge\bar\sigma by Cauchy-Schwarz, requiring no covariance assumption. ∑mcmσm\sum_m c_m\sigma_m is an assumption-free upper bound on std(Rc)\mathrm{std}(R^c). Table 7 (0.158→0.3070.158\to0.307) and the §5.1 routing statistics (weights 0.28/0.31/0.40≠0.330.28/0.31/0.40\ne0.33, 57.0/28.4/14.6%57.0/28.4/14.6\% dominance, cross-query Pearson [−0.16,0.24][-0.16,0.24], HHI 0.689→0.7530.689\to0.753) all stand.

Revised. Our experiments confirm the spread proxy is not the effective learning signal because GRPO cancels scale. We drop the wordings "wastes reward budget", "VW-exclusive signal annealing" and "2.2× amplification". The §5.1 the example has σ=(0.484,0,0.484)\sigma=(0.484,0,0.484) with k=2k=2 equal nonzero σ\sigma, so RV=(M/k)RU+const=1.5 RU+constR^V=(M/k)R^U+\mathrm{const}=1.5\,R^U+\mathrm{const} and the advantages are strictly identical ($ | \Delta A | _{\max}\approx7\times10^{-16}$) independent of covariance. The ratio 0.484/0.323=1.5=M/k0.484/0.323=1.5=M/k is exactly the rescaling normalization removes. This delineates where VW acts rather than refuting it. Genuinely VW-specific and retained as a measured fact is the weight direction adapting during training (HHI$0.689\to0.753$) as generators stop responding. We present it only as a possible mechanism and leave the causal study of why VW wins to dedicated follow-up work.

RQ2. Fraction and magnitude

Two independent runs give a candiffer\mathrm{can_differ} fraction of 0.330 / 0.329. About 67% of groups therefore give strictly identical advantages, and 33% is the upper bound where VW can change the gradient. This is consistent with a reliable gain on that subset. Within these groups the two advantages differ in direction governed by the dispersion of the nonzero σ\sigma, not in amplitude. Whether this directional difference drives the observed gain is a possible mechanism that we will study specifically, not a conclusion we establish here.

Thanks for the runs and the confirmation, which resolves my reliability concern. But the gain is small and its mechanism is now unexplained after the signal amplification and annealing claims were withdrawn. So, I keep my current score.

Best regards,

Reviewer fFZ6

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.