- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to train top-k selection modules in language model systems, including document rerankers in RAG and expert routers in MoE. It argues that ordinary backpropagation mainly adjusts how selected items are combined, while providing limited direct feedback on the selection decision itself. The authors formulate training as minimizing expected task loss over subsets sampled without replacement. The resulting gradient combines a policy gradient term that updates selection probabilities with a supervised term that improves predictions using the selected items. A tunable coefficient on the policy gradient controls the bias–variance tradeoff.
The paper analyzes the relationship to an all-candidate reference objective and derives convergence bounds under stated assumptions. Experiments cover synthetic regression, RAG reranker training with frozen language models, and small MoE models trained from scratch on Wikitext-103. All experiments select three items while varying the candidate pool size. The reported results improve selection accuracy, QA exact match, and language modeling perplexity over the evaluated baselines, including a supervised-only variant, with a small nonzero policy-gradient weight generally performing best.
Critical strengths
- The distinction between learning which items to select and how to combine them is useful. It identifies a limitation of training through the selected outputs alone and motivates giving the selector direct feedback on its decisions through the final task loss.
- The explicit treatment of the bias–variance tradeoff is practically well motivated. It recognizes that an unbiased gradient can be difficult to use when its variance is high, and provides a simple way to balance noisy selection feedback with the supervised training signal.
- The evaluation examines both selection quality and prediction performance. Synthetic regression allows direct evaluation against known selection targets, while RAG and MoE test the approach with frozen and jointly trained predictors, respectively. These complementary settings strengthen the empirical case for the method.
Critical weaknesses
- The benefit of computing probabilities for unordered sets is not yet clear. A simpler hybrid estimator can use the probability of the actual sampling order while retaining the same sampling procedure, prediction loss, and supervised gradient. The current supervised-only comparison supports adding policy feedback, but does not establish how much additional benefit comes from accounting for all possible sampling orders.
- Practical scalability beyond k=3 is not sufficiently demonstrated. All experiments vary the candidate pool size while fixing the number of selected items at three. The paper does not report the cost of computing and differentiating the set probabilities in Eq. 8, or the resulting training overhead. Results at larger k, together with runtime measurements, would help establish whether the method remains practical as the selection budget grows.
- The preference for a very small may depend on the loss baseline. The method uses the average loss of other examples as a reference for the current selection. Since examples differ in difficulty, the policy feedback may contain noise unrelated to selection quality. It remains unclear how much the benefit of shrinking the policy term comes from compensating for this noise.
- The incremental benefit of policy feedback beyond exploration control remains unclear. The supervised-only baseline already includes stochastic selection and tuned sampling temperatures, and some additional gains on real tasks are modest. Although the improvements are consistent, the current evidence leaves open whether similar gains could be achieved through simpler adjustments to exploration. This makes the practical value of the additional policy-gradient term harder to assess.
Questions for authors
-
Does summing over all sampling orders improve the results? In one regression or MoE experiment, could you compare the proposed method with a version that uses only the probability of the actual sampled order for the policy gradient? Keep the other settings the same. This would show how much the extra step of combining all possible orders adds to the final performance.
-
How much extra computation does the method require? For example, a set of eight items has 8! = 40320 possible sampling orders, so explicitly summing over them could be expensive. Could you clarify how set probabilities are computed in practice and compare per-step training time with supervised-only training at k=3,4,8, including the time spent on set probabilities? Reporting task performance at one larger k would also help show whether the gains justify the extra cost.
-
Would a different loss baseline change which works best? The current baseline uses losses from other examples, which may differ in difficulty. In one regression experiment, could you instead use each example’s own past losses as its baseline? Comparing the two choices would help show whether a very small still works best when the baseline better reflects the current example’s difficulty.
-
Could you provide stronger evidence for the gains over supervised-only training? Some improvements are modest, particularly in the RAG language-modeling experiments. Could you compare the hybrid and supervised-only methods on one additional benchmark? Alternatively, you could repeat one representative experiment and report the average performance and variation across runs. These results would help establish whether adding the policy-gradient term provides a consistent benefit.
4 comments
Rebuttal by Authors (cont.)
[3-seed results and scaled-up MoE] In our original submission, we fixed the random seed to 0 for all methods, so the models are trained from the same initialization. To further demonstrate the statistical significance of the results, we run additional experiments with three random seeds and report the average and standard deviation of the metrics. We also add experiments for for the multi-layer MoE setting, so we now have 10 MoE architectures. The results are shown in the tables below and updated in our revision, where we observe consistent improvements in our method.
MSE loss ($\times$1e-3) on the regression task.
| m=64 | m=128 | m=256 | m=512 | |
|---|---|---|---|---|
| Deterministic | 1.02 (.01) | 2.02 (.01) | 5.71 (.02) | 5.97 (.02) |
| Gaussian | 1.38 (.02) | 2.16 (.02) | 5.13 (.03) | 6.18 (.04) |
| DefaultMoE | 1.19 (.01) | 1.43 (.01) | 1.87 (.02) | 3.30 (.03) |
| SparseMixer | 1.40 (.01) | 1.92 (.02) | 3.12 (.03) | 4.32 (.04) |
| Supervised-only | 0.99 (.01) | 1.42 (.01) | 1.92 (.02) | 4.68 (.03) |
| Ours | 0.95 (.02) | 1.16 (.01) | 1.49 (.03) | 2.85 (.02) |
Top-1 selection accuracy on the regression task.
| m=64 | m=128 | m=256 | m=512 | |
|---|---|---|---|---|
| Deterministic | 89.0 (.1) | 66.9 (.3) | 4.3 (.5) | 0.7 (.3) |
| Gaussian | 80.9 (.3) | 64.1 (.7) | 15.1 (1.6) | 2.2 (1.4) |
| DefaultMoE | 80.5 (.2) | 68.7 (.3) | 53.4 (.8) | 41.2 (1.5) |
| SparseMixer | 86.3 (.4) | 74.3 (.5) | 46.2 (.9) | 26.8 (1.1) |
| Supervised-only | 89.8 (.2) | 80.2 (.6) | 60.4 (1.3) | 11.9 (1.6) |
| Ours | 91.1 (.3) | 84.7 (.6) | 67.0 (1.1) | 47.8 (1.3) |
Perplexity on Wikitext-103 by MoE (one layer).
| m=8 | m=16 | m=32 | m=64 | m=128 | |
|---|---|---|---|---|---|
| Deterministic | 86.30 (.21) | 84.58 (.19) | 86.36 (.41) | 86.01 (.17) | 85.77 (.23) |
| Gaussian | 85.45 (.43) | 82.72 (.68) | 82.56 (.47) | 82.05 (.16) | 83.43 (.38) |
| DefaultMoE | 87.21 (.54) | 89.61 (.28) | 87.45 (.49) | 87.05 (.29) | 84.56 (.47) |
| SparseMixer | 87.92 (.67) | 86.23 (.52) | 86.31 (.61) | 86.21 (.23) | 86.23 (.33) |
| Supervised-only | 84.76 (.43) | 80.87 (.32) | 77.86 (.22) | 74.67 (.45) | 74.04 (.39) |
| Ours | 84.09 (.59) | 79.98 (.41) | 76.63 (.39) | 73.01 (.56) | 71.64 (.52) |
Perplexity on Wikitext-103 by MoE (two layers).
| m=8 | m=16 | m=32 | m=64 | m=128 | |
|---|---|---|---|---|---|
| Deterministic | 71.04 (.09) | 69.38 (.10) | 68.42 (.25) | 69.09 (.43) | 61.88 (.14) |
| Gaussian | 69.54 (.26) | 67.21 (.12) | 67.93 (.24) | 66.36 (.39) | 59.07 (.28) |
| DefaultMoE | 70.68 (.24) | 70.56 (.18) | 69.26 (.51) | 67.43 (.34) | 63.11 (.57) |
| SparseMixer | 71.22 (.39) | 68.34 (.51) | 68.66 (.30) | 69.67 (.37) | 63.42 (.33) |
| Supervised-only | 69.07 (.42) | 64.97 (.73) | 62.87 (.69) | 59.69 (.26) | 53.64 (.36) |
| Ours | 68.49 (.41) | 64.15 (.68) | 61.56 (.52) | 58.12 (.55) | 51.01 (.32) |
Perplexity on wikitext-103 by RAG (Qwen-2.5-1.5B and Llama-3.2-3B).
| Qwen | Llama | |||||
|---|---|---|---|---|---|---|
| m=8 | m=16 | m=32 | m=8 | m=16 | m=32 | |
| Policy-only | 7.51 (.07) | 7.58 (.07) | 7.88 (.08) | 6.70 (.06) | 7.10 (.05) | 7.49 (.05) |
| ICRALM | 6.64 (.05) | 7.02 (.07) | 7.67 (.07) | 5.38 (.05) | 5.57 (.02) | 6.08 (.03) |
| REPLUG | 6.59 (.06) | 6.88 (.05) | 7.62 (.06) | 5.37 (.04) | 5.58 (.02) | 5.96 (.04) |
| Supervised-only | 6.51 (.05) | 6.78 (.06) | 7.26 (.07) | 5.32 (.04) | 5.55 (.03) | 5.88 (.03) |
| Ours | 6.38 (.02) | 6.65 (.05) | 7.15 (.03) | 5.26 (.05) | 5.43 (.04) | 5.76 (.02) |
EM accuracy on NQ.
| Qwen | Llama | |||||
|---|---|---|---|---|---|---|
| m=8 | m=16 | m=32 | m=8 | m=16 | m=32 | |
| Policy-only | 25.1 (.6) | 29.5 (.4) | 30.5 (.4) | 29.9 (.3) | 31.7 (.5) | 33.3 (.4) |
| ICRALM | 27.4 (.3) | 29.7 (.3) | 30.5 (.5) | 30.6 (.3) | 34.2 (.4) | 35.1 (.2) |
| REPLUG | 27.2 (.3) | 29.9 (.3) | 31.4 (.5) | 31.1 (.2) | 32.9 (.6) | 34.7 (.3) |
| Supervised-only | 27.6 (.4) | 31.2 (.5) | 32.5 (.3) | 31.5 (.3) | 34.3 (.3) | 35.4 (.4) |
| Ours | 28.7 (.4) | 32.1 (.3) | 33.5 (.4) | 32.4 (.3) | 35.2 (.3) | 36.7 (.2) |
Answer doc recall on NQ.
| Qwen | Llama | |||||
|---|---|---|---|---|---|---|
| m=8 | m=16 | m=32 | m=8 | m=16 | m=32 | |
| Policy-only | 55.9 (.5) | 62.5 (.6) | 67.4 (.6) | 55.4 (.5) | 63.0 (.4) | 67.4 (.3) |
| ICRALM | 65.3 (.3) | 74.2 (.5) | 81.3 (.2) | 65.1 (.4) | 73.4 (.1) | 79.2 (.2) |
| REPLUG | 64.9 (.3) | 73.4 (.4) | 79.4 (.3) | 64.8 (.2) | 73.1 (.2) | 78.3 (.3) |
| Supervised-only | 65.7 (.5) | 75.3 (.3) | 82.9 (.3) | 65.6 (.3) | 74.1 (.2) | 81.8 (.3) |
| Ours | 67.0 (.4) | 75.9 (.7) | 84.0 (.9) | 66.8 (.3) | 75.2 (.3) | 82.8 (.4) |
EM accuracy on TriviaQ.
| Qwen | Llama | |||||
|---|---|---|---|---|---|---|
| m=8 | m=16 | m=32 | m=8 | m=16 | m=32 | |
| Policy-only | 56.4 (.6) | 58.1 (.5) | 61.1 (.5) | 70.7 (.4) | 71.1 (.4) | 72.9 (.5) |
| ICRALM | 60.2 (.3) | 61.3 (.3) | 62.1 (.2) | 72.2 (.2) | 74.4 (.2) | 74.3 (.4) |
| REPLUG | 60.5 (.4) | 61.7 (.3) | 63.3 (.4) | 72.8 (.2) | 74.7 (.3) | 75.3 (.3) |
| Supervised-only | 61.4 (.2) | 62.7 (.3) | 65.1 (.3) | 73.4 (.5) | 75.2 (.4) | 75.7 (.3) |
| Ours | 62.3 (.3) | 63.9 (.3) | 66.3 (.4) | 74.0 (.3) | 75.9 (.3) | 76.9 (.4) |
Answer doc recall on TriviaQ.
| Qwen | Llama | |||||
|---|---|---|---|---|---|---|
| m=8 | m=16 | m=32 | m=8 | m=16 | m=32 | |
| Policy-only | 80.3 (.4) | 83.7 (.5) | 86.8 (.4) | 80.3 (.5) | 83.8 (.3) | 86.1 (.3) |
| ICRALM | 86.2 (.3) | 89.1 (.2) | 90.8 (.3) | 86.8 (.2) | 90.0 (.2) | 91.2 (.3) |
| REPLUG | 86.6 (.3) | 89.7 (.2) | 91.0 (.3) | 86.5 (.3) | 90.2 (.4) | 91.6 (.4) |
| Supervised-only | 86.9 (.2) | 90.4 (.3) | 92.4 (.3) | 86.9 (.3) | 91.1 (.5) | 92.2 (.4) |
| Ours | 87.5 (.4) | 91.6 (.4) | 94.1 (.2) | 88.0 (.4) | 91.9 (.3) | 93.4 (.3) |
Thank you for the detailed response and substantial additional experiments. The new results have largely addressed my concerns and strengthened the empirical evidence side. I encourage the authors to incorporate these results and clarifications into the revised manuscript. I am happy to raise my score.
Reply to Reviewer Response
Thank you again for your valuable time and review! We are glad to know that the concerns have been addressed. Your insights strengthen this paper, and we will keep the new results and clarifications in our next version.
Rebuttal by Authors
We appreciate the reviewer for the comments, and we address the concerns and clarification questions as follows.
[Probability of the actual sampling order] We thank the reviewer for this insightful question. Following the reviewer’s suggestion, denoting the sampled ordered sequence as S, we use the probability of S as πθ(I) in policy gradient computation, and the rest stays the same. We compare this 'Sequence probability' variant with our proposed method on the regression task.
MSE loss ($\times$1e-3, ↓) on the regression task.
The results show that computing the exact policy gradient (i.e., calculating the set probability rather than the sequence probability) is important. Theoretically, the update of the ‘Sequence probability’ baseline may not correspond to an objective function, and this would yield a biased gradient for the surrogate objective. We have also added the results to Table 4 in the revised paper.
[Memory and time] We thank the reviewer for mentioning this. We would like to clarify that computing π(I) does not require enumerating the k⋅k! orderings: the denominator of each ordering depends only on which experts were drawn before, so the sum factorizes over subsets and can be computed by dynamic programming in O(k⋅2k) per token per selection module. This yields 2,048 rather than 322,560 terms at k=8.
These costs of computing are negligible compared with the dominating costs of running forward and backward passes to optimize the model. Furthermore, the cost is independent of the candidate pool size and model width, so the costs become even more negligible as the model size scales up, a scenario where m-choose-k problems become exponentially more difficult, and our method has more advantage.
To further demonstrate scalability, we report the extra time and peak GPU memory needed for set probability compared to using the O(k) ordered sequence probability. The results are averaged across 20 steps. We use A100 GPU, m=64, batch size = 128, one MoE layer. We observe that in each training step, our method adds 3.2 ms at k=3 and 9.8 ms at k=8, with at most 0.23 GB of additional peak memory. We have added these discussions to Appendix C of the revision.
[Leave-one-out baseline] We thank the reviewer for this suggestion. We would like to highlight that our method is general, and we can use any baseline for the gradient estimation. The leave-one-out baseline is a common and classical baseline for policy gradient. Following the reviewer’s suggestion, we run the suggested experiment on the regression task, replacing the leave-one-out baseline with an exponential moving average of each example’s own past losses.
MSE loss ($\times$1e-3, ↓) for two baselines at m∈{128,512}.
We observe that the optimal λ=0.05, so the variance of the policy term is dominated by subset sampling rather than example difficulty. Therefore, the preference for small λ is not an artifact of the baseline. We have also added this table to Table 3 of the revision.
[Benefit of policy gradient] We thank the reviewer for this question. First, we’d like to clarify that policy gradient provides signals on which subsets should be selected, whereas the supervised-only baseline only differentiates through the aggregation of already selected items, thus only providing signals on how to aggregate the selected items. Second, the supervised-only baseline already incorporates simple adjustments to facilitate exploration, including adding random noise to the item scores and tuning the sampling temperatures from a grid. Another baseline adds Gaussian noise with learned standard deviations [1] for exploration. Therefore, our improvements are relative to advanced baselines that already incorporate exploration. Empirically, we show that the policy term helps improve performance consistently over all the datasets.
[1] Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017.