acceptodds
Under review as a conference paper at ICLR 2027

Batch Diversity under Distortion Constraints for Language Models

Abstract

A batch of responses to one prompt is more useful when the responses differ and each response stays close in distribution to the model. We introduce batch diversity, which quantifies the spread of responses within a batch, and distortion, which quantifies the deviation of the batch's average feature distribution from the model's. We characterize the maximum diversity at every distortion bound. Release selects a batch from a pool of independent responses under the bound, and Release diversity converges to the supremum on the support of the model's features as the pool grows and the optimization error vanishes. Diversity-Adaptive Release (DAR) chooses the number of additional responses from a pilot and keeps the bound. On two language models, Release has higher diversity than independent sampling at zero distortion, and DAR has lower generation cost than Release. Larger distortion bounds tend to raise utility and answer coverage and lower the constant-reward fraction, majority accuracy, and response accuracy.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
batch diversity, distributional distortion, Wasserstein distance, language model sampling, inference-time selection, survey sampling, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Rwj6OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed an algorithm with the goal of sampling a diverse batch of responses under distortion constraints. They first define the notions of distortion and diversity in the probability space, specifically, the original model distribution and the joint distribution for batch sampling. Based on these definitions, they derive the best-diversity can be achieved under distortion constraints, and the lowest distortion under diversity constraints. The method is applied at test time. In practice, the algorithm samples redundant independent responses, and calculate the empirical desired joint distribution and subsample the batch of responses. They also propose an adaptive sampling algorithm along with the non-adaptive version. Through experiments, they show the effectiveness of their method.

Critical strengths
  1. The method is mathematically correct. And the motivation of explicitly controlling the diversity and distortion bound can be useful in some scenarios.
Critical weaknesses
  1. The main concern is that the notion of distortion is defined in the probability space, i.e., regarding the model's distribution as the anchor, measuring the distance with it. This does not align with the usual motivation in this area where people care about the quality of responses when encouraging diversity. Controlling the distortion is just to stay close with the model's distribution, it does not capture the quality of the responses generally and heavily depends on the model's original distribution. From this point of view, I think the overall method framework is not that applicable.

  2. Another concern is the gap between theoretical and practical finite data setting. In the perfect setting, one can derive the bound on distortion and diversity. However, in the finite data setting, the results are not guaranteed even for a weaker version. Proposition 3.2 only holds when NN is infinity. And there is no guidance on how to select the hyper parameters, for example, given a model and task, it is not clear which ρ\rho is appropriate. In practice, the method is basically doing subsampling at test time: first sample a lot of responses and then subsample a batch of response. I didn't see the advantage of the proposed method compared with simple submodular methods.

Questions for authors

No

1 comment

We thank the reviewer for the comments. A revised PDF is uploaded, with changes in blue. We respectfully disagree with the assessment and respond to its four concerns below. In the tables, values in brackets are 95% confidence intervals and pp means percentage points.

Response to the comment that distortion does not capture the quality of responses.

Measuring distortion against the model's distribution lets the method preserve quality without committing to one quality score: at ρ = 0 every returned response has exactly the model's distribution (Proposition C.3), so the expected value of any quality measure of a response (accuracy, a reward, a judge's score) is the same as under ordinary sampling. A positive ρ is a budget that the user may spend for more diversity. Every method that samples from a fixed model, including MMR and k-DPP, depends on that model's distribution.

The experiments at ρ = 0 are consistent with this. The table gives Release (N = 64) minus independent sampling for the eight metrics of Table 32.

MetricLlamaQwen
Response accuracy (pp)−0.13 [−0.28, 0.00]−0.02 [−0.04, 0.00]
NoveltyBench utility+0.161 [−0.010, 0.338]+0.049 [−0.176, 0.267]
NoveltyBench distinct+0.065 [−0.022, 0.155]+0.055 [−0.043, 0.150]
Embedding diversity+0.150 [0.119, 0.180]+0.145 [0.123, 0.168]
Embedding distortion−0.018 [−0.031, −0.005]−0.021 [−0.030, −0.012]
Constant-reward groups (pp)−4.83 [−6.17, −3.58]−1.10 [−1.93, −0.40]
pass@8 (pp)+2.13 [1.33, 3.01]+0.09 [−0.03, 0.26]
majority@8 (pp)+3.31 [1.80, 4.80]+0.08 [0.00, 0.24]

Every difference is in the favorable direction or within 0.13 pp of zero. After Holm correction over all comparisons of that table, the differences in diversity and in constant-reward groups hold on both models, and those in pass@8 and majority@8 on Llama (Appendix E.7). Qwen already has 96–97% pass@8.

Response to the comment that the results are not guaranteed with finite data.

The guarantee on distortion is a finite-sample statement. For a feature map fixed in advance, D ≤ ρ holds for every pool size N (Proposition 3.1) and every response limit T (Proposition 3.2). On every finite pool, Release returns a selection that satisfies the pool-level constraint and approximates the most diverse one (problem (8)), exactly for class features (Appendix C.1) and with a certified gap below 10⁻⁶ in the embedding experiments (Appendix D.2). Only the comparison of this diversity with its supremum on the support of the model's features is asymptotic, as the paper says. In a check added in the revision (Appendix F.13; observed class distribution of each prompt as the population), the mean diversity of Release at ρ = 0 is within 0.88% (B = 8) and 1.1% (B = 16) of the mean supremum at N = 64 on Llama. The revision states "for every N" and "for every T" in both propositions.

Response to the comment that there is no guidance on how to select the hyperparameters.

ρ = 0 needs no selection and is the setting of the comparisons above. For more diversity, ρ is a budget with a direct meaning: how far the returned responses may move from the model's distribution, in units of the model's own spread. With class features, ρ = √(2ε(1 − 1/N)) guarantees that on every prompt Release moves at most a share ε of the probability mass between classes, so the accuracy of a returned response changes by at most ε relative to its pool. No such bound exists for the temperature or for the λ of MMR. The revision adds this rule (Section 3.1).

Response to the comment that the method is subsampling at test time and has no advantage over simple submodular methods.

Release, MMR, and k-DPP all select a batch from a pool, and the paper compares them on the same 64-response pools (Section 4.3). The contribution of the paper is what the selection does to the responses. MMR and k-DPP place no constraint on the distribution of the selected responses, and Release selects for diversity subject to a bound on distortion. The table shows the difference on Llama (Tables 3 and 38).

MetricRelease, ρ = 0MMR, λ = 0.3MMR, λ = 0.5MMR, λ = 0.7k-DPP
GSM8K: response accuracy minus that of 128 other independent responses (pp)−0.1−5.3−0.2+3.7−2.1
MATH-500: the same difference (pp)−0.0−5.3−1.0+4.3−2.8
GSM8K majority@8 (%)90.586.090.088.086.5
MATH-500 majority@8 (%)63.450.055.064.061.5

Release at ρ = 0 leaves the accuracy of the responses within 0.1 pp, whereas MMR shifts it by between −5.3 and +4.3 pp depending on λ and k-DPP by −2.1 and −2.8 pp. In majority@8, Release at ρ = 0 is within 0.6 pp of the best setting of MMR on each dataset, with no λ to choose. On Qwen, where pass@8 is 96–97%, MMR and k-DPP shift the accuracy by at most 0.6 pp.

Reviewer_NnXHOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work studies how to select a diverse batch of language model responses to a single prompt from a larger pool of N candidates without distorting the model’s underlying output distribution. The authors formalise the trade-off between batch diversity and marginal distortion by deriving the theoretical upper bound on diversity for any distortion budget. They introduce two subset-selection algorithms: Release, which solves a convex program to achieve asymptotically optimal batch diversity within a given distortion bound, and Diversity-Adaptive Release (DAR), which uses a small pilot sample to allocate generation budgets per prompt. They boost diversity on open-ended generation and math reasoning using Llama-3.1-8B and Qwen3-8B and improves pass@8.

Critical strengths
  1. Unlike prior works that rely on asymptotic heuristics, this paper derives the exact supremum of batch diversity under 2-Wasserstein distortion constraints

  2. The introduction of Diversity-Adaptive Release (DAR) solves the primary system bottleneck of candidate pool generation.

  3. The authors connect their diversity metrics to RL alignment, showing that raising the distortion bound to spread batches over rarer classes significantly reduces the fraction of uninformative "constant-reward" groups in GRPO

  4. The authors evaluate their methods on both standard instruct and reasoning models across multiple domains, report descriptive 95% confidence intervals, and transparently document all experimental details, including solver times and failed baseline generations

Critical weaknesses
  1. Coarse Mathematical Evaluation (Only pass@8pass@8): The reasoning evaluation is restricted to a single, fixed batch size B=8. The paper does not show how diversity or accuracy scales across a spectrum of batch sizes (such as pass@1pass@1, pass@2pass@2, etc.), leaving it unclear if the gains of Release hold at smaller, more computationally practical batch sizes.

  2. No Active Policy Gradient Training: While the paper proves that Release and DAR improve GRPO group reward statistics (Table 4), the authors do not conduct actual policy gradient training.

  3. No Evaluation of Reasoning Trace ("Thinking Block") Diversity: Since multiple distinct thinking paths can lead to the same final math answer, measuring diversity only on the final extracted answer severely underestimates the semantic diversity of the reasoning traces.

  4. Heavy Sensitivity to Manual Feature Map Design: The metrics of diversity and distortion rely entirely on the choice of the feature map. The authors use BGE embeddings for open-ended text and one-hot indicators for math. The system's performance is highly dependent on manual engineering.

  5. Missing relevant baselines like Ahmed et al. which provides multiple ways to measure diversity in addition to another relevant solution to compare against.

  6. Minor point: The abstract does not clearly delineate the algorithms. The names are confusing as they look like normal nouns which make the sentences grammatically incorrect. Please rephrase or at least use proper visual indicators. Also the sentence motivating DAR is confusing and the benefit is not clear until much further into the paper.

Ahmed, Eltayeb, et al. "Intent factored generation: Unleashing the diversity in your language model." arXiv preprint arXiv:2506.09659 (2025).

Questions for authors

Q1. How sensitive is the method to temperature and sampling method?

Q2. Can you please add the relevant baselines or at least rebut why you think you should not?

I think the paper has clear merits and I am willing to increase my score.

6 comments

We thank the reviewer for the constructive comments. A revised PDF is uploaded, with changes in blue. We answer the two questions first and then the weaknesses in the order of the review. The new results are collected in the new Appendix F. In the tables, values in brackets are 95% confidence intervals and pp means percentage points.

Response to Q1, on the sensitivity to temperature and sampling method.

The gain of Release over random batches from the same pool is positive in every setting that we tried. We crossed four temperatures (0.6, 0.8, 1.0, 1.2) with three sampling methods (full distribution, top-p 0.9, min-p 0.1), which gives 12 settings for each model.

On NoveltyBench we generated 16 responses per prompt (100 held-out prompts) in each setting and compared Release at ρ = 4 with random batches of four from the same 16 responses. The table reports the gain of Release. The temperature rows average over the three sampling methods, and the sampling-method rows average over the four temperatures.

SettingLlama, distinctLlama, utilityQwen, distinctQwen, utility
Temperature 0.6+0.42+0.81+0.37+0.74
Temperature 0.8+0.38+0.73+0.39+0.80
Temperature 1.0+0.36+0.58+0.44+0.90
Temperature 1.2+0.37+0.48+0.37+0.72
Full distribution+0.38+0.59+0.41+0.80
Top-p 0.9+0.38+0.60+0.39+0.80
Min-p 0.1+0.39+0.76+0.38+0.77

The table shows averages. Taken one by one, the 12 settings on the two models give 24 cases. In every one of them Release at ρ = 4 is higher than random batches: the gain is between 0.31 and 0.51 in distinct and between 0.26 and 1.03 in utility, and its confidence interval is above zero. We also repeated the comparison with Release at ρ = 0. The gain is again positive in all 24 cases, and its confidence interval is above zero in 14 of them for distinct and in 13 for utility.

On the mathematical tasks we ran the same grid on Llama at ρ = 0. In 11 settings the generator of the paper drew a new pool of 64 responses for each of the 200 held-out problems, and the setting of the paper (temperature 1.0, full distribution) uses the pools of the paper. Release is compared with independent sampling from the same pool. The table gives Release minus independent sampling, exact over the selection. The values in the row of the paper's setting differ from Table 32, which was estimated by Monte Carlo.

TemperatureSampling methodpass@8 (pp)majority@8 (pp)Constant-reward groups of 16 (pp)
0.6full distribution+1.39 [0.80, 2.04]+2.57 [1.20, 4.02]−4.56 [−5.69, −3.49]
0.6top-p 0.9+1.57 [0.92, 2.30]+1.48 [0.13, 2.83]−4.18 [−5.30, −3.12]
0.6min-p 0.1+1.68 [1.02, 2.41]+1.50 [0.18, 2.87]−4.70 [−5.88, −3.57]
0.8full distribution+2.52 [1.65, 3.44]+2.50 [1.15, 4.03]−5.89 [−7.15, −4.62]
0.8top-p 0.9+1.50 [0.86, 2.18]+2.35 [1.00, 3.68]−4.51 [−5.57, −3.47]
0.8min-p 0.1+1.53 [0.89, 2.32]+2.42 [1.05, 3.78]−4.89 [−6.05, −3.78]
1.0full distribution (pools of the paper)+2.15 [1.39, 2.97]+3.28 [1.85, 4.81]−5.19 [−6.39, −4.04]
1.0top-p 0.9+2.30 [1.55, 3.16]+1.66 [0.56, 2.76]−5.79 [−6.99, −4.62]
1.0min-p 0.1+2.04 [1.25, 2.85]+2.25 [0.83, 3.74]−5.51 [−6.60, −4.38]
1.2full distribution+2.90 [2.10, 3.78]+3.24 [1.79, 4.85]−3.03 [−3.89, −2.16]
1.2top-p 0.9+2.63 [1.76, 3.57]+2.22 [0.84, 3.68]−6.00 [−7.21, −4.87]
1.2min-p 0.1+2.12 [1.31, 2.99]+2.75 [1.40, 4.11]−5.37 [−6.53, −4.21]

In all 12 settings Release has higher pass@8 (by 1.4 to 2.9 pp) and majority@8 (by 1.5 to 3.3 pp) and fewer constant-reward groups (by 3.0 to 6.0 pp), and every interval excludes zero. The level of accuracy depends on the setting while the sign of the gain does not: at temperature 1.2 with the full distribution, 15% of the attempts reached the length limit, and pass@8 is 80.4% for Release and 77.5% for independent sampling. With temperature 0.8 and min-p 0.1, one problem had fewer than 8 complete responses and is left out. For compute reasons this grid covers Llama only. Qwen already reaches 96–97% pass@8 on these problems.

Section 4.3 now states this for the mathematical tasks, and Appendix F.2 reports every setting.

Response to Q2 and to the comment on missing baselines such as Ahmed et al.

We added Intent Factored Generation (IFG; Ahmed et al., 2025) as a baseline on both tasks of the paper. At the end of this response we also explain how the baselines of the paper were chosen.

For NoveltyBench (B = 4, 100 held-out prompts), the IFG paper does not provide a prompt for open-ended chat, so we wrote one. The model first writes the intent of its answer itself, on one line that starts with "Intent:" and lists a few keywords, and then writes the answer after that line. The intent is sampled at temperature 1.0 or 1.3, and the answer at 0.7 for Llama and 0.6 for Qwen. Only the answer is scored. The temperatures were fixed before the runs, and the IFG rows show the better of the two intent temperatures (1.3 for Llama and 1.0 for Qwen).

ModelMethodResponses generateddistinctutility
LlamaIFG162.385.20
LlamaRelease, ρ = 4, on 16 independent responses163.105.49
LlamaRelease, ρ = 4, as in the paper643.145.74
QwenIFG161.984.53
QwenRelease, ρ = 4, on 16 independent responses162.305.87
QwenRelease, ρ = 4, as in the paper642.446.21
Release as in the paper − IFGdistinctutility
Llama+0.76 [0.61, 0.91]+0.54 [0.15, 0.95]
Qwen+0.46 [0.32, 0.61]+1.68 [1.32, 2.04]

"On 16 independent responses" means that Release selects from 16 responses generated with the sampling settings of the paper, the same number as IFG generates. For the rows with 16 responses, the standard error of each mean over the prompts is about 0.10 for distinct and 0.22 for utility.

For the mathematical tasks (ρ = 0) we used the MATH code released by the IFG authors, with their few-shot prompt and their alternating intent and step temperatures. We also followed their random search for the two temperatures: ten pairs drawn uniformly, with the intent temperature in (0, 1.1) and the step temperature in (0, 0.7). We kept the pair with the best pass@8 on our 200 calibration problems. In the order intent/step, it is 0.34/0.53 for Llama, 0.97/0.48 for non-thinking Qwen, and 0.64/0.56 for thinking Qwen. An IFG batch consists of B of its attempts on a problem, drawn at random among its first 64 complete attempts (those that ended within the length limits). Release and independent sampling use the pools of the paper, generated at temperature 1.0 for Llama and 0.6 for Qwen.

ModelMethodpass@8 (%)majority@8 (%)Constant-reward groups, B = 16 (%)Response accuracy, B = 16 (%)
LlamaIFG87.576.234.767.3
LlamaRelease87.278.419.058.9
LlamaIndependent85.175.124.458.9
QwenIFG, non-thinking95.491.472.487.3
QwenIFG, thinking96.092.575.788.9
QwenRelease96.796.094.295.6
QwenIndependent96.695.995.295.6
LlamaIFG − Release (pp)+0.3 [−2.0, 2.6]−2.2 [−4.9, 0.5]+15.6 [10.8, 20.7]+8.3 [5.9, 10.6]

The Llama rows are computed on the 196 (B = 8) and 193 (B = 16) problems for which IFG produced at least B complete attempts, and the Qwen rows on all 200 problems. For Qwen, the non-thinking mode is the procedure of the IFG paper. In thinking mode with an intent line first, the responses of Qwen are shorter than in our pools (629 generated tokens on average versus about 3,900). The shorter responses are a limitation of our adaptation.

As for the choice of baselines, the paper takes representatives from three groups of methods in its introduction, and IFG now joins the prompting group.

GroupBaselineWhy this baseline
PromptingIn-context regeneration (Zhang et al., 2025)NoveltyBench's own protocol for asking for a different answer
PromptingVerbalized sampling (Zhang et al., 2026)A recent prompting method against repeated answers
PromptingIFG (Ahmed et al., 2025), addedFirst an intent, then the response given the intent, each with its own temperature
Joint decodingArithmetic sampling (Vilnis et al., 2023)Keeps the distribution of each response, as Release does at ρ = 0
Joint decodingStochastic beam search (Kool et al., 2019)Samples sequences without replacement
Selection from a poolMMR (Carbonell & Goldstein, 1998)Classical diversity-based selection, on the same 64-response pools as Release
Selection from a poolk-DPP (Kulesza & Taskar, 2011)Classical diverse-subset sampling, on the same pools

In the revision, IFG is added to Figure 3 and Table 22, Section 4.3 states how the baselines were chosen, and Appendix F.3 gives the comparison on the mathematical tasks, the IFG prompts, and the temperature search.

Response to the comment that the mathematical evaluation uses a single batch size (W1).

The gains also hold at smaller batch sizes. In fact the gain in pass@B, the reduction in constant-reward groups, and the gain in diversity all become larger as B decreases. The tables compare Release with independent sampling from the same 64-response pool on Llama at ρ = 0, on the 100 GSM8K and 100 MATH-500 held-out problems.

GSM8K, Bpass@B, Release (%)pass@B, independent (%)Gain (pp)majority@B, Release (%)majority@B, independent (%)Gain (pp)
290.684.6+6.02 [4.47, 7.63]72.171.8+0.25 [−1.17, 1.63]
494.691.8+2.75 [1.75, 3.85]85.181.8+3.31 [1.24, 5.40]
895.995.1+0.82 [0.32, 1.42]90.287.7+2.54 [0.81, 4.40]
MATH-500, Bpass@B, Release (%)pass@B, independent (%)Gain (pp)majority@B, Release (%)majority@B, independent (%)Gain (pp)
260.554.9+5.62 [4.32, 7.18]41.840.6+1.27 [−0.28, 2.85]
469.764.7+4.97 [3.41, 6.73]51.250.9+0.37 [−2.05, 2.81]
875.171.6+3.48 [2.10, 5.02]63.559.5+4.00 [1.70, 6.31]
GRPO groups, both datasets, BConstant-reward groups, Release (%)Constant-reward groups, independent (%)Div, Release − independent
262.774.4+0.209 [0.188, 0.233]
443.653.6+0.116 [0.106, 0.125]
830.438.0+0.063 [0.058, 0.067]
1621.226.4+0.031 [0.029, 0.032]

For majority@B the confidence interval of the gain contains zero at B = 2 on both datasets and at B = 4 on MATH-500. On Qwen the differences in pass@B are at most 0.2 pp at these batch sizes, because pass@B is already 96–97% for both methods. The values differ from Tables 3 and 4, which were estimated by Monte Carlo.

Section 4.3 now states this, and Appendix F.4 reports both models and datasets, ρ ∈ {0, 1, 2}, and pools of 64, 128, and 4B responses.

Response to the comment that no policy-gradient training was conducted (W2).

We ran this training with Llama-3.1-8B-Instruct (LoRA, GRPO, six seeds, ρ = 0). Every arm has the same budget of generated rollout tokens (5×10⁷ on GSM8K and 10⁸ on MATH), with 64 prompts per step and groups of 16. The table reports the accuracy of the last model evaluated within the budget.

ArmGSM8K accuracy (%)MATH accuracy (%)Updates within the budget, GSM8KUpdates within the budget, MATH
Plain GRPO, 16 attempts per prompt89.0255.61224255
DAR's pool, every attempt used, classes spread evenly over the groups89.5655.66211294
Release's pool of 64, every attempt used, classes spread evenly over the groups89.2555.48208271
Control: 64 attempts, every attempt used, advantages over the whole pool89.5955.13209270
Control: 64 attempts, every attempt used, random groups of 1689.1055.88208268
DAR, 16 selected from its pool, the rest discarded88.3254.826396
Release, 16 selected from 64, the rest discarded87.9354.345359

When 16 responses are selected and the rest are discarded, as in Table 4, the accuracy is below plain GRPO. At equal tokens the arms that discard attempts make about four (Release) and three (DAR) times fewer updates, because the discarded attempts also cost tokens. These arms end 0.7 to 1.3 pp below plain GRPO.

When every attempt of the same pools is used, the accuracy is at or above plain GRPO on GSM8K and within the standard deviation of plain GRPO across seeds on MATH (0.38 pp on GSM8K and 1.46 pp on MATH). Among the Release and DAR arms, the pool of DAR gives the highest accuracy on both datasets. On GSM8K it is 0.54 pp above plain GRPO and higher in each of the six seeds, and on MATH the two are 55.66% and 55.61%. The two control arms that also use every attempt are within 0.6 pp of plain GRPO.

MATH here means the MATH problems with numeric answers, and we trained Llama only.

The result is now reported in Section 4.3, and Appendix F.5 lists every arm and seed.

Reviewer_psspOpenReview ↗
Rating
2(+1)
Confidence
1(−2)
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper formalizes batch sampling from an LLM through two quantities measured in a feature space.

Diversity is the normalized expected pairwise distance within a batch, and distortion is the W2W_2 distance between the batch's averaged marginal and the model distribution. It derives the closed-form optimal diversity–distortion frontier and a finite-pool selection rule that guarantees bounded distortion.

Two methods follow: Release, a linear-programming selector over a pool of i.i.d. samples, and DAR, which adapts the pool size from a small pilot. On NoveltyBench, math pass@k/maj@k, and GRPO group statistics with Llama-3.1-8B and Qwen3-8B, Release at zero distortion increases embedding diversity by about 15–18%. Larger distortion budgets trade accuracy for coverage, and DAR cuts generation cost by about 60% at some loss of diversity.

Critical strengths
  • The paper has a clean formulation. Separating diversity (the joint dependence) from distortion (the averaged marginal) is a principled way to state the diversity–faithfulness trade-off. The closed-form frontier (Thm 2.3, Cor 2.4) is tight and attained.

  • Proofs are careful (measurability, finite-pool feasibility, convergence along all N). Certified optimality gaps are reported. The appendix is unusually candid about negative or neutral results, for example, the matched-cost controls for DAR and the order-dependent NoveltyBench partitions.

Critical weaknesses
  1. Presentation. Section 2 is notation-dense with little intuition, and the key practical facts (the one-hot reduction, the inactive constraint, matched controls) are buried in a 45-page appendix.

  2. The GRPO analysis is speculative.

  • No training is performed.
  • Even at ρ = 0 the group members are negatively dependent. That changes the bias and variance of the group-mean baseline and of the normalized advantage, and this is not analyzed.
  • At ρ > 0 the groups are off-policy.
  • The claim that groups become “more informative” is unsupported.
  1. Limited novelty at ρ = 0.
  • The theory reduces to a variance decomposition (Lemma A.2) plus a triangle inequality and a radial-scaling construction (Eq. 16).
  • With one-hot features, which are used for all downstream tasks, Release at ρ = 0 is stratified or systematic sampling over clusters.
  • The paper should state this explicitly and compare against the obvious baseline “cluster the pool, then stratified-sample”, which needs no LP.
  1. ρ > 0 lacks semantic meaning and practical guidance.

  2. Weak downstream evidence where the guarantee holds (ρ = 0).

  3. Dependence on features and the diversity measure. The connection between the optimized objective and the evaluated metric is loose.

Questions for authors
  1. Can you provide a principled or task-driven way to choose ρ? Can distortion be tied to a task quantity, such as a bound on the change in expected reward via a Lipschitz assumption on features?

  2. How does Release at ρ = 0 differ, theoretically and empirically, from cluster-then-stratified sampling with one-hot features? Please add this baseline, together with equal-budget selection baselines on the same 64-sample pool.

  3. For ρ > 0 on NoveltyBench, where the pool-dependent classes void the guarantee, what does “distortion ≤ ρ” actually certify?

  4. At ρ = 0, how does negative dependence within a GRPO group affect the expectation of the policy-gradient estimator with a group-mean or standardized baseline? Is it still unbiased?

  5. Given Tables 16–19, would a non-adaptive, cost-matched pool size suffice, making the pilot policy unnecessary?

  6. What is the total wall-clock cost, including embedding, classification, and the LP, relative to generation?

7 comments

We thank the reviewer for the careful and detailed review. A revised PDF is uploaded, with changes in blue. The review rates the concerns on soundness as minor or fixable, and we have acted on each of its six questions and six weaknesses. Following the review, we ran the GRPO training (W2), derived the expectation of the policy-gradient estimator under negative dependence (Q4), added the stratified baseline for one-hot features (Q2), give a rule for choosing ρ with class features (Q1), estimated the wall-clock cost from measured components (Q6), evaluated on harder problems (W5), rescored the batches with encoders that no method uses for selection (W6), and revised Sections 2 to 4 (W1). Each answer below says where the change appears in the revision. The new results are collected in the new Appendix F, organized by the reviewers' questions, and the main text states the main conclusions. We answer the questions first and then the weaknesses. In the tables, values in brackets are 95% confidence intervals and pp means percentage points.

Response to Q1, on how to choose ρ, and to the comment that ρ > 0 lacks meaning and guidance (W4).

ρ can be chosen from the amount of change that the task tolerates. A larger ρ allows more diversity (Theorem 2.3), and with one-hot features Release moves at most a share ρ²/(2(1 − 1/N)) of the probability mass between classes. One can therefore fix a tolerated share ε and take the largest ρ that keeps the bound at most ε, ρ = √(2ε(1 − 1/N)). For example, ε = 1 pp gives ρ ≈ 0.14 at N = 64.

The tolerance itself is left to the user. We do not say which ε is appropriate for which task, and this remains a limitation. With embedding features the bound below has an unknown constant, and ρ = 0 is the default.

Distortion can indeed be tied to a task quantity. For a reward g that is L-Lipschitz in the features, |E_Π̄ g − E_P₀ g| ≤ L·s₀·ρ at every finite N. The inequality follows from the bound D ≤ ρ of Proposition 3.1, because W₁ ≤ W₂. With one-hot features, Release moves at most a share ρ² s_x²/(2(1 − 1/N)) of the probability mass between classes relative to the pool, where s_x² = 1 − Σ_j p̂_j² and p̂_j are the class shares of the pool. The accuracy of a returned response therefore changes by at most that amount. Since s_x² ≤ 1, this gives the bound used above.

The table shows the rule at work (Llama, Release with N = 64 and B = 8, the 200 held-out problems). The bound is the mean over prompts of ρ² s_x²/(2(1 − 1/N)), and the observed change is the accuracy of a returned response minus that of its pool.

ρBound on the change in accuracy (pp)Observed change (pp)Divpass@8 (%)majority@8 (%)
00.000.001.03685.576.9
0.10.24−0.131.04285.576.8
0.20.95−0.511.06285.676.7
0.32.13−1.151.09385.876.5
0.55.91−3.221.19186.274.8
123.7−11.871.56287.162.9

The bound holds on every prompt, and the observed change is about half of it. Between ρ = 0.2 and 0.5 the diversity is 2.5% to 15% above its value at ρ = 0, for a change in accuracy of 0.5 to 3.2 pp.

With one-hot features this bound also gives ρ > 0 a meaning: through it, ρ limits the share of probability mass that Release may move between classes. For a tolerance of a few pp the rule gives values well below the ρ = 1, 2, 4 used in the paper, so at those values the bound does not guarantee such a tolerance. The values 1, 2, and 4 were chosen to cover the trade-off between diversity and distortion and are not recommendations for tasks with a correct answer. The two bounds and the choice of ρ from a tolerance are now in Section 3.1, the table above is in Appendix F.8, and the remaining limitation is stated in the limitations.

Response to Q2, on the difference from cluster-then-stratified sampling and on equal-budget selection baselines.

With one-hot features at ρ = 0 the two coincide, and the difference lies outside that case: Release applies the same optimization, problem (8), to features without given clusters and to ρ > 0 (see W3 for features without given clusters). In the one-hot case Release is proportional stratified sampling with systematic rounding: Eq. (37) gives the expected number a_j = B·p_j of batch members in class j, Eq. (25) forces the actual number K_j to be ⌊a_j⌋ or ⌈a_j⌉, and members are chosen uniformly within a class. For class features we run this closed form, and the LP of Release is needed only for features without given classes. Any randomized rounding with these counts and E K_j = a_j has the same distribution of the count in each class, and therefore the same diversity, pass@B, and constant-reward fraction. We confirmed this with the cube method (Deville and Tillé, 2004) on every 64-response mathematical pool with more than one answer class (195 pools of Llama and 20 of Qwen).

A stratified baseline can therefore differ from Release only in how it rounds. As such a baseline we added largest-remainder rounding (Llama, N = 64, the 100 GSM8K and 100 MATH-500 held-out problems together).

MetricReleaseLargest remainderLargest remainder − Release
pass@8 (%)85.5087.25+1.75 pp [0.50, 3.31]
majority@8 (%)76.8877.71+0.83 pp [−0.13, 1.94]
Div, B = 81.0360.862−0.174 [−0.232, −0.122]
Constant-reward groups, B = 16 (%)21.2522.31+1.06 pp [−0.40, 2.49]
Response accuracy, B = 16 (%)56.9357.67+0.74 pp [0.53, 0.94]
Pools with nonzero distortion, B = 80189+189

With largest-remainder rounding the counts are fixed given the pool, except for ties that are broken at random. The rule is therefore not at zero distortion, and it changes the distribution of the responses: its response accuracy is 0.74 pp above that of Release on the same pools (B = 16). Without zero distortion it can reach a higher pass@8. By the argument of Example B.3, when the correct responses form one class, no selection rule at zero distortion relative to the pool has higher pass@B than Release.

MMR and k-DPP in the paper are equal-budget selection baselines on the same 64-response pools. The results of MMR and k-DPP at ρ = 0 are collected below from Tables 3, 4, 22, and 38.

LlamaRelease, ρ = 0MMR, λ = 0.3MMR, λ = 0.5MMR, λ = 0.7k-DPP
GSM8K pass@8 (%)95.993.096.097.095.0
GSM8K majority@8 (%)90.586.090.088.086.5
MATH-500 pass@8 (%)74.967.074.075.072.5
MATH-500 majority@8 (%)63.450.055.064.061.5
GSM8K constant-reward groups (%)17.721.023.030.023.0
MATH-500 constant-reward groups (%)24.925.026.028.025.0
GSM8K: response accuracy minus that of 128 other independent responses (pp)−0.1−5.3−0.2+3.7−2.1
MATH-500: the same difference (pp)−0.0−5.3−1.0+4.3−2.8
NoveltyBench distinct2.793.123.072.942.81
NoveltyBench utility5.125.155.365.685.16

MMR and k-DPP have no constraint on the distribution of the responses. On Llama their response accuracy moves by −5.3 to +4.3 pp, whereas that of Release stays within 0.1 pp. On Qwen, where pass@8 is 96–97%, the shifts in accuracy are at most 0.6 pp, pass@8 and majority@8 of the selectors are within 1.0 pp of Release, and the constant-reward fractions of the selectors lie between 3.4 pp below and 0.6 pp above the fraction of Release. On NoveltyBench at ρ = 0 the selectors are above Release. The reduction is now stated in Section 3.1, and the rounding baseline is reported in Appendix F.9.

Response to Q3, on what "distortion ≤ ρ" certifies on NoveltyBench when ρ > 0.

In that setting the bound certifies only a statement about the partition of the pool. Averaged over the selection, the class shares of a batch returned by Release are within total variation ρ² s_x²/(2(1 − 1/N)) of the class shares of the pool. For DAR the reference is the distribution R_T of Eq. (9) and not the pool. The bound does not certify a distance to the model's distribution, because the classes are formed from the pool and Proposition 3.1 does not apply.

At the values used in Figure 3 the bound reaches 1 once s_x² ≥ 0.49 at ρ = 2 and once s_x² ≥ 0.12 at ρ = 4. In that case the constraint is inactive, and Release returns the most diverse batch that the classes of the pool allow (Appendix E.4). Section 4.3 now says this where Figure 3 is discussed and presents the NoveltyBench results at ρ > 0 as empirical.

Response to Q4, on how negative dependence affects the policy-gradient estimator, and to the related point of W2.

In general the estimator is not unbiased, for either kind of group. With the group-mean baseline, the factor on ∇J is the same for every prompt for independent groups, whereas for Release it can differ between prompts. With standardized advantages the factor depends on the prompt in both cases.

The setting is as follows. A group returned by Release at ρ = 0 with class features from a pool of N ≥ B ≥ 2 responses is exchangeable, each Y_i ∼ π when averaged over pools, and given the classes of the group its responses are independent draws from π within their classes. The reward is 1 on the gold class g and 0 otherwise, J = p_g with 0 < p_g < 1, and s = ∇log π. Then

E[(1/B) Σ_i (r_i − r̄) s_i] = ((B − 1)/B)·(∇J − Σ_c δ(c) ∇p_c), where δ(c) = P(r_j = 1 | c(Y_i) = c) − p_g for j ≠ i.

QuantityIndependent groupsRelease, pool of N
δ0nonzero in general
Group-mean baseline, two classes: factor on ∇J1 − 1/B1 − Var(r̄)/(p_g(1 − p_g)), between 1 − 1/B and 1 − 1/N
The same at B = 16, N = 640.9375between 0.9375 and 0.984
Var(r̄)p_g(1 − p_g)/Bp_g(1 − p_g)/N + E[f(1 − f)]/B², never larger
Estimator with standardized advantages: expectationF·∇JF·∇J + Σ_{c≠g} e_c ∇p_c
F, for B ≥ 3above 1above 1

The notation in the table is as follows.

  • f is the fractional part of B·p̂_g, where p̂_g is the share of correct answers in the pool.
  • F = E[σ̂²/(σ̂ + ε)]/(p_g(1 − p_g)) for the advantage (r_i − r̄)/(σ̂ + ε), where σ̂ is the standard deviation of the rewards in the group (σ̂² = r̄(1 − r̄)) and ε = 10⁻⁴.
  • e_c is the mean advantage of a response in class c minus the p-weighted mean of these means over the wrong classes.

We add three remarks:

  • With two classes the direction of the gradient is kept within a prompt. With several wrong classes the group-mean expectation has a further term −((B − 1)/B)·Σ_{c≠g} ε_c ∇p_c, where ε_c is δ(c) minus its p-weighted mean over the wrong classes.
  • The baseline r̄ is unbiased for p_g in both cases.
  • The standardized advantage has mean 0 and variance E[σ̂²/(σ̂ + ε)²] for both kinds of group. Up to a term of order ε, this variance is the share of groups that are not constant-reward, which is larger for Release (Table 4).

To give an idea of the sizes, we computed these quantities for N = 64, taking the class shares of each stored pool as the class distribution of the policy, so they are not averaged over new pools.

QuantityValue
Largest absolute ε_c, mean over Llama's 176 pools with both correct and incorrect answers, B = 160.025
Ratio of F for Release to F for independent groups, mean over the same 176 pools of Llama, B = 81.14
The same ratio, Llama, B = 161.10
The same ratio, mean over Qwen's 17 pools with both correct and incorrect answers, B = 81.11
The same ratio, Qwen, B = 161.17
Largest absolute e_c, mean on Llama, B = 8 (where F ≈ 2.5)0.17

The statement, its proof, and this computation are now in Appendix F.6, and two sentences are added to Section 4.3.

Response to Q5, on whether a non-adaptive, cost-matched pool size would suffice.

At ρ = 0 a cost-matched fixed pool size does not differ from DAR beyond the confidence intervals, with one exception that favors DAR. The table gives the control minus DAR at ρ = 0, taken from the tables of the paper. The control is Release with a mixture of two adjacent fixed pool sizes whose mean cost matches that of DAR on calibration prompts. The token differences of the control on held-out prompts are reported in the same tables.

MetricLlamaQwen
Embedding diversity (Table 16)+0.002 [−0.007, 0.012]+0.006 [−0.002, 0.015]
NoveltyBench utility (Table 17)+0.082 [−0.103, 0.258]+0.140 [−0.036, 0.311]
pass@8, pp (Table 18)−0.0 [−1.0, 0.8]−0.0 [−0.2, 0.0]
Constant-reward GRPO groups, pp (Table 19)+3.3 [1.2, 5.6]+0.0 [−0.9, 1.2]

The only difference whose confidence interval excludes zero is the constant-reward fraction on Llama, where DAR has fewer constant-reward groups. Against the best mixture of any two pool sizes at the cost of DAR, the difference is 3.6 pp [1.5, 5.0] at T = 64 and 3.2 pp [1.1, 4.7] at T = 128. In training, however, groups selected by DAR did not improve on plain GRPO (see W2).

At ρ > 0 DAR has no consistent downstream advantage. On the mathematical tasks with Llama, the best mixture of two pool sizes at the cost of DAR has 10 to 25 pp higher majority@8 at ρ = 1 and 2. On Qwen at ρ = 2 the embedding diversity of DAR is 0.062 [0.032, 0.094] above that of the mixture control of the table, but DAR also generates 13.8 thousand more tokens per prompt [9.2, 18.1] (Table 16). Section 4.2 now summarizes the matched-cost controls together with this conclusion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.