acceptodds
Under review as a conference paper at ICLR 2027

DeSPO: Decoupling Rollout Budget from Set-Valuation Horizon in Policy Optimization

Abstract

Set-level objectives capture interactions among responses in reinforcement learning for large language models (LLMs). Yet scoring only the realized response set ties its valuation scope to the rollout budget. We introduce Decoupled Set Policy Optimization (DeSPO), separating the real rollout budget G from an independently chosen set-valuation horizon K. DeSPO optimizes probability weights over the generated responses using task rewards and the expected utility of K draws with replacement from the same fixed pool. The resulting target guides policy updates without additional text generation. For a determinantal utility, we derive an exact finite-support expansion showing that changing the horizon reweights interactions of different orders and is not generally equivalent to scaling a diversity bonus. Experiments across tasks, budgets, and model scales show that varying K at fixed G shifts learned quality–spread operating points, with effects that persist under fresh sampling. In a 3B setting, DeSPO achieves higher observed Mean RM and semantic spread than DQO with half as many real training rollouts. These results identify valuation horizon as a control axis distinct from rollout budget.

Primary area:
reinforcement learning
Keywords:
large language models, reinforcement learning, set-level policy optimization, diverse generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mPWmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DeSPO separates the number of generated candidates from the horizon used to value repeated draws from those candidates. It derives an exact determinantal objective, uses it to construct local target weights, and fits the policy to the resulting reallocation. The experiments show useful control over response quality and diversity, with benefits that vary by task.

Critical strengths
  • The paper introduces a useful way to change how candidate responses are valued without generating more text. The exact expansion explains why changing K is generally different from increasing the weight of a diversity reward.
  • The experiments show quality and diversity gains at matched rollout budgets and with fewer rollouts. The reward-only and KL-matched controls help isolate the effect, while evaluation on fresh generations shows that it carries over to the trained model.
Critical weaknesses
  • Open-ended answer quality is evaluated with the same reward model used for training. The additional BGE-M3 evaluation checks diversity, not quality.
  • A human comparison with baseline outputs, assessing both answer quality and useful variety, would help verify whether the reported gains agree with human judgments.
  • I am not yet convinced that the exact determinant is worth its computational cost as the candidate pool grows. In Table F3, its gains over Occupancy in Mean RM, Best@16, and LogDet all have confidence intervals that include zero. The second-order approximation is also worth testing in a full training run; the current comparison only examines local target directions.
  • The choice of local prior in Equation 1 needs more explanation. Why use a softmax over mean token log-probabilities, and how does this length normalization affect the target allocation when candidate responses differ substantially in length?
Questions for authors

See above

3 comments

Thank you for your careful assessment. We are glad you see value in varying the valuation horizon while reusing the same generated responses. We especially appreciated your observation that the effect “carries over to the trained model” when it generates fresh responses. That makes the quality of those answers, and the choices they offer a reader, a natural place to start.

W1 and W2. Do the outputs offer better answers and useful choices?

Our original evaluation used the training reward model to score answer quality. To assess the two criteria you raised independently of that model, we asked Gemini (gemini-3.8-flash) to score the generated answers. With four training rollouts per prompt, Gemini gave DeSPO higher mean scores than DQO for both answer quality and useful diversity on all three tasks. This held at both K=4 and K=8. Your proposed human comparison would tell us whether readers share that assessment.

We used the same 128 prompts per task for all 3B checkpoints, with 16 responses per prompt. Gemini scored each answer’s quality and each set’s useful diversity on separate 1–5 scales. Only substantive differences among answers scoring at least 4 on quality with no hard failure earn diversity credit. For summaries, that can mean highlighting different facts supported by the article shown to the judge. We added the scoring criteria and task-specific examples to Appendix C.5.

Scores are means ± sample SD across three training seeds. Within each task and metric, bold marks the highest mean and underlining the second highest.

DatasetMethodGKQuality ↑Useful diversity ↑
CommonGenDQO4—4.3006 ± 0.03701.9067 ± 0.1084
CommonGenDeSPO444.4164 ± 0.08632.0567 ± 0.1546
CommonGenDQO8—4.5376 ± 0.02422.2119‾\underline{2.2119} ± 0.1763
CommonGenDeSPO484.5149‾\underline{4.5149} ± 0.05522.2677 ± 0.1602
DollyDQO4—4.3520 ± 0.01781.5581 ± 0.0357
DollyDeSPO444.4199 ± 0.04291.6098 ± 0.0548
DollyDQO8—4.5135 ± 0.00811.6835‾\underline{1.6835} ± 0.0630
DollyDeSPO484.4742‾\underline{4.4742} ± 0.02551.7243 ± 0.0504
CNN/DailyMailDQO4—4.1407 ± 0.02561.7157 ± 0.0579
CNN/DailyMailDeSPO444.2460 ± 0.06091.7887 ± 0.0671
CNN/DailyMailDQO8—4.3551 ± 0.03361.8324‾\underline{1.8324} ± 0.0459
CNN/DailyMailDeSPO484.3304‾\underline{4.3304} ± 0.04341.8785 ± 0.0637

The table below gives the differences between DeSPO G4K8 and each DQO baseline, in points on the five-point scales.

TaskDQO baselineΔQualityΔUseful diversity
CommonGenG=4+0.2143+0.3610
CommonGenG=8−0.0227+0.0558
DollyG=4+0.1222+0.1662
DollyG=8−0.0393+0.0408
CNN/DailyMailG=4+0.1897+0.1628
CNN/DailyMailG=8−0.0247+0.0461

Table D7 in the revised paper gives all four configurations. Increasing K from 4 to 8 raised both mean scores on every task while keeping G fixed at four. With half as many training rollouts per prompt, DeSPO G4K8 came within 0.04 points of DQO G8 on mean quality and scored higher on mean useful diversity on all three tasks. We highlight this comparison in Table 2 of the main text.

W3. What does the full determinant buy over a second-order approximation?

At G=4, keeping the full determinant added 0.30 seconds per roughly 20-second training refresh (1.52%) over the second-order approximation (P2). For that small cost, Full retains information about triples that P2 discards. Across the three trained configurations, P2 at β=1 had the highest mean RM, P2 at β=2 had the highest spread, and Full fell between them on both measures.

For this comparison we trained a separate Full reference alongside P2 on CommonGen 1B. All three settings used the same FSDP2 runtime and optimizer, G=4, K=8, three training seeds and N=16.

Same-runtime methodMean RM ± SD ↑BGE LogDet ± SD ↑
Full, α=114.1674 ± 0.63695.8197 ± 0.3305
P2, β=114.3093 ± 0.08265.3604 ± 0.2038
P2, β=213.5607 ± 0.39346.3021 ± 0.2640

Appendix F.6 reports the training comparison. The ordering also held on the additional inputs. Table F3 leaves the Full–Occupancy ranking open, with intervals including zero for Mean RM, Best@16, and LogDet.

What P2 misses. P2 keeps the constant, singleton, and pair terms of the exact polynomial before taking its logarithm.

+(K)_2\sum_{i<j}w_iw_j\det L_{\\{i,j\\}}.$$ Appendix A.8 illustrates the difference with these two unit-diagonal Gram matrices. $$L^+=\begin{pmatrix}1&1/2&1/2\\\\1/2&1&1/2\\\\1/2&1/2&1\end{pmatrix}, \qquad L^-=\begin{pmatrix}1&1/2&1/2\\\\1/2&1&-1/2\\\\1/2&-1/2&1\end{pmatrix}.$$ Every singleton and pair minor is identical across the two matrices, so P2 gives them equal value for every allocation and every coefficient choice. Their full determinants are 1/2 and 0. At K≥3 the third-order contribution to the difference is (K)₃w₁w₂w₃/2. Full can therefore distinguish two pools with identical pairwise determinant terms. At rank≤2 or K≤2, the truncated polynomial equals the full polynomial. **Cost at the training scale.** We profiled each method in two separate timing repetitions with eight warm-up and 32 measured refreshes, including critic updates and checkpoint work. Appendix E.7 gives the timings. Percentages use unrounded means. | Method | Seconds per training refresh ↓ | Full's relative extra time | |---|---:|---:| | Full | 20.18 | — | | P2 | 19.88 | 1.52% | | Occupancy | **19.57** | 3.11% | For larger pools, we timed the complete K=8 target solve on one CPU core. Principal minors took 24.824 seconds per prompt at G=16. With the spectral evaluator, **the solve stayed below a second per prompt** at G=32 (0.144 s) and G=64 (0.708 s). ## W4. Why normalize the local prior by response length? **Two answers with the same average token log-probability receive the same prior score, even if one is twice as long.** If the geometric-mean token probability is c, summing log-probabilities gives T log c. Averaging gives log c. Applying softmax to these averages assigns each candidate a prior weight proportional to its geometric-mean token probability. Only response tokens enter the average. We clarified this choice after Eq. 1. $$z_{\mathrm{old},i}=\frac{1}{T_i} \sum_t m_{i,t}\log\pi_{\mathrm{old}}(y_{i,t}\mid x,y_{i,<t}), \qquad p_i=\operatorname{softmax}_i(z_{\mathrm{old}}).$$ We tested how much that choice changes the allocation by holding the same 512 training-derived pools, rewards, geometry, and optimizer settings fixed. A sum-token prior assigned more weight to shorter answers. The resulting K=8 targets differed from the mean-token targets by 1.1325 in L1 distance among converged solves. Expected length is the prior-weighted response length in tokens. L1 is the target distance from the mean-token reference. | Prior | Expected length | Solves (K=4) | Solves (K=8) | L1 (K=4) | L1 (K=8) | |---|---:|---:|---:|---:|---:| | Mean-token | 113.679 | 512/512 | 512/512 | 0 | 0 | | Uniform | 116.295 | 512/512 | 512/512 | 0.4990 | 0.4987 | | Sum-token, floor 1e−12 | 104.878 | 509/512 | 500/512 | 1.1329 | 1.1325 | L1 is averaged over converged solves on existing responses. We then trained a uniform-prior control under the same settings as Full. Appendix F.7 gives both comparisons. **The mean-token prior gave higher RM and BGE spread in all three paired runs.** Normalizing by length removes the direct penalty on longer candidates, while the two trained policies can still generate different response lengths. | Prior in Full G4K8 | Mean RM ± SD ↑ | BGE LogDet ± SD ↑ | Mean generated tokens | |---|---:|---:|---:| | Mean-token | **14.1674** ± 0.6369 | **5.8197** ± 0.3305 | 101.549 | | Uniform | 13.5971 ± 0.1882 | 5.3359 ± 0.0859 | 96.161 |

Dear Reviewer mPWm,
We sincerely thank you for your careful assessment and constructive feedback, and for recognizing the value of varying the valuation horizon while reusing the same generated responses, as well as your observation that the effect "carries over to the trained model" on fresh responses. We have worked to address each of your main concerns in our response: (1) regarding whether the outputs offer better answers and useful choices (W1, W2), we added an evaluation independent of the training reward model, in which Gemini scores answer quality and useful diversity for the 3B policies across three training seeds. With four training rollouts per prompt, DeSPO scores higher than DQO on both measures on all three tasks, and DeSPO G4K8 comes within 0.04 points of DQO G8 on mean quality while scoring higher on useful diversity with half as many rollouts (Table 2, Table D7, scoring criteria in Appendix C.5); (2) regarding what the full determinant buys over a second-order approximation (W3), keeping it adds 0.30 seconds per roughly 20-second training refresh (1.52%), and Appendix A.8 gives two pools with identical pairwise terms that only the full determinant can tell apart (training comparison in Appendix F.6); and (3) regarding why the local prior is normalized by response length (W4), we clarified this choice after Eq. 1 and added a control (Appendix F.7): a sum-token prior assigns more weight to shorter answers, and the mean-token prior gave higher reward-model score and BGE spread than a uniform prior in all three paired runs.
As the author–reviewer discussion period is drawing to a close, we would be grateful to know whether these responses have resolved your main concerns, particularly those regarding the quality and useful diversity of the generated answers, the benefit of the full determinant, and the length-normalized prior. We remain happy to clarify any remaining points before the deadline.
Thank you again for helping us improve this work.
Best regards, The Authors

Reviewer_BG9xOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the coupling between response-generation cost and set-level valuation in diversity-aware large language model (LLM) post-training. The authors propose Decoupled Set Policy Optimization (DeSPO), which separates the actual rollout budget (G) from an independently controlled set-valuation horizon (K). Given a fixed pool of (G) generated responses, DeSPO optimizes their probability allocation by jointly considering task rewards, semantic diversity, and local KL regularization. The set-level utility is evaluated over (K) hypothetical draws with replacement, allowing the valuation horizon to change without requiring additional response generation. The authors further derive an exact finite-support expansion for the expected determinant and use the resulting target allocation to guide policy updates. The theoretical analysis shows that changing (K) reweights interactions of different orders rather than simply rescaling a diversity coefficient. Experiments across multiple open-ended generation tasks and mathematical reasoning benchmarks, including comparisons with PPO, DQO, and SetPO, demonstrate improvements in quality–diversity trade-offs at matched rollout budgets and, in selected settings, with fewer real training responses. The experiments also show that the effects of varying (K) persist under fresh sampling. However, these improvements are task-dependent: increasing (K) can sometimes reduce response quality and does not provide additional observed correctness gains on GSM8K.

Critical strengths
  1. The paper introduces an interesting approach to set-level policy optimization by decoupling the actual rollout budget (G) from the set-valuation horizon (K), allowing the valuation horizon to vary without generating additional responses. The exact finite-support expansion provides a concrete technical contribution, enabling exact evaluation for small candidate pools and explaining how changing (K) affects the relative weighting of interaction orders. This supports acceptance because it offers a meaningful approach to set-level valuation while potentially improving rollout efficiency.

  2. The paper provides experimental evaluations across multiple generation tasks and mathematical reasoning, with comparisons against PPO, DQO, and SetPO. The fixed-interface horizon sweeps, reward-only controls, target-KL matching, alternative utilities, and fresh-sampling evaluations help investigate the effects of different G and K settings and provide useful insights into the roles of valuation, fitting, and allocation magnitude. The results demonstrate quality–diversity improvements in selected settings with fewer real rollouts, while also revealing task-dependent limitations. This supports acceptance by providing empirical evidence of the method's potential benefits and the conditions under which horizon control is useful.

Critical weaknesses
  1. The primary quality evaluator for open-ended generation is the reward model used during training. Therefore, improvements in reward-model scores do not necessarily demonstrate better factuality, instruction satisfaction, or overall usefulness. Stronger independent evaluation is needed to establish whether the reported quality improvements generalize beyond the training reward model.

  2. DeSPO differs from existing methods in both its set-valuation mechanism and actor optimization procedure. Although the within-DeSPO experiments provide evidence of the effect of varying (K), the comparisons with existing methods do not fully establish how much improvement comes specifically from horizon decoupling rather than differences in probability reallocation, policy fitting, or hyperparameter tuning. Moreover, some of the strongest results are achieved with (G=K=8), while noticeable performance differences remain between (G=4, K=8) and (G=8, K=8). This raises concerns about the general effectiveness and specific contribution of the proposed decoupling mechanism.

  3. The improvements in quality and diversity are task-dependent. Increasing (K) can improve semantic diversity while reducing response quality, and it does not provide additional observed correctness gains on GSM8K. The paper also provides limited guidance on selecting an appropriate valuation horizon under quality constraints or accounting for the additional computational cost of hyperparameter tuning.

  4. Although DeSPO reuses existing rollout samples without introducing additional trainable parameters, its exact set-value evaluation has exponential computational complexity, (O(G2^G)). This limits its practicality for larger candidate pools, and the computational trade-off between exact valuation and reduced response generation remains insufficiently explored.

Questions for authors
  1. Appendix C.4 acknowledges that benchmark test results informed broader project development. Could the authors clarify which design decisions were influenced by these results and whether the central performance comparisons can be further validated on an untouched evaluation set?

  2. The experimental results show that increasing (K) can improve semantic diversity but may also reduce response quality, with no additional observed correctness gains on GSM8K. Could the authors suggest a practical validation procedure for selecting an appropriate (K) while maintaining a specified minimum quality level?

  3. How do the computational costs of embedding generation, subset evaluation, and target optimization scale separately with (G) and (K)? Under what conditions would these additional computational costs offset the savings from reduced response generation?

5 comments

Thank you for your thoughtful review. We were pleased that you found the exact expansion “a concrete technical contribution” and that the experiments helped identify when changing the valuation horizon is useful. In particular, you noted that in some settings DeSPO improved both quality and diversity with fewer real rollouts. We begin by examining those gains with an evaluator independent of the training reward model.

W1. Does the quality gain extend beyond the training reward model?

Yes. At G=4, Gemini scored both DeSPO configurations higher than DQO on all three tasks. This held for both mean answer quality and useful diversity. On CommonGen, for example, DQO scored 4.3006 for quality and 1.9067 for useful diversity. DeSPO at K=8 scored 4.5149 and 2.2677 on the same scales.

We evaluated the 3B checkpoints with gemini-3.8-flash, using three training seeds, 128 fixed prompts selected per task, and 16 evaluation responses per prompt. The judge scores each answer's quality from 1 to 5 and assesses useful diversity once for the whole set on a separate 1–5 scale. Quality reflects the task, including concept coverage on CommonGen, instruction satisfaction on Dolly, and faithfulness to the source article visible to the model on CNN/DailyMail. Only substantive differences among answers scoring at least 4 with no hard failure earn diversity credit.

Table R1. Independent evaluation of the 3B checkpoints. Table 2 in the revised paper compares DeSPO G4K8 with both DQO budgets. Table D7 gives all four configurations shown below. Each entry is the mean ± sample SD across three training seeds. Within each task and metric, bold marks the highest mean and underlining the second highest.

DatasetMethodGKQuality ↑Useful diversity ↑
CommonGenDQO4—4.3006 ± 0.03701.9067 ± 0.1084
CommonGenDeSPO444.4164 ± 0.08632.0567 ± 0.1546
CommonGenDQO8—4.5376 ± 0.02422.2119‾\underline{2.2119} ± 0.1763
CommonGenDeSPO484.5149‾\underline{4.5149} ± 0.05522.2677 ± 0.1602
DollyDQO4—4.3520 ± 0.01781.5581 ± 0.0357
DollyDeSPO444.4199 ± 0.04291.6098 ± 0.0548
DollyDQO8—4.5135 ± 0.00811.6835‾\underline{1.6835} ± 0.0630
DollyDeSPO484.4742‾\underline{4.4742} ± 0.02551.7243 ± 0.0504
CNN/DailyMailDQO4—4.1407 ± 0.02561.7157 ± 0.0579
CNN/DailyMailDeSPO444.2460 ± 0.06091.7887 ± 0.0671
CNN/DailyMailDQO8—4.3551 ± 0.03361.8324‾\underline{1.8324} ± 0.0459
CNN/DailyMailDeSPO484.3304‾\underline{4.3304} ± 0.04341.8785 ± 0.0637

With half as many training rollouts per prompt as DQO G8, DeSPO G4K8 scored higher on useful diversity on every task (+0.0408 to +0.0558), with mean quality 0.0227 to 0.0393 points lower on the five-point scale. On the paper’s original 3B evaluation, this same four-versus-eight comparison favored DeSPO on both mean RM and MiniLM LogDet on CommonGen and Dolly (Table D4).

Q1. Which decisions used benchmark results, and what do the additional inputs tell us?

Benchmark results informed method and hyperparameter development. Appendix C.4 now distinguishes the SetPO pilot from the earlier control calibration. The reward-only and target-KL calibration used 512 training-derived prompts. For the revised SetPO baseline in Part 2, W2, the weight pilot used 256 historical test-derived prompts as development data. The early records give an incomplete account of K and checkpoint selection.

We evaluated frozen 3B checkpoints on two additional sets of 512 prompts each, using 16 outputs per prompt. The CommonGen models received new concept combinations. The Dolly models received instructions from Alpaca, a different distribution. We found no normalized-prompt overlap with the recoverable canonical data. The results below also appear in Table D8. They are means across three training seeds.

Additional inputsTraining methodMean RM ↑MiniLM LogDet ↑BGE LogDet ↑
CommonGen conceptsDQO G814.53325.85895.2097
CommonGen conceptsDeSPO G4K815.09496.81685.5018
Alpaca instructionsDQO G814.46704.62834.0779
Alpaca instructionsDeSPO G4K814.47964.77774.1835

Both encoders measured greater spread for DeSPO on both additional sets. Mean RM also favored DeSPO on the new CommonGen combinations and was close on the Alpaca instructions (14.4796 versus 14.4670). We later reused the inputs for ablations, which limits their use as an untouched test of the overall method. For the separate K-selection check in Part 2, we fixed the choice before scoring the terminal prompts.

W2. What does decoupling G and K contribute?

With four candidates, increasing K improved both Gemini scores on all three tasks. Moving from K=4 to K=8 raised mean quality by 0.0985 on CommonGen, 0.0543 on Dolly, and 0.0844 on CNN/DailyMail. Useful diversity rose by 0.2110, 0.1145, and 0.0898, respectively. G determines which answers enter the pool. K sets the number of hypothetical draws with replacement used to value allocations within each pool. If a pool lacks a useful idea, increasing G gives the policy a chance to generate it. On CommonGen 1B at fixed K=8, increasing G from 4 to 8 raised mean RM from 13.886 to 16.223 and MiniLM LogDet from 6.890 to 7.373 (Table E3).

Under the same actor loss, Full had higher BGE spread on CommonGen 1B at LOO temperatures 150, 300, 600, and the target-KL-matched 627.334. At τ=150 and 300, LOO had higher mean RM than Full, peaking at 300. A separate control also matching post-AdamW step length gave Full +0.9089 RM (95% interval [0.3568, 1.9018]) and +1.4245 BGE LogDet [1.0535, 1.9481], with both gains positive in all three pairs. Appendix F.5 gives these controls. Table R1 in Part 1 compares the trained methods end to end.

To decide whether a longer horizon helps, keep G and the diversity coefficient α fixed when comparing K. Changing α scales the structural term as a whole. Changing K alters the relative weights on pairs, triples, and higher-order interactions. Raising α can also cost quality. At K=8, doubling α from 1 to 2 gave the highest spread and lowest RM in the grid below (Table E11 in the revision, CommonGen 1B, G=4, three runs per cell).

KαMean RM ↑BGE LogDet ↑
40.514.16904.9569
4113.95535.2158
4214.17665.6526
80.514.43255.0723
8113.88655.7051
8211.58136.6700

As a separate baseline check, we compared DeSPO with the revised SetPO combined-PPO adapter on Dolly 1B at G=4 and N=16 (Table D9). Both DeSPO configurations achieved higher RM and MiniLM spread than both SetPO settings in every paired run.

MethodMean RM ± SD ↑MiniLM LogDet ± SD ↑
Revised SetPO, default weight 0.111.3223 ± 0.11604.2585 ± 0.0274
Revised SetPO, selected weight 0.0311.3375 ± 0.10974.2656 ± 0.0072
DeSPO G4K411.7515 ± 0.23924.3942 ± 0.0175
DeSPO G4K811.4765 ± 0.10024.4311 ± 0.0910

W3 and Q2. How should K be chosen when quality has a minimum requirement?

Set G from the rollout budget. At that G, choose the highest-spread K that meets the application’s quality floor on development data. Fix the metric, floor and candidate horizons in advance. Break ties toward smaller K. If neither qualifies, keep the floor and report no eligible setting. For tasks scored on correctness, use correctness as the quality metric. On GSM8K, raising K to 8 gave no additional observed correctness gain.

The choice of metric also matters within the same task. On CNN/DailyMail 3B, moving from K=4 to K=8 at G=4 lowered mean RM by 0.305 and MiniLM LogDet by 0.012 (Table E1). Gemini judged the K=8 outputs higher on quality (+0.0844) and useful diversity (+0.0898). MiniLM measures embedding dispersion across responses. Gemini credits distinct content only among answers that meet the task’s quality criterion.

For CommonGen, we reused the RM thresholds from our earlier sensitivity analysis and fixed them before scoring these splits. We compared K∈{4,8} over three training runs, with 16 outputs per prompt on 256 development and 256 separate terminal prompts.

KDevelopment RM ↑Development BGE ↑Terminal RM ↑Terminal BGE ↑
413.91465.196914.00615.2248
813.87735.693013.88955.7234
Prespecified RM thresholdFrozen selectionSelected terminal RMSelected terminal BGE
13.5K813.88955.7234
14.0No feasible K——
14.5No feasible K——

The rule selected K=8 at 13.5 and neither horizon at 14.0. At 13.5, both passed on development data and K=8 won on spread. It reached 13.8895 RM and 5.7234 BGE on the terminal set. K=4 subsequently reached 14.0061 on the terminal set, after the selection was fixed. The threshold applies to the three-run mean. We added this selection rule to Section 6. Table E10 gives the development and terminal scores.

This two-setting comparison reused six existing checkpoints. Starting from scratch at the same replication budget would require two horizon settings × three training seeds, plus development and terminal evaluation. The per-refresh savings in Part 3 apply once a configuration is chosen.

W4 and Q3. When does exact valuation pay for itself?

With exact valuation included, DeSPO G4K8 saved about 14 seconds per training refresh at 1B and 30 seconds at 3B compared with DQO G8. Generating and scoring four fewer candidates more than paid for exact valuation at these measured scales. Section 6 summarizes the savings, and Table E12 gives the measured times per refresh. Percentages use the unrounded timings.

ModelDQO G8 (s/refresh) ↓DeSPO G4K8 (s/refresh) ↓Time saved
1B33.01418.97142.54%
3B67.99038.28743.69%

For a larger pool, reducing G saves training time when the time saved on generation and scoring exceeds the extra time training waits for target construction. Only the part of a CPU solve that delays the next training stage counts toward that comparison. Changing K at fixed G on a given pool reuses the responses, scores, and embeddings and requires a new target solve.

The principal-minor implementation enumerates subsets through order D=min(K,G). The same objective has an exact spectral form. If μ are the eigenvalues of the weighted Gram matrix, eⱼ is the jth elementary symmetric polynomial and (K)ⱼ is a falling factorial, then

\qquad \mu=\operatorname{eig}\\!\left( \operatorname{diag}(\sqrt w)L\operatorname{diag}(\sqrt w)\right).$$ The spectral value and gradient cost O(G³+GD²) arithmetic operations per evaluation. The full solve also depends on the number of iterations and numerical precision. For the component timings in Appendix E.7 (Tables E13–E15), we profiled eight fixed CommonGen prompts with a fixed 1B actor. Generation was run separately at each G. Scoring and encoding use nested prefixes of a fresh G=64 pool. ### Component timings on generated CommonGen responses GPU means are in seconds per batch of eight prompts on one RTX PRO 6000 Blackwell, with one warm-up and three measured repetitions. | G | Generation | Policy scoring | Reward scoring | Encoding | |---|---:|---:|---:|---:| | 4 | 0.840 | 0.616 | 0.082 | 0.022 | | 8 | 0.910 | 1.230 | 0.163 | 0.042 | | 16 | 0.979 | 2.462 | 0.325 | 0.084 | | 32 | 1.129 | 4.926 | 0.650 | 0.165 | | 64 | 1.439 | 9.858 | 1.302 | 0.331 | CPU means are in seconds per prompt on one pinned AMD EPYC 9V74 core with one BLAS thread. The K=8 breakdown separates preparation, a single value-and-gradient evaluation, and the complete target solve. Each complete solve includes its own evaluator construction. | G | Preparation | Value and gradient | Complete solve | |---|---:|---:|---:| | 4 | 0.000178 | 0.000060 | 0.002895 | | 8 | 0.001974 | 0.001002 | 0.079419 | | 16 | 0.324835 | 0.182713 | 24.823664 | | 32 | 0.000108 | 0.000588 | 0.143679 | | 64 | 0.000207 | 0.002014 | 0.707735 | Complete solve times across all 15 combinations are given in seconds per prompt. | G | K=4 | K=8 | K=16 | |---|---:|---:|---:| | 4 | 0.002888 | 0.002895 | 0.002854 | | 8 | 0.048698 | 0.079419 | 0.079889 | | 16 | 1.491533 | 24.823664 | 52.972872 | | 32 | 0.150559 | 0.143679 | 0.155406 | | 64 | 0.697549 | 0.707735 | 0.814253 | **At K=8, the complete target solve took about 3 milliseconds per prompt at G=4 and less than a second at both G=32 and G=64** on the profiled CPU core. At G=16, the principal-minor solve took 24.82 seconds per prompt. The implementation switches from principal minors to the spectral evaluator above G=16. All 360 measured cases passed the feasibility and objective checks following 120 warm-ups. Separate stress tests found cancellation at K=32 and K=64 and failures with concentrated priors. ### Spectral value and gradient For D=min(K,G), initialize e₀=1 and the remaining coefficients to zero. The recurrence for elementary symmetric polynomials is $$e_j^{(t)}=e_j^{(t-1)}+\mu_t e_{j-1}^{(t-1)}.$$ With A=diag(√w)Ldiag(√w)=Qdiag(μ)Qᵀ, the eigenvalue and weight derivatives are $$h_\ell=\frac{\partial H_K}{\partial\mu_\ell} =\sum_{j=1}^{D}(K)_j e_{j-1}(\mu_{\setminus\ell}), \qquad \frac{\partial H_K}{\partial w_i} =\frac{1}{w_i}\sum_{\ell=1}^{G}Q_{i\ell}^{2}\mu_\ell h_\ell.$$ We added the spectral derivation to Appendix A.9 and the implementation details to Appendix B.7. Prefix and suffix polynomial products compute the exclusion coefficients. The matrix gradient Qdiag(h)Qᵀ also applies at repeated eigenvalues. The weight derivative above assumes positive w. The optimizer checks feasibility and numerical acceptance separately.
Reviewer_3eArOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Decoupled Set Policy Optimization (DeSPO), which separates the real rollout budget GG from the set-valuation horizon KK in LLM post-training. DeSPO optimizes probability allocations over GG generated responses using task rewards, expected set value under KK draws with replacement, and KL regularization. The resulting target guides policy updates without additional text generation. Experiments show improvements in response quality and semantic spread over existing methods, although the effects depend on the task and valuation horizon.

Critical strengths
  • Clear separation of rollout budget and set valuation, supported by an exact formulation
    • The paper separates the real rollout budget GG from the set-valuation horizon KK, allowing the same candidate pool to be valued under different horizons without additional response generation. Proposition 1 provides an exact expansion of HKH_K with at most 2G2^G subset terms instead of enumerating GKG^K ordered outcomes, making the computation practical for small candidate pools. The accompanying analysis also explains how KK changes the relative weights of different interaction orders, rather than simply scaling a fixed diversity bonus. These are clear conceptual and mathematical contributions.
  • Promising empirical results with useful ablation studies
    • DeSPO shows improvements in response quality and semantic spread in several settings. In particular, the 3B CommonGen and Dolly results improve both measures over DQO while using half as many real training rollouts. The fixed-GG horizon sweeps, reward-only controls, and target-KL-matched comparisons also provide useful evidence that set valuation and horizon choice affect the learned policy under the proposed fitting procedure.
Critical weaknesses
  • Limited explanation of the advantage over realized-group credit
    • The analysis establishes that the proposed valuation differs from DQO and that changing KK is not generally equivalent to rescaling a diversity coefficient. However, these differences alone do not explain why DeSPO should achieve a better quality–diversity trade-off. It remains unclear which specific limitation of DQO’s realized-group credit is addressed by repeated-draw valuation.
  • The effects of set valuation and actor updates are not sufficiently separated
    • The main DeSPO–DQO comparisons change both the set-level learning signal and the actor-update rule. Although the existing ablations are useful, they do not compare the proposed valuation with DQO-style credit under the same full-target actor update. Therefore, the source of the improvement over DQO remains unclear.
Questions for authors
  • When should repeated-draw valuation be preferable to realized-group credit?
    • Could the authors provide a simple example or comparative analysis showing when and why the proposed valuation should improve the quality–diversity trade-off over DQO? For example, extending the redundancy example in Section 4.1 to compare both methods on the same candidate pool would help clarify the benefit beyond the mathematical differences between their objectives.
  • Comparison with DQO-style credit under the same actor update.
    • Could the authors compare DeSPO with a DQO-style target using the same full-target actor update? Specifically, using DQO’s realized-group leave-one-out credit ciDQOc_i^{\mathrm{DQO}}, including both the task reward and the diversity term, we can construct a target allocation as
\frac{p_i \exp(c_i^{\mathrm{DQO}}/\tau)} {\sum_j p_j \exp(c_j^{\mathrm{DQO}}/\tau)}.$$ Then, the actor can be updated using $\ell_x(\theta; \hat{w}_{\mathrm{DQO}} - p)$, following Equation 6. Using the same rollout budget and training settings would help isolate the contribution of the proposed set valuation from differences in actor updates. Since the two target constructions can have different credit scales, could the authors tune $\tau$ for this baseline, explicitly including values below $600$, rather than directly reusing DeSPO’s default?
3 comments

Thank you for your careful reading. We are glad you see the distinction between rollout budget and valuation horizon, together with the exact expansion, as “clear conceptual and mathematical contributions.” You also singled out the 3B results on CommonGen and Dolly, where DeSPO improved both mean reward-model score and semantic spread over DQO with half as many training rollouts. We begin with your question about what the two valuation rules do on the same pool.

W1 and Q1. When does repeated-draw valuation help?

Following your suggestion, we extended the three-candidate example in Section 4.1. Appendix A.7 works out how the two rules move probability on that same pool. The distinction matters when a response is unique in the observed group but already receives most of the local probability. Giving it still more weight can make future draws from the pool less diverse, even when every answer is equally good.

What deletion credit sees. Suppose A1 and A2 express the same idea and have identical unit embeddings. B expresses a second idea and has a unit embedding orthogonal to theirs. All three have equal task reward. For this pool, the regularized determinant det(I + L) is 6, where L is the Gram matrix. Removing one A leaves an A and a B, whose determinant is 4. Removing B leaves two copies of A, whose determinant is 3. Thus the DQO-style leave-one-out (LOO) credits are log(6/4) = log(1.5) for each A and log(6/3) = log(2) for B. B earns the larger credit because removing it eliminates the only representative of its idea. These credits stay the same for any probability allocation on the three candidates.

What changes when we value the allocation. Let the local prior be (0.1, 0.1, 0.8). If we draw twice with replacement, the chance of seeing both ideas is only 2 × 0.8 × 0.2 = 0.32. Moving some probability from B to the A candidates raises that chance while leaving expected task reward unchanged. The maximum for two draws is 0.50, reached at a 50/50 split between the ideas. DeSPO’s target (Full, Eq. 5) stays much closer to the prior because moving probability carries a KL penalty.

For any horizon K, write q for the total probability on B. In Section 4.1's exact expansion, A1 and A2 contribute zero pair volume together. Each A forms a complementary pair with B, so the sum of the two weighted pair terms is q(1−q). This rank-two pool has no higher-order terms. The expected determinant is therefore

HK(w)=1+K+K(K−1)q(1−q).H_K(w)=1+K+K(K-1)q(1-q).

For K≥2, the slope of its logarithm has the sign of 1−2q. Starting at a prior with 0 < q < 0.5, both targets favor giving B more weight. At the illustrated prior, q=0.8, the deletion credit still favors B, while the repeated-draw term favors A. With equal rewards, unit diversity coefficients and τ=600, the two targets move in opposite directions.

TargetProbability assigned to B
Prior0.800000
Reward-plus-LOO target0.800077
Full target, K=40.799723
Full target, K=80.799502

The small shifts in the table reflect the strong KL penalty. From the same prior, LOO reinforces the prevalent idea, while Full makes it more likely that two draws cover both ideas. Here G=3 throughout. Increasing K from 4 to 8 changes the size of the reallocation using the same three embeddings and rewards. If the two A candidates have reward rA and B has reward rB, transferring probability ε from B to A changes expected reward by ε(rA−rB). When B is substantially better, Eq. 5 weighs that cost against the gain in set value. The quality–diversity benefit is easiest to see when multiple answers satisfy the task and one of their ideas already dominates the allocation.

W2 and Q2. What changes when both targets use the same actor update?

We ran the control you proposed, including the task reward and temperatures below 600. With four rollouts per prompt and the same actor loss (Eq. 6), exact set valuation produces more widely spread response embeddings on CommonGen and Dolly. Section 5.4 now reports the CommonGen comparison. Appendix F.5 gives the sweep, matched steps, and cross-task results.

For the DQO-style target, we combine task reward r and realized-group leave-one-out (LOO) credit c with the local prior p.

p_i\exp\\!\left((r_i+\beta c_i^{\mathrm{LOO}})/\tau\right), \qquad \beta=1.$$ DeSPO’s exact valuation (Full) uses Eq. 5 with K=8 and the same prior. At each rollout refresh, both methods freeze the target displacement and fit it with the same listwise actor loss (λ=1). $$\mathcal L=-\frac{1}{B_p}\sum_{u=1}^{B_p}\sum_{i=1}^{G} (\widehat w_{u,i}-p_{u,i})z_{\theta,u,i},$$ Here z is the mean response-token log-probability and Bp is the number of prompts in the batch. **On CommonGen 1B, Full gives higher BGE spread at every tested LOO temperature.** To account for the different credit scales, we tested LOO at τ=150, 300 and 600, plus a temperature that matches Full’s target KL. Each setting used three training runs, 186 rollout refreshes per run and 16 evaluation responses per prompt. We report mean reward-model score (RM) and embedding spread (BGE LogDet). | Signal | τ | Mean RM ↑ | BGE LogDet ↑ | |---|---:|---:|---:| | Full, K=8 | 600 | 13.8865 | **5.7051** | | LOO | 150 | 14.1591 | 4.8003 | | LOO | 300 | **14.3617** | 5.0470 | | LOO | 600 | 13.8639 | 4.8297 | | LOO, target-KL matched | 627.334 | 14.1672 | 4.8766 | Lowering τ helps LOO. Its best setting on both metrics is τ=300, where LOO scores 0.4752 higher on RM and Full scores 0.6581 higher on BGE. At the shared τ=600, RM differs by only 0.0226 and Full gains 0.8754 on BGE. Matching target KL also preserves the spread advantage. **At the same optimizer step length, Full gains on both metrics in all three paired CommonGen runs.** Matching target KL controls how far the allocation moves from the prior. AdamW can still turn those displacements into parameter updates of different lengths. In a separate control, we restarted both methods from the same base actor at τ=600 with the same learning rate, clipping and AdamW settings. After preconditioning and weight decay, we scaled each update to the same length (0.0122747512, the median of 16 proposal norms from both methods in training-only calibration). All 4,464 updates met the 0.1% tolerance (744 steps per run, maximum relative error 0.0555%). The first two rows give means ± SD over three training runs. The last gives the paired difference and its 95% interval. | Common-step control | Mean RM ↑ | BGE LogDet ↑ | |---|---:|---:| | Full G4K8 | **16.2162** ± 0.7108 | **5.1362** ± 0.2204 | | LOO G4, τ=600 | 15.3073 ± 0.3151 | 3.7117 ± 0.2939 | | Full−LOO | +0.9089 [0.3568, 1.9018] | +1.4245 [1.0535, 1.9481] | Full−LOO differences by paired run. | Paired run | ΔMean RM | ΔBGE LogDet | |---|---:|---:| | 1 | +0.3589 | +1.0497 | | 2 | +0.4549 | +1.9512 | | 3 | +1.9130 | +1.2726 | Averaging over the three runs for each prompt, Full has higher BGE spread on **all 1,024 main evaluation prompts** and higher mean RM on **994 of them**. MiniLM LogDet increases in all three pairs as well (mean +2.6554, 95% interval [2.3108, 3.1214]). Both gains also held on the additional evaluation inputs. Full’s responses averaged 23.56 more tokens, which may contribute to both gains. **The spread gain extends to Dolly at both tested temperatures.** Full gains 0.14–0.16 in BGE LogDet on Dolly, while LOO scores about 0.30 higher on mean RM. On CNN/DailyMail, the two methods give similar BGE LogDet scores at both temperatures (within 0.022). Entries are Full−LOO differences under the ordinary optimizer, with three 1B training runs per method and setting. All 95% intervals in this reply use a paired seed/prompt bootstrap and are exploratory, without adjustment for multiple comparisons. | Task | LOO τ | ΔMean RM | ΔBGE LogDet | |---|---:|---:|---:| | Dolly | 600 | −0.2905 [−0.8975, 0.1099] | +0.1628 [0.0769, 0.2752] | | Dolly | 300 | −0.3082 [−1.0281, 0.3490] | +0.1427 [0.0870, 0.2310] | | CNN/DailyMail | 600 | −0.1109 [−0.5955, 0.2178] | −0.0214 [−0.0712, 0.0428] | | CNN/DailyMail | 300 | +0.0155 [−0.2019, 0.3553] | −0.0025 [−0.0249, 0.0148] | **On CommonGen, the valuation itself contributes to the spread gain.** Full retains its advantage when LOO uses its best tested temperature and, in a separate comparison at τ=600, when both actors take equally long parameter steps.

Dear Reviewer 3eAr,
We sincerely thank you for your careful review and constructive feedback, and for recognizing the separation of rollout budget and valuation horizon, together with the exact expansion, as "clear conceptual and mathematical contributions." We have worked to address each of your main concerns in our response: (1) regarding when repeated-draw valuation helps (W1, Q1), we extended the three-candidate example as you suggested (Section 4.1, Appendix A.7): when the only answer expressing one idea already holds 80% of the local probability, leave-one-out credit still adds weight to it, while DeSPO's exact valuation shifts weight toward the other idea, making two draws more likely to cover both at the same expected reward; and (2) regarding what changes when both targets use the same actor update (W2, Q2), we ran the control you proposed, including the task reward and temperatures below 600. On CommonGen 1B, exact valuation gives higher BGE spread at every tested leave-one-out temperature (τ = 150, 300, 600, and a target-KL-matched setting), while leave-one-out reaches its highest reward-model score at τ = 300. When the optimizer step length is also matched, exact valuation gains +0.9089 in mean reward-model score and +1.4245 in BGE LogDet, with both gains in all three paired runs. The spread gain also holds on Dolly (+0.14–0.16 BGE LogDet), and the two methods are close on CNN/DailyMail (Section 5.4, Appendix F.5).
As the author–reviewer discussion period is drawing to a close, we would be grateful to know whether these responses have resolved your main concerns, particularly those regarding when repeated-draw valuation differs from leave-one-out credit and whether the gain remains under the same actor update. We remain happy to clarify any remaining points before the deadline.
Thank you again for helping us improve this work.
Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response on new results and revisions

We have uploaded the revised PDF, with changes marked in blue throughout the main text and appendices.

We thank the AC and all three reviewers for their careful reading. Reviewer 3eAr describes the separation of rollout budget and valuation horizon, together with the exact expansion, as “clear conceptual and mathematical contributions.” BG9x highlights the gains in quality and diversity with fewer training rollouts in selected settings. mPWm notes that the effect carries over to fresh answers from the trained policy.

Can a policy learn more from four generated answers per training prompt (G=4) by valuing eight draws from that same pool (K=8)? The reviewers’ suggestions led us to evaluate fresh answers independently of the training reward model and to compare valuation rules under the same actor loss.

QuestionNew result
Does a longer horizon improve the answers?At G=4, increasing K from 4 to 8 raised both mean quality and mean useful diversity on CommonGen, Dolly, and CNN/DailyMail. Gemini evaluated the 3B policies across three training seeds, crediting substantive differences among answers that satisfy the task. Against DQO G4, DeSPO G4K8 gained 0.12–0.21 points in quality and 0.16–0.36 in useful diversity on five-point scales (Table D7).
How close do four rollouts come to eight?Against DQO G8, DeSPO G4K8 had higher mean useful diversity on all three tasks with half the training rollouts. Gemini rated its mean quality 0.023–0.039 points lower on the five-point scale (Table 2).
Does the valuation rule matter under the same actor loss?On CommonGen 1B, exact valuation gave higher BGE spread at every tested leave-one-out temperature. Leave-one-out credit reached its highest mean reward-model score (RM) at τ=300. In a separate control matching optimizer step length, DeSPO gained 0.9089 RM and 1.4245 BGE LogDet, with both gains in every paired run (Section 5.4, Appendix F.5).
What does exact valuation cost in training?Including target construction, DeSPO G4K8 used 42.54% less time per refresh at 1B and 43.69% less at 3B than DQO G8 on CommonGen. At fixed G=4 on CommonGen 1B, keeping the full determinant added 0.30 seconds per roughly 20-second refresh over the second-order approximation (Section 6, Appendix E.7).
How should we choose K when quality must meet a minimum?Choose the highest-spread K that clears the quality floor on development prompts. On CommonGen 1B, both horizons cleared 13.5 RM. The rule selected K=8 for its higher spread, and it reached 13.89 RM on separate evaluation prompts. At a floor of 14.0, neither tested horizon qualified on development data (Section 6, Table E10).

The example suggested by 3eAr shows why the choice of horizon matters. When the sole answer expressing one idea already has 80% of the probability, deletion credit still favors it. Valuing repeated draws shifts weight toward the other idea, making two draws more likely to cover both at the same expected reward (Section 4.1, Appendix A.7). With four rollouts available, a practitioner can test an eight-draw horizon at the same generation budget and keep it when the resulting policy meets the task’s quality requirement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.