- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
DeSPO separates the number of generated candidates from the horizon used to value repeated draws from those candidates. It derives an exact determinantal objective, uses it to construct local target weights, and fits the policy to the resulting reallocation. The experiments show useful control over response quality and diversity, with benefits that vary by task.
Critical strengths
- The paper introduces a useful way to change how candidate responses are valued without generating more text. The exact expansion explains why changing K is generally different from increasing the weight of a diversity reward.
- The experiments show quality and diversity gains at matched rollout budgets and with fewer rollouts. The reward-only and KL-matched controls help isolate the effect, while evaluation on fresh generations shows that it carries over to the trained model.
Critical weaknesses
- Open-ended answer quality is evaluated with the same reward model used for training. The additional BGE-M3 evaluation checks diversity, not quality.
- A human comparison with baseline outputs, assessing both answer quality and useful variety, would help verify whether the reported gains agree with human judgments.
- I am not yet convinced that the exact determinant is worth its computational cost as the candidate pool grows. In Table F3, its gains over Occupancy in Mean RM, Best@16, and LogDet all have confidence intervals that include zero. The second-order approximation is also worth testing in a full training run; the current comparison only examines local target directions.
- The choice of local prior in Equation 1 needs more explanation. Why use a softmax over mean token log-probabilities, and how does this length normalization affect the target allocation when candidate responses differ substantially in length?
Questions for authors
See above
3 comments
W3. What does the full determinant buy over a second-order approximation?
At G=4, keeping the full determinant added 0.30 seconds per roughly 20-second training refresh (1.52%) over the second-order approximation (P2). For that small cost, Full retains information about triples that P2 discards. Across the three trained configurations, P2 at β=1 had the highest mean RM, P2 at β=2 had the highest spread, and Full fell between them on both measures.
For this comparison we trained a separate Full reference alongside P2 on CommonGen 1B. All three settings used the same FSDP2 runtime and optimizer, G=4, K=8, three training seeds and N=16.
| Same-runtime method | Mean RM ± SD ↑ | BGE LogDet ± SD ↑ |
|---|---|---|
| Full, α=1 | 14.1674 ± 0.6369 | 5.8197 ± 0.3305 |
| P2, β=1 | 14.3093 ± 0.0826 | 5.3604 ± 0.2038 |
| P2, β=2 | 13.5607 ± 0.3934 | 6.3021 ± 0.2640 |
Appendix F.6 reports the training comparison. The ordering also held on the additional inputs. Table F3 leaves the Full–Occupancy ranking open, with intervals including zero for Mean RM, Best@16, and LogDet.
What P2 misses. P2 keeps the constant, singleton, and pair terms of the exact polynomial before taking its logarithm.
+(K)_2\sum_{i<j}w_iw_j\det L_{\\{i,j\\}}.$$ Appendix A.8 illustrates the difference with these two unit-diagonal Gram matrices. $$L^+=\begin{pmatrix}1&1/2&1/2\\\\1/2&1&1/2\\\\1/2&1/2&1\end{pmatrix}, \qquad L^-=\begin{pmatrix}1&1/2&1/2\\\\1/2&1&-1/2\\\\1/2&-1/2&1\end{pmatrix}.$$ Every singleton and pair minor is identical across the two matrices, so P2 gives them equal value for every allocation and every coefficient choice. Their full determinants are 1/2 and 0. At K≥3 the third-order contribution to the difference is (K)₃w₁w₂w₃/2. Full can therefore distinguish two pools with identical pairwise determinant terms. At rank≤2 or K≤2, the truncated polynomial equals the full polynomial. **Cost at the training scale.** We profiled each method in two separate timing repetitions with eight warm-up and 32 measured refreshes, including critic updates and checkpoint work. Appendix E.7 gives the timings. Percentages use unrounded means. | Method | Seconds per training refresh ↓ | Full's relative extra time | |---|---:|---:| | Full | 20.18 | — | | P2 | 19.88 | 1.52% | | Occupancy | **19.57** | 3.11% | For larger pools, we timed the complete K=8 target solve on one CPU core. Principal minors took 24.824 seconds per prompt at G=16. With the spectral evaluator, **the solve stayed below a second per prompt** at G=32 (0.144 s) and G=64 (0.708 s). ## W4. Why normalize the local prior by response length? **Two answers with the same average token log-probability receive the same prior score, even if one is twice as long.** If the geometric-mean token probability is c, summing log-probabilities gives T log c. Averaging gives log c. Applying softmax to these averages assigns each candidate a prior weight proportional to its geometric-mean token probability. Only response tokens enter the average. We clarified this choice after Eq. 1. $$z_{\mathrm{old},i}=\frac{1}{T_i} \sum_t m_{i,t}\log\pi_{\mathrm{old}}(y_{i,t}\mid x,y_{i,<t}), \qquad p_i=\operatorname{softmax}_i(z_{\mathrm{old}}).$$ We tested how much that choice changes the allocation by holding the same 512 training-derived pools, rewards, geometry, and optimizer settings fixed. A sum-token prior assigned more weight to shorter answers. The resulting K=8 targets differed from the mean-token targets by 1.1325 in L1 distance among converged solves. Expected length is the prior-weighted response length in tokens. L1 is the target distance from the mean-token reference. | Prior | Expected length | Solves (K=4) | Solves (K=8) | L1 (K=4) | L1 (K=8) | |---|---:|---:|---:|---:|---:| | Mean-token | 113.679 | 512/512 | 512/512 | 0 | 0 | | Uniform | 116.295 | 512/512 | 512/512 | 0.4990 | 0.4987 | | Sum-token, floor 1e−12 | 104.878 | 509/512 | 500/512 | 1.1329 | 1.1325 | L1 is averaged over converged solves on existing responses. We then trained a uniform-prior control under the same settings as Full. Appendix F.7 gives both comparisons. **The mean-token prior gave higher RM and BGE spread in all three paired runs.** Normalizing by length removes the direct penalty on longer candidates, while the two trained policies can still generate different response lengths. | Prior in Full G4K8 | Mean RM ± SD ↑ | BGE LogDet ± SD ↑ | Mean generated tokens | |---|---:|---:|---:| | Mean-token | **14.1674** ± 0.6369 | **5.8197** ± 0.3305 | 101.549 | | Uniform | 13.5971 ± 0.1882 | 5.3359 ± 0.0859 | 96.161 |Dear Reviewer mPWm,
We sincerely thank you for your careful assessment and constructive feedback, and for recognizing the value of varying the valuation horizon while reusing the same generated responses, as well as your observation that the effect "carries over to the trained model" on fresh responses. We have worked to address each of your main concerns in our response: (1) regarding whether the outputs offer better answers and useful choices (W1, W2), we added an evaluation independent of the training reward model, in which Gemini scores answer quality and useful diversity for the 3B policies across three training seeds. With four training rollouts per prompt, DeSPO scores higher than DQO on both measures on all three tasks, and DeSPO G4K8 comes within 0.04 points of DQO G8 on mean quality while scoring higher on useful diversity with half as many rollouts (Table 2, Table D7, scoring criteria in Appendix C.5); (2) regarding what the full determinant buys over a second-order approximation (W3), keeping it adds 0.30 seconds per roughly 20-second training refresh (1.52%), and Appendix A.8 gives two pools with identical pairwise terms that only the full determinant can tell apart (training comparison in Appendix F.6); and (3) regarding why the local prior is normalized by response length (W4), we clarified this choice after Eq. 1 and added a control (Appendix F.7): a sum-token prior assigns more weight to shorter answers, and the mean-token prior gave higher reward-model score and BGE spread than a uniform prior in all three paired runs.
As the author–reviewer discussion period is drawing to a close, we would be grateful to know whether these responses have resolved your main concerns, particularly those regarding the quality and useful diversity of the generated answers, the benefit of the full determinant, and the length-normalized prior. We remain happy to clarify any remaining points before the deadline.
Thank you again for helping us improve this work.
Best regards, The Authors
Thank you for your careful assessment. We are glad you see value in varying the valuation horizon while reusing the same generated responses. We especially appreciated your observation that the effect “carries over to the trained model” when it generates fresh responses. That makes the quality of those answers, and the choices they offer a reader, a natural place to start.
W1 and W2. Do the outputs offer better answers and useful choices?
Our original evaluation used the training reward model to score answer quality. To assess the two criteria you raised independently of that model, we asked Gemini (gemini-3.8-flash) to score the generated answers. With four training rollouts per prompt, Gemini gave DeSPO higher mean scores than DQO for both answer quality and useful diversity on all three tasks. This held at both K=4 and K=8. Your proposed human comparison would tell us whether readers share that assessment.
We used the same 128 prompts per task for all 3B checkpoints, with 16 responses per prompt. Gemini scored each answer’s quality and each set’s useful diversity on separate 1–5 scales. Only substantive differences among answers scoring at least 4 on quality with no hard failure earn diversity credit. For summaries, that can mean highlighting different facts supported by the article shown to the judge. We added the scoring criteria and task-specific examples to Appendix C.5.
Scores are means ± sample SD across three training seeds. Within each task and metric, bold marks the highest mean and underlining the second highest.
The table below gives the differences between DeSPO G4K8 and each DQO baseline, in points on the five-point scales.
Table D7 in the revised paper gives all four configurations. Increasing K from 4 to 8 raised both mean scores on every task while keeping G fixed at four. With half as many training rollouts per prompt, DeSPO G4K8 came within 0.04 points of DQO G8 on mean quality and scored higher on mean useful diversity on all three tasks. We highlight this comparison in Table 2 of the main text.