- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces VerDICT, a decentralized, inference-only framework for collective preference optimization. The method combines EGGROLL, a low-rank evolution strategies algorithm, with a pairwise preference objective designed to approximate maximal lotteries. Each worker evaluates responses generated by a perturbed policy against responses sampled from the current policy, and the resulting preference votes determine the shared parameter update. The method avoids backward passes and a separate reference model, while allowing parameter updates to be reconstructed from shared random seeds and logged votes.
The authors evaluate VerDICT on synthetic preference games, TL;DR summarization, and two human preference datasets (CoVal and MultiPref), using Qwen3-0.6B, Qwen3-8B, and Llama-3.1-8B-Instruct. The synthetic experiments show that VerDICT approaches the maximal lottery in rock-paper-scissors and selects the Condorcet winner in a voting game involving a dominated alternative. On language tasks, VerDICT generally outperforms EGGROLL-DPO in judged win rates while requiring less computation per training step.
Critical strengths
The paper addresses an interesting problem at the intersection of decentralized training, preference optimization, and social choice theory. Using maximal lotteries as the target provides a clear motivation for handling inconsistent or cyclic preferences, rather than fitting a scalar reward that may fail to represent majority preferences. The inference-only training procedure also provides practical advantages in memory requirements and communication costs.
Critical weaknesses
- A central motivation of the paper is to learn maximal lotteries, particularly when preferences are cyclic. However, the strongest evidence for this property comes from synthetic games involving only a few alternatives. In the rock-paper-scissors experiment, VerDICT requires multiple opponent samples to approach the maximal lottery, while produces cycling behavior. Notably, all language-task experiments use , for which the algorithm reduces to self-play against the current policy. The paper does not provide a convergence guarantee for these dynamics. Furthermore, Appendix B.1 demonstrates that even with , the implemented algorithm can converge to a biased solution in a non-uniform cyclic game. This raises questions about whether the proposed approximation reliably preserves the desired social choice properties beyond simple synthetic settings.
- Another critial weakness is that this paper does not deal with decentralized collective preference aggregation. In the experiments, this paper draws the preference votes from a shared learned judge rather than heterogeneous preferences. This experiments primarily demonstrate that evolution strategies can optimize an aggregated preference signal using scalar feedback. So experiments should involve distinct participant preference functions and group-level evaluation
Questions for authors
- Appendix D.2 includes GD Online ML on synthetic games, but not on language tasks. How does VerDICT compare with gradient-based optimization of the same sampled-opponent objective on realistic tasks?
- Some synthetic experiments use different sampling budgets or perturbation settings for VerDICT and EGGROLL-DPO. For example, the rock-paper-scissors experiments use 256 prompts per member for VerDICT and 64 for EGGROLL-DPO. Could the authors provide results under matched preference-query budgets?
- VerDICT is faster per step than EGGROLL-DPO, but the gradient-based baselines process substantially fewer preference comparisons per step and often require fewer total GPU-hours in Table 16. Can the authors provide performance-versus-compute curves under matched total computational budgets, including judge evaluation costs?
6 comments
Revised PDF Upload with Additional Experimental Results (Text in Blue)
We have uploaded a revised document. To answer W1, we have added guarantees for an idealised VerDICT update in Appendix section B.2.
To address W2, we have added a free-text game with simulated voter groups with distinct rankings and a known maximal lottery in Appendix section C.4.
To answer Q1, we have added Table 8 which shows GD Online ML is higher than VerDICT on TL;DR and MultiPref and level on CoVal and Community Alignment, with fewer GPU-hours.
To answer Q2, we have added Table 19, which shows the results hold with matched 64 prompts for both methods in five of six cells. The exception is rock-paper-scissors with Llama-3.1-8B-Instruct, where VerDICT ends on a single move. Appendix F.4 attributes this to the higher matched learning rate.
To answer Q3, we have added Table 9 and Figure 9. On TL;DR at Qwen3-0.6B, GD Online DPO reaches VerDICT’s final win rate with 47 times fewer GPU-hours, and VerDICT is above EGGROLL-DPO at every checkpoint.
Reminder to Engage in Discussion
We would like to kindly prompt our reviewer to respond to our responses in order for us to proceed with further discussions before the opportunity ends, thank you.
Rebuttal by Authors
We thank the reviewer for the thorough and detailed review, and for acknowledging the advantages of VerDICT in terms of targeting the maximal lottery and leveraging the inference-only nature of EGGROLL.
W1
We appreciate the concern that our evidence of approximation of maximal lottery properties is only demonstrated on games, and recognise that a theoretical analysis of the convergence properties of VerDICT would help strengthen our social choice claim. We address this concern in the general comment as well, and we will present a section in the appendix of our paper showing a theoretical analysis of an idealised version of the update, and list conditions the implementation does not meet by Monday the 16th of November.
We also acknowledge that the text tasks all run nb=1, and this is self-play. We tested the effect of using multiple opponents on the text tasks (Table 5). This did not change win rates much and cost more per step, so we set nb to 1. This achieves the Condorcet winner as shown in the voting game, but it does not achieve convergence under a cycle, as shown with RPS with nb=1. Regarding the biased solution shown in B.1, we agree, and only claim that VerDICT approximates the maximal lottery.
Dealing with Preference Aggregation
We appreciate and understand the reviewer’s concern regarding whether using a single trained judge is sufficient to represent true decentralised collective preference aggregation. Regarding decentralisation, EGGROLL enables individual workers to submit scalar votes which are aggregated at the end of the scoring process, making VerDICT’s training process decentralised.
Regarding collective preference aggregation, we apologise for not clarifying the mechanism by which we are evaluating candidate responses in the VerDICT algorithm. The judge that we train for the human preference data represents the probability that a random individual from the population prefers answer A to answer B, thus it is a fair representation of a decentralised collective preference.
We further clarify this with an example: assume we have 3 individuals who each have the following preferences:
User 1: A≻B
User 2: A≻B
User 3: A≻B
If you sample the individuals uniformly, this can be represented by P(A≻B)=66. Or, if you want to assume that individuals are stochastic, then:
P1(A≻B)=30
P2(A≻B)=30
P3(A≻B)=90
If you sample the individuals uniformly, this can be represented by P(A≻B)=50. To emphasise again, P(A≻B) is not the probability a unique individual prefers A to B. It is the probability that a random citizen from our sample prefers A to B. We apologise for not being clear enough, and appreciate that this does not fully address the concern. We will add a text task judged by simulated voters with distinct preferences with a known maximal lottery to a section in the appendix by Monday the 16th of November.
Q1: GD Online ML on Language Tasks
We agree that Appendix D.2 only covered the games. We have now run GD Online ML on all four text tasks with Qwen3-0.6B. We find that GD on the VerDICT objective beats VerDICT on TL;DR and MultiPref, and level on CoVal and Community Alignment. The win rates are 95.3% against 83.8% on TL;DR and 66.2% against 60.4% on MultiPref, and GD Online ML used 3.4 to 9.1 times fewer GPU-hours. We ran one untuned learning rate and only tested the 0.6B model. This shows that the objective also works under GD, but EGGROLL enables inference-only computation and workers that send only votes.
Q2: Matched Sampling Budgets
We agree that some synthetic experiments use different sampling budgets for VerDICT vs EGGROLL-DPO, and thank the reviewer for pointing this out. We reran VerDICT at EGGROLL-DPO’s 64 prompts, finding VerDICT maintains its lead against EGGROLL-DPO despite this. On RPS at Qwen3-0.6B, VerDICT’s final exploitability is 0.027 against 0.542 for EGGROLL-DPO.
Q3: Performance-Vs-Compute Curves
We agree both that the gradient baselines process fewer comparisons per step and often require fewer GPU hours. We appreciate the reviewer’s suggestion to produce performance-vs-compute curves under matched computational budgets to demonstrate the performance of these methods more clearly. We have created these curves for Qwen3-0.6B on TL;DR, finding that GD Online DPO reaches VerDICT’s final win-rate with 47 times fewer GPU-hours, including the judge evaluation costs. VerDICT is ahead of EGGROLL-DPO at every checkpoint.
We will share the full results of these three experiments in additional appendix sections by Monday the 16th of November.
We once again thank the reviewer for the review and hope that they improve their support of the paper. We invite the reviewer to notify us of further arguments against the paper or additional issues they wish to point out.