acceptodds
Under review as a conference paper at ICLR 2027

VerDICT: Verifiable Decentralised Inference-only Collective Training

Abstract

As AI systems increasingly serve users with differing interests, training shared models requires both efficient participation and principled preference aggregation. We introduce VerDICT, a decentralised training approach that allows participants to contribute computation while expressing their preferences over model outputs. VerDICT combines EGGROLL with a scalar pairwise preference objective targeting maximal lotteries, which accommodate cyclic preferences and select the Condorcet winner when one exists. Training operates entirely through inference, requiring no backward passes or separate reference model, and shares baseline responses across workers to reduce computational costs. Each worker sends only its votes, and every update is a deterministic function of the shared seed and the logged votes, so it can be recomputed and checked. We assume workers report their preferences honestly. On controlled preference games VerDICT approaches the maximal lottery where DPO under the same optimiser does not, and on summarisation and two human preference datasets it improves judged quality over the initial policy at lower per-step cost than EGGROLL-DPO.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
pluralistic alignment, maximal lotteries, social choice theory, preference learning, evolution strategies, LLM post-training, intransitive preferences, RLHF

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_XsBWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces VerDICT, a decentralized, inference-only framework for collective preference optimization. The method combines EGGROLL, a low-rank evolution strategies algorithm, with a pairwise preference objective designed to approximate maximal lotteries. Each worker evaluates responses generated by a perturbed policy against responses sampled from the current policy, and the resulting preference votes determine the shared parameter update. The method avoids backward passes and a separate reference model, while allowing parameter updates to be reconstructed from shared random seeds and logged votes.

The authors evaluate VerDICT on synthetic preference games, TL;DR summarization, and two human preference datasets (CoVal and MultiPref), using Qwen3-0.6B, Qwen3-8B, and Llama-3.1-8B-Instruct. The synthetic experiments show that VerDICT approaches the maximal lottery in rock-paper-scissors and selects the Condorcet winner in a voting game involving a dominated alternative. On language tasks, VerDICT generally outperforms EGGROLL-DPO in judged win rates while requiring less computation per training step.

Critical strengths

The paper addresses an interesting problem at the intersection of decentralized training, preference optimization, and social choice theory. Using maximal lotteries as the target provides a clear motivation for handling inconsistent or cyclic preferences, rather than fitting a scalar reward that may fail to represent majority preferences. The inference-only training procedure also provides practical advantages in memory requirements and communication costs.

Critical weaknesses
  1. A central motivation of the paper is to learn maximal lotteries, particularly when preferences are cyclic. However, the strongest evidence for this property comes from synthetic games involving only a few alternatives. In the rock-paper-scissors experiment, VerDICT requires multiple opponent samples nb=4n_b=4 to approach the maximal lottery, while nb=1n_b=1 produces cycling behavior. Notably, all language-task experiments use nb=1n_b=1, for which the algorithm reduces to self-play against the current policy. The paper does not provide a convergence guarantee for these dynamics. Furthermore, Appendix B.1 demonstrates that even with nb=4n_b=4, the implemented algorithm can converge to a biased solution in a non-uniform cyclic game. This raises questions about whether the proposed approximation reliably preserves the desired social choice properties beyond simple synthetic settings.
  2. Another critial weakness is that this paper does not deal with decentralized collective preference aggregation. In the experiments, this paper draws the preference votes from a shared learned judge rather than heterogeneous preferences. This experiments primarily demonstrate that evolution strategies can optimize an aggregated preference signal using scalar feedback. So experiments should involve distinct participant preference functions and group-level evaluation
Questions for authors
  1. Appendix D.2 includes GD Online ML on synthetic games, but not on language tasks. How does VerDICT compare with gradient-based optimization of the same sampled-opponent objective on realistic tasks?
  2. Some synthetic experiments use different sampling budgets or perturbation settings for VerDICT and EGGROLL-DPO. For example, the rock-paper-scissors experiments use 256 prompts per member for VerDICT and 64 for EGGROLL-DPO. Could the authors provide results under matched preference-query budgets?
  3. VerDICT is faster per step than EGGROLL-DPO, but the gradient-based baselines process substantially fewer preference comparisons per step and often require fewer total GPU-hours in Table 16. Can the authors provide performance-versus-compute curves under matched total computational budgets, including judge evaluation costs?
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the thorough and detailed review, and for acknowledging the advantages of VerDICT in terms of targeting the maximal lottery and leveraging the inference-only nature of EGGROLL.

W1

We appreciate the concern that our evidence of approximation of maximal lottery properties is only demonstrated on games, and recognise that a theoretical analysis of the convergence properties of VerDICT would help strengthen our social choice claim. We address this concern in the general comment as well, and we will present a section in the appendix of our paper showing a theoretical analysis of an idealised version of the update, and list conditions the implementation does not meet by Monday the 16th of November.

We also acknowledge that the text tasks all run nb=1n_b=1, and this is self-play. We tested the effect of using multiple opponents on the text tasks (Table 5). This did not change win rates much and cost more per step, so we set nbn_b to 1. This achieves the Condorcet winner as shown in the voting game, but it does not achieve convergence under a cycle, as shown with RPS with nb=1n_b=1. Regarding the biased solution shown in B.1, we agree, and only claim that VerDICT approximates the maximal lottery.

Dealing with Preference Aggregation

We appreciate and understand the reviewer’s concern regarding whether using a single trained judge is sufficient to represent true decentralised collective preference aggregation. Regarding decentralisation, EGGROLL enables individual workers to submit scalar votes which are aggregated at the end of the scoring process, making VerDICT’s training process decentralised.

Regarding collective preference aggregation, we apologise for not clarifying the mechanism by which we are evaluating candidate responses in the VerDICT algorithm. The judge that we train for the human preference data represents the probability that a random individual from the population prefers answer A to answer B, thus it is a fair representation of a decentralised collective preference.

We further clarify this with an example: assume we have 3 individuals who each have the following preferences:

User 1: A≻BA\succ B

User 2: A≻BA\succ B

User 3: A≻BA\succ B

If you sample the individuals uniformly, this can be represented by P(A≻B)=66P(A\succ B)=66%. Or, if you want to assume that individuals are stochastic, then:

P1(A≻B)=30P_1(A\succ B)=30%

P2(A≻B)=30P_2(A\succ B)=30%

P3(A≻B)=90P_3(A\succ B)=90%

If you sample the individuals uniformly, this can be represented by P(A≻B)=50P(A\succ B)=50%. To emphasise again, P(A≻B)P(A\succ B) is not the probability a unique individual prefers A to B. It is the probability that a random citizen from our sample prefers A to B. We apologise for not being clear enough, and appreciate that this does not fully address the concern. We will add a text task judged by simulated voters with distinct preferences with a known maximal lottery to a section in the appendix by Monday the 16th of November.

Q1: GD Online ML on Language Tasks

We agree that Appendix D.2 only covered the games. We have now run GD Online ML on all four text tasks with Qwen3-0.6B. We find that GD on the VerDICT objective beats VerDICT on TL;DR and MultiPref, and level on CoVal and Community Alignment. The win rates are 95.3% against 83.8% on TL;DR and 66.2% against 60.4% on MultiPref, and GD Online ML used 3.4 to 9.1 times fewer GPU-hours. We ran one untuned learning rate and only tested the 0.6B model. This shows that the objective also works under GD, but EGGROLL enables inference-only computation and workers that send only votes.

Q2: Matched Sampling Budgets

We agree that some synthetic experiments use different sampling budgets for VerDICT vs EGGROLL-DPO, and thank the reviewer for pointing this out. We reran VerDICT at EGGROLL-DPO’s 64 prompts, finding VerDICT maintains its lead against EGGROLL-DPO despite this. On RPS at Qwen3-0.6B, VerDICT’s final exploitability is 0.027 against 0.542 for EGGROLL-DPO.

Q3: Performance-Vs-Compute Curves

We agree both that the gradient baselines process fewer comparisons per step and often require fewer GPU hours. We appreciate the reviewer’s suggestion to produce performance-vs-compute curves under matched computational budgets to demonstrate the performance of these methods more clearly. We have created these curves for Qwen3-0.6B on TL;DR, finding that GD Online DPO reaches VerDICT’s final win-rate with 47 times fewer GPU-hours, including the judge evaluation costs. VerDICT is ahead of EGGROLL-DPO at every checkpoint.

We will share the full results of these three experiments in additional appendix sections by Monday the 16th of November.

We once again thank the reviewer for the review and hope that they improve their support of the paper. We invite the reviewer to notify us of further arguments against the paper or additional issues they wish to point out.

AuthorsOpenReview ↗

Revised PDF Upload with Additional Experimental Results (Text in Blue)

We have uploaded a revised document. To answer W1, we have added guarantees for an idealised VerDICT update in Appendix section B.2.

To address W2, we have added a free-text game with simulated voter groups with distinct rankings and a known maximal lottery in Appendix section C.4.

To answer Q1, we have added Table 8 which shows GD Online ML is higher than VerDICT on TL;DR and MultiPref and level on CoVal and Community Alignment, with fewer GPU-hours.

To answer Q2, we have added Table 19, which shows the results hold with matched 64 prompts for both methods in five of six cells. The exception is rock-paper-scissors with Llama-3.1-8B-Instruct, where VerDICT ends on a single move. Appendix F.4 attributes this to the higher matched learning rate.

To answer Q3, we have added Table 9 and Figure 9. On TL;DR at Qwen3-0.6B, GD Online DPO reaches VerDICT’s final win rate with 47 times fewer GPU-hours, and VerDICT is above EGGROLL-DPO at every checkpoint.

AuthorsOpenReview ↗

Reminder to Engage in Discussion

We would like to kindly prompt our reviewer to respond to our responses in order for us to proceed with further discussions before the opportunity ends, thank you.

Reviewer_zTDFOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Distributed collective training traditionally uses gradient descent methods, which are bandwidth heavy. Prior work, EGGROLL, switches to an evolution strategies method, which allows for verification. This work improves upon the performance of EGGROLL-DPO at a lower per-step cost. The authors approach this work with a social choice lens, which is suited for the problem of dealing with the issue of users with conflicting preferences contributing to training. VerDICT targets a maximal lottery. The authors demonstrate their method with two toy preference games in addition to a subset of the TL;DR dataset and two human preference datasets. On these datasets, the authors show general improvements in performance against EGGROLL-DPO and GD Online IPO, while coming closest to GD Online DPO.

Critical strengths
  • The paper’s empirical gains match its overall narrative, a pluralistic improvement upon the existing EGGROLL work. On games with known answers, the authors show that VerDICT is able to approach the maximal lottery on rock-paper-scissors, whereas EGGROLL-DPO cannot. I thought Figure 3 made this clear. On the text tasks, VerDICT has a higher mean win rate than EGGROLL-DPO in nearly all settings and runs faster. They use ablations to show that the improvements on the games come from the objective.

  • The objective is grounded in social choice, and targets the maximal lottery. This is an interesting frame for the problems posed by decentralized training. The social choice grounding can be seen in Table 7 of the appendix, which shows the properties of the maximal lottery compared with the Borda count. Dealing with the inconsistency of opinions is key with the conflicting signals decentralized training can create - a good domain for social choice.

Critical weaknesses
  • The pluralistic claim is only tested in the games, and the text experiments use one training judge per dataset. The paper’s claims of consistency are not rigorously tested on real data. Table 8 shows <5% cyclical disagreement. Appendix Table 9 shows VerDICT minimally changes how often it picks the Condorcet winner. The authors use a metric of win rate to judge performance, and with that metric the results are mixed w.r.t. gradient-based outcomes.

  • Verification is not tested in any main body experiment. The method is measured on win rate and cost. But as seen in Appendix Table 16, the gradient baselines use fewer training hours than VerDICT in every timed run with mixed results on win rate

Questions for authors

My primary issue with this paper is the unclear story on beating baselines. Gradient descent is not consistently beaten on win rate, and it seems consistency is at best modestly improved from EGGROLL-DPO in real datasets. Why does this not matter, or do you have numbers that make the story clearer?

  • Could you provide numbers on how the method would perform at the scale of real-world experiments with judges that disagree? If that isn’t feasible, could you provide theoretical analysis?
  • How does VerDICT compare with gradient methods at matched compute? If it continues to be mixed, why should your method be used in place of existing ones in practical settings?
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the thoughtful and considerate review, and for highlighting the alignment of our empirical data with the narrative of our paper as well as the positive feedback about our social choice framing.

Pluralistic Claim Only Tested in Games

We will first address the reviewer’s concern regarding the fact that pluralism is only tested in games. We respond to this concern in our general comment above, agreeing that a demonstration of pluralism would help strengthen our claim to provide a method for pluralistic alignment.

GD is Not Consistently Beaten by VerDICT

We will next address the concern that our comparisons with GD show that GD is not consistently beaten by VerDICT on win-rate. We also respond to this concern in our general comment above, highlighting that we do not claim to have a method which outperforms GD, but rather a method that combines several properties desirable for collective training. On the comment about matched compute, we agree this is important to include for fair comparison and so will add experimental results for this on matched GPUs to a section of the appendix by Monday the 16th of November. We have compared performance-vs-compute curves for Qwen3-0.6B on TL;DR, finding that GD Online DPO reaches VerDICT’s final win-rate with 47 times fewer GPU-hours, including the judge evaluation costs. VerDICT is ahead of EGGROLL-DPO at every checkpoint.

How the Method Would Perform at the Scale of Real-World Experiments

We understand this question as asking for an experiment that combines the property of our games which have multiple disagreeing voters with the scale of our human data experiments. We apologise for not clarifying the mechanism by which we are evaluating candidate responses in the VerDICT algorithm. The judge that we train for the human preference data represents the probability that a random individual from the population prefers answer A to answer B, thus it is a fair representation of a decentralised collective preference. Because the judge is trained to return a preference of one response over another P(A≻B)P(A \succ B), this is enough to learn a function that returns the probability that a randomly sampled individual prefers A to B. We will provide a text task with simulated voters to a section in the paper appendix to further address this directly, as well as a section in the appendix demonstrating a theoretical analysis of an idealised version of the algorithm by Monday the 16th of November.

If Performance is Mixed at Matched Compute, Why Should Our Method Be Used In Place Of Existing Ones In Practical Settings? It is the decentralisation that would let VerDICT enable verifiable decentralised pluralistic alignment training. In VerDICT, the workers run inference-only and send only votes, with no gradients or parameters. This reduces cost and enables verification, which we will show experimentally in the appendix of the paper that we will add by Monday the 16th of November.

Verification Experiment

To clarify the concern about verification, we agree that we did not include experimental results which show VerDICT to be verifiable as we claim it to be. We will include the results of a simple verification experiment into a section of the appendix of the paper by Monday the 16th of November. We perform an exact replay of a 300-step TL;DR training run, and show that the model is identical on replaying the training process. We also show that 84 tampered versions of the log change the weights of the model 79 cases, where the edit changed a member’s votes. In 5 cases, the log was changed but each member’s total votes stayed the same.

We once again thank the reviewer for the review and hope that they improve their support of the paper. We invite the reviewer to notify us of further arguments against the paper or additional issues they wish to point out.

AuthorsOpenReview ↗

Revised PDF Upload with Additional Experimental Results (Text in Blue)

We have uploaded a revised document. In Appendix section B.5, we have added a verification experiment replaying a 300-step run from the same seed that logs votes and reproduces the same model weights. 79 of 84 edits to the vote log changed the weights, and the other five left every member’s vote total unchanged.

On matched compute, we have added results for this in Appendix D.3, table 9 and figure 9. We show GD Online DPO reaches VerDICT’s final TL;DR win rate with 47 times fewer GPU-hours, and VerDICT is above EGGROLL-DPO at every checkpoint.

Regarding disagreeing voters on text, we have added a free-text game with simulated voter groups and a known maximal lottery in Appendix section C.4. VerDICT moves the answer to the Condorcet winner, including when the Borda winner is different, and EGGROLL-DPO stays close to the base model. On the cyclic profile no method holds the mixed lottery. Appendix B.2 gives the theoretical analysis.

AuthorsOpenReview ↗

Reminder to Engage in Discussion

We would like to kindly prompt our reviewer to respond to our responses in order for us to proceed with further discussions before the opportunity ends, thank you.

Reviewer_JsTjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VerDICT is a decentralized LLM training method (building upon EGGROLL) with an objective modified that encourages the output distribution to approximate a maximal lottery. Since computing a full preference matrix is intractable, VerDICT’s approximation samples opponents, collects noisy preference feedback and estimates an update using evolution strategies. The optimization procedure is low-rank and inference-only; participants’ feedback are decentralized and model updates can be reconstructed and verified from logged votes and shared random seeds. VerDICT’s Condorcet consistency is empirically demonstrated on 2 controlled preference games; the authors also show competitive performance on human preference datasets (CoVal and MultiPref) and in text summarization.

Critical strengths

S1. Problem and approach. This paper addresses an important shortcoming in how human preferences are presently aggregated and collapsed into a scalar. Current paradigms could plausibly result in arbitrary deterministic tie-breaking rules being learned, and Borda-like behaviour. VerDICT’s approach to instead learn an LLM whose output distribution approximates maximal lotteries is interesting and apt, since their existence is guaranteed even if no Condorcet winner exists. This is a promising way to respect majority rule and handles cyclical preferences.

S2. Framing, presentation and literature review. I found this paper clearly written and the literature review—from game-theoretic equilibria to social choice theory, to decentralized LLM training techniques—thorough and well-organized. Each of the 3 experimental settings considered are very well explained and appropriately visualized, esp. RPS in Tab. 1 and the Condorcet vs. Borda behaviour in Tab. 2. Empirical claims are well substantiated by exact (in toy cases) or approximated (proxy) metrics (CoVal, MultiPref), and effectively communicated in tabular and graphical formats.

Critical weaknesses

W1. The VerDICT objective is approximate (as acknowledged by the authors), therefore, there are no guarantees that VerDICT would empirically converge to learning maximal lotteries. Therefore, it might be helpful to strengthen the empirical analyses, e.g., by comparing the output distribution of VerDICT to an LP-estimate (see Q1.), or by providing more detailed description of the experimental parameters (e.g., number of votes, independence of votes, support) minimal necessary for this to empirically hold.

W2. The motivation for this work could use further elaboration, in the context general human preference data and LLM tasks. For example, what do maximal lotteries mean in the context of text summarization? What are the real-world tasks of interest especially given that cyclical preferences are rare in real human data?

Questions for authors

Q1. I understand that the authors perform “supervised evaluation” in the 2 toy settings where the maximal lottery is known. On the 2 human preference datasets where this is not the case, the authors use proxies like held-out judge win rate and exploitability (CoVal only). Have the authors considered some approximate supervised evaluation via linear programming? E.g. construct the pairwise majority-margin matrix; solve the LP to estimate the maximal lottery; measure VerDICTs output distribution over the same responses; compare the LP-pseudolabel and VerDICT results? This would increase confidence in claims that VerDICT does empirically converge to learning maximal lotteries in less structured settings.

4 comments
AuthorsOpenReview ↗

Rebuttal by Authors

We thank the reviewer for the careful and considerate review, and for the positive feedback on our framing and presentation.

Pluralistic Claim Only Tested in Games

We will first address the reviewer’s concern regarding the fact that pluralism is only tested in games. We respond to this concern in our general comment above, agreeing that a demonstration of pluralism would help strengthen our claim to provide a method for pluralistic alignment.

The VerDICT Objective is Approximate

There is no guarantee to VerDICT algorithm as we implemented it, but we will present a section in the appendix of our paper showing a theoretical analysis of an idealised version of the algorithm by Monday the 16th of November, listing conditions the implementation requires in terms of the number of votes, independence of votes, and support of the opponent distribution as you suggest, and those it does not meet.

Linear Programming Experiment

We ran the described experiment and will include details in the appendix of our paper by Monday the 16th of November. No model trained with any of the methods in our paper significantly changes its distribution over the four candidate responses, we think it makes sense small updates will not change how likely it is our model produces outputs created by the process that generated the CoVal data by much. We will also include an additional experiment comparing the answers generated by the trained with the linear programme maximal lottery.

Maximal lotteries and text summarisation

First, the TL;DR task serves as a check to show that VerDICT can learn an ordinary text task. Next we wish to highlight that maximal lotteries are great for situations where there is no clear agreement. When there is a clear agreement, maximal lotteries select that answer because maximal lotteries are majority and Condorcet consistent. If there exists a way to summarize that is believed to be superior, then people agree on the best way to summarize. Maximal lotteries would give you precisely that summary, or a distribution over summaries.

See Maura-Rivero et al. (2025), appendix A.4., which considers the objection that the output space is very large. They claim that the large possible combination of tokens makes it unlikely that a Condorcet winner preferred by a majority would exist. Tthis does not mean Condorcet principles are irrelevant because the Smith set, the smallest non-empty set of alternatives such that every alternative inside the set beats every alternative outside it, generalises the Condorcet winner.

Maura-Rivero et al. (2025) provide an example. For the prompt “Summarize the French Revolution” for example no single summary might be completely preferred, but a Smith set could separate preferred from dispreferred summaries. The summaries in this set would reflect the core of the preferred summaries despite the noise of LLM generations. While real Condorcet winners may be rare over the full space of possible responses, properties related to the Smith set like Condorcet consistency and the majority criterion ensure the LLM prioritises outputs a majority prefers over the alternatives.

The real world tasks of interests

We would first like to highlight that since VerDICT is a general training method, and we demonstrate it can learn regular tasks, we want to make sure that we have a method that handles cyclical preferences when they arise. We are presenting a combined method that enables decentralised training while also handling social-choice problems effectively when they arise. While cyclic preferences do not arise all the time, it is a desirable property of a method that handles preference aggregation.

Cyclic preferences do occur in human data. In CoVal there are 1.3%, and 7.1% of prompts have no Condorcet winner. At scale that proportion would be far from insignificant, and should be handled appropriately. Furthermore, McCune & McCune (2021) find Condorcet cycles in the Minneapolis Ward 2 city council race, McCune (2023) finds no Codorcet winner in the Oakland 2022 District 4 school director election, and Kurrild-Klitgaard (2001) finds a cycle in survey of Danish prime ministerial election data.

We once again thank the reviewer for their review and hope that these additions strengthen their support of the paper. We invite the reviewer to notify us of further arguments against the paper or additional issues they wish to point out.

References

  1. Kurrild-Klitgaard, P. (2001). An Empirical Example of the Condorcet Paradox of Voting in a Large Electorate. Public Choice, 107(1), 135–145. https://doi.org/10.1023/A:1010304729545
  2. McCune, D., & McCune, L. (2021). The Curious Case of the 2021 Minneapolis Ward 2 City Council Election (arXiv:2111.09846). arXiv. https://doi.org/10.48550/arXiv.2111.09846
  3. McCune, D. (2023). Ranked Choice Bedlam in a 2022 Oakland School Director Election (arXiv:2303.05985). arXiv. https://doi.org/10.48550/arXiv.2303.05985
AuthorsOpenReview ↗

Revised PDF Upload with Additional Experimental Results (Text in Blue)

We have uploaded a revised document. In Appendix section B.2, we have added guarantees for an idealised VerDICT update, which we hope the reviewer finds helpful in response to W1. The terms in Theorem B.3 correspond to the three quantities the reviewer named: votes per opponent, independence of the votes used for selection and scoring (Assumption 4), and support of the opponent distribution (Assumption 3). Remark B.7 lists where our implementation differs.

On Q1, see Appendix section E.4 and Table 15. On the four annotator candidates no trained policy differs from the initial policy, as we state in our first response. On the 36-answer comparison, VerDICT's generations lose least often to a draw from the linear programme lottery. (0.428, against 0.375 for the initial policy, 0.393 for EGGROLL-DPO and 0.390 for GD Online DPO). No policy reaches 0.500.

On W2, Appendix C.4 adds a text game with a known lottery.

AuthorsOpenReview ↗

Reminder to Engage in Discussion

We would like to kindly prompt our reviewer to respond to our responses in order for us to proceed with further discussions before the opportunity ends, thank you.

Thank you for additionally analysing the idealised VerDICT update and comparing with a maximal lottery computed by linear programming. These additions partly address my concern about how closely VerDICT approximates a maximal lottery and clarify where the implementation departs from theoretical assumptions. Convergence on real preference data remains a relevant, open question. I continue to think the paper is a good candidate for acceptance and maintain my score of 3.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Comment

We thank the reviewers for their considerate comments and suggestions, and appreciate that all three reviewers agree with us that the problem and social choice framing for this setting are worthwhile.

We first would like to clarify the contribution of our paper. We apologise for not making our contribution clear, and appreciate our reviewers pointing this ambiguity out to us. The combination of decentralised training, verifiability, inference-only computation, and collective preference aggregation is the core of our contribution. To our knowledge, no method currently offers the combination of these things. Techniques offer decentralised training, or maximal lottery approximation, or high win-rate, but they all come with tradeoffs, and VerDICT unifies these properties. The fact that we have a method that combines all of these things makes it feasible to do decentralised pluralistic alignment which approximates the maximal lottery.

We claim that in games where the maximal lottery is known, VerDICT approximates it and EGGROLL-DPO does not, and that VerDICT performs competitively on text tasks. We do not claim that our algorithm as implemented converges to the maximal lottery, that our method is more compute-efficient than gradient descent, or that our current text tasks demonstrate pluralism by themselves.

We note two shared concerns raised in more than one review that we wish to address in this general comment separately from our responses to the individual reviews.

The First is That Pluralism is Only Shown in Games

We agree with this. Unfortunately, as identified by reviewers zTDF and JsTj, CoVal and MultiPref did not contain many examples which demonstrate approximation of the maximal lottery. We also acknowledge XsBW’s comment that our setup only supplies evidence of maximal lottery approximation from synthetic games.

As shown in Table 8, it is clear that situations where the Condorcet winner and Borda winner differ are scarce. To address this, we are adding a free-answer text task judged by simulated voters in which there is a known maximal lottery, the results of which we will share by Monday the 16th of November.

The Second Is That GD Is Not Consistently Beaten By VerDICT

We do not try to claim in the paper that VerDICT beats GD on win rate or compute. We want to emphasise that our contribution is more than just training accuracy. We present a unique algorithm designed to enable decentralised training, approximation of the maximal lottery, and good win-rates simultaneously. Because our method combines these properties, it takes a step towards decentralised pluralistic alignment with theoretically sound preference aggregation. That is our contribution.

We agree that it is important to highlight the disparity between the respective win-rates of each optimiser for the sake of comparison, so we are sharing the win-rates of VerDICT, EGGROLL-DPO, and GD at matched GPUs in an updated version of our paper by Monday the 16th of November. We have compared performance-vs-compute curves for Qwen3-0.6B on TL;DR, finding that GD Online DPO reaches VerDICT’s final win-rate with 47 times fewer GPU-hours, including the judge evaluation costs. VerDICT is ahead of EGGROLL-DPO at every checkpoint.

We once again thank the reviewers for their thoughtful engagement, and are happy to clarify other things during the discussion phase that will help the reviewers clarify these points which are not addressed in our individual responses.

AuthorsOpenReview ↗

Revised PDF Upload with Additional Experimental Results (Text in Blue)

We have uploaded a revised document containing experimental results requested by our reviewers. We thank the reviewers for suggesting these and hope that these help in the discussion process. Changes to the main body text, which include links to their respective sections, have been written in blue to be easily identifiable.

  • In Appendix section B.2, we have added guarantees for an idealised VerDICT update.
  • In Appendix section B.5, we have added a verification experiment.
  • In Appendix section C.4, we have added a free-text game with simulated voter groups and a known maximal lottery
  • In Appendix section D.2.1, we have added a win-rate vs GPU-hours on TL;DR
  • In Appendix section D.3, we have added a comparison with a linear programme maximal lottery on CoVal.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.