- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
MoDA addresses the tendency of aligned language models to produce similar answers to open-ended prompts. It assigns numbered modes to a shared policy and rewards each mode for producing an answer that differs from the others, provided a reference-calibrated quality threshold is met. The authors evaluate diversity on open-ended generation tasks and check standard model capabilities, with ablations for role prompting and components of the reward.
Critical strengths
- The conditioning mechanism is simple. Numbered modes give the user a way to ask one model for multiple directions without building separate models for those directions. I can imagine this being useful in brainstorming systems, where repeating the same suggestion with different wording is a common problem.
- The paper makes a real attempt to rule out an easy explanation for its diversity gains. Role prompting alone is included, and the ablations show that the explicit diversity term matters. The quality-only and diversity-only variants are also useful: the latter obtains high diversity but degrades ordinary accuracy, which makes the quality gate more convincing.
- There is substantial experimental coverage. Infinite-Chat is a natural place to test this idea, and the results against DivPO are large: SBERT diversity increases from 0.274 to 0.482. I also like that the authors include general-capability results rather than evaluating generated variety in isolation.
- Keeping the modes in the same policy is an attractive design choice. It avoids maintaining a collection of independently trained agents, even though the training itself still requires multiple candidate generations per prompt.
Critical weaknesses
- I have some doubts about using embedding diversity as the main evidence of useful diversity. Two responses can be far apart in embedding space and still add little to a brainstorming session. A more telling test would ask people to compare complete sets of ideas for novelty and usefulness. The quality results already suggest why this matters: for the thinking-enabled Qwen3-8B setting, the native NoveltyBench quality score drops from 4.90 for the initial model to 3.39 for MoDA, despite the large rise in diversity. This deserves more attention in a paper motivated by scientific and creative work.
- It is also unclear how much of the result comes from stable specialization of the numbered modes. The single-role ablation changes the training configuration. I would prefer an inference-only test using the final six-mode checkpoint: sample six times under one mode and compare that set against one sample from each of the six modes. That comparison would tell us more directly whether the learned mode identities are contributing something beyond stochastic decoding.
- One practical concern is the training cost of scoring several generated responses and their pairwise differences. The method uses six responses per prompt in its reported setup. A training-time comparison with DivPO or repeated role prompting would help quantify what the diversity gains cost. The paper already evaluates inference under different modes, so the missing piece is mainly the training overhead.
Questions for authors
- Could the authors compare whole sets of generated hypotheses or ideas using human evaluation, especially for NoveltyBench, where the native quality result drops?
- For the final six-mode checkpoint, what is the diversity of six samples from one fixed mode compared with one sample per mode?
- Figure 1 appears to be a raster image, with some details becoming blurry when enlarged. I suggest replacing it with a vector figure.
- The title seems to have an unusual line break after “ALIGNMENT,” leaving a large blank space. Could the authors check whether this is a formatting issue?
6 comments
Reply to Reviewer geTb (cont.)
W2 + Q2 (“stable specialization of the numbered modes”): We thank you for suggesting additional evaluation. As per you suggested, we sample six times under one mode (fixed-role) and compare that set against one sample from each of the six modes (mixed-role). We repeat this for all six modes on 100 prompts from the held-out Infinite-Chat.
| Set (k = 6) | Disc. | SBERT | E-Vendi |
|---|---|---|---|
| Mixed: 6 different roles | 0.468 [0.455, 0.482] | 0.478 [0.454, 0.499] | 3.343 [3.221, 3.457] |
| Fixed: one role ×6, mean over roles 1–6 | 0.463 [0.449, 0.477] | 0.440 [0.419, 0.458] | 3.160 [3.054, 3.259] |
| Fixed: Role 1 ×6 | 0.451 [0.434, 0.468] | 0.373 [0.346, 0.401] | 2.785 [2.657, 2.916] |
| Fixed: Role 2 ×6 | 0.447 [0.432, 0.463] | 0.430 [0.401, 0.459] | 3.075 [2.936, 3.214] |
| Fixed: Role 3 ×6 | 0.467 [0.453, 0.482] | 0.456 [0.430, 0.480] | 3.252 [3.122, 3.374] |
| Fixed: Role 4 ×6 | 0.466 [0.451, 0.481] | 0.460 [0.435, 0.485] | 3.286 [3.157, 3.407] |
| Fixed: Role 5 ×6 | 0.470 [0.455, 0.485] | 0.466 [0.442, 0.489] | 3.305 [3.184, 3.419] |
| Fixed: Role 6 ×6 | 0.477 [0.462, 0.491] | 0.452 [0.428, 0.475] | 3.261 [3.141, 3.373] |
| Mixed − fixed (mean over roles) | +0.005 [+0.004, +0.007] | +0.038 [+0.032, +0.044] | +0.182 [+0.155, +0.210] |
| Mixed − Role 1 | +0.017 [+0.009, +0.025] | +0.104 [+0.083, +0.126] | +0.558 [+0.462, +0.655] |
| Mixed − Role 2 | +0.021 [+0.014, +0.028] | +0.047 [+0.029, +0.067] | +0.267 [+0.189, +0.347] |
| Mixed − Role 3 | +0.001 [-0.006, +0.008] | +0.022 [+0.007, +0.037] | +0.091 [+0.028, +0.158] |
| Mixed − Role 4 | +0.002 [-0.005, +0.010] | +0.017 [+0.002, +0.032] | +0.057 [-0.015, +0.129] |
| Mixed − Role 5 | -0.002 [-0.009, +0.006] | +0.011 [-0.008, +0.029] | +0.038 [-0.037, +0.113] |
| Mixed − Role 6 | -0.008 [-0.016, -0.000] | +0.026 [+0.010, +0.042] | +0.082 [+0.007, +0.157] |
We observe that, on average, mixed-role sets are more diverse than single-role sets: +0.033 SBERT (+7.5% relative) and +0.15 E-Vendi (+4.8%), with significant CIs; Disc. changes little (+0.005). Additionally, the effect depends on the role. Role 1 and Role 2 produce significantly less diverse sets on their own, while the other four come close to the mixed sets. Role conditioning therefore does shape the output distribution. We will frame our claim of role specialization accordingly.
W3 (“Training cost”): We reported our training condition in Appendix D.1. A single MoDA training for 2 epochs runs approximately 24 hours on 4 H200 GPUs. DivPO training takes 22.5 hours on 6 H200 GPUs. Repeated role prompting is an inference-time only method.
Q3: We have replaced Figure 1 with a vector figure with more details. Please check out our updated revision.
Q4: There is a linebreak after “via” and we have removed it in the revision.
We hope these updates address your concerns and are happy to provide anything further during the discussion period. Thank you for your time in engaging in the discussion period with us. If there is any further information you require to consider raising your score, please don’t hesitate to let us know!
Supplementary Questions of Submission47285 by Reviewer geTb
Supplementary Questions:
SQ1: From your new ablation comparing fixed‑mode sampling versus mixed‑mode sampling: Roles 3‑6 achieve diversity close to the full mixed‑role condition, while Role 1 and Role 2 are noticeably weaker. Could the authors unpack what differentiates Role 1 / Role 2 from the rest of the learned modes? Are there observable differences in topic preference, style, or answer length that explain this performance gap?
SQ2: Your human evaluation is still ongoing. Assuming human raters confirm higher idea‑level diversity for MoDA, how do the authors plan to reconcile this with the observed quality drop on NoveltyBench for thinking‑enabled Qwen3‑8B? Would you suggest practical inference‑time knobs (e.g. mode subset selection, diversity‑weight sampling) to mitigate quality degradation for real‑world brainstorming use‑cases?
Reply to Reviewer geTb
Thank you for the thoughtful review. We especially appreciate your recognition of our substantial experimental coverage. We also appreciate you emphasizing that “keeping the modes in the same policy is an attractive design choice” to “avoids maintaining a collection of independently trained agents,” which makes MoDA an alignment approach that won’t compromise the model’s own capability. Your suggestions led to some useful new analyses, which we walk through below.
W1+Q1: Thank you for your feedback on the diversity evaluations. We re-measured diversity with two embedders from different model families than the training reward (all-MiniLM-L6-v2): BAAI/bge-large-en-v1.5 and Qwen3-Embedding-0.6B.
Thinking = True, Base model = Qwen3-8B
Thinking = False, Base model = Qwen3-8B
Thinking = False, Base model = Llama-3.1-8B
Thinking = False, Base model = GLM-4-9B
MoDA is still leading the board across models and configurations. This result is strong evidence that MoDA promotes generative diversity across model families.
We also want to clarify that not all the evaluation index has the same index and has the same origin as the training signal. The training uses all-MiniLM-L6-v2 embedding, but our discriminator uses all-mpnet-base-v2 embedder to construct its feature vector (Appx D.4), and therefore serves as an additional measure for diversity. Additionally, NoveltyBench has its naive distinct_k metrics, which calls LLM-as-Judge to judge the number of distinct answers. The result below shows that MoDA improves distinct_k value in NoveltyBench across model families, beating DivPO and DARLING(WildChat) in both thinking enabled and disabled settings. All values are mean ± std over n=3.
Thinking = True, Base model = Qwen3-8B
Thinking = False, Base model = Qwen3-8B
Thinking = False, Base model = Llama-3.1-8B
Thinking = False, Base model = GLM-4-9B
Together, the diversity increase across different families of metrics indicates that MoDA produces genuinely different responses rather than just spreading out in the reward embedder’s space.
Additionally, we agree that human evaluation is the gold standard to judge diversity. Adding this evaluation would greatly improve the soundness of our diversity claim. We asked 10 participants to compare MoDA with the strongest baseline, DARLING, on 26 prompts from NoveltyBench and PreScience. The participants are asked to read three answers from each model without knowing which system wrote which. For every prompt they will judge which set had more genuinely different ideas, which was more original, and which was better overall. Each prompt is rated by at least three people. We estimate that human evaluation would take about a week to complete.