acceptodds
Under review as a conference paper at ICLR 2027

Forty Shades of Blue: Quality-Diversity Alignment via Mode-Conditioned Reinforcement Learning

Abstract

A notable byproduct of LLM alignment training is mode collapse: the progressive loss of output diversity that narrows a model's expressivity at inference time. This degradation is especially limiting for applications requiring open-ended exploration and pluralistic perspectives, such as scientific ideation and creative writing. We present MoDA (Mode-conditioned Diversity Alignment), an online post-training RL algorithm that jointly optimizes generation quality and diversity, inspired by the coordination perspective in multi-agent reinforcement learning (MARL). MoDA trains a single shared LLM policy conditioned on abstract numbered roles, where each role acts as an agent competing to produce outputs distinct from the others. This formulation encourages mode-conditioned agents to explore complementary regions of the high-quality output space without requiring hand-crafted personas or architectural modifications. MoDA employs a prompt-adaptive quality gating mechanism that calibrates a reference quality threshold and grants diversity rewards only to responses that meet the threshold, preventing reward-hacking behaviors that compromise response quality. To study quality-diversity tradeoffs, we evaluate MoDA on a comprehensive suite of benchmarks spanning seven general capability tasks and four domain-specific diversity tasks in scientific ideation and creative writing. MoDA improves SBERT diversity by 265% on the Infinite-Chat held-out prompts, while increasing average general capability pass@1 by 10.3% over the Qwen3-8B baseline. Compared with the strongest DivPO baseline, MoDA improves SBERT diversity from 0.274 to 0.482 (+75.9%) and E-Vendi from 2.86 to 4.4 (+53.8%), while improving average general capability pass@1 by 7.0%. Overall, MoDA provides a drop-in alternative to standard post-training methods that preserves and expands the model's expressive output space while improving quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diverse generation, multiagent reinforcement learning, alignment

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_geTbOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

MoDA addresses the tendency of aligned language models to produce similar answers to open-ended prompts. It assigns numbered modes to a shared policy and rewards each mode for producing an answer that differs from the others, provided a reference-calibrated quality threshold is met. The authors evaluate diversity on open-ended generation tasks and check standard model capabilities, with ablations for role prompting and components of the reward.

Critical strengths
  1. The conditioning mechanism is simple. Numbered modes give the user a way to ask one model for multiple directions without building separate models for those directions. I can imagine this being useful in brainstorming systems, where repeating the same suggestion with different wording is a common problem.
  2. The paper makes a real attempt to rule out an easy explanation for its diversity gains. Role prompting alone is included, and the ablations show that the explicit diversity term matters. The quality-only and diversity-only variants are also useful: the latter obtains high diversity but degrades ordinary accuracy, which makes the quality gate more convincing.
  3. There is substantial experimental coverage. Infinite-Chat is a natural place to test this idea, and the results against DivPO are large: SBERT diversity increases from 0.274 to 0.482. I also like that the authors include general-capability results rather than evaluating generated variety in isolation.
  4. Keeping the modes in the same policy is an attractive design choice. It avoids maintaining a collection of independently trained agents, even though the training itself still requires multiple candidate generations per prompt.
Critical weaknesses
  1. I have some doubts about using embedding diversity as the main evidence of useful diversity. Two responses can be far apart in embedding space and still add little to a brainstorming session. A more telling test would ask people to compare complete sets of ideas for novelty and usefulness. The quality results already suggest why this matters: for the thinking-enabled Qwen3-8B setting, the native NoveltyBench quality score drops from 4.90 for the initial model to 3.39 for MoDA, despite the large rise in diversity. This deserves more attention in a paper motivated by scientific and creative work.
  2. It is also unclear how much of the result comes from stable specialization of the numbered modes. The single-role ablation changes the training configuration. I would prefer an inference-only test using the final six-mode checkpoint: sample six times under one mode and compare that set against one sample from each of the six modes. That comparison would tell us more directly whether the learned mode identities are contributing something beyond stochastic decoding.
  3. One practical concern is the training cost of scoring several generated responses and their pairwise differences. The method uses six responses per prompt in its reported setup. A training-time comparison with DivPO or repeated role prompting would help quantify what the diversity gains cost. The paper already evaluates inference under different modes, so the missing piece is mainly the training overhead.
Questions for authors
  1. Could the authors compare whole sets of generated hypotheses or ideas using human evaluation, especially for NoveltyBench, where the native quality result drops?
  2. For the final six-mode checkpoint, what is the diversity of six samples from one fixed mode compared with one sample per mode?
  3. Figure 1 appears to be a raster image, with some details becoming blurry when enlarged. I suggest replacing it with a vector figure.
  4. The title seems to have an unusual line break after “ALIGNMENT,” leaving a large blank space. Could the authors check whether this is a formatting issue?
6 comments
AuthorsOpenReview ↗

Reply to Reviewer geTb

Thank you for the thoughtful review. We especially appreciate your recognition of our substantial experimental coverage. We also appreciate you emphasizing that “keeping the modes in the same policy is an attractive design choice” to “avoids maintaining a collection of independently trained agents,” which makes MoDA an alignment approach that won’t compromise the model’s own capability. Your suggestions led to some useful new analyses, which we walk through below.

W1+Q1: Thank you for your feedback on the diversity evaluations. We re-measured diversity with two embedders from different model families than the training reward (all-MiniLM-L6-v2): BAAI/bge-large-en-v1.5 and Qwen3-Embedding-0.6B.

Thinking = True, Base model = Qwen3-8B

ModelSBERT · BGE-largeSBERT · Qwen3-EmbEVendi · BGE-largeEVendi · Qwen3-Emb
Qwen3-8B w/ thinking0.080 ± 0.0000.123 ± 0.0011.49 ± 0.001.76 ± 0.01
SSoT w/ thinking0.111 ± 0.0010.178 ± 0.0011.62 ± 0.001.99 ± 0.01
MoDA w/ thinking0.192 ± 0.0050.269 ± 0.0062.15 ± 0.032.63 ± 0.03
DivPO w/ thinking0.107 ± 0.0020.167 ± 0.0031.64 ± 0.011.99 ± 0.01
Persona Sampling w/ thinking0.125 ± 0.0010.215 ± 0.0021.83 ± 0.002.53 ± 0.01

Thinking = False, Base model = Qwen3-8B

ModelSBERT · BGE-largeSBERT · Qwen3-EmbEVendi · BGE-largeEVendi · Qwen3-Emb
Qwen3-8B0.075 ± 0.0010.114 ± 0.0021.45 ± 0.001.69 ± 0.01
MoDA0.176 ± 0.0010.267 ± 0.0032.09 ± 0.002.70 ± 0.02
DARLING (Mix)0.141 ± 0.0030.244 ± 0.0051.89 ± 0.022.64 ± 0.04
DARLING (WildChat)0.098 ± 0.0010.162 ± 0.0021.60 ± 0.002.05 ± 0.01
DivPO0.131 ± 0.0010.208 ± 0.0011.81 ± 0.002.32 ± 0.01

Thinking = False, Base model = Llama-3.1-8B

ModelSBERT · BGE-largeSBERT · Qwen3-EmbEVendi · BGE-largeEVendi · Qwen3-Emb
Llama-MoDA0.192 ± 0.0020.313 ± 0.0032.17 ± 0.013.04 ± 0.02
Llama-3.1-8B0.145 ± 0.0020.232 ± 0.0031.83 ± 0.012.37 ± 0.03

Thinking = False, Base model = GLM-4-9B

ModelSBERT · BGE-largeSBERT · Qwen3-EmbEVendi · BGE-largeEVendi · Qwen3-Emb
GLM-MoDA0.182 ± 0.0020.296 ± 0.0022.18 ± 0.013.09 ± 0.02
GLM-4-9B0.148 ± 0.0010.285 ± 0.0021.96 ± 0.013.03 ± 0.02

MoDA is still leading the board across models and configurations. This result is strong evidence that MoDA promotes generative diversity across model families.

We also want to clarify that not all the evaluation index has the same index and has the same origin as the training signal. The training uses all-MiniLM-L6-v2 embedding, but our discriminator uses all-mpnet-base-v2 embedder to construct its feature vector (Appx D.4), and therefore serves as an additional measure for diversity. Additionally, NoveltyBench has its naive distinct_k metrics, which calls LLM-as-Judge to judge the number of distinct answers. The result below shows that MoDA improves distinct_k value in NoveltyBench across model families, beating DivPO and DARLING(WildChat) in both thinking enabled and disabled settings. All values are mean ± std over n=3.

Thinking = True, Base model = Qwen3-8B

Modeldistinct_k (official)
Qwen3-8B w/ thinking4.63 ± 0.21
SSoT w/ thinking2.73 ± 0.06
MoDA w/ thinking6.10 ± 0.08
DivPO w/ thinking5.64 ± 0.08

Thinking = False, Base model = Qwen3-8B

Modeldistinct_k (official)
Qwen3-8B4.33 ± 0.09
MoDA7.45 ± 0.06
DARLING (WildChat)6.48 ± 0.03
DivPO6.85 ± 0.06

Thinking = False, Base model = Llama-3.1-8B

Modeldistinct_k (official)
Llama-MoDA7.50 ± 0.07
Llama-3.1-8B5.96 ± 0.12

Thinking = False, Base model = GLM-4-9B

Modeldistinct_k (official)
GLM-MoDA9.80 ± 0.02
GLM-4-9B7.69 ± 0.14

Together, the diversity increase across different families of metrics indicates that MoDA produces genuinely different responses rather than just spreading out in the reward embedder’s space.

Additionally, we agree that human evaluation is the gold standard to judge diversity. Adding this evaluation would greatly improve the soundness of our diversity claim. We asked 10 participants to compare MoDA with the strongest baseline, DARLING, on 26 prompts from NoveltyBench and PreScience. The participants are asked to read three answers from each model without knowing which system wrote which. For every prompt they will judge which set had more genuinely different ideas, which was more original, and which was better overall. Each prompt is rated by at least three people. We estimate that human evaluation would take about a week to complete.

AuthorsOpenReview ↗

Reply to Reviewer geTb (cont.)

W2 + Q2 (“stable specialization of the numbered modes”): We thank you for suggesting additional evaluation. As per you suggested, we sample six times under one mode (fixed-role) and compare that set against one sample from each of the six modes (mixed-role). We repeat this for all six modes on 100 prompts from the held-out Infinite-Chat.

Set (k = 6)Disc.SBERTE-Vendi
Mixed: 6 different roles0.468 [0.455, 0.482]0.478 [0.454, 0.499]3.343 [3.221, 3.457]
Fixed: one role ×6, mean over roles 1–60.463 [0.449, 0.477]0.440 [0.419, 0.458]3.160 [3.054, 3.259]
Fixed: Role 1 ×60.451 [0.434, 0.468]0.373 [0.346, 0.401]2.785 [2.657, 2.916]
Fixed: Role 2 ×60.447 [0.432, 0.463]0.430 [0.401, 0.459]3.075 [2.936, 3.214]
Fixed: Role 3 ×60.467 [0.453, 0.482]0.456 [0.430, 0.480]3.252 [3.122, 3.374]
Fixed: Role 4 ×60.466 [0.451, 0.481]0.460 [0.435, 0.485]3.286 [3.157, 3.407]
Fixed: Role 5 ×60.470 [0.455, 0.485]0.466 [0.442, 0.489]3.305 [3.184, 3.419]
Fixed: Role 6 ×60.477 [0.462, 0.491]0.452 [0.428, 0.475]3.261 [3.141, 3.373]
Mixed − fixed (mean over roles)+0.005 [+0.004, +0.007]+0.038 [+0.032, +0.044]+0.182 [+0.155, +0.210]
Mixed − Role 1+0.017 [+0.009, +0.025]+0.104 [+0.083, +0.126]+0.558 [+0.462, +0.655]
Mixed − Role 2+0.021 [+0.014, +0.028]+0.047 [+0.029, +0.067]+0.267 [+0.189, +0.347]
Mixed − Role 3+0.001 [-0.006, +0.008]+0.022 [+0.007, +0.037]+0.091 [+0.028, +0.158]
Mixed − Role 4+0.002 [-0.005, +0.010]+0.017 [+0.002, +0.032]+0.057 [-0.015, +0.129]
Mixed − Role 5-0.002 [-0.009, +0.006]+0.011 [-0.008, +0.029]+0.038 [-0.037, +0.113]
Mixed − Role 6-0.008 [-0.016, -0.000]+0.026 [+0.010, +0.042]+0.082 [+0.007, +0.157]

We observe that, on average, mixed-role sets are more diverse than single-role sets: +0.033 SBERT (+7.5% relative) and +0.15 E-Vendi (+4.8%), with significant CIs; Disc. changes little (+0.005). Additionally, the effect depends on the role. Role 1 and Role 2 produce significantly less diverse sets on their own, while the other four come close to the mixed sets. Role conditioning therefore does shape the output distribution. We will frame our claim of role specialization accordingly.

W3 (“Training cost”): We reported our training condition in Appendix D.1. A single MoDA training for 2 epochs runs approximately 24 hours on 4 H200 GPUs. DivPO training takes 22.5 hours on 6 H200 GPUs. Repeated role prompting is an inference-time only method.

Q3: We have replaced Figure 1 with a vector figure with more details. Please check out our updated revision.

Q4: There is a linebreak after “via” and we have removed it in the revision.

We hope these updates address your concerns and are happy to provide anything further during the discussion period. Thank you for your time in engaging in the discussion period with us. If there is any further information you require to consider raising your score, please don’t hesitate to let us know!

Reviewer_geTbOpenReview ↗

Supplementary Questions of Submission47285 by Reviewer geTb

Supplementary Questions:

SQ1: From your new ablation comparing fixed‑mode sampling versus mixed‑mode sampling: Roles 3‑6 achieve diversity close to the full mixed‑role condition, while Role 1 and Role 2 are noticeably weaker. Could the authors unpack what differentiates Role 1 / Role 2 from the rest of the learned modes? Are there observable differences in topic preference, style, or answer length that explain this performance gap?

SQ2: Your human evaluation is still ongoing. Assuming human raters confirm higher idea‑level diversity for MoDA, how do the authors plan to reconcile this with the observed quality drop on NoveltyBench for thinking‑enabled Qwen3‑8B? Would you suggest practical inference‑time knobs (e.g. mode subset selection, diversity‑weight sampling) to mitigate quality degradation for real‑world brainstorming use‑cases?

Reviewer_8DoUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This is a paper on RL post-training methods. Introducing the MARL competitive perspective into alignment: A single shared strategy is conditioned on numbered role tokens. Six "agent intelligences" each provide a response to the same prompt. The reward consists of a prompt-adaptive quality gate (threshold anchoring freezing the reference strategy's smin/s̄/smax), a nearest neighbor embedding distance diversity term, and three types of reward hacking penalties, optimized by GRPO; evaluated on 7 general benchmarks and 4 diversity benchmarks, claiming that the diversity of SBERT increased from 0.274 to 0.482, and the average general ability increased by +10.3 pt.

Critical strengths

1.ole-only (injecting roles without training) demonstrates that the diversity gain mainly comes from training. The diversity-only variant shows a quality collapse under the standard mode, proving that the quality threshold is necessary.

2.The quality-only variant shows that the diversity degrades, proving that the explicit diversity goal is necessary - all three key factors are independently isolated, and the causal chain is relatively complete.

3.The dual-mode reasoning design (reverting to standard behavior when omitting the role) provides practical properties that the standard method does not have during training.

Critical weaknesses

1.The training signal and the evaluation index have the same origin, and there is a suspicion of cyclic verification: The diversity reward is optimized by the nearest neighbor distance of all-MiniLM-L6-v2 embedding, while the main indicator SBERT distance is in the same embedding space as E-Vendi. Qualitative evidence such as "40 kinds of blue "and" six different books "are just the surface entity enumeration most easily captured by embedding distance. The available evidence cannot rule out that the model learns to be "spread out in the embedding space" rather than "semantically truly different".

2.The MARL framework is closer to packaging than mechanism: there is no communication, no joint action, and no shared environment between roles. The so-called competition is only reflected in the normalization of the advantage and the nearest neighbor distance within the group. Key choices underpinning the design also lack justifiability -- role count 6 comes from memory constraints rather than diversity saturation analysis, role number-diversity payoff curves are missing, and no seed-level variance is reported for emergent role specializations.

3.The attribution of generic capability improvement is not isolated: The average pass@1 increased by 10.3 percentage points, and TruthfulQA increased from 22.2 to 57.9, which was abnormal. The paper lacked "pure GRPO with the same data and no diversity target" control, so it could not be excluded that the gain came from the instruction distribution of Tulu3 mixed data rather than the method itself. At the same time, the decline in quality in PreScience is downplayed in the discourse, and the claim of quality-diversity co-promotion does not hold in all fields.

Questions for authors

1.Could you re-measure the diversity gains in a measurement space that shares nothing with the training reward — e.g., diversity computed with a different embedding family, plus an LLM-judged or human-rated conceptual diversity score on a subset of prompts? If the SBERT/E-Vendi gains largely vanish under an orthogonal measurement space, the claim of semantic diversity improvement is falsified — the model would have learned to spread out in the reward embedder's space, not to produce genuinely different ideas.

2.Could you train a control that keeps your full reward (quality gate + nearest-neighbor diversity term) but removes role conditioning, sampling the six group responses i.i.d. without role tokens? If diversity improves equally without roles, the MARL/mode-conditioning framing is falsified as a mechanism — the gains would come from the group-relative diversity reward alone, and the multi-agent narrative should be dropped. (A role-count sweep over N ∈ {2, 4, 6, 12} would additionally show whether 6 is a saturation point or merely a GPU constraint.)

3.Could you train pure GRPO on the same 10K-prompt Tulu3/Infinite-Chat mixture with the quality reward only (no diversity term, no role conditioning) and report the seven general-capability benchmarks? If this control reproduces most of the +10.3 pt average gain (especially TruthfulQA 22.2→57.9), the capability-improvement claim is falsified — the gains would be attributable to the data mixture rather than to MODA, and the paper's contribution would narrow to diversity alone.

7 comments
AuthorsOpenReview ↗

Reply to Reviewer 8DoU

We thank the reviewer for the careful reading. We appreciate your recognition for our exhaustive ablation studies to isolate each component of the reward, and the practical value of our dual-mode design. All requested details are provided below and we will incorporate them into the revision based on your feedback.

W1 and Q1 (“ The training signal and the evaluation index have the same origin”): We appreciate your observations. We re-measured diversity with two embedders from different model families than the training reward (all-MiniLM-L6-v2): BAAI/bge-large-en-v1.5 and Qwen3-Embedding-0.6B. MoDA is still leading the board across models and configurations. Thinking = True, Base model = Qwen3-8B

ModelSBERT · BGE-largeSBERT · Qwen3-EmbEVendi · BGE-largeEVendi · Qwen3-Emb
Qwen3-8B w/ thinking0.080 ± 0.0000.123 ± 0.0011.49 ± 0.001.76 ± 0.01
SSoT w/ thinking0.111 ± 0.0010.178 ± 0.0011.62 ± 0.001.99 ± 0.01
MoDA w/ thinking0.192 ± 0.0050.269 ± 0.0062.15 ± 0.032.63 ± 0.03
DivPO w/ thinking0.107 ± 0.0020.167 ± 0.0031.64 ± 0.011.99 ± 0.01
Persona Sampling w/ thinking0.125 ± 0.0010.215 ± 0.0021.83 ± 0.002.53 ± 0.01

Thinking = False, Base model = Qwen3-8B

ModelSBERT · BGE-largeSBERT · Qwen3-EmbEVendi · BGE-largeEVendi · Qwen3-Emb
Qwen3-8B0.075 ± 0.0010.114 ± 0.0021.45 ± 0.001.69 ± 0.01
MoDA0.176 ± 0.0010.267 ± 0.0032.09 ± 0.002.70 ± 0.02
DARLING (Mix)0.141 ± 0.0030.244 ± 0.0051.89 ± 0.022.64 ± 0.04
DARLING (WildChat)0.098 ± 0.0010.162 ± 0.0021.60 ± 0.002.05 ± 0.01
DivPO0.131 ± 0.0010.208 ± 0.0011.81 ± 0.002.32 ± 0.01

Thinking = False, Base model = Llama-3.1-8B

ModelSBERT · BGE-largeSBERT · Qwen3-EmbEVendi · BGE-largeEVendi · Qwen3-Emb
Llama-MoDA0.192 ± 0.0020.313 ± 0.0032.17 ± 0.013.04 ± 0.02
Llama-3.1-8B0.145 ± 0.0020.232 ± 0.0031.83 ± 0.012.37 ± 0.03

Thinking = False, Base model = GLM-4-9B

ModelSBERT · BGE-largeSBERT · Qwen3-EmbEVendi · BGE-largeEVendi · Qwen3-Emb
GLM-MoDA0.182 ± 0.0020.296 ± 0.0022.18 ± 0.013.09 ± 0.02
GLM-4-9B0.148 ± 0.0010.285 ± 0.0021.96 ± 0.013.03 ± 0.02

MoDA is still leading the board across models and configurations. This result is strong evidence that MoDA promotes generative diversity across model families.

We also want to clarify that not all the evaluation index has the same index and has the same origin as the training signal. The training uses all-MiniLM-L6-v2 embedding, but our discriminator uses all-mpnet-base-v2 embedder to construct its feature vector (Appx D.4), and therefore serves as an additional measure for diversity. Additionally, NoveltyBench has its naive distinct_k metrics, which calls LLM-as-Judge to judge the number of distinct answers. The result below shows that MoDA improves distinct_k value in NoveltyBench across model families, beating DivPO and DARLING(WildChat) in both thinking enabled and disabled settings. All values are mean ± std over n=3.

Thinking = True, Base model = Qwen3-8B

Modeldistinct_k (official)
Qwen3-8B w/ thinking4.63 ± 0.21
SSoT w/ thinking2.73 ± 0.06
MoDA w/ thinking6.10 ± 0.08
DivPO w/ thinking5.64 ± 0.08

Thinking = False, Base model = Qwen3-8B

Modeldistinct_k (official)
Qwen3-8B4.33 ± 0.09
MoDA7.45 ± 0.06
DARLING (WildChat)6.48 ± 0.03
DivPO6.85 ± 0.06

Thinking = False, Base model = Llama-3.1-8B

Modeldistinct_k (official)
Llama-MoDA7.50 ± 0.07
Llama-3.1-8B5.96 ± 0.12

Thinking = False, Base model = GLM-4-9B

Modeldistinct_k (official)
GLM-MoDA9.80 ± 0.02
GLM-4-9B7.69 ± 0.14

Together, the diversity increase across different families of metrics indicates that MoDA produces genuinely different responses rather than just spreading out in the reward embedder’s space.

AuthorsOpenReview ↗

Reply to Reviewer 8DoU (cont.)

Q2 (“Additional control experiment”): Thank you for suggesting these experiments. As we mentioned in our overall reply, we agree that additional control experiments without role conditioning can isolate the effect of role conditioning.

We trained a control model with the same quality-gating diversity reward function without role conditioning for 2 epochs, and we compared it with MoDA trained for 2 epochs.

General capability tasks

Modelgsm8kmmlugpqaboolqhellaswagtruthfulqaIFEvalavg
Qwen3-8B (base)89.285.042.987.382.572.176.5276.5
No-role92.582.049.085.078.973.875.7976.7
MODA (2 ep)92.683.045.083.777.971.871.7275.1

Paired per-prompt differences in diversity tasks (95% bootstrap CI over 100 prompts; Self-BLEU: negative = more diverse). IC = Infinite-Chat. NB = NoveltyBench.

DomainMetricNo-role − BaseMODA - Base
ICDisc.+0.022 [+0.015, +0.029]+0.044 [+0.035, +0.052]
ICSBERT+0.066 [+0.054, +0.079]+0.125 [+0.111, +0.140]
ICE-Vendi+0.46 [+0.38, +0.54]+0.86 [+0.76, +0.97]
NBDisc.+0.027 [+0.015, +0.040]+0.044 [+0.033, +0.055]
NBSBERT+0.074 [+0.051, +0.099]+0.105 [+0.084, +0.127]
NBE-Vendi+0.68 [+0.53, +0.85]+0.87 [+0.73, +1.02]

The result shows that the no-role model retains the general reasoning capability (+0.2%) as we expected, but doesn’t achieve the same level of diversity as the MoDA model. On Infinite-Chat, the no-role model’s diversity gain over base is about half of MoDA’s gain. On NoveltyBench, the no-role model’s diversity gain over base is about 60-70% of MoDA’s gain. Therefore, this result suggests that our quality-gating reward function itself is a meaningful contribution in retaining the reasoning general capability while promoting generative diversity, while adding mode-conditioning further incentivizes exploration. We will add these results into our ablation studies in the main text.

Per your suggestion, we also ran the control experiment without role conditioning and with quality reward only, and we have the evaluation results below.

Modelgsm8kmmlugpqaboolqhellaswagtruthfulqaIFEvalAvg. Pass@1
Qwen3-8B (base)89.285.042.987.382.572.176.5276.5
Quality-only, no roles90.585.041.987.680.673.175.2376.3
MODA (2 ep)92.683.045.083.777.971.871.7275.1

As the results show, training with quality-only reward retains the base model’s general capability. Since MoDA’s main contribution is to promote diversity while retaining quality, this ablation shows that capability gains over the base model are likely driven by RL post-training, and our main contribution is achieving greater diversity while preserving these gains.

W2 (“The MARL framework is closer to packaging than mechanism): We have a single-role ablation experiment in Section 5.2 and Appendix E.3. We show that the single-role trained model (Discriminator: 0.428, SBERT: 0.239, E-Vendi: 2.57) falls behind in the diversity metrics compared to MoDA with six-role (Discriminator: 0.444, SBERT: 0.257, E-Vendi: 2.69). We will add an additional control experiment with 3 roles to further support the scaling law of roles, which would take 24 hours to finish. We will post the results once the training is complete.

AuthorsOpenReview ↗

Reply to Reviewer 8DoU (cont.)

W3 (“The attribution of generic capability improvement is not isolated”): We reviewed the generated traces and evaluation script, and made two updates:

  • Answer parsing. The base models often put their final answer in bold (e.g., (i.e. (A) …)), which our original parser failed to parse. We fixed the parser to handle this and re-evaluated all models.

  • Llama-3.1-8B configuration. We examined the MMLU and GPQA traces by Llama-3.1-8B and found that our evaluation settings didn’t match the model’s default decoding setting. We first added the instruction “Answer with the option letter” to the MMLU and GPQA prompts to ensure its answers follow the multiple-choice format our parser expects. We also rerun the llama model evaluation with the model’s default decoding configuration temperature = 0.6 and top_p = 0.9 instead of our own decoding setting (temperature = 1.0 and top_p = 1.0). Here is the evaluation result with updated configurations.

Qwen3-8B, thinking off

ModelGSM8KMMLUGPQABoolQHellaSwagTruthfulQAIFEvalAvg.Δ Avg. vs base
Qwen3-8B89.2 ± 0.985.0 ± 3.642.9 ± 3.587.3 ± 0.682.5 ± 0.472.1 ± 1.678.4 ± 1.876.8 ± 0.8—
DARLING (Mixture)70.3 ± 1.366.0 ± 4.739.4 ± 3.583.4 ± 0.776.1 ± 0.469.6 ± 1.653.0 ± 2.165.4 ± 0.9-11.3
DARLING (WildChat)90.0 ± 0.886.0 ± 3.537.9 ± 3.486.9 ± 0.677.9 ± 0.469.6 ± 1.673.0 ± 1.974.5 ± 0.8-2.3
DivPO89.3 ± 0.985.0 ± 3.640.4 ± 3.584.5 ± 0.678.5 ± 0.469.9 ± 1.677.3 ± 1.875.0 ± 0.8-1.8
MODA91.9 ± 0.886.0 ± 3.552.5 ± 3.586.3 ± 0.678.7 ± 0.473.1 ± 1.676.9 ± 1.877.9 ± 0.8+1.2

Qwen3-8B, thinking on

ModelGSM8KMMLUGPQABoolQHellaSwagTruthfulQAIFEvalAvg.Δ Avg. vs base
Qwen3-8B95.0 ± 0.694.0 ± 2.444.4 ± 3.587.9 ± 0.682.4 ± 0.475.4 ± 1.579.7 ± 1.779.8 ± 0.7—
SSoT88.6 ± 0.993.0 ± 2.657.1 ± 3.587.2 ± 0.680.1 ± 0.472.7 ± 1.661.9 ± 2.177.2 ± 0.7-2.6
DivPO94.2 ± 0.692.0 ± 2.750.5 ± 3.686.2 ± 0.680.1 ± 0.472.5 ± 1.677.6 ± 1.879.0 ± 0.7-0.8
MODA95.4 ± 0.693.0 ± 2.653.5 ± 3.587.6 ± 0.682.1 ± 0.470.6 ± 1.679.5 ± 1.780.3 ± 0.7+0.4

With these updates, our results are consistent across all model families. MODA retains the base model’s capabilities on Qwen3-8B (+1.2%) and improves on Llama-3.1-8B (+6.8%) and GLM-4-9B (+22.2%), while DARLING and DivPO both cause some regression (-11.3% and -1.8%).

ModelMetricGSM8KMMLUGPQABoolQHellaSwagTruthfulQAIFEvalAvg.Δ Avg. vs base
Llama-3.1-8Bpass@1 ± SE77.3 ± 1.251.0 ± 5.023.2 ± 3.083.8 ± 0.668.1 ± 0.557.2 ± 1.773.2 ± 1.962.0 ± 0.9 (54.3)—
MODA (Llama-3.1-8B)pass@1 ± SE82.6 ± 1.049.0 ± 5.033.3 ± 3.481.5 ± 0.770.3 ± 0.567.3 ± 1.673.8 ± 1.965.4 ± 1.0 (60.9)+3.4

On the llama model, MoDA improves general capability pass@1 by 3.4% over the Llama-3.1-8B baseline. So the paper’s main point holds that MoDA adds diversity without compromising general capability.

the decline in quality in PreScience is downplayed: We have discussed this in Section 5 Generative Diversity Evaluation, and we can make this more clear in the paper. but note that this is just one benchmark, and we have other improvements as shown by other benchmarks.

We hope these updates address your concerns and are happy to provide anything further during the discussion period. Thank you for your time in engaging in the discussion period with us. Please don’t hesitate to let us know if there is any further information you require to consider raising your score!

Reviewer_pqpeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • An online GRPO method for quality-diverse generation. A single policy is conditioned on abstract role prompts (“You are role i”).
  • The reward combines a quality score and a diversity term (each response’s embedding distance to its nearest neighbor in the group), plus penalties for length, language mixing, and formatting.
  • At inference, adding role prompts gives a diverse candidate set; omitting them recovers single-response behavior.
Critical strengths
  • Simple modifications to DARLING (prefixes) to make sampling of diverse generations more "controllable"
  • Broad evaluation that covers three model families.
Critical weaknesses
  • Several baseline scores fall below chance on multiple-choice benchmarks? Why does Llama-3.1-8B gets GPQA 6.1% and MMLU 16%? Similarly Qwen3-8B gets TruthfulQA-MC1 22.2% and HellaSwag 48.4%. So I am concerned about the soundness of the evals (answer-extraction or formatting failures).

  • I would have appreciated if the ablations focused on prefix-conditioning mechanism that is the main sales pitch. Like everything kept the same, w/ vs w/o prefix-conditioning. Ultimately that's the main addition over DARLING control to establish the effect.

  • Minor: Fig1 is clearly LLM generated. I'd try porting it in a format that is better quality.

Questions for authors

None

6 comments
AuthorsOpenReview ↗

Reply to Reviewer pqpe

We thank the reviewer for a careful and engaged reading, and for highlighting the controllability of MoDA and the breadth of our evaluation. We believe that addressing the weaknesses you pointed out is especially helpful in improving the manuscript.

W2 (Mode-conditioning mechanism ablation): We thank the reviewers for prompting this additional experiment. We trained a control model with the same quality-gating diversity reward function without role conditioning for 2 epochs, and we compared it with MoDA trained for 2 epochs.

General capability tasks

Modelgsm8kmmlugpqaboolqhellaswagtruthfulqaIFEvalavg
Qwen3-8B (base)89.285.042.987.382.572.176.5276.5
No-role92.582.049.085.078.973.875.7976.7
MODA (2 ep)92.683.045.083.777.971.871.7275.1

Paired per-prompt differences in diversity tasks (95% bootstrap CI over 100 prompts; Self-BLEU: negative = more diverse). IC = Infinite-Chat. NB = NoveltyBench.

DomainMetricNo-role − BaseMODA - Base
ICDisc.+0.022 [+0.015, +0.029]+0.044 [+0.035, +0.052]
ICSBERT+0.066 [+0.054, +0.079]+0.125 [+0.111, +0.140]
ICE-Vendi+0.46 [+0.38, +0.54]+0.86 [+0.76, +0.97]
NBDisc.+0.027 [+0.015, +0.040]+0.044 [+0.033, +0.055]
NBSBERT+0.074 [+0.051, +0.099]+0.105 [+0.084, +0.127]
NBE-Vendi+0.68 [+0.53, +0.85]+0.87 [+0.73, +1.02]

The result shows that the no-role model retains the general reasoning capability (+0.2%) as we expected, but doesn’t achieve the same level of diversity as the MoDA model. On Infinite-Chat, the no-role model’s diversity gain over base is about half of MoDA’s gain. On NoveltyBench, the no-role model’s diversity gain over base is about 60-70% of MoDA’s gain. Therefore, this result suggests that our quality-gating reward function itself is a meaningful contribution in retaining the reasoning general capability while promoting generative diversity, while adding mode-conditioning further incentivizes exploration. We will add these results into our ablation studies in the main text.

W3 (Formatting): We have formatted the figure 1 to make it more presentable and informative in our updated revision. Please check out the revised version.

AuthorsOpenReview ↗

Reply to Reviewer pqpe (cont.)

W1 (multiple-choice benchmarks): Thanks for flagging the multiple-choice scores. Thank you for your great suggestions around looking into the general capabilities scores. We want to clarify that our main claim is capability retention rather than capability improvement. Our contribution is achieving greater diversity while preserving the general reasoning capability of base model.

We reviewed the generated traces and evaluation script, and made two updates:

  • Answer parsing. The base models often put their final answer in bold (e.g., (i.e. (A) …)), which our original parser failed to parse. We fixed the parser to handle this and re-evaluated all models.

  • Llama-3.1-8B configuration. We examined the MMLU and GPQA traces by Llama-3.1-8B and found that our evaluation settings didn’t match the model’s default decoding setting. We first added the instruction “Answer with the option letter” to the MMLU and GPQA prompts to ensure its answers follow the multiple-choice format our parser expects. We also rerun the llama model evaluation with the model’s default decoding configuration temperature = 0.6 and top_p = 0.9 instead of our own decoding setting (temperature = 1.0 and top_p = 1.0). Here is the evaluation result with updated configurations.

Qwen3-8B, thinking off

ModelGSM8KMMLUGPQABoolQHellaSwagTruthfulQAIFEvalAvg.Δ Avg. vs base
Qwen3-8B89.2 ± 0.985.0 ± 3.642.9 ± 3.587.3 ± 0.682.5 ± 0.472.1 ± 1.678.4 ± 1.876.8 ± 0.8—
DARLING (Mixture)70.3 ± 1.366.0 ± 4.739.4 ± 3.583.4 ± 0.776.1 ± 0.469.6 ± 1.653.0 ± 2.165.4 ± 0.9-11.3
DARLING (WildChat)90.0 ± 0.886.0 ± 3.537.9 ± 3.486.9 ± 0.677.9 ± 0.469.6 ± 1.673.0 ± 1.974.5 ± 0.8-2.3
DivPO89.3 ± 0.985.0 ± 3.640.4 ± 3.584.5 ± 0.678.5 ± 0.469.9 ± 1.677.3 ± 1.875.0 ± 0.8-1.8
MODA91.9 ± 0.886.0 ± 3.552.5 ± 3.586.3 ± 0.678.7 ± 0.473.1 ± 1.676.9 ± 1.877.9 ± 0.8+1.2

Qwen3-8B, thinking on

ModelGSM8KMMLUGPQABoolQHellaSwagTruthfulQAIFEvalAvg.Δ Avg. vs base
Qwen3-8B95.0 ± 0.694.0 ± 2.444.4 ± 3.587.9 ± 0.682.4 ± 0.475.4 ± 1.579.7 ± 1.779.8 ± 0.7—
SSoT88.6 ± 0.993.0 ± 2.657.1 ± 3.587.2 ± 0.680.1 ± 0.472.7 ± 1.661.9 ± 2.177.2 ± 0.7-2.6
DivPO94.2 ± 0.692.0 ± 2.750.5 ± 3.686.2 ± 0.680.1 ± 0.472.5 ± 1.677.6 ± 1.879.0 ± 0.7-0.8
MODA95.4 ± 0.693.0 ± 2.653.5 ± 3.587.6 ± 0.682.1 ± 0.470.6 ± 1.679.5 ± 1.780.3 ± 0.7+0.4

With these updates, our results are consistent across all model families. MODA retains the base model’s capabilities on Qwen3-8B (+1.2%) and improves on Llama-3.1-8B (+6.8%) and GLM-4-9B (+22.2%), while DARLING and DivPO both cause some regression (-11.3% and -1.8%).

ModelMetricGSM8KMMLUGPQABoolQHellaSwagTruthfulQAIFEvalAvg.Δ Avg. vs base
Llama-3.1-8Bpass@1 ± SE77.3 ± 1.251.0 ± 5.023.2 ± 3.083.8 ± 0.668.1 ± 0.557.2 ± 1.773.2 ± 1.962.0 ± 0.9 (54.3)—
MODA (Llama-3.1-8B)pass@1 ± SE82.6 ± 1.049.0 ± 5.033.3 ± 3.481.5 ± 0.770.3 ± 0.567.3 ± 1.673.8 ± 1.965.4 ± 1.0 (60.9)+3.4

On the llama model, MoDA improves general capability pass@1 by 3.4% over the Llama-3.1-8B baseline. So the paper’s main point holds that MoDA adds diversity without compromising general capability.

We hope these updates address your concerns and are happy to provide anything further during the discussion period. Thank you for your time in engaging in the discussion period with us. If there is any further information you require to consider raising your score, please don’t hesitate to let us know!

Dear Reviewer pqpe,

Thank you again for your thoughtful feedback. As the rebuttal period draws to a close, we wanted to follow up on our response and highlight the additional evidence relevant to your main concerns:

  • Multiple-choice benchmarks: We made two updates to improve the parser and decoding configurations for our general capability evaluations. With these updates, the ranking among models are largely unchanged across all model families. Our main conclusions still holds. MODA retains the base model’s capabilities on Qwen3-8B (+1.2%) and improves on Llama-3.1-8B (+6.8%) and GLM-4-9B (+22.2%), while DARLING and DivPO both cause some regression (-11.3% and -1.8%).

  • Mode-conditioning ablation: We trained a control model with the same quality-gating diversity reward function without role conditioning for 2 epochs, and we compared it with MoDA trained for 2 epochs. The result shows that the no-role model retains the general reasoning capability (+0.2%) as we expected, but doesn’t achieve the same level of diversity as the MoDA model. On Infinite-Chat, the no-role model’s diversity gain over base is about half of MoDA’s gain. On NoveltyBench, the no-role model’s diversity gain over base is about 60-70% of MoDA’s gain.

  • Formatting: We have formatted the figure 1 to make it more presentable and informative in our updated revision.

If any concern remains unresolved, we would appreciate the opportunity to address it before the discussion ends. If these results and clarifications address your main concerns, we would be grateful if you could consider raising your score in light of the additional evidence.

Comments on the paper

5 comments
AuthorsOpenReview ↗

General reply to all reviewers

We thank all reviewers for their time. Your constructive suggestions allow us to make significant improvements to our work. We have addressed all the points raised, and we're happy to answer any follow up questions. We would like to discuss some common issues raised by the reviewers below.

Diversity evaluation: We appreciate the observations by reviewer 8DoU and geTb that our training signal and some of the diversity evaluation share the same embedding model. We re-measured diversity with two embedders from different model families than the training reward (all-MiniLM-L6-v2): BAAI/bge-large-en-v1.5 and Qwen3-Embedding-0.6B.

Thinking = True, Base model = Qwen3-8B

ModelSBERT · BGE-largeSBERT · Qwen3-EmbEVendi · BGE-largeEVendi · Qwen3-Emb
Qwen3-8B w/ thinking0.080 ± 0.0000.123 ± 0.0011.49 ± 0.001.76 ± 0.01
SSoT w/ thinking0.111 ± 0.0010.178 ± 0.0011.62 ± 0.001.99 ± 0.01
MoDA w/ thinking0.192 ± 0.0050.269 ± 0.0062.15 ± 0.032.63 ± 0.03
DivPO w/ thinking0.107 ± 0.0020.167 ± 0.0031.64 ± 0.011.99 ± 0.01
Persona Sampling w/ thinking0.125 ± 0.0010.215 ± 0.0021.83 ± 0.002.53 ± 0.01

Thinking = False, Base model = Qwen3-8B

ModelSBERT · BGE-largeSBERT · Qwen3-EmbEVendi · BGE-largeEVendi · Qwen3-Emb
Qwen3-8B0.075 ± 0.0010.114 ± 0.0021.45 ± 0.001.69 ± 0.01
MoDA0.176 ± 0.0010.267 ± 0.0032.09 ± 0.002.70 ± 0.02
DARLING (Mix)0.141 ± 0.0030.244 ± 0.0051.89 ± 0.022.64 ± 0.04
DARLING (WildChat)0.098 ± 0.0010.162 ± 0.0021.60 ± 0.002.05 ± 0.01
DivPO0.131 ± 0.0010.208 ± 0.0011.81 ± 0.002.32 ± 0.01

Thinking = False, Base model = Llama-3.1-8B

ModelSBERT · BGE-largeSBERT · Qwen3-EmbEVendi · BGE-largeEVendi · Qwen3-Emb
Llama-MoDA0.192 ± 0.0020.313 ± 0.0032.17 ± 0.013.04 ± 0.02
Llama-3.1-8B0.145 ± 0.0020.232 ± 0.0031.83 ± 0.012.37 ± 0.03

Thinking = False, Base model = GLM-4-9B

ModelSBERT · BGE-largeSBERT · Qwen3-EmbEVendi · BGE-largeEVendi · Qwen3-Emb
GLM-MoDA0.182 ± 0.0020.296 ± 0.0022.18 ± 0.013.09 ± 0.02
GLM-4-9B0.148 ± 0.0010.285 ± 0.0021.96 ± 0.013.03 ± 0.02

MoDA is still leading the board across models and configurations. This result is strong evidence that MoDA promotes generative diversity across model families.

We also want to clarify that not all the evaluation index has the same index and has the same origin as the training signal. The training uses all-MiniLM-L6-v2 embedding, but our discriminator uses all-mpnet-base-v2 embedder to construct its feature vector (Appx D.4), and therefore serves as an additional measure for diversity. Additionally, NoveltyBench has its naive distinct_k metrics, which calls LLM-as-Judge to judge the number of distinct answers. The result below shows that MoDA improves distinct_k value in NoveltyBench across model families, beating DivPO and DARLING(WildChat) in both thinking enabled and disabled settings. All values are mean ± std over n=3.

Thinking = True, Base model = Qwen3-8B

Modeldistinct_k (official)
Qwen3-8B w/ thinking4.63 ± 0.21
SSoT w/ thinking2.73 ± 0.06
MoDA w/ thinking6.10 ± 0.08
DivPO w/ thinking5.64 ± 0.08

Thinking = False, Base model = Qwen3-8B

Modeldistinct_k (official)
Qwen3-8B4.33 ± 0.09
MoDA7.45 ± 0.06
DARLING (WildChat)6.48 ± 0.03
DivPO6.85 ± 0.06

Thinking = False, Base model = Llama-3.1-8B

Modeldistinct_k (official)
Llama-MoDA7.50 ± 0.07
Llama-3.1-8B5.96 ± 0.12

Thinking = False, Base model = GLM-4-9B

Modeldistinct_k (official)
GLM-MoDA9.80 ± 0.02
GLM-4-9B7.69 ± 0.14

Together, the diversity increase across different families of metrics indicates that MoDA produces genuinely different responses rather than just spreading out in the reward embedder’s space.

AuthorsOpenReview ↗

General reply to all reviewers (cont.)

General capability evaluation: Thank you for your great suggestions around looking into the general capabilities scores. We want to clarify that our main claim is capability retention rather than capability improvement. Our contribution is achieving greater diversity while preserving the general reasoning capability of base model.

We reviewed the traces for all the general capability benchmarks and we confirmed that the low scores partially come from a parsing issue. Base model produces answer in bold (i.e. (A) …), which the old parser couldn’t read.

Qwen3-8B, thinking off

ModelGSM8KMMLUGPQABoolQHellaSwagTruthfulQAIFEvalAvg.Δ Avg. vs base
Qwen3-8B89.2 ± 0.985.0 ± 3.642.9 ± 3.587.3 ± 0.682.5 ± 0.472.1 ± 1.678.4 ± 1.876.8 ± 0.8—
DARLING (Mixture)70.3 ± 1.366.0 ± 4.739.4 ± 3.583.4 ± 0.776.1 ± 0.469.6 ± 1.653.0 ± 2.165.4 ± 0.9-11.3
DARLING (WildChat)90.0 ± 0.886.0 ± 3.537.9 ± 3.486.9 ± 0.677.9 ± 0.469.6 ± 1.673.0 ± 1.974.5 ± 0.8-2.3
DivPO89.3 ± 0.985.0 ± 3.640.4 ± 3.584.5 ± 0.678.5 ± 0.469.9 ± 1.677.3 ± 1.875.0 ± 0.8-1.8
MODA91.9 ± 0.886.0 ± 3.552.5 ± 3.586.3 ± 0.678.7 ± 0.473.1 ± 1.676.9 ± 1.877.9 ± 0.8+1.2

Qwen3-8B, thinking on

ModelGSM8KMMLUGPQABoolQHellaSwagTruthfulQAIFEvalAvg.Δ Avg. vs base
Qwen3-8B95.0 ± 0.694.0 ± 2.444.4 ± 3.587.9 ± 0.682.4 ± 0.475.4 ± 1.579.7 ± 1.779.8 ± 0.7—
SSoT88.6 ± 0.993.0 ± 2.657.1 ± 3.587.2 ± 0.680.1 ± 0.472.7 ± 1.661.9 ± 2.177.2 ± 0.7-2.6
DivPO94.2 ± 0.692.0 ± 2.750.5 ± 3.686.2 ± 0.680.1 ± 0.472.5 ± 1.677.6 ± 1.879.0 ± 0.7-0.8
MODA95.4 ± 0.693.0 ± 2.653.5 ± 3.587.6 ± 0.682.1 ± 0.470.6 ± 1.679.5 ± 1.780.3 ± 0.7+0.4
  • Llama model evaluation configuration: We thank the reviewers for their insights on the evaluation results of the Llama model. We closely examined the Llama-3.1-8B generation traces on MMLU and GPQA and made two refinements to align our configuration with the model’s default decoding configuration.
    • Answer format. We added the instruction “Answer with the option letter” to the MMLU and GPQA prompts. This makes sure its answers follow the multiple-choice format our parser expects.
    • Decoding settings. We now use Llama’s default decoding configuration (temperature = 0.6, top_p = 0.9) in place of our default decoding setting (temperature = 1.0, top_p=1.0). Here is the evaluation result with updated configurations. Avg. in parentheses is the temperature-1.0 value.
ModelMetricGSM8KMMLUGPQABoolQHellaSwagTruthfulQAIFEvalAvg.Δ Avg. vs base
Llama-3.1-8Bpass@1 ± SE77.3 ± 1.251.0 ± 5.023.2 ± 3.083.8 ± 0.668.1 ± 0.557.2 ± 1.773.2 ± 1.962.0 ± 0.9 (54.3)—
MODA (Llama-3.1-8B)pass@1 ± SE82.6 ± 1.049.0 ± 5.033.3 ± 3.481.5 ± 0.770.3 ± 0.567.3 ± 1.673.8 ± 1.965.4 ± 1.0 (60.9)+3.4

With the updated configuration, MoDA improves general capability pass@1 by 3.4% over the Llama-3.1-8B baseline. The paper’s main conclusions are therefore unchanged. We will carefully include these updated results and the full evaluation configuration in the camera-ready version.

AuthorsOpenReview ↗

General reply to all reviewers (cont.)

Mode-conditioning mechanism ablation: We thank the reviewers for prompting this additional experiment. We trained a control model with the same quality-gating diversity reward function without role conditioning for 2 epochs, and we compared it with MoDA trained for 2 epochs.

General capability tasks

Modelgsm8kmmlugpqaboolqhellaswagtruthfulqaIFEvalavg
Qwen3-8B (base)89.285.042.987.382.572.176.5276.5
No-role92.582.049.085.078.973.875.7976.7
MODA (2 ep)92.683.045.083.777.971.871.7275.1

Paired per-prompt differences in diversity tasks (95% bootstrap CI over 100 prompts; Self-BLEU: negative = more diverse). IC = Infinite-Chat. NB = NoveltyBench.

DomainMetricNo-role − BaseMODA - Base
ICDisc.+0.022 [+0.015, +0.029]+0.044 [+0.035, +0.052]
ICSBERT+0.066 [+0.054, +0.079]+0.125 [+0.111, +0.140]
ICE-Vendi+0.46 [+0.38, +0.54]+0.86 [+0.76, +0.97]
NBDisc.+0.027 [+0.015, +0.040]+0.044 [+0.033, +0.055]
NBSBERT+0.074 [+0.051, +0.099]+0.105 [+0.084, +0.127]
NBE-Vendi+0.68 [+0.53, +0.85]+0.87 [+0.73, +1.02]

The result shows that the no-role model retains the general reasoning capability (+0.2%) as we expected, but doesn’t achieve the same level of diversity as the MoDA model. On Infinite-Chat, the no-role model’s diversity gain over base is about half of MoDA’s gain. On NoveltyBench, the no-role model’s diversity gain over base is about 60-70% of MoDA’s gain. Therefore, this result suggests that our quality-gating reward function itself is a meaningful contribution in retaining the reasoning general capability while promoting generative diversity, while adding mode-conditioning further incentivizes exploration. We will add these results into our ablation studies in the main text.

Human evaluation: We agree that human evaluation is the gold standard to judge diversity. Adding this evaluation would greatly improve the soundness of our diversity claim. We asked 10 participants to compare MoDA with the strongest baseline, DARLING, on 26 prompts from NoveltyBench and PreScience. The participants are asked to read three answers from each model without knowing which system wrote which. For every prompt they will judge which set had more genuinely different ideas, which was more original, and which was better overall. Each prompt is rated by at least three people. We estimate that human evaluation would take about a week to complete, and we will include the results in the camera-ready version.

Formatting: We have formatted the figure 1 to make it more presentable and informative in our updated revision. Please check out the revised version.

We appreciate your feedback and please check our manuscripts for updated Figure 1 and Table 1, as well as additional ablation study results. Thank you for your time in engaging in the discussion period with us.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.