acceptodds
Under review as a conference paper at ICLR 2027

CADI: Restoring Output Diversity in Large Language Models via Calibration and Self-Distillation

Abstract

Large language models (LLMs) often concentrate their outputs on a small set of canonical responses, causing repeated sampling to explore only a narrow subset of valid outputs. Inference-time methods can improve diversity, but their gains depend on persistent prompting or decoding strategies rather than being internalized in model parameters, while training-based methods alter the model itself but typically require carefully designed diversity-specific rewards. In this paper, we propose CADI, a two-stage framework that restores access to valid alternatives suppressed by concentrated model preferences. First, diversity calibration uses simple candidate-selection tasks to reduce overly concentrated model preferences while preserving the model's original preference structure. Second, online guided self-distillation contrasts plain and guidance-conditioned response sets generated by the current model, and distills the guided behavior only when it achieves a Pareto improvement over unguided generation. Experiments on NoveltyBench and Artificial Hivemind show that CADI consistently outperforms strong training-based baselines in semantic diversity and utility across two LLM backbones, while largely preserving response quality and general capabilities. Our code is available at https://anonymous.4open.science/r/CADI-D4B3.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Output Diversity

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_QdCSOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses limited semantic diversity when sampling responses to open-ended prompts. CADI first calibrates candidate preferences, then distills guided responses selected for diversity, utility, and quality into a model that needs no guidance at inference. An idealized coverage analysis motivates the approach; experiments on two open-ended benchmarks and appendix studies report positive results.

Critical strengths
  1. The simple, base-anchored calibration objective is well motivated. It reduces concentration on held-out candidate sets, suggesting the effect extends beyond the exact training choices (p. 27, Table 5).

  2. The evaluation considers semantic coverage, utility, and quality across two backbones, with additional larger-model and reasoning-trajectory checks (p. 7, Table 1; Appendix C). This gives the paper a meaningful empirical signal.

Critical weaknesses
  1. The gain in useful diversity at comparable quality is not yet secure. Some advantages over strong baselines are small, one backbone shows a quality trade-off, and the main results lack paired uncertainty and independent scorer checks (page 7, Table 1; pages 18–19). These checks would clarify how much of the claimed benefit is robust.

  2. Stage 2 adds a modest improvement over Stage 1 while requiring external guidance and substantial extra generation and training. Without total resource costs and a matched-budget comparison, its practical value is hard to judge.

  3. The proposed mechanism and generalization claim are broader than the direct evidence: the proof uses an idealized semantic distribution, and the main open-ended evaluations cover a limited set of prompts (pages 3–6). Appendix results help, but a category-level analysis and quality-matched decoding comparison would better define where the method works.

Questions for authors
  1. Which gains over the strongest baselines remain under paired uncertainty estimates and an independent semantic/quality check of the existing outputs (page 7, Table 1)?

  2. Which models and definitions score diversity, utility, and quality during Stage 2, and do these evaluators overlap with those used at test time (pages 5–6, 18–19)?

  3. What are Stage 2's total generation, compute, and API costs, and does its gain remain worthwhile against a matched-budget baseline (pages 7, 18, 28)?

  4. How do gains vary by prompt category, and do they persist against a temperature/decoding sweep at matched quality?

  5. While this does not constitute a specific weakness or a point for questioning, I believe the Llama3.1/Qwen2.5 used in the paper are already outdated. Besides, benchmarks such as GSM8K, MMLU and HumanEval seems to be easy for modern LLMs. I would like to see more recent models and challenged benchmarks, though this is not a requirement.

5 comments

We sincerely thank you for the thoughtful evaluation and constructive feedback. We greatly appreciate your valuable suggestions and provide detailed responses to each of your concerns below.


Q1. Statistical Reliability and Independent Validation

Thank you for this valuable suggestion. We provide additional prompt-paired uncertainty estimates and independent validation to assess the reliability of CADI's improvements.

(1) Prompt-Paired Confidence Intervals. Following your suggestion, we computed 95% confidence intervals for Full CADI relative to the strongest external training baseline for each metric using 10,000 prompt-paired bootstrap replicates. The complete results are also presented in Figure 4 of the revised manuscript.

BackboneMetricExternal BaselineCADI − Baseline (95% CI)
Qwen2.5-7BDistinct₁₀DARLING[-0.137, +0.375]
Qwen2.5-7BQuality₁₀GAPO[-0.052, +0.275]
Qwen2.5-7BUtility₁₀DARLING[+0.058, +0.747]
Qwen2.5-7BHeterogeneityDARLING[-0.224, +1.329]
Llama-3.1-8BDistinct₁₀DARLING[+0.779, +1.650]
Llama-3.1-8BQuality₁₀DQO[-1.156, -0.384]
Llama-3.1-8BUtility₁₀DARLING[-0.239, +0.633]
Llama-3.1-8BHeterogeneityDARLING[+1.843, +4.157]

Full CADI achieves higher mean scores than the strongest external baselines across all four metrics on Qwen, and across Distinct₁₀, Utility₁₀, and Heterogeneity on Llama. The paired 95% confidence intervals remain entirely above zero for Utility₁₀ on Qwen and for Distinct₁₀ and Heterogeneity on Llama, providing statistical support for these improvements. The remaining positive mean differences have intervals overlapping zero, while Llama's Quality₁₀ is lower than DQO's.

(2) Independent Semantic and Quality Validation. To independently verify the results in Table 1, we re-evaluate the existing NoveltyBench outputs of Full CADI and the strongest training baselines under matched decoding settings and a fixed budget of 10 responses per prompt. We use GPT-6 as an independent evaluator. Under the same generation budget and decoding settings, we independently assessed valid semantic coverage and mean class quality, and recomputed utility accordingly.

BackboneMethodValid Coverage@10 ↑Mean Class Quality ↑Utility↑
Qwen2.5-7BDARLING5.296.854.16
CADI5.727.344.53
Llama-3.1-8BDARLING6.217.584.98
CADI7.187.445.32

On Qwen, CADI improves Valid Coverage@10 by 0.43, Mean Class Quality by 0.49, and Utility by 0.37 over DARLING. On Llama, CADI achieves 0.97 higher valid semantic coverage and 0.34 higher Utility, with only a 0.14 decrease in mean class quality. These results provide complementary evidence that CADI's semantic coverage and utility advantages persist under an independent evaluation protocol.


Q2. Stage 2 Evaluation Metrics and Evaluator Overlap

Thank you for raising this important question. Stage 2 follows the NoveltyBench evaluation protocol, using the released DeBERTa-v3-large-generation-similarity classifier for semantic clustering and Skywork-Reward-Gemma-2-27B-v0.2 for response quality scoring.

Specifically, diversity measures the number of distinct semantic classes, quality averages the scores of the first response in each observed class, and utility rewards the early discovery of distinct, high-quality responses using a discount factor of 0.8. The complete mathematical definitions are provided in Appendix C.2. For each training prompt, we compare eight unguided and eight guidance-conditioned responses, retaining only those satisfying the Pareto margins specified in Section 3.2 and Appendix C.1.

We acknowledge that these evaluators are also used for NoveltyBench testing, although the training and evaluation prompts are disjoint. To address potential evaluator-specific bias, we additionally conducted an independent GPT-6-based evaluation of existing CADI and DARLING outputs, as detailed in our response to Q1. CADI retains higher valid semantic coverage and utility on both backbones under this independent evaluation, providing complementary evidence that its improvements are not solely attributable to the original evaluators.

Q3. Stage 2 Computational Costs and Matched-Budget Controls

Thank you for this suggestion. We provide the realized Stage 2 training costs and additional controlled comparisons to assess whether its incremental gains justify the computational overhead.

(1) Training Efficiency and Resource Costs. Stage 2 generates eight unguided and eight guidance-conditioned responses per screened prompt. We employ early stopping based on held-out validation performance, selecting checkpoints at step 10 for Llama and step 15 for Qwen. The following table summarizes the actual generation volume, average screening acceptance rates, and resource costs.

BackboneSelected StepTotal GenerationsAvg. Acceptance RateGPU-hoursAPI Cost
Qwen2.5-7B1528.3k13.58%5.4 GPU·h$27.30
Llama-3.1-8B109.8k26.23%3.8 GPU·h$14.34

Total generations include both unguided and guidance-conditioned responses as recoreded in our log. Training was conducted on NVIDIA H100 GPUs. Guidance can also be generated by locally hosted models or the model being trained, we used an external API primarily to accelerate training and report its cost for transparency.

(2) Matched-Budget Distillation Controls. To assess whether simpler training strategies can recover similar improvements, we compare Full CADI with hard-target SFT and fixed-teacher context distillation. Both controls start from the same Stage 1 checkpoint and use the same accepted training trajectories, with comparable optimization budgets and identical evaluation protocols. Hard-target SFT applies cross-entropy supervision to accepted responses, while fixed-teacher distillation uses soft token targets from the frozen Stage 1 model with query-specific guidance.

MethodLlama Δ Distinct₁₀Llama Δ Utility₁₀Qwen Δ Distinct₁₀Qwen Δ Utility₁₀
Full CADI+0.18+0.13+0.15+0.18
Hard-target SFT+0.12+0.03+0.09−0.14
Fixed-teacher distillation+0.09+0.05+0.08+0.04

All improvements are relative to Stage 1. Hard-target SFT recovers 60–67% of Full CADI's Distinct₁₀ gains but substantially less of its Utility₁₀ improvements, even reducing Utility₁₀ on Qwen. Fixed-teacher distillation recovers 50–53% of the Distinct₁₀ gains and 22–38% of the Utility₁₀ gains.


Q4. Prompt-Category Analysis and Decoding Robustness

Thank you for this valuable suggestion. We provide additional category-level analysis and quality-matched comparisons across decoding configurations to assess the robustness of CADI's improvements.

(1) Performance Across Prompt Categories. We further analyze the results on NoveltyBench-Curated across four prompt categories, using the existing per-prompt evaluation results under the default decoding configuration (T=1.0, top-k=10, top-p=1.0). The table reports improvements of Full CADI over Base.

Prompt CategoryQwen Δ Distinct₁₀Qwen Δ Utility₁₀Llama Δ Distinct₁₀Llama Δ Utility₁₀
Randomness+1.000+0.856+1.267+1.107
Factual Knowledge+1.286+0.965+1.286+0.824
Creative Writing+0.667+0.750+1.444+0.784
Subjectivity+0.700+0.517+1.767+0.543

Following NoveltyBench's four-category taxonomy[1], we group the six fine-grained labels in the released dataset by merging creation and naming into Creative Writing, and recommendation and subjective opinion into Subjectivity.

CADI consistently improves both Distinct₁₀ and Utility₁₀ across all four categories on both backbones. The largest Distinct₁₀ gains occur in Factual Knowledge on Qwen (+1.286) and Subjectivity on Llama (+1.767), while Utility₁₀ improves across all categories. These results suggest that CADI's benefits extend across different types of open-ended prompts rather than being concentrated in a particular task category.

(2) Decoding Robustness at Matched Quality. We also evaluate CADI across six decoding configurations per backbone, varying temperature and top-k/top-p truncation. As detailed in Appendix C.3 (Tables 4–5) of the revised manuscript, CADI achieves the highest Utility₁₀ in 11 of the 12 settings and improves both Distinct₁₀ and Utility₁₀ over Stage 1 in all settings.

More importantly, under untruncated decoding (top-k disabled, top-p=1.0), we adjust sampling temperatures to compare Base and CADI at nearly identical response quality, using 10 responses per prompt. On Qwen, CADI (T=0.85) improves Distinct₁₀ by 1.31 and Utility₁₀ by 0.738 over Base (T=0.8), at comparable Quality₁₀ (7.593 vs. 7.570). Similarly, on Llama, CADI (T=0.8) improves Distinct₁₀ by 0.73 and Utility₁₀ by 0.612 over Base (T=1.0), at nearly identical Quality₁₀ (7.497 vs. 7.456).

Q5. Evaluation on More Recent Models and Challenging Benchmarks

Thank you for this valuable suggestion. We agree that evaluating more recent models and challenging benchmarks would further strengthen our conclusions. Accordingly, we have extended our experiments to Gemma-4-12B-it[2], comparing Base, CADI Stage 1, Full CADI, and DARLING. Beyond MMLU-Pro, we additionally evaluate SWE-bench Verified[3] and BIG-Bench Extra Hard (BBEH)[4] to assess general capabilities on more challenging tasks. The results are also included in Section 4.3 (Table 2) of the revised manuscript.

MethodDistinct₁₀ ↑Quality₁₀ ↑Utility₁₀ ↑Heterogeneity ↑MMLU-Pro ↑SWE-bench Verified ↑BBEH ↑
Base2.749.383.7712.7977.244.253.0
DARLING3.488.454.2414.3573.943.251.9
CADI Stage 13.739.134.5315.2877.844.452.6
Full CADI3.899.164.6815.9877.445.252.8

On Gemma-4-12B-it, Full CADI improves Distinct₁₀ by 1.15 and Utility₁₀ by 0.915 over Base, while reducing Quality₁₀ by only 0.224. It also outperforms DARLING in semantic diversity, utility, and response quality. Importantly, CADI largely preserves general capabilities across all three benchmarks, achieving the highest SWE-bench Verified score of 45.2%. These additional results further support the effectiveness of CADI on a more recent backbone and challenging evaluation tasks.

[1] Zhang, Yiming, et al. "NoveltyBench: Evaluating Language Models for Humanlike Diversity." Proceedings of the Second Conference on Language Modeling (COLM), 2025.
[2] Gemma Team, et al. "Gemma 4 Technical Report." arXiv, 2026, arXiv:2607.02770.
[3] Chowdhury, Neil, et al. "Introducing SWE-bench Verified." OpenAI, 13 Aug. 2024.
[4] Kazemi, Mehran, et al. "BIG-Bench Extra Hard." Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, 2025, pp. 26473–26501.


Thank you again for your time and thoughtful consideration. We would be happy to address any remaining concerns or provide further clarification. We sincerely hope that our responses help resolve your concerns and encourage you to reconsider your assessment.

Reviewer_EnyUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CADI, a two-stage training framework designed to improve output diversity in LLMs by reducing overly concentrated response distributions. The first stage calibrates candidate-level probabilities toward a balanced distribution using a KL-based training objective, while the second stage employs guided self-distillation to internalize diverse generation behaviors without requiring inference-time guidance. Experiments across multiple benchmarks demonstrate improvements in semantic diversity and utility while largely preserving response quality and general capabilities.

Critical strengths

S1. The paper provides a probabilistic formulation of output diversity and introduces a calibration objective based on length-normalized candidate likelihoods. This offers a simple and interpretable approach to reducing excessive probability concentration while preserving the base model's relative preferences.

S2. The proposed framework addresses concentrated output distributions by combining diversity calibration with guided self-distillation. Rather than relying solely on inference-time prompting or decoding strategies, CADI attempts to internalize diverse generation behavior into the model parameters.

S3. The paper evaluates CADI across multiple diversity benchmarks and compares it against various existing methods for improving output diversity. The inclusion of ablation studies, general-capability evaluations, and additional analyses of valid response accessibility strengthens the empirical assessment.

Critical weaknesses

W1. Gap Between Theoretical Analysis and Stage 1 Calibration.
The theoretical analysis in Section 2 establishes the relationship between probability concentration and expected coverage over semantic response classes. However, Stage 1 optimizes length-normalized candidate-selection distributions, which are not directly equivalent to the semantic response distributions considered in the theory. Consequently, the analysis does not establish why calibrating candidate-level probabilities should improve diversity in open-ended generation. A clearer connection between the theoretical formulation and the actual training objective would strengthen the methodological justification.

W2. Insufficient Justification for the Two-Stage Framework.
The relationship between diversity calibration and guided self-distillation remains insufficiently explained. Stage 1 reduces preference concentration through simple candidate-selection tasks, whereas Stage 2 distills guidance-conditioned responses selected through a Pareto criterion. Although the ablation results demonstrate the importance of Stage 1, they do not fully explain how calibration specifically facilitates subsequent self-distillation beyond providing a different initialization. Further justification of the sequential design and the complementary roles of the two stages would strengthen the proposed framework.

W3. Limited Incremental Gains and Unclear Advantages Over Existing Methods.
Most of CADI's diversity improvements already emerge from Stage 1, while Stage 2 provides relatively modest additional gains. Moreover, competing methods such as DARLING and DQO achieve better results on certain lexical diversity or response quality metrics. Given the additional complexity of guided response generation, Pareto filtering, and iterative distillation, the practical advantages of Stage 2 remain unclear. Comparisons of computational efficiency, training stability, and diversity–quality trade-offs would help establish its added value.

W4. Limitations of Uniformity-Based Diversity Calibration.
The proposed calibration encourages more balanced probabilities over valid candidates, but reducing probability concentration does not necessarily correspond to improving meaningful output diversity. Although the paper evaluates semantic diversity, broader aspects such as creativity, novelty, and context-dependent appropriateness may require non-uniform preferences over alternatives. The relationship between probability balancing and these aspects of diversity deserves further discussion, particularly regarding when a concentrated response distribution may be appropriate.

Questions for authors

Q1. Relationship Between Candidate Calibration and Open-Ended Diversity.
Under what conditions does calibrating a finite candidate-selection distribution improve diversity over the broader semantic response space? Could optimizing toward a predefined candidate pool inadvertently restrict alternatives outside that pool?

Q2. Context-Dependent Distributional Preferences.
How does CADI account for naturally different response distributions across linguistic, cultural, or task contexts? For example, could calibration distort appropriate response preferences in multilingual settings where English and Chinese prompts induce different distributions?

Q3. Relative Advantages Over Existing Methods.
Under what conditions does CADI offer advantages over DARLING or DQO, particularly when these methods achieve better lexical diversity or response quality? How do the approaches compare in computational cost and training stability?

Q4. Necessity of Sequential Training.
What motivates separating diversity calibration and guided self-distillation rather than jointly optimizing them? Would joint training or applying guided self-distillation directly to the base model achieve comparable results, and what evidence supports the proposed sequential design?

3 comments

We sincerely thank you for the thoughtful evaluation and constructive feedback. We greatly appreciate your valuable suggestions and provide detailed responses to each of your concerns below.


Q1. Generalization Beyond Candidate Calibration

Thank you for this insightful question. Our theoretical analysis establishes that reducing probability concentration improves expected coverage over a fixed semantic response space, but does not assume that calibration on finite candidate sets necessarily guarantees improved open-ended diversity. Instead, we use candidate-selection tasks as a controlled proxy for calibrating model preferences, without restricting the response space at inference time.

To mitigate the risk of suppressing valid alternatives, CADI employs a base-anchored calibration target that preserves the relative preferences of the base model, together with token-level KL regularization to limit distributional drift.

We provide three complementary empirical observations:

  • Held-out generalization (Appendix C.9): On unseen candidate categories, values, and prompt templates, Stage 1 reduces preference-distribution JSD by 66.5% on Qwen and 71.2% on Llama, with improvements in all 96 evaluated conditions per backbone.
  • Open-ended diversity (Table 1): Despite training only on candidate-selection tasks, Stage 1 improves NoveltyBench Distinct₁₀ by 0.85 on Qwen and 1.29 on Llama.
  • Valid alternative accessibility (Figure 5, Appendix C.6): On open-ended story continuations, Full CADI improves valid-token Recall@1000 from 91.28% to 98.71% on Qwen and from 88.99% to 96.17% on Llama, while preserving or improving top-ranked validity.

These results suggest that CADI's calibration effects extend beyond the predefined candidate pools, improving access to valid alternatives rather than merely redistributing probability among training candidates.


Q2. Context-Dependent Distributional Preferences

Thank you for raising this concern. CADI does not impose a globally uniform distribution across linguistic or cultural contexts. Instead, its base-anchored calibration target is prompt- and candidate-conditioned:

qα∗(x,Ax)=αq0(x,Ax)+(1−α)uKq_\alpha^*(x,A_x)=\alpha q_0(x,A_x)+(1-\alpha)u_K

Here, q0(x,Ax)q_0(x,A_x) captures context-dependent preferences. With α=0.6\alpha=0.6, calibration preserves the original candidate ranking while reducing excessive concentration. Token-level KL regularization further limits distributional drift, helping preserve context-specific preferences.

Multilingual and Cultural Evaluation. To empirically examine this concern, we constructed 50 English–Chinese parallel prompt pairs, comprising 20 general tasks, 10 Chinese-cultural contexts, 10 English-speaking cultural contexts, and 10 task/audience-constrained prompts. We evaluated Base, Stage 1, and Full CADI on Qwen2.5-7B and Llama-3.1-8B using identical decoding settings (T=1.0, top-k=10, top-p=1.0), generating 10 responses per prompt and language (6,000 responses in total).

A blinded GPT-6 evaluator assessed language adherence, contextual appropriateness, task validity, and usefulness. Responses satisfying the applicable validity criteria and achieving usefulness ≥3/5 were counted as valid and useful. We additionally measured semantic coverage among valid responses.

BackboneLanguageValid & useful (%) Base → Stage 1 → FullValid Coverage@10 Base → Stage 1 → FullFull − Base (95% CI, pp)
Qwen2.5-7BEnglish97.0 → 98.6 → 99.03.00 → 3.20 → 3.28+2.0 [0.4, 4.0]
Chinese94.6 → 95.8 → 95.42.94 → 2.89 → 2.96+0.8 [−0.8, 3.0]
Llama-3.1-8BEnglish98.2 → 97.4 → 98.42.88 → 3.22 → 3.28+0.2 [−0.2, 0.8]
Chinese85.4 → 88.4 → 87.22.68 → 2.86 → 2.92+1.8 [−1.2, 5.2]

Confidence intervals use 10,000 bootstrap replicates over the 50 paired prompts. The full evaluation prompts are available in our anonymous repository.

Context-Specific Results. Importantly, the improvements do not require eliminating language-dependent differences. Full CADI increases valid semantic coverage over Base in both languages on both backbones, while maintaining comparable overall validity rates. Within Chinese-language cultural tasks, Qwen maintains 99–100% validity across both cultural categories. For Llama, validity changes from 92% to 89% in Chinese-cultural contexts and from 95% to 99% in English-speaking cultural contexts, indicating some context-specific variation rather than uniformly improved performance.

Q3. Relative Advantages Over Existing Methods

Thank you for this important question. We further clarify CADI's advantages over DARLING and DQO in terms of diversity–quality trade-offs, computational efficiency, and training stability.

(1) Diversity–Quality Trade-off. CADI primarily aims to improve semantic coverage and utility rather than maximizing lexical diversity or response quality in isolation. As shown in Table 1, Full CADI outperforms both DARLING and DQO in Distinct₁₀, Utility₁₀, and Heterogeneity on both backbones. While DARLING achieves stronger lexical diversity on Qwen and DQO retains higher response quality on Llama, CADI offers stronger semantic coverage and utility. At nearly identical response quality under untruncated decoding (7.158 vs. 7.126), CADI also improves Distinct₁₀ by 0.57 and Utility₁₀ by 0.269 over DARLING on Llama.

(2) Computational Efficiency. Notably, even Stage 1 calibration alone achieves highly competitive performance at substantially lower training cost. The following table compares its performance with DARLING and DQO under the default evaluation settings, alongside the relative GPU costs recorded in our training experiments.

BackboneMethodDistinct₁₀ ↑Quality₁₀ ↑Utility₁₀ ↑Heterogeneity ↑Relative GPU Cost
Qwen2.5-7BCADI Stage 15.417.664.7119.101×
DARLING5.437.124.5218.9032.9×
DQO5.067.494.4118.208.5×
Llama-3.1-8BCADI Stage 17.037.435.6023.101×
DARLING6.008.065.5321.4029.7×
DQO5.548.165.1519.308.2×

All training experiments were conducted using NVIDIA H100 GPUs. Remarkably, Stage 1 alone achieves higher Utility₁₀ and Heterogeneity than both baselines on both backbones, while DQO and DARLING require 8.2–8.5× and approximately 30× the GPU compute of CADI Stage 1, respectively. It also achieves higher Distinct₁₀ on Llama and nearly matches DARLING on Qwen.

These efficiency differences partly reflect the additional optimization and evaluation overhead of the baselines. DQO jointly trains a policy and a separate critic, while DARLING requires repeated reward evaluation and semantic classification during training.

Stage 2 introduces only a modest additional cost: 5.4 GPU-hours on Qwen and 3.8 GPU-hours on Llama. Thus, Stage 1 provides an efficient standalone solution, while Stage 2 further improves diversity and utility when additional training resources are available.

(3) Training Stability. In our training and baseline reproduction experiments, CADI, DARLING, and DQO all exhibited stable optimization behavior without notable training divergence. We did not observe substantial instability that would distinguish these methods. We additionally report prompt-paired 95% confidence intervals in Figure 4 of the revised manuscript, supporting CADI's incremental diversity and utility improvements over Stage 1. These intervals characterize evaluation uncertainty rather than variability across independently trained runs.

Q4. Necessity of Sequential Training

Thank you for this insightful question. Our sequential design is motivated by the complementary roles of the two stages. Stage 1 first reduces excessive preference concentration, making valid but underrepresented alternatives more accessible. Stage 2 then builds on this calibrated distribution to discover and selectively distill query-specific improvements. This ordering provides a more diverse starting point for guided self-distillation.

Empirical Evidence. To assess whether Stage 1 is necessary, we directly apply Stage 2 to the original Base model using the same training procedure (w/o Stage 1 in Figure 6). The results are summarized below.

BackboneMethodDistinct₁₀Quality₁₀Utility₁₀Heterogeneity
Qwen2.5-7BStage 2 Only4.547.684.1317.23
Full CADI5.567.684.8919.40
Llama-3.1-8BStage 2 Only5.707.814.9819.77
Full CADI7.217.395.7324.40

Without Stage 1, guided self-distillation yields limited semantic diversity gains. In contrast, the full sequential framework improves Distinct₁₀ by 1.02 and 1.51, Utility₁₀ by 0.76 and 0.75, and Heterogeneity by 2.17 and 4.63 points on Qwen and Llama, respectively.

Why Sequential Rather Than Joint Optimization? The two stages operate on different objectives: Stage 1 calibrates candidate-level preferences toward a base-anchored target, whereas Stage 2 performs on-policy distillation using dynamically refreshed teacher distributions. Separating them allows calibration to establish a more diverse initialization before iterative distillation. Joint optimization is possible in principle, but would introduce additional optimization complexity, requiring careful balancing of potentially competing calibration and distillation objectives, as well as extensive tuning of their relative weights and training schedules. Our ablation results directly support the importance of calibration before guided self-distillation.


Thank you again for your time and thoughtful consideration. We would be happy to address any remaining concerns or provide further clarification. We sincerely hope that our responses help resolve your concerns and encourage you to reconsider your assessment.

Reviewer_ubj9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CADI, a two-stage framework for improving LLM output diversity while preserving response quality and general capabilities. The first stage, diversity calibration, reduces overly concentrated model preferences through simple candidate-selection tasks while retaining the original preference structure. The second stage, online guided self-distillation, selectively distills guidance-conditioned responses that improve diversity, utility, and quality into an unguided student. Experiments across multiple LLM backbones on NoveltyBench and Artificial Hivemind demonstrate improvements in semantic diversity and utility over existing training-based methods.

Critical strengths
  1. Well-motivated problem I appreciate the paper's focus on output homogenization, a meaningful challenge for open-ended generation. The theoretical analysis provides a clear connection between probability concentration and limited output diversity, offering useful motivation for the proposed approach.
  2. Simple and intuitive methodology The two-stage framework is conceptually straightforward yet interesting.
  3. Comprehensive empirical evaluation Besides the concern below, I think the paper provides extensive comparisons with relevant training-based baselines across multiple model scales, along with ablations, sensitivity studies, and preference-order analyses.
Critical weaknesses
  1. Limited incremental benefit of Stage 2. I find the contribution of Stage 2 somewhat underwhelming. Table 1 shows that Stage 1 already accounts for most of the diversity improvements, while Stage 2 provides only modest additional gains (+0.18/+0.15 in Distinct10 across the two backbones). Given the additional costs of external guidance generation, sampling, verification, and iterative distillation, it remains unclear whether these gains justify the added complexity. A compute-matched comparison and statistical analysis would help establish the necessity of Stage 2.

  2. Insufficient analysis of the diversity–quality trade-off. A central goal of CADI is to improve diversity while preserving response quality. However, Quality10 decreases for three of the four evaluated model scales, including a 0.39-point drop on Llama-3.1-8B. I find this particularly interesting because Stage 2 explicitly filters guided responses based on improvements in diversity, utility, and quality. The observed regressions raise questions about how reliably these improvements transfer to the final unguided model. A deeper analysis of the diversity–quality trade-off would strengthen the paper's claims.

  3. Limited comparisons with inference-time diversity methods. The paper motivates CADI partly by highlighting the limitations of inference-time approaches, yet the main experiments focus almost exclusively on training-based baselines. I would have liked to see comparisons with simple, tuned decoding strategies, such as temperature and top-k/top-p sampling, under comparable quality constraints. Although these approaches do not internalize diversity into model parameters, such comparisons would help clarify the practical advantages of CADI over simpler alternatives.

Questions for authors

See weakness above.

4 comments

We sincerely thank you for the thoughtful evaluation and constructive feedback. We greatly appreciate your valuable suggestions and provide detailed responses to each of your concerns below.


Q1. Incremental benefit and cost of Stage 2

Thank you for raising this important concern. We agree that Stage 1 accounts for most of the diversity improvements. To better establish the contribution and cost-effectiveness of Stage 2, we provide additional statistical analysis, alternative distillation controls, and realized training costs below.

(1) Stage 2 Gains and Statistical Reliability. Although the incremental Distinct₁₀ gains are modest, Stage 2 consistently improves semantic coverage and utility across both backbones. Using 10,000 prompt-paired bootstrap replicates, we find that the 95% confidence intervals for Distinct₁₀, Utility₁₀, and Heterogeneity are all above zero.

MetricLlama-3.1-8B Δ [95% CI]Qwen2.5-7B Δ [95% CI]
Distinct₁₀+0.18 [+0.104, +0.370]+0.15 [+0.083, +0.358]
Utility₁₀+0.13 [+0.069, +0.286]+0.18 [+0.094, +0.364]
Heterogeneity+1.30 [+1.062, +1.538]+0.30 [+0.141, +0.459]

While Stage 1 already accounts for most of the diversity gains, Stage 2 further improves the diversity–utility balance. Importantly, its contribution also depends on the decoding policy. As detailed in Appendix C.3 of the revised manuscript, under untruncated sampling at T = 1.0, Stage 2 improves Quality₁₀ by 0.231 on Qwen and 0.299 on Llama, compared with only 0.033 and 0.052 under combined top-k/top-p truncation. This pattern is consistent with truncation already suppressing low-probability continuations and leaving less room for further quality improvements. Thus, the modest gains observed under truncated decoding do not fully reflect Stage 2's benefits across decoding settings.

(2) Alternative Distillation Controls. During the development of CADI, we explored alternative training strategies.

To further substantiate our design choices, we compare Full CADI with hard-target SFT and fixed-teacher context distillation. To provide an approximately compute-matched comparison, both controls are initialized from the same Stage 1 checkpoint and trained on the same accepted trajectories as Full CADI, using comparable optimization budgets and identical evaluation protocols. Hard-target SFT applies cross-entropy to accepted response tokens, while fixed-teacher distillation uses the frozen Stage 1 teacher with the original query-specific guidance and the same token-level KL objective.

MethodLlama Δ Distinct₁₀Llama Δ Utility₁₀Qwen Δ Distinct₁₀Qwen Δ Utility₁₀
Full CADI (final checkpoint)+0.18+0.13+0.15+0.18
Hard-target SFT+0.12+0.03+0.09−0.14
Fixed-teacher context distillation+0.09+0.05+0.08+0.04

All differences are relative to Stage 1. Hard-target SFT recovers 60–67% of Full CADI's Distinct₁₀ gains but substantially less of its Utility₁₀ improvements, even reducing Utility₁₀ on Qwen. Fixed-teacher distillation recovers 50–53% of the Distinct₁₀ gains and 22–38% of the Utility₁₀ gains. These results suggest that simpler distillation strategies capture part of the improvement but do not recover the full diversity–utility gains of CADI.

(3) Training Efficiency and Resource Costs. We report the realized Stage 2 training costs below. We use early stopping based on held-out validation performance, retaining the best checkpoint. The selected checkpoints are at step 10 for Llama and step 15 for Qwen, with acceptance rates averaged across screening steps.

BackboneSeleted stepAcceptance rate (per step)GPU-hoursAPI Cost
Qwen2.5-7B1513.58%5.4 GPU·h$27.30
Llama-3.1-8B1026.23%3.8 GPU·h$14.34

Training was conducted on NVIDIA H100 GPUs. Guidance can also be generated by a locally hosted model or the model itself. We used an external API primarily to accelerate training and report its cost for transparency.

Overall, these results show that Stage 2 provides statistically supported improvements with relatively modest additional computational costs.

Q2. Diversity–quality trade-off

Thank you for highlighting this important concern. We agree that preserving response quality while improving semantic diversity is essential. We provide further analysis to clarify the observed quality changes and their relationship to Stage 2.

(1) Understanding Quality Changes. The Pareto criterion ensures that accepted guided responses outperform unguided responses from the same frozen checkpoint, but does not guarantee quality improvements after distillation on unseen prompts. Moreover, Quality₁₀ averages quality across observed semantic classes, so expanding coverage to additional valid classes may reduce the average quality score.

Importantly, most of the quality decrease on Llama occurs during Stage 1 rather than Stage 2. We summarize the quality changes and decoding sensitivity below.

BackboneQuality₁₀: Base → Stage 1 → Full (Default)Stage 2 Δ Quality₁₀ (Untruncated)Stage 2 Δ Quality₁₀ (Combined Truncation)
Llama-3.1-8B7.78 → 7.43 → 7.39+0.299+0.052
Qwen2.5-7B7.58 → 7.66 → 7.68+0.231+0.033

As shown in Figure 4 of the revised manuscript, the prompt-paired 95% confidence intervals for Stage 2's Quality₁₀ changes include zero on both backbones, indicating no clear statistical evidence of a systematic quality regression attributable to Stage 2.

(2) Quality–Diversity Trade-off Across Decoding Settings. The decoding sweep in Appendix C.3 further shows that Stage 2 yields larger quality improvements without truncation (T = 1.0, top-k disabled, top-p = 1.0). These gains become smaller under combined truncation (T = 1.0, top-k = 10, top-p = 0.9), consistent with truncation already suppressing low-probability continuations and leaving less room for further quality improvements. Importantly, Stage 2 continues to improve Distinct₁₀ and Utility₁₀ under both settings.

Furthermore, at comparable response quality, CADI covers 0.73 more semantic classes per prompt than Base on Llama (Quality₁₀: 7.497 vs. 7.456). After filtering responses with quality scores below 3, CADI still retains 4.54 distinct semantic classes per prompt, compared with 3.55 for Base. We additionally conducted independent GPT-6-based assessments, using ten judgments per additional semantic class with majority voting, and found that 83.7% of the assessed classes were both valid and useful. These results provide further evidence that CADI's increased semantic coverage is not merely driven by lower-quality responses.

These results indicate that Stage 2 provides additional diversity and utility gains without a reliably detected quality regression relative to Stage 1, while its quality benefits are more pronounced under untruncated decoding.

Q3. Comparison with Inference-Time Decoding Strategies

Thank you for this valuable suggestion. To examine whether simple decoding adjustments can achieve comparable diversity–quality trade-offs, we systematically evaluate Base and CADI across six decoding configurations, varying temperature and top-k/top-p truncation. The complete results, including all training baselines, are provided in Appendix C.3 (Tables 4–5) of the revised manuscript. We summarize the key NoveltyBench results below.

BackboneDecodingMethodDistinct₁₀ ↑Quality₁₀ ↑Utility₁₀ ↑
QwenT=1.0, k=10, p=1.0Base4.567.584.05
CADI5.567.684.89
T=0.8, untruncatedBase4.047.573.68
CADI4.977.694.38
T=1.0, untruncatedBase4.797.374.09
CADI6.327.054.74
T=1.2, untruncatedBase6.065.823.63
CADI7.625.163.99
T=1.0, p=0.9Base4.097.693.82
CADI5.137.864.59
T=1.0, k=10, p=0.9Base3.877.813.76
CADI4.917.864.45
LlamaT=1.0, k=10, p=1.0Base5.747.784.92
CADI7.217.395.73
T=0.8, untruncatedBase5.047.944.52
CADI6.737.505.38
T=1.0, untruncatedBase6.007.464.77
CADI7.267.165.45
T=1.2, untruncatedBase7.114.243.32
CADI8.403.743.58
T=1.0, p=0.9Base5.207.914.50
CADI7.127.405.59
T=1.0, k=10, p=0.9Base4.877.994.45
CADI6.257.695.32

Untruncated: top-k disabled, top-p = 1.0. For top-p-only sampling, top-k is disabled. Bold indicates the better value within each Base–CADI pair.

Across all 12 backbone–decoding combinations, CADI consistently improves both Distinct₁₀ and Utility₁₀ over Base. Increasing temperature can also improve Base's diversity, but often at a substantial quality cost.

More importantly, CADI retains its advantages at comparable response quality. We conduct quality-matched comparisons on NoveltyBench by adjusting the sampling temperature for each model, with top-k disabled and top-p = 1.0, using 10 responses per prompt. On Qwen, CADI (T = 0.85) achieves higher Distinct₁₀ than Base (T = 0.8), with 5.35 vs. 4.04 at nearly identical Quality₁₀ (7.593 vs. 7.570), alongside a Utility₁₀ gain of 0.738. Similarly, on Llama, CADI (T = 0.8) achieves 6.73 Distinct₁₀ compared with Base (T = 1.0) at 6.00, with comparable Quality₁₀ (7.497 vs. 7.456) and a Utility₁₀ gain of 0.612.

These results indicate that CADI provides a more favorable diversity–utility trade-off than the tested decoding adjustments alone.


Thank you again for your time and thoughtful consideration. We would be happy to address any remaining concerns or provide further clarification. We sincerely hope that our responses help resolve your concerns and encourage you to reconsider your assessment.

Thanks for the detailed response and additional experiments. I think the new results largely address my concerns about W2 and W3. My main reservation is still about the contribution of Stage 2. While the new ablations provide better evidence for its effectiveness, most of the improvements still come from Stage 1, and the additional gains remain relatively modest. I'm still not fully convinced that Stage 2 adds enough to justify the more complex two-stage framework. Thus, I'll maintain my original rating.

Reviewer_kur3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CADI combines candidate-preference calibration with online guided self-distillation to improve output diversity. Stage 1 fits an induced candidate-preference distribution to a mixture of the initial preferences and a uniform distribution, with a token-level KL anchor. Stage 2 compares plain and guidance-conditioned response sets from the same frozen model snapshot, filters guidance using diversity, utility, and quality margins, and distills the guided token distributions into an unguided student. GPT-5.5 supplies the guidance. Results on NoveltyBench and Artificial Hivemind support the approach under the reported decoding policy, with much of the improvement already present after Stage 1. The appendices provide substantial supporting analyses. My remaining concerns concern decoding robustness, uncertainty in the incremental Stage 2 gains, and the diversity-quality trade-off on Llama.

Critical strengths
  1. Stage 1 is simple and produces substantial gains. On Llama, Distinct_10 increases from 5.74 to 7.03 and Utility_10 from 4.92 to 5.60 before Stage 2 (Table 1, p. 7). The full method has the highest reported Distinct_10, Utility_10, and Heterogeneity among the compared trained variants for both main backbones, although it does not lead every metric.
  2. The evaluation covers two main backbones and four training-based baselines. Figure 5 ablates Stage 1, the base-aware target, the token-level anchor, and Pareto filtering. Figure 6 studies calibration hyperparameters; Appendix C.7 varies the calibration candidate count. Appendix C.4 additionally compares full CADI with the base models at 32B and 70B, and Figure 3 supports general-capability retention on the tested tasks.
  3. The appendices contain meaningful validation. Appendix C.2 reports 79% accuracy and F1 0.811 for the semantic classifier on a disjoint 100-pair human-labeled test set. Appendix C.6 includes correct-only, matched-count GSM8K comparisons, clustering-threshold sensitivity, and paired bootstrap intervals. Appendices C.5 and C.8 also quantify uncertainty in ranking and held-out calibration analyses. These results strengthen the evidence beyond unfiltered diversity scores alone. The authors release code, configurations, and training data, and select the final Stage 2 checkpoint using validation data only (p. 7).
Critical weaknesses
  1. The robustness of the diversity-quality advantage to decoding remains insufficiently characterized. The open-ended evaluation uses temperature 1.0, top-p 1.0, and top-k 10 (Section 4.1), consistent with the public NoveltyBench evaluation script. Table 1 directly supports gains at this setting. However, the manuscript does not report a decoding sweep or quality-matched comparisons against simply adjusting the base model's decoding. Because truncation and temperature can change the accessible response distribution, comparisons across temperature and top-k/top-p settings, including no top-k truncation, would better establish the practical advantage of training. This concern concerns generalization across decoding policies, not the validity of the observed gains at the reported setting.
  2. The precision and resource efficiency of the incremental Stage 2 gains are not established by the main results. Relative to Stage 1, Distinct_10 increases by 0.18 on Llama and 0.15 on Qwen, and Utility_10 by 0.13 and 0.18 (Table 1). These are positive point estimates, but the manuscript does not give paired intervals for these main-benchmark differences or quantify variability across training runs. Existing component ablations support the design, but do not resolve this uncertainty. Stage 2 adds guidance generation, plain/guided sampling, scoring, filtering, and distillation. The public configurations specify a maximum of 50 updates and a prompt batch size of 64; this is not a missing-configuration criticism. Reporting realized acceptance rates and total generation, evaluation, and training resources alongside the incremental gains would clarify the practical value of this stage.
  3. The diversity-quality trade-off on Llama deserves a more direct evaluation. In Table 1, Llama's Quality_10 decreases from 7.78 to 7.39 overall, with most of the decrease already present after Stage 1 (7.43); Utility_10 nevertheless increases from 4.92 to 5.73. Qwen improves both quality and utility. Thus, the evidence does not support a general claim that CADI harms quality, nor does it establish that the small Stage 2 quality change is significant. Quality_10 averages the first representative of each observed semantic class (Appendix C.2), so the change can also reflect which classes are discovered. Quality-matched coverage or coverage after an explicit quality threshold, together with an independent quality check of newly discovered classes, would help interpret the Llama trade-off. The existing human-label check validates semantic equivalence, while the correct-only GSM8K analysis addresses a separate reasoning setting. This request supplements those controls; it does not imply that the per-prompt Pareto filter is invalid or that it promises a held-out student-level guarantee.
Questions for authors
  1. How do the base model, CADI, and strong training baselines compare on quality-matched diversity and utility curves across decoding policies, especially with top-k truncation removed?
  2. Can the authors provide prompt-paired confidence intervals for full CADI versus Stage 1 and versus the strongest comparator for each main metric, and distinguish evaluation sampling uncertainty from training-run variability?
  3. Given the released Stage 2 configurations, what are the realized update counts, per-round screening acceptance rates, and total resource costs for the reported runs? As an additional control, would generic guidance or hard-target distillation of the same accepted responses recover a similar fraction of the Stage 2 gains?
  4. How does the Llama quality difference change when coverage is measured at matched quality or after quality filtering, and do independent judgments confirm the usefulness of the additional semantic classes?
4 comments

We appreciate your careful review and constructive feedback. Below, we respectfully provide our detailed responses to address your concerns.


Q1. Diversity and utility at comparable quality

(1) Quality-Matched Comparisons. Thank you for this suggestion. We evaluate CADI and the baselines at comparable response quality by varying sampling temperature. The table reports directly measured results under untruncated decoding (top-k=-1, top-p = 1.0), with temperature varied as indicated. Distinct₁₀, Quality₁₀, and Utility₁₀ are evaluated on NoveltyBench, while the other diversity metrics follow the evaluation protocols in Section 4.1.

BackboneMethodT (Untruncated)Div-BLEU ↑EAD ↑Sent-BERT ↑Distinct₁₀ ↑Quality₁₀ ↑Utility₁₀ ↑Heterogeneity ↑
Llama-3.1-8BBase1.0051.3671.8730.676.007.464.7721.94
Llama-3.1-8BCADI0.8055.7172.5435.186.737.505.3823.59
Llama-3.1-8BDARLING1.0065.2281.9632.626.697.135.1822.58
Llama-3.1-8BCADI1.0066.8484.7741.397.267.165.4525.54
Qwen2.5-7BBase0.8035.6255.3222.384.047.573.6815.69
Qwen2.5-7BCADI0.8544.8363.1225.595.357.594.4218.94
Qwen2.5-7BGAPO1.0047.2165.8727.415.517.124.3619.51
Qwen2.5-7BCADI1.0052.0569.2629.346.327.054.7421.72

On Llama, CADI improves Distinct₁₀ and Utility₁₀ by 0.73 and 0.61 over Base at nearly identical quality (7.50 vs. 7.46). Compared with DARLING under identical decoding settings, CADI improves Distinct₁₀ by 0.57 and Utility₁₀ by 0.27, with a quality difference of 0.03. On Qwen, CADI improves Distinct₁₀ and Utility₁₀ by 1.31 and 0.74 over Base at comparable quality (7.59 vs. 7.57). CADI also outperforms GAPO on both metrics under identical decoding settings.

(2) Robustness Across Decoding Policies. We have included a comprehensive decoding sweep in Appendix C.3 of the revised manuscript, covering six configurations per backbone with varying temperatures and top-k/top-p truncation settings. CADI achieves the highest Utility₁₀ in 11 of the 12 settings and consistently improves both Distinct₁₀ and Utility₁₀ over Stage 1 across all settings. These results demonstrate that CADI's gains are robust across different decoding policies rather than being tied to a particular sampling configuration.


Q2. Prompt-paired confidence intervals

Thank you for raising this point. To ensure consistent comparisons, we fixed the training seeds and followed the same evaluation protocol across methods. Each score in Table 1 is averaged over five evaluations of the same checkpoint using different decoding seeds. Following your suggestion, we computed 95% confidence intervals for metric differences using 10,000 prompt-paired bootstrap replicates. These intervals quantify evaluation uncertainty conditional on fixed checkpoints and sampled responses. Due to computational constraints, we conducted one training run per configuration. Therefore, these intervals do not capture training-run variability across independently trained checkpoints.

BackboneMetricCADI − Stage 1 (95% CI)External baselineCADI − baseline (95% CI)
Qwen2.5-7BDistinct₁₀[+0.083, +0.358]DARLING[-0.137, +0.375]
Qwen2.5-7BQuality₁₀[-0.036, +0.163]GAPO[-0.052, +0.275]
Qwen2.5-7BUtility₁₀[+0.094, +0.364]DARLING[+0.058, +0.747]
Qwen2.5-7BHeterogeneity[+0.141, +0.459]DARLING[-0.224, +1.329]
Llama-3.1-8BDistinct₁₀[+0.104, +0.370]DARLING[+0.779, +1.650]
Llama-3.1-8BQuality₁₀[-0.121, +0.107]DQO[-1.156, -0.384]
Llama-3.1-8BUtility₁₀[+0.069, +0.286]DARLING[-0.239, +0.633]
Llama-3.1-8BHeterogeneity[+1.062, +1.538]DARLING[+1.843, +4.157]

Relative to Stage 1, Full CADI achieves higher mean Distinct₁₀, Utility₁₀, and Heterogeneity on both backbones, with all six paired 95% confidence intervals above zero. The Quality₁₀ intervals include zero on both backbones. Against DARLING, the 95% confidence intervals also lie above zero for Utility₁₀ on Qwen ([+0.058, +0.747]) and for Distinct₁₀ and Heterogeneity on Llama ([+0.779, +1.650] and [+1.843, +4.157], respectively).

Q3. Stage 2 efficiency and alternative controls

Thank you for these suggestions.

(1) Training Efficiency and Resource Costs. We report the realized Stage 2 training statistics and computational costs below. We adopt early stopping based on held-out validation performance. The selected checkpoints are at step 10 for Llama-3.1-8B and step 15 for Qwen2.5-7B. Acceptance rates are averaged across screening steps. Training was conducted on NVIDIA H100 GPUs, with an external API used for guidance generation. Guidance can also be generated by a locally hosted model or the model itself, which we empirically found to yield comparable performance. We used an external API for the reported runs primarily to accelerate training, and report its cost for transparency.

BackboneSelected stepAcceptance rate (per step)GPU-hoursAPI Cost
Qwen2.5-7B1513.58%5.4 GPU·h$27.30
Llama-3.1-8B1026.23%3.8 GPU·h$14.34

(2) Alternative Distillation Controls. We appreciate this suggestion. During the early development of CADI, we explored several alternative training strategies. To further substantiate our design choices, we provide additional comparisons with hard-target SFT and fixed-teacher context distillation. Both controls start from the same Stage 1 checkpoint and use the same accepted training trajectories as Full CADI.

  • Hard-target SFT: Applies cross-entropy supervision to the accepted response tokens.
  • Fixed-teacher context distillation: Uses the frozen Stage 1 model with the original query-specific guidance to provide soft token targets, retaining the same token-level KL objective as Full CADI.

Llama-3.1-8B

MethodΔ Distinct₁₀Δ Utility₁₀
Full CADI+0.18+0.13
Hard-target SFT+0.12+0.03
Fixed-teacher context distillation+0.09+0.05

Qwen2.5-7B

MethodΔ Distinct₁₀Δ Utility₁₀
Full CADI+0.15+0.18
Hard-target SFT+0.09−0.14
Fixed-teacher context distillation+0.08+0.04

All differences are relative to Stage 1. Hard-target SFT recovers 67% and 60% of Full CADI's Distinct₁₀ gains on Llama and Qwen, respectively, but only 22% of its Utility₁₀ gain on Llama, while reducing Utility₁₀ on Qwen. Fixed-teacher distillation recovers 50–53% of the Distinct₁₀ gains and 22–38% of the Utility₁₀ gains. These results indicate that simpler distillation strategies capture part of the diversity improvement but do not recover the full diversity–utility gains achieved by CADI.

We also explored generic diversity guidance in preliminary experiments and observed comparable performance to query-specific guidance. However, its Pareto screening acceptance rate was only approximately 20–25% of that achieved by query-specific guidance, requiring substantially more generation and screening to collect comparable amounts of accepted training data. Given this additional computational overhead, we did not pursue generic guidance for full Stage 2 training.

Q4. Quality-Controlled Coverage and Validity

Thank you for this suggestion. As shown in our response to Q1, CADI covers 0.73 more semantic classes per prompt than Base on Llama at comparable mean Quality₁₀ (7.497 vs. 7.456), with top-k truncation disabled. Under the same untruncated decoding setting as DARLING, CADI also achieves higher Distinct₁₀ (7.26 vs. 6.69) at nearly identical Quality₁₀ (7.158 vs. 7.126).

Quality filtering. After excluding responses with quality scores below 3, CADI retains 4.54 distinct semantic classes per prompt, compared with 3.55 for Base, demonstrating that its coverage advantage persists even after removing low-quality responses.

Independent validation. To independently assess the validity and usefulness of the additional semantic classes, we conducted ten separate GPT-6 evaluations per class and aggregated the judgments through majority voting like prior works[1][2]. Over 83.7% of the evaluated classes were judged to be both valid and useful. Together, these results provide further evidence that CADI expands meaningful semantic coverage rather than merely generating lower-quality variations.

[1] Ruan, Zhiwen, et al. "G2: Guided generation for enhanced output diversity in LLMs." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.
[2] Fawzi, Fares, et al. "SCRIBE: Structured chain reasoning for interactive behaviour explanations using tool calling." Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025.


Thank you again for your time and thoughtful consideration. We would be happy to address any remaining concerns or provide further clarification. We sincerely hope that our responses help resolve your concerns and encourage you to reconsider your assessment.

Comments on the paper

4 comments

We appreciate your careful review and constructive feedback. Below, we respectfully provide our detailed responses to address your concerns.


Q1. Comparison with Decoding-Tuned Base Models

Thank you for this valuable suggestion. To examine whether decoding adjustments alone can achieve comparable diversity–quality trade-offs, we systematically evaluate Base and CADI across six decoding configurations per backbone, varying temperature and top-k/top-p truncation. All comparisons use the same generation budget of 10 responses per prompt on NoveltyBench. The complete results, including all training baselines, are provided in Appendix C.3 (Tables 4–5) of the revised manuscript.

BackboneDecodingMethodDistinct₁₀ ↑Quality₁₀ ↑Utility₁₀ ↑
QwenT=1.0, k=10, p=1.0Base4.567.584.05
CADI5.567.684.89
T=0.8, untruncatedBase4.047.573.68
CADI4.977.694.38
T=1.0, untruncatedBase4.797.374.09
CADI6.327.054.74
T=1.2, untruncatedBase6.065.823.63
CADI7.625.163.99
T=1.0, p=0.9Base4.097.693.82
CADI5.137.864.59
T=1.0, k=10, p=0.9Base3.877.813.76
CADI4.917.864.45
LlamaT=1.0, k=10, p=1.0Base5.747.784.92
CADI7.217.395.73
T=0.8, untruncatedBase5.047.944.52
CADI6.737.505.38
T=1.0, untruncatedBase6.007.464.77
CADI7.267.165.45
T=1.2, untruncatedBase7.114.243.32
CADI8.403.743.58
T=1.0, p=0.9Base5.207.914.50
CADI7.127.405.59
T=1.0, k=10, p=0.9Base4.877.994.45
CADI6.257.695.32

Untruncated sampling uses top-k disabled and top-p = 1.0; top-p-only sampling also disables top-k. Bold indicates the better result within each Base–CADI pair.

Across all 12 backbone–decoding combinations, CADI consistently achieves higher Distinct₁₀ and Utility₁₀ than Base. Increasing temperature also improves Base's diversity, but often at a substantial quality cost.

Quality-Matched Comparisons. More importantly, we compare Base and CADI at nearly identical response quality by adjusting their sampling temperatures with top-k disabled and top-p = 1.0. On Qwen, CADI (T = 0.85) achieves higher Distinct₁₀ than Base (T = 0.8), with 5.35 vs. 4.04 at comparable Quality₁₀ (7.593 vs. 7.570), alongside a Utility₁₀ gain of 0.738. Similarly, on Llama, CADI (T = 0.8) achieves 6.73 Distinct₁₀ compared with Base (T = 1.0) at 6.00, with comparable Quality₁₀ (7.497 vs. 7.456) and a Utility₁₀ gain of 0.612.

These results show that CADI provides a more favorable diversity–utility trade-off than the tested decoding adjustments alone, even without top-k truncation and under the same generation budget.

Q2. Comparison with Prior Calibration and Stage 2 Distillation Controls

Thank you for this valuable suggestion. We provide additional comparisons to clarify the contributions of both stages, together with statistical analysis of Stage 2's incremental gains.

(1) Comparison with Prior Calibration. The closest related calibration approach is [1], which encourages a uniform distribution over valid candidates. To isolate the effect of our base-anchored target, we compare CADI Stage 1 with a uniform-target variant (α = 0), while retaining the same token-level KL regularization and training settings.

BackboneMethodDistinct₁₀ ↑Quality₁₀ ↑Utility₁₀ ↑Heterogeneity ↑HumanEval ↑MMLU-Pro ↑GSM8K ↑
Qwen2.5-7BUniform Target6.856.604.2026.0561.0040.0079.00
CADI Stage 15.417.664.7119.1065.8543.2785.60
Llama-3.1-8BUniform Target7.966.255.1528.9557.0033.0080.40
CADI Stage 17.037.435.6023.1060.9837.2983.55

Although uniform-target calibration achieves higher raw diversity, it substantially reduces response quality and general capabilities. In contrast, CADI Stage 1 improves Quality₁₀ by 1.06 and 1.18 points and Utility₁₀ by 0.51 and 0.45 on Qwen and Llama, respectively, while consistently preserving stronger general capabilities. These results support our base-anchored calibration design, which balances semantic coverage with response quality and capability preservation.

(2) Alternative Distillation Controls. We further compare Full CADI with two alternative strategies using the same Stage 1 initialization and accepted training trajectories, with comparable optimization budgets and identical evaluation protocols.

  • Hard-target SFT: Applies cross-entropy supervision to accepted response tokens.
  • Fixed-teacher context distillation: Uses the frozen Stage 1 model with the original query-specific guidance to provide soft token targets, retaining the same token-level KL objective as Full CADI.
MethodLlama Δ Distinct₁₀Llama Δ Utility₁₀Qwen Δ Distinct₁₀Qwen Δ Utility₁₀
Full CADI+0.18+0.13+0.15+0.18
Hard-target SFT+0.12+0.03+0.09−0.14
Fixed-teacher distillation+0.09+0.05+0.08+0.04

All differences are relative to Stage 1. Hard-target SFT recovers 60–67% of Full CADI's Distinct₁₀ gains but substantially less of its Utility₁₀ improvements, even reducing Utility₁₀ on Qwen. Fixed-teacher distillation recovers 50–53% of the Distinct₁₀ gains and 22–38% of the Utility₁₀ gains. These results suggest that simpler distillation strategies capture part of the improvement but do not recover the full diversity–utility gains of CADI.

(3) Statistical Reliability of Stage 2. We additionally compute 95% confidence intervals for Stage 2's incremental gains using 10,000 prompt-paired bootstrap replicates.

MetricLlama Δ [95% CI]Qwen Δ [95% CI]
Distinct₁₀+0.18 [+0.104, +0.370]+0.15 [+0.083, +0.358]
Utility₁₀+0.13 [+0.069, +0.286]+0.18 [+0.094, +0.364]
Heterogeneity+1.30 [+1.062, +1.538]+0.30 [+0.141, +0.459]

All six 95% confidence intervals exclude zero, supporting the incremental diversity and utility gains of Stage 2. Furthermore, the Appendix C.3 in our revised manuscript shows that Stage 2 improves both Distinct₁₀ and Utility₁₀ across all 12 backbone–decoding combinations. These results indicate that Stage 2's benefits persist across different decoding conditions rather than being limited to the default setting.

[1] Zhang, Yiming, et al. "Forcing Diffuse Distributions out of Language Models." Conference on Language Modeling (COLM), 2024.


Thank you again for your time and thoughtful consideration. We would be happy to address any remaining concerns or provide further clarification. We sincerely hope that our responses help resolve your concerns and encourage you to reconsider your assessment.

Thanks for the additional experiments, which address most of my concerns. I’m raising my score to 3, though I still find the novelty somewhat limited and the gains from Stage 2 modest.

We sincerely thank you for carefully reviewing our additional experiments and reconsidering your assessment. We acknowledge that Stage 1 accounts for most of the diversity gains, while Stage 2 serves as a complementary refinement, providing modest but statistically supported improvements in diversity and utility. We will further clarify this distinction and the respective contributions of both stages in our revised manuscript.

Once again, we sincerely appreciate your time, thoughtful feedback, and constructive engagement throughout the discussion!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.