acceptodds
Under review as a conference paper at ICLR 2027

When Does Conditional Vendi Measure Posterior Diversity?

Abstract

Conditional generation requires separating diversity within a condition from variation across conditions. We characterise the discrepancy between averaged conditional kernel entropy and joint-minus-conditioning entropy. For a finite condition catalogue, we obtain an attainable certificate determined by the conditioning kernel, an exact equality criterion, and sharp recovery limits when only the aggregate score is retained. Global spectral truncation creates a second obstruction: even with orthogonal condition features, its rank budget can erase or reverse conditional diversity. A conditional tail decomposition retains an explicit approximation bound. Exact controls isolate these mechanisms, and an audit of 1,200 released images finds 31 native-generator ranking reversals across 112 kernel and subset specifications. The initial correction certifies none of these reversals, while a post hoc refinement certifies all on the same banks. A prospective Qwen3 study with 4,096 stochastic outputs certifies a finite-bank temperature ordering, with no native ranking reversal. Population entropy rankings remain unresolved. The analysis identifies which conclusions aggregate scores support and gives an evaluation protocol based on direct grouping, explicit compression error and sampling uncertainty.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Conditional Vendi Score, conditional diversity, kernel entropy, generative model evaluation, posterior diversity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_MJLeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines when a joint-minus-conditioning kernel entropy can be interpreted as average within-condition diversity. It separates three sources of discrepancy: overlap in the conditioning representation, global spectral truncation, and finite-sample estimation. The main results include a sharp conditioning certificate, an exact equality characterization, recovery limits from aggregate scores, and counterexamples showing that global truncation can destroy the intended conditional interpretation.

Critical strengths
  1. The paper proves that the joint-minus-conditioning (“conditional Vendi”) score does not generally equal average conditional entropy when condition features overlap, and gives a computable bound C(p,L) on the discrepancy.

  2. The paper identifies global spectral truncation as an independent failure mode: even with orthogonal condition labels, a global top- tt budget can erase conditional entropy. The rank-budget law and finite Gaussian counterexample are particularly useful because they expose when truncated aggregation can be negative or misleading, and the blockwise tail-aggregation remedy preserves the conditional decomposition with explicit error accounting.

Critical weaknesses
  1. The central certificate in Theorem 3.1 follows from data processing and entropy-of-mixing inequalities, while the blockwise tail construction uses the standard entropy grouping identity. These are useful applications, but their inclusion does not by itself establish a substantial theoretical advance. In my view, the more distinctive contributions are the quantitative overlap obstruction, the equality characterization, the aggregate-score recovery limit, and the truncation counterexample. I would encourage the authors to organize the theoretical narrative around these results and explain more precisely what they establish beyond existing kernel-entropy and quantum-information analyses.

  2. The paper states that attainability of the conditioning bound and the minimax recovery lower bound rely on output ensembles that can realize mutually orthogonal states. However, a fixed CLIP, DINO, or other feature map generally restricts the admissible covariance operators. It remains unclear how closely realistic generators can approach the unrestricted worst case. A systematic comparison of the actual discrepancy with the certificate. For example, through the ratio ((G-H_{\mathrm{post}})/C(p,L))—would help assess its practical tightness. A stronger theoretical extension would characterize the gap under restrictions induced by the output representation or derive tighter certificates using additional output information.

  3. In the Qwen confirmation study, the empirical entropy difference is approximately (0.110) nats, whereas the reported population difference interval is ([-2.423,2.642]). This is an informative negative result, but it leaves a substantial gap between finite-bank certification and reliable comparison of the underlying generators. The rank-based bias allowance alone is approximately (2.195) nats for (r=512) and (n=64), far exceeding the observed difference. The rare-tail impossibility result does not fully explain this limitation: it concerns a class without a common rank or tail restriction, whereas the linear-feature experiments have a known finite-dimensional representation. It would be valuable to establish whether the unresolved rankings primarily reflect insufficient data or conservatism in the analysis.

  4. The 31 reversals among 112 GenEval specifications are correlated sensitivity analyses on the same output banks, rather than independent estimates of a failure rate. The initial tolerance certifies none of these reversals. The subsequent tighter tolerances demonstrate improved identification on the observed banks but do not establish prospective reliability. The Qwen confirmation study provides prospective evidence of reduced distortion, although the native temperature ranking is already correct. The fresh-image selection study is useful, but the held-out gain is approximately (0.0078) nats, direct grouped scoring achieves the same gain, and the population intervals remain inconclusive. A more decisive experiment would compare selection errors, abstention rates, and held-out gains across independent output banks under a fixed evaluation budget.

Questions for authors

The term “posterior diversity” should be used cautiously. Equality between the two entropy targets establishes agreement about a representation-dependent diversity functional. It does not establish posterior fidelity, calibration, or useful sample diversity. The manuscript acknowledges this distinction, but most experiments evaluate generator distributions without a known posterior reference. The title and application claims should remain consistent with that scope.

3 comments
AuthorsOpenReview ↗

Response to Reviewer MJLe

Your distinction between standard ingredients and the more specific results has guided the revision. The introduction now centres the contribution on the quantitative overlap obstruction, equality characterisation, recovery limits from an aggregate score, and the separate truncation obstruction. We identify data processing, entropy of mixing and tail aggregation as established tools used in this analysis.

1. Fixed representations and realised tightness. We agree that unrestricted attainability does not establish sharpness for every fixed encoder. We have now recomputed (G−H^group)/C(G-\hat H_{\mathrm{group}})/C across all 336 score rows of the original GenEval grid. These retain all three systems, both representations, both output kernels, seven prompt subsets and four conditioning kernels. Appendix K.1 (Table 17) reports absolute gaps and certificates alongside the ratios.

Output representation and kernelObserved gap / boundAbsolute gap, nats
CLIP, linear0.403 to 0.5540.030 to 1.648
CLIP, Gaussian0.393 to 0.5220.029 to 1.607
DINOv2, linear0.819 to 0.9800.057 to 3.176
DINOv2, Gaussian0.426 to 0.5510.030 to 1.679

The certificate ranges from 0.064 to 3.799 nats in each stratum. DINOv2-linear ratios on the full 100-prompt bank range from 0.819 to 0.879, so the high ratios are not confined to small subsets. These descriptive results show that substantial fractions of the bound occur with the fixed empirical feature maps. They concern score magnitude, not improved ranking certification. Similar gaps can cancel in a score difference. The configurations share outputs and are not independent replications. A representation-specific sharpness theorem or tighter output-dependent certificate remains beyond the present results.

2. Population uncertainty and the role of the bias allowance. You are correct that the unrestricted rare-tail construction does not explain the finite-dimensional Qwen result. We have separated these points in the revision. At n=64n=64 and r=512r=512, the bias allowance is 2.195487 nats and the concentration radius is 0.168481 per temperature. The former contributes 86.7% of the difference-interval half-width. This identifies the dominant term in the interval width, without measuring the actual population bias or showing that more data are unnecessary.

The new retrospective Qwen curves use the first 1, 2, 4, 8, 16, 32 and 64 stored draws per prompt. The observed difference stays positive from 2 to 64, but both entropy estimates still increase from 32 to 64 draws, by 0.033791 and 0.050260 nats. No population interval resolves the ordering. The full-sample Vendi ratio is 1.116 with interval [0.089, 14.043]. Thus the new analysis locates the dominant allowance and shows sample-size sensitivity. It does not determine whether a sharper analysis would certify this comparison at the existing budget.

3. Decision reliability and analysis history. We retain the original 0-of-31 certification result and label the tighter GenEval tolerances as post hoc. Replaying the unmodified scores reproduces all 31 reversals among 112 correlated configurations, but this supplies no independent failure-rate estimate. The prospective Qwen study preserves the native ordering. The fresh-image illustration concerns one aggregate decision, a held-out gain of 0.007782 nats, and the same gain under direct scoring. It does not establish an advantage over that baseline or repeated-study reliability. We have not added the proposed independent-bank selection experiment, so selection errors, abstention rates and gains across independent studies remain unresolved.

4. Posterior interpretation. The abstract and setup now make explicit that posterior diversity is an application when the sampled conditional law is the specified Bayesian posterior. The generator experiments assess representation-dependent conditional diversity. Agreement between the entropy targets establishes neither posterior fidelity nor calibration or output quality. These scope statements also govern the revised practical recommendations.

Reviewer_MJLeOpenReview ↗

response to authors

The paper establishes a solid theoretical account of conditioning distortion, recovery limits, and the global truncation problem for finite condition catalogues. But there are still some key concerns remain unresolved: population entropy rankings are still open, the empirical evidence is narrow and heavily controlled, and the finite-sample intervals are often too wide to support the very comparisons the paper is trying to justify. The counterexample to Jalali et al. may be important, but its assumptions and scope still need careful check.

AuthorsOpenReview ↗

Scope of the truncation counterexample

Thank you for distinguishing the theoretical result from the empirical evidence. We agree that our experiments do not establish population entropy rankings or tell us how often rankings change across generators.

On the counterexample, Section 4 and Example C.3 address a specific claim: Eq. (19) of Theorem 7 in Appendix B of Jalali et al. The prompt is equally likely to be 0 or 4, and the output is an independent draw from the same two points. Both kernels are Gaussian with bandwidth one, both feature maps have unit norm, and each prompt component has zero variance. With two eigenvalues retained and the omitted mass redistributed equally, the discrepancy is 0.693147 nats, while the stated upper bound is 0.026417 nats.

Independence makes the distinction particularly clear: before truncation, the joint-minus-prompt entropy equals the within-prompt entropy exactly. The discrepancy arises because retaining two eigenvalues of the joint operator is different from retaining two in each conditional operator. The example contradicts the truncated aggregation bound as stated; it does not contradict the untruncated score or establish how often truncation matters in practice.

Reviewer_iP3FOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper analyzes the difference between the conditional Vendi score, a metric intended to measure generated output diversity while accounting for differences among prompts, and an alternative diversity metric, which averages within-prompt entropy. The authors provide bounds on this discrepancy based on prompt similarities and weights. The paper also analyzes the effects of computational approximation and small sample sizes. Experiments to test the practical distinction between the two metrics include those on image generators and Qwen3-0.6B; some of the image model comparisons show differences in metric ordering, while Qwen3 rankings does not. The alternative diversity metric is recommended when repeated outputs are available.

Critical strengths

The authors distinguish the effects of various evaluation conditions, such as similarity between prompts, computational approximation, and small sample sizes. This helps clarify when the two metrics (conditional Vendi score and averaged within-prompt entropy) can disagree.

Critical weaknesses
  1. The primary contribution is an analysis of an existing metric, the conditional Vendi score of Jalali et al. (2026). It is unclear what aspects of the analyses provide novel insights beyond existing work -- this is in particular to the concluding recommendation, which is an established practice of measuring diversity within each prompt and aggregating scores (as in e.g., Nguyen & Dieng (2025) for averaged conditional Vendi entropy). The fact that conditional Vendi does not necessarily equal the latter metric is recognized in existing work, as the paper acknowledges in the appendix with reference to Bach (2022).

  2. The logical flow of the manuscript is difficult to follow at times. The practical importance of the metrics focused on and the analysis of their discrepancy are largely discussed in later sections (5 & 8). The "operational consequence" in section 5, however, is difficult to parse through. I recommend putting more leading motivation both within the abstract and the introduction, for conditional Vendi scores and similar prompt-based diversity metrics as well as discussions on implications for practitioners. Given that the paper's primary contribution is analysis, the empirical breadth would benefit from being expanded upon, as the Qwen3 0.6B experiments show no difference between the two metrics' ordering of the temperature settings.

Minor: Some of the prose can be simplified for better clarity, e.g., "finite-bank," "no native ranking reversal," "certificate abstains."

Questions for authors
  1. To my understanding, a core motivation behind conditional Vendi is for evaluations where repeated outputs are not available for prompts. The recommendation that an alternate prompt-diversity metric be preferable when repeated outputs are available is not contradicted (and explicitly mentioned) in Jalali et al. (2026). Are there any non-intuitive setting where the paper's analysis results in a different recommendation?
  2. Following the above, which practical recommendations are new relative to prior work, e.g., from Jalali et al. (2026) and Nguyen & Dieng (2025)?
  3. Given that the Qwen3 0.6B experiments did not show any difference in metric ordering across temperature settings, what are the implications for when/how often the discrepancy does indeed affect practical evaluation decisions?
2 comments
AuthorsOpenReview ↗

Response to Reviewer iP3F

We agree that the contribution should be distinguished more clearly from established within-prompt scoring and from the known failure of a general kernel-entropy chain rule. The revision introduces the evaluation question before the theory and identifies direct grouped entropy as established practice, including Nguyen and Dieng (2025). It also discusses Bach (2022) and the repeated-output setting recognised by Jalali et al. (2026).

1. What the analysis adds. The contribution is a quantitative characterisation of the information retained by the aggregate score. Section 3 gives the equality criterion through the overlap graph of condition features, a quantitative obstruction when overlapping conditions have different output states, and an aggregate-score recovery limit. For mm conditions with fixed (p,L)(p,L), arbitrary conditional density operators in dimension at least mm, and only (G,p,L)(G,p,L) retained, the optimal worst-case error is C(p,L)/2C(p,L)/2. A fixed encoder can restrict this admissible class. This limits recovery from the scalar score, not methods retaining conditional samples or Gram blocks. We identify the standard quantum-information ingredients explicitly.

2. A practical consequence beyond recommending repeats. Orthogonal condition labels eliminate the conditioning discrepancy, but they do not protect against global spectral truncation. The rank-budget law in Section 4 gives

\qquad H_{\mathrm{post}}=\log r.$$ For example, with 100 conditions and a shared uniform conditional state of rank 10, keeping 100 global spectral components yields $G_t=0$, although ten components per conditional block retain the full target $\log10$. This is an analytical example, not an additional experiment. It shows why a global spectral budget can remove within-condition variation even when the label representation is exact. The resulting recommendation is to preserve the conditional block decomposition and account for omitted tail entropy when approximating that target. The finite Gaussian example also identifies a concrete failure of the published truncated interpretation. The paper therefore analyses when existing evaluation and compression procedures support their intended interpretation. It does not claim to introduce direct grouping or to improve on it when the same repeated outputs are available. For data without repeats, a smooth conditional score remains a well-defined functional, but the within-prompt population interpretation requires additional justification. **3. What the Qwen result implies.** The unchanged temperature ordering limits the empirical conclusion. This study demonstrates a score discrepancy without a selection reversal. For example, in the confirmation study the unmodified score gives 0.850 nats for the deterministic greedy outputs, whose exact-prompt target is zero. Retrospective sample-size curves now show the same observed temperature ordering from 2 to 64 outputs per prompt, with population intervals remaining inconclusive. The 31 GenEval reversals concern correlated configurations of the same images. Together with the Qwen result, they show that ranking changes can occur and are not inevitable. They do not estimate how often such changes occur in independent evaluations. We retain that limitation and the original correction's 0-of-31 certification outcome. The revised discussion separates observed score distortion, ranking changes and population comparisons, and replaces shorthand such as “certificate abstains” with a direct statement that the interval does not resolve the ordering.

I thank the authors for their clarifications and revisions. I maintain my score as it is still unclear to me the practicality of understanding aggregate vs within-prompt entropy, given the mixed empirical results.

Reviewer_m7ckOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at diversity evaluation for conditional generative models from a different angle, by studying whether joint-minus-conditioning kernel entropy equals the expected entropy within each condition. The authors show that these quantities do not generally agree when condition features overlap, and provide a sharp bound on their difference, conditions for equality, and limits on recovering the expected within-condition entropy from the aggregate score alone. They also study spectral truncation and show that truncating the joint spectrum is not equivalent to truncating each conditional spectrum separately and taking the weighted average. Synthetic examples and image- and text-generation experiments illustrate how these choices can affect diversity scores and model rankings.

Critical strengths
  • The authors look at the problem of diversity evaluation for conditional models from a different angle, by asking whether the joint-minus-conditioning score captures the expected within-condition entropy. The distinction between variation within a fixed prompt and variation across prompts is useful, and the paper provides a clear theoretical characterization of when these quantities agree and how much they can differ.

  • The paper also studies the effect of global spectral truncation and shows that truncating the joint spectrum is not equivalent to truncating each conditional spectrum separately and taking the weighted average. The examples make this distinction clear, even when the condition features are orthogonal.

Critical weaknesses

The paper compares the matrix-based conditional entropy from quantum information theory, adopted by Jalali et al. (2026) from Sánchez Giraldo et al. (2014),

H_{\mathrm{CV}}=H(C_{X,T})-H(C_T), \tag{1}

with the average exact-prompt entropy promoted by the authors,

=\mathbb E_{t\sim P_T}\left[H(C_{X\mid T=t})\right]. \tag{2}$$ The quantitative bounds are useful, and the experimental protocols are documented. However, I have some concerns about the recommendation to treat the second quantity as the reference for evaluating prompt-aware diversity. My main concerns are as follows. **1. Treating Definition (2) as the reference for conditional diversity is debatable.** Even if the Vendi score is accepted as an unconditional diversity measure, it does not follow that averaging its logarithm over exact prompts is the right notion of conditional diversity. Consider a deterministic model A (e.g., an LLM with zero-temperature decoding) that produces **substantially different**, valid outputs for semantically equivalent prompts, and a collapsed model B that produces the same output for every prompt. The recommended score assigns $$H_{\mathrm{within}}(A) = H_{\mathrm{within}}(B) = 0, \quad Vendi_{\mathrm{within}}(A) = Vendi_{\mathrm{within}}(B) = 1.$$ I find this counterintuitive. Both models have zero randomness at any fixed prompt, but they behave completely differently across related prompts, and averaging at the exact-prompt level discards precisely this distinction. Pooling over a **given** semantic category $Z=q(T)$, i.e., $H_{\mathrm{pooled}}=\mathbb E_{z\sim P_Z}[H(C_{X\mid Z=z})]$, could distinguish the two models. However, the paper provides no principled rule for choosing the partition, thereby assuming away a central difficulty of the task: deciding which prompts should be treated as the same condition. Conditional-Vendi (Jalali et al., 2026) instead incorporates prompt similarity through the conditioning kernel. The grouping-based alternative does not avoid a comparable choice: exact-prompt grouping cannot distinguish models A and B, and pooling can do so only after deciding which prompts belong together. The paper should present this trade-off explicitly in the introduction rather than treating Definition (2) as the reference by default. **2. Definition (2) is a special case of Conditional-Vendi with a Dirac text kernel.** If we consider Definition (1) with the Dirac text kernel $$k_T(t,t')=\begin{cases}1 & \text{if } t=t',\\ 0 & \text{otherwise,}\end{cases}$$ which treats two prompts as similar only if they are identical, is exactly Definition (2). Conditional-Vendi (Jalali et al., 2026) generalizes this to the Gaussian text kernel $$k_T(t,t')=\exp\left(-\frac{\|\psi(t)-\psi(t')\|^2}{2\sigma^2}\right),$$ where $\psi$ is the text embedding. As $\sigma\to 0$, this kernel tends to the Dirac kernel, and Conditional-Vendi recovers the proposed metric. The two definitions are therefore not different notions of conditional diversity but members of the same family, and the gap in Theorem 3.1 reflects the choice of bandwidth. This raises the question of whether removing the bandwidth and using the special case is desirable. With a zero bandwidth, paraphrases are treated as unrelated prompts, which is exactly why models A and B above receive the same score. I would like the authors to justify why the limiting case is preferable to the general one, rather than treating the gap as a distortion of Conditional-Vendi. The fact that definition 2 is a special case deserves to be explicitly mentioned in the abstract or intro rather than the Appendix or in explanation of a theorem. **3. Convergence analysis of the recommended score.** Although the paper provides statistical guarantees, their practical informativeness is limited: even the prospective study with 4,096 stochastic outputs leaves the population ranking unresolved. Moreover, the recommended estimator requires repeated outputs at each exact prompt and is identically zero when only one output per prompt is available. The finite-sample intervals in Section 5 and Appendix D also require a known population rank bound, i.e., a bounded feature dimension. As the paper itself notes, this does not hold for the Gaussian kernel, yet Gaussian output kernels are used in the experiments. Ospanov & Farnia (2025, Figure 1) show that the Vendi score with a Gaussian kernel does not converge on ImageNet within the available sample sizes, and averaging the same order-one estimate over prompts, each with fewer samples, is unlikely to avoid this issue. I believe that checking the convergence of the recommended score on ImageNet with a Gaussian kernel on DINOv2 features, with class labels as conditions, as suggested in [2], is essential to verify the paper's claims numerically. Finally, results are reported in nats, while the Vendi score is the exponential of entropy, so additive errors become multiplicative. For example, the difference interval $[-2.42, 2.64]$ nats in Appendix H.3 allows the ratio of the two Vendi scores to lie anywhere between $0.09$ and $14.0$. **4. Related works.** The discussion of order-$\alpha$ Vendi scores should explicitly compare them with the related RKE score [1]. The order-two score $1/\mathrm{Tr}(\rho^2)$, described as inverse purity in Remark A.4 and Proposition D.3, is the RKE score introduced in [1], where its computation from kernel pairs and its finite-sample convergence were also studied. This work should be cited and discussed. Similarly, Cluster-Vendi, defined and evaluated in Jalali et al. (2026), averages per-cluster Vendi scores rather than per-cluster entropies, but it is directly relevant to the grouping-based alternative and should be discussed. Overall, I think this work has some merit, and I credit the authors for analyzing the relationship between these two definitions. However, I believe the concerns above need to be addressed before the manuscript is ready for publication at a top venue such as ICLR. --- [1] Jalali et al. An Information-Theoretic Evaluation of Generative Models in Learning Multi-modal Distributions. NeurIPS 2023. [2] Stein et al. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models. NeurIPS 2023.
Questions for authors
  • Why should the expected exact-prompt entropy be the preferred target for evaluating conditional diversity? Models A and B in my first concern both receive zero exact-prompt entropy, although they behave completely differently across semantically equivalent prompts. Should the variation of model A across closely related prompts be treated as distortion or as meaningful diversity?

  • Given that Definition (2) is the zero-bandwidth limit of Conditional-Vendi with a Gaussian text kernel (Theorem 3.1 and Proposition B.10), why is the limiting case preferable to a positive bandwidth?

  • How does the grouping-based alternative relate to Cluster-Vendi in Jalali et al. (2026), which averages per-cluster Vendi scores rather than per-cluster entropies?

  • How many outputs per prompt are needed for direct grouped entropy to provide a reliable population comparison? Given that the population rankings remain unresolved in your experiments, can you show how the estimates and rankings change as the number of outputs per prompt increases, and report the resulting intervals on the Vendi scale rather than in nats? What do you recommend when only one output per prompt is available, as is common in practice?

  • Does the recommended score converge on ImageNet with a Gaussian kernel on DINOv2 features (e.g., with class labels as conditions)? Can you show how the estimate changes as the number of samples per class increases, for several kernel bandwidths? Since the known rank bound does not hold for the Gaussian kernel, which statistical guarantee applies to the Gaussian-kernel settings used in your experiments?

3 comments
AuthorsOpenReview ↗

Response to Reviewer m7ck

We agree that exact-prompt entropy is not universally preferable. The revised introduction distinguishes that target from semantic-group diversity and states the Dirac-kernel equivalence explicitly.

1. Exact prompts, semantic groups and bandwidth. Your models A and B both have zero entropy at a fixed prompt. This measures variation with the complete prompt held fixed, but does not distinguish behaviour across paraphrases. When that variation matters, one appropriate target is entropy within a declared semantic group, computed from its mixture of output laws. Choosing those groups is a modelling decision that our analysis does not resolve.

The exact-prompt target is indeed the Dirac-kernel special case. With condition states ρt\rho_t and weights ptp_t,

\qquad S(\rho_{XT}^{\mathrm{Dirac}})-H(p)=\sum_t p_tS(\rho_t).$$ For distinct embeddings in a fixed finite catalogue, a Gaussian prompt kernel also approaches this case as its bandwidth tends to zero. If embeddings coincide, the limit groups those collisions. We do not recommend zero bandwidth for every evaluation objective. These choices belong to the same kernel family but can answer different questions. Our discrepancy is relative to a specified target, without implying that paraphrase variation is undesirable or a positive-bandwidth score invalid. The equality characterisation and recovery limits specify when that score determines the chosen target. The abstract and practical recommendations now reflect this scope. **2. Cluster-Vendi and RKE.** Section 2 now discusses both. For the same groups, weights and output kernel, Cluster-Vendi averages scores, $\sum_zp_z e^{S(\rho_z)}$, whereas exponentiating averaged entropy gives $\exp(\sum_zp_zS(\rho_z))$, their weighted geometric mean. Jensen's inequality orders these aggregates for one system, but their rankings across systems can differ. Neither determines the appropriate grouping. We credit Nguyen and Dieng (2025) for averaged conditional Vendi entropy. We also credit Jalali, Li and Farnia (2023) for order-two RKE, its kernel-pair computation and finite-sample analysis. Their order-two RKE is $-\log\operatorname{Tr}(\rho^2)$, whose exponential, RKE-MC, is inverse purity. An order-two guarantee does not settle an order-one entropy ranking. **3. Outputs per prompt and the Vendi scale.** Appendix K.2 analyses the existing Qwen outputs at $n=1,2,4,8,16,32,64$, retaining all 32 fixed prompts and the first $n$ stored draws. This analysis is retrospective. The entropy differences, temperature 1.1 minus 0.7, are: | Outputs per prompt | 1 | 2 | 4 | 8 | 16 | 32 | 64 | |---|---:|---:|---:|---:|---:|---:|---:| | Difference, nats | 0 | .0318 | .0618 | .0749 | .0783 | .0932 | .1096 | Both entropy estimates still rise from 32 to 64 draws, so these curves do not establish convergence. At 64 draws, the ratio of exponentiated average entropies is 1.116, with the original 95% population ratio interval **[0.089, 14.043]**. Every tested size remains inconclusive. We distinguish pointwise intervals from intervals adjusted across all six sizes with $n\geq2$. Nested samples are not independent replications. There is no universal sufficient sample count independent of the kernel, distribution and comparison margin. With one output per exact prompt, empirical entropy is zero by construction, not evidence of a deterministic generator. For this population target, we recommend repeated draws or justified structural assumptions. Pooling changes the target. A smooth-kernel score can still be reported as its own functional, without claiming that it identifies exact-prompt population entropy. **4. Gaussian kernels and ImageNet.** The Qwen and fresh-image population intervals use normalised **linear** CLIP output features, with known rank at most 512. Gaussian prompt conditioning does not change that output-rank bound. For the general-population Gaussian output kernels used elsewhere, we do not apply the rank-512 guarantee. Their reported comparisons concern the observed empirical distributions. Theorem A.1 supplies pointwise consistency under its stated assumptions, but no practical uniform sample-size rate. We have not run the requested ImageNet/DINOv2 Gaussian-kernel experiment. The Qwen curves answer the question for the existing linear-feature study, while behaviour on ImageNet across Gaussian bandwidths remains untested. Neither averaging over conditions nor a finite empirical Gram rank supplies the missing population-rank guarantee.

I thank the authors for their detailed response. The revised introduction now states that the exact-prompt entropy is not universally preferable and that it is the Dirac-kernel special case of Conditional-Vendi, and Section 2 now discusses Cluster-Vendi and RKE. However, several of my concerns remain.

The exact-prompt average as the reference. The response states that the exact-prompt entropy is not universally preferable and that choosing semantic groups is a modelling decision that the analysis does not resolve. Hence, the exact-prompt average is one possible target rather than the posterior diversity that Conditional-Vendi is expected to measure, and I believe the title and the main claims of the paper, which treat it as the reference, are not supported.

The gap is not a distortion. Since Definition (2) is the Dirac-kernel special case of Conditional-Vendi, the gap between the two should not be treated as a distortion or an error. In fact, as stated after Theorem 3.1, the gap equals I(A:Y∣B)I(A:Y\mid B), i.e., the information that the outputs carry about the exact prompt beyond its kernel representation. To see why this matters, consider models A and B in my review: both have zero exact-prompt entropy, while with a Gaussian text kernel the gap is positive for model A and zero for model B. Therefore, the gap captures the variation across related prompts that the exact-prompt average misses, although it can also include prompt following when similar prompts differ in a meaningful detail. I believe the terms "conditioning distortion" and "distortion witness" should be revised accordingly.

Convergence and the number of outputs per prompt. The new results in Appendix K.2 confirm my concern: both entropy estimates still increase from 32 to 64 outputs per prompt, and the 95% interval for the ratio of the Vendi scores, [0.089, 14.04], is inconclusive at every tested size. The estimator is also zero with one output per prompt, which is common in practice.

Gaussian kernel on ImageNet. I specifically asked for the ImageNet experiment with a Gaussian kernel because the Gaussian kernel is infinite-dimensional, and the finite-sample guarantees in the paper, which require a known rank bound, do not hold in this case. The Qwen results use linear CLIP features with rank at most 512 and therefore do not answer this question. As the response states, the Gaussian-kernel comparisons concern only the observed empirical distributions, so they do not support conclusions about the population.

Reporting on the Vendi scale. Since the paper focuses on the Vendi score, which is the exponential of the entropy and is interpreted as the effective number of distinct outputs, I suggest that the authors report all numerical results on this scale rather than in nats. For example, the interval [−2.42, 2.64] nats corresponds to a ratio of the Vendi scores between 0.09 and 14.0, which is much easier to interpret.

Overall, the response clarifies the scope of the paper, but I believe the main claims are not supported by the analysis and the experiments. I therefore keep my score.

AuthorsOpenReview ↗

What the conditioning gap measures

Thank you for pressing this distinction. In your example, both models have a within-prompt Vendi score of 1: neither varies when the prompt is held fixed. Yet model A varies across paraphrases, and that variation can matter. We agree that “conditioning gap” is more accurate than “conditioning distortion.” The gap is an error only when the stated goal is to measure diversity at a fixed condition.

A more accurate title would be When Does Conditional Vendi Measure Within-Condition Diversity? A Bayesian inverse problem gives one reason to ask this question: for a fixed observation, how much do draws from its posterior vary? Our generator experiments do not establish posterior fidelity. Theorem 3.1 addresses the comparison itself: for a finite set of conditions, the untruncated Conditional Vendi score lies between the weighted geometric mean of the within-condition Vendi scores and that mean multiplied by exp⁡(C(p,L))\exp(C(p,L)).

On the Vendi scale, the observed Qwen score at temperature 1.1 was 1.116 times the score at 0.7. Both entropy estimates still rose from 32 to 64 outputs per prompt, and the 95% population ratio interval at 64 was [0.089, 14.04]. The data therefore leave the population ordering unresolved. With one output per prompt, the empirical within-prompt score is 1 by construction. We have not tested convergence on ImageNet with a Gaussian output kernel. The Qwen study uses linear features and does not answer that question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.