- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces CHORD, an effective metric for evaluating open-ended generation that makes distributional evaluation sensitive to global coherence. The method is straightforward: place each passage in a specific prompt, extract the frozen LLM's last-token hidden state from a late layer, and compute RBF-MMD between the generated and human-reference corpora. The paper then null-standardizes the distance and tests whether it distinguishes harmful coherence perturbations from matched benign paraphrases. The authors create nine types of perturbations, spanning relation, discourse, structural, and mixture errors. They show that CHORD detects all nine at high severity, whereas standard likelihood-based and generic distributional metrics miss some, particularly the more difficult relation perturbations. The analysis also finds that the representation matters more than the distributional distance, and that CHORD aligns with human judgments in unconditional generation on OpenWebText.
Critical strengths
- The paper is well-written and easy to follow, with extensive experiments demonstrating CHORD’s effectiveness and examining why it works. These include comparisons across backbone models and distributional metrics, along with detailed ablation studies of prompting strategies, hidden-state extraction methods, and representation choices. The paper also evaluates CHORD on open-ended text generation using different generative models and incorporates human judgments to assess the proposed metric’s robustness.
Critical weaknesses
- The novelty of the proposed method is limited, as it primarily combines existing techniques: a coherence-oriented prompt adapted from PromptEOL, hidden-state extraction from a pretrained Qwen3.5-27B model, and the established RBF-MMD distance. As shown in Figure 3(a), the Qwen-based representation performs similarly across different distance metrics, suggesting that the representation itself accounts for most of the improvement. Although Figure 4(b) supports the effectiveness of coherence prompting, Table 8 shows that alternative approaches using the same backbone, including direct LLM judging and G-Eval, can also distinguish all nine perturbation types from benign paraphrases at the highest severity level. CHORD yields stronger statistical responses; however, the authors do not sufficiently demonstrate whether this translates into meaningful practical advantages.
- CHORD uses Qwen3.5-27B (or other modern models), while Table 1, the main results table, compares it with methods built on much older and smaller models, such as GPT-2 and BERT. Given the substantial differences in backbone capabilities, these improvements are not entirely surprising. Although Appendix E includes stronger, modern embedding baselines, the comparison still does not fully isolate the proposed methodology’s contribution. More controlled comparisons that use the same backbone across different evaluation methods would be more convincing.
Questions for authors
-
The evaluation relies largely on LLM-generated or rule-based perturbations, such as word shuffling, repetition, and inserting unrelated sentences, which create relatively obvious errors. These perturbations may not capture the subtle coherence failures found in real-world text. Evaluating naturally occurring or human-constructed coherence errors would offer stronger evidence of generalization.
-
The experiments in Table 2 and Figure 5 primarily assess GPT-2 variants and relatively small diffusion- or flow-based generators. It remains unclear whether CHORD can reliably distinguish outputs from stronger LLMs, where coherence differences are likely subtler. Moreover, because CHORD uses Qwen3.5-27B as its evaluator backbone, it would be valuable to test whether the metric systematically favors or penalizes text generated by models from the same family. These evaluations would clarify CHORD's practical robustness.
-
Another minor concern is that Figure 3(b) could be clearer. Several curves and their uncertainty bands overlap substantially, especially those for Energy Distance and RBF-MMD. Making the curves and uncertainty regions easier to distinguish would improve readability.
7 comments
Response to Reviewer ZQRR (2/4)
W1 (continued). Comparison with LLM judges
Regarding comparison with LLM-judge, CHORD and the same-backbone judge show comparable coherence-detection performance in the reported comparisons. Both detect all nine perturbation types at the highest severity (Table 13). In the new long-document evaluation across multiple severities, both detect 36/40 conditions at 1K and 35/40 at 2K (Table 9).
CHORD nevertheless offers four concrete advantages:
-
Distributional information beyond coherence ratings. Two corpora can receive similar per-text coherence ratings while differing substantially in topic composition or style. CHORD captures these differences while retaining sensitivity to additional coherence damage. In Appendix J, its median null-standardized responses to topic and cross-outlet shifts are 82 and 86, compared with 4.6 and 5.5 for the direct coherence judge. CHORD also detects added contradiction, broken transition, topic drift, and sentence permutation in at least two-thirds of the 22 corpus pairs (Table 21). This broader distributional objective is central to its design.
-
Lower inference cost. With all methods served by the same vLLM engine on a single A100-80GB GPU, scoring 100 passages with the 27B backbone at concurrency 32 takes 6.4 seconds for CHORD, compared with 12.9 for G-Eval and 16.0 for the direct judge. Under sequential requests, the times are 9.7, 23.0, and 154.8 seconds, respectively. CHORD is the fastest method at every evaluated model size and concurrency level (Appendix G.3, Table 15).
-
Effective distilled evaluators. Both the 2B and 0.8B distilled CHORD encoders retain detection of all nine perturbation types at the highest severity; direct judging and G-Eval with the corresponding small models each detect only five. At concurrency 32, the CHORD students score 100 passages in 0.72 and 0.46 seconds, respectively.
-
Larger selectivity relative to reference sampling variation. With the same 27B backbone, CHORD achieves larger selectivity Δz = z_harm − z_benign across all nine perturbation types. For contradiction, the selectivity is 78.3 for CHORD, 15.9 for the direct judge, and 9.3 for G-Eval (Table 13). This quantifies stronger separation of coherence damage from benign rewriting, relative to each metric’s reference-null variation, alongside the direct detection comparisons above.
We have added a dedicated “Relation to LLM judging” paragraph in Section 5 explaining these evaluation goals and practical advantages.
W2. Controlled comparisons using the same backbone
Concern: Comparisons with older backbones do not fully isolate the contribution of CHORD’s methodology.
The original paper already includes controlled ablations separating backbone, extraction, and distance. Specifically:
- Figure 3(a), Section 4.3.1: crosses four representations with four distributional distances, holding the representation fixed when comparing distances.
- Figure 4(b), Section 4.3.2: compares extraction methods within Qwen3.5-27B and GPT-2-large, holding the backbone and distance fixed.
- Appendix H: examines target attributes, prompt formats, wording, and extraction methods.
- Appendix G: compares hidden-state features with token-space measures and direct judging using the same Qwen3.5-27B backbone.
We have additionally extended the controlled extraction comparison to four modern backbones (Appendix H.1, Table 19). Within each backbone, we hold the extraction layer, RBF-MMD distance, and evaluation protocol fixed, changing only the extraction method.
The table below reports detected relation and discourse conditions out of 12:
| Backbone | Coherence-prompted last token | Unprompted last token | Unprompted mean pooling |
|---|---|---|---|
| Qwen3.5-27B | 10 | 1 | 0 |
| Qwen3.8-27B | 8 | 1 | 0 |
| Gemma-3-27B | 9 | 0 | 0 |
| Mistral-Small-3.2-24B | 9 | 3 | 0 |
Table 19 also applies MAUVE’s distance to the same unprompted representations and computes generative perplexity with the same backbone. These configurations detect none of the 12 relation or discourse conditions on any of the four backbones, while still detecting many structural and mixture conditions (generative perplexity: 23–24 of 40 overall).
These comparisons demonstrate that access to a modern backbone alone does not account for CHORD’s sensitivity: the extraction method determines whether the relevant coherence distinctions become detectable.
If the reviewer has a further controlled comparison in mind, could they specify which component or confounding factor remains unresolved by these experiments? We would welcome that clarification.
Response to Reviewer ZQRR (3/4)
Q1. Naturally occurring and subtler coherence failures
Question: Can CHORD detect coherence failures beyond obvious synthetic perturbations?
The controlled suite includes relation-level errors such as contradiction and causal reversal, designed to preserve local fluency while changing relationships across sentences. We have added two complementary evaluations.
Human validation (Appendix C, Table 7). Two annotators independently evaluated 100 passages: 20 from each of the four LLM-edited perturbation types and 20 benign paraphrases. Across harmful conditions, 149 of 160 judgments (93%) confirmed the intended error; 38 of 40 judgments (95%) confirmed meaning preservation for benign paraphrases. Agreement on validity was 97% (Cohen’s κ = 0.75).
Naturally occurring model responses (Appendix D.2, Table 10). We evaluate HelpSteer2[1] responses with human coherence ratings, without injecting errors. Low-coherence responses (ratings ≤2) and medium-coherence responses (rating 3) are paired with rating-4 responses to the same prompts, with a response-length ratio of at most two. Each group contains 381 pairs.
CHORD detects both groups relative to their paired high-coherence controls. Unprompted mean pooling from the same backbone detects only the low-coherence group; MAUVE, generative perplexity, and unigram entropy detect neither. Detecting the medium-coherence group provides evidence that CHORD captures subtler human-rated differences in naturally generated responses.
Q2. Stronger generators and potential same-family bias
Question: Does CHORD distinguish outputs from stronger generators, and does the Qwen evaluator favor or penalize Qwen-generated text?
We have added an evaluation of 21 autoregressive and diffusion generators, including models up to 27B parameters, and a comparison across three evaluator families (Appendix M). Each generator continues the same 128-token DCLM [2] prefixes. We evaluate 800-token continuations over ten non-overlapping folds of 500 matched prefixes.
Stronger generators (Appendix M.2, Table 26). CHORD distinguishes held-out human continuations from those of the 24–27B generators, and larger models within each tested autoregressive family are closer to the human reference. The comparison also uses Qwen3.5-27B for all three metrics (CHORD, MAUVE, and generative perplexity) so the evaluators share the same backbone.
The results expose practical differences between metrics. Generative perplexity ranks Dream-7B third among the 21 generators, despite substantial repetitive degeneration: 256 of its 400 lowest-perplexity continuations have distinct-3 scores below the first percentile of human continuations. CHORD ranks Dream-7B 14th, penalizing these outputs. The expanded evaluation therefore tests CHORD on modern generators and longer outputs where favorable perplexity can still accompany poor generation quality.
Encoder–generator family overlap (Appendix M.1, Table 25). We score ten generators from the Qwen, Gemma, Mistral, and Llama families with Qwen3.5-27B, Gemma-3-27B, and Mistral-Small-3.2-24B encoders. Qwen and Gemma produce identical rankings, and Mistral agrees closely with both (Spearman ρ = 0.93).
In particular, the Qwen evaluator ranks Gemma-3-27B and Mistral-Small-3.1-24B ahead of Qwen3.5-27B, exactly as the Gemma evaluator does. Qwen3.5-27B ranks third under both encoders and fourth under Mistral. These results provide no evidence of systematic favoritism toward Qwen generators by the default Qwen evaluator and show substantial ranking consistency across evaluator families.
Reference
[1] Wang et al. HelpSteer2: Open-source dataset for training top-performing reward models, NeurIPS 2024
[2] Li et al. DataComp-LM: In search of the next generation of training sets for language models, NeurIPS 2024 (Datasets and Benchmarks Track)
Response to Reviewer ZQRR (1/4)
We thank the reviewer for recognizing CHORD’s effectiveness and the extensive analysis of its representations. We have clarified the contribution and practical advantages, added further controlled comparisons using the same backbones, evaluated naturally occurring coherence differences and stronger generators, and examined encoder–generator family overlap. We address each concern below.
W1. Contribution and practical advantages over LLM judges
Concern: CHORD combines established components, and its practical advantages over same-backbone judging are unclear.
Our contribution is to make distributional evaluation of open-ended generation sensitive to global coherence. In tasks such as unconditional generation and prefix continuation, evaluating a generator requires considering both the quality of individual outputs and how well their distribution matches human text. Generative perplexity provides an incomplete account of this objective: predictable, repetitive outputs can receive favorable scores. MAUVE addresses the broader objective by comparing generated and human-text distributions in an embedding space, capturing differences that individual quality scores alone do not characterize.
However, this distributional comparison can detect only the distinctions preserved by its representation. Global coherence is a consequential blind spot in commonly used feature spaces. Our experiments show that MAUVE’s GPT-2 features can respond similarly to contradiction-inducing edits and meaning-preserving paraphrases. The metric therefore fails to distinguish coherence degradation from benign textual variation. CHORD addresses this gap through coherence-oriented representation extraction, evaluated using controlled perturbations, matched benign controls, and human judgments.
The reviewer’s observation that CHORD performs well across several distances supports this contribution: the improvement comes primarily from making coherence distinctions visible in the representation used for distributional comparison. Figure 3(a) demonstrates this directly: GPT-2 features with RBF-MMD detect only three perturbation types, whereas coherence-oriented Qwen features with MAUVE’s distance detect all nine. The same-backbone comparisons below further isolate the contribution of extraction from backbone capability.
We explicitly credit PromptEOL and the established distributional distances on which we build. The contribution of this metric is to identify a missing evaluation capability and provide a validated way to supply it. Its simple construction enables that capability without requiring a new encoder architecture or divergence function.
(continued in the next comment)