acceptodds
Under review as a conference paper at ICLR 2027

Coherence-Aware Distributional Evaluation of Open-Ended Text Generation

Abstract

Existing metrics for open-ended text generation measure likelihood, lexical diversity, or distributional similarity in generic representation space, yet they can miss fundamental dimensions of quality. A prominent blind spot is global coherence: a generated passage may be locally fluent while remaining globally contradictory, causally inconsistent, or topically disconnected. Such failures can still preserve the token-level and lexical statistics that existing metrics rely on. We identify representation as a central bottleneck in detecting these failures and introduce CHORD (Coherence-aware Hidden-state Open-generation Reference Distance), a coherence-sensitive distributional metric. CHORD encodes generated and human-written corpora in the hidden-state space of a frozen LLM using a coherence-eliciting prompt, and compares the resulting distributions using MMD with an RBF kernel. To validate that the metric responds to coherence degradation but not generic textual change, we construct a counterfactual evaluation suite that pairs graded coherence-degrading perturbations with meaning-preserving controls. CHORD selectively detects relation, discourse, structural, and mixture failures that perplexity, entropy, MAUVE, FBD, and MMD-based baselines either miss or cannot separate from benign rewriting. Factorial ablations show that representation is the primary source of coherence sensitivity, while RBF-MMD improves sample efficiency once the relevant distinctions become visible. Larger backbones capture finer-grained distinctions, but coherence prompting improves selectivity only when the backbone can follow the prompt. On unconditional generation and prefix continuation, CHORD yields model rankings that strongly align with human judgments of whether outputs make sense and appear human-written. Together, these results establish representation design as central to reliable distributional evaluation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Evaluation, Coherence, Open-ended Text Generation, Diffusion Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ZQRROpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CHORD, an effective metric for evaluating open-ended generation that makes distributional evaluation sensitive to global coherence. The method is straightforward: place each passage in a specific prompt, extract the frozen LLM's last-token hidden state from a late layer, and compute RBF-MMD between the generated and human-reference corpora. The paper then null-standardizes the distance and tests whether it distinguishes harmful coherence perturbations from matched benign paraphrases. The authors create nine types of perturbations, spanning relation, discourse, structural, and mixture errors. They show that CHORD detects all nine at high severity, whereas standard likelihood-based and generic distributional metrics miss some, particularly the more difficult relation perturbations. The analysis also finds that the representation matters more than the distributional distance, and that CHORD aligns with human judgments in unconditional generation on OpenWebText.

Critical strengths
  1. The paper is well-written and easy to follow, with extensive experiments demonstrating CHORD’s effectiveness and examining why it works. These include comparisons across backbone models and distributional metrics, along with detailed ablation studies of prompting strategies, hidden-state extraction methods, and representation choices. The paper also evaluates CHORD on open-ended text generation using different generative models and incorporates human judgments to assess the proposed metric’s robustness.
Critical weaknesses
  1. The novelty of the proposed method is limited, as it primarily combines existing techniques: a coherence-oriented prompt adapted from PromptEOL, hidden-state extraction from a pretrained Qwen3.5-27B model, and the established RBF-MMD distance. As shown in Figure 3(a), the Qwen-based representation performs similarly across different distance metrics, suggesting that the representation itself accounts for most of the improvement. Although Figure 4(b) supports the effectiveness of coherence prompting, Table 8 shows that alternative approaches using the same backbone, including direct LLM judging and G-Eval, can also distinguish all nine perturbation types from benign paraphrases at the highest severity level. CHORD yields stronger statistical responses; however, the authors do not sufficiently demonstrate whether this translates into meaningful practical advantages.
  2. CHORD uses Qwen3.5-27B (or other modern models), while Table 1, the main results table, compares it with methods built on much older and smaller models, such as GPT-2 and BERT. Given the substantial differences in backbone capabilities, these improvements are not entirely surprising. Although Appendix E includes stronger, modern embedding baselines, the comparison still does not fully isolate the proposed methodology’s contribution. More controlled comparisons that use the same backbone across different evaluation methods would be more convincing.
Questions for authors
  1. The evaluation relies largely on LLM-generated or rule-based perturbations, such as word shuffling, repetition, and inserting unrelated sentences, which create relatively obvious errors. These perturbations may not capture the subtle coherence failures found in real-world text. Evaluating naturally occurring or human-constructed coherence errors would offer stronger evidence of generalization.

  2. The experiments in Table 2 and Figure 5 primarily assess GPT-2 variants and relatively small diffusion- or flow-based generators. It remains unclear whether CHORD can reliably distinguish outputs from stronger LLMs, where coherence differences are likely subtler. Moreover, because CHORD uses Qwen3.5-27B as its evaluator backbone, it would be valuable to test whether the metric systematically favors or penalizes text generated by models from the same family. These evaluations would clarify CHORD's practical robustness.

  3. Another minor concern is that Figure 3(b) could be clearer. Several curves and their uncertainty bands overlap substantially, especially those for Energy Distance and RBF-MMD. Making the curves and uncertainty regions easier to distinguish would improve readability.

7 comments
AuthorsOpenReview ↗

Response to Reviewer ZQRR (1/4)

We thank the reviewer for recognizing CHORD’s effectiveness and the extensive analysis of its representations. We have clarified the contribution and practical advantages, added further controlled comparisons using the same backbones, evaluated naturally occurring coherence differences and stronger generators, and examined encoder–generator family overlap. We address each concern below.

W1. Contribution and practical advantages over LLM judges

Concern: CHORD combines established components, and its practical advantages over same-backbone judging are unclear.

Our contribution is to make distributional evaluation of open-ended generation sensitive to global coherence. In tasks such as unconditional generation and prefix continuation, evaluating a generator requires considering both the quality of individual outputs and how well their distribution matches human text. Generative perplexity provides an incomplete account of this objective: predictable, repetitive outputs can receive favorable scores. MAUVE addresses the broader objective by comparing generated and human-text distributions in an embedding space, capturing differences that individual quality scores alone do not characterize.

However, this distributional comparison can detect only the distinctions preserved by its representation. Global coherence is a consequential blind spot in commonly used feature spaces. Our experiments show that MAUVE’s GPT-2 features can respond similarly to contradiction-inducing edits and meaning-preserving paraphrases. The metric therefore fails to distinguish coherence degradation from benign textual variation. CHORD addresses this gap through coherence-oriented representation extraction, evaluated using controlled perturbations, matched benign controls, and human judgments.

The reviewer’s observation that CHORD performs well across several distances supports this contribution: the improvement comes primarily from making coherence distinctions visible in the representation used for distributional comparison. Figure 3(a) demonstrates this directly: GPT-2 features with RBF-MMD detect only three perturbation types, whereas coherence-oriented Qwen features with MAUVE’s distance detect all nine. The same-backbone comparisons below further isolate the contribution of extraction from backbone capability.

We explicitly credit PromptEOL and the established distributional distances on which we build. The contribution of this metric is to identify a missing evaluation capability and provide a validated way to supply it. Its simple construction enables that capability without requiring a new encoder architecture or divergence function.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer ZQRR (2/4)

W1 (continued). Comparison with LLM judges

Regarding comparison with LLM-judge, CHORD and the same-backbone judge show comparable coherence-detection performance in the reported comparisons. Both detect all nine perturbation types at the highest severity (Table 13). In the new long-document evaluation across multiple severities, both detect 36/40 conditions at 1K and 35/40 at 2K (Table 9).

CHORD nevertheless offers four concrete advantages:

  1. Distributional information beyond coherence ratings. Two corpora can receive similar per-text coherence ratings while differing substantially in topic composition or style. CHORD captures these differences while retaining sensitivity to additional coherence damage. In Appendix J, its median null-standardized responses to topic and cross-outlet shifts are 82 and 86, compared with 4.6 and 5.5 for the direct coherence judge. CHORD also detects added contradiction, broken transition, topic drift, and sentence permutation in at least two-thirds of the 22 corpus pairs (Table 21). This broader distributional objective is central to its design.

  2. Lower inference cost. With all methods served by the same vLLM engine on a single A100-80GB GPU, scoring 100 passages with the 27B backbone at concurrency 32 takes 6.4 seconds for CHORD, compared with 12.9 for G-Eval and 16.0 for the direct judge. Under sequential requests, the times are 9.7, 23.0, and 154.8 seconds, respectively. CHORD is the fastest method at every evaluated model size and concurrency level (Appendix G.3, Table 15).

  3. Effective distilled evaluators. Both the 2B and 0.8B distilled CHORD encoders retain detection of all nine perturbation types at the highest severity; direct judging and G-Eval with the corresponding small models each detect only five. At concurrency 32, the CHORD students score 100 passages in 0.72 and 0.46 seconds, respectively.

  4. Larger selectivity relative to reference sampling variation. With the same 27B backbone, CHORD achieves larger selectivity Δz = z_harm − z_benign across all nine perturbation types. For contradiction, the selectivity is 78.3 for CHORD, 15.9 for the direct judge, and 9.3 for G-Eval (Table 13). This quantifies stronger separation of coherence damage from benign rewriting, relative to each metric’s reference-null variation, alongside the direct detection comparisons above.

We have added a dedicated “Relation to LLM judging” paragraph in Section 5 explaining these evaluation goals and practical advantages.

W2. Controlled comparisons using the same backbone

Concern: Comparisons with older backbones do not fully isolate the contribution of CHORD’s methodology.

The original paper already includes controlled ablations separating backbone, extraction, and distance. Specifically:

  • Figure 3(a), Section 4.3.1: crosses four representations with four distributional distances, holding the representation fixed when comparing distances.
  • Figure 4(b), Section 4.3.2: compares extraction methods within Qwen3.5-27B and GPT-2-large, holding the backbone and distance fixed.
  • Appendix H: examines target attributes, prompt formats, wording, and extraction methods.
  • Appendix G: compares hidden-state features with token-space measures and direct judging using the same Qwen3.5-27B backbone.

We have additionally extended the controlled extraction comparison to four modern backbones (Appendix H.1, Table 19). Within each backbone, we hold the extraction layer, RBF-MMD distance, and evaluation protocol fixed, changing only the extraction method.

The table below reports detected relation and discourse conditions out of 12:

BackboneCoherence-prompted last tokenUnprompted last tokenUnprompted mean pooling
Qwen3.5-27B1010
Qwen3.8-27B810
Gemma-3-27B900
Mistral-Small-3.2-24B930

Table 19 also applies MAUVE’s distance to the same unprompted representations and computes generative perplexity with the same backbone. These configurations detect none of the 12 relation or discourse conditions on any of the four backbones, while still detecting many structural and mixture conditions (generative perplexity: 23–24 of 40 overall).

These comparisons demonstrate that access to a modern backbone alone does not account for CHORD’s sensitivity: the extraction method determines whether the relevant coherence distinctions become detectable.

If the reviewer has a further controlled comparison in mind, could they specify which component or confounding factor remains unresolved by these experiments? We would welcome that clarification.

AuthorsOpenReview ↗

Response to Reviewer ZQRR (3/4)

Q1. Naturally occurring and subtler coherence failures

Question: Can CHORD detect coherence failures beyond obvious synthetic perturbations?

The controlled suite includes relation-level errors such as contradiction and causal reversal, designed to preserve local fluency while changing relationships across sentences. We have added two complementary evaluations.

Human validation (Appendix C, Table 7). Two annotators independently evaluated 100 passages: 20 from each of the four LLM-edited perturbation types and 20 benign paraphrases. Across harmful conditions, 149 of 160 judgments (93%) confirmed the intended error; 38 of 40 judgments (95%) confirmed meaning preservation for benign paraphrases. Agreement on validity was 97% (Cohen’s κ = 0.75).

Naturally occurring model responses (Appendix D.2, Table 10). We evaluate HelpSteer2[1] responses with human coherence ratings, without injecting errors. Low-coherence responses (ratings ≤2) and medium-coherence responses (rating 3) are paired with rating-4 responses to the same prompts, with a response-length ratio of at most two. Each group contains 381 pairs.

CHORD detects both groups relative to their paired high-coherence controls. Unprompted mean pooling from the same backbone detects only the low-coherence group; MAUVE, generative perplexity, and unigram entropy detect neither. Detecting the medium-coherence group provides evidence that CHORD captures subtler human-rated differences in naturally generated responses.

Q2. Stronger generators and potential same-family bias

Question: Does CHORD distinguish outputs from stronger generators, and does the Qwen evaluator favor or penalize Qwen-generated text?

We have added an evaluation of 21 autoregressive and diffusion generators, including models up to 27B parameters, and a comparison across three evaluator families (Appendix M). Each generator continues the same 128-token DCLM [2] prefixes. We evaluate 800-token continuations over ten non-overlapping folds of 500 matched prefixes.

Stronger generators (Appendix M.2, Table 26). CHORD distinguishes held-out human continuations from those of the 24–27B generators, and larger models within each tested autoregressive family are closer to the human reference. The comparison also uses Qwen3.5-27B for all three metrics (CHORD, MAUVE, and generative perplexity) so the evaluators share the same backbone.

The results expose practical differences between metrics. Generative perplexity ranks Dream-7B third among the 21 generators, despite substantial repetitive degeneration: 256 of its 400 lowest-perplexity continuations have distinct-3 scores below the first percentile of human continuations. CHORD ranks Dream-7B 14th, penalizing these outputs. The expanded evaluation therefore tests CHORD on modern generators and longer outputs where favorable perplexity can still accompany poor generation quality.

Encoder–generator family overlap (Appendix M.1, Table 25). We score ten generators from the Qwen, Gemma, Mistral, and Llama families with Qwen3.5-27B, Gemma-3-27B, and Mistral-Small-3.2-24B encoders. Qwen and Gemma produce identical rankings, and Mistral agrees closely with both (Spearman ρ = 0.93).

In particular, the Qwen evaluator ranks Gemma-3-27B and Mistral-Small-3.1-24B ahead of Qwen3.5-27B, exactly as the Gemma evaluator does. Qwen3.5-27B ranks third under both encoders and fourth under Mistral. These results provide no evidence of systematic favoritism toward Qwen generators by the default Qwen evaluator and show substantial ranking consistency across evaluator families.

Reference

[1] Wang et al. HelpSteer2: Open-source dataset for training top-performing reward models, NeurIPS 2024

[2] Li et al. DataComp-LM: In search of the next generation of training sets for language models, NeurIPS 2024 (Datasets and Benchmarks Track)

Reviewer_Rxr7OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an approach to evaluating textual coherence in open-ended generation. The approach is based on an embedding function over the text and a distance metric over the text embedding (in practice, this is a paragraph embedded into a single vector). Experiments show that the "richness" or feature representation capacity of the embeddings is key for the distance metric performance in detecting overall and specifically hardest (relations contradictions/causal reversal) types of in-coherences.

Critical strengths
  • A metric to assess different cases of coherence beyond sentence ordering.
Critical weaknesses
  • The motivation for the proposed metric is not well described. The introduction sates the lack of references and mentions that distributional approaches are useful in this case. However, some approaches at instance level also work without references. So, the motivation of why corpus level is the focus should be better formulated.
  • The proposed method for obtaining embedding representations builds on existing work. Specific prompt and last token representation adapted from existing work.
  • The finding that smaller models produce representations that do not have discriminative features for some (harder) in-coherent phenomena is not surprising.
  • Incoherent texts for evaluation are synthetic and this may bias evaluation.
Questions for authors
  • Please introduce properly acronyms from the beginning.
  • Is still a human reference corpus needed? I we want to evaluate coherence in generated summaries, do we need a corpus of human authored summaries?
  • Table 4 should be in the main paper.
6 comments
AuthorsOpenReview ↗

Response to Reviewer Rxr7 (1/2)

We thank the reviewer for engaging in our work. We would like to clarify our main objectives, contributions, and a few important details to better facilitate the discussions.

W1–W2. Research objective and contribution

The central contribution of this paper is to make distributional evaluation of open-ended generation sensitive to global coherence. This distributional objective is essential to our study and to assessing its novelty.

The rapid development of diffusion and flow-based language models has renewed interest in unconditional generation and prefix continuation. These systems are commonly evaluated using generative perplexity and lexical-diversity statistics. However, recent studies cited in our Introduction show that improvements in these metrics need not translate into better generation quality: predictable, repetitive outputs can receive favorable perplexity scores.

Our study builds on the distributional evaluation paradigm exemplified by MAUVE[1]. MAUVE compares generated and human-text distributions in an embedding space, accounting for both outputs unlike human text and inadequate coverage of the human distribution. A generator that repeatedly produces a narrow set of individually good passages can therefore differ substantially from the target distribution. Reference-free instance-level coherence scores assess individual output quality, but do not measure this distributional mismatch. Their availability consequently does not remove the motivation for corpus-level evaluation. The analogy is FID[2] in image generation: individually plausible images do not establish that a generator matches the target image distribution.

The effectiveness of this comparison depends on which distinctions the embedding space preserves, which is the central claim of our paper. Global coherence is a critical gap in the generic representations commonly used by these distributional metrics such as MAUVE. Our experiments show that MAUVE’s GPT-2 features, for example, respond similarly to contradiction-inducing edits and meaning-preserving paraphrases. We address this gap by constructing coherence-oriented representations and validating their selectivity against matched benign controls. Figure 3(a) demonstrates the importance of this change: GPT-2 features with RBF-MMD detect only three perturbation types, whereas coherence-oriented Qwen features with MAUVE’s distance detect all nine.

We explicitly build on PromptEOL and established distributional distances. Our contribution is to identify a consequential blind spot in distributional evaluation and provide a validated metric that addresses it. The use of established components does not diminish this contribution: the experiments establish how representation design makes previously missed coherence failures detectable within the distributional framework.

We have revised the Introduction and Section 3.1 to make this objective and contribution explicit.

W3. What the model-scale ablation establishes

Concern: Smaller models failing to capture harder coherence phenomena is unsurprising.

The model-scale experiment is an ablation supporting the design of the evaluator. The paper’s contribution does not rest on the observation that larger models perform better.

The experiments establish that backbone capability and representation extraction jointly determine coherence sensitivity. With the Qwen3.5-27B backbone, layer, and distance held fixed, coherence-prompted extraction detects 10 of 12 relation and discourse conditions, compared with one for unprompted last-token extraction and zero for mean pooling (Table 19). This advantage extends across four modern backbones. Yet the same prompting strategy can reduce selectivity on GPT-2, where mean pooling performs best (Table 16).

These results answer a practical design question: using a stronger backbone does not by itself produce an effective coherence-aware distributional metric. The relevant information must be exposed by the representation.

Furthermore, our distilled 2B and 0.8B encoders both detect all nine perturbation types at the highest severity, demonstrating that the teacher’s sensitivity can be transferred to substantially smaller models.

(continued in the next comment)

AuthorsOpenReview ↗

Response to Reviewer Rxr7 (2/2)

W4. Evidence beyond synthetic perturbations

Concern: Synthetic incoherent texts may bias evaluation.

We have added human validation of the edited passages and an evaluation on naturally occurring model responses.

Human validation (Appendix C, Table 7). Two annotators independently evaluated 100 passages: 20 from each of the four LLM-edited perturbation types and 20 benign paraphrases. Across the harmful conditions, 149 of 160 judgments (93%) confirmed the intended error; 38 of 40 judgments (95%) confirmed meaning preservation for benign paraphrases. Agreement on validity was 97% (Cohen’s κ = 0.75).

Naturally occurring coherence differences (Appendix D.2, Table 10). We additionally evaluate on HelpSteer2, a dataset of model responses with human coherence ratings, without injecting errors. The low-coherence and medium-coherence groups each contain 381 responses paired with rating-4 responses to the same prompts, with a response-length ratio of at most two. CHORD detects both groups relative to their paired high-coherence controls. Unprompted mean pooling from the same backbone detects only the low-coherence group; MAUVE, generative perplexity, and unigram entropy detect neither.

These results show that CHORD’s sensitivity extends beyond the synthetic editing procedure to independently human-rated differences in model outputs.

Q1. Acronym definitions

We have expanded the remaining acronyms at their first occurrence in the abstract and introduction.

Q2. Human reference corpus

Yes. The human reference corpus is an essential part of CHORD’s design: it defines the target distribution against which generated text is evaluated. For generated summaries, the corresponding reference would be a corpus of human-authored summaries representative of the intended domain and format. These need not be paired one-to-one with the generated summaries.

This requirement follows directly from the distributional evaluation objective shared with MAUVE. CHORD removes the need for aligned reference outputs while retaining a reference corpus that defines the comparison target. We have clarified this distinction in the Introduction and Section 3.1.

Q3. Table 4 should be in the main paper

We have added a compact version as Table 1 in the main paper, covering all nine perturbation types and the benign control. Full examples remain in Appendix C.

Summary and Further Discussion

We believe these responses address the concerns raised by clarifying the distributional evaluation objective and the contribution of coherence-sensitive representation design, establishing the role of the model-scale ablation, and adding both human validation and evidence from naturally occurring model responses. We have also clarified the reference-corpus requirement and acronym definitions, and moved perturbation examples into the main paper. Following our clarification of the distributional evaluation objective and the additional same-backbone comparisons, could the reviewer clarify which aspect of the contribution you still find insufficiently established? This would help us address your remaining concern more directly. We would appreciate the reviewer’s reassessment of the contribution and welcome any remaining questions for us to address during the discussion.

Reference

[1] Pillutla et al. MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers. NeurIPS 2021.

[2] Heusel et al. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. NeurIPS 2017.

AuthorsOpenReview ↗

A Brief Follow-up for Reviewer Rxr7

Dear Reviewer Rxr7,

Thank you again for your time and your comments, which helped us state our motivation much more clearly.

In brief, the revision:

  1. Clarifies the motivation: why corpus-level distributional evaluation is not covered by instance-level coherence scores (Introduction, Section 3.1).
  2. Adds same-backbone comparisons showing that coherence sensitivity comes from representation design rather than backbone scale (Table 19).
  3. Adds human validation of the perturbations and an evaluation on naturally occurring, human-rated responses (Tables 7 and 10).

If any concerns remain, we would be very glad to hear them and will respond promptly. Even a brief note would be much appreciated.

Best regards,

The Authors

Reviewer_HwXcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper focuses on a previously underexplored dimension of open-ended text generation - global coherence. Existing distributional metrics can only detect differences preserved by their feature spaces, and cannot effectively capture discourse-level perturbations. Dedicated coherence metrics focus on individual samples, which is not designed to compare generated and human corpora as distributions. The paper proposes CHORD, a distributional metric that captures coherence differences between generated and human text. Evaluation shows that CHORD can detect a significantly broader range of coherence failures.

Critical strengths
  • This is a well-motivated problem, and the core design to allow the metric to separate a coherence-degrading perturbation from benign rewrites is intuitive and informative.
  • Strong results: the controlled perturbations include nine different coherence failures, and relation-level errors can maintain local fluency. Evaluation is conducted with varying model sizes, prompts, layers, distance functions, etc., and Table 1 shows strong evidence that CHORD outperforms existing metrics in detecting such global coherence failures. Section 4.5 presents high correlation with human system rankings.
  • The paper also shows two smaller distilled student encoder (2B, 0.8B) from CHORD-27B. The two student encoders significantly reduce runtime and GPU memory usage, while being able to detect all nine perturbation types. These are practical alternatives to the full-size encoder.
Critical weaknesses
  • It's unclear that CHORD has clear advantage over the same-backbone LLM judging. Table 8 shows that Qwen3.5-27B LLM judge and G-Eval can also detect all nine perturbation types at highest severity, while CHORD produce larger null-standardized responses. Results at more subtle severities or fewer perturbations in the text corpus would be persuasive.
  • Generalizability: The evaluation set are relative old English web text, and most are short passages (at least four sentences and 100–400 words). Performance on an unfamiliar domain, longer text documents, and some more subtle coherence failures may not generalize.
Questions for authors
  • How does CHORD compare with LLM judges on detection power of the coherence failures? For example, by varying sample size, perturbation severity, etc., can CHORD detect failures that LLM judge with the same model cannot?
  • Appendix C listed the filtering step to check the synthetic perturbation and benign edits. Are there a human evaluation of a random set to verify that the intended coherence failures are present in the perturbed set and that the semantic meaning is preserved for all benign edits?
  • How sensitive is CHORD to domain differences? What happens when the generated passages are coherent but come from a different domain or style from the human reference? Can the metric separate the shifts from global coherence?
5 comments
AuthorsOpenReview ↗

Response to Reviewer HwXc (1/3)

We thank the reviewer for recognizing the motivation, controlled evaluation, and practical value of the distilled encoders. We have added comparisons of inference cost and smaller evaluators, evaluations on long documents and human-rated model responses, human validation of the perturbations, and experiments on topic and style shifts.

W1 / Q1. Comparison with same-backbone LLM judges

Question: Does CHORD offer practical advantages over same-backbone judging, particularly for subtler coherence failures?

CHORD and the same-backbone judge show comparable coherence-detection performance in the reported comparisons. Both detect all nine perturbation types at the highest severity (Appendix G.2, Table 13). Our new long-document evaluation also compares detection across multiple severities: CHORD and the direct judge each detect 36/40 conditions on 1K documents and 35/40 on 2K documents (Appendix D.1, Table 9). These results support the effectiveness of both approaches.

CHORD’s advantages concern its broader evaluation objective, inference efficiency, and the ability to transfer its sensitivity to smaller encoders:

  1. Distributional information beyond per-text coherence ratings. CHORD measures differences between generated and reference corpora in a coherence-sensitive representation space. Two corpora can receive similar coherence ratings while differing substantially in topic composition or style. In our new topic/style-shift experiments, CHORD responds strongly to these differences while retaining sensitivity to additional coherence damage; the coherence judges respond much less strongly to the shifts themselves (Appendix J, Table 21). This is a central capability of distributional evaluation that aggregating per-text coherence ratings does not provide.

  2. Lower inference cost. With all methods served by the same vLLM engine on a single A100-80GB GPU, scoring 100 passages with the 27B backbone at concurrency 32 takes 6.4 seconds for CHORD, compared with 12.9 for G-Eval and 16.0 for the direct judge. Under sequential requests, the corresponding times are 9.7, 23.0, and 154.8 seconds. CHORD is the fastest method at every evaluated backbone size and concurrency level (Appendix G.3, Table 15).

  3. Effective distilled evaluators. Both distilled CHORD encoders, at 2B and 0.8B, retain detection of all nine perturbation types at the highest severity. Direct judging and G-Eval with the corresponding small models each detect only five. The students also reduce inference time substantially: at concurrency 32, CHORD scores 100 passages in 0.72 seconds with 2B and 0.46 seconds with 0.8B (Table 15).

  4. Larger selectivity relative to reference sampling variation. With the same 27B backbone, CHORD achieves larger selectivity Δz = z_harm − z_benign across all nine perturbation types (Table 13). For contradiction, its selectivity is 78.3, compared with 15.9 for the direct judge and 9.3 for G-Eval. This establishes stronger separation of coherence damage from benign rewriting, relative to each metric’s reference-null variation.

We have added a dedicated “Relation to LLM judging” paragraph in Section 5 to explain these complementary evaluation goals and practical advantages.

AuthorsOpenReview ↗

Response to Reviewer HwXc (2/3)

W2. Generalization beyond short passages from the default sources

Concern: Performance may not generalize to unfamiliar data, longer documents, or subtler coherence failures.

We have added two complementary evaluations in Appendix D.

Long documents from another web corpus (Appendix D.1, Table 9). We reconstruct the counterfactual evaluation on complete DCLM-baseline-1.0 documents[1] in two length ranges: 896–1024 and 1792–2048 GPT-2 tokens, with median lengths of 39 and 78 sentences. Each length setting uses disjoint pools of 500 seeds, 3,600 reference documents, and 2,000 replacement documents. Relation and discourse edits affect 1/2/3 sentences at 1K and 2/4/6 at 2K, approximately matching their edited fractions. Benign paraphrases follow the same schedules. CHORD detects 36/40 conditions at 1K and 35/40 at 2K, matching the direct judge and exceeding G-Eval (33/40 at both lengths) and the other baselines (at most 27/40). CHORD detects two of the three contradiction conditions at each length.

Naturally occurring, human-rated responses (Appendix D.2, Table 10). We evaluate HelpSteer2[2] model responses without injecting errors. Low-coherence responses (ratings ≤2) and medium-coherence responses (rating 3) are paired with rating-4 responses to the same prompts, with a response-length ratio of at most two. Each group contains 381 pairs. CHORD detects both the low- and medium-coherence groups relative to their paired high-coherence controls. Unprompted mean pooling from the same backbone detects only the low-coherence group; MAUVE, generative perplexity, and unigram entropy detect neither. The medium-coherence result provides evidence that CHORD captures subtler human-rated differences in naturally generated responses.

Q2. Human validation of perturbations and benign rewrites

Question: Have human annotators verified that the intended errors are present and benign edits preserve meaning?

Yes. We have added human validation in Appendix C, Table 7. We sample 20 passages from each of the four LLM-edited perturbation types at the highest severity and 20 benign paraphrases, yielding 100 items stratified by source. Two annotators independently evaluate every item, viewing the original and edited passages with edited sentences highlighted.

Across the harmful conditions, 149 of 160 judgments (93%) confirm the intended error. For benign paraphrases, 38 of 40 judgments (95%) confirm meaning preservation. The annotators agree on the validity label for 97% of items (Cohen’s κ = 0.75). They also assess the edited sentences in isolation: fluency receives positive judgments in 88–98% of cases across harmful types and 100% for benign paraphrases.

These results support both the intended coherence changes and the use of paraphrases as meaning-preserving controls.

Q3. Sensitivity to domain and style differences

Question: What happens when coherent generated passages differ in domain or style from the human reference? Can CHORD separate these shifts from coherence differences?

CHORD does respond to domain and style mismatch. This is expected: as a distributional metric, it is coherence-aware rather than coherence-only. Its reference corpus defines the target distribution, and coherent text can still differ from that distribution in meaningful ways such as in topic and style.

We examine this explicitly in Appendix J. We construct 20 ordered topic-shift pairs across five Daily Mail sections and two cross-outlet pairs comparing Daily Mail Sport with CNN Sport. For each pair, we evaluate the original target corpus and versions with added contradiction, causal reversal, broken transition, topic drift, or sentence permutation.

The experiment measures two quantities separately: the response to the original corpus shift, and the additional response after coherence damage is introduced.

  • Shift alone: CHORD’s median null-standardized responses are 82 for topic shifts and 86 for cross-outlet shifts. The direct coherence judge yields 4.6 and 5.5, and G-Eval yields 5.6 and 0.6.
  • Additional coherence damage: CHORD detects contradiction, broken transition, topic drift, and sentence permutation in at least two-thirds of the 22 pairs; causal reversal is detected in 11/22 pairs.
  • Comparison with generic representations: Unprompted mean pooling from the same backbone and MAUVE also respond to the original shifts, but show much weaker sensitivity to the added coherence damage (Table 21).

Thus, CHORD captures distributional mismatch while preserving sensitivity to coherence failures within the shifted corpus. The paired comparisons isolate the additional response to coherence damage; a standalone CHORD score measures the combined discrepancy from the reference. We have clarified this interpretation in Section 3.1 and the Discussion.

AuthorsOpenReview ↗

Response to Reviewer HwXc (3/3)

Summary and Further Discussion

We believe the clarifications and additional experiments address the concerns raised: they establish CHORD’s practical advantages and complementary objective relative to same-backbone judging, extend evaluation to longer documents and naturally occurring coherence differences, validate the synthetic edits through human judgments, and demonstrate sensitivity to both domain shifts and additional coherence damage. We would appreciate the reviewer’s assessment of whether these responses resolve the concerns, and welcome any remaining questions for us to address during the discussion.

Reference

[1] Li et al. DataComp-LM: In search of the next generation of training sets for language models, NeurIPS 2024 (Datasets and Benchmarks Track)

[2] Wang et al. HelpSteer2: Open-source dataset for training top-performing reward models, NeurIPS 2024

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.