acceptodds
Under review as a conference paper at ICLR 2027

: Meta-Evaluating Metrics for Multi-lingual Style-Personalized Generation

Abstract

Meta-evaluation of metrics and LLM-judges for text generation is primarily done using metric-human correlations, even though prior work has shown that correlation does not give an appropriate estimate of a metric's evaluation capability, and can be unstable across settings such as different datasets or languages. We propose a regression-based meta-evaluation of metrics that learns on calibrated metric outputs to predict human ratings. We ground this measurement relative to a trivial constant-mean baseline and the human agreements, to provide a comparable measurement of metric capability across evaluation settings. To test our meta-evaluation framework, we developed a large-scale multi-lingual style personalized text generation (SPTG) benchmark, collecting multiple annotations over 10 topologically diverse languages and four evaluation dimensions of varying subjectivity. We meta-evaluated 32 metrics from three style evaluation paradigms (string-based, embedding-based, and LLM-judges) and their ensembles over these 40 evaluation settings. Our analysis revealed that even simple strong-based and embedding-based measurements can match strong LLM-judges when calibrated to human ratings. We also identified characteristics of human annotations that determine the performance of metrics, as well as when a metric ensemble outperforms its individual constituents.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
meta evaluation, style personalized text generation, subjective metric evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_oD3fOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a regression-based framework for meta-evaluating automatic metrics for style-personalized text generation. Metric outputs are calibrated and used to predict reliability-weighted human judgments. Held-out MAE is then normalized relative to a constant-mean predictor and an individual-annotator reference. The authors introduce a ten-language dataset containing 10,038 instances, each with two candidate generations and three-annotator judgments across style, factual consistency, reference-content leakage, and usability. The study evaluates 32 metrics and their ensembles. Its main findings are that inexpensive metrics can be competitive after supervised calibration and regression, and that ensembling is particularly beneficial for style evaluation.

Critical strengths
  1. A valuable multilingual and multiply annotated evaluation resource. The dataset covers ten languages and separates style, factual consistency, leakage, and usability. Three-annotator judgments and the reported reliability statistics support a richer investigation than a single overall-quality label. The breadth of metric coverage and disclosure of judge-output failures further increase the empirical value.

  2. A useful investigation of supervised predictive signal and metric combinations. The paper systematically compares inexpensive string metrics, semantic and style embeddings, LLM judges, and ensembles. Table 7 reports substantial style gains for the full ensemble, with Ψ increasing from approximately 0.43 for the strongest individual metrics covering all ten languages to 0.78. The instance-grouped folds and training-fold-only calibration and regression are appropriate safeguards. These results motivate further investigation of locally supervised combinations of metrics.

Critical weaknesses
  1. The interpretation and cross-setting comparability of Ψ require stronger validation. Equation 2 uses a constant-mean baseline while evaluating absolute error. Under MAE, a median is an optimal constant predictor. For example, in a hypothetical distribution with 90% label 0 and 10% label 4, constant-mean and constant-median MAEs are 0.72 and 0.40. A uninformative constant predictor can therefore improve over the proposed reference. This does not invalidate Ψ as a score relative to a chosen mean baseline, but it limits its interpretation as performance beyond an appropriate no-information baseline. Section 7.2 acknowledges dependence on label skew and annotator agreement; however, the manuscript does not establish that the normalization supports the cross-setting comparability claimed in the introduction and problem definition. Reporting the underlying errors, normalization denominators, and sensitivity to the constant baseline would make this claim assessable. For a fixed setting with a common positive denominator, Ψ preserves the MAE ordering, so this concern primarily affects absolute interpretation and cross-setting aggregation.

  2. The main comparisons need more independent and controlled validation. Appendix A describes selecting a single ExtraTrees regressor after comparing 16 approaches on the same cross-validation setting within the approximately 100-instance-per-language test split. This creates a risk of selection optimism; the fold-level fitting safeguards in Section 5.2 do not by themselves establish that method selection was independent of final performance estimation. Moreover, the comparison between correlation and the proposed framework changes several factors simultaneously: Section 5.1 uses unweighted human means, whereas Section 5.2 uses reliability-weighted targets, calibration, and nonlinear regression. The observed ranking changes therefore do not isolate which component reveals additional useful signal. Independent regressor selection, matched-target ablations, and uncertainty estimates for the headline metric differences are needed to substantiate the comparative conclusions.

Questions for authors
  1. Selection and evaluation independence. Was the ExtraTrees selection in Appendix A performed independently of the data used to report the final meta-evaluation results? Please clarify the roles of the train/dev/test splits and whether annotator-reliability weights were estimated within each outer training fold. If method selection used the reported evaluation data, do the central findings persist with an independent development split or nested cross-validation?

  2. Attribution of ranking changes. On matched examples and human targets, how do direction correction, simple or monotonic calibration, and nonlinear regression affect the main metric comparisons? Please specify the calibration algorithm. Results using both unweighted and reliability-weighted targets would help separate changes in the evaluation target from changes in the predictive method. Uncertainty estimates for the principal comparisons would also clarify whether the observed differences are robust.

  3. Interpretation of the normalized score. Please report per-setting metric MAE, constant-mean and constant-median MAEs, the human-reference MAE, and the normalization denominator. How sensitive are the main conclusions to the baseline choice? Please also clarify the interpretation of the human reference given that metric predictions target a three-annotator consensus, while the human-reference error compares one annotator with the other two. Is clipping to [0,1] restricted to Table 8’s display, and does Table 7 aggregate unclipped values?

  4. Variance-decomposition diagnostic. Equation 1 and Section 5.1 describe nonnegative incremental R², but Figure 6 contains negative values. What statistic is actually plotted, and how is it aggregated? If it is based on held-out predictions or a different signed comparison, please provide the corresponding definition and clarify whether the conclusions about unique metric contributions and ensemble complementarity remain supported.

3 comments

(1/2)

We would like to thank the reviewer for their time and consideration in providing helpful feedback on our work. We address the reviewer's remarks as follows:

The interpretation and cross-setting comparability of Ψ require stronger validation. Equation 2 uses a constant-mean baseline while evaluating absolute error. Under MAE, a median is an optimal constant predictor. For example, in a hypothetical distribution with 90% label 0 and 10% label 4, constant-mean and constant-median MAEs are 0.72 and 0.40. A uninformative constant predictor can therefore improve over the proposed reference. This does not invalidate Ψ as a score relative to a chosen mean baseline, but it limits its interpretation as performance beyond an appropriate no-information baseline. Section 7.2 acknowledges dependence on label skew and annotator agreement; however, the manuscript does not establish that the normalization supports the cross-setting comparability claimed in the introduction and problem definition. Reporting the underlying errors, normalization denominators, and sensitivity to the constant baseline would make this claim assessable. For a fixed setting with a common positive denominator, Ψ preserves the MAE ordering, so this concern primarily affects absolute interpretation and cross-setting aggregation.

(question) Interpretation of the normalized score. Please report per-setting metric MAE, constant-mean and constant-median MAEs, the human-reference MAE, and the normalization denominator. How sensitive are the main conclusions to the baseline choice? Please also clarify the interpretation of the human reference given that metric predictions target a three-annotator consensus, while the human-reference error compares one annotator with the other two.

We have performed the analysis using the median as the trivial baseline choice in the normalization step, on the same k-fold validation splits as in the original submission. While the change in baseline affects the raw Ψ values, it does not change the rankings of metrics obtained from the meta-evaluation. The order of metrics is almost identical under the median baseline compared to the reported rankings with the mean baselines, with Spearman rank correlations of 0.93 (style), 0.95 (factual), 0.98 (leakage), and 0.96 (usability).

Using the mean across the training folds of the data is also a viable baseline for our meta-evaluation and does not affect our findings. Therefore, we have decided to keep the main results from our work unchanged, and have included the analysis of the influence of changing the normalization baseline from mean to median in Appendix C.3 (Lines 1058-86).

The main comparisons need more independent and controlled validation. Appendix A describes selecting a single ExtraTrees regressor after comparing 16 approaches on the same cross-validation setting within the approximately 100-instance-per-language test split. This creates a risk of selection optimism; the fold-level fitting safeguards in Section 5.2 do not by themselves establish that method selection was independent of final performance estimation. [...]

(question) Selection and evaluation independence. Was the ExtraTrees selection in Appendix A performed independently of the data used to report the final meta-evaluation results? Please clarify the roles of the train/dev/test splits and whether annotator-reliability weights were estimated within each outer training fold. If method selection used the reported evaluation data, do the central findings persist with an independent development split or nested cross-validation?

We conducted extensive analysis across a wide range of regression models (7 additional models from the four regression families explored earlier) and show that the proposed meta-evaluation framework is robust to the choice of regression model. Overall, the ranking of the 32 individual metrics remains consistent across the regression model’s Ψ values, with a rank correlation of Spearman’s ρ=0.84 to ρ=1.0 across regression models. We have included a detailed analysis in Appendix C.2 of our revised submission (Figure 6 and Lines 1030-56).

(2/2)

[...] Moreover, the comparison between correlation and the proposed framework changes several factors simultaneously: Section 5.1 uses unweighted human means, whereas Section 5.2 uses reliability-weighted targets, calibration, and nonlinear regression. The observed ranking changes therefore do not isolate which component reveals additional useful signal. Independent regressor selection, matched-target ablations, and uncertainty estimates for the headline metric differences are needed to substantiate the comparative conclusions.

(question) Attribution of ranking changes. On matched examples and human targets, how do direction correction, simple or monotonic calibration, and nonlinear regression affect the main metric comparisons? Please specify the calibration algorithm. Results using both unweighted and reliability-weighted targets would help separate changes in the evaluation target from changes in the predictive method. Uncertainty estimates for the principal comparisons would also clarify whether the observed differences are robust.

We used isotonic regression calibration (fit to the training folds of the 5-fold validation) to map the continuous metric predictions to the human ratings (a step function with a median of 3 to 5 distinct values). We conducted an ablation study on the two components of our proposed meta-evaluation: calibration and nonlinear regression (using LASSO as the best-ranking linear regressor from Appendix A).

Here’s the Spearman agreement of the final meta-metric and the ablated meta-metric (including the standard Speraman’s ρ meta-evaluation from Section 5.1 in the last row:

Meta-metricρ Styleρ Factualρ Leakageρ Usability
Ψ1.001.001.001.00
− calibration (ExtraTrees, raw metric values)0.990.980.961.00
− tree-based regression (Lasso, raw metric values)0.880.920.910.92
Spearman |ρ| (Section 5.1)0.630.910.880.85

The table shows the difference in agreement between an ablated meta-evaluation based on single-metric rankings and the proposed meta-evaluation metric. Calibration does not affect the tree-based regressors as much (ρ = 0.96 to 1.00), since trees are invariant to monotone input transforms such as the isotonic calibration. Replacing trees with a linear fit shifts the ranking by ρ = 0.88 to 0.92 and removes the ensemble gains

We have additionally conducted a sensitivity analysis of the meta-evaluation measurement with respect to the size of the calibration data in Appendix C.1, as well as assessed robustness to the choice of regression model (when applied to the calibrated data) in Appendix C.2 of the revised submission.

(question) Variance-decomposition diagnostic. Equation 1 and Section 5.1 describe nonnegative incremental R², but Figure 6 contains negative values. What statistic is actually plotted, and how is it aggregated? If it is based on held-out predictions or a different signed comparison, please provide the corresponding definition and clarify whether the conclusions about unique metric contributions and ensemble complementarity remain supported.

The plot shows the average difference between two non-negative R^2 terms across all other evaluation metrics. This difference reflects the average unique variance a metric captures relative to all other metrics. Formally, for a metric p, it is computed as ∑q∈M,q≠pΔRp∣q2−ΔRq∣p2\sum_{q\in\mathcal{M}, q\neq p}\Delta R^2_{p|q} - \Delta R^2_{q|p}. We have clarified how this aggregate R^2 is computed in the captions of Figures 1 and 9 in our revised submission.

[...] Is clipping to [0,1] restricted to Table 8’s display, and does Table 7 aggregate unclipped values?

We clipped the values in Table 8 for readability. But we have now updated it to include the unclipped values in the revised submission. You are correct in understanding that Table 7’s aggregate was done on the unclipped values.

Thank you again for your constructive feedback. If you find that our responses have resolved your main concerns, we would be grateful if you could consider updating your assessment accordingly.

Thank you for your response. I appreciate the clarification and find it reasonable. However, with respect, I would like to maintain my original evaluation.

Reviewer_V8dxOpenReview ↗
Rating
4
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a critical bottleneck in the Natural Language Generation (NLG) domain: the lack of reliable meta-evaluation for automatic evaluation metrics, particularly when applied to subjective tasks such as style-personalized generation. Traditional meta-evaluation methods, which heavily rely on correlation coefficients (such as Spearman's ρ), suffer from extreme instability and sensitivity to data distribution. The main contributions of this work are threefold: A Novel Meta-Evaluation Framework: The authors propose a regression-based meta-evaluation framework. It calibrates individual metric outputs to human rating scales and trains a regressor to predict human consensus. The core of this framework is a normalized quality score, Ψ (Equation 2), which quantifies the extent to which a metric closes the gap between a random baseline and human evaluator consensus. This framework offers significantly higher stability and comparability than standard correlation coefficients.

A Large-Scale, Multilingual Benchmark Dataset: The authors construct a benchmark dataset for the "Style-Personalized Text Generation (SPTG)" task, spanning 10 typologically diverse languages. It contains multi-annotator human ratings across four dimensions: style accuracy, factual accuracy, information leakage, and usability. This dataset provides a solid empirical foundation for multilingual metric evaluation.

Comprehensive Meta-Evaluation Experiments: The paper systematically evaluates 32 distinct automatic evaluation metrics, covering classic string-matching baselines, embedding-based similarities, and advanced Large Language Model (LLM) judges. The findings expose critical limitations in existing evaluation paradigms and offer solid heuristics on how to select and ensemble metrics to best approximate human judgment.

Critical strengths

Clear Problem Definition and Strong Motivation: The paper clearly articulates the systematic flaws of current meta-evaluation practices in NLG—especially their over-reliance on correlation metrics—and introduces a principled alternative. This work is highly valuable for driving more reliable and reproducible evaluations in the NLG community.

Robust and Novel Methodology: The proposed regression-based meta-evaluation framework, particularly the normalized quality score Ψ, is elegantly designed. It solves the issue of comparability across different tasks and languages. By anchoring performance against the Maximum Achievable Agreement of Humans (MAEH), it provides an intuitive, absolute physical meaning to metric performance.

Thorough Experimental Design and Complete Evidence Chain: Dataset Construction: The authors put substantial effort into constructing the high-quality, multilingual, multi-dimensional SPTG dataset. The validation pipeline (translation, manual proofreading, and multi-annotator consensus scoring) is well-documented in the appendix, establishing strong credibility.

In-Depth Analysis: The analysis in Section 7 is insightful. For instance, the authors find that once calibrated, certain simple, non-LLM embeddings (such as LaBSE) can match or even exceed the performance of state-of-the-art LLM judges in specific dimensions like "Style". This challenges the common assumption that LLM judges are unilaterally superior across all evaluation tasks.

Ensembling Insights: The study reveals that metric ensembling yields substantial gains in dimensions with sparse signal distributions (like "Style"), but provides marginal utility in dimensions where signals are highly concentrated (like "Information Leakage"). This provides practical guidance for practitioners on when to deploy multi-metric ensembles.

Critical weaknesses

Boundaries of Framework Applicability: While the regression-based meta-evaluation framework is powerful, its validity inherently depends on the availability of sufficient human-annotated data for calibration and training. As the authors acknowledge in Section 8, the method "still requires human annotations to calibrate the measurement." For low-resource languages or entirely new tasks where human labels are scarce, the utility of this framework might be constrained. The paper would benefit from a more rigorous sensitivity analysis demonstrating how much human data is required to yield a stable Ψ score.

Subjectivity in Defining and Evaluating "Style": Although the paper breaks down the evaluation into four dimensions, "Style Accuracy" remains inherently subjective and multi-faceted. The empirical results show that "Style" is consistently the most challenging dimension for all metrics to capture. While this is an inherent property of the task rather than a flaw of the paper, the authors could discuss how future work might deconstruct "Style" into finer, more objective sub-dimensions (e.g., formality, humor, conciseness) to make metric evaluation even more precise.

Questions for authors

Regarding the Negative Correlation of LaBSE in Frisian and Japanese: In Section 8, you mention that LaBSE has a negative raw correlation with style in Frisian and Japanese, which is a very interesting phenomenon. Could you elaborate on this finding? Does it suggest that, in the cultural or linguistic context of these two languages, the semantic similarity captured by LaBSE conflicts with human-perceived stylistic similarity?

Analysis of LLM Judge Failure Rates: Table 9 in the appendix shows that some open-source models (like the Llama series) have very high parsing failure rates in pairwise comparison tasks. Did this affect their performance in the meta-evaluation? Have you considered using more robust constraints or post-processing on the LLM judge outputs to reduce these failures?

Choice of Regression Models: In Appendix A, you compare 16 regression models and ultimately select ExtraTrees. The results show that tree-based models generally perform better. Could you briefly analyze why tree-based models are more suitable for this task of mapping metric outputs to human scores compared to linear models or neural networks?

Discussion on the "Usability" Dimension: In Figure 2, the evaluation results for the "Usability" dimension are quite interesting, with performance differences between various metric types (including LLMs) appearing small. Does this imply that "Usability" is a more composite but weaker signal? What are your thoughts on this result?

3 comments

(1/3)

We thank the reviewer for their encouraging assessment and for the constructive questions, which helped us improve the paper. We address the reviewer's remarks as follows:

Boundaries of Framework Applicability: While the regression-based meta-evaluation framework is powerful, its validity inherently depends on the availability of sufficient human-annotated data for calibration and training. As the authors acknowledge in Section 8, the method "still requires human annotations to calibrate the measurement." For low-resource languages or entirely new tasks where human labels are scarce, the utility of this framework might be constrained. The paper would benefit from a more rigorous sensitivity analysis demonstrating how much human data is required to yield a stable Ψ score.

We agree with the reviewer that a sensitivity analysis of the amount of human data versus the stability of the meta-evaluation’s Ψ score would be helpful to readers. Therefore, we have conducted a sensitivity analysis of the number of annotated instances in the matched test set and across the entire dataset (for all non-LLM metrics). This analysis shows that different evaluation dimensions require varying amounts of data to reach stability, with leakage achieving results close to its asymptotic value across the entire dataset in just 40 annotations (from 20 instances). We also found that, compared to using 20 instances for calibration, using 160 annotations (from 80 instances rated for both paraphrases separately) yields an m_{string+embedding} ensemble that captures 71% of the total gain in style, 59% in factual accuracy, and 80% in leakage. The remaining 1600+ instances add a small gain to the meta-evaluation score. We have included this analysis in our revised submission in Appendix C.1 (Figure 5 and Lines 964-1029).

Subjectivity in Defining and Evaluating "Style": Although the paper breaks down the evaluation into four dimensions, "Style Accuracy" remains inherently subjective and multi-faceted. The empirical results show that "Style" is consistently the most challenging dimension for all metrics to capture. While this is an inherent property of the task rather than a flaw of the paper, the authors could discuss how future work might deconstruct "Style" into finer, more objective sub-dimensions (e.g., formality, humor, conciseness) to make metric evaluation even more precise.

Evaluating (and even defining) style remains a major challenge in the community, and, as the reviewer has pointed out, a primary reason is that it is subjective and multifaceted. Therefore, to capture the native speaker’s perception of style within the task of style-personalized text generation, the annotation guidelines instructed them to treat the style reference document as their own writing and to judge whether the generated style-personalized output “sounds like me”. We also provide them with a detailed guideline of what style represents at different granularities, from word-level, phrase-level, sentence-level, and document-level characteristics. It is the same definition used to generate the paraphrases (as provided in the prompt, Figure 14), and we have made this explicit in our revised submission (Section 4.2, Lines 193-97).

Taking humor as an example, most people would readily agree on what humor means, but whether a given sentence is funny varies from person to person. Referring to Kane’s argument-based view of validity [1], the construct of style is shared, and the disagreement arises at the scoring stage, where the raters assign a score to the observation (in our case, a 0-4 Likert scale). This is also consistent in our data, with substantial agreement on style annotations overall (α = 0.76) and wide variation across languages (0.58 to 0.95; Table 1). From a meta-evaluation perspective, our proposed framework accounts for this by normalizing the extent to which annotators predict each other’s scores.

Determining the objective sub-dimensions and how annotators rate them in each language requires qualitative and stylometric analyses of the annotations. Separating the subjective ratings from the objective components is a study in its own right, and we leave it to future work. Our annotated data also include open-ended comments for each rater’s pointwise annotation, which can aid this downstream analysis. We thank the reviewer for their feedback and have incorporated it into our revised submission as a future research direction (Lines 523-26).

  1. Kane, M.T., 2013. Validating the interpretations and uses of test scores. Journal of educational measurement, 50(1), pp.1-73.

(2/3)

(Question) Regarding the Negative Correlation of LaBSE in Frisian and Japanese: In Section 8, you mention that LaBSE has a negative raw correlation with style in Frisian and Japanese, which is a very interesting phenomenon. Could you elaborate on this finding? Does it suggest that, in the cultural or linguistic context of these two languages, the semantic similarity captured by LaBSE conflicts with human-perceived stylistic similarity?

While cultural context could potentially explain (partially) the different behavior of LaBSE in Frisian and Japanese, we found it to also be an artifact of annotators’ behavior on an edge case in our annotations. For about a fifth of the instances that were labeled to have a non-zero leakage score (i.e., content was leaked from the style reference r to the generated output), the annotators in Japanese and Frisian gave a higher style score to these instances than the non-leakage outputs. The mean style rating for leaked/non-leaked instances was 3.47/1.94 in fy and 2.42/0.78 in ja; compared to other languages, where the annotators gave a score close to zero for the leaked instances 0.2/1.5 for leaked/non-leaked instances. Compared to the raw negative correlations of -0.52 and -0.54, removing the leaked instances yielded -0.29 and -0.34, respectively. Similarly, removing instances from this edge case of leakage in the output (as the guidelines do not provide an annotation convention for this scenario), LaBSE’s correlation across the other languages ranges from −0.47 to +0.09 (with negative correlation in 5 languages).

(question) Analysis of LLM Judge Failure Rates: Table 9 in the appendix shows that some open-source models (like the Llama series) have very high parsing failure rates in pairwise comparison tasks. Did this affect their performance in the meta-evaluation? Have you considered using more robust constraints or post-processing on the LLM judge outputs to reduce these failures?

We limited the investigation to the same prompt to ensure consistent, comparable metric evaluation, while treating any failures (parsing or truncation) as part of the judge’s practical reliability. Future work could include prompt-optimization algorithms such as GEPA [1] and APO [2] to further improve the viability of LLM judges for this task; we have included this as a limitation and a suggestion in Section 8 of our revised submission (Lines 536-39).

  1. Agrawal, L.A., Tan, S., Soylu, D., Ziems, N., Khare, R., Opsahl-Ong, K., Singhvi, A., Shandilya, H., Ryan, M.J., Jiang, M. and Potts, C., 2026, April. Gepa: Reflective prompt evolution can outperform reinforcement learning. In International Conference on Learning Representations (Vol. 2026, pp. 8479-8565).
  2. Pryzant, R., Iter, D., Li, J., Lee, Y., Zhu, C. and Zeng, M., 2023, December. Automatic prompt optimization with “gradient descent” and beam search. In Proceedings of the 2023 conference on empirical methods in natural language processing (pp. 7957-7968).

(3/3)

(question) Choice of Regression Models: In Appendix A, you compare 16 regression models and ultimately select ExtraTrees. The results show that tree-based models generally perform better. Could you briefly analyze why tree-based models are more suitable for this task of mapping metric outputs to human scores compared to linear models or neural networks?

We selected our final regression model from the 16 choices by evaluating them on the same task: minimizing MAE loss over isotonic-calibrated features to predict the average human rating, computed on a 5-fold validation of the test-split instances. There are two reasons why tree-based models fit the metric features better. Firstly, because the training data for fitting the regression models is small (160 instances) and includes 32 highly correlated features, unregularized metrics overfit (LR mean rank 13.2), and the MLP requires significantly more data than is available (MLP mean rank 14.5). We also found that linear models extrapolate beyond the rating scale in 22 to 34 percent of factual and leak predictions, whereas the tree predictors take the mean of the training labels and are therefore bounded within the rating scale by construction.

We have additionally conducted a robustness analysis on the impact of the choice of regression model on the meta-evaluation of metrics (Appendix C.2, Lines 1030-56). Our experiments reveal that the proposed meta-evaluation framework is robust to the choice of metrics, as other regressors also align closely with the selected ExtraTrees regressor (ρ = 0.84 to ρ = 1.0 when ranking the 32 individual metrics).

(question) Discussion on the "Usability" Dimension: In Figure 2, the evaluation results for the "Usability" dimension are quite interesting, with performance differences between various metric types (including LLMs) appearing small. Does this imply that "Usability" is a more composite but weaker signal? What are your thoughts on this result?

We agree with the reviewer that usability is a composite evaluation dimension. The way it was defined for the annotators to rate, it captures the degree of editing required to make the test usable (i.e., their version of a good style-personalized text that reflects style from r on the content from o). As editing generated output can involve fixing hallucinations, removing information leakage, and/or addressing style mismatches, usability somewhat aggregates the other three dimensions. Your point about it being a weaker signal is also true, as usability has the lowest agreement (α = 0.75) among all the dimensions.

Thank you again for your constructive feedback; we hope our responses and revised submission appropriately addressed your valuable remarks.

Reviewer_iZoiOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a meta-evaluation methodology and study. The authors propose to evaluate metrics by first calibrating them against human annotation scales. Then a regression approach is used as meta-metric, i.e., the target human annotation labels are predicted from the calibration output. The authors applied the meta-evaluation in the context of a style generation task in multiple languages. The authors create the task and the multi-lingual corpus which consists of instances: a source text (e.g., Wikipedia article) and a reference style (a Reddit post). This corpus is first generated in English and then translated to various languages (automatic + human revision). The generators receive source and reference as input and the task is to generate a text with the content of the source text with the style of the reference. The outputs by the generators are annotated by human annotators across 4 dimensions (style, usability, faithfulness, leakage). Several LLMs are used and evaluate several metrics (string, embeddings, and LLM-based). The authors show that correlation of metrics with human judgements is not always a good indicative of metric performance and that their approach better accounts for labels skewness and inter-annotator agreement.

Critical strengths
  • Findings that in some cases (languages/dimensions) simpler metrics perform similar to LLM-based ones.
  • The analysis of metric ensembe and individual metric contribution is interesting.
Critical weaknesses
  • Overall the meta-evaluation setup seems looks a bit synthetic, since the corpus built to the dimensions. The combination of e.g., CNN/DM documents with Reddit styles. Similarly the "leakage" dimension is relevant to the model used for personalised style generation (SPTG) task in this study (i.e., other approaches to style generation might not use the reference and thus no leakage). The usability dimension is also unclear what evaluates, what is the usability of rewriting a CNN/DM article into Reddit style? The analysis in multiple languages is interesting; however, it feels that there is no clear trends/patterns w.r.t them, and results presented are averaged.

  • Presentation should be improved. Equation 2, what is MAE_m? How is MAE_H computed? Moreover, the previous paragraph to this equation and metric quality explanation should be improved.

Questions for authors
  • Line 025, "strong-based" => string-based
  • Line 041, what does it mean "it does not have a reference point for human"?
  • Line 139, m_p and m_q not defined.
  • Line 147, "candidate paraphrases"? are this the stylised versions?
  • Table 1, => larger font
6 comments

(1/3)

We would like to thank the reviewer for their time and consideration in providing helpful feedback on our work. We address the reviewer's remarks as follows:

Overall the meta-evaluation setup seems looks a bit synthetic, since the corpus built to the dimensions. The combination of e.g., CNN/DM documents with Reddit styles. [...]

We would like to argue against the design choice of using style personalized text generation (SPTG) as the case study for our meta-evaluation method. We selected the SPTG task to capture subjective perception of style obtained from human ratings, and do not expect to capture human behavior from our task design. The cross-domain formulation of the task, hence, by design helps capture the subjectivity in style in the human annotations by decoupling style from the content.

Prior work in style embeddings has adopted this decoupling strategy to train the embeddings to capture style and not content in the latent space [1,2]. The choice to perform text style transfer in a cross-domain setting is not novel to this work–it is a common practice adopted in the community, with works like Krishna et al. [3], who study cross-domain test style transfer across 11 unrelated domains such as Shakespeare, Bible, AAE tweets and lyrics; and Jangra et al. [4], who studied the problem of style personalized text generation across domains such as reddit, emails, lyrics and amazon reviews. Text style transfer has also studied arbitrary style transfer, without any constraints on the domains [5,6].

Our focus on meta-evaluating style metrics stems from the aim to highlight the limitations of existing meta-evaluation methods. As style is inherently a subjective attribute of language [7,8,9] and prior work in style evaluation has found existing measures inadequate for evaluating style [4], we chose style as the focus of our meta-evaluation.

  1. Wegmann, A., Schraagen, M. and Nguyen, D., 2022, May. Same author or just same topic? towards content-independent style representations. In Proceedings of the 7th Workshop on Representation Learning for NLP (pp. 249-268).
  2. Patel, A., Zhu, J., Qiu, J., Horvitz, Z., Apidianaki, M., McKeown, K. and Callison-Burch, C., 2025, April. StyleDistance: Stronger content-independent style embeddings with synthetic parallel examples. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) (pp. 8662-8685).
  3. Krishna, K., Wieting, J. and Iyyer, M., 2020, November. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 737-762).
  4. Jangra, A., Sarrafzadeh, B., Cucerzan, S., de Wynter, A. and Jauhar, S.K., 2026, October. Evaluating style-personalized text generation: Challenges and directions. In Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (pp. 3854-3878).
  5. Reif, E., Ippolito, D., Yuan, A., Coenen, A., Callison-Burch, C. and Wei, J., 2022, May. A recipe for arbitrary text style transfer with large language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) (pp. 837-848).
  6. Riley, P., Constant, N., Guo, M., Kumar, G., Uthus, D.C. and Parekh, Z., 2021, August. TextSETTR: Few-shot text style extraction and tunable targeted restyling. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 3786-3800).
  7. Fish, S., 2021. Is there a text in this class?. In Campus wars (pp. 49-56). Routledge.
  8. Irvine, J.T., 2001. " Style" as distinctiveness: the culture and ideology of linguistic differentiation (pp. 21-43). na.
  9. Campbell-Kibler, K., 2007. Accent,(ING), and the social logic of listener perceptions. American speech, 82(1), pp.32-64.

(2/3)

[…] Similarly the "leakage" dimension is relevant to the model used for personalised style generation (SPTG) task in this study (i.e., other approaches to style generation might not use the reference and thus no leakage). [...]

You are correct; the leakage dimension is an evaluation dimension relevant to the model used to generate style-personalized text. Typical text style transfer literature [1,2] does not study leakage because it was limited to text rewriting tasks due to the limitations of language-generation frameworks at the time. However, leakage is an important evaluation dimension for the SPTG task that is not captured by other style-transfer metrics, such as style accuracy and content preservation. Additionally, from a meta-evaluation perspective, leakage is an evaluation dimension not explored in prior work and serves as a reference point for our investigation, as the most objective among the four studied (α=0.85; see Table 1).

  1. Krishna, K., Wieting, J. and Iyyer, M., 2020, November. Reformulating unsupervised style transfer as paraphrase generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 737-762).
  2. Jangra, A., Nema, P. and Raghuveer, A., 2022, December. T-STAR: Truthful style transfer using AMR graph as intermediate representation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 8805-8825).

[...] The usability dimension is also unclear what evaluates, what is the usability of rewriting a CNN/DM article into Reddit style? [...]

Usability measures how much editing a writer would need to do before adopting the text; it is a property of the generated output, and not the usefulness of the writing scenario. Text style transfer is closely associated with creative writing, and in AI-collaborative creative writing, the degree of editing required for the generated output (usability) is an essential part of the writing process [1] and, hence, a measure of task success. Usability provides a holistic and complementary measure alongside the other evaluation dimensions of style, factual, and leakage, and is therefore considered in our annotation and meta-evaluation process. We have emphasized this in our revised submission (Lines 199-202).

  1. Akoury, N., Wang, S., Whiting, J., Hood, S., Peng, N. and Iyyer, M., 2020, November. Storium: A dataset and evaluation platform for machine-in-the-loop story generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (pp. 6470-6484).

Presentation should be improved. Equation 2, what is MAE_m? How is MAE_H computed? Moreover, the previous paragraph to this equation and metric quality explanation should be improved.

MAE_m is the mean absolute error for the metric ‘m’ being normalized. MAE_H (H stands for human) is an oracle estimate of how good a regressor can be; we trained the same regressor with a leave-one-out strategy, predicting the mean of two annotators’ responses using a regressor trained on the ratings of the third annotator’s responses. These lower and upper estimates help normalize and interpret the meta-evaluation Ψ values. We have clarified this in the revised submission, rewriting the suggested passages (Lines 287-307).

(question) Line 041, what does it mean "it does not have a reference point for human"?

The statement, “correlation-based meta-evaluation does not have a reference point for human agreement, making it incomparable across tasks, annotators, and datasets”, is meant to express that, by itself, the absolute value of a metric’s correlation with human ratings does not provide us with an estimate of how good that metric is, as the notion of agreement varies across tasks. For instance, an agreement of Spearman’s rho (ρ) of 0.5 can be interpreted only with respect to the annotator's agreement and task subjectivity. On a subjective task where annotators themselves only agree at about 0.55, the evaluation metric with 0.5 correlation would be considered good; whereas the same metric would be considered weak if the annotators agree at ρ=0.95. Current meta-evaluation studies do not take this human reference when evaluating against subjective human ratings. Our formulation of the regression-based meta-evaluation takes this human oracle into account (via the normalization step in Equation 2, as described in the response above).

(question) Line 147, "candidate paraphrases"? are this the stylised versions?

Yes, the candidate paraphrases are the two model-generated outputs that we ask the human raters to first provide pointwise annotations and then collect pairwise preference annotations. We have clarified this in our revised submission (Lines 149-51).

(3/3)

[...] The analysis in multiple languages is interesting; however, it feels that there is no clear trends/patterns w.r.t them, and results presented are averaged.

We have presented the averaged results due to space constraints, and provide the per-language results in Figures 12 and Tables 8 in Appendix C.

Unlike some other properties of language, such as entailment [1], style is a property of text that does not universally align across languages and is rather culturally-shaped [2]. The background of the speakers (social and cultural) shapes their language use, and thus gets reflected in their annotations [3,4]. Prior work in NLP also shows that stylistic markers such as formality and politeness are realized and perceived differently across languages [5,6].

Our findings also reflect this. Take one of the best-studied pairs, English and Frisian: both are closely related West Germanic languages that share the same script and are spoken in geographically close regions. However, their meta-evaluation varies sharply, with the m_{all} ensemble achieving Ψ = 0.00/0.18/0.32/0.14 (style/factual/leakage/usability) in English and 1.25/0.68/0.88/1.0 in Frisian (Table 8).

Linguistic proximity therefore does not predict metric quality. As we discuss in Section 7.2, this variation instead can be explained by the properties of the annotations: label skews and annotator agreement (Section 7.2, Lines 471-79). Identifying why annotators behave differently across languages would require re-annotation and qualitative and stylometric analysis; that is an undertaking worth a paper on its own, and hence we leave it to future work. We have reflected this in our revised submission as well (Lines 523-26).

  1. Conneau, A., Rinott, R., Lample, G., Williams, A., Bowman, S.R., Schwenk, H. and Stoyanov, V., 2018. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 conference on empirical methods in natural language processing (pp. 2475-2485).
  2. Connor, U. (1996a). Contrastive rhetoric: Cross-cultural aspects of second language writing. Cambridge: Cambridge University Press. doi:10.1017/CBO9781139524599
  3. Hovy, D. and Yang, D., 2021, June. The importance of modeling social factors of language: Theory and practice. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human language technologies (pp. 588-602).
  4. Plank, B., 2022, December. The “problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 conference on empirical methods in natural language processing (pp. 10671-10682).
  5. Havaldar, S., Pressimone, M., Wong, E. and Ungar, L., 2023, December. Comparing styles across languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (pp. 6775-6791).
  6. Briakou, E., Agrawal, S., Tetreault, J. and Carpuat, M., 2021, November. Evaluating the evaluation metrics for style transfer: A case study in multilingual formality transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 1321-1336).

Other typos and minor errors (from Questions for Authors)

We have addressed the typos and minor suggestions in the revised manuscript by changing “strong-based” to “string-based” in the abstract, explicitly defining the two arbitrary metrics m_p and m_q in the problem definition, and increasing the font size of Table 1.

Thank you again for your constructive feedback. If you find that our responses have resolved your main concerns, we would be grateful if you could consider updating your assessment accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.