- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a regression-based framework for meta-evaluating automatic metrics for style-personalized text generation. Metric outputs are calibrated and used to predict reliability-weighted human judgments. Held-out MAE is then normalized relative to a constant-mean predictor and an individual-annotator reference. The authors introduce a ten-language dataset containing 10,038 instances, each with two candidate generations and three-annotator judgments across style, factual consistency, reference-content leakage, and usability. The study evaluates 32 metrics and their ensembles. Its main findings are that inexpensive metrics can be competitive after supervised calibration and regression, and that ensembling is particularly beneficial for style evaluation.
Critical strengths
-
A valuable multilingual and multiply annotated evaluation resource. The dataset covers ten languages and separates style, factual consistency, leakage, and usability. Three-annotator judgments and the reported reliability statistics support a richer investigation than a single overall-quality label. The breadth of metric coverage and disclosure of judge-output failures further increase the empirical value.
-
A useful investigation of supervised predictive signal and metric combinations. The paper systematically compares inexpensive string metrics, semantic and style embeddings, LLM judges, and ensembles. Table 7 reports substantial style gains for the full ensemble, with Ψ increasing from approximately 0.43 for the strongest individual metrics covering all ten languages to 0.78. The instance-grouped folds and training-fold-only calibration and regression are appropriate safeguards. These results motivate further investigation of locally supervised combinations of metrics.
Critical weaknesses
-
The interpretation and cross-setting comparability of Ψ require stronger validation. Equation 2 uses a constant-mean baseline while evaluating absolute error. Under MAE, a median is an optimal constant predictor. For example, in a hypothetical distribution with 90% label 0 and 10% label 4, constant-mean and constant-median MAEs are 0.72 and 0.40. A uninformative constant predictor can therefore improve over the proposed reference. This does not invalidate Ψ as a score relative to a chosen mean baseline, but it limits its interpretation as performance beyond an appropriate no-information baseline. Section 7.2 acknowledges dependence on label skew and annotator agreement; however, the manuscript does not establish that the normalization supports the cross-setting comparability claimed in the introduction and problem definition. Reporting the underlying errors, normalization denominators, and sensitivity to the constant baseline would make this claim assessable. For a fixed setting with a common positive denominator, Ψ preserves the MAE ordering, so this concern primarily affects absolute interpretation and cross-setting aggregation.
-
The main comparisons need more independent and controlled validation. Appendix A describes selecting a single ExtraTrees regressor after comparing 16 approaches on the same cross-validation setting within the approximately 100-instance-per-language test split. This creates a risk of selection optimism; the fold-level fitting safeguards in Section 5.2 do not by themselves establish that method selection was independent of final performance estimation. Moreover, the comparison between correlation and the proposed framework changes several factors simultaneously: Section 5.1 uses unweighted human means, whereas Section 5.2 uses reliability-weighted targets, calibration, and nonlinear regression. The observed ranking changes therefore do not isolate which component reveals additional useful signal. Independent regressor selection, matched-target ablations, and uncertainty estimates for the headline metric differences are needed to substantiate the comparative conclusions.
Questions for authors
-
Selection and evaluation independence. Was the ExtraTrees selection in Appendix A performed independently of the data used to report the final meta-evaluation results? Please clarify the roles of the train/dev/test splits and whether annotator-reliability weights were estimated within each outer training fold. If method selection used the reported evaluation data, do the central findings persist with an independent development split or nested cross-validation?
-
Attribution of ranking changes. On matched examples and human targets, how do direction correction, simple or monotonic calibration, and nonlinear regression affect the main metric comparisons? Please specify the calibration algorithm. Results using both unweighted and reliability-weighted targets would help separate changes in the evaluation target from changes in the predictive method. Uncertainty estimates for the principal comparisons would also clarify whether the observed differences are robust.
-
Interpretation of the normalized score. Please report per-setting metric MAE, constant-mean and constant-median MAEs, the human-reference MAE, and the normalization denominator. How sensitive are the main conclusions to the baseline choice? Please also clarify the interpretation of the human reference given that metric predictions target a three-annotator consensus, while the human-reference error compares one annotator with the other two. Is clipping to [0,1] restricted to Table 8’s display, and does Table 7 aggregate unclipped values?
-
Variance-decomposition diagnostic. Equation 1 and Section 5.1 describe nonnegative incremental R², but Figure 6 contains negative values. What statistic is actually plotted, and how is it aggregated? If it is based on held-out predictions or a different signed comparison, please provide the corresponding definition and clarify whether the conclusions about unique metric contributions and ensemble complementarity remain supported.
3 comments
(2/2)
[...] Moreover, the comparison between correlation and the proposed framework changes several factors simultaneously: Section 5.1 uses unweighted human means, whereas Section 5.2 uses reliability-weighted targets, calibration, and nonlinear regression. The observed ranking changes therefore do not isolate which component reveals additional useful signal. Independent regressor selection, matched-target ablations, and uncertainty estimates for the headline metric differences are needed to substantiate the comparative conclusions.
(question) Attribution of ranking changes. On matched examples and human targets, how do direction correction, simple or monotonic calibration, and nonlinear regression affect the main metric comparisons? Please specify the calibration algorithm. Results using both unweighted and reliability-weighted targets would help separate changes in the evaluation target from changes in the predictive method. Uncertainty estimates for the principal comparisons would also clarify whether the observed differences are robust.
We used isotonic regression calibration (fit to the training folds of the 5-fold validation) to map the continuous metric predictions to the human ratings (a step function with a median of 3 to 5 distinct values). We conducted an ablation study on the two components of our proposed meta-evaluation: calibration and nonlinear regression (using LASSO as the best-ranking linear regressor from Appendix A).
Here’s the Spearman agreement of the final meta-metric and the ablated meta-metric (including the standard Speraman’s ρ meta-evaluation from Section 5.1 in the last row:
| Meta-metric | ρ Style | ρ Factual | ρ Leakage | ρ Usability |
|---|---|---|---|---|
| Ψ | 1.00 | 1.00 | 1.00 | 1.00 |
| − calibration (ExtraTrees, raw metric values) | 0.99 | 0.98 | 0.96 | 1.00 |
| − tree-based regression (Lasso, raw metric values) | 0.88 | 0.92 | 0.91 | 0.92 |
| Spearman |ρ| (Section 5.1) | 0.63 | 0.91 | 0.88 | 0.85 |
The table shows the difference in agreement between an ablated meta-evaluation based on single-metric rankings and the proposed meta-evaluation metric. Calibration does not affect the tree-based regressors as much (ρ = 0.96 to 1.00), since trees are invariant to monotone input transforms such as the isotonic calibration. Replacing trees with a linear fit shifts the ranking by ρ = 0.88 to 0.92 and removes the ensemble gains
We have additionally conducted a sensitivity analysis of the meta-evaluation measurement with respect to the size of the calibration data in Appendix C.1, as well as assessed robustness to the choice of regression model (when applied to the calibrated data) in Appendix C.2 of the revised submission.
(question) Variance-decomposition diagnostic. Equation 1 and Section 5.1 describe nonnegative incremental R², but Figure 6 contains negative values. What statistic is actually plotted, and how is it aggregated? If it is based on held-out predictions or a different signed comparison, please provide the corresponding definition and clarify whether the conclusions about unique metric contributions and ensemble complementarity remain supported.
The plot shows the average difference between two non-negative R^2 terms across all other evaluation metrics. This difference reflects the average unique variance a metric captures relative to all other metrics. Formally, for a metric p, it is computed as . We have clarified how this aggregate R^2 is computed in the captions of Figures 1 and 9 in our revised submission.
[...] Is clipping to [0,1] restricted to Table 8’s display, and does Table 7 aggregate unclipped values?
We clipped the values in Table 8 for readability. But we have now updated it to include the unclipped values in the revised submission. You are correct in understanding that Table 7’s aggregate was done on the unclipped values.
Thank you again for your constructive feedback. If you find that our responses have resolved your main concerns, we would be grateful if you could consider updating your assessment accordingly.
Thank you for your response. I appreciate the clarification and find it reasonable. However, with respect, I would like to maintain my original evaluation.
(1/2)
We would like to thank the reviewer for their time and consideration in providing helpful feedback on our work. We address the reviewer's remarks as follows:
We have performed the analysis using the median as the trivial baseline choice in the normalization step, on the same k-fold validation splits as in the original submission. While the change in baseline affects the raw Ψ values, it does not change the rankings of metrics obtained from the meta-evaluation. The order of metrics is almost identical under the median baseline compared to the reported rankings with the mean baselines, with Spearman rank correlations of 0.93 (style), 0.95 (factual), 0.98 (leakage), and 0.96 (usability).
Using the mean across the training folds of the data is also a viable baseline for our meta-evaluation and does not affect our findings. Therefore, we have decided to keep the main results from our work unchanged, and have included the analysis of the influence of changing the normalization baseline from mean to median in Appendix C.3 (Lines 1058-86).
We conducted extensive analysis across a wide range of regression models (7 additional models from the four regression families explored earlier) and show that the proposed meta-evaluation framework is robust to the choice of regression model. Overall, the ranking of the 32 individual metrics remains consistent across the regression model’s Ψ values, with a rank correlation of Spearman’s ρ=0.84 to ρ=1.0 across regression models. We have included a detailed analysis in Appendix C.2 of our revised submission (Figure 6 and Lines 1030-56).