acceptodds
Under review as a conference paper at ICLR 2027

EcomWriteBench: A Real-World Benchmark for E-commerce Writing Tasks

Abstract

Large language models (LLMs) are increasingly used throughout e-commerce workflows to generate content that informs purchasing decisions and supports customer interactions. Evaluating these systems requires accounting for user needs, product evidence, structural constraints, and business requirements that are not fully captured by general-purpose writing benchmarks. We introduce EcomWriteBench, a real-world benchmark containing 3,861 queries from 69 e-commerce writing tasks across 13 categories and five languages. Through task-level extraction, cross-task consolidation, and refinement guided by human feedback, EcomWriteBench constructs a shared pool of 37 abilities and identifies the subset applicable to each task. Ability-wise scoring enables us to compare state-of-the-art LLMs on EcomWriteBench and characterize their relative strengths and weaknesses across fine-grained e-commerce writing abilities. We further propose a preference-calibrated weight-learning framework that converts human pairwise preferences into interpretable scoring rubrics through global, category-level, and task-level aggregation weights. This framework provides a reusable approach for translating human preferences into automated evaluation by combining ability discovery, ability-wise scoring, and preference-calibrated aggregation, and can potentially be applied to a broad range of benchmarks beyond EcomWriteBench. Empirically, our scoring framework achieves 90.42% agreement with human preferences, compared with 78.54% for conventional LLM-as-a-judge scoring.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, E-commerce Writing, Preference-calibrated Rubric Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jif7OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces EcomWriteBench, a multilingual benchmark for e-commerce writing. It develops a framework that identifies writing task-relevant abilities and learns hierarchical aggregation weights from human pairwise preferences to provide fine-grained evaluation of LLM-generated content. Although the benchmark addresses a relevant practical problem, the methodological novelty and experimental validation are insufficient to support the paper's main claims.

Critical strengths
  1. This paper addresses an important problem in evaluating LLM-generated content for real-world e-commerce applications.
  2. The benchmark covers diverse e-commerce scenarios and multiple languages, providing potentially useful evaluation resources.
  3. The proposed ability-wise evaluation framework offers more interpretable assessments than conventional holistic scoring approaches.
  4. Experiments include various LLM configurations and provide detailed comparisons across different evaluation dimensions.
Critical weaknesses
  1. The proposed method largely relies on existing techniques, including LLM-based ability extraction, fine-grained evaluation, and preference-based weight learning, with limited methodological contributions.
  2. The 90.42% agreement reported in Table 1 is based on the same preference pairs used for training, making it difficult to assess whether the improvement over the baselines generalises to unseen queries.
  3. Table 3 shows only a marginal improvement from global to category-level weights (84.64% to 84.84%), suggesting that the proposed hierarchical weighting offers little benefit for unseen tasks.
  4. The benchmark queries have been rewritten, translated, and filtered, but it is unclear whether the resulting dataset still reflects real-world e-commerce scenarios.
  5. The evaluation relies heavily on LLM judges, but the paper lacks sufficient evidence that their ability-level scores are consistent with human assessments.
  6. The performance differences among the top models in Table 4 are very small, and statistical significance tests are needed to support the reported rankings.
Questions for authors

See above

6 comments

We thank the reviewer for acknowledging the practicability of our proposed benchmark and our interpretable evaluation rubrics. We address your concerns below.


W1. The proposed method largely relies on existing techniques, including LLM-based ability extraction, fine-grained evaluation, and preference-based weight learning, with limited methodological contributions.

A. To the best of our knowledge, we are the first to jointly formulate direct preference-to-rubric learning over a shared fine-grained ability space and hierarchical weight adaptation across heterogeneous evaluation tasks. Our key methodological contribution is a unified framework that converts human pairwise preferences directly into interpretable, task-adaptive scoring rubrics, rather than merely learning preference predictions or relying on human-assigned ability-level ratings.

Specifically, our method differs from existing preference-calibrated evaluation approaches in three respects (Appendix A.2.5, Table 12).

First, direct preference-to-rubric learning optimizes non-negative weights over fine-grained abilities using human pairwise preferences, producing explicit response-level scoring functions without requiring human ability-level annotations.

Second, cross-task rubric transfer is enabled by a shared vocabulary of 37 abilities across 69 heterogeneous tasks. Global and category-level weights can be applied to unseen tasks without target-task preference annotations (Table 3), whereas HD-Eval and AutoMetrics construct task-specific evaluation criteria or metrics that do not directly transfer.

Third, hierarchical weight sharing enables effective adaptation under limited preference supervision. Our new low-resource experiments demonstrate that hierarchical learning outperforms independently fitted task weights by 5.59 percentage points when only 5% of training queries are retained (Table 13; see Reviewer XkPY W5).

EcomWriteBenchLLM-RubricMetaMetricsHD-EvalAutoMetrics
Native supervisionPairwise preferencesPer-annotator ratingsScalar ratings or pairwiseScalar ratingsScalar ratings
Token log-probabilitiesNot neededNeededNot neededNot neededNot needed
Features shared across tasksYesNoYesNoNo
Preparation for each taskAbility selection1 LLM callNone1,370 LLM calls1,341 LLM calls
Learned aggregationNon-negative hierarchical weightsNeural networkLinear weights or treesWhite-box modelSparse linear model

We further validate the aggregation method itself: under identical three-judge ability-scoring features, our approach achieves 88.96% held-out preference agreement, outperforming the strongest alternative aggregator (86.91%) by 2.05 percentage points (Table 11; see Reviewer XkPY Q3). These results demonstrate the empirical value of our integrated preference-calibrated rubric-learning framework beyond its individual components.


W2. The 90.42% agreement reported in Table 1 is based on the same preference pairs used for training, making it difficult to assess whether the improvement over the baselines generalises to unseen queries.

A. We conduct additional evaluation under the held-out five-fold cross validation. Conventional LLM-as-a-judge scoring and uniform aggregation have no fitted parameters and are therefore already out-of-sample. Four compared preference-learning methods are also reported. The results are shown below (and also Table 1 in our revised manuscript). We show that our EcomWriteBench rubric outperforms all baselines and achieves the highest held-out agreement.

MethodAgreement ↑Gain (EcomWrite − Baseline) ↑
Conventional LLM-as-a-judge0.7854+10.42 pt
Uniform ability aggregation0.8252+6.44 pt
AutoMetrics0.7406+14.90 pt
MetaMetrics0.7558+13.38 pt
HD-Eval0.8241+6.55 pt
LLM-Rubric0.8634+2.62 pt
EcomWriteBench rubric0.8896—

W3. Table 3 shows only a marginal improvement from global to category-level weights (84.64% to 84.84%), suggesting that the proposed hierarchical weighting offers little benefit for unseen tasks.

A. We acknowledge that the 0.19-point improvement from global to category-level weights is not statistically significant (95% CI [−0.20, 0.59], bootstrap over tasks). However, Table 3 is intended to compare two variants of our proposed framework for unseen-task transfer, rather than to demonstrate the superiority of category-level weights over global weights.

To further validate unseen-task generalization, we additionally introduce uniform ability aggregation as a baseline in Table 3. Both global and category-level weights transfer to unseen tasks without task-specific preference annotations and outperform uniform aggregation (82.59%) by 2.05 points (95% CI [1.16, 2.95]) and 2.24 points, respectively. These results support the effectiveness of preference-calibrated weights for unseen-task evaluation.

WeightsAgreement ↑Gain vs. Uniform ↑
Uniform0.8259—
Global0.8464+2.05 pt
Category0.8484+2.24 pt

The marginal difference between global and category-level weights is also consistent with Figure 7, where the selected category-to-global regularization coefficient is λ=0\lambda=0. This suggests that category-level preference supervision may already be sufficient to learn reliable weights, reducing the need for shrinkage toward global weights.


W4. The benchmark queries have been rewritten, translated, and filtered, but it is unclear whether the resulting dataset still reflects real-world e-commerce scenarios.

A. EcomWriteBench is constructed from real-world e-commerce workflows, and our data-processing pipeline preserves the original task objectives and evaluation requirements (Appendix A.1.1–A.1.2). Specifically, (1) Rewriting replaces privacy-sensitive and organization-specific information while preserving task variables, factual dependencies, numerical constraints, and output requirements. (2) Translation transfer queries from multi-lingual source tasks to equally distributed multi-lingual queries within each task, with human verification of semantic fidelity and task transferability. (3) Filtering and sampling remove malformed or unusable queries, followed by random sampling within each task under task-specific quotas.

Importantly, all 69 original tasks across 13 e-commerce categories are retained, preserving the diversity of real-world business objectives and evaluation requirements. These procedures maintain the benchmark's grounding in authentic e-commerce workflows while ensuring data quality and multilingual coverage.


W5. The evaluation relies heavily on LLM judges, but the paper lacks sufficient evidence that their ability-level scores are consistent with human assessments.

A. We would like to clarify that our framework does not require LLM-generated ability scores to independently match human-assigned scores for each ability. Instead, a key advantage of our approach is that it directly calibrates the aggregation of ability scores using human pairwise preferences, without requiring costly human annotations at the ability level. Collecting reliable human scores for every applicable ability across 69 heterogeneous tasks and thousands of model responses would require substantial annotation effort and is infeasible within the rebuttal period.

Our primary objective is to align the final rubric scores with human preferences, rather than independently reproduce human judgments on every ability dimension. We directly evaluate this alignment through preference agreement on held-out queries. Furthermore, our additional single-judge experiments (Table 15, Appendix A.2.9) show that task-specific calibration consistently improves agreement over uniform aggregation across all three judges, with gains of 6.23–8.67 percentage points.

These results support the effectiveness and robustness of preference-calibrated aggregation without requiring human ability-level supervision. While direct human validation of individual ability scores could provide complementary evidence for their diagnostic interpretability, it is not a prerequisite for learning human-aligned aggregation weights.

W6. The performance differences among the top models in Table 4 are very small, and statistical significance tests are needed to support the reported rankings.

A. We revise Table 4 and now report 95% confidence intervals for all overall scores, from a bootstrap over tasks. GPT-5.6-Sol, Gemini-3.1-Pro, GLM-5.3 and Kimi-K3 form a leading group, and its gap to the next configuration, Claude-Opus-4.8, is significant (0.122, 95% CI [0.081, 0.164]) (Section 4.4).

ModelReasoningOverall score (0–10)95% CI
GPT-5.6-Solmax9.581[9.499, 9.660]
Gemini-3.1-Prohigh9.577[9.500, 9.653]
GLM-5.3max9.539[9.454, 9.624]
Kimi-K3max9.535[9.463, 9.608]
Claude-Opus-4.8max9.414[9.327, 9.503]
Gemma-4-31Benabled9.379[9.285, 9.475]
Qwen3.8-27Bxhigh9.338[9.230, 9.446]
Claude-Sonnet-5max9.313[9.224, 9.406]
MiniMax-M3enabled9.215[9.056, 9.357]
Gemma-4-31Bdisabled9.184[9.066, 9.302]
Qwen3.5-397B-A17Benabled9.180[9.043, 9.314]
Qwen3.6-35B-A3Benabled9.179[9.054, 9.307]
DeepSeek-V4-Promax8.961[8.822, 9.100]
Qwen3.5-397B-A17Bdisabled8.937[8.780, 9.090]
DeepSeek-V4-Flashmax7.785[7.438, 8.110]

Thanks for your insightful and constructive comments. We sincerely hope that our explanations and additional experiments solve your concerns.

Reviewer_4GCGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces EcomWriteBench, a benchmark containing 3,861 queries across 69 e-commerce writing tasks and five languages. It constructs 37 fine-grained evaluation abilities through task-level extraction and human-guided refinement. The authors further propose a hierarchical preference-calibration method that learns global, category-level, and task-specific scoring weights from human pairwise preferences. Experiments on 15 LLM configurations demonstrate useful ability-level differences and improved agreement with human preferences.

Critical strengths
  1. The benchmark covers diverse real-world e-commerce writing tasks across multiple languages. The practical motivation and task coverage are valuable.

  2. The ability-based evaluation framework provides more informative diagnostics than a single overall score. The 37-ability taxonomy and task-specific rubrics are reasonably designed.

  3. The preference-calibration experiments provide useful evidence for task-specific weighting. The five-fold evaluation shows improved agreement on held-out queries from observed tasks.

Critical weaknesses
  1. The methodological novelty is not sufficiently distinguished from existing preference-calibrated evaluation methods, such as LLM-Rubric, MetaMetrics, HD-Eval, and AutoMetrics.

  2. The reported 90.42% preference agreement is measured on the calibration data. Although cross-validation is provided, the paper does not report the same held-out comparison against conventional LLM-as-a-judge and stronger learned evaluation baselines.

  3. The generalization to unseen tasks is limited. Category-level weights achieve 84.84% agreement compared with 84.64% for global weights, providing little evidence that category-specific calibration transfers meaningfully to new tasks.

  4. The evaluation remains dependent on LLM-generated ability scores. The reported judge agreement is moderate for several pairs, especially on subjective criteria.

  5. The main preference-calibration setting retains only unanimously preferred response pairs. This may favor relatively unambiguous comparisons, while the reliability of the learned rubrics on subjective or disputed preferences remains less clear.

Questions for authors
  1. Provide comparisons with stronger preference-calibrated evaluation methods.

  2. Clarify the held-out improvements over conventional LLM-as-a-judge scoring.

  3. Discuss the limited generalization gains on unseen tasks.

  4. Provide further evidence for robustness to judge selection and preference disagreement.

7 comments

We thank the reviewer for acknowledging the practicability of our proposed benchmark and our informative scoring rubrics. We address your concerns below.


W1. The methodological novelty is not sufficiently distinguished from existing preference-calibrated evaluation methods, such as LLM-Rubric, MetaMetrics, HD-Eval, and AutoMetrics.

A. To the best of our knowledge, we are the first to jointly formulate direct preference-to-rubric learning over a shared fine-grained ability space and hierarchical weight adaptation across heterogeneous evaluation tasks. Our key methodological contribution is a unified framework that directly converts human pairwise preferences into interpretable, task-adaptive scoring rubrics, rather than merely predicting preferences or requiring human-assigned ability-level ratings.

Our method differs from existing approaches in three respects (Appendix A.2.5, Table 12).

First, direct preference-to-rubric learning optimizes non-negative ability weights from human pairwise preferences, producing explicit response-level scoring functions without requiring human ability-level annotations.

Second, cross-task rubric transfer is enabled by a shared vocabulary of 37 abilities across 69 heterogeneous tasks. Global and category-level weights can be applied to unseen tasks without target-task preference annotations (Table 3), whereas HD-Eval and AutoMetrics construct task-specific criteria or metrics that do not directly transfer.

Third, hierarchical weight sharing enables effective task-specific adaptation under limited preference supervision. Our new low-resource experiments show that hierarchical learning outperforms independently fitted task weights by 5.59 percentage points with only 5% of training queries (Table 13; see Reviewer XkPY W5).

EcomWriteBenchLLM-RubricMetaMetricsHD-EvalAutoMetrics
Native supervisionPairwise preferencesPer-annotator ratingsScalar ratings or pairwiseScalar ratingsScalar ratings
Token log-probabilitiesNot neededNeededNot neededNot neededNot needed
Features shared across tasksYesNoYesNoNo
Preparation for each taskAbility selection1 LLM callNone1,370 LLM calls1,341 LLM calls
Learned aggregationNon-negative hierarchical weightsNeural networkLinear weights or treesWhite-box modelSparse linear model

The effectiveness of our aggregation method is further validated under controlled comparisons: using identical three-judge ability-scoring features, our framework achieves 88.96% held-out preference agreement, outperforming the strongest alternative aggregator (86.91%) by 2.05 percentage points (Table 11; see Reviewer XkPY Q3). End-to-end comparisons also demonstrate superiority over all four existing evaluation methods (Table 1; see Reviewer XkPY W2/W3).


W2. The reported 90.42% preference agreement is measured on the calibration data. Although cross-validation is provided, the paper does not report the same held-out comparison against conventional LLM-as-a-judge and stronger learned evaluation baselines.

A. We conduct additional evaluation under the held-out five-fold cross validation. Conventional LLM-as-a-judge scoring and uniform aggregation have no fitted parameters and are therefore already out-of-sample. Four compared preference-learning methods are also reported. The results are shown below (and also Table 1 in our revised manuscript). We show that our EcomWriteBench rubric outperforms all baselines and achieves the highest held-out agreement.

MethodAgreement ↑Gain (EcomWrite − Baseline) ↑
Conventional LLM-as-a-judge0.7854+10.42 pt
Uniform ability aggregation0.8252+6.44 pt
AutoMetrics0.7406+14.90 pt
MetaMetrics0.7558+13.38 pt
HD-Eval0.8241+6.55 pt
LLM-Rubric0.8634+2.62 pt
EcomWriteBench rubric0.8896—

W3. The generalization to unseen tasks is limited. Category-level weights achieve 84.84% agreement compared with 84.64% for global weights, providing little evidence that category-specific calibration transfers meaningfully to new tasks.

A. We acknowledge that the 0.19-point improvement from global to category-level weights is not statistically significant (95% CI [−0.20, 0.59], bootstrap over tasks). However, Table 3 is intended to compare two variants of our proposed framework for unseen-task transfer, rather than to demonstrate the superiority of category-level weights over global weights.

To further validate unseen-task generalization, we additionally introduce uniform ability aggregation as a baseline in Table 3. Both global and category-level weights transfer to unseen tasks without task-specific preference annotations and outperform uniform aggregation (82.59%) by 2.05 points (95% CI [1.16, 2.95]) and 2.24 points, respectively. These results support the effectiveness of preference-calibrated weights for unseen-task evaluation.

WeightsAgreement ↑Gain vs. Uniform ↑
Uniform0.8259—
Global0.8464+2.05 pt
Category0.8484+2.24 pt

The marginal difference between global and category-level weights is also consistent with Figure 7, where the selected category-to-global regularization coefficient is λ=0\lambda=0. This suggests that category-level preference supervision may already be sufficient to learn reliable weights, reducing the need for shrinkage toward global weights.


W4. The evaluation remains dependent on LLM-generated ability scores. The reported judge agreement is moderate for several pairs, especially on subjective criteria.

A. We acknowledge that LLM judges exhibit greater disagreement on subjective abilities. However, our framework does not require individual LLM-generated ability scores to independently match human-assigned scores. Instead, we directly calibrate their aggregation using human pairwise preferences, avoiding the substantial cost of collecting human ability-level annotations. Our primary objective is therefore to achieve reliable agreement with human preferences at the aggregated evaluation level.

To assess robustness to judge selection, we conduct additional single-judge experiments (new Table 15, Appendix A.2.9). Task-specific calibration consistently outperforms uniform aggregation across all three judges, improving held-out preference agreement by 6.23–8.67 percentage points. Moreover, single-judge model rankings correlate strongly with the three-judge ranking (Spearman 0.94–0.99).

JudgeConventionalUniformGlobalCategoryTask
Gemini-3.1-Pro0.71680.75460.79490.81340.8306
Qwen3.6-35B-A3B0.67500.73290.77840.80100.8196
Gemma-4-31B0.65230.74350.77890.79240.8058

These results show that the effectiveness of preference-calibrated aggregation and the resulting model-level conclusions are robust to the choice of LLM judge, despite disagreement in intermediate ability scores. While human validation of individual ability scores could further assess their diagnostic interpretability, such annotations are not required for learning human-aligned aggregation weights.


W5. The main preference-calibration setting retains only unanimously preferred response pairs. This may favor relatively unambiguous comparisons, while the reliability of the learned rubrics on subjective or disputed preferences remains less clear.

A. To assess robustness beyond unanimous preferences, we evaluate weights trained exclusively on strict pairs against the broader majority-preference set, which also includes disputed comparisons (Appendix A.2.4, Table 9). Task-specific weights achieve 77.31% held-out agreement, significantly outperforming uniform aggregation (72.71%) by 4.60 percentage points (95% CI [3.50, 5.81]). This demonstrates that preference-calibrated aggregation remains effective beyond the strict preference setting.

PairsConventionalUniformGlobalCategoryTask
Majority set0.69220.72710.75110.76280.7731

Q1. Provide comparisons with stronger preference-calibrated evaluation methods.

A. Please see our response to W2.


Q2. Clarify the held-out improvements over conventional LLM-as-a-judge scoring.

A. Please see our response to W2.


Q3. Discuss the limited generalization gains on unseen tasks.

A. Please see our response to W3.


Q4. Provide further evidence for robustness to judge selection and preference disagreement.

A. Please see our responses to W4 and W5.


Thanks for your insightful and constructive comments. We sincerely hope that our explanations and additional experiments solve your concerns.

Thank you for the additional experiments. They help address my concerns about judge selection and evaluation beyond unanimous preferences. However, the reliability of the individual ability scores as diagnostic measures remains unclear, and I would still like to see separate results on disputed preference pairs. The evidence for an additional benefit from category-level weighting is also limited. These remaining concerns lead me to keep my original score.

Reviewer_XkPYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper introduces EcomWriteBench, a benchmark of 3,861 queries across 69 e-commerce writing tasks, 13 categories, and five languages.
  • It proposes a shared vocabulary of 37 abilities, grouped into 6 families, for fine-grained evaluation focused on writing and generation tasks specific to e-commerce.
  • A preference calibrated weight learning framework is introduced that converts human pairwise preferences to learn global, category-level, and task-specific aggregation weights that can provide interpretable scoring rubrics.
  • 15 LLM configurations using three LLM judges and deterministic checks are evaluated.
  • Results show 90.42% in-sample preference agreement, with additional cross-validation and unseen-task experiments.
Critical strengths
  • This paper introduces a practical benchmark with diverse task coverage such as product content, recommendations, customer service, logistics, and merchant operations.
  • This paper also discusses ability-wise scoring provides useful diagnostics beyond aggregate model rankings. It also provides an interpretable scoring framework with reasonable mathematical foundations.
  • Multilingual coverage increases its potential applicability however the exact split is not provided.
  • L1 and L2 optimization formulations appear mathematically sound and non-negative weights provide interpretable scoring rubrics.
  • The paper includes detailed ability definitions, scoring prompts, ablations, and sensitivity analyses and evaluates global, category-level, and task-specific preference alignment.
Critical weaknesses
  • The main methodological claims are supported by rather limited experimental evidence.
  • The headline 90.42% agreement is measured on the calibration pairs, not held-out data.
  • Missing held-out comparisons against conventional LLM judges, uniform weighting, and stronger preference-learning baselines.
  • Unseen-task generalization improves by only ~0.2 percentage points (Table 3), without significance testing.
  • Selected λ=0\lambda=0 raises questions about the contribution of hierarchical regularization.
  • Model rankings lack confidence intervals, despite very small differences among leading models.
  • Benchmark characterization and multilingual analysis is insufficient.
  • Table 4 reports aggregate results without per-language breakdowns.
  • Translation quality, original language distribution, and cross-language task difficulty are not adequately analyzed.
  • Query filtering and original-index sampling may introduce selection bias.
  • Category-specific ability weights in Figure 3 lack sufficient qualitative explanations.
  • Human preference annotation consistency and judge reliability need stronger validation.
  • To claim that the framework substantially improves upon existing rubric-based evaluation methods might need more evidence.
Questions for authors
  • Are the 3,861 queries distributed across languages or translated from common source queries? Please report per-language sample sizes, performance and judge consistency.

  • Why do Content & Intent dominate product authoring and customer service, while Style & Language dominate user reviews and repurchase tasks? Can representative examples and weight-stability analyses be provided?

  • Report held-out agreement and raw score distributions for holistic LLM judging, uniform aggregation, and learned rubrics using identical splits. How does the method compare against simpler preference-learning approaches?

  • In Table 3, Unseen-task generalization, is the ~0.2-percentage-point improvement statistically significant? Provide confidence intervals and category-wise results.

  • Report model evaluation results separately for all five languages, with confidence intervals? Are the overall model rankings consistent across languages?

  • How sensitive are results to annotator disagreement, judge selection, and preference filtering? What benefit does hierarchical regularization provide when the selected λ=0\lambda=0?

7 comments

We thank the reviewer for acknowledging the practicability of our proposed benchmark and our interpretable scoring rubrics. We address your concerns below.


W1. The main methodological claims are supported by rather limited experimental evidence.

A. We substantially strengthen empirical validation through three sets of experiments: (1) Preference-calibrated rubric learning: 88.96% held-out agreement, outperforming existing evaluation methods by up to 14.90 points (W2/W3) and the strongest matched-feature aggregator by 2.05 points (Q3); (2) Cross-task transfer: a significant 2.05-point improvement over uniform aggregation on unseen tasks (W4); and (3) Hierarchical adaptation: a 5.59-point improvement over independent task-level fitting with only 5% of training queries (W5).

We further provide model-score confidence intervals (W6), multilingual evaluations (W7/W8), and annotation and judge robustness analyses (W12). Together, these results provide stronger empirical support for our framework.


W2/W3. The headline 90.42% agreement is measured on the calibration pairs, not held-out data. Missing held-out comparisons against conventional LLM judges, uniform weighting, and stronger preference-learning baselines.

A. We conduct additional evaluation under the held-out five-fold cross validation. Conventional LLM-as-a-judge scoring and uniform aggregation have no fitted parameters and are therefore already out-of-sample. Four compared preference-learning methods are also reported. The results are shown below (and also Table 1 in our revised manuscript). We show that our EcomWriteBench rubric outperforms all baselines and achieves the highest held-out agreement.

MethodAgreement ↑Gain (EcomWrite − Baseline) ↑
Conventional LLM-as-a-judge0.7854+10.42 pt
Uniform ability aggregation0.8252+6.44 pt
AutoMetrics0.7406+14.90 pt
MetaMetrics0.7558+13.38 pt
HD-Eval0.8241+6.55 pt
LLM-Rubric0.8634+2.62 pt
EcomWriteBench rubric0.8896—

W4. Unseen-task generalization improves by only ~0.2 percentage points (Table 3), without significance testing.

A. We acknowledge that the 0.19-point improvement from global to category-level weights is not statistically significant (95% CI [−0.20, 0.59], bootstrap over tasks). However, Table 3 is intended to compare two variants of our proposed framework for unseen-task transfer, rather than to demonstrate the superiority of category-level weights over global weights.

To further validate unseen-task generalization, we additionally introduce uniform ability aggregation as a baseline in Table 3. Both global and category-level weights transfer to unseen tasks without task-specific preference annotations and outperform uniform aggregation (82.59%) by 2.05 points (95% CI [1.16, 2.95]) and 2.24 points, respectively. These results support the effectiveness of preference-calibrated weights for unseen-task evaluation.

WeightsAgreement ↑Gain vs. Uniform ↑
Uniform0.8259—
Global0.8464+2.05 pt
Category0.8484+2.24 pt

The marginal difference between global and category-level weights is also consistent with Figure 7, where the selected category-to-global regularization coefficient is λ=0\lambda=0. This suggests that category-level preference supervision may already be sufficient to learn reliable weights, reducing the need for shrinkage toward global weights.

W5. Selected λ=0\lambda=0 raises questions about the contribution of hierarchical regularization.

A. The selected λ=0\lambda=0 suggests that category-level preference supervision may already be sufficient to learn reliable weights without shrinkage toward global weights. To further investigate the benefit of hierarchical regularization, we conduct additional experiments under limited preference supervision.

Low-resource experimental setting. We retain 5%, 10%, 25%, and 50% of the training queries within each task and compare three approaches: (1) Global, which learns shared weights across all tasks; (2) Independent, which fits task-specific weights without hierarchical regularization; and (3) Hierarchical, which learns task-specific weights with global-to-category and category-to-task regularization. All methods are evaluated on held-out queries.

Retained queriesPairs per taskGlobalIndependentHierarchicalSelected (λ,μ)(\lambda,\mu)
5%110.83450.78450.8404(3, 3)
10%210.84500.80410.8536(1, 1)
25%520.85110.84000.8662(3, 1)
50%1050.85060.85990.8726(1, 0.3)

The results demonstrate that hierarchical regularization is particularly beneficial under limited supervision. Compared with independently fitted task weights, its improvement increases from 0.4 percentage points with full annotations to 5.6 points with only 5% of training queries. Moreover, positive sharing coefficients are selected across all low-resource settings, indicating that leveraging higher-level weights becomes more beneficial when task-level supervision is scarce. These findings support the value of adaptive hierarchical weight sharing, even though category-to-global shrinkage is unnecessary in the full-data setting.


W6. Model rankings lack confidence intervals, despite very small differences among leading models.

A. We revise Table 4 and now report 95% confidence intervals for all overall scores, from a bootstrap over tasks. GPT-5.6-Sol, Gemini-3.1-Pro, GLM-5.3 and Kimi-K3 form a leading group, and its gap to the next configuration, Claude-Opus-4.8, is significant (0.122, 95% CI [0.081, 0.164]) (Section 4.4).

ModelReasoningOverall score (0–10)95% CI
GPT-5.6-Solmax9.581[9.499, 9.660]
Gemini-3.1-Prohigh9.577[9.500, 9.653]
GLM-5.3max9.539[9.454, 9.624]
Kimi-K3max9.535[9.463, 9.608]
Claude-Opus-4.8max9.414[9.327, 9.503]
Gemma-4-31Benabled9.379[9.285, 9.475]
Qwen3.8-27Bxhigh9.338[9.230, 9.446]
Claude-Sonnet-5max9.313[9.224, 9.406]
MiniMax-M3enabled9.215[9.056, 9.357]
Gemma-4-31Bdisabled9.184[9.066, 9.302]
Qwen3.5-397B-A17Benabled9.180[9.043, 9.314]
Qwen3.6-35B-A3Benabled9.179[9.054, 9.307]
DeepSeek-V4-Promax8.961[8.822, 9.100]
Qwen3.5-397B-A17Bdisabled8.937[8.780, 9.090]
DeepSeek-V4-Flashmax7.785[7.438, 8.110]

W7/W8. Benchmark characterization and multilingual analysis is insufficient. Table 4 reports aggregate results without per-language breakdowns.

A. The benchmark is characterized by category, task, language, and query length (refer to Figure 2 and also Appendix A.1.1–A.1.3). To strengthen the multilingual analysis, we additionally report per-language results for all 15 model configurations in the table below (and also Table 6, Appendix A.2.2 in our revised), including 95% confidence intervals obtained by bootstrapping over 69 tasks (20,000 resamples).

Each task originates in its own source language. Its queries are distributed approximately evenly across five languages (Chinese, English, German, French, and Dutch), with translation applied where necessary. All translations were verified by human reviewers for semantic fidelity and task transferability. The new results provide a more fine-grained characterization of multilingual model performance and enable comparisons beyond the aggregate scores in Table 4.

ModelReasoningOverallChineseEnglishGermanFrenchDutch
GPT-5.6-Solmax9.5819.689 ± 0.0739.638 ± 0.0879.567 ± 0.0919.516 ± 0.0989.496 ± 0.106
Gemini-3.1-Prohigh9.5779.683 ± 0.0729.625 ± 0.0879.572 ± 0.0869.529 ± 0.0919.475 ± 0.091
GLM-5.3max9.5399.711 ± 0.0679.630 ± 0.0839.508 ± 0.1139.455 ± 0.1019.393 ± 0.106
Kimi-K3max9.5359.621 ± 0.0759.603 ± 0.0829.497 ± 0.0849.520 ± 0.0799.436 ± 0.092
Claude-Opus-4.8max9.4149.587 ± 0.0739.478 ± 0.0919.367 ± 0.1089.359 ± 0.1009.276 ± 0.113
Gemma-4-31Benabled9.3799.583 ± 0.0789.421 ± 0.0999.360 ± 0.1129.292 ± 0.1109.240 ± 0.126
Qwen3.8-27Bxhigh9.3389.630 ± 0.0699.531 ± 0.1029.235 ± 0.1369.237 ± 0.1399.056 ± 0.153
Claude-Sonnet-5max9.3139.415 ± 0.0929.392 ± 0.0949.305 ± 0.1149.284 ± 0.1029.169 ± 0.117
MiniMax-M3enabled9.2159.352 ± 0.1429.290 ± 0.1519.162 ± 0.1829.123 ± 0.1849.146 ± 0.178
Gemma-4-31Bdisabled9.1849.445 ± 0.1049.197 ± 0.1279.155 ± 0.1419.133 ± 0.1278.988 ± 0.145
Qwen3.5-397B-A17Benabled9.1809.520 ± 0.0829.366 ± 0.1248.992 ± 0.1809.058 ± 0.1838.961 ± 0.174
Qwen3.6-35B-A3Benabled9.1799.546 ± 0.0879.357 ± 0.1208.983 ± 0.1759.085 ± 0.1548.922 ± 0.167
DeepSeek-V4-Promax8.9619.197 ± 0.1299.029 ± 0.1458.876 ± 0.1758.905 ± 0.1358.800 ± 0.171
Qwen3.5-397B-A17Bdisabled8.9379.281 ± 0.1288.991 ± 0.1828.811 ± 0.1818.870 ± 0.1928.731 ± 0.174
DeepSeek-V4-Flashmax7.7858.349 ± 0.2158.057 ± 0.2867.563 ± 0.3957.610 ± 0.3967.344 ± 0.517

W9. Translation quality, original language distribution, and cross-language task difficulty are not adequately analyzed.

A. Our source language is determined at the task level, with each task originating from a single language. The overall original distribution over the five languages are relatively equal.

For dataset construction, queries within each task are distributed across all five evaluation languages through translation, resulting in approximately balanced coverage: Chinese (774), English (773), German (772), French (772), and Dutch (770). All translations were verified by human reviewers for semantic fidelity and preservation of task-specific requirements, including factual dependencies and output constraints.

For cross-language analysis, since all five language subsets share the same task definitions, evaluation objectives, and constraints, the benchmark is designed to maintain comparable task difficulty across languages. Please also refer to W7 for our additionally reported per-language performance. The strong cross-language ranking correlations (Spearman ρ = 0.90–0.99) further support consistent relative model evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.