- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces EcomWriteBench, a multilingual benchmark for e-commerce writing. It develops a framework that identifies writing task-relevant abilities and learns hierarchical aggregation weights from human pairwise preferences to provide fine-grained evaluation of LLM-generated content. Although the benchmark addresses a relevant practical problem, the methodological novelty and experimental validation are insufficient to support the paper's main claims.
Critical strengths
- This paper addresses an important problem in evaluating LLM-generated content for real-world e-commerce applications.
- The benchmark covers diverse e-commerce scenarios and multiple languages, providing potentially useful evaluation resources.
- The proposed ability-wise evaluation framework offers more interpretable assessments than conventional holistic scoring approaches.
- Experiments include various LLM configurations and provide detailed comparisons across different evaluation dimensions.
Critical weaknesses
- The proposed method largely relies on existing techniques, including LLM-based ability extraction, fine-grained evaluation, and preference-based weight learning, with limited methodological contributions.
- The 90.42% agreement reported in Table 1 is based on the same preference pairs used for training, making it difficult to assess whether the improvement over the baselines generalises to unseen queries.
- Table 3 shows only a marginal improvement from global to category-level weights (84.64% to 84.84%), suggesting that the proposed hierarchical weighting offers little benefit for unseen tasks.
- The benchmark queries have been rewritten, translated, and filtered, but it is unclear whether the resulting dataset still reflects real-world e-commerce scenarios.
- The evaluation relies heavily on LLM judges, but the paper lacks sufficient evidence that their ability-level scores are consistent with human assessments.
- The performance differences among the top models in Table 4 are very small, and statistical significance tests are needed to support the reported rankings.
Questions for authors
See above
6 comments
W3. Table 3 shows only a marginal improvement from global to category-level weights (84.64% to 84.84%), suggesting that the proposed hierarchical weighting offers little benefit for unseen tasks.
A. We acknowledge that the 0.19-point improvement from global to category-level weights is not statistically significant (95% CI [−0.20, 0.59], bootstrap over tasks). However, Table 3 is intended to compare two variants of our proposed framework for unseen-task transfer, rather than to demonstrate the superiority of category-level weights over global weights.
To further validate unseen-task generalization, we additionally introduce uniform ability aggregation as a baseline in Table 3. Both global and category-level weights transfer to unseen tasks without task-specific preference annotations and outperform uniform aggregation (82.59%) by 2.05 points (95% CI [1.16, 2.95]) and 2.24 points, respectively. These results support the effectiveness of preference-calibrated weights for unseen-task evaluation.
| Weights | Agreement ↑ | Gain vs. Uniform ↑ |
|---|---|---|
| Uniform | 0.8259 | — |
| Global | 0.8464 | +2.05 pt |
| Category | 0.8484 | +2.24 pt |
The marginal difference between global and category-level weights is also consistent with Figure 7, where the selected category-to-global regularization coefficient is . This suggests that category-level preference supervision may already be sufficient to learn reliable weights, reducing the need for shrinkage toward global weights.
W4. The benchmark queries have been rewritten, translated, and filtered, but it is unclear whether the resulting dataset still reflects real-world e-commerce scenarios.
A. EcomWriteBench is constructed from real-world e-commerce workflows, and our data-processing pipeline preserves the original task objectives and evaluation requirements (Appendix A.1.1–A.1.2). Specifically, (1) Rewriting replaces privacy-sensitive and organization-specific information while preserving task variables, factual dependencies, numerical constraints, and output requirements. (2) Translation transfer queries from multi-lingual source tasks to equally distributed multi-lingual queries within each task, with human verification of semantic fidelity and task transferability. (3) Filtering and sampling remove malformed or unusable queries, followed by random sampling within each task under task-specific quotas.
Importantly, all 69 original tasks across 13 e-commerce categories are retained, preserving the diversity of real-world business objectives and evaluation requirements. These procedures maintain the benchmark's grounding in authentic e-commerce workflows while ensuring data quality and multilingual coverage.
W5. The evaluation relies heavily on LLM judges, but the paper lacks sufficient evidence that their ability-level scores are consistent with human assessments.
A. We would like to clarify that our framework does not require LLM-generated ability scores to independently match human-assigned scores for each ability. Instead, a key advantage of our approach is that it directly calibrates the aggregation of ability scores using human pairwise preferences, without requiring costly human annotations at the ability level. Collecting reliable human scores for every applicable ability across 69 heterogeneous tasks and thousands of model responses would require substantial annotation effort and is infeasible within the rebuttal period.
Our primary objective is to align the final rubric scores with human preferences, rather than independently reproduce human judgments on every ability dimension. We directly evaluate this alignment through preference agreement on held-out queries. Furthermore, our additional single-judge experiments (Table 15, Appendix A.2.9) show that task-specific calibration consistently improves agreement over uniform aggregation across all three judges, with gains of 6.23–8.67 percentage points.
These results support the effectiveness and robustness of preference-calibrated aggregation without requiring human ability-level supervision. While direct human validation of individual ability scores could provide complementary evidence for their diagnostic interpretability, it is not a prerequisite for learning human-aligned aggregation weights.
W6. The performance differences among the top models in Table 4 are very small, and statistical significance tests are needed to support the reported rankings.
A. We revise Table 4 and now report 95% confidence intervals for all overall scores, from a bootstrap over tasks. GPT-5.6-Sol, Gemini-3.1-Pro, GLM-5.3 and Kimi-K3 form a leading group, and its gap to the next configuration, Claude-Opus-4.8, is significant (0.122, 95% CI [0.081, 0.164]) (Section 4.4).
| Model | Reasoning | Overall score (0–10) | 95% CI |
|---|---|---|---|
| GPT-5.6-Sol | max | 9.581 | [9.499, 9.660] |
| Gemini-3.1-Pro | high | 9.577 | [9.500, 9.653] |
| GLM-5.3 | max | 9.539 | [9.454, 9.624] |
| Kimi-K3 | max | 9.535 | [9.463, 9.608] |
| Claude-Opus-4.8 | max | 9.414 | [9.327, 9.503] |
| Gemma-4-31B | enabled | 9.379 | [9.285, 9.475] |
| Qwen3.8-27B | xhigh | 9.338 | [9.230, 9.446] |
| Claude-Sonnet-5 | max | 9.313 | [9.224, 9.406] |
| MiniMax-M3 | enabled | 9.215 | [9.056, 9.357] |
| Gemma-4-31B | disabled | 9.184 | [9.066, 9.302] |
| Qwen3.5-397B-A17B | enabled | 9.180 | [9.043, 9.314] |
| Qwen3.6-35B-A3B | enabled | 9.179 | [9.054, 9.307] |
| DeepSeek-V4-Pro | max | 8.961 | [8.822, 9.100] |
| Qwen3.5-397B-A17B | disabled | 8.937 | [8.780, 9.090] |
| DeepSeek-V4-Flash | max | 7.785 | [7.438, 8.110] |
Thanks for your insightful and constructive comments. We sincerely hope that our explanations and additional experiments solve your concerns.
We thank the reviewer for acknowledging the practicability of our proposed benchmark and our interpretable evaluation rubrics. We address your concerns below.
W1. The proposed method largely relies on existing techniques, including LLM-based ability extraction, fine-grained evaluation, and preference-based weight learning, with limited methodological contributions.
A. To the best of our knowledge, we are the first to jointly formulate direct preference-to-rubric learning over a shared fine-grained ability space and hierarchical weight adaptation across heterogeneous evaluation tasks. Our key methodological contribution is a unified framework that converts human pairwise preferences directly into interpretable, task-adaptive scoring rubrics, rather than merely learning preference predictions or relying on human-assigned ability-level ratings.
Specifically, our method differs from existing preference-calibrated evaluation approaches in three respects (Appendix A.2.5, Table 12).
First, direct preference-to-rubric learning optimizes non-negative weights over fine-grained abilities using human pairwise preferences, producing explicit response-level scoring functions without requiring human ability-level annotations.
Second, cross-task rubric transfer is enabled by a shared vocabulary of 37 abilities across 69 heterogeneous tasks. Global and category-level weights can be applied to unseen tasks without target-task preference annotations (Table 3), whereas HD-Eval and AutoMetrics construct task-specific evaluation criteria or metrics that do not directly transfer.
Third, hierarchical weight sharing enables effective adaptation under limited preference supervision. Our new low-resource experiments demonstrate that hierarchical learning outperforms independently fitted task weights by 5.59 percentage points when only 5% of training queries are retained (Table 13; see Reviewer XkPY W5).
We further validate the aggregation method itself: under identical three-judge ability-scoring features, our approach achieves 88.96% held-out preference agreement, outperforming the strongest alternative aggregator (86.91%) by 2.05 percentage points (Table 11; see Reviewer XkPY Q3). These results demonstrate the empirical value of our integrated preference-calibrated rubric-learning framework beyond its individual components.
W2. The 90.42% agreement reported in Table 1 is based on the same preference pairs used for training, making it difficult to assess whether the improvement over the baselines generalises to unseen queries.
A. We conduct additional evaluation under the held-out five-fold cross validation. Conventional LLM-as-a-judge scoring and uniform aggregation have no fitted parameters and are therefore already out-of-sample. Four compared preference-learning methods are also reported. The results are shown below (and also Table 1 in our revised manuscript). We show that our EcomWriteBench rubric outperforms all baselines and achieves the highest held-out agreement.