acceptodds
Under review as a conference paper at ICLR 2027

RubricFly: Rubric-Guided Adaptive Evaluation and Judge Learning for Multilingual Translation

Abstract

As large language models improve at translation, static benchmarks lose diagnostic value: overall scores separate models less and reveal little about which capabilities remain weak. Sustaining that value requires tests that target specific capabilities, such as meaning preservation, register, and cultural adaptation, with question-specific criteria, and that are renewed as models improve. Yet translation benchmarks are mostly authored and updated by hand, and automatic generation for translation either optimizes difficulty alone or yields a fixed pool of items that is not renewed from observed model behavior. Model failures could guide renewal, but only if the Judge reliably separates genuine failures from valid translations; rejecting a valid rendering misstates a model's weaknesses and misdirects the next round of tests. We present Rubricfly, the first LLM-driven translation evaluation framework that combines automated question and rubric generation with feedback-guided benchmark renewal and human-guided Judge rule learning. Shared testpoints specify the capability under test and the response requirements that guide both construction and evaluation. A production loop turns model failures and coverage gaps into new questions and rubrics, and a Judge loop distills human corrections into reusable decision rules without updating model parameters. We build fly-corpus, a 500-question benchmark covering bidirectional translation and cross-lingual rewriting between Chinese and English, French, Japanese, Spanish, and Korean. Under a common GPT-5.5 Medium judging protocol, the 47 questions introduced by renewal average 43.32 across ten models, against 80.95 for the saturated questions they replace; full-feedback replacements are harder than those generated without model feedback (43.06 vs 51.25) and separate models more than difficulty-only ones. Rule-guided rechecking reduces essential-item errors from 307 to 291 under three-draw voting, more than rechecking without rules, although response-level errors are unchanged. Together, these results show how capability diagnosis can drive test renewal and how human corrections can improve the judgments on which renewal depends.

Primary area:
datasets and benchmarks
Keywords:
multilingual evaluation, machine translation, adaptive benchmark generation, rubric-guided evaluation, LLM-as-a-judge

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_w7gWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RubricFly, a framework for constructing and updating multilingual translation evaluations. Questions and their rubrics are built around shared "testpoints" that specify the capability being tested. The framework has two separate feedback loops: one generates new questions from observed failures and coverage gaps, and the other turns human corrections into rules for rechecking Judge decisions. The authors construct a 500-question benchmark covering translation and cross-lingual rewriting across five Chinese-centered language pairs and evaluate 13 models. They compare three generation strategies on 47 replacement questions and evaluate Judge rule learning against blinded human labels. Full-feedback generation produces the greatest observed separation between models. Rule-guided rechecking modestly improves essential-item agreement, but does not reduce response-level errors.

Critical strengths
  1. The testpoint design gives rubric-level diagnosis a concrete use in benchmark construction. A failed requirement can be traced to a capability and used to formulate another test, rather than simply contributing to an aggregate score. This is a useful organizing idea for benchmarks that need to evolve as models improve.

  2. The authors put substantial effort into auditing the benchmark and evaluating the Judge. Blinded human labels, question-level cross-fitting, and rechecking controls are meaningful strengths. I also appreciate that the paper reports cases where the method does not help, including unchanged response-level errors and ranking improvements that do not hold across evaluation settings.

Critical weaknesses
  1. The renewal comparison is too limited to establish why full feedback works better. Each strategy contributes only 15–16 selected questions, and the allocation is not randomized. Differences in language, task, or candidate selection could account for part of the observed separation. Full feedback also receives score-spread information that the difficulty-only baseline does not receive. Thus, the experiment does not separate the value of capability diagnosis from the value of explicitly targeting model discrimination. The authors acknowledge that this comparison is descriptive, but it is nonetheless the main evidence offered for the renewal mechanism, so its limitations carry substantial weight.

  2. The Judge improvement is small and does not carry through to response-level correctness. Learned cards reduce essential-item errors from 307 to 291, but their advantage over the rechecking controls is not statistically established in Table 26. They also exchange 14 false failures for 14 false passes, leaving the total number of response errors unchanged. This matters because a single essential-item decision can determine the score and influence which failures guide renewal. The post-hoc choice of gated rechecking, noted in Appendix E.4, adds uncertainty to the result. I therefore see evidence for a modest improvement in item agreement, but not yet for more reliable evaluation or renewal decisions overall.

Questions for authors
  1. How are the three renewal strategies distributed across language pairs and task families? Can the existing candidate records support a comparison before final selection or an analysis controlling for these differences?

  2. Appendix E.1 and Appendix E.4: Which methodological choices were fixed before W0 outcomes were inspected? Are any reported results from an evaluation block that was untouched by both rule induction and policy selection?

  3. Given that response-level errors remain unchanged, is there evidence that the corrected judgments produce more accurate capability diagnoses or better renewal targets? If this was not measured, would the authors agree that the demonstrated benefit is limited to essential-item agreement?

1 comment

Thank you for the careful assessment. Existing records show greater model separation for full feedback in a balanced candidate comparison and improved MUST-item judgments on both development questions and historical W0.

1. Does the separation advantage persist before final selection?

The archived candidate evaluation contains 20 questions per strategy, with two translation and two rewriting questions per language pair. All ten models have complete results: 600 evaluations under each judge.

Using StrictlyHybrid scores on a 0–100 scale, mean within-question SD across models is:

StrategySonnet judgeGPT judge
Static balancing20.7224.10
Difficulty only27.6424.78
Full feedback32.4133.68

Full feedback has the greatest observed separation before final selection, with matched language/task counts, under both judges.

The final replacement distribution is below. Cells give translation/rewriting counts; Chinese is the other language.

LanguageStaticDifficulty onlyFull feedback
English4/23/34/2
French2/32/23/2
Spanish1/21/21/2
Japanese1/01/20/2
Korean0/00/00/0
Total8/77/98/8

Score-spread feedback is an explicit part of the full policy. The balanced comparison shows that the policy’s observed separation advantage is already present before final release selection.

2. What improves on development questions and W0 separately?

For the frozen Qwen3.6-27B judge, three-draw voting shows:

  • Development 80: MUST errors decrease 244→234 across 4,386 labeled decisions, with 23 repairs and 13 new errors.
  • Historical W0, 20 questions: MUST errors decrease 63→57 across 1,062 labeled decisions, with six repairs and no new errors.

Net corrections relative to initial judging are:

Rechecking armDevelopment 80Historical W0
Learned cards+10+6
Resampling+2+4
Scoped placebo+5+3
Blank placebo+1+4

Cards achieve the largest observed net correction in each subset. The item-level improvement is present in development cross-validation as well as W0.

Among responses with complete MUST labels, errors change 121/474→125/474 on development and 29/120→25/120 on W0. The pooled total of 150 combines opposite response-level changes; improved MUST-item agreement is the consistent positive finding across both subsets.

3. What was fixed, and how should these results be interpreted?

Model parameters, rubric text, and scoring are fixed across arms. Development evaluation uses four-fold cross-fitting: 60 questions for rule induction and 20 for evaluation per fold. W0 uses rules induced from the other 80 questions.

All 20 W0 questions had appeared in earlier development and evaluation, so W0 is reported as a historically reused evaluation set. Gating was selected after inspecting always-on rechecking results.

The calibration comparisons are exploratory: development question-level paired p-values against resampling, scoped placebo, and blank placebo are 0.116, 0.428, and 0.093, respectively. The balanced renewal comparison is nonrandomized and evaluates the full feedback policy, including both capability diagnosis and score-spread information. We will use coupled-round experiments to examine whether calibration improves downstream diagnoses and renewal targets.

Reviewer_RwDEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The authors introduce RubricFly, an LLM-based translation evaluation framework that automatically generates a multilingual translation benchmark by updating questions and rubrics based on model failures and coverage gaps. They also integrate a Judge loop to distill human corrections into reusable decision rules without updating model parameters. They also build a 500-question benchmark covering bidirectional translation and cross-lingual rewriting between Chinese, English, French, Japanese, Spanish, and Korean. The results show that feedback-guided benchmark renewal generates more challenging and discriminative questions, while rule-guided judge calibration slightly reduces rubric-level evaluation errors.

Critical strengths
  • The motivation is clear. The paper presents an interesting framework to adaptively update the questions and rubrics based on previous failures and coverage gaps to better challenge the models.
  • The benchmark covers diverse task families, generation routes, and rubric items. This also involves 39 multilingual experts to validate the quality of the rubrics and judgments.
  • The experiments cover multiple models and language pairs.
Critical weaknesses
  • The results show that GPT-5.6-Sol achieves the highest score on the benchmark. However, the evaluation is judged by GPT-5.5, which may introduce biases and affect the reliability of the results. Also the evaluation focuses on Chinese-centered language pairs. Does the framework generalize to other language pairs (e.g., En-Ja, Fr-Es) that do not involve Chinese?
  • The framework involves multiple powerful LLMs (Gemini 3.1 Pro, Doubao Seed Pro, DeepSeek Pro, GPT-5.5, Qwen3.6-27B) and human experts (39 experts) for rubric validation and judge calibration. Is it necessary to have different LLMs for different steps? It is unclear how scalable the framework is beyond the 500-question benchmark. A cost analysis would be interesting to include.
  • Although the authors acknowledge the lack of joint feedback loops and multi-round evaluation, this remains a major limitation of the paper. The main motivation is to continuously generate more challenging questions and rubrics as models improve. A single update of the questions and rubrics does not sufficiently demonstrate the effectiveness of the framework over multiple rounds.
Questions for authors

Refer to the weaknesses. In addition, will the benchmark be publicly accessible?

2 comments

Thank you for the questions about judge preference, coverage, scalability, and availability.

1. Could GPT-family preference explain GPT-5.6-Sol’s first place?

The leading result is reproduced by a non-GPT judge. We compared existing Claude Sonnet 4.6 Medium and GPT-5.5 Medium judgments on 435 questions × ten models = 4,350 identical responses, with matching questions, rubrics, weights, and MUST/PLUS categories.

Under the same StrictlyHybrid formula, GPT-5.6-Sol, GLM-5.3, Qwen-3.8, and Kimi-K3 occupy the first four positions in the same order under both judges. GPT-5.6-Sol scores 65.92 under Sonnet and 65.85 under GPT. Its lead over GLM is approximately 0.3 versus 11.45 points, respectively: first place is consistent across judges, while the margin depends on the judge.

In the expert-reference panel, GPT-5.5 xhigh receives 65.21 from experts versus 54.92 from the GPT judge, moving from first to second; Claude Opus 4.8 moves from fourth to first. This provides an additional direct comparison of family preference against expert judgments.

2. Does the framework generalize beyond Chinese-centered pairs?

The framework does not require Chinese as a central language. Its testpoint and rubric schemas specify source/target languages, linguistic phenomena, intended operations, and success criteria, and can also accommodate English–Japanese or French–Spanish. We will expand coverage to additional languages and non-Chinese-centered pairs.

3. Are multiple LLMs and 39 experts necessary? What does evaluation cost?

We recommend different model families for construction and judging to increase their independence; model roles remain configurable according to cost and requirements. Automated construction and scoring do not require per-item expert review. The 39 experts supported benchmark validation and supplied calibration reference labels across two study rounds. Routine evaluation runs do not require this staffing.

For a fixed benchmark, we estimate CNY 1.78 per scheduled question per evaluated model for response generation and judging. This is based on a recorded Qwen-MT-Plus run costing CNY 889.31 for 500 scheduled questions: 9.9* for generation, 879.24 for judging, and 0.10 for preflight, with 498/500 responses fully graded. This estimate covers evaluation API charges.

4. Does one update establish sustained evolution?

The balanced 60-question candidate comparison shows the greatest observed model separation for full feedback under both Sonnet and GPT, supporting its ability to produce a useful next evaluation batch.

Current experiments evaluate renewal and calibration separately, with development cross-validation and historically reused W0 reported separately. We will conduct coupled multi-round experiments to test whether calibrated feedback produces sustained improvements in subsequent test quality and model separation. Sustained gains remain an open empirical question for these experiments.

5. Will the benchmark be public?

We plan to post the manuscript on arXiv soon and publicly release the code and benchmark resources.

Thanks to the authors for their efforts to address my concerns. I appreciate the additional analyses and clarifications. However, I still think multi-round evaluation is needed to show whether the framework can continuously improve the benchmark over time, as this is one of the main motivations of the paper. Also, evaluating non-Chinese-centered language pairs would help verify whether the framework works well beyond the current setting. Given these remaining concerns, I will maintain my original score.

Reviewer_NhiEOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes RubricFly for evaluating multilingual translation and cross-lingual rewriting. It generates questions and MUST/PLUS rubrics from a testpoint that brings together the target competence, linguistic phenomenon, and observable requirements, and assigns a weighted score only when all MUST items are passed. It defines two feedback loops: question renewal based on model failures, and judge calibration that incorporates experts' corrections to judgments into rule cards. It evaluates several models on fly-corpus, which consists of rubric items, and separately evaluates the effects of question renewal and judge calibration.

Critical strengths

Building an evaluation resource that includes expert audits of the initial questions and rubrics, along with reference annotations collected with model identities and judge verdicts hidden, is useful. It is also a strength that the paper makes the evaluation criteria explicit through testpoints grounded in translation studies and MUST/PLUS rubrics, and frames the problem as jointly improving evaluation questions and the judge. And it is written to be easy to read and follow, so that even those unfamiliar with the relevant research can easily grasp the problem and design method addressed in the paper.

Critical weaknesses

Building an evaluation resource that has undergone expert review is a strength, and the problem formulation of jointly improving evaluation questions and the judge seems useful, but I find the following aspects insufficiently convincing, which makes me doubt the framework as a whole.

  1. The novelty of the core components seems very limited. Constraint-based translation evaluation has been addressed in IFMTBench (Sun et al., 2026), iterative generation of difficult questions in MT-Breaker (Zouhar et al., 2025), and joint updating of examples and rubrics in GrowLoop (Lin et al., 2026). Although there are differences in the testpoint and MUST/PLUS design, it is unclear what new evaluation capability is gained by combining existing components. I believe the research novelty of the current contribution is insufficient to justify acceptance at ICLR.
  • References:
    • IFMTBench: Sun, Mingrui, et al. "IFMTBench: A Comprehensive Benchmark for Multilingual Translation Instruction Following." arXiv preprint arXiv:2605.28218 (2026).
    • MT-Breaker: Zouhar, Vilém, et al. "Generating Difficult-to-Translate Texts." Proceedings of the First Workshop on Multilingual Multicultural Evaluation. 2026.
    • GrowLoop (Lin et al., 2026): Lin, Yihang, et al. "GrowLoop: Self-Evolving Conversation Evaluation Seeded by Human." arXiv preprint arXiv:2605.28882 (2026).
  1. The reported agreement on MUST items is 93.4%, but this does not seem to sufficiently establish the reliability of the actual scores. A response receives a score of 0 if it fails even one MUST item, so is it correct that judge–expert agreement on whether a response passes all MUST items is 71.2%? I think it is necessary to check not only item-level accuracy but also how well the scores and model rankings agree with those based on expert judgments. As it stands, I am not convinced at all.

  2. The benefit of the two feedback loops working together has not been demonstrated. Since question renewal and judge learning were each evaluated for only one step using different models, the claim that improving the judge also improves the next round of question generation is not yet supported. For testpoints, a comparison against direct rubric generation is also essential to establish what advantages they offer.

  3. It is difficult to interpret the drop in scores before and after replacement as an effect of question renewal. Since easy questions were selected for replacement in the first place, a decrease in scores is expected to some extent. The comparison between strategies is also limited to 16 questions per group, and the new questions lack the same level of independent expert validation as the initial data. Increased difficulty must be distinguished from improved evaluation quality. I understand to some extent why the number of questions is limited, given the need for expert involvement, but the number of questions per group still needs to be increased.

  4. It is unclear whether rule cards are more effective than simple rechecking. The number of response-level errors remains the same at 150.

Questions for authors
  1. For the same responses with expert labels, could you report the per-model Strictly Hybrid scores computed from GPT-5.5 Medium judgments and expert labels, along with the rank correlation? I would like to see how MUST-gate agreement affects the actual scores and model rankings.

  2. When rule learning -> failure reassessment -> question renewal are connected using the same judge, does the evaluation quality of the subsequent questions improve compared with rechecking without rules? Could you also compare against generating rubrics directly from questions without testpoints to show the additional benefit of the core design?

  3. Citation revision recommendation: I recommend aligning the years and sources in the references with the versions cited. If citing the arXiv version of Auto-rubric, please use 2025, the year of its initial posting; for DyVal, please either cite the official ICLR 2024 publication or, if retaining the arXiv version, use 2023, the year of its initial posting.

6 comments

Thank you for the detailed review. We address each concern below.

1. What does RubricFly add beyond existing work?

RubricFly turns failures of specific translation requirements into specifications for new tests and their assessment criteria, and learns judge corrections under fixed criteria. The cited works address different evaluation problems or update mechanisms:

Prior workMain focusRubricFly’s additional capability
IFMTBenchTranslation constraint following, using deterministic gates and continuous rubric judgments.Converts failed requirements and coverage gaps into new question–rubric specifications. MUST/PLUS roles reflect task necessity, allowing context-dependent semantic requirements to be essential.
MT-BreakerIteratively refines source texts using model translations and selects difficult examples through quality estimation.Generates targeted questions together with explicit assessment criteria, making failures diagnosable at the requirement level and reusable for subsequent generation.
GrowLoopLearns and revises rubrics for conversational human-likeness from human preferences, alongside case evolution.Grounds translation assessment in source evidence and task requirements. Judge calibration updates interpretation rules while keeping rubric text, weights, and model parameters fixed.

The testpoint provides the concrete mechanism: it links a target capability, an eliciting source phenomenon, and observable success conditions. The same specification guides question and rubric construction, while failed requirements inform subsequent generation targets. Thus, feedback determines both what to test next and how the resulting responses should be assessed. We have used this design to construct 500 questions with 8,643 rubric items, evaluate multiple models, and renew the benchmark.

2. How closely do actual scores and rankings match experts?

We recomputed StrictlyHybrid scores using expert labels and GPT-5.5 Medium judgments on identical responses to 66 questions from six models. Scores are out of 100; pass counts indicate satisfaction of every MUST criterion.

ModelExpert scoreGPT scoreExpert passesGPT passes
GPT-5.5 xhigh65.2154.9244/6637/66
DeepSeek V4 Pro62.0054.5242/6637/66
Grok 4.5 high60.9848.2842/6633/66
Claude Opus 4.857.6463.6539/6643/66
Gemini 3.1 Pro53.0845.6936/6631/66
HY 3.0 high52.9149.6236/6634/66

Gate decisions agree on 282/396 responses (71.21%), and 10/15 model-pair orderings agree. These directly characterize response-level and ranking agreement alongside the 93.43% MUST-item agreement.

3. What does the testpoint ablation show? Have the loops been evaluated jointly?

We compared rubric generation with and without testpoints on 20 identical questions, using DeepSeek-V4.1-Flash (Max) for both conditions. Claude Sonnet 4.6 assessed anonymized pairs in both presentation orders. With versus without testpoints, question-averaged atomicity was 53.8% versus 46.5%, and observable pass/fail boundaries were 95.5% versus 92.7%. Both differences persisted after excluding case 010.

  • Case 019: The source says “私だけが、時が止まったみたいに、そこに取り残されている” (“Only I was left there, as if time had stopped”). The testpoint explicitly includes this meaning. Without it, r10 (MUST) checks first-person narration but no dedicated item checks this proposition. With it, r5 (MUST) explicitly checks “only I,” “as if time had stopped,” and “left behind.”
  • Case 010: The source lists Unit 1 as “기술=초임계” (supercritical) and Unit 2 as “기술=아임계” (subcritical), and requests factual correction. The testpoint identifies Unit 2 as supercritical. Without it, r3/r5 (MUST) incorrectly require both units to be subcritical; with it, r2/r4 (MUST) require both to be supercritical, consistent with manufacturer documentation.

These model-assisted results support specific benefits in making requirements explicit and individually assessable. Current experiments evaluate the components separately. We will further test calibration→reassessment→renewal across coupled rounds, including comparison with rule-free rechecking.

4. Does renewal offer more than increased difficulty?

In the balanced 60-question candidate set, each strategy contributes 20 questions with identical language/task counts. Before final selection, full feedback produces the greatest mean within-question score dispersion under both Sonnet (32.41) and GPT (33.68).

Among final replacements, difficulty-only generation produces lower mean scores than full feedback (36.15 versus 43.06) but less dispersion (22.59 versus 35.16) and more all-zero questions (5 versus 0). Full feedback’s observed benefit therefore includes greater model separation and fewer uniformly failed tests.

The comparison uses balanced, nonrandomized candidate sets; replacements have not undergone an independent expert audit equivalent to the initial benchmark.

5. Are cards better than ordinary rechecking?

Under three-draw voting with the frozen Qwen3.6-27B judge:

  • Development 80: MUST errors decrease 244→234 across 4,386 labeled decisions. Cards yield +10 net corrections, versus +2/+5/+1 for resampling/scoped placebo/blank placebo.
  • Historical W0, 20 questions: MUST errors decrease 63→57 across 1,062 labeled decisions, with six repairs and no new errors.

Cards achieve the largest observed net correction in both subsets, including development cross-validation. Response-level errors change 121/474→125/474 on development and 29/120→25/120 on W0; the common improvement is at the MUST-item level.

W0 had appeared in earlier development and evaluation and is reported separately as a reused evaluation set. Development paired p-values against the three controls are 0.116/0.428/0.093, so the comparisons remain exploratory.

6. Reference corrections

We will use 2025 for Auto-Rubric’s initial arXiv posting and the official ICLR 2024 publication for DyVal.

Thank you for the detailed response and the additional analyses. Because of a heavy reviewing load, I had to reread the paper, and I apologize for the late reply. I hope we can move the discussion forward quickly from here. In my view, only the citation corrections have been resolved. All five of the concerns I raised remain unresolved. Moreover, the additional results seem to confirm my concerns rather than resolve them. Expert and judge decisions often disagree at the response level, and the model rankings flip in several places. In particular, a model ranked in the middle by the experts receives the highest score from the judge, so it is hard to say that the judge reliably reproduces the ranking. Since you state that the effect of running the two feedback loops together has not yet been evaluated, the paper's central claim remains unsupported. The testpoint comparison is based on LLM judgments over a small number of items, and the differences are small. The finding that score differences between models widen after the item update is meaningful, but it is also measured with an LLM judge whose reliability has not been established, and there is no expert validation of the new items. The rule cards reduce errors only marginally, the difference from a simple re-check is not statistically significant, and on some datasets response-level errors actually increase. What is your view on this? On novelty, you explained how the work differs from prior studies, but no experimental evidence demonstrating this difference has been added. Would additional experiments on this be possible? Therefore, I believe it is right to keep my current score for now.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.