- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RubricFly, a framework for constructing and updating multilingual translation evaluations. Questions and their rubrics are built around shared "testpoints" that specify the capability being tested. The framework has two separate feedback loops: one generates new questions from observed failures and coverage gaps, and the other turns human corrections into rules for rechecking Judge decisions. The authors construct a 500-question benchmark covering translation and cross-lingual rewriting across five Chinese-centered language pairs and evaluate 13 models. They compare three generation strategies on 47 replacement questions and evaluate Judge rule learning against blinded human labels. Full-feedback generation produces the greatest observed separation between models. Rule-guided rechecking modestly improves essential-item agreement, but does not reduce response-level errors.
Critical strengths
-
The testpoint design gives rubric-level diagnosis a concrete use in benchmark construction. A failed requirement can be traced to a capability and used to formulate another test, rather than simply contributing to an aggregate score. This is a useful organizing idea for benchmarks that need to evolve as models improve.
-
The authors put substantial effort into auditing the benchmark and evaluating the Judge. Blinded human labels, question-level cross-fitting, and rechecking controls are meaningful strengths. I also appreciate that the paper reports cases where the method does not help, including unchanged response-level errors and ranking improvements that do not hold across evaluation settings.
Critical weaknesses
-
The renewal comparison is too limited to establish why full feedback works better. Each strategy contributes only 15–16 selected questions, and the allocation is not randomized. Differences in language, task, or candidate selection could account for part of the observed separation. Full feedback also receives score-spread information that the difficulty-only baseline does not receive. Thus, the experiment does not separate the value of capability diagnosis from the value of explicitly targeting model discrimination. The authors acknowledge that this comparison is descriptive, but it is nonetheless the main evidence offered for the renewal mechanism, so its limitations carry substantial weight.
-
The Judge improvement is small and does not carry through to response-level correctness. Learned cards reduce essential-item errors from 307 to 291, but their advantage over the rechecking controls is not statistically established in Table 26. They also exchange 14 false failures for 14 false passes, leaving the total number of response errors unchanged. This matters because a single essential-item decision can determine the score and influence which failures guide renewal. The post-hoc choice of gated rechecking, noted in Appendix E.4, adds uncertainty to the result. I therefore see evidence for a modest improvement in item agreement, but not yet for more reliable evaluation or renewal decisions overall.
Questions for authors
-
How are the three renewal strategies distributed across language pairs and task families? Can the existing candidate records support a comparison before final selection or an analysis controlling for these differences?
-
Appendix E.1 and Appendix E.4: Which methodological choices were fixed before W0 outcomes were inspected? Are any reported results from an evaluation block that was untouched by both rule induction and policy selection?
-
Given that response-level errors remain unchanged, is there evidence that the corrected judgments produce more accurate capability diagnoses or better renewal targets? If this was not measured, would the authors agree that the demonstrated benefit is limited to essential-item agreement?
Thank you for the careful assessment. Existing records show greater model separation for full feedback in a balanced candidate comparison and improved MUST-item judgments on both development questions and historical W0.
1. Does the separation advantage persist before final selection?
The archived candidate evaluation contains 20 questions per strategy, with two translation and two rewriting questions per language pair. All ten models have complete results: 600 evaluations under each judge.
Using StrictlyHybrid scores on a 0–100 scale, mean within-question SD across models is:
Full feedback has the greatest observed separation before final selection, with matched language/task counts, under both judges.
The final replacement distribution is below. Cells give translation/rewriting counts; Chinese is the other language.
Score-spread feedback is an explicit part of the full policy. The balanced comparison shows that the policy’s observed separation advantage is already present before final release selection.
2. What improves on development questions and W0 separately?
For the frozen Qwen3.6-27B judge, three-draw voting shows:
Net corrections relative to initial judging are:
Cards achieve the largest observed net correction in each subset. The item-level improvement is present in development cross-validation as well as W0.
Among responses with complete MUST labels, errors change 121/474→125/474 on development and 29/120→25/120 on W0. The pooled total of 150 combines opposite response-level changes; improved MUST-item agreement is the consistent positive finding across both subsets.
3. What was fixed, and how should these results be interpreted?
Model parameters, rubric text, and scoring are fixed across arms. Development evaluation uses four-fold cross-fitting: 60 questions for rule induction and 20 for evaluation per fold. W0 uses rules induced from the other 80 questions.
All 20 W0 questions had appeared in earlier development and evaluation, so W0 is reported as a historically reused evaluation set. Gating was selected after inspecting always-on rechecking results.
The calibration comparisons are exploratory: development question-level paired p-values against resampling, scoped placebo, and blank placebo are 0.116, 0.428, and 0.093, respectively. The balanced renewal comparison is nonrandomized and evaluates the full feedback policy, including both capability diagnosis and score-spread information. We will use coupled-round experiments to examine whether calibration improves downstream diagnoses and renewal targets.