- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AutoRubric-T2I learns an explicit reward function for text-to-image alignment from image preference pairs. A frozen VLM scores images against generated rubrics, and L1-regularized logistic regression selects and weights rubric features to predict preferences. Iterative refinement expands the pool using misranked pairs selected by a curriculum. The resulting weighted pointwise reward is evaluated on preference benchmarks and used to fine-tune SD3.5-Medium with Flow-GRPO. Experiments with three judges, component ablations, and downstream evaluations report gains over the implemented reward-model and rubric baselines.
Critical strengths
- A coherent approach to a meaningful problem. Preference-supervised selection and weighting directly address the usefulness of individual rubrics, while retaining explicit criteria and a frozen judge. Updating the pool from the current evaluator's errors provides a concrete mechanism for improving its coverage.
- Useful empirical support across evaluation and optimization. The paper evaluates three judges and demonstrates downstream improvements on TIIF and UniGenBench++. Component ablations and cost measurements help assess the practical value of the design beyond preference prediction alone.
Critical weaknesses
- The comparison does not clearly establish the benefit of the proposed scoring design. Assessing sparse weighting and rubric refinement requires an effective rubric baseline. Yet Figure 3 and Table 2 adapt Auto-Rubric's pairwise procedure to pointwise evaluation without adequately explaining the adaptation. This makes it difficult to attribute the performance gap to the proposed learning mechanism. The choice of preference accuracy further limits the conclusion: it evaluates ranking, which either scoring mode can support, and does not independently establish an advantage of pointwise scoring. The results therefore leave the benefit of the scoring design insufficiently established.
- Unclear model selection weakens the evidence for transfer and downstream improvement. The limited-supervision and transfer claims depend on which labels are used to select the rubric set. Appendix G (p.19, L1006–1008) specifies 256 validation pairs from the other source corpus, while Appendix H (p.20, L1039–1043) specifies the same source. This leaves the supervision budget and transfer setting unclear. Downstream selection is also incompletely reported: Appendix B gives a training range and evaluation interval, but not the checkpoint-selection rule for Figure 4 and Tables 3–4. This matters for gains as small as 0.4–0.7 percentage points over AutoRule. Independent support from the human study is limited: its 600 judgments cover only 20 prompts, with no selection rationale.
Questions for authors
- Which validation source was used for each result? Please reconcile Appendices G and H, specify all labels used for model selection, and identify the checkpoint-selection rule for Figure 4 and Tables 3–4.
- How was Auto-Rubric adapted to pointwise evaluation, and what evidence shows that the adaptation preserves its effectiveness? Please also reconcile the conflicting descriptions of RubricRL's offline caching and generation during RL in §2.3 and Appendix C.
- Some of the existing methods such as ARR [1] also develop visual preference rubrics through generation, verification, and refinement for evaluation and downstream optimization. Discussing their relationship to the learned global weights and curriculum mining here would help position the contribution.
[1] Juanxi Tian et al. Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria. arXiv preprint arXiv:2605.08354v1. 2026.
Thank you for the detailed review. In short: Table 2 uses Auto-Rubric's native pointwise mode, and pairwise re-evaluation confirms our rubrics still lead by 3.3–8.1 points across all benchmarks; all results use cross-dataset validation and a fixed step-500 checkpoint; and we will fix Appendices C, G, and H, include the new RL reward curves in the main text, add prompt-level bootstrap CIs for the human study, and discuss ARR [1].
W1. Scoring design and Auto-Rubric baseline
How Auto-Rubric was run. Our comparison tests how rubrics are learned (sparse positive weights + hard-pair mining), not whether pointwise beats pairwise scoring. Although the Auto-Rubric paper describes a pairwise protocol, its OpenJudge code defaults to pointwise grading (
grader_mode = GraderMode.POINTWISEinopenjudge/generator/llm_grader_generator.py, commit4dcb30d). Table 2 thus uses Auto-Rubric's native implementation, passing image pairs and labels to its generator as in the paper (clarified in Appendix D).Pairwise verification. Rerunning both methods under pairwise scoring with the same prompt (averaging both orders) confirms pointwise mode did not disadvantage Auto-Rubric:
*Tie-adjusted as in Table 2 (26.5 if ties count as errors).
On MMRB2, Auto-Rubric shifts judge accuracy by ≤1.3 points (lowering it pointwise), while ours adds 4.4–5.2 points in both modes. Pairwise Auto-Rubric also drops below the judge alone on HPDv3 and Pick-a-Pic, whereas we lead Auto-Rubric across all three benchmarks by 4.4–8.4 points pointwise and 3.3–8.1 points pairwise.
Why pointwise scoring. Pairwise judging is easier (1.9–4.2 points higher with our rubrics) as it sees both images together. We use pointwise scoring because Flow-GRPO normalizes rewards across K=24 rollouts (Appendix B), needing 24 judge calls per group vs. 276 pairs (552 order-balanced). Its utility is shown in Figure 4, beating scalar rewards in all eight settings.
W2. Model selection, checkpoints, and human study
Validation labels (Appendices G & H). All results use cross-dataset validation: rubrics trained on 256 HPDv3 pairs pick their round on 256 Pick-a-Pic pairs, and vice versa. These 256 pairs are the sole selection labels (never entering generation or weight fitting), totaling 512 pairs per run with zero MMRB2 overlap. We will fix the same-dataset split in Appendix H and ambiguous text in Appendix G, and state the 256 + 256 budget in the main text.
Checkpoint rule. Figure 4 and Tables 3–4 evaluate the fixed step-500 checkpoint without benchmark selection. By step 500, runs reach 85–95% of their 800-step max reward, while longer training induces reward hacking (e.g., HPSv3 destabilizes around step 700). We will include these new reward curves in the main text and state the step-500 rule in Appendix B. Across three seeds (see Reviewer 2), ours leads TIIF mean in all four settings (69.0 overall vs. 67.4 AutoRule and 66.4 scalar), though single-setting gaps to AutoRule fall within seed spread.
Human study. The 20 prompts were uniformly sampled from UniGenBench and TIIF, showing four anonymized step-500 images in random order. Treating this as supporting evidence and bootstrapping over prompts for prompt-level correlation, AutoRubric-T2I is chosen 44.8% of the time (95% CI: [39.6%, 46.7%]). We will add the sampling rule and CI to the revision.
Q1 & Q2. Validation, checkpoints, Auto-Rubric, and RubricRL
Validation labels, checkpoints, and Auto-Rubric's pointwise mode are covered in W1–W2. For RubricRL, §2.3 is accurate and Appendix C will be fixed: RubricRL caches prompt-specific checklists offline and runs only scoring during RL (Appendix C followed an earlier arXiv draft). Unlike RubricRL's per-prompt, equally weighted checklists, we learn one shared rubric set with sparse fitted weights.
Q3. Relation to ARR [1]
We will cite and discuss concurrent work ARR [1] in Section 2 and Appendix C. Both refine visual rubrics from small preference sets, but ARR merges verified rubrics into one structured prompt for pairwise win/loss decisions without per-rubric weights. We score each rubric independently, fit sparse positive global weights (+1.2 points on MMRB2), and mine misranked pairs (+1.3 points over random-pair refinement, Figure 5). On Qwen3-VL-8B MMRB2, ARR reports 62.7% (+5.1 over its pairwise judge), comparable to our pairwise accuracy of 64.3–64.6% (+4.9–5.2 over ours), while our per-rubric formulation also serves as a pointwise RL reward.
[1] Juanxi Tian et al. Auto-Rubric as Reward: From Implicit Preferences to Explicit Multimodal Generative Criteria. arXiv:2605.08354v1, 2026.