acceptodds
Withdrawn from ICLR 2027

Cross-Case Rubric Learning for Consistent Visual Preference Evaluation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal large language models (MLLMs) can produce plausible rubric-based assessments of visual creatives, yet these often lack cross-case consistency and fail to translate into accurate preference evaluations. To bridge this gap, we introduce a curated dataset of 7,044 same-product image pairs, with preference labels derived directly from aggregate click-through rates (CTR) in real user exposure logs. We then propose Cross-Case Rubric Learning (CCRL), a framework that connects rubric-based visual reasoning with these behavioral outcomes. CCRL decouples preference evaluation into identifying visual differences, selecting relevant criteria, and assigning dimension-level scores that are deterministically aggregated. We optimize this scoring with a leave-one-out policy-gradient (RLOO) estimator, combining local rewards with conditional cross-case feedback. Specifically, independently assessed visual relationships act as relational constraints, penalizing inconsistent scores for equivalent visual elements across different cases. CCRL achieves the highest reported pairwise accuracy among the evaluated baselines. We assess intermediate evaluation quality using expert-reference rubric micro-F1 and full-program compliance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Large Language Models, Visual Preference Evaluation, Rubric Learning, Cross-Case Consistency, Reinforcement Learning, Relational Regularization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_hmCiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Cross-Case Rubric Learning (CCRL) for rubric-based visual preference evaluation, together with a dataset of 7,044 same-product image pairs labelled by observed CTR ordering, a 54-rubric taxonomy, and 723 expert-annotated pairs. The authors argue that outcome-only supervision is underdetermined, since compensating criterion errors leave the final margin unchanged and per-case rewards do not constrain consistency across related cases. CCRL therefore decouples evaluation into visual difference identification, rubric selection, and bounded criterion scores that are deterministically aggregated into a preference margin, and optimizes this decomposition with a case-by-rollout RLOO estimator in which independently assessed cross-case relations penalize inconsistent scores for equivalent visual elements. Experiments report 58.37% pairwise accuracy against 56.71% for Local-RLOO, with expert-reference rubric micro-F1 and full-program compliance as process metrics.

Critical strengths
  1. A useful diagnosis. Section 4 separates output validity, label accuracy, full-program compliance, and swap consistency, and shows that better original-order accuracy does not imply order invariance.
  2. An inspectable formulation. The decision is an explicit sum of signed criterion contributions, each a binary applicability gate times a bounded score, and Section 3 states precisely how contributions must invert under an A/B swap.
  3. Appropriate data design. Same-product comparisons, matched exposure windows, and item-disjoint splits.
Critical weaknesses
  1. The consistency claim is never measured. Table 3 omits CCRL although Section 1 lists image-identity swap consistency as a metric, and neither rubric micro-F1 nor full-program compliance shows that criterion scores stay stable across cases. Identity consistency alone is also uninformative, since a fixed label-independent ordering attains it perfectly, so consistent-and-correct rates with label balance and fixed-position baselines are needed. Tied margins, explicit ties, and invalid outputs are never defined in the accuracy or consistency computation.
  2. Graph construction and reward settings are underspecified. The assessor and its protocol are unnamed, and values or selection rules for τ\tau, λ\lambda, αi\alpha_i and beb_e, together with block size and degree statistics, are absent. Eq. 4 covers equality edges only, a zero gate nulls the penalty so any score escapes it, and Eq. 7 is unnormalized by node degree. No random-edge control is reported, and Local-RLOO differs from CCRL in more than the relational term.
  3. CTR labels are not shown to be reliable. At the 100-exposure floor and a typical 2% CTR, sampling noise on the observed gap is near 2pp against a 0.1pp retention threshold, so retained pairs need not have a reliable ordering. No gap-stratified accuracy or noise ceiling is reported.
  4. Annotations are not mapped to splits. The test split, expert subset, and evaluation cohort are each 723 pairs, and SFT trains on "versioned reference responses", leaving it unclear which records served training, selection, and evaluation, or whether test-pair CoTs were excluded.
Questions for authors
  1. Can CCRL be added to Table 3 with swapped-order accuracy, identity consistency, and consistent-and-correct rates? How are tied margins, explicit ties, and invalid outputs scored?
  2. Which assessor builds the graph, and what are the values of τ\tau, λ\lambda, αi\alpha_i and beb_e? Is there a random-edge control matched in edge count and weight?
  3. Where do the Gate Transition applicability judgements come from, and how is a criterion prevented from escaping the constraint once its gate is zero?
  4. How does accuracy vary with the observed CTR gap, and what noise ceiling does that imply for the benchmark?
  5. Which records served SFT, model selection, and evaluation, and were test-pair CoTs excluded?
Reviewer_5Np1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Cross-Case Rubric Learning (CCRL) for visual preference prediction. It introduces a dataset of 7044 advertising image pairs labeled by real-world CTR. The method uses an MLLM to compare images and assign rubric scores, then applies cross-case consistency constraints with RLOO training. Experiments show that CCRL improves pairwise accuracy from 56.71% to 58.37% over Local-RLOO.

Critical strengths
  1. Interesting training design. The paper connects rubric scores across different cases through a relational graph and introduces a graph-coupled RLOO objective with a policy-gradient derivation.
  2. Structured evaluation. The paper evaluates preference accuracy, rubric selection, output validity and program compliance.
Critical weaknesses
  1. Limited generalization. The method is evaluated only on advertising image preference prediction but its effectiveness on broader tasks (eg, image aesthetic assessment, editing quality, VLM-as-a-judge, etc) remains unclear.
  2. Insufficient justification for the method design. The paper directly sums rubric contributions to predict preferences without sufficient justification or comparison with alternative aggregation methods. Moreover, it is unclear whether the proposed graph-coupled RLOO is necessary compared with simpler approaches, such as cross-case consistency regularization with SFT or direct CTR-based preference training.
  3. Limited experimental evidence. Graph-RLOO improves accuracy over Local-RLOO by only 1.66 percentage points. The paper lacks direct evidence that Graph-RLOO improves cross-case rubric consistency, including its A/B swap consistency results. Important graph-related ablations are also missing.
Questions for authors
  1. Method necessity: How does Graph-RLOO compare with simpler alternatives, such as Cross-Case Consistency Loss + SFT, direct CTR-based MLLM training without rubric generation, or randomly constructed cross-case graph edges?
  2. Score aggregation: Why are rubric contributions directly summed with equal weights? Have the authors considered learnable rubric weights or other aggregation methods?
  3. Consistency evaluation: Does Graph-RLOO actually improve cross-case rubric consistency and A/B swap consistency? Can the authors provide direct measurements of rubric-score differences across related cases?
Reviewer_F4hrOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies visual preference prediction for e-commerce creatives using 7,044 pairs of images for the same product, with the preferred image determined by aggregate click-through rate. Cross-Case Rubric Learning makes the evaluator identify visual differences, select criteria from a 54-rubric taxonomy, score each criterion, and derive the final decision deterministically from those scores. Its Graph-RLOO training objective combines per-case prediction and grounding rewards with penalties from a graph of cross-case relations.

Critical strengths
  • The preference labels come from real click and exposure logs rather than model-generated judgments. Same-product comparisons, matched exposure windows, and item-disjoint splits reduce several obvious confounds.
  • The structured evaluator is genuinely inspectable: its final prediction is computed from criterion-level outputs, and the paper separately reports output validity, final accuracy, rubric selection, and full-program compliance.
  • Cross-case constraints address a real weakness of local outcome supervision. The graph-coupled RLOO construction provides candidate-level credit while each case is still processed independently at inference time.
  • The diagnosis in Table 3 is useful. It shows that improving original-order accuracy with Local-RLOO does not improve identity consistency after swapping image positions.
Critical weaknesses
  • The final Graph-RLOO model is not included in the swap experiment in Table 3. Because swap inconsistency is a central motivation for CCRL, this result should be reported for the proposed model rather than only for SFT, Local-RLOO, and Swap-RLOO.
  • Graph-RLOO improves accuracy over Local-RLOO by 1.66 percentage points, but the paper reports neither multiple training runs nor confidence intervals or significance tests. It is therefore unclear whether this relatively small gain is stable.
  • The construction of the relational graph is underspecified. Section 5.1 says that relations may come from a frozen vision-language prior or human annotation, but does not give the exact division of labor, relation counts and accuracies, construction cost, or safeguards against relations crossing the data splits.
  • The filtering rule requires 100 exposures per image and a CTR gap of 0.1 percentage points, but it does not test whether each gap is statistically significant. Calling these pairs behaviorally distinct may therefore overstate the reliability of some labels.
  • The absolute results remain modest: accuracy is 58.37%, and only 25.17% of outputs pass every programmatic check. Together with the use of population-level CTR, which can hide differences between user groups, this limits the current system's practical use.
Questions for authors

Please refer to the Weaknesses section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.