- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes Cross-Case Rubric Learning (CCRL) for rubric-based visual preference evaluation, together with a dataset of 7,044 same-product image pairs labelled by observed CTR ordering, a 54-rubric taxonomy, and 723 expert-annotated pairs. The authors argue that outcome-only supervision is underdetermined, since compensating criterion errors leave the final margin unchanged and per-case rewards do not constrain consistency across related cases. CCRL therefore decouples evaluation into visual difference identification, rubric selection, and bounded criterion scores that are deterministically aggregated into a preference margin, and optimizes this decomposition with a case-by-rollout RLOO estimator in which independently assessed cross-case relations penalize inconsistent scores for equivalent visual elements. Experiments report 58.37% pairwise accuracy against 56.71% for Local-RLOO, with expert-reference rubric micro-F1 and full-program compliance as process metrics.
Critical strengths
- A useful diagnosis. Section 4 separates output validity, label accuracy, full-program compliance, and swap consistency, and shows that better original-order accuracy does not imply order invariance.
- An inspectable formulation. The decision is an explicit sum of signed criterion contributions, each a binary applicability gate times a bounded score, and Section 3 states precisely how contributions must invert under an A/B swap.
- Appropriate data design. Same-product comparisons, matched exposure windows, and item-disjoint splits.
Critical weaknesses
- The consistency claim is never measured. Table 3 omits CCRL although Section 1 lists image-identity swap consistency as a metric, and neither rubric micro-F1 nor full-program compliance shows that criterion scores stay stable across cases. Identity consistency alone is also uninformative, since a fixed label-independent ordering attains it perfectly, so consistent-and-correct rates with label balance and fixed-position baselines are needed. Tied margins, explicit ties, and invalid outputs are never defined in the accuracy or consistency computation.
- Graph construction and reward settings are underspecified. The assessor and its protocol are unnamed, and values or selection rules for , , and , together with block size and degree statistics, are absent. Eq. 4 covers equality edges only, a zero gate nulls the penalty so any score escapes it, and Eq. 7 is unnormalized by node degree. No random-edge control is reported, and Local-RLOO differs from CCRL in more than the relational term.
- CTR labels are not shown to be reliable. At the 100-exposure floor and a typical 2% CTR, sampling noise on the observed gap is near 2pp against a 0.1pp retention threshold, so retained pairs need not have a reliable ordering. No gap-stratified accuracy or noise ceiling is reported.
- Annotations are not mapped to splits. The test split, expert subset, and evaluation cohort are each 723 pairs, and SFT trains on "versioned reference responses", leaving it unclear which records served training, selection, and evaluation, or whether test-pair CoTs were excluded.
Questions for authors
- Can CCRL be added to Table 3 with swapped-order accuracy, identity consistency, and consistent-and-correct rates? How are tied margins, explicit ties, and invalid outputs scored?
- Which assessor builds the graph, and what are the values of , , and ? Is there a random-edge control matched in edge count and weight?
- Where do the Gate Transition applicability judgements come from, and how is a criterion prevented from escaping the constraint once its gate is zero?
- How does accuracy vary with the observed CTR gap, and what noise ceiling does that imply for the benchmark?
- Which records served SFT, model selection, and evaluation, and were test-pair CoTs excluded?