acceptodds
Under review as a conference paper at ICLR 2027

AxisRubric: Rubric Construction via Axis Discovery for Judging Multimodal Generation

Abstract

Rubric-based rewards improve comparative preference prediction for multimodal generation by decomposing human preferences into explicitly checkable evaluation axes. Their accuracy therefore depends on how effectively these axes capture the factors that shape human preference. A largely overlooked challenge in discovering preference-relevant axes for reward modeling in multimodal generation is the cross-modal observability gap. That is, textual instructions and predefined rubrics may miss delivery-quality differences, such as rendering artifacts, audio clarity, and motion coherence, which are observable only in the generated multimodal outputs. Existing methods use instruction-derived, expert-authored, or output-aware rubrics, yet none fully closes the gap. Our preliminary study shows that even when the rubric author sees the generated outputs, it still focuses on what the instruction requests, and up to 32% of human-validated delivery-quality axes are not discovered. Moreover, the discovered axes are not equally predictive of human preference, nor are they applicable to every instance. These observations reveal two key failure modes: axis omission, where preference-relevant axes are missing from the rubric; dilution, where redundant, poorly predictive, or inapplicable axes weaken the influence of informative ones. To close the gap, we propose AxisRubric, a three-phase multi-agent framework for automatic rubric construction via axis discovery. To address omission, Phase 1 runs independent instruction-blind and instruction-informed observer agents that discover evaluation axes, which we consolidate into an evolving benchmark-level taxonomy. To address dilution, Phase 2 deactivates misleading axis categories and calibrates the weights of the remaining axes, and Phase 3 selects the axes applicable to each individual instance and rewrites them as instance-specific rubric items. Across twelve preference benchmarks spanning image, audio, and video generation, AxisRubric outperforms all seven evaluated baselines. It improves preference accuracy over the strongest automatic rubric construction baseline by 0.08 on average (from 0.66 to 0.74) and by up to 0.18 (from 0.70 to 0.88). Notably, these gains hold across 21 pairs of rubric-author and output-judge models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
LLM-as-a-Judge, Agentic Judge, Rubric-based Reward, Rubric Generation, Reward Modeling, Multi-Agent Systems, Multimodal Generation Evaluation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_C3aYOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

this paper proposes AxisRubric, which constructs preference rubrics with or without instruction-informed observers, iterative taxonomy refinement, preference-based axis calibration, and instance-specific rubric generation. Across 12 image, audio, and video benchmarks, the primary configuration reports average preference accuracy of 0.74 versus 0.66 for the selected automatic rubric baselines. Modality-average gains persist across 21 configurations; ablations favor taxonomy evolution and instance-specific rubrics.

Critical strengths

the idea of separately observing outputs without instructions is a good and straightforward solution. Taxonomy consolidation and instance-specific questions connect these observations to an explicit, inspectable preference decision.

also the ablations and experiments are insightful in general.

Critical weaknesses
  1. i think the biggest problem is that the size of the evaluation set is too small, each benchmark has only 120 test examples, with several gains of 1–3 percentage points, should expand the evaluation protocol.
  2. the baselines are generally weak, no latest omni model compared with, seems no other strong API models even as an upper-bound.
Questions for authors

do the main gains persist across multiple 40/120 splits and full taxonomy-generation runs, with paired confidence intervals? what's the margin of performance between the API models and your trained models?

Reviewer_TcHBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AxisRubric constructs rubrics for pairwise preference prediction over generated images, audio, and video. It uses separate instruction-blind and instruction-informed observers to discover delivery-quality and instruction-following axes, consolidates them into an evolving taxonomy, calibrates category/axis weights against human preferences, and generates instruction-specific rubric items for a separate judge. Experiments cover twelve benchmarks, with 40 instances per benchmark for discovery/calibration and 120 held-out instances for evaluation. The primary configuration reports average preference accuracy of 0.74 versus 0.66 for the selected strongest automatic rubric-construction baseline in each modality; additional experiments vary rubric authors and judges and ablate framework components.

Critical strengths
  1. The separation of instruction following from delivery quality addresses a useful evaluation problem: checking only explicitly requested content can miss perceptual defects that affect human preference. The instruction-blind observer is a plausible way to recover such information.

  2. The evaluation spans image, audio, and video, and includes component ablations and multiple author/judge configurations. In particular, the consolidation and per-instance rubric ablations provide useful evidence that how evaluation criteria are organized and instantiated matters, beyond merely listing generic quality criteria.

Critical weaknesses
  1. The evaluation is too small to establish the reliability of several reported improvements. Appendix A.1 samples only 120 evaluation instances per benchmark rather than providing a large-scale evaluation of the available benchmark data. A two-percentage-point improvement corresponds to roughly two or three additional correct predictions on such a set. The paper does not report confidence intervals for method differences or variability across independently sampled discovery/evaluation splits. Repeating judgments with fixed rubrics or changing author/judge models does not measure uncertainty due to which 40 discovery instances and 120 evaluation instances were selected. This limits confidence in the benchmark-level comparisons and their representativeness.

  2. The comparisons do not sufficiently isolate the proposed construction method from target-benchmark adaptation. AxisRubric discovers its taxonomy on 40 instances from each target benchmark and calibrates it using their preference labels, whereas several baselines, such as Holistic, Fixed, and instruction-derived rubrics, do not use an equivalent target-label adaptation step. Sharing the judge and judging protocol controls one source of variation, but does not resolve this information-budget difference. The without-calibration ablation in Table 9 is useful and loses only 0.01-0.02 in average accuracy; however, it still discovers axes from the 40 target-benchmark output pairs. Consequently, the evidence primarily supports within-benchmark adaptation, and it remains unclear how much advantage persists under matched adaptation budgets or transfer to a new benchmark.

Additional presentation concern: The figure layout is overly crowded, especially the method overview in Figure 2. Numerous nested boxes, arrows, and dense annotations are compressed into a small area, leaving the text difficult to read at normal page scale and making the information flow hard to follow. The layout needs a clearer visual hierarchy and more spacing. Splitting the overview into simpler panels, enlarging labels, and moving secondary details to the appendix would substantially improve readability.

Questions for authors
  1. Evaluation scale and sampling uncertainty: Can the authors evaluate on substantially larger representative held-out sets (ideally the full appropriate evaluation splits), release the sampled instance IDs and sampling procedure, and repeat the discovery/calibration and evaluation with independently sampled splits? Please report paired confidence intervals for improvements over the strongest baselines. I would like to know whether the conclusions, particularly the smaller gains, persist when both the 40-instance construction set and the test set change.

  2. Matched target-data access: Can the authors give suitable baselines the same 40 target-benchmark pairs and preference labels, for example through calibration of fixed/TIFA rubric weights or few-shot adaptation of the holistic judge, and compare them with both calibrated and uncalibrated AxisRubric under matched inference budgets? Not all baselines require target-specific calibration, so this comparison is needed to separate better rubric discovery from the benefit of target-data access. Please also clarify transfer performance when the taxonomy and weights are frozen from another benchmark, without target-specific rediscovery or calibration.

  3. Incremental value of discovered axes: Can the authors isolate the contribution of newly discovered axes using comparisons matched for rubric-item count and judge calls, alongside accuracy-versus-number-of-axes and pruning/addition analyses? Predicting human preference above chance with an individual axis does not establish its incremental value conditional on the other axes, since correlated axes may duplicate votes. The one-round and consolidation ablations are informative, but also change axis definitions and organization. Could a small subset of axes, or an equally long set of generic quality criteria, achieve similar accuracy?

Reviewer_uwoxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AxisRubric, a multi-agent framework for automatically constructing evaluation rubrics for multimodal generation. The authors identify two limitations of existing rubric-based evaluation methods: axis omission, where important quality dimensions are missing, and dilution, where irrelevant or poorly weighted axes weaken preference prediction. To address these issues, AxisRubric introduces a three-phase pipeline: (1) discovering evaluation axes through independent instruction-blind and instruction-informed observers, (2) calibrating axis weights using human preference labels, and (3) generating instance-specific rubrics from the discovered taxonomy.

The authors evaluate AxisRubric on twelve human preference benchmarks covering image, audio, and video generation. The method achieves an average preference accuracy improvement of 0.08 over the strongest automatic rubric-construction baselines, with consistent gains across 21 rubric-author and judge configurations. Additional ablation studies and robustness analyses further examine the contributions of taxonomy evolution, axis calibration, and per-instance rubric generation.

Critical strengths
  1. Well-motivated problem formulation and systematic rubric construction. The paper identifies a meaningful limitation of existing rubric-based evaluation methods: textual instructions often fail to capture important quality differences observable only in generated multimodal outputs. The distinction between axis omission and dilution provides a clear motivation for the proposed framework. The combination of axis discovery, preference-based calibration, and instance-specific rubric generation offers a systematic approach to improving rubric quality and interpretability.

  2. Broad empirical evaluation with consistent improvements. The paper evaluates AxisRubric across twelve benchmarks spanning image, audio, and video generation, demonstrating consistent improvements over multiple rubric-construction baselines. The results are further supported by experiments with 21 rubric-author and judge configurations, component ablations, and robustness analyses. This relatively comprehensive evaluation provides meaningful evidence that the proposed framework can improve agreement with human preferences across different multimodal generation tasks.

Critical weaknesses
  1. Insufficient isolation of the core axis-discovery mechanism. Although the paper motivates AxisRubric through the cross-modal observability gap, the ablation studies do not directly isolate the contribution of the proposed instruction-blind and instruction-informed dual-observer design. In particular, the paper lacks a controlled comparison against a single output-aware observer with a matched computational budget, while keeping taxonomy evolution, calibration, and instance-specific rubric generation unchanged. Since the proposed framework combines multiple components, it remains unclear whether the improvements specifically arise from the novel axis-discovery mechanism or from the broader rubric refinement pipeline. This weakens the evidence supporting the paper's central methodological contribution.

  2. Limited evidence of generalization and practical efficiency. AxisRubric constructs and calibrates a separate taxonomy for each benchmark using labeled preference examples, rather than demonstrating that the discovered evaluation axes transfer across different datasets or generation domains. Moreover, the multi-agent discovery process and repeated rubric-based judgments introduce additional computational overhead, but the paper does not provide a systematic comparison of inference latency or total evaluation cost. Consequently, it remains unclear how effectively the proposed framework generalizes to new preference distributions and whether its accuracy improvements justify the additional complexity in practical reward-modeling applications.

Questions for authors
  1. Contribution of the dual-observer design (experimental validation): Could the authors provide an ablation comparing the proposed instruction-blind and instruction-informed observers against a single output-aware observer that jointly receives the instruction and generated outputs, under a matched computational budget? Keeping the remaining taxonomy evolution, calibration, and rubric generation components identical would help determine whether separating the two observation perspectives is essential to the reported improvements.

  2. Generalization of discovered axes (experimental validation): Since the taxonomy and axis weights are constructed separately for each benchmark, how well do the discovered axes generalize to unseen datasets within the same modality? Could the authors evaluate a taxonomy discovered and calibrated on one benchmark directly on another, without additional axis discovery or preference-label calibration? This would help establish whether AxisRubric learns transferable evaluation criteria or primarily adapts to benchmark-specific preference distributions.

  3. Computational cost and practical applicability (clarification and experimental evidence): Could the authors report the computational cost of taxonomy discovery, calibration, and per-instance rubric-based judging, including LLM calls, token consumption, and end-to-end latency? A comparison with the strongest baselines under comparable evaluation budgets would clarify whether the accuracy gains justify the additional computational overhead, particularly for large-scale reward modeling.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.