- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
this paper proposes AxisRubric, which constructs preference rubrics with or without instruction-informed observers, iterative taxonomy refinement, preference-based axis calibration, and instance-specific rubric generation. Across 12 image, audio, and video benchmarks, the primary configuration reports average preference accuracy of 0.74 versus 0.66 for the selected automatic rubric baselines. Modality-average gains persist across 21 configurations; ablations favor taxonomy evolution and instance-specific rubrics.
Critical strengths
the idea of separately observing outputs without instructions is a good and straightforward solution. Taxonomy consolidation and instance-specific questions connect these observations to an explicit, inspectable preference decision.
also the ablations and experiments are insightful in general.
Critical weaknesses
- i think the biggest problem is that the size of the evaluation set is too small, each benchmark has only 120 test examples, with several gains of 1–3 percentage points, should expand the evaluation protocol.
- the baselines are generally weak, no latest omni model compared with, seems no other strong API models even as an upper-bound.
Questions for authors
do the main gains persist across multiple 40/120 splits and full taxonomy-generation runs, with paired confidence intervals? what's the margin of performance between the API models and your trained models?