- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AudioRubrics adds evolving, audio-conditioned rubrics to GRPO, alongside answer and length rewards. Criteria are generated from current rollouts and filtered by their ability to distinguish responses. Experiments on three benchmarks show improvements over outcome-only training.
Critical strengths
- Updating criteria as the policy improves addresses a real limitation of static rewards. Table 6 provides evidence that evolution contributes beyond the initial rubrics.
- The gains cover both reasoning and perception. The weaker-judge experiment is also useful: it shows when the approach stops working.
Critical weaknesses
- Figure 11 gives the judge each rollout's correctness label. The rubric reward could partly reproduce answer supervision rather than assess acoustic grounding. A label-blind comparison or an independent audit of explanations would help establish the claimed mechanism.
- Appendix A says reward weights follow the benchmark sweeps in Section 4.3, without identifying a separate validation set. This leaves uncertainty about whether the reported improvements include test-set tuning.
- Online rubric generation depends on Gemini-3.1-Pro, yet its cost is not reported. The added expense matters when the margin over existing methods is small; a comparison with static rubrics under the same total budget would be useful.
Questions for authors
- which data split was used to select the rubric and length reward weights?
- does removing rollout correctness labels from the judge prompt preserve the gains?
- Can you train with an open 7B-8B audio/text judge only, and report MMAU/MMAR/MMSU vs Gemini-as-judge?
- What is human agreement with the judge on 200 traces (faithful vs fluent-but-wrong)? Do evolved rubrics mention events absent from the clip?