acceptodds
Under review as a conference paper at ICLR 2027

Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning

Abstract

Audio reasoning is essential for machine understanding of the acoustic world. Reinforcement learning with verifiable rewards can elicit such reasoning, yet existing reward designs are complementary in their limitations: outcome-based rewards supervise only the final answer and let the model reach it without attending to the audio, whereas process-based rewards score the reasoning itself but rely on coarse, hand-crafted, and fixed criteria that neither adapt to each question nor stay grounded in the acoustic evidence. Moreover, questions differ in what they demand, with some hinging on perception and others on multi-step reasoning, and any static criterion weakens as the policy improves. Supervising the reasoning process with fine-grained, audio-grounded, and adaptive rewards is therefore crucial, yet challenging since such rewards are impractical to design by hand for every sample. To this end, we introduce AudioRubrics, a reinforcement learning framework that supervises audio reasoning with evolving, audio-grounded rubric rewards. AudioRubrics synthesizes per-sample rubrics from the raw waveform and, conditioned on the model's own rollouts, regenerates and reweights criteria per group, supplying a continuous learning signal that keeps targeting the current policy's weaknesses as static criteria saturate. Comprehensive evaluations across three audio reasoning benchmarks reveal that AudioRubrics substantially outperforms a wide range of open-source and training-based baselines. Furthermore, our analysis shows that the gains scale with the capability of the rubric generator and judge, and AudioRubrics converges to a stable reasoning length that avoids both degenerate collapse and unbounded growth. The improvement in audio perception further demonstrates the effectiveness of anchoring supervision in the acoustic evidence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
audio reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_efzFOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AudioRubrics adds evolving, audio-conditioned rubrics to GRPO, alongside answer and length rewards. Criteria are generated from current rollouts and filtered by their ability to distinguish responses. Experiments on three benchmarks show improvements over outcome-only training.

Critical strengths
  1. Updating criteria as the policy improves addresses a real limitation of static rewards. Table 6 provides evidence that evolution contributes beyond the initial rubrics.
  2. The gains cover both reasoning and perception. The weaker-judge experiment is also useful: it shows when the approach stops working.
Critical weaknesses
  1. Figure 11 gives the judge each rollout's correctness label. The rubric reward could partly reproduce answer supervision rather than assess acoustic grounding. A label-blind comparison or an independent audit of explanations would help establish the claimed mechanism.
  2. Appendix A says reward weights follow the benchmark sweeps in Section 4.3, without identifying a separate validation set. This leaves uncertainty about whether the reported improvements include test-set tuning.
  3. Online rubric generation depends on Gemini-3.1-Pro, yet its cost is not reported. The added expense matters when the margin over existing methods is small; a comparison with static rubrics under the same total budget would be useful.
Questions for authors
  1. which data split was used to select the rubric and length reward weights?
  2. does removing rollout correctness labels from the judge prompt preserve the gains?
  3. Can you train with an open 7B-8B audio/text judge only, and report MMAU/MMAR/MMSU vs Gemini-as-judge?
  4. What is human agreement with the judge on 200 traces (faithful vs fluent-but-wrong)? Do evolved rubrics mention events absent from the clip?
Reviewer_iQ81OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces AudioRubrics, a reinforcement learning framework that enhances audio reasoning via evolving, audio-grounded rubric rewards. The approach employs a powerful AudioLLM to generate question specific evaluation criteria directly from raw audio and dynamically refines these rubrics according to the policy's rollouts. The resulting process rewards are combined with outcome rewards and an overthinking penalty to promote accurate, evidence-grounded, and concise reasoning. Experiments on MMAU, MMAR, and MMSU show consistent improvements over vanilla GRPO and competitive audio reasoning baselines, supported by ablation studies and analyses of rubric evolution.

Critical strengths

The paper presents an interesting framework that dynamically evolves audio-grounded rubrics in response to the model's current reasoning performance. By generating, filtering, and reweighting evaluation criteria, the method offers more informative process-level supervision than fixed reward functions. And AudioRubrics achieves consistent gains over vanilla GRPO and competitive baselines on MMAU, MMAR, and MMSU. Ablation studies further demonstrate the contributions of evolving rubrics and the overthinking penalty, while qualitative examples illustrate how the evaluation criteria adapt to increasingly challenging reasoning tasks.

Critical weaknesses

The training data are derived from AVQA, which originally includes questions that require visual or joint audio-visual information. It remains unclear whether all converted questions can be reliably answered using audio alone. The authors should clarify whether audio-insufficient samples were filtered out or validated, since such samples may undermine the reliability of the proposed audio-grounded rubric rewards.

Questions for authors

The proposed method depends on Gemini-3.1-Pro for rubric generation, judging, and reweighting throughout RL training. Could the authors report the total training time and token consumption, and compare the computational cost against vanilla GRPO?

Reviewer_KxwhOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AudioRubrics adds a rubric reward to outcome-based GRPO for audio question answering. Before training, Gemini-3.1-Pro listens to each clip and writes five weighted, question-specific rubrics, one per fixed category. During training, for each group of 8 rollouts, the same model sees the audio, the ground-truth answer, the rollouts and their correctness, proposes up to 3 new positive or negative rubrics, and judges all rubrics on all rollouts. Rubrics with zero within-group variance are removed, the top 5 by standard deviation are kept, and Gemini re-assigns their weights. The final reward is 0.9 accuracy + 0.1 format + 0.5 rubric score + 0.15 (1 - length/256). With Qwen2.5-Omni-7B trained on 40,176 AVQA-derived questions for 400 steps, the method reports 78.00 / 65.80 / 65.86 on MMAU test-mini, MMAR and MMSU, which is +2.8 / +3.6 / +2.7 points over the authors' GRPO baseline. Further analyses cover reward weights, a weaker judge (GPT-audio-1.5), response length, rubric adoption over training, and a 3B policy.

Critical strengths
  1. Clear, consistent gains over a matched GRPO baseline, with honest negative results. Table 6 separates plain RL, static rubrics, evolving rubrics and the length penalty, and every full-method number is above the internal GRPO baseline (+2.8 MMAU, +3.6 MMAR, +2.7 MMSU), with a similar trend for the 3B model (Table 8). The paper also reports that a weaker judge removes the benefit (Figure 3b) and that the rubric reward alone makes responses grow without bound (Figure 7). These results make the recipe and its limits easy to understand.
  2. A practical adaptation of rubric rewards to audio. The judge receives the raw waveform rather than a caption or transcript, so rubrics can name concrete sounds (e.g., "footsteps crunch through dry vegetation" in Figure 6). The length penalty addresses a real failure mode of rubric rewards. For people building audio reasoning models with a strong proprietary audio judge, this is a usable recipe.
Critical weaknesses
  1. The core method is close to DR Tulu's RLER, and the difference is not stated. DR Tulu (Shao et al., 2025, cited only in Related Work) already keeps a per-prompt rubric buffer, asks an LM to propose positive and negative rubrics by contrasting the current rollouts, removes zero-variance rubrics, ranks the rest by standard deviation and keeps the top K. This is Section 3.2 steps (i) and (ii). The evolving-rubric system prompt in Figure 10 follows DR Tulu's prompt section by section ("Discriminative Power", "Novelty & Non-Redundancy", "Avoid Mirror Rubrics", "Conservative Negative Rubrics", "1-5 total rubrics", "more positive / more negative / empty"), with audio dimensions added. Section 2.2 also reuses DR Tulu's text on rubrics as rewards almost word for word. Meanwhile, Table 1 marks "Evolve with Policy" as unique to AudioRubrics without listing DR Tulu, RLCER (Sheng et al., 2026), OnlineRubrics (Rezaei et al., 2025) or AutoRubric-R1V (Jia et al., 2025). The real additions are the audio input to the judge, judge-assigned reweighting, and the length penalty. That is a reasonable engineering adaptation, but the paper presents the whole framework as new.
  2. The "evolving" mechanism is barely isolated, and model selection used the test sets. With 400 steps and batch size 8 (Table 7), training covers at most 3,200 prompt groups, under 8% of the 40,176 questions. Unless batch size is per device, most questions are seen once, so the carried-over rubric set from a "previous iteration" of the same question rarely exists, and Figure 4 cannot show rubrics "upgrading" for the same question. In Table 6, adding evolution changes MMAU / MMAR / MMSU by +1.0 / +1.0 / -0.2, from a single run; on MMAU test-mini (1,000 items) one standard error is about 1.3 points. In addition, gamma and delta were chosen by sweeping on MMAU, MMAR and MMSU themselves (Tables 4-5; Appendix A says the final values "follow the sensitivity analyses"). The margins over the best baselines are small (0.9 over CESAR on MMAU, 0.5 over Audio-Thinker on MMAR), and the method is below CESAR on MMSU reasoning (80.45 vs. 81.07) and MMAR music (51.94 vs. 55.83). The "best on all benchmarks" claim is therefore not reliable under this protocol.
  3. Better audio grounding is claimed but not tested. The judge sees the ground-truth answer and each rollout's correctness (Figure 11), so it can easily write and pass rubrics that justify the correct option rather than check what is in the audio. Omni-R1 (Rouditchenko et al., 2025), a baseline in this paper, found that much of the GRPO gain on MMAU/MMAR comes from better text reasoning, and that text-only fine-tuning also helps. Yet the paper runs no no-audio or mismatched-audio test, no check of rationale faithfulness, and no human check of judge accuracy. The MMSU perception gain is used to claim that "correct answers rest on faithfully perceived evidence", which accuracy alone cannot show. Finally, the judge (Gemini-3.1-Pro) is itself much stronger than the policy on these benchmarks (e.g., MMSU 82.6), so part of the gain may simply be distillation from a stronger teacher; there is no compute-matched baseline using the same teacher (e.g., SFT on Gemini rationales, or Gemini as an outcome-only verifier).
Questions for authors
  1. What exactly is new beyond RLER apart from the audio input, reweighting and length penalty? Please add DR Tulu and the other evolving-rubric methods to Table 1, and acknowledge the source of the Figure 10 prompt and the Section 2.2 text.
  2. Is batch size 8 global or per device? How many unique questions were seen, and how many were seen more than once? What fraction of reward computations used a rubric set carried over from an earlier visit? Does Figure 4 count rubrics proposed in the same call or rubrics that survived from earlier visits?
  3. Were gamma, delta, L and the checkpoint chosen on MMAU/MMAR/MMSU? If so, can you report results with values chosen on a held-out split of AVQA, and give results over several seeds?
  4. Do you have accuracy with silent or mismatched audio for GRPO and AudioRubrics, or any human check of whether the judge's rubric verdicts match the audio?
  5. How does the method compare with SFT on Gemini-generated rationales, or GRPO with the same compute spent on Gemini? Please report the number of Gemini calls, API cost and wall-clock time, and the exact model version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.