acceptodds
Under review as a conference paper at ICLR 2027

AudioRSI: Recursively Self-Improving Audio Reasoners

Abstract

Reinforcement learning has made Audio Large Language Models (Audio LLMs) better reasoners, but the reward that drives it still comes from outside the model: a final-answer check, a hand-crafted process score, or an external LLM judge that stays fixed while the policy improves. We show that an Audio LLM can supply this reward itself, and can do so recursively: each improved model becomes the critic that trains its successor, so the training signal improves along with the model. We introduce AudioRSI (Recursively Self-Improving Audio Reasoners), a reinforcement learning framework in which a single model plays two roles: as an actor it generates reasoning paths, and as a critic it compares them pairwise, turning its own judgments into a process-level reward. The critic is periodically replaced by the current policy, which closes the recursive loop. Because the critic hears the same recording as the actor, it can reject reasoning that cites acoustic evidence the audio does not contain, rather than reward reasoning that merely sounds plausible. The recursion is what sustains progress: under a frozen external judge, out-of-distribution accuracy plateaus during training, while a critic refreshed from the improving policy keeps it rising. Across four out-of-distribution benchmarks, all disjoint from the AVQA training distribution, AudioRSI outperforms strong baselines including Audio Flamingo 3 and Gemini 2.5 Pro on MMAU test-mini, ranks first on MMSU, achieves a clear lead on the challenging Multi-Audio split of MMAU-Pro, and leads the 7B open-source tier on MMAR among methods without external reward models. With no reward specifying how to reason, AudioRSI also develops structured reasoning behaviors (logical reasoning, perception-aware analysis, and reduced hallucination), corroborated by an external LLM-as-Judge evaluation and a keyword-based behavior analysis.

Primary area:
reinforcement learning
Keywords:
recursive self-improvement, self-critique, self-rewarding, audio large language models, reinforcement learning, process reward, audio reasoning, multimodal reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_AQhNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper targets a key limitation in audio QA: standard RL rewards only check final answers and formats, rewarding lucky guesses equally without distinguishing genuine reasoning. To avoid training an external reward model, the authors propose AudioRSI, where the model generates multiple reasoning trajectories as an Actor and evaluates them pairwise using a frozen copy of itself as a Critic.

Critical strengths
  1. Audio-conditioned self-critique requires neither a separately trained reward model nor human step-level labels.
Critical weaknesses
  1. Self-critique reliability is inadequately validated. The human audit covers only four cases. Critic quality is not tracked across checkpoints. Better answering has not been shown to yield better CoT supervision.

  2. The added benefit and cost are unclear. MMAU exceeds the quoted Omni-R1 (AVQA) score by only 0.3 points; MMSU omits that baseline.

Questions for authors
  1. A key premise of this work seems to be that a multimodal model’s ability to verify/supervise its own chain of thought (CoT) is more accurate than its ability to generate it. Did the authors empirically validate this assumption, or provide a detailed justification for why it holds?

  2. Please report Omni-R1's MMSU Perception and Reasoning scores under the same protocol. Does additional CoT supervision yield reliable speech-understanding gains?

  3. Please give more figures about the training of AudioRSI(reward and others) to help me better understand how RSI works.

Reviewer_JfzsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AudioRSI trains an audio reasoning model using outcome rewards and pairwise self-evaluation rewards, periodically refreshing the critic from the improving policy. Experiments across four benchmarks show gains over outcome-only training. Ablations and external preference evaluation support the approach, while the specific benefit of recursive critic improvement remains insufficiently isolated.

Critical strengths
  1. The combined training recipe consistently improves over outcome-only training across four audio benchmarks, with gains of 1.5, 0.44, 1.4, and 3.1 percentage points. This controlled comparison provides credible evidence of practical value beyond simply comparing against the original pretrained model.
  2. The external preference evaluation complements benchmark accuracy and supports improvements in response quality. Using a separate judge and randomized response order makes this evidence more informative than evaluation through the training critic alone, although it does not by itself isolate improvements in reasoning quality.
Critical weaknesses
  1. The process-reward contribution is not sufficiently separated from reward rescaling. Equation 10 centers rewards without standard-deviation normalization, so a correctness-aligned critic could amplify advantages and alter the policy–KL balance. Consequently, gains over outcome-only training do not yet establish that the critic supplies useful supervision beyond outcome reweighting.1
  2. The evidence specifically supporting recursive critic improvement is less comprehensive than the evidence supporting the overall recipe. The refresh analysis is limited to one benchmark, with no demonstrated robustness across training seeds. Moreover, changing reference responses complicates attribution even when critic weights are frozen.
Questions for authors
  1. Clarification: Were the outcome-only baseline and AudioRSI matched for advantage scale and effective KL regularization? Please clarify whether the critic meaningfully distinguishes responses with the same correctness label, since this would help establish information beyond a rescaled outcome reward.
  2. Clarification: In the frozen-critic and refresh ablations, how are reference responses generated, and does their generating policy change during training? This determines whether the comparison isolates critic updates or combines them with changes in the critic’s reference distribution.
  3. Focused evidence request: Can existing logs or a limited matched control show whether the gains persist after accounting for advantage scaling? Such evidence could materially improve my assessment of the claimed mechanism; a broad new benchmark suite is not necessary.
Reviewer_vquVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AudioRSI trains an audio-language model using its own judgments of its reasoning. For each question, the model generates eight reasoning traces and answers. A frozen copy of the model listens to the audio and compares every ordered pair of responses. These judgments add a reward to the usual answer-correctness and format rewards. Every 200 training steps by default, the updated model replaces the judge if its answer accuracy on a training batch has not fallen. Training still requires ground-truth answers.

Critical strengths
  1. Gains on four benchmarks under the same training and evaluation settings. The Ke-Omni-R comparison keeps the base model, data, and evaluation fixed. MMAR shows the largest overall gain. On MMSU, reasoning improves from 78.06% to 80.54%, while perception falls from 47.09% to 45.62%. These results show a useful, though uneven, effect from the added reward.

  2. The judge can check reasoning against the audio. It receives the recording, which lets it check claims about what was heard. The paper also tests fixed judges, fixed reference responses, and different update intervals. This is a useful extension of Self-Rewarding Language Models and the external text judge used by Audio-Thinker.

Critical weaknesses
  1. The paper does not show whether the gains justify the extra training compute. Appendix B.4 reports 56 judge calls per question on top of eight generated answers, and 3–4 days on eight H200 GPUs for 6,400 steps. Appendix A.1 explicitly says that a controlled runtime comparison with outcome-only training is missing. The extra calls show added work, but do not tell us the runtime multiplier. Gains over Ke-Omni-R are +1.50 points on MMAU, +0.44 on MMSU, +1.4 on MMAU-Pro, and a stronger +3.1 on MMAR. It remains unclear whether the baseline would do as well with the same total compute. Figures 3–5 use training steps rather than compute on the horizontal axis. The fixed-judge experiment also pays for judging, so it does not answer this question.
  2. Better answers do not directly show a better judge. The update rule checks answer accuracy, not judgment accuracy. The paper does not test successive judges on the same held-out preference set. Its fixed judge is Ke-Omni-R, so that comparison changes both the starting judge and whether it is updated. The human check covers four examples, only one of which compares two correct answers. Keyword counts and external-judge preferences can also reflect longer or more polished responses. These limits matter because improving the judge is the paper's central claim. Checking the judge with missing or shuffled audio would also help, given Omni-R1's finding that text-only training can improve audio benchmark scores.
Questions for authors
  1. Compute measurement: How much training time and how many GPU-hours do AudioRSI and your Ke-Omni-R baseline use on the same hardware and schedule? Appendix B.4 already reports AudioRSI's total time. Please add the baseline time and a breakdown of answer generation, judging, and model updates, including input/output token counts.

  2. Equal-compute comparison: Does AudioRSI still win if outcome-only training gets the same total GPU-hour budget, through more training steps or more sampled answers on the same data? Please show accuracy against GPU-hours. How do training time and accuracy change when using fewer than 56 pairwise judgments?

  3. Judge improvement: Do later judges make better decisions on the same held-out examples, especially when both answers are correct? How often does the update check accept a new judge? What happens if the initial judge stays fixed throughout training?

  4. Evaluation details: Which MMAU version, prompts, decoding settings, and checkpoint-selection rule did you use? Please explain the base-model score of 65.2% under your setup versus the quoted 71.5%. Also specify the AVQA subset and changes from the released Ke-Omni-R recipe, which uses AVQA and MusicBench. Your reproduced baseline is the clearest basis for an equal-compute comparison; the quoted Omni-R1 scores use a different setup.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.