acceptodds
Withdrawn from ICLR 2027

Mega-ASR: Towards In-the-wild^2 Speech Recognition via Scaling up Real-world Acoustic Simulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Automatic speech recognition degrades nonlinearly in the wild: as acoustic corruption compounds, local confusions give way to omissions, repetitions, empty outputs, and semantically plausible but unsupported transcriptions. We present MEGA-ASR, a learnability-aligned framework that treats this transition as a linked problem of coverage, grounding, and optimization. First, VOICES-IN-THE-WILD-2M composes seven atomic acoustic effects into 54 controllable scenarios, providing hard but learnable supervision for compound environments. Second, Acoustic-to-Semantic Progressive Supervised Fine-Tuning (A2S-SFT) establishes acoustic grounding before adapting semantic recovery: it first trains the acoustic interface, then the language-side decoder, and finally aligns both end to end. Third, Dual-Granularity WER-Gated Policy Optimization (DG-WGPO) starts from this grounded policy and changes its preference with the error regime, favoring token-level refinement when speech remains recoverable and structural reconstruction when it does not. The three stages therefore form a capability chain rather than independent training add-ons. Compared to the baseline, Qwen3-ASR-1.7B, MEGA-ASR achieves 45.69% versus 54.01% WER on VOiCES R4-B-F and 21.10% versus 29.34% WER on NOIZEUS Station-0dB. On the mixed real/simulated compositional settings, it reduces WER by 65.8%/52.5% relative to Gemini-3-Flash and by 69.4%/69.1% relative to Whisper Large-v3.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Automatic Speech Recognition, Speech Processing, Robust Learning, Data Augmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jx9hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes MEGA-ASR for robust speech recognition through three contributions: a large-scale compound-degradation speech dataset, a three-stage acoustic-to-semantic fine-tuning strategy (A2S-SFT), and a WER-gated policy optimization method (DG-WGPO). Experiments on standard ASR and robustness benchmarks show improved performance over the Qwen3-ASR-1.7B backbone, reducing robust macro-average WER from 7.93% to 6.70%. Ablations and cross-backbone evaluations provide additional validation, though the individual contributions of the proposed mechanisms warrant further investigation.

Critical strengths
  1. Comprehensive evaluation and practical relevance. The paper evaluates MEGA-ASR across standard ASR datasets, established robustness benchmarks, and compound-degradation scenarios, providing evidence of improved robustness under challenging acoustic conditions.
  2. Systematic component analysis. The paper includes ablations of staged fine-tuning and reward components, as well as cross-backbone evaluations. These experiments help assess the contributions of different components and suggest that the approach may generalize beyond a single model configuration.
Critical weaknesses
  1. Limited methodological novelty. The proposed A2S-SFT and DG-WGPO primarily combine staged fine-tuning with heuristic reward components. The current experiments do not fully establish that these specific design choices are necessary or consistently superior to simpler alternatives under strictly matched training budgets.
  2. Insufficient evidence for the claimed mechanisms. Although the paper reports improvements and component ablations, the gains from WER-gated reward fusion over fixed fusion are relatively small. Stronger multi-seed comparisons, paired significance tests on performance differences, and analysis of candidate-ranking quality would be needed to substantiate the central methodological claims.
Questions for authors
  1. Clarification: Were the A2S-SFT and DG-WGPO comparisons matched for total training steps, compute budget, and data exposure? If so, could you clarify these controls? This would help determine whether the reported gains arise from the proposed mechanisms rather than additional optimization.
  2. Clarification: Were the improvements of WER-gated reward fusion over fixed fusion statistically significant across multiple random seeds? Given the relatively small differences in the ablation results, this would materially affect my assessment of DG-WGPO's contribution.
  3. Clarification: How well does the proposed approach generalize to real-world acoustic conditions that are independent of the synthetic degradation pipeline? In particular, what evidence supports the representativeness of the real-world evaluation subset and the absence of speaker or source leakage?
Reviewer_Ye7gOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a large scale simulated corpus for training and benchmarking robust speech recognition models, and a supervised finetuning and reinforcement learning optimization framework to create highly robust ASR model, called Mega-ASR.

The dataset covers 54 realistic compound scenarios. The SFT stage first finetunes the encoder and aligner of Qwen3-ASR followed by the LLM side adaptation and finally full adaptation. Finally a WER-gated policy optimization is run, which dynamically switches the rewards based on the dataset's error regime, thus favouring local refinement when speech is less corrupted to sentence level when speech is highly corrupted.

The experiments show very good performance on robust ASR benchmarks.

Critical strengths
  1. Co-designing the data simulation and the finetuning stages and learnability controls for various stages and rewards is a good approach.
  2. The dataset creation is principled and such a large scale dataset is highly valuable.
  3. The multi-stage finetuning leads to significant performance improvements.
Critical weaknesses
  1. Even though the dataset is named "in-the-wild", it is a simulated dataset, hence the sim-to-real gap will definitely exist. There could be certain rare scenarios which might not be covered by the 54 scenarios.
  2. This work involves significant engineering and computational overhead, and training maybe sensitive to careful hyperparameter tuning at various stages.
  3. The high WER rule based filtering or reward switching, while shown to be effective, could be susceptible to legitimate removal of rare and valuable corner cases such as whispering or accents, etc. rather than actual audio corruption.
Questions for authors

See weaknesses above.

Reviewer_fKWGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work tries to improve the robustness of the Encoder-Adaptor-LLM-based ASR. They first establish VOICES-IN-THE-WILD corpus, which adopts 7 acoustic effects under 54 controllable scenarios. Then, they conduct multi-stage training for the Encoder-Adaptor-LLM ASR model. The multi-stage training first train the Encoder-Adaptor, then the LLM decoder, finally jointly finetune (all stages are with LoRA applied). Their base model is Qwen3-ASR. Finally RL training is applied, with algorithm from GRPO and WER-related reward applied. They focuses on English and Mandarin ASR only, and evaluations are mainly separated into two stages: original multi-copora and noise-controlled scenarios.

Critical strengths

None

Critical weaknesses

1. FIrst, the paper is hard to follow.

Here are some examples in the Abstract that I found it hard to interpret:

L16: We present MEGA-ASR, a learnability-aligned framework that treats this transition as a linked problem of coverage, grounding, and optimization. -> what is learnability-aligned?

L20: Acoustic-to-Semantic Progressive Supervised Fine-Tuning (A2S-SFT) establishes acoustic grounding before adapting semantic recovery -> What is acoustic grounding and semantic recovery?

There are plenty of these fancy but unclear wordings that make the paper hard to follow.

2. The work is technical rather than introducing novel idea or interesting findings.

Collecting enough data for robustness -> multi-stage training -> RL finetuning, based on a strong enough backbone model is quite typical. I understand that the authors put a lot of efforts into each stages, but they are more like technical or engineering improvements rather than interesting findings.

3. The authors introduce a lot of details in their training, but they are not properly elaborated or studied.

For example, why do we need so many stages during SFT? and why progressively adding more and more noisy data? Any ablations on this? The designed RL reward is complicated. Can you elaborate more on the behind ideas and verify them with studies?

Questions for authors

Please address the above questions arised in the weaknesses part.

Apart from the above, it would be better to see the if the authors could share or summarize their opinions on:

What do you think is the most critical part when curating the noisy data? Anything interesting?

Do you have any interesting findings during the multi-staged training (both SFT and RL)? What are the behind intuitions? Any evidence?

Does the model still have the similar generalizability as Qwen3-ASR? (multi-lingual capability, OOD scenario) What are the trade-offs?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.