acceptodds
Under review as a conference paper at ICLR 2027

When Reward Doesn't See Everything: Modality-Blind Credit Assignment for Simultaneous Speech-To-Speech Translation

Abstract

Reinforcement learning is increasingly used to fine-tune simultaneous speech-to-speech translation (S2ST) models. These models produce both text and audio at the same time, but the reward functions used to train them are computed almost entirely from text. We show that this mismatch causes a failure the training process cannot detect on its own, in which the training reward rises steadily while the model's own voice quality collapses beneath it. We test this by applying group relative policy optimization (GRPO) with a text-only bilingual evaluation understudy (BLEU) reward to Hibiki-Zero, a 3-billion-parameter S2ST model. During training, the reward score rises steadily, but translation quality on held-out data does not follow it, and audio quality falls sharply. Speaker similarity drops by more than 40%, and naturalness, measured by the UTokyo-SaruLab mean opinion score (UTMOS), falls from 3.410 to 2.610. We trace this to GRPO's credit assignment, which broadcasts a single scalar advantage derived from the text reward across every token the model produces, including the audio tokens the reward never examines. This is not simply a missing safeguard: the collapse persists even when we restore GRPO's Kullback-Leibler (KL) divergence penalty, which limits how far the model may drift from its starting point. Holding the reward fixed, we substitute a learned value function that scores each position separately, using proximal policy optimization with generalized advantage estimation (PPO+GAE). This removes the collapse entirely, and it holds when group size is matched to GRPO's. We further test four cheaper alternatives, including one with no learned value estimator at all. All four collapse and are statistically indistinguishable from GRPO. This shows that what matters is not whether a value estimator is present, but whether credit is assigned by comparing samples.

Primary area:
reinforcement learning
Keywords:
Simultaneous Speech-to-Speech Translation, Reward Hacking, Reinforcement Learning Fine-Tuning, Credit Assignment, Policy Optimization, Multi-Stream Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_u4AAOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper studies reinforcement learning for simultaneous speech-to-speech translation when the policy jointly generates text and audio, but the training reward is computed only from the text stream. Using Hibiki-Zero, the authors show that GRPO can improve its text-derived training reward while substantially degrading generated speech quality, including speaker similarity and UTMOS. They compare vanilla and KL-regularized GRPO with PPO+GAE, several lightweight value-function approximations, and a segment-level critic-free alternative. Their main empirical finding is that PPO+GAE avoids the observed audio degradation under a matched text-only reward, whereas the tested GRPO-family alternatives do not. The paper further performs robustness checks across reward formulations, group sizes, random seeds, language pairs, evaluation sets, and the weight of the text-derived advantage applied to the audio stream.

Critical strengths
  1. The paper identifies an important and practically relevant failure mode in multistream RL.
  2. The authors test KL regularization, matched-reward PPO+GAE, matched group size, several lightweight critic/value approximations, a segment-level critic-free baseline, and a direct intervention on the audio-stream advantage.
Critical weaknesses
  1. The central causal claim is not supported by the paper’s own experiments. The paper attributes the failure to cross-modal advantage broadcasting, but directly removing this broadcasting (cq=0c_q=0) does not prevent the collapse. The actual mechanism therefore remains unresolved.

  2. It is unclear whether PPO+GAE actually solves the RL problem rather than simply avoiding learning. The paper emphasizes that PPO+GAE prevents audio degradation, but Figure 2 shows its training reward remains essentially flat, and the matched-reward PPO+GAE does not clearly improve translation quality over the original model. Preventing collapse by making little useful optimization progress is much weaker than the claimed solution.

  3. The conceptual novelty is limited once the causal claim is weakened. The paper itself discusses closely related failures in TTS and joint audio-video generation, including prior work that already motivates modality-specific credit assignment. If the proposed new mechanism is not actually established, what remains is mainly an empirical case study showing that one GRPO setup degrades audio while PPO does not.

Questions for authors
  1. What controlled experiment isolates credit assignment alone while keeping the remaining optimization procedure fixed?

  2. Do human listeners confirm the substantial audio-quality degradation suggested by UTMOS and speaker similarity?

  3. Which conclusions do the authors believe are supported specifically for Hibiki-Zero, and which are intended to generalize to multistream/critic-free RL more broadly?

Reviewer_3fAGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper is substantially an analysis of the RL technique already introduced in the literature in Hibiki-Zero, which is a GRPO based on BLEU score (n-gram translation quality metric) with some variants (mainly other PO techniques). The paper claims the identification of a failure mode of RL when the reward is represented by a text-only translation quality metric, which is that the output speech becomes substantially worse. Experiments builds on the open-weight Hibiki-Zero and are done on one (or two? see below) language pairs using EuroParl-ST.

Critical strengths
  • The systematic analysis of different PO techniques applied to Simultaneous Speech-to-Speech Translation is new and potentially interesting
Critical weaknesses
  • Outdated metric for translation quality: BLEU instead of COMET and, in general, neural-based metrics that are the de-facto standard in the last years speech translation evaluation [1,2] given the collectively recognized limitations of n-gram based metrics, including their poor agreement with human evaluators [3]. However, the paper focuses only on this quality metric.
  • Technical novel is limited, the analysis of the different PO strategies is potentially interesting but its impact is severely limited by the adoption of a single (and outdated, see above) quality metric.
  • The main claim regarding the RL "structural weakness" appears to be overstated and not validated with different models and/or data scale and/or training languages. Moreover, this "structural weakness" is not observed in the paper introducing the BLEU-based GRPO (Hibiki-Zero), as no significant speech quality degradation is observed.
  • Experimental settings are unspecified: the selection of "held-out set of 150 Europarl-ST utterances" is completely unmotivated, especially since EuroParl-ST does have a test set. This completely prevents reproducibility. Moreover, no statistics about its duration is reported.
  • Paper writing is unnecessary verbose, and despite that, many key information are missing such as the evaluation framework (e.g., how latency is logged, whether it considers computational overhead or not [4]), and the FLEURS dataset just appears in the middle of the Results discussion (Section 4), without being mentioned in the experimental settings (not even Section 3.1), as well the related language pair (Spanish-English) that appears in the text twice but it is never reported in the Figures/Results (nor in their captions).
  • Multiple citations of datasets (no citation for the main dataset of the paper, EuroParl-ST, nor for FLEURS), and models (e.g., Magistral--even if it appears in the reference list) are missing or wrong (e.g., BLEU citation is missing at first occurrence and appears only when mentioning ASR-BLEU metric, which is not correct)

Reference

[1] Speech Translation and Metrics in 2026: Findings of the IWSLT Campaign (Adelani et al., IWSLT 2026)

[2] Findings of the IWSLT 2025 Evaluation Campaign (Abdulmumin et al., IWSLT 2025)

[3] Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust (Freitag et al., WMT 2022)

[4] How “Real” is Your Real-Time Simultaneous Speech-to-Text Translation System? (Papi et al., TACL 2025)

Questions for authors
  • Can the authors explain how can they be sure that the identified "failure" is dictated by the BLEU-based GRPO instead of a wrong training setting (e.g., due to the small training set used)?
  • Why did the authors chose BLEU score over other better quality metrics?
  • Why did the authors extracted 150 examples from Europarl-ST instead of using the test set?
  • What is the total duration of the test set?
  • Since the authors mention "We train on Europarl-ST alone, rather than Hibiki-Zero’s own Europarl-ST/CoVoST2 mixture, so that any collapse can be attributed to the training procedure rather than to heterogeneous data quality" (lines 185-186), what did the author do for ensuring the data quality of the extracted training set (line 182-184)?
  • Where are the FLEURS results?
  • What is the novelty of the proposed PO variants?
  • Since the main claim of the paper is the identification of RL failure regarding speech output quality, why not using an additional reward for naturalness itself as a solution (for instance, [1])?

[1] Lee, D., Cho, Y., Park, S., Kim, H., Yoon, S. (2026) NaturalFlow: Reducing Disruptive Pauses for Natural Speech Flow in Simultaneous Speech-to-Speech Translation. Proc. Interspeech 2026 [Long Track], 1646-1655, doi: 10.21437/Interspeech.2026-540

Reviewer_YZtROpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2(+1)
Read review
Paper summary

This paper studies audio degradation during RL fine-tuning of simultaneous speech-to-speech translation models with text-only rewards. Using Hibiki-Zero, the authors show that GRPO can degrade naturalness and speaker similarity despite improving the training reward. PPO+GAE preserves audio quality in the reported experiments, whereas KL-regularized GRPO and four lightweight alternatives do not. The paper attributes this difference to credit assignment and investigates it through reward-formulation, group-size, and audio-advantage-weight ablations.

Critical strengths

The paper highlights a practically important evaluation blind spot. Measuring naturalness and speaker similarity independently of the text reward reveals substantial degradation that reward-based evaluation would miss.

The training trajectories and targeted ablations provide useful diagnostic evidence beyond an isolated failed run. In particular, they show that restoring the tested KL penalty or applying several lightweight advantage corrections is insufficient to preserve audio quality.

Critical weaknesses

The causal conclusion is stronger than the evidence. In Table 3, removing the direct audio advantage entirely ((c_q=0)) does not protect speech: UTMOS falls to 2.749, compared with 3.040 at (c_q=1). This challenges broadcasting as the identified mechanism, rather than merely complicating its dose-response relationship. The parameter-norm analysis does not rule out shared-parameter interference, since functional changes depend on update direction, not just magnitude. The GRPO–PPO comparison establishes an algorithm-dependent difference, but does not establish that group-relative credit assignment is the underlying cause.

PPO’s stability is not demonstrated under comparably effective reward optimization. Its reward curve in Figure 2 shows little sustained improvement, and matched-reward PPO has Text-BLEU 26.79 versus 27.39 for the baseline (Table 2). Its preserved audio quality could therefore reflect more conservative updates rather than better credit assignment. Matching the reward formula and nominal KL coefficient does not resolve this alternative without comparing achieved reward gains and actual policy movement.

Questions for authors

How do you reconcile the (c_q=0) result with the proposed broadcasting mechanism? What evidence distinguishes this explanation from interference caused by text updates through shared parameters, beyond comparing parameter norms?

Does PPO preserve audio quality when achieving training-reward gains comparable to GRPO? Please provide same-reward learning curves and realized per-stream KL, and clarify whether reward formulation and group size were matched simultaneously in a single comparison.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.