- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper studies reinforcement learning for simultaneous speech-to-speech translation when the policy jointly generates text and audio, but the training reward is computed only from the text stream. Using Hibiki-Zero, the authors show that GRPO can improve its text-derived training reward while substantially degrading generated speech quality, including speaker similarity and UTMOS. They compare vanilla and KL-regularized GRPO with PPO+GAE, several lightweight value-function approximations, and a segment-level critic-free alternative. Their main empirical finding is that PPO+GAE avoids the observed audio degradation under a matched text-only reward, whereas the tested GRPO-family alternatives do not. The paper further performs robustness checks across reward formulations, group sizes, random seeds, language pairs, evaluation sets, and the weight of the text-derived advantage applied to the audio stream.
Critical strengths
- The paper identifies an important and practically relevant failure mode in multistream RL.
- The authors test KL regularization, matched-reward PPO+GAE, matched group size, several lightweight critic/value approximations, a segment-level critic-free baseline, and a direct intervention on the audio-stream advantage.
Critical weaknesses
-
The central causal claim is not supported by the paper’s own experiments. The paper attributes the failure to cross-modal advantage broadcasting, but directly removing this broadcasting () does not prevent the collapse. The actual mechanism therefore remains unresolved.
-
It is unclear whether PPO+GAE actually solves the RL problem rather than simply avoiding learning. The paper emphasizes that PPO+GAE prevents audio degradation, but Figure 2 shows its training reward remains essentially flat, and the matched-reward PPO+GAE does not clearly improve translation quality over the original model. Preventing collapse by making little useful optimization progress is much weaker than the claimed solution.
-
The conceptual novelty is limited once the causal claim is weakened. The paper itself discusses closely related failures in TTS and joint audio-video generation, including prior work that already motivates modality-specific credit assignment. If the proposed new mechanism is not actually established, what remains is mainly an empirical case study showing that one GRPO setup degrades audio while PPO does not.
Questions for authors
-
What controlled experiment isolates credit assignment alone while keeping the remaining optimization procedure fixed?
-
Do human listeners confirm the substantial audio-quality degradation suggested by UTMOS and speaker similarity?
-
Which conclusions do the authors believe are supported specifically for Hibiki-Zero, and which are intended to generalize to multistream/critic-free RL more broadly?