- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how English-to-Chinese speech translation can preserve source endpoints and salient internal pauses. It constructs synthetic speech supervision aligned to the source timeline. It then trains direct generation with a joint text-speech model and a four-stage curriculum. GSPO combines translation-quality, global-duration, and pause rewards. It also shows tradeoffs in translation quality and naturalness.
Critical strengths
-
The paper clearly distinguishes total-duration agreement from internal-pause alignment. It makes the source's absolute timeline an explicit generation constraint. The definition helps evaluate dubbing with fixed timing requirements, beyond translation meaning alone.
-
Source-aligned supervision is a substantive contribution. The authors construct a large training corpus. The final model generates directly and avoids the construction pipeline repeating search at inference.
-
The final model jointly generates translated text and speech. Inference does not repeat the candidate search and audio corrections used to construct the training data. This demonstrations take effort to build but development is simpler. This is a practical advantage.
Critical weaknesses
-
Table 6 shows contributions of the reward components. It does not separately compare reward aggregation, normalization, and modality masks. The evidence therefore supports the combined recipe but does not isolate the contribution of the specialized allocation mechanism.
-
The human evaluation lacks utterance counts, ratings per output, and confidence intervals. These issues limit assessment of the combined tradeoff among timing, content accuracy, and naturalness.
-
The synthetic-supervision pipeline rejects examples that cannot meet local duration constraints. The retained data may therefore favor expressions that are easier to synchronize. The authors disclose this possibility, but the experiments do not isolate its effect.
Questions for authors
-
Are there existing results using a scalar reward, shared normalization, or no modality masks? Please provide available results, or supply reward scales, advantage statistics for each token group.
-
Please report the utterance count, ratings per output, and rating coverage for each system in the existing human evaluation. Please provide confidence intervals for paired differences using the collected ratings.
2 comments
Thank you for your reponse. The results support modality-specific masking with better translation scores and similar timing metrics. I maintain my rating.
Thank you for your constructive feedback and suggestions for improving experimental isolation and reporting. We address each concern below.
1. Reward allocation, normalization, and empirical statistics.
We agree that Table 6 isolates individual reward components but does not independently examine reward aggregation and normalization. To further investigate reward allocation, we conducted an additional ablation applying all three rewards to all output tokens.
Compared with modality-specific masking, this variant yields worse QE (4.12 → 4.35), while maintaining similar Dur@5% (92.2% → 92.1%) and pause F1 (0.550 → 0.553). These results support modality-specific masking as a favorable quality–timing trade-off.
We also provide the requested training diagnostics from the final model's Stage-4 run:
2. Human-evaluation coverage.
Each system was evaluated on 100 utterances, with at least two human raters assessing each output and consistent evaluation coverage maintained across all systems. We will explicitly include these details in the revised evaluation protocol to improve transparency and reproducibility.
3. Selection effects.
Our test evaluation is not restricted to samples accepted by the data construction pipeline.
As shown in Table 13, the Stage-3 pipeline produces accepted outputs for only 47.5% of test inputs, whereas TransDubber successfully generates outputs for 99.4%. This demonstrates that the model's inference coverage extends substantially beyond the subset accepted by the construction pipeline.