acceptodds
Under review as a conference paper at ICLR 2027

Learning Timeline-Preserving Speech-to-Speech Translation

Abstract

Speech-to-speech translation (S2ST) increasingly preserves *what* is said and *how* it sounds, but not necessarily *when* speech and silence occur. For dubbing and other temporally grounded applications, matching total duration is not enough: speech and pauses must also occur at corresponding moments. We formulate *timeline-preserving S2ST*, which aims to preserve the source endpoint and salient internal pauses on a shared absolute timeline. We present TransDubber, an end-to-end autoregressive system that learns this temporal structure directly from aligned target waveforms, without inference-time frame-level timing control or post-hoc temporal correction. To provide this supervision, we construct 642K exactly duration-matched English-Chinese speech pairs by translating and synthesizing target speech on source-defined temporal scaffolds. Training combines a progressive supervised curriculum with verifier-guided reinforcement learning to balance translation quality and temporal alignment. On a 4,353-utterance test set spanning read and spontaneous speech, TransDubber increases the proportion of translations within 2% of the source duration from 25.6% to 61.5% and pause-alignment F1 from 0.118 to 0.550 compared with Uniss, while maintaining competitive translation quality. Our analysis shows that aligned supervision establishes timeline following, while reinforcement learning refines temporal precision and partially offsets the translation-quality cost of strict synchronization. Together, these results demonstrate the effectiveness of our data and training framework for timeline-preserving S2ST. We will release the training data, model checkpoints, and code upon publication. An anonymous demo is available at https://anonymous.4open.science/w/transdubber-demo-38E6.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Speech-to-Speech Translation, Dubbing

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_KrGQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how English-to-Chinese speech translation can preserve source endpoints and salient internal pauses. It constructs synthetic speech supervision aligned to the source timeline. It then trains direct generation with a joint text-speech model and a four-stage curriculum. GSPO combines translation-quality, global-duration, and pause rewards. It also shows tradeoffs in translation quality and naturalness.

Critical strengths
  1. The paper clearly distinguishes total-duration agreement from internal-pause alignment. It makes the source's absolute timeline an explicit generation constraint. The definition helps evaluate dubbing with fixed timing requirements, beyond translation meaning alone.

  2. Source-aligned supervision is a substantive contribution. The authors construct a large training corpus. The final model generates directly and avoids the construction pipeline repeating search at inference.

  3. The final model jointly generates translated text and speech. Inference does not repeat the candidate search and audio corrections used to construct the training data. This demonstrations take effort to build but development is simpler. This is a practical advantage.

Critical weaknesses
  1. Table 6 shows contributions of the reward components. It does not separately compare reward aggregation, normalization, and modality masks. The evidence therefore supports the combined recipe but does not isolate the contribution of the specialized allocation mechanism.

  2. The human evaluation lacks utterance counts, ratings per output, and confidence intervals. These issues limit assessment of the combined tradeoff among timing, content accuracy, and naturalness.

  3. The synthetic-supervision pipeline rejects examples that cannot meet local duration constraints. The retained data may therefore favor expressions that are easier to synchronize. The authors disclose this possibility, but the experiments do not isolate its effect.

Questions for authors
  1. Are there existing results using a scalar reward, shared normalization, or no modality masks? Please provide available results, or supply reward scales, advantage statistics for each token group.

  2. Please report the utterance count, ratings per output, and rating coverage for each system in the existing human evaluation. Please provide confidence intervals for paired differences using the collected ratings.

2 comments

Thank you for your constructive feedback and suggestions for improving experimental isolation and reporting. We address each concern below.

1. Reward allocation, normalization, and empirical statistics.

We agree that Table 6 isolates individual reward components but does not independently examine reward aggregation and normalization. To further investigate reward allocation, we conducted an additional ablation applying all three rewards to all output tokens.

Reward allocationBLEU ↑QE ↓Dur@5% ↑[email protected] ↑
Modality-specific masks20.804.1292.20.550
All three rewards → all tokens19.404.3592.10.553

Compared with modality-specific masking, this variant yields worse QE (4.12 → 4.35), while maintaining similar Dur@5% (92.2% → 92.1%) and pause F1 (0.550 → 0.553). These results support modality-specific masking as a favorable quality–timing trade-off.

We also provide the requested training diagnostics from the final model's Stage-4 run:

ComponentLogged reward mean ± SDAdvantage SDAssigned tokensLoss weight
Quality0.903 ± 0.0350.62Translation text0.2 / 2.2
Pause−0.178 ± 0.0770.85Speech codec1.0 / 2.2
Duration−0.032 ± 0.0380.41Speech codec1.0 / 2.2

2. Human-evaluation coverage.

Each system was evaluated on 100 utterances, with at least two human raters assessing each output and consistent evaluation coverage maintained across all systems. We will explicitly include these details in the revised evaluation protocol to improve transparency and reproducibility.

3. Selection effects.

Our test evaluation is not restricted to samples accepted by the data construction pipeline.

As shown in Table 13, the Stage-3 pipeline produces accepted outputs for only 47.5% of test inputs, whereas TransDubber successfully generates outputs for 99.4%. This demonstrates that the model's inference coverage extends substantially beyond the subset accepted by the construction pipeline.

Thank you for your reponse. The results support modality-specific masking with better translation scores and similar timing metrics. I maintain my rating.

Reviewer_HLgiOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses the task of timeline-preserving speech-to-speech translation (S2ST), where the generated target speech is expected to not only match the total utterance duration of the source speech (global isochrony) but also preserve the internal speech-silence intervals on a shared timeline.

The authors develop a three-stage pipeline to synthesize aligned parallel speech corpora: extract source voice-activity scaffolds; translate individual speech segments; generate natural-pace target speech (stage 2) and timeline-preserving target speech (stage 3) with local character budgets and constrained durations, using VoxCPM2 TTS.

Built upon Qwen3-ASR-1.7B and the CosyVoice3 speech codec and flow-matching decoder, the proposed model (TRANSDUBBER) generates pause-marked source ASR transcripts followed by interleaved translated text and speech-codec tokens. Training proceeds through a four-stage curriculum: (1) pause-aware ASR and text translation grounding, (2) natural-pace S2ST warm-up, (3) timeline-preserving supervised fine-tuning (SFT), and (4) Group Sequence Policy Optimization (GSPO) with decoupled, token-masked rewards for translation quality, global duration agreement, and local pause alignment. On a 4,353-utterance English-to-Chinese evaluation set drawn from CoVoST2 and GigaSpeech, TRANSDUBBER increases strict duration matching (Dur@2%\text{Dur@2\%}) from 25.6% to 61.5% and pause-alignment F1 ([email protected]\text{[email protected]}) from 0.118 to 0.550 over the strongest expressive S2ST baseline (UniSS), while maintaining comparable translation quality.

Critical strengths
  1. The motivation and problem is practical. Moving beyond utterance-level duration matching to internal speech-silence timeline preservation addresses a genuine limitation in automatic dubbing, where speaking through dramatic or respiratory pauses disrupts scene timing.
  2. The proposed synthetic data construction pipeline is reasonable. The authors promise to release this dataset, which could be a valuable contribution to the community.
  3. The training process is formulated as a multi-stage SFT and RL curriculum, which separates modality acquisition from temporal synchronization. The authors provide an ablation study to demonstrate the effectiveness of the progressive curriculum, which is informative.
Critical weaknesses
  1. In Section 3.5 (Eq. 1 and Eq. 17), RqualR_{\text{qual}} is masked solely to the Chinese text tokens (MtextM_{\text{text}}), whereas RdurR_{\text{dur}} and RpauseR_{\text{pause}} are masked solely to the speech-codec tokens (MspeechM_{\text{speech}}). However, under the interleaved 5:10 text-to-speech decoding schedule, the length and syllabic density of the generated Chinese text MtextM_{\text{text}} fundamentally constrain the achievable duration and pause budget of the speech sequence. By blocking temporal gradients (Adur,ApauseA_{\text{dur}}, A_{\text{pause}}) from updating MtextM_{\text{text}}, the text policy cannot learn to select more concise synonyms when the source speaking window is short or slightly more descriptive phrasing when the window is long. There is no ablation comparing this hard modality split against propagating duration/pause advantages to both MtextM_{\text{text}} and MspeechM_{\text{speech}}.
  2. In Section 4.1 and Table 2, COMET and MetricX-QE are computed on the intended-text track (the model's intermediate text output) rather than an ASR transcription of the generated waveform. So the quality of the speech generation part is still not clear to me.
  3. Human evaluation seems to be limited to translation quality and speech naturalness. It would be ideal to have human evaluation on the claimed timeline-preserving capability
Questions for authors
  1. There will be cases where an English utterance has a pause between two clauses that require reordering in natural Mandarin (e.g., a long post-verbal conditional or temporal clause following a pause). Does the data construction pipeline also cover such cases?
1 comment

Thank you for your constructive feedback and specific suggestions. We address each concern below.

1. Temporal feedback to text tokens.

We agree that lexical choices can constrain speech timing. To investigate reward allocation, we compared our modality-specific masking strategy with a variant that applies all three rewards to all output tokens.

Reward allocationBLEU ↑QE ↓Dur@5% ↑[email protected] ↑
Modality-specific masks20.804.1292.20.550
All three rewards → all tokens19.404.3592.10.553

The all-token variant achieves comparable temporal alignment but lower translation quality. These results support modality-specific masking as a favorable quality–timing trade-off.

2. Quality of generated speech.

While Table 2 evaluates intended-text quality, Appendix G (Table 9) additionally evaluates ASR transcriptions of the generated waveforms, thereby accounting for speech-realization errors.

TransDubber achieves COMET scores of 0.745/0.809 on CoVoST2/GigaSpeech, compared with 0.738/0.732 for UniSS. We will highlight this complementary evaluation more prominently in the revised manuscript.

3. Human evaluation of timeline preservation.

We directly evaluate source-timeline correspondence using duration accuracy and pause-alignment metrics (Appendix E). In addition, our human naturalness evaluation explicitly considers rhythm, phrasing, and pauses (Appendix F).

These evaluations serve complementary purposes: objective metrics quantify temporal correspondence with the source, while human ratings assess the perceptual naturalness of the generated speech.

4. Cross-pause reordering.

Our current pipeline does not explicitly account for semantic reordering across pause boundaries. We acknowledge this limitation and will discuss it in the Limitations and Future Work section of the revised manuscript.

Reviewer_6aZAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper introduces timeline-preserving speech-to-speech translation (S2ST), it's helpful for scenarios like video dubbing.

  • The authors build a synthetic EN-to-CN corpus of 642K duration-matched pairs. The pipeline segments the source audio with VAD, translates each region under a per-region budget, synthesizes it with VoxCPM2 in the source speaker's voice, and renders it back into the source timeline, followed by the quality filtering.

  • The proposed TransDubber is initialized from Qwen3-ASR-1.7B, which first generates English transcript with <pause> markers, then interleaves Chinese text and speech tokens, then uses a flow-matching decoder to turn it into audio.

  • The authors applied the detailed ablations, comparing the proposed model with baselines, as well as the cascaded systems. Ablations attribute gains to the timeline-preserving SFT data, with RL adding further refinement.

Critical strengths
  • The stage-wise analysis makes clear where the timing ability comes from. The paper is also clearly written and easy to follow.
  • Preserving internal pauses, besides total duration, matters for dubbing.
Critical weaknesses

The paper is clearly written, and I can see the substantial effort that went into data preparation, model training, and evaluation. However, the following flaws make me hesitant to accept it.

  • The contribution is mainly data, no novelty is kind of limited. The ablations show that most of the timing gains come from SFT on the synthetic duration-matched data. Beyond building aligned data and fine-tuning on it, the paper offers few generalizable insights for the broader ICLR audience.
  • The generality and evaluation are not yet convincing. All training targets come from a single synthetic pipeline. I would like to raise my score (to weak accept) if real human dubbing or other language experiments are presented. The cascaded system still wins on translation quality and naturalness, so it is unclear whether an end-to-end model is the right choice here.
Questions for authors
  • Will the dataset, data pipeline, and model be released? This would significantly affect how I weigh the data contribution.
  • Similar to the question in the Critical Weaknesses part, how does the model perform on real-world cases?
2 comments

Thank you for recognizing the practical motivation and our stage-wise analysis. We appreciate your constructive feedback and address your concerns below.

1. Release.

Yes. Upon publication, we will release the full codebase (including the data construction pipeline), model weights, and training/test data to facilitate reproducibility and future research.

2. Contribution beyond data.

While aligned supervision provides the foundation, the effectiveness of our approach also depends on a progressive training curriculum and modality-specific multi-objective RL. The results in Table 6 further highlight the importance of reward design. Taken together, these findings suggest that performance gains stem not only from corpus construction, but also from the training sequence and the coordinated optimization of semantic and temporal objectives.

3. Real-world evaluation.

To evaluate generalization to real-world dialogue, we randomly sampled 100 utterances from the MELD test set and performed direct inference without further training. We report duration-stratified results alongside the corresponding GigaSpeech test subsets.

Test setDurationQE ↓Dur@2% / 5% / 10% ↑tIoU ↑[email protected] ↑
MELD5–8 s3.2676.9 / 98.7 / 98.70.5190.734
GigaSpeech5–8 s3.3074.1 / 98.4 / 100.00.3750.607
MELD8–12 s4.9971.4 / 85.7 / 90.50.5840.848
GigaSpeech8–12 s3.6170.6 / 97.5 / 99.90.3600.633

These results demonstrate the model's ability to follow source timelines in real-world dialogue without adaptation. We acknowledge that longer MELD clips pose greater challenges, which are further compounded by their out-of-distribution (OOD) nature, leading to slight decreases in translation quality and duration accuracy.

Due to resource constraints, conducting large-scale training on additional languages is not feasible in the short term. Nevertheless, extending our approach to a broader range of languages remains an important direction for future work.

4. End-to-end vs. cascaded system.

While the cascaded system achieves higher translation quality and naturalness, TransDubber offers key advantages in avoiding error accumulation, reducing multi-stage latency, and ensuring temporal alignment (92.2% vs. 13.6% Dur@5% and 0.550 vs. 0.076 pause F1, Table 2). Moreover, the cascaded system produces accepted outputs for only 47.5% of inputs (Table 13), further highlighting the practical advantages of our unified end-to-end framework.

Dear Reviewer 6aZA, thank you again for your time and constructive feedback.

As the end of the discussion is approaching, we wanted to gently check if our rebuttal has resolved your concerns. Please let us know if any further clarification is needed. We are fully available to answer any remaining questions before the deadline.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback. In response, we provide additional evaluation on 100 randomly sampled MELD test utterances without fine-tuning, an all-token reward-allocation ablation, and empirical reward/advantage statistics. We also clarify waveform-based translation evaluation, human-evaluation coverage, and inference coverage relative to the construction pipeline.

We confirm that we will release all code, including the data-construction pipeline, model weights, and training/test data upon publication.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.