acceptodds
Under review as a conference paper at ICLR 2027

Structured Prosody Modeling for Long-Form Text-to-Speech via Semantic-Conditioned Cross-Scale Regularization

Abstract

Despite recent advances in neural text-to-speech synthesis, long-form generation still suffers from prosodic degradation, including inappropriate pauses, pitch drift, unstable energy patterns, and rhythmic discontinuities. We argue that these failures arise because prosody is often modeled as a flat acoustic sequence, a global style variable, or a reference-conditioned latent factor, without explicitly enforcing structural consistency across phrase-, sentence-, and discourse-level contexts. To address this limitation, we formulate long-form prosody modeling as a cross-scale structural-consistency problem and propose SPM-TTS, a structured prosody modeling framework for neural speech synthesis. SPM-TTS couples three components: 1. **Segmental Information-Guided Generation**: introduces linguistically motivated boundaries, segmental position encodings, and boundary-conditioned acoustic prediction to improve local prosodic organization. 2. **Semantic-conditioned Prosody Consistency Regularization**: uses global prosodic anchors, semantic offsets, and adaptive consistency weights to suppress harmful long-range drift while preserving semantically justified local prosodic variation. This regularization is applied only during training and introduces no additional decoding cost at inference time. 3. **Neural codec tokenizer**: provides expressive acoustic representations that support stable reconstruction and prosody-aware generation. Experiments on AISHELL-3 and long-form Chinese inputs show improvements in speech quality, intelligibility, speaker similarity, and prosodic stability over representative baselines, with more stable pitch and energy trajectories in long-form generation. Ablation studies further indicate that segment-aware local modeling, semantic-conditioned global regularization, and codec-level acoustic representation provide complementary gains. These results suggest that explicit cross-scale structure provides an effective inductive bias for robust long-form TTS.

Primary area:
generative models
Keywords:
text-to-speech, long-form speech synthesis, prosody modeling, semantic conditioning, neural codec

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bB7TOpenReview ↗
Rating
3(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SPM-TTS, a structured prosody modeling framework for long-form text-to-speech synthesis. The method combines Segmental Information-Guided Generation (SIG), Semantic-conditioned Prosody Consistency Regularization (S-PCR), and neural codec representations to improve local prosodic transitions and global consistency. Experiments on AISHELL-3 and long-form Chinese texts demonstrate improvements in speech quality, intelligibility, and prosodic stability compared with several representative baselines.

Critical strengths

The paper addresses prosodic inconsistency in long-form TTS, an important challenge in speech synthesis. The distinction between local segment-level modeling and global prosodic consistency is well motivated.

Critical weaknesses

Several important aspects of S-PCR remain unclear, including the construction of the global prosodic anchor, the definitions of the smoothing and variance losses, and the computation of differentiable prosodic statistics from predicted codec representations. These omissions make it difficult to fully assess and reproduce the proposed method.

Questions for authors

no

Reviewer_kwG3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SPM-TTS for long-form speech synthesis by combining Segmental Information-Guided Generation (SIG), Semantic-conditioned Prosody Consistency Regularization (S-PCR), and a neural audio codec. The reported results are promising, but several concerns remain regarding the implementation and experimental validation.

Critical strengths
  • The paper addresses an important problem in long-form TTS, particularly prosodic drift and cross-sentence consistency.

  • Combining linguistic segmentation with prosody regularization is a reasonable approach, and the paper provides several experimental comparisons.

Critical weaknesses
  • The paper evaluates 40 long-form passages and reports F0 RMSE and Energy RMSE, but it does not clearly explain whether corresponding reference recordings are available or how temporal alignment is performed. Since these metrics require reference signals, the evaluation protocol needs clarification. Moreover, AISHELL-3 primarily contains short utterances, raising questions about the model's ability to generalize to passages of 200–2000 words.

  • S-PCR extracts F0 and energy statistics from generated speech, but the paper does not explain how gradients propagate through discrete acoustic tokens and prosody extraction. Furthermore, the learnable semantic offset in Eq. (5) may directly absorb the difference between local and global statistics, potentially weakening the intended regularization. The authors should clarify the optimization procedure and analyze the role of these offsets.

  • SIG largely relies on segmentation markers and positional encodings, while SpeechTokenizer is an existing component. S-PCR is based on semantic-weighted matching of local and global prosody statistics. Stronger comparisons with existing segmentation methods and standard global prosody regularization are needed. The configurations in Table 3 should also be clarified, with independent and combined component ablations.

  • The evaluation mainly relies on PESQ, STOI, and F0-related metrics, which do not directly capture cross-sentence pauses, rhythmic continuity, or long-term prosodic drift. Length-stratified evaluations and boundary-level prosody measurements would strengthen the claims. In addition, the MOS results lack confidence intervals and significance tests, and the training details remain incomplete.

Questions for authors

see Weaknesses

2 comments
AuthorsOpenReview ↗

Response to Reviewer kwG3: Clarification of S-PCR Optimization and Long-form Evaluation

We sincerely thank the reviewer for the thoughtful and constructive feedback, and for recognizing the value of long-form prosodic consistency and linguistic-segmentation-based regularization. We address all concerns below and acknowledge areas requiring clearer disclosure.

1. Long-form Evaluation and Reference Alignment

We first clarify our two evaluation settings: AISHELL-3 is used for standard objective evaluation, while long-form testing uses a separate held-out set of 40 Chinese passages, split into four length bins (200–500, 500–1000, 1000–1500, 1500–2000 words, 10 passages each) with manually verified coherent discourse structure.

We agree the manuscript insufficiently documents paired reference recordings, their provenance, and the temporal alignment pipeline for reference-based F0F_0 and energy metrics. We will explicitly clarify reference sources, text-recording correspondence, alignment procedures, and duration mismatch handling, and distinguish reference-dependent acoustic errors from reference-free long-range prosodic stability measures.

2. S-PCR Optimization and Semantic-conditioned Offsets

We agree the S-PCR optimization path requires more precise explanation, especially with discrete codec tokens.

A key detail of Eq. (5) is that the semantic offset is predicted solely from text-side representations: δi=MLP([hi;hg;hi−hg]).\delta_i=\mathrm{MLP}([h_i;h_g;h_i-h_g]). It does not take generated segment prosodic statistics (FiF_i) as input. The shared semantic mapping captures systematic, semantically appropriate prosodic deviations, producing a semantic-conditioned target rather than uniform relaxation of the global constraint.

We acknowledge the optimization behavior and offset magnitude are under-explained. We will clarify the full optimization procedure, stop-gradient role, differentiable prosody computation (distinguished from offline WORLD/pyworld evaluation), and the gradient path to the acoustic generator, with component-wise ablations comparing zero offsets, semantic offsets, and global moment matching.

3. SIG, Existing Components, and Ablation Design

We do not claim SpeechTokenizer as a novel neural codec; it provides the acoustic representation for generation.

SIG’s contribution lies in integrating semantic segmentation, explicit boundary markers, segmental position encoding, and segment-conditioned acoustic prediction to organize local generation by linguistic structure, complementing S-PCR’s cross-segment consistency constraint.

We agree comparisons to existing segmentation and global prosody regularization lack detail. We will clarify Table 3 configurations (independent vs. progressive components) and add isolated and combined ablations of SIG, S-PCR, and the codec, including conventional global moment matching baselines, to disentangle individual effects.

4. Evaluation Metrics, MOS, and Reproducibility

We agree PESQ, STOI, and F0F_0 metrics alone cannot fully capture pause placement, rhythmic continuity, or long-range prosodic drift. We will supplement evaluation with segment- and boundary-level measurements: pause detection, pause duration consistency, and long-range F0F_0/energy drift, alongside length-stratified analysis across the four passage groups.

For MOS: 24 native Mandarin listeners provided 600 ratings across 25 unique samples (5 per system variant). We acknowledge limited stimulus diversity restricts the strength of perceptual conclusions. We will explicitly state this limitation, add confidence intervals and appropriate statistical analysis, and expand stimulus coverage where feasible.

We will also fully document training configurations: model architecture, semantic encoder, segmentation procedure, tokenizer settings, optimization hyperparameters, and all regularization terms (including smoothness and variance loss definitions).

5. Closing Remarks

We thank the reviewer for identifying these critical issues. The core motivation of SPM-TTS remains valid: local prosody should be semantic-conditioned, while long-form generation should maintain appropriate global consistency.

We acknowledge the original submission lacks sufficient implementation detail and targeted evaluation to fully validate this mechanism. We will improve methodological transparency, clarify the boundaries of current empirical evidence, and strengthen controlled evaluations accordingly.

We hope these clarifications address the reviewer’s concerns and demonstrate the promise of semantic-conditioned cross-scale prosody modeling for long-form speech synthesis.

AuthorsOpenReview ↗

Additional Experimental Evidence and Clarifications Following the Review

Dear Reviewer kwG3,

Thank you again for your constructive feedback. We have taken your concerns seriously and conducted additional experiments and a targeted audit of the recovered implementation and evaluation records. We would like to share several new findings that were not available in our initial response.

1. Additional evaluation of text segmentation (SIG).

We evaluated a frozen learned BiLSTM boundary predictor on 528 unique texts from 113 speakers, covering 3,862 interior character positions. The normalized texts do not overlap with its training and development manifests. These recordings are from a previously excluded subset of the official training corpus, not the official test split.

The learned predictor achieves 94.27% precision, 95.76% recall, and 95.01% F1 (speaker-cluster bootstrap 95% CI: 94.07–95.99), compared with 20.50% F1 for fixed five-character segmentation and 39.87% for a random control matched to its predicted boundary counts. The paired F1 improvement over this random control is 55.14 percentage points (95% CI: 52.79–58.08).

Importantly, our implementation audit identified a distinction that we now explicitly acknowledge. The currently recovered primary inference script uses punctuation-based segmentation with a 12-character limit, rather than loading this BiLSTM predictor. We evaluated this recovered rule on the same texts and obtained only 4.48% F1. We therefore do not attribute the BiLSTM result to the historical end-to-end generator or claim that it validates acoustic interleaving. Rather, it provides independent evidence that learned boundary prediction is substantially more effective than simple heuristic segmentation. We will make this implementation distinction explicit in the revised description of SIG.

2. Reference-based metric verification.

We reproduced the located codec-reconstruction evaluator, obtaining PESQ-WB 4.21398 and STOI 0.99022 for one reference–reconstruction pair. We also conducted a controlled duration-handling test: appending a one-second repeated tail to an identity signal did not change either score because the evaluator truncates both signals to the shorter duration. This demonstrates an important limitation of the recovered evaluation protocol. We agree that codec reconstruction cannot establish text-conditioned TTS quality, and that the original aggregate synthesis metrics require traceable generated–reference pairs and appropriate duration handling.

3. Clarification of remaining experimental limitations.

Our provenance audit could not establish reliable run-level links between the historical Table 3 configurations, their checkpoints, and the Figure 2 generated audio. We have also not recovered the complete matched S-PCR training and ablation records, the text-to-reference mapping and per-passage results for the 40 long-form passages, or the listener-level MOS records. We do not want to overstate the evidence or treat these missing records as verified results.

We are committed to revising the manuscript to clarify the recovered implementation, distinguish directly reproduced findings from historical results with unresolved provenance, and complete the missing controlled S-PCR ablations, traceable long-form evaluation, and listener-level perceptual analysis before retaining the corresponding quantitative claims in a complete subsequent version.

We recognize that the additional boundary experiment does not, by itself, establish the full end-to-end effectiveness of SPM-TTS. Nevertheless, we hope that the new controlled evidence, the implementation audit, and our willingness to explicitly correct the scope of the claims demonstrate our serious effort to address your concerns.

We sincerely appreciate your consideration and respectfully ask whether these additional findings and clarifications could lead you to reconsider your assessment of the submission.

Reviewer_Xu2yOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a SPM-TTS model targeting the prosodic degradation in long-form TTS via 2 components: Segmental-Information-Guided Generation (SIG), which inserts boundary tokens and segmental position encodings and conditions acoustic prediction on segments; Semantic-conditioned Prosody Consistency Regularization (S-PCR), which aligns segment-level F0/energy statistics with a global target plus a semantics-predicted offset and uses adaptive weights plus smoothness/variance terms. A SpeechTokenizer-based neural-codec representation is utilized as the underlying tokenizer.

Critical strengths

The prosodic degradation problem in long-form TTS is a task valuable to be studied.

Critical weaknesses
  1. Lack of technical novelty and the relevant literature: The paper does not establish what is new relative to existing work. Segment-boundary conditioning is a long-standing idea in TTS [1,2], and hierarchical/multi-scale prosody modeling is equally well studied [3,4]. The proposed S-PCR is, at its core, global F0/energy moment matching with a learned semantic offset and adaptive weight, which is very close to global-variance methods [5] and prior prosody-consistency losses [6].
  2. Lack of method description and reproducibility: The paper is far below the expected standard of a research submission. The main SPM-TTS pipeline—backbone architecture, parameter count, training data, text-segmentation procedure, semantic encoder, SpeechTokenizer configuration, optimization, and inference—is not described; in particular, the paper never explains how a text segment X_i is aligned to the acoustic frames/tokens over which F_i is computed (forced alignment, duration predictor, attention, or a heuristic). In total, the main body is under eight pages, with no algorithm/pseudocode and large portions of the method left implicit.
  3. Lack of sufficient and reliable empirical evidence:
  • (i) For evaluation metrics, the MOS study uses only five samples per variant (25 in total); the PESQ and STOI are intrusive metrics that require time-aligned reference speech; the paper does not state which reference is used, how generated and reference signals are aligned, or how duration/prosody mismatch is handled.
  • (ii) For experiment coverage, the only ablation (Table 3) is a coarse additive sweep rather than a component-isolated analysis, and the in-house “fixed-block AR-Diffusion” baseline is unspecified and not shown to share the backbone or training budget of SPM-TTS. The external systems are explicitly not budget-matched.
  • (iii) Finally, the central claims about pauses, rhythm, and discourse-level consistency are not measured.

References

  • [1] M. Ostendorf and N. Veilleux, “A Hierarchical Stochastic Model for Automatic Prediction of Prosodic Boundary Location,” Computational Linguistics, vol. 20, no. 1, pp. 27–54, 1994.
  • [2] M. Kune?ová and J. Matou?ek, “Neural Speech Synthesis with Enriched Phrase Boundaries,” Proc. Interspeech 2023, pp. 4833–4837.
  • [3] G. Sun, Y. Zhang, R. J. Weiss, Y. Cao, H. Zen, and Y. Wu, “Fully-Hierarchical Fine-Grained Prosody Modeling for Interpretable Speech Synthesis,” Proc. ICASSP 2020, pp. 6264–6268.
  • [4] X. Li, C. Song, J. Li, Z. Wu, J. Jia, H. Meng, “Towards Multi-Scale Style Control for Expressive Speech Synthesis,” Proc. Interspeech 2021, pp.4673-4677.
  • [5] T. Toda and K. Tokuda, “Speech Parameter Generation Algorithm Considering Global Variance for HMM-Based Speech Synthesis,” Proc. Interspeech 2005, pp. 2801–2804.
  • [6] R. Liu, J. Xi, Z. Jiang, and H. Li, “FluentEditor: Text-based Speech Editing by Considering Acoustic and Prosody Consistency,” Proc. Interspeech 2024, pp. 3435–3439.
Questions for authors
  1. Exactly how are PESQ and STOI computed on generated speech? What is the reference signal, how are the two signals time-aligned, and how is duration/prosody mismatch handled?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 1: Clarification of Technical Contributions, Methodology, and Evaluation

We thank the reviewer for the careful assessment and clarify the intended contribution and limits of the current evidence.

1. Contribution and prior work

We do not claim that boundary markers, hierarchical prosody modeling, global-variance constraints, or neural codecs are novel individually. The intended contribution is S-PCR's semantic-conditioned, segment-specific consistency target, integrated with segment-aware acoustic generation.

Unlike fixed global moment matching, S-PCR does not force every segment statistic FiF_i to match one global statistic FgF_g. It predicts

\hat F_i=\mathrm{sg}(F_g)+\delta_i,$$ with a semantic-distance-dependent weight $$w_i=\mathrm{clip}(\exp(-\tau\|h_i-h_g\|_2),w_{\min},w_{\max}).$$ The target and constraint strength thus vary by segment, preserving appropriate local deviations while discouraging cross-segment drift. SIG supplies the segment structure through semantic segmentation, `⟨SEG⟩` markers, segmental position encodings, and segment-conditioned acoustic prediction. Our claim concerns this specific formulation and integration, not the invention of its individual ingredients. We acknowledge that the original Related Work did not make this distinction sufficiently clear. ### 2. Method and reproducibility SIG and acoustic-token generation are used at inference; S-PCR is a training-time regularizer removed at inference. SpeechTokenizer with residual vector quantization provides the acoustic representation. S-PCR uses segment/global semantic representations, segment-level F0/energy means and standard deviations, a semantic offset predictor, adaptive weights, and smoothness/variance terms. The submission omits: the backbone and semantic encoder, segmentation implementation, tokenizer configuration, optimization settings, definitions of $L_{\mathrm{smooth}}$ and $L_{\mathrm{var}}$, and the gradient path from S-PCR to the acoustic generator. The role of $\mathrm{sg}(\cdot)$ also needs clarification. These are genuine reproducibility gaps and cannot be inferred from the figures. The method description must state the actual implementation, how predicted prosodic statistics are obtained, and how the regularizer updates the generator. ### 3. Evaluation and claims **MOS.** The appendix reports 24 listeners, 25 distinct samples across five variants (five per variant), and 600 ratings. We agree that 600 ratings are not 600 independent stimuli and that five samples per variant provide limited diversity. MOS should therefore be treated as preliminary perceptual evidence, not a large-scale estimate of generalization. We will state this limitation and avoid overstating its statistical strength. **PESQ/STOI.** The manuscript does not sufficiently specify the reference waveform, temporal alignment, or handling of duration mismatch. Using the same text inputs and reference prompts does not resolve this; a prompt is not necessarily the paired target reference. This reporting gap must be addressed by specifying the actual reference, resampling, alignment, and duration handling. **Long-form prosody.** The test set contains 40 held-out Chinese passages, with 10 in each of four length ranges (200–500, 500–1000, 1000–1500, and 1500–2000 words). Table 4 reports WER, F0 RMSE, and energy RMSE. We agree these metrics do not directly measure pause placement/duration, rhythm continuity, or discourse-level appropriateness. The current results support claims about the reported intelligibility and F0/energy measures, but do not alone establish all broader pause- and discourse-level claims. We will narrow the claims and identify direct pause/rhythm evaluation as a limitation. ### 4. Baselines The paper identifies F5-TTS, CosyVoice3, and Qwen3-TTS as reference systems, not training-budget-matched comparisons. We agree this distinction should be prominent. The in-house fixed-block AR-Diffusion baseline also lacks sufficient architectural and training details to establish strict parity. External comparisons should be interpreted as references; the ablation table shows trends associated with SIG, S-PCR, and the tokenizer, but does not establish fully isolated effects under a documented, budget-matched protocol. Our claim is not that all aspects of long-form prosody are solved: SPM-TTS studies a semantic-dependent, segment-specific regularization target to balance local prosodic variation and utterance-level consistency. We agree that the submission needs clearer implementation disclosure and more cautious interpretation of the current evaluation. We hope this clarification makes the intended contribution and limits of the evidence explicit.

Comments on the paper

1 comment

Dear reviewers,

The authors have submitted their rebuttals. I would be grateful if you could take a moment to review them carefully and respond as appropriate. Thank you for your ongoing engagement and support.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.