acceptodds
Under review as a conference paper at ICLR 2027

FibreTTS: A Fibre-Bundle View on Content-Dependent Control in Text-to-Speech

10 views

Abstract

Controllable text-to-speech (TTS) aims to preserve target words while expressing requested attributes such as emotion, speaking rate, and emphasis. Yet language-model-based systems often produce different effects for the same instruction across texts, and stronger control can introduce content errors. To explain this interaction, we propose FibreTTS, drawing on fibre-bundle theory to distinguish content from reading while allowing their relationship to vary. Content representations form the base space, and the possible readings of each representation form its fibre. We construct text- and reference-associated coordinates that explain most of the observed representation variance, providing empirical support for this structured description. Building on this model, we examine two complementary properties and their implications for control. Local triviality provides shared coordinates for what is spoken and how it is read within a content neighborhood, while allowing the same reading change to have different acoustic realizations across texts. Fibre transport, defined through a chosen connection, guides adaptation of a source reading change to target content while retaining the target's own baseline reading. Experiments on two TTS systems, using public emotional speech data and private financial outbound-call data, support the coordinates' behavioral relevance and reveal the effects of direct cross-text control reuse. Transport-inspired residual updates improve the instruction-following scores on outputs whose original control effect was weak, and the improvements are statistically significant on both systems and datasets.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, controllability, text-to-speech generation, language models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_G44MOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes FibreTTS, a fibre-bundle-based framework for understanding content-dependent control in text-to-speech synthesis. It models linguistic content as the base space and possible speaking styles as fibres, providing a geometric perspective on how the same control instruction can produce different effects across texts. The authors construct content- and reading-associated coordinates through logit decomposition and investigate the limitations of directly transferring control representations between texts. Based on fibre transport, they further propose a residual correction method to adapt successful control effects from one text to another.

Critical strengths
  1. The paper investigates an interesting and meaningful problem in controllable TTS, i.e., why the same control instruction produces different effects across texts.
  2. The proposed residual correction method achieves improvements in instruction following while largely preserving linguistic content.
Critical weaknesses

I am not familiar with the fibre bundle theory, but I have several concerns:

  1. The paper develops a fibre-bundle formulation involving connections and parallel transport, but the actual method is implemented as a residual correction in the logit space. I want to know how the proposed update approximates the theoretically defined parallel transport.
  2. The empirical validation relies primarily on a low-rank decomposition of text- and reference-associated logit variations. However, a successful additive decomposition does not establish the existence of a fibre-bundle structure, local triviality, or a meaningful content projection, as described in the paper. These claims should be proved given that the authors introduce the fibre bundle theory.
  3. Without key controls/ablations such as simple residual transfer, random donor, and predictor-only, it is impossible to determine which part contributes to the performance improvement.
  4. There is no human evaluation regarding intelligibility or naturalness, e.g., reporting Naturalness MOS or attribute-related evaluation such as emotion before and after the transport, which is crucial for TTS research. Without human evaluation, the theoretical hypothesis is unreliable.
Questions for authors

The author needs to explain how the actual residual correction approximates connection-based parallel transport, rather than relying merely on conceptual analogies.

And I want to emphasize that my foremost concern is that the paper introduces an unnecessarily complex theoretical framework to explain a relatively straightforward problem. While the fibre-bundle perspective is interesting, its practical necessity and contribution remain unclear. The proposed method is essentially a residual correction in the logit space, without a rigorous connection to the theoretical formulation of parallel transport.

I think it is a bad practice to package a simple phenomenon as a complex theory.

The authors should avoid overcomplicating the presentation with mathematical concepts that do not directly contribute to the proposed method or its effectiveness and explain the necessity of utilizing such a complex mathematical theory in this paper.

Reviewer_VzAkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FibreTTS, a geometric framework inspired by fibre-bundle theory to analyze and improve content-dependent style/attribute control in language-model-based TTS systems (CosyVoice2 and IndexTTS-2.5). The authors model the space of speech representations as a fibre bundle where the base space B represents spoken content (text) and the fibres F represent possible readings (prosody, emotion, speaking style). Empirically, they extract the raw vocabulary logits right before the first speech token across a full grid of target texts and reference audios (T x R), apply a two-way additive decomposition (row/column means) followed by rank-20 SVD to obtain linear subspaces D_B and D_F, and show that these 40 dimensions explain 75-81% of logit variance (and 88-94% with nearest-neighbor residual corrections). They then show that perturbing B hurts CER and replacing a receiver's F coordinates with a donor's across all decoding steps degrades both CER and instruction following (IF). Finally, motivated by parallel transport, they fit a quadratic ridge regression predictor q(b, f) on the projection residual and transfer the residual deviation from an IF=5 donor to an IF=1-3 receiver at the first speech token, reporting statistically significant IF gains on ESD and a private Fintech outbound-call dataset.

Critical strengths
  • Investigating why the same style prompt, reference audio, or steering vector behaves inconsistently across different target texts—and why strong style control often triggers word errors—is an important and practical question for autoregressive LM-based TTS.
  • Running controlled grid experiments across two recent LM-based TTS architectures (CosyVoice2 and IndexTTS-2.5), three control interfaces (reference audio, text prompting, and activation steering), and two distinct domains (ESD emotional speech and conversational call-center speech) provides a broad empirical testbed.
  • The two-factor variance analysis of first-token logits (Figure 2 and Appendix F) offers a clean diagnostic showing that while marginal text and reference effects are largely additive, the remaining interaction term varies systematically with text similarity.
Critical weaknesses
  • There is a major disconnect between the heavy differential geometry formalism (smooth manifolds, local trivializations, transition maps g_VU, Ehresmann connections, and horizontal lifts along paths) and what is actually computed in the paper. In practice, no local charts, transition functions, or path-integrated parallel transport are ever estimated. Section 2.3.2 is simply a global two-way ANOVA (subtracting text and reference means on a balanced grid) followed by rank-20 PCA/SVD (ht,r = mu + D_B bt + D_F fr + eps). More importantly, while Section 4.1 formulates parallel transport as transforming the fibre coordinate f along a base path, the actual update in Section 4.2 (Eq. 10) leaves the receiver's B and F coordinates completely untouched and only modifies the orthogonal projection residual r in ker(D^+) using a polynomial regression fit q(b, f). Calling a two-way SVD decomposition plus bilinear residual regression a "fibre bundle with parallel transport" feels like heavy mathematical over-packaging that obscures rather than clarifies the underlying linear algebra.
  • The evaluation of the transport intervention in Section 4.3 (Table 1) suffers from a severe regression-to-the-mean and post-hoc selection bias. Specifically, the authors generate outputs for the grid, score them with the Gemini-3.1-Pro judge (1-5 scale), and then exclusively select low-scoring samples (IF = 1-3) as receivers and top-scoring samples (IF = 5) as donors. When you filter a stochastic autoregressive TTS generator and a stochastic LLM judge strictly for outputs that scored poorly (1-3) on the first draw, applying any small perturbation to the initial logits (or simply re-sampling with a new random seed) and re-evaluating will naturally produce a higher mean score due to regression to the mean. There is no control baseline that re-samples or applies a norm-matched random residual perturbation to these IF = 1-3 receivers, nor is there an evaluation of what happens when the transport update is applied to all samples (including IF = 4-5 receivers).
  • The comparison between "Direct F reuse fails" (Section 3.3, Figure 3c-d) and "Fibre transport succeeds" (Section 4.3, Table 1) is confounded by completely different intervention protocols. As noted in Section 3.3.1 and Appendix E, direct F reuse overwrites the model's logits with the static first-step basis D_F at every single autoregressive decoding step (t = 1 ... T), which unsurprisingly destroys autoregressive generation and causes CER to jump by +70 to +100 percentage points (Table 4). In contrast, the proposed transport intervention in Section 4.2 is applied only once at the very first speech token (scaled by k = 0.5), leaving all subsequent decoding steps untouched. You cannot attribute the difference in CER and IF to "fibre transport vs. unadapted reuse" when one intervention is applied at every token and the other is only a half-step nudge at token 0.
  • The entire pipeline is transductive and relies on oracle evaluation labels at test time. As stated in Appendix C.1, all target texts and references in the transport experiment come from the exact same T x R grid used to fit D_B, D_F, and q(b, f). Furthermore, performing the update requires already having an IF = 5 donor and knowing that the receiver scored IF = 1-3 via the Gemini judge, making it unclear how this method could be used as a practical inference-time control mechanism on unseen texts without first generating and judging a grid of candidates.
Questions for authors
  1. In Section 4.3 (Table 1), because receivers are post-hoc selected to have IF <= 3 and donors to have IF = 5, how much of the reported IF increase is simply regression to the mean from re-generating and re-judging low-scoring samples? What is the mean IF change on the exact same IF = 1-3 receiver set if you (a) simply re-sample from the unmodified model or add a random residual vector of the same norm ||k(r_hat - r)|| at the first speech token, and (b) apply your transport update across all samples without filtering by initial IF score?
  2. Why was direct F reuse (Section 3.3) applied at every decoding step while the transport update (Section 4.2) was applied only at the first speech token? What happens if direct F reuse (or transferring f_donor - f_receiver) is also applied with k = 0.5 only at the first speech token?
  3. In Section 4.1, parallel transport is derived as an update to the fibre coordinates f, yet in Section 4.2 (Eq. 10), the intervention leaves f_receiver unchanged and only modifies the orthogonal residual r outside the span of [D_B, D_F]. How does modifying only the unmodeled residual r correspond to parallel transport on the fibre F?
Reviewer_Xh1YOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper apply the idea from fibre-bundle thory to TTS. However, many of parts are undefined and the manuscript does not explain the rationale behind their design choices.

Critical strengths

Applying fibre-bundle theory to controllable TTS is interesting idea. If it works on theory, we could sample more instruction following samples.

Critical weaknesses

However, this paper have definite problem about the writing, especially for formulation and experimental setup.

First, the manuscript using the first logit before the speech,hh. Since the speech utterance usually starts with preceding silence, ususally first logit is highly focus on silence tokens. However, the manuscript is explaining that they can decompose the effects of text and reference from it. I find this setup difficult to understand. Also, even if it affects the final quality, for LLM-TTS, stochastic sampling process is also affect a lot during the generation, however it is not mentioned in the manuscript.

Second, to make concrete fibre transport in theory, we could make a map between two fribre from different bb. However, the paper did not mention the proposed SVD coordinates decompose the fibre well, and give stable map. In addition, while the text-to-speech is one-to-many problem even if the same condition is given, it does not mention that kind of generative modeling nature in the manuscript.

Third, I also have concerns about the test setups.

  1. I find the setup is difficult to understand, during the experimental setup, donors have high IF with 5, and receivers have lower IF. Are they generated samples or data from the datasets?
  2. For the IF value, the authors mentioned p-test, however, while the IF is based on integer, and it is measured on small samples, the authors should mention confidence interval similar to mean opinion scores.

Forth, some terms are not defined through the manuscript or using uncommon wordings.

  1. I am not used to use the word "Reading", I think utterance is more appropriate word.
  2. For figure1, explained variance of which value is not described.
Questions for authors

The main concern is that I cannot understand the fibre setup, why it uses the logits preceding speech tokens. As mentioned in the weakness section, my current understanding is that the method relies on logits associated with seemingly uninformative silence tokens. The authors should clarify the motivation behind this design choice and explain why these logits are expected to contain meaningful information. In addition, to prove this setup is meaningful, the authors need to visualize the original vector (or logit) distribution, SVD directions, data which have same coordinates, and etc, to show it to readers. Also, it should be great to make a demo page.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.