acceptodds
Under review as a conference paper at ICLR 2027

Steering intonational tunes in text-to-speech synthesis

Abstract

Text-to-speech systems built on residual vector quantization often generate audio in two stages: a transformer backbone emits the first level of each frame's token stack, and a smaller depth model generates the rest autoregressively. We ask which components of such a model are responsible for prosodic intonation, more precisely for generating the English *nuclear tune* (for instance, the falling pitch melody that distinguishes an assertion from a question's rising melody), and whether interventions on those components can be used to control intonation. Prompt-based control mostly fails; only a final question mark is a reliable lever. Interchange interventions in CSM and S2 locate the tune: the backbone's hidden state carries it to the waveform through the depth model, and the first level of the stack carries little or none of it. Steering the backbone state with a difference-of-means vector computed on new material generated with a final period or question mark moves the tune, but the push toward an assertion degrades the voice. Steering vectors computed instead from the backbone states associated with human imitations of theoretically-specified tunes, whose text carries no punctuation, and applied inside the generation loop, move most recordings to the other condition in both systems (even when fitted without the speaker being steered), at a small cost in predicted naturalness. These imitation-derived vectors are aligned with the one obtained from the two punctuated conditions far beyond a shuffled-label null, so the backbone state of a punctuated prompt already represents intonation. The steered recordings broadly become falling or rising, according to the intent, but the edge tone moves more coherently than the pitch accent, and the target tune cannot be selected precisely beyond this broad tendency.

Primary area:
interpretability and explainable AI
Keywords:
text-to-speech synthesis, intonation, causal analysis, activation steering, controllable speech generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_6xgJOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the transmission and control of intonation within CSM and Fish Audio S2 by swapping audio codes, replacing the inputs to the downstream generation module, and editing backbone states. For the main statement–question contrast, replacing the backbone state transfers most of the measured final-word pitch difference, whereas replacing the first-level audio codes has a smaller effect. Editing directions extracted from human speech also change the output contours, but do not reliably produce the specified intonation. The experiments use three sentences and ten speakers, with acoustic, speech-classification, and audio-quality metrics, but no listening evaluation.

Critical strengths

The paired input-replacement experiments directly compare the effects of backbone states and first-level audio codes on generated pitch. Re-running the downstream generation module with its original inputs also provides a control for variation caused by random sampling. The consistent pattern across the two systems provides more direct empirical evidence about the information-flow pathways under the tested conditions than representation classification alone.

Critical weaknesses
  1. The explanatory contribution is limited, and the scope of the conclusions is insufficiently supported. Input replacement and directional activation editing have established precedents. The paper’s main new insight concerns the roles of different inputs in two models for a specific intonation contrast, without distinguishing whether this division arises from the codec, model architecture, or training. A small intervention effect from first-level audio codes also does not directly establish that they “carry little to no intonation.”

  2. The main generation experiments cover only ten speakers and three similar short sentences. Repeated generation and multiple editing strengths do not broaden the textual coverage. The existing held-out-speaker evaluations and held-out-sentence tests for the offline punctuation-derived direction do not directly establish that the human-derived direction generalizes across sentences when applied during generation. The evidence therefore supports findings within a restricted setting, but is insufficient to establish a general mechanism of intonation or a reusable control principle.

  3. The evidence for the benefits of control and the claimed audio-quality advantage is insufficient. The paper provides no listening evaluation of generated speech, and automatic pitch metrics and template labels do not establish that listeners perceive the intended intonation. It also does not sufficiently explain its relationship to existing F0-conditioned control methods or demonstrate additional benefits through a comparable baseline, such as direct F0 editing.

  4. Offline editing uses a direction derived from punctuation conditions, whereas editing during generation uses a direction derived from human recordings. Both the editing procedure and the source of the direction therefore change, so the lower audio-quality cost cannot be attributed solely to applying edits during generation. Although the authors acknowledge that reliable selection of specific intonation patterns remains unsuccessful, the perceptual value and audio-quality advantage of the remaining coarse-grained control still require stronger evidence.

Questions for authors
  1. Does “first-level audio codes carry little to no intonation” refer to a small effect under the tested interventions, or to a limited amount of intonational information? What evidence distinguishes these interpretations and separates the contributions of the codec, model architecture, and training?

  2. Beyond the existing held-out-speaker evaluations, has the human-derived editing direction been tested on sentences used neither to extract the direction nor to select the editing strength? If not, which conclusions should be restricted to the three carrier sentences used here?

  3. How does the proposed method relate to existing F0-conditioned control methods? With comparable target information and magnitudes of pitch change, what evidence supports additional control or audio-quality benefits over direct F0 editing?

  4. Have offline editing and editing during generation been compared using the same editing direction and similar magnitudes of pitch change? If not, how can a change in the direction’s source be ruled out as an explanation for the audio-quality difference, or should the corresponding attribution be qualified?

  5. Could the authors provide a small set of paired unedited and edited audio examples, including both successes and failures? What evidence currently supports that listeners perceive the intended intonation and the audio-quality changes indicated by the automatic metrics?

1 comment

W1/Q1, "level 0 carries little to no intonation." We agree that this is a statement about these interventions, and the paper now says so (§7). Our evidence is interventional: swapping level 0 between a matched assertion and question moves the tune by ΔΦ\Delta_\Phi = 0.00 (CSM) and 0.07 (S2), and re-running the depth model with the other utterance's level 0 by 0.01 and 0.07, while the other utterance's backbone state moves it 0.98 and 0.95 (Table 1). The cross-sentence control shows that level 0 does carry the words (App. E.4, Table 8). §5 attributes the division to the trained models, not to the factorization. Our methods cannot currently separate the effect of the codec, the model architecture and the training, and §7 now states this.

W2/Q2, held-out sentences for the human vector in the loop. Added, exactly as asked (App. G, Table 9): for each sentence, the tune vector is refitted without it, the strength is chosen by the criterion on the other two sentences, and the held-out sentence is read in the loop. Held out it flips 20–57% (CSM) and 27–40% (S2) of recordings, against 30–47% and 23–40% in sample. The steering functions were also held out by speaker (Table 11). The conclusions remain restricted to three carrier sentences, as the limitations state.

W3/Q5, listening evaluation Listening test added (App. J, Table 13, Figure 4; see the general comment).

W3/Q3, F0-editing baseline. We do not include it, for three reasons: (1) our question is what the model's own states encode; (2) we want to approach intonational tunes as phonological abstractions, not as arbitrary pitch trajectories (there is empirical evidence that human subjects are not able to reproduce arbitrary variations in pitch trajectory heard in auditory stimuli); and (3) the actual pitch trajectory depends on each speaker and sentence, complicating the prescription of pitch targets.

W4/Q4, same direction under both procedures. Yes: both vectors are now run under both procedures (App. I, Table 12). The voice-quality difference follows the procedure, not the direction's source (see our response to c9AE W1). The attribution in the paper is changed accordingly.

Reviewer_c9AEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how CSM and Fish Audio S2 generate pitch patterns at the ends of sentences, such as the rise that can make a statement sound like a question. The authors generate pairs of recordings with the same words and voice, changing only the final punctuation. They then exchange internal states or audio tokens between the recordings. Replacing the backbone’s hidden state largely transfers the pitch pattern. Replacing only the first audio token at each frame has little effect.

The authors also change pitch by adding a fixed vector to the backbone’s hidden states. They compute this vector by subtracting the average states for falling-pitch examples from those for rising-pitch examples. Vectors computed from human recordings change the final pitch pattern when applied during speech generation, including for speakers excluded from the vector calculation. However, the generated pitch patterns often differ from the specific patterns the authors intended to produce. Automatic quality measures suggest a modest quality cost at the selected settings.

Critical strengths
  • The experiments help explain which inputs the depth model uses to generate the final pitch pattern. Replacing the backbone state transfers almost the full difference between statements and questions, while replacing the first audio token has little effect (Table 1). Rerunning the model without replacing either input helps rule out random sampling as the explanation. Tests using different sentences also show that the first audio token still matters for producing the words.

  • The vectors computed from human recordings change pitch without retraining the TTS models. The effect remains when the target speaker’s recordings are excluded from the vector calculation. It also exceeds the changes obtained by simply generating the same sentence again with a different random seed (Appendix F). Although adding vectors to hidden states is an established method, these results show where it can help control pitch and where it falls short.

Critical weaknesses
  • The experiments do not establish why the later procedure causes less damage to voice quality. The authors change both how they compute the vector and how they apply it. The earlier procedure uses generated recordings and changes each frame while keeping later backbone states fixed. The later procedure uses human recordings and allows the changed audio tokens to influence subsequent generation. Appendix E includes several useful checks, but does not separate these two changes. It therefore remains unclear which change produces the improvement.

  • Two implementation issues leave uncertainty about the S2 results. The authors use 21.0 rather than 21.53 frames per second to locate the frames they modify. They also report that some recordings change when generation is replayed without any edit, possibly because model state was carried over from a previous recording (Appendices D.6 and F). Comparing edited recordings with the replayed version is a useful precaution, but corrected checks are needed to confirm that these issues do not materially affect the findings.

Questions for authors
  • Could you compare the two generation procedures using the same vector computed from generated recordings? Comparing vectors from human and generated recordings within the same procedure would also help establish which change improves voice quality.

  • Please clarify which S2 results are compared with the original recordings and which are compared with the replayed recordings. Can you confirm the main results after correcting the frame rate and resetting model state between recordings?

1 comment

W1/Q1, vector source vs procedure. We ran the full 2×2 in both systems (App. I, Table 12), and also the punctuation contrast as a steering function. The procedure explains the voice-quality difference: every in-loop cell has higher HNR than every offline cell (16.1–17.6 vs 12.7–14.9 dB). The vector's source does not: with the procedure fixed, the human and punctuation vectors are within 0.8 dB. What does matter is the form: the punctuation function (time-dependent vector), fitted on the same recordings as the punctuation vector, is closer to natural target recordings in all eight cells. The submission's conjecture about why the offline edit degrades the voice is withdrawn.

W2/Q2, S2 implementation. Both issues are fixed and every S2 result recomputed, except four appendix numbers reported from the submitted set: the edge probe's accuracy (App. B.4), the no-edit baseline and the held-out speakers (App. G), and the accent vector's flip rates (App. K).

  • Frame rate: all S2 windows, onsets and vectors use 21.53 Hz (App. E.6). Table 1 is recomputed: every S2 ΔΦ\Delta_\Phi within 0.04 of the submitted value.
  • Replay: the mismatch was not model state carried between generations. One activation function in S2's codec is compiled by TorchScript, which runs it unoptimized on its first call in a process and optimized afterwards, with slightly different rounding; the first voice prompt encoded in each process therefore got slightly different codes. With TorchScript disabled, the regenerated S2 set replays exactly: 360/360 no-edit replays in fresh processes reproduce their recordings token for token, and a second, independent generation is identical in codes, backbone states and audio (180/180; App. G). So original and replayed recordings now coincide, and every recomputed S2 comparison is against recordings that replay exactly.
Reviewer_CptSOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper looks at how nuclear intonation is represented and controlled in two hierarchical TTS models, CSM and Fish Audio S2. Using interchange interventions on codec tokens and backbone hidden states, the authors conclude that the assertion-interrogation contrast sits mostly in the backbone and gets passed down through the depth model, not in the first-level semantic tokens. They then try difference-of-means steering, with vectors built from punctuation-conditioned generations and from human imitations of specified tunes. This moves the broad rise/fall of the output with moderate success and mostly keeps predicted naturalness and speaker similarity intact, but the speech usually doesn't land on the intended phonological tune.

Critical strengths
  • The causal analysis is the best part. Interchange interventions plus re-running the depth model under controlled inputs tell us a lot more about what each component does than another round of probing would.
  • Two architectures with different codec structures give the same picture, and the matched-pair and leave-one-speaker-out setups make that more convincing.
  • The controls are thorough (intervention strength, acoustic quality, speaker similarity, alternative steering directions). The implementation checks in the appendix are good and help with reproducability.
  • I also liked that the paper reports steering mostly moves edge tones instead of producing full tunes. That's a useful result for anyone working on controllable speech generation.
Critical weaknesses

W1. The evidence is narrow. The main generation experiments use three short English sentences, ten speakers and one binary contrast. I can't tell whether any of this carries over to other sentence structures, longer utterances, other prosodic positions or other intonational categories. The extra experiment on a tune with no matching punctuation cue is small too.

W2. This is my main concern. Steering sucess is measured by whether the pitch excursion on the final word crosses a preset threshold, which is not the same as producing the intended tune. According to Figure 3 and Appendix F.2, only 11% (CSM) and 1% (S2) of recordings steered toward a rise match the prototypical lhh category. Steering toward a fall gives hll/hlh in just 2% and 0%. So what the method actually does is coarse pitch-contour manipulation, with very little control over specific patterns. The authors do admit this, but it's implications for the central contribution deserve much more weight than a caveat in the text.

W3. There's no listening test. Everything rests on acoustic measurements, automatically assigned tune labels and predicted quality (UTMOS). For a paper about a linguistically meaningful attribute, I want to know wether people actually hear the intended category or meaning. Similar F0 trajectories don't guarantee that.

W4. The writing makes the paper harder than it needs to be. The grammar is fine and the detail is all there, but the intro throws in a lot of specialized terms before giving any picture of what the interventions actually do. The method section is squeezed and there's no architecture overview, so I spent a while reconstructing the setup. Figures 2 and 3 pack several acoustic measures, directions and tune categories into one plot and I honestly had trouble finding the takeaway. Most important, the paper never makes clear up front that flipping a broad rising/falling condition and producing a specific nuclear tune are different things. A cleaner story, simpler plots and a clear seperation between mechanistic findings and demonstrated control would help a lot.

W5. Two method issues. In-loop steering needs an initial generation that is force-aligned to find the intervention window, so it isn't really an inference-time control method as it stands. Also, Appendix D.6 says the frame rate used for the S2 windows doesn't match the actual codec frame rate. Everything here is about pitch near the end of the final word, so a mismatch like that could matter. Please quantify it instead of leaving it as an implementation note.

Questions for authors
  1. Given that so few steered recordings hit the intended tune, what supports the controllability claims beyond changing the direction of final-word pitch? A clearer numerical split between simple condition flips and real tune realization would help.
  2. Have you tried unseen sentence structures, different lexical content, longer utterances, or nuclear accents in other prosodic contexts? Without that I can't tell if the directions encode transferable intonation information or just features of this small controlled set.
  3. Can you add a listening study? Do listeners identify the intended category in the steered speech? Rating naturalness and perceived meaning together would strengthen the conclusions a lot.
  4. How much does the S2 frame-rate mismatch (App. D.6) change things? The word-final region is short, so could you rerun the key localization and steering experiments with the correct frame rate and check whether the results hold?
  5. What's the practical advantage over simple text-based control? For assertion vs. interrogation, punctuation already works reliably, and the method doesn't consistently produce the other target tunes. Is the main contribution meant to be a mechanistic analysis of hierarchical speech generation or a useful intonation-control method? Right now it reads like it wants to be both.
1 comment

W1/Q2, narrow evidence. New: held-out sentences for the tune vector in the loop (Table 9); the steering functions refitted and dosed on five speakers and applied to the other five (Table 11); steering between all five tune classes, not one binary contrast (Table 2). Longer utterances and other prosodic positions remain untested, as the limitations state. We explained in the general comment the reasons to work with this limited dataset.

W2/Q1, flip ≠ tune. We agree, and the revision separates the two throughout. Every steering result is now reported as the share in the target tune class (Table 2), next to the flip rate where the target is a rise or a fall (Table 12). With the vectors, the target class is reached in at most 14% of held-out-voice recordings at the chosen strength (Table 12), as you noted. Steering functions, which follow the class difference over the word instead of averaging it, reach lhh in 33% (CSM) and 44% (S2) and the rise-fall in 18% and 38% (Table 2). The advantage is concentrated in the directions toward the scooped rise and the rise-fall: by the share alone the function is ahead in 13 of 18 directions in S2 and 9 in CSM. Intervals over voices for this per-direction comparison did not fit in the discussion period; we will add them to the camera-ready version. Steering assertions to lhh and questions to the rise-fall, each from its own class, they bring 39% and 43% of assertions to lhh and 19% and 42% of questions to the rise-fall (Table 10; read with each voice at the dose chosen without it, 37% and 42%, 20% and 42%; before steering, 10% and 4% of assertions were already lhh, 2% of questions in the rise-fall). For scale: natural imitations of the scooped rise are labelled lhh 74% of the time, and the models' own questions 41% and 60% (App. C).

W3/Q3, listening study. Added (App. J, Table 13, Figure 4); see the general comment. Listeners judged statement vs question and distortion (simple judgements listeners were already familiar with). This was a variation of Experiment 1 in (Sostarics and Cole, 2023), which establishes that the likelihood of a "question" judgement increases continuously with the size of the rise (their Figure 2), among other things. The limitations emphasize this point.

W4, writing. §6 is now built around the target tune class, with flips reported separately; the submission's Figures 2 and 3 are moved out of the main text and replaced by a per-class table (Table 2) and a figure of produced against heard outcomes (Figure 4); the appendix is cut from 40 to 23 pages. The introduction is rewritten to state the argument before the technical terms, the setup now opens with a diagram of the generation pipeline, its notation and where each intervention cuts it (§3, Figure 1), and the prompting results have their own section (§4).

W5, method issues. Forced alignment: we agree. In-loop steering is used here as a causal test of what the backbone state encodes, not proposed as an inference-time control method; a deployable version would need the word boundary online. The Limitations now say so (§8). S2 frame rate (Q4): fixed and rerun at 21.53 Hz, all but four appendix numbers, which are marked (App. E.6). In Table 1, every S2 ΔΦ\Delta_\Phi moves by at most 0.04 (full swap 0.91 → 0.95; level 0, 0.09 → 0.07), and no steering conclusion changes.

Q5, mechanism or control. For the assertion/question contrast, punctuation is the better control (§4). The contribution is the mechanism: where the contrast is computed (§5), and that differences read from human speech move production into the corresponding class across voices and speakers (§6). We tried to make this very clear in the new version of the introduction. The human-fitted vector aligns with the punctuation vector far beyond a shuffled-label null (cosine 0.66 vs at most 0.31 in CSM; 0.52 vs 0.20 in S2; App. G).

Reviewer_VzrTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how intonational tunes are represented and generated in hierarchical RVQ-based TTS systems, focusing on CSM and Fish Audio S2. Through interchange interventions, the authors identify backbone hidden states as the primary carriers of intonational information, while Level-0 tokens contribute little to intonation generation. The paper further explores difference-of-means activation steering based on human imitations of different intonational tunes to manipulate generated pitch contours without model retraining. Although the proposed approach demonstrates promising control over rising and falling intonation, its ability to generate specific linguistic tune categories remains limited.

Critical strengths
  1. The paper investigates an underexplored aspect of TTS controllability, namely linguistic intonation, which is important for natural and expressive speech generation.

  2. I appreciate the use of interchange interventions to investigate the respective roles of backbone hidden states and different RVQ token levels in intonation generation.

  3. I find the use of human imitations to derive steering vectors interesting. This approach enables inference-time manipulation without requiring additional model training.

Critical weaknesses
  1. Limited methodological novelty and missing baselines. I think the proposed steering method primarily relies on conventional difference-of-means activation steering. Although applying this technique to intonation control is interesting, the paper does not sufficiently demonstrate its advantages over existing prosody-control approaches. I would like to see comparisons with representative reference-based, prompt-based, and explicit prosody-control methods to better establish its effectiveness and practical value.

  2. Limited effectiveness in controlling linguistic intonation. I find that the proposed method mainly changes the overall rising or falling tendency of the final pitch contour, rather than reliably generating the intended nuclear tune. For example, the steered outputs frequently fall into high-plateau or mixed-edge categories instead of the target linguistic categories. This raises questions about whether the proposed method achieves genuine linguistic intonation control or merely modifies local pitch patterns.

  3. Insufficient perceptual evaluation. I think the evaluation relies too heavily on acoustic measurements and automatic predictors, without any human listening experiments. It remains unclear whether the generated speech actually conveys the intended intonation, whether the modifications sound natural, and whether listeners can reliably distinguish the target tune categories. Subjective evaluations would provide more convincing evidence of the effectiveness of the proposed method.

  4. Limited experimental scope and generalizability. I am concerned that the experiments are restricted to American English, three carrier sentences, and ten speakers. Although leave-one-speaker-out experiments are included, I do not think the current evidence is sufficient to establish generalization across diverse linguistic contexts, sentence structures, and languages. Broader evaluations are needed to support the paper's claims.

Questions for authors

Suggestions:

  1. I find the paper difficult to follow, particularly the Abstract and Introduction. The authors introduce too many technical terms without sufficient explanation, making it difficult for readers to understand the motivation, methodology, and main contributions. I suggest reorganizing these sections and providing clearer explanations of the key concepts and overall research pipeline.
  2. I suggest evaluating the effects of different steering strengths, intervention locations, and tune-pair selections to better understand the controllability and robustness of the proposed method.
3 comments

W1, novelty and baselines. Our claim is about what the model represents, not a control method competing with F0-conditioned TTS: states the model forms while reading human speech, when added during generation, move its production into the corresponding tune class (§6), for voices and speakers they were not fitted on (Tables 9, 11). The relevant baselines are therefore the prompt-side routes the model already offers, which now have their own section (§4, App. D): none of 18 style tags moves the tune (Figure 6, Table 6); a recorded demonstration of the tune as voice prompt transfers a little over half of the punctuation effect in both systems; lexical lead-ins have a limited effect on the pitch accent only present in CSM. Only the final punctuation mark works reliably. We did not run an explicit F0-conditioned baseline, for three reasons: (1) our question is what the model's own states encode; (2) we want to approach intonational tunes as phonological abstractions, not as arbitrary pitch trajectories (there is empirical evidence that human subjects are not able to reproduce arbitrary variations in pitch trajectory heard in auditory stimuli); and (3) the actual pitch trajectory depends on each speaker and sentence, complicating the prescription of pitch targets.

W2, rise/fall vs linguistic tunes. This is now the main analysis of §6. Read by tune class (Table 2): the vector reaches the level classes (the low fall lll, the high rise hhl/hhh) about as well as the function, but the function reaches the classes defined by contour shape: lhh 0.33 vs 0.13 (CSM) and 0.44 vs 0.24 (S2); the rise-fall 0.18 vs 0.10 and 0.38 vs 0.07. The high-plateau outcomes you noticed (the class now named, after Cole et al. 2023, the high rise hhl/hhh) are overshoot: at strength 2.5 the vector puts 79% (CSM) and 76% (S2) of recordings in that class, but at distance 2.34 and 1.74 from its centroid (in the Euclidean space of normalized trajectories), against 0.63 and 0.55 for natural ones. The functions still miss most of the time; the shortcomings of the method might be due to imprecise representation of intonation in these systems, which has not been explicitly trained to produce pragmatically meaningful intonation.

W3, listening test. Added (App. J, Table 13, Figure 4); please see the general comment.

W4, scope. We explained in the general comment our reasons to work with this relatively limited dataset. The choice to work with 10 speakers was partly imposed by our computational constraints, but as we said in the paper, we chose them to represent a variety of mean F0. To justify generalization, we computed our metrics with held-out sentences in the loop (Table 9) and held-out speakers for the steering functions (Table 11). The steering now covers all 18 directed pairs of five tune classes (Table 2, Figure 13). The material is still English, three sentences and ten voices; the limitations say so.

Q1, clarity. The introduction is rewritten around the paper's argument (where the contrast is computed; why that alone does not show that the backbone encodes intonation; what steering from human speech adds). The limits of prompting now have their own section (§4), the setup opens with a plain description of how the generators work and a diagram of the pipeline with its notation (§3, Figure 1), and the appendix is cut and reorganized.

Q2, strengths, locations, tune pairs. Strengths: every method's dose is chosen by one criterion (§6), and the measures it weighs are shown over the full strength range for the tune vector in the loop (Figure 12) and the punctuation vector offline (Figure 11). Locations: the window-dose and positional swaps (App. E.2, Figures 9–10). Tune pairs: all 18 directed pairs (Figure 13).

W1, novelty and baselines. Our claim is about what the model represents, not a control method competing with F0-conditioned TTS: states the model forms while reading human speech, when added during generation, move its production into the corresponding tune class (§6), for voices and speakers they were not fitted on (Tables 9, 11). The relevant baselines are therefore the prompt-side routes the model already offers, which now have their own section (§4, App. D): none of 18 style tags moves the tune (Figure 6, Table 6); a recorded demonstration of the tune as voice prompt transfers a little over half of the punctuation effect in both systems; lexical lead-ins have a limited effect on the pitch accent only present in CSM. Only the final punctuation mark works reliably. We did not run an explicit F0-conditioned baseline, for three reasons: (1) our question is what the model's own states encode; (2) we want to approach intonational tunes as phonological abstractions, not as arbitrary pitch trajectories (there is empirical evidence that human subjects are not able to reproduce arbitrary variations in pitch trajectory heard in auditory stimuli); and (3) the actual pitch trajectory depends on each speaker and sentence, complicating the prescription of pitch targets.

W2, rise/fall vs linguistic tunes. This is now the main analysis of §6. Read by tune class (Table 2): the vector reaches the level classes (the low fall lll, the high rise hhl/hhh) about as well as the function, but the function reaches the classes defined by contour shape: lhh 0.33 vs 0.13 (CSM) and 0.44 vs 0.24 (S2); the rise-fall 0.18 vs 0.10 and 0.38 vs 0.07. The high-plateau outcomes you noticed (the class now named, after Cole et al. 2023, the high rise hhl/hhh) are overshoot: at strength 2.5 the vector puts 79% (CSM) and 76% (S2) of recordings in that class, but at distance 2.34 and 1.74 from its centroid (in the Euclidean space of normalized trajectories), against 0.63 and 0.55 for natural ones. The functions still miss most of the time; the shortcomings of the method might be due to imprecise representation of intonation in these systems, which has not been explicitly trained to produce pragmatically meaningful intonation.

W3, listening test. Added (App. J, Table 13, Figure 4); please see the general comment.

W4, scope. We explained in the general comment our reasons to work with this relatively limited dataset. The choice to work with 10 speakers was partly imposed by our computational constraints, but as we said in the paper, we chose them to represent a variety of mean F0. To justify generalization, we computed our metrics with held-out sentences in the loop (Table 9) and held-out speakers for the steering functions (Table 11). The steering now covers all 18 directed pairs of five tune classes (Table 2, Figure 13). The material is still English, three sentences and ten voices; the limitations say so.

Q1, clarity. The introduction is rewritten around the paper's argument (where the contrast is computed; why that alone does not show that the backbone encodes intonation; what steering from human speech adds). The limits of prompting now have their own section (§4), the setup opens with a plain description of how the generators work and a diagram of the pipeline with its notation (§3, Figure 1), and the appendix is cut and reorganized.

Q2, strengths, locations, tune pairs. Strengths: every method's dose is chosen by one criterion (§6), and the measures it weighs are shown over the full strength range for the tune vector in the loop (Figure 12) and the punctuation vector offline (Figure 11). Locations: the window-dose and positional swaps (App. E.2, Figures 9–10). Tune pairs: all 18 directed pairs (Figure 13).

Thank the authors for the detailed rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the improved presentation and expanded evaluations.

However, my main concerns remain. The technical contribution is still limited, and the additional results further confirm that the proposed steering method struggles to reliably control specific linguistic tune categories. Moreover, the limited experimental scope restricts the generalizability of the findings.

While I acknowledge the improvements, I do not think they sufficiently address my primary concerns. Therefore, I will maintain my original score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to the reviews

We appreciate the reviewers' detailed comments. In addressing them, we made substantial changes that we believe clarify our goals and strengthen our contribution: the revised paper clarifies its aim and scope, and introduces new analytic tools and results that directly address the issues raised.

Aim of the paper: We have clarified that the aim is mechanistic interpretation: what are the mechanisms that compute intonation inside staged hierarchical generators? Answers to this question are critical to inform future work on prosody control, when the goal is neural systems capable of the rich and nuanced prosodic expression that conveys meaning in English dialogue and narration. The introduction was rewritten accordingly.

Scope of the analysis: We seek approaches to speech generation that can produce the full range of prosodic expression documented in human speakers. Punctuation (period vs. question mark) captures only a subset of it. Two findings from intonation research are relevant. First, the pitch pattern over the final region of the sentence (typically the final stressed word) conveys critical information about dialogue acts and informational focus, while pitch over preceding material conveys style, but not logical or situational meaning. Second, there are roughly 8 (or slightly fewer) acoustically distinct pitch patterns associated with distinct meanings, most of which punctuation does not distinguish. Steering based on punctuation is therefore inherently limited for the broader goal of full prosodic expression.

Number of speakers. We demonstrate steering with human intonation using an admittedly small corpus of human productions and paired perceptual ratings (Cole et al., 2023). It is the only open-access corpus to date that establishes the robust pitch-pattern distinctions of the (American) English intonation system, and the only empirically validated labeled intonation dataset that generalizes across speakers and listeners. Public speech databases are large, but none has intonational annotation validated beyond the fragile criterion of inter-annotator agreement. Cole's method scales up, so applications with a larger corpus are within reach.

Types of sentence materials. The corpus is also limited in sentence structure: the critical region for pitch analysis is the final word, a design that was key to Cole's study but is not necessary. We are developing a pipeline using the Wavelet Toolkit (Suni et al., 2017) to identify prosodic phrases and, within them, the critical region, which will allow analysis of sentences of varying length and structure.

New methods and results.

  1. Listening test (App. J, Table 13, Figure 4). 35 Prolific listeners heard 60 trials each, over 20 conditions: per model, the unsteered A and I recordings and 4 in-loop steering methods in both directions (A→I toward a rise lhh, I→A toward a fall hll/hlh). Listeners hear the effect: with the new steering functions (time-dependent) fitted on human imitations, the share heard as a question rises by 45 to 53 percentage points for steered assertions and falls by 48 to 56 for steered questions. Reported distortion does not rise significantly for any method except S2's tune vector toward an assertion (p=0.03p = 0.03). Across conditions, the acoustic share of rising recordings predicts the share heard as a question (r=0.94r = 0.94).

  2. Tune categories, not only rise vs fall (§6, Table 2, App. H). We now steer between every pair of the five tune classes listeners distinguish (18 directed pairs) and report the share reaching the target class, alongside the flip rate for rises and falls. A word-averaged steering vector mostly sets the word's pitch level; a steering function, which follows the class difference over the word, reaches the scooped rise lhh and the rise-fall hll/hlh 1.8 to 5.5 times as often (Table 2). For calibration, the labeller assigns a natural human rise to lhh 74% of the time, and the models' own questions 41% (CSM) and 60% (S2) (App. C).

  3. Vector × procedure, fully crossed (App. I, Table 12). Both vectors and the punctuation function, run offline and in the loop, in both systems.

  4. Held-out sentences and speakers (App. G, Table 9; App. H, Table 11). The tune vector refitted without a sentence, its strength chosen without it, behaves on that sentence as in sample. The steering functions, refitted and dosed on five speakers, steer the other five.

  5. S2 fixed and rerun (App. E.6). Every S2 window now uses the codec's true 21.53 Hz, and the replay mismatch, from TorchScript in S2's codec, is removed at the root (see the response to c9AE). Every S2 result, including Table 1, was recomputed, except four appendix numbers marked as from the submitted set; no conclusion changes.

The appendix was also shortened and reorganized, as CptS and VzrT asked for a clearer paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.