- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates the transmission and control of intonation within CSM and Fish Audio S2 by swapping audio codes, replacing the inputs to the downstream generation module, and editing backbone states. For the main statement–question contrast, replacing the backbone state transfers most of the measured final-word pitch difference, whereas replacing the first-level audio codes has a smaller effect. Editing directions extracted from human speech also change the output contours, but do not reliably produce the specified intonation. The experiments use three sentences and ten speakers, with acoustic, speech-classification, and audio-quality metrics, but no listening evaluation.
Critical strengths
The paired input-replacement experiments directly compare the effects of backbone states and first-level audio codes on generated pitch. Re-running the downstream generation module with its original inputs also provides a control for variation caused by random sampling. The consistent pattern across the two systems provides more direct empirical evidence about the information-flow pathways under the tested conditions than representation classification alone.
Critical weaknesses
-
The explanatory contribution is limited, and the scope of the conclusions is insufficiently supported. Input replacement and directional activation editing have established precedents. The paper’s main new insight concerns the roles of different inputs in two models for a specific intonation contrast, without distinguishing whether this division arises from the codec, model architecture, or training. A small intervention effect from first-level audio codes also does not directly establish that they “carry little to no intonation.”
-
The main generation experiments cover only ten speakers and three similar short sentences. Repeated generation and multiple editing strengths do not broaden the textual coverage. The existing held-out-speaker evaluations and held-out-sentence tests for the offline punctuation-derived direction do not directly establish that the human-derived direction generalizes across sentences when applied during generation. The evidence therefore supports findings within a restricted setting, but is insufficient to establish a general mechanism of intonation or a reusable control principle.
-
The evidence for the benefits of control and the claimed audio-quality advantage is insufficient. The paper provides no listening evaluation of generated speech, and automatic pitch metrics and template labels do not establish that listeners perceive the intended intonation. It also does not sufficiently explain its relationship to existing F0-conditioned control methods or demonstrate additional benefits through a comparable baseline, such as direct F0 editing.
-
Offline editing uses a direction derived from punctuation conditions, whereas editing during generation uses a direction derived from human recordings. Both the editing procedure and the source of the direction therefore change, so the lower audio-quality cost cannot be attributed solely to applying edits during generation. Although the authors acknowledge that reliable selection of specific intonation patterns remains unsuccessful, the perceptual value and audio-quality advantage of the remaining coarse-grained control still require stronger evidence.
Questions for authors
-
Does “first-level audio codes carry little to no intonation” refer to a small effect under the tested interventions, or to a limited amount of intonational information? What evidence distinguishes these interpretations and separates the contributions of the codec, model architecture, and training?
-
Beyond the existing held-out-speaker evaluations, has the human-derived editing direction been tested on sentences used neither to extract the direction nor to select the editing strength? If not, which conclusions should be restricted to the three carrier sentences used here?
-
How does the proposed method relate to existing F0-conditioned control methods? With comparable target information and magnitudes of pitch change, what evidence supports additional control or audio-quality benefits over direct F0 editing?
-
Have offline editing and editing during generation been compared using the same editing direction and similar magnitudes of pitch change? If not, how can a change in the direction’s source be ruled out as an explanation for the audio-quality difference, or should the corresponding attribution be qualified?
-
Could the authors provide a small set of paired unedited and edited audio examples, including both successes and failures? What evidence currently supports that listeners perceive the intended intonation and the audio-quality changes indicated by the automatic metrics?
W1/Q1, "level 0 carries little to no intonation." We agree that this is a statement about these interventions, and the paper now says so (§7). Our evidence is interventional: swapping level 0 between a matched assertion and question moves the tune by ΔΦ = 0.00 (CSM) and 0.07 (S2), and re-running the depth model with the other utterance's level 0 by 0.01 and 0.07, while the other utterance's backbone state moves it 0.98 and 0.95 (Table 1). The cross-sentence control shows that level 0 does carry the words (App. E.4, Table 8). §5 attributes the division to the trained models, not to the factorization. Our methods cannot currently separate the effect of the codec, the model architecture and the training, and §7 now states this.
W2/Q2, held-out sentences for the human vector in the loop. Added, exactly as asked (App. G, Table 9): for each sentence, the tune vector is refitted without it, the strength is chosen by the criterion on the other two sentences, and the held-out sentence is read in the loop. Held out it flips 20–57% (CSM) and 27–40% (S2) of recordings, against 30–47% and 23–40% in sample. The steering functions were also held out by speaker (Table 11). The conclusions remain restricted to three carrier sentences, as the limitations state.
W3/Q5, listening evaluation Listening test added (App. J, Table 13, Figure 4; see the general comment).
W3/Q3, F0-editing baseline. We do not include it, for three reasons: (1) our question is what the model's own states encode; (2) we want to approach intonational tunes as phonological abstractions, not as arbitrary pitch trajectories (there is empirical evidence that human subjects are not able to reproduce arbitrary variations in pitch trajectory heard in auditory stimuli); and (3) the actual pitch trajectory depends on each speaker and sentence, complicating the prescription of pitch targets.
W4/Q4, same direction under both procedures. Yes: both vectors are now run under both procedures (App. I, Table 12). The voice-quality difference follows the procedure, not the direction's source (see our response to c9AE W1). The attribution in the paper is changed accordingly.