- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
ProbeCRC steers speech attributes in a frozen autoregressive TTS model using layer-wise linear probes and regularized hidden-state interventions. Conformal Risk Control selects a shared steering strength for an external-classifier-defined failure loss. On the evaluated Fish-Speech model, the authors report bidirectional emotion, gender, and age changes with better quality than the tested steering baselines at the reported operating points. For emotion and gender, mean held-out classifier failure remains below the requested risk levels across 50 speaker splits. Listening judgments also support perceptible attribute changes. Age is evaluated perceptually, and the risk guarantee concerns classifier-defined expected failure rather than perceptual or quality guarantees.
Critical strengths
-
Reliable strength selection is a meaningful TTS problem: weak steering may fail, while excessive steering can damage speech. Section 4.4 turns this trade-off into an explicit risk-selection task. Probes and CRC are established tools, but their combination for speech steering is useful; Appendix A.2 makes the calibration more credible by handling non-monotonic losses.
-
Sections 5–6 combine automatic metrics, repeated speaker splits, and listening judgments across three attributes, with recent controllable-TTS baselines such as IndexTTS2 and VoxCPM2. This is a reasonable evaluation of control on the chosen model. The evidence remains limited to one TTS model, and CRC validation covers emotion and gender rather than a perceptual guarantee for every attribute.
Critical weaknesses
-
The CRC theorem and monotone envelope are sound under their stated conditions, but Equations (3) and (11) are not clearly connected to Appendix B.5's speaker-level protocol. The calibration unit, within-speaker aggregation, and infeasible-grid handling need specification before the implemented guarantee can be checked.
-
Section 4.1 identifies Slow Transformer intervention during semantic-token generation, but Sections 4.2–4.3 leave the probe-training activation source, token selection and aggregation, and precise hook unspecified. Equation (10) gives a local probe-logit target, not an exact end-to-end change. These details are needed to reproduce and interpret the implemented steering.
Questions for authors
-
What is the exchangeable calibration unit, and how are within-speaker losses and infeasible candidates handled? Figure 1 selects the largest feasible lambda, whereas Equation (4) selects the smallest; which rule is implemented?
-
From which tokens and inference stage are probe activations extracted, how are they aggregated, and where are interventions applied? Is the shared-logit target intended only for each local injection?
-
Does the quality advantage persist at matched external-classifier failure rates? Please distinguish that controlled loss from perceptual attribute failure and speech-quality preservation.
5 comments
Response to Reviewer 8Cwm Part 2/4
W2 + Q2: Probe activation extraction, intervention placement, and interpretation of the shared-logit target
Reply:
Probe activation extraction and aggregation. Activations are extracted from the residual stream at the output of each layer of the Slow Transformer, before the final RMSNorm, during autoregressive semantic-token generation. Specifically, we collect hidden states from decoding steps, excluding the initial prefill stage and the first generated token. For each utterance, the collected token-level activations are mean-pooled over time, producing a single representation per layer, which is used to train the corresponding linear probe.
We use temporal mean pooling because the attributes considered in our study, such as emotion, gender, and age, are utterance-level characteristics that are generally expressed throughout the generated speech rather than being associated with individual tokens.
Intervention placement. During inference, the computed steering vector is added to the residual stream after each Slow Transformer block, across all layers. Interventions are applied during both prefill and autoregressive decoding, using a fixed layer-specific steering vector at every token position. The Fast Transformer is not directly modified.
Shared-logit target. The shared parameter λ specifies a local target change in probe-logit space for each layer, rather than a guaranteed cumulative change across the network. Each layer's intervention is computed independently using its corresponding probe, according to the regularized formulation in Eq. (10).
Importantly, λ represents the desired local logit shift, rather than an exact change in the probe output. Regularization and interactions between successive layers may affect the resulting activations.
Our goal is to develop a general steering framework that can be applied across different models, where the effects of interventions may vary not only between architectures but also across layers and attributes within the same model.
Rather than assuming a fixed relationship between λ and the resulting output, we use CRC to calibrate a single shared steering-strength parameter based on observed output-level steering failures. This allows the steering strength to be selected according to a specified risk threshold, without requiring the intervention effects to be identical across layers or models.
We have clarified activation extraction, aggregation, and intervention placement in Appendix B.1, and the local interpretation of the shared-logit target in Section 4.3 of the revised manuscript.
Q3: Speech quality at matched classifier-defined failure rates and distinction between evaluation criteria
Reply:
Identical CRC protocol. We first applied the same speaker-level CRC procedure to probe-guided steering and both difference-of-means (DoM) baselines, using the same 50 speaker-disjoint calibration/test splits. Probe-guided steering and single-layer DoM are evaluated on a common candidate-strength grid, while multi-layer DoM uses its method-specific grid, extended to |λ| ≤ 0.5 (steps of 0.02 up to 0.1, followed by steps of 0.1).
At α = 0.05, probe-guided steering achieves higher held-out WV-MOS across all feasible comparisons and is the only method with a feasible operating point for all four steering targets. Restricting probe-guided steering to the single-layer DoM grid changes its WV-MOS by at most 0.03 relative to the denser grid used in the main experiments, without changing the relative ranking of the methods.
However, since identical target risks do not necessarily imply identical empirical failure rates, we conducted two additional comparisons to directly assess speech quality at comparable levels of steering success.
Response to Reviewer 8Cwm Part 3/4
Matched empirical failure rates. To compare speech quality at comparable levels of steering effectiveness, independently of CRC-based strength selection, we evaluated classifier-defined failure rates and WV-MOS on identical utterances, using the same candidate-strength grids as above. For each method, we interpolated WV-MOS at the first crossing of the target empirical failure rate of 5%. Extending the multi-layer DoM grid does not change the matched-failure results, since the first crossing occurs within the original strength range.
The table below reports the difference in WV-MOS between probe-guided steering and each baseline at this matched failure rate. Positive values indicate higher speech quality for probe-guided steering, while negative values favor the baseline. Values in brackets denote 95% paired speaker-level bootstrap confidence intervals.
| Target | vs. Single-layer DoM | vs. Multi-layer DoM |
|---|---|---|
| Happy | −0.18 [−0.24, 0.01]* | +0.35 [−0.05, 0.59]* |
| Sad | +0.19 [0.08, 0.28] | +0.35 [0.20, 0.49] |
| Female | +0.21 [0.16, 0.27] | +0.23 [0.18, 0.28] |
| Male | +0.11 [0.06, 0.22] | 0.00 [−0.05, 0.10] |
*For happy steering, the baselines do not reach a 5% failure rate in all bootstrap samples.
At matched classifier-defined failure rates, probe-guided steering achieves significantly higher WV-MOS than both baselines for sad and female steering, and than single-layer DoM for male steering. For male steering, no statistically significant difference in WV-MOS is observed relative to multi-layer DoM.
For happy steering, the comparison is less conclusive: single-layer steering attains higher WV-MOS and multi-layer steering lower WV-MOS, with both confidence intervals including zero. UTMOSv2 consistently favors probe-guided steering, with statistically significant improvements in most comparisons, further supporting its ability to preserve speech quality.
Quality conditional on steering success. To assess whether the quality advantage persists among successfully steered outputs, we restricted the evaluation to held-out utterances classified as successfully steered at each method's CRC-selected strength, using the same candidate-strength grids as above.
At α = 0.05, the success-conditioned WV-MOS results are:
| Target | Probe-guided | Single-layer DoM | Multi-layer DoM |
|---|---|---|---|
| Happy | 3.57 | Infeasible | Infeasible |
| Sad | 4.33 | 4.03 | 3.80 |
| Female | 4.81 | 4.49 | Infeasible |
| Male | 4.60 | 4.27 | 4.20 |
Since the successfully steered subsets may differ across methods, this analysis complements rather than replaces the matched-failure comparison.
Probe-guided steering achieves higher WV-MOS across all feasible comparisons and is the only method with a feasible operating point for happy steering. These results show that the quality advantage persists when conditioning on classifier-defined steering success, rather than only in the overall averages.
Response to Reviewer 8Cwm Part 1/4
We thank the reviewer for the constructive feedback and the opportunity to clarify the theoretical guarantee, implementation details, and evaluation of ProbeCRC. We have revised the paper and conducted additional analyses to address the concerns raised below.
W1 + Q1: Calibration unit, exchangeability, infeasible candidates, and strength-selection rule
Reply:
Calibration unit and exchangeability. In the original submission, each calibration loss corresponded to an individual utterance (a speaker–sentence pair, with 10 utterances per speaker), while calibration and test sets were split at the speaker level. We agree that speaker-disjoint splitting alone does not establish exchangeability at the utterance level, since utterances from the same speaker may be dependent.
Following your comment, we have revised our CRC calibration procedure to use speakers, rather than individual utterances, as the calibration units. For each calibration speaker i, the loss in Eq. (11) is the steering-failure rate over that speaker's utterances:
Li(λ)=mi1∑j=1miℓij(λ),
where ℓᵢⱼ(λ) is the binary classifier-defined steering failure for utterance j of speaker i, and mᵢ is the number of utterances for speaker i (mᵢ = 10 for all speakers in our experiments).
We apply the monotone upper envelope (Eq. 21) to this speaker-level loss curve and run CRC with n in Eq. (3) equal to the number of calibration speakers (100 for emotion and 99 for gender, following our 80/20 speaker-disjoint calibration/test split).
Under the assumption that speakers are exchangeable and that the same loss construction is applied to calibration and test speakers, the speaker-level loss functions satisfy the exchangeability requirement of CRC without requiring independence between utterances of the same speaker. The guarantee therefore concerns the expected classifier-defined failure rate of a new speaker, subject to the remaining assumptions of Theorem 1.
All CRC results in the revised paper (Fig. 5, Fig. 8, Tables 1 and 3, and the new ablations) use this speaker-level procedure.
The empirical results are robust to this change. Across the same 50 splits, four steering directions, and seven target risk levels, the speaker-level and utterance-level procedures select the same most frequent steering strength in 13 of the 28 settings, with differences of at most 0.2 in the remaining settings (one or two grid steps). These differences reflect the combined effects of applying the monotonicity correction to speaker-averaged losses and using a larger finite-sample correction due to the smaller number of calibration units.
Importantly, these differences do not substantially affect steering effectiveness or speech quality. In the evaluated feasible settings, the mean held-out risk remains below α, and WV-MOS is essentially unchanged. For example, at α = 0.1, the held-out risk for happy steering is 0.062 (speaker-level) versus 0.050 (utterance-level), and for female steering it is 0.078 for both procedures.
Infeasible candidates. If no candidate steering strength satisfies the CRC threshold, the run is flagged as infeasible and no risk-control guarantee is claimed. Infeasible runs are reported separately and excluded from the risk estimates of certified operating points.
For ProbeCRC, all 2,800 speaker-level calibration runs (4 directions × 50 splits × 14 target risks, α = 0.05, 0.10, …, 0.70) were feasible. Within this range, infeasibility was observed only for the difference-of-means baselines. Specifically, for happy steering, neither the single-layer nor the multi-layer difference-of-means baseline yielded a feasible candidate in any of the 50 splits for any evaluated α ≤ 0.3 under the same calibration protocol.
Feasibility also depends on calibration size: with few calibration speakers, the finite-sample correction term 1/(n+1) alone can exceed stringent target risks. For example, with 25 calibration speakers, the correction is approximately 0.038, making certification at α = 0.025 impossible under the adopted procedure. These results illustrate why feasibility must be explicitly checked rather than assumed.
In the revised paper, we have revised the calibration procedure and clarified the exchangeability assumption, the scope of the risk guarantee, and the handling of infeasible settings.
Selection rule. The implementation follows Eq. (4) and selects the smallest feasible strength:
λ^α=argminλ∈Λ∣λ∣
subject to
n+11∑i=1nLi(λ)+n+11≤α.
Increasing |λ| corresponds to stronger steering toward the target. CRC selects the smallest |λ| satisfying the risk constraint.
The reference to the largest feasible value in Figure 1 is a labeling error, which we have corrected.