acceptodds
Under review as a conference paper at ICLR 2027

ProbeCRC: Reliable Probe-Guided Speech Attribute Steering with Conformal Risk Control

Abstract

Modern text-to-speech models generate highly natural speech, yet controlling specific speaker attributes typically requires dedicated model adaptation and training. Existing inference-time alternatives can substantially degrade speech quality and provide no guarantees on the resulting attribute change. We introduce ProbeCRC, a probe-guided activation-steering framework with risk-controlled strength selection for pretrained autoregressive text-to-speech models. Layer-wise linear probes identify attribute-related directions, which we use to construct regularized hidden-state interventions. A shared steering parameter, defined in probe-logit space, translates the desired steering strength into layer-specific perturbations. We calibrate this parameter on held-out data using Conformal Risk Control (CRC), enabling attribute manipulation under a user-specified bound on failure risk. Experiments demonstrate effective bidirectional shifts in gender, emotion, and age. ProbeCRC outperforms baseline steering approaches and satisfies the prescribed risk guarantees while minimally affecting speech quality. Demo samples are available online.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
speech generation, text-to-speech, activation steering, controllable speech synthesis, linear probes, conformal risk control

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8CwmOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ProbeCRC steers speech attributes in a frozen autoregressive TTS model using layer-wise linear probes and regularized hidden-state interventions. Conformal Risk Control selects a shared steering strength for an external-classifier-defined failure loss. On the evaluated Fish-Speech model, the authors report bidirectional emotion, gender, and age changes with better quality than the tested steering baselines at the reported operating points. For emotion and gender, mean held-out classifier failure remains below the requested risk levels across 50 speaker splits. Listening judgments also support perceptible attribute changes. Age is evaluated perceptually, and the risk guarantee concerns classifier-defined expected failure rather than perceptual or quality guarantees.

Critical strengths
  • Reliable strength selection is a meaningful TTS problem: weak steering may fail, while excessive steering can damage speech. Section 4.4 turns this trade-off into an explicit risk-selection task. Probes and CRC are established tools, but their combination for speech steering is useful; Appendix A.2 makes the calibration more credible by handling non-monotonic losses.

  • Sections 5–6 combine automatic metrics, repeated speaker splits, and listening judgments across three attributes, with recent controllable-TTS baselines such as IndexTTS2 and VoxCPM2. This is a reasonable evaluation of control on the chosen model. The evidence remains limited to one TTS model, and CRC validation covers emotion and gender rather than a perceptual guarantee for every attribute.

Critical weaknesses
  • The CRC theorem and monotone envelope are sound under their stated conditions, but Equations (3) and (11) are not clearly connected to Appendix B.5's speaker-level protocol. The calibration unit, within-speaker aggregation, and infeasible-grid handling need specification before the implemented guarantee can be checked.

  • Section 4.1 identifies Slow Transformer intervention during semantic-token generation, but Sections 4.2–4.3 leave the probe-training activation source, token selection and aggregation, and precise hook unspecified. Equation (10) gives a local probe-logit target, not an exact end-to-end change. These details are needed to reproduce and interpret the implemented steering.

Questions for authors
  • What is the exchangeable calibration unit, and how are within-speaker losses and infeasible candidates handled? Figure 1 selects the largest feasible lambda, whereas Equation (4) selects the smallest; which rule is implemented?

  • From which tokens and inference stage are probe activations extracted, how are they aggregated, and where are interventions applied? Is the shared-logit target intended only for each local injection?

  • Does the quality advantage persist at matched external-classifier failure rates? Please distinguish that controlled loss from perceptual attribute failure and speech-quality preservation.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 8Cwm Part 1/4

We thank the reviewer for the constructive feedback and the opportunity to clarify the theoretical guarantee, implementation details, and evaluation of ProbeCRC. We have revised the paper and conducted additional analyses to address the concerns raised below.

W1 + Q1: Calibration unit, exchangeability, infeasible candidates, and strength-selection rule

Reply:

Calibration unit and exchangeability. In the original submission, each calibration loss corresponded to an individual utterance (a speaker–sentence pair, with 10 utterances per speaker), while calibration and test sets were split at the speaker level. We agree that speaker-disjoint splitting alone does not establish exchangeability at the utterance level, since utterances from the same speaker may be dependent.

Following your comment, we have revised our CRC calibration procedure to use speakers, rather than individual utterances, as the calibration units. For each calibration speaker i, the loss in Eq. (11) is the steering-failure rate over that speaker's utterances:

Li(λ)=1mi∑j=1miℓij(λ),L_i(\lambda)=\frac{1}{m_i}\sum_{j=1}^{m_i}\ell_{ij}(\lambda),

where ℓᵢⱼ(λ) is the binary classifier-defined steering failure for utterance j of speaker i, and mᵢ is the number of utterances for speaker i (mᵢ = 10 for all speakers in our experiments).

We apply the monotone upper envelope (Eq. 21) to this speaker-level loss curve and run CRC with n in Eq. (3) equal to the number of calibration speakers (100 for emotion and 99 for gender, following our 80/20 speaker-disjoint calibration/test split).

Under the assumption that speakers are exchangeable and that the same loss construction is applied to calibration and test speakers, the speaker-level loss functions satisfy the exchangeability requirement of CRC without requiring independence between utterances of the same speaker. The guarantee therefore concerns the expected classifier-defined failure rate of a new speaker, subject to the remaining assumptions of Theorem 1.

All CRC results in the revised paper (Fig. 5, Fig. 8, Tables 1 and 3, and the new ablations) use this speaker-level procedure.

The empirical results are robust to this change. Across the same 50 splits, four steering directions, and seven target risk levels, the speaker-level and utterance-level procedures select the same most frequent steering strength in 13 of the 28 settings, with differences of at most 0.2 in the remaining settings (one or two grid steps). These differences reflect the combined effects of applying the monotonicity correction to speaker-averaged losses and using a larger finite-sample correction due to the smaller number of calibration units.

Importantly, these differences do not substantially affect steering effectiveness or speech quality. In the evaluated feasible settings, the mean held-out risk remains below α, and WV-MOS is essentially unchanged. For example, at α = 0.1, the held-out risk for happy steering is 0.062 (speaker-level) versus 0.050 (utterance-level), and for female steering it is 0.078 for both procedures.

Infeasible candidates. If no candidate steering strength satisfies the CRC threshold, the run is flagged as infeasible and no risk-control guarantee is claimed. Infeasible runs are reported separately and excluded from the risk estimates of certified operating points.

For ProbeCRC, all 2,800 speaker-level calibration runs (4 directions × 50 splits × 14 target risks, α = 0.05, 0.10, …, 0.70) were feasible. Within this range, infeasibility was observed only for the difference-of-means baselines. Specifically, for happy steering, neither the single-layer nor the multi-layer difference-of-means baseline yielded a feasible candidate in any of the 50 splits for any evaluated α ≤ 0.3 under the same calibration protocol.

Feasibility also depends on calibration size: with few calibration speakers, the finite-sample correction term 1/(n+1) alone can exceed stringent target risks. For example, with 25 calibration speakers, the correction is approximately 0.038, making certification at α = 0.025 impossible under the adopted procedure. These results illustrate why feasibility must be explicitly checked rather than assumed.

In the revised paper, we have revised the calibration procedure and clarified the exchangeability assumption, the scope of the risk guarantee, and the handling of infeasible settings.

Selection rule. The implementation follows Eq. (4) and selects the smallest feasible strength:

λ^α=arg⁡min⁡λ∈Λ∣λ∣\hat{\lambda}_{\alpha}=\arg\min_{\lambda\in\Lambda}|\lambda|

subject to

1n+1∑i=1nL~i(λ)+1n+1≤α.\frac{1}{n+1}\sum_{i=1}^{n}\widetilde{L}_i(\lambda)+\frac{1}{n+1}\leq\alpha.

Increasing |λ| corresponds to stronger steering toward the target. CRC selects the smallest |λ| satisfying the risk constraint.

The reference to the largest feasible value in Figure 1 is a labeling error, which we have corrected.

AuthorsOpenReview ↗

Response to Reviewer 8Cwm Part 2/4

W2 + Q2: Probe activation extraction, intervention placement, and interpretation of the shared-logit target

Reply:

Probe activation extraction and aggregation. Activations are extracted from the residual stream at the output of each layer of the Slow Transformer, before the final RMSNorm, during autoregressive semantic-token generation. Specifically, we collect hidden states from decoding steps, excluding the initial prefill stage and the first generated token. For each utterance, the collected token-level activations are mean-pooled over time, producing a single representation per layer, which is used to train the corresponding linear probe.

We use temporal mean pooling because the attributes considered in our study, such as emotion, gender, and age, are utterance-level characteristics that are generally expressed throughout the generated speech rather than being associated with individual tokens.

Intervention placement. During inference, the computed steering vector is added to the residual stream after each Slow Transformer block, across all layers. Interventions are applied during both prefill and autoregressive decoding, using a fixed layer-specific steering vector at every token position. The Fast Transformer is not directly modified.

Shared-logit target. The shared parameter λ specifies a local target change in probe-logit space for each layer, rather than a guaranteed cumulative change across the network. Each layer's intervention is computed independently using its corresponding probe, according to the regularized formulation in Eq. (10).

Importantly, λ represents the desired local logit shift, rather than an exact change in the probe output. Regularization and interactions between successive layers may affect the resulting activations.

Our goal is to develop a general steering framework that can be applied across different models, where the effects of interventions may vary not only between architectures but also across layers and attributes within the same model.

Rather than assuming a fixed relationship between λ and the resulting output, we use CRC to calibrate a single shared steering-strength parameter based on observed output-level steering failures. This allows the steering strength to be selected according to a specified risk threshold, without requiring the intervention effects to be identical across layers or models.

We have clarified activation extraction, aggregation, and intervention placement in Appendix B.1, and the local interpretation of the shared-logit target in Section 4.3 of the revised manuscript.

Q3: Speech quality at matched classifier-defined failure rates and distinction between evaluation criteria

Reply:

Identical CRC protocol. We first applied the same speaker-level CRC procedure to probe-guided steering and both difference-of-means (DoM) baselines, using the same 50 speaker-disjoint calibration/test splits. Probe-guided steering and single-layer DoM are evaluated on a common candidate-strength grid, while multi-layer DoM uses its method-specific grid, extended to |λ| ≤ 0.5 (steps of 0.02 up to 0.1, followed by steps of 0.1).

At α = 0.05, probe-guided steering achieves higher held-out WV-MOS across all feasible comparisons and is the only method with a feasible operating point for all four steering targets. Restricting probe-guided steering to the single-layer DoM grid changes its WV-MOS by at most 0.03 relative to the denser grid used in the main experiments, without changing the relative ranking of the methods.

However, since identical target risks do not necessarily imply identical empirical failure rates, we conducted two additional comparisons to directly assess speech quality at comparable levels of steering success.

AuthorsOpenReview ↗

Response to Reviewer 8Cwm Part 3/4

Matched empirical failure rates. To compare speech quality at comparable levels of steering effectiveness, independently of CRC-based strength selection, we evaluated classifier-defined failure rates and WV-MOS on identical utterances, using the same candidate-strength grids as above. For each method, we interpolated WV-MOS at the first crossing of the target empirical failure rate of 5%.  Extending the multi-layer DoM grid does not change the matched-failure results, since the first crossing occurs within the original strength range.

The table below reports the difference in WV-MOS between probe-guided steering and each baseline at this matched failure rate. Positive values indicate higher speech quality for probe-guided steering, while negative values favor the baseline. Values in brackets denote 95% paired speaker-level bootstrap confidence intervals.

Targetvs. Single-layer DoMvs. Multi-layer DoM
Happy−0.18 [−0.24, 0.01]*+0.35 [−0.05, 0.59]*
Sad+0.19 [0.08, 0.28]+0.35 [0.20, 0.49]
Female+0.21 [0.16, 0.27]+0.23 [0.18, 0.28]
Male+0.11 [0.06, 0.22]0.00 [−0.05, 0.10]

*For happy steering, the baselines do not reach a 5% failure rate in all bootstrap samples.

At matched classifier-defined failure rates, probe-guided steering achieves significantly higher WV-MOS than both baselines for sad and female steering, and than single-layer DoM for male steering. For male steering, no statistically significant difference in WV-MOS is observed relative to multi-layer DoM.

For happy steering, the comparison is less conclusive: single-layer steering attains higher WV-MOS and multi-layer steering lower WV-MOS, with both confidence intervals including zero. UTMOSv2 consistently favors probe-guided steering, with statistically significant improvements in most comparisons, further supporting its ability to preserve speech quality. 

Quality conditional on steering success. To assess whether the quality advantage persists among successfully steered outputs, we restricted the evaluation to held-out utterances classified as successfully steered at each method's CRC-selected strength, using the same candidate-strength grids as above.

At α = 0.05, the success-conditioned WV-MOS results are:

TargetProbe-guidedSingle-layer DoMMulti-layer DoM
Happy3.57InfeasibleInfeasible
Sad4.334.033.80
Female4.814.49Infeasible
Male4.604.274.20

Since the successfully steered subsets may differ across methods, this analysis complements rather than replaces the matched-failure comparison.

Probe-guided steering achieves higher WV-MOS across all feasible comparisons and is the only method with a feasible operating point for happy steering. These results show that the quality advantage persists when conditioning on classifier-defined steering success, rather than only in the overall averages.

Reviewer_JHn8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors give ProbeCRC, a probe-guided activation-steering framework for controllable autoregressive text-to-speech. ProbeCRC combines (i) layer-wise linear probes to find attribute-relevant directions, (ii) a shared steering parameter in probe-logit space, and (iii) Conformal Risk Control (CRC) to select the steering strength at a user-specified bound on steering-failure risk. The method intervenes on Fish-Speech Slow Transformer, showing bidirectional control over emotion, gender and age, showing improved control-quality trade-offs versus difference-of-means steering and stronger response than controllable TTS baselines, while empirically achieving low average classifier-defined failure risk on held-out data through CRC.

Critical strengths
  1. Combining probe-guided steering where the global control parameter is defined in probe-logit space togethwe with the CRC to calibrate steering strength is quite well motivated integration of TTS control. Also casting strength selection as risk control (rather than ad-hoc tuning) addresses a practical pain point in activation steering and introduces explicit reliability targets into controllable generation.

  2. The experiments cover three attributes - emotion, gender, and age and evaluate both steering effectiveness and speech preservation. The use of automatic quality metrics, word error rate, speaker-similarity measures, and human listening judgments provides complementary evidence. In particular, Figures 2 and 3 show a favorable attribute-control versus speech-quality trade-off compared with mean-difference steering.

  3. Having a reliable, inference-time control with guarantees is important for practical TTS deployment; the CRC-based calibration is a meaningful step toward trustworthy controllability.

  4. The approach is lightweight (no model fine-tuning) and potentially applicable to other generative backbones, which broadens impact.

Critical weaknesses

Major Weaknesses

  1. The regularized intervention in Eq. (10) simplifies to a scaled probe-weight direction. The paper lacks any direct comparisons with normalized probe-based steering and probe-probability-based strength selection, making it difficult to isolate the contribution of the proposed formulation and CRC. More abltations are definitely required.

  2. The CRC guarantee is defined using a binary loss from an external attribute classifier, rather than a direct measurement of perceptual attribute success. The resulting CRC guarantee therefore concerns classifier-defined failure and does not automatically imply an equivalent guarantee for human-perceived attribute changes. Moreover, although the authors use speaker-disjoint calibration/test splits, the paper does not clearly establish the exchangeability assumptions at the level of the individual calibration losses when multiple utterances originate from the same speaker. The distinction between speaker-level and utterance-level calibration is important for interpreting the theoretical guarantee. Finally, age steering is evaluated perceptually but is not included in the CRC risk-control experiments. These limitations narrow the scope of the reliability claims.

Minor Weaknesses

  1. Experiments are restricted to Fish-Speech. Evaluation on another TTS architecture would strengthen generalizability. Althought I don't expect the authors to do fresh experiments, but any results on some other TTS arch. is greatly appreciated.

  2. Sensitivity studies for the regularization parameter η\eta, selected intervention layers, and shared versus layer-specific steering strengths are missing.

  3. The listening study includes only 15 participants and four samples per condition. Inter-rater agreement and statistical significance are not reported.

  4. The paper does not clearly specify how interventions are applied across token positions during autoregressive generation, which may affect reproducibility.

  5. Although Appendix A.2 introduces a conservative upper-envelope correction, its effect on selected steering strengths and speech quality is not quantified.

Questions for authors
  1. Since Eq. (10) reduces to a scaled probe-weight direction, can the authors compare ProbeCRC against direct and normalized probe-direction steering using the same intervened layers and strength-selection protocol? Please also report the regularization coefficient η\eta used in the experiments and its sensitivity.

  2. What is the precise unit of each calibration loss, a speaker or an utterance? If multiple utterances from the same speaker contribute separate losses, how is the exchangeability assumption justified? How is the procedure handled when no candidate steering strength satisfies the CRC threshold?

  3. Can the authors provide evidence that the classifier-defined failure risk correlates with human-perceived steering success, particularly at CRC-selected operating points? Also, is the absence of CRC experiments for age due solely to the lack of a reliable age evaluator, and how should this limitation qualify the general reliability claim?

10 comments
AuthorsOpenReview ↗

Response to Reviewer JHn8 Part 1/9

We thank the reviewer for the detailed and constructive feedback. We have conducted additional experiments, expanded our ablation studies, and revised the theoretical formulation and experimental protocol to address the concerns raised below.

Major W1-a + Minor W2-a + Q1: Regularization sensitivity and probe-direction steering

Reply:

We agree that the intervention in Eq. (10) is proportional to the linear-probe direction. The contribution of our formulation lies not in introducing a different steering direction, but in determining the layer-specific intervention magnitudes through a shared target change in probe-logit space and a regularized objective.

Regularization (η) and probe-direction scaling. We compared our regularized intervention, defined in Eq. (10),

vℓ∗(λ)=(wℓwℓ⊤+ηI)−1wℓλ.v_\ell^*(\lambda)=\left(w_\ell w_\ell^\top+\eta I\right)^{-1}w_\ell\lambda.

with alternative regularization strengths and two simplified probe-based steering variants. In our main experiments, we use η = 0.01. We evaluate its sensitivity by considering η = 0.1, 1, 10, 100, alongside:

  • Direct probe-weight steering: vℓ(λ)∝λwℓv_\ell(\lambda)\propto\lambda w_\ell, corresponding to the relative layer-wise scaling obtained in the limit η → ∞ of Eq. (10), after global rescaling.
  • Normalized probe-direction steering: vℓ(λ)∝λwℓ/∣wℓ∣2v_\ell(\lambda)\propto\lambda w_\ell/|w_\ell|_2, assigning equal perturbation magnitudes across layers.

All variants use the same intervened layers, the same speaker-level CRC protocol, and the same 50 speaker-disjoint calibration/test splits. All variants are globally rescaled to inject the same total L2 perturbation per unit λ. Thus, the comparison isolates the effect of how the perturbation is distributed across layers, rather than differences in overall intervention magnitude.

ProbeCRC, the alternative regularization strengths, and direct probe-weight steering are evaluated on a shared 14-point strength grid, rather than the 26-point grid used for ProbeCRC in the main experiments. All grids share the same maximal perturbation magnitude; therefore, adding intermediate strengths cannot restore feasibility when the corrected calibration risk at the maximal strength already exceeds the CRC threshold.

Comparison under CRC.

Held-out WV-MOS at the CRC-selected strength (speaker-level CRC, 50 splits). Values are averaged over the splits in which a feasible strength was found; “–” indicates that no certified operating point was found in most splits.

TargetαOurs (η=0.01)η=0.1η=1η=10η=100DirectNormalized
Happy0.053.533.16–––––
0.103.943.85–––––
0.204.184.202.09––––
0.404.294.294.221.09––4.00
Sad0.054.314.304.294.104.094.074.25
0.104.324.324.344.294.284.284.33
0.204.364.364.354.294.294.284.33
0.404.364.364.354.364.344.364.37

For happy steering, η = 0.01 provides the most robust CRC feasibility across risk targets while generally achieving the highest speech quality. Larger regularization coefficients and simplified probe-direction steering variants exhibit reduced feasibility and greater quality degradation. For sad steering, all variants remain feasible, with η = 0.01 achieving the highest speech quality at the most stringent risk target and comparable quality at higher risk targets.

AuthorsOpenReview ↗

Response to Reviewer JHn8 Part 2/9

Comparison without CRC. To isolate the contribution of our steering formulation independently of CRC-based strength selection, we compared the fixed-strength curves of all variants on the full evaluation dataset, using the same total L2 perturbation budget per unit λ. We report classifier-defined success rates and WV-MOS at individual strengths. For each variant, we additionally report WV-MOS at the weakest grid strength whose classifier-defined success rate is at least 0.97.

VariantHappy: peak success (WV-MOS)Happy: success / WV-MOS at λ_maxHappy: WV-MOS at success ≥ 0.97Sad: peak success (WV-MOS)Sad: success / WV-MOS at λ_maxSad: WV-MOS at success ≥ 0.97
Ours (η=0.01)0.979 (3.48)0.978 / 3.353.661.000 (4.09)1.000 / 3.924.31
η=0.10.977 (3.46)0.974 / 3.053.691.000 (4.10)1.000 / 3.814.29
η=10.973 (3.43)0.862 / 1.923.431.000 (3.91)1.000 / 3.084.28
η=100.958 (3.55)0.629 / 1.08not reached1.000 (3.96)1.000 / 1.814.09
η=1000.951 (3.87)0.578 / 0.79not reached1.000 (3.70)0.998 / 1.424.04
Direct0.951 (3.31)0.556 / 0.80not reached1.000 (3.19)1.000 / 1.414.06
Normalized0.968 (3.34)0.700 / 1.42not reached1.000 (3.69)0.999 / 2.424.24

For happy steering, η = 0.01 and η = 0.1 provide the most favorable quality–success trade-off, while larger regularization coefficients and simplified probe-direction steering exhibit reduced success and substantial quality degradation at stronger interventions. For sad steering, all variants achieve high success rates, but η = 0.01 best preserves speech quality, particularly at larger steering strengths. Overall, these results support our regularization choice independently of CRC-based strength selection.

Summary. These results highlight the importance of the layer-wise magnitude allocation induced by Eq. (10), beyond the choice of probe directions alone. Small regularization coefficients yield a more favorable effectiveness–quality trade-off independently of CRC-based strength selection. Under CRC, they also provide more robust feasibility and better speech quality, whereas larger coefficients and alternative probe-direction formulations exhibit substantial quality degradation and non-monotonic responses that limit feasibility.

We have added these ablations to Appendix C.5.

AuthorsOpenReview ↗

Response to Reviewer JHn8 Part 3/9

Major W1-b: Probe-probability-based strength selection and the contribution of CRC

Reply:

Probe-probability-based strength selection. To isolate the contribution of CRC, we compared it with three selection rules based on internal probe predictions, in the spirit of Gao et al. (2026). For each utterance and candidate strength, we re-run steered generation and compute each layer's target-attribute probability by applying the sigmoid to the linear probe output on time-averaged hidden states. These states are collected before the intervention at the current layer, but after interventions at preceding layers. The per-utterance probability is the mean of the probabilities across all 28 layers.

For a target probability p∗=1−αp^*=1-\alpha, we select the weakest strength satisfying one of three criteria: (i) the mean probe probability over calibration speakers reaches p∗p^*; (ii) coverage-based selection: at least a fraction p* of calibration utterances are classified as the target attribute by the probe (probability ≥ 0.5), i.e., CRC’s criterion applied to the probe’s predictions instead of the output classifier’s; or (iii) the probe probability for each utterance reaches p∗p^*, using λmax⁡\lambda_{\max} if the threshold is not reached. All methods use the same probe-guided intervention, candidate-strength grid, 50 speaker-disjoint splits, and held-out external-classifier and WV-MOS evaluation.

For happy steering at α = 0.05, the most frequently selected strengths across splits are λ = 1.2 for mean-probability selection and λ = 0.8 for coverage-based selection, compared with λ = 2.4 for CRC. The per-utterance rule selects a mean strength of 1.07, averaged over held-out utterances and splits. These probe-based rules yield held-out failure rates of 18–30%, exceeding the target in all 50 splits. At α = 0.1–0.3, the target is exceeded in 94–100% of splits.

For sad steering, the calibration-based probe rules (mean probability and coverage) exceed the target in 74–100% of splits at α = 0.05, 0.1, 0.3. The per-utterance rule exceeds the target in 90–96% of splits at α = 0.05, 0.1, but performs better at α = 0.3, with a mean held-out failure rate of 26.2%, exceeding the target in 20% of splits. At α = 0.2, the mean-probability rule selects exactly the same strength as CRC (|λ| = 0.5) in all 50 splits, resulting in identical held-out failure rates (16.1% on average, exceeding the target in 14% of splits). In contrast, the coverage-based and per-utterance rules exceed the target in 92% and 70% of splits, respectively.

Overall, CRC achieves mean held-out failure rates below the target across all evaluated settings, whereas the probe-based rules exhibit less consistent risk control. Using only the last layer or the layer with the highest probe validation accuracy leads to the same overall conclusions.

These results show that high internal probe confidence does not necessarily imply successful steering at the speech-output level. While the probes identify effective intervention directions, CRC calibrates their strength using observed output-level failures. This comparison is included in Appendix C.4.

Comparison with fixed strengths and validation-based selection. To further isolate the contribution of CRC-based strength selection, we compared CRC with fixed steering strengths and two validation-based selection strategies: standard empirical validation and validation using the same monotonicity correction as CRC. All comparisons use speaker-level calibration and the same 50 speaker-disjoint calibration/test splits.

Fixed steering strengths. We evaluated predefined fixed steering strengths to assess the importance of adapting the intervention to the desired risk level. For happy steering at α = 0.2, CRC selects λ = 1.3 in 45/50 splits, achieving a mean held-out failure rate of 14.8% and a mean WV-MOS of 4.20. A weaker fixed strength of λ = 1.0 yields a slightly higher WV-MOS (4.26), but its mean held-out failure rate of 21.8% exceeds the target, and it satisfies the target in only 42% of splits (with failure rates of up to 36.8%). Conversely, stronger fixed interventions satisfy the target but may unnecessarily degrade speech quality or compromise speaker identity. These results illustrate the benefit of adapting the steering strength to the desired risk level, rather than relying on a fixed strength that may be either insufficient or unnecessarily strong.

Crucially, rather than relying on a fixed strength whose effectiveness is established empirically on previously evaluated samples, CRC uses calibration data to select a steering strength with a finite-sample guarantee on the expected classifier-defined failure rate for unseen speakers, under the stated CRC assumptions. This allows users to specify a desired risk level without knowing in advance which steering strength will achieve it on new data.

Reviewer_TWfTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProbeCRC, a framework for controlling speech attributes through probe-guided activation steering and conformal risk control (CRC). The main idea is to define steering strength in probe-logit space and calibrate it to satisfy a target failure-risk constraint. The results are promising, particularly in preserving speech quality while changing the target attribute. However, I have some concerns about the incremental contribution of the method and the evidence supporting its risk-control claims.

Critical strengths
  1. The paper addresses a practical problem: stronger activation steering can improve attribute control but may also degrade speech quality.

  2. The use of CRC to select steering strength is an interesting direction, as it provides a more principled alternative to manually tuning this parameter.

  3. The evaluation goes beyond automatic attribute classification and includes speech quality, content preservation, and human listening tests.

Critical weaknesses
  1. The contribution of the steering design is not entirely clear. The regularized intervention has a simple closed-form solution that amounts to scaling the linear-probe direction, so it is unclear what advantage it offers over directly scaling the probe weights. The paper also overlooks EmoShift (ICASSP 2026), which studies lightweight activation steering for emotion-aware speech synthesis. A clearer comparison with existing methods and targeted ablations would help establish the novelty of the approach.

  2. The benefit of CRC over simpler alternatives is unclear. The paper does not compare CRC with fixed steering strengths or validation-based selection. Without these comparisons, it is difficult to determine whether CRC provides a meaningful practical improvement.

  3. The risk guarantee may not fully reflect perceptual reliability. Steering failure is defined using an external speech classifier, so the guarantee applies to classifier-based predictions rather than directly to human perception. If the classifier disagrees with human judgments, satisfying the risk constraint does not necessarily mean that listeners perceive the intended attribute change.

Questions for authors
  1. The CRC guarantee assumes that the steering loss is monotonic with respect to the steering strength, but the paper shows that this does not always hold in practice. Could the authors clarify how the monotonicity correction affects the risk guarantee, and whether the guarantee still applies to the original steering failure rate?

  2. The steering failure rate is measured using external classifiers, while human evaluation is conducted separately. Do the authors observe a strong agreement between classifier predictions and human judgments, especially for emotion and age steering? It would be helpful to understand how well the reported risk reflects actual perceptual success.

  3. The method combines probe-guided steering, regularization, and CRC-based strength selection. Could the authors provide ablation results to better understand how much each component contributes, particularly whether CRC improves the trade-off between steering effectiveness and speech quality compared with using a fixed steering strength?

9 comments
AuthorsOpenReview ↗

Response to Reviewer TWfT Part 1/7

We thank the reviewer for the constructive feedback and the opportunity to clarify the methodological contributions of ProbeCRC. We have conducted additional experiments and revised the manuscript to address the concerns raised below.

W1 + Q3-a: Contribution of the steering design, comparison with existing methods, and intervention ablations

Reply:

While we agree that the closed-form intervention is proportional to the linear-probe direction, we would like to clarify that our contribution extends beyond the choice of steering direction.

In particular, our contribution lies in formulating activation steering as a shared target change in probe-logit space, rather than directly scaling the probe weights. This formulation yields a regularized intervention with layer-specific magnitudes determined by the learned probe parameters, providing a principled way to distribute the intervention across layers. We further integrate Conformal Risk Control (CRC) to select the shared steering strength according to a user-specified bound on classifier-defined steering-failure risk.

To disentangle the contributions of the steering mechanism and regularization, we conducted additional controlled ablation experiments using speaker-level calibration and 50 identical speaker-disjoint calibration/test splits. The underlying trade-off between steering strength, attribute change, and speech quality is illustrated in Figures 2 and 3 in the paper.

Candidate-strength grids. To ensure controlled comparisons, we use shared candidate-strength grids wherever applicable. For the steering-mechanism ablation, probe-guided steering and single-layer DoM use the same grid, while multi-layer DoM uses its method-specific grid, extended to ∣λ∣≤0.5\lvert\lambda\rvert\leq 0.5. For the regularization ablation, ProbeCRC, the alternative regularization strengths, and direct probe-weight steering use a common 14-point grid rather than the 26-point grid used in the main experiments. All grids within the regularization ablation share the same maximal perturbation magnitude; therefore, adding intermediate strengths cannot restore feasibility when the corrected calibration risk at the maximal strength exceeds the CRC threshold.

Steering mechanism. To isolate the contribution of the proposed intervention, we applied the same speaker-level CRC procedure to probe-guided steering and both difference-of-means (DoM) baselines, using identical calibration/test splits and the same monotonicity correction. For a controlled comparison, probe-guided steering and single-layer DoM use the same candidate-strength grid, while multi-layer DoM uses its method-specific grid, which we extended to ∣λ∣≤0.5\lvert\lambda\rvert\leq 0.5 (0.02 steps up to 0.1, then 0.1 steps); already at λ=0.1\lambda=0.1 the multi-layer DoM perturbation exceeds the largest probe-guided perturbation in total ℓ2\ell_2. WV-MOS is averaged over the 50 splits. “Infeasible” indicates that no certified operating point was found in any of the 50 splits; all reported values were feasible in every split.

At α=0.05\alpha=0.05, the held-out WV-MOS results are:

TargetProbe-guidedSingle-layer DoMMulti-layer DoM
Happy3.56InfeasibleInfeasible
Sad4.334.023.78
Female4.804.48Infeasible
Male4.594.274.20

Probe-guided steering achieves higher speech quality across all feasible comparisons, and is the only method with a feasible operating point for all four targets. For multi-layer DoM, larger strengths degrade the output rather than strengthening the target attribute (e.g., 91–100% failure for happy at λ≥0.2\lambda\geq 0.2, and a failure rate rising to 15% for female at λ=0.5\lambda=0.5). Because CRC, through the monotonicity correction applied identically to all methods, certifies a strength only if it remains safe at all larger strengths, this non-monotonic behavior removes feasibility. Restricting probe-guided steering to the single-layer DoM grid changes its WV-MOS by at most 0.03 relative to the denser grid used in the main experiments, without changing the relative ranking of the methods. This indicates that the observed quality advantage is robust to the difference in grid resolution.

AuthorsOpenReview ↗

Response to Reviewer TWfT Part 2/7

Comparison without CRC. To isolate the contribution of the steering mechanism independently of CRC-based strength selection, we additionally compared probe-guided steering and both DoM baselines at matched empirical classifier failure rates, using the same candidate-strength grids as in the CRC comparison above, including the extended multi-layer DoM grid (∣λ∣≤0.5\lvert\lambda\rvert\leq 0.5).

For each method, we evaluated classifier-defined failure and speech quality across steering strengths on identical utterances. We interpolated WV-MOS at matched failure rates using the first crossing of the target failure rate and computed 95% paired speaker-level bootstrap confidence intervals.

At a matched classifier failure rate of 5%, the differences in WV-MOS (probe-guided minus baseline; positive values favor probe-guided steering) are:

Targetvs. Single-layer DoMvs. Multi-layer DoM
Happy−0.18 [−0.24, 0.01]*+0.35 [−0.05, 0.59]*
Sad+0.19 [0.08, 0.28]+0.35 [0.20, 0.49]
Female+0.21 [0.16, 0.27]+0.23 [0.18, 0.28]
Male+0.11 [0.06, 0.22]0.00 [−0.05, 0.10]

*For happy steering, the baselines do not reach a 5% failure rate in all bootstrap samples.

Probe-guided steering significantly improves WV-MOS for sad and female steering against both baselines, and for male steering against single-layer DoM. For male steering, no statistically significant difference in WV-MOS is observed relative to multi-layer DoM. For happy steering, the comparison is less conclusive: single-layer DoM achieves higher WV-MOS at the matched failure rate, whereas multi-layer DoM achieves lower WV-MOS, with both confidence intervals including zero. UTMOSv2 consistently favors probe-guided steering, with statistically significant improvements in most comparisons, further supporting its ability to preserve speech quality.

These results demonstrate that the quality advantage of probe-guided steering over both DoM baselines is not solely attributable to CRC-based strength selection. Importantly, extending the multi-layer DoM grid does not change these matched-failure results, since the first crossing of the target failure rate occurs within the original strength range. In contrast, the deterioration at larger strengths affects CRC feasibility through the monotonicity correction.

Success-conditioned quality. We additionally compared WV-MOS restricted to held-out utterances classified as successfully steered at each method's CRC-selected strength (α=0.05\alpha=0.05, using the same grids as above). Probe-guided steering achieves higher WV-MOS across all feasible comparisons: 4.33 versus 4.03 and 3.80 for sad steering, 4.81 versus 4.49 for female steering, and 4.60 versus 4.27 and 4.20 for male steering, compared with single-layer and multi-layer DoM, respectively. For happy steering, probe-guided steering is the only feasible method (WV-MOS 3.57). Thus, the quality advantage persists when conditioning on classifier-defined steering success. Since the successfully steered subsets may differ across methods, this analysis complements rather than replaces the matched-failure comparison.

AuthorsOpenReview ↗

Response to Reviewer TWfT Part 3/7

Regularization (η\eta) and probe-direction scaling. We compared our regularized intervention, defined in Eq. (10),

vℓ∗(λ)=(wℓwℓ⊤+ηI)−1wℓλ.v_\ell^*(\lambda)=\left(w_\ell w_\ell^\top+\eta I\right)^{-1}w_\ell\lambda.

with alternative regularization strengths and two simplified probe-based steering variants. In our main experiments, we use η=0.01\eta=0.01. We evaluate its sensitivity by considering η∈{0.1,1,10,100}\eta\in\{0.1,1,10,100\}, alongside:

  • Direct probe-weight steering: vℓ(λ)∝λwℓv_\ell(\lambda)\propto\lambda w_\ell, corresponding to the relative layer-wise scaling obtained in the limit η→∞\eta\to\infty of Eq. (10), after global rescaling.
  • Normalized probe-direction steering: vℓ(λ)∝λwℓ/∥wℓ∥2v_\ell(\lambda)\propto\lambda w_\ell/\lVert w_\ell\rVert_2, assigning equal perturbation magnitudes across layers.

All variants use the same intervened layers, the same speaker-level CRC protocol, and the same 50 speaker-disjoint calibration/test splits. All variants are globally rescaled to inject the same total ℓ2\ell_2 perturbation per unit λ\lambda. Thus, the comparison isolates the effect of how the perturbation is distributed across layers, rather than differences in overall intervention magnitude.

Comparison under CRC.

Held-out WV-MOS at the CRC-selected strength (speaker-level CRC, 50 splits). Values are averaged over the splits in which a feasible strength was found; “–” indicates that no certified operating point was found in most splits.

TargetαOurs (η=0.01)η=0.1η=1η=10η=100DirectNormalized
Happy0.053.533.16–––––
0.103.943.85–––––
0.204.184.202.09––––
0.404.294.294.221.09––4.00
Sad0.054.314.304.294.104.094.074.25
0.104.324.324.344.294.284.284.33
0.204.364.364.354.294.294.284.33
0.404.364.364.354.364.344.364.37

For happy steering, η=0.01\eta=0.01 provides the most robust CRC feasibility across risk targets while generally achieving the highest speech quality. Larger regularization coefficients and simplified probe-direction steering variants exhibit reduced feasibility and greater quality degradation.

For sad steering, all variants remain feasible, with η=0.01\eta=0.01 achieving the highest speech quality at the most stringent risk target and comparable quality at higher risk targets.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Revisions and Response to Reviewers

We thank the reviewers for their time and effort in evaluating our paper. Their constructive feedback has helped us refine the theoretical formulation, strengthen the empirical validation, and clarify the methodological motivation of our work, as reflected in the revised manuscript. Their main concerns included exchangeability and its implications for risk guarantees, the contribution of the proposed steering formulation and CRC-based strength selection, the relationship between classifier-defined risk and human perception, the generalizability of the approach, and the reproducibility of the implementation. We have systematically addressed these concerns through six major revisions:

  1. Speaker-Level Calibration and Theoretical Risk Guarantees. We revised our CRC framework to use speakers, rather than individual utterances, as the calibration units, accounting for dependencies between utterances from the same speaker. Under the assumption of exchangeable speakers and the remaining CRC conditions, this formulation controls the expected classifier-defined failure rate of a new speaker without requiring independence between utterances from the same speaker. We further clarified the scope of the risk guarantee, the handling of infeasible operating points, the strength-selection rule, and the effects of the monotonicity correction.

  2. Expanded Ablation Studies and Clarification of Novelty. We conducted extensive ablation experiments comparing our regularized intervention with direct and normalized probe-direction steering, alternative regularization coefficients, different subsets of intervention layers, and alternative layer-wise magnitude allocations under a shared global steering strength. These experiments clarify the contribution of the proposed layer-wise intervention formulation and demonstrate its importance for achieving stringent risk targets while preserving speech quality.

  3. Practical Benefit of Conformal Risk Control. We added comparisons with fixed steering strengths, standard validation-based selection, monotonicity-corrected validation, and probe-probability-based strength selection. These experiments demonstrate the importance of calibrating steering strength using output-level failures rather than internal probe confidence, and highlight the practical benefit of CRC's finite-sample correction, particularly when calibration data are limited.

  4. Strengthened Human Evaluation and Perceptual Validation. We conducted additional analyses of the human listening study, including classifier–human agreement, correlations between automatic metrics and human ratings, inter-rater agreement, statistical significance testing, and confidence intervals. These analyses reveal strong agreement between classifier predictions and human judgments, supporting the perceptual relevance of our automatic evaluation metrics and the effectiveness of the proposed steering approach. We also investigated automatic age classification as a potential basis for extending risk control to age steering.

  5. Controlled Comparisons of Steering Effectiveness and Speech Quality. We extended our comparison with difference-of-means baselines using identical speaker-level CRC protocols, matched empirical classifier failure rates, and quality evaluations conditional on successful steering.

  6. Generalization Across Architectures and Improved Reproducibility. We extended our experimental evaluation to CosyVoice3, providing preliminary evidence that ProbeCRC generalizes beyond Fish-Speech. We also clarified the probe-training activation source, token selection and aggregation, intervention placement during autoregressive generation, and the local interpretation of the shared probe-logit target, improving the reproducibility and interpretation of the proposed framework.

Together, these revisions provide a clearer theoretical formulation, stronger empirical validation, and a more comprehensive understanding of the complementary contributions of probe-guided intervention and conformal risk control.

We have revised the manuscript to incorporate the additional analyses, experiments, and clarifications described in our responses below. All changes are highlighted in blue in the revised manuscript.

We sincerely appreciate the thoughtful and constructive feedback provided by all the reviewers. In response, we have uploaded a significantly revised version of our paper, which carefully addresses the review comments. We believe these revisions have led to meaningful improvements in several key aspects, thanks to your valuable suggestions.

Please do not hesitate to share any additional questions or concerns.

Thank you once again for your time, understanding, and insightful feedback.

Area_Chair_YV9GOpenReview ↗

Please acknowledge author response

Dear Reviewers,

Please acknowledge the authors' responses and adjust your scores if appropriate.

Best,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.