acceptodds
Under review as a conference paper at ICLR 2027

Edit Who Speaks, Control How They Speak: Global Timbre Editing and Local Instruction Control for TTS

Abstract

Instruction-based text-to-speech (TTS) mainly focuses on cloning persistent speaker identity or controlling local speech expression. Voice cloning preserves the speaker identity from a reference voice, whereas text-based voice design generates a voice from a natural-language description. Neither provides a mechanism for explicitly editing the timbre of a given reference voice. Meanwhile, utterance-level instructions control speech expressiveness, but cannot specify how it should vary across segments. We introduce EDICT, a framework that decouples persistent speaker identity from transient expressive instructions while coupling them through an editable acoustic representation. EDICT first maps a reference voice and structured timbre edits to an edited codec-token representation. This representation is shared across all text segments to anchor a consistent target voice. For each segment, a separate natural-language instruction is injected into a frozen TTS backbone to control local expression. To enable instruction switching without disrupting acoustic continuity, EDICT rebuilds the KV cache at each segment boundary. It retains a bounded acoustic context to preserve local speech history while refreshing the instruction condition. This design provides independent control over what the voice sounds like and how it speaks across segments. Evaluations on our proposed TimbreEdit-Bench and IntraTTS-Bench demonstrate improved timbre-editing accuracy and segment-level instruction adherence over baseline methods, while maintaining cross-segment voice consistency and speech quality.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Controllable text-to-speech, timbre editing, instruction following, expressive speech synthesis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_RkY2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents EDICT, a text-to-speech framework combining reference-conditioned timbre editing with within-utterance expressive control.

A notable design is its representation of relative timbre-editing requests as structured attribute operations, explicitly specifying which dimensions to modify, the direction and magnitude of changes, and which attributes to preserve. A trainable editor uses these operations and reference speech to generate a reusable edited codec reference. A frozen TTS backbone then uses this reference to maintain a consistent voice, rebuilding its KV cache at instruction boundaries with updated conditions and retained acoustic context to support local control and continuity.

The authors introduce two bilingual benchmarks, TimbreEdit-Bench and IntraTTS-Bench, and validate the framework through automatic metrics, listening tests, and ablations, supplemented by ESD and MED-TTS experiments. Results show improved target-voice matching and a favorable balance between local instruction adherence, speaker consistency, and transition quality, with trade-offs in preserving unedited attributes and speech content accuracy.

Critical strengths

1.Structured relative timbre editing provides a useful control interface. Explicit operations specify which attributes to change or preserve, producing an edited voice reference reusable across texts and delivery conditions.

2.KV cache reconstruction is supported by informative ablations. Comparisons with persistent and truncated caches demonstrate its benefits, while results across two TTS backbones and MED-TTS provide further evidence of effectiveness.

Critical weaknesses

1.Selective timbre editing is insufficiently validated. EDICT preserves unedited attributes less reliably than Qwen3-TTS-VD, and evaluation checks edit direction but not the specified magnitude. Real-speech experiments also lack fine-grained attribute evaluation, leaving uncertainty about targeted editing while preserving other attributes.

2.Direct comparisons with the closest methods are missing. The experiments do not directly compare against relative timbre-editing methods such as VoxEditor or the KV-cache control method of Kang et al., limiting assessment of the contribution beyond prior work.

Questions for authors

1.Selective editing on real speech. The real-speech experiments in H.1 lack attribute-level evaluation, and target-speaker similarity alone does not establish selective editing. Could the authors evaluate single- or dual-attribute edits on real references, measuring both the requested changes and preservation of unedited attributes, and report their joint success rate?

2.Effectiveness of edit magnitude. The structured operations distinguish one-level and two-level changes, whereas Up/Down evaluation checks only direction. For the same reference and attribute, do different magnitude instructions produce ordered and perceptually distinguishable changes? Could the authors provide a targeted evaluation?

3.Comparison with the closest methods. Table 6 supports recomputation over truncating and reusing old KV states, but the advantage over the KV-cache control method of Kang et al. remains unclear. Could the authors provide a direct comparison and a TED-TTS baseline using the same edited reference? For timbre editing, could they compare against an editing–synthesis pipeline incorporating a method such as VoxEditor, or explain the specific limitations preventing such a comparison?

1 comment
AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer RkY2,

Thank you again for your thoughtful feedback. Our responses include attribute-level evaluation on real speech references, an acoustic analysis of pitch-edit magnitude, and comparisons with Kang. The corresponding revisions are highlighted in blue in the updated manuscript.

As the discussion period draws to a close, we would be grateful if you could take a look at our responses and the revised manuscript.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_xABpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates natural-language instruction-guided TTS, aiming to combine global timbre editing with local expressive control. The paper makes three main contributions:

  1. Global Timbre Editing: Natural-language editing instructions are mapped to structured operations over nine timbre attributes. Based on Qwen3-TTS-Base, the timbre editor is trained using SFT and Reward-Weighted DPO (wDPO) to generate an edited codec reference that serves as a shared timbre anchor for subsequent speech synthesis.
  2. Segment-Level Instruction Control: Using a frozen TTS backbone, KV Cache Reconstruction rebuilds the cache at instruction-switching boundaries while retaining selected initial and recent acoustic context to reduce control leakage and maintain acoustic continuity.
  3. Evaluation Benchmarks: TimbreEdit-Bench and IntraTTS-Bench are constructed to evaluate timbre editing and intra-utterance instruction control, respectively, and to further assess their combined effectiveness.

Experimental results show that EDICT-Stage I outperforms the compared instruction-conditioned systems in target-timbre similarity and edit accuracy, while EDICT-Stage II improves local instruction adherence and transition quality over TED-TTS. The complete EDICT system further improves target-timbre matching and transition quality, but at the cost of reduced segment-level instruction accuracy and content fidelity.

Critical strengths
  1. The paper combines global timbre editing with local expressive control, allowing users to modify specified attributes of a reference voice while controlling emotion, speaking rate, and prosody across different segments of the same utterance, providing a more fine-grained approach to TTS control.
  2. The paper achieves promising results in timbre editing. In Table 1, EDICT-Stage I achieves a WavLM SIM-T of 0.819 under Structured Instructions, outperforming Qwen3-TTS-VD (0.701). In Table 2, its Category Accuracy reaches 89.3%, compared with 53.2% for Qwen3-TTS-VD, demonstrating the effectiveness of structured timbre editing.
  3. The effectiveness of KV Cache Reconstruction is supported by extensive experiments. In Table 3, with Qwen3-TTS-VD as the backbone, EDICT-Stage II improves SegAcc from 74.25% (TED-TTS) to 82.30% and Trans from 78.03% to 82.00%. Table 6 further compares different caching strategies and acoustic context configurations, providing direct evidence for the effectiveness of this mechanism.
  4. The paper constructs bilingual evaluation benchmarks covering Chinese and English and validates the local control mechanism on both Qwen3-TTS-VD and CosyVoice 2. It also provides comprehensive experimental evidence through real speech references, external benchmarks, subjective tests, and ablation studies.
Critical weaknesses
  1. The paper aims to modify specified timbre attributes while preserving others. However, in Table 2, EDICT-Stage I achieves Preservation scores of 66.7% (Structured) and 70.1% (Descriptive), lower than Qwen3-TTS-VD's 83.3% and 85.0%, respectively. The wDPO preference scoring relies on target-speaker similarity without explicitly accounting for attribute preservation. Although Table 5 demonstrates improvements in SIM-T through DPO and wDPO, it does not evaluate their effects on attribute-level editing accuracy or preservation. Therefore, it remains unclear whether the improvements in target-voice similarity also lead to more accurate and selective timbre editing.
  2. Both the training data and TimbreEdit-Bench are based on synthetic speech generated by Qwen3-TTS-VD. Although Appendix H.1 includes experiments using real speech references from ESD, the evaluation covers only 20 speakers and 200 editing pairs, providing limited evidence of generalization to diverse real-world voices.
  3. Table 1 compares EDICT-Stage I, trained on 1M editing pairs with subsequent wDPO, against baselines without comparable task-specific fine-tuning. The paper lacks a data-matched Qwen3-TTS-Base baseline trained directly on natural-language editing instructions, making it difficult to disentangle the benefits of task-specific fine-tuning from those of the proposed timbre-editing design. Direct comparisons with specialized methods such as VoxEditor and VoiceShop are also missing.
  4. The evaluation of joint timbre editing and local instruction control is insufficiently comprehensive. Although Table 4 reports overall target-voice similarity (SIM-T), inter-segment speaker similarity (SIM-I), instruction adherence (SegAcc), and transition quality (Trans), it does not evaluate attribute-level editing accuracy or preservation, as reported in Table 2. Therefore, it remains unclear whether the requested global timbre edits are consistently maintained across segments with different local instructions, and whether unspecified timbre attributes remain unchanged during these expressive variations.
  5. IntraTTS-Bench contains only utterances with two or three text segments, which is insufficient to establish the stability and scalability of the proposed method for longer utterances involving frequent instruction switches.
  6. IntraTTS-Bench jointly generates segment texts and instructions (Appendix C.2), potentially introducing correlations between textual semantics and requested expressions. The paper lacks controlled experiments that change only one segment's instruction while keeping other inputs fixed, or evaluate instructions that conflict with the text's implied emotion. This limits the evidence for independent segment-level control.
Questions for authors
  1. Timbre-editing instructions are converted into structured attribute operations using Qwen3-14B. How does the system handle ambiguous instructions or requests that cannot be directly mapped to the nine predefined timbre attributes?
  2. Global timbre editing and local delivery instructions may affect overlapping acoustic properties. For example, a global instruction may request a lower pitch register, while a local instruction calls for excited, high-pitched delivery. How does the system reconcile these two levels of control in such cases?
  3. KV Cache Reconstruction requires rebuilding the cache at each instruction switch. What are the associated inference latency and GPU memory overhead compared with existing caching strategies?
  4. EDICT reconstructs the KV cache at each segment boundary using the first S and most recent K generated codec positions. Could repeated reconstruction lead to progressive degradation or drift in timbre consistency, instruction adherence, or content accuracy as the number of segments increases? Have the authors evaluated these effects across successive segments in longer utterances?
2 comments
AuthorsOpenReview ↗

Response to Reviewer xABp (1/3)

Thank you for the constructive review. We have updated the manuscript PDF, with revisions highlighted in blue for easy reference. We reproduce the relevant results below using the revised manuscript's table numbers.

R1. Selective editing and preference optimization (W1)

Joint success requires every requested edit and keep operation to pass on the same output. The updated Table 2 shows why Preservation alone is insufficient: Qwen3-TTS-VD has higher Preservation but much lower Joint success.

Table 2. Attribute-operation accuracy and joint success (%). Gemini compares source/generated audio without the target recording. Up/Down measures ordinal direction, Category measures categorical targets, and Preservation measures unchanged attributes. Direct-Instruct uses descriptive instructions.

SystemInstructionUp ↑Down ↑Category ↑Preserv. ↑Joint ↑
Qwen3-TTS-VDStructured67.064.353.283.37.4
Qwen3-TTS-VDDescriptive60.063.049.885.011.2
Qwen3-TTS-BaseStructured62.354.546.445.73.6
Qwen3-TTS-BaseDescriptive66.054.550.045.65.0
CosyVoice 2Structured68.165.557.768.210.2
CosyVoice 2Descriptive72.273.369.066.213.6
Direct-Instruct SFT—65.467.878.160.218.2
EDICT-Stage IStructured70.880.689.366.729.2
EDICT-Stage IDescriptive77.979.191.770.130.6

The new Table 7 evaluates the attribute-level effects of preference optimization. wDPO improves Preservation and Joint over SFT under both instruction forms, complementing the SIM-T results in Table 6. Preservation remains imperfect: the reward does not explicitly optimize it, and Descriptive Preservation is slightly below DPO's.

Table 7. Attribute-level training ablations (%). Up/Down measures ordinal direction; Category measures categorical targets; Preservation measures unchanged attributes. Joint requires all edit and keep operations to pass on the same output.

SystemInstructionUp ↑Down ↑Category ↑Preserv. ↑Joint ↑
SFT (1M)Structured68.671.878.357.622.2
SFT (1M)Descriptive70.469.786.262.123.2
DPO (M = 16)Structured71.075.182.763.927.2
DPO (M = 16)Descriptive75.677.286.471.627.8
wDPO (M = 16)Structured70.880.689.366.729.2
wDPO (M = 16)Descriptive77.979.191.770.130.6

R2. Synthetic benchmarks and real references (W2)

Synthetic profiles enable controlled nine-attribute requests, followed by audio screening; constructing equivalent real-data benchmarks requires detailed attribute annotation. ESD tests transfer to human recordings. Table 15 adds attribute-level evaluation on 200 requests; Joint uses 188 containing both edited and preserved attributes.

Table 15. Attribute-level results on real ESD references (%). There are 200 requests; Joint uses the 188 with both edit and keep requirements, excluding 12 requests that edit all nine attributes.

SystemInstructionUp ↑Down ↑Category ↑Preserv. ↑Joint ↑
Qwen3-TTS-VDStructured56.865.457.784.88.5
Qwen3-TTS-VDDescriptive62.563.758.885.410.1
Qwen3-TTS-BaseStructured61.656.365.051.74.3
Qwen3-TTS-BaseDescriptive58.855.157.750.73.7
CosyVoice 2Structured57.967.660.081.811.7
CosyVoice 2Descriptive62.370.659.385.012.8
EDICT-Stage IStructured76.580.486.265.428.2
EDICT-Stage IDescriptive78.482.786.063.927.7

The new Figure 12 shows that only 6/90 same-language speaker pairs differ on one or two attributes. These results support compositional editing on real references, with broader speaker diversity and small-edit coverage remaining limited.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer xABp,

Thank you again for your thoughtful feedback. Our responses include a data-matched Direct-Instruct SFT baseline, controlled single-segment instruction interventions, and additional analyses of segment-position performance and inference efficiency. The corresponding revisions are highlighted in blue in the updated manuscript.

As the discussion period draws to a close, we would be grateful if you could take a look at our responses and the revised manuscript.

Thank you again for your time and consideration.

Best regards,
The Authors

Reviewer_Vq6COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

EDICT combines global voice-attribute editing with local instruction control for TTS. A reference-conditioned editor first changes selected attributes through supervised training and preference optimization. During synthesis, the system reconstructs the attention cache at instruction boundaries using the edited reference and a short acoustic context. The evaluation covers two TTS backbones, synthetic and real recordings, and human judgments.

Critical strengths

Cache reconstruction improves segment control and transitions across two TTS backbones, with ablations separating its effect from simple cache truncation (Tables 3 and 6). Timbre editing is also evaluated beyond the encoder used for preference training, through an independent encoder, attribute judgments, real recordings, and a small listening study.

Critical weaknesses

Editing and preservation are reported separately (Table 2), leaving unclear how often a request succeeds on both. The real-voice tests mostly involve large changes to several attributes, and the combined system lacks attribute-level checks across locally controlled segments. Comparisons with closely related voice-editing and KV-swapping methods are also missing, making the added contribution harder to assess.

Questions for authors

Could you report joint edit-and-preserve success for SFT and wDPO, including smaller edits on real recordings? How does EDICT compare with VoxEditor and the cited KV-swapping method under compatible settings? In the combined system, how well are edited and preserved voice attributes maintained across segments with different local instructions?

2 comments
AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer Vq6C,

Thank you again for your thoughtful feedback. Our responses include additional edit-and-preservation evaluations on synthetic and real references, as well as comparisons with Kang. The corresponding revisions are highlighted in blue in the updated manuscript.

As the discussion period draws to a close, we would be grateful if you could take a look at our responses and the revised manuscript.

Thank you again for your time and consideration.

Best regards,
The Authors

AuthorsOpenReview ↗

Follow-up Before the Discussion Deadline

Dear Reviewer Vq6C,

As the discussion period will close shortly, we would be grateful if you could take a look at our point-by-point responses and revised manuscript before the deadline, if time permits. Revisions are highlighted in blue for ease of reference.

Thank you again for your time and consideration.

Best regards, The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

Follow-up on our responses

Thank you again for your thoughtful feedback. We have posted point-by-point responses to all three reviews and uploaded a revised manuscript with changes highlighted in blue. Our Summary of Revisions and Clarifications gives an overview of the new experiments and clarifications. If any concerns remain or you have further questions, we would be happy to discuss them.

AuthorsOpenReview ↗

A Brief Follow-up

Dear Reviewers,

Thank you again for your time and thoughtful feedback. If any further clarification on our responses or the updated manuscript would be helpful, we would be happy to provide it. For ease of reference, revisions are highlighted in blue in the updated manuscript.

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.