acceptodds
Under review as a conference paper at ICLR 2027

Learning Persistent and Editable Voice Profiles with Generative Retrieval

Abstract

Existing controllable text-to-speech systems can design voices or modify reference speakers, but these capabilities are often mediated by implicit voice representations distributed across conditioning and generation paths without exposing a persistent state that can be compiled, recovered or edited. In this paper, we introduce GRAPE-TTS, a framework for Generative Retrieval and Acoustic Profile Editing, which represents each voice as a persistent acoustic profile comprising a short sequence of residual-quantization identifiers. A supervised residual vector-quantization codec maps speaker embeddings into profiles, and an assembler converts each profile into acoustic conditioning for a TTS backbone. All voice operations are then acts of generative retrieval over this shared state. Extensive experiments demonstrate that GRAPE-TTS achieves strong performance across voice design, cloning, and editing, while preserving the naturalness and intelligibility of the backbone. These results establish explicit acoustic profiles as a unified and direct representation for controllable speech synthesis.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
text-to-speech, generative retrieval, discrete profiles, voice design, voice editing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vPAeOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a concept of "voice profile", which is the speaker embedding output from ECAPA-TDNN, with further revisions. Specifically, the authors trained a VQ-VAR model to decompose the underlying semantics of the speaker embeddings, with additional supervision, such as age, gender, pitch, etc.

Critical strengths

(1) We problem formulation makes sense: multiple speech generation tasks should share some common input information regardless of the specific task. (2) Experimentally, the paper provides sufficient baselines, together with sufficient details in the appendix.

Critical weaknesses

(1) Using a time-invariant speaker embedding as the "voice profile" is not very solid to me. There should be quite a lot of dynamics for the speaking style from the speaker that can hardly be represented by a speaker embedding. (2) I have some concerns about the soundness of the experimental results. (2.1) In Table 2, the proposed method doesn't show a clear strength over the baselines. (2.2) In Table 1, the metric to compare is exactly the metric to select the candidate, which is clearly a leakage. (2.3) For a voice generation paper, the paper doesn't disclose any basic intelligibility number, aka, WER/CER of the generated content. (3) The paper title mentions retrieval, but it's not very clear to me where the retrieval happens; (4) The proposed model is not sufficiently tested on a wide range of benchmarks.

Questions for authors

As in weakness.

3 comments

We thank the reviewer for the focused criticism. The concerns about the profile's scope, empirical advantages, metric sharing, intelligibility, retrieval terminology, and benchmark coverage identify important limitations in the original presentation. We address each concern below with expanded evaluations and specific revisions.

W1. A persistent time-invariant acoustic profile cannot by itself represent utterance-level dynamics such as emotion trajectories, local prosodic variation, emphasis, and other fine-grained aspects of speaking style

We agree that a time-invariant embedding cannot encode an utterance-level style trajectory by itself. The profile is intentionally time-invariant, while the complete synthesis system contains time-varying stages. The profile is a persistent acoustic state that is independent of the target text. Trait supervision associates levels 0 to 3 with gender, age, pitch, and speaking rate. It does not store local prosodic dynamics.

The assembler maps the profile to a 64-dimensional PCA coefficient sequence, which inverse PCA converts into a mel carrier spanning approximately eight seconds. Content timing, emphasis, and local prosodic variation enter through the target text and the proxy backbone model during synthesis. Dynamic style therefore lies outside the profile's storage scope, while parts of its realization remain available in later stages of the system.

The independent instruction-following evaluation provides evidence for the intended design-text scope, with DSD of 91.7 and RP of 85.3. These measurements do not establish control over emotion arcs or detailed local prosody. We will state this boundary explicitly as a limitation and avoid implying that the persistent profile contains a complete dynamic style trajectory.

W2.1. Table 2 does not demonstrate a clear empirical advantage over the baselines, and the respective evaluation domains and source-speaker fidelity limitations of GRAPE-TTS require clearer interpretation

The purpose of Table 2 is to evaluate backbone-capability retention under shared-profile conditioning. GRAPE-TTS uses one profile space for design, reference recovery, and editing, with frozen ZipVoice serving as the proxy backbone model. The experiment tests whether this shared representation retains the backbone's general synthesis ability and where its representation resolution limits cloning fidelity.

MethodELDERLY67 SIMSeed-zh SIMSeed-en SIMMacro SIMUTMOS
GRAPE-TTS.580.611.561.5843.184
ZipVoice.409.750.666.6083.222

GRAPE-TTS remains close to ZipVoice in Macro SIM and UTMOS, with differences of .024 and .038. On ELDERLY67, GRAPE-TTS obtains .580, compared with .409 for ZipVoice and .465 for the strongest original baseline. These results show that the shared profile can improve registered-domain elderly-speaker recovery while retaining much of the backbone's overall synthesis capability.

The Seed results expose the current limitation clearly. GRAPE-TTS trails ZipVoice by .139 on Seed-zh and .105 on Seed-en. Discrete RVQ encoding and the PCA carrier realization restrict the source-speaker detail available to the backbone. The fidelity of carrier realization and conditioning of the proxy backbone model also limit the final output. We will present Table 2 as evidence of this capability-retention and resolution tradeoff, with no claim that GRAPE-TTS replaces dedicated zero-shot cloning systems.

ELDERLY67 is a registered-domain, speaker-seen evaluation. Speaker overlap with the registry is intentional. All 134 evaluation recordings are excluded from training, and the reference and held-out recordings are distinct live phone recordings from the same speakers. Our zero-shot speaker evaluation relies on Seed-zh and Seed-en, where we report the remaining similarity gaps explicitly.

W2.2. Table 1 uses the same trait measurements for candidate selection and system evaluation, creating a dependent evaluation that requires matched candidate budgets and independent judges or encoders

The trait probes are trained on registry and synthetic entries that share no items with the evaluation sets. The main concern is therefore the dependence created by using the same trait measurements for selection and reporting. We address this dependence through matched candidate budgets, an independent judge, and independent speaker encoders.

First, the contribution of selection is directly disclosed. The autoregressive generator produces NN candidates, and a fixed disclosed rule selects one profile. The N=1N=1 and N=6N=6 comparison isolates this contribution.

SystemCandidatesGenderPitchRateAge
GRAPE-TTS1.932.570.656.560
GRAPE-TTS6.966.829.873.776
Qwen3-TTS-vd1.958.541.620.369
Qwen3-TTS-vd6.992.785.804.613

The matched-N=6N=6 comparison gives GRAPE-TTS and Qwen3-TTS-vd the same candidate budget and the same frozen selection probes. GRAPE-TTS maintains higher pitch, rate, and age accuracy under this controlled comparison. Spark-TTS with six candidates and structured ground-truth trait labels leads on gender, pitch, and rate, while GRAPE-TTS retains a 10.8-point age advantage. Spark-TTS serves as a structured-conditioning reference because it does not consume the free-text design descriptions used by GRAPE-TTS.

Second, the expanded evaluation includes an independent Qwen2.5-Omni-7B judge under the official InstructTTSEval evaluation setup.

SystemDSDRP
GRAPE-TTS91.785.3
Qwen3-TTS-vd85.369.3

This judge provides trait and instruction-following evidence outside the probes used for candidate selection.

Third, the expanded similarity evaluation re-scores the cloning and editing outputs with two WavLM encoders that were not used in training, selection, or identity gating. Seed-en and Seed-zh cloning rankings correlate with the ECAPA rankings at Spearman .86 to .93. Together, these controls separate the effects of candidate budget, selection metrics, and independent evaluation.

W2.3. The submission does not disclose basic WER or CER results for a voice-generation system, making it difficult to assess whether profile conditioning and proxy-backbone synthesis preserve intelligibility

We agree that a voice-generation paper should report intelligibility. Our expanded evaluation adds the following results after synthesis through the proxy backbone model:

Evaluation setMetricGRAPE-TTSBaseline range
Seed-enWER3.1%1.8% to 26.3%
Seed-zhCER4.9%4.5% to 269.9%

The expanded benchmark table also includes per-system WER results for all 15 cloning baselines. GRAPE-TTS lies within the corresponding baseline ranges on both language evaluations.

The original submission omitted these results because final intelligibility is primarily determined by two system factors. The first is the frozen backbone's own synthesis capability. The second is how faithfully the carrier conditions the proxy backbone model, which is a limitation of our current realization. These measurements therefore characterize the complete synthesis system more directly than the profile mechanism alone.

This rationale explains the original omission, and we agree that WER and CER are still essential system-level metrics. The expanded results quantify the actual intelligibility of the complete system, including the cost introduced by carrier realization and conditioning of the proxy backbone model. We will promote the per-task results to a main-text table and revise the abstract so that its intelligibility wording matches these measurements.

The raw research carrier has no valid transcription score because it is 16 kHz band-limited analysis audio that causes the transcriber to fail. We will report it as n/a and will not use it as intelligibility evidence.

W3. The title refers to generative retrieval, but the paper does not clearly identify where retrieval occurs or distinguish codebook addressing from nearest-entry speaker lookup in a speaker database

Retrieval refers to quantization to frozen RVQ codebook entries. The current system does not perform nearest-entry speaker lookup in a speaker database. The underlying operation is vector quantization, and the term generative retrieval describes continuous profile generation followed by snapping to codebook entries.

This operation appears in both profile-formation paths:

  1. Reference recovery: a live reference is embedded by the frozen speaker encoder, corrected by the registrar, and quantized by the codec into an eight-code profile.
  2. Text-based design: the autoregressive generator produces continuous level vectors, which are snapped to frozen codebook entries to form an eight-code profile.

The two paths therefore produce the same discrete profile object. A profile encoded from real audio and a profile generated from a description share the same representation and can occupy the same registry-addressing structure.

We will define retrieval at first use, distinguish codebook addressing from speaker-database retrieval, and add a compact algorithm box to Section 3 that shows both profile-formation paths. The 99.3% nearest-entry result is only a registry-addressing diagnostic. It is not part of the current speaker-recovery path and does not measure exact all-eight code agreement.

W4. The evaluation does not appear to cover a sufficiently wide range of benchmarks, task families, languages, public baselines, and independently validated evaluation settings

The expanded evaluation covers 4,991 cases across four task families and 18 public baselines. The complete coverage is as follows:

Task familyBenchmark or evaluation setCompositionCounted cases
Cloningseed-tts-eval EnglishZero-shot cloning500
Cloningseed-tts-eval ChineseZero-shot cloning500
CloningELDERLY67Registered-domain elderly speakers67
Designhard2416 English joint-trait descriptions and 8 miscellaneous descriptions24
DesignBreezeBluePublic voice-design benchmark900
DesignParaSpeechCaps-styleStyle-prompted captions500
Designzh_tmplChinese design templates300
Instruction followingOfficial InstructTTSEval suiteInstruction-following evaluation1,200
EditingFive-operation editing suite200 speakers across three domains and five operations1,000
Total4,991

The four design sources contain 1,724 prompts. Cloning contributes 1,067 counted cases, instruction following contributes 1,200, and editing contributes 1,000. ELDERLY67 contains 67 speakers and 134 distinct reference and held-out recordings, with one case counted per speaker in the total above.

The evaluation also includes targeted comparisons for the mechanisms questioned in the reviews. VoiceSculptor has been evaluated within its supported Chinese voice-design setting on the shared 300-case subset:

SystemGenderPitchRateAge
VoiceSculptor.803.507.347.403
GRAPE-TTS.910.857.793.733

Independent WavLM speaker-verifier re-scoring is described in W2.2. A MOS/SMOS listening study is prepared and will be reported. We will consolidate the benchmark sources, case counts, baseline coverage, and task definitions into a central evaluation table so that the scale and scope of the evaluation are visible in the revision.

Reviewer_E5reOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes GRAPE-TTS, which represents a voice as an "acoustic profile": eight residual-VQ identifiers (2,048 entries each) obtained by quantizing a frozen 192-d ECAPA-TDNN speaker embedding. Levels 0-3 receive auxiliary gender/age/pitch/rate classification losses, and a symmetric-KL term smooths code assignments across recordings of the same speaker. Three operations act on this profile: (i) voice design, where a Qwen3-0.6B-conditioned autoregressive transformer with a per-level flow-matching head generates eight continuous vectors that are snapped to the nearest codebook entries ("generative retrieval"); (ii) cloning, where a residual MLP "registrar" corrects the averaged reference embedding before quantization; and (iii) editing, where a classifier replaces one designated identifier, filtered by identity gates. A low-rank additive "assembler" maps the profile to PCA coefficients that are inverted into a fixed-length (768-frame) mel "carrier". The carrier is fed as the acoustic prompt to a frozen ZipVoice together with a fixed prompt string. Pitch and rate are controlled by shifting or time-scaling this carrier. Experiments report voice-design accuracy with six-candidate selection (Table 1), InstructTTSEval compatibility under a local Qwen2.5-Omni-7B judge, ECAPA-SIM cloning on ELDERLY67/Seed-zh/Seed-en (Table 2), pitch/rate/age editing (Table 3), and two ablations (Table 4).

Critical strengths
  1. A clear problem statement with a concrete, inspectable object. The paper asks a fair question: can design, cloning, and editing share one explicit voice state, instead of each task hiding the voice in its own conditioning path? Turning the voice into eight identifiers gives an object that can be stored, compared, and searched. Section 2.4 and Eq. 13 also state clearly that additive assembly gives locality only in coefficient space, not acoustic disentanglement. This framing is more careful than in many controllable-TTS papers.
  2. Candid reporting of some negative or mixed results. The paper reports the single-candidate numbers (Table 4a), the clear Seed-zh/Seed-en similarity loss against its own backbone (0.750 to 0.611 and 0.666 to 0.561, Table 2), and the low age-edit hit rate (29.5%). It also keeps carrier-level pitch/rate control separate from code replacement in the text. These disclosures make the work easier to assess and are the main reason I see value in the direction.
Critical weaknesses
  1. Limited novelty: the system is an assembly of known parts, and the closest prior work is not discussed.** Each component is established: speaker-verification embeddings for multi-speaker TTS (Jia et al., 2018), residual VQ (SoundStream/EnCodec), auxiliary supervision on chosen RVQ levels (SpeechTokenizer), RQ semantic IDs (TIGER), continuous autoregression with a flow-matching head, a residual MLP for embedding correction, and a frozen zero-shot TTS backbone. The paper itself notes that the trait labels "do not establish complete acoustic disentanglement" (Sec. 2.4), so the claimed addressability is mostly a naming of RVQ levels. "Generative retrieval" here means generating a continuous vector and snapping it to the nearest codebook entry. That is ordinary vector quantization; no voice is retrieved from a store (App. A.3 says as much). More importantly, generating new speakers in a speaker-embedding space, with or without text descriptions, and editing voices in such a space is an established line of work, e.g., TacoSpawn (speaker generation, ICASSP 2022), PromptSpeaker (ASRU 2023), PromptTTS++ (ICASSP 2024), and VoxGenesis (2024). None is cited or compared. Spark-TTS is cited, but its fixed-length global tokens are already a compact, storable, attribute-controllable voice state, so the distinction drawn in Sec. 1 is not convincing. The paper never shows that the discrete, trait-organized profile beats the obvious simpler baseline: generate or store the continuous 192-d ECAPA vector (equally persistent, 768 bytes in float32) and edit it along attribute directions. Without that comparison, the core contribution is unclear.
  2. The headline numbers come mainly from selection, carrier post-processing, and a self-referential ECAPA metric, not from the profile. With one candidate, GRAPE is below Spark-TTS on pitch (57.0 vs. 79.3) and rate (65.6 vs. 71.7). The Table 1 gains appear only after six-candidate selection, scored by the same probes that pick the winner (Table 4a, App. B.2), while every baseline gets a single sample. Rate looks set by construction: ZipVoice derives output duration from prompt duration and a text-length ratio, and GRAPE keeps the prompt text fixed while time-scaling the carrier. So the rate RDE of 0.001 (Table 3) and the extra temporal variants used in design mostly reflect that rule, not profile control. Identity results are also circular. The same ECAPA model defines the profile, is a training loss for the assembler (L_sim,car and L_sim,flow, Eq. 37), gates the edits, and is the only similarity metric. Even with this advantage, GRAPE loses about 0.14 SIM to its own backbone on Seed-zh (Table 2). An independent verifier, a budget-matched baseline, and a listening test are needed.
  3. The paper's own evidence contradicts the selective-editing and persistence claims. With the identity gates disabled, replacing only the level-1 "age" code drops id_self to 0.0994 (Table 4b), so the speaker effectively changes. This fits Fig. 5: levels 0 and 1, nominally gender and age, have perplexities of about 1,341 and 235, so they carry most of the identity. The selectivity therefore comes from gates that reject or soften edits, which explains the low 29.5% hit rate on a single target class. That rate is not significantly different from Step-Audio-EditX (23.5%, n = 200). The id_self = 0.885 in Table 3 is also measured against GRAPE's own re-rendered voice and compared with baselines evaluated on different operations. Persistence, the paper's title claim, is never measured: it reports no rate at which two recordings of a speaker yield the same profile. The only related number, "complete agreement of 0.0098" (App. E.1), is undefined and, if it means cross-recording agreement, contradicts the claim.
  4. The abstract and contributions say GRAPE "preserves the naturalness and intelligibility of the backbone" and "achieves high naturalness through the voice conversion". CER is defined in App. B.1 but never reported for any task. UTMOS is given only for cloning. There is no voice-conversion experiment and no MOS, SMOS, or other human evaluation anywhere.
Questions for authors
  1. Do you have any result comparing the RVQ profile with a continuous baseline that stores or generates the ECAPA vector directly and edits it along learned attribute directions? If not, what evidence supports the claim that discrete, trait-organized codes are needed?
  2. For held-out speakers with several recordings, how often do all eight codes match, and how often does each level match? What exactly are the "matching values" 0.2426/0.2410 and the "complete agreement" 0.0098 in App. E.1?
  3. Do you override ZipVoice's default prompt-to-output duration rule? If not, please confirm that rate RDE and the rate gains from temporal variants follow directly from carrier time-scaling. What is the design rate accuracy with temporal variants turned off?
  4. Can you report cloning and editing similarity with an encoder not used in training or gating (e.g., the WavLM verifier of Seed-TTS-eval), together with CER/WER for all tasks? How were the trait probes trained, and do they share data or labels with the codec's trait supervision?
  5. For each row, which edit operations and which cases are averaged into id_self? Is GRAPE's id_self computed against the re-rendered unedited profile or the source recording? What are the source-relative similarity and the pre-edit young-adult rate of the 200 speakers?
6 comments

We thank the reviewer for the detailed criticism and precise questions. The concerns about novelty, candidate selection, speaker similarity, editing, persistence, rate control, and intelligibility have helped us identify where the paper required clearer claims and additional evidence. We respond to each point below and report the corresponding expanded evaluations and planned revisions.

W1. The novelty is unclear because the method combines established components, uses ordinary vector quantization under the name generative retrieval, shares compact voice-state representations with methods such as Spark-TTS, and omits relevant prior work on speaker generation

Our contribution is the shared persistent profile and the operations defined over it. One 88-bit, trait-organized profile supports reference recovery from real audio, generation from text, and subsequent identifier replacement. The registrar aligns live-reference embeddings before quantization, and the full editing configuration applies identity gates after identifier replacement. We do not claim RVQ, speaker embeddings, or flow matching as individually novel components.

Addressability. The trait associations are supported by intervention evidence. Gender is level-locked to L0, and L1 carries the principal age response. Pitch and rate show weaker level locality, and the current system uses carrier realization for these controls where appropriate. We will present these differences explicitly and avoid implying complete acoustic disentanglement across all eight levels.

Generative retrieval. This term refers to generation followed by nearest-neighbor snapping to frozen RVQ codebook entries. It does not refer to nearest-neighbor retrieval of a speaker from a speaker database. We will define the term at first use and use codebook addressing where that wording is clearer.

Spark-TTS. Its global tokens already provide a compact voice state, and we will acknowledge this explicitly. The distinction studied in our paper is functional. GRAPE-TTS organizes the stored profile into trait-associated levels and uses the same object for reference recovery, text-based design, and identifier replacement. The revised comparison will focus on these operational differences without implying that compact voice tokens are inherently unable to influence voice attributes.

Missing prior work. We will cite TacoSpawn, PromptSpeaker, PromptTTS++, and VoxGenesis. The revised related-work section will compare these methods by voice-state representation, profile creation, storage, and supported editing operations. The continuous-representation comparison requested by the reviewer is addressed directly in Q1.

W2. The headline improvements may arise primarily from candidate selection and carrier post-processing, while the reliance on ECAPA for speaker-similarity evaluation raises concerns about self-referential evidence and insufficient independent validation

Candidate selection

Candidate selection occurs during profile creation using a fixed and disclosed rule. We report both N=1N=1 and N=6N=6 so that the contribution of selection can be measured directly.

SystemCandidatesGenderPitchRateAge
GRAPE-TTS1.932.570.656.560
GRAPE-TTS6.966.829.873.776
Qwen3-TTS-vd1.958.541.620.369
Qwen3-TTS-vd6.992.785.804.613

The matched-N=6N=6 comparison gives GRAPE-TTS and Qwen3-TTS-vd the same candidate budget and the same frozen selection probes. GRAPE-TTS maintains higher pitch, rate, and age accuracy under this controlled comparison. Spark-TTS with six candidates and structured ground-truth trait labels leads on gender, pitch, and rate, while GRAPE-TTS retains a 10.8-point age advantage. We treat Spark-TTS as a complementary structured-conditioning reference because it does not consume the free-text design descriptions used by GRAPE-TTS.

An independent Qwen2.5-Omni-7B evaluation provides an additional result outside the selection probes:

SystemDSDRP
GRAPE-TTS91.785.3
Qwen3-TTS-vd85.369.3

Rate realization

Rate-edit RDE and design-time rate accuracy measure different claims. The editing RDE is exact by construction because the operation applies a mel-domain time-axis transformation. Design-time rate accuracy also includes profile generation and candidate selection. Disabling only the temporal variants reduces design rate accuracy from .873 to .711, which remains above Qwen3-TTS-vd at one candidate, with .620. The complete controlled comparison is given in Q3.

Expanded speaker-similarity evaluation

ECAPA is no longer the only speaker verifier used to evaluate the similarity conclusions. In an expanded evaluation covering every cloning and editing result, we re-scored the outputs with two encoders that were not used in training, candidate selection, or identity gating:

  • The official seed-tts-eval WavLM-large speaker verifier
  • microsoft/wavlm-base-plus-sv

For Seed-en and Seed-zh cloning, system rankings correlate with the ECAPA rankings at Spearman .86 to .93. The age-editing identity comparison also remains consistent across all three encoders:

SystemECAPAseed-tts-eval WavLM-largewavlm-base-plus-sv
GRAPE-TTS.885.942.963
StepAudioEditX.702.661.920

These expanded results show that the principal similarity rankings persist across independently trained encoders. The full scores will be included in the revision.

W3. The gates-off identity collapse, low age-hit rate, inconsistent idselfid_{\text{self}} comparisons, and undefined 0.0098 agreement statistic weaken the paper's claims about selective editing and persistent profiles

The gates-off result is an ablation of the complete editing configuration. L1 identifier replacement retains embedding cosine similarity of .73 to .79, yet rendering the modified profile with both identity gates disabled reduces idselfid_{\text{self}} to .099. This result shows that embedding-space proximity alone is insufficient to guarantee speaker identity after decoding and synthesis through the proxy backbone model.

The full age-editing configuration therefore includes identifier replacement, a decoded-space identity gate, a rendered-space identity gate, and a fixed fallback. Both gates use fixed global constants shared across all test cases. Figure 5 and the gates-off result also indicate that early RVQ levels carry identity information. We will narrow the text to the acoustic selectivity measured by the experiments and present both gates as explicit parts of the editing method.

The absolute age-hit rate remains limited, and we will report it as such. On the same 200-speaker subset and under the same scorer, the comparison is:

SystemAge hitidselfid_{\text{self}}
GRAPE-TTS.295.885
StepAudioEditX.235.717

The six-point age-hit difference is descriptive at this sample size. The larger difference in own-render identity shows that the complete GRAPE-TTS age operation better preserves the identity of its unedited base render in this evaluation. The mean predicted P(young-adult)P(\text{young-adult}) over the 200 eligible speakers' base renders is .218. DSP does not provide the corresponding age-editing operation.

Exact code agreement and profile recoverability are separate measurements. The definitions and all eight per-level results are given in Q2. The operation sets and comparison targets behind each editing identity result are given in Q5.

W4. The abstract claims preserved naturalness and intelligibility even though the submission reports no basic WER or CER results, no MOS or SMOS evaluation, and no standalone voice-conversion experiment

We agree that the abstract's naturalness and intelligibility statement required stronger supporting evidence. We have expanded the intelligibility evaluation by measuring WER and CER after synthesis through the proxy backbone model.

Evaluation setMetricGRAPE-TTSBaseline range
Seed-enWER3.1%1.8% to 26.3%
Seed-zhCER4.9%4.5% to 269.9%

The measured WER and CER values lie within the corresponding baseline ranges and show that the shared profile and synthesis through the proxy backbone model retain usable intelligibility in these evaluations. The raw research carrier is 16 kHz analysis audio and cannot be scored reliably by the transcriber. We will mark it as n/a and will not use it as intelligibility evidence.

A per-task WER and CER table will be added to the main text. UTMOS also provides an automatic quality measurement, with GRAPE-TTS at 3.184 and ZipVoice at 3.222. WER, CER, and UTMOS do not measure human-rated naturalness. A MOS/SMOS study is prepared and will be reported. Until those results are included, the abstract will be qualified to match the transcription and automatic quality evidence.

The submission does not contain a standalone voice-conversion experiment. We will ensure that the task descriptions and abstract claims cover only the design, cloning, reference-recovery, and editing evaluations actually presented.

Reviewer_DaGUOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes GRAPE-TTS, which compresses a given voice to an acoustic profile of 8 discrete codes and uses the same profile in a unified manner for voice design (from a text description), voice cloning (from a reference clip) and voice editing. The profile is a frozen speaker embedding quantized with an 8-codebook residual VQ; first 4 also trained to predict gender, age, etc. For voice design, given a text description, a flow-matching model generates six candidate profiles, scores them using attribute classifiers and keeps the best one (a best-of-6 selection). For editing, age is changed by swapping the age code, and pitch and rate by directly modifying mel-spectrograms. To generate speech, an assembler model turns profile codes into a mel-spectrogram of a fixed sentence (the "carrier"), which then serves as the voice prompt for a frozen ZipVoice TTS model. Its voice design and instruction-following scores are the best among the compared systems, but the comparison is unfair because GRAPE-TTS uses the same classifiers for best-of-6 selection as the ones used to score the results, while each baseline gets a single output. With one candidate, it lags behind Spark-TTS on pitch and rate accuracy. In cloning, it has the best speaker similarity on an elderly-speaker test set but is worse than its ZipVoice backbone on the standard Seed-zh and Seed-en sets. In editing, it has the highest identity retention and a 29.5% age-edit success rate. There are no listening tests, and no CER or WER is reported even though the abstract says intelligibility is preserved.

Critical strengths
  1. The problem is well-motivated. The paper separates the applications (design, cloning, editing) from the voice representation itself, and argues that the representation should be an explicit object that can be saved and reused. The approach is quite simple and practical. The profile is just eight integers that don't depend on the text, all three operations share them, and they are able to condition an off-the-shelf TTS model that is never retrained. As far as I could find, quantizing a speaker embedding with RVQ codes supervised by attribute labels is novel.
Critical weaknesses
  1. The main results compare against baselines unfairly. from an unfair comparison. GRAPE-TTS uses best-of-n selection (for voice design, using the same classifiers used to score at evaluation time!) while each baseline doesn't get such selection.
    1. For voice design, much of the improvement is due to this unfair best-of-n-selection step. If only one candidate is allowed (Table 4a), GRAPE-TTS is 22.3 points behind Spark-TTS on pitch accuracy and 6.1 points behind on speaking rate accuracy (Table 1).
    2. For editing, the gains come from a filtering step (the "identity gates"). With those off, the age edit almost never succeeds (0.5% of cases) and the output no longer resembles the original speaker (Table 4b). There is also some inconsistency; GRAPE-TTS's identity retention is measured on age edits, but FireRedTTS-3 and the signal-processing reference have no age results, so theirs probably comes from pitch or rate edits?
  2. Whether their profile is a better unified representation for design, cloning and editing is not compared against the obvious alternatives. Closely related prior work is missing.
    1. The discrete profile is never compared with simpler options, such as the continuous speaker embedding itself without their quantization, or a cascade of existing systems. For example, the Qwen3-TTS README says one can save a designed clip as a cloning prompt for a consistent character voice, yet such a simple baseline is not reported! In cloning, GRAPE-TTS is below its own ZipVoice backbone in speaker similarity on Seed-zh and Seed-en. It only succeeds on cloning on ELDERLY67, a test set the paper never describes, and Appendix D.2 even mentions specific training on elderly-speech corpora.
    2. Closely related prior work is not cited: PromptTTS++ (https://arxiv.org/abs/2309.08140) and PromptSpeaker (https://arxiv.org/abs/2310.05001) which generate representations from text descriptions; VoiceShop (https://arxiv.org/abs/2404.06674) which edits a speaker's age and gender, and the closest work, VoiceSculptor (https://arxiv.org/abs/2601.10629), which designs a voice from a description, can edit individual attributes, and renders the result "into a prompt waveform" for a cloning model, much like GRAPE-TTS uses its carrier to prompt ZipVoice. It should also be a baseline.
  3. The paper is very hard to read. The method section uses many new names like "registrar", "assembler", "carrier", and "generative retrieval" and gives many equations with little explanation.. The parts that matter for the results are never explained, such as the identity gates and the two generator outputs used for candidate selection. The design test sets (hard24, breezeblue, zh_tmpl) are also never described.
Questions for authors
  1. How did each baseline take the voice description as input? For example, Spark-TTS's released code only takes gender. pitch and speed. How was it run on the descriptive and role-play instructions?
  2. What are the hard24, breezeblue, zh_tmpl and ELDERLY67 test sets? Do any ELDERLY67 speakers also appear in the training elderly-speech corpora?
7 comments

We thank the reviewer for the detailed assessment. The concerns about candidate selection, identity during editing, alternative profile representations, writing, and evaluation coverage are well taken. We have expanded the evaluation and reorganized the response below around the original weaknesses and questions. In particular, the expanded evaluation makes the fairness of the baseline comparison explicit through both native-system evaluation and matched-budget controls.

W1. Validity of claims and experimental design

W1.1 The fairness of the main design comparison is unclear because GRAPE-TTS uses multi-candidate generation and trait-based selection, while the baselines are not evaluated with the same candidate budget and selection procedure

The original Table 1 is a fair system-level comparison because every method is evaluated through its native voice-conditioning path and official inference pipeline. Each baseline receives the input format and generation procedure for which it was designed. For GRAPE-TTS, the model generates all NN candidates end-to-end, and a fixed selection rule uses frozen trait probes to choose the output. No human or external reranker is involved. The table therefore evaluates the complete systems, including their respective generation and selection procedures.

To complement this system-level comparison with a controlled comparison, our expanded evaluation includes a matched-N=6N=6 comparison that gives GRAPE-TTS and Qwen3-TTS-vd the same candidate budget and the same frozen trait probes for selection. The two comparisons answer complementary questions: the first compares complete systems under their intended operating configurations, while the second tests whether GRAPE-TTS retains its advantages after candidate generation and selection are equalized.

First, Table 4a isolates the selection stage under the same autoregressive generator. Increasing the candidate budget improves all four trait metrics for GRAPE-TTS.

SystemCandidatesGenderPitchRateAge
GRAPE-TTS1.932.570.656.560
GRAPE-TTS6.966.829.873.776
Qwen3-TTS-vd1.958.541.620.369
Qwen3-TTS-vd6.992.785.804.613

The N=1N=1 result is therefore the mechanism ablation. It also provides a lower-compute operating point for generating a reusable profile.

Second, the matched-N=6N=6 control makes the candidate-generation and selection conditions equivalent for GRAPE-TTS and Qwen3-TTS-vd. Both systems receive six candidates and one selected output under the same frozen selection probes. GRAPE-TTS maintains higher pitch, rate, and age accuracy under this controlled setting. Together with the native-system evaluation, this provides a fair comparison at both the complete-system level and the matched-selection level.

Spark-TTS provides a complementary reference under its native structured-label input format. At six candidates, it leads on gender, pitch, and rate, while GRAPE-TTS retains a 10.8-point advantage on age. This comparison benefits Spark-TTS by supplying ground-truth trait labels through the pipeline for which it was designed.

Third, the expanded evaluation includes an independent judge using Qwen2.5-Omni-7B. This control changes the evaluation scorer, which is separate from matching the candidate budget.

SystemDSDRP
GRAPE-TTS91.785.3
Qwen3-TTS-vd85.369.3

These results provide an additional evaluation view under a judge that is independent of the trait probes used for selection.

Candidate generation and selection occur during profile creation. Once selected, the resulting 88-bit profile can be stored and reused without repeating this search for every synthesized utterance. We will revise the experimental section to distinguish clearly between the native-system comparison, the single-candidate generator ablation, the matched-N=6N=6 control, and the independent-scorer evaluation.

W1.2 The editing results do not sufficiently establish identity preservation because the roles of the identity gates, the choice of unedited comparison target, and the similarity to the real source recording are not clearly distinguished

The full age-editing configuration comprises identifier replacement followed by decoded-space and rendered-space identity gates. Section 3.3 under-specified this procedure, and we will expand it into an explicit algorithm. Table 4b evaluates this configuration by disabling both identity gates.

Age-editing configurationAge hitidselfid_{\text{self}}
Full operator.295.885
Identity gates disabled.005.099

The rendered-space gate checks the synthesized voice because agreement between profile embeddings does not guarantee identity preservation after carrier realization and speech synthesis. The gates-off result shows the practical importance of this check. The revision will also present both identity gates as explicit components of the editing configuration, including their thresholds and fallback behavior.

We further separate two quantities that the original draft did not distinguish sufficiently: idselfid_{\text{self}} compares an edited render with the system's own unedited base render under the same assembler and proxy backbone model. It measures identity change introduced by editing. idsrcid_{\text{src}} compares the edited render with the real source recording. It additionally includes representation, carrier-realization, and synthesis losses.

Expanded own-render identity evaluation

MethodCompared operationsMean idselfid_{\text{self}}Evaluation size
GRAPE-TTSPitch-up/down, rate-up/down, age.7971,000
StepAudioEditXAge on the elderly-overlap subset with base renders.70167
FireRedTTS-3Pitch/rate.451774
DSPPitch/rate.684800

For StepAudioEditX, source-relative pooling covers rate and age over 600 cases. Across the evaluated baselines, idsrcid_{\text{src}} ranges from .664 to .717, compared with .290 for GRAPE-TTS. These two measurements answer different questions. GRAPE-TTS maintains identity well relative to its own base render, while its source-relative score exposes substantial information loss between the real source recording and the reconstructed voice.

On the same 200-speaker age subset and under the same scorer, the comparison is:

SystemAge hitidselfid_{\text{self}}
GRAPE-TTS.295.885
StepAudioEditX.235.717

The age-hit difference is descriptive at this sample size. The larger identity gap provides evidence that the complete GRAPE-TTS age operator preserves its base-render identity effectively in this evaluation. The revision will report both own-render and source-relative identity so that editing consistency and source fidelity remain clearly separated.

W2. Comparison with alternative representations

W2.1 The paper does not sufficiently compare the discrete RVQ profile with reusable continuous representations or saved waveform prompts, and the cloning results also raise concerns about fidelity to the source speaker

Our expanded representation evaluation includes two relevant comparisons. The first reuses a generated waveform as a cloning prompt. The second replaces hard RVQ addressing with soft codebook addressing while keeping the autoregressive generator, assembler, candidate selection, and evaluation scorer fixed.

ConfigurationGenderPitchRateAge
Qwen3-TTS-vd design-to-clip-to-clone cascade.968.552.484.329
Soft codebook addressing.817.769.776.692
Hard RVQ addressing.966.829.873.776

The cascade retains gender information effectively, while pitch, rate, and especially age degrade after the designed voice is reused as a waveform prompt. This result quantifies attribute-control loss through waveform-prompt reuse. Our stored RVQ profile additionally provides explicit levels for subsequent identifier replacement.

The soft-addressing comparison isolates hard codebook assignment within the current architecture. Hard assignment improves all four trait metrics. This supports the usefulness of discrete addressing in our design, without by itself establishing that discreteness is necessary for every persistent voice representation.

A direct baseline that generates, stores, and edits the raw 192-dimensional ECAPA embedding remains an appropriate additional control. We will include it in the revision to evaluate the RVQ representation against direct continuous-speaker-embedding storage.

Cloning quality and backbone retention

The purpose of the cloning experiment is to test whether a shared persistent profile can condition the frozen ZipVoice backbone while retaining its general synthesis capability. The results support substantial retention at the macro level, together with a clear resolution limitation on the more demanding Seed evaluation.

MethodELDERLY67 SIMSeed-zh SIMSeed-en SIMMacro SIMUTMOS
GRAPE-TTS.580.611.561.5843.184
ZipVoice.409.750.666.6083.222

GRAPE-TTS improves registered-domain ELDERLY67 cloning while remaining close to the frozen backbone in Macro SIM and UTMOS. The larger Seed-specific gaps show that the shared representation does not preserve all source-speaker detail. Discrete RVQ encoding and the PCA carrier realization restrict the acoustic resolution available to the cloning backbone. We will revise the discussion to present this tradeoff explicitly and avoid framing the method as a general replacement for dedicated zero-shot cloning systems.

W2.2 The related-work discussion and empirical comparisons do not adequately cover recent voice-design and voice-editing methods, including PromptTTS++, PromptSpeaker, VoiceShop, and VoiceSculptor

We will add PromptTTS++, PromptSpeaker, VoiceShop, and VoiceSculptor to the related-work discussion. The revised comparison will focus on how each method represents voice state, whether reference recovery and text-based design share that state, and where the editing intervention occurs.

VoiceSculptor is especially relevant because it connects design and editing to waveform-prompt-based cloning. We therefore evaluated it on the shared Chinese zh_tmpl subset using its supported Chinese conditioning format:

SystemGenderPitchRateAge
VoiceSculptor.803.507.347.403
GRAPE-TTS.910.857.793.733

This provides a direct empirical comparison on the 300 Chinese design cases. The revised related-work section will combine this result with a method-level comparison of waveform-prompt conditioning and trait-supervised RVQ profiles.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.