- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces a concept of "voice profile", which is the speaker embedding output from ECAPA-TDNN, with further revisions. Specifically, the authors trained a VQ-VAR model to decompose the underlying semantics of the speaker embeddings, with additional supervision, such as age, gender, pitch, etc.
Critical strengths
(1) We problem formulation makes sense: multiple speech generation tasks should share some common input information regardless of the specific task. (2) Experimentally, the paper provides sufficient baselines, together with sufficient details in the appendix.
Critical weaknesses
(1) Using a time-invariant speaker embedding as the "voice profile" is not very solid to me. There should be quite a lot of dynamics for the speaking style from the speaker that can hardly be represented by a speaker embedding. (2) I have some concerns about the soundness of the experimental results. (2.1) In Table 2, the proposed method doesn't show a clear strength over the baselines. (2.2) In Table 1, the metric to compare is exactly the metric to select the candidate, which is clearly a leakage. (2.3) For a voice generation paper, the paper doesn't disclose any basic intelligibility number, aka, WER/CER of the generated content. (3) The paper title mentions retrieval, but it's not very clear to me where the retrieval happens; (4) The proposed model is not sufficiently tested on a wide range of benchmarks.
Questions for authors
As in weakness.
3 comments
W2.2. Table 1 uses the same trait measurements for candidate selection and system evaluation, creating a dependent evaluation that requires matched candidate budgets and independent judges or encoders
The trait probes are trained on registry and synthetic entries that share no items with the evaluation sets. The main concern is therefore the dependence created by using the same trait measurements for selection and reporting. We address this dependence through matched candidate budgets, an independent judge, and independent speaker encoders.
First, the contribution of selection is directly disclosed. The autoregressive generator produces candidates, and a fixed disclosed rule selects one profile. The and comparison isolates this contribution.
| System | Candidates | Gender | Pitch | Rate | Age |
|---|---|---|---|---|---|
| GRAPE-TTS | 1 | .932 | .570 | .656 | .560 |
| GRAPE-TTS | 6 | .966 | .829 | .873 | .776 |
| Qwen3-TTS-vd | 1 | .958 | .541 | .620 | .369 |
| Qwen3-TTS-vd | 6 | .992 | .785 | .804 | .613 |
The matched- comparison gives GRAPE-TTS and Qwen3-TTS-vd the same candidate budget and the same frozen selection probes. GRAPE-TTS maintains higher pitch, rate, and age accuracy under this controlled comparison. Spark-TTS with six candidates and structured ground-truth trait labels leads on gender, pitch, and rate, while GRAPE-TTS retains a 10.8-point age advantage. Spark-TTS serves as a structured-conditioning reference because it does not consume the free-text design descriptions used by GRAPE-TTS.
Second, the expanded evaluation includes an independent Qwen2.5-Omni-7B judge under the official InstructTTSEval evaluation setup.
| System | DSD | RP |
|---|---|---|
| GRAPE-TTS | 91.7 | 85.3 |
| Qwen3-TTS-vd | 85.3 | 69.3 |
This judge provides trait and instruction-following evidence outside the probes used for candidate selection.
Third, the expanded similarity evaluation re-scores the cloning and editing outputs with two WavLM encoders that were not used in training, selection, or identity gating. Seed-en and Seed-zh cloning rankings correlate with the ECAPA rankings at Spearman .86 to .93. Together, these controls separate the effects of candidate budget, selection metrics, and independent evaluation.
W2.3. The submission does not disclose basic WER or CER results for a voice-generation system, making it difficult to assess whether profile conditioning and proxy-backbone synthesis preserve intelligibility
We agree that a voice-generation paper should report intelligibility. Our expanded evaluation adds the following results after synthesis through the proxy backbone model:
| Evaluation set | Metric | GRAPE-TTS | Baseline range |
|---|---|---|---|
| Seed-en | WER | 3.1% | 1.8% to 26.3% |
| Seed-zh | CER | 4.9% | 4.5% to 269.9% |
The expanded benchmark table also includes per-system WER results for all 15 cloning baselines. GRAPE-TTS lies within the corresponding baseline ranges on both language evaluations.
The original submission omitted these results because final intelligibility is primarily determined by two system factors. The first is the frozen backbone's own synthesis capability. The second is how faithfully the carrier conditions the proxy backbone model, which is a limitation of our current realization. These measurements therefore characterize the complete synthesis system more directly than the profile mechanism alone.
This rationale explains the original omission, and we agree that WER and CER are still essential system-level metrics. The expanded results quantify the actual intelligibility of the complete system, including the cost introduced by carrier realization and conditioning of the proxy backbone model. We will promote the per-task results to a main-text table and revise the abstract so that its intelligibility wording matches these measurements.
The raw research carrier has no valid transcription score because it is 16 kHz band-limited analysis audio that causes the transcriber to fail. We will report it as n/a and will not use it as intelligibility evidence.
W3. The title refers to generative retrieval, but the paper does not clearly identify where retrieval occurs or distinguish codebook addressing from nearest-entry speaker lookup in a speaker database
Retrieval refers to quantization to frozen RVQ codebook entries. The current system does not perform nearest-entry speaker lookup in a speaker database. The underlying operation is vector quantization, and the term generative retrieval describes continuous profile generation followed by snapping to codebook entries.
This operation appears in both profile-formation paths:
- Reference recovery: a live reference is embedded by the frozen speaker encoder, corrected by the registrar, and quantized by the codec into an eight-code profile.
- Text-based design: the autoregressive generator produces continuous level vectors, which are snapped to frozen codebook entries to form an eight-code profile.
The two paths therefore produce the same discrete profile object. A profile encoded from real audio and a profile generated from a description share the same representation and can occupy the same registry-addressing structure.
We will define retrieval at first use, distinguish codebook addressing from speaker-database retrieval, and add a compact algorithm box to Section 3 that shows both profile-formation paths. The 99.3% nearest-entry result is only a registry-addressing diagnostic. It is not part of the current speaker-recovery path and does not measure exact all-eight code agreement.
W4. The evaluation does not appear to cover a sufficiently wide range of benchmarks, task families, languages, public baselines, and independently validated evaluation settings
The expanded evaluation covers 4,991 cases across four task families and 18 public baselines. The complete coverage is as follows:
| Task family | Benchmark or evaluation set | Composition | Counted cases |
|---|---|---|---|
| Cloning | seed-tts-eval English | Zero-shot cloning | 500 |
| Cloning | seed-tts-eval Chinese | Zero-shot cloning | 500 |
| Cloning | ELDERLY67 | Registered-domain elderly speakers | 67 |
| Design | hard24 | 16 English joint-trait descriptions and 8 miscellaneous descriptions | 24 |
| Design | BreezeBlue | Public voice-design benchmark | 900 |
| Design | ParaSpeechCaps-style | Style-prompted captions | 500 |
| Design | zh_tmpl | Chinese design templates | 300 |
| Instruction following | Official InstructTTSEval suite | Instruction-following evaluation | 1,200 |
| Editing | Five-operation editing suite | 200 speakers across three domains and five operations | 1,000 |
| Total | 4,991 |
The four design sources contain 1,724 prompts. Cloning contributes 1,067 counted cases, instruction following contributes 1,200, and editing contributes 1,000. ELDERLY67 contains 67 speakers and 134 distinct reference and held-out recordings, with one case counted per speaker in the total above.
The evaluation also includes targeted comparisons for the mechanisms questioned in the reviews. VoiceSculptor has been evaluated within its supported Chinese voice-design setting on the shared 300-case subset:
| System | Gender | Pitch | Rate | Age |
|---|---|---|---|---|
| VoiceSculptor | .803 | .507 | .347 | .403 |
| GRAPE-TTS | .910 | .857 | .793 | .733 |
Independent WavLM speaker-verifier re-scoring is described in W2.2. A MOS/SMOS listening study is prepared and will be reported. We will consolidate the benchmark sources, case counts, baseline coverage, and task definitions into a central evaluation table so that the scale and scope of the evaluation are visible in the revision.
We thank the reviewer for the focused criticism. The concerns about the profile's scope, empirical advantages, metric sharing, intelligibility, retrieval terminology, and benchmark coverage identify important limitations in the original presentation. We address each concern below with expanded evaluations and specific revisions.
W1. A persistent time-invariant acoustic profile cannot by itself represent utterance-level dynamics such as emotion trajectories, local prosodic variation, emphasis, and other fine-grained aspects of speaking style
We agree that a time-invariant embedding cannot encode an utterance-level style trajectory by itself. The profile is intentionally time-invariant, while the complete synthesis system contains time-varying stages. The profile is a persistent acoustic state that is independent of the target text. Trait supervision associates levels 0 to 3 with gender, age, pitch, and speaking rate. It does not store local prosodic dynamics.
The assembler maps the profile to a 64-dimensional PCA coefficient sequence, which inverse PCA converts into a mel carrier spanning approximately eight seconds. Content timing, emphasis, and local prosodic variation enter through the target text and the proxy backbone model during synthesis. Dynamic style therefore lies outside the profile's storage scope, while parts of its realization remain available in later stages of the system.
The independent instruction-following evaluation provides evidence for the intended design-text scope, with DSD of 91.7 and RP of 85.3. These measurements do not establish control over emotion arcs or detailed local prosody. We will state this boundary explicitly as a limitation and avoid implying that the persistent profile contains a complete dynamic style trajectory.
W2.1. Table 2 does not demonstrate a clear empirical advantage over the baselines, and the respective evaluation domains and source-speaker fidelity limitations of GRAPE-TTS require clearer interpretation
The purpose of Table 2 is to evaluate backbone-capability retention under shared-profile conditioning. GRAPE-TTS uses one profile space for design, reference recovery, and editing, with frozen ZipVoice serving as the proxy backbone model. The experiment tests whether this shared representation retains the backbone's general synthesis ability and where its representation resolution limits cloning fidelity.
GRAPE-TTS remains close to ZipVoice in Macro SIM and UTMOS, with differences of .024 and .038. On ELDERLY67, GRAPE-TTS obtains .580, compared with .409 for ZipVoice and .465 for the strongest original baseline. These results show that the shared profile can improve registered-domain elderly-speaker recovery while retaining much of the backbone's overall synthesis capability.
The Seed results expose the current limitation clearly. GRAPE-TTS trails ZipVoice by .139 on Seed-zh and .105 on Seed-en. Discrete RVQ encoding and the PCA carrier realization restrict the source-speaker detail available to the backbone. The fidelity of carrier realization and conditioning of the proxy backbone model also limit the final output. We will present Table 2 as evidence of this capability-retention and resolution tradeoff, with no claim that GRAPE-TTS replaces dedicated zero-shot cloning systems.
ELDERLY67 is a registered-domain, speaker-seen evaluation. Speaker overlap with the registry is intentional. All 134 evaluation recordings are excluded from training, and the reference and held-out recordings are distinct live phone recordings from the same speakers. Our zero-shot speaker evaluation relies on Seed-zh and Seed-en, where we report the remaining similarity gaps explicitly.