acceptodds
Withdrawn from ICLR 2027

Learn What to Say: Generalizable Active Enrollment for Few-Shot Voice Personalization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Few-shot voice personalization can reproduce a target speaker from only a few seconds of reference audio, yet users receive little guidance about what they should say. We formulate active voice enrollment as a budget-constrained sequential selection problem and introduce LEAP (Language-Transferable Enrollment via Active Policy Learning). LEAP predicts the marginal utility of each candidate utterance conditioned on the content already selected, enabling it to construct complementary enrollment sets without synthesizing speech or measuring downstream quality at inference time. We evaluate LEAP across Korean, Japanese, and Vietnamese, multiple voice-personalization models, matched recording budgets, and human listening studies. LEAP improves automatic personalization utility over random selection, natural user-selected enrollment, phoneme coverage, language-specific handcrafted objectives, and a learned static-ranking baseline. Crucially, when trained only on outcomes from two source languages, LEAP outperforms a handcrafted objective designed specifically for the held-out target language; its gains also persist for an unseen voice-personalization model. Human listeners prefer outputs generated from actively selected enrollment for speaker similarity, naturalness, and content clarity. These results show that enrollment utility contains transferable structure: when only a few seconds are available, performance depends not only on how much a user speaks, but also on what the system asks them to say. The selection summary and lexicon file for LEAP are available at https://anonymous.4open.science/r/Learn-What-to-Say-F86C.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Few-Shot Voice Personalization, Korean Enrollment Lexicon, Enrollment Selection, Budget-Constrained Linguistic Content Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_X4RYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LEAP, a method for selecting informative enrollment utterances for few-shot voice personalization under a limited recording budget. LEAP predicts the marginal utility of candidate utterances based on linguistic features and the content already selected, using a learned set-conditioned scoring function. The policy is trained offline using downstream personalization outcomes and operates without additional synthesis or quality evaluation at inference time. The method is evaluated on Korean, Japanese, and Vietnamese using three voice conversion models (Seed-VC, OpenVoice V2, and FreeVC) under recording budgets of 3 to 10 seconds. The experiments include comparisons with random selection, phoneme coverage, handcrafted linguistic objectives, and learned static ranking, as well as cross-language and cross-model transfer evaluations. The reported results show consistent improvements in normalized automatic metrics and moderate preferences for LEAP in human listening tests.

Critical strengths

Well-motivated problem setting

The paper addresses the practical problem of selecting enrollment content under limited recording budgets, rather than focusing solely on improving voice personalization models. This is a relevant and relatively underexplored direction.

Broad experimental evaluation

The experiments cover multiple languages, voice conversion models, recording budgets, and transfer settings. Human listening tests and comparisons with learned and handcrafted baselines provide useful evidence that enrollment content selection affects downstream personalization quality.

Critical weaknesses

Limited methodological novelty and lack of enrollment-time adaptation

Although LEAP is formulated as an active enrollment policy, selection relies entirely on predefined linguistic annotations and the remaining budget, without incorporating speaker-specific information or feedback from recorded audio. Consequently, a fixed candidate inventory and budget yield the same enrollment script for every speaker, and the entire selection process can be completed before recording begins. While the set-conditioned sequential selection is valid, the method is closer to learned budget-constrained subset selection than an enrollment policy that adapts to individual speakers. Its main components, including novelty and redundancy modeling, marginal utility prediction, and greedy selection, are also established techniques. The paper does not clearly establish a methodological distinction from existing subset-selection approaches.

Insufficient comparison with simpler set-aware methods and limited interpretability of improvements

Although LEAP outperforms learned static ranking and handcrafted coverage baselines, the evaluation lacks comparisons with simpler learned set-aware approaches, such as weighted coverage or submodular selection. It is therefore unclear whether the proposed architecture is necessary to obtain the reported gains. Furthermore, the main results report normalized composite and component scores without corresponding absolute speaker similarity, CER/WER, or naturalness values, making the practical magnitude of the improvements difficult to interpret. The method also depends on curated, language-specific linguistic annotations and prompt inventories, leaving the cost of extending it to new languages insufficiently examined.

Questions for authors
  1. Since LEAP does not use speaker-specific information or recorded audio, does it generate identical enrollment scripts for all speakers given the same inventory and budget? What advantage does sequential selection offer over offline script optimization?

  2. Can the authors compare LEAP with simpler learned set-aware methods, such as weighted coverage or submodular selection, using the same linguistic features and training supervision?

  3. Can the authors report raw speaker similarity, CER/WER, and NISQA scores? These would help clarify the practical significance of improvements reported using normalized metrics.

  4. How much manual effort is required to construct the language-specific annotations and candidate inventories? Does LEAP remain effective with automatically generated or less curated prompts?

8 comments
AuthorsOpenReview ↗

Response to Reviewer X4RY (Part 1)

We thank the reviewer for recognizing the practical relevance of enrollment-content selection and the breadth of the multilingual evaluation. We agree that the paper should more precisely distinguish transferable conditional script construction from speaker-adaptive enrollment, compare against stronger learned set-aware baselines, report raw metric magnitudes, and quantify the cost of constructing target-language resources.

Q1. Is LEAP an Enrollment Policy or an Offline Script Optimizer?

We agree that, for an identical language, candidate inventory, and recording budget, LEAP is deterministic and produces the same ordered script for every user. LEAP does not use speaker-specific audio or recording feedback at selection time; it should therefore not be interpreted as a speaker-adaptive enrollment policy. We will revise the framing accordingly. LEAP is an offline, transferable conditional script-construction method. It sequentially constructs a script for a supplied inventory and budget by re-scoring remaining candidates according to already selected linguistic evidence and remaining recording time. Its practical value is not per-speaker variation under fixed conditions, but its ability to construct a script for a changed prompt inventory, lexical realization, or recording budget without target-condition personalization outcomes. The submitted results already support this transfer capability. When recording budgets are held out during training and model selection, LEAP exceeds learned static ranking by +0.026 at 5 seconds and +0.016 at 10 seconds. For unseen lexical realizations, it remains ahead by +0.029 at 3 seconds and +0.019 at 10 seconds. We agree that learned static ranking is not a sufficient fixed-script comparator because it does not jointly optimize complementarity. We therefore add an offline jointly optimized fixed-script baseline for the primary in-language setting. For each language-inventory-budget condition, it selects one shared subset by maximizing mean downstream utility over training speakers and the same source voice-personalization models used for LEAP training. The resulting script is deployed unchanged to all held-out speakers and evaluation models. It has access to the same training-speaker outcomes, candidate inventory, and budget information as LEAP, but cannot be adapted to a new inventory or budget without collecting downstream outcomes for that new condition. The jointly optimized fixed script achieved 0.681 average utility across Korean, Japanese, Vietnamese, and all four budgets, compared with 0.694 for LEAP ((Δ=+0.013), paired 95% CI: [+0.007, +0.019]). At 3, 5, 7, and 10 seconds, it achieved 0.635, 0.674, 0.695, and 0.719, compared with LEAP's 0.652, 0.687, 0.709, and 0.727, respectively. Thus, LEAP's contribution is not speaker adaptation; it is transferable conditional utility learning that outperforms a shared script optimized on in-distribution training speakers while constructing scripts for changed inventories or budgets without target-condition outcomes.

AuthorsOpenReview ↗

Response to Reviewer X4RY (Part 2)

Q2. Does LEAP Outperform Simpler Learned Set-Aware Objectives?

We agree that comparisons with learned static ranking and handcrafted coverage alone do not isolate whether LEAP requires richer conditional utility modeling. The submitted ablations already show that removing the novelty and overlap features reduces average utility from 0.694 to 0.676, while set-independent scoring yields 0.669. LEAP also selects less redundant evidence than learned static ranking, with mean marginal overlap of 0.29 versus 0.46. However, these analyses do not fully exclude stronger learned set-aware objectives. We therefore add learned weighted coverage, facility-location selection, learned submodular coverage, and an additive conditional scorer. All methods use the same candidate inventories, linguistic descriptors, training-speaker utility supervision, recording budgets, and greedy feasibility rule. Learned weighted coverage learns evidence-family weights; facility location learns weighted representativeness; learned submodular coverage learns diminishing-return coverage weights; and the additive conditional scorer receives the same selected-set summary as LEAP but excludes candidate-set cross-feature interactions. Learned weighted coverage, facility-location selection, learned submodular coverage, and the additive conditional scorer achieved average utilities of 0.676, 0.678, 0.681, and 0.682, respectively, compared with 0.694 for full LEAP. LEAP therefore exceeded the strongest comparator, the additive conditional scorer, by +0.012 (paired 95% CI: [+0.003, +0.021]).

MethodAvg. utilityDifference from LEAP
Learned weighted coverage0.676−0.018
Facility-location selection0.678−0.016
Learned submodular coverage0.681−0.013
Additive conditional scorer0.682−0.012
LEAP0.694—

These match comparisons show that the improvement is not explained solely by learned evidence weights, conventional diminishing-return coverage, or additive state conditioning. We will accordingly limit the novelty claim to transferable nonlinear conditional utility learning, rather than greedy selection or redundancy reduction itself.

AuthorsOpenReview ↗

Response to Reviewer X4RY (Part 3)

Q3. What Is the Raw Practical Magnitude of the Automatic-Metric Improvements?

We agree that normalized scores facilitate aggregation across language-model conditions but can obscure absolute magnitudes. We therefore now report raw speaker similarity, content error, and NISQA-TTS scores alongside normalized utility. Speaker similarity is cosine similarity between frozen ECAPA-TDNN embeddings; content preservation is CER for the multilingual primary benchmark, with WER additionally reported where reliable tokenization is available; naturalness is the raw NISQA-TTS score. At the 3-second budget, averaged across the Korean, Japanese, and Vietnamese primary-language conditions and evaluated voice-conversion backends, LEAP improved mean ECAPA cosine similarity from 0.741 to 0.758, reduced CER from 9.8% to 8.7%, and improved NISQA-TTS from 3.42 to 3.55, relative to learned static ranking. At 10 seconds, it improved cosine similarity from 0.770 to 0.778, reduced CER from 6.9% to 6.4%, and improved NISQA-TTS from 3.67 to 3.73. The corresponding English-transfer WER decreased from 8.1% to 7.3%. These raw values show that the largest practical gains occur in the most constrained enrollment regime, consistent with the submitted normalized budget curves. We will provide per-language, per-budget raw metrics and paired confidence intervals in the appendix.

BudgetMethodECAPA cosine ↑CER ↓NISQA–TTS ↑
3 sStatic ranking0.7419.8%3.42
3 sLEAP0.7588.7%3.55
10 sStatic ranking0.7706.9%3.67
10 sLEAP0.7786.4%3.73
Reviewer_YdUgOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the selection of recording content for few-shot voice personalization under a limited recording budget. It proposes LEAP, a set-conditioned policy that predicts the marginal utility of candidate utterances based on their linguistic annotations, previously selected content, and the remaining budget. The policy learns from offline personalization outcomes but requires neither candidate audio nor downstream synthesis at selection time. Experiments cover Korean, Japanese, and Vietnamese, with additional transfer to English, multiple voice-personalization backends, and human listening evaluations. The paper addresses a meaningful problem and provides evidence that complementary enrollment content improves personalization. However, the practical advantage of the active-enrollment formulation over jointly optimized fixed scripts remains insufficiently established.

Critical strengths
  1. Meaningful research question. Improving reference recordings is a useful direction complementary to improving the personalization model itself. The paper investigates whether choosing what users say can improve quality without increasing their recording budget.
  2. Well-motivated modeling of complementarity. An utterance’s value depends on the evidence already present in the enrollment set. Modeling conditional marginal utility is therefore more appropriate than treating every utterance independently or maximizing phoneme coverage alone.
  3. Broad evaluation with perceptual support. The paper evaluates generalization across speakers, budgets, languages, lexical realizations, and downstream models. The human listening study provides evidence beyond automatic metrics: listeners prefer LEAP over learned static ranking in 59.7% of speaker-similarity comparisons.
Critical weaknesses
  1. The practical value of the active-enrollment formulation is not fully established. LEAP does not use speaker-specific audio or recording feedback at selection time. Its selected script could therefore be precomputed and reused for users sharing the same language, candidate inventory, and budget. It is unclear what additional benefit the proposed formulation offers over a fixed enrollment script jointly optimized on training speakers. The static-ranking baseline does not fully address this concern, because a fixed script can also account for complementarity among utterances. While transfer across languages and candidate inventories remains valuable, the current results more clearly establish the usefulness of content-aware script optimization than the necessity of an active enrollment policy.
  2. The demonstrated gains do not fully establish benefits in a real enrollment workflow. The experiments focus on controlled recordings and short estimated-duration budgets. In practice, reading difficulty, pauses, pronunciation errors, and re-recordings may affect user effort and recording quality. It would also be useful to understand whether a slightly longer, simpler recording could achieve comparable quality. These considerations limit how directly the reported gains translate into practical enrollment benefits.
  3. English evaluation is limited to an additional transfer setting. The paper includes automatic results for transfer to English, so English is not absent from the evaluation. However, it is not included in the primary in-language training and human listening studies. This leaves the evidence less complete for English, particularly regarding perceptual improvements and the comparison with an offline-optimized enrollment script.
Questions for authors
  1. Under identical candidate inventories and budgets, does LEAP select the same script for every user? If so, should the method be understood primarily as an offline, transferable script optimizer rather than a user-adaptive enrollment policy?
  2. Why is English included only as an additional held-out transfer target rather than a primary evaluation language? Are in-language training results or human listening judgments available to complement the automatic transfer results?
  3. Can the existing budget curves quantify how much additional recording time a simpler enrollment strategy needs to match LEAP’s quality? This would help assess the practical magnitude of the improvement.
4 comments
AuthorsOpenReview ↗

Response to Reviewer YdUg (Part 1)

We thank the reviewer for recognizing the practical importance of enrollment content selection, the role of set-conditioned complementarity, and the breadth of our multilingual evaluation. We agree that the paper should more precisely distinguish transferable conditional script construction from speaker-adaptive enrollment, and should more directly quantify real-workflow robustness and the value of the time saved by better content selection.

Q1. Is LEAP a User-Adaptive Policy or a Transferable Script Optimizer?

We agree that, for an identical language, candidate inventory, and recording budget, LEAP is deterministic and selects the same ordered script for every user. LEAP does not observe speaker-specific audio or recording feedback at selection time; it should therefore not be interpreted as a user-adaptive policy. We will revise the framing accordingly. LEAP is an offline, transferable conditional script-construction policy: it sequentially constructs a script for a supplied candidate inventory and budget by re-scoring each remaining candidate according to the linguistic evidence already selected and the remaining recording time. Its practical value is not that it produces different scripts for users under unchanged conditions, but that it can construct a new script when the available inventory, lexical realization, or recording budget changes, without collecting downstream personalization outcomes in that target condition. The submitted results already support these forms of transfer. When a recording budget is held out during training and model selection, LEAP exceeds learned static ranking by +0.026 at the 5-second budget and +0.016 at the 10-second budget. When candidate prompts are replaced with unseen lexical realizations that preserve the same linguistic requirements, LEAP remains ahead by +0.029 at 3 seconds and +0.019 at 10 seconds. We agree that learned static ranking is not a sufficient fixed-script comparator because it does not jointly optimize complementarity. We therefore add an offline jointly optimized fixed-script baseline for the primary in-language setting. For each language-inventory-budget condition, it selects one shared subset by maximizing mean downstream utility over training speakers and the same source voice-personalization models used to train LEAP. That single script is then deployed unchanged to all held-out speakers and evaluation models. It has access to the same training-speaker outcomes, candidate inventory, and budget information as LEAP, but cannot be adapted to a new inventory or budget without collecting downstream outcomes for that new condition. Averaged across Korean, Japanese, Vietnamese, and four budgets, the jointly optimized fixed script achieves 0.681, compared with 0.694 for LEAP ((Δ=+0.013), paired 95% CI [+0.007, +0.019]). At 3, 5, 7, and 10 seconds, it obtains 0.635, 0.674, 0.695, and 0.719, compared with LEAP's 0.652, 0.687, 0.709, and 0.727, respectively.

Jointly optimized fixed scripts versus LEAP. Both methods use the same training-speaker outcomes, candidate inventory, and estimated-duration budget; the fixed script is deployed unchanged to all held-out speakers.

BudgetJointly optimized fixed scriptLEAPΔ
3 s0.6350.652+0.017
5 s0.6740.687+0.013
7 s0.6950.709+0.014
10 s0.7190.727+0.008
Avg.0.6810.694+0.013

Thus, LEAP's contribution is not speaker adaptation. Rather, it is transferable conditional utility learning: it outperforms a shared script optimized on in-distribution training speakers, while constructing scripts for changed inventories or budgets without target-condition personalization outcomes.

AuthorsOpenReview ↗

Response to Reviewer YdUg (Part 2)

Q2. Why Is English Evaluated Only as a Held-Out Transfer Target?

English was intentionally used as an additional held-out target to test whether a policy trained on Korean, Japanese, and Vietnamese can transfer without English personalization outcomes, validation signals, calibration statistics, or checkpoint selection. This provides a stricter test of generalization than an in-language evaluation. The submitted English transfer result is positive: LEAP achieves 0.635, compared with 0.612 for learned static ranking, a gain of +0.023 (95% CI: [+0.011, +0.035]). However, we agree that transfer-only automatic results do not provide the same completeness as an English in-language evaluation with perceptual evidence. We therefore add a speaker-disjoint English in-language evaluation, including the jointly optimized fixed-script comparator described in Q1, together with a blinded English listening study. We collected 432 additional speaker-similarity judgments using the same listener-recruitment, randomization, and mixed-effects analysis protocol as in the primary-language study. In the English speaker-disjoint in-language setting, LEAP achieves a composite utility of 0.690, compared with 0.672 for learned static ranking and 0.681 for the jointly optimized fixed script. Thus, LEAP retains a +0.009 advantage over the fixed-script comparator. In the blinded English evaluation, listeners prefer LEAP over the fixed script in 58.1% of the 432 retained speaker-similarity judgments (95% CI: [53.4%, 62.7%]). These additions distinguish English outcome-free transfer from English in-language performance and provide corresponding perceptual evidence for improved speaker similarity.

AuthorsOpenReview ↗

Response to Reviewer YdUg (Part 3)

Q3. How Much Additional Recording Time Does a Simpler Strategy Require to Match LEAP?

We agree that practical value should be quantified not only through matched-budget improvements, but also through the additional recording time required by simpler alternatives to achieve comparable quality. The submitted budget curves permit this estimate. At the 3-second budget, LEAP achieves a normalized composite utility of 0.652. Learned static ranking increases from 0.619 at 3 seconds to 0.659 at 5 seconds; linear interpolation of these measured values indicates that it requires approximately 4.65 seconds to match LEAP’s 3-second utility—an additional 1.65 seconds, or approximately 55% more recording time. The language-specific handcrafted baseline requires approximately 4.90 seconds (+1.90 seconds), and phoneme coverage requires approximately 5.88 seconds (+2.88 seconds) to reach the same utility. We present these values as descriptive interpolations from measured budget curves, rather than separately evaluated intermediate-budget experiments. We also acknowledge that controlled recordings and estimated-duration budgets do not fully represent a prospective enrollment workflow. The submitted duration analysis shows that LEAP’s gain is not explained by longer realized recordings: at the 3-second budget, LEAP uses 2.95 seconds on average, compared with 2.96 seconds for learned static ranking, with comparable budget-overrun rates of 15.5% and 15.7%. We therefore conducted a prospective counterbalanced crossover study with 12 additional native speakers (four per language), in which participants recorded only the method-specific prompts selected under the matched 3-second final-reference budget, rather than a complete candidate inventory. We also evaluated a separate failure-and-one-re-record protocol. Prompt–transcript mismatches were first flagged by automatic ASR and independently confirmed as pronunciation errors by two native-speaking annotators; disagreements were adjudicated by a third annotator. Recording failures and realized-duration overruns were handled under the same protocol, while final reference-audio budgets remained matched across methods. In the prospective study, LEAP achieved 0.687, compared with 0.669 for learned static ranking (Δ=+0.018Δ=+0.018, 95% CI: [+0.006, +0.030]). Under the separate failure-and-one-re-record protocol, LEAP retained an advantage (0.681 vs. 0.663; Δ=+0.018Δ=+0.018, 95% CI: [+0.005, +0.031]), with comparable re-recording rates (6.3% vs. 6.1%).

MethodEstimated time to match LEAP at 3 sAdditional recording time
LEAP3.00 s—
Learned static ranking4.65 s+1.65 s (+55%)
Language-specific handcrafted baseline4.90 s+1.90 s (+63%)
Phoneme coverage5.88 s+2.88 s (+96%)

These analyses quantify the practical recording-time value of content selection while preserving the correct scope: LEAP is an open-loop, text-conditioned optimizer, not an acoustic-feedback-driven adaptive enrollment system.

Thank you again for your review. We have posted a consolidated response addressing the distinction between transferable script construction and speaker-adaptive enrollment, including a jointly optimized fixed-script comparator, English in-language and perceptual evaluation, and a recording-time-equivalence analysis.

We would be grateful if you could revisit the response when convenient and consider whether the additional evidence addresses your concerns. Thank you for your time and consideration.

Reviewer_sW3bOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper formulates few-shot voice enrollment as a budget-constrained sequential selection problem and proposes LEAP, a set-conditioned policy that predicts the marginal utility of candidate utterances using linguistic annotations. The method learns from offline personalization outcomes and selects complementary enrollment utterances without running downstream voice models at inference time. Experiments across multiple languages and voice-conversion models demonstrate improvements over handcrafted and learned baselines, including transfer to unseen languages and models.

Critical strengths
  • The paper addresses a practically meaningful but relatively underexplored problem: selecting informative enrollment utterances rather than improving the voice-personalization model itself. The set-conditioned formulation provides a reasonable approach to exploiting complementary linguistic evidence under strict recording budgets.

  • The experimental evaluation is extensive, including multiple languages, voice-conversion backends, recording budgets, cross-language transfer, and human listening studies. The consistent improvements over learned static ranking, particularly under restrictive recording budgets, provide convincing evidence for the usefulness of conditional utterance selection.

Critical weaknesses
  • The claimed language transferability relies heavily on carefully curated, language-specific candidate lexicons and structured phonological, prosodic, and articulatory annotations. Although LEAP does not require target-language personalization outcomes, constructing these annotations may require substantial linguistic expertise. The absence of experiments with noisy or automatically generated annotations limits the practical generalizability of the framework to genuinely low-resource languages.

  • Despite framing the problem broadly as few-shot voice personalization, the evaluation exclusively considers voice-conversion systems (Seed-VC, OpenVoice V2, and FreeVC), without examining modern zero-shot text-to-speech models. Since voice conversion and TTS employ substantially different reference-conditioning mechanisms, it remains unclear whether the learned enrollment preferences transfer to widely deployed generative TTS systems.

  • The methodological novelty beyond conventional budgeted subset selection is somewhat limited. In particular, the handcrafted baseline already performs greedy, set-conditioned coverage optimization, while the learned static-ranking baseline does not account for selected-set information. Comparisons with stronger learned submodular objectives, adaptive coverage weighting, or diversity-aware subset-selection methods would better establish whether LEAP learns meaningful higher-order complementarity beyond conventional redundancy reduction.

  • The evaluation relies on retrospectively selecting subsets from fully prerecorded, quality-controlled candidate inventories, with only six held-out test speakers per primary language. Moreover, LEAP does not condition on the actual audio or speaker-specific characteristics during enrollment. Consequently, its robustness to real-world pronunciation variability, recording failures, and heterogeneous speaker characteristics remains unclear, despite these factors being central to practical interactive enrollment.

Questions for authors
  • How sensitive is LEAP to errors or omissions in the linguistic annotations? Can the authors quantify the annotation effort required for a new language and evaluate transfer using automatically generated descriptors?

  • Do the learned enrollment policies generalize to modern zero-shot TTS systems, such as F5-TTS or XTTS, rather than only voice-conversion architectures? Are there preliminary results suggesting that the selected linguistic evidence remains useful across these different conditioning mechanisms?

  • Given that the handcrafted baseline already performs set-conditioned greedy selection, how much of LEAP's improvement can be attributed to learned conditional interactions rather than simply learning better weights for predefined linguistic coverage features? Can the authors compare against stronger learned set-selection baselines?

  • How robust is LEAP when users mispronounce selected prompts, deviate from their estimated durations, or exhibit speaker-specific pronunciation patterns? Would incorporating acoustic feedback after each recording substantially improve the proposed selection policy?

6 comments
AuthorsOpenReview ↗

Response to Reviewer sW3b (Part 1)

We thank the reviewer for recognizing the practical significance of enrollment content selection, the motivation for set-conditioned linguistic evidence, and the breadth of our multilingual and cross-model evaluation. We appreciate the four questions, which help us clarify LEAP's scope and identify the additional evidence needed to substantiate its practical generalizability.

Q1. Sensitivity to Linguistic Annotations and Annotation Effort

LEAP requires a target-language candidate inventory and text-side linguistic descriptors, but does not require target-language personalization outcomes. We therefore evaluated both the feasibility of automatic resource construction and sensitivity to incomplete descriptors.

Automatically constructed resources. Averaged across the Korean, Japanese, and Vietnamese held-out-speaker test sets under matched recording-budget conditions, LEAP using automatically constructed candidate inventories and descriptors achieved a normalized utility of 0.684, compared with 0.694 using curated resources, retaining 98.6% of the original utility. Under the same automatically constructed resources, LEAP continued to outperform learned submodular coverage (0.684 vs. 0.670; Δ=+0.014Δ=+0.014, 95% CI [+0.004,+0.024][+0.004, +0.024]). Thus, its advantage is not restricted to manually curated inventories. Our pipeline automatically harvests candidate prompts from language-specific text sources subject to length and coverage constraints, obtains pronunciation representations through G2P or phonological parsing, and extracts the structured descriptors used by LEAP. Relative to curated annotations, the automatically generated descriptors achieved F1 = 0.91. Language-specific resource preparation decreased from 20 to 4 person-hours (80% reduction), with the remaining effort limited to final candidate validation and annotation quality control.

Resource settingPreparation time / languageDescriptor F1 vs. curatedLEAP utility
Curated inventory and descriptors20 h—0.694
Automatically constructed resources with light human verification4 h0.910.684

Sensitivity to incomplete descriptors. In a controlled missingness analysis, LEAP achieved normalized utilities of 0.688, 0.681, and 0.674 under 10%, 20%, and 30% descriptor missingness, respectively, compared with 0.694 using complete annotations. Even at 30% missingness, it remained ahead of learned submodular coverage (0.674 vs. 0.660). We will additionally report per-family sensitivity to distinguish the effects of omissions in different descriptor families.

Scope of transfer. Our submitted outcome-free transfer evaluation (Table 3) already demonstrates leave-one-language-out transfer to Korean, Japanese, and Vietnamese, as well as transfer to English as an additional target trained using all three non-English source languages. In every setting, the target language contributes no downstream personalization outcomes, normalization statistics, validation scores, or model-selection signals. Appendix A.1 describes how language-specific analyses are mapped into shared canonical evidence descriptors with applicability masks, enabling a single learned policy to operate across languages. We will clarify that LEAP supports outcome-free target-language transfer, rather than annotation-free deployment: it retains its advantage when candidate resources and descriptors are constructed automatically with light human validation. Reliable G2P or phonological analysis remains a limitation for languages without suitable automatic tools, rather than a requirement for manually curated target-language personalization data.

AuthorsOpenReview ↗

Response to Reviewer sW3b (Part 2)

Q2. Generalization Beyond Voice-Conversion Architectures

We agree that successful transfer across voice-conversion systems alone does not establish generalization to zero-shot text-to-speech architectures, which use different reference-conditioning mechanisms.

Existing cross-backend evidence. Our submitted leave-one-model-out evaluation already demonstrates consistent outcome-free transfer across three distinct voice-conversion architectures. LEAP improves over learned static ranking by +0.022 on Seed-VC, +0.023 on OpenVoice V2, and +0.022 on FreeVC, with statistically significant paired improvements (Tables 21 and 24). In every fold, the held-out backend contributes no utility supervision, calibration statistics, validation signals, or model-selection signals.

Extension to zero-shot TTS. To examine transfer beyond voice conversion, we additionally evaluate XTTS-v2 on the Korean and Japanese held-out-speaker test sets, using matched reference-audio budgets and identical synthesis texts. Both selection policies are frozen before XTTS-v2 evaluation, and XTTS-v2 outcomes are excluded from policy training, calibration, and model selection. LEAP achieves an average normalized composite utility of 0.662, compared with 0.643 for learned submodular coverage, yielding a paired gain of +0.019 (95% CI: [+0.007, +0.031]). The improvement is consistent across all three evaluation dimensions:

  • Speaker similarity: ECAPA-TDNN cosine similarity increases from 0.742 to 0.758.
  • Intelligibility: transcription word-error rate decreases from 6.9% to 6.1%.
  • Naturalness: mean naturalness increases from 3.74 to 3.86 / 5. These results provide evidence that LEAP’s learned enrollment preferences are not limited to voice-conversion-specific reference representations. The improvement under XTTS-v2 supports the hypothesis that complementary linguistic evidence can retain value across the evaluated voice-conversion and zero-shot TTS conditioning mechanisms. We will limit our generalization claim to the tested architectures and supported languages, rather than asserting universal applicability across all zero-shot TTS systems.
AuthorsOpenReview ↗

Response to Reviewer sW3b (Part 3)

Q3. Stronger Learned Set-Aware Selection Baselines

We appreciate this important distinction. Our claim is not that set-aware greedy selection itself is novel, but that downstream enrollment utility can be learned as a function of the candidate, the selected linguistic evidence, and the remaining budget. The submitted ablations already support the value of conditional utility modeling: removing novelty and overlap features reduces average utility from 0.694 to 0.676, while set-independent scoring yields 0.669 (Table 4). LEAP also achieves lower marginal evidence overlap than learned static ranking (0.29 vs. 0.46; Table 25). These analyses establish the contribution of conditioning relative to static ranking, but do not fully isolate richer learned interactions from stronger learned coverage objectives. We therefore added learned weighted coverage, learned submodular coverage, and an additive conditional scorer. All methods use matched candidate inventories, linguistic descriptors, downstream utility supervision, and recording budgets. Learned weighted coverage, learned submodular coverage, and the additive conditional scorer achieve utilities of 0.676, 0.681, and 0.682, respectively, compared with 0.694 for full LEAP. The strongest comparator is the additive conditional scorer; LEAP improves over it by +0.012 (paired 95% CI: [+0.003, +0.021]). Learned submodular coverage is the strongest matched diminishing-return and diversity-aware set-selection baseline in our comparison.

MethodAvg. utilityDifference from LEAP
Learned weighted coverage0.676−0.018
Learned submodular coverage0.681−0.013
Additive conditional scorer0.682−0.012
LEAP0.694—

The additive conditional scorer uses the same evidence families and selected-set summary as LEAP, but permits only additive evidence-family contributions rather than learned cross-feature candidate–set interactions. Its gap to LEAP therefore isolates the benefit of richer learned conditional interactions beyond learned evidence weights, conventional diminishing-return objectives, and additive state conditioning. These matched comparisons show that LEAP’s improvement cannot be explained solely by learned coverage weights or standard submodular redundancy reduction. We will clarify that the methodological contribution is transferable conditional utility learning, rather than greedy selection itself or a claim to identify a unique higher-order interaction mechanism.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.