- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LEAP, a method for selecting informative enrollment utterances for few-shot voice personalization under a limited recording budget. LEAP predicts the marginal utility of candidate utterances based on linguistic features and the content already selected, using a learned set-conditioned scoring function. The policy is trained offline using downstream personalization outcomes and operates without additional synthesis or quality evaluation at inference time. The method is evaluated on Korean, Japanese, and Vietnamese using three voice conversion models (Seed-VC, OpenVoice V2, and FreeVC) under recording budgets of 3 to 10 seconds. The experiments include comparisons with random selection, phoneme coverage, handcrafted linguistic objectives, and learned static ranking, as well as cross-language and cross-model transfer evaluations. The reported results show consistent improvements in normalized automatic metrics and moderate preferences for LEAP in human listening tests.
Critical strengths
Well-motivated problem setting
The paper addresses the practical problem of selecting enrollment content under limited recording budgets, rather than focusing solely on improving voice personalization models. This is a relevant and relatively underexplored direction.
Broad experimental evaluation
The experiments cover multiple languages, voice conversion models, recording budgets, and transfer settings. Human listening tests and comparisons with learned and handcrafted baselines provide useful evidence that enrollment content selection affects downstream personalization quality.
Critical weaknesses
Limited methodological novelty and lack of enrollment-time adaptation
Although LEAP is formulated as an active enrollment policy, selection relies entirely on predefined linguistic annotations and the remaining budget, without incorporating speaker-specific information or feedback from recorded audio. Consequently, a fixed candidate inventory and budget yield the same enrollment script for every speaker, and the entire selection process can be completed before recording begins. While the set-conditioned sequential selection is valid, the method is closer to learned budget-constrained subset selection than an enrollment policy that adapts to individual speakers. Its main components, including novelty and redundancy modeling, marginal utility prediction, and greedy selection, are also established techniques. The paper does not clearly establish a methodological distinction from existing subset-selection approaches.
Insufficient comparison with simpler set-aware methods and limited interpretability of improvements
Although LEAP outperforms learned static ranking and handcrafted coverage baselines, the evaluation lacks comparisons with simpler learned set-aware approaches, such as weighted coverage or submodular selection. It is therefore unclear whether the proposed architecture is necessary to obtain the reported gains. Furthermore, the main results report normalized composite and component scores without corresponding absolute speaker similarity, CER/WER, or naturalness values, making the practical magnitude of the improvements difficult to interpret. The method also depends on curated, language-specific linguistic annotations and prompt inventories, leaving the cost of extending it to new languages insufficiently examined.
Questions for authors
-
Since LEAP does not use speaker-specific information or recorded audio, does it generate identical enrollment scripts for all speakers given the same inventory and budget? What advantage does sequential selection offer over offline script optimization?
-
Can the authors compare LEAP with simpler learned set-aware methods, such as weighted coverage or submodular selection, using the same linguistic features and training supervision?
-
Can the authors report raw speaker similarity, CER/WER, and NISQA scores? These would help clarify the practical significance of improvements reported using normalized metrics.
-
How much manual effort is required to construct the language-specific annotations and candidate inventories? Does LEAP remain effective with automatically generated or less curated prompts?
8 comments
Response to Reviewer X4RY (Part 2)
Q2. Does LEAP Outperform Simpler Learned Set-Aware Objectives?
We agree that comparisons with learned static ranking and handcrafted coverage alone do not isolate whether LEAP requires richer conditional utility modeling. The submitted ablations already show that removing the novelty and overlap features reduces average utility from 0.694 to 0.676, while set-independent scoring yields 0.669. LEAP also selects less redundant evidence than learned static ranking, with mean marginal overlap of 0.29 versus 0.46. However, these analyses do not fully exclude stronger learned set-aware objectives. We therefore add learned weighted coverage, facility-location selection, learned submodular coverage, and an additive conditional scorer. All methods use the same candidate inventories, linguistic descriptors, training-speaker utility supervision, recording budgets, and greedy feasibility rule. Learned weighted coverage learns evidence-family weights; facility location learns weighted representativeness; learned submodular coverage learns diminishing-return coverage weights; and the additive conditional scorer receives the same selected-set summary as LEAP but excludes candidate-set cross-feature interactions. Learned weighted coverage, facility-location selection, learned submodular coverage, and the additive conditional scorer achieved average utilities of 0.676, 0.678, 0.681, and 0.682, respectively, compared with 0.694 for full LEAP. LEAP therefore exceeded the strongest comparator, the additive conditional scorer, by +0.012 (paired 95% CI: [+0.003, +0.021]).
| Method | Avg. utility | Difference from LEAP |
|---|---|---|
| Learned weighted coverage | 0.676 | −0.018 |
| Facility-location selection | 0.678 | −0.016 |
| Learned submodular coverage | 0.681 | −0.013 |
| Additive conditional scorer | 0.682 | −0.012 |
| LEAP | 0.694 | — |
These match comparisons show that the improvement is not explained solely by learned evidence weights, conventional diminishing-return coverage, or additive state conditioning. We will accordingly limit the novelty claim to transferable nonlinear conditional utility learning, rather than greedy selection or redundancy reduction itself.
Response to Reviewer X4RY (Part 3)
Q3. What Is the Raw Practical Magnitude of the Automatic-Metric Improvements?
We agree that normalized scores facilitate aggregation across language-model conditions but can obscure absolute magnitudes. We therefore now report raw speaker similarity, content error, and NISQA-TTS scores alongside normalized utility. Speaker similarity is cosine similarity between frozen ECAPA-TDNN embeddings; content preservation is CER for the multilingual primary benchmark, with WER additionally reported where reliable tokenization is available; naturalness is the raw NISQA-TTS score. At the 3-second budget, averaged across the Korean, Japanese, and Vietnamese primary-language conditions and evaluated voice-conversion backends, LEAP improved mean ECAPA cosine similarity from 0.741 to 0.758, reduced CER from 9.8% to 8.7%, and improved NISQA-TTS from 3.42 to 3.55, relative to learned static ranking. At 10 seconds, it improved cosine similarity from 0.770 to 0.778, reduced CER from 6.9% to 6.4%, and improved NISQA-TTS from 3.67 to 3.73. The corresponding English-transfer WER decreased from 8.1% to 7.3%. These raw values show that the largest practical gains occur in the most constrained enrollment regime, consistent with the submitted normalized budget curves. We will provide per-language, per-budget raw metrics and paired confidence intervals in the appendix.
| Budget | Method | ECAPA cosine ↑ | CER ↓ | NISQA–TTS ↑ |
|---|---|---|---|---|
| 3 s | Static ranking | 0.741 | 9.8% | 3.42 |
| 3 s | LEAP | 0.758 | 8.7% | 3.55 |
| 10 s | Static ranking | 0.770 | 6.9% | 3.67 |
| 10 s | LEAP | 0.778 | 6.4% | 3.73 |
Response to Reviewer X4RY (Part 1)
We thank the reviewer for recognizing the practical relevance of enrollment-content selection and the breadth of the multilingual evaluation. We agree that the paper should more precisely distinguish transferable conditional script construction from speaker-adaptive enrollment, compare against stronger learned set-aware baselines, report raw metric magnitudes, and quantify the cost of constructing target-language resources.
Q1. Is LEAP an Enrollment Policy or an Offline Script Optimizer?
We agree that, for an identical language, candidate inventory, and recording budget, LEAP is deterministic and produces the same ordered script for every user. LEAP does not use speaker-specific audio or recording feedback at selection time; it should therefore not be interpreted as a speaker-adaptive enrollment policy. We will revise the framing accordingly. LEAP is an offline, transferable conditional script-construction method. It sequentially constructs a script for a supplied inventory and budget by re-scoring remaining candidates according to already selected linguistic evidence and remaining recording time. Its practical value is not per-speaker variation under fixed conditions, but its ability to construct a script for a changed prompt inventory, lexical realization, or recording budget without target-condition personalization outcomes. The submitted results already support this transfer capability. When recording budgets are held out during training and model selection, LEAP exceeds learned static ranking by +0.026 at 5 seconds and +0.016 at 10 seconds. For unseen lexical realizations, it remains ahead by +0.029 at 3 seconds and +0.019 at 10 seconds. We agree that learned static ranking is not a sufficient fixed-script comparator because it does not jointly optimize complementarity. We therefore add an offline jointly optimized fixed-script baseline for the primary in-language setting. For each language-inventory-budget condition, it selects one shared subset by maximizing mean downstream utility over training speakers and the same source voice-personalization models used for LEAP training. The resulting script is deployed unchanged to all held-out speakers and evaluation models. It has access to the same training-speaker outcomes, candidate inventory, and budget information as LEAP, but cannot be adapted to a new inventory or budget without collecting downstream outcomes for that new condition. The jointly optimized fixed script achieved 0.681 average utility across Korean, Japanese, Vietnamese, and all four budgets, compared with 0.694 for LEAP ((Δ=+0.013), paired 95% CI: [+0.007, +0.019]). At 3, 5, 7, and 10 seconds, it achieved 0.635, 0.674, 0.695, and 0.719, compared with LEAP's 0.652, 0.687, 0.709, and 0.727, respectively. Thus, LEAP's contribution is not speaker adaptation; it is transferable conditional utility learning that outperforms a shared script optimized on in-distribution training speakers while constructing scripts for changed inventories or budgets without target-condition outcomes.