acceptodds
Under review as a conference paper at ICLR 2027

Exploring Token-Space Manipulation in Latent Audio Tokenizers

Abstract

Neural audio codecs provide compact discrete representations for speech generation and manipulation. However, most codecs organize tokens as frame-level sequences, making it difficult to study or intervene on global factors of variation. In this work, we propose the Latent Audio Tokenizer for Token-space Editing (LATTE) that appends a fixed set of learnable latent tokens to the audio feature sequence and retains only these tokens for quantization and decoding. This design produces a compact, non-temporally aligned bottleneck in which each token can aggregate global information across the full utterance. We show that the resulting tokenizer preserves competitive reconstruction quality in low-bitrate speech coding settings while enabling simple token-space interventions. In particular, we find that swapping selected latent token positions between utterances can modify global attributes, such as speaker identity and background noise, and we evaluate these interventions on voice conversion and denoising tasks. Our results suggest that compact latent audio tokenizers can support controllable audio manipulation without supervision in task-specific editing models.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
latent tokenization, audio tokenizer, voice conversion, controllable tokenizer

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_sahgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LATTE, which introduces learned discrete latent slots between a frozen FocalCodec encoder and decoder. The authors estimate slot importance using groups defined by attributes such as speaker identity, gender, accent, and noise. They then edit speech by replacing selected slots with tokens from a reference utterance. Experiments cover speech reconstruction, slot-importance analysis, denoising, and controlled voice conversion.

Critical strengths
  1. The speaker-editing experiment includes informative internal controls. In Table 4, replacing five high-importance slots yields a target-speaker similarity of 90.0, compared with 67.8 for random slots and 68.1 for the least-important slots at the same swap count. This directly supports the usefulness of the importance ranking.

  2. The paper connects representation analysis with actual interventions. The authors examine importance profiles across several attributes and test whether these associations support editing. They also report the loss in reconstruction intelligibility and acknowledge that speaker identity, gender, and accent remain entangled, giving a useful account of both the representation’s capabilities and its limitations.

Critical weaknesses
  1. The practical editing benefit of non-temporally aligned slots over existing representations is not yet sufficiently established. Learned queries for aggregating audio information have precedents, including ALMTokenizer. An important contribution here is therefore the evidence that this slot structure makes attributes easier to analyze and edit. Appendix Figure 5 compares importance profiles with frame-level FocalCodec, but the paper does not compare their actual editing performance under a common reference condition and evaluation protocol. The controls in Table 4 show that slot selection matters within LATTE, but do not establish how much benefit comes from the non-temporally aligned representation itself. Given the reduction in reconstruction intelligibility, this trade-off needs more direct experimental support.

  2. The selectivity of attribute editing is not fully established. The experiments show that selected slots affect the target attribute, but provide less complete evidence that other information is preserved. The denoising experiment compares against no-swap reconstruction without random-slot or least-important-slot controls at the same swap count. It is therefore difficult to determine whether the improvement specifically depends on the identified noise-related slots. The internal controls for speaker editing are stronger, but the source and reference utterances contain the same text. Consequently, dWER cannot distinguish preservation of source content from the introduction of content from the reference. The authors explicitly frame this as a controlled intervention, which is reasonable. Nevertheless, the current setting provides limited evidence about content preservation, leaving the broader selectivity of the representation uncertain.

Questions for authors
  1. Could you provide a small-scale editing comparison between LATTE and the existing FocalCodec model using the same test utterances, reference audio, and task metrics? FocalCodec could use an editing strategy appropriate for its frame-level representation, with the relevant settings explained. This would help establish the practical benefit of the non-temporally aligned slots.

  2. For denoising, could you report random-slot and least-important-slot controls with the same number of replaced slots? For speaker editing, could you provide an additional check of source-content preservation, for example using a small set of references with different text and reporting both content preservation and target-speaker similarity? These checks would help distinguish target-attribute changes from the transfer of other reference information.

  3. Does the bitrate reported in Table 1 account for the tokens produced by overlapping windows and padding of short utterances? Please report the effective bitrate for each test set, as this affects the interpretation of the reconstruction quality–coding cost trade-off.

3 comments
AuthorsOpenReview ↗

Reply to reviewer sahg (part 1/2)

We thank the reviewer for the constructive questions; we ran the three suggested comparisons.

(W1, Q1) Do non-aligned slots make editing easier than frame-level FocalCodec under the same protocol?

TLDR: Yes: LATTE changes the speaker by swapping 1.2% of its tokens, while FocalCodec needs about half of its frames.

To test the benefit of non-aligned slots, we edit the token sequences extracted using FocalCodec on the same 164 VCTK pairs, with the same reference audio and metrics as Table 4.

FocalCodec@50 also uses 250 tokens of 13 bits per 5 s (same as LATTE), so the share of swapped tokens is directly comparable to LATTE.

A share of the source's FocalCodec frames was replaced with the target's, chosen at random (10 draws) or by the same importance score (Fig. 5). Because the two recordings differ in timing, frames were aligned in two ways: by cutting both to the same length, and by time-warping (DTW), which works here because both say the same sentence. Table R4 (the importance-chosen rows are now in Table 4; the full table is Table 6 in App. F):

SystemPositionsTokens changedSim-TSim-SUTMOSdWER
FocalCodecnone0%70.896.53.943.8
FocalCodec, truncatedimportance / random10%71.7 / 72.596.1 / 95.43.92 / 3.705.6 / 20.4
FocalCodec, truncatedimportance / random50%86.3 / 84.685.1 / 83.03.63 / 3.0545.0 / 100.3
FocalCodec, DTW-alignedimportance / random10%71.9 / 73.196.2 / 95.83.93 / 3.893.7 / 6.5
FocalCodec, DTW-alignedimportance / random50%85.5 / 86.186.1 / 85.43.78 / 3.7414.4 / 16.8
FocalCodec, all framestarget re-coded100%96.669.23.9513.7
LATTEnone0%68.196.24.1110.4
LATTEguided / random1.2% (k=3k=3)89.2 / 68.574.9 / 95.74.11 / 4.0912.7 / 14.1
LATTEguided / random2% (k=5k=5)89.7 / 68.874.6 / 95.44.16 / 4.0812.3 / 16.3
kNN-VC (external reference)––94.569.63.4210.9

LATTE reaches Sim-T 89.2 by swapping 1.2% of its tokens. FocalCodec needs about half of its frames for a similar value (84.6–86.3). Without time-warping its quality drops (UTMOS 3.1–3.6) and dWER rises to 45–100; time-warping largely avoids this only because both recordings say the same sentence.

Choosing frames by importance moves the speaker no more than choosing them at random (App. F), which matches Fig. 5: FocalCodec spreads speaker information across all frames.

AuthorsOpenReview ↗

Reply to reviewer sahg (part 2/2)

(W2, Q2) Is editing selective: do the denoising gains need the noise-ranked slots, and is source content preserved with a different reference sentence?

TL;DR: Yes: the noise-ranked slots beat the controls, and source content survives a reference that says a different sentence.

Denoising controls. Reported in full in the reply to EqRf (W1): the noise-ranked slots beat random, least-important and speaker-ranked slots in 6 of 8 settings; the exception swaps a single slot for WHAM! noise.
Non-parallel conversion. To check that content is preserved and not copied from the reference, each of the same 164 source recordings was paired with a target that differs in both speaker and sentence (fixed list, seed 0). dWER is measured against the source transcript (did the content survive?) and against the target transcript (did the target's words leak in?), in Table R5 (now Table 7 in App. G):

RowSim-TSim-SUTMOSdWER vs sourcedWER vs target
No swap71.896.24.1110.38127.2
Guided, k=3k=388.680.64.0811.65127.9
Guided, k=5k=589.280.14.0714.17126.8

The guided swap raises Sim-T by about 17 points, close to the 21 points with same-sentence pairs. dWER against the target transcript stays where it is without swapping (about 127; above 100 because the sentences are unrelated), so the target's words do not leak into the output. dWER against the source rises by 1.3 (k=3k=3) and 3.8 (k=5k=5) points.

(Q3) Does the Table 1 bitrate count overlapping windows and the padding of short utterances?

TL;DR: No: 650 bps is the nominal rate; counting overlap and padding, the evaluated rate is 1,051 to 1,911 bps (Table R6).

The 650 bps in Table 1 is the nominal rate of one 5 s window and does not include overlap or padding. Two things raise the actual rate: the evaluation moves the 5 s window forward 1 s at a time, so windows overlap by 4 s, and utterances shorter than 5 s still use a full window of 250 tokens. Table R6 (now Table 8 in App. H) gives the rate computed from the tokens actually produced:

SetUtterancesShare under 5 sEffective bps, as evaluated (1 s hop)Nominal bps
LibriSpeech test-clean2,62042%1,911650
VoiceBank82498%1,343650
Libri1Mix3,00077%1,051650

We note that both of these issues are consequences of the network that we designed to investigate the properties of latent tokenizers in audio, and with further engineering efforts this can plausibly be mitigated. We also note that this does not impact the main findings of how latent tokens for audio represent the information present in the audio.

Dear reviewer,

Thank you for your constructive review. We were wondering if you had any feedback for our rebuttal, given that the author-reviewer discussion period is coming to a close soon.

Best regards,

The authors

Reviewer_EqRfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LATTE, a TiTok-style latent-bottleneck speech tokenizer built on a frozen FocalCodec. A compressor appends LL learnable queries to FocalCodec's WavLM layer-6 features, keeps only the query outputs, quantizes them with Binary Spherical Quantization (d=13d=13, 8192-entry vocabulary; L=rτ=250L=r\tau=250 tokens per 5 s chunk, i.e. nominally 650 bps), and a decompressor reconstructs the codec feature sequence for the frozen FocalCodec decoder. Training uses only a feature-space MSE plus the BSQ regularizer. A post-hoc slot-importance score is used to locate slots associated with speaker, gender, accent, and noise level, and importance-ranked slots are swapped between utterances to perform zero-shot denoising and parallel voice conversion.

Critical strengths
  1. The matched-budget comparison in Table 4 is the right experiment: swapping only 5 of 250 importance-ranked slots (γ=0.10\gamma=0.10) gives Sim 90.0, whereas swapping 5 random or 5 least-important slots gives 67.8/68.1, with UTMOS (4.16) and dWER (12.32) largely preserved. This ~22-point gap shows that the importance score finds functionally relevant positions rather than any edit being a generic perturbation.
Critical weaknesses
  1. The random/least-important controls are reported only for VC. In Table 3, the source's slots are replaced by those of an unmatched clean utterance, so the gains could come from importing any clean-utterance tokens, not specifically noise slots. Without random-slot and least-important-slot swaps at the same γ\gamma, Table 3 does not show that the noise profile is what matters. The metric choice adds to this confound. DNSMOS is a single no-reference score, and Sim drops consistently.
  2. ALMTokenizer is acknowledged as the audio adaptation of the same learnable-query bottleneck idea, but it is excluded because code is unavailable. A comparison on its reported benchmarks, or a discussion of what LATTE adds beyond FocalCodec + TiTok-style queries + BSQ, is needed to establish novelty on the tokenizer side.
  3. The paper describes the reconstruction as "matching or approaching frame-level baselines", the dWER gap as "slightly looser phonetic reconstruction", and the overall result as "strong controllability". But dWER is 2.5× FocalCodec's at the same nominal bitrate.
  4. All models use r=50r=50 Hz and d=13d=13. There is no ablation of token rate/number of slots, code dimension, or the feature-MSE-only objective. The asymmetric encoder/decoder design is motivated but not tested, since BASE and LARGE change both widths at once.
Questions for authors

None

3 comments
AuthorsOpenReview ↗

Reply to reviewer EqRf (part 1/2)

We thank the reviewer for their detailed feedback.

(W1) Without controls, does Table 3 show that the noise-ranked slots matter, and is DNSMOS enough?

TLDR: We would like to note that Table 3 now includes the controls, which the noise-ranked slots beat in 6 of 8 settings, and dWER and similarity complement DNSMOS.

To test whether the noise profile specifically matters, we swapped the same number of slots in every row of Table 3, chosen in four ways:

  • by noise importance (as in the paper),
  • at random (10 draws),
  • the least important,
  • the most speaker-related (as a cross-check).

All four take tokens from the same clean reference. Table R3 gives DNSMOS; without swapping it is 3.513 on VoiceBank and 3.155 on Libri1Mix:

SetProfile, γ\gamma (kk)GuidedRandomLeastCrossGuided − random [95% CI]
VoiceBankWHAM!, 0.02 (1)3.4873.5033.4923.501−0.016 [−0.028, −0.004]
VoiceBankWHAM!, 0.10 (2)3.5873.5023.4713.504+0.085 [0.062, 0.109]
VoiceBankwhite, 0.02 (1)3.5803.5033.5003.501+0.077 [0.054, 0.101]
VoiceBankwhite, 0.10 (3)3.5883.5073.5023.526+0.081 [0.056, 0.107]
Libri1MixWHAM!, 0.02 (1)3.1383.1653.1653.159−0.028 [−0.041, −0.015]
Libri1MixWHAM!, 0.10 (2)3.3593.1703.1653.161+0.190 [0.163, 0.216]
Libri1Mixwhite, 0.02 (1)3.3943.1653.1613.159+0.228 [0.201, 0.257]
Libri1Mixwhite, 0.10 (3)3.3993.1743.1603.162+0.225 [0.198, 0.253]

We observe that in six of eight settings the guided swap beats every control, by 0.06 to 0.08 DNSMOS on VoiceBank and 0.19 to 0.23 on Libri1Mix; in all six, the paired 95% confidence interval against random swaps is above zero. The controls change DNSMOS by at most 0.05, so copying tokens from a clean utterance does not by itself improve quality. The exception is WHAM! at γ=0.02\gamma=0.02, where only one slot is swapped and the guided swap does slightly worse than random on both sets; the revision reports it.
Metrics. dWER and speaker similarity for the noise-ranked swaps are in the submitted Table 13 (App. J): on VoiceBank, dWER falls from 16.30 to 9.54 with WHAM! noise, and speaker similarity drops because the clean reference is deliberately unmatched in text and speaker (Section 6.3, App. D).

AuthorsOpenReview ↗

Reply to reviewer EqRf (part 2/2)

(W2) What does LATTE add beyond FocalCodec, TiTok-style queries and BSQ, given ALMTokenizer?

TLDR: The paper does not claim a better tokenizer; it studies where such a tokenizer stores speaker, accent and noise information, and tests this by editing.

We would like to point out that tokenizers built on learned queries exist for images (TiTok) and audio (ALMTokenizer), and LATTE builds on them. The paper asks a different question: where does such a tokenizer store speaker, accent and noise information? The design is set up to answer it:

  • The encoder and decoder are frozen, so any structure in the slots comes from the new middle layer alone.
  • The importance ranking is tested by editing, against random and least-important swaps of the same size, and now also against shuffled labels (reply to rdE8).
  • The new comparison with FocalCodec (reply to Reviewer ‘sahg’) shows what this improves: LATTE is able to convert the speaker by swapping only 3 of 250 tokens (similarity 68.1→89.268.1 \to 89.2), while FocalCodec needs about half of its frames (125 of 250 frames, Sim-T 84.6–86.3) and stays intelligible only when both recordings say the same sentence and are time-aligned.

ALMTokenizer is evaluated on reconstruction and language-model tasks, not on where information is stored or on editing. The ALMTokenizer repository has no code or checkpoints, and its published numbers use different training data, bitrates and protocols, so a comparison would not be reliable.

(W3) Does the wording overstate reconstruction quality, given the higher dWER?

TLDR: The revision replaces these phrases with the measured numbers.

In the revision we replaced "matching or approaching frame-level baselines", "slightly looser phonetic reconstruction" and "strong controllability" with the measured values:

We see that UTMOS, DNSMOS and speaker similarity are in fact on par with or above FocalCodec@50; dWER for LATTE LARGE is 2.7×2.7\times FocalCodec@50 on LibriSpeech (5.82 vs 2.18), 2.0×2.0\times on VoiceBank (16.30 vs 8.08, still the second-lowest of all listed codecs) and 1.4×1.4\times on Libri1Mix (39.07 vs 27.89), at an actual bitrate above the nominal 650 bps (reply to sahg, Q3); and control that is partial and varies by factor, as Sections 6.2 and 7 already say.

(W4) Why are token rate, slot count, code size, training objective and encoder/decoder asymmetry not ablated?

TLDR: Each of these ablations needs a full retraining, so they are now listed as future work.

Each model took about 2.55 million optimizer steps on 4 GPUs (about 115 epochs), so ablations of token rate, slot count, code size or training loss cannot be trained within the discussion period. The revision no longer suggests that different encoder and decoder widths help, since BASE and LARGE change both at once, and lists these ablations as future work. We hope the new controls and comparisons address the reviewer's concerns, and would be grateful if the reviewer would reconsider the score.

Dear reviewer,

Thank you for your constructive review. We were wondering if you had any feedback for our rebuttal, given that the author-reviewer discussion period is coming to a close soon.

Best regards,

The authors

Reviewer_rdE8OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Frame-aligned codecs spread global attributes across many time steps. The paper replaces FocalCodec's frame-level bottleneck with a TiTok-style one: a fixed set of learned latent queries attends over the frozen WavLM layer-6 features, only the query positions are kept, they are quantized with BSQ at 13 bits, and a decompressor rebuilds the feature sequence for the frozen decoder. Because no temporal alignment is imposed, each slot can pool information from the whole utterance. The analysis scores each slot by the leading eigenvalue of the between-group covariance of mean codes, for speaker, gender, accent and noise partitions. Slots ranked by that score are then swapped between utterances for denoising and parallel voice conversion, with random and least-important controls at matched budgets.

Critical strengths

1. The design isolates the bottleneck properly. Encoder and decoder stay frozen, only the chain between them is learned, and the same importance scoring is run on the original frame-level tokenization as a reference. That means the observed slot structure is attributable to the bottleneck rather than to a new codec.

2. The probe is causal rather than correlational. Estimating a slot ranking and then intervening with matched budgets is the right way to test whether a profile means anything, and most work in this area stops at the profile.

3. The denoising number is real. On VoiceBank, swapping a few upper-ranked slots takes dWER from 16.30 to 9.54 and DNSMOS above every listed baseline, with no training and a clean reference that is unmatched in both text and speaker. I did not expect a token swap to close that much of the gap.

Critical weaknesses

1. The conversion controls destroy the audio rather than hold it fixed: random and least-important swapping give a similarity of 67.8 and 68.1 against 89.5 for the guided swap, so the comparison shows that arbitrary tokens break speech, not that these slots carry speaker. There is no external conversion baseline either, and Table 4 puts resynthesis codecs and the swap in one similarity column whose reference utterance differs between the two blocks.

2. The importance score has no null. It is a leading eigenvalue estimated over 250 slots, so some spread is expected even without factor structure; shuffling the labels would calibrate it. The speaker profile is also fit on VCTK, the same population the conversion pairs come from.

3. The reconstruction claim leans on the metrics the bottleneck is most likely to flatter. dWER is worse everywhere, 5.82 against 2.18 on clean speech, while UTMOS and DNSMOS are reference-free and can reward a smoother signal. No confidence intervals appear in Tables 1 to 4.

Questions for authors

1. In each row of Table 4, which pair of utterances is the similarity computed over, and are the codec rows comparable with the swap rows at all?

2. What does the importance profile look like under shuffled factor labels, and is the speaker profile estimated on speakers disjoint from the conversion pairs?

3. Do the UTMOS and DNSMOS gains survive at matched dWER? If the perceptual gain and the intelligibility loss are two sides of one smoothing effect, the reconstruction argument is weaker than it reads.

3 comments
AuthorsOpenReview ↗

Reply to reviewer rdE8 (part 1/2)

We thank the reviewer for their careful review; the three points raised led to most of the new experiments.

(W1, Q1) Do the controls just break the audio, and are the similarity columns of Table 4 comparable across rows?

TL;DR: No: after a random or least-important swap the output keeps the source speaker and its quality, and only the top-ranked slots move it to the target; Table 4 now uses one similarity definition in every row.

To check whether the control experiments (swapping tokens based on reverse importance order, or random importance) break the audio, each output was compared with both the source and the target speaker. Sim-T is speaker similarity to the target (WavLM-SV, against the first 5 s of the target utterance), Sim-S is similarity to the source, and every row uses the same 164 pairs.

After a random or least-important swap, we observe that the output still sounds like the source speaker (Sim-S 95.4–96.1, against 96.2 without swapping) and keeps its quality (UTMOS 4.08–4.10, against 4.11). We would like to note that it does not just move toward the target (Sim-T 68.0–68.8, against 68.1 without swapping). Swapping the top-ranked slots moves it there (Sim-T 89.2–89.7) at the same quality (Table R1):

RowkUTMOSSim-TSim-SdWER (corpus)
FocalCodec@50, resynthesis of source–3.9470.896.53.78
LATTE, no swap04.1168.196.210.38
Guided (γ=0.05\gamma=0.05)34.1189.274.912.65
Guided (γ=0.07\gamma=0.07)44.1089.374.912.93
Guided (γ=0.10\gamma=0.10)54.1689.774.612.32
Random (10 draws)34.0968.595.714.08
Random (10 draws)44.0868.695.615.03
Random (10 draws)54.0868.895.416.26
Least important34.1068.096.111.74
Least important44.0968.096.113.20
Least important54.1068.196.112.03
kNN-VC (external reference)–3.4294.569.610.87

We emphasize that guided swaps beat random ones by about 21 points of Sim-T at every kk (95% confidence intervals between 18 and 24). Sim-T values differ slightly from the submitted Table 4 (for example 89.7 instead of 90.0 at k=5k=5) because every row now uses the same 164 pairs and the same Sim-T definition.
Q1 (which pairs similarity is computed over). The revised Table 4 computes Sim-T and Sim-S as above in every row, including FocalCodec@50 and kNN-VC, so the codec rows and the swap rows are now directly comparable.
External baseline. We have added an experiment on kNN-VC, a dedicated conversion system which given the same 5 s reference, matches the target speaker more closely than LATTE at k=5k=5 (+4.8 Sim-T) but scores clearly lower in terms of sound quality, i.e., UTMOS (−0.74).

AuthorsOpenReview ↗

Reply to reviewer rdE8 (part 2/2)

(W2, Q2) Is the importance score calibrated against shuffled labels, and is the speaker profile fit on the test speakers?

TLDR: Yes, every profile beats all 1,000 label shuffles, and profiles fit without the test speakers give the same conversion results.

Shuffled labels. To better calibrate the importance score (Eqn 9), we shuffled the labels of each factor 1,000 times and recomputed the importance profile. We observe that, for every factor, the real profile is more concentrated than all 1,000 shuffled ones (lower entropy, higher Gini), and its top score exceeds the top score of any shuffled profile (Table R2, now Table 5 in App. E):

FactorEntropy, observedEntropy, shuffled 95%Gini, observedGini, shuffled 95%Max gg, observedMax gg, largest shuffled
Speaker (LibriTTS)5.4855.507–5.5120.1330.074–0.0910.1080.089
Gender (LibriTTS)4.4085.407–5.4440.6260.219–0.2650.1030.004
Speaker (VCTK)5.1835.465–5.4770.4190.166–0.1850.1050.028
Gender (VCTK)4.5485.403–5.4410.6250.224–0.2690.0730.005
Accent (VCTK)5.1935.449–5.4740.4020.171–0.2090.0870.035
Noise, white4.8505.471–5.4900.5560.140–0.1730.2260.016
Noise, WHAM!4.7845.472–5.4900.5740.140–0.1710.1550.016

Every slot swapped in Tables 3 and 4 scores higher than in all 1,000 shuffles (p≤0.001p \le 0.001, the smallest value 1,000 shuffles can give). Speaker on LibriTTS is the weakest case, but its top score still exceeds the largest shuffled one (0.108 against 0.089).
Disjoint speakers. As the reviewer notes, the original profile was estimated on all 107 VCTK speakers, including the 8 used for conversion. Conversion was re-run with two profiles that leave them out:

  • VCTK without the 8 conversion speakers (99 speakers): it ranks the slots almost exactly as before and picks the same top five, so the swaps and results are unchanged (Sim-T 89.2 at k=3k=3, 89.7 at k=5k=5).
  • LibriTTS dev-clean (40 speakers, no VCTK data): it picks partly different slots but works as well (Sim-T 89.3 at k=3k=3 and 89.5 at k=5k=5, against about 68 for both controls).

So the conversion result does not depend on having seen the test speakers.

(W3, Q3) Do the UTMOS and DNSMOS gains survive at matched dWER, and where are the confidence intervals?

TLDR: Yes: LATTE still scores higher on utterances that both codecs reconstruct with zero dWER, and Table 3 now reports 95% confidence intervals.

Q3, matched dWER. To separate quality from intelligibility, we restricted the comparison to utterances that both LATTE LARGE and FocalCodec@50 reconstruct with zero dWER.

We observe that LATTE still scores higher: +0.16 UTMOS on LibriSpeech (1,283 utterances), +0.13 DNSMOS on VoiceBank (401) and +0.11 DNSMOS on Libri1Mix (227), all with 95% confidence intervals above zero. On the full test sets, the gain is similar in every dWER bin, and how much higher LATTE's dWER is on an utterance does not predict how much higher it scores (Spearman correlation ∣ρ∣≤0.05|\rho| \le 0.05), so the higher quality scores are not a side effect of lower intelligibility. Speaker similarity to the original recording is also on par (97.4 for both on LibriSpeech).
Confidence intervals. In Table 3 we now report 95% bootstrap confidence intervals over test utterances (10,000 resamples), paired between methods, and the Table 4 caption gives the paired interval for guided vs random swaps (18 to 24 Sim-T points). For Table 1, per-system intervals would not show whether LATTE's quality advantage over FocalCodec@50 survives its higher dWER, so the paired matched-dWER comparison above tests this directly; every gain there has a 95% confidence interval above zero. All these intervals capture variation across test utterances, not across training runs, which would need several full trainings; the revision lists this as a limitation. We hope these results address the reviewer's concerns, and would be grateful if the reviewer would reconsider the score in light of them.

Dear reviewer,

Thank you very much for your constructive review. We were wondering if you had any feedback for our rebuttal, given that the author-reviewer discussion period is coming to a close soon.

Best regards,

Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the rebuttal

We thank the reviewers for their feedback. The reviewers found the probe "causal rather than correlational" (rdE8) and the matched-budget comparison "the right experiment" (EqRf), and noted that the paper "connects representation analysis with actual interventions" (sahg). All three rated the paper clear and easy to follow, with soundness concerns that are minor or fixable.

The main concerns and the responses are summarized below; Tables R1–R6 are in the replies.

  • Do the control experiments just break the audio? (rdE8, EqRf). No. We observe that after a random or least-important swap the output still sounds like the source speaker, at the same quality; swapping the top-ranked slots moves it to the target speaker (similarity 89–90 against 68–69; Table R1). Table 4 now uses one similarity definition in every row and adds kNN-VC, a dedicated conversion system.
  • Is the importance score calibrated, and does it depend on the test speakers? (rdE8). Against 1,000 label shuffles, every real profile is more concentrated than any shuffled one, and every swapped slot is significant (Table R2). Profiles estimated without the eight test speakers, or on LibriTTS, give the same conversion results (similarity 89–90).
  • Does the denoising gain come from the noise-ranked slots? (EqRf, sahg). In six of eight settings, yes: they beat random, least-important and speaker-ranked swaps of the same size by 0.06 to 0.08 DNSMOS on VoiceBank and 0.19 to 0.23 on Libri1Mix, while the controls change DNSMOS by at most 0.05 (Table R3). The exception, a single-slot swap for WHAM! noise, is reported in the revision.
  • What do non-aligned slots offer over frame-level tokens? (sahg, EqRf). On the same pairs, LATTE changes the speaker by swapping just 3 out of its 250 tokens. FocalCodec needs to modify about half of its frames and stays intelligible only when both recordings say the same sentence and are time-aligned (Table R4). With a reference that says a different sentence, LATTE still changes the speaker and does not pick up the reference's words (Table R5).
  • Are the audio reconstruction quality gains in Table 1 a side effect of the higher dWER? (rdE8, EqRf, sahg). We observe that on utterances that both LATTE and FocalCodec@50 reconstruct with zero dWER, LATTE still scores 0.11 to 0.16 higher in UTMOS and DNSMOS, with 95% confidence intervals above zero.
  • What does LATTE add over ALMTokenizer? (EqRf, sahg). We would like to note that in this paper, we do not claim LATTE to be a better tokenizer. The contribution of LATTE is to show where speaker, accent and noise information ends up in this kind of tokenizer and to test it by editing. We have adapted the introduction to better reflect this.
  • What is the actual bitrate? (sahg). The 650 bps in Table 1 is nominal. Counting overlapping windows and the padding of short utterances, the rate as evaluated is 1,051 to 1,911 bps depending on the test set (Table R6). We have modified Table 1 to report this.

We thank the reviewers again for their constructive feedback, which has helped us to improve the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.