- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes LATTE, which introduces learned discrete latent slots between a frozen FocalCodec encoder and decoder. The authors estimate slot importance using groups defined by attributes such as speaker identity, gender, accent, and noise. They then edit speech by replacing selected slots with tokens from a reference utterance. Experiments cover speech reconstruction, slot-importance analysis, denoising, and controlled voice conversion.
Critical strengths
-
The speaker-editing experiment includes informative internal controls. In Table 4, replacing five high-importance slots yields a target-speaker similarity of 90.0, compared with 67.8 for random slots and 68.1 for the least-important slots at the same swap count. This directly supports the usefulness of the importance ranking.
-
The paper connects representation analysis with actual interventions. The authors examine importance profiles across several attributes and test whether these associations support editing. They also report the loss in reconstruction intelligibility and acknowledge that speaker identity, gender, and accent remain entangled, giving a useful account of both the representation’s capabilities and its limitations.
Critical weaknesses
-
The practical editing benefit of non-temporally aligned slots over existing representations is not yet sufficiently established. Learned queries for aggregating audio information have precedents, including ALMTokenizer. An important contribution here is therefore the evidence that this slot structure makes attributes easier to analyze and edit. Appendix Figure 5 compares importance profiles with frame-level FocalCodec, but the paper does not compare their actual editing performance under a common reference condition and evaluation protocol. The controls in Table 4 show that slot selection matters within LATTE, but do not establish how much benefit comes from the non-temporally aligned representation itself. Given the reduction in reconstruction intelligibility, this trade-off needs more direct experimental support.
-
The selectivity of attribute editing is not fully established. The experiments show that selected slots affect the target attribute, but provide less complete evidence that other information is preserved. The denoising experiment compares against no-swap reconstruction without random-slot or least-important-slot controls at the same swap count. It is therefore difficult to determine whether the improvement specifically depends on the identified noise-related slots. The internal controls for speaker editing are stronger, but the source and reference utterances contain the same text. Consequently, dWER cannot distinguish preservation of source content from the introduction of content from the reference. The authors explicitly frame this as a controlled intervention, which is reasonable. Nevertheless, the current setting provides limited evidence about content preservation, leaving the broader selectivity of the representation uncertain.
Questions for authors
-
Could you provide a small-scale editing comparison between LATTE and the existing FocalCodec model using the same test utterances, reference audio, and task metrics? FocalCodec could use an editing strategy appropriate for its frame-level representation, with the relevant settings explained. This would help establish the practical benefit of the non-temporally aligned slots.
-
For denoising, could you report random-slot and least-important-slot controls with the same number of replaced slots? For speaker editing, could you provide an additional check of source-content preservation, for example using a small set of references with different text and reporting both content preservation and target-speaker similarity? These checks would help distinguish target-attribute changes from the transfer of other reference information.
-
Does the bitrate reported in Table 1 account for the tokens produced by overlapping windows and padding of short utterances? Please report the effective bitrate for each test set, as this affects the interpretation of the reconstruction quality–coding cost trade-off.
3 comments
Reply to reviewer sahg (part 2/2)
(W2, Q2) Is editing selective: do the denoising gains need the noise-ranked slots, and is source content preserved with a different reference sentence?
TL;DR: Yes: the noise-ranked slots beat the controls, and source content survives a reference that says a different sentence.
Denoising controls. Reported in full in the reply to EqRf (W1): the noise-ranked slots beat random, least-important and speaker-ranked slots in 6 of 8 settings; the exception swaps a single slot for WHAM! noise.
Non-parallel conversion. To check that content is preserved and not copied from the reference, each of the same 164 source recordings was paired with a target that differs in both speaker and sentence (fixed list, seed 0). dWER is measured against the source transcript (did the content survive?) and against the target transcript (did the target's words leak in?), in Table R5 (now Table 7 in App. G):
| Row | Sim-T | Sim-S | UTMOS | dWER vs source | dWER vs target |
|---|---|---|---|---|---|
| No swap | 71.8 | 96.2 | 4.11 | 10.38 | 127.2 |
| Guided, | 88.6 | 80.6 | 4.08 | 11.65 | 127.9 |
| Guided, | 89.2 | 80.1 | 4.07 | 14.17 | 126.8 |
The guided swap raises Sim-T by about 17 points, close to the 21 points with same-sentence pairs. dWER against the target transcript stays where it is without swapping (about 127; above 100 because the sentences are unrelated), so the target's words do not leak into the output. dWER against the source rises by 1.3 () and 3.8 () points.
(Q3) Does the Table 1 bitrate count overlapping windows and the padding of short utterances?
TL;DR: No: 650 bps is the nominal rate; counting overlap and padding, the evaluated rate is 1,051 to 1,911 bps (Table R6).
The 650 bps in Table 1 is the nominal rate of one 5 s window and does not include overlap or padding. Two things raise the actual rate: the evaluation moves the 5 s window forward 1 s at a time, so windows overlap by 4 s, and utterances shorter than 5 s still use a full window of 250 tokens. Table R6 (now Table 8 in App. H) gives the rate computed from the tokens actually produced:
| Set | Utterances | Share under 5 s | Effective bps, as evaluated (1 s hop) | Nominal bps |
|---|---|---|---|---|
| LibriSpeech test-clean | 2,620 | 42% | 1,911 | 650 |
| VoiceBank | 824 | 98% | 1,343 | 650 |
| Libri1Mix | 3,000 | 77% | 1,051 | 650 |
We note that both of these issues are consequences of the network that we designed to investigate the properties of latent tokenizers in audio, and with further engineering efforts this can plausibly be mitigated. We also note that this does not impact the main findings of how latent tokens for audio represent the information present in the audio.
Dear reviewer,
Thank you for your constructive review. We were wondering if you had any feedback for our rebuttal, given that the author-reviewer discussion period is coming to a close soon.
Best regards,
The authors
Reply to reviewer sahg (part 1/2)
We thank the reviewer for the constructive questions; we ran the three suggested comparisons.
(W1, Q1) Do non-aligned slots make editing easier than frame-level FocalCodec under the same protocol?
TLDR: Yes: LATTE changes the speaker by swapping 1.2% of its tokens, while FocalCodec needs about half of its frames.
To test the benefit of non-aligned slots, we edit the token sequences extracted using FocalCodec on the same 164 VCTK pairs, with the same reference audio and metrics as Table 4.
FocalCodec@50 also uses 250 tokens of 13 bits per 5 s (same as LATTE), so the share of swapped tokens is directly comparable to LATTE.
A share of the source's FocalCodec frames was replaced with the target's, chosen at random (10 draws) or by the same importance score (Fig. 5). Because the two recordings differ in timing, frames were aligned in two ways: by cutting both to the same length, and by time-warping (DTW), which works here because both say the same sentence. Table R4 (the importance-chosen rows are now in Table 4; the full table is Table 6 in App. F):
LATTE reaches Sim-T 89.2 by swapping 1.2% of its tokens. FocalCodec needs about half of its frames for a similar value (84.6–86.3). Without time-warping its quality drops (UTMOS 3.1–3.6) and dWER rises to 45–100; time-warping largely avoids this only because both recordings say the same sentence.
Choosing frames by importance moves the speaker no more than choosing them at random (App. F), which matches Fig. 5: FocalCodec spreads speaker information across all frames.