- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Laconic looks at a trade-off between token rate (ie, number of tokens transmitted per second) and token dimensionality, in the world of neural speech codecs. The paper trains models at fixed token rates and dimensionality, performing a grid search and discovering a region of consistently low error, with marked degredation as token rate gets extremely low (1 tok/s).
Critical strengths
This is potentially a very rich question, trading off temporal frequency for geometric structure. At low-enough rates, the representation has to encompass more and more temporal structure, and presumably would require higher dimensional vectors to do that well.
Critical weaknesses
-
There is extensive literature on the information content of speech for compression which should be relevant to the question at hand; some greater relationship drawn to the context beyond WavLM would be desirable. How much information is actually being carried in the tokens? What does that tell you about the allowed shape of the tokens? Are there known limits to information rate elsewhere in the speech coding literature which would be relevant to the question you are asking?
-
The motivation of the paper isn't clear. Why do the authors care whether token dimensionality is larger or smaller? Why do they care whether the token rate is higher or lower? My expectation is that the goal is to reduce context size for a downstream WavLM-like system; in which case, it would be helpful to see some end-to-end metrics that demonstrate the utility of swapping in the larger vectors at lower rate.
-
This work would benefit from a more information-theoretic discussion. Token rate and dimensionality both have implications for overall bitrate of the stream. There is extensive work on low-bitrate coding of speech signals, including noticing when a bitrate of a (sub)-vector can be reduced as low as zero (ie, dropping out entirely, and thus reducing the apparent dimension of the vector), for example in the Opus encoder. In this case, we can imagine that some lower-bound on token rate is needed to represent the signal, while the dimensionality of the representation varies dynamically as needed.
In the end, a single system is trained and tested; better theoretical grounding could help overcome that shortcoming. As it is, I'm left wondering whetehr these results are actually useful or an artifact of the particular system that was chosen for training.
Questions for authors
-
It would be helpful to better explain the purpose of this work. For example, it isn't clear to me that this is particularly helpful in the problem of low-rate speech coding (there is no discussion of overall bitrate), but seems more geared towards reducing the context-length of a downstream transformer by trying to reduce overall token count?
-
At the extreme, we can imagine sending a single second of audio by simply transposing it as a single 16k-dim'l vector... Does the system improve as we approach the dimensionality of the raw audio?
2 comments
Critical Weaknesses:
A-W1: There is extensive literature on the information content of speech for compression which should be relevant to the question at hand; some greater relationship drawn to the context beyond WavLM would be desirable. How much information is actually being carried in the tokens? What does that tell you about the allowed shape of the tokens? Are there known limits to information rate elsewhere in the speech coding literature which would be relevant to the question you are asking?
How much information is in the tokens and what is says about the allowed token shape
To quantify this, we use a single delta rule throughout: standardise each dimension, one absolute step Δ for each of the dimensions, rate = summed per-dimension empirical entropy, with the utterance tokens amortised. Measured: 5.95 bits/dim in the lower dimensions vs 4.24 in the higher dimensions.
Known limits elsewhere in the literature: We compare Laconic with other Codecs, namely Mimi and FlexiCodec in the table below. Uncoded bits: naive uncompressed representation. Entropy-coded: theoretical limit with ideal entropy coding. Gzip: actual bitrate obtained by compressing the representation with gzip.
| System | tok/s | uncoded b/s | entropy-coded bits/s | gzip b/s | WER | SIM-o |
|---|---|---|---|---|---|---|
| Laconic Q, d=64 | 3.13 | 2,191 | 1,089 | 1,175 | 3.04 | 0.514 |
| Mimi, 8 cb | 12.5 | 1,100 | 1,043 | 1,249 | 4.18 | 0.570 |
| FlexiCodec, 24 cb | 4.17 | 1,213 | 1,150 | 1,322 | 3.84 | 0.573 |
| Laconic Q, d=64 | 6.25 | 3,791 | 2,019 | 2,168 | 2.55 | 0.531 |
| Mimi, 32 cb | 12.5 | 4,400 | 4,233 | 5,119 | 2.46 | 0.696 |
We add a bitrate table which highlights the information in tokens per second. At matched bitrate (~1,050 b/s) Laconic is the most intelligible of the three (WER 3.04 vs 4.18 and 3.84) and the least speaker-faithful (0.514 vs 0.570, 0.573).
A-W2: The motivation of the paper isn't clear. Why do the authors care whether token dimensionality is larger or smaller? Why do they care whether the token rate is higher or lower? My expectation is that the goal is to reduce context size for a downstream WavLM-like system; in which case, it would be helpful to see some end-to-end metrics that demonstrate the utility of swapping in the larger vectors at lower rate.
it would be helpful to see some end-to-end metrics that demonstrate the utility of swapping in the larger vectors at lower rate.
We indeed show this in the paper on both speech reconstruction (Figure 5), ASR (Figure 3b), and Text-To-Speech (Figure 6)
A-W3: This work would benefit from a more information-theoretic discussion. Token rate and dimensionality both have implications for overall bitrate of the stream. There is extensive work on low-bitrate coding of speech signals, including noticing when a bitrate of a (sub)-vector can be reduced as low as zero (ie, dropping out entirely, and thus reducing the apparent dimension of the vector), for example in the Opus encoder. In this case, we can imagine that some lower-bound on token rate is needed to represent the signal, while the dimensionality of the representation varies dynamically as needed.
This work would benefit from a more information-theoretic discussion.
We will include an information-theocratic discussion in-line with the results and table we include above, under the first weakness.
we can imagine that some lower-bound on token rate is needed to represent the signal, while the dimensionality of the representation varies dynamically as needed.
We believe this is exactly what we show in the paper. Understanding tasks such as ASR saturate with few dimensions (Figure 4). More demanding tasks such as reconstruction need more dimensions (Figure 5).
Questions For Authors:
A-Q1: It would be helpful to better explain the purpose of this work. For example, it isn't clear to me that this is particularly helpful in the problem of low-rate speech coding (there is no discussion of overall bitrate), but seems more geared towards reducing the context-length of a downstream transformer by trying to reduce overall token count?
Yes, our work is geared towards reducing the context length of a downstream transformer by reducing overall token count. We will make this clearer in the introduction.
A-Q2: At the extreme, we can imagine sending a single second of audio by simply transposing it as a single 16k-dim'l vector... Does the system improve as we approach the dimensionality of the raw audio?
We indeed looked at representing the utterances at much higher dimensionalities, as shown in Figure 3 (a) row “WavLM-Stack”. For completeness, we also experimented with transposing all the features (whether WavLM or raw waveform samples) in a second to obtain 1 token per second at a dimensionality of 16k. That leads to much worse results: using the raw waveform at 16k gave 115.5% WER, and WavLM-Stack gave 11.7% WER, compared to Laconic WER which is below 4%.