acceptodds
Under review as a conference paper at ICLR 2027

NanoVoco: Upsampling-Aware Rank-Width Reallocation for Lightweight Vocoding

Abstract

Deploying text-to-speech (TTS) systems on resource-constrained edge devices requires lightweight neural vocoders. Under extreme compression, reducing channel width degrades synthesis quality. Although depthwise-separable convolutions (DSCs) allow wider layers at comparable parameter budgets, replacing standard convolutions (StdCs) with DSCs degrades quality even further in our experiments. We observe that this replacement involves reallocating parameters from convolutional rank to channel width, and find that the benefits of this rank-width reallocation depend strongly on the temporal upsampling rule. Replacing zero-insertion (ZI) with sample-repetition (SR) upsampling substantially improves the benefits of rank-width reallocation. We observe this interaction across different discriminator designs, generator backbones, and parameter budgets, with the strongest effect under extreme compression. Based on this observation, we propose NanoVoco, with only 72.3K parameters and UTMOS-measured synthesis quality comparable to the evaluated Parallel WaveGAN model and higher than FreGrad. Integrated into a complete TTS pipeline, its quantized implementation enables real-time streaming speech synthesis on a microcontroller using only on-chip memory.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Neural vocoding, Speech synthesis, Efficient neural networks, On-device inference, Depthwise-Separable Convolution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Mce4OpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how the upsampling method affects the parameter reallocation benefit between convolutional rank and channel width in extremely compressed neural vocoders: under ZI, using DSC to save parameters for channel widening instead degrades quality, whereas switching to SR clearly improves the benefit. The authors define and compare rank-reduction cost, widening gain, and net reallocation benefit via A/B/C matched configurations, supported by rank sweeps, ZI→SR interpolation, architecture transfer, and multi-budget experiments, finding the interaction strongest at about 72K parameters and weaker at larger budgets. Based on this, they propose NanoVoco (SR, wide DSC refiner, DynamicTanh, 72.3K parameters), which achieves UTMOS 3.71 on LJSpeech, comparable to the evaluated Parallel WaveGAN and higher than FreGrad, and is deployed on STM32H747XI for real-time streaming synthesis using only on-chip memory. Overall, the experiments are solid and the engineering is complete, but evidence for perceptual quality and generality is insufficient, and the methodological novelty is weaker than the analysis work.

Critical strengths
  1. The experimental isolation is solid and is the strongest part of the paper: the A/B/C controlled experiments separate the rank reduction loss, the widening gain, and the net benefit, and the rank sweep shows that the benefit is all negative under ZI and all positive under SR, making the core observation highly credible.

  2. MCU deployment is a strong engineering contribution: the complete TTS pipeline actually runs on a Cortex-M7 (STM32H747XI, 480 MHz) using only on-chip Flash and SRAM; RTF 0.9876, no I2S underflow across 640 requests, and zero SDRAM usage, clearly adding to the paper's practical value.

Critical weaknesses
  1. The perceptual evaluation is not sufficient to support the core quality claims. The biggest weakness is the lack of human listening tests: the claim that a 72.3K-parameter model is comparable in quality to much larger vocoders is based mainly on UTMOS, evaluated on only 128 utterances from LJSpeech. UTMOS can serve as a screening metric, but it is not sufficient in a vocoder paper that interprets relatively small score differences as architectural improvements. Moreover, several core comparisons involve external checkpoints with different training distributions and frontends, so I suggest adding a small-scale MOS/MUSHRA test comparing at least HiFi-GAN V2, Parallel WaveGAN, and one lightweight baseline.

  2. The novelty and generality of the design principle are limited. Combining SR upsampling with separable convolution already has a precedent in nVOC-22, so the main novelty here lies in explaining and controlled-characterizing its interaction with rank-width reallocation, rather than in the component combination itself. This characterization is useful but remains largely empirical: it does not sufficiently explain why SR so markedly changes the return from widening a rank-constrained generator in the ~72K regime; the strongest effect is highly budget dependent, and most sweep and transfer experiments use only a single training seed. I therefore currently regard it as a careful characterization of an architectural phenomenon rather than a broadly established principle for efficient vocoding.

Questions for authors
  1. Human perceptual evaluation: Could the authors provide human MOS/CMOS/MUSHRA comparing NanoVoco with the strongest relevant baselines? In particular, I would like to confirm whether the roughly 3.71 vs. 3.70 UTMOS difference with Parallel WaveGAN, and the improvements over compressed alternatives, are reflected in human judgments. This would materially strengthen my confidence in the main quality claim.

  2. Generalization beyond LJSpeech: Does the upsampling–reallocation interaction persist on multi-speaker or otherwise substantially different datasets? The transfer experiments vary only architecture and adversarial supervision, not the speech distribution; one additional dataset would also help determine whether the effect is specific to LJSpeech.

  3. Relation to nVOC-22: Could the authors more clearly distinguish this paper's contribution from nVOC-22? As I understand it, nVOC-22 already combines repetition upsampling with separable convolution, whereas this paper extends separable processing more broadly and focuses on the rank-width interaction. Please clarify which architectural elements are new and which part of the contribution should primarily be regarded as analysis rather than architecture.

4 comments

Thank you so much, Mce4!

We sincerely appreciate your recognition of our experimental isolation and MCU deployment.

We fully acknowledge the weaknesses and questions you raised and have revised our manuscript to address them based on your valuable feedback.

Critical Weaknesses 1: "The perceptual evaluation is not sufficient to support the core quality claims. ... I suggest adding a small-scale MOS/MUSHRA test comparing at least HiFi-GAN V2, Parallel WaveGAN, and one lightweight baseline."

We agree that human listening tests would provide valuable perceptual evidence, and we acknowledge their absence as a limitation of this work. We are currently conducting a small-scale MOS listening test comparing HiFi-GAN V2, Parallel WaveGAN, and a lightweight baseline (V2-iSTFTNet). However, the process has taken longer than anticipated due to ethical considerations. We will report the results as soon as they become available.

In the meantime, we have expanded the objective evaluation to include M-STFT, MCD, Periodicity, ViSQOL, and PESQ, along with comparisons against lightweight iSTFT-based vocoders (see Table 1 in the revised paper). Although these objective metrics cannot replace human judgments, they provide complementary perspectives on synthesis quality and enable a more comprehensive evaluation.

Critical Weaknesses 2.1: "Combining SR upsampling with separable convolution already has a precedent in nVOC-22, so the main novelty here lies in explaining and controlled-characterizing its interaction with rank-width reallocation, rather than in the component combination itself..."

nVOC-22 finds that SR-DSC is more effective than ZI-StdC. However, it does not experimentally investigate what interaction exists between the upsampler and StdC or DSC, why it occurs, when it is beneficial, or how we can use it. This work sets out to answer these questions.

(Analytically)

  1. "What interaction between upsampler and StdC or DSC?" Our StdC-to-DSC experiments show that SR exhibits a lower rank-1 cost and a higher widening gain than ZI, which we term the upsampling–reallocation interaction.
  2. "Why does it occur?" Our ZI-to-SR experiments support a possible explanation: the upsampler’s built-in filtering may help DSCs utilize additional channel width more effectively while alleviating the quality penalty of kernel-rank reduction.
  3. "When is it benefitial?" This interaction is observed on a multi-speaker dataset (VCTK), under different supervision (Avocodo), with a different architecture (BigVGAN), and across a wider range of parameter budgets (from 72K to 926K).

(Architecturally)

  1. "How can we use it?" We propose NanoVoco, a 72K-parameter vocoder that remains compatible with larger vocoder models.

Critical Weaknesses 2.2: "... This characterization is useful but remains largely empirical: it does not sufficiently explain why SR so markedly changes the return from widening a rank-constrained generator in the ~72K regime..."

In the revised paper, we propose a supported explanation for why SR has lower rank-reduction cost and higher width-expansion gain (upsampling-reallocation interaction)

From a signal-processing perspective, sample repetition can be modeled as zero insertion followed by a fixed zero-order hold filter, which is essentially a low-pass filter. To find whether the interaction is related to the upsampler's built-in filtering, we design ZI-to-SR experiments.

The results show that StdC performance remains relatively stable as the upsampler progressively introduces fixed filtering, suggesting that this filtering provides little additional quality benefit to StdC. By contrast, both DSC configurations improve substantially, particularly the widened DSC.

These observations suggest that the importance of StdC's full filtering flexibility may diminish when the upsampler provides built-in filtering. Meanwhile, reallocating parameters to channel-width expansion appears to be more beneficial than retaining the full filtering flexibility under this setting.

Together, these observations are consistent with our proposed explanation for the upsampling–reallocation interaction (revised paper, Figs. 4 and 5).

Question 1: "Could the authors provide human MOS/CMOS/MUSHRA comparing NanoVoco with the strongest relevant baselines?"

See our response to Critical Weaknesses 1.

Question 2: "Does the upsampling–reallocation interaction persist on multi-speaker or otherwise substantially different datasets? ..."

We trained six configurations on the multi-speaker VCTK dataset, using 100 speakers for training and nine held-out speakers for evaluation, following a setup similar to that of HiFi-GAN (see Fig. 6(a) in the revised paper).

ModelUTMOS (mean ± std)
ZI-StdC2.1261 ± 0.5081
ZI-DSC-fixwidth1.3803 ± 0.1805
ZI-DSC2.6836 ± 0.3882
SR-StdC2.2377 ± 0.4396
SR-DSC-fixwidth1.5658 ± 0.3278
SR-DSC2.8799 ± 0.3885

(Under ZI)

the rank-reduction cost is ZI-StdC - ZI-DSC-fixwidth = 2.1261 - 1.3803 = 0.7458

the widening gain is ZI-DSC - ZI-DSC-fixwidth = 2.6836 - 1.3803 = 1.3033.

(Under SR)

the rank-reduction cost is SR-StdC - SR-DSC-fixwidth = 2.2377 - 1.5658 = 0.6719 (< 0.7458)

the widening gain is SR-DSC - SR-DSC-fixwidth = 2.8799 - 1.5658 = 1.3141 (> 1.3033).

These results show that the upsampling–reallocation interaction persists on the multi-speaker VCTK dataset, with SR exhibiting a lower Rank-1 Cost and a slightly higher Widening Gain. However, the magnitude of the interaction is relatively small, particularly for Widening Gain. This suggests that the benefit of the interaction may also depend on dataset characteristics, as well as on the choice of upsampler.

Question 3: "Could the authors more clearly distinguish this paper's contribution from nVOC-22? Please clarify which architectural elements are new and which part of the contribution should primarily be regarded as analysis rather than architecture."

See our response to Critical Weaknesses 2.1.

Reviewer_Mce4OpenReview ↗

Raising Score and Noting Assessment Uncertainty

Thank you to the authors for revising the paper and providing a response.

I will raise my score. My main basis is Fig. 2 and Fig. 3 in the revised manuscript: these two figures clearly address the main concerns I raised earlier, and the authors' additional clarifications are convincing to me. I therefore believe the revision represents a substantive improvement over the initial submission, and raising my score is appropriate.

At the same time, I would like to candidly note the uncertainty in my assessment, for the authors and the AC's reference:

  1. Domain familiarity: I am not especially familiar with the specific subfield of this paper. My review therefore focuses mainly on the dimensions I can reliably assess, such as writing clarity, completeness of experimental reporting, consistency between figures and claims, and reproducibility information; my confidence is more limited on judgments such as domain novelty and the marginal contribution relative to the most related work.

  2. The field is relatively old: This direction has a long research history, and judging whether the incremental contribution of this paper relative to existing work is sufficiently significant requires a deep grasp of the field's lineage. I am not fully confident on this point, so my score should be read as a judgment based on the dimensions noted above, rather than a comprehensive verdict on the paper's contribution to the field.

In light of the above, I suggest the AC appropriately weigh the opinions of reviewers more familiar with this subfield when calibrating novelty and contribution judgments; my raised score mainly reflects that the specific concerns addressed by Fig. 2/Fig. 3 have been resolved. If the authors or the AC consider it helpful, I can further specify exactly which points changed my view.

Thank you to the authors for their careful revision, and to the AC for organizing the review.

Dear Mce4,

Thank you so much for your acknowledgment of our revised manuscript!

We are very glad that our response helped clarify our work.

We sincerely appreciate your time and valuable insights, which have helped us improve our paper.

Best Regards, Authors

Reviewer_ydowOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper describes an ultra lightweight mel spectrogram inversion module for speech reconstruction. The primary innovation is the modification of the up sampling step in hifi gan, using sample-and-hold updating instead of zero insertion. This change allows the use of lighter weight depth convolutions instead of full convolutions, which greatly reduces the model parameter count. Some of the reduced parameter count is spent widening the network, recovering some quality.

Critical strengths

I really appreciate seeing cool approaches to lightweight modeling! Down and Up sampling are a common operations in lightweight models, and I expect that this observation could improve model quality at small size in many contexts.

Critical weaknesses

That said, the writing in this paper could use some serious work. Jargon is introduced early in the introduction with no explanation (rank reduction cost, widening benefit), and no real explanation of the basic model architecture is given.

At the same time, the primary to for confirming the observations is needlessly complex and doesn't really provide that much additional insight. All we really need to see here is that the smaller model with the three changes applied (sample reporting, depth wise convs, and modifying model width) give good quality relative to the baseline, along with some simple ablations. The rank reduction scheme requires a lot of explanation and doesn't provide much insight.

Instead, I would love to hear /why/ the DSC seems to require the sample replacement strategy. The motivation for zero insertion (presumably) comes from signal processing, where it is easy to filter out the high frequency components introduced by zero insertion, while sample repetition introduces low frequency artifacts which are harder to correct. This suggests that the full convolution (or pair of full convolutions) may have sufficient power to learn the needed low pass filter, while the DSC might lack the needed flexibility. A more detailed understanding of what's actually needed/missing (rather than an empirical rank expiration) might suggest further low cost architectural improvements.

Questions for authors

None at this time. However, please improve the introduction (explain motivation, provide overview of the model and methods sufficient to understand the what the paper is doing), and check the paper for typos and latex errors.

2 comments

Thank you so much, ydow!

We sincerely appreciate your recognition of our lightweight modeling!

We fully acknowledge the weaknesses you raised and have revised our manuscript to address them based on your valuable feedback.

Critical Weaknesses 1: "... Jargon is introduced early in the introduction with no explanation (rank reduction cost, widening benefit) ..."

We have added the necessary explanations in the Introduction. The terms "Rank-1 Cost" and "Widening Gain" are now introduced as follows:

"... we decompose the transition from StdC to DSC into two steps. First, we replace StdC with DSC while keeping the channel width fixed, a process called rank reduction, which reduces the number of temporal filters. We refer to the resulting quality penalty as the Rank-1 Cost. Second, we reinvest the parameters saved by rank reduction into channel-width expansion. We refer to the resulting quality improvement as the Widening Gain. ..."

We also refer readers to a diagram illustrating where these terms come from and what they represent. More detailed explanations are provided in Section 4, with the mathematical definitions given in Appendix B.

Critical Weaknesses 2: "... and no real explanation of the basic model architecture is given." and "please provide overview of the model and methods sufficient to understand what the paper is doing."

  • We redrew Fig. 1 to illustrate the core observation and note that this paper also investigates why, when, and how it occurs.
  • We added Fig. 2 to illustrate the exploration process of lightweight modeling, leading to the motivation for this work.
  • In Figs. 3 and 4, we added illustrations of how we conduct the corresponding experiments.
  • We added Fig. 5 to illustrate an explanation of the core interaction.
  • We better organized Figs. 6 and 7 for clearer illustration.
  • We also added a detailed HiFi-GAN architecture diagram in Appendix A (Fig. 8), which shows our baseline model. All our adjustments are based on this model.

Critical Weaknesses 3: " ... the primary to for confirming the observations is needlessly complex and doesn't really provide that much additional insight. ..." and "... The rank reduction scheme requires a lot of explanation and doesn't provide much insight."

We really appreciate your feedback! We agree that the rank-reduction experiments are unnecessary, so we removed this part from our storyline. We also improved our storyline; see our response to Critical Weaknesses 5.

Critical Weaknesses 4: "Instead, I would love to hear /why/ the DSC seems to require the sample replacement strategy. ..."

Same-width DSC has more constrained temporal filtering capabilities than StdC. While StdC can learn more flexible temporal filters, it remains unclear whether such flexibility is always necessary. The main idea of our work is to identify settings where this additional filtering flexibility may be less important.

The upsample-filtering application is the case we found, where part of the filtering can be handled by the upsampler. From a signal-processing perspective, sample repetition can be modeled as zero insertion followed by a fixed zero-order hold filter, which is essentially a low-pass filter.

Our ZI-to-SR experiments (revised paper, Fig. 4) show that StdC performance remains relatively stable as the upsampler progressively introduces fixed filtering, suggesting that this filtering provides little additional quality benefit to StdC. By contrast, both DSC configurations improve substantially, particularly the widened DSC.

These observations suggest that the importance of StdC's full filtering flexibility may diminish when the upsampler provides built-in filtering. Meanwhile, reallocating parameters to channel-width expansion appears to be more beneficial than retaining the full filtering flexibility under this setting.

Together, these observations are consistent with our proposed explanation for the upsampling–reallocation interaction (revised paper, Figs. 4 and 5).

Critical Weaknesses 5: "However, please improve the introduction (explain motivation, provide overview of the model and methods sufficient to understand the what the paper is doing)..."

We now have a clearer motivation and storyline, as follows:

A lightweight vocoder is needed for TTS edge deployment.

  • This work sets out to explore a lightweight modeling method.
  • nVOC-22 finds that SR-DSC is more effective than ZI-StdC, but it does not investigate what interaction exists between the upsampler and StdC or DSC, why it occurs, when it is beneficial, or how we can use it. This work sets out to answer these questions.
  • To understand why SR-DSC is better than ZI-StdC, we designed a StdC-to-DSC experiment and pointed out an upsampling-reallocation interaction.
  • To explore an explanation for the interaction, we designed a ZI-to-SR experiment and the result suggests a possible explanation: the upsampler's built-in filtering may help DSCs utilize additional channel width more effectively while alleviating the quality penalty of kernel-rank reduction.
  • To clarify the scope of this interaction, we experimented with different datasets, supervision strategies, architectures, and parameter budgets.
  • To demonstrate how can we use this interaction, we propose NanoVoco and evaluate its performance.

We also updated our visual presentation as mentioned in our response to Critical Weaknesses 2.

Critical Weaknesses 6: "... and check the paper for typos and latex errors."

We double-checked those errors in the revised manuscript.

Reviewer_keLjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why compressing a time-domain GAN vocoder (HiFi-GAN) to ∼72K\sim 72\text{K} parameters via depthwise-separable convolutions (DSC) and channel widening (w0=32→84w_0 = 32 \to 84, termed rank-width reallocation) collapses synthesis quality under standard transposed-convolution / zero-insertion (ZI) upsampling (UTMOS drops from 2.752.75 to 1.411.41). Using an A/B/C causal decomposition into a rank-reduction cost (A→BA \to B at fixed width w0=32w_0 = 32) and a channel-widening gain (B→CB \to C at rank 1), the authors show that replacing ZI with parameter-free sample-repetition (SR) upsampling halves the rank-reduction cost (−1.46-1.46 to −0.73-0.73 UTMOS) and restores the widening gain (+0.11+0.11 to +1.40+1.40 UTMOS), yielding a net reallocation benefit of +0.66+0.66 UTMOS (SR-DSC reaches 3.333.33 UTMOS at 69.1K69.1\text{K} parameters). They further evaluate intermediate rank factorizations, ZI-to-SR linear interpolations, transfer to Avocodo supervision and a BigVGAN-derived backbone, and a parameter-budget sweep (72K–926K72\text{K}\text{--}926\text{K}). Finally, replacing all 77 intermediate LeakyReLU activations in SR-DSC with learnable DynamicTanh (DyT) layers yields NanoVoco (72,31072{,}310 parameters, 3.713.71 UTMOS on LJSpeech), which is quantized to A16W8 and deployed as part of an end-to-end on-device TTS pipeline (Flite 2.2 + EfficientSpeech + NanoVoco) on an STM32H747XI Cortex-M7 microcontroller.

Critical strengths
  1. Clean Empirical Decomposition and Transparent Reporting: The A/B/C experimental design (StdC w0=32→w_0=32 \to DSC w0=32→w_0=32 \to DSC w0=84w_0=84) cleanly separates the effect of kernel rank reduction from channel widening under a fixed parameter budget. The paper is also commendable for its thoroughness in reporting negative/boundary results in the appendices (such as DyT not improving ZI baselines, non-monotonic scaling across larger budgets, and A16W8 quantization degradation).
  2. End-to-End Microcontroller TTS Demonstration: Deploying a complete text-to-speech pipeline (Flite 2.2 + EfficientSpeech + NanoVoco) strictly within the on-chip Flash and SRAM of an STM32H747XI Cortex-M7 @ 480 MHz provides concrete systems validation beyond simulated parameter counts.
Critical weaknesses
  1. Absence of a Theoretical/Mechanistic Explanation & Unisolated Post-Expansion vs. Residual Block Effects:
    • Despite its analytical framing, the paper offers no mathematical, polyphase-filtering, spectral, or gradient-flow explanation for why ZI causes rank-1 DSC refiners to collapse or why SR fixes it. In Section 4.2 (Fig. 5), the widening gain jumps from +0.11+0.11 at λ=0\lambda = 0 (ZI) to +1.30+1.30 at λ=0.25\lambda = 0.25 (93%93\% of the full SR gain at the first nonzero step!) and stays flat (1.30,1.25,1.30,1.401.30, 1.25, 1.30, 1.40) for λ≥0.25\lambda \ge 0.25. At λ=0\lambda = 0, r−1r - 1 of every rr samples entering the post-expansion depthwise filter are exact zeros (so only 2 of 16 filter taps touch nonzero values at r=8r = 8, and the 1×11\times 1 pointwise conv applies identical channel weights across all rr polyphase offsets).
    • Furthermore, by grouping the 1 post-expansion convolution and 18 dilated residual-block convolutions per stage into a single "refiner," the A/B/C study never tests whether ZI-DSC fails simply because the 4 post-expansion convolutions apply rank-1 depthwise filters directly to sparse zero-inserted sequences (i.e., the paper never evaluates ZI with dense StdC in the 4 post-expansion layers and wide DSC in the 72 residual convolutions).
  2. Exclusive Reliance on 16 kHz UTMOS22-strong Without Human Listening Tests or High-Frequency Spectral Metrics:
    • No human subjective evaluation (MOS, CMOS, SMOS, or MUSHRA) was conducted anywhere in the study (lines 495, 981), and all evaluations use 128 utterances from a single speaker (LJSpeech-1.1).
    • All perceptual quality claims in Figures 2–7 and Tables 1, 4–7, 10, and 14 rely solely on UTMOS22-strong (wav2vec 2.0-based), which downsamples 22.05 kHz22.05\text{ kHz} audio to 16 kHz16\text{ kHz} (8 kHz8\text{ kHz} Nyquist frequency) before scoring—completely discarding the 8.0–11.025 kHz8.0\text{--}11.025\text{ kHz} band where upsampling imaging artifacts, tonal replicas, and high-frequency phase distortions occur in 22.05 kHz22.05\text{ kHz} vocoders. Standard objective spectral metrics (M-STFT, Mel L1/MCD, PESQ, F0F_0/V-UV error) are also omitted from the main experiments and Table 1.
  3. Vanishing Net Gains Outside 72K HiFi-GAN Because SR Consistently Degrades StdC (Tables 6 & 7):
    • At ∼250K\sim 250\text{K} and ∼535K\sim 535\text{K} parameters (Table 6), ZI-DSC (3.5993.599 and 3.7953.795) already outperforms ZI-StdC (3.3673.367 and 3.6983.698), showing that ZI rank-width reallocation does not fail above 72K72\text{K}.
    • Across all four parameter budgets and BigVGAN, SR consistently degrades standard dense convolutions (StdC) by −0.091-0.091 to −0.190-0.190 UTMOS. Because Eq. (4) measures the SR reallocation benefit relative to SR-StdC, a large fraction of the reported positive interaction is driven by SR hurting StdC. On BigVGAN (Table 7), the proposed SR-DSC (2.507±0.36942.507 \pm 0.3694) beats the unmodified standard ZI-StdC baseline (2.500±0.33042.500 \pm 0.3304) by only +0.007+0.007 UTMOS, and on 926K HiFi-GAN (Table 6), SR-DSC (4.1124.112) beats ZI-StdC (4.0874.087) by only +0.025+0.025 UTMOS.
  4. MCU Flash Bloat (870.6 KB870.6\text{ KB} Flash for a 72.3K72.3\text{K}-Param Model), Missing CPU/MCU Baselines, and Redundant Bias Parameters:
    • CPU & MCU Overhead of Wide DSC + DyT: On desktop CPU (Table 1), NanoVoco (72.3K72.3\text{K} params, 42.98×42.98\times real time) is only 10.2%10.2\% faster than HiFi-GAN V2 (926K926\text{K} params, 39.00×39.00\times real time) despite having 12.8×12.8\times fewer parameters, and lightweight Fourier/hybrid vocoders (Vocos, iSTFTNet) as well as the authors' own 71.8K71.8\text{K} ZI-StdC (w0=32w_0 = 32) and 69.1K69.1\text{K} SR-DSC (LeakyReLU) controls are omitted from Table 1 and MCU benchmarks. On the STM32H747XI (Table 13), NanoVoco (72,31072{,}310 params) occupies 870,632 bytes870{,}632\text{ bytes} of Flash (12.04 bytes/parameter12.04\text{ bytes/parameter}!)—3×3\times larger than unquantized FP32 weights and 2.96×2.96\times larger than EfficientSpeech (294.1 KB294.1\text{ KB} Flash for 266.4K266.4\text{K} params)—because 576,666 bytes576{,}666\text{ bytes} (66.2%66.2\% of NanoVoco's Flash) are consumed by DyT lookup tables. The MCU firmware saturates 99.39%99.39\% of Flash and 97.62%97.62\% of SRAM, runs at a razor-thin RTF=0.9876\text{RTF} = 0.9876 (max 0.99170.9917) with a 744.1 ms744.1\text{ ms} startup delay, and drops to 3.2273.227 UTMOS / 9.599%9.599\% WER end-to-end (Table 14).
    • Distorted Rank Sweep & Redundant Biases (Eqs. 2, 8–10, Tables 2–3): In Table 3, 14 of 16 blocks at ρ=75%\rho = 75\% fall back to dense StdC (so width grows 32→33→34→52→8432 \to 33 \to 34 \to 52 \to 84), and because the implementation adds R~ℓ(Cin,ℓ+Cℓ)\widetilde{R}_\ell (C_{\text{in},\ell} + C_\ell) linearly redundant biases per factorized layer (omitted in Eqs. 2 & 10), several factorized layers in Table 3 (and conv_post in Avocodo/BigVGAN) have more parameters than a dense layer while having lower rank.
Questions for authors
  1. Isolating Post-Expansion vs. Residual Convolutions Under ZI (Clarification): What happens at ∼72K\sim 72\text{K} parameters under ZI if only the 4 post-expansion convolutions remain dense (StdC, which takes very few parameters at narrow stride stages) while the 72 residual-block convolutions use wide DSC? Does keeping the 4 post-expansion convolutions dense already eliminate most of the ZI-DSC collapse?
  2. Full-Bandwidth (22.05 kHz22.05\text{ kHz}) Objective Metrics & Lightweight Baselines (Clarification): Since UTMOS22-strong resamples to 16 kHz16\text{ kHz} (ignoring 8–11.025 kHz8\text{--}11.025\text{ kHz}), can you report full-bandwidth 22.05 kHz22.05\text{ kHz} M-STFT, Log-Mel L1 / MCD, and F0F_0 periodicity errors across the A/B/C models and Table 1, as well as comparing against lightweight iSTFT vocoders (Vocos / iSTFTNet) and your own 71.8K71.8\text{K} ZI-StdC and 69.1K69.1\text{K} SR-DSC (LeakyReLU) models on CPU/MCU latency and memory?
  3. Why Do DyT Lookup Tables Require 576,666 Bytes576{,}666\text{ Bytes} of Flash (Clarification)? In Table 13b, DyT lookup tables consume 576,666 B576{,}666\text{ B} of Flash for 77 layers (1,5661{,}566 channels). How are these tables indexed and parameterized, and why not use a single shared unit-tanh⁡\tanh lookup table (e.g., 2–4 KB2\text{--}4\text{ KB}) with per-channel affine scaling α,γ,β\alpha, \boldsymbol{\gamma}, \boldsymbol{\beta} in SRAM/registers instead of 576.7 KB576.7\text{ KB} of Flash?
  4. Streaming Chunk Size on the MCU (Clarification): What mel-frame chunk size and receptive-field causal/ring-buffer padding strategy does NanoVoco use on the STM32H747XI to achieve bitwise parity with full-utterance host A16W8 inference?
7 comments

Thank you so much, keLj!

We sincerely appreciate your recognition of our Empirical Decomposition, Transparent Reporting, and End-to-End Microcontroller TTS Demonstration.

We fully acknowledge the weaknesses and questions you raised and have revised our manuscript to address them based on your valuable feedback.

Critical Weaknesses 1.1: "the paper offers no mathematical, polyphase-filtering, spectral, or gradient-flow explanation for why ZI causes rank-1 DSC refiners to collapse or why SR fixes it."

We added a possible mechanism behind this interaction. (See revised paper section 5) From a signal-processing perspective, SR can be viewed as ZI followed by fixed filtering, which may reduce the filtering demands on learned convolutions. To investigate this effect, we progressively transition from ZI to SR by increasing the values at inserted positions.

The results show that introducing modest nonzero values rapidly recovers most of the Widening Gain, while the Rank-1 Cost decreases more gradually as the transition continues. These observations suggest that upsampler's built-in filtering may help DSC refiners utilize additional channel width more effectively while alleviating the quality penalty of rank reduction.

Critical Weaknesses 1.2: “the paper never evaluates ZI with dense StdC in the 4 post-expansion layers and wide DSC in the 72 residual convolutions.”

We added this partial-DSC ablation. (See revised paper: Fig. 2(c)) Keeping the four post-expansion convolutions dense while using the DSC Resblock reduces the initial width from 84 to 60 at ~72K-parameter budget. Its UTMOS is 1.43, compared with 1.41 for fully DSC and 2.75 for the original ZI–StdC configuration. Thus, retaining these dense post-expansion layers does not substantially recover the lost quality.

Critical Weaknesses 2.1: "No human subjective evaluation (MOS, CMOS, SMOS, or MUSHRA) was conducted anywhere in the study (lines 495, 981), and all evaluations use 128 utterances from a single speaker (LJSpeech-1.1)."

We agree that human listening tests would provide valuable perceptual evidence, and we acknowledge their absence as a limitation of this work. We are currently conducting subjective listening experiments, but the process has taken longer than anticipated due to ethical considerations. We will report the results as soon as they become available.

In the meantime, we have expanded the objective evaluation to include M-STFT, MCD, Periodicity, ViSQOL, and PESQ, along with comparisons against more lightweight iSTFT-based vocoders (see Table 1 in the revised paper). Although these objective metrics cannot replace human judgments, they provide complementary perspectives on synthesis quality and enable a more comprehensive evaluation.

In addition, we conducted experiments on the multi-speaker VCTK dataset to examine whether the core interaction persists (see Fig. 6).

Critical Weaknesses 2.2: "Standard objective spectral metrics (M-STFT, Mel L1/MCD, PESQ, F0 errors) are also omitted from the main experiments and Table 1."

We have added M-STFT, MCD, and Periodicity (F0 errors) as suggested, along with ViSQOL and PESQ (see Table 1 in the revised paper). We have also documented the evaluation protocols, including the full-bandwidth STFT resolutions and metric implementation details (see Appendix D).

NanoVoco achieves the best UTMOS, M-STFT, Periodicity, ViSQOL, and PESQ scores among the ablation models, despite its lower RTFX of 42.98.

Notably, the quality collapse observed in Fig. 2 is most pronounced in UTMOS rather than uniformly reflected across all metrics. Compared with ZI-StdC, ZI-DSC exhibits substantially lower UTMOS and worse M-STFT and Periodicity scores, but achieves better MCD, ViSQOL, and PESQ scores. This discrepancy highlights that different metrics capture different aspects of synthesis quality.

Critical Weaknesses 3.1: "At 250K and 535K parameters (Table 6), ZI-DSC already outperforms ZI-StdC, showing that ZI rank-width reallocation does not fail above."

That's a correct observation!

We also observe that the Reallocation Benefit under ZI fluctuates as the parameter budget increases (see the black curve in Fig. 7(b)). Therefore, we do not claim that Rank-Width Reallocation always fails under ZI. Instead, we focus on our finding that SR consistently achieves a higher Reallocation Benefit than ZI across the evaluated parameter budgets (see Fig. 7(b)).

Rank-Width Reallocation yields positive benefits at some parameter budgets but negative benefits at others. One possible explanation is that its effectiveness depends on multiple factors. Our experiments suggest that the choice of upsampler is an important factor in the evaluated settings.

Critical Weaknesses 3.2: "Across all four parameter budgets and BigVGAN, SR consistently degrades standard dense convolutions (StdC)."

That's also a correct observation!

From a signal-processing perspective, SR is equivalent to ZI followed by a fixed zero-order-hold filter, which suppresses spectral images but also introduces some passband attenuation. Our ZI-to-SR experiments (Revised paper, Fig. 4(c), blue curve) show that introducing this fixed filtering does not improve, and may even slightly reduce, the UTMOS performance of StdC refiners.

One possible reason why SR hurts StdC is that StdCs already have sufficient flexibility to learn appropriate temporal filters under ZI. Therefore, the additional fixed filtering introduced by SR may be unnecessary and could impose excessive smoothing, slightly degrading synthesis quality.

In contrast, SR improves DSC because DSCs have more constrained temporal filtering capabilities than StdCs. SR's built-in filtering may reduce the filtering burden on DSC refiners, improving their performance. Moreover, the additional channel width enabled by DSCs may be utilized more effectively under SR, leading to a larger benefit.

These hypotheses are consistent with the observations from our ZI-to-SR experiments (see Figs. 4 and 5 in the revised paper).

Critical Weaknesses 4.1: "lightweight Fourier/hybrid vocoders (Vocos, iSTFTNet) as well as the authors' own ZI-StdC and SR-DSC (LeakyReLU) controls are omitted from Table 1 and MCU benchmarks. "

We have added the suggested lightweight Fourier/hybrid vocoders, including Vocos, iSTFTNet, and LightVoc, as well as our ZI-StdC and SR-DSC (LeakyReLU) controls, to Table 1.

Table 1 shows that iSTFT-based vocoders achieve high computational efficiency. However, these models could not be deployed on our target MCU due to insufficient on-chip memory. In contrast, NanoVoco prioritizes parameter efficiency while maintaining competitive synthesis quality and enabling real-time inference on our target MCU.

In addition, we report MCU inference throughput and memory usage for NanoVoco, ZI-StdC, and SR-DSC (LeakyReLU) in Table 2. (also see below:)

ModelVocoder RTFX ↑TTS RTFX ↑Flash usage (KiB) ↓Size reduction
NanoVoco1.0911.0131781.0Baseline
SR-DSC1.0050.9401179.2-601.8 (-33.8%)
ZI-StdC1.0350.966922.2-858.8 (-48.2%)

SR-DSC reduces the Flash usage by 601.8 KiB, mainly because it replaces the DyT lookup tables, which occupy approximately 563.2 KiB, with LeakyReLU. Although LeakyReLU is mathematically simpler, NanoVoco evaluates DyT using highly optimized table lookups. In our MCU implementation, these lookups are slightly faster than the sign checking and scaling required by LeakyReLU. Therefore, SR-DSC uses substantially less storage but is slightly slower than NanoVoco.

ZI-StdC further reduces the Flash usage by removing the DyT tables and using narrower dense convolutional blocks. It performs approximately 31.6% fewer MAC operations than NanoVoco. However, fewer MACs do not directly translate into lower latency on a microcontroller. The narrow layers provide less parallel work, while data rearrangement, quantization, loop control, and memory access still introduce similar fixed costs. NanoVoco's wider DSC layers make better use of SIMD instructions and reuse intermediate data more efficiently. Consequently, ZI-StdC remains approximately 5.1% slower than NanoVoco in vocoder throughput.

Critical Weaknesses 4.2.1: "In Table 3, 14 of 16 blocks at ρ = 75% fall back to dense StdC"

We agree that the nominal rank ratio does not correspond to a uniform reduction in the actual ranks of all convolutional layers. Due to differences in kernel sizes and channel widths, as well as the layer-wise dense-fallback rule, many layers remain standard convolutions at intermediate nominal rank ratios. Therefore, the rank sweep does not represent a smooth, uniformly controlled transition in kernel rank, which limits its interpretability.

Since the upsampling-reallocation interaction is already supported by the StdC-to-DSC experiment (as shown in Fig. 3), we have removed the rank sweep from the revised manuscript.

Critical Weaknesses 4.2.2: "because the implementation adds R~ℓ(Cin,ℓ+Cℓ)\widetilde{R}\ell (C{\text{in},\ell} + C_\ell) linearly redundant biases per factorized layer (omitted in Eqs. 2 & 10), several factorized layers in Table 3 (and conv_post in Avocodo/BigVGAN) have more parameters than a dense layer while having lower rank."

Thank you for pointing out that unnecessary parameter!

We double-checked the implemented code and found that our dsc indeed involved redundant bias, because we chose (bias = True) for each depthwise convolution and pointwise convolution. For NanoVoco, we identified 1,641 redundant depthwise-bias parameters, representing approximately 2.27% of the reported 72,310 parameters.

The same situation is also in our Avocodo, BigVGAN settings. We also acknowledge that factorizing the single-output conv_post layers does not provide a rank-reduction or parameter-efficiency advantage, since their kernel slices already have a maximum rank of one.

Importantly, all reported parameter counts include these biases, and all experimental results were obtained using the original implementations. Therefore, this inefficiency does not introduce an error into the reported parameter counts or invalidate the measured comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.