- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a memory-efficient fine-tuning method for BitNet models with ternary weights, targeting deployment on edge devices. It addresses the memory overhead of retaining full-precision latent weights during fine-tuning. The key idea is that latent weights close to the quantization boundaries are more likely to change their ternary values under perturbations. The method therefore prioritizes these weights for zeroth-order updates and progressively freezes weights that move away from the boundaries, reducing the number of full-precision parameters that must be stored. Latent weights outside the active set no longer require full-precision copies, further reducing memory requirements.
Critical strengths
- The parameter-selection criterion is directly tied to the ternary quantization mechanism, providing a clear rationale for selecting the active weights.
- Mask construction avoids the initial backpropagation overhead and can be performed offline.
- The method achieves substantial reductions in the fine-tuning memory footprint.
Critical weaknesses
- Permanent freezing may prematurely exclude useful parameters. As the model parameters and quantization thresholds evolve, weights excluded earlier may become useful later. However, the algorithm provides no mechanism for reactivating them.
- Evidence for practical edge deployment and efficiency remains incomplete. The paper does not sufficiently establish training latency, energy consumption, or overall benefits on real edge hardware. Zeroth-order optimization requires multiple forward passes, so a smaller memory footprint does not necessarily translate into shorter training time.
- Performance robustness under extreme sparsity is insufficiently evaluated. The main experiments use a trainable latent-weight fraction of ρ = 0.05, while ρ = 0.01 is discussed only in the memory-footprint analysis. The paper does not systematically characterize performance degradation at lower trainable fractions or provide trade-off curves relating sparsity, performance, and memory footprint.
- Generalization to larger models remains unverified. Evaluation is limited to models with 1B–3B parameters. Given the limited availability of larger open-source BitNet models, it remains unclear whether the findings extend to models with 7B parameters or more. Task coverage is also limited to classification, instruction following, and mathematical reasoning.
Questions for authors
Under the same trainable-parameter fraction, number of perturbations, and hyperparameter-tuning budget, how large and consistent are the gains from boundary-proximity selection over random selection? Do the improvements arise from selecting more useful weights, or primarily from reducing the number of trainable parameters and the variance of the zeroth-order gradient estimator? Does updating weights that are more likely to cross quantization boundaries necessarily lead to greater reductions in the task loss?
W1 — No weight reactivation
This is a design choice. In fact, we considered allowing frozen weights to re-enter the active set, but this requires continually monitoring all latent weights for boundary proximity, reintroducing the cost freezing is meant to avoid. The one-way, shrinking mask is what preserves TerMeZO's memory savings. In practice, the initial fraction ρ0 should be set by examining the latent-weight distribution around the decision boundaries, and selecting based on flip probability under perturbation.
W2 — Practical gains beyond memory on edge deployment
Our motivation is primarily about memory. We aim to make fine-tuning possible under tight memory budgets, so that fine-tuning a heavily quantized model costs roughly as much memory as running inference on it. As shown in Tables 2 and 4, the memory savings are large enough that TerMeZO can fine-tune a bigger and more accurate model using far less memory than BP. In particular, TerMeZO fine-tunes Falcon-E-3B with 1.3 GB of peak memory, while BP needs 16.1 GB for Falcon-E-1B.
A smaller memory footprint implies that fine-tuning can run on smaller, lower-power devices, which may bring energy benefits, although measuring them would require a hardware-specific evaluation. We deem this to be beyond the scope of this work. We have made these points clearer in the revised paper.
In terms of latency, we acknowledge that this is a limitation of the proposed scheme as well as all ZO-based methods. In this regard, we note that recent works on accelerating ZO optimization, such as adaptive update rules, apply directly to our method and could help close the latency gap with BP. ZO methods also work with non-differentiable objectives and RL-based fine-tuning (Qiu et al., 2025), which gives further reasons to use them.
W3 — Ablation on ρ0
We have added two ablations that we agree were missing. The first, included in Figure 1, studies the effect of the fraction ρ0 of latent weights included in the mask, with ρ0=0 corresponding to fine-tuning only the full-precision parameters. The second, included in Table 5, compares fine-tuning only the ternary weights against fine-tuning only the full-precision weights (norm layers, embeddings, and the LM head).
Effect of ρ0 on TerMeZO. GSM8K test accuracy (%) (Figure 1 in the revised paper).
Ablation on Falcon-E-3B, GSM8K. Test accuracy (%) (Table 5 in the revised paper).
W4 — Generalization to larger models
We agree with the reviewer that testing scalability is important. The larger publicly available ternary models (e.g., Bonsai) are released without their latent weights, so we could not evaluate them. Instead, we added experiments on Qwen3-8B with INT4 and FP4 weight quantization on PubMedQA, a biomedical task (Appendix H.3). These results provide preliminary evidence that TerMeZO extends to larger models and to other quantization schemes.
Test accuracy (%) on PubMedQA dataset for Qwen3-8B. The model weights are quantized to INT4 and FP4 before fine-tuning.
Q1 — Boundary vs. random selection
We thank the reviewer for this question. We ran the suggested test, including S-MeZO with random selection under the same settings (ρ, number of perturbations, tuning budget) using the Falcon-E-1B model on two datasets, GSM8K and MagiCoder. As shown in the table below, the proposed boundary-proximity selection consistently outperforms random selection at the same ρ. We added this to the revised paper in Appendix H.2.
Falcon-E-1B fine-tuned with different fractions ρ0 of updated ternary latents. GSM8K test accuracy (%) and HumanEval pass@1 (%) after MagiCoder fine-tuning.