acceptodds
Under review as a conference paper at ICLR 2027

Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Cost

Abstract

Post-Training Quantization (PTQ) is essential for deploying Large Language Models (LLMs) on memory-constrained devices, yet it renders models static and difficult to fine-tune. Standard fine-tuning paradigms, including Reinforcement Learning (RL), fundamentally rely on backpropagation and high-precision weights to compute gradients. Thus they cannot be used on quantized models, where the parameter space is discrete and non-differentiable. While Evolution Strategies (ES) offer a backpropagation-free alternative, optimization of the quantized parameters can still fail due to vanishing or inaccurate gradient estimation. This paper introduces Quantized Evolution Strategies (QES), an optimization paradigm that performs full-parameter fine-tuning directly in the quantized space. QES is based on two innovations: (1) it integrates accumulated error feedback to preserve high-precision weight updating signals, and (2) it utilizes a stateless seed replay to reduce memory usage to low-precision inference levels. QES significantly outperforms the state-of-the-art zeroth-order fine-tuning method on a variety of tasks, making direct fine-tuning for quantized models possible. It therefore opens up the possibility for scaling up LLMs entirely in the quantized space.

Primary area:
foundation or frontier models, including LLMs
Keywords:
evolution strategies, large language models, quantization, fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_abPHOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Quantized Evolution Strategies for directly fine-tuning low-bit LLMs without backpropagation or full-precision optimizer states. The main idea is to use accumulated error feedback to avoid losing small updates during quantization, and stateless seed replay to save memory by reconstructing residuals instead of storing them. Experiments show clear improvements over QuZO across different tasks and quantization formats, especially on reasoning tasks.

Critical strengths
  1. The paper addresses a practical problem by enabling direct fine-tuning of low-bit LLMs without full-precision weights or optimizer states, making fine-tuning more accessible in resource-constrained settings.

  2. Combining accumulated error feedback with stateless seed replay is a simple but insightful design that balances optimization accuracy, memory usage, and computation by reconstructing high-precision residuals instead of storing them.

  3. The experiments show substantial improvements over QuZO, while the ablation studies provide useful evidence for the method's effectiveness and practical trade-offs.

Critical weaknesses
  1. While QES shows clear improvements over QuZO, it would be helpful to include a simple baseline that adds standard error feedback to QuZO. This comparison would help clarify which improvements come from error feedback itself and which come from the specific design of QES.

  2. The explanation of why QES improves optimization could be supported by more direct evidence. The paper argues that accumulated error feedback prevents small updates from being lost during quantization. It would be useful to show the fraction of weights actually updated per iteration or measure the gap between intended and actual updates to better support this explanation.

  3. Section 4.4 shows that Seed Replay performs close to Full Residual in most cases, but there is a noticeable gap on Qwen2.5-3B W8A8 Countdown (21.35% vs. 31.70%). It would be helpful to explain what causes this gap and when replay errors become significant.

Questions for authors
  1. How much of QES's improvement comes from error feedback itself? Would adding standard error feedback to QuZO give similar results?

  2. Could the authors clarify how much GPU memory QES actually saves, considering the extra computation introduced by seed replay?

3 comments
AuthorsOpenReview ↗

Response to Reviewer abPH (Part 1/3)

We thank reviewer abPH for the constructive review, and we are encouraged to see that reviewer abPH finds the problem practical, the combination of accumulated error feedback with stateless seed replay simple but insightful, and the improvements over QuZO and the ablations useful.

Here we respond to the reviewer's concerns and questions:

W1 & Q1: How much of QES's improvement comes from error feedback...

Thank you for asking for this ablation study; we ran it, together with the full set of variants around it.

We would like to reemphasize that error feedback is the core of QES. QES trains the quantized model directly, so each weight can only sit on a fixed grid of levels. On 3B INT4 GSM8K, a single ES step typically asks a weight to move by only a few percent of the gap between two levels, while rounding to nearest moves a weight only if the change reaches half a gap; for 99.999% of the weights, one step falls short of that. Without error feedback, almost every step is therefore rounded away (Proposition 2). Error feedback keeps the remainder and moves the weight by one code once the accumulated remainder reaches half a step. Seed replay is how QES keeps that remainder without storing it. Removing error feedback therefore removes most of what QES does, as the following new experiment shows:

SettingQESQES without error feedbackStarting point
3B INT4 GSM8K, generation 20075.157.448.2
SFT, RoBERTa-large INT8 (4 tasks x 3 seeds, average)57.847.346.5 (zero-shot)

On these settings, error feedback accounts for most of QES's gain over its starting point. The per-task SFT results are in the updated Table 1 (our reply to Reviewer YR3d); QES is ahead of the run without error feedback on all four tasks.

We also construct another set of experiments by adding error feedback to QuZO. QuZO differs from QES in two more ways. It rounds its update stochastically (SR) rather than to nearest (RTN). Its estimator also re-rounds the perturbation with an independent random draw, so it updates along a copy that matches the evaluated perturbation δ\delta only on average; QES updates along δ\delta itself. This also explains why QuZO collapses: both choices make it apply updates that are partly wrong. Stochastic rounding keeps each update right on average, but it cannot make the small moves QuZO's update asks for. In our experiments, run at the same step size as QES, it instead moves about 1% of the weights per step by a full level, and only 16% of the change actually applied goes in the direction QuZO's update asked for; the rest is rounding noise. This is the variance explosion of Section 5 (Eq. 9): the noise adds up from step to step and soon outweighs the small update QuZO is trying to make. The re-rounded estimator adds a second error, a component of every step that no rollout evaluated. Once these errors dominate, the weights drift away from the solution, as in Table 2 and Figure 2, where QuZO is unstable and frequently collapses on the INT4 lattice.

We crossed these factors on 3B INT4 GSM8K with the experiment below. "Exact" error feedback stores the full fp32 residual. At QES's step size (σ=5×10−3\sigma = 5\times10^{-3}, α=3×10−4\alpha = 3\times10^{-4}; Table 2's QuZO used σ=10−3\sigma = 10^{-3}, α=10−5\alpha = 10^{-5}):

EstimatorRoundingError feedbackBest evalBehaviour
QESRTNreplay (K=50K = 50)75.1rises and holds
QESRTNexact74.0rises and holds
QESRTNnone62.0flat for 100 generations, then rises slowly
QuZORTNexact68.3rises, then falls to 39.7
QuZOSRexact14.2collapses to 0
QuZOSRnone14.2collapses to 0
QESSRexact6.0collapses to 0

At a 10x smaller step (α=3×10−5\alpha = 3\times10^{-5}), where stochastic rounding moves fewer codes, 60 generations:

EstimatorRoundingError feedbackBest evalBehaviour
QuZOSRnone46.1swings, ends at 37.1
QuZOSRexact64.6brief peak, then collapses to 10.3

Based on the experiment we can see:

  • Standard error feedback does not rescue QuZO. With QuZO's stochastic rounding, error feedback has no deterministic remainder to carry, only rounding noise. At QES's step size QuZO collapses with or without it. At the smaller step error feedback adds a brief peak (64.6) and then collapses (10.3)
  • Stochastic rounding is what breaks, as Section 5 predicts. At QES's step size every stochastic-rounding variant, including QES's own estimator with exact error feedback, falls to 6–14% after the first update and ends at 0%. Round-to-nearest with error feedback learns at the same step without returning and holds its level through generation 200
  • QES's estimator makes the gain last. With RTN and exact error feedback, QuZO's estimator peaks at 68.3 and falls to 39.7 within 100 generations, while QES's estimator holds at 74.0. One reading consistent with this: QuZO's update direction is a re-rounded copy, so part of every step was never evaluated, and error feedback accumulates that part instead of discarding it
AuthorsOpenReview ↗

Response to Reviewer abPH (Part 2/3)

W2: The explanation of why QES improves optimization...

Thank you for this suggestion. We agree that direct evidence is more convincing than the argument alone, so we measured exactly these two quantities on two settings.

The intuition. A quantized weight can only sit on a fixed set of levels, like the rungs of a ladder. Each ES step asks every weight to move by a tiny amount, typically only 1–4% of the distance to the next rung. If each request is simply rounded, the weight is rounded back to the rung it already sits on, and nothing changes. Error feedback keeps a running tally of what was asked for but not yet applied. Once the tally reaches half a rung, the weight moves one rung and the tally is reduced accordingly. So without error feedback, small updates are lost; with it, they add up.

We measured the following metrics during training:

  1. how many weights actually change in that step;
  2. how many weights end up at a different level from where they started;
  3. how well the actual change agrees in direction with the change ES asked for, from 0 (unrelated) to 1 (exactly the requested direction).
GSM8K, QESGSM8K, no error feedbackSFT, QESSFT, no error feedback
Weights changed per step0.0164%0.00057%2.0%0.018%
Weights moved overall2.9%0.11%91%22%
Direction agreement (0 to 1)0.240.030.610.08
Accuracy75.157.457.847.3

GSM8K: Qwen2.5-3B INT4 after 200 generations. SFT: RoBERTa-large INT8, seed 42 at the end of training, averaged over 4 tasks; the accuracy row is the 3-seed mean of the updated Table 1 (included in the response to reviewer YR3d).

What it shows. Without error feedback, 99.9% of what ES asks for is rounded away: after 200 generations on GSM8K, only 0.11% of the weights sit at a different level from where they started. With error feedback, QES changes 29x more weights per step on GSM8K and 110x more on SFT, and the weights move the way ES asked (agreement 0.24 vs 0.03, and 0.61 vs 0.08). Accuracy follows. Rounding at random, as QuZO does, is not a substitute. We measured the same way on the QuZO runs of W1, it moves many weights (about 1% of the weights per step), but the change it applied is 2.5x (GSM8K) and 7.8x (SFT) larger than the update it asked for, and the excess part is rounding noise.

Agreement stays below 1 on GSM8K partly by design: ES's requests are noisy, and QES slowly forgets old requests (γ=0.9\gamma = 0.9) so that the noise does not pile up. Storing the tally exactly instead of replaying it gives the same agreement (0.24), so replay is not the cause. How many weights change per step also depends on the setting: about 1% per step on Countdown in every format (Table 7), and 0.016% on 3B INT4 GSM8K.

We will add this discussion to the paper.

Q2: Could the authors clarify how much GPU memory ...

Thank you. QES saves the whole residual that exact error feedback must keep, one value per weight, and holds only a 29.7 KB seed-and-reward history instead (Table 8). Its saved peak memory is summarized below (Table 9, total VRAM in GB, INT4):

ModelQESFull Residual (FP16)Saved by QES
Qwen2.5-1.5B1.0713.5112.44
Qwen2.5-3B1.9267.0945.17
Llama-3.1-8B5.19017.34012.15

The saving grows with model size, while QES's state does not. The price is update time: on one GPU, replay adds 6.1%, 7.8% and 12.9% to QuZO's iteration time at 1.5B, 3B and 8B (Table 10), using a scratch buffer of 20–70 MB that is freed after each chunk. A shorter window (K=20K = 20) brings the 3B overhead to 4.2%, and sharding the update over 8 GPUs makes it as fast as QuZO's (16.88 vs 16.85 s). Our reply to Reviewer sDyK gives the full timing comparison.

AuthorsOpenReview ↗

Response to Reviewer abPH (Part 3/3)

W3: Section 4.4 shows that Seed Replay performs close to Full Residual ...

Thank you for pointing at this cell. We reran it with different seeds, at the paper's setting:

3B W8A8 CountdownQES (seed replay)Full Residual
Table 621.3531.70
Rerun, seed 1008621.3520.65
Rerun, seed 4223.3524.00

Across these runs Full Residual ranges from 20.65 to 31.70 and QES from 21.35 to 23.35. The two methods overlap, and the gap in Table 6 is one draw from a wide spread rather than a systematic loss from replay. We will add the reruns to Table 6 and report the spread.

The sensitivity in tuning. We found that 3B W8A8 Countdown is a very sensitive setting. On Countdown about 1% of the weights change in every generation, as in Table 7, roughly 60x more than on 3B INT4 GSM8K. W8A8 also rounds the activations in every forward pass, so a small difference in the weights can change the generated answers. Any small difference between two runs therefore grows quickly. Replay and an exactly stored residual, started from the same seed with identical weights, already differ in 26% of the weights by generation 25, as much as two QES runs with different seeds (28%), while their accuracies stay within a few points (15.2 vs 14.4 at generation 100). Full Residual's random initial residual alone makes 2.4% of its weights differ from QES's after the first update, and even the hardware and software setup changes the rollouts at generation 0 (population reward 0.1616 vs 0.1629 at the same seed). This is also what Section 4.4 found: on the same W8A8 configuration on GSM8K, the two methods are within 0.76 points (80.82 vs 81.58), so the Countdown gap is specific to this cell rather than a failure of replay.

When replay errors become significant. Replay rebuilds the residual exactly unless an update hits the quantization boundary, and it leaves out what is older than the window KK. Its error therefore matters only with a short window or frequent boundary hits. At the paper's K=50K = 50 neither applies: boundary hits are 4.2×10−64.2\times10^{-6} of attempted changes on this cell, and on 3B INT4 GSM8K replay and the exact residual agree on every weight for the first two updates, differ in one weight out of 2.77 billion at the third, and reach the same accuracy (73.8 vs 73.5 at generation 149–150). With a short window the loss becomes visible: in Table 7, K=10K = 10 gives 13.05 against 16.00 at K=50K = 50.

We will add these discussions to the revision.

Reviewer_sDyKOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed QES, a memory-efficient fine-tuning method tailored to quantized LLMs. Specifically, QES leverages gradient-free evolution strategies to directly optimize discrete model weights, while using accumulated error feedback to correct quantization errors dynamically and stateless seed replay to improve memory efficiency. Extensive experimental results demonstrate that QES excels in both classification and complex reasoning tasks while keeping the memory footprint at the low-precision inference level.

Critical strengths
  1. QES performs well on both classification and reasoning tasks while maintaining memory efficiency.
  2. The paper is technically sound with sufficient analysis of the proposed method.
Critical weaknesses
  1. Training time is significantly increased. A common weakness of gradient-free methods is longer training time due to slower convergence [1]. The proposed method further increases the per-iteration wall-clock time to 13−17×13-17\times. Although a sharding strategy with 8 GPUs mitigates this issue, it is not a promising solution under a memory-efficient setting.
  2. Missing comparison with zeroth-order methods using n-SPSA. QES uses stateless seed replay to materialize the error vector on-the-fly within a limited window during training. This resembles the nn-SPSA setting in zeroth-order optimization methods [1], which also saves the random seeds and scalar-form directional derivative estimates for the last nn updates. However, the comparison experiments consider only a single perturbation case and therefore may be unfair.
  3. Missing discussions on compatibility with codebook-based quantization methods. The proposed method considers only scalar-based quantization methods, such as GPTQ and LLM-compressor. Nevertheless, the formulation of QES should also be compatible with codebook-based methods like AQLM [2] or NF4 [3].

References

[1] Fine-Tuning Language Models with Just Forward Passes (NeurIPS 2023)

[2] Extreme Compression of Large Language Models via Additive Quantization (ICML 2024)

[3] QLoRA: Efficient Fine-tuning of Quantized LLMs (NeurIPS 2023)

Questions for authors

Based on the weaknesses, I hope the authors can respond to the following questions to address my major concerns.

  1. Is there an alternative approach to mitigate the training-time issue without increasing the compute budget while maintaining the memory-efficient setting?
  2. Could you provide comparisons with nn-SPSA variants of zeroth-order methods under a fair setting?
  3. Could you briefly discuss whether QES is suitable for codebook-based quantization methods?
5 comments
AuthorsOpenReview ↗

Response to Reviewer sDyK (Part 1/3)

We thank reviewer sDyK for the careful review, and we are encouraged to see that reviewer sDyK finds that QES performs well on both classification and reasoning tasks while staying memory-efficient, and that the method is technically sound with sufficient analysis.

Here we respond to the reviewer's concerns and questions:

W1 & Q1: Training time is ...

Thank you for raising the concern about the computational cost. We want to clarify that the 13–17x figure you mentioned is for the weight update step alone using one gpu, which only constitutes about 6-12% of the total wall-clock time of each iteration. In cases where more gpus are used, this update time becomes even more negligible (see Table 10 in the appendix). A training iteration is dominated by the rollouts, i.e. generating the N=50N = 50 perturbed members' answers, which QES and QuZO run identically. To make the comparison clearer, we calculated the total wall-clock time for each iteration using one gpu:

Per iteration (Table 10, one H100, INT4, N=50N = 50, K=50K = 50, seconds):

ModelRolloutsQuZO updateQES updateQES iteration vs QuZO iteration
Qwen2.5-1.5B17847.02115.6+6.1%
Qwen2.5-3B317416.85264.7+7.8%
Llama-3.1-8B401443.29565.9+12.9%

The added time buys memory: QES holds 29.7 KB of state, whereas storing the residual, the alternative that makes the update cheap, takes 5.17 GB more at 3B in FP16 (Full Residual, Table 9) and twice that, 10.3 GB, in FP32.

Below are ways to further speed up the QES update step without more compute or persistent memory.

A shorter window. With γ=0.9\gamma = 0.9, a residual older than KK generations is scaled by γK\gamma^{K}. At 3B, K=20K = 20 cuts the single-GPU update from 264.7 s to 150.9 s (Appendix E.3), i.e. +4.2% per iteration instead of +7.8%, for 1.25 accuracy points on 1.5B INT4 Countdown (Section 4.4). We also reran 3B INT4 GSM8K with K=20K = 20, where it costs nothing as shown in the table below.

Replay window KK (γ=0.9\gamma = 0.9)1.5B INT4 Countdown (Tables 6, 7)3B INT4 GSM8K
2014.7575.7
5016.0073.8
Exactly stored residual18.0573.5

Batched replay. The sequential update replays the KN perturbations one at a time; batched replay regenerates all of them for a group of layers in a single pass. It performs the same computation and gives identical weights, so it adds no compute and no hardware; it is faster only by leveraging the batched compute advantage of GPU. On one H100 with 3B INT4 and K=50, the update can be further accelerated to 104.2 s vs 264.7 s, which is 2.5x faster. The only cost is temporary memory during the update, which is freed right after it and, in our runs, fits in the GPU memory the inference engine leaves outside its own budget. How many perturbations are batched at once can be tuned to trade memory for speed. When rollouts are already spread over several GPUs, the update can also be split across them: on 8 GPUs, QES's update takes 16.88 s against QuZO's 16.85 s at 3B (Table 10, right).

One other thing we would like to mention is the time-to-accuracy. QES gets more out of each iteration, especially in reasoning tasks. Without error feedback, each small update is either rounded away (round-to-nearest) or buried in rounding noise (QuZO's stochastic rounding). On 3B INT4 GSM8K, QES reaches 65.0% by generation 25. QES without error feedback is still at 57.4% at generation 200, and no plain QuZO run on this cell ever scores above 48.6% (Table 2). So even at +6–13% wall-clock time per iteration (using one gpu), QES passes the best accuracy of either baseline within 25 generations, while plain QuZO never improves on where it starts.

Last, one computational benefit of gradient-free methods is that the calculations of backpropagation are no longer needed. A gradient-based method usually needs 4X more FLOPs to process one rollout compared to gradient-free methods [1]. According to the experiments in [1] and [2], ES can outperform gradient-based RL in many LLM fine-tuning tasks given the same number of rollouts.

[1] Yulu Gan and Phillip Isola, “Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights”, ICML 2026.

[2] Xin Qiu et al., “Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning”, ICML 2026.

AuthorsOpenReview ↗

Response to Reviewer sDyK (Part 2/3)

W2 & Q2: Missing comparison with zeroth-order ...

Thank you for pointing us to n-SPSA. In MeZO's Definition 1, n-SPSA averages the SPSA estimate over nn random perturbations within one step, at 2n2n forwards per step. Our comparisons already used it, i.e., QuZO shares the exact same mechanism with n-SPSA, with the same nn for every method: each reasoning run, QES and QuZO alike, evaluates N=50N = 50 members, i.e. n=25n = 25 antithetic directions per step. On SFT, QES, QuZO and the MeZO n-SPSA row of the updated Table 1 (our reply to Reviewer YR3d) all use n=4n = 4 at the same 12,000 forwards.

To further verify the effect of nn, we conducted a round of ablation study: RoBERTa-large, 4 tasks (SNLI, MNLI, RTE, SST-5) each with 3 seeds, 12,000 training forwards per run. Every method gets its own step-size grid at every nn (at least 3 points, extended when the winner sits at an edge) and is selected on validation set only. Mean test accuracy:

nnQES (8-bit)QuZO (n-SPSA, 8-bit)MeZO n-SPSA (FP32)
1––58.3
255.349.459.6
457.850.558.6
854.051.059.3
1658.054.258.6
Weight memory1 Byte/param1 Byte/param4 Bytes/param

At equal bit-width, QES beats n-SPSA (QuZO) at every nn, by 3.0 to 7.3 points. More directions per step help QuZO a little, but they do not close the gap. This is because its error comes from rounding, not from estimator variance. With 8-bit weights, QES comes within 0.6 points of full-precision n-SPSA (MeZO using FP32) at n=16n = 16, with a quarter of its weight memory.

We also investigated the method storing the last updates' seeds and scalars. This is MeZO's App. B.2, which rebuilds gradient history (momentum) from stored seeds and projected gradients instead of a dense buffer; we test it with a window, so its cost matches QES. We refer to it as B.2. B.2 and QES replay the same kind of record, but they rebuild different things. B.2 rebuilds a smoothed gradient, which still passes through the quantizer every step, so whatever rounding cuts off is lost (round-to-nearest) or replaced by noise (stochastic rounding). QES instead rebuilds the part of its recent updates that rounding has cut off, and adds it back before the next rounding, so small updates add up until they move a weight instead of being lost. We tested B.2 on QuZO at matched cost: the same window as QES's KK, the same seed-and-scalar state (~1 KB on SFT, 29.7 KB on reasoning) and the same replay work, with β∈\beta \in {0.5, 0.9} and the step size tuned.

SFT (RoBERTa-large, n=4n = 4, 12,000 forwards, window 16 = QES's KK, mean test over 3 seeds):

MethodStateSNLIMNLIRTESST-5Avg
QuZO–54.951.059.037.150.5
QuZO + B.2 momentum~1 KB58.652.658.742.853.2
QES~1 KB64.357.164.445.657.8

Reasoning (Qwen2.5-3B INT4, GSM8K, window 50 = QES's KK):

MethodTrain accBest evalFinal eval
QuZO (α\alpha = 3e-5)28.946.137.1
+ B.2, β\beta = 0.5, α\alpha = 1.5e-50.244.00.0
+ B.2, β\beta = 0.5, α\alpha = 5e-50.046.20.0
+ B.2, β\beta = 0.9, α\alpha = 3e-6 (selected)17.247.80.8
+ B.2, β\beta = 0.9, α\alpha = 1e-50.046.90.0
QES71.671.471.4

Based on the results, QES consistently outperforms QuZO and QuZO+B.2 in all the testing cases.

Momentum smooths the estimate but leaves the rounding error uncorrected; error feedback is what lets a low-bit model follow the ES trajectory.

We will add all the new experiments and discussions to the revision.

AuthorsOpenReview ↗

Response to Reviewer sDyK (Part 3/3)

W3 & Q3: Missing discussions on compatibility with codebook-based ...

Thank you for the suggestion. QES needs two things from a format: a fixed decode from code to weight, and codes that are ordered, so that changing a code by +/-1 is the smallest possible move on the grid. Error feedback then carries whatever a single step cannot apply, in code units. Uniform integer grids satisfy both, and so do scalar codebooks such as NF4. NF4's 16 levels are sorted quantiles of a normal distribution, scaled by a per-block absmax, so QES runs on the 4-bit index without any change to the algorithm. Only the decode becomes a table lookup. We have added a new experiment of QES on NF4 format:

QES on NF4. RoBERTa-large with every weight matrix in QLoRA's NF4 (block size 64), same tasks, seeds, 12,000-forward budget and validation-set-only selection as the updated Table 1 (our reply to Reviewer YR3d). QES and QuZO each searched 18 settings in the same two passes: a first pass, then a refinement at smaller σ\sigma after the first pass either moved no codes or moved whole levels destructively. Mean test accuracy over 3 seeds; the share of weights updated and the direction agreement between the applied and the intended update (cosine similarity, 1 = same direction, 0 = unrelated) are from seed 42:

NF4SNLIMNLIRTESST-5AvgWeights movedCos Sim
Zero-shot47.045.852.337.845.7––
QuZO39.139.050.834.740.947.1%0.21
QES without error feedback (γ=0\gamma = 0)49.648.051.741.847.82.4%0.08
QES53.450.857.042.550.924.4%0.36

The INT8 mechanism carries over. Without error feedback almost nothing moves (2.4% of the codes). QES moves a quarter of the codes along the ES direction and gains 5.2 points over zero-shot. The gain is smaller than at INT8 (57.8 from 46.5), as expected: one NF4 level is about an order of magnitude wider than an INT8 step.

We also investigated why the QuZO baseline collapses on this format. In each step, QuZO's update asks each weight to move by far less than one NF4 level (about 0.5% of a level on average). Stochastic rounding cannot make such a small move: it leaves almost every weight where it is and moves a random few (about 0.05% of the weights per step) by a full level. This is right on average, but the change actually applied agrees only weakly with the update QuZO wanted (cosine similarity 0.21, where 1 means the same direction and 0 means unrelated). This noise adds up over training (Section 5, Eq. 9), and because an NF4 level is about ten times wider than an INT8 step, each random move does more damage. In contrast, QES instead rounds to the nearest level and keeps the leftover; a weight moves only after its leftovers add up to half a level, and then it moves in the direction the updates asked for.

Vector codebooks (AQLM). Here a code indexes a vector built from learned codebooks, and indices carry no order, so a +/-1 change is not a small move and rounding with a residual is not defined on the codes. However, we have 2 routes remaining. One is to run ES on AQLM's continuous parameters, its codebooks and scales, which are small and need no rounding. The other is to keep the error-feedback residual in weight space and periodically re-encode, which is costly with beam-search encoding. We will add this discussion and the NF4 results to the paper.

Reviewer_YR3dOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses fine-tuning post-training-quantized LLMs directly in the discrete integer space, where ZO/ES updates either stagnate (deterministic rounding of sub-lattice steps) or diffuse (stochastic rounding, Θ(√T) deviation). QES applies standard ES on the integer codes with two additions: accumulated error feedback (a residual carries the fractional part of each intended update — the 1-bit-SGD trick transplanted to a lattice) and stateless seed replay (the residual is rematerialised by replaying the last K generations, cutting optimizer state from O(d) to O(K)). Prop. 2 characterizes a dead zone below which a coordinate provably never moves. Experiments span RoBERTa-large SFT and Qwen2.5-1.5B/3B RLVR reasoning (INT4/INT8/W8A8), with large gains over QuZO (e.g. 3B INT4 GSM8K 48.45 → 77.56) and two transferred-hyperparameter case studies at 8B/32B.

Critical strengths
  1. The mechanism is simple and the paper explains why it should work. The two-failure-mode diagnosis (stall vs diffuse) is correct and well argued; the connections to Delta-Sigma noise shaping and error feedback in gradient compression are the right ones, and the borrowings are candidly credited.
  2. Proposition 2 is a genuinely useful characterization, not decoration. The dead-zone bound explains why a ZO method on INT4 can look like "slow convergence" while actually at a fixed point, and it predicts that γ = 0.9 lowers the resolvable signal by an order of magnitude — consistent with the empirical collapse at γ = 0.58.
  3. Seed replay answers the right engineering problem with measured costs. The state inventory acknowledges the 20–70 MB replay scratch honestly, and the paper reports the 13–17× slower update before arguing it amortizes (end-to-end +6–13%) because rollouts dominate.
Critical weaknesses
  1. The SFT baseline grid (Table 1) runs at a budget ~100× below the protocol it claims to follow, and two MeZO cells are at chance. MeZO reports SNLI 34.0 / MNLI 34.0 (three-way chance) and SST-5 21.7 vs a published 40.8; the FP32 "upper bound" scores 50.0 on SNLI vs published ~77.5 — so QES "beats the upper bound," which is false labeling. The headline claim that error feedback recovers more signal than smoothed ZO estimation is measured against baselines that failed to train.
  2. The QuZO baseline on the reasoning tasks is not established as fairly tuned. Its collapse to 4.40% on 3B W8A8 GSM8K — below the 24.49% untuned base — is treated as QuZO brittleness; the other reading is a mis-tuned step size. Per-cell σ/α are documented for QES only, with no grid-search description for QuZO, and the central claim is the size of the QuZO–QES gap.
  3. No ablation isolates error feedback from the rest of the configuration. The natural test — QES at γ = 0 (pure round-to-nearest, the stagnation regime of §3.2) at fixed N, α, σ, and generations — is absent. Without it, gains cannot be attributed to the residual mechanism rather than to deterministic rounding, N = 50 antithetic pairs, or per-cell tuned α.
Questions for authors
  1. Q1 the SFT baselines run ~100× below their published protocol, with two cells at chance). Table 1's MeZO reports SNLI 34.0 and MNLI 34.0 (three-way chance) and SST-5 21.7 vs a published 40.8; the FP32 "upper bound" row scores 50.0 on SNLI vs ~77.5 published full fine-tuning on the same backbone. Please give the budget and hyperparameters actually used for each baseline, explain the discrepancy with the MeZO protocol A.2 says you follow, and re-run Table 1 with each baseline at its published budget and tuning. If the 1500-step budget is intended as part of the comparison, say so explicitly and drop the "upper bound" label — QES "beating the upper bound" is currently a false statement.
  2. Q2 the QuZO baseline is not established as fairly tuned). How were QuZO's σ and learning rate chosen for each of the six (model, format) cells in Table 2? Was there a grid search comparable to the SFT setting? Please report the searched values and add a cell showing QuZO at its best-found setting on 3B W8A8 GSM8K, where it currently scores 4.40% against a 24.49% untuned base. The central quantitative claim of the paper is the size of the QuZO–QES gap.
  3. Q3 no ablation isolates error feedback. Please add QES with γ = 0 (pure deterministic round-to-nearest, i.e. exactly the stagnation regime of §3.2) at the same N, α, σ, and generation count, on at least one reasoning cell (3B INT4 GSM8K and Countdown). Without this cell, the reported gains cannot be attributed to the residual mechanism rather than to deterministic rounding, N = 50 antithetic pairs, or per-cell tuned α.
4 comments
AuthorsOpenReview ↗

Response to Reviewer YR3d (Part 1/3)

We thank reviewer YR3d for the careful review, and we are encouraged to see that reviewer YR3d finds it to be a simple and well explained method.

Here we respond to the reviewer's questions:

W1 & Q1 the SFT baselines run ~100x below…

Thank you for checking Table 1 against the published numbers. We re-audited our implementations and found the cause being the discrepancies in library versions and model-loading. As suggested, we have changed our implementations to use the same open-source codes and environments of the relevant papers, and reran the experiments for all the algorithms. The zero-shot accuracy now matches the published values within 0.4 points. The reasoning experiments use a separate vLLM-based pipeline and are not affected: Table 2's 3B W8A8 Countdown base (8.20%) reproduces exactly with our current code. The updated Table 1 results are shown below:

Updated Table 1. RoBERTa-large, k=16, LM-BFF prompts, 1,000-example test sets (RTE: 277), mean over 3 data seeds, one configuration per method shared by all four tasks. The last three rows are references: MeZO's own protocol (100k steps, learning rate chosen per task), as we ran it and as published, and the published first-order baseline.

MethodState / paramSNLIMNLIRTESST-5Avg
Zero-shot, FP32 / W8–50.4 / 50.549.2 / 49.251.3 / 51.635.5 / 34.646.6 / 46.5
First-orderFP32 (Adam), 1k steps16 Byte76.267.165.146.063.6
QAT W8 (STE, FP32 master, Adam)16 Byte78.468.467.947.065.4
MeZO FP32, 12k forwards4 Byte65.659.861.446.358.3
n-SPSA (n=4) FP32, 12k forwards4 Byte68.460.462.543.358.6
QuZO W8, 12k forwards1 Byte54.951.059.037.150.5
QES W8, 12k forwards1 Byte64.357.164.445.657.8
QES without error feedback (γ\gamma=0)1 Byte51.850.051.635.647.3
MeZO FP32, 100k steps, our run (seed 42)4 Byte69.659.264.345.359.6
MeZO FP32, 100k steps, reported by Malladi et al.4 Byte68.558.764.045.559.2
First-order FP32, reported by Malladi et al.16 Byte77.570.066.447.565.3

On 8-bit weights, QES beats QuZO on every task, by 7.3 points on average, and comes within 0.5 points of full-precision MeZO (both using 12k forwards) with a quarter of its weight memory. Our MeZO run also reproduces the published MeZO numbers (59.6 vs 59.2). We will update Table 1, its caption and Section 4.2 accordingly.

Budgets and hyperparameters. FT and QAT follow MeZO's FT protocol: 1,000 Adam steps over MeZO's grid, lr {1e-5, 3e-5, 5e-5} x batch {2, 4, 8}. Apart from the 100k-step MeZO references, every zeroth-order row gets the same 12,000 training forwards: MeZO 6,000 steps x 2 forwards over lr {1e-7, 1e-6, 1e-5} and n-SPSA 1,500 x 8 over lr {1e-7, 1e-6, 1e-5, 2e-5, 4e-5}; QuZO 1,500 x 8 over α\alpha, σ\sigma, schedule and full-batch SST-5 (15 configurations); QES the same 12,000 forwards over these plus nn (1,500 x 8 or 750 x 16), KK and the set of trained tensors (34 configurations). Each method's configuration is chosen once on seed 42 and kept fixed for seeds 13 and 21. This protocol replaces Appendix A.2 and Table 3 in the revision.

AuthorsOpenReview ↗

Response to Reviewer YR3d (Part 2/3)

W2 & Q2 the QuZO baseline is not established as…

Thanks for your comment. We want to clarify that we did run a hyperparameter sweep for QuZO to obtain its best performance, as detailed below:

How QuZO's σ\sigma and α\alpha were chosen. Before Table 2 we screened QuZO's step size on Countdown: α\alpha ∈ {1e-5, 1e-4, 3e-4} with σ\sigma ∈ {1e-3, 5e-3}, on 1.5B INT4, 1.5B INT8, 3B INT4 and 3B INT8 (6 runs). Every run with α\alpha ≥ 1e-4 collapsed to 0%; α\alpha = 1e-5 was the only value that did not, and it was used with σ\sigma = 1e-3 for every INT4 and INT8 cell on both tasks. The two W8A8 cells used σ\sigma = 0.01, α\alpha = 1e-3. For this response we searched QuZO on 3B W8A8 GSM8K over σ\sigma ∈ {0.001, 0.01, 0.1} x α\alpha ∈ {1e-3, 1e-4, 1e-5}, selecting by training accuracy, and σ\sigma = 0.01, α\alpha = 1e-3 is the best of the nine. For comparison, QES's σ\sigma and α\alpha (Table 4) were set once per (model, format) cell without a comparable search, and reused for GSM8K.

The best-found setting is the one the paper already used. QuZO thus received more tuning than QES, and the search did not find a better setting for QuZO.

Why 4.40%. We double checked the training logs of QuZO in this setup, and found that the training accuracy plateaued in the early stage and collapsed to 0.3% by generation 100 (best evaluation 4.40%) due to increasingly wrong answer formats. QuZO's training are also unstable on this cell: training accuracy 10 generations apart differ by 7 to 19 points on average in the early stage. We have tried a hyperparameter grid search for QuZO in this setup (as detailed above), but did not find a better-performing configuration. Therefore, we attribute this performance collapse to the unstable training behaviors of QuZO in learning and preserving correct answer format.

Every Table 2 entry we could trace, QuZO and QES alike, is the best evaluation of its run.

AuthorsOpenReview ↗

Response to Reviewer YR3d (Part 3/3)

W3 & Q3 no ablation isolates error feedback…

Thank you for specifying the control; we have added the ablation studies exactly as suggested. Error feedback is the core of QES, and removing it (γ\gamma = 0, pure round-to-nearest, the stagnation regime of Section 3.2) removes most of QES's gain. On 3B INT4 GSM8K both runs use N = 50 members, σ\sigma = 5e-3, α\alpha = 3e-4 and the same seed; the only difference is γ\gamma: 0.9 with error feedback, or 0 without it. We ran the run without error feedback for 500 generations, 2.5x longer than QES, so that it is not cut short.

SettingQES (γ\gamma = 0.9)No error feedback (γ\gamma = 0)Starting point
3B INT4 GSM8K, generation 20075.157.448.2
SFT, RoBERTa-large INT8 (4 tasks x 3 seeds; Q1)57.847.346.5 (zero-shot)

On GSM8K, the run without error feedback stays near the starting level for 100 generations and then rises slowly. Its best evaluation over all 500 generations (62.0) is below what QES reaches by generation 25 (65.0). On SFT, it stays essentially at zero-shot.

We measured the Proposition 2 during the training. A quantized weight can only sit on a fixed grid of levels. Rounding to nearest moves it to the next level only if its update is at least half the gap between two levels; anything smaller is rounded back to where the weight already is. We measured how large the ES updates are on 3B INT4 GSM8K: in a typical step each weight is asked to move by only about 4% of the gap (0.038 gaps, averaged over all weights as a root mean square), far below the half gap needed. As a result:

  • Without error feedback, more than 99.9% of every step's update is rounded away: only 0.00057% of the weights change in a step.
  • With error feedback, the rounded-away part is kept and added to the following steps, so a weight moves once its small updates add up to half a gap. QES changes 0.0164% of the weights per step, 29x more.
  • These moves also follow the update ES asks for. Their agreement in direction with the intended update (cosine similarity: 1 means the same direction, 0 unrelated) is 0.24 for QES against 0.03 without error feedback, at generation 200.
  • Over the whole run, QES had moved more weights away from their starting values by generation 34 (0.41%) than the run without error feedback did in 500 generations (0.38%).

This matches the decay ablation in the paper (Table 7). A smaller γ\gamma keeps less of the rounded-away part and widens the dead zone (Eq. 13): on 1.5B INT4 Countdown, lowering γ\gamma from 0.9 to 0.58 drops accuracy from 16.00 to 4.55, and Table 7 shows that the drop comes from γ\gamma rather than the shorter replay window. γ\gamma = 0 keeps nothing, which is the extreme case of that trend.

We also investigated the alternative explanations you list. Deterministic rounding and the population (N = 50 members) are the same in both runs, so they cannot explain the gap. Nor can the step size. A larger step without error feedback does get past the dead zone, but then every step moves weights by a whole level along one noisy estimate, and the model breaks: at 3x the step size, accuracy is 52.2 after one update and 0.0 at every evaluation from generation 20 on; at 10x it is 0.0 from the first update. A smaller step only moves fewer weights. Error feedback is what lets QES take small, safe steps and still apply them.

Our reply to Reviewer abPH adds stochastic rounding with and without error feedback, and QuZO's estimator combined with error feedback. Only the variants with QES's estimator and round-to-nearest error feedback, replayed or exactly stored, rise and hold their accuracy. We will add all the new experiments and discussions to the revision.

Reviewer_YR3dOpenReview ↗

Response to author

Thank you for the detailed rebuttal and for the additional experiments and clarifications. I appreciate the authors' transparency about the experimental configuration and the effort invested in the revision. I maintain weak accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.