- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents Quantized Evolution Strategies for directly fine-tuning low-bit LLMs without backpropagation or full-precision optimizer states. The main idea is to use accumulated error feedback to avoid losing small updates during quantization, and stateless seed replay to save memory by reconstructing residuals instead of storing them. Experiments show clear improvements over QuZO across different tasks and quantization formats, especially on reasoning tasks.
Critical strengths
-
The paper addresses a practical problem by enabling direct fine-tuning of low-bit LLMs without full-precision weights or optimizer states, making fine-tuning more accessible in resource-constrained settings.
-
Combining accumulated error feedback with stateless seed replay is a simple but insightful design that balances optimization accuracy, memory usage, and computation by reconstructing high-precision residuals instead of storing them.
-
The experiments show substantial improvements over QuZO, while the ablation studies provide useful evidence for the method's effectiveness and practical trade-offs.
Critical weaknesses
-
While QES shows clear improvements over QuZO, it would be helpful to include a simple baseline that adds standard error feedback to QuZO. This comparison would help clarify which improvements come from error feedback itself and which come from the specific design of QES.
-
The explanation of why QES improves optimization could be supported by more direct evidence. The paper argues that accumulated error feedback prevents small updates from being lost during quantization. It would be useful to show the fraction of weights actually updated per iteration or measure the gap between intended and actual updates to better support this explanation.
-
Section 4.4 shows that Seed Replay performs close to Full Residual in most cases, but there is a noticeable gap on Qwen2.5-3B W8A8 Countdown (21.35% vs. 31.70%). It would be helpful to explain what causes this gap and when replay errors become significant.
Questions for authors
-
How much of QES's improvement comes from error feedback itself? Would adding standard error feedback to QuZO give similar results?
-
Could the authors clarify how much GPU memory QES actually saves, considering the extra computation introduced by seed replay?
3 comments
Response to Reviewer abPH (Part 2/3)
W2: The explanation of why QES improves optimization...
Thank you for this suggestion. We agree that direct evidence is more convincing than the argument alone, so we measured exactly these two quantities on two settings.
The intuition. A quantized weight can only sit on a fixed set of levels, like the rungs of a ladder. Each ES step asks every weight to move by a tiny amount, typically only 1–4% of the distance to the next rung. If each request is simply rounded, the weight is rounded back to the rung it already sits on, and nothing changes. Error feedback keeps a running tally of what was asked for but not yet applied. Once the tally reaches half a rung, the weight moves one rung and the tally is reduced accordingly. So without error feedback, small updates are lost; with it, they add up.
We measured the following metrics during training:
- how many weights actually change in that step;
- how many weights end up at a different level from where they started;
- how well the actual change agrees in direction with the change ES asked for, from 0 (unrelated) to 1 (exactly the requested direction).
| GSM8K, QES | GSM8K, no error feedback | SFT, QES | SFT, no error feedback | |
|---|---|---|---|---|
| Weights changed per step | 0.0164% | 0.00057% | 2.0% | 0.018% |
| Weights moved overall | 2.9% | 0.11% | 91% | 22% |
| Direction agreement (0 to 1) | 0.24 | 0.03 | 0.61 | 0.08 |
| Accuracy | 75.1 | 57.4 | 57.8 | 47.3 |
GSM8K: Qwen2.5-3B INT4 after 200 generations. SFT: RoBERTa-large INT8, seed 42 at the end of training, averaged over 4 tasks; the accuracy row is the 3-seed mean of the updated Table 1 (included in the response to reviewer YR3d).
What it shows. Without error feedback, 99.9% of what ES asks for is rounded away: after 200 generations on GSM8K, only 0.11% of the weights sit at a different level from where they started. With error feedback, QES changes 29x more weights per step on GSM8K and 110x more on SFT, and the weights move the way ES asked (agreement 0.24 vs 0.03, and 0.61 vs 0.08). Accuracy follows. Rounding at random, as QuZO does, is not a substitute. We measured the same way on the QuZO runs of W1, it moves many weights (about 1% of the weights per step), but the change it applied is 2.5x (GSM8K) and 7.8x (SFT) larger than the update it asked for, and the excess part is rounding noise.
Agreement stays below 1 on GSM8K partly by design: ES's requests are noisy, and QES slowly forgets old requests () so that the noise does not pile up. Storing the tally exactly instead of replaying it gives the same agreement (0.24), so replay is not the cause. How many weights change per step also depends on the setting: about 1% per step on Countdown in every format (Table 7), and 0.016% on 3B INT4 GSM8K.
We will add this discussion to the paper.
Q2: Could the authors clarify how much GPU memory ...
Thank you. QES saves the whole residual that exact error feedback must keep, one value per weight, and holds only a 29.7 KB seed-and-reward history instead (Table 8). Its saved peak memory is summarized below (Table 9, total VRAM in GB, INT4):
| Model | QES | Full Residual (FP16) | Saved by QES |
|---|---|---|---|
| Qwen2.5-1.5B | 1.071 | 3.511 | 2.44 |
| Qwen2.5-3B | 1.926 | 7.094 | 5.17 |
| Llama-3.1-8B | 5.190 | 17.340 | 12.15 |
The saving grows with model size, while QES's state does not. The price is update time: on one GPU, replay adds 6.1%, 7.8% and 12.9% to QuZO's iteration time at 1.5B, 3B and 8B (Table 10), using a scratch buffer of 20–70 MB that is freed after each chunk. A shorter window () brings the 3B overhead to 4.2%, and sharding the update over 8 GPUs makes it as fast as QuZO's (16.88 vs 16.85 s). Our reply to Reviewer sDyK gives the full timing comparison.
Response to Reviewer abPH (Part 3/3)
W3: Section 4.4 shows that Seed Replay performs close to Full Residual ...
Thank you for pointing at this cell. We reran it with different seeds, at the paper's setting:
| 3B W8A8 Countdown | QES (seed replay) | Full Residual |
|---|---|---|
| Table 6 | 21.35 | 31.70 |
| Rerun, seed 10086 | 21.35 | 20.65 |
| Rerun, seed 42 | 23.35 | 24.00 |
Across these runs Full Residual ranges from 20.65 to 31.70 and QES from 21.35 to 23.35. The two methods overlap, and the gap in Table 6 is one draw from a wide spread rather than a systematic loss from replay. We will add the reruns to Table 6 and report the spread.
The sensitivity in tuning. We found that 3B W8A8 Countdown is a very sensitive setting. On Countdown about 1% of the weights change in every generation, as in Table 7, roughly 60x more than on 3B INT4 GSM8K. W8A8 also rounds the activations in every forward pass, so a small difference in the weights can change the generated answers. Any small difference between two runs therefore grows quickly. Replay and an exactly stored residual, started from the same seed with identical weights, already differ in 26% of the weights by generation 25, as much as two QES runs with different seeds (28%), while their accuracies stay within a few points (15.2 vs 14.4 at generation 100). Full Residual's random initial residual alone makes 2.4% of its weights differ from QES's after the first update, and even the hardware and software setup changes the rollouts at generation 0 (population reward 0.1616 vs 0.1629 at the same seed). This is also what Section 4.4 found: on the same W8A8 configuration on GSM8K, the two methods are within 0.76 points (80.82 vs 81.58), so the Countdown gap is specific to this cell rather than a failure of replay.
When replay errors become significant. Replay rebuilds the residual exactly unless an update hits the quantization boundary, and it leaves out what is older than the window . Its error therefore matters only with a short window or frequent boundary hits. At the paper's neither applies: boundary hits are of attempted changes on this cell, and on 3B INT4 GSM8K replay and the exact residual agree on every weight for the first two updates, differ in one weight out of 2.77 billion at the third, and reach the same accuracy (73.8 vs 73.5 at generation 149–150). With a short window the loss becomes visible: in Table 7, gives 13.05 against 16.00 at .
We will add these discussions to the revision.
Response to Reviewer abPH (Part 1/3)
We thank reviewer abPH for the constructive review, and we are encouraged to see that reviewer abPH finds the problem practical, the combination of accumulated error feedback with stateless seed replay simple but insightful, and the improvements over QuZO and the ablations useful.
Here we respond to the reviewer's concerns and questions:
Thank you for asking for this ablation study; we ran it, together with the full set of variants around it.
We would like to reemphasize that error feedback is the core of QES. QES trains the quantized model directly, so each weight can only sit on a fixed grid of levels. On 3B INT4 GSM8K, a single ES step typically asks a weight to move by only a few percent of the gap between two levels, while rounding to nearest moves a weight only if the change reaches half a gap; for 99.999% of the weights, one step falls short of that. Without error feedback, almost every step is therefore rounded away (Proposition 2). Error feedback keeps the remainder and moves the weight by one code once the accumulated remainder reaches half a step. Seed replay is how QES keeps that remainder without storing it. Removing error feedback therefore removes most of what QES does, as the following new experiment shows:
On these settings, error feedback accounts for most of QES's gain over its starting point. The per-task SFT results are in the updated Table 1 (our reply to Reviewer YR3d); QES is ahead of the run without error feedback on all four tasks.
We also construct another set of experiments by adding error feedback to QuZO. QuZO differs from QES in two more ways. It rounds its update stochastically (SR) rather than to nearest (RTN). Its estimator also re-rounds the perturbation with an independent random draw, so it updates along a copy that matches the evaluated perturbation δ only on average; QES updates along δ itself. This also explains why QuZO collapses: both choices make it apply updates that are partly wrong. Stochastic rounding keeps each update right on average, but it cannot make the small moves QuZO's update asks for. In our experiments, run at the same step size as QES, it instead moves about 1% of the weights per step by a full level, and only 16% of the change actually applied goes in the direction QuZO's update asked for; the rest is rounding noise. This is the variance explosion of Section 5 (Eq. 9): the noise adds up from step to step and soon outweighs the small update QuZO is trying to make. The re-rounded estimator adds a second error, a component of every step that no rollout evaluated. Once these errors dominate, the weights drift away from the solution, as in Table 2 and Figure 2, where QuZO is unstable and frequently collapses on the INT4 lattice.
We crossed these factors on 3B INT4 GSM8K with the experiment below. "Exact" error feedback stores the full fp32 residual. At QES's step size (σ=5×10−3, α=3×10−4; Table 2's QuZO used σ=10−3, α=10−5):
At a 10x smaller step (α=3×10−5), where stochastic rounding moves fewer codes, 60 generations:
Based on the experiment we can see: