- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes TernRefine, a method for improving an already ternary-quantized language model. It uses one gradient computation on the quantized model to rank small weight changes, then uses separate validation data to decide how many changes to apply. The three allowed weight values and the number of side-state weights in each group remain unchanged. Experiments on three backbone show lower language-modeling loss and small average improvements on downstream tasks.
Critical strengths
-
The method improves an already compressed model while keeping the same ternary values and the same number of side-state weights in each group. This shows that some performance can be recovered without increasing weight precision.
-
The controlled experiments show that gradients computed on the quantized model guide weight changes better than gradients from the full-precision model. This provides useful evidence about how to refine models after quantization.
Critical weaknesses
My main weakness is that I am not convinced by the practical value demonstrated in this setting. In the MMLU row of Table 4, TernRefine improves the Qwen PT² baseline from 22.90% to 22.95%, a gain of only 0.05 percentage points. Since MMLU is a four-option multiple-choice benchmark, uniform random guessing has an expected accuracy of 25%. Both the ternary baseline and the refined model therefore perform below this level. If the existing baseline and author's methods are lower than random guessing, how can we know that this method is really effective? Given that the proposed method barely improves a baseline that is already below chance, I find it difficult to conclude from these results that the method offers meaningful practical value.
Questions for authors
-
Both the baseline and your method score below random guessing on MMLU. What practical value does the improvement provide?
-
Why keep the number of side-state weights fixed? What happens if you remove this restriction?
-
Why not test BitNet b1.58 or other models trained with ternary weights? Can your method improve them?
4 comments
Response to Question 2: Motivation for Fixed Side-State Cardinality
Thank you for this important question. Our motivation is to investigate whether task performance can be improved by optimizing the allocation of existing ternary side states, without increasing their number or changing the quantization parameters.
Why preserve side-state cardinality? In a conventional ternary representation with codebook , side states correspond to nonzero weights. Their number determines the nonzero-state occupancy and can influence storage and computation in implementations that exploit sparsity. Preserving this number allows us to improve the model while maintaining its original nonzero-state budget. More importantly, the quality of a ternary representation depends not only on how many side states are available, but also on where they are assigned. TernRefine specifically investigates this remaining optimization freedom.
Why impose exact cardinality preservation? Fixing the number of side states isolates the effect of task-aware reassignment from changes in representation occupancy. This directly motivates our donor–receiver relocation mechanism, where each side-state activation is paired with a deactivation within the same group. Consequently, any improvement is achieved by redistributing existing states rather than introducing additional occupied states.
What happens if the constraint is removed? Relaxing cardinality preservation would enlarge the feasible discrete search space and could potentially yield greater accuracy improvements. However, our current formulation and donor–receiver selection rule are specifically designed for exact groupwise occupancy preservation. Removing this constraint would change the set of feasible updates and require a different update-selection strategy, potentially introducing additional optimization and validation considerations. While unrestricted refinement might achieve higher accuracy, it addresses a different optimization problem from our central objective: improving task performance solely through the redistribution of existing ternary states under fixed groupwise occupancy. We therefore deliberately focus on this constrained setting rather than unrestricted codeword reassignment.
For the affine ternarization case, where the center codeword need not be numerically zero, we provide a more detailed explanation of the representation-level motivation in our response to Reviewer VmTV.
Response to Question 3: Evaluation on BitNet b1.58
Thank you for this valuable suggestion. We would first like to clarify that TernRefine is specifically designed for post-training quantization (PTQ) refinement rather than quantization-aware training (QAT). This scope is explicitly established in our Introduction (Section 1), PTQ formulation (Section 3.1), refinement procedure (Section 4), and experimental setup (Section 5.1).
PTQ and native ternary training follow different optimization paradigms. PTQ converts an already pretrained model into a low-bit representation without retraining its original weights, potentially leaving discrete assignment errors introduced during conversion. In contrast, BitNet b1.58 incorporates ternary quantization into training, allowing its parameters to adapt to quantization constraints throughout the training process. Consequently, the two approaches involve substantially different training costs and optimization histories, and the remaining opportunities for discrete refinement may differ. We therefore would not necessarily expect the improvements observed on PTQ-derived models to transfer directly to a natively ternary-trained model.
Nevertheless, we appreciate your suggestion to investigate whether our refinement mechanism can transfer beyond its intended PTQ setting. We therefore conducted an additional experiment on the official BitNet b1.58 2B4T checkpoint, preserving its ternary codebook, quantization scales, and groupwise side-state cardinality, without retraining.
To adapt TernRefine to BitNet's representation, we used the BF16 master weights solely to assign fixed candidate signs to center-state receivers, without re-quantizing the original packed weights. We partitioned the Q/K weight matrices into groups of 128 coordinates only to define occupancy-preserving relocations, without changing BitNet's native quantization granularity or scales. TernRefine was applied to the Q/K projections across all 30 layers, using one quantized-point gradient and validation-based patch selection. The selected patch contains 60 legal relocations, modifying 120 ternary coordinates, with exact groupwise cardinality preservation.
1. Language Modeling Results
| Model | WikiText-2 NLL | WikiText-2 PPL | C4 NLL | C4 PPL |
|---|---|---|---|---|
| Original BitNet | 3.562749 | 35.2600 | 3.416008 | 30.4476 |
| BitNet + TernRefine | 3.560127 | 35.1677 | 3.414667 | 30.4068 |
TernRefine reduces held-out NLL by 0.00262 on WikiText-2 and 0.00134 on C4. These modest improvements provide initial evidence that occupancy-preserving discrete refinement can also reduce language-modeling loss on a natively ternary-trained model under the tested protocol.
2. Downstream Evaluation
We further evaluated the fixed refined checkpoint on six full zero-shot benchmarks, covering 30,737 paired evaluation examples. No downstream data was used for fitting or patch selection.
| Task | Original BitNet (%) | + TernRefine (%) | Δ (pp) |
|---|---|---|---|
| HellaSwag | 68.373 | 68.333 | -0.040 |
| PIQA | 77.149 | 77.258 | +0.109 |
| ARC-Easy | 74.495 | 74.832 | +0.337 |
| ARC-Challenge | 49.573 | 49.147 | -0.427 |
| WinoGrande | 71.981 | 72.691 | +0.710 |
| MMLU | 52.122 | 52.165 | +0.043 |
| Macro Average | 65.616 | 65.738 | +0.122 |
Four of the six task-level point estimates improve, while two decrease. However, all six paired 95% bootstrap confidence intervals include zero, indicating that the observed downstream changes are not statistically reliable under this experimental protocol.
Interpretation. These results demonstrate that the proposed discrete refinement can be applied to a natively ternary-trained model, although the observed improvements are considerably smaller than those obtained in our PTQ experiments. This outcome is consistent with the different optimization histories of PTQ-derived and natively ternary-trained models, without establishing the training paradigm as the cause of the smaller gains.
These results extend the empirical scope of TernRefine beyond PTQ-derived models. Even on a natively ternary-trained BitNet checkpoint, occupancy-preserving discrete relocation reduces held-out language-modeling loss without retraining or modifying the original quantization representation. TernRefine is primarily designed for post-PTQ refinement; the BitNet experiment serves to examine its applicability and limitations beyond this intended setting.
Thank you for taking the time to review our paper. We have provided detailed responses to your concerns and would greatly appreciate any further thoughts you may have, if time permits.
Thank you for your time and consideration.
Response to Question 1: Practical Significance Beyond MMLU
Thank you for raising this important concern. Under aggressive post-training ternarization, substantial performance degradation on challenging, knowledge-intensive tasks such as MMLU is understandable. PTQ must approximate pretrained high-precision weights using an extremely limited ternary representation, without the end-to-end adaptation available during quantization-aware training. As shown in Table 4, Qwen3-8B's MMLU accuracy decreases from 72.95% in full precision to 22.90% after PT² ternarization. Recovering the full capabilities lost under such severe quantization is therefore considerably more challenging than improving language-modeling loss within the existing ternary representation.
The practical value of TernRefine should consequently be evaluated through its ability to recover performance under these constraints, rather than by MMLU alone. Moreover, motivated by your third question, we additionally evaluated TernRefine on the natively ternary-trained BitNet b1.58 2B4T, whose baseline achieves 52.12% MMLU accuracy, substantially above random guessing. TernRefine further reduces held-out language-modeling loss on this stronger ternary model, as detailed in our response to Question 3.
1. Downstream performance beyond random guessing.
Our Table 4 evaluates six downstream benchmarks covering different capabilities, including commonsense reasoning, physical understanding, scientific knowledge, and language comprehension. The following table compares the original PT² baselines with their TernRefine-refined counterparts, alongside the corresponding random-guessing references.
Random references follow Appendix A.7. TR denotes TernRefine applied to the PT² baseline. Bold values indicate the higher accuracy within each baseline–refinement pair.
These results show that, despite the substantial degradation introduced by extreme ternarization, the resulting models retain measurable capabilities above uniform random-guessing references on several nontrivial benchmarks. TernRefine further improves performance on tasks where the quantized models already demonstrate non-random capabilities. For example, HellaSwag improves from 41.54% to 44.81% on Llama-2-7B, PIQA from 62.68% to 64.40%, and ARC-Easy from 33.46% to 37.46% on Qwen3-8B (+4.00 percentage points). These improvements demonstrate that the benefits of TernRefine extend beyond language-modeling loss and are not limited to tasks where the baseline performs near random guessing.
2. Evidence of downstream improvements.
The six-task macro accuracy increases from 36.34% to 37.74% on Qwen3-8B and from 41.99% to 43.30% on Llama-2-7B. Moreover, our paired bootstrap analysis on Llama-2-7B shows a positive macro improvement of +1.309 percentage points (95% CI: [+0.895, +1.719]). None of these downstream tasks was used for fitting or patch selection. These findings complement the held-out NLL/PPL improvements reported in Table 1.
Overall, TernRefine demonstrates that useful task performance can still be recovered within the strict representation constraints of post-training ternarization. Although extreme PTQ may substantially degrade difficult knowledge-intensive tasks, the improvements across language modeling and multiple downstream benchmarks establish the value of refining existing discrete assignments without increasing their representation capacity.