acceptodds
Under review as a conference paper at ICLR 2027

TernRefine: Gradient-Guided Fixed-Capacity Refinement of Ternary LLMs

Abstract

Post-training ternarization produces a compact discrete model, but the resulting ternary assignment need not be optimal for the target task. We introduce TernRefine, a refinement method that reallocates side-state occupancy after quantization while preserving the ternary codebook and the exact number of side states in every quantization group. TernRefine computes a single task gradient at the deployed quantized model, uses the first-order score −⟨G, ∆Q⟩ to rank legal relocations, and selects how many relocations to apply using a separate validation split. We find that the point at which the gradient is evaluated matters substantially. In a controlled study over five fitting offsets, gradients computed at the quantized model and at the corresponding full-precision model produce patches with less than 2% Jaccard overlap and lead to opposite average outcomes on held-out data. Ranking with the quantized model gradient improves WikiText-2 in all five runs, whereas ranking with the full-precision gradient degrades it on average. TernRefine also improves WikiText-2 and C4 on OPT-350M, Llama-2-7B, and Qwen3-8B after affine ternarization. Starting from PT2 models, all three fitted patches improve both datasets on Llama-2-7B and Qwen3-8B, and the selected models show positive average transfer on standard zero-shot tasks. Overall, these results show that ternary models can retain useful task structure after quantization and that gradients evaluated at the deployed discrete state provide an effective signal for refining that structure.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, model quantization, ternary quantization, post-training quantization, model compression, discrete optimization, task-aware refinement

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hLXiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TernRefine, a method for improving an already ternary-quantized language model. It uses one gradient computation on the quantized model to rank small weight changes, then uses separate validation data to decide how many changes to apply. The three allowed weight values and the number of side-state weights in each group remain unchanged. Experiments on three backbone show lower language-modeling loss and small average improvements on downstream tasks.

Critical strengths
  1. The method improves an already compressed model while keeping the same ternary values and the same number of side-state weights in each group. This shows that some performance can be recovered without increasing weight precision.

  2. The controlled experiments show that gradients computed on the quantized model guide weight changes better than gradients from the full-precision model. This provides useful evidence about how to refine models after quantization.

Critical weaknesses

My main weakness is that I am not convinced by the practical value demonstrated in this setting. In the MMLU row of Table 4, TernRefine improves the Qwen PT² baseline from 22.90% to 22.95%, a gain of only 0.05 percentage points. Since MMLU is a four-option multiple-choice benchmark, uniform random guessing has an expected accuracy of 25%. Both the ternary baseline and the refined model therefore perform below this level. If the existing baseline and author's methods are lower than random guessing, how can we know that this method is really effective? Given that the proposed method barely improves a baseline that is already below chance, I find it difficult to conclude from these results that the method offers meaningful practical value.

Questions for authors
  1. Both the baseline and your method score below random guessing on MMLU. What practical value does the improvement provide?

  2. Why keep the number of side-state weights fixed? What happens if you remove this restriction?

  3. Why not test BitNet b1.58 or other models trained with ternary weights? Can your method improve them?

4 comments

Response to Question 1: Practical Significance Beyond MMLU

Thank you for raising this important concern. Under aggressive post-training ternarization, substantial performance degradation on challenging, knowledge-intensive tasks such as MMLU is understandable. PTQ must approximate pretrained high-precision weights using an extremely limited ternary representation, without the end-to-end adaptation available during quantization-aware training. As shown in Table 4, Qwen3-8B's MMLU accuracy decreases from 72.95% in full precision to 22.90% after PT² ternarization. Recovering the full capabilities lost under such severe quantization is therefore considerably more challenging than improving language-modeling loss within the existing ternary representation.

The practical value of TernRefine should consequently be evaluated through its ability to recover performance under these constraints, rather than by MMLU alone. Moreover, motivated by your third question, we additionally evaluated TernRefine on the natively ternary-trained BitNet b1.58 2B4T, whose baseline achieves 52.12% MMLU accuracy, substantially above random guessing. TernRefine further reduces held-out language-modeling loss on this stronger ternary model, as detailed in our response to Question 3.

1. Downstream performance beyond random guessing.

Our Table 4 evaluates six downstream benchmarks covering different capabilities, including commonsense reasoning, physical understanding, scientific knowledge, and language comprehension. The following table compares the original PT² baselines with their TernRefine-refined counterparts, alongside the corresponding random-guessing references.

BenchmarkEvaluated CapabilityRandom ReferenceQwen PT²Qwen + TRLlama PT²Llama + TR
HellaSwagCommonsense event continuation with adversarial distractors25.00%30.32%32.19%41.54%44.81%
PIQAPhysical commonsense and practical problem solving50.00%56.20%56.57%62.68%64.40%
ARC-EasyGrade-school scientific knowledge and reasoning≈25.02%33.46%37.46%42.09%44.36%
ARC-ChallengeChallenging scientific questions and reasoning≈25.02%23.38%24.91%23.89%25.82%
WinoGrandeCommonsense-based pronoun resolution50.00%51.78%52.33%58.72%57.09%
MMLUMultidisciplinary knowledge and understanding25.00%22.90%22.95%23.02%23.30%
Macro AverageSix-task unweighted average—36.34%37.74%41.99%43.30%

Random references follow Appendix A.7. TR denotes TernRefine applied to the PT² baseline. Bold values indicate the higher accuracy within each baseline–refinement pair.

These results show that, despite the substantial degradation introduced by extreme ternarization, the resulting models retain measurable capabilities above uniform random-guessing references on several nontrivial benchmarks. TernRefine further improves performance on tasks where the quantized models already demonstrate non-random capabilities. For example, HellaSwag improves from 41.54% to 44.81% on Llama-2-7B, PIQA from 62.68% to 64.40%, and ARC-Easy from 33.46% to 37.46% on Qwen3-8B (+4.00 percentage points). These improvements demonstrate that the benefits of TernRefine extend beyond language-modeling loss and are not limited to tasks where the baseline performs near random guessing.

2. Evidence of downstream improvements.

The six-task macro accuracy increases from 36.34% to 37.74% on Qwen3-8B and from 41.99% to 43.30% on Llama-2-7B. Moreover, our paired bootstrap analysis on Llama-2-7B shows a positive macro improvement of +1.309 percentage points (95% CI: [+0.895, +1.719]). None of these downstream tasks was used for fitting or patch selection. These findings complement the held-out NLL/PPL improvements reported in Table 1.

Overall, TernRefine demonstrates that useful task performance can still be recovered within the strict representation constraints of post-training ternarization. Although extreme PTQ may substantially degrade difficult knowledge-intensive tasks, the improvements across language modeling and multiple downstream benchmarks establish the value of refining existing discrete assignments without increasing their representation capacity.

Response to Question 2: Motivation for Fixed Side-State Cardinality

Thank you for this important question. Our motivation is to investigate whether task performance can be improved by optimizing the allocation of existing ternary side states, without increasing their number or changing the quantization parameters.

Why preserve side-state cardinality? In a conventional ternary representation with codebook {−1,0,+1}\{-1,0,+1\}, side states correspond to nonzero weights. Their number determines the nonzero-state occupancy and can influence storage and computation in implementations that exploit sparsity. Preserving this number allows us to improve the model while maintaining its original nonzero-state budget. More importantly, the quality of a ternary representation depends not only on how many side states are available, but also on where they are assigned. TernRefine specifically investigates this remaining optimization freedom.

Why impose exact cardinality preservation? Fixing the number of side states isolates the effect of task-aware reassignment from changes in representation occupancy. This directly motivates our donor–receiver relocation mechanism, where each side-state activation is paired with a deactivation within the same group. Consequently, any improvement is achieved by redistributing existing states rather than introducing additional occupied states.

What happens if the constraint is removed? Relaxing cardinality preservation would enlarge the feasible discrete search space and could potentially yield greater accuracy improvements. However, our current formulation and donor–receiver selection rule are specifically designed for exact groupwise occupancy preservation. Removing this constraint would change the set of feasible updates and require a different update-selection strategy, potentially introducing additional optimization and validation considerations. While unrestricted refinement might achieve higher accuracy, it addresses a different optimization problem from our central objective: improving task performance solely through the redistribution of existing ternary states under fixed groupwise occupancy. We therefore deliberately focus on this constrained setting rather than unrestricted codeword reassignment.

For the affine ternarization case, where the center codeword need not be numerically zero, we provide a more detailed explanation of the representation-level motivation in our response to Reviewer VmTV.

Response to Question 3: Evaluation on BitNet b1.58

Thank you for this valuable suggestion. We would first like to clarify that TernRefine is specifically designed for post-training quantization (PTQ) refinement rather than quantization-aware training (QAT). This scope is explicitly established in our Introduction (Section 1), PTQ formulation (Section 3.1), refinement procedure (Section 4), and experimental setup (Section 5.1).

PTQ and native ternary training follow different optimization paradigms. PTQ converts an already pretrained model into a low-bit representation without retraining its original weights, potentially leaving discrete assignment errors introduced during conversion. In contrast, BitNet b1.58 incorporates ternary quantization into training, allowing its parameters to adapt to quantization constraints throughout the training process. Consequently, the two approaches involve substantially different training costs and optimization histories, and the remaining opportunities for discrete refinement may differ. We therefore would not necessarily expect the improvements observed on PTQ-derived models to transfer directly to a natively ternary-trained model.

Nevertheless, we appreciate your suggestion to investigate whether our refinement mechanism can transfer beyond its intended PTQ setting. We therefore conducted an additional experiment on the official BitNet b1.58 2B4T checkpoint, preserving its ternary codebook, quantization scales, and groupwise side-state cardinality, without retraining.

To adapt TernRefine to BitNet's representation, we used the BF16 master weights solely to assign fixed candidate signs to center-state receivers, without re-quantizing the original packed weights. We partitioned the Q/K weight matrices into groups of 128 coordinates only to define occupancy-preserving relocations, without changing BitNet's native quantization granularity or scales. TernRefine was applied to the Q/K projections across all 30 layers, using one quantized-point gradient and validation-based patch selection. The selected patch contains 60 legal relocations, modifying 120 ternary coordinates, with exact groupwise cardinality preservation.

1. Language Modeling Results

ModelWikiText-2 NLLWikiText-2 PPLC4 NLLC4 PPL
Original BitNet3.56274935.26003.41600830.4476
BitNet + TernRefine3.56012735.16773.41466730.4068

TernRefine reduces held-out NLL by 0.00262 on WikiText-2 and 0.00134 on C4. These modest improvements provide initial evidence that occupancy-preserving discrete refinement can also reduce language-modeling loss on a natively ternary-trained model under the tested protocol.

2. Downstream Evaluation

We further evaluated the fixed refined checkpoint on six full zero-shot benchmarks, covering 30,737 paired evaluation examples. No downstream data was used for fitting or patch selection.

TaskOriginal BitNet (%)+ TernRefine (%)Δ (pp)
HellaSwag68.37368.333-0.040
PIQA77.14977.258+0.109
ARC-Easy74.49574.832+0.337
ARC-Challenge49.57349.147-0.427
WinoGrande71.98172.691+0.710
MMLU52.12252.165+0.043
Macro Average65.61665.738+0.122

Four of the six task-level point estimates improve, while two decrease. However, all six paired 95% bootstrap confidence intervals include zero, indicating that the observed downstream changes are not statistically reliable under this experimental protocol.

Interpretation. These results demonstrate that the proposed discrete refinement can be applied to a natively ternary-trained model, although the observed improvements are considerably smaller than those obtained in our PTQ experiments. This outcome is consistent with the different optimization histories of PTQ-derived and natively ternary-trained models, without establishing the training paradigm as the cause of the smaller gains.

These results extend the empirical scope of TernRefine beyond PTQ-derived models. Even on a natively ternary-trained BitNet checkpoint, occupancy-preserving discrete relocation reduces held-out language-modeling loss without retraining or modifying the original quantization representation. TernRefine is primarily designed for post-PTQ refinement; the BitNet experiment serves to examine its applicability and limitations beyond this intended setting.

Thank you for taking the time to review our paper. We have provided detailed responses to your concerns and would greatly appreciate any further thoughts you may have, if time permits.

Thank you for your time and consideration.

Reviewer_VmTVOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper introduces a method to improve the performance of ternary LLMs while preserving quantization and sparsity. Ternary LLMs are models whose weights are divided into groups, where the weights within each group can only take values from the set Cg={μg−αg,μg,μg+αg}\mathcal{C}_g = \{\mu_g-\alpha_g, \mu_g, \mu_g+\alpha_g\}. The weights other than μg\mu_g are called side states. The idea is to calculate the gradient of the loss of the given ternary model and update the weights by reassigning them within each group, while preserving the elements of SgS_g and the number of side states based on the gradient, and hence preserving the same level of quantization and sparsity. The paper provides a principled method for doing so and presents experiments showing that the method outperforms the baselines.

Critical strengths
  1. The proposed method preserves the same level of quantization, the same set of states for each group, and the number of side states. This can be quite useful, as the method itself is cheap, and preserving sparsity and quantization can be important to practitioners if the method scales well.

  2. The paper is well written, the method is easy to understand, and the notation is clear and easy to follow while preserving rigour.

  3. The experiments are conducted carefully, with fitting and selection data kept separate, and the results are reported clearly.

Critical weaknesses

The core idea of the paper is already presented in Li et al. (2026), which the paper also cites. The idea of computing the gradient and reassigning the weights seems quite natural after reading Li et al. (2026), and the subsequent adaptive patch growth step is essentially early stopping. I therefore do not think the paper, in its current form, is sufficiently novel for the venue.

I could point out other technical issues, but since the review asks me to focus on issues that could change my overall assessment, I would like to make the originality and novelty of the idea my main concern and keep the discussion limited to that.

Questions for authors
  1. Could the authors clarify how their method differs from the ideas presented in Li et al. (2026) and whether the changes are substantial?

  2. Preserving the number of side states is understandable when one of the states is zero (e.g., μg=0\mu_g = 0), as this preserves the sparsity of the quantized model. Could the authors clarify the motivation for preserving the number of side states in the affine case, where doing so does not preserve sparsity? Why not allow the number of side states to change to potentially further improve performance?

3 comments

Response to Question 1: Substantive Novelty and Distinction from Li et al. (2026)

Thank you for this important comment. We believe an important distinction lies in the nature of the two contributions. Li et al. (2026) primarily provide a theoretical explanation of existing QAT behavior. As they explicitly state in Section 1.1 (Related Work), their goal is “rather than proposing a new quantization algorithm, we study a mechanism explaining when and why QAT can correct a PTQ failure.” Their contribution is to explain how gradients evaluated at quantized weights guide continuous latent-weight updates toward low-loss regions. In contrast, TernRefine proposes a concrete optimization algorithm for directly refining an already ternarized model under exact groupwise occupancy constraints.

A distinct constrained optimization problem. Li et al. explain why quantized-point gradients are informative during QAT, but do not investigate how to use them to optimize an already constructed discrete representation without latent-weight updates. TernRefine addresses this different problem: improving task performance by redistributing existing side-state assignments while preserving the codebook, quantization parameters, and groupwise occupancy. This constraint has a practical representation-level motivation, further explained in our response to your second question.

From quantized-point gradients to feasible discrete updates. Under exact cardinality constraints, individual occupancy updates are infeasible: activating a side state requires simultaneously deactivating another within the same group. Consequently, a favorable continuous gradient direction need not correspond to a feasible discrete improvement. TernRefine addresses this challenge through structured donor–receiver relocations. By projecting task gradients onto legal exchanges, we derive a separable first-order objective that enables efficient identification of the best first-order relocation within each group, without continuous optimization or exhaustive pairwise enumeration.

Our substantive contribution is therefore not a new explanation of quantized-point gradients, but the formulation and structured solution of task-guided discrete refinement under fixed groupwise occupancy. Our controlled experiments support the effectiveness of this approach. We also agree that APG employs a conventional validation-based stopping principle and do not claim early stopping itself as a standalone innovation. We will revise the paper to make the distinction between the theoretical insight of Li et al. and our algorithmic contribution explicit.

Response to Question 2: Motivation for Fixed Side-State Cardinality in Affine Ternarization

Thank you for this insightful question. We agree that affine ternarization does not necessarily preserve numerical weight sparsity. However, its shared groupwise center parameters allow the zero-valued states of the underlying ternary representation to remain exploitable for potentially efficient storage and computation. Affine ternary weights can be equivalently represented using zero-centered ternary states and shared groupwise offsets. Since these offsets are already part of the quantization parameters, no additional per-weight offsets are required. The original affine computation can be preserved through groupwise corrections, allowing suitable implementations to exploit the zero-valued ternary states with limited additional parameter-storage overhead.

This provides a practical motivation for our fixed-cardinality constraint beyond merely restricting the search space. Preserving side-state cardinality maintains the number of nonzero ternary states within each group, which can support a fixed nonzero-entry budget and predictable resource requirements in implementations that exploit these states. The actual storage and computational benefits remain implementation-dependent.

We acknowledge that relaxing the cardinality constraint could yield further accuracy improvements. However, as emphasized in our response to your first question, TernRefine deliberately investigates a distinct constrained optimization problem: whether task performance can be improved solely by reallocating existing side states without changing their groupwise occupancy. Exact cardinality preservation isolates the effect of reassignment from changes in the number of occupied side states. This constraint directly motivates our donor–receiver relocation mechanism, in which each activation is paired with a deactivation within the same group.

We will clarify that fixed cardinality preserves the nonzero-state count of the underlying ternary representation rather than numerical weight sparsity, without claiming implementation-independent storage or computational gains.

Thank you for your replies. I have understood the clarifications regarding the distinction from Li et al. (2026) and the reasoning about preserving the number of side states in the affine case. The explanations have resolved my main doubts; I increased my score accordingly.

Thank you for your thoughtful follow-up and reconsideration. We sincerely appreciate your constructive feedback and consideration.

Reviewer_tCqKOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TernRefine modifies the discrete ternary assignments of an already post-training-quantized language model without increasing its codebook or side-state capacity. It scores legal relocation proposals using a gradient evaluated at the deployed quantized model, rather than at the full-precision source, and selects how many moves to apply through adaptive validation (APG). Experiments span OPT-350M, Llama-2-7B and Qwen3-8B, different quantizers, held-out language modeling datasets and standardized downstream tasks. The discrete-codebook and cardinality constraints are not incidental: they prevent hiding an improvement in additional storage or precision. I therefore separate evidence for the quantized-state gradient ranking from the operational performance of adaptive prefix validation, which is much more computationally intensive than the single backward pass highlighted in the formulation.

Critical strengths
  1. The decisive evidence is the matched evaluation-point experiment, not just the final perplexity. With the same proposal pool and edit density, Table 2 reports OPT-350M mean W2 delta NLL of -0.0627 for quantized-point gradients versus +0.1075 for full-precision-point gradients; the latter frequently worsens the deployed model. This supports the method's actual mechanism. The candidate-matched evaluation-point control is particularly persuasive because different proposal pools could otherwise account for the gain. Table 2 reports sign-consistent improvement across validation, Wikitext-2 and C4 in multiple fits, whereas the corresponding FP-gradient ranking can damage the deployed model. This probes the proposed mechanism rather than comparing unrelated PTQ recipes.

  2. Fixed-capacity refinement is a meaningful constraint: gains are not purchased by hidden higher-precision weights or auxiliary trainable adapters. For Qwen3-8B under PT2 W2, Table 1 reports perplexity 49.204 before refinement versus 32.844 +/- 0.426 afterward, a substantial reduction in quantization damage. The magnitude of the Qwen3 improvement would be much less informative if the procedure expanded the effective bit budget. Fixed cardinality ensures the effect is attributable to a better local discrete assignment. The full-precision reference also makes it possible to quantify remaining damage, rather than mistaking a large relative recovery for complete functional restoration.

  3. The appendix reports unfavorable module/allocation controls rather than only favorable endpoints, and downstream tasks are not used to select interventions. These choices improve interpretability of the results. The poor global allocation is a useful counterexample to the stronger—but unjustified—claim that any sequence of Q-point edits is safe. In practice the layer budget and stopping rule may matter as much as the gradient source. Reporting failure modes increases confidence that the authors inspected interventions rather than selecting only positive trials.

Critical weaknesses
  1. The method is sensitive to allocation and model/task conditions. Appendix A.3 shows a global Q-point allocation that improves fitting loss but worsens Llama held-out W2 NLL by +0.0824; balancing edits per layer changes that to -0.0472. Appendix A.4 similarly warns that expanding to all linear module groups can hurt. Thus deployed-state gradients are helpful but do not make selection plug-and-play. The +0.0824 W2 regression under global allocation is particularly revealing because the fitting objective improves. An effective deployment recipe needs to state how a practitioner detects and prevents this reversal without silently optimizing on the downstream test distribution. The proposed layer cap is promising but should be justified as a transferable rule rather than a case-specific repair.

  2. The phrase 'one task gradient' should not imply negligible pipeline cost. At Qwen3-8B scale, APG's validation forwards account for approximately 93% of a 116-second pipeline (Section 5.4/Appendix A.5). Full inference/validation work and memory should be compared with budget-matched PTQ corrections, not solely the number of backwards passes. The reported 93% share of validation time means wall-clock superiority cannot be inferred from counting gradient evaluations. If a competitor performs more backwards but much less validation, relative deployment cost might reverse. A direct breakdown by gradient work, candidate processing and evaluation forwards—plus peak memory—would turn an abstract efficiency claim into a reproducible systems result.

  3. Some extreme ternary configurations remain far from full-precision downstream utility, even after large perplexity improvements. The reported gains should be scoped to the tested quantizers and task metrics; loss recovery is not equivalent to recovering general model capability. Wikitext-2/C4 perplexity is necessary but not sufficient for quality of instruction following, factual recall or reasoning under extreme quantization. Since downstream tasks are already measured, a table identifying null and negative task deltas could reveal what types of damage Q-point edits do and do not repair. Model and selected projection scope should likewise be stated explicitly.

Questions for authors
  1. Please distinguish fixed relocation design from adaptive stopping: which settings transfer unchanged across models, and which are chosen using W2/C4? A concise procedural rule relating per-layer caps, patch density and held-out loss would make the +0.0824 versus -0.0472 reversal reproducible and interpretable.

  2. For every measured model scale, give wall time and memory broken into gradient, proposal selection and all APG validation passes, including validation-token counts. Compare with the strongest viable post-PTQ correction under matched total compute rather than an equal number of backward passes.

  3. Please provide task-by-task downstream changes, including neutral and negative cases, and quantify whether validation perplexity improvements predict task accuracy gains. Which model/quantizer/module settings should a practitioner expect not to benefit?

7 comments

Response to Question 1: Relocation Rules and Adaptive Stopping

Thank you for this important question. The relocation and stopping rules serve different purposes. CPSR defines the legal donor–receiver exchanges, QGP ranks them using one gradient evaluated at the quantized model, and APG determines how many ranked exchanges to apply. The codebook, quantization parameters, and groupwise side-state cardinality remain fixed throughout. Within each layer, we retain one candidate per eligible group and rank these candidates once. Neither the gradient nor the ranking is updated during patch selection.

We use a layer-balanced allocation with (K) relocations per eligible layer, corresponding to (K|\mathcal M|) relocations and (2K|\mathcal M|) modified coordinates in total. APG evaluates a predefined sequence of prefixes and returns the preceding prefix when validation NLL first fails to improve. The stopping rule is fixed; only the selected (K) depends on validation loss. This is a practical stopping criterion, not a claim that the selected prefix is globally optimal.

In our cross-model E1 experiments, Llama-2-7B and Qwen3-8B use the same group size (128), Q/K module scope, single-gradient ranking procedure, and candidate schedule (K\in{0,1,2,4,8,16,32,64}). For Llama-2-7B (32 layers), validation NLL increases from 7.600570 at (K=32) to 7.602278 at (K=64), so APG selects (K^=32), corresponding to 1,024 relocations. For Qwen3-8B (36 layers), validation NLL increases from 4.211204 at (K=8) to 4.213169 at (K=16), yielding (K^=8), or 288 relocations. Thus, different patch sizes arise from the same selection procedure rather than model-specific choices based on W2/C4 evaluation results.

The allocation reversal in Appendix A.3 illustrates why the layer-balanced constraint matters. Under the same 384-relocation budget on Llama-2-7B, we obtain:

AllocationΔValidation NLLΔW2 NLLΔC4 NLL
Global Q-point+0.0869+0.0824−0.0907
Layer-balanced Q-point−0.0604−0.0472−0.0680

Although global allocation reduces fitting loss, it worsens validation and W2 NLL. Layer-balanced allocation avoids this reversal by limiting the concentration of edits within individual layers. The global strategy performs better on C4 in this comparison, so we do not claim that layer balancing is universally optimal. Rather, this result shows that useful local QGP scores do not necessarily remain beneficial when edits are combined without controlling their distribution.

Finally, APG uses only its designated selection-validation windows to choose (K). In E1, the untouched W2 evaluation windows are disjoint token segments from the same WikiText-2 validation source, while C4 uses a separate validation source. Neither evaluation set is used for stopping or patch selection. The stronger PT² replications additionally record fitting/selection indices and disjointness checks.

Together, these results clarify the division of responsibilities: QGP provides the fixed ranking, layer-balanced allocation controls the distribution of edits, and APG selects the patch extent using validation loss. The computational cost of this validation procedure is addressed separately in our response to Question 2.

Response to Question 2 (Part 1/2): Runtime, Memory, and APG Workload

Thank you for emphasizing total refinement cost rather than backward-pass count alone. This first part reports measured runtime, stage-wise GPU memory, and all APG validation passes across three model scales.

1. Measured Refinement Cost and APG Workload

All measurements use an NVIDIA A100-SXM4-80GB. Table Q2-1 includes every APG evaluation, including K=0K=0 and the first non-improving prefix. Times are in seconds; GPU memory is peak allocated/reserved GiB.

Table Q2-1. Refinement runtime and validation workload.

ModelFitGradientQGPProposal selectionAPG forwardOther processingWall timePeak GPU memoryAPG passesValidation positions (input/prediction)
OPT-350M—0.12615.7680.8630.548—17.3050.951 / 0.994816,384 / 16,384
Llama-2-7B03.79313.1930.18116.10952.95586.23122.074 / 22.8286196,608 / 196,512
173.74213.0390.23018.79654.24390.05022.074 / 22.8287229,376 / 229,264
334.25414.6580.1478.05959.24786.36522.074 / 22.828398,304 / 98,256
Qwen3-8B02.8847.4770.00218.713130.426159.50231.453 / 32.3186196,608 / 196,512
171.7556.7010.00315.596111.972136.02731.460 / 32.2795163,840 / 163,760
331.5138.4110.22115.626122.294148.06531.460 / 32.2795163,840 / 163,760

Each Llama/Qwen APG pass processes 32,768 input / 32,752 prediction positions. The selected K∗K^* values are 8/16/1 for Llama and 8/4/4 for Qwen (fits 0/17/33); OPT evaluates K={0,1,2,4,8,16,32,64}K=\{0,1,2,4,8,16,32,64\} and selects 32. The OPT prediction-position total follows its original profiling convention.

Table Q2-2. Stage-wise GPU peak allocated/reserved memory (GiB).

ModelFitGradientQGP scoringProposal selectionAPG validation
OPT-350M—0.951 / 0.9940.635 / 0.9940.637 / 0.9940.756 / 0.994
Llama-2-7B022.074 / 22.82812.567 / 22.82812.567 / 22.82813.177 / 22.828
1722.074 / 22.82812.567 / 22.82812.567 / 22.82813.177 / 22.828
3322.074 / 22.82812.567 / 22.82812.567 / 22.82813.177 / 22.828
Qwen3-8B031.453 / 32.31815.274 / 32.31815.274 / 32.31818.172 / 32.318
1731.460 / 32.27915.273 / 32.27915.273 / 32.27918.171 / 32.279
3331.460 / 32.27915.273 / 32.27915.273 / 32.27918.171 / 32.279

Accounting boundaries. “Other processing” includes costs not assigned to the separately timed stages; it is a wall-time residual, not a directly profiled single operation. For Llama, it includes state auditing, serialization, and orchestration following incremental patch transitions. For Qwen, the near-zero proposal-selection timings cover prefix construction only; patch application and full-state audits fall in the residual. Its timestamp-derived wall intervals end with a verified, serialized patch, before final dense-weight writeback. These measurements exclude initial PTQ, model/data loading, and held-out evaluation. Reserved GPU memory includes allocator caching. The earlier Llama compatibility-selection timing included repeated dense patch application and audits; an incremental implementation preserved the fit-0 selected state and validation NLL in an equivalence check.

Response to Question 2 (Part 2/2): Matched-Compute Comparison

This second part compares correction quality under measured compute budgets, following the runtime and memory breakdown in Part 1.

2. Controlled Correction-Budget Comparison with ReQuant

We compare against ReQuant (Ma et al., 2026), a relevant post-PTQ fixed-grid discrete correction method. Within each ordinary-affine Q/K experiment, both methods use the same quantized starting state, evaluation protocol, and A100 GPU. The recorded budgets include TernRefine's gradient, QGP, proposal selection, and APG validation, and ReQuant's statistics preparation, initialization computations, and coordinate updates.

Table Q2-3. Matched recorded correction budgets and extended ReQuant runs. Negative ΔNLL indicates improvement.

ModelMethodMeasured correction time (s)ΔWikiText-2 NLLΔC4 NLL
OPT-350MTernRefine-APG1.841−0.1532−0.1249
ReQuant, matched time1.791+0.0923+0.1734
ReQuant, extended optimization38.792+0.0190+0.1134
Qwen3-8BTernRefine-APG12.753−0.5179−0.5139
ReQuant, matched time12.663+0.0177+0.0019
ReQuant, extended optimization599.780−0.1791−0.1325

TernRefine yields lower held-out NLL than the tested ReQuant adaptation at both matched recorded budgets. Even with substantially longer ReQuant runs, the final NLL remains higher in these experiments. ReQuant is not uniformly ineffective: Appendix A.6 reports positive results for its original calibration settings, which use a different starting state and budget and should be considered separately.

3. Interpretation and Compute Boundaries

The evidence supports a favorable quality–compute tradeoff under the controlled affine correction workloads, not a claim of fully matched end-to-end PT² cost. The Table Q2-3 budgets exclude initial PTQ and held-out evaluation and do not share the complete audit/serialization boundaries used in Table Q2-1; the comparison is therefore matched within its recorded correction stages, not across every state-processing cost. A fully audited equal-total-cost comparison, including all patch application and finalization, remains distinct from the measurements reported here.

The high Qwen “Other processing” values also show why a single-backward-pass description cannot stand in for runtime accounting. One backward pass is used to form the fixed QGP ranking; APG forward evaluations and state processing add substantial cost. TernRefine nevertheless improves held-out NLL under the tested correction budgets while preserving the original ternary codebook and exact groupwise side-state cardinality.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Revised Manuscript: Summary of Changes and Additional Evidence

Dear Reviewers and Area Chair,

Thank you for the constructive reviews and discussion. We have uploaded a revised manuscript, with changes highlighted in blue. The abstract, core method, and original experimental results remain unchanged. The revision clarifies our problem setting, brings key experimental evidence into the main text, and provides additional evaluations of the method's scope.

1. Motivation and method (Secs. 1–4).

We clarify the motivation for fixed-capacity refinement, where improvements arise from reallocating existing ternary side states rather than increasing their number. The distinction from prior work is also made more explicit. While Li et al. (2026) investigate quantized-point gradients in QAT with latent-weight updates, TernRefine uses these gradients to rank legal donor–receiver relocations in an already quantized model under exact groupwise cardinality constraints. We further clarify the receiver-sign convention, layer-balanced allocation, and the computational roles of QGP and APG.

2. Main experimental evidence (Sec. 5; Fig. 3).

We bring previously reported per-fit results from Appendices A.2 and A.8 into the main text through Fig. 3. Under matched candidate pools and budgets, Q-point ranking outperforms FP-point ranking on WikiText-2 across all eight OPT-350M and Llama-2-7B comparisons. For the stronger PT² initialization, all three independently fitted patches improve both WikiText-2 and C4 on Llama-2-7B and Qwen3-8B.

We also incorporate the original module-scope results into Sec. 5.4, showing that refinement quality depends on the editable modules and the composition of individual relocations.

3. Method scope and additional evaluation (Appendices A.9–A.12).

The additional experiments clarify the objectives and operating boundaries of TernRefine.

Appendix A.9 examines the relationship between validation-loss selection and downstream performance. APG selects patch extent using validation language-modeling loss, rather than downstream accuracy. The independent prefix analysis illustrates this distinction and characterizes the behavior of larger, post-selection prefixes.

Appendix A.10 extends the evaluation to a natively trained ternary model, BitNet b1.58. The results show that capacity-preserving refinement remains applicable beyond post-training ternarization, with modest held-out language-modeling improvements in this setting.

Appendix A.11 provides additional runtime and memory measurements, distinguishing the single backward pass used for ranking from the full refinement pipeline. Appendix A.12 reports a correction-budget comparison with ReQuant, with the differences in cardinality constraints and timing boundaries stated explicitly.

Together, these additions clarify the setting in which TernRefine operates, the objective used to select its patches, and the scope of the reported comparisons.

We hope these revisions make the method and its experimental evidence easier to assess. Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.