acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Quantized LLM Fine-Tuning: An Adaptive-Grouping Alternating Zeroth-Order Algorithm in Continuous Mapping Space

Abstract

Fine-tuning quantized Large Language Models (LLMs) is crucial for model adaptation on resource-constrained edge devices, where zeroth-order (ZO) optimization has emerged as a memory-friendly alternative to first-order methods. Existing ZO fine-tuning methods typically focus on either perturbing discrete quantized weights or coarse-grained (per-tensor/channel) continuous scales. In this paper, we rethink this fundamental choice and recognize that while fine-tuning should ideally target the dequantized weights, the discrete nature of weights restricts optimization to a finite search space, and tuning coarse-grained scales is limited to a constrained continuous subspace that lacks the expressive capacity of the full parameter space. To mitigate these issues, we propose to break down scales and zero-points to a fine-grained level and fine-tune them in a continuous mapping space. Motivated by this, we propose CoZO, an adaptive-grouping alternating zeroth-order algorithm that optimizes these fine-grained parameters while adaptively balancing search-space expressiveness and estimation stability. Crucially, we provide a theoretical convergence guarantee for CoZO. Extensive experiments demonstrate that CoZO achieves superior stability and effectiveness, consistently outperforming discrete-perturbation and scale-tuning baselines under stringent memory constraints.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Quantized Large Language Models, Zeroth-Order Optimization, Memory-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bk4mOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CoZO, a memory-efficient ZO fine-tuning method for quantized LLMs. It optimizes fine-grained dequantization scales and zero-points in a continuous space, using alternating optimization and adaptive grouping to balance expressiveness and estimation variance. Theoretical analysis and experiments support its effectiveness.

Critical strengths

The paper is clearly written and well-structured, with its main claims supported by theoretical analysis or experiments. The topic is well-motivated and relevant to the community, as memory-efficient LLM fine-tuning can make LLM adaptation more accessible to users with limited computational resources.

Critical weaknesses
  1. The novelty is somewhat limited. Compared with QZO, the main difference is the introduction of learnable zero-points, which are common in quantization literature. Alternating optimization and adaptive grouping seem more like add-ons rather than being closely tied to the core motivation.
  2. The evaluation relies mainly on OPT and LLaMA-2, which are not sufficiently representative of modern LLM architectures. A similar concern applies to the selection of downstream datasets, which are relatively outdated and limited in diversity.
  3. The paper lacks comparisons with representative PTQ methods, such as OmniQuant, SpinQuant, and FlatQuant. These methods could also serve as meaningful baselines when applied to fine-tuned models and evaluated on downstream tasks. Such comparisons would help better establish the advantages of the proposed method.
Questions for authors
  1. Have you repeated your main experiments with multiple random seeds? Unlike other research topics, ZO optimization is sensitive to random seeds, leading to noticeable performance variations. Although Table 10 reports results across different seeds, the evaluation is still somewhat limited, especially considering that the improvements from alternating optimization and adaptive grouping are relatively modest.
  2. From my understanding, CoZO uses the loss difference as the signal for adaptive grouping. However, the theoretical variance bound depends on the gradient norm, which does not necessarily mean that a single loss difference reliably reflects estimation variance. Moreover, different perturbation directions may produce very different loss differences, especially when using only one direction per update as in MeZO. Could you clarify the theoretical justification and reliability of this strategy?
7 comments

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper. Below, we provide point-by-point responses to all the weaknesses and questions raised. We have also revised the original manuscript accordingly, with all changes highlighted in blue.

W1:The novelty is somewhat limited. Compared with QZO, the main difference is the introduction of learnable zero-points, which are common in quantization literature. Alternating optimization and adaptive grouping seem more like add-ons rather than being closely tied to the core motivation.

We thank the reviewer for the thoughtful comments on the novelty of our method. We agree that learnable zero-points are not a new concept in themselves. However, the core contribution of this work is not simply the introduction of zero-points, but rather a set of interconnected optimization designs in continuous space that address the limitations of the optimization space in zeroth-order fine-tuning of quantized models. We would like to further elaborate on the following aspects.

  • Motivation for Introducing Zero-Points.

    Existing zeroth-order fine-tuning methods that optimize only scales are significantly limited in the directions along which they can adjust model weights. As analyzed in Appendix C.9, zero-points provide additional translation directions that cannot be achieved through scale-only optimization. Our introduction of zero-points is therefore not simply intended to increase the number of trainable parameters, but is motivated by an analysis of the optimization structure of the continuous mapping space.

  • Design Motivation for Alternating Optimization and Adaptive Grouping.

    However, while fine-grained continuous parameterization enhances expressiveness, it also increases the search dimension of zeroth-order gradient estimation. As the reviewer pointed out in Q1, zeroth-order optimization inherently suffers from estimation instability, which becomes more pronounced as the optimization dimension increases. To address this issue, we introduce alternating optimization to reduce the search dimension of each gradient estimation step, together with adaptive grouping to dynamically balance the expressiveness of the search space and estimation stability throughout training. Therefore, these two designs are not merely additional modules, but interconnected solutions to the dimensionality and stability challenges arising in zeroth-order optimization within fine-grained continuous spaces.

W2:The evaluation relies mainly on OPT and LLaMA-2, which are not sufficiently representative of modern LLM architectures. A similar concern applies to the selection of downstream datasets, which are relatively outdated and limited in diversity.

We agree with the reviewer's suggestion to extend our evaluation to more advanced models and a broader range of downstream tasks. To this end, we conducted additional experiments on Qwen3-8B and introduced the GSM8K mathematical reasoning benchmark, using both Strict EM and Flexible EM metrics to evaluate the applicability of CoZO across newer model architectures and different task types. The corresponding experimental results have been added to Appendix C.7 of the revised manuscript.

Table: Task Evaluation Results on Qwen3-8B

SST2CBWICWSCGSM8K strict EMGSM8K flexible EM
Zero-Shot51.4951.7952.0455.1769.9872.71
QZO78.3269.6460.6657.6967.8571.27
QuZO80.3980.3655.1761.5454.0657.32
CoZO81.7783.9360.8263.4674.4575.73
Reviewer_aFVwOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CoZO, a zeroth-order fine-tuning method for post-training-quantized LLMs that freezes the integer codes WqW_q and optimizes the dequantization parameters SS (scale) and ZZ (zero-point), grouped into dd blocks with an adaptive split/merge rule every KK steps. §3.1 frames the design space as a discrete mapping space (perturbing WqW_q) versus a full continuous mapping space (varying S,ZS, Z, which can express any weight matrix), and §4 gives convergence results under an adaptive-grouping analysis. Experiments on OPT-6.7B / Llama-2-7B (4-bit, OmniQuant) show CoZO beating QuZO and QZO by 2.4–5.8 points on the 8-task average at roughly one-third of QuZO's peak memory, and essentially tying 16-bit MeZO on OPT-6.7B.

Critical strengths
  1. The mapping-space framing in §3.1 is genuinely clarifying. Showing that the continuous space equals all achievable dequantized weights turns a vague intuition into a precise statement about what each existing ZO family can reach; the zero-point projection analysis (C.9) is a small, independently checkable contribution on its own.
  2. The empirical margins on the two 7B-class models are large and internally consistent (+4.9/+5.8 over QuZO/QZO on OPT-6.7B), and CoZO at 4 bits essentially ties 16-bit MeZO (70.57 vs 70.46) — the practically interesting claim that quantized ZO fine-tuning need not cost full-precision accuracy.
  3. Honest accounting. Memory, training time, and the QZO baseline's 4× step advantage are all reported openly; the evaluation surface (6 backbones, 8 tasks, ablations, seeds, sensitivity) is broad for a 9-page paper.
Critical weaknesses
  1. The convergence theorems do not cover the algorithm as actually run. Thm 4.7/4.8 require K = Ω(T) (Θ(1) regroups total), but the experiments use K = 125 with T = 5k/8k — 40–64 regrouping events, a number growing linearly in T. The bounded-regrouping conditions appear only in an appendix and the key assumption ET=O(1)E_T = O(1) is neither proved under constant K nor measured.
  2. The §3.1 variance comparison does not support its conclusion. The two bounds are stated at different dimensions; at the same dimension the gap is a constant factor 4 arising from two slack triangle inequalities in the proof, not from quantization. Two upper bounds that cannot be ordered do not establish a stability ordering.
  3. The adaptive-grouping trigger is structurally asymmetric and may never split. The window mean of a nonnegative right-skewed |g| sits below the p-quantile, so merge is reachable while split is nearly unreachable — biasing dd downward. The sensitivity tables (identical numbers across p ∈ {0.7, 0.8, 0.9} and across d0 ∈ {4, 8, 16}) are consistent with an inert split branch. Report the dtd_t trajectory and split/merge event counts.
  4. The central explanatory quantity — expressiveness via dd — is never measured. dtd_t and dmaxd_{max} are never reported, so the reader cannot tell how much of the gain over scale-only QZO comes from finer grouping versus simply doubling the continuous degrees of freedom (S and Z); the missing control is S-only vs S-and-Z at matched parameter count.
Questions for authors
  1. Q1 the theory does not cover the algorithm as run. Thm 4.7/4.8 require K = Ω(T), but the experiments use K = 125 with T = 5k/8k (40–64 regroups, growing linearly in T). Please either rerun with K = cT for a constant c and report those numbers, or state and prove a variant that permits a fixed K with Θ(T/K) regroups and an explicit per-event loss-increase bound — and say how ETE_T scales in T/K, since the proof's ET=O(1)E_T = O(1) cannot hold for 40–64 events without an additional decay assumption.
  2. Q2 the variance comparison does not support its conclusion. Eq. (10) and Eq. (11) are stated at different dimensions (dim(θ) vs the weight dimension m). Please redo the comparison at a single dimension, and separate the constant factor 4 in Eq. (11) — which comes from the two slack inequalities in Eq. (32) — from the genuinely quantization-dependent terms. If the honest conclusion is that quantized RGE's variance is inflated "by a bounded factor plus error terms" rather than by a dimension-dependent penalty, Remark 3.5 should say so, and §3.1 should be presented as a heuristic rather than a proven explanation.
  3. Q3 the split branch is nearly unreachable. The trigger statistic |g| is nonnegative and strongly right-skewed, so the window mean sits above the (1−p)-quantile for p > 0.5. Please report the dtd_t trajectory and the split/merge/maintain event counts for one representative run: how can the split branch (Algorithm 1, lines 13–14) fire at all? If d decreases monotonically, what does "expand the search space and further improve the expressiveness" (§3.2) refer to in the runs you report? And please explain the bit-identical sensitivity results across p ∈ {0.7, 0.8, 0.9} and across d0 ∈ {4, 8, 16}.
  4. Q4 the central quantity d is never measured. Please provide accuracy-versus-d curves for (a) S-only and (b) S-and-Z at matched parameter count, and report the realized d in your main runs. Without these, the reader cannot tell how much of the gain over QZO comes from finer granularity versus simply doubling the number of continuous degrees of freedom. Also define the "non-grouped setting" numerically, and run the three ablations in Fig. 3 from a single common base configuration so they can be read as a decomposition.
8 comments

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper. Below, we provide point-by-point responses to all the weaknesses and questions raised. We have also revised the original manuscript accordingly, with all changes highlighted in blue.

W1 & Q1: The convergence theorems do not cover the algorithm as actually run.

We thank the reviewer for the careful analysis of the relationship between the theoretical conditions and experimental settings. We understand the reviewer's concerns regarding the grouping interval KK and the cumulative regrouping error ETE_T. We address these concerns from the following three aspects.

  • Theoretical Condition K=Ω(T)K=\Omega(T) and Experimental Settings.

    In our existing experiments, we use K=125K=125 for both T=5000T=5000 and T=8000T=8000, corresponding to ratios of 1/401/40 and 1/641/64, respectively. We understand that this may make the grouping interval KK appear relatively small compared with the total number of training steps TT. However, we would like to clarify that K=Ω(T)K=\Omega(T) in Theorems 4.7 and 4.8 does not require KK to be close to TT, but rather requires their ratio to be bounded below by a positive constant in the asymptotic sense. Therefore, our existing settings under finite training budgets are not inconsistent with this theoretical condition. As the training budget increases further, KK can be increased accordingly to maintain the required asymptotic relationship.

    To further address the reviewer's concern, we conducted additional experiments with a larger proportionality constant cc. Specifically, on OPT-1.3B, we set c=1/5c=1/5, resulting in K=1000K=1000 for a total of T=5000T=5000 training steps, and compare the results with those obtained using the original setting of K=125K=125. These experiments further evaluate the impact of different grouping adjustment frequencies on model performance.

    Table: Comparison of Different Grouping Intervals on OPT-1.3B.

    SST2CBWSC
    K=12585.8971.4357.69
    K=100087.7371.4358.65
  • Theoretical Analysis of the Cumulative Regrouping Error ETE_T.

    We thank the reviewer for the concern regarding the cumulative regrouping error. First, we would like to clarify that the 40–64 operations in our experiments correspond to grouping decisions rather than actual regrouping operations. Since Split and Maintain do not immediately change the dequantized weights, additional loss increments can only arise from Merge operations. With the initial number of groups set to d0=2d_0=2, as in our paper, even in the most frequent merging scenario, at most 20–32 Merge operations can occur among the 40–64 grouping decisions. More importantly, the actual operation statistics provided in our response to W3 show that the vast majority of decisions during training are Maintain, with the number of Merge operations typically below 5.

    From a theoretical perspective, Appendix D.4 already provides an explicit analysis of the cumulative regrouping error and its corresponding upper bound. Specifically, Eqs. (98)–(100) establish that, under the corresponding assumptions, the loss increment induced by each Merge operation has a uniform upper bound eme_m. Consequently, Eqs. (101)–(102) further yield:

    ET≤NTmergeem≤⌈TK⌉em.E_T \leq N_T^{\mathrm{merge}}e_m\leq \left\lceil\frac{T}{K}\right\rceil e_m.

    When K=cTK=cT, we directly obtain:

    ET≤⌈1c⌉em=O(1).E_T\leq\left\lceil\frac{1}{c}\right\rceil e_m=O(1).

    Therefore, under the condition K=cTK=cT, the cumulative regrouping error is guaranteed to remain bounded without requiring any additional assumption that the loss increment induced by each regrouping operation gradually decays during training. Moreover, the small number of Merge operations observed in practice further suggests that estimating the cumulative error solely based on the number of grouping checks may be overly conservative.

  • Further Analysis under a Fixed KK.

    In fact, our theoretical analysis is not limited to the case of K=Ω(T)K=\Omega(T). In Eq. (103) of Appendix D.4, we also discuss convergence under a fixed KK. This analysis shows that K=Ω(T)K=\Omega(T) is a sufficient condition, rather than a necessary condition, for maintaining the convergence rate. Even with a fixed KK, as long as the cumulative parameter displacements induced by Merge operations and their squared terms satisfy the boundedness conditions, ET=O(1)E_T=O(1) can still be guaranteed, thereby preserving the corresponding convergence rate.

W2 & Q2: The §3.1 variance comparison does not support its conclusion.

We thank the reviewer for the careful examination of our variance bound derivations. We would like to further clarify the purpose of our theoretical analysis.

  • Consistent Dimensional Settings in Eqs. (10) and (11).

    We would like to emphasize that our variance analysis aims to reveal the quantization errors introduced by discrete perturbations, rather than directly compare the optimization stability of CoZO and QuZO. To this end, Definitions 2.1 and 2.2 both consider a general (d)-dimensional parameter space θ∈Rd\theta\in\mathbb{R}^d. Therefore, the theoretical derivations of Eqs. (10) and (11) are based on the same dimensional setting, which we hope clarifies the reviewer's concern regarding dimensional inconsistency.

  • Constant Factor and Quantization Errors in the Variance Bounds.

    Regarding the constant factor of 4 raised by the reviewer, we would like to clarify that this factor primarily arises from inequality relaxations in the proof, whereas the additional quantization-error terms in Eq. (11) are the terms directly associated with discrete perturbations. Our analysis aims to reveal, from a theoretical perspective, the potential additional errors introduced by discrete perturbations, thereby providing theoretical motivation for continuous-space optimization, rather than establishing a strict ordering of the actual estimator variances based on the two upper bounds. We appreciate the reviewer for pointing out the potential ambiguity in our original wording. Accordingly, we have revised Remark 3.5 to further distinguish the effects of inequality relaxations from those of quantization errors, making the corresponding theoretical conclusions more precise and rigorous.

Reviewer_1HLvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies zeroth-order (ZO) fine-tuning of weight-quantized LLMs. It argues that existing approaches are limited either by the finite search space of discrete weights (QuZO, ZOQO) or by the low expressiveness of coarse per-channel/per-tensor scales (QZO). The proposed method, CoZO, freezes the integer codes and uses SPSA to optimize grouped scales and zero-points in continuous space. Scales and zero-points are updated alternately, and the number of groups is adjusted with a split/merge rule driven by statistics of the SPSA finite-difference term. The authors give variance bounds comparing SPSA with quantized RGE and an O(dmax⁡/T)O(\sqrt{d_{\max}/T}) convergence result for the alternating, adaptively grouped procedure. Experiments use 4-bit OmniQuant models (OPT-1.3B to 13B, Llama-2-7B, Qwen2-7B) on SuperGLUE-style tasks and SQuAD, where CoZO generally outperforms QZO and QuZO with similar memory to QZO.

Critical strengths

The method is simple and practical: it keeps the quantized integer codes fixed and only adds continuous dequantization parameters, so it stays compatible with a quantized representation while expanding the reachable weight space beyond scale-only tuning. The appendix argument that zero-points add a translation direction that scaling cannot provide (especially for zero-coded weights) is a clean justification. Across several models and tasks, CoZO consistently improves over QZO and QuZO at a memory cost comparable to QZO, which is a useful empirical data point for memory-constrained adaptation of quantized models.

Critical weaknesses
  • The comparison does not control for quantization granularity. CoZO tunes grouped scales and zero-points, while the main continuous baseline (QZO) tunes only per-channel scales of the same OmniQuant model. The gains may therefore come simply from having more continuous parameters rather than from the proposed method. A fair comparison requires baselines at matched granularity, e.g., a standard group-wise quantized model (W4-g128) with scale/zero-point tuning. The paper should also report the final number of groups and the resulting storage overhead per weight, which are currently missing, so it is unclear whether CoZO even ends up finer than per-channel (it starts from two groups and uses the same memory as QZO).

  • The evaluation relies on outdated models and tasks, and the gains are concentrated on the oldest model family. The main results use OPT and Llama-2 on the SuperGLUE-style protocol of MeZO (2023); Qwen2-7B appears only in the appendix on three tasks. Large margins appear mainly on OPT (e.g., +9.2 on SST-2 and +6.9 on RTE for OPT-6.7B). On Llama-2-7B and Qwen2-7B, the improvements over the best quantized baseline are mostly within 0.3–3 points on small evaluation sets, and CoZO loses to QuZO on Llama-2 SST-2. Recent models such as Llama-3 and Qwen3 are known to be more sensitive to low-bit quantization, so it is unclear whether the method generalizes to the settings practitioners care about now. Evaluation on these models, and on more current tasks such as reasoning or generation benchmarks, is needed to improve the paper.

Questions for authors
  • Under a matched quantization setting (e.g., the same W4-g128 OmniQuant base), how does CoZO compare with fixed-grouping scale/zero-point tuning, both at the standard group size and at the granularity CoZO ends up at? This would isolate the contribution of adaptive grouping from the effect of simply adding parameters.
4 comments

We sincerely thank the reviewer for the time and effort devoted to reviewing our paper. Below, we provide point-by-point responses to all the weaknesses and questions raised. We have also revised the original manuscript accordingly, with all changes highlighted in blue.

W1 & Q1: The comparison does not control for quantization granularity.

We thank the reviewer for the insightful comments on the experimental settings and grouping mechanism. We would like to clarify the following points and provide additional experimental evidence.

  • Ablation Studies and Motivation for Adaptive Grouping.

    We understand the reviewer's concern that the performance gains may primarily come from the increased number of trainable parameters rather than alternating optimization and adaptive grouping. To address this concern, following the suggestion of reviewer aFVw, we have further refined our ablation studies and updated the corresponding descriptions and Figure 3 to more clearly distinguish and evaluate the individual contributions of each component. The revised results are presented in Section 5.4.

    Furthermore, we would like to emphasize that adaptive grouping is designed not only to balance optimization stability and the expressiveness of the search space, but also to reduce the uncertainty and tuning costs associated with manually selecting the grouping granularity. To further validate the role of adaptive grouping, we conducted additional experiments comparing extreme fixed-grouping configurations with adaptive grouping. These results further demonstrate that CoZO's performance gains are not solely attributable to an increase in the number of trainable parameters.

    Table: Comparison of Extreme Fixed Grouping and CoZO on OPT-1.3B.

    SST2CBWSC
    d=185.5567.8652.88
    d=6482.8062.5050.00
    CoZO85.8971.4357.69
  • Additional Experiments with Matched Grouping Granularity and Parameter Counts.

    We would like to clarify that QZO and CoZO use the same initial quantized model to ensure a consistent quantization baseline. We agree with the reviewer's suggestion to further control for grouping granularity and the number of trainable parameters. To this end, we conducted additional fixed-grouping experiments using the same W4-g128 OmniQuant base model, including scale/zero-point tuning at the standard quantization granularity (without additional grouping) and fixed-grouping methods with the number of groups matched to CoZO's final grouping configuration.

    Table: Performance Comparison of Different Grouping Configurations on OPT-1.3B.

    Grouping ConfigurationSST-2CBWSC
    Without Additional Grouping (d=1)82.9164.2959.62
    Fixed Grouping Matched to CoZO's Final Granularity82.1162.5060.58
    CoZO83.0367.8660.58
  • Final Number of Groups and Storage Overhead.

    • Final Number of Groups.

      We would first like to clarify that our grouping mechanism operates within each channel, meaning that each channel is further divided into groups. Therefore, any grouping configuration with (d > 1) has a finer granularity than per-channel quantization. In our main experiments, we consistently set the initial number of groups to (d_0 = 2). We report the final number of groups for a subset of experiments. The results show that CoZO's final number of groups varies across downstream datasets and that its final grouping granularity is indeed finer than per-channel granularity.

      Table: Final Number of Groups Across Different Tasks on OPT-6.7B.

      SST2RTEWSC
      Final Number of Groups448
    • Storage Overhead.

      In practice, the number of trainable scale and zero-point parameters in CoZO remains relatively small compared with the number of model weights. We limit the number of trainable parameters by setting a maximum number of groups ((d_{\max})), preventing unrestricted parameter growth as the grouping granularity becomes finer. To provide a more intuitive assessment of the actual storage cost, we further report the additional storage overhead relative to the quantized model weights (bits/weight), as shown in the table below. Notably, although CoZO incurs higher parameter storage overhead than QZO, this additional cost remains small compared with the baseline storage of 4 bits per weight. This also helps explain why CoZO and QZO exhibit very similar peak GPU memory usage in practice.

      Table: Additional Parameter Storage Overhead on OPT-6.7B (bits/weight).

      SST2RTEWIC
      QZO0.0058590.0058590.005859
      CoZO0.0234380.0234380.046875

W2:The evaluation relies on outdated models and tasks, and the gains are concentrated on the oldest model family.

We thank the reviewer for the valuable suggestions on extending the evaluation to more advanced models and diverse tasks. We would like to clarify the motivation behind our original experimental settings and provide additional evidence of CoZO's generalization to newer models and different task types.

  • Cross-Model Performance under Unified Training Settings. Our original evaluation on OPT and Llama-2 follows the established experimental settings of existing zeroth-order fine-tuning methods, including QZO and QuZO, to ensure consistent and directly comparable evaluations. We also note that we use the same training hyperparameters across the OPT and Llama-2 model families, without architecture-specific tuning. Although the magnitude of improvement varies across model families, CoZO achieves competitive results and outperforms existing zeroth-order fine-tuning methods on most tasks, with slightly lower performance in certain cases, such as SST-2 on Llama-2 compared with QuZO. These results demonstrate the effectiveness of CoZO across different model families under unified training settings.

  • Additional Experiments on More Advanced Models and Diverse Tasks.

    We agree with the reviewer's suggestion to extend the evaluation to more advanced models and a broader range of downstream tasks. To this end, we conducted additional experiments on Qwen3-8B and introduced the GSM8K mathematical reasoning benchmark, reporting both Strict EM and Flexible EM metrics. These experiments further evaluate the applicability of CoZO to newer model architectures and different task types. The corresponding results have been added to Appendix C.7 of the revised manuscript.

    Table: Task Evaluation Results on Qwen3-8B

    SST2CBWICWSCGSM8K strict EMGSM8K flexible EM
    Zero-Shot51.4951.7952.0455.1769.9872.71
    QZO78.3269.6460.6657.6967.8571.27
    QuZO80.3980.3655.1761.5454.0657.32
    CoZO81.7783.9360.8263.4674.4575.73

Dear Reviewer,

We would like to kindly follow up on our rebuttal response. We would appreciate any further feedback or concerns you may have, and we would be happy to provide additional clarification if needed.

Thank you very much for your time and consideration!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.