acceptodds
Desk rejected at ICLR 2027

QAM: Quadratic-Accurate Checkpoint Merging via Sequential Consistency

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Saved checkpoints record states along a training trajectory, but generally do not determine the updates at states that would be visited under a different schedule. We study how accurately these checkpoints can reconstruct the endpoint of a sequential reference with prescribed update strengths. Under a common local transition model, two checkpoint-index moment conditions characterize all convex merges that agree with this reference through second order. We then prove an information limit that for nondegenerate profiles, no algorithm using only a fixed-length gradient-descent (GD) history with step size can achieve endpoint error uniformly over a fixed class of smooth, strongly convex losses. The lower bound follows from two losses with identical GD checkpoint histories but sequential reference endpoints separated by . Quadratic-Accurate Merging (QAM) achieves a matching uniform endpoint error bound. Its explicit coefficients also define the unique profile-dependent merge that exactly matches the sequential GD reference across all fixed quadratic objectives. Across two public Adam checkpoint trajectories (SmolLM3-3B and OpenEuroLLM-Prelude-9B), three windows and three profiles per model, and 15 tasks, QAM shows mixed results for short windows and broader advantages over Warmup-Stable and Merge (WSM) for longer windows. Matched-moment GSM8K diagnostics further show that local consistency alone does not fully determine downstream scores. These results characterize the reconstruction limits of saved histories, provide a coefficient rule that attains the optimal rate, and assess its practical utility.

Primary area:
foundation or frontier models, including LLMs
Keywords:
pretraining, model merging, learning rate schedule

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_uyReOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper focus on how to merge the checkpoints more effectively and improve the performance.

Critical strengths
  1. The paper provides a clear theoretical analysis of checkpoint merging through sequential consistency.
  2. The theoretical results are complete. The paper characterizes second-order-consistent merge checkpoint and shows that the proposed method attains this optimal rate as the lower bound match the upper bound.
  3. The proposed method is easy to use and does not require any detailed information, such as gradients, optimizer states, Hessian estimation, or additional training.
Critical weaknesses
  1. There is a mismatch between the theory and experiments. The theory assumes a common local transition and mainly analyzes GD, while the experiments use sparse Adam pretraining checkpoints.
  2. The empirical evaluation is limited to two completed pretraining trajectories and do not include new pretraining experiments. Maybe using some LLM pretraining experiments and collect some real checkpoint data will provide more evidence that this method is useful.
Questions for authors
  1. Can the authors directly training some LLM model instead of using old data and comparing the endpoint error of QAM, WSM, and other merged methods? I understand the computational resource may limited but I think this is needed. Maybe some small models is enough.
  2. Since distributions with the same first two moments can have substantially different downstream performance, what property of the full QAM coefficient distribution is expected to explain its empirical advantage beyond second-order consistency? You can provide some explanations.
3 comments

Thank you for your positive feedback on the theoretical analysis and practical simplicity of QAM, as well as your constructive suggestions. Below, we provide point-by-point responses to your questions and weaknesses you identified.


Question 1: Can the authors train small language models and directly compare the endpoint errors?

A1: Following your suggestion, we have added controlled experiments on three newly trained small Transformers with approximately 0.11M, 0.82M, and 4.79M parameters. The new endpoint study focuses on the paired QAM–WSM comparison to directly test the difference in error orders predicted by Eq. (6) and Theorem 2.

We collect new GD checkpoints and construct paired WSM and QAM merges. From the same starting state, we also train a separate model copy with learning rates hWjhW_j, recomputing every gradient along its own trajectory to produce the sequential reference endpoint ϕkW\phi_k^W. We then measure the parameter distance em(h)=∥∣θm−ϕkW∥∣2,m∈{WSM,QAM}e_m(h)=\||\theta_m-\phi_k^W\||_2,\quad m\in\{\mathrm{WSM},\mathrm{QAM}\} while varying h=2−rh=2^{-r}, r=6,…,15r=6,\ldots,15, with the objective, starting state, window length, and profile fixed. Across five seeds, three starting states, three window lengths, three profiles, and the three model scales, we evaluate 405 configurations.

QAM achieves smaller endpoint error in 3,904/4,050 comparisons, including every configuration at every tested h≤2−8h\le 2^{-8}. At the smallest step size, the median local error exponents are 1.990 for WSM and 2.979 for QAM, supporting the predicted quadratic and cubic scaling. All 146 exceptions occur at the two largest step sizes. Full experimental settings and results, including the complete slope ranges, are provided in Section 4.1 and Appendix B. These experiments provide direct validation of the theoretical predictions on fresh training runs.


Question 2: What properties of QAM's full coefficient distribution could explain downstream gains beyond second-order moment matching?

A2: QAM's additional structural property is exact reconstruction of the sequential GD endpoint across all fixed quadratic objectives. Its generating polynomial reproduces the complete sequential-update polynomial, and Theorem 5 shows that this requirement uniquely determines the profile-dependent linear coefficients. This provides a selection principle beyond second-order moment matching.

However, we have not established that a specific higher-order property of these coefficients causes the observed downstream gains. Understanding the reasons behind QAM's observed downstream gains remains an important direction for future work.


Weakness 1: The connection between theoretical validation and practical evaluation.

Response: We agree that the original experiments did not directly validate the endpoint-error predictions under the stated local-transition assumptions. Therefore, we have added controlled GD experiments to explicitly test these predictions, as described in A1. The original 3B/9B Adam experiments provide empirical evidence for the practical utility of the resulting coefficient rule. Extending the quantitative endpoint analysis to different optimizers (and for larger models) calls for finer local models that captures optimizer-state evolution and adaptive updates. We now identify this as a direction for future research in the revised conclusion.


Weakness 2: Limited practical evaluation and the need for additional training experiments.

Response: We agree that evaluation on additional independent pretraining histories would strengthen the evidence for QAM's practical generality. Within the resources and time available for this revision, we prioritized fresh small-model training and direct endpoint comparisons. We identify broader evaluation across independent pretraining histories and training settings as an important direction for further study.

Thank you for the detailed responses and additional experiments. The clarifications have addressed my concerns, and I will maintain my positive score.

AuthorsOpenReview ↗

Thank You Note to Reviewer uyRe

We again express our gratitude for all your comments, suggestions, and acknowledgment of our responses.

Reviewer_hPvcOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper is about merging checkpoints from a pretrained-model trajectory. The authors argue that WSM's tail-sum view only reweights the recorded updates. It does not reproduce a sequential reference in which each transition is recomputed at the new state. They show that WSM has a second-order interaction bias. The authors propose QAM: take independent Bernoulli variables with update strengths W, and use the distribution of their sum as checkpoint weights. This matches the first two index moments of the reference and gives local consistency. Experiments compare paired WSM/QAM merges on SmolLM3-3B and OpenEuroLLM-Prelude-9B, over three windows, three profiles, and 15 tasks. QAM is mixed at short windows and tends to do better at intermediate/long windows, especially on GSM8K. A same-moment MaxEnt control suggests that matching the first two moments does not determine downstream ranking.

Critical strengths
  1. Theorem 2 gives a necessary and sufficient condition for local second-order consistency, Theorem 4 establishes a checkpoint-only reconstruction limit, and QAM attains the matching upper bound. This separates what can be recovered from saved checkpoints and what cannot.

  2. QAM is computationally simple and practical: coefficients are obtained by a scalar convolution over W, requiring only saved checkpoints, no gradients, optimizer states, or additional training. The quadratic-exactness uniqueness result further gives a principled selection among second-order-consistent rules.

Critical weaknesses
  1. The experiments do not directly test the central theoretical claim. The theory is about closeness to the sequential reference endpoint, but the experiments compare downstream scores of WSM and QAM. Real Adam histories need not satisfy the common-transition assumption, and this endpoint is never computed.

  2. The MaxEnt control weakens the empirical case for QAM over other moment-matched weights. It does not affect the theoretical uniqueness result, but it means the experiments do not show a robust downstream advantage beyond matching the first two moments.

Questions for authors
  1. Could the authors add a small controlled experiment (synthetic quadratic/GD or a small model) that actually computes the sequential reference endpoint and reports the parameter distance to it for WSM and QAM as h varies?

  2. How should we read the same-moment MaxEnt controls, which sometimes score comparably or slightly higher than QAM?

3 comments

Thank you for your positive feedback on the theoretical results and practical simplicity of QAM, as well as your constructive suggestions. Below, we provide point-by-point responses to your questions and the weaknesses you identified.


Question 1: Can the authors add a small controlled experiment that computes the sequential reference endpoint and compares the parameter distances of WSM and QAM as the step size varies?

A1: Following your suggestion, we have added controlled experiments on three newly trained small Transformers. We collect new GD checkpoints and construct paired WSM and QAM merges. Meanwhile, we independently train a separate model copy with learning rates hWjhW_j, recomputing every gradient at its own current parameters to produce the sequential reference endpoint ϕkW\phi_k^W.

We measure the parameter distance em(h)=∥∣θm−ϕkW∥∣2,m∈{WSM,QAM}e_m(h)=\||\theta_m-\phi_k^W\||_2,\quad m\in\{\mathrm{WSM},\mathrm{QAM}\} while varying h=2−rh=2^{-r}, r=6,…,15r=6,\ldots,15, with the objective, starting state, window length, and profile fixed. Across five seeds, three starting states, three window lengths, three profiles, and the three model scales, we evaluate 405 configurations.

These experiments directly test both the endpoint distances and their scaling with hh. QAM achieves smaller endpoint error in 3,904/4,050 comparisons, including every configuration at every tested h≤2−8h\le 2^{-8}. To examine the predicted error orders, we compute the per-configuration local slope pm(h)=log⁡2(em(2h)/em(h))p_m(h)=\log_2(e_m(2h)/e_m(h)). At the smallest step size, the median slopes are 1.990 for WSM and 2.979 for QAM, supporting the predicted quadratic and cubic scaling. Details are updated in the paper. (Section 4.1 and Appendix B.)


Question 2: How should we interpret the same-moment MaxEnt controls, which sometimes perform comparably to or slightly better than QAM?

A2: The MaxEnt controls show that matching the first two checkpoint-index moments alone does not fully determine downstream performance (Tables 4 and 10). This suggests that other factors may also contribute to QAM’s observed downstream gains.


Weakness 1: The experiments do not directly test the central endpoint-reconstruction claim, and the Adam histories need not satisfy the common-transition assumption.

Response: We agree that the original downstream comparisons did not directly validate the endpoint-error predictions under the stated local-transition assumptions. Therefore, we have added the controlled GD experiments described in A1 to explicitly test these predictions. The 3B/9B Adam experiments provide empirical evidence for the practical utility of the resulting coefficient rule.

We also agree that the Adam histories need not satisfy our current dynamics-modeling. Indeed, extending the quantitative endpoint analysis to different optimizers calls for finer local models of training dynamics that capture optimizer-state evolution and adaptive updates. We now identify this as a direction for future research in the revised conclusion.


Weakness 2: The MaxEnt controls do not establish a robust downstream advantage of QAM over other moment-matched rules.

Response: We agree with this assessment and explicitly acknowledge it in Section 5. QAM's full coefficient rule is selected by a stronger structural requirement: exact reconstruction of the sequential GD endpoint across all fixed quadratic objectives. Theorem 5 shows that this requirement uniquely determines the profile-dependent linear coefficients.

This provides a principled justification for choosing QAM among second-order-consistent rules, but does not guarantee a downstream advantage over every moment-matched alternative. Understanding the reasons behind QAM's observed downstream gains remains an important direction for future work.

AuthorsOpenReview ↗

Looking Forward to Your Feedback

As the rebuttal stage is nearing its end, we would like to briefly follow up on our response and the revised manuscript. We would appreciate your thoughts on whether the response address your concerns, and would be happy to clarify any remaining points before the deadline. Thank you again for your time!

Thank you for the detailed responses and additional experiments. The clarifications have addressed my concerns, and I will retain my original score.

Reviewer_CLU5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Quadratic-Accurate Merging (QAM), a training-free checkpoint merging method that approximates sequential training trajectories through moment matching. The authors establish theoretical guarantees, including exact reconstruction for quadratic objectives and an information-theoretic lower bound. Experiments on SmolLM3-3B and OpenEuroLLM-Prelude-9B demonstrate improvements over Warmup-Stable and Merge (WSM) in several settings.

Critical strengths
  1. Solid theoretical formulation. The paper provides a mathematically well-structured analysis of checkpoint merging, including moment-matching conditions, quadratic exactness, and reconstruction error bounds.

  2. Practical efficiency. QAM requires neither additional training nor gradient or Hessian computations, making it straightforward to integrate into existing checkpoint-based workflows.

  3. Controlled experimental comparisons. The experiments compare QAM and WSM using identical checkpoints and update profiles, helping isolate the effects of different merging coefficients.

Critical weaknesses
  1. Unclear applicability and practical boundaries. The paper does not sufficiently clarify under what training settings and conditions QAM is expected to be effective. Given the variations in optimization algorithms, checkpoint intervals, and training dynamics, it remains unclear when QAM can consistently outperform existing methods and what its practical limitations are.

  2. Inconsistent empirical improvements. Although QAM performs well for longer checkpoint windows, it does not consistently outperform WSM, particularly for shorter windows. This raises concerns about the robustness and general applicability of the method.

  3. Limited experimental validation. The experiments involve only two pretrained models and lack direct comparisons with actual sequential training trajectories. Moreover, some improvements may be influenced by generation failures in the WSM baseline rather than the proposed merging principle itself.

Questions for authors
  1. What explains QAM's weaker performance in short checkpoint windows, and how should practitioners determine when QAM is preferable to WSM?

  2. Can the authors provide broader experiments across model scales and checkpoint configurations, while separating genuine performance gains from improvements caused by baseline generation failures?

3 comments
AuthorsOpenReview ↗

Comment Part I

Thank you for your careful assessment and constructive feedback. Below, we provide point-to-point responses to your questions and the weaknesses you identified.


Question 1: What explains the weaker short-window performance, and how to choose between QAM and WSM?

A1: This is an insightful question. We address it from several perspectives below, while also responding to Weaknesses 1 and 2.

First, our theory does not establish a relationship between merging-window length and downstream performance or predict that QAM must outperform WSM at a particular window length. Moreover, the “short”, “intermediate”, and “long” windows are experimental settings selected within each training history to compare the methods across different merging configurations. They are not universal categories, and the corresponding checkpoint counts and token spans are not identical across the two tested models.

Empirically, QAM's disadvantage relative to WSM in the tested short-window settings is modest overall, with mixed rankings across configurations. The median absolute capability-group difference is approximately 0.35 percentage points, with QAM scoring lower in 21/36 comparisons. At intermediate and long windows, QAM shows more consistent gains across the tested configurations, winning 28/36 and 29/36 comparisons, respectively (Section 4.2.3). This favorable pattern also persists after excluding Math (GSM8K), as discussed in A2. These results suggest that, within the tested histories, QAM may offer a broader range of favorable merging-window settings than WSM, with more consistent gains at intermediate and long windows while remaining competitive in short windows. However, scores also vary across model histories, tasks, and profiles, and our experiments do not isolate the reasons for the observed short-window rankings. Window length alone may therefore be insufficient to explain these outcomes. (This also responds to Weakness 2.) In practice, the window length, profile, and merging rule should be selected jointly to optimize downstream validation performance for the intended tasks.

In addition, comparisons with individual checkpoints provide further evidence of QAM's practical utility. QAM exceeds the mean downstream score of its constituent checkpoints in 105/108 capability-group comparisons and the latest checkpoint in 219/270 task–configuration comparisons (Section 4.2.2 and Table 2). On GSM8K, intermediate- and long-window QAM with a fixed linear profile also exceeds the best tested WSM configuration on both models (Table 3). These constituent-checkpoint comparisons support QAM's practical utility in its own right, while the GSM8K results provide additional evidence of gains over the best tested WSM baseline.

Finally, establishing more general selection criteria and understanding the reasons behind these gains require further research. Extending the quantitative endpoint analysis to different optimizers calls for finer local models that capture optimizer-state evolution and adaptive updates. Studies that independently vary checkpoint granularity, window span, and profiles would further clarify when QAM is preferable. Our work provides a starting point for further research on optimizer-specific modeling and the practical scope of checkpoint merging. (This also responds to Weakness 1.)


Question 2: Can the authors broaden the experiments and distinguish performance gains from improvements caused by WSM generation failures?

A2: Our existing results provide two complementary checks on whether QAM's gains depend on degraded WSM baselines.

(1) Comparison against the best tested WSM. The largest Prelude gains include recovery from substantial generation degradation, as documented in Appendix F. However, Table 3 compares fixed linear-profile QAM against the best WSM result across all nine tested window/profile configurations. QAM still obtains higher GSM8K scores at both intermediate and long windows on both models, with improvements of 1.55–2.12 percentage points, averaged over flexible and strict extraction. These improvements do not depend on selecting the degraded long-window WSM configuration as the baseline.

(2) Results beyond Math (GSM8K). After excluding GSM8K, QAM still scores higher than matched WSM in 22/30 intermediate-window comparisons and 23/30 long-window comparisons (Section 4.2.3). The favorable pattern is therefore not confined to the GSM8K results discussed in the generation diagnostics.

Together, these comparisons provide evidence of practical benefits beyond recovery from the severely degraded WSM configuration. We address the request for broader experimental coverage in our response to Weakness 3.

AuthorsOpenReview ↗

Comment Part II

Weakness 3: Limited model coverage and lack of direct comparisons with sequential training trajectories.

Response: To directly test the endpoint-reconstruction claim, we have added controlled experiments on three newly trained Transformers with approximately 0.11M, 0.82M, and 4.79M parameters. We collect fresh GD checkpoints and construct paired WSM and QAM merges. From the same starting state, we independently train a separate model copy with learning rates hWjhW_j, recomputing every gradient along its own trajectory to obtain the sequential reference endpoint. We then measure the parameter distances of both merges to this endpoint while varying h=2−rh=2^{-r}, r=6,…,15r=6,\ldots,15.

Across five seeds, three starting states, three window lengths, three profiles, and three model scales, we evaluate 405 configurations. QAM achieves smaller endpoint error in 3,904/4,050 comparisons, including every configuration at every tested h≤2−8h\le 2^{-8}. At the smallest step size, the median local error exponents are 1.990 for WSM and 2.979 for QAM, supporting the predicted quadratic and cubic scaling. All 146 exceptions occur at the two largest step sizes. Full settings and results are provided in Section 4.1 and Appendix B.

These experiments directly address the missing comparison with sequential reference trajectories. We agree that additional downstream experiments across independent pretraining histories and checkpoint configurations are needed to assess how broadly QAM's observed empirical advantages generalize. Understanding the reasons behind these gains is also an important direction for further research.

AuthorsOpenReview ↗

Looking Forward to Your Feedback

As the rebuttal stage is nearing its end, we would like to briefly follow up on our response and the revised manuscript. We would appreciate your thoughts on whether the response address your concerns, and would be happy to clarify any remaining points before the deadline. Thank you again for your time!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Comment

We thank all reviewers for their time, effort, and constructive feedback. We have provided individual replies to each reviewer and welcome any further questions or discussion.

The main addition to the revised manuscript is a set of controlled experiments directly evaluating the endpoint reconstruction errors of WSM and QAM, presented in Section 4.1 and Appendix B. We have also reorganized the conclusion to clarify the main findings and directions for future research. We have uploaded the updated manuscript, with newly added content highlighted in blue for ease of review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.