- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper focus on how to merge the checkpoints more effectively and improve the performance.
Critical strengths
- The paper provides a clear theoretical analysis of checkpoint merging through sequential consistency.
- The theoretical results are complete. The paper characterizes second-order-consistent merge checkpoint and shows that the proposed method attains this optimal rate as the lower bound match the upper bound.
- The proposed method is easy to use and does not require any detailed information, such as gradients, optimizer states, Hessian estimation, or additional training.
Critical weaknesses
- There is a mismatch between the theory and experiments. The theory assumes a common local transition and mainly analyzes GD, while the experiments use sparse Adam pretraining checkpoints.
- The empirical evaluation is limited to two completed pretraining trajectories and do not include new pretraining experiments. Maybe using some LLM pretraining experiments and collect some real checkpoint data will provide more evidence that this method is useful.
Questions for authors
- Can the authors directly training some LLM model instead of using old data and comparing the endpoint error of QAM, WSM, and other merged methods? I understand the computational resource may limited but I think this is needed. Maybe some small models is enough.
- Since distributions with the same first two moments can have substantially different downstream performance, what property of the full QAM coefficient distribution is expected to explain its empirical advantage beyond second-order consistency? You can provide some explanations.
3 comments
Thank you for the detailed responses and additional experiments. The clarifications have addressed my concerns, and I will maintain my positive score.
Thank You Note to Reviewer uyRe
We again express our gratitude for all your comments, suggestions, and acknowledgment of our responses.
Thank you for your positive feedback on the theoretical analysis and practical simplicity of QAM, as well as your constructive suggestions. Below, we provide point-by-point responses to your questions and weaknesses you identified.
Question 1: Can the authors train small language models and directly compare the endpoint errors?
A1: Following your suggestion, we have added controlled experiments on three newly trained small Transformers with approximately 0.11M, 0.82M, and 4.79M parameters. The new endpoint study focuses on the paired QAM–WSM comparison to directly test the difference in error orders predicted by Eq. (6) and Theorem 2.
We collect new GD checkpoints and construct paired WSM and QAM merges. From the same starting state, we also train a separate model copy with learning rates hWj, recomputing every gradient along its own trajectory to produce the sequential reference endpoint ϕkW. We then measure the parameter distance em(h)=∥∣θm−ϕkW∥∣2,m∈{WSM,QAM} while varying h=2−r, r=6,…,15, with the objective, starting state, window length, and profile fixed. Across five seeds, three starting states, three window lengths, three profiles, and the three model scales, we evaluate 405 configurations.
QAM achieves smaller endpoint error in 3,904/4,050 comparisons, including every configuration at every tested h≤2−8. At the smallest step size, the median local error exponents are 1.990 for WSM and 2.979 for QAM, supporting the predicted quadratic and cubic scaling. All 146 exceptions occur at the two largest step sizes. Full experimental settings and results, including the complete slope ranges, are provided in Section 4.1 and Appendix B. These experiments provide direct validation of the theoretical predictions on fresh training runs.
Question 2: What properties of QAM's full coefficient distribution could explain downstream gains beyond second-order moment matching?
A2: QAM's additional structural property is exact reconstruction of the sequential GD endpoint across all fixed quadratic objectives. Its generating polynomial reproduces the complete sequential-update polynomial, and Theorem 5 shows that this requirement uniquely determines the profile-dependent linear coefficients. This provides a selection principle beyond second-order moment matching.
However, we have not established that a specific higher-order property of these coefficients causes the observed downstream gains. Understanding the reasons behind QAM's observed downstream gains remains an important direction for future work.
Weakness 1: The connection between theoretical validation and practical evaluation.
Response: We agree that the original experiments did not directly validate the endpoint-error predictions under the stated local-transition assumptions. Therefore, we have added controlled GD experiments to explicitly test these predictions, as described in A1. The original 3B/9B Adam experiments provide empirical evidence for the practical utility of the resulting coefficient rule. Extending the quantitative endpoint analysis to different optimizers (and for larger models) calls for finer local models that captures optimizer-state evolution and adaptive updates. We now identify this as a direction for future research in the revised conclusion.
Weakness 2: Limited practical evaluation and the need for additional training experiments.
Response: We agree that evaluation on additional independent pretraining histories would strengthen the evidence for QAM's practical generality. Within the resources and time available for this revision, we prioritized fresh small-model training and direct endpoint comparisons. We identify broader evaluation across independent pretraining histories and training settings as an important direction for further study.