- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how selecting data subsets and optimizing their weights using a Gaussian gradient sketch affects subsequent certification of the reconstruction residual. For exact least-squares fitting within a fixed affine family, it derives a distributional characterization of the selected residual's sketched-to-exact squared-norm ratio. This yields a distinction between the dimensions needed for accurate estimation and those needed for valid uncorrected confidence bounds: relative distortion can vanish while coverage approaches zero. The paper also presents a model-specific correction and an independent Gaussian audit after committing the selected residual, with extensions for simultaneous claims, numerical error, and repeated decisions. Experiments on Qwen2.5-1.5B/0.5B using GSM8K and MBPP subsets examine residual estimation, reweighting, conservative uniform certificates, and downstream performance. These experiments provide descriptive evidence of sketch optimism, while the independent-audit results are analytical power calculations based on measured residuals.
Critical strengths
The paper clearly distinguishes estimation accuracy from post-selection coverage. The exact affine-family analysis and its two-threshold consequence provide a useful explanation of why an accurate sketch estimate can still yield an unreliable confidence bound. The assumptions and limitations of the different guarantees are stated explicitly.
Critical weaknesses
-
The main certification mechanisms lack direct empirical validation. The exact correction assumes least-squares fitting within a fixed affine family, whereas the real-model experiments use sketch-selected supports and constrained, regularized, normalized weights. The independent-audit protocol is not run in these experiments; Table 3 reports analytical power calculations from known residual norms. The observed coverage of conservative uniform bounds therefore does not directly validate either principal calibration route. A comparison of realized coverage, bound tightness, and computational cost across the proposed certification approaches is missing.
-
The practical usefulness of the certificates remains insufficiently demonstrated. Even with a tolerance equal to the full-update norm, only 6 of 27 primary uniform bounds meet the tolerance criterion, coming from behavior-only and random selection; the joint variants do not. Functional diagnostics remain at the numerical precision floor, leaving the empirical fidelity evidence confined to parameter-update space. The study uses only 128 tasks, one rank-8 adapter location, and 64 training steps, with no statistically established downstream advantage. These results leave limited evidence that the certificates support useful decisions in realistic training settings.
-
Model coverage is limited to small, older models. All real-model experiments use Qwen2.5-0.5B and Qwen2.5-1.5B, from a family released in September 2024. Despite the availability of newer open models such as Qwen3, released in April 2025, the paper provides no evaluation on a newer model family or a substantially larger model. This limits evidence that the observed certificate usefulness and computational trade-offs extend to contemporary language-model training.
Questions for authors
- Can you directly evaluate the independent-audit protocol on the constrained selectors, reporting repeated-audit coverage, certification rates, and measured overhead relative to uniform certification and exact residual computation?
- What evidence connects the chosen residual tolerance to meaningful training fidelity, beyond the one-step parameter identity and functional measurements at the numerical precision floor?
- Why is the evaluation restricted to Qwen2.5-0.5B/1.5B, and do the empirical findings hold on newer and larger models?
3 comments
Functional decisions and newer-model evidence (Weaknesses 2-3/Q2-Q3)
Weakness 2 and Q2: What connects the tolerance to meaningful fidelity beyond the numerical floor?
We acknowledge this concern and added actual FP32 full-pool/subset updates, not only the parameter identity. For each model/adapter variant, a common step is selected on all 837 validation tasks using fixed KL/NLL and resolution criteria, then frozen before testing. Repeated identical-update forwards measure the numerical floor. Every cell evaluates full-vocabulary, teacher-forced distributions on all 1,319 GSM8K and 500 MBPP test tasks.
| Model | Held-out cells above floor | Largest mean KL (budget 0.01) | Residual/logit Spearman range |
|---|---|---|---|
| Qwen2.5-0.5B | 225/225 | 0.00291 | 0.817-1.000 |
| Qwen2.5-1.5B | 225/225 | 0.00232 | 0.767-1.000 |
| Qwen3-8B | 225/225 | 0.000521 | 0.717-0.983 |
The repeated-update controls are below the registered 1e-8 logit floor, and all 675 cells exceed the resulting 1e-7 threshold. The held-out KL is KL(full update || subset update), not the validation pilot's KL(base || full update). All 75 nine-method Spearman comparisons exceed the registered 0.65 descriptive criterion. These configuration comparisons share tasks and gradients; KL is a benchmark-balanced mean, not a per-task maximum.
A validation-only policy then chooses tau=0.1 ||g_V|| using mean KL<=0.01 and relative logit RMS<=0.1, where relative error divides subset-versus-full discrepancy by full-update-versus-base discrepancy. It accepts 29/135, 36/135, and 92/135 main-adapter cells, respectively, with zero observed violations among accepted cells. Alpha=0.2 does not satisfy the validation rule. This connects the residual to resolvable output changes and selective decisions. Its scope is measured local additive-update fidelity, not semantic equivalence or preservation throughout AdamW training. Section 5.3 and Table 3 (page 8), Appendix J.3 (page 21), and Appendix K.4 (pages 25-26) give the definitions and results.
Weakness 3 and Q3: Do the findings extend to newer and larger models?
We appreciate the request to go beyond the smaller exact-gradient reference models. Qwen3-8B now participates in the same calibration, functional, tolerance, and cost protocols, including rank-8, rank-16, and four-location adapters. The training workload uses 7,097 examples and 444 steps, with longer horizons reaching 1,331 steps.
For completeness, equal-GSM8K/MBPP macro rewards are:
| Model | Base | Joint | Full-pool matched |
|---|---|---|---|
| Qwen2.5-0.5B | 20.04% | 20.04% | 20.41% |
| Qwen2.5-1.5B | 53.83% | 48.34% | 48.45% |
| Qwen3-8B | 72.10% | 61.06% | 61.40% |
These comparisons use fixed training seed 4401; task-bootstrap intervals describe that seed, and exact seed sign-flip p-values are 1. Downstream superiority is not established. Retention is benchmark-dependent: Qwen3 main-run HumanEval base/joint scores are 86.59/47.56%, while MMLU subject-macro scores are 73.03/73.34%. Section 5.5 and Table 5 (pages 9-10), with Appendix K.6 (pages 27-29), report these separate outcomes.
The newer model supports transfer of calibration and measured local fidelity, not a universal learning benefit.
Thank you for your time and we hope these additions address all three questions and would be grateful if you would reconsider your assessment.
Thank you for the author's response. In my view, this work still requires further refinement and optimization to meet the acceptance criteria for ICLR; therefore, I maintain my rating.
Direct calibration, certification rates, and overhead (Weakness 1/Q1)
Thank you for recognizing the exact affine analysis and its explanation of why accurate estimates can yield unreliable certificates. We agree that the two calibration routes needed experiments matching their respective assumptions.
Weakness 1 and Q1: Can the independent audit be evaluated directly, including coverage, decision rates, tightness, and cost?
Yes. We now commit 45 constrained deployed residuals per model, from nine selectors and five selection projections, before drawing 2,000 fresh shared-projection trials at each of 64, 128, and 256 audit rows. At 128 rows:
Both columns use simultaneous endpoints with per-residual risk 0.05/45. Coverage counts the joint event per projection; certification averages the 45 residuals' trial rates at tau=0.5 ||g_V||. This is a sensitivity tolerance, separate from the validation-selected policy below. Marginal endpoints have median bound/exact-norm ratios 1.112-1.114. Section 5.3, Table 2, and Figure 2 (page 7), with Appendix K.3 (page 25), report the repeated experiment and its uncertainty.
To address the fixed-affine mismatch separately, we also ran unconstrained, unregularized fits on independently fixed uniform/high-loss gradient families. Corrected coverage is 93.8-97.2% over the 36 original-grid settings, with 1,000 trials each. Under the i.i.d. Gaussian assumption, this is an implementation and calibration check of the exact pivot, not evidence for Rademacher or structured sketches. At k=64, m=512, naive coverage is 1.0-1.2% versus 94.6-94.8% after correction across the three models. Section 5.2 (pages 5-7) and Table 1 (page 6) provide the corresponding implementation evidence for the other principal calibration route.
Uniform and exact cost comparisons. We acknowledge your point that usable decisions must be evaluated alongside cost. The six-of-27 small-pool observation is retained in Appendix L (page 30); the new comparison uses the full training pool. On that pool, full-span uniform certification uses 13,824 rows and accepts 5/9, 5/9, and 7/9 residuals at the same half-update tolerance on selection seed 4501. Uniform projection preparation takes approximately 105, 161, and 403 ms, respectively. These cover a larger residual class than a committed audit.
For 1.5B, sealed/exact sparse queries take 0.310/0.539 ms after preparation. Sealed projection and target preparation cost 1.427 ms plus 0.054 ms map generation, versus 0.452 ms exact target preparation. With the residual already resident, exact is cheaper: 0.026 ms versus 0.090 ms fresh generation/application. Section 5.4 and Table 4 (pages 8-9), with Appendix J.4 (page 21), separate these measured access regimes rather than imply universal acceleration. Our second comment answers Q2 and Q3 on functional usefulness and model breadth.