acceptodds
Under review as a conference paper at ICLR 2027

Reference Accuracy and Predictive Value in Low-Precision Training

Abstract

Direct low-precision write-back can erase nonzero optimizer proposals. A high-precision reference can evaluate that event on a projected state, but aggregate agreement need not imply improvement over simpler controls. In a controlled grid, 55/72 cells have measured and predicted post-initialization crossings spanning ; 52/55 lie within 15%, while four cells disagree on crossing category. A reused-source NeoX audit passes temporal skill against a constant, yet an outcome-blind secondary comparison favors a historical template (RMSE 0.00360) over the corrected source (0.00438). A prospective Pythia-410M checkpoint-restart stage evaluates 15 holdout targets across three ages. Its arithmetic-margin selector meets study-specific absolute error/coverage requirements on every target but has higher error than always predicting unchanged on the same accepted events. Every unit-gain source correction also loses to its template. Weak residual alignment limits even a separate hindsight-optimal scalar gain to less than 3.4% RMSE improvement per observed comparison, below the fixed 10% requirement; this is a post-outcome geometric result. The complete original protocol already rejects the forecast, and finalization includes a disclosed analysis-binding repair. Matched rounding interventions separately show large fixed-run policy contrasts. These empirical results characterize this reference construction and its controls, rather than a general failure of reference forecasting.

Primary area:
optimization
Keywords:
low-precision training, numerical precision, reference-based prediction, selective classification, empirical evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_PCx8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper is written very badly and for me as a human it is close to impossible to understand even what it is about. I spent some time on it but unfortunately it was too frustrating for me to actually get to the point were I understand the bare minimum. It also seems very LLM written iwht paragraphs like "what the controls establish".

I am not willed to spend more time on this.

Critical strengths

See above

Critical weaknesses

See above

Questions for authors

What is this paper about generally? Could you provide a human written abstract with a problem statement?

2 comments
AuthorsOpenReview ↗

Response to PCx8: rewritten abstract and presentation

Suppose a model stores a weight to two decimal places. If a step proposes moving the weight from 1.00 to 1.003, rounding to the nearest hundredth leaves it at 1.00. We tested whether predicting how often this happens helps choose a rounding rule.

We used AI in drafting and editing the revised abstract, as disclosed in the paper.

In low-precision training, a weight can round back to its previous value after a small update. We test whether predicting how often this happens helps choose a rounding rule. In Pythia-410M continuations from a common checkpoint, a higher-precision pilot accurately predicted the fraction of matrix weights left unchanged at each of the first 64 updates. Across three schedules with optimizer state retained or reset, nearest rounding gave lower validation loss at 256 updates than stochastic rounding or retaining 32-bit weights. The same ordering held at 512 updates with retained state. Accurately forecasting the unchanged fraction did not identify a benefit from switching policies in these runs.

Dear authors, thank you for the rebuttal. It seems that you radically rewrote the abstract, paper and also changed the title. I am struggling to locate the differences.

Could you please summarize all the changes somewhere?

Reviewer_wd91OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper audits whether a high-precision reference trajectory, projected onto a low-precision grid, provides predictive value about update invisibility (frozen coordinates) in a separate low-precision run. It distinguishes exact per-coordinate rounding events, aggregate agreement of the projected proxy, and incremental skill over simple controls. The proxy closely matches onset timing in controlled toy settings, but in a prospective Pythia-410M checkpoint-restart study a margin-based selector that satisfies pre-specified absolute error/coverage gates is worse than always predicting “unchanged”, and a unit-gain source correction is worse than a historical temporal template. The paper also reports matched RNE-vs-SR write-back interventions and an appendix allocation study.

Critical strengths

S1. The paper clearly separates three claims: exactness of the rounding event, aggregate agreement of the reference proxy, and incremental predictive value over simple controls. It shows that the first two do not imply the third.

S2. The evaluation is rigorous and transparent. It uses pre-fixed gates, matched baselines (a constant label on the same accepted events and a historical template), and full reporting of negative results and post-outcome repairs.

S3. The paper diagnoses its negative results rather than merely reporting them. The excess-risk identity and the hindsight scalar-gain bound attribute the failure to weak alignment between the source correction and the target residuals.

Critical weaknesses

W1. The practical motivation is unclear. Obtaining the reference requires a full high-precision run, while remedies such as SR, ECO, or master weights need no forecast at all. The paper should specify which training decision this forecast would actually change.

W2. The negative findings are informative but largely specific to the evaluated reference construction, and it remains unclear whether this construction represents a sufficiently strong or practically relevant forecasting approach. The paper also offers limited guidance on how to design a reference predictor with meaningful incremental value.

W3. The Pythia experiments involve only three shared fresh-source trajectories across 15 holdouts, limiting the independence of the evidence. Moreover, optimizer restarts may affect source–target alignment, but the paper does not provide an ablation against uninterrupted training to assess this effect.

W4. The presentation obscures the main claim. Heavy protocol jargon, many loosely related studies (SR, native-hybrid, INT3, allocation), and reliance on a 39-page appendix make the contribution hard to assess. Post-outcome repairs and the inconclusive Study 1 further weaken the paper’s preregistration narrative.

Questions for authors

Please see details in Weakness section.

1 comment
AuthorsOpenReview ↗

Response to wd91: controls, optimizer state and longer runs

W1. Choice of training policy. We tested whether a high predicted fraction of unchanged weights justifies switching to stochastic rounding or retaining FP32 weights. Both policies can be applied directly. Nearest rounding gave lower 256-update validation loss than either alternative on all three Pythia-410M schedules, with either preserved or reset optimizer state (Table 1). The original references ran for 512 updates from released checkpoints, and the new ones for 64 updates after shared adaptation. On one exploratory schedule, using a pilot to allocate a 10% master-weight budget gave slightly higher loss and took more time than random allocation (Table 6).

W2. Choice of reference predictor. A target-specific reference did improve prediction in our separate NeoX experiment. Across three targets, predicting the fraction of unchanged weights from historical low-precision curves gave mean RMSE 0.00360. Adding a correction from one reusable reference raised the error to 0.00438. Using a reference matched to each target lowered it to 0.00283 and required a separate reference run per target (original Section 4.1, revised Section 5).

For the original Pythia correction, fitting its scale to each observed target reduced raw RMSE relative to history by at most 3.38%, below the specified 10% improvement (Appendix C). A stronger reusable predictor could estimate target-specific deviations from history using information available before training. Testing that forecast on fresh schedules and comparing its policy choice and cost with short direct trials would address its practical value.

W3. Shared references and optimizer restarts. The original 15 holdouts shared three fresh references, one per checkpoint age. The added study uses three source–target schedule pairs from one adapted checkpoint. Each pair uses the same projected starting weights, batches and learning rates with Adam's moments and counter either preserved or reset. Reset changed the nearest-minus-FP32 loss difference by −0.000560, +0.000034 and −0.001308 nats per token across the three pairs (Table 3).

Reset increased absolute forecast error in every pair. Relative forecast skill increased because the temporal standard deviation of the target fraction grew roughly thirteenfold (Section 4.1). Averaging the other two low-precision trajectories gave lower error than the reference in every condition. This retrospective comparison includes runs completed later (Section 4.2). Preserving state through weight projection differs from uninterrupted pretraining, so the effect of the original restart remains unresolved.

W4. Presentation and analysis timing. The abstract and introduction present the training decision, and Table 1 gives the matched policy results. We specified the policy and reset contrasts before the 256-update runs. The 512-update extension followed those results and reused their validation set. In all three preserved-state blocks, nearest rounding still gave lower loss than FP32 and both stochastic-rounding streams (Appendix F). Appendix C describes the post-outcome correction to the links between holdouts and development runs and reports the original and amended negative decisions. Study 1 was inconclusive under its registered criteria.

Reviewer_1U4BOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper evaluates whether high-precision training traces can predict when weight updates are lost to rounding during low-precision training. Experiments examine the accuracy and added predictive value of these reference-based estimates, as well as the effects of rounding policies on training performance.

Critical strengths
  • The paper empirically investigates the effects of quantization on training.
Critical weaknesses
  • The motivation and practical implications of the research question are unclear. What practical benefit would follow from predicting low-precision training behavior using a high-precision pilot run while focusing exclusively on write-back rounding? Once the intended benefit is clarified, the paper should directly evaluate whether it can be achieved and, where appropriate, explore methodological improvements toward that goal.
  • The connection between the experimental settings and practical benefits is unclear. Before presenting each experiment’s results, the paper should explain what conclusions different outcomes would support and why those conclusions matter in practice.
Questions for authors
  • If a high-precision training run is already required to obtain the reference updates, why not train in high precision and quantize the resulting model? Under what circumstances would the proposed reference-based approach be preferable?
  • Section 3 compares update-invisibility predictions derived from high-precision training traces with observations from low-precision training. How could agreement between these quantities inform a practical training decision?
  • What practical decisions or benefits are informed by the experiments in Sections 4 and 5?
1 comment
AuthorsOpenReview ↗

Response to 1U4B: training decision and policy comparisons

Training in high precision and quantizing afterward. We did not compare end-to-end training cost or final quality against training in higher precision and quantizing for deployment. The proposed use of the pilot is to choose how to store weights during later training runs. Our reference runs used 512 updates from released checkpoints in the original study and 64 updates after a shared adaptation in the revision. Reuse could spread the cost over several training jobs (Sections 1–2).

Using the forecast to choose a rounding policy. Section 3 of the original submission tested whether the pilot could predict when many weights would round back to their previous values. The matched Pythia-410M continuations test whether switching policies improves validation loss when the forecast is accurate. Policies share starting weights and minibatches. After 256 updates, nearest rounding gave lower loss than FP32 updates under all three schedules with either retained or reset optimizer state. Stochastic rounding had loss 0.02524–0.02868 nats per token above nearest (revised Table 1). In a retrospective comparison, nearest rounding also had the lowest validation loss at update 64 in all six cases (revised Section 3).

With retained state, the ordering persisted at 512 updates under both stochastic-rounding streams. We specified the extension after seeing the 256-update results and reused their validation set (Appendix F).

Decisions addressed in the original Sections 4 and 5. Section 4 tested whether a reference adds predictive information. In Pythia, always predicting “unchanged” made fewer errors on the same accepted events. Using historical low-precision curves alone gave lower forecast error than adding the reference correction. Section 5 tested the training effect of changing the rounding rule. Stochastic rounding gave lower validation loss than nearest in the GPT-2-family runs from scratch (revised Section 5 and Appendix C).

We also tested where to allocate a limited amount of higher-precision storage. On one exploratory schedule, the pilot selected the 10% of weight tiles with the largest estimated rounding damage. This gave validation loss 2.80131 in 272 seconds, including the pilot. Randomly choosing the same number gave 2.80100 in 205 seconds. Both choices reduced loss from 2.80954 with nearest rounding alone (Appendix D, Table 6).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revision and new matched experiments

We tested whether the predicted fraction of weights left unchanged by rounding justifies switching to stochastic rounding or retaining FP32 weights. In Pythia-410M continuations, the higher-precision pilots predicted the target fraction with root-mean-square error below 0.00044 over the first 64 updates. The three policies started from the same rounded checkpoint and used the same minibatches within each training schedule (Sections 2–4).

After 256 updates, nearest rounding gave the lowest validation loss on all three schedules with either retained or reset optimizer state. Every FP32 control improved from initialization. Nearest rounding had loss 0.00486–0.00617 nats per token below FP32, and stochastic rounding had loss 0.02524–0.02868 above nearest (Table 1). At 512 updates with retained state, both stochastic-rounding streams had loss 0.03562–0.03823 above nearest. This extension used the same validation set and was specified after the 256-update results (Appendix F). In earlier GPT-2-family experiments from scratch, stochastic rounding gave lower loss than nearest (Appendix C).

In the original Pythia study, historical curves alone had lower forecast error than those corrected by the high-precision reference (Section 5). In a retrospective comparison of the new continuations, nearest rounding also gave the lowest validation loss at the first measurement, after 64 updates, in all six conditions (Section 3). In a separate exploratory experiment on one schedule, a high-precision pilot selected 10% of weight tiles to retain in higher precision. Training with the selected tiles gave loss 2.80131 in 272 seconds including acquisition. Random selection at the same budget gave 2.80100 in 205 seconds (Table 6).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.