- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 1(−1)
- Soundness
- 3
Read review
Paper summary
This paper studies what stochastic information is necessary to characterize future learning outcomes. The authors introduce an observable-relative framework that determines when a summary of stochastic updates, such as their mean and covariance, is sufficient to predict a given future outcome. They theoretically show that matching any finite number of moments does not guarantee identical learning outcomes. Building on this framework, they propose treating higher-order properties of gradient distributions as control variables to improve future learning performance. Experiments on small learning systems, neural networks, and LLM training are provided to support their theoretical findings.
Critical strengths
The paper provides a well-structured theoretical framework for understanding how stochastic update laws affect future learning outcomes. I appreciate the connection between observable-relative equivalence and stochastic-law control. The authors also provide extensive theoretical analysis and experiments across different settings, including neural networks and Qwen RLVR, to support their findings.
Critical weaknesses
Limited practical applicability.
My main concern is the gap between observable-relative distinguishability and practical stochastic-law control. While the paper provides a theoretical characterization of how stochastic update laws affect future learning outcomes, I think more discussion is needed on how to efficiently select and control realizable moment-matched update laws, especially considering the computational cost of estimating backward observables. The control experiments are mainly limited to small, carefully constructed learning systems, leaving the practical applicability of the proposed framework unclear.
Questions for authors
How can the proposed framework guide practical stochastic-law control, given the computational cost of estimating backward observables and selecting realizable moment-matched update laws? Could the authors provide more discussion or evidence beyond the small, carefully constructed learning systems?
4 comments
Stochastic law control improves Qwen's held-out math accuracy at the same GPU budget (2/2)
What does the search cost?
We gave each shortlisting procedure up to 14.8 GPU-hours per seed to train with its candidates and choose one final winner on validation. Standard tuning spent about 12.6 GPU-hours training six runs and used the remaining 2.2 to generate and verify answers on the validation panel. Both shortlisting procedures fit within that cap once we counted training, their share of building and scoring the answer bank, and the validation used to choose a winner. They ran the same number of updates, and we tracked GPU time directly because the runs can generate answers of different lengths. Once each procedure had picked a best candidate, we evaluated it on the same held-out questions outside the search budget.
We repeated the search for each seed, and once a configuration is chosen, a single 200-update run takes about 2.1 GPU-hours. To see what the shorter search gave up, we also trained using all 16 candidates to completion and scored them on the same held-out questions. Training all 16 took 34.0 GPU-hours per seed. The shortlist included the best-scoring model on 16 of 20 seeds, and the model we had already picked on validation finished only 0.55 points behind the best observed test score on average, with a one-sided upper bound of 0.90 points.
Thank you again for your thoughtful review. We welcome any further questions and respectfully ask you to positively reconsider your rating in light of this evidence about the framework's practical value.
Dear Reviewer V7Q6,
Thank you again for your review. With the discussion period ending on November 18th, we wanted to follow up once more. Your main concern was about the practical benefit of the framework beyond small constructed systems, and we ran new experiments in light of your review to investigate the applicability of stochastic-law control and its computational cost.
In brief, our rebuttal reports a Qwen2.5-1.5B RLVR experiment (20 paired seeds, equal GPU budget, with a search-matched control) in which choosing the update law improved held-out GSM8K accuracy by 3.1 points over tuned standard training.
Does this resolve your concerns? We are glad to clarify anything before the discussion closes. Thanks again for your thoughtful review. Your recommendations helped enrich the work, and we hope you will reconsider your score in light of the new findings.
Regards,
The authors
Thank you for your detailed response and for conducting additional experiments to address my concerns. I appreciate the authors' efforts and the new evidence provided.
As noted in my original review, this topic is outside my primary area of expertise, and I have already communicated my limited confidence to the AC. While I acknowledge the additional experimental evidence, I do not feel sufficiently convinced to change my overall assessment at this stage. I will therefore maintain my original rating and defer to the AC and other reviewers with more relevant expertise in evaluating the paper's contributions.
Thank you again for your time and efforts.
Stochastic law control improves Qwen's held-out math accuracy at the same GPU budget (1/2)
Thank you for recognizing the value of the theoretical framework and the breadth of the experiments. To address your concern about practical applicability, we used the framework to guide the choice of update laws which in turn improved Qwen2.5-1.5B's final math accuracy within the same GPU budget compared to two baseline procedures.
Specifically, choosing how to weight the four gradient estimates at each update raised held-out GSM8K accuracy from 58.3 to 61.4, a gain of 3.1 percentage points against the standard baseline.
The three procedures we tested are:
In both 16-candidate searches (in procedures 2 and 3), we used the same screen at update 64, then trained the four finalists to update 200, and chose one using the same validation panel. Within each of 20 independent seeds, the methods started from the same model initialization, followed the same training prompt order, and answered the same held-out questions. We used the differences within each seed to get 95 confidence intervals of [+2.0,+4.2] points against standard tuning and [+0.9,+2.9] against averaging with the same shortlisting procedure.
Where do the 16 weightings come from?
For each prompt at a fixed model state, we combine four independent draws from the same gradient distribution. We choose weights ci with ∑i=14ci=1 and ∑i=14ci2=1, so every candidate has the same expected gradient and the same variance in every direction as a single source gradient. The higher moments can still vary. In any direction with nonzero variance, ∑i=14ci3 scales skewness and ∑i=14ci4 scales excess kurtosis. Two of our weightings, for instance, share a skewness multiplier of 0.625 and have excess kurtosis multipliers of 0.438 and 0.578. We use the framework to choose the 16 weightings before training, and we spread them across both multipliers so the screen can compare a range of noise shapes, which enables real control.
How to shortlist four finalists?
Section 4 shows that choosing the law that gives the lowest immediate loss can lead to worse terminal performance. In Qwen, our goal is to pick the run that will solve the most held-out math problems after 200 updates. At update 64, we look for an early sign of that ability. A model can start assigning more probability to correct solutions before it generates them reliably.
For each of the 48 development questions, we save two verifier-approved answers and two plausible wrong ones. Every candidate's checkpoint at step 64 scores the same answers, and for each question we subtract the wrong answers' mean log probability per token from that of the correct answers. We rank checkpoints by the average gap across questions. Because every checkpoint scores the same saved answers, the screen takes only a few batched forward passes.
To see whether those scores helped us decide which candidates to shortlist, we compared them with a simpler rule that ranked candidates by the fraction of training answers that passed the verifier over the first 64 updates. Under the same search budget, the saved-answer scores helped us choose candidates that achieved 0.7 percentage points higher held-out accuracy at update 200 (95 CI [+0.1,+1.3]). This is the practical payoff of Section 4’s nonmyopic principle. The final task told us what to look for early, and that signal helped us spend the remaining training budget on runs that went on to solve more problems.