acceptodds
Under review as a conference paper at ICLR 2027

What Stochastic Information Matters for Learning? Observable-Relative Equivalence and Control

Abstract

Two stochastic learning rules can have the same conditional update mean and full covariance at every common learner state yet lead to different finite-horizon outcomes. We establish that the stochastic information needed to describe learning depends on the future quantity being asked about. We characterize when a summary of today’s update is sufficient to predict a future learning outcome. For every finite order , we construct SGD systems on the same smooth, strongly convex objective whose temporal polynomial statistics agree through order , while their probabilities of the same finite-horizon event differ by arbitrarily close to one. No fixed finite moment hierarchy is universally sufficient. When summary-equivalent laws are realizable choices, unresolved stochastic structure becomes a control variable that can improve learning. At the same learner state, the law minimizing immediate error differs from the law minimizing error after eight further SGD steps; choosing for future loss reduces terminal risk by 8.35% relative to state-aware immediate-loss control in our exact system. In three neural settings, Gaussianizing updates while preserving their conditional mean and covariance moves boundary-crossing risk toward the Gaussian prediction in 59 of 60 systems, closing 59% to 75% of the discrepancy. In closed-loop on-policy learning, matched local mean and covariance lead to different future evidence distributions. In same-prompt Qwen RLVR, moment-matched gradient laws produce substantially distinguishable applied updates after clipping and AdamW. The future learning question determines both the stochastic resolution needed for prediction and the realizable law worth choosing for control.

Primary area:
learning theory
Keywords:
stochastic optimization, learning theory, stochastic gradients, reinforcement learning, RLVR

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_V7Q6OpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
1(−1)
Soundness
3
Read review
Paper summary

This paper studies what stochastic information is necessary to characterize future learning outcomes. The authors introduce an observable-relative framework that determines when a summary of stochastic updates, such as their mean and covariance, is sufficient to predict a given future outcome. They theoretically show that matching any finite number of moments does not guarantee identical learning outcomes. Building on this framework, they propose treating higher-order properties of gradient distributions as control variables to improve future learning performance. Experiments on small learning systems, neural networks, and LLM training are provided to support their theoretical findings.

Critical strengths

The paper provides a well-structured theoretical framework for understanding how stochastic update laws affect future learning outcomes. I appreciate the connection between observable-relative equivalence and stochastic-law control. The authors also provide extensive theoretical analysis and experiments across different settings, including neural networks and Qwen RLVR, to support their findings.

Critical weaknesses

Limited practical applicability.

My main concern is the gap between observable-relative distinguishability and practical stochastic-law control. While the paper provides a theoretical characterization of how stochastic update laws affect future learning outcomes, I think more discussion is needed on how to efficiently select and control realizable moment-matched update laws, especially considering the computational cost of estimating backward observables. The control experiments are mainly limited to small, carefully constructed learning systems, leaving the practical applicability of the proposed framework unclear.

Questions for authors

How can the proposed framework guide practical stochastic-law control, given the computational cost of estimating backward observables and selecting realizable moment-matched update laws? Could the authors provide more discussion or evidence beyond the small, carefully constructed learning systems?

4 comments
AuthorsOpenReview ↗

Stochastic law control improves Qwen's held-out math accuracy at the same GPU budget (1/2)

Thank you for recognizing the value of the theoretical framework and the breadth of the experiments. To address your concern about practical applicability, we used the framework to guide the choice of update laws which in turn improved Qwen2.5-1.5B's final math accuracy within the same GPU budget compared to two baseline procedures.

Specifically, choosing how to weight the four gradient estimates at each update raised held-out GSM8K accuracy from 58.358.3\\% to 61.461.4\\%, a gain of 3.1 percentage points against the standard baseline.

The three procedures we tested are:

  1. Standard Baseline: (Accuracy: 58.358.3\\%). At each update, sample four gradient estimates and average them equally. Make 200 updates at each of six prespecified learning rates, then choose the best run on validation.
  2. Shortlist learning rates and schedules (Accuracy: 59.559.5\\%). Keep equal averaging and screen 16 configurations, each with a learning-rate schedule set before training that specifies the rate at every update through update 200.
  3. Stochastic law control through gradient weightings (Accuracy: 61.461.4\\%). Keep the training schedule fixed and screen 16 ways of combining the four gradients, each of which preserves the conditional mean and covariance.

In both 16-candidate searches (in procedures 2 and 3), we used the same screen at update 64, then trained the four finalists to update 200, and chose one using the same validation panel. Within each of 20 independent seeds, the methods started from the same model initialization, followed the same training prompt order, and answered the same held-out questions. We used the differences within each seed to get 9595\\% confidence intervals of [+2.0,+4.2][+2.0, +4.2] points against standard tuning and [+0.9,+2.9][+0.9, +2.9] against averaging with the same shortlisting procedure.

Where do the 16 weightings come from?

For each prompt at a fixed model state, we combine four independent draws from the same gradient distribution. We choose weights cic_i with ∑i=14ci=1\sum_{i=1}^{4} c_i = 1 and ∑i=14ci2=1\sum_{i=1}^{4} c_i^2 = 1, so every candidate has the same expected gradient and the same variance in every direction as a single source gradient. The higher moments can still vary. In any direction with nonzero variance, ∑i=14ci3\sum_{i=1}^{4} c_i^3 scales skewness and ∑i=14ci4\sum_{i=1}^{4} c_i^4 scales excess kurtosis. Two of our weightings, for instance, share a skewness multiplier of 0.6250.625 and have excess kurtosis multipliers of 0.4380.438 and 0.5780.578. We use the framework to choose the 16 weightings before training, and we spread them across both multipliers so the screen can compare a range of noise shapes, which enables real control.

How to shortlist four finalists?

Section 4 shows that choosing the law that gives the lowest immediate loss can lead to worse terminal performance. In Qwen, our goal is to pick the run that will solve the most held-out math problems after 200 updates. At update 64, we look for an early sign of that ability. A model can start assigning more probability to correct solutions before it generates them reliably.

For each of the 48 development questions, we save two verifier-approved answers and two plausible wrong ones. Every candidate's checkpoint at step 64 scores the same answers, and for each question we subtract the wrong answers' mean log probability per token from that of the correct answers. We rank checkpoints by the average gap across questions. Because every checkpoint scores the same saved answers, the screen takes only a few batched forward passes.

To see whether those scores helped us decide which candidates to shortlist, we compared them with a simpler rule that ranked candidates by the fraction of training answers that passed the verifier over the first 64 updates. Under the same search budget, the saved-answer scores helped us choose candidates that achieved 0.7 percentage points higher held-out accuracy at update 200 (9595\\% CI [+0.1,+1.3][+0.1,+1.3]). This is the practical payoff of Section 4’s nonmyopic principle. The final task told us what to look for early, and that signal helped us spend the remaining training budget on runs that went on to solve more problems.

AuthorsOpenReview ↗

Stochastic law control improves Qwen's held-out math accuracy at the same GPU budget (2/2)

What does the search cost?

We gave each shortlisting procedure up to 14.8 GPU-hours per seed to train with its candidates and choose one final winner on validation. Standard tuning spent about 12.6 GPU-hours training six runs and used the remaining 2.2 to generate and verify answers on the validation panel. Both shortlisting procedures fit within that cap once we counted training, their share of building and scoring the answer bank, and the validation used to choose a winner. They ran the same number of updates, and we tracked GPU time directly because the runs can generate answers of different lengths. Once each procedure had picked a best candidate, we evaluated it on the same held-out questions outside the search budget.

We repeated the search for each seed, and once a configuration is chosen, a single 200-update run takes about 2.1 GPU-hours. To see what the shorter search gave up, we also trained using all 16 candidates to completion and scored them on the same held-out questions. Training all 16 took 34.0 GPU-hours per seed. The shortlist included the best-scoring model on 16 of 20 seeds, and the model we had already picked on validation finished only 0.55 points behind the best observed test score on average, with a one-sided 9595\\% upper bound of 0.90 points.

Thank you again for your thoughtful review. We welcome any further questions and respectfully ask you to positively reconsider your rating in light of this evidence about the framework's practical value.

Dear Reviewer V7Q6,

Thank you again for your review. With the discussion period ending on November 18th, we wanted to follow up once more. Your main concern was about the practical benefit of the framework beyond small constructed systems, and we ran new experiments in light of your review to investigate the applicability of stochastic-law control and its computational cost.

In brief, our rebuttal reports a Qwen2.5-1.5B RLVR experiment (20 paired seeds, equal GPU budget, with a search-matched control) in which choosing the update law improved held-out GSM8K accuracy by 3.1 points over tuned standard training.

Does this resolve your concerns? We are glad to clarify anything before the discussion closes. Thanks again for your thoughtful review. Your recommendations helped enrich the work, and we hope you will reconsider your score in light of the new findings.

Regards,

The authors

Thank you for your detailed response and for conducting additional experiments to address my concerns. I appreciate the authors' efforts and the new evidence provided.

As noted in my original review, this topic is outside my primary area of expertise, and I have already communicated my limited confidence to the AC. While I acknowledge the additional experimental evidence, I do not feel sufficiently convinced to change my overall assessment at this stage. I will therefore maintain my original rating and defer to the AC and other reviewers with more relevant expertise in evaluating the paper's contributions.

Thank you again for your time and efforts.

Reviewer_k79sOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper discusses what is required from a stochastic learning algorithm to predict a given objective. In particular, clearly only conditional mean and covariance (or other summary statistics) are not sufficient to obtain unique outcomes, as two different laws can have same summary statistics.

Critical strengths

The manuscript is well-written and addresses important practical question on learning algorithms.

Critical weaknesses

Despite the importance of the topic and provocative style of the text, the main message is not new nor surprising: summaries of laws do not characterize laws and hence not outcomes of the learning algorithms. As such, the novelty lies in characterizing when summary statistics would be sufficient information. It is also difficult to judge theoretical contribution of the paper. It contains two theorems. However, theorem 2 seems trivial, as its central message is that agreement of finite amount of moments do not imply equivalence of the tails.

Questions for authors

The appendix was missing in the submission, and hence it is difficult to assess whether theoretical results are correct and what is the novelty behind them. For example, is Theorem 1 consequence of existing results in the literature or does the proof contain any real novelty?

2 comments

Reviewer k79s rebuttal (1/2)

Thank you for recognizing the importance of the question and the clarity of the paper. We kindly disagree with your reading of its central claim. The fact that different laws can share finitely many moments is familiar. The question we answer is when hidden differences change the course of learning and how a learner can use them to improve the outcome.

You point that the paper's characterization of when summary statistics are sufficient could be a novelty. Indeed, it is one part of the picture as theorem 1 gives an exact criterion and measures how much a summary leaves unresolved. Theorem 2 shows why this matters even in strongly convex, contractive SGD.

Sections 3 and 4 carry this further. They show, while fixing the loss function, when the very same summary is enough and when it falls short, and they demonstrate that choosing among laws with matching moments can improve final predictions. We would welcome your assessment of these contributions, which the current review leaves unaddressed.

The supplementary appendix was uploaded with the submission. Sections A.2 and A.8 contain the full proofs of Theorems 1 and 2.

 

1. Why Theorem 2 isn’t trivial

On a bounded interval, matching the first kk moments brings two laws within O(1/k)O(1/k) in Wasserstein distance (Kong and Valiant, 2017 [1]). In our contractive SGD example, that O(1/k)O(1/k) bound holds for the distribution of the iterate at every individual step. Yet for every finite kk, one learning event can be nearly certain under one law and nearly impossible under the other. That gap is the starting point of our framework. To know whether a summary is enough, we have to ask what we want to predict about the run.

[1] Kong, Weihao, and Gregory Valiant. Spectrum estimation from samples. (2017): 2218-2247.

2. What Theorem 1 contributes

Theorem 1 uses classical duality. The contribution is reducing the learning question to the backward observable, the expected future outcome after each possible update. This gives an exact sufficiency test. Sections 3 and 4 use it to locate sufficiency boundaries and guide law selection. On compact support with a continuous observable, the largest gap over unrestricted laws sharing a summary is twice the best uniform approximation error using the retained functions and constants. For mean and full covariance, the test is whether the observable is quadratic on the update space. This quantifies what summaries miss and emphasizes the need for a framework that starts with the learning outcome. Propositions 1 and 2 connect this gap to trajectory error and realizable control.

3. When the same summary suffices

Sections 3 and 4 develop two central contributions that the review does not discuss. In Section 3, we show that the same summary can be exactly sufficient for one learning process and insufficient when we change what happens next even with the loss function fixed. We establish this boundary in two ways, by changing how training continues and by changing how much data a downstream learner receives. With fixed future sampling, matched mean and covariance give identical expected final loss at every horizon. When sampling responds to state, the same laws give different losses. In the second example, the data laws agree for one or two labeled examples. A third reveals dependence and changes prediction risk.

Reviewer k79s rebuttal (2/2)

4. How choosing a law improves learning

Section 4 shows that the freedom a summary leaves can be used to improve learning. Even with the conditional mean and full covariance fixed, a learner can improve its final prediction by choosing among feasible update laws. We show that the right choice can depend on both the learner’s current state and how much training follows. One bit of classifier state cuts terminal MSE by 4.64% against the globally best fixed schedule, found by exhaustive search. Both start with the same law and preserve exposure and pairwise sampling. Ignoring the state removes the gain.

At the same ReLU state, eight more SGD steps reverse which law is best. Choosing for later loss cuts terminal risk by 8.35% against state-aware immediate loss control, which ties the best fixed law. Practitioners can compare candidates from the same state under a common continuation. Finite rollouts recovered the preferred law at 99% simultaneous confidence.

5. Why this matters for modern training

Mean and full covariance do not determine how often Qwen RLVR produces a nonzero gradient. At the same state and prompt, the rate is 78.66% with four completions and 98.68% when combining four independent gradients from sixteen. All 32 prompts agree in direction. All 27 observed combined zeros came from four zero sources. Applied updates remain distinguishable after clipping and AdamW (p=0.001p = 0.001).

At matched moments, changing the law moves path risk toward the Gaussian prediction in 59 of 60 neural systems and changes future evidence failure by 3.074 percentage points after 32 policy gradient updates.

In one controlled follow-up, the chance of collecting all the evidence varied by less than 0.15 percentage points within each group of laws matched through third order, with 99% simultaneous confidence. Earlier laws matched on mean and covariance spanned about 2.03 points on the same outcome. A third order model predicts 35 new randomized laws without refitting (R2=0.99744R^2 = 0.99744).

In short, Theorem 2 proves the phenomenon exists. We then probe it across increasingly complex learning settings, measure how much it changes outcomes, and engineer the randomness in an update to improve learning. We respectfully ask you to reconsider the novelty, informativeness, and overall assessment in light of the detailed analyses and demonstrated control gains.

Reviewer_AGEfOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This theory-heavy paper studies stochastic learning rules that have the same mean and covariance at every state the algorithm visits, but still lead to different outcomes in the long run. In the context of reinforcement learning, for instance, the actual trajectory through the state space that the learner visits affects the quality of the estimated value function. The paper focuses on establishing whether a particular stochastic summary is sufficient to contain all the information needed to answer the question the learner cares about (i.e., the future quantity that needs to be predicted).

Critical strengths

The motivation of the work is clear, although it could have been placed better in the wider scientific context.

Stochastic update rules are widespread, so insights into what information needs to be retained in a specific learning context are valuable.

The paper considers quite a wide range of specific instantiations of learning problems and provides results for them. Besides the theorems, some numerical examples showcase the results.

Critical weaknesses

The paper could have been written in more accessible language: quite some concepts are used without introducing them. It jumps between high-level intuition in the introduction to quite specific examples of learning settings, which makes it hard to follow for a non-expert.

The potential impact of the work is not described in sufficient detail, for instance how practitioners could build on these results.

Questions for authors

I am wondering about how these results could impact reinforcement learning (outside the RLVR setting), as they for instance seem directly tied to deciding how to explore. Can the authors comment on that?

1 comment
AuthorsOpenReview ↗

Choosing the Update Law Shapes RL Exploration and Improves Qwen’s Math Accuracy

Thank you for recognizing the importance of the question, the paper’s contributions, and the range of settings we study. Your question about exploration points at what practitioners can gain from the framework. We demonstrate that using the framework to guide the choice of update laws can improve what a trained model does. In our Qwen RLVR experiments (in the paper and in our rebuttal to Reviewer V7Q6), that choice raised a language model’s accuracy on mathematical reasoning tasks by a statistically significant margin. Moreover, the policy-gradient experiment we describe below shows how the same idea helps with exploration by making the trained agent more likely to find all the evidence it needs.

Think of an agent that can query three sensors, each covering a different part of a room. In our simulation, each query selects one sensor and returns its evidence label, AA, BB, or CC. The agent starts out choosing AA 8080\\% of the time, and it earns a reward during training when a round of three queries returns one of each label. After 32 policy updates, we take the learned policy and ask how likely it is to find all three labels in its next nine queries. If pip_i is the probability of choosing sensor ii, the probability of finding all three is $ \Pr(\text{see all three labels in nine queries}) = 1 - \sum_{i\in\{A,B,C\}} (1-p_i)^9 + \sum_{i\in\{A,B,C\}} p_i^9.$ Thus, even at the final training step, the chance of seeing all three labels depends nonquadratically on how the update changes the policy. By Theorem 1, this means that matching mean and covariance alone gives no general guarantee of matching the chance of discovery. We tested whether that difference matters for two realizable update rules. We kept the task and sampling budget fixed and changed how we combined the gradients. Both update rules drew four independent gradient estimates G1,…,G4G_1,\ldots,G_4 at each update. One used G1G_1, and the other used (G1+G2+G3−G4)/2(G_1+G_2+G_3-G_4)/2. In both cases the weights sum to one, and so do their squares, giving us two different update laws with exactly the same mean and full covariance at any given policy. As a result and across roughly one million fresh simulated training runs per rule, the learned policies had 76.4076.40\\% and 78.4578.45\\% chances of finding all three labels, respectively. So changing how we combined the gradients shifted the chance of complete discovery by 2.05 percentage points (9999\\% CI [1.99,2.11][1.99, 2.11]).

We could also predict what would happen with new weightings. For each of 64 weightings c⃗\vec {c} (each c⃗\vec {c} gives a distinct update rule ∑iciGi\sum_i c_i G_i), we computed two numbers, ∑ici3\sum_i c_i^3 and ∑ici4\sum_i c_i^4, which describe how the weights scale the third and fourth cumulants of the gradient distribution. We fit a quadratic regression that uses those two numbers to predict the probability of finding all three labels after training. On 16 unseen weightings, the predictions had R2=0.9903R^2 = 0.9903 and a root-mean-square error of just 0.05 percentage points. Once we fit the regression, we could estimate how well a new weighting would cover the three sources from the weights alone. That gives us a way to screen new weightings before spending compute training with each one.

In a larger RL problem, suppose the agent rarely reaches a state where it could learn something valuable. A practitioner can start from the same policy and training budget, vary how the policy gradients are combined while preserving their mean and covariance (or even higher cumulants), and measure how often each learned policy reaches that state within a fixed number of interactions. That gives the practitioner a way to choose an update rule based on what they want the trained agent to discover.

In addition to the traditional RL example above, we have also demonstrated practical gains in a modern RLVR setting. As detailed in our separate response to Reviewer V7Q6, stochastic-law selection improved Qwen2.5-1.5B's held-out GSM8K accuracy by 3.1 percentage points over the standard equal-gradient-averaging baseline, under the same GPU search budget. This provides a concrete example of how choosing the stochastic update law with a future learning objective in mind can improve the trained model's performance. We provide the experimental protocol, compute accounting, and statistical results in our rebuttal to Reviewer V7Q6.

In an effort to make the paper clearer and more accessible to non-experts, we will include the three-sensor example in the introdcution and carry it into the formal setup. We will use it to define the update law as the distribution of the learner’s next update, the observable as the future outcome we care about, and a summary as the information we retain about the update law. We will also open each experiment with the question it answers before introducing the technical setup to avoid any jumps in narrative.

Thank you again for the thoughtful review.

Comments on the paper

2 comments

Thank you for reading the appendix carefully and for sharing your assessment of the proofs. We appreciate your willingness to revisit your evaluation.

Your review/follow-up focuses on Theorem 2, so we want to emphasize that the paper goes substantially beyond it, both in theory and in experiments. Theorem 2 is the starting point. It shows how large the effect of changing the update law can be, even in contractive SGD. If that effect is something we should already expect, that gives us all the more reason to study when it matters and what we can do about it. Those are the questions the rest of the paper takes on. We show when a summary of the update law is enough, bound how the differences it misses can add up over training, and demonstrate that choosing among laws with the same summary can improve learning.

Since your latest comment focuses only on Theorem 2, we would like to briefly restate the paper's results and bring your attention to the new control results:

When is a summary enough?

Theorem 1 gives an exact test for whether a summary is sufficient for a given future learning outcome. When the summary falls short, the theorem also tells us how much disagreement it can hide.

What happens over a whole training run?

Proposition 1 bounds how the differences a summary misses can add up along the states the learner visits. It also gives conditions under which matching mean and covariance keeps the error small over a fixed training time. Section 3 shows why the rest of the run matters. The same summary can go from exactly sufficient to insufficient when we change how training continues, even with the loss held fixed.

How can choosing an update law improve learning?

Section 4 shows two ways to improve the final outcome while preserving the conditional mean and full covariance.

  • A rule that uses just one bit of information about the learner’s state lowers terminal MSE by 4.64% relative to the best state-blind schedule.
  • At the same learner state, eight more SGD steps reverse which law is best. Choosing for the later outcome lowers terminal risk by 8.35% relative to a state-aware rule that chooses for immediate loss.

How far do these effects reach?

  • Neural networks. At matched mean and covariance, changing the update law moves neural path risk toward the Gaussian prediction in 59 of 60 systems.
  • RL exploration. In closed-loop policy gradient, changing the law shifts the probability of missing at least one type of evidence by 3.074 percentage points. Our response to Reviewer AGEf describes a follow-up showing that the effect persists when both laws use the same number of gradient estimates.
  • Language-model training. In Qwen RLVR, the nonzero-gradient rate changes from 78.66% to 98.68%, and the applied-update distributions remain distinguishable after clipping and AdamW. As reported in our response to Reviewer V7Q6, choosing how to combine gradients improved Qwen2.5-1.5B’s held-out GSM8K accuracy by 3.1 percentage points over tuned equal-gradient averaging under the same GPU search budget, across 20 paired seeds.

We would particularly value your view of the trajectory bounds and the results on choosing among laws with the same summary. How do these contributions factor into your assessment of the paper’s novelty and your overall recommendation?

Thank you for the clarification, and I agree that Theorem 2 is not the only contribution of the paper. As said before, I think it is borderline case, and I leave the decision to the editor whether to accept or reject.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.