- Rating
- 2
- Confidence
- 1
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies tabular policies for iterative symbolic tasks through deterministic policy codes, a Gibbs distribution over those codes, and product-distribution variational approximations. It proves a non-reward-decreasing path to perfect reward for untied (0,0) quasigroup folds, derives exact counting/partition results for parity, and investigates Monte Carlo traps, coordinate-ascent variational inference, and entropy/curriculum effects in transformers. The strongest contribution is the tractable landscape analysis; the connection from spin-glass states to actual training traps is less firmly established.
Critical strengths
-
Substantive constructive and counting results. The constructive escape path for untied (0,0) quasigroup folds and exact parity counting are substantive, problem-specific results; exhaustive enumeration at T=2,3,4 agrees with the latter. They are not immediate corollaries of standard fully observed policy-gradient convergence results.
-
The policy-code perspective makes a useful restricted model accessible. It allows comparisons among exact tabular optimization, Monte Carlo and transformer experiments. Closest work already studies restricted-policy myopic optima and entropy-dependent optimization landscapes; the present paper's specific combinatorics and quasigroup construction remain distinct, rather than making the general existence of traps or benefits of entropy new.
Critical weaknesses
-
The implications for practical RL training are insufficiently developed. The non-decreasing escape-path theorem gives an interesting structural result, but it remains unclear what diagnosis or training decision it supports in large-scale RL. The construction uses exact continuation values and independent edits to table cells; the paper does not explain how these operations translate to a shared-parameter model trained from sampled terminal rewards. Training stagnation may also reflect the dataset or the training infrastructure. The analysis should explain when the proposed mechanism is relevant, what observations could distinguish it from these alternatives, and what intervention would follow. The common-curriculum entropy experiment supports an improvement on the tested symbolic tasks, but does not establish this broader diagnostic or practical guidance.
-
The pure-state connection in Section 3.3 is overstated. The argument moves from weakly correlated pure states to fully factorized distributions "by definition", then identifies them with relevant variational optima. Weak correlation, or even pairwise independence, does not imply joint factorization. Figure 2 supports empirical agreement between cell marginals and CAVI endpoints, rather than the stronger identification of joint distributions. This limits the proposed explanation of training traps; the independent escape-path theorem and parity counting remain valid contributions.
Questions for authors
-
Practical implications: What concrete guidance do the results provide for diagnosing or addressing stagnation in large-scale RL training? Please discuss the conditions under which the analyzed mechanism could apply to shared-parameter models, and how it could be distinguished from limitations arising from the dataset or training infrastructure.
-
Scope of the theoretical connection: Does Section 3.3 establish a correspondence of joint distributions, variational stationary points, or only empirical marginal agreement? Please clarify the supported claim and revise the inference from weak correlation to factorization accordingly.
3 comments
(1) Practical implications: First, regarding parameter sharing, we would like to emphasize that our bound, which is exact for untied/unshared models, also holds approximately for tied models (shared) at low temperatures (since double visits to policies are uncorrelated at low temperature when policies are deterministic/codes). We further comment that, as part of future work, we developed ways to encourage tying/sharing parameters by augmenting the RLVR loss, such that the bound is exact with respect to the spin system with (Eq. (7)).
More generally, our results do exactly what the referee asks – they separate between difficulties of optimization coming from parametrization and noise, from those which are an inherent property of the landscape. As such, they allow us to understand when it is worthwhile to invest compute in solving the former issues, and when to accept traps as a reality of training, and think about how we change the task or the loss.
In practice, this is exactly the process that allowed us to train the transformer. The new version contains an ablation study of 334 different transformer RLVR runs that have failed (App. I.5, Table 7), prior to our understanding that extremely high regulators are useful. Following these disappointing results, we studied the tabular models to establish the existence of suboptimal local maxima or prohibitive diffusive barriers in the reward landscape. Having demonstrated that there are no such problems (parity has mild diffusive barriers, but the other tasks we studied do not, see revised manuscript App. I.5, Fig. 5), we deduced that this is mainly a problem of exact-gradient RLVR updates not finding the optimum from random initializations. The final clue was noting that at least for some tabular models, regulator values close to collapse were beneficial. Applying those insights to transformers, we managed to train them so that they solve the task.
A more general practical pipeline we can envision is the following. Given a pure RLVR problem that fails to optimize yet has a deterministic code-like solution, consider first our spin-glass model over different families of codes containing the desired code. Study this problem using Monte Carlo or analytical techniques to unravel potential pitfalls in the landscapes (e.g. ones coming from shortcut heuristics which are suboptimal local maxima of the reward, as in our correlated parity task). Once these are addressed, look for regulators or other "hacks" which can improve RLVR. We can envision this pipeline leading to various disciplined advancements in pure RLVR, some of which may carry over to RLVR on pre-trained models.
(2) Scope of the theoretical connection: We thank the referee for this question, as the text may have mixed good approximations with proof level results. The exact result of Sec. 3.2 is that the RLVR loss coincides with a mean-field approximation to the spin-glass model. In such spin-systems, mean-field approximations coincide with the best product distribution. In the theory of spin-glasses, such product/mean-field states are good approximations to the states reached in practice using local Monte Carlo updates. As in physics, by good approximation we do not mean that the actual distribution is exactly a product state. What we mean is that when compared with experiments, the mean-field/product-probability approximates relevant observables well. The relevant observable here is the reward. In the revised manuscript, we clarify this (Sec. 3.3), and provide further evidence that this is the case. To ground this comparison, the revised manuscript defines traps explicitly (App. G.1) and specifies how they are identified in the Monte Carlo runs behind Fig. 2 (App. I.2). Establishing this as a mathematical proof (in the thermodynamic limit) is beyond the scope of this work. Nonetheless, App. G of the revised submission gives an exact expression for the reward gap between a trap and the product of its marginals (Theorem G.1), which we find to be below relative to the trap's reward. Moreover, CAVI started from this product policy barely changes its reward (Fig. 2), and ends within (parity) and (tied models) of the trap's own Monte Carlo reward (App. I.2), so that each trap we find corresponds to an RLVR fixed point with the same reward.
Thank you for the thoughtful response. Your clarification that the pure-state approximation concerns the reward addresses my second question. I also appreciate how the theoretical analysis motivated the experiments with strong entropy regularization. I remain uncertain, however, about how far the tabular results can explain the behavior of transformer-based language models during RL training. On balance, I will keep my rating at 2. I should emphasize that my familiarity with this area is limited.
Response to reviewer EMuf
We thank the referee for their valuable insights, yet respectfully disagree with their following remark: "The strongest contribution is the tractable landscape analysis; the connection from spin-glass states to actual training traps is less firmly established." We prove a lower bound on the RLVR loss via an auxiliary spin-glass model. As spin-glass models have been studied extensively in physics, this is not only a rigorous result but also a potentially valuable bridge for cross-fertilization between physics and computer science. Furthermore, the revised manuscript provides further numerical and analytical evidence for the relation between pure states and RLVR (Sec. 3.3, Fig. 2 and App. G), as detailed in our response to Q2 below.
We thank the referee for highlighting the policy-code perspective as a strength of our work. We would, however, like to clarify how we see our contribution, since the referee's summary places its novelty mainly in the specific combinatorics and the quasigroup construction. We agree that the general existence of traps, and the fact that entropy can aid optimization, are not new, and we do not claim them; the quasigroup construction, in turn, serves as a test bed. Our main contribution is a framework: an exact mapping from entropy-regularized RLVR to a spin-glass model, which, as discussed above, lower-bounds the RLVR loss and makes the toolbox of spin-glass theory available for analyzing RLVR. This framework yields results that go beyond the works cited by the referee, which study the behavior of specific optimizers (DeWeese & Qu, myopic optima under restricted policies; Ahmed et al., entropy and the optimization landscape), and in one case point in the opposite direction. First, although our tabular policies are myopic in DeWeese & Qu's sense, we prove (Theorem 4.1) that for a broad class of untied models the reward has no suboptimal local maxima, so that failures there must come from the optimization rather than from the landscape. Second, the framework allows us to anticipate when entropy helps: for parity, the first-order transition we compute exactly is what led us to the strong regulators with which transformers learn the task.
Questions are addressed bellow.