- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to estimate the difficulty of each problem for the current model at a low cost during RLVR training, helping schedulers select training problems and allocate rollouts. The authors first construct a similarity graph from problem embeddings and then share training feedback across related problems. As training progresses, the method updates the latent-state assignments of problems and the success probability of each state to predict the success probability of each problem. The authors integrate the method into three schedulers, GVM, PCL, and GRESO, and evaluate it using two base models and four mathematical reasoning benchmarks. The results show that, under the same total generation budget, the method improves final performance in most settings while introducing little additional estimation overhead.
Critical strengths
- The paper addresses a practically important problem, and the method has low overhead. RLVR scheduling requires identifying problems suitable for the current model, but generating additional responses specifically to estimate success probabilities increases training costs. The proposed method uses feedback already collected during training and shares this information across related problems, reducing the need for additional probing. Table 2 reports low additional estimation overhead, which makes the method practically useful.
- The method works with different schedulers, and the experimental results are reasonably convincing. The authors replace the original difficulty estimators in GVM, PCL, and GRESO and compare performance under the same total generation budget. Improvements are observed in most experimental settings, suggesting that the benefits are not limited to a particular scheduling approach. The authors also analyze the effects of graph structure, initialization, instructions, and embedding models, and examine whether problems within the same cluster have similar difficulty. These analyses provide support for the method’s design.
Critical weaknesses
-
It remains unclear how the method adapts to changes in problem difficulty during training. Through analyses of instructions, embeddings, and clustering, the authors provide evidence that the relationships captured by the graph go beyond simple semantic similarity. However, two problems involving similar knowledge do not necessarily remain equally likely to be solved by the current model. As training progresses, the model may acquire the skills needed for one problem while still struggling with the other. In this case, the fixed graph structure continues to encourage information sharing between them. Although the method allows problems to change their state assignments, the paper lacks an illustrative example showing whether these adjustments can correct inappropriate sharing in a timely manner. A related concern is how historical feedback is retained. The main method continually accumulates previous success and failure counts. As historical evidence grows, new feedback has less influence on the estimates; it is also unclear how counts previously assigned to a state are handled when a problem later changes its state assignment. The authors discuss stale feedback in the appendix and provide a windowed variant, so my concern is not that changes in model capabilities have been ignored. Rather, it is how well the main method continues to reflect current success probabilities after such changes occur.
-
The reported correlation metric does not directly establish the accuracy of problem-level rankings. In Table 2, MAE is computed at the individual-problem level, whereas Pearson correlation is computed after aggregation into batches of 256 problems. A method may accurately predict the average success rate of a batch while failing to identify which problems within that batch are harder or easier. Therefore, higher batch-level correlation does not directly imply more accurate problem-level rankings. On Qwen, the windowed variant reduces MAE from 0.183 to 0.172, while correlation decreases from 0.836 to 0.712. Based on these results, the authors suggest that the windowed variant may improve numerical predictions while weakening ranking quality. I find this explanation plausible, but the current metrics are insufficient to establish it. Further comparisons are needed of how the two variants rank individual problems, which problems their schedulers ultimately select, and how they allocate rollouts.
Questions for authors
- How does the method adjust when neighboring problems no longer have similar difficulty? Please clarify the meaning of a “latent state”: does it primarily represent a knowledge category, a problem-solving skill, or a group of problems with similar success probabilities? If the model learns to solve one of two connected problems while the other remains difficult, how does the method reduce inappropriate information sharing between them? Could you provide a concrete example showing the empirical success rates, predicted success probabilities, and state assignments of these two problems over training? This would help me understand whether feedback can effectively correct limitations imposed by the static graph.
- How do historical counts adapt to changes in model capabilities and problem assignments? When the model’s capabilities improve substantially, or when a problem moves to another state, does the main method downweight old counts or reassign the relevant counts to the new state? If not, why should the accumulated success estimate still represent the current model rather than an average across previous training stages? As counts accumulate, the Beta distribution becomes increasingly concentrated; does this decreasing uncertainty still reflect the reliability of the current estimate? I would appreciate a clear explanation of the conditions under which this online approximation is reasonable.
- Could you directly compare problem-level rankings and actual selection decisions? At the same model checkpoint and on the same set of candidate problems, could you compare the problem-level ranking accuracy of the cumulative and windowed variants, and examine how their selected problems or rollout allocations differ? If the windowed variant predicts individual success probabilities more accurately on average but leads to worse scheduling outcomes, in which success-probability ranges do the most consequential errors occur? For example, does it incorrectly classify problems suitable for training as too difficult or too easy? Such an analysis would explain the differences between the two methods more directly than batch-level correlation.
2 comments
-
For Q1, We suppose that latent states represent underlying patterns among training samples that are difficult to describe directly in natural language, rather than necessarily corresponding to a specific knowledge category or reasoning skill. Based on our analysis of the clustering structure in Appendix G.2, we conjecture that these underlying patterns encode at least information related to knowledge categories and difficulty. The results show that the initial clustering structure is not only strongly associated with mathematical subject categories but can also distinguish different difficulty levels within the same subject. We have addressed other issues in our response to W1.
-
For Q2:
- We argue that temporal lag tends to introduce a systematic bias that affects samples similarly. Consequently, although absolute success-probability estimates may be biased, the relative difficulty rankings among samples can remain largely stable.
- Mathematically, a more concentrated Beta posterior indicates more accumulated statistical evidence, which generally leads to more reliable estimates. We believe Figure 2 provides empirical support for this. At the initial stage, γ does not take effect, and the estimates rely entirely on graph-based inference. As training progresses, the Beta posterior uncertainty generally decreases, while our method's performance steadily improves.
- We have addressed other issues in our response to W1.
-
For Q3:
-
First, regarding the differences in sample-level ranking accuracy between the two variants, we have addressed this issue in our response to W2.
-
Compared with the cumulative estimator, the windowed variant achieves better prediction accuracy for samples whose empirical success probabilities are close to 0 or 1.
-
The main difference across cumulative and windowed variants lies in rollout allocation, particularly for extremely difficult problems, while their sample-selection decisions remain largely similar.
- Rollout allocation. We observe that the windowed variant provides more accurate success-probability estimates for extremely difficult problems, with empirical success rates close to 0. However, it tends to allocate more rollouts to these problems than our cumulative estimator, then lending to excessive allocation. Through manual inspection of the training logs, we find that this counterintuitive behavior arises from the interaction between probability estimation and the downstream allocation strategy. Due to feedback sharing among neighboring samples and latent states, predicted success probabilities generally approach but do not reach exactly 0. Existing rollout-allocation frameworks are not explicitly adapted to this characteristic and may still allocate substantial budgets to effectively unsolvable problems. Consequently, better numerical prediction accuracy does not necessarily translate into more effective exploration, potentially wasting rollouts on overly difficult problems rather than more learnable ones.
- Sample selection. We find that the cumulative and windowed variants have a high overlap in their selected Top 100 samples from the candidate pool of each batch. Therefore, the two variants do not lead to substantial differences in the samples selected for training.
-
We sincerely thank the reviewer for the careful evaluation of our manuscript and the detailed, constructive feedback. Your comments, especially Question 3, have led us to think more deeply about the underlying principles of our algorithm, and have even helped us identify promising directions for future work.