acceptodds
Under review as a conference paper at ICLR 2027

Efficient RLVR Scheduling via Graph-Structured Online Difficulty Estimation

Abstract

Reinforcement learning with verifiable rewards (RLVR) improves the mathematical reasoning capabilities of large language models but relies on costly rollout exploration. Assigning the same exploration budget to samples with different difficulty levels is inefficient: easy samples may receive redundant rollouts, whereas difficult but learnable samples may receive too little exploration. Existing adaptive schedulers address this mismatch through curriculum-based sample selection or non-uniform rollout allocation based on estimated sample difficulty. However, obtaining reliable online difficulty estimates remains challenging: dedicated probing adds substantial generation overhead, whereas history-based estimators face a cold start with no initial observations and stale feedback, and typically ignore relations among samples. To address these limitations, we propose a plug-and-play graph-based online difficulty estimator for RLVR in mathematical reasoning that shares rollout feedback across related samples and continuously updates their difficulty estimates, mitigating cold start and staleness without dedicated probing. Specifically, we first construct a difficulty-aware sample graph based on semantic and reasoning similarities. Based on this graph, we introduce latent difficulty states and use a Potts prior to encourage neighboring samples to share the same state. We then employ a state-level Beta-Binomial model to aggregate the rollout outcomes associated with each state. Finally, we use an online mean-field variational algorithm to continuously update the latent-state assignments and state-level difficulty as new feedback arrives. Our framework can be integrated into sample-selection and rollout-allocation schedulers, enabling difficulty-adaptive exploration without dedicated probing. Experiments across multiple base models, RL schedulers, and mathematical reasoning benchmarks demonstrate that our framework achieves better performance under matched rollout budgets.

Primary area:
reinforcement learning
Keywords:
Large Language Model Reasoning, Reinforcement Learning with Verifiable Rewards, Adaptive Scheduling

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_zRA1OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to estimate the difficulty of each problem for the current model at a low cost during RLVR training, helping schedulers select training problems and allocate rollouts. The authors first construct a similarity graph from problem embeddings and then share training feedback across related problems. As training progresses, the method updates the latent-state assignments of problems and the success probability of each state to predict the success probability of each problem. The authors integrate the method into three schedulers, GVM, PCL, and GRESO, and evaluate it using two base models and four mathematical reasoning benchmarks. The results show that, under the same total generation budget, the method improves final performance in most settings while introducing little additional estimation overhead.

Critical strengths
  1. The paper addresses a practically important problem, and the method has low overhead. RLVR scheduling requires identifying problems suitable for the current model, but generating additional responses specifically to estimate success probabilities increases training costs. The proposed method uses feedback already collected during training and shares this information across related problems, reducing the need for additional probing. Table 2 reports low additional estimation overhead, which makes the method practically useful.
  2. The method works with different schedulers, and the experimental results are reasonably convincing. The authors replace the original difficulty estimators in GVM, PCL, and GRESO and compare performance under the same total generation budget. Improvements are observed in most experimental settings, suggesting that the benefits are not limited to a particular scheduling approach. The authors also analyze the effects of graph structure, initialization, instructions, and embedding models, and examine whether problems within the same cluster have similar difficulty. These analyses provide support for the method’s design.
Critical weaknesses
  1. It remains unclear how the method adapts to changes in problem difficulty during training. Through analyses of instructions, embeddings, and clustering, the authors provide evidence that the relationships captured by the graph go beyond simple semantic similarity. However, two problems involving similar knowledge do not necessarily remain equally likely to be solved by the current model. As training progresses, the model may acquire the skills needed for one problem while still struggling with the other. In this case, the fixed graph structure continues to encourage information sharing between them. Although the method allows problems to change their state assignments, the paper lacks an illustrative example showing whether these adjustments can correct inappropriate sharing in a timely manner. A related concern is how historical feedback is retained. The main method continually accumulates previous success and failure counts. As historical evidence grows, new feedback has less influence on the estimates; it is also unclear how counts previously assigned to a state are handled when a problem later changes its state assignment. The authors discuss stale feedback in the appendix and provide a windowed variant, so my concern is not that changes in model capabilities have been ignored. Rather, it is how well the main method continues to reflect current success probabilities after such changes occur.

  2. The reported correlation metric does not directly establish the accuracy of problem-level rankings. In Table 2, MAE is computed at the individual-problem level, whereas Pearson correlation is computed after aggregation into batches of 256 problems. A method may accurately predict the average success rate of a batch while failing to identify which problems within that batch are harder or easier. Therefore, higher batch-level correlation does not directly imply more accurate problem-level rankings. On Qwen, the windowed variant reduces MAE from 0.183 to 0.172, while correlation decreases from 0.836 to 0.712. Based on these results, the authors suggest that the windowed variant may improve numerical predictions while weakening ranking quality. I find this explanation plausible, but the current metrics are insufficient to establish it. Further comparisons are needed of how the two variants rank individual problems, which problems their schedulers ultimately select, and how they allocate rollouts.

Questions for authors
  1. How does the method adjust when neighboring problems no longer have similar difficulty? Please clarify the meaning of a “latent state”: does it primarily represent a knowledge category, a problem-solving skill, or a group of problems with similar success probabilities? If the model learns to solve one of two connected problems while the other remains difficult, how does the method reduce inappropriate information sharing between them? Could you provide a concrete example showing the empirical success rates, predicted success probabilities, and state assignments of these two problems over training? This would help me understand whether feedback can effectively correct limitations imposed by the static graph.
  2. How do historical counts adapt to changes in model capabilities and problem assignments? When the model’s capabilities improve substantially, or when a problem moves to another state, does the main method downweight old counts or reassign the relevant counts to the new state? If not, why should the accumulated success estimate still represent the current model rather than an average across previous training stages? As counts accumulate, the Beta distribution becomes increasingly concentrated; does this decreasing uncertainty still reflect the reliability of the current estimate? I would appreciate a clear explanation of the conditions under which this online approximation is reasonable.
  3. Could you directly compare problem-level rankings and actual selection decisions? At the same model checkpoint and on the same set of candidate problems, could you compare the problem-level ranking accuracy of the cumulative and windowed variants, and examine how their selected problems or rollout allocations differ? If the windowed variant predicts individual success probabilities more accurately on average but leads to worse scheduling outcomes, in which success-probability ranges do the most consequential errors occur? For example, does it incorrectly classify problems suitable for training as too difficult or too easy? Such an analysis would explain the differences between the two methods more directly than batch-level correlation.
2 comments

We sincerely thank the reviewer for the careful evaluation of our manuscript and the detailed, constructive feedback. Your comments, especially Question 3, have led us to think more deeply about the underlying principles of our algorithm, and have even helped us identify promising directions for future work.

  • For W1:
    • We acknowledge that our method does not dynamically remove edges after the initial graph construction, which may indeed lead to some inappropriate information sharing. However, we would like to emphasize that the graph structure only provides a relational prior rather than a fixed difficulty assignment. Each sample's soft latent-state assignment is updated online based on information from its graph neighbors and the currently available rollout feedback. Therefore, even when two neighboring samples gradually develop different difficulty levels during training, our method can partially correct inappropriate information sharing by adjusting their latent-state assignments. Furthermore, graph sparsification can reduce interference from weakly related nodes. By analyzing the training logs, we find that, during the later stages of training, approximately 20.6% of neighboring sample pairs in the sparsified graph exhibit empirical success-rate differences exceeding 25 percentage points, compared with approximately 36.5% without graph sparsification. These results suggest that graph sparsification can reduce the proportion of potentially inappropriate information sharing.
    • We have added a concrete case study in Appendix K of the revised manuscript to illustrate how our method adjusts latent-state assignments and partially corrects inappropriate information sharing when two neighboring samples develop different difficulty levels during training.
    • When a sample gradually shifts toward another latent state, the main method's previously accumulated success and failure counts remain in the states to which they were originally assigned, rather than being retrospectively redistributed. Meanwhile, subsequent rollout feedback is used to update the corresponding state-level success and failure counts according to the sample's updated soft latent-state assignments.
    • Regarding whether the main method can continue reflecting sample difficulty under the current policy, we provide a trajectory-wide analysis of difficulty tracking and sample-level ranking in Section 4.3 of the revised manuscript. Our method shows improving difficulty-ranking performance throughout training (Figure 2 and Figure 6). These results suggest that, despite using a fixed graph structure, our method can continue tracking changes in sample difficulty under the evolving policy by incorporating newly observed rollout feedback.
  • For W2, To address this concern, we have updated the experimental analysis in Section 4.3 of the revised manuscript and directly compared the sample-level ranking performance of different methods in Figure 2. Specifically, we introduce pairwise accuracy and Kendall's τb\tau_b to directly evaluate the agreement between predicted success-probability rankings and empirical success-rate rankings within each batch. The results show that our method also achieves strong performance in sample-level ranking and generally outperforms low-overhead estimators such as VIP and PCL across most training stages. Regarding the differences between the cumulative and windowed variants in downstream sample selection and rollout allocation, we provide further discussion in our response to Q3 below.
  • For Q1, We suppose that latent states represent underlying patterns among training samples that are difficult to describe directly in natural language, rather than necessarily corresponding to a specific knowledge category or reasoning skill. Based on our analysis of the clustering structure in Appendix G.2, we conjecture that these underlying patterns encode at least information related to knowledge categories and difficulty. The results show that the initial clustering structure is not only strongly associated with mathematical subject categories but can also distinguish different difficulty levels within the same subject. We have addressed other issues in our response to W1.

  • For Q2:

    • We argue that temporal lag tends to introduce a systematic bias that affects samples similarly. Consequently, although absolute success-probability estimates may be biased, the relative difficulty rankings among samples can remain largely stable.
    • Mathematically, a more concentrated Beta posterior indicates more accumulated statistical evidence, which generally leads to more reliable estimates. We believe Figure 2 provides empirical support for this. At the initial stage, γ does not take effect, and the estimates rely entirely on graph-based inference. As training progresses, the Beta posterior uncertainty generally decreases, while our method's performance steadily improves.
    • We have addressed other issues in our response to W1.
  • For Q3:

    • First, regarding the differences in sample-level ranking accuracy between the two variants, we have addressed this issue in our response to W2.

    • Compared with the cumulative estimator, the windowed variant achieves better prediction accuracy for samples whose empirical success probabilities are close to 0 or 1.

    • The main difference across cumulative and windowed variants lies in rollout allocation, particularly for extremely difficult problems, while their sample-selection decisions remain largely similar.

      • Rollout allocation. We observe that the windowed variant provides more accurate success-probability estimates for extremely difficult problems, with empirical success rates close to 0. However, it tends to allocate more rollouts to these problems than our cumulative estimator, then lending to excessive allocation. Through manual inspection of the training logs, we find that this counterintuitive behavior arises from the interaction between probability estimation and the downstream allocation strategy. Due to feedback sharing among neighboring samples and latent states, predicted success probabilities generally approach but do not reach exactly 0. Existing rollout-allocation frameworks are not explicitly adapted to this characteristic and may still allocate substantial budgets to effectively unsolvable problems. Consequently, better numerical prediction accuracy does not necessarily translate into more effective exploration, potentially wasting rollouts on overly difficult problems rather than more learnable ones.
      • Sample selection. We find that the cumulative and windowed variants have a high overlap in their selected Top 100 samples from the candidate pool of each batch. Therefore, the two variants do not lead to substantial differences in the samples selected for training.
Reviewer_nMp3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper focuses on how to estimate the difficulty of training examples at a lower cost, thereby enabling a more rational allocation of the generation budget in reinforcement learning for large models. The authors first organise questions with similar semantic and reasoning structures into a graph. They then combine a shared latent difficulty state, Bayesian modelling and online variational inference to use training feedback from existing questions to update the probability of correct answers for other questions. This estimator can be integrated into existing scheduling methods to help determine which questions to train on and how many responses to generate for each question.

Critical strengths

The issues addressed in this paper are of practical significance. Consequently, utilising feedback generated during the training process represents a promising avenue for research. This paper presents a concrete implementation. The method can be integrated with different schedulers, offering considerable flexibility in its application. The authors retained the original scheduling logic, replacing only the difficulty estimation component, and conducted comparisons under the same generation budget. This helps to determine whether the benefits stem from the new estimator.

Critical weaknesses

I am not yet convinced by the paper’s explanation of its innovativeness. Section 5.1 of the reference paper ‘Adaptive Rollout Allocation for Online Reinforcement Learning with Verifiable Rewards’ already employs Gaussian processes on prompt embeddings to update the success probabilities of unobserved samples based on feedback from observed samples. Merely emphasising that feedback can be propagated between relevant samples is insufficient to demonstrate the paper’s independent contribution relative to prior work.

Questions for authors
  1. Apart from consistency with the original baseline, what evidence is there to suggest that this method remains effective on updated or larger-scale policy models?

  2. If the model is retrained using different random seeds, will the improvements shown in Table 1 still hold? Could you provide the mean, standard deviation, or confidence intervals from repeated training runs?

  3. The paper constructs a graph using instruction-aware embeddings and cosine similarity. However, semantic similarity does not necessarily imply similarity in problem difficulty.

  4. Can this work actually solve the cold-start problem? If so, how does it address it?

Please address the weaknesses outlined above. If these concerns are adequately addressed, I would consider increasing my score.

4 comments

We sincerely thank the reviewer for the careful evaluation of our manuscript and the constructive feedback. We understand the reviewer's concerns regarding the novelty of our method, its effectiveness on larger models, the robustness of our experimental results, and the cold-start problem. To address these concerns, we further clarify the key distinctions between our approach and prior work and provide additional experimental analyses and supporting evidence.

  • For W1, we acknowledge that VIP has already introduced cross-sample feedback propagation. However, our contribution goes beyond information propagation, focusing on more reliable sparse-feedback modeling and a potentially generalizable probabilistic inference framework. Our key contributions are twofold:

    • More robust modeling of sparse feedback. Unlike VIP, which uses a Gaussian process to predict clipped and logit-transformed empirical success rates, we directly model rollout success and failure counts through a state-level Beta–Binomial model. By aggregating feedback through soft latent-state assignments, our method mitigates the instability of per-sample estimates under limited rollout budgets (Sections 3.4–3.5).

    • A unified and extensible probabilistic inference framework. We formulate difficulty estimation as an online graph-structured latent-variable inference problem, jointly incorporating relational priors, soft state assignments, and state-level observation models under the ELBO. By separating the relational prior from the observation likelihood, our framework can potentially accommodate different reward types by replacing the likelihood and re-deriving the corresponding posterior updates, providing a reusable modeling foundation for broader LLM post-training scenarios.

  • For Q1, we additionally conducted a lightweight experiment on a larger-scale model. The results show that our method continues to outperform the original baseline on the newer and larger Qwen3.5-9B model under the GVM framework.

MATHAIME 24AIME 25Olym
Base86.261.744.667.7
GVM86.866.750.472.0
GVM+Ours87.366.753.372.4

Meanwhile, prior studies such as PCL (Gao et al., 2026) and GRESO (Zheng et al., 2026) have provided empirical evidence for the effectiveness of difficulty-aware scheduling on larger models. Our contribution is complementary: rather than relying on fixed difficulty estimates tied to a particular model, our estimator continuously updates sample difficulty using rollout feedback from the evolving policy. This allows it to provide difficulty estimates for adaptive scheduling even as model capabilities and sample difficulty distributions change. Additionly, he improvements on Qwen3.5-9B provide additional evidence that this mechanism remains effective on a stronger policy model.

  • For Q2, to evaluate the robustness of the improvements reported in Table 1, we repeated training using Qwen2.5-Math-1.5B as the base model under the GVM framework with three different random seeds (42, 43, and 1109). We report the mean performance, standard deviation, and confidence intervals calculated from these repeated runs:
MATHAIME 24AIME 25Olym
GVM71.2±0.60 (69.7,72.7)11.1±1.2 (8.12,14.1)11.3±0.83 (9.18,13.32)35.7±0.74 (33.86,37.54)
GVM+Ours74.3±0.50 (73.02,75.52)16.4±1.1(13.78,18.99)13.1±1.2 (10.07,16.04)72.4±0.82 (36.1,40.15)
  • For Q3, first, we would like to clarify that we do not assume that semantic similarity necessarily implies difficulty similarity.

    • In fact, in our method, the initial graph structure only provides a relational prior rather than a fixed difficulty assignment. Meanwhile, the initial semantic similarity does not permanently determine whether two samples should share difficulty-related evidence.

    • Second, the results in Table 5 validate the effectiveness of the difficulty-aware instruction, suggesting that it helps the embedding representations capture certain difficulty-related characteristics rather than semantic information alone.

    • Furthermore, Figure 4 in Appendix B shows that adjacent nodes in our graph exhibit significantly smaller differences in empirical success rates than randomly paired nodes. This provides further evidence that our constructed graph preserves meaningful difficulty-related relationships throughout training.

  • For Q4, We do not claim that our method completely solves the cold-start problem. However, it can substantially alleviate the difficulty of estimating sample difficulty during the early stages of training. As shown in the early-stage portion of Figure 2 in our revised manuscript, our method demonstrates a clear advantage over other low-overhead difficulty estimation methods. This suggests that our approach can provide more informative difficulty estimates even when direct observations are sparse at the beginning of training. We suppose that this advantage mainly comes from two aspects.

    • First, by propagating historical rollout feedback through graph neighbors and shared latent states, our method improves the utilization of the limited feedback available during early training. Even when a sample has not been directly sampled, its success probability can still be estimated using observations from related samples.

    • Second, state-level difficulty modeling aggregates successful and unsuccessful rollout observations across related samples. This helps mitigate sampling errors caused by sparse feedback and reduces the influence of stochastic fluctuations from individual samples on early-stage difficulty estimates.

Thank you for the detailed responses and clarifications. The authors have adequately addressed my concerns, and I am satisfied with the explanations and additional evidence provided. I will increase my overall score accordingly. Good Luck!

Thank you so much for your positive feedback and for your recognition of our work. We greatly appreciate your insightful comments, which help us improve the manuscript. We are sincerely grateful for your time and support throughout the review process.

Reviewer_jeQ8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes a difficulty-aware scheduling in reinforcement learning with verifiable feedback, mainly designed via instruction-conditioned embeddings and their graph. The graph is updated via the current policy's realized difficulty during the training, updated via online mean-field variational inference. Experiments report improved accuracy across three schedulers, two small base models, and four mathematics benchmarks under matched rollout counts.

Critical strengths
  1. The proposed difficulty estimation method yields a consistent gain over the baseline method, demonstrating the empirical gain of using graph-structured online difficulty estimation.

  2. A variety of ablation studies and analyses on the prompt states and their difficulty estimations support the practical gains of the graph-structured mechanism. For example, estimation error analysis by KK and within-cluster difficulty measurement supplement core assumptions of the method.

Critical weaknesses

The paper comprehensively demonstrates the effectiveness of the proposed method already with diverse ablation studies in the Appendix. There are two points for improvement:

  1. In the default cumulative variant, state-level Beta concentration grows with accumulated rollout evidence, reducing the influence of a comparable amount of recent feedback. This matters because scheduling decisions depend on difficulty under the current policy: stale estimates could cause the scheduler to overlook newly learnable prompts or allocate exploration to prompts that have already become easier. Although the downstream gains and the windowed comparison are encouraging, the reported batch-level correlations do not establish how reliably the cumulative estimator tracks prompt-level difficulty changes, particularly for infrequently revisited prompts

  2. The improved accuracy under matched rollout budgets and low estimator overhead provide encouraging evidence of efficiency. However, empirical success rates are Monte Carlo estimates whose sampling variance decreases with the number of independent rollouts per prompt under a fixed policy, making this count a key experimental variable. Without examining this dependence, it remains unclear how strongly the benefit of cross-prompt feedback sharing depends on sparse observations and whether it persists when more accurate direct estimates are available.

Questions for authors
  1. How does the estimator’s accuracy depend on the number of observed rollouts per prompt? In particular, what evidence distinguishes the benefit of sharing feedback across related prompts from the benefit of simply collecting more rollouts for each prompt?

  2. For the Sample Before Step (SBS) baseline, does each training step involve fresh rollouts from the current policy for the entire prompt pool, a candidate subset, or only the selected training batch?

  3. After a prompt is used in a training step, when does it become eligible for selection again under each integrated scheduler?

1 comment

We appreciate Reviewer's careful reading of our paper and the thoughtful comments. We are grateful for these constructive suggestions.

  • For W1: Generally speaking, as the outcomes become more and more certain, the influence of new feedback should naturally decrease. We have discussed this point in Appendix A.3. In addition, the updated Figure 2 in the revision shows that our algorithm also achieves reliability superior to methods with comparable overhead at the sample-ranking level.
  • For W2 and Q1: We have added experiments in Appendix C. Following the setup of Section 4.3 and taking Qwen 2.5-Math-1.5B as the base model, we increased the sampling budget up to 32 rollouts to analyze how different numbers of rollouts affect our algorithm. Our experiments show that increasing the number of rollouts improves the precision of the feedback and thus the accuracy of our algorithm, while the gains gradually diminish. Moreover, the ablation on γ in Figure 6 demonstrates that the pure graph-based algorithm remains effective throughout the entire training process, which illustrates the benefit of sharing feedback across correlated prompts.
  • For Q2: Sample Before Step refers to generating fresh rollouts only for the selected training batch; we will clarify this point in the revision.
  • For Q3: GVM does not involve sequential scheduling and is only used to determine how many rollouts each sample receives. Both PCL and GRESO select prompts from the current candidate batch. After a prompt has been used, it becomes eligible for selection again when it is encountered in the next round of data traversal.
Reviewer_xbmtOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies online difficulty estimation for RLVR scheduling. The authors build a sample graph from semantic and reasoning similarity, assign prompts to latent difficulty states with a Potts prior, pool rollout outcomes with a state-level Beta-Binomial model, and update the assignments online by mean-field variational inference. The estimator is plugged into GVM, PCL, and GRESO. On Qwen2.5-Math-1.5B and Llama-3.2-1B trained on NuminaMath, it improves downstream accuracy under a matched rollout budget and gives lower-cost difficulty estimates than probing.

Critical strengths
  1. Sharing sparse rollout feedback across related math problems is a reasonable way to reduce dedicated probing, and using one plug-and-play estimator inside both sample selection and rollout allocation makes the idea easy to evaluate.

  2. The gains hold on two model families and three schedulers (Table 1), and Table 2 shows a better batch-level correlation than several low-cost estimators with small extra overhead.

  3. The paper is clearly written and well organized, and the appendices provide thorough derivations and hyperparameter analyses that support reproducibility.

Critical weaknesses
  1. The related work and baselines are not comprehensive enough. Many recent studies have explored online difficulty estimation, adaptive task selection, and rollout allocation for RLVR, including but not limited to [1-5], yet none of them are discussed or compared. Thus, the conceptual and empirical advantages of the proposed graph-based estimator over these methods remain unclear.

  2. In Section 3.6, observed samples still average the graph prediction with their historical success rate (γ=0.5\gamma = 0.5), which partially reintroduces stale feedback, and the static semantic graph may not always reflect policy-dependent difficulty changes during training. Besides, the presentation of several figures (e.g., Figures 1-3) could be further polished to improve readability, such as enlarging small text labels and making the visual layout cleaner.

[1] BOTS: A Unified Framework for Bayesian Online Task Selection in LLM Reinforcement Finetuning

[2] Small generalizable prompt predictive models can steer efficient RL post-training of large reasoning models

[3] Improving Sampling Efficiency in RLVR through Adaptive Rollout and Response Reuse

[4] Dynamics-Predictive Sampling for Active RL Finetuning of Large Reasoning Models

[5] Improving data efficiency for LLM reinforcement fine-tuning through difficulty-targeted online data selection and rollout replay

Questions for authors
  1. Can the authors add a discussion of [1-5] and, if possible, compare with at least one of the online difficulty-estimation methods among them?

  2. In Section 3.6, observed samples average the graph prediction with their historical success rate (γ=0.5\gamma = 0.5). How sensitive is the performance to γ\gamma, and how much of the gain comes from the graph inference itself versus this historical smoothing?

  3. Since the sample graph is fixed before training while difficulty is policy-dependent, do graph neighbors remain correlated in pass rate as the policy updates?

3 comments

We sincerely thank you for the time and effort devoted to reviewing our paper, as well as for the constructive and insightful comments, which have greatly helped us improve the quality of this work. We respond to each point below.

  • For W1 and Q1: We have reproduced DPS for direct comparison and provided discussions of the remaining related works in the updated revision (Section 2.2). For DPS, to ensure a fair comparison, we map our difficulty estimates onto the same three-category classification using thresholds of 0.25 and 0.75; we find that our method outperforms DPS in almost all settings. Overall, these results demonstrate the effectiveness of our method. Here is the results:
ModelStepsOurs AccuracyDPS AccuracyOurs Macro-F1DPS Macro-F1
Qwen-2.5 Math-1.5BEarly53.54%41.26%0.46930.1947
Qwen-2.5 Math-1.5BMiddle70.19%66.41%0.67140.6025
Qwen-2.5 Math-1.5BLast74.26%67.93%0.71380.6520
Qwen-2.5 Math-1.5BFull66.00%58.53%0.64960.5305
Llama-3.2 1B-InstructEarly61.57%57.97%0.44710.2540
Llama-3.2 1B-InstructMiddle74.94%72.40%0.64140.6023
Llama-3.2 1B-InstructLast75.81%74.62%0.65970.6372
Llama-3.2 1B-InstructFull70.77%68.33%0.57500.5686
  • For W2 and Q2: We have added Figure 2 and Figure 4 in the revision, and we analyze the performance of our algorithm without accumulated historical evidence to demonstrate the effectiveness of the graph-based inference itself; we have also added a discussion of the hyperparameter γ in the appendix . Overall, the early-step stage in Figure 2 and the Cold Start setting in Figure 4 involve little or no accumulated historical evidence, yet our method still achieves gains over methods with comparable overhead, demonstrating the benefit of the graph-inference algorithm. Moreover, the performance of our algorithm varies little for γ ∈ [0.5, 0.8]. We have also added more figures to improve readability.

  • For Q3: The pass rates of neighboring samples remain correlated, although with a gradually decreasing trend, and they always stay within a relatively acceptable range. Additional experiments (Figure 4) on Qwen show that the mean absolute difference in empirical pass rates between graph neighbors is about 0.0771 ± 0.1102 before training and about 0.1150 ± 0.1587 in the late stage of training, indicating that graph neighbors remain correlated to a certain extent as the policy updates. We provide further discussion in Appendix B of the revision. Meanwhile, inspired by zRA1's comment, we have also included a case study in the appendix (Appendix K) discussing how our algorithm corrects the difficulty estimates when the pass rates of neighboring samples become uncorrelated.

Thanks for the detailed response and additional experiments. My main concerns have been well addressed, and I am happy to recommend acceptance.

Just one minor question: since the pass-rate correlation between graph neighbors slightly weakens in later stages, would decaying the Potts coupling strength β\beta during training help, or is a fixed β\beta already sufficient?

We tune this hyperparameter on the existing training trajectories and analyze the MAE between the predicted and actual results during the iterative process. We make two very simple attempts: linear decay and exponential decay. Overall, exponential decay performs worse than the fixed setting, while linear decay is just slightly better than the fixed setting. I personally think that focusing on this hyperparameter is of relatively little significance, because the MAE improvement is limited (the highest group does not exceed 2.72%), and designing a corresponding adaptive method requires more hyperparameters.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.