acceptodds
Desk rejected at ICLR 2027

MechDS: Mechanistic RLVR Data Selection via Reasoning Component Dependence

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has become a central paradigm for improving the reasoning capabilities of large language models, yet effective data selection remains a key challenge. Existing RLVR data selection methods typically estimate the value of training data through extrinsic proxy signals, while the model's internal reasoning mechanisms remain largely unexplored. We study this missing dimension through reasoning component dependence (RCD), which measures how strongly a training sample relies on components critical to reasoning during generation. To examine information beyond extrinsic proxies, we compare RLVR training on samples with similar behavioral entropy but different reasoning component dependence. Empirically, a consistent pattern emerges across entropy-matched comparisons: low-dependence samples yield better RLVR performance than high-dependence samples, while high-dependence samples converge faster and exhibit smaller answer-reasoning gaps. These findings motivate MechDS, a dynamic data selection strategy that adaptively balances stability, learning potential, and exploration according to behavioral entropy and reasoning component dependence. Experiments on three models and three benchmarks show that MechDS consistently improves RLVR performance over strong data selection baselines, demonstrating that reasoning component dependence provides mechanistic information beyond extrinsic signals for selecting effective RLVR data.

Primary area:
reinforcement learning
Keywords:
Data Selection, RLVR

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_wkGzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studied the RLVR data selection from the views of Reasoning Component Dependence (RCD) and Behavioral Entropy (BE). It first analyzed the different roles of training samples: low-BE high-RCD samples stabilize early optimization, mid-BE low-RCD samples provide the main performance gains, and high-BE low-RCD samples support late-stage exploration. Further, based on the above analysis, this paper proposed a stage-adaptive data selection strategy Called MechDS. Experiments on three models and three benchmarks show the effectiveness of MechDS.

Critical strengths
  1. The paper introduces an interesting perspective by incorporating mechanistic information into RLVR data selection. The analysis of high-RCD and low-RCD samples under different behavioral entropy levels is particularly interesting.
  2. The performance of the proposed method is promising. Specially, MechDS-Dynamic with a 10% selection budget surpasses DEPO with a 30% budget.
Critical weaknesses
  1. The generalizability of the stage-adaptive selection strategy deserves further investigation. Could the paper report the trained model on other domain benchmarks like LiveCodeBench and Crux?
  2. The design of the proposed method seems a little complex. Additional comparisons with alternative scheduling strategies could strengthen the justification for the proposed design.
Questions for authors
  1. The computational overhead of the proposed method should be reported and compared.
Reviewer_tEqhOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MechDS, an RLVR data selection method that combines reasoning component dependence (RCD) with behavioral entropy (BE). RCD measures changes in the model’s next-token distributions after ablating selected attention heads. MechDS uses these two signals to select training data according to the training stage. Experiments on three models and three mathematical reasoning benchmarks show improvements over the evaluated data selection baselines.

Critical strengths

The paper combines reasoning component dependence with behavioral entropy for RLVR data selection. Comparisons within the same BE bands examine the effects of training on subsets with different RCD levels. The full method achieves strong performance across multiple models and mathematical reasoning benchmarks. The ablation studies also support the benefits of combining both signals, selecting reasoning-relevant attention heads, and adapting data selection across training stages.

Critical weaknesses

MechDS combines an ablation-based component dependence score with a predefined selection schedule in the BE–RCD space. Although the full method outperforms CircuitSeer, the comparison does not isolate the added value of RCD relative to CircuitSeer’s attention variance score within the same selection framework. Moreover, the BE–RCD training patterns motivating the schedule are demonstrated only on Qwen2.5-Math-7B-Base.

Questions for authors
  • The paper provides ablation studies supporting the effectiveness of RCD, BE, and the dynamic selection strategy, but has not established whether RCD is more effective. Could the authors keep the other components of MechDS unchanged and replace only RCD with CircuitSeer's attention variance score for comparison? If the two approaches achieve similar performance, what is the additional contribution of RCD?
  • Figure 2 shows that high-RCD samples converge faster, while low-RCD samples achieve better final performance. However, this analysis is conducted only on Qwen2.5-Math-7B. Do the other two models exhibit the same patterns? Could the authors provide corresponding experiments to verify whether the empirical findings underlying MechDS's dynamic selection strategy are consistent across models?
Reviewer_GmKcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates leveraging internal model information for RLVR data selection. The authors first identify important components via attention head ablation on trajectories filtered for answer correctness and reasoning validity. They then define Reasoning Component Dependence (RCD) using the KL divergence of token distributions between the original model and the ablated model. The authors observe that within similar Behavioral Entropy (BE) bins, low-RCD data generally yields higher final accuracy, while high-RCD data reaches target performance faster with a smaller answer-reasoning gap. Motivated by these observations, MechDS selects data at different training stages following a predefined curriculum using precomputed BE/RCD scores. Experiments are conducted across three models and three mathematical reasoning benchmarks.

Critical strengths
  1. Compared with DEPO, MechDS-Dynamic achieves clear improvements in average Mean@16 across the three models, and MechDS-Static also outperforms major baselines. This indicates that the performance gain does not fully rely on dynamically changing training data.
  2. The ablation study is comprehensive. The authors compare variants including BE-only, RCD-only, random attention heads, low-importance attention heads, and versions with different curriculum stages removed. This work further introduces a dependence metric based on intervened output distribution shifts and investigates its training effects in RLVR, which is a valuable extension for research.
Critical weaknesses
  1. The paper selects high- and low-RCD data within a broad bin of normalized BE but does not report the actual BE distributions of the two groups. Falling into the same bin does not guarantee matched means, quantiles or tails. Besides, the two groups may differ in initial success rate, problem type, length and data diversity. Figure 4 shows reward and length trends during training and cannot substitute for controls over these confounding variables at sampling time. Therefore, current evidence supports that “RCD carries additional predictive value within coarse BE stratification”, yet it cannot rule out explanations from common proxy signals and their combinations.
  2. MechDS-Dynamic simultaneously varies the timing and frequency of sample exposure as well as the overall data distribution encountered throughout training. Removing the Stability or Exploration stage also alters the aggregate sample distribution. As such, existing results do not sufficiently establish that the proposed sequence “early stabilization — intermediate learning — late exploration” itself is the critical factor.
Questions for authors
  1. Are the high- and low-RCD subsets matched in their actual BE, initial success rate, sequence length and problem-type distributions? Do the conclusions hold under stricter matching?
  2. Does Figure 3 always use the same probe and trajectories to compute head importance? Is the overlap of identified component sets stable when switching probes or using bootstrap resampling?
  3. Does the dynamic scheme retain its advantage when the total samples and their frequencies are kept identical and only the curriculum order is shuffled?
  4. Could the authors provide conditional reasoning validity rates grouped by method, cross-Judge rankings, and seed variance for the main results?
  5. The total number of epochs, actual training steps, checkpoint selection rules, and the training configuration for DeepSeek-R1-Distill-Llama-8B need clarification. The training parameter section in Appendix A currently explicitly describes only the two Qwen models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.