acceptodds
Under review as a conference paper at ICLR 2027

TD-Grokking: Learning from Zero-Reward Problems by Training-Time Decomposition

Abstract

Reinforcement learning with verifiable rewards (RLVR) has advanced LLM reasoning, but it cannot learn directly from difficult problems for which every sampled solution fails: their uniformly zero rewards provide no contrastive training signal. We propose TD-Grokking, a training-time decomposition framework for learning from these zero-reward problems. Guided by a reference solution, it rewrites the local reasoning requirements of each original problem as self-contained subproblems with verifiable answers. Joint training on the original problems and their subproblems allows the model to learn from verifiable local signals while continuing to practice complete solutions. Across all 15 model–benchmark combinations spanning three model families and five mathematics and science reasoning benchmarks, TD-Grokking outperforms vanilla GRPO by 4.30% on average. It also consistently improves performance across four medical question-answering benchmarks. Further analysis shows that training on subproblems alone enables 77 of 507 previously zero-reward original problems to produce at least one verified correct solution, despite never training directly on those problems. These results support training-time decomposition as a practical way to obtain useful outcome-reward signals from difficult reasoning tasks. Our code and datasets are available at https://anonymous.4open.science/r/TD-Grokking-6567/.

Primary area:
reinforcement learning
Keywords:
Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VCPyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes TD-Grokking, which uses reference-solution-guided decomposition to turn empirically zero-reward problems into self-contained subproblems with verifiable answers, then jointly trains on the original problems and their subproblems using GRPO. Experiments across three model families and mathematical, scientific, and medical benchmarks report consistent improvements, while subproblem-only training and a dynamic variant examine transfer and training efficiency.

Critical strengths
  1. The paper addresses a practically relevant limitation of RLVR: the absence of informative rewards on consistently unsolved problems.
  2. The method is straightforward to implement and shows empirical gains across multiple model families and reasoning domains.
  3. The SFT, hint-based, generic-data, and subproblem-only controls provide useful evidence about training-time decomposition.
Critical weaknesses
  1. The main comparisons do not establish an advantage under comparable training budgets. Table 1 uses 3,788 training records for vanilla GRPO and 18,505 for TD-Grokking, but training updates, rollout tokens, and checkpoint-selection rules are not reported for the Table 2 baselines. This particularly weakens the comparison with QuestA, where several gains are relatively small.

  2. The reliability of generated subproblem answers is not quantified. Appendix A.3 describes format and compatibility checks rather than independent correctness validation, while Appendix D acknowledges incorrect targets and answer-normalization issues. Since these answers define the training rewards and diagnostic scores, a focused correctness audit is needed to assess the impact of such errors.

  3. The recovery diagnostic does not distinguish local-skill transfer from simplified-root practice. Listing 2 requires the final subproblem to retain the original target and explicitly supplies solution-derived intermediate results. The recovery of 77 previously zero-reward roots therefore supports transfer within the same problem families, but does not isolate the contribution of local-skill practice.

  4. Evaluation-set independence is insufficiently documented. Appendix D notes the absence of a documented overlap audit between training roots, generated subproblems, and evaluation benchmarks. Appendix A.6 also leaves the precise evaluation split and any model-selection use of the 1,000 PubMedQA items unclear, limiting confidence that the reported gains reflect held-out generalization.

Questions for authors

My main concern is the experimental validation, particularly the comparability of training budgets, evaluation-set independence, and whether the transfer analysis isolates the contribution of local-skill practice. Please address the Critical Weaknesses above.

Reviewer_NhRAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses problems where reasoning LLMs fail to produce any correct solution, and therefore has no training signal. It addresses this by prompting a stronger model with a reference solution to break down the problem into subproblems. It then trains on these subproblems and the roots. These models then are tested in math and science competition questions, as well as some medical questions. It beats ordinary training and a few competing alternative methods.

Critical strengths

It is a simple intervention that solves a real problem. There are plenty of problems in the real world that the LLM cannot solve, and this intervention is only changing the data on which it’s trained. Moreover, this improvement persists across the 3 models the authors tried, and it does better across different types of problems as well. Finally the authors compare against a couple of competing methods, and show it outperforms those.

Critical weaknesses

The main comparison feels a bit unfair, as the vanilla GRPO is trained only on roots with 0 reward, so beating that baseline just means that a model learns from learnable data better than from unlearnable data. It feels like it would be better to compare against training for the same amount of time on a similar number of questions that are of a more suitable difficulty. The MWP-GRPO control doesn’t feel like it covers this, as it just easy word problems that it probably always solves. It would also be great to see training on step questions from unrelated hard problems, to isolate how much the steps matching the roots really matters.

Also, by definition the model is given the answer on the last step question, so even in the steps only version the model is trained on providing the correct answer to the original hard root. So it seems like it would be valuable to isolate how much of the improvement comes from that.

Each method was also only trained once, so on 30 AIME questions the boost is roughly 2 additional correct answers. It’s not clear in the paper how the checkpoints were chosen, and how much of this can be explained by a good checkpoint. Moreover TD-Grokking gets almost 5x more prompts compared to the ones it is compared against.

Also, for the medical results, we see the exact same score of 39.59% for 3 differently trained models, which is unlikely due to chance. Also Base MedQA is scoring basically at complete random guessing level, depending on if it was the 4-option or 5-option variant. Also on PubMedQA the score is roughly the baseline in the sense of the bigger class has 55% of the answers and we see a score of 54%.

Finally, the Qwen3 technical report gives higher scores on AIME than reported here, I do not understand why this is the case.

Questions for authors

How much training did each method receive, what was the sampling procedure, and how was the checkpoint that was reported chosen?

How come, in the medical table, there are identical scores but different standard deviations? This seems extremely unlikely to happen by chance.

Why are the Qwen3 scores lower than the official ones?

Reviewer_4SoSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TD-Grokking, a training-time decomposition framework that tackles the zero-reward challenge in reinforcement learning with verifiable rewards (RLVR). The key idea is to break down difficult problems into smaller, self-contained subproblems with verifiable answers, allowing models to receive meaningful reward signals even when the original problems are too challenging to solve directly. TD-Grokking incorporates both original problems and their subproblems into training, while keeping the inference process unchanged. Experiments across different model families and reasoning benchmarks show that the proposed method consistently outperforms standard GRPO, achieving an average improvement of 4.30 percentage points. Notably, models trained exclusively on subproblems are able to solve 77 out of 507 previously unsolved original problems, suggesting that reasoning skills learned from simpler tasks can effectively transfer to more complex problems.

Critical strengths
  1. A simple and practical solution to an important challenge in RLVR. The paper addresses the zero-reward problem, a key bottleneck in RLVR where training provides little useful feedback when all sampled solutions are incorrect. TD-Grokking tackles this issue by introducing verifiable subproblems, while keeping the standard GRPO training procedure and final-answer rewards largely unchanged. The approach is straightforward and does not require additional components at inference time, making it a potentially general solution for improving RLVR on challenging reasoning tasks.
  2. Extensive experiments across models, benchmarks, and domains. The experimental evaluation is thorough, covering three model families and five mathematics and science reasoning benchmarks, with 15 model–benchmark combinations in total. The authors also extend their evaluation to four medical question-answering benchmarks, where the method continues to show consistent improvements. The breadth of these experiments, together with the positive results across different settings, provides convincing evidence of the method's effectiveness and generalizability.
  3. Interesting findings on knowledge transfer from subproblems to original problems. An appealing aspect of the paper is its investigation of whether reasoning abilities learned from subproblems can transfer to more difficult original problems. In particular, the subproblem-only training experiment shows that 77 out of 507 previously zero-reward problems become solvable, even though the model is never directly trained on these original problems. This observation provides concrete evidence that learning from simpler, verifiable subproblems can help models develop the ability to solve harder tasks, lending further support to the central idea of TD-Grokking.
Critical weaknesses
  1. Potential influence of knowledge distillation from stronger models. The proposed approach relies on a stronger external model to generate reference solutions when they are unavailable, while DeepSeek-V3.2-Exp is used to rewrite the subproblems. As a result, the training process incorporates additional data synthesized by stronger models, which may contribute to the observed performance gains. It is therefore unclear to what extent these improvements can be attributed to the proposed training-time decomposition strategy, rather than the knowledge transferred from stronger models. Further analysis would help clarify the contribution of each factor.
  2. Lack of comparison with closely related work. The paper would benefit from a more thorough discussion and comparison with From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning. Both approaches share the central idea of constructing verifiable subproblems to improve reinforcement learning for reasoning tasks. Given this conceptual overlap, a direct comparison would help clarify the key methodological differences and better establish the novelty and effectiveness of the proposed approach.
  3. Insufficient evidence for generalizable reasoning skill acquisition. The 77/507 experiment provides some evidence of transfer from subproblem training to the original root problems. However, the evaluation is conducted on root problems whose derived subproblems have already been used during training, rather than on entirely unseen root–subproblem families. Although the model is not directly trained on the root problems themselves, exposure to their associated subproblems may still introduce problem-specific information. Consequently, the current results do not fully distinguish generalizable reasoning improvements from gains driven by familiarity with particular problems. Evaluating the model on unseen root–subproblem families would provide stronger evidence that the proposed method promotes transferable reasoning skills.
Questions for authors
  1. It would be helpful to further clarify the respective contributions of training-time decomposition and the additional synthetic data generated by stronger models. Could the authors consider conducting ablation studies to better isolate these two factors?
  2. The distinction between TD-Grokking and From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning could be discussed in greater detail. Would it be possible to include a direct empirical comparison to better highlight the differences between the two approaches?
  3. To better understand the generalization capabilities of TD-Grokking, it would be valuable to evaluate its performance on previously unseen root–subproblem families. Such an evaluation could help determine whether the observed improvements stem from transferable reasoning abilities or from problem-specific knowledge learned through derived subproblems.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.