acceptodds
Under review as a conference paper at ICLR 2027

Curriculum Learning Teaches LLMs Long-Horizon Implicit Reasoning, Up To A Limit

Abstract

LLMs “reason” by externalizing intermediate steps as tokens through chain-of-thought (CoT), yet even frontier reasoning models hit abrupt performance cliffs on long-horizon tasks. CoT commits to one step at a time, and the model's implicit reasoning—reasoning within a single forward pass—sets how far the model can search ahead to decide the correct next move. Models with weak implicit reasoning are more likely to commit to wrong paths and must attempt to recover via backtracking, while models with strong implicit reasoning identify the correct proof strategy directly, shortening the CoT. However, prior work shows LLMs struggle to reason implicitly beyond shallow depths, with no relief from scale. We investigate whether curriculum learning can extend this depth by fine-tuning on gradually increasing vocabulary size and lookahead—the number of steps a model must search ahead to identify the correct next move. At matched compute, curriculum-trained models achieve lookaheads where no-curriculum training fails—Qwen3-0.6B reaches accuracy at lookaheads up to with curriculum, compared to without. However, progress slows with training compute, following a saturating exponential that suggests a maximum intrinsic lookahead to which models can learn to reason implicitly. Curriculum step sizes that are too large stall progression, larger models reach higher fitted ceilings, and pretrained initialization far outpaces training from scratch. Finally, the capability acquired on synthetic data transfers to natural language reasoning, with curriculum-trained models outperforming models trained without curriculum at matched compute on natural language reasoning benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
implicit reasoning, long-horizon reasoning, curriculum learning, large language models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Q8ceOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors explore whether curriculum learning can improve long horizon implicit reasoning in language models. They use a synthetic task, using a DAG for the structure and then translated into english, this allows them to control the generation of the training data. The curriculum gradually increases the maximum lookahead and the vocabulary of the attribute names in the data. They show this curriculum enables models to solve problems requiring greater lookahead, then those trained on a uniform distribution of lengths and vocabulary from the same dataset. They thoroughly investigate how multiple design choices affect their conclusions, including model size, initialisation and curriculum step size. They also demonstrate improvements on reasoning benchmarks compared without training with a curriculum.

Critical strengths
  • Strong benefits of curriculum learning: Section 4.1 provides clear evidence of the benefits of the authors' approach when compared to training without a curriculum. For a lookahead length of 96 and 128, there is a clear speedup of training with the curriculum. At length 256, the model with curriculum achieves 98%, whilst the model without curriculum only achieves 45% accuracy. These results show clear training efficiencies from their design, as well as increasing the model's capacity to correctly answer harder problems.

  • Thorough exploration of design choices: The authors dive deeply into investigating the impacts of the curriculum step size, vocabulary, model size, and the differences of their conclusions when comparing between its effectiveness fir starting from a pre-trained checkpoint and random initialisation. These findings and results are useful for future work on this topic.

  • Downstream Reasoning Accuracy: Section 4.6 extends the findings from being interesting in isolation about curriculum design for long horizon implicit reasoning, but shows that the benefits actually transfer to downstream reasoning tasks using Qwen3 1.7B. Although the benefits are spiky depending on the benchmark, this section adds relevance of the synthetic training findings to downstream reasoning tasks.

Critical weaknesses
  • Robustness Across Training Seeds: The experiments in the paper only use one training seed, which raises some concerns about the robustness of the gains and convergence times. For Figure 2 and Table 1, I am concerned that the different training seeds may change the reported speedup gains and also could change how the model with no curriculum (which currently plateaus at 45%) performs on the harder problems (L = 256).
Questions for authors
  • Could the authors perform their experiments across additional training seeds? This would address my robustness concern. Some important experiments to run over multiple seeds are Section 4.1, Figure 2, Table 1, and also the downstream reasoning in Table 4. If these could both be performed over multiple seeds, it would strengthen the paper.
4 comments
AuthorsOpenReview ↗

Response to Reviewer Q8ce (1/2)

We thank the reviewer for the thoughtful feedback. Below we address each concern in turn and attempt to implement the reviewer's suggestions. Our changes and additions to the paper that are most relevant to the reviews are highlighted in blue in the revised PDF. If any concerns remain after our response, we would be grateful if the reviewer could raise them, so that we have the opportunity to address them before the end of the discussion period.

1. The comparison in Section 4.1 (Figure 2 and Table 1) and the downstream results in Table 4 use a single training seed and may change with the seed.

We agree that a single training seed cannot show whether the advantage of curriculum learning is robust to training randomness. To address this, we repeated the comparison of training with and without curriculum at maximum lookahead L=256L=256 (Section 4.1, Figure 2) with two additional seeds (2345 and 3456) to stand alongside the seed in the original submission (1234), for a total of three seeds. We prioritized this comparison because we consider it the most impactful one in Section 4.1, since it is the only lookahead at which training without curriculum does not reach 98% at all in the given budget. We added a new Table 9 to Appendix C.1 (Lines 956-963), which reports each seed and the mean ±\pm standard deviation.

Table 9 (Appendix C.1). Training Qwen3-0.6B\texttt{Qwen3-0.6B} with and without curriculum at maximum lookahead L=256L=256.

Seed 1234 (paper)Seed 2345Seed 3456Mean ±\pm std
With curriculum: cumulative training compute (PFLOPs) to reach 98% eval259K284K267K270K ±\pm 13K
Without curriculum: evaluation accuracy at matched training compute (the compute at which training with curriculum ends for the same seed)45.2%52.2%48.4%48.6% ±\pm 3.5

In Section 4.1 (Lines 271-274), we revised the description of the L=256L=256 comparison to report these results:

...training without curriculum plateaus at $\sim$45-52$\%$ accuracy across seeds and never converges within the compute budget, while training with curriculum continues climbing to 9898\\%. Over three seeds, training with curriculum reaches 9898\\% after 270K ±\pm 13K PFLOPs (mean ±\pm standard deviation; Table 9).

Training with curriculum is consistent across seeds. At the same cumulative training compute, evaluation accuracy with curriculum differs between any two seeds by 2.3 points on average and by at most 6.9 points.

Training compute is matched within each seed, so the training run without curriculum is given the same cumulative compute at which the training run with curriculum ended for that seed. At that point, the model trained without curriculum is at 48.6±3.548.6 \pm 3.5\\% evaluation accuracy on average.

We revised Figure 2, which shows evaluation accuracy vs. cumulative training compute for training with and without curriculum at L∈96,128,256L \in \\{96, 128, 256\\}. Its L=256L=256 panel now shows all three seeds for both settings and their mean, each in the color of its setting.

We also agree that the speedups observed in Table 1 at the smaller lookaheads (L=96L=96-128128) may change with the training seed. Training without curriculum fixes the maximum lookahead from the start, so each entry in Table 1 for training without curriculum requires a separate training run. Replicating all of these runs with two additional seeds requires a large amount of compute, which was not possible within the length of the rebuttal period. For the camera-ready version, we will provide the full replication of Table 1. Given how consistent the L=256L=256 comparison is across seeds, where the compute to reach 9898\\% evaluation accuracy has a standard deviation of about 55\\% of its mean, we strongly suspect that the behavior reported in Table 1 will hold beyond slight seed-to-seed variation.

AuthorsOpenReview ↗

Response to Reviewer Q8ce (2/2)

For the downstream results in Table 4, we repeated the experiment with the same two additional training seeds to stand alongside the seed in the original submission. For each seed, we fine-tuned Qwen3-1.7B\texttt{Qwen3-1.7B} with curriculum learning to L=48L=48, following the procedure in Section 4.6, and without curriculum learning for the same training compute (137.1K, 155.0K and 128.0K PFLOPs for the three seeds). Table R1 reports accuracy as mean ±\pm standard deviation over seeds, and the increase in accuracy due to the curriculum for each seed.

Table R1. Downstream reasoning accuracy (%) for Qwen3-1.7B\texttt{Qwen3-1.7B} at L=48L=48 over three training seeds. Increase in accuracy is the accuracy with curriculum minus the accuracy without curriculum. *: the two models trained with the same seed differ significantly on the corresponding benchmark (two-sided exact McNemar test on paired items, p<0.05p<0.05).

BenchmarkWith curriculumWithout curriculumIncrease in seed 1234 (paper) / 2345 / 3456Mean increase
ProofWriter OWA51.1 ±\pm 3.342.0 ±\pm 2.4+14.3* / +4.3* / +8.5*+9.0
ProofWriter CWA66.7 ±\pm 1.759.8 ±\pm 5.6+12.8* / +1.4 / +6.5*+6.9
ParaRules53.2 ±\pm 2.447.4 ±\pm 3.1+4.7* / +3.0* / +9.7*+5.8
ZebraLogic-MC53.5 ±\pm 3.747.2 ±\pm 1.3+5.2* / +11.5* / +2.2+6.3
PrOntoQA58.2 ±\pm 1.754.0 ±\pm 2.0+5.1* / +3.3* / +4.3*+4.2
FOLIO46.5 ±\pm 0.840.7 ±\pm 0.8+4.4* / +5.4* / +7.4*+5.7
JustLogic53.7 ±\pm 0.447.2 ±\pm 1.3+7.4* / +7.5* / +4.9*+6.6
ProverQA46.6 ±\pm 1.648.3 ±\pm 0.9−1.4-1.4 / 0.0 / −3.7-3.7*−1.7-1.7
CLUTRR26.9 ±\pm 1.526.1 ±\pm 1.7+1.9 / −1.6-1.6 / +2.1+0.8
LogicBench-BQA73.2 ±\pm 0.570.5 ±\pm 1.5+2.4* / +1.8 / +3.8*+2.7
RuleTaker61.0 ±\pm 1.160.5 ±\pm 2.5+1.2 / −2.6-2.6* / +2.9+0.5
Web of Lies57.3 ±\pm 0.655.8 ±\pm 2.5+0.2 / +0.4 / +4.0*+1.5
LogiQA43.0 ±\pm 0.542.4 ±\pm 1.0−0.9-0.9 / +0.8 / +1.9+0.6
NLGraph44.1 ±\pm 0.544.5 ±\pm 1.7−2.5-2.5* / +1.2 / +0.1−0.4-0.4
Mean of 1452.5 ±\pm 0.549.0 ±\pm 0.8+3.9 / +2.6 / +3.9+3.5

In Section 4.6 (Lines 496-500), we replaced the single-seed summary with a summary of these results:

At L=48L=48, averaged over three training seeds, training with curriculum outperforms training without curriculum on 12 of the 14 benchmarks, with a mean accuracy of 52.552.5\\% against 49.049.0\\%. The increase in accuracy is positive in every seed on nine benchmarks, and it is statistically significant in every seed on ProofWriter OWA, ParaRules, PrOntoQA, FOLIO, and JustLogic. Accuracies on ProverQA and NLGraph are lower on average with curriculum.

The results are stable across training seeds. Averaged over the 14 benchmarks, the standard deviation of accuracy across seeds is 0.50.5 percentage points for the curriculum-trained model and 0.80.8 points for the model trained without curriculum, compared with an increase in accuracy of +3.5+3.5 points. In each seed, the curriculum-trained model outperforms the model trained without curriculum on at least 11 of the 14 benchmarks.

A paired bootstrap over evaluation items, pooling the three seeds, yields a 9595\\% confidence interval of [+3.0,+3.9][+3.0, +3.9] points for the increase in mean accuracy. After Holm correction over the 14 benchmarks, nine benchmarks show a significant increase in accuracy (ProofWriter OWA, ProofWriter CWA, ParaRules, ZebraLogic-MC, PrOntoQA, FOLIO, JustLogic, LogicBench-BQA and Web of Lies), and the other five show no significant difference.

Having replicated the downstream experiment with three training seeds, we also revised the paper as follows:

In Table 4 (Section 4.6, Lines 432-448), the accuracy values have been changed to be the average accuracy over three training seeds.

In Appendix F (Lines 1423-1440), we added a new Table 23, which reports the accuracy with and without curriculum and the increase in accuracy due to curriculum learning for each training seed.

Thank you for engaging with a thorough response and additional experiments! Overall, I consider my concerns largely resolved and thus I remain supportive of acceptance and will keep my original score.

AuthorsOpenReview ↗

Follow-up for Reviewer Q8ce: Fitted Ceiling with a Second Training Seed

We thank the reviewer for the continued support of our submission, and are glad to hear that we were able to largely address the concerns.

We wanted to provide some new results on the effect of the training seed on the fitted ceiling. We are repeating the s=1s=1 training of Qwen3-0.6B\texttt{Qwen3-0.6B} from Section 4.2 with a second seed (2345), with all other hyperparameters and the curriculum configuration held fixed. This training run has now reached L=158L=158, where a ceiling can already be identified, and we report its current fit below.

For the second seed, the saturating exponential also fits better than the power law (ΔAICc=204.4\Delta\text{AICc} = 204.4), with the fitted equation

L≈315.7(1−exp⁡(−(C/4.80×105)0.552))L \approx 315.7\Big(1-\exp\big(-(C/4.80\times10^{5})^{0.552}\big)\Big)

Its fitted asymptote is Lmax⁡≈315.7L_{\max} \approx 315.7 (9595\\% bootstrap interval [280,369][280, 369]), compared with Lmax⁡≈323.6L_{\max} \approx 323.6 ([307.6,345.4][307.6, 345.4]) for the original seed. The interval of the second seed contains the fitted asymptote of the original seed. Under the same Diebold-Mariano test (Appendix D.1), the saturating exponential fitted on the first 7070\\% of stages predicts the remaining stages significantly more accurately than the power law (p=0.046p = 0.046), as was the case for the original seed. In Section 4.3 (Lines 397-402) of the revised paper, we also extended the s=8s=8 training of Qwen3-0.6B\texttt{Qwen3-0.6B} to 1.19M PFLOPs and fitted both forms, and the saturating exponential again fits its trajectory better than the power law (ΔAICc=29.0\Delta\text{AICc} = 29.0), with Lmax⁡≈631L_{\max} \approx 631. Together, these results suggest that, for a given model, a lookahead ceiling exists under different curriculum step sizes and different training seeds. At the same step size, the value of the fitted ceiling is also largely consistent across training seeds.

For the camera-ready version, we will extend this training run to 1.5M PFLOPs and report the fitted asymptote Lmax⁡L_{\max} for both seeds.

Reviewer_YsTWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether curriculum learning can improve the long-horizon implicit reasoning capabilities of large language models (LLMs), defined as the ability to look ahead within a single forward pass to identify the correct next reasoning step. To this end, the authors propose a curriculum learning method based on a modified DeepRD synthetic dataset, which progressively increases both the lookahead depth and the attribute vocabulary size. They systematically investigate the effects of curriculum step size, model scale, and pretrained initialization on implicit reasoning capabilities. The experimental results demonstrate the effectiveness of the proposed method.

Critical strengths
  1. The proposed curriculum learning method is clearly presented, and its key design choices are analyzed relatively systematically. The method progressively increases both the lookahead depth of reasoning tasks and the attribute vocabulary size. The authors further investigate the effects of curriculum step size and vocabulary growth strategies on training performance, which helps clarify the role of curriculum learning in long-horizon implicit reasoning tasks.
  2. The experiments evaluate not only performance on synthetic reasoning tasks but also the transfer of the acquired capabilities to natural language reasoning tasks.
  3. This paper is easy to follow.
Critical weaknesses
  1. The main downstream experiments use only a single random seed, making it difficult to determine whether the results can be consistently reproduced across different training initializations, data samples, or optimization trajectories. Confidence intervals over evaluation samples cannot substitute for an assessment of the uncertainty arising from training randomness. The authors should use multiple independent random seeds in key experiments, particularly for the main comparisons between curriculum and non-curriculum training and for the fitted saturation curves. This would improve the reliability of the main experimental findings.
  2. The main experiments are based on relatively constrained synthetic reasoning graph structures, with training instances primarily consisting of a target path and a distractor branch. It therefore remains unclear whether the proposed method can effectively scale to reasoning tasks with higher branching factors and more complex search structures. Although the authors conduct ablation experiments related to the number of branches, the branch structure and attribute vocabulary size are varied simultaneously, making it difficult to disentangle the effects of these factors. The authors should further evaluate the method on different graph structures and branching factors while controlling for other variables.
  3. The paper discusses related work on implicit reasoning and curriculum learning, but its coverage of recent research on curriculum learning for LLM reasoning remains insufficient. For example, [1] investigates the impact of curriculum reinforcement learning from easy to hard tasks on LLM reasoning capabilities. The authors should incorporate a discussion of this work and clarify the distinctive contributions of the proposed method relative to existing curriculum learning strategies.

Reference

[1] Curriculum reinforcement learning from easy to hard tasks improves llm reasoning. International Conference on Learning Representations, 2026.

Questions for authors

Please refer to Weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer YsTW (1/5)

We thank the reviewer for the thoughtful feedback. Below we address each concern in turn and attempt to implement the reviewer's suggestions. Our changes and additions to the paper that are most relevant to the reviews are highlighted in blue in the revised PDF. If any concerns remain after our response, we would be grateful if the reviewer could raise them, so that we have the opportunity to address them before the end of the discussion period.

1. The downstream experiments use a single seed, so it is unclear whether the results reproduce across training runs.

1.1) Confidence intervals over evaluation items do not capture uncertainty from training randomness.

We agree that confidence intervals over evaluation items measure only the uncertainty from the evaluation set. To measure the uncertainty from training randomness, we repeated the downstream experiment with two additional training seeds (2345 and 3456) to stand alongside the seed in the original submission (1234). Since the seed also affects the data generator, the model is trained on a different sample of the synthetic data under each seed. For each seed, we fine-tuned Qwen3-1.7B\texttt{Qwen3-1.7B} with curriculum learning to L=48L=48, following the procedure in Section 4.6, and without curriculum learning for the same training compute (137.1K, 155.0K and 128.0K PFLOPs for the three seeds). Table R1 reports accuracy as mean ±\pm standard deviation over seeds, and the increase in accuracy due to the curriculum for each seed.

Table R1. Downstream reasoning accuracy (%) for Qwen3-1.7B\texttt{Qwen3-1.7B} at L=48L=48 over three training seeds. Increase in accuracy is the accuracy with curriculum minus the accuracy without curriculum. *: the two models trained with the same seed differ significantly on the corresponding benchmark (two-sided exact McNemar test on paired items, p<0.05p<0.05).

BenchmarkWith curriculumWithout curriculumIncrease in seed 1234 (paper) / 2345 / 3456Mean increase
ProofWriter OWA51.1 ±\pm 3.342.0 ±\pm 2.4+14.3* / +4.3* / +8.5*+9.0
ProofWriter CWA66.7 ±\pm 1.759.8 ±\pm 5.6+12.8* / +1.4 / +6.5*+6.9
ParaRules53.2 ±\pm 2.447.4 ±\pm 3.1+4.7* / +3.0* / +9.7*+5.8
ZebraLogic-MC53.5 ±\pm 3.747.2 ±\pm 1.3+5.2* / +11.5* / +2.2+6.3
PrOntoQA58.2 ±\pm 1.754.0 ±\pm 2.0+5.1* / +3.3* / +4.3*+4.2
FOLIO46.5 ±\pm 0.840.7 ±\pm 0.8+4.4* / +5.4* / +7.4*+5.7
JustLogic53.7 ±\pm 0.447.2 ±\pm 1.3+7.4* / +7.5* / +4.9*+6.6
ProverQA46.6 ±\pm 1.648.3 ±\pm 0.9−1.4-1.4 / 0.0 / −3.7-3.7*−1.7-1.7
CLUTRR26.9 ±\pm 1.526.1 ±\pm 1.7+1.9 / −1.6-1.6 / +2.1+0.8
LogicBench-BQA73.2 ±\pm 0.570.5 ±\pm 1.5+2.4* / +1.8 / +3.8*+2.7
RuleTaker61.0 ±\pm 1.160.5 ±\pm 2.5+1.2 / −2.6-2.6* / +2.9+0.5
Web of Lies57.3 ±\pm 0.655.8 ±\pm 2.5+0.2 / +0.4 / +4.0*+1.5
LogiQA43.0 ±\pm 0.542.4 ±\pm 1.0−0.9-0.9 / +0.8 / +1.9+0.6
NLGraph44.1 ±\pm 0.544.5 ±\pm 1.7−2.5-2.5* / +1.2 / +0.1−0.4-0.4
Mean of 1452.5 ±\pm 0.549.0 ±\pm 0.8+3.9 / +2.6 / +3.9+3.5

In Section 4.6 (Lines 496-500), we replaced the single-seed summary with a summary of these results:

At L=48L=48, averaged over three training seeds, training with curriculum outperforms training without curriculum on 12 of the 14 benchmarks, with a mean accuracy of 52.552.5\\% against 49.049.0\\%. The increase in accuracy is positive in every seed on nine benchmarks, and it is statistically significant in every seed on ProofWriter OWA, ParaRules, PrOntoQA, FOLIO, and JustLogic. Accuracies on ProverQA and NLGraph are lower on average with curriculum.

The results are stable across training seeds. Averaged over the 14 benchmarks, the standard deviation of accuracy across seeds is 0.50.5 percentage points for the curriculum-trained model and 0.80.8 points for the model trained without curriculum, compared with an increase in accuracy of +3.5+3.5 points. In each seed, the curriculum-trained model outperforms the model trained without curriculum on at least 11 of the 14 benchmarks.

A paired bootstrap over evaluation items, pooling the three seeds, yields a 9595\\% confidence interval of [+3.0,+3.9][+3.0, +3.9] points for the increase in mean accuracy. After Holm correction over the 14 benchmarks, nine benchmarks show a significant increase in accuracy (ProofWriter OWA, ProofWriter CWA, ParaRules, ZebraLogic-MC, PrOntoQA, FOLIO, JustLogic, LogicBench-BQA and Web of Lies), and the other five show no significant difference.

AuthorsOpenReview ↗

Response to Reviewer YsTW (2/5)

Having replicated the downstream experiment with three training seeds, we also revised the paper as follows:

In Table 4 (Section 4.6, Lines 432-448), the accuracy values have been changed to be the average accuracy over three training seeds.

In Appendix F (Lines 1423-1440), we added a new Table 23, which reports the accuracy with and without curriculum and the increase in accuracy due to curriculum learning for each training seed.

1.2) Key experiments, including the curriculum comparison and the fitted asymptotes, should use multiple seeds.

We thank the reviewer for this suggestion, and we agree that repeating experiments with additional seeds would help to bolster the comparison of training with and without curriculum and the results for the fitted asymptotes.

For the comparison, we repeated the Section 4.1 experiment at maximum lookahead L=256L=256 with two additional seeds for both settings, for a total of three seeds. We added a new Table 9 to Appendix C.1 (Lines 956-963), which reports each seed and the mean ±\pm standard deviation.

Table 9 (Appendix C.1). Training Qwen3-0.6B\texttt{Qwen3-0.6B} with and without curriculum at maximum lookahead L=256L=256.

Seed 1234 (paper)Seed 2345Seed 3456Mean ±\pm std
With curriculum: cumulative training compute (PFLOPs) to reach 98% eval259K284K267K270K ±\pm 13K
Without curriculum: evaluation accuracy at matched training compute (the compute at which training with curriculum ends for the same seed)45.2%52.2%48.4%48.6% ±\pm 3.5

Averaged over seeds, training with curriculum reaches 9898\\% evaluation accuracy after 270K ±\pm 13K PFLOPs. Training compute is matched within each seed, so the training run without curriculum is given the same cumulative compute at which the training run with curriculum ended for that seed. At that point, training without curriculum plateaus at 48.6±3.548.6 \pm 3.5\\% on average and never reaches 9898\\% evaluation accuracy within the compute budget.

We revised Figure 2, which shows evaluation accuracy vs. cumulative training compute for training with and without curriculum at L∈96,128,256L \in \\{96, 128, 256\\}. In its L=256L=256 panel, the trajectory for each seed is now a thin, semi-transparent line and the bold line is the mean trajectory across seeds.

For the fitted asymptotes, repeating the s=1s=1 training of Qwen3-0.6B\texttt{Qwen3-0.6B} with a second seed requires about 1.5M PFLOPs, which is not feasible within the rebuttal period. For the camera-ready version, we will train this second seed to 1.5M PFLOPs, refit the saturating exponential, and report the fitted asymptote Lmax⁡L_{\max} for both seeds. (Update: This training run has now reached L=158L=158, where a ceiling can already be identified, and we report its current fit in our follow-up comment.)

AuthorsOpenReview ↗

Response to Reviewer YsTW (3/5)

2. The experiments rely on relatively constrained synthetic graph structures.

2.1) It is unclear whether the method scales to higher branching factors and more complex search structures.

We agree that our training instances have a simple structure, a correct chain and a distractor chain. We focused on the simplest search structure deliberately. Our study asks whether LLMs can learn to reason implicitly to greater depths through curriculum learning, and a correct chain with one distractor chain is the simplest structure that requires “searching ahead”. With this structure, the complexity of an instance depends on a single quantity, the lookahead, which is the number of steps a model must search ahead to identify the correct next step.

The original submission shows that models trained on instances with three and four branches still advance through the curriculum, reaching L=72L=72 and L=64L=64 within 100K PFLOPs (Section 4.1). It does not, however, show whether the advantage of curriculum learning extends to instances with more branches.

To test this, we repeated the comparison of Section 4.1 on instances with four branches (b=4b=4 in the notation of Section 4.1). We fine-tuned Qwen3-0.6B\texttt{Qwen3-0.6B} on these instances with and without curriculum at maximum lookaheads L=80L=80 and L=128L=128. We added a new Figure 11 to Appendix C.2 (Lines 1040-1053), which shows evaluation accuracy versus cumulative training compute for these four training runs.

In Section 4.1 (Lines 308-328), we added the following summary of this comparison:

On instances with b=4b=4 and a fixed attribute vocabulary, training with and without curriculum follows the same pattern as with two branches (Appendix C.2, Figure 11). At L=80L=80, both reach 9898\\% evaluation accuracy, and training with curriculum is 1.76×1.76\times more compute-efficient. At L=128L=128, training without curriculum plateaus between 2222\\% and 3333\\% evaluation accuracy, while training with curriculum reaches 97.897.8\\%. The advantage of curriculum learning therefore extends to instances with more branches, and the qualitative gap appears at a smaller maximum lookahead than with two branches (L=128L=128 instead of L=256L=256).

A new paragraph in Appendix C.2 (Lines 1014-1018) reports the training compute of each of the four training runs. Table R2 compares training on instances with two branches and with four branches at L=128L=128.

Table R2. Training Qwen3-0.6B\texttt{Qwen3-0.6B} at maximum lookahead L=128L=128 on instances with two and four branches.

Two branches (Table 1)Four branches (Figure 11)
With curriculum98% after 81.8K PFLOPs97.8% after 390K PFLOPs
Without curriculum98% after 186K PFLOPsplateaus at 22-33% from 80K PFLOPs onward

With four branches, training with curriculum also needs more compute to reach the same lookahead (Table R2). This and the plateau at a smaller lookahead both suggest that training on instances with four branches is harder for the model.

More complex search structures, such as branches that fork again or paths that merge, are a natural extension of our study, and we leave them to future work.

Reviewer_CFAXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether curriculum learning can improve the ability of large language models (LLMs) to perform long-horizon implicit reasoning without generating intermediate chain-of-thought tokens. The authors propose a curriculum that progressively increases both the required lookahead depth and the attribute vocabulary size in synthetic proof-planning tasks derived from DeepRD.

Critical strengths
  1. Strong empirical evidence that curriculum learning improves long-horizon implicit reasoning. The main contribution is the substantial performance improvement over compute-matched non-curriculum training. The large accuracy gap at lookahead 256 provides convincing evidence that the proposed curriculum can overcome optimization difficulties that direct training encounters. The experiments across multiple target depths, together with ablations on vocabulary growth and curriculum step size, further strengthen this finding.

  2. Systematic experimental analysis and evidence of downstream transfer. The paper goes beyond demonstrating a single performance improvement by investigating curriculum step size, model scaling, initialization, and compute-dependent behavior across Qwen3 and Pythia models. The downstream evaluation on 14 benchmarks further strengthens the practical relevance of the study, with statistically significant improvements on several deductive reasoning tasks.

Critical weaknesses
  1. The evidence is insufficient to establish an intrinsic upper limit on implicit reasoning depth. A central claim is that models have a maximum intrinsic lookahead, supported by fitting a saturating exponential to the relationship between training compute and achieved lookahead. Although this model provides better fits and held-out predictions than the tested power-law baseline, these results do not establish the existence of a fundamental reasoning limit. First, the fitted asymptotes are extrapolated from finite training trajectories, and several models have not approached their predicted ceilings. Second, the apparent ceiling may depend on the training procedure, including curriculum step size, optimization settings, and data distribution. Indeed, the paper shows that changing the curriculum step size substantially affects the achieved lookahead at a fixed compute budget. Furthermore, the per-stage iteration analysis does not identify a finite ceiling for Pythia-2.8B. Consequently, the evidence supports diminishing returns under the investigated training configurations, but the stronger interpretation of a model-specific intrinsic reasoning limit remains speculative. This weakens one of the paper's central conceptual contributions.

  2. Downstream generalization is not sufficiently established, particularly regarding robustness and reasoning complexity. Although curriculum learning outperforms the non-curriculum baseline on 11 of 14 downstream benchmarks, the evaluation has several limitations. First, all downstream results are obtained from a single training seed. The reported statistical tests account for evaluation-item variability but do not establish robustness to training randomness. Second, many evaluated benchmarks involve synthetic or structured logical reasoning closely related to the training task. Improvements on these benchmarks do not necessarily establish transfer to more complex, open-ended reasoning problems. Moreover, the gains are not consistently monotonic across curriculum stages, and the instruction-only control outperforms the final curriculum model on several benchmarks. Finally, most downstream tasks are evaluated through constrained answer likelihoods rather than free-form reasoning. While this evaluation protocol is reasonable for isolating answer accuracy, it does not directly demonstrate that improved implicit lookahead reduces chain-of-thought length, backtracking, or error accumulation in realistic long-horizon reasoning.

These limitations weaken the connection between the substantial improvements on the synthetic task and the broader claims about general-purpose reasoning capabilities.

Questions for authors
  1. Clarification regarding the intrinsic reasoning limit. How robust are the estimated maximum lookahead values to changes in curriculum step size, optimization hyperparameters, and training randomness? In particular, could the authors provide evidence that the apparent saturation persists under alternative curriculum configurations rather than reflecting the limitations of the specific training procedure? This would materially affect whether the fitted asymptotes can be interpreted as intrinsic limits rather than procedure-dependent estimates.

  2. Clarification regarding downstream transfer and robustness. Do the downstream improvements remain consistent across multiple independent training seeds? Additionally, could the authors analyze performance improvements as a function of the reasoning depth or complexity of downstream examples? Such an analysis would help establish whether increased implicit lookahead is responsible for the observed gains, rather than a more general effect of the curriculum on optimization or adaptation to structured reasoning data.

8 comments
AuthorsOpenReview ↗

Response to Reviewer CFAX (1/7)

We thank the reviewer for the thoughtful feedback. Below we address each concern in turn and attempt to implement the reviewer's suggestions. Our changes and additions to the paper that are most relevant to the reviews are highlighted in blue in the revised PDF. If any concerns remain after our response, we would be grateful if the reviewer could raise them, so that we have the opportunity to address them before the end of the discussion period.

1. Evidence for a ceiling on implicit reasoning depth is insufficiently established.

1.1) The fitted asymptotes are extrapolated from finite trajectories, so the results may only show diminishing returns.

We agree that some of our models end well below their fitted asymptotes, so for these models the fitted asymptote is an extrapolation. This is a matter of compute, since the cost of each additional stage of the curriculum grows steeply as the model approaches its fitted asymptote, and we could not train every model to that point within our available budget.

However, three of our models do closely approach their fitted asymptotes. Pythia-1.4B\texttt{Pythia-1.4B}, 410M\texttt{410M} and 160M\texttt{160M} achieve L=75L=75, 4646 and 99, against fitted asymptotes Lmax⁡L_{\max} of 81.981.9, 51.251.2 and 9.79.7. The original submission did not highlight this and left out the fit for Pythia-160M\texttt{Pythia-160M}, so we added the following to Section 4.4 (Lines 425-427):

Pythia-160M\texttt{Pythia-160M}, 410M\texttt{410M}, and 1.4B\texttt{1.4B} end training at 9393\\%, 9090\\%, and 9292\\% of their fitted asymptotes (Figure 13). Pythia-2.8B\texttt{Pythia-2.8B} ends at only 5757\\%, so its trajectory has not plateaued and its fitted asymptote is an extrapolation.

For Pythia-1.4B\texttt{Pythia-1.4B}, 410M\texttt{410M} and 160M\texttt{160M}, the last 10% of the stages consumes most of the total training compute: 2.75M of 5.0M PFLOPs, 1.19M of 2.0M PFLOPs and 0.67M of 1.0M PFLOPs, respectively. We added this result to Appendix D.2 (Lines 1308-1311):

In all three trajectories that end within 1010\\% of their fitted asymptotes, roughly the last tenth of completed stages consumes more than half of the budget: 5555\\% for Pythia-1.4B\texttt{Pythia-1.4B} (L=68L=68-7575), 5959\\% for Pythia-410M\texttt{Pythia-410M} (L=42L=42-4646), and 6767\\% for Pythia-160M\texttt{Pythia-160M} (L=9L=9). Qwen3-0.6B\texttt{Qwen3-0.6B} ends further from its fitted asymptote, at 7575\\%, but its per-stage cost also rises steadily, from 183183 PFLOPs per stage over L=1L=1-1919 to 22.322.3K over L=220L=220-244244.

In Lines 1298-1301 and 1313-1332, we replaced Figure 14, which showed the compute per completed stage only for Pythia-1.4B\texttt{Pythia-1.4B}, with a new figure containing results for Pythia-160M\texttt{Pythia-160M}, 410M\texttt{410M}, 1.4B\texttt{1.4B} and Qwen3-0.6B\texttt{Qwen3-0.6B}. In every panel of the new figure, the mean compute per stage increases throughout training and is highest in the last bar, which covers roughly the last tenth of completed stages.

Training therefore slows sharply as achieved lookahead nears Lmax⁡L_{\max}. To show this even more clearly, we added a new Figure 13 to Appendix D (Lines 1206-1220), which plots achieved lookahead as a fraction of the fitted asymptote against cumulative training compute for all five trajectories trained with s=1s=1. In this figure, each of the three trajectories rises steeply at first and then flattens just below its fitted asymptote.

Together, we feel these three models strongly suggest that the fitted ceiling is not an artifact of extrapolation.

We also tested the alternative that our results only show diminishing returns. Diminishing returns without a ceiling correspond to an unbounded form, in which achieved lookahead keeps growing with compute, only more slowly. In Section 4.2, we compare the saturating exponential with a power law, the simplest unbounded form. We also fit a shifted power law and a saturating power law, which contain a plain power law and a logarithmic curve, respectively, and the logarithmic form results in a worse fit than the saturating exponential on every trajectory. Because held-out errors along a single trajectory are serially correlated, we compare the power law and the saturating exponential with a Diebold-Mariano test on the number of training iterations needed to complete each stage (Appendix D.1). On the held-out stages, the saturating exponential makes significantly more accurate predictions for Qwen3-0.6B\texttt{Qwen3-0.6B} (p=7×10−10p = 7 \times 10^{-10}), Pythia-410M\texttt{Pythia-410M} (p=0.018p = 0.018) and 1.4B\texttt{1.4B} (p=1.5×10−4p = 1.5 \times 10^{-4}), and on all three trajectories its advantage grows with lookahead. It also makes significantly more accurate predictions for the new s=8s=8 trajectory of Qwen3-0.6B\texttt{Qwen3-0.6B} (p=0.034p = 0.034).

AuthorsOpenReview ↗

Response to Reviewer CFAX (2/7)

Our claim is that our results suggest that a ceiling exists, and the original submission describes it as a model- and procedure-specific ceiling (Section 4.2). To further elaborate on our findings, we perform additional experiments to show that the ceiling persists under a different curriculum step size. For the camera-ready version, we will continue training Qwen3-0.6B\texttt{Qwen3-0.6B} and Pythia-2.8B\texttt{Pythia-2.8B} with additional compute and report how close they come to their fitted asymptotes.

1.2) The ceiling may depend on the training procedure, such as the curriculum step size, the optimization hyperparameters and the training seed.

We agree that more experiments are needed to show that the ceiling persists across training configurations. We cannot test every curriculum design and hyperparameter setting, but we nevertheless conducted new experiments on two of the factors listed by the reviewer: the curriculum step size and the training seed.

For curriculum step size, we extended the s=8s=8 training of Qwen3-0.6B\texttt{Qwen3-0.6B}, fitted both forms to its trajectory (RMSE 5.345.34 for the saturating exponential versus 7.327.32 for the power law), and added the following paragraph to Section 4.3 (Lines 397-402):

The fitted ceiling depends on the step size. We extend the s=8s=8 training of Qwen3-0.6B\texttt{Qwen3-0.6B} beyond L=256L=256 to 1.19M PFLOPs, where it reaches L=368L=368, and fit both forms of Section 4.2. The saturating exponential again fits better than the power law (ΔAICc=29.0\Delta\text{AICc} = 29.0; Appendix D), with Lmax⁡≈631L_{\max} \approx 631, against ≈324\approx 324 for s=1s=1. Because large step sizes stall, as described above [a step size equal to the target lookahead is equivalent to training without curriculum, which stalls at L=256L=256 (Section 4.1)], the fitted ceiling cannot keep increasing with ss, and there must exist some intermediate step size that yields the highest ceiling. The existence of such a highest ceiling supports the hypothesis that the model has a maximum intrinsic lookahead.

In Appendix D (Lines 1230-1232), we also added a held-out comparison for the s=8s=8 trajectory:

When both forms are fitted to the training iterations per stage over the first 7070\\% of stages, the saturating exponential predicts the remaining stages significantly more accurately (p=0.034p = 0.034; Appendix D.1).

The original title of Section 4.3 only stated that small step sizes are required for the curriculum to progress. Since the fitted ceiling is higher with s=8s=8 than with s=1s=1, we felt that this title did not fully reflect the content of the section, and we changed it (Line 378) to "The Step Size Must Be Neither Too Large nor Too Small".

To elaborate on the effect of the training seed on the lookahead asymptote, we repeated the Section 4.1 training with curriculum to maximum lookahead L=256L=256 with two additional seeds (2345 and 3456) alongside the seed in the original submission (1234). We added the two new seeds to Figure 2, and a new Table 9 in Appendix C.1 (Lines 956-963) reports the cumulative training compute that each seed needs to reach 9898\\% evaluation accuracy. Since the seed also affects the data generator, the model is trained on a different sample of the synthetic data under each seed. The three seeds advance through the curriculum at nearly the same rate. At the same cumulative training compute, the achieved lookahead of any two seeds differs by 7.7 on average, which is a difference within one stage of the curriculum (s=8s=8). These results suggest that the fitted ceiling is likely invariant to the training seed.

Fitting the asymptote itself under a second seed or under different optimization settings requires a new s=1s=1 training run of Qwen3-0.6B\texttt{Qwen3-0.6B} that closely approaches the asymptote, which takes about 1.5M PFLOPs of compute and was not feasible within the rebuttal period. For the camera-ready version, we will repeat the s=1s=1 training of Qwen3-0.6B\texttt{Qwen3-0.6B} under a second seed and under different optimization settings, and report their fitted asymptotes alongside the current results. (Update: The training run with the second seed has now reached L=158L=158, where a ceiling can already be identified, and we report its current fit in our follow-up comment.)

AuthorsOpenReview ↗

Response to Reviewer CFAX (3/7)

1.3) The per-stage analysis does not identify a finite ceiling for Pythia-2.8B\texttt{Pythia-2.8B}.

We thank the reviewer for raising this point, and we agree that the trajectory of Pythia-2.8B\texttt{Pythia-2.8B} has not plateaued by the end of its training. It ends its training at L=133L=133 after 2.56M PFLOPs, which is 5757\\% of its Lmax⁡≈235L_{\max} \approx 235. Its fitted Lmax⁡L_{\max} is therefore an extrapolation, which Section 4.4 now states explicitly (Lines 425-427, quoted in our response to 1.1).

This is because larger Pythia\texttt{Pythia} models have higher fitted lookahead asymptotes and evidently need more compute to approach these asymptotes. With 1M PFLOPs, Pythia-1.4B\texttt{Pythia-1.4B} and 2.8B\texttt{2.8B} achieved 7171\\% and 4040\\% of their fitted asymptotes, respectively. Pythia-2.8B\texttt{Pythia-2.8B} is therefore still early in its training trajectory, where the power law and the saturating exponential fit about equally well. Pythia-1.4B\texttt{Pythia-1.4B} illustrates this pattern. When we fitted both forms to only the early part of its training trajectory, up to L=35L=35, the difference between them was small, with an RMSE of 0.390.39 for the saturating exponential versus 0.470.47 for the power law (a ratio of 1.21.2). Over its full trajectory, the gap grows to a ratio of 5.65.6 (0.530.53 versus 2.982.98), so the two forms become clearly distinguishable only late in training. The full Pythia-2.8B\texttt{Pythia-2.8B} trajectory resembles the early part of the Pythia-1.4B\texttt{Pythia-1.4B} trajectory, with RMSEs of 2.152.15 and 2.672.67, again a ratio of about 1.21.2.

Training Pythia-2.8B\texttt{Pythia-2.8B} until it reaches about 9090\\% of its fitted asymptote, as Pythia-160M\texttt{Pythia-160M}, 410M\texttt{410M} and 1.4B\texttt{1.4B} did, requires a large amount of additional compute, which is not feasible within the rebuttal period.

We also added the following explanation to Appendix D.1 (Lines 1281-1283):

This is expected for the largest model. Pythia-2.8B\texttt{Pythia-2.8B} ends its training at 5757\\% of its fitted asymptote Lmax⁡L_{\max}, well below the fractions reached by Pythia-160M\texttt{Pythia-160M}, Pythia-410M\texttt{Pythia-410M} and Pythia-1.4B\texttt{Pythia-1.4B}, so its trajectory does not yet distinguish between the two forms.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.