acceptodds
Under review as a conference paper at ICLR 2027

Where to Pause and How to Resume: Towards Human-Inspired Iterative Chain-of-Thought

22 views

Abstract

Iterative chain-of-thought (CoT) reasoning extends reasoning depth by periodically summarizing intermediate steps and resuming generation within a bounded context. However, existing iterative CoT approaches typically construct supervision through length-based segmentation, which can interrupt reasoning at arbitrary positions and provide limited guidance on how to resume from summaries. We propose PhaseCoT, a data-centric refinement that improves iterative reasoning along two dimensions: where to pause and how to resume. PhaseCoT segments reasoning trajectories at semantic milestones, so that each segment corresponds to a coherent reasoning phase, and further rewrites segment heads to introduce context-aware continuation from prior summaries. We construct supervised fine-tuning datasets for standard CoT, conventional length-driven iterative CoT, and PhaseCoT, and evaluate multiple models on MATH500, AIME22, and AMC23. Results show that PhaseCoT improves mathematical reasoning over both standard CoT and length-driven iterative CoT in most model-benchmark comparisons, which suggests that effective iterative reasoning requires not only longer reasoning horizons, but also semantically grounded pauses and context-aware resumptions.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Iterative Chain-of-Thought, Chain-of-Thought Reasoning, Large Language Models, Mathematical Reasoning, Supervised Fine-Tuning, Reasoning Data Construction, Long Chain-of-Thought, Test-Time Scaling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_MH66OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper propose PhaseCoT, that reduces the length iterative reasoning based on where to pause and how to resume. PhaseCoT segments reasoning trajectories at semantic milestones and rewrites segment heads to introduce context-aware continuation from prior summaries.

Critical strengths

None

Critical weaknesses
  1. The main idea of the paper is to make CoT "effective"; however, Table 1 shows that the model trained on the proposed dataset generates more tokens.
  2. The "semantic milestone" is not defined, and the LLM is asked to identify a milestone. There is no way to evaluate if the LLM has identified a milestone correctly. SiS_i includes the summaries of all the previous milestones, which might contain the incorrect explored path. Including that in the training dataset does not make sense, and summarizing milestones removes the self-reflection that the models do to update their reasoning.
Questions for authors
  1. How does the model perform if the model is instructed to generate small CoTs and be more effective?
  2. What is the accuracy on the datasets without the no-think mode? This will help to identify whether CoT is helping the model or not.
1 comment
AuthorsOpenReview ↗

Revised Manuscript: Changes Addressing Your Review (in Orange)

Dear Reviewer MH66,

We have uploaded a revised version of the paper. The change made in response to your comments is shown in orange and tagged [MH66]; changes for the other reviewers use different colors.

"Effective" versus "efficient" (W1). The Introduction now states explicitly that "effective" refers to reasoning accuracy: PhaseCoT does not aim to shorten iterative CoT, but to let it pause and resume in a more human-like way so that it reasons more accurately.

Two additions made for other reviewers also bear on your comments:

  • Cost of the additional tokens (W1). Section 6.2 and Appendix M (in magenta) measure inference memory and speed with Qwen3-14B on AIME22. Because each round runs in a bounded context, the iterative formats hold 33–34% less peak KV cache per generation than Vanilla and roughly double batched throughput. Because PhaseCoT generates more tokens, a single generation is 14% slower than with Vanilla.
  • Evaluating semantic milestones (W2). Section 6.4 (in magenta) trains a Random-Matched control. It has the same number of phases and matched segment lengths as PhaseCoT, but places the boundaries at random. PhaseCoT outperforms it by 7.43 points on average, which shows that placing the boundaries at semantic milestones drives the gains. This complements the boundary-quality audit in Appendix H.

Thank you again for your time. We hope the revision clarifies the goal of our work, and we would be grateful if you could take it into account in your final assessment.

Reviewer_Ki88OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes PhaseCoT, a data-construction refinement of length-driven iterative CoT such as InftyThink [1]. Instead of cutting reasoning traces every ~4K tokens, it uses an LLM to cut at "semantic milestones". It then summarizes each prefix, and rewrites the opening paragraph of each later segment so that it starts with a short recap of the problem and the summary. Models trained with SFT on 11.8K OpenR1-Math traces show higher average accuracy than both vanilla CoT and InftyThink on MATH500, AIME22 and AMC23 across six backbones. The paper also includes budget-controlled baselines, a 2×2 component ablation, and a fairly detailed audit of the LLM-generated data.

Critical strengths
  • The idea is simple and well motivated, and the paper is clearly written.
  • The controls are better than usual. In particular, the budget-dominated baselines (Table 2) and the Random-matched boundary control (Table 11) rule out several obvious confounds.
Critical weaknesses

W1. No length bound on segments. The segmenter prompt asks for "3–8 milestones" with no length limit, so segments grow with trace length. Fig. 3 shows AIME segments reaching 8–9K tokens, right at the 8192 per-round cap (Table 7). This undermines the bounded-context premise of iterative CoT.

W2. Gains may reflect format failures, not reasoning. Only 126/480 (InftyThink) and 209/480 (PhaseCoT) AIME22 generations terminate normally (App. H.1), and Sec. 5 analyzes only these. The 126 InftyThink survivors produce just 51 pauses, so the baseline seems not to have learned to pause. This would also explain why it falls below Vanilla on Qwen3-8B-Base (49.75 vs. 52.80).

W3. Unequal budget against Vanilla. Vanilla is capped at 16K tokens in total, while iterative methods get 8K per round over an unreported number of rounds. Vanilla truncation on AIME is likely non-trivial but is not reported. Fig. 5 varies k, not the per-sample cap, so it does not resolve this.

Questions for authors
  1. What are the max number of rounds and the total token budget for the iterative methods at evaluation? What fraction of Vanilla generations are truncated at 16K, and how does Vanilla perform with the same total budget?
  2. How are abnormally terminated generations scored in Table 1? Could you break down accuracy by normal vs. abnormal termination, and describe what failure mode InftyThink actually exhibits (no pause within 8192 tokens, or hitting the round limit)?
  3. What fraction of PhaseCoT segments exceed 4K / 8K tokens, at training time and at test time? Have you tried a hybrid rule, e.g., the nearest semantic boundary within a 2–6K window, falling back to a length cut?
2 comments
AuthorsOpenReview ↗

Revised Manuscript: Changes Addressing Your Review (in Blue)

Dear Reviewer Ki88,

We have uploaded a revised version of the paper. All changes made in response to your comments are shown in blue and tagged [Ki88]; changes for the other reviewers use different colors. We summarize below where each of your concerns is addressed.

Abnormal termination and format failures (W2, Q2).

  • Section 4.1 (new paragraph "Generation budgets and termination") defines abnormal termination and how such generations are scored: an answer produced before the length limit is extracted as usual, and all other abnormally terminated generations count as incorrect.
  • Section 5 now explains why the analysis uses the Base models. The Instruct models end almost every round normally: abnormal termination stays below 5% for every format, and PhaseCoT still has the highest average accuracy on all three Instruct backbones. The Base models, which have not been post-trained, more often run into the length limit, and this makes differences in convergence visible.
  • New Appendix J (Generation Convergence Rate) compares InftyThink and PhaseCoT on Qwen3-14B-Base with two metrics (Figure 6): the sample-level convergence rate, i.e., the share of generations in which every round ends normally, and the round-level convergence rate, i.e., the share of generations entering round r that finish it normally. PhaseCoT converges more often on all three benchmarks, and the gap widens with difficulty: 441 vs. 356 of the 640 AMC23 generations (68.9% vs. 55.6%), and 209 vs. 126 of the 480 AIME22 generations (43.5% vs. 26.3%). At the round level, PhaseCoT has the higher rate in every round except the fifth round on AMC23, which few generations reach. Both formats are trained on the same trajectories. Together with the Instruct results, this shows that the gains of PhaseCoT do not depend on format failures of the baseline, and that the PhaseCoT format is easier for the model to learn.
  • Appendix I gives the details: the behavior of Base and Instruct models, the Instruct-model comparison (47.06 vs. 44.97, 63.30 vs. 59.63, and 71.82 vs. 68.86), and the converged generations used in Section 5 and Appendix H.1.

Evaluation budgets (W3, Q1). Section 4.1 and Appendix I report the budgets. Vanilla uses 16K tokens; 32K brings no further gain. On AIME22 the truncation rate is about 17.1% for Qwen3-14B-Base and 1.67% for Qwen3-14B, and no generation is cut off while still reasoning productively. The iterative formats use 8K tokens per round with no round limit. Almost all generations end within five rounds, and a 16K per-round budget brings no further gain.

Segment length and the hybrid rule (W1, Q3).

  • Section 3.1 now documents that each CoT is segmented several times and the segmentation that best meets our requirements, including the segment length, is kept; no training segment exceeds 8K tokens.
  • Section 6.4 and Table 4 report the Hybrid control you suggested: the semantic milestone within a 2K–6K window, with a length-based fallback. Hybrid lies between InftyThink and PhaseCoT on every benchmark (63.14 vs. 61.47 and 66.66 on average). Construction details, and our discussion of the bounded context as a relative notion, are in Appendix L.

Thank you again for these suggestions, which helped us make the paper clearer and more complete. We would be grateful if you could take the revised version into account in your final assessment.

Area_Chair_wTuWOpenReview ↗

Reminder: Rebuttal Discussion Deadline

Dear Reviewer,

The discussion deadline is only a few hours away. Could you please respond to the authors' rebuttal before the deadline?

Best regards, Area Chair

Reviewer_CjLsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PhaseCoT, a data-centric approach to improving iterative chain-of-thought reasoning through two complementary components: semantic milestone-based segmentation and context-aware segment-head rewriting. Given long-form mathematical reasoning trajectories, PhaseCoT identifies semantically coherent reasoning phases, generates cumulative summaries of prior phases, and rewrites the beginning of each subsequent segment to support continuation from the summarized context. The authors construct supervised fine-tuning datasets from 11,776 examples and evaluate the approach against vanilla CoT and length-driven InftyThink on three mathematical reasoning benchmarks using six model configurations. The results show improved average accuracy across the evaluated backbones, while budget comparisons and component ablations provide evidence that the gains are not explained solely by additional training or inference computation. The study suggests that both semantically grounded stopping points and context-aware continuation can improve learned iterative reasoning.

Critical strengths
  1. A well-motivated refinement of iterative reasoning supervision. PhaseCoT addresses two concrete limitations of length-driven segmentation: pauses may occur before a reasoning phase is complete, and subsequent segments may lack the context needed to continue from compressed summaries. The combination of semantic milestone-based segmentation and context-aware segment-head rewriting provides a coherent and practically useful solution to these problems.

  2. A reasonably comprehensive empirical evaluation. The method is evaluated across multiple model backbones and mathematical reasoning benchmarks. Training-budget and inference-budget comparisons, together with component ablations, provide evidence that the observed improvements are not simply attributable to additional computation and that both proposed components contribute to performance.

Critical weaknesses
  1. The methodological novelty is meaningful but incremental. The general idea of organizing long reasoning into segments and continuing from compressed context builds on existing work on iterative reasoning and context management. The paper's more distinctive contribution is the combination of semantic milestone-based segmentation with explicit rewriting of subsequent segment heads. While the ablations support the usefulness of this design, the overall contribution is better characterized as a targeted refinement of existing iterative reasoning approaches than as a fundamentally new reasoning paradigm.

  2. Training robustness and preprocessing costs remain insufficiently characterized. The reported results are based on one training run per configuration, which limits the assessment of variability across training seeds. Moreover, the proposed pipeline relies on large external models for segmentation, summarization, and rewriting, but the total data-construction cost is not fully quantified. These limitations make it harder to assess the robustness and practical cost-effectiveness of the approach, although they do not directly negate the reported accuracy improvements.

Questions for authors
  1. Robustness across training seeds: Were the main results evaluated with multiple independent training seeds? If so, could the authors report the variability of the performance gains across runs? If not, how should the stability of the reported improvements be assessed given that the current results use one training run per configuration?

  2. Data-construction cost: Could the authors quantify the computational cost of constructing the PhaseCoT dataset, including the calls to the segmentation, summarization, and rewriting models? How does this overhead compare with the training and inference costs saved or incurred by the resulting models?

Reviewer_MXB8OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper revisits iterative chain-of-thought (CoT) reasoning under a bounded active context. Building on length-based iterative CoT, PhaseCoT uses LLM-identified semantic milestones to segment long reasoning traces and rewrites the opening paragraph of each subsequent segment to support continuation from a compressed summary. These transformations yield phase-level SFT examples that teach models when to summarize and how to resume. Across six Llama/Qwen backbones and three math benchmarks, the method generally improves accuracy over vanilla CoT and InftyThink.

Critical strengths
  1. A well-targeted problem formulation. The paper identifies two concrete deficiencies of length-based iterative CoT—interrupting incomplete reasoning phases and resuming from summaries without contextual re-grounding. Both are consequential for learning coherent iterative behavior.

  2. Consistent empirical gains with meaningful budget controls. PhaseCoT gives the best average accuracy on all six backbones; on Qwen3-14B-Base, it reaches 66.66%, versus 61.47% for InftyThink and 60.20% for vanilla CoT. Additional training-budget and test-time token-budget analyses make the improvement harder to explain by extra computation alone.

  3. Mechanistic evidence beyond benchmark accuracy. A 2×2 ablation supports independent benefits from semantic segmentation and rewriting. Boundary audits, a summary-conditioned resumption-gap metric, and test-time pause judgments substantiate that the trained model learns more semantically coherent stopping behavior.

Critical weaknesses
  1. Limited methodological novelty. The main contribution is a relatively simple LLM-based data transformation—semantic boundary annotation followed by paragraph-level rewriting—on top of an established iterative-CoT framework. Its empirical utility is clear, but the underlying methodological advance is modest.

  2. The motivating long-context advantage is not directly established. The evaluation demonstrates higher math accuracy but does not isolate tasks whose required reasoning exceeds a vanilla model's single-window capacity, nor report matched end-to-end latency or peak-memory savings. Thus, the claim about improved long-horizon reasoning under context constraints is stronger than the evidence presented.

  3. The causal role of semantic placement remains incompletely isolated. The random-matched segmentation control is evaluated for boundary quality, but not for downstream SFT accuracy. Since semantic segmentation also changes the number and lengths of training phases, the accuracy contribution of where boundaries are placed is not fully separated from changes in the training-instance distribution.

Questions for authors
  1. Can the authors train an otherwise identical model using the random-matched or paragraph-aligned segmentation control (with the same summary/rewrite pipeline) and report downstream accuracy? This would more directly isolate the value of semantic boundary placement.

  2. Can the authors evaluate explicitly beyond-single-window reasoning cases under comparable total inference budgets, and report peak KV-cache memory and wall-clock latency? This would establish whether the proposed mechanism addresses its motivating context-limit bottleneck in practice.

Comments on the paper

1 comment

Dear Reviewers,

I hope you're doing well.

Please respond to the authors' rebuttal if you have not already done so.

Thank you for your valuable time and engagement.

Best regards, Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.