acceptodds
Withdrawn from ICLR 2027

Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Long chain-of-thought reasoning (Long CoT) is now fundamental to state-of-the-art LLMs, especially in mathematical reasoning. However, LLM generation is highly sequential, and long CoTs lead to a high latency. We propose to train Divide-and-Conquer CoT (DC-CoT) to reduce the latency. With DC-CoT, the model can act as a director that identifies distinct subtasks that can be performed in parallel in its reasoning process, and then spawns workers to execute the subtasks. Our goal is to achieve high accuracy, with a low longest path length, which is a theoretical measure of the latency needed for the response. We start with a long CoT base model (DeepScaleR-1.5B-Preview), and first use SFT with a small curated demonstration set to initialize its ability to spawn workers in a certain format. Because SFT degrades the accuracy significantly, we design a multi-stage RL algorithm, with various data filtering strategies, to recover the accuracy while decreasing the longest path length. Across several benchmarks including AIME 2024 and HMMT 2025, DC-CoT achieves similar accuracy as DeepScaleR-1.5B-Preview while decreasing longest path length by 35-40%.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Reinforcement Learning, LLMs, Long CoT, Latency, Multi-agent

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_KcHDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper trains a long-CoT model to split parts of its reasoning across parallel workers. A single model acts as both director and workers. SFT teaches the parallel format, followed by four stages of RL that reward correctness and penalize longest path length (LPL). An optional high length penalty stage further reduces LPL. On six math benchmarks, the method reduces LPL by about 35–40% relative to the original model, with mostly small changes in accuracy. Experiments on AIME 2024 also report lower wall-clock latency.

Critical strengths
  1. The method gives useful accuracy–latency tradeoffs across several math benchmarks. The paper includes sequential baselines trained with a high length penalty. On AIME 2024, DC-CoT-HLP achieves 40.85% accuracy versus 39.03% for DSR-HLP-12K, with lower LPL (Table 1). Table 5 also reports lower actual latency, from 48.94 to 35.49 seconds. These results support a practical benefit in the tested setting.

  2. The paper carefully handles the different contexts used for worker generation and director aggregation. The training masks and position IDs in Section 4.1.2 match these two operations, while inference uses standard vLLM calls. This is a useful implementation detail for training a shared model to perform both roles.

Critical weaknesses
  1. The comparison does not isolate the benefit of the DC design from the additional training. The sequential HLP baselines receive 200 steps of further RL, while DC-CoT-HLP first receives SFT and 1,640 steps of RL before its 200-step HLP stage (Section 5.1 and Appendix I). Matching the final HLP stage is helpful, but does not control for this difference. Also, on AIME 2024, total output tokens fall from 9,540 for DSR-32K to 6,831 for DC-CoT, whose LPL is 5,974 (Tables 1 and 4). Much of the LPL reduction therefore comes with shorter outputs. A sequential baseline with comparable training is needed to establish the added value of the DC design.

  2. LPL leaves out costs that matter for this architecture. It counts the decoding path but omits additional prefill and scheduling costs. Section 4.1.2 explains why worker generation and director aggregation require different context representations, so worker KV caches cannot simply all be reused unchanged. Tables 5–6 partly address this concern by reporting wall-clock time, but only on AIME 2024, without a breakdown of these costs. This limits how confidently the LPL gains across the other benchmarks can be interpreted as actual latency gains.

Questions for authors
  1. With comparable SFT data and total training compute, does DC-CoT still improve the accuracy–latency tradeoff over a sequential model using the same RL and length-penalty schedule? A comparison on one existing benchmark would help separate the DC contribution from the effect of further training.

  2. For the same trained DC-CoT checkpoint, how does actual latency change when workers run in parallel versus one after another, with their input contexts kept identical? Please break down prefill, decoding, and scheduling time, and state the prefix-cache reuse settings. This would measure the gain from parallel execution and the overhead omitted by LPL.

Reviewer_AxBpOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary
  1. The paper proposes DC-CoT, which lets a long-CoT reasoning model act as a "director" mid-reasoning: it spawns 3 parallel "workers" to handle subtasks, then aggregates their outputs and continues reasoning. The goal is to reduce the longest path length (LPL), a proxy for latency.
  2. Training starts from DeepScaleR-1.5B-Preview. The authors first perform SFT on 3,411 sequential CoTs rewritten into the parallel format by Claude Sonnet 4.5. Since SFT noticeably degrades accuracy, they then apply a 4-stage RL procedure (DAPO → CISPO, switching data filtering from Include-Easy to Remove-Easy, and finally relaxing the length limit), followed by a high-length-penalty (HLP) stage, to recover accuracy and reduce LPL.
  3. Across 6 math benchmarks, the authors report that DC-CoT matches the accuracy of DSR-32K while reducing LPL by 35–40%, and that DC-CoT-HLP matches DSR-HLP-12K in accuracy while reducing LPL by about 20%.
Critical strengths
  1. The problem is well motivated. Latency of long CoTs becomes a pressing issue as token budgets grow, and introducing parallel reasoning on top of an RL-trained long-CoT model is a reasonable direction.
  2. The implementation is clean. Inference relies only on repeated calls to INFER and runs on unmodified vLLM, unlike Multiverse, which requires a modified inference engine. During training, worker tokens are duplicated and paired with a custom attention mask and position IDs, keeping training consistent with inference.
  3. Experiments are documented in detail. The authors report training phenomena such as DAPO's steadily increasing entropy and the accuracy plateau caused by Include-Easy, and provide entropy percentiles, per-stage curves, and simplified variants (Appendices G, H, K).
Critical weaknesses
  1. Limited methodological novelty. The fork-join structure, in which a director spawns sub-threads and resumes reasoning after they join, has already appeared in several prior works: Skeleton-of-Thought and APAR target instruction-following models; SPRINT and Multiverse use SFT to teach reasoning models to parallelize; APR and Parallel-R1 train parallel reasoning with RL; and the concurrent AsyncThink and NPR are also trained end-to-end with RL. The main differences in this paper are that it starts from an RL-trained long-CoT model and does not modify vLLM at inference time. The former is a difference in experimental setting and the latter is an engineering convenience; neither constitutes a new parallel reasoning mechanism. The real contribution lies in the training recipe, but it reads more like a tuning log of one specific training run than a generalizable principle.
  2. The multi-stage recipe relies on manual intervention, and its scalability is questionable. Every change (DAPO → CISPO, Include-Easy → Remove-Easy, relaxing L_max and L_cutoff) was made only after observing an accuracy plateau on the validation set, each time continuing from the latest checkpoint; the learning rate and batch size were even changed at step 200 within Stage 2. The overall approach is "SFT first hurts accuracy, then RL is tailored to repair it," and explanations such as DAPO's entropy increase being harmful are largely speculative (Appendix K even suggests the opposite, that the entropy increase in Stage 1 may be beneficial). All experiments use a single 1.5B model, only math problems, and a fixed number of 3 workers. The Limitations section claims that larger models will benefit significantly, but no experiment supports this, and there is no evidence that such a finely tuned recipe transfers to larger models and data scales.
  3. The latency gains come mainly from the length penalty rather than parallelism, and the model rarely truly divides and conquers. On AIME 2024, compared with DSR-HLP-12K, which was also trained with a length penalty, DC-CoT-HLP reduces LPL by only about 21% (6023 → 4740); most of the 35–40% reduction claimed in the abstract comes from shorter overall generation. In Appendix M, only 6 of 50 rollouts actually decompose the problem into subtasks, while most of the rest perform redundant verification in parallel. At larger batch sizes (Table 6, N = 16), DC-CoT is even slower than DSR-HLP-12K.
  4. Limited accuracy gains. Relative to DSR-32K, DC-CoT has lower accuracy on 5 of 6 benchmarks (e.g., −1.49 on Minerva Math), and the only benchmark where it is higher, AIME 2024, is the same set used for validation during training.
Questions for authors
  1. Larger and other base models. Can the main conclusions be reproduced on at least one larger model (e.g., 7B)? Would the same 4-stage switching still be needed, and can the switching points be determined in advance rather than by observing plateaus?
  2. Direct comparison with parallel reasoning methods. Can you compare against at least one of Multiverse, APR, Parallel-R1, or NPR using the same base model? Also, how does simply prompting DeepScaleR to spawn workers, without any training, perform? This would test the claim that parallel thinking must be learned through training.
  3. Role of the SFT stage. What are the accuracy and LPL of the model after SFT alone? Could the model learn to spawn workers directly through RL with a format reward, skipping SFT? Since the SFT data is rewritten by Claude Sonnet 4.5, how much of the gain comes from distillation from a stronger teacher model?
  4. Improvement in divide-and-conquer ability. Only 6 of 50 rollouts in Appendix M actually decompose the problem. Can you evaluate separately on the decomposable problems (10 of 50 by your count) to check whether the model really divides the work on those problems?
Reviewer_ga9yOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This work introduces DC‑CoT, a divide-and-conquer reasoning paradigm that enables intra-trace parallelism for long CoT. Instead of generating reasoning steps strictly sequentially, the model acts as a director that spawns independent worker sub-traces via special tags. Workers run in parallel without communication, and the director aggregates their outputs to continue solving the main problem. The training pipeline consists of SFT on re-formatted parallel reasoning traces followed by a multi-stage RL schedule switching from DAPO to CISPO with carefully designed sample filtering. The reward jointly optimizes answer correctness, format compliance and longest path length (LPL), a proxy for wall-clock latency. Evaluated on competition math benchmarks, DC‑CoT maintains or slightly improves pass@1 accuracy while reducing LPL by 35–40% and real inference latency by around 25% compared with a strong long-CoT baseline. On the whole, this work reads more like an engineering practice report rather than an academic paper.

Critical strengths
  1. Carefully engineered multi-stage RL recipe: The authors show that naive SFT hurts performance, and a staged transition from DAPO to CISPO paired with easy/hard sample filtering is critical to balance parallelism and accuracy. This is practically useful for RL on reasoning models and provides actionable recipes for follow-up work.
  2. Solid empirical gains on math benchmarks: Results on AIME24, AMC23 and HMMT demonstrate a favorable Pareto frontier: better or matching accuracy with substantially reduced latency. Ablations validate the importance of each RL stage and the length penalty.
  3. Clear qualitative analysis of worker usage: The paper distinguishes two major use cases of workers: decomposing independent subproblems and running separate verification paths. This characterization helps readers understand where the method gains its speedup.
Critical weaknesses
  1. Heavy dependency on a custom rollout engine is understated. DC‑CoT cannot be deployed with vanilla vLLM generation. When the director emits <spawn_workers>, the inference stack must intercept the tag, fork parallel worker requests, collect results and resume the director. Without this runtime intervention, the tags become plain text and the parallel mechanism collapses. The paper should quantify how much performance degrades under naive offline simulation and discuss deployment constraints more prominently.
  2. The benefit is limited by problem structure and worker isolation. Workers are stateless and cannot communicate with one another. This means the method only helps for problems that can be split into independent subproblems or redundant verification paths. For reasoning tasks with dense sequential dependencies, there is little room for parallel speedup. The paper does not systematically analyze the distribution of decomposable versus non-decomposable problems across benchmarks, which limits the generalizability claim.
  3. Limited task scope. All experiments are confined to competition mathematics. It remains unclear whether the paradigm transfers to code generation, multi-hop QA, or tool-using agents, where sub-task decomposition and aggregation have different failure modes.
Questions for authors

What is the difference between the proposed system and the agent+sub-agent system?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.