acceptodds
Under review as a conference paper at ICLR 2027

Logic Density Pruning: Mitigating Overthinking via Step-Level Chain of Thought Compression

Abstract

Large reasoning models often improve on complex tasks by generating longer Chains-of-Thought (CoT). However, prolonged reasoning trajectories do not strictly guarantee improved utility. The phenomenon of generating redundant paraphrasing, loose explanations, and unproductive branches, which substantially inflates token usage and latency, is often termed *overthinking*. Prior work attempts to control reasoning length via prompt engineering, teacher-driven rewriting and distillation, or training-based constraints, but it commonly lacks an interpretable metric over reasoning steps that attributes each reasoning step to the final answer, making it difficult to reliably balance compression and fidelity. Compared to these methods, this paper proposes Logic Density Pruning (LDP), a CoT compression and training framework. Rather than relying on token-level importance, LDP computes step importance based on a unified criterion, *logic density*, which is constructed from two complementary scores at the step level—Mutual Information (MI) and attention citation. Based on this logic density, LDP performs reasonable CoT compression. To reduce the risk of mistakenly removing truly critical steps, this paper introduces a step perturbation check that verifies whether removing a candidate step induces a significant shift in the model’s subsequent generation distribution, and selectively recovers such steps. Notably, when the compressed CoTs are applied to Qwen3-8B, reasoning tokens are reduced by 80.2% (from 295.5 to 58.6) on CommonsenseQA, with only a 1.9% performance drop being observed.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Chain-of-Thought, Reasoning, Overthinking, Model Compression

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GRKoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This work proposes Logic Density Pruning (LDP), a method that reduces redundant reasoning in LLMs by assessing the importance of complete reasoning steps. LDP combines a kernel-based dependence score and an attention-based citation score for step evaluation, employs adaptive thresholding for pruning, and uses a Step Perturbation Check to safeguard potentially important steps. Evaluations on three benchmarks and four reasoning models across commonsense, strategic, and mathematical reasoning tasks demonstrate its effectiveness.

Critical strengths
  1. LDP evaluates the importance of complete reasoning steps by combining statistical dependence and attention-based evidence, with a step perturbation check to safeguard potentially important steps. This step-level design offers a principled alternative to token-level pruning and provides a practical means of reducing redundant reasoning while retaining answer-relevant information.

  2. This work evaluates LDP on three benchmarks spanning commonsense, strategic, and mathematical reasoning, using four reasoning models. The results demonstrate substantial reductions in reasoning length across multiple settings, often with relatively small accuracy degradation. This breadth of evaluation supports the practical relevance and generalizability of the proposed approach.

Critical weaknesses
  1. LDP combines a kernel-based dependence score with attention-based evidence and a hidden-state perturbation check. However, statistical dependence and attention weights do not necessarily reflect a step's causal contribution to answer correctness, while hidden-state changes do not directly establish whether removing a step harms reasoning. This limits confidence in the reliability of the pruning mechanism and the strength of the paper's central methodological claims.

  2. Although the experiments cover three benchmarks and four models, the results show varying accuracy changes across settings, including noticeable drops in some cases. Moreover, reductions in reasoning-token counts do not by themselves establish proportional improvements in end-to-end latency or computational cost. More comprehensive evaluation of answer reliability, inference efficiency, and robustness would strengthen the case for practical adoption.

If the authors address my concerns, I'll raise my score.

Questions for authors

Could the authors provide evidence that the Step Perturbation Check reliably identifies reasoning steps whose removal would affect answer correctness?

Reviewer_ydnPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Logic Density Pruning (LDP), a framework for compressing chain-of-thought training data and fine-tuning models to generate shorter reasoning traces. LDP combines a kernel-based representation dependence score with attention citation, applies adaptive thresholding to select reasoning steps, and uses a Step Perturbation Check (SPC) to recover potentially important deleted steps. Experiments cover four models and three benchmarks, showing substantial reductions in reasoning tokens with varying accuracy changes. The problem is relevant, and several compression results are promising. However, the validity of the central scoring mechanism, the effectiveness of SPC as a fidelity check, and the fairness and reproducibility of the experimental comparisons require further evidence.

Critical strengths
  1. Practical and relevant objective. Reducing redundant reasoning while preserving accuracy is important for efficient deployment. Using compressed traces as supervision provides a clear path toward inference-time savings.
  2. Reasonable choice of compression unit. Selecting complete reasoning steps can preserve textual coherence better than independent token deletion and makes the resulting traces easier to inspect.
  3. Promising results across multiple models. For example, Qwen3-8B on CommonsenseQA reduces average reasoning length from 295.5 to 58.6 tokens while accuracy decreases from 80.1% to 78.2%. Qwen3-14B on GSM8K reduces length from 626.1 to 308.9 tokens while accuracy increases from 86.6% to 87.2%.
  4. An attempt to control supervision length in ablations. Table 3 compares the scoring branches using approximately matched retained training-token budgets and a shared recovery procedure. This is a useful control for evaluating the fusion rule.
Critical weaknesses
  1. The validity of the representation dependence score is insufficiently established. Section 4.2.2 mean-pools each step and answer into a vector, then treats corresponding hidden coordinates as paired observations. It is unclear why coordinates within a single representation pair constitute appropriate statistical observations for measuring step–answer dependence, particularly when the representations may come from different layers. The paper appropriately acknowledges that this is neither an exact MI estimator nor a standard HSIC estimator. Nevertheless, its usefulness as a step-importance measure needs validation beyond downstream compression results. Comparisons with simple representation similarity, sensitivity to layer and bandwidth choices, and correlation with answer changes under deletion would help establish what the score captures.
  2. SPC does not directly establish answer fidelity. SPC compares the final-token hidden states of the next generated sentence with and without a candidate step. These continuations can correspond to different reasoning stages: without the step, the model may regenerate it, while with the step, it may proceed to the following one. Consequently, a large representation difference need not indicate that the deleted step is necessary for the answer, and a small difference does not guarantee answer preservation. Moreover, checks use the original prefix independently for each step, whereas the final trace can delete multiple steps jointly. The paper should validate whether SPC predicts final-answer changes and whether its decisions remain reliable under combined deletions.
  3. The ablations do not isolate key components or exclude simpler explanations. Table 3 compares MI + SPC, Attention + SPC, and Fusion + SPC, supporting the fusion rule under that setting. However, it does not isolate the contribution of SPC itself or compare against simple step-selection strategies. A fusion-without-SPC ablation and length-matched random or position-based selection would be particularly informative. These controls are needed to distinguish the value of the proposed importance criterion from the general benefit of training on shorter traces. The paper should also explicitly confirm that all relevant methods share the same initial shortest-correct-trace selection.
Questions for authors
  1. What justifies treating hidden coordinates as paired observations, and how does the resulting score compare with simpler representation-similarity measures? What answer text is encoded for scoring?
  2. How accurately does SPC predict final-answer changes or correctness degradation? Have you evaluated its behavior when multiple steps are removed jointly?
  3. Can you provide fusion-without-SPC and length-matched random or position-based selection baselines? Do all methods use identical initial trajectory selection?
  4. Can you report accuracy–length curves over multiple budgets and clarify how baseline compression settings were selected?
  5. Can you provide the complete data splits, per-run configurations, and truncation policy? In particular, how is answer supervision preserved when a training sequence exceeds 512 tokens?
Reviewer_2KF1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes Logic Density Pruning (LDP), which compresses structured chain-of-thought trajectories for supervised fine-tuning. It combines a kernel-based step–answer dependence proxy with attention citation scores, applies gated scoring and Otsu thresholding, and uses a Step Perturbation Check to recover selected steps before training on the compressed traces. Experiments across four models and three reasoning benchmarks report shorter outputs with varying accuracy trade-offs, alongside comparisons with existing compression methods and a matched-supervision-budget fusion ablation.

Critical strengths

Transparent reporting of mixed outcomes. The tables report accuracy losses and settings where competing methods perform better, alongside favorable results, making the limitations of the gains visible.

Critical weaknesses
  1. The central claim of identifying essential reasoning steps is not established. The preliminary analysis labels steps using LDP itself, without independently validating their redundancy. Readable CoT need not faithfully describe decision-making (Turpin et al.), and filler tokens can support computation in specifically trained models (Pfau et al.). SPC tests deletion effects on next-sentence representations, but their relationship to final-answer correctness is unvalidated. Compression gains therefore do not establish the proposed interpretation of logic density or reliable critical-step recovery.

  2. The added benefit and generality are insufficiently validated. Previous literature has explored step-level compression, including StepEntropy. The matched-supervision-budget ablation supports fusion, but covers one model, does not isolate SPC, gating or penalty effects, and does not match inference costs. The comparative accuracy–length trade-offs leave LDP's distinctive advantage unclear. The pipeline prescribes structured traces; despite its sentence-segmentation fallback, effectiveness on native unnumbered reasoning and robustness to segmentation choices are not established. Applicability to more recent reasoning models is also unverified. These results support compression within the tested pipeline, but do not sufficiently establish the added contribution or generality needed for acceptance.

Questions for authors
  1. SPC criterion. Does next-sentence representation distance predict changes in final-answer correctness, rather than differences in reasoning stage?

  2. Scoring necessity and stability. How sensitive are results to the Gaussian bandwidth of 5050 and the principal score thresholds? In a representative setting, how does the full rule compare with simpler fusion without VIP gating or the softplus penalty, and with a budget-matched alternative to Otsu thresholding?

  3. Trajectory-selection control. Do unpruned SFT and LDP use exactly the same shortest-correct source trajectories, questions and final answers? The selection policy already changes supervision quality and length before pruning.

  4. Ablation protocol and scope. What does “previous ablation” refer to, and how does the budget-constrained ablation relate to the default pipeline? Do fusion gains persist at comparable inference budgets and beyond Qwen3-8B, including on a more recent reasoning checkpoint of comparable size?

  5. Output-format dependence. With questions, source model and training budget controlled, how does LDP perform on native reasoning traces generated without the prescribed step template?

Presentation comments. These are separate from the assessment-changing questions above.

  1. Related-work positioning. The main text gives insufficient detail on the methodological differences from the closest step-compression methods.

  2. Implementation detail. Low-level implementation details interrupt the conceptual presentation, while the scoring rule's rationale remains insufficiently explained.

  3. Comparative table formatting. Table 2 follows its stated LDP-versus-baseline bolding rule, but this makes the best results across methods harder to identify.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.