- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SLIDER, an interpretability framework based on Partial Information Decomposition (PID), to analyze the contributions of individual reasoning steps to the final answer in large language models. Specifically, the proposed method decomposes the answer-relevant information shared between the current reasoning step and preceding steps into redundant, unique, and synergistic information. Furthermore, it introduces step-level and trajectory-level metrics to quantify repetitive reasoning. Experimental results demonstrate the effectiveness of the proposed method in identifying repetitive reasoning steps and improving reasoning efficiency.
Critical strengths
- Compared with traditional mutual information or conditional mutual information, PID distinguishes redundant, unique, and synergistic information, providing a different perspective for understanding the contributions of individual reasoning steps.
- The paper evaluates the proposed method from multiple perspectives through theoretical analysis, synthetic data experiments, and evaluations of reasoning trajectories generated by real-world language models.
Critical weaknesses
-
Although the authors demonstrate a strong correlation between Trajectory-RRI and reasoning length, they have not sufficiently established whether this metric can accurately measure reasoning redundancy independently of reasoning length.
-
The reliability of PID estimation requires further validation. SLIDER heavily relies on accurate estimation of the PID components. However, the proposed method involves a relatively complex estimation pipeline, including generating up to 50 numerical variants using an LLM, encoding them with all-MiniLM-L6-v2, discretizing the continuous representations into 10 clusters using K-means, and estimating the joint probability distribution. The rationale behind each component of this pipeline requires further justification.
-
The effectiveness of Step-RRI in detecting repetitive reasoning requires more comprehensive evaluation. Additional evaluation metrics and stronger baselines should be included to better demonstrate the effectiveness of the proposed method.
-
The validity of the structure-preserving sample generation strategy requires further investigation. Since a single reasoning trajectory is insufficient to estimate the joint probability distribution, the authors use GPT-4o-mini or GPT-5-mini to generate multiple numerical variants while preserving the same reasoning structure. However, a concern is whether these generated trajectories can accurately reflect the actual reasoning behavior of language models across different problems. In practice, even when two problems share the same structure and differ only in numerical values, language models may follow different reasoning paths, perform different intermediate computations, and exhibit different repetitive reasoning behaviors.
Questions for authors
See Weakness
4 comments
Response to Reviewer SCTU: Part 2/2
On weakness 3: Additional evaluation metrics
To further validate the effectiveness of Step-RRI in detecting repetitive reasoning, we incorporated two additional evaluation metrics: (1) an LLM-as-a-judge repetitiveness score and (2) a human-annotated repetitiveness score. We evaluated example reasoning traces from both QwQ-32B and DeepSeek-R1-Distill-Qwen-32B.
We observed statistically significant positive Spearman correlations () between Step-RRI and both the LLM and human repetitiveness scores, outperforming the baseline InfoGain. These positive correlations further support the effectiveness of Step-RRI in identifying repetitiveness in reasoning trajectories. Appendix A.5 of the revised manuscript describes the shared scoring criteria used by the LLM judge and human annotators to rate step-level repetitiveness.
Table 3. Correlation between Step-RRI and step-level repetitiveness ratings.
| Method | Human Spearman’s ρ | Human p-value | LLM Spearman’s ρ | LLM p-value |
|---|---|---|---|---|
| −InfoGain | 0.424 | 0.055 | 0.388 | 0.082 |
| Step-RRI (ours) | 0.652 | 0.001 | 0.562 | 0.008 |
On weakness 4: Validity of structure-preserving sample generation
We agree that a model can follow different reasoning paths when the numbers in a problem change. However, our sampling strategy is designed to estimate PID for a fixed reference trajectory. The resulting PID estimates characterize how answer-relevant information is distributed across steps within the reference reasoning structure. If a model takes a different path when solving a variant independently, that path falls outside the scope of this estimate.
We use GPT-4o-mini or GPT-5-mini to generate numerical variants of each reasoning trajectory. The prompt templates in Appendix D.1 STRUCTURE-PRESERVING SAMPLE GENERATION instruct the models to change only numerical values, preserve the structure and order of the reasoning steps, and recompute intermediate results for each variant. We manually inspected a subset of the generated samples and refined the prompts to ensure that the resulting variants satisfied these constraints. We then discard variants whose step count differs from that of the original trajectory. Thus, we restrict each variant to numerical changes while preserving the reference trajectory’s reasoning structure.
Dear reviewer,
As the review period is coming to an end, we would really appreciate it if you could let us know whether our rebuttal has addressed your concerns. We are happy to respond to any additional questions. Thank you so much for your time and consideration.
Thank you for your response. The authors have largely addressed my concerns, so I have decided to raise my score.
Response to Reviewer SCTU: Part 1/2
We thank the reviewer for their thoughtful comments.
We have made several edits to the manuscript (marked in blue), including an analysis of SLIDER on PERL traces with natural errors (Section 5 and Appendix A.1), experiments on removing repetitive reasoning steps based on Step-RRI (Appendix A.2), a comparison of redundant information alone with Step-RRI (Appendix A.3), and experiments on how Trajectory-RRI contributes beyond reasoning length (Appendix A.4). We also report additional evaluation metrics (Appendix A.5), present additional sensitivity analysis (Appendix A.6), and expand the discussion of SLIDER’s limitations (Appendix B).
On weakness 1: Trajectory-RRI and reasoning length
We agree that the correlation between Trajectory-RRI and token count alone leaves open how much information Trajectory-RRI provides about redundancy beyond reasoning length. Though average Trajectory-RRI is positively correlated with average reasoning length, we argue that Trajectory-RRI can also capture differences in repetitiveness that length alone misses. A trajectory could be long due to the genuine demands of the problem, e.g., complexity (desirable) or due to unnecessary steps (undesirable), and these two aspects cannot be distinguished by looking at the total length alone.
On the other hand, Trajectory-RRI systematically captures how high the redundancy Red(A:S<i,Si) of the current step Si is in comparison to its informativeness (Uni(A:Si∣S<i) or Syn(A:S<i,Si)) about the final answer A.
We illustrate this with two examples in Appendix A.4.1 from the redundancy subset of PRMBench in which Step-RRI identifies the annotated redundant steps. Problem 1 requires more intermediate calculations and has a longer reasoning trajectory with only one annotated redundant step. Problem 2 has a shorter trajectory with more annotated redundant steps. In this pair, ranking the trajectories by Trajectory-RRI agrees with the annotated redundancy counts. Thus, a longer reasoning trace can contain fewer redundant steps than a shorter one.
On weakness 2: On PID estimation
We acknowledge that estimating PID involves a complex pipeline. Estimating PID terms is computationally challenging and requires nontrivial optimization. This task becomes even more difficult in the continuous high-dimensional domain. However, PID’s ability to quantify the unique, redundant, and synergistic contributions about a target has motivated its use in several machine learning applications involving high-dimensional real-valued data, including analyzing multimodal interactions [1], measuring the information spectrum of large vision-language models [2], and interpreting diffusion models [3]. In this work, we leverage PID to interpret reasoning traces and to quantify repetitiveness using a new measure, Step-RRI.
Each component in SLIDER makes the PID estimation tractable at the individual problem level. The sample generator produces variants that share the same reasoning structure. Next, we use an encoder to map text to embeddings so that semantically similar passages can be grouped even when their wording differs. Finally, we cluster the embeddings and estimate a joint probability distribution for each problem, which serves as the input to the PID optimization. We conduct two sensitivity analyses to assess how the number of generated variants and the number of clusters affect PID estimates.
Number of variants (Table 4, Appendix A.6): In the revised manuscript, we now include a sensitivity analysis of PID values for different numbers of variants N∈{50,100,500}. We observe that the dominant PID component remains the same at all sample sizes.
[From Previous Draft] Number of clusters (Table 5, Appendix D.2): We vary the number of clusters to assess the stability of the results. We observe that the patterns of redundancy, uniqueness, and synergy are largely preserved across different cluster choices.
In conclusion, SLIDER provides a practical framework for estimating PID on textual data at an individual problem level and can be extended to incorporate other PID estimators.
References:
[1]Liang, Paul, et al. "Multimodal learning without labeled multimodal data: Guarantees and applications." International Conference on Learning Representations. Vol. 2024. 2024.
[2]Xiu, Lixin, Xufang Luo, and Hideki Nakayama. "A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models." arXiv preprint arXiv:2603.29676 (2026).
[3]Dewan, Shaurya, et al. "Diffusion pid: Interpreting diffusion via partial information decomposition." Advances in Neural Information Processing Systems 37 (2024): 2045-2079.