- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper defines PADC, a conceptual framework for determining the correctness of reasoning traces beyond just final answer accuracy. PADC identifies potential failure points at different stages of processing: Preservation (P) tests whether task-relevant information is present in the external representations of the problem’s state; Access (A) tests whether the model can accurately read out the state; Downstream dependence (D) tests whether the recovered state exerts a causal effect on model behavior; Correct computation tests whether models display correct counterfactual behavior (based on manipulations of the recovered state). The authors then perform experiments based on each of the four PADC dimensions. They show that the same reasoning trajectory rendered as image or text can produce different results; they then show that erasing task-relevant information has variable effects across tasks. Finally, they show that models can read out task states without being able to compose them, and that task-relevant information can have a selective causal effect on model answers without necessarily supporting correct counterfactual behavior.
Critical strengths
-
The framework provides a useful taxonomy of the failure points in model reasoning: “is the problem’s state represented at all,” “can the model access it,” “do model answers depend causally on the accessed state,” and “does the model correctly perform computations over it” are an intuitive progression of questions that other researchers could adopt.
-
The authors generally take care in their experimental design (which is especially evident in the Appendix). Each experiment is thoroughly explained and attempts to isolate a single PADC property; they also perform cross-model replications and try a variety of tasks. The inclusion of visual CoT alongside textual CoT is interesting, and the difference in performance between the two (despite controlling for the information presented) indicates that access is an important factor in model success.
Critical weaknesses
-
While the framework itself is clear and could make for a strong contribution, the properties are mostly treated as independent throughout the experiments, when they actually depend on each other. For example, downstream dependence is only meaningful if the model can access the state; access only really makes sense when the information is preserved. Each property is tested on different items, tasks, and sometimes different models; no task instance is traced through all four stages, meaning that failures at a given stage can’t really be separated from failures at the upstream stages. Furthermore, because each property is tested using behavioral readouts, there is some degree of circularity; for example, how can preservation be accurately measured using model readouts if access and downstream dependence have not been established?
-
It isn’t clear that the experiments actually isolate the properties they purport to test. Preservation seems to be defined as a property of the reasoning trace artifact rather than a property of the model. Since the traces are not actually generated by the model, it seems that Preservation holds by construction. Furthermore, Preservation is measured by how much the model relies on the erased information (correct me if I’m wrong); this just seems like another form of Downstream dependence to me rather than a measure of whether task-relevant information survives in the trace. The Access experiments don’t necessarily measure actual model access; directly asking models to read out the state doesn’t necessarily tell us if the model can access that state while solving a task (since it’s an entirely different context, and models could read out the state without actually representing it during intermediate reasoning).
Questions for authors
Clarification requests:
- Can the authors clarify how Preservation differs from Downstream dependence? They seem to be testing very similar ideas in the experiments.
- Can the authors clarify their choice to ablate externally generated CoT / VoT to test for preservation, as opposed to probing for information in the model’s internal states while it solves tasks? I initially got the impression that Preservation would be about testing whether relevant information is present at all throughout the reasoning process. It’s a bit unclear to me how using the model to read ablated traces actually tests that idea.
Suggestions for future work:
- Can the authors provide a diagnosis using the full PADC framework on a single set of test items (i.e., with each stage conditioned on the success of the prior stage)? For example, the authors observe that VoT underperforms on BFS; could PADC be applied there to explain why?
4 comments
W2. It isn’t clear that the experiments actually isolate the properties they purport to test. Preservation seems to be defined as a property of the reasoning trace artifact rather than a property of the model. Since the traces are not actually generated by the model, it seems that Preservation holds by construction. Furthermore, Preservation is measured by how much the model relies on the erased information (correct me if I’m wrong); this just seems like another form of Downstream dependence to me rather than a measure of whether task-relevant information survives in the trace. The Access experiments don’t necessarily measure actual model access; directly asking models to read out the state doesn’t necessarily tell us if the model can access that state while solving a task (since it’s an entirely different context, and models could read out the state without actually representing it during intermediate reasoning).
Q1. Can the authors clarify how Preservation differs from Downstream dependence? They seem to be testing very similar ideas in the experiments.
The definition of P and its experimental procedure need to be explained at two separate levels:
-
Representation-level Preservation refers to whether the information and structure required by the task are retained in the external representation. For example, in BFS, we can check whether nodes, edges, and adjacency relations are fully preserved by comparing them with the task state. This is a property of the representation itself. It does not depend on the model's answer behavior.
-
Behavioral Consequence refers to the actual impact of this information on downstream predictions. We separately remove task-relevant and task-irrelevant structures. We then compare the changes in the model's support for the correct answer. If removing relevant structures causes a larger decrease, it indicates that these structures have an actual effect on the model's predictions.
The two are connected through the task-relevant structures being examined. The representation-level check determines whether the information is preserved. The erasure experiment measures the behavioral change after removing this information. Therefore, "independent of answer behavior" describes the representation-level definition of P, while relevant/irrelevant erasure provides behavioral evidence. Whether the information is fully preserved must be determined by comparing the representation with the task state.
D further examines whether the model can adjust its answer when the task state changes. After confirming that the information is preserved and passes the A check, we change the task state to another valid state. We then examine whether the answer probability changes in the expected direction and exceeds a predefined threshold.
For example, in BFS, the erasure experiment examines whether removing a critical edge reduces the support for the original correct answer. D examines whether changing the graph shifts the answer distribution toward the correct answer for the new state. These two measurements focus on different aspects: the effect of removing structures and the model's directional response to changes in the task state.
Q2. Can the authors clarify their choice to ablate externally generated CoT / VoT to test for preservation, as opposed to probing for information in the model’s internal states while it solves tasks? I initially got the impression that Preservation would be about testing whether relevant information is present at all throughout the reasoning process. It’s a bit unclear to me how using the model to read ablated traces actually tests that idea.
PADC studies the model–representation interface. It focuses on how task states are preserved in representations and how models read and use these states. Its main contribution is a diagnostic framework that connects four dimensions: information preservation, access, predictive dependence, and computational correctness. These stages examine different properties of the interface. The internal neural computation process of the model is beyond the direct scope of this work.
We use externally constructed CoT/VoT to keep the reasoning content fixed. The same canonical trajectory is encoded in two formats. We then apply targeted erasure interventions to specific relations and use matched control interventions. Preservation focuses on task-relevant information in the external representation. Under the intact condition, information preservation is guaranteed by construction. The erasure experiments measure how relevant structures affect model predictions. This setup allows us to compare different representation formats and the effects of their interventions. Freely generated model trajectories may also differ in correctness, completeness, and generation paths. These issues require separate investigation.
The differences between PADC and related work are summarized below:
| Related Research | Main Research Question | How PADC Differs |
|---|---|---|
| Causal Abstraction / Interchange Intervention(Lanham et al.,2023; Paul et al.,2024) | Do variables in a representation have a causal effect on model outputs? | PADC checks information preservation and access before intervening on task states. It also examines computational correctness. |
| Amnesic Probing (Tutek et al., 2025) | How does model performance change after specific information is removed? | PADC measures information preservation, state recovery, and predictive dependence separately. It distinguishes the effects of information removal from evidence of state recoverability. |
| Control Tasks (Jia et al., 2026) | Do probing results come from the target information or other cues? | PADC uses matched control conditions at each stage to test the validity of its diagnostic results. |
| CoT Faithfulness / Unfaithfulness (Wu et al., 2024; Hu et al., 2024) | Does the reasoning trace faithfully reflect the basis of the model's answer? | PADC separately examines state recovery, predictive dependence, and computational correctness. It also establishes a conditional diagnostic order. |
This diagnostic order also defines what each measurement means. A checks whether the model can recover the state from the prompt. After a sample passes A, D checks whether the model's prediction changes in the expected direction when the task state changes. C then checks whether the answers are correct under both the original and counterfactual states.
For example, a model may correctly read the adjacency relations in BFS but still give the original answer after a critical edge is changed. In this case, predictive dependence needs further examination. Alternatively, the model may change its answer but still produce an incorrect result. In this case, computational correctness needs further examination.
- Lanham, T., et al. (2023). Measuring Faithfulness in Chain-of-Thought Reasoning. arXiv:2307.13702.
- Paul, D., West, R., Bosselut, A., & Faltings, B. (2024). Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning. Findings of EMNLP 2024, 15012–15032.
- Tutek, M., Hashemi Chaleshtori, F., Marasović, A., & Belinkov, Y. (2025). Measuring Chain of Thought Faithfulness by Unlearning Reasoning Steps. EMNLP 2025, 9935–9960.
- Jia, J., Benton, J., & Easley, E. (2026). Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning. arXiv:2605.24286.
- Wu, W., Mao, S., Zhang, Y., Xia, Y., Dong, L., Cui, L., & Wei, F. (2024). Mind’s Eye of LLMs: Visualization-of-Thought Elicits Spatial Reasoning in Large Language Models. NeurIPS, 37.
- Hu, Y., Shi, W., Fu, X., Roth, D., Ostendorf, M., Zettlemoyer, L., Smith, N. A., & Krishna, R. (2024). Visual Sketchpad: Sketching as a Visual Chain of Thought for Multimodal Language Models. NeurIPS, 37.
Follow-up on Our Revised Manuscript
Dear Reviewer 3K78,
As the discussion period draws to a close, we would appreciate any feedback you may have on our responses. It would be helpful to know whether our clarifications and additional experiments have addressed your concerns, or whether any questions remain.
We understand that your time is limited and would be grateful for even a brief update. We would be happy to provide any further clarification.
Thank you again for your time and constructive feedback.
Best regards, The Authors
Thank you for your valuable and constructive comments. We reply to your concerns as follows:
We added a repair comparison experiment on 167 BFS instances. We compared single-stage repair strategies with repairs selected based on PADC diagnoses. The results are shown below:
PADC-guided repair increased the number of correct answers from 53 to 87. Accuracy improved by 20.36%. C-only repair was the best single-stage repair. PADC-guided repair produced 7 more correct answers, with an accuracy gain of 4.19%. These results support selecting repair stages based on diagnostic results. D-only repair performed worse than the original VoT. This shows that repairing a fixed stage does not always improve performance.
We also tracked the pass counts during sequential diagnosis and repair. Only instances that passed the previous stage, either before or after repair, proceeded to the next stage. The results are shown below:
Access was the main bottleneck under the current checks and repairs. After P repair, all 167 instances passed the Preservation check. However, only 15 passed the Access check before A repair. A repair increased this number to 56. Even after A repair, 111 instances still failed the check. Next, 56 instances entered D, and 45 passed after repair. These 45 instances then entered C, and 34 passed after repair.
Note: C follows the paired-state correctness criterion used in the paper. An instance must be answered correctly under both the original state and its corresponding counterfactual state to pass this check. Only 45 instances reached C because of the earlier checks. Of these, 34 passed the paired-state correctness check after repair. The other 11 had an incorrect answer in at least one state.
This does not mean that only 34 of all 167 instances were ultimately answered correctly. The first table reports single-state task accuracy across all 167 instances. PADC-guided repair yielded correct answers for 87 of them. The two measures cover different sets of instances and use different correctness criteria.
An instance may produce a correct answer on the original task but fail an earlier check. Such an instance does not proceed to C. An instance may also reach C and answer the original task correctly but fail on the counterfactual state. It would not pass the C check.