- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Reasoning DAG Probing, which probes LLM hidden states for node depth, pairwise distance, and adjacency in externally constructed reasoning DAGs. Across several tasks and models, the authors report that DAG-related information is most recoverable in intermediate layers, increases with model scale, and tends to emerge during generation. They also show that activation steering can improve some initially incorrect generations.
Critical strengths
- The paper presents a clear and potentially useful framework for studying structured reasoning beyond linear chain-of-thought.
- The experiments cover multiple tasks, model scales, layers, and generation stages.
- The layerwise and generation-time analyses provide informative empirical patterns.
Critical weaknesses
-
Recovering DAG labels from hidden states does not show that the model actually reasons according to the DAG. In ProofWriter, early nodes may systematically occur earlier in the input, while final conclusions occur later. Position, wording, or formatting cues could therefore predict depth without understanding dependencies. The authors should repeat the experiments after randomizing node order and controlling such superficial cues.
-
Some DAG annotations, especially for GSM8K, are generated by an LLM rather than directly provided by the dataset. The paper lacks sufficient human alignment, inter-annotator agreement, or annotation-quality analysis to establish that these DAGs faithfully represent the intended reasoning process.
Questions for authors
Same as weakness
3 comments
Follow-up on Input-Order Controls (W1).
While we previously argued that input-order cues are limited because intermediate conclusions and the final reasoning structure are not provided in the theory, we wanted to address your concern as directly as possible. We therefore added a randomized-order control: for each ProofWriter example, we shuffled complete theory sentences, recomputed hidden states from the shuffled input, retrained and re-evaluated the same set of probes. We report layers 24-28 because they are the strongest region for Qwen3-14B in the original setting.
| layer | original DAG-F1 | randomized DAG-F1 | original depth spearman | randomized depth spearman | original distance spearman | randomized distance spearman | original adjacency F1 | randomized adjacency F1 |
|---|---|---|---|---|---|---|---|---|
| 24 | 0.631 | 0.624 | 0.894 | 0.878 | 0.687 | 0.679 | 0.667 | 0.665 |
| 25 | 0.638 | 0.635 | 0.905 | 0.896 | 0.701 | 0.698 | 0.668 | 0.669 |
| 26 | 0.635 | 0.627 | 0.902 | 0.885 | 0.696 | 0.681 | 0.667 | 0.667 |
| 27 | 0.637 | 0.629 | 0.904 | 0.891 | 0.699 | 0.686 | 0.667 | 0.665 |
| 28 | 0.630 | 0.622 | 0.903 | 0.887 | 0.686 | 0.676 | 0.657 | 0.653 |
DAG-F1 and all three probe-specific metrics remain very similar after randomization. Given that the node-only, bag-of-words, and label-shuffled baselines reported in the paper are much lower, this suggests that the recovered structure is not explained by input sentence order.
Response to authors
I have no further concerns and will keep my score!
Thank you for your constructive review and for recognizing the quality of our paper. Your comments helped us identify where the paper would benefit from clearer explanation and additional validation statistics.
W1: Recoverability does not imply actual use.
You are right that probing alone does not prove that the model uses the recovered DAG structure. This is why we include generation-time steering as a causal test of behavioral relevance in Section 3.7. In addition to Section 3.7, we added matched controls to isolate whether the effect is DAG-specific: random norm-matched steering and shuffled-DAG-label steering.
Random norm-matched perturbations stay close to no-steer, suggesting same-sized activation changes are not sufficient to reliably improve nor damage generation. Shuffled-DAG-label steering is consistently worse than no-steer, indicating that corrupted structural targets can harm generation. Gold-DAG steering improves substantially, supporting the view that the recovered DAG structure is behaviorally relevant to the model's reasoning rather than only passively decodable.
W1: ProofWriter input order and superficial cues.
Your concern about superficial cues is reasonable, but the input-order cue is limited in our setting. The theory contains facts and rules, while sentences directly in the input are either distractors outside the gold DAG or peripheral premise nodes. Deeper intermediate conclusions and the final reasoning structure are not provided; they must be inferred by the model. Thus, sentence position in the theory does not directly reveal the node's reasoning depth. For wording and formatting cues, we include a lexical-only bag-of-words baseline. It remains substantially below the full contextual probe, suggesting shallow textual cues alone do not explain the recoverable DAG structure.
W2: GSM8K annotation quality.
Your concern about LLM-generated GSM8K DAG annotations is valid. Among 8,792 GSM8K datapoints, we randomly sampled 800 for manual validation. 744 annotations were directly correct; the remaining 56 differed from our validators' initial expected decomposition but were still self-coherent and correct reasoning DAGs for the solution. This suggests the GSM8K DAG annotations are reliable for our analysis, and we will report these validation statistics in the appendix.
In the uploaded PDF, we prioritized figure updates for visual inspection. All reported analyses, clarifications, and validation statistics will be incorporated into the final revision.