- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposed Inferential Information Flow, a reasoning-aware diagnostic framework to mesures whether a large language model preserves selectivity toward proof-relevant evidence during the reasoning process. It demonstrated that the selectivity are emphasized in the middle stage but degraded afterward. With this insight, the authors proposed FRGI, i.e., factor-rule grouped intervention method to select evidences and rules adaptively follows a 'what-where-how' design. Experimental results on three typical QA datasets under three LLMs demonstrates that with the assistance of FRGI, the reasoning accuracy is substantially improved.
Critical strengths
This paper is well organized and well motivated. The proposed FRGI which does not require re-training and proof annotations is interesting.
Critical weaknesses
-
Are the five indicators, i.e., NCE-IIF, CLUB-stype probe, ValueIIF, ICA, and TripletAcc, independent and truly complementary? Would they be biased? They are proxy diagnostics inspired by mutual information. The direct information flow and the actual reasoning process are still uncertain.
-
The above five indicators can demonstrates the similarity, ranking of the proof-relevant information. However, representations collapse cannot be truly proved by merely these indicators.
-
It is suggested to add ablation studies on the reasoning process with or without FRGI and test whether the IIF and its five indicators are truly improved through FRGI.
-
Experiments on only three logical reasoning benchmarks is limited. More experiments on wider range of reasoning benchmarks should be given.
Questions for authors
How do the authors validate the IIF truly capture reasoning-relevant information loss and the proposed FRGI improves both these indicators and reasoning performance?
Response to Reviewer
Thank you for asking questions. We address the four concerns below and report the paired intervention results in Q1.
W1. Complementarity and interpretation of the diagnostics.
The five diagnostics capture different aspects of proof-evidence selectivity using a shared set of proof-span annotations. NCE-IIF measures how much contrastive score mass is assigned to proof spans. The CLUB-style probe measures how well similarity scores predict proof-span membership. ValueIIF measures selectivity in the value channel, ICA measures alignment with the centroids of proof spans and distractors, and TripletAcc measures how often a proof span ranks above a distractor.
Figure 8 shows that agreement among these diagnostics varies across backbones. On Mistral-7B, pairwise correlations across layers are at least 0.71. On Qwen3-8B, the correlation between ValueIIF and ICA is 0.06. The diagnostics thus provide complementary views of the representations within a shared evaluation setup. Their scores are interpreted relative to the annotated proof. When multiple proofs are valid, a span outside the annotated proof may still support another derivation.
W2. What the layerwise decline establishes.
We interpret the post-peak decline as reduced separation between annotated proof spans and distractors in the measured representations. Appendix C.4 shows this decline in raw ICA and TripletAcc profiles before aggregation, and Appendix D.5 examines it directly in hidden-state and value space. These results support a claim about evidence selectivity at the measured layers. The model's actual use of that evidence during deduction is a separate question.
W3. Ablation with and without FRGI.
We recomputed all five diagnostics on the same ProverQA examples under vanilla inference and full FRGI, using the same proof-span annotations and readout layers. Q1 reports the paired comparison alongside task accuracy.
W4. Benchmark coverage and transfer to FOLIO.
The main FRGI accuracy evaluation covers ProverQA, PrOntoQA, and ProofWriter. Appendix D.5 extends the raw evidence-separation analysis to JustLogic and LogicSkills, bringing the diagnostic evaluation to five benchmarks.
To evaluate FRGI on an additional reasoning benchmark, we ran Qwen3-8B on the combined FOLIO training and validation splits (N=1,208) using greedy decoding. We used each premise as a candidate evidence span and assigned fact and rule roles directly from the premise text.
Full FRGI reaches 56.46% accuracy, improving over vanilla inference by 4.31 percentage points and zero-shot CoT by 1.08 points. The cascade and value hooks apply updates on 28.48% and 25.17% of inputs, respectively.
Q1. How do we validate what the IIF diagnostics capture, and does FRGI improve them alongside task accuracy?
We use annotated proof spans and distractors to test whether the representations distinguish evidence relevant to the annotated deduction. To assess whether FRGI improves that distinction, we evaluate Qwen3-8B on the same ProverQA examples with and without FRGI under beam search (k=4). The table reports diagnostic scores averaged over layers 24 and 28. Gold proof-span annotations are used only for evaluation.
Full FRGI improves all five diagnostics at the measured readout layers. For example, TripletAcc rises from 0.543 to 0.824 on a 0–1 scale, indicating that proof spans rank above distractors more often. Task accuracy rises from 38.70% to 65.60% on the same examples. In this setting, the accuracy gain is accompanied by improved selectivity for annotated proof evidence across all five diagnostics.
Thank you again for your feedback. We hope the new ablation and FOLIO results address your questions about what the diagnostics capture and how FRGI transfers. We welcome any follow-up questions and would appreciate your reconsidering the rating.