- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper identifies a consistent recognition-utilization gap in LLM-based graph prediction and links it to marginal response compression in softmax-normalized attention. It proposes SAIL, a utilization-alignment framework leveraging unnormalized structural accumulation and residual score correction. SAIL improves graph prediction performance and complements existing presentation methods, yet shows limited extrapolation to unseen multiplicities.
Critical strengths
- Paradigm-shifting conceptual contribution: It decouples "presentation alignment" from "utilization alignment", breaking the field's long-standing path dependence on optimizing input formatting and attention allocation, and opens up a brand-new optimization dimension for graph-LLM research.
- Complete and rigorous argumentation: The work forms a closed logic loop: controlled diagnosis of the recognition-utilization gap via GRAPHMACC, theoretical explanation of marginal response compression, SAIL solution design, and complementarity verification with existing presentation methods.
Critical weaknesses
- Overstated mechanistic causality: The conclusion of marginal response compression is built on the idealized local exchangeability assumption. No controlled experiments on real LLMs confirm it as the dominant cause of the gap, and alternative factors like feed-forward layers and numerical reasoning deficits are not ruled out.
- Severe out-of-range extrapolation failure: SAIL only works reliably within the multiplicity range covered by training, and performance degrades sharply at unseen larger counts, making it hard to adapt to real-world graphs with heavy-tailed degree distributions. The root cause of this failure is also not deeply analyzed.
Questions for authors
-
To what extent is marginal response compression the dominant cause of the recognition-utilization gap in real LLMs, versus alternative factors such as feed-forward computation or numerical reasoning limitations?
-
What is the root cause of SAIL’s sharp performance degradation at unseen multiplicities? Does it stem from the QueryInt head, training distribution bias, or the unnormalized accumulation design itself?
3 comments
Response to Reviewer fkvJ (2/2)
W2 & Q2: What causes SAIL's sharp degradation at unseen multiplicities, and does it limit real-world use?
A. Thank you for your comments. The degradation comes from the QueryInt head, which ignores the magnitude relation between counts, rather than from unnormalized accumulation or the training distribution, and it has limited impact on real graphs. Following the setting of Appendix G.2 (Qwen3-4B, training with counts and candidates 1, , m, and testing extended to m+2), we examined the three possible causes you mentioned one by one. Here, "trained with candidates m+1, m+2" only extends the training candidates to m+2, while the training samples remain c m. "Linear probe" trains a linear regression to predict the count on the frozen accumulated state, using only c m. "Numeric-candidate readout" keeps the accumulated state unchanged and scores each candidate by its distance to the predicted count. The columns of the table below are the accuracies within the training range (c m) and on the unseen counts m+1 and m+2, and the fraction of unseen counts predicted as m, all averaged over m.
| Setting | c m | c = m+1 | c = m+2 | Unseen counts predicted as m (m+1 / m+2) |
|---|---|---|---|---|
| Untreated LLM | 54.4 | 24.1 | 78.8 | 8.6 / 3.0 |
| SAIL | 86.1 | 5.9 | 5.7 | 90.8 / 85.9 |
| SAIL, trained with candidates m+1, m+2 | 87.5 | 1.3 | 1.6 | 97.4 / 92.5 |
| Linear probe on frozen accumulated state | 99.9 | 96.8 | 81.9 | 2.9 / 0.0 |
| Frozen accumulated state + numeric-candidate readout | 99.8 | 95.7 | 80.4 | 3.9 / 0.0 |
First, on unseen counts, about 90% of the predictions of SAIL fall on m, the largest count within the training range. Second, on the frozen accumulated state, a linear probe trained only on samples with c m still correctly reads out 96.8% of the counts at m+1, showing that the accumulated state retains the information of unseen counts and that the problem does not lie in unnormalized accumulation itself. On the other hand, even when m+1 and m+2 are also used as candidates during training, the results remain nearly unchanged. Finally, keeping the accumulated state unchanged and only scoring the candidates by their numerical values, the accuracy on unseen counts recovers to 95.7% and 80.4% even with the same training on c m, showing that the problem does not lie in the training distribution itself either. Therefore, the extrapolation failure arises because QueryInt treats 1 to m as unrelated categories and is not aware of the magnitude relation between counts. This is also consistent with our theory: unnormalized accumulation makes each occurrence bring the same increment, but does not guarantee that SAIL can read out the magnitude relation within it.
For heavy-tailed degree distributions on real graphs, we computed, under the text-graph setting, the fraction of test nodes whose number of neighbors exceeds the maximum in training. This fraction is 0 on Cora, WikiCS, and arXiv, and 1.4% on PubMed, showing that the training data in the native tasks largely cover the count range at test time and that the extrapolation issue caused by heavy-tailed degree distributions has limited impact. For scenarios that require extrapolation to larger counts, we will explore magnitude-aware readouts in future work.
In the revised version, we make the following changes:
- add the above diagnostic table to Appendix G.2 and state the cause of the extrapolation failure;
- add the statistics of the test-node count range under the text-graph setting to Appendix G.2.
Thank you for your rebuttal. My concerns has been addressed, therefore I maintain my positve score.
Response to Reviewer fkvJ (1/2)
We thank the reviewer for the constructive comments. We have uploaded a revised version with all changes highlighted in blue, and below we address your concerns one by one.
W1 & Q1: Is marginal response compression the dominant cause of the recognition-utilization gap in real LLMs?
A. Thank you for your comments. No, compression is not the dominant cause but one of several, together with later components and numerical comparison limitations. We agree that Theorem 2.3 is an idealized local analysis. The original submission consistently describes compression as one possible contributor to the gap (Section 1, Section 5, and Appendix B) rather than the only or dominant cause. To answer "to what extent," we add controlled experiments on Qwen3-4B (as in our response to Reviewer ppCm Q4) from three aspects.
1. The condition for compression holds in a real LLM. Lemma B.2 shows that compression requires a_R > a_B, i.e., the single-occurrence attention weight of the target category is higher than that of the replaced category. We measured these two weights on GraphMACC.
About 77% of the samples satisfy a_R > a_B, and the ratio between the two increases with degree, showing that the condition for compression holds for most samples in a real LLM.
2. Normalization is one cause of compression, but later components such as feed-forward layers may also contribute. We measure compression by the decay ratio, defined as the geometric mean of the successive ratios between the hidden-state distances of adjacent counts at the answer position, where a value below 1 indicates compression. The intervention fixes the attention weight of each occurrence of the target category to its value at count 1 without renormalization, and is applied either at layer 21 only (8 heads) or at layer 21 and all subsequent layers (same heads), without the SAIL readout. The inputs are matched samples with d = 4 and d = 8 constructed by the GraphMACC generator.
The decay ratio of SAIL is measured on its accumulated structural state. After removing the normalization at layer 21, the decay ratio at that layer rises from 0.825 to 0.901, showing that normalization indeed causes compression. However, even when every layer from layer 21 on is de-normalized, compression still gradually reappears in later layers, and the decay ratio at the last layer is 0.769, comparable to 0.774 without intervention. Since the intervention changes only the attention normalization, this remaining compression comes from other components, possibly including the feed-forward layers. Meanwhile, de-normalization without a readout does not improve count accuracy (from 42.7 to 40.3), showing that the recovered count information still requires a readout to be utilized. SAIL retains count information through unnormalized accumulation together with a readout, raising count accuracy to 50.5 on the same inputs. The component ablations in our responses to Reviewer ppCm W1 & Q3 and Reviewer 8F43 W3 & Q3 further show that unnormalized accumulation brings gains in Overall and in the utilization-related tasks (Count and Compare) that cannot be replaced by supervised training of the head alone.
3. Numerical reasoning limitations also exist on comparison tasks. Keeping the untreated Qwen3-4B, the prompt, and the scoring unchanged, we only provided the true counts of each neighbor category of the center node before the question, and made predictions on GraphMACC.
After the counts are given, Count improves substantially, showing that the model has the numerical ability to read out counts, and that the difficulty on Count mainly comes from extracting counts from the serialized structure, which is consistent with the compression discussed above. In contrast, Compare remains close to chance, showing that the numerical comparison ability itself is also limited. Therefore, compression is one cause of the gap, but numerical reasoning limitations also play a role on comparison tasks.
In the revised version, we make the following changes: