acceptodds
Under review as a conference paper at ICLR 2027

Serialization Is Not Enough: From Presentation to Utilization in LLM-Based Graph Prediction

Abstract

Graph serialization makes structural evidence explicit to large language models (LLMs), whereas this explicit presentation does not ensure that the evidence is reliably used for prediction. We study this distinction through degree profiles, which record category-specific neighbor counts. Constructing and using a controlled diagnostic GraphMACC, which separates recognition of category presence from calculation and comparison of category counts, we find a consistent recognition-utilization gap across LLMs and representative presentation-alignment methods. We further examine a possible contributor to this gap. Under a local exchangeability assumption, normalized attention exhibits marginal response compression, where successive occurrences of the same structural category induce progressively smaller changes in the aggregated representation, reducing the separation between adjacent counts. This motivates utilization alignment, which directly targets how presented structural evidence is accumulated and propagated for prediction. We instantiate this idea with SAIL (Structural Accumulation and Integration for LLMs), which preserves occurrence-level structural information through unnormalized accumulation and uses the resulting structural state to correct the original LLM candidate scores. Under mild conditions, we further show that the accumulated differences can be mapped to improved prediction margins for adjacent counts. Experiments substantiate that SAIL improves category-count utilization across LLM families and betters graph prediction across node-, edge-, and graph-level tasks. Its gains are also complementary to existing presentation-alignment methods and persist under fine-tuning and in-context learning.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
Graph Prediction, Large Language Models, Graph Serialization, Utilization Alignment, Structural Accumulation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_fkvJOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper identifies a consistent recognition-utilization gap in LLM-based graph prediction and links it to marginal response compression in softmax-normalized attention. It proposes SAIL, a utilization-alignment framework leveraging unnormalized structural accumulation and residual score correction. SAIL improves graph prediction performance and complements existing presentation methods, yet shows limited extrapolation to unseen multiplicities.

Critical strengths
  1. Paradigm-shifting conceptual contribution: It decouples "presentation alignment" from "utilization alignment", breaking the field's long-standing path dependence on optimizing input formatting and attention allocation, and opens up a brand-new optimization dimension for graph-LLM research.
  2. Complete and rigorous argumentation: The work forms a closed logic loop: controlled diagnosis of the recognition-utilization gap via GRAPHMACC, theoretical explanation of marginal response compression, SAIL solution design, and complementarity verification with existing presentation methods.
Critical weaknesses
  1. Overstated mechanistic causality: The conclusion of marginal response compression is built on the idealized local exchangeability assumption. No controlled experiments on real LLMs confirm it as the dominant cause of the gap, and alternative factors like feed-forward layers and numerical reasoning deficits are not ruled out.
  2. Severe out-of-range extrapolation failure: SAIL only works reliably within the multiplicity range covered by training, and performance degrades sharply at unseen larger counts, making it hard to adapt to real-world graphs with heavy-tailed degree distributions. The root cause of this failure is also not deeply analyzed.
Questions for authors
  1. To what extent is marginal response compression the dominant cause of the recognition-utilization gap in real LLMs, versus alternative factors such as feed-forward computation or numerical reasoning limitations?

  2. What is the root cause of SAIL’s sharp performance degradation at unseen multiplicities? Does it stem from the QueryInt head, training distribution bias, or the unnormalized accumulation design itself?

3 comments
AuthorsOpenReview ↗

Response to Reviewer fkvJ (1/2)

We thank the reviewer for the constructive comments. We have uploaded a revised version with all changes highlighted in blue, and below we address your concerns one by one.

W1 & Q1: Is marginal response compression the dominant cause of the recognition-utilization gap in real LLMs?

A. Thank you for your comments. No, compression is not the dominant cause but one of several, together with later components and numerical comparison limitations. We agree that Theorem 2.3 is an idealized local analysis. The original submission consistently describes compression as one possible contributor to the gap (Section 1, Section 5, and Appendix B) rather than the only or dominant cause. To answer "to what extent," we add controlled experiments on Qwen3-4B (as in our response to Reviewer ppCm Q4) from three aspects.

1. The condition for compression holds in a real LLM. Lemma B.2 shows that compression requires a_R > a_B, i.e., the single-occurrence attention weight of the target category is higher than that of the replaced category. We measured these two weights on GraphMACC.

Degreea_R / a_BFraction with a_R > a_B
42.570.75
84.660.80
167.350.76
All3.970.77

About 77% of the samples satisfy a_R > a_B, and the ratio between the two increases with degree, showing that the condition for compression holds for most samples in a real LLM.

2. Normalization is one cause of compression, but later components such as feed-forward layers may also contribute. We measure compression by the decay ratio, defined as the geometric mean of the successive ratios between the hidden-state distances of adjacent counts at the answer position, where a value below 1 indicates compression. The intervention fixes the attention weight of each occurrence of the target category to its value at count 1 without renormalization, and is applied either at layer 21 only (8 heads) or at layer 21 and all subsequent layers (same heads), without the SAIL readout. The inputs are matched samples with d = 4 and d = 8 constructed by the GraphMACC generator.

SettingL21L22L23L24LastCount acc.
Untreated LLM0.8250.8320.8150.7940.77442.7
De-norm. at L210.9010.8800.8430.8150.76741.0
De-norm. from L21 on0.9010.8810.8460.8270.76940.3
SAIL0.949----50.5

The decay ratio of SAIL is measured on its accumulated structural state. After removing the normalization at layer 21, the decay ratio at that layer rises from 0.825 to 0.901, showing that normalization indeed causes compression. However, even when every layer from layer 21 on is de-normalized, compression still gradually reappears in later layers, and the decay ratio at the last layer is 0.769, comparable to 0.774 without intervention. Since the intervention changes only the attention normalization, this remaining compression comes from other components, possibly including the feed-forward layers. Meanwhile, de-normalization without a readout does not improve count accuracy (from 42.7 to 40.3), showing that the recovered count information still requires a readout to be utilized. SAIL retains count information through unnormalized accumulation together with a readout, raising count accuracy to 50.5 on the same inputs. The component ablations in our responses to Reviewer ppCm W1 & Q3 and Reviewer 8F43 W3 & Q3 further show that unnormalized accumulation brings gains in Overall and in the utilization-related tasks (Count and Compare) that cannot be replaced by supervised training of the head alone.

3. Numerical reasoning limitations also exist on comparison tasks. Keeping the untreated Qwen3-4B, the prompt, and the scoring unchanged, we only provided the true counts of each neighbor category of the center node before the question, and made predictions on GraphMACC.

InputCountCompare
Original prompt36.7047.89
Original prompt + explicit counts87.0253.97

After the counts are given, Count improves substantially, showing that the model has the numerical ability to read out counts, and that the difficulty on Count mainly comes from extracting counts from the serialized structure, which is consistent with the compression discussed above. In contrast, Compare remains close to chance, showing that the numerical comparison ability itself is also limited. Therefore, compression is one cause of the gap, but numerical reasoning limitations also play a role on comparison tasks.

In the revised version, we make the following changes:

  1. state more clearly in Section 1, Section 2, and Section 5 that marginal response compression is one cause of the gap;
  2. add the above three experiments and their analysis to Appendix B, and state in Limitations that later components such as feed-forward layers and numerical comparison limitations may also affect the gap.
AuthorsOpenReview ↗

Response to Reviewer fkvJ (2/2)

W2 & Q2: What causes SAIL's sharp degradation at unseen multiplicities, and does it limit real-world use?

A. Thank you for your comments. The degradation comes from the QueryInt head, which ignores the magnitude relation between counts, rather than from unnormalized accumulation or the training distribution, and it has limited impact on real graphs. Following the setting of Appendix G.2 (Qwen3-4B, training with counts and candidates 1, …\ldots, m, and testing extended to m+2), we examined the three possible causes you mentioned one by one. Here, "trained with candidates m+1, m+2" only extends the training candidates to m+2, while the training samples remain c ≤\le m. "Linear probe" trains a linear regression to predict the count on the frozen accumulated state, using only c ≤\le m. "Numeric-candidate readout" keeps the accumulated state unchanged and scores each candidate by its distance to the predicted count. The columns of the table below are the accuracies within the training range (c ≤\le m) and on the unseen counts m+1 and m+2, and the fraction of unseen counts predicted as m, all averaged over m.

Settingc ≤\le mc = m+1c = m+2Unseen counts predicted as m (m+1 / m+2)
Untreated LLM54.424.178.88.6 / 3.0
SAIL86.15.95.790.8 / 85.9
SAIL, trained with candidates m+1, m+287.51.31.697.4 / 92.5
Linear probe on frozen accumulated state99.996.881.92.9 / 0.0
Frozen accumulated state + numeric-candidate readout99.895.780.43.9 / 0.0

First, on unseen counts, about 90% of the predictions of SAIL fall on m, the largest count within the training range. Second, on the frozen accumulated state, a linear probe trained only on samples with c ≤\le m still correctly reads out 96.8% of the counts at m+1, showing that the accumulated state retains the information of unseen counts and that the problem does not lie in unnormalized accumulation itself. On the other hand, even when m+1 and m+2 are also used as candidates during training, the results remain nearly unchanged. Finally, keeping the accumulated state unchanged and only scoring the candidates by their numerical values, the accuracy on unseen counts recovers to 95.7% and 80.4% even with the same training on c ≤\le m, showing that the problem does not lie in the training distribution itself either. Therefore, the extrapolation failure arises because QueryInt treats 1 to m as unrelated categories and is not aware of the magnitude relation between counts. This is also consistent with our theory: unnormalized accumulation makes each occurrence bring the same increment, but does not guarantee that SAIL can read out the magnitude relation within it.

For heavy-tailed degree distributions on real graphs, we computed, under the text-graph setting, the fraction of test nodes whose number of neighbors exceeds the maximum in training. This fraction is 0 on Cora, WikiCS, and arXiv, and 1.4% on PubMed, showing that the training data in the native tasks largely cover the count range at test time and that the extrapolation issue caused by heavy-tailed degree distributions has limited impact. For scenarios that require extrapolation to larger counts, we will explore magnitude-aware readouts in future work.

In the revised version, we make the following changes:

  1. add the above diagnostic table to Appendix G.2 and state the cause of the extrapolation failure;
  2. add the statistics of the test-node count range under the text-graph setting to Appendix G.2.

Thank you for your rebuttal. My concerns has been addressed, therefore I maintain my positve score.

Reviewer_8F43OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper distinguishes presenting graph structure to an LLM from using that structure for prediction. GraphMACC evaluates category recognition, counting, and count comparison, while a theoretical analysis studies marginal response compression in normalized attention. The proposed SAIL method sums node-level LLM representations outside the frozen backbone and uses a supervised head to correct candidate scores.

Critical strengths
  1. The problem formulation is useful. Separating category recognition from counting and comparison provides a more informative diagnosis than overall prediction accuracy alone.

  2. The diagnostic design and statistical reporting are thorough. GraphMACC controls degree and shares graph inputs across tasks, while covering multiple topologies, serializations, and node orders. Graph-level bootstrap resampling preserves dependencies between queries, and chance-adjusted metrics and counting errors complement exact accuracy.

  3. The trainable module is relatively small. SAIL freezes the backbone and trains approximately 1.06M/1.67M QueryInt parameters, while TrajAgg introduces no trainable parameters.

Critical weaknesses
  1. The independent value of LLM trajectories over simpler supervised structural representations remains unclear. Response only already reaches 92.85%/43.30%/57.97% on existence/counting/comparison, showing a substantial contribution from supervised prediction. Without controls using initial embeddings, attribute encodings, or comparable graph aggregation models, the benefit specifically attributable to LLM trajectories is not isolated (Tables 5 and 39–40).

  2. The native-task comparisons mix structural design with additional supervision. SAIL trains on native task labels, whereas the main presentation-alignment baselines retain the frozen prediction head and sometimes perform near chance. The absence of native-task Response only results makes it difficult to separate structural accumulation from supervised adaptation.

  3. The ablations do not fully isolate unnormalized accumulation. Mean aggregation reuses the fixed scale estimated from Sum, while No relation encoding also removes endpoint-membership information. The resulting drops therefore mix scaling and information changes and cannot be attributed solely to normalization-induced count compression (Appendix F.3).

Questions for authors
  1. Were alternatives to LLM trajectories, such as initial node embeddings or attribute encodings, evaluated with the same aggregation module, prediction head, and training budget?

  2. Was Response only evaluated on the native benchmarks using the same labels and training budget as full SAIL? If so, how did the results compare?

  3. Appendix F.3 states that Mean aggregation uses the structural scale estimated from Sum. Could the authors explain this choice and clarify whether the comparison was also evaluated with a separately estimated scale for each variant?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 8F43 (1/3)

We thank the reviewer for the constructive comments. We have uploaded a revised version with all changes highlighted in blue, and below we address your concerns one by one.

W1 & Q1: Do LLM trajectories outperform simpler structural representations under the same setup?

A. Thank you for your comments. Yes, LLM trajectories outperform both initial embeddings and attribute vectors under the same setup. We agree, however, that the original submission only compared SAIL with Response only and did not replace the LLM trajectories with simpler structural representations, and therefore could not isolate the contribution of the LLM trajectories. As suggested, we keep the aggregation module, prediction head, and training budget unchanged and replace only the node representations in the structural state, adding two controls using initial embeddings and attribute encodings, as well as a comparison with GNNs.

1. Initial embeddings. This control is consistent with our response to Reviewer ppCm Q8. On Qwen3-4B, h(0)h^{(0)} is the initial embedding at the LLM input layer, h(ℓ)h^{(\ell)} is the hidden state at layer 21, and uattnu_{\mathrm{attn}} is the attention output of the 8 selected heads at that layer. The experiment follows the ablation setting of Table 5 on GraphMACC. Best results are in bold.

Structural stateOverallExistenceCountCompare
Untreated LLM56.4184.6336.7047.89
Mean of three (SAIL)70.4192.6147.1071.53
Response only64.7092.8543.3057.97
h(0)h^{(0)} only63.8092.6742.6356.10
h(ℓ)h^{(\ell)} only67.3192.6444.9964.30
uattnu_{\mathrm{attn}} only66.6492.8044.1762.96
Mean of h(0)h^{(0)} and h(ℓ)h^{(\ell)}64.4292.6041.8758.79
Mean of h(0)h^{(0)} and uattnu_{\mathrm{attn}}66.8092.7844.7662.85
Mean of h(ℓ)h^{(\ell)} and uattnu_{\mathrm{attn}}67.4192.7745.0864.39
Concatenation of three70.1792.8146.6271.08

Using only the initial embedding (h(0)h^{(0)}) gives results comparable to Response only, showing that non-contextualized node representations cannot bring gains beyond supervised training itself. After introducing the LLM-contextualized trajectories h(ℓ)h^{(\ell)} and uattnu_{\mathrm{attn}}, the performance clearly improves, the mean of the three channels (Full SAIL) achieves the highest Overall, Count, and Compare, and every single channel is lower than the mean of the three channels. Therefore, the gain of SAIL beyond Response only comes from the LLM trajectories rather than from the aggregation module or the head itself.

2. Attribute encodings. On the native datasets, we replaced the LLM trajectories with the node attribute vectors provided by each dataset. The experiment is conducted on Qwen3-4B following the individual training setting in our response to Reviewer ppCm Q6, and reports accuracy on the four node classification datasets. Best results are in bold.

Structural stateCoraPubMedWikiCSarXiv
Untreated LLM68.8088.0076.2153.08
Attribute vectors73.4887.7880.3067.20
SAIL (LLM trajectories)75.1888.3280.7668.75

On all four datasets, SAIL with LLM trajectories outperforms the variant using attribute vectors. This shows that the gain of SAIL cannot be fully obtained from raw node representations with the same accumulation and head, and that the LLM trajectories provide information beyond the attribute vectors. On GraphMACC, the node attribute (color) directly determines the answer, so attribute encodings would amount to providing the label, and we therefore did not conduct this control on GraphMACC.

3. Graph aggregation models. As in our response to Reviewer ppCm W2 & Q2 & Q7, we added a GCN reference to Table 2 under the text-graph setting. For a fair comparison, GCN matches the receptive field of the text graphs, using a single layer on the node and edge datasets, where SAIL reads only the one-hop neighborhood, and multiple layers to cover the complete molecular graph. On GraphMACC, Cora, PubMed, WikiCS, arXiv, Citation2, and MolHIV, GCN obtains 93.10, 78.30, 75.58, 78.31, 62.93, 0.0503, and 71.57, while SAIL obtains 75.00, 72.16, 89.40, 78.76, 58.82, 0.4350, and 66.32 with Qwen3-4B, and 81.41, 73.96, 89.60, 79.16, 65.98, 0.3175, and 70.15 with Qwen3-8B. SAIL outperforms GCN on 7 of the 14 combinations, showing that, given the same structural information, SAIL is already comparable to models dedicated to graph-structured data.

In the revised version, we make the following changes:

  1. briefly describe in Section 4.3 the gain of LLM trajectories over initial embeddings and attribute vectors;
  2. add the control tables for initial embeddings and attribute vectors to Appendix F.3, with the corresponding analysis;
  3. add GCN to Table 2 and describe its setting in Appendix D.3.
AuthorsOpenReview ↗

Response to Reviewer 8F43 (2/3)

W2 & Q2: Do SAIL's native gains come from structural accumulation or from additional supervision, and was Response only evaluated on the native benchmarks?

A. Thank you for your comments. Both contribute, and structural accumulation brings gains beyond supervision. Response only was not evaluated on the native datasets in the original submission, and after adding it under the same labels and training budget, SAIL still outperforms Response only on 12 of the 14 combinations. To evaluate SAIL more accurately, we changed the native datasets to the complete text-graph setting following the comments of Reviewer ppCm W2 & Q2 & Q7, and supplemented the Response only control on the native datasets to separate the gains from supervised training of the head and from structural accumulation. Response only and SAIL use the same labels and training budget, and both follow the global training protocol in our response to Reviewer ppCm Q6. GraphMACC reports U, the four node classification datasets report accuracy, Citation2 reports MRR, and MolHIV reports AUC. Best results are in bold.

Qwen3-4B

MethodGraphMACCCoraPubMedWikiCSarXivCitation2MolHIV
Untreated LLM42.2968.8088.0076.2153.080.376459.86
Response only62.0572.5088.2478.0159.530.359563.18
SAIL75.0072.1689.4078.7658.820.435066.32

Qwen3-8B

MethodGraphMACCCoraPubMedWikiCSarXivCitation2MolHIV
Untreated LLM47.6270.5089.2076.1259.640.084057.46
Response only65.6373.0689.4478.4065.790.089368.22
SAIL81.4173.9689.6079.1665.980.317570.15

As shown in the table above, Response only improves over the untreated LLM on most datasets, but under the same supervision, SAIL still outperforms Response only on 12 of the 14 combinations, and per-sample paired tests show that the difference is significant (p < 0.05) on 9 of them. Therefore, the gains of SAIL on the native datasets cannot be fully explained by supervised training of the head, and structural accumulation brings additional improvement beyond supervision. In addition, under the text-graph setting, the presentation baselines are no longer close to chance (see our response to Reviewer ppCm W2 & Q2 & Q7). These methods involve no training, and we selected their hyperparameters on a shared validation set. More importantly, SAIL is not intended to replace presentation methods under equal supervision (Section 1), but to provide a complementary utilization perspective, which is supported by the combination results of SAIL with each presentation method under the text-graph setting (Table 3 of the revised version).

In the revised version, we make the following changes:

  1. add the Response only results on the native datasets and their paired tests against SAIL to Appendix F.3, and describe the experimental configuration;
  2. add the hyperparameters of each presentation baseline and the validation set used for their selection to Appendix D.3.
AuthorsOpenReview ↗

Response to Reviewer 8F43 (3/3)

W3 & Q3: Do the Mean and No relation encoding ablations confound scale and information changes?

A. Yes, both ablations mixed the two, but after separating them, unnormalized accumulation still contributes to count-related tasks. We agree that Sum and Mean share the structural scale estimated from the Sum fitting states, so the scale difference cannot be separated from the aggregation. The original submission used the shared scale to change only the aggregation, but overlooked the effect of the scale. We also agree that No relation encoding removes the endpoint-membership information.

1. Scale of the Mean ablation. As suggested, we let each variant estimate its scale only from its own fitting states. The scales of the two are then estimated in the same way, so the effect of the aggregation can be measured on its own, as shown in the table below.

VariantScaleOverallExistenceCountCompare
Full SAIL (Sum)Sum-fitted70.4192.6147.1071.53
Mean (original)Sum-fitted64.4092.4244.5156.26
Mean (revised)Mean-fitted65.7592.7346.0258.49

After each variant estimates its own scale, the results of Mean (revised) are largely consistent with the original ablation, and the related ablation conclusions remain unchanged.

2. Separating the relation encodings. We separated the two relation encodings and removed only R_count or only R_map, with all other settings identical to Full SAIL, as shown in the table below.

VariantOverallExistenceCountCompare
Full SAIL70.4192.6147.1071.53
w/o R_count70.3492.6646.7971.55
w/o R_count, Sum → Mean64.3792.5444.7855.80
w/o R_map66.0292.7047.1758.18
w/o both (original No relation encoding)66.0592.7047.0258.44

Removing only R_map clearly lowers Compare, close to the result of removing both, whereas removing only R_count leaves all tasks nearly unchanged. Therefore, the drop of the original No relation encoding mainly comes from the missing endpoint-membership information, reflecting the role of relation encoding rather than unnormalized accumulation, which is consistent with the attribution in Appendix F.3 of the original submission. On the other hand, both the accumulated magnitude of Sum and R_count carry neighbor-count information, so when only R_count is removed, the count information can still be retained by the accumulated magnitude. We therefore further replaced Sum with Mean after removing R_count (with the same setting as Mean (revised)). In this case, Existence remains nearly unchanged while Count and Compare clearly drop, showing that without relying on R_count, unnormalized accumulation itself still contributes to count-related tasks.

In the revised version, we make the following changes:

  1. revise the Mean ablation in Appendix F.3 so that Mean also uses only the structural scale estimated from its own fitting states, and update the Mean row in Table 5;
  2. add the above three variants that separate the relation encodings to the ablation table in Appendix F.3, and state that the drop of the original No relation encoding mainly comes from the missing endpoint-membership information.

Thank you for the detailed responses and additional experiments. The clarifications regarding the new cross-cohort and ablation results have adequately addressed my major concerns. I appreciate the authors' efforts and have raised my score accordingly.

Reviewer_ppCmOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper asks whether making graph structure explicit to an LLM also makes the model use it. The authors build GRAPHMACC, a controlled diagnostic that varies category-specific neighbor counts while holding total degree fixed, with matched existence/calculation/comparison prompts. Across Qwen3-4B/8B and four presentation-alignment methods, they find a consistent recognition–utilization gap: models detect category presence far better than they recover counts. They attribute this to marginal response compression under softmax normalization (Theorem 2.3) and propose SAIL, which sums occurrence-level trajectory messages without normalization and uses the result to correct candidate scores. Experiments span GRAPHMACC, six native benchmarks, 13 backbones, and combinations with presentation alignment, fine-tuning, and ICL. The diagnostic is cleanly designed and the sweep is broad; my concerns are about metric validity, baseline fairness, and whether the evidence supports the mechanism.

Critical strengths

S1. The GRAPHMACC diagnostic is the paper's strongest contribution: holding total degree fixed while varying only the target-category count, and switching only the prompt across existence/calculation/comparison, cleanly isolates "can the model see the category" from "can it use the count" — consequential because it turns a vague complaint about LLM graph reasoning into a controlled, falsifiable measurement.

S2. The diagnostic result is unusually well substantiated — it replicates across 13 backbones, four presentation-alignment methods, two serializations, three node orders, and sampled real topologies — which is consequential because it shows the recognition–utilization gap is a property of the current paradigm rather than of one model or one prompting trick, and that alone justifies the paper's framing even where the proposed remedy is debatable.

Critical weaknesses

W1. The paper's central mechanism is not supported by its own ablations: SAIL adds a trained re-ranker on top of a frozen LLM, and "Response only," which discards the accumulated structural state entirely, already recovers most of the gain (64.70 vs. 70.41 overall) while "Structure only" is worse still, so the claimed benefit of unnormalized accumulation over normalized aggregation is never isolated from the benefit of simply training a head on the LLM's hidden states.

W2. The evidence for "graph prediction" beyond the synthetic diagnostic is not trustworthy: several native benchmarks sit at or below chance (PubMed 34.10% on 3 classes, Qwen3-8B Cora 13.90% on 7 classes), reported MRR values of 2.91, 4.06 and 5.97 exceed the metric's upper bound of 1, and 1–4 point differences inside run-to-run variance are presented as wins, which is consequential because the paper's generality claim rests entirely on these numbers.

Questions for authors

Q1. How is the Citation2 MRR computed? Table 2 reports 2.91 and 4.06 for SAIL and the appendix goes up to 5.97, but MRR cannot exceed 1 and the untuned LLM is 0.82 in the same column. Please recompute it and state how ties and the 1,000 negatives are handled.

Q2. PubMed sits at 34.10% on 3 classes, Qwen3-8B Cora at 13.90% (below the 14.3% chance level), WikiCS at 14.98% on 10 classes, and LoRA plus Graph Linearization reaches 11.47% on Cora. Given that, what makes the 1–4 point differences on these benchmarks evidence of anything, and where are the paired tests and the GNN baseline?

Q3. All four presentation baselines are training-free while SAIL trains 1–1.7M parameters on the same seven datasets. "Response only" already gets 64.70 overall against 70.41 for full SAIL, and "Structure only" is worse than that. So what in the results actually shows that unnormalized accumulation, rather than a trained head on hidden states, is doing the work? Also, with ∆max = 32 against O(1) logits, can you report the distribution of |MLP|/|s_LLM| so readers can judge whether this is a correction or a replacement?

Q4. Theorem 2.3 fixes the attention scores and values as the count varies — precisely what a real model changes. Lemma B.2 then shows the direction depends on sign(a_R − a_B), so compression is not even guaranteed in your own construction. And Figure 4 shows the adjacent distances still shrinking after SAIL (8.59 > 4.80 > 2.54). What measurement — layer-wise ∆h, a count probe, an intervention that removes normalization — actually confirms the mechanism in a real LLM?

Q5. Three backbones report exactly 50.00 on both binary tasks, which looks like constant answering rather than measured recognition. And ΔG is negative for 6 of 11 backbones in Table 41 while U improves. Why should readers trust a gap metric that rewards failing at recognition, and why is the chance-adjusted version confined to Table 8 while every headline number uses the raw one?

Q6. "Full SAIL" in Tables 5/39/40 is trained on GRAPHMACC alone (U = 59.32, AccE = 92.61) while Table 2 uses the jointly trained model (U = 51.62, AccE = 84.99). The shared model is 7.7 points worse on the benchmark it was trained on. Is that negative transfer, and which of the two do you consider the paper's method?

Q7. Graph-KV falls to 4.31% on Arxiv against 15.26% for the untuned LLM, GaLA's head weights and SLASH's locations and γ are tuned on GRAPHMACC, and Graph Linearization uses PageRank for one backbone and degree for the other. Can any of these reproduce a published gain on any benchmark under an equal tuning budget?

Q8. Several pieces remain unexamined: averaging h⁽⁰⁾, h⁽ℓ⁾, and u_attn mixes three different spaces by simple mean, layer 21 and the eight heads per backbone appear hand-picked, and the fixed a_T scale lets graph-level queries conflate molecular size with composition. What is the sensitivity to each, and how much of SAIL survives without them?

6 comments
AuthorsOpenReview ↗

Response to Reviewer ppCm (1/5)

We thank the reviewer for the constructive comments. We have uploaded a revised version with all changes highlighted in blue, and below we address your concerns one by one.

W2 & Q2 & Q7: Does SAIL provide trustworthy gains on native tasks over properly reproduced baselines?

A. Thank you for pointing this out. Yes, under the text-graph setting, SAIL outperforms all reproduced presentation baselines on all 14 combinations with significant gains. We agree, however, that the native-benchmark scores in the original submission are close to chance level, where 1-4 point differences are insufficient to demonstrate the effectiveness of SAIL. This is because the original submission described nodes with numerical features (Appendix D.2), to avoid introducing bias by designing text ourselves for datasets without node text (e.g., MolHIV). Scores under this setting cannot reflect the actual effectiveness of each method, so we now describe each node with its complete text (e.g., the title and abstract of each paper in Cora) and construct the text from the dataset metadata when node text is unavailable (Appendix D.2 of the revised version). The main results of Table 2 under this text-graph setting are shown below. GraphMACC reports U, the four node classification datasets report accuracy, Citation2 reports MRR (for its scale, see your W2 & Q1), and MolHIV reports AUC. Best results are in bold and second-best in italics.

Qwen3-4B

MethodGraphMACCCoraPubMedWikiCSarXivCitation2MolHIV
Untreated LLM42.2968.8088.0076.2153.080.376459.86
Graph Linearization46.4169.6088.6076.9352.800.085956.70
Graph-KV16.5966.4083.7068.7955.080.030556.45
GaLA41.8968.6087.8076.1252.940.334959.23
SLASH40.6569.2085.4076.4047.500.171364.59
SAIL75.0072.1689.4078.7658.820.435066.32
GCN93.1078.3075.5878.3162.930.050371.57

Qwen3-8B

MethodGraphMACCCoraPubMedWikiCSarXivCitation2MolHIV
Untreated LLM47.6270.5089.2076.1259.640.084057.46
Graph Linearization50.7070.9088.5077.8759.360.005759.94
Graph-KV27.1968.6085.9074.0754.320.165563.09
GaLA46.8570.2089.3075.9759.100.077256.97
SLASH47.0070.5088.8076.0659.420.081257.92
SAIL81.4173.9689.6079.1665.980.317570.15
GCN93.1078.3075.5878.3162.930.050371.57

Under the text-graph setting, all scores return to a reasonable range (e.g., Graph-KV on arXiv rises from 4.31 to 54.32) and node classification is clearly above chance level. Excluding GCN, SAIL outperforms the presentation baselines on all 14 combinations. In per-sample paired tests, which do not include GCN, its gains over the untreated LLM and the strongest baseline are significant (p < 0.05) on 12 and 11 of the 14 combinations, respectively. Since the untreated LLM and the presentation baselines use deterministic inference, a one-sided one-sample t-test using the standard deviation across the seeds of SAIL further shows that its gain over the strongest baseline is significant (p < 0.05) on all 14 combinations, so the gains are not an artifact of run-to-run variance (Appendix E.1).

We treat GCN as a reference for dedicated graph models rather than as a baseline comparable to the presentation methods, and for a fair comparison its receptive field is matched to the text graphs. SAIL outperforms GCN on 7 of the 14 combinations, showing that it is already comparable to a dedicated graph model.

Regarding Q7, the hyperparameters of all baselines are selected on a shared validation set (Appendix D.3). Graph Linearization, Graph-KV, and SLASH each reproduce a gain over the untreated LLM on some datasets. On the datasets shared with the original baseline papers, the reproduced results are on the same scale as the reported values and some are higher (e.g., GaLA on arXiv and SLASH on Cora), with the differences arising from different backbones and evaluation subsets.

In the revised version, we make the following changes:

  1. replace Table 2 with the results under the text-graph setting, add GCN, and report the significance tests in Appendix E.1;
  2. disclose the text construction for each dataset and the settings of each baseline in Appendix D.2 and D.3;
  3. rerun the other experiments under the text-graph setting, including Table 3 (combination with presentation alignment), Table 4 (FT/ICL), and Table 6 (general LLMs), and all results have been updated in the revised version.
AuthorsOpenReview ↗

Response to Reviewer ppCm (2/5)

W1 & Q3: Does unnormalized accumulation contribute beyond simply training a head on the LLM's hidden states, and does the trained head correct or replace the LLM scores?

A. Yes, unnormalized accumulation contributes beyond the trained head, mainly on the utilization task Compare. The correction is small on Existence and Count but closer to a replacement on Compare. We agree, however, that the overall gain also includes the contribution of training the head. The original submission stated that the ablations "isolate" the effect of unnormalized accumulation, which is too strong, and we withdraw this claim.

1. Unnormalized accumulation versus a trained head. The key control for separating the two is Mean, which uses the same head and training settings as Full SAIL and only replaces unnormalized accumulation with normalized aggregation. The table below restates the ablations in Table 5 and Appendix F.3, where Mean (revised) uses the corrected scale (see our response to Reviewer 8F43 W3 & Q3) and Δ\Delta is the change relative to the untreated LLM. These ablations are trained and evaluated only on GraphMACC with Qwen3-4B, so they are unaffected by the text-graph setting. Best results are in bold.

VariantOverallExistenceΔ\Delta ExistenceCountΔ\Delta CountCompareΔ\Delta Compare
Full SAIL70.4192.61+7.9847.10+10.4071.53+23.64
Mean (revised)65.7592.73+8.1046.02+9.3258.49+10.60
Response only64.7092.85+8.2243.30+6.6057.97+10.08
Structure only62.4988.68+4.0538.90+2.2059.90+12.01
Untreated LLM56.4184.63-36.70-47.89-

The Overall gap between Full SAIL and Response only appears small because Overall averages the recognition and utilization tasks. On Existence, Response only is comparable to Full SAIL, and the gain mainly comes from the head. On Compare, the improvement of Full SAIL over Response only exceeds the gain of the head itself, while Mean (revised), which only switches to normalized aggregation, falls back to the level of Response only, showing that this part of the gain cannot be obtained by training a head. Structure only is also weaker than Full SAIL, showing that the accumulated state is utilized only when fed into the head together with the original LLM response.

The hyperparameters of the training-free presentation baselines are selected on a shared validation set (see our response to your W2 & Q2 & Q7). More importantly, SAIL is not intended to replace presentation methods under equal supervision (Section 1) but to provide a complementary utilization perspective, as supported by the results of combining SAIL with each presentation method under the text-graph setting (Table 3 of the revised version).

2. Correction or replacement. We report the magnitude of the correction relative to the original scores for Qwen3-4B on GraphMACC. Norm ratio is the L2 norm of the correction divided by that of the centered original scores over the candidates, reported as the median (90th percentile in parentheses). Near Δ\Delta max is the fraction of corrections reaching 95% of Δ\Delta max, Flipped is the fraction of predictions changed by the correction, and Right / wrong flips are the average numbers of predictions flipped to correct and to incorrect per run.

MetricExistenceCountCompare
Norm ratio0.00 (0.01)0.33 (1.17)1.60 (16.64)
Near Δ\Delta max0%0%0%
Flipped2.0%34.4%33.4%
Right / wrong flips21.7 / 9.2223.5 / 13.9438.1 / 73.1

No correction approaches Δ\Delta max, so Δ\Delta max is only an upper bound and does not determine the correction magnitude. The correction is nearly zero on Existence, mostly smaller than the original scores on Count, and usually larger than the original scores on Compare, where it is closer to a replacement. Among the flipped predictions, those flipped to correct far outnumber those flipped to incorrect.

In the revised version, we make the following changes:

  1. revise the claims throughout the paper that the ablations "isolate" the effect of unnormalized accumulation, and clarify the contributions of the head and unnormalized accumulation;
  2. add the table of correction magnitudes to Appendix F.3 and state that Δ\Delta max is only an upper bound.
AuthorsOpenReview ↗

Response to Reviewer ppCm (3/5)

W2 & Q1: How is the Citation2 MRR computed, given that the reported values exceed 1?

A. Thank you for pointing this out. The reported values are MRR multiplied by 100, so 2.91 is in fact 0.0291. Although the caption of Table 2 in the original submission states "Scores are percentages," the Citation2 column is still easy to misunderstand. Consistent with the OGB evaluator, each query ranks one positive against the 1,000 negatives provided by OGB by the score difference between Yes and No, and tied negatives are counted as one half in the rank. The new Table 2 reports MRR on a 0 to 1 scale. Regarding the scores close to chance level, we have replaced the numerical node features with textual descriptions, the scores of all methods return to a reasonable range, and the conclusions remain unchanged (see our response to your W2 & Q2 & Q7).

In the revised version, we make the following changes:

  1. report MRR on a 0 to 1 scale in all Citation2 tables;
  2. state in Appendix D.2 that, following the OGB evaluator, tied negatives are counted as one half in the rank.

Q4: Is marginal response compression confirmed in a real LLM beyond the idealized theorem?

A. Partially yes. In Qwen3-4B, the compression condition holds for most samples and removing normalization reduces compression, but normalization is not its only cause. We also agree that Theorem 2.3 is an idealized analysis with fixed attention scores and values, and the paper therefore presents compression only as one possible contributor to the gap.

1. The condition for compression in Lemma B.2. We measured the single-occurrence attention weights of the target category (a_R) and the replaced category (a_B) in the untreated Qwen3-4B on GraphMACC.

Degreea_R / a_BFraction with a_R > a_B
42.570.75
84.660.80
167.350.76
All3.970.77

About 77% of the samples satisfy the compression condition a_R > a_B, and the ratio between the two increases with degree, consistent with the intuition that the queried target category usually receives larger attention weights. Therefore, although Lemma B.2 itself does not guarantee the direction of compression, its condition holds for most samples in a real LLM.

2. Adjacent distances in Figure 4. The adjacent distances in Figure 4 still shrink because SAIL is a residual correction on the original scores and does not remove the existing decreasing trend. As shown below, SAIL increases every adjacent distance, and the more severe the original compression, the larger the increase.

Adjacent countsUntreated LLMSAILRelative increase
1 vs 28.408.59+2.3%
2 vs 34.144.80+15.8%
3 vs 42.112.54+20.2%

3. Measurements in a real LLM. As suggested, we add layer-wise Δ\Delta h measurements and a de-normalization intervention on Qwen3-4B. We measure compression by the decay ratio, defined as the geometric mean of the successive ratios between the hidden-state distances of adjacent counts at the answer position, where a value below 1 indicates compression. The intervention fixes the attention weight of each occurrence of the target category to its value at count 1 without renormalization, and is applied either at layer 21 only (8 heads) or at layer 21 and all subsequent layers (same heads), without the SAIL readout. The inputs are matched samples with d = 4 and d = 8 constructed by the GraphMACC generator.

SettingL21L22L23L24LastCount acc.
Untreated LLM0.8250.8320.8150.7940.77442.7
De-norm. at L21 (no readout)0.9010.8800.8430.8150.76741.0
De-norm. from L21 on (no readout)0.9010.8810.8460.8270.76940.3
SAIL0.949----50.5

The decay ratio of SAIL is measured on its accumulated structural state. De-normalization clearly slows the shrinking of the distances at layers 21 to 24, showing that normalization is one cause of compression in a real LLM. However, compression still exists at the last layer and count accuracy does not improve, and only adding the SAIL readout clearly improves it. Together with the Response only result in your W1 & Q3, this shows that neither training only the readout nor only removing normalization is sufficient to utilize the count information, and the gain of SAIL comes from the combination of unnormalized accumulation and the readout.

In the revised version, we make the following changes:

  1. state in Section 2 that Theorem 2.3 is a local analysis with fixed attention and values, and that Lemma B.2 requires a_R > a_B for compression;
  2. revise the Figure 4 description in Section 4.3 and add the distance-increase table to Appendix F.1;
  3. add the attention-weight comparison and the de-normalization intervention to Appendix B.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.