acceptodds
Under review as a conference paper at ICLR 2027

On the Token Collapse Phenomenon in Textual Graph Soft Prompting

Abstract

Recent methods for injecting graph structure into LLM prompts include graph soft prompting, in which a graph neural network (GNN) learns graph-specific tokens that capture relational information among entities in textual prompts. A major development in this line of work is G-Retriever, which combines graph soft prompts with a verbalized description of the same graph to strengthen the textual prompt signal. In this paper, we conduct an in-depth analysis of G-Retriever and uncover a pathological behavior we term token collapse. Across three benchmarks, three graph encoders, two LLM families, and both frozen and LoRA-adapted models, G-Retriever's graph tokens become almost indistinguishable across inputs. We provide evidence that the same failure recurs in subsequent methods to G-Retriever, even with multi-token pooling. Under the standard setup, swapping a token with the dataset mean, a token from another example, or one derived from a corrupted graph produces no measurable change in answer accuracy. Moreover, a learned constant prompt that never accesses the graph matches or surpasses G-Retriever on all three benchmarks. These findings indicate that, rather than conveying graph content as intended, the collapsed token functions primarily as a generic soft prompt. We examine conditions linked to this collapse and propose a two-stage procedure that yields non-collapsed tokens while maintaining competitive performance in most cases. However, token variability alone does not prove that the model is using graph information: via interventions and decoding probes, we show that depending on the configuration, a token can reflect the question, an answer prior, or graph-derived content that the LLM may or may not actually leverage. Overall, our results offer a practical approach to evaluating graph-token alignment rather than inferring it from task accuracy alone. Code to reproduce our experiments is available at https://anonymous.4open.science/r/On-the-Token-Collapse-Phenomenon-in-Textual-Graph-Soft-Prompting-6136/.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
graph soft prompting, soft prompting, knowledge graph question answering, graph neural networks, RAG

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_dmzTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper audits the graph-token interface of G-Retriever and identifies "token collapse": although the GNN's representations differ across inputs, the projected graph tokens are nearly identical. The phenomenon is consistent across benchmarks, graph encoders, LLM families, and tuning regimes, and also appears in several later methods. Replacing the token with a fixed or mismatched alternative does not change accuracy, and a graph-free learned constant performs on par with G-Retriever, suggesting that the token acts mainly as a soft prompt while graph information is conveyed through the verbalised graph. The authors further propose a two-stage pipeline that produces non-collapsed tokens, but show that these tokens may encode the question, an answer prior, or graph information, which the LLM uses only in some configurations.

Critical strengths
  1. An insightful and practically important finding. The paper shows that strong task accuracy can coexist with a graph token that carries little usable graph-specific information. Comparing encoder and token geometry locates the collapse at the projection step. This is a useful caution for the graph soft-prompting literature.
  2. A well-designed diagnostic protocol. Separating token variation, recoverable content, and downstream use is a valuable conceptual contribution. CONST, which shares G-Retriever’s projector and changes only the token’s source, is a clean graph-versus-no-graph control. The idea of introducing a separate prompt token to carry the task context, so that the graph token has a simpler, more focused role, is also appealing.
  3. Broad replication and transparency. The phenomenon is replicated across many settings. The experiments cover three benchmarks, three graph encoders, two LLM families, and frozen and LoRA settings, and recurs in several later methods (GRAG, GRETPool, GraSP).
Critical weaknesses

W1. Collapse is defined only in relative terms. Cosine and residual ratio are measured against a mean token 22–97× the norm of a word embedding. In absolute terms the residual is still about 1.1–7.5 word-embedding norms (Table 11).

W2. Less collapse does not mean better performance. In Table 3, the most collapsed configurations perform best (85.56, 84.48), while the least collapsed one (cosine 0.323) falls to 57.76, near majority class. This contradicts the framing of collapse as a pathology.

W3. The proposed pipeline is not a convincing remedy. It only matches G-Retriever on WebQSP and drops on SceneGraphs (72.50 vs. 81.35). It also trails the graph-free CONST on WebQSP (71.25) and SceneGraphs (81.40). The soft prompt is relocated to a second token; the graph token does not become more useful. On WebQSP the token mainly encodes the question and answer (ranks 65/60) rather than the graph (315 vs. control 407). Stage 1 memorises answers (top-1 about 0.9 on training vs. 0.5 on validation), and under cross-fitting ExplaGraphs accuracy falls below the baseline (76.35 vs. 80.87).

W4. Key comparisons are single-seed. G-Retriever’s ExplaGraphs initialisations range from 85.56 to 89.71, while CONST (85.56) and the pipeline (85.74) each have one seed, so the claim that CONST matches G-Retriever on all benchmarks does not hold on ExplaGraphs. Multi-seed results and a per-dataset power analysis are needed.

W5. Presentation. The presentation of the method figure (Figure 2) is somewhat rough and could be improved. Most tables are in the appendix, with substantial duplication, and Appendix E contains a broken “Table ??” reference.

Questions for authors

Seen in Weaknesses.

5 comments

Thank you for the detailed review. We have conducted three additional experiments to address your concerns -- residual amplification (W1), a graph-free CONST control for Qwen2.5-3B (W3) and multi-seed evaluations (W4).

W1: collapse is defined only in relative terms.

We agree that a small residual relative to the mean token can still be substantial in absolute terms and could contain information that the LLM uses. To test this directly, we amplified the input-specific residual while keeping the trained model and textual inputs fixed.

Setting. We decompose each graph token as ti=μ+rit_i = \mu + r_i, where μ\mu is the mean token over the training questions (all questions for ExplaGraphs; a random sample of 2,000 for WebQSP) and rir_i is the input-specific part. Using the five ExplaGraphs G-Retriever initializations and the four WebQSP ones, we replace each token with:

  • Own: μ+k⋅ri\mu + k \cdot r_i, preserving its original residual;
  • Swapped: μ+k⋅rj\mu + k \cdot r_j, using another example's residual;
  • Random: μ+k⋅∥ri∥u\mu + k \cdot \lVert r_i \rVert u, where uu is a random unit vector, matching the norm of the original residual.

We test k∈{1,2,5,10,20}k \in \{1, 2, 5, 10, 20\}. At k=1k = 1, Own corresponds to the original token and Swapped to SWAP. We report accuracy on ExplaGraphs and Hit@1 on WebQSP, and compare variants with the exact McNemar test.

Results. Numbers are in the table below. At k=1k = 1, the size of the trained model produces, replacing the input-specific part with another example has no significant effect in any run (on WebQSP +0.37 / −1.60 / −0.92 / −1.11, p ≥ .07). On WebQSP, once the part is amplified, the answers depend on which part is supplied: another example's part is significantly worse than the question's own in three of the four runs, for example at k=5k = 5 by 3.69 / 3.13 / 1.90 / 2.58 points (p < .001 / .02 / .14 / .046). On ExplaGraphs we did not detect a difference between the two at any kk in any of the five runs (largest gap 3.07 points, p = .15), although larger values of kk hurt accuracy. A random direction of the same norm hurts less than the learned part at large kk: at k=20k = 20 by 13.94 / 11.79 / 15.11 / 10.69 points on WebQSP (p < 10−1510^{-15}) and by 8.30 / 7.04 / 8.12 points in ExplaGraphs runs 2–4 (p < .005).

ExplaGraphs, accuracy (run 1 / 2 / 3 / 4 / 5):

kkownanother example'srandom, same norm
185.38 / 86.10 / 87.91 / 89.17 / 85.7484.12 / 84.84 / 88.27 / 89.17 / 85.9285.38 / 85.74 / 88.45 / 89.53 / 85.92
284.84 / 84.66 / 87.00 / 89.53 / 85.9284.12 / 84.30 / 88.09 / 88.99 / 86.4685.74 / 85.56 / 87.91 / 89.35 / 86.46
582.49 / 80.51 / 85.38 / 88.81 / 85.0283.94 / 77.44 / 85.02 / 87.73 / 84.6684.84 / 82.67 / 86.28 / 88.81 / 84.84
1074.55 / 68.95 / 76.90 / 86.64 / 75.4574.01 / 67.33 / 78.70 / 84.66 / 76.5375.63 / 74.91 / 79.06 / 86.82 / 76.71
2054.87 / 50.00 / 52.35 / 64.26 / 54.3353.61 / 50.72 / 53.43 / 66.25 / 51.8153.61 / 58.30 / 59.39 / 72.38 / 55.23

WebQSP, Hit@1 (run 1 / 2 / 3 / 4):

kkownanother example'srandom, same norm
170.64 / 71.74 / 70.33 / 70.4571.01 / 70.15 / 69.41 / 69.3569.35 / 70.45 / 71.44 / 70.33
270.02 / 70.88 / 67.63 / 71.2570.02 / 68.12 / 66.40 / 70.1569.72 / 71.19 / 68.55 / 69.90
567.14 / 59.40 / 60.63 / 63.0263.45 / 56.27 / 58.72 / 60.4468.73 / 68.06 / 65.91 / 68.86
1054.18 / 36.61 / 42.69 / 46.5051.78 / 34.46 / 39.99 / 42.9461.00 / 54.48 / 57.92 / 57.99
2025.06 / 15.42 / 17.94 / 20.5223.83 / 13.88 / 16.28 / 18.1239.00 / 27.21 / 33.05 / 31.20

These results confirm part of your concern: the residual is not empty, and its content can affect predictions when amplified. Our claim is narrower. At the scale learned by standard G-Retriever, replacing the input-specific residual has no resolved answer-level effect in these configurations. The token therefore behaves primarily as a shared prompt under its normal operating conditions, despite containing non-zero input-specific variation.

We will include this analysis in the revision and report the absolute residual magnitude in word-embedding units more prominently alongside the relative collapse measures.

The other weaknesses are addressed in following comments.

W2: less collapse does not mean better performance.

We agree, and this is one of the central findings of the paper. We do not argue that reducing token collapse necessarily improves task accuracy. We use ”pathological” to describe a mismatch between the intended/widely accepted role of the graph token of providing input-dependent graph information, and its observed function as a largely graph-independent soft prompt.

Our CONST and fixed-weight intervention results show that a model can achieve high accuracy even when the input-specific graph-token variation contributes little measurable information to its answers. The relevant question is therefore not whether collapse lowers accuracy, but whether the graph-token pathway performs the function attributed to it.

Our contribution is to identify and characterize token collapse, demonstrate its functional consequences through controlled interventions and provide an evaluation protocol that distinguishes token variability, graph-information recoverability, downstream utilization and task performance. The two-stage pipeline serves as a diagnostic proof of concept rather than a general performance remedy, and it serves as a starting point for future research to exploit graph-information within the supplied tokens.

W3: the pipeline is not a convincing remedy.

The pipeline was designed primarily to prevent collapse and investigate what non-collapsed graph tokens encode and whether the LLM uses them (RQ4). It is not intended as a generally superior GraphQA architecture. On the representation side, it achieves the intended change: token cosine decreases from 0.9960.996 to 0.1420.142 on WebQSP and to 0.099−−0.2000.099--0.200 across our tested configurations (Table 4).

Accuracy against G-Retriever and CONST. With Llama-2, the pipeline maintains similar accuracy to G-Retriever on ExplaGraphs, is approximately one point lower on WebQSP across runs (W4) and performs worse on SceneGraphs (72.5072.50 vs. 81.3581.35). We acknowledge that the pipeline does not consistently improve accuracy, nor does it consistently outperform CONST. The largest improvements in Table 4 occur with Qwen2.5-3B, for which we had not included CONST. We have now trained this missing control with the same training budget as the Qwen configurations (30 epochs, early stopping patience of 4):

Qwen2.5-3BExplaGraphsWebQSP
G-Retriever (Table 4)74.3757.13
CONST (new)85.3856.94
Ours (Table 4)83.9465.79

On ExplaGraphs, graph-free CONST scored 85.38, compared with 83.94 for our pipeline (exact McNemar p = .28) and 74.37 for G-Retriever. This shows that a graph-free soft prompt can achieve a similar gain in this setting. The pipeline’s graph token was decodable, and graph replacement produced no detectable accuracy effect (Table 66). On WebQSP, our pipeline outperformed CONST by 8.85 Hit@1 points, while CONST and G-Retriever scored 56.94 and 57.13, respectively (p = .89).

Does the graph token become more useful? Moving the soft prompt to a second token is what frees the graph token to carry input-specific information, and in several settings the LLM then uses it, while G-Retriever's token has no resolved effect: replacing the graph supplied through the token (GSWAP) costs 5.30 points on SceneGraphs with Llama-2, 6.50 on SceneGraphs with Qwen and 2.09 on WebQSP with Qwen (Table 4). On WebQSP with Qwen, this use comes with the accuracy gain over CONST above. Of course, here we mainly uncover this phenomenon, and understanding how to control what these tokens have to encode becomes the real challenge for future works.

What the WebQSP token encodes. We agree, and this is why Table 5 labels the standard WebQSP token MASKED rather than USED. The same architecture trained without question conditioning is used (GSWAP −2.33, Table 4), so what a non-collapsed token carries, and whether it is used, depends on the training inputs. These experiments show that removing collapse does not determine what the graph token encodes or whether the LLM uses it. The resulting behavior depends on the training inputs and benchmark, which motivates evaluating graph-token use directly rather than inferring it from representation geometry or accuracy.

Stage 1 memorisation. We ran the cross-fitting of Appendix P to test exactly this: Stage 1 and Stage 2 see disjoint halves of the training questions, so Stage 2 cannot rely on answers that Stage 1 memorised. The WebQSP token is still used (GSWAP −2.83, Table 69), so its use is not only memorised answers. On ExplaGraphs, as you note, cross-fitting costs accuracy (76.35 vs 80.87, Table 72).

As this might cause confusion, we will better state these limits in Section 4.4. More generally, our audit lets us check whether a training change makes the LLM use graph information supplied through the token, rather than inferring it from accuracy.

W4: key comparisons are single-seed.

We agree. We have now trained additional runs using the same training recipe as the paper, varying only the random seed. Run 1 corresponds to the main results in Tables 2 and 4. G-Retriever's first four ExplaGraphs initializations were already reported in Tables 14–17; the remaining runs are new. SceneGraphs remains at one run per model for now, since a single run takes more than 12 hours.

ExplaGraphs, accuracy:

ModelRunsMean ±\pm SD
G-Retriever85.56 / 86.28 / 87.91 / 89.71 / 85.7487.04±1.7687.04 \pm 1.76
CONST85.56 / 85.02 / 85.20 / 88.63 / 87.1886.32±1.5586.32 \pm 1.55
Ours85.74 / 87.55 / 85.56 / 84.66 / 87.1886.14±1.2086.14 \pm 1.20

WebQSP, Hit@1:

ModelRunsMean ±\pm SD
G-Retriever70.33 / 71.62 / 70.58 / 70.5870.78±0.5770.78 \pm 0.57
CONST71.25 / 69.90 / 70.15 / 70.3970.42±0.5970.42 \pm 0.59
Ours69.90 / 69.23 / 69.23 / 70.2769.66±0.5269.66 \pm 0.52

Across initializations, CONST and G-Retriever are within one point of each other on both benchmarks: CONST minus G-Retriever is −0.72 on ExplaGraphs (95% Welch interval −3.14 to +1.70) and −0.35 on WebQSP (−1.36 to +0.65). These intervals do not establish statistical equivalence, particularly on ExplaGraphs, but they show that the two methods have similar mean performance at the resolution supported by the available runs. We will update our manuscript with this fact. We will also report that our pipeline is 1.12 points below G-Retriever on WebQSP (−2.07 to −0.17) and 0.77 points below CONST (−1.72 to +0.19).

Per-dataset power analysis. We also estimated the smallest detectable effect at 80%80\% power and α=0.05\alpha = 0.05, expressed in accuracy points:

ComparisonExplaGraphsWebQSPSceneGraphs
Between models (Welch)3.361.38n/a (1 run)
MEAN (McNemar)1.522.281.66
SWAP (McNemar)1.962.381.69
GSWAP (McNemar)1.752.421.68

Between-model estimates use the available training runs; intervention estimates use the observed paired prediction changes in G-Retriever run 1. These estimates clarify the resolution of our experiments. The intervention tests have approximately 80%80\% estimated power for effects of 1.5–2.41.5\text{–}2.4 points, depending on the dataset and perturbation. Smaller effects may remain undetected, particularly on WebQSP. We will report these values alongside our two-point criterion in Section 4.1 and avoid interpreting nonsignificant differences as proof of zero effect.

Importantly, the central finding does not require CONST to achieve exactly the same accuracy as G-Retriever. The key observation is that a learned graph-independent prompt recovers much of G-Retriever's performance, while replacing the graph-conditioned token variation has little resolved effect in the standard configurations. The new multi-seed comparisons support this interpretation more reliably than the original single-run results.

W5: presentation.

Thank you. We will improve Figure 2, reduce duplication in the appendix and correct the broken reference in Appendix G (which should point to Table 11).

We hope the new experiments and clarifications address your main concerns and would appreciate your reconsideration of the overall assessment.

Reviewer_UD4XOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines token collapse in graph soft prompting, focusing on G-Retriever. The authors find that graph tokens become nearly identical after projection, despite being computed from different input graphs. Replacing these tokens with their mean, tokens from other examples, or a learned graph-independent prompt generally leaves task performance largely unchanged. The authors also introduce a two-stage training procedure that reduces collapse. Through interventions and decoding probes, they show that token variation, recoverable graph information, and the LLM’s use of that information need to be evaluated separately.

Critical strengths
  1. The paper identifies an important gap in how graph soft prompting is evaluated. When the graph is also provided as text, good answer accuracy does not establish that the graph encoder contributes useful information. The combination of token replacement experiments and a constant-prompt baseline makes this concern concrete. The constant prompt is a particularly useful control because it retains the projector parameterization while removing the graph input, providing a more direct comparison than ordinary prompt tuning.

  2. The analysis goes beyond documenting collapse. The proposed training procedure produces more varied tokens, but the authors also check what those tokens contain and whether the LLM uses their content. The results show that reducing collapse does not consistently improve accuracy or establish graph use.

Critical weaknesses

The evidence for using graph content is stronger than the evidence for using graph structure. GSWAP replaces entities, relations, and topology together, so a performance drop cannot distinguish reliance on relational structure from reliance on entity semantics or compatibility between the graph and the question. Similarly, recovering an embedding of the verbalized graph mainly demonstrates recoverable semantic content.

Questions for authors

See weaknesses

2 comments

Thank you for the review and for highlighting the distinction between graph content and graph structure.

Graph content vs graph structure.

We agree that GSWAP alone cannot establish the use of relational structure, since it changes the whole graph, i.e. changes the node attributes, relation attributes and topology together. Our existing interventions provide more targeted evidence: NODE replaces node features with those from other graphs while preserving the topology, whereas EDGE rewires connectivity while preserving node features and edge count.

We report these interventions for five configurations of our pipeline (differences from REAL, exact McNemar test; appendix table numbers in brackets):

ModelGSWAPNODE (content)EDGE (topology)
Ours, SceneGraphs, Llama-2 [T34]−5.30***−5.10***−1.05 (p = .19)
Ours, WebQSP, Llama-2 [T43]−1.97†−1.23 (p = .22)−0.80 (p = .42)
Ours, WebQSP, no question conditioning [T62]−2.33*−2.09*−0.49 (p = .64)
Ours, WebQSP, Qwen2.5-3B [T69]−2.09**−0.80 (p = .25)−1.35*
Ours, SceneGraphs, Qwen2.5-3B [T69]−6.50***−4.40***−1.65*

On SceneGraphs (with both LLMs) and on WebQSP without question conditioning, NODE produces much larger drops than EDGE and reproduces most of the GSWAP effect. On WebQSP with Qwen, EDGE (−1.35) is larger than NODE (−0.80), but both stay below our two-point criterion. Rewiring the edges has no resolved effect for any of the five pipeline models. These results suggest that the observed graph-conditioned sensitivity is driven more strongly by node-feature content than by connectivity in several configurations, although smaller structural effects cannot be excluded. This is consistent with our Stage-1 supervision, which does not explicitly target topology. The encoder is aligned with the verbalized-graph embedding on ExplaGraphs and with the answer on WebQSP and SceneGraphs.

The "Stage 1 + GELU, projector only" models provide a useful contrast where the predictions exhibit greater sensitivity to connectivity perturbations. They use the same frozen Stage 1 encoder with a single graph token and no separate prompt token, and their tokens do not collapse. GSWAP was not evaluated for them in the paper, but NODE and EDGE were:

Stage 1 + GELU, projector onlytoken cosineNODE (content)EDGE (topology)
ExplaGraphs [T3, T25]0.323−1.26 (p = .69)−6.14*
SceneGraphs [App. G, T33]0.519−8.90***−2.45*

In both configurations, rewiring the edges changed accuracy, although on ExplaGraphs this model is close to majority-class accuracy. These results show that non-collapsed tokens can retain sensitivity to connectivity changes even without explicit topology supervision. However, sensitivity to random rewiring does not by itself establish structural reasoning. Explicit topology supervision within the stage 1 training would be a useful next step.

We hope this addresses your concern and clarifies what our interventions establish. We are happy to discuss any further questions.

Thanks for the response. As shown in my score, I am already on the positive side. I will maintain my score.

Reviewer_SyPSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether graph soft tokens in textual GraphQA actually preserve and contribute graph-specific information. The authors identify a “token collapse” phenomenon in G-Retriever, where projected graph tokens become nearly identical across inputs. Through graph-free controls, fixed-weight token/graph interventions, likelihood analysis, and decoding probes, the paper shows that task accuracy alone can substantially overstate the role of graph-token information. A two-stage training pipeline is further used to demonstrate that token variability, graph-information recoverability, downstream use, and task performance are distinct properties.

Critical strengths
  1. The paper provides convincing evidence that strong GraphQA performance does not necessarily imply meaningful use of graph soft tokens. In particular, the combination of matched graph-free controls and fixed-weight interventions makes the analysis substantially stronger than relying on token similarity alone.

  2. The distinction between token variation, recoverable graph information, actual downstream use, and task accuracy is conceptually useful. The experiments support this distinction across multiple encoders, LLM settings, and intervention types.

Critical weaknesses
  1. The novelty relative to recent work on weak graph-language alignment could be articulated more clearly. Prior work has already shown text-dominant behavior and that graph-language models may underutilize graph information; the main additional contribution here appears to be the token-level collapse analysis and causal audit of the graph-token interface.

  2. The strongest evidence concerns G-Retriever. The cross-method audit of later graph-prompting approaches is more limited, so the broader claim that token collapse is a recurring phenomenon across textual graph soft prompting is currently supported less strongly than the G-Retriever-specific conclusions

Questions for authors
  1. Please see Weakness 1
  2. Does token collapse persist in graph-language models trained with explicit graph-reconstruction/alignment objectives, rather than answer-only supervision?
3 comments

Thank you for the review. We address the novelty and generalization concerns below and report a new experiment testing whether an explicit graph-reconstruction objective prevents collapse.

W1/Q1: novelty with respect to work on weak graph-language alignment.

We agree that Section 2 does not distinguish our contribution clearly enough from recent work, and we will expand it using the extended discussion in Appendix S. Zhang et al. (2026) show that autoregressive training can favour a text-dominant solution, and Petkar et al. (2026) show that single-modality and raw soft-prompt baselines remain competitive. Our work complements these findings by examining what happens specifically at the graph-token--LLM interface, rather than inferring graph use from model-level performance. This reveals three aspects not established by the prior observations:

  1. Where input-specific variation is suppressed. Graph-encoder outputs remain distinct across inputs (cosine 0.420.42-0.690.69), while the projected tokens become nearly identical (0.9850.985–0.9980.998, Table 1). This localizes the observed collapse to the projection interface.

  2. What the collapsed token contributes instead. Fixed-weight interventions distinguish the shared token component, which the LLM relies on (zeroing the token costs 59.659.6-82.082.0 points in accuracy, Appendix J), from input-specific variation, whose replacement produces no resolved answer-level effect in the tested settings (Table 2). CONST provides a complementary control as it retains the projector but removes graph input entirely, while achieving comparable performance to G-Retriever. Thus, the interpretation of the collapsed token as a soft prompt is supported by controlled experiments, rather than inferred from accuracy or token similarity alone.

  3. What happens once the collapse is prevented. Via our two-stage pipeline experiments, we show that non-collapsed tokens can encode a question prior, an answer prior, recoverable graph information that the LLM does not measurably use or graph-conditioned information that changes the LLM’s prediction (Table 5).

Our contribution is therefore not simply another observation of weak graph-language alignment, but the identification and localization of token collapse together with controlled experiments establishing the functional role of the graph-token.

W2: limited cross-method analysis.

We agree that our most detailed analysis concerns G-Retriever. We chose it as the primary case study because it is an influential work and provides a representative architecture combining graph soft tokens with verbalized graph input. To investigate whether collapse is specific to this implementation, Appendix F extends the geometry and intervention analyses to GRAG, GRETPool, and GraSP and we find the same trends as our G-Retriever experiments. All exhibit near-identical projected tokens and token replacement produces no resolved answer-level effect (Table 10).

We address Q2 in a following comment.

[1] Zhang, Z., Wang, X., Zhang, M., Tan, J., & Shi, C. (2026, April). Toward graph-tokenizing large language models with reconstructive graph instruction tuning. In Proceedings of the ACM Web Conference 2026 (pp. 430-441).

Q2: does collapse persist with explicit graph-alignment or reconstruction objectives?

Table 3 answers part of this for alignment, showing that a graph encoder trained with an explicit contrastive alignment objective (Stage 1, InfoNCE) produces highly distinct representations (cosine 0.0240.024). However, after passing these representations through G-Retriever's Sigmoid projector and training with the answer objective, the resulting tokens are again nearly identical (cosine 0.9920.992). An aligned encoder is therefore not enough when the projector and the answer objective stay the same. Replacing Sigmoid with GELU reduces token cosine to 0.3230.323 (Table 3), while our two-token pipeline reaches 0.0990.099–0.2000.200 (Table 4).

Additional experiment. To investigate the stronger case of joint training with explicit reconstruction objectives, we conducted an additional experiment inspired by RGLM-Similarizer [1]. RGLM introduces graph-reconstruction supervision at the LLM's graph-token outputs, although its original setting differs from textual GraphQA: it uses node-level graph tokens for tasks such as node classification and link prediction, without simultaneously supplying the graph as verbalized text.

Setting. We retained G-Retriever's architecture and training recipe, while adding a latent-space reconstruction loss based on RGLM-Similarizer. Specifically, a linear head maps the LLM's last-layer hidden state at the graph-token position to a target representation from a frozen graph encoder (our Stage-1 encoder trained without question conditioning). The reconstruction loss is one minus the cosine similarity between the predicted and target representations, with weight 11 added to the original answer loss.

We trained three independent runs on ExplaGraphs and evaluated token geometry, reconstruction quality and the fixed-weight interventions from our paper.

run 1 / 2 / 3
accuracy88.63 / 88.27 / 88.45
token cosine0.9955 / 0.9955 / 0.9954
reconstruction cosine on test questions0.40 / 0.37 / 0.46
Δ\DeltaMEAN+0.18 / +0.54 / +0.54
Δ\DeltaSWAP−0.54 / +1.44 / +0.00
Δ\DeltaGSWAP−0.54 / +1.44 / −0.18

None of the interventions is statistically significant (p > .05 in all runs).

The reconstruction objective learns a nontrivial signal, but collapse persists. On test questions, the reconstruction head reaches cosine similarities of 0.400.40, 0.370.37, and 0.460.46 with the target, compared to 0.110.11 for the mean-target prediction baseline. Nevertheless, the projected tokens collapse and remain highly similar across inputs (cosine approximately 0.99550.9955 versus 0.9970.997 for standard G-Retriever), and MEAN, SWAP and GSWAP produce no resolved answer-level effects.

This gives a more specific answer to your question: even when explicit supervision makes a graph representation recoverable from the LLM's graph-token hidden state, it need not prevent geometric collapse or make graph-conditioned token variation measurably useful for answering questions. Note that this experiment adapts a latent-space reconstruction objective to G-Retriever; it does not evaluate RGLM in its original setting or establish that reconstruction objectives generally fail to prevent collapse.

We will include these results in the final version of the paper.

We hope these additional results clarify the novelty and scope of our contribution. We appreciate your consideration and would be happy to clarify any remaining questions.

[1] Zhang, Z., Wang, X., Zhang, M., Tan, J., & Shi, C. (2026, April). Toward graph-tokenizing large language models with reconstructive graph instruction tuning. In Proceedings of the ACM Web Conference 2026 (pp. 430-441).

Thanks to the authors for the detailed response and additional experiments. I will maintain my score

Comments on the paper

1 comment

Dear Reviewers and AC,

Thank you for your constructive feedback and careful evaluation of our work. We appreciate that the reviewers recognized the importance of our findings on token collapse, the value of our diagnostic protocol and the distinction between graph-token variability, recoverable information, downstream use and task performance. We were also glad that the reviewers found our proposed separation of tokens and training procedure appealing.

In response to your comments, we have run several experiments and analyses, and updated the paper with meaningful clarifications. We have uploaded a revised manuscript with textual changes highlighted in blue. Below we summarise the changes to the paper we have already made:

  1. Novelty and scope (Sections 2 and 5). We clarify what the token-level audit adds to prior work on weak graph–language alignment. We also state the limits of the cross-method audit: one run per configuration on WebQSP, using controlled adaptations and without likelihood measurements.
  2. Method and interpretation (Section 3.3, Figure 2, and Section 5). We explain the two stages step by step and redraw Figure 2. We distinguish sensitivity to node features from sensitivity to connectivity, and clarify that sensitivity to rewiring alone does not establish structural reasoning.
  3. Additional controls and statistical resolution (Sections 4.1, 4.3, and 4.4; Appendix T). We report five training runs on ExplaGraphs and four on WebQSP, alongside per-dataset power estimates and a matched Qwen2.5-3B CONST control. The multi-run results show similar mean scores for CONST and G-Retriever without establishing equivalence. SceneGraphs remains single-run per model.
  4. Residuals and reconstruction (Table 11 and Appendices S and U). We correct the word-normalised token norms and report absolute residual magnitudes. Appendix S tests residual replacement at the learned scale and under amplification; Appendix U reports three runs with a reconstruction objective adapted from RGLM.

We also corrected the broken table reference in Appendix G. Appendices S–U and Tables 74–80 are new, and the extended related work moved from Appendix S to Appendix V. We kept Tables 1–73 unchanged in numbering so that references in the rebuttal remain valid. We plan to consolidate overlapping appendix tables and update cross-references in the camera-ready version.

We hope these revisions and additional experiments address your concerns and help clarify the contributions and limitations of our work. We appreciate your consideration and welcome any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.