- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper audits the graph-token interface of G-Retriever and identifies "token collapse": although the GNN's representations differ across inputs, the projected graph tokens are nearly identical. The phenomenon is consistent across benchmarks, graph encoders, LLM families, and tuning regimes, and also appears in several later methods. Replacing the token with a fixed or mismatched alternative does not change accuracy, and a graph-free learned constant performs on par with G-Retriever, suggesting that the token acts mainly as a soft prompt while graph information is conveyed through the verbalised graph. The authors further propose a two-stage pipeline that produces non-collapsed tokens, but show that these tokens may encode the question, an answer prior, or graph information, which the LLM uses only in some configurations.
Critical strengths
- An insightful and practically important finding. The paper shows that strong task accuracy can coexist with a graph token that carries little usable graph-specific information. Comparing encoder and token geometry locates the collapse at the projection step. This is a useful caution for the graph soft-prompting literature.
- A well-designed diagnostic protocol. Separating token variation, recoverable content, and downstream use is a valuable conceptual contribution. CONST, which shares G-Retriever’s projector and changes only the token’s source, is a clean graph-versus-no-graph control. The idea of introducing a separate prompt token to carry the task context, so that the graph token has a simpler, more focused role, is also appealing.
- Broad replication and transparency. The phenomenon is replicated across many settings. The experiments cover three benchmarks, three graph encoders, two LLM families, and frozen and LoRA settings, and recurs in several later methods (GRAG, GRETPool, GraSP).
Critical weaknesses
W1. Collapse is defined only in relative terms. Cosine and residual ratio are measured against a mean token 22–97× the norm of a word embedding. In absolute terms the residual is still about 1.1–7.5 word-embedding norms (Table 11).
W2. Less collapse does not mean better performance. In Table 3, the most collapsed configurations perform best (85.56, 84.48), while the least collapsed one (cosine 0.323) falls to 57.76, near majority class. This contradicts the framing of collapse as a pathology.
W3. The proposed pipeline is not a convincing remedy. It only matches G-Retriever on WebQSP and drops on SceneGraphs (72.50 vs. 81.35). It also trails the graph-free CONST on WebQSP (71.25) and SceneGraphs (81.40). The soft prompt is relocated to a second token; the graph token does not become more useful. On WebQSP the token mainly encodes the question and answer (ranks 65/60) rather than the graph (315 vs. control 407). Stage 1 memorises answers (top-1 about 0.9 on training vs. 0.5 on validation), and under cross-fitting ExplaGraphs accuracy falls below the baseline (76.35 vs. 80.87).
W4. Key comparisons are single-seed. G-Retriever’s ExplaGraphs initialisations range from 85.56 to 89.71, while CONST (85.56) and the pipeline (85.74) each have one seed, so the claim that CONST matches G-Retriever on all benchmarks does not hold on ExplaGraphs. Multi-seed results and a per-dataset power analysis are needed.
W5. Presentation. The presentation of the method figure (Figure 2) is somewhat rough and could be improved. Most tables are in the appendix, with substantial duplication, and Appendix E contains a broken “Table ??” reference.
Questions for authors
Seen in Weaknesses.
5 comments
W2: less collapse does not mean better performance.
We agree, and this is one of the central findings of the paper. We do not argue that reducing token collapse necessarily improves task accuracy. We use ”pathological” to describe a mismatch between the intended/widely accepted role of the graph token of providing input-dependent graph information, and its observed function as a largely graph-independent soft prompt.
Our CONST and fixed-weight intervention results show that a model can achieve high accuracy even when the input-specific graph-token variation contributes little measurable information to its answers. The relevant question is therefore not whether collapse lowers accuracy, but whether the graph-token pathway performs the function attributed to it.
Our contribution is to identify and characterize token collapse, demonstrate its functional consequences through controlled interventions and provide an evaluation protocol that distinguishes token variability, graph-information recoverability, downstream utilization and task performance. The two-stage pipeline serves as a diagnostic proof of concept rather than a general performance remedy, and it serves as a starting point for future research to exploit graph-information within the supplied tokens.
W3: the pipeline is not a convincing remedy.
The pipeline was designed primarily to prevent collapse and investigate what non-collapsed graph tokens encode and whether the LLM uses them (RQ4). It is not intended as a generally superior GraphQA architecture. On the representation side, it achieves the intended change: token cosine decreases from to on WebQSP and to across our tested configurations (Table 4).
Accuracy against G-Retriever and CONST. With Llama-2, the pipeline maintains similar accuracy to G-Retriever on ExplaGraphs, is approximately one point lower on WebQSP across runs (W4) and performs worse on SceneGraphs ( vs. ). We acknowledge that the pipeline does not consistently improve accuracy, nor does it consistently outperform CONST. The largest improvements in Table 4 occur with Qwen2.5-3B, for which we had not included CONST. We have now trained this missing control with the same training budget as the Qwen configurations (30 epochs, early stopping patience of 4):
| Qwen2.5-3B | ExplaGraphs | WebQSP |
|---|---|---|
| G-Retriever (Table 4) | 74.37 | 57.13 |
| CONST (new) | 85.38 | 56.94 |
| Ours (Table 4) | 83.94 | 65.79 |
On ExplaGraphs, graph-free CONST scored 85.38, compared with 83.94 for our pipeline (exact McNemar p = .28) and 74.37 for G-Retriever. This shows that a graph-free soft prompt can achieve a similar gain in this setting. The pipeline’s graph token was decodable, and graph replacement produced no detectable accuracy effect (Table 66). On WebQSP, our pipeline outperformed CONST by 8.85 Hit@1 points, while CONST and G-Retriever scored 56.94 and 57.13, respectively (p = .89).
Does the graph token become more useful? Moving the soft prompt to a second token is what frees the graph token to carry input-specific information, and in several settings the LLM then uses it, while G-Retriever's token has no resolved effect: replacing the graph supplied through the token (GSWAP) costs 5.30 points on SceneGraphs with Llama-2, 6.50 on SceneGraphs with Qwen and 2.09 on WebQSP with Qwen (Table 4). On WebQSP with Qwen, this use comes with the accuracy gain over CONST above. Of course, here we mainly uncover this phenomenon, and understanding how to control what these tokens have to encode becomes the real challenge for future works.
What the WebQSP token encodes. We agree, and this is why Table 5 labels the standard WebQSP token MASKED rather than USED. The same architecture trained without question conditioning is used (GSWAP −2.33, Table 4), so what a non-collapsed token carries, and whether it is used, depends on the training inputs. These experiments show that removing collapse does not determine what the graph token encodes or whether the LLM uses it. The resulting behavior depends on the training inputs and benchmark, which motivates evaluating graph-token use directly rather than inferring it from representation geometry or accuracy.
Stage 1 memorisation. We ran the cross-fitting of Appendix P to test exactly this: Stage 1 and Stage 2 see disjoint halves of the training questions, so Stage 2 cannot rely on answers that Stage 1 memorised. The WebQSP token is still used (GSWAP −2.83, Table 69), so its use is not only memorised answers. On ExplaGraphs, as you note, cross-fitting costs accuracy (76.35 vs 80.87, Table 72).
As this might cause confusion, we will better state these limits in Section 4.4. More generally, our audit lets us check whether a training change makes the LLM use graph information supplied through the token, rather than inferring it from accuracy.
W4: key comparisons are single-seed.
We agree. We have now trained additional runs using the same training recipe as the paper, varying only the random seed. Run 1 corresponds to the main results in Tables 2 and 4. G-Retriever's first four ExplaGraphs initializations were already reported in Tables 14–17; the remaining runs are new. SceneGraphs remains at one run per model for now, since a single run takes more than 12 hours.
ExplaGraphs, accuracy:
| Model | Runs | Mean SD |
|---|---|---|
| G-Retriever | 85.56 / 86.28 / 87.91 / 89.71 / 85.74 | |
| CONST | 85.56 / 85.02 / 85.20 / 88.63 / 87.18 | |
| Ours | 85.74 / 87.55 / 85.56 / 84.66 / 87.18 |
WebQSP, Hit@1:
| Model | Runs | Mean SD |
|---|---|---|
| G-Retriever | 70.33 / 71.62 / 70.58 / 70.58 | |
| CONST | 71.25 / 69.90 / 70.15 / 70.39 | |
| Ours | 69.90 / 69.23 / 69.23 / 70.27 |
Across initializations, CONST and G-Retriever are within one point of each other on both benchmarks: CONST minus G-Retriever is −0.72 on ExplaGraphs (95% Welch interval −3.14 to +1.70) and −0.35 on WebQSP (−1.36 to +0.65). These intervals do not establish statistical equivalence, particularly on ExplaGraphs, but they show that the two methods have similar mean performance at the resolution supported by the available runs. We will update our manuscript with this fact. We will also report that our pipeline is 1.12 points below G-Retriever on WebQSP (−2.07 to −0.17) and 0.77 points below CONST (−1.72 to +0.19).
Per-dataset power analysis. We also estimated the smallest detectable effect at power and , expressed in accuracy points:
| Comparison | ExplaGraphs | WebQSP | SceneGraphs |
|---|---|---|---|
| Between models (Welch) | 3.36 | 1.38 | n/a (1 run) |
| MEAN (McNemar) | 1.52 | 2.28 | 1.66 |
| SWAP (McNemar) | 1.96 | 2.38 | 1.69 |
| GSWAP (McNemar) | 1.75 | 2.42 | 1.68 |
Between-model estimates use the available training runs; intervention estimates use the observed paired prediction changes in G-Retriever run 1. These estimates clarify the resolution of our experiments. The intervention tests have approximately estimated power for effects of points, depending on the dataset and perturbation. Smaller effects may remain undetected, particularly on WebQSP. We will report these values alongside our two-point criterion in Section 4.1 and avoid interpreting nonsignificant differences as proof of zero effect.
Importantly, the central finding does not require CONST to achieve exactly the same accuracy as G-Retriever. The key observation is that a learned graph-independent prompt recovers much of G-Retriever's performance, while replacing the graph-conditioned token variation has little resolved effect in the standard configurations. The new multi-seed comparisons support this interpretation more reliably than the original single-run results.
W5: presentation.
Thank you. We will improve Figure 2, reduce duplication in the appendix and correct the broken reference in Appendix G (which should point to Table 11).
We hope the new experiments and clarifications address your main concerns and would appreciate your reconsideration of the overall assessment.
Thank you for the detailed review. We have conducted three additional experiments to address your concerns -- residual amplification (W1), a graph-free CONST control for Qwen2.5-3B (W3) and multi-seed evaluations (W4).
W1: collapse is defined only in relative terms.
We agree that a small residual relative to the mean token can still be substantial in absolute terms and could contain information that the LLM uses. To test this directly, we amplified the input-specific residual while keeping the trained model and textual inputs fixed.
Setting. We decompose each graph token as ti=μ+ri, where μ is the mean token over the training questions (all questions for ExplaGraphs; a random sample of 2,000 for WebQSP) and ri is the input-specific part. Using the five ExplaGraphs G-Retriever initializations and the four WebQSP ones, we replace each token with:
We test k∈{1,2,5,10,20}. At k=1, Own corresponds to the original token and Swapped to SWAP. We report accuracy on ExplaGraphs and Hit@1 on WebQSP, and compare variants with the exact McNemar test.
Results. Numbers are in the table below. At k=1, the size of the trained model produces, replacing the input-specific part with another example has no significant effect in any run (on WebQSP +0.37 / −1.60 / −0.92 / −1.11, p ≥ .07). On WebQSP, once the part is amplified, the answers depend on which part is supplied: another example's part is significantly worse than the question's own in three of the four runs, for example at k=5 by 3.69 / 3.13 / 1.90 / 2.58 points (p < .001 / .02 / .14 / .046). On ExplaGraphs we did not detect a difference between the two at any k in any of the five runs (largest gap 3.07 points, p = .15), although larger values of k hurt accuracy. A random direction of the same norm hurts less than the learned part at large k: at k=20 by 13.94 / 11.79 / 15.11 / 10.69 points on WebQSP (p < 10−15) and by 8.30 / 7.04 / 8.12 points in ExplaGraphs runs 2–4 (p < .005).
ExplaGraphs, accuracy (run 1 / 2 / 3 / 4 / 5):
WebQSP, Hit@1 (run 1 / 2 / 3 / 4):
These results confirm part of your concern: the residual is not empty, and its content can affect predictions when amplified. Our claim is narrower. At the scale learned by standard G-Retriever, replacing the input-specific residual has no resolved answer-level effect in these configurations. The token therefore behaves primarily as a shared prompt under its normal operating conditions, despite containing non-zero input-specific variation.
We will include this analysis in the revision and report the absolute residual magnitude in word-embedding units more prominently alongside the relative collapse measures.
The other weaknesses are addressed in following comments.