- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
The paper follows the GNN-as-prefix line (Rel-LLM style), where GNN embeddings of a relbench entity subgraph are projected into the input space of a frozen LLM as "soft prompt" tokens. It claims such freely projected prompts are "semantically blind" (off the "semantic manifold" of token embeddings) and instead mixes each token, through a learned gate, with a convex combination over a fixed dictionary of embedded schema descriptions (names and templated statistics), plus a task-conditioned smoothing of the tokens and cross-database pretraining. A "Maurey-type" bound and gains over GNN baselines and Rel-LLM on the RelBench aggregates are also reported.
This sits at the border of RDL (GNNs for DBs) and NLP/LLMs, which is a bit new to me (though I know RDL well), so I might be missing something, but the whole paper reads to me as over-complicated, obfuscating instead of explaining -- an undefined core idea, theory merely as decoration, and ablations that don't really answer the essential question...
Critical strengths
-
The evaluation is broad (V1 and V2, zero-shot and task-specific) and the aggregates look good at face value (though you lose to both RelGNN and Rel-LLM on Amazon, H&M and Avito click) - if the comparison with Rel-LLM is clean (see questions), this would be a solid empirical result, which keeps me from a clear reject.
-
The mechanism underneath (a frozen schema-text codebook as a prior/regularizer on the prompt projector) is cheap and reasonable, and an honest version framed as such could be a nice short empirical/workshop contribution, I think.
Critical weaknesses
-
The intro is not very understandable to me - what is the "structured semantic manifold"? It sounds very flowery but doesn't really explain... Why say on-manifold instead of in-distribution? You don't seem to assume anything about that manifold anyway - as far as I can tell, you build a convex hull over the "anchors" and talk inside it only, with a Lipschitz constant that is never estimated (and App. F.4 then confirms the "manifold" talk is just informal, saying you don't claim free prompts lie outside the token-embedding hull, or at any particular distance from it, and redefines blindness as "the absence of a certificate", i.e. not a performance claim at all) -- so the abstract's "we show" isn't really shown anywhere, I think? Plus the anchors are averages of token embeddings (Eq. 3), which the LLM (presumably) never sees as input during training either... Sec. 4 then embeds the schema text and stacks it into a matrix A (schema names/statistics being the standard embeddings in RDL foundation models anyway, so I don't see much novelty in 4.1 afaik), 4.2 seems like a rather ad-hoc mixing of the graph tokens with their neighbourhood means (conditioned on the task text and the seed), which per C.3 doesn't even help Rel-LLM, and the core 4.3 assigns the GNN embedding (via a softmax) to a convex combination of these text embeddings -- which looks to me just like a classic attention head (with a fixed basis), i.e. more of a restriction of the normal projection you argue against (and which is moreover kept and mixed back in through the gate), so what is the difference between "assignment" and "projection" actually? Couldn't a plain projection learn that itself through the loss anyway? And being in the convex hull of these (averaged) description texts doesn't make a prompt in-distribution either, I guess (your own Prop. 2 says as much about the token hull) The theory in Sec. 5 then doesn't help: Theorem 1 assumes the target already in the hull, so how does the sparsification bound relate to the anchoring at all? It looks merely like theory as decoration to me...
-
The experiments also don't settle it for me. The anchoring ablations keep the anchoring in place and only damage it (random/rotated dictionaries, other assignment rules...), which mostly shows that damaging it hurts (and even then the random, meaningless tokens cost only ~~5 points, so I'm not sure what the "semantics" even does here...) The essential ablation seems missing to me -- provide the same information (schema text, names, stats) and simply feed it to the LLM next to the GNN node embeddings with some plain projection -- i.e. no anchoring. Even worse, all the controls behave suspiciously similarly (~6 points down whatever is removed, and "both off" with no anchors beating each single ablation - Sec. 6.3 reads this as coupled pathways, I'd rather read it as noise, but I might be wrong...), so I'm afraid the margins might measure the (un)tuning of the controls rather than the components. With no seeds/std reported and Rel-LLM taken from its published numbers rather than rerun in your pipeline, the headline aggregates are hard for me to trust.
Questions for authors
- Does the plain baseline above (schema text fed next to the GNN prompt of Rel-LLM, no anchoring) perform worse than RelAnchor? This is the result that would change it for me...
- Your "both off" control (Table 18), i.e. your pipeline without any anchors, sits several points below the published Rel-LLM on all six of those classification tasks (Table 1) - what explains that gap, if the anchoring is the ingredient?
- The cross-DB pretraining apparently applies to RelAnchor only (C.3) - please give the numbers behind C.3's "comparable" from-scratch claim (with seeds... and say whether the test metrics use the full test split).
- Why does Event (an events table that is mostly anonymized) show the largest classification gains, with the zero-shot model beating every task-specific baseline on both tasks (and your own task-specific model on user-repeat), while losing on the Event regression task?
Suggestion: say plainly what the method is (a frozen schema-codebook prior on the projector), drop the manifold/blindness/paradigm-shift vocabulary from the title and abstract, and cut Sec. 5 to a remark (or actually measure L_S).
9 comments
Rebuttal to Reviewer p2CC – Part(2/5)
2. The gap between "Both off" and published Rel-LLM (Critical Question 2)
"Both off" (no semantic injection: , ) sits several points below published Rel-LLM on classification. The gap has three identified sources, none related to anchoring.
(a) Pretraining vs. from-scratch.
- "Both off" follows LODO pretraining plus target fine-tuning, while published Rel-LLM trains from scratch on each target task. As App. C.3 discusses, pretraining mainly accelerates convergence rather than improving final performance, so the two results use different training protocols and are not a strictly matched comparison.
- Our Rel-LLM + LODO pretraining control shows that cross-database pretraining does not automatically improve free-projection models: it obtains 76.16 mean AUROC and 0.9002 , versus 77.82 and 0.8847 for published Rel-LLM.
- Although this is not a strictly paired comparison under an identical implementation, it reinforces the need to distinguish training protocols when interpreting the gap.
(b) Retention module. Even with both semantic pathways zeroed, Query-Conditioned Retention (Sec. 4.2) still mixes graph tokens with neighbourhood summaries. This changes the prompt layout relative to Rel-LLM's bare projection. The mixing was designed to work with the semantic pathways; without them, it introduces noise rather than useful context.
(c) Hyperparameters. "Both off" uses the hyperparameters tuned for full RelAnchor (learning rate, dropout, batch size) and is not separately tuned. Rel-LLM's published numbers reflect its own hyperparameter search.
The correct comparison for isolating anchoring is therefore not "both off" vs. published Rel-LLM. It is the controlled ablations of Tables 15–16 and Fig. 3, where every variant shares architecture, pretraining, and hyperparameters and only the component under study changes. Under those conditions, every degraded anchoring variant loses to full RelAnchor on every task.
On the pretraining difference in (a), see also Section 1 above: RelAnchor trained from scratch beats our from-scratch Rel-LLM rerun by 2.71 mean AUROC and 1.51 mean MAE. This further supports the contribution of Schema Anchoring even without cross-database pretraining.
3. From-scratch training numbers with seeds (Critical Question 3)
Below are RelAnchor results trained from scratch (no LODO pretraining) over 3 seeds. Test metrics use the full test split.
| Method | Mean AUROC (6 cls.) ↑ | Mean MAE (4 reg.) ↓ |
|---|---|---|
| RelAnchor (pretrained) | 84.26 | 15.97 |
| RelAnchor (from scratch), seed 45 | 84.07 | 16.05 |
| RelAnchor (from scratch), seed 46 | 83.34 | 16.31 |
| RelAnchor (from scratch), seed 47 | 84.15 | 15.67 |
| RelAnchor (from scratch), mean ± std | 83.86 ± 0.45 | 16.01 ± 0.32 |
| Rel-LLM (published, matched tasks) | 82.58 | 16.59 |
| Rel-LLM (from scratch, our rerun) | 81.36 | 17.56 |
As stated in App. C.3, from-scratch RelAnchor achieves final performance comparable to the pretrained version; LODO pretraining primarily accelerates convergence.
Rebuttal to Reviewer p2CC – Part(3/5)
4. The Event anomaly (Critical Question 4)
We thank the reviewer for highlighting this. Understanding these results requires distinguishing three things: the relational information Event preserves, the role of cross-database pretraining, and the different requirements of classification and regression.
- Anonymized event content does not eliminate user-behavior and relational information.
- Many event-content fields use anonymized identifiers, but Event still preserves users, event participation, invitations, interests, friendships, and timestamps.
- All three Event tasks target users, so the available signal is not limited to anonymized event content.
- RelAnchor can combine the entity roles and relations expressed by the schema with the behavioral structure encoded by the GNN, which provides a foundation for cross-database transfer.
- Cross-database pretraining may be an important source of the classification gains. The zero-shot model has undergone supervised pretraining on other databases; it simply has not seen Event's target-task labels. It can therefore transfer learned patterns of user activity, continued participation, and interaction behavior, which plausibly explains its strong zero-shot classification on Event.
- The slight zero-shot advantage on user-repeat may relate to target-task supervision and label quality.
- The difference is only 0.65 AUROC (88.96 vs. 88.31). On user-ignore, by contrast, fine-tuning raises AUROC from 88.35 to 92.92, so the benefit of fine-tuning is task-dependent.
- On user-repeat, overfitting, optimization variability, or target-label quality may prevent fine-tuning from improving on the pretrained model's generalization.
- This is not uncommon when fine-tuning pretrained models, and a similar pattern appears in the reported RT results.
- Transferable ranking ability does not guarantee numerical accuracy in count regression.
- User-repeat and user-ignore use AUROC, which mainly measures how well the model ranks positives against negatives. User-attendance instead requires predicting the actual number of events a user will attend in the next seven days, evaluated by MAE.
- Behavioral representations learned across databases may rank participation tendencies well while still needing adaptation to Event-specific count scales and target distributions.
- Strong zero-shot ranking and weaker count regression can therefore coexist. Notably, zero-shot RelAnchor's user-attendance MAE of 0.285 still beats the other zero-shot methods (0.379 and 0.461).
We will include this task-level discussion in the revised paper.
Rebuttal to Reviewer p2CC – Part(1/5)
We thank the reviewer for the detailed reading and for stating what would change their assessment. The key new results are:
1. The missing ablation: schema text alongside a free projection (Critical Question 1)
We agree this experiment is important, and we constructed exactly the baseline described:
This isolates whether the information content of the schema text alone, rather than how anchoring integrates it, explains the gains.
This baseline is a classic prompt-engineering strategy: supplying domain context to activate knowledge acquired during pretraining. It is beneficial, but activating relevant domain knowledge alone is insufficient for the LLM to understand individual graph tokens.
RelAnchor instead uses learned assignment to express each node's semantics as a weighted combination of schema anchors. Schema concepts thus participate directly in constructing graph-token representations, rather than merely serving as additional context. This experiment highlights the fundamental distinction between providing the LLM with domain knowledge and anchoring specific graph representations to it. The former improves prediction; the latter lets the LLM leverage relational representations more effectively. That is precisely the motivation for moving from free projection to learned assignment.
This is the functional difference between "assignment" and "projection" that we should have emphasized more clearly:
This is a hard structural constraint, not a soft preference the loss could discover.