acceptodds
Under review as a conference paper at ICLR 2027

Semantically Anchored Prompts for Relational Deep Learning

Abstract

Recent work injects GNN embeddings of relational databases into frozen large language models (LLMs) as soft prompts through a learned projection. We show that such prompts are semantically blind: the projection places them off the manifold of pre-trained token embeddings, so the LLM cannot tell a customer from a transaction and its world knowledge is never activated. We propose RelAnchor, which learns an assignment instead of a projection. Schema Anchoring embeds the database's table and column descriptions with the frozen input embedding layer, yielding an on-manifold anchor basis that spans exactly the semantics of the database. Adaptive Composition expresses each row as a row-specific convex mixture over these anchors, gated with its graph representation, so that every prompt preserves relational distinctions while staying anchored to named schema concepts near the manifold. We prove that anchoring costs at most in prompt fidelity with active anchors, governed by the LLM's Lipschitz constant on the compact on-manifold anchor hull rather than the unbounded ambient space. On RelBench, RelAnchor outperforms GNN-only baselines and the state-of-the-art graph-prompted LLM on the aggregate classification and regression metrics. Code is available at https://github.com/nmksjx/RelAnchor.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Relational Deep Learning, Large Language Models, Prompt Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_p2CCOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper follows the GNN-as-prefix line (Rel-LLM style), where GNN embeddings of a relbench entity subgraph are projected into the input space of a frozen LLM as "soft prompt" tokens. It claims such freely projected prompts are "semantically blind" (off the "semantic manifold" of token embeddings) and instead mixes each token, through a learned gate, with a convex combination over a fixed dictionary of embedded schema descriptions (names and templated statistics), plus a task-conditioned smoothing of the tokens and cross-database pretraining. A "Maurey-type" bound and gains over GNN baselines and Rel-LLM on the RelBench aggregates are also reported.

This sits at the border of RDL (GNNs for DBs) and NLP/LLMs, which is a bit new to me (though I know RDL well), so I might be missing something, but the whole paper reads to me as over-complicated, obfuscating instead of explaining -- an undefined core idea, theory merely as decoration, and ablations that don't really answer the essential question...

Critical strengths
  1. The evaluation is broad (V1 and V2, zero-shot and task-specific) and the aggregates look good at face value (though you lose to both RelGNN and Rel-LLM on Amazon, H&M and Avito click) - if the comparison with Rel-LLM is clean (see questions), this would be a solid empirical result, which keeps me from a clear reject.

  2. The mechanism underneath (a frozen schema-text codebook as a prior/regularizer on the prompt projector) is cheap and reasonable, and an honest version framed as such could be a nice short empirical/workshop contribution, I think.

Critical weaknesses
  1. The intro is not very understandable to me - what is the "structured semantic manifold"? It sounds very flowery but doesn't really explain... Why say on-manifold instead of in-distribution? You don't seem to assume anything about that manifold anyway - as far as I can tell, you build a convex hull over the "anchors" and talk inside it only, with a Lipschitz constant that is never estimated (and App. F.4 then confirms the "manifold" talk is just informal, saying you don't claim free prompts lie outside the token-embedding hull, or at any particular distance from it, and redefines blindness as "the absence of a certificate", i.e. not a performance claim at all) -- so the abstract's "we show" isn't really shown anywhere, I think? Plus the anchors are averages of token embeddings (Eq. 3), which the LLM (presumably) never sees as input during training either... Sec. 4 then embeds the schema text and stacks it into a matrix A (schema names/statistics being the standard embeddings in RDL foundation models anyway, so I don't see much novelty in 4.1 afaik), 4.2 seems like a rather ad-hoc mixing of the graph tokens with their neighbourhood means (conditioned on the task text and the seed), which per C.3 doesn't even help Rel-LLM, and the core 4.3 assigns the GNN embedding (via a softmax) to a convex combination of these text embeddings -- which looks to me just like a classic attention head (with a fixed basis), i.e. more of a restriction of the normal projection you argue against (and which is moreover kept and mixed back in through the gate), so what is the difference between "assignment" and "projection" actually? Couldn't a plain projection learn that itself through the loss anyway? And being in the convex hull of these (averaged) description texts doesn't make a prompt in-distribution either, I guess (your own Prop. 2 says as much about the token hull) The theory in Sec. 5 then doesn't help: Theorem 1 assumes the target already in the hull, so how does the sparsification bound relate to the anchoring at all? It looks merely like theory as decoration to me...

  2. The experiments also don't settle it for me. The anchoring ablations keep the anchoring in place and only damage it (random/rotated dictionaries, other assignment rules...), which mostly shows that damaging it hurts (and even then the random, meaningless tokens cost only ~~5 points, so I'm not sure what the "semantics" even does here...) The essential ablation seems missing to me -- provide the same information (schema text, names, stats) and simply feed it to the LLM next to the GNN node embeddings with some plain projection -- i.e. no anchoring. Even worse, all the controls behave suspiciously similarly (~6 points down whatever is removed, and "both off" with no anchors beating each single ablation - Sec. 6.3 reads this as coupled pathways, I'd rather read it as noise, but I might be wrong...), so I'm afraid the margins might measure the (un)tuning of the controls rather than the components. With no seeds/std reported and Rel-LLM taken from its published numbers rather than rerun in your pipeline, the headline aggregates are hard for me to trust.

Questions for authors
  • Does the plain baseline above (schema text fed next to the GNN prompt of Rel-LLM, no anchoring) perform worse than RelAnchor? This is the result that would change it for me...
  • Your "both off" control (Table 18), i.e. your pipeline without any anchors, sits several points below the published Rel-LLM on all six of those classification tasks (Table 1) - what explains that gap, if the anchoring is the ingredient?
  • The cross-DB pretraining apparently applies to RelAnchor only (C.3) - please give the numbers behind C.3's "comparable" from-scratch claim (with seeds... and say whether the test metrics use the full test split).
  • Why does Event (an events table that is mostly anonymized) show the largest classification gains, with the zero-shot model beating every task-specific baseline on both tasks (and your own task-specific model on user-repeat), while losing on the Event regression task?

Suggestion: say plainly what the method is (a frozen schema-codebook prior on the projector), drop the manifold/blindness/paradigm-shift vocabulary from the title and abstract, and cut Sec. 5 to a remark (or actually measure L_S).

9 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer p2CC – Part(1/5)

We thank the reviewer for the detailed reading and for stating what would change their assessment. The key new results are:

  • (R1) the requested "plain-baseline" ablation;
  • (R2) from-scratch numbers with seeds;
  • (R3) an explanation of the gap between our pipeline without anchors and published Rel-LLM.

1. The missing ablation: schema text alongside a free projection (Critical Question 1)

We agree this experiment is important, and we constructed exactly the baseline described:

  • the same metadata set Ameta\mathcal{A}_{\text{meta}} is serialized as a text string and embedded via the frozen token-embedding table EE;
  • these tokens are concatenated into the LLM input next to Rel-LLM's freely projected GNN tokens, with no anchoring;
  • the projector, GNN, and task head are identical to Rel-LLM.

This isolates whether the information content of the schema text alone, rather than how anchoring integrates it, explains the gains.

MethodMean AUROC (6 cls.) ↑Mean MAE (4 reg.) ↓
Rel-LLM (our rerun)81.3617.56
Rel-LLM + schema text appended82.3016.95
RelAnchor (no pretraining)84.0716.05
RelAnchor (full)84.2615.97
  • Simply providing schema-text tokens does help: +0.94 AUROC and −0.61 MAE over Rel-LLM.
  • It still falls short of RelAnchor trained from scratch, which adds a further +1.77 AUROC and −0.90 MAE.

This baseline is a classic prompt-engineering strategy: supplying domain context to activate knowledge acquired during pretraining. It is beneficial, but activating relevant domain knowledge alone is insufficient for the LLM to understand individual graph tokens.

  • Appended schema text tells the LLM which entities, attributes, and relations exist in the database.
  • The freely projected graph tokens, however, still lack any explicit correspondence to these concepts.
  • So even with the relevant context available, the LLM cannot reliably identify the entity roles and relational meanings that individual graph tokens represent.

RelAnchor instead uses learned assignment to express each node's semantics as a weighted combination of schema anchors. Schema concepts thus participate directly in constructing graph-token representations, rather than merely serving as additional context. This experiment highlights the fundamental distinction between providing the LLM with domain knowledge and anchoring specific graph representations to it. The former improves prediction; the latter lets the LLM leverage relational representations more effectively. That is precisely the motivation for moving from free projection to learned assignment.

This is the functional difference between "assignment" and "projection" that we should have emphasized more clearly:

  • A free projection could in principle learn to land near schema-text embeddings, but the downstream loss never incentivizes it to. The loss cares only about the final prediction and is agnostic to whether intermediate representations are interpretable through the LLM's pretrained knowledge.
  • Anchoring constrains the output space to the schema-specific anchor hull, so the LLM is guaranteed to receive tokens it can parse through pretrained associations.

This is a hard structural constraint, not a soft preference the loss could discover.

AuthorsOpenReview ↗

Rebuttal to Reviewer p2CC – Part(2/5)

2. The gap between "Both off" and published Rel-LLM (Critical Question 2)

"Both off" (no semantic injection: α=0\alpha = 0, sG=0s_G = 0) sits several points below published Rel-LLM on classification. The gap has three identified sources, none related to anchoring.

(a) Pretraining vs. from-scratch.

  • "Both off" follows LODO pretraining plus target fine-tuning, while published Rel-LLM trains from scratch on each target task. As App. C.3 discusses, pretraining mainly accelerates convergence rather than improving final performance, so the two results use different training protocols and are not a strictly matched comparison.
  • Our Rel-LLM + LODO pretraining control shows that cross-database pretraining does not automatically improve free-projection models: it obtains 76.16 mean AUROC and 0.9002 μn\mu_n, versus 77.82 and 0.8847 for published Rel-LLM.
  • Although this is not a strictly paired comparison under an identical implementation, it reinforces the need to distinguish training protocols when interpreting the gap.

(b) Retention module. Even with both semantic pathways zeroed, Query-Conditioned Retention (Sec. 4.2) still mixes graph tokens with neighbourhood summaries. This changes the prompt layout relative to Rel-LLM's bare projection. The mixing was designed to work with the semantic pathways; without them, it introduces noise rather than useful context.

(c) Hyperparameters. "Both off" uses the hyperparameters tuned for full RelAnchor (learning rate, dropout, batch size) and is not separately tuned. Rel-LLM's published numbers reflect its own hyperparameter search.

The correct comparison for isolating anchoring is therefore not "both off" vs. published Rel-LLM. It is the controlled ablations of Tables 15–16 and Fig. 3, where every variant shares architecture, pretraining, and hyperparameters and only the component under study changes. Under those conditions, every degraded anchoring variant loses to full RelAnchor on every task.

On the pretraining difference in (a), see also Section 1 above: RelAnchor trained from scratch beats our from-scratch Rel-LLM rerun by 2.71 mean AUROC and 1.51 mean MAE. This further supports the contribution of Schema Anchoring even without cross-database pretraining.


3. From-scratch training numbers with seeds (Critical Question 3)

Below are RelAnchor results trained from scratch (no LODO pretraining) over 3 seeds. Test metrics use the full test split.

MethodMean AUROC (6 cls.) ↑Mean MAE (4 reg.) ↓
RelAnchor (pretrained)84.2615.97
RelAnchor (from scratch), seed 4584.0716.05
RelAnchor (from scratch), seed 4683.3416.31
RelAnchor (from scratch), seed 4784.1515.67
RelAnchor (from scratch), mean ± std83.86 ± 0.4516.01 ± 0.32
Rel-LLM (published, matched tasks)82.5816.59
Rel-LLM (from scratch, our rerun)81.3617.56

As stated in App. C.3, from-scratch RelAnchor achieves final performance comparable to the pretrained version; LODO pretraining primarily accelerates convergence.

AuthorsOpenReview ↗

Rebuttal to Reviewer p2CC – Part(3/5)

4. The Event anomaly (Critical Question 4)

We thank the reviewer for highlighting this. Understanding these results requires distinguishing three things: the relational information Event preserves, the role of cross-database pretraining, and the different requirements of classification and regression.

  1. Anonymized event content does not eliminate user-behavior and relational information.
    • Many event-content fields use anonymized identifiers, but Event still preserves users, event participation, invitations, interests, friendships, and timestamps.
    • All three Event tasks target users, so the available signal is not limited to anonymized event content.
    • RelAnchor can combine the entity roles and relations expressed by the schema with the behavioral structure encoded by the GNN, which provides a foundation for cross-database transfer.
  2. Cross-database pretraining may be an important source of the classification gains. The zero-shot model has undergone supervised pretraining on other databases; it simply has not seen Event's target-task labels. It can therefore transfer learned patterns of user activity, continued participation, and interaction behavior, which plausibly explains its strong zero-shot classification on Event.
  3. The slight zero-shot advantage on user-repeat may relate to target-task supervision and label quality.
    • The difference is only 0.65 AUROC (88.96 vs. 88.31). On user-ignore, by contrast, fine-tuning raises AUROC from 88.35 to 92.92, so the benefit of fine-tuning is task-dependent.
    • On user-repeat, overfitting, optimization variability, or target-label quality may prevent fine-tuning from improving on the pretrained model's generalization.
    • This is not uncommon when fine-tuning pretrained models, and a similar pattern appears in the reported RT results.
  4. Transferable ranking ability does not guarantee numerical accuracy in count regression.
    • User-repeat and user-ignore use AUROC, which mainly measures how well the model ranks positives against negatives. User-attendance instead requires predicting the actual number of events a user will attend in the next seven days, evaluated by MAE.
    • Behavioral representations learned across databases may rank participation tendencies well while still needing adaptation to Event-specific count scales and target distributions.
    • Strong zero-shot ranking and weaker count regression can therefore coexist. Notably, zero-shot RelAnchor's user-attendance MAE of 0.285 still beats the other zero-shot methods (0.379 and 0.461).

We will include this task-level discussion in the revised paper.

Reviewer_zwd9OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes RelAnchor, a framework for relational prediction with frozen LLMs. It constructs semantic anchors from database schema metadata and learns row-specific compositions of these anchors, which are combined with GNN representations to form soft prompts. The model is evaluated in both cross-database zero-shot and task-specific settings on RelBench.

Critical strengths
  1. Strong predictive performance. RelAnchor achieves competitive results across a broad set of relational prediction tasks, with particularly encouraging zero-shot performance on unseen databases and schemas.

  2. An intuitive schema-aware prompt design. Constructing fixed anchors from schema metadata and learning their instance-dependent composition provides an interesting way to integrate relational representations with a pretrained LLM. The paper also includes extensive ablations examining the proposed components.

Critical weaknesses
  1. The cross-schema transfer mechanism is insufficiently specified. The paper does not clearly explain how the trainable relational encoder, particularly its column encoders and heterogeneous GNN parameters, handles unseen feature schemas and relation types. This is essential to its zero-shot claims.

  2. The sources and robustness of the performance gains remain unclear. The task-specific improvements over Rel-LLM are largely concentrated on F1 and Event, for which prior work has raised evaluation concerns. RelBench has also released dataset updates. Furthermore, many substantially different ablations exhibit surprisingly similar performance degradation, raising questions about what drives the improvements.

Questions for authors

Q1. Cross-schema GNN transfer. How are the PyTorch Frame column encoders and heterogeneous GNN parameters shared across databases with different feature types, distributions, node types, and edge types? How can the pretrained encoder directly process an unseen schema without target-specific training?

Q2. Dataset versions and evaluation reliability. Previous work, including RDBLearn, has raised leakage concerns regarding F1 and Event. Which RelBench version was used, and how were the reported baseline results obtained? Could the authors provide results excluding these potentially problematic datasets and clarify their consistency with updated evaluations?

Q3. Ablation behavior. Why do substantially different ablations consistently produce large and similar performance drops? In particular, even replacing a learned scalar gate with fixed values results in substantial degradation. Could the authors clarify the initialization, pretraining, and hyperparameter-tuning procedures for these variants, and report multi-seed results?

Q4. Pretraining and learned representations. Appendix C.3 states that training from scratch achieves comparable final performance. Could the authors provide quantitative results? It would also be helpful to report the learned gate values, retention weights, and anchor-assignment distributions, as well as zero-shot ablations isolating the contribution of schema anchoring.

Q5. Theoretical relevance. Theorem 1 assumes the target prompt already belongs to the convex hull of the anchors. What justifies this assumption for useful or optimal prompts? Since the actual model uses dense assignments and gated prompts that need not belong to this hull, how does the theorem characterize the proposed method?

7 comments
AuthorsOpenReview ↗

Response to Reviewer zwd9 – Part(1/5)

We appreciate the reviewer's detailed reading and address each concern below.


Q1. Cross-schema GNN transfer

We agree this deserves explicit clarification. Cross-schema generalization of the underlying relational encoder is not RelAnchor's primary contribution. For a fair comparison, we follow Rel-LLM's architecture and adopt RT's cross-database weight-transfer strategy for Leave-One-Database-Out (LODO) evaluation. Our contribution is Schema Anchoring, which connects relational representations to the frozen LLM's pretrained semantic space.

Architecture follows Rel-LLM; weight transfer follows RT.

  • We reuse Rel-LLM's PyTorch Frame column encoders, heterogeneous GNN, natural-language task instructions, and task-type-specific prediction heads.
  • PyTorch Frame dispatches columns by data type (Table 10): numerical (linear), categorical (embedding lookup), and textual (frozen sentence encoder, all-mpnet-base-v2). Missing numerical values are mean-imputed.
  • The task is specified by a natural-language instruction I\mathcal{I} read by the frozen LLM. One prediction head is shared across tasks of each type (one for binary classification, one for regression).
  • For LODO, following RT, applicable pretrained parameters are reused on the target database. Parameters with no pretrained counterpart remain randomly initialized.

This lets the model process unseen schemas without target-specific training, without requiring a newly designed cross-schema encoder.

Unseen relation types.

  • Following Rel-LLM, the heterogeneous GNN (nodes = rows, edges = foreign-key relations) has separate message-passing parameters for each foreign-key relation type.
  • Under RT-style transfer, parameters for compatible relation types are reused, while those for unseen relation types remain randomly initialized during zero-shot inference.
  • We do not explicitly design shared cross-database message-passing functions.

RelAnchor nevertheless achieves strong zero-shot performance under this protocol.

Schema Anchoring adapts to the target schema without retraining. The anchor matrix AA is recomputed from the target schema metadata via frozen embedding lookup, with no training (App. C.2, lines 1428–1435). Because assignment is a softmax over whichever dictionary is supplied, the query network needs no change when MM changes.

In summary, we reuse Rel-LLM's architecture and RT's weight-transfer strategy rather than introducing a new cross-schema GNN or transfer mechanism. RelAnchor's contribution is making the resulting relational representations accessible to the frozen LLM through schema-specific anchors:

  • the anchor matrix AA and entity graph are built per target database;
  • reusable pretrained parameters are transferred;
  • unseen relation-specific parameters remain randomly initialized.

This separation lets us evaluate Schema Anchoring without conflating it with improvements to the relational encoder.

AuthorsOpenReview ↗

Response to Reviewer zwd9 – Part(2/5)

Q2. Dataset versions and evaluation reliability

RelBench version. We use the public relbench v2.1.1 package with the official temporal splits. This version includes the maintainers' fix for temporal leakage in rel-event/user-ignore (PR #357, merged February 2026, first included in v2.1.0), so our results on this task use the corrected data.

Baseline provenance.

  • For GraphSAGE, HGT, HGTPE, RelGNN, RelGT, and Rel-LLM (Wu et al., 2025), we report published results. Our local reproductions using the environments of the original repositories were consistent with these numbers, so we retain the published values.
  • LightGBM and DS+LightGBM correspond to the original tabular features and automated feature-engineering pipelines, respectively.
  • All RelAnchor results are from our own experiments, with validation-selected checkpoints evaluated on the test split (App. C.5, lines 1507–1535).

Leakage and evaluation concerns regarding F1 and Event. We acknowledge these concerns, but the identified issues do not necessarily affect every task in these databases.

  • Event user-ignore: the temporal leakage was fixed in v2.1.0, which we use.
  • Event user-repeat: the training-label generation issue was only addressed in v3.0.0, so we exclude this task.
  • F1: the unusually high ~99.61 AUROC discussed by RDBLearn is on driver-top3. Given additional concerns about inconsistent evaluation-time information across models, we conservatively exclude the entire F1 database rather than only that task.
  • Event user-attendance: we found no specific reports of leakage or task-level corrections, so we retain it.

Recomputing the aggregates from the per-task results in Tables 1–2 leaves 6 databases, 9 classification tasks, and 8 regression tasks:

Table R4: Task-specific performance excluding tasks with known label issues or evaluation concerns

MethodMean AUROC ↑ (9 cls.)Raw Avg. MAE ↓ (8 reg.)Normalized MAE μn\mu_n ↓
LightGBM61.5015.27961.0000
DS+LightGBM75.5412.05840.8709
GraphSAGE76.1013.70740.9126
RelGNN77.3513.51910.8552
RelGT76.0413.48890.8614
Rel-LLM77.2413.34900.8764
RelAnchor78.4811.47880.7993

Raw Avg. MAE is the unweighted mean of the original MAEs over the 8 regression tasks. μn\mu_n follows App. C.1 (Eq. 14), dividing each task's MAE by the LightGBM MAE before averaging.

  • RelAnchor still achieves the best results on all three aggregate metrics. Its mean AUROC of 78.48 exceeds Rel-LLM by 1.24 and RelGNN by 1.13 points, and its μn\mu_n of 0.7993 is the best of all methods.
  • As a stricter check, we also excluded F1 and Event entirely (8 classification, 7 regression tasks). RelAnchor still achieves the highest mean AUROC (76.68) and the lowest μn\mu_n (0.7609).

RelAnchor's aggregate advantage therefore does not depend on the disputed F1 and Event tasks, and it holds under different exclusion criteria.

AuthorsOpenReview ↗

Response to Reviewer zwd9 – Part(3/5)

Q3. Ablation behavior

We respectfully disagree that "substantially different ablations consistently produce large and similar performance drops." The ablations differ meaningfully in magnitude and, more importantly, in their per-task patterns.

The aggregate drops are not similar. In Table 14, mean AUROC drops range from −4.50 (generic vocabulary) to −6.96 (fixed α\alpha=0.10). Mean MAE increases range from +5.9% (task-shared retention) to +23.3% (fixed α\alpha=1.00). That is a 1.55× range on AUROC and a 3.9× range on MAE, far from "surprisingly similar."

Per-task patterns differ qualitatively (Figs. 12–17, Tables 15–21):

  • Generic vocabulary (Table 15) concentrates damage on F1 (−5.41 dnf, −13.41 top3) while barely affecting Stack (−2.30 engagement, −3.83 badge). Schema-specific naming matters most when tables are described by short, highly specific identifiers, as in F1.
  • Table-shared mixture (Table 16) hits regression hardest (+21.9% MAE on driver-position, +14.9% on item-ltv), while classification losses are moderate and uniform (−2.71 to −13.12). Row specificity matters most for continuous targets.
  • Local-only retention (Table 20) produces the single largest drop (−19.86 on driver-top3) but is competitive on Stack engagement (−2.63). Multi-view neighbourhood context is critical where the label depends on relations beyond the seed entity.

These are qualitatively different failure modes that isolate different mechanisms. When components are complementary, every control is expected to lose; the diagnostic value lies in where each control loses, not merely that it loses.

Gate ablation. The optimal fixed value is task-dependent (Table 19, Fig. 16): 0.75 for driver-dnf, 0.25 for driver-top3, 0 for item-churn, and 0.50 for item-ltv.

  • Any single fixed value is a compromise, and a wrong one is costly: item-ltv MAE rises from 47.28 to 61.03 at α\alpha=1.0.
  • The learned gate finds the per-model optimum at the cost of one scalar parameter (Sec. 6.3, lines 438–443; App. E.6).

Initialization, training, and tuning. All variants share full RelAnchor's setup, and only the ablated mechanism changes (App. E.1, Table 13):

  • training recipe: LODO pretraining followed by target-task fine-tuning;
  • update budget: 128K steps;
  • model selection: validation every 2K updates, early stopping after 3 evaluations;
  • the same per-task hyperparameter grid.

The gate is initialized at σ(a)=0.1\sigma(a)=0.1 (App. B.4, line 1209).

Multi-seed results (seeds 45, 46, 47; to be added in the revision):

  • In every seed, full RelAnchor outperforms all four controls on the six-task classification mean, by 4.21–5.99 AUROC points on average.
  • Seed-level standard deviations (0.04–0.64) are substantially smaller than these gaps, supporting the robustness of our conclusions.
VariantMean AUROC (seed 45 / 46 / 47)Mean ± std
RelAnchor (full)84.26 / 83.00 / 83.8383.69 ± 0.64
Generic vocab.79.76 / 79.83 / 78.8679.48 ± 0.54
Uniform mixture78.09 / 78.28 / 77.8278.07 ± 0.23
Node off (α=0\alpha=0)77.85 / 77.78 / 77.7777.80 ± 0.04
Local only77.64 / 77.30 / 78.1777.70 ± 0.44

Each seed's AUROC is averaged over the six classification tasks; std is the sample standard deviation across the three seed-level averages.

Reviewer_KhqBOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RelAnchor, which uses relational database schema descriptions to construct semantic anchors in the embedding space of a frozen LLM.

Instead of relying only on projected GNN embeddings the method builds row-specific mixtures of schema anchors and graph representations.

Evaluation on RelBenchV1 and V2 shows improvements in classification and regression tasks.

Critical strengths

Interesting and practical idea. Using database schema information to connect relational representations to the language model makes sense.

Good results across a broad set of relational prediction tasks.

The rotated-anchor and generic-vocabulary ablations provide useful evidence that schema-specific anchors matter.

Critical weaknesses

I find the theoretical justification less convincing. The main theorem bounds the approximation of a point already in the convex hull of the anchors. This does not establish that the actual gated prompts lie on the LLM's semantic manifold or activate pretrained world knowledge.

The performance gains are encouraging, but the comparison to Rel-LLM is not fully controlled.

RelAnchor uses cross-database pretraining, while published Rel-LLM results are from task-specific training. Appendix discusses this, but does not provide detailed numerical results for the matched controls.

Questions for authors

Can the authors provide a direct comparison to a free-projection baseline with the same backbone GNN, retention module, pretraining, and training budget?

How close are the actual learned prompts to the anchor hull? What are the learned gate values across tasks?

4 comments
AuthorsOpenReview ↗

Response to Reviewer KhqB – Part(1/2)

We appreciate the careful reading and address each concern below.


W1: Scope and strength of the theoretical justification

The reviewer states that Theorem 1 "bounds the approximation of a point already in the convex hull" and "does not establish that the actual gated prompts lie on the LLM's semantic manifold or activate pretrained world knowledge." We respectfully argue that the theorem and its supporting propositions establish more than this, and that the ablations supply the interventional evidence for world-knowledge activation.

  1. The value lies in the constant, not the containment. Theorem 1 bounds ∥F(p)−F(p∗)∥\|F(\mathbf{p}) - F(\mathbf{p}^*)\| by LSDA/2kL_S D_A / \sqrt{2k}, where LSL_S is the frozen LLM's Lipschitz constant restricted to the anchor hull SS. Free-projection prompts can lie anywhere in Rdℓ\mathbb{R}^{d_\ell} and can only invoke Lglob≥LSL_{\mathrm{glob}} \geq L_S. That constant compounds across layers and grows with input norm, so the free-projection bound "may be vacuous" (Remark 1, App. F.3). The theorem is therefore a separation result between two prompt-construction paradigms: anchored composition gets tighter, manifold-governed fidelity control. It is not a tautology about hull membership.
  2. Anchors are on the manifold by construction, not by assumption. Each anchor is the mean of frozen input-token embeddings of a real schema-metadata string (Eq. 3, Sec. 4.1, lines 201–209). These are points the LLM was pretrained to read; they carry the associated world knowledge and cannot drift during training. S=conv⁡({aj})S = \operatorname{conv}(\{\mathbf{a}_j\}) is thus a compact, on-manifold region with finite LSL_S (App. F.3, Eq. 19). We deliberately do not identify SS with the token-embedding hull HH. Prop. 2 (App. F.4) shows HH is full-dimensional, so membership in it carries no semantic information.
  3. Gated prompts are provably contracted toward SS. Every gated token satisfies dist⁡(pu,S)≤(1−α)dist⁡(g~u,S)\operatorname{dist}(\mathbf{p}_u,S) \leq (1-\alpha)\operatorname{dist}(\tilde{\mathbf{g}}_u,S) (Prop. 4, App. F.6). Since the learned gate α=σ(a)>0\alpha = \sigma(a) > 0 on every task (Table 19, Fig. 16), every prompt token is strictly closer to SS than the free graph vector. The ungated neighbourhood token sG\mathbf{s}_G is a pure anchor composition and lies exactly in SS (Eq. 13, App. B.4).
  4. The rotated-anchor ablation provides direct causal evidence for world-knowledge activation. We agree that no theorem alone proves world knowledge is activated; that requires an empirical intervention. The rotated-anchor ablation (Sec. 6.3, Fig. 3, Table 15) is designed for exactly this.
    • A random orthogonal QQ preserves every norm and pairwise inner product of the dictionary (Prop. 6, App. F.8). The mixture geometry is therefore identical, but the anchors leave the LLM's native embedding coordinates.
    • The resulting 5.75-point mean AUROC drop (Table 14) is direct causal evidence. What drives performance is alignment with the pretrained coordinate system (semantic grounding that can activate world knowledge), not mere membership in some convex set.
    • No distance measurement to HH could produce this kind of interventional evidence (App. F.4, lines 2415–2420).

Summary. The argument has four layers:

  • (i) Theorem 1: anchored composition invokes the tighter on-manifold LSL_S rather than the potentially vacuous LglobL_{\mathrm{glob}}.
  • (ii) Anchor construction guarantees that SS is on-manifold.
  • (iii) Prop. 4 certifies that gated prompts stay near SS.
  • (iv) The rotated-anchor ablation causally confirms that on-manifold alignment activates world knowledge.

We will restructure Sec. 5 to present this chain explicitly.

AuthorsOpenReview ↗

Response to Reviewer KhqB – Part(2/2)

W2: Controlled comparison with Rel-LLM

We acknowledge the presentation gap: App. C.3 (lines 1443–1474) reports the matched controls only qualitatively. We will tabulate full per-task numbers in the revision. Each of the two controls isolates one potential confound:

  • RelAnchor without pretraining (target task only, same anchoring) "achieves comparable final performance", and "the performance advantage over Rel-LLM is preserved under training from scratch" (lines 1450–1455). The gains therefore cannot be explained solely by the pretraining data; they stem primarily from anchoring.
  • Rel-LLM + Retention (the Sec. 4.2 module added to Rel-LLM's free projection) "yields no performance gain" (line 1457). This matches our analysis that retention in the off-manifold projection space "can attenuate useful variation without a compensating alignment benefit" (lines 1463–1465).

Table R1: Controlled comparison isolating pretraining and anchoring

MethodPretrainingPrompt typeMean AUROC ↑μn\mu_n ↓
Rel-LLM (published)None (task-specific)Free projection77.820.8847
Rel-LLM + RetentionNone (task-specific)Free projection77.430.8968
Rel-LLM + LODO pretrainingLODOFree projection76.160.9002
RelAnchor w/o pretrainingNone (task-specific)Schema anchoring79.870.7883
RelAnchor (full)LODOSchema anchoring80.580.7847

Q1: Free-projection baseline with the same backbone, Retention, pretraining, and training budget

Our paper already includes this control: the "Both off" configuration (Tables 14 and 18).

  • It sets the node gate to α=0\alpha = 0 and replaces the neighbourhood semantic token with a zero vector. This removes Schema Anchoring while keeping the freely projected graph vectors.
  • The GNN, Retention, pretraining protocol, and training budget are unchanged.
  • Mean AUROC drops from 84.26 to 78.83 (−5.43), and mean MAE rises from 15.97 to 17.31 (+8.4%).

Table R2: Free projection vs. RelAnchor under matched settings (AUROC ↑, from Table 18)

MethodF1 dnfF1 top3Stack eng.Stack badgeAmz. u-churnAmz. i-churnMean ↑
Free projection (Both off)71.1878.0988.3286.1568.5780.6578.83
RelAnchor (full)81.4990.7791.2990.2670.0281.7284.26

Q2: Proximity to the anchor hull and learned gate values

Learned gate values. RelAnchor uses a single learnable scalar α=σ(a)\alpha = \sigma(a) per model, shared across nodes. The fixed-gate sensitivity analysis over all ten ablation tasks (Table 19, Fig. 16) shows:

  • The learned gate outperforms every fixed α∈{0,0.10,0.25,0.50,0.75,1.00}\alpha \in \{0, 0.10, 0.25, 0.50, 0.75, 1.00\} on all ten tasks.
  • The best fixed value varies substantially across tasks: F1 driver-dnf favors 0.75 and driver-top3 favors 0.25; Amazon item-churn favors 0 and item-ltv favors 0.50.
  • The best-to-worst gap among fixed values is 2.78–4.98 AUROC points on classification. On Amazon item-ltv, α=1.00\alpha = 1.00 raises MAE by 11.02 relative to α=0.50\alpha = 0.50.

A fixed gate therefore cannot accommodate differing task requirements, whereas the learned gate adapts the injection strength to the training objective.

Proximity to the anchor hull: theory.

  • For pu=(1−α)g~u+αsup_u = (1-\alpha)\tilde{g}_u + \alpha s_u with su∈Ss_u \in S, Prop. 4 (App. F.6) gives dist(pu,S)≤(1−α) dist(g~u,S)\mathrm{dist}(p_u,S) \leq (1-\alpha)\,\mathrm{dist}(\tilde{g}_u,S).
  • Since α=σ(a)∈(0,1)\alpha = \sigma(a) \in (0,1), this bound is strictly smaller than the original distance whenever that distance is positive.
  • The neighbourhood token sG=αG⊤As_G = \alpha_G^\top A is ungated and lies exactly in SS (Eq. 13).

Proximity to the anchor hull: measurement. As requested, we extracted the learned gates and measured dist(pu,S)/dist(g~u,S)\mathrm{dist}(p_u,S)/\mathrm{dist}(\tilde{g}_u,S) on the test sets of six classification tasks. The ratio is computed for nodes with a positive denominator, and Prop. 4 bounds it above by 1−α1-\alpha.

Table R3: Learned gate values and relative distances to the anchor hull

TaskLearned α\alphaMean ratio ↓Median ratio ↓90th pctl. ↓
F1 driver-dnf0.360.6280.6280.629
F1 driver-top30.150.8330.8320.833
Stack user-engagement0.190.7960.7960.796
Stack user-badge0.130.8590.8590.859
Amazon user-churn0.220.7610.7610.762
Amazon item-churn0.040.9500.9500.951
  • Mean ratios of 0.628–0.950 correspond to a 5.0%–37.2% reduction in distance to the hull relative to the freely projected graph vectors.
  • These values are consistent with the Prop. 4 bounds, confirming the gate's contraction on the learned prompts.
  • The varying magnitudes across tasks reflect adaptive adjustment of the semantic injection strength.

Hi, Thank you for providing a detailed response. The new baseline is very helpful. Revised framing is also appreciated. Your responses addresses all my concerns. Thanks again!!

AuthorsOpenReview ↗

Thank You for Your Appreciation and Raising Score

Thank you very much for your encouraging response, and your positive assessment. We are delighted that the new baseline, additional experiments, and revised framing have addressed all your concerns.

Thank you again for your time, consideration, and support!

Comments on the paper

1 comment
AuthorsOpenReview ↗

Thank You and Summary of Updates (Revised PDF Available)

We sincerely thank all reviewers for their careful and thoughtful evaluations, as well as their valuable comments and suggestions throughout the discussion.

Based on the reviews and our discussions so far, we have conducted additional experiments, refined the analyses in both the main text and the appendix, and submitted an updated PDF of the manuscript for your consideration.

We look forward to continuing the discussion with all reviewers and warmly invite you to participate in the ongoing discussion at your earliest convenience. We will actively respond to any further questions or suggestions.

Once again, we sincerely appreciate your time and effort!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.