acceptodds
Under review as a conference paper at ICLR 2027

Toward Effective Relational Foundation Model: A Retrieval and Memory Based Approach

Abstract

Relational Foundation Models (RFMs) aim to learn generalizable representations across different relational databases, whose critical information is distributed across tabular entity attributes, primary–foreign key connections, and multi-hop relational contexts. While existing methods have achieved strong performance on relational databases, our empirical experiments reveal three fundamental limitations: (1) Flattening relational databases obscures relational information, making primary–foreign key connections and relational context less explicit. (2) Graph-based methods process relations indiscriminately, introducing task-irrelevant relational context which contributes little to target tasks. (3) Existing pretraining objectives under-utilize relational information, limiting the models' ability to capture transferable relational information. To address these challenges, we propose RaMA (Retrieval and Memory Assistant), an RFM that utilizes relational knowledge from records stored in a dynamically evolving external relational memory. (1) RaMA preserves relational information through a relation-based model backbone and overall framework. (2) RaMA retrieves task-relevant relational information through a dynamically evolving external relational memory, where task-aware queries retrieve records for adaptive fusion with local context and reward-guided feedback constantly refines memory effectiveness. (3) RaMA explicitly learns relational information by jointly optimizing masked attribute reconstruction and masked foreign-key recovery. Extensive experiments on RelBench show that RaMA consistently outperforms baselines across classification, regression, and relation prediction, with up to 2.45% AUROC improvement and significantly lower MAE. It further improves zero-shot transfer by up to 3.24% on unseen databases, while retaining efficient adaptation with moderate time and GPU-memory costs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Relational Database, Graph Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_rf7YOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

RaMA, a foundation model for relational databases. RaMA builds on RelGNN's atomic-route formulation and uses a GNN to encode entities and their relational paths. It also introduces an external memory bank that stores entity attributes and relational records spanning one or two hops. The model selects Top-k records using learnable query projections, similarity scoring, and metadata MLPs. It then aggregates the retrieved records through attention and fuses them with local graph representations using a gating network.

Training consists of three stages. First, the model constructs relational graphs from source databases and initializes the external memory. Second, it performs pre-training through masked attribute reconstruction and foreign-key recovery, while jointly training the GNN, retrieval scoring, and fusion modules. The memory bank is updated using heuristic utility scoring. The model then attributes the gain to retrieved records based on retrieval attention and the weight assigned to the retrieval branch by the gating network. Finally, the pre-trained parameters and memory bank are used for downstream tasks, with supervised fine-tuning or few-shot inference.

The experiments cover multiple downstream tasks. The authors report that RaMA outperforms all listed baselines on the six tasks in Table 1 and achieves better results in leave-one-database-out zero-shot transfer. The ablation study evaluates the method by replacing the backbone and comparing static and dynamic memory, providing evidence of its effectiveness.

Critical strengths

S1: The problem is meaningful. Extending retrieval-augmented learning from text and graphs to route-level retrieval in relational databases is a reasonable research direction.

S2: Temporal leakage is explicitly addressed. The memory uses only temporally valid records from the training splits of source databases. During pre-training, retrieved records that reveal masked attributes or foreign keys are excluded.

S3: The experimental evaluation is comprehensive. It covers three task types (classification, regression, and relation prediction), leave-one-database-out zero-shot transfer, few-shot learning and in-context learning, seven ablation variants, sensitivity analysis, and efficiency analysis. Standard deviations over three random seeds are reported for the main results.

Critical weaknesses

W1: The source of the memory gains is unclear. The reported gains appear to come entirely from records dynamically admitted during training. The paper does not specify the provenance of these records or whether they include data from the target database. The update rule is heuristic, and credit assignment is not performed at the individual-record level.

W2: The method is incompletely specified. The paper does not clearly explain how retrieval is integrated into the GNN. Several components are described only through function symbols. “Local retrieval” is mentioned only in the hyperparameter table, and the prediction procedures for zero-shot and in-context learning are not defined.

W3: Key baselines are missing, making it difficult to attribute the gains. The paper does not compare RaMA with RelGNN or with simple memory-update strategies. The zero-shot evaluation compares RaMA with only two baselines. The claim of consistent superiority in few-shot settings holds only for the fine-tuned version.

W4: TabICL v1.1 does not support regression, yet regression results are reported for it.

Questions for authors

Q1. What is the source of the performance gains from the memory module? Static Memory performs similarly to the model without memory. For the supervised experiments in Table 1, please report which databases the records added to memory originate from, whether records from the target database's training split can enter the memory, and how the model performs when all memory records originating from the target database are excluded.

Q2. How are zero-shot and in-context learning (ICL) predictions implemented? For each task type in Tables 2 and 3, please provide the complete inference procedure, including how a prediction head for a new task is obtained without access to target-task labels; how classification probabilities, regression predictions, and relation-ranking results are derived from the model outputs; and how ICL uses labels from the support set to make predictions without updating model parameters.

Q3. How is the retrieval module integrated with the GNN? Does retrieval occur at every message-passing layer? When synchronizing the retrieval indices, are memory-record embeddings recomputed using the current model parameters? What exactly does “local retrieval” in Table 6 refer to, and how does model performance change when this mechanism is removed?

Q4. What additional value does the proposed memory mechanism provide over simpler alternatives? Please compare RaMA with RelGNN under matched model width, pre-training data, and downstream adaptation budgets. Also, compare the proposed memory module with capacity-matched alternatives that use random admission or frequency-based admission. In addition, please explain why the heterogeneous GNN variant with memory performs worse than the memory-free RDL-GNN baseline.

6 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer rf7Y by Authors

W1/Q1·Short answer. Memory contains only temporally eligible training records; excluding target-database records retains most performance.

Thank you for raising the issue. For Table 1, memory records come from rel-f1, rel-trial, rel-amazon, and rel-event. Only temporally eligible target-database training records may enter memory. Memory construction, updates, and retrieval exclude test labels and future outcomes.

Excluding target-database records. We exclude target-database records from memory initialization and updates, keeping other settings unchanged.

TaskMetricTarget eligibleTarget records excluded
Driver-DNFAUROC (%) ↑78.9377.34
Studies-has-dmcAUROC (%) ↑63.0161.75
Review-ratingMAE ↓0.51130.5227
AttendanceMAE ↓0.22050.2298

Target-database records provide useful evidence, but RaMA retains much of its performance without them in external memory.

W2/Q3·Short answer. Retrieval follows the final GNN layer and reuses stored embeddings. We clarified the role of Local Retrieval.

We appreciate these specific questions. Retrieval is performed after all message-passing layers, and the previously stored embeddings are reused, even after parameters have been updated.

Local Retrieval. For each query, RaMA constructs a temporary pool of relational records from the target database under temporal constraints and selects task-relevant evidence. Source retrieval selects evidence from the cross-database memory. Both representations are then integrated through relation-aware fusion, making local retrieval an integral core part of RaMA's pipeline.

W2/Q2·Short answer. Zero-shot uses source-trained heads. ICL uses labeled support examples with fixed parameters and memory.

We appreciate your comments on the inference protocol.

Zero-shot inference. RaMA uses prediction-time query information and a head trained on source tasks of the same type without target-task labels. We compute AUROC from the positive-class logit, MAE on the original target scale, and MAP from the top KK candidate target nodes ranked by predicted score.

In-context learning. A frozen encoder produces the query representation zqz_q and support representations ziz_i. Query-to-support attention aggregates values that include the support label embedding e(yi)e(y_i). The classification head maps the concatenated query representation and support summary to the positive-class probability pqp_q.

αi=softmax⁡i((WQzq)⊤(WKzi)d)\alpha_i = \operatorname{softmax}_i\left(\frac{(W_Q z_q)^\top(W_K z_i)}{\sqrt{d}}\right)

cq=∑iαiWV[zi;e(yi)]c_q = \sum_i \alpha_i W_V[z_i;e(y_i)]

pq=σ(Dcls([zq;cq]))p_q = \sigma(D_{\mathrm{cls}}([z_q;c_q]))

W3/Q4·Short answer. RaMA outperforms the added RelGNN baseline and simpler memory policies. We clarify the heterogeneous GNN ablation setting and ICL results.

Thank you for pointing out our omission of the RelGNN results. We excluded RelGNN because it does not establish a unified cross-database pre-training and transfer framework. The table below reports the supervised RelGNN results under same settings.

TaskMetricRelGNN
Driver-DNFAUROC (%) ↑69.80
Studies-has-dmcAUROC (%) ↑76.48
Review-ratingMAE ↓0.5313
AttendanceMAE ↓0.2496
Site-Spon-RunMAP (%) ↑13.26
Paper-Paper-CoMAP (%) ↑12.03

Memory admission. RaMA achieves the highest MAP compared with simpler update policies.

PolicyAdmissionEvictionPaper-Paper-Co MAP (%)
RaMARelevance+feedback+supportLowest utility13.08
Relevance-onlyHighest relevanceLowest relevance7.81
Random replacementRandomRandom1.42
Random admissionRandomLowest utility1.91
Frequency-based admissionMost frequentLowest utility5.21

Heterogeneous GNN. The heterogeneous GNN ablation uses GraphSAGE without memory. GraphSAGE does not explicitly construct atomic routes, which serve as the basic units in our memory mechanism, so the memory cannot be added.

Few-shot performance. Although RaMA-ICL underperforms some fine-tuned baselines, RaMA achieves the best performance within both the ICL and fine-tuning groups under comparable evaluation settings.

W4·Short answer. We fitted a linear median regression head using frozen TabICL v1.1 embeddings.

Thank you for pointing out that TabICL v1.1 is a classification model. To support regression, we use its pretrained encoder to extract row embeddings before classification. For each task, we fit a separate linear median-regression head using only the corresponding training embeddings and regression labels. Thus, the adaptation preserves TabICL's learned representations without modifying its pretrained parameters.

Overall Response

We sincerely thank the reviewer and the AC again. We appreciate the reviewer's recognition. The main concerns regarding memory provenance, retrieval implementation, experimental protocols, and memory update effectiveness have been addressed through implementation clarifications and additional experiments.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Thank you again for your time and thoughtful feedback on our work. We sincerely appreciate your comments and have provided detailed responses in our rebuttal.

We would be very grateful to know whether our responses have adequately addressed your concerns and helped clarify the contributions of our work. We genuinely hope that our explanations will help strengthen your confidence in the paper, and that you might take these clarifications into account when reconsidering your overall assessment.

If any concerns remain or if you have further questions, we would be more than happy to provide additional explanations or evidence. We highly value your feedback and would welcome any further discussion.

Thank you very much for your time and consideration.

Thanks for your response, which has helped clarify the issues and deepen my understanding of the paper. I have a few further questions that I would appreciate your help in clarifying.

Inconsistent Numbers and Unspecified Source Corpus (W1/Q1):

  • Could you clarify the significant numerical discrepancies between Table 1 of the paper and your rebuttal (RaMA's AUROC against Studies-has-dmc)?

Unclear Memory-Embedding Refresh and Index Synchronization (W2/Q3):

  • If the query embeddings are updated as the model parameters change, while the memory embeddings remain encoded using outdated parameters, how can the authors ensure that both embeddings remain in a consistent representation space?
  • Does SynchronizeIndices recompute the embeddings of the memory records?
Reviewer_bR8fOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces a novel variant of relational foundation models (RFMs), named as RaMA (Retrieval and Memory Assistant), which exploits relational knowledge stored in a dynamically evolving external relational memory to overcome some limitations of prior approaches.

To overcome the limitations of conventional approaches, RaMA incorporates three primary design principles:

  • it captures cross-table dependencies by modeling relational databases as graphs and employing an atomic-route-oriented backbone, overcoming the limitations of flattening relational databases of prior methods;

  • it retrieves only task-relevant relational information instead of generic relational context of existing graph-based relational models;

  • it learns relational information explicitly by including the relational structure in their learning objective, thus overcoming the predominance of semantic-oriented learning (e.g., attribute-level reconstruction) of prior methods.

The authors evaluate RaMA on RelBench across six tasks encompassing classification, regression, and relation prediction. Their method outperforms conventional methods, such as Graph Neural Network (GNNs), and Tabular Foundation Model (TFMs), and existing RFM baselines in supervised, zero-shot, and few-shot settings while maintaining favorable computational efficiency.

Critical strengths
  • Extensive evaluation: The authors evaluate RaMA against a comprehensive set of state-of-the-art baselines (e.g., TabPFN, RelGT, Griffin) across diverse tasks on RelBench. The model demonstrates strong leave-one-database-out zero-shot transfer and in-context learning capabilities with controlled GPU memory overhead during adaptation.

  • Grounded motivation: The individual components of the proposed architecture (Section 4) are explicitly justified by preliminary empirical findings in Section 3, establishing a clear link between observed limitations and methodological solutions. In addition, the paper provides theoretical bounds on retrieval contamination (Theorem 4.1) and an information-theoretic lower bound on relational preservation (Proposition 4.2).

  • Ablation study: The authors effectively isolate the contributions of the stages of their proposed method (i.e., the memory store, backbone, and joint loss) thorough ablation studies in Section 5.3.

Critical weaknesses
  • Redundant framing of research questions: The manuscript presents two disjointed sets of research questions: three empirical questions in Section 3 (e.g., How to model relational databases?) and five evaluation questions in Section 5 (e.g., Does RaMA outperform strong baselines...?). This dual setup creates unnecessary narrative friction. I recommend reframing the structure of the paper away from sequential "historical storytelling" and instead organizing the research questions directly around how each proposed stage overcomes specific limitations of prior methods.

  • Figure Readability and referencing: Several figures (e.g., Figures 1 and 3) are densely packed with small fonts and complex diagrams, making them difficult to parse. Simplifying the visual elements to highlight key data flows would improve readability. Furthermore, ensure that all figure panels introduced in Section 3 are explicitly cited and discussed in the main text body

Questions for authors
  1. When adapting to an unseen database with significantly different schemas or naming conventions, how robust is the structural and semantic query projection in filtering records?

  2. How does the batch-level learning feedback scale computationally when pre-training on databases with significantly larger tables and higher-degree node connectivity?

  3. Could the narrative in Section 3 be consolidated into Section 4 to streamline the transition between preliminary observations and architectural choices?

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer bR8f by Authors

W1 · Short answer. We improved the wording in Section 3 and Section 5 to clarify their purposes.

Thank you for the suggestion. The questions in Section 3 are intended to clarify the key challenges addressed by our method, whereas those in Section 5 aim to comprehensively evaluate the method's effectiveness. Although the two sections address distinct questions, their similar organization may have obscured their different purposes. We have improved the transitional sentences in both sections to clarify their distinct purposes.

W2 · Short answer. We simplified Figures 1 and 3 and cited each panel of Figure 2.

We appreciate your suggestions on figure readability and references. We simplified visual elements in Figures 1 and 3 to improve their readability and added explicit references to each panel of Figure 2 in the text.

Q1 · Short answer. RaMA transfers across schemas and naming conventions.

We appreciate your focus on retrieval robustness across schemas and naming conventions.

Table 2 reports the cross-schema baseline comparison. The semantic projection can use attribute values, and the structural projection captures the connectivity and direction of foreign-key routes. We compare the original names with two one-to-one renamings. One changes the naming style (e.g., review_history to reviewHistory). The other replaces the original names with non-descriptive identifiers (e.g., review_history to t000001).

Held-out taskMetricOriginal scoreStyle scoreStyle top-kk kept (%)ID scoreID top-kk kept (%)
Rel-F1 Driver-Top3AUROC (%) ↑87.4782.0798.7383.4995.42
Rel-Stack User-EngagementAUROC (%) ↑76.2269.9676.1263.9469.02
Rel-Event User-AttendanceMAE ↓0.30400.278176.440.275477.68

Both renaming schemes lower MAE on the attendance task. Top-kk retention measures the proportion of originally retrieved source records still in the top kk after renaming, averaged over queries. Retention is high under the two renaming settings, indicating that retrieval remains relatively stable under renaming.

Q2 · Short answer. With fixed neighborhoods, feedback time varies modestly across table sizes and the tested degree groups.

We appreciate your question about the cost of feedback on larger tables and higher-degree nodes.

At scheduled feedback steps, RaMA adds one forward pass with source-memory retrieval and gradient computation disabled. Feedback time includes this pass, loss attribution, and candidate bookkeeping. We compare Books 5-core with full Books using the same product queries and sampled neighborhoods, then compare three historical degree groups within Books 5-core. The sampled edge count shows how much of the neighborhood the pass processes.

ComparisonSettingReview rowsMedian historical degreeMedian sampled edges per queryExtra pass median (ms/batch)Attribution and bookkeeping median (ms/batch)Feedback median (ms/batch)Feedback p95 (ms/batch)
Table sizeBooks 5-core20,862,04032360.168.2426.81,085.2
Table sizeFull Books, same sampled neighborhoods37,452,33832361.362.9417.11,093.2
Product degreeTypical20,862,040918104.729.6133.3731.3
Product degreeHigh20,862,04014432113.028.6140.5727.2
Product degreeExtreme20,862,0402,008.532110.929.5139.9752.2

Q3 · Short answer. We linked Section 4’s design choices to Section 3’s empirical findings.

Thank you for suggesting a closer connection between the empirical analysis and the methodology.

Section 3 presents the empirical findings together to help readers quickly grasp the key insights motivating RaMA's design, while Section 4 details the model architecture and its design choices. We have added explicit connections in Section 4 to link design choices to their corresponding empirical findings in Section 3.

Overall Response

We sincerely thank the reviewer and the AC again. We appreciate the reviewer's recognition of RaMA's well-grounded motivation, comprehensive experimental evaluation, and effective component ablations. The main concerns regarding retrieval robustness, feedback scalability, empirical motivation, and presentation clarity have been addressed through additional experiments, feedback-time evaluations, and improved figures and organization.

I'm satisfied with the rebuttal performed by the authors and I'm keeping my positive score.

Reviewer_tApoOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces RaMA, a relational foundation model that uses external memory to reuse information across databases. Both the backbone and memory represent relationships through atomic routes. For each prediction, the model retrieves records based on their semantic and structural relevance and combines them with local context through a learned gate. Training feedback guides which records are retained in memory, while pretraining combines masked attribute reconstruction with foreign-key recovery. The authors evaluate RaMA on classification, regression, and relation prediction tasks from RelBench, including zero-shot and few-shot transfer to unseen databases. They report gains over several strong baselines and use ablations to examine the contributions of memory, its update mechanism, and the pretraining objectives.

Critical strengths
  • Using atomic routes for both the backbone and memory gives the model a natural way to combine retrieved records with local relational context.

  • The reported gains span classification, regression, and relation prediction. The ablations also show that removing memory or keeping it static reduces performance.

  • The evaluation goes beyond supervised prediction to examine transfer to unseen databases, with explicit temporal constraints and exclusion of held-out databases from source memory.

Critical weaknesses
  • Appendix E.2 describes the zero-shot and ICL evaluation protocols, but does not fully specify how predictions are produced for unseen tasks or how labeled support examples are used to predict query labels.

  • The memory ablations are useful, but do not show whether the proposed update policy outperforms simpler alternatives, such as relevance-only maintenance or random replacement.

  • Although RelGNN appears in the motivation experiments, it is absent from the main comparison in Table 1 despite its close connection to the atomic-route backbone.

  • Some Driver-DNF accuracies in Figure 6(a) differ from Table 1 without explanation. For example, RelGT is plotted around 76 AUROC, whereas Table 1 reports 71.91, making the accuracy–time comparison difficult to interpret.

Questions for authors
  1. Appendix E.2 describes the evaluation protocol, but how does the model produce predictions for unseen tasks without target-label training? For ICL, what operation combines the query representation with the labeled support examples?
  2. Beyond the static-memory and loss-based-only ablations, does the memory update policy outperform relevance-only maintenance or random replacement under the same memory budget and update schedule?
  3. RelGNN appears in the motivation experiments but not in Table 1. Could you explain this omission and provide a comparison under matched settings?
  4. RelGT is plotted at approximately 76 AUROC on Driver-DNF in Figure 6(a), while Table 1 reports 71.91. Do these results use different configurations?
2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer tApo by Authors

W1 / Q1 · Short answer. Zero-shot uses source-trained heads. ICL attends to labeled support examples.

We appreciate the opportunity to clarify the zero-shot and in-context prediction protocols.

Zero-shot inference. RaMA encodes the query with information available at prediction time and selects the head trained on labeled source tasks of the same type. It uses no target-task labels. We compute AUROC from the positive-class logit, MAE on the original target scale, and MAP from the top KK candidate target nodes ranked by predicted score.

αi=softmax⁡i((WQzq)⊤(WKzi)d)\alpha_i = \operatorname{softmax}_i\left(\frac{(W_Q z_q)^\top (W_K z_i)}{\sqrt{d}}\right)

cq=∑iαiWV[zi;e(yi)]c_q = \sum_i \alpha_i W_V [z_i; e(y_i)]

pq=σ(Dcls([zq;cq]))p_q = \sigma\left(D_{\mathrm{cls}}([z_q; c_q])\right)

W2 / Q2 · Short answer. RaMA achieves the highest MAP among the compared memory update methods.

We appreciate the suggestion to compare the update rule with simpler policies. The results below show that RaMA achieves the best performance under the same initial memory, memory capacity, and update interval.

PolicyAdmissionEvictionPaper-Paper-Cocitation MAP (%) ↑
RaMARelevance, learning feedback, and supportLowest utility13.08
Relevance-onlyHighest relevanceLowest relevance7.81
Random replacementUniform among eligibleUniform among eligible1.42
Random admissionUniform among eligibleLowest utility1.91
Frequency-based admissionHighest observation countLowest utility5.21

W3 / Q3 · Short answer. We add supervised RelGNN results under the same evaluation protocol.

Thank you for pointing out our omission of the RelGNN results. RelGNN was not included in the original comparison because it primarily focuses on task-specific supervised relational learning rather than cross-database pre-training and transfer. The table below reports the supervised RelGNN results using RaMA's downstream splits, prediction cutoffs, and metrics.

TaskMetricRelGNN
Driver-DNFAUROC (%) ↑69.80
Studies-has-dmcAUROC (%) ↑76.48
Review-ratingMAE ↓0.5313
AttendanceMAE ↓0.2496
Site-Spon-RunMAP (%) ↑13.26
Paper-Paper-CocitationMAP (%) ↑12.03

W4 / Q4 · Short answer. Correction of the RelGT plotting error on Driver-DNF.

Thank you for pointing out the discrepancy in the Driver-DNF result, which was caused by a transcription error during the preparation of Fig. 6. Specifically, we mistakenly plotted RelGT's result on Studies-has-dmc instead of Driver-DNF. We have corrected the RelGT result on Driver-DNF to 71.91 AUROC / 0.86 h.

Overall Response

We sincerely thank the reviewer and the AC again. We appreciate the reviewer's recognition of RaMA's methodological design, performance improvements across diverse tasks, and cross-database transfer abilities. The main concerns regarding zero-shot and ICL protocols, memory update effectiveness, baseline completeness, and result consistency have been addressed through detailed clarifications, additional experiments, and correction of the RelGT plotting error.

I thank the authors for their response, which addresses some of my questions. I will update my score accordingly.

Comments on the paper

1 comment
Area_Chair_fkT1OpenReview ↗

Please Engage with the Authors

Dear Reviewers,

We're halfway through the discussion stage. The authors have provided their responses to your reviews. Please check whether their responses have resolved the concerns raised in your review. If not, please raise and leave some time for the authors to respond again. If yes, please respond to the authors.

Many thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.