acceptodds
Under review as a conference paper at ICLR 2027

Pretraining in Relational Databases Across Tasks and Entity Types

Abstract

Relational databases support many predictive tasks over shared histories, yet relational deep learning models are typically trained separately for each target and entity type. We study whether a conventional relational encoder can instead be pretrained once on many query-defined forecasting tasks and reused. Query-Conditioned Relational Pretraining (QCRP) generates future-prediction objectives from structured query specifications and trains them jointly through a shared query-conditioned predictor. Evaluated on RelBench, frozen QCRP encoders improve over existing relational pretraining objectives on standard tasks. Because standard targets largely overlap with pretraining objectives, we introduce 24 strictly held-out tasks, where QCRP also demonstrates strong transfer. A single encoder pretrained across entity types remains competitive with type-specific specialists. We further find that query-defined tasks share low-dimensional structure, which the query encoder captures to support zero-shot prediction for unseen queries, although its advantage over nearest-query substitution is limited. These results suggest that task-diverse pretraining is a practical route to reusable database-level encoders, and that task specification is a promising interface for future work.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Relational deep learning, Self-supervised pretraining, Graph Neural Networks, Multi-task learning, Representation learning

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_AumwOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies pretraining for deep learning on relational databases. Here, graph neural networks (GNNs) learn from linked tables to forecast future outcomes for entities such as customers or products, usually with one model per task. The main contribution is Query-Conditioned Relational Pretraining (QCRP). QCRP generates many forecasting tasks from structured query fields (entity type, relational path, aggregation, filter, horizon) and trains one encoder on all of them through a shared query-conditioned predictor. It can also train a single encoder across entity types. On Rel-Amazon and Rel-HM, frozen QCRP encoders beat earlier relational pretraining methods on 6 standard RelBench tasks and 24 new held-out tasks. A single multi-root checkpoint stays close to type-specific specialists, and the query interface supports limited zero-shot prediction.

Critical strengths

1- Consistent frozen-encoder gains, including on held-out tasks. With GraphSAGE, QCRP beats TVE and TVE+CTR on most of the six standard tasks (Table 1). On the 24 held-out tasks, QCRP-root has the best overall mean rank (2.12), ahead of supervised training (2.79) (Table 3). Both QCRP variants beat both TVE baselines on all 12 Amazon held-out tasks (Appx. D.3). Only a 16.9K-parameter head is trained per task, versus 1.36M–2.25M for supervised training (Sec. 3.3). This makes the method practical for reuse.

2-An honest look at benchmark overlap. Table 2 shows that standard RelBench labels can be recovered almost perfectly from TVE pretraining statistics. For example, logistic regression reaches ROC-AUC 1.000 on churn even after removing the matching coordinates. The authors use this to motivate their held-out suite. This analysis is useful to the community beyond this paper.

Critical weaknesses

1- Part of the gain may come from target overlap, not transfer. All six standard tasks are included in QCRP's own pretraining supervision (Sec. 3.2). The largest gain, on item-LTV, comes from a price-times-count target that QCRP pretrains on directly but TVE does not. The authors note this "does not isolate its effect" (Sec. 3.2). The 24 held-out tasks were picked by an AI agent and share query primitives with pretraining (Appx. B). The authors also say the held-out tasks "may share predictive structure with pretraining supervision" (Sec. 3.3). After full fine-tuning, the advantage over TVE+CTR mostly disappears (Table 12). So it is unclear how much of the frozen gain reflects a better representation rather than wider target coverage.

2- The evidence is narrow, and the second and third contributions are weak. The study uses only two databases with two entity types each, and three seeds, with no significance tests ("bold does not imply statistical significance", Table 1). Multi-root training makes Amazon customer worse on all 40 of its pretraining objectives (Sec. 4.2). Zero-shot prediction barely beats swapping in the nearest pretraining query: Spearman rho 0.42 vs 0.41, AUC 68.4 vs 67.1 (Table 4). It is also far below fitted probes on classification (AUC 81.1). The paper's own conclusion says this "leaves open whether it generalizes beyond relationships already represented by nearby pretraining tasks" (Sec. 6).

Questions for authors

1- (Clarification) Can you compare QCRP with TVE when both use the same target pool, or with QCRP after removing all pretraining queries that exactly match or nearly match the downstream tasks? This would show whether the gains come from the training method or from target coverage. 2- (Clarification) How were the 24 held-out tasks chosen by the AI agent? Were any candidate tasks dropped after the prevalence and variance filters? A held-out set built from different query primitives would make the transfer claim stronger.

3-(Clarification) Do the frozen-encoder and multi-root results hold on more RelBench databases with more than two entity types, such as rel-stack or rel-f1?

1 comment

We thank the reviewer for their detailed and thoughtful assessment. We particularly appreciate their recognition of the consistent frozen-encoder gains across standard and held-out tasks, the parameter efficiency of our approach, and the value of our analysis of target overlap in existing relational pretraining benchmarks. We have provided a general response to shared questions as a standalone comment, which we refer to as G1 through G4.

W1/Q1 — Target overlap versus representation learning.

We agree that broader target coverage contributes to QCRP's improvements, and conducted two additional analyses to quantify its role.

First, our recoverability analysis (G1) confirms substantial overlap: ground-truth pretraining targets recover held-out labels with mean R2R^2 of 0.906 and AUC of 97.03%. However, recoverability from learned predictions is considerably lower (0.328 R2R^2 and 79.86% AUC), demonstrating that target overlap does not automatically translate into predictive performance.

Second, matched-task-pool ablations (G2) show that improvements persist even without expanding TVE's original supervision. Better target processing and decoder design increase mean R2R^2 from 0.2803 to approximately 0.304 and AUC from 79.96% to approximately 80.9%. Expanding the task pool provides further gains to approximately 0.310 R2R^2 and 81.2% AUC.

We acknowledge that QCRP's advantage over TVE+CTR narrows substantially after full fine-tuning. This is understandable because all methods fine-tune the same downstream architecture and objective, differing primarily in their pretrained initialization. End-to-end adaptation can reduce the influence of these initialization differences, leading to similar downstream performance. Our primary goal, however, is learning reusable representations that transfer effectively without updating the encoder.

W2 — Limited experimental scope and multi-root generalization.

We extended evaluation from two to four RelBench databases by adding Rel-Stack and Rel-RateBeer (G3). Rel-Stack uses 232 pretraining tasks across user and post roots; Rel-RateBeer uses 225 tasks across users, beers, and brewers. Across seven additional downstream tasks, both QCRP variants outperform random-initialized frozen encoders on every task and outperform fully supervised training on three tasks each. These are preliminary single-seed results.

These results support competitive multi-root consolidation rather than uniformly positive transfer. We observed root-dependent trade-offs as reported in Section 4.2: multi-root pretraining improves Amazon product on 37/38 objectives but worsens Amazon customer on all 40. Similarly, zero-shot transfer remains limited. We view improving cross-root optimization and zero-shot generalization as important directions for future work.

Q2 — Held-out task selection and candidate filtering.

The 24 held-out tasks use the same task-construction procedure described in Section 2.1, with their exact definitions excluded from pretraining (G4). They intentionally share query primitives with pretraining objectives: the intended test is generalization to unseen compositions within the forecasting language. This does not guarantee statistical independence from pretraining targets; G1 quantifies the resulting overlap through ground-truth and learned-prediction recoverability analyses.

Reviewer_9pLSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes QCRP, a query-conditioned relational pretraining framework that generates future-prediction tasks from relational databases. It investigates representation transfer, multi-root training, and zero-shot generalization to unseen queries.

Critical strengths
  1. The paper explores a promising approach to generating diverse pretraining tasks and shows encouraging improvements, particularly on regression tasks. The extensive analyses of task correlations and target overlap are informative.

  2. The paper provides a careful empirical study of multi-root training, including different training strategies, optimization dynamics, and parameter sharing.

Critical weaknesses
  1. Unclear pretraining protocol. The construction of pretraining tasks and entity–cutoff samples (which entity index, at which time) is insufficiently explained.

  2. Questionable held-out generalization. Section 3.2 demonstrates that downstream targets can be almost perfectly recovered from other pretraining statistics, even after removing directly matching coordinates. However, Section 3.3 only excludes exact task definitions, leaving substantial uncertainty about whether the held-out tasks provide genuinely new predictive information.

  3. Unclear sources of improvement. Multi-root training does not consistently outperform root-specific models. More broadly, it is unclear whether the gains mainly arise from richer, highly correlated supervision and broader entity–cutoff coverage rather than query conditioning or cross-root knowledge transfer.

  4. Limited experimental scope. Despite the extensive task-level evaluation, all experiments are conducted on only two relational databases, each with two root entity types. This limits the evidence for the general applicability and scalability of the proposed pretraining framework to more diverse schemas and root types.

Questions for authors
  1. How are pretraining entities, index and timestamps, and task definitions selected? How are temporal boundaries enforced relative to downstream splits?

  2. Can the authors apply the target-recovery analysis from Section 3.2 to the 24 held-out tasks? How are the held-out tasks determined?

  3. Can the authors provide a controlled comparison using the same pretraining targets and entity–cutoff coverage, but without query conditioning (maybe similar to simply extend training samples), to better isolate the contributions of the proposed method?

3 comments

We thank the reviewer for their thoughtful assessment and constructive feedback. We particularly appreciate their recognition of the promising task-generation approach, the encouraging improvements on regression tasks, and the informative analysis of task correlations and target overlap. We also appreciate their acknowledgment of our careful empirical investigation of multi-root training, including optimization dynamics and parameter sharing. We have provided a general response to shared questions as a standalone comment, which we refer to as G1 through G4.

W1/Q1 — Pretraining protocol and temporal boundaries.

Following TVE, we use a database-wide anchor grid rather than constructing a separate cohort for each downstream task. For official validation cutoff TvalT_{\mathrm{val}} and prediction horizon Δ\Delta, training anchors are Tval−kΔT_{\mathrm{val}}-k\Delta for positive integers k within the available database history. We use Δ=91\Delta=91 days for Rel-Amazon and Δ=7\Delta=7 days for Rel-HM.

Each example consists of a root-table entity ID and an anchor timestamp; the same entity may appear at multiple anchors. No pretraining target window extends into the downstream validation prediction window. The same entity–cutoff selection and temporal boundaries apply to QCRP and all pretraining baselines, including TVE, MaskAE, and CTR (G4).

W2/Q2 — Held-out task overlap and generalization.

We agree that held-out tasks may share predictive structure with pretraining objectives. Our intended setting is generalization to unseen task definitions within the same relational forecasting language, rather than to entirely unrelated tasks.

To quantify overlap, we conducted additional recoverability experiments (G1). Ground-truth QCRP pretraining targets recover held-out labels with mean R2R^2 of 0.906 and AUC of 97.03%. However, using learned predictions instead of future ground-truth statistics reduces recoverability to 0.328 R2R^2 and 79.86% AUC. Thus, overlap in target definitions does not automatically imply accurate forecasting from historical information. We will clarify this distinction in the paper.

The 24 held-out tasks are generated using the same task-construction procedure as pretraining, with exact task definitions excluded. Shared primitives are retained intentionally to evaluate generalization within the query language (G4). Candidate-filter counts remain missing and require verification; we do not infer them from the final selected pool.

W3/Q3 — Attribution of performance gains.

We conducted new matched-task-pool ablations (G2). Even under TVE's original supervision, improved target processing and decoder design increased mean R2R^2 from 0.2803 to approximately 0.304 and AUC from 79.96% to approximately 80.9%. Expanding the task pool provides additional improvements, reaching approximately 0.310 R2R^2 and 81.2% AUC.

The shared query-conditioned decoder achieves comparable frozen-representation quality to separate task-specific MLP heads, while providing an additional zero-shot prediction interface. On the QCRP pool, its zero-shot predictions achieve Spearman 0.415 / AUC 69.37%, compared with 0.297 / 64.41% when averaging private heads.

W4 — Limited experimental scope.

We extended evaluation from two to four RelBench databases, adding Rel-Stack and Rel-RateBeer (G3). RateBeer includes three prediction roots—users, beers, and brewers—providing a stronger test of multi-root pretraining. Across seven additional downstream tasks, both root-specific and multi-root QCRP outperform random-initialized frozen encoders on every task, and each variant outperforms fully supervised training on three tasks. These results support broader applicability and competitive performance from one multi-root checkpoint. G3 reports the experiment settings and the full result tables.

Thank you for the detailed rebuttal, especially the clarifications on the pretraining protocol, the additional ablations, and the experiments on more datasets. These have addressed several of my original concerns.

I still have two remaining concerns:

  1. Target overlap and pretraining supervision. I appreciate the recoverability analysis, which clarifies that strong target overlap does not imply trivial prediction or direct label leakage. However, the agent-assisted task generation and selection may introduce strong, difficult-to-interpret priors. Given the substantial target overlap, I remain uncertain how much of the improvement reflects generalizable representations rather than closely aligned pretraining supervision.

  2. Multi-root pretraining. While combining multiple root-specific models into one checkpoint is useful, multi-root training rarely outperforms the specialists. I remain uncertain about its broader benefits, especially given the limited evidence of positive transfer across roots.

Overall, I appreciate the additional experiments and clarifications. However, these remaining concerns still limit my assessment of the contributions. I am inclined to maintain my current rating, but remain open to other reviewers' perspectives and further discussion.

Thank you for considering the additional experiments and for clarifying your two remaining concerns.

  1. Task selection and target overlap. The agent proposed queries in a predefined forecasting language, based on the task-generation grammar discussed in Section 2.1. We selected 24 for diversity and balance across roots and task types. We acknowledge that the bias in the agent selection process is unmeasured, but this selection process is done independently of model performance. The 24 tasks therefore test transfer to unseen definitions within a curated sample. Our evaluation on official RelBench tasks separately tests transfer to objectives defined independently of this process.

    TVE gives a reason to expect some overlap even for independently chosen tasks. Its premise is that different labels depend on the same future records, and that a few aggregates can preserve much of the information needed to compute them. Both TVE and QCRP include pretraining objectives that predict simple SQL aggregates over those records. These aggregates as practical approximations to sufficient statistics. Truong et al. give a concrete example:

    • The sum of samples is sufficient for exponential, Poisson, and Bernoulli distributions.
    • The max is sufficient for a uniform distribution over a bounded interval.
    • The mean is sufficient for a normal distribution with known variance.

    Even with TVE’s pretraining targets fixed, changing target processing and decoder design improves frozen-probe mean R2R^2 from 0.280 to 0.305. Further expanding the pool to QCRP by including filtering tasks and conditional tasks raises it further to 0.310 (G2). Thus, closer alignment from the expanded pool explains part of the gain, while the held-out evaluation remains subject to the selection caveat above.

  2. Value of multi-root pretraining. When a database supports forecasts about several entity types, each root-specific specialist model must learn its own encoder, including its own transformations for the history those entities share. We measured the trade-off under matched per-root exposure. Across 221 pretraining objectives on Rel-Amazon and Rel-HM, sharing reduces mean validation loss by 3.51% for Amazon products and increases it by only 0.20–0.78% on each of the other three roots. On the six official downstream tasks, the shared GraphSAGE encoder matches or exceeds its specialist on three. We agree that positive transfer depends on the root.

    The broader finding is that independently optimizing an encoder for every root often buys little in these experiments, even though each specialist can learn an entirely different representation. This finding gives a practical reason to pursue a database-level foundation model for multi-entity forecasting. Typically, each additional prediction root requires another specialist encoder to deploy, maintain and update. The multi-root model brings operational benefits that matter even when predictive accuracy stays the same. For organizations considering a single encoder for forecasts across a complex database, the operational benefit of sharing an encoder therefore grows with the number of roots it serves. Our comparison with specialists quantifies the predictive trade-off involved in that deployment decision.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General response (G1): Held-out Task Overlap and Transfer

We thank all reviewers for their constructive feedback, thoughtful questions, and valuable suggestions. We appreciate the opportunity to clarify our contributions and provide additional experimental evidence. To address the main concerns, we conducted additional experiments investigating (G1) held-out target overlap, (G2) attribution of performance gains, (G3) generalization to additional databases, and (G4) clarifications of the pretraining and task-selection protocols.

We extended the target-recoverability analysis in Section 3.2 to the 24 held-out tasks on the validation sets. We compare QCRP and TVE’s pretraining statistics. We distinguish two settings: recovering held-out labels from ground truth future statistics (oracle recovery) and recovering them from the model’s prediction based on historical inputs.

Recovery fromQCRP R2R^2 ↑\uparrowTVE R2R^2 ↑\uparrowQCRP AUC (%) ↑\uparrowTVE AUC (%) ↑\uparrow
Oracle future targets0.9060.78397.0396.67
Model-predicted targets, refitted recovery0.3280.25079.8678.90

1. Oracle recoverability. Oracle recovery confirms substantial shared predictive structure among relational forecasting objectives. QCRP’s expanded pool provides higher recoverability, particularly for contextualized regression targets. High recoverability does not mean task leakage: a holdout task: count⁡(rating)>5\operatorname{count}(\mathrm{rating}) > 5 can be perfectly recovered from the ground-truth target of a pretraining task count⁡(rating)\operatorname{count}(\mathrm{rating}), even though the task itself is never seen during training. However, recoverability from future ground-truth statistics is fundamentally different from predicting those statistics using historical information.

2. Recoverability from model predictions. We repeated the analysis using predictions from pretrained models instead of ground-truth targets: Our results demonstrate transfer to unseen task definitions within a shared relational forecasting language, not to statistically unrelated tasks. The controlled experiments in G2 further examine how task coverage relates to frozen-representation quality.

These findings also reflect our intended generalization setting: transfer to unseen task specifications within a compositional language of relational forecasting queries, rather than to entirely unseen query primitives. This setting is motivated by a recurring pattern in existing relational forecasting benchmarks: many predictive targets can be expressed as aggregations over relational records, optionally combined with filters, contextual attributes, and thresholded readouts. QCRP explicitly exploits this inductive bias by expanding the coverage of meaningful forecasting objectives beyond TVE's fixed statistics. While the 24 held-out task definitions are excluded from pretraining, they may share primitives and predictive structure with pretraining objectives by design. We will clarify oracle recoverability versus forecasting from historical inputs and include the oracle-versus-model-prediction recovery results in the appendix.

AuthorsOpenReview ↗

General response (G2): Attribution of Performance Gains

We conducted additional ablations separating target preprocessing, decoder architecture, task-pool expansion, and query conditioning. All results aggregate performance across 24 held-out tasks using three seeds and root-specific pretraining.

1. Matched-task-pool ablations.

ConfigurationPoolMean R2R^2 ↑\uparrowAUC (%) ↑\uparrow
TVE (MinMax + cosine)TVE0.280379.96
TVE + MSETVE0.277179.96
TVE + log/std + MSETVE0.302880.57
K-linear, no queryTVE0.298380.66
K-MLP, no queryTVE0.304380.87
K-MLP + queryTVE0.303580.93
Shared MLP + queryTVE0.304780.89
Shared MLP, no queryTVE0.264378.95
K-MLP + queryQCRP0.310481.25
Shared MLP + query (paper)QCRP0.310381.20

These results identify three sources of improvement:

  • Target processing and training formulation: Under TVE's task pool, log/standardized targets with MSE improve mean R2R^2 from 0.2803 to 0.3028 and AUC from 79.96% to 80.57%. The scalar multi-task training formulation with MLP predictors provides further improvements, reaching approximately 0.304 R2R^2 and 80.9% AUC. These comparisons evaluate training configurations rather than isolating every individual optimization choice.
  • Task-pool expansion: Holding the K-MLP + query architecture fixed, expanding from TVE's task pool to QCRP improves mean R2R^2 from 0.3035 to 0.3104 and AUC from 80.93% to 81.25%. This confirms that broader task coverage contributes additional gains.
  • Shared query-conditioned prediction: The shared decoder achieves frozen-representation quality comparable to private task-specific MLP heads. Query conditioning allows the shared decoder to distinguish prediction objectives, avoiding the substantial degradation observed when the query is removed (R2R^2 0.3047 vs. 0.2643 under the TVE task pool).

2. Role of query conditioning and shared decoder. The shared decoder provides a unified interface for predicting unseen task specifications without fitting additional heads. We compare it against two ways of reusing pretrained private MLP heads for zero-shot prediction.

Prediction interfaceSpearman ρ\rho ↑\uparrowAUC (%) ↑\uparrow
Shared MLP + query encoder0.41569.37
K-MLP, mean predictions0.29764.41
K-MLP, nearest head0.23860.41

The shared query-conditioned decoder substantially outperforms both private-head reuse strategies. Sharing decoder parameters encourages the model to distinguish tasks through their structured query representations rather than relying on separate task-specific prediction heads.

Together, these experiments show the frozen-encoder improvements arise from target processing, decoder design, and broader task coverage, while query conditioning and shared decoder enables a zero-shot transfer. The revised paper will summarize the contributions of target processing, decoder architecture, and expanded task coverage in Section 3.2. The matched-task-pool ablations will appear in a new Appendix D.6, and the shared-versus-private-head zero-shot comparison in Appendix D.5.

AuthorsOpenReview ↗

General response (G3): Broader evaluation

To address the limited experimental scope, we extend our evaluation from two to four RelBench databases, including Rel-Stack and Rel-RateBeer. In particular, Rel-RateBeer is the only RelBench database with official entity-level forecasting tasks spanning three distinct root tables (users, beers, and brewers), providing a stronger test of multi-root pretraining.

We retain the relational encoder architecture and frozen-probe evaluation protocol described in Sections 2–3 and Appendix A, with dataset-specific task pools, prediction horizons, and training adjustments.

Rel-Stack. We construct 232 pretraining tasks (96 user-rooted and 136 post-rooted), including 142 regression and 90 binary classification objectives, using relational paths of up to two hops and 91-day prediction windows.

Methoduser-badge AUCuser-engagement AUCpost-votes MAE
QCRP root (frozen probe)87.7090.200.06422
QCRP multi (frozen probe)87.4490.000.06435
Random encoder (frozen probe)84.1586.210.06790
Supervised (end-to-end)88.4389.890.06627

Rel-RateBeer. We construct 225 pretraining tasks across three roots (88 user, 46 beer, and 91 brewer), including 216 regression and 9 binary classification objectives, using one-hop relational paths. Prediction horizons are 90 days for users and beers and 365 days for brewers.

The brewer root has substantially fewer entity–timestamp training pairs. To avoid oversampling this root, rather than introducing extra hyperparameters to balance the loss, we just allocate samples in a batch respecting the original count distribution across roots in joint multi-root training.

Methoduser-churn AUCuser-count MAEbeer-churn AUCbrewer-dormant AUC
QCRP root (frozen probe)92.887.15878.3079.66
QCRP multi (frozen probe)94.047.20777.3477.64
Random encoder (frozen probe)85.9011.05474.0774.01
Supervised (end-to-end)93.846.60680.5678.83

Across the seven additional downstream tasks, both QCRP variants outperform random-initialized frozen encoders on all seven tasks, and each outperforms end-to-end supervised training on three. Notably, the RateBeer multi-root model achieves 94.04% user-churn AUC, exceeding both its root-specific counterpart (92.88%) and supervised training (93.84%).

The results extend our findings to more diverse relational schemas, including a database with three prediction roots. Importantly, one multi-root checkpoint retains competitive downstream performance across all three RateBeer roots. Upon acceptance, we will extend the Rel-Stack and Rel-RateBeer evaluations to three seeds and report the full results and dataset-specific training settings in a new Appendix D.7.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.