acceptodds
Under review as a conference paper at ICLR 2027

Improving Relational Forecasting with Target History

Abstract

Relational learning, namely machine learning for relational databases, has attracted increased interest in recent years. However, despite the inherently temporal nature of forecasting tasks in relational learning, modeling of the temporal component has received limited attention. In particular, existing models often do not employ an entity’s past target values in a principled way, even though these values may provide a strong predictive signal. In this work, we investigate multiple model-agnostic strategies for incorporating an entity’s target history into predictions. We show that a simple strategy consistently improves the performance of multiple representative models on RelArena- forecasting tasks, with the best resulting model achieving new state-of-the-art results among open-source model submissions. Finally, we conduct a case study showing how the current typically used evaluation protocol hinders performance on certain datasets by forbidding the use of information between the test threshold and the sample timestamp. We hope our findings will draw the community’s attention to the importance of the temporal component in relational forecasting tasks and contribute to the development of better benchmarking protocols.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
relational learning, forecasting, target history, RelArena, RelBench

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7TVzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper studies how an entity’s past target values can improve predictions on relational databases. The authors use task-specific SQL queries to reconstruct historical targets from existing records and compare lagged values, summary statistics, and combinations of the two. They then add the same set of 16 lag features to different models. Across 21 tasks in RelArena-α, this improves the overall performance of three model families. The paper also presents a case study on how freezing the database limits the information available at prediction time.

Critical strengths
  1. The method is simple and easy to add to different models. The same history features improve the overall performance of TabPFN-Rel, GraphSAGE, and LightGBM, suggesting that the approach is useful across different model families.

  2. The paper draws a useful distinction between the data split cutoff and the actual prediction time. It points out that freezing the database can exclude information already available when a prediction is made, which helps connect benchmark design with applications where data keeps arriving.

Critical weaknesses
  1. The core approach reconstructs historical targets and adds lagged values or summary statistics to existing models. While the results show that this works, the paper needs to clarify what is new compared with existing approaches that reuse past labels or construct lag features.

  2. The method can access the task’s SQL definition to reconstruct additional historical targets, while some baselines only use the provided task tables. The gains therefore reflect several factors at once: access to task-specific information, broader historical coverage, and the way that history is represented.

  3. The paper mainly reports overall gains from a shared history representation, without closely examining how missing history, entity activity, or changes in behavior affect performance. This leaves its usefulness for new entities, sparse histories, and distribution shifts unclear.

  4. In the case study that removes the time cutoff, only the target history is updated, while the other relational inputs remain frozen. This shows the value of more recent history, but does not yet provide a full comparison where all methods can use updated data.

Questions for authors

Please refer to the weaknesses.

2 comments

Dear Reviewer 7TVz,

Thank you for your thorough review. We address your concerns below.

W1. We discuss this in Section 2.3. In short, we explicitly construct target lags via SQL queries over the RDB, while other approaches either use labels from the provided task tables or rely on an LLM to generate features. In theory, an LLM can construct target lags too, but this is not guaranteed.

W2. We note that DFS with depth 4 and a 2-hop GNN (assuming no subgraph sampling) already access all the database records required to compute target lags on all tasks except rel-trial/site-success, though they aggregate them in a way that does not necessarily retain information about target lags. Thus, for TabPFN-Rel and GraphSAGE, our approach mostly provides a better representation of records the models already access, while additionally using the target-defining SQL query as task information that these baselines do not use. For LightGBM, our approach also provides information from additional database records.

W3. In this work, we focus on established benchmarks and tasks and report the overall gains in the corresponding setups. If you have any specific suggestions for an experiment, either on the existing benchmark or on another dataset, that would address your concern, please let us know and we will do our best to conduct it.

W4. The goal of the case study is to show that models can benefit from removing clipping. For simplicity of implementation, we indeed only updated the target history computation. Removing clipping from other inputs will likely boost performance even further, but we believe that our current implementation already achieves the goal of the case study. Thus, we acknowledge that the case study does not provide a full comparison in which all methods take full advantage of removing clipping, but this was never the goal of the case study.

Please also note that we have revised the paper, with changes highlighted in blue (for revised figures and tables, the captions are highlighted).

Best regards,

Authors

Thanks for the response. I agree that the clipping case study shows why more recent target history can help. On W2, your reply clarifies which records the baselines can access under the stated assumptions. I’m still unsure how much of the gain comes from the task-specific SQL, the additional history it recovers, or the way that history is represented. I’ll keep my original rating.

Reviewer_bZRYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper adds an entity's past outcomes to relational forecasting models by rerunning the task's SQL query over earlier time windows. It compares lag features and summaries, then selects 16 raw lags using validation results. On 21 RelArena-alpha tasks, these features improve aggregate results for multiple models. A separate rel-f1 case study showed further gains when history can use records available up to the prediction time.

Critical strengths
  1. The same 16 features help three different models in aggregate. That's a useful result for building stronger forecasting baselines.

  2. The experiments were well designed and executed

Critical weaknesses
  1. There's no direct comparison with lags taken from the supplied labels, while TabPFN-Rel already offers that. This makes it hard to tell what comes from reconstructing additional outcomes versus simply giving the model explicit features.
Questions for authors

It's unclear whether Tables 1-3 used the split threshold or db.max_timestamp to anchor history. I'd appreciate more detail on which cutoff was used and whether the results hold with the cutoff in Eq. 1.

2 comments

Dear Reviewer bZRY,

Thank you for your review and valuable comments. We address your concerns below.

W1: Comparison against TabPFN-Rel with its built-in lag features. We have revised our paper to include these experiments. The corresponding method is called tabpfn-rel-tt, and it achieves 1,7301{,}730 Elo compared to 1,7731{,}773 Elo for tabpfn-rel-th. We believe these results support our position regarding the utility of explicitly constructing target history. Thank you for suggesting this ablation.

Q1: Threshold ambiguity. We used db.max_timestamp instead of a validation or test threshold since RelArena-α\alpha did not provide a direct API for accessing these thresholds. We have added a clarifying footnote in the revised paper. Moreover, we note that db.max_timestamp actually matches the validation/test threshold in most cases.

Please also note that we have revised the paper, with changes highlighted in blue (for revised figures and tables, the captions are highlighted).

Best regards,

Authors

Dear Reviewer bZRY,

Since the discussion period is ending soon, please let us know if our response addresses your concerns, and if you have any further comments.

Thank you for your time and effort in reviewing our paper.

Best regards,

Authors

Reviewer_jrntOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that relational forecasting models underuse an entity's own past outcomes. Since RelBench targets are defined by task-specific SQL queries, the authors re-run the same query at earlier timestamps (in steps of the prediction horizon Δ) to rebuild a history of past targets for each entity–timestamp pair, without being limited to the rows in the supplied training table. They compare raw lags (L ∈ {1,…,64}), eight summary statistics over 64 lags, and combinations of the two. They choose one configuration (16 raw lags) by validation Elo with TabPFN-Rel, then add it unchanged to LightGBM, GraphSAGE and TabPFN-Rel on the 21 RelArena-α forecasting tasks, keeping each backbone's search space and tuning budget. Test Elo goes up for all three models (+282 LightGBM, +28 GraphSAGE, +77 TabPFN-Rel), and TabPFN-Rel-th is reported as the top open-source model submission. A case study on rel-f1, the only database with several timestamps per evaluation split, shows that letting the history features use records between the split threshold and the prediction timestamp improves all nine model–task pairs. From this the authors argue that the freeze-at-threshold protocol should be revisited.

Critical strengths

The main thing is that the intervention is simple, cheap and controlled, and it still moves strong baselines. The history is built from the task's own SQL definition. The representation is picked once on validation data with one backbone and then frozen, and the backbones keep their RelArena-α search spaces and budgets, with five seeds for both the "-th" and plain variants (Sec. 4). That makes the comparison against the unmodified models fairly clean. The LightGBM result is the most convincing part of the paper. With only entity-table features plus 16 lags it goes from 1,396 to 1,678 Elo and passes RelGT, RDBLearn and RelGNN (Obs. 5). Some per-task gains are far larger than the seed spread, e.g. rel-hm/item-sales MAE 0.0750 → 0.0352 and rel-amazon/item-ltv 56.52 → 44.18 (Table 2).

Critical weaknesses

W1. The paper never isolates its central claim, that the gain comes from reconstructing history beyond the supplied labels. The Introduction and Sec. 2.3 separate the proposed features from existing mechanisms because those "rely on the entity–timestamp pairs present in the supplied task tables" and do not "evaluate the task definition at additional historical timestamps". But the paper also says that TabPFN-Rel already reuses training labels through DFS by default in RelArena-α, and has built-in lag features (most recent labels and their ages) that are turned off by default (Sec. 2.3). The obvious ablation is missing: lags restricted to labels already in the training table, or TabPFN-Rel with its own lag option enabled. Without it, we can't tell how much of the +77 Elo is due to SQL reconstruction and how much to presenting already-available labels in a better format. Table 1 makes this more pressing. For both L=1, A=0 and L=0, A=64, candidate and baseline Elo are exactly the same (1,870 vs. 1,870). Outside rel-f1, validation has a single timestamp equal to the threshold (Sec. 3), so lag 1 is y(e, τ−Δ). For entities in the training table, that is the label at the last training timestamp, which TabPFN-Rel's default label-table DFS can already aggregate, and the eight statistics overlap heavily with standard DFS aggregation primitives. A Δ of exactly zero looks more like redundancy (or features that had no effect) than the reading in Obs. 2 and Obs. 6 that recency "alone is insufficient" but "complementary" with longer history. The persistence-vs-const-entity comparison in Obs. 6 is also confounded. For the 12 binary tasks, persistence outputs h₁ ∈ {0,1}, which gives a heavily tied ROC-AUC, while const-entity outputs a continuous per-entity mean. So the 1,278 vs. 1,302 gap may reflect score granularity more than how informative recency is.

W2. The aggregate Elo gains, and the selection behind them, are not separated from noise. Elo is computed on 5-seed task means, with no per-task significance (Sec. 4), so any small difference counts as a full win or loss. Many of the counted wins are well inside one standard deviation in Table 2. For GraphSAGE (+28 Elo) these include rel-amazon/item-churn 0.8296 → 0.8301, rel-stack/user-badge 0.8891 → 0.8893 and rel-stack/user-engagement 0.9066 → 0.9074, while it loses clearly on rel-event/user-repeat (0.7659 → 0.7386). One case looks informative. On rel-trial/study-outcome, persistence scores exactly 0.5000, which suggests h₁ is unavailable for basically every test entity and the model falls back to a constant. Yet TabPFN-Rel-th still gets a "win" there (0.7184 ± 0.0067 → 0.7254 ± 0.0033), most likely from seed variance. The selection in Table 1 has the same problem. Baseline Elo moves between 1,847 and 1,870 across rows only because each pool contains a different candidate, so pool composition alone causes about 23 Elo of variation. That is similar to the gaps between the top candidates (L=16: 75, L=2/A=64: 70, L=8: 56). Obs. 1 (non-monotonic effect of history length) and the choice of (16, 0) may therefore not be reproducible. The claims built on small Elo margins are the most exposed: GraphSAGE-th "ranks above unmodified TabPFN-Rel" (1,732 vs. 1,713, Obs. 4) and the "state-of-the-art among open-source submissions" framing in the abstract.

Questions for authors

Could you report an ablation that separates SQL reconstruction from label reuse? Concretely: (a) the same 16-lag representation filled only from labels in the supplied training table, and/or (b) TabPFN-Rel with its built-in lag features enabled (Sec. 2.3), both under the same tuning budget. It would also help to give, per task, the fraction of non-missing lags that come from timestamps or entities missing from the training table. Please also explain why L=1, A=0 and L=0, A=64 give exactly the same validation Elo as the baseline in Table 1. Are the predictions identical, and are the new columns dropped or ignored somewhere in the TabPFN-Rel/DFS pipeline?

4 comments
AuthorsOpenReview ↗

Official Comment (Part 1)

Dear Reviewer jrnt,

Thank you for your thorough and insightful review. We address your concerns and questions below.

W1.1 and Q1: Comparison against TabPFN-Rel with its built-in lag features. We have revised our paper to include these experiments. The corresponding method is called tabpfn-rel-tt, and it achieves 1,7301{,}730 Elo compared to 1,7731{,}773 Elo for tabpfn-rel-th. We believe these results support our position regarding the utility of explicitly constructing target history. Thank you for suggesting this ablation.

W1.2 and Q3: The same validation Elo for the baseline and target-history-augmented methods. First, we provide a small table showing that for each method there is at least one task where its (mean−std,mean+std)(\text{mean} - \text{std}, \text{mean} + \text{std}) interval does not overlap with the baseline interval. Second, after updating the results on the rel-trial/study-* tasks (see our response to W2.2 for details), the gaps between augmented and baseline methods in Table 1 are nonzero for both (L,A)=(1,0)(L, A) = (1, 0) and (L,A)=(0,64)(L, A) = (0, 64). Thus, task-level gains and losses appear to have canceled out in the aggregate Elo.

dataset/taskBaselineL=1,A=0L=1, A=0L=0,A=64L=0, A=64
rel-amazon/user-ltv (MAE)12.0048±0.010712.0048 \pm 0.010711.9396±0.007611.9396 \pm 0.007612.0099±0.008812.0099 \pm 0.0088
rel-event/user-ignore (ROC-AUC)0.8454±0.00310.8454 \pm 0.00310.8404±0.00680.8404 \pm 0.00680.8327±0.00760.8327 \pm 0.0076

W1.3: Persistence baseline. Thank you for highlighting this. To break ties in binary classification tasks, we updated the persistence baseline to use 0.9⋅persistence+0.1⋅fallback0.9 \cdot \text{persistence} + 0.1 \cdot \text{fallback} as predictions, where the fallback is given by the entity or global mean. As expected, this modification improves the performance of this baseline, and the updated version now ranks above the const-entity baseline.

W2.1: Statistical significance of Elo gains. We acknowledge this problem. However, uncertainty in method rankings appears to be a broader issue in the current RelArena-α\alpha evaluation. For example, in the original paper, 4 out of 6 reported open-source model submissions, tabpfn-rel, graphsage, relgt, and rdblearn, had overlapping confidence intervals. To reduce evaluation noise, we ran our methods with 5 seeds rather than 1 seed as in the original RelArena-α\alpha setup.

W2.2: rel-trial/study-outcome and rel-trial/study-adverse tasks. Indeed, for these two tasks, there is no target history available for any entity. We were aware of this at the time of submission, but re-evaluated the models for simplicity. Since this can introduce noise into the comparisons, we decided to replace the results of all TH-augmented methods on these two tasks with the corresponding baseline results in the revised paper.

Q2: Fraction of non-missing lags that are missing in the training table. Following your suggestion, we conducted the following analysis. Among non-missing lag cells at offsets 1…L1 \ldots L, we computed the percentage of lags whose corresponding (entity, timestamp) pair has no label in the task tables. In other words, this quantity measures the percentage of targets we need to explicitly compute and cannot simply take from the task tables. We computed it for different prediction scenarios (train, validation and test) and different values of LL, and report the results below. Let us summarize our key observations.

First, we note that for the train and validation scenarios, most entries are either zero or near-zero. This is because RelBench v1 constructs train/val/test task tables in a very specific way. In short, for a given split, it uses the same timestamp grid for all entities and retains all eligible (entity, timestamp) pairs where a label can be computed. Moreover, the interval between consecutive timestamps equals the prediction horizon Δ\Delta, and for all datasets except rel-f1, the single validation timestamp is given by the last training timestamp plus Δ\Delta.

However, we still argue that explicitly computing target lags is better than reusing labels from task tables. The latter approach implicitly relies on how task tables are constructed. For RelBench v1, most of the required target lags are available in the task tables, but this might not hold for other datasets. For example, one may have non-uniform timestamp grids or use only a subset of eligible (entity, timestamp) pairs. In such cases, the percentage of required lags that cannot be reused from task tables should be larger. Thus, we argue that explicitly computing target lags is a more robust solution that does not depend on how the train/val/test tables are constructed.

AuthorsOpenReview ↗

Official Comment (Part 2)

Second, we note that for the test scenario, most entries are 100%100\%. This is because for most tasks, the gap between the test and validation cutoffs is not a multiple of the prediction horizon Δ\Delta. For example, on the rel-event dataset, the prediction horizon Δ\Delta is equal to 7 days, while the gap between the validation and test cutoffs is equal to 8 days. Thus, while our table reports 100%100\%, the task tables are likely to contain something very similar to the target lags we use. However, explicitly computing lags allows us to use exactly the information we need, and not just something similar. We believe this observation further supports the utility of our proposed method.

L=1L=1L=16L=16L=64L=64L=1L=1L=16L=16L=64L=64L=1L=1L=16L=16L=64L=64
Splittraintraintrainvalvalvaltesttesttest
rel-amazon/user-churn0.000.000.000.000.000.00100.00100.00100.00
rel-amazon/item-churn0.000.000.000.000.000.00100.00100.00100.00
rel-amazon/user-ltv0.000.000.000.000.000.00100.00100.00100.00
rel-amazon/item-ltv0.230.350.670.000.000.24100.00100.00100.00
rel-avito/user-visits20.5831.8231.820.0013.1413.14100.00100.00100.00
rel-avito/user-clicks18.8129.4429.440.0011.6511.65100.00100.00100.00
rel-avito/ad-ctr16.8328.0428.040.0010.1610.16100.00100.00100.00
rel-event/user-repeat0.000.000.000.000.000.00100.00100.00100.00
rel-event/user-ignore0.000.000.000.000.000.00100.00100.00100.00
rel-event/user-attendance0.000.000.000.000.000.00100.00100.00100.00
rel-f1/driver-dnf0.180.080.08100.00100.00100.00100.00100.00100.00
rel-f1/driver-top30.000.000.00100.00100.00100.00100.00100.00100.00
rel-f1/driver-position0.380.350.40100.00100.00100.00100.00100.00100.00
rel-hm/user-churn0.000.000.000.000.000.000.000.000.00
rel-hm/item-sales1.9216.3560.160.000.0018.750.000.0017.19
rel-stack/user-engagement0.000.000.000.000.000.00100.00100.00100.00
rel-stack/user-badge0.000.000.000.000.000.00100.00100.00100.00
rel-stack/post-votes0.000.000.000.000.000.00100.00100.00100.00
rel-trial/study-outcomeN/AN/AN/AN/AN/AN/AN/AN/AN/A
rel-trial/study-adverseN/AN/AN/AN/AN/AN/AN/AN/AN/A
rel-trial/site-success0.000.000.000.000.000.00100.00100.00100.00

Please also note that we have revised the paper, with changes highlighted in blue (for revised figures and tables, the captions are highlighted).

Best regards,

Authors

Dear Reviewer jrnt,

Since the discussion period is ending soon, please let us know if our response addresses your concerns, and if you have any further comments.

Thank you for your time and effort in reviewing our paper.

Best regards,

Authors

Thanks for your rebuttal and additional experiments. I'd like to keep my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.