- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
This paper studies how an entity’s past target values can improve predictions on relational databases. The authors use task-specific SQL queries to reconstruct historical targets from existing records and compare lagged values, summary statistics, and combinations of the two. They then add the same set of 16 lag features to different models. Across 21 tasks in RelArena-α, this improves the overall performance of three model families. The paper also presents a case study on how freezing the database limits the information available at prediction time.
Critical strengths
-
The method is simple and easy to add to different models. The same history features improve the overall performance of TabPFN-Rel, GraphSAGE, and LightGBM, suggesting that the approach is useful across different model families.
-
The paper draws a useful distinction between the data split cutoff and the actual prediction time. It points out that freezing the database can exclude information already available when a prediction is made, which helps connect benchmark design with applications where data keeps arriving.
Critical weaknesses
-
The core approach reconstructs historical targets and adds lagged values or summary statistics to existing models. While the results show that this works, the paper needs to clarify what is new compared with existing approaches that reuse past labels or construct lag features.
-
The method can access the task’s SQL definition to reconstruct additional historical targets, while some baselines only use the provided task tables. The gains therefore reflect several factors at once: access to task-specific information, broader historical coverage, and the way that history is represented.
-
The paper mainly reports overall gains from a shared history representation, without closely examining how missing history, entity activity, or changes in behavior affect performance. This leaves its usefulness for new entities, sparse histories, and distribution shifts unclear.
-
In the case study that removes the time cutoff, only the target history is updated, while the other relational inputs remain frozen. This shows the value of more recent history, but does not yet provide a full comparison where all methods can use updated data.
Questions for authors
Please refer to the weaknesses.
2 comments
Thanks for the response. I agree that the clipping case study shows why more recent target history can help. On W2, your reply clarifies which records the baselines can access under the stated assumptions. I’m still unsure how much of the gain comes from the task-specific SQL, the additional history it recovers, or the way that history is represented. I’ll keep my original rating.
Dear Reviewer 7TVz,
Thank you for your thorough review. We address your concerns below.
W1. We discuss this in Section 2.3. In short, we explicitly construct target lags via SQL queries over the RDB, while other approaches either use labels from the provided task tables or rely on an LLM to generate features. In theory, an LLM can construct target lags too, but this is not guaranteed.
W2. We note that DFS with depth 4 and a 2-hop GNN (assuming no subgraph sampling) already access all the database records required to compute target lags on all tasks except
rel-trial/site-success, though they aggregate them in a way that does not necessarily retain information about target lags. Thus, for TabPFN-Rel and GraphSAGE, our approach mostly provides a better representation of records the models already access, while additionally using the target-defining SQL query as task information that these baselines do not use. For LightGBM, our approach also provides information from additional database records.W3. In this work, we focus on established benchmarks and tasks and report the overall gains in the corresponding setups. If you have any specific suggestions for an experiment, either on the existing benchmark or on another dataset, that would address your concern, please let us know and we will do our best to conduct it.
W4. The goal of the case study is to show that models can benefit from removing clipping. For simplicity of implementation, we indeed only updated the target history computation. Removing clipping from other inputs will likely boost performance even further, but we believe that our current implementation already achieves the goal of the case study. Thus, we acknowledge that the case study does not provide a full comparison in which all methods take full advantage of removing clipping, but this was never the goal of the case study.
Please also note that we have revised the paper, with changes highlighted in blue (for revised figures and tables, the captions are highlighted).
Best regards,
Authors