acceptodds
Under review as a conference paper at ICLR 2027

FoodTrade: A Benchmark for Graph-Based Learning on Temporal Food Trade Networks

Abstract

This work introduces FoodTrade, a dynamic temporal network dataset covering global trade in four staple grains from 1992 to 2023, comprehensively curated from multiple public sources, preprocessed, and ready-to-use for graph-based machine learning. The dataset presents a combination of challenges often encountered in machine learning, such as a noisy, low-sample, high-dimensional, and heavy-tailed structure with temporal distribution shifts and the presence of hidden confounders. With these characteristics, FoodTrade offers a new and difficult benchmark for graph-based learning at a time when the community is increasingly concerned about stagnated progress due to limitations of current benchmarks. We analyze the evolution of the link structure and the presence of network shocks, formulate predictive tasks at the node and edge levels, build an evaluation framework, and benchmark several baselines across different model classes. While temporal GNNs may seem like a natural choice for modeling the food trade network, they are not currently the best-performing model. Heuristics, econometric, and tabular ML methods often perform as well as or better than GNNs. Therefore, this dataset contributes to the call for more meaningful benchmarks that facilitate effective evaluation of graph-learning methods beyond simple solutions. We outline potential directions for better modeling of the food trade network and improving predictions of critical variables. Therefore, the dataset paves the way for developing specialized predictive tools that invite collaboration with domain experts to advance impactful data-driven AI solutions to assess global food security.

Primary area:
learning on graphs and other geometries & topologies
Keywords:
temporal graphs, gnns, dataset, benchmark, food trade network, food security

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_kneHOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The data introduces the FoodTrade dataset, a dynamic network covering global trade for four staple grains, as an additional benchmark for temporal graph-based machine learning algorithms. The dynamic network includes additional node and edge level attributes which provide a richer network than existing trade-based networks.

An initial benchmarking for two regression problems show that otherwise SotA machine learning algorithms for dynamic networks struggle on the FoodTrade dataset, especially compared to a moving average. This is also observed for the trade network on the Temporal Graph Benchmark website so this result is not so surprising. The dynamic network is split into train/validation/test using rolling windows, which is consistent but may not work best for each of the tested models.

Critical strengths

New trade dataset highlighting the issues, even in smaller datasets, for SotA graph-based machine learning with additional node and edge level covariate information posing a newer challenge. It is important that research introducing novel datasets for benchmarking are introduced widely to focus research effort. I do not want my comments elsewhere to downplay the importance of work like this.

Critical weaknesses

As a benchmarking dataset, this does not offer much extra beyond existing dynamic trade datasets. For instance, Structure reducibility of mulitlayer networks, De Domenico et al. (2015) introduces a multilayer trade network over ten years. The Temporal Graph Benchmark website has a small trade example for a node classification task but with a only single edge weight, so a multilayer dynamic network is a natural comparison.

The most interesting addition are the node and edge covariate information, but an EDA for this information is lacking in the dataset description.

Questions for authors
  • What were the motivation for including a rolling window for the train/validation/test setup? The regression can be done using the temporal sequence of network snapshots, so it felt like this approach restricted how machine learning algorithms could be applied to the data.
  • Can you provide more details about the other possible tasks for this dataset beyond classification and prediction? The shock response simulation and nutrition classification felt like a stretch for what is possible using this dataset. If these are possible, then this would be a big selling point for the FoodTrade data.
3 comments

We thank the reviewer for their insightful comments. We address their concerns below and would be happy to answer any further questions.

  • Inter-layer similarity analysis (W1)

The paper by De Domenico et al. (2015) and the FoodTrade benchmark have significantly different aims. The former quantifies inter-layer similarity using (unweighted) Jensen-Shannon (JS) distances and studies how a high number of layers (240 for the trade network) can be reduced to fewer but more distinctive ones. Our paper provides a benchmark for temporal graph learning with four layers of staple crops, which have potentially different trade dynamics. As two crop layers can have very similar binary topology yet differ significantly in trade-volume distributions or response to shocks, we evaluate FoodTrade inter-layer similarity by the Jaccard and weighted JS distances, which, between all pairs of crop layers across all years, are 0.75 and 0.85, respectively, marking high difference in the trade networks. Fig. 17 shows the JS distance at the crop-pair and year level. Additionally, De Domenico et al. (2015) use a trade network only for 2010, not over ten years. It is also not explicitly stated whether the edge weights are trade volumes or values, and whether the network is directed or undirected.

  • Covariate EDA (W2)

We divide node and edge covariates into categories and add them to the feature metadata. Fig. 18 provides an overview of the relationship across the high-dimensional feature space through within- and cross-category correlations. We also ablate the contribution of different categories to model performance in Table 19. Since the utility of features can vary across different tasks supported by the benchmark, we retain all the features at the dataset level, so purposeful feature selection and further feature engineering can be carried out as appropriately needed. Recent works have addressed feature selection particularly in the context of graph learning [1,2]. FoodTrade provides a new setting in which such methods can account for temporal roles and whether features are used locally or through message passing.

[1] Adaptive Node Feature Selection for Graph Neural Networks (ICML 2026)

[2] Let Your Features Tell The Differences: Understanding Graph Convolution By Feature Splitting (ICLR 2025)

  • Motivation for rolling windows (Q1)

Training/validation/test target years are chronologically separated to ensure evaluation on future unseen years, while a fixed context length provides a controlled comparison across model classes by giving each model access to the same amount of historical information. The evaluation framework then constructs this information as required by the model, e.g., a table, temporal sequence, or sequence of graphs (network snapshots). Rolling windows are a standard forecasting construction in time series forecasting [3,4]. including economic and trade domains. It is particularly useful here for converting a short temporal sequence into multiple supervised forecasting instances, while maintaining a common interface across vastly different model classes and enabling their direct comparison.

[3] Z-GCNETs: Time Zigzags at Graph Convolutional Networks for Time Series Forecasting (ICML 2021)

[4] Nowcasting trade in value added indicators. (OECD Statistics Working Papers 2023)

  • Higher-order tasks (Q2)

We agree that shock-response simulation and nutritional adequacy assessment are more complex than the node- and edge-level tasks evaluated here. These higher-order tasks are naturally compositional: they depend on accurate estimation of the quantities benchmarked in this work, so errors in basic forecasting would propagate into downstream simulations or classifications. For example, a trade cascade can arise when changes in domestic production or supply affect trade flows, subsequently altering supply in other countries and propagating through the network. Reliable simulation of such dynamics first requires accurate prediction of node- and edge-level quantities across varying network conditions. Otherwise, errors compounding over multiple steps can become difficult to diagnose. The present contribution of FoodTrade is to establish this benchmark foundation: a curated, attributed temporal graph dataset, standardized prediction tasks, and an extensive baseline evaluation of core quantities such as country-level supply and bilateral trade volumes. Establishing reliable performance on these building blocks is a necessary step toward more complex cascade and food-security tasks, which the dataset and evaluation framework are designed to support.

Thank you for your response.

Firstly, my W1 comment was supposed to say "over ten years ago" not "over ten years", so apologises for any confusion there. While the initial De Domenico et al. (2015) paper had a different aim, this dataset has been studied later as part of dynamic multilayer network analysis. It is not identical but the similarities are certainly enough to be cited.

I have reservations regarding forcing rolling windows for training/validation/test. I understand that this is commonly used, but by insisting on its inclusion in the benchmarking, this naturally restricts methods that can be tested. If the windowing is useful, then that can always be incorporated by a new technique, rather than being compulsory.

I am still not convinced that this dataset provides enough novelty and, while I hope I am wrong, I don't see it supplanting the TGB Trade benchmark. However, I may have underplayed it significance in bringing together a few existing dynamic multilayer networks into one place, so have improved my rating.

Regarding W1, we agree and have now cited De Domenico et al. (2015) as related work on multi-layer trade networks. We further clarify the rolling-window setup and address novelty concerns relative to TGB-Trade as follows.

  • Clarification on rolling windows

We would like to clarify that rolling windows for train/validation/test are not enforced by the dataset itself. FoodTrade remains accessible as a sequence of network snapshots, and users can define train/validation/test splits over arbitrary years and apply models directly to the full sequence. The rolling-window construction is instead part of the default benchmark protocol. It converts the relatively short annual sequence into multiple supervised forecasting instances and provides a common historical context across substantially different model classes. This is particularly useful in the low-sample regime of FoodTrade, while allowing each model to receive the same information in its required form, e.g., as a table, temporal sequence, or sequence of graphs. Thus, the benchmark provides a standardized comparison, without restricting future methods from using varying historical contexts.

  • Novelty relative to TGB-Trade

The goal of FoodTade is not to supplant the TGB Trade benchmark, but rather to complement it, as the two address different learning problems. TGB-Trade aggregates all agricultural products into a single monetary value network and evaluates next-year prediction of how a country's trade is distributed across partners as a ranking task. FoodTrade instead preserves crop-specific networks and evaluates quantitative prediction of country-level caloric supply and bilateral trade volumes, together with substantially richer node and edge covariates. The crop-trade domain additionally gives the benchmark meaningful downstream applications such as studying trade dependence, rerouting, and shock-response cascades.

Beyond the food trade domain, as a general discrete-time temporal graph-learning benchmark, evaluation across several model classes exposes where current temporal GNNs fail to exploit the available temporal, structural, and covariate information effectively. For example, on the Twitter Tennis benchmark [1], using two experimental settings [1,2], we find that a simple moving-average baseline outperforms their best reported temporal GNNs (MSE: 0.286 vs. 2.006 and 0.331 vs. 0.440, respectively). Such results highlight the need for challenging benchmarks that reveal these limitations and support the development of improved temporal GNN architectures and training strategies.

[1] PyTorch Geometric Temporal: Spatiotemporal Signal Processing with Neural Machine Learning Models (CIKM 2021)

[2] Permutation Equivariant Neural Controlled Differential Equations for Dynamic Graph Representation Learning (NeurIPS 2025)

Reviewer_yKkeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces FoodTrade, a new benchmark dataset for machine learning on temporal food trade networks represented as graphs. The dataset covers global trade flows for four staple food crops from 1992 to 2023 and models the problem as an attributed temporal graph with 204 countries, 172,000 edges, 32 temporal snapshots, 110 node features, and 40 edge features. The authors describe an extensive data pre-processing pipeline, an exploratory analysis, benchmark tasks (node-level supply forecasting and edge-level trade volume forecasting), an evaluation framework, and comparisons across heuristic approaches, econometric models, tabular models, sequential models, and graph neural networks (GNNs). An important empirical finding is that simple heuristic baselines often outperform current temporal GNNs, highlighting both the difficulty of the benchmark and the limitations of existing graph learning methods.

Critical strengths
  1. The work integrates numerous public data sources, involves significant data harmonization and cleaning, addresses reporting inconsistencies, and provides a ready-to-use benchmark, complete with code and evaluation infrastructure. In my opinion this represents a good engineering contribution.

  2. Compared to previous benchmarks on trade networks, this dataset includes a significantly broader set of node and edge attributes, enabling the study of representations that simultaneously leverage network topology; temporal evolution; socioeconomic variables of individual countries; characteristics of bilateral trade.

Critical weaknesses
  1. The primary contribution is the dataset and the benchmark itself. The paper does not introduce a novel learning methodology, architecture, training strategy, or theoretical contribution. The main scientific contribution is therefore the dataset, the benchmark design, and the empirical evaluation, rather than a new modeling approach.

  2. Limited analysis of GNN failure.

  3. Limited discussion on statistical significance. Although means and standard deviations are reported, the paper does not present statistical significance tests or confidence intervals for comparing the different methods. Given the limited temporal dimension of the sample, a more rigorous statistical validation would be desirable.

Questions for authors
  1. Can you provide an ablation study that quantifies how much of the performance depends on temporal persistence versus the graph structure?

  2. Have you evaluated longer forecasting horizons (e.g., 2 to 5 years), where persistence typically becomes less competitive?

  3. How robust are the conclusions when using error metrics other than sMAPE?

  4. Can you quantify the contribution of different feature families (commercial, macroeconomic, demographic, and gravity-based variables) through systematic ablation studies?

  5. Did you observe significant performance differences between countries that are highly central within the network and those that are peripheral?

Additionally:

  1. Include more in-depth diagnostic studies to understand why GNNs struggle.
  2. Report the computational costs for all baselines.
4 comments

We thank the reviewer for their insightful comments. We address their concerns below and would be happy to answer any further questions.

  • Contributions besides modelling approach (W1)

We respectfully disagree that not introducing a new modeling approach limits the contributions of this work. In fact, the reviewer themselves acknowledge the significant engineering contribution in the benchmark construction and empirical discovery. Upon the reviewer’s suggestion, we have also conducted further post-hoc error analysis (Fig. 6 & 7), established the statistical significance of results (Table 13), and ablated feature category contributions to performance (Table 19). We identify and explore a key challenge of the dataset (i.e., heterogeneity, among others), and outline directions for new modeling approaches such as Mixture-of-Experts-inspired solutions to preserve heterogeneity. Overall, this makes the FoodTrade benchmark a significant scientific contribution, enabling progress that existing benchmarks do not adequately support.

  • GNN failure analysis (W2)

We hypothesize that heterogeneity of the data, particularly the target values, is an important contributing factor. The core-periphery structure, where trade between a few large exporters and many smaller importers varies substantially in both magnitude and temporal persistence, is not well-aligned with conventional message passing in GNNs. Aggregating information across such heterogeneous edges may smooth out or obscure distinctive features that are important for predicting trade flows. Our edge-volume-percentile (Fig. 6) and node-centrality-based (Fig. 7) analyses are consistent with this hypothesis. Of course, these diagnostics do not necessarily establish information loss due to aggregation. It remains an open question how mechanisms such as attention and gating affect what information individual architectures retain during aggregation. These diagnostics do, however, motivate examining how the inductive biases of models affect performance. Such analysis is currently beyond the scope of this paper, as we focus on providing the community with a strong, ready-to-use benchmark with comprehensive baseline evaluation. This benchmark enables further diagnostic work that can inform the design of architecture and training strategies that can tackle such complex tasks more effectively.

  • Statistical significance of results (W3)

We estimate test-sample uncertainty using 95% confidence intervals (CI) for sMAPE obtained from 10000 repetitions of sampling with replacement (bootstrapping) in Table 13. Among all model pairs, 75.6%, 70.5%, and 74.3% of the pairs show a statistically significant difference for the Supply, Supply/cap, and TradeVolume prediction tasks. After ranking models by performance, comparing only adjacent models yields 16.6%, 16.6%, and 33.3% significant pairs, respectively. This shows that the most significant differences are between models that are farther apart in the ranking, which, together with Table 2, establish that temporal GNNs are significantly worse than heuristic baselines. We outline more details in Section B.2.

  • Structural vs temporal ablation (Q1)

Our evaluation of graph-agnostic baselines, including sequence and tabular models, already quantifies performance in the absence of graph structure. However, since they change the model class, we complement the results with another ablation that retains the graph structure but removes temporal information. For this, we compare the performance of MPGNNs using a window size of one (thus including only current-year features for next year’s prediction) with and without target values in Table 15. In all cases, removing the target lag worsens performance. Compared to the results in Table 2, which use a window size of 3 years as context, a window size of one with the target value worsens performance for the edge task but improves for the node supply/cap task, indicating that more historical information may contribute differently for different tasks. Furthermore, the information levels used to construct tabular inputs isolate the effect of target lags, node and edge features. The combination of all three yields the best performance for GRU on the edge task, whereas for the node tasks, Random Forest achieves the best performance using only the ‘target’ level, which excludes node features (Table 16).

  • Longer forecasting horizons (Q2)

While the performance of persistent-forecast and moving-average decreases for direct-step forecasting of longer horizons, they remain the best baselines, as shown in Table 20.

AuthorsOpenReview ↗

Rebuttal (continued)

  • Metric robustness (Q3)

We select sMAPE as the main metric because it is scale-independent and each test target contributes equally to the aggregated error. Given the large variation in target values of the tasks, sMAPE is suitable for general evaluation. A complementary metric is wMAPE, which weighs percentage error proportional to the absolute target value during aggregation. We replicate Table 2 using wMAPE as the metric instead of sMAPE in Table 17. As expected, the wMAPE error values are generally lower than the corresponding sMAPE values because the highly concentrated trade links (e.g., top 10% of the links by volume that constitute 85% of global wheat trade in 2023, see Fig. 16), which are more stable and relatively easier to predict, dominate the overall error, while the remaining trade links are not well-represented. On the contrary, sMAPE reflects performance across both large and small trade links. To establish robustness of model performance to the evaluation metric, we report the Kendall’s Tau correlation coefficient (Table 18) over all models, showing similar ranking. Persistent forecast, moving average, and GRU retain the top three positions across tasks. Furthermore, taking the performance gap into account makes the model rankings almost identical across the two metrics.

  • Feature category ablation (Q4)

Table 9 ablates three feature inclusion settings for the Gravity model. We further divide node and edge features into six and five categories, respectively, as part of feature metadata. We ablate their contribution to the performance of the best model after naive baselines, i.e., GRU, in Table 19, which shows the performance change in percentage points (increase in sMAPE) compared to the results in Table 2, where all features were used. For the edge trade volume task, removing features representative of political relationships and trade regulations increased error the most, whereas for the node supply task, removing demographic features increased error the most.

  • Node centrality based error analysis (Q5)

Yes, with countries ranked by centrality percentile, the average test error for GRU varies significantly between the countries with highest and lowest degree centrality. Averaging is done across all observations paired by country and test year assigned to each percentile bin of width one. The error distribution over node centrality across all models and both node tasks is shown in Fig. 7. The average error for specific percentiles representing countries that are most and least central in the network is as follows:

Centrality percentileSupply/cap (Avg. sMAPE)Supply (Avg. sMAPE)
<10%12.7817.76
<20%12.5516.19
>80%86.7489.58
>90%133.14127.91
  • Computational cost

We now report the computational cost in Table 21.

Thank you for the detailed rebuttal. The additional analyses address most of my concerns, particularly regarding statistical validation, ablation studies, and diagnostic analyses. They strengthen the paper and make the results more convincing. However, my overall assessment remains largely the same, as my main concern was never about the experimental evaluation itself. Rather, the core contribution is still primarily the dataset and benchmark rather than a novel methodology. For this reason, I maintain my original rating.

Thank you for your response. We appreciate that you are satisfied with our additional analysis. We agree that our core contribution is a dataset and benchmark, but the absence of a novel method does not limit the scientific contribution of this work. Please note that ICLR explicitly lists this as a relevant topic in the call for papers and the contributions should be evaluated in that context.

Reviewer_CTStOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

FoodTrade integrates FAOSTAT and CEPII data to construct annual, crop-specific trade networks spanning 1992–2023. The dataset release includes node and edge attributes, preprocessing metadata, and code for forecasting both country-level crop supply and bilateral trade volumes. Notably, the volume forecasting task assumes that the trade network topology for the target year is known a priori. Across a comprehensive set of baselines, simple persistence and moving average models prove remarkably difficult to outperform on most tasks. Furthermore, the study disaggregates prediction errors to distinguish between continuing and newly emerging trade links.

Critical strengths
  1. The dataset curation is substantial. Covering four staple crops across 204 countries over 32 years, and providing documented node and edge features, it offers a valuable testbed for methods dealing with short time series and noisy economic data.
  2. The paper establishes genuinely strong simple baselines. Table 3 clearly illustrates how a model can achieve high overall performance while failing on newly emerging links, a crucial distinction that should be preserved in future applications of this benchmark.
Critical weaknesses
  1. Generalizability beyond this domain remains unproven. While Table 1 lists 204 nodes and 172,409 edges, the 32 annual snapshots and four crop graphs share the same countries, data sources, and temporal resolution. They do not constitute four independent domains. A small graph can serve as a deliberately challenging benchmark, but these experiments offer limited insight into scalability or whether model rankings transfer to other discrete-time dynamic graphs. More importantly, the evaluation focuses solely on supply and trade-volume regression. Link prediction is deferred, and no edge classification or graph-level structural tasks are tested in Section 4. Consequently, the work makes a stronger case as a food-trade forecasting resource than as a broadly applicable graph-learning benchmark.
  2. The evaluation leaves key comparisons unresolved. For volume prediction, Section 4 provides the target year's edge set, framing this as a conditional magnitude estimation task rather than true next-year network forecasting. Appendix A.1 mentions linear interpolation for time-varying covariates but does not clarify whether future observations relative to the forecast origin are used. Furthermore, Table 8 reports a maize trade-volume sMAPE of 55.35 ± 8.91 for TGRU, compared to 110.49 for moving average and 113.86 for persistence. If accurate, this contradicts the claim that naive baselines are generally superior. If it is a reporting error, the benchmark conclusions should be revisited based on corrected results.
Questions for authors
  1. What is the intended scope of the benchmark claim? Could the authors include a topology-oriented task, such as next-year link prediction, and compare dataset/model rankings against at least one established discrete-time dynamic graph benchmark? If this falls outside the current scope, please frame the contribution more narrowly as a crop-trade regression resource.
  2. At each forecast origin, are interpolation, imputation, and feature construction strictly limited to information available at that time? Why is the target year's edge set assumed known, and how would performance change if link existence also had to be predicted?
  3. Is the maize TGRU result in Table 8 correct? Please reconcile this with the paper’s broader conclusions and identify the best method for each crop and task using a consistent selection criterion.
4 comments

We thank the reviewer for their insightful comments. We address their concerns below and would be happy to answer any further questions.

  • Scalability, transferability & scope (W1+ Q1)

Firstly, we agree that the four crop networks do not constitute independent but related domains, which makes this data more interesting for a multiplex analysis in future. Cross-layer dependencies are especially relevant under shocks, where shortages in one staple crop may alter production, demand, or trade in others, creating non-trivial multi-relational prediction problems. We quantify inter-crop-layer similarity of the trade networks in Fig. 17.

Secondly, Foodtrade establishes difficulty as a small-scale benchmark and is not meant to stress test million to billion edge scalability. Therefore, it is complementary to existing large scale temporal benchmarks such as TGB.

Thirdly, to address the reviewer’s question about cross-domain transfer of model ranking, we evaluate the heuristic baselines on Twitter Tennis [1], a discrete-time dynamic graph for temporal node regression, against the temporal GNNs benchmarked on the dataset in [1] and [2]. To compare against the results in each paper, we adopt the settings of their experiments from their provided code. As the following table shows, moving average also outperforms the models evaluated in both papers.

SettingsBest reported modelPersistent-forecastMoving-average
[1]2.0060.3000.286
[2]0.4400.5180.331

In addition, most papers, to the best of our knowledge, do not benchmark across multiple model classes such as tabular, sequence and GNNs, which FoodTrade does with appropriately constructed inputs. This establishes that simpler baselines should not be easily disregarded.

We distinguish the intended scope of the benchmark from the scope of the current paper. This paper introduces the dataset, evaluation framework, and a first comprehensive baseline study, which constitute a significant analytical and engineering contribution. Moreover, the challenges posed by FoodTrade as discussed in Section 3.2 (such as limited temporal samples, distribution shift, heavy tails, noisy attributes, and data heterogeneity) can also arise beyond food-trade forecasting. Therefore, the benchmark is a resource both for future food-security applications and a testbed for methods designed to handle these interacting challenges in graph learning.

[1] PyTorch Geometric Temporal: Spatiotemporal Signal Processing with Neural Machine Learning Models (CIKM 2021)

[2] Permutation Equivariant Neural Controlled Differential Equations for Dynamic Graph Representation Learning (NeurIPS 2025)

  • Link prediction (W1 + W2 + Q2)

We now include results for link prediction and the two-step hurdle model, i.e., link prediction followed by edge regression on the predicted links, in Tables 4 and 5, respectively. When we assume an oracle for link existence, we specifically isolate the difficulty of the magnitude component. We expect the performance to worsen if both components are predicted, as errors from the link prediction model are propagated into the edge regression task.

  • Missing value imputation (W2 + Q2)

The imputation step is part of the dataset curation and should be distinguished from information leakage in the supervised benchmark. It reconstructs missing covariates that could otherwise be observed or available according to their temporal role, as defined in the metadata. For example, population is explicitly treated as future-known because short-horizon population estimates are typically highly predictable and commonly available from demographic projections, so using its target-year value is consistent with the task definition. Historical-dynamic features, such as GDP, are only provided from context years, with temporal roles identified in the feature metadata. The feature metadata can be easily updated to conservatively label any features as solely historical. Target values are never imputed. Thus, imputation affects the completeness of permitted covariates, while the rolling-window construction determines which features and years are available to each prediction. For dynamic-historical features, which constitute the majority of the covariates, feature construction is strictly restricted to context-year information available before the prediction year, preventing target-year values from entering the corresponding prediction.

AuthorsOpenReview ↗

Rebuttal (continued)

  • Maize-GRU result (W1 + Q2)

The anomaly result TRGU for maize trade volume prediction is reported correctly, but it results from unstable model training. The model selection criteria are consistent across all experiments, and as both validation and test sets show the same highly oscillating trend during training, the model with a low error is selected based on validation across hyperparameter tuning. We also inspect other models’ training curves and generally find more stable training. Evaluating TGRU experiments with different hyperparameter settings shows a more stable training curve with test sMAPE of 139.54 +- 2.56.

Thank you for the response. My main remaining concern is that the evidence for cross-domain generalizability is still weak — the comparison on Twitter Tennis alone is not sufficient to demonstrate that model rankings are transferable to other discrete-time dynamic graphs. Meanwhile, the "known target-year edge set" setup means the task remains essentially conditional magnitude estimation, which still falls short of full next-year network forecasting. I hope the authors will more explicitly narrow the scope of the benchmark in the next version. Hence, I keep my original rating of 2.

We thank the reviewer for their feedback. We have provided the requested link prediction setup and additional tasks where naive baselines outperform temporal GNNs and believe that those answer the reviewer’s original questions. While we agree that the transferability of model rankings would be valuable work, it does not contribute to the central objective of our paper. Our primary contribution is to introduce a new, challenging temporal graph dataset, together with meaningful prediction tasks and an evaluation framework that can foster methodological advances in graph learning. We compare different methods on FoodTrade to highlight the associated challenges.

Regarding next-year network forecasting, we agree that this is an interesting long-term objective. However, it is extremely challenging. It is an established strategy to solve complex problems like this by first addressing meaningful, more tractable subproblems. For that reason, we have additionally evaluated both link prediction and a two-stage forecasting approach (Tables 4–5), as requested. It is one of our main findings that these sub-tasks are already highly challenging on their own. Our proposed tasks can serve as methodological building blocks and provide valuable insights for more ambitious forecasting approaches.

Importantly, full network forecasting is not our ultimate objective: advancing methods for assessing food security is. Predicting country-level caloric supply, one of our central benchmark tasks, is directly relevant to this societally important goal - more so than full network forecasting.

We respectfully maintain that FoodTrade’s value as a challenging new benchmark should be assessed in light of these intended contributions, rather than contingent on solving substantially broader problems that it is designed to help the community address.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank the reviewers for their insightful feedback and summarize the rebuttal discussion as follows. All three reviewers are satisfied with the additional clarifications and experimental results, mainly:

  • Link prediction and removing the oracle network structure assumption for edge regression (Reviewer CTSt W1+W2+Q2): We include results for link prediction and the two-step hurdle model, i.e., link prediction followed by edge regression on the predicted links, in Tables 4 and 5, respectively.

  • Diagnostic analyses (Reviewer yKke W2+Q5): We analyze the error distribution based on node centrality (Fig. 7) and edge trade volumes (Fig. 6), which motivate our hypothesis regarding low performance of GNNs.

  • Covariate analysis (Reviewer kneH W2 & Reviewer yKke Q4): We ablate the contribution of different feature categories to model performance in Table 19 and summarize the relationship across the high-dimensional feature space through within- and cross-category correlations (Fig. 18).

Other questions regarding statistical validation (Table 13), ablations (Tables 15, 16, and 19), metric robustness (Tables 17-18), longer forecasting horizons (Table 20), and inter-layer network similarity (Fig. 17) were also addressed satisfactorily. We conclusively clarify our position regarding the unsettled points for each reviewer.

  • Reviewer CTSt insists that we demonstrate cross-domain generalizability such that model rankings are transferable to discrete-time dynamic graphs. However, this does not directly pertain to the primary contribution of the paper, i.e., a new, challenging temporal graph dataset, together with meaningful prediction tasks and an evaluation framework that can foster methodological advances in (i) graph learning and (ii) food security assessment. Therefore, we maintain that it should be assessed in light of these intended contributions, rather than contingent on solving a different problem that it is designed to help the community address. Also, please note that, in their response, the reviewer has not acknowledged that we explicitly provide results for link prediction and edge regression without conditional magnitude estimation (Tables 4-5) as requested.

  • Reviewer yKke shows concern about the core contribution being primarily the dataset and benchmark rather than a novel methodology, while they acknowledge that the detailed rebuttal strengthens the paper and there remains no concern regarding experimental evaluation. However, we note that ICLR explicitly lists ‘dataset and benchmark’ as a relevant topic in the call for papers. Therefore, the absence of a novel method does not limit the scientific contributions of this work, which should be evaluated in the appropriate context.

  • Reviewer kneH expresses reservations regarding the novelty of FoodTrade relative to TGB-trade and the role of rolling windows in the experimental setup. We clarify that rolling windows are not strictly enforced and that the dataset remains accessible as a sequence of network snapshots for the user to define splits as desired. The benchmark provides a standardized comparison, without restricting future methods. Secondly, we thoroughly distinguish TGB-Trade and FoodTrade in terms of learning tasks, trade subject, covariate information, and downstream applications.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.