- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes EvoST for traffic forecasting when sensor networks expand across successive periods. It inherits embeddings for existing nodes, initializes embeddings for new nodes, and updates a shared weekly time representation. A linearized kernel graph convolution avoids explicitly constructing a dense adjacency matrix. On three PEMS-based datasets, EvoST reports the best average MAE, RMSE, and MAPE among the listed methods.
Critical strengths
- The problem is practically meaningful. When new sensors are added, a forecasting model has to deal with both a changing parameter shape and a changing dependency structure. The paper focuses on this issue directly instead of treating network expansion as a minor implementation detail.
- The proposed update rule is simple enough to be useful. At a period transition, old rows of the node embeddings are copied, new rows are appended, and the time dictionary can be reused because it does not depend on the number of sensors. This makes the method easier to deploy than approaches that require a separate replay or pattern-retrieval pipeline.
- The empirical results are encouraging. EvoST has the lowest average errors in Table 1 on all three datasets, and the EAC+ experiment is useful because it shows that the evolving graph module is not tied exclusively to the EvoST backbone. The memory measurements on PEMS12-Stream also support the motivation for avoiding an explicit dense adjacency.
Critical weaknesses
- The continual-learning protocol is not sufficiently clear. The paper does not specify how much data from existing sensors is available when a new period begins. It also reports aggregate current-period errors without separate results for existing and newly added sensors or a forgetting metric.
- The ablations do not fully separate the contributions of the different transfer mechanisms. The retraining variant evaluates the whole inheritance strategy, while removing the graph convolution does not directly compare the proposed linearized kernel with an exact or standard adaptive graph.
- The comparison with STBP needs clarification. STBP is reported with an unfrozen backbone on two datasets because the original version produced large errors. The authors should explain this modification and provide results under a consistent comparison setting.
- The efficiency analysis does not report the random-feature dimension used in the kernel approximation. A controlled study of approximation accuracy and memory usage would make the scalability claim easier to verify.
- The figure descriptions are incomplete. Some figures do not provide sufficiently clear captions. The authors should add complete captions to improve readability.
Questions for authors
- What data from existing sensors are available during training at each period transition? Please report performance separately for old and new sensors and, if possible, include a forgetting measure.
- Can the authors provide ablations that separately test inherited node embeddings, inherited time embeddings, and inherited shared parameters?
- What random-feature dimension is used, and how does changing this dimension affect accuracy and memory?
- Please clarify the STBP implementation on each dataset and report results under comparable training settings.
9 comments
Response to reviewer 34qY
Response to W3:
We agree with this concern and understand the importance of fairness in baseline comparison. In Response of Q4, we clarify our settings of STBP and report the experimental results of both frozen and unfrozen versions of STBP in our initial manuscript. Besides, we tune the hyperparameter of STBP and report the results of both frozen and unfrozen versions. Finally, we explain our motivation of modifying the original version of STBP and diagnose the reason of abnormal accuracy metrics. These analyses guarantee the fairness in baseline comparison and performance advantage of our method.
We also want to advertise our source codes, as provided in the anonymous repository. Current public codes of continual traffic forecasting methods mostly follow the pipeline of TrafficStream. It is valuable and important, but is kinda old and not easy to read. Our framework is based on the pipeline of LargeST and has many useful modifications by ourselves to guarantee the efficiency and readability, and is easy to use. We carefully sort source codes of all baselines into our framework in a unified way. We are hopeful that our codes will benefit further researches in the continual forecasting filed.
We have added clarifications and diagnoses of STBP results in section 5.2, A.4.3 and revised the results of STBP on PEMS12-Stream dataset in Table 1 and Table 10 in the revised manuscript.
Response to W4:
We sincerely thank you for pointing out this question and make clarification in Response of Q3. The random feature dimension used in the kernel approximation is 64. In Response of Q3, we compare the approximation kernel with exact kernel. We also compare approximation kernel itself under different random feature dimensions in aspects of prediction error, training epoch time, and GPU memory usage.
We have added the used random feature dimension in section 5.1 and the controlled study of our approximation kernel in section 5.4 in the revised manuscript.
Response to W5:
We sincerely thank you for pointing out this concern. In the revised manuscript, we have added complete captions to improve readability, including Figure 5, 6, 7, 8, 9, 10, 11, 12.
Response to reviewer 34qY
Response to Q1:
We sincerely thank you for pointing out this question and make clarification as following.
During training at each period transition, the current period data of existing sensors are available. Thus, our model is trained on data of all nodes in each period, including existing sensors and newly added sensors. This setting is different from TrafficStream and STKEC which are trained only on newly added sensors, while in accordance with EAC and STBP which are trained on all sensors.
We report performance separately for old and new sensors in the following table. It can be seen that our method has satisfied results on both old, new, and all sensors. Considering the application of forecasting in real-world continual scenario, we think the model tends to predict on current period. It does not function on historical periods. In addition, the protocols are also not compatible between current and historical periods since the sensor counts are different. Thus, we are not able provide forgetting measure.
We have added clarification of data access and provide separate results of old and new nodes in section A.4.1 in the revised manuscript.
Prediction performance of EvoST on old, new, and all nodes:
PEMS-Stream
| Method | Metric | 3 | 6 | 12 | Avg. |
|---|---|---|---|---|---|
| EvoST-Old Nodes | MAE | 11.06 ± 0.03 | 11.72 ± 0.05 | 12.75 ± 0.08 | 11.71 ± 0.05 |
| RMSE | 18.69 ± 0.05 | 19.97 ± 0.07 | 21.68 ± 0.10 | 19.90 ± 0.07 | |
| MAPE(%) | 14.22 ± 0.14 | 15.07 ± 0.17 | 16.69 ± 0.22 | 15.15 ± 0.14 | |
| EvoST-New Nodes | MAE | 8.73 ± 0.02 | 9.18 ± 0.02 | 9.94 ± 0.04 | 9.18 ± 0.03 |
| RMSE | 14.54 ± 0.03 | 15.28 ± 0.04 | 16.42 ± 0.08 | 15.28 ± 0.04 | |
| MAPE(%) | 29.11 ± 0.44 | 31.75 ± 0.93 | 36.89 ± 1.27 | 32.26 ± 0.86 | |
| EvoST-All Nodes | MAE | 11.12 ± 0.03 | 11.83 ± 0.05 | 12.95 ± 0.07 | 11.83 ± 0.05 |
| RMSE | 18.60 ± 0.06 | 19.96 ± 0.09 | 21.78 ± 0.10 | 19.89 ± 0.08 | |
| MAPE(%) | 14.67 ± 0.13 | 15.60 ± 0.17 | 17.39 ± 0.21 | 15.69 ± 0.14 |
PEMS06-Stream
| Method | Metric | 3 | 6 | 12 | Avg. |
|---|---|---|---|---|---|
| EvoST-Old Nodes | MAE | 11.10 ± 0.03 | 11.90 ± 0.04 | 13.13 ± 0.09 | 11.90 ± 0.05 |
| RMSE | 18.13 ± 0.03 | 19.60 ± 0.06 | 21.64 ± 0.12 | 19.59 ± 0.07 | |
| MAPE(%) | 16.33 ± 0.12 | 17.41 ± 0.09 | 19.32 ± 0.19 | 17.48 ± 0.11 | |
| EvoST-New Nodes | MAE | 9.60 ± 0.20 | 10.46 ± 0.31 | 11.65 ± 0.51 | 10.44 ± 0.32 |
| RMSE | 15.17 ± 0.29 | 16.68 ± 0.46 | 18.47 ± 0.72 | 16.63 ± 0.48 | |
| MAPE(%) | 15.76 ± 1.78 | 17.52 ± 2.55 | 20.67 ± 4.05 | 17.67 ± 2.62 | |
| EvoST-All Nodes | MAE | 11.02 ± 0.03 | 11.81 ± 0.05 | 13.01 ± 0.10 | 11.81 ± 0.06 |
| RMSE | 17.99 ± 0.03 | 19.46 ± 0.07 | 21.47 ± 0.14 | 19.44 ± 0.08 | |
| MAPE(%) | 16.49 ± 0.31 | 17.67 ± 0.36 | 19.73 ± 0.54 | 17.74 ± 0.37 |
PEMS12-Stream
| Method | Metric | 3 | 6 | 12 | Avg. |
|---|---|---|---|---|---|
| EvoST-Old Nodes | MAE | 11.85 ± 0.04 | 12.94 ± 0.07 | 14.63 ± 0.12 | 12.94 ± 0.07 |
| RMSE | 21.18 ± 0.42 | 23.72 ± 1.20 | 27.61 ± 2.77 | 23.92 ± 1.51 | |
| MAPE(%) | 22.15 ± 0.26 | 24.02 ± 0.19 | 27.16 ± 0.31 | 24.13 ± 0.20 | |
| EvoST-New Nodes | MAE | 10.87 ± 0.04 | 11.98 ± 0.04 | 13.91 ± 0.05 | 12.03 ± 0.04 |
| RMSE | 20.74 ± 0.05 | 23.23 ± 0.11 | 27.16 ± 0.20 | 23.42 ± 0.11 | |
| MAPE(%) | 17.31 ± 0.32 | 19.00 ± 0.23 | 21.83 ± 0.29 | 19.09 ± 0.26 | |
| EvoST-All Nodes | MAE | 11.83 ± 0.04 | 12.89 ± 0.08 | 14.52 ± 0.15 | 12.89 ± 0.09 |
| RMSE | 21.44 ± 0.38 | 23.89 ± 1.15 | 27.70 ± 2.79 | 24.08 ± 1.48 | |
| MAPE(%) | 21.79 ± 0.23 | 23.59 ± 0.13 | 26.63 ± 0.46 | 23.69 ± 0.22 |
Response to reviewer 34qY
Dear reviewer 34qY,
We sincerely thank for your insightful and valuable comments from which we learn a lot to improve the manuscript. We will provide point-by-point responses as following.
We also sincerely apologize for our late reply. We are hopeful our responses will address your concerns. If you have further questions, we are still available and willing to make timely and detailed replies.
Best regards,
All authors
Response to W1:
We sincerely thank you for pointing out this concern. We provide separate results for existing and newly added sensors in Response of Q1. When a new period begins, the model is trained on data of all sensors, including all existing sensors and new sensors. Thus, our setting is different from TrafficStream and STKEC while in accordance with EAC and STBP. The main challenge we want to address is how to adapt to the evolving distributions in expanding networks, including both the spatial and temporal aspects. The most important contribution of our work is proposing the expansible adaptive adjacency learning method, which deals with a long-standing critical limitation in continual learning scenario caused by the changing sensor counts across periods. Thus, our method enables adaptive adjacency learning deployable in continual scenarios, aligning the model architecture design in classic traffic forecasting and continual forecasting scenarios.
Except for EvoST, we also provide Ada-ST and EAC+ in the initial manuscript and GWNet+ in this discussion phase. These models both equip with the expansible adaptive adjacency learning and can be directly applied in continual learning scenario. Specifically, Ada-ST shows that the classic surrogate STGNN used in TrafficStream, STKEC, and naive Retrain-ST, Pretrain-ST, Online-ST-AN, Online-ST-NN can be boosted through adaptive adjacency learning. EAC+ shows that the expansible adaptive adjacency learning plays function in harmony with the learnable node prompts proposed in EAC. The two learnable techniques are not conflicting and can both improve the performance of continual learning models. GWNet+ shows that modern STGNNs which has fixed adaptive adjacency learning can be adapted to continual learning scenario, thus aligning the advanced model designs in stationary traffic forecasting and continual scenarios. The last but not least, EvoST further provides a scalable solution of expansible adaptive adjacency learning through linearized kernel graph convolution, which is more efficient when dealing with large-scale node counts or longer period span.
In addition to the spatial modeling, the temporal component in EvoST is also an useful solution to learn temporal patterns and temporal distribution shifts. It is simple yet effective without requiring pattern bank and complex pipeline.
We have added contents to articulate the continual learning protocol, data access, and the performance results of old and new old in section A.4.1 in the revised manuscript. We also added contents to stress our contributions to the continual learning scenario in section 1, 2, 4, 5.2, 6 in the revised manuscript.
Performance comparison of GWNet and GWNet+ on PEMS-Stream:
Response to W2:
We agree with this concern and provide further comparison results in Response of Q2 and Q3. In Response of Q2, we conduct ablations on inherited node embeddings, temporal components, and shared parameters, separately. In Response of Q3, we compare our approximation kernel with exact kernel, and also the approximation kernel itself under different number of random features, according to the prediction accuracy, training epoch time and GPU memory usage.
We have added ablation variants in section 5.3 and comparisons to exact kernel in section 5.4 in the revised manuscript.