- Rating
- 1
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper addresses approximate unlearning for time series forecasting. The authors propose RDTU, which combines a retained-set NTK predictor, a structural irreplaceability score, and a conditional residual diffusion model to construct counterfactual labels, which then guide a lightweight unlearning update. Experiments on four datasets demonstrate improved functional agreement with exact retraining while maintaining forecasting utility and efficiency.
Critical strengths
- The problem to be addressed is a real need in the time series forecasting.
- The novelty of the proposed method is adequate.
Critical weaknesses
- The base forecast model used in the experiments is a Transformer. However, I cannot find any detailed introduction, neither in the main text nor in the appendix, about the structure and scale of the Transformer. The scale of the model influences the discrepancy of NTK and the performance of the diffusion module a lot for unlearning.
- If the proposed method is considered 'agnostic to the original model', experiments on different types of models, including LSTM, TCN, PatchTST, etc., should be conducted.
- One real-world motivation for unlearning is privacy, as mentioned in the Introduction. The experimental setting should align with this claim. Take the traffic dataset (which can be treated as multi-dimension time series forecasting)as an example, if one sensor is considered to have a privacy issue, then what ought to be removed from the training data is the whole corresponding channel, not some random time windows.
- The scale of test data is TOO SMALL, which brings 3 major issues: (a) Consider the time consumption for retraining in Table 3. If the longest time to retrain the model is less than 40 seconds, the practical benefit of learning is questionable. (b) In larger scale datasets, the error of Nystrom can be greater. In small datasets, this problem is hidden. (c) Larger datasets may require the diffusion to have more sampling steps, which may deteriorate the usability of the proposed method.
Based on the critical flaws w.r.t. experimental settings, I tend to give a score between 1 and 2 in the first round. However, I am open to increase my score if the above mentioned concerns can be adequately addressed.
Questions for authors
- Why use a diffusion model instead of some more efficient generative models like flow matching or rectified flow? I think it is a good idea to investigate whether these models could reduce the sampling steps while maintaining similar performance in RDTU.
- Instead of using NTK, can we simply use a lightweight NN model, like DLinear or an LSTM, to form the base forecast?
- Please also refer to my concerns listed in the weaknesses.
10 comments
Response to Reviewer wSef [part 2]
Reviewer wSef W3
One real-world motivation for unlearning is privacy, as mentioned in the Introduction. The experimental setting should align with this claim. Take the traffic dataset (which can be treated as multi-dimension time series forecasting)as an example, if one sensor is considered to have a privacy issue, then what ought to be removed from the training data is the whole corresponding channel, not some random time windows.
Thanks for the question. We designed experiments for both single-variable and multi-variable scenarios. For the single-variable data, we selected the complete data of 10 users for the experiment, and randomly selected all the data of one of the users for forgetting. For multivariate data, we also selected 10 users and randomly chose the data from a certain sensor/dimension of one of the users for forgetting. The detailed results are shown in the following table. It can be observed that our method has significant advantages and performance in this scenario.
| Method | Air U/N | Air F-P Gap (%) | Air NRMSE | Air Time (s) | Traffic U/N | Traffic F-P Gap (%) | Traffic NRMSE | Traffic Time (s) |
|---|---|---|---|---|---|---|---|---|
| Original | 0.5988 ± 0.0666 | -- | 0.2217 ± 0.0011 | -- | 0.7319 ± 0.0621 | -- | 0.3153 ± 0.0100 | -- |
| Retrain | 0.5568 ± 0.0229 | -- | 0.2250 ± 0.0048 | 192.7964 ± 58.2205 | 0.6135 ± 0.0312 | -- | 0.3163 ± 0.0058 | 119.0407 ± 30.3364 |
| SSD | 0.5613 ± 0.0363 | 59.6761 ± 21.2877 | 0.6040 ± 0.1651 | 0.6852 ± 0.1042 | 0.5628 ± 0.0516 | 78.3237 ± 45.0656 | 0.6812 ± 0.2257 | 0.5832 ± 0.1281 |
| NegGrad+ | 0.5571 ± 0.0402 | 859.6122 ± 91.8490 | 7.3509 ± 0.7707 | 81.3413 ± 3.1315 | 0.5695 ± 0.0452 | 861.6575 ± 138.9572 | 2.6960 ± 0.7017 | 33.8003 ± 1.5389 |
| Zero-label | 0.5519 ± 0.0269 | 66.2707 ± 8.5350 | 0.6293 ± 0.0529 | 15.4022 ± 0.8456 | 0.5832 ± 0.0625 | 79.9763 ± 38.9463 | 0.3794 ± 0.0479 | 6.6967 ± 0.6467 |
| Random-label | 0.5590 ± 0.0250 | 68.4220 ± 8.0251 | 0.6503 ± 0.0512 | 17.3528 ± 2.0106 | 0.5809 ± 0.0559 | 81.2705 ± 38.6683 | 0.3959 ± 0.0480 | 6.8879 ± 0.6449 |
| SCRUB-R | 0.5531 ± 0.0352 | 24.3947 ± 0.9358 | 0.3234 ± 0.0013 | 97.3652 ± 5.0673 | 0.6251 ± 0.0518 | 30.8972 ± 9.0358 | 0.3624 ± 0.0103 | 47.1269 ± 2.1898 |
| TS-Unlearn | 0.5598 ± 0.0302 | 70.3468 ± 7.2247 | 0.6664 ± 0.0474 | 16.6571 ± 2.3968 | 0.6133 ± 0.0613 | 81.0927 ± 35.6063 | 0.3869 ± 0.0449 | 7.6543 ± 1.2011 |
| RDTU | 0.5507 ± 0.0437 | 6.1794 ± 1.3391 | 0.2293 ± 0.0044 | 3.4558 ± 0.1674 | 0.6083 ± 0.0693 | 23.1773 ± 9.9836 | 0.3267 ± 0.0138 | 5.2726 ± 0.1996 |
| Method | EEG U/N | EEG F-P Gap (%) | EEG NRMSE | EEG Time (s) | HAR70+ U/N | HAR70+ F-P Gap (%) | HAR70+ NRMSE | HAR70+ Time (s) |
|---|---|---|---|---|---|---|---|---|
| Original | 0.5670 ± 0.0737 | -- | 0.0622 ± 0.0078 | -- | 0.5981 ± 0.0471 | -- | 0.1244 ± 0.0070 | -- |
| Retrain | 0.5597 ± 0.0510 | -- | 0.0580 ± 0.0083 | 114.1573 ± 15.6318 | 0.5847 ± 0.0712 | -- | 0.1219 ± 0.0071 | 227.1485 ± 27.6805 |
| SSD | 0.5548 ± 0.0410 | 84.4921 ± 2.5816 | 0.6527 ± 0.0776 | 0.4189 ± 0.0993 | 0.5407 ± 0.0284 | 26.5559 ± 10.4653 | 0.5155 ± 0.0971 | 0.6252 ± 0.1200 |
| NegGrad+ | 0.5658 ± 0.0523 | 1011.6810 ± 254.6942 | 4.6332 ± 1.1332 | 26.7890 ± 2.1131 | 0.5636 ± 0.0332 | 656.0686 ± 353.6126 | 1.3259 ± 0.6860 | 62.9475 ± 1.3050 |
| Zero-label | 0.5863 ± 0.0514 | 14.4248 ± 2.1941 | 0.0891 ± 0.0090 | 4.2419 ± 0.3346 | 0.5830 ± 0.0710 | 26.6644 ± 11.6035 | 0.1225 ± 0.0070 | 12.6703 ± 3.7334 |
| Random-label | 0.5884 ± 0.0799 | 14.7301 ± 4.5221 | 0.0861 ± 0.0117 | 4.8149 ± 0.9715 | 0.5807 ± 0.0729 | 54.3932 ± 16.2876 | 0.1470 ± 0.0098 | 14.4579 ± 4.7531 |
| SCRUB-R | 0.5646 ± 0.0543 | 15.8047 ± 2.6305 | 0.0988 ± 0.0085 | 30.9713 ± 1.5701 | 0.5813 ± 0.0582 | 18.2323 ± 6.5166 | 0.1306 ± 0.0069 | 76.7681 ± 3.9383 |
| TS-Unlearn | 0.5899 ± 0.0624 | 15.4128 ± 5.2699 | 0.0901 ± 0.0101 | 4.9452 ± 0.9133 | 0.5802 ± 0.0670 | 45.1040 ± 13.5515 | 0.1389 ± 0.0082 | 14.2351 ± 3.7874 |
| RDTU | 0.5597 ± 0.0491 | 14.2764 ± 10.4858 | 0.0629 ± 0.0094 | 4.4898 ± 0.2362 | 0.5721 ± 0.0734 | 16.5749 ± 6.0840 | 0.1233 ± 0.0077 | 6.7578 ± 0.1847 |
Response to Reviewer wSef [part 3]
Reviewer wSef W4
The scale of test data is TOO SMALL, which brings 3 major issues: (a) Consider the time consumption for retraining in Table 3. If the longest time to retrain the model is less than 40 seconds, the practical benefit of learning is questionable. (b) In larger scale datasets, the error of Nystrom can be greater. In small datasets, this problem is hidden. (c) Larger datasets may require the diffusion to have more sampling steps, which may deteriorate the usability of the proposed method.
Thanks for the question. We conducted experiments of two larger common time series forecasting benchmark datasets(Ettm1, Weather), the results of the transformer base model shows below and more results of other base model have been added to Appendex.
| Method | ETTm1 NRMSE | ETTm1 F-P Gap% | ETTm1 U/N | ETTm1 Time | Weather NRMSE | Weather F-P Gap% | Weather U/N | Weather Time |
|---|---|---|---|---|---|---|---|---|
| Original | 0.448866 | 13.760 | 0.829193 | — | 0.180288 | 21.297 | 0.842255 | — |
| Retrain | 0.453708 | 0.000 | 0.530762 | 47.291 | 0.200102 | 0.000 | 0.562683 | 44.892 |
| SSD | 0.849014 | 29.356 | 0.501617 | 1.209 | 0.299896 | 41.233 | 0.590256 | 1.297 |
| NegGrad+ | 1.184357 | 217.511 | 0.713577 | 36.925 | 0.347505 | 92.723 | 0.680862 | 40.030 |
| Zero Label | 0.502257 | 21.676 | 0.738297 | 3.074 | 0.208587 | 33.876 | 0.720856 | 3.259 |
| Random Label | 0.529354 | 22.316 | 0.727142 | 3.025 | 0.204099 | 34.047 | 0.724121 | 3.290 |
| SCRUB-R | 0.477872 | 33.176 | 0.671707 | 4.592 | 0.200495 | 31.707 | 0.687903 | 5.142 |
| TS-Unlearn | 0.469907 | 28.120 | 0.787582 | 17.818 | 0.280045 | 59.479 | 0.673721 | 22.547 |
| Ours | 0.461662 | 17.459 | 0.527878 | 10.251 | 0.186133 | 16.820 | 0.660202 | 13.914 |
We can find that our model still performs well on larger scale datasets, although the error of Nystrom is greater, the residual capture of diffusion can effectively make up for this part. In addition, we have optimized the implementation of diffusion in terms of efficiency, such as DDPM acceleration. Since time-series data is inherently not overly complex, the optimized diffusion can effectively mitigate the time impacts caused by multi-step sampling and other issues.
Reviewer wSef Q1
Why use a diffusion model instead of some more efficient generative models like flow matching or rectified flow? I think it is a good idea to investigate whether these models could reduce the sampling steps while maintaining similar performance in RDTU.
Thanks for the question. We conducted ablation experiments by replacing diffusion with flow matching and rectified flow, and the specific results are shown in the table below. It can be observed that, without significant differences in running time, flow matching and rectified flow did not achieve better performance than diffusion. On one hand, this is because in the diffusion part, we used DDPM to accelerate and improve the running efficiency of diffusion. On the other hand, we believe that diffusion can better capture this uncertain residual relationship.
| Method | Air Quality U/N | Air Quality F-P Gap | Air Quality NRMSE | Air Quality Time | Traffic U/N | Traffic F-P Gap | Traffic NRMSE | Traffic Time | EEG U/N | EEG F-P Gap | EEG NRMSE | EEG Time | HAR70 U/N | HAR70 F-P Gap | HAR70 NRMSE | HAR70 Time |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| NTK + Diffusion (Ours) | 0.5001 | 4.6312 | 0.2600 | 11.710 | 0.5420 | 9.9251 | 0.3935 | 4.261 | 0.5244 | 3.7526 | 0.1110 | 2.322 | 0.5178 | 9.7703 | 0.3424 | 5.744 |
| NTK + VP Flow Matching | 0.5383 | 5.3320 | 0.2686 | 8.921 | 0.5421 | 10.3264 | 0.3961 | 7.441 | 0.5268 | 3.9202 | 0.1102 | 2.896 | 0.5216 | 10.5770 | 0.3491 | 6.822 |
| NTK + Rectified Flow | 0.5314 | 5.3066 | 0.2668 | 8.911 | 0.5431 | 10.3287 | 0.3955 | 7.382 | 0.5289 | 3.8073 | 0.1100 | 2.895 | 0.5232 | 10.4007 | 0.3486 | 6.870 |
Response to Reviewer wSef [part 1]
Reviewer wSef W1
Transformer: 4 layers, hidden dimension 128, 8 heads, feed-forward dimension 256, dropout 0.1, and GELU activation. It is trained with AdamW for at most 80 epochs using batch size 512, learning rate 3×10−4, weight decay 10−4.
LSTM: 2 layers, hidden dimension 256, dropout 0.1, unidirectional recurrence, and a linear prediction head. It is trained with AdamW for at most 200 epochs using batch size 512, learning rate 10−3, and weight decay 10−4.
TCN: 4 residual blocks, 128 channels, 2 causal convolutions per block, kernel size 3, dilation factors {1,2,4,8}, dropout 0.1, weight normalization, and ReLU activation. It is trained with AdamW for at most 200 epochs using batch size 512, learning rate 10−3, and weight decay 10−4.
PatchTST: 2 encoder layers, hidden dimension 128, 8 heads, feed-forward dimension 256, patch length 4, stride 2, dropout 0, and GELU activation, with channel independence, RevIN, and a flatten-and-linear prediction head. It is trained with AdamW for at most 200 epochs using batch size 512, learning rate 10−3, and weight decay 10−4.
Reviewer wSef W2
Thanks for the suggestion. We conducted supplementary experiments on different base models, and the specific results of LSTM are shown in the following table, results of TCN, PatchTST are added in Appendix. LSTM: