acceptodds
Under review as a conference paper at ICLR 2027

ADDP: Coordinated Historical Pruning via a Shared Continuous Routing Signal for Irregular Multivariate Time Series Forecasting

Abstract

Effectively utilizing historical observations is fundamental to irregular multivariate time series forecasting. However, real-world time series often contain redundant segments that distract model attention and degrade forecasting performance. Existing pruning methods for low-information data typically rely on independently designed decision rules for different components, making it difficult for different pruning strategies to coordinate or conflict on the same sample. We propose Adaptive Data-Dependent Pruning (ADDP), a framework that uses a shared signal to guide multiple pruning decisions during training. Specifically, ADDP summarizes each sample's historical variation in a shared signal that guides subsequent pruning decisions. Lightweight learnable calibrators then use this signal to determine the redundancy of historical segments, the amount of information shared between aggressive and conservative pruning views, and the weights assigned to these views. Based on these decisions, ADDP constructs two complementary views that retain different amounts of historical information and jointly trains a shared forecasting model. Experiments on PhysioNet, Activity, USHCN, and MIMIC-III show that ADDP consistently outperforms competing methods, achieving the lowest mean MSE on each of the four datasets. On MIMIC-III, ADDP reduces MSE by 5.05% relative to Hi-Patch, the strongest evaluated baseline.

Primary area:
learning on time series and dynamical systems
Keywords:
irregular time series, multivariate time series forecasting, continuous volatility indexing, adaptive pruning, robust statistics, Transformer, missing data, attention collapse

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_tTExOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Adaptive Data-Dependent Pruning (ADDP), a training framework for irregular multivariate time-series forecasting. ADDP includes patch scoring, the size of a shared core retained across two training views, and the temperature of a smooth worst-view objective. Conservative and aggressive views share the same forecasting backbone, while inference uses the complete observed history in a single pass.

Critical strengths
  1. Clear motivation. Historical observations can contain redundant segments, and the same pruning rule may not work well for every sample. Using a history-dependent signal to adjust pruning is a reasonable way to address this problem.

  2. Clear presentation and visualization. The figures and tables make the ADDP pipeline easy to follow and clearly present the component analyses, helping readers understand the shared signal and how it guides the three decisions in ADDP.

  3. Solid experiments. The paper evaluates ADDP on four common datasets and compares it with a broad range of baselines. The backbone comparisons, component ablations, and sensitivity analyses provide useful support for the reported improvements.

Critical weaknesses
  1. It is unclear how some "learnable calibrators" actually learn. The paper says these calibrators are updated through the training objective, but does not explain how gradients pass through these decisions.

  2. The data splits need more detail. For Activity, four-second windows are generated with a 500 ms stride, so neighboring windows overlap substantially. It is important to clarify how these windows are separated into training, validation, and test sets before drawing conclusions from the reported improvements.

Questions for authors
  1. How are the calibrators updated? Could the authors explain how the patch-scoring and shared-core calibrators are updated during training?
5 comments
AuthorsOpenReview ↗

Response to Reviewer tTEx

Response to Reviewer tTEx

Thank you for highlighting the clear motivation, the presentation of the ADDP pipeline, and the breadth of our experimental evaluation. In response to your questions about calibrator updates and the Activity data split, we recorded gradients during training and retrained both methods on an Activity split with disjoint recording IDs.


W1 & Q1. How do the scoring and shared-core calibrators learn with hard ranking and rounding?

A single optimizer updates the forecasting backbone and all three calibrators in the same training run. The scoring calibrator receives gradients through continuous patch scores, and the shared-core calibrator receives gradients through the shared-core ratio before rounding. Sections 3.6–3.8 already define the continuous mappings for all three calibrators. In the forward pass, ADDP uses hard ranking and top-kk selection to choose patches and rounding to determine the size of the shared core. Gradients stop at these discrete operations. The table below shows which loss updates each parameter group.

Parameter groupOutput in the forward passLoss used for the update
Scoring calibrator (ah,aℓ,au)(\mathbf a_h,\mathbf a_\ell,a_u)Continuous patch scores sps_p used for rankingScore margin loss Lscore\mathcal L_{\mathrm{score}}
Shared-core calibrator (ac,bc)(a_c,b_c)Shared-core ratio ρ(γ)\rho(\gamma) before roundingShared-core regularizer Lcore\mathcal L_{\mathrm{core}}
Temperature calibrator (aτ,bτ)(a_\tau,b_\tau)Temperature τ(γ)\tau(\gamma) used to combine the two view lossesMain forecasting loss Lpred\mathcal L_{\mathrm{pred}}
Forecasting backbonePredictions for the two training viewsMain forecasting loss Lpred\mathcal L_{\mathrm{pred}}

The full training objective is

=\mathcal L_{\mathrm{pred}} +\lambda_s\mathcal L_{\mathrm{score}} +\lambda_c\mathcal L_{\mathrm{core}}.$$ **Scoring calibrator.** Section 3.6 already defines the softmax weights for the two scoring channels and the recency weight $\lambda_u=\operatorname{softplus}(a_u)$. Once the forward pass identifies the kept and dropped patches, the score margin loss uses the continuous scores of the two groups directly. $$\mathcal L_{\mathrm{score}} =\left[m+\bar s_{\mathrm{drop}}-\bar s_{\mathrm{keep}}\right]_+.$$ If the average score of the kept patches falls short of the required margin above the average score of the dropped patches, this loss updates the scoring weights through the continuous scores of both groups. Hard ranking determines group membership, and the margin loss adjusts the scores. **Shared-core calibrator.** Section 3.7 already defines the continuous shared-core ratio. $$\rho(\gamma) =\rho_{\min}+(\rho_{\max}-\rho_{\min}) \operatorname{sigmoid}\!\left(a_c-\operatorname{softplus}(b_c)\gamma\right).$$ The forward pass uses this ratio to determine how many patches the two views share. The shared-core regularizer acts directly on $\rho(\gamma)$ before rounding. This gives $a_c,b_c$ a gradient and discourages extreme shared-core ratios. **Temperature calibrator.** Section 3.8 already defines how to combine the losses from the two views. For a sample with forecasting losses $\ell_1$ and $\ell_2$, the combined loss is $$\tau(\gamma)\log\!\left[ \frac{\exp(\ell_1/\tau(\gamma))+\exp(\ell_2/\tau(\gamma))}{2} \right].$$ A lower temperature gives more weight to the view with the higher loss. Because $\tau(\gamma)$ enters this calculation directly, the main forecasting loss can update the temperature parameters. **Gradients and changes during training.** We recorded the fraction of training batches with nonzero gradients for each calibrator on Activity. | Activity calibrator | Gradient source | Training batches with nonzero gradients | |---|---|---:| | Scoring calibrator | Score margin loss | 99.25% | | Shared-core calibrator | Shared-core regularizer | 100% | | Temperature calibrator | Main forecasting loss | 99.89%–100% | | Activity calibration quantity | Epoch 1 | Epoch 49 | |---|---:|---:| | Weight on within-patch variation in the high-roughness channel | 0.6400 | 0.8632 | | Weight on patch-mean deviation in the low-roughness channel | 0.4316 | 0.6861 | | Recency weight $\lambda_u$ | 0.3057 | 3.5411 | | Mean shared-core ratio $\rho$ | 0.4263 | 0.4419 | | Mean loss temperature $\tau$ | 1.4747 | 2.2747 | --- Due to the character limit, please see the remaining questions in the following comment.

Continued from previous comment

In this run, the scoring and shared-core calibrators received gradients in 99.25% and 100% of training batches, respectively. By epoch 49, the recency weight had increased from 0.3057 to 3.5411, and the mean fraction of patches shared by the two views had moved from 42.63% to 44.19%. These results show that the continuous gradient paths are active in training and that both patch scoring and the shared-core ratio change over the course of training. We will specify the full training objective in §§3.6–3.8 and show which loss updates each parameter group in the pseudocode in Appendix A.


W2. How were the overlapping Activity windows split, and what happens when recording IDs are kept in separate splits?

The original experiment generated windows before randomly splitting them, so overlapping windows from the same recording could appear in different splits.

We split the data by recording ID and retrained both methods. The 25 recording IDs (A01–E05) represent five people with five recordings each. We assigned 15 recording IDs to training, 5 to validation, and 5 to testing before generating windows. All windows from a given recording stay in the same split. Different recordings from the same person can appear in different splits. We computed normalization statistics using only historical observations from the training recordings. Both methods used the same recording-ID split, training budget, and evaluation procedure.

Activity recording-ID splitTest MSE ↓
Backbone0.0025566
Backbone + ADDP0.0023427

On the held-out recordings, ADDP reduced test MSE from 0.0025566 to 0.0023427, an 8.37% reduction. The revised “Dataset Tasks” subsection describes the original window-level split, the recording-ID assignments, the window generation order, and the normalization protocol. “Activity Evaluation with Held-Out Recordings” reports the paired results. The code is available in the anonymous ADDP repository.


Finally, thank you once again for the patient and thorough reading of our paper and recognizing its strengths. Your thoughtful questions and suggestions helped us improve the manuscript. We would be happy to continue the discussion if you have any further questions!

Thank you for the detailed response and for rerunning the Activity experiment. The explanation of the score margin loss and the shared-core regularizer clarifies how the two calibrators are updated despite the discrete selection steps. The gradient measurements also address my question about whether these parameters receive updates in practice. The new experiment with disjoint recording IDs addresses my concern about overlapping windows across splits. It is reassuring that ADDP still improves over the backbone under this split. Please include the full training objective and the recording-level split protocol in the revised paper. The response addresses my main concerns. I remain in favor of acceptance.

Reviewer_w1XBOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed ADDA for pruning in irregular multivariate time series forecasting. This method measures a roughness score from individual sample history and then utilizes said score to guide patch selection, shared-context size, and the loss weight for a dual-view training. One view keeps more history and the other has a higher retention but both views use the same forecaster. At test time, the model uses the full history with no changes. Results on four benchmark datasets demonstrated better performance in terms of MSE than the studied baselines. Finally, an ablation study supports the utility of various components such as the routing and the dual-view design.

Critical strengths
  • The paper is relevant to ICLR audience and has enough technical novelties.
  • The paper offers an interesting diagnostic study based on data points removal which provided motivation for ADDA.
  • The paper offers substantial empirical evidence.
  • The improvements seem reasonable and positive in majority of studied scenarios with reported results over multiple runs with standard deviation (though significant testing was not done).
Critical weaknesses
  • The current diagnostic is useful motivation, but repeating it for ADDA vs. the baseline transformer would substantially strengthen the message of the paper because it would connect its proposed training algorithm to the optimization and attention behaviour it claims are problems with other methods.

  • ADDA is positioned as an adaptive historical pruning but the paper does not clarify its computational implications. The current method constructs two masked training views while retaining the original token structure. It is unclear whether pruning reduces computational cost or instead introduces a roughly dual-view training overhead. Please provide some justification e.g., theoretical complexity relative to backbones, empirical training time, memory usage, parameter count, or latency compared to the baselines.

  • There is no reference to the code. I urge authors to make code publicly available under an anonymous repo and later on in a repository.

nit: images are rendering with delay for me which makes reading the paper less pleasant. Perhaps the quality of the image can be adjusted a bit to have the same message but renders without delay.

Questions for authors
  • How does ADDA training impact robustness of the model at inference time. Right now the original signal with full history is tested without any changes but I am curious if ADDA can improve harder setups e.g., with wider observation gaps or shorter histories.

  • The learned calibration parameters are central to the ADDA. How does lambda evolve during training? It would be helpful to see trajectory of lambda for various datasets and backbones

  • Does ADDA train models end-to-end? If so, how does it achieve this with non-differentiable operations like ranking e.g., in training calibrators

  • Why the improvement is less profound on USHCN benchmark? with brings up a bigger question on applicability domain of ADDA. The paper mentions irregular multivariate forecasting as its claim to fame but its applicability domain is not clearly defined. It would be extremely helpful to clarify the conditions under which ADDP is expected to help e.g., any requirements on historical length, observation density, miss data points, temporal irregularity rate, and redundancy in the signal. Similarly, it would be helpful to mention settings for which the roughness-based routing may be less reliable e.g, highly noisy or extremely sparse data, regular time-series, or settings with distribution shift.

10 comments
AuthorsOpenReview ↗

Response to Reviewer w1XB

Response to Reviewer w1XB

Thank you for highlighting the technical novelty, the diagnostic study based on observation removal, and the empirical results across repeated runs. We added same-backbone diagnostics, measurements of training cost, results under incomplete test histories, and traces of the calibrators and their gradients during training.


W1 & Q1. How does ADDP affect optimization and attention behavior? How robust is it at inference?

Optimization and attention behavior. Table 14 in the Diagnostic Definitions subsection of the appendix already reports an observation-removal experiment with iTransformer on Activity. As the fraction of history retained falls from 40% to 20% and then 10%, gradient norms at initialization decrease and attention concentrates on fewer positions.

Activity history retained (Table 14)Gradient norm at initializationNormalized attention entropyEffective attention span NeffN_{\mathrm{eff}}
100%4.61 ± 0.780.99411.82
60%4.02 ± 0.600.99211.76
40%3.87 ± 0.650.99511.85
20%3.19 ± 0.770.7105.84
10%1.46 ± 0.600.2141.70

Following your suggestion, we compared PatchTST and PatchTST + ADDP on the same USHCN split. At each retention level, both models received the same test histories.

USHCN test historyPatchTST MSE ↓PatchTST + ADDP MSE ↓MSE reductionGradient norm (PatchTST → +ADDP)Attention entropy (PatchTST → +ADDP)
Full history0.5258750.5198091.15%0.888 → 0.6720.772 → 0.660
Randomly retain 50% of history0.6581140.57480712.66%1.338 → 0.9930.737 → 0.639
Randomly retain 25% of history0.8159710.72952810.59%1.491 → 2.0350.594 → 0.521

Given the same history, the model trained with ADDP makes more accurate predictions and has more concentrated attention in all three settings. Its gradient norm is lower with the full history and at 50% retention, and higher at 25% retention. At 25% retention, ADDP still reduces MSE by 10.59%, showing that better predictions can coincide with a larger gradient norm. The revised appendix reports the paired results and test-history conditions after Diagnostic Definitions.

Robustness at inference. Using the PhysioNet checkpoints from seed 0, we tested shorter histories, randomly missing observations, and wider observation gaps (contiguous intervals masked at test time).

PhysioNet test historyBackbone MSE ↓ADDP MSE ↓MSE reduction
Full history0.0041670.0038587.42%
Most recent 75% of history0.0042660.0040744.50%
Most recent 50% of history0.0044240.0042214.59%
Most recent 25% of history0.0048940.0047293.37%
Randomly remove 10% of observations0.0042330.0039616.43%
Randomly remove 30% of observations0.0044660.0041976.02%
Randomly remove 50% of observations0.0048610.0045506.40%
Mask a contiguous 2 h interval0.0041980.0039126.81%
Mask a contiguous 4 h interval0.0042190.0039486.42%
Mask a contiguous 8 h interval0.0043270.0040775.78%

ADDP retains its accuracy gains when the test history is incomplete. On PhysioNet, it lowers MSE by 3.37%–6.81% across all nine missing-history settings. On USHCN, it lowers MSE by 12.66% and 10.59% when only half or a quarter of the history is randomly retained, respectively. In these settings, a model already trained with ADDP can make more accurate predictions from shorter histories, fewer observations, or longer gaps without being retrained for those conditions.


Due to the character limit, please see the remaining questions in the following comment.

Continued from previous comment


W2. What are the computational implications of the two masked training views?

Methodology and Limitations explain that the two training views share a forecasting backbone and that inference uses a single forward pass over the full history. The current implementation masks removed observations while keeping the token count fixed. For a backbone with full attention, each view therefore still incurs O(L2d)O(L^2d) attention cost.

We measured the costs using the same PhysioNet inputs, forecasting backbone, batch size, hardware, and numerical precision.

MethodTraining time per step ↓Throughput (original samples/s) ↑Peak GPU memory ↓Inference latency per batch ↓
Full-history backbone45.56 ms495.601.307 GiB6.59 ms
Random single-view pruning59.41 ms370.291.307 GiB7.28 ms
Random dual views60.86 ms371.522.593 GiB6.28 ms
ADDP74.51 ms320.442.594 GiB6.01 ms

Compared with the full-history backbone, ADDP takes 63.5% longer per training step and uses about twice the peak GPU memory. Random dual views already use 2.593 GiB, close to ADDP's 2.594 GiB. This indicates that most of the extra memory comes from training on two views. The calibrators add just 10 scalar parameters, equivalent to about 0.006% of the 165,634 backbone parameters in the Activity configuration.

Compared with random dual views, ADDP takes 13.65 ms (22.4%) longer per training step and uses just 0.001 GiB more peak GPU memory. Table 9 already reports the accuracy comparison. Full ADDP achieves a test MSE of 0.003860**, which is 6.01% lower than the 0.004107 obtained with random dual views and mean loss. This extra cost is incurred during training. At deployment, ADDP still needs only one forward pass over the full history. We consider the extra 13.65 ms per step a reasonable trade-off for a 6.01% reduction in test MSE, given the nearly identical peak GPU memory and the same inference procedure. Model Analyses summarizes the comparison, Training Cost and Convergence gives the timing table, and Limitations describes masking.


W3. Code and reproducibility

The submitted supplementary material already includes the ADDP implementation, data preprocessing, and training configurations. And we also agree that sharing a direct link makes the code easier to find. Anonymous ADDP repository Thank you for the suggestion!


Nit. Figure loading


Yep... We noticed this too when we went back through the paper after the deadline T_T. We've already replaced the figures with faster-loading vector versions. Thanks for pointing this out!


Q2 & Q3. How do the calibration parameters evolve, and how are they updated with non-differentiable ranking?

How the parameters change (Q2). Sections 3.6–3.8 already define the scoring weights, the shared-core ratio, and the loss temperature. We tracked these quantities through a full training run on Activity. The recency weight (denoted λu\lambda_u in the manuscript) controls how strongly the score favors recent patches, ρ\rho is the fraction of patches shared by the two views, and τ\tau controls how their losses are combined. The last two columns report averages across samples.

EpochEnergy weight in the high-roughness channelDeviation weight in the low-roughness channelRecency weight λu\lambda_uMean ρ\rhoMean τ\tau
10.64000.43160.30570.42631.4747
100.65400.58750.45600.42261.9510
200.68540.63581.26020.43032.1421
300.78450.65932.08320.43572.2432
400.84380.67402.84780.43962.2817
490.86320.68613.54110.44192.2747

The two scoring channels put more weight on different cues as training progressed, consistent with why we designed them separately. The channel for rougher histories gave more weight to variation within a patch. The channel for smoother histories gave more weight to how far the patch mean was from the overall history mean. The recency weight λu\lambda_u rose from 0.3057 to 3.5411, giving recent patches a larger boost later in training. The average proportion of patches shared by the two views stayed around 42%–44%. Mean temperature rose early and then leveled off. For a given difference between the two view losses, a higher temperature gives the views more similar weights. Calibrator Trajectories and Gradient Diagnostics reports these Activity measurements.


Due to the character limit, please see the remaining questions in the following comment.

Continued from previous comment


How the parameters are updated (Q3). A single optimizer updates the forecasting backbone and all three calibrators in the same training run. In the forward pass, ADDP constructs the two views using hard ranking, rounding, and top-kk selection. Gradients stop at these discrete operations. The scoring and shared-core calibrators are updated by auxiliary losses that act directly on continuous values. The main forecasting loss updates the temperature calibrator.

Parameter groupRole in the forward passGradient source
Forecasting backboneMakes predictions for both training viewsMain forecasting loss Lpred\mathcal L_{\mathrm{pred}}
Scoring calibrator (ah,aℓ,au)(\mathbf a_h,\mathbf a_\ell,a_u)Computes continuous patch scores sps_p for rankingScore margin loss Lscore\mathcal L_{\mathrm{score}}
Shared-core calibrator (ac,bc)(a_c,b_c)Computes the shared-core ratio ρ(γ)\rho(\gamma) before roundingShared-core regularizer Lcore\mathcal L_{\mathrm{core}}
Temperature calibrator (aτ,bτ)(a_\tau,b_\tau)Combines the two forecasting losses using τ(γ)\tau(\gamma)Main forecasting loss Lpred\mathcal L_{\mathrm{pred}}
Hard ranking, rounding, and top-kk selectionConstruct the two views from the scores and core sizeGradients stop at these operations

The full training objective is

+\lambda_s\mathcal L_{\mathrm{score}} +\lambda_c\mathcal L_{\mathrm{core}}.$$ **Scoring calibrator.** Section 3.6 already defines the softmax weights for the two scoring channels and the recency weight $\lambda_u=\operatorname{softplus}(a_u)$. Once the forward pass has identified the kept and dropped patches, the score margin loss uses the continuous scores of those two groups directly. $$\mathcal L_{\mathrm{score}} =\left[m+\bar s_{\mathrm{drop}}-\bar s_{\mathrm{keep}}\right]_+.$$ If the kept patches do not outscore the dropped patches by the required margin on average, this loss updates the scoring parameters through their continuous scores. Group membership is set in the forward pass, so the backward pass only needs gradients with respect to the scores. **Shared-core calibrator.** Section 3.7 already defines the continuous shared-core ratio. $$\rho(\gamma)=\rho_{\min}+(\rho_{\max}-\rho_{\min}) \operatorname{sigmoid}\!\left(a_c-\operatorname{softplus}(b_c)\gamma\right).$$ The forward pass uses this ratio to determine how many patches are in the core. The shared-core regularizer acts on $\rho(\gamma)$ before rounding. This supplies gradients to $a_c,b_c$ and keeps the ratio away from settings where everything or nothing is shared. **Temperature calibrator.** Section 3.8 already defines how the two view losses are combined. For a sample with forecasting losses $\ell_1$ and $\ell_2$, the combined loss is $$\tau(\gamma)\log\!\left[ \frac{\exp(\ell_1/\tau(\gamma))+\exp(\ell_2/\tau(\gamma))}{2} \right].$$ A lower temperature puts more weight on the view with the higher loss. Since $\tau(\gamma)$ enters this calculation directly, the main forecasting loss can update the temperature parameters. | Activity calibrator diagnostic | Gradient source | Training batches with nonzero gradients | |---|---|---:| | Scoring calibrator | Score margin loss | 99.25% | | Shared-core calibrator | Shared-core regularizer | 100% | | Temperature calibrator | Main forecasting loss | 99.89%–100% | In practice, all three calibrators received gradients in nearly every batch. Hard ranking can be used to build the views because the scoring and shared-core calibrators each have a loss that acts directly on their continuous outputs. The revised Methodology gives the training objective, and Notation and Pipeline Pseudocode identifies the loss used to update each parameter group. --- Due to the character limit, please see the remaining questions in the following comment.
Reviewer_TbiJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ADDP, a training framework for irregular multivariate time series forecasting that adaptively selects historical information. The authors argue that applying the same pruning rule to all inputs and independently determining patch selection, information retention, and view weighting can limit effective learning. ADDP computes a roughness index from historical observations and uses it to coordinate patch ranking, the amount of information shared between two training views, and the emphasis placed on the higher-loss view. The two views retain different amounts of historical information and share the same forecasting model. At inference, the model uses the full history. The paper reports improved forecasting performance across four datasets. Its main contribution is a training strategy that coordinates information selection and augmentation through a shared input-dependent signal.

Critical strengths

1.The paper addresses a practically relevant problem. Observation density does not necessarily reflect predictive information in irregular time series, so adapting historical selection to each input is a reasonable direction. 2. The reported improvements cover multiple datasets and backbones. The paper also includes comparisons with random pruning and independent routing signals to examine the contributions of its design.

Critical weaknesses

1.The severity of the underlying problem and the necessity of the proposed solution are not sufficiently established.

  • The paper provides limited direct evidence of when and how much historical redundancy or independently controlled pruning decisions hinder learning. It is also unclear whether roughness reliably reflects redundancy, and why it should jointly determine patch selection, shared-core retention, and loss weighting. The connection between the identified limitations and the proposed design therefore remains largely heuristic.
  1. The source of the gains and their value relative to the added complexity remain unclear.
  • Lower forecasting error is encouraging, but it does not establish that removing redundancy improves optimization or representation learning. Changes in initialization gradient norms and attention entropy do not directly demonstrate this either. Since dual-view training requires additional computation, comparisons under matched compute budgets are needed to establish its advantage over simpler alternatives such as random pruning.
Questions for authors

1.Can the authors provide direct evidence that redundant history and independently controlled pruning decisions hinder learning?

  • I would like to understand what constitutes harmful redundancy and how substantial its effect is. Could controlled experiments that vary redundancy or distinguish meaningful changes from noise show that roughness-based coordination specifically addresses this problem?
  1. How can we determine whether the gains come from the proposed coordination and justify the additional training cost?
  • Comparisons with random pruning and simple dual-view training, using the same backbone and compute budget, would help separate the proposed mechanism from general augmentation effects. Under these conditions, does ADDP also improve convergence speed or training stability, beyond reducing final forecasting error?
8 comments
AuthorsOpenReview ↗

Response to Reviewer TbiJ

Response to Reviewer TbiJ

Thank you for highlighting the practical relevance of adaptive history selection, the improvements across datasets and backbones, and the comparisons with random pruning and independent routing signals. We respond point by point using comparisons already reported in the paper and new experiments.


W1 & Q1 | When does historical redundancy hurt, and why coordinate the three pruning decisions?

How much does redundant history hurt? Table 1 already shows that ADDP achieves lower test MSE than the strongest baseline on each of the four datasets.

Dataset (Table 1)Strongest baselineBaseline test MSE ↓ADDP test MSE ↓MSE reduction (Table 1)
PhysioNetAPN0.00406 ± 0.000100.003860 ± 0.0000504.93%
MIMIC-IIIHi-Patch0.0992 ± 0.00140.09419 ± 0.000205.05%
ActivityHyperIMTS0.00253 ± 0.000300.002287 ± 0.0000099.60%
USHCNiTransformer0.5068 ± 0.01290.4954 ± 0.00402.25%

Table 6 in Appendix B.3 already compares four history retention strategies using the same forecasting backbone. Relative to full-history training, random pruning reduces test MSE by 5.46%, and ADDP reduces it by 8.31%.

PhysioNet training strategy (Table 6, five seeds)Test MSE ↓MSE reduction with ADDP
Full history0.004210 ± 0.0000018.31%
Random pruning0.003980 ± 0.0000023.02%
Fixed pruning0.004150 ± 0.0000026.99%
ADDP0.003860 ± 0.000050—

To measure the effect of redundancy directly, we modified only the training histories on USHCN. At each ratio (25%, 50%, and 100%), we added either complete duplicate patches or an equal length of empty padding to the training inputs. Both conditions used clean histories with the corresponding length of empty padding for validation and testing, and the forecasting targets stayed the same.

Duplication ratioBackbone test MSE (padding → duplication)Backbone changeADDP test MSE (padding → duplication)ADDP changeADDP vs. backbone under duplication
25%0.501186 → 0.514930+2.74%0.514218 → 0.514019−0.04%0.18% lower
50%0.496781 → 0.504622+1.58%0.496767 → 0.508893+2.44%0.85% higher
100%0.505352 → 0.513346+1.58%0.496727 → 0.506531+1.97%1.33% lower

Adding duplicate patches only during training increases the backbone's MSE on clean test histories by 1.58%–2.74%. At 25% duplication, the backbone's test MSE increases by 2.74%, while ADDP's test MSE is nearly unchanged relative to its own padding control. At 50% and 100% duplication, test MSE increases for both methods.

What does roughness capture? We separately tested how γ\gamma responds to repeated observations and random noise. We added repeated observations, random noise, or a mixture of the two while retaining the original histories and forecasting targets.

Historical inputMean γ\gammaChange from original history
Original history0.3459—
Add 50% repeated observations0.2491−28.0%
Add 50% random noise0.4720+36.5%
Add 25% repeated observations and 25% random noise0.3987+15.2%

This is consistent with how we designed γ\gamma. Repeating individual observations reduced local variation and lowered γ\gamma, while random noise increased both. Duplicating whole patches preserved the variation within each patch and had little effect on γ\gamma.

Why share the routing signal? Both training views come from the same input history. Which patches to prioritize, how many to keep in both views, and how much weight to give each view’s loss are related decisions. Using the same roughness score γ\gamma (§3.5, Eq. 3) lets ADDP adjust all three together based on how much that history changes. For example, when the history varies more, ADDP gives more weight to variation within patches when scoring them, keeps fewer patches in common between the views, and puts more weight on the view with the higher prediction loss. Table 11 shows that on Activity, using independent routing signals increased test MSE by 2.24% relative to using a shared γ\gamma.

DecisionResponse to higher γ\gammaPaper location
Patch scoringPlace more emphasis on variation energy and observation density§3.6, Eqs. 9–11
Shared coreRetain fewer high-scoring patches in both views§3.7
View weightingLower the loss temperature to give the higher-loss view more weight§3.8

Due to the character limit, please see the remaining questions in the following comment.

Continued from previous comment


Table 11 already compares shared routing, independent routing, and single-component adaptation. On Activity, using independent routing signals increases test MSE by 2.24% relative to shared γ\gamma. Table 12 already shows that a fixed scoring channel, a standard-deviation-based routing index, and separate component-specific indices all yield higher test MSE on PhysioNet.

Dataset and paper locationConfigurationTest MSE ↓MSE increase over the full configuration in the same table
Activity, Table 11Shared γ\gamma across all three components0.002280—
Independent signals0.0023312.24%
Scoring-only adaptation0.0023482.98%
Temperature-only adaptation0.0023563.33%
Shared-core-only adaptation0.0023713.99%
PhysioNet, Table 12Full ADDP0.003860 ± 0.000050—
Single fixed scoring channel0.004740 ± 0.00045022.8%
Standard deviation in place of γ\gamma0.004380 ± 0.00041013.5%
Separate component-specific indices0.00437013.2%

We also evaluated all three two-component combinations on USHCN. Scoring plus temperature yielded the lowest MSE among these combinations. Adding the shared core further reduced MSE from 0.496474 to 0.495416.

New USHCN ablationTest MSE ↓MSE increase over full ADDP
Scoring + shared core0.5314187.27%
Scoring + temperature0.4964740.21%
Shared core + temperature0.5075902.46%
Scoring + shared core + temperature0.495416—

The revised Methodology explains the shared roughness index. Model Analyses summarizes the results, and Coordinating Adaptive Decisions reports the routing and component comparisons. Additional Experimental Details reports the two perturbation experiments.


W2 & Q2 | Where do the gains come from, and are they worth the training cost?

Contribution beyond random augmentation. Within the dual-view framework, adaptive ranking and view weighting each reduce test MSE. Figure 3(a) and Table 9 already report this step-by-step comparison. With mean loss, switching from random views to adaptive ranking reduces MSE from 0.004107 to 0.003950. Fixed-temperature weighting and full ADDP reduce it further to 0.003916 and 0.003860, respectively.

PhysioNet dual-view ablation (Table 9)Test MSE ↓
Random dual views + mean loss0.004107
Adaptive ranking + mean loss0.003950
Adaptive ranking + fixed temperature0.003916
Full ADDP0.003860

Training cost. We measured the following costs using the same PhysioNet inputs, forecasting backbone, batch size, hardware, and numerical precision.

MethodTraining time per step ↓Throughput (original samples/s) ↑Peak GPU memory ↓Inference latency per batch ↓
Full-history backbone45.56 ms495.601.307 GiB6.59 ms
Random single-view pruning59.41 ms370.291.307 GiB7.28 ms
Random dual views60.86 ms371.522.593 GiB6.28 ms
ADDP74.51 ms320.442.594 GiB6.01 ms

Compared with random dual views, ADDP takes 22.4% (13.65 ms) longer per training step, uses 0.001 GiB more peak GPU memory, and reduces test MSE by 6.01%. Compared with the full-history backbone, ADDP takes 63.5% longer per step, uses about twice the peak GPU memory, and reduces test MSE by 8.31% (Table 6). Within dual-view training, we consider a 6.01% reduction in test MSE a reasonable trade-off for an extra 13.65 ms per step with nearly identical peak memory. At inference, ADDP uses the single forward pass over the full history described in Methodology, with measured latency close to that of the full-history backbone. Model Analyses summarizes the accuracy and cost comparison, and Training Cost and Convergence gives the timing table.

Convergence and stability (Q2). The table reports the best PhysioNet validation MSE reached by each epoch. The percentage column uses t-PatchGNN as its reference.

EpochRandom pruning ↓t-PatchGNN ↓ADDP ↓ADDP reduction vs. t-PatchGNN
50.0047970.0045200.0041508.19%
100.0040770.0042580.0038409.82%
150.0039450.0042580.0038409.82%
200.0039010.0041470.0038207.89%
250.0038630.0041470.0037409.81%
300.0038050.0040760.0036909.47%

ADDP reached a validation MSE below 0.003850 by epoch 10. Random pruning first did so at a reported checkpoint at epoch 30, and t-PatchGNN remained above it through epoch 30.


Due to the character limit, please see the remaining questions in the following comment.

Continued from previous comment


Table 1 already reports final test MSE over five random seeds on each of the four datasets, quantifying how much ADDP's final forecasting error varies across runs.

DatasetADDP test MSE (mean ± standard deviation) ↓
PhysioNet0.003860 ± 0.000050
MIMIC-III0.09419 ± 0.00020
Activity0.002287 ± 0.000009
USHCN0.4954 ± 0.0040

Across the four datasets, the standard deviation of ADDP's final test MSE ranges from 0.21% to 1.30% of the corresponding mean. Model Analyses summarizes convergence, Training Cost and Convergence reports the random-pruning comparison and timing measurements, and Main Results reports the five-seed test results.


Thank you again for your constructive and valuable questions. We hope the additional experiments and clarifications address your concerns. If you have any other questions or suggestions, we look forward to continuing the conversation!


Comments on the paper

3 comments

Dear Reviewers TbiJ, w1XB, and tTEx,

The authors have responded to your questions. Please review their responses and consider whether your concerns have been addressed. You may also take a look at the other reviewers' comments and the revised manuscript and share any further feedback.

Best regards,

Your AC

AuthorsOpenReview ↗

Submission 10085 | Summary of Review and Discussion

First, we sincerely thank the AC for promoting the discussion, and the three reviewers for their careful evaluation during the review and rebuttal periods. After the discussion, w1XB and tTEx both raised their scores from 3 (weak accept) to 4 (clear accept), while TbiJ maintained a score of 2 (weak reject).

ReviewerScoreAcknowledged strengthsMain concerns and our responsesDiscussion outcome
TbiJ2The problem is practically meaningful; the results cover multiple datasets and backbones; the paper also compares random pruning and independent routing.To test what roughness can distinguish, whether ADDP can mitigate the impact of duplicated history, and whether the additional training cost is worthwhile, we added controlled heart-rate experiments, duplicated-segment experiments, and equal-training-time comparisons.Acknowledged stronger evidence and clearer costs, and raised three follow-up questions.
w1XB3 → 4Acknowledged the technical contribution of the method, the history-removal diagnostic, and the results from multiple runs.On the Activity dataset and iTransformer backbone used in the original diagnostic, we compared prediction results under the same history retention rates and clarified when ADDP still yields benefits. We also added cost measurements, calibrator training details, and an anonymous code link.Explicitly stated that all concerns had been addressed, raised the score, and continued to support acceptance.
tTEx3 → 4Acknowledged the clear motivation, easy-to-read flowchart, and solid experiments.Clarified which losses update the calibrators under hard sorting and rounding, and reported gradients measured during training. Activity was re-split by recording ID before generating windows, and both methods were retrained on the same split.Confirmed that the main concerns had been addressed, raised the score, and continued to support acceptance.

Main Results Added During the Discussion

  1. Given the same amount of training time, ADDP still predicts more accurately. Under the 50% duplicated-history training setting on Activity, ADDP achieved lower best validation MSE than Backbone, random single-view, and random dual-view within 4, 6, and 9 minutes. Within the common 10-minute cap, each method selected its checkpoint by validation MSE. ADDP's test MSE was 0.002870, 8.37% lower than the strongest baseline, random dual-view.
  2. In the controlled heart-rate experiment, roughness distinguishes sustained changes from transient spikes. We constructed two types of changes with similar magnitudes in PhysioNet: one persisted into the forecasting interval, and the other reverted before forecasting. Using only the heart rate within the event segment to compute γ\gamma, the AUROC for distinguishing the two types of changes was 0.790. With a fixed predictor and a fixed number of retained segments, ADDP's segment selection outperformed random selection on all three seeds, with average forecasting MSE 10.71% and 11.84% lower when retaining two segments and one segment, respectively.
  3. The impact of duplicated history varies by setting. In the 50% duplication training control on Activity, compared with each method's equal-length empty-padding control, ADDP reduced the MSE increase caused by duplication by 36.32%. On USHCN, the effect varied with the duplication ratio. When revisiting 50% duplicated inputs with a frozen model, we found that copies of the same segment occupied retention slots in the aggressive view. This is a limitation of our duplicated-data design; real-world observations are extremely unlikely to contain 50% identical data. After replacing the excess copies with an equal amount of other history segments, the MSE of that view decreased by 3.60%.
  4. Applicable scope of ADDP. On Activity, even when only 20% of the history was retained, ADDP's test MSE remained 22.71% lower than the original iTransformer. When only 10% was retained, the original iTransformer had lower error. This is also intuitive: when the data is already very scarce, further pruning reduces the amount of effective data and thus harms forecasting. We use these results in the introduction to illustrate the applicable scope of the method and provide the full comparison in the appendix.
  5. The calibrator update paths and the Activity data split have been written into the revised manuscript. The two calibrators receive gradients through the margin loss on continuous segment scores and the regularizer on the shared ratio before rounding, respectively, and the training records show that both paths are active. After splitting Activity by recording ID, ADDP's test MSE decreased from 0.0025566 to 0.0023427, a reduction of 8.37%. The revised manuscript reports the full training objective, recording assignment, and window generation procedure.
AuthorsOpenReview ↗

Thank you again

Finally, thank you for taking the time to read our paper and consider our rebuttal!

All authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.