- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper looks at impute-then-forecast pipelines where the forecaster is a frozen time-series foundation model (TSFM) that can only be queried. The imputer is still trained on reconstruction, but at every step PILOT uses finite differences, inside a small subspace spanned by the current and recent updates, to estimate whether the update would increase the forecast loss on the training set. If so, it applies a minimally corrected update instead. To make this estimate from part of the data, it samples extra probe windows without replacement and uses a finite-population confidence set. The theory covers the subspace construction, the finite-difference error, the estimator, and a one-step protection guarantee. Experiments include a synthetic quadratic testbed and 100 real configurations (5 univariate series, 5 imputers, 4 TSFMs, one seed), where PILOT lowers test MSE in 77 of 100 cases with a median change of −0.40%.
Critical strengths
- The problem proposed by this paper is real and the formulation is reasonable. Steering imputer updates with a frozen, query-only forecaster, and treating the probe windows as a finite population, is a sensible idea, and the synthetic testbed (Fig. 5, Tables 4–9) behaves as the theory predicts. This is the part I would build on. However, the theory says nothing about test performance (Appendix H), so the paper still stands or falls with its real-data results.
Critical weaknesses
-
The results do not seem to come from the method described in the paper. I checked the released code:
- Every config sets
preserve_norm: true, which rescales the corrected step to the norm of the raw step. This is not described in the paper. It also explains Table 7: the PILOT column is exactly for every , which is what a norm-preserving orthogonal step gives in this testbed. The minimum-change projection of Eq. (21) would give a smaller value. - In
src/models/jointModel.py, inside thesafe_releasebranch,gate_conflict = dot.item() < 0.0runs after the if/else in every case, so the robust release test credited in Table 1b is never used. - Sec. 4.1 describes time-ordered train/val/test ranges with training-only standardization, but
src/data_loader.pysplits overlapping windows at random and keeps only the first 2,000 points of each series. PILOT is also applied for a single continuation epoch.
Since the experiments are what carry the paper, this is the main reason for my rating.
- Every config sets
-
Even taken as reported, the real-data evidence is thin. The median gain is 0.4%, so many of the 77 wins could be noise. The ablation and the sensitivity scan use a single configuration, the defaults are the best test-MSE setting on every axis of that scan (Table 12), and the six-probe-batch run returns exactly the B1 number (0.45952). There is also no cost comparison (query count or wall-clock) and no comparison with backpropagating through the forecaster, although all four TSFMs are open-weight. Without these, it is hard to see PILOT as a practical alternative to gradient access, which is its stated motivation.
Questions for authors
Clarification questions:
- Were all reported results produced with
preserve_norm: true, the random window split and the 2,000-point cap that are in the released code? If so, please describe them in the paper and also report results for the method and protocol as written (the minimum-change step of Eq. (21) and time-ordered splits). - Which version of the code produced Tables 1a and 1b? In the released code,
gate_conflict = dot.item() < 0.0overrides the safe-release test, and thegate_mode="mean"path (B5) does not setgate_conflictat all.
6 comments
About W2
We fully understand your concerns regarding the end-to-end experimental results. Below, we report a diagnostic experiment that directly examines the update mechanism, providing additional evidence alongside our existing results.
The experiment uses the full Current Velocity dataset, with a randomly initialized SAITS imputer and a Toto2 predictor whose pretrained weights are frozen. We fix the random seed to 42, use an input length of 96 and a prediction horizon of 1, and train with plain SGD (learning rate 0.001, batch size 128) for three epochs, totaling 366 steps.
To control for differences in training trajectories, at each step we evaluate the original imputation-gradient update, the true forecast-gradient half-space projection, and the PILOT update from the same pre-update parameters. The true forecast gradient used by the projection baseline is computed by backpropagation over the entire training set and verified using finite differences. The forecast loss after each candidate update is also evaluated over the entire training set. After these evaluations, only the original imputation-SGD update is actually applied. Thus, the table below compares one-step effects from identical starting points.
The PILOT configuration shown uses a target subspace rank of , four additional probe batches (512 windows), a finite-difference step size of , and . During startup, the original experiment uses the effective rank available at that point, subsequently reaching .
Let denote the full-training forecast MSE after a candidate update minus the MSE before the update. Negative values indicate improvement.
| One-step update | Mean (×10⁻⁴) ↓ | Steps with ↓ | Steps with ↓ |
|---|---|---|---|
| Original imputation-gradient SGD | −3.499 | 132/366 | 131/366 |
| True forecast-gradient half-space projection | −5.259 | 96/366 | 3/366 |
| PILOT (rank 3, four probe batches) | −5.400 | 0/366 | 0/366 |
These results directly support PILOT's mechanism. The original imputation update increases the forecast loss at 132 steps (36.1%), and all 132 updates also exhibit directional conflict with the true forecast gradient, confirming local conflicts between the imputation and forecasting objectives. At the same 366 starting points, PILOT produces no forecast-loss increase exceeding and achieves a larger average loss reduction than the original update.
This effect cannot be explained by simply halting updates: PILOT intervenes at 154 steps, of which only seven produce zero displacement, while the remaining 147 interventions retain nonzero updates. This supports PILOT's ability to mitigate conflicts with the forecasting objective by correcting update directions.
Overall, this experiment shows that for the real-world dataset and model configuration examined, PILOT can mitigate objective conflicts confirmed by the true gradient, providing direct mechanistic evidence for its local protective effect.
Response to Authors' Rebuttal
Thank you for the response and the new diagnostics. The preserve_norm clarification is helpful, but my main concerns remain and I still tend to rejection.
-
Released code vs. the response. The response says the shuffled split, the 2,000-point cap and the
safe_releasebranch were not used in the main experiments. However, the README andconfigs/experiment.yaml(labelled "Main experiment") describe the main pipeline with the shuffled split, thesafe_releasegate and a single continuation epoch, andsrc/data_loader.pyapplies the cap and the shuffled split unconditionally. The configuration and code revision behind the 100-pair results and Table 1b are still not identified. -
Method as run vs. method as written.
preserve_normwas enabled in the main experiments, and no real-task result is given for the Eq. (21) step described in the paper. -
Real-task evidence is unchanged. The new diagnostic measures one-step training loss in a different setting (horizon 1, plain SGD), so it does not speak to the small test-MSE effect (median −0.40%). The six-probe-batch run that equals the B1 reference (0.45952) in Table 12 is also not explained.
We thank the reviewer for recognizing the value of our theory and controlled experiments. We address your questions below.
1. Released code, experimental configurations, and norm preservation
Thank you for carefully examining the released code. These issues arose because we did not thoroughly check the experimental configuration files before release, leaving test branches from the experimental design stage mixed with the formal experiment configurations. The
safe_release, sequence truncation, and random partitioning of overlapping windows that you identified were not enabled in the real-task main experiments. Their purposes during experimental design were as follows:safe_releasebranch: This branch was used to test the utility of the robust release module. The overwritten robust criterion and missing mean-gate branch that you identified occur in this legacy test code.preserve_normwas indeed enabled in the main experiments. Its purpose was to ensure a fair comparison between the baseline and PILOT: for nonzero corrections that satisfy the numerical threshold, the update norm is restored to that of the current optimizer's original candidate displacement. This controls for the additional variable introduced by changes in step size and isolates the effect of directional correction. The applied update consists of the projection in Eq. (21), followed by norm restoration; the minimum-change property applies to the projection stage before rescaling. Directional protection continues to hold after norm restoration: the robust margin is positively homogeneous, R(αx)=αR(x) for α>0, so a projected update satisfying R(x)≥0 continues to satisfy the same protection constraint after positive rescaling. The directional protection result therefore holds under the original coverage conditions. We apologize for omitting this implementation detail from the paper and will state it explicitly in the method and experimental settings.2. Empirical gains and stability
A core contribution of this work is to make conflicts between imputer updates and the forecasting objective measurable using only forward queries to the forecaster, and to construct corrections with a clear geometric interpretation. The controlled experiments provide direct evidence for this mechanism: the behavior in Fig. 5(a) agrees with the predicted 90° first-order conflict boundary; the central-difference error in Fig. 5(c) exhibits second-order scaling, with a fitted slope of 2.00; and, when all remaining windows are queried to eliminate sampling error, the fitted slopes of the loss changes with respect to step size in Fig. 5(d) are 1.04 and 2.00 for the raw and corrected updates, respectively. This is consistent with the correction removing the harmful first-order term and leaving a second-order contribution. Fig. 5(e) examines the correspondence between empirical coverage and nominal levels under the specified finite-population construction.
The value of these results is that the individual components of the mechanism can be tested against known reference quantities, rather than assessed solely through final test MSE. The controlled experiments establish local behavior under a specified construction, while translating that behavior into sustained gains on real tasks remains a further challenge. Real training also involves uncertainty-set calibration, curvature effects at finite step sizes, multi-step optimization trajectories, and differences between training and test distributions.
We acknowledge your concerns about the real-task results. Given the computational cost of the full experimental matrix, we are unable to complete additional systematic experiments within this rebuttal period. In future work, we will consider paired repetitions across multiple random seeds, together with mechanism diagnostics during real training, to investigate the conditions under which the protective behavior observed in controlled experiments translates into stable forecasting gains.