acceptodds
Under review as a conference paper at ICLR 2027

PILOT: Probe-guided Imputation via Local Subspace Optimization for Time-Series Forecasting

Abstract

Missing values can substantially degrade the forecasting performance of time series foundation models (TSFMs), while conventional impute-then-forecast pipelines optimize reconstruction rather than downstream forecasting. End-to-end task-aware training could address this mismatch but requires differentiating through the TSFM, which may be too large or accessible only as a black box. We propose Probe-guided Imputation via Local Subspace Optimization for Time-Series Forecasting (PILOT), a black-box-compatible method that improves imputation using only forward queries to the TSFM. PILOT estimates forecast-loss changes in a local optimization subspace using finite differences on the current batch and a probe sample, and applies a robust correction when the proposed update may conflict with the full-training forecasting objective. We establish conflict-inner-product preservation, second-order finite-difference accuracy, finite-population estimation guarantees, and directional protection under confidence-set coverage. Controlled experiments confirm the predicted conflict boundary and scaling, while experiments on five real-world time series show that PILOT improves test forecast MSE in 77 of 100 paired configurations.

Primary area:
learning on time series and dynamical systems
Keywords:
time series forecasting, missing value imputation, zeroth-gradient estimation, task-aware learning, black-box compatibility

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_imw7OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper looks at impute-then-forecast pipelines where the forecaster is a frozen time-series foundation model (TSFM) that can only be queried. The imputer is still trained on reconstruction, but at every step PILOT uses finite differences, inside a small subspace spanned by the current and recent updates, to estimate whether the update would increase the forecast loss on the training set. If so, it applies a minimally corrected update instead. To make this estimate from part of the data, it samples extra probe windows without replacement and uses a finite-population confidence set. The theory covers the subspace construction, the finite-difference error, the estimator, and a one-step protection guarantee. Experiments include a synthetic quadratic testbed and 100 real configurations (5 univariate series, 5 imputers, 4 TSFMs, one seed), where PILOT lowers test MSE in 77 of 100 cases with a median change of −0.40%.

Critical strengths
  1. The problem proposed by this paper is real and the formulation is reasonable. Steering imputer updates with a frozen, query-only forecaster, and treating the probe windows as a finite population, is a sensible idea, and the synthetic testbed (Fig. 5, Tables 4–9) behaves as the theory predicts. This is the part I would build on. However, the theory says nothing about test performance (Appendix H), so the paper still stands or falls with its real-data results.
Critical weaknesses
  1. The results do not seem to come from the method described in the paper. I checked the released code:

    • Every config sets preserve_norm: true, which rescales the corrected step to the norm of the raw step. This is not described in the paper. It also explains Table 7: the PILOT column is exactly η2/2\eta^2/2 for every η\eta, which is what a norm-preserving orthogonal step gives in this testbed. The minimum-change projection of Eq. (21) would give a smaller value.
    • In src/models/jointModel.py, inside the safe_release branch, gate_conflict = dot.item() < 0.0 runs after the if/else in every case, so the robust release test credited in Table 1b is never used.
    • Sec. 4.1 describes time-ordered train/val/test ranges with training-only standardization, but src/data_loader.py splits overlapping windows at random and keeps only the first 2,000 points of each series. PILOT is also applied for a single continuation epoch.

    Since the experiments are what carry the paper, this is the main reason for my rating.

  2. Even taken as reported, the real-data evidence is thin. The median gain is 0.4%, so many of the 77 wins could be noise. The ablation and the sensitivity scan use a single configuration, the defaults are the best test-MSE setting on every axis of that scan (Table 12), and the six-probe-batch run returns exactly the B1 number (0.45952). There is also no cost comparison (query count or wall-clock) and no comparison with backpropagating through the forecaster, although all four TSFMs are open-weight. Without these, it is hard to see PILOT as a practical alternative to gradient access, which is its stated motivation.

Questions for authors

Clarification questions:

  1. Were all reported results produced with preserve_norm: true, the random window split and the 2,000-point cap that are in the released code? If so, please describe them in the paper and also report results for the method and protocol as written (the minimum-change step of Eq. (21) and time-ordered splits).
  2. Which version of the code produced Tables 1a and 1b? In the released code, gate_conflict = dot.item() < 0.0 overrides the safe-release test, and the gate_mode="mean" path (B5) does not set gate_conflict at all.
6 comments

We thank the reviewer for recognizing the value of our theory and controlled experiments. We address your questions below.

1. Released code, experimental configurations, and norm preservation

Thank you for carefully examining the released code. These issues arose because we did not thoroughly check the experimental configuration files before release, leaving test branches from the experimental design stage mixed with the formal experiment configurations. The safe_release, sequence truncation, and random partitioning of overlapping windows that you identified were not enabled in the real-task main experiments. Their purposes during experimental design were as follows:

  • Random partitioning of overlapping windows: The generated overlapping windows were randomly assigned to the data splits to reduce the distribution shift associated with chronological splitting and examine the effect of this factor in isolation.
  • The safe_release branch: This branch was used to test the utility of the robust release module. The overwritten robust criterion and missing mean-gate branch that you identified occur in this legacy test code.
  • Sequence truncation: Restricting the input to the first 2,000 time points was intended for rapid, small-scale validation of the experimental design.

preserve_norm was indeed enabled in the main experiments. Its purpose was to ensure a fair comparison between the baseline and PILOT: for nonzero corrections that satisfy the numerical threshold, the update norm is restored to that of the current optimizer's original candidate displacement. This controls for the additional variable introduced by changes in step size and isolates the effect of directional correction. The applied update consists of the projection in Eq. (21), followed by norm restoration; the minimum-change property applies to the projection stage before rescaling. Directional protection continues to hold after norm restoration: the robust margin is positively homogeneous, R(αx)=αR(x)R(\alpha x)=\alpha R(x) for α>0\alpha>0, so a projected update satisfying R(x)≥0R(x)\geq0 continues to satisfy the same protection constraint after positive rescaling. The directional protection result therefore holds under the original coverage conditions. We apologize for omitting this implementation detail from the paper and will state it explicitly in the method and experimental settings.

2. Empirical gains and stability

A core contribution of this work is to make conflicts between imputer updates and the forecasting objective measurable using only forward queries to the forecaster, and to construct corrections with a clear geometric interpretation. The controlled experiments provide direct evidence for this mechanism: the behavior in Fig. 5(a) agrees with the predicted 90° first-order conflict boundary; the central-difference error in Fig. 5(c) exhibits second-order scaling, with a fitted slope of 2.00; and, when all remaining windows are queried to eliminate sampling error, the fitted slopes of the loss changes with respect to step size in Fig. 5(d) are 1.04 and 2.00 for the raw and corrected updates, respectively. This is consistent with the correction removing the harmful first-order term and leaving a second-order contribution. Fig. 5(e) examines the correspondence between empirical coverage and nominal levels under the specified finite-population construction.

The value of these results is that the individual components of the mechanism can be tested against known reference quantities, rather than assessed solely through final test MSE. The controlled experiments establish local behavior under a specified construction, while translating that behavior into sustained gains on real tasks remains a further challenge. Real training also involves uncertainty-set calibration, curvature effects at finite step sizes, multi-step optimization trajectories, and differences between training and test distributions.

We acknowledge your concerns about the real-task results. Given the computational cost of the full experimental matrix, we are unable to complete additional systematic experiments within this rebuttal period. In future work, we will consider paired repetitions across multiple random seeds, together with mechanism diagnostics during real training, to investigate the conditions under which the protective behavior observed in controlled experiments translates into stable forecasting gains.

About W2

We fully understand your concerns regarding the end-to-end experimental results. Below, we report a diagnostic experiment that directly examines the update mechanism, providing additional evidence alongside our existing results.

The experiment uses the full Current Velocity dataset, with a randomly initialized SAITS imputer and a Toto2 predictor whose pretrained weights are frozen. We fix the random seed to 42, use an input length of 96 and a prediction horizon of 1, and train with plain SGD (learning rate 0.001, batch size 128) for three epochs, totaling 366 steps.

To control for differences in training trajectories, at each step we evaluate the original imputation-gradient update, the true forecast-gradient half-space projection, and the PILOT update from the same pre-update parameters. The true forecast gradient used by the projection baseline is computed by backpropagation over the entire training set and verified using finite differences. The forecast loss after each candidate update is also evaluated over the entire training set. After these evaluations, only the original imputation-SGD update is actually applied. Thus, the table below compares one-step effects from identical starting points.

The PILOT configuration shown uses a target subspace rank of r=3r=3, four additional probe batches (512 windows), a finite-difference step size of 0.0010.001, and δ=0.3\delta=0.3. During startup, the original experiment uses the effective rank available at that point, subsequently reaching r=3r=3.

Let ΔQ\Delta Q denote the full-training forecast MSE after a candidate update minus the MSE before the update. Negative values indicate improvement.

One-step updateMean ΔQ\Delta Q (×10⁻⁴) ↓Steps with ΔQ>10−8\Delta Q > 10^{-8} ↓Steps with ΔQ>10−6\Delta Q > 10^{-6} ↓
Original imputation-gradient SGD−3.499132/366131/366
True forecast-gradient half-space projection−5.25996/3663/366
PILOT (rank 3, four probe batches)−5.4000/3660/366

These results directly support PILOT's mechanism. The original imputation update increases the forecast loss at 132 steps (36.1%), and all 132 updates also exhibit directional conflict with the true forecast gradient, confirming local conflicts between the imputation and forecasting objectives. At the same 366 starting points, PILOT produces no forecast-loss increase exceeding 10−810^{-8} and achieves a larger average loss reduction than the original update.

This effect cannot be explained by simply halting updates: PILOT intervenes at 154 steps, of which only seven produce zero displacement, while the remaining 147 interventions retain nonzero updates. This supports PILOT's ability to mitigate conflicts with the forecasting objective by correcting update directions.

Overall, this experiment shows that for the real-world dataset and model configuration examined, PILOT can mitigate objective conflicts confirmed by the true gradient, providing direct mechanistic evidence for its local protective effect.

Reviewer_imw7OpenReview ↗

Response to Authors' Rebuttal

Thank you for the response and the new diagnostics. The preserve_norm clarification is helpful, but my main concerns remain and I still tend to rejection.

  1. Released code vs. the response. The response says the shuffled split, the 2,000-point cap and the safe_release branch were not used in the main experiments. However, the README and configs/experiment.yaml (labelled "Main experiment") describe the main pipeline with the shuffled split, the safe_release gate and a single continuation epoch, and src/data_loader.py applies the cap and the shuffled split unconditionally. The configuration and code revision behind the 100-pair results and Table 1b are still not identified.

  2. Method as run vs. method as written. preserve_norm was enabled in the main experiments, and no real-task result is given for the Eq. (21) step described in the paper.

  3. Real-task evidence is unchanged. The new diagnostic measures one-step training loss in a different setting (horizon 1, plain SGD), so it does not speak to the small test-MSE effect (median −0.40%). The six-probe-batch run that equals the B1 reference (0.45952) in Table 12 is also not explained.

Reviewer_ANF4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PILOT, a method that provides task-aware training feedback to an imputer through forward-only queries to a frozen time-series forecaster. It estimates directional derivatives of the forecasting loss in a low-dimensional subspace containing the candidate update and historical applied updates, then uses finite-population probing and robust projection to correct potentially harmful updates. The authors report lower test forecasting MSE in 77 of 100 imputer–forecaster–dataset configurations.

Critical strengths

The problem is practically motivated: training an imputer with downstream feedback when the forecaster exposes only an inference interface is a useful research direction. PILOT preserves the modular deployment pipeline, and including the candidate update in the subspace exactly preserves the inner product relevant to first-order conflict. Conditioning on the current optimization batch before sampling the remaining windows also provides a principled finite-population estimator without requiring independence among overlapping windows. The paper is transparent about important limitations, distinguishing single-step training-objective protection from improved test performance and reporting both degradations and the distribution of gains.

Critical weaknesses
  1. The conditional protection guarantee lacks mechanism-level validation on real tasks. Theorem 4 assumes that the confidence set achieves the stated coverage, but the implementation’s Hotelling-based radius does not guarantee nominal coverage for an arbitrary fixed finite population. Unbiased estimation of the mean and covariance does not resolve this gap. The calibration experiments use only a Gaussian perturbation construction, without reporting empirical coverage, false-release rates, or residual directional conflict after correction during real training. Consequently, improvements in test MSE provide evidence of downstream utility but do not directly validate the proposed protection mechanism.

  2. The experiments do not establish that the correction delivers consistent incremental benefits that justify its cost. All real-task experiments use only seed 42, with a median improvement of 0.40%. The sole ablation configuration achieves a 13.64% improvement, substantially larger than the dataset-wide median improvement of 0.11%. Although B3–B6 provide useful component ablations, the experiments lack simple controls such as skipping conflicting updates, accepting or rejecting updates based on before-and-after forecasting loss, and matching PILOT’s applied update norm. It therefore remains unclear whether the gains mainly result from more conservative updates. Meanwhile, each protected step under the default configuration evaluates six parameter perturbations across multiple batches, yet total training query counts and runtime are not reported. The absence of repeated-seed, query-budget-matched comparisons leaves the method’s consistency, necessity, and computational value unresolved.

Questions for authors
  1. At several fixed training checkpoints on EWELD Load and Current Velocity, how closely does empirical confidence-set coverage match the nominal level under repeated probe draws, with the model state, optimization batch, and candidate subspace held fixed? What are the conflict rate among released updates, the residual directional-conflict rate after correction, and the zero-update frequency? Please use the full-training finite-difference coordinate mean as the reference and separately report actual forecasting-loss changes before and after applying updates.

  2. Under matched query budgets, does PILOT consistently outperform skipping conflicting updates, accepting or rejecting updates based on before-and-after forecasting loss, and matching the applied update norm? Do these findings hold across 3–5 paired random seeds, and do the component conclusions extend to configurations with typical small gains and degradations?

  3. What are the total window-level forecast evaluation counts and training runtimes for PILOT and its comparators? Were the default hyperparameters selected exclusively using validation data, and were consistent checkpoint-selection rules applied to all methods?

5 comments

We thank the reviewer for recognizing the forward-query setting, subspace construction, and finite-population estimator. We address your questions on mechanism calibration, control experiments, and computational cost below.

1. Conditional protection guarantees and mechanism validation on real tasks

Theorem 4 describes single-step directional protection conditional on confidence-set coverage. Finite-population sampling provides conditionally unbiased estimates of the mean and covariance, while the Hotelling-type radius is the calibration choice used in the implementation. Your observation about coverage for an arbitrary fixed finite population is correct: its calibration should be assessed through repeated sampling from the population in question.

Our controlled experiments provide an analytically tractable instance of this validation. In Fig. 5(e), finite populations are repeatedly generated under a Gaussian perturbation construction and probe windows are sampled. Empirical coverage fluctuates around the nominal levels and reaches 1 under full probing.

The fixed-checkpoint diagnostics you propose are the key next step in extending this validation from controlled constructions to real training. We will consider conducting this evaluation on EWELD Load and Current Velocity. Given the cost of querying the full training set and repeating the probe draws, we are unable to complete these additional experiments within this rebuttal period.

2. Simple controls, stability of gains, and the scope of the ablations

Regarding whether the gains mainly come from more conservative updates, the implementation's preserve_norm option rescales nonzero corrections that satisfy the numerical threshold to the norm of the current optimizer's candidate displacement. Its purpose is to control for step-size reduction and examine the effect of changing the update direction.

The control experiments and ablation studies you suggest are valuable, and these baselines were among those we considered including, but omitted because of computational cost. In future research, we will consider systematically evaluating the stability of gains across correction strategies, step-size reduction rules, and probe budgets. Together with mechanism diagnostics during real training, this would help identify when the protective behavior observed in controlled experiments translates into stable forecasting gains.

3. Query costs, hyperparameter selection, and checkpoint-selection rules

We welcome the suggestion to evaluate computational cost. In future work, we will use a consistent accounting procedure to report total window-level forecast evaluations, per-step runtime, and total training runtime for each method, and examine the trade-off between probe budget and forecasting gains.

For checkpoint selection, all compared methods use early stopping with patience=5 and max_epochs=50.

For the default hyperparameters, we did not perform systematic tuning; instead, we selected reasonable initial values based on experience.

About Q2

We thank the reviewer for suggesting additional experiments to further validate PILOT's mechanism and stability. We agree that evaluation across additional random seeds is important. However, our current computational budget did not allow us to complete these repeated experiments during the rebuttal period, and we apologize for this limitation. Below, we report a diagnostic experiment that directly examines the update mechanism, providing additional evidence alongside our existing results.

The experiment uses the full Current Velocity dataset, with a randomly initialized SAITS imputer and a Toto2 predictor whose pretrained weights are frozen. We fix the random seed to 42, use an input length of 96 and a prediction horizon of 1, and train with plain SGD (learning rate 0.001, batch size 128) for three epochs, totaling 366 steps.

To control for differences in training trajectories, at each step we evaluate the original imputation-gradient update, the true forecast-gradient half-space projection, and the PILOT update from the same pre-update parameters. The true forecast gradient used by the projection baseline is computed by backpropagation over the entire training set and verified using finite differences. The forecast loss after each candidate update is also evaluated over the entire training set. After these evaluations, only the original imputation-SGD update is actually applied. Thus, the table below compares one-step effects from identical starting points.

The PILOT configuration shown uses a target subspace rank of r=3r=3, four additional probe batches (512 windows), a finite-difference step size of 0.0010.001, and δ=0.3\delta=0.3. During startup, the original experiment uses the effective rank available at that point, subsequently reaching r=3r=3.

Let ΔQ\Delta Q denote the full-training forecast MSE after a candidate update minus the MSE before the update. Negative values indicate improvement.

One-step updateMean ΔQ\Delta Q (×10⁻⁴) ↓Steps with ΔQ>10−8\Delta Q > 10^{-8} ↓Steps with ΔQ>10−6\Delta Q > 10^{-6} ↓
Original imputation-gradient SGD−3.499132/366131/366
True forecast-gradient half-space projection−5.25996/3663/366
PILOT (rank 3, four probe batches)−5.4000/3660/366

These results directly support PILOT's mechanism. The original imputation update increases the forecast loss at 132 steps (36.1%), and all 132 updates also exhibit directional conflict with the true forecast gradient, confirming local conflicts between the imputation and forecasting objectives. At the same 366 starting points, PILOT produces no forecast-loss increase exceeding 10−810^{-8} and achieves a larger average loss reduction than the original update.

This effect cannot be explained by simply halting updates: PILOT intervenes at 154 steps, of which only seven produce zero displacement, while the remaining 147 interventions retain nonzero updates. This supports PILOT's ability to mitigate conflicts with the forecasting objective by correcting update directions.

Overall, this experiment shows that for the real-world dataset and model configuration examined, PILOT can mitigate objective conflicts confirmed by the true gradient, providing direct mechanistic evidence for its local protective effect.

About Q1

Thank you for suggesting a fixed-checkpoint diagnostic. We have conducted additional experiments on the two datasets you specified, EWELD Load and Current Velocity, directly measuring confidence-set coverage and directional conflicts among released updates and after correction, while separately evaluating actual changes in forecasting loss.

We used SAITS with frozen Toto2 and pre-specified three states—early, middle, and late—along a shared five-epoch Raw-SGD trajectory for each dataset. At each state, we fixed the model parameters, optimization batch, missingness masks, candidate update, and historical subspace. Using the full-training finite-difference coordinate mean as the reference, we performed 1,000 independent probe draws, each sampling without replacement from the remaining windows. The supplementary diagnostic used rank=3, a finite-difference step size of 0.001, a probe-batch ratio of 10%, batch size=256, context/horizon=96/1, and seed=42. Checkpoints were not selected based on conflict labels.

The empirical coverage rates are as follows:

Dataset / training stepNominal 70%Nominal 90%Nominal 95%
EWELD Load / 368.0%88.4%93.0%
EWELD Load / 14070.0%91.9%95.7%
EWELD Load / 28068.8%88.9%94.7%
Current Velocity / 367.4%88.2%94.0%
Current Velocity / 15368.5%89.0%94.8%
Current Velocity / 30570.2%90.6%95.6%

At every state, the absolute difference between empirical coverage and the nominal level was at most 2.6 percentage points.

The middle and late EWELD states contained directional conflicts against the full-training reference, allowing us to directly evaluate conflict handling. At the default nominal coverage level of 70%, all 1,000 probe decisions at each of these two states triggered correction, with no false releases. The residual conflicts after correction were 26/1,000 (2.6%) and 33/1,000 (3.3%), respectively. All corrected updates were nonzero, retaining an average of 97.0% and 78.2% of the original update norm, respectively. At the early EWELD state and all three Current Velocity states, the raw updates were released, with a conflict rate among released updates of 0/1,000 at each state. The zero-update frequency was 0 at all six states.

We also independently applied the candidate updates from the same parameter state for the first five pre-specified probe draws at each state, and computed the actual change in forecasting loss over the full training set, ΔL=Lafter−Lbefore\Delta L=L_{\text{after}}-L_{\text{before}}. The PILOT values below are averages over the five draws:

Dataset / training stepRaw update ΔLPILOT update ΔL
EWELD Load / 3−0.00664808−0.00664808
EWELD Load / 140+0.00027050−0.00000938
EWELD Load / 280+0.00118356−0.00000897
Current Velocity / 3−0.00672169−0.00672169
Current Velocity / 153−0.00074233−0.00074233
Current Velocity / 305−0.00028391−0.00028391

At the two conflicting EWELD states, PILOT changed the average forecasting-loss outcome from an increase to a decrease while retaining nonzero updates. Of the 10 evaluated candidates at these states, nine decreased the loss and one increased it by approximately 1.80×10⁻⁶. At the other four states, the original safe updates were retained and the actual loss decreased. These results directly demonstrate release, conflict correction, and finite-step behavior at the evaluated states.

Reviewer_HtbEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies missing-value imputation for time-series forecasting with frozen time-series foundation models (TSFMs). The authors propose PILOT, a training wrapper that improves the imputer using only forward queries to the frozen forecaster.

Critical strengths

Clean, well-motivated problem formulation with strong internal validation. The black-box, forward-query-only setting is realistic and increasingly relevant (API-only or inference-optimized TSFM deployments), and the mismatch between reconstruction quality and downstream forecasting utility is crisply characterized (Error Propagation eq., Fig. 3a). The controlled experiments reproduce the exact theoretical predictions — the 90° conflict boundary, O(ε²) finite-difference scaling (slope 2.00), first- vs. second-order step-size scalings, and nominal coverage — which gives credible internal validity to the single-step protection mechanism. This makes the paper's central mechanism verifiable rather than merely asserted. Honest, broad, and well-scoped evaluation. The 100 paired configurations across five datasets, five imputers, and four TSFMs, the component ablation isolating each design choice (trajectory-informed basis, robust release, robust projection), the 19-point one-factor sensitivity, and — notably — an unusually candid limitations section (Appendix H, explicitly disclaiming any end-to-end improvement guarantee and documenting optimizer-state mismatch) together show a careful, non-overclaiming empirical discipline. The result that PILOT is "essentially free" for reconstruction quality is a useful negative-cost finding.

Critical weaknesses

The practical effect is small and concentrated, and the statistics are not yet established. The headline is 77/100 wins with mean −1.84% and median −0.40%; the mean is more than four times the median, so a few outlier pairs (e.g., large EWELD Load gains) carry the average, while the typical pair changes by well under half a percent. Current Velocity — one of the five datasets — is close to a coin flip (11/20 wins, per-dataset means near −0.1% median). All real-task numbers come from seed 42 only, with no significance testing or confidence intervals, and the ablation/sensitivity rest on a single configuration. As it stands, the central empirical claim cannot yet be distinguished from sampling noise, which is consequential because the method's value proposition is precisely a measurable downstream gain. Guarantees are single-step and conditional, and the wrapper's cost is never quantified.

Questions for authors

Algorithm 1 (line 9) computes Σ̂_FP,t from the outer product of the probe mean (Ĉ_{P,t}Ĉ_{P,t}^⊤/m_t), which differs from Eq. (15), the finite-population-corrected sample covariance ((M_t/N)²(1−f_t)/m_t · S_{P,t}). Which is implemented in the released code, and does the choice affect the empirical coverage in Fig. 5(e)? What is PILOT's actual added cost per step and over training (number of forecaster forward passes, wall-clock, total) relative to B1? With four 128-window probe batches, the overhead is not negligible; how does the mean −1.84% gain trade against it, and what is the Pareto frontier over probe budget?

4 comments

post comments below your review discussing if your (human-generated) assessment of the paper differs from any AI tools you may have used. The differences between my reviews and those generated by AI include the article ratings(my init rating is 2(weak reject and the author solve my concerns,so I raise my scores to 3)), the actual strengths and weaknesses of the articles, and the authors' questions all of which show clear differences. For this article, if we were to use AI to generate peer review comments directly, the results would be as follows: 表格

项目评分 / 结论
Presentation2(可读懂但密度高,多处笔误与数字不一致)
Informativeness2(有启发性:插补的一阶方向冲突 + 低秩子空间探针,但组件均为成熟思想组合)
Soundness2(可修复问题:单种子实验、无显著性检验、Algorithm 1 与 Eq.15 协方差公式不一致)
Rating3 Weak acceptance(临界,取决于多种子统计与成本 / 收益量化)

两个关键优点: ① 设定新颖且真实(黑盒冻结 TSFM + 仅前向查询),受控实验精确复现了理论预测(90° 冲突边界、ε²/η² 标度、覆盖率),机制验证扎实;② 评测全面(100 组配对、组件消融、敏感性)。

两个关键弱点: ① 实证增益小且集中(中位数 −0.40%,21/100 组变差,均值由少数离群对拉动;Current Velocity 近乎抛硬币 11/20),且全部真实任务只用 seed 42、无显著性检验;② 理论保证仅单步、条件性,且全文未报告 PILOT 相对 B1 的额外查询 / 计算开销,也未对比可微联合训练上界 —— 这直接削弱 "查询高效黑盒" 的核心卖点。

给作者的问题(表单第 7 项)聚焦 5 个实质问题: 6 处小笔误清单(如 "5×5×4=100"、Fig.5 (f) 的 89°/85° 表述、Table 12 重复 ε=0.002 等)。

We thank the reviewer for recognizing the problem formulation, theoretical mechanism, and experimental validation. Your comments on statistical stability, the covariance implementation, and computational cost help us clarify the scope of our findings.

1. Theoretical guarantees and experimental stability

A core contribution of this work is to make conflicts between imputer updates and the forecasting objective measurable using only forward queries to the forecaster, and to construct corrections with a clear geometric interpretation. The controlled experiments provide direct evidence for this mechanism: the behavior in Fig. 5(a) agrees with the predicted 90° first-order conflict boundary; the central-difference error in Fig. 5(c) exhibits second-order scaling, with a fitted slope of 2.00; and, when all remaining windows are queried to eliminate sampling error, the fitted slopes of the loss changes with respect to step size in Fig. 5(d) are 1.04 and 2.00 for the raw and corrected updates, respectively. This is consistent with the correction removing the harmful first-order term and leaving a second-order contribution. Fig. 5(e) examines the correspondence between empirical coverage and nominal levels under the specified finite-population construction.

The value of these results is that the individual components of the mechanism can be tested against known reference quantities, rather than assessed solely through final test MSE. The controlled experiments establish local behavior under a specified construction, while translating that behavior into sustained gains on real tasks remains a further challenge. Real training also involves uncertainty-set calibration, curvature effects at finite step sizes, multi-step optimization trajectories, and differences between training and test distributions.

We acknowledge your concerns about the real-task results. Given the computational cost of the full experimental matrix, we are unable to complete additional systematic experiments within this rebuttal period. In future work, we will consider paired repetitions across multiple random seeds, together with mechanism diagnostics during real training, to investigate the conditions under which the protective behavior observed in controlled experiments translates into stable forecasting gains.

2. The discrepancy between Algorithm 1 and Eq. (15)

Thank you for identifying this discrepancy. We have checked that the released code uses the sample covariance with the finite-population correction defined in Eq. (15), rather than the outer product of the probe mean. Specifically, the implementation centers each probe window's coordinates by subtracting the probe mean and computes

SP,t=1mt−1∑i∈Pt(C^t,i−CˉP,t)(C^t,i−CˉP,t)⊤,S_{P,t}=\frac{1}{m_t-1}\sum_{i\in P_t} (\hat C_{t,i}-\bar C_{P,t})(\hat C_{t,i}-\bar C_{P,t})^\top,

then constructs

Σ^FP,t=(MtN)21−ftmtSP,t,ft=mtMt.\widehat\Sigma_{\mathrm{FP},t} =\left(\frac{M_t}{N}\right)^2\frac{1-f_t}{m_t}S_{P,t}, \qquad f_t=\frac{m_t}{M_t}.

The implementation also adds a small diagonal regularization term for numerical stability.

The released experimental script corresponding to Fig. 5(e) uses the same covariance estimator to compute empirical confidence-set coverage. The expression in Algorithm 1 therefore needs to be corrected to match Eq. (15) and the implementation.

3. Additional computational cost and the benefit–cost trade-off

We agree with this comment. The current evaluation does not systematically quantify the training overhead of additional forward queries or establish the benefit–cost Pareto frontier across probe budgets. It therefore leaves the practical value of PILOT under different computational budgets incompletely characterized.

We will include computational efficiency and budget allocation in future research, considering systematic reporting of additional forward-query counts, per-step runtime, and total training runtime, alongside the trade-off between probe budget and forecasting gains. This will help establish where PILOT is useful under different deployment constraints.

Thank you for your detailed response. Taking into account the overall innovation, the practicality of the conclusions, and the thoroughness of the experiments, I have raised my score to 3 points.

Thank you for your thoughtful reassessment and constructive feedback. We are glad that our response helped address your concerns. We will incorporate these clarifications into the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.