acceptodds
Withdrawn from ICLR 2027

Not All Variables Agree: Reliability-Aware Variable-Wise Gradient Surgery for Multivariate Time-Series Forecasting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

In data-driven training, multivariate time-series forecasting is usually optimized with a scalar loss averaged over samples, variables, and horizons. This averaging is convenient, but the optimizer sees only the aggregated gradient, which does not reveal whether the variable-wise contributions align or oppose one another. To quantify how often this disagreement arises, we measure the variable-wise gradients directly and find that 30.6% of their pairwise cosine similarities are negative on average across seven datasets. However, conflict and harm are not the same thing. Under shared training 35 of the 64 variables do worse than a full-input single-target oracle, and the harmed fraction is not reliably predicted by how often gradients conflict. We propose Per-Variable Surgery (PV-Surgery), an optimizer-side training strategy for backbones with cache-compatible layers. One backward pass builds variable-wise gradient proxies from output-side signals and keeps the pointwise forecasting loss. Reliability-aware selection targets layers whose proxy sums closely approximate their shared-gradient slices. Conditional pooling forms anchor and conflict pools without dropping variables. Common-direction surgery aligns variable or pooled gradients with their normalized mean and restores input norms to avoid reweighting. In experiments across five backbones, seven datasets, and four horizons, PV-Surgery lowers MSE by 3.61% and MAE by 2.93% on average. For multivariate forecasting, this indicates that the variable-wise structure hidden by mean-loss training is a usable optimization signal.

Primary area:
learning on time series and dynamical systems
Keywords:
Multivariate time-series forecasting, Gradient surgery, Gradient conflict, Multi-task optimization, Optimizer-side training strategy

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iFWoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PV-Surgery, a training method for multivariate time-series forecasting. It estimates variable-wise gradients in a single backward pass, selects layers, conditionally groups gradients, corrects their directions, and restores the norms of the surgery inputs. The paper also distinguishes gradient conflict from actual performance degradation under shared training. Across five backbones, seven datasets, and four forecasting horizons, it reports average relative reductions of 3.61% in MSE and 2.93% in MAE compared with standard MSE training.

Critical strengths

1. Useful distinction between gradient conflict and harmful sharing.\textbf{1. Useful distinction between gradient conflict and harmful sharing.} The paper goes beyond reporting negative gradient cosines by comparing shared training with full-input single-target models and examining different partner compositions. These diagnostics provide useful evidence that conflict frequency alone is insufficient to determine whether sharing helps or hurts prediction. This meaningfully advances understanding of variable-wise optimization in forecasting.

2. Broad empirical evaluation and direct proxy validation. \textbf{2. Broad empirical evaluation and direct proxy validation. } The method is evaluated across 140 backbone–dataset–horizon settings, with additional baselines and component ablations. The paper also directly compares gradient proxies with exact per-variable gradients, providing useful evidence about proxy quality beyond forecasting performance alone. These experiments help assess the method’s effectiveness and limitations across the evaluated settings.

Critical weaknesses

1. Agreement of the gradient sum does not necessarily establish the accuracy of individual proxies.\textbf{1. Agreement of the gradient sum does not necessarily establish the accuracy of individual proxies.} Equation 6 checks whether the variable-wise proxies sum to the reference gradient. However, individual proxy errors can cancel, producing an accurate sum despite inaccurate variable-wise estimates. Figure 5 and Appendix J provide useful evidence of higher fidelity after layer selection, but selection also depends on output-layer inclusion and gradient norms. It therefore remains unclear whether the reconstruction-error score itself helps identify more accurate proxies.

2. The additional benefit of directional correction needs clearer isolation and repeated-seed evidence. \textbf{2. The additional benefit of directional correction needs clearer isolation and repeated-seed evidence. } Beyond directional correction, the method replaces gradients with proxy sums and sets current data gradients to zero on unsupported hooked blocks. The output-cache ablation removes several components together, so it does not separate these effects from directional correction. Appendix I provides a useful comparison using the same proxy construction, layer-selection rules, and gradient assembly, but pooling, candidate selection, and aggregation scale also differ across methods. Moreover, the main results and ablations use only seed 42, and the gradient-method comparison is limited to MICN. These limitations make it difficult to assess the additional gains and smaller ablation differences with confidence.

Questions for authors

1. Does the reliability score identify accurate individual proxies? \textbf{1. Does the reliability score identify accurate individual proxies? } With complete and consistently partitioned linear-layer caches, Equation 4 appears to recover the summed reference gradient despite downstream variable mixing. What causes nonzero reconstruction error on supported layers in practice? Does this error predict per-variable proxy fidelity beyond output-layer inclusion and gradient-norm-based selection?

2. How much does directional correction contribute? \textbf{2. How much does directional correction contribute? } Can you provide a control that keeps proxy construction and gradient assembly unchanged while turning off directional correction, i.e., setting ΔS=0\Delta_{\mathcal S}=0? How often does the assembly rule materially change unselected gradients or zero otherwise nonzero reference gradients? Does the output-cache ablation retain sum-loss training?

3. Are the additional gains stable across seeds? \textbf{3. Are the additional gains stable across seeds? } Can you provide paired multi-seed results for PV-Surgery, standard MSE training, MGDA, and GradVac on representative MICN settings? These would help assess whether the additional improvements and small ablation differences are consistent across runs.

Reviewer_yVCzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

In multivariate time-series forecasting, the gradient step is averaged across variables. This can generate conflicts where gradients point to negatively correlated directions. However, such conflicts do not necessarily harm the accuracy of the forecasting. In this work, authors propose a gradient surgery technique to discern between these two cases and improve the gradient effectiveness.

Critical strengths

-Gradient surgery is a relevant topic in multivariate time series forecasting which is itself a cumbersome topic where prediction accuracy is still a challenge.

-The authors recognise the difference between gradient conflict and prediction harm and diagnose it from data.

Critical weaknesses

-The proposed method lacks motivation. It is mainly a composition of engineered gradient modifications lacking theoretical or empirical basis. This is crucial, the method is a chaining of unrelated engineering techniques that happen to lower the MAE by some mild margin.

-Given the specificity of the solution, the results that follow are insuficiently compelling. Mainly, a 3% MAE improvement over vanilla gradient averaging is not convincing. These empirical results are less encouraging when considered along with the comparison to other gradient surgery methods where the proposed method does not show any clear advantage.

-Authors report dense, redundant information. This redundancy of metrics and data organization hinders the lack of significant advantage in empirical results.

Questions for authors

The process here aims to make surgery in the gradient by chaining very specific techniques. Could you provide a motivation on the ideas behind each surgery step?

Reviewer_rawROpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies variable-wise gradient conflict and harm from shared forecasting training using exact gradients, full-input single-target references, and partner-group experiments. It finds that conflict and average alignment do not reliably identify harmed variables or helpful partners. PV-Surgery constructs gradient proxies from one sum-loss backward pass, selects a parameter subspace, pools compatible rows, applies common-direction surgery, and assembles the optimizer gradient. Across five backbones, seven datasets, and four horizons, it reports average relative MSE/MAE reductions of 3.61%/2.93%, alongside ablations, operator comparisons, and mechanism diagnostics.

Critical strengths
  1. The diagnosis provides a useful conceptual contribution. The paper goes beyond counting conflicts, uses target-specific references and repeated partner-group experiments, and reports negative findings. This helps the field formulate the shared-training problem more accurately.

  2. The framework makes variable-level optimization usable across backbones. The single-backward proxy pipeline shows practical potential across different architectures, with exact-gradient fidelity measurements, alternative operators, and a complete cost analysis. This breadth and transparency make it a useful foundation for further work.

Critical weaknesses
  1. Although conflict/alignment do not reliably identify harmful sharing, intervention still relies on proxy cosines, common directions, and violation counts. Dynamic pooling and input-norm restoration do not explain why useful sharing is preserved; common-direction aggregation also introduces unit-gradient balancing. For example, even with exact gradients 10,-1,-1, the selector favors −12 over the original sum of 8 because it harms fewer objectives, although this increases total loss to first order. The paper acknowledges residual per-variable violations, but the relationship to average MSE and the “safe/anchored” characterization remains unclear.

  2. An accurate proxy sum can conceal inaccurate individual directions, and after variable mixing an internal variable-axis contribution need not belong to the corresponding output objective. The fidelity measurements support usable proxies in some subspaces, but do not establish reconstruction error as a reliable indicator of that attribution. Assembly additionally replaces all hooked gradients and writes zeros on unsupported blocks, introducing separate update-suppression effects. These gaps limit the interpretation of the gains as reliable variable-wise coordination and the methodological advance over grouping/balancing approaches such as MTLinear.

Questions for authors
  1. Is candidate selection intended to optimize the number of improving variables, fairness, or average forecasting error? Does the implementation contain an additional total-gradient safeguard relevant to the example above? Please clarify the precise meaning of “safe” and “anchored.”

  2. After matching sum/mean loss, clipping, and aggregation scale, how do simple unit-gradient aggregation, MTLinear's error-balancing component, and existing operators compare? What happens when surgery is disabled while proxy assembly remains active?

  3. Main results and operator comparisons use seed 42. Could authors provide matched-seed differences for representative large-gain, small-gain, and regression settings, especially the PV-MGDA/GradVac margin and MICN-Exchange-720?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.