acceptodds
Under review as a conference paper at ICLR 2027

Gradient Toxicity: When Auxiliary Heads Collapse the Shared Encoder

Abstract

Multi-taskoptimizationevaluateshowtaskssharegradients,buttaskcompatibility doesnotdeterminewhetherthesharedrepresentationsurvives.Weintroducegra dient toxicityasaseconddiagnosticaxis: thesignedcontributionofanauxiliary updatetorepresentation-variancecontraction. Anexactvarianceidentityshows whyconflictinformationalonecannotcertifysurvival,andtenmulti-taskoptimiz erscollapsein10/10seedsatthetestedhighdoseswhiledetachmentcollapsesin none.Becausetheaxisismeasuredbyacheapfrozenprobe, thevalidityofthat probemustbeestablishedratherthanassumed.Wederiveacomputablemargin conditionunderwhichtheprobeagreeswiththeinstantaneousvariancederiva tive,anddecomposeitsfailureintorestorationmisalignment,encodergeometry, andmismatchbetween theprobeand theoperatinggradient. Underonecon trolledtoxifier,predicteddosestrackmeasuredscattercrossingsonbatterydata butoverestimatethemby30–116×onRadioMLandby3.9–32.6×onNYUv2, andchangingonlythecalibrationtimeswitchesaHARfilterfrom10/10to0/10 collapses.Realauxiliaryheadsseparateunsignedradialloadingfromsignedcon traction,andrepresentationsurvival fromtaskutility: onNYUv2,depthRMSE improvesfrom1.33to1.09astherepresentationcontracts.Underacorrectedcell disjointbatteryprotocol,detachmentreducesstate-of-healthRMSEfrom5.57to 2.10percentagepoints. Selectivefilteringretainsausefulauxiliaryeffectwhere detachmentremovesit,butnotateverydose.Theseresultsestablishrepresenta tionsurvivalasaseparateevaluationrequirementandprobevalidityasaquantity toaudit.

Primary area:
optimization
Keywords:
multi-task learning, representation collapse, gradient conflict, phase transition, spectral filtering, stop-gradient, self-supervised learning, time-series, battery health prediction, capacity law

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_32JYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The submission argues that successful representation learning lies core of successful multi-task learning (MTL), but that prior work concentrated on the aspect if how MTL aligh with each other or now, while the more important question is whether the resulting representation benefit or not, for which they take the representation variance as a proxy and suggests two additional measures. However, since these terms are not formally defined, and the assumption on e.g. the architecture are not specified, it is hard to judge the merits and correctness of the contribution.

The submission reports on experiments on four rather small-scale MTL setups, where they demonstrate that the new measures reflect different phenomena and can support a filtering mechanism to prevent representation collapse. However, this section as well lacks too much important information to allow proper assessment.

Critical strengths
  • The submission revisits a common (but rarely justified) assumption about multi-task learning.
  • The submission tries to formally derives their expression and measure from first principles.
Critical weaknesses
  • The manuscript writing is too unclear for publication. It relies on technical background and assumptions about specific MTL architectures and methods without stating them, and it argues about concepts without defining them. The writing style resembles a project report more than a scientific paper, concentrating on a proposed tool and reporting experimental findings rather than putting the work into context of prior work and demonstrating advantages.

  • The impact of the contribution is not made clear. The submission puts forward a measure and suggests this should be reported when evaluating MTL, but the practical benefits this would have remain unclear.

Questions for authors

I do not have any questions that could affect my assessment. The manuscript would need to be completely rewritten, starting with a description of the studied setups, definition of the core quantities and quantitative success criteria. Afterwards, it could undergo a new round of reviews. This is not possible in the form of an author response, so please do not try.

Reviewer_DKG5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper contests that conflict in task gradients is not enough to capture the negative effects of auxiliary tasks in multi-task learning (MTL) since conflict-free gradients may still cause the shrinkage of shared representations. The authors formalize this distinction by a batch-variance functional and employ the concept of gradient toxicity for representation contraction. They provide theoretical results that demonstrate that the task-gradient Gram matrix is not able to tell whether representation scatter will increase or decrease, and offer a frozen-probe diagnostic with a sufficient margin condition that can be used to predict the direction of change. Further theoretical analysis of simplified scalar and linear models.

The empirical study is conducted in six application domains: human activity recognition, RadioML, battery state-of-health estimation, NYUv2, CelebA, and financial ranking. The proposed MTL optimization methods are compared to each other within the context of a designed variance-shrinking auxiliary objective, and frequent representation collapse at high auxiliary weights is reported. They also test the proposed diagnostic at training checkpoints, and report large estimation errors, high sensitivity to experimental conditions, and poor alignment with the true representation dynamics. Lastly, they explore the possibilities of improvement of selected settings by detachment of gradients and a spectral toxicity filter (STF), with an admission of the limitations and the negative outcomes. The paper includes code and comprehensive experimental records. Its main contribution is a distinction between gradient conflict and representation contraction, and an empirical study of the reliability and problems of a proposed diagnostic.

Critical strengths

[S1] This paper shows an important limitation of conventional multi-task learning diagnostics i.e non-conflicting task gradients do not guarantee that auxiliary training preserves useful shared representations. This distinction is relevant to understanding negative effects of auxiliary tasks and provides a useful perspective for evaluating MTL methods beyond task-level performance.

[S2] The authors also evaluate their proposed diagnostic across multiple domains and report important limitations, including sensitivity to warm-up duration, substantial estimation errors, and weak agreement with the actual representation dynamics. The zero-dose controls and additional intervention experiments make the empirical investigation more informative. I particularly appreciate that the authors report results that challenge their proposed method rather than presenting only favorable outcomes. This strengthens the credibility of the empirical analysis, even though the findings also expose significant limitations in the paper’s main methodological claims.

Critical weaknesses

[W1] The experiments are not good evidence that this is actually removing useful information from the representations. The paper relies on the standard deviation of feature representations to measure collapse, but this can be reduced even if the features are just made smaller. This is not always due to the loss of useful information by the model. The authors' own experiments reveal that the scatter of the features in the data set can be dramatically reduced, and some tasks still perform as well or better. Furthermore, detachment will always meet the collapse criterion, since the encoder is trained according to the single task training setup, used as the reference. If no other assessments are done, like linear probing on standardized features or scale-invariant ones, it is hard to draw a conclusion that the reported collapse indicates true loss of the representational quality. The results of several of the MTL methods are also disappointing, adding to doubts regarding the reliability of the comparisons.

[W2] The proposed diagnostic is inadequate to back the claims of the paper. The authors' own experiments yield an agreement with the actual direction of representation change of approximately 51% at the test checkpoints, near chance. In addition, the proposed certificate seems to have computations that would be necessary to make to measure the actual change directly, making it unclear what benefit this certificate would do in practice. Additional theoretical results are based on simplified models of scalar and linear networks, and have only limited relevance to the behavior of actual neural networks. Overall, while interesting, there is a lack of diagnostic clarity and compelling evidence that the intervention described is clearly superior to simpler alternatives in reducing the gradient conflict or in bringing about representation contraction.

Questions for authors

[Q1] Could the authors explain how the auxiliary weight alpha is handled by each of the ten MTL methods? In particular, do methods such as MGDA, IMTL-G, Aligned-MTL, Nash-MTL, and FAMO still show collapse when alpha is very small? I am also concerned that several methods achieve almost chance-level accuracy on RadioML and HAR. Were these results checked for training instability, implementation issues, or insufficient hyperparameter tuning?

[Q2] Can the authors provide additional results using measures that are less sensitive to feature scale, such as effective rank or participation ratio? Linear-probe accuracy on standardized frozen representations would also be helpful. These experiments could clarify whether the observed collapse actually means that the model loses useful information or whether the features are simply becoming smaller.

[Q3] I am not sure what practical advantage the proposed certificate offers over directly measuring the change in representation scatter. The certificate seems to require computations similar to those needed to obtain the actual quantity. Can the authors explain how the certificate can be used efficiently without having to compute these quantities?

[Q4] In the corrected battery experiments, the jointly trained model has higher covariance trace and participation rank than the single-task model, even though detachment improves SOH RMSE from 5.57 to 2.10. How do the authors interpret this result as evidence of representation collapse? Could the improvement instead be explained by negative transfer from the auxiliary tasks?

[Q5] Do the authors have an example where the measured collapse causes a loss of useful information that is not reflected in the primary-task performance? For example, does it affect transfer to a new task or the performance of a new classifier trained on the learned representations? Such evidence would help explain why representation scatter should be evaluated separately from task performance.

Reviewer_c15GOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies a structural blind spot in the standard task-gradient conflict diagnosis used in multi-task learning (e.g., PCGrad, CAGrad): low—or even improving—task losses do not guarantee the survival of the shared encoder's representation. The authors propose gradient toxicity as a second diagnostic axis, formalizing the signed contribution of the auxiliary update to representation covariance contraction through the exact identity V̇ = r − αc, and prove that conflict information (the Gram matrix) cannot determine the sign of this change; empirically, under the controlled variance-shrinkage toxifier, ten mainstream optimizers collapse in 10/10 seeds at tested high doses across three domains, while detachment collapses in none. Because practical decisions rely on a cheap frozen probe (the radial projection norm ratio), the paper further derives a computable local validity boundary (Theorem 1) that decomposes the probe's error into restoration misalignment, encoder geometry, and probe mismatch, and exposes its systematic failures across domains: predictions agree on battery data but overestimate the crossing by 30–116× on RadioML, changing only the calibration time flips HAR outcomes, and on HAR, the zero-dose control shows that the probe statistic itself can drift by orders of magnitude. As an intervention test under the controlled high-dose toxifier, STF reduces collapse from 10/10 to 0/10 while simultaneously demonstrating that representation survival and task utility must be evaluated separately—the rank filter clears the scatter threshold yet performs worse than detachment on task metrics. The end product is an audit protocol (Algorithm 1) requiring three questions to be answered distinctly: does the update contract the representation, is the probe trustworthy, and does the resulting representation support the task?

Critical strengths
  • S1. The paper establishes a precise structural limitation of task-gradient conflict diagnostics. Through the exact variance identity V̇ = r − αc and Proposition 1, it shows that identical task-gradient Gram information can correspond to opposite changes in representation variance, and demonstrates the practical scale of this gap with controlled experiments: ten mainstream multi-task optimizers collapse in 10/10 seeds at tested high doses while detachment collapses in none. This formalizes representation survival as an evaluation axis separate from task conflict and shows why task-gradient conflict information alone is insufficient to certify representation survival.

  • S2. Beyond proposing a diagnosis, the paper systematically audits the diagnostic tool itself. Theorem 1 provides a computable sufficient condition under which the cheap frozen probe agrees in sign with the exact variance derivative, decomposing the discrepancy into restoration misalignment, encoder geometry, and probe mismatch. The empirical characterization of its failure modes is systematic and non-obvious: with the same controlled toxifier, probe error varies by domain (accurate on battery data vs. 30–116× overestimation on RadioML) and by calibration time alone (flipping HAR outcomes from 10/10 collapses to 0/10), while the HAR zero-dose control shows that the probe statistic itself can drift by orders of magnitude. The resulting three-way evaluation structure—representation survival, probe validity, and task utility—together with Algorithm 1, carries explicit validity boundaries rather than a universal threshold; the negative result that rank-based filtering clears the scatter criterion yet underperforms detachment on task metrics directly establishes that survival is not task utility.

Critical weaknesses
  • W1. The general theory does not justify the full operational STF rule. The identity dV/dt=r−αcdV/dt = r - \alpha c and Theorem 1 are general local results, but STF-cal uses a narrower rule: it interprets the unsigned probe ρ\rho through the one-shot threshold αρ=1\alpha \rho = 1, while the rank rule has theorem-level support only for a specified PSD operator. The paper itself notes that empirical PCA bases require separate identification and that no tested implementation has a universal trajectory guarantee from the projected norm ratio alone. Thus the paper supports a useful audit framework and a special-case intervention, but not STF as a generally reliable diagnostic and intervention method.

  • W2. The real-head evidence is too narrow to support broad deployment claims. RadioML SNR regression provides a genuine real-head case: at α=3000\alpha = 3000, joint training collapses in 10/10 seeds, while STF-cal and the rank-3 arm reduce this to 1/10 and 0/10, respectively. However, the other tested real heads do not collapse at their reported doses, and the successful case uses one auxiliary head at an extreme dose whose relation to natural loss scales is not established. In the fully labeled protocol, rank-3 filtering also preserves scatter while underperforming detachment on task accuracy. The evidence therefore supports a narrow toxicity case, but not the broader deployment value of STF.

Questions for authors
  • Q1. Scope of the operational STF rule. The paper presents dV/dt=r−αcdV/dt = r - \alpha c and Theorem 1 as general local results, while the operational STF-cal procedure uses the unsigned probe ρ\rho, the one-shot threshold αρ=1\alpha \rho = 1, and a PCA-based rank rule. Which assumptions connect these operational choices to the general identities? In particular, is the αρ=1\alpha \rho = 1 gate intended as a theorem-backed certificate only under the specified operator assumptions, or as an empirical heuristic for arbitrary auxiliary heads? This distinction materially affects how broadly the STF claims should be interpreted.

  • Q2. Interpretation and generality of the real SNR result. Table 3 and Figure 3 provide a genuine real-head example: RadioML SNR regression collapses under joint training at α=3000\alpha = 3000, while STF reduces or eliminates collapse depending on the arm. How should this dose be interpreted relative to the natural scale of the two losses, and what evidence separates gradient toxicity from a task-specific loss-scaling or optimization failure? Since the other tested real auxiliary heads do not collapse at their reported doses, what supports extending the SNR result to auxiliary heads more generally?

  • Q3. Validity of one-shot calibration. The deployed controller freezes its gate and rank after a short warm-up probe. What existing analysis in the paper shows that the calibrated ρ\rho predicts the later representation dynamics? In my HAR runs, the calibration margin stayed below one while the post-training ρ\rho changed by several orders of magnitude. Is the one-shot rule claimed as a deployment-level certificate or only as a local heuristic? This distinction materially changes whether STF supports the paper’s practical claim.

Comments on the paper

1 comment
Area_Chair_3m2LOpenReview ↗

Reviewer–Author Discussion Period Ending Soon

Dear Reviewers,

With the discussion period coming to a close, I encourage you to revisit the authors’ responses and let us know if you have any further comments.

Thank you for your careful reviews and engagement in the discussion!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.