acceptodds
Withdrawn from ICLR 2027

Residual Aggregation as Nonlinear Scalarization: Exact Gradient Reallocation and Shared-Encoder Transmission

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Block-structured objectives can encode the same residual evidence yet allocate gradient mass differently depending on how block losses are scalarized. Let block residuals be and losses . We compare linear scalarization with weighted power scalarization , where . We derive an exact pairwise log-odds law showing that native-to-composed allocation distortion separates into relative weight mismatch and times the log-residual ratio. The law yields matching conditions, exact worst-case distortion and inverse rigidity, exponent identification, and state sensitivity. Power-mean and Jacobian-conditioned Gram analyses connect the same scalarization change to candidate rankings and shared-parameter updates. Common-state audits on Energy, Parkinsons, and CelebA-40 quantify the predicted reallocation and its transmission through shared encoders. In paired 200-epoch regression runs, native consistently lowers task-1 error, whereas composed lowers task-2 and worst-task error on both datasets. Macro-average ordering differs across datasets, and the Parkinsons contrast largely attenuates under validation-selected stopping. Together, theory and experiments provide a quantitative account of when scalarization geometry produces persistent, task-selective optimization effects.

Primary area:
optimization
Keywords:
Optimization, Loss Functions, Gradient Dynamics, Gradient Allocation, Loss Aggregation, Multi-Task Learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_VoD7OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes how linear and weighted-power scalarizations redistribute gradients over identical block residuals. It derives an exact log-odds allocation law, matching conditions, ranking-reversal criteria, and a Jacobian Gram identity for shared-encoder transmission. Experiments on Energy, Parkinsons, and CelebA-40 examine these mechanisms and downstream task trade-offs.

Critical strengths
  1. The mathematical analysis is precise. The log-odds law separates weight mismatch from exponent mismatch and residual heterogeneity, providing a clear characterization of gradient reallocation and equivalence conditions.

  2. The empirical reporting is transparent. The authors distinguish numerical consistency checks from independent evidence and acknowledge task trade-offs and checkpoint-selection sensitivity rather than claiming universal improvements.

Critical weaknesses
  1. The incremental contribution and downstream significance remain uncertain. Several results follow directly from scalarization derivatives or the chain rule. Without scale-matched retraining or simple task-reweighting controls, the independent contribution of relative gradient allocation to training outcomes remains unclear.

  2. The downstream evidence is limited. Training endpoints rely mainly on two regression datasets with fixed splits and ten seeds, while CelebA-40 provides common-state diagnostics rather than endpoint comparisons. On Parkinsons, validation selection usually chooses epoch 0, weakening the fixed-budget findings.

Questions for authors
  1. Do the downstream task trade-offs persist against scale-matched objectives and simpler linear task-reweighting baselines?

  2. Can the authors demonstrate that the observed endpoint effects remain robust across additional data splits or training budgets?

Reviewer_V8jeOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper compares two methods of combining multiple task losses into a single objective in multi-objective training: additive residual aggregation and the nonlinear/weighted-power/direct-sum scalarization. They show mathematically when the two methods are equivalent, or how they change the gradient allocation of “blocks”. Small experiments validate their mathematical derivations and show that these differences can affect training and task performance, but neither approach seems consistently better.

Critical strengths

Theoretically understanding how different ways of aggregating task losses affect optimization in multi-task learning is an interesting research question.

Critical weaknesses

The clarity of the paper is the main concern. It is possible I missed the point or misunderstood the contributions because of this.

  • There are many undefined symbols and concepts in the abstract alone. Things like “residual evidence,” “allocation distortion,” and “inverse rigidity” are introduced without clear explanations. It’s unclear to me what the research question is after reading the abstract.
  • Equation (1) is again unclear. What exactly constitutes a block? The authors list sensors, tasks, modalities, etc., but no concrete learning problem illustrating the objective and residuals. What are a and b? What is an additive rule, and what is a Euclidean direct sum rule? And why does each prefer a different residual? What is a residual head? This seems to be a central concept mentioned throughout the paper yet never defined. And why are they considered “identical”? Equation 2 is again unclear; symbols are either undefined, or the definition is unclear. It’s also unclear whether the author propose these two methods or does it exist, what they means and how they are used, etc.
  • The naming of the two methods is inconsistent throughout the paper. It seems to me that the nonlinear scalarization, direct sum organization, and weighted power scalarization all refer to the same thing, for example.
  • Figure 1 is unclear. The terms and formulas are repeated, but no intuitive explanation of the mechanism is provided. I don’t really see the point of including this figure.
  • Throughout Section 1, I still don’t understand what problem it is attempting to address and what the impact is. Why is it important to study these particular scalarizations? What is the surprising and novel part? Different loss aggregations having different gradients and optima seems expected. And what practical insight follows?
  • The main message of the theory, as far as I can tell, is that different the exponent of the aggregate residuals changes their relative gradient weighting and can change candidate rankings. Most of the theory seems to be using algebraic properties or equivalence conditions of the two objective functions; I’m not sure if there are more insights.
  • The experiments are mostly verification of the above theory claims, which are more so directly from the definitions. Again I’m not sure whether there are any surprising or significant results. The only more practical experiment training with the two objectives seem inconclusive and rather small scaled.
Questions for authors

See weakness.

Reviewer_dyhJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies how linear and weighted-power aggregation of block residuals affect gradient allocation, candidate rankings, and shared-parameter updates. It derives exact equivalence conditions, distortion bounds, inverse-identification results, and sensitivity estimates. Experiments examine gradient transmission and downstream task trade-offs.

Critical strengths

The paper provides a precise characterization of aggregation-induced gradient reallocation, including useful exact bounds and inverse-identification conditions. Common-state comparisons and paired training experiments are carefully designed. The authors clearly distinguish mathematical identities from empirical findings and appropriately qualify their conclusions.

Critical weaknesses

The analysis is careful, and I recognize the value of the exact bounds and inverse-identification results. However, the overall contribution appears incremental: it primarily refines our understanding of implicit task weighting within a fixed power family, while the experiments document state-dependent gradient effects and task trade-offs. I found these results useful, but the additional insight is not substantial enough to support acceptance. I have no major correctness concerns.

Questions for authors

No specific questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.