acceptodds
Desk rejected at ICLR 2027

Same Model, Different Interactions: Hidden History in Multi-Task Learning

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Multi-task learning routinely characterizes the interaction between tasks from an instantaneous snapshot of the current model, using quantities such as task losses, gradients, or pairwise gradient alignment. Such snapshots describe how tasks relate at a given moment, but treating them as sufficient to determine how those relations evolve under continued training is an assumption, not a guarantee. We show it fails in the strongest possible sense: for any current model, one can construct distinct optimization states that implement it exactly, are indistinguishable under any model-derived instantaneous representation, yet evolve into different future task–response maps. What is missing is therefore not a better observable but the optimization history itself. That decisive history lives in the optimizer's momentum, which carries a hidden, task-attributed record of past training rather than unstructured noise. As a consequence, the same pair of visible tasks can help each other under one hidden history and harm each other under another. More strongly, two indistinguishable states can call for different next tasks to train: we quantify this with an Aliasing Decision Gap and prove that any policy acting on the snapshot alone incurs an irreducible regret, which we confirm across model families and task suites over many seeds. Task interaction is therefore not an intrinsic property of a task pair but a history-conditioned relation carried behind an identical model and exposed only through history.

Primary area:
optimization
Keywords:
multi-task learning, task interaction, optimization history, momentum, model snapshot, aliasing decision gap

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qzXvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the influence of optimization history on task interactions in multi-task learning (MTL). The authors argue that instantaneous model information, such as task losses and gradients, is insufficient to characterize future task interactions when stateful optimizers are used. By analyzing task-attributed optimizer momentum, the paper demonstrates that different optimization histories can lead to different task interactions and subsequent training decisions, even when the current model parameters are identical. The authors further introduce the Aliasing Decision Gap and establish a theoretical lower bound on the regret of snapshot-only task-selection policies.

Critical strengths
  1. The paper presents an interesting perspective on multi-task learning by investigating the role of historical optimization information, rather than relying solely on instantaneous task gradients or losses.
  2. The analysis is thought-provoking, and the empirical investigations across different models and tasks provide evidence that optimization history can substantially influence task interactions and subsequent training decisions.
Critical weaknesses
  1. The paper primarily focuses on analyzing how historical optimization information influences multi-task learning and demonstrates the limitations of snapshot-based task selection. However, it does not provide concrete strategies or mechanisms for leveraging this information to improve MTL. Consequently, the practical implications of the analysis remain unclear, making the overall contribution feel somewhat incomplete.
  2. The observation that instantaneous model information is insufficient to characterize future optimization dynamics under stateful optimizers is relatively intuitive, given the dependence of their updates on historical states. Although the paper extends this observation to task interactions in MTL, it remains unclear what fundamentally new insights are established beyond this known property of optimization dynamics.
Questions for authors
  1. How can the theoretical findings of this paper be translated into practical guidance for improving multi-task learning?
  2. Could the authors suggest or demonstrate a concrete history-aware MTL strategy motivated by their analysis, such as history-aware task selection or task weighting?
  3. Regarding Theorem 4.1, the result appears rather general: when two indistinguishable states require different optimal actions, a policy restricted to their common observation necessarily incurs regret. What additional insights does this theorem provide specifically for multi-task learning? Can it offer more concrete guidance for the design of effective MTL algorithms?
  4. The current analysis does not appear to explicitly address the influence of task-specific loss scales. In practical MTL, differences in loss magnitudes and gradient scales can significantly affect optimization dynamics and task interactions. How are these factors accounted for in the proposed theoretical framework, and to what extent might they influence the reported conclusions?

In general, I find the perspective of this paper interesting and theoretically thought-provoking. However, my main concern is that the analysis primarily establishes the importance of historical optimization information without explaining how such information can be effectively utilized to improve multi-task learning. Moreover, some of the theoretical conclusions appear relatively intuitive or general, raising questions about the novelty and significance of the contribution. I believe the paper would be substantially strengthened by providing more concrete and constructive insights into how historical information should inform multi-task optimization. In particular, a plausible history-aware strategy or a deeper theoretical characterization of its potential benefits would help establish the practical significance of the findings. I would be open to increasing my score if the authors could provide convincing clarification or supporting evidence addressing these concerns.

Reviewer_DHXXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that the instaneous snapshot of a multi-task model (which is generally adopted schemes for existing multi-task optimization literatures) does not identify how tasks will interact under continued training. The authors claim decisive history lives in the optimizer’s momentum carries a hidden and task-attributed record of past training rather than unstructured noise. Experiments results of Llama and Qwen familes on 15-task P3 and SuperNI, and NYUv2 provide the detailed analysis to support this claim.

Critical strengths
  • While existing multi-task optimization literature largely focuses on how to optimize the model with respect to the task gradient set, this paper directly questions whether these methods identify future dynamics. This is the clear and intersting question. Intervention design is clean and memoryless-SGD control correctly shows the effect requires optimizer state.

  • Comprehensive evaluation results and good breath across setups. Analysis covers three strong LLM backbones, two instruction-tuning task suites, and vision MTL setup, with appropriate confidence intervals and several sensitivity checks.

Critical weaknesses
  • Several claims in the paper are oversold. The analyses basically support the claim that "future dynamics of the model caused by momentum of the optimizer", but phrases such as "fails in the strongest possible sense, prove ... irreducible regret, task interaction is not an intrinsic of a task pairs" are somewhat overstated.

  • Practicality is limited. Standard Multi-task optimization already uses mixed-task batches and typically combines task gradients before the optimizer step, where the optimizer state (momentum) is already used. In standard multi-task optiization with mixed-task batches, where task gradients are combined before one optimizer step, how should past task k wrote this component and the next-task decision be defined, and do the conclusions carry over?

Questions for authors

Can you show a simple history-aware task-selection or weighting rule that improves final multi-task performance over snapshot-based methods?

Reviewer_VyaFOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to show that a model’s current parameters are not sufficient to determine how tasks will interact during subsequent training, because the optimizer state carries training history that the parameters do not capture. To support this claim, the paper constructs pairs of states with identical parameters but different AdamW first moments, and decomposes the first moment by the task that produced each gradient, showing that these states lead to different future task responses. Lastly, it shows that such states may require different next-task choices, proving that any policy relying only on the current model incurs irreducible regret.

Critical strengths

I consider that the question addressed about what state multi-task optimization methods should condition on is relevant.

Critical weaknesses
  • The central finding is expected by construction and is not new. With a stateful optimizer, the training state comprises the parameters and the optimizer moments, so states with identical parameters but different first moments will evolve differently. This is confirmed by the memoryless-SGD control (Fig. 1e). Moreover, the observation that, in MTL with momentum-based optimizers, the optimizer state carries descent directions from previously tasks into subsequent task updates is not new [1]. The paper does not show what is surprising or non-trivial beyond these observations.
  • The practical relevance is not demonstrated. The aliased states are constructed by transplanting the first moment of one trajectory onto the parameters of another, yielding states not encountered in standard training. No history-aware method is proposed or evaluated, so the claim that task selection and weighting should condition on optimizer history remains untested.

References

  1. L. Pascal, P. Michiardi, X. Bost, B. Huet, M. A. Zuluaga. “Improved optimization strategies for deep Multi-Task Networks.” arXiv:2109.11678, 2021.
Questions for authors
  • Can a task-selection or weighting policy that conditions on the optimizer state be shown to improve final multi-task performance over snapshot-based methods?
  • How do the response differences and the decision gap compare with ordinary variability, such as seed-to-seed differences in standard training or differences between existing task-selection policies?
  • Given that momentum is known to carry previous tasks’ descent directions into subsequent updates, what does the paper establish that goes beyond this observation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.