acceptodds
Under review as a conference paper at ICLR 2027

Beyond Gradient Conflict: Multi-Task Landscape-based Optimization via Geometric Independent-task Confluence

Abstract

Multi-task learning (MTL) is largely dominated by gradient-based methods that reconcile conflicting objectives through a single shared update direction. However, under strong task conflict, this shared-trajectory constraint often leads to unstable optimization and degraded performance. We introduce **LOGIC**, **L**andscape-based **O**ptimization via **G**eometric **I**ndependent-task **C**onfluence, a conceptual shift in MTL that replaces a single shared gradient compromise with independently optimized task trajectories that are later coordinated in parameter space. Rather than enforcing agreement at every optimization step, LOGIC allows each task to evolve along its own loss landscape before geometrically aligning task-adapted solutions through a curvature-weighted centroid, with periodic re-anchoring of the task trajectories. By decoupling task-specific optimization from cross-task coordination, LOGIC stabilizes training and enables consistent positive transfer under strong task conflict. Across four benchmarks, three backbones, and both vision and language tasks, LOGIC consistently outperforms thirteen strong gradient-based baselines and frequently surpasses single-task learning, while exhibiting substantially smoother and more stable convergence dynamics.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multi Task Learning, Gradient Conflict, Computer Vision, Advanced Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_TWL1OpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LOGIC, a multi-task learning optimization method that moves beyond per-step gradient conflict resolution. Its main contributions are two-fold. First, a shift from gradient-space aggregation to parameter-space coordination: task conflict is handled by aligning independently optimized task trajectories rather than directly reconciling conflicting gradients. Second, LOGIC computes a curvature-weighted centroid of task-specific parameter copies, using relative Hessian-trace estimates to give more influence to locally stable task trajectories.

Critical strengths

1.The paper reframes multi-task optimization from finding a shared gradient compromise at each step to coordinating independently optimized task trajectories in parameter space, which is a meaningful shift in perspective. 2. The method leverages the geometry of task-specific parameter solutions through a curvature-weighted centroid rather than relying only on local gradient directions.

Critical weaknesses
  1. The baseline coverage is not fully convincing. The paper compares against many standard gradient-based MTL methods, but the baseline set still feels incomplete. In particular, it lacks the latest 2025/2026 gradient-based MTL baselines, and it does not sufficiently cover methods that are closer in spirit to LOGIC’s core design, such as copy-based, trajectory-based, or model-merging-style approaches.
  2. The evidence for the copy-based trajectory mechanism is still limited. Since a central claim of LOGIC is to maintain task-specific parameter copies and coordinate their trajectories in parameter space, the paper should include stronger comparisons with methods that also use parameter copies or local task trajectories. The current ablations show that synchronization and curvature weighting matter, but they do not fully rule out the possibility that much of the gain comes from the multi-copy training structure itself.
  3. The theoretical and Hessian-based analysis feels relatively thin. The method uses Hessian trace as a curvature proxy, but the analysis does not go much beyond this scalar estimate. There is limited discussion of Hessian spectrum, top eigenvalues, conditioning, negative curvature, trace-estimation noise, or the empirical relationship between curvature weights and final performance. The theory also does not appear to provide a rigorous convergence guarantee, so the main support for the method remains largely empirical.
Questions for authors

Please refer to weakness also. It includes but not limited to questions like:

  1. Could you include more recent and complete gradient-based MTL baselines, especially methods from 2025 and 2026?
  2. Could you compare LOGIC with stronger copy-based or trajectory-coordination baselines?
  3. Can you provide deeper evidence that the Hessian-trace weighting is the main driver of the gains?
3 comments
AuthorsOpenReview ↗

Response to Reviewer TWL1 - ** Part 1**

We thank the reviewer for the supportive, constructive feedback, which greatly strengthened the paper. We addressed every comment and added evaluations beyond those requested:

(1) Positioning. LOGIC optimizes each task independently and synchronizes the resulting solutions geometrically in parameter space. To our knowledge, it is also the first MTL method to decouple shared-encoder learning from synchronization of the task-specific copies. The shared encoder moves toward a curvature-weighted centroid at every step, while the copies are synchronized on a progressively less frequent schedule. This keeps task heads coupled to an up-to-date encoder while giving task-specific copies more room to explore between synchronization steps, with bounded drift.

(2) Baselines more than doubled (13 → 31), spanning gradient-based, trajectory-based, task-grouping and model-merging methods, and the latest 2025–2026 methods (Rep-MTL, SAMO, ConsMTL, LDC-MTL, CORE-MTL, ATM). LOGIC ranks first in Δm% and mean rank.

(3) A 40-task stress test. On the challenging large-scale CelebA benchmark (App. A.9), LOGIC achieves the best Δm% (2.36) and mean rank (3.90): a 20% smaller gap to single-task performance than the strongest competitor (SAMO-FairGrad, 2.95), with the lowest cross-seed std (±0.11 vs. ≥±0.28).

Changes are marked in red in the revised PDF; we respond to each point below.

W1. Recent baselines.

We thank the reviewer for raising this concern. We added a significant number of recent multi-task literature to Sec. 2 and to Tables 1–2, including Rep-MTL (ICCV 2025), SAMO (ICCV 2025), ConsMTL (CVPR 2025), LDC-MTL (ECCV 2026), CORE-MTL (ICML 2026) and ATM (ICIAP 2025). All baselines were rerun with their official code and default hyperparameters under our protocol (ResNet-50, HPS, 200 epochs, three seeds, identical STL reference). This matters because Δm% is measured against each protocol's own single-task baseline. DiLoCo and ForkMerge, which were not designed for MTL, were adapted to the multi-task setting. Under the same protocol, LOGIC achieves the best Δm% on both NYUv2 and Cityscapes: −2.78 and −3.40, respectively.

The table below summarizes Δm% (↓, relative to STL) for the added baselines:

MethodΔm% NYUv2Δm% Cityscapes
SAMO-FairGrad−1.56−3.16
Rep-MTL−0.07−1.32
Aligned-MTL0.39−2.06
ConsMTL−0.24−2.59
CORE-MTL−0.71−2.67
LDC-MTL1.29−1.48
Sequential-Reptile3.04−0.73
Auto-λ2.36−2.73
ATM−0.09−0.57
DiLoCo3.672.83
LOGIC (Ours)−2.78−3.40

As requested, we also evaluated LOGIC against four model-merging methods: TIES-Merging (2023), AdaMerging (2024), SIMO (2025) and SyMerge (2026) (Table 11, Appendix). Consistent with the literature, merging methods generally trail standard MTL training, although recent methods such as SIMO narrow the gap.

W2. Comparison with stronger copy- and trajectory-based baselines.

The table in our response to W1 includes several copy- and trajectory-based methods, as well as model-merging approaches and recent 2025–2026 baselines. We also compared LOGIC with model-merging, auxiliary-task-learning and task-grouping methods (Table 11, Appendix). The table below summarizes Δm% for these additional methods: The table below summarizes Δm% (↓, relative to STL) for these additional methods on NYUv2:

MethodΔm% (NYUv2)
ForkMerge1.72
TIES-Merging46.03
AdaMerging26.31
SIMO10.03
SyMerge25.09
TAG2.95
ETG1.94
LOGIC (Ours)−2.78

W3. Evidence for the copy-based mechanism.

Is the gain just the multi-copy structure? Our ablation study separates the effect of the copies themselves from the effect of the coordination/synchronization mechanism, and shows that the copies alone do not produce the gain. Training per-task copies without re-anchoring, using equal-weight averaging, reaches Δm% = +29.52, far worse than joint training (Δm% = +2.32). The opposite control, averaging without per-task copies (Lookahead on joint training, App. A.4), also fails (Δm% = +2.59). Only when the copies are properly coordinated, through re-anchoring and the curvature-based centroid, does every configuration surpass single-task learning (Δm% from −2.09 to −2.78). The gain therefore comes from how the copies are coordinated, which is LOGIC's core contribution.

The paper also characterizes the opposite limit. By Prop. 2, synchronizing at every step (T = 1) reduces LOGIC exactly to weighted gradient aggregation, the family to which the gradient-based baselines belong (e.g., SAMO-FairGrad, the strongest of them, reaches Δm% = −1.56). LOGIC's synchronization interval thus spans a range between two limits: no coordination, where copies drift freely (Δm% = +29.52), and no task-specific trajectory (T = 1), where LOGIC becomes a weighted-aggregation method - same family as the baselines.

AuthorsOpenReview ↗

Response to Reviewer TWL1 - ** Part 2**

W4. Hessian analysis and theory.

(i) Why the trace, and why Hutchinson. Computing the full Hessian spectrum is infeasible for a ResNet-50 encoder with millions of parameters, and it would have to be repeated throughout training; top-eigenvalue estimates are possible but, like the trace, are scalar summaries. Hutchinson's estimator gives an unbiased estimate of the trace from Hessian-vector products alone, without ever forming the Hessian, at the cost of M extra backward passes per refresh. Refreshed every Δ_curv steps, this cost is negligible (App. A.5). App. A.5 also discusses the trace's main limitation, cancellation between eigenvalues of opposite sign, and alternatives that avoid it (top eigenvalue, Frobenius norm, Gauss-Newton). We therefore use the trace only as a relative signal: each task is compared with its own recent history, so a systematic cancellation shared with that history is absorbed by the baseline. The trace serves as a stability signal, not as a flatness certificate.

(ii) Trace-estimation noise. LOGIC is robust to estimator noise at three levels. First, both the per-task baseline and the weights are updated by exponential moving averages, which suppress the noise of the Hutchinson estimator (App. A.2.2). Second, the theory bounds its effect: Prop. 5 shows that the noise in the update direction decays as σ²/T, and Prop. 6 bounds how far the weights can move regardless of the estimated values (at most 0.14 over 1000 steps). Third, changing the number of Hutchinson probes (M ∈ {8, 16, 32}) changes mIoU by only about 0.2.

(iii) Weights and performance. On both benchmarks the weights move from uniform toward depth, the relatively flatter task (ĉ ≈ 0.43 vs. 1/3 on NYUv2; 0.58 vs. 1/2 on Cityscapes; App. A.2.2), and on NYUv2 depth is where LOGIC's largest gains over the baselines appear (Sec. 4.3). Freezing the weights at their converged values gives only Δm% = −0.10 (vs. −2.78 for full LOGIC), so the gain comes from how the weights evolve, not from fixed coefficients.

(iv) Theory and convergence. We agree; App. A.1 ("What we do not claim") states these limits: we prove no convergence or optimality result. The coupled, non-linear dynamics of K task copies, a moving anchor and evolving weights make a convergence proof very hard, and MTL has no canonical scalar objective to converge on. Our theory instead characterizes what the update computes: weighted aggregation at T = 1 (Prop. 2), plus a second-order term D that no single-trajectory method with nonnegative task weights can produce (Props. 3–4). To assess its size, App. A.2.5 measures D directly with Hessian-vector products: it exceeds the first-order term within 50 epochs, reaching a final ratio near 7 on NYUv2 and 70 on Cityscapes, so the update departs substantially from weighted aggregation. As App. A.2.5 notes, reading D as a second-order term is reliable only early in training; later, these ratios measure deviation from aggregation. Convergence guarantees remain future work.

Is Hessian-trace weighting the main driver?

Not on its own. The foundation is the coordination of per-task trajectories: with uniform weights, LOGIC already reaches Δm% = −2.09, better than every baseline in Table 1. Curvature weighting is a significant second component. (i) When copies are never re-anchored, it reduces Δm% from +29.52 to +8.71, so the weights identify which copies to trust. (ii) With synchronization, it improves Δm% from −2.09 to −2.78, and each component contributes (absolute curvature alone: −2.45; Table 4). The gain is positive on all five seeds (paired t-test, p ≤ 0.005). (iii) The benefit comes from how the weights evolve, not from fixed coefficients (W4 (iii)).

We hope the new clarifications resolve the reviewers' concerns. We are happy to discuss any remaining questions.

Thanks for your detailed reply. I will maintain my positive rating of the paper.

Reviewer_fYe1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LOGIC, a multi-task optimization framework that does not aggregate task gradients at a single shared point in parameter space. It keeps one copy of the shared encoder per task, and each copy is trained independently on its own task loss with its own optimizer. The authors claim this paradigm encourages the model steering toward robust, better-generalized regions. Experiments cover classical multi-task optimization benchmark, including NYUv2, Cityscapes, and PASCAL-Context, and demonstrating lowest multi-seed standard deviation, and numerically stable training curves

Critical strengths
  • Straightforward and efficient. strong headline results.

  • The authors honestly state what is not theoretically claimed, such as convergence or optimality analysis.

Critical weaknesses
  • The main claim of the paper is curvature-based geometric coordination, but the ablation table shows most gains come from periodic averaging of independently trained task copies. Specifically, in table 4, UWA with synchroinzation already reaches performance that beats every baseline in table 1. In contrast curvature weighting adds only small gains. This evidence supports that the core contribution is DiLoCo-style periodic model averaging applied across tasks.
Questions for authors

Given that uniform averaging with synchronization already reaches −2.09, is the 0.69 gain from curvature weighting statistically significant across seeds? Could you report more seeds or a paired test?

2 comments
AuthorsOpenReview ↗

Response to Reviewer fYe1

We thank the reviewer for the supportive, constructive feedback, which greatly strengthened the paper. We addressed every comment and added evaluations beyond those requested:

(1) Positioning. LOGIC optimizes each task independently and synchronizes the resulting solutions geometrically in parameter space. To our knowledge, it is also the first MTL method to decouple shared-encoder learning from synchronization of the task-specific copies. The shared encoder moves toward a curvature-weighted centroid at every step, while the copies are synchronized on a progressively less frequent schedule. This keeps task heads coupled to an up-to-date encoder while giving task-specific copies more room to explore between synchronization steps, with bounded drift.

(2) Baselines more than doubled (13 → 31), spanning gradient-based, trajectory-based, task-grouping and model-merging methods, and the latest 2025–2026 methods (Rep-MTL, SAMO, ConsMTL, LDC-MTL, CORE-MTL, ATM). LOGIC ranks first in Δm% and mean rank.

(3) A 40-task stress test. On the challenging large-scale CelebA benchmark (App. A.9), LOGIC achieves the best Δm% (2.36) and mean rank (3.90): a 20% smaller gap to single-task performance than the strongest competitor (SAMO-FairGrad, 2.95), with the lowest cross-seed std.

Changes are marked in red in the revised PDF.

W1. Most gain comes from synchronized averaging; curvature weighting adds little.

(i) Per-task coordination is the core contribution, and the ablation confirms it. With uniform weights, coordinating independent task trajectories in parameter space already reaches Δm% = −2.09 ± 0.35 on NYUv2. That outperforms every competing method in Table 1 (best competitor: SAMO-FairGrad, Δm% = −1.56 ± 0.69). The gain does not come from slow-weight averaging. Lookahead on joint training averages slow weights on a single shared model and gives Δm% = +2.59, worse than joint training (Δm% = +2.32). The gain appears only when each task has its own trajectory. We have rewritten the contributions to lead with per-task trajectory coordination. Curvature weighting is now presented as a second component, with its claim sized to the statistics in point (iii).

(ii) Relation to DiLoCo. We agree that the two methods are related: both train copies separately and periodically average them. They differ in when the shared model moves. DiLoCo couples two events. Every T steps it averages the copies, takes one outer step on the shared model, and resets the copies to it; between these rounds the shared model stays fixed. LOGIC decouples them. At every iteration, the shared encoder moves toward the centroid of the per-task copies while the copies keep training on their own loss landscapes. Re-anchoring (i.e., synchronization), which resets the copies to the shared encoder, runs on its own, growing schedule. The shared encoder therefore follows the centroid's whole trajectory instead of jumping to its endpoint once per synchronization round.

This matters in our setting because only the encoder is replicated, while the task heads are trained by backpropagation through the shared encoder. Under DiLoCo's schedule, the heads would train for a whole round on a frozen, outdated encoder and then face a sudden jump in features; under LOGIC, they always train on the current shared encoder. To test whether this difference matters in practice, we added a DiLoCo-across-tasks baseline: one encoder copy per task, uniform averaging, a shared-encoder update only at synchronization, and a reset after each round. It reaches Δm% = +3.67 on NYUv2, against −2.09 for LOGIC with uniform weights and −2.78 for full LOGIC. Since both DiLoCo and uniform LOGIC use uniform weights, this difference isolates the effect of LOGIC's decoupled update schedule.

(iii) Statistical significance. We agree that three seeds are not enough to establish that the gain from curvature weighting (Δm% = −2.09 → −2.78) is significant. For the revision, we ran five seeds of both uniform weighting and full LOGIC. A paired t-test over the five seeds shows that the gain is statistically significant (p ≤ 0.005);

(iv) What the curvature weights contribute. Two existing controls suggest the weights carry real information. When copies are never re-anchored, curvature weighting reduces Δm% from +29.52 to +8.71, which suggests the weights identify which copies to trust. Freezing the weights at their converged values gives Δm% = −0.10, so the gain does not come from a fixed set of coefficients.

Q1. Statistical significance of the 0.69 gain. Following (iii) - the results are in the table below -

Uniform + syncLOGIC (full)
Δm% (mean ± std, 5 seeds)−2.10 ± 0.34−2.79 ± 0.32

Paired gain: 0.69; 95% CI of the gain [0.45, 0.93]. LOGIC is better in 5/5 seeds (paired t-test, t(4) = 7.98, p ≤ 0.005).

We hope the new experiments and clarifications resolve the reviewer's concerns. We are happy to discuss any questions.

Thank you to the authors for the additional experiments. The authors' response has resolved most of my concerns. I found no major flaws in this paper, and therefore I have maintained my final rating to 3.

Reviewer_19DJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an MTL method based on geometry-aware trajectories, which trains weights separately for multiple subtasks and then aggregates them. The proposed method utilizes more memory and achieves good performance on several datasets.

Critical strengths

The proposed method allows for a relatively long synchronization interval during the optimization process of each task.

Critical weaknesses

There is an overstatement of its contribution. The authors mention that "LOGIC reverses the merge-optimize order of existing MTL methods." However, meta-learning-based MTL methods also allow each task (or set of subtasks) to update independently in the inner loop, and then aggregate shared weights in the outer loop, such as Sequential Reptile (ICLR 2022). Efficient Task Grouping (TPAMI 2025) also optimizes each task first and then integrates them. Furthermore, although the authors mention that federated learning methods have different objectives, they also involve separate multi-step optimization followed by aggregation, and are worth comparing, such as TA-FMTL (MM Asia 2025).

The synchronization interval needs to be complete. The ablation study includes multi-task data and delta m%.

There are absolutely no references after 2025. For example, Rep-MTL (ICCV 2025), SAMO (ICCV 2025), ...

Questions for authors

Please address weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 19DJ

We thank the reviewer for the supportive, constructive feedback, which greatly strengthened the paper. We addressed every comment and added evaluations beyond those requested:

(1) Positioning. Our LOGIC optimizes each task independently and synchronizes the resulting solutions geometrically in parameter space. To our knowledge, it is also the first MTL method to decouple shared-encoder learning from synchronization of the task-specific copies. The shared encoder moves toward a curvature-weighted centroid at every step, while the copies are synchronized on a progressively less frequent schedule. This keeps task heads coupled to an up-to-date encoder while giving task-specific copies more room to explore between synchronization steps, with bounded drift.

(2) Baselines more than doubled (13 → 31), spanning gradient-based, trajectory-based, task-grouping and model-merging methods, and the latest 2025–2026 methods (Rep-MTL, SAMO, ConsMTL, LDC-MTL, CORE-MTL, ATM). LOGIC ranks first in Δm% and mean rank.

(3) A 40-task stress test. On the challenging large-scale CelebA benchmark (App. A.9), LOGIC achieves the best Δm% (2.36) and mean rank (3.90): a 20% smaller gap to single-task performance than the strongest competitor (SAMO-FairGrad, 2.95), with the lowest cross-seed std (±0.11 vs. ≥±0.28).

Changes are marked in red in the revised PDF; we respond to each point below.

W1. Overstated contribution.

We removed "reverses the merge-optimize order": optimizing tasks separately before aggregating is not new by itself. To make LOGIC's contribution precise:

LOGIC's novelty lies in how the separated tasks are coordinated. None of the cited methods coordinates them the way LOGIC does. TA-FMTL separates clients, not tasks, and still merges task losses at every local step. Efficient Task Grouping decides which tasks to train together, not how to train them, so LOGIC could serve as its optimizer. Sequential Reptile runs one trajectory through batches from all tasks, so the tasks are mixed rather than separated.

LOGIC differs from current copy-based/trajectory-based approaches in two ways. First, current methods update the shared encoder weights only at the outer step, and the same step coordinates the copies. LOGIC decouples shared-encoder learning from copy synchronization. The encoder moves toward the copies' curvature-weighted centroid at every step, while the copies re-synchronize on an increasingly spaced schedule. The task heads therefore always train on an up-to-date encoder, and the copies can explore longer while synchronization keeps their drift bounded. Second, LOGIC weights the copies by relative geometry-aware curvature instead of averaging them uniformly. Our ablation study and appendix show that per-task coordination, not slow-weight averaging, drives the gain.

We added these clarifications to the Contributions and Related Work sections and compared LOGIC with all works requested by the reviewer (and additional ones).

W2. Synchronization interval.

We added the requested details to Appendix A.5. We tested a full grid of 15 schedules: minimum intervals of 1, 2, 3, 4 and 5 epochs, each combined with maximum intervals of 10, 15 and 20 epochs. Across all 15 configurations, mIoU changed by less than 0.5, depth AbsErr by less than 0.01, normal mean error by less than 0.3, and Δm% by less than 0.2. LOGIC is therefore robust to the choice of synchronization interval.

W3. References after 2025.

We thank the reviewer for raising this concern. We added a significant number of recent multi-task literature to Sec. 2 and to Tables 1–2, including Rep-MTL (ICCV 2025), SAMO (ICCV 2025), ConsMTL (CVPR 2025), LDC-MTL (ECCV 2026), CORE-MTL (ICML 2026) and ATM (ICIAP 2025). All baselines were rerun with their official code and default hyperparameters under our protocol (ResNet-50, HPS, 200 epochs, three seeds, identical STL reference). This matters because Δm% is measured against each protocol's own single-task baseline. Under the same protocol, LOGIC achieves the best Δm% on both NYUv2 and Cityscapes: −2.78 and −3.40, respectively.

The table below summarizes Δm% (↓, relative to STL) for the added baselines:

MethodNYUv2Cityscapes
SAMO-FairGrad−1.56−3.16
Rep-MTL−0.07−1.32
Aligned-MTL0.39−2.06
ConsMTL−0.24−2.59
CORE-MTL−0.71−2.67
LDC-MTL1.29−1.48
Sequential-Reptile3.04−0.73
Auto-λ2.36−2.73
ATM−0.09−0.57
DiLoCo3.672.83
LOGIC (Ours)−2.78−3.40

As requested, we also tested TAG (2021), ETG (2025) and TA-FMTL (2025). On NYUv2, TAG reaches a Δm% of 2.95 and ETG 1.94 (Table 11). We added model merging techniques as well (Table 11). On Cityscapes, TA-FMTL reaches −0.31 (Table 2).

We hope the new experiments and clarifications resolve the reviewer's concerns. We are happy to discuss any remaining questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.