- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes LOGIC, a multi-task learning optimization method that moves beyond per-step gradient conflict resolution. Its main contributions are two-fold. First, a shift from gradient-space aggregation to parameter-space coordination: task conflict is handled by aligning independently optimized task trajectories rather than directly reconciling conflicting gradients. Second, LOGIC computes a curvature-weighted centroid of task-specific parameter copies, using relative Hessian-trace estimates to give more influence to locally stable task trajectories.
Critical strengths
1.The paper reframes multi-task optimization from finding a shared gradient compromise at each step to coordinating independently optimized task trajectories in parameter space, which is a meaningful shift in perspective. 2. The method leverages the geometry of task-specific parameter solutions through a curvature-weighted centroid rather than relying only on local gradient directions.
Critical weaknesses
- The baseline coverage is not fully convincing. The paper compares against many standard gradient-based MTL methods, but the baseline set still feels incomplete. In particular, it lacks the latest 2025/2026 gradient-based MTL baselines, and it does not sufficiently cover methods that are closer in spirit to LOGIC’s core design, such as copy-based, trajectory-based, or model-merging-style approaches.
- The evidence for the copy-based trajectory mechanism is still limited. Since a central claim of LOGIC is to maintain task-specific parameter copies and coordinate their trajectories in parameter space, the paper should include stronger comparisons with methods that also use parameter copies or local task trajectories. The current ablations show that synchronization and curvature weighting matter, but they do not fully rule out the possibility that much of the gain comes from the multi-copy training structure itself.
- The theoretical and Hessian-based analysis feels relatively thin. The method uses Hessian trace as a curvature proxy, but the analysis does not go much beyond this scalar estimate. There is limited discussion of Hessian spectrum, top eigenvalues, conditioning, negative curvature, trace-estimation noise, or the empirical relationship between curvature weights and final performance. The theory also does not appear to provide a rigorous convergence guarantee, so the main support for the method remains largely empirical.
Questions for authors
Please refer to weakness also. It includes but not limited to questions like:
- Could you include more recent and complete gradient-based MTL baselines, especially methods from 2025 and 2026?
- Could you compare LOGIC with stronger copy-based or trajectory-coordination baselines?
- Can you provide deeper evidence that the Hessian-trace weighting is the main driver of the gains?
3 comments
Response to Reviewer TWL1 - ** Part 2**
W4. Hessian analysis and theory.
(i) Why the trace, and why Hutchinson. Computing the full Hessian spectrum is infeasible for a ResNet-50 encoder with millions of parameters, and it would have to be repeated throughout training; top-eigenvalue estimates are possible but, like the trace, are scalar summaries. Hutchinson's estimator gives an unbiased estimate of the trace from Hessian-vector products alone, without ever forming the Hessian, at the cost of M extra backward passes per refresh. Refreshed every Δ_curv steps, this cost is negligible (App. A.5). App. A.5 also discusses the trace's main limitation, cancellation between eigenvalues of opposite sign, and alternatives that avoid it (top eigenvalue, Frobenius norm, Gauss-Newton). We therefore use the trace only as a relative signal: each task is compared with its own recent history, so a systematic cancellation shared with that history is absorbed by the baseline. The trace serves as a stability signal, not as a flatness certificate.
(ii) Trace-estimation noise. LOGIC is robust to estimator noise at three levels. First, both the per-task baseline and the weights are updated by exponential moving averages, which suppress the noise of the Hutchinson estimator (App. A.2.2). Second, the theory bounds its effect: Prop. 5 shows that the noise in the update direction decays as σ²/T, and Prop. 6 bounds how far the weights can move regardless of the estimated values (at most 0.14 over 1000 steps). Third, changing the number of Hutchinson probes (M ∈ {8, 16, 32}) changes mIoU by only about 0.2.
(iii) Weights and performance. On both benchmarks the weights move from uniform toward depth, the relatively flatter task (ĉ ≈ 0.43 vs. 1/3 on NYUv2; 0.58 vs. 1/2 on Cityscapes; App. A.2.2), and on NYUv2 depth is where LOGIC's largest gains over the baselines appear (Sec. 4.3). Freezing the weights at their converged values gives only Δm% = −0.10 (vs. −2.78 for full LOGIC), so the gain comes from how the weights evolve, not from fixed coefficients.
(iv) Theory and convergence. We agree; App. A.1 ("What we do not claim") states these limits: we prove no convergence or optimality result. The coupled, non-linear dynamics of K task copies, a moving anchor and evolving weights make a convergence proof very hard, and MTL has no canonical scalar objective to converge on. Our theory instead characterizes what the update computes: weighted aggregation at T = 1 (Prop. 2), plus a second-order term D that no single-trajectory method with nonnegative task weights can produce (Props. 3–4). To assess its size, App. A.2.5 measures D directly with Hessian-vector products: it exceeds the first-order term within 50 epochs, reaching a final ratio near 7 on NYUv2 and 70 on Cityscapes, so the update departs substantially from weighted aggregation. As App. A.2.5 notes, reading D as a second-order term is reliable only early in training; later, these ratios measure deviation from aggregation. Convergence guarantees remain future work.
Is Hessian-trace weighting the main driver?
Not on its own. The foundation is the coordination of per-task trajectories: with uniform weights, LOGIC already reaches Δm% = −2.09, better than every baseline in Table 1. Curvature weighting is a significant second component. (i) When copies are never re-anchored, it reduces Δm% from +29.52 to +8.71, so the weights identify which copies to trust. (ii) With synchronization, it improves Δm% from −2.09 to −2.78, and each component contributes (absolute curvature alone: −2.45; Table 4). The gain is positive on all five seeds (paired t-test, p ≤ 0.005). (iii) The benefit comes from how the weights evolve, not from fixed coefficients (W4 (iii)).
We hope the new clarifications resolve the reviewers' concerns. We are happy to discuss any remaining questions.
Thanks for your detailed reply. I will maintain my positive rating of the paper.
Response to Reviewer TWL1 - ** Part 1**
We thank the reviewer for the supportive, constructive feedback, which greatly strengthened the paper. We addressed every comment and added evaluations beyond those requested:
(1) Positioning. LOGIC optimizes each task independently and synchronizes the resulting solutions geometrically in parameter space. To our knowledge, it is also the first MTL method to decouple shared-encoder learning from synchronization of the task-specific copies. The shared encoder moves toward a curvature-weighted centroid at every step, while the copies are synchronized on a progressively less frequent schedule. This keeps task heads coupled to an up-to-date encoder while giving task-specific copies more room to explore between synchronization steps, with bounded drift.
(2) Baselines more than doubled (13 → 31), spanning gradient-based, trajectory-based, task-grouping and model-merging methods, and the latest 2025–2026 methods (Rep-MTL, SAMO, ConsMTL, LDC-MTL, CORE-MTL, ATM). LOGIC ranks first in Δm% and mean rank.
(3) A 40-task stress test. On the challenging large-scale CelebA benchmark (App. A.9), LOGIC achieves the best Δm% (2.36) and mean rank (3.90): a 20% smaller gap to single-task performance than the strongest competitor (SAMO-FairGrad, 2.95), with the lowest cross-seed std (±0.11 vs. ≥±0.28).
Changes are marked in red in the revised PDF; we respond to each point below.
W1. Recent baselines.
We thank the reviewer for raising this concern. We added a significant number of recent multi-task literature to Sec. 2 and to Tables 1–2, including Rep-MTL (ICCV 2025), SAMO (ICCV 2025), ConsMTL (CVPR 2025), LDC-MTL (ECCV 2026), CORE-MTL (ICML 2026) and ATM (ICIAP 2025). All baselines were rerun with their official code and default hyperparameters under our protocol (ResNet-50, HPS, 200 epochs, three seeds, identical STL reference). This matters because Δm% is measured against each protocol's own single-task baseline. DiLoCo and ForkMerge, which were not designed for MTL, were adapted to the multi-task setting. Under the same protocol, LOGIC achieves the best Δm% on both NYUv2 and Cityscapes: −2.78 and −3.40, respectively.
The table below summarizes Δm% (↓, relative to STL) for the added baselines:
As requested, we also evaluated LOGIC against four model-merging methods: TIES-Merging (2023), AdaMerging (2024), SIMO (2025) and SyMerge (2026) (Table 11, Appendix). Consistent with the literature, merging methods generally trail standard MTL training, although recent methods such as SIMO narrow the gap.
W2. Comparison with stronger copy- and trajectory-based baselines.
The table in our response to W1 includes several copy- and trajectory-based methods, as well as model-merging approaches and recent 2025–2026 baselines. We also compared LOGIC with model-merging, auxiliary-task-learning and task-grouping methods (Table 11, Appendix). The table below summarizes Δm% for these additional methods: The table below summarizes Δm% (↓, relative to STL) for these additional methods on NYUv2:
W3. Evidence for the copy-based mechanism.
Is the gain just the multi-copy structure? Our ablation study separates the effect of the copies themselves from the effect of the coordination/synchronization mechanism, and shows that the copies alone do not produce the gain. Training per-task copies without re-anchoring, using equal-weight averaging, reaches Δm% = +29.52, far worse than joint training (Δm% = +2.32). The opposite control, averaging without per-task copies (Lookahead on joint training, App. A.4), also fails (Δm% = +2.59). Only when the copies are properly coordinated, through re-anchoring and the curvature-based centroid, does every configuration surpass single-task learning (Δm% from −2.09 to −2.78). The gain therefore comes from how the copies are coordinated, which is LOGIC's core contribution.
The paper also characterizes the opposite limit. By Prop. 2, synchronizing at every step (T = 1) reduces LOGIC exactly to weighted gradient aggregation, the family to which the gradient-based baselines belong (e.g., SAMO-FairGrad, the strongest of them, reaches Δm% = −1.56). LOGIC's synchronization interval thus spans a range between two limits: no coordination, where copies drift freely (Δm% = +29.52), and no task-specific trajectory (T = 1), where LOGIC becomes a weighted-aggregation method - same family as the baselines.