acceptodds
Under review as a conference paper at ICLR 2027

RepDebt: The Debt Direction Surgery Cannot Replay

Abstract

Multi-task learning hopes similar tasks reinforce each other; interference is just as documented, and few instruments measure either effect. We built one — and it surfaces a failure mode direction surgery cannot see. SphereBench-54 is a benchmark of 54 datasets over 7 modality families, mixed so edge-modality influence stays measurable. We train 423 models under one protocol: 99 jointly on all 54, dense controls parameter-matched at 1.5M, plus 324 per-dataset specialists. We vary only the input representation of four point-cloud datasets (flat serialization, multi-view rendering, Hilbert-curve ordering); the other fifty are byte-identical. This harmless-looking choice decides the fate of plain shared trunks (one encoder, no routing). Our best such trunk swings from the best main-scan macro-accuracy among parameter-matched jointly trained models, 0.636 under one serialization, to 0.392 under another (paired Wilcoxon, ). Expert-routed architectures (MMoE, PLE, Cross-Stitch) at equal or greater budgets move by at most 1.62 points; no expert cross-serialization comparison survives Bonferroni correction. This is, to our knowledge, the first controlled measurement of interference cascading through a shared encoder into byte-identical untouched tasks (the 14 vision datasets drop from 0.46 to 0.19). Gradient probes of retained checkpoints show why: a norm-dominated rank-one capture leaves the collapsed trunk's joint gradient nearly rank-one (mean pairwise task-gradient cosine +0.83 versus 0.08 elsewhere); routed and latent-bottleneck models keep per-task gradients orthogonal. Follow-up controls close the loop: the frozen readout is a minor tax (4.8 of 24.5 points), direction surgery repays nothing, per-task gradient-norm equalization repays 23.7 — the carrier is the norm-dominated rank-one update — and a vanilla dense trunk without the spherical-iterative core never incurs the debt: it is specific to shared iterative dynamics, not dense sharing. Routing buys robustness, not sample efficiency. How far this generalizes is what the benchmark is for. Code, configurations, and the evaluation protocol are released at https://anonymous.4open.science/r/RepDebt-A449.

Primary area:
datasets and benchmarks
Keywords:
multi-task learning, task interference, representation learning, benchmark, multimodal learning, input serialization, gradient conflict, rank collapse, mixture-of-experts, spherical latent space, parameter efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8hhUOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SphereBench-54, a controlled multi-task benchmark spanning 54 public datasets across 7 modality families, with 423 matched-compute models used to study cross-task interference under shared encoders. It identifies a failure mode termed representation debt, where changing only the serialization of four point-cloud datasets can strongly degrade otherwise byte-identical untouched tasks, with macro-accuracy dropping from 0.636 to 0.392 in the best dense shared trunk. Diagnostic analyses attribute this to norm-dominated, nearly rank-one joint gradients in shared iterative architectures; routing, latent bottlenecks, gradient-norm equalization, and non-iterative dense trunks substantially reduce or avoid the effect.

Critical strengths

The paper presents a carefully controlled and unusually large empirical study, with 423 main training arms (482 including follow-up experiments), matched-compute comparisons, multiple architectures, three serialization settings, and targeted intervention studies. The analysis goes beyond reporting performance drops by experimentally probing the proposed mechanism and showing that architectural isolation and specific interventions can substantially mitigate the observed representation-debt effect.

Critical weaknesses

The demonstrated failure mode is currently tied to the specific classification setting and spherical-iterative shared architecture studied here. The authors themselves note that extension beyond classification remains future work, so it is unclear how broadly the phenomenon generalizes to other tasks and model families.

Although the overall experimental campaign is extensive, the main comparisons use only three seeds per configuration. Given some of the relatively small differences among the more robust architectures, additional repetitions would provide stronger evidence about the stability of those comparisons.

Questions for authors

Please follow the weaknesses and strengths.

The demonstrated failure mode is currently tied to the specific classification setting and spherical-iterative shared architecture studied here. The authors themselves note that extension beyond classification remains future work, so it is unclear how broadly the phenomenon generalizes to other tasks and model families.

Although the overall experimental campaign is extensive, the main comparisons use only three seeds per configuration. Given some of the relatively small differences among the more robust architectures, additional repetitions would provide stronger evidence about the stability of those comparisons.

Reviewer_wNW6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies a failure mode in multi-task learning where changing the input representation of a few point-cloud datasets appears to hurt performance on many other datasets that were not changed. The authors build SphereBench-54, a 54-dataset benchmark across multiple modalities, and compare shared-trunk models against routed or more isolated architectures under different point-cloud serializations. Their main claim is that dense shared iterative models can accumulate “representation debt”: a bad representation induced by a few tasks contaminates the shared encoder and degrades other tasks.

Critical strengths

1.The paper studies cross-task contamination in multi-task learning, specifically how a shared encoder can be affected by a small subset of tasks and then hurt otherwise unchanged tasks. 2. The reported 24.5-point collapse is large enough to be hard to dismiss, making the observed failure case worth investigating even if the proposed explanation is not fully conclusive.

Critical weaknesses
  1. The mechanism is not directly established. The paper’s explanation around “representation debt” and norm-dominated rank-one capture is plausible, but much of the evidence is indirect: gradient cosine, latent rank, kNN probes, and recovery via norm equalization.
  2. The presentation and baselines are not strong enough for the claims. Several figures are visually crowded and hard to read, and some details appear poorly rendered, e.g. Figure 5 seems to omit values after “±”. More importantly, the baseline coverage feels incomplete: the paper compares a few routed architectures, but does not provide a sufficiently broad set of standard MTL optimizers, loss-balancing methods, shared-trunk variants, or point-cloud-specific controls.
Questions for authors
  1. Can you provide a more direct causal test for the proposed mechanism?
  2. Could you expand the baseline comparison to cover standard multi-task optimization and loss-balancing methods?
Reviewer_k6WXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes a novel benchmark to evaluate the effects of input representation across tasks. The proposed benchmark contains 54 datasets over 7 modality families, and they evaluate on 6 architectures under the same protocol, and they find that the performance degradation exists when the input representation changes, and the gradient-norm domination is a key factor to this degradation.

Critical strengths
  • The proposed benchmark contains 54 datasets over 7 modality families, which provides an fair empirical evaluation.
  • The evaluation of task interference from the perspective of input representation is under-explored in previous works.
Critical weaknesses
  • My main concern is that the issue pointed by the benchmark may be uncommon, and can be overcomed using existing techniques. For example, as shown in Section G.5, no collapse is observed when using a Transformer as the shared backbone. Furthermore, routing also improves robustness against changes in input representation. Both are widely used techniques in modern deep learning. The main experiments only use 6 different architectures, and severe collapse is observed in spherical-iterative class of models. The value of the benchmark beyond this special model class is unclear to me.
  • The writing makes the paper difficult to follow. First, the structure of this paper is not reasonable. The abstract contains too much information while the introduction provides insufficient context. Moreover, some expressions are unclear and indirect. For example, “world line” and “pays the tax.” It creates barriers to understanding for readers. The statement “train 423 models” may be misleading, as readers could interpret it as using 423 different architectures rather than 423 training runs on 6 different models.
Questions for authors
  1. Is the collpase issue a common problem in MTL or a vunlerability of some special classes of architectures?
  2. Inspired by the experiments on Transformers, if we use architectures that are more suitable for the corresponding modalities, can the collapse issue be significantly mitigated? Do the experimental results suggest a robust and general architecture for multimodal learning?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.