- Rating
- 3
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces SphereBench-54, a controlled multi-task benchmark spanning 54 public datasets across 7 modality families, with 423 matched-compute models used to study cross-task interference under shared encoders. It identifies a failure mode termed representation debt, where changing only the serialization of four point-cloud datasets can strongly degrade otherwise byte-identical untouched tasks, with macro-accuracy dropping from 0.636 to 0.392 in the best dense shared trunk. Diagnostic analyses attribute this to norm-dominated, nearly rank-one joint gradients in shared iterative architectures; routing, latent bottlenecks, gradient-norm equalization, and non-iterative dense trunks substantially reduce or avoid the effect.
Critical strengths
The paper presents a carefully controlled and unusually large empirical study, with 423 main training arms (482 including follow-up experiments), matched-compute comparisons, multiple architectures, three serialization settings, and targeted intervention studies. The analysis goes beyond reporting performance drops by experimentally probing the proposed mechanism and showing that architectural isolation and specific interventions can substantially mitigate the observed representation-debt effect.
Critical weaknesses
The demonstrated failure mode is currently tied to the specific classification setting and spherical-iterative shared architecture studied here. The authors themselves note that extension beyond classification remains future work, so it is unclear how broadly the phenomenon generalizes to other tasks and model families.
Although the overall experimental campaign is extensive, the main comparisons use only three seeds per configuration. Given some of the relatively small differences among the more robust architectures, additional repetitions would provide stronger evidence about the stability of those comparisons.
Questions for authors
Please follow the weaknesses and strengths.
The demonstrated failure mode is currently tied to the specific classification setting and spherical-iterative shared architecture studied here. The authors themselves note that extension beyond classification remains future work, so it is unclear how broadly the phenomenon generalizes to other tasks and model families.
Although the overall experimental campaign is extensive, the main comparisons use only three seeds per configuration. Given some of the relatively small differences among the more robust architectures, additional repetitions would provide stronger evidence about the stability of those comparisons.