acceptodds
Under review as a conference paper at ICLR 2027

Equivalent Predictors, Different Updates: Canonical-Coordinate Strategy Fusion for Long-Tailed Recognition

Abstract

Long-Tailed Recognition (LTR) increasingly relies on heterogeneous learning strategies with complementary strengths across class-frequency regions. Gradient-based strategy fusion coordinates these strategies through their full-parameter gradients. We identify a non-invariance: functionally equivalent parameterizations of an expert-private classifier can preserve the predictor, loss, and shared-parameter gradient while producing different coordinated updates. The chosen private coordinates therefore become an unintended factor in how LTR strategies are coordinated, a phenomenon we call coordinate leakage. We propose Canonical-Coordinate Strategy Fusion (CCSF), which transports the affected gradients to a common reference coordinate, applies the original fusion rule and optimizer there, and maps the resulting update back to parameter storage. Controlled interventions across multiple full-gradient coordinators reveal the predicted coordinate sensitivity, trace its effect on expert complementarity across frequency regions, and show that CCSF numerically recovers the corresponding reference trajectories. Experiments evaluate the complete configuration on CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018. Code is provided in the supplementary material.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Long-tailed learning, Multi-objective optimization

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_E3FZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper identifies a geometric non-invariance termed coordinate leakage: functionally equivalent parameterizations of an expert-private classifier leave predictions, losses, and shared-parameter gradients intact while rescaling private gradient components. Because full-gradient coordinators rely on gradient norms, inner products, and projection geometries, this private rescaling alters the fusion geometry and yields divergent model updates. To resolve this issue, the authors introduce Canonical-Coordinate Strategy Fusion. CCSF maps private gradients to a canonical reference coordinate, applies the original fusion rule and optimizer in that reference coordinate, and transports the resulting parameter increments back to native storage.

The motivation seems interesting. However, the paper is difficult to follow. The performance improvement is marginal.

Critical strengths

The paper uncovers an overlooked geometric issue in full-gradient multi-strategy fusion: function-preserving reparameterizations of private classifier heads inadvertently alter cross-objective coordination.

Critical weaknesses
  1. Overall improvements on large-scale datasets such as ImageNet-LT and iNaturalist 2018 are almost the same as the TS-MOF baseline.
  2. The performance of this method is not good. Many methods proposed in recent years have a better performance.
  3. The framework is validated within a specific two-stage pipeline (Stage 1 representation frozen; Stage 2 joint fine-tuning of shared decoder alongside LT, BCL, and LOS heads).
Questions for authors

Refer to [Weaknesses].

1 comment

Thank you for your detailed response and the additional experiments. I appreciate the authors' efforts to address my concerns.

However, I remain concerned about the practical effectiveness of the proposed method. In particular, the performance improvements on large-scale datasets are marginal (0.2% on ImageNet-LT and 0.1% on iNaturalist 2018), which limits the evidence for its practical advantages. While the additional experiments help clarify the methodological contribution, they do not fully resolve my concerns regarding the limited empirical gains.

Therefore, I will maintain my original rating.

Reviewer_G9D4OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies gradient-based fusion (training one model on several objectives at once and merging their per-objective gradients into a single update with a multi-task rule such as PCGrad or MGDA) of several long-tailed recognition (LTR) strategies in a two-stage setting. A frozen Stage 1 encoder feeds a shared decoder and three expert heads (an adaptive LT objective, BCL and LOS). A multi-objective coordinator combines their full gradients. The authors observe that storing an expert's private classifier as U=aWU = aW, evaluated as W=U/aW = U/a, leaves the predictor, loss and shared gradients unchanged but rescales the private gradient by 1/a. Fusion rules that read norms, cosines or projections (PCGrad, MGDA, CAGrad, Aligned-MTL, and a stateful importance-based coordinator) then produce different shared updates for functionally identical models. The authors call this "coordinate leakage."

The proposed fix, CCSF, maps affected private gradients to a reference coordinate before fusion and maps the update back afterward. It provably reproduces the reference trajectory for any fixed, known, invertible linear map. The paper reports 58.0% on CIFAR-100-LT (IR=100), 56.5% on ImageNet-LT and 72.4% on iNaturalist 2018, slightly above TS-MOF.

Critical strengths
  1. A correct and clearly stated observation. Coordinators that read full gradients, including expert-private blocks, are not invariant to function-preserving reparameterizations of those blocks. Proposition 1 makes this concrete: the cross-task inner product is preserved while norms, cosines and projection coefficients change.
Critical weaknesses
  1. Leakage is author-applied. Every transformed condition is a map the authors impose themselves (U = 2W, diagonal scales, orthogonal or QR rebasing), and only a = 2 is tested. There is no discussion of a relevant setting landing in a non-reference coordinate on its own. TS-MOF, the cited source of the setting, trains only independent heads, so the leakage cannot arise there either (that paper claims to do gradient combination in a setting where there are no conflicting gradients which is an issue in of itself).

  2. CCSF is not what is improving the numbers. By design, CCSF reports exactly the same accuracies as the untransformed Canonical run (Tables 4 and 11, Appendix C.4). The "improvements" over TS-MOF in Tables 1–2 must therefore simply be implementation improvements in the authors' Canonical pipeline (a shared decoder, an adaptive LT loss and a tuned coordinator). These are never ablated, and they are not a contribution of CCSF.

.

Questions for authors

Is there a way to apply CCSF style fix without needing private information about how the gradients are stored or that does not simply reduce to the version of TS-MOF with a gradient correction? I believe this would basically be a new algorithm, but it would be necessary for consideration of publication.

Reviewer_oVYyOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a parameterization-sensitivity issue in gradient-based strategy fusion for long-tailed recognition. The authors observe that a function-preserving reparameterization of an expert-private classifier can leave the predictions, losses, and shared-parameter gradients unchanged while altering private-gradient magnitudes and the full-gradient geometry used by a multi-objective coordinator. They call the resulting dependence of coordinated updates on private storage coordinates coordinate leakage.

To address this issue, the paper proposes Canonical-Coordinate Strategy Fusion (CCSF). It maps affected private gradients into a common reference coordinate, applies the original fusion rule and optimizer in that coordinate, and transports the resulting parameter increments back to storage. Theoretical results characterize the sensitivity and establish reference-consistent execution under fixed, known, invertible coordinate maps. Controlled experiments examine multiple coordinators and recovery of reference trajectories, while benchmark evaluations report long-tailed recognition results on CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018.

Critical strengths

A clearly identified optimization phenomenon supported by theory and controlled evidence.

The paper shows that, in full-gradient fusion of heterogeneous long-tailed learning strategies, functionally equivalent private parameterizations can alter gradient geometry and consequently affect coordinated shared updates. The mathematical analysis, experiments across different gradient coordinators, and CCSF reference-trajectory recovery provide coherent evidence for this phenomenon. Importantly, the paper connects the established broader issue of parameterization sensitivity to shared optimization and expert complementarity in long-tailed recognition, rather than merely illustrating different updates for a single model.

Critical weaknesses

1. Practical relevance and scope of the coordinate-consistency guarantee remain unclear.

CCSF assumes a known, fixed, invertible transformation between the stored and reference coordinates. Although the paper provides controlled tests beyond uniform scaling, it does not sufficiently establish how often the relevant equivalent private-coordinate changes arise naturally in practical long-tailed strategy fusion, or how an appropriate reference coordinate would be identified in less controlled settings. Thus, while the theoretical phenomenon and correction are convincing under the stated assumptions, the practical necessity and applicability of CCSF need clearer motivation.

2. Performance attribution and cross-benchmark benefits are insufficiently clarified.

The matched experiments show that CCSF recovers the Canonical training trajectory; this demonstrates coordinate consistency, but not an improvement over the reference execution itself. The main benchmark comparisons report gains over TS-MOF, yet Appendix C.3 describes a controlled TS-MOF reproduction with a KPS/BCL/LOS strategy configuration and its original coordination protocol, whereas the complete CCSF configuration uses an adaptive LT objective and its specified fusion setup. This makes it difficult to attribute the entire reported improvement specifically to coordinate correction. In addition, the Overall gains on ImageNet-LT and iNaturalist 2018 are only 0.2 and 0.1 percentage points, respectively, and Few-class accuracy decreases on ImageNet-LT while increasing on iNaturalist 2018. The source and consistency of these performance effects require clearer explanation.

Questions for authors

1. Practical occurrence and reference selection. In realistic long-tailed multi-strategy training, under what circumstances do equivalent private-coordinate changes arise without being deliberately introduced? How would the method identify or maintain the canonical reference when the relevant transformation is not already known and fixed?

2. Attribution and variation across datasets. Could the authors separate the gains specifically due to correcting coordinate leakage from gains due to differences in strategy objectives, fusion rules, or other configurations relative to TS-MOF? How do they explain the small Overall improvements on the two large-scale datasets and the different directions of the Few-class changes on ImageNet-LT and iNaturalist 2018?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.