acceptodds
Withdrawn from ICLR 2027

When Aggregation Hides Directional Structures in Federated Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Federated learning (FL) relies on aggregating decentralized client updates into a global update, yet this many-to-one aggregation can obscure important directional structures contained in individual client updates. In particular, distinct client directional configurations may yield similar aggregate updates because opposing directions partially cancel each other, making the underlying directional relationships less observable from the aggregate representation alone. We characterize this phenomenon as Hidden Directional Opposition (HDO), which captures client-level directional opposition that exists before aggregation but becomes partially unobservable after aggregation. To analyze such hidden structures in a shared coordinate system, we construct a geometry-aware tangent-space directional representation that reduces endpoint-dependent interference while retaining relative directional displacement. We further introduce a direction-preserving adaptive optimization mechanism that adapts update magnitudes without introducing additional directional rotation. Experiments on CIFAR-10 and CIFAR-100 show that HDO consistently emerges under heterogeneous federated training, becomes more pronounced with stronger client heterogeneity, and exhibits distinct temporal and layer-wise patterns. These results highlight that federated aggregation determines not only the global update but also which client-level directional structures remain observable at the server.

Primary area:
optimization
Keywords:
Federated Learning, Federated Aggregation, Client Heterogeneity, Hidden Directional Opposition, Directional Representation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_5sWdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper defines Hidden Directional Opposition from negative pairwise inner products and aggregate directional energy. Client endpoints are projected onto a sphere and mapped to a common tangent space. A2 then applies scalar adaptive scaling to the aggregated first moment. Experiments characterize directional opposition across heterogeneity levels, training rounds, and network layers.

Critical strengths
  1. The energy decomposition separates positive and negative pairwise contributions to the aggregate update.

  2. The experiments cover CIFAR-10 and CIFAR-100 with multiple seeds. The paper also distinguishes tangent-step direction preservation from predictive performance and acknowledges that FedAdam achieves higher accuracy.

Critical weaknesses
  1. Table 1 is inconsistent with the stated definitions. For nonnegative weights summing to one, EQ−EA=∑i<jpipj∥vi−vj∥2≥2D−E_Q-E_A=\sum_{i<j}p_ip_j\|v_i-v_j\|^2\geq2D^-. However, CIFAR-10/C2 at α=0.1\alpha=0.1 reports EQ−EA=1.509E_Q-E_A=1.509 and D−=4.750D^-=4.750. Several other rows also violate this inequality. Different scaling or aggregation conventions need to be disclosed; otherwise, the central empirical characterization is not reliable.

  2. The distinction between HDO and existing conflict measures remains insufficiently established. D−D^- is a magnitude-weighted sum of negative pairwise inner products, and r−r^- normalizes it using aggregate energy. Both are computed from client updates. The paper does not demonstrate what additional information they provide beyond conventional conflict and cancellation statistics.

  3. The spherical representation can discard the directional opposition being studied. For Δ1=ax\Delta_1=ax and Δ2=−ax\Delta_2=-ax with 0<a<10<a<1, both projected endpoints equal xx, so both tangent representations vanish. The paper needs to delimit its analysis to angular opposition and explain what information is lost through projection. Table D1 provides little evidence of an advantage over raw updates.

  4. A2 preserves the aggregated first-moment direction, not the individual client directions lost during averaging. It does not use D−D^- or r−r^- to recover or respond to hidden opposition. The direction-preservation experiment verifies scalar collinearity, but does not establish preservation of client-level information.

  5. Scalar adaptive scaling has precedents in AdaSGD and VectorAdam, which are not discussed. The comparisons also lack a Euclidean scalar-adaptive baseline or a tangent-space momentum baseline, leaving the contributions of geometry, momentum, scalar adaptation, and angular clipping unresolved.

  6. Important implementation details are missing, including the momentum transport rule, server learning rates, moment coefficients, angular cap, and numerical safeguards for the spherical maps. The auxiliary methods SW2 and FMO are also insufficiently specified. These omissions prevent reproduction and make the optimizer comparisons difficult to assess.

Questions for authors
  1. How can Table 1 be reconciled with EQ−EA≥2D−E_Q-E_A\geq2D^-? Are the columns computed from the same vectors, weights, normalization, and temporal aggregation?

  2. What information does HDO add beyond magnitude-weighted negative inner products and standard cancellation measures? Do existing results isolate this distinction?

  3. How much update energy is removed by spherical projection? How should radial opposition, which maps to zero, be interpreted within the proposed observability framework?

  4. What client-level directional information does A2 retain beyond the aggregated first moment? Is there an existing diagnostic demonstrating this retention?

  5. How does A2 differ from existing scalar-adaptive optimizers, and are there controlled comparisons separating geometry from momentum and adaptive scaling?

  6. Please specify the momentum transport, server hyperparameters, angular cap, and the procedures used by SW2 and FMO.

  7. Please retain the standard ICLR body font and formatting. The axis labels and tick values in several figures are too small to read comfortably.

Reviewer_jcPCOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates directional observability during the server-side aggregation of learning information in federated learning. To this end, the paper introduces Hidden Directional Opposition(HDO) to quantitatively analyze directional opposition among clients. Furthermore, the paper proposes a geometry-aware tangent-space directional representation that preserves displacement information while mitigating interference arising from client-specific endpoint dependencies. This representation also serves as a tool for systematically examining directional relationships among clients. Finally, the paper presents direction-preserving adaptive optimization to prevent unnecessary directional rotations introduced by conventional adaptive optimization methods while preserving the directional integrity of model updates.

Critical strengths
  1. This paper attempts to characterize and analyze gradient conflicts arising during the aggregation stage of federated learning by introducing HDO. It also seeks to uncover the hidden structure of these conflicts by developing a geometry-aware tangent-space directional representation.
Critical weaknesses
  1. The paper lacks clarity and readability, making it difficult for readers to follow. First, A2 is introduced for the first time at line 135 without any explanation of what it represents. Additionally, the font sizes in Figures 2, 3, and 5 are too small, making it difficult to read the axis labels, legends, and their corresponding values. The experimental section also lacks a detailed explanation of what C2 represents.
  2. The proposed methods lack sufficient explanations and theoretical justification. For instance, Dt−D_t^- and rt−r_t^- are introduced merely as quantities used to define HDO, without adequately explaining their meanings, why they are necessary, or what makes them important. Similarly, Section 3.4 does not provide sufficient explanations of the individual components of the proposed direction-preserving adaptive optimization method. More importantly, the paper lacks a clear explanation or theoretical justification for why the proposed method actually preserves the direction of model updates.
  3. The experimental findings do not appear to provide sufficiently meaningful insights. In Section 4.2, the paper states that "EQ remains consistently larger than the observable aggregate energy EA throughout training." However, this relationship appears to follow directly from the mathematical formulation, making the experimental observation somewhat trivial. Similarly, Section 4.3 reports that "hidden opposition becomes more pronounced under stronger client heterogeneity." Since greater client heterogeneity would naturally be expected to increase directional conflicts among client updates, it is unclear what additional insight this experimental result provides.
  4. The results in Section 4.6 raise concerns about the effectiveness of the proposed direction-preserving adaptive optimization method. In particular, the proposed method exhibits substantially worse performance than FedAdam, making it difficult to conclude that the experimental results support the paper's claims regarding the benefits of preserving update directions.
Questions for authors
  1. What do Dt−D_t^- and rt−r_t^- represent, and what is the significance of using these quantities to define HDO?
  2. Are there any experimental results on more diverse and challenging image classification datasets beyond CIFAR, such as Places365 or SUN397?
Reviewer_kNSwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates federated aggregation from a directional observability perspective and introduces Hidden Directional Opposition (HDO) to characterize client-level directional opposition whose observability is reduced by aggregation. To analyze these directional relationships, the paper constructs a geometry-aware tangent-space directional representation that reduces endpoint-dependent interference while preserving relative directional information. It further proposes a direction-preserving adaptive optimization mechanism that adjusts update magnitudes without introducing additional directional rotation.

Critical strengths

1.This paper studies directional information obscured through aggregation. This perspective provides a useful complement to existing studies of heterogeneous federated optimization. 2. HDO explicitly relates client directional opposition to the information retained by the aggregation mapping, providing a complementary view of heterogeneous federated optimization by distinguishing directional structure that exists at the client level from that which remains observable in the aggregate. 3.Experiments show that HDO persists under heterogeneous training and varies systematically across heterogeneity levels, training stages, and network layers, offering useful insights into the behavior of heterogeneous client updates.

Critical weaknesses

1.The experimental scope limits the generality of the conclusions. All federated experiments focus on CIFAR-10 and CIFAR-100 using 10 clients, and all multi-seed experiments are conducted with three random seeds (3, 4, and 5). A relatively small client population, and a limited number of random seeds. Further evaluation across diverse model architectures and larger client populations would help establish whether the observed directional structures generalize beyond the current settings. 2.This paper do not establish a universal causal relationship between HDO magnitude and predictive performance. The authors should further investigate the relationship between HDO and optimization outcomes, and clarify the conditions under which HDO can serve as a reliable indicator of performance degradation.

Questions for authors

1.Table C2 analyses the relationship between directional disagreement measures and HPO However, Table C2 does not fully establish whether HDO provides additional insights beyond existing measures. The authors should provide evidence showing specific cases in which HDO reveals optimization-relevant information that conventional metrics fail to capture, and clarify how such information can address aggregation-related issues in federated learning. 2.The tangent-space analysis and pairwise HDO computation also introduce additional computational cost at larger scales. HDO estimation requires access to individual client updates, which may introduce additional communication, and scalability challenges. A more quantitative assessment of communication cost, computational cost, and the practical feasibility would improve the paper. 3.Table 2 show that FedAdam achieves higher performance than A2, under what conditions does preserving directional information improve optimization? 4.The contributions of the individual components have not been sufficiently validated. It remains unclear whether the performance improvements primarily stem from the geometry-aware tangent-space directional representation, the direction-preserving adaptive optimization mechanism, or other factors. The authors should conduct more comprehensive ablation studies to evaluate the individual and combined effects of these components.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.