acceptodds
Under review as a conference paper at ICLR 2027

On the Consistency of Federated LoRA under Secure Aggregation via Functional Sketching

Abstract

Federated low-rank adaptation (LoRA) on heterogeneous devices faces two coupled aggregation inconsistencies. Independently averaging factors generally differs from averaging their products, while additive masking before local factor multiplication introduces non-canceling cross terms. We propose SAGE-LoRA, a functional-sketch framework that applies secure aggregation (SecAgg) to linear measurements of locally paired products rather than factor coordinates. Budget-aware paired blocks support heterogeneous local ranks, while fixed-width sketches decouple per-client measurement size from local rank and cohort size. An anchored right sketch identifies a cohort-level range, and a range-locked left sketch from the same surviving cohort enables fixed-rank reconstruction. Clients evaluate both sketches through factor chains before masking, without materializing dense updates. We establish exact measurement aggregation and sequential aggregate confidentiality under the universal SecAgg assumptions, and derive reconstruction and time-averaged stationarity bounds that distinguish approximation from optimization error. Across eight commonsense tasks with Qwen2.5-1.5B-Instruct and rank-8 adapters, SAGE-LoRA with SecAgg achieves 76.37% mean accuracy, within 0.66 percentage points of unprotected FlexLoRA and over 5 points above unprotected FLoRA and FSLoRA. Under the evaluated representations and precisions, logical communication is 67.3% and 60.1% lower than FLoRA and FSLoRA, respectively, including second-stage projection traffic. The paired mean accuracy change from plaintext is only 0.10 percentage points. On five GLUE tasks, plaintext SAGE-LoRA improves the macro score over FLoRA, FSLoRA, and FlexLoRA by 0.42–2.40 points. These results support SAGE-LoRA as a practical exploration of SecAgg for federated LoRA that reconciles product consistency, aggregate confidentiality, and compact communication.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
federated learning, federated fine-tuning, low-rank adaptation, secure aggregation, parameter-efficient fine-tuning, privacy-preserving machine learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_EV8ROpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SAGE-LoRA, a functional-sketch framework for secure federated LoRA aggregation. The authors identified two long-standing issues in existing approaches: factor averaging is inconsistent with product averaging, and pre-factor masking introduces non-canceling cross terms. SAGE-LoRA addresses these issues by securely aggregating linear sketches of locally formed LoRA products instead of factor coordinates, enabling product-consistent aggregation with heterogeneous ranks. The paper provides theoretical analyses on aggregation correctness, reconstruction error, convergence, and confidentiality under SecAgg assumptions. Experiments on Qwen2.5-1.5B-Instruct demonstrate competitive performance and reduced communication overhead. Overall, the paper has several practical issues regarding deployment overhead, communication evaluation, security scope, and scalability.

Critical strengths
  1. The paper proposed a systematic approach to solve the aggregation inconsistency in FedIT and security-interface inconsistency in additive secure aggregation.

  2. The proposed approach performs linear sketching before masking, avoiding bilinear mask interaction terms while maintaining compatibility with additive SecAgg.

  3. The paper combines theoretical analysis with experiments covering utility, communication efficiency, heterogeneous ranks, reconstruction fidelity, and security-utility trade-offs.

Critical weaknesses
  1. The novelty of SAGE-LoRA is insufficiently established. The aggregation inconsistency illustrated in Fig. 1(a) has already been extensively investigated in prior works (e.g., FedEx-LoRA (ACL 2025), LoRA-FAIR (ICCV 2025), Te-LoRA (ICCV 2025), FedRot-LoRA (ICML 2026), etc.). More importantly, FedASK (NeurIPS 2025) already employs a two-stage product-sketching framework for privacy-preserving federated LoRA. (Q1)

  2. The practical reliability and efficiency of the two-stage secure aggregation protocol remain insufficiently evaluated. (Q2-4)

  3. The reported communication savings are partially confounded by heterogeneous numerical representations and precision settings. (Q5)

  4. Weak security model, evaluation generality. (Q6-7)

Questions for authors
  1. Given the long history of the aggregation inconsistency, what specific limitations remain unresolved, and how does SAGE-LoRA overcome them? For example, what are the essential technical differences and advantages of SAGE-LoRA beyond replacing DP with secure aggregation? Under what conditions does the proposed anchored, range-locked sketching provide measurable utility advantages over the other priors, especially FedASK? A more detailed methodological comparison and controlled experimental evaluation would strengthen the novelty claims.

  2. The paper introduces two update modes, but the practical selection criteria remain unclear. What are their respective advantages and limitations in terms of convergence, computational cost, and adaptation performance?

  3. How sensitive is SAGE-LoRA to the sketch dimensions? The reconstruction quality depends on the selected sketch widths s_R and s_L. Could the authors provide an ablation study showing how different sketch dimensions affect reconstruction error, downstream accuracy, and communication cost?

  4. Compared with a matched one-stage secure aggregation protocol, what additional latency, communication, synchronization, and authentication overhead does the sequential design introduce? Although Table 15 reports aggregate execution times, could the authors provide stage-wise measurements and evaluate round-completion rates, waiting times, retries, and overall training throughput under different client dropout rates?

  5. Are the reported communication advantages maintained under matched numerical precision? Could the authors provide BF16-to-BF16 or FP32-to-FP32 comparisons, in addition to the existing scalar-count analysis, and report actual transmitted bytes including numerical encoding, cryptographic metadata, and secure aggregation protocol overhead?

  6. The paper explicitly limits its formal analysis to semi-honest aggregate confidentiality. What information about individual clients may still be inferred from repeated authorized aggregate releases, varying participating cohorts, or published global models? Could the authors provide a more concrete evaluation of these residual privacy risks and discuss the requirements for extending the protocol to stronger adversarial settings?

  7. The evaluation is mainly conducted on Qwen2.5-1.5B-Instruct with rank-8 LoRA adapters. How does SAGE-LoRA scale to substantially larger models, more participating clients, and more heterogeneous computation and communication environments? Could the authors evaluate its memory, communication, and execution-time requirements under these settings?

6 comments
AuthorsOpenReview ↗

Responses to Q1–Q7: Novelty, Security, and Experimental Validation

We thank the reviewer. We address the seven questions below.

Q1. Novelty beyond FedASK. We agree that FedASK (NeurIPS 2025) already provides two-stage product sketching and a non-DP option; we do not claim these operations as new. Appendix E.2 states that both produce the same reconstruction under an identical target, range, and truncation rule.

SAGE specifies a heterogeneous paired-product interface: independently trained client blocks are measured through fixed-width factor chains, with sequential SecAgg over the same weighted cohort. Its public-adapter anchor retains existing global directions. In a matched receiver comparison, reconstructed products differ by at most 4.66×10−84.66\times10^{-8} of the target norm; anchoring reduces relative error from 3.333×10−33.333\times10^{-3} to 2.374×10−32.374\times10^{-3} (28.8%) for either receiver (Tables 22 and 24). On OpenChat-7B, SAGE and non-DP FedASK obtain 89.19%/89.22% commonsense and 47.56%/47.61% MMLU accuracy, indicating comparable utility. FedEx-LoRA's rank-eight correction followed by rank-eight truncation gives 0.00691 relative error, close to SAGE's 0.00693. Unequal-rank FedIT also shows a round-two discrepancy of 0.19597, versus 0.00460 at common rank eight (Table 6).

Q2. Complete versus residual updates. Section 2.1 and Appendix F.1 define complete mode as Pt=0P^t=0 and residual mode as Pt=MtP^t=M^t. They share a measurement interface but differ in optimization trajectory. A complete adapter may have rank at most rir_i, while its displacement from the global adapter can have rank ri+kr_i+k. Appendix C.9 analyzes the modes separately. Some complete-update runs could not maintain eight measured directions for all 100 rounds, whereas corresponding residual runs completed all rounds.

Q3. Sketch-dimension sensitivity. Figure 9 compares fixed and adaptive measurements. On identical round-ten products, fixed balanced, adaptive Gaussian, and adaptive anchored designs obtain relative errors 0.045307, 0.008676, and 0.006933. Their training-trajectory accuracies are 76.29%, 76.45%, and 76.26%. Thus, improved matrix reconstruction does not guarantee higher task accuracy. Compact 14/14 removes the six completion rows transmitted in 14/20.

Q4. Sequential protocol reliability and cost. Stage II authenticates the Stage-I survivor set and protected count before share release; any post-Stage-I dropout aborts the round. Appendix E.2 reports gRPC median stage durations of 0.445 s and 0.728 s. Appendix F.9 reports that two secure sums take 81.96 s versus 65.15 s for one sum in the tested 40-client single-host setup. These are measured costs, not a general dropout-rate model. We acknowledge the extra projection broadcast, second sum, synchronization, and zero post-Stage-I dropout tolerance. The second projection is derived from the first aggregate; a fixed one-stage sketch cannot adapt to that cohort-specific range.

Q5. Matched-precision communication. We separate native logical payload, common BF16 payload, and serialized TCP traffic. Under common BF16 accounting, SAGE 14/20 requires 10.04 MB/client-round versus 23.97 MB for FLoRA and 19.61 MB for FSLoRA. Compact transposed 14/14 requires 8.40 MB; matched direct 14/14 requires 7.40 MB. Rank-eight factor communication remains smaller.

Q6. Privacy and stronger adversaries. Section 2.2 and Appendix E consider a semi-honest server colluding with clients within the SecAgg threshold. Proposition E.1 requires auxiliary-input simulation, sequential composition, fresh stage randomness, and pre-release authentication of the same cohort and protected count. Aggregates and the global adapter remain observable. SecAgg is not differential privacy and does not prevent inference from released models or differencing across overlapping cohorts. The guarantee excludes malicious input substitution, poisoning, Sybil attacks, traffic analysis, side channels, and denial of service. Authentication binds roster and count, not the benignness of private inputs.

Q7. Model and cohort scaling. We evaluate OpenChat-3.5-0106 (7B), adapting 64 query/value matrices. SAGE obtains 89.19% commonsense and 47.56% MMLU accuracy; Appendix F.8 reports 14.25 GiB peak allocated GPU memory. Appendix F.9 evaluates 10, 20, and 40 concurrent clients; at 40 clients, two secure sums take 81.96 s in the tested single-host setup. These results show fixed-width messages independent of cumulative client rank, while coordination cost grows with cohort size.

SAGE thus provides product-consistent measurements for heterogeneous paired LoRA under sequential SecAgg, with explicit reconstruction, security, utility, and system-cost analyses.

Thank you for the detailed response. Some key concerns still remain:

  1. The novelty is still not clear. Can you explicitly indicate what remains unsolved in the previous works and how it motivates SAGE-LORA? What makes the previous works incapable in solving aggregation inconsistency and security-interface inconsistency, and what is the evidence that supports your argument?

  2. The criteria for selecting the update mode (complete-update mode or residual mode) is still unclear. What criteria should practitioners use to select an update mode before training? Algorithm 2 still takes the update mode as an input, then what criteria should practitioners use to switch modes?

AuthorsOpenReview ↗

SAGE-LoRA's Novelty and Practical Update-Mode Selection

Thank you for these focused questions. We clarify SAGE's additional contribution and the criteria for choosing its update mode.

1. What remains unresolved, and what does SAGE add?

Prior work already addresses aggregation inconsistency in several settings. SAGE asks how to exploit the evolving public adapter to reconstruct heterogeneous product averages from fixed-width secure sums, with error control and consistent sequential execution.

Securely summing factors cancels masks correctly, but multiplying their means generally differs from averaging their products (Eq. 1; Table 6). Multiplying individually masked factors instead introduces the cross terms in Eq. 2. SAGE evaluates paired-product measurements locally before masking, so the secure sum measures the intended product average. Measurement aggregation is exact; rank-limited reconstruction remains approximate.

FedEx-LoRA and FLoRA recover product averages through residual correction and stacking; FlexLoRA compresses the product average. Alignment/refinement methods, including FedRot-LoRA, LoRA-FAIR and Te-LoRA, address factor mismatch. Fixed-factor/core methods make aggregation linear by restricting the trained subspaces. SAGE retains independently trainable factor pairs, permits unequal local ranks, and obtains a prescribed-rank return through fixed-width measurements without dense client products or common factor coordinates (Secs. 3.1-3.3).

FedASK is the closest counterpart: its two-stage construction, non-DP option and fixed-projection linear measurements already support this basic sketching route. SAGE's principal methodological addition is public-adapter-anchored measurement geometry with change-dependent reconstruction analysis. FedASK's random probe does not explicitly include the public adapter's row space. SAGE includes that space and samples complementary directions for new variation, without restricting local training to either subspace. For public adapter MM and target SS, with orthonormal Ω\Omega containing row(M)\mathrm{row}(M) and orthonormal QQ spanning SΩS\Omega, Proposition C.3 gives ∥(I−QQ⊤)S∥F≤∥(S−M)(I−ΩΩ⊤)∥F≤∥S−M∥F.\|(I-QQ^\top)S\|_F \le \|(S-M)(I-\Omega\Omega^\top)\|_F \le \|S-M\|_F. Thus, missed-range error is controlled by the change from the public adapter: when SS is close to MM, this bound is small. The analysis separates this error from rank truncation.

At matched 14/14 width, rank-eight return and payload, anchoring lowers reconstruction error by 20.1% on identical client products (Sec. 5.3). The separate probe/receiver control (Table 1b) shows the benefit with both receivers, identifying measurement design as the source of improvement and its applicability to FedASK.

For the adaptive decoder, the second sum measures coordinates in the range selected by the first. SAGE specifies how both sums retain one weighted target: the same clients reuse their frozen products and weights. Proposition E.1 states the SecAgg and pre-release authentication assumptions for sequential aggregate confidentiality; Fig. 6 tests target mismatch. These conditions matter because linearity within each secure sum alone does not ensure consistency between the two sums. The contribution combines anchored reconstruction, its error analysis and an explicit secure execution for heterogeneous products.

2. How should practitioners choose the update mode?

Select by the local adaptation objective and resource budget (Sec. 2.1; Eq. 22):

  • Complete updates: choose this mode to jointly readapt the entire local adapter within rank rir_i. Clients initialize from [M]min⁡(ri,k)[M]_{\min(r_i,k)} and train both factors; the aggregate replaces the global adapter. If ri≥kr_i\ge k, the full global adapter is retained at initialization. If ri<kr_i<k, this choice accepts truncation of the initial public adapter.

  • Residual updates: choose this mode when clients should retain the full public adapter during local training and devote their trainable rank to an additive correction. Clients freeze MM and train fresh rank-rir_i blocks around it. This is particularly relevant when ri<kr_i<k, provided the device can store and evaluate the frozen rank-kk base as well as the trainable blocks. Retention here concerns the local base; the server still returns rank kk.

We recommend complete updates by default, as in our main experiments. The modes optimize different parameterizations: complete displacement has rank at most ri+kr_i+k, versus rir_i for a fresh residual (Appendix C.9). These rank bounds do not establish an accuracy ranking.

Algorithm 2 takes the mode as input because it executes one round of the chosen parameterization. Select one mode before training and keep it fixed across the run and cohort. It has no automatic switching policy: numerical sketch rank below kk triggers abort, not a mode change. Both modes share the secure measurement interface.

Reviewer_KcWoOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the author argues that directly averaging independently trained LoRA factors fails to preserve the average of their products, and applying additive masks before factor multiplication introduces cross terms that do not cancel. To address these issues, the paper proposes SAGE-LoRA, which securely aggregates compact linear sketches of locally computed adapter products and reconstructs a fixed-rank global adapter while supporting heterogeneous client ranks. The paper provides theoretical guarantees for aggregation consistency, reconstruction error, and confidentiality under stated assumptions, with experiments showing competitive accuracy and reduced logical communication relative to selected baselines.

Critical strengths
  • Overall, the paper is well organized.
  • Evaluating linear product sketches via low-rank factor chains (Bi(AiΩt)B_i(A_i\Omega^t) and (ΨtBi)Ai(\Psi^t B_i)A_i) before masking neatly avoids both factor-averaging mismatch and bilinear mask cross terms without forming dense matrices.
  • The proposed emthod has rigorous theoretical scaffolding. The orthogonal error decomposition (Theorem C.2), stationarity analysis (Theorem 4.4), and sequential composition proof (Proposition E.1) are mathematically sound and well-organized.
  • The authors discussed implementation limitations in the appendices, including the post-unmasking check, FP32/BF16 precision gap, discarded HtH_t rows, and zero inter-stage dropout tolerance.
Critical weaknesses
  • Proposition E.1 requires Stage II to authenticate the Stage-I survivor set and protected count before releasing the second aggregate, yet Appendix E.2 admits the Flower reference checks consistency only "after unmasking." Because pre-release same-cohort binding is essential to prevent cohort-differencing leakage, the results in Table 14 only measure a simplified wrapper rather than validating the formal protocol. The authors should either implement pre-release binding or clearly separate the formal protocol from the evaluated reference in the main text.

  • While motivated by factor-averaging inconsistency (EfacE_{fac}), the paper lacks discussion and comparison against methods specifically designed to correct exact-product or aggregation-bias errors (e.g., residual-correction approaches). Furthermore, factor-space baselines like FedIT and FedRot-LoRA match or outperform SAGE-LoRA on homogeneous tasks (Tables 1–2) and beat it by up to 2.64% under heterogeneous ranks (Table 8). A sharper analytical and empirical contrast against exact/bias-corrected methods is needed to clarify the true novelty.

  • In Table 13, non-adaptive Matched random bilateral sketching achieves 76.36% accuracy, slightly outperforming the proposed adaptive two-stage SAGE-LoRA (76.33%) despite higher matrix reconstruction error. Since random bilateral sketches can be generated via public seeds in a single stage, the heavy costs of the two-stage design, doubled SecAgg rounds, 6.41 GiB of projection downlink, a 135x server runtime increase (Table 15), and zero tolerance to inter-stage client dropouts (Section 2.2), somehow lack end-to-end utility justification.

  • The claimed 67.3% and 60.1% byte savings over FLoRA and FSLoRA conflate representation size with precision, as baselines use FP32 while SAGE-LoRA uploads BF16 sketches (Appendix F.4). Moreover, setting sR=14,sL=20s_R = 14, s_L = 20 broadcasts and aggregates 6 completion rows (HtH_t) that the decoder Gt†=[IsR,0]G_t^\dagger = [I_{s_R}, 0] immediately discards (Section 3.3), wasting 30% of Stage-II bandwidth. Finally, switching from complete updates (homogeneous) to residual updates (heterogeneous) confounds the evaluation in Table 8.

Questions for authors
  • Can you implement the pre-release authenticated binding required by Proposition E.1, or revise the main-text claims to reflect the post-unmasking wrapper?

  • Given Table 13, why not use a single-stage seeded random bilateral sketch to eliminate the second SecAgg round, the 6.41 GiB projection broadcast, and the inter-stage dropout fragility?

  • How do the communication numbers look under equal precision (all BF16) with sL=sR=14s_L = s_R = 14, and how does Complete-update (Eq. 20) perform under heterogeneous ranks?

  • Please also refer to the weakness section.

4 comments
AuthorsOpenReview ↗

Formal Security Scope, Residual Correction, and Protocol Costs

We thank the reviewer for recognizing SAGE-LoRA's mathematical structure and for raising important questions about security implementation, exact-product aggregation, and protocol costs. The revision clarifies the formal guarantees and adds controlled comparisons.

1. Pre-release authentication and implementation scope. We agree that the post-unmasking check in the Flower utility reference does not instantiate the pre-release authentication required by Proposition E.1. The revision distinguishes the formal protocol, the Flower utility/performance reference, and a separate authenticated gRPC implementation. The formal protocol requires Stage-II authentication of the Stage-I survivor roster and protected count before shares are released. The gRPC tests accept intact requests and reject tampered rosters, inconsistent signed counts, removed proofs, and attempts to bypass the release check. We do not claim that the Flower utility implementation itself provides this pre-release guarantee. Proposition E.1 remains conditional on the SecAgg backend satisfying the stated auxiliary-input simulation and sequential-composition assumptions.

2. Exact-product aggregation and FedEx-LoRA. Appendix B compares FLoRA, FlexLoRA, FedEx-LoRA, FedRot-LoRA, Fed-SB, and SAGE by their aggregation objects. FedEx-LoRA can recover the exact product mean through a residual-correction statistic beyond the two factor sums. On the same round-ten FedIT products, rank-eight and rank-sixteen corrections yield relative errors 0.00200 and 0.00071 before truncation; after rank-eight truncation, the error is 0.00691, close to SAGE's 0.00693 (Appendix B.9 and Table 1). This comparison clarifies SAGE's fixed-width measurement objective; it does not claim residual correction is incapable of exact aggregation. Table 6 also reports a round-two FedIT discrepancy of 0.19597 for unequal ranks versus 0.00460 for common rank eight.

3. Why adaptive two-stage measurement? A fixed bilateral sketch is a valid alternative. Appendix F.5 compares it directly: fixed random and anchored bilateral measurements achieve 76.36% and 76.33% on their respective training trajectories. We do not claim an accuracy advantage for the adaptive protocol. Its second projection is instead derived from the range of the current first aggregate, so the second sum supplies coordinates of the same weighted product target in that range. At the 14/14 budget, fixed balanced, adaptive Gaussian, and adaptive anchored designs obtain relative reconstruction errors 0.045307, 0.008676, and 0.006933, respectively (Figure 9). The benefit is matrix fidelity, not guaranteed task-accuracy improvement. We explicitly report the additional projection broadcast, second secure sum, synchronization, and zero tolerance to post-Stage-I dropout.

4. Communication accounting. We separate native logical traffic, common BF16 payload, and serialized TCP traffic. Under common BF16 accounting, Table 19 reports 10.04 MB/client-round for SAGE 14/20 versus 23.97 MB for FLoRA and 19.61 MB for FSLoRA. Compact transposed 14/14 requires 8.40 MB; matched direct 14/14 requires 7.40 MB. The six completion rows in 14/20 are counted although they do not change decoded coordinates in exact arithmetic; 14/14 removes this overhead. Logical payload is not conflated with encoded Flower traffic.

5. Complete and residual updates. Section 2.1 and Appendix F.1 distinguish complete updates (Pt=0P^t=0) from residual updates (Pt=MtP^t=M^t). They share a measurement interface but have different optimization trajectories. A complete adapter may have rank at most rir_i, while its displacement from the global adapter can have rank ri+kr_i+k. Appendix C.9 analyzes the two modes separately. Some complete-update runs could not maintain eight measured directions for all 100 rounds, whereas the corresponding residual-update runs completed all rounds. These results clarify the numerical-rank constraint rather than implying equivalent behavior across update modes.

The revision distinguishes formal guarantees, implementation evidence, and measured system costs. SAGE integrates heterogeneous paired-product measurements with a same-cohort sequential SecAgg interface; it does not claim uniformly superior accuracy or lower overhead.

Thank you again for your constructive feedback. We have revised the manuscript and provided detailed responses to your concerns, particularly regarding the security assumptions, implementation scope, and protocol trade-offs. If you have an opportunity to review our response and revisions, we would appreciate hearing whether any concerns remain unresolved or require further clarification. We would be happy to address them.

I appreciate the author's point-by-point clarification. My concerns have been addressed, and I have no further questions. I raise my score to 3.

Thank you for your careful review and thoughtful reassessment. We are pleased that our clarifications have addressed your concerns, and we sincerely appreciate your constructive feedback and the time you devoted to reviewing our work.

Reviewer_VSyFOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets federated LoRA with independently trained, unequal-rank adapters under additive secure aggregation (SecAgg). Clients compute linear sketches of their locally paired products (DiΩ,ΨDi)(D_iΩ, ΨD_i) without forming dense updates. These are aggregated in two sequential SecAgg stages: the right probe is anchored on the global adapter, and the left projection is derived from the Stage-I range. The server then reconstructs a rank-k adapter.

Critical strengths

Writing is clear and easy to follow. The scope of the security claim is honest: the authors say SecAgg is not DP, the inter-stage binding has zero dropout tolerance, and the evaluated Flower reference checks the binding only after unmasking (App. E.2).

Critical weaknesses
  • The method is FedASK's non-DP pipeline plus SecAgg, and the FedASK comparison uses its DP variant. FedASK already has clients upload B_k(A_kΩ), QR the sum, upload the Q-projected product, and return B = QUΣ^{1/2}, A = Σ^{1/2}Vᵀ. That is Eqs. 9–14 here. FedASK also has a non-DP mode (acknowledged on p.25). Both of its stages are plain sums, so additive masking applies directly. The main new element, the anchored Ω, does not change accuracy: random bilateral sketching gets 76.36 vs 76.33 (Table 13). The 0.63–0.73-point gains over "FedASK" (Table 17) compare against a row identical to "FedASK (DP)" at ε=8 in Table 14 (75.64±0.40; also the same 51.9 s server time, Table 15). So they do not isolate anything SAGE adds.

  • The results do not show that product consistency is what SecAgg needs. Eq. 2 shows that masking before local multiplication fails. But FedIT averages B and A separately, so additive masks cancel on each sum. FedRot-LoRA aligns client-side (Tables 3–4) and then averages. Both are SecAgg-compatible as described. In Table 1, FedIT (76.78) and FedRot (76.63) beat SAGE+SecAgg (76.37) at 8.12 vs 14.58 GiB (Table 10). On GLUE, FedIT, zero-padding and FedRot all beat it (Table 2). FedRot leads by 2.64 points under heavy-tail ranks (Table 8), where zero-padding is missing. The abstract compares instead against FlexLoRA and against FLoRA/FSLoRA. Those two lose ~15 points on HellaSwag despite FLoRA's exact aggregation (Table 1), which suggests a tuning problem; all methods share one learning rate. The 67.3%/60.1% savings also mix BF16 sketches with FP32 baselines. From the scalar counts on p.31, I get 58%/49% at equal precision, and SAGE sends 2.3× FedIT's scalars.

Other Issues

  • The SecAgg reference encodes at 24-bit quantization mod 2^32 (App. E.2), but Table 14 lists the BF16 logical upload for "SAGE+SecAgg". Server time rises from 86.5 s to 11,701.5 s (Table 15).
  • Zero-padding equals FedIT exactly in Table 1 (homogeneous ranks) but not in Table 2 (MRPC 84.90 vs 86.12), although F.2 calls GLUE homogeneous-rank.
  • The abstract's GLUE gap to FlexLoRA is 0.43 from the table means, not 0.42. The abstract also omits FedIT, zero-padding and FedRot, which beat SAGE on GLUE.
  • PowerSGD (Vogels et al., NeurIPS 2019) uses the same warm-started, two-pass, sum-compatible range/coefficient scheme. It is worth citing.
Questions for authors
  1. clarification: Do you agree that FedIT, zero-padding and FedRot-LoRA can be run under additive SecAgg unchanged? If not, what breaks? If they can, is there a setting where E_fac measurably hurts? Reporting ‖E_fac‖/‖D̄‖ for FedIT in your runs, next to SAGE's 0.0032 relative error, would show whether the inconsistency matters here.

  2. suggestion: The six H rows (s_L=20 vs s_R=14) are discarded by the decoder but still transmitted and released. What are the cost and accuracy at s_L = s_R?

3 comments
AuthorsOpenReview ↗

Clarifying SAGE's Contribution and Matched Aggregation Evidence

We thank the reviewer for the careful assessment. The revised manuscript clarifies the scope of SAGE-LoRA, adds controlled comparisons, and separates product consistency from downstream task utility.

1. Relationship to FedASK and novelty. We agree that FedASK (NeurIPS 2025) already provides two-stage product sketching and a non-DP option. We therefore do not claim two-stage sketching, QR-based range extraction, or linear aggregation of fixed-projection product measurements as independent novelties. Appendix E.2 now states explicitly that, given the same weighted target, measured range, and truncation rule, FedASK and SAGE produce the same reconstruction in exact arithmetic.

SAGE instead specifies an interface for independently trained, heterogeneous-rank paired LoRA blocks: clients evaluate fixed-width product measurements through factor chains, and both measurements are protected by sequential SecAgg and bound to the same weighted survivor cohort. The public-adapter anchor incorporates existing global directions into range estimation. The contribution is this heterogeneous paired-product/SecAgg interface and its conditional sequential-confidentiality analysis, not a claim of universal accuracy superiority over FedASK.

We added a controlled receiver comparison. Table 24 shows that FedASK and SAGE reconstruct products differing by at most 4.66×10−84.66\times10^{-8} of the target norm. Replacing the Gaussian probe with the public-adapter anchor reduces relative error from 3.333×10−33.333\times10^{-3} to 2.374×10−32.374\times10^{-3} (28.8%) for either receiver (Tables 1 and 22). On OpenChat-7B, SAGE and non-DP FedASK also obtain nearly identical accuracy: 89.19% vs. 89.22% on commonsense and 47.56% vs. 47.61% on MMLU. These results isolate the probe benefit without attributing it exclusively to the SAGE decoder.

2. SecAgg compatibility versus product consistency. We agree that FedIT, zero-padding, and FedRot-LoRA can apply additive SecAgg to their respective factor-coordinate messages. Our claim is not that these methods cannot use SecAgg. Rather, SecAgg preserves the sum of submitted coordinates; it does not turn a factor-product aggregation rule into the weighted mean of locally paired products. FedRot-LoRA reduces rotational misalignment but its factor-product rule is not identically the product mean.

Appendix B.9 derives the FedIT discrepancy and a pairwise-dispersion bound. Table 6 reports relative discrepancy 0.19597 for unequal ranks at round 2, versus 0.00460 at common rank eight; at round 10 the values are 0.02437 and 0.00739. On the same round-ten FedIT products, Table 1 reports errors of 7.39×10−37.39\times10^{-3} for factor averaging, 6.93×10−36.93\times10^{-3} for SAGE, and 6.89×10−36.89\times10^{-3} for the rank-eight oracle. The discrepancy is measurable, particularly under unequal ranks, although its magnitude varies during training and does not alone determine task accuracy. We do not claim product consistency guarantees the highest downstream score: the revised GLUE results explicitly show FedIT, zero-padding, and FedRot-LoRA above SAGE.

3. Communication and matched widths. We now distinguish native-precision logical traffic, common-precision BF16 payload, and serialized TCP traffic. The abstract's native-representation savings are not presented as equal-precision compression. Under common BF16 accounting, Table 19 reports 10.04 MB/client-round for 14/20 SAGE versus 23.97 MB for FLoRA and 19.61 MB for FSLoRA (58.1% and 48.8% lower). Compact transposed 14/14 requires 8.40 MB; the matched direct 14/14 comparison requires 7.40 MB. We also report that SAGE exceeds compact rank-eight factor communication.

The six completion rows in 14/20 are included in traffic accounting although the exact decoder does not use them. The equal-width 14/14 experiments remove these rows. Table 18 reports 12.097 GiB over 100 rounds for transposed 14/14 and 14.580 GiB for 14/20.

4. Baseline results and related work. The revised manuscript reports non-DP FedASK explicitly and separates the direct-orientation matched comparison from the transposed benchmark. On GLUE, SAGE scores 81.22, exceeding FLoRA, FSLoRA, and FlexLoRA by 2.40, 1.37, and 0.43 points, respectively, while remaining below FedIT, zero-padding, and FedRot-LoRA. PowerSGD is now cited; Appendix B.9 distinguishes its distributed gradient compression from the locally paired LoRA-product measurements and same-cohort sequential interface studied here.

We believe the revision establishes a more precise claim: SAGE is a product-consistent, heterogeneous-rank measurement interface for aggregate-only SecAgg, with explicit reconstruction and system trade-offs, rather than a universally superior LoRA optimizer.

I thank the authors for their detailed response, It resolves most of my concerns. I've increased my score.

We sincerely thank you for your careful review, constructive feedback, and thoughtful reassessment. We are pleased that our response and revisions have addressed most of your concerns. Your comments helped us clarify SAGE-LoRA's contribution and its relationship to prior work, and improve the presentation of our theoretical and empirical results. We greatly appreciate the time and effort you devoted to reviewing our work and helping us improve the manuscript.

Comments on the paper

2 comments

We sincerely thank the reviewers and the Area Chairs for their constructive feedback. In response, we have substantially revised the manuscript to clarify the research motivation, distinguish SAGE-LoRA from prior aggregation and sketching methods, strengthen the theoretical analysis, and provide more direct evidence for its design choices. We would like to highlight the central contribution and the main improvements.

1. A distinct aggregation interface for heterogeneous federated LoRA.

The central motivation of SAGE is not to rediscover the factor-averaging inconsistency, which has been studied by FedEx-LoRA, LoRA-FAIR, Te-LoRA, FedRot-LoRA, and other works. Rather, we address the joint requirements of independently trained heterogeneous LoRA factors, product-consistent aggregation, fixed-width communication, and additive secure aggregation.

Existing approaches follow different design paths: residual correction recovers product averages through additional statistics; factor-alignment methods reduce coordinate misalignment; stacking and explicit product aggregation preserve local pairing but introduce larger intermediate representations or server-side processing; and shared-factor methods constrain updates to common subspaces. SAGE provides a complementary design that evaluates linear measurements of locally paired products through factor chains, without materializing dense client-side products or requiring common factor coordinates.

2. Clarifying the distinction from FedASK.

We acknowledge that FedASK (NeurIPS 2025) already introduces two-stage product sketching and supports a non-DP update option. We do not claim either operation as an independent novelty. SAGE instead develops their integration with heterogeneous paired-factor adaptation, public-adapter-anchored range estimation, fixed-rank reconstruction, and a same-weighted-cohort sequential SecAgg interface.

The public-adapter anchor incorporates existing global directions into the measurement space, while complementary directions capture new product variation. The first aggregate identifies the current product range, and the second measures its coordinates for the same weighted target. This design separates local adaptation rank, measurement width, and returned rank. We further provide a conditional sequential aggregate-confidentiality analysis for the resulting protocol. The matched FedASK/SAGE receiver experiments clarify which reconstruction benefits arise from probe geometry and which properties belong to the integrated protocol design.

3. Theoretical foundations.

The revised manuscript establishes exact aggregation of local product measurements, an orthogonal reconstruction-error decomposition separating missed-range and rank-truncation errors, and a missed-range bound for public-adapter anchoring. It further derives a time-averaged stationarity guarantee under explicit assumptions on federated direction bias, variance, and reconstruction distortion. Proposition E.1 analyzes sequential aggregate confidentiality under the stated SecAgg simulation, composition, and pre-release authentication assumptions. These results connect the aggregation interface, reconstruction mechanism, optimization behavior, and security requirements within a unified analysis.

4. Stronger empirical validation and transparent trade-offs.

We expanded the evaluation to directly examine product-aggregation discrepancy, matched product reconstruction, probe geometry, heterogeneous-rank adaptation, task utility, communication, and protocol execution. The revised experiments show that public-adapter anchoring improves reconstruction under matched measurement budgets and that the resulting rank-eight reconstruction approaches the corresponding low-rank reference. Evaluations on Qwen2.5-1.5B-Instruct and OpenChat-7B demonstrate competitive utility under the evaluated settings. We also report the costs of sequential aggregation, the limitations under heterogeneous ranks, and the distinction between Flower utility evaluation and the separate authenticated gRPC release implementation.

Importantly, we do not claim that SAGE universally outperforms existing LoRA aggregation methods, that two-stage sketching is unique to our work, or that SecAgg provides differential privacy. The revised manuscript explicitly identifies the assumptions, implementation boundaries, and communication and availability costs.

We believe these revisions establish SAGE-LoRA as a technically grounded contribution: a product-consistent, heterogeneous-rank measurement and reconstruction framework integrated with additive secure aggregation, supported by theoretical characterization and systematic empirical evaluation. We appreciate the reviewers' feedback, which helped us sharpen the contribution and improve the rigor and clarity of the paper.

Area_Chair_8MdHOpenReview ↗

Discussion Phase

As the discussion period is rapidly coming to a close (Nov 18 AoE), I would like to remind the reviewers to please participate in discussion and acknowledge any comments posted by the authors. A secondary, important reminder: full disclosure of any AI usage is required and the reviewer must include their own human-written assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.