- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes SAGE-LoRA, a functional-sketch framework for secure federated LoRA aggregation. The authors identified two long-standing issues in existing approaches: factor averaging is inconsistent with product averaging, and pre-factor masking introduces non-canceling cross terms. SAGE-LoRA addresses these issues by securely aggregating linear sketches of locally formed LoRA products instead of factor coordinates, enabling product-consistent aggregation with heterogeneous ranks. The paper provides theoretical analyses on aggregation correctness, reconstruction error, convergence, and confidentiality under SecAgg assumptions. Experiments on Qwen2.5-1.5B-Instruct demonstrate competitive performance and reduced communication overhead. Overall, the paper has several practical issues regarding deployment overhead, communication evaluation, security scope, and scalability.
Critical strengths
-
The paper proposed a systematic approach to solve the aggregation inconsistency in FedIT and security-interface inconsistency in additive secure aggregation.
-
The proposed approach performs linear sketching before masking, avoiding bilinear mask interaction terms while maintaining compatibility with additive SecAgg.
-
The paper combines theoretical analysis with experiments covering utility, communication efficiency, heterogeneous ranks, reconstruction fidelity, and security-utility trade-offs.
Critical weaknesses
-
The novelty of SAGE-LoRA is insufficiently established. The aggregation inconsistency illustrated in Fig. 1(a) has already been extensively investigated in prior works (e.g., FedEx-LoRA (ACL 2025), LoRA-FAIR (ICCV 2025), Te-LoRA (ICCV 2025), FedRot-LoRA (ICML 2026), etc.). More importantly, FedASK (NeurIPS 2025) already employs a two-stage product-sketching framework for privacy-preserving federated LoRA. (Q1)
-
The practical reliability and efficiency of the two-stage secure aggregation protocol remain insufficiently evaluated. (Q2-4)
-
The reported communication savings are partially confounded by heterogeneous numerical representations and precision settings. (Q5)
-
Weak security model, evaluation generality. (Q6-7)
Questions for authors
-
Given the long history of the aggregation inconsistency, what specific limitations remain unresolved, and how does SAGE-LoRA overcome them? For example, what are the essential technical differences and advantages of SAGE-LoRA beyond replacing DP with secure aggregation? Under what conditions does the proposed anchored, range-locked sketching provide measurable utility advantages over the other priors, especially FedASK? A more detailed methodological comparison and controlled experimental evaluation would strengthen the novelty claims.
-
The paper introduces two update modes, but the practical selection criteria remain unclear. What are their respective advantages and limitations in terms of convergence, computational cost, and adaptation performance?
-
How sensitive is SAGE-LoRA to the sketch dimensions? The reconstruction quality depends on the selected sketch widths s_R and s_L. Could the authors provide an ablation study showing how different sketch dimensions affect reconstruction error, downstream accuracy, and communication cost?
-
Compared with a matched one-stage secure aggregation protocol, what additional latency, communication, synchronization, and authentication overhead does the sequential design introduce? Although Table 15 reports aggregate execution times, could the authors provide stage-wise measurements and evaluate round-completion rates, waiting times, retries, and overall training throughput under different client dropout rates?
-
Are the reported communication advantages maintained under matched numerical precision? Could the authors provide BF16-to-BF16 or FP32-to-FP32 comparisons, in addition to the existing scalar-count analysis, and report actual transmitted bytes including numerical encoding, cryptographic metadata, and secure aggregation protocol overhead?
-
The paper explicitly limits its formal analysis to semi-honest aggregate confidentiality. What information about individual clients may still be inferred from repeated authorized aggregate releases, varying participating cohorts, or published global models? Could the authors provide a more concrete evaluation of these residual privacy risks and discuss the requirements for extending the protocol to stronger adversarial settings?
-
The evaluation is mainly conducted on Qwen2.5-1.5B-Instruct with rank-8 LoRA adapters. How does SAGE-LoRA scale to substantially larger models, more participating clients, and more heterogeneous computation and communication environments? Could the authors evaluate its memory, communication, and execution-time requirements under these settings?
6 comments
Thank you for the detailed response. Some key concerns still remain:
-
The novelty is still not clear. Can you explicitly indicate what remains unsolved in the previous works and how it motivates SAGE-LORA? What makes the previous works incapable in solving aggregation inconsistency and security-interface inconsistency, and what is the evidence that supports your argument?
-
The criteria for selecting the update mode (complete-update mode or residual mode) is still unclear. What criteria should practitioners use to select an update mode before training? Algorithm 2 still takes the update mode as an input, then what criteria should practitioners use to switch modes?
SAGE-LoRA's Novelty and Practical Update-Mode Selection
Thank you for these focused questions. We clarify SAGE's additional contribution and the criteria for choosing its update mode.
1. What remains unresolved, and what does SAGE add?
Prior work already addresses aggregation inconsistency in several settings. SAGE asks how to exploit the evolving public adapter to reconstruct heterogeneous product averages from fixed-width secure sums, with error control and consistent sequential execution.
Securely summing factors cancels masks correctly, but multiplying their means generally differs from averaging their products (Eq. 1; Table 6). Multiplying individually masked factors instead introduces the cross terms in Eq. 2. SAGE evaluates paired-product measurements locally before masking, so the secure sum measures the intended product average. Measurement aggregation is exact; rank-limited reconstruction remains approximate.
FedEx-LoRA and FLoRA recover product averages through residual correction and stacking; FlexLoRA compresses the product average. Alignment/refinement methods, including FedRot-LoRA, LoRA-FAIR and Te-LoRA, address factor mismatch. Fixed-factor/core methods make aggregation linear by restricting the trained subspaces. SAGE retains independently trainable factor pairs, permits unequal local ranks, and obtains a prescribed-rank return through fixed-width measurements without dense client products or common factor coordinates (Secs. 3.1-3.3).
FedASK is the closest counterpart: its two-stage construction, non-DP option and fixed-projection linear measurements already support this basic sketching route. SAGE's principal methodological addition is public-adapter-anchored measurement geometry with change-dependent reconstruction analysis. FedASK's random probe does not explicitly include the public adapter's row space. SAGE includes that space and samples complementary directions for new variation, without restricting local training to either subspace. For public adapter and target , with orthonormal containing and orthonormal spanning , Proposition C.3 gives Thus, missed-range error is controlled by the change from the public adapter: when is close to , this bound is small. The analysis separates this error from rank truncation.
At matched 14/14 width, rank-eight return and payload, anchoring lowers reconstruction error by 20.1% on identical client products (Sec. 5.3). The separate probe/receiver control (Table 1b) shows the benefit with both receivers, identifying measurement design as the source of improvement and its applicability to FedASK.
For the adaptive decoder, the second sum measures coordinates in the range selected by the first. SAGE specifies how both sums retain one weighted target: the same clients reuse their frozen products and weights. Proposition E.1 states the SecAgg and pre-release authentication assumptions for sequential aggregate confidentiality; Fig. 6 tests target mismatch. These conditions matter because linearity within each secure sum alone does not ensure consistency between the two sums. The contribution combines anchored reconstruction, its error analysis and an explicit secure execution for heterogeneous products.
2. How should practitioners choose the update mode?
Select by the local adaptation objective and resource budget (Sec. 2.1; Eq. 22):
-
Complete updates: choose this mode to jointly readapt the entire local adapter within rank . Clients initialize from and train both factors; the aggregate replaces the global adapter. If , the full global adapter is retained at initialization. If , this choice accepts truncation of the initial public adapter.
-
Residual updates: choose this mode when clients should retain the full public adapter during local training and devote their trainable rank to an additive correction. Clients freeze and train fresh rank- blocks around it. This is particularly relevant when , provided the device can store and evaluate the frozen rank- base as well as the trainable blocks. Retention here concerns the local base; the server still returns rank .
We recommend complete updates by default, as in our main experiments. The modes optimize different parameterizations: complete displacement has rank at most , versus for a fresh residual (Appendix C.9). These rank bounds do not establish an accuracy ranking.
Algorithm 2 takes the mode as input because it executes one round of the chosen parameterization. Select one mode before training and keep it fixed across the run and cohort. It has no automatic switching policy: numerical sketch rank below triggers abort, not a mode change. Both modes share the secure measurement interface.
Responses to Q1–Q7: Novelty, Security, and Experimental Validation
We thank the reviewer. We address the seven questions below.
Q1. Novelty beyond FedASK. We agree that FedASK (NeurIPS 2025) already provides two-stage product sketching and a non-DP option; we do not claim these operations as new. Appendix E.2 states that both produce the same reconstruction under an identical target, range, and truncation rule.
SAGE specifies a heterogeneous paired-product interface: independently trained client blocks are measured through fixed-width factor chains, with sequential SecAgg over the same weighted cohort. Its public-adapter anchor retains existing global directions. In a matched receiver comparison, reconstructed products differ by at most 4.66×10−8 of the target norm; anchoring reduces relative error from 3.333×10−3 to 2.374×10−3 (28.8%) for either receiver (Tables 22 and 24). On OpenChat-7B, SAGE and non-DP FedASK obtain 89.19%/89.22% commonsense and 47.56%/47.61% MMLU accuracy, indicating comparable utility. FedEx-LoRA's rank-eight correction followed by rank-eight truncation gives 0.00691 relative error, close to SAGE's 0.00693. Unequal-rank FedIT also shows a round-two discrepancy of 0.19597, versus 0.00460 at common rank eight (Table 6).
Q2. Complete versus residual updates. Section 2.1 and Appendix F.1 define complete mode as Pt=0 and residual mode as Pt=Mt. They share a measurement interface but differ in optimization trajectory. A complete adapter may have rank at most ri, while its displacement from the global adapter can have rank ri+k. Appendix C.9 analyzes the modes separately. Some complete-update runs could not maintain eight measured directions for all 100 rounds, whereas corresponding residual runs completed all rounds.
Q3. Sketch-dimension sensitivity. Figure 9 compares fixed and adaptive measurements. On identical round-ten products, fixed balanced, adaptive Gaussian, and adaptive anchored designs obtain relative errors 0.045307, 0.008676, and 0.006933. Their training-trajectory accuracies are 76.29%, 76.45%, and 76.26%. Thus, improved matrix reconstruction does not guarantee higher task accuracy. Compact 14/14 removes the six completion rows transmitted in 14/20.
Q4. Sequential protocol reliability and cost. Stage II authenticates the Stage-I survivor set and protected count before share release; any post-Stage-I dropout aborts the round. Appendix E.2 reports gRPC median stage durations of 0.445 s and 0.728 s. Appendix F.9 reports that two secure sums take 81.96 s versus 65.15 s for one sum in the tested 40-client single-host setup. These are measured costs, not a general dropout-rate model. We acknowledge the extra projection broadcast, second sum, synchronization, and zero post-Stage-I dropout tolerance. The second projection is derived from the first aggregate; a fixed one-stage sketch cannot adapt to that cohort-specific range.
Q5. Matched-precision communication. We separate native logical payload, common BF16 payload, and serialized TCP traffic. Under common BF16 accounting, SAGE 14/20 requires 10.04 MB/client-round versus 23.97 MB for FLoRA and 19.61 MB for FSLoRA. Compact transposed 14/14 requires 8.40 MB; matched direct 14/14 requires 7.40 MB. Rank-eight factor communication remains smaller.
Q6. Privacy and stronger adversaries. Section 2.2 and Appendix E consider a semi-honest server colluding with clients within the SecAgg threshold. Proposition E.1 requires auxiliary-input simulation, sequential composition, fresh stage randomness, and pre-release authentication of the same cohort and protected count. Aggregates and the global adapter remain observable. SecAgg is not differential privacy and does not prevent inference from released models or differencing across overlapping cohorts. The guarantee excludes malicious input substitution, poisoning, Sybil attacks, traffic analysis, side channels, and denial of service. Authentication binds roster and count, not the benignness of private inputs.
Q7. Model and cohort scaling. We evaluate OpenChat-3.5-0106 (7B), adapting 64 query/value matrices. SAGE obtains 89.19% commonsense and 47.56% MMLU accuracy; Appendix F.8 reports 14.25 GiB peak allocated GPU memory. Appendix F.9 evaluates 10, 20, and 40 concurrent clients; at 40 clients, two secure sums take 81.96 s in the tested single-host setup. These results show fixed-width messages independent of cumulative client rank, while coordination cost grows with cohort size.
SAGE thus provides product-consistent measurements for heterogeneous paired LoRA under sequential SecAgg, with explicit reconstruction, security, utility, and system-cost analyses.