acceptodds
Under review as a conference paper at ICLR 2027

FedNASC: Noise-Aware Semantic Control for Federated LoRA Adaptation

Abstract

Federated LoRA adaptation of pretrained Vision Transformers must distinguish genuine cross-client semantic variation from finite-sample prototype noise. This is difficult when local class support is small, because unstable prototypes can mimic genuine within-class domain modes. We propose FedNASC, a noise-aware semantic controller that explicitly estimates this instability from split-half prototype disagreement. Each client augments its class prototype with a split-half instability statistic, and the server forms a matched class-wise noise floor, subtracts it from observed dispersion, and maps the calibrated excess to a score controlling prototype-bank capacity, classifier anchoring, semantic regularization, and prototype-readout fusion. Under a fixed-map perturbation model, we show that the untruncated calibrated difference estimates population representation dispersion in expectation up to controlled noise-dependent terms. Population dispersion in turn determines the weighted Euclidean single-prototype consensus error and bounds its unit-norm counterpart. We further derive a stationarity bound for the implemented optimization and server pipeline with explicit implementation residuals. Across CIFAR-100, HAM10000, DomainNet, and NICO++, FedNASC outperforms the strongest of six federated adaptation baselines by 1.4-2.4 percentage points across all ten settings. Ablations show that noise calibration contributes most under label skew, whereas adaptive bank capacity contributes most under domain shift, consistent with the intended roles of the two mechanisms.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Federated Learning, Parameter-Efficient Fine-Tuning, Low-Rank Adaptation (LoRA), Prototype Learning, Personalized Federated Learning, Statistical Heterogeneity, Vision Transformers

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_p4NeOpenReview ↗
Rating
1
Confidence
3
Informativeness
1(−1)
Presentation
1(−1)
Soundness
1(−1)
Read review
Paper summary

This paper addresses a key challenge in federated LoRA adaptation of pretrained Vision Transformers: distinguishing genuine cross-client semantic variation from finite-sample prototype noise in cross-client class prototype dispersion. To address this issue, the authors propose FedNASC. The method estimates prototype instability by randomly splitting each client's samples within a class into two near-balanced subsets and measuring the split-half prototype disagreement. The server then constructs a matched class-wise noise floor and subtracts it from the observed dispersion. The calibrated excess dispersion is mapped to a control score. This score is used to adjust the prototype-bank capacity, classifier anchoring, semantic regularization, and prototype-readout fusion. The paper compares FedNASC with six federated adaptation baselines across ten settings on CIFAR-100, HAM10000, DomainNet, and NICO++. FedNASC achieves mean accuracy improvements of 1.39–2.44 percentage points over the strongest baseline in each setting. Functional ablations show that removing noise calibration leads to the largest performance drop under the label-skew setting on CIFAR-100. In contrast, restricting the model to a single prototype per class has the greatest impact under the domain-shift setting on DomainNet.

Critical strengths
  1. Clear value in distinguishing the sources of prototype dispersion: The paper highlights that genuine cross-client semantic variation and finite-sample prototype noise may lead to similar patterns of prototype dispersion, but they require different treatments. This distinction is important for deciding whether cross-client knowledge should be preserved or suppressed. It provides a clear motivation for noise-aware semantic control.
  2. Coherent connection between statistical estimation and subsequent model operations: The method estimates noise through split-half calibration. It then uses the calibrated excess to allocate prototype-bank capacity and adjust semantic operations. The theoretical analysis links population dispersion to the single-prototype consensus error. This provides a justification for preserving multiple representative directions. Together, these components form a relatively coherent design framework.
  3. Performance results and functional ablations are largely consistent with the proposed mechanisms: The paper reports the highest mean accuracy across all ten settings, with improvements of 1.39–2.44 percentage points over the strongest baseline. The functional ablations show that removing noise calibration has a larger impact on CIFAR-100, while restricting the model to a single prototype per class has a larger impact on DomainNet. These differences support the design motivations of the two mechanisms. Specifically, noise calibration helps address sparse class support, while multiple prototypes help preserve persistent domain-specific class structure.
Critical weaknesses
  1. The implementation source and method-specific configurations of HiLoRA are unclear, making it difficult to verify this key baseline comparison. The appendix states that all methods use the same backbone, data splits, and training budget within a unified codebase. However, it does not provide sufficient details about the implementation source of HiLoRA, LoRA rank settings at each tier, cascaded tier-wise optimization, LoRA-Subspace Adaptive Clustering, or cross-tier orthogonality constraints. Using a unified training budget does not replace the need to verify whether each baseline is faithfully implemented. HiLoRA is the strongest or one of the strongest competing methods in several settings. Therefore, this lack of information directly affects the credibility of the reported performance gains. The available information is sufficient to request clarification from the authors, but it does not justify any conclusion about result fabrication.
  2. The applicability of the core calibration interpretation to actual federated training still lacks direct evidence. The main estimation results rely on a fixed-map perturbation model. However, the proposed method trains the encoder and computes prototypes using the same local samples. The appendix introduces a representation-reuse remainder and corresponding stability bounds. Thus, the authors have not completely overlooked this issue. However, it remains unclear whether this remainder is sufficiently small in actual ViT-LoRA training. The learned-feature bootstrap diagnostics also perform resampling on fixed learned features. They do not directly examine data dependence during the full training process. In addition, the authors acknowledge that the current score cannot distinguish data heterogeneity from encoder drift. This limitation weakens the interpretation of the score as a measure of genuine cross-client semantic variation.
  3. Figure 1 lacks sufficient clarity and visual consistency, and its production process needs clarification. Figure 1 and Figure 2 show noticeable differences in font style, font size, line sharpness, and compression quality. Some small text and symbols are difficult to read. Figure 1 is intended to illustrate the core motivation of the paper. These presentation issues may prevent readers from clearly understanding the distinction between noise and semantic structure. The visual style may raise questions about possible AI-assisted generation. However, visual differences alone cannot establish how the figure was produced. This concern should be treated as an issue of presentation quality and production transparency.
Questions for authors
  1. How were the HiLoRA results obtained? Please clarify whether HiLoRA was evaluated using a public implementation, an independent reproduction, or another source. Please provide details on its hierarchical architecture, LoRA rank settings at each tier, cascaded tier-wise optimization, LoRA-Subspace Adaptive Clustering, and orthogonality regularization weights. Which components were preserved from the original method? Which components were modified to match the unified backbone or training budget? If available, please provide an anonymous link to the implementation, configuration files, and logs for each random seed. This information will directly affect my assessment of the main experimental comparisons.
  2. Is there evidence that the representation-reuse remainder and encoder drift do not dominate the class-wise score during actual training? Please clarify whether the bounds on the representation-reuse remainder in the appendix provide meaningful quantitative guarantees under the experimental settings. Are there any diagnostic results based on actual ViT-LoRA training trajectories, such as evaluations with independent samples, cross-fitting, or other analyses? If the current evidence only supports the fixed-map setting, please explicitly discuss this limitation. In particular, how does it affect the interpretation of the class-wise score and its control effectiveness during training?
  3. Was Figure 1 created using generative AI, and can the authors provide the original source files? Please clarify which tools were used to create Figure 1, whether any generative AI content was involved, and what verification steps were performed by the authors. If available, please provide an anonymous link to the editable source file, vector graphics, or plotting scripts. This would help verify the text, symbols, and relationships shown in the figure. If the original files are unavailable, please describe the figure creation and export process.
2 comments

I have read the authors’ responses and supplementary materials. I strongly criticize the authors’ handling of content verification, generative AI use, and disclosure concerning the central motivational figure. The current response has not resolved my serious concerns about the overall reliability of the paper.

In Figure 1, the Bird column for Client 3 contains a deer with antlers, a category mismatch the authors have acknowledged. The paper reports experiments on real image datasets, yet the figure uses generated images rather than corresponding examples from those datasets. The authors then failed to check whether even birds and deer were placed in the correct categories. Why does a column labelled Bird contain a deer with antlers? What exactly was checked during the claimed verification of the scientific content? This error should have been caught by the most basic element-by-element inspection, yet it remained in the figure explaining the paper’s central motivation. I find this unacceptable.

Scientific figures, formulas, symbols, icons, and layout are all parts of a paper and require the authors’ careful attention and responsibility. The authors cannot simply treat outputs from generative tools as scientifically adequate representations. An image containing an explicit category error passed the claimed verification and entered the formal submission. This exposes serious shortcomings in the review of AI-generated content and in research quality control. It raises substantial doubts for me about whether the authors carefully verified the experimental materials and reported results.

The original disclosure did not state that ChatGPT was used to generate the thumbnails and modify the entire Figure 1. These specific uses were disclosed only after the reviewer asked. The incomplete disclosure further undermines transparency, but the more serious issue is the clear gap between the inadequate verification and the claim that the scientific content had been checked. I strongly object to this approach to handling scientific content.

Accurately presenting scientific content, truthfully describing the involvement of tools, and carefully verifying the final submission are basic requirements of research integrity and author accountability. These problems have seriously undermined my trust in the authors’ verification statements and raised substantial doubts about the standards applied to reviewing the paper’s content and its overall reliability. A general statement that the authors “checked the content and take responsibility” is insufficient to restore that trust.

Based on the current manuscript, the authors’ responses, and the supplementary materials, I maintain my original rating and will not increase it. I leave the final decision to the AC’s overall assessment.

We thank the reviewer for the follow-up.

Figure 1. Before submission, we checked the intended calibration formula and notation against the manuscript and traced the arrows from client-side estimation to server-side aggregation. Our final check missed the misplaced thumbnails. We identified and reported them when we re-examined the figure for our earlier response. We chose schematic illustrations because details in the 32×32 CIFAR-100 images were difficult to see at the size used in Figure 1, and we wanted a consistent style across panels. The thumbnails illustrate generic categories and were used only in the figure. We will correct their placement, check each thumbnail against its label, and export the figure directly from the corrected Visio source using fonts consistent with Figure 2.

AI disclosure. Our original statement omitted the use of ChatGPT for Figure 1. It did not state or imply that the figure was produced without AI assistance, and it contained no inaccurate description of how AI tools were used. The ICLR 2027 author policy lists creating or modifying scientific figures and images under recommended disclosure, and the Program Chairs invited authors to provide or update disclosures during the discussion period. We will add the following wording to the disclosure:

We designed Figure 1 in Microsoft Visio and used ChatGPT to generate the schematic thumbnails and to restyle the entire figure after export from Visio. The thumbnails do not come from the evaluated datasets.

This update follows the process set by the Program Chairs. The omission concerns how a schematic figure was produced and has no bearing on the experimental data, code or reported results.

HiLoRA comparison (Q1). As detailed in our earlier response, we independently implemented HiLoRA and provided both configurations, per-seed results for all ten Table 1 settings, and logs for twelve additional runs using HiLoRA's original optimizer, learning rates and rank. The Table 1 configuration outperformed the original configuration in all four settings tested, by 2.24–7.48 percentage points, so Table 1 reports the stronger configuration. Anonymous implementation, configurations and results

Representation reuse and encoder drift (Q2). As detailed in our earlier response, the appendix bound gives no informative numerical guarantee on the fitted trajectories, and our diagnostics on actual ViT-LoRA training show that encoder differences can contribute to the dispersion remaining after calibration. The sampling correction still matters in practice: removing the noise floor reduced personalized accuracy by 1.09 percentage points on CIFAR-100 and 0.63 on DomainNet (Table 2).

In the original review, the concern about the credibility of the reported gains was based on Q1: the reviewer stated that the HiLoRA information "directly affects the credibility of the reported performance gains" and "will directly affect my assessment of the main experimental comparisons." We have provided that information, and the follow-up does not comment on it! We therefore ask directly: after examining the HiLoRA materials, did the reviewer find any discrepancy, such as a configuration that differs from the reported protocol or a rerun log that does not match its reported value? If so, could the reviewer identify the setting and value so we can address it directly? If a concern remains about the trajectory diagnostics in Q2, it would likewise help to know which result or interpretation is in question.

Reviewer_jmGPOpenReview ↗
Rating
3(−1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies class-level semantic reliability in federated ViT-LoRA adaptation. Clients upload class prototypes and split-half statistics. The server estimates a matched noise floor, νcν_c, subtracts it from the observed cross-client dispersion, dcd_c, and obtains calibrated excess dispersion, ece_c, and a score, scs_c. The score controls prototype-bank capacity, classifier anchoring, client-side semantic guidance, and prototype-readout fusion. The contributions include a split-half calibration procedure, a score-driven semantic controller, and a theoretical analysis. The experiments cover label skew on CIFAR-100 and HAM10000, domain shift on DomainNet, and context shift on NICO++. Table 1 reports the highest mean accuracy for FEDNASC in all ten settings, with improvements of 1.39 to 2.44 percentage points over the strongest baseline.

Critical strengths
  1. Cross-client prototype disagreement can reflect both semantic structure and finite-sample noise. The split-half statistic provides a simple and interpretable client-side estimate of this noise. This gives the central mechanism a clear motivation and a measurable target.

  2. The paper reports the highest mean accuracy across four benchmarks and ten settings, along with results before and after fine-tuning, participation-rate experiments, per-domain and per-context results, and functional ablations. The theoretical analysis connects calibrated dispersion to population dispersion, multi-prototype representation error, and the direction of score-controlled operations. Although the analysis relies on local assumptions, it provides a structured account of the method.

Critical weaknesses
  1. Some controller parameters, including the scarcity scale n0n_0, the anchoring strength η0\eta_0, and the readout parameter β\beta, are fixed across experiments. The robustness of these choices across datasets and distribution-shift settings, as well as possible interactions among parameters, remains unclear.
Questions for authors
  1. Could the authors report sensitivity analyses for the remaining fixed controller parameters, particularly , n0n_0, η0\eta_0, and β\beta, and clarify whether the reported results are robust across datasets and distribution-shift settings?
Reviewer_YqMeOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes FedNASC, a federated LoRA adaptation framework designed to distinguish genuine cross-client semantic variation from finite-sample noise in class prototypes. The key observation is that high cross-client prototype dispersion can arise either because different clients contain genuinely different within-class modes or because some clients have too few samples to estimate stable prototypes. FedNASC estimates prototype instability locally using split-half disagreement, constructs a class-specific noise floor at the server, subtracts this from the observed cross-client dispersion, and converts the residual into a semantic score. This score controls prototype-bank capacity, classifier anchoring, semantic regularization, and prototype-assisted inference.

The method is evaluated using pretrained ViTs with federated LoRA adaptation under label skew, domain shift, and context shift across CIFAR-100, HAM10000, DomainNet, and NICO++.

Critical strengths

The paper identifies a meaningful problem in prototype-based federated learning: whether observed prototype variation is sufficiently reliable to be interpreted as genuine semantic heterogeneity. The authors explicitly distinguish between true semantic variation and finite-sample sampling noise. Although the proposed noise-estimation procedure is relatively simple, the underlying idea is intuitive and practically useful.

Critical weaknesses
  1. The performance gain is fairly modest relative to the added complexity. FedNASC improves over the strongest baseline by about 1.4–2.4 percentage points, but adds prototype estimation, server-side calibration, adaptive prototype banks, extra semantic communication

  2. All experiments use the same ImageNet-21K-pretrained ViT-B/16 with rank-8 LoRA. So although the paper presents FedNASC as a general framework for federated LoRA adaptation, it is unclear whether the conclusions hold for other backbone architectures, model sizes, or LoRA configurations

Questions for authors

what if the split-half estimate itself is noisy because it is random split?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.