acceptodds
Under review as a conference paper at ICLR 2027

Shared Prompt Carriers with Bounded Task Residuals for Federated Continual Learning

Abstract

Federated continual learning (FCL) requires clients to learn from evolving and heterogeneous tasks while transferring knowledge without sharing data. Prompt-based adaptation is parameter-efficient, but fully shared prompts can cause task interference, whereas isolated prompts limit reuse of common structure. We introduce FedSCR, a prompt parameterization built on a shared carrier with bounded low-rank task residuals. The shared carrier remains trainable across clients, while each task retains a compact offset whose norm is bounded relative to the carrier norm captured at warmup. Experiments on ImageNet-R and DomainNet show competitive accuracy and transfer compared with representative federated continual learners under heterogeneous task arrivals. Analyses indicate that most of the accuracy observed in our experiments is already present in the shared carrier, while residuals provide compact task state for controlled retention. FedSCR offers a simple way to balance cross-client knowledge transfer and controlled task retention in FCL.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
federated learning, continual learning, prompt tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_q86hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

FedSCR adapts a frozen pretrained vision transformer through a shared prompt carrier and bounded low-rank residuals associated with client-task slots. Each residual uses a carrier norm captured after warmup, receives two adaptation exposures, and is then frozen under the ordinary stream. Owner-bound updates avoid dilution of slot-specific state. A separate anchor-recovery branch adjusts old residuals when a carrier-drift condition is met before a revisit. Experiments use slot-aware ImageNet-R and DomainNet protocols, with carrier-only and lifecycle ablations, a controlled drift-revisit test, and resource accounting.

Critical strengths
  1. The paper is unusually explicit about its evaluation assumptions, including task-slot metadata, class masks, single-owner residual updates, and the distinction between tensor payload and measured network communication. This transparency makes the results easier to assess.

  2. The shared-carrier design achieves competitive accuracy with relatively compact trainable state. The carrier-only control, paired differences, and per-seed results provide useful evidence about which parts of the design account for performance.

Critical weaknesses
  1. The reported evidence does not establish a consequential benefit from the bounded residual mechanism that distinguishes the method. In Table 2, full FedSCR reaches 86.874% AIA versus 86.784% for the shared carrier alone. Appendix C, Table 10 gives the paired improvement as only 0.091 percentage points, with a 95% paired interval of [-0.236, +0.417]. The corresponding BT values in Table 2 are almost identical (1.358 versus 1.357). The reported intervals for removing warmup, using an unbounded rank-one residual, and diluting owner updates also cross zero. These results do not prove that the mechanisms are useless, but they leave their practical advantage unresolved. 
  2. The ordinary main protocols follow the no-trigger path, so they do not demonstrate the usefulness of the anchor-recovery mechanism during natural learning. The positive recovery result instead comes from a prescribed 10% earlier-task-gradient-aligned carrier perturbation, followed by direct application of recovery (Section 4.4 and Appendix C.3). That test does not establish whether the calibrated trigger detects harmful drift, avoids unnecessary interventions, or improves retention over a naturally evolving long stream. Every nonempty residual update group has exactly one owner, and all five clients participate synchronously for only 15 rounds. The multi-contributor factor-aggregation extension is explicitly unevaluated. Consequently, the empirical evidence does not yet establish robustness to unknown task identity, realistic participation, or prolonged task accumulation.
Questions for authors
  1. Under matched seeds, rank, tuning budget, and optimization, what is the gain of bounded rank-two residuals over both a carrier-only model and unbounded rank-two residuals? Is the advantage reproduced on DomainNet?

  2. Can you demonstrate the complete trigger-and-recovery pipeline on naturally induced carrier drift, reporting trigger frequency, false interventions, earlier-task accuracy, and comparison with always-on recovery? The current constructed perturbation test does not evaluate the trigger itself.

  3. How much of the improvement remains without oracle client-task output-block selection, or under a substantially longer stream with partial participation? Please distinguish evidence for the current slot-aware setting from claims about broader FCL deployment.

2 comments

Thank you for recognizing the shared-carrier results and explicit evaluation setting. We address the residual controls, natural recovery behavior and inference without oracle output-block selection below.

  1. Matched rank-two comparison

To assess the contribution of task residuals and their amplitude bound, we compare bounded rank-two residuals with a carrier-only model and a same-rank unbounded control. The unbounded control removes radial saturation while matching rank, local gain, initialization, data order, optimization, and lifecycle. Both residual designs use the same fixed training configuration without separate retuning. On ImageNet-R:

DesignAIA (%)FM (lower better)BT (higher better)
Bounded rank two86.874-0.3791.358
Unbounded rank two86.689-0.3831.525
Carrier only86.784-0.3611.357

Paired AIA differences are +0.186 pp over unbounded (95% paired-t interval [-0.187, +0.558]) and +0.091 pp over carrier only [-0.236, +0.417]. The corresponding warmup and owner-update ablation differences are +0.083 pp [-0.185, +0.352] and +0.154 pp [-0.064, +0.372]. The close AIA values indicate that the shared learning path supplies much of the observed performance, while the residual adds an explicit budget for task-specific state. Freezing preserves the learned task offset, warmup fixes its scale reference, and ownership governs its writes during adaptation. These roles distinguish the state-management design from its incremental accuracy contribution in this comparison.

A separate paired DomainNet control study gives the following means ± sample SD:

DesignAIA (%)FAA (%)
Bounded rank two76.092 ± 0.59376.133 ± 0.207
Unbounded rank two76.157 ± 0.61776.223 ± 0.024
Carrier only76.152 ± 0.67876.217 ± 0.085

The paired AIA differences are -0.065/-0.060 pp over unbounded/carrier only. The shared carrier attains similar AIA. Residual magnitudes are normalized by the carrier norm captured after each task's warmup. Their logged maxima are 0.094-0.095 with bounding and 2.471-2.557 without saturation. The unbounded model therefore uses substantially larger task offsets while attaining similar AIA, making the bound's control of task-state magnitude directly observable.

  1. Natural trigger frequency and always-on recovery

In a 36-round ImageNet-R stream revisiting five fixed tasks under ordinary synchronous, full-participation updates, AUTO triggered 0 times across 18 eligible rounds; AUTO and OFF both reached 86.822% earlier-task accuracy. Always-on recovery executed twice and reached 86.635%. Forced recovery from identical OFF checkpoints changed earlier-task accuracy by -0.020/+0.032 pp.

AUTO followed the same trajectory as OFF because the two-consecutive-exceedance condition was never met. ALWAYS measures the cumulative effect of intervening at both revisit boundaries, while forced recovery from identical OFF checkpoints isolates the immediate effect of each correction. The small immediate changes show why the intervention decision matters alongside the correction rule. The trigger uses carrier movement with a threshold fixed from a separate calibration run; test accuracy is used to evaluate outcomes. Appendix C.6 gives the full comparison.

The separate automatic-path test under amplified federated updates produced a -0.018 pp final earlier-task difference. The paper's +1.328 pp result directly invokes recovery after a prescribed gradient-aligned perturbation. Old-task gradients construct that perturbation; recovery uses saved prompt targets. This controlled result demonstrates correction through old-task residual factors while the shared carrier remains fixed. The natural stream evaluates when the automatic policy intervenes under ordinary updates.

  1. Removing oracle output-block selection and deployment scope

We evaluated inference without sample client/task/output-block IDs, keeping the trained models unchanged. Image-only readout uses all seen prompt rows and trained output blocks, merging repeated physical classes by maximum logit. ImageNet-R stream AIA is:

ReadoutFedSCR (%)POWDER (%)
Slot-aware86.75185.793
Raw image-only73.91875.333

Expanding the candidate outputs changes both absolute accuracy and the method comparison: FedSCR's AIA difference from POWDER changes from +0.957 pp under slot-aware readout to -1.415 pp under raw image-only readout. To understand this change, we examined errors at frozen endpoint checkpoints. Even when the correct task head predicts the right class, a higher score for a wrong class from another head can change the final prediction when all blocks compete.

We therefore evaluate an optional calibration that learns a temperature and offset for each task head before merging scores by physical class. This preserves class rankings within each head while adjusting relative scores across heads, directly targeting the competition identified above. The backbone, prompts, residuals, and classifier remain frozen, and all three methods receive the same calibration procedure. The separate endpoint FAA results are:

MethodRaw image-only FAA (%)+ Calibration (%)
FedSCR72.02978.056
Carrier only72.19077.795
POWDER73.56677.993

Calibration raises FedSCR's endpoint FAA by 6.027 pp with representations fixed, showing the value of aligning scores across heads. All three methods benefit, supporting comparison against identically calibrated baselines. They use the same training-only data split and calibration budget; training-side validation selects the calibration, and test data are excluded from fitting and selection. This optional readout extension requires historical training images. The calibrated readout supports inference without output-block IDs under the original synchronous, full-participation training protocol.

Reviewer_v24FOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

FedSCR combines a continually trained shared prompt carrier with rank-two residuals for individual client–task slots. Residual magnitudes are bounded relative to a captured carrier norm; each residual undergoes finite adaptation before freezing, with conditional anchor recovery after carrier drift. Experiments compare seven methods on ImageNet-R and DomainNet under task-aware federated protocols, supplemented by carrier-only controls, paired analyses, resource accounting, and an artificial drift–revisit test.

Critical strengths
  1. The parameterization provides explicit, compact task state. Rank and amplitude bounds, fixed reference scales, and ownership rules make residual storage and updates well defined. This is a practical contribution for controlling task-specific state while preserving a shared learning path.
  2. The evaluation contains unusually informative controls. Matched protocols, per-seed results, carrier-only ablations, and paired intervals make the contribution assessable. FedSCR improves mean AIA over POWDER by approximately 1.03 points on ImageNet-R and 0.78 points on DomainNet, supporting the effectiveness of the overall shared-carrier implementation within the stated setting.
Critical weaknesses
  1. The proposed residual mechanism contributes little demonstrated benefit. Full FedSCR exceeds the carrier-only control by just 0.091 AIA points, with a paired 95% interval of [-0.236, 0.417] (Table 10). Warmup and owner-update ablations are similarly unresolved, and the unbounded control changes rank as well as the bound. Most observed performance is therefore explained by the simpler carrier, leaving the necessity of the paper's principal additions unestablished.
  2. Recovery is validated only under a constructed intervention. Main experiments never trigger recovery; the stress test directly applies it after a prescribed 10% drift aligned with earlier-task gradients (Sections 4.1 and C.3). This demonstrates conditional correction, but does not validate the proposed trigger-and-recovery procedure under ordinary federated updates. That distinction matters because recovery supplies the clearest evidence that residual state improves retention.
Questions for authors
  1. Evidence needed for assessment: Do paired carrier-only and same-rank unbounded controls on DomainNet or the alternative task construction demonstrate a reproducible retention or accuracy benefit from bounded residuals?
  2. Recovery clarification: When are anchors captured relative to drift, how is the trigger threshold calibrated, and does the complete automatic procedure improve retention under naturally occurring carrier updates without using old-task gradients to construct the perturbation?
2 comments

Thank you for identifying compact task state and the shared learning path as practical contributions. We have added a same-rank control, a DomainNet matched comparison and a natural-update recovery diagnostic to clarify what each component contributes.

  1. Same-rank and carrier-only controls

The new unbounded control retains rank two, local gain, initialization, data order, optimization and lifecycle, removing only radial saturation. Both arms use the same fixed training settings. On ImageNet-R:

DesignAIA (%)FM (lower better)BT (higher better)
Bounded rank two86.874-0.3791.358
Unbounded rank two86.689-0.3831.525
Carrier only86.784-0.3611.357

Paired AIA differences are +0.186 pp over unbounded (95% paired-t interval [-0.187, +0.558]) and +0.091 pp over carrier only [-0.236, +0.417]. Removing warmup gives a full-minus-control difference of +0.083 pp [-0.185, +0.352]; diluting owner updates gives +0.154 pp [-0.064, +0.372]. The close AIA values place the shared carrier at the center of the observed performance; the residual mechanisms make task-state budgets and updates explicit. The revised Table 2 and Appendix C.3 give the full controls. Warmup establishes the fixed scale reference, and ownership preserves each task's updates during its adaptation period.

A separate paired DomainNet control study gives the following means ± sample SD:

DesignAIA (%)FAA (%)
Bounded rank two76.092 ± 0.59376.133 ± 0.207
Unbounded rank two76.157 ± 0.61776.223 ± 0.024
Carrier only76.152 ± 0.67876.217 ± 0.085

The paired AIA differences are -0.065/-0.060 pp over unbounded/carrier only. The shared carrier already attains a similar AIA in this comparison. Logged normalized residual maxima are 0.094-0.095 with bounding and 2.471-2.557 without saturation. The sharp amplitude difference alongside similar AIA identifies control of task-state magnitude as the clearest observed effect of saturation here.

  1. Recovery under ordinary carrier updates

We evaluated OFF, AUTO and always-on recovery in a 36-round ImageNet-R stream with ordinary federated updates. AUTO triggered 0 times across 18 eligible rounds; AUTO and OFF both reached 86.822% earlier-task accuracy. Always-on recovery executed twice and reached 86.635%. Applying recovery to identical OFF checkpoints changed earlier-task accuracy by -0.020/+0.032 pp at the revisit boundaries. The always-on comparison motivates evaluating the intervention decision alongside the correction rule. Appendix C.6 reports these natural-stream observations.

The separate automatic-path test under amplified federated updates produced a -0.018 pp final earlier-task difference. The paper's +1.328 pp gain comes from directly invoking recovery after the prescribed perturbation. Old-task gradients construct that perturbation; recovery itself optimizes saved prompt targets. The controlled gain shows that stored task targets can guide correction after the prescribed perturbation; the natural stream characterizes when the automatic policy intervenes.

  1. Anchor timing and threshold calibration

In the ordinary stream, anchors save effective prompts after aggregation, before subsequent monitoring and revisits. In the controlled test, the target is saved before the added perturbation. The trigger uses the 95th percentile of carrier-movement observations from a separate calibration run and requires two consecutive threshold exceedances. Recovery adjusts old-task residual factors toward the saved targets while fixing the carrier, classifier and backbone.

The revised appendix records the full schedules for the natural and amplified-update tests, and Eq. 8 now matches the implemented per-family MSE. The recovery experiments use the experimental implementation; the original uploaded code supplement omits this branch.

Thanks for the responses. My concerns have been resolved. Therefore, I maintain my positive score.

Reviewer_F8H6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies how to balance knowledge sharing and task retention in federated continual learning (FCL). It introduces FedSCR, which uses a shared carrier with bounded low-rank residuals for individual tasks. The carrier is continuously updated across clients, while the residuals are trained for a limited period and then frozen. The method is evaluated on ImageNet-R and DomainNet, showing improvements in accuracy and knowledge transfer over existing FCL methods.

Critical strengths
  1. Competitive performance across benchmarks.
  • FedSCR achieves the highest AIA on both ImageNet-R and DomainNet among the compared methods. It also shows improvements in forward transfer, suggesting that the shared carrier helps transfer knowledge to new tasks.
Critical weaknesses

1. Unclear contribution of task residuals to task retention.

  • FedSCR combines a shared carrier with bounded low-rank task residuals to balance cross-client knowledge transfer and task retention. However, even after the residuals are frozen, the effective prompts for earlier tasks continue to change as the shared carrier changes, making it unclear how much freezing contributes to retaining earlier-task performance. Moreover, the ablation results in Table 2 show little difference between Full FedSCR and the Shared carrier only variant, including forgetting and transfer metrics. Overall, the experimental evidence does not sufficiently show the contribution of the proposed method.

2. Limited methodological novelty.

  • The use of shared and task-specific prompts, along with low-rank adaptation, has already been studied in prior works. FedSCR mainly changes how task residuals are constrained and managed, but it is unclear whether these changes provide a meaningful improvement over existing approaches.
Questions for authors
  1. Residual bounds under high heterogeneity.
  • FedSCR limits the size of task residuals relative to the shared carrier. However, clients or tasks with very different data distributions may need larger residuals to adapt effectively. How does this constraint balance reducing task interference and allowing sufficient task-specific adaptation?
  1. Separate residuals for similar tasks.
  • FedSCR uses separate residuals for each client–task pair, even when tasks are similar or share classes. Since similar tasks may benefit from sharing task-specific knowledge, why does FedSCR keep their residuals separate instead of allowing them to share information?
2 comments

Thank you for recognizing the benchmark results. FedSCR combines continuing cross-client sharing with compact task state whose rank, amplitude and update lifetime are explicit. The complete system improves AIA over POWDER by 1.028 percentage points (pp) on ImageNet-R and 0.779 pp on DomainNet.

  1. What freezing preserves and what the residual contributes

Freezing preserves the task-specific offset relative to the shared carrier. For each task, the effective prompt is X = S + R. During ordinary training after freezing, R remains fixed while S continues learning, so delta X = delta S. Each task therefore retains its learned specialization while continuing to receive cross-client updates. Later tasks update the common carrier and their own eligible residuals; they leave the earlier task's offset intact.

The fixed reference scale is important here. Both the residual factors and the carrier norm captured after warmup are held fixed once the row freezes. Consequently, subsequent changes in the carrier's magnitude cannot rescale the stored offset. This gives the task a stable parameter component alongside the evolving shared component.

To examine the residual's empirical contribution, we added an unbounded rank-two control that matches initialization, data order, optimization, and lifecycle. It removes radial saturation while preserving rank and local gain, with all other training settings fixed. On ImageNet-R:

DesignAIA (%)FM (lower better)BT (higher better)
Bounded rank two86.874-0.3791.358
Unbounded rank two86.689-0.3831.525
Carrier only86.784-0.3611.357

The paired AIA differences are +0.186 pp over unbounded (95% paired-t interval [-0.187, +0.558]) and +0.091 pp over carrier only [-0.236, +0.417]. The close results identify continuing shared adaptation as a major source of stream accuracy. The residual adds compact task-specific state with explicitly controlled capacity, amplitude, and update eligibility. Against dense offsets, the bounded rank-two design achieves +1.419 pp AIA [0.640, 2.199] with about 16 times fewer residual factors, illustrating its accuracy/capacity tradeoff. The revised Table 2 and Appendix C.3 report these controls.

The recovery experiment demonstrates a further use of this task-specific state: it provides an adjustment path for earlier tasks while the shared carrier is held fixed. After the prescribed 10% earlier-task-gradient-aligned carrier perturbation, directly applying anchor recovery raises earlier-task accuracy from 81.908% to 83.236% (+1.328 pp). Recovery adjusts old-task residual factors toward previously saved effective-prompt targets, keeping the carrier, classifier, and backbone fixed. Old-task training images define the stress-test perturbation; the correction itself uses the stored prompt targets. This controlled comparison shows how residual factors can support task-specific correction alongside a shared learning path.

  1. Contribution relative to prior work

FedSCR's contribution is the concrete coupling of continuing federated sharing with budgeted task state. CODA-Prompt supplies the image-query readout, DualPrompt separates shared and task-specific prompts, and NB-LoRA studies rank/norm constraints and reference-norm budgets. These precedents motivate the components used in FedSCR.

The specific design places the shared carrier and task residual within the same prompt-value, key, and attention-vector families. Each effective family therefore combines a common component that continues learning with a compact offset learned for the corresponding task. Rank controls the residual's capacity; the captured reference norm and amplitude budget control its magnitude; exposure counts and ownership govern when and by whom it can be updated.

These choices work together. Capturing the reference norm once after warmup gives each task a fixed budget even as later federated updates change the carrier. Finite adaptation then preserves the learned offset, while owner-bound updates preserve its task-specific update semantics. The residual can also be reopened for correction toward a saved effective-prompt target.

This organization differs from mixing complete task prompts. POWDER transfers knowledge through correlations between task prompts; FedSCR maintains a common carrier that continues receiving updates across clients and tasks. Its empirical value is reflected in the system-level AIA gains, the compact residual-versus-dense comparison, and the controlled recovery result described above.

  1. Adaptation under heterogeneity

The bound applies to the task-specific offset, while the shared carrier continues adapting to aggregated client data. Adaptation is therefore distributed between a continually learned shared component and a residual optimized from the task's own data. Each task learns its own residual direction and magnitude within the available budget.

The budget rho makes the tradeoff between specialization and sharing explicit. Increasing rho permits a stronger task-specific departure from the carrier; decreasing it places greater emphasis on the shared component. The bound is scaled by the carrier norm captured after the task's warmup, so its absolute magnitude is tied to the learned prompt scale. A common value of rho thus specifies a relative budget across tasks.

We selected this budget through validation. In the rank-two structural screen, rho=0.10 gave higher validation AIA than either rho=0.05 or rho=0.20, supporting an intermediate specialization budget for that setting. The selected value was then held fixed for the ImageNet-R and DomainNet evaluations. This makes the balance between task-specific adaptation and shared learning an explicit, validation-controlled design choice.

  1. Why similar tasks retain separate residuals

A residual row records a client-task pair's adaptation state, including its owner, captured scale, and exposure count. Sharing classes does not imply that two tasks have the same data distribution or have reached the same adaptation stage. For example, one task may already have completed adaptation while another task with overlapping classes has just arrived. Separate rows let the former retain its learned offset while the latter continues adapting.

Merging those residuals would couple their parameter updates and freezing schedules. Keeping them separate allows each task to follow its own learning history while preserving the common carrier as the cross-client sharing path.

Sharing also occurs during prompt composition. The image-derived query combines available effective prompt rows, allowing information from different tasks to contribute to prediction. Thus, task-specific write ownership coexists with shared computation: ownership determines which task may modify an offset, while the carrier and query-based composition determine how learned prompt information is reused.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.