acceptodds
Under review as a conference paper at ICLR 2027

SCOUT: Measuring When Cross-Model Transfer Helps Federated Lightweight Learners

Abstract

Cross-model transfer can improve lightweight learners, yet the utility of an individual intervention can depend on receiver state and update scale. We study this dependence in federated learning through SCOUT, a receiver-centric matched-intervention protocol comparing a candidate-transfer branch with a matched NoTransfer control from the same pre-intervention state. Across 75 audited states, FL-normalized transfer has positive mean utility but negative observed utility in 21 states. Changing only the update scale reverses utility sign in 29/75 states (38.7%) and makes Fixed-step preferable in 24/75 states (32.0%). A retrospective statewise operator oracle adds mean bidirectional over the best fixed operator (22.2%). On 50 canonical states, eight pre-intervention features with Bayesian Ridge and regularized logistic models do not reliably predict FL-normalized utility or harm under seed-grouped held-out evaluation; relative operator preference is not tested. Five matched 20-round repeated-transfer policy pairs have positive final held-out effects, while one-step audits separately contain negative observed utilities. These results distinguish intervention utility, retrospective choice value, single-operator predictability, and trajectory-level outcome.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
federated learning, cross-model transfer, transferability, intervention utility, multimodal retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ib8vOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

Paper Summary:

The paper introduces SCOUT, a receiver-centric protocol for evaluating cross-model knowledge transfer in federated multimodal retrieval. Rather than assessing transfer methods solely by their average performance, SCOUT measures the utility of an individual transfer intervention by comparing matched Transfer and NoTransfer branches starting from the same pre-intervention state. The study investigates how transfer utility varies across receiver states and update scales, whether it can be predicted from pre-intervention evidence, and how one-step intervention effects relate to long-term training outcomes.

Contributions:

  1. A controlled protocol is developed to isolate the utility of cross-model transfer from ordinary federated learning progress.
  2. Experiments reveal that transfer utility depends on both receiver state and update scale. A retrospective statewise operator oracle quantifies the potential benefit of adaptive operator selection, without claiming a deployable selector has been learned.
  3. The predictive capability of pre-intervention features is evaluated under seed-grouped held-out testing, while one-step intervention utility is distinguished from repeated-transfer policy outcomes over longer training trajectories.

Validation:

Experiments are conducted on MS COCO and Flickr30K using a frozen CLIP ViT-L/14 foundation model and a CLIP ViT-B/32 lightweight receiver in a 20-client federated setting. Across 75 audited receiver states, FL-normalized transfer achieves positive mean utility but produces negative utility in 21 states. Changing only the update scale reverses the utility sign in 29 states (38.7%). A retrospective operator oracle improves mean utility by 22.2% over the best fixed operator. However, prediction using eight pre-intervention features yields weak held-out performance. In a separate 20-round evaluation, all five repeated-transfer policies achieve positive final held-out effects, with a mean improvement of 0.34 percentage points in Avg. R@1. These findings demonstrate that positive long-term policy outcomes can coexist with harmful individual transfer interventions, while the effectiveness of prospective intervention selection remains unestablished.

Critical strengths
  1. SCOUT compares matched Transfer and NoTransfer branches from the same pre-intervention state, helping isolate the effect of knowledge transfer from ordinary federated learning progress. This controlled design provides a more reliable basis for assessing when transfer is beneficial or harmful.
  2. The paper shows that transfer effectiveness varies across receiver states and update scales, including sign reversals in 29 of 75 audited states when only the update scale changes. This finding challenges the assumption that a fixed transfer strategy is uniformly beneficial and motivates more adaptive evaluation and selection methods.
Critical weaknesses
  1. Although SCOUT investigates whether pre-intervention features can predict transfer utility, the reported held-out results are weak. This limits the practical value of the proposed evaluation framework for identifying beneficial transfer interventions before applying them, weakening the paper’s central motivation for adaptive transfer.
  2. The evaluation focuses on two image–text retrieval datasets and a specific CLIP teacher–receiver configuration in a 20-client federated setting. The paper therefore provides limited evidence that its findings about state-dependent transfer utility and update-scale sensitivity generalize to other model architectures, modalities, or federated learning settings. This constrains the broader significance of the conclusions.
Questions for authors
  1. How does the proposed framework enable practical transfer selection given that the evaluated pre-intervention features fail to predict transfer utility reliably on held-out data?
  2. To what extent do the observed state-dependent transfer effects generalize beyond the specific CLIP teacher–receiver configuration and the two evaluated image–text retrieval datasets?
Reviewer_rUE3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCOUT, a matched-intervention protocol for measuring transfer utility at individual federated learner states. Experiments across 75 states show that two update-scale rules can produce utility gains with opposite signs. The paper also examines retrospective operator choice, prediction before intervention, and repeated transfer over 20 rounds.

Critical strengths
  1. The matched control separates transfer effects from ordinary federated training progress. Transfer and NoTransfer start from the same checkpoint and use the same downstream execution settings, including client selection and restored random states. This makes the comparison more informative than comparing consecutive rounds, where transfer effects would be mixed with ordinary optimization.
  2. The paper is careful about what each experiment supports. It does not treat the retrospective oracle as a deployable selector, prediction failure as proof of unpredictability, or positive terminal outcomes as evidence that harmful interventions caused later gains. Grouping prediction examples by seed also avoids splitting states from the same training trajectory across training and evaluation. These choices make the results easier to interpret.
Critical weaknesses
  1. The stability and practical importance of the main effects are unclear. The mean FL-normalized utility is 2.198×10−42.198\times10^{-4} MRR, and the operator oracle adds 4.88×10−54.88\times10^{-5}. The reported 22.2% improvement is relative to the small mean intervention gain, rather than overall retrieval performance.

    The authors appropriately describe the 21 harmful states and 29 sign reversals as descriptive observations. Still, uncertainty estimates are needed to assess whether these counts reflect stable differences between states and operators. Paired query bootstrapping is already used for the prediction targets, but corresponding uncertainty is not reported for the main utility and sign-reversal results. The oracle also selects and evaluates the better operator using the same query outcomes. This describes the best choice on those queries, but does not show that the preference holds on independent queries.

  2. The experiments offer limited insight beyond sensitivity to update scale. Both operators use the same transfer direction and differ only in scale. The study uses one teacher–receiver pair, one relation construction, and one non-IID configuration. It shows that the two scaling rules behave differently at the audited states, but does not establish whether this behavior is specific to foundation-model transfer or also occurs with ordinary receiver-loss corrections.

    The prediction experiment uses 50 states from ten seed groups, eight selected features, and two simple model families. Quantities such as relation-gradient norm, gradient compatibility, and applied correction scale are excluded. The authors avoid broad claims about unpredictability, but the negative prediction result still tells us relatively little about what determines transfer utility. A scale sweep and a matched receiver-only correction would help explain the observed behavior without requiring a deployable selector.

Questions for authors
  1. How many harmful states and operator sign reversals remain stable under paired query resampling or a small practical-effect threshold? Please specify the bootstrap unit and explain how dependence between queries sharing an image is handled.
  2. Does the oracle advantage persist when operator preference is determined on one query subset and evaluated on another? Please also report uncertainty across complete seed groups.
  3. Do similar statewise reversals occur for a norm-matched update using the receiver’s ordinary retrieval gradient? How do the results change across a wider range of correction scales?
  4. Please specify the downstream replay horizon, trainable receiver parameters, local training settings, and separation between calibration, foundation-selection, intervention-evaluation, and final-test data. These details are needed to reproduce the intervention comparison and understand its scope.
Reviewer_JMYKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SCOUT, a receiver-centric matched-intervention protocol for auditing cross-model transfer utility in federated multimodal retrieval. SCOUT forks the same pre-intervention checkpoint into Transfer and NoTransfer branches and measures their difference in bidirectional MRR under matched downstream execution.

Critical strengths

The matched-intervention design is aligned with the research question.

The predictive evaluation includes appropriate safeguards against leakage.

The paper is well written, with a clear and easy-to-follow presentation.

Critical weaknesses

Gradient alignment concerns a local direction for the calibration loss and does not guarantee improved evaluation MRR. Since the operators differ only in update scale, sign reversals alone do not establish the role of teacher-provided knowledge.

Section 4.1 and Figure 2 primarily motivate MRR through improved measurement resolution. However, the metrics assign different values to rank positions and can produce opposite utility signs for the same ranking changes.

The definition of “foundation-induced margin increase,” the relationship between raw and safety-selected foundation states, and the computation of several historical evidence features remain ambiguous. Figure 1 provides a reasonable conceptual overview but cannot replace these execution details.

Lines 370–372 suggest that Fixed-step ties may arise from an inactive gate. However, the two operators share the gate, while Table 2 reports nonzero FL-normalized utility in all 75 states.

Questions for authors

Refer to the above text.

Comments on the paper

1 comment

Dear all,

We're the half-way through the discussion phase. The authors have provided their response to your comments. Some of you have started involving discussion, but all of us can take action now.

If the provided responses do not address (even part of) your concerns, this is your chance to raise it. Please engage with the authors as soon as possible, we need to keep the conversation going!

Thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.