acceptodds
Under review as a conference paper at ICLR 2027

JAM: Joint Allocation and Matching for Sample Sharing under Emerging Environments

Abstract

In edge deployments without cloud connectivity, clients encounter different new environments at different times, such as distribution shifts caused by lighting or terrain. Directing information about environments already encountered by some clients to those that have not yet encountered them can support early adaptation. With limited communication, however, each sender must decide whom to send to, how much to send, and what to send. Two challenges arise: class labels do not readily reveal environmental differences within a class, and the same sample has different value to different clients. We propose JAM (Joint Allocation and Matching), which measures differences using features from the common initial model rather than class labels and jointly decides whom to send to, how much to send, and what to send within the budget. Multi-seed experiments on three datasets show that, compared with random sharing, separate allocation and selection, sample selection, and prototype exchange, JAM can significantly improve deployment performance in new environments on datasets with identifiable within-class environmental deviations, without sacrificing average old-environment loss. Quantizing feature averages in JAM control metadata to 4-bit reduces communication while retaining the main adaptation gains.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
sample sharing and selection, collaborative edge learning, communication-efficient learning, federated learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_SpxTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes JAM for communication-constrained edge learning during cloud outages. JAM uses features extracted by the common pre-deployment model to construct per-class summaries and jointly determines which receiver to serve, how much budget to allocate, and which samples to transmit through a dynamic marginal-gain greedy strategy. Experiments on CIFAR-100, PACS, and Office-Home show clear gains on CIFAR-100 and PACS, while improvements on Office-Home are negligible; the paper also studies mechanism variants, robustness, statistical significance, and communication quantization.

Critical strengths
  1. The paper formulates a practically relevant joint allocation-and-selection problem in which the usefulness of a sample explicitly depends on the target receiver rather than only on the sender-side sample quality.
  2. The analysis provides a clear theoretical justification for jointly modeling receiver-sample pairs.
Critical weaknesses
  1. The main setting assumes fully connected clients, labeled post-deployment samples, a shared frozen feature extractor, and extensive exchange of per-class statistics. Moreover, Office-Home is explicitly modified to guarantee full class coverage from the first round. These assumptions may substantially simplify the practical edge-learning scenario.
  2. The proposed greedy optimization lacks an optimality guarantee and always fills the predefined sending dose.
  3. JAM provides significant improvements on CIFAR-100 and PACS, but its improvements on Office-Home are around 10^-3 and statistically insignificant, suggesting that the method is effective only when the assumed feature-summary signal is sufficiently strong.
  4. Summary SNR is useful for post-hoc explanation but has limited deployability as an applicability indicator. Its empirical computation uses environment identities and statistics from the full original training pool, whereas these quantities are unavailable to JAM during actual deployment.
Questions for authors

Please see Weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer SpxT (1/3) — Comment 1

Response to Reviewer SpxT (1/3)

Thank you for your clear review and for recognizing the formulation and theoretical motivation of joint allocation. Your comments on the deployment assumptions, the greedy procedure, the Office-Home result, and the use of SNR helped us clarify when the method applies. We added comparisons at the same sending dose under both BN settings and training comparisons on the original Office-Home stream. We address each concern below.

Comment 1. Communication, labels, the feature extractor, and first-round class coverage

We study how to use limited sample transmissions to improve adaptation to new environments when cloud synchronization is interrupted and clients can still communicate with one another. All methods use the same fully connected point-to-point network, candidate receivers, and sender budget. This lets us compare which images to send, to whom, and in what quantities. Clients exchange statistics to construct a common network reference and update their prediction models locally. They must complete the all-to-all exchange before selecting samples in each round. Our conclusions therefore apply to small fully connected networks that can complete this exchange in time.

The shared frozen feature extractor puts the clients' per-class feature averages in the same representation space so they can be compared directly. Clients continue training their local prediction models. Freezing the feature extractor for selection and fixing BN running statistics during adaptation are separate settings. Public class labels define the common classification task. Hidden environment information, such as species and capture locations, is used only for splitting and evaluation. Selection must determine which already observed images can complement a receiver's local data.

In the first round on Office-Home, each client receives one image for each of the 65 classes, so every class has an initial summary. All methods use the same data stream. The first-round class coverage still leaves each client without local data from its Deficit environment, so the adaptation problem with limited observations per class remains. We retain this controlled setting to examine what happens when each class has initial observations and subsequent arrivals are sparse.

In the additional experiments, JAM's whole-process Novel Loss is significantly lower than that of all three Need baselines at the same sending dose on class-balanced PACS and the five-superclass iWildCam task under both BN settings. See Section 2 and Tables 3–4 of our general response for the data construction and results. In the class-balanced PACS JPEG comparison with fixed BN running statistics, Random-JPEG sends 1 more image per sender per round after converting the control overhead using the mean JPEG size of the candidate pool. JAM-INT4-JPEG still achieves significantly lower whole-process Novel Loss. See the end of Section 3 of our general response for the budget calculation, transmission counts, and loss difference. These results support the comparison of selection methods under the shared conditions above. The revised manuscript will state the conditions explicitly: sharing labeled images must be authorized, and clients must be able to finish exchanging statistics before sample selection in each round.

AuthorsOpenReview ↗

Response to Reviewer SpxT (2/3) — Comments 2–3

Response to Reviewer SpxT (2/3)

Comment 2. Greedy optimality and the sending dose per round

Proposition 1 shows that the single-pair cost generally cannot be written as the sum of independent receiver and sample terms. This provides a structural reason to select them jointly. JAM uses a dynamic greedy heuristic. Each time it selects an image, it updates the receiver's projected feature average and selects the feasible pair with the largest current marginal gain. The greedy procedure does not guarantee a globally optimal set of pairs. We evaluate its performance through comparisons at the same sending dose and the pairing control.

The additional experiments test this directly. Tables 3–4 in our general response show that JAM achieves significantly lower whole-process Novel Loss than the per-receiver Need-Herding baseline on class-balanced PACS and the five-superclass iWildCam task under both BN settings. We also kept the shared images and each receiver's per-class receipt counts the same and permuted the recipient assignments. This permutation offset about 26.5%–34.1% of JAM's Novel Loss gain over random sharing. All five seeds favored JAM in each of the four pairing comparisons, and all four differences were statistically significant after Holm correction (Table 5 in our general response). These results support the value of joint selection and of the specific receiver–sample matches for adaptation.

Our comparisons at the same sending dose give each sender the same number of transmissions per effective communication round across methods. This separates the effect of sending more samples from the effect of selecting better samples. With the total sending dose fixed, JAM still jointly decides how many images to send to each receiver, how many to send from each class, and which images to send. A fixed dose thus preserves the allocation and matching problem studied in the paper and lets us compare methods with the same transmission count.

The CIFAR-100 budget sensitivity experiment in the submitted manuscript provides additional evidence for this choice. Increasing the budget from 8 to 32 images per sender per round reduces Novel Loss from 2.5355 to 2.5172 for JAM and from 2.5506 to 2.5430 for random sharing. These are means over three seeds. In the tested budget range, increasing the number of shared samples helps both methods, and JAM retains its selection advantage.

Comment 3. Gains on Office-Home and possible explanations

Our results show the same pattern. Under the current Office-Home protocol, JAM shows no consistent advantage at the same sending dose. We retain this result to report performance when few observations are available per class. The task has 65 public classes. Each client receives 321 natural arrivals in total, or about 4.94 images per class on average. With few observations per class, estimates of feature averages are more uncertain. The lower Summary SNR is consistent with the smaller gains from selection and offers a possible explanation for this pattern.

The additional experiments test selection gains on other tasks. On class-balanced PACS and the five-superclass iWildCam task, JAM achieves significantly lower whole-process Novel Loss than all three Need baselines at the same sending dose under both standard BN and fixed BN running statistics. With the shared images and per-class receipt counts held fixed, the comparison also shows a statistically significant benefit from JAM's specific recipient assignments. See Tables 3–5 in our general response. We report these results alongside Office-Home to show where joint selection yielded gains under the tested conditions and where the sharing methods performed similarly.

AuthorsOpenReview ↗

Response to Reviewer SpxT (3/3) — Comment 4

Response to Reviewer SpxT (3/3)

Comment 4. Using Summary SNR to assess applicability at deployment

We agree that Summary SNR cannot serve as an applicability indicator at deployment given the inputs it requires. Summary SNR is an offline diagnostic for how well per-class feature summaries can distinguish environments. Sample selection in JAM does not use Summary SNR, and running JAM does not require calculating it first. Calculating it requires environment identities and the full training pool, which are available for this experimental diagnostic. Appendix A.3 and the mechanism diagnostics section of the submitted manuscript describe this role. We will clarify the distinction from the inputs used for selection in the revised manuscript.

Selection uses public class labels and features extracted from available images with the frozen feature extractor. The network reference aggregates per-class statistics from all natural arrivals up to the current round. Each receiver summary also includes previously received shared images. Selection can proceed without hidden environment identities, future arrivals, the full original training pool, or a prior estimate of Summary SNR.

Summary SNR helps explain how readily feature differences across environments can be distinguished from finite-sample estimation error. It helps interpret observations across tasks and supplies no validated threshold for deployment gains. The practical value of JAM is evaluated through comparisons at the same sending dose and pairing controls that hold the shared images and receipt counts fixed, as shown in Tables 3–5 of our general response.

Thank you for taking the time to review our work. Your comments helped clarify the conditions under which the method applies. We welcome further discussion of any assumptions that warrant additional testing.

Thanks for the authors' responses. The author has addressed part of my concerns, and I will keep the score unchanged.

Reviewer_J5TeOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes JAM (Joint Allocation and Matching), a sample-sharing framework for decentralized edge learning under asynchronous environmental shifts and limited communication budgets. JAM addresses three coupled decisions: which receivers should receive samples, how many samples should be allocated to each receiver, and which specific samples should be transmitted.

The proposed approach uses a frozen feature extractor from the initial model to construct per-class feature statistics. It formulates a joint receiver–sample selection objective based on minimizing the discrepancy between receivers' feature means and the network reference. A dynamic marginal-gain greedy algorithm is then employed to select receiver–sample pairs under a fixed communication budget.

Experiments on CIFAR-100, PACS, and Office-Home demonstrate improvements over several sample-sharing and prototype-based baselines, particularly on CIFAR-100 and PACS. Additional ablations investigate the contributions of allocation and matching, environmental identifiability, and communication overhead.

Critical strengths
  1. A meaningful problem formulation with coupled allocation and matching decisions. The paper studies an important problem in decentralized learning: how to exploit environmental information observed by other clients before the same conditions are encountered locally.
  2. Comprehensive experimental analysis and relatively transparent reporting. The paper evaluates JAM against multiple baselines and investigates the effects of receiver allocation, sample matching, dynamic scoring, backbone architectures, and communication budgets.
Critical weaknesses

W1. Limited methodological novelty and insufficient validation of the optimization objective.

The main methodological components—class-wise feature statistics, feature-mean discrepancy minimization, and marginal-gain greedy selection—are relatively straightforward.

Proposition 1 establishes that the single-pair cost is generally non-separable into independent receiver and sample terms. While mathematically valid, this observation follows directly from expanding the squared Euclidean distance. It does not establish that the proposed joint greedy algorithm is superior to other receiver-conditioned selection strategies.

More importantly, the proposed feature-discrepancy objective is not convincingly connected to downstream adaptation performance.

In Appendix D.2, the authors report that changes in the JAM objective and changes in the actual post-update task loss agree in direction in only 526 out of 1,065 effective single-round permutation comparisons, approximately 49.4%. This suggests that improvements in the surrogate objective are not reliable indicators of immediate task-loss improvements.

W2. Limited evidence of practical benefits under realistic communication constraints and deployment conditions.

The experiments demonstrate improvements under equal sample-transmission budgets, but practical communication efficiency depends on total transmitted bytes, including control metadata.

On CIFAR-100, JAM-INT4 achieves a Novel Loss of 2.5323 using approximately 300 KB per client per round, whereas Random-U48 achieves 2.5354 using approximately 295 KB. The difference is small, indicating that sending additional random samples can recover most of the proposed method's advantage under approximately equal total communication.

Questions for authors

Q1. How well does the surrogate objective reflect actual adaptation utility? Q2. How much improvement is specifically attributable to receiver–sample matching? Q3. Does JAM remain competitive under equal total communication budgets? Q4. How sensitive is JAM to its deployment assumptions?

6 comments
AuthorsOpenReview ↗

Response to Reviewer J5Te (1/6) — Comment 1

Response to Reviewer J5Te (1/6)

Thank you for your careful review and for recognizing the value of the problem formulation and the transparency of the experimental analysis. You identified a central question for this work: whether the sample selection objective leads to better adaptation in practice. Your comments prompted two additional comparisons: a per-receiver herding baseline at the same sending dose and an experiment that permutes recipients while preserving the shared images. To address your question about total communication, we used the submitted manuscript's accounting for control metadata and added a communication budget comparison with JPEG images. We respond to each point below.

Comment 1. Methodological novelty and validation of the selection objective

We agree that each component has precedents. Our contribution is to formulate them as a joint selection problem under a sender's total budget and test the practical value of making these decisions jointly. Within each sender's budget, JAM jointly decides which receivers to serve, how many images to send to each, and which images to send. It uses per-class feature averages to score each pair and updates the receiver's estimated feature average as it selects each image. Proposition 1 shows that the pair cost is generally non-separable into independent receiver and sample terms, motivating a direct comparison of the pairs. We test this selection method against Need-Herding at the same sending dose. Need-Herding first allocates quotas to receivers based on need, then sets image counts by class subject to candidate availability. Finally, it uses herding with a linear kernel to select images one at a time for each receiver and class. Both methods use the same feature extractor, initial model, data stream, candidate pool, and sending budget. JAM also jointly determines the number of images per class, and its score accounts for the norm of each candidate feature vector.

On class-balanced PACS and the five-superclass iWildCam task at the same sending dose, JAM achieves significantly lower whole-process Novel Loss than Need-Herding under both standard BN and BN with fixed running statistics. JAM also significantly reduces whole-process Novel Loss and Pre Loss (Pre L, the pre-arrival Deficit cross-entropy) relative to Need-SES and Need-CF-CL on both tasks under both BN settings. Tables 3 (BN with fixed running statistics) and 4 (standard BN) in our General Response give the full results and the families of comparisons used for Holm correction.

To examine the connection between the feature-discrepancy objective and adaptation performance, we also permute recipients to test the value of the receiver–sample pairs selected by the objective. JAM uses discrepancies in per-class feature averages to select shared images that complement a receiver's local observations. After these images enter the training set, they can continue to contribute to training in later rounds. The single-round diagnostic in Appendix D.2 of the submitted manuscript shows that the objective has weak predictive power for the change in loss immediately after a few SGD steps. A comparison over the full adaptation process is therefore needed to evaluate the practical effect of these choices.

We keep the images sent in each round, the total number of transmissions, and the per-class number of images each receiver receives fixed. We then permute the recipients of individual images and rerun the entire adaptation process. On class-balanced PACS and the five-superclass iWildCam task, JAM's original pairs yield lower whole-process Novel Loss under both BN settings. All five seed-wise differences favor JAM in each of the four comparisons, and all four comparisons are significant after Holm correction. See Question 2 for details. With the same images and receipt counts, retaining the receiver–sample pairs selected by the objective still improves adaptation, providing direct experimental evidence for its usefulness.

AuthorsOpenReview ↗

Response to Reviewer J5Te (2/6) — Comment 2

Response to Reviewer J5Te (2/6)

Comment 2. Communication cost and deployment conditions

We agree that sending additional random samples recovers most of the gain on CIFAR-100. In the original experiment, an uncompressed image occupies only 3 KiB. The bytes saved by omitting INT4 control metadata cover 32 additional transmissions, raising the random baseline's sending dose from 16 to 48 images per round.

To examine how this trade-off changes across tasks, both methods send JPEG images. We use the mean JPEG size of each candidate pool to convert the saved bytes into an additional random transmission count, rounded down to whole images in each round. On class-balanced PACS under BN with fixed running statistics, the calculation allows 1 additional image. JAM-INT4-JPEG and Random-JPEG therefore send 5 and 6 images per sender per round, respectively. JAM still reduces whole-process Novel Loss by 0.024818, with all five seed-wise differences favoring JAM (p=0.001360p=0.001360). It also significantly improves pre-arrival accuracy, pre-arrival cross-entropy, and whole-process Deploy Loss.

Question 3 gives the budget accounting rule, transmission counts, and training results for each task. Question 4 summarizes the results with longer adaptation, both BN settings, and a stronger initial model.

AuthorsOpenReview ↗

Response to Reviewer J5Te (3/6) — Question 1

Response to Reviewer J5Te (3/6)

Question 1. The selection objective and adaptation utility

On class-balanced PACS and the five-superclass iWildCam task, the sharing decisions made using the JAM objective improve performance over the full adaptation process. Comparison with baselines that send the same number of samples measures JAM's overall gain. The permutation comparison further shows that keeping the specific receiver–sample pairs selected by the objective lowers whole-process Novel Loss even when the shared images and per-class receipt counts are fixed. The surrogate objective evaluates the value of sending a candidate image to each receiver based on the discrepancy between the receiver's per-class feature averages and the network reference. Selection uses public class labels and frozen features. Environment identities are used only for post hoc diagnostics. Comment 1 explains the single-round diagnostic and what it measures.

The transmission destinations show that the objective sends more relevant images to receivers that have yet to encounter the corresponding environment. Among the transmissions that can affect the pre-arrival evaluation, these images account for 83.33% with JAM and 63.83% with JAM-Permuted on class-balanced PACS, and 70.00% and 53.67% on the five-superclass iWildCam task. On iWildCam, shared samples account for 10.41% of all sample accesses during training for both methods. Shared samples from the receiver's Deficit environment account for 6.82% of all training accesses with JAM and 5.22% with JAM-Permuted. These observations indicate that feature discrepancies help identify images that expand the range of environments covered by the receivers' training data.

These selections also improve task performance. We keep the images sent in each round, the total number of transmissions, and the per-class number of images each receiver receives fixed, permute the recipients of individual images, and rerun the entire adaptation process. On class-balanced PACS and the five-superclass iWildCam task, preserving JAM's original receiver–sample pairs yields lower whole-process Novel Loss under both standard BN and BN with fixed running statistics. All five seed-wise differences favor JAM in each of the four comparisons, and all four comparisons are significant after Holm correction. Table 5 in our General Response gives the full effect sizes and statistical results. Question 2 explains how this comparison isolates the contribution of matching. With the same shared images and receipt counts, the receiver–sample pairs selected by the objective yield better adaptation.

The gains appear before data from the target environment arrive naturally at the receiver. On the five-superclass iWildCam task, JAM improves pre-arrival accuracy over the permutation control by 1.0417 and 1.5083 percentage points under standard BN and BN with fixed running statistics, respectively (Holm-adjusted pp values of 0.004530 and 0.030978). Pre-arrival cross-entropy decreases by 0.044495 and 0.099243, and whole-process Deploy Loss, which accounts for Old environments, decreases by 0.008692 and 0.018942. All four comparisons are also significant after Holm correction. The gains from the selected pairs thus cover both early adaptation to new environments and performance across the deployment period.

Comparisons with strong baselines at the same sending dose further support the practical value of the selection method. With the same initial model, data stream, feature extractor, and sending budget, JAM achieves significantly lower whole-process Novel Loss than Need-SES, Need-CF-CL, and Need-Herding on both tasks under both BN settings. It also significantly lowers pre-arrival cross-entropy relative to Need-SES and Need-CF-CL. Tables 3 and 4 in our General Response provide the full five-seed results and the families of comparisons used for Holm correction.

Reviewer_r1xBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies edge clients that lose cloud access and see new environments at different times. Each sender has a small per-round budget of raw samples, and JAM decides who to send to and which samples, by greedily reducing the distance between each receiver's per-class feature mean (in frozen G0 features) and a network-wide per-class mean. Experiments on CIFAR-100, PACS and Office-Home show small gains over random sharing and a few adapted baselines on the first two, and no gain on Office-Home.

Critical strengths
  1. The setting is reasonable and the method is simple and cheap.
  2. The reporting is honest. Paired tests with Holm correction, the null Office-Home result and the byte-matched Random-U48 run are all shown, which I appreciate.
  3. The ablations (JAM-Matching, JAM-Allocation, JAM-Permuted, Static-JAM) try to separate allocation from matching, and the communication ledger checks out.
Critical weaknesses
  1. The gains are really small. Novel loss goes from 2.549 to 2.530 on CIFAR-100 and 0.796 to 0.776 on PACS. Pre-arrival accuracy, which is the main motivation, is not significantly better than Need-SES or Need-CF-CL on any dataset, and on PACS frozen G0 beats every method including JAM (71.94 vs 70.95). Also whole-process accuracy is never reported, only loss.

  2. The communication story doesn't hold up well. At matched bytes on CIFAR-100, Random-U48 is basically the same as JAM-INT4 (2.5354 vs 2.5323). On PACS/Office-Home the samples are charged as uncompressed pixels (150KB each), which is why the metadata looks free there. With JPEG the same bytes would buy several extra random samples.

  3. JAM needs every client's per-class stats every round plus all-to-all summary exchange. That's pretty much a server step and doesn't fit the intermittent point-to-point motivation. Only fully connected networks with 8-10 clients are tested.

  4. Novelty is limited. Greedily matching a running mean to a target mean is basically kernel herding with a linear kernel (Chen et al. 2010, Bach et al. 2012), here per receiver with a shared budget, and this isn't discussed. Prop. 1 is just expanding the square and says nothing about how good the greedy solution is. Also pulling everyone toward the network mean assumes every client wants the network mixture, which happens to be exactly how Novel is defined in the eval.

  5. The paper's own ablations don't really support the pair-coupling claim. On PACS, JAM-Permuted keeps almost all the gain (0.0197 vs 0.0205). In the single-round test, the objective and the task loss agree in sign in 526 of 1065 cases, which is chance.

  6. The setup is small and synthetic. G0 is trained for 5 rounds, adaptation is 40 SGD steps total, and on Office-Home each client sees about 5 samples per class, so the null result there (and the SNR story) mostly comes from the protocol. No benchmark has actual lighting/weather/location shifts (ImageNet-C, WILDS, DomainNet). Related work on data sharing in non-IID FL is missing, e.g. Zhao et al. 2018, FedMix (Yoon et al. 2021), D2D data sharing (Cai et al. 2020) and Gen-FedSD (Morafah et al. 2024). Privacy of sending raw samples isn't discussed either. Minor: "Equation equation 23" all over, and a stray "zℓ. (25)" at the end of A.2.

Questions for authors
  1. Can you report whole-process Novel and Deploy accuracy for all methods?
  2. What does a byte-matched random baseline look like on PACS/Office-Home with JPEG samples?
  3. How do you build the network reference when links are intermittent or the graph is sparse?
  4. How does JAM compare to per-receiver kernel herding with the same budget?
8 comments
AuthorsOpenReview ↗

Response to Reviewer r1xB (1/8) — Comment 1

Response to Reviewer r1xB (1/8)

Thank you for your detailed and candid review. We appreciate your assessment of the problem setting, transparent reporting, and ablation design. Your concerns about pre-arrival accuracy, communication accounting, evidence for pairing, and experimental scale motivated most of the additional experiments in this response. We examined early adaptation in the original tasks, evaluated sample selection at the same sending dose on class-balanced PACS and the five-superclass iWildCam task, and reported the requested whole-process accuracy. Sections 1–2 of our general response describe the data streams and training settings.

Comment 1. Size of the gains and pre-arrival accuracy

We agree that both the size of the gains and pre-arrival accuracy warrant closer examination. Tables 3–4 in our general response report five-seed results with 4 local SGD steps per client per round. On class-balanced PACS and the five-superclass iWildCam task, JAM significantly lowers whole-process Novel Loss relative to all three Need baselines under both BN settings. JAM also significantly lowers pre-arrival cross-entropy (Pre L) relative to Need-SES and Need-CF-CL. Relative to Need-Herding, the reduction in Pre L is significant in both tasks under standard BN and in iWildCam with fixed BN running statistics.

The absolute loss reductions in the submitted manuscript were modest on CIFAR-100 and PACS. On both datasets, JAM significantly improved whole-process Novel Loss and Final Accuracy relative to Need-SES and Need-CF-CL, and the Pre Accuracy (Pre A) differences you identified were not statistically significant. On class-balanced PACS with fixed BN running statistics, JAM improves Pre A by 2.6333 percentage points over Need-SES (Holm-adjusted p=0.008968p=0.008968) and by 1.7635 percentage points over Need-CF-CL (p=0.086099p=0.086099, not statistically significant). Under standard BN, neither Pre A comparison reaches statistical significance.

Under the original PACS protocol, accuracy on new environments had already declined before shared images entered training. By the final round, JAM's Deficit accuracy was 3.80 percentage points higher than that of the frozen model. We found substantial differences between the class proportions of early arrivals and the test set. In the original setting, these differences coincided with within-class environmental shifts, making it difficult to distinguish their effects on early adaptation. Repeatedly sampling from a small set of images may also affect BN running statistics. In a single-seed Local-U0 comparison using the original data stream, fixing those statistics raised Pre A by 1.49 percentage points with 4 updates per round, and the effect varied with the number of updates (Section 1 of our general response). We then balanced the seven classes in each arrival round and gave them equal weight within each test domain. On class-balanced PACS, JAM's Pre A is 68.36% under standard BN and 69.22% with fixed BN running statistics. Both are significantly higher than that of the frozen model in the same setting (67.59%, with Holm-adjusted p=0.009312p=0.009312 and 0.0000270.000027, respectively, in the supplementary tests). Both BN settings share the class-balanced stream and initial model. The original and class-balanced protocols also differ in the training data for G0G_0, the number of arrivals, and the test weights (Section 2 of our general response).

Our response to Question 1 reports the whole-process Novel and Deploy accuracy omitted from the submitted manuscript, averaged over rounds 1–9. It presents results for the submitted settings alongside those from the additional experiments.

AuthorsOpenReview ↗

Response to Reviewer r1xB (2/8) — Comments 2–3

Response to Reviewer r1xB (2/8)

Comment 2. Image encoding and communication gains

We agree that Random-U48 and JAM-INT4 have similar whole-process Novel Loss on CIFAR-100 at approximately the same total communication cost (2.5354 vs. 2.5323). An uncompressed image in the original experiment occupies only 3,072 B. With 20 public classes and 10 clients, Appendix C of the submitted manuscript accounts for 201,600 B of INT4 control overhead per client per effective communication round, counting both sending and receiving. Counting both ends of every image transmission as well, each sender can send ⌊201,600/(2×3,072)⌋=32\lfloor 201{,}600/(2\times 3{,}072)\rfloor=32 additional images per round. Increasing the random baseline's sending dose from 16 to 48 images per round recovers most of JAM's gain at the same sending dose.

Following your suggestion, we recomputed the sending doses for PACS and Office-Home using JPEG encoding. Their mean JPEG sizes are similar, at about 14.7 kB and 13.8 kB, respectively. Both tasks use the same number of clients and the same feature dimension, but the summaries for 65 classes in Office-Home use more bytes than those for 7 classes in PACS. Using the mean JPEG size of the candidate pool in each task and rounding down in each round, a sender can send 18 additional images in Office-Home and 1 in PACS. The summary overhead for the five-superclass iWildCam task is equivalent to about 0.72 images at the mean JPEG size. Under the same rule, the number of additional images is 0.

These comparisons show that omitting the summaries buys few extra random transmissions when there are relatively few public classes and clients exchanging statistics, and encoded images remain relatively large. When clients have complementary observations within classes, these conditions make it easier for JAM to retain a selection gain after accounting for summary overhead. The comparison on class-balanced PACS with fixed BN running statistics illustrates this. JAM-INT4-JPEG and Random-JPEG send 5 and 6 images per sender per round, respectively. JAM still has a 0.024818 lower whole-process Novel Loss, with all five seed-wise differences favoring JAM (p=0.001360p=0.001360). The two methods perform similarly on Office-Home. See Question 2 for the budget conversion and training results.

Comment 3. Full connectivity and exchange of statistics

Thank you for raising this point. Each client updates its model locally and can compute the common network reference after exchanging statistics over point-to-point links. We agree that this still requires all clients to complete the exchange before sample selection in each round. The communication accounting in Appendix C.1 includes the bytes sent and received in this exchange. With a fixed summary size, the network-wide volume of statistics exchanged per round grows approximately quadratically with the number of clients.

The submitted experiments use fully connected networks with 8 or 10 clients. All methods have the same reachable receivers and sending budgets, allowing us to test how the choice of recipients, the number of samples each receives, and the particular samples sent affect adaptation to new environments given the same communication opportunities. The submitted manuscript considers cloud outages during which clients can still communicate, but it could state the requirement of completing each round's network-wide exchange and the resulting limitations more clearly. We will clarify this point in the introduction and limitations of the revised manuscript and restrict our experimental conclusions to small, fully connected networks that can complete the exchange in time. Question 3 discusses the open issues for sparse or intermittent links.

AuthorsOpenReview ↗

Response to Reviewer r1xB (3/8) — Comment 4

Response to Reviewer r1xB (3/8)

Comment 4. Relation to herding with a linear kernel and the evaluation target

Matching per-class feature averages is closely related to herding with a linear kernel. JAM jointly selects receivers and images within each sender's budget. Proposition 1 shows that the cost of a pair generally cannot be separated into independent receiver and image terms. We evaluate the benefit empirically at the same sending dose. In Table 3 of our general response, with fixed BN running statistics, JAM lowers Novel Loss relative to Need-Herding by 0.013611 on class-balanced PACS and 0.060195 on the five-superclass iWildCam task (five seeds, Holm-adjusted p=0.030721p=0.030721 and 0.0006290.000629, respectively). Both comparisons under standard BN in Table 4 are also statistically significant. See Question 4 for the comparison settings.

The common network reference reflects our task: each client needs to adapt to environments emerging across the network. The Novel distribution represents those new environments and measures each client's classification performance across them. Each client has so far observed only some of them locally. In iWildCam, for example, clients may observe species within an existing class and camera locations that other clients encounter only later. Moving a receiver's per-class feature averages toward the network reference uses observations already available elsewhere in the network to supplement what the receiver has seen.

The network reference aggregates local statistics for all training images that arrived naturally across the network through the current round, weighting each client's statistics by its sample counts. These arrivals include Old. Novel evaluation uses independent test images and weights the two new environments equally. The gains also extend to Deploy, which includes Old. With fixed BN running statistics on the five-superclass iWildCam task, JAM's Deploy Loss is 0.080076 lower than that of random sharing at the same sending dose, with all five seed-wise differences favoring JAM.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General Response (1/5) — Overview, original PACS and BN

General Response and Additional Experiments (1/5)

We thank the three reviewers for reading the paper carefully and raising specific questions. Their comments on pre-arrival accuracy, communication accounting, the pairing ablation, and deployment assumptions prompted most of the additional experiments in this response. We now report whole-process accuracy, include a per-receiver herding baseline at the same sending dose, recalculate how many extra images can be sent in place of the control metadata when using JPEG, and reassess receiver–sample matching on two new tasks using five seeds. This feedback helped us improve the experimental design and clarify the paper. We report the new experiments and analyses here and plan to upload a revised manuscript in the next few days. Sections 1–2 describe the shared background and changes to the data streams. Section 3 presents the main results. We welcome further questions.

As r1xB noted, JAM's Pre Accuracy in the original PACS protocol was 70.95%, compared with 71.94% for Frozen G0G_0. Increasing the number of updates per round from 4 to 16 under the original protocol reduced JAM's whole-process Novel Loss across three seeds, but its Pre Accuracy remained below that of the frozen model. This prompted us to examine the class proportions of early arrivals and the behavior of batch normalization (BN). We then compared two BN settings on class-balanced PACS and iWildCam. With BN running statistics fixed to those of G0G_0, JAM achieved significantly lower whole-process Novel Loss than all three baselines at the same sending dose on both tasks. Pre Accuracy increased significantly in three comparisons, and Pre Loss decreased significantly in five.

1. Class imbalance and BN in the original PACS protocol

On PACS, G0G_0 was trained on art painting and cartoon images (Old). Starting in round 0, one client group receives photo images locally and the other receives sketch images locally (Seen). For each group, the other new domain (Deficit) first arrives locally in round 4. The seven prediction classes stay the same. In round 0, clients have not yet trained on received shared images. JAM and Local-U0, which trains only on local natural arrivals, both achieve 70.84% average accuracy across the two new domains, compared with 71.94% for the frozen model. The initial drop therefore occurs during local adaptation before shared images enter training.

The original protocol fixes the number of arrivals from each environment in each round. Within each phase, class quotas follow the proportions of images available from each source and are distributed across rounds. Old arrivals cover all seven classes in every round, with different counts by class. About 20% of the images in each class and domain are held out for testing. Single-domain scores reflect the class frequencies in that domain. Novel weights the two new domains equally, and Deploy combines overall test-set class frequencies with round-specific environment weights. The protocol does not balance class proportions. Early training and test sets can therefore differ considerably in class composition, making it difficult to determine how much of the early performance reflects adaptation to within-class environment shifts.

For example, for client 0 with seed 42, 20 of the 32 first-round arrivals are in the person class (62.5%). The same class accounts for 86/333 (25.83%) of the photo test set and 32/786 (4.07%) of the sketch test set. Sketch had not yet arrived locally for this client. With uniform sampling over images, frequent classes appear more often in training batches, while fewer images are available to estimate feature averages for infrequent classes.

The 32 first-round images are sampled repeatedly to form 4 batches of 64 images each. Under standard BN, training uses the current batch means and variances and updates the running means and variances used at evaluation. Running statistics updated from a small set of early images may poorly represent later environments. When fine-tuning pretrained models with BN, Keras recommends retaining the existing running statistics.

We therefore tested BN with fixed running statistics. Local training and evaluation both use the per-channel means and variances stored in G0G_0, while the BN affine parameters and other weights continue to update. In the Local-U0 comparison for seed 42, G0G_0, the data stream, and the evaluation protocol were held fixed. Fixing the BN statistics increased Pre Accuracy from 68.98% to 70.47% with 4 steps per round and decreased it from 68.76% to 67.70% with 16 steps per round.

AuthorsOpenReview ↗

General Response (2/5) — Class-balanced streams and the BN comparison

General Response and Additional Experiments (2/5)

2. Class-balanced streams and comparison of BN settings

To reduce the effect of differences in class proportions between training and testing on the comparison of early adaptation, we redistributed the PACS arrivals and weighted all seven classes equally within each test domain. We kept 8 clients and assigned each client 3 images from each of the seven classes in every arrival round. Deficit still first arrives locally in round 4. For example, in each of the first four rounds, the 6 photo images of each class are assigned to the 4 clients that first encounter photo. Two receive 1 image each, and the other two receive 2 each. Across the 8 clients, each domain contributes 6 images per class in every arrival round. We set the splits and evaluation rules before training. Table 1 compares the two protocols.

Table 1. Arrivals, initial training, and testing on PACS

ItemOriginal PACSClass-balanced PACS
Arrivals per client per arrival round32 images21 images, 3 from each of 7 classes
Old / Seen / Deficit images per round16/16/0 in rounds 0–3 and 16/8/8 in rounds 4–810/11/0 or 11/10/0 early, and 10/6/5, 10/5/6, or 11/5/5 later
Total natural arrivals2,304 images1,512 images
Old data for G0G_02,361 training images48 training and 6 validation images per class from each of the two Old domains, totaling 672 training and 84 validation images
Held-out test set1,998 images: 410 art painting, 469 cartoon, 333 photo, and 786 sketchSame images. All classes receive equal weight within each domain. Environment weights for Novel and Deploy follow the original protocol.

The sketch domain has only 64 training images in the house class. With 8 clients, allocating 4 images per class per round would require 72 of them. The design in Table 1 requires 54. We initialized ResNet-18 with ImageNet weights and trained a new G0G_0 for 5 rounds with standard BN. This protocol also changes the initial model, number of arrivals, and evaluation weights. We therefore reran all methods and compare them within the class-balanced protocol, since the results from the original imbalanced stream are not directly comparable.

To evaluate adaptation to changes in real capture locations, we constructed a five-superclass task from the WILDS iWildCam dataset. Following the CIFAR-100 design, the model always predicts the same five superclasses, while the underlying species and camera-trap locations vary by environment. This simulates encountering new species within the same superclass as clients enter new environments. The 15 species and 136 camera-trap locations are divided into three disjoint groups. As on PACS, we initialized ResNet-18 with ImageNet-pretrained weights and trained it on Old data for 5 rounds with standard BN to obtain the common initial model G0G_0. The other two groups serve as Seen and Deficit, with their roles reversed across client groups. Each of the 6 clients receives 30 images per round, 6 per superclass. Within each superclass, the Old / Seen / Deficit counts are 2/4/0 before round 4 and 2/2/2 thereafter. Across the network, each species contributes 12 images per round. Initial training, validation, natural arrivals, and testing use 1,000, 200, 1,620, and 600 images, respectively. Images from the same capture date or camera-triggered burst are kept in the same split. Each burst in the arrival stream is assigned to a single client, preventing near-duplicate frames from appearing in both training and testing.

Within each new task, all adaptation methods start from the same G0G_0, use the same data stream and evaluation protocol under both BN settings, and make 4 updates per round. During sharing, each sender can transmit up to 5 images per round. The only labels available to JAM and the baselines for selection are the public class labels (7 classes on PACS and 5 superclasses on iWildCam). Hidden environment information, such as species and capture location, is used only to construct the stream and for post hoc evaluation.

AuthorsOpenReview ↗

General Response (3/5) — Frozen model and fixed BN

General Response and Additional Experiments (3/5)

3. Adaptation results under the new protocols

Table 2 uses Frozen G0G_0 in each task as the reference and reports JAM's means across five seeds (40, 42, 44, 45, and 46) under both BN settings. Shared images are transmitted without JPEG encoding. Pre Accuracy (Pre A) and Pre Loss (Pre L) are the Deficit accuracy and cross-entropy at round 3, before the Deficit environment naturally arrives at the receiver. Nov L and Dep L are Novel Loss and Deploy Loss averaged across rounds 1–9.

Table 2. Frozen model and JAM under both BN settings within each task

TaskConditionPre A (%)Pre LNov LDep L
Class-balanced PACSFrozen G0G_067.58831.0201121.0201120.805751
Class-balanced PACSJAM, standard BN68.36230.9622770.8100350.678520
Class-balanced PACSJAM, fixed BN running statistics69.22240.8649660.6684860.549993
Five-superclass iWildCamFrozen G0G_031.75002.1478032.1478031.692797
Five-superclass iWildCamJAM, standard BN30.35001.8958441.6123621.276685
Five-superclass iWildCamJAM, fixed BN running statistics36.83331.6348881.4432291.208087

Under standard BN, JAM has higher Pre Accuracy than Frozen G0G_0 on class-balanced PACS. On the five-superclass iWildCam task, JAM has lower Pre Accuracy than the frozen model, while cross-entropy at the same time decreases from 2.147803 to 1.895844. JAM has lower whole-process Novel and Deploy Loss than the frozen model on both tasks. With fixed BN running statistics, JAM achieves a better mean on each of the four metrics than the respective frozen model on both tasks. Under the original Office-Home protocol, fixing BN running statistics reduces each of the three adaptation methods' Novel Loss by about 0.26 relative to standard BN. JAM and Neutral Random still perform similarly at the same sending dose.

Under both BN settings, JAM has lower Nov L and Dep L than Neutral Random and the three Need baselines in all five seeds. Need-Herding first determines the number of images of each class for each receiver, then selects images one at a time using herding with a linear kernel to bring the per-class feature averages closer to the network reference. Table 3 reports five-seed differences with fixed BN running statistics. In Tables 3 and 4, positive values favor JAM, and Pre A differences are in percentage points. Parentheses show Holm-adjusted pp values, and bold indicates adjusted p<0.05p<0.05.

Table 3. Five-seed differences between JAM and three baselines with fixed BN running statistics

Task / BaselinePre A increase (Holm pp)Pre L reduction (Holm pp)Nov L reduction (Holm pp)
Class-balanced PACS / Need-SES2.6333 (0.008968)0.076300 (0.003273)0.035027 (0.002704)
Class-balanced PACS / Need-CF-CL1.7635 (0.086099)0.064427 (0.025128)0.032443 (0.004045)
Class-balanced PACS / Need-Herding0.3723 (0.489764)0.023538 (0.113898)0.013611 (0.030721)
Five-superclass iWildCam / Need-SES5.4333 (0.045951)0.263821 (0.000473)0.074784 (0.001107)
Five-superclass iWildCam / Need-CF-CL4.3833 (0.000629)0.319133 (0.001064)0.146502 (0.006350)
Five-superclass iWildCam / Need-Herding2.6000 (0.086099)0.226010 (0.001064)0.060195 (0.000629)

Both tables use two-sided paired tt-tests across seeds. For Table 3, the 12 comparisons of Pre A and Nov L across the two tasks and three baselines form one Holm correction family. After analyzing the main results, we also tested cross-entropy at the same pre-arrival evaluation point. The Pre L analyses in both tables use all five seeds. For each task, the 12 Pre L comparisons across both BN settings and six reference methods form one Holm correction family. The reference methods are the three Need baselines shown in the tables, Local-U0, Neutral Random, and Frozen G0G_0.

With fixed BN running statistics, all six Nov L comparisons are statistically significant. Pre A is significantly higher than Need-SES on PACS and than Need-SES and Need-CF-CL on iWildCam. Pre L is significantly lower than Need-SES and Need-CF-CL on both tasks. Against Need-Herding, the Pre L reduction is statistically significant on iWildCam and does not reach significance on class-balanced PACS.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.