- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes JAM for communication-constrained edge learning during cloud outages. JAM uses features extracted by the common pre-deployment model to construct per-class summaries and jointly determines which receiver to serve, how much budget to allocate, and which samples to transmit through a dynamic marginal-gain greedy strategy. Experiments on CIFAR-100, PACS, and Office-Home show clear gains on CIFAR-100 and PACS, while improvements on Office-Home are negligible; the paper also studies mechanism variants, robustness, statistical significance, and communication quantization.
Critical strengths
- The paper formulates a practically relevant joint allocation-and-selection problem in which the usefulness of a sample explicitly depends on the target receiver rather than only on the sender-side sample quality.
- The analysis provides a clear theoretical justification for jointly modeling receiver-sample pairs.
Critical weaknesses
- The main setting assumes fully connected clients, labeled post-deployment samples, a shared frozen feature extractor, and extensive exchange of per-class statistics. Moreover, Office-Home is explicitly modified to guarantee full class coverage from the first round. These assumptions may substantially simplify the practical edge-learning scenario.
- The proposed greedy optimization lacks an optimality guarantee and always fills the predefined sending dose.
- JAM provides significant improvements on CIFAR-100 and PACS, but its improvements on Office-Home are around 10^-3 and statistically insignificant, suggesting that the method is effective only when the assumed feature-summary signal is sufficiently strong.
- Summary SNR is useful for post-hoc explanation but has limited deployability as an applicability indicator. Its empirical computation uses environment identities and statistics from the full original training pool, whereas these quantities are unavailable to JAM during actual deployment.
Questions for authors
Please see Weaknesses.
4 comments
Response to Reviewer SpxT (2/3) — Comments 2–3
Response to Reviewer SpxT (2/3)
Comment 2. Greedy optimality and the sending dose per round
Proposition 1 shows that the single-pair cost generally cannot be written as the sum of independent receiver and sample terms. This provides a structural reason to select them jointly. JAM uses a dynamic greedy heuristic. Each time it selects an image, it updates the receiver's projected feature average and selects the feasible pair with the largest current marginal gain. The greedy procedure does not guarantee a globally optimal set of pairs. We evaluate its performance through comparisons at the same sending dose and the pairing control.
The additional experiments test this directly. Tables 3–4 in our general response show that JAM achieves significantly lower whole-process Novel Loss than the per-receiver Need-Herding baseline on class-balanced PACS and the five-superclass iWildCam task under both BN settings. We also kept the shared images and each receiver's per-class receipt counts the same and permuted the recipient assignments. This permutation offset about 26.5%–34.1% of JAM's Novel Loss gain over random sharing. All five seeds favored JAM in each of the four pairing comparisons, and all four differences were statistically significant after Holm correction (Table 5 in our general response). These results support the value of joint selection and of the specific receiver–sample matches for adaptation.
Our comparisons at the same sending dose give each sender the same number of transmissions per effective communication round across methods. This separates the effect of sending more samples from the effect of selecting better samples. With the total sending dose fixed, JAM still jointly decides how many images to send to each receiver, how many to send from each class, and which images to send. A fixed dose thus preserves the allocation and matching problem studied in the paper and lets us compare methods with the same transmission count.
The CIFAR-100 budget sensitivity experiment in the submitted manuscript provides additional evidence for this choice. Increasing the budget from 8 to 32 images per sender per round reduces Novel Loss from 2.5355 to 2.5172 for JAM and from 2.5506 to 2.5430 for random sharing. These are means over three seeds. In the tested budget range, increasing the number of shared samples helps both methods, and JAM retains its selection advantage.
Comment 3. Gains on Office-Home and possible explanations
Our results show the same pattern. Under the current Office-Home protocol, JAM shows no consistent advantage at the same sending dose. We retain this result to report performance when few observations are available per class. The task has 65 public classes. Each client receives 321 natural arrivals in total, or about 4.94 images per class on average. With few observations per class, estimates of feature averages are more uncertain. The lower Summary SNR is consistent with the smaller gains from selection and offers a possible explanation for this pattern.
The additional experiments test selection gains on other tasks. On class-balanced PACS and the five-superclass iWildCam task, JAM achieves significantly lower whole-process Novel Loss than all three Need baselines at the same sending dose under both standard BN and fixed BN running statistics. With the shared images and per-class receipt counts held fixed, the comparison also shows a statistically significant benefit from JAM's specific recipient assignments. See Tables 3–5 in our general response. We report these results alongside Office-Home to show where joint selection yielded gains under the tested conditions and where the sharing methods performed similarly.
Response to Reviewer SpxT (3/3) — Comment 4
Response to Reviewer SpxT (3/3)
Comment 4. Using Summary SNR to assess applicability at deployment
We agree that Summary SNR cannot serve as an applicability indicator at deployment given the inputs it requires. Summary SNR is an offline diagnostic for how well per-class feature summaries can distinguish environments. Sample selection in JAM does not use Summary SNR, and running JAM does not require calculating it first. Calculating it requires environment identities and the full training pool, which are available for this experimental diagnostic. Appendix A.3 and the mechanism diagnostics section of the submitted manuscript describe this role. We will clarify the distinction from the inputs used for selection in the revised manuscript.
Selection uses public class labels and features extracted from available images with the frozen feature extractor. The network reference aggregates per-class statistics from all natural arrivals up to the current round. Each receiver summary also includes previously received shared images. Selection can proceed without hidden environment identities, future arrivals, the full original training pool, or a prior estimate of Summary SNR.
Summary SNR helps explain how readily feature differences across environments can be distinguished from finite-sample estimation error. It helps interpret observations across tasks and supplies no validated threshold for deployment gains. The practical value of JAM is evaluated through comparisons at the same sending dose and pairing controls that hold the shared images and receipt counts fixed, as shown in Tables 3–5 of our general response.
Thank you for taking the time to review our work. Your comments helped clarify the conditions under which the method applies. We welcome further discussion of any assumptions that warrant additional testing.
Thanks for the authors' responses. The author has addressed part of my concerns, and I will keep the score unchanged.
Response to Reviewer SpxT (1/3) — Comment 1
Response to Reviewer SpxT (1/3)
Thank you for your clear review and for recognizing the formulation and theoretical motivation of joint allocation. Your comments on the deployment assumptions, the greedy procedure, the Office-Home result, and the use of SNR helped us clarify when the method applies. We added comparisons at the same sending dose under both BN settings and training comparisons on the original Office-Home stream. We address each concern below.
Comment 1. Communication, labels, the feature extractor, and first-round class coverage
We study how to use limited sample transmissions to improve adaptation to new environments when cloud synchronization is interrupted and clients can still communicate with one another. All methods use the same fully connected point-to-point network, candidate receivers, and sender budget. This lets us compare which images to send, to whom, and in what quantities. Clients exchange statistics to construct a common network reference and update their prediction models locally. They must complete the all-to-all exchange before selecting samples in each round. Our conclusions therefore apply to small fully connected networks that can complete this exchange in time.
The shared frozen feature extractor puts the clients' per-class feature averages in the same representation space so they can be compared directly. Clients continue training their local prediction models. Freezing the feature extractor for selection and fixing BN running statistics during adaptation are separate settings. Public class labels define the common classification task. Hidden environment information, such as species and capture locations, is used only for splitting and evaluation. Selection must determine which already observed images can complement a receiver's local data.
In the first round on Office-Home, each client receives one image for each of the 65 classes, so every class has an initial summary. All methods use the same data stream. The first-round class coverage still leaves each client without local data from its Deficit environment, so the adaptation problem with limited observations per class remains. We retain this controlled setting to examine what happens when each class has initial observations and subsequent arrivals are sparse.
In the additional experiments, JAM's whole-process Novel Loss is significantly lower than that of all three Need baselines at the same sending dose on class-balanced PACS and the five-superclass iWildCam task under both BN settings. See Section 2 and Tables 3–4 of our general response for the data construction and results. In the class-balanced PACS JPEG comparison with fixed BN running statistics, Random-JPEG sends 1 more image per sender per round after converting the control overhead using the mean JPEG size of the candidate pool. JAM-INT4-JPEG still achieves significantly lower whole-process Novel Loss. See the end of Section 3 of our general response for the budget calculation, transmission counts, and loss difference. These results support the comparison of selection methods under the shared conditions above. The revised manuscript will state the conditions explicitly: sharing labeled images must be authorized, and clients must be able to finish exchanging statistics before sample selection in each round.