- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper targets multi-site AIoT federation: sites of one organisation observe the same machines through different sensor subsets, each running a small language model of width tied to its hardware budget. The authors formalise this as feature-view heterogeneity (every client covers all classes but none observes all evidence) and a pre-registered 756-cell campaign shows weight-space resize-and-average and logit-consensus distillation both fail. They propose RelFed: each client re-expresses its pooled hidden state as cosine similarities to a shared anchor set of M=200 task-domain texts, producing a representation whose dimension depends only on M. A single shared linear head (C × M) is aggregated by plain sample-weighted FedAvg, with no resize, no interpolation, no distillation; communication is ~16 KB/round. On 20 Newsgroups RelFed gains +3.69 pp, with 37/41 paired wins and an honest operating envelope over ~120 cells.
Critical strengths
-
Conceptual contribution: feature-view heterogeneity as a distinct regime, with the right diagnostic pre-registered campaigns. Naming feature-view heterogeneity and showing that the two natural workarounds fail (756-cell resize-and-average campaign + logit-distillation campaign) is a clean piece of conceptual work. The 11 resize operators and the +0.16 pp best noise-level result are unusually thorough.
-
Methodologically tight and exactly-parameter-federating. Building on Moschella et al. (2023), the paper lifts relative representations from a post-hoc stitching tool to a train-time substrate. The single linear head W_{\text{rel}} \in \mathbb R^{C \times M} is aggregated by plain sample-weighted FedAvg; the 16 KB per client per round budget (229–542× smaller than shape-incompatible LoRA exchange, 4–6 orders of magnitude below full-model exchange) is rare and very concrete for AIoT deployment.
Critical weaknesses
-
The confirmed-effect subset is narrow. Table 1 reports +3.69 pp on 20 Newsgroups with CI [+2.98, +4.11] and 5/5 seeds, but Yahoo (+1.27, CI [-0.10, +2.70], 3/5) and HAR (+0.94, CI [-0.78, +2.42], 6/8) are explicitly labelled "consistent trends, not confirmed effects". The 41 paired runs and the 37/41 sign test are genuinely strong paired evidence, but the more conservative reading is that the confirmed improvement is concentrated on 20 Newsgroups. The operating envelope (Table 10) reinforces this: AG-News at -0.98 pp CI [-1.57, -0.39], Yahoo-seq256 at -0.33, HAR-Qwen-det at -0.89, and DBpedia +0.61 against an 82.3 % floor.
-
The protocol scaling is narrow and the proprietary industrial pilot is unquantified. The main protocol uses N=5 clients with 400 examples each; client-count scaling is deferred. The report explicitly defers client-count scaling, and the claim "federating frequently doubles the gain of federating rarely" only varies aggregation round rather than client count. The proprietary multi-site industrial pilot that motivates the paper is included as motivation only — no reproducible number rests on it — so the generalisation from public benchmarks (20NG / Yahoo / HAR) to a real factory fleet is unverified.
Questions for authors
-
Could you explicitly report a confirmed-effect subset — e.g. all rows in Table 10 whose bootstrap CI excludes zero and whose per-seed sign-consistency is \ge 4/n? The 37/41 figure is correct as a paired count, but the headline narrative is more interpretable next to a count of confirmed-cell wins.
-
The main protocol uses N=5 clients with 400 examples each. Could you add a small client-scaling experiment (e.g. N \in 5, 10, 20 on 20NG) and a longer-input + same-data arm so the reader can separate “more data” from “more clients”?
-
The proprietary industrial pilot is mentioned only as motivation. Could you report order-of-magnitude numbers (federated floor, paired delta, sign-consistency across the available deployments) without revealing the system? That would let the reader judge how far the public-benchmark results travel toward the real multi-site AIoT deployment that motivates the paper.
3 comments
The statement of AI documentation
I have filled out the AI documentation. The review was independently assessed by me. An LLM was only used for language polishing; all technical evaluation and the final score are my own judgment. There is no mismatch between my human assessment and the AI-assisted drafting. I will actively join the discussion with authors and the AC during the discussion period.
Thank you for your prompt reply and for carrying out the extra experiments.
W1/Q1 (Confirmed-effect subset). This is the analysis that has been performed. Applying the criterion to the 17 configurations with at least five paired seeds yielded four confirmed positive cells: 20NG mixed (+3.69, 5/5), movie-review sentiment with the Qwen fleet (+2.83, 5/5), 20NG with the Qwen fleet (+2.79, 5/5), and the sensor task with the Qwen fleet (+1.39, 5/5). Two of the four are still 20NG; however, the movie-review and sensor results from the two modalities and fleets are in line. The confirmed subset is no longer 20NG-only. The 37/41 paired count remains a useful summary; the full table plus the operating envelope is what I will read when judging the general claim.
W2/Q2 (Client Scaling). The N=5/10/20 experiment is the appropriate design, and the two arms are "more data" and "more clients". At 400 examples per site, the gains are +3.69 at N=5, +1.90 at N=10 (CI +0.10 to +3.39, 4/5), and +3.67 at N=20; at a fixed total data, they are +2.83 at N=10 and +0.86 at N=20, both with intervals above zero. This supports the claim that the effect is not merely a data effect, and that it survives 100 examples per site. Two additions would help here: please include the table with per-seed values in the response, and explain the N = 10 dip at 400 examples per site, since the non-monotonicity between N = 10 and N = 20 is not obvious. The fixed-total arm is the simpler test, and the expected shrinkage is observed.
W3/Q3 (Industrial Pilot). I accept that the industrial data will not be disclosed. The paper now presents the pilot as motivation only and states that there is no claim based on it; that is the correct solution. I do not need the proprietary numbers and will not penalise the paper for not including them. Do not state in the abstract or introduction that the pilot study provides quantitative verification of the method.
General. The additional experiments are real work, and I appreciate the care taken in running them. After re-reading them, however, I do not think they yet change my overall assessment. The new experiments answer the questions I asked, and I do not dispute their execution. What they do not change is that the confirmed evidence for the reference mixed fleet still comes from one dataset, while the two new confirmed cells use a single model family; the fixed-total arm also shrinks to +0.86 pp. I therefore keep the rating at 2 for this round, and the decisive item is a confirmed cell on a second dataset under the mixed fleet rather than further reporting of the existing ones.
We thank the reviewer for the careful reading and for the precise suggestions, which we have adopted.
Q1 / W1 (confirmed-effect subset). This suggestion strengthens the paper, and we now report the confirmed-effect subset explicitly. Applying the reviewer's criterion (bootstrap 95% interval excludes zero and per-seed sign consistency at least 4 of n) to all 17 configurations with at least five paired seeds, four configurations are confirmed positive: 20 Newsgroups with the mixed fleet (+3.69 pp, [+2.98, +4.11], 5/5), movie-review sentiment with the Qwen fleet (+2.83 pp, [+0.95, +4.71], 5/5), 20 Newsgroups with the Qwen fleet (+2.79 pp, [+2.04, +3.31], 5/5), and the sensor task with the Qwen fleet (+1.39 pp, [+0.64, +2.14], 5/5). Every one of these passes with all five seeds positive, and together they span two modalities (text and serialized sensor windows) and two independent backbone fleets, so the effect is neither specific to 20 Newsgroups nor to one architecture mix. This is exactly the behaviour the method predicts: RelFed transfers feature-view evidence, so the gain is largest where partial views leave local training short of the task ceiling, as the operating envelope in the appendix maps out. We now report this subset in the abstract and the main results, alongside the 37/41 paired count.
W2 / Q2 (client scaling, more data vs more clients). We ran exactly this experiment on 20 Newsgroups with five paired seeds per cell. With 400 examples per site, so that total data grows with the fleet, RelFed gains +3.69 pp at N = 5 (CI [+2.98, +4.11], 5/5), +1.90 pp at N = 10 (CI [+0.10, +3.39], 4/5) and +3.67 pp at N = 20 (CI [+3.02, +4.42], 5/5). With 2,000 examples in total, so that each site holds 200 and then 100 examples, it gains +2.83 pp at N = 10 (CI [+2.29, +3.52], 5/5) and +0.86 pp at N = 20 (CI [+0.56, +1.15], 5/5). All five cells have an interval above zero. The gain therefore persists as the fleet quadruples at fixed data per site, so it is not a data effect, and it survives even when each of twenty sites keeps only 100 examples. All fleets remain architecture-heterogeneous, with backbones assigned round-robin over the same four-model pool.
Q3 (industrial pilot). Beyond the accuracy range already stated in Section 4.1, we are not able to release numbers from the industrial deployment under the terms of that collaboration, and we do not have the right to publish them, since the deployment data partly include healthcare data collected in Tokyo and are subject to the corresponding privacy regulations. We have therefore revised the paper so that the pilot appears only as motivation for the problem setting, and we state explicitly that no claim in the paper rests on it. Every reported number comes from public benchmarks and regenerates from the released artifacts.