- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper studies personalized federated evaluation and proposes a combined training and inference-management framework. FedGIM combines proximal regularization, private prediction heads, and class-balanced objectives. FedIM adds test-time adaptation, drift monitoring, computational escalation, output perturbation, and selective referral. The paper argues that client-matched evaluation exposes label-prior information and proposes pooled evaluation with feature-based head routing. Experiments are conducted on an endoscopic screening task. My main concerns are the justification of the evaluation setting, limited methodological novelty, and insufficient evidence that the inference-management layer provides a useful advantage.
Critical strengths
A useful evaluation sanity check. The prior-only experiment demonstrates that client identity and client-specific prevalence can contribute substantially to aggregate discrimination. This is a useful reminder to distinguish image-derived predictive information from between-client distributional information.
Attention to deployment behavior beyond accuracy. Considering drift, computational cost, abstention, and referral is relevant to medical AI deployment. Reporting results across five seeds and including component ablations are positive aspects of the evaluation.
Critical weaknesses
The central leakage claim is insufficiently justified for personalized federated deployment. In a system deployed locally at a participating hospital, the client identity is naturally known and determines which personalized model is available. Using that hospital’s training-derived class prior is not automatically test-label leakage. The prior-only control demonstrates predictive information in client identity, but does not establish that using this information is illegitimate under the intended deployment conditions.
Pooled evaluation with unknown client identity addresses a different inference problem. The paper needs to explain why this is the appropriate target setting and distinguish evaluation shortcuts from legitimate personalization. Otherwise, performance differences may reflect the removal of available deployment information and the introduction of routing errors, rather than correction of an invalid evaluation protocol.
The deployment architecture also remains unclear. Private heads are described as remaining local, whereas the router selects among client-specific heads. The paper should explain where routing occurs and how those heads are accessed. Separately, output-level differential privacy requires a specified recipient, adversary, and protected information. Its necessity is not established merely by invoking federated learning, especially when predictions remain within the originating institution.
Limited novelty and empirical evidence do not establish the value or generality of the combined framework. FedGIM combines established proximal regularization, private-head personalization, and class balancing. FedIM similarly assembles established adaptation, drift detection, quantization, model fusion, output perturbation, and selective prediction techniques. The paper does not clearly identify a new coordination principle or demonstrate an advantage attributable to that coordination.
All empirical validation concerns one endoscopic screening task. Although Algorithm 1 is written in generic notation, the experiments do not establish that its choices, thresholds, and interactions transfer to other tasks or acquisition conditions. This is insufficient support for the broad model-agnostic deployment claims.
Moreover, Table 2 shows that adding FedIM to FedGIM reduces AUROC from 0.8572 to 0.8419 and F1 from 0.7218 to 0.6984, while increasing ECE from 0.0612 to 0.0923. Such losses might be justified by a demonstrated operational benefit, but the paper does not establish that benefit against appropriate matched alternatives. Figure 5 compares FedGIM plus FedIM with FedAvg, changing both training and inference simultaneously. A comparison against the same FedGIM model with a standard confidence-based referral policy is needed to isolate the contribution of inference management.
Questions for authors
Deployment and leakage: Is client identity known at deployment? If inference occurs at the originating hospital, why is selecting its personalized head considered leakage? Please distinguish known-client personalization from inference on samples with unknown provenance.
Routing architecture: Where does the pooled router run, and how does it access private heads that are stated to remain local? What deployment scenario requires this routing mechanism?
Novelty and usefulness: What specific methodological contribution goes beyond combining established components? Does FedIM improve risk–coverage or latency–utility behavior over the same trained model with standard selective prediction, under matched privacy and computational constraints?
Privacy assumptions: Who receives the perturbed outputs, what information is protected, and how is the sensitivity bound Δ established for the released log probabilities? The stated Gaussian guarantee depends on this bound.
Response to GHzj: deployment setting, novelty, routing architecture, privacy assumptions
Thank you for the thorough review. We take the concerns about deployment setting, novelty and privacy seriously.
1. Leakage and known-client deployment. We agree: if inference runs at the originating hospital, selecting its personalized head is legitimate personalization, not leakage. Our claim should have been limited to samples of unknown provenance, where the head must be inferred. We will rewrite the framing to distinguish the two settings, stop describing known-client gains as artifactual, and report pooled and within-client AUROC under known client identity, plus oracle-identity pooled evaluation to separate client information from routing error.
2. Routing architecture. [Answer from your actual design.] In our setup, "private" means the head is excluded from aggregation during training (Eqs. 4, 20); it does not mean the head is unavailable at inference. The router (Eq. 5) uses only training-set feature centroids and runs [where, e.g., at a coordinating service holding the heads, or within a consortium of participating sites]. The motivating scenario is [e.g., pooled screening or referral of cases whose originating site is unknown]. We will state this architecture explicitly and acknowledge where it conflicts with stricter notions of head locality.
3. Novelty. We agree that each component (proximal regularization, private heads, class balancing, TTA, drift detection, quantized escalation, output perturbation, selective prediction) is established, and we do not claim otherwise. Our contribution is their coordination: drift detection gates adaptation, confidence gates escalation, and referral operates on the released privacy-perturbed posterior, so decisions share state. We acknowledge the paper does not yet demonstrate an advantage attributable to this coordination. The matched comparison below tests it directly, and we will narrow our claims if it does not hold.
4. Table 2 and Figure 5. We acknowledge the losses (AUROC 0.8572 to 0.8419, F1 0.7218 to 0.6984, ECE 0.0612 to 0.0923) and that Figure 5 changes training and inference together. We will add, on the same FedGIM model, no management vs. standard calibrated confidence-based referral vs. full FedIM, at matched coverage, computational cost and privacy setting, with risk-coverage and latency-utility curves.
5. Single task. We agree that one endoscopic task does not support broad model-agnostic claims. We will [add a second dataset if feasible / otherwise] restrict our claims to this setting and list the limitation explicitly.
6. Privacy assumptions. We will specify the recipient of the perturbed outputs, the adversary and the protected information, define the neighboring relation, and establish Δ by clipping released log-probabilities to [−B,0] (sensitivity at most BC), then recompute σ and re-run the privacy-utility experiments. [If output-level DP cannot be justified for the intended deployment, we will present it as an optional component rather than a core claim.]
We thank you for the specific questions, which will make the paper more precise.