- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes FedHIT, a two-stage workflow for query-specific diagnosis in federated learning. It first ranks client updates in each communication round by their alignment with the query-loss gradient, then applies SOURCE to the local training trajectories of selected clients to identify influential samples. The two stages explain different models: client scores concern the round-local global update, while sample scores concern the selected client's local endpoint. The paper evaluates client-ranking agreement, sample attribution, removal counterfactuals, and attribution cost on three medical image datasets.
Critical strengths
- The distinction between the two attribution targets is explicit and useful. Sections 2.1 and 3 carefully avoid interpreting a local sample score as that sample's influence on the final global model. The removal surrogate and the sign convention are also explained clearly, which helps make the output interpretable.
- The client-level results are encouraging within the studied setting. Section 4.1 reports mean rank correlations of 0.936–0.988 on Matek-19, with similarly strong results on the other datasets in Appendix G.1. Table 1 provides a complementary directional removal check. The cost analysis also includes trajectory replay and distinguishes measured runtime from FLOPs rather than reporting theoretical savings alone.
Critical weaknesses
- The evidence for the hierarchy itself is less convincing than the evidence for its individual components. Level 2 directly applies SOURCE (Section 3), and Section 4.2 deliberately evaluates it independently of client routing. In the cost comparison, both methods expand three clients per round; the main difference is SOURCE versus explicit sample-wise retraining (Section 4.4 and Appendix D.1). This establishes an efficient implementation relative to a costly reference, but does not establish how much client routing improves the diagnostic result. For example, there is no comparison with randomly routed clients or with SOURCE applied to all clients at a matched inspection budget, nor a quantitative data-curation or error-diagnosis outcome. Given that combining the two levels is the main contribution, this missing evaluation substantially limits the case for the method.
- The sample-level evidence and reporting leave considerable uncertainty. Figure 3 gives mean LDS values of only 0.078–0.162, without error bars or variation over training/partition seeds. Appendix G.2's removal table is explicitly a selected illustrative query-round, so it does not substantiate the stated consistency across queries and rounds. In addition, the paper does not specify actual learning rates, optimizer and batch size, segment/checkpoint settings, curvature sample cap, or key attribution-baseline settings. These omissions make it difficult to assess whether the improvements are reliable and whether the comparison can be reproduced.
Questions for authors
- Could you quantify what routing contributes using the same SOURCE implementation with top-score, random, and broader/all-client routing, preferably reporting a concrete diagnosis or curation outcome at comparable cost? A routing-budget sweep would also clarify the quality/cost trade-off.
- Can you report query/round-aggregated sample-removal results and uncertainty across independent training or partition seeds, and provide the training, segmentation, curvature, and baseline hyperparameters? For LDS, which client-round trajectories are evaluated, what query measurement is used, and how are those results aggregated?
- How is the Level-1 score related to the renormalized LOCO intervention when client weights differ? Writing u = sum_i alpha_i Delta_i, first-order expansion at the round start gives LOCO loss change approximately [s_i + alpha_i g_q^T u]/(1-alpha_i), whereas the score is s_i = -alpha_i g_q^T Delta_i. Equal weights preserve the ordering under this approximation, but unequal weights need not. Please clarify whether the reported partitions have equal client sizes and whether ranking agreement persists for unequal weights.
- Please reconcile the local sample-score definition in Equations (2)–(3) and Algorithm 1, which uses 1/n_c, with Equation (5), which introduces alpha_c/n_c. That extra factor does not affect within-client rankings, but it changes the scale of the claimed local removal estimate. Which version is implemented?