- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies model partition in resource- and data-heterogeneous federated learning and proposes MDMP, a metadata-guided post-training partitioning method for heterogeneous federated learning. MDMP calculates condition gradients of structural units and calibrates a global model for response concentration and separation to group those units into different modules. Then, the considered model partition problem was reformulated to solve a metadata-aware set-cover problem for each client. Experiments on 10-client MNIST, CelebA, and CIFAR-10 evaluate measured MACs, mean client accuracy, worst-client drops, and stability. At the 95% target budget, MDMP has only 6.38-9.24% computation savings with at most a 0.94 percentage-point mean accuracy drop, while the maximum client drop remains below the stated 5-point safety limit.
Critical strengths
- The paper addresses an important gap in heterogeneous federated learning. Whereas existing methods primarily adapt sub-model size to client capacity, MDMP introduces a data-aware perspective that jointly considers clients’ computational budgets and the data conditions they need to learn. This formulation is practically meaningful because enabling resource-constrained clients to participate can improve participation diversity and overall model utility.
- MDMP develops a coherent methodological pipeline that connects condition-gradient analysis, capacity-balanced module construction, and client-specific module selection. This design systematically translates client-side data and resource heterogeneity into executable personalized sub-networks, rather than treating model-size adaptation as an isolated pruning problem.
- The experiments provide strong deployment-oriented evidence across multiple forms of heterogeneity, including label and morphological variation on MNIST, attribute-based feature heterogeneity on CelebA, and CLIP-derived visual conditions on CIFAR-10. The evaluation reports executable MACs, mean client accuracy, per-client degradation, and worst-client performance drop. Measuring actual sub-network computation—rather than relying only on retained parameter ratios—directly supports the paper’s efficiency claims.
- The module-removal test further verifies whether the learned modules capture their intended data conditions, while supplementary analyses examine client-level retention, module granularity, resource usage, and selection–test consistency. Together, these analyses provide useful mechanistic and robustness evidence beyond aggregate accuracy comparisons.
Critical weaknesses
- The central concentration objective appears mathematically inconsistent with its stated purpose. Equation (2) defines as negative normalized entropy, while is minimized. Therefore, minimizing the displayed objective appears to encourage high-entropy, diffuse responses rather than concentrated ones. The authors should clarify the exact implementation and indicate whether the sign in Equation (2) is a typographical error.
- The scalability of the method has not been sufficiently established. As the numbers of structural units, metadata conditions, and clients increase, the associated computational and communication overheads may grow substantially. Experiments limited to 10 clients and ResNet-18 are insufficient to demonstrate that MDMP remains practical for larger models and larger-scale federated learning systems.
- The assumption that metadata-conditioned gradient responses reveal functional modularity is not sufficiently justified. Although the simulation provides some evidence of gradient-response “disentanglement,” the paper does not clearly explain why such patterns should correspond to meaningful functional modularity in actual neural networks. A stronger theoretical motivation or empirical validation of this connection is needed.
- The algorithm is difficult to reproduce because several key definitions and implementation details are underspecified. The paper introduces numerous symbols and multiple optimization stages without providing a concise and unified technical description. A clear algorithmic summary, together with precise definitions of the main variables, objectives, and optimization steps, would substantially improve reproducibility.
Questions for authors
- Equation (2) defines , while is minimized. For a binary condition, a uniform response gives , whereas a one-hot response gives . Thus, minimizing the displayed objective appears to maximize entropy rather than concentrate responses. Is the sign in Eq. (2) incorrect, or is a maximization step used in the implementation?
- Since the proposed MDMP is based on clients’ metadata, it is unclear how they can get such information. Please clarify how metadata is obtained at clients in practical deployment, and indicate which metadata are assumed available without an additional recognition model.
- The primary feature set is described as selecting “at most” . If all responses are zero, or if several conditions have identical responses, the set may be empty. How are empty sets handled in implementation?
- Since there exist numerous mathematical symbols in the paper, which causes the method MDMP hard to follow, could the authors provide an overall technical scheme for MDMP?
- The number of modules , response threshold , feature-selection limit , coverage parameter , and capacity factor can all affect the final sub-networks. The paper provides an ablation on module-count for CelebA, but it is unclear whether the conclusions are stable on MNIST and CIFAR-10. A sensitivity study or ablation table would help separate robust behavior from dataset-specific tuning.
5 comments
Response (Part 2/3)
R:** Thank you for the suggestion. The module construction of MDMP is based on condition loss sensitivity, response similarity, and module removal verification. BIMT [1] demonstrates instances where neural networks, under appropriate training constraints, form interpretable modules, share features, and collaborate for prediction, providing empirical evidence for organizing modules based on internal computational functions.
Sensitivity and grouping basis: Appendix indicates that under the same small perturbation budget, the maximum first-order change of the conditional loss is . Therefore, the normalized conditional gradient can describe the local sensitivity of the structural unit; Taylor pruning [2] and Task2Vec [3] also study the relationship between tasks and structures by utilizing the derivatives of losses. For unit vectors, maximizing the cosine similarity is equivalent to minimizing the squared distance (Appendix ), enabling the structurally similar conditional responses to form functional modules.
Prediction contribution verification: Appendix further compares the conditional losses before and after the removal of the module. If removing a module leads to an increase in the loss under condition , it indicates that when other modules are retained, this module is helpful for the prediction of that condition.This links the local sensitivity described by the gradient with the actual predicted changes.
[1] Liu et al. Seeing Is Believing: Brain-Inspired Modular Training for Mechanistic Interpretability. Entropy, 2024.
[2] Molchanov et al. Pruning Convolutional Neural Networks for Resource Efficient Inference. ICLR, 2017.
[3] Achille et al. Task2Vec: Task Embedding for Meta-Learning. ICCV, 2019.
Weakness 4 & Question 4: The algorithm is difficult to reproduce because several key definitions and implementation details are underspecified. The paper introduces numerous symbols and multiple optimization stages without providing a concise and unified technical description. A clear algorithmic summary, together with precise definitions of the main variables, objectives, and optimization steps, would substantially improve reproducibility. Could the authors provide an overall technical scheme for MDMP?
R: Thank you for the suggestion. To reduce the cognitive burden caused by mathematical symbols, we have provided the overall framework diagram and corresponding explanations in Appendix . The MDMP consists of three parts:
(a) functional module construction, which starts from a pretrained federated global model and performs response calibration, conditional-response measurement, and module partitioning in advance; (b) Client sub-module determination, which selects modules from the existing library based on client's metadata distribution and MACs budget; (c) Participation by resource-constrained clients, which assigns submodels that satisfies MACs budget to clients, enabling local model training. Note that in the above three steps, step (a) and (b) should be "trained" on some clients in advance before later deployment in FL to allow resource-constrained clients to participant.
Question 2: Since the proposed MDMP is based on clients’ metadata, it is unclear how they can get such information. Please clarify how metadata is obtained at clients in practical deployment, and indicate which metadata are assumed available without an additional recognition model.
R: Thank you for the suggestion. This paper primarily utilizes the existing labeled or directly calculable attributes. When such attributes are unavailable, they are estimated using the model.CelebA directly reads the official attribute annotations and uses 39 attributes other than the task objective "Smiling" as metadata; MNIST uses six morphological measurements from Morpho-MNIST as metadata, and new samples can be calculated through the corresponding algorithm.CIFAR-10 reuses the eight attribute word list made public by LM4CV. It calculates the matching scores for each image through the frozen CLIP, and then constructs high/low conditions based on the threshold determined by the training set. Therefore, it requires additional model inference.
For new datasets lacking pre-existing attributes, a process based on the LM4CV approach can be adopted: LLM generates candidate attributes based on the category description → freeze CLIP to extract text-image features → learn K attribute directions using the training set category labels, match the candidate text and filter the attributes → calculate the matching scores for each image and discretize them into metadata.
Question 3: The primary feature set is described as selecting “at most . If all responses are zero, or if several conditions have identical responses, the set may be empty. How are empty sets handled in implementation?
Response (Part 3/3)
R: Thank you for the question. The main condition set is selected in descending order of module relative response . "At most " specifies the maximum set size. Sort all candidate conditions in descending order of response, and handle zero responses and tied responses using a fixed index sequence.
Specifically, stable sorting preserves the predefined condition order when responses are equal and returns conditions. Here, is the total number of candidate conditions, and the code parameter major_max corresponds to . If all input responses are zero, the fixed index order determines the condition labels. The corresponding code can be found in the file "mnist_resnet_modules_code/mnist_resnet_modules/modules.py" in the directory "MNIST_experiments", specifically on lines 134 and 140.
Before module construction, a numerical validity check is also performed: if the maximum row norm of the response matrix does not exceed , the code will report an error and terminate the process. The corresponding code is located in "mnist_resnet_modules_code/mnist_resnet_modules/modules.py" in the directory "MNIST_experiments", specifically on lines 100 and 101.
Question 5: The number of modules , response threshold , feature-selection limit , coverage parameter , and capacity factor can all affect the final sub-networks. The paper provides an ablation on module-count for CelebA, but it is unclear whether the conclusions are stable on MNIST and CIFAR-10. A sensitivity study or ablation table would help separate robust behavior from dataset-specific tuning.
R: Thank you for the suggestion. We conducted additional experiments on the number of modules for MNIST and CIFAR-10, as well as a sensitivity analysis of the remaining parameters on MNIST.
Module count : Table 2 compares the results of MNIST and CIFAR-10 under different numbers of modules and two levels of MACs Budget. Acc. represents the average test accuracy range of the client, and All-client feasible is the number of configurations where all clients meet the budget constraint divided by the total number of configurations.
Table 2. Module-count sensitivity. \begin{array}{llcrr}\text{Dataset} & M & \text{MACs Budget} & \text{Acc. (\\%)} & \text{All-client feasible} \\\\ \hline \text{MNIST} & 4,6,10,12 & 95\\% & 97.57\text{–}99.44 & 4/4 \\\\ \hline \text{MNIST} & 4,6,10,12 & 90\\% & 97.03\text{–}98.90 & 3/4 \\\\ \hline \text{CIFAR-10} & 12,14,16,18 & 95\\% & 86.80\text{–}88.13 & 4/4 \\\\ \hline \text{CIFAR-10} & 12,14,16,18 & 90\\% & 85.38\text{–}87.00 & 4/4 \\\\ \end{array}
Among the 16 new configurations, 15 achieve full client budget-based allocation, with the average accuracy drop not exceeding . It supports effective accuracy at different module granularities - a trade-off between computational cost and accuracy. The complete results can be found in Appendix I.9 Table .
Other parameters:** Table 3: Fixed on MNIST, default , , and . . Change the parameters item by item. Values represent the parameter values, Acc. indicates the average test accuracy range under the corresponding MACs budget, Feasible reports the number of feasible clients in order of values, and there are 10 clients in each group.
**Table 3. Parameter sensitivity on MNIST. \begin{array}{llrrl}\text{Parameter} & \text{Values} & \text{Acc (95\\%)} & \text{Acc (90\\%)} & \text{Feasible (90\\%)} \\\\ \hline \tau_{\mathrm{resp}} & 10^{-6},0.01,0.05 & 97.46\text{–}99.02 & 96.46\text{–}98.70 & 10,9,2 \\\\ \hline \rho_j^{\ell}=c/J_{\ell} & c=0.25,0.5,0.75 & 98.76\text{–}99.48 & 96.98\text{–}97.73 & 10,10,5 \\\\ \hline \alpha & 1.5,2,2.5 & 98.76\text{–}99.20 & 95.67\text{–}97.24 & 10,10,9 \\\\ \hline s_{\max} & 2,3,4,5 & 97.56\text{–}98.04 & 96.68\text{–}97.22 & 10,10,10,10 \\\\ \end{array}
Under the MACs budget, all configurations achieve / feasible clients, with the average accuracy ranging from to . Under the MACs budget, the average accuracy ranges from to , and all settings maintain 10/10 feasible clients.
Response (Part 1/3)
Thank you to the reviewers for your valuable feedback on our paper. We sincerely thank you for taking the time to review our work, and we are grateful for your insightful comments.In this rebuttal, we address the key concerns raised and seek to clarify the relevant issues while incorporating the suggestions to improve our paper.
Weakness 1 & Question 1: The central concentration objective appears mathematically inconsistent with its stated purpose. Equation (2) defines Lcon as negative normalized entropy, while Lcal is minimized. Therefore, minimizing the displayed objective appears to encourage high-entropy, diffuse responses rather than concentrated ones. The authors should clarify the exact implementation and indicate whether the sign in Equation (2) is a typographical error.
R: Thank you for raising this question. We verified Eq. (2): the outer minus sign makes the expression normalized entropy, and minimizing it promotes response concentration. The implementation directly minimizes the calibration objective containing this entropy term.
For structural unit q and single-valued factor ℓ, the normalized response entropy is Hqℓ=−log∣Jℓ∣∑j∈Jℓϕq,jℓ,callogϕq,jℓ,cal We have $\mathcal{L}_{\mathrm{con}}^{\mathrm{cal}} =\frac{1}{Q|\mathcal{I}_s|}\sum_{q=1}^{Q}\sum_{\ell\in\mathcal{I}_s}H_q^\ell$ after taking its average to Eq (2). Since the uniform response ϕ=(1/2,1/2), we have
Hqℓ=−log221log21+21log21=1, Then, the one-hot response ϕ=(1,0) is Hqℓ=−log21log1+0log0=0. Therefore, −1 and 0 are the results when the outermost negative sign is not taken into account. According to Eq (2), they should be 1 and 0 respectively when calculated completely. Since Lconcal is included in Lcal with a positive weight, minimizing the concentration term in this objective will encourage more concentrated conditional responses.
Weakness 2: The scalability of the method has not been sufficiently established. As the numbers of structural units, metadata conditions, and clients increase, the associated computational and communication overheads may grow substantially. Experiments limited to 10 clients and ResNet-18 are insufficient to demonstrate that MDMP remains practical for larger models and larger-scale federated learning systems.
R: Thank you for suggesting that we add more extensive validation. On the MNIST, CelebA and CIFAR-10 datasets, we increased the number of clients from 10 to 20 and re-executed the entire process of MDMP.
In Table 1, Global and MDMP represent the average test accuracy of the calibrated global model and the assigned sub-models respectively, Δ is the difference between the two, and Feas is the number of clients that meet the accuracy and MACs budget. Under the 90% MACs budget for MNIST, one client uses the complete model, and its accuracy is included in the average value, but it is not counted as feasible within the MACs budget.
Table 1. MDMP with 20 clients. \begin{array}{lcrrrc}\text{Dataset} & \text{MACs Budget} & \text{Global (\\%)} & \text{MDMP (\\%)} & \Delta & \text{Feas} \\\\ \hline \text{MNIST} & 95\\% & 99.69 & 99.23 & 0.47 & 20/20 \\\\ \hline \text{MNIST} & 90\\% & 99.69 & 97.09 & 2.61 & 19/20 \\\\ \hline \text{CelebA} & 95\\% & 92.51 & 92.50 & 0.01 & 20/20 \\\\ \hline \text{CelebA} & 90\\% & 92.51 & 92.50 & 0.01 & 20/20 \\\\ \hline \text{CIFAR-10} & 95\\% & 83.73 & 82.59 & 1.14 & 20/20 \\\\ \hline \text{CIFAR-10} & 90\\% & 83.73 & 81.20 & 2.53 & 20/20\end{array}
The results show that under a 95% MACs budget, all 20 clients in each of the three datasets obtained feasible sub-models, with the average accuracy decreasing by only 0.47, 0.01, and 1.14 respectively; under a 90% MACs budget, 19/20, 20/20, and 20/20 clients were feasible respectively, and the average accuracy decreased by no more than 2.61. This indicates that even when the number of clients increases to 20, MDMP remains practical.
Weakness 3: The assumption that metadata-conditioned gradient responses reveal functional modularity is not sufficiently justified. Although the simulation provides some evidence of gradient-response “disentanglement,” the paper does not clearly explain why such patterns should correspond to meaningful functional modularity in actual neural networks. A stronger theoretical motivation or empirical validation of this connection is needed.