- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper addresses the problem of selecting and sharing pretrained models under risk constraints imposed on individual data groups (contract cells). Using a separate (local) model for each cell can lead to substantial storage and deployment costs, whereas sharing models across multiple cells can potentially reduce these costs but introduces the risk of degraded performance on individual data groups. In distributed systems, using a single model for all data groups does not guarantee compliance with risk constraints, while deploying local models for every group may be expensive. The authors formulate the problem of determining which models can be shared across which groups and at what cost. The proposed approach operates at deployment time and uses a pool of trained candidate models. Based on certification results, the method selects a subset of shared models and assigns them to admissible cells. The authors propose a method for grouping cells and assigning a common model to each group based on K-means clustering.
The considered problem is decomposed into two subproblems: selecting a subset of models and assigning the selected models to individual cells. The first subproblem is combinatorial, and the authors propose using exhaustive search or greedy algorithm to solve it. For the second subproblem, the selection of the least expensive certified model for each cell is proposed.
The practical significance of the proposed approach lies in its potential to reduce the costs of storing and deploying models while preserving formal risk constraints for the data groups being served. The problem under consideration is how to certifiably allocate a limited set of models across groups under risk constraints with minimal deployment costs. This direction is relevant to machine learning systems in which reliability requirements are specified separately for different data segments and storage and serving resources are limited.
The scientific novelty of the work lies in formulating the problem of certified model sharing under per-group risk constraints and proposing the CSDS (Certified Shared Decoder Selection) procedure for it. The certification techniques used (Hoeffding risk bounds), candidate-pool generation methods, shared-model subset selection methods, and model to cell assignment procedure themselves are not novel.
The problem considered is practically relevant and the paper presents a mathematically well-structured framework for certified model sharing under per-cell risk constraints. Its value is the formalization of the deployment-time model selection problem and the theoretically justified decomposition of certification into individual cell-action pairs. However, the methodological novelty remains insufficiently established. In particular, the experiments do not demonstrate the evidence that the proposed architecture offers an advantage due to the new algorithmic solutions, and not due to the problem statement itself, the certification requirement, or the quality of the candidate pool.
Overall, the paper presents a coherent and potentially useful framework, but the extent of its contribution beyond the integration of existing certification and optimization techniques remains open. It would be more convincing if it included more evidence demonstrating the clear advantages of the CSDS, as well as a more systematic analysis of the factors determining its efficiency.
Critical strengths
- A rigorous and practically relevant problem formulation.
- A clear decomposition of the CSDS framework into distinct stages: candidate pool generation, statistical certification of the cell-model assignments, and cost-aware selection of the model for its assignment to cells.
- A theoretically justified certification strategy. The authors show that the risk constraints and the objective function are separable across contract cells. This allows the feasibility of an deployment configuration to be guaranteed by certifying individual cell-action pairs.
- A formal proof of optimality for the assignment subproblem. The authors establish that, for a fixed subset of shared models, the optimal assignment can be computed in polynomial time by independently selecting the least costly feasible model for each cell.
Critical weaknesses
-
It remains unclear how significant the combination of known methods in the proposed CSDS architecture is. In particular, the paper could better clarify which aspects of the proposed approach go beyond a direct application of existing principles of statistical certification and risk control to the model-sharing problem.
-
The benefit of subset optimization over simpler selection policies is not demonstrated. The experimental results show no advantage of the proposed subset-selection procedure over a simpler fixed-K policy on the same candidate pool: both achieve identical results in the reported comparisons. So, the practical or theoretical benefit of optimizing the number of shared models remains unclear. The authors should discuss about the main contribution whether lies in the subset-selection algorithm itself or in candidate generation and certification.
-
The paper does not address how the proposed certification guarantees and deployment decisions would behave under distribution or conceptual shift. In particular, it remains unclear how the statistical guarantees should be adjusted, whether model assignments should be updated dynamically, and how such updates could be certified under risk constraints.
-
The performance of CSDS appears to depend substantially on the quality and diversity of the source candidate pool generated before deployment. Since CSDS selects and assigns existing predictors rather than improving their predictive capabilities, the CSDS quality is inherently constrained by the candidates. However, the paper does not investigate how the CSDS performance depends on the performance of the candidate models, particularly weak models. The additional experiment should distinguish the effect of individual model performance and effect of model diversity within the pool.
-
It is unclear how the proposed cost function in Equation (2) relates to actual costs in industrial deployment. In particular, the relationship between the theoretical coding-cost objective and practical considerations such as storage requirements, inference latency, computational resources, and serving costs should be clarified. Without such an analysis, the CSDS grounding to industrial deployment infrastructure remains uncertain.
-
In the CSDS, a cell is admitted for service only if its local model is certified. If the local model fails certification, the cell is referred for further processing even when a shared model has been successfully certified for that cell. Such restriction is the authors’ choice and is not a shortcoming of the work. Nevertheless, an ablation study relaxing this restriction would be valuable. In particular, how allowing a cell to be served whenever at least one shared model is certified for it affects service coverage, deployment cost, and the resulting risk guarantees.
-
Insufficient practical guidance on where the architecture's improvements should be focused. From the perspective of researchers seeking to apply CSDS to their own problems, it remains unclear which component deserves the greatest attention: candidate-pool generation, the choice of statistical certification procedure, or the optimization algorithm used to select the shared models. The paper would benefit from more explicit practical recommendations, supported by the experimental results, on which components are most influential and are likely to yield the greatest improvements.
Questions for authors
How important is the combination of known methods in the proposed CSDS architecture?
What measurable benefits does the subset optimization procedure provide?
What assumptions underlie the statistical certification guarantees when the data distribution changes after deployment? Does CSDS require recertification before updating model assignments?
Since CSDS selects and assigns existing predictors rather than improving their predictive capabilities, how sensitive are its results to the quality and diversity of the initial candidate pool?
What is the intended interpretation of the cost function in Equation (2) in terms of actual deployment infrastructure? Is it intended as a theoretical proxy for deployment complexity?
See Critical Weaknesses for more details.
2 comments
Thank you for the response and the additional analyses. The new experiments and clarifications address several of my concerns and make the paper better and the contribution of CSDS clearer. The additional comparisons of subset-selection strategies clarify the role of the optimization component that subset-selection procedure is not the primary source of the reported improvements. So the main contribution lies primarily in candidate generation and statistical certification, rather than in the subset-selection algorithm itself. If so, my concerns are related to the methodological novelty, contribution to the new knowledge and the limited evidence regarding candidate-pool quality and practical deployment costs. The paper is well-organized but I'm not sure that the theoretical and methodological ideas are sufficient for the conference. So, my decision is still between weak rejection and borderline.
Response to Reviewer w4WE: component ranking, admission ablation, candidate-pool analysis
Dear Reviewer w4WE,
Thank you for the careful and well-organized review. It prompted three analyses that we think settle your main questions: a ranking of components, an ablation of local admission, and a test of candidate-pool quality and diversity.
Components and subset optimization (W7, W2/Q2). In order of size (bits/sample, Covertype unless stated): candidate columns 1.7–2.2 (choosing by tune risk rather than cost gain; random columns 0.9–1.2); certificate and admission 21–35% of cost on Covertype and 37–41% on Electricity; subset search at most 0.18; Holm instead of Bonferroni at most 0.03. The fixed-K policy in the submitted paper coincided with exact enumeration, so it could not show a benefit; we replace it with controls that do not enumerate: greedy keeps 99.6% of the exact solver's gain over the tuned router (+2.699 vs +2.708; same subset in 123/150 runs; ~3 ms vs ~0.1 s) and a frozen ranking 94.4%. The main contribution therefore lies in candidate generation and certification.
Known methods (W1/Q1). We do not claim a new statistical tool. What goes beyond direct application is cell separability: m(K+1) pair-level bounds cover every deployment plan at once, so subset choice and assignment can use the calibration data freely, with local admission inside the same failure budget.
Local admission (W6). Thank you for this ablation; it produced a clear improvement. Letting a cell without a certified local predictor be served by a certified shared action, with voluntary referral, is never worse than the original rule at referral prices 4, 8, 16: 0.12–0.65 bits lower on Covertype and 0.76–1.21 on Electricity, with no held-out violation among the newly served cells (1–2% and 7% of cells); only certified pairs are served, so the guarantee keeps its form. We recommend it as the default.
Candidate pool (W4/Q4). Selection by lowest tune risk costs 1.7–2.2 bits more than ranking by cost gain (3.0 on Electricity), random columns 0.9–1.2 more, and a coverage rule meant to add diversity performs like the tune-risk rule: what repairs the dependence is the ranking criterion, not diversity. Deliberately degraded candidates are a natural extension.
Cost function (W5/Q5). Eq. (2) is a description-length proxy: bits for the shared and local models, the routing index and the data, plus a referral price κ; latency, compute and serving cost are not modeled. On Covertype CSDS stays ahead at every price we tried; on Electricity up to κ≈16, and 18.7 with voluntary referral.
Distribution shift (W3/Q3). The guarantee assumes calibration and deployment data are exchangeable within each cell; after a shift, assignments are recertified on fresh labelled calibration data, which is cheap (m(K+1) bounds, millisecond selection). Descriptively, served pairs have median margin τ− test risk ≈0.15 (Covertype) and 0.23 (Electricity), and the smallest uniform risk shift producing a first violation is ≈0.06 and 0.11; this measures slack, not robustness.
Main results. On Covertype, CSDS beats the tuned router in all 30 seeds of each setting, by 2.68–3.70 bits, with no held-out violation under the headline certificate. Electricity, with the least calibration data per cell (about 130 samples), is the closest comparison and the one where the tighter certificate helps most (37–41%). After recomputing the main tables from the cached calibration artifacts, CSDS uses about 56% fewer bits than all-local deployment there (all 30 seeds) and about 15% fewer than the tuned router (+0.708 bits, 25 of 30 seeds), and is level with the certified MoE router (0.090 bits, 18/30).
Thank you again for the detailed engagement.