- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies how to reuse multiple learnwares when each one only partly meets the user's needs. It proposes Distribution Filling, which reweights points within the specifications to match the user distribution. The resulting weights are combined with local compatibility to select a model or combine predictions for each input. Experiments on tabular data, images, and sales prediction show improvements over existing learnware reuse methods.
Critical strengths
-
The problem is clear and the method is easy to follow. The paper considers cases where each learnware only partly meets the user's needs. It reweights points within the specifications to combine useful parts from different learnwares and reduce the influence of irrelevant regions. It only needs to solve a convex optimization problem once for each user task, without training an extra selector.
-
The experiments are extensive, which cover tabular data, images, and a sales prediction task. The paper also includes ablation studies, parameter analysis, and runtime comparisons. These experiments provide useful support for the method.
Critical weaknesses
-
The contribution of pointwise filling to the overall improvement is unclear. The method shows large gains over JobSelector, but only gains 0.14–0.71 percentage points over the simple baseline using local compatibility alone. The paper compares whole-specification and pointwise weighting on some datasets, but this direct comparison is missing from the main benchmark.
-
The link between distribution matching and prediction ability is not well established. A model may cover an input region but still make poor predictions there. The paper acknowledges this difference, but does not sufficiently explain when the distribution-based weights are reliable and when they may fail.
Questions for authors
-
Could you compare whole-specification weighting plus local compatibility with pointwise weighting plus local compatibility on the main benchmark? This would help clarify the contribution of pointwise filling.
-
Under what conditions does distribution coverage provide useful evidence of prediction ability? Could you discuss a case where this relationship may fail and how it would affect the method?
-
Besides the accuracy–cost trade-off metioned in the paper, are there other practical considerations for choosing DF–Select over DF–MoE, for example when the candidate models’ probabilities are not well calibrated?
4 comments
Response to Reviewer vN88 (Part 2/2)
Continuation of response 2
2. "The link between distribution matching and prediction ability is not well established."
The first term bounds weighted regional prediction error; measures reuse-weight mismatch. The extra DF–Select term accounts for committing to one model. DF–MoE has low convex-loss risk when regional errors and weight mismatch are small; DF–Select has low risk when its weights are also concentrated. With predictive utility defined as one minus prediction loss, the DF–Select bound also gives conditions for high achieved utility within the framework of Section 4.1.
As an illustrative failure case, suppose the same input region has label A in the source but label B for the user. Its input distribution can match perfectly while the source classifier still predicts A. DF may then select that model or assign substantial fusion weight to its incorrect output. The model's regional error is then large despite a good input match. We added the assumptions, risk bounds and their interpretation to Section 4.4 (Assumptions 1–2, Theorem 1, Eqs. (11)–(13)) of the revision, with the proof in Appendix C.2.
3. "Could you compare whole-specification weighting plus local compatibility with pointwise weighting plus local compatibility on the main benchmark?"
Thank you for this suggestion. We added a paired experiment on reconstructed main-benchmark tasks comparing whole-specification weighting (Whole-local) with pointwise weighting (DF–MoE), holding candidates, local compatibility and probability fusion fixed. Accuracy (%) is averaged over 17 tabular or 13 image datasets.
| Setting | Whole-local | DF–MoE | Gain (pp) |
|---|---|---|---|
| Tabular / Subsumption | 84.51 | 89.20 | +4.70 |
| Tabular / Instance recurrence | 81.34 | 82.44 | +1.09 |
| Image / Subsumption | 89.21 | 89.88 | +0.68 |
| Image / Instance recurrence | 75.64 | 75.75 | +0.10 |
DF–MoE improves over Whole-local in 45/60 dataset-scenario pairs, with the largest mean gain under tabular subsumption. This supports the benefit of adjusting useful and irrelevant source regions separately while keeping local compatibility fixed. We added this comparison to Appendix G.2 (Table 9) of the revision.
4. "...are there other practical considerations for choosing DF–Select over DF–MoE, for example when the candidate models’ probabilities are not well calibrated?"
As described in Section 4.3 (Eqs. (9)–(10)), DF–Select chooses a model using input-distribution information and returns its class prediction, whereas DF–MoE combines aligned probabilities. Reliability of the candidates' probabilities is therefore an additional practical consideration.
We tested a simple overconfidence setting across all 60 main-benchmark dataset-scenario pairs. We first temperature-scaled candidate probabilities using labeled samples separate from evaluation, then increased all candidates' confidence by dividing their temperatures by four, keeping their predicted classes and reuse weights fixed. Results are averaged over all 60 pairs.
| Probability setting | DF–MoE accuracy (%) | DF–Select accuracy (%) |
|---|---|---|
| Before confidence increase | 85.15 | 81.48 |
| After confidence increase | 84.75 | 81.48 |
Increasing confidence reduces DF–MoE accuracy by 0.40 percentage points while DF–Select remains unchanged. DF–Select is therefore a practical option when class predictions are trusted but confidence values are unreliable.
We hope these clarifications and revisions address your concerns.
Thank you for your detailed response. It has addressed my concerns. I will keep my rating at 3.
Thank you for reviewing our responses and confirming that they have addressed your concerns. We appreciate your time and constructive feedback, which helped us improve the paper.
Response to Reviewer vN88 (Part 1/2)
Thank you for recognizing the clarity and soundness of our work and the extensive evaluation, and for the constructive suggestions on clarifying DF's contribution and use. We address each concern below.
As shown in the original Table 9, DF–MoE improves over Uniform-local in 49/60 dataset-scenario pairs. The original Fig. 4 and Appendix G.2 further compare pointwise and whole-specification weighting at fixed local compatibility, with gains of 18.40/7.36 points on CJS and 1.95/0.84 on EuroSAT (subsumption/instance recurrence).
Learnware pools accumulate models developed independently for different historical tasks, so the prevalence of a capability in the pool need not reflect its importance to a new user. This mismatch arises naturally across sites or operating regimes: in the original Fig. 1, the factory needs complementary low- and high-load capabilities despite overlapping source coverage of medium-load conditions.
To clarify DF's contribution to complementary reuse, we conducted an additional experiment on 13 tabular datasets with five candidates over six classes A–F. Each of D and F is covered by only one candidate; the other classes by several. We varied total D/F demand, split equally, while fixing predictions and local compatibility. Results are dataset-averaged accuracy (%) and paired gains (pp), reported as mean ± sample SD over three seeds.
These results show that DF adds substantial value beyond local compatibility when demand concentrates on reliable capabilities supplied by few learnwares. Local compatibility indicates where a model is applicable; DF adapts its task-level contribution to user demand and coverage supplied by other candidates. Here, DF raises the D/F candidates' weights, directing local fusion toward their reliable complementary predictions. We added the results and mechanism analysis to Section 5.5 (Fig. 4) and Appendix G.3 (Fig. 9 and Table 13) of the revision.
Section 4.1 (Eqs. (2)–(3)) values complementary capabilities, motivating DF's pointwise allocation and local compatibility. Proposition 1 explains the allocation mechanism. Our added analysis bounds the prediction loss of the final DF–Select and DF–MoE rules using the fitted reuse weights in Eqs. (8)–(9). It separates regional model error, reuse-weight mismatch, and an additional term for selecting one model, making explicit when complementary coverage supports reliable prediction.
We assume (i) the user density is a mixture pu=∑iaipirel of relevant source regions, which may overlap; and (ii) model i has expected loss at most ϵi on its region under the user's label relationship, for a loss in [0,1]. The first assumption captures tasks assembled from portions of source tasks. The second assumption is reasonable when models perform well on the relevant source regions and the input–label relationships remain stable from source to user, so that their regional competence can transfer to the user task.
The reference weight ri†(x)=aipirel(x)/pu(x) measures region i's share at input x. Let δ=EX[21∑i∣ri(X)−ri†(X)∣] measure the discrepancy of the actual DF weights from these reference weights. Theorem 1 bounds DF–Select for any such loss, including zero–one loss:
RSelect≤∑iaiϵi+δ+EX[1−maxiri(X)].
For aligned outputs in a common convex space and a convex loss, such as normalized Brier loss for probability fusion, the theorem also gives:
RMoE≤∑iaiϵi+δ.
Response 2 continues in Part 2.