- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper proposed a variational-EM energy model for weak multi-instance multi-label learning where bag instances can be incorrect while at the same time label also can be incorrect. The proposed method conducts label denoising, instance denoising, and correlation denoising to combat incorrect label and instance assignment during learning.
Critical strengths
(1) This paper provided their clear research position that doubly degraded weak multi-instance multi-label learning is realistic but understudied field.
Critical weaknesses
(1) Experimental setting is both shallow and narrow, and less credible. First, it seems that the authors flipped each label independently according to label flip rate. This setting ignores the label correlation in multi-label setting, so it is far from realistic noise but rather artificial noise. It is doubtful whether this experiment result could be generalized into general case or real case. Second, several baseline methods (e.g., SCE and GCE) modified cross entropy loss thus it is specialied for single-label learning, not multi-label learning. However, implementation details are missing so it is doubtful whether fair comparison was actually made.
(2) This paper failed to prove the efficacy of their proposed method. In Table 1, the authors did not report all 20 baseline methods exhaustively. It is not enough to report just the best case. Also, it could be observed that the proposed method did not show remarkable performance gain compared to best case baseline. What is more, in some datasets, the performance gain of VEM diminishes when label noise rate increases. This results is contradictory since VEM is proposed to combat wMIML scenario, but the authors did not provide any reason nor analysis about this. Furthermore, ablation results in Table 2 and Figure 2 indicate that the components in their proposed method do not leads to performance improvement. Where does the overall performance indeed come from?
Questions for authors
See weaknesses
3 comments
Response to Reviewer 6fxd (2/3)
1. SCE/GCE
Both were designed for single-label classification. We use them on per-label sigmoid binary tasks: GCE or SCE is computed independently on each of the label positions (GCE additionally weights positives by ), never as a -way softmax, and the losses are averaged over positions; the other baselines we use per position (CUE, BML, JAL) follow the same pattern. Concretely, on position the logit gives GCE with and on the positive side, and SCE with , averaged over the positions. The revision will list both formulas and their weight settings in the appendix.
2. Fairness of the comparison
Fairness also depends on the tuning budget. We gave the baselines extra budget: four (lr, hidden width) grid points for GCE and MaxPool each, selected on validation. Taking the strongest baseline per dataset, three of the five datasets are no better than the original value, and the average gain is only 0.46 points, so is already close to their optimum. After tuning, VEM still leads by 2.5 to 3.0 points on bibtex, CAL500 and LetterCarroll, while on Reuters and enron the lead narrows from to . The main-table lead therefore does not come from under-tuned baselines, but we report honestly that these two cells are tuning-sensitive. One asymmetry we state ourselves: per-dataset configuration gives VEM points (Appendix Table 6), larger than the for the baselines; the gain concentrates in the five datasets with a structural mismatch, and although each direction comes from the diagnosis in advance, VEM is indeed tuned one level more than the baselines, and we do not report only the half that favours us. At the same time, with no per-dataset selection at all, the unified configuration already beats the best baseline in 8 of 30 settings and wins 208 of 360 pairwise comparisons against the 12 deep baselines; per-dataset selection raises this to 20/30 and 314/360, and each intervention follows a structural statistic (a deeper encoder for sparse features, a lower Hill weight when the support is uninformative, a rebalanced positive weight under saturation) rather than a grid search.
3. Table 1
For space, the main text lists only the strongest baseline per setting; the complete 20-method by 30-setting matrix is in Appendix C, and each cell of the main table also names that baseline (Reuters: MaxPool, MaxPool, Hill across the three levels), so the change of winner is readable from the main table itself. The strongest baseline is not one method: 23 cells without ties are shared by six methods (GCE 9, MaxPool 7, MLDN 3, MIMLVAE 2, Hill and KISAR 1 each), and the other 7 cells have several methods tied for the maximum (13 to 14 at each OpenMIC level, 2 to 3 at each Scene level, 12 at emotions 0.5); counting ties in full, the optimal set across 30 cells involves 15 methods. The winner changes with the level on three benchmarks (Reuters from MaxPool to Hill, bibtex from MaxPool to KISAR, FashionMNIST from MIMLVAE to MLDN), and Scene is tied at all three levels. We therefore compare against the per-cell best baseline rather than requiring a clean sweep.
4. Size of the gain
Under a common 0.5-point tolerance (the 20/30 in the paper counts ; here the same tolerance is applied to the original model and to the concept-head version), the original model wins 14 cells, ties 12 and loses 4, with a sign test . On the three benchmarks that keep their lead after fair tuning (bibtex, CAL500, LetterCarroll) the lead at is 2.5 to 3.0 points. Adding the concept-level posterior branch of 4.7 raises the mean margin over the strongest baseline from to , with cell counts 17/9/4 against the best baseline (13/14/3 against the original VEM) and ; the pooled mean rises from to points. These are not single runs: Appendix Table 8 gives three-seed means and standard deviations for all 30 main-table settings (maximum ), and the concept head uses two seeds per level (Appendix G), with a median per-cell of 0.003 and a maximum of 0.011 at level 0.3; the conclusions at the level of means are unchanged. The branch is reported separately as an extension and does not replace the main table, so that the 30-cell comparison and the ablation anchor stay intact; if you consider it should become the main method, we will replace the main table and re-run the dependent analyses in the final version.
Response to Reviewer 6fxd (3/3)
1. Gain versus noise level
The observation that the gain diminishes as the noise level rises holds on three benchmarks, and we do not avoid it: LetterCarroll falls from to , enron from to , and Corel5K from to ; bibtex holds only partially, from to and then flat. This is not a contradiction: the claim is superiority over the per-cell best baseline at the same noise level, not that the margin grows monotonically with noise. Stacking the ten curves hides three other regimes: OpenMIC has 13 to 14 methods tied at every level and emotions differs by less than 0.05 points, so these two are ceilings where zero gain reflects no room rather than decay; Reuters ( to ) and CAL500 ( to ) move the other way, with the margin widening; FashionMNIST lags throughout, from to , which is a structural mismatch. From 0.3 to 0.5, on LetterCarroll VEM goes from 0.558 to 0.397 and the best baseline from 0.531 to 0.391; on enron 0.641 to 0.599 against 0.624 to 0.594; on bibtex both drop by 0.097; on Corel5K VEM drops from 0.392 to 0.360 while the baseline rises slightly from 0.385 to 0.387. The baselines do not become stronger; VEM degrades slightly faster. We have no mechanistic explanation for the faster degradation and report it as it stands. The revision states the coverage condition in 3.4 and separates the two boundary cases, support saturation (CAL500) and single-concept structure (FashionMNIST), in 4.4.
2. Ablation
Apart from the full VEM, Table 2 of the original had only three rows (single-energy posterior, no Hill, plus VEM4): the single-energy and no-Hill rows stay within and VEM4 averages . All three were measured with the structure already present, so near-zero effects do not show that the components carry the gain, nor do they answer where the performance comes from: the structure itself was never ablated. We ran that ablation over 30 settings: replacing energy pooling with mean pooling lowers mean F1 from 0.6713 to 0.6416, below the mean of the per-cell best baselines (0.6644), cuts winning cells () from 20/30 to 10/30 and raises cells lost by more than half a point from 4 to 12. The revision reports the three layers separately: structure points (30 cells), support-aware reweighting (9 cells), contrastive term (30 cells). The loss terms have small marginal effects because they are not independent sources of gain: the support term reweights the negative half of reconstruction, the contrastive term estimates the KL already in the bound, and removing one leaves the others approximating the same quantity. Figure 2 is the per-dataset view of the single-energy row, where the dual-energy and single-energy posteriors differ by points (five benchmarks positive, one neutral, four slightly negative); the main text has always treated it as a secondary effect relative to reconstruction.
3. Where the performance comes from
From the energy pooling structure, whose contribution rises monotonically with the label space: points for , for and for , and in each bin the version without the structure falls below the best baseline. The locus of efficacy is therefore the structure, not the marginal terms. We also acknowledge two limits: FashionMNIST remains unsolved, and the revision does not improve it (three levels at , two of them worse than the original); suppressing the max-pooling branch recovers only about 1 point of the 4.3-point gap. And leading on most benchmarks is not a mechanism-level proof: what we provide is that the structure carries the main contribution and that the label-level criterion is identifiable, not a complete causal chain.
4. Presentation
We agree that the paper was hard to follow. The revision adds a Remark in 3.3 separating the modelling choice from the objective, moves the applicability conditions from the analysis into the method section (3.4), rewrites the ablation in 4.5 as a margin attribution and moves the layered and binned tables to the appendix, and adds the realised noise statistics (Appendix F) together with the structural discussion in 4.4.
5. Reproducibility
Every number above can be reproduced; the scripts and result tables will be released with the revision.
Response to Reviewer 6fxd (1/3)
We thank the reviewer for the detailed reading. We address each point below, stating the definition, the evidence and the scope of the claim.
1. Noise construction
The corruption is not symmetric per-bit flipping. For each bag, r is a budget over its positive labels: about 40% are deleted and 60% are moved to a concept that no instance in the bag supports. On the instance side we append 0.5ni real instances drawn from the same corpus and delete none of the originals. Labels are corrupted in training only, and evaluation always uses clean test labels; instance distractors are injected on both sides, each from the pool of its own split, so both see the same noise distribution. Appending rather than substituting is deliberate: substitution would remove genuine evidence and confound instance corruption with label ambiguity, whereas appending keeps the evidence and adds only distractors, which is exactly the coexistence of relevant and irrelevant instances that defines wMIML. At r=0.3 the realised per-bit error rate is 2.5% to 21.3%, and 56.1% to 100% of bags are affected. Together the two channels produce missing and spurious labels, the two errors annotators actually make. The three levels are not arbitrary: the comparable protocols we checked use 10 to 40% (Bai et al., IJCAI'20) and 30 to 50% (Xia et al., ICCV'23), and our 0.2/0.3/0.5 sits inside that range. We stress that this is controlled synthesis, not field-collected noise: its value lies in matching the error types of real annotation and in applying one identical corruption to every method, not in reproducing the noise of a particular deployment.
2. Label correlation
We quantified it. Apart from emotions, whose mean absolute ϕ is 0.125, the other nine benchmarks stay at or below 0.059, and strongly correlated pairs account for at most 1.4%. At that magnitude, per-bit independent sampling and correlation-aware sampling give nearly indistinguishable label distributions. We did try generating correlated noise from the co-occurrence distribution, but co-occurrence in these data is itself weak, so the resulting noise differed little from independent flipping and does not amount to a test of meaningful strength; we therefore report it as a boundary rather than claim coverage. One clarification: we never claimed correlation denoising. The factorized posterior does not model label correlation, as stated in the Limitations; our target is joint denoising on the instance and label sides.
3. Generality
On scale first: ten benchmarks (5 to 374 classes, 800 to 14915 bags), three noise levels and 20 baselines, 30 settings in total, and five metrics (micro-F1, MAP, Hamming loss, one-error, macro-F1). This is a controlled empirical study. A fair comparison requires controlling how noise enters, so that the same construction applies to every method; missing and spurious labels are the two errors annotators actually make, which also makes more prior work comparable. We do include real data. OpenMIC's labels come from the original files, where unsure entries were counted as positive; its 94% coverage (18.8 of 20 labels per bag) is itself real annotation noise, and at that density the all-positive prediction already reaches 0.9696, so the cell offers no room. CAL500 has 8.9 positive labels per instance and near-saturated support. For the same reason, verifying denoising on real noise requires manually verified clean test labels; otherwise one cannot separate gains due to denoising from gains due to fitting the test noise, and such a benchmark does not exist in MIML. Hence, for a fair comparison, we did not add more real datasets.