- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper learns an instance classifier when each bag only has the sum of hidden integer labels, like counts, signed counts or digit sums. FS-Conv turns each instance prediction into a distribution over its contribution. It convolves these into a distribution over the bag total, and trains on the observed total’s likelihood. The convolution is not new, and the authors say so. The pitch is a shared interface. The binary case reduces to known count losses. A toy example shows two models with the same mean and variance that give the observed total different probabilities. Moment-based losses miss this. Over three seeds, FS-Conv is best on MNIST and SVHN digit sums and on signed MNIST counts.
Critical strengths
- The paper gives a clean, modular formulation that separates the instance predictor, the contribution-PMF map , and the convolution backend.
- The analysis explains why the discrete likelihood should behave differently from moment objectives.
Critical weaknesses
- Methodological novelty is limited. The likelihood is the classical convolution of per-instance PMFs. Its Bernoulli case coincides with the CountLoss and LLP-PVC count likelihoods, the known-sign case reduces to a shifted count, and DeepProbLog already showed that digits can be learned from sums, so the contribution rests mainly on the unifying interface and the empirical comparison.
- The vision experiments compare only against two moment-based objectives, with no comparison to DeepProbLog-style exact inference or to the full CountLoss or LLP-PVC pipelines.
- Neither the central factorization assumption nor the broader scope is tested. Robustness to within-bag dependence is not measured, and the only real-world task shows no advantage.
Questions for authors
- How stable are the results with independently resampled image splits and bag sets, ideally with more seeds and paired tests?
- For digit sums, is this the same likelihood DeepProbLog computes exactly? If so, what does FS-Conv add beyond organizing the computation? A direct comparison would help.
2 comments
Thank you to the authors for their responses and additional experiments. While I appreciate their efforts to address my concerns, I believe that the novelty of the proposed method remains questionable. In my view, the contributions and experimental validation are still insufficient to meet the standards of a top-tier conference.
Thank you for the detailed review.
Q1. Stability with resampled splits and bags, more seeds, paired tests. Done: 20 seeds on MNIST and 10 on SVHN, each with its own image split and bags.
FS-Conv also has lower expected-sum MAE and discrete NLL than both baselines in every seed of all three tasks (Table 12). On MNIST, FS-Conv itself is bimodal across seeds (at least 0.95 in ten, 0.70–0.86 in the other ten). These runs use each method's own checkpoint rule and 50 epochs. With one rule for all methods (a second script, 20 seeds) the gap is the same: 0.861 vs 0.706, +0.155 [+0.102, +0.208], 18 of 20 seeds. With 16 times longer training both likelihoods improve and the gap narrows but persists: 0.932 vs 0.849, +0.084, paired t p=0.02, 15 of 20 seeds (Tables 17–18).
Q2 / W2. Is this DeepProbLog's likelihood? Comparison with CountLoss, LLP-PVC. Yes, it is the same likelihood. Against DeepProbLog 2.1 with its exact engine, probabilities and gradients agree to within 2e-16 for sums of two to four digits. On DeepProbLog's single-digit addition task (configuration of its released example, same initial weights and data order) the two reach 0.9738 ± 0.0035 and 0.9758 ± 0.0027 test digit accuracy, in 681 s and 160 s on one CPU thread (Tables 25–26). With the two encodings we tried, DeepProbLog needs minutes to compile four digits; for five, the faster one exceeded a 64 GB memory limit and the slower was not run. Our bags have ten; our layer evaluates the same likelihood in polynomial time on a GPU. That computation is not new either: Section 2 now cites De Smet & Zuidberg Dos Martires (NeurIPS 2024), whom we had missed.
For binary labels our likelihood is the count likelihood of CountLoss and LLP-PVC (Eq. 11); the full LLP-PVC method needs class histograms, which a digit sum does not provide. On the binary tasks we did not run their full pipelines; Section 3.4 limits the claim to the likelihood components.
W1. Limited novelty. We agree that neither the likelihood nor its computation is new, and Section 2 now says so. The main contribution is empirical: with the instance model and the data protocol held fixed, the paper indicates when the exact likelihood improves instance recovery and when it does not. On digit sums the mean gap to Gaussian-AMLE grows with the number of digit values (none for K ≤ 3; 0.853 vs 0.680 at K=10, confirmed with 20 seeds) and is largest for small bags (Section 4.4). FS-Conv has no advantage for large bags with few training bags, on binary Criteo bags, or under the dependence below.
W3. Factorization untested; no advantage on the real task.
Dependence. New stress test (Appendix C.4): a synthetic count task in which a shared bag-level random effect of scale τ is added to every instance logit; 10 seeds.
A larger τ also adds label noise that the features cannot explain (last row; majority-class rate 0.60, 0.60, 0.59, 0.57). Relative to that ceiling, Mean-MSE loses 1–4 points and the two likelihoods 3–7 for τ > 0: the mean of the total is a sum of marginal means under any dependence, while both likelihoods assume the factorized variance. The result does not favor our method; the abstract and Section 5 now say that the exact likelihood is not robust to this misspecification. The coverage of the model's 90% intervals for held-out totals falls from 0.93 to 0.31, which flags a miscalibrated aggregate model without instance labels but is not a test for dependence.
Real-world tasks. We added two (Section 4.5). On review star totals (K=5) FS-Conv is ahead of Gaussian-AMLE by 1.4, 1.2 and 0.6 points of rating accuracy at n=4, 8 and 16 (all five seeds, paired t p < 0.003), with indistinguishable accuracy at n=32 and 64. On PanNuke nuclei counts (K=16) it lowers the image-total MAE from 3.68 to 3.09 (all five seeds). On Criteo (binary labels, bags of 128) FS-Conv is slightly below proportion matching (ROC-AUC 0.749 vs 0.752), and we claim no advantage there.