- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes PTSR, a method based on paired tripled differences that recovers the conditional law of a response variable in an aggregate learning setting where each group shares an unknown offset. The poor writing and description of the setting and the missing motivation of the offset itself and thus of the main research problem, leave it open whether this work actually addresses an impactful research question. Together with the almost non-existent proper baselines and the evaluation focusing only on two seemingly cherry-picked synthetic data tasks selected by the authors, it is hard to judge whether the paper constitutes a contribution at all. Neither the writing nor the actual methodology and experimental setup of the paper hold up to the standards of this conference, which is why I recommend a clear rejection.
Critical strengths
The paper proposes to learn shape and orientation of a response distribution while eliminating shared offsets. The method itself is justified and leads to an estimation of the conditional distribution while simultaneously removing the offset.
Critical weaknesses
Baselines and experimental setup: The paper lacks a clear and thorough experimental evaluation, especially with respect to baselines and benchmarks. The authors claim "two heavy-tailed benchmarks" in the abstract. However, these are synthetic experiments, constructed by the authors with no justification why they should be a fair benchmark. In fact, the scenarios consider a specific heavy-tailed (-stable) distribution which seems to be chosen specifically to accommodate the paper setting without addressing or discussing its suitability in practical applications. There are no actual benchmarks or real-world datasets used, not even from the referenced related work.
Furthermore, the paper introduces several baselines (mostly without any reference or justification) of which three are "privileged" baselines that use some notion of ground truth information and another two are trained on a reduced pair pool. While some of those are based on related work (the others are not referenced at all), they are implicitly flagged as "not comparable" and not adequately compared. The only remaining actual comparable baseline is then the Winsorized GMM for which neither an adequate justification nor any reference is given. Possible baselines to add (beyond properly evaluating the cited ones [1,2,3]) would include for example:
- Delaigle et al. [7], who provide a density estimator based on pairwise differences (not triplets), which is also considered in this paper. They also already provide the estimator based on cosine/Fourier-features
- Li et al. [8], who provide a nonparametric estimator for two or more available measurements also using characteristic functions
Overall, the experimental section seems constructed and cherry-picked specifically to highlight the performance of the proposed PTSR method. While Figure 4 shows a qualitative evaluation of the method regarding calibration, the quantitative comparison in the paper, especially with respect to the dataset and baseline selection, is not suitable for a venue like this. This would either require a detailed setup against benchmarks and baselines or alternatively good justification why this does not apply together with more thorough qualitative evaluation of the method and a placement within the broader literature of learning from aggregates.
Writing and style of the paper The paper is very poorly written and thus extremely difficult to follow and understand. Neither the motivation nor the research question are adequately introduced or explained. The paper uses seemingly complex scientific words and definitions without actually explaining the context or providing the thorough definition. While most of the definitions come from signal processing and might be known in that community they are neither cited nor sufficiently explained here for the reader coming more from the probabilistic methods community (which this paper is classified under). Also nothing of the definitions and the naming is similar to the cited related work [1,2,3]. The following constitutes a non-exhaustive list of poor wording and styling:
- The abstract mentions a conditional law, but so far no covariates are introduced
- Many words are thrown in the abstract and introduction that are not properly defined (or maybe defined later) and not really clear from the context, e.g., "triple contrasts", "remaining orientation" (orientation of what), "totals" (not defined in introduction and not a consistent wording across literature), "identified up to reflection by pair spectra" (before defining pair spectra), "orbit" (used in contributions but never explained until later), "spatial coverage" (how is the scenario now spatial),...
- "Baseline", "offset", and "location" are three words used for the same thing in the introduction
- Section 4.2: Several metrics are introduced but only one is actually defined in the main paper
- 2.1 uses the notation , while Proposition 1 suddenly changes it. Also the proof of Prop. 1 is very informal.
- l. 162 "[...] share an anchor context , as in the recovery benchmark". Since the recovery benchmark is not defined (and also is not actually a benchmark), it is not clear what an anchor context is
- Algorithm 1: The input requires fixed frequencies and candidate bandwidths, however these are only defined in the appendix (without an actual selection heuristic) and should be in the main part
- Algorithm 1: three out of five equations are actually in the appendix. Since the algorithms constitutes the contribution the equations should be in the main text. Also the algorithm is written very informal
- l. 267 "Covariates are scrambled Sobol points": Why choose a quasi-random Sobol sequence, as opposed to random sampling. The choice should be justified and the Sobol sequence equipped with a reference.
- Figure 2: "Coordinates [...] and ticks are shared with labelled only in the Truth column". This is obvious and does not need to go in the Figure description as it serves now purpose. At the same time raises the question why is not also shared, since both panels have the same -axis
- The contributions section names sections 3,5,6, thus in the wrong order
- l. 169 The space of is undefined
- Wording errors such as "rspctively" in l. 148
- ...
While individual mistakes are not a problem, the writing of the paper overall does not hold up to scientific standards and makes it very hard for any reader to understand what the contribution and proposed method actually is.
Missing references and literature The references are very sparse and there are many missing references, as well as important related work that is not cited. Examples of missing references:
- l. 272 Winsorized GMM not referenced (or adequately explained). Same for the other benchmarks, i.e., Nuisance-profiled MLE, Shift-known MLE, Raw ECF, ...
- l. 29 "Recovering [...] is an inverse problem" not referenced of justified
- l. 267 Sobol points (sequence) not explained or referenced
- l. 40 "For a family identified up to reflection by pair spectra, this leaves two candidate orientations." Claim not supported by a reference
- While the main contribution is concerned with offsets across replicate groups, no reference is given to support the relevance of the setting in practice
Exemplary additional relevant related work:
- Tanaka et al. [4], while focusing on the spatial setting, propose a probabilistic model to model different levels of granularity (aggregates) based on Gaussian processes
- Javanmard et al. [5] consider an aggregate learning setting and study the impact of designing a loss function either on the instance-level or the bagged level
- Yousefi et al. [6] consider an aggregated learning setting and propose to a linear combination of latent Gaussian processes as a model.
- ...
References
[1] Zhang, Y. et al. Learning from Aggregate Observations, NeurIPS 2020.
[2] Bhowmik, A. et al. Frequency domain predictive modelling with aggregated data. UAI 2017.
[3] Law, H. et al. Variational Learning on aggregate outputs with gaussian processes. NeurIPS 2018.
[4] Tanaka, Y. et al. Refining Coarse-Grained Spatial Data Using Auxiliary Spatial Data Sets with Various Granularities. AAAI 2019.
[5] Javanmard, A. et al. Learning from aggregate responses: Instance level versus bag level loss functions. ICLR 2024.
[6] Yousefi, F. et al. Multi-task learning for aggregated data using gaussian processes, NeurIPS 2019.
[7] Delaigle A. et al. On deconvolution with repeated measurements. An. Statist. 36(2).
[8] Li, T. et al. Nonparametric estimation of the measurement error model using multiple indicators. Journal of Multivariate Analysis 1998.
Questions for authors
-
While the examples in l.25-l.30 are examples of aggregates, it is not clear at all why a shared offset makes sense here. It would be good to further motivate this to highlight the relevance of the actual research problem.
-
You claim that estimating the response is difficult under heavy tails (l.13) and that "in general the mean [...] may not exist" (l.79). Since estimation under heavy tails is a proposed advantage of PTSR: Why does the heavy-tailed assumption matter in your specific setting beyond the constructed synthetic dataset. Are there any examples of real-world data from aggregates where heavy-tailed data are present or the data has undefined mean (i.e., Cauchy data)?
-
l. 40 "For a family identified up to reflection by pair spectra". Since this seems to be the main assumption on the distributional family, how restrictive is this assumption and what families are included here beyond the -stable family in the experiments. Do any other works consider similar restrictions or similar distributional families?
36 comments
I acknowledge the author's response and raised concerns and provide answers to several points below:
- You are correct that Bhowmik et al. [2] was published in AISTATS 2017 not in UAI 2017 as I have mistakenly stated and it is absolutely fair to point that out. However, trying to discredit the whole review on this basis seems to push it quite far, although the motivation behind discrediting seems quite clear given the ratings.
- I did not (and did not mean to) question the authors' scientific integrity and standards in my comment. I state that the paper does not hold up to the (scientific) standards, explicitly mentioning that this goes beyond small mistakes such as a false attribution of a reference. In my opinion, the overall standard of ICLR is not met, neither in writing nor in the baselines and experiments, as also pointed out by reviewer u6v4. This does not imply that the authors purposefully committed misconduct or violated scientific integrity; it just means that the paper does not meet the standards.
- While the review could be seen as detailed, it mentions overall only three main weaknesses. The "[...] sheer volume of criticisms" you mention seems overstated, especially since W1 aligns with reviewer u6v4, which you already seemed to have answered.
Of course, there is no timeline for when to reply, but I am happy to participate in additional rebuttal and am looking forward to further constructive comments when the authors "once again reconvene".
Hi, one of the authors here. Thanks for writing back :)
We appreciate the message and wanted to briefly clarify your three points to avoid any further misunderstandings before we post our consolidated rebuttal:
Regarding (1): To be clear, we did not suggest that this single error discredits your entire review, nor were we attempting to dismiss your remaining concerns. On the contrary, we had already responded to the other reviewers and did not want to leave your review unanswered in the meantime. This is precisely why we explained at the end of our preliminary comment that we had already discussed and consolidated our responses, but needed additional time for the coauthors to meet again and organise a clear rebuttal. Our intention is to give your review the same careful attention as the others, not to dismiss or discredit it.
Regarding your remark that "the motivation behind discrediting seems quite clear given the ratings," , it's best to and let's try to avoid such assumptions about each other's motivations where possible and keep the discussion focused on the scientific arguments and evidence.
Regarding (2): This was clear and not misunderstood on our end, we do understand that in your assessment, you feel our paper falls short in your view on what ICLR's (scientific) writing/exposition standards require. Either way, this is why we hope our consolidated rebuttal will give us the space to explain our reasoning and evidence more clearly, discuss the specific points of disagreement, find common ground, and hopefully address your concerns through constructive discussion.
Regarding (3): Yes, we recognise that you have organised your criticisms into three main categories, which is why we referred to the "scientific, methodological, and presentation related criticisms". To clarify "sheer volume of criticisms," we meant the numerous individual points raised within some categories, which take time to address carefully. We want to give your review the proper attention and address your comments thoughtfully, rather than post a rushed or fragmented reply.
Finally, we want to reiterate that we genuinely do appreciate the time you have taken to read and review our paper, as well as your willingness to continue the discussion. We want to make it clear our intention is to engage with all your concerns seriously and to explain the reasoning and choices underlying the work.
With the consolidated rebuttal, we hope our subsequent discussions will help clarify the precise points of disagreement and establish common ground where possible. We believe constructive scientific discussion is valuable even when assessments and views differ. So we look forward to engaging with your feedback in detail when we post our full response.
Thanks again for your time, and wishing you a good weekend!
Concern Regarding Review Accuracy and Integrity
We wish to flag a specific factual error that raises concerns about the accuracy and reliability of this review to the ACs, SACs and anyone else reading this:
The reviewer cites [2] (Bhowmik et al., Frequency Domain Predictive Modelling with Aggregated Data), as appearing at UAI 2017. This is incorrect. The paper was published at AISTATS 2017 (or PMLR, Vol. 54, pp. 971–980 as cited in our submission). This is directly verifiable from the official proceedings (https://proceedings.mlr.press/v54/bhowmik17a). And can be double checked against UAI's accepted papers (https://www.auai.org/uai2017/accepted.php).
This is an objectively false bibliographic attribution and it is particularly concerning that a reviewer who explicitly questions our referencing and scientific standards presents factually false publication information in their own assessment. We regard this as a potential review-integrity concern and request that the ACs and SACs examine whether the review meets ICLR's standards of responsible reviewing. Factually incorrect citations are a serious academic integrity violation, and we believe the same standards expected from submissions should apply equally to reviews.
We have already discussed and consolidated our responses to most of the reviewer's scientific, methodological, and presentation related criticisms. However, we have not had sufficient time yet to organize these into a clear and easy to read coherent point-by-point rebuttal due to the sheer volume of criticisms. We will post our response by the coming weekend, when all authors can once again reconvene to properly finalize it.
(after further discussion with the reviewer, we would like to retract parts of this statement such as the reliability concerns because we believe the reviewer was reviewing in good faith and are very grateful for the reviewer for providing us detailed comments that is helpful to understand their perspective)