- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper considers Pareto Set Identification (PSI), the task of identifying the exact Pareto set with a pre-specified confidence probability using as few measurements as possible. The paper studies the setting where some measurements of the candidates already exist, allocated unevenly across candidates, and asks how to choose the remaining measurements.
Given this setting, the paper proposes two ways to estimate how many more samples are needed before the procedure can stop. (1) The first counts samples until the confidence intervals of candidates stop overlapping compared with other candidate. To rule out a candidate, it is enough that one other candidate is confidently better on every objective. (2) The second counts samples until the data makes every wrong answer unlikely enough. It finds the wrong Pareto set that is hardest to tell apart from the current estimate and asks how many more samples would push it below a confidence threshold.
The paper claims that although both perspectives lead to the same high-probability identification guarantee, they can value the same historical dataset in opposite ways. To understand this difference, the authors analyze the mathematical properties of the two criteria and construct an example where they favor opposite ways of splitting the past samples across candidates. They then develop two adaptive sampling strategies, each planning toward one criterion, and compare them against existing PSI methods on the number of additional samples needed. The strategy based on the first criterion reduces additional sampling cost when historical measurements are concentrated on one candidate, but this advantage disappears when history is spread evenly. The strategy based on the second criterion shows no consistent improvement over existing methods.
Critical strengths
I think the problem setting is somewhat novel, and the authors make a nice observation about the asymmetric value of previously collected data.
Critical weaknesses
Readability This paper looks too much like an AI gen paper, even without attempt of careful revising to make it readable for human, and is extremely difficult to follow. As a concrete example:
- "uneven historical measurements" is unclear what it indicates
- "Exclusion needs one verified dominator; acceptance must rule out every possible dominator": It is understandable only when dominator means, a solution dominate the point that is going tobe excluded, that said, everything is stated without clear explaination at all.
- Planning the cheapest remaining route is therefore natural: It is unclear what you mean by cheapest
These examples are just from the first couple of paras, and it is all of such hard to understand phrases, making the current draft clearly not acceesible for people, even though the AI usage statement has made stated the fact. Given the paper has not used 9 pages, it is hard to believe any effort has been made to increase readability of the paper. This is the paper I found most hard to read among the papers I review. As its current form, it is hard for me to believe that this paper be considered as a contribution to human knowledge other than generative AI pipeline's artifact.
Practical Impact. Another key limitation I found is that it seems quite intuitive that the first perspective on cost is easier to approach than the second (which aims to rule out all incorrect answers and may therefore be less straightforward). To this end, the empirical results are not particularly surprising. The main insight is essentially a theoretical indication, together with empirical validation, that historical data do have an impact on subsequent sampling strategies. However, to the best of my understanding, this does not provide much new insight into either the PSI problem itself or future algorithm design for PSI. Happy to reconsider this evaluation if the author provide further rebuttal on this aspect.
Questions for authors
- Could the author comment on why there are such scenarios that you have some history measurements and you wish to continuoue sample ? Such a scenario is slightly less obvious to me.
- Following up the 2nd weakness, what contribution the author believe this paper can contribute to the PSI community?
Response to Reviewer 6Gmc
Your point: the writing is difficult to read and appears unrevised. I take responsibility for the paper's content. I rewrote the opening and defined the terms before the formulas. Before: “uneven historical measurements.” After: “Earlier measurements then remain available, often in different numbers for different candidates.” Where: Introduction and Table 1.
Why resume with history? A pilot can exhaust its budget before identifying the Pareto set. Additional funding permits further measurements under the same conditions. History consists of observations already purchased; additional cost counts new measurements. Where: Introduction and Section 2.
What does the PSI community learn? Historical measurements affect both unfinished comparisons and the stopping threshold. With a fixed instance and sampler, we prove that an ellipsoid threshold can create growing additional cost from an easily excluded candidate's history. Predetermined historical scales remove this growth. Matched experiments then test the allocation effect separately. Where: Sections 5.4, 4.4; Appendix AA.
The added selector chooses between certificate and calibrated boundary targets after checking both against the same stopping rule. Its fresh-seed mean improvement over boundary planning is 0.83%, concentrated in part of the tested suite. Where: Sections 4.6, 5.8.
Recalibration removes an earlier certificate gain, and the original information gain does not replicate. The paper provides a calibration mechanism, an implementable target-selection rule, and experiments showing where each helps.