acceptodds
Under review as a conference paper at ICLR 2027

Replanning Precision for Pareto Identification with Accumulated Evidence

Abstract

How should existing measurements guide the next experiment in Pareto set identification? We distinguish the remaining cost of completing a sufficient certificate from the information needed to exclude incorrect answers. A three-candidate construction reverses their preferred history allocation: concentration completes a rectangular certificate, while balanced history requires less additional Gaussian information cost. We explain this reversal through convexity and symmetry: averaging histories over an information-preserving symmetry cannot increase residual information cost, whereas alternative certificate routes can favor concentration. For fixed routes, we establish complementary bottlenecks and a suffix identity under which frozen progress scores tie across required actions. We implement certificate and information replanning with primal–dual allocation bounds and an anytime-valid Gaussian stopping rule. Across 1,068 synthetic trajectories, certificate replanning has lower observed mean cost on two targeted historical suites; an information planner's initial gain disappears on fresh seeds, and native PSIPS uses fewer capped measurements on cold-start suites. The results separate history value, certificate geometry, and sequential execution as distinct design problems.

Primary area:
optimization
Keywords:
Pareto set identification, multi-objective optimization, sequential experimental design, adaptive sampling, sample complexity

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_6GmcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper considers Pareto Set Identification (PSI), the task of identifying the exact Pareto set with a pre-specified confidence probability using as few measurements as possible. The paper studies the setting where some measurements of the candidates already exist, allocated unevenly across candidates, and asks how to choose the remaining measurements.

Given this setting, the paper proposes two ways to estimate how many more samples are needed before the procedure can stop. (1) The first counts samples until the confidence intervals of candidates stop overlapping compared with other candidate. To rule out a candidate, it is enough that one other candidate is confidently better on every objective. (2) The second counts samples until the data makes every wrong answer unlikely enough. It finds the wrong Pareto set that is hardest to tell apart from the current estimate and asks how many more samples would push it below a confidence threshold.

The paper claims that although both perspectives lead to the same high-probability identification guarantee, they can value the same historical dataset in opposite ways. To understand this difference, the authors analyze the mathematical properties of the two criteria and construct an example where they favor opposite ways of splitting the past samples across candidates. They then develop two adaptive sampling strategies, each planning toward one criterion, and compare them against existing PSI methods on the number of additional samples needed. The strategy based on the first criterion reduces additional sampling cost when historical measurements are concentrated on one candidate, but this advantage disappears when history is spread evenly. The strategy based on the second criterion shows no consistent improvement over existing methods.

Critical strengths

I think the problem setting is somewhat novel, and the authors make a nice observation about the asymmetric value of previously collected data.

Critical weaknesses

Readability This paper looks too much like an AI gen paper, even without attempt of careful revising to make it readable for human, and is extremely difficult to follow. As a concrete example:

  • "uneven historical measurements" is unclear what it indicates
  • "Exclusion needs one verified dominator; acceptance must rule out every possible dominator": It is understandable only when dominator means, a solution dominate the point that is going tobe excluded, that said, everything is stated without clear explaination at all.
  • Planning the cheapest remaining route is therefore natural: It is unclear what you mean by cheapest

These examples are just from the first couple of paras, and it is all of such hard to understand phrases, making the current draft clearly not acceesible for people, even though the AI usage statement has made stated the fact. Given the paper has not used 9 pages, it is hard to believe any effort has been made to increase readability of the paper. This is the paper I found most hard to read among the papers I review. As its current form, it is hard for me to believe that this paper be considered as a contribution to human knowledge other than generative AI pipeline's artifact.

Practical Impact. Another key limitation I found is that it seems quite intuitive that the first perspective on cost is easier to approach than the second (which aims to rule out all incorrect answers and may therefore be less straightforward). To this end, the empirical results are not particularly surprising. The main insight is essentially a theoretical indication, together with empirical validation, that historical data do have an impact on subsequent sampling strategies. However, to the best of my understanding, this does not provide much new insight into either the PSI problem itself or future algorithm design for PSI. Happy to reconsider this evaluation if the author provide further rebuttal on this aspect.

Questions for authors
  • Could the author comment on why there are such scenarios that you have some history measurements and you wish to continuoue sample ? Such a scenario is slightly less obvious to me.
  • Following up the 2nd weakness, what contribution the author believe this paper can contribute to the PSI community?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 6Gmc

Your point: the writing is difficult to read and appears unrevised. I take responsibility for the paper's content. I rewrote the opening and defined the terms before the formulas. Before: “uneven historical measurements.” After: “Earlier measurements then remain available, often in different numbers for different candidates.” Where: Introduction and Table 1.

Why resume with history? A pilot can exhaust its budget before identifying the Pareto set. Additional funding permits further measurements under the same conditions. History consists of observations already purchased; additional cost counts new measurements. Where: Introduction and Section 2.

What does the PSI community learn? Historical measurements affect both unfinished comparisons and the stopping threshold. With a fixed instance and sampler, we prove that an ellipsoid threshold can create growing additional cost from an easily excluded candidate's history. Predetermined historical scales remove this growth. Matched experiments then test the allocation effect separately. Where: Sections 5.4, 4.4; Appendix AA.

The added selector chooses between certificate and calibrated boundary targets after checking both against the same stopping rule. Its fresh-seed mean improvement over boundary planning is 0.83%, concentrated in part of the tested suite. Where: Sections 4.6, 5.8.

Recalibration removes an earlier certificate gain, and the original information gain does not replicate. The paper provides a calibration mechanism, an implementable target-selection rule, and experiments showing where each helps.

Reviewer_C8wrOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper studies how to choose additional experiments when some measurements have already been collected, with the goal of identifying the Pareto set in a noisy multi objective problem.

Critical strengths

Planning the cheapest remaining route might be useful in some real applications to reduce the computational cost.

Critical weaknesses

I’m really having difficulty understanding several of the statements, as many of them seem confusing to me. For example, the statement, “History placement matters because Pareto decisions have different logical requirements. Exclusion needs one verified dominator; acceptance must rule out every possible dominator,” is confusing to me. I believe the fundamental logic is the same in both cases, as both rely on comparing objective values to determine Pareto dominance. The difference lies in the difficulty of identifying Pareto optimal and non-optimal solutions, rather than in the underlying logic itself.

Additionally, several mathematical notations throughout the paper are ambiguous and lack formal definitions, making the manuscript difficult to follow: n0n^0: This notation is confusing. While 00 appears to be a specific symbol/notation in your context, writing it as n0n^0 conventionally denotes a 0th0^{\text{th}} order or exponentiation (which equals 11). ll, ilil, and jljl: It is unclear whether ilil and jljl represent a product of variables or multi index notation (e.g., index i,li, l). Furthermore, while ll likely represents the objective index, it is never explicitly defined. Relying on readers to infer symbol definitions makes following the mathematical framework unnecessarily challenging.

Questions for authors

Did the author check the notations definition before submission?

2 comments
AuthorsOpenReview ↗

Response to Reviewer C8wr

Your point: acceptance and exclusion use the same dominance relation. Yes. The revised example states the comparisons each decision requires. If either A or B is better than C on every objective, either can establish C's exclusion. To certify A as nondominated, every competitor must be ruled out as its dominator. These are different quantifiers over the same relation. Where: Introduction; Section 2, “Sufficient certificates.”

Your point: symbols lack clear definitions. The submitted notation needed correction. We now separate historical counts, current counts, planned targets, and actual additional observations. Candidate indices are i and j; the objective index is ell. Commas separate candidate and objective indices. The superscript “init” denotes history; parenthesized superscripts label alternative history vectors. Where: Section 2 and Table 1.

Your point: readers must infer technical words. A certificate is a set of verified comparisons identifying the Pareto set. A route selects those comparisons. A witness is a dominating candidate used for an exclusion. Measurement cost counts new observations at their stated prices; decision computation is reported separately. Where: Introduction; Sections 2, 5.8.

The added selector uses the same notation throughout: it constructs target counts, checks their feasibility, and samples toward the selected target. Updated observations determine actual stopping. Where: Section 4.6.

Thank you for the response and the updated version. The notation has been corrected, making the paper more readable. Accordingly, I have increased my score from 1 to 2.

Reviewer_Y1NgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies Pareto set identification with accumulated measurements. It distinguishes the cost of completing sufficient certificates from information needed to exclude alternative Pareto sets, proves that these objectives can prefer different history allocations, and develops certificate/information replanning with valid sequential stopping. Synthetic Gaussian experiments demonstrate localized certificate-planning gains but no reproducible advantage from information planning.

Critical strengths

None. The potential benefits of the proposed framework are not sufficiently substantiated by the current experimental evidence.

Critical weaknesses
  1. Theory–execution gap. Theorem 3 uses a stylized radius and an unmatched information threshold (lines 151–154), so it does not establish a same-confidence reversal in sequential cost. No finite-sample or asymptotic cost guarantee connects the online planners to the historical lower bound.

  2. Unreliable information-planning gains. On fresh seeds, historical cost is 43.83 versus 43.61 for APE-common; competing plans cost 11.47% more than certificate-only planning (Table 2). Thus, the information formulation and hybrid do not demonstrate consistent improvements in sequential efficiency.

  3. Incomplete comparisons in the main setting. Native PSIPS is tested only on cold starts, with different stopping calibration; Pareto Track-and-Stop is absent (Tables 1–2). Comparisons under matched confidence requirements and accumulated history are needed to establish competitiveness against these relevant methods.

  4. Narrow empirical support. Experiments use small synthetic, two-objective Gaussian instances, unit costs, and twelve seed blocks per stage. Certificate gains concentrate in a post-hoc concentrated-history stratum; balanced-history mean costs slightly worsen (Figure 2). Generalization across geometries, larger problems, and realistic measurement conditions remains unestablished.

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer Y1Ng

Your point: planning reversal does not establish sequential efficiency. The revision adds execution bounds and calibrated target selection. The reversal remains a result about planning costs. The finite expected-cost and baseline-relative bounds do not match the historical information lower bound. Where: Section 4.3; Appendices L, AD.

Your point: information and hybrid gains do not replicate. We retain both negative results. Historical-scale correction also removes the certificate advantage in the targeted suite. The new selector checks certificate and boundary targets against one calibrated stopping rule, then chooses the cheaper feasible target. A fresh-seed comparison finds a 0.83% mean reduction over boundary planning alone. The full results retain configurations with no improvement and cases worse than APE. Where: Sections 5.6, 5.4, 4.6, 5.8; Appendix AI.

Your point: historical baselines and matched comparisons are missing. We added history-initialized PTS and independently implemented PSIPS-D. We separate common-stopping allocation comparisons from complete procedures. Their empirical stopping tests accept almost all tested histories immediately; certificate planning does not beat those procedures end to end here. Where: Section 5.5; Appendix K.

Your point: empirical support is narrow. We added larger candidate sets, more objectives, matched geometries, fresh-seed confirmation, and bounded historical bias. The new selector experiment remains a two-objective Gaussian study. Where: Sections 5.2, 5.3; Appendices Z, AI.

The contribution is a measured calibration mechanism, executable calibrated targets, and controlled allocation comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.