acceptodds
Under review as a conference paper at ICLR 2027

From Pareto Fronts to Statistical Evidence: Selective Inference for Models and Policies

Abstract

Multi-objective evaluation often uses the same noisy data both to select a Pareto shortlist and to support claims about the selected models or policies. This creates a post-selection problem: uncertainty statements that ignore how the shortlist was chosen need not provide the advertised reliability for the report that is actually published. We develop report-conditional inference for complete Pareto reports around a structural observation: the reported candidates themselves provide the dominance witnesses needed to reconstruct the entire selection event. Along an inferential line, each required dominance relation occupies an interval, so a report of size among candidates can be reconstructed from only directed relations. This yields support construction in with near-optimal dependence on for fixed and sparse reports. Under joint Gaussianity, the reconstructed support gives exact conditional inference and enables scalar carving, which reuses selection data while preserving the same reported labels. Across Gaussian designs, model libraries, and policy evaluations, the method maintains report-level calibration while recovering substantially more information than matched splitting. In the primary image setting, carving reduces mean paired interval width by 33.6%; in a 1,024-configuration Bank library, report-sensitive reconstruction yields a end-to-end median acceleration over all-pair compression.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
selective inference, Pareto fronts, uncertainty quantification, data carving, multi-objective evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AdJ1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the post-selection inference problem in multi-objective evaluation and presents an efficient approach to generate a Pareto report of size s among K candidates and a near-optimal algorithm to reconstruct the complete same-report selective support.

Critical strengths
  1. The paper addresses an important problem in model evaluation and selection.
  2. The proposed approach can significantly reduce the cost of generating Pareto report and reconstruct supports. The method is designed elegantly with strong theoretical support.
  3. The experimental study shows that the approach achieves meaningful gain. The carving also reduces mean pair interval, making the estimation more precise than sample splitting.
Critical weaknesses
  1. The paper assumes X (includes the stacked scores S and performance estimates T) are jointly Gaussian, with covariance known, for support reconstruction. This is a quite strong assumption. iThe paper should clarify the assumption and discuss how practical the assumption is in real world applications. Failing to do so will affect the applicability of the work.

  2. The paper should further discuss the related work in multi-objective evaluation and clarify its novelty/advantages compared to those approaches.

Questions for authors

Please see the listed weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer AdJ1 #1

CW1. Role and Practical Scope of Joint Gaussianity and Known Covariance

We appreciate the reviewer's question regarding the assumptions and their practical applicability. The structural reconstruction is distribution-independent; joint Gaussianity and known covariance are required for exact conditional inference.

Role of the assumptions. Given an affine score line, the Pareto dominance structure alone establishes the complete-report event using s(K−1)s(K-1) directed relations. Joint Gaussianity ensures independence between the target statistic and its covariance-projection residual, while known covariance determines the projection direction and variance. Together, they yield the exact conditional distribution over the reconstructed support. Scalar carving further uses the nested-sample covariance model to reuse selection data while preserving conditional validity (Sections 2–4.1; Appendices A.2 and C).

Practical applicability and calibration. For real applications, Gaussian approximation is applied to averages of per-observation scores, with jointly estimated covariance capturing dependence across models and objectives. Independent confirmation on real model libraries achieves 96.85% conditional simultaneous coverage on Image and 97.00% on Bank, with multiplicity-adjusted error upper bounds below 5% in both cases. The primary image experiment also demonstrates a 33.6% reduction in mean paired marginal width relative to splitting.

Table A1. Conditional coverage and interval width with estimated covariance.

MethodImage coverageImage width / splittingBank coverageBank width / splitting
Carving96.85%0.6823397.00%0.44152
Matched splitting94.85%1.0000094.50%1.00000
Global max-T92.70%0.3824094.05%0.28670
LSI89.35%0.3627494.50%0.29064

Both tasks use n=5000n=5000, f=0.9f=0.9, and a 5% coordinate simultaneous-inference budget. Coverage is evaluated over 2,000 conditional repetitions per frozen report; widths are mean paired ratios from 100 prespecified records. The Bank control does not establish multiplicity-adjusted global/LSI conditional undercoverage.

Approximation limits and finite-sample alternative. The revised Sections 4.2 and 7 explicitly discuss small-sample undercoverage and the conditions required for Gaussian approximation. For bounded i.i.d. scores with known ranges, Appendix N.2 provides an independent-holdout procedure with finite-sample protection, at a measured width cost of 1.644–2.959 times Gaussian splitting across the original six settings.

These results distinguish the distribution-free structural contribution, exact Gaussian inference, and empirically validated real-score applicability, while providing a finite-sample alternative when Gaussian approximation is insufficient.

See Sections 2–4.2 and 7, Tables 3–4, and Appendices A.2, C, N.2, Q.1 (Tables 49–50, pp. 59–61), and Q.5 for details.

AuthorsOpenReview ↗

Response to Reviewer AdJ1 #2

CW2. Novelty Relative to Multi-Objective Evaluation Methods

We appreciate the reviewer's suggestion to clarify the distinction from existing multi-objective evaluation methods. Our central contribution is efficient, selection-aware inference with conditional guarantees for the complete published Pareto report, addressing a different statistical target from existing approaches.

Distinction from prior work. Pareto Testing and reliability-graph methods validate configurations against risk constraints, while Pareto-set identification aims to recover the population-optimal set. Stability-based policy evaluation addresses data reuse through algorithmic stability. In contrast, our framework reconstructs the complete selection event of the observed Pareto shortlist, enabling inference on its population performance and prespecified comparisons conditional on the exact reported label set. Unlike these approaches, it does not require identifying the population Pareto front.

Technical novelty and theoretical guarantees. Our key insight is self-certification: every omitted candidate is dominated by at least one reported candidate. This reduces complete-event reconstruction to s(K−1)s(K-1) directed relations, achieving O(Ksd+Kslog⁡(Ks+1))O(Ksd+Ks\log(Ks+1)) complexity while preserving all disconnected support components and dominance-witness switches. An Ω(Ks)\Omega(Ks) explicit-output lower bound establishes near-optimal dependence for fixed d≥2d\geq2 and 2≤s≤K/22\leq s\leq K/2. The resulting exact support enables Gaussian conditional inference and scalar carving to reuse evaluation data without changing the published report.

Computational validation. The trained Bank library demonstrates the benefit of report-sensitive reconstruction while preserving complete inferential outputs.

Table A2. End-to-end computation on the trained Bank library.

Library size KKReport size ssReport-sensitive (s)All-pair (s)Median speedup
6436–408.448.511.01×
12855–6114.2714.571.01×
25690–10025.8727.651.07×
512127–16546.5958.911.26×
1,024180–23474.21145.081.95×

Raw-row-to-report timings begin with loaded, warmed models. Speedups are medians of paired ratios.

At K=1024K=1024, the framework achieves 1.95× end-to-end acceleration, with a separate 4.84× gain in support reconstruction, while retaining the full selection event and inferential outputs.

The contribution therefore extends beyond Pareto evaluation itself: it establishes a computationally tractable connection between complete Pareto selection, exact selective inference, and data-efficient statistical reporting.

Section 3 and Appendix A.3 establish the theoretical results. Expanded Section 6 distinguishes our statistical target from existing multi-objective methods and clarifies the connection to Le Duy and Takeuchi's parametric programming. Sections 5.1–5.3 provide computational and inferential evidence (Table 1, p. 6; Appendix P.3, pp. 57–58).

Code, experimental configurations, and numerical audits are available in the anonymous repository.

Thanks the authors for the rebuttal response. I will maintain my score.

Dear Reviewer AdJ1,

 

Thank you for your thoughtful review and for taking the time to follow up on our rebuttal. We appreciate your encouraging feedback and are glad that the new experiment addressed your concern.

 

Best regards,
The Authors

Reviewer_pafYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies inference after Pareto selection, where the same evaluation data are used to select a shortlist and to report the performance of its members, and targets report-conditional coverage, i.e., validity over repeated evaluations that yield the same complete label set. The authors observe that the reported labels themselves certify the domination of every omitted candidate, so the selection event can be reconstructed from s(K−1)s(K−1) directed relations and, along an inferential line, is a finite union of intervals. This gives an O(Ksd+Kslog(Ks+1))O(Ksd + Ks log(Ks+1)) construction together with an $Ω(Ks) $explicit-output lower bound. Under joint Gaussianity with known covariance, the support yields exact truncated-normal inference, which is combined with scalar carving to reuse the selection data. Experiments on Gaussian simulations, CIFAR-10 (21 RobustBench models), Bank Marketing (16 models and a 1,024-configuration library) and policy evaluation compare primarily against matched splitting.

Critical strengths

S1. A clean separation between report-conditional and aggregate protection. In Prop. 4 and Tables 3/33, global max-T and LSI attain conditional coverage of only .878/.889 on a two-label report that occurs with probability .164 (and as low as .467 at δ = 4), while their aggregate coverage stays at the nominal level. This shows convincingly that aggregate validity does not carry over to a specific published report, which is the premise of the paper.

S2. Correct structural results with a tight complexity analysis and careful auditing. Thm. 1, Prop. 1 and the construction in A.3 establish optimality up to a log factor, with disconnected supports, ties and singular covariance handled explicitly. The empirical work is unusually thorough: all cells are reported, Monte Carlo uncertainty is given throughout, and 318 supports are cross-checked against an independent backend.

Critical weaknesses

CW1. The report-conditional guarantee yields no observable benefit in any of the real settings studied, while it does cost precision.

Global max-T and LSI offer no report-conditional guarantee, so some loss of precision is a reasonable price for one. On real data, however, nothing is gained in return. In App. P.1, all four methods keep conditional risk below .05 in each of the six real report groups, and global/LSI fail in none of them. The costs, by contrast, are visible. In the Gaussian designs (Tables 8/9), carving's simultaneous bands are 20–32% wider than those of global max-T. On CIFAR, carving's width ratio (.666, Table 4A) is on par with global max-T (.663) and significantly larger than LSI (.649; App. J.3), and it gains only 0.207 true comparisons per report over global max-T (about 1.7%, Table 4B). On Bank, it yields fewer true comparisons than global/LSI in all three groups, e.g., 14.32 vs. 15.36 (Table 45).

The practical need for the report-conditional target therefore rests on the synthetic two-candidate design (Prop. 4, Tables 3/33). The abstract reports gains only against splitting, and Sec. 5.2 states that carving "exceeds splitting in true-comparison yield in each" group without mentioning that it trails the global methods on Bank. The computational speedup is also limited, at 1.01–1.07× for K ≤ 256 (Table 1B), whereas the paper's real pools contain 21 and 16 models. The paper should identify realistic settings in which the guarantee justifies its cost, for example by reporting global max-T/LSI alongside splitting in the abstract and main tables, and by exhibiting a real setting where their report-conditional coverage fails.

CW2. The evidence for the plug-in Gaussian approximation on real data does not support the calibration claim in the abstract.

On CIFAR with n = 500–1000, carving's FCR is .067–.087, and the FCR intervals of all six small-sample cells lie above .05 (App. J.5). On real Bank records, the miss rate reaches 7.2% even with the oracle covariance, and 10.8% at n = 1000, f = .8 (Table 41, App. O.6), so the error stems from the score distribution rather than from covariance estimation. The bound in Thm. 2 is trivial (equal to 1) at these sample sizes (p. 53). Prop. 2 further requires non-rare reports; if the realized report probabilities are very small, as K = 1024 with s ≈ 200 suggests, the intervals in the scaling study lack validity support (see Q1). The abstract's claim that, "across Gaussian designs, model libraries, and policy evaluations, the method maintains report-level calibration" should be restricted to the large-sample regime (n ≥ 2500), and the finite-sample alternative in App. N.2, together with its cost, should be discussed in the main text.

Questions for authors

Q1. What are the estimated probabilities of the realized complete reports in the 1,024-configuration library, and are the intervals calibrated there? If not, what does this experiment establish beyond computational speed?

Q2. Among the 14 CIFAR label groups in Table 18 and the Bank groups, is the report-conditional coverage of global max-T or LSI significantly below .95 in any of them? Such a case would directly address CW1.

Q3. Did you consider conditioning on a coarser event, e.g., only on whether the target label is selected? Such an event has higher probability and could yield shorter intervals and better asymptotic behavior. What makes conditioning on the complete label set necessary?

6 comments
AuthorsOpenReview ↗

Response to Reviewer pafY #1

CW1. Observable Benefit, Precision, and Scale

We appreciate the reviewer's focus on practical benefit and computational efficiency. Our results demonstrate the value of complete-report protection on real predictions, alongside measurable information-recovery and scalability gains.

Direct evidence of conditional protection. We retain all 21 image models and select on clean accuracy and negative Brier score at n=5000n=5000, f=0.9f=0.9. A selection-only rule based on report frequencies and population dominance freezes the complete report before independent confirmation, without inspecting coverage outcomes. Across unconditional draws, global max-T and LSI achieve 99.50% and 99.05% simultaneous coverage, respectively. Conditional on the frozen report, their coverage drops to 92.70% and 89.35%, while carving achieves 96.85%.

Table P1. Conditional simultaneous coverage and interval width on the frozen image report.

MethodConditional coverageAdjusted error intervalWidth / splitting
Carving96.85%[2.115%, 4.481%]0.68233
Matched splitting94.85%[3.804%, 6.778%]1.00000
Global max-T92.70%[5.689%, 9.178%]0.38240
LSI89.35%[8.710%, 12.836%]0.36274

All methods use the same frozen report and 5% coordinate simultaneous-inference budget. Coverage uses 2,000 accepted conditional repetitions; widths use 100 prespecified paired records. Two-sided MC intervals are simultaneously adjusted across 16 risks, separately from the inference budget. The report occurs in 13/2,000 independent draws (0.65%).

Carving's adjusted conditional-error upper bound is 4.481%, below the 5% target, while global max-T and LSI have lower bounds of 5.689% and 8.710%. This provides multiplicity-adjusted evidence of conditional undercoverage for both simultaneous baselines. Carving improves conditional coverage by 4.15 and 7.50 percentage points, respectively, while remaining 31.8% narrower than matched splitting. The additional width relative to global/LSI therefore provides verified protection for the specific published report (Section 5.2, Table 3, p. 7; Appendix Q.1, Table 49, p. 60).

Information recovery and strong-baseline comparisons. Under the stated nested Gaussian model, carving intervals are no wider than matched splitting for the same report, target, error budget, and common known covariance. Empirically, carving reduces primary-image mean paired marginal width by 33.6% and supports 0.684 additional true comparisons per report (paired 95% MC interval [0.633, 0.735]). It improves comparison yield over splitting in all six original frozen-report groups. Primary-image simultaneous width ratios are 0.666/0.663/0.649 for carving/global max-T/LSI; carving supports 0.207 more true comparisons than global max-T ([0.174, 0.240]). On Bank, global/LSI achieve higher comparison yields. These results distinguish the information-recovery gain relative to splitting from the precision cost of stronger conditional protection (Sections 4.1 and 5.2–5.3; Table 4, p. 8; Table 46, p. 56).

Scalability. Computational gains increase with library size as report-sensitive reconstruction replaces all-pair comparisons. For K≤256K\leq256, shared inference costs limit end-to-end acceleration. At K=1024K=1024, the trained Bank library achieves 4.84× faster support reconstruction and 1.95× median end-to-end acceleration, while preserving complete inferential outputs. These results demonstrate the computational advantage at the large-library scale targeted by our structural contribution (Section 5.1, Table 1, p. 6). The statistical validity of the large-report experiment is addressed separately in Q1.

Together, these findings establish three complementary benefits: verified conditional protection on a real Pareto report, improved information recovery relative to splitting, and efficient complete-report inference at scale.

AuthorsOpenReview ↗

Response to Reviewer pafY #2

CW2. Evidence for Plug-in Calibration

We appreciate the reviewer's focus on real-data calibration. Independent conditional confirmation supports empirical calibration on the verified complete reports, while additional experiments characterize the approximation's limitations.

Independent calibration evidence. Across the six original image/Bank reports, carving achieves adjusted joint-error upper bounds of 2.40–3.97%, below the 5% target under the 0.025/0.025 coordinate–comparison allocation. Further independent confirmation establishes 96.30%, 96.85%, and 97.00% conditional simultaneous coverage on the five-objective image, two-objective image, and two-objective Bank tasks, respectively. Their multiplicity-adjusted error upper bounds are 4.512%, 4.481%, and 4.305%, all below 5%.

Table P2. Independently confirmed conditional calibration on real model reports.

Frozen-report taskConditional repetitionsSimultaneous coverageAdjusted error upper bound
Image, five objectives4,00096.30%4.512%
Image, two objectives2,00096.85%4.481%
Bank, two objectives2,00097.00%4.305%

All experiments use n=5000n=5000, f=0.9f=0.9, and a separate 5% coordinate simultaneous-inference budget. The five-objective study uses one-sided MC bounds adjusted across eight risks; the two-objective studies use two-sided intervals adjusted across a separate family of 16 risks.

Approximation limits. All six tested image settings with n=500n=500–10001000 exhibit undercoverage (FCR 0.067–0.087). Bank's oracle-covariance diagnostic also shows 7.2% overall misses, reaching 10.8% at n=1000n=1000, f=0.8f=0.8, indicating that covariance estimation is not the sole source of approximation error. The revised abstract and Section 7 distinguish exact Gaussian validity from empirically confirmed real-score calibration, without asserting a universal sample-size threshold.

Theoretical scope and finite-sample alternative. Proposition 2 establishes hard-pivot asymptotics under positive limiting report probability; Proposition 3 gives exact nested-Gaussian carving; and Theorem 2 bounds plug-in carving error. The latter's numerical envelope reaches 1 at the evaluated scales, making independent confirmation the primary empirical calibration evidence. For bounded i.i.d. scores, Appendix N.2 provides a finite-sample valid independent-holdout alternative, with widths 1.644–2.959 times Gaussian splitting across the original six settings. Unlike the plug-in asymptotic argument, this guarantee requires no lower bound on report probability, including for the rare K=1024K=1024 reports examined in Q1.

These results establish empirical calibration in independently verified real settings, identify the operating limits of Gaussian approximation, and provide a finite-sample alternative when stronger guarantees are required.

See Section 5.2, Section 4.2, Appendices P.1 and Q.1–Q.2 for calibration results, and Appendices J.5, N.2, O.6, and Q.5 (pp. 65–66) for approximation diagnostics and validity conditions.

AuthorsOpenReview ↗

Response to Reviewer pafY #3

Q1. Probabilities, Calibration, and Contribution of the 1,024-Configuration Experiment

We appreciate the reviewer's questions regarding report frequency, statistical validity, and scalability. The large-library experiments establish efficient exact support reconstruction and complete-output preservation, complemented by a finite-sample valid inference option for rare reports.

Report probability. The three original K=1024K=1024 reports contain 180, 234, and 190 labels, respectively. None recurs in 2,000 independent selection draws, giving estimated frequencies of 0/2000 and a simultaneous 95% upper bound of 0.2045% per report. This confirms that the reports are rare, without implying zero probability. The audit preserves the original model roster, objectives, selection size, and offset rule (Appendix Q.4).

Conditional validity and finite-sample confirmation. Report-conditional coverage of the original plug-in carving intervals has not been established at this scale. We therefore additionally evaluate the independent bounded-score holdout procedure from Appendix N.2, which provides finite-sample protection without requiring a lower bound on report probability.

For each frozen report, 2,000 independent repetitions use the original 15,211-row score population, five objectives, n=5000n=5000, f=0.6f=0.6, and 2,000 holdout records independent of selection. The procedure covers all 900, 1,170, and 950 reported coordinates in 100% of repetitions, with a multiplicity-adjusted 95% conditional-error upper bound of 0.274% per report. Because inference uses independent holdout data, conditional validation does not require repeatedly selecting the rare report.

Table P3. Conditional coverage and interval width in the 1,024-configuration library.

Report size / coordinatesProcedureSimultaneous coverageMean coordinate widthPaired width ratio
180 / 900Bounded holdout100.00%0.076023.011
180 / 900Gaussian splitting2.95%0.04251.000
234 / 1,170Bounded holdout100.00%0.077317.384
234 / 1,170Gaussian splitting2.90%0.04321.000
190 / 950Bounded holdout100.00%0.07677.810
190 / 950Gaussian splitting5.00%0.04301.000

Coverage requires simultaneous protection of every coordinate in the frozen report. Each procedure is evaluated on 2,000 repetitions per report, with a 5% report-level error budget. MC correction covers six report–procedure risks. Paired width ratios average within-record ratios and differ from ratios of mean widths.

Precision and scalability. Finite-sample protection incurs a measurable width cost: mean coordinate widths are 0.0760–0.0773, compared with 0.0425–0.0432 for Gaussian splitting. The larger paired width ratios reflect near-zero Gaussian widths on some coordinates. Separately, the report-sensitive and all-pair backends agree on all 5,436 target instances, including supports, endpoints, p-values, Holm decisions, and report summaries. Witness compression achieves 4.84× faster support reconstruction and 1.95× median end-to-end acceleration at K=1024K=1024.

These findings establish computationally efficient complete-report reconstruction at scale, exact agreement with all-pair inference outputs, and an independently validated finite-sample protection option for rare reports, with its precision cost explicitly measured.

See Section 5.1, Table 1 (p. 6), Appendix P.3, and Appendix Q.4, Table 58 (pp. 64–65), for the complete protocols and results.

Q2. Conditional Undercoverage in the Original and Added Report Groups

We appreciate the request for direct evidence of conditional undercoverage in real model libraries. The newly confirmed 21-model image report provides statistically significant evidence of conditional undercoverage for both global max-T and LSI.

Original report groups. Among the 14 original CIFAR label groups, the smallest one-sided exact p-value is 0.06077, and Holm correction across 28 tests yields no rejection. The six original independently confirmed groups likewise show no significant global/LSI undercoverage.

Reviewer_FftrOpenReview ↗
Rating
3
Confidence
3(+1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses the problem of reporting confidence intervals for models that were selected as a Pareto front. A library of candidate models is evaluated on several objectives using one test set, the non-dominated models are reported, and intervals for their scores are computed from the same test set. The authors point out that this is a post-selection problem: models with overestimated test-set scores are more likely to be on the front, so the scores of the reported models are too high on average and nominal 95% intervals miss the true value more often than 5% of the time. To correct for this the authors use data carving, where the front is selected on a fraction of the test set, the estimate uses the whole test set, and the contribution of the selection fraction is treated as a normal variable restricted to the set of values that would have produced the same front. Carving the estimate this way requires knowing that set exactly, and for a Pareto front it has a complicated form, since as a model's score on one objective varies the model can enter and leave the front several times, each time because of a different competitor, so the set is a union of separate intervals with gaps between them. The main technical contribution is an exact and fast way to compute it, based on the fact that every omitted model is dominated by a reported model, so only the reported models need to be compared against the rest. From this they obtain exact intervals under a Gaussian model with known covariance, together with an error budget that covers all intervals and pairwise comparisons in a report at once. They evaluate on image-classifier and tabular model libraries and conclude that conditioning on the published front gives intervals that are calibrated and narrower than splitting.

Critical strengths

The paper extends data carving from single-objective to multi-objective model selection, which gives practitioners a method they can apply to multi-objective model comparisons. Carving is an established way to get valid intervals after selecting the best candidates on one metric while still using the whole test set, and it is the only such method whose intervals are narrow enough to be useful; the paper's own comparisons show this, with carving intervals about a third narrower than sample splitting at the same validity. Until now it could not be applied when the shortlist is a Pareto front, which is how multi-objective comparisons are usually published. The method here needs only the scores, an estimate of their covariance and a selection fraction, and is not tied to any model class or objective, so it can be applied to a model comparison without further development. This matters to my assessment because the contribution is a practical procedure for a situation that occurs routinely, not only a theoretical construction.

The selection event of a Pareto front is computed exactly and cheaply. Carving requires the set of score values under which the published front would have been the same. For a Pareto front that set is a union of disconnected intervals, and a direct construction compares all pairs of candidates. The authors show that it is enough to compare the reported candidates against all other candidates, because every omitted candidate is dominated by a reported one, which brings the cost from quadratic in the library size to linear in the front size times the library size, with no approximation involved. Exactness is necessary here: the paper's learned-law baseline, which approximates the same quantity, fails its tolerance in every run. This matters to my assessment because the construction is general: any conditional inference procedure for Pareto selection needs this set, not only the carving used here.

Critical weaknesses

The positioning of the paper overstates the results. None of the reported numbers are wrong, but the abstract and introduction present the method as more precise and better calibrated than existing methods, and the experiments do not support that reading. Specifically: the precision claim is made against sample splitting, which carving is proven never to exceed in width, so the comparison can only come out one way. The methods a practitioner would otherwise use, global max-T and locally simultaneous inference, are compared with carving only in the appendix, and there carving is no narrower on the image pool and wider, with fewer true claims, on the Bank library (Tables 4A, 25, 45). The calibration claim is stated without qualification, while all image cells with n of 1000 or fewer undercover (false coverage rate 0.067 to 0.087 against 0.05, Table 19); this is a limitation of the plug-in Gaussian approximation rather than of carving, since splitting undercovers equally there, but the main text should say so instead of describing it as affecting "some" settings.

The real-data experiments show that the per-report guarantee is not significant in practice. On the image and Bank model libraries, the simultaneous methods (global max-T and locally simultaneous inference) are at or within one Monte Carlo standard error of nominal coverage on every frozen report (Table 44), and all joint error bounds are below 0.05 (Table 43). The only setting in which they undercover on a published front is the two-model Gaussian example (Table 3, Proposition 4). The paper should either show a realistic library where the simultaneous methods fail on a published front, or state that the practical difference is small at these scales.

Questions for authors
  1. Table 22 reports a "membership" method that is not defined anywhere in the paper. What does it condition on, and how do its width and coverage compare with carving on the image pool?

  2. How does the self-certifying construction relate to the parametric-programming approach of Le Duy and Takeuchi, which also computes the selection event along the line of the test statistic as a union of intervals?

  3. Will code and the audit artefacts referenced in the text (the companion numerical-validation record, the source CSVs for figures, the 180-test regression suite) be released? The empirical section depends on these for verification and none are attached.

4 comments
AuthorsOpenReview ↗

Response to Reviewer Fftr #1

CW1. Positioning, Precision, and Calibration

We appreciate the reviewer's careful assessment. Our results distinguish information recovery relative to matched splitting from report-conditional protection relative to global max-T and LSI.

Precision and information recovery. The 33.6% reduction in the abstract refers specifically to mean paired marginal width versus matched splitting. Under the nested Gaussian model with common known covariance, carving is theoretically no wider than splitting for the same report, target, and error budget. The experiments quantify the practical benefit: carving supports 0.684 additional true comparisons per report (paired 95% MC interval [0.633, 0.735]). This empirical gain is not determined by the theoretical bound alone.

Stronger baseline comparisons. On the primary image task, normalized simultaneous widths are 0.666/0.663/0.649 for carving/global max-T/LSI, respectively. Carving nevertheless supports 0.207 more true comparisons than global max-T (paired interval [0.174, 0.240]). On Bank, global/LSI achieve higher comparison yields, while carving improves over splitting across all six frozen-report groups. These results clarify the trade-off between precision, supported comparisons, and conditional protection (Sections 4.1 and 5.2–5.3; Table 4, p. 8; Table 46, p. 56).

Calibration and validity. Exact conditional validity holds under the stated Gaussian model; real-score experiments use plug-in approximations. The primary image experiment achieves FCR 0.05033 (95% MC interval [0.04411, 0.05656]). Independent confirmation further establishes 96.85% conditional simultaneous coverage, with an adjusted error upper bound below 5% (CW2). The approximation limits are explicitly documented: all six image settings with n = 500–1000 have FCR 0.067–0.087, and Bank's oracle-covariance diagnostic also shows undercoverage. Theorem 2's conservative bound is uninformative at the evaluated scales; a finite-sample bounded-score holdout alternative is provided with its measured width cost (Section 4.2; Appendix N.2). The revised abstract, Section 7, and Appendix Q.5 (pp. 65–66) clarify these conditions.

Together, these findings establish measurable information recovery and independently verified report-conditional protection, while explicitly characterizing their precision costs and applicability.

CW2. Practical Value of Complete-Report Protection

The new independent confirmation experiment demonstrates a practical advantage of complete-report conditioning on real model predictions, directly addressing whether aggregate coverage protects a specific published Pareto shortlist.

Experimental design and conditional coverage. All 21 image models are retained, with selection based on clean accuracy and negative Brier score. A selection-only screen freezes the complete report before examining coverage outcomes. At n = 5000 and f = 0.9, 2,000 accepted repetitions yield 96.85% conditional simultaneous coverage for carving, compared with 92.70% for global max-T and 89.35% for LSI. Yet across unconditional draws, global max-T and LSI achieve 99.50% and 99.05% coverage, illustrating the distinction between aggregate and report-conditional reliability.

Statistical significance and precision. After simultaneous Monte Carlo correction, carving's conditional-error upper bound is 4.481%, below the 5% target, whereas global max-T and LSI have lower bounds of 5.689% and 8.710%, respectively. Thus, both simultaneous baselines exhibit statistically supported conditional undercoverage on this real report. Carving also reduces mean paired interval width by 31.8% versus matched splitting, although global/LSI produce narrower intervals.

Table F2. Conditional coverage and interval width on the frozen image report.

MethodConditional coverageAdjusted error intervalWidth / splitting
Carving96.85%[2.115%, 4.481%]0.68233
Matched splitting94.85%[3.804%, 6.778%]1.00000
Global max-T92.70%[5.689%, 9.178%]0.38240
LSI89.35%[8.710%, 12.836%]0.36274

All methods use the same 5% coordinate simultaneous-inference budget. Coverage uses 2,000 conditional repetitions; widths use 100 prespecified paired records. Two-sided MC intervals are adjusted across 16 risks, separately from the inference budget.

AuthorsOpenReview ↗

Response to Reviewer Fftr #2

The report occurs in 13/2,000 independent selection draws (0.65%). The original six groups and Bank control remain reported and show no multiplicity-adjusted evidence of global/LSI conditional undercoverage. The new experiment therefore identifies a concrete real-world setting where complete-report conditioning provides additional protection, rather than claiming a uniform advantage across reports.

The key contribution is statistical reliability for the specific Pareto report being published, with a directly measured trade-off between conditional coverage and interval precision.

See Section 5.2, Table 3 (p. 7) and Appendix Q.1, Table 49 (pp. 59–61) for the protocol and complete results.

Q1. Membership Conditioning and the Matched Comparison

We appreciate the opportunity to clarify the distinction between membership and complete-report conditioning. Membership conditions on Mk={k∈P^(S)}M_k=\{k\in\hat P(S)\}, whereas our method conditions on EP={P^(S)=P}E_P=\{\hat P(S)=P\}, with SS including the selection offset. The original membership baseline uses hard conditioning, including the projection residual, while our method employs full-report scalar carving. On the primary image pool, their pooled marginal coverages are 94.64% and 94.96%, with normalized widths of 0.89624 and 0.66392, respectively (Table 54, p. 63).

Matched comparison under scalar carving. To isolate the conditioning event from the data-reuse mechanism, we additionally apply scalar carving to both methods. For the fixed clean-accuracy target, full-report and membership conditioning achieve 95.00% and 95.15% coverage over 2,000 conditional repetitions. Across 400 prespecified width records, the mean membership/full-report width ratio is 0.99645 (95% MC interval [0.99466, 0.99824]). Thus, complete-report conditioning provides the stronger report-level guarantee at a width cost of approximately 0.35%, while remaining 28.1% narrower than matched splitting.

Definitions and conditioning distributions appear in Appendices J.1 and Q.3; matched results are in Tables 55–56 (pp. 63–64), with a summary in Section 5.3.

Q2. Relationship to Le Duy and Takeuchi's Parametric Programming

We appreciate the connection to parametric programming. Both approaches recover selection events along a one-dimensional inferential line, but our contribution lies in the Pareto-specific structural characterization and its computational complexity.

Le Duy and Takeuchi recover selection regions by tracing parametric solution paths. In contrast, a complete Pareto report is self-certifying: its reported candidates provide the dominance witnesses needed to characterize every omitted candidate. This reduces reconstruction to s(K−1)s(K-1) directed relations, yielding complexity O(Ksd+Kslog⁡(Ks+1))O(Ksd+Ks\log(Ks+1)) while preserving all witness switches and disconnected support components. An Ω(Ks)\Omega(Ks) explicit-output lower bound further establishes near-optimal dependence for fixed dd and sparse reports.

The revised Section 6 discusses this relationship and cites their AISTATS 2021 and JMLR 2022 papers. The structural results and proofs appear in Section 3 and Appendix A.3.

Q3. Code and Audit Materials

The anonymous repository now provides the code and audit materials required to reproduce and verify the reported results. Release r20261110.2 includes inference implementations, frozen configurations and score pools, 14 figure-source CSVs, numerical-validation records, and executable reproduction commands, extending the manuscript's r20261110.1 package.

The current release passes 214 tests, comprising the unchanged 145-test core suite and 69 extended tests, with complete inventories and execution logs. The previously cited 180-test count originated from an earlier development record without a recoverable itemized inventory; the current 214-test suite provides the auditable reference.

Independent clean-extraction validation reproduced all 8,000 two-objective confirmation records, 400 width records, and 6,000 large-library bounded-holdout records byte-for-byte, supporting reproducibility of the principal new experiments.

Detailed verification instructions and audit records are provided in REPRODUCIBILITY.md, NUMERICAL_VALIDATION.md, ALL_TEST_INVENTORY.csv, and validation/github_clean/VALIDATION_SUMMARY.json. The manuscript's Reproducibility Statement and Appendix Q.6 (p. 66) describe the core release, with additional materials documented in the repository.

CW1. Separating the gain from splitting is what I was looking for. Thanks for clarifying.

CW2. My concern was that the paper showed global max-T and LSI failing only in synthetic tests. However, the new tests address it. I would add two facts next to that result in the main text: the report occurs in a low percentage of selection draws (13 of 2,000, 0.65%), and the carving interval on it is wider than global max-T (0.682 against 0.382 of the splitting width).

Q1. Good to know the cost of conditioning on the full report. That resolves my worry.

Q2. Thanks. The clarification is what I need.

Q3. Thanks for releasing the code.

I will keep my rating and raise my confidence. Thanks for clarifying all of the points.

Dear Reviewer Fftr,

 

Thank you for your thoughtful review and for taking the time to follow up on our rebuttal. We appreciate your encouraging feedback and are glad that the new experiment addressed your concern.

Regarding CW2, we will revise the main text as you suggested and report both how often the Pareto report occurred and how the interval widths compare alongside the coverage results.

 

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Reviewer Responses and Manuscript Revisions

We thank the Area Chair and all three reviewers for their time and careful feedback. We have uploaded a revised manuscript with changes marked in red. The main text ends on page 9. Section and table numbers below refer to the revised manuscript.

  1. Conditional coverage on a real Pareto report (Fftr CW2 and pafY CW1, Q2). We froze a report from the 21-model image library using a selection-only screen and evaluated it in 2,000 independent conditional repetitions. Carving achieved 96.85% simultaneous coverage, compared with 92.70% for global max-T and 89.35% for locally simultaneous inference (LSI). Both baselines exceeded 99% aggregate coverage, yet their conditional undercoverage on this report remained significant after multiplicity correction. Carving was 31.8% narrower than matched splitting and wider than both baselines. The original report groups and Bank control showed no significant conditional undercoverage for either baseline (Section 5.2, Table 3, and Appendix Q.1).

  2. Precision and the choice of conditioning event (Fftr CW1, Q1 and pafY CW1, Q3). On the primary image task, carving reduced mean paired marginal width by 33.6% and supported 0.684 more true comparisons per report than matched splitting. Simultaneous width ratios on the image task were 0.666 for carving, 0.663 for global max-T, and 0.649 for LSI. On Bank, both baselines supported more true comparisons than carving. In a matched scalar-carving comparison on one image target, conditioning on the complete shortlist cost about 0.35% in width relative to conditioning on target membership alone and remained 28.1% narrower than matched splitting (Sections 5.2–5.3, Table 4, and Appendix Q.3, Tables 55–56).

  3. Gaussian assumptions and finite-sample validity (Fftr CW1, pafY CW2, and AdJ1 CW1). Support reconstruction uses only Pareto dominance along a given affine score line. Exact conditional inference requires joint Gaussianity and known covariance. With estimated covariance at n=5,000n=5{,}000, carving attained 96.30–97.00% conditional simultaneous coverage across three independently confirmed reports. All six image settings at n=500n=500–1,0001{,}000 had false coverage rates of 6.7–8.7% against a 5% target. For bounded i.i.d. scores with known ranges, the finite-sample independent-holdout alternative used intervals 1.64–2.96× as wide as Gaussian splitting across six settings (Sections 4.2 and 7 and Appendices N.2 and Q.1–Q.2).

  4. Scalability and rare reports (pafY CW1, Q1 and AdJ1 CW2). In the 1,024-configuration Bank library, witness compression accelerated support reconstruction by 4.84× and yielded a 1.95× median end-to-end speedup, with agreement on all 5,436 audited targets. None of the three reports reappeared in 2,000 selection draws. Plug-in carving coverage remains unverified for these reports. The independent bounded-score holdout covered every reported coordinate in all 2,000 repetitions per report, with wider intervals than Gaussian splitting (Table 1 and Appendix Q.4, Table 58).

  5. Technical novelty and reproducibility (Fftr Q2–Q3 and AdJ1 CW2). Section 6 clarifies the connection to parametric programming and explains how dominance witnesses from the reported candidates reconstruct the complete Pareto selection event using only s(K−1)s(K-1) directed relations. It also distinguishes inference on a published shortlist from population Pareto-set identification and risk-constrained validation. The updated release r20261110.2 expands the supplementary package’s 145-test core suite to 214 tests and provides frozen protocols, figure-source CSVs, numerical audits, and reproduction commands (anonymous repository).

We respond to each reviewer in their own thread.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.