- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses the post-selection inference problem in multi-objective evaluation and presents an efficient approach to generate a Pareto report of size s among K candidates and a near-optimal algorithm to reconstruct the complete same-report selective support.
Critical strengths
- The paper addresses an important problem in model evaluation and selection.
- The proposed approach can significantly reduce the cost of generating Pareto report and reconstruct supports. The method is designed elegantly with strong theoretical support.
- The experimental study shows that the approach achieves meaningful gain. The carving also reduces mean pair interval, making the estimation more precise than sample splitting.
Critical weaknesses
-
The paper assumes X (includes the stacked scores S and performance estimates T) are jointly Gaussian, with covariance known, for support reconstruction. This is a quite strong assumption. iThe paper should clarify the assumption and discuss how practical the assumption is in real world applications. Failing to do so will affect the applicability of the work.
-
The paper should further discuss the related work in multi-objective evaluation and clarify its novelty/advantages compared to those approaches.
Questions for authors
Please see the listed weaknesses.
4 comments
Response to Reviewer AdJ1 #2
CW2. Novelty Relative to Multi-Objective Evaluation Methods
We appreciate the reviewer's suggestion to clarify the distinction from existing multi-objective evaluation methods. Our central contribution is efficient, selection-aware inference with conditional guarantees for the complete published Pareto report, addressing a different statistical target from existing approaches.
Distinction from prior work. Pareto Testing and reliability-graph methods validate configurations against risk constraints, while Pareto-set identification aims to recover the population-optimal set. Stability-based policy evaluation addresses data reuse through algorithmic stability. In contrast, our framework reconstructs the complete selection event of the observed Pareto shortlist, enabling inference on its population performance and prespecified comparisons conditional on the exact reported label set. Unlike these approaches, it does not require identifying the population Pareto front.
Technical novelty and theoretical guarantees. Our key insight is self-certification: every omitted candidate is dominated by at least one reported candidate. This reduces complete-event reconstruction to directed relations, achieving complexity while preserving all disconnected support components and dominance-witness switches. An explicit-output lower bound establishes near-optimal dependence for fixed and . The resulting exact support enables Gaussian conditional inference and scalar carving to reuse evaluation data without changing the published report.
Computational validation. The trained Bank library demonstrates the benefit of report-sensitive reconstruction while preserving complete inferential outputs.
Table A2. End-to-end computation on the trained Bank library.
| Library size | Report size | Report-sensitive (s) | All-pair (s) | Median speedup |
|---|---|---|---|---|
| 64 | 36–40 | 8.44 | 8.51 | 1.01× |
| 128 | 55–61 | 14.27 | 14.57 | 1.01× |
| 256 | 90–100 | 25.87 | 27.65 | 1.07× |
| 512 | 127–165 | 46.59 | 58.91 | 1.26× |
| 1,024 | 180–234 | 74.21 | 145.08 | 1.95× |
Raw-row-to-report timings begin with loaded, warmed models. Speedups are medians of paired ratios.
At , the framework achieves 1.95× end-to-end acceleration, with a separate 4.84× gain in support reconstruction, while retaining the full selection event and inferential outputs.
The contribution therefore extends beyond Pareto evaluation itself: it establishes a computationally tractable connection between complete Pareto selection, exact selective inference, and data-efficient statistical reporting.
Section 3 and Appendix A.3 establish the theoretical results. Expanded Section 6 distinguishes our statistical target from existing multi-objective methods and clarifies the connection to Le Duy and Takeuchi's parametric programming. Sections 5.1–5.3 provide computational and inferential evidence (Table 1, p. 6; Appendix P.3, pp. 57–58).
Code, experimental configurations, and numerical audits are available in the anonymous repository.
Thanks the authors for the rebuttal response. I will maintain my score.
Dear Reviewer AdJ1,
Thank you for your thoughtful review and for taking the time to follow up on our rebuttal. We appreciate your encouraging feedback and are glad that the new experiment addressed your concern.
Best regards,
The Authors
Response to Reviewer AdJ1 #1
CW1. Role and Practical Scope of Joint Gaussianity and Known Covariance
We appreciate the reviewer's question regarding the assumptions and their practical applicability. The structural reconstruction is distribution-independent; joint Gaussianity and known covariance are required for exact conditional inference.
Role of the assumptions. Given an affine score line, the Pareto dominance structure alone establishes the complete-report event using s(K−1) directed relations. Joint Gaussianity ensures independence between the target statistic and its covariance-projection residual, while known covariance determines the projection direction and variance. Together, they yield the exact conditional distribution over the reconstructed support. Scalar carving further uses the nested-sample covariance model to reuse selection data while preserving conditional validity (Sections 2–4.1; Appendices A.2 and C).
Practical applicability and calibration. For real applications, Gaussian approximation is applied to averages of per-observation scores, with jointly estimated covariance capturing dependence across models and objectives. Independent confirmation on real model libraries achieves 96.85% conditional simultaneous coverage on Image and 97.00% on Bank, with multiplicity-adjusted error upper bounds below 5% in both cases. The primary image experiment also demonstrates a 33.6% reduction in mean paired marginal width relative to splitting.
Table A1. Conditional coverage and interval width with estimated covariance.
Both tasks use n=5000, f=0.9, and a 5% coordinate simultaneous-inference budget. Coverage is evaluated over 2,000 conditional repetitions per frozen report; widths are mean paired ratios from 100 prespecified records. The Bank control does not establish multiplicity-adjusted global/LSI conditional undercoverage.
Approximation limits and finite-sample alternative. The revised Sections 4.2 and 7 explicitly discuss small-sample undercoverage and the conditions required for Gaussian approximation. For bounded i.i.d. scores with known ranges, Appendix N.2 provides an independent-holdout procedure with finite-sample protection, at a measured width cost of 1.644–2.959 times Gaussian splitting across the original six settings.
These results distinguish the distribution-free structural contribution, exact Gaussian inference, and empirically validated real-score applicability, while providing a finite-sample alternative when Gaussian approximation is insufficient.
See Sections 2–4.2 and 7, Tables 3–4, and Appendices A.2, C, N.2, Q.1 (Tables 49–50, pp. 59–61), and Q.5 for details.