- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the reliability diagnosis of low-order interaction surrogate models under variations in compound parameter distributions. The authors construct linear surrogates using functional ANOVA and polynomial moment features, while constraining the true interaction response via simultaneous empirical Bernstein confidence intervals. By employing robust re-fitting linear programming to derive upper bounds on error, and utilizing left-null-space dual certificates to establish lower bounds on error for the entire surrogate class, the approach enables the diagnosis of specific issues: surrogate sufficiency, coefficient mismatch, feature insufficiency, or a lack of sufficient evidence.
The experiments encompass combinations of image pre-processing steps and language decoding parameters, demonstrating applications such as feature expansion, re-fitting after transfer, and the selection of operator sequences. The provided guarantees are restricted to product mixtures of the marginal distributions for which evaluations have been performed.
Critical strengths
-
Category-level diagnostics are more valuable than the validation error of an individual model. The paper draws a clear distinction between cases where "the current coefficients are insufficient" and those where "no set of coefficients can meet the requirements." By establishing lower bounds for entire feature categories—while allowing for an "unresolved" status—this diagnostic approach becomes more rigorous and interpretable than simply examining fitting residuals.
-
There is a strong alignment between the experiments and the theoretical claims. Elements such as independent fitting, the use of certified random samples, simultaneous coverage, the leveraging of covariance in shared evaluations, and the construction of distributions that share the same mean but differ in higher-order moments all directly support the core arguments. Sensitivity analyses regarding sample size and confidence levels further bolster the credibility of the results.
Critical weaknesses
-
The theoretical contributions and the scope of applicability are not sufficiently highlighted. The core framework combines ANOVA, empirical Bernstein bounds, robust Chebyshev approximation, and LP duality. The paper needs to articulate more clearly what new theoretical capabilities or practical diagnostic advantages it offers compared to existing methods for testing models with set-based constraints. Furthermore, the guarantees provided are specific to fixed evaluation instances and designated distribution families; they do not constitute general out-of-distribution (OOD) generalization guarantees. Additionally, the empirical analysis is limited to two-parameter interactions.
-
The certification process is computationally expensive, yet there is no comparative analysis to justify this cost. Some experiments require hundreds of full-dataset evaluations, yet the paper fails to compare its approach against alternatives—such as direct importance-weighted estimation or vertex-response caching—under equivalent computational budgets and precision requirements. Consequently, it remains unclear under what types of workloads the upfront certification costs are justified. Moreover, there is insufficient disclosure regarding numerical error budgets, the verification of LP feasibility, and key experimental configurations—details that are particularly critical for a paper centered on "rigorous certification."
Questions for authors
Compared to existing set-membership/model-invalidation and robust approximation methods, what new theoretical capabilities does this paper introduce? Could you illustrate this with specific differences in results or concrete examples? Given the same inference budget and certification precision, what advantages does this approach offer over direct estimation or vertex caching? Please provide data on the number of function calls, actual execution time, and cost amortization.
5 comments
2/ Comparison to vertex caching and direct importance weighting under equal budget (Weakness 2 & Question 2). On ResNet-50 under ImageNetV2 resampling/JPEG shifts, the table below compares direct vertex response caching and . One replicate experiment evaluates all 10K images using both operator orders for a total of 20K forward passes.
| Representation | Separate fitting passes | Shared offline certification passes | Certified upper bound over product mixtures (pp) |
|---|---|---|---|
| Direct vertex response table | 0 | 15.36 million (768 replicates) | 0.0406 |
| Two-term surrogate | 80K (4 replicates) | 15.36 million (768 replicates) | 0.1349 |
Within the evaluated product-mixture family, the 16-vertex table retains all grid estimates without representation truncation and thus achieves the tighter upper bound. In comparison, compresses the response into two fitted coefficients multiplying moment features, while still meeting the 0.15 pp tolerance. Both approaches allow for the subsequent product-mixture queries to be answered in 3-5 microseconds without any model calls. Beyond response querying, our framework provides a certified choice of representation, which determines when interaction terms can be omitted altogether ( in ResNet-50 tone/noise and MMLU), when coefficients can be adjusted, and which moment features are essential. In fact, the computational cost required to derive these structural conclusions is significantly lower than the full evaluation figures shown in the table, as Figure 6 (Appendix B.3) already certifies both class exclusions at 192 replicates and adequacy at 384 replicates. Also, larger decision margins in operator-order selection allow for the use of even coarser bounds (Section 6).
In addition to the evaluated mixture hull, the matched-mean construction in Section 5.2 includes a distribution with second shifted-Legendre moment . This distribution belongs neither to the family of probability mixtures of exponential tilts (see Appendix F.2) nor to the 16-vertex interpolation guarantee. To cover new target families outside this hull via direct importance weighting, one would either need to collect fresh evaluations or to reuse evaluation data from a proposal distribution that covers the new targets. By contrast, retains only the two scalar coefficients from its initial 4-replicate fit to generate predictions based on the moments of a target distribution. This process requires no new model calls and no refitting at all, while a separate 64-replicate (1.28 million pass) evaluation independently certifies that the transfer error for the matched-mean pairs remains within 0.092 pp.
3/ Runtime and workload (Weakness 2 & Question 2). Collecting all 26.88 million forward passes for the four primary ImageNetV2 settings took 16.9 GPU hours in total. In contrast, solving each prepared class-lower-bound or robust-refit LP over the 16 non-reference targets takes about 1 ms on a single CPU thread. Thus, the collection of evaluation data dominates runtime, which is necessary in any case to reduce finite-sample uncertainty and also already shared across all shifted targets under our framework. Without a reusable surrogate model, evaluating new distribution shifts would ask for fresh GPU-based benchmarking for every uncached shift category. In contrast, our upfront inference distributes across two regimes: (1) Lightweight fitting or re-fitting (4-16 replicates or 80K-320K passes in ResNet-50) takes only minutes to estimate a set of shared coefficients to serve repeated queries across shifting distributions with no additional GPU calls. (2) Full offline certification (192-768 replicates) serves as a one-time qualification audit for a stable predictor. A single shared audit can then justify omitting the interaction term () and establish a minimal moment basis or operator order, which covers all continuous product mixture distributions of the audited marginals without further model evaluations.
4/ Numerical error budgets and solver-independent verification (Weakness 2). Neither our structural exclusions nor our candidate adequacy certificates depend on floating-point LP solver tolerances. For lower bounds, our central class and term-omission exclusions rely on closed-form analytic dual vectors whose feasibility identities ( and ) hold in exact arithmetic and don't rely on numerical LP solvers. For upper bounds, any fitted or HiGHS-refitted coefficient vector is treated only as a candidate. Then, it is verified post-hoc through in Eq. (3) using interval enclosures of and with outward rounding. Therefore, at worst, HiGHS stopping tolerances can yield a slightly more conservative upper bound but never an invalid certificate. Through independent cross-verification using exact rational arithmetic, we reproduced the central structural and candidate bounds reported in the paper. In the primary vision evaluation, the total numerical allowance added to each score interval in Algorithm 1 remains strictly below pp, which is orders of magnitude below the 0.15 pp tolerance.
Under a 10-page budget, we will clarify the connection to set-membership invalidation and cost amortization in the main text. We can also include the caching comparison, forward pass and runtime summary, and verification details reported above in Appendix C.3.
Thank you very much for your thorough review and for highlighting the value of class-wide diagnostics and the close alignment between our theory and experiments! Our goal is certified model reduction of the interaction surrogate for a frozen predictor's pipeline, so as to determine which distributional moments a compact interaction surrogate must retain, when coefficient refitting suffices, and when its feature representation must expand.
1/ Capabilities beyond a coordinate-wise set membership test and scope of guarantees (Weakness 1 & Question 1). A coordinate-wise set membership test starts from an uncertainty box and asks whether a chosen model class contains a response compatible with it. With this, we turn shared randomized pipeline evaluations into simultaneous finite-sample certificates of adequacy, parameter misfit, and feature inadequacy, which provides three capabilities for choosing and maintaining a compact surrogate.
Regarding scope, our experiments condition on fixed benchmark instances to isolate shifts in pipeline parameter distributions from the variability introduced by benchmark sampling. To achieve a population-level guarantee, these conditional certificates would need to be combined with an outer sampling bound over independently drawn instances. Across parameter distributions, the guarantees apply to continuous product mixtures of the evaluated marginals (Section 5.1) and specified perturbation families certified in Section 5.2, instead of unconstrained out-of-distribution shifts. Also, because the LPs in Section 3 require only predictions linear in coefficients (Xb) and valid simultaneous response intervals, higher order tensor-moment dictionaries for three or more operators can be integrated into the same LP framework directly. As the number of operators increases, the main scaling challenge lies in controlling joint importance weights and score variances across larger target and feature sets, which actually further highlights the importance of determining when low order interaction terms shall suffice.