- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how imperfect provenance affects the value of budgeted shadow evaluations for online model selection. Under bounded i.i.d. losses and marginal-preserving independent resampling, it introduces the posterior-contamination spectrum , where is the conditional probability of resampling given the observed provenance. This quantity controls both achievable comparison precision and information-theoretic lower bounds, yielding minimax regret characterizations up to logarithmic factors. The paper constructs channels with identical matching accuracy and mutual information but different regret orders, and develops calibrated weighting and order-only algorithms that exploit provenance quality. Validation consists of controlled synthetic experiments on two-action service regret, multi-action certification and evaluation savings, and target-preservation audits. The results illustrate two-action service gains and improved certification, while highlighting limited additional multi-action regret gains and the importance of post-budget continuation.
Critical strengths
- Matching upper and lower bounds establish the posterior-contamination spectrum as a principled characterization of provenance value.
- The equal-accuracy and equal-mutual-information separation shows why aggregate provenance metrics can miss substantial differences in learning performance.
- Calibrated and order-only algorithms make the theory constructive, with the latter retaining polynomial regret rates without numerical calibration.
- The explicit treatment of target preservation clarifies when provenance-based selection and weighting remain statistically valid.
Critical weaknesses
- Restrictive observation model. The characterization assumes marginal-preserving independent resampling, excluding score-dependent sample retention and distribution-shifting mismatches that can invalidate provenance-based weighting.
- Fragile fast-rate regime. Any fixed positive intrinsic noise floor restores square-root asymptotics; the experiments do not establish that realistic evaluations satisfy the conditions needed for faster rates.
- Limited evidence of service gains. Simultaneous intervals for tested-gap maximum-risk improvements include zero, and no multi-action provenance-versus-pooled regret improvement survives multiple-testing correction.
- Provenance semantics are assumed. The algorithms require calibrated contamination probabilities or a correct quality ordering, while learning these reliably from limited audits remains outside the guarantees.
Questions for authors
-
In the clock and request-ID constructions, what practical constraint prevents exposing the readiness bit or complete audit readout? Since these stronger interfaces provide an exactly clean stratum, their availability and cost affect the practical relevance of the intermediate-rate regimes.
-
Using the noise-floor transition analysis, how small must be for provenance-aware learning to improve meaningfully over the marginal baseline at the reported horizons and budgets? What supports this noise scale in the motivating applications?
-
The multi-action experiments show no statistically significant provenance-specific regret improvement. Can the recorded trajectories establish whether the bottleneck is conservative confidence widths, query scheduling, or pooled service orientation, and explain when improved comparison precision actually changes production decisions?
2 comments
Thank you for the substantive revisions. The E1 replication strengthens the two-action evidence, and E2/E3 clarify the decision bottlenecks and provide encouraging multi-action results. My concerns are partially addressed, although practical evidence for the required noise levels, metadata costs, and reliable provenance calibration remains limited. The multi-action gains also remain sensitive to the inference method. I acknowledge these improvements while maintaining my rating of 3 (Weak acceptance).
Thank you for your careful review, for recognizing the matching upper and lower bounds, and for your helpful questions about interfaces, noise floors, and service decisions. We address these questions and the supporting evidence below.
Exact matching information and cost We recommend exposing exact matching information when available at the same cost (Section 5). The clock and request-ID constructions permit stronger readouts; partial logs are interface choices. Section 5 describes the clean-stratum benefit; Appendix E.2 shows that full audit readout weakly dominates its binary coarsenings and prices evaluation, joining, binding and audit work. We added an explicit scope statement in Section 5: the evidence is a controlled comparison of constructed interfaces; choosing partial metadata in deployment depends on its actual costs and constraints. Shadow savings are a separate resource benefit under a cap; regret plus evaluation/metadata cost is a different objective (Appendix A.2).
Noise-floor scales at the reported horizon Theorem 1 includes ν>0, and Appendix E.1 derives the transition budget. For a fixed channel and fixed N,v, a positive floor restores square-root asymptotics. We added finite-horizon comparisons in Appendix E.1: at N=v=2, γ=1, α=1/4 and T=B=2^18, production-to-paired scale ratios of 3, 5 and 10 require approximately ν≤0.036, 0.0087 and 0.0010. The numerator is the marginal scale sqrt(NT); these comparisons omit constants and logarithms.
We added E4 (Section 6.4; Appendix F.7). For α=1/4 at full budget, measured MOSS/floor-aware maximum tested-gap mean-regret ratios are 21.7/21.0/9.5/3.7 at ν=0/10-3/10-2/10^-1. The corresponding unweighted/floor-aware ratios are 5.25/4.80/2.23/1.01; these measure provenance's value within paired feedback, a different baseline. These 128-cluster results are a post-plan descriptive enlargement; the planned 32-cluster analysis reports nonmonotone small-floor ratios and a setting where fixed-scale EO exceeds unweighted regret. Small budgets can favor MOSS. These synthetic thresholds provide audit benchmarks; application-specific benefits require measuring the pipeline's paired disagreement and floor.
Confidence widths, query scheduling, and service decisions Appendices C.3/C.10 separate query-set and orientation cost. The handoff study in Section 6.2/F.5 identifies continuation as the largest service effect, with provenance-specific contrasts unresolved after correction.
We added E2 trajectory diagnostics (Section 6.4; Appendix F.7). At full budget, calibrated/rank PCE serve the same actions as pooled PCE on every path. Confidence intervals remain too wide to resolve the smallest gap, so no run certifies; earlier elimination saves calibrated PCE 124.3 shadows on average in the N=4, α=1/4 full-budget cell. Query-set cost accounts for 84.4%/77.4% of pooled pre-exhaustion regret for N=4/8. At N=4, α=1/4 and B=T/4, the only cell with changed service, four paths differ after exhaustion and a later phase completion.
We added E3 to test provenance-aware orientation, retaining each variant's own elimination bank and a common MOSS handoff. At N=4, T=2^20, α=1/4 and full budget, maximum tested-gap mean regret is 431.24/422.15 for calibrated/rank variants versus 511.36 pooled. Six of eight planned contrasts pass bootstrap-Holm; all projected contrast intervals include zero. Mean shadow use is unchanged and no run certifies. A separate pooled-bank/calibrated-orientation control isolates orientation and gives 400.42 descriptively; the eight planned tests concern complete variants.
We added a pre-specified E1 replication with 128 fresh clusters. At T=B=2^18 and α=1/4, weighted/order-only maxima are 15.41/16.41 versus 61.70 unweighted, supported by bootstrap-Holm and Student-t/Bonferroni projected contrast intervals. At α=3/4, bootstrap-Holm supports both gains while projected contrast intervals include zero.
Conditions and robustness Section 5 requires conditional mean preservation; Appendix E.2 gives a selection-induced gap reversal even with correct matching. We added Proposition 6 in Appendix E.3: under Definition 1, stationary, mean-preserving interfaces with fresh tags and an excess-moment envelope admit upper bounds: calibrated PCE uses known envelope values; ordered PCE uses its monotone order. The matching converse remains resampling-specific. Within these conditions, new Proposition 7 retains the erased-pairing rate up to constants/logarithms for PCE banks with fixed misspecified semantics. E4/E5 measure the finite-horizon sensitivity of fixed-scale EO separately. The sharper spectral rates use calibrated probabilities or a correct order; independent audits assess these inputs before transfer across model pairs.