acceptodds
Under review as a conference paper at ICLR 2027

The Decision Value of Imperfect Provenance: A Minimax Characterization for Shadow Evaluation

Abstract

Online systems use shadow evaluations to compare candidate models or agents with the production choice. Same-request comparisons can cancel shared variation, but imperfect pairing leaves their quality uncertain. We study how provenance determines the value of a fixed shadow budget under bounded i.i.d. losses and marginal-preserving independent resampling. Two channels can have the same erased-score law, matching accuracy, and mutual information about request matching, yet yield constant versus square-root minimax regret at full budget. The relevant quantity is the posterior-contamination spectrum , where is the probability of independent resampling conditional on the observed provenance. This spectrum governs achievable comparison precision and the hard-family information bound for complete adaptive transcripts, giving minimax regret bounds that match up to logarithmic factors. It leads to calibrated inverse-variance weighting and an order-only method that retains the polynomial regret rates without calibration. In controlled experiments at , full budget, and a near-clean tail exponent of , the maximum tested-gap mean regret is for calibrated weighting, for unweighted pairing, and for order-only service.

Primary area:
learning theory
Keywords:
Online Learning, Multi-Armed Bandits, Minimax Regret, Heteroscedastic Feedback, Best-Arm Identification

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fC5YOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies how imperfect provenance affects the value of budgeted shadow evaluations for online model selection. Under bounded i.i.d. losses and marginal-preserving independent resampling, it introduces the posterior-contamination spectrum �0��0�, where �1��1� is the conditional probability of resampling given the observed provenance. This quantity controls both achievable comparison precision and information-theoretic lower bounds, yielding minimax regret characterizations up to logarithmic factors. The paper constructs channels with identical matching accuracy and mutual information but different regret orders, and develops calibrated weighting and order-only algorithms that exploit provenance quality. Validation consists of controlled synthetic experiments on two-action service regret, multi-action certification and evaluation savings, and target-preservation audits. The results illustrate two-action service gains and improved certification, while highlighting limited additional multi-action regret gains and the importance of post-budget continuation.

Critical strengths
  • Matching upper and lower bounds establish the posterior-contamination spectrum as a principled characterization of provenance value.
  • The equal-accuracy and equal-mutual-information separation shows why aggregate provenance metrics can miss substantial differences in learning performance.
  • Calibrated and order-only algorithms make the theory constructive, with the latter retaining polynomial regret rates without numerical calibration.
  • The explicit treatment of target preservation clarifies when provenance-based selection and weighting remain statistically valid.
Critical weaknesses
  • Restrictive observation model. The characterization assumes marginal-preserving independent resampling, excluding score-dependent sample retention and distribution-shifting mismatches that can invalidate provenance-based weighting.
  • Fragile fast-rate regime. Any fixed positive intrinsic noise floor restores square-root asymptotics; the experiments do not establish that realistic evaluations satisfy the conditions needed for faster rates.
  • Limited evidence of service gains. Simultaneous intervals for tested-gap maximum-risk improvements include zero, and no multi-action provenance-versus-pooled regret improvement survives multiple-testing correction.
  • Provenance semantics are assumed. The algorithms require calibrated contamination probabilities or a correct quality ordering, while learning these reliably from limited audits remains outside the guarantees.
Questions for authors
  1. In the clock and request-ID constructions, what practical constraint prevents exposing the readiness bit or complete audit readout? Since these stronger interfaces provide an exactly clean stratum, their availability and cost affect the practical relevance of the intermediate-rate regimes.

  2. Using the noise-floor transition analysis, how small must ν\nu be for provenance-aware learning to improve meaningfully over the marginal baseline at the reported horizons and budgets? What supports this noise scale in the motivating applications?

  3. The multi-action experiments show no statistically significant provenance-specific regret improvement. Can the recorded trajectories establish whether the bottleneck is conservative confidence widths, query scheduling, or pooled service orientation, and explain when improved comparison precision actually changes production decisions?

2 comments

Thank you for your careful review, for recognizing the matching upper and lower bounds, and for your helpful questions about interfaces, noise floors, and service decisions. We address these questions and the supporting evidence below.

Exact matching information and cost We recommend exposing exact matching information when available at the same cost (Section 5). The clock and request-ID constructions permit stronger readouts; partial logs are interface choices. Section 5 describes the clean-stratum benefit; Appendix E.2 shows that full audit readout weakly dominates its binary coarsenings and prices evaluation, joining, binding and audit work. We added an explicit scope statement in Section 5: the evidence is a controlled comparison of constructed interfaces; choosing partial metadata in deployment depends on its actual costs and constraints. Shadow savings are a separate resource benefit under a cap; regret plus evaluation/metadata cost is a different objective (Appendix A.2).

Noise-floor scales at the reported horizon Theorem 1 includes ν>0, and Appendix E.1 derives the transition budget. For a fixed channel and fixed N,v, a positive floor restores square-root asymptotics. We added finite-horizon comparisons in Appendix E.1: at N=v=2, γ=1, α=1/4 and T=B=2^18, production-to-paired scale ratios of 3, 5 and 10 require approximately ν≤0.036, 0.0087 and 0.0010. The numerator is the marginal scale sqrt(NT); these comparisons omit constants and logarithms.

We added E4 (Section 6.4; Appendix F.7). For α=1/4 at full budget, measured MOSS/floor-aware maximum tested-gap mean-regret ratios are 21.7/21.0/9.5/3.7 at ν=0/10-3/10-2/10^-1. The corresponding unweighted/floor-aware ratios are 5.25/4.80/2.23/1.01; these measure provenance's value within paired feedback, a different baseline. These 128-cluster results are a post-plan descriptive enlargement; the planned 32-cluster analysis reports nonmonotone small-floor ratios and a setting where fixed-scale EO exceeds unweighted regret. Small budgets can favor MOSS. These synthetic thresholds provide audit benchmarks; application-specific benefits require measuring the pipeline's paired disagreement and floor.

Confidence widths, query scheduling, and service decisions Appendices C.3/C.10 separate query-set and orientation cost. The handoff study in Section 6.2/F.5 identifies continuation as the largest service effect, with provenance-specific contrasts unresolved after correction.

We added E2 trajectory diagnostics (Section 6.4; Appendix F.7). At full budget, calibrated/rank PCE serve the same actions as pooled PCE on every path. Confidence intervals remain too wide to resolve the smallest gap, so no run certifies; earlier elimination saves calibrated PCE 124.3 shadows on average in the N=4, α=1/4 full-budget cell. Query-set cost accounts for 84.4%/77.4% of pooled pre-exhaustion regret for N=4/8. At N=4, α=1/4 and B=T/4, the only cell with changed service, four paths differ after exhaustion and a later phase completion.

We added E3 to test provenance-aware orientation, retaining each variant's own elimination bank and a common MOSS handoff. At N=4, T=2^20, α=1/4 and full budget, maximum tested-gap mean regret is 431.24/422.15 for calibrated/rank variants versus 511.36 pooled. Six of eight planned contrasts pass bootstrap-Holm; all projected contrast intervals include zero. Mean shadow use is unchanged and no run certifies. A separate pooled-bank/calibrated-orientation control isolates orientation and gives 400.42 descriptively; the eight planned tests concern complete variants.

We added a pre-specified E1 replication with 128 fresh clusters. At T=B=2^18 and α=1/4, weighted/order-only maxima are 15.41/16.41 versus 61.70 unweighted, supported by bootstrap-Holm and Student-t/Bonferroni projected contrast intervals. At α=3/4, bootstrap-Holm supports both gains while projected contrast intervals include zero.

Conditions and robustness Section 5 requires conditional mean preservation; Appendix E.2 gives a selection-induced gap reversal even with correct matching. We added Proposition 6 in Appendix E.3: under Definition 1, stationary, mean-preserving interfaces with fresh tags and an excess-moment envelope admit upper bounds: calibrated PCE uses known envelope values; ordered PCE uses its monotone order. The matching converse remains resampling-specific. Within these conditions, new Proposition 7 retains the erased-pairing rate up to constants/logarithms for PCE banks with fixed misspecified semantics. E4/E5 measure the finite-horizon sensitivity of fixed-scale EO separately. The sharper spectral rates use calibrated probabilities or a correct order; independent audits assess these inputs before transfer across model pairs.

Thank you for the substantive revisions. The E1 replication strengthens the two-action evidence, and E2/E3 clarify the decision bottlenecks and provide encouraging multi-action results. My concerns are partially addressed, although practical evidence for the required noise levels, metadata costs, and reliable provenance calibration remains limited. The multi-action gains also remain sensitive to the inference method. I acknowledge these improvements while maintaining my rating of 3 (Weak acceptance).

Reviewer_jkMNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
1
Soundness
3
Read review
Paper summary

The paper studies stochastic multi-armed bandits with limited shadow evaluations, motivated by applications such as comparing competing models on production requests. Unlike standard bandits, the learner may observe the losses of two actions on the same request, yielding potentially informative paired comparisons, or on independent requests, yielding less informative comparisons. The learner receives a provenance tag that provides imperfect information about whether the evaluations were genuinely paired.

The main contribution is a minimax characterization of the value of provenance through the posterior-contamination spectrum, H(s)=E[1/(s+Z)]H(s)=\mathbb{E}[1/(s+Z)], where ZZ represents the conditional probability of independent resampling given the observed provenance. The authors show that the behavior of this spectrum determines the attainable regret rates, including constant, intermediate, and square-root regimes. They further propose calibrated and order-only learning procedures and provide theoretical results for best-action certification, alongside experiments evaluating regret, certification, and shadow-evaluation usage.

Critical strengths

1. Interesting and practically relevant problem setting. The paper studies an important but underexplored aspect of shadow evaluation: the value of comparing models on the same request depends on the quality of the available provenance information. This is relevant to model selection and evaluation pipelines where paired evaluations are costly or imperfect.

2. A compelling theoretical characterization. The posterior-contamination spectrum provides a useful way to characterize the statistical value of provenance. The identification of distinct minimax regret regimes, including intermediate rates between constant and square-root regret, is an interesting theoretical contribution.

3. Broad theoretical development. Beyond the main minimax characterization, the paper develops calibrated and order-only learning procedures and studies best-action certification. The combination of achievable guarantees and information-theoretic lower bounds strengthens the theoretical contribution.

4. Attention to practical implementation issues. The paper considers limited shadow budgets, budget handoff, and whether provenance mechanisms preserve the intended comparison target. These considerations help connect the abstract model to real evaluation systems.

Critical weaknesses

1. The paper is very poorly written and difficult to follow. The abstract and introduction do not communicate the problem and its significance effectively. Important results are introduced with limited intuition, and the dense notation and concentration of theoretical statements make the paper unnecessarily difficult to understand. The paper would benefit substantially from restructuring and rewriting.

2. The assumptions and their implications are insufficiently explained. In particular, Definition 1 imposes a structural condition on the second moments of paired loss differences relative to the suboptimality gaps. The paper does not adequately explain how restrictive this assumption is, which practical settings satisfy it, or how the results might change when it fails.

3. The interpretation of the main theoretical results is underdeveloped. Although the spectrum provides an elegant characterization, the paper does not sufficiently explain its implications for practical provenance mechanisms. In particular, it would help to clarify when two mechanisms with similar conventional measures of quality can lead to substantially different regret rates, and what this means for designing real-world evaluation systems.

4. The empirical evidence could be better connected to the theoretical claims. The experiments cover several relevant scenarios, but the relationship between the theoretical improvements, observed regret, certification performance, and shadow-budget savings is not always clear. A more focused discussion of what the experiments validate—and what they do not establish—would strengthen the paper.

Questions for authors

1. Practical relevance of Definition 1: How restrictive is the paired-loss moment condition in Definition 1? Could the authors provide concrete examples of model-evaluation settings in which it holds, and discuss what happens when it is violated?

2. Interpretation of the spectrum: Can the authors provide a more intuitive explanation of why the behavior of H(s)=E[1/(s+Z)]H(s)=\mathbb{E}[1/(s+Z)], particularly the distribution of ZZ near zero, determines the minimax regret rate? Can two provenance mechanisms with similar average contamination or matching accuracy nevertheless yield substantially different regret rates?

3. Knowledge of the provenance mechanism: The calibrated procedure assumes knowledge of the conditional contamination probabilities, whereas the order-only procedure requires an ordering of provenance quality. How realistic are these information assumptions, and how sensitive are the guarantees to errors in calibration or ordering?

4. Practical implications of the results: Under what conditions should a practitioner expect provenance-aware evaluation to improve production regret, rather than primarily improve certification or reduce shadow-evaluation usage? Can the authors clarify which of these benefits are theoretically guaranteed and which are supported only by the experiments?

2 comments

Thank you for your careful review, for recognizing the spectrum characterization and the calibrated and order-only methods, and for your thoughtful questions. We address your presentation concerns and four questions below.

Presentation We clarified the abstract and introduction around the production/shadow decision and the value of nearly-clean comparisons. Sections 2 and 4.1 add intuition for the moment condition and spectrum; Section 4.2 distinguishes calibrated and ordered inputs, and Section 4.3 previews the theorem. We added the assumptions guide in E.3 and experiments in Section 6.4/F.7; A.1 collects the notation.

Q1: Definition 1 and its practical scope Definition 1 concerns how much candidates differ on the same request, beyond their mean losses. We added a concrete interpretation in Section 2/B.3: for binary losses, γ=1 and ν=0 require the optimum to win at least a (1+1/v)/2 fraction of disagreements, when disagreement has positive probability. Frequent, nearly balanced wins can violate this condition despite similar mean losses.

For a candidate set containing a Bayes classifier, B.3 gives (γ,v,ν)=(1,1/m₀,0) under a positive label margin m₀. We added local-disagreement examples and a Tsybakov-margin derivation with γ in (1,2), ν=0, excluding mass at η=1/2. These classification examples give sufficient conditions; applying them to a pipeline calls for auditing paired disagreements and the floor separately.

Every bounded loss law is covered with ν=1, retaining the production-only guarantee. At fixed channel and N,v, positive ν restores square-root minimax asymptotics; Appendix E.1 gives the crossover. We added E4 to test finite-horizon sensitivity; F.7 reports the planned 32-cluster analysis and labels the 128-cluster enlargement descriptive.

Q2: Why near-zero contamination matters Section 4.1 defines H(s)=E[1/(s+Z)]. We added intuition in Sections 1 and 4.1: a record with contamination z has second-moment envelope s+z. When z is comparable to or below s, it retains nearly clean precision; the frequency of such records determines the benefit as s decreases. It also limits information in complete adaptive transcripts (Section 4.3).

Equation (16) holds average contamination, tag marginal, and erased scores fixed while changing regret exponents. Corollary 3 gives equal 75% matching accuracy and mutual information yet constant versus square-root known-channel regret at full budget on its two-action family. Preserving nearly-clean distinctions therefore matters.

Q3: Supplied semantics and errors Section 4.2 takes calibrated probabilities or a correct ordering as inputs. In Section 5's constructed clock interface, rates determine calibration and wait allowances determine ordering. Appendix E.1 shows that small relative calibration errors retain constant-factor precision, while an upward additive shift can cap it.

We added Proposition 7 in E.3: in the fixed resampling model, with fresh tags and preserved means, PCE banks retain erased-pairing rates up to constants and confidence logarithms for arbitrary fixed calibration maps in [0,1] or fixed tag orders. We added E5 for fixed-scale EO and order-only service. In the planned 32-cluster sample, fourfold overestimation at α=3/4 gives 80.75 versus 72.99 unweighted. The 128-cluster enlargement is descriptive; 20% tag flips bring ordered service close to pooling (Section 6.4; F.7). The sharper rates use calibrated probabilities or a correct order; deployment calls for independent audits.

Q4: Production regret, certification, and resources Theorem 1 gives class-level minimax guarantees. At fixed N,v, Corollary 1 gives polynomial gains with zero floor, γ<2, 0<α<1, and budget above its stated threshold; ordered PCE retains the exponent. Certification has its own boundary (D.9). Service gains require better comparisons to change served actions.

Section 6.2 reports certification and shadow savings with unchanged regret; its handoff study separates unresolved provenance contrasts from the MOSS-continuation benefit (F.5).

We added E1-E3 in Section 6.4/F.7. The pre-specified E1 replication uses 128 fresh clusters at T=B=2^18: at α=1/4, weighted/order-only maximum tested-gap mean regret is 15.41/16.41 versus 61.70 unweighted, supported by bootstrap-Holm and projected contrast intervals. At α=3/4, bootstrap-Holm tests pass while projected contrast intervals include zero. E2 finds identical full-budget service under pooled orientation: confidence widths block certification, while earlier elimination occasionally saves shadows. E3 retains each variant's bank and a common handoff: 6/8 contrasts across both budgets pass bootstrap-Holm, while all projected contrast intervals include zero; the pooled-bank orientation control is descriptive.

Certification and shadow savings remain separate benefits under the resource cap.

I would like to thank the author(s) for their clarification. I maintain my positive opinion about this paper. However, the paper is still not properly written and needs a more thorough rewriting.

Reviewer_oZofOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies stochastic bandit learning over two or more candidate models. On each round, one model serves a request and at most one other model is evaluated in the background. Regret counts only production loss relative to the best fixed model; background evaluations consume a separate budget. The two evaluations may use the same request or independent requests from the same population. Afterward, a log supplies information about matching. Given numerical mismatch probabilities, the method builds confidence intervals from reliability-weighted loss differences and eliminates worse models; another version requires only a correct reliability ordering. Under an unbiased-comparison assumption and a condition relating same-request noise to model-quality gaps, the paper characterizes minimax regret through H(s)=E[1/(s+Z)]H(s)=\mathbb E[1/(s+Z)], where ZZ is the conditional mismatch probability and ss is the relevant comparison-noise scale. Upper and lower bounds match up to the stated logarithmic factors. Validation consists of proofs and synthetic experiments, including settings with four and eight models.

Critical strengths

The strongest contribution is the matching characterization of the observation model. Extra observations, loss-difference estimators, inverse-variance weighting, and confidence-based elimination are established ideas; the oracle upper bound explicitly builds on SODA (Thune and Seldin, 2018). The substantial additional result is that the same precision quantity also controls a lower bound for arbitrary adaptive policies using both scores, the complete log, free production observations, and adaptive spending. The checked lower-bound construction rules out obtaining a better polynomial rate simply by using those observations differently.

The paper explains which aspect of provenance matters. Its equal-accuracy and equal-mutual-information example gives different regret rates even with identical score laws after the records are erased. The order-only method then retains the polynomial rates without numerical calibration, although it still needs a correct quality ordering. The experiments use matched scores and budgets to isolate provenance, and some synthetic settings show supported gains in identifying the best model and saving evaluations. These are useful findings even though they are narrower than lower production regret.

Critical weaknesses

The practical significance of the observation model needs stronger justification. Under the stated assumptions, the full log is equivalent to its supplied conditional mismatch probability; the paper does not generally learn or validate these probabilities from arbitrary logs. The alternative still needs a correct supplied ordering. Mismatched evaluations must preserve the request population, and conditioning on the record must preserve the mean loss difference. Faster rates additionally require same-request squared differences to shrink as average model quality approaches a tie; similar average accuracy does not imply this. A fixed positive noise floor restores square-root asymptotic worst-case regret, although finite-sample gains may remain. Section 5 constructs valid interfaces, but does not establish when these assumptions describe an existing pipeline or why uncertain pairing is preferable to retaining exact matching information. The exposition also obscures the simple observation model and familiar elimination machinery beneath numerous variants and technical terms. The theoretical characterization supports weak acceptance, but the application case does not support a stronger recommendation.

The evidence for lower production regret is much weaker than the evidence for identifying the best model and saving evaluations. Across 1,200 tested two-action parameter combinations, only one calibrated-versus-unweighted comparison survives correction, and none does for the order-only method. No multi-model provenance-versus-pooled regret comparison survives the stated correction. Some larger gains come from changing the learner used after the budget runs out, rather than from provenance. Moreover, unused shadow budget has no value in the main regret objective: a learner can purchase additional evaluations within its cap and ignore them. Evaluation savings are therefore a separate resource benefit, not automatically an improvement in the optimized objective. Appendix E.2 discusses costs, but the main theorem does not optimize production regret plus evaluation cost. The unresolved comparisons do not prove absence of benefit; they limit the demonstrated practical gain.

Questions for authors

Clarification of the intended application: Why must matching remain uncertain when Section 5's mechanisms already track readiness or request identities? What prevents exposing the exact match indicator, and what concrete cost advantage does imperfect metadata offer? Building on Appendix E.2, please distinguish benefits under the paper's separate shadow budget from benefits under an objective that also charges for evaluations and metadata. Paid-observation objectives are established, for example in Seldin et al. (2014), Prediction with Limited Advice and Multiarmed Bandits with Paid Observations. I am asking which existing application claim is supported, not requesting a new theorem for that objective; taking observation cost to zero would still require preserving this paper's one-shadow restriction and observation law.

Clarification of the supplied information and structural assumptions: Where would the numerical mismatch probabilities, or the correct ordering of records, come from in the intended application? What existing evidence supports their validity across model pairs, preservation of the mean loss difference after conditioning on a record, and shrinking same-request differences near a tie? The fact that a record predicts matching does not itself establish the latter two properties. If support is limited to the constructed examples, please clarify that scope.

1 comment

Thank you for your careful review, for recognizing the matching minimax characterization and complete-transcript lower bound, and for your constructive questions. We address your questions and clarify the supporting evidence below.

Exact matching, application scope, and costs (Q1) The clock and request-ID constructions in Section 5 allow fuller feedback: the readiness bit exposes a clean stratum, and the full audit readout simulates either binary coarsening. Retaining a partial readout is an interface choice. We added explicit guidance in Section 5 to expose exact information when available at the same cost. Section 6.3 provides controlled audits of these interfaces; the cited systems supply implementation primitives. Whether incomplete metadata offers a deployment cost advantage depends on pipeline-specific constraints and measured evaluation, waiting, binding, and audit costs.

Appendix E.2 accounts for evaluations, joining, binding, and auditing, and compares precision at fixed work. The objective is production regret under a separate shadow cap. Saving evaluations is a separate resource benefit: unused observations could be purchased and ignored. We added this distinction in Section 5 and the connection to Seldin et al. (2014) in Appendix A.2. Optimizing regret plus evaluation/metadata cost is a different objective; a zero-cost comparison must preserve the one-shadow restriction and observation law.

Sources and scope of the structural assumptions (Q2) Sections 4.1–4.3 specify supplied information: the oracle knows the channel, calibrated PCE receives contamination probabilities, and ordered PCE receives a correct quality order. In Section 5, clock rates determine calibration, wait allowances determine ordering, and independent request-ID audits define the strata. These constructions justify semantics within their designs; transfer across model pairs requires independent validation. The model assumes a channel fixed across pairs, fresh tags after endpoint selection, and marginal-preserving independent resampling.

Matching prediction, conditional mean preservation, and shrinking paired differences are separate requirements. Section 5 requires the conditional mean contrast to remain the production gap for every pair and completed history. Section 6.3 checks 24 conditional-gap identities in the constructed audit and shows that outcome-dependent selection can reverse a gap despite correct matching. We added an assumptions guide and robustness analysis in Appendix E.3, separating comparison validity from precision.

Section 2 explains Definition 1: binary paired second moments equal disagreement probabilities, so similar average accuracy alone is insufficient. Appendix B.3 gives a sufficient example containing the Bayes classifier under a positive label margin; we added local-disagreement and Tsybakov-margin explanations. At fixed channel and N,v, a positive floor restores square-root asymptotic minimax regret (Section 5; Appendix E.1), although finite-horizon gains remain possible. New E4 tests finite-horizon floor sensitivity; its 128-cluster enlargement is descriptive, separate from the planned 32-cluster analysis. Applying these rates requires measuring the pipeline's paired moments and noise floor.

Production regret versus certification and resource savings Section 6.2 shows certification and shadow-use gains with unchanged production regret. The broad service grid has one corrected calibrated-versus-unweighted improvement and zero order-only improvements across 1,200 cells (Appendix F.1); multi-action provenance comparisons remain unresolved, and MOSS handoff explains the large partial-budget gain (F.5).

We added E1–E3 in Section 6.4 and Appendix F.7. The pre-specified E1 replication uses 128 fresh clusters at T=B=2^18. At α=1/4, maximum tested-gap mean regrets are 15.41/16.41/61.70 for weighted/order-only/unweighted service. All four planned contrasts over α=1/4,3/4 pass bootstrap–Holm; projected Student-t/Bonferroni contrast intervals exclude zero only at α=1/4. E2 records identical full-budget service actions despite occasional earlier elimination and shadow savings. E3 uses four actions and 32 clusters: six of eight planned contrasts pass bootstrap–Holm, while all projected contrast intervals include zero. Each variant retains its own elimination bank and uses the same MOSS handoff; the separate pooled-bank orientation control is descriptive. These results show how using provenance in service can change production decisions, with the stated inference limits.

We also clarified the abstract and introduction, the spectrum intuition in Section 4.1, and the policy inputs in Section 4.2.

Comments on the paper

1 comment

We thank the reviewers for their thoughtful feedback. Our minimax characterization shows why provenance tags with the same matching accuracy can yield different regret rates depending on how often they identify nearly clean comparisons. During rebuttal, we clarified when comparisons preserve the mean production loss gap and added five studies examining service decisions, noise floors, and errors in calibration or ordering. In our pre-specified two-action replication (T=B=218T=B=2^{18}, α=1/4\alpha=1/4), the maximum mean regret across tested gaps fell from 61.7061.70 under unweighted pairing to 15.4115.41 with calibrated weighting and 16.4116.41 using only the quality ordering. The multi-action studies also showed when better comparisons can save shadow evaluations without changing production decisions. We appreciate Reviewer jkMN's feedback on the presentation. We have revised the abstract and introduction and will further refine the structure and polish the exposition throughout the camera-ready version to make the assumptions and main results easier to follow.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.