acceptodds
Under review as a conference paper at ICLR 2027

When Can Old Evaluations Certify a New Model? Matching Label-Complexity Bounds Under Evaluator-Residual Drift

Abstract

Releasing a model update requires certifying that its risk on the current population stays below a threshold. Trusted labels are expensive, whereas a cheap evaluator, such as a language-model judge, scores every example. Reusing evaluator errors from earlier audits is tempting, yet it is unclear when such evidence may replace current labels. The answer depends on the status given to history. If the errors can change invisibly, no label-free test detects the change, and every valid, useful certifier must keep buying labels at a rate we characterize; if a bound on the change is assumed, label-free certification is valid at an explicit error cost. For the middle ground, where history is informative but untrusted, we propose portfolio vigilance, a sequential certifier that mixes a betting expert guided by history with one that learns only from current labels; history affects only how it bets, so validity holds for any history. The contribution is not prior-informed betting or expert mixtures themselves, but the distinction between historical evidence that may enter validity and historical evidence that may only guide evidence accumulation. In a canonical model, accurate history shortens decisions but never raises the evidence growth rate, whereas stale history can destroy it. On held-out CIFAR-10N and DICES-990 data, portfolio vigilance needs (95% confidence interval (CI) ) and () times the labels of a matched prediction-powered monitor with no observed false certification, and fewer labels on all six external blocks. When historical advice is corrupted, it stays within of its better component, whereas trusting history alone costs up to times as much. In post-confirmatory repeated-judge experiments on DICES-990 and ToxicChat, changing the rubric of a fixed LLM judge moves its scores more than its run-to-run variation. After the change, the portfolio needs () and () times the labels of the matched monitor, and fewer labels than trusting history alone.

Primary area:
learning theory
Keywords:
Deep Learning, Model update certification, learning theory, trustworthy ai, AI safety, uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_k8KSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper looks at the situations in which historical evaluator information can be safely reused in order to verify that an updated model meets a previously defined risk threshold. It is based on the assumption that trusted human labels are costly, whereas a less expensive evaluator, for example an LLM judge, can assess the evaluation population. The major problem here is that the cheap evaluator's errors may vary over time, which means that previous calibration could become unreliable.

The authors identify three scenarios: unrestricted hidden evaluator drift (M0), trusted transport assuming a bound on the changes in error from historical to current values (M1), and untrusted historical advice (M2). They derive a lower bound for the requirement of fresh labels in the case of hidden drift and determine the conditions under which historical information can be used to support certification without the need for additional labeling.

In the case of the untrusted-history setting, the authors suggest using Portfolio Vigilance, a method that involves two sequential betting procedures: one based on past evaluator errors and the other using only the current trusted labels. This approach keeps the error guarantees valid at all times even if the historical advice is wrong.

The approach has been tested using CIFAR-10N, DICES-990, some additional external datasets, synthetic corruption experiments, and changes to the LLM-judge rubric on DICES-990 and ToxicChat. The paired geometric-mean label requirements reported are 0.465 and 0.740 times the values of a PPRM-style baseline for CIFAR-10N and DICES-990, respectively. The authors also show that the portfolio is more robust than an approach that relies only on history when evaluator advice becomes stale.

Critical strengths

1. A problem of great practical importance which can be clearly expressed in statistical terms.

The paper looks at a significant issue in the evaluation of models, namely whether historical information from evaluators is still useful when there are changes in the models, the populations in question, or the evaluation methods themselves. I really value the way the three different information regimes—M0, M1, and M2—are distinguished, since this makes it clear when historical information can properly be used to provide a statistical guarantee and when it should merely be employed to guide the gathering of evidence. The impossibility result concerning hidden drift has a important practical implication: historical calibration cannot simply take the place of up-to-date, trustworthy labels when changes in evaluator error are not observable. This way of framing things is helpful for the design of reliable evaluation and model-release processes.

2. The combination of strong validity guarantees and meaningful empirical label savings.

The framework that has been proposed remains valid at any time as long as the historical advice is arbitrary and the assumptions regarding sampling and predictability are satisfied. This is an important feature since inaccurate historical information may reduce efficiency without causing the certificate to be invalidated. The empirical results are also promising. On the main datasets, the method makes use of considerably fewer trusted labels than the matched PPRM-style comparator, and the experiments involve corrupted historical advice, external datasets, and multiple evaluations by the LLM judges under changing criteria. The experiments show that it is possible to improve efficiency by combining historical guidance with a component that is independent of history. It is also a strength of the authors to have explicitly reported any comparisons that were unsuccessful or inconclusive.

Critical weaknesses

1. The methodological innovation is in fact quite limited and the comparison with other methods based on history is incomplete.

Although the way in which the problem is stated is useful, the principal algorithm makes use of some well-known elements: prediction-powered evaluation, sequential betting, anytime-valid e-processes, and fixed mixtures of experts. The paper itself acknowledges these earlier works. Theorem 4 can be obtained from standard arguments involving nonnegative supermartingales and the union bound, and the log-2 regret guarantee in Theorem 5 is a direct consequence of mixing two evidence processes in equal quantities. In a similar manner, the impossibility and transport results employ fairly standard indistinguishability and error-budget arguments.

The key innovation seems to be the application of and arrangement of these ideas around the concept of historical evaluator drift, rather than involving the introduction of a completely new certification method. Specifically, Chen and Weng (2026) have already looked at the possibility of using potentially unreliable external information to assist with anytime-valid betting. In order to show what contribution Portfolio Vigilance makes beyond a simple betting mixture based on historical information, a more direct empirical comparison with existing advice-guided or simulator-guided betting methods—adapted to the same finite-population task—would be needed. This is important since the algorithmic contribution is a major reason for recommending acceptance.

2. The experimental setup does not show the (practical) benefits that are shown for model-update certification.

The main experiments are carried out on fixed evaluation datasets rather than on actual sequences of model updates. The risk thresholds are determined by adding a predefined margin to the historical ledger risk, and all of the CIFAR-10N decisions in the primary experiment are safe. Although this approach allows for controlled comparisons, it offers only limited evidence regarding the model's performance when facing genuinely unsafe updates, changes in the deployed user population, or difficult cases near the certification boundary. The LLM experiments are more directly relevant since they alter the evaluation criteria of a fixed judge rather than assessing successive versions of the model, and these were carried out after the primary protocol had been fixed.

The baseline comparison also requires attention. Instead of being an exact implementation of the original method, the main point of comparison is a version based on the PPRM approach. Moreover, the headline figures for label reductions are derived from paired geometric-mean ratios, which are not the same as the reduction in the average number of labels bought. For instance, Table 1 shows 1,182.2 compared with 1,398.3 mean labels for DICES-990, representing a smaller relative reduction than the headline 26% geometric-mean improvement. Although the results are still promising, the practical conclusions could be strengthened by including further comparisons and by providing a more clear explanation of the total labelling costs.

Questions for authors

Question 1: The aspect of novelty in comparison with existing advice-guided betting methods (Section 5, Appendix B).

The paper admits that prior-informed betting and mixtures of e-processes are well-established techniques. However, what specific advantage does Portfolio Vigilance have when compared to simply combining a history-informed betting process with an existing anytime-valid, prediction-powered monitor? In particular, could the authors compare it with an appropriately adapted version of Chen and Weng (2026), or with another history-informed betting baseline, using the same finite-population protocol? Such a comparison would help to determine whether the empirical improvement is mainly due to the proposed formulation or instead to the known benefits of using auxiliary forecasts to select betting strategies.

Question 2: Behaviour under real-model updates and in making difficult certification decisions (Sections 6 to 7).

What happens to the performance of the proposed method when the historical and current evaluations relate to actually different versions of the model, rather than to fixed datasets or changes in the judge's rubric? Given that the main CIFAR-10N experiment includes only safe decisions, could the authors also give a more balanced evaluation involving unsafe candidates and risk thresholds near the actual risk? In particular, I should like to know about the resulting label requirements, the abstention rates, and the false-decision behaviour.

Question 3: A explanation of the headline label-efficiency gains (Tables 1 till 2, and Appendix G).

The main findings are based on paired geometric-mean label ratios, while the arithmetic mean of the label counts indicates somewhat different levels of reduction, especially in the case of DICES-990. Could the authors please explain which of the two statistics most accurately reflects the expected total annotation cost in a real-world deployment? Moreover, how sensitive are the results to using a more powerful history-informed comparator instead of the adapted PPRM-style monitor? Providing these explanations would aid in evaluating the practical importance of the improvements reported.

5 comments

We thank the reviewer for the detailed assessment of novelty and practical relevance. We address the requested comparisons and clarifications below.

W1. The components and mixture argument are standard. We revised Section 1 and Appendix B to state that directly. The paper's analysis concerns when historical residuals may enter validity, the i.i.d. vigilance lower bound and matching certifier, and the canonical identity relating advice error to evidence growth. Portfolio vigilance implements the distinction between history as evidence and history as advice in a finite-pool audit.

The reason to mix the two bettors is that we do not know which will be cheaper on the current pool. Useful history can give ledger-only a head start. Stale advice can make it slow, while the current-label bettor continues to learn. Their wealth mixture stays within log⁡2\log 2 evidence of the better component on every path. This does not say it uses fewer labels than both components on every dataset.

On stable CIFAR-10N, ledger-only uses 248.9 labels versus the portfolio's 279.9. On stable DICES-990, their means are close: 1,181.4 versus 1,182.2. Much of the primary saving over PPRM-style therefore comes from history-guided stakes. After the repeated-judge rubric change, portfolio-minus-ledger differences are -140.7 [-256.2, -46.2] on DICES-990 and -12.7 [-20.8, -5.9] on ToxicChat. The portfolio also improves over robust-only on DICES, by -109.1 [-199.3, -32.6]; that comparison is unresolved on ToxicChat. Ledger-only is itself anytime-valid under arbitrary advice. Stale advice can damage its efficiency without invalidating its current-label bets.

We added the requested advice-guided comparator on the same pools, orders, thresholds, candidate sharing, and error budgets. "Chen and Weng (adapted)" follows their Algorithm 1 structure: moment forecasts, exponential weights from past Gaussian log scores, and clipped Kelly-style stakes. We replace the i.i.d. null with the remaining-pool null and return fixed-threshold decisions. The primary replay uses three forecasts: ledger-adjusted proxy, proxy only, and neutral. The checkpoint and judge studies add a historical-loss forecast. Appendix M gives the configurations and the adaptation's validity argument.

Primary datasetPortfolio labelsAdapted comparator labelsPaired geometric ratio [95% CI]
CIFAR-10N279.87373.200.756 [0.581, 0.918]
DICES-9901,182.151,396.100.724 [0.542, 0.919]

Neither method falsely certifies in this replay. The portfolio has the one DICES false rejection already reported in Table 1; the adaptation has none. Ledger-only is cheaper than the adaptation on both primary datasets. Against the adaptation, checkpoint and judge-study intervals do not resolve a difference. This comparison supports the implementation under the stated protocol, not a claim about the authors' original software or every possible advice-guided method. The repeated-judge and corrupted-advice studies provide the evidence for the hedge when advice is stale.

W2. We added trained-model transitions, controlled unsafe updates, and a CIFAR-10N boundary replay in Appendix M. The DICES checkpoint study uses TF-IDF/SGD snapshots and re-scores the stored labeled audit set for each current snapshot. Portfolio costs are 41.53 and 42.23 labels, versus 49.67 for PPRM-style. All cases are safe, and ledger-only and robust-only are within 0.4 observed labels of the portfolio. This does not test carrying the earlier snapshot's residuals forward unchanged.

The unsafe and boundary results are:

SettingUnsafe evidence for the portfolioMean labels: portfolio / PPRM-style
20NG, far from boundarySix unsafe blocks, all 30 runs rejected7.97 / 44.93
20NG, near boundary at common thresholdTwo unsafe baseline blocks, all 10 runs rejected125.80 / 135.00
20NG, near-boundary updated subsetSeven dependent checkpoint-block cases, all 35 runs rejected103.80 / 122.00
CIFAR-10N, threshold 0.18087Two unsafe and four safe blocks2,209.17 / 2,461.17
CIFAR-10N, threshold 0.16087All six blocks unsafe1,275.80 / 2,461.17

For each 20NG study, thresholds and advice are frozen before its test split is opened. Later category pairs were nevertheless chosen after earlier results had been seen. The first two studies use baseline ledger risk plus 0.05; the third uses each checkpoint's ledger risk plus 0.02. The regressions are class-imbalanced training interventions, not deployment failures.

All seven near-boundary unsafe updated cases share one block. They are dependent cases, with risk excesses of 0.006 to 0.026. Robust-only is slightly cheaper on this subset, using 103.26 labels. Across all nine unsafe cases of that study, portfolio, robust-only, and PPRM-style use 107.16, 106.73, and 124.22 labels.

Across all 48 safe and unsafe cases in the third study, mean costs are 106.05 for portfolio, 106.42 for robust-only, 109.43 for ledger-only, 117.07 for the adaptation, and 132.33 for PPRM-style. The portfolio has the lowest observed mean, but its small difference from robust-only is descriptive. The closest unsafe case takes 116 of 122 rows. Such cases can require almost full enumeration; this count is not predicted by the i.i.d. Theorem 1.

The CIFAR replay contains 240 method runs, with no false certification, false rejection, or unresolved decision. The portfolio releases all 20 safe-block runs and rejects all 40 unsafe-block runs, using 10.2% and 48.2% fewer labels than PPRM-style at the two thresholds. Ledger-only uses 2,183.23 and 1,280.67 labels, and robust-only 2,247.73 and 1,388.97. The thresholds were selected after inspecting the final roles, so this is a sensitivity replay.

All added update/replay studies resolve correctly at full budget. Full enumeration would also resolve a finite pool, so these counts do not establish power under a small label cap. Section 7.1 reports where the original studies abstain. At 10% budget, portfolio abstention is 46.7% on CIFAR-10N and 86.7% on DICES, versus 86.7% and 99.6% for PPRM-style. At 80%, the fractions are 0% and 20.4%, versus 0% and 43.8%. An abstention is never a release. The primary DICES study includes 65 unsafe candidate decisions among 240; the primary CIFAR study has only safe cases.

Section 6 also explains the PPRM adaptation. It keeps the prediction-plus-residual estimator and CM-EB-style radius, but estimates a fixed finite-pool mean rather than running deployed risk. Pools and orders are matched; the original procedure is not reproduced exactly.

For annotation costs, we now give both measures in the abstract, Section 7.1, and Appendix H:

DatasetGeometric-mean ratio [95% CI]Ratio of total labels [95% CI]Reduction: geometric / total
CIFAR-10N0.465 [0.327, 0.575]0.487 [0.387, 0.579]53.5% / 51.3%
DICES-9900.740 [0.618, 0.877]0.845 [0.724, 0.931]26.0% / 15.5%

The total-label ratio measures spending over these blocks. DICES savings are smaller on this measure because India-locale blocks with ratios near one account for 65.9% of PPRM-style labels. We retain the prespecified geometric mean and state the total saving separately. Primary costs are unpenalized, with the known DICES false rejection reported alongside them.

Q1. Portfolio vigilance is a specific combination of a history-guided and a current-label e-process. The mixture argument is established. We use it to avoid choosing a bettor in advance, and analyze the historical-residual problem around that choice. Chen and Weng aggregate forecasts; we mix component wealths, giving the pathwise log⁡2\log 2 guarantee. We claim neither a new mixture principle nor superiority over arbitrary history-plus-monitor combinations.

The adapted comparison is reported under W1. The portfolio is cheaper than that adaptation on the primary pools, while ledger-only is cheaper than both on stable CIFAR-10N. Under stale advice, the portfolio improves over ledger-only; on repeated-judge DICES it also improves over robust-only. The stable-data gains over PPRM-style largely come from history-guided stakes.

Q2. Appendix M reports the three requested settings: trained checkpoint changes on DICES, unsafe updated classifiers on 20NG, and safe and unsafe cases in the CIFAR replay. The full results and design qualifications are under W2.

Near the boundary, correct decisions are costly. On the seven unsafe updated 20NG cases, the portfolio uses 103.80 labels, versus 122.00 for PPRM-style and 103.26 for robust-only. It buys 116 of 122 rows in the closest case. The CIFAR replay buys about 90% of the pool at the ledger-risk threshold. Early-stopping savings are therefore limited in these cases.

There are no false or unresolved decisions at full budget in the added studies. The seven 20NG cases share one block, CIFAR thresholds are post hoc, and regressions are controlled. We report these limits with the results. Section 7.1's budget frontiers show the abstentions that full-budget counts cannot reveal.

Q3. For spending over the tested block mix, use the ratio of total labels: 0.487 [0.387, 0.579] on CIFAR-10N and 0.845 [0.724, 0.931] on DICES-990, or 51.3% and 15.5% savings. The geometric mean instead describes multiplicative improvement with equal weight per block. We now report both. Costs under a different deployment mix or unequal label prices would require the corresponding weights.

The component comparison changes how we interpret the saving. Ledger-only is cheapest on stable CIFAR-10N and close to the portfolio on stable DICES. After the rubric change, the portfolio uses 140.7 fewer labels on DICES and 12.7 fewer on ToxicChat, with intervals excluding zero. It improves over robust-only on DICES, with an unresolved comparison on ToxicChat. The adapted Chen and Weng comparison favors the portfolio on the primary pools and is unresolved on the checkpoint/judge studies. The gain over PPRM-style on stable data largely reflects useful history; the additional case for the portfolio is its behavior when that advice is stale. W1 gives the matched comparison and its limits.

Reviewer_RDLJOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies when historical audit evidence can be reused to certify current risk under evaluator drift. It distinguishes M0, where hidden residual drift prevents validating transport from cheap observations alone; M1, where a trusted transport bound permits reuse at an explicit error cost; and M2, where history serves as advice without being assumed valid evidence. Theorem 1 gives a fresh-label lower bound for procedures satisfying both honesty and liveness in a Bernoulli submodel. For M2, an equal-weight mixture of a history-guided ledger expert and an online robust expert is anytime-valid under the stated sampling and betting assumptions, even when historical advice is wrong. Its log evidence is within log 2 of the better component. In the canonical model, useful history can provide a finite head start without improving the robust expert's asymptotic evidence-growth rate, while stale advice can impair the ledger expert. Prespecified CIFAR-10N and DICES-990 audits report label-cost ratios of 0.465 and 0.740 relative to a PPRM-style comparator, with no observed false certifications. Additional studies examine external blocks, corrupted advice, non-LLM evaluator replacement, and repeated LLM judging. In the latter study, the portfolio improves over both components on DICES; on ToxicChat it improves over ledger-only and is statistically tied with robust-only.

Critical strengths
  1. A precise distinction between trusted evidence and untrusted advice. Proposition 2 explains why cheap observations alone cannot establish transport, while the M2 guarantee shows how history can still guide efficient evidence collection without being trusted for validity. The lower bound and canonical analysis help clarify the cost of obtaining useful certification under these information constraints. This is relevant to audit design when evaluator behavior may change.
  2. The reporting distinguishes confirmatory results from exploratory evidence. The primary protocol is prespecified, later analyses are labeled, uncertainty estimates accompany empirical comparisons, and the reduction from 50 planned random orders to 5 in the repeated-judge study is disclosed. The ToxicChat tie is reported rather than treated as a universal win. The exploratory Folktables false-certification result is also stated in the main text (§4), with its detailed design in Appendix L.
Critical weaknesses
  1. The empirical evidence covers a narrower set of deployment changes than the motivation. The LLM study changes the rubric of a fixed judge model. Appendix J also studies a non-LLM evaluator replacement, from a unigram TF-IDF logistic model to a larger unigram/bigram model, but this does not establish behavior under an LLM judge-model upgrade. The primary DICES experiment uses a TF-IDF proxy, while CIFAR-10N uses class and annotation-related side information; neither is a direct end-to-end study of a deployed LLM update audited through a changing LLM judge. These limits concern empirical applicability rather than the conditional validity theorem. Similarly, the up-to-75% Folktables false-certification result is an exploratory stress test with injected hidden failures at a post hoc margin of 0.05; it should not be read as a natural deployment failure rate. At the prespecified margin of 0.02, those bridge methods have zero observed false certifications.
  2. The practical boundary for choosing trusted transport needs more guidance. M2 remains available when no defensible bridge exists and does not require verifying that historical advice is correct. The unresolved operational issue is when a team has sufficient external justification to use M1 and claim its zero-label savings: which assumptions must be documented, how the transport/error budget should be justified, and what current-label checks could assess those assumptions. A worked example would make the distinction easier to use without suggesting that cheap diagnostics can overcome Proposition 2.
Questions for authors
  1. Can you provide a practical example of documenting an M1 transport assumption and its error budget, together with a fallback to M2 when that justification is unavailable?
  2. Do you have evidence from an LLM judge-model upgrade, extending the existing fixed-model rubric study and non-LLM evaluator replacement?
  3. Theorem 1 uses an i.i.d. Bernoulli submodel, whereas the procedure audits a finite pool without replacement. What lower bound or approximation is appropriate for interpreting finite-pool sample efficiency?
  4. Section 7.1 attributes the ledger-only advantage on the primary datasets to useful historical advice and the portfolio's hedge cost. Beyond the canonical bound, can you characterize the finite-sample distribution of that additional label cost and when it is practically negligible?
  5. What size of portfolio-versus-robust difference can the repeated-judge design resolve on ToxicChat? A precision or power analysis would help interpret the reported tie.
3 comments

We thank the reviewer for the assessment and questions about the theory and evaluation. We address the weaknesses and questions below.

W1. Appendix M now includes trained predictor checkpoints, a replacement of the LLM judge model, and controlled unsafe updates on 20 Newsgroups. The DICES checkpoint and judge-replacement studies use previously inspected roles and all-safe blocks. The 20 Newsgroups studies test rejection after induced regressions. These additions extend the evidence beyond a rubric change, but none tests an end-to-end deployed LLM update. Section 8 states that limit.

We also revised the Folktables passage in Section 4. At the prespecified margin of 0.02, the bridge certifies nothing, and no bridge-based or fresh-label method false-certifies. The proxy-only control still falsely certifies every injected block. The 75% result uses injected failures at the exploratory margin of 0.05. It shows what can happen when an invalid bridge is trusted, rather than estimating how often this happens in deployment. The distinct-judge results are under Q2.

W2. We added the documentation rule to Appendix F and a pointer after Theorem 3. Before inspecting target labels, a team must record the historical upper bound UH(k)U_H(k) with error δH\delta_H, the bridge radius ρk\rho_k with its justification and error δT\delta_T, and the certificate check Q‾tar(k)+UH(k)+ρk≤τk\overline Q_{\mathrm{tar}}(k)+U_H(k)+\rho_k\le\tau_k. The certificate's error is at most δH+δT\delta_H+\delta_T, without an independence assumption, under the simultaneous guarantees required by the theorem.

A small observed score change does not supply the bridge justification. Appendix F now gives a conditional example in which an unchanged pool and trusted-loss vector justify it after an evaluator-only replacement; we work through it under Q1. The retrospective radii in our experiments serve a different purpose: they show what a bridge would need to cover, using target labels to check it.

If no radius can be justified, the team uses M2 and treats the ledger as advice. Vigil-Cell in Appendix L illustrates how current labels can assess a proposed bridge, but those checks spend the labels that zero-label transport was intended to save.

Q1. Consider an evaluator replacement on the same finite pool, with the candidate, trusted losses, and item weights unchanged. Write Q‾H\overline Q_H and Q‾T\overline Q_T for the old and new all-pool cheap means. Then μET−μEH=Q‾H−Q‾T\mu_E^T-\mu_E^H=\overline Q_H-\overline Q_T. The radius ρ=[Q‾H−Q‾T]+\rho=[\overline Q_H-\overline Q_T]_+ is therefore a deterministic bridge, with δT=0\delta_T=0. An earlier residual bound UHU_H at error δH=δ\delta_H=\delta permits zero-fresh-label certification when Q‾T+UH+ρ≤τ\overline Q_T+U_H+\rho\le\tau.

The justification is the unchanged trusted-loss vector and weights. It must be documented; it cannot be inferred from score stability. This is a conditional application of Theorem 3, not an empirical finding from the model-update experiments. If the candidate, population, or trusted-label process changes, this argument no longer supplies the bridge. Without another justification, the fallback is M2.

The existing data show why that distinction matters. The repeated-judge diagnostic uses an optimistic plug-in rule, replacing UHU_H by the historical residual mean. Its fractions are not validated certificate rates. On DICES-990, radii of 0.12 and 0.10 first cover every observed cell under rubrics A and B, yet leave only 0% and 2% zero-label-certifiable. On Folktables, the calibrated width 0.048 exceeds the prespecified margin 0.02. On ToxicChat, retrospective radii of 0.05 and 0.02 leave 62.5% and 100% certifiable, but were checked with target labels. Appendix F now separates these diagnostics from a bridge justified before target-label inspection.

Q2. Yes. We replaced gpt-5.6-luna with gpt-6.1-terra, keeping the Judge-B rubric, output schema, and conversation identifiers fixed. All 990 requests returned schema-valid judgments identified as the new model. The audit has two checkpoint transitions, six final blocks per transition, five seeds, and five methods: 300 method runs.

Mean labelsPortfolio, refreshed ledgerPortfolio, stale ledgerRobust-onlyAdapted Chen and WengPPRM-style
Baseline→updated41.3042.0341.7343.0749.67
Updated→induced regression42.0342.8042.5043.2749.67

All 12 transition-block cases are safe, and every method resolves them correctly. The stale portfolio/PPRM-style ratios are 0.821 [0.698, 0.949] and 0.838 [0.706, 0.969]. Against robust-only, the ratios are 1.010 [0.987, 1.032] and 1.010 [0.989, 1.032]; against the adapted Chen and Weng method, 0.975 [0.932, 1.013] and 0.990 [0.947, 1.016]. The intervals do not resolve a difference against either betting comparator.

This adds a distinct-judge test with correct releases and lower label costs than PPRM-style. It uses a TF-IDF predictor on inspected DICES roles, so it does not test an LLM predictor or an unsafe deployed update. Bootstrap units are the six blocks per transition. The seeds are nested orders, and the transitions reuse items rather than providing independent replications. Intervals are not multiplicity-adjusted. Appendix M reports the study and the stale-versus-refreshed comparison.

Q3. Theorem 1 gives an i.i.d. Bernoulli bound, and the original submission explicitly distinguishes it from sampling a finite pool without replacement. We now add a direct finite-pool bound in Appendix F, rather than use a variance correction to reinterpret the i.i.d. stopping-time result.

Consider two worlds with fixed safe and unsafe loss counts, uniformly hidden among NN rows and independent of identical cheap information. An auditor can adaptively choose unqueried rows and stop early. Let CC denote certification, with Pu(C)≤δP_u(C)\le\delta and Ps(C)≥1−βP_s(C)\ge1-\beta. At prefix length mm, the loss count is hypergeometric. If πm(δ)\pi_m(\delta) is the largest safe-world certification probability of a level-δ\delta test of that count, then

EsT≥∑m=0N−1[1−β−πm(δ)]+.\mathbb E_s T\ge\sum_{m=0}^{N-1}[1-\beta-\pi_m(\delta)]_+.

The proof bounds certification by each prefix and sums the probability of continuing. It permits adaptive stopping and pool exhaustion. A joint-loss-vector extension counts each purchased row once when it reveals several candidate outcomes. The bound averages over the hidden assignments; it does not impose the same cost on every fixed safe pool.

For the five existing N=5,000N=5{,}000, δ=.05\delta=.05, β=.10\beta=.10 scaling settings, the finite-pool bounds are 141.76, 53.50, 24.55, 14.23, and 6.75 rows. The corresponding saved Portfolio means are 1,145.93, 468.01, 220.14, 121.89, and 54.08. Appendix F reports these calculations using the frozen runs, without new experiments.

This supplies a finite-pool information bound, but not a matching efficiency characterization. The summed prefix bound can be loose. Its two-world test knows the exact counts; Portfolio must control false certification for all unsafe risks above the threshold. The scalar comparison also assumes uninformative cheap features and one candidate. We therefore retain the i.i.d. reference as a separate comparison and make no finite-pool near-optimality claim.

Q4. Section 7.1 and Appendix H now report the extra labels per primary block-seed pair, measured as portfolio minus ledger-only:

DatasetMeanMedianInterquartile rangeRangePortfolio no costlier
CIFAR-10N31.017[9, 37][-165, 275]6.7%
DICES-9900.81[0, 9][-175, 168]48.3%

On DICES-990 the average cost is close to one label. On CIFAR-10N it is larger. In the corrupted-advice grid, the portfolio/cheaper-component label ratio ranges from 0.912 to 1.080.

We also compare against the cheaper component selected separately on each repeated-judge run. This is a hindsight benchmark; a deployed method cannot make that choice in advance. Wrong or unresolved decisions are charged the whole block. On ToxicChat A→B, the portfolio costs 5.90 extra labels on average [2.475, 10.000], with median 2, interquartile range [1, 4.25], 90th percentile 13.8, and 95th percentile 23.25. The mean per-run cost ratio is 1.14. On DICES-990 the difference is -23.13 [-64.77, 15.18], with unresolved sign.

The general guarantee remains on evidence: Tmix(A)≤min⁡{TL(2A),TR(2A)}T_{\mathrm{mix}}(A)\le\min\{T_L(2A),T_R(2A)\} pathwise. Its label cost depends on how long a component takes to double its evidence. In the canonical model's positive-growth region, the displayed expectation upper bounds differ by log⁡2/GL(η)\log 2/G_L(\eta). That comparison of bounds does not establish the same bound on the difference of actual expectations. We have no universal constant label-overhead guarantee or general condition under which the overhead is negligible.

Q5. The post-confirmatory precision analysis gives a portfolio-minus-robust difference of +0.05 labels on ToxicChat A→B, with 95% block-bootstrap interval [-4.60, 4.10]. It averages the five orders within each block, then resamples the eight block means using 10,000 replicates. The manuscript now uses this recorded interval consistently.

The interval permits a few-label saving or cost. A normal-approximation calculation gives half-width 4.62 labels, from 1.96(6.67)/81.96(6.67)/\sqrt{8}. This describes precision; it is neither a minimum detectable effect nor a formal power analysis. We cannot claim superiority or equivalence. Equivalence would require a prespecified practical margin and an appropriate analysis. More random orders would not provide more independent evaluation units.

Reviewer_Qz9ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper studies when historical evaluation results can reduce the number of new human labels needed to certify that a model’s current risk is below a threshold. It distinguishes three settings: (1) unrestricted evaluator drift, where fresh labels are necessary; (2) an assumed bound on historical-to-current drift, which can permit certification without new labels; (3) and informative but untrusted history, which can guide evaluation without determining its validity. For the third setting, the authors propose portfolio vigilance, a sequential test combining a history-guided betting strategy with one that learns from current labels. They establish false-certification control under uniform sampling from a fixed pool and show that the mixture loses at most a constant amount of log-evidence relative to its better component. Experiments on CIFAR-10N, DICES, and ToxicChat report reduced labeling costs, including under corrupted historical advice and changes to an LLM judge’s rubric.

Critical strengths
  1. I think the paper investigates an important and practical question.
  2. The central theoretical arguments appear sound under the stated assumptions.
Critical weaknesses
  1. The writing and clarification should be largely improved. For example, the line "under M0: ≥kl..\geq kl.. ... fresh labels per coordinate" and "Declared status of history" are quite confusing terminologies. And "Prediction-powered inference, e-values, betting confidence sequences, prior-informed betting, and expert mixtures all predate this work; the contribution is the information-model distinction, its consequences, and a certifier that makes it operational.";"The failure is not only theoretical: in an exploratory Folktables stress test with hidden injected failures, a certifier that trusts a calibrated bridge without verifying it falsecertifies up to 75% of the unsafe blocks, exactly as often as a no-label oracle, while fresh-label baselines catch every on" are suspiciously ai-generated sentences.
  2. For the evaluator-only shift experiments, can historical examples be re-scored using the current evaluator? If so, refreshed historical residuals seem a practical alternative and should be included explicitly in the interpretation of the gains from hedging against stale advice.
  3. The experiments support label-efficient auditing and robustness to controlled evaluator changes. However, they do not directly evaluate the motivating scenario of certifying successive model updates using historical audits. An experiment across model checkpoints, where evaluator errors may change as model outputs change, would strengthen the practical evidence.
Questions for authors

See weaknesses.

3 comments

We thank the reviewer for the comments on clarity, refreshed history, and model-update evaluation. We address each point below.

W1. We rewrote the main text and appendices, retaining the section order, algorithms, citations, and core results. The introduction now defines the ledger and residual correction before the information models. Section 4 identifies the i.i.d. Bernoulli model used for the lower bound, so it is not confused with the finite-pool procedure. Section 5 explains when bets are chosen, how candidates share a purchased label, how the first bet is initialized, and why an evidence guarantee does not imply the same guarantee in label units. We split the longer results paragraphs into the comparison being made, its uncertainty, and its interpretation.

We also redrew Figure 1. It now says "Information Model" and "Declared information model (M0, M1, or M2)". The caption defines a monitored coordinate and Bernoulli relative entropy, and the Stage II tags point to Theorems 4 and 5. Other captions define their decision and cost units. We rewrote both passages you quoted: the contribution statement now identifies the inherited tools and states the paper's contribution directly; the Folktables passage identifies 0.02 as the prespecified margin and 0.05 as the exploratory margin. The abstract now opens with the certifier.

W2. Yes. The B→B condition re-scores the historical ledger with the current rubric. We had described it as a stationary control; Section 7.3 and Appendix K now discuss it as the refreshed-history alternative.

With refreshed history, ledger-only uses 1,274.7 labels on DICES-990 and 130.6 on ToxicChat. The portfolio uses 1,214.3 and 130.7. All are below the stale A→B portfolio's 1,305.2 and 137.6 labels. These are comparisons of condition means, not a paired inferential contrast.

The judge-replacement study provides a paired comparison on the same pools and orders. Refreshed ledger-only uses 41.93 and 42.70 labels; the stale portfolio uses 42.03 and 42.80. In each transition, refreshed ledger-only minus stale portfolio is -0.10 labels, with 95% block-bootstrap interval [-1.77, 1.33]. The observed costs are close, but this does not establish equivalence. There are no wrong or unresolved decisions in these runs. Refreshing also reduces portfolio costs to 41.30 and 42.03; the stale/refreshed ratios are 1.021 [1.000, 1.057] and 1.022 [1.001, 1.057].

These results favor refreshing when the old inputs can be re-scored. The portfolio can use that refreshed advice, or retain stored advice when refreshing is unavailable. In either case it buys current labels to establish the certificate. Re-scoring the ledger alone does not justify transport to a changed population or label process. The post hoc mean absolute ledger-to-target residual gaps are 0.03265 with stale scores and 0.03406 with refreshed scores; these diagnostics use target labels.

W3. We added a checkpoint study on DICES-990 using three trained TF-IDF/SGD snapshots: a baseline, an ordinary update, and a regression induced by class-imbalanced training. Both weights and predictions change. We audit each later snapshot on six hash blocks of 41 to 55 final-role conversations, with the threshold set to the source snapshot's ledger risk plus 0.05.

The portfolio resolves all 30 runs per transition correctly, 60 across both transitions. It uses 41.53 and 42.23 labels, versus 49.67 for PPRM-style. The paired geometric ratios are 0.812 [0.686, 0.946] and 0.827 [0.698, 0.963]. Ledger-only uses 41.20 and 41.90 labels, and robust-only 41.23 and 41.93. The portfolio's observed extra cost is below 0.4 labels, with intervals that do not resolve a difference. Its comparisons with the adapted Chen and Weng method are also unresolved: 0.964 [0.877, 1.033] and 1.002 [0.958, 1.044].

The historical resource reused here is a labeled audit set. We recompute its losses and proxy scores for the snapshot under audit. This tests audit-set reuse across checkpoints; it does not test carrying the previous snapshot's residuals forward unchanged. The final-role items had been inspected in earlier studies, so the experiment is additional evidence, not a fresh confirmatory test.

All 12 transition-block cases are safe, and the portfolio buys about 83% and 84% of these small pools. The study shows correct release after a trained-model update and modest savings over PPRM-style. The controlled 20 Newsgroups studies test rejection of unsafe updates separately; we discuss them under Reviewer k8KS, W2. Appendix M and the Limitations state these distinctions.

I appreciate the authors’ detailed response. I will maintain my score, as I believe the writing and clarity of the presentation still need improvement.

Thank you for taking the time to consider our response. Could you please let us know which parts of the revised manuscript would benefit most from further clarification? That would help us focus our next round of revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.