- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper looks at the situations in which historical evaluator information can be safely reused in order to verify that an updated model meets a previously defined risk threshold. It is based on the assumption that trusted human labels are costly, whereas a less expensive evaluator, for example an LLM judge, can assess the evaluation population. The major problem here is that the cheap evaluator's errors may vary over time, which means that previous calibration could become unreliable.
The authors identify three scenarios: unrestricted hidden evaluator drift (M0), trusted transport assuming a bound on the changes in error from historical to current values (M1), and untrusted historical advice (M2). They derive a lower bound for the requirement of fresh labels in the case of hidden drift and determine the conditions under which historical information can be used to support certification without the need for additional labeling.
In the case of the untrusted-history setting, the authors suggest using Portfolio Vigilance, a method that involves two sequential betting procedures: one based on past evaluator errors and the other using only the current trusted labels. This approach keeps the error guarantees valid at all times even if the historical advice is wrong.
The approach has been tested using CIFAR-10N, DICES-990, some additional external datasets, synthetic corruption experiments, and changes to the LLM-judge rubric on DICES-990 and ToxicChat. The paired geometric-mean label requirements reported are 0.465 and 0.740 times the values of a PPRM-style baseline for CIFAR-10N and DICES-990, respectively. The authors also show that the portfolio is more robust than an approach that relies only on history when evaluator advice becomes stale.
Critical strengths
1. A problem of great practical importance which can be clearly expressed in statistical terms.
The paper looks at a significant issue in the evaluation of models, namely whether historical information from evaluators is still useful when there are changes in the models, the populations in question, or the evaluation methods themselves. I really value the way the three different information regimes—M0, M1, and M2—are distinguished, since this makes it clear when historical information can properly be used to provide a statistical guarantee and when it should merely be employed to guide the gathering of evidence. The impossibility result concerning hidden drift has a important practical implication: historical calibration cannot simply take the place of up-to-date, trustworthy labels when changes in evaluator error are not observable. This way of framing things is helpful for the design of reliable evaluation and model-release processes.
2. The combination of strong validity guarantees and meaningful empirical label savings.
The framework that has been proposed remains valid at any time as long as the historical advice is arbitrary and the assumptions regarding sampling and predictability are satisfied. This is an important feature since inaccurate historical information may reduce efficiency without causing the certificate to be invalidated. The empirical results are also promising. On the main datasets, the method makes use of considerably fewer trusted labels than the matched PPRM-style comparator, and the experiments involve corrupted historical advice, external datasets, and multiple evaluations by the LLM judges under changing criteria. The experiments show that it is possible to improve efficiency by combining historical guidance with a component that is independent of history. It is also a strength of the authors to have explicitly reported any comparisons that were unsuccessful or inconclusive.
Critical weaknesses
1. The methodological innovation is in fact quite limited and the comparison with other methods based on history is incomplete.
Although the way in which the problem is stated is useful, the principal algorithm makes use of some well-known elements: prediction-powered evaluation, sequential betting, anytime-valid e-processes, and fixed mixtures of experts. The paper itself acknowledges these earlier works. Theorem 4 can be obtained from standard arguments involving nonnegative supermartingales and the union bound, and the log-2 regret guarantee in Theorem 5 is a direct consequence of mixing two evidence processes in equal quantities. In a similar manner, the impossibility and transport results employ fairly standard indistinguishability and error-budget arguments.
The key innovation seems to be the application of and arrangement of these ideas around the concept of historical evaluator drift, rather than involving the introduction of a completely new certification method. Specifically, Chen and Weng (2026) have already looked at the possibility of using potentially unreliable external information to assist with anytime-valid betting. In order to show what contribution Portfolio Vigilance makes beyond a simple betting mixture based on historical information, a more direct empirical comparison with existing advice-guided or simulator-guided betting methods—adapted to the same finite-population task—would be needed. This is important since the algorithmic contribution is a major reason for recommending acceptance.
2. The experimental setup does not show the (practical) benefits that are shown for model-update certification.
The main experiments are carried out on fixed evaluation datasets rather than on actual sequences of model updates. The risk thresholds are determined by adding a predefined margin to the historical ledger risk, and all of the CIFAR-10N decisions in the primary experiment are safe. Although this approach allows for controlled comparisons, it offers only limited evidence regarding the model's performance when facing genuinely unsafe updates, changes in the deployed user population, or difficult cases near the certification boundary. The LLM experiments are more directly relevant since they alter the evaluation criteria of a fixed judge rather than assessing successive versions of the model, and these were carried out after the primary protocol had been fixed.
The baseline comparison also requires attention. Instead of being an exact implementation of the original method, the main point of comparison is a version based on the PPRM approach. Moreover, the headline figures for label reductions are derived from paired geometric-mean ratios, which are not the same as the reduction in the average number of labels bought. For instance, Table 1 shows 1,182.2 compared with 1,398.3 mean labels for DICES-990, representing a smaller relative reduction than the headline 26% geometric-mean improvement. Although the results are still promising, the practical conclusions could be strengthened by including further comparisons and by providing a more clear explanation of the total labelling costs.
Questions for authors
Question 1: The aspect of novelty in comparison with existing advice-guided betting methods (Section 5, Appendix B).
The paper admits that prior-informed betting and mixtures of e-processes are well-established techniques. However, what specific advantage does Portfolio Vigilance have when compared to simply combining a history-informed betting process with an existing anytime-valid, prediction-powered monitor? In particular, could the authors compare it with an appropriately adapted version of Chen and Weng (2026), or with another history-informed betting baseline, using the same finite-population protocol? Such a comparison would help to determine whether the empirical improvement is mainly due to the proposed formulation or instead to the known benefits of using auxiliary forecasts to select betting strategies.
Question 2: Behaviour under real-model updates and in making difficult certification decisions (Sections 6 to 7).
What happens to the performance of the proposed method when the historical and current evaluations relate to actually different versions of the model, rather than to fixed datasets or changes in the judge's rubric? Given that the main CIFAR-10N experiment includes only safe decisions, could the authors also give a more balanced evaluation involving unsafe candidates and risk thresholds near the actual risk? In particular, I should like to know about the resulting label requirements, the abstention rates, and the false-decision behaviour.
Question 3: A explanation of the headline label-efficiency gains (Tables 1 till 2, and Appendix G).
The main findings are based on paired geometric-mean label ratios, while the arithmetic mean of the label counts indicates somewhat different levels of reduction, especially in the case of DICES-990. Could the authors please explain which of the two statistics most accurately reflects the expected total annotation cost in a real-world deployment? Moreover, how sensitive are the results to using a more powerful history-informed comparator instead of the adapted PPRM-style monitor? Providing these explanations would aid in evaluating the practical importance of the improvements reported.
5 comments
W2. We added trained-model transitions, controlled unsafe updates, and a CIFAR-10N boundary replay in Appendix M. The DICES checkpoint study uses TF-IDF/SGD snapshots and re-scores the stored labeled audit set for each current snapshot. Portfolio costs are 41.53 and 42.23 labels, versus 49.67 for PPRM-style. All cases are safe, and ledger-only and robust-only are within 0.4 observed labels of the portfolio. This does not test carrying the earlier snapshot's residuals forward unchanged.
The unsafe and boundary results are:
| Setting | Unsafe evidence for the portfolio | Mean labels: portfolio / PPRM-style |
|---|---|---|
| 20NG, far from boundary | Six unsafe blocks, all 30 runs rejected | 7.97 / 44.93 |
| 20NG, near boundary at common threshold | Two unsafe baseline blocks, all 10 runs rejected | 125.80 / 135.00 |
| 20NG, near-boundary updated subset | Seven dependent checkpoint-block cases, all 35 runs rejected | 103.80 / 122.00 |
| CIFAR-10N, threshold 0.18087 | Two unsafe and four safe blocks | 2,209.17 / 2,461.17 |
| CIFAR-10N, threshold 0.16087 | All six blocks unsafe | 1,275.80 / 2,461.17 |
For each 20NG study, thresholds and advice are frozen before its test split is opened. Later category pairs were nevertheless chosen after earlier results had been seen. The first two studies use baseline ledger risk plus 0.05; the third uses each checkpoint's ledger risk plus 0.02. The regressions are class-imbalanced training interventions, not deployment failures.
All seven near-boundary unsafe updated cases share one block. They are dependent cases, with risk excesses of 0.006 to 0.026. Robust-only is slightly cheaper on this subset, using 103.26 labels. Across all nine unsafe cases of that study, portfolio, robust-only, and PPRM-style use 107.16, 106.73, and 124.22 labels.
Across all 48 safe and unsafe cases in the third study, mean costs are 106.05 for portfolio, 106.42 for robust-only, 109.43 for ledger-only, 117.07 for the adaptation, and 132.33 for PPRM-style. The portfolio has the lowest observed mean, but its small difference from robust-only is descriptive. The closest unsafe case takes 116 of 122 rows. Such cases can require almost full enumeration; this count is not predicted by the i.i.d. Theorem 1.
The CIFAR replay contains 240 method runs, with no false certification, false rejection, or unresolved decision. The portfolio releases all 20 safe-block runs and rejects all 40 unsafe-block runs, using 10.2% and 48.2% fewer labels than PPRM-style at the two thresholds. Ledger-only uses 2,183.23 and 1,280.67 labels, and robust-only 2,247.73 and 1,388.97. The thresholds were selected after inspecting the final roles, so this is a sensitivity replay.
All added update/replay studies resolve correctly at full budget. Full enumeration would also resolve a finite pool, so these counts do not establish power under a small label cap. Section 7.1 reports where the original studies abstain. At 10% budget, portfolio abstention is 46.7% on CIFAR-10N and 86.7% on DICES, versus 86.7% and 99.6% for PPRM-style. At 80%, the fractions are 0% and 20.4%, versus 0% and 43.8%. An abstention is never a release. The primary DICES study includes 65 unsafe candidate decisions among 240; the primary CIFAR study has only safe cases.
Section 6 also explains the PPRM adaptation. It keeps the prediction-plus-residual estimator and CM-EB-style radius, but estimates a fixed finite-pool mean rather than running deployed risk. Pools and orders are matched; the original procedure is not reproduced exactly.
For annotation costs, we now give both measures in the abstract, Section 7.1, and Appendix H:
| Dataset | Geometric-mean ratio [95% CI] | Ratio of total labels [95% CI] | Reduction: geometric / total |
|---|---|---|---|
| CIFAR-10N | 0.465 [0.327, 0.575] | 0.487 [0.387, 0.579] | 53.5% / 51.3% |
| DICES-990 | 0.740 [0.618, 0.877] | 0.845 [0.724, 0.931] | 26.0% / 15.5% |
The total-label ratio measures spending over these blocks. DICES savings are smaller on this measure because India-locale blocks with ratios near one account for 65.9% of PPRM-style labels. We retain the prespecified geometric mean and state the total saving separately. Primary costs are unpenalized, with the known DICES false rejection reported alongside them.
Q1. Portfolio vigilance is a specific combination of a history-guided and a current-label e-process. The mixture argument is established. We use it to avoid choosing a bettor in advance, and analyze the historical-residual problem around that choice. Chen and Weng aggregate forecasts; we mix component wealths, giving the pathwise guarantee. We claim neither a new mixture principle nor superiority over arbitrary history-plus-monitor combinations.
The adapted comparison is reported under W1. The portfolio is cheaper than that adaptation on the primary pools, while ledger-only is cheaper than both on stable CIFAR-10N. Under stale advice, the portfolio improves over ledger-only; on repeated-judge DICES it also improves over robust-only. The stable-data gains over PPRM-style largely come from history-guided stakes.
Q2. Appendix M reports the three requested settings: trained checkpoint changes on DICES, unsafe updated classifiers on 20NG, and safe and unsafe cases in the CIFAR replay. The full results and design qualifications are under W2.
Near the boundary, correct decisions are costly. On the seven unsafe updated 20NG cases, the portfolio uses 103.80 labels, versus 122.00 for PPRM-style and 103.26 for robust-only. It buys 116 of 122 rows in the closest case. The CIFAR replay buys about 90% of the pool at the ledger-risk threshold. Early-stopping savings are therefore limited in these cases.
There are no false or unresolved decisions at full budget in the added studies. The seven 20NG cases share one block, CIFAR thresholds are post hoc, and regressions are controlled. We report these limits with the results. Section 7.1's budget frontiers show the abstentions that full-budget counts cannot reveal.
Q3. For spending over the tested block mix, use the ratio of total labels: 0.487 [0.387, 0.579] on CIFAR-10N and 0.845 [0.724, 0.931] on DICES-990, or 51.3% and 15.5% savings. The geometric mean instead describes multiplicative improvement with equal weight per block. We now report both. Costs under a different deployment mix or unequal label prices would require the corresponding weights.
The component comparison changes how we interpret the saving. Ledger-only is cheapest on stable CIFAR-10N and close to the portfolio on stable DICES. After the rubric change, the portfolio uses 140.7 fewer labels on DICES and 12.7 fewer on ToxicChat, with intervals excluding zero. It improves over robust-only on DICES, with an unresolved comparison on ToxicChat. The adapted Chen and Weng comparison favors the portfolio on the primary pools and is unresolved on the checkpoint/judge studies. The gain over PPRM-style on stable data largely reflects useful history; the additional case for the portfolio is its behavior when that advice is stale. W1 gives the matched comparison and its limits.
We thank the reviewer for the detailed assessment of novelty and practical relevance. We address the requested comparisons and clarifications below.
W1. The components and mixture argument are standard. We revised Section 1 and Appendix B to state that directly. The paper's analysis concerns when historical residuals may enter validity, the i.i.d. vigilance lower bound and matching certifier, and the canonical identity relating advice error to evidence growth. Portfolio vigilance implements the distinction between history as evidence and history as advice in a finite-pool audit.
The reason to mix the two bettors is that we do not know which will be cheaper on the current pool. Useful history can give ledger-only a head start. Stale advice can make it slow, while the current-label bettor continues to learn. Their wealth mixture stays within log2 evidence of the better component on every path. This does not say it uses fewer labels than both components on every dataset.
On stable CIFAR-10N, ledger-only uses 248.9 labels versus the portfolio's 279.9. On stable DICES-990, their means are close: 1,181.4 versus 1,182.2. Much of the primary saving over PPRM-style therefore comes from history-guided stakes. After the repeated-judge rubric change, portfolio-minus-ledger differences are -140.7 [-256.2, -46.2] on DICES-990 and -12.7 [-20.8, -5.9] on ToxicChat. The portfolio also improves over robust-only on DICES, by -109.1 [-199.3, -32.6]; that comparison is unresolved on ToxicChat. Ledger-only is itself anytime-valid under arbitrary advice. Stale advice can damage its efficiency without invalidating its current-label bets.
We added the requested advice-guided comparator on the same pools, orders, thresholds, candidate sharing, and error budgets. "Chen and Weng (adapted)" follows their Algorithm 1 structure: moment forecasts, exponential weights from past Gaussian log scores, and clipped Kelly-style stakes. We replace the i.i.d. null with the remaining-pool null and return fixed-threshold decisions. The primary replay uses three forecasts: ledger-adjusted proxy, proxy only, and neutral. The checkpoint and judge studies add a historical-loss forecast. Appendix M gives the configurations and the adaptation's validity argument.
Neither method falsely certifies in this replay. The portfolio has the one DICES false rejection already reported in Table 1; the adaptation has none. Ledger-only is cheaper than the adaptation on both primary datasets. Against the adaptation, checkpoint and judge-study intervals do not resolve a difference. This comparison supports the implementation under the stated protocol, not a claim about the authors' original software or every possible advice-guided method. The repeated-judge and corrupted-advice studies provide the evidence for the hedge when advice is stale.