- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper investigates requalification under distribution shift. Some alternative that was previously rejected is recommended to be reactivated by a proposer model. A gate then accepts or rejects the switch. They state a necessary and sufficient condition for when a gate decreases expected regret: if and only if the proposer's precision π is below q_w H_w / (q_w H_w + q_c H_c), where q_c and q_w are the rates at which the gate blocks correct and wrong proposals and H_c and H_w are the average costs of those blocks. The paper measures these quantities on held-out data, and compare the gate with controls like random rate-matched blockers.
Critical strengths
- It would be interesting to measure the cost of vetoes in some domains.
- The math seems correct.
Critical weaknesses
- The paper's focus seems unmotivated. I did not get a good sense of why this question is particularly important or interesting to study.
- "Under distribution shift, that balance can change abruptly because the proposer itself can move from unreliable to highly reliable." This sort of move would be surprising! If a proposer is unreliable in-distribution, it seems unlikely that it would become highly reliable out-of-distribution. It would be kind of like a child that has only been taught English and failed to learn to speak it correctly turning out to be very good at speaking Spanish without any training.
- The paper's experiments are all on synthetic data, but the kind of question it studies would only be interesting if it were on real data, and there's little reason to think that experiments on synthetic data are any real guide to performance on real data in this case.
- The controlled-precision experiment results seem trivial. Here the only thing that's changed is the mixture proportion pi, so q_c, q_w, H_c, and H_w remain the same, and the results match Δ = ρ[π q_c H_c − (1−π) q_w H_w] with sampling errors.
- The paper's headline effect is very small: the change in regret is -0.0034.
- The paper seems substantially AI-generated with little human vetting. Tells include a substantial dressing-up of mostly trivial and uninteresting results, extensive references to the history of the project ("An early version of this project," etc.), and many new coinages (QRL, ContextEvidenceML, FullValidityVeto, EvidenceRank, DelayMatchedRank, LearnedDeferred, etc.).
Questions for authors
- Since only π changes between calibration and testing, how could the calibration-only prediction have been falsified?
- Can you test on a real gating task?
2 comments
Reply
Thank you. I'm going to keep my score. It looks like the human authors aren't paying any attention to this paper at all. The replies here are entirely AI-generated, and the paper has been completely overhauled in response to the reviews. It's not worth my attention to engage with this.
Motivation. Whenever a deployed model is retrained, the old and new models disagree on some inputs, and for each of those inputs someone must decide whether to accept the new answer. The gate is that decision rule, and the question is when it lowers regret.
"The proposer itself can move from unreliable to highly reliable." The sentence was badly put, and your analogy is fair for a fixed proposer on a new distribution. In the setting we now study the proposer is the retrained model, and its precision is the share of its disagreements with the deployed model on which it is right. That share depends on how stale the deployed model has become, and it moved a great deal within single streams: across periods from 0.22 to 0.86 on Elec2, 0.23 to 0.83 on Rialto and 0.08 to 0.66 on INSECTS-abrupt. The revision replaces the sentence with these observed ranges.
Q1. How could the calibration-only prediction have been falsified? In the submitted experiment it could not; you were right, and the controlled-precision result is now labelled an identity check. In the new experiments the predictions were posted in this forum before scoring, and many failed: label-free monitoring of the gate effect on HuffPost and arXiv (r = -0.19 and -0.42); transport of the frozen consequence estimates on HuffPost (r = 0.10); the label-budget rule on arXiv; the fallback M7 on covtype and poker; and two of the three mechanism predictions tested on INSECTS. Three later policies gave no confirmed result. Only one of our five adaptive policies met its primary criterion.
Q2. A real gating task. The new experiments use model update on public data: news topics by month (HuffPost), arXiv categories by month, and twelve further evaluable streams. The observations are real. The update schedule and the gate are ours, and several streams are in file order, not time order. It is not a deployed system, and we say so.
Effect size. We now report effects relative to the regret of deploying the update with no gate. The learned gate changed regret by +4.8% (HuffPost), +22.0% (arXiv), -51% (covtype and poker), -31% (INSECTS-abrupt), and +18% and +7% (the other two INSECTS streams). The sign changes with the regime, which is now the paper's main finding.
AI assistance and vetting. The submission declared AI assistance for writing, ideation and execution, drafting, synthetic data and proofs. The same holds for the new experiments, and the revision states it. We take responsibility for every claim. Each commitment used code hashes posted before scoring; second runs reproduced the verdicts, with small floating-point differences in 9 of 54 C3 periods. Failures are reported alongside successes. The revision removes the project-history passages and the coined names you listed, replaces them with standard descriptions, and organises the experiments around one question.
We would be grateful to know whether this addresses your concerns.