acceptodds
Under review as a conference paper at ICLR 2027

Certified Adaptive Refresh: Anytime-Valid Monitoring for Federated Conformal RAG

Abstract

Question-answering services built on retrieval-augmented generation (RAG), in which a language model answers from retrieved documents, are inspected continuously and upgraded repeatedly, so their reliability guarantee must survive both. We study federated conformal RAG: retrieval nodes holding private corpora score a shared list of candidate answers with a shared language model and send only compressed scores to a hub, which returns an answer set; a miss is a set that omits the true answer. We formulate monitoring as a sequential test: raise an alarm when misses exceed a certified upper bound on the miss rate (the allowance), while keeping the probability of any false alarm over the whole deployment below a preset budget; an alarm is false when no deployed configuration has drifted above its allowance. Existing tools solve only parts of this problem. A conformal certificate bounds the miss rate of one frozen configuration at one look fixed in advance, so it supplies no anytime-valid alarm, one that stays valid however often it is checked; a threshold calibrated for one model certifies nothing about its replacement; and re-certifying each upgrade at the full error level compounds failures. We propose Anytime-FC-RAG, a protocol with three auditable rules: pick the next configuration before drawing its fresh calibration data, charge every certificate attempt to one trajectory-wide budget , and fix the deployed threshold, allowance and monitoring bet before each query arrives. One betting wealth process , evidence that grows in expectation only when misses exceed the allowance, then runs across every deployment without reset. Our certified adaptive-refresh composition theorem shows that, for an evidence budget , alarming when first reaches has false-alarm probability under a random number of history-selected, evidence-driven refreshes of the model, retriever, corpus, score or threshold; pays for certificates that are wrong by chance. With an exact order-statistic certificate the guarantee is distribution-free and architecture-agnostic; it concerns misses against the allowance, not per-input coverage or distribution shift as such. Replaying Qwen2.5 on MMLU-Pro at a 5% total budget, all 210 runs whose shift an independent audit rated harmful alarm within 5,000 queries, and none of 222 benign or unshifted runs does. In a swap of the deployed model, calibrating the incoming model on fresh data holds a downgrade's miss rate at .089, under its .10 target, whereas calibrating it on the outgoing model's scores lets it rise to .154.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
anytime-valid inference, conformal prediction, e-processes, sequential testing, distribution-free uncertainty quantification, model monitoring, federated learning, retrieval-augmented generation, calibration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qUg9OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies sequential monitoring of prediction-set miscoverage in a federated conformal RAG system. Retrieval nodes score a shared list of candidate answers using local context and upload compressed scores to a central hub, which returns a calibrated answer set at last. The proposed protocol aims to monitor the process for missing answers via a combination of techniques. Theorem is proposed to bound the probability of any alarm over the deployment by the sum of monitoring and error budgets, with configuration changes. Emprical results reports synthetic stress tests, detector comparisons, and a model-swap experiment.

Critical strengths

(1) The proposal explicitly accounts for both certification failures and sequential false alarms. A theorem is provided for a trajectory-level false-alarm bound under the stated conditions, allowing history-dependent configuration changes and refreshes.

(2) The synthetic studies illustrate failure modes from unbudgeted certification and outcome-dependent decisions. These results support the feasibility of the mechanism in the tested settings.

Critical weaknesses

(1) The nontrivial methodological increment needs further clarification. The paper acknowledges that it inherits FC-RAG's inference mechanism and leverages established calibration and sequential-testing methods. The main part of paper does not sufficiently explain the obstacle that prevents a straightforward combination and what is gained by such combination, in both methodology and theorem. This is a major concern about novelty and significance.

(2) The federated RAG motivation and the usefulness of the inherited FC-RAG architecture are under-justified. The core theorem operates through generic error indicators, certificates, allowances, and predictability conditions. It does not appear to rely on the specific LM retrieval or federated structure. The paper does not sufficiently motivate why the proposal is useful in practice. Inheriting architecture from FC-RAG also does not establish its practical value. A comparison with feasible alternatives under the same information-sharing constraints would help, which is missing from the main context.

(3) The simulations do not adequately isolate the added value of composition. Section 5 includes detector comparisons and a model swap, however, several stress tests contrast compliant operation with invalid procedures, such as unbudgeted certification or decisions made after observing the outcome. These demonstrate the importance of the constraints, but not the benefit over combinations. Table 2 primarily compares detection on shifted streams. A more informative baseline would combine fresh state-matched calibration with trajectory-wide certification accounting and budgeted restarts of an existing monitor, whose proper option is already noted in Section 4.3. It should also examine late-onset shifts after long benign operation. The evaluation does not provide a full scope of contribution.

(4) The practical feasibility and empirical validation of repeated adaptive refresh remain insufficiently evaluated. Smaller certification budgets across refreshes require larger labeled calibration or looser allowances, but the resulting tradeoff is not quantified. The monitor tests the certified allowance, not the target miscoverage rate. Moreover, the main-text model swap does not establish behavior under repeated history-dependent configuration selection. The headline Qwen replays use fixed 30-question subject pools, so their audits concern fixed-pool risk. Repeated paths should not be interpreted as independent deployments. Appendix F.6 also acknowledges that the calibration/deployment sampling design does not by itself establish Assumption 1's conditional premise.

Questions for authors

(1) Please clarify the contribution beyond combining existing ingredients, both theoretically and experimentally. What technical obstacle prevents an existing betting-based risk monitor, combined with fresh state-matched calibration and conditional error-budget allocation? Which step requires a new argument, and what experimental comparison isolates the proposed protocol's added value over such combination?

(2) Which difficulty in the core theoretical analysis specifically requires federation or retrieval? I did not thoroughly check the theorem proofs, which I assume is consistent. But what practical evidence supports the usefulness of the inherited FC-RAG architecture, a comparison with feasible alternatives under the same information-sharing constraints would help, and what downstream decision benefits from its answer sets? Please distinguish the general monitoring contribution from the application-specific contribution.

(3) The weaknesses mention above on simulations should also be settled by authors.

4 comments
AuthorsOpenReview ↗

Response to Reviewer qUg9 (1/3)

Thank you for the review, and for the baseline you proposed: fresh state-matched calibration with trajectory-wide accounting and budgeted restarts of an existing monitor. We ran exactly that, on synthetic streams and real scores, and it became the centerpiece of the revision. We also took the presentation score to heart: the main text now says what the obstacle is (Section 1), what composition gains (new Section 4.4) and which experiments isolate it (Section 5).

In brief.

  • The obstacle: the obvious proof of the combination does not work, by an explicit example; the theorem needs a different argument, and its bound is tight (W1, Q1).
  • The gain over your baseline: the never-reset mixture of a restart's own processes alarms no later than the restart on every path (proved). With a refresh every 100 queries the never-reset default monitor detects .999 of changes against .006 and .045 for the two valid restarts; on real scores, .400, .213 and .565 of harmful paths against .031, .135 and .215 (W3).
  • The application: on a second benchmark with eight topic nodes, a routing hub rule gains 6 to 10 points of accuracy over no retrieval, within 2 points of pooled retrieval that no federation may use (W2).
  • Feasibility: with every component refreshed along each path, 5 to 131 attempts a path, every budget holds; none of the 13,536 cells with a stated bound exceeds it at the 1% level (W4).

Terms.

  • Allowance: the certified upper bound on a state's miss rate.
  • Wealth: the monitor's running evidence; it grows in expectation only when misses exceed the allowance, and the alarm fires when it reaches 1/δe1/\delta_ e.
  • Budgeted restart: a fresh monitor at each refresh with a share of δe\delta_ e, either summable (6δe/(π2m2)6\delta_ e/(\pi^2m^2) for the mm-th restart) or capped (δe\delta_ e split over a number of refreshes declared in advance).
  • Default monitor: our equal mix of the submission's adaptive bettor (aGRAPA) at several memory scales and a mixture over start times.
  • State attestation: each node's reported component digest is checked against the certified one.
  • Design and evaluation splits: on QASC, designs were piloted on the development questions and evaluated on the training questions.

W1. [The obstacle to a straightforward combination, and what is gained]

What is new. The cited monitors are valid for one fixed deployed model. Theorem 4.2 makes a betting risk monitor with state-matched certificates and error spending valid when the model, retriever, corpus, hub rule or threshold is changed on the strength of what was observed, any number of times.

The obstacle: each tool is valid under a condition that the other two break once refreshes depend on observed outcomes.

ToolValid whenWhat breaks it
Conformal certificatethe state is fixed before its calibration data are drawnchoosing the state, or keeping the threshold, with data already seen (wrong on up to .0775 at a level of .025; on at least .9999 of draws when a stronger model inherits the old threshold)
Betting monitor (Ville's inequality)every bet is fair given the pastthis holds only while the certificate in force is correct, which nobody observes
Error budgetthe number of certificates is fixed in advancerefreshes triggered by outcomes make it random (100 refreshes at the full .05: some certificate wrong with probability .9941)

The obvious proof does not work. "Assume every certificate is correct, apply Ville's inequality, add the budget" fails: given that every certificate is correct, the wealth is not a supermartingale, because whether a later certificate is attempted depends on earlier outcomes (Proposition B.4, an explicit example). The proof instead kills the wealth on an event fixed before each query, and the bound δe+δcal\delta_ e+\delta_ {cal} is tight (Proposition B.6, Remark B.7).

The gain. In method: one alarm, one threshold 1/δe1/\delta_ e and one budget for the whole deployment, however often and on whatever evidence the system is refreshed. Against the closest valid alternative, a budgeted restart: the never-reset mixture of a restart's own processes alarms no later than the restart on every path (Proposition C.2), restarts have exact detection ceilings (Corollary C.6), and the measured gains are in the table under W3.

In the revision: Sections 1, 4.4, 5; Appendices B.4, C.5, F.12.

AuthorsOpenReview ↗

Response to Reviewer qUg9 (2/3)

W2. [Federated RAG motivation; usefulness of the inherited architecture; feasible alternatives under the same information-sharing constraints]

General and specific. The theorem uses only miss indicators, certificates, allowances and predictability; the paper now says so ("The guarantee needs no federation; federated RAG is its application", Section 1).

Why this application. Its components belong to different parties and change at different times (each node's corpus and retriever, the shared model, the hub's rule), and the hub cannot see inside a node, so even uncertified changes occur; state attestation then cuts the post-change queries served above the allowance from .298 to .004 (Section 4.4; Appendix F.17).

The comparison under the same constraints, on a second benchmark where retrieval matters (QASC: 7,671 facts in twelve topics, the eight largest held by eight nodes, which send compressed scores and never documents; 7B scorer at 90% coverage):

SystemTop-1Mean set of 8Score vectors a query
No retrieval.6063.201
One node alone (average over nodes).558–1
Plurality vote of the nodes' top answers.590–8 answers
Mean over all eight nodes.6043.038
Silent-node rule (nodes without a relevant passage stay silent).6712.481.58
Most similar node alone.6792.541
Not available to a federation: one retriever over the pooled corpora.6902.481

Routing hub rules gain, and the certificate covers whichever rule is deployed (Remark D.4). The silent-node rule gains +.098, +.077 and +.065 over no retrieval for the 0.5B, 1.5B and 7B scorers (95% intervals above zero), comes within two points of pooled retrieval, and keeps the smallest sets while sending a fifth of the score vectors of the mean; routing to the single most similar node is about a point more accurate with slightly larger sets. Averaging every node does not gain on this split (it does for two of three scorers on the smaller design split): here the value lies in retrieving at the right node. On MMLU-Pro retrieval lowers accuracy, as the paper states.

In the revision: Sections 1, 5; Appendices D.5, F.16 (Table R9).

W3. [Baselines that isolate the added value: budgeted restarts and late-onset shifts]

We ran exactly this baseline: the same bettor with fresh state-matched calibration and trajectory-wide accounting, never reset or restarted at each refresh with a summable or capped share of the budget, on the same paths (116 cells of 5,000 paths, follow-up studies, and real scores).

Share of changes detectedNever reset (default)Restart, summableRestart, capped
Refresh every 100 queries, excess .04.999.006.045
The same with the allowance known; proved ceiling for any restarted bettor1.000.009 (ceiling .122).060 (ceiling .524)
Change after 50,000 benign queries (refresh every 1,000), excess .04.998.270.906
Real scores, refreshes chosen from history: QASC evaluation / design split, MMLU-Pro.400 / .213 / .565.031 / .135 / .215–
  • The bettor held fixed: never resetting the five-scale mixture detects more than its summable restart in 58 of 84 cells with harm, and fewer in none.
  • Late onset: detection on at least .998 of paths with median delays of 1,094, 1,706 and 1,894 queries after 500, 10,000 and 50,000 benign queries, and a proved delay bound that grows only logarithmically in the onset time (Proposition D.2). On real scores a small excess (two to four points) appearing after many refreshes takes thousands of queries to see; refreshing on the shifted traffic repairs the state, leaving .103 of queries above the allowance against .435 with stale blocks (Appendix F.18).
  • Monitors of the same risk null (the windowed betting rate of Timans et al., a running-risk test) run inside the composition and stay valid; for a fixed state the windowed rate matches our default, so the gain we claim is across refreshes.
  • Where a restart can be sooner (a scheduled refresh falling exactly on the change, by up to 294 queries in median), keeping that restart's own processes running is sooner in 15 of the 19 such cases, level in 3, and never later than the restart.

In the revision: Sections 3.3, 4.4, 5; Appendices C.5, D.4, F.12, F.13.

AuthorsOpenReview ↗

Response to Reviewer qUg9 (3/3)

W4. [Practical feasibility and validation of repeated adaptive refresh]

The trade-off is now quantified, and repeated history-dependent refresh is validated on real scores. In your order:

  1. The trade-off, quantified over 100 refreshes: the allowance grows from .172 to .286 with blocks of 100 and only from .121 to .150 with blocks of 1,000, while the 7B scorer's QASC set stays at 2.49 of 8 options; blocks taken from the labeled traffic need 1,000 extra labels a path instead of 21,000.
  2. The allowance, not the target, is what the monitor tests, by design (Section 4.4). Where the target itself matters, a strict order statistic tests it, at a cost of .3 of an option with blocks of 1,000.
  3. Repeated history-dependent selection, new on real scores: a controller moves among 36 states along each path, 5 to 131 certificate attempts a path (in one version every refresh's time and kind, the threshold rule included, come from the path's own record). Both budgets hold in every arm: some certificate wrong on .021 to .027 of paths, the alarm with every certificate correct on at most .018, against .025 each.
  4. Fixed 30-question pools. Agreed for those replays, which the paper now describes as such; every new real-score study draws questions i.i.d. from a fixed population, with a fresh block in every replication.
  5. Repeated paths are not independent deployments. Agreed: the 432 paths share fixed states and pools. With fresh blocks per replication, detection is a smooth function of the exact excess: .489 at one to two points, .932 at two to four, .997 at four to eight (QASC: .450, .903, .990).
  6. The premise of Appendix F.6. Redone with Assumption 1 holding by construction: the legal order is wrong at its level (9,050 and 9,031 of 720,000 at .0125 each), the alarm with both certificates correct stays at or below .0140, and the default monitor detects the post-swap drift on .88 to .89 of paths.

In the revision: Sections 4.4, 5; Appendices F.1, F.5, F.6, F.14 to F.18.

Q1. [The contribution beyond combining existing ingredients; the obstacle; the step that needs a new argument; the comparison that isolates the added value]

The new step is bounding the alarm jointly with the correctness of the certificates in force, without conditioning on it (the contribution and the obstacle are under W1).

  • Why it is needed: correctness is never observed, and whether a later certificate is attempted depends on earlier outcomes, so the bets cannot be called fair given that the certificates are correct.
  • The argument: the wealth is killed on an event fixed before each query (measurable then, though never observed), which yields δe+δcal\delta_ e+\delta_ {cal}; the bound is tight, and exactly δe+δcal−δeδcal\delta_ e+\delta_ {cal}-\delta_ e\delta_ {cal} under the timing of Algorithm 1 (Propositions B.4, B.6; Remark B.7).
  • The comparison that isolates the added value keeps everything but the reset: the never-reset mixture of a restart's own processes is never later than the restart in any of 3,064,721 restart alarms (gains under W3).

Q2. [What requires federation or retrieval; evidence for the inherited architecture; the downstream decision; general against application-specific]

  • What requires federation or retrieval in the analysis? Nothing; Section 1 now says so.
  • Evidence for the architecture against feasible alternatives: the table under W2.
  • Downstream decisions that use the answer sets, against the same decisions made from the hub's confidence at matched coverage (QASC, 7B scorer; values from the supplement's results, summarized in Appendix F.16):
Silent-node ruleMean over nodes
Answered automatically (the set has one option).376.226
Handled correctly (answered rightly, or the right answer in the list passed on): set, confidence.900, .865.900, .882
Wrong automatic answers after all traffic moves to topics no node holds: set, confidence.044, .084.027, .028

Answer or escalate handles more traffic correctly than the matched confidence rule in all 15 systems (by .014 to .045), and its wrong automatic answers can never exceed the monitored miss rate; a shortlist from the set holds the right answer more often than equally long top lists in 15 of 15 systems.

General against application-specific. General: the protocol, Proposition 4.1 with Theorem 4.2, the restart, late-onset and uncertified-change results. Application-specific: what a state is in federated conformal RAG (the hub rule included), runtime checks for changes the hub cannot see, the communication term, and the evidence on two benchmarks.

In the revision: Sections 1, 5; Appendices D.5, C.7, F.16.

Q3. [The weaknesses on simulations]

With frequent refreshes the never-reset monitor detects .999 of changes against .006 and .045 for valid restarts, and on real scores every budget holds (details under W3 and W4).

AuthorsOpenReview ↗

Polite follow-up regarding our rebuttal

Dear Reviewer,

Thank you once more for the constructive comments on our work.

Since the discussion period closes on Nov 18, we wanted to politely follow up to learn whether our point-by-point replies have resolved your concerns.

Should you have any outstanding questions, or if there is anything further we can clarify in these final days, we would be delighted to reply!

Kind regards,
The Authors

Reviewer_JL7cOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the monitoring issue of RAG in deployment. Standard conformal prediction can only be applied to a fixed check; it cannot provide effective alerts when models, retrievers, corpora, or thresholds change, nor under repeated checks. This paper proposes the Anytime-FC-RAG deployment protocol and a composition theorem, formalizing monitoring alarms as a sequential test and providing finite-sample exact, distribution-free certificates. Experiments show that the method maintains the preset false-alarm budget under any number of refreshes and model replacements, effectively distinguishes harmful from benign shifts, and alarms only on miss rates exceeding the allowance rather than on score changes alone; its detection ability is close to that of oracle detectors given the true post-change rate.

Critical strengths
  1. Instead of simply applying the anytime-valid tool, the four components of "finite sample exact certificate", "summable union bound", "Ville inequality" and "never reset wealth process" are stitched together into a auditable protocol. Certificate-preserving operations and guarantee-breaking practices are clearly delineated.
  2. It is clear that the goal of reliability monitoring is to determine whether the missed answer rate exceeds the certified allowance, rather than the change of score distribution. Updating the node corpus, changing the quantization noise or changing the version of the retriever may change the score distribution, but as long as the missed answer rate is within its allowance, the monitor should remain silent.
Critical weaknesses
  1. In federated RAG, node-side retriever/corpus updates may occur outside the hub’s timing audit; the protocol audits pre-query measurability, not within-epoch distributional stability or state–certificate matching. If a node update shifts inside an epoch without a fresh state-matched certificate, Assumption 1 and the deployment null need not hold, and the wealth monitor does not by itself detect this violation. Thus the “no weakening with the number of refreshes” claim is conditional on unobserved runtime assumptions; without runtime audit/enforcement of node changes and epoch score stability, practical assurance is limited.
  2. The experiments validate only a single-component student-model swap and synthetic interface stress tests; they do not exercise history-dependent refreshes of the retriever, corpus, or threshold under realistic federated operation. Thus, although the composition theorem may hold under its stated assumptions, the practical benefit of the full adaptive-refresh framework remains empirically unsupported.
Questions for authors
  1. Assumption 1 requires that, within an epoch, each deployed true-answer score follows the calibration law FrF_r conditional on FtpreF_t^{pre}. Is there any mechanism in the protocol that can detect that FrF_r has drifted during deployment? If not, what behavior will the monitor show when the assumption 1 is violated
  2. The paper's core claim is that the operator can replace any of the model, retriever, corpus, or threshold, but the experiments only perform student-model replacement; refreshes of the retriever, corpus, and threshold are not validated on real data streams. Could the authors explain why student-model replacement is representative of refreshes to the other components?
3 comments
AuthorsOpenReview ↗

Response to Reviewer JL7c (1/2)

Thank you for a precise review. Your account of what the guarantee covers and what it assumes is exactly right, and both weaknesses pointed to real gaps, which the revision closes.

In brief.

  • Node changes the hub does not certify (W1, Q1): the paper now states what holds when Assumption 1 fails inside an epoch, and adds two runtime checks, each with its own guarantee. State attestation (each node's reported component digest checked against the certified one) cuts the post-change queries served above the allowance from .298 to .004 on a second benchmark with real retrieval; label-free flags (indicators the hub sees without labels) catch .999 of node losses and at least .998 of model changes.
  • Refreshes of every component (W2, Q2): on real retrieval, a controller refreshes the retrieval policy, hub rule, corpus, threshold rule, model and bit budget along each path, 5 to 131 times (in one version every refresh's time and kind come from the path's own record), and both budgets hold in every arm.

Terms. The allowance is the certified upper bound on a deployed state's miss rate; the hub rule is how the hub combines the nodes' scores; on QASC, designs were piloted on the development questions (design split) and evaluated on the training questions (evaluation split).

W1. [Node-side changes outside the hub's audit; no runtime audit or enforcement]

Your description is accurate: the protocol audits timing, not within-epoch stability or state–certificate matching, and the wealth monitor alone does not detect a violation of Assumption 1 as such. The revision adds four things.

1. What holds under a violation (Remark C.9). Theorem 4.2 uses Assumption 1 only to obtain Equation (5). If a node changes without a new certificate and the miss rate stays within the allowance, the null still holds and the false-alarm bound is intact (between target and allowance the monitor is silent by design); if it rises above the allowance, the null is false and any alarm is a true detection.

2. The monitor under violation, on real retrieval. On QASC (eight topic nodes) corpora, nodes, retrievers, models or bit budgets change with the old certificate left in force: 320 variants of 2,000 replications on each split. The monitor alarms on .0004 of 427,100 benign uncertified changes and detects .566 of the harmful ones within 4,500 queries, and .976 or more of those above four points of excess.

3. State attestation (Remark C.11). Each node attaches a digest of its corpus version, retriever, model and encoder, and the hub compares it with the digest recorded in the certificate before using the message. A mismatch or a missing message is known before the outcome, so it may trigger a certified refresh at no cost to the alarm's bound (Corollary 4.3(f)). Measured on the same 320 variants, with the digests deciding when to refresh:

  • every declared change is caught at its first query and recertified from the next (304 of 304 changed variants), and no unchanged variant triggers a refresh;
  • the post-change queries served above the allowance fall from .298 with no refresh to .004 (at most .005 for any hub rule and kind of change);
  • validity is untouched: refreshed certificates wrong on 2,214 of 608,000 at a level of .0038, and no cell's alarm with every certificate correct above δe\delta_ e;
  • the cost is a block of 1,000 labels per declared change.

4. Label-free flags (Remark C.12). A second betting process watches an indicator the hub sees without labels, against a bound certified from the same block. A crossing triggers recertification, never an alarm, and its false-trigger probability is bounded by Theorem 4.2 applied to the indicator.

Flag 1: no node's best passage reaches the gateFlag 2: the hub's score of its top option
Fires with nothing changed (bound .05).0024.0020
Harmful changes flagged.999 of node lossesat least .998 of model changes
Queries served above the allowance after the change, without → with a refresh at the flag.716 → .008 (two nodes); .243 → .012 (eight).250 → .007

Scope, stated in the paper. Attestation trusts each node's report (misreporting is Byzantine, outside the scope) and covers the retriever changes the flags mostly miss; harm confined to one node's topic moves the pooled miss rate only a little (at most .014), which is where Flag 1, catching .999 of node losses, matters.

In the revision: Section 4.4; Appendices C.7, F.17 (Table R11).

AuthorsOpenReview ↗

Response to Reviewer JL7c (2/2)

W2. [Only a student swap and synthetic stress tests; no history-dependent refreshes of the retriever, corpus or threshold]

Correct for the submission; the revision adds history-dependent refreshes of every component on real scores, on two benchmarks.

Design (QASC). A controller walks 36 states: three Qwen2.5 scorers; three retrieval policies, each pairing a retriever setting with a hub rule; the nodes' own or extended corpora; 12 or 6 bits per score. Refreshes are triggered by evidence (the wealth has grown by a warning factor), by cost (a cheaper state is kept only if its block shows smaller sets, every attempt charged), and in one version by an exogenous clock standing for new retriever and corpus versions. Every block of 1,000 questions is drawn after the state is chosen; one wealth process runs throughout; every miss rate is exact; 20,000 paths per arm.

QASC, evaluation splitNatural nullBoundary null (every epoch exactly at its allowance)
Certificate attempts a path7.15.3
Some attempted certificate wrong (budget .025).0222.0209
Alarm with every certificate correct (budget .025).0007.0167
Either (budget .05).0229.0376
  • Every refresh from history (no clock). A review after 150 observed misses reads the epoch's record and changes the threshold rule, else the corpus (using a signal the hub sees without labels), else the retrieval policy. These reviews changed the retrieval policy on .913 of the null paths, the corpus on .797 and the threshold rule on .518, and both budgets hold over runs of up to 100,000 queries: some certificate wrong on at most .0248 of paths, an alarm with every certificate correct on at most .0175.
  • Long runs: over 100,000 queries with about 70 to 130 attempts a path, some certificate is wrong on at most .0272 of paths and the alarm fires with every certificate correct on at most .0182, against .025 each.
  • With the clock the retrieval policy changes on .98 of the null paths and the corpus on .91; the model on .012 and the bit budget on about .001, since evidence refreshes are rare under the null.

Realistic operation. Every real-score experiment replays cached scores of real models on real retrieval; the conclusion lists a live federation among what remains open.

In the revision: Section 5, Table 1; Appendices F.15, F.18 (Table R12).

Q1. [Is there a mechanism that can detect that the calibration law has drifted during deployment? If not, how does the monitor behave when Assumption 1 is violated?]

In the submission, no. In the revision, two mechanisms with stated guarantees: state attestation, which sees every declared node-side change or outage before the next outcome, and label-free flags, with a bounded false-trigger probability. Either may trigger a certified refresh at no cost to the alarm's bound (W1). On QASC the post-change queries served above the allowance are .298 with no refresh, .024 with a refresh at the first of two flags (the three hub rules where both exist), and .004 with attestation.

The monitor under violation tests the miss rate against the allowance and nothing else:

What the drift doesNull (5)MonitorMeasured on QASC
The miss rate stays within the allowancestill truesilent; the false-alarm bound is intactalarms on .0004 of 427,100 such changes
It rises above the allowancefalsean alarm is a true detection.566 detected within 4,500 queries; .976 or more above four points

Between the target and the allowance the monitor is silent by design; where that band matters, a strict order statistic tests the target itself, at .3 of an option.

In the revision: Section 4.4; Appendices C.7, F.17.

Q2. [Why is student-model replacement representative of refreshes to the other components?]

For validity, because neither Proposition 4.1 nor Theorem 4.2 looks inside the state: the certificate needs only that the block is scored by the state to be deployed and drawn after that state is chosen. The revision checks this instead of asserting it. Over 30 ordered pairs of states (20,000 fresh blocks each), whichever component differs (scorer, retrieval policy with hub rule, corpus, bit budget), the legal order is wrong on 14,853 (MMLU-Pro), 15,130 and 14,985 (QASC's two splits) of 600,000 blocks at a level of .025, and at its level for every pair.

The components differ in what the illegal order does. Calibrating on the outgoing state's scores is wrong on every block when a stronger scorer comes in (exact miss rate .229 to .303 against a target of .10), on .58 to .74 of blocks when one scorer moves from the lead retrieval setting to the second (at most .0002 the other way), and never when a weaker scorer comes in. So the student swap is where breaking the rule is most visibly harmful, not a special case for validity.

In the revision: Section 5; Appendices F.15, F.18.

AuthorsOpenReview ↗

Polite follow-up regarding our rebuttal

Dear Reviewer,

Thank you once more for the constructive comments on our work.

Since the discussion period closes on Nov 18, we wanted to politely follow up to learn whether our point-by-point replies have resolved your concerns.

Should you have any outstanding questions, or if there is anything further we can clarify in these final days, we would be delighted to reply!

Kind regards,
The Authors

Reviewer_LU8LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Anytime-FC-RAG maintains a betting wealth process on binary prediction-set misses. Each new score state is selected before acquiring fresh calibration data, every certificate attempt spends part of a trajectory-wide calibration-error budget, and the threshold, allowance, and bet are fixed before the next query. A single wealth process continues across epochs. On the events where the certificates are correct, the conditional miss mean is bounded by the active allowance; a killed-supermartingale argument and a union bound then give an overall alarm bound of delta_e + delta_cal. The paper supplies an order-statistic/Beta certificate, several operational corollaries, synthetic failure-mode studies, cached Qwen2.5/MMLU-Pro monitoring experiments, and a model-swap experiment. In the fixed-state Qwen replay, all 210 paths classified as harmful by an independent risk audit alarm by 5,000 queries, while no compatible or unshifted paths alarm. The authors also report a fresh-question confirmation for the strongest selected shift.

Critical strengths
  1. The statistical contract is explicit. The filtration, pre-query timing, conditional null, and certificate budgets make it possible to see exactly what the guarantee does and does not cover.

  2. The composition proof is short and readable. It handles a random, history-dependent sequence of calibration attempts without assuming independence across attempts. Charging discarded attempts avoids a realistic source of selection bias.

  3. The paper distinguishes miss-rate violations from arbitrary score drift, and marginal conformal coverage from the conditional-mean property required by the monitor.

Critical weaknesses
  1. The theoretical advance is limited. The composition largely follows from standard risk certificates, predictable betting, and error spending, with no direct comparison against an adapted sequential risk monitor.

  2. The model-swap experiment does not establish the certificate assumptions. Its calibration and deployment construction does not verify the conditional score-law requirement needed for the claimed guarantee.

  3. The practical benefit remains unclear. Large answer sets and the growing calibration cost of repeated refreshes leave the tradeoff between useful predictions, monitoring sensitivity, and labeling effort insufficiently tested.

Major Concerns

  1. The incremental theoretical contribution needs a stronger case.

The main construction combines a classical high-probability risk certificate, predictable betting, Ville's inequality, and a summable allocation of certificate errors. Once Equations (4) and (5) are assumed, the proof in Appendix B.2 is a direct supermartingale-and-union-bound argument. This is a clean result, but the paper needs to explain what substantial difficulty is resolved beyond applying established tools through an explicit interface.

The authors acknowledge the inherited components, which I appreciate. Still, Table 5 mainly contrasts the proposed combination with individual published methods in their original settings. It does not show why a standard sequential risk monitor equipped with state-matched certificates and ordinary error spending cannot provide essentially the same operational guarantee. Monitoring harmful risk changes while ignoring benign distribution changes is already the target of Podkopaev and Ramdas (2022), which the paper cites. The new part is the certificate composition across adaptive state changes, rather than that monitoring objective itself.

  1. The actual model-swap experiment does not validate the certificate assumptions.

Proposition 4.1 needs conditionally i.i.d. calibration scores, and Assumption 1 transports their law to deployment. Appendix F.6 instead uses disjoint, subject-stratified calibration blocks and separate fixed deployment pools. Its final paragraph explicitly states that this construction does not establish Assumption 1. The boundary-targeted mixtures are also fitted by Monte Carlo and do not certify the null inequality. The resulting experiment is informative about matching a threshold to the incoming model. It is weaker evidence for the advertised certified transition. An observed alarm rate of .025 or .015 cannot establish false-alarm control if the experimental null has not been shown to satisfy the premise being tested. Likewise, 200 replay paths do not replace repetitions over fresh calibration blocks when the certificate's error probability is itself part of the claim.

  1. The cost of certification and its effect on useful monitoring are not established.

There is a growing cost to repeated refresh. As the error allocation per attempt shrinks, a fixed-size calibration block produces a larger allowance; the appendix notes that increasing block sizes can compensate. The claim that the guarantee does not weaken with more refreshes is true for the error probability, but it does not imply preserved sensitivity. The experiments should show how block size, labeled-data cost, allowance inflation, and detection delay evolve over many refreshes.

The prediction task is also very permissive: the fixed-state Qwen answer sets contain 8.63 of ten options on average. The paper correctly calls these sets inefficient. However, this leaves the application benefit uncertain. It would be more convincing to show the coverage/set-size/detection tradeoff for a useful scorer and realistic label budget, rather than relying mainly on a system that retains almost every answer. The synthetic resource study, where bandwidth changes the allowance and cost but not the generated errors, does not resolve this issue.

  1. The strongest comparisons do not test the claimed advantage over the closest alternatives.

CTM tests exchangeability, so its response to a miss-preserving score change is expected behavior under its own objective. The authors explain this difference. These experiments demonstrate target specificity, but not superiority over another monitor of the same risk null. Similarly, CUSUM and Shiryaev-Roberts are given the true alternative but use thresholds calibrated for a finite horizon. Their loss of false-alarm control when those thresholds are retained at a longer horizon is unsurprising. This comparison is useful as an illustration of finite-horizon thresholding, but it does not establish an advantage over established time-uniform risk monitoring. Conversely, the oracle methods are much faster in the reported delay comparison and substantially better near the boundary, which deserves comparable emphasis.

  1. The headline real-model results remain difficult to reproduce independently.

Appendix G states that the two large score caches are not redistributed, so the Qwen replay cannot be rerun from the supplementary bundle alone. Retained summaries, provenance records, and hashes are useful for integrity checks, but a hash check cannot reproduce the scores or verify that the sampling and monitoring code generated the reported outcomes. Please release the synthetic score arrays where permitted, or provide a complete regeneration route with exact question identities, passages, prompt templates, model/retriever revisions, and seed mappings. The summary-only model-swap miss rates also prevent a paired uncertainty analysis of the .089-versus-.154 contrast. Retaining per-path results would fix that limitation.

Technical and Presentation Issues

  1. Separate the sources of randomness beside the headline 210/210 and 0/222 results. These are replay paths sharing fixed score states and question pools, not 432 independent model deployments or calibration experiments. The five allowances and the within-cell sampling structure matter for interpreting the apparent complete separation.

  2. The model-swap study preselects the incoming model and its calibration block before replay; evidence adapts the swap time. This is a valid special case to study, but it does not exercise history-dependent model selection followed by newly acquired calibration data. Describe the experiment at that level of specificity.

  3. In Figure 2(a) and Table 1, distinguish certificate failure from an actual alarm. The near-one rate for the unallocated scheme is dominated by certificate failures; Appendix F.3 reports that the monitor's alarm rate stays at most .0262 in that sweep. It should not be read as a near-one observed false-alarm rate.

  4. The false-alarm theorem is independent of federation once the miss sequence and certified allowances are supplied. Clarify which empirical benefit requires FC-RAG specifically. No network deployment is necessary to validate the abstract theorem, but the application framing should not imply that networked adaptive federation has been demonstrated.

  5. The oracle KL calculation in Appendix D.3 starts from a stationary alternative. It is not a delay characterization for a process carrying pre-change wealth and an estimated bet across many epochs. Keep this distinction explicit when discussing sustained-excess detection.

Questions for authors
  1. What concrete advantage does the composition provide over a standard conditional-risk e-process with state-matched certificates and error spending, or over the valid budgeted-restart construction in Appendix C.1?

  2. Can you repeat the adaptive-refresh experiment with fresh calibration and deployment scores drawn from the same conditional law in each epoch, repeating the calibration randomness as well as the query paths?

  3. How do labeled-data requirements, allowances, set sizes, and detection delays evolve over many refreshes and after long benign periods, especially when the true miss rate lies above alpha but below the allowance?

  4. Can the Qwen score caches, or all inputs and scripts needed to regenerate them, be supplied along with per-path results for the paired model-swap comparison?

9 comments
AuthorsOpenReview ↗

Response to Reviewer LU8L (1/5)

Thank you for a careful, fair and specific review; every point led to new theory, a new experiment or a correction.

In brief.

  • Theory (W1, W4, Q1): a standard risk monitor with state-matched certificates and error spending is an instance of Theorem 4.2; the theorem is what makes it valid when the model is changed on the strength of what was observed, which the cited monitors leave open. Against the valid budgeted restart, the never-reset mixture of the restart's own processes alarms no later on every path, restarts have exact detection ceilings, and a late onset costs only a logarithm.
  • The swap (W2, W5, Q2): redone as you asked, with calibration and deployment drawn i.i.d. from one population and the calibration randomness repeated. Certificates err at their stated rates, and the alarm with both certificates correct stays at or below .0140 (budget .025).
  • Cost and usefulness (W3, W6, Q3): over 100 refreshes with blocks of 1,000 the allowance grows from .121 to .150 while a 7B scorer's set stays at 2.49 of 8 options on a new benchmark where retrieval helps; blocks taken from the labeled traffic need 1,000 extra labels a path instead of 21,000.
  • Reproducibility (W8, Q4): the caches, per-path results and replay scripts are in the supplement.

Terms.

  • Allowance: the certified upper bound on a state's miss rate.
  • Budgeted restart: a fresh wealth process at each refresh with a share of the evidence budget δe\delta_ e, either summable (6δe/(π2m2)6\delta_ e/(\pi^2m^2) for the mm-th) or capped (δe\delta_ e split over a number of refreshes declared in advance).
  • Default monitor: our equal mix of the submission's adaptive bettor, aGRAPA, at several memory scales and a mixture over start times.
  • Kept mixture: the never-reset mixture of a restart's own processes, each kept betting.
  • Design and evaluation splits: on QASC, designs were piloted on the development questions and evaluated on the training questions.

W1. [Limited theoretical advance; no comparison with an adapted sequential risk monitor]

The comparison is now in the paper, and the advance is what the theorem makes valid (details under W4 and W7).

  • An adapted risk monitor is an instance of Theorem 4.2: run inside the composition, the betting rate of Timans et al. and a running-risk test in the style of Podkopaev and Ramdas stay valid. For a fixed state the windowed rate matches our default near the allowance (.764 against .758), and we claim no advantage there.
  • The advantage is across refreshes: with a refresh every 100 queries and an excess of .04, one never-reset process detects .999 of changes, and a summable or capped restart of the same bettor .006 and .045; with the allowance known, we prove that no restarted bettor can exceed .122 and .524.

In the revision: Sections 1, 4.4, 5; Appendices B.4, C.5, F.12, F.13.

W2. [The model-swap experiment does not establish the certificate assumptions]

Agreed; the evidence now comes from designs that satisfy the premise by construction (details under W5). Questions are drawn i.i.d. from a fixed population for calibration and deployment alike, every replication draws its own blocks, and every miss rate is exact. The legal order is then wrong at its level for every one of 30 ordered pairs of states (14,853 of 600,000 at .025 on MMLU-Pro), and in the redrawn swap the alarm with both certificates correct stays at or below .0140 (budget .025).

In the revision: Section 5; Appendices F.6, F.15, F.18.

W3. [The practical benefit remains unclear]

The revision adds a benchmark where the benefit can be shown (details under W6, W12 and Q3). On QASC (7,671 science facts in twelve topics, the eight largest held by eight nodes), with a 7B scorer:

  • Useful sets: 2.48 of 8 options at exact 90% coverage; on MMLU-Pro, where retrieved passages lower accuracy, the best system keeps 5.97 of 10 (the submission's replays: 8.63).
  • A useful decision: answering when the set has one option and escalating otherwise handles more traffic correctly than a matched confidence rule in all 15 systems (by .014 to .045), and its wrong automatic answers can never exceed the monitored miss rate.
  • The cost of refreshing, measured: over 100 refreshes with blocks of 1,000 the allowance grows from .121 to .150 while the set stays at 2.49 options; blocks taken from the labeled traffic need 1,000 labels a path beyond monitoring instead of 21,000.

In the revision: Section 5; Appendices F.14, F.16.

AuthorsOpenReview ↗

Response to Reviewer LU8L (2/5)

W4. [The incremental theoretical contribution needs a stronger case]

What is new beyond the short argument you describe: the theorem keeps a betting risk monitor valid when the deployed state is changed on the strength of what was observed; the cited monitors are stated for one fixed model (Timans et al. leave model updating to future work). Four results make the case (Sections 1, 4.4; proofs in the appendix).

1. The obvious proof does not work. "Condition on every certificate being correct, apply Ville's inequality on that event, add the calibration budget" fails because whether a later certificate is attempted depends on earlier outcomes. In an explicit deployment that satisfies (4) and (5), the wealth is not a supermartingale given that event, and the conditional alarm probability is δe/1−δcal(1−δe)>δe\delta_ e/\\{1-\delta_ {cal}(1-\delta_ e)\\}>\delta_ e (Proposition B.4). The theorem instead kills the wealth on an event fixed before each query. The bound δe+δcal\delta_ e+\delta_ {cal} is tight, and exactly δe+δcal−δeδcal\delta_ e+\delta_ {cal}-\delta_ e\delta_ {cal} under the timing of Algorithm 1 (Proposition B.6, Remark B.7). We claim no new proof technique; the contribution is the interface under which the short argument is correct.

2. Each rule is needed: dropping it is measured to fail (exact miss rates):

ShortcutMeasured
Calibrate the incoming model on the outgoing model's scorescertificate wrong on at least .9999 of draws when a stronger model comes in (MMLU-Pro); on every block on QASC
Choose the best of several states on the block that certifies itwrong on up to .0775 at a level of .025
Redraw blocks until the set looks small, without charging the trieswrong on .130 (MMLU-Pro) and .1615 (QASC) at a level of .025
Restart the monitor at the full budget at each refreshfalse alarms on up to .68 of boundary paths; .35 to .38 with about 70 attempts on real scores
All rules kept, every new experimentcertificates wrong on 152,793 of 10,826,942 draws, where the levels give 152,911; none of 13,536 cells with a stated bound above it at the 1% level

3. A standard risk monitor with state-matched certificates and error spending is an instance of Theorem 4.2, which is what proves that it keeps the guarantee across adaptive state changes (new row in Table 5); the paper credits Podkopaev and Ramdas with the monitoring objective.

4. What composition gains is proved and measured: domination of any budgeted restart (Proposition C.2), exact ceilings on restarts (Corollary C.6) and a delay bound logarithmic in the change time (Proposition D.2); measured gains under Q1.

In the revision: Sections 1, 4.4, 5; Table 5; Appendices B.4, C.5, D.4, F.11 to F.18.

W5. [The model-swap experiment does not validate the certificate assumptions]

Agreed on every count, and the swap is redone so that it does (also your Q2). Appendix F.6 now says what the submitted run is: one special case on two calibration blocks, with disjoint stratified pools and Monte Carlo-fitted boundary arms, which shows that a threshold must be matched to the incoming model.

The redone swap and its companions: calibration and deployment questions are i.i.d. draws from one known population; every replication draws its own blocks (20,000 paths per arm; 20,000 blocks per ordered pair of states); every miss rate is exact.

MMLU-ProQASC, evaluation split
Legal order, 30 ordered pairs, level .02514,853 of 600,000 wrong15,130 of 600,000 wrong
The swap, both blocks redrawn, level .0125 each9,050 and 9,031 of 720,000 wronga swap of scorers: .011 to .014 wrong
Alarm with every certificate correct, budget .025at most .0140 in the swap; at most .0153 with refreshes chosen from historyat most .0182 (5 to 130 attempts a path)

What the redrawn swap gives: the legal order holds the downgrade's miss rate at .099 on average against .164 for calibrate-then-replace (paired difference +.0648 over 20,000 redrawn blocks; the submitted .089 against .154 was one draw); an alarm or a wrong certificate occurs on at most .0399 of a cell's paths (budget .05); and the default monitor detects the post-swap drift on .88 to .89 of paths, cumulative aGRAPA on .61 to .66 (its submitted .925 and .955 belonged to that run's two blocks). The abstract and Section 5 report these.

In the revision: abstract; Section 5; Appendices F.1, F.6, F.15, F.18.

AuthorsOpenReview ↗

Response to Reviewer LU8L (3/5)

W6. [The cost of certification and its effect on useful monitoring]

Agreed: "does not weaken" holds for the error probability, and Section 1 now calls it a false-alarm guarantee. The cost to sensitivity is measured over 100 refreshes (summable schedule, target .10, δcal=.025\delta_ {cal}=.025):

Attempt1520100
Level of the attempt.0156.1×10−46.1\times10^{-4}3.8×10−53.8\times10^{-5}1.5×10−61.5\times10^{-6}
Allowance, blocks of 100.172.218.251.286
Allowance, blocks of 1,000.121.133.141.150
QASC, 7B scorer: mean set of 8, blocks of 1,0002.492.492.492.49
Synthetic: a shift to a miss rate of .20 detected (median delay), blocks of 1,0001.000 (197).999 (501).998 (772).999 (1,090)
The same, blocks of 100.145000
  • Block size is the lever: with blocks of 1,000 a shift to .20 is detected throughout; with blocks of 100 the allowance passes .20 by the fifth attempt, so that shift is within the null.
  • Labels. Blocks taken from the labeled traffic, scored in shadow by the new state, are valid fresh blocks when the traffic is stationary over the block and its epoch (Remark C.8). On QASC with 20 refreshes a path they need 1,000 labels beyond monitoring instead of 21,000; both budgets hold in all 24 stationary cells, with one query in ten labeled as well; and after a full shift of the traffic they repair the state on all but 6 of 24,000 paths.
  • Detection after many refreshes, on real scores. A small excess appearing late is detected slowly: on QASC's evaluation split the default detects .41 of the paths above their allowance after about 2 attempts and .03 to .07 after 24 to 118 (design split .31 to .34), budgets intact; Proposition D.2 locates the cost. Refreshing on the shifted traffic is the remedy: the controller refreshes on .985 of paths, a median of 582 queries after the change, and .103 of queries are served above the allowance, against .435 with stale blocks.

In the revision: Section 5 ("Refreshes cost little"); Appendices C.6, F.14, F.18 (Tables R7, R8, R13).

W7. [The strongest comparisons do not test the claimed advantage]

Agreed on each point. The CTM comparison shows target specificity only, since CTM responds to miss-preserving score changes by design, and the CUSUM and Shiryaev–Roberts comparison shows what finite-horizon thresholds cost, not an advantage over time-uniform risk monitoring; Appendices F.7 and F.13 say so. Section 5 now gives the oracle detectors' lead near the allowance and a monitor of the same risk null. On the paired streams (allowance .10, post-change rate .123):

MonitorDetectedMedian delay
CUSUM / Shiryaev–Roberts (told the post-change rate).917 / .9361,128 / 1,123
Timans et al., window 365 (same risk null).7641,315
Our default monitor.7581,625
Cumulative aGRAPA (submitted).6032,057
Running-risk test (same risk null).5602,106

Paired on the same streams, the default trails CUSUM by .159 (95% interval [.142, .175]) and Shiryaev–Roberts by .178 [.161, .195], and leads cumulative aGRAPA by .156 [.138, .173]. Among monitors of the same risk null none dominates, so the advantage we claim is across refreshes (Q1).

In the revision: Section 5; Appendices F.7, F.13 (Table R6).

W8. [The headline real-model results are difficult to reproduce independently]

The supplement now contains the score caches, a regeneration route and the per-path results (also your Q4).

  • Caches: the three score caches behind the Qwen2.5 runs, with a manifest row for each of their 21,616 vectors (question, node and corpus digests, passages by digest, model, retriever and dataset revisions).
  • Per-path results for all 2,400 paths of the model swap; the score tables of every new experiment on both benchmarks; the code, configurations, retained results and the protocol.
  • Replays from these files alone, on one CPU core: the 14-subject grid (every integer and string identical, floating-point values within a relative 5×10−115\times10^{-11}) and QASC's main study on either split.
  • Limits, stated: regenerated scores are bit-exact on the same GPU family and batching (1,800 of 1,800 vectors); across card families they differ by .0025 to .0028 per score on average, with the same top choice on at least 99.5% of vectors.

The paired analysis: within the submitted run, calibrate-then-replace raises the downgrade's miss rate by +.0650 (95% interval [+.0641, +.0658], conditional on its two blocks), on 200 of 200 paths; over 20,000 redrawn blocks, by +.0648 on average.

In the revision: Reproducibility statement; Appendices F.6, F.15, G.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to all reviewers (1/2)

We thank the reviewers for constructive reports. Reviewer LU8L finds the statistical contract explicit and the proof short and readable; Reviewer JL7c finds certificate-preserving operations and guarantee-breaking practices clearly delineated; Reviewer qUg9 credits a trajectory-level bound under history-dependent changes. The protocol, Theorem 4.2 and its proof are unchanged; the revision adds results the theorem covers, a default monitor for long deployments and a second benchmark, answering four shared questions.

Terms. The allowance is the certified upper bound on a deployed state's miss rate; the hub rule is how the hub combines the nodes' scores; state attestation checks each node's reported component digest against the certified one, and a label-free flag watches an unlabeled indicator with its own false-trigger budget, either triggering a certified refresh; on QASC, designs were piloted on the development questions (design split) and evaluated on the training questions (evaluation split).

Shared questionWhat the revision showsWhere
What does the composition give that its ingredients do not?The cited risk monitors are valid for one fixed model; the theorem keeps a betting risk monitor with state-matched certificates and error spending valid when any component is changed on the strength of what was observed. The obvious proof does not work, by an explicit example, and the bound δe+δcal\delta_ e+\delta_ {cal} is tight.Sections 1, 4.4; Appendix B.4
What does one never-reset process gain over a monitor restarted at each refresh with a share of the budget?The never-reset mixture of a restart's own processes alarms no later than the restart on every path (proved); a restarted monitor has an exact detection ceiling; a late change costs only a logarithm of its onset time. With a refresh every 100 queries the never-reset default monitor detects .999 of changes, the two valid restarts .006 and .045.Sections 4.4, 5; Appendices C.5, D.4, F.12
Do the budgets hold, under the stated assumptions, when refreshes are chosen from history?New designs draw questions i.i.d. from a known population, so every miss rate is exact; every component is refreshed along each path (by evidence, cost, an exogenous clock or the path's own record), 5 to 131 certificate attempts a path. None of the 13,536 cells with a stated bound exceeds it at the 1% level.Section 5, Table 1; Appendices F.11, F.15, F.18
Is the application useful, and what if a node changes without a new certificate?On QASC with eight topic nodes a routing hub rule gains 6 to 10 points of accuracy over no retrieval, within 2 points of pooled retrieval; a 7B scorer keeps 2.48 of 8 options at 90% coverage. State attestation cuts the post-change queries served above the allowance from .298 to .004.Sections 4.4, 5; Appendices C.7, F.16, F.17

The new experiments were specified before they ran (on QASC, before their runs on the evaluation split), most with an expected outcome, and no expectation about validity failed (Appendix F.19). No submitted number of a section, table, figure, equation or theorem has changed.

A. [What is new]

Theory, each result short and proved in the appendix:

ResultWhere
Conditioning on correct certificates, applying Ville and adding the budget does not work, by an explicit example; δe+δcal\delta_ e+\delta_ {cal} is tightPropositions B.4, B.6; Remark B.7
One never-reset process alarms no later than any budgeted restart; exact ceilings on what a restarted bettor can detectPropositions C.2, C.4, C.5; Corollary C.6
A delay bound for an excess that begins at an unknown time ν\nu, growing with ν\nu only through 2log⁡(ν/L+3)2\log(\nu/L+3) natsProposition D.2
Calibration blocks taken from the labeled traffic are valid fresh blocks when the traffic is stationary over the block and its epochRemark C.8
A node that changes without a new certificate: what holds, state attestation, and label-free flags with their own false-trigger boundSection 4.4; Remarks C.9 to C.12

Experiments, on cached scores of real models or synthetic streams with exact rates: every budget beside its measured rate (certificates wrong on 152,793 of 10,826,942 draws, where their levels give 152,911; Appendix F.11); never reset against budgeted restarts and monitors of the same risk null (F.12, F.13); the cost of refreshing (F.14); the first benchmark redrawn (F.15); a second benchmark where retrieval matters (F.16); node changes with no new certificate (F.17); refreshes chosen from history among 36 states (F.18).

Reproducibility. The supplement now holds the three score caches behind the Qwen2.5 runs (21,616 vectors, each with a manifest row), the per-path results of the swap, the score tables of every new experiment, the protocol, and scripts that replay results from these files on one CPU core.

AuthorsOpenReview ↗

Response to all reviewers (2/2)

B. [What we corrected]

Reviewer LU8L's checks led to three corrections; each keeps the finding and states it exactly, in the paper.

  1. The model swap, with both calibration blocks redrawn on every path. The legal order holds the downgrade's miss rate at .099 on average, against .164 for calibrate-then-replace (paired difference +.0648; the submitted .089 against .154 was one draw); the alarm with both certificates correct stays at or below .0140 (budget .025); and the default monitor detects the post-swap drift on .88 to .89 of paths, the submitted bettor (cumulative aGRAPA) on .61 to .66 (its .925 and .955 belonged to that run's two blocks). Abstract; Section 5; Appendices F.6, F.15.
  2. The headline replays (all 210 harmful paths alarm, none of 222 benign) share fixed states and 30-question pools. With questions and blocks redrawn in every replication, detection rises smoothly with the exact excess over the allowance, to .932 at two to four points. Section 5; Appendices F.5, F.15.
  3. The monitor for long deployments is now an equal mix of a five-scale aGRAPA mixture and a start-time mixture, which detects a change after 50,000 benign queries on .998 of paths and has a proved delay bound; the cumulative aGRAPA bettor alone loses power after long benign periods. Sections 3.3, 4.4; Appendices D.4, F.12.

Scope, stated plainly. For a fixed state, a time-uniform risk monitor such as the windowed rate of Timans et al. runs inside the composition and matches our default; the gain we claim is across refreshes. On MMLU-Pro retrieval lowers accuracy, so the application is shown on QASC. Every real-score experiment replays cached scores of real models; no live federation was run.

C. [How the paper changed]

Every new or updated passage is in blue; the main text uses the tenth page.

  • Abstract: the swap's redrawn means; one sentence on the second benchmark.
  • Section 1: The pieces do not compose by themselves; the guarantee named precisely as a false-alarm guarantee; the second benchmark.
  • Section 3.3: the default monitor for long deployments.
  • Section 4.4 (new): What composition gains, and how it handles uncertified changes.
  • Section 5 (now "Experiments"): validity of the added studies; three rows in Table 1; captions that separate certificate failure from alarms; the replays' units of replication; the redrawn swap; the oracle detectors' lead and a monitor of the same null; three new paragraphs (never reset against restarts; the second benchmark; what refreshes cost).
  • Section 6: the second benchmark's result; a live federation listed as open.
  • Appendix: new subsections B.4, C.5 to C.7, D.4, D.5 and F.11 to F.19; a row and a note in Table 5 and a note in Table 6; notes in C.1, D.3, F.1, F.3, F.5, F.6, F.7 and G; every table now sits beside its section.

Dear all,

We're the half-way through the discussion phase. The authors have provided their response to your comments. Some of you have started involving discussion, but all of us can take action now.

If the provided responses do not address (even part of) your concerns, this is your chance to raise it. Please engage with the authors as soon as possible, we need to keep the conversation going!

Thanks,

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.