- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper studies sequential monitoring of prediction-set miscoverage in a federated conformal RAG system. Retrieval nodes score a shared list of candidate answers using local context and upload compressed scores to a central hub, which returns a calibrated answer set at last. The proposed protocol aims to monitor the process for missing answers via a combination of techniques. Theorem is proposed to bound the probability of any alarm over the deployment by the sum of monitoring and error budgets, with configuration changes. Emprical results reports synthetic stress tests, detector comparisons, and a model-swap experiment.
Critical strengths
(1) The proposal explicitly accounts for both certification failures and sequential false alarms. A theorem is provided for a trajectory-level false-alarm bound under the stated conditions, allowing history-dependent configuration changes and refreshes.
(2) The synthetic studies illustrate failure modes from unbudgeted certification and outcome-dependent decisions. These results support the feasibility of the mechanism in the tested settings.
Critical weaknesses
(1) The nontrivial methodological increment needs further clarification. The paper acknowledges that it inherits FC-RAG's inference mechanism and leverages established calibration and sequential-testing methods. The main part of paper does not sufficiently explain the obstacle that prevents a straightforward combination and what is gained by such combination, in both methodology and theorem. This is a major concern about novelty and significance.
(2) The federated RAG motivation and the usefulness of the inherited FC-RAG architecture are under-justified. The core theorem operates through generic error indicators, certificates, allowances, and predictability conditions. It does not appear to rely on the specific LM retrieval or federated structure. The paper does not sufficiently motivate why the proposal is useful in practice. Inheriting architecture from FC-RAG also does not establish its practical value. A comparison with feasible alternatives under the same information-sharing constraints would help, which is missing from the main context.
(3) The simulations do not adequately isolate the added value of composition. Section 5 includes detector comparisons and a model swap, however, several stress tests contrast compliant operation with invalid procedures, such as unbudgeted certification or decisions made after observing the outcome. These demonstrate the importance of the constraints, but not the benefit over combinations. Table 2 primarily compares detection on shifted streams. A more informative baseline would combine fresh state-matched calibration with trajectory-wide certification accounting and budgeted restarts of an existing monitor, whose proper option is already noted in Section 4.3. It should also examine late-onset shifts after long benign operation. The evaluation does not provide a full scope of contribution.
(4) The practical feasibility and empirical validation of repeated adaptive refresh remain insufficiently evaluated. Smaller certification budgets across refreshes require larger labeled calibration or looser allowances, but the resulting tradeoff is not quantified. The monitor tests the certified allowance, not the target miscoverage rate. Moreover, the main-text model swap does not establish behavior under repeated history-dependent configuration selection. The headline Qwen replays use fixed 30-question subject pools, so their audits concern fixed-pool risk. Repeated paths should not be interpreted as independent deployments. Appendix F.6 also acknowledges that the calibration/deployment sampling design does not by itself establish Assumption 1's conditional premise.
Questions for authors
(1) Please clarify the contribution beyond combining existing ingredients, both theoretically and experimentally. What technical obstacle prevents an existing betting-based risk monitor, combined with fresh state-matched calibration and conditional error-budget allocation? Which step requires a new argument, and what experimental comparison isolates the proposed protocol's added value over such combination?
(2) Which difficulty in the core theoretical analysis specifically requires federation or retrieval? I did not thoroughly check the theorem proofs, which I assume is consistent. But what practical evidence supports the usefulness of the inherited FC-RAG architecture, a comparison with feasible alternatives under the same information-sharing constraints would help, and what downstream decision benefits from its answer sets? Please distinguish the general monitoring contribution from the application-specific contribution.
(3) The weaknesses mention above on simulations should also be settled by authors.
4 comments
Response to Reviewer qUg9 (2/3)
W2. [Federated RAG motivation; usefulness of the inherited architecture; feasible alternatives under the same information-sharing constraints]
General and specific. The theorem uses only miss indicators, certificates, allowances and predictability; the paper now says so ("The guarantee needs no federation; federated RAG is its application", Section 1).
Why this application. Its components belong to different parties and change at different times (each node's corpus and retriever, the shared model, the hub's rule), and the hub cannot see inside a node, so even uncertified changes occur; state attestation then cuts the post-change queries served above the allowance from .298 to .004 (Section 4.4; Appendix F.17).
The comparison under the same constraints, on a second benchmark where retrieval matters (QASC: 7,671 facts in twelve topics, the eight largest held by eight nodes, which send compressed scores and never documents; 7B scorer at 90% coverage):
| System | Top-1 | Mean set of 8 | Score vectors a query |
|---|---|---|---|
| No retrieval | .606 | 3.20 | 1 |
| One node alone (average over nodes) | .558 | – | 1 |
| Plurality vote of the nodes' top answers | .590 | – | 8 answers |
| Mean over all eight nodes | .604 | 3.03 | 8 |
| Silent-node rule (nodes without a relevant passage stay silent) | .671 | 2.48 | 1.58 |
| Most similar node alone | .679 | 2.54 | 1 |
| Not available to a federation: one retriever over the pooled corpora | .690 | 2.48 | 1 |
Routing hub rules gain, and the certificate covers whichever rule is deployed (Remark D.4). The silent-node rule gains +.098, +.077 and +.065 over no retrieval for the 0.5B, 1.5B and 7B scorers (95% intervals above zero), comes within two points of pooled retrieval, and keeps the smallest sets while sending a fifth of the score vectors of the mean; routing to the single most similar node is about a point more accurate with slightly larger sets. Averaging every node does not gain on this split (it does for two of three scorers on the smaller design split): here the value lies in retrieving at the right node. On MMLU-Pro retrieval lowers accuracy, as the paper states.
In the revision: Sections 1, 5; Appendices D.5, F.16 (Table R9).
W3. [Baselines that isolate the added value: budgeted restarts and late-onset shifts]
We ran exactly this baseline: the same bettor with fresh state-matched calibration and trajectory-wide accounting, never reset or restarted at each refresh with a summable or capped share of the budget, on the same paths (116 cells of 5,000 paths, follow-up studies, and real scores).
| Share of changes detected | Never reset (default) | Restart, summable | Restart, capped |
|---|---|---|---|
| Refresh every 100 queries, excess .04 | .999 | .006 | .045 |
| The same with the allowance known; proved ceiling for any restarted bettor | 1.000 | .009 (ceiling .122) | .060 (ceiling .524) |
| Change after 50,000 benign queries (refresh every 1,000), excess .04 | .998 | .270 | .906 |
| Real scores, refreshes chosen from history: QASC evaluation / design split, MMLU-Pro | .400 / .213 / .565 | .031 / .135 / .215 | – |
- The bettor held fixed: never resetting the five-scale mixture detects more than its summable restart in 58 of 84 cells with harm, and fewer in none.
- Late onset: detection on at least .998 of paths with median delays of 1,094, 1,706 and 1,894 queries after 500, 10,000 and 50,000 benign queries, and a proved delay bound that grows only logarithmically in the onset time (Proposition D.2). On real scores a small excess (two to four points) appearing after many refreshes takes thousands of queries to see; refreshing on the shifted traffic repairs the state, leaving .103 of queries above the allowance against .435 with stale blocks (Appendix F.18).
- Monitors of the same risk null (the windowed betting rate of Timans et al., a running-risk test) run inside the composition and stay valid; for a fixed state the windowed rate matches our default, so the gain we claim is across refreshes.
- Where a restart can be sooner (a scheduled refresh falling exactly on the change, by up to 294 queries in median), keeping that restart's own processes running is sooner in 15 of the 19 such cases, level in 3, and never later than the restart.
In the revision: Sections 3.3, 4.4, 5; Appendices C.5, D.4, F.12, F.13.
Response to Reviewer qUg9 (3/3)
W4. [Practical feasibility and validation of repeated adaptive refresh]
The trade-off is now quantified, and repeated history-dependent refresh is validated on real scores. In your order:
- The trade-off, quantified over 100 refreshes: the allowance grows from .172 to .286 with blocks of 100 and only from .121 to .150 with blocks of 1,000, while the 7B scorer's QASC set stays at 2.49 of 8 options; blocks taken from the labeled traffic need 1,000 extra labels a path instead of 21,000.
- The allowance, not the target, is what the monitor tests, by design (Section 4.4). Where the target itself matters, a strict order statistic tests it, at a cost of .3 of an option with blocks of 1,000.
- Repeated history-dependent selection, new on real scores: a controller moves among 36 states along each path, 5 to 131 certificate attempts a path (in one version every refresh's time and kind, the threshold rule included, come from the path's own record). Both budgets hold in every arm: some certificate wrong on .021 to .027 of paths, the alarm with every certificate correct on at most .018, against .025 each.
- Fixed 30-question pools. Agreed for those replays, which the paper now describes as such; every new real-score study draws questions i.i.d. from a fixed population, with a fresh block in every replication.
- Repeated paths are not independent deployments. Agreed: the 432 paths share fixed states and pools. With fresh blocks per replication, detection is a smooth function of the exact excess: .489 at one to two points, .932 at two to four, .997 at four to eight (QASC: .450, .903, .990).
- The premise of Appendix F.6. Redone with Assumption 1 holding by construction: the legal order is wrong at its level (9,050 and 9,031 of 720,000 at .0125 each), the alarm with both certificates correct stays at or below .0140, and the default monitor detects the post-swap drift on .88 to .89 of paths.
In the revision: Sections 4.4, 5; Appendices F.1, F.5, F.6, F.14 to F.18.
Q1. [The contribution beyond combining existing ingredients; the obstacle; the step that needs a new argument; the comparison that isolates the added value]
The new step is bounding the alarm jointly with the correctness of the certificates in force, without conditioning on it (the contribution and the obstacle are under W1).
- Why it is needed: correctness is never observed, and whether a later certificate is attempted depends on earlier outcomes, so the bets cannot be called fair given that the certificates are correct.
- The argument: the wealth is killed on an event fixed before each query (measurable then, though never observed), which yields ; the bound is tight, and exactly under the timing of Algorithm 1 (Propositions B.4, B.6; Remark B.7).
- The comparison that isolates the added value keeps everything but the reset: the never-reset mixture of a restart's own processes is never later than the restart in any of 3,064,721 restart alarms (gains under W3).
Q2. [What requires federation or retrieval; evidence for the inherited architecture; the downstream decision; general against application-specific]
- What requires federation or retrieval in the analysis? Nothing; Section 1 now says so.
- Evidence for the architecture against feasible alternatives: the table under W2.
- Downstream decisions that use the answer sets, against the same decisions made from the hub's confidence at matched coverage (QASC, 7B scorer; values from the supplement's results, summarized in Appendix F.16):
| Silent-node rule | Mean over nodes | |
|---|---|---|
| Answered automatically (the set has one option) | .376 | .226 |
| Handled correctly (answered rightly, or the right answer in the list passed on): set, confidence | .900, .865 | .900, .882 |
| Wrong automatic answers after all traffic moves to topics no node holds: set, confidence | .044, .084 | .027, .028 |
Answer or escalate handles more traffic correctly than the matched confidence rule in all 15 systems (by .014 to .045), and its wrong automatic answers can never exceed the monitored miss rate; a shortlist from the set holds the right answer more often than equally long top lists in 15 of 15 systems.
General against application-specific. General: the protocol, Proposition 4.1 with Theorem 4.2, the restart, late-onset and uncertified-change results. Application-specific: what a state is in federated conformal RAG (the hub rule included), runtime checks for changes the hub cannot see, the communication term, and the evidence on two benchmarks.
In the revision: Sections 1, 5; Appendices D.5, C.7, F.16.
Q3. [The weaknesses on simulations]
With frequent refreshes the never-reset monitor detects .999 of changes against .006 and .045 for valid restarts, and on real scores every budget holds (details under W3 and W4).
Polite follow-up regarding our rebuttal
Dear Reviewer,
Thank you once more for the constructive comments on our work.
Since the discussion period closes on Nov 18, we wanted to politely follow up to learn whether our point-by-point replies have resolved your concerns.
Should you have any outstanding questions, or if there is anything further we can clarify in these final days, we would be delighted to reply!
Kind regards,
The Authors
Response to Reviewer qUg9 (1/3)
Thank you for the review, and for the baseline you proposed: fresh state-matched calibration with trajectory-wide accounting and budgeted restarts of an existing monitor. We ran exactly that, on synthetic streams and real scores, and it became the centerpiece of the revision. We also took the presentation score to heart: the main text now says what the obstacle is (Section 1), what composition gains (new Section 4.4) and which experiments isolate it (Section 5).
In brief.
Terms.
W1. [The obstacle to a straightforward combination, and what is gained]
What is new. The cited monitors are valid for one fixed deployed model. Theorem 4.2 makes a betting risk monitor with state-matched certificates and error spending valid when the model, retriever, corpus, hub rule or threshold is changed on the strength of what was observed, any number of times.
The obstacle: each tool is valid under a condition that the other two break once refreshes depend on observed outcomes.
The obvious proof does not work. "Assume every certificate is correct, apply Ville's inequality, add the budget" fails: given that every certificate is correct, the wealth is not a supermartingale, because whether a later certificate is attempted depends on earlier outcomes (Proposition B.4, an explicit example). The proof instead kills the wealth on an event fixed before each query, and the bound δe+δcal is tight (Proposition B.6, Remark B.7).
The gain. In method: one alarm, one threshold 1/δe and one budget for the whole deployment, however often and on whatever evidence the system is refreshed. Against the closest valid alternative, a budgeted restart: the never-reset mixture of a restart's own processes alarms no later than the restart on every path (Proposition C.2), restarts have exact detection ceilings (Corollary C.6), and the measured gains are in the table under W3.
In the revision: Sections 1, 4.4, 5; Appendices B.4, C.5, F.12.