acceptodds
Under review as a conference paper at ICLR 2027

Safe to Omit, Worth Omitting? Keeping the Answer and Cutting Latency When an LLM Agent Skips Another's Message

Abstract

In collaborative multi-agent systems, a receiver agent often waits for a message from a helper agent, which can be either natural-language text or its latent KV cache. While this communication can improve performance, it also adds latency. We investigate when a receiver agent can skip the helper agent's message safely (the final answer rarely changes) and beneficially (the time saved exceeds the cost of deciding). We propose a dynamic skipping mechanism that reads the receiver agent's confidence without the message and applies a finite-sample statistical test. This test certifies a confidence threshold at which, with high probability, at most 5% of skipped answers change; if no threshold passes, the system keeps communicating. We evaluate our mechanism on five benchmarks: three multiple-choice benchmarks with three model pairs each, plus an extractive (SQuAD) and a math (GSM8K) benchmark. Three main insights emerge. First, oracle headroom, a common metric for routing among paths, is weak evidence that messages help. Control inputs without answer-specific content, matched to change answers as often as real messages, give comparable headroom and reproduce a median 76% of the oracle's gain over the receiver agent. Whether text messages beat these controls with statistical significance depends on how they are matched. Second, whether a message can be skipped depends on how often and how predictably it alters the receiver agent's answer, not on its overall accuracy. Third, safety does not determine efficiency: at batch size one, skipping text messages saves hundreds of milliseconds per query, whereas skipping latent KV cache saves far less. Certification extends to SQuAD, which a label-free rule chose from eight candidate settings, while on GSM8K no threshold passes. Finally, in eleven tests on unseen questions, every certified policy we could test kept its change rate's 95% upper bound below 5%, though only four tests had a pass criterion set in advance, and the sealed KV-cache result holds only under our answer parser. The code is available at: https://anonymous.4open.science/r/skip-message-cert.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large language models, LLM agents, multi-agent communication, latent communication, KV cache, selective prediction, distribution-free risk control, finite-sample certification, inference latency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5yjKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates when an LLM receiver can skip a helper’s message while usually preserving the answer of the always-communicating system. The proposed method uses receiver confidence and finite-sample testing to choose a skipping threshold, then measures net latency savings after accounting for the probe. The empirical results cover text and KV-cache communication, with successful certification in several multiple-choice settings and one extractive setting.

Critical strengths

S1. I have to admit that the problem formulation is clear. Preserving a deployed system’s behavior without requiring correctness labels is a useful goal. The paper also correctly separates this goal from preserving accuracy and obtaining a net speedup.

S2. The empirical analysis contains useful observations. Specifically, the headroom controls challenge a potentially misleading interpretation of oracle gains, and the text-versus-KV comparison shows why similar skipping coverage need not produce similar latency savings.

S3. The analysis parts are good. The authors include sveral useful ablations which help distinguish stronger evidence from exploratory results.

Critical weaknesses

W1. One of my biggest concerns is the limited experimental coverage, despite the additional benchmarks. Although the MMLU-Pro and the cross-family experiments are useful additions (yet they are not included in the abstract), but the successful results still mostly concern short, parsed answers in a two-agent, single-message setup. Besidse, the main models are at most around 8B with non-thinking mode and the GSM8K score fails to support certification. SQuAD establishes that the approach can work outside multiple choice, but it is one setting selected from eight pilot candidates, not broad evidence for open-ended generation.

W2. My another biggest concern is the takeaway message of this paper. I merit the technical contribution of this paper, i.e., the proposed method is useful in the empirical results. However, the paper is mainly talking about an empirical lesson on evaluating communication, rather than something new and insightful to the audience like me. A rebuttal that clarifies the distinctive contribution that make the work more insightful could move me toward acceptance. Note that a stronger wording alone would not resolve these concerns, but a better formulation of the existing evidence might.

Questions for authors

Q1. What is the main contribution you want readers to take away? Is it a new communication-specific risk formulation, the headroom critique, or an empirical account of when skipping pays? Please make the distinction from CALM and existing risk-controlled routing precise.

Q2. Which existing result is the strongest evidence for preserving useful collaboration? The Llama Text settings seem promising. Can the rebuttal use these results to explain the joint trade-off between answer changes, retained accuracy gain, and net latency, rather than mainly emphasizing certification success?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5yjK

Thank you for the constructive review, and for recognizing the problem formulation and the headroom controls as strengths. Your two questions shaped the framing of the revision.

Q1 / W2: the main takeaway. In one sentence: a message can be skipped safely when it rarely and predictably changes the receiver's answer, a property that oracle headroom misses and accuracy does not reveal, and whether the skip then pays is a separate, measured question. The results support this in three steps. Headroom arises without answer-specific content, so it cannot tell useful messages from volatile ones (§4.1). Certification follows the disagreement rate and its predictability, not accuracy, and two label-free quantities predict the outcome (§4.2, App. D). Where a message does help, skipping keeps most of the gain and saves time (§4.3–4.4). Two new experiments test this last step with the procedure fixed in advance: a three-agent system and a 32B pair keep 98% and 100% of the gain (General response, parts 2 and 3). The revised abstract and introduction lead with this claim.

Relation to CALM and risk-controlled early exiting. We share their risk-control framework (Learn then Test with a consistency risk relative to a fixed reference) and claim no new test; neither guarantee requires the reference to be more accurate. CALM bounds the expected inconsistency over all inputs; we bound the change rate among skipped questions. Within each decoding step, an early-exit gate reads intermediate states of the computation it may cut short; our gate reads a separate message-free prefill, which in our implementation adds a forward pass to every query, so whether skipping pays must be measured (§4.4). A message also comes from another agent: it may carry no answer-specific content (§4.1) or make the system worse (§4.2), so we measure gain retention separately (§4.3).

Relation to certified routing (Uddin & Bauer, 2026): they certify the correctness of the routed answer, which needs labels; our change-rate certificate bounds agreement with a deployed path that may be less accurate than the receiver alone, and needs none.

Q2: the joint trade-off in the Llama text settings. Agreed. Among the original experiments, Llama-8B/ARC is the clearest case: on the 1,172 test questions, text adds 3.41 points [1.71, 5.12] over the receiver alone; the policy skips 67.7% of questions, changes 1.9% of skipped answers, and keeps 3.24 points [1.62, 4.86]; on the 128-question development timing panel it saves 419 ms per query. The table below collects both settings (see also Tables 2 and 3 of the revision):

Llama-3.1-8B receiver, text from the large helperOBQAARC
Skipped (development)59.0%68.9%
Changed among skipped: development / unseen1.6% / 0.7%1.9% / 1.9%
Text's gain over the receiver: development / unseen (points)4.45 / 2.697.02 / 3.41
Gain kept by the policy: development / unseen106% / 85%90% / 95%
Policy minus receiver on unseen questions, points [95% CI]2.28 [−0.00, 4.44]3.24 [1.62, 4.86]
Mean saving per query, ms [95% CI]319 [261, 372]419 [362, 474]
Label-light variant: labels, coverage, gain kept (development)12, 45.0%, 106% (ε = 0.01)1, 40.8%, 100% (ε = 0.02)

The three quantities move separately. The change rate is what the test certifies. The gain kept depends on whether the skipped changes would have been corrections. The saving depends on how long the helper takes to generate. The label-light threshold trades saving for gain: on OBQA it lowers the saving from 319 to 218 ms.

W1: scope. Agreed. The revised abstract names MMLU-Pro (one of the three multiple-choice benchmarks) and the cross-family pairs; §6 calls the SQuAD result an existence result and explains the GSM8K fallback. We also ran a three-agent experiment in which a verifier holds information the answerer lacks: with the procedure frozen in advance, the certified policy skips the verifier on 40% of 2,000 sealed questions and keeps 12.55 of its 12.80-point gain while cutting mean latency by 483 ms (General response, part 2). Two further experiments extend the scope to larger models and batched serving: with a 32B helper and receiver, the certified policy skips the helper on 68% of held-out questions and keeps the whole gain, and with early return in static batches of 1 to 32, skipping lowers mean request latency by 35 to 67% (General response, part 3).

We hope the reorganized evidence and the new experiments address your concerns, and we would be glad to clarify anything further during the discussion.

AuthorsOpenReview ↗

A note on our general response

We thank you again for your constructive review. Your questions regarding the paper's primary takeaways and scope significantly informed the framing of our revision.

We have provided a general response to all reviewers ("Response to common concerns and summary of major revisions"). This response explains our choice of evaluation setting and what the paper tests beyond it (addressing W1), articulates our core contributions across four key findings (addressing W2 and Q1), and summarizes the major updates to the manuscript.

We would be glad to clarify any further points as needed.

Thanks you for the detailed response (and the general response), which has addressed most of my concerns. I still expect the authors to further refine their manuscript, especially on the presenting pipeline, i.e., showing their central claims more clearly with in a well-formed way. I will increase the rating in terms of the dedicated response.

We thank you for the thoughtful follow-up. We agree that the manuscript would benefit from presenting its central claims more directly. In the final version, we will restructure the results section to open each subsection with the specific finding it supports, immediately followed by the empirical evidence. Furthermore, we will include a brief overview mapping our four core findings to their respective experiments, ensuring the connection between our claims and the evidence is immediately apparent. Thank you again for your constructive engagement throughout the review process.

Reviewer_Tj4gOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a confidence-based dynamic message-skipping method for multi-agent LLM systems. It uses statistical hypothesis testing to control the rate of answer changes caused by skipping communication, thereby reducing inference latency.

Critical strengths
  1. The paper evaluates different model scales, communication mechanisms, and multiple benchmarks.
Critical weaknesses
  1. All certified C2C communication paths achieve lower accuracy than receiver-only inference, raising the question of whether directly removing the helper would be more reasonable than dynamically skipping its messages. Moreover, the helper and receiver generally have access to the same input information, lacking the complementary knowledge, tool use, and task specialization commonly found in realistic multi-agent systems.

  2. The proposed method mainly applies existing confidence estimation, Learn-Then-Test, and binomial hypothesis testing techniques to message skipping, with limited algorithmic novelty.

Questions for authors
  1. Can you add more comparisons with stronger single-agent models ?

  2. The experiments primarily focus on two-agent systems, single-round communication, and multiple-choice tasks, making it unclear whether the method generalizes to more complex systems.

5 comments
AuthorsOpenReview ↗

Response to Reviewer Tj4g

Thank you for the review. We agree with two of your points and address the others with results from the paper and the revision.

W1: certified C2C paths are less accurate than the receiver alone. Agreed. §4.2 states this, and an operator with labels should retire such a path. The revision presents these certificates as identifying redundant paths. The evidence that skipping keeps useful collaboration comes from text paths that help (Table 2; general response). With a Llama-3.1-8B receiver, the certified policies keep an estimated 85% and 95% of text's gain on unseen questions while skipping 59% and 68% of them. Removing the helper there would cost 2.7 and 3.4 points. With a 32B helper and a 32B receiver, removing the helper would cost 2.55 points [0.94, 4.30] on held-out questions; the certified policy keeps all of it while skipping the helper on 68% of them (General response, part 3).

W1: the helper and receiver see the same input. This is true of the main settings. One extension gives the helper information the receiver lacks. In Text+fact (App. G.2), the helper sees the science fact behind each OBQA question, and the receiver sees it only through the message. Text then raises development accuracy from 617 to 670 of 742; on the 744 held-out questions the deployed policy skips 75.0% and keeps 82% of the gain (Table 2). A stronger helper, whose messages change 13 to 21% of the receiver's answers, is not certified: no threshold passes, as the frequency and predictability of its answer changes predict (App. D). We tested this setting directly on a three-agent system with distinct roles and asymmetric information: a background agent, a verifier that alone sees the supporting facts, and an answerer that decides after the first message whether to wait for the verifier. The verifier raises sealed-test accuracy by 12.8 points and changes 16% of answers, but predictably: with the procedure frozen in advance, the certified policy skips it on 40% of questions and keeps 12.55 of the 12.80 points (General response, part 2). Tool use is not covered, and §6 now says so.

W2: algorithmic novelty. Agreed: we use Learn then Test with an exact binomial test on answer disagreement, within the risk-control framework that CALM also uses (§5). The contribution is what the test reveals about communication: (i) oracle headroom, a common argument for keeping several paths, arises largely without answer-specific content (§4.1); (ii) whether a message can be skipped is set by how often and how predictably it changes the answer, not by its accuracy, and two label-free quantities predict it (§4.2, App. D); (iii) where a message helps, skipping keeps most of its gain (§4.3); (iv) a certified skip need not pay (§4.4). The revised §5 states the relation to CALM precisely.

Q1: stronger single models. New Table 15 (Appendix E) reports them. On OBQA development questions, Qwen3-8B alone answers 83.2%, Qwen2.5-7B alone 86.3% (scored by the argmax over option-label probabilities, not by free generation), and the pair with text 86.9%; on ARC, 89.6%, 87.6%, and 91.6%. At 32B, on held-out OBQA questions, the receiver alone answers 90.3%, the helper alone 91.3%, and the pair with text 92.9% (General response, part 3; Appendix N). We do not claim that collaboration beats a stronger single model. The question we study is narrower: given a deployed system that communicates, when can it skip a message without changing its answers?

Q2: beyond two agents, one round, and multiple choice. The paper also covers an extractive task (SQuAD, certified) and a numeric one (GSM8K, which falls back because a number restated after a derivation carries no residual uncertainty; App. G.4). The three-agent experiment above extends the setting beyond two agents, and its certified policies meet every accuracy criterion on a sealed test. Multi-round communication and open-ended generation remain open, and §6 says so.

We would be glad to clarify any of these points further.

There are concerns remain:

  1. All certified C2C paths are less accurate than receiver-only inference, and the authors fail to demonstrate why dynamically skipping messages is preferable to simply removing the helper agent.

  2. The new three-agent experiment remains a single-round multiple-choice pipeline, with no evaluation of tool use, iterative communication, or complex collaborative workflows.

  3. The proposed method primarily combines existing confidence estimation, Learn-Then-Test, and binomial testing techniques without introducing a substantial new algorithmic contribution.

  4. The additional single-agent comparisons use inconsistent evaluation protocols and do not control computational budgets, making the claimed value of collaboration difficult to establish.

  5. he method fails to certify any threshold on GSM8K, raising concerns about its applicability beyond multiple-choice and extractive QA tasks.

  6. The additional confidence probe incurs computational overhead on every query, while the latency savings for C2C are marginal and inconsistent.

These issues concern are important to the paper. I think the paper remains below the acceptance threshold and will maintain my score.

AuthorsOpenReview ↗

Response to Reviewer Tj4g's follow-up (1/2)

Thank you for the follow-up. Points 1, 2, 3, and 5 return to concerns raised in the review and answered in our reply (W1, W2, Q2); points 4 and 6 are new. Three of the six (1, 5, and 6) concern results that our abstract itself reports.

1. Skipping versus removing the helper. In every certified C2C setting the receiver alone was more accurate, as our abstract and §4.2 state, so removing the helper is reasonable there when accuracy is the objective. Many deployments instead must preserve the existing system's answers, a standard objective at top venues: speculative decoding (Leviathan et al., ICML 2023) accelerates a model without changing its output distribution, CALM (Schuster et al., NeurIPS 2022) certifies consistency with the full model rather than accuracy, and positive-congruent training (Yan et al., CVPR 2021) argues that avoiding new errors on inputs the old model got right can matter as much as reducing overall error. For such deployments, removal is not a safe substitute: fixed receiver-only execution fails the 5% answer-change test in all 14 OBQA, ARC, and large-pair MMLU-Pro settings (Figure 4a). Where the helper raises accuracy, removing it costs 2.7 and 3.4 points (Llama-8B receiver, unseen questions), 2.55 points (32B pair, held-out questions), and 12.8 points (three-agent system, sealed test). The certified policies skip the helper on 59%, 68%, 68%, and 40% of those questions and keep an estimated 85%, 95%, 100%, and 98% of the gain (Table 2; Appendices M and N).

2. Scope: single-round, multiple-choice evaluation. The three-agent experiment addresses the concern in your review that the helper and receiver see the same input: the verifier alone holds the supporting facts. It is single-round and multiple-choice. Single-round evaluation centered on multiple-choice benchmarks is standard in recent top-venue work on LLM communication and routing:

  • C2C (Fu et al., ICLR 2026), the method we study, reports its main results on four multiple-choice benchmarks with one message from helper to receiver; our OBQA and ARC benchmarks come from that evaluation (§3).
  • Activation communication (Ramesh & Li, ICML 2025) passes information between agents in a single transfer, which the authors describe as "not an iterative approach", and evaluates on five MMLU subjects among its reasoning benchmarks.
  • RouteLLM (Ong et al., ICLR 2025) routes each query to one model and evaluates on MMLU, MT Bench, and GSM8K.

Multiple choice is also a common way to obtain finite-sample guarantees for LLM outputs, because answers can be compared exactly: Ye et al. (NeurIPS 2024) convert all their tasks to multiple choice to apply conformal prediction, and KnowNo (Ren et al., CoRL 2023) casts each planning step as a multiple-choice question for the same purpose.

Tool use, multi-round communication, and open-ended generation are therefore not requirements at top venues. C2C, activation communication, RouteLLM, and Trust or Escalate (Jung et al., ICLR 2025) use neither tools nor multi-round communication, and C2C's main results and Ye et al. involve no open-ended generation. Beyond this common setting, the paper adds an extractive task (SQuAD, certified), a numeric task (GSM8K, a prespecified check reported as a fallback), and the three-agent system with asymmetric information.

These three settings are outside the paper's claims, and §6 (Discussion and limitations) lists this scope as a limitation: no setting uses tools or several rounds, and the tasks are multiple choice plus one extractive setting. We state them so that readers know where the guarantees apply.

References are listed in part 2.

Reviewer_Hmu9OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a receiver agent in a collaborative multi-agent system can skip the message sent by a helper agent, where the message is either natural-language text or the helper's KV cache. The goal is to reduce the latency of waiting for the message without noticeably changing the final answer. The proposed approach reads the receiver agent's confidence without the message and skips the message when the receiver is confident enough to answer alone. The skipping threshold is certified with a finite-sample statistical test, so that with high probability the fraction of skipped answers that change stays within a preset tolerance. The procedure needs only unlabeled data, and if no threshold passes, the system keeps communicating as before. The method is evaluated mainly on Qwen model pairs and multiple-choice benchmarks, and the paper further examines whether oracle headroom shows that messages are useful and how much latency skipping actually saves.

Critical strengths

CS1. The paper addresses a practical problem in collaborative multi-agent systems: a receiver agent that waits for a helper agent's message incurs added latency, so when can the message be skipped? The paper also shows that oracle headroom, commonly used to motivate keeping several paths, is weak evidence of message utility, because control inputs without answer-specific content give comparable headroom. This finding is a direct caution for work that relies on headroom to justify routing or multi-agent collaboration.

CS2. The experiments are solid. Each frozen policy is tested out of sample on questions never used before, every setting is reported whether it is certified or falls back, and the code is released together with per-question confidence scores and routes. This lets readers see clearly under which conditions the method works and under which it does not.

Critical weaknesses

CW1. The certificate bounds answer changes, not accuracy, so it cannot guarantee that a helpful path's gain is preserved.

The paper certifies how often skipped answers change relative to the deployed reference, while the decision to skip is made without seeing the message, and the paper acknowledges that a Text message that overturns a confident answer is invisible to the probe. Among the certified settings, the C2C paths themselves are less accurate than the receiver agent alone. For the helpful Text paths, only the large pair is certified, and κα exceeds Text's gain over the receiver agent, with the large/OBQA/Text policy losing accuracy against the reference out of sample. A helpful path therefore still needs an accuracy check with labels.

CW2. The confidence score (ProbeMax) that decides whether to skip a message may be affected by the model's prior knowledge of the benchmarks.

In the experiments, the large pair can skip both messages on every benchmark, while the small pair can skip neither. For the Qwen receiver agents, a large share of the skipped development questions have zero uncertainty, and the OBQA data are all drawn from its official training set. A substantial part of the skipped questions may therefore come from the model's memorization of the benchmarks. Since all models are at most 8B and no closed commercial model is included, it is difficult to judge whether these results would hold on unlabeled deployment traffic and on larger models.

Questions for authors
  1. Section 4.3 shows that labels on a few skipped disagreements bound the accuracy loss, but this analysis is post hoc and is shown only for three of the helpful Text paths. Could this procedure be prespecified and evaluated on all helpful paths, including the medium pair's more accurate Text path, which is not certified, with the gain kept reported on unseen questions? If a few labels reliably preserve the gain of a message, my concern that the certificate does not preserve a helpful path's gain would be largely resolved.
  2. Section 6 states that the Llama-8B receiver agent has no zero-uncertainty predictions, yet is certified and holds out of sample. Does this rule out an effect of the model's prior knowledge of the benchmarks on the certification outcomes?
  3. Section 4.2 states that the certificate covers the skipped questions as a whole, not each subgroup, and that on MMLU-Pro, 12 of 72 skipped engineering C2C answers change. Using the existing calibration split, could certification be run separately for each MMLU-Pro discipline, reporting how many disciplines are certified on their own and how much coverage is lost as a result?
  4. All models are at most 8B, and latency is measured at batch size one, the only mode the C2C release supports. Since the Text path does not depend on C2C, could the certification outcomes and latency savings of the Text path be evaluated on larger open models or closed commercial models (such as GPT or Claude), and under batched inference? If the method still saves substantial latency under these conditions, which are closer to real deployment, it would clearly improve my assessment of the paper's practical value.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Hmu9 (1/2)

Thank you for the careful review, and for recognizing the headroom control and the out-of-sample reporting as strengths. Your questions led to the main new analyses and experiments in the revision. In brief: with the rules fixed in advance as Q1 asks, the primary policies pass a one-point lost-correction bound on a new three-agent system and on a 32B pair, using 40 and 21 calibration labels, and keep 98% and 100% of the gain (point estimates); skipping also saves time under batching and at 32B (Q4); and the per-discipline analysis (Q3) is now in the main text.

CW1 / Q1: the certificate bounds changes, not accuracy. We agree, and the revised §4.3 states it directly. Two clarifications of scope:

  • Helpful text paths are certified beyond the large Qwen pair. With a Llama-3.1-8B receiver, text raises development accuracy by 4.4 points (OBQA) and 7.0 points (ARC). Both policies are certified and pass held-out tests whose pass rule was fixed in advance (Table 3). On unseen questions they keep 85% and 95% of the gain while skipping 59% and 68% of questions (Table 2; general response).
  • As you note, κα exceeds the gain in the three large-pair Qwen text settings. On unseen questions, κ̂α is well below the gain only for Text+fact (3.75 vs. 8.06 points; nominal certificate) and close to it for Llama-8B (2.96 vs. 2.69 on OBQA; 3.39 vs. 3.41 on ARC). The retentions in Table 2 are therefore measured, not guaranteed.

On the label-light certificate: App. E evaluates all six helpful text paths (Tables 13–14); three of them meet its counting rule (coverage of at least 20% and ε at most half the gain). The two ARC paths are infeasible at ε = 0.01 because their 448 calibration questions fall short of the 688 the test needs, and large/MMLU-Pro's gain (1.40 points) is below 2ε. On the 744 held-out OBQA questions, the three counted policies lose at most two reference corrections each (95% upper bounds 0.75 to 0.97%).

You are right that this analysis was post hoc. We therefore froze the procedure before generating any calibration output (prompts, score, candidate thresholds, ε ∈ {0.01, 0.02}, the label rule, and the pass rules on sealed questions) and applied it to a new three-agent system on QASC. On 2,000 sealed questions, the ε = 0.01 policy loses 5 corrections (upper bound 0.58%), skips 40% of questions, and keeps 12.55 of the 12.80-point gain (General response, part 2). We then ran the same frozen procedure on a 32B helper and receiver pair on OBQA. On 744 held-out questions, evaluated once, the ε = 0.01 policy loses one correction (upper bound 0.75%), skips 68% of questions, and keeps the whole 2.55-point gain (General response, part 3).

On OBQA, the medium pair's text path fails the change-rate test at every threshold (AUROC 0.787), so the joint certificate cannot include it. Run on its own, the lost-correction test does not rescue it either: after labels on 36 calibration disagreements, no threshold passes at ε = 0.01 or 0.02, so the system keeps communicating. We report this as a post-hoc analysis.

Other helpful text paths. The small- and medium-pair text paths fail the change-rate test, so the joint certificate cannot include them; on medium/OBQA, the path you named, the loss-only test also falls back (above). At ε = 0.02, Llama-8B/ARC certifies a two-point bound with one label (Tables 13–14). In the two prospective tests, both ε policies pass the held-out change-rate and lost-correction tests (General response, parts 2 and 3).

AuthorsOpenReview ↗

Response to Reviewer Hmu9 (2/2)

CW2 / Q2: prior knowledge of the benchmarks. Your factual points are correct. 60 to 92% of skipped Qwen development questions have zero uncertainty, and our OBQA splits come from the official training split: its test split has only 500 questions, and we needed 4,208 for fit, calibration, and development. The Llama-8B result shows that certification does not require zero uncertainty. It does not rule out contamination, and the revision says so. On model scale, the revision adds a 32B pair (General response, part 3).

The revised §6 separates two questions. Validity: the certificate holds for traffic exchangeable with the calibration questions, whether or not the model has seen them. Transfer: traffic the model has not seen may be less confident and skipped less, so the threshold should be recertified on deployment traffic. Two results bear on transfer. On the sealed ARC-Challenge test split, the questions below the frozen thresholds do not differ significantly from the training split (Fisher p = 0.65 and 0.68). With the options rotated, which changes the input the model may have memorized, the five frozen large-pair policies still change only 1.7 to 2.6% of skipped answers, all with 95% upper bounds below 5% (Table 28).

Q3: per-discipline certification. This analysis is in Appendix C (p. 23 of the submission). Run within each of the 14 MMLU-Pro disciplines, the test certifies three disciplines for text (biology, economics, engineering) and three for C2C (health, other, psychology). With δ also divided by 14, it certifies none for text and one for C2C. Skipping only within disciplines certified on their own lowers overall development coverage from 40.5% to about 11% for both references; with δ also divided by 14, it falls to 0% for text and 3.7% for C2C. The revision adds this to §4.2 and Table 10: the pooled certificate covers the skipped set as a whole, and subgroup guarantees cost coverage.

Q4: larger and closed models, batching. We added a certified 32B setting and a static-batch latency evaluation (General response, part 3; Appendices N and O). Larger models. With a 32B helper and a 32B receiver on OBQA, the procedure frozen in advance yields a policy that skips the helper on 68% of held-out questions, changes two answers, keeps the whole 2.55-point gain, and cuts mean latency by 56%. Batching. We batched the probe and replayed the four deployed text policies in batches of 1, 8, and 32. With early return, skipping lowers mean request latency by 310 to 1,728 ms (35 to 67%) in all twelve settings. Closed models. For a Text helper, the method needs only the helper's text, so a closed helper fits the same algorithm once the policy is recalibrated with it. A closed receiver must expose enough option-label probabilities at the probe position to compute the score, which generated-token or top-k probabilities do not always provide, and C2C needs both access to the helper's KV cache and a receiver that can consume it. We did not run closed models; §6 states these conditions.

We hope these results address your concerns, and we would be glad to clarify anything further during the discussion.

Thank you for the detailed response and the substantial new experiments. The prospective tests on the three-agent system and the 32B pair, together with the batched serving results, address the main concern in my review. I appreciate that the revision now separates the validity of the certificate from its transfer to traffic the model has not seen.

The revision clearly strengthens the paper, and I appreciate the effort that went into it. Weighing the overall contribution and presentation, I still consider the paper as slightly below the acceptance threshold, so I will keep my original score.

Comments on the paper

6 comments
AuthorsOpenReview ↗

General response (3/3): larger models and batched serving

Two further experiments answer the questions on model scale and batching (Hmu9 Q4, Tj4g Q1, 5yjK W1). Both are reported in full in the revised PDF (Appendices N and O).

A 32B pair. We scaled both models: Qwen2.5-32B-Instruct as helper and Qwen3-32B as receiver, on the text path, on OBQA with the paper's splits (2,100 fit, 1,366 calibration, 742 development, and 744 held-out questions). The paper already uses OBQA, so this experiment extends model scale, not data. The procedure is that of part 2 (same score, tests, label rule, and pass rules). It was frozen before any calibration output, and the held-out questions were evaluated once. Calibration used gold labels on 21 of the 1,366 calibration questions. On the held-out questions, text raises accuracy from 90.32% (receiver alone) to 92.88%, a gain of 2.55 points [0.94, 4.30]:

PolicySkip helperChanged / skippedLost corrections / 744Gain over receiver alone, points [95% CI]Dev-panel latency saving, ms [95% CI]
Joint, ε = 0.0168.4%2 / 5091 (upper bound 0.75%)2.55 [0.94, 4.17]1,377 [1,219, 1,533]
Joint, ε = 0.0279.8%6 / 5944 (upper bound 1.37%)2.28 [0.67, 3.90]1,525 [1,377, 1,667]
Label-free84.9%9 / 63262.15 [0.67, 3.76]1,618 [1,485, 1,750]

The primary policy (ε = 0.01) meets every criterion fixed in advance. It skips the 32B helper on 68% of questions and changes two answers; its accuracy equals that of always sending text (92.88%), so it keeps the whole gain, and mean latency falls from 2,441 to 1,064 ms (56%). All three change-rate upper bounds are below 5% (at most 2.69%), and both joint policies keep the lost-correction bound below their ε. Text changes the receiver's answer on 45 of the 744 held-out questions (6%); the primary policy skips the helper on 68% of questions at the accuracy of always sending text.

Batched serving. We replayed the four deployed text policies (the large pair and the Llama-3.1-8B receiver, on OBQA and ARC) with their thresholds unchanged, on the full development sets (742 and 299 questions), in static batches of 1, 8, and 32 in Hugging Face Transformers. In each batch the probe runs for all requests, the skipped requests are answered and returned, and the helper and receiver then run for the rest. The arms alternate on the same node, and each request's completion time is measured. Mean saving per request against the always-communicate reference, in ms [95% CI]:

PolicyBatch size 1Batch size 8Batch size 32
large / OBQA421 [402, 441]606 [565, 647]1,061 [1,000, 1,120]
large / ARC581 [558, 604]900 [852, 948]1,728 [1,585, 1,873]
Llama-8B / OBQA310 [289, 331]421 [382, 461]797 [727, 866]
Llama-8B / ARC409 [372, 445]650 [590, 717]1,427 [1,275, 1,592]

With early return, skipping lowers mean request latency in all twelve settings, by 35 to 67% of the reference latency, and the absolute saving grows with batch size. Among skipped requests, 1.6 to 3.3% of answers differ from the reference at the same batch size. If the whole batch must instead return together, the saving interval stays above zero in ten of the twelve settings (Appendix O).

AuthorsOpenReview ↗

General response (2/3): a prospective test on a new three-agent system

We fixed the full procedure before generating any calibration output and evaluated it once on a sealed test. The system and the dataset are new to this work. Of QASC's 9,051 questions after deduplication, 8,000 are split into 2,000 fit, 3,000 calibration, 1,000 development, and 2,000 sealed questions. Agent A (Qwen2.5-7B-Instruct) writes background knowledge. Agent B (Qwen2.5-7B-Instruct) alone sees the two supporting facts; it is instructed to check and extend A's note without saying which choice is correct. The receiver C (Qwen3-8B) reads A, and the probe decides whether C waits for B.

Calibration fixed three policies: the label-free change-rate certificate, and the joint certificate at ε = 0.01 (primary) and ε = 0.02, which used gold labels on 40 of the 3,000 calibration questions. On the sealed test, B raises accuracy from 63.80% (A+C) to 76.60% (A+B+C), a gain of 12.80 points [11.30, 14.40]:

PolicySkip BChanged / skippedLost corrections / 2,000Gain over A+C, points [95% CI]Dev-panel latency saving, ms [95% CI]
Joint, ε = 0.0139.9%5 / 7985 (upper bound 0.58%)12.55 [11.10, 14.15]483 [365, 604]
Joint, ε = 0.0249.6%15 / 99114 (upper bound 1.17%)12.10 [10.65, 13.65]618 [501, 732]
Label-free54.7%29 / 1,0942611.55 [10.15, 13.05]697 [581, 818]

Every criterion fixed in advance is met. The change-rate upper bound is below 5% for all three policies (at most 3.78%), both joint policies keep the lost-correction bound below ε, every policy keeps more than half of the reference gain at 95% confidence, and every latency saving has a lower end above zero. The primary policy skips B on 40% of questions, keeps 12.55 of the 12.80 points, and cuts mean latency from 2,056 to 1,573 ms (23.5%). B changes 16.4% of C's answers (328 of 2,000), yet only 5 of these changes fall among the 798 skipped questions: the probe locates where B matters, which is the predictability that §4.2 identifies as what makes a message skippable.

Latency comes from the end-to-end panel fixed in the protocol: 128 development questions, A and B generating live, the policy and the reference alternating per request at batch size one, with cold requests kept. Nearly all of the saving comes from skipping B (about 1.3 s each time); the probe costs about 40 ms per request. New Appendix M of the revised PDF reports the frozen protocol, every calibration test, the latency panel, the controls the protocol fixes, and all deviations.

AuthorsOpenReview ↗

General response (1/3): useful collaboration, the role of labels, and what the paper claims

We thank the Area Chair and the reviewers. They agree that the formulation is clear (5yjK), that the headroom control is a useful caution for routing and multi-agent work (Hmu9, 5yjK), and that the reporting and ablations are thorough (Hmu9, 5yjK). The shared concern is whether certified skipping preserves useful collaboration or only identifies redundant paths. We answer it with results the original text under-reported, three new experiments, and a revised presentation.

New experiments in this revision (changed text is blue in the revised PDF; table numbers in our replies refer to it):

  1. A prospective test on a new three-agent system (Appendix M; part 2). With the certification rules fixed before calibration, the certified policy skips a verifier that alone holds the supporting facts on 40% of 2,000 sealed questions and keeps 12.55 of its 12.80-point gain.
  2. A 32B helper and receiver pair (Appendix N; part 3). The same procedure skips the helper on 68% of 744 held-out questions and keeps the whole 2.55-point gain.
  3. Batched serving (Appendix O; part 3). With early return in static batches of 1, 8, and 32, skipping lowers mean request latency by 35 to 67% in all twelve settings.

1. Where text helps, skipping keeps most of its gain on unseen questions. Four deployed text policies (two with nominal certificates) improve accuracy over the receiver alone and have unseen questions available (Table 2, new in §4.3):

PolicyUnseen setSkippedChangedReceiver aloneAlways textPolicyText − receiverPolicy − receiverKept
Llama-8B / OBQA / Textheld-out OBQA (744)59.1%3/44080.3883.0682.662.69 [0.40, 4.97]2.28 [−0.00, 4.44]85%
Llama-8B / ARC / TextARC test (1,172)67.7%15/79481.3184.7384.563.41 [1.71, 5.12]3.24 [1.62, 4.86]95%
large / OBQA / Textheld-out OBQA (744)80.0%12/59583.0686.9686.023.90 [1.88, 5.78]2.96 [1.21, 4.57]76%
large / OBQA / Text+factheld-out OBQA (744)75.0%16/55883.0691.1389.658.06 [5.91, 10.22]6.59 [4.57, 8.47]82%

Accuracies in %; gains in points with paired-bootstrap 95% CIs. Three of the four retained gains exclude zero; for Llama-8B/OBQA the interval includes zero (lower end −0.003). Kept is a point estimate. On the 128-question development panels, these policies save 319 to 517 ms per query. The C2C certificates, by contrast, identify paths less accurate than the receiver alone; the revision presents them as a redundancy result, not as preserved collaboration.

2. A few labels can certify the accuracy loss. The change-rate certificate bounds the accuracy loss by κα (Eq. 2), which protects the gain only when κα is below it. On unseen questions, κ̂α is well below the gain only for Text+fact (3.75 vs. 8.06 points), whose certificate is nominal, so all four retentions above are measured, not guaranteed. Because accuracy can change only where the receiver alone and the reference disagree, labels on the lowest-uncertainty skipped disagreements certify the loss directly. In App. E, 7 to 12 such labels certify a loss of at most one point for three of the six helpful text paths (two nominal) while keeping 98 to 106% of the development gain. To test this prospectively, we froze the full procedure (prompts, score, tests, label rule, and pass rules) before generating any calibration output and ran it on the three-agent system and on the 32B pair. Both pass, with 40 and 21 calibration labels (parts 2 and 3).

3. What the paper claims. We do not claim a new statistical test; we share the risk-control framework of CALM (§5). Our claims concern communication: (i) oracle headroom is weak evidence that messages help; (ii) whether a message can be skipped is set by how often and how predictably it changes the answer, which two label-free quantities predict; (iii) where messages help, skipping keeps most of the gain, with a label-light certificate when the label-free bound is too loose; (iv) a safe skip need not be a profitable one.

Other revisions. Abstract and contributions rewritten around these claims; new Table 2 and two paragraphs in §4.3 on helpful paths; per-discipline MMLU-Pro coverage in §4.2 and Table 10; a paragraph in §5 on the relation to CALM, with the last column of Table 34 (formerly 31) removed; §6 extended on information symmetry, model scale, closed models, batching, and pretraining exposure; Appendix E adds the medium-pair lost-correction test and single-model baselines (Table 15).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.