- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates when an LLM receiver can skip a helper’s message while usually preserving the answer of the always-communicating system. The proposed method uses receiver confidence and finite-sample testing to choose a skipping threshold, then measures net latency savings after accounting for the probe. The empirical results cover text and KV-cache communication, with successful certification in several multiple-choice settings and one extractive setting.
Critical strengths
S1. I have to admit that the problem formulation is clear. Preserving a deployed system’s behavior without requiring correctness labels is a useful goal. The paper also correctly separates this goal from preserving accuracy and obtaining a net speedup.
S2. The empirical analysis contains useful observations. Specifically, the headroom controls challenge a potentially misleading interpretation of oracle gains, and the text-versus-KV comparison shows why similar skipping coverage need not produce similar latency savings.
S3. The analysis parts are good. The authors include sveral useful ablations which help distinguish stronger evidence from exploratory results.
Critical weaknesses
W1. One of my biggest concerns is the limited experimental coverage, despite the additional benchmarks. Although the MMLU-Pro and the cross-family experiments are useful additions (yet they are not included in the abstract), but the successful results still mostly concern short, parsed answers in a two-agent, single-message setup. Besidse, the main models are at most around 8B with non-thinking mode and the GSM8K score fails to support certification. SQuAD establishes that the approach can work outside multiple choice, but it is one setting selected from eight pilot candidates, not broad evidence for open-ended generation.
W2. My another biggest concern is the takeaway message of this paper. I merit the technical contribution of this paper, i.e., the proposed method is useful in the empirical results. However, the paper is mainly talking about an empirical lesson on evaluating communication, rather than something new and insightful to the audience like me. A rebuttal that clarifies the distinctive contribution that make the work more insightful could move me toward acceptance. Note that a stronger wording alone would not resolve these concerns, but a better formulation of the existing evidence might.
Questions for authors
Q1. What is the main contribution you want readers to take away? Is it a new communication-specific risk formulation, the headroom critique, or an empirical account of when skipping pays? Please make the distinction from CALM and existing risk-controlled routing precise.
Q2. Which existing result is the strongest evidence for preserving useful collaboration? The Llama Text settings seem promising. Can the rebuttal use these results to explain the joint trade-off between answer changes, retained accuracy gain, and net latency, rather than mainly emphasizing certification success?
4 comments
A note on our general response
We thank you again for your constructive review. Your questions regarding the paper's primary takeaways and scope significantly informed the framing of our revision.
We have provided a general response to all reviewers ("Response to common concerns and summary of major revisions"). This response explains our choice of evaluation setting and what the paper tests beyond it (addressing W1), articulates our core contributions across four key findings (addressing W2 and Q1), and summarizes the major updates to the manuscript.
We would be glad to clarify any further points as needed.
Thanks you for the detailed response (and the general response), which has addressed most of my concerns. I still expect the authors to further refine their manuscript, especially on the presenting pipeline, i.e., showing their central claims more clearly with in a well-formed way. I will increase the rating in terms of the dedicated response.
Thank you
We thank you for the thoughtful follow-up. We agree that the manuscript would benefit from presenting its central claims more directly. In the final version, we will restructure the results section to open each subsection with the specific finding it supports, immediately followed by the empirical evidence. Furthermore, we will include a brief overview mapping our four core findings to their respective experiments, ensuring the connection between our claims and the evidence is immediately apparent. Thank you again for your constructive engagement throughout the review process.
Response to Reviewer 5yjK
Thank you for the constructive review, and for recognizing the problem formulation and the headroom controls as strengths. Your two questions shaped the framing of the revision.
Q1 / W2: the main takeaway. In one sentence: a message can be skipped safely when it rarely and predictably changes the receiver's answer, a property that oracle headroom misses and accuracy does not reveal, and whether the skip then pays is a separate, measured question. The results support this in three steps. Headroom arises without answer-specific content, so it cannot tell useful messages from volatile ones (§4.1). Certification follows the disagreement rate and its predictability, not accuracy, and two label-free quantities predict the outcome (§4.2, App. D). Where a message does help, skipping keeps most of the gain and saves time (§4.3–4.4). Two new experiments test this last step with the procedure fixed in advance: a three-agent system and a 32B pair keep 98% and 100% of the gain (General response, parts 2 and 3). The revised abstract and introduction lead with this claim.
Relation to CALM and risk-controlled early exiting. We share their risk-control framework (Learn then Test with a consistency risk relative to a fixed reference) and claim no new test; neither guarantee requires the reference to be more accurate. CALM bounds the expected inconsistency over all inputs; we bound the change rate among skipped questions. Within each decoding step, an early-exit gate reads intermediate states of the computation it may cut short; our gate reads a separate message-free prefill, which in our implementation adds a forward pass to every query, so whether skipping pays must be measured (§4.4). A message also comes from another agent: it may carry no answer-specific content (§4.1) or make the system worse (§4.2), so we measure gain retention separately (§4.3).
Relation to certified routing (Uddin & Bauer, 2026): they certify the correctness of the routed answer, which needs labels; our change-rate certificate bounds agreement with a deployed path that may be less accurate than the receiver alone, and needs none.
Q2: the joint trade-off in the Llama text settings. Agreed. Among the original experiments, Llama-8B/ARC is the clearest case: on the 1,172 test questions, text adds 3.41 points [1.71, 5.12] over the receiver alone; the policy skips 67.7% of questions, changes 1.9% of skipped answers, and keeps 3.24 points [1.62, 4.86]; on the 128-question development timing panel it saves 419 ms per query. The table below collects both settings (see also Tables 2 and 3 of the revision):
The three quantities move separately. The change rate is what the test certifies. The gain kept depends on whether the skipped changes would have been corrections. The saving depends on how long the helper takes to generate. The label-light threshold trades saving for gain: on OBQA it lowers the saving from 319 to 218 ms.
W1: scope. Agreed. The revised abstract names MMLU-Pro (one of the three multiple-choice benchmarks) and the cross-family pairs; §6 calls the SQuAD result an existence result and explains the GSM8K fallback. We also ran a three-agent experiment in which a verifier holds information the answerer lacks: with the procedure frozen in advance, the certified policy skips the verifier on 40% of 2,000 sealed questions and keeps 12.55 of its 12.80-point gain while cutting mean latency by 483 ms (General response, part 2). Two further experiments extend the scope to larger models and batched serving: with a 32B helper and receiver, the certified policy skips the helper on 68% of held-out questions and keeps the whole gain, and with early return in static batches of 1 to 32, skipping lowers mean request latency by 35 to 67% (General response, part 3).
We hope the reorganized evidence and the new experiments address your concerns, and we would be glad to clarify anything further during the discussion.