acceptodds
Under review as a conference paper at ICLR 2027

CONFORMAL RISK CONTROL FOR CERTIFYING STREAMED GENERATION UNDER CONNECTIVITY LOSS

Abstract

Conformal risk control certifies a threshold with a finite-sample guarantee, but only when the exchangeable unit is plentiful. We study it where the unit is scarce: a drive trace on a cellular network. A language-model reply streamed to a phone on a train is a sequential decision with a hidden hazard. The link can die before the last token arrives, and the serving stack learns of it last. Buffering cannot help, be- cause future tokens do not exist yet. Reacting costs a loss-detection timeout plus a fallback start. We build FORETOKEN, a runtime that admits a response only under a completion certificate. The reply must either finish inside a certified floor on the link’s remaining life, or hand off within that floor to an on-device drafter whose measured speculative-decoding acceptance rate clears a quality gate. The floor is a conformal-risk-control threshold over a grid of horizons with the trace as the exchangeable unit, so its realized per-second violation rate is bounded rather than assumed. On 21 public 5G traces (11.75 active hours, 2,830 replayed responses), FORETOKEN admits 96.7 % of launches with a response-level violation rate of 0.40 % against a 5% budget (trace-clustered 95% interval 0.18 to 0.69 %). It declines 92 launches, of which 92.4 % stall, and it holds p99 excess wait at 445 ms against 1,345 ms for a reactive hybrid. Three findings cut against our own premise. An RSRP-drop precursor covers 29.3 % of hard outages at a 19.5 % matched-control rate, and a learned ranker beats it at every false-alarm budget a phone would run. With 20 calibration traces the finite-sample term, not the hazard, decides what a 5% budget can certify: a one-second floor, eight seconds from 6%, and nothing at all below twenty traces. And when the budget sits at the hazard’s own scale, calibrated-probability thresholds exceed it on up to 14 of 21 test traces while the conformal floor declines to certify. A completion guarantee does not need the outage to be predictable. It needs the hazard on the seconds it admits to be measured, bounded, and paid for.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
conformal risk control, conformal prediction, uncertainty quantification, distribution-free guarantees, calibration, speculative decoding, LLM serving, streaming generation, mobile connectivity, cellular networks

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_2UvVOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2(−1)
Presentation
2(−1)
Soundness
2
Read review
Paper summary

This paper proposes FORETOKEN, a framework for handling connectivity loss during streamed LLM generation. It combines conformal risk control with speculative decoding and on-device fallback to make risk-aware response admission decisions.

Critical strengths

The paper provides a relatively comprehensive evaluation contained comparisons with multiple baselines, ablation studies, and sensitivity analyses.

Critical weaknesses

I found the paper difficult to follow, especially the explanation of the completion certificate and the certified floor. Figure 1 gives a basic overview, but it does not clearly show how the certifier, cloud verifier, local drafter, and fallback actions work together. The paper also takes some time to clarify that the formal guarantee concerns trace-level outage risk rather than the completion of an entire response.

Proposition 1 applies existing conformal risk control theory to network traces. The analysis in Section 2 shows that signal-based outage prediction is unreliable, which motivates the work, but this observation does not itself constitute a new method. I am not convinced that combining CRC with speculative decoding and fallback provides a sufficiently distinct methodological contribution.

I am not sure what the completion certificate adds to the actual serving process. In Table 2, removing the certificate leaves both mean wait and p99 wait unchanged at 16.2 ms and 445 ms. The latency improvement mainly comes from keeping the local drafter ready. While Section 5.4 shows the risk-control properties of CRC, it remains unclear when this certificate leads to better serving decisions than a simpler system with a pre-warmed local model.

Questions for authors

Section 3.3 uses q(α)=αq(\alpha)=\alpha to measure fallback quality, but the experiments fix α=0.60\alpha=0.60. Figure 7 only varies this assumed value. Is there evidence from actual model outputs that acceptance rates reflect the quality of on-device generation across different tasks and contexts?

With 20 calibration traces, a 5% risk budget gives only a one-second certified floor. Under the 10 kbps interruption definition, no positive floor can be certified at the same budget. Would the certificate remain useful on networks with different outage patterns? Could the authors test this using traces from other operators or regions?

Reviewer_XWfgOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper explores LLM generation with speculative decoding in a cloud-offloading scenario where the drafter would run on a local device and the verifier model runs in the cloud. In this framing, they study the problem of estimating if a link is likely to break before the current generation is complete, and if so, attempt to switch to using the draft model to finish the completion. They show that this handoff approach can reduce wait times compared to using the cloud-only approach as it can finish on the local device. The paper uses real network traces but does not model accept rate of speculative decoding.

Critical strengths

The paper addresses an important problem of keeping cloud-backed LLMs to have a hand-off to continue generation when links break.

Critical weaknesses

I found this paper hard to read. I spent multiple iterations re-reading the first 3 paragraphs in the introduction to get a sense of the exact problem being explored here. Many of the section headers do not help the reader make progress through the paper. The notion of a certificate seems like a construction that does not provide a lot of value in the actual problem being studied. Instead, I found this made it harder to unpack the underlying problem being studied.

The main claim in the paper is that if the link breaks, the approach can fall back to the small draft model running on the phone. However, this changes the generation and violates the standard speculative decoding framing. From what I can tell, quality degradation for switching to the draft model is discussed but not measured in the paper. Instead, the paper uses a simulated replay of real network traces but do not model the token generation, accept rate, etc.. itself. The reported quality in table 1, is not end-to-end quality, but instead a ratio of cloud tokens vs device tokens with a quality ranking for each component. However, this quality is a free parameter the paper sweep that has no bearing to actual end-to-end quality.

Questions for authors

Can you validate the fallback using actual language models? What is the the downstream accuracy/quality impact of switching to the draft model mid-generation in this specific scenario?

What is the precise guarantee for a complete response?

I don't really understand this notion of a certificate. What does this give you beyond having a fallback ready?

Reviewer_UUdpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes FORETOKEN, a framework for reliable LLM streaming under intermittent connectivity. It uses conformal risk control to certify a safe connectivity horizon and combines speculative decoding with on-device fallback. Experiments on 21 real 5G traces show a 96.7% admission rate, a 0.40% response-level violation rate, and reduced tail latency compared with reactive baselines.

Critical strengths

Practical problem formulation. The paper addresses an important reliability challenge for streaming LLM applications and provides an interesting application of conformal risk control.

Comprehensive empirical analysis. The evaluation includes multiple baselines, ablations, sensitivity analyses, and an explicit examination of finite-sample limitations.

Critical weaknesses

Limited novelty and theoretical guarantees. The main conformal guarantee follows directly from existing conformal risk control theory. Moreover, the guarantee concerns per-second outage risk rather than response-level completion, making the central certification claim weaker than its framing suggests.

Limited practical validation. Results rely entirely on trace-driven simulation using only 21 traces from one dataset. Key assumptions, including fallback latency, speculative acceptance, and quality, are not validated in an actual deployment. In particular, treating speculative acceptance rate as a quality bound lacks sufficient justification.

Questions for authors

Can the authors provide a formal response-level completion guarantee rather than only a per-second risk bound?

How well does speculative acceptance rate correlate with actual generation quality? Can this be validated using real model outputs?

How robust are the results across different cellular datasets, devices, and real-world deployment conditions?

Reviewer_uVW5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper seems to use conformal risk control technique to tune hyperparameters in a cellular network.

Critical strengths

None

Critical weaknesses

This paper is very hard to read as many undefined terminologies are used in a very confusing way (e.g., "floor"). Besides the writing issue, this paper seems to be a mere application of conformal risk control that tunes hyperparameters in cellular network systems that does not involve any meaningful innovations. Also, while the abstract implies that this work is trying to address the limitation of conformal risk control that requires "plentiful exchangeable units" (though I'm not sure what the authors are trying to refer to through "units"), I cannot find any clear way of resolving such limitation nor even clear description of the limitation throughout the main text.

Questions for authors

In which sense the authors are resolving "plentiful exchangeable units" limitation of conformal risk control in this work? What is the main message of this work? Would the message of this work be generally useful in more broader engineering/machine learning fields? To me, it seems the contribution is very limited to cellular network system without any meaningful innovations hence this work is hard to be appreciated to general readers/participants of ICLR.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.