- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2(−1)
- Presentation
- 2(−1)
- Soundness
- 2
Read review
Paper summary
This paper proposes FORETOKEN, a framework for handling connectivity loss during streamed LLM generation. It combines conformal risk control with speculative decoding and on-device fallback to make risk-aware response admission decisions.
Critical strengths
The paper provides a relatively comprehensive evaluation contained comparisons with multiple baselines, ablation studies, and sensitivity analyses.
Critical weaknesses
I found the paper difficult to follow, especially the explanation of the completion certificate and the certified floor. Figure 1 gives a basic overview, but it does not clearly show how the certifier, cloud verifier, local drafter, and fallback actions work together. The paper also takes some time to clarify that the formal guarantee concerns trace-level outage risk rather than the completion of an entire response.
Proposition 1 applies existing conformal risk control theory to network traces. The analysis in Section 2 shows that signal-based outage prediction is unreliable, which motivates the work, but this observation does not itself constitute a new method. I am not convinced that combining CRC with speculative decoding and fallback provides a sufficiently distinct methodological contribution.
I am not sure what the completion certificate adds to the actual serving process. In Table 2, removing the certificate leaves both mean wait and p99 wait unchanged at 16.2 ms and 445 ms. The latency improvement mainly comes from keeping the local drafter ready. While Section 5.4 shows the risk-control properties of CRC, it remains unclear when this certificate leads to better serving decisions than a simpler system with a pre-warmed local model.
Questions for authors
Section 3.3 uses to measure fallback quality, but the experiments fix . Figure 7 only varies this assumed value. Is there evidence from actual model outputs that acceptance rates reflect the quality of on-device generation across different tasks and contexts?
With 20 calibration traces, a 5% risk budget gives only a one-second certified floor. Under the 10 kbps interruption definition, no positive floor can be certified at the same budget. Would the certificate remain useful on networks with different outage patterns? Could the authors test this using traces from other operators or regions?