- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
QUOTA is a token-level router for small-large model collaboration. At each decoding step it scores the SLM's next-token distribution with entropy, maps the score to a percentile using ECDF calibrated only on unlabeled prompts, and lets the LLM emit the next token when the percentile falls in the top-q tail of a user-specified LLM token quota q; after the one-token takeover, routing is recomputed from the updated prefix. Experiments report better quality-latency tradeoffs than previous methods
Critical strengths
- The motivation is clear, supported by formalization and validated by ablation studies. The experimental setup with MR is convincing.
- The training-free pipeline is complete and sample-efficient, offering strong ease of use.
Critical weaknesses
- Some details are unclear. I do not understand how KV cache management is performed in the proposed framework, where the SLM executes most of the tokens and hands over to the LLM at certain positions. At this point, the LLM's KV cache is interrupted, and the tokens executed by the SLM have no cache. How is this situation handled? Does the LLM perform a prefill again here (for the tokens previously generated by the SLM)? Or are the SLM and LLM structurally coupled to ensure that the LLM also saves the KV cache at every token? Additionally, while this paper reports the time cost, the memory pressure brought by the two models and the two KV caches should also be reported in the main results (e.g., peak memory usage).
- Cross-domain generalization of the calibration data. The paper relies on the calibration data to estimate the distribution, which may cause cross-domain distribution shift. For example, if the calibration is done on natural text (e.g. wikitext) but the inference is run on math data, can the LLM participation ratio still stay consistent with the requested budget? The overall model performance may also degrade due to inaccurate estimation of the positions where the LLM is invoked.
- From the main results (Table 1), the proposed method does not show a clear advantage over GlimpRouter. The accuracy is only slightly higher while the latency is also higher. Can the authors clarify this?
Questions for authors
see Weaknesses.