- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The hypothesis of this paper is: giving a model a richer input context can reduce the output decoding steps needed to solve a task. The paper introduces effective cost, and quantifying the tradeoff between footprint and work. The paper widely discussed the hypothesis in the experiment part, from whether useful information works, to the trade-off part, and discussed rebuilding an active cache.
Critical strengths
- The author deep dived into the hypothesis, from whether useful information works, to the trade-off part, and discussed rebuilding an active cache. The whole storyline is quite complete.
- The core question is well-motivated: separating per-step KV-cache footprint cost from required output steps K gives a clear view of context-generation trade-offs.
Critical weaknesses
- In 4.1, there exists three setups: compact, neutral rich and useful rich. However, the comparing between neutral rich and useful rich are unfair, since they are comparing against two different criteria. One only need start and end, the other one requires full-path. Another thing is, when calculating the inference time, the author only considered "first token produced during prompt processing", but didn't consider the time of reading tokens (prefill). The comparison between useful-rich and compact to show that useful-rich makes more sense doesn't stand.
- It seems that in 4.2, the prefill time was taken into consideration again. It would be better to discussed all with prefill and w/o prefill.
- Before running the question, the model won't know the optimized K. It looks like a retrospective replay.
Questions for authors
In weekness.