acceptodds
Withdrawn from ICLR 2027

When Does More Information Pay Off? Effective Cost in LLM Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

More context is not always cheaper. Extra information can reduce how much an LLM needs to generate, but a longer prompt also makes each cached decoding step more expensive. We study when the saved computation outweighs the cost of carrying additional context. We define effective cost as the smallest physical budget needed to reach a common target, such as a desired response behavior or task score. Under regularity conditions, one representation provides at least as much attainable utility as another across budgets and decision objectives exactly when it reaches every response behavior attainable from the other at no greater effective cost. We then derive when a larger context is worthwhile and develop rules for jointly choosing the input context and the amount of generation, including the cost of constructing or replacing the cached state. In controlled routing, useful extra information lowers the cost of reaching a common numerical success threshold despite a longer prompt. On 576 held-out MuSiQue questions with supplied intermediate answers, jointly choosing how many paragraphs to retain and how long to generate improves mean answer F1 by 0.040 over choosing one evidence scope per budget, averaged across three prespecified profiled GPU-time budgets. Finally, rebuilding an active KV cache from a shorter prompt yields decoding savings that exceed the measured rebuilding cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Inference Efficiency, Test-Time Compute, Context Selection, Resource-Constrained Inference, KV-Cache Management

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xgysOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The hypothesis of this paper is: giving a model a richer input context can reduce the output decoding steps needed to solve a task. The paper introduces effective cost, and quantifying the tradeoff between footprint and work. The paper widely discussed the hypothesis in the experiment part, from whether useful information works, to the trade-off part, and discussed rebuilding an active cache.

Critical strengths
  1. The author deep dived into the hypothesis, from whether useful information works, to the trade-off part, and discussed rebuilding an active cache. The whole storyline is quite complete.
  2. The core question is well-motivated: separating per-step KV-cache footprint cost from required output steps K gives a clear view of context-generation trade-offs.
Critical weaknesses
  1. In 4.1, there exists three setups: compact, neutral rich and useful rich. However, the comparing between neutral rich and useful rich are unfair, since they are comparing against two different criteria. One only need start and end, the other one requires full-path. Another thing is, when calculating the inference time, the author only considered "first token produced during prompt processing", but didn't consider the time of reading tokens (prefill). The comparison between useful-rich and compact to show that useful-rich makes more sense doesn't stand.
  2. It seems that in 4.2, the prefill time was taken into consideration again. It would be better to discussed all with prefill and w/o prefill.
  3. Before running the question, the model won't know the optimized K. It looks like a retrospective replay.
Questions for authors

In weekness.

Reviewer_bScHOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes a cost modeling framework for LLMs, based on the precept that there can be trade offs between the size and utility of its context (e.g. K/V cache) for a given downstream task. The claimed key contribution of the paper is "effective cost for representations, which specify how task information is supplied and retained. Effective cost is the least budget needed to reach a target using the allowed continuation procedures."

The submission is unclear, poorly structured and poorly motivated, appears to have been mostly AI-generated without proofreading. The experiments poorly support the claims of the paper and do not refer to previous works from the literature. My recommendation is rejection.

Critical strengths

None

Critical weaknesses

For a paper claiming to model and optimize the footprint/utility in LLM inference, it does not explain what the cost of each operation is, either in compute/memory complexity, power footprint, etc. Also does not mention what the incremental cost of e.g. a longer K/V cache represents. Does not define any key term such as "representation", "target", and seems to ignore that LLMs are stochastic. Does not explain under which aspect their work is different than previous literature and does not ground/compare their results in/with previous literature.

Questions for authors

In one sentence that a human can understand, what problem does this paper solve and how does it solve it better than the previous literature?

Reviewer_64weOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces effective cost, defined as the minimum physical inference budget required to achieve a target response behavior or task utility. It develops theoretical conditions for trading off context footprint against required continuation work, including the costs of acquiring or rebuilding KV caches. Based on this framework, the authors propose jointly selecting evidence scope and generation length under a fixed budget. Experiments across different benchmarks demonstrate the proposed tradeoffs.

Critical strengths

Interesting problem formulation. The paper considers the interaction between information availability and inference computation, rather than optimizing prompt length or generation cost independently. Controlled experimental analysis. The authors independently profile execution costs and evaluate different information–computation tradeoffs using held-out data and additional sensitivity analyses.

Critical weaknesses

It relies on unrealistic assumptions about information availability and usefulness. The framework assumes that useful information and its impact on required generation work can be estimated, without explaining how to identify such information or why adding it should reduce output length. The experiments largely depend on predefined shortcuts, ground-truth intermediate reasoning answers, and predetermined compact representations. Missing important baselines and insufficient justification for KV-cache rebuilding. The rebuilding experiment does not compare against existing alternatives such as query-aware sparse attention or KV eviction/compression, which are essentially more general ways of KV compaction. Limited practical relevance and insufficient evidence of meaningful serving improvements. The experiments use small models, batch sizes of one or four, and PyTorch SDPA rather than optimized serving frameworks. The primary QA gains depend on extremely short generation checkpoints and selected budget boundaries, while the evaluation does not establish significant improvements under realistic KV cache management, continuous batching, or production serving workloads, making cost formulation less useful.

Questions for authors

see weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.