acceptodds
Withdrawn from ICLR 2027

The Price of Verification in Budgeted Perception: Checking Costs What Looking Costs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

An agent under a per-decision token budget holds a cheap description of each observation and re-reads the observation when a question demands it. We price the half of that bargain that is left unpriced: establishing whether the description is true. On an interface that meters tokens and re-transmits the observation on every call grounded in it, any verifier whose verdict depends on the observation must pay that transmission, so the price of verification relative to the price of the channel is bounded below and tends to one: checking costs what looking costs. Both premises are measured rather than assumed. The observation is most of the bill, identified inside a single run as the difference between a grounded and an ungrounded verdict call, and the measured ratio clears the floor those terms predict on every open backbone we test. That an observation is re-charged at all is audited on every provider endpoint the router will pin, and the audit turns on a distinction the obvious test misses: asked to serve the identical call twice, four of seven backbones have an endpoint that returns the observation as a cache read; asked a different question about the same frame (which is what a verifier does), only two do. So the assumption holds, for the call the theorem is about, on five of seven backbones, and where it fails it fails at the discount our own proposition predicts in advance. That cache then expires within minutes, which is shorter than the deferral a stored description exists to create. Pinning the provider also separates what belongs to the model from what belongs to the vendor: between endpoints of one model the dollar price of an identical call varies twofold and one endpoint tokenises the same image nine per cent shorter, while the ratio the theorem bounds moves by 0.012. A price, however, is only half a verifier. Normalising by how much a verdict actually discriminates separates two quantities that behave quite differently: the nominal ratio is pinned by the interface, moving little across an order of magnitude of scale and uncorrelated with it, while the discrimination varies widely between backbones and is a property of the individual model. The price of a verdict belongs to the interface and its value to the model; only the second can improve, and yield saturates on most backbones before it does.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
multimodal agents, verification cost, token-metered interfaces, prompt caching, streaming memory, hallucination checking, inference economics, Youden's J, open-weight VLMs, cost model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_cAkAOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the of verifying whether a description of an observation is accurate, when an agent must re-read the original observation to verify it. The results includes lower bounds and empirical analysis on the verification cost relative to the observation cost.

Critical strengths
Critical weaknesses

A major weakness for this paper is clarity. This paper does not clearly state what problem it is studying or clearly describe the problem setup, methodology, or results. The exposition is extremely obfuscated and the explanations and intuitions given use imprecise language, unexplained jargon, and makes claims without proper justification.

The lack of clarity begins in the abstract. It is extremely unclear to me what this means: “A price, however, is only half a verifier. Normalising by how much a verdict actually discriminates separates two quantities that behave quite differently: the nominal ratio is pinned by the interface, moving little across an order of magnitude of scale and uncorrelated with it, while the discrimination varies widely between backbones and is a property of the individual model.”

The main paragraph describing contributions in the introduction: “The verification price is therefore not a small additive constant on top of the channel’s price: it is bounded below by the channel’s own observation term, and the ratio approaches one exactly as the interface becomes more image-dominated.”

"What compression drops first (§4.5, Proposition 3): a derivation of why a capped description sheds the object–attribute pairing before either marginal, together with the measurement that refutes its natural rival, the typicality account."

Finally, the paper also not self-contained. It relies heavily, both in discussion of related work, exposition in various places, and statement and proofs of certain results on references of five different concurrent submissions. The "admission criterion of the companion analysis" is referred to often, but is not clear. Relationship to this concurrent work is only treated in the Appendix, not at all in the main body, and in similarly unclear language.

Questions for authors
Reviewer_nRFbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

In multi-model inference APIs where we pay by token, the standard assumption is that referring to the raw observation/image is costly. As such, one maintains and refers against some description of the observation to minimize direct access. The core assumption is that: "a description of a frame costs a fraction of what the frame costs". This work tries to validate whether creating that caption is worth it vs directly verifying the claim you care about.

Critical strengths

I do not understand the core premise of the paper. So I don't have much to comment on this.

Critical weaknesses

First of all, it is very difficult to understand the core interaction the paper is modeling. It would really help the reader to understand how the authors imagine users to be interacting with the model. If the premise is that passing images/video around all the time is expensive, and its cheaper to use a constant number of calls to create a terse but accurate description of the media, and in all subsequent uses, refer to that description instead of the raw media, then this is not the interaction this paper is considering. Rather, they consider a setting where we pass the media (incur c_obs) and create some caption. But then every time we need to answer the question, we also pass in the media alongside our query? And the core observation is that the costs in both are dominated by the observation? This does not seem that surprising; especially not warranting a research paper. Further, what is the utility of creating the caption - the only reason to do it, is to cache and reuse it. A more meaningful comparison would be whether the cost savings of captioning once and reusing justify the possibly poorer performance relative to directly passing the observation every time you need it. But this is not what worked out here.

Questions for authors

Please see the weakness I listed and justify what insight you are providing.

Reviewer_e9pxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper examines verification costs in token-metered multimodal interfaces. It derives an observation-cost lower bound and evaluates seven vision-language models across nineteen provider endpoints. The experiments distinguish nominal cost from verification quality and examine caching, batching, and prompt sensitivity.

Critical strengths
  • The endpoint audit usefully distinguishes caching identical requests from reusing the same image across different questions.

  • The experiments examine several important sensitivities, including provider choice, verifier prompts, and negative-claim construction.

Critical weaknesses
  • Some conclusions do not follow from the cost bound. For a fixed caption cost, shortening the verifier’s text decreases v/c, while leaving its lower bound unchanged, contrary to repeated statements in the paper. Moreover, expensive verification alone does not imply that verification is inadmissible: this also depends on downstream benefits and the unmeasured opportunity cost \mu.

  • The description-saturation and binding-loss claims lack robust support. The shipped yield predicate can confuse co-occurrence with object–colour binding, and alternative scoring changes the apparent saturation pattern.

Questions for authors
  1. After correcting the direction of the prompt-shortening argument, which design conclusions remain?

  2. How does caption yield correspond to the downstream accuracy gain in the admission criterion, and what supports the recommendation against verification without measuring \mu?

  3. Does the saturation claim survive validated binding evaluation on the 200-probe dataset, and how should the mechanism be revised given Appendix D.10?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.