acceptodds
Withdrawn from ICLR 2027

Compute, Fuse, or Acquire? Test-Time Decision-Information Bottlenecks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Test-time scaling is usually treated as allocating more computation, or more interaction with the world. We show that this conflates three distinct bottlenecks. Computational regret is imperfect inference over the current interface; representation regret is evidence the model has already observed but its interface discards; observation regret is evidence never observed. The algebra separating them is elementary, but the consequence is not: computational, representation and observation regret require distinct remedies, namely compute, fusion and acquisition respectively. This yields a prediction a compute-versus-interaction view cannot state, namely that performance can improve at exactly fixed evidence. We test the predictions on InfoGap, a benchmark whose floors are designed and, for vision, tunable. Across three VLMs, varying the designed single-view Bayes accuracy moves the observed sampling plateau with it, separating an information ceiling from a capability limit; fixed-view sampling stays at that ceiling while one relevant view lifts accuracy from 0.49 to 0.99. Holding the observed evidence identical and changing only how two channels are combined moves accuracy from 0.50 to 1.00, which acquisition cannot explain. A bottleneck-matching router is a proof of concept rather than a transferable diagnostic: it reaches 0.94 within a known task mixture at half the generated-token cost of Best-of-16, but leave-one-category-out shows it identifies task families, not instances.

Primary area:
learning theory
Keywords:
Test-time scaling, decision-information bottlenecks, Bayes risk, information acquisition, relational fusion, vision-language models, active perception, test-time inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_u2uyOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper models test-time decision-making with a latent state S, a goal G, base observations X_base, and an interface statistic Z that a test-time procedure conditions on. Proposition 1 telescopes excess risk into three terms:

  • computational: R(δ_K) − R*(Z,G);
  • representation: R*(Z,G) − R*(X_base,G);
  • observation: R*(X_base,G) − R*(S,G).

Under log loss, the last two sum to I(Y;S|Z,G), which splits by the chain rule. Theorem 1 has three parts:

  • Zero "decision deficiency" for all bounded losses holds exactly when the posterior is sufficient: p(y|Z,G) = p(y|S,G).
  • The log-loss deficiency equals I(Y;S|Z,G).
  • For bounded losses, the deficiency is bounded above via Pinsker, with a Fano lower bound for deterministic finite-label tasks.

Two further results motivate "bottleneck matching":

  • Proposition 2: sampling more from the same state yields no new information (conditional data processing).
  • Proposition 3: each operation class moves only its own term's floor. The classes are compute; fuse, which re-encodes within σ(X_base,G); and acquire, which expands the information set.

Bottleneck matching spends the next unit on whichever operation has the largest estimated risk reduction minus λ·cost, and stops otherwise.

ReLATE implements this as a one-step controller over a frozen VLM or VLA policy. Two pilot samples produce cheap features, and a learned router picks one of three arms:

  • compute: majority vote over K fresh samples;
  • acquire: a probe ranked by decision-relevant information gain (DRIG);
  • stop.

Fuse is validated separately rather than routed, and the iterative form is not evaluated (L346–350). InfoGap has three parts:

  • Exact factored POMDPs, where every Bayes risk and mutual information is computed by enumeration.
  • Procedural two-view vision scenes. The families are occlusion, before/after, relational XOR, targeted probe, a tunable occlusion-leak family, and a compute-limited hard-count family. A constraint solver verifies single-view Bayes accuracy ≤ 0.6 and joint ≥ 0.95.
  • Ancillary wrist-camera masking on OpenVLA-OFT in LIBERO.

Experiments cover:

  • the information ceiling (Table 1, Fig. 2);
  • fusion at fixed evidence (Fig. 7);
  • decision-relevant vs. generic acquisition (Fig. 8);
  • the router on a held-out pool of 1,800 instances (Tables 4–5);
  • cross-model transfer (Table 8);
  • embodied sensitivity (Table 6).
Critical strengths

The decomposition gives a clean vocabulary for something test-time-scaling papers routinely blur: evidence the system already holds but discards (fixable without acting on the world) versus evidence it never observed. The formal statements are correct under carefully stated assumptions. The paper uses conditional independence (Y ⊥ X_base | (S,G), ξ ⊥ (S,Y) | (Z,G); L211–214) rather than a naive Markov chain. Theorem 1 usefully ties the deficiency across loss families to a single conditional mutual information.

The best experiment is the occlusion-leak sweep. A real VLM's sampling plateau tracks a designed Bayes bound within 0.02 at four levels, which cleanly separates an information ceiling from a capability limit. The effect also runs in both directions:

  • On synthetic Task D, Best-of-K goes from 0.494 to 1.000 while exact-DRIG probing stays at 0.494 (Table 3).
  • On the compute-limited slice of the VLM pool, always-acquire gets 0.093 where the router gets 0.865 (L1234–1236).

Opening all four probes (0.793) is worse than opening the one decision-relevant probe (1.000), a nice demonstration that acquisition has to be targeted.

The 2026 works it positions against (e.g., Li & Shi; Sevetlidis 2026a/b; Jeong & Yoon) exist and are described accurately. The related-work section is direct about which identities are not new.

Critical weaknesses

Theory. The theoretical contribution is thin for a paper submitted under learning theory:

  • Proposition 1 is a telescoping sum (L195).
  • Theorem 1(ii) is the identity "log-loss Bayes risk = conditional entropy," which the paper credits to Sevetlidis 2026b (L157–160).
  • Theorem 1(i), called "the nontrivial content" (L234), reduces to Brier-score propriety plus the tower property and the law of total variance (L897–913).
  • Proposition 2 is conditional data processing.
  • Proposition 3(ii) is immediate, because Z′ = X_base is admissible.
  • Proposition 4 is Howard/Lindley value of information.

There is no generalization, sample-complexity, or rate result. The "decision deficiency" terminology is not related to Le Cam's deficiency or to the Variational Deficiency Bottleneck (Banerjee & Montúfar). "Fuse" is not positioned against the literature on early and late multimodal fusion.

Evidence. More importantly, the empirical tests of P1–P3 mostly replay how the benchmark was built:

  • The solver enforces chance-level single views and near-perfect joint views.
  • Each XOR view carries zero bits (L1074), so per-view late voting must sit at 0.5.
  • Acquisition hands the model the missing view.
  • The trivial "All views (K=1)" baseline already scores 1.000 everywhere at 64 tokens (Table 1).

Compute is only ever scaled over the starved single view. That cannot distinguish "sampling cannot help" from "sampling a view with zero decision information cannot help," and the second is true by definition.

The flagship P2 result (0.500 → 1.000 at fixed evidence) depends on the paper's taxonomy. A two-axis view could just as well call "one joint prompt instead of per-view votes" a better use of inference over evidence already held. The additive-vs-parity control (E.6) attributes the gain to relational structure, but it does not settle this. The paper itself concedes the result is "a restricted-class gap, not a bound on representation regret" (L445–446). Yet the abstract says "acquisition cannot explain" it and calls it a prediction a compute-versus-interaction view "cannot state" (L018–025).

The line between computational and representation regret also depends on an analyst-chosen Z, which can be moved for an end-to-end VLM. And the regret terms are never quantified on a real model: the logged regret fields "were not computed under the measurability convention of §3.1" (L1716–1718).

Router. The operational half does not deliver the three-way choice in the title:

  • The router selects only among compute, acquire and stop; fuse is "validated separately, not as a routing arm."
  • The relational encoder is validated on synthetic data only (L1600–1601).
  • The iterative controller is not evaluated.

The router itself is effectively a 30-cell lookup table: an explicit per-cell lookup scores 0.937 (L1279–1281). A category-only baseline comes within 0.007. Leave-one-category-out falls to 0.695, with hard-count collapsing to 0.299 after 452 of 720 instances are misrouted to acquire (L1286–1295). The 0.938 headline is just the weighted average under an unmotivated 40/40/20 regime mix (0.4·1.000 + 0.4·0.865 + 0.2·0.958). The "half the generated-token cost" claim uses configured per-arm costs that count output tokens only. The authors state that the sign of the total-cost difference "is not identified" (L1153–1154).

Scope. The evidence base is narrow:

  • Vision is procedural only, with no natural images.
  • The tunable-p sweep is reported only for Qwen3-VL-8B (Fig. 2), although the abstract opens that sentence with "Across three VLMs."
  • Transfer (Table 8) covers only the chance-level categories.
  • Embodied evidence is ancillary, and its baseline was not reproduced (LIBERO-10 at 0.26 vs. about 0.95 published).
  • The per-scene records behind Fig. 7 and the multi-model transfer logs are not in the supplement (L492–493).

The writing is dense, heavily hedged, and full of internal codenames (E5–E9, R10/R11, script names). The hard-count and saturated families make up 60% of the router pool, yet they are never specified beyond one-line labels.

Questions for authors
  1. Please scale compute on the joint interface where there is headroom, e.g., before/after joint at 0.929 (Fig. 7), and report it next to Table 1.
  2. Table 3 reports single-inference accuracy 0.961 [0.954, 0.968] on Task B (relational XOR). Appendix C says binary-label families are sampled at C_info ∈ {0.2, 0.5, ln 2} nats with Y = g(S) (L1093–1099). For binary Y, H(Y|Z,G) ≥ 0.2 nats implies Bayes error ≥ h⁻¹(0.2) ≈ 0.05, so accuracy is at most about 0.95. Which instances and ceilings does Table 3 use? For example, does column B pool the 4-label additive control?
  3. Why does acquisition accuracy fall from 1.000 to 0.966 as the leak level rises (L435–436), when joint Bayes accuracy is 1.0 throughout?
Reviewer_BWSYOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper decomposes the regret of test-time computation into three sources---computational, representational, and observational. The first considers the gap between the final iterate of the model (δK\delta_K) and true risk with under representation ZZ, the second is what's lost by processing the original features XX into ZZ, and the third is regret incurred by the gap between the XX and the 'true latent state' SS. An algorithm, ReLATE, is proposed that estimates how additional computation, ‘fusion’, or information acquisition reduces the corresponding risks. Presumably, this would need to be done iteratively, but the paper presents only a “one-step” form. Experiments are conducted; however, the ‘fuse’ action is replaced with a ‘stop’ action, and ‘fuse’ is tested in an isolated experiment.

Critical strengths

I think the paper is gesturing at an interesting problem: how regret is affected by various aspects of modern architectures such as representation and number of compute steps, and asking if we can decompose / disentangle those sources of error.

Critical weaknesses
  1. Theoretical formulation: the theoretical formulation is not sound. I have the following issues.
  • The involved variables and the underlying generative process (which seems to exist since there is a "true latent state" SS) are not clearly defined. For instance, regarding S, G, X, and Z: Are they continuous or discrete? Presumably they are random, but I'm not sure that is stated. Is there an underlying distribution (MDP?) that involves these variables that the model is trying to match, or are these purely model artifacts? I don't think ξk\xi_k is ever defined: this is the randomness involved in the test-time compute?
  • I don't understand why the goal variable GG is necessary. Since there is a target decision YY, why is the goal simply not to predict YY correctly? Or, as I often see G on the RHS of conditioning statements, maybe G is something more similar to a prompt? But then ZZ is also said to be a prompt, so I don't know why there would be the need to introduce two prompts then.
  • I don't understand why the regret is only defined for the last computational step (K). Shouldn't it involve some expression that decomposes into per-computation-step regret?
  • If I understand correctly, Prop 2 is just saying that we can't learn anything from samples when the distribution from which we are sampling is conditioned on Z and G, which are already known? This seems like an obvious statement, and I don't see how it fits into the contribution since presumably we are doing additional computation between sampling steps?, i.e. δk\delta_k --> δk+1\delta_{k+1}?
  1. It is not clear how ReLATE is actually able to estimate the three regrets in line 2 of the algorithm. Leaving the issues of reliable finite-sample estimation alone, the paper says ΔR^\hat{\Delta R} is estimated from a 'pilot budget'. Doesn't estimating these risks require the very actions that you are taking the argmax over? So should I think of the pilot budget like a calibration set? This may be sensible---I've seen similar things done for early-stopping of test-time compute---but there's a big underlying issue of distribution shift. How do I know that my 'pilot budget' represents the later risk estimates of interest? And if it there is zero distribution shift, then hasn't the pilot estimation step simply done all the `work' the algorithm needs to do in the first place?
Questions for authors

Questions are provided within the Weaknesses section.

Reviewer_dM9fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies when test-time resources should be spent on additional computation, fusion of existing evidence, or acquisition of new observations. It decomposes excess risk into computational, representation, and observation regret using Bayes-risk references for the deployed interface, available observations, and full latent state. Under the stated assumptions, the authors characterize decision sufficiency and relate log-loss deficiency to conditional mutual information.

The InfoGap benchmark examines these ideas through synthetic tasks, procedural visual problems, and information-channel interventions on a frozen vision-language-action policy in LIBERO. The experiments show sampling plateaus that approximately follow designed single-view information limits, gains from complementary observations, and advantages of joint conditioning over late voting. A one-step router selects compute, acquire, or stop. It performs well within the evaluated task mixture, although the reported controls indicate that it primarily learns task-family structure rather than transferable instance-level bottleneck diagnosis.

Critical strengths

The paper gives a precise operational distinction between imperfect inference over a current interface, decision-relevant information discarded by that interface, and information that has not been observed. These cases require different interventions even though they may all appear as failure to obtain the correct answer. The risk decomposition and the careful separation of exact log-loss identities from claims about classification accuracy provide a useful framework for interpreting failures of test-time scaling.

The tunable information-ceiling experiment is the strongest empirical result. Rather than showing only that sampling saturates near chance, the authors vary the designed single-view Bayes accuracy and find that the observed sampling plateau tracks it reasonably closely, while acquiring the complementary view remains effective. The category-based and leave-one-category-out controls also provide an unusually candid account of the narrower scope of the router result.

Critical weaknesses

The fixed-evidence fusion experiment more directly establishes the benefit of joint processing over a restricted late-voting procedure than a reduction in representation regret. A channelwise but lossless interface can preserve all observed information while late voting still fails, as the paper itself acknowledges. Thus, although the experiment supports the operational importance of combining evidence jointly, the representation component of the proposed decomposition is not cleanly isolated empirically. This limits how strongly the fixed-evidence result validates the proposed decomposition rather than a restricted decision-rule comparison.

The router performs only slightly better than the category-based policy and the per-cell lookup, while its performance declines substantially when evaluated on an unseen category. The learned router also excludes fusion, the operation that distinguishes the proposed framework from a compute-versus-acquire controller. In addition, the efficiency comparison is limited to normalized generated-output-token cost and does not establish total serving cost. These limitations substantially reduce the evidence for transferable instance-level bottleneck diagnosis or an end-to-end compute, fuse, and acquire controller.

Questions for authors
  1. Can the authors provide a direct empirical intervention in which the deployed interface itself discards decision-relevant information, so that the resulting gain more clearly identifies representation regret rather than only the restricted-class gap between late voting and joint conditioning?

  2. In the targeted-probe experiment, the selector is trained on context geometry and achieves very high held-out probe-selection accuracy. How much of the reported advantage should therefore be interpreted as supervised task-specific relevance selection rather than estimation of decision-relevant information gain in a more general sense? An equally supervised relevance-selector baseline would help clarify this point.

  3. The router is trained using the best realized utility among stochastic arm outcomes. How stable are the resulting routing labels under repeated arm evaluations or different generation seeds, and how much of the router’s apparent advantage over fixed strategies remains when labels are based on estimated expected utility rather than a single realized outcome?

Comments on the paper

1 comment

We would like to sincerely thank all three reviewers and the Area Chair for taking the time to read and evaluate our work.

We were encouraged by the reviewers' recognition of the importance of distinguishing different bottlenecks in test-time scaling and the usefulness of our decision-theoretic perspective. We also appreciate the positive feedback on our information-ceiling experiments.

Although some aspects of the theoretical formulation may not have come across as clearly as we intended, the reviews have given us a better understanding of where the paper needs to be strengthened. In particular, we agree that the work would benefit from deeper theoretical developments, more direct empirical evidence for representation regret, and a more thorough investigation of how these different bottlenecks can be identified in practice.

After careful consideration, we have decided to withdraw the current submission. Rather than making incremental changes during the rebuttal period, we would like to take more time to develop these ideas and address the reviewers' concerns more thoroughly.

We remain excited about the research question and believe that the feedback we received will help us make this work stronger.

Thank you again for your thoughtful comments, time, and consideration. We genuinely appreciate your efforts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.