acceptodds
Under review as a conference paper at ICLR 2027

Reobserve: Allocating a Fixed Budget Between Visual Readings and Reasoning Chains

Abstract

A vision-language system answers a question about an image by encoding the image once and then sampling several reasoning chains over that single encoding, which are then combined by a majority vote. Under a fixed budget one can instead read the image over again. Chains built on one reading share whatever that reading got wrong, so they are not independent observations: with readings and chains on each, the variance of the vote mean is , whose first term, the between-reading variance, does not shrink however many chains are added. The decomposition and its cost-optimal split, , are classical results of two-stage sampling and we claim neither. What the paper adds is the object: a reading as the cluster, a definition of reading diversity under which the between-reading variance is a property of the system, and the measurement of the two components on nine multimodal benchmarks under three ways of reading again. Reobserve estimates the components by a nested calibration and enumerates the feasible integer schedules. On every measured grid the accuracy falls with the vote-mean variance, so the variance-optimal schedule is also the accuracy-optimal one on every benchmark, and the oracle row is the same schedule. The size of the two components says where reallocation is possible: on seven of nine benchmarks the default single-reading schedule sits points below the best equal-cost schedule, and on the two with a between-reading variance below there is nothing to reallocate. Most of the gain is reading again at all: one chain per reading is within points of the calibrated rule at the anchor cost, and the split matters only when a reading costs about twice a chain. Where a question's mean support is on the wrong side of one half, less variance lowers its accuracy.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Multimodal reasoning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Lfx9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces Reobserve, which decomposes voting variance into between-reading and within-reading components and uses a labeled calibration set to allocate a fixed budget between visual readings and reasoning chains per reading. Experiments report an average gain of 1.84 percentage points across nine benchmarks and two primary readers. However, important concerns remain regarding baseline implementations, budget accounting, and the connection between variance minimization and accuracy.

Critical strengths
  • Applies two-stage sampling to analyze correlated errors caused by shared visual readings, yielding a simple, interpretable budget-allocation rule that requires no additional training.

  • Covers multiple benchmarks, readers, budgets, and reading distributions, with negative controls, calibration ablations, and theoretical derivations providing broad experimental coverage.

Critical weaknesses
  • External baselines do not retain their core mechanisms: Appendix D explicitly reduces reward-model and weighted-voting methods to schedule selection, then evaluates all selected schedules using majority voting. This does not represent the original methods’ performance, so comparisons against “eighteen rules” do not adequately establish superiority over existing methods.

  • The budget comparison imposes artificial restrictions: baselines that require no calibration are nevertheless charged Reobserve’s calibration cost, while the default is capped at 16 chains and cannot spend the remaining budget at higher budget levels. The increasing advantage with larger budgets therefore partly reflects restrictions on the baseline rather than more effective compute allocation.

  • Minimizing average variance is not equivalent to maximizing voting accuracy. Theorem 1 provides only a conditional lower bound, and the authors show that reducing variance can be harmful when mean chain accuracy is below one half. Yet the experiments do not report the prevalence of such questions, their margin distributions, or stratified gains. The optimization also ignores how incorrect votes are distributed across multiple candidate answers.

  • The method’s incremental value is limited: both the variance decomposition and optimal cluster size come from classical two-stage sampling, while fixing one chain per reading trails the full method by only 0.07 percentage points on average across the seven main benchmarks in Table 4. The authors need to establish a consistent advantage over this simple strategy sufficient to justify labeled calibration and variance estimation.

Questions for authors

Please refer to Weaknesses.

Reviewer_zky3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to allocate a fixed inference budget between visual re-readings and reasoning chains in a two-stage vision-language system. The authors decompose the variance of chain correctness into between-reading and within-reading components, derive a cost-aware allocation rule, and propose Reobserve to estimate these components using a labeled calibration set and select an appropriate schedule. Experiments on nine multimodal benchmarks show that Reobserve improves over the default single-reading self-consistency strategy and achieves performance close to an oracle schedule. However, its advantage over simple fixed re-reading is limited, and the way calibration costs are accounted for raises concerns about the fairness of the comparisons.

Critical strengths
  1. The paper explores an interesting direction in multimodal test-time scaling by investigating how to allocate computational resources between visual perception and reasoning. Rather than simply increasing the number of reasoning chains, it highlights the importance of revisiting visual inputs, which provides a valuable perspective on improving the efficiency of multimodal reasoning.
  2. The paper provides a two-level variance decomposition that explains why additional reasoning chains cannot eliminate errors shared through the same visual reading. This analysis offers a useful theoretical perspective on the limitations of conventional self-consistency methods.
Critical weaknesses
  1. The paper subtracts Reobserve's calibration cost from the budget of every baseline, even though most baselines do not require calibration. This effectively reduces the inference budget available to the baselines. More importantly, the paper's own full accuracy grids show that a simple (8,1) fixed re-reading schedule already matches Reobserve on MathVista and outperforms its selected (7,1) schedule on MMMU. Therefore, the reported advantage of Reobserve over fixed re-reading may largely result from the way the calibration cost is accounted for rather than from the proposed variance-based allocation.
  2. Most of the improvement comes from increasing the number of readings, while fixing k=1 loses only 0.07 points relative to the full rule. Moreover, directly selecting the schedule by calibration accuracy is only 0.18 points worse on average. Thus, the paper currently provides stronger evidence for the usefulness of visual re-reading than for the necessity or superiority of the proposed variance-based allocation rule.
Questions for authors
  1. Can the authors rerun the main comparison without charging Reobserve's calibration cost to baselines, and include a strong cost-matched fixed-re-reading baseline that uses the maximum feasible number of readings with k=1? How much of the reported gain remains under this comparison?
  2. The theoretical accuracy bound assumes that the correct answer receives more than half of the votes, whereas the experiments use plurality/majority voting over multiple answer choices. Can the authors clarify whether the theoretical bound directly applies to their actual multiclass aggregation protocol?
Reviewer_vrJUOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses a specific inefficiency in test-time scaling for multimodal reasoning: self-consistency samples multiple reasoning chains from a single visual reading, but chains sharing the same reading inherit any misperception in that reading. The authors formalize this by decomposing vote variance into a between-reading term and a within-reading term (Proposition 1), drawing directly on the law of total variance and the two-stage sampling literature (Cochran, Kish, Searle). They show that more chains cannot reduce the between-reading component, which in their measurements on seven of nine benchmarks is of the same order as the within-reading component. From this decomposition they derive (i) a lower bound on vote accuracy that has a ceiling at one reading (Theorem 1), and (ii) an exact efficiency identity showing that any (r,k) schedule's variance equals the continuous optimum scaled by two factors—one for unspent budget and one for distance from the optimal k* (Theorem 2). The proposed method, Reobserve, estimates the two variance components on a small labelled calibration set and selects the feasible integer schedule minimizing predicted variance, with no training or changes to the reader, reasoner, or aggregator. The empirical work compares Reobserve to eighteen baselines under a matched cost cap on nine benchmarks with two Qwen2.5-VL readers and additional reader/reasoner variations, reporting an average gain of 1.84 points over the default (2.37 points on the seven main benchmarks), with no gain on two negative controls where σ_V² is negligible. The paper also includes extensive ablations isolating the contributions of extra readings, fewer chains, calibration size, and robustness to misspecified inputs.

Critical strengths

The most consequential strength is the clean, actionable theoretical decomposition combined with a method that follows directly from it. Proposition 1 is not new mathematics, but the decision to treat the reading as the cluster and the chain as the within-cluster unit is the right abstraction for the problem, and it yields a concrete, closed-form rule for budget allocation (Equation 6, Algorithm 1). Theorem 2's efficiency identity is particularly useful: it explains why the default (1,K) is inefficient (it leaves the between-reading term untouched) and how much is lost by any other split, including a second-order sensitivity result (Proposition 2) showing that errors in the variance ratio are absorbed by the flatness of the bracket near k*. This means the method is robust to small calibration sets (a point the authors confirm empirically): 64 calibration questions select the same schedules as 512. For a practitioner, this is a genuinely operational result, not just an asymptotic claim.

The second critical strength is the empirical rigor of the comparison and the honesty of the negative controls. The paper does not simply report aggregate gains; it isolates the seven benchmarks where σ_V² is non-negligible and explicitly treats SEED and RealWorldQA as negative controls where no rule should (and does not) gain more than 0.2 points. The gain decomposition (Table 15) shows that 74–88% of the gain comes from additional readings, not from reducing chains, which directly validates the theoretical framing. The oracle gap is small (within 0.15 points at the main budget), and the ablations in Table 4 are informative: ignoring the reading-grouping collapses the gain to the default, confirming that the variance model is responsible. This level of self-scrutiny is not universal in the test-time scaling literature.

Critical weaknesses

The most material weakness is that the core statistical framework is entirely classical, and the paper does not engage deeply with the existing test-time augmentation literature that already demonstrates the central empirical claim. Kozodoi et al. (2026, "Test-Time Augmentation for LLMs: When Input Diversity Beats Output Diversity at Matched Compute") show across six tasks that input-side diversity Pareto-dominates self-consistency on cost-effectiveness, delivering roughly 1.8× more accuracy per dollar. That paper is not cited in the present submission. The distinction the authors could draw is that Reobserve provides a variance-based allocation rule rather than a fixed augmentation budget, which is a genuine contribution. But without positioning against this closely related work, the paper's novelty claim ("none models the variance between readings of the input") is incomplete. A reviewer familiar with the TTA literature will ask: is the contribution the observation that input diversity helps, or the specific scheduling rule? The paper should make this distinction explicit and demonstrate why the scheduling rule adds value beyond simply spending a fixed budget on crops or prompt rewrites.

The second weakness is that the separate reader–reasoner pipeline limits the scope of the claims, and the paper does not test whether the same variance decomposition holds for unified multimodal reasoning models. The experiments use Qwen2.5-VL as a reader producing a textual description and a separate text-only Qwen2.5-7B as the reasoner. This is a legitimate design, but many current multimodal systems perform perception and reasoning in a single forward pass or interleaved chain. The paper acknowledges this limitation in the conclusion, but it means the central quantity σ_V²(q) may have a different structure (and different estimability) when "reading" and "chain" are not separable stages. Without at least a small experiment on a unified VLM, the generality of the measurement remains untested. Given that the paper's headline claim is that "between-reading variance matters in current VLM systems", evidence from a unified model would substantially strengthen that claim.

Questions for authors
  1. Kozodoi et al. (2026) report that visual TTA outperforms self-consistency at matched compute on multimodal QA. How does Reobserve differ from simply applying a fixed number of visual augmentations and spending the remainder on chains? Is the variance rule's advantage over, say, a (4,4) split primarily due to better calibration of the ratio, or would a fixed split chosen by cross-validation perform comparably? A direct comparison against a TTA-style baseline with a tuned fixed split would clarify the marginal value of the variance model.

  2. Estimability of σ_V² in unified models. In a unified VLM that interleaves perception and reasoning, the notion of a "reading" as a separable textual description may not apply. Could the authors comment on whether the two-level decomposition is still identifiable, for example, by treating different image crops or prompt rewrites as readings even when the model does not produce an explicit description? Have the authors attempted a pilot experiment on a model like InternVL3 or Qwen2.5-VL in a unified prompting mode?

  3. The bound in Theorem 3 is described as "loose as a number", and the ceiling at r=1 requires σ_V²(q) < δ_q(1−μ_q) for the tightness construction. On the seven main benchmarks, what fraction of questions satisfy this condition? If the bound is numerically loose for most questions, does it play any role beyond motivating the direction of the reallocation, or is the efficiency identity (Theorem 2) doing all the practical work?

  4. The paper selects one schedule per benchmark and identifies per-question scheduling as future work. Given that Theorem 1's sign condition (μ_q > ½) varies across questions, how much additional gain might per-question allocation provide? Is the calibration cost per question prohibitive, or could a lightweight predictor of σ_V²(q) be trained from the calibration pool?

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response 2: What the paper finds and what is new, in plain terms

Dear AC and reviewers,

Thank you for the reviews and the discussion. As the discussion closes, we restate in plain terms what the paper finds and what is new.

The problem. A vision-language system answers a question about an image in two steps: it reads the image, then reasons over what it read. The common way to spend more compute is self-consistency: read once, sample many reasoning chains, and vote. But every chain built on one reading starts from that reading. If the reading misses a detail, all chains miss it, and more chains cannot correct it. Asking more people to reason from the same blurry photo does not remove the blur; taking another photo does.

What we find. The uncertainty of the vote has two parts: one that changes from reading to reading, and one that changes from chain to chain. More chains shrink only the second part; only more readings shrink the first. On seven of nine benchmarks the first part is of the same order as the second, 0.48 to 1.68 times its size. As a result, with one reading there is a ceiling on vote accuracy that no number of chains can lift (Theorem 1). In practice, going from 16 to 64 chains on one reading adds only 0.15 points.

What is new.

  1. The two-part view applied to multimodal voting, and what it implies. The split of the variance is classical two-stage sampling (Cochran; Kish), and the paper credits it in Section 2. New are its consequences for a vote: the ceiling at one reading (Theorem 1), the exact loss of any split between readings and chains (Theorem 2), and the measurement of the reading part on nine benchmarks with readers from five model families.
  2. A rule that splits a fixed budget. From two numbers measured on a small labeled set, and the cost of a reading relative to a chain, Reobserve picks how many readings to take and how many chains to run on each. When readings barely differ, it keeps one reading; when a reading is costly, it runs more chains on each. Errors in the two numbers cost little (Proposition 2), so 64 calibration questions are enough.
  3. Evidence that the rule matters, beyond reading more often. That input diversity helps is also known from test-time augmentation. A fixed recipe, however, does not adapt: where the best split moves, a fixed one chain per reading trails by 0.40 to 0.98 points, and in all six deployment conditions the rule also beats choosing the split by calibration accuracy, at 1/6.7 of its calibration cost (Table R2).

What it buys. At the same cost, Reobserve improves on the single-reading vote by 1.84 points over nine benchmarks, and by 2.37 points on the seven where readings vary. On the two benchmarks where readings do not vary, it keeps the single reading and changes nothing. It needs no training, its lead grows with the budget (2.22 and 2.56 points at the two larger budgets), and it holds in unified models that reason directly from the image: 2.14 points on Qwen2.5-VL-7B and 2.22 on InternVL3-8B (Table R3). Reward models and visual re-reading methods, run with their full native mechanisms, still trail it.

What the reviews credit. Reviewer vrJU calls the reading-as-cluster view "the right abstraction for the problem", Reviewer zky3 finds that revisiting the visual input "provides a valuable perspective on improving the efficiency of multimodal reasoning", and Reviewer Lfx9 credits "a simple, interpretable budget-allocation rule that requires no additional training". In the terms of the ICLR reviewer guidelines, the paper contributes new knowledge, theoretical and empirical, with a direct use for practitioners: in multimodal voting, one reading caps accuracy, the cap matters on most benchmarks, and a cheap rule turns this into a better split of the budget.

We thank the AC and the three reviewers again for their time and careful reading.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.