- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper introduces Reobserve, which decomposes voting variance into between-reading and within-reading components and uses a labeled calibration set to allocate a fixed budget between visual readings and reasoning chains per reading. Experiments report an average gain of 1.84 percentage points across nine benchmarks and two primary readers. However, important concerns remain regarding baseline implementations, budget accounting, and the connection between variance minimization and accuracy.
Critical strengths
-
Applies two-stage sampling to analyze correlated errors caused by shared visual readings, yielding a simple, interpretable budget-allocation rule that requires no additional training.
-
Covers multiple benchmarks, readers, budgets, and reading distributions, with negative controls, calibration ablations, and theoretical derivations providing broad experimental coverage.
Critical weaknesses
-
External baselines do not retain their core mechanisms: Appendix D explicitly reduces reward-model and weighted-voting methods to schedule selection, then evaluates all selected schedules using majority voting. This does not represent the original methods’ performance, so comparisons against “eighteen rules” do not adequately establish superiority over existing methods.
-
The budget comparison imposes artificial restrictions: baselines that require no calibration are nevertheless charged Reobserve’s calibration cost, while the default is capped at 16 chains and cannot spend the remaining budget at higher budget levels. The increasing advantage with larger budgets therefore partly reflects restrictions on the baseline rather than more effective compute allocation.
-
Minimizing average variance is not equivalent to maximizing voting accuracy. Theorem 1 provides only a conditional lower bound, and the authors show that reducing variance can be harmful when mean chain accuracy is below one half. Yet the experiments do not report the prevalence of such questions, their margin distributions, or stratified gains. The optimization also ignores how incorrect votes are distributed across multiple candidate answers.
-
The method’s incremental value is limited: both the variance decomposition and optimal cluster size come from classical two-stage sampling, while fixing one chain per reading trails the full method by only 0.07 percentage points on average across the seven main benchmarks in Table 4. The authors need to establish a consistent advantage over this simple strategy sufficient to justify labeled calibration and variance estimation.
Questions for authors
Please refer to Weaknesses.
General Response 2: What the paper finds and what is new, in plain terms
Dear AC and reviewers,
Thank you for the reviews and the discussion. As the discussion closes, we restate in plain terms what the paper finds and what is new.
The problem. A vision-language system answers a question about an image in two steps: it reads the image, then reasons over what it read. The common way to spend more compute is self-consistency: read once, sample many reasoning chains, and vote. But every chain built on one reading starts from that reading. If the reading misses a detail, all chains miss it, and more chains cannot correct it. Asking more people to reason from the same blurry photo does not remove the blur; taking another photo does.
What we find. The uncertainty of the vote has two parts: one that changes from reading to reading, and one that changes from chain to chain. More chains shrink only the second part; only more readings shrink the first. On seven of nine benchmarks the first part is of the same order as the second, 0.48 to 1.68 times its size. As a result, with one reading there is a ceiling on vote accuracy that no number of chains can lift (Theorem 1). In practice, going from 16 to 64 chains on one reading adds only 0.15 points.
What is new.
What it buys. At the same cost, Reobserve improves on the single-reading vote by 1.84 points over nine benchmarks, and by 2.37 points on the seven where readings vary. On the two benchmarks where readings do not vary, it keeps the single reading and changes nothing. It needs no training, its lead grows with the budget (2.22 and 2.56 points at the two larger budgets), and it holds in unified models that reason directly from the image: 2.14 points on Qwen2.5-VL-7B and 2.22 on InternVL3-8B (Table R3). Reward models and visual re-reading methods, run with their full native mechanisms, still trail it.
What the reviews credit. Reviewer vrJU calls the reading-as-cluster view "the right abstraction for the problem", Reviewer zky3 finds that revisiting the visual input "provides a valuable perspective on improving the efficiency of multimodal reasoning", and Reviewer Lfx9 credits "a simple, interpretable budget-allocation rule that requires no additional training". In the terms of the ICLR reviewer guidelines, the paper contributes new knowledge, theoretical and empirical, with a direct use for practitioners: in multimodal voting, one reading caps accuracy, the cap matters on most benchmarks, and a cheap rule turns this into a better split of the budget.
We thank the AC and the three reviewers again for their time and careful reading.