acceptodds
Under review as a conference paper at ICLR 2027

ReSight-SMC: Two-Stage Power Sampling via Island SMC with Visual Scouts

Abstract

Power sampling has emerged as a powerful training-free approach to LLM reasoning, eliciting capabilities comparable to reinforcement learning by sharpening the model distribution over complete responses. Despite this success, power sampling remains underexplored in large vision-language models (LVLMs). We first transfer Power-SMC to LVLM decoding by defining a sequence-power target over complete responses conditioned on both the image and the prompt. This direct multimodal transfer provides a strong training-free baseline, but leaves two aspects of finite-particle multimodal inference unaddressed. At the particle level, global resampling can collapse genealogies, while particle-based power sampling does not diversify trajectories through distinct visual cues in multimodal decoding, limiting exploration under a finite particle budget. At the answer level, sequence-level sharpening makes distinct reasoning trajectories compete even when they support the same answer. We introduce ReSight-SMC, a verifier-free two-stage power sampler for LVLM inference. Its first stage uses ancestry-isolated SMC islands to preserve independent trajectory families and routes a bounded set of prefix-conditioned visual scouts to prefix-relevant image regions while discouraging redundant overlap. Each scout temporarily increases attention to the image tokens and further emphasizes its routed region. Exact importance correction preserves the base LVLM sequence-power target. The second stage aggregates terminal importance mass by canonical answer, powers the resulting answer marginal, and samples an answer together with a supporting trajectory. Across four LVLM backbones and five benchmarks, ReSight-SMC achieves stronger aggregate performance than Power-SMC over both the reasoning and perception benchmark groups. Without post-training, it remains competitive in aggregate with backbone-matched models trained using reinforcement learning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
large vision-language models, training-free inference, sequential Monte Carlo, power sampling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JuZxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReSight-SMC, a training-free test-time scaling framework for large vision-language models. Building upon Power-SMC, the method introduces ancestry-isolated particle islands and prefix-conditioned visual scouts to improve trajectory exploration, followed by answer-marginal power sharpening to aggregate support across reasoning paths. Experiments on four LVLM backbones and five benchmarks demonstrate consistent improvements in aggregate accuracy over Power-SMC.

Critical strengths
  1. The proposed framework achieves consistent aggregate improvements across multiple LVLM backbones and benchmarks without additional training.

  2. The component ablations provide evidence for the effectiveness of the proposed modules.

Critical weaknesses
  1. The methodological novelty of the paper appears relatively limited, as several key components are closely related to existing approaches. The sequence-level power sampling framework and importance correction are largely inherited from Power-SMC, while island-based resampling is a well-established technique in the SMC literature. The visual attention reactivation mechanism is closely related to PAI [1], which already introduces training-free attention amplification over image tokens. Furthermore, the answer-level aggregation follows a similar motivation to Self-Consistency[2] and more recent marginal-sharpening approaches. Although combining these techniques into a unified framework for LVLM inference is potentially useful, I find the overall methodological contribution somewhat incremental.

  2. The experimental comparisons could be more comprehensive, particularly given the close connections to existing methods. While the authors compare against Power-SMC and low-temperature sampling, several relevant training-free approaches are missing from the evaluation. For instance, incorporating visual attention enhancement strategies such as PAI into the Power-SMC baseline would help determine whether the proposed visual scouts offer advantages beyond simpler attention amplification. Similarly, comparisons with alternative answer aggregation strategies, such as weighted voting or existing marginal-sharpening approaches, would better establish the necessity of the second stage. The current ablations demonstrate that individual components contribute to the final performance, but do not sufficiently distinguish the proposed mechanisms from closely related alternatives.

  3. Although ReSight-SMC consistently improves aggregate accuracy over Power-SMC, the absolute gains are relatively modest, ranging from approximately 1.0 to 1.7 pp across the four evaluated backbones. Meanwhile, the reported inference latency increases from 7.85s for Power-SMC to 10.37s for ReSight-SMC. Since both methods use the same number of particles but incur different computational costs, matching the particle budget alone does not necessarily constitute a fair efficiency comparison. It would be useful to evaluate Power-SMC with a larger particle budget or other compute-matched configurations to determine whether the additional complexity of ReSight-SMC is justified.

  4. The effectiveness of the visual scouting mechanism could be better supported by more direct analysis. The paper argues that prefix-conditioned visual scouts encourage exploration of distinct visual evidence and mitigate the declining influence of image tokens during long reasoning trajectories. However, most of the supporting experiments focus on downstream accuracy rather than directly measuring whether the selected regions contain relevant evidence or whether the resulting trajectories exhibit better visual grounding. In particular, a comparison with random region assignment would help clarify how much of the improvement comes from the proposed attention-based routing strategy rather than the attention intervention itself. Moreover, visual scouting is only activated after a fixed decoding checkpoint and does not contribute to shorter responses, which limits its applicability to some of the evaluated tasks.

[1] Paying More Attention to Images: A Training-Free Method for Alleviating Hallucination in LVLMs. ECCV 2024 [2] Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023

Questions for authors

See weakness.

Reviewer_67J1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article extends Power-SMC to vision-language models by sampling responses from a sharpened distribution p(y)αp(y)^\alpha. It introduces three components: independent particle groups to preserve diversity, visual scouts that increase attention to image regions with importance correction, and answer-level weighted voting with sharpening γ=2\gamma=2. The method requires no training and is evaluated on four Qwen-VL models across four datasets using four seeds. MMStar is further divided into reasoning and perception subsets.

Critical strengths

Careful implementation and transparent reporting. The paper provides a sound importance correction for adaptive scout proposals (Appendices B and D). All results include four seeds with standard deviations. The authors also report negative findings, such as scouts never activating on RealWorldQA, making the results easier to assess.

Critical weaknesses
  1. The most important baseline, majority voting with the same sampling budget, is missing. The second stage essentially performs weighted voting. On RealWorldQA, it explains the entire gain over Power-SMC because scouts never activate. Without majority voting or best-of-N over 32 samples, the added value of SMC remains unclear. Current decoding baselines use only one sample, and the 3B model barely outperforms low-temperature decoding (51.3 vs. 51.1). The comparison with RL-trained models is also unfair in compute, using 32 particles (10.4s) against single-sample RL decoding (4.1s).

  2. The contribution of visual scouts is limited. Removing scouts reduces the average score only from 65.05 to 64.42, a small difference relative to the reported seed variation. Scouts activate in only 31% of MMStar-P runs and never on RealWorldQA, weakening the claim that they drive the perception gains. Overall improvements over Power-SMC are also modest at 1.0 to 1.7 points.

Questions for authors

Clarifications That Could Change My Assessment:

  1. How does majority voting over 32 independent samples at temperatures (T=0.5) and (T=1) perform across the five evaluation columns? How does weighted voting with γ=1\gamma=1 compare?

  2. At matched inference cost, such as TRACE-RL with 32-sample voting, does ReSight-SMC still outperform or match the RL baselines?

  3. Would triggering scouts earlier, such as at token 8, improve performance on direct-answer perception tasks?

Suggestions :

  1. Visual attention boosting is closely related to PAI (Paying More Attention to Image, ECCV 2024), which deserves discussion. Adding a greedy-decoding baseline and visualizations of scout-selected regions would also strengthen the evaluation.
Reviewer_fUknOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ReSight-SMC, a training-free inference framework for large vision-language models (LVLMs) that extends sequence-level power sampling to multimodal reasoning. The method combines two stages to improve exploration and answer selection under a finite particle budget.

In the first stage, the authors introduce an island-based Sequential Monte Carlo (SMC) sampler, where particles are divided into independently resampled groups to mitigate genealogical collapse. They further introduce prefix-conditioned visual scouts, which select image regions based on attention-derived relevance and temporarily amplify attention to the corresponding visual tokens. An importance-weight correction is applied to maintain the original sequence-power target despite these modified proposal distributions.

In the second stage, the method aggregates the importance weights of completed trajectories according to their canonical answers and applies an additional power transformation to the resulting answer distribution. This is intended to leverage the collective support of different reasoning trajectories leading to the same answer.

Experiments are conducted on four Qwen-based LVLM backbones across five evaluation groups derived from MathVista, LogicVista, MMStar, and RealWorldQA. The results show consistent aggregate improvements over the Power-SMC baseline, with additional ablations examining the contributions of island resampling, visual scouting, and answer-level sharpening.

Critical strengths

S1. A principled extension of sequence-level power sampling to multimodal inference.

The paper provides a mathematically grounded formulation of sequence-power sampling conditioned on both visual and textual inputs. In particular, the importance-weight correction allows the proposed visual interventions to modify the sampling proposals while preserving the underlying sequence-power target. The theoretical analysis and derivations in the appendix strengthen the methodological soundness of the framework.

S2. A reasonable approach to mitigating particle degeneracy and improving trajectory diversity.

The island-based sampling strategy addresses a well-known limitation of SMC algorithms: repeated resampling can cause a small number of ancestral trajectories to dominate the particle population. Restricting resampling within independent islands is a reasonable approach to preserving multiple trajectory families. The authors also provide genealogy statistics that support the intended effect.

S3. Relatively comprehensive evaluations and component-level analyses.

The experiments cover four LVLM backbones and multiple benchmarks involving visual reasoning and perception. The paper includes comparisons with the original Power-SMC approach, low-temperature sampling, and selected reinforcement-learning-based checkpoints. It also provides component ablations, particle genealogy measurements, output coverage analyses, and additional sensitivity studies in the appendix.

Critical weaknesses

W1. The improvements over Power-SMC are relatively modest, and the contribution of the visual scouting component is particularly limited.

While ReSight-SMC consistently improves aggregate accuracy, the absolute improvements over Power-SMC range from approximately 1.0 to 1.7 percentage points across the four backbones.

More importantly, the component ablations on Qwen2.5-VL-7B show that removing visual scouts reduces the aggregate accuracy from 65.05% to 64.42%, corresponding to only a 0.63-point improvement. This is relatively small considering the additional complexity introduced by region routing, attention modification, and dual decoder states.

Furthermore, the visual scouting mechanism is never activated on RealWorldQA and is activated for only 31.2% of question-seed runs on MMStar-P. This raises concerns about whether visual scouting is sufficiently general or effective across different types of multimodal tasks.

W2. The effectiveness of the visual scouting mechanism is not convincingly demonstrated.

The paper assumes that increasing attention to selected image regions can encourage more useful visual reasoning trajectories. However, attention-based relevance is not necessarily equivalent to causal importance or task-relevant visual evidence.

The current experiments mainly report downstream accuracy improvements, without directly verifying whether visual scouts actually discover previously overlooked visual evidence or improve visual grounding. For example, it would be useful to measure whether routed regions overlap with ground-truth evidence regions, whether visual scouts reduce hallucinations, or whether they produce substantially different reasoning trajectories from the unmodified proposals.

Moreover, the paper lacks comparisons with simpler visual interventions, such as randomly selected regions, uniformly distributed scout regions, or alternative attention-based routing strategies. Such comparisons are important to determine whether the proposed prefix-conditioned routing mechanism is necessary.

W3. The inference overhead is substantial relative to the reported accuracy improvements.

According to the efficiency analysis, ReSight-SMC requires 10.37 seconds per example, compared with 7.85 seconds for Power-SMC and 4.11 seconds for standard sampling.

This corresponds to approximately 32% additional latency over Power-SMC and 2.52 times the latency of standard sampling. Given the relatively modest aggregate accuracy gains, the accuracy-efficiency tradeoff remains insufficiently justified.

A more convincing evaluation should include accuracy-versus-computation curves under matched latency, FLOPs, or effective token-generation budgets. It is currently unclear whether the same computational resources could yield comparable or better performance simply by increasing the number of particles in Power-SMC or using more samples for self-consistency.

Questions for authors

Q1. Necessity of prefix-conditioned visual routing.

Have the authors considered simpler baselines, such as random region selection, fixed spatial coverage, or uniform visual attention amplification? How much of the visual scouting improvement comes specifically from prefix-conditioned routing rather than the additional attention allocated to visual tokens?

Q2. Evidence for improved visual grounding.

Can the authors provide direct evidence that visual scouting improves the model's use of visual information? For example, do scout trajectories attend to more relevant regions, recover previously missed visual evidence, or produce more visually grounded reasoning than standard particles?

Q3. Comparison under matched inference computation.

How does ReSight-SMC compare with Power-SMC using a larger particle budget or with self-consistency under matched wall-clock latency or FLOPs? Would the reported improvements remain significant when comparing methods with equivalent computational resources?

Comments on the paper

1 comment
Area_Chair_Ua4vOpenReview ↗

Reminder: Please Respond to Author Rebuttals

Dear Reviewers,

As the discussion period is coming to an end, please review the authors’ responses and provide a brief follow-up on whether your concerns have been addressed.

Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.