acceptodds
Under review as a conference paper at ICLR 2027

Rewind to Rethink: Localized Repair for Multimodal Reasoning

Abstract

Multimodal reasoning models can continue to rely on an early judgment even after generating additional reasoning, allowing an initial error to persist into the final answer. We define step-to-step interventional support using offline Shapley attribution: preceding steps are retained or masked, and their marginal contributions to the mean log-likelihood of fixed subsequent text are averaged across coalitions. The resulting matrix reveals how an early judgment can continue to support later reasoning and the final answer, a pattern we call answer inheritance. Inspired by this observation, we propose BANKROLL, a training-free method that rolls back to regenerate an earlier judgment together with the reasoning built on it. Cashout monitors intermediate answers to determine when thinking stops. Refinance uses online local masking tests as a heuristic to select a rollback point, truncates the chain before that step, and samples alternative continuations. Writeoff retains the original chain if no rollback point is chosen, no continuation is selected, or the probe-score acceptance check fails. Experiments across five benchmarks and three multimodal models show that BANKROLL consistently improves task performance over standard chain-of-thought (CoT).

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal Reasoning, Shapley Attribution, Localized Reasoning Repair, Inference-Time Early Stopping, Training-Free Inference

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_DA8zOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how an early judgment can continue to influence later reasoning and the final answer in multimodal models. It uses an offline Shapley analysis to examine this pattern, called answer inheritance. The paper then proposes BANKROLL, a training-free framework for stopping and repairing reasoning. Cashout uses intermediate answer probes to decide when to stop. Refinance uses local masking tests to select a rollback point and samples alternative continuations. Writeoff compares probe scores to decide whether to accept the repair. Experiments cover five benchmarks and three multimodal models. BANKROLL improves performance over standard CoT in all 15 settings. Ablation studies further examine the stopping rules, rollback selection, and repair acceptance.

Critical strengths

1.The repair method is well motivated, and the ablations support its design. Rolling back before a selected step allows the model to revise an earlier visual judgment and the reasoning that follows it. This requires no additional training. The random-pivot control keeps the candidate set and repair procedure unchanged. Selecting the pivot with the proposed score improves accuracy by 1.03 and 1.80 percentage points on M³CoT and MMStar. This provides useful evidence that the rollback position matters.

2.The evaluation is broad and shows consistent gains over standard CoT. The experiments cover reasoning, visual hallucination, and text recognition across three models. The same BANKROLL settings are used across models and benchmarks. On MMMU-Pro, accuracy rises from 54.62% to 69.02% for Qwen3.5-9B and from 39.13% to 49.77% for GLM-4.1V-9B-Thinking. These results suggest that the benefit extends beyond a single model or task.

Critical weaknesses
  1. The repair scores are not well tested. The masking score shows how much a step supports the current answer. A correct step can also receive a high score. For open-ended tasks, masking can change the trial answer, so the scores may compare different answers. Writeoff also uses a score recorded before the repair is complete. That score may refer to a different answer from the final output. The paper does not show how often the selected steps contain errors or whether the saved scores help choose correct repairs.

  2. It is unclear whether local rollback is better than simpler alternatives. The paper does not compare with self-refinement. It also lacks a baseline that samples full reasoning chains and picks the one with the highest probe score. Appendix F shows only one example using a reflection prompt. The random-pivot test compares rollback positions within BANKROLL, but does not address these other methods. On M³CoT, using one continuation gives 80.24% accuracy, below 81.02% without Refinance. A stronger comparison is needed to show what local rollback adds beyond sampling and selecting answers.

Questions for authors
  1. How often does the selected step contain an actual error? Does scoring the completed repair work better than using the earlier score?

  2. Could you compare BANKROLL with a self-refinement baseline and with full-chain sampling selected by the same answer-probe score? These comparisons would help clarify the benefit of local rollback.

5 comments

Thank you for these comments and suggestions. We address the two points in the order raised below.

1. Selected-step and repair-score evidence

In a manual check of 100 randomly sampled repair records, 23 selected steps contained an actual error and 77 did not. A selected step is an intervention point, not an error label: Refinance regenerates that step and its suffix.

Replacing score-based pivot selection with uniform random selection from the same shortlist reduces accuracy from 82.70% to 81.67% on M³CoT and from 79.00% to 77.20% on MMStar (Table 2). This supports the score-based selection heuristic. Removing Writeoff’s score-based acceptance check lowers accuracy from 82.70% to 82.53% on M³CoT and from 79.00% to 78.27% on MMStar. Writeoff uses the probe score recorded at candidate selection, whose predicted answer may differ from the completed repair’s final answer. Replacing this saved score with a score recomputed after the repair is complete changes M³CoT from 82.70% to 82.61% and leaves MMStar and MMMU-Pro unchanged.

We clarify these points in Writeoff and Core Ablations, with supporting analyses in the From Attribution to Local Repair appendix.

2. Local rollback versus simpler alternatives

Probe-select (2) samples two complete chains and uses the same answer probe as BANKROLL. For the reported comparison, Self-Refine generates an initial response for every example and performs one round of self-checking and revision; both generations are counted in TpT_p.

Across 15 model–benchmark settings, BANKROLL scores higher than Self-Refine in 14/15 and than Probe-select (2) in 13/15. Self-Refine is higher on Qwen HallusionBench (81.39% vs. 81.28%); Probe-select is higher on GLM HallusionBench (75.92% vs. 75.50%) and InternVL MMMU-Pro (56.00% vs. 55.90%). BANKROLL uses 2.8–15.6% more tokens than Self-Refine but 38.0–57.0% fewer than Probe-select. The results support retained-prefix repair, with the Self-Refine comparison carrying additional generation cost.

With (R,s)=(1,3)(R,s)=(1,3), accuracy is 80.24% on M³CoT and 77.93% on MMStar, compared with 81.02% and 77.47% without Refinance; the default (R,s)=(3,3)(R,s)=(3,3) reaches 82.70% and 79.00%. Thus, the current evidence supports selecting among multiple local continuations rather than rollback alone. Because changing the pivot also changes the continuation, random pivoting is a same-procedure control rather than a paired-content comparison.

These comparisons are reported in Main Results and Core Ablations, with candidate-count and random-pivot analyses in the Hyperparameter Sensitivity and From Attribution to Local Repair appendices.

Reviewer_DA8zOpenReview ↗

Reply to Author

Thank you for the additional experiments and clarifications. The new baseline comparisons address much of my concern about the value of local repair. I have two more points that I’m unclear about.

  1. The response says Self-Refine runs only when BANKROLL triggers repair. However, Section 4.1 says it generates and revises an answer for every example. Which setup produced the reported results? Please make the paper and response consistent.
  2. Does rescoring completed repairs give similar results on CC-OCR? The reported comparison covers only multiple-choice tasks. My concern about scores referring to different answers mainly applies to open-ended tasks.

Thank you for the follow-up questions. We clarify the Self-Refine protocol first, and then report the separate open-ended check on CC-OCR.

1. Self-Refine protocol

The reported Self-Refine results use the all-example protocol described in Section 4.1. For every example, Self-Refine first generates an initial response from the image and question, and then performs one round of self-checking and revision; both generations are included in its reported token count. We apologize for the confusion caused by our imprecise earlier wording. Self-Refine is run on every example, independently of whether BANKROLL repairs that example, consistent with Section 4.1.

2. Rescoring completed repairs on CC-OCR

On CC-OCR, post-completion rescoring gives essentially the same aggregate result: the official score changes from 79.55 under selection-time scoring to 79.59 under post-completion scoring (+0.04). To isolate score timing, we kept the original chain, selected continuation, and final-answer handling fixed, changing only the score supplied to Writeoff’s acceptance comparison.

BenchmarkMetricSelection-timePost-completionChange
CC-OCROfficial 0--100 score79.5579.59+0.04
M$^3$CoTAccuracy82.70%82.61%$-$0.09 pp
MMStarAccuracy79.00%79.00%0.00 pp
MMMU-ProAccuracy69.02%69.02%0.00 pp

The CC-OCR comparison directly tests whether the selection-time score can correspond to a different answer after completion. Among 516 repair decisions, 142 had no valid decision-point probe readout, leaving 374 comparable cases. The decision-point and post-completion probe answers differed in 115 of these cases (30.7%). Despite this local mismatch, only 19 of the 512 repairs accepted under selection-time scoring (3.7%) would receive a different accept/rollback decision when the post-completion score was used. Thus, probe answers can change between selection and completion, but post-completion rescoring does not show a consistent aggregate advantage.

Reviewer_256XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses offline Shapley attribution over reasoning steps to illustrate “answer inheritance”: an early judgment keeps supporting later reasoning and the final answer. It then proposes BANKROLL, a training-free procedure for multimodal reasoning models with three parts: Cashout stops thinking based on answer probes; Refinance truncates before the step whose masking most lowers the probe score and regenerates from there; Writeoff accepts the repair only if its probe score does not drop much. BANKROLL improves on standard CoT across five benchmarks and three models, with the largest gains on MMMU-Pro.

Critical strengths
  • The proposed method is training-free. Its hyperparameters were tuned on one model and transferred unchanged to the other models and benchmarks.

  • The repair is selective, since it fixes clearly more wrong answers than it breaks. BANKROLL also matches self-consistency on M3CoT at a fraction of the token cost.

Critical weaknesses
  • The observation and the method are close to prior work. Liao et al. (2025) already show that errors from early steps persist, and the paper cites them for this point. Locating a faulty step and resampling from the clean prefix is the design of Thought-ICS (Samanta et al., ICML 2026), which is not cited. The Shapley evidence consists of three selected chains. The method itself does not use Shapley values but single-step masking, which the paper presents as the weaker signal.

  • The source of the large gains is unclear. Where BANKROLL gains most over standard CoT, simple early exit (DEER) or Concise CoT already recovers most of the gap. This points to a weak standard-CoT baseline (over-long chains or answer extraction) rather than a benefit of rollback. The paper does not report the generation budget or the truncation and parse-failure rates. The ablations also exclude MMMU-Pro, where the headline gains occur.

  • The ablations do not show that each component is needed. Two ablated variants score slightly higher than the full method, most gaps are within the sampling noise of a single run, and no confidence intervals are reported. This evidence does not justify 14 hyperparameters. Nor does it justify a betting-style stopping rule that, by the paper’s own statement, gives no guarantee on answer error.

  • The comparisons are not budget-matched. BANKROLL uses more tokens than standard CoT on two of the three models. Yet it is not compared with confidence-based selection at a similar budget, such as best-of-2 chains chosen by the same probe or DeepConf (ICLR 2026). Random pivot selection performs only slightly worse, so the value of the masking-based localization is unclear.

Questions for authors

Please answer the questions raised in my review.

1 comment

Thank you for these comments and suggestions. We address the four points in the order raised below.

1. Prior work and the role of Shapley

Liao et al. (2025) establish early-error persistence; our offline Shapley analysis measures support from earlier steps to fixed later text in frozen multimodal chains. The three displayed chains illustrate persistent, fading, and weak cross-phase dependencies; permutation-sampling stability tests whether this contrast persists.

Thought-ICS (Samanta et al., ICML 2026) shares prefix-preserving suffix regeneration but uses textual self-diagnosis and one suffix regeneration per round. BANKROLL selects a restart step via single-step masking, scores short continuations with answer probes, and applies probe-based acceptance. We added the citation and clarified this distinction in Related Work.

Shapley and LOO analyze fixed later-text log-likelihood offline; online masking measures current-answer sensitivity with context intact, providing a lower-cost selection signal without predecessor-coalition evaluation.

2. Source of the MMMU-Pro gains

The MMMU-Pro gain is not explained solely by a weak Standard CoT baseline. BANKROLL scores 69.02% versus 54.62% (Standard), 62.72% (Concise), and 65.66% (DEER). More directly, w/o Refinance reaches 66.13%; across 1,730 questions, Refinance corrects 78 errors and reverses 28 correct answers, yielding +50 questions (+2.89 pp) as TpT_{\rm p} rises from 2,844 to 4,049.

Thinking stops at 4,096 tokens before an uncapped summary. Standard/BANKROLL limit-hit rates are Qwen 50.8%/21.2%, GLM 20.3%/10.8%, and InternVL 45.8%/12.9%; Standard Qwen has 0.40% no-letter extractions. TpT_{\rm p} includes discarded rollback work.

3. Component contributions and search budget

The components act at different stages: masking identifies where to restart, Refinance searches over short repairs, and Writeoff decides whether to keep one. BANKROLL scores 82.70/79.00/69.02 on M³CoT/MMStar/MMMU-Pro with 1,853/1,796/4,049 tokens. Removing Refinance lowers accuracy to 81.02/77.47/66.13 at 1,387/1,408/2,844 tokens; random candidate/pivot selection reaches 81.36/77.00/67.11 and 81.67/77.20/67.05. Removing masking gives 82.79/78.60/67.80 but costs 3,542/3,806/5,241 tokens. Removing Cashout adds 24.9% tokens and lowers both non-M³CoT scores; removing acceptance or last-probe override lowers accuracy at unchanged token counts.

The 14 hyperparameters were selected on Qwen3.5-9B development data, fixed before M³CoT test inference, and transferred unchanged. R=1 lowers M³CoT/MMStar accuracy to 80.24/77.93, while R=5 uses more tokens and reaches 82.18/78.20. Cashout-Cert improves the empirical accuracy--generation trade-off, but its thresholds provide no answer-error bound.

4. Full-chain baseline comparisons

We compare selection among complete chains with local repair from a retained prefix. Probe-select (2) samples two complete chains and chooses between them with BANKROLL’s answer probe; one-round Self-Refine independently generates and revises a response. Across 15 model--benchmark settings, BANKROLL is higher than Self-Refine in 14/15 settings, with 2.8--15.6% more reported generated tokens; the exception is Qwen3.5-9B on HallusionBench (81.28/1,941 vs. 81.39/1,826). It is higher than Probe-select in 13/15 settings and uses 38.0--57.0% fewer tokens. Probe-select is higher on GLM HallusionBench (75.92 vs. 75.50) and InternVL MMMU-Pro (56.00 vs. 55.90); its full-chain protocol generates 1.61--2.32× BANKROLL’s tokens. Counts follow each method’s actual protocol; BANKROLL’s TpT_{\rm p} includes answers, probes, discarded candidates, and rejected Writeoff work, while reused prefixes add no generated tokens.

The random-pivot control keeps the same candidate shortlist, truncation, continuation settings, and Writeoff rule. On Qwen3.5-9B, BANKROLL exceeds random pivot by 1.03, 1.80, and 1.97 points on M³CoT (82.70 vs. 81.67), MMStar (79.00 vs. 77.20), and MMMU-Pro (69.02 vs. 67.05), with token counts within 4% (1853/1861, 1796/1736, and 4049/4053).

The revised manuscript reports these results in Main Results and Core Ablations. The attribution distinctions and stability analysis are covered in Related Work, Analysis and Motivation, and the attribution appendix; generation controls, token accounting, and search-budget sensitivity are covered in the Experimental Setup, Implementation Details, and Hyperparameter Sensitivity appendices.

Comments on the paper

1 comment

Thank you for these comments and suggestions.

1. Complete-chain selection versus local repair

Probe-select (2) samples two complete chains and chooses between them with BANKROLL’s answer probe; Self-Refine independently generates and revises one response. BANKROLL retains the prefix before the selected rollback point, samples up to three short continuations, and completes only the selected one.

Across 15 model--benchmark settings, BANKROLL scores higher than Probe-select (2) in 13/15 and than Self-Refine in 14/15. Probe-select is higher on GLM HallusionBench (75.92 vs. 75.50) and InternVL MMMU-Pro (56.00 vs. 55.90); Self-Refine is higher on Qwen HallusionBench (81.39 vs. 81.28). Under the reported accounting, BANKROLL uses 38.0--57.0% fewer generated tokens than Probe-select but 2.8--15.6% more than Self-Refine. These results support retained-prefix repair without fully expanding discarded chains.

2. End-to-end inference cost

Generated-token counts do not capture prefill or masked-context recomputation. On the M³CoT test set, BANKROLL averages 69 s/example end to end, versus 50 s for Standard CoT (mean Tp=1853T_{\rm p}=1853 vs. 1431). This measurement includes answer probes, masked-context recomputation, and discarded short continuations; for this configuration, the lower token count does not translate into lower wall-clock time.

3. Attribution stability and online selection

The qualitative attribution patterns persist as the permutation budget increases. Across the three frozen chains and additional M³CoT development examples, early-step support sometimes extends into the summary, fades before it, or remains within the thinking segment, while individual cell strengths and signs vary.

We recomputed every target row under four settings: 1 pair/13, 4 pairs/13, 4 pairs/1301, and 4 pairs/2601, using each permutation and its reverse. The same row normalization and signed-square-root visualization were used throughout. In geography-3180, T19 remains positive into the summary for 7/7, 7/7, 7/7, and 6/7 targets; six targets stay positive throughout, while Y4 becomes negative in the last setting and the band fades with four pairs. Mathematics-989 and Temporal-commonsense-18 show no sustained earlier-step band into the summary. These results support a stable qualitative contrast in the offline attribution analysis: support from earlier steps can persist into later reasoning, fade before the summary, or remain local to the thinking segment. This pattern motivates local repair, while Refinance uses a distinct online fixed-letter masking score to select the pivot. The random-pivot comparison separately evaluates the contribution of this masking-based pivot-selection heuristic.

4. Implementation and development details

We count literal reflection-marker matches in paragraph-level thinking steps, rank positive-count steps with earlier positions breaking ties, and retain up to three candidates. Zero-count steps do not fill the shortlist; if all counts are zero, repair is skipped.

BANKROLL was developed on all 669 Qwen3.5-9B M³CoT development examples. Rules and hyperparameters were fixed before the 2,318-example test evaluation and transferred unchanged to the other benchmarks and models, without model-specific retuning. DEER and No-Think use development-selected thresholds 0.995 and 0.99; confidence is normalized over valid answer options. DEER stops when an intermediate probe reaches its threshold; No-Think uses the initial direct answer when it meets its threshold and otherwise runs Standard CoT.

The revised manuscript reports these results in the Experiments section and Table 1. The stability analysis is discussed in the Stability across Permutation Samples subsection; generation cost, development-split selection, candidate nomination, and baseline thresholds are documented in the Experimental Setup and Implementation Details appendices.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.