- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how an early judgment can continue to influence later reasoning and the final answer in multimodal models. It uses an offline Shapley analysis to examine this pattern, called answer inheritance. The paper then proposes BANKROLL, a training-free framework for stopping and repairing reasoning. Cashout uses intermediate answer probes to decide when to stop. Refinance uses local masking tests to select a rollback point and samples alternative continuations. Writeoff compares probe scores to decide whether to accept the repair. Experiments cover five benchmarks and three multimodal models. BANKROLL improves performance over standard CoT in all 15 settings. Ablation studies further examine the stopping rules, rollback selection, and repair acceptance.
Critical strengths
1.The repair method is well motivated, and the ablations support its design. Rolling back before a selected step allows the model to revise an earlier visual judgment and the reasoning that follows it. This requires no additional training. The random-pivot control keeps the candidate set and repair procedure unchanged. Selecting the pivot with the proposed score improves accuracy by 1.03 and 1.80 percentage points on M³CoT and MMStar. This provides useful evidence that the rollback position matters.
2.The evaluation is broad and shows consistent gains over standard CoT. The experiments cover reasoning, visual hallucination, and text recognition across three models. The same BANKROLL settings are used across models and benchmarks. On MMMU-Pro, accuracy rises from 54.62% to 69.02% for Qwen3.5-9B and from 39.13% to 49.77% for GLM-4.1V-9B-Thinking. These results suggest that the benefit extends beyond a single model or task.
Critical weaknesses
-
The repair scores are not well tested. The masking score shows how much a step supports the current answer. A correct step can also receive a high score. For open-ended tasks, masking can change the trial answer, so the scores may compare different answers. Writeoff also uses a score recorded before the repair is complete. That score may refer to a different answer from the final output. The paper does not show how often the selected steps contain errors or whether the saved scores help choose correct repairs.
-
It is unclear whether local rollback is better than simpler alternatives. The paper does not compare with self-refinement. It also lacks a baseline that samples full reasoning chains and picks the one with the highest probe score. Appendix F shows only one example using a reflection prompt. The random-pivot test compares rollback positions within BANKROLL, but does not address these other methods. On M³CoT, using one continuation gives 80.24% accuracy, below 81.02% without Refinance. A stronger comparison is needed to show what local rollback adds beyond sampling and selecting answers.
Questions for authors
-
How often does the selected step contain an actual error? Does scoring the completed repair work better than using the earlier score?
-
Could you compare BANKROLL with a self-refinement baseline and with full-chain sampling selected by the same answer-probe score? These comparisons would help clarify the benefit of local rollback.
5 comments
Reply to Author
Thank you for the additional experiments and clarifications. The new baseline comparisons address much of my concern about the value of local repair. I have two more points that I’m unclear about.
- The response says Self-Refine runs only when BANKROLL triggers repair. However, Section 4.1 says it generates and revises an answer for every example. Which setup produced the reported results? Please make the paper and response consistent.
- Does rescoring completed repairs give similar results on CC-OCR? The reported comparison covers only multiple-choice tasks. My concern about scores referring to different answers mainly applies to open-ended tasks.
Thank you for the follow-up questions. We clarify the Self-Refine protocol first, and then report the separate open-ended check on CC-OCR.
1. Self-Refine protocol
The reported Self-Refine results use the all-example protocol described in Section 4.1. For every example, Self-Refine first generates an initial response from the image and question, and then performs one round of self-checking and revision; both generations are included in its reported token count. We apologize for the confusion caused by our imprecise earlier wording. Self-Refine is run on every example, independently of whether BANKROLL repairs that example, consistent with Section 4.1.
2. Rescoring completed repairs on CC-OCR
On CC-OCR, post-completion rescoring gives essentially the same aggregate result: the official score changes from 79.55 under selection-time scoring to 79.59 under post-completion scoring (+0.04). To isolate score timing, we kept the original chain, selected continuation, and final-answer handling fixed, changing only the score supplied to Writeoff’s acceptance comparison.
| Benchmark | Metric | Selection-time | Post-completion | Change |
|---|---|---|---|---|
| CC-OCR | Official 0--100 score | 79.55 | 79.59 | +0.04 |
| M$^3$CoT | Accuracy | 82.70% | 82.61% | $-$0.09 pp |
| MMStar | Accuracy | 79.00% | 79.00% | 0.00 pp |
| MMMU-Pro | Accuracy | 69.02% | 69.02% | 0.00 pp |
The CC-OCR comparison directly tests whether the selection-time score can correspond to a different answer after completion. Among 516 repair decisions, 142 had no valid decision-point probe readout, leaving 374 comparable cases. The decision-point and post-completion probe answers differed in 115 of these cases (30.7%). Despite this local mismatch, only 19 of the 512 repairs accepted under selection-time scoring (3.7%) would receive a different accept/rollback decision when the post-completion score was used. Thus, probe answers can change between selection and completion, but post-completion rescoring does not show a consistent aggregate advantage.
Thank you for these comments and suggestions. We address the two points in the order raised below.
1. Selected-step and repair-score evidence
In a manual check of 100 randomly sampled repair records, 23 selected steps contained an actual error and 77 did not. A selected step is an intervention point, not an error label: Refinance regenerates that step and its suffix.
Replacing score-based pivot selection with uniform random selection from the same shortlist reduces accuracy from 82.70% to 81.67% on M³CoT and from 79.00% to 77.20% on MMStar (Table 2). This supports the score-based selection heuristic. Removing Writeoff’s score-based acceptance check lowers accuracy from 82.70% to 82.53% on M³CoT and from 79.00% to 78.27% on MMStar. Writeoff uses the probe score recorded at candidate selection, whose predicted answer may differ from the completed repair’s final answer. Replacing this saved score with a score recomputed after the repair is complete changes M³CoT from 82.70% to 82.61% and leaves MMStar and MMMU-Pro unchanged.
We clarify these points in Writeoff and Core Ablations, with supporting analyses in the From Attribution to Local Repair appendix.
2. Local rollback versus simpler alternatives
Probe-select (2) samples two complete chains and uses the same answer probe as BANKROLL. For the reported comparison, Self-Refine generates an initial response for every example and performs one round of self-checking and revision; both generations are counted in Tp.
Across 15 model–benchmark settings, BANKROLL scores higher than Self-Refine in 14/15 and than Probe-select (2) in 13/15. Self-Refine is higher on Qwen HallusionBench (81.39% vs. 81.28%); Probe-select is higher on GLM HallusionBench (75.92% vs. 75.50%) and InternVL MMMU-Pro (56.00% vs. 55.90%). BANKROLL uses 2.8–15.6% more tokens than Self-Refine but 38.0–57.0% fewer than Probe-select. The results support retained-prefix repair, with the Self-Refine comparison carrying additional generation cost.
With (R,s)=(1,3), accuracy is 80.24% on M³CoT and 77.93% on MMStar, compared with 81.02% and 77.47% without Refinance; the default (R,s)=(3,3) reaches 82.70% and 79.00%. Thus, the current evidence supports selecting among multiple local continuations rather than rollback alone. Because changing the pivot also changes the continuation, random pivoting is a same-procedure control rather than a paired-content comparison.
These comparisons are reported in Main Results and Core Ablations, with candidate-count and random-pivot analyses in the Hyperparameter Sensitivity and From Attribution to Local Repair appendices.