- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how many visual tokens or latents should be allocated to intermediate states in interleaved visual reasoning. Through capacity interventions on fixed reasoning trajectories, the authors find that reconstruction fidelity does not reliably predict downstream answer utility, that useful capacity varies across states, and that capacity choices interact across reasoning steps. VisCal-Ext trains an external capacity controller using trajectory-level utility while keeping the backbone frozen. VisCal-OPD moves capacity selection before visual generation and jointly optimizes capacity, visual content, and subsequent text through outcome-weighted branch distillation. Experiments across four models and eight transfer benchmarks report an average accuracy improvement of 5.14 percentage points at matched visual capacity and an estimated 65.1% reduction in intermediate visual capacity at the released models’ accuracy.
Critical strengths
Beyond measuring the relationship between capacity and answer utility, the authors include question-image controls, nuisance controls, and free-generation validation. The external-controller experiments further hold architecture and training procedures fixed while changing the supervision target, providing meaningful evidence for utility-based supervision and joint allocation. This is the main reason I lean toward acceptance. Results across four backbones and eight transfer benchmarks, together with Standard OPD, single-step branching, and capacity-only ablations, make the contribution more substantial than capacity tuning for a particular model. The appendix also reports end-to-end costs, allowing readers to distinguish reductions in visual capacity from actual computational savings.
Critical weaknesses
Standard OPD uses the native capacity policy, whereas VisCal-OPD trains across varying capacities. The comparison does not fully isolate adaptive allocation from the benefits of capacity-varied training. The capacity-only ablation freezes both visual and text generation. It therefore does not establish whether both components need adaptation or whether updating either one would suffice.
Questions for authors
Does VisCal-OPD outperform a state-independent capacity schedule trained with the same capacity-varied branches and supervision? With capacity learning fixed, how does updating only visual or only text generation compare with updating both?
4 comments
Thanks for your detailed responses, which partially solve my concern. I will maintain my score.
Dear Reviewer K1LC,
Thank you for your reply and for maintaining your positive score. We understand that a remaining concern may be that the two new comparisons in our response (the state-independent schedule and updating only one generator) were evaluated on the in-distribution Zebra-CoT split. To address this, we are evaluating the same variants on the eight transfer benchmarks. We will post the results here before the discussion period ends.
(Follow-up to our message above.)
Dear Reviewer K1LC,
Thank you for your patience. We evaluated the variants from our response on the eight transfer benchmarks at the original model's budget. The table gives VisCal-OPD's accuracy gain over each variant (pp).
| VisCal-OPD over | Model | 4 visual-state-centric benchmarks | All 8 benchmarks (95% CI) |
|---|---|---|---|
| Capacity + image generator | Bagel-Zebra-CoT | +1.63 | +1.71 [0.98, 2.43] |
| Capacity + text generator | Bagel-Zebra-CoT | +2.94 | +2.20 [1.47, 2.90] |
| State-independent schedule | Bagel-Zebra-CoT | +0.92 | +0.60 [−0.08, 1.27] |
| State-independent schedule | DeltaV | +1.95 | +1.38 [0.69, 2.06] |
As on the held-out split, both components need adaptation: updating either one alone does not suffice. VisCal-OPD stays significantly ahead of each single-generator variant.
Against the state-independent schedule, VisCal-OPD stays significantly ahead on DeltaV, so adaptive allocation adds a gain beyond capacity-varied training on the transfer benchmarks as well. On Bagel-Zebra-CoT, it is still ahead, but the gap is not significant. As with the main results (Sec. 6.2), this gap is larger on the four visual-state-centric benchmarks, which depend most on generated visual states.
We added the per-benchmark results for each variant to App. F.8 (Table 26) of the revised paper.
We hope these results address your remaining concern. Thank you again for your constructive suggestions.
Dear Reviewer K1LC,
Thank you for the careful and constructive review. We are grateful that you think our controlled comparisons provide meaningful evidence for utility-based supervision and joint allocation. We ran both suggested experiments and report them below:
W1/Q1: Adaptive allocation vs. capacity-varied training
Thank you for this helpful question. VisCal-OPD outperforms a state-independent capacity schedule trained with the same capacity-varied branches and supervision. In this new variant, the capacity head reads only the step index and the capacity price. It does not read the question or the current reasoning state. At a given step and price, it assigns the same capacity to every question. Everything else is the same as VisCal-OPD.
The table gives VisCal-OPD's accuracy gain over this variant on the same held-out Zebra-CoT split as Table 4 (pp, 95% CI). γ sets the visual budget: γ = 1 is the original model's budget, and γ = 0 is the lowest budget the model reaches.
VisCal-OPD is significantly ahead at every budget in both models, so adaptive allocation adds a gain beyond capacity-varied training.
We added this variant to Table 4, Sec. 6.4, and App. F.4.
W2/Q2: Updating only visual or only text generation
Thank you for this suggestion. At every tested budget, updating both generators is better than updating either one alone. With capacity learning kept the same as in VisCal-OPD, we trained two new variants: one updates only the image generator, and the other only the text generator. We used Bagel-Zebra-CoT because its image and text generators have separate parameters. In DeltaV, image and text generation share one backbone, so updating one also changes the other.
The table gives VisCal-OPD's accuracy gain over each variant on the same held-out Zebra-CoT split as Table 4 (Bagel-Zebra-CoT, pp, 95% CI; γ as in W1/Q1).
Both components need adaptation; updating either one alone does not suffice.
We added both variants to Table 4, Sec. 6.4, and App. F.4.