- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes ActVis, a training-free framework for maintaining visual evidence during multimodal chain-of-thought reasoning. Instead of simply accumulating intermediate observations, ActVis introduces an editable Evidence Canvas that allows evidence to be retained, revised, merged, or removed as reasoning progresses. It also uses an entropy-guided evidence selection mechanism and a Step Audit to assess the usefulness and reliability of new observations.
The authors evaluate ActVis on several MLLM backbones across seven reasoning tasks, including geometry, graph reasoning, chess, and general multimodal benchmarks. The reported results show improvements over several existing Visual CoT baselines, with particularly large gains on some graph reasoning tasks. Additional experiments investigate the effects of structured evidence maintenance, entropy-guided selection, and different canvas representations.
Critical strengths
-
Meaningful motivation. I find the idea of actively maintaining visual evidence reasonable and intuitive. During multi-step reasoning, it is not enough to simply accumulate observations. The model should also preserve useful evidence and revise or remove incorrect or outdated information. I think this is a meaningful problem that deserves further investigation.
-
Practical framework with promising results. ActVis provides a training-free approach that can be applied to different frozen MLLMs. The authors also conduct a reasonable number of experiments across several benchmarks and backbones. Some improvements are substantial, suggesting that explicit evidence management can be useful for visual reasoning.
Critical weaknesses
- Unclear evaluation reliability due to discrepancies in baseline performance and experimental settings.
I have concerns about the large discrepancies between the reported baseline results and publicly available performance numbers. For example, Table 1 reports only 27.6% MathVista accuracy for Qwen2.5-VL-7B-Instruct, while the official model card reports 68.2% on MathVista testmini. Even after applying ActVis, the best reported result for this backbone is only 48.0%.
These results may come from different benchmark subsets, prompts, or evaluation settings, but the paper does not explain this clearly. The discrepancy makes it difficult to assess how meaningful the reported improvements are.
There is also an inconsistency in the maximum context length: the main paper reports 8,192 tokens, while the supplementary material specifies 65,536. This is particularly important because context exhaustion is used to explain the poor performance of the append-only baseline. I think these evaluation details need to be clarified before the experimental conclusions can be fully trusted.
- Limited novelty and insufficient evidence isolating the contribution of active maintenance.
Although the motivation is reasonable, many of the underlying ideas are closely related to existing work, including Visual Sketchpad, See It, Say It, Sorted, Trace, Verify, and Correct, EVRI, and Entropy-Gradient Grounding. ActVis extends these ideas by maintaining an editable evidence state, but I consider the overall contribution relatively incremental.
More importantly, the experiments do not sufficiently isolate the benefit of active evidence maintenance itself. The comparison with append-only memory may largely reflect the advantage of a shorter, more compact context rather than the proposed revision and correction mechanisms. A simple fixed-budget memory baseline, under comparable inference budgets, would make the contribution much clearer.
I also have concerns about the entropy-guided selection mechanism. Reducing predictive entropy does not necessarily mean that the new evidence is correct. Incorrect evidence may also make the model more confident in a wrong answer. Although the authors introduce additional confidence filtering and a Step Audit, these mechanisms still rely heavily on the same MLLM. They may help in ambiguous cases, but the paper does not clearly demonstrate when they successfully correct errors rather than reinforce them.
I would like to see stronger evidence-level or intervention-based experiments, rather than relying mainly on improvements in final accuracy, to establish the actual contribution of these mechanisms.
Questions for authors
-
Baseline performance. Could the authors explain why the reported backbone performance differs so substantially from publicly available results? For example, Qwen2.5-VL-7B-Instruct achieves 27.6% on MathVista in this paper, compared with 68.2% on the officially reported testmini split. Which dataset splits, prompts, image preprocessing settings, and evaluation protocols were used? Are these results directly comparable?
-
Context length and append-only comparison. The main paper specifies a maximum context length of 8,192 tokens, while Appendix A.1 reports 65,536. Which setting was actually used? How many append-only trajectories encountered truncation or context-length failures? Could the authors provide a controlled comparison against a simple fixed-budget memory baseline to separate the effect of active maintenance from context compression?
-
Contribution of maintenance and verification. Could the authors provide more direct evidence that REVISE, MERGE, and DELETE improve reasoning? For example, how frequently are these operations used, and how often do they successfully correct erroneous evidence? An intervention experiment with deliberately corrupted or replaced evidence would be particularly informative.
-
Entropy-guided selection. How well does entropy reduction correlate with the correctness or usefulness of selected evidence? Since the same MLLM is used for reasoning and auditing, how does the framework avoid reinforcing confident but incorrect predictions? Also, the supplementary material describes both logits-based entropy estimation and a heuristic uncertainty proxy. Which was used in the main experiments? For open-ended questions, how well does the top-128 next-token approximation represent uncertainty over complete answers?
-
Tool contributions and result consistency. Could the authors clarify the capabilities of the external visual tools on graph and symbolic reasoning tasks, particularly whether they can directly produce algorithmic results? This would help separate tool-based computation from evidence maintenance. Also, Table 1 reports 91.34% Maxflow accuracy for Qwen2.5-VL-7B with ActVis (HTML), while Table 2 reports 82.03%. Which value is correct?
We thank the reviewer for the detailed and constructive comments.
Q1 While preparing this response we found that image loading had failed in our baseline reproduction for the multimodal tasks, so those runs received text-only input. This lowered the numbers of both ActVis and the baselines. We re-ran the affected comparisons and updated Tables 1-4 and Figure 3.
Q2 Context length and append-only comparison.
We have clarified our experimental setup: the maximum output length per response is 8,192, and the maximum total context length per task is 65,536. All other numerical discrepancies have been corrected. The baseline models we compare against all use Qwen2.5, Qwen3, and InternVL as their base models. The agent framework involves multiple reasoning steps, all of which are performed by the base model without any API calls. The inference GPU we used is an RTX 3090, with a maximum VRAM capacity of 24 GB per card. In Table 1, the Visual Sketchpad itself is a non-structured context compression baseline. The experimental results in Section 4.2 show that the lack of constraints in context compression actually reduces the intrinsic reasoning ability of the base model. Besides, We have refined the original Table 2 in Section 4.2 with a comparison of ActVis against add-only and sliding-window strategies, with structured evidence maintenance and all other components held constant. For all methods, generated output was capped at 8,192 tokens per call, and the combined input–output context window was capped at 65,536 tokens. Under these limits, ActVis selectively maintains key evidence and improves accuracy on several tasks. Table 2 also suggests that an overly long context can interfere with reasoning, while retaining only the latest observations can discard critical evidence and lead to errors. Section 4.3 further illustrates this with a case study on predicting the winner from a chess position (Figure 4).
Q3 Contribution of maintenance and verification. We counted the usage frequency of ADD, RETAIN, REVISE, DELETE based on the Qwen3 model in 7 test tasks. The results are as follows. It can be seen that RETAIN and ADD account for the largest proportion. ┌─────────┬─────────────────┬─────────────────┬─────────────────┬─────────────────┐ │ Format │ ADD │ RETAIN │ REVISE │ DELETE │ ├─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┤ │ JSON │ 337 (33.01%) │ 590 (57.79%) │ 74 (7.25%) │ 20 (1.96%) │ │ HTML │ 298 (28.35%) │ 667 (63.46%) │ 67 (6.37%) │ 19 (1.81%) │ ├─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┤ │ Overall │ 635 (30.65%) │ 1257 (60.67%) │ 141 (6.81%) │ 39 (1.88%) │ └─────────┴─────────────────┴─────────────────┴─────────────────┴─────────────────┘
Q4 Entropy-guided selection. For deployable open-source models, we use rule-based entropy selection. For closed-source models, since the internal states of the models are unavailable, the method degrades to an LLM-as-a-judge heuristic proxy mode. All baseline models selected in the paper are open-source models, so no other models are used for scoring. In addition, Table 4 also shows the performance of the models on different tasks when the entropy-guided selection mechanism is removed. The experimental results show that removing the gating mechanism degrades the model to some extent into an “add-only” mode, introducing reasoning noise and potential misjudgments, which makes it difficult to improve model accuracy. However, our method currently indeed cannot filter out incorrect but highly confident reasoning. We sincerely thank the reviewer for this comment; this is a highly research-worthy issue and one we will further address in future work.
Q5 Tool contributions and result consistency. Different tasks require different tools. Tasks such as drawing auxiliary lines for geometry problems and constructing graph connectivity are well suited to being drawn directly with code. In addition to Python tools, our proprietary vision tools include models such as SAM, DepthAnything, and OCR. This is because our general vision tasks involve determining the relative positions of objects (front, back, left, and right), counting objects, recognizing and comparing numbers in images, and image stitching. We hope to place greater emphasis on the model’s own visual reasoning rather than purely text-based code reasoning, and directly invoking vision tools can reduce token consumption and produce richer evidence. However, statistics show that approximately 6% of errors stem from incorrect observations obtained after tool invocation. Therefore, the tools themselves also exhibit certain execution deviations, such as inaccurate localization, which hinder reasoning.