acceptodds
Under review as a conference paper at ICLR 2027

ActVis: Active Maintenance of Structured Visual Evidence for Efficient Visual Chain-of-Thought Reasoning

Abstract

Multimodal chain-of-thought reasoning requires models to acquire and reuse visual evidence across multiple inference steps. Existing training-free Visual CoT methods mainly optimize when and where new visual evidence should be acquired, while previously acquired observations are typically appended to a linear reasoning trajectory without explicit maintenance. As reasoning proceeds, useful evidence may become obscured, whereas redundant, outdated, or conflicting observations continue to occupy the multimodal context. We introduce ActVis, a training-free framework for active visual evidence maintenance. ActVis organizes task-critical observations in a structured and editable Evidence Canvas, where each evidence unit remains grounded in its visual source and can be added, retained, revised, merged, or deleted as reasoning evolves. ActVis further employs an entropy-guided evidence gate to estimate the marginal utility of candidate observations based on predictive uncertainty, retaining informative evidence, filtering redundancy, and revising conflicts. Extensive experiments across multiple multimodal large language models and diverse reasoning tasks demonstrate that ActVis consistently improves reasoning accuracy over linear text–image reasoning and training-free Visual CoT baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal chain-of-thought, MLLMs

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_3LbxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ActVis, a training-free framework for maintaining visual evidence during multimodal chain-of-thought reasoning. Instead of simply accumulating intermediate observations, ActVis introduces an editable Evidence Canvas that allows evidence to be retained, revised, merged, or removed as reasoning progresses. It also uses an entropy-guided evidence selection mechanism and a Step Audit to assess the usefulness and reliability of new observations.

The authors evaluate ActVis on several MLLM backbones across seven reasoning tasks, including geometry, graph reasoning, chess, and general multimodal benchmarks. The reported results show improvements over several existing Visual CoT baselines, with particularly large gains on some graph reasoning tasks. Additional experiments investigate the effects of structured evidence maintenance, entropy-guided selection, and different canvas representations.

Critical strengths
  1. Meaningful motivation. I find the idea of actively maintaining visual evidence reasonable and intuitive. During multi-step reasoning, it is not enough to simply accumulate observations. The model should also preserve useful evidence and revise or remove incorrect or outdated information. I think this is a meaningful problem that deserves further investigation.

  2. Practical framework with promising results. ActVis provides a training-free approach that can be applied to different frozen MLLMs. The authors also conduct a reasonable number of experiments across several benchmarks and backbones. Some improvements are substantial, suggesting that explicit evidence management can be useful for visual reasoning.

Critical weaknesses
  1. Unclear evaluation reliability due to discrepancies in baseline performance and experimental settings.

I have concerns about the large discrepancies between the reported baseline results and publicly available performance numbers. For example, Table 1 reports only 27.6% MathVista accuracy for Qwen2.5-VL-7B-Instruct, while the official model card reports 68.2% on MathVista testmini. Even after applying ActVis, the best reported result for this backbone is only 48.0%.

These results may come from different benchmark subsets, prompts, or evaluation settings, but the paper does not explain this clearly. The discrepancy makes it difficult to assess how meaningful the reported improvements are.

There is also an inconsistency in the maximum context length: the main paper reports 8,192 tokens, while the supplementary material specifies 65,536. This is particularly important because context exhaustion is used to explain the poor performance of the append-only baseline. I think these evaluation details need to be clarified before the experimental conclusions can be fully trusted.

  1. Limited novelty and insufficient evidence isolating the contribution of active maintenance.

Although the motivation is reasonable, many of the underlying ideas are closely related to existing work, including Visual Sketchpad, See It, Say It, Sorted, Trace, Verify, and Correct, EVRI, and Entropy-Gradient Grounding. ActVis extends these ideas by maintaining an editable evidence state, but I consider the overall contribution relatively incremental.

More importantly, the experiments do not sufficiently isolate the benefit of active evidence maintenance itself. The comparison with append-only memory may largely reflect the advantage of a shorter, more compact context rather than the proposed revision and correction mechanisms. A simple fixed-budget memory baseline, under comparable inference budgets, would make the contribution much clearer.

I also have concerns about the entropy-guided selection mechanism. Reducing predictive entropy does not necessarily mean that the new evidence is correct. Incorrect evidence may also make the model more confident in a wrong answer. Although the authors introduce additional confidence filtering and a Step Audit, these mechanisms still rely heavily on the same MLLM. They may help in ambiguous cases, but the paper does not clearly demonstrate when they successfully correct errors rather than reinforce them.

I would like to see stronger evidence-level or intervention-based experiments, rather than relying mainly on improvements in final accuracy, to establish the actual contribution of these mechanisms.

Questions for authors
  1. Baseline performance. Could the authors explain why the reported backbone performance differs so substantially from publicly available results? For example, Qwen2.5-VL-7B-Instruct achieves 27.6% on MathVista in this paper, compared with 68.2% on the officially reported testmini split. Which dataset splits, prompts, image preprocessing settings, and evaluation protocols were used? Are these results directly comparable?

  2. Context length and append-only comparison. The main paper specifies a maximum context length of 8,192 tokens, while Appendix A.1 reports 65,536. Which setting was actually used? How many append-only trajectories encountered truncation or context-length failures? Could the authors provide a controlled comparison against a simple fixed-budget memory baseline to separate the effect of active maintenance from context compression?

  3. Contribution of maintenance and verification. Could the authors provide more direct evidence that REVISE, MERGE, and DELETE improve reasoning? For example, how frequently are these operations used, and how often do they successfully correct erroneous evidence? An intervention experiment with deliberately corrupted or replaced evidence would be particularly informative.

  4. Entropy-guided selection. How well does entropy reduction correlate with the correctness or usefulness of selected evidence? Since the same MLLM is used for reasoning and auditing, how does the framework avoid reinforcing confident but incorrect predictions? Also, the supplementary material describes both logits-based entropy estimation and a heuristic uncertainty proxy. Which was used in the main experiments? For open-ended questions, how well does the top-128 next-token approximation represent uncertainty over complete answers?

  5. Tool contributions and result consistency. Could the authors clarify the capabilities of the external visual tools on graph and symbolic reasoning tasks, particularly whether they can directly produce algorithmic results? This would help separate tool-based computation from evidence maintenance. Also, Table 1 reports 91.34% Maxflow accuracy for Qwen2.5-VL-7B with ActVis (HTML), while Table 2 reports 82.03%. Which value is correct?

1 comment

We thank the reviewer for the detailed and constructive comments.

Q1 While preparing this response we found that image loading had failed in our baseline reproduction for the multimodal tasks, so those runs received text-only input. This lowered the numbers of both ActVis and the baselines. We re-ran the affected comparisons and updated Tables 1-4 and Figure 3.

Q2 Context length and append-only comparison.

We have clarified our experimental setup: the maximum output length per response is 8,192, and the maximum total context length per task is 65,536. All other numerical discrepancies have been corrected. The baseline models we compare against all use Qwen2.5, Qwen3, and InternVL as their base models. The agent framework involves multiple reasoning steps, all of which are performed by the base model without any API calls. The inference GPU we used is an RTX 3090, with a maximum VRAM capacity of 24 GB per card. In Table 1, the Visual Sketchpad itself is a non-structured context compression baseline. The experimental results in Section 4.2 show that the lack of constraints in context compression actually reduces the intrinsic reasoning ability of the base model. Besides, We have refined the original Table 2 in Section 4.2 with a comparison of ActVis against add-only and sliding-window strategies, with structured evidence maintenance and all other components held constant. For all methods, generated output was capped at 8,192 tokens per call, and the combined input–output context window was capped at 65,536 tokens. Under these limits, ActVis selectively maintains key evidence and improves accuracy on several tasks. Table 2 also suggests that an overly long context can interfere with reasoning, while retaining only the latest observations can discard critical evidence and lead to errors. Section 4.3 further illustrates this with a case study on predicting the winner from a chess position (Figure 4).

Q3 Contribution of maintenance and verification. We counted the usage frequency of ADD, RETAIN, REVISE, DELETE based on the Qwen3 model in 7 test tasks. The results are as follows. It can be seen that RETAIN and ADD account for the largest proportion. ┌─────────┬─────────────────┬─────────────────┬─────────────────┬─────────────────┐ │ Format │ ADD │ RETAIN │ REVISE │ DELETE │ ├─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┤ │ JSON │ 337 (33.01%) │ 590 (57.79%) │ 74 (7.25%) │ 20 (1.96%) │ │ HTML │ 298 (28.35%) │ 667 (63.46%) │ 67 (6.37%) │ 19 (1.81%) │ ├─────────┼─────────────────┼─────────────────┼─────────────────┼─────────────────┤ │ Overall │ 635 (30.65%) │ 1257 (60.67%) │ 141 (6.81%) │ 39 (1.88%) │ └─────────┴─────────────────┴─────────────────┴─────────────────┴─────────────────┘

Q4 Entropy-guided selection. For deployable open-source models, we use rule-based entropy selection. For closed-source models, since the internal states of the models are unavailable, the method degrades to an LLM-as-a-judge heuristic proxy mode. All baseline models selected in the paper are open-source models, so no other models are used for scoring. In addition, Table 4 also shows the performance of the models on different tasks when the entropy-guided selection mechanism is removed. The experimental results show that removing the gating mechanism degrades the model to some extent into an “add-only” mode, introducing reasoning noise and potential misjudgments, which makes it difficult to improve model accuracy. However, our method currently indeed cannot filter out incorrect but highly confident reasoning. We sincerely thank the reviewer for this comment; this is a highly research-worthy issue and one we will further address in future work.

Q5 Tool contributions and result consistency. Different tasks require different tools. Tasks such as drawing auxiliary lines for geometry problems and constructing graph connectivity are well suited to being drawn directly with code. In addition to Python tools, our proprietary vision tools include models such as SAM, DepthAnything, and OCR. This is because our general vision tasks involve determining the relative positions of objects (front, back, left, and right), counting objects, recognizing and comparing numbers in images, and image stitching. We hope to place greater emphasis on the model’s own visual reasoning rather than purely text-based code reasoning, and directly invoking vision tools can reduce token consumption and produce richer evidence. However, statistics show that approximately 6% of errors stem from incorrect observations obtained after tool invocation. Therefore, the tools themselves also exhibit certain execution deviations, such as inaccurate localization, which hinder reasoning.

Reviewer_DuUoOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ActVis maintains a small, editable evidence canvas during visual reasoning rather than accumulating the full observation history. Each entry records its visual source and interpretation, and can be revised, merged or deleted. A self-assessed score filters tool actions; an entropy gate and Step Audit then decide whether new evidence should enter the canvas.

The framework uses frozen MLLMs and retains at most three active evidence entries. Experiments on three models cover graph, geometry and chess reasoning alongside MathVista, Mantis and MMVP. The strongest reported gains are on graph tasks and against an append-only evidence baseline.

Critical strengths

Keeping visual evidence editable rather than simply accumulating it is a reasonable idea. Acquiring an observation and deciding whether it remains useful are different problems, and the canvas makes that distinction explicit. The graph results are promising: Qwen3-VL reaches 100% on Maxflow and Connectivity, with some gains transferring to InternVL3.5. These results suggest that a compact state can help tasks with clear intermediate facts.

Critical weaknesses

The append-only comparison is my main concern. On Qwen3-VL, its MathVista accuracy is 20.4 versus 67.8 for the base model. The paper attributes the degradation partly to exhausting the 8K context limit. This makes it difficult to tell whether the large gains come from active evidence maintenance or simply avoiding overflow. A longer-context append-only run and a three-entry sliding window would be more informative controls. The paper also does not show how frequently revision, merging and deletion occur or what happens when each is disabled. Those comparisons are needed to support the claimed distinction from bounded memory.

The efficiency claim is not established either. Chess reasoning consumes roughly 77–94K tokens versus 8.7K for the base setting; savings relative to an ungated canvas do not imply lower overall cost. It is unclear whether the additional answer-probe passes are fully counted, and no wall-clock comparison is provided. Performance is also uneven: InternVL3.5 Geometry falls from 54.17 to 35.98 with the JSON canvas, and both formats trail the base model on MathVista. Together with the missing implementation details and inconsistent table entries, these issues prevent a confident assessment of the mechanism and its benefits.

Questions for authors

How much of the gain remains against longer-context append-only and bounded sliding-window baselines? Could you report the frequency and accuracy impact of revision, merging and deletion? This would most directly test whether active maintenance contributes beyond context control.

Please supply the referenced appendix, including prompts, thresholds and the open-ended entropy probe. Can you reconcile the table discrepancies and report end-to-end costs including probes, audits and tools? An explanation of the InternVL regressions would also help.

1 comment

We thank the reviewer for the detailed and constructive comments.

Summary of changes. While preparing this response we found that image loading had failed in our baseline reproduction for the multimodal tasks, so those runs received text-only input. This lowered the numbers of both ActVis and the baselines. We re-ran the affected comparisons and updated Tables 1-4 and Figure 3.

Q1 We previously treated Visual Sketchpad as an append-only baseline in Table 2, which was imprecise. Visual Sketchpad continually appends new observations, but updates its reasoning history in an unstructured manner. We have therefore replaced the original Table 2 in Section 4.2 with a comparison of ActVis against add-only and sliding-window strategies, with structured evidence maintenance and all other components held constant. For all methods, generated output was capped at 8,192 tokens per call, and the combined input–output context window was capped at 65,536 tokens. Under these limits, ActVis selectively maintains key evidence and improves accuracy on several tasks. Table 2 also suggests that an overly long context can interfere with reasoning, while retaining only the latest observations can discard critical evidence and lead to errors. Section 4.3 further illustrates this with a case study on predicting the winner from a chess position (Figure 4).

Q2 We have submitted the updated version of the appendix, which includes our prompt words and the trajectories corresponding to the cases in 4.3. Furthermore, we have updated table 3 in 4.3, which includes the average canvas length, average output length, and the average total token consumption per task for each of the three methods. The experimental results prove that our maintenance method can incorporate denser reasoning evidence within a concise reasoning context. However, since ActVis needs the canvas organization mechanisms, more Agent interaction prompts are required. Therefore, in terms of the total tokens, we are not the best.

Reviewer_ucDpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ActVis, a training-free framework for active visual evidence maintenance in multimodal chain-of-thought reasoning. The authors argue that existing Visual CoT methods primarily focus on acquiring new visual information, while previously obtained evidence is often accumulated without explicit management, potentially introducing redundancy, outdated observations, and conflicting information.

To address this issue, ActVis introduces a structured and editable Evidence Canvas that maintains task-relevant visual observations through five operations: ADD, RETAIN, REVISE, MERGE, and DELETE. The framework further incorporates an entropy-guided evidence selection mechanism to estimate the predictive utility of new observations, together with a Step Audit module to assess their reliability. The maintained evidence state is then reinjected into subsequent reasoning steps without updating model parameters.

The authors evaluate ActVis on seven benchmarks covering geometry, graph reasoning, game-state analysis, and general multimodal understanding, using three MLLM backbones. Results show substantial accuracy improvements on several tasks compared with conventional Visual CoT baselines. Additional experiments compare active maintenance against append-only accumulation, examine JSON and HTML evidence representations, and analyze the effects of structured evidence and entropy-guided selection on accuracy and inference efficiency.

Critical strengths

1. A practical and training-free framework for managing visual evidence during reasoning.

The paper presents a practical framework that organizes intermediate visual observations into a structured, editable Evidence Canvas, enabling evidence reuse and refinement throughout multi-step reasoning. The integration of evidence selection, validation, and maintenance into a unified inference-time workflow is potentially useful, particularly because it requires no additional model training or architectural modifications. Despite the limited methodological novelty, this design offers a reasonably general and interpretable way to manage intermediate visual information in existing MLLMs.

2. Substantial empirical improvements across multiple models and reasoning tasks.

ActVis demonstrates notable accuracy improvements across several challenging reasoning benchmarks and different MLLM backbones. In particular, the substantial gains on geometry, graph reasoning, and multi-image understanding tasks suggest that explicitly managing intermediate evidence can be beneficial. The evaluation also includes comparisons with representative Visual CoT methods, append-only evidence accumulation, and component ablations. Although these experiments do not fully isolate the contributions of individual mechanisms, the observed improvements provide meaningful evidence of the practical effectiveness of the overall framework.

Critical weaknesses

1. Insufficiently justified motivation and limited methodological novelty.

The paper builds on the well-established problem of visual forgetting in multimodal reasoning, but does not provide sufficient evidence that the accumulation of redundant, outdated, or conflicting visual evidence is a significant and distinct bottleneck in existing Visual CoT methods. Moreover, the proposed ADD/RETAIN/REVISE/MERGE/DELETE operations largely resemble existing dynamic memory management mechanisms and are primarily implemented through LLM-based state reconstruction and heuristic filtering, without a clearly novel vision-specific maintenance algorithm. The distinction from existing agent memory frameworks is insufficiently established. Consequently, both the necessity and methodological novelty of the proposed approach remain unclear.

2. Insufficient experimental evidence to isolate the benefits of active evidence maintenance.

Although ActVis achieves substantial accuracy improvements, the experiments do not convincingly demonstrate that these gains originate from the proposed evidence maintenance mechanism. The main comparison against append-only accumulation is potentially confounded by context-length limitations, as acknowledged by the authors. Comparisons with simple context compression, sliding-window memory, or fixed-budget evidence selection under matched computational budgets are missing. Furthermore, the paper lacks operation-level ablations demonstrating the necessity of REVISE, MERGE, and DELETE, as well as direct evaluations of evidence correction and conflict resolution. The inconsistent context-length settings reported in the main paper and supplementary material further complicate interpretation. Therefore, the experiments do not adequately establish that active evidence maintenance provides benefits beyond conventional context management and evidence filtering.

Questions for authors
  1. Motivation and problem justification (Clarification). The paper argues that redundant, outdated, or conflicting visual evidence is a key bottleneck in existing Visual CoT methods. Can the authors provide empirical evidence showing how frequently these issues occur and how much they contribute to reasoning failures? In particular, how can these failures be distinguished from general visual forgetting or context-length limitations?

  2. Methodological novelty and evidence maintenance (Clarification). The proposed ADD, RETAIN, REVISE, MERGE, and DELETE operations resemble existing dynamic memory management mechanisms in LLM agents. What are the specific vision-oriented designs that distinguish ActVis from these approaches? Since the supplementary material suggests that these operations are primarily implemented through LLM-based canvas reconstruction, what mechanisms ensure reliable evidence revision, merging, and conflict resolution beyond prompt-based instructions?

  3. Necessity of active maintenance (Additional experimental evidence). The substantial improvements over append-only baselines may partly result from reducing context length rather than actively revising or removing evidence. Can the authors compare ActVis against simpler alternatives, such as sliding-window memory, fixed-budget evidence retention, or LLM-based history summarization, under comparable context and computational budgets? Such comparisons are important for establishing whether active evidence maintenance is necessary.

  4. Ablation of key mechanisms (Additional experimental evidence). While the paper evaluates structured evidence and entropy-guided selection, it does not isolate the contributions of individual maintenance operations. Can the authors report ablations that independently disable REVISE, MERGE, and DELETE, along with statistics on their usage and effectiveness? Additionally, how much improvement does the entropy-guided gate provide over simpler relevance- or confidence-based evidence selection?

  5. Experimental consistency and fairness (Clarification). There appear to be inconsistencies between the main paper and supplementary material: the maximum context length is reported as 8,192 and 65,536, respectively, while the Qwen2.5-VL ActVis-HTML Maxflow accuracy differs between Tables 1 and 2 (91.34% vs. 82.03%). Could the authors clarify the actual experimental settings and correct these discrepancies? Furthermore, how are additional MLLM calls, token budgets, and failed executions accounted for when comparing ActVis with the baselines, particularly given the reported memory constraints of DaP-ICoT?

3 comments

We sincerely thank the reviewer for the detailed and constructive comments.

Summary of changes. While preparing this response, we found that image loading had failed in our baseline reproduction for the multimodal tasks, so those runs received text-only input, which lowered the numbers of both ActVis and the baselines. We re-ran the affected comparisons and updated Tables 1-4 and Figure 3.

Q1. Motivation and problem justification

We re-explained the example in the Introduction. In Figure 1, a model first extracts the graph's edges correctly. In a later step it contradicts them (it states that node 2 has no outgoing edges). This wrong statement stays in the context, and every later augmenting-path step conditions on it.

Failure statistics. We manually analyzed 75 failed Visual Sketchpad trajectories on Qwen3-VL-8B:

  • 52% began with an incorrect visual observation that was never corrected later, which is the persistence of a wrong claim that active maintenance targets.
  • 9% came from visual forgetting or deviation from an initially correct observation, the case drawn in Figure 1.
  • 13.33% were calculation or tool errors, and about 6% hit the token or reply budget.

Our framework filters out initial incorrect perceptions with low confidence through evidence maintenance and gating, reducing the proportion of trajectories that continue on an erroneous reasoning path.

Q2. Novelty and reliability of evidence maintenance

The current memory mechanisms and context compression of LLM agents are mainly for token budget, recall or efficiency, rather than for determining whether a certain item still aligns with visual reasoning. These frameworks are mostly used for memory management in multi-round conversations or across tasks. We are more focused on evidence maintenance and organization between different steps within a single task, so that the evidence can always be aligned with the reasoning. This is also inspired by related memory work, and we have added an explanation of this branch to the updated Introduction.

Our framework goes beyond prompting: revisions are checked by the runtime (Appendix A.3).

  • Tool outputs are stored with a SHA-256 digest that is re-verified before reuse.
  • A revision or deletion must name an existing evidence unit and a newly admitted source that differs from the old one, with a non-empty reason.
  • An edit must quote an exact substring of the new tool output.
  • Requests failing these checks are ignored.
  • The Step Audit judges the action, accepts or rejects generated artifacts, and records inconsistencies.

Q3. Necessity of active maintenance

We previously treated Visual Sketchpad as an append-only baseline in Table 2, which was imprecise. Visual Sketchpad continually appends new observations, but updates its reasoning history in an unstructured manner. We have therefore replaced the original Table 2 in Section 4.2 with a comparison of ActVis against add-only and sliding-window strategies, with structured evidence maintenance and all other components held constant. For all methods, generated output was capped at 8,192 tokens per call, and the combined input-output context window at 65,536 tokens. Under these limits, ActVis selectively maintains key evidence and improves accuracy on several tasks. Table 2 also suggests that an overly long context can interfere with reasoning, while retaining only the latest observations can discard critical evidence and lead to errors. Section 4.3 further illustrates this with a case study on predicting the winner from a chess position (Figure 4).

Q4. Ablation of mechanisms

We counted the usage of each operation on Qwen3 across the 7 tasks. RETAIN and ADD account for the largest proportions.

Format | ADD | RETAIN | REVISE | DELETE JSON | 337 (33.01%) | 590 (57.79%) | 74 (7.25%) | 20 (1.96%) HTML | 298 (28.35%) | 667 (63.46%) | 67 (6.37%) | 19 (1.81%) Overall | 635 (30.65%) | 1257 (60.67%) | 141 (6.81%) | 39 (1.88%)

Q5. Experimental consistency and fairness

We have clarified our experimental setup: the maximum output length per response is 8,192 tokens, and the maximum total context length per task is 65,536. All other numerical discrepancies have been corrected. The baselines we compare against all use Qwen2.5, Qwen3, or InternVL as base models. The agent framework involves multiple reasoning stages, all performed by the base model without any API calls. We used RTX 3090 GPUs with 24 GB of VRAM per card.

DaP-ICoT performs full attention over all input images to extract important tokens, so VRAM overhead rises sharply when multiple images are provided or when the model must view images repeatedly, leading to failures. This is most visible on Mantis, where every task provides 2-5 images: our archived DaP-ICoT runs hit CUDA out-of-memory errors on 78/217 examples (35.94%) with Qwen2.5-VL and 58/217 (26.73%) with Qwen3-VL. Under limited resources, DaP-ICoT is therefore not a cost-effective choice.

Thank you for the response. I still have two main concerns:

  1. Is active evidence maintenance really necessary? Over 91% of the operations are ADD or RETAIN, while REVISE and DELETE are rarely used, and MERGE is not reported. If these three operations are removed, how much does performance drop? Without this comparison, it is unclear whether the improvement comes from active evidence maintenance or simply selecting useful evidence.

  2. Are the experimental comparisons reliable? The authors acknowledge that some experiments failed to load images, and DaP-ICoT encountered OOM errors on many Mantis examples. How much do the corrected results differ from the original results? Does ActVis still outperform DaP-ICoT when comparing only successfully executed examples?

These two issues are important for assessing the paper's core contribution and experimental validity.

We thank the reviewer for raising this important question and agree that it should be answered directly.

  1. Is active evidence maintenance really necessary?

Operation statistics. Due to time constraints, we supplemented the results of disabling RETAIN and DELETE for the original REVISE and DELETE tasks in the Qwen3 HTML format. The comparison results are as follows: Qwen3-VL-8B — HTML (Winner ID, Matched Examples)

| Operation | Examples | Base Accuracy | ActVis Accuracy | Gain | +-----------+----------+------------------+------------------+-----------+ | REVISE | 31 | 11/31 (35.48%) | 14/31 (45.16%) | +9.68 pp | +-----------+----------+------------------+------------------+-----------+-----------+ | DELETE | 16 | 8/16 (50.00%) | 9/16 (56.25%) | +6.25 pp | +-----------+----------+------------------+------------------+-----------+

We all agree that ADD and RETAIN are dominant. We do not consider this a weakness, but an expected outcome: if evidence is rejected at the entropy gating stage, it will not enter the subsequent visual evidence organizer and will not be counted as one of these five operations. However, it largely complements REVISE and DELETE, which are corrective operations triggered only when a previous claim is refuted, and this is precisely the failure mode driving our work (Figure 1). At the same time, our five operations target visual evidence without textual reasoning. However, even when visual evidence is RETAINED, the text on the canvas—such as latest observation, key finding, and notes—is still updated, and these textual updates also guide the model to shift its attention to the existing visual evidence. Since we retain evidence at the individual level, changes in attention cannot be directly reflected in the RETAIN operation. The scope of our “proactive maintenance” is not limited to these five visual operations; it also includes admission control, budget retention, and issue tracking (resolved and unresolved issues). In addition, the comparative experiments between ActVis and Add-only FIFO further demonstrate the value of our proactive maintenance framework. MERGE is an operation for newly added evidence: if the organizer finds that it is complementary to previous evidence, it merges the two pieces of evidence into a single evidence set. This requirement is relatively stringent, so this operation did not appear in the current round of experiments.

  1. Are the experimental comparisons reliable?

We counted the tasks in the DaP-ICoT benchmark that did not encounter OOM when using the Qwen3 model, and recalculated the accuracy of our method and DaP-ICoT on this subset as follows: On tasks without OOM, our method still outperforms DaP-ICoT almost across the board, which demonstrates our overall value. However, DaP-ICoT’s more fine-grained visual evidence exploration still offers insights we can learn from.

| Task | Examples | DaP-ICoT | ActVis JSON | ActVis HTML | +------------+----------+-------------------+-------------------+--------------------+ | Winner ID | 235 | 54/235 (22.98%) | 119/235 (50.64%) | 128/235 (54.47%) | +------------+----------+-------------------+-----------------+ | MathVista | 958 | 761/958 (79.44%) | 729/958 (76.10%) | 742/958 (77.45%) | +------------+----------+-------------------+--------------+ | Mantis | 159 | 119/159 (74.84%) | 123/159 (77.36%) | 125/159 (78.62%) | +------------+----------+-------------------+-------------------+ | MMVP | 296 | 226/296 (76.35%) | 229/296 (77.36%) | 227/296 (76.69%) |

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.