acceptodds
Under review as a conference paper at ICLR 2027

Buried in Textual Debt: Context Pruning with Visual Evidence Preservation for MLLM Agents

Abstract

Multimodal Large Language Models (MLLMs) are increasingly deployed as multi-step agents, where explicit reasoning enhances task decomposition and tool coordination but also accumulates self-generated text. Over long trajectories, this text can dominate the context and suppress visual evidence, creating textual debt. Our analysis reveals that explicit reasoning steps are superfluous when task-relevant visual evidence is reliably grounded, yet stale hypotheses propagate errors into downstream inference when grounding remains ambiguous. Pruning must therefore remove redundant text without discarding visual evidence. We propose SPARE, a Kullback-Leibler (KL)-guided framework for pruning accumulated reasoning in multimodal tool-use agents. SPARE uses a compact task-state summary as privileged diagnostic context. For each candidate segment, it replays the same model under the original and summary-conditioned contexts. Reverse-KL divergence from on-policy self-distillation (OPSD) then tests whether the summary sufficiently covers the segment without disrupting future reasoning. We further fine-tune the summarizer with supervised fine-tuning (SFT), enabling more compact summaries, broader coverage, and more aggressive pruning. Across five multi-step visual tool-use benchmarks, SPARE achieves the highest average accuracy among pruning methods for all three backbones, improving task accuracy by up to 13.56 percentage points. In the aggregate trade-off analysis between accuracy and pruning, it removes 57.81% of reasoning tokens while surpassing the baseline Pareto frontier. This favorable trade-off shows that reducing textual dominance restores reliance on visual evidence and mitigates over-conditioning on self-generated language.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
MLLM-based Agents, Context Pruning, Visual Grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_71XaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies interference caused by accumulated reasoning text in multi-step multimodal agents and proposes SPARE for managing reasoning history. After generating a temporary task-state summary, the model replays historical reasoning with and without summary conditioning and uses a truncated reverse-KL score to measure token-level sensitivity. Low-sensitivity segments are removed, while high-sensitivity segments are compressed through rule-based extraction of visual evidence. Original tool calls, observations, and images are preserved. The authors also investigate SFT for improving summarization and evaluate accuracy, reasoning-token pruning, and textual interference across three backbones and multiple benchmarks.

Critical strengths

SPARE preserves original visual and tool evidence while modifying only self-generated reasoning history. This focuses the investigation on managing potentially misleading textual context rather than further compressing visual inputs that may already receive insufficient attention.

For the 30B backbone, accuracy improves by 13.56 percentage points on GTA and 12.72 percentage points on MNMS. Comparisons with complete deletion, random deletion, evidence-only reconstruction, and KL-guided selection provide initial support for selective compression over indiscriminate removal.

Critical weaknesses

Insensitivity to adding a summary does not imply insensitivity to deleting historical reasoning. The implemented score additionally relies on top-20 truncation, a fixed fallback for unavailable teacher log probabilities, and event-level normalization. Although the appendix appropriately acknowledges that the theoretical bound does not justify the actual deletion rule, the experiments do not sufficiently establish whether the score reliably predicts the consequences of removing a segment.

In Table 1, average accuracy improves by only 0.32 percentage points for the 8B backbone and decreases for Nemotron; the 30B backbone also shows a substantial decline on V*. Meanwhile, each pruning event requires summary generation and two forward replays, whose costs are excluded from the reported persistent-history pruning rate. It therefore remains unclear under which tasks and trajectory lengths SPARE provides a reliable accuracy–compute advantage.

Questions for authors

Have you measured the relationship between the KL score and the actual change in answer quality after deleting a segment? At matched retained-context lengths, does the proposed selection rule consistently outperform random selection, conventional summarization, or simpler importance heuristics?

What are the changes in end-to-end latency, total processed tokens, or FLOPs when summary generation and both replay passes are included? Could you also provide confidence intervals for the main accuracy differences to establish whether the smaller gains are robust?

Reviewer_9WgWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates textual debt in multi-step MLLM agents, where accumulated reasoning can reinforce outdated assumptions and distract models from visual evidence. The authors propose SPARE, a context-pruning framework that uses a temporary task-state summary to measure the sensitivity of historical reasoning through summary-conditioned reverse KL divergence. Low-sensitivity reasoning segments are removed, while high-sensitivity segments are compressed into concise visual evidence using rule-based extraction. Original tool calls, observations, and images are preserved. The authors further improve summarization through supervised fine-tuning.

Across three MLLM backbones and four benchmarks, SPARE shows promising accuracy–pruning trade-offs. On the three benchmarks used for aggregate analysis, it improves accuracy from 48.81% to 50.34% while pruning 57.81% of historical reasoning tokens.

Critical strengths
  1. A meaningful problem formulation. The paper identifies an important failure mode in long-horizon multimodal agents: accumulated self-generated reasoning can introduce redundancy and reinforce incorrect assumptions. Focusing compression on historical reasoning rather than visual inputs is a sensible design choice, particularly for tasks that depend on repeated visual inspection.

  2. A practical approach to selective context management. SPARE combines summary-conditioned sensitivity estimation with evidence-preserving reconstruction. It retains tool interactions and original visual observations while selectively removing reasoning text. The method requires no auxiliary model at inference time and can be integrated into existing tool-use pipelines without retraining the underlying agent.

  3. Encouraging empirical results with useful ablations. The paper evaluates three MLLM backbones and compares SPARE against random deletion, complete reasoning removal, and evidence-only reconstruction. SPARE improves Qwen3-VL-30B's average accuracy from 56.53% to 59.24% while pruning 69.40% of reasoning tokens.

Critical weaknesses
  1. The KL-based sensitivity signal is not directly aligned with reasoning importance. SPARE measures how token predictions change when a task-state summary is added, rather than how removing a reasoning segment affects subsequent decisions. Moreover, the replay concatenates historical reasoning without explicitly preserving the original interleaved tool observations in the replay sequence. This raises questions about whether the signal reflects a segment's actual contribution in the agent's multimodal context.

  2. The claimed efficiency benefits lack end-to-end cost measurements. The reported pruning rate measures the reduction in persistent reasoning tokens but excludes temporary summaries and the additional computation required for two forward replays of historical reasoning. Since the pruned text has already been generated, these reductions do not necessarily translate into lower total inference cost. Without latency, or total token processing, it is difficult to assess the practical efficiency gains.

  3. Performance gains are inconsistent, and stronger matched-budget comparisons are needed. SPARE substantially improves some benchmarks but degrades others. For example, Qwen3-VL-30B drops from 74.87% to 67.02% on V*, while Nemotron's overall accuracy decreases slightly.

Questions for authors
  1. How well does summary-conditioned KL predict the actual importance of historical reasoning segments?

The proposed signal measures sensitivity to adding a summary, which differs from the downstream effect of deleting a segment. Could the authors directly measure how individual segment removals affect next-step predictions or final task accuracy, and evaluate the correlation with their KL scores? It would also be useful to compare against simpler selection criteria such as recency, or entropy etc.

  1. Does SPARE provide net inference savings after accounting for the additional computation?

Each pruning event requires summary generation and two forward replays of the historical reasoning. Could the authors report end-to-end latency and/or total processed tokens compared with Full Trace and simpler pruning baselines?

  1. How reliably does the reconstruction procedure preserve correct visual evidence, and how much does the temporary summary itself contribute?

Since visual evidence is extracted programmatically from prior model-generated reasoning, how does SPARE distinguish grounded observations from hallucinated or outdated interpretations? Could the authors provide a quantitative audit of extraction precision and information loss?

Reviewer_5VKDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how accumulated reasoning can distract multimodal agents from visual evidence. It calls this problem “textual debt” and proposes SPARE to prune the reasoning history. SPARE generates a task-state summary and compares token predictions with and without this summary. It uses the resulting KL scores to select reasoning segments. Low-score segments are removed, while selected segments are reduced to short evidence records through fixed rules. Tool calls, observations, and images remain unchanged. The paper evaluates three backbones on four benchmarks. It also studies summary fine-tuning and analyzes changes in stale-answer copying and visual attention.

Critical strengths
  1. The method addresses a useful problem with a clear design. It separates past reasoning from tool outputs and images. This allows the agent to remove stale text while keeping the original evidence available.
  2. The experiments include useful controls. The paper compares full history, complete deletion, random deletion, evidence-only compression, and KL-guided deletion. Across the three benchmarks in Figure 2, SPARE improves aggregate accuracy from 48.81% to 50.34% while removing 57.81% of reasoning tokens.
Critical weaknesses
  1. The link between the KL score and safe deletion is unclear. The score measures how predictions change when a summary is added. It does not directly measure whether deleting a segment will hurt later decisions. The appendix checks summary coverage, but does not show that removing low-KL segments causes fewer errors than removing high-KL segments.
  2. The gains are small in some settings, and an important baseline is missing. Average accuracy rises by 0.32 points on Qwen3-VL-8B and falls by 0.64 points on Nemotron. The main ablation uses 59 GTA examples, with only one more correct answer than Full Trace. No uncertainty estimates are reported. A comparison with replacing old reasoning by a persistent summary is also missing.
Questions for authors
  1. Does deleting low-KL segments cause fewer errors than deleting high-KL segments? Can you test this while matching the amount of text removed?
  2. How does SPARE compare with replacing old reasoning by a persistent task-state summary? Please use the same trigger points and keep tool outputs and images unchanged.
  3. Does summary fine-tuning also improve Full Trace without pruning? This comparison would help separate better summarization from broader changes to the agent.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.