acceptodds
Under review as a conference paper at ICLR 2027

When History Is Multimodal: Rethinking Context Management for Long-Horizon Agents

Abstract

Long-horizon agents need a context manager to compress growing interaction histories into a bounded working context, via passive strategies or active strategies that decide how memory is accessed and reorganized. Meanwhile, prior optical-memory work mainly treats pixels as a dense codec for textualized histories, often presupposing that rendering context into optical memory incurs a significant performance drop relative to text, thus coupling this representation with SFT, self-distillation, or reinforcement learning to close this gap, leaving unresolved (i) how visual rendering performs as a context manager under a fair, controlled comparison, and (ii) whether this carrier offers a native advantage when history is inherently multimodal. In this paper, we formulate context management as a budget-constrained history transformation and introduce Visual Rendering (VR) as a representational context manager. Under a shared harness, policy model, trigger, and task domain, we evaluate VR on four text-centric and three multimodal benchmarks against four baselines (No Compression, Discard-All, Sliding Window, Summarization), finding visual memory is a natural carrier of native visual evidence. Building on this finding, we propose **VERA** (**V**isual **E**vidence-**R**etaining strategy for long-horizon **A**gents), a training-free context manager built on deterministic rendering with no exposed memory operations: on text-centric benchmarks it renders textual history as VR does, while on multimodal benchmarks it retains native visual observations instead of translating them into text. Across nearly all benchmarks, VERA cuts cumulative non-cache tokens by 31.5%–63.1% versus No Compression, matches existing managers on text-centric tasks, and achieves the highest accuracy among all baselines on multimodal tasks, supporting a modality-preserving view of long-horizon context management.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-horizon agents, Context management, Context compression, Visual memory, Multimodal learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9JDqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether rendering the interaction history of long-horizon agents into images is a good context manager. The authors propose VERA, a training-free rendering manager that 1) keeps native images from tools such as image search, crop, and zoom, 2) renders table outputs as compact tables, and 3) downsamples older history. Experiments with Gemini-3.5-flash on four text and three multimodal benchmarks show that visual rendering uses the fewest tokens and that VERA achieves the best accuracy on the multimodal benchmarks.

Critical strengths
  1. The question of what to keep when the agent history itself is multimodal is interesting and well-motivated.

  2. The paper is clearly written, and the controlled setting (same harness, same policy model, same trigger threshold) makes the comparison easy to follow.

  3. The evaluation covers both text-only and multimodal agentic benchmarks, and the token cost is reported for every method.

Critical weaknesses
  1. The two conclusions of the paper, that visual context is competitive with text context and that keeping native images helps when the history is multimodal, both rest on one specific harness implementation, VERA. Many of its design choices can change the results: older pages are downsampled to 0.75x, at most 10 images are supplied per call, an unspecified number of recent turns are kept as raw text, and the history is re-organized with role and tool labels. The paper says Visual Rendering changes only the representation (L72 to L76), but these choices also change how much information is kept. It is necessary to test other harness implementations, or at least to give a hyperparameter analysis of these key designs.

  2. The results in Table 1 are counter-intuitive. Among the text-based methods, Discard-All is the best, Summarization and Sliding Window come next, and No Compression is the worst (69.0, and only 59.8 on BrowseComp-EN). With such an unusual ordering, the advantage or disadvantage of Visual Rendering does not look reliable, and it is hard to explain why text performs poorly while Visual Rendering performs better on average with the same information. The gain of Visual Rendering over Discard-All is also only 0.5 points without variance, and Visual Rendering is the worst method on XBench-DeepResearch and Seal-0.

  3. The novelty is limited. Rendering history into images has been studied by many recent works, and VERA mainly adds deterministic engineering choices (keeping native images, table rendering, resolution decay). The unified definition in Section 3 is mostly notation, and the "Mechanistic Analysis" listed as a contribution is only the retry-probability formula in Appendix B.

  4. The multimodal comparison is confounded. Sliding Window and Summarization drop or verbalize the images after compression, while VERA keeps them, so the gain in Table 2 cannot be attributed to rendering itself. A text-based manager that also keeps the native images is needed. It is also unclear how images are converted to text in the Text-only Rendering ablation (who writes the captions and with which prompt).

  5. The experiments use only one policy model, Gemini-3.5-flash. Reading rendered text depends heavily on the visual ability of the model, so the conclusions should be checked on at least one open-source VLM and one stronger model. Learned optical-context methods such as AgentOCR are discussed but not compared; training-free ones such as OCR-Memory could at least be included.

  6. The RQ3 analysis has low statistical power. With 100 questions and avg@2, one point has a standard error of about 4 to 5 points, so the non-monotonic trend in Table 5 may be noise. Finding 3 should be stated more carefully.

  7. The cost analysis is incomplete. Only non-cache tokens are reported, but rendering breaks the prefix cache while No Compression can reuse it heavily. Wall-clock time and rendering overhead should also be reported.

Minor: "Eq. equation 3" appears several times (around L200, L205, L221). Equation 7 is used in Section 4.2 but only defined in Appendix B. Table 5 uses "1/Vis. Ratio" while the text uses "Compression Factor". The caption of Figure 2 says 0x is excluded, but it is plotted.

Questions for authors

Please refer to Weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer 9JDq

We thank the reviewer for an exceptionally detailed review, which led us to find and correct an error. Full tables and references are in the General Response (GR).

W1. One harness and specific design choices.

We added DeepSeek-V4.1-Flash under Hermes and DeepSeekHarness (DSH) (GR-R1). Visual rendering stays competitive with the strongest text manager in every setting and is the cheapest under both harnesses, so we claim competitiveness, not superiority. Design choices: the original task and the latest tool-call round stay verbatim; in the compressed history, the 5 most recent rounds are rendered at 1.0× and older pages at 0.75×; at most 10 images are supplied per call, dropping the oldest pages first. Role/tool labels only mark provenance; text managers get the same information via message roles and tool names. All values are in Appendix D.2; the resolution sweep is in W6.

W2. Counter-intuitive Table 1.

Timed-out No Compression runs on BrowseComp-EN had been scored as incorrect instead of being re-run under our retry protocol (Appendix D.4), which caused the anomalous 59.8. We apologize for the error. After re-running them, No Compression is 68.0, and Table 1 is corrected. The remaining gap is consistent with long-horizon degradation: over 14,920 trajectories, No Compression reaches 94.1% on trajectories of 31–60 turns but only 31.7% beyond 100 turns, in line with reports that discard-style management is strong on deep search (DeepSeek-V3.2 [5]). The gain over Discard-All is small, and VERA is weakest on XBench-DeepResearch (69.0 vs. 71.7) and Seal-0 (54.1 vs. 56.5); we now state this explicitly. All new results use avg@3 and report mean ± std over repeats (GR-R1).

W3. Novelty.

Contributions (GR): (C1) the formulation, which separates managers from boundary controls and fixed from active or memory-routed managers, and thus decides what can be compared under one budget; (C2) a unified protocol, which shows that with capable VLMs training-free rendering is already competitive with text managers, so the visual–text gap reported by AgentOCR [4] and CAPS [6] is not inherent to the carrier; (C3) VERA, whose core is preserving native visual evidence. Rendering and resolution decay are credited to prior work [1, 2, 3], and "Mechanistic Analysis" is removed from the contributions.

W4. Confound in the multimodal comparison.

Summary+Native and Window+Native keep exactly VERA's native images at a matched budget, and all three are on par (GR-R3). The Table 2 gain therefore comes mainly from preserving native evidence (C3). This also explains MM-BrowseComp in GR-R1: Sliding Window keeps its recent native images and is strong, while Discard-All drops them all and is far weaker (29.7 vs. 46.3–52.7). In Text-only Rendering, image_search images are replaced by captions from the Hermes vision tool (Qwen3.7-Plus) with its default prompt "Fully describe and explain everything about this image, then answer the following question: {question}"; crops and zooms are omitted, keeping their coordinates and source; tables keep their original text form (Appendix D.2).

W5. One policy model; AgentOCR / OCR-Memory.

We added DeepSeek-V4.1-Flash, an open-source model from a second family (GR-R1). Other open-source VLMs need local deployment; with limited compute and slow local inference, we could not finish all controls during the rebuttal, and we list the dependence on dense-text reading as a limitation. OCR-Memory [3] is not training-free: it LoRA-fine-tunes a DeepSeek-OCR [2] retriever that returns verbatim text, so the agent never consumes images. AgentOCR [4] is learned. Both fall outside our same-policy, training-free class and are compared conceptually in Appendix J.

W6. Statistical power of RQ3.

We re-ran the sweep with DeepSeek-V4.1-Flash, avg@3, scoring timeouts as incorrect (hence 82.3 at 0.75× vs. 84.0 in GR-R1, same runs). Mean ± std at 0 / 0.25 / 0.5 / 0.7 / 0.75 / 0.8 / 1.0×: 77.7±1.5 / 77.0±3.1 / 81.7±3.4 / 84.3±2.7 / 82.3±1.6 / 80.3±2.5 / 78.0±2.2. As on Gemini, intermediate scales are best: 0.7× and 0.75× beat 1.0× and no older history by 4–7 pp, above the std. We keep 0.75× (within one std of the best, chosen beforehand on Gemini) and call the exact optimum exploratory in Finding 3.

W7. Cost and wall-clock.

Now reported (GR-R2). VERA costs 58% / 30% less than No Compression: its cache-hit rate is lower (70% vs. 88% on Hermes), but its input is 2–5× smaller. It is not slower end to end, and rendering takes 3.7–17.4 s per question.

Minor. All fixed: equation references, the reset-retry equation (old Eq. 7) now defined in Sec. 4.2, "compression factor" used throughout (now Table 7), and the resolution-figure caption (now Fig. 3).

We are grateful for this review: it caught a real error and made the paper clearer. We hope the revised claims and new evidence address your concerns, and we would appreciate it if you would consider revisiting your score.

Thanks for the detailed comments! However, the contribution and novelty of this paper may not convince me. Currently, I will maintain my score.

Thank you for engaging with our rebuttal and for acknowledging the additional experiments. We fully respect your assessment. Following your earlier comments, we have substantially narrowed and restructured our contributions. Since we no longer claim rendering itself as novel, we would like to briefly clarify what the revised paper contributes.

C1. A formulation of context management.

We define context management as a budget-constrained history transformation. A passive manager deterministically maps the history at each trigger to a nonempty, budgeted representation. This definition does two things:

  • it separates passive managers from active or memory-routed managers (e.g., OCR-Memory, AgentOCR) and from full-retention and reset controls;
  • it determines which methods can be fairly compared under one budget.

This directly addresses your concern that Section 3 was mostly notation: the formulation now fixes the scope and the comparison set.

C2. A unified protocol for fairly comparing passive managers.

Text and visual managers are compared under the same policy, trigger and budget, with cache-aware cost. The comparison now covers two policy models and three agent harnesses (Hermes, DSH, Apodex). Training-free Visual Rendering is competitive with existing compression strategies on text-centric tasks while being the cheapest:

  • −51% to −82% total tokens relative to No Compression;
  • a prefix-cache hit rate of 70–73%, on par with Sliding Window and Discard-All.

Hence, with capable VLMs, the visual–text gap that motivates trained optical memories is not inherent to the visual carrier.

C3. Preserving native visual evidence in a single visual carrier.

The unified protocol reveals that, when the history is multimodal, preserving native visual evidence (searched images, crops, zoomed views, tables) matters more than the choice of compression strategy. This is the confound you identified, and your suggested control confirmed it.

VERA follows directly from this finding. It keeps the evidence on the rendered canvas together with the history:

  • It clearly outperforms the variant without native evidence (Vanilla Visual: +6.7 pp on Apodex, +7.3 pp on Hermes).
  • It is comparable to text managers that are explicitly given the same native images.
  • It needs no summarization call and no separate image channel. Within VERA's budget, native-image baselines sometimes could not fit at all, or exceeded the image cap during the run.

In short, the paper's message is: define the problem, compare fairly, find that native evidence is what matters, and preserve it in the cheapest carrier. We believe these empirical findings are useful to practitioners designing agent memory, beyond the rendering technique itself.

We are grateful for your review: it identified the key confound and an aggregation error, and made the paper more rigorous and more honest. We hope the restructured contributions better convey the value of this work. If they do, we would be grateful if you would consider revisiting your score.

Reviewer_UAppOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how visual representations can be used to manage the interaction history of long-horizon agents. The authors formulate context management as a budget-constrained transformation of historical information and propose VERA, a training-free strategy that renders older textual history as images while preserving native visual observations and spatially structured analytical artifacts, keeping recent interactions in their original form. Using a unified Hermes-agent harness and the Gemini-3.5-flash-thinking model, the authors compare VERA against four baselines across four text-centric benchmarks and three multimodal benchmarks. Experiments show that VERA achieves competitive accuracy on text-centric tasks while reducing cumulative non-cache token consumption, and attains the highest reported performance among the compared methods on all three multimodal benchmarks. Ablation studies further examine the effects of preserving native visual evidence and varying the resolution of rendered older history.

Critical strengths
  1. VERA requires neither additional model training nor learned memory components. Under the same agent harness and policy model, it achieves both the lowest cumulative non-cache token consumption and performance improvements across the reported benchmarks.
  2. The paper is clearly written and easy to follow.
Critical weaknesses
  1. All experiments rely exclusively on a single model (Gemini-3.5-flash-thinking). Because the effectiveness of visual rendering heavily depends on the underlying VLM's vision encoder, dense-text OCR capability, and multi-image attention mechanism, it remains unverified whether the competitiveness of training-free visual rendering on text-centric tasks, as well as the 0.75×0.75\times scaling conclusion, generalizes to other model families or open-source VLMs with weaker visual reading capabilities (e.g., the Qwen-VL series). Methodologically, VERA essentially consists of deterministic layout rules, offering relatively limited algorithmic novelty.
  2. This paper measures efficiency solely by cumulative non-cache tokens (TncT^{\text{nc}}). However, append-only baselines that do not rewrite history (such as No Compression) naturally enjoy high prefix-cache hit rates, whereas VERA replaces history with newly rendered canvases and downsamples older canvases upon triggering compression, which invalidates the prefix cache. Because the paper does not report cache-inclusive total tokens (TallT^{\text{all}}), actual API costs accounting for cache pricing, or end-to-end latency, this single metric fails to reflect the true inference overhead.
Questions for authors
  1. Under a comparable post-compression token budget, how does VERA perform compared to a baseline that combines textual summarization with preserving the exact same native visual evidence? In addition, in the text-only rendering ablation, how are images, cropped regions, and tables specifically converted into text?
  2. How does the system handle cases where the historical content requires multiple rendered images (e.g., when exceeding the image cap)? What are the base canvas dimensions and font sizes? These hyperparameter choices determine how much evidence can be retained and are essential for reproducing the core comparisons. Furthermore, what are the empirical guidelines for selecting these parameters?

The above details require further clarification.

2 comments
AuthorsOpenReview ↗

Response to Reviewer UApp

We thank the reviewer for the careful reading and concrete questions. Full tables and references are in the General Response (GR).

W1(a). Single model and dependence on visual-reading ability.

We added the open-source DeepSeek-V4.1-Flash and DeepSeekHarness (DSH) (GR-R1).

  • On BrowseComp, VERA stays competitive with the strongest text manager. With DeepSeek + Hermes it reaches 84.0 ± 2.8 vs. 81.2 for Summarization; on DSH all methods are within 4 pp, with VERA 2.0 pp below Summarization. On MM-BrowseComp with DeepSeek, VERA (52.7) is on par with the best text managers.
  • VERA is the cheapest method under both harnesses (GR-R2).

Other open-source VLMs (e.g., Qwen-VL) need local deployment, which our limited compute and slow inference did not allow within the rebuttal period. We agree that the benefit depends on the VLM's dense-text reading ability. We state this explicitly as a condition of applicability in the revised Limitations, and we do not claim that the scaling conclusion transfers to weaker VLMs.

W1(b). Novelty.

Our contributions (GR) are (C1) the formulation of context management as a budget-constrained history transformation, (C2) a unified protocol for fairly comparing passive context managers, and (C3) VERA, whose core is preserving native visual evidence alongside rendered history. The layout rules are implementation details rather than claimed contributions, and rendering and resolution decay are credited to prior work [1, 2, 3].

W2. Cache-inclusive tokens, actual cost and latency.

Now reported (GR-R2). On matched instances, compared with No Compression, VERA uses 82% / 51% fewer total tokens and costs 58% / 30% less under cache-discounted pricing (Hermes / DSH).

We agree that rewriting the history invalidates the cached prefix at each compression. Fortunately, compression is triggered only 1.9 / 0.3 times per question on average and the context is append-only between triggers, so VERA's cache-hit rate (70% / 73%) matches Sliding Window and Discard-All. Summarization is the method that most disrupts caching (41%).

E2E time on matched instances is not higher for VERA, and rendering overhead is 3.7–17.4 s per question.

Q1(a). Summarization + the same native visual evidence.

Implemented at matched post-compression prompt budget (±2%) (GR-R3).

ArmStrict common subset (46 units, 31 Q)Mean ± std over 3 repeats
VERA38/46 (82.6%)82.6 ± 11.0
Summary+Native37/46 (80.4%)84.0 ± 12.5
Window+Native36/46 (78.3%)80.6 ± 11.4

With the same native images, the three managers are on par (they differ by 1–2 questions out of 46). The multimodal gain in Table 2 therefore comes mainly from preserving native visual evidence (C3), not from rendering per se. Consistently, Sliding Window, which keeps the native images of its recent window, is the strongest text baseline on MM-BrowseComp. Rendering's contribution is reaching this accuracy at lower cost: it needs no summarization call and is cheaper than Sliding Window (GR-R2). We have revised the Table 2 discussion accordingly. We thank the reviewer for suggesting this control; it makes the attribution in Table 2 much clearer.

Q1(b). Text-only rendering ablation.

Thank you for this question; we should have described it precisely. Text-only Rendering uses the same renderer as VERA and differs only in the compressed history:

  • image_search images are replaced by captions from the Hermes vision tool (Qwen3.7-Plus) with its default prompt "Fully describe and explain everything about this image, then answer the following question: {question}".
  • Crops and zooms are omitted; their textual metadata (coordinates, source) remains.
  • Tables keep their original text form instead of being rendered as tables.

The original question images and the verbatim recent context are identical to VERA. Details are in Appendix D.2.

Q2. Image cap, canvas parameters and guidelines.

VERA and Text-only Rendering share these settings, now listed in Appendix D.2:

  • Pages are 1024×2048 px and paginated, so a long history spans several images.
  • Text is 16 pt at 72 DPI with 20 pt line spacing (1.25×), in DejaVu Sans by default.
  • The original task and the most recent tool-call round stay verbatim.
  • In the compressed history, the 5 most recent tool-call rounds are rendered at 1.0×, and older pages are scaled to 0.75× (768×1536 px).
  • At most 10 images are supplied per call; beyond this cap, the oldest pages are dropped first.

Guidelines: the font must stay legible after the vision encoder resizes the page. In our probe QA (v3WU, W2), 16 pt text on 1024-px pages keeps most facts readable at 1.0× and 0.75× (13/18, 12/18) but not at 0.5× (5/18). This is why only older pages are downsampled, and only to 0.75×. The 7-scale sweep is in the appendix.

We hope these clarifications and new results address your concerns, and we would appreciate it if you would consider revisiting your score.

Dear Reviewer UApp,

Thank you again for your thoughtful review. As the discussion period ends soon, we would like to briefly highlight how we addressed your main points:

  • Q1 (summarization + the same native images): We implemented this baseline at a matched prompt budget (GR-R3). Your suggestion led us to sharpen the paper's claim: in multimodal histories, preserving native visual evidence matters most, and VERA does so within a single visual carrier, without a summarization call or a separate image channel.
  • W2 (cache-inclusive cost): We now report cached / non-cached tokens, reference $, cache-hit rate and E2E time. VERA uses 51–82% fewer total tokens, and its cache-hit rate (70–73%) is on par with other compressors (GR-R2).
  • W1 (single model): We added DeepSeek-V4.1-Flash and two additional harnesses (DSH and the lightweight Apodex). VERA remains competitive and is the cheapest in every setting.
  • Q2 (canvas / image-cap details): All parameters and selection guidelines are now in Appendix [X].

If any concern remains, we would be glad to clarify. If our responses have addressed your concerns, we would be grateful if you would consider revisiting your assessment.

Reviewer_v3WUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes VERA, aiming for a more efficient and better context management by replacing textual/visual history evidence in rendered images. They test one gemini model with one harness framework. Results show that VERA saves tokens by around 30~60%, and achieves the highest accuracy among all compared methods.

Critical strengths
  1. The idea of context management using rendered images to replace raw visual/textual history evidence is straightforward and easy to understand.
Critical weaknesses
  1. Experiment is only run on 1 model + 1 harness. More settings are needed to show the method generalizes beyond this specific setting.

  2. What’s the information loss when compressing history into an image? A direct way to measure it is to compare the raw evidence with the recovered ones from the compressed image (eg by using vlm-as-a-judge).

  3. What’s the actual cost of each experiment? One concern is that by replacing history with rendered images, one can lose the advantage of calling api with kv-cache, which is much cheaper than non-cache calling. How do you manage this tradeoff?

Questions for authors

See weakness above.

3 comments
AuthorsOpenReview ↗

Response to Reviewer v3WU

We thank the reviewer for the clear and actionable questions. Full tables are in the General Response (GR).

W1. One model + one harness.

We added DeepSeek-V4.1-Flash and DeepSeekHarness (DSH). All runs are avg@3 (GR-R1).

  • On BrowseComp with DeepSeek + Hermes, VERA is the most accurate at 84.0 ± 2.8, compared with 81.2 / 76.3 / 73.7 / 65.6 for Summarization / No Compression / Sliding Window / Discard-All.
  • On DSH, all methods are within 4 pp; VERA (82.1) is 2.0 pp below Summarization.
  • On MM-BrowseComp with DeepSeek, VERA (52.7) is on par with Sliding Window (51.7) and No Compression (49.0).

VERA is the cheapest method under both harnesses (GR-R2). Our revised claim is therefore "competitive accuracy at the lowest cost," not universal superiority.

W2. Information loss.

We followed your suggestion and measured recoverability with a VLM-as-a-judge. From held-out trajectories we extracted atomic facts in the history (numbers, entities, URLs) and asked the policy to answer fact-level probes from each representation. A judge then scores each answer against the raw evidence.

Preliminary result (9 held-out trajectories, 18 facts common to all representations):

RepresentationProbe QA correct
Raw text (upper bound)18/18
Canvas 1.0×13/18
Canvas 0.75×12/18
Canvas 0.5×5/18
600-word summary3/18

Rendering is not lossless, and loss grows as resolution drops. However, the canvas retains far more checkable evidence than a summary. This motivates keeping recent turns raw and decaying only older pages.

We also tried free-form full reconstruction. It proved unreliable as a metric: even reconstructions from raw text supported only 3/7 judged facts. We therefore adopt the fact-level protocol.

W3. Actual cost and the KV-cache trade-off.

We now report cached, non-cached and output tokens, reference $ under cache-discounted pricing, cache-hit rate and E2E time (GR-R2). On matched instances, compared with No Compression, VERA uses 82% / 51% fewer total tokens and costs 58% / 30% less (Hermes / DSH).

We manage the trade-off in three ways:

  1. Compression is triggered rarely (1.9 / 0.3 times per question, averaged over all questions). Between triggers the context is append-only, so VERA's cache-hit rate (70.1% / 73.1%) is on par with Sliding Window and Discard-All.
  2. The absolute input is much smaller. A cheaper cached token cannot offset a 5× larger context: No Compression has an 88% hit rate yet costs 2.4× more on Hermes.
  3. Summarization, not rendering, is the method that most disrupts caching (41.3% hit rate).

E2E time on matched instances is not higher for VERA (368 s vs. 384 s on Hermes; 498 s vs. 562 s on DSH). Rendering itself costs 3.7–17.4 s per question.

We hope these results address your soundness concerns, and we would be grateful if you would consider updating your assessment. We are happy to clarify anything further.

I appreciate your reply and experiments added.

W1. I appreciate the experiment on DSH + DS model. I understand this is might be the best you can do during rebuttal. However, for comprehensiveness of experiments, I suggest running on more models+harness.

W2. It seems that information loss is significant for some setting. It leads to another question that how does such information loss affect task performance. I suggest adding it to analysis to deepen the understanding of your method.

We sincerely thank the reviewer for the follow-up and for acknowledging the new experiments.

W1. More models and harnesses.

We agree that broader coverage strengthens the conclusions. In addition to Gemini + Hermes, DeepSeek + Hermes and DeepSeek + DSH, we have now completed a third, lightweight harness (Apodex).

On timing. The Apodex runs were launched in parallel with the other rebuttal experiments. Porting the context managers into a new harness required additional engineering, so these runs started later and finished only recently, which is why they were not included in our first response. They use exactly the same manager implementations, trigger, budget, question set, repeats and judge as the other settings. We will release the configurations and per-run logs with the revised supplementary material.

The table reports avg@3 ± std.

MethodBC · ApodexMMBC · Apodex
No Compression79.3 ± 2.937.7 ± 2.8
Discard-All58.7 ± 4.222.3 ± 2.4
Sliding Window67.3 ± 1.336.0 ± 3.8
Summarization73.0 ± 4.434.3 ± 3.6
Vanilla Visual (no native evidence)–40.0 ± 3.3
VERA80.3 ± 1.946.7 ± 3.2

Vanilla Visual ablates native visual evidence and is therefore evaluated only on the multimodal benchmark. On Apodex, VERA achieves the highest accuracy on both benchmarks. On Hermes, it is the most accurate on BC and on par with the best text manager on MM-BrowseComp; on DSH, it is on par with the text managers. It is the cheapest manager in every setting.

Given the remaining time in the discussion period, we will add more policy models in the camera-ready version: (i) an open-source VLM served locally (Qwen-VL family), which also removes provider-side latency confounds, and (ii) a stronger proprietary model. Both will be run on BrowseComp with all five managers and avg@3.

W2. How does information loss affect task performance?

Thank you for this suggestion. We agree that fact-level recoverability and end-to-end accuracy are different quantities. Our resolution sweep and Probe QA cannot be chained into a causal curve: they intervene on different parts of the context (RQ3 downsamples only far history and keeps recent turns raw), they use different histories, and task completion is additionally affected by timeouts. We therefore do not claim that information loss is harmless.

Current evidence. We expanded the probe study to 40 text histories with 240 planned facts; this supersedes the preliminary 18-fact result in our earlier reply.

  • Text histories: among 24 completed token-matched pairs, rendered history answers 122/144 probes, versus 71/144 for a summary of equal prompt size.
  • Multimodal histories: the matched subset is small and shows parity with summarization (24/48 vs. 24/48). Summary + native images is on par with rendering (20/24 vs. 19/24; 4 histories).

These results use a full-fidelity diagnostic renderer and are model-judged; human validation is pending.

Rendering thus preserves substantially more checkable evidence for text-heavy histories, but not for multimodal ones. This agrees with our revised framing: in multimodal histories, what matters most is preserving native visual evidence, which VERA does within a single visual carrier.

Planned analysis (camera-ready).

  1. Failure attribution. In a random, stratified sample of completed-but-incorrect trajectories, we will trace answer-critical evidence across compression boundaries. Each failure will be classified as compression-related loss, retrieval failure, evidence-use error, or indeterminate, with infrastructure failures reported separately.
  2. Counterfactual check. We will restore the critical evidence in raw form and compare against a control given the same extra budget, to test whether the loss causally changes outcomes.

We will add this as a dedicated analysis section. We thank the reviewer again: this question clarifies that rendering fidelity and task effectiveness should be evaluated separately.

We hope the additional harness, the cost analysis, and the expanded information-loss study address your concerns about soundness. If so, we would be sincerely grateful if you would consider raising your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank all reviewers. All three asked about (i) generalization beyond one model and harness and (ii) cost under prefix caching; UApp and 9JDq also asked for (iii) text managers that keep the same native images. All changes are in the revised PDF.

Contributions (clarified).

  • C1. Formulation. Context management as a budget-constrained history transformation, separating passive from active managers and from boundary controls.
  • C2. Unified protocol. A fair comparison of text and visual passive managers (same policy, trigger, budget) on two models and two harnesses with cache-aware cost. It shows that, with capable VLMs, a carefully designed training-free renderer is competitive with text managers: the visual–text gap that motivates trained optical memories (AgentOCR [4], CAPS [6]) is not inherent to the carrier.
  • C3. VERA. Its core is keeping native visual evidence (searched images, crops, zooms, tables) alongside the rendered history rather than verbalizing it.

Rendering text into images and resolution decay come from prior work (Glyph [1], DeepSeek-OCR [2], OCR-Memory [3]); layout rules are implementation details, and "Mechanistic Analysis" is dropped.

R1. Additional model, harness and multimodal benchmark

We added DeepSeek-V4.1-Flash and DeepSeekHarness (DSH). All runs use 100 questions × 3 repeats (mean ± std); BrowseComp (BC) uses the 100 randomly sampled questions from RQ3. Each instance has a 1-hour limit. On BC, accuracy uses each method's completed instances (with DeepSeek, timeouts are first re-run, leaving <10% per method); on MM-BrowseComp (MMBC), timeouts count as incorrect.

MethodBC Gemini + HermesBC DeepSeek + HermesBC DeepSeek + DSHMMBC DeepSeek + Hermes
No Compression67.0 ± 5.376.3 ± 3.180.2 ± 2.049.0 ± 1.0
Discard-All72.3 ± 8.365.6 ± 2.982.1 ± 1.929.7 ± 3.1
Sliding Window71.2 ± 6.073.7 ± 1.583.5 ± 1.051.7 ± 3.2
Summarization70.6 ± 1.581.2 ± 1.984.1 ± 1.246.3 ± 4.5
VERA74.3 ± 3.084.0 ± 2.882.1 ± 1.052.7 ± 3.8

On BC DeepSeek + Hermes, VERA leads every baseline, from +2.8 pp over Summarization to +18.4 over Discard-All. On DSH, VERA is 2.0 pp below the best (Summarization). On MMBC, VERA is on par with Sliding Window and No Compression, and 10.7 pp above rendering without native images.

R2. Cache-inclusive cost and latency

For fairness, we use BC instances completed by every method and where every compression method triggered (Hermes N=143, DSH N=120), averaged per instance. H = Hermes, D = DSH. Reference prices (USD per M tokens): 0.15 cached, 1.50 non-cached, 9 output.

MethodH tokH hitH $H E2E (s)D tokD hitD $D E2E (s)
No Compression1089K88.4%0.477384307K83.0%0.190562
Discard-All425K71.6%0.457491217K74.4%0.190646
Sliding Window469K71.8%0.374334240K72.5%0.197581
Summarization442K41.3%0.642388250K60.0%0.275569
VERA196K70.1%0.202368150K73.1%0.134498

All managers, including Sliding Window, act only at the 64k-token trigger and are append-only in between; VERA compresses only 1.9 / 0.3 times per question (over all questions), so its hit rate matches Sliding Window and Discard-All. No Compression has the highest hit rate, but its input is 2–5× larger, so VERA still costs 58% / 30% less. Summarization disrupts caching most; rendering takes 3.7–17.4 s per question.

R3. Text managers that keep the same native images (MMBC)

Summary+Native and Window+Native pass exactly VERA's native images at a prompt budget matched to VERA (±2%). Each trajectory continues from the same prefix with one intervention at the first compression. Common subset: 46 question × repeat units (31 questions).

ArmCorrect (of 46)Mean ± std
VERA38 (82.6%)82.6 ± 11.0
Summary+Native37 (80.4%)84.0 ± 12.5
Window+Native36 (78.3%)80.6 ± 11.4

With equal native evidence, the three managers are on par, so the multimodal gain comes mainly from preserving native evidence (C3). This also explains MMBC in R1: Sliding Window keeps its last five rounds' native images and is strong; Discard-All drops them all and is by far the weakest (29.7). Rendering matches this accuracy at lower cost, with no summarization call (R2).

References

[1] Cheng et al. Glyph. arXiv:2510.17800, 2025.
[2] Wei et al. DeepSeek-OCR. arXiv:2510.18234, 2025.
[3] Li et al. OCR-Memory. ACL 2026.
[4] Feng et al. AgentOCR. arXiv:2601.04786, 2026.
[5] DeepSeek-AI. DeepSeek-V3.2. arXiv:2512.02556, 2025.
[6] Fan et al. Reading is not Reasoning (CAPS). arXiv:2608.08960, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.