acceptodds
Withdrawn from ICLR 2027

DocHarness-RSI: Failure-Guided Harness Evolution around Frozen Vision-Language Models for Long Multimodal Document Understanding

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Answering questions over long multimodal documents requires locating sparse evidence scattered across complex layouts and hundreds of pages, all within a fixed page and context budget. Frontier Vision-Language Models (VLMs) reason well once the right pages are in view, yet the harness that selects and organizes those pages is typically a static retriever or a heuristic truncation rule, leading to missed evidence, incomplete cross-page coverage, and diluted context. We argue that this harness, rather than the model, is the primary bottleneck, and that optimizing it around a frozen VLM unlocks substantial headroom with no parameter updates. We present DocHarness-RSI, which treats the context-construction harness as an executable program and improves it recursively. The inner harness implements a Retrieve-Complete-Purify procedure: multi-channel retrieval surfaces candidate pages, question-conditioned expansion completes cross-page evidence chains, and utility-based purification removes low-value pages so the budget is spent on what matters. The outer loop is failure-guided: it inspects residual errors on development data, traces each one to the responsible harness stage, mutates only that stage, and admits a candidate program only after it passes dual-gated verification. Because the search is anchored to the currently binding bottleneck, effort migrates across stages as earlier failures are resolved, which keeps the harness compact while steadily raising both evidence recall and context purity. On long-document understanding benchmarks, the evolved harness delivers consistent gains over strong retrieval and truncation baselines, showing that executable harness evolution is an effective optimization layer for frozen VLM agents. Code will be released.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Document Understanding, LLM, Harness

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_yrLHOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

DocHarness-RSI addresses long multimodal document understanding from a context-construction perspective rather than through model training. To achieve this, the method keeps the VLM frozen and optimizes the harness that determines which pages are presented to the model under a fixed page budget. The harness consists of a three-stage Retrieve–Complete–Purify (RCP) pipeline. An outer failure-guided optimization loop further improves the harness by identifying the dominant failure mode and modifying the corresponding RCP component.

Critical strengths
  • The paper proposes an approach for long-document understanding that does not require training or fine-tuning the VLM and operates under a limited context budget.
  • The paper provides extensive ablation studies covering different components of the proposed method.
  • The paper is well written and clearly presented.
  • The proposed harness outperformes TextRAG baseline under the same frozen-VLM setting.
Critical weaknesses

Weaknesses:

  • Paper novelty: the idea of doing long reasoning under limited budget using the traning-free harness process orginaly proposed in the VideoHarness-RSI, paper, and this paper only adapt to the long document undertanding domain, completely the same gaol. Apart from that the way how they adapted to the new domian has also minimum novelity, and it sounds more engineering, bringing and combaning the existing established ideas in document and multimodal RAG, such as multimodal retrieval and evidence/context filtering. Therefore, it remains unclear what fundamentally new methodological contribution this work provides beyond adapting and combining existing techniques for the long-document setting.
  • Unfair sota comparison on table 5: The models compared in Table 5 use different backbones, such as DocAgent. It is therefore unclear whether the performance improvements stem from the proposed method itself or from differences in the underlying backbones. Similarly, the weaker performance of other methods may be attributable to their backbones rather than to the methods themselves. This makes the comparison potentially unfair and weakens the validity of the claims regarding state-of-the-art performance.
Questions for authors

Please see the weakness, and also clarify that what benfits this paper brings to the community apart from a SOTA perfromance if we consider the results as sota?

Reviewer_xFRsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

DocHarness-RSI optimizes document context construction around frozen vision-language models through a Retrieve–Complete–Purify pipeline. A failure-guided outer loop modifies individual pipeline components and selects candidates through development and gating evaluations. Experiments report improvements on MMLongBench-Doc and transfer to LongDocURL.

Critical strengths

Practical problem formulation. Optimizing context construction around frozen VLMs addresses an important deployment problem without requiring model fine-tuning. The modular pipeline provides an understandable structure for improving document evidence selection.

Promising benchmark results. The method reports 75.15% accuracy on MMLongBench-Doc, exceeding SLEUTH by 2.37 percentage points, and 66.36% on LongDocURL, exceeding DocAgent by 2.58 points. These results suggest useful improvements across two document benchmarks.

Informative evidence diagnostics. Table 3 measures both coverage and purity, revealing that purification increases evidence precision from 18.59% to 83.86% while reducing recall from 90.38% to 80.79%. This analysis helps explain the trade-off behind the smaller final context.

Critical weaknesses

The formal method does not accurately describe the primary implementation. Algorithm 1 and Equation (7) present purification as greedy utility-based selection. Appendix A.2 instead describes a GPT-5.6-Terra verifier inspecting up to 24 candidate pages, with the utility function used only after service failures. This distinction materially changes the method’s computational requirements and the interpretation of its ablations.

The final page budget does not establish comparable total inference resources. Although the answerer receives at most eight pages, the purification VLM can inspect 24 pages beforehand. Purification alone averages 40.56 seconds on MMLongBench-Doc, compared with 5.13 seconds for answering (Table 9). The appendix usefully reports these costs, but comparisons matching total visual exposure, model calls, or token expenditure are needed to assess the accuracy–cost advantage.

The benefit of recursive program evolution is not sufficiently isolated. Table 4 provides component-removal results, but no comparison against a static RCP pipeline with the same verifier or a search baseline with the same proposal budget. The proposer configuration, candidate counts, concrete code mutations, and repeated-search variability are also insufficiently specified. Consequently, the gains cannot be confidently attributed to failure-guided code evolution.

The separation between search and evaluation needs clarification. Document-disjoint development, gate, and held-out partitions are asserted, but their sizes and the partition underlying each reported result are not clearly documented. Additionally, Section 4.1 describes LongDocURL as evaluation-only, while Section 4.3 describes its failure distribution as prompting scope selection. This may reflect imprecise wording, but it leaves the zero-shot transfer protocol unclear.

Failure attribution is not established causally. Appendix B.4 acknowledges that “Context Dilution” includes incorrect answers despite complete evidence and can therefore capture intrinsic reasoning or perception failures. Using this proxy to identify the responsible pipeline stage requires validation. Otherwise, the claimed diagnosis-driven optimization may respond to errors that context purification cannot resolve.

Questions for authors

Which purification implementation produces the main results? Please reconcile Algorithm 1 with Appendix A.2 and compare the VLM verifier against the deterministic utility-based selector.

How does DocHarness-RSI compare with a static RCP pipeline using the same verifier and with search baselines given identical proposal and inference budgets? Please report repeated-search variability and total search cost.

What are the document and question counts for the development, gate, and held-out sets, and which partitions support Tables 1–4? Were the LongDocURL diagnostics strictly post hoc, with no influence on program selection or design?

How are failures assigned to the three categories, and how reliably do these assignments identify beneficial interventions? In particular, how are context dilution and intrinsic answerer errors distinguished?

Reviewer_xET4OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DocHarness-RSI, a framework for improving long-document understanding by optimizing the context-construction pipeline around a frozen vision-language model (VLM), without updating the model’s parameters.

The method has two parts. The inner Retrieve-Complete-Purify pipeline retrieves candidate pages using lexical, dense, and visual signals, expands the candidates to recover evidence spread across pages, and then removes irrelevant or redundant pages to fit a fixed page budget. The outer loop diagnoses errors on development data, asks an LLM to propose targeted changes to the responsible component, and retains changes only when they improve development performance without degrading a separate gate set.

On MMLongBench-Doc, the authors report 75.15% overall accuracy, compared with 72.78% for the strongest listed baseline, SLEUTH. On LongDocURL, the proposed method achieves 66.36%, compared with 63.78% for DocAgent, without adapting the harness to the target benchmark. Ablations suggest that multimodal retrieval, cross-page completion, purification, and failure-guided evolution each contribute to the final result.

The paper’s main contribution is an approach for evolving executable document-context construction around a frozen VLM, supported by experiments across two long-document benchmarks. It also reports substantial API usage, which is relevant to evaluating the practical cost of the method.

Critical strengths
  1. A practical and clearly motivated problem. Selecting the right evidence pages is a real challenge in long multimodal documents. The separation into evidence miss, cross-page incompleteness, and context dilution gives the method a useful structure and makes its design choices easy to understand.

  2. Promising results with informative ablations. The method improves over the listed baselines on both MMLongBench-Doc and LongDocURL. The component analysis suggests that multimodal retrieval, cross-page completion, and purification address different failure modes. The attempt to validate changes on a document-disjoint gate set is also a sensible safeguard against overfitting the development set.

Critical weaknesses
  1. The novelty of the outer-loop optimization is not fully isolated. The main performance improvements appear to rely heavily on combining multiple retrieval channels, expanding evidence across sections, and using an LLM to remove irrelevant pages. These are useful design choices, but the paper should better distinguish the value of failure-guided recursive evolution from the value of simply adding these components. The outer-loop ablation helps, but more repeated runs and comparisons against alternative harness-search strategies would make the central claim stronger.

  2. The cost and reproducibility trade-off needs more attention. The appendix reports an average of approximately 62,966 tokens per MMLongBench-Doc question and 76,045 per LongDocURL question across the stated evaluation pipelines, with purification taking around 40 seconds per question on average. These costs may be substantial for practical use. In addition, the system depends on remote model APIs, including an LLM-based purification stage. The paper should better quantify how much the gains cost, how sensitive results are to model or API changes, and whether a cheaper alternative can retain most of the improvement.

Questions for authors
  1. How much of the final improvement is specifically attributable to failure-guided recursive harness evolution, compared with using the final Retrieve-Complete-Purify pipeline directly or optimizing the same components with a simpler search strategy? Can the authors report results across multiple independent search runs?

  2. Can the authors provide a more direct accuracy-cost comparison against the strongest baselines, including total API cost or a comparable cost estimate, rather than token counts alone? How much does performance change if the LLM-based purification stage is replaced with a cheaper model or a non-LLM selection method?

  3. How robust is the evolved harness to changes in the downstream VLM, retrieval encoders, and remote API versions? The cross-model results are encouraging, but it would help to clarify which parts of the harness are model-independent and which require retuning.

Reviewer_GDxrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper argues that in long multimodal document QA, the main bottleneck is which pages get shown to the model, not the model's reasoning ability. It proposes a Retrieve-Complete-Purify pipeline that selects pages for a frozen VLM, wrapped in an outer loop where an LLM rewrites the code of whichever stage error analysis blames, keeping a change only if it passes a dev split and a held-out gate split. With GPT-5.6 as the frozen answerer, the method reports state-of-the-art results on MMLongBench-Doc and zero-shot transfer to LongDocURL, plus ablations on each stage and on the outer loop.

Critical strengths

The oracle analysis is useful. It shows that, with the same model and the same page budget, feeding the right pages closes a large share of the error, and that a small, clean context can beat a larger one with higher recall.

Critical weaknesses

My first concern is that the paper's claimed contribution, failure-guided recursive self-improvement, is never really demonstrated. The final pipeline is assembled from standard parts (hybrid retrieval, neighbour expansion, LLM reranking), the search is short, and the accepted path simply walks through the three modules the authors designed in advance. No code diffs, no examples of what the LLM changed, and no comparison to a hand-built version of the same pipeline. I can not tell the "evolution" apart from manual engineering.

The second concern is that the results look driven by the strongest closed-source model rather than by the method. Even the trivial BM25 baseline gets a large boost from GPT-5.6, the key Purify stage is essentially GPT-5.6 reranking many full-page images at a cost far beyond the baselines, and simple alternatives such as a strong visual retriever plus the same reranker, or full-context input, are missing.

On top of that, the paper does not compare with recent long-document methods such as DocSeeker[1], Doc-V*[2] and MM-Doc-R1[3]. The paper itself claims the harness is model-agnostic, so a same-backbone comparison is feasible.

Overall, the paper reads more as a strong-model benchmark result than as a validated new method.

[1] DocSeeker: Structured Visual Reasoning with Evidence Grounding for Long Document Understanding

[2] Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA

[3] MM-Doc-R1: Training Agents for Long Document Visual Question Answering through Multi-turn Reinforcement Learning

Questions for authors
  1. What exactly did the LLM proposer change in each accepted generation, and how does the final pipeline differ from one designed by hand?

  2. Was LongDocURL used in any form during search or failure analysis (the failure-distribution figure appears to need its evidence labels), and if so, how is the transfer still zero-shot?

  3. Please report the results over multiple runs with variance, different splits, more backbones.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.