- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
DocHarness-RSI addresses long multimodal document understanding from a context-construction perspective rather than through model training. To achieve this, the method keeps the VLM frozen and optimizes the harness that determines which pages are presented to the model under a fixed page budget. The harness consists of a three-stage Retrieve–Complete–Purify (RCP) pipeline. An outer failure-guided optimization loop further improves the harness by identifying the dominant failure mode and modifying the corresponding RCP component.
Critical strengths
- The paper proposes an approach for long-document understanding that does not require training or fine-tuning the VLM and operates under a limited context budget.
- The paper provides extensive ablation studies covering different components of the proposed method.
- The paper is well written and clearly presented.
- The proposed harness outperformes TextRAG baseline under the same frozen-VLM setting.
Critical weaknesses
Weaknesses:
- Paper novelty: the idea of doing long reasoning under limited budget using the traning-free harness process orginaly proposed in the VideoHarness-RSI, paper, and this paper only adapt to the long document undertanding domain, completely the same gaol. Apart from that the way how they adapted to the new domian has also minimum novelity, and it sounds more engineering, bringing and combaning the existing established ideas in document and multimodal RAG, such as multimodal retrieval and evidence/context filtering. Therefore, it remains unclear what fundamentally new methodological contribution this work provides beyond adapting and combining existing techniques for the long-document setting.
- Unfair sota comparison on table 5: The models compared in Table 5 use different backbones, such as DocAgent. It is therefore unclear whether the performance improvements stem from the proposed method itself or from differences in the underlying backbones. Similarly, the weaker performance of other methods may be attributable to their backbones rather than to the methods themselves. This makes the comparison potentially unfair and weakens the validity of the claims regarding state-of-the-art performance.
Questions for authors
Please see the weakness, and also clarify that what benfits this paper brings to the community apart from a SOTA perfromance if we consider the results as sota?