acceptodds
Under review as a conference paper at ICLR 2027

ReprForge: Efficient Index Rebuilding for Visual Document Retriever Upgrades

Abstract

Upgrading a visual document retriever usually requires re-encoding the entire collection. However, this process repeats costly visual computation, which may remain valid after the upgrade. Reusing such computation requires determining which visual results the target model needs and whether they remain valid. In this paper, we present ReprForge, which answers these questions by tracing how visual outputs are produced and consumed. From one forward pass, it identifies the complete visual state required by later stages and its computational dependencies, then retains this state during index construction. At upgrade time, it checks the dependencies as loaded and the processed page inputs. When both remain unchanged, the target model resumes from the retained state with newly generated text embeddings and positions; otherwise, it performs full encoding. This links reuse to the actual computation rather than a manually chosen cache boundary. On 76 released upgrade pairs, reuse decisions agree with independent bitwise checks on tested pages. On 20,946 pages under a common processing configuration and a fixed input contract, four real upgrades achieve 4.35–4.48× rebuild speedups and reduce total build and rebuild time, including the initial build and state retention, by 61%, with bitwise-identical indexes under the tested settings. These results show that exact index updates do not require repeating the full encoding process: intermediate visual computation can remain reusable even when the final document representations change.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
visual document retrieval, index rebuilding, retriever upgrades, intermediate computation reuse, multimodal retrieval, efficient inference

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fqduOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the cost of rebuilding the index when a visual document retriever (e.g., a ColPali/ColQwen-style multi-vector retriever) is upgraded. The authors observe that many retriever upgrades change only the language-side adapter or the retrieval head while leaving the visual computation unchanged, and propose ReprForge:

  1. Discovering the state to retain. From one traced forward pass, ReprForge identifies every visual output read by later stages (the "visual interface", e.g., the three DeepStack features that Qwen3-VL injects into its language layers), together with the parameters and buffers that produce them (the "state dependencies"). This visual state is saved during the initial index build.
  2. Checking whether the state can be reused at upgrade time. The check has two levels. At the model level, the state dependencies of the source and target models are compared after loading. At the page level, the processed visual input is compared, either by rerunning the target processor on every page or, when the deployment guarantees unchanged inputs, through a "fixed input contract".
  3. Resuming with the target model. If both checks pass, encoding resumes from the retained state using the target model's own text embeddings, positions, language layers, and retrieval head; otherwise, the page is encoded in full from the raw image.

Experimental results:

  • State discovery takes less than 1.5 seconds on each of five retriever configurations.
  • On 76 released upgrade pairs, the reuse decisions agree with independent bitwise comparisons.
  • On a 20,946-page collection, using a common processing configuration (602,112 pixels) and the fixed input contract, four upgrades achieve 4.35–4.48× rebuild speedups with indexes bitwise identical to full encoding. Total time for one initial build plus four upgrades falls by 61%, at the cost of 63.9 GB of additional storage.
Critical strengths
  1. Thorough verification of exact reconstruction.
    • Across all 2,840 ViDoRe v1 pages (about 1.54 billion representation elements), the maximum absolute error is zero, and all 4 × 20,946 page representations in the main experiment are bitwise identical to full encoding.
    • The authors show that this is not because the source and target indexes are already similar: before replay, they agree on the top-1 result for only 80.5% of queries, and no query keeps the same ordered top-10. After replay, all 1,033 queries recover the target's ordered top-10 exactly.
  2. Identifying page input as a separate validity condition. Ten upgrade pairs keep the visual weights unchanged but differ in processing configuration (e.g., pixel budget) and therefore cannot be reused directly (Table 2). This is a useful reminder for practitioners that comparing weights alone does not guarantee reusability. The ablation showing that exact reconstruction fails without the DeepStack features (Appendix C.3) is also a concrete and useful observation.
Critical weaknesses

W1. The central claim that "reusing visual computation requires more than saving the final output of the vision encoder" is not well supported: in the paper's own experiments, saving the final output is exactly what most cases require

The introduction presents identifying the complete visual state as the core difficulty the method addresses (using Qwen3-VL's DeepStack as the example) and motivates trace-based automatic discovery on this basis. However, the paper's own data indicate that cases requiring multiple visual interfaces are the exception:

  • The main results do not exercise this claim. Among the five configurations in Table 1, Qwen2.5-VL, PaliGemma, and Idefics3/SmolVLM each have a single visual interface, i.e., only the final output of the visual branch (including the merger) needs to be cached; only Qwen3-VL requires four. Of the 76 upgrade pairs in Table 2, 74 belong to model families without DeepStack (Qwen2/2.5-VL, PaliGemma, Idefics3), and the main experiment in Table 3 (20,946 pages, four upgrades) uses only the single-interface Qwen2.5-VL.
  • Automatic discovery matches the manual specification exactly. In Appendix C.1, the automatically discovered interface and dependencies are identical to the manual specification (391 dependencies for Qwen2.5-VL and 316 for Qwen3-VL-4B), and Appendix C.4 shows that a replay boundary later than post-vision saves only about 1% more time per page. For existing architectures, then, choosing the cache location by hand is not difficult, and the practical benefit of automatic discovery is not demonstrated.

W2. The statistic that "107 of 151 resolvable adapter releases contain no vision-side updates" does not support the motivation that visual computation is usually reusable across upgrades

The 151 adapters were collected by searching for model-family names (Appendix I.1). Most of them are independent fine-tunes of the same Qwen or PaliGemma base by different publishers, not successive versions along an upgrade chain that a single deployment would go through. Because the sample is drawn from derivatives of the same base that share a frozen vision tower, finding that the vision side is mostly unchanged is largely a consequence of how the sample was selected, and it does not show that visual computation usually stays unchanged in real upgrades.

The main experiment has the same issue. Of the four "real upgrade" targets, only ColQwen2.5 v0.2 is a successor of the source model v0.1; Metric-AI, T-Systems, and ColNomic are models independently trained by other publishers on the same base, which is closer to switching models than to upgrading.

In addition, some of the paper's more detailed analyses seem to point to a more conservative conclusion. Among the 18 historical version pairs in Appendix I.1, 5 match in both weights and processing fields. In Appendix H.3, under each release's native processing configuration, the visual input matches the source on only 15/200, 15/200, and 30/200 pages for v0.2, T-Systems, and ColNomic, respectively.

W3. The contribution is mainly an engineering implementation, with limited methodological novelty

In implementation terms, the fixed-input-contract path used in the main experiment can be summarized as follows: save the output of the visual branch during the initial build; at upgrade time, compare once the parameters and buffers that produce this output in the source and target models; if they match, load the cached state and run only the language side and the retrieval head. This is essentially memoization with dependency checking. The tracing implementation and bitwise verification are solid, but the paper offers no insight beyond the engineering, such as an analysis or characterization of which kinds of upgrades preserve the validity of intermediate representations. The main contributions lie in the system implementation and careful timing measurements.

In the related work, the paragraph on reuse of intermediate computation cites only HELIX (2018) and materialized-view maintenance (1995). Similar mechanisms already exist in recent LLM/VLM inference systems and are not discussed:

  • Activated LoRA (arXiv 2504.12397) modifies how the adapter is applied so that an adapter, once switched in, can directly reuse the base model's KV cache without recomputation;
  • VLCache (arXiv 2512.12977) reuses the encoder cache and KV cache of recurring multimodal inputs across requests and has been integrated into SGLang;
  • LRAgent (arXiv 2602.01053) splits the cache in multi-LoRA settings into a component shared from the base model and an adapter-specific component, and shares the former.

ReprForge's increment lies mainly in applying the mechanism of "reuse shared upstream computation and recompute only the changed downstream part" to offline index rebuilding, with weight comparison used to decide reuse across versions.

Questions for authors
  1. Necessity of automatic discovery. On which existing or foreseeable architectures is manually identifying the visual interface genuinely difficult or error-prone, so that automatic discovery becomes necessary? Appendix K states that the method requires the visual interface to appear as module outputs. Does this mean the method covers precisely those architectures where the boundary is also easy to identify by hand?
  2. Real upgrade chains. Could you report, for successive versions from the same publisher (e.g., colqwen2 v0.1 → v1.0, colpali v1.2 → v1.3), what fraction of pages can be reused under each release's native processing configuration, and the corresponding end-to-end speedups?
  3. Gains under native configurations. The 4.35–4.48× speedups in the main experiment rely on forcing a common 602,112-pixel configuration and the fixed input contract. What are the end-to-end speedups if each of the four targets uses its own released processing configuration with the generic validation path? Since the common configuration changes the target's retrieval quality (ColNomic nDCG@10 drops by 1.83 points), how do the authors view the fact that what is exactly reproduced is a target model with a modified configuration?
  4. Relation to inference-side cache reuse. Compared with inference-side work that reuses shared computation, such as Activated LoRA, VLCache, and LRAgent, what is the essential methodological difference of ReprForge beyond the application setting?
Reviewer_26vrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The manuscript address the problem of efficiently rebuilding visual document retreival indexes when the underlying retreival model is upgraded. the authors propose RepForge, which traces a forward pass to automatically identify and cahce intermediate visual representations and their dependencies, and determines at upgrade time what cached representations can be reused. When the relevant model components and the processed inputs remain unchanged, these cahced representations are reused to reconstruct the target models embeddings, avoiding redundant computation. They demonstrate correct reuse decisions over 76 upgrade pairs with gains of around 5x over full recomputation and large reduction in cumulative build and rebuild time under tested settings.

Critical strengths

None

Critical weaknesses

Limited novelty and unclear motivation - I am not able to understand the novelty beyond caching intermediate results and determining at update time which cached portions are reusable and which need to be recomputed. The forward-pass mechanism for automatically determining a replay boundary seems to be the main technical contribution. However, the need for this automation is not sufficiently motivated. The main counterexample presented to simply caching the final visual features is Qwen3-VL, which consumes intermediate visual representations. However, this can be handled through model-specific rules specifying what needs to be cached. It is unclear whether these custom feature-consumption patterns are sufficiently widespread and difficult to maintain as manual rules. Table 1 lists interfaces containing either one or four visual outputs - something that can be addressed through manual configs. The manuscript does not convincingly demonstrate where automatic discovery provides a substantial advantage over manual specification. A detailed analysis of update patterns and the resulting difficulties for manual caching would help establish the need for the proposed system.

Lack of strong baselines - The primary efficiency baseline is a full index rebuild. This demonstrates the benefit of skipping reusable computation, but does not establish the benefit of the proposed automatic discovery mechanism over a manually configured caching system or other systems present in literature that tackle similar problems. The manuscript doesnt go in detail about advantages of automatic discovery (say quantifiable metrics for integration effort, maintenance, human error in config specification, etc). There is no experimental comparison against other relevant computation reuse methods, or a discussion of methodological differences from these approaches, or a discussion of absence of such methods and establishing the need for the proposed system.

Unclear Presentation - Section 3.1 introduce the tags but doesnt detail how these tags interact, how they are used to calcualte the "visual frontier", visual interface and replay boundary. These appear to be the main contributions of the proposed work and so a precise algorithm/pseudocode in the main paper is important for understanding and assessing the method. The brief discussion in L363-365 on rotary position buffers and evaluation of this kind should be explored more in the main paper as they could help explain why automatic dependency discovery might offer advantage over simpler manual configs. The presentation of Table 1 also lacks context - all configurations achieve exact recovery, and all traces take less than 1.5 seconds, but it is unclear how meaningful these result are in absence of comparison with strong baselines. also, would a trace time of 10 seconds significantly affect the intended deployment of this system? The state sizes are also reported without enough discussion of what constitutes a good storage–recomputation trade-off for each architecture. Similar criticism holds for results in table 2.

Cache-selection tradeoff - If automatic replay-boundary discovery is the main novelty, a detailed analysis of the trade-off between saved-cache size and recomputation time would help assess its value. There is also a third factor: the nature of subsequent updates, which determines whether a retained state remains reusable. Section 3.4 specify that similar measurements are recorded so that deployment system can choose which states to retain, but does not provide or evaluate a general selection policy, which leaves unclear as to how to make these measurements actionable across different storage budgets and update patterns. An exploration of the Pareto frontier across the three axes would help understand the need for automatic discovery. Some analysis along these lines are covered in the appendices, however they should be surfaced in the main paper to support technical novelty and should be more broader to understand when different retention strategies are preferable.

Moreover, a single forward pass does not establish coverage of unseen data-dependent paths. Even though these limitations are acknowledged in Appendix K, an operational handling of such changes remains insufficiently explained. This significantly limits the scope of the system.

Questions for authors
  • What is the exact mechanism of calculation of the replay boundary and whats the granularity of such a boundary?
  • What are concrete examples where the proposed automatic tracing mechanism identifies a reusable state or dependency that would be difficult to specify through manual caching rules?
  • How does the proposed mechanism generalize across different update patterns, and how should the replay boundary be selected in practice?
Reviewer_SivsOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents a method for caching intermediate states when constructing indexable representations. IIUC, an analogy would be a KV cache for text generation, where a prefix can be reused across different inputs. In this case there are two main differences:

  1. We are caching intermediate layers of a multimodal embedding system rather than a prefix for text generation.
  2. We are reusing the cache across upgrades, so there is likely a large delay from when the cached representation is stored and would need to be reused.

Part of their approach is the automatic detection of which states need to be stored --- these are likely shareable directly by the consumer, but the automatic detection alleviates the need directly to do so.

AFAICT, the systems angle of caching these representations is the only contribution in the paper, with explanation why this caching is doable and the latency savings.

Critical strengths

S1 - Efficient indexing is an important research area, and it is particularly beneficial to have research on the multimodal setting.

S2 - The approach yields substantial empirical latency savings.

Critical weaknesses

W1 - The paper does not provide analysis indicating under which conditions the caching approach is cost-efficient compared to recomputing. For instance, if recomputing is only done once per year, then recomputing may be somewhat slow but less expensive by dollar cost.

W2 - The approach is only applicable when the upgrade shares some overlapping computation, and will not work if the whole model was finetuned or if the model was switched. Alternative research directions could be towards document embeddings that never need to be upgraded because they already capture the majority of useful features, or split architectures that allow query embeddings to be updated while document embeddings stay fixed. The latter is already supported in various enterprise embedding offerings. In summary, the paper should further motivate why this particular setting is appealing.

W3 - It is unclear how significant is the contribution of automatic tracing, given this should be deterministic if the user shares details from finetuning.

Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.