acceptodds
Withdrawn from ICLR 2027

MINER: Mining Multi-Modal Internal Representation for Efficient Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

12 views

Abstract

Visual document retrieval has become essential for accessing information in visually rich documents. Existing approaches fall into two camps. Late-interaction retrievers achieve strong quality through fine-grained token-level matching but store hundreds of vectors per page, incurring large index footprints and high serving costs. By contrast, dense single-vector retrievers retain storage and latency advantages but consistently lag in quality because they compress all information into a single final-layer embedding. In this work, we first conduct a layerwise diagnostic on single-vector retrievers, revealing that retrieval-relevant signal resides in internal representations. Motivated by these findings, we propose **MINER** (**M**ining Multimodal **I**nternal Represe**N**tation for **E**fficient **R**etrieval), a lightweight plug-in module that probes and fuses internal signals across transformer layers into a single compact embedding without modifying the backbone or sacrificing single-vector efficiency. The first *Retrieval-Aligned Layer Probing* stage attaches a lightweight probe at each layer, surfacing which dimensions carry retrieval-relevant information. The subsequent *Adaptive Sparse Multi-Layer Fusion* stage applies performance-adaptive neuron-level masking to the selected layers and fuses the surviving signals into the final dense vector. Across ViDoRe V/V/V, MINER outperforms existing dense single-vector retrievers on the majority of benchmarks, with up to 4.5% nDCG@ improvement over its corresponding backbone. In settings where models support both late-interaction and dense retrieval, MINER substantially narrows the nDCG@ performance gap while retaining the storage and serving efficiency of dense retrieval. Our source code is available at [this anonymous repository](https://anonymous.4open.science/r/MINER-8F41).

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi-modal Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_JmkTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MINER, a lightweight plug-in framework that enhances single-vector visual document retrieval by extracting and fusing retrieval-relevant information from intermediate transformer layers. It combines CKA-based layer selection, alignment-aware probing, performance-adaptive neuron masking, and cross-layer fusion. Experiments on ViDoRe V1/V2/V3 show improvements across several dense retrieval backbones while preserving the index storage advantages of single-vector retrieval. The approach is practically motivated, but the novelty of its individual components, the fairness of the training comparisons, and the justification for several design choices require stronger evidence.

Critical strengths

1.The paper addresses the trade-off between retrieval quality and efficiency in visual document retrieval, aiming to improve single-vector retrieval without increasing index storage costs. 2.The experiments cover three backbones and ViDoRe V1/V2/V3, along with ablation studies, efficiency comparisons, and hyperparameter sensitivity analyses.

Critical weaknesses

1.Limited methodological novelty. Exploiting intermediate Transformer representations for retrieval and feature aggregation has been extensively studied. MLR (EMNLP 2025) investigates multi-layer representation aggregation for single-vector retrieval. Furthermore, SV-RAG (ICLR 2025) and URaG (AAAI 2026) demonstrate the retrieval capabilities of intermediate representations in MLLMs. Consequently, MINER's core motivation and overall methodology substantially overlap with prior work, with its contributions appearing largely limited to specific layer-selection and neuron-masking strategies.

2.Prior studies suggest that selecting an appropriate intermediate representation may outperform using the final-layer embedding. For example, SV-RAG reports better retrieval performance from an intermediate layer (Layer 21) than from the final layer, while FreeRet (ICML 2026) identifies lexicalization pressure in the final MLP of generative MLLMs, which can degrade semantic retrieval representations. However, MINER does not compare against a properly trained best-single-layer baseline or establish the necessity of including final-layer features in the fusion. It therefore remains unclear whether the proposed multi-layer fusion offers meaningful advantages over simpler intermediate-layer selection strategies.

3.The alignment ratio is defined as the quotient of cosine similarity and CKA, but its theoretical motivation and advantages over simpler layer-selection criteria are not sufficiently established. Moreover, replacing continuous feature weights with binary masks between the probing and fusion stages may introduce a representation mismatch.

4.MINER introduces additional contrastive training, making it difficult to distinguish the benefits of intermediate representation mining from those of additional supervised optimization. Performance gains are relatively modest in several settings.

References:

MLR (EMNLP 2025): Investigating Multi-layer Representations for Dense Passage Retrieval https://aclanthology.org/2025.findings-emnlp.1333/

SV-RAG (ICLR 2025): SV-RAG: LoRA-Contextualizing Adaptation of MLLMs for Long Document Understanding https://proceedings.iclr.cc/paper_files/paper/2025/hash/0d40c7b4d78bf7f08ba532542818f6c0-Abstract-Conference.html

URaG (AAAI 2026): URaG: Unified Retrieval and Generation in Multimodal LLMs for Efficient Long Document Understanding https://ojs.aaai.org/index.php/AAAI/article/view/39729

FreeRet (ICML 2026): FreeRet: MLLMs as Training-Free Retrievers https://proceedings.mlr.press/v306/zhu26ba.html

Questions for authors

1.What are the fundamental methodological differences between MINER and existing approaches exploiting intermediate representations for retrieval, such as MLR, SV-RAG, and URaG?

2.Why is multi-layer fusion necessary instead of directly selecting the most retrieval-effective intermediate layer? Could the authors compare MINER with a best-single-layer baseline under matched training data and computational budgets?

3.Given FreeRet's findings on lexicalization pressure in the final MLP, why does MINER still use final-layer representations as cross-modal anchors and include them in the fusion? Would excluding the final layer improve retrieval performance?

4.Would a lightweight projection head trained on the best intermediate layer or the final layer with the same contrastive objective achieve comparable improvements?

5.Do the proposed alignment ratio and performance-adaptive masking provide significant benefits over fixed layer selection and simple weighted fusion?

Reviewer_qYfdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a layer-fusion approach for Visual Document Retrieval task. This approach extracts retrieval-relevant features from intermediate layers of the transformer backbone and fuses the extracted features into a single dense representation per query/document. The extraction are done by applying a masking or a linear transformation on top of per-layer readout (mean pooling/last token) embedding. The fusion is a elementwise weighted sum of extracted features from all layers. The extraction and fusion layers are trained on top of a frozen VDR model (three backbones were experimented) using the same training data as the original VDR trained on.

Critical strengths
  • The approach is parameter-efficient, only some lightweight layers/hyper-parameters added to each layer of a frozen pretrained VDR model. Similar to other parameter-efficient tuning methods, this proposed method might help to scale the training to larger batch size and larger number of negatives, which are critical knobs in contrastive training methods. The training cost is also low (given a pretrained VDR model is provided).

  • The approach is practical. it keeps the backbone frozen and the embedding size unchanged, so index storage is identical, and adds only small query latency. It improves retrieval in some settings, most notably ViDoRe V2.

Critical weaknesses
  • The effectiveness gains are limited. on ViDoRe V1 (nDCG@5), MINER is roughly on par with its baselines for Jina (+0.1) and MoCa (−0.1). On ViDoRe V3 (nDCG@10), gains are small (+0.4 to +1.2). larger gains appear only on ViDoRe V2, and are significant only for Jina and Eager. all three benchmarks also have small corpus size (about 1k–5k pages). Real search systems index millions of documents or more, where many near-duplicate distractors make retrieval much harder and ANN search adds its own error. It is unclear whether small gains measured on a few thousand pages survive at that scale.

  • Table 2 omits the backbone baseline (the frozen pretrained models). Adding it from Table 1 shows that All Norm and ConcateLinear fall well below the frozen backbone on all three models (e.g., Eager: 47.8 vs. 56.0). Both variants can see the final layer and can represent the identity mapping, so this suggests they were poorly initialized or tuned, not that they are inherently weaker. This also raises the question of whether other ablated baselines are properly tuned.

  • The provided codebase is incomplete (e.g., config files and code inside src/model_definition are missing) and non-executable at the time of reviewing.

Questions for authors
  • What validation set is used for hyper parameter tuning in Section 4.2? Can you confirm it doesn't overlap ViDoRe V1, V2 or V3?
  • What was the latency setup? Please give a breakdown into encoding, fusion and search, plus hardware, Qdrant version and index configuration (HNSW or exact, multi-vector settings).
  • Can you add the backbone row to Table 2, and explain why All Norm and ConcateLinear score below the frozen backbone?
  • How would the gains change on larger corpora (e.g., millions scale), for example with all subsets pooled into one corpus or with distractor pages added?
Reviewer_qT5QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MINER to improve single-vector visual document retrieval using intermediate representations of frozen multimodal backbones. Specifically, it uses CKA and an alignment ratio to select and partition layers, followed by retrieval-aligned probing, adaptive neuron masking, and cross-layer fusion. Experiments on three backbones show improved retrieval performance while retaining the original embedding dimensionality and index size.

Critical strengths
  1. The proposed method addresses a practical trade-off between retrieval quality and index storage without requiring backbone updates.
  2. The evaluation covers three backbones and ViDoRe V1–V3, with consistent average improvements on V2 and V3. In addition, training details are provided, which ensure the reproducibility of the paper.
Critical weaknesses
  1. The experiments do not fully isolate the contribution of intermediate representations from additional retrieval adaptation. Appendix D reports 40 epochs for each training stage, while the adapted baselines in Table 2 all use multiple layers. How does MINER compare with a lightweight head trained only on the frozen final-layer representation, using the same data, contrastive objective, and a comparable optimization budget?

  2. The specific benefit of performance-adaptive masking remains unclear. All Base provides some evidence for probe-based masking, but the adaptive rule is not compared against a fixed alternative. Moreover, All Neurons removes both probing and masking from the full configuration. Please compare the full MINER setup against no masking and a fixed retention rule, keeping the probing and fusion-training settings consistent and matching the overall sparsity budget for the fixed-rule comparison.

  3. The reported index storage and end-to-end latency support MINER’s retrieval efficiency, but its additional model overhead is not fully quantified. To be specific, NormProbe introduces layer-specific D×DD\times D projections, yet the added parameter counts and peak inference memory are not reported. These measurements, together with an encoding/fusion versus database search latency breakdown, would provide a more complete assessment of the “lightweight” claim.

Questions for authors
  1. How does MINER compare with a lightweight head trained only on the frozen final-layer representation, using the same data, contrastive objective, and a comparable training budget?

  2. With the probing and fusion setup unchanged, how does performance change when adaptive masking is replaced by no masking or a fixed retention rule?

  3. Could the authors report the additional parameter count and peak memory overhead of MINER relative to the dense backbone, and break down query latency into encoding/fusion and database search time?

Reviewer_TYdtOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MINER improves single-vector visual document retrieval by probing and fusing intermediate representations from a frozen backbone. CKA and an alignment-ratio diagnostic guide layer selection and probe allocation, followed by performance-adaptive neuron masking and learned fusion. Experiments cover three backbones and ViDoRe V1/V2/V3. On V2, average gains over Jina and Eager are 3.6% and 4.5% relative, respectively; Jina retains the dense index footprint with approximately 1.07× query response time.

Critical strengths

1.A clever and practically valuable quality–efficiency trade-off. MINER improves retrieval while preserving single-vector indexing and output dimensionality. MINER-Jina raises V2 average nDCG@5 from 53.3 to 55.2 with modest latency overhead. Consistent gains across three backbones make this practically appealing. 2. Interesting empirical findings on the value of intermediate representations. All Neurons outperforms the dense baselines, and full MINER improves further. The probe and fusion comparisons highlight the importance of processing and combining internal representations, offering useful insights for retrieval.

Critical weaknesses
  1. The added value of the diagnostics is not fully isolated. Table 2 keeps the selected layers fixed, and all CKA-selected sets in Table 9 are contiguous suffixes of the backbone. While CKA may provide a useful way to choose the depth range, the experiments do not establish an advantage over simple depth-selection heuristics. Likewise, All Base / All Norm support using mixed probe types, but do not isolate the AR-derived allocation from alternative assignments with the same probe counts.
  2. Evaluation details and the scope of improvement need clarification. Section 5.1 specifies the validation split used for masking, but the CKA/AR estimation data, hyperparameter-selection procedure, and reported-checkpoint rule remain unclear. Gains also vary: V1 averages change from 84.3 to 84.4 for Jina and 85.7 to 85.6 for MoCa. The paper acknowledges these near-parity results and reports per-query paired t-tests; confidence intervals would further clarify the magnitude and precision of the benefits.
Questions for authors
  1. What evidence supports CKA-guided range selection beyond a depth heuristic? Can the AR-derived allocation be compared with alternative allocations using the same candidate layers, probe counts, and training budget?

  2. Section 5.1 specifies the validation data used for masking, Appendix D provides training schedules, and Section 5.5 and Appendix J report sensitivity analyses. Could you clarify the dataset, split, and sample count used for CKA/AR estimation, how the default cutoff and sparsity floor were chosen, and whether the reported results use the final or a validation-selected checkpoint?

  3. Can you provide confidence intervals for the main average gains, specifying the estimation unit and dataset weighting?

  4. Do you have a final-layer-only adaptation baseline that retains the backbone’s original embedding pipeline and receives the same supervision with a comparable training budget? This comparison could affect my assessment by distinguishing the benefit of intermediate-layer access from that of additional adaptation. If unavailable, please discuss this alternative explanation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.