acceptodds
Under review as a conference paper at ICLR 2027

Learning Sparse Multimodal Representations under Retrieval Budgets

Abstract

Sparse multimodal retrievers are typically trained to preserve dense representations, ranking structure, or average retrieval cost, yet these objectives do not determine which semantic neighbors survive when inverted-index traversal is terminated under a finite budget. We formulate Budgeted Neighborhood Preservation (BNP), which learns sparse representations to retain the local retrieval neighborhood of a frozen dense teacher within the candidate set produced under bounded posting access. BNP combines neighborhood distillation with differentiable posting-work control and a budget-aware reachability objective that estimates whether teacher neighbors can be exposed before traversal stops. At inference, retrieval uses hard Top- sparse codes and standard bounded inverted-index traversal, without access to the teacher neighborhood. Across five multimodal retrieval tasks with Qwen3-VL-Embedding-2B, BNP improves top-100 teacher-neighborhood coverage by 1.2–2.4 percentage points over the strongest cost-aware PUMA adaptation at a 20k-posting budget, while also improving ground-truth candidate inclusion on all five tasks. The advantage is largest under constrained budgets and narrows as traversal work increases. Final ranking gains are task-dependent, showing that preserving candidate accessibility and optimizing relevance are related but distinct objectives. These results suggest that sparse retrieval representations should be evaluated and learned according to what remains reachable under the execution budget used at inference.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal retrieval, sparse representation learning, budgeted neighborhood preservation, candidate generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jtkDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Budgeted Neighborhood Preservation (BNP), which learns sparse multimodal representations under limited posting budgets. It combines sparse autoencoding, neighborhood distillation, posting-cost control, and a budget-aware reachability loss. Experiments on various benchmarks and multiple backbones show consistent improvements in neighborhood coverage.

Critical strengths

1. Meaningful motivation. The main innovation is the reachability loss, which explicitly encourages sparse representations to preserve dense-teacher neighbors under limited posting budgets, rather than simply preserving similarity scores or minimizing retrieval cost.

2. Solid experiments. The evaluation includes multiple benchmarks, backbones, retrieval budgets, and ablations, with additional analysis of traversal policies and million-item retrieval.

Critical weaknesses

1. Poor presentation. The paper could be better organized and presented. In particular, Figure 1 does not clearly highlight the main innovation in the sparse training stage. Several tables are poorly formatted and difficult to read. The reference list is also rather limited, with important citations for the backbones and datasets missing. I encourage the authors to improve the overall presentation.

2. Incremental novelty and limited ranking improvements. The method builds heavily on existing sparse autoencoding and retrieval-aware training approaches such as PUMA, with the reachability loss being the main new component. While BNP improves neighborhood coverage, the gains over the strongest baseline are relatively small, and final nDCG even decreases on three of the five benchmarks. This limits the significance of the improvements in practical retrieval quality.

Questions for authors

1. Baseline implementation. Could the authors provide more details on how the baselines, especially PUMA-A and PUMA-A+FLOPS. The training configurations are unclear, including the training datasets, batch size, and other hyperparameters.

2. Loss ablation. Could the authors provide a more systematic ablation study of each loss term in the overall training objective,? This would help clarify the contribution of each loss, especially the proposed L_reach.

3. Broader benchmarks. The evaluation mainly focuses on traditional image retrieval benchmarks. Could the authors evaluate BNP on a broader and more recent multimodal retrieval benchmark, such as MMEB v2, to further demonstrate its generalization?

Reviewer_2JvCOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies sparse multimodal retrieval under a fixed posting-access budget. It proposes Budgeted Neighborhood Preservation (BNP), combining teacher-neighborhood distillation, posting-cost control, and a reachability loss to preserve useful candidates before traversal stops. Experiments show improved teacher-neighborhood coverage and candidate inclusion, although final ranking gains are mixed.

Critical strengths
  1. The paper identifies a useful distinction between preserving sparse similarity scores and preserving the candidates actually reached during retrieval. This is relevant to systems that terminate index traversal under a finite budget. The candidate-loss analysis helps explain where useful neighbors disappear.

  2. The experiments cover five retrieval tasks and several embedding backbones. Holding the traversal policy and posting budget fixed provides a controlled comparison between representations. The budget curves and objective ablations help establish when the proposed approach is useful.

  3. An anonymous implementation is included in the supplementary material. It contains the sparse encoder, training objectives, bounded inverted-index retrieval, and coverage evaluation. The code is compact and makes the main components easy to inspect.

Critical weaknesses
  1. The methodological novelty appears limited. Post-hoc Top-k sparsification, teacher distillation, support alignment, and document-frequency-based cost control already have close precedents in PUMA and DF-FLOPS. The reachability term is the main addition, but its advantage over a simpler teacher-neighbor support-overlap loss under the same cost constraint is not established.

  2. The practical gains over the strongest baseline are modest. At the default budget, teacher-neighborhood coverage improves by 1.2–2.4 points, while final nDCG decreases on three of the five tasks. Improving agreement with the teacher is useful, but the results provide limited evidence of a consistent benefit to final retrieval quality.

  3. The reachability approximation does not capture important details of the actual traversal. It depends on shared support and estimated posting volume, without explicitly modeling an item’s position within a posting list or the list-access order. The paper does not directly establish how accurately this score predicts whether a neighbor will actually be visited.

  4. The supplementary implementation contains a scale mismatch in the budget calculation. Document frequencies are computed as mean activation proportions, but the training budget is expressed in posting counts; in the supplied toy configuration, estimated volume is at most 32 against a budget of 1,500. Consequently, the reachability budget factor is always one, removing the intended budget dependence and raising a concern about consistency between the implementation and the paper.

Questions for authors
  1. How does the reachability loss compare with a simple teacher-neighbor support-overlap objective under identical neighborhood supervision and cost constraints?

  2. Can you demonstrate a clearer downstream relevance benefit at matched latency, given that nDCG decreases on three of the five tasks at the default budget?

  3. How well does the predicted reachability score agree with actual neighbor visitation under the two traversal policies?

  4. How are activation proportions converted into posting counts in the experiments? Were the reported results produced using the same budget calculation as the supplementary code?

Reviewer_2sP2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at how to find candidates for multimodal search when the inverted index has a limited access budget. The authors point out that keeping dense vectors intact, preserving the full sparse ranking, or controlling average search cost does not guarantee that useful neighbors will be found before the budget runs out. They propose BNP, which trains a Top-k sparse autoencoder on frozen multimodal embeddings. Its training combines neighborhood distillation, reconstruction, a cost constraint based on changing document frequencies, and a differentiable objective that encourages useful neighbors to be reachable within the budget.

Across five tasks, using Qwen3-VL-Embedding-2B and a budget of 20k posting accesses, BNP improves coverage of the dense teacher's top 100 neighbors by 1.2–2.4 percentage points over PUMA-A+FLOPS. It also raises the chance of including a labeled relevant item on every task. Their analysis of missed neighbors suggests that most of the gain comes from reducing the number of teacher neighbors that are never visited before the budget runs out. Final ranking results vary by task, and the coverage advantage generally gets smaller as the access budget grows.

Critical strengths
  1. The paper clearly separates keeping similarity rankings intact when all scores can be computed from making candidates reachable when index traversal is limited. Figure 3 and Table A9 divide teacher neighbors into three groups: retained, never visited, and visited but later dropped. They show that BNP's main gain comes from reducing the number that are never visited. For example, on CIRR this share falls from 17.8% to 15.7%, and on VisualNews it falls from 65.7% to 62.5%.
  2. In Table 1, BNP improves both teacher-neighborhood coverage and the chance of including a relevant item on all five tasks, without increasing average posting accesses. The budget curves, ablations, and results with other backbones provide additional support.
Critical weaknesses
  1. Equation 20 uses one estimated access fraction, (\rho(q)), for each query, together with the sparse dimensions shared by the query and a teacher neighbor. So the objective directly encourages more overlap between supports and adds a budget signal through the total posting volume. However, it does not explicitly account for where a neighbor appears in each ordered posting list or how far each list is actually traversed. Table 2 shows that adding this term helps, but it does not show whether the gain comes from modeling the budget or simply from encouraging extra support alignment.
  2. Table A5 reports the mean and standard deviation across three runs under a hard 20k posting limit, but gives (20.00\pm0.10)k accesses for VisualNews/PUMA-A. If every query in every run is capped at 20k, each run's average must also be at or below 20k. A mean this close to the cap and the reported standard deviation do not seem compatible; ordinary rounding does not appear sufficient to explain the difference. The authors should check other near-cap entries as well.
  3. The paper also does not give enough detail about the training splits, optimizer, learning rate, batch size, number of training epochs, main loss weights and temperatures, or how PUMA-A was adapted and tuned.
Questions for authors
  1. Keeping the other losses and training settings the same, what happens if the reachability term is replaced with a simple support-overlap objective for teacher neighbors? How much of the gain remains if (rho(q)) is fixed or removed?
  2. How closely does the reachability score (r(q,x)) track whether a neighbor is actually visited before the budget runs out? Does this relationship hold under both the default round-robin traversal and the alternative traversal policy? Table A13 does not include the strongest baseline, PUMA-A+FLOPS.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.