Learning Sparse Multimodal Representations under Retrieval Budgets

Sparse multimodal retrievers are typically trained to preserve dense representations, ranking structure, or average retrieval cost, yet these objectives do not determine which semantic neighbors survive when inverted-index traversal is terminated under a finite budget. We formulate Budgeted Neighborhood Preservation (BNP), which learns sparse representations to retain the local retrieval neighborhood of a frozen dense teacher within the candidate set produced under bounded posting access. BNP combines neighborhood distillation with differentiable posting-work control and a budget-aware reachability objective that estimates whether teacher neighbors can be exposed before traversal stops. At inference, retrieval uses hard Top-$k$ sparse codes and standard bounded inverted-index traversal, without access to the teacher neighborhood. Across five multimodal retrieval tasks with Qwen3-VL-Embedding-2B, BNP improves top-100 teacher-neighborhood coverage by 1.2–2.4 percentage points over the strongest cost-aware PUMA adaptation at a 20k-posting budget, while also improving ground-truth candidate inclusion on all five tasks. The advantage is largest under constrained budgets and narrows as traversal work increases. Final ranking gains are task-dependent, showing that preserving candidate accessibility and optimizing relevance are related but distinct objectives. These results suggest that sparse retrieval representations should be evaluated and learned according to what remains reachable under the execution budget used at inference.