GATOR: Granularity-Aware Token-Optimized Routing for Efficient Multimodal Retrieval

Multimodal agents rely on external memory to access evidence beyond a finite context window, yet reading retrieved evidence remains costly. A document page may be represented as a caption, full text, or an image, with costs differing by more than an order of magnitude. Retaining rich representations can waste budget, whereas uniformly choosing cheap ones can discard answer-critical evidence. We propose GATOR, a cost-aware router that complements relevance ranking with query-dependent granularity selection. Under an explicit token budget, its greedy selector balances relevance, redundancy, and cost, retaining at most one representation per page while promoting coverage across pages. A lightweight scorer learns representation preferences from automatically derived weak supervision and on-policy distillation, without LLM calls during training. The same selector accepts host retrieval scores, enabling integration with existing memory systems without retraining or re-indexing them. On UniDoc and MMDocRAG, standalone GATOR reduces token-equivalent evidence cost by $41.3\%$ and $59.4\%$ relative to Vanilla RAG while improving answer F1. Mounted on Mem0 and A-Mem, it reduces evidence cost by up to $88.5\%$, with observed F1 changes ranging from $-0.011$ to $+0.042$.