acceptodds
Under review as a conference paper at ICLR 2027

GATOR: Granularity-Aware Token-Optimized Routing for Efficient Multimodal Retrieval

Abstract

Multimodal agents rely on external memory to access evidence beyond a finite context window, yet reading retrieved evidence remains costly. A document page may be represented as a caption, full text, or an image, with costs differing by more than an order of magnitude. Retaining rich representations can waste budget, whereas uniformly choosing cheap ones can discard answer-critical evidence. We propose GATOR, a cost-aware router that complements relevance ranking with query-dependent granularity selection. Under an explicit token budget, its greedy selector balances relevance, redundancy, and cost, retaining at most one representation per page while promoting coverage across pages. A lightweight scorer learns representation preferences from automatically derived weak supervision and on-policy distillation, without LLM calls during training. The same selector accepts host retrieval scores, enabling integration with existing memory systems without retraining or re-indexing them. On UniDoc and MMDocRAG, standalone GATOR reduces token-equivalent evidence cost by and relative to Vanilla RAG while improving answer F1. Mounted on Mem0 and A-Mem, it reduces evidence cost by up to , with observed F1 changes ranging from to .

Primary area:
foundation or frontier models, including LLMs
Keywords:
Multimodal RAG, Agent Memory, Granularity Selection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_rWorOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GATOR, a post-retrieval router that selects among caption, full-text, and image representations of each page under a token budget, which is an important dimension for balancing performance and cost but neglected in previous methods. The selection mechanism combines a training strategy to train an MLP-based scorer with a cost-aware selection mechanism. Experiments on UniDoc and MMDocRAG show that GATOR substantially reduces evidence cost while maintaining or even improving answer quality.

Critical strengths
  1. This paper is based on a strong motivation, identifying an important and underexplored problem: selecting the appropriate representation granularity of retrieved multimodal evidence.
  2. The proposed GATOR mechanism considers multiple factors and can be utilized in a standalone manner or integrated with existing retrieval and memory systems.
  3. Experiments on 2 benchmarks show that GATOR can reach a better performance-cost tradeoff
Critical weaknesses

Presentation issue

  1. Why are most of the loss definitions deferred to the appendix? Moreover, the value of the margin parameter (\mu) in the margin loss appears to be unspecified.
  2. Given the large number of components in GATOR, Section 2 would be easier to follow if the method were explained alongside Figure 2. Currently, Figure 2 is referenced only once at the beginning of the section and is not used to guide the subsequent technical discussion.

Efficiency and Cost Accounting

  1. Although GATOR reduces the reported inference-time evidence cost, it requires additional preprocessing of the original data to construct multiple representations and token-number measurement. The paper should discuss this preprocessing overhead and how it trades off against—or is amortized by—the inference-time savings.

  2. The token-equivalent cost does not fully capture modality-dependent computational overhead. Images require additional processing by a vision encoder, whereas text is processed through token embeddings, resulting in different latency and resource costs. Moreover, output tokens also contribute to actual end-to-end efficiency. Evaluations based on wall-clock latency, throughput, memory consumption, or actual API cost would make the efficiency claims more convincing.


Concerns on Generalization

  1. It is unclear whether evaluation on only two benchmarks is sufficient to support the generalization claims.

  2. Unlike a general-purpose embedding retriever, GATOR’s learned scorer must be trained on dataset-specific supervision. It is therefore unclear how the method would perform in a new domain or on a dataset without annotations. Figure 3(b) already suggests that cross-dataset transfer can substantially alter the quality–cost trade-off.

  3. In addition to its trainable parameters, GATOR employs a relatively complex selection mechanism with multiple components and hyperparameters. This raises concerns about its robustness across datasets. How sensitive is the method to these hyperparameters? Do they need to be retuned for each dataset, and what is the computational or annotation cost of such tuning?

  4. Can GATOR be integrated with existing general-purpose agent harness frameworks? The paper would benefit from discussing the required interface and engineering modifications.


Limited Baselines

  1. The baseline comparison is relatively weak. The paper does not compare against existing adaptive granularity-selection or routing methods, such as OMGM (ACL 2025), Mix-of-Granularity (COLING 2025), and UniversalRAG (ACL 2026).

  2. I also suggest including an agentic baseline. For example, the model could initially receive only low-cost page summaries and then autonomously decide which pages should be expanded into full text or images.

Questions for authors

Please refer to the weakness.

Reviewer_QRfbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes GATOR, a granularity-aware routing framework that selects query-relevant evidence from captions, full text, and images under a token budget. It combines cost-aware greedy selection with lightweight learning and on-policy distillation to reduce retrieval costs while preserving answer quality.

Critical strengths
  1. The paper addresses an important efficiency challenge in multimodal retrieval by jointly considering relevance, redundancy, and representation cost.
  2. GATOR shows strong experimental results.
  3. The framework can be flexibly integrated with existing memory systems such as Mem0 and A-Mem without retraining or re-indexing, demonstrating practical applicability.
Critical weaknesses
  1. Evaluation focuses on two document QA benchmarks, leaving performance on more diverse multimodal tasks and real-world agent workloads unclear. Can the authors show real-world performance ?
  2. Cross-dataset transfer shows noticeable performance degradation in some settings, raising concerns about the robustness of learned granularity preferences. Can authors comment on this ?
Questions for authors

See weaknesses.

Reviewer_mFUnOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GATOR is a retrieval selection framework that use greedy selector to weighs context importance based on relevancy, redundancy, adjusted with the token cost. Compared to prior work, they are not changing what gets retrieved but instead choosing how the retrieved context is read via caption / full text, or multimodal context. GATOR works by do per version scoring then greedily adding the best version of context per page until the budget / unit limit is reached. The use case of GATOR is on top of available multimodal agents and memory ssytems that retrieve document pages, and it can return mixed-granularity context pertaining the same accuracy at a lower evidence cost. Compared to baselines, GATOR reduces cost by 41% on UniDoc and 59% on MMDocRAG relative to Vanilla RAG, while also raising F1 scores. It also works better coupled with prior memory framework like Mem0 and A-Mem and reduces cost by 21-89%.

Critical strengths
  1. The idea is simple and easy to plug in. With retrieval decides which page to read, GATOR offers cheap motivated way on how expensively to read it under multiple formats. They are also being explicit about the budgeting which makes efficiency becomes the cornerstone of their contribution (Section 1 and Section 2.1). The easy-to-plug manner on top of memory layers like Mem0 and A-Mem without requiring re-indexing is also something that further boosts the adoptability.

  2. Evaluation is fairly clean and result is consistent as shown. In their mounting experiments, their setup see the same retrieved pages with same model so the savings really come from the selection as shown in Table 2. The delta cost / unit breakdown also clearly separates fewer pages vs cheaper pages (as shown in App A.8 further). The gains are also generalized across datasets, models, and price controls (Table 1 and 4 + Fig. 3b).

Critical weaknesses
  1. It is not clear whether the gains come from choosing the granularity. GATOR changes 2 things at once, which the (1) pages it picks and (2) how it reads each page (choosing caption, text, r image). vanilla RAG as the baseline does neither so when GATOR wins, we don't know whether it is because of wither of them. The problem is, (1) already helps since picking pages better with BM25 already raises score form 0.194 -> 0.274 (table 1).Same problem appears when GATOR is added on top of Mem0, the trained scorer isn't even used there and Mem0 is being setup to keep the duplicate pages, and since GATOR'S one version per page rule removes those duplicates automatically, some of savings are simply for dropping repeats.

  2. The results are less solid than they look. Every experiment was run only once so we do not know the stochasticity on a rerun. Then, one result looks too good to be true -> removing small helper loss (OPD) makes score collapse from 0.296 -> 0.102 (Table 6). The cost is estimated and not measured since every iamge is assumed to cost 1.440 tokens (Eq. 14) but actual model would probably uses more. So the "%cheaper" numbers may not match real cost.

Questions for authors
  1. If we swap GATOR's page ranking with vanilla RAG's and run vanilla at the same cost, does the gain still hold? GATOR's scorer is trained on gold pages while vanilla and BM25 aren't so better ranking alone helps a lot, if GATOR's granularity choices still beat them when the page ranking held fixed, the main claim stands but if not the paper is really about a supervised page ranker only.

  2. How does GATOR compare to the memory host simply cut down to the same number of units, or deduplicated by page? What values were used for δ and M? If GATOR beats a truncated or deduplicated host then the claim holds. If it matches, the mounting results show only that keeping fewer units is cheaper.

  3. Why does removing OPD drop F1 from 0.296 to 0.102 (Table 6)? Does this hold across seeds and with retrained ablations?

  4. Table 4 doesn’t match Table 1 under either weighting of the two datasets (expected F1 0.242 or 0.228, reported 0.235). Which questions are in it?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.