acceptodds
Withdrawn from ICLR 2027

TRIAD: Multimodal Retrieval across Abstraction, Constraints, and State

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Multimodal retrieval increasingly supports agentic systems, where user requests can involve cross-level relations, combine multiple constraints, and evolve across dialogue turns. In these settings, a highly similar result may still violate the user's requirements, while retrieved content may directly inform downstream actions without human screening. We introduce TRIAD, a multimodal retrieval benchmark for evaluating how well retrieval systems satisfy these complex, evolving requests. TRIAD comprises three tracks: abstraction, testing bidirectional retrieval between visual instances and concepts at different hierarchical levels; constraints, testing the joint satisfaction of positive and negative conditions; and state, testing the integration of requirements across dialogue turns and the updating of earlier conditions when users revise their requests. The benchmark contains six subsets, each with 1,000 queries and a fixed pool of 1,000 candidates per query, evaluated using a common multi-positive mean average precision (mAP) protocol. Simpler reference tasks and hard negatives grouped by the requirements they violate help characterize specific retrieval difficulties. We evaluate 20 configurations of 17 systems and find that no system leads across all six subsets. Performance declines with hierarchical distance, and negative-only queries generally underperform positive-only queries with the same number of constraints. Across paired tasks expressing the same final intent, multi-turn retrieval yields lower mAP in 39 of 40 configuration–domain comparisons. The benefits of explicit reasoning also vary across models and tasks. These findings highlight the gap between retrieving similar content and satisfying a complete request.

Primary area:
datasets and benchmarks
Keywords:
multimodal, embedding, benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_YXUBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces TRIAD, a multimodal retrieval benchmark motivated by complicated agentic settings. It covers three tracks; 1. abstraction, instance to concept or concept to instance retrieval based on WordNet hierarchies, 2. constraints, product retrieval under multiple positive and negative conditions, and 3. state, the same constraint queries spread across dialogue turns, including revisions. The benchmark has six subsets of 1,000 queries, each with a fixed 1,000-candidate pool, and use multi-positive mAP as a main metric. The authors evaluate 20 configurations of 17 embedding models, and find that no model leads on all subsets, performance drops with hierarchical distance, with negative constraints, and in multi-turn settings. Also paper demonstrates that explicit reasoning helps inconsistently.

Critical strengths
  1. The paper addresses a timely and practically relevant question, as agentic retrieval beyond text domain become more popular and new way of retrieval.
  2. The evaluation spans 20 configurations of recent embedders including dual-tower, discriminative reasoning embedders.
  3. The paper designed careful diagnostic design and also detailed ablations, including paired single-turn reference, hard negatives, last-turn only ablations to help deeper understanding of the embedders' behavior. Several informative findings, such as most of the multi-turn loss are from revision session, are because of this design.
Critical weaknesses
  1. The main agentic motivation is not reflected in the evaluation design. The paper motivates TRIAD by demonstrating the agentic retrieval is more complicated and arguing that traditional retrieval based on self-contained query is limited. However, retrieval pipelines routinely use query rewriting or expansion, conversational query reformulation, and LLM reranking or filtering. The paper cites such work but does not evaluate it. The main results (Table 2) are almost entirely single-pass embedding models, plus one reranker with no frontier agent, LLM-based rewriter, or MLLM-as-filter pipeline is tested. Lack of this recent standard baselines hurts the contribution of the benchmark paper and also matters for the state track. For example, the paired single-turn query is essentially an oracle rewrite of the dialogue, so a simple “LLM condenses dialogue → embed” baseline could close much of the reported multi-turn gap.

  2. The clarity of the benchmark design are not convincingly established. 2.1. The three “axes” are not separate capabilities at the same level and make the per-track headline numbers confusing. For example, “State” (update intent while preserving other constraints) is by construction the constraints track spread over turns, so state performance inherits constraint difficulty. 2.2. Hop distance follows WordNet’s granularity, not a clear semantic notion. For example, in the example of Figure 1, saddlebill→stork→wading bird makes wading bird a 2-hop ancestor, but another reasonable taxonomy could place it 1 hop away. The finding that performance declines with distance may therefore partly is confounded and reflect taxonomy artifacts and an underspecified query rather than a capability gap.

  3. The domains are narrow and the curation is model-dependent. Fashion and Cars are highly category-specific, which limits generality. The pipeline relies heavily on Qwen-family models: Qwen3-VL-Embedding selects the ImageNet images, and Qwen3.5/3.6 annotate attributes, single-model annotation for Cars.

  4. The dialogues are templated. State dialogues come from deterministic templates without paraphrasing (App. A.3). This limits the practical dialogue, which contradicts with the main motivation and limits how well the track represents real evolving user requests.

  5. Some part of the writing is hard to follow. For example, the introduction reports findings (“instance-to-concept retrieval quality decreases with hierarchical distance”) before explaining how the benchmark is built or defining hierarchical distance. Also, key information such as how the dataset is built, how dialogues are generated, and how quality is ensured are mostly deferred to the appendix, so the main text is not self-contained enough to judge validity.

Questions for authors
  • Can you evaluate agentic or LLM-in-the-loop pipeline (e.g., rewrite + embed + MLLM filtering/reranking) to support the agentic framing as well?
  • How does a simple query-rewriting baseline perform on the state track, e.g., an MLLM condensing the dialogue into a single-turn request before embedding and see If it approaches the paired single-turn mAP?
  • Given that Qwen3-VL-Embedding selected the abstraction images and Qwen3.x models annotated attributes, have you checked for bias toward Qwen-family or Qwen-backbone embedders?
  • As a suggestions, adding domains beyond Fashion and Cars, and LLM-paraphrased or human-written dialogues for the state track, would strengthen generality claims.
  • Consider moving a some of the construction and QA procedures in the appendix into the main text, as construction part is the core part deciding the reliability of the benchmark

[minor]

  • I2C/C2I first appear at line 160 without expansion. The D1–D7 abbreviation (line 178) might be hard to follow inline without explanation
Reviewer_4fiDOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TRIAD evaluates multimodal retrieval across hierarchical abstraction, positive and negative constraints, and multi-turn intent updates. It contains six subsets with 1,000 queries each and evaluates 20 configurations of 17 systems. The experiments identify difficulties with distant ancestor concepts, negation, and revised dialogue requirements.

Critical strengths
  1. Typed negatives and paired single-/multi-turn requests make the benchmark useful for diagnosing specific failures, rather than simply comparing overall scores.
  2. The controls strengthen the findings. Multi-turn degradation persists in 38/40 comparisons with matched candidate pools, and the hierarchical-distance gap remains after removing positives at the other distance.
Critical weaknesses

The main concern is whether the relevance labels consistently follow from the visible request. Appendix E.1 identifies five queries with potentially consequential ambiguity, but leaves them unchanged. For example, “not in blue” refers to different watch parts without specifying which part in the query. Such cases may penalize models for underspecified requests. The reported 98.83% accuracy measures visual attributes, not relevance labels. The human relevance study also provides structured-condition aids unavailable to the models, leaving uncertainty about whether the implicit preservation rules are clear from the actual inputs.

Questions for authors

Do the main results hold after correcting or excluding the identified ambiguous queries and other instances of the same patterns? How are users expected to infer which unmentioned attributes must be preserved? Does human agreement with the relevance labels remain high without the structured-condition aids?

Reviewer_LByLOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TRIAD, a multimodal benchmark designed to evaluate three capabilities: Abstraction, Constraints, and State. Abstraction evaluates whether a retrieval system can navigate hierarchical relationships in WordNet, such as retrieving an image of a bird given a sparrow image or retrieving images of descendant classes given a more general concept. Constraints assesses whether retrieval systems can satisfy multiple positive requirements, exclude forbidden attributes, and preserve attributes that users have not requested to change, focusing on the fashion and car domains. State evaluates the ability to handle multi-turn requests, including accumulating constraints across turns and revising previously specified values while preserving unrelated requirements. The authors develop benchmark datasets for each of these three capabilities, evaluate various models on the resulting tasks, analyze their performance, and compare their findings with previous benchmarks.

Critical strengths

1- The paper evaluates a large pool of models and provides a wide range of analyses. In my opinion, its detailed analysis and comparison with previous benchmarks are interesting and offer valuable insights into the strengths and limitations of existing models.

2- The pipeline introduced for the Abstraction component could be particularly useful for automated data generation. In my opinion, this approach could support future work on generating large-scale datasets to improve model performance in such scenarios.

Critical weaknesses

1- My main concern is the novelty of this work. Most of the capabilities evaluated by the proposed benchmark have already been explored in previous studies, and it is unclear what additional scientific insights are gained by combining them into a single framework. I am not convinced that this unification, in its current form, constitutes a sufficiently significant contribution for acceptance at ICLR. Furthermore, the constraint-based retrieval experiments are primarily limited to fashion and cars, which restricts the diversity of the evaluated scenarios and raises questions about the benchmark's generalizability to more complex, real-world retrieval tasks.

2- Another concern is the validity of the WordNet-based data generation pipeline used for the abstraction experiments. Automatically traversing parent–child relationships in the WordNet hierarchy does not necessarily guarantee that the resulting queries and retrieval tasks are semantically meaningful. For example, moving upward through the hierarchy may produce overly broad concepts for which the intended retrieval targets or relevance judgments become ambiguous. Even when the underlying taxonomic relationships are valid, the generated queries may not faithfully represent the intended levels of abstraction. Without sufficient validation of the generated samples and their relevance labels, the reported results may be misleading and comparisons between models may not reliably reflect their actual abstraction and retrieval capabilities.

Questions for authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.