- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper introduces TRIAD, a multimodal retrieval benchmark motivated by complicated agentic settings. It covers three tracks; 1. abstraction, instance to concept or concept to instance retrieval based on WordNet hierarchies, 2. constraints, product retrieval under multiple positive and negative conditions, and 3. state, the same constraint queries spread across dialogue turns, including revisions. The benchmark has six subsets of 1,000 queries, each with a fixed 1,000-candidate pool, and use multi-positive mAP as a main metric. The authors evaluate 20 configurations of 17 embedding models, and find that no model leads on all subsets, performance drops with hierarchical distance, with negative constraints, and in multi-turn settings. Also paper demonstrates that explicit reasoning helps inconsistently.
Critical strengths
- The paper addresses a timely and practically relevant question, as agentic retrieval beyond text domain become more popular and new way of retrieval.
- The evaluation spans 20 configurations of recent embedders including dual-tower, discriminative reasoning embedders.
- The paper designed careful diagnostic design and also detailed ablations, including paired single-turn reference, hard negatives, last-turn only ablations to help deeper understanding of the embedders' behavior. Several informative findings, such as most of the multi-turn loss are from revision session, are because of this design.
Critical weaknesses
-
The main agentic motivation is not reflected in the evaluation design. The paper motivates TRIAD by demonstrating the agentic retrieval is more complicated and arguing that traditional retrieval based on self-contained query is limited. However, retrieval pipelines routinely use query rewriting or expansion, conversational query reformulation, and LLM reranking or filtering. The paper cites such work but does not evaluate it. The main results (Table 2) are almost entirely single-pass embedding models, plus one reranker with no frontier agent, LLM-based rewriter, or MLLM-as-filter pipeline is tested. Lack of this recent standard baselines hurts the contribution of the benchmark paper and also matters for the state track. For example, the paired single-turn query is essentially an oracle rewrite of the dialogue, so a simple “LLM condenses dialogue → embed” baseline could close much of the reported multi-turn gap.
-
The clarity of the benchmark design are not convincingly established. 2.1. The three “axes” are not separate capabilities at the same level and make the per-track headline numbers confusing. For example, “State” (update intent while preserving other constraints) is by construction the constraints track spread over turns, so state performance inherits constraint difficulty. 2.2. Hop distance follows WordNet’s granularity, not a clear semantic notion. For example, in the example of Figure 1, saddlebill→stork→wading bird makes wading bird a 2-hop ancestor, but another reasonable taxonomy could place it 1 hop away. The finding that performance declines with distance may therefore partly is confounded and reflect taxonomy artifacts and an underspecified query rather than a capability gap.
-
The domains are narrow and the curation is model-dependent. Fashion and Cars are highly category-specific, which limits generality. The pipeline relies heavily on Qwen-family models: Qwen3-VL-Embedding selects the ImageNet images, and Qwen3.5/3.6 annotate attributes, single-model annotation for Cars.
-
The dialogues are templated. State dialogues come from deterministic templates without paraphrasing (App. A.3). This limits the practical dialogue, which contradicts with the main motivation and limits how well the track represents real evolving user requests.
-
Some part of the writing is hard to follow. For example, the introduction reports findings (“instance-to-concept retrieval quality decreases with hierarchical distance”) before explaining how the benchmark is built or defining hierarchical distance. Also, key information such as how the dataset is built, how dialogues are generated, and how quality is ensured are mostly deferred to the appendix, so the main text is not self-contained enough to judge validity.
Questions for authors
- Can you evaluate agentic or LLM-in-the-loop pipeline (e.g., rewrite + embed + MLLM filtering/reranking) to support the agentic framing as well?
- How does a simple query-rewriting baseline perform on the state track, e.g., an MLLM condensing the dialogue into a single-turn request before embedding and see If it approaches the paired single-turn mAP?
- Given that Qwen3-VL-Embedding selected the abstraction images and Qwen3.x models annotated attributes, have you checked for bias toward Qwen-family or Qwen-backbone embedders?
- As a suggestions, adding domains beyond Fashion and Cars, and LLM-paraphrased or human-written dialogues for the state track, would strengthen generality claims.
- Consider moving a some of the construction and QA procedures in the appendix into the main text, as construction part is the core part deciding the reliability of the benchmark
[minor]
- I2C/C2I first appear at line 160 without expansion. The D1–D7 abbreviation (line 178) might be hard to follow inline without explanation