- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a unique task, query-free shared visual cue discovery. In this task, a model receives a shuffled collection of images and identifies subsets linked by recurring visual evidence. It also describes the visual cue shared by each subset. To support this task, the authors construct CUESEL, a dataset derived from VIST.
Critical strengths
-
The paper presents a technically coherent factorization of a difficult structured-output problem. The proposed CUEMEM separates support recovery from cue description instead of asking a VLM to autoregressively serialize hundreds of image identifiers and descriptions in one sequence.
-
The empirical analysis is comprehensive. The authors include clustering-based baselines, task-adapted SigLIP representations, a output-interface control, cross-backbone evaluation, manual annotation and cue-compatibility audits, runtime analysis, and qualitative failure cases. These controls make it possible to distinguish the large benefit of structured factorization from the smaller incremental contribution. Although I have the task-validity concerns as in the section below.
Critical weaknesses
- The task formulation imposes an artificial exclusive partition on an inherently overlapping discovery problem. Shared visual cues are generally non-exclusive. The same image may simultaneously share an object instance with one subset. CUESEL converts the reference supports into a single partition in which each image belongs to exactly one group, and trains all cross-group image pairs as negatives.
However, I believe this creates a mismatch between the stated open-ended task and its evaluation. A visually valid cue discovered across reference groups is penalized, while performance partly measures recovery of the benchmark chosen packing partition rather than discovery of valid shared evidence. This concern is consequential because it affects both the pairwise supervision and the main PGF1 metric. The paper may consider either introduce overlapping or multi-reference evaluation (or substantially narrow its claims to recovery of a controlled primary partition in synthetically packed collections?).
- The practical need for query-free discovery is not established relative to query-based retrieval. From an industry perspective, if a user already knows the desired relations, such as the same object, clothing pattern, lighting condition, or scene types, then the text-conditioned retrieval provides a more direct and manageable formulation. Query-free discovery could potentially useful for unknown patterns (please let me know if other scenes), but the paper does not show that its discovered groups are useful to users or downstream systems. I feel this is the real valuable part of this work (Especially to these easy to observe patterns, which can be found by human quickly and then text query is more convenient. I understant this could make the evalulation process challenging.).
This limitation is especially critical because test collections may contain up to 520 predicted cue groups, many corresponding to generic properties such as color, lighting, indoor scenes, or outdoor scenes. Producing hundreds of unranked groups may create substantial cognitive burden. No user study/downstream test demonstrates improved browsing, retrieval or query formulation relative to language/text-based query search. I believe this seriously undermine the significance of this work. If the issue is reasonably explained or clarified, I will consider increasing the score.
Questions for authors
-
Why is an exclusive partition an appropriate target when one image can support multiple valid cues? Can the authors provide a more realistic overlap-aware or multi-reference annotations for a representative test subset?
-
What proportion of negative cross-group pairs actually shares an alternative visible cue, and how sensitive is relation learning to these false-negative pair labels?
-
What user or downstream task benefits from producing query-free cue groups rather than using language-conditioned retrieval?
3 comments
Dear Reviewer,
Thank you again for your constructive feedback. We have revised the manuscript in response to the comments from all reviewers. Additions and revised text are marked in red for ease of review. The main revisions include:
- Terminology: We replaced “long” with “large” when referring to image collections, retaining “long” only when discussing VLM input sequences or context length.
- Related work: We added references and discussion on set-level visual understanding and collection organization, clarifying their connections to and differences from our setting.
- Annotation and ethics: We clarified the annotation process, annotator compensation, bias-control safeguards, information collection, and the institutional determination regarding IRB requirements.
- Additional experiments: We expanded the appendix with cue-category support recovery and hard-negative analyses, paired statistical tests and three-seed results, a matched pooled-hidden-state control, output-budget sensitivity, and overlap-aware evaluation. These studies further assess cue-specific recovery, statistical reliability, the contribution of Q/K interactions, and robustness under output constraints and additional reference relations.
We would be grateful if you could review the revised manuscript when time permits. Please let us know if any questions or concerns remain; we would be happy to address them. If these revisions have satisfactorily addressed your concerns, we would appreciate your reconsideration of the paper’s overall assessment in light of the revised manuscript and additional evidence.
Best regards,
The Authors
Thank you for the detailed response and additional experiments. I appreciate the extended annotations and overlap-aware evaluation, which provide meaningful evidence that CueMem relative advantage is not eliminated by the added reference relations. However, a few concerns remain.
(1) The event-centric application is a concrete use case. Could the authors clarify which user task it supports, how the paper’s cue groups are converted into events? Is there evidence that users discover useful, previously unknown patterns through these cues?
(2) the AP decoder still produces disjoint supports, so the method cannot represent multiple overlapping cues involving the same image. The label-flipping experiment measures robustness to supervision perturbations, but does not resolve this representational limitation.
Overlapping Cues, Label Ambiguity, and Practical Scope
We thank the reviewer for the careful review and thoughtful comments. We address the concerns point by point below.
Q1. Exclusive partition and overlap-aware evaluation.
Visual cues are non-exclusive. CueSel's primary partition is a controlled evaluation convention, not a claim that each image has only one valid relation. It gives a reproducible packing-based target, but PGF1/CSA can penalize valid alternatives. We will narrow the abstract, task/evaluation discussion, and conclusion accordingly.
We extended the annotations for 134 test collections comprising 28,584 images, with each collection containing at most 300 images. This yielded 469 additional relations, including 148 spanning multiple primary reference groups.
We score the same predictions against primary and extended references. oPGF1 allows overlapping supports without first-assignment exclusivity; additional-relation recall measures recovery of newly annotated supports at IoU >= 0.5.
The method ranking remains unchanged under oPGF1, supporting its robustness to the additional reference relations. CueMem also maintains a paired primary-PGF1 advantage of +0.028 [0.018, 0.038] over Supervised CBC on this subset.
CueMem's AP decoder produces disjoint supports, and additional-relation recall evaluates how well these supports match alternative annotated relations. We will include this study, the metric definitions, and the subset-selection protocol in the revised paper, and identify overlap-aware learning and decoding as future directions.
Q2. Alternative cues in negative pairs and label sensitivity.
In the existing audit of 200 randomly sampled cross-group image pairs, 14 (7%) shared an alternative visible cue, providing an empirical estimate of label ambiguity under the audit protocol. Although labeled negative under the primary partition, these pairs can support valid alternative relations in open-ended cue discovery.
We also retrain after flipping the top 5% or 10% of cross-group pairs, ranked by frozen SigLIP cosine similarity, from negative to positive.
Both methods show performance degradation under label perturbation. CueMem's PGF1 decreases by 0.112 and 0.218 at the 5% and 10% flip levels, respectively, while remaining above Supervised CBC in both settings. Its smaller performance drops indicate greater robustness to this hard-pair label perturbation.
This label-sensitivity stress test evaluates robustness to changes in the supervision of visually similar cross-group pairs. We will include the protocol and results in the revised paper, with overlap-aware supervision as a promising direction for incorporating alternative cue relations.
Q3. Practical value versus language-conditioned retrieval.
We agree that text retrieval is preferable when users know what they seek. Query-free discovery is complementary: surfacing recurring patterns before a query is formulated. An unfamiliar archive may reveal the same landmark across weather conditions or a material detail across otherwise different scenes. Support-linked descriptions could become browsable index entries or candidate retrieval queries, rather than replacing search. Building on CueMem, we have developed an app that proactively discovers event-related cues in users' image collections and organizes the associated events into a dedicated, event-centric interface for browsing and exploration.
We appreciate the concern about presenting hundreds of unranked groups. CueSel's maximum of 520 reference groups characterizes a stress-test collection rather than a recommended display size. A practical interface could rank groups by support size, remove redundant entries, and reveal additional groups progressively instead of displaying them all at once.
The paper's Visual Haystacks stress test provides downstream evidence for structural evidence selection: with the same frozen Qwen3.5-4B, Q/K-based selection improves average answer accuracy from 59.38% to 62.66%. This result supports the value of selecting relevant visual evidence before answering in a query-conditioned setting. We will highlight this finding alongside the browsing and query-formulation motivations, clarifying how interpretable support-description pairs can serve as building blocks for collection exploration and retrieval.