acceptodds
Under review as a conference paper at ICLR 2027

CueMem: Discovering Shared Visual Cues in Long Image Collections

Abstract

Large visual collections contain recurring evidence that links images beyond their dominant content, enabling collection-level organization and exploration. In this paper, we define query-free shared visual cue discovery: given a long image collection, a model must identify image subsets grounded in recurring, interpretable visual cues and describe what each subset shares, without cue-specific queries or a predefined cue inventory. Conventional VLMs must recover sparse recurring evidence amid increasingly many distractors, yet standard token-level supervision does not directly separate supporting images from irrelevant ones. To study this problem, we introduce CUESEL, a diagnostic benchmark with 4k packed collections and 73k human-filtered cue groups spanning object, appearance, instance, and scene cues. To overcome the lack of explicit support modeling, we propose CUEMEM, which transforms internal VLM query/key interactions into cue-induced image affinities and uses inferred supports to ground cue generation. It achieves 0.747 Pairwise Grouping F1 (PGF1), compared with 0.405 for the strongest supervised flat VLM. Its advantage over flat VLMs grows as cue support becomes sparse relative to the collection, indicating that explicit relation learning and support-conditioned generation help VLMs recover recurring evidence under increasing visual distraction. Code and model data will be released.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
visual collection mining, visual cue discovery, query-free visual grouping, vision-language model

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_zgw4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a unique task, query-free shared visual cue discovery. In this task, a model receives a shuffled collection of images and identifies subsets linked by recurring visual evidence. It also describes the visual cue shared by each subset. To support this task, the authors construct CUESEL, a dataset derived from VIST.

Critical strengths
  1. The paper presents a technically coherent factorization of a difficult structured-output problem. The proposed CUEMEM separates support recovery from cue description instead of asking a VLM to autoregressively serialize hundreds of image identifiers and descriptions in one sequence.

  2. The empirical analysis is comprehensive. The authors include clustering-based baselines, task-adapted SigLIP representations, a output-interface control, cross-backbone evaluation, manual annotation and cue-compatibility audits, runtime analysis, and qualitative failure cases. These controls make it possible to distinguish the large benefit of structured factorization from the smaller incremental contribution. Although I have the task-validity concerns as in the section below.

Critical weaknesses
  1. The task formulation imposes an artificial exclusive partition on an inherently overlapping discovery problem. Shared visual cues are generally non-exclusive. The same image may simultaneously share an object instance with one subset. CUESEL converts the reference supports into a single partition in which each image belongs to exactly one group, and trains all cross-group image pairs as negatives.

However, I believe this creates a mismatch between the stated open-ended task and its evaluation. A visually valid cue discovered across reference groups is penalized, while performance partly measures recovery of the benchmark chosen packing partition rather than discovery of valid shared evidence. This concern is consequential because it affects both the pairwise supervision and the main PGF1 metric. The paper may consider either introduce overlapping or multi-reference evaluation (or substantially narrow its claims to recovery of a controlled primary partition in synthetically packed collections?).

  1. The practical need for query-free discovery is not established relative to query-based retrieval. From an industry perspective, if a user already knows the desired relations, such as the same object, clothing pattern, lighting condition, or scene types, then the text-conditioned retrieval provides a more direct and manageable formulation. Query-free discovery could potentially useful for unknown patterns (please let me know if other scenes), but the paper does not show that its discovered groups are useful to users or downstream systems. I feel this is the real valuable part of this work (Especially to these easy to observe patterns, which can be found by human quickly and then text query is more convenient. I understant this could make the evalulation process challenging.).

This limitation is especially critical because test collections may contain up to 520 predicted cue groups, many corresponding to generic properties such as color, lighting, indoor scenes, or outdoor scenes. Producing hundreds of unranked groups may create substantial cognitive burden. No user study/downstream test demonstrates improved browsing, retrieval or query formulation relative to language/text-based query search. I believe this seriously undermine the significance of this work. If the issue is reasonably explained or clarified, I will consider increasing the score.

Questions for authors
  1. Why is an exclusive partition an appropriate target when one image can support multiple valid cues? Can the authors provide a more realistic overlap-aware or multi-reference annotations for a representative test subset?

  2. What proportion of negative cross-group pairs actually shares an alternative visible cue, and how sensitive is relation learning to these false-negative pair labels?

  3. What user or downstream task benefits from producing query-free cue groups rather than using language-conditioned retrieval?

3 comments
AuthorsOpenReview ↗

Overlapping Cues, Label Ambiguity, and Practical Scope

We thank the reviewer for the careful review and thoughtful comments. We address the concerns point by point below.

Q1. Exclusive partition and overlap-aware evaluation.

Visual cues are non-exclusive. CueSel's primary partition is a controlled evaluation convention, not a claim that each image has only one valid relation. It gives a reproducible packing-based target, but PGF1/CSA can penalize valid alternatives. We will narrow the abstract, task/evaluation discussion, and conclusion accordingly.

We extended the annotations for 134 test collections comprising 28,584 images, with each collection containing at most 300 images. This yielded 469 additional relations, including 148 spanning multiple primary reference groups.

We score the same predictions against primary and extended references. oPGF1 allows overlapping supports without first-assignment exclusivity; additional-relation recall measures recovery of newly annotated supports at IoU >= 0.5.

MetricCueMemSupervised CBCFlat Qwen3.5
PGF1, primary partition0.74500.71700.3789
oPGF1, overlap-aware0.73730.71130.3729
Additional-relation recall @0.50.18190.12390.0516

The method ranking remains unchanged under oPGF1, supporting its robustness to the additional reference relations. CueMem also maintains a paired primary-PGF1 advantage of +0.028 [0.018, 0.038] over Supervised CBC on this subset.

CueMem's AP decoder produces disjoint supports, and additional-relation recall evaluates how well these supports match alternative annotated relations. We will include this study, the metric definitions, and the subset-selection protocol in the revised paper, and identify overlap-aware learning and decoding as future directions.

Q2. Alternative cues in negative pairs and label sensitivity.

In the existing audit of 200 randomly sampled cross-group image pairs, 14 (7%) shared an alternative visible cue, providing an empirical estimate of label ambiguity under the audit protocol. Although labeled negative under the primary partition, these pairs can support valid alternative relations in open-ended cue discovery.

We also retrain after flipping the top 5% or 10% of cross-group pairs, ranked by frozen SigLIP cosine similarity, from negative to positive.

Training labelsCueMem PGF1Supervised CBC PGF1
Original primary-partition labels0.74720.7170
5% highest-similarity negative pairs flipped0.63550.4573
10% highest-similarity negative pairs flipped0.52880.3553

Both methods show performance degradation under label perturbation. CueMem's PGF1 decreases by 0.112 and 0.218 at the 5% and 10% flip levels, respectively, while remaining above Supervised CBC in both settings. Its smaller performance drops indicate greater robustness to this hard-pair label perturbation.

This label-sensitivity stress test evaluates robustness to changes in the supervision of visually similar cross-group pairs. We will include the protocol and results in the revised paper, with overlap-aware supervision as a promising direction for incorporating alternative cue relations.

Q3. Practical value versus language-conditioned retrieval.

We agree that text retrieval is preferable when users know what they seek. Query-free discovery is complementary: surfacing recurring patterns before a query is formulated. An unfamiliar archive may reveal the same landmark across weather conditions or a material detail across otherwise different scenes. Support-linked descriptions could become browsable index entries or candidate retrieval queries, rather than replacing search. Building on CueMem, we have developed an app that proactively discovers event-related cues in users' image collections and organizes the associated events into a dedicated, event-centric interface for browsing and exploration.

We appreciate the concern about presenting hundreds of unranked groups. CueSel's maximum of 520 reference groups characterizes a stress-test collection rather than a recommended display size. A practical interface could rank groups by support size, remove redundant entries, and reveal additional groups progressively instead of displaying them all at once.

The paper's Visual Haystacks stress test provides downstream evidence for structural evidence selection: with the same frozen Qwen3.5-4B, Q/K-based selection improves average answer accuracy from 59.38% to 62.66%. This result supports the value of selecting relevant visual evidence before answering in a query-conditioned setting. We will highlight this finding alongside the browsing and query-formulation motivations, clarifying how interpretable support-description pairs can serve as building blocks for collection exploration and retrieval.

Dear Reviewer,

Thank you again for your constructive feedback. We have revised the manuscript in response to the comments from all reviewers. Additions and revised text are marked in red for ease of review. The main revisions include:

  1. Terminology: We replaced “long” with “large” when referring to image collections, retaining “long” only when discussing VLM input sequences or context length.
  2. Related work: We added references and discussion on set-level visual understanding and collection organization, clarifying their connections to and differences from our setting.
  3. Annotation and ethics: We clarified the annotation process, annotator compensation, bias-control safeguards, information collection, and the institutional determination regarding IRB requirements.
  4. Additional experiments: We expanded the appendix with cue-category support recovery and hard-negative analyses, paired statistical tests and three-seed results, a matched pooled-hidden-state control, output-budget sensitivity, and overlap-aware evaluation. These studies further assess cue-specific recovery, statistical reliability, the contribution of Q/K interactions, and robustness under output constraints and additional reference relations.

We would be grateful if you could review the revised manuscript when time permits. Please let us know if any questions or concerns remain; we would be happy to address them. If these revisions have satisfactorily addressed your concerns, we would appreciate your reconsideration of the paper’s overall assessment in light of the revised manuscript and additional evidence.

Best regards,
The Authors

Thank you for the detailed response and additional experiments. I appreciate the extended annotations and overlap-aware evaluation, which provide meaningful evidence that CueMem relative advantage is not eliminated by the added reference relations. However, a few concerns remain.

(1) The event-centric application is a concrete use case. Could the authors clarify which user task it supports, how the paper’s cue groups are converted into events? Is there evidence that users discover useful, previously unknown patterns through these cues?

(2) the AP decoder still produces disjoint supports, so the method cannot represent multiple overlapping cues involving the same image. The label-flipping experiment measures robustness to supervision perturbations, but does not resolve this representational limitation.

Reviewer_BP8nOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how a model can identify recurring visual cues in a large image collection without being given a specific query. The output consists of image groups and descriptions of the visual evidence shared within each group.

The authors introduce CueSel, a benchmark constructed by packing human-annotated groups from VIST into shuffled collections. They also propose CueMem, which extracts image-level affinities from a VLM’s internal query/key interactions, trains these affinities with pairwise group supervision, and uses affinity propagation to recover image groups. The model then generates a description conditioned on each predicted group.

CueMem achieves 0.747 Pairwise Grouping F1, compared with 0.405 for the strongest flat VLM and 0.717 for a supervised clustering-and-captioning pipeline. The paper also examines longer collections, alternative clustering methods, output-format controls, and human assessments of cue descriptions.

Critical strengths
  1. The benchmark offers a useful and carefully documented diagnostic. CueSel tests a substantial length shift: training collections average about 31 images, whereas test collections average about 238. Recovering small groups under this shift is a meaningful challenge. The annotation filtering, source-story and exact-image split isolation, and manual audits are valuable safeguards. The authors also acknowledge that their reference partition is not the only possible interpretation of a collection.
  2. The method performs well against several relevant controls. The improvement over flat Qwen3.5 is large, and CueMem remains ahead when support prediction and captioning are separated for the flat baseline. More importantly, the paper includes a supervised SigLIP-based clustering pipeline using the same AP decoder. CueMem improves PGF1 from 0.717 to 0.747 and CSA from 0.708 to 0.748 against this stronger comparison. Removing relation supervision reduces PGF1 to 0.706, providing evidence that the pairwise objective contributes beyond the structured output procedure.
Critical weaknesses
  1. The benchmark leaves an important ambiguity between cue discovery and recovery of source-related image groups. Reference groups are annotated within short VIST source sets and then packed into larger collections. This is a reasonable diagnostic construction, but it may make shared event, scene, or photographic context a strong shortcut for recovering the partition. The high performance of generic SigLIP features is consistent with this possibility, although it does not prove it. The manual audits establish that most reference groups share visible evidence; they do not establish that the model must identify the stated cue to recover those groups. In addition, the disjoint partition excludes overlapping cues and penalizes some valid alternative groupings. The authors acknowledge these limitations, but they still constrain how broadly the results demonstrate query-free visual cue discovery.
  2. The specific advantage of CueMem over a strong structured pipeline needs firmer evidence. Much of the headline gain comes from replacing flat autoregressive grouping with explicit support inference and programmatic serialization. The remaining improvement over supervised clustering is 0.030 PGF1, and the relation-loss contribution is 0.041. These are worthwhile gains, but the paper does not report confidence intervals or repeated-training variability for these comparisons. The supervised SigLIP baseline also adapts a projection over frozen global features, whereas CueMem adapts internal VLM representations jointly with generation. Consequently, the comparison does not isolate whether Q/K interactions are especially effective or whether broader task-specific adaptation accounts for the difference. A matched representation control would strengthen the methodological contribution.
Questions for authors
  1. Can you provide evidence that successful grouping depends on the annotated cue rather than source-event or scene similarity? For example, how do CueMem and supervised clustering compare on visually similar images from different reference groups, or on cue categories such as instance details versus broad lighting and scene cues?
  2. Are the gains over supervised clustering and the no-relation-loss variant consistent across training seeds? Please report paired uncertainty estimates over test collections and, if available, repeated-training results.
  3. Have you compared Q/K affinities with affinities derived from pooled hidden states of the same VLM, using comparable adaptation and the same support decoder? This would help establish the value of extracting relations specifically from Q/K interactions.
  4. How often do the generation budgets truncate outputs, particularly for the longest collections? Does the advantage over the flat and support-only baselines persist when all methods receive enough output tokens to enumerate their predicted groups? Please also clarify whether PGF1 uses every group inferred by AP or only groups that are emitted before generation stops.
2 comments
AuthorsOpenReview ↗

Cue Specificity, Statistical Reliability, and Matched Controls

We thank the reviewer for the constructive questions and recognition of the benchmark safeguards. We agree that the large gain over flat generation must be distinguished from the incremental gains within structured pipelines. Below, CBC denotes Cluster-based Captioning; Supervised CBC uses the adapted SigLIP projection.

Q1. Annotated cues versus source/scene shortcuts.

We provide two complementary analyses. First, cue-category support recovery uses a predicted support's IoU with its reference group, with recovery at IoU >= 0.5. It measures support recovery, not whether the generated description names the cue.

Cue categoryCueMemSupervised CBC
Same instance0.7710.570
Shape0.7480.539
Texture/material0.6610.472
Color0.7890.634
Scene0.7760.715
Shared object0.7550.631
Light/weather0.7960.699
Person0.8130.696

The gains are larger for instance (+0.201), shape (+0.209), and texture (+0.189) than for scene cues (+0.061), supporting fine-grained discrimination beyond broad scene similarity.

Second, we identify hard negatives as the top 5% of cross-reference-group image pairs ranked by frozen DINOv3 cosine similarity within each collection. Their merge rate—the fraction assigned to the same predicted group—is 0.0047 for CueMem, compared with 0.0098 for Supervised CBC and 0.0291 for flat Qwen3.5. Together, these analyses provide evidence that CueMem distinguishes cue-specific relations beyond broad visual similarity. We will incorporate both analyses into the experimental section of the revised paper.

Q2. Paired uncertainty and repeated training.

On 510 test collections, we compute paired per-collection differences, bootstrap 95% confidence intervals (CI), and two-sided paired Wilcoxon tests. All listed comparisons have p < 1e-16 for both metrics.

CueMem minusDelta PGF1 [95% CI]Delta CSA [95% CI]
Flat Qwen3.5+0.340 [0.317, 0.362]+0.377 [0.354, 0.399]
No relation supervision+0.038 [0.035, 0.042]+0.023 [0.021, 0.026]
CBC (SigLIP)+0.061 [0.055, 0.067]+0.063 [0.059, 0.067]
Supervised CBC+0.022 [0.016, 0.027]+0.059 [0.054, 0.063]
CBC (DINOv3)+0.096 [0.090, 0.101]+0.080 [0.076, 0.084]

Separately, three-training-seed summaries are:

MethodPGF1 mean +/- SDCSA mean +/- SD
CueMem0.7487 +/- 0.00150.7495 +/- 0.0014
No relation supervision0.7093 +/- 0.00300.7279 +/- 0.0009
Supervised CBC0.7214 +/- 0.00380.6959 +/- 0.0105

Across-seed mean gains are +0.0273 PGF1 over Supervised CBC and +0.0394 over the no-relation variant. The collection-level CIs quantify test-collection uncertainty, not training variability. They use rerun prediction sets, whereas the original tables report original runs; this accounts for differences from the original +0.030/+0.041 PGF1 gaps. We will report the run identities and distinguish these aggregation units in the revision.

Q3. Matched pooled-hidden-state control.

We replace Q/K patch interactions with cosine affinities from mean-pooled hidden states of the same VLM. We retain the LoRA adaptation, relation BCE, generation objective, and AP support decoder, changing the affinity source.

Affinity sourcePGF1CSA
Mean-pooled hidden states0.61840.6502
Q/K patch interactions0.7470.748

The +0.129 PGF1 gap supports Q/K patch interactions over this matched global-pooling control, rather than attributing the gain solely to broader VLM adaptation. We will include the control and its configuration.

Q4. Truncation, output budget, and PGF1.

CueMem reaches the generation budget on 6.1% of test collections, primarily those with at least 257 images, compared with 51.2% for flat Qwen3.5. Increasing the standard budget from 4096 to 8192 tokens yields only marginal changes for CueMem: PGF1 rises from 0.7472 to 0.7485, and CSA from 0.7498 to 0.7525. Flat Qwen3.5 remains at 0.4053 PGF1, with 50.6% of collections still reaching the budget, as the additional output largely repeats previously emitted groups. CueMem therefore retains its advantage under the larger budget.

PGF1 is computed from emitted supports, not the full set of groups inferred by AP. Images omitted from the output are assigned to singleton groups using the same repair procedure for all methods. Unreported AP supports receive no credit, and omitted within-reference-group relations count as false negatives. We will clarify the generation budgets and scoring procedure in the revised paper.

Dear Reviewer,

Thank you again for your constructive feedback. We have revised the manuscript in response to the comments from all reviewers. Additions and revised text are marked in red for ease of review. The main revisions include:

  1. Terminology: We replaced “long” with “large” when referring to image collections, retaining “long” only when discussing VLM input sequences or context length.
  2. Related work: We added references and discussion on set-level visual understanding and collection organization, clarifying their connections to and differences from our setting.
  3. Annotation and ethics: We clarified the annotation process, annotator compensation, bias-control safeguards, information collection, and the institutional determination regarding IRB requirements.
  4. Additional experiments: We expanded the appendix with cue-category support recovery and hard-negative analyses, paired statistical tests and three-seed results, a matched pooled-hidden-state control, output-budget sensitivity, and overlap-aware evaluation. These studies further assess cue-specific recovery, statistical reliability, the contribution of Q/K interactions, and robustness under output constraints and additional reference relations.

We would be grateful if you could review the revised manuscript when time permits. Please let us know if any questions or concerns remain; we would be happy to address them. If these revisions have satisfactorily addressed your concerns, we would appreciate your reconsideration of the paper’s overall assessment in light of the revised manuscript and additional evidence.

Best regards,
The Authors

Reviewer_qGpLOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose CueSel, a benchmark for analyzing the extraction of common visual cues in large (/long) image collections. They test several existing VLMs on this benchmark. In addition, they propose CueMem to adapt VLMs to the task introduced in the paper and reflected in the benchmark.

Critical strengths

The research question contributes to a field which is novel. The proposal of the benchmark is rigorous and interesting, with annotated collections of images. The definition of the task is clear. CueMem shows a clear contribution.

Critical weaknesses

The authors seem not to be aware of recent relevant literature that addresses the understanding of large collection of images. I will provide below 5 relevant examples, but I strongly believe that not contextualizing the work in the broader existing literature is a strong weakness. First of all, how does this paper advance compared to the work and tasks proposed here? And also, could some of this work be used as baselines in this paper? If not, why? If yes, maybe the proposed task can improve the performance or the overview on the tasks proposed in existing literature?

[1] Riccio, P., Galati, F., Schweighofer, K., Garcia, N., & Oliver, N. M. (2026, March). Imageset2text: Describing sets of images through text. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 40, No. 11, pp. 8731-8739).

[2] Dunlap, L., Zhang, Y., Wang, X., Zhong, R., Darrell, T., Steinhardt, J., ... & Yeung-Levy, S. (2024, June). Describing differences in image sets with natural language. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 24199-24208). IEEE.

[3] Deng, B., Peng, S., Genova, K., Wetzstein, G., Snavely, N., Guibas, L., & Funkhouser, T. (2025, October). Visual chronicles: Using multimodal llms to analyze massive collections of images. In 2025 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 12769-12778). IEEE.

[4] Liu, M., Zhong, Z., Li, J., Franchi, G., Roy, S., & Ricci, E. (2026). Organizing unstructured image collections using natural language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 8907-8918).

[5] Van Noord, N. (2023, October). Prototype-based dataset comparison. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV) (pp. 1944-1954). IEEE.

Questions for authors
  1. Is it possible to integrate the missing literature mentioned above, either with a better contextualization or with experiments (or explaining why this would not be feasible)
  2. Can the authors explain and justify the usage of the word "long"? If the collections are shuffled, wouldn't "large" be more appropriate?
  3. Who are the human annotators? Are they authors? If so, could this bias the annotation? If not, was IRB approval received? If not, why?
4 comments
AuthorsOpenReview ↗

Related Work, Terminology, and Annotation Ethics

We thank the reviewer for the positive assessment of our task formulation, benchmark rigor, and CueMem's contribution, as well as for the helpful pointers to related work.

Q1. Missing literature and additional comparisons.

We have reviewed all five suggested works. We will add a subsection, Set-level Visual Understanding and Collection Organization, distinguishing their input requirements and outputs from our setting: discovering image supports and describing their shared evidence within a single shuffled collection, without cue-specific queries.

Imageset2Text [1] produces collection-level text rather than per-image group assignments, so its native output does not support PGF1 or CSA. We therefore compare descriptions using GT-cue semantic matching, assessing agreement between generated descriptions and annotated cues. On 510 test collections, CueMem scores 0.847, versus 0.581 for Imageset2Text.

X-Cluster [4], from Organizing Unstructured Image Collections Using Natural Language, is a relevant training-free grouping baseline. We evaluate it on all test collections containing at most 50 images. For each collection, we select the clustering criterion with the highest reference PGF1 and report PGF1 and CSA for that same partition. This oracle selection deliberately favors X-Cluster; it is not an inference-time procedure.

MethodPGF1CSA
X-Cluster, oracle-best criterion0.62340.5772
CueMem0.91650.8540

We restrict this comparison to the small-collection subset because X-Cluster's image-wise LLM processing across multiple criteria incurs substantial API cost at CueSel's average test size of 238 images. We will disclose the subset and criterion-selection protocol, and will not extrapolate these results to the full test split.

The other works provide important context but do not directly produce the evaluated support-description output. VisDiff [2] describes differences between two supplied image sets; adapting it would require introducing a single-collection support-discovery procedure. Visual Chronicles [3] studies large-collection analysis using metadata and user queries, whereas CueSel supplies shuffled images without those inputs. ProtoSim [5] compares datasets through prototypes, rather than returning per-image cue supports with natural-language explanations. These interface differences motivate discussion rather than an unmodified PGF1/CSA comparison.

We will revise Related Work to position CueSel as a controlled diagnostic of joint support recovery and cue description, rather than implying that understanding or organizing image collections is unexplored.

Q2. Why use "long" for shuffled collections?

We agree that "large collections of shuffled images" is more precise and will adopt it in the title and collection-level discussion. We will retain "long" only when explicitly referring to the VLM input sequence or context length; it does not imply temporal order.

Q3. Annotators, bias, and IRB status.

The annotators were not the authors. They were engaged through our standard annotation process and compensated under the applicable employment/vendor arrangement. The task involved labeling visible evidence in publicly available VIST images, not studying the annotators themselves. We retained task annotations without collecting finer-grained personal, behavioral, or sensitive information. Our institutional ethics office determined that this activity does not constitute human-subjects research and therefore does not require IRB review; we do not claim that an IRB approval was issued.

To mitigate annotation bias, each source set was independently reviewed by at least three trained annotators. Sets with inconsistent judgments or support selections were excluded rather than resolved by majority vote, and the remaining annotations underwent a second quality-control pass. The initial candidate pool was approximately 1.3 times the size of the retained pool. We will document these safeguards and the annotator and ethics disclosures in Section 3.2 and the Ethics Statement of the revised paper.

Ref:

[1] Riccio et al. ImageSet2Text: Describing Sets of Images through Text. AAAI 2026. https://doi.org/10.1609/aaai.v40i11.37826

[2] Dunlap et al. Describing Differences in Image Sets with Natural Language. CVPR 2024. https://doi.org/10.1109/CVPR52733.2024.02284

[3] Deng et al. Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Images. ICCV 2025. https://doi.org/10.1109/ICCV51701.2025.01186

[4] Liu et al. Organizing Unstructured Image Collections using Natural Language. https://doi.org/10.48550/arXiv.2410.05217

[5] van Noord. Prototype-based Dataset Comparison. ICCV 2023. https://doi.org/10.1109/ICCV51070.2023.00186

Dear Reviewer,

Thank you again for your constructive feedback. We have revised the manuscript in response to the comments from all reviewers. Additions and revised text are marked in red for ease of review. The main revisions include:

  1. Terminology: We replaced “long” with “large” when referring to image collections, retaining “long” only when discussing VLM input sequences or context length.
  2. Related work: We added references and discussion on set-level visual understanding and collection organization, clarifying their connections to and differences from our setting.
  3. Annotation and ethics: We clarified the annotation process, annotator compensation, bias-control safeguards, information collection, and the institutional determination regarding IRB requirements.
  4. Additional experiments: We expanded the appendix with cue-category support recovery and hard-negative analyses, paired statistical tests and three-seed results, a matched pooled-hidden-state control, output-budget sensitivity, and overlap-aware evaluation. These studies further assess cue-specific recovery, statistical reliability, the contribution of Q/K interactions, and robustness under output constraints and additional reference relations.

We would be grateful if you could review the revised manuscript when time permits. Please let us know if any questions or concerns remain; we would be happy to address them. If these revisions have satisfactorily addressed your concerns, we would appreciate your reconsideration of the paper’s overall assessment in light of the revised manuscript and additional evidence.

Best regards,
The Authors

Reviewer_qGpLOpenReview ↗

Response to Authors

I thank the authors for engaging with my review and for providing the revised version of the manuscript. One thing that is not yet clear to me is whether the quantitative comparison with X-Cluster and ImageSet2Text provided in the response is currently missing from the paper (appendix, most likely) because they are deemed not relevant for the narrative of this work, or for other reasons. I have currently updated my score from weak reject to weak accept.

Thank you for the follow-up and for your reassessment of our work. We viewed these comparisons as complementary to the main experimental narrative: ImageSet2Text produces collection-level descriptions rather than image-level support assignments, while the X-Cluster comparison uses a restricted subset and oracle criterion selection. We therefore initially reported them as supplementary evidence in the response rather than incorporating them into the manuscript’s appendix.

Following your feedback, we agree that these comparisons provide useful context and should also be documented in the paper. We have now included both in Appendix D.17, “Comparisons with Collection-Level Baselines,” together with their evaluation protocols and scope limitations. The additions are marked in red in the updated manuscript.

We sincerely appreciate your careful reading and constructive suggestions throughout the review process. Your feedback has helped us clarify the positioning of our work and strengthen the empirical comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.