acceptodds
Withdrawn from ICLR 2027

A Plateau Is Not a Ceiling: Evaluating Headroom in Conversational Image Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A late-turn plateau in conversational image retrieval summarizes the outcome of an interaction, but does not establish which opportunities for improvement have been exhausted. Historical success, query construction, and target access affect that interpretation in different ways. We study retrieval headroom through three controlled comparisons: continuations without additional target observations, alternative representations of a fixed dialogue, and additional target-grounded observations. On FS-COCO-Dialog, reordering a frozen top-100 shortlist after four turns raises cumulative H@10 from 39.7 to 50.5 while current H@10 falls from 30.9 to 6.0. On DAR-preprocessed VisDial states, mean pooling raises current H@10 from 62.2 to 74.9 under ChatIR-FT and from 51.6 to 79.9 under BLIP-ITC, with a positive gain also on prefixes that are never truncated. Across broader fixed-dialogue comparisons, the preferred construction depends on the encoder and input regime. Additional evidence remains useful beyond the tested dialogue representations: under held-out interface selection, a paired human sketch improves all twelve dialogue-state–retrieval-space pairs by 9.8–22.7 points. On the same episodes, rewriting without target access matches or exceeds the sketch on concatenated queries in three of four spaces but adds little to an already reformulated state, an interaction that is significant in every retrieval space. These comparisons separately probe historical exposure, representation headroom, and evidence-access headroom, without assuming that they form independent causes of failure. The results support evaluating saturation relative to the states and observations tested, rather than inferring an information ceiling from a flattened retrieval curve. Code is available at https://anonymous.4open.science/r/iclr2027-DA65/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
conversational image retrieval, multimodal learning, representation learning, retrieval evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_TAtwOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper argues that a late-turn plateau in conversational image retrieval does not imply an information ceiling. The paper is diagnostic rather than methodological. Through controlled comparisons on FS-COCO-Dialog and VisDial, it separates three sources of remaining headroom for the retrieval task: historical exposure, query representation under fixed observations, and additional target-grounded evidence. The paper concludes that saturation should be assessed relative to tested states and observation regimes rather than inferred from a flattened retrieval curve.

Critical strengths
  1. Clear conceptual separation and systematic framing It cleanly distinguishes historical exposure, query representation, and evidence access, which are often conflated in the past conversational retrieval evaluation.

  2. Strong and carefully controlled experiments The paper uses matched cohorts, paired bootstrap intervals, multiple retrieval backbones, and held-out interface selection.

  3. Practical evaluation recommendations Based on the above experiments, the paper offers a practical check that could make future claims of interaction gains more reliable.

Critical weaknesses
  1. Limited novelty of the individual findings. Each of the three conclusions is close to what one would expect and some are already discussed in prior works (PlugIR and MAQIU). The paper's value therefore rests mainly on the broader framing and the rigor of the controls. In addition, the motivation would be strengthened by documenting more concretely how often plateaus in prior literature have been interpreted as information ceilings.

  2. Generated benchmark with uncertain answer fidelity. A substantial portion of the representation and sketch analysis relies on FS-COCO-Dialog, a generated benchmark whose answer fidelity is uncertain: the two automated judges agree on only 57.8% of sampled answers. This may complicate the interpretation of the measured headroom, since some gains could partly reflect robustness to noisy or erroneous turns. It would be helpful to discuss this limitation explicitly and to clarify the extent to which the VisDial results mitigate this concern.

Questions for authors
  1. Could the authors clarify whether the premise that plateaus are commonly interpreted as information ceilings reflects a documented consensus in prior work? Specific examples would help assess the novelty and strength of this motivation.

  2. Could the authors explicitly discuss the limitation of uncertain answer fidelity on FS-COCO-Dialog and clarify the extent to which the VisDial results mitigate this concern?

Reviewer_qXw5OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether late-turn performance plateaus in conversational image retrieval indicate exhausted retrieval opportunities. It compares historical and current retrieval success, alternative representations of fixed dialogue observations, and additional target-grounded evidence. Experiments on FS-COCO-Dialog and VisDial show that cumulative success can increase while current retrieval deteriorates, query construction can substantially change performance, and additional evidence can remain useful after dialogue reformulation. The central comparison shows that rewrite gains diminish after PlugIR-style reformulation, while human sketches retain substantial gains.

Critical strengths
  1. The matched comparison accounts for the strength of the dialogue representation. Comparing rewriting and sketch fusion on the same episodes before and after reformulation helps assess whether additional evidence remains useful beyond a weak text baseline. The interaction is consistently observed across four retrieval spaces.

  2. The central experiments include useful validation and failure analysis. Training-only interface selection, an untouched replication cohort, repeated rewrites, and wrong-target sketches strengthen the main comparison. Reporting both recovered and lost targets also makes the limitations of sketch fusion explicit.

Critical weaknesses
  1. The scientific contribution is too limited to justify acceptance. PlugIR already distinguishes current from cumulative retrieval and demonstrates the benefit of dialogue reformulation, while text-sketch complementarity has been studied in prior work, including Sangkloy et al. (2022). The main extension here is the matched finding that rewriting and sketch fusion provide comparable gains from weak dialogue representations, but their benefits diverge after reformulation. This is a useful controlled extension. However, I find that it provides limited additional understanding beyond the established importance of query representation and complementary observations. Even when evaluated as an empirical analysis paper, the advance is insufficiently significant to support acceptance.

  2. The robustness of the central sketch finding to dialogue errors remains uncertain. The main rewrite-sketch comparison relies on generated dialogues whose factual accuracy is not independently established. The automated fidelity analyses do not directly verify this comparison after reformulation, and the human VisDial experiments use privileged image-derived descriptions rather than sketches. This does not invalidate the gains on the recorded trajectories, but leaves uncertainty about their magnitude under accurate dialogue. This is a secondary concern about the generality of the empirical finding.

Questions for authors

Robustness to dialogue errors: Does the advantage of sketch fusion over another dialogue-only rewrite after reformulation persist on factually accurate dialogues identified through a manual audit of a random sample of the existing evaluation set? Please report the audited and retained sample sizes, the gains from both additions, and uncertainty estimates. A representative analysis using one retrieval backbone would help assess the generality of the central finding.

Reviewer_jxeVOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper argues that a plateau in conversational image retrieval does not necessarily indicate that all useful retrieval information has been exhausted. It studies three sources of remaining headroom: (i) historical exposure, where cumulative success can improve without a stronger current ranking; (ii) representation headroom, where the same observed dialogue can yield different retrieval states under different query constructions; and (iii) evidence-access headroom, where additional target-grounded observations may improve retrieval after dialogue representations have been strengthened.

Critical strengths
  1. The paper distinguishes historical credit, representation quality, and access to additional target evidence. These factors are often conflated when interpreting a flat retrieval curve, and the proposed controls make the distinctions concrete.

  2. For the representation experiments, the target, gallery, observed dialogue, and retrieval encoder are held fixed while query construction changes. This makes the comparison more interpretable than a comparison in which the system also receives different dialogue observations.

Critical weaknesses
  1. PlugIR already distinguishes current from cumulative retrieval and studies dialogue reformulation, while pooling, query rewriting, and sketch-based retrieval are established families of techniques. The paper's main contribution appears to be a unified experimental decomposition and a careful set of controlled comparisons. That can be valuable, but the paper should make clearer what generalizable evaluation principle or new scientific conclusion is introduced beyond organizing existing controls along three axes. At present, the framework risks being read as a taxonomy of familiar interventions rather than a sufficiently distinct methodological contribution.

  2. A sketch is created with access to the target image and may contain information absent from the dialogue, restate existing facts in a more retrieval-friendly form, or correct inaccurate dialogue answers. The paper appropriately acknowledges that it cannot fully distinguish these possibilities. Therefore, the sketch gains support the narrower claim that target-grounded evidence can improve retrieval under the tested setup. They do not directly quantify the benefit of an ordinary additional user turn or establish a practical acquisition policy. The difference in user effort and information access should be made explicit in the headline interpretation.

  3. The Qwen3-VL rewrite is instructed to include only visual facts stated in the dialogue. However, the paper does not provide a human evaluation of whether these rewrites preserve the dialogue's information, omit important details, or introduce unsupported attributes. Because the key comparison contrasts target-blind rewrite with target-grounded sketch, uncertainty about rewrite fidelity weakens the interpretation of the residual sketch advantage. A small, stratified human audit would make this comparison more convincing.

Questions for authors
  1. What is the central methodological contribution of the three-axis headroom framework beyond the current/cumulative distinction and query-reformulation analyses already studied in PlugIR and related work? Can the authors state a reusable protocol or principle that would transfer to other retrieval settings?

  2. Can the authors evaluate target-blind rewrite fidelity, including unsupported additions and omissions, using human judgments or a carefully validated protocol?

  3. Can the authors further separate genuinely new target information from re-expression or correction of existing dialogue information in the sketch comparison? If this is not possible, can the claims be consistently limited to privileged target-access headroom?

Reviewer_sZrNOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, the authors try to figure out the core bottleneck of the plateau in conversational image retrieval performance. For this, the authors analyze whether the bottleneck is caused by a lack of information or by how the conversation is represented.

Critical strengths
  1. The authors evaluate previous and current retrieval performance separately.
  2. The authors analyze how retrieval performance changes when the same conversation is represented in different ways. The authors also examine whether adding new information can further improve performance.
Critical weaknesses
  1. I think the main findings of this work have already been reported in [1]. Although the authors cite [1], I still do not clearly understand how the findings of this work differ with [1]. In particular, [1] has already shown that cumulative success rates can increase without actual improvements in current retrieval performance. It has also analyzed how reformulation of conversation affects retrieval performance. Therefore, I am not sure what the main novelty of this work is.
  2. The authors evaluate conversations with up to 20 turns (Section 5.3). However, I wonder whether the same findings would hold for more longer conversations.
  3. I think the authors should further analyze whether the performance improvements arise from new information or from representing the same information. I think more controlled experiments could help clarify this.

[1] Saehyung Lee, Sangwon Yu, Junsung Park, Jihun Yi, and Sungroh Yoon. Interactive text-to-image retrieval with large language models: A plug-and-play approach. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pp. 791–809, 2024.

Questions for authors
  1. The authors repeat the same information throughout the paper. The writing quality could be improved by reducing this repetition.
  2. The authors can mention somewhere in the paper that H@10 means Hit@10.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.