acceptodds
Withdrawn from ICLR 2027

Beyond Resolution: Object-to-Image Ratio Mismatch in Instance Retrieval

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Visual instance retrieval often fails when the same object appears at different apparent sizes in the query and gallery. We show that the dominant cause is usually not resolution loss but object-to-image (O2I) ratio mismatch: the object occupies different fractions of the two images. On a controlled benchmark of 3,021 Objaverse objects rendered at five camera distances, more than 80% of the cross-distance degradation is attributable to O2I mismatch rather than resolution for 9 of 12 pretrained backbones; multi-scale architectures cut the resolution-only effect to single digits yet remain equally susceptible. The failure is also asymmetric: tight queries retrieve more reliably against wide gallery images than the reverse. Guided by this analysis, query-side scale augmentation and an OWLv2 crop reranker reach state of the art on ILIAS 100M (29.2 mAP@1000 before reranking, 42.0 after) without training or modifying the precomputed gallery index, and a LoRA fine-tune matches the query-side gains at a single forward pass, showing that O2I robustness is learnable.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
instance retrieval, image retrieval, global descriptors, object-to-image ratio, scale mismatch, vision foundation models, test-time augmentation, reranking, representation analysis

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2FfTOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper investigates object-to-image (O2I) ratio mismatch in visual instance retrieval. Using controlled renderings of 3,021 Objaverse objects at five camera distances, it argues that differences in the fraction of the image occupied by an object explain much of the retrieval degradation commonly attributed to resolution loss. It also reports directional asymmetry between tight-to-wide and wide-to-tight retrieval. Motivated by these observations, the authors evaluate query-side scale averaging, attention-masked augmentation, detector-based crop reranking, and LoRA adaptation. On ILIAS 100M, masked augmentation improves mAP@1000 from 26.5 to 29.2, while crop reranking achieves 42.0.

Critical strengths
  1. A useful diagnostic question. Distinguishing lost visual detail from sensitivity to object framing is relevant to understanding global retrieval representations. The controlled rendering setup and fixed-pixel canvas experiment provide complementary evidence that resolution loss alone cannot explain the observed behavior.

  2. Segnificant experimental coverage. The study evaluates twelve backbones and includes augmentation controls, query-expansion comparisons, O2I-stratified results, and additional experiments on Oxford and Paris. These analyses go beyond simply demonstrating that cropping improves retrieval.

  3. Practical improvements. Query-side augmentation improves full-scale ILIAS retrieval while preserving the existing gallery index. The LoRA experiment also suggests that some of these gains can be obtained with a single inference pass, although it requires adapting the gallery representation.

Critical weaknesses
  1. The quantitative causal attribution is stronger than the experimental controls support. Section 4.2 interprets the improvement after cropping and resizing as the contribution of O2I mismatch, and the remaining error as resolution-only failure. However, cropping also changes spatial sampling, object coverage across tokens, and the subsequent resizing pipeline. The rendering setup additionally retains perspective differences, as acknowledged in Section 3. Thus, the experiment demonstrates that canonical cropping recovers substantial performance, but does not uniquely establish a decomposition of errors into O2I and resolution components. A factorial experiment varying object detail and frame occupancy independently would provide stronger support for the headline attribution.

Similarly, Appendix L.2 compares attention masking with oracle cropping. The difference includes changes in object sampling and spatial layout, so interpreting it as a lower bound attributable specifically to O2I requires additional justification. Moreover, this experiment uses ground-truth boxes and the ILIAS core gallery; it does not directly decompose the actual OWLv2 reranking improvement on ILIAS 100M.

  1. Limited methodological novelty and insufficient direct comparison with closely related methods. Scale averaging, regional descriptors, and crop-based maximum-similarity matching are established retrieval techniques. Patchify, including its detector-proposal variant, is particularly close to the proposed reranker. The paper acknowledges this relationship but does not compare these methods under a shared backbone, shortlist, and computational budget. The comparison against a PE-L + AMES leaderboard result changes multiple components simultaneously and therefore does not isolate the advantage of the proposed approach. The paper’s contribution must consequently rest primarily on the diagnostic analysis, making the attribution concerns particularly consequential.

  2. The fixed-pixel evidence is narrower than the broader claims. Section 4.3 provides a useful control, but evaluates only DINOv3-L and reports cosine-similarity drift rather than retrieval accuracy. A reduction in self-similarity does not necessarily imply worse rankings if similarities to other instances also change. Reporting retrieval metrics under this control, and extending it to representative additional backbones, would better connect the mechanism to the claimed retrieval failure.

  3. The asymmetry is observed but not sufficiently explained. Cosine similarity between a query and its positive is symmetric. Directional differences in retrieval therefore depend on how the query compares with competing gallery instances. The results establish asymmetric rankings under the chosen protocol, but the interpretation as a directional property of object-support encoding needs analysis of positive and negative similarity distributions, rather than aggregate MRR alone.

  4. The practical efficiency comparison remains incomplete. GP and MSA require five and three query encodings, respectively. The reranker uses approximately 31 crop encodings per candidate image, in addition to detection. Caching can amortize this work, but does not establish practical latency or storage requirements for previously unseen candidates. End-to-end latency, cache assumptions, storage, and accuracy–cost comparisons are needed. Appendix M also shows that LoRA requires re-encoding the gallery; its single-pass inference advantage should therefore be distinguished from the index-preserving benefit of query-side augmentation.

  5. Related work could better distinguish O2I mismatch from task-dependent resolution requirements. CARES (Context-Aware Resolution Selector for VLMs) studies selecting the minimal sufficient resolution for a given image-question pair. Although it addresses VQA rather than instance retrieval, discussing this work would help clarify the distinction between preserving task-relevant visual detail and matching object-to-image ratios. Using such aproach could work in O2I as well, with classifier training and I think that without implementing it, the authors failed to mention it.

Questions for authors

I acknowledge I might not be an expert but please adress the weaknesses i've stated.

Reviewer_iD1zOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies a critical problem in image retrieval: when one object have difference size in different images, it is hard to be retrieved accurately. The authors first argue that the issue is because of the mismatch in O2I ratio instead of resolution loss. They construct a small benchmark to prove the statement. They also benchmark some pretrained backbone models to show the similar capacity trend.

Based on this observation, the authors proposed some solutions to mitigate the issue. First is query-side scale augmentation, second is masked scale augmentation, and their is a object crop reranking. A evaluation on about 1200 queries and 100M image gallery searching validate the solution effectiveness.

Critical strengths
  1. The paper proposed a problem that is important to information retrieval. The O2I 3D rendering also helps isolate the scale problem from pixel loss and resolution reduction.
  2. Based on the observation, the paper proposes some practical solution.
  3. Beyond the synthetic data, the paper studies the real dataset and show improvement from the practical solutions.
Critical weaknesses
  1. The scale mismatch is a well-known problem in information retrieval. The scale and view mismatch are typical problems in real applications like street to shop retrieval. I did not see this is a newly discovered problem.
  2. It is also well-known that the pretrained backbone models have scale and view mismatch problem. It is because encoding an image into a global embedding will average the information and mismatch with the close view. There are varying solutions including pre-processing, fine-tuning with conditional embedding, disentanglement, etc. This paper does not compare with any of these solutions.
  3. The main solutions proposed in this paper are based on resizing images, averaging embeddings from different scales, masking padded regions, and comparing object crops. These are reasonable techniques, but the ideas are already familiar in image retrieval and computer vision.
  4. The evaluation and ablation study are limited. only about 1200 queries are studied in ILIAS 100M. And there is no ablation study on whether the method is effective on different backbones.
Questions for authors
  1. Prior works have investigated the scale-invariant representations, object-centric matching, local feature retrieval, etc. what is the fundamental novelty of this paper beyond prior works?
  2. Pretrained backbone models are not built for specific downstream tasks. instead of comparing with pretrained backbones, did the authors tried fine-tuned models or to fine-tune the backbones with prior techniques that address the scale mismatch?
  3. In Table 13, the cost of query side process (GP, MSA) are 5 times and 3 times. the cost of candidate side (reranking) is 32 times. so overall on both side the cost is 3x5x32 if applying all three techniques? Is this cost reasonable? Correct me if I am wrong.
Reviewer_DG7XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on object-to-image(O2I) ratio mismatch in visual instance retrieval. The authors construct a benchmark using 3021 Objaverse objects and evaluate 12 pretrained backbones. The authors find that O2I mismatch has a larger impact on retrieval performance than resolution loss. Based on this observation, the authors propose two query-side interventions(GP and MSA), and an OWLv2-based crop reranker. On ILIAS 100M benchmark, GP and MSA achieve 1.8 and 2.7 points absolute gain on mAP@1000 respectively, while the OWLv2-based reranker outperforms the previous best by 2.6 points. They also explore LoRA finetuning to improve O2I robustness, achieving 29.7 mAP@1000 on ILIAS 100M.

Critical strengths
  1. The authors construct a controlled benchmark and evaluate 12 different backbones. The results provide useful insights into how O2I mismatch affects retrieval performance across different models.
  2. The authors proposed several methods to improve O2I robustness, including query-side interventions(GP and MSA), an OWLv2-based crop reranker, and LoRA fine-tuning. These methods are relatively simple, and the results on ILIAS 100M show that they are effective.
Critical weaknesses
  1. The authors claim that 80% of the retrieval degradation is from O2I mismatch, instead of resolution loss. However, the method in Section 4.2 does not consider O2I mismatch and resolution loss interaction. In Section 4.3 where object pixels are fixed and O2I changes, only DINOv3-L is evaluated, and retrieval performance is not reported.
Questions for authors
  1. Can the authors further extend the experiment in Section 4.3 to more backbones and report the retrieval metrics?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.