acceptodds
Withdrawn from ICLR 2027

Image Retargeting: A Dataset and Metrics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Image retargeting adaptively resizes images for different display formats while preserving important content and perceptual quality. However, its development and evaluation remain limited by the lack of a comprehensive dataset with annotations and task-specific metrics. To address this gap, we construct a benchmark containing 5,000 input images, 20,000 instance-level annotations, and 20,000 pixel-level reference outputs. The dataset covers diverse input aspect ratios, instance counts, foreground layouts, and retargeting-sensitive content. To enable comprehensive and diagnostic evaluation, we further introduce a protocol comprising four scores. Semantic Faithfulness (SMF) and Layout Faithfulness (LOF) use the instance-level annotations to evaluate foreground preservation and reference-layout fidelity, while Scene and Texture Consistency (STC) and Structural and Visual Fidelity (SVF) assess aspects of visual quality not captured by these annotations. To the best of our knowledge, this is the first comprehensive image-retargeting benchmark to jointly provide instance-level annotations, pixel-level reference outputs, and annotation-driven evaluation metrics. Experiments on seven representative methods demonstrate that the proposed metrics agree more strongly with human judgments than existing evaluation measures. Our dataset and code will be made publicly available to the community.

Primary area:
datasets and benchmarks
Keywords:
image retargeting, dataset, metrics

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YzmsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper constructs an image retargeting benchmark, which covers diverse input aspect ratios, instance counts, foreground layouts, and retargeting-sensitive content. The paper further introduces an evaluation protocol comprising four scores, including Semantic Faithfulness (SMF) and Layout Faithfulness (LOF), Scene and Texture Consistency (STC) and Structural and Visual Fidelity (SVF).

Critical strengths

The construction of 5,000 input images with 20,000 instance-level annotations (across 4 target aspect ratios) and 20,000 pixel-level reference outputs is a substantial engineering effort. The human-in-the-loop annotation pipeline (bounding, refinement, quality check) is thoughtfully designed and the matching accuracy (93–95%) is reported transparently. Evaluating seven representative methods (from classical cropping/scaling/seam carving to modern generative approaches like PruneRepaint, InGAN, GPDM, GPNN) across four aspect ratios provides useful empirical insights into the trade-offs between content preservation, layout fidelity, and visual quality.

Critical weaknesses

The pixel-level reference outputs are generated using Gemini 2.5 Flash, and the STC/SVF scores are also computed using Gemini. The reference outputs inherit any biases or systematic errors of the model. The reproducibility is compromised since results depend on a proprietary API that may change over time. The paper does not analyze how alternative generative models (e.g., other vision-language models) would produce different references or scores.

Questions for authors

Can the authors provide statistics on scene types, object categories, complexity levels, and demographic diversity (if human faces are present)? This is important for assessing the benchmark's coverage. Since the matching algorithm is not perfect, could erroneous matches artificially inflate or deflate metric scores for certain methods? Has the authors conducted any error analysis on the matching pipeline?

Reviewer_jmQbOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces a foreground-centric image-retargeting resource with 5,000 source images, 20,000 source/target-ratio instance-annotation sets and 20,000 Gemini-generated, manually selected reference images. An annotation pipeline initializes mask groups and target layouts, followed by human adjustments. The protocol combines saliency-weighted aligned-mask IoU (SMF), reference-layout centroid distance (LOF), and Gemini 2.5 Pro scores for scene/texture consistency (STC) and structural/visual fidelity (SVF). Seven methods are benchmarked on 200 images at four ratios; eight evaluators rate 50 images’ method outputs, giving 350 outputs per ratio for correlation analysis. Table 2 reports output-level agreement with criterion-specific MOS; pixel-reference training utility is explicitly unevaluated.

Critical strengths

S1:The annotation resource addresses a real reusable-evaluation gap. A 5,000-image, four-target-ratio collection with explicit instance and layout annotations is materially more useful than rankings tied to a few fixed outputs. Three annotators inspect initialized annotations and 35.3% require adjustments (Sec. 3.2), suggesting substantive curation rather than an unchecked automatic dump. S2:The evaluation is diagnostic and uses output-level human correlations. SMF/LOF distinguish foreground integrity from reference layout, complemented by visual-plausibility axes. LOF SRCC 0.698–0.800 and SMF 0.702–0.827 across ratios support criterion-specific agreement on the tested outputs. Table 3 shows different methods lead different axes, which is useful evidence against collapsing all retargeting quality into one number.

Critical weaknesses

W1:The reusable metric specification is incomplete in cases central to content loss. SMF/LOF are written over matched pairs, but the paper does not fully specify unmatched-instance penalties, saliency-weight renormalization, match thresholds, zero center-vector handling or a resolution-invariant LOF normalization. If weights are renormalized over survivors, losing an object could improve LOF; if they are not, the unmatched contribution still needs definition. Mask shape alone also cannot guarantee semantic identity. Reported 93–95% matching accuracy lacks denominators and method-stratified extractor failure rates. These are substantive reproducibility/validity gaps for the proposed benchmark. W2:Human agreement is promising but not yet a sufficiently calibrated reusable validation. Correlations pool seven methods for only 50 source images per ratio, so between-method differences may dominate within-method quality discrimination; no source-clustered intervals, within-method/leave-one-method-out analyses or rater agreement are reported. STC/SVF use an unspecified proprietary VLM scoring protocol without a reproducible prompt/model-version/repetition contract or comparator for those two axes. LOF targets one preferred layout despite multiple valid compositions. Thus the current evidence supports agreement on this output pool more securely than general-purpose benchmark reliability.

Questions for authors

Q1:Please give executable SMF/LOF definitions for missing, unmatched or extra instances, including normalization and edge cases. How were the three reported matching accuracies validated, and how are extractor misses distinguished from genuine object loss? A small controlled deletion/deformation/relocation test would help validate the intended responses. Q2:Using existing human ratings, can you report image-clustered uncertainty, rater reliability and within-method or leave-one-method-out correlations? Please include per-image ranking/pairwise agreement and state what raters could see, to determine whether agreement extends beyond pooled method separation. Q3:Please provide exact STC/SVF prompts, scoring anchors, model version/date, settings, parsing/failure rules and cost. On a small fixed output set, can repeated calls or one compatible evaluator comparison establish scoring stability and human agreement? Q4:When the pipeline expands the frame, how are target aspect ratio/resolution preserved and grouping, clipping or overlap conflicts resolved? Please report available annotation-agreement checks and explain how LOF treats valid alternative compositions. Q5:How were the 200 benchmark and 50 human-rating inputs sampled from 5,000, and are their distributions representative? Please specify Cropping/Scaling and per-method settings, plus Gemini-reference selection criteria/rejection rates and reference–annotation consistency checks. Q6:Please provide source/annotation/synthetic-image licenses, release/split manifests, and whether AIR-DR training includes any benchmark evaluation images. What consent or approval assessment applies to annotation and human rating?

Reviewer_XjEBOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investgates image retargeting. It introduces 5,000 source images, 20,000 target-layout annotations, and 20,000 generated reference outputs across four aspect ratios. It proposes two mask-based scores and two VLM-based scores, then evaluates seven retargeting methods and compares the scores with human judgments.

Critical strengths
  1. This dataset provides target-specific instance layouts provide information beyond image-level preference labels.
  2. This paper separates foreground preservation, layout agreement, and visual quality, which makes trade-offs easier to interpret.
  3. The manually adjusted annotations are more trustworthy.
Critical weaknesses
  1. Table 1 compares only with three datasets containing fewer than 100 inputs. The review on previous works is clearly not comprehensive enough.
  2. Section 4.3 computes SMF from mask overlap after translation and isotropic scaling. This measures foreground shape preservation, but cannot directly establish preservation of identity, appearance, text, or fine semantic details.
  3. The criteria for unmatched reference objects, hallucinated or duplicated objects, empty detections, and the criterion for a “valid” match are not specified.
  4. The topic seems to be off topic for ICLR.
Questions for authors

Please see the weakness section above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.