acceptodds
Under review as a conference paper at ICLR 2027

METICULOUS MAGNIFIER: EVALUATING FINE- GRAINED SUBJECT-DETAIL PRESERVATION IN SUBJECT-DRIVEN GENERATION

Abstract

Subject-driven generation (SDG) achieves strong high-level semantic consistency, yet preserving and evaluating localized details such as text and logos remains challenging. We study this gap by introducing DreamBench-Detail a benchmark of detail-rich product subjects, and a controlled protocol that induces ordered levels of localized detail degradation while holding generation settings fixed. Under this protocol, embedding-based metrics show weak score separation, while holistic MLLM evaluators assign tied scores to approximately 75% of comparisons. When contextual variation is introduced, the evaluated metrics either remain weak or lose ranking accuracy. To address these limitations, we propose Meticulous Magnifier, which identifies reference-specific salient targets, localizes them in both images, and aggregates rubric-guided crop-level judgments. Across multiple distortion levels and SDG models, Meticulous Magnifier achieves 78.54% of Pairwise Accuracy with a 6.69% tie rate. It retains 75.33% accuracy under additional contextual variation, substantially outperforming the evaluated baselines.

Primary area:
datasets and benchmarks
Keywords:
Subject driven generation, image generation, image evaluation, mllm-as-a-judge

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_guT2OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies fine-grained subject fidelity evaluation in subject-driven image generation, focusing on text, patterns, and local structure that holistic similarity metrics may overlook. The authors introduce DreamBench-Detail, comprising 439 subjects across 25 product categories and 643 reference–prompt pairs. A controlled protocol generates outputs from locally degraded references to assess metric sensitivity to an assumed degradation ordering. The proposed Meticulous Magnifier identifies multiscale reference-specific targets, localizes and crops corresponding regions, and aggregates five-level MLLM judgments. Validation includes baseline comparisons across three generation models, a context-variation test, component and evaluator ablations, and human inspection of bounding boxes. The method achieves 78.54% intervention-consistent pairwise accuracy in the main controlled setting and 75.33% under context variation, outperforming the evaluated baselines.

Critical strengths
  1. The paper addresses a practically meaningful evaluation gap. Overall subject similarity does not guarantee faithful preservation of text, patterns, or local structure, and this distinction is particularly consequential for applications such as product presentation. By focusing on subjects with rich instance-specific details, DreamBench-Detail encourages evaluation beyond broad semantic consistency.

  2. The evaluation pipeline is clearly structured and supported by promising initial results. Decomposing a reference into reusable targets, followed by localization, cropping, and local assessment, makes the evaluation criteria more explicit and facilitates inspection of individual errors. Under the proposed controlled protocol, the method improves over the strongest baseline, with similar trends observed across different generation models. These findings support the potential value of the approach.

Critical weaknesses
  1. The representativeness of the controlled degradation protocol for real generation errors remains unestablished. Naturally occurring errors in subject-driven generation include localized color loss in one element of a repeated pattern, omission of individual components, inappropriate deformation, and changes in spatial relationships. Such errors can occur while the overall subject appearance remains largely consistent, as also illustrated by the local pattern-preservation issue in Figure 4. In contrast, the evaluation conditions are primarily constructed by reducing reference-image resolution and applying localized perturbations. The paper does not establish that these interventions adequately cover naturally occurring generation errors. Consequently, the experiments primarily demonstrate sensitivity to a particular degradation protocol, while it remains unclear whether the observed gains transfer to evaluating real-world subject-driven image generation. This limits how strongly the central experiments support the method’s practical utility.

  2. The broad claims about subject-detail evaluation lack error-type-specific evidence and targeted baselines. The degradation setup and examples prominently feature changes to text and other high-frequency details, but the results do not sufficiently distinguish performance on textual and non-textual errors. It therefore remains possible that much of the aggregate improvement comes from assessing text fidelity. OCR-based character accuracy or edit distance would be important targeted baselines for textual content, while typography, color, patterns, and deformation require corresponding validation. Without these comparisons, the method’s incremental value over existing evaluation approaches and its scope of applicability remain insufficiently established. The breadth of the application claims exceeds the evidence currently provided.

Questions for authors
  1. Is there independent human-annotated evidence validating the metric on outputs generated normally from clean references, with errors arising naturally during generation? Additionally, within the existing controlled data, what fractions of human comparisons agree with, tie, or reverse the assumed None–Mild–Strong ordering, particularly across different prompts? Reference degradation severity alone does not guarantee the corresponding quality ordering for every pair of generated outputs.

  2. What are the main error types and their frequencies in the evaluation data? Could the authors report performance separately for textual errors, local color or pattern errors, component omissions, and geometric deformation, and include character-accuracy or edit-distance baselines on the text subset, drawing on OCR-based evaluation protocols used in works such as TextDiffuser/MARIO-Eval and AnyText? Does the method retain a consistent advantage when textual targets are excluded? This OCR comparison concerns textual content fidelity; it does not assume that OCR replaces evaluation of typography, layout, or visual style.

  3. How are undetected targets scored, and do they remain in the denominator of the final average? How does the method distinguish generation-induced component omissions from invisibility caused by viewpoint or occlusion? Furthermore, when most of a crop is preserved but one element of a repeated pattern loses color or becomes deformed, can the local judge reliably detect that error? The existing bounding-box quality study does not directly establish these aspects of evaluation validity.

  4. Cropping improves accuracy over the version without localization by 3.84 percentage points, while Mixed improves over Fine-only by only 0.68 percentage points. Given the dependence among comparisons sharing subjects, prompts, and images, could the authors provide subject-clustered confidence intervals and, where necessary, repeated-run results? Do these results support the claim that all pipeline components are essential?

22 comments

We thank the reviewer for the detailed review and address the concerns and questions below.

W1. Representativeness of the Controlled Degradation Protocol

The representativeness of the controlled degradation protocol for real generation errors remains unestablished. Naturally occurring errors in subject-driven generation include localized color loss in one element of a repeated pattern, omission of individual components, inappropriate deformation, and changes in spatial relationships. Such errors can occur while the overall subject appearance remains largely consistent, as also illustrated by the local pattern-preservation issue in Figure 4. In contrast, the evaluation conditions are primarily constructed by reducing reference-image resolution and applying localized perturbations. The paper does not establish that these interventions adequately cover naturally occurring generation errors. Consequently, the experiments primarily demonstrate sensitivity to a particular degradation protocol, while it remains unclear whether the observed gains transfer to evaluating real-world subject-driven image generation. This limits how strongly the central experiments support the method’s practical utility.

We thank the reviewer for raising this important concern. We conducted three complementary analyses to assess whether MetMag's effectiveness extends beyond the controlled degradation protocol. First, our protocol produces diverse errors, including color/pattern changes (11.1%), component omissions (11.2%), and geometric deformations (21.8%). Second, on naturally generated SDG images, MetMag achieves 83.9% agreement with individual human judgments, compared with 68.1% for ID-Sim and 66.3% for Keypoints, while reproducing human-derived model rankings. Third, MetMag achieves stronger correlations with human Concept Preservation ratings on DreamBench++ than the evaluated baselines, suggesting applicability to broader subject-preservation differences. Together, these findings provide evidence of MetMag's practical utility beyond the specific controlled degradation setting.

1. Diversity of Errors Under the Controlled Degradation Protocol

Although our protocol is designed to isolate fine-grained detail degradation, the resulting errors are not limited to resolution loss. They also include localized color/pattern changes, component omissions, and geometric deformations, as highlighted by the reviewer.

To quantify these error types, we used GLM-5.3-Flash to categorize the visual discrepancy descriptions recorded during MetMag's crop-level evaluations into five categories: textual, color/pattern, omission, geometric, and other. We included evaluations rated Good, Fair, Poor, or Very Poor, yielding 18,223 descriptions.

TextualColor / PatternOmissionGeometricOther
55.3%11.1%11.2%21.8%0.5%

These results show that approximately 45% of the recorded discrepancies concern non-textual attributes, including the color/pattern errors, omissions, and geometric deformations identified by the reviewer. These frequencies are derived from the evaluator's discrepancy descriptions rather than independent error annotations and therefore do not represent the distribution of errors in naturally generated SDG images.

Thus, while our protocol does not aim to reproduce the full distribution of naturally occurring errors, it induces diverse failure modes beyond simple resolution degradation.

2. Human Validation of Detail Preservation on Naturally Generated Images

To directly evaluate MetMag on naturally occurring detail-preservation errors, we conducted an additional blind two-alternative forced-choice (2AFC) human study using SDG images generated from clean Amazon product references, without controlled degradation.

We sampled 100 image pairs, stratified across model pairings, from outputs of GPT-Image-2 [High], FLUX.1-Kontext, and Qwen-Image-Edit evaluated in Figure 5. Five annotators independently compared the fine-grained detail preservation of each pair, yielding 500 human preference votes.

2.1. Agreement with Individual Human Judgments

We first measured how accurately each metric's pairwise preferences agree with individual human judgments on naturally generated images.

MetricHuman Agreement (%)95% CI (%)
Keypoints66.3[57.9, 74.4]
ID-Sim68.1[59.7, 75.9]
MetMag (Ours)83.9[78.1, 89.0]

Agreement is calculated over individual decisive human judgments, excluding human and metric ties. Confidence intervals are estimated using image-pair-level bootstrap resampling.

MetMag achieves 83.9% agreement with human judgments, outperforming ID-Sim and Keypoints by 15.8 and 17.6 percentage points, respectively. This provides direct evidence that MetMag can distinguish naturally occurring differences in fine-grained subject-detail preservation, independently of the controlled degradation ordering.

2.2. Agreement with Human-Derived Model Rankings

We further examined whether MetMag's evaluations recover the relative detail-preservation performance of different SDG models.

ModelHuman Decisive Win Rate (%)MetMag Win Rate (%)Rank (Human / MetMag)
GPT-Image-2 [High]88.894.41 / 1
FLUX.1-Kontext47.843.22 / 2
Qwen-Image-Edit12.312.43 / 3

Human preferences followed the ordering GPT-Image-2 [High] > FLUX.1-Kontext > Qwen-Image-Edit, exactly matching MetMag's ranking. Moreover, the human and MetMag win rates differed by at most 5.6 percentage points across all three models. This ordering is also consistent with the MetMag-based leaderboard and Bradley–Terry Elo rankings in Figure 5.

Importantly, these images contain naturally occurring generation errors rather than artificially induced degradation. The agreement with independent human judgments at both the individual-image and model-ranking levels demonstrates that MetMag's detail-preservation assessments transfer beyond the controlled degradation protocol.

3. Complementary Evaluation of Broader Subject-Preservation Errors on DreamBench++

Beyond localized detail errors, naturally generated SDG images can also exhibit broader subject-preservation differences, such as changes in overall structure, appearance, and spatial relationships. Although these are not the primary focus of MetMag, its multi-scale Salient Target List includes coarser structural regions alongside fine-grained details, potentially allowing it to capture such discrepancies.

To examine this broader applicability, we evaluated MetMag against human Concept Preservation (CP) ratings on naturally generated images from DreamBench++, which assesses overall subject preservation.

3.1. Object Subjects

We evaluated 65 object subjects (585 images) across three SDG models and measured Pearson correlations with human CP ratings.

Generation ModelCLIP-IDINOv2ID-SimKeypointsMetMag (Ours)
DB-LoRA-SDXL0.5290.4520.6000.3320.644
IP-Adapter-Plus-ViT-H-SDXL0.3000.2700.2770.0560.388
IP-Adapter-ViT-G-SDXL0.3300.2570.3270.2130.394

MetMag achieves the highest Pearson correlation across all three generation models, indicating stronger alignment with human assessments of broader subject preservation than the evaluated embedding-based and keypoint-based metrics.

3.2. Human Subjects

We further evaluated 20 human subjects (180 images) generated with IP-Adapter-Plus-ViT-H-SDXL, including photorealistic humans, cartoon characters, and other stylized subjects.

MetricPearson rSpearman ρ
CLIP-I0.2760.315
DINOv2-0.118-0.007
MetMag (Ours)0.5510.461

MetMag achieves the strongest correlations with human CP ratings on this diverse human-subject subset, extending its applicability beyond detail-rich product subjects.

Together, these DreamBench++ results provide complementary evidence that MetMag's evaluation capability is not confined to fine-grained detail errors but can also reflect broader subject-preservation differences in naturally generated images. While these experiments do not independently isolate spatial-relationship errors, they demonstrate alignment with human judgments under broader subject-consistency criteria.

4. Summary

Our additional analyses address the reviewer's concern from three complementary perspectives:

  1. Error diversity: The controlled protocol produces textual, color/pattern, omission, and geometric discrepancies rather than only resolution-related degradation.
  2. Real-world detail preservation: On naturally generated SDG images, MetMag achieves 83.9% agreement with individual human judgments and reproduces human-derived model rankings, without relying on artificial degradation.
  3. Broader subject preservation: On DreamBench++, MetMag achieves the strongest correlations with human Concept Preservation ratings across the evaluated object and human settings.

These results support the practical applicability of MetMag to naturally generated SDG images, while acknowledging that the controlled protocol does not reproduce the full distribution of real-world generation errors.

We have incorporated these additional analyses into the revised manuscript and clarified the scope and limitations of our controlled degradation protocol.

Reviewer_XfTMOpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper investigates the evaluation of fine-grained subject-detail preservation in subject-driven image generation. It introduces DreamBench-Detail, a product-focused benchmark, together with a controlled reference-degradation protocol for testing metric sensitivity. The proposed Meticulous Magnifier generates reference-specific visual targets, localizes and crops corresponding regions in the reference and generated images, and aggregates rubric-based VLM judgments. Experiments demonstrate improvements over the evaluated baselines in ranking accuracy relative to the expected degradation order, score separation, and robustness to additional contextual variation.

Critical strengths
  1. The benchmark and controlled evaluation protocol are valuable. DreamBench-Detail contains 439 detail-rich subjects across 25 product categories. By controlling reference-detail degradation while fixing other generation settings, the protocol provides a systematic way to investigate metric sensitivity to detail loss.
  2. The framework is clear, with informative ablation studies. Decomposing evaluation into target selection, localization, and local judging provides a clear design rationale. The ablations help distinguish the contributions of per-target score aggregation and cropping, while the context-variation experiments provide additional evidence of robustness.
Critical weaknesses
  1. The advantage of VLM-based local judging over alternative local similarity measures is insufficiently established. The experiments do not fully distinguish whether the improvements primarily arise from selecting and cropping relevant regions or from the VLM judge itself. Comparing DINOv3 and DreamSim on the same corresponding crop pairs would help establish the benefit of VLM-based local evaluation. Reporting computational costs would also clarify the practical trade-off.
  2. The completeness and human relevance of the generated target list remain unvalidated. The human study in Appendix G mainly examines whether predicted boxes accurately cover specified parts. It does not establish whether the selected targets cover the details humans consider important. Important attributes omitted from the list may receive no explicit assessment. A small study comparing independently annotated human targets with VLM-generated targets, including coverage, importance, and omission patterns, would strengthen confidence in the metric.
Questions for authors
  1. Have the authors considered comparing VLM judgments with DINOv3 or DreamSim similarities computed on the same corresponding crop pairs? This would help separate the benefits of region selection and cropping from those of VLM-based judging. For a future extension to human subjects, could identity-specific metrics such as LVFace or ArcFace complement the evaluator?

  2. To what extent does the VLM-generated target list cover the details humans consider important? Appendix G validates localization accuracy for specified targets, but not the completeness of target selection. Could the authors conduct a small-scale study in which annotators independently identify key details, and report target coverage, important-detail omission rates, and representative omission cases?

  3. Have the authors considered using the proposed metric as a reward in a small-scale fine-tuning experiment with Flow-GRPO, DanceGRPO, or DiffusionNFT, compared against existing metrics used as rewards? Improvements on held-out subjects, confirmed through independent human judgments or text-fidelity measures while maintaining prompt adherence and overall image quality, would provide stronger evidence of practical utility. A small-scale experiment under just one of the optimization frameworks would suffice, rather than running all three."

8 comments

We thank the reviewer for the detailed review and address the concerns and questions below.

W1, Q1.1. Advantage of VLM-based Local Judging over Local Similarity Measures

The advantage of VLM-based local judging over alternative local similarity measures is insufficiently established. The experiments do not fully distinguish whether the improvements primarily arise from selecting and cropping relevant regions or from the VLM judge itself. Comparing DINOv3 and DreamSim on the same corresponding crop pairs would help establish the benefit of VLM-based local evaluation. Reporting computational costs would also clarify the practical trade-off.

Have the authors considered comparing VLM judgments with DINOv3 or DreamSim similarities computed on the same corresponding crop pairs? This would help separate the benefits of region selection and cropping from those of VLM-based judging.

We thank the reviewer for these suggestions. We conducted two additional experiments to isolate the contribution of VLM-based judging and quantify its computational overhead.

  1. VLM-based judging vs. crop-level similarity: Using identical corresponding crop pairs, MetMag achieves higher Pairwise Accuracy (83.12% vs. 75.44% for DINOv3 and 78.74% for DreamSim) and substantially higher AUCz (59.16 vs. 27.43 and 27.96). These results demonstrate that MetMag's advantage extends beyond region selection and cropping.
  2. Computational cost: MetMag requires 12.30 s/pair, approximately 1.2× the runtime of crop-level DINOv3 and DreamSim. Localization accounts for 79.4% of its runtime, while VLM-based crop judging accounts for only 20.2%.

We provide the detailed results below.

Q1.1. VLM-Based Local Judging vs. Local Similarity Measures

To isolate the contribution of VLM-based judging, we introduced two additional baselines, DINOv3 (bbox-crop) and DreamSim (bbox-crop). Both compute embedding similarities using the same corresponding crop pairs as MetMag, retaining the region selection and localization stages while replacing only the VLM-based judge.

We evaluated all methods on the same human-verified benchmark comprising 555 degradation sequences.

MetricPairwise Acc. (%)Tie Rate (%)AUCz
CLIP52.430.1210.93
DINOv260.420.008.55
DreamBench++12.0779.1612.07
VIEScore16.0474.2912.63
Keypoints72.551.3830.25
DINOv3 (bbox-crop)75.440.4827.43
DreamSim (bbox-crop)78.740.1227.96
MetMag (Ours)83.124.5059.16

Both crop-level embedding baselines achieve competitive Pairwise Accuracy, confirming that localized similarity evaluation is a meaningful alternative to holistic image comparison. However, MetMag outperforms both methods despite using identical corresponding crop pairs, achieving 4.38 percentage points higher accuracy than the stronger baseline, DreamSim.

More importantly, MetMag achieves substantially stronger normalized score separation, with an AUCz of 59.16 compared with 27.43 for DINOv3 and 27.96 for DreamSim. Since AUCz measures how well correct pairwise rankings are maintained as the required score margin increases relative to each metric's score standard deviation, these results indicate that MetMag provides more robust separation between degradation levels.

Together, these findings demonstrate that MetMag's improvements cannot be attributed solely to region selection and cropping. We attribute the additional benefit to rubric-guided VLM judgments, which explicitly assess the preservation of reference-specific visual details rather than relying solely on embedding similarity.

Q1.2. Computational Costs

We additionally measured the average evaluation runtime per reference–generated image pair across 643 pairs, using optimized inference implementations, including vLLM for MLLM-based methods. The one-time cost of Salient Target List generation is excluded, as these lists are precomputed and provided with the benchmark.

MethodTime (s/pair)
CLIP0.034
DINO0.032
DreamBench++0.108
VIEScore0.220
Keypoints3.229
DINOv3 + Crop10.341
DreamSim + Crop10.327
MetMag (Ours)12.298

MetMag requires 12.30 s/pair, approximately 1.2× the runtime of crop-level DINOv3 and DreamSim.

We further decomposed MetMag's runtime into individual components:

MetMag ComponentTime (s/pair)Share (%)
Reference detection4.13833.7
Generated-image detection5.62345.7
Crop-level judging2.48420.2
Engine switching0.0520.4
Total12.298100.0

Localization accounts for 79.4% of MetMag's runtime, while VLM-based crop judging accounts for only 20.2%. Thus, the primary computational bottleneck is localization rather than the VLM judge itself.

Overall, MetMag achieves higher Pairwise Accuracy and substantially stronger normalized score separation than crop-level embedding baselines, at approximately 19% additional runtime. These results establish the benefit of VLM-based local judging while identifying localization as the main opportunity for future efficiency improvements.

W2, Q2. Completeness and Human Relevance of the Generated Target List

The completeness and human relevance of the generated target list remain unvalidated. The human study in Appendix G mainly examines whether predicted boxes accurately cover specified parts. It does not establish whether the selected targets cover the details humans consider important. Important attributes omitted from the list may receive no explicit assessment. A small study comparing independently annotated human targets with VLM-generated targets, including coverage, importance, and omission patterns, would strengthen confidence in the metric.

To what extent does the VLM-generated target list cover the details humans consider important? Appendix G validates localization accuracy for specified targets, but not the completeness of target selection. Could the authors conduct a small-scale study in which annotators independently identify key details, and report target coverage, important-detail omission rates, and representative omission cases?

Thank you for this suggestion. We conducted the requested study on 50 subjects, where human annotators independently identified the attributes they considered most important for each subject. We then compared these annotations with the VLM-generated Salient Target Lists.

CriterionOKOmissionHallucination
Human-annotated attributes (317 items)313 (98.7%)4 (1.3%)–
VLM-generated labels (706 labels)701 (99.3%)–5 (0.7%)

Coverage of important attributes. The VLM-generated lists cover 98.7% of human-annotated important attributes, demonstrating high agreement with human-selected evaluation targets.

Omission patterns. The four omitted attributes comprise two small product logos, a small subtitle beneath a product name, and a handle. Importantly, we verified that all four omitted attributes fall within broader coarse-grained targets. Thus, although these attributes are not explicitly selected as individual targets, their corresponding regions remain included in MetMag's evaluation. However, this does not guarantee sensitivity to discrepancies in the omitted attributes, and dedicated targets could provide more reliable fine-grained assessment.

Hallucination. We additionally examined the validity of VLM-generated labels. Only 0.7% refer to attributes absent from the reference image or contain typographical errors.

Overall, these results demonstrate that the target-generation stage captures nearly all human-identified important attributes while introducing very few invalid targets, complementing the bounding-box localization validation already presented in Appendix G.

Reviewer_udAYOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DreamBench-Detail, a benchmark for evaluating fine-grained subject-detail preservation in subject-driven image generation. It proposes a controlled degradation protocol to assess the sensitivity of existing evaluation metrics and identifies limitations in embedding-based and holistic MLLM-based approaches. To address these limitations, the authors propose Meticulous Magnifier, which identifies salient visual details, localizes corresponding regions, and aggregates crop-level MLLM judgments. Experiments across multiple generation models demonstrate improved pairwise ranking accuracy and robustness to contextual variations, supported by ablation studies and model comparisons.

Critical strengths
  1. Clear motivation and valuable benchmark. The paper addresses an important limitation of existing subject-driven generation metrics in capturing fine-grained visual details. DreamBench-Detail provides a targeted benchmark and controlled evaluation protocol for systematically studying this issue.

  2. Promising empirical results. The proposed method achieves improved pairwise ranking accuracy and robustness to contextual variations. The ablation studies also support the effectiveness of structured, localized evaluation, although stronger validation against human judgments is still needed.

Critical weaknesses

W1. Limited methodological novelty. The proposed framework mainly combines existing techniques, including MLLM-based attribute identification, open-vocabulary grounding, crop-level evaluation, and score aggregation. The methodological novelty beyond structured evaluation prompting remains limited. W2. Insufficient validation against human judgments. The evaluation assumes that stronger reference degradation leads to worse detail preservation in generated images. However, this ordering may not always hold due to generative model behavior. Direct comparisons with human judgments on naturally generated images are needed to establish the validity of the proposed metric. W3. Incomplete comparisons with recent methods. The experiments omit recent relevant evaluation methods, particularly ID-Sim (CVPR 2026), which also addresses fine-grained identity similarity in personalized image generation. ID-Sim: An Identity-Focused Similarity Metric W4. Limited justification of score aggregation. The method assigns equal weights to all identified details, although different attributes may contribute differently to subject identity. The effects of overlapping targets, irrelevant details, and localization errors on the final score remain insufficiently analyzed. W5. Limited contribution of multi-scale evaluation. Table 4 shows only a marginal improvement in Pairwise Accuracy from combining coarse and fine targets compared with fine-only evaluation, while AUCz slightly decreases. The necessity of the multi-scale design requires stronger justification.

Stronger human evaluation and evidence of superiority over 2026 SOTA under naturally occurring generation errors could potentially raise my recommendation

Questions for authors

refer to the weakness

10 comments

We thank the reviewer for the detailed review and address the five concerns below.

W1: Methodological Novelty

Limited methodological novelty. The proposed framework mainly combines existing techniques, including MLLM-based attribute identification, open-vocabulary grounding, crop-level evaluation, and score aggregation. The methodological novelty beyond structured evaluation prompting remains limited.

Our primary contribution is the systematic identification and characterization of an underexplored limitation of existing subject-driven generation (SDG) evaluation metrics: their limited sensitivity to fine-grained differences in subject details. To investigate this limitation, we introduce DreamBench-Detail and a controlled evaluation protocol that enable systematic measurement of metric sensitivity to localized detail degradation. We thank the reviewer for the opportunity to clarify these contributions.

To address this limitation, we propose Meticulous Magnifier (MetMag), a framework that enables fine-grained evaluation of subject-detail preservation. While its individual components are established techniques, our contribution lies in enabling systematic, reference-specific evaluation of individual subject details. In this framework, salient details are explicitly identified, spatially grounded, and independently evaluated before aggregation.

Our ablation study (Table 3) demonstrates the effectiveness of this approach: reference-specific per-target assessment improves pairwise accuracy from 12.07–16.04% to 78.98%, even without localization. Adding crop-based evaluation further increases accuracy to 83.12%, demonstrating the additional benefit of localization.

Together, these results establish our contributions: (1) identifying and characterizing an overlooked evaluation gap, (2) enabling its systematic investigation through a targeted benchmark and controlled protocol, and (3) providing an empirically validated framework for fine-grained subject-detail evaluation. We will revise the manuscript to articulate these contributions more clearly.

W2: Validation Against Human Judgments

W2. Insufficient validation against human judgments. The evaluation assumes that stronger reference degradation leads to worse detail preservation in generated images. However, this ordering may not always hold due to generative model behavior. Direct comparisons with human judgments on naturally generated images are needed to establish the validity of the proposed metric.

We conducted additional human studies to validate (1) the assumed degradation ordering in controlled experiments and (2) MetMag's alignment with human judgments on naturally generated images. On naturally generated outputs, MetMag achieves 83.9% agreement with individual human choices, compared with 68.1% for ID-Sim and 66.3% for Keypoints. Furthermore, its advantage over existing metrics persists when controlled evaluation is restricted to human-verified comparisons. We detail these complementary evaluations below.

1. Human Validation of the Controlled Degradation Ordering

1.1. Same-prompt comparisons (Table 1).

We examined whether the expected None > Mild > Strong ordering holds in generated outputs. Three annotators independently evaluated all 643 SDG sequences, each containing outputs generated from the three reference-degradation levels under identical generation settings. We aggregated their judgments by majority vote.

ClassificationCountPercentage
Agree55586.3%
Tie exists457.0%
Reversed121.9%
Strange images132.0%
No consensus182.8%

The expected ordering holds for 86.3% (555/643) of sequences. While violations occur, as the reviewer correctly notes, the ordering is valid for the large majority of generated outputs.

To determine whether these violations affect our conclusions, we reevaluated all metrics on the 555 human-verified sequences.

MetricPairwise Acc. (%)Tie Rate (%)AUC_z
CLIP52.430.1210.93
DINO60.420.008.55
DreamBench++12.0779.1612.07
VIEScore16.0474.2912.63
Keypoints72.551.3830.25
MetMag (Ours)83.124.5059.16

MetMag's Pairwise Accuracy increases from 78.54% to 83.12% (+4.58 pp), and AUC_z increases from 53.23 to 59.16. It outperforms the strongest baseline, Keypoints, by 10.57 percentage points in accuracy.

These results demonstrate that violations of the assumed ordering do not explain MetMag's performance advantage.

1.2. Cross-prompt comparisons (Table 2).

We additionally validated the ordering assumption under contextual variation. We sampled 300 cross-prompt pairs, evenly distributed across None vs. Mild, None vs. Strong, and Mild vs. Strong comparisons (100 pairs each), with sampling distributed across products.

Six annotators independently compared each pair under blinded conditions. We assigned signed votes according to the expected ordering (+1: agree, 0: indistinguishable, −1: reverse) and classified each pair by the sign of the aggregated vote.

ClassificationCountPercentage
Agree24782.3%
Tie / no net signal113.7%
Reverse4214.0%

The expected ordering holds in 82.3% of cross-prompt comparisons, despite additional contextual variation.

We then compared metric performance on the full 300-pair sample and the 247 human-agreed pairs.

MetricAcc. (300)Acc. (247)AUC_z (300)AUC_z (247)
CLIP52.0055.0630.9233.78
DINOv253.6760.3228.6232.86
DreamBench++22.3325.1022.3325.10
VIEScore25.3327.9417.5319.19
Keypoints63.0070.8537.5243.74
MetMag (Ours)78.3389.0756.7967.15

On human-agreed pairs, MetMag achieves 89.07% accuracy, outperforming Keypoints by 18.22 percentage points. Its advantage therefore persists—and becomes more pronounced—when the expected ordering is independently verified by humans.

2. Human Validation on Naturally Generated Images

To directly assess agreement with human judgments beyond controlled degradation, we conducted an independent human study using naturally generated SDG outputs from clean references, without artificial degradation.

We evaluated three representative models: GPT-Image-2 [High], FLUX.1-Kontext, and Qwen-Image-Edit.

Human study protocol. We sampled 100 image pairs, stratified across the three model pairings. Five annotators independently judged which output better preserved the reference subject's fine-grained details, yielding 500 individual judgments.

2.1. Agreement with individual human judgments.

We compared each metric's pairwise preferences against individual human choices, excluding human ties or undecidable judgments and metric ties.

EvaluatorAgreement95% pair-bootstrap CI
Keypoints267/403 (66.3%)57.9%–74.4%
ID-Sim277/407 (68.1%)59.7%–75.9%
MetMag (Ours)338/403 (83.9%)78.1%–89.0%

Agreement is calculated over individual annotator votes, not unique image pairs. The number of valid comparisons differs because metric ties are excluded separately for each evaluator.

MetMag achieves 83.9% agreement with individual human judgments, outperforming ID-Sim by 15.8 percentage points and Keypoints by 17.6 percentage points.

The 95% confidence intervals were estimated using 1,000 bootstrap resamples clustered by image pair to account for dependencies among annotations of the same pair. MetMag's interval lies entirely above the reported intervals of both baselines.

These results provide direct evidence that MetMag more reliably identifies which naturally generated image better preserves the reference subject's fine-grained details, without relying on artificial reference degradation.

2.2. Agreement with human-derived model rankings.

We further compared Bradley–Terry Elo ratings derived from human preferences with those derived from metric-based preferences on the same sampled comparisons. Ratings use a 400-point Elo scale anchored at 1000.

ModelHuman EloKeypoints EloID-Sim EloMetMag Elo
GPT-Image-2 [High]1189.21043.21079.31382.2
FLUX.1-Kontext991.61127.31034.8979.7
Qwen-Image-Edit819.2829.5886.0638.1

Human preferences establish the ranking GPT-Image-2 [High] > FLUX.1-Kontext > Qwen-Image-Edit, which MetMag reproduces exactly. ID-Sim also recovers this ordering, whereas Keypoints incorrectly ranks FLUX.1-Kontext above GPT-Image-2.

The human-derived Elo ratings have non-overlapping 95% confidence intervals: GPT-Image-2 [High] (1152.8–1231.5), FLUX.1-Kontext (954.6–1027.1), and Qwen-Image-Edit (777.2–854.9).

Although MetMag's Elo gaps are not identical to those derived from human preferences, it correctly captures the model ordering and the pronounced advantage of GPT-Image-2 over FLUX.1-Kontext.

The model-level decisive win rates also show consistent trends:

ModelHuman win rate (%)MetMag win rate (%)
GPT-Image-2 [High]88.894.4
FLUX.1-Kontext47.843.2
Qwen-Image-Edit12.312.4

Together, the individual-comparison and model-level analyses demonstrate that MetMag aligns with human preferences on naturally occurring generation errors, rather than only on controlled reference degradation.

Summary

These additional studies address both concerns raised by the reviewer.

  1. Validity of controlled degradation: Human annotators confirm the expected ordering in 86.3% of same-prompt sequences and 82.3% of cross-prompt comparisons. Restricting evaluation to human-verified comparisons preserves and strengthens MetMag's advantage over existing metrics.

  2. Agreement with natural human judgments: On naturally generated images, MetMag achieves 83.9% agreement with individual human choices, compared with 68.1% for ID-Sim and 66.3% for Keypoints. It also reproduces the human-derived model ranking.

We will incorporate these findings into the revised manuscript and use the 555 human-verified sequences as the primary controlled evaluation subset of DreamBench-Detail.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We would first like to express our gratitude to the Program Chairs, Senior Area Chairs, and Area Chairs for their efforts, as well as to the reviewers for their careful reading and insightful comments on our paper.

We appreciate the reviewers for highlighting the strengths of our work:

  • An important and practically meaningful evaluation gap. The reviewers noted that high-level subject similarity does not guarantee faithful preservation of text, logos, patterns, and local structure, which is particularly consequential for applications such as product presentation.

  • A valuable benchmark and controlled protocol. DreamBench-Detail, with 439 detail-rich subjects across 25 product categories, and the controlled degradation protocol provide a systematic way to measure metric sensitivity to detail loss while holding other generation settings fixed.

  • A clear evaluation framework with informative ablations. Decomposing evaluation into target selection, localization, and crop-level judging makes the evaluation criteria explicit and individual errors inspectable, and the ablations and context-variation experiments support its effectiveness.

Based on the reviewers' constructive feedback, we conducted additional experiments and provided clarifications during the rebuttal. We summarize them below and incorporate them into the revised manuscript.

  • Human validation of the controlled protocol. Annotators confirmed the expected None > Mild > Strong ordering in 86.3% of same-prompt sequences and 82.3% of cross-prompt pairs. On the 555 human-verified sequences, which now serve as the primary benchmark, MetMag reaches 83.12% Pairwise Accuracy, compared with 72.55% for the strongest baseline.

  • Alignment with humans on naturally generated images. Without any artificial degradation, MetMag agrees with 83.9% of individual human choices, compared with 68.1% for ID-Sim and 66.3% for Keypoints, and reproduces the human-derived model ranking. It also achieves the highest correlation with human ratings on DreamBench++, including human subjects, where it complements ArcFace and LVFace.

  • Comparison with additional baselines. MetMag outperforms ID-Sim (CVPR 2026) (83.12% vs. 64.44%), crop-level DINOv3 and DreamSim computed on identical crop pairs (75.44% and 78.74%, with AUCz 59.16 vs. 27.43 and 27.96), and OCR-based NED on text-dominant subjects (88.06% vs. 83.89%).

  • Effectiveness beyond textual fidelity. About 45% of observed discrepancies are non-textual, and MetMag achieves the highest Pairwise Accuracy and AUCz across textual, color/pattern, omission, and geometric-dominant subjects. On subjects without OCR-detectable text, it retains the highest AUCz (50.44 vs. 25.39), although Keypoints achieves higher accuracy in this subset.

  • Reliability of the salient target list. The generated target lists cover 98.7% of the attributes that human annotators independently marked as important, with only 0.7% hallucinated labels.

  • Robustness of the evaluation pipeline. Accuracy remains stable under alternative aggregation weights, removal of near-duplicate targets, injection of target-agnostic crops, and alternative handling of undetected targets. It stays at or above 80.0% under all tested localization errors and at 76.70% under 30% occlusion. On the subtle None-to-Mild comparison, MetMag reaches 67.57%, compared with 53.87% for the best baseline.

  • Statistical analysis and refined claims. Subject-clustered bootstrap confidence intervals confirm significant gains from semantic target selection, fine-grained targets, and localization with cropping. Since combining coarse and fine targets is not significant on the main benchmark, we no longer claim that every component is essential, and instead present coarse targets as complementary for broader errors such as spatial relationships (74.24% vs. 70.20% in a patch-swapping study).

  • Practical utility as a reward signal. As a selection signal, MetMag raises human-judged identity preservation from 44.4% to 77.8% with best-of-N sampling and 88.9% with zeroth-order search, above ImageReward (55.6% and 66.7%). In offline DPO on FLUX.2-Klein-9B, the MetMag-trained model is preferred by humans over the original model and over DINOv2- and ImageReward-based DPO on held-out subjects, while prompt adherence and aesthetic quality are maintained.

  • Clarification of contributions. We clarified that our main contributions are identifying an overlooked evaluation gap, measuring it with a targeted benchmark and controlled protocol, and providing a framework validated against human judgments. The main claims of the paper remain unchanged, and most of the new results are included in the appendix.

We hope our responses have addressed the concerns raised, and we would be glad to discuss any remaining questions during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.