acceptodds
Under review as a conference paper at ICLR 2027

Do Generative Priors Align with Human Naturalness Perception?

Abstract

Visual generative models learn probability distributions over natural images, but whether their native priors directly capture human perception of image naturalness remains unresolved. Here, we probe these priors through native prediction errors across 25 open image and video generators. Because raw single-image losses are dominated by scene content and visual complexity, we evaluate directional loss differences using content-preserving, paired relational interventions that selectively disrupt facial configurations or physical illumination consistency while limiting changes in low-level image statistics. In representative conditions, these loss differences reproduce selective human sensitivities and tolerances, capturing the classic Thatcher effect on faces and shape-dependent responses to illumination inconsistencies. Crucially, they reliably track continuous human naturalness judgments across individual stimulus pairs (peaking at on faces and on physical scenes). Across the evaluated models, overall sensitivity to these violations broadly covaries with human alignment, yet the two dissociate along denoising schedules, with fine-grained perceptual alignment typically peaking before sensitivity. Importantly, generative loss differences retain positive partial correlations with human judgments after controlling for feature distances from frozen vision encoders and standard image quality metrics. Together, these findings demonstrate that learning visual distributions yields generative loss landscapes that capture distinct aspects of human naturalness perception.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Diffusion Models, Human Visual Perception, Model Evaluation, Human-AI Alignment, Visual Psychophysics

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6zwWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper asks whether the native prediction losses of visual generative models align with human naturalness perception. Using content-preserving relational interventions (Thatcherized faces and swapped shadows, reflections, and lighting in rendered scenes), the authors compare paired loss differences from 25 image and video generators with human ratings. They report that these differences track item-level human judgments (up to r = .84 on faces and .64 on illumination) and reproduce effects such as the Thatcher illusion. They also find that the losses retain partial correlations with human judgments after controlling for vision encoders and IQA metrics, and that sensitivity and alignment peak at different points along the denoising schedule.

Critical strengths

S1. Design that directly addresses density confounds. The authors first show that native losses are dominated by low-level statistics: on CSIQ, loss-DMOS correlations reverse sign for blur and JPEG, and deleting a shadow lowers loss (App. F). Their content-preserving pixel swaps with paired loss differencing are a principled response to this failure mode, which makes the main results more credible.

S2. Rigorous experiments and reproducibility. The psychophysics uses within-subject paired ratings and counterbalanced assignment. Bootstrap intervals cover participants, scenes, and model/layer selection, and encoder baselines are favored by best-layer selection, which keeps the comparison conservative. The released code, stimuli, model scores, and anonymized ratings further support credibility and reproducibility.

S3. Broad model coverage. 25 open generators across 11 families span pixel/latent space, diffusion/flow, and image/video, so the conclusions do not hinge on a single architecture.

Critical weaknesses

W1. Limited technical contribution. The paper does not propose a new model, training objective, or method for improving human alignment. Its contribution is an evaluation protocol and analysis, and both its tool and its question have precedents. Paired denoising-loss contrasts have already been used to evaluate physical plausibility in video diffusion models [1]. The relation between probabilities under generative models and human perceptual sensitivity has also been studied [2], although this work is not cited. The main novel element therefore appears to be the combination of content-preserving interventions with item-level human ratings. This may provide useful insights, but the technical contribution appears limited, and the authors should discuss how their findings differ from [2].

[1] J. Yuan et al. LikePhys: Evaluating intuitive physics understanding in video diffusion models via likelihood preference. ICLR 2026. [2] A. Hepburn et al. On the relation between statistical learning and perceptual distances. ICLR 2022.

W2. Headline correlations overstate the degree of alignment. The reported pooled correlations (r = .84 for faces and r = .64 for illumination, from the best models) appear to be driven in part by mean differences across conditions (e.g., upright vs. inverted stimuli). After condition centering, the 25-model median decreases from .75 to .41 on faces and from .33 to .27 on illumination. More importantly, within-condition model–human alignment remains substantially below human split-half reliability (e.g., .26 vs. .96 for shadows; Fig. S12). These results suggest that the headline correlations may overstate item-level alignment. Claims that models “reliably track” human judgments should therefore be qualified, and within-condition alignment should be emphasized as the primary measure.

W3. Inconsistent coupling between sensitivity and human alignment. The association between model sensitivity and human alignment is not consistent across stimulus conditions. While the relationship is observed for inverted faces and shadows, it is negligible for light direction (r = .05) and negative for reflections (r = -.37; Fig. S13). Although Sec. 4.2 acknowledges this, the abstract and discussion still state that sensitivity broadly covaries with human alignment. This variability weakens that claim, and its scope should be narrowed accordingly in the abstract and discussion.

W4. Mismatch between human and model evaluation tasks. The human participants were explicitly instructed to attend to specific properties of the stimuli (App. B.2), making their task closer to targeted detection, whereas model responses are evaluated using whole-image denoising loss. This discrepancy complicates the interpretation of model–human alignment, as the model metric may be influenced by image regions unrelated to the manipulated property. Evaluating denoising loss within the manipulated regions, alongside the full-image metric, would provide a more controlled comparison.

Questions for authors

Q1. Could the authors clarify how the proposed paired-loss probing differs methodologically from LikePhys and YoCausal, beyond the comparison with human judgments? For example, are there design choices (e.g., content-preserving interventions, timestep and noise aggregation) that the authors consider essential and that were absent in prior work?

Q2. App. I and Fig. S13 report results within each condition, but the task-level analyses are not repeated on condition-centered scores. Do the main conclusions hold when the comparison with encoder and IQA baselines (Fig. 5) and the cross-model relation between sensitivity and alignment (Fig. 3) use condition-centered alignment?

Q3. What do the authors believe drives the negative sensitivity–alignment association for reflections? Is it concentrated in specific model families, or related to the models' response to hue differences noted in Sec. 4.1?

Q4. Do the model–human correlations hold when the loss difference is restricted to the manipulated regions, and how much signal comes from outside these regions? Since spatial loss maps are already available (Fig. 2, App. M), this analysis should be feasible without new model runs.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 6zwW (1/4)

Thank you for your thorough review and concrete suggestions. We completed all requested analyses (condition-centered controls in W2, reflection diagnostics in W3, and inside/outside region evaluations in W4) and updated our claims, citations, and figures accordingly. Since Q1–Q4 map directly onto W1–W4 (thank you for organizing your review so clearly), we address each pair together below. Please also see our General Response for the distinction between SmS_m and rmr_m (G1) and the revision summary (G4).

W1 / Q1: Technical contribution and methodological relation to prior work

Methodological comparison with LikePhys and YoCausal (W1, Q1): We agree that core probing mechanics (matched-noise sampling, timestep aggregation, and paired denoising contrasts) follow the same formulation as prior work (Section 3.2). Technically, our setup differs in two respects: retaining continuous loss differences (ΔL\Delta L) instead of binary win/loss preferences, and using content-preserving factorial pairs rather than generated or unconstrained videos. As noted in our General Response (G1), these two choices suppress low-level confounds while spanning graded human tolerances, enabling us to measure violation sensitivity (SmS_m) and item-level human alignment (rmr_m) independently on the exact same stimuli and uncover their dissociations across conditions, model scale, and denoising schedules (Figures 4–5, Appendix K). In addition, as highlighted in G3, our systematic factorial design enables a fine-grained analysis of human–model agreements and divergences across individual scene factors (Figure 1B, Sections 4.1 and 5, Appendix J, Figures S15–S16).

Comparison with Hepburn et al. [2]: Thank you for pointing out Hepburn et al. [2], which we now cite and discuss in Section 2.1. Although both works relate learned image statistics to human perception at a broad level, they address fundamentally different questions. Hepburn et al. [2] investigate how a reference image's probability modulates local sensitivity to perturbations, evaluating unsigned distances (induced by autoencoders) between reference and distorted images against human ratings of generic distortions. We instead test whether signed changes in native prediction losses directly track perceived naturalness under higher-level relational manipulations of facial configuration and scene illumination, and identify where stronger overall violation responses fail to translate into closer item-level agreement with human judgments.

AuthorsOpenReview ↗

Response to Reviewer 6zwW (2/4)

W2 / Q2: Pooled vs. within-condition alignment and condition-centered task analyses

We agree. In the revised Abstract and Section 4.2, we replace peak-correlation emphasis with the full 25-model ranges (Thatcher r=.30–.84r = .30\text{--}.84, Illumination .04–.64.04\text{--}.64), remove "reliably track" phrasing, and explicitly note the remaining gap relative to human split-half reliability. Moreover, to separate between-condition mean differences from item-level agreement, Section 4.2 now reports condition-centered correlations directly alongside task-pooled metrics (all 25 models remain positive, with medians of .411.411 on Thatcher and .271.271 on Illumination, peaking at .534.534 and .619.619), complemented by full within-condition breakdowns in Appendices G–I (Figures S12–S14) and condition-centered task checks in Appendix S (Table S13).

We adopt this co-reporting strategy rather than dropping task-pooled alignment altogether because condition-level contrasts (such as orientation dependence in the Thatcher effect) are themselves a core part of the phenomena under study. Presenting pooled, condition-centered, and within-condition results side by side lets readers evaluate both condition-level structure and item-level ranking transparently.

With respect to the cast-shadow figure you cited (.26.26 vs. .96.96 human reliability), we also note that the .26.26 all-model median reflects wide spread across our 25-model panel (−.03-.03 to .80.80), whereas the five top Illumination models reach a median of .76.76 (and .56.56 on condition-centered Illumination, peaking at .62.62). Presenting both the full model range and top-five medians (Appendix I, Figure S14) clarifies that stronger generators show substantial within-condition agreement while still falling short of the human split-half reliability reference.

Condition-centered checks for main task-level conclusions (Q2): To test whether our main task-level conclusions hold after removing between-condition mean differences, we repeated both Figure 3 (baseline comparisons, formerly Figure 5) and Figure 4 (SmS_m–rmr_m coupling, formerly Figure 3) using condition-centered residuals (subtracting condition-specific means from human ratings and all model/baseline scores). Both conclusions remain supported:

  • Encoder and IQA baselines (Figure 3): Preserving the task-pooled selection procedure (repeating best-layer and top-five selection within each of 10,000 participant-and-scene bootstrap draws) and evaluating on condition-centered scores, generative losses retained positive partial correlations beyond encoders, FR-IQA, and NR-IQA across all six comparisons (all 95% CIs for median paired differences exclude zero):

    Task & Baseline (XX)Centered r(h,G∣X)r(h, G \mid X)Centered r(h,X∣G)r(h, X \mid G)Median Paired Diff. (G−XG - X)
    Thatcher
    Vision encoders.404 [.283,.453].404\ [.283, .453].110 [.044,.235].110\ [.044, .235]+.290 [.087,.363]+.290\ [.087, .363]
    FR-IQA.467 [.339,.510].467\ [.339, .510].020 [−.084,.124].020\ [-.084, .124]+.454 [.265,.549]+.454\ [.265, .549]
    NR-IQA.506 [.374,.560].506\ [.374, .560].069 [−.003,.168].069\ [-.003, .168]+.412 [.263,.512]+.412\ [.263, .512]
    Illumination
    Vision encoders.511 [.370,.564].511\ [.370, .564].353 [.164,.435].353\ [.164, .435]+.149 [.008,.305]+.149\ [.008, .305]
    FR-IQA.519 [.367,.585].519\ [.367, .585].097 [−.054,.247].097\ [-.054, .247]+.421 [.154,.598]+.421\ [.154, .598]
    NR-IQA.565 [.463,.637].565\ [.463, .637].008 [−.027,.057].008\ [-.027, .057]+.556 [.435,.619]+.556\ [.435, .619]

    Reselecting encoder layers and top-five models directly on centered alignment yields consistent results—positive partial correlations beyond encoders (.362.362 Thatcher; .499.499 Illumination) and continued superiority over FR/NR-IQA—except that paired generative-minus-encoder differences span zero (.137 [−.031,.248].137\ [-.031, .248] and .148 [−.019,.276].148\ [-.019, .276]).

  • Cross-model sensitivity–alignment relation (Figure 4): Using condition-centered alignment, cross-model correlations with SmS_m remained positive under the joint family-, participant-, and scene-bootstrap: r=.743 [.172,.933]r = .743\ [.172, .933] (Thatcher) and r=.737 [.363,.918]r = .737\ [.363, .918] (Illumination), closely matching pooled correlations (.663.663 and .821.821).

We summarize this in Section 4.2 and provide the full analysis in Appendix S (Table S13).

AuthorsOpenReview ↗

Response to Reviewer 6zwW (3/4)

W3 / Q3: Condition dependence of sensitivity–alignment coupling and reflection diagnostics

Thank you for highlighting this inconsistency. We agree that stating that sensitivity "broadly covaries" with human alignment overstates the cross-model relationship given its condition-level variability. As noted in General Response (G1), this condition-level dissociation, alongside schedule divergence and reflection scaling, is central to our main takeaway.

In the revision, we removed the "broadly covaries" framing from the Abstract, Introduction, Section 4.3 (formerly Section 4.2), and Discussion:

  1. Abstract & Introduction: We now state explicitly that while sensitivity and alignment associate positively across each full domain, cross-model coupling breaks down for specific violation types and decouples along denoising schedules.
  2. Section 4.3 & Figure 4C: We added a condition-resolved forest plot (Figure 4C) alongside the task-pooled scatters and report exact condition-level correlations and 95% joint family/participant/scene CIs in the main text (cast shadows: .810 [.424,.960].810\ [.424,.960]; inverted faces: .723 [.118,.923].723\ [.118,.923]; upright faces: .667 [−.215,.904].667\ [-.215,.904]; light direction: .051 [−.417,.570].051\ [-.417,.570]; surface reflections: −.371 [−.704,.364]-.371\ [-.704,.364]).
  3. Discussion: We replaced the opening summary with a unified discussion of sensitivity–alignment dissociation across conditions, model scale on reflections (Appendix K, Figure S22), and denoising schedules (Section 4.3; Figure 5, formerly Figure 4).

Drivers of the negative reflection association (Q3): Because the 95% CI spans zero ([−.704,.364][-.704, .364]), we do not interpret −.371-.371 as a reliable inverse law. New leave-one-family-out and factor-level diagnostics across the 96 reflection pairs (summarized in Appendix H) clarify what drives this point estimate:

  1. Model-family influence: Across the 11 model families, removing the three pixel-space JiT models (which combine low reflection sensitivity with relatively high alignment) shifts the correlation from r=−.371r = -.371 to r=−.029r = -.029, whereas excluding any of the other 10 families yields r=−.458r = -.458 to −.328-.328.
  2. Factor dependence and sensitivity balance: Excluding color manipulations moves the correlation only partway toward zero (r=−.173r = -.173), showing that hue insensitivity alone does not explain the trend. Evaluating alignment within each manipulation type (32 pairs each) shows that the relation with overall reflection sensitivity turns positive on rotation trials (r=.291r = .291) while remaining negative on shape (−.434-.434) and color (−.371-.371). Moreover, regressing out mean differences across the three reflection manipulation types from both human and model scores before computing alignment shifts the correlation to r=−.031r = -.031, and additionally removing the main effect of floor roughness shifts it to +.382+.382 (95% CI [−.064,.792][-.064, .792]).

These diagnostics reinforce our central conclusion: average violation sensitivity cannot be treated as a general proxy for human alignment, as the coupling between the two depends on both the stimulus condition and how responses are distributed across manipulation factors.

Reviewer_z5yQOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper use a proposed evaluation framework to evaluate 25 generative models, trying to answer the question of whether the learned priors can naturally align with human perception of image naturalness. The two target domains are disrupt facial configurations and physical illumination consistency. The authors conducted controlled experiment and comprehensive statistical analysis to show generative prior retain more positive partial correlations with human judgments compared to frozen encoders.

Critical strengths

S1. I like the statistical analysis done by the authors. The human studiy is carefullly designed and broad empirical evaluation are reported with detailed analyses

S2. Very well written and easy to follow the ideas and flow.

Critical weaknesses

W1. The evaluation is limited. Only two perceptual domains are covered. It's a bit stretched for this to represent/generalize to entire human naturalness. There are many other factors, semantic validity, occlusion, texture, etc. Then, what about things human can't see, the invisible details/noise, etc. How does the generative prior align with human on this.

W2. The comparison with frozen encoders is not as informative to me. The model is not trained to indicate this naturalness with the vector distance. What about some linear models trained on top of human judgements, or discriminative models

Questions for authors

Q1. For the human scores, are the scorers told to look for certain cues, like lighting, reflection? Are the scorers progressively getting better during the study, picking up the task and very niche details. Could the results be biased?

Q2. Could the authors elaborate more on one claim made in the paper, $\textit{adherence to objective physical laws does not directly imply alignment with human perceptual naturalness}$. Reflections and lighting are also adherent to objective physical laws, only with the understanding of those physical law one could tell if it's natural. This doesn't really convince me how the designed investigation is materially different from pervious work.

Q3. What's the purpose of using video models for images?

6 comments
AuthorsOpenReview ↗

Response to Reviewer z5yQ (1/2)

Thank you for your constructive review and insightful questions. Below, we address each point in turn, including two new post-hoc analyses: nested-CV Ridge probes (W2) and a trial-order learning check (Q1). Please also refer to our General Response (G1–G4) for cross-reviewer clarifications on SmS_m vs. rmr_m, encoder baselines, factorial scope, and the revised section/figure map.

W1: Perceptual scope and generalization

We agree that those untested dimensions and imperceptible changes fall outside our scope. Rather than aiming for broad domain coverage, our study isolates higher-level relational naturalness while minimizing low-level statistical shifts, using systematic factorial variation across facial configurations and three illumination paradigms (13 scene factors, 44 levels; Appendices A and J, Figures S15–S19). We now state this scope and design rationale explicitly in Sections 1 and 5 (see also G3).

W2: Human-supervised linear readouts and discriminative models

We agree that frozen encoders are not trained to express naturalness via vector distances. As clarified in our General Response (G2), we therefore do not treat unsigned encoder distances as upper bounds on encoder capacity, and instead position them strictly as diagnostic zero-shot controls for representational change magnitude (Sections 3.4 and 4.2, Appendix D.1).

Regarding discriminative baselines, Figure 3 (formerly Figure 5) already evaluates pretrained FR-IQA and NR-IQA metrics (including LPIPS, DISTS, HyperIQA, MUSIQ, and LIQE), which show weak human alignment and leave generative models with strong positive partial correlations. To also check whether frozen encoders perform better when scored directionally without supervised fitting, we added zero-shot directional text-anchored VLM scoring (1b97 W2; Appendix Q, Table S10), which did not consistently outperform unsigned distance across tasks.

To address your question on supervised linear models, we also tested nested-CV ridge regression on signed feature differences (Δϕ=ϕmodified−ϕoriginal\Delta\phi = \phi_{\mathrm{modified}} - \phi_{\mathrm{original}}) across all 12 frozen encoders. Under scene-grouped CV, encoder readouts achieved r=.91r = .91, .86.86, and .70.70 for Shadow, Reflection, and Light direction, respectively. However, an image-blind ridge model using only categorical scene-factor metadata and candidate pairwise interactions already achieved r=.75r = .75, .89.89, and .72.72.

In our compact factorial design, a fitted readout can predict human ratings simply by identifying scene configurations and learning their rating associations during training, without the pretrained model itself assigning those configurations a naturalness ordering. High CV accuracy therefore cannot distinguish a native naturalness preference from supervised factor decoding. While supervised readouts are informative about feature predictability, they do not test whether pretrained models align with human naturalness without learning the mapping from our human judgments, so we focus our comparisons on unfitted baselines and diagnostic controls.

AuthorsOpenReview ↗

Response to Reviewer z5yQ (2/2)

Q1: Participant instructions and learning during the study

As detailed in Appendix B.2 and now summarized directly in Section 3.1, participants received illustrated instructions and practice trials before the main session specifying the target attribute to evaluate (internal facial features for Thatcher; shadow consistency, reflection matching, or central-object lighting consistency across between-subject Illumination groups). In the main session, images were presented individually in randomized order without side-by-side comparisons and rated on a 5-point naturalness scale.

Specifying the target attribute was necessary in crowdsourced testing so non-expert observers would not default to incidental cues (e.g., attractiveness or rendering style), while revealing nothing about whether an individual image was modified. Crucially, explicit instructions did not override genuine perceptual tolerances: observers still overlooked inverted facial manipulations (Thatcher effect) and strongly penalized teapot shadow swaps while remaining largely tolerant under Spot (Figure 1B).

To test whether participants progressively learned the manipulations during the main session, we estimated separate trial-order slopes for original and modified ratings with participant and image fixed effects:

Rsi=αs+βi+γorigtsiI(orig)+γmodtsiI(mod)+ϵsiR_{si} = \alpha_s + \beta_i + \gamma_{\mathrm{orig}} t_{si} I(\mathrm{orig}) + \gamma_{\mathrm{mod}} t_{si} I(\mathrm{mod}) + \epsilon_{si}

where RsiR_{si} is participant ss's naturalness rating for image ii, αs\alpha_s and βi\beta_i are participant and image fixed effects, tsi∈[0,1]t_{si} \in [0, 1] is normalized trial order across the main session, I(orig)I(\mathrm{orig}) and I(mod)I(\mathrm{mod}) are indicator functions for original and modified images, γorig\gamma_{\mathrm{orig}} and γmod\gamma_{\mathrm{mod}} are the corresponding trial-order slopes, and ϵsi\epsilon_{si} is the residual error. Evaluating the slope difference γorig−γmod\gamma_{\mathrm{orig}} - \gamma_{\mathrm{mod}} (where positive values indicate widening original-minus-modified separation) with participant-clustered standard errors and Holm correction across the five conditions showed that all 95% confidence intervals spanned zero (Upright: +0.043 [−0.091,+0.177]+0.043\ [-0.091, +0.177]; Inverted: −0.084 [−0.241,+0.073]-0.084\ [-0.241, +0.073]; Shadow: −0.020 [−0.195,+0.155]-0.020\ [-0.195, +0.155]; Reflection: +0.079 [−0.135,+0.293]+0.079\ [-0.135, +0.293]; Light direction: +0.324 [−0.107,+0.755]+0.324\ [-0.107, +0.755]; all Holm-adjusted p≥.652p \ge .652), with consistent results without participant screening. Thus, we find no evidence of progressive mid-session learning biasing human ratings.

Q2: Physical correctness versus perceived naturalness

We agree that lighting and reflections are governed by objective physical laws, and every modification in our Illumination task introduces an optical violation. However, detecting a physical violation differs from predicting its perceptual impact: human observers strongly penalize some optical inconsistencies while overlooking others (Ostrovsky et al., 2005; Nightingale et al., 2019). Accordingly, we systematically vary scene factors so that the same physical violation (e.g., a 180∘180^\circ shadow flip) ranges from clearly unnatural under an asymmetric teapot to barely noticeable under Spot (Section 3.1, Figure 1B), testing whether model losses track human judgments across individual pairs.

As highlighted in our General Response (G1), this directly distinguishes our protocol from benchmarks such as LikePhys and YoCausal, which evaluate binary preference over physically valid vs. invalid videos and compare aggregate model rankings against generated-video evaluations. Measuring violation sensitivity (SmS_m) and item-level alignment (rmr_m) separately on the same stimuli reveals that physical violation detection does not imply human perceptual weighting. This dissociation is most striking on reflections, where larger models gain sensitivity (r=.783r = .783) without gaining alignment (r=−.256r = -.256; Appendix K, Figure S22), and also emerges across denoising schedules (Figure 5, formerly Figure 4). We have sharpened this distinction in Sections 2.2 and 3.1.

Q3: Why evaluate video models on images?

While temporal and motion dynamics are beyond our scope, we included nine video generative models to test a specific hypothesis about spatiotemporal pre-training. Because video models learn from continuous temporal sequences depicting changing viewpoints, lighting, and motion, they might internalize richer 3D physical and geometric regularities that yield stronger or qualitatively different alignment even on static scenes. Evaluating both model classes on identical static interventions tested this directly. Empirically, video models showed no systematic advantage or qualitative difference over image models (Sections 3.2 and 4.2, Figure 4, Table S1), indicating that spatiotemporal training alone does not automatically produce more human-aligned static naturalness priors. We have added this motivation and finding to Sections 3.2 and 4.2.

Thank for your response. My questions regarding the choices made in the paper are clarified. I like the rigorous experimental design but my main concern is still not addressed. The scope limitation allows more controlled experiments and better isolated of individual factors, but the claim is still overstated, this is not equal to human naturalness. Reframe the claim would be better

Reviewer_1b97OpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether the native prediction losses of visual generative models align with human naturalness perception. The authors introduce content-preserving paired interventions involving facial configurations and illumination consistency, and evaluate 25 image and video generative models. Experiments show that paired loss differences correlate with human naturalness judgments and capture perceptual patterns that are not fully explained by existing vision encoders and image quality metrics.

Critical strengths
  1. Novel and interesting research question. The paper investigates whether generative priors inherently capture human perceptual regularities, providing insights into what visual generative models learn beyond generation quality.

  2. Carefully designed experiments. The content-preserving paired interventions help reduce confounding factors, while human psychophysical evaluations provide direct evidence for assessing perceptual alignment.

  3. Extensive empirical evaluation. The study evaluates 25 generative models across diverse architectures and compares them against vision encoders and image quality metrics, providing evidence that native generative losses capture complementary perceptual information.

Critical weaknesses
  1. Limited diversity of perceptual tasks. The experiments focus on only two perceptual domains: facial configurations (Thatcher effect) and illumination consistency. These tasks cover a relatively narrow subset of human naturalness perception. Can the authors clarify whether the observed alignment generalizes to other perceptual dimensions, such as object geometry, spatial relationships, textures, and motion?

  2. Potentially unfair baseline comparisons. The generative approach uses directional loss differences, whereas vision encoder baselines primarily rely on feature distances that measure the magnitude of change rather than its direction. This asymmetry may favor the proposed approach. Could the authors consider stronger baselines with directional scoring mechanisms to establish whether generative priors provide genuinely unique advantages?

  3. Potential confounding effects from image manipulations. Although the paired interventions preserve much of the original image content, local flipping operations may introduce boundary artifacts, texture discontinuities, or low-level statistical changes. It remains unclear whether increased generative losses primarily reflect violations of perceptual relationships or sensitivity to these artifacts. Additional controls would strengthen the causal interpretation.

  4. Limited validation of native loss as a perceptual prior. The paper interprets native prediction errors as empirical surrogates for image plausibility, but denoising and flow-matching losses are not necessarily equivalent to calibrated image likelihoods. The authors should clarify how strongly the observed loss differences support claims about learned generative priors rather than architecture- or objective-specific prediction errors.

  5. Insufficient evidence for generalization beyond controlled stimuli. The experiments rely on carefully constructed paired modifications, whereas real-world unnatural images often involve multiple interacting distortions or complex semantic inconsistencies. Furthermore, the paper shows that raw native losses perform poorly on a standard image distortion benchmark. How broadly can the proposed findings generalize to natural, unconstrained images?

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer 1b97 (1/2)

Thank you for your thoughtful review and constructive feedback. Below we address each of your points directly—highlighting our new zero-shot directional text-prompt evaluation across seven VLMs (Appendix Q, Table S10) and our clarifications on perceptual scope, editing artifacts, and native prediction losses. Please also see our General Response for the distinction between SmS_m and rmr_m (G1), encoder baseline framing (G2), factorial design rationale (G3), and the updated section/figure map (G4).

W1: Scope across perceptual domains

We cannot claim generalization to untested domains such as motion, general textures, or broader geometric and spatial tasks; indeed, even within Illumination (G3), alignment is not uniform across cues. Rather than claiming broad domain coverage, our core contribution is a controlled diagnostic framework that isolates relational plausibility to pinpoint where generative priors agree with versus diverge from human perception. Specifically, varying 13 scene factors (44 levels) across constrained shape/rotation, elevation/light-angle, and surface parameters (Appendix A) lets us inspect agreement and divergence factor by factor (e.g., matching rotation asymmetry and shadow softness while diverging on shape swaps and subtle hue shifts; Figure 1B). To highlight this diagnostic contribution in the revision, we clarified our relational scope and generalization limits in Sections 1, 3.1, and 5, linked Section 4.1 to the factor breakdowns, and reorganized Appendix J around stimulus-paired agreement (Figure S15) and divergence (Figure S16) summaries (full profiles in Figures S17–S19).

W2: Directional scoring and fairness of encoder baselines

Thank you for this important point. As clarified in G2, our goal is not to claim that generative models outperform encoder representations; within our zero-shot scope (without fitting readouts to human ratings), we frame encoder distances strictly as diagnostic controls for change magnitude rather than capacity bounds (Sections 1, 3.4, 4.2; Appendix D.1).

Still, to test whether lacking directionality makes unsigned distance an overly weak zero-shot baseline, we evaluated all seven vision–language encoders with text towers using task-informed natural vs. unnatural prompts (e.g., "The shadow shapes on the tabletop look natural [unnatural] relative to the object shapes"): s=voriginal⊤tc−vmodified⊤tc,tc=tnatural−tunnaturals = v_{\text{original}}^\top t_c - v_{\text{modified}}^\top t_c, \quad t_c = t_{\text{natural}} - t_{\text{unnatural}} Below are pooled Pearson rr values compared against unsigned cosine distance on the same ℓ2\ell_2-normalized final embeddings v(x)v(x) (with ℓ2\ell_2-normalized tnaturalt_{\text{natural}} and tunnaturalt_{\text{unnatural}}):

Vision–Language EncoderThatcher: Unsigned DistanceThatcher: Signed Text Score (Δr\Delta r)Illumination: Unsigned DistanceIllumination: Signed Text Score (Δr\Delta r)
CLIP ViT-B/16.650.650.715.715 (+.065+.065).390.390.214.214 (−.177-.177)
SigLIP 2 (Base/16).445.445.624.624 (+.179∗+.179^*).470.470.175.175 (−.295∗-.295^*)
PE-Core (Base/16).530.530.633.633 (+.102∗+.102^*).454.454.298.298 (−.155-.155)
PE-Core (Giant/14).660.660.720.720 (+.060+.060).445.445.328.328 (−.117-.117)
Qwen3-VL-Embedding (2B).402.402.549.549 (+.147∗+.147^*).256.256−.040-.040 (−.296∗-.296^*)
Qwen3-VL-Embedding (8B).527.527.663.663 (+.136∗+.136^*).426.426.195.195 (−.231-.231)
Jina CLIP v2.551.551.686.686 (+.135∗+.135^*).443.443.240.240 (−.203-.203)

*Simultaneous 95% bootstrap CI for Δr\Delta r excludes zero.

On Thatcher, directional text scoring improves pooled correlation across all seven models (five significant), but this reflects coarse upright–inverted separation and disappears after condition centering (Δr=−.053 to +.031\Delta r = -.053\text{ to }+.031). On Illumination, it underperforms unsigned distance across all seven models (r=−.040 to .328r = -.040\text{ to }.328, with two Δr\Delta r values significantly negative). Four simpler prompt variants (generic antonym pairs and a coarser scene-level prompt) showed the same pattern. Because unsigned distance provides the stronger prompt-free baseline on Illumination, we retain it in the main analysis and report the full evaluation in Appendix Q (Table S10; summarized in Section 3.4).

AuthorsOpenReview ↗

Response to Reviewer 1b97 (2/2)

W3: Boundary artifacts and low-level confounds

We agree that paired manipulations cannot completely eliminate editing artifacts or subtle low-level shifts. However, four converging lines of evidence argue against a simple artifact-based explanation and support sensitivity to contextual and relational structure:

  1. Stimulus construction (Appendix A): Thatcherized faces blend inverted eye and mouth regions using Gaussian feathering along elliptical contours, while Illumination scenes elevate objects above the floor so floor swaps alter shadows and reflections without changing object–floor contact contours.
  2. Orientation control: Inverted Thatcher stimuli are exact 180∘180^\circ rotations of upright pairs and therefore share identical local edits and boundary transitions. Yet the 25-model median sensitivity drops sharply from 2.242.24 (upright) to 0.840.84 (inverted), closely mirroring human perceptual attenuation (Figure 1B).
  3. Scene-dependent responses: The same floor-flipping procedure elicits strong penalties under asymmetric objects (e.g., teapot) but minimal penalties under shapes lacking in-plane asymmetries (e.g., Spot; Figure 1B), and spatial loss maps show loss increases extending to unmodified scene elements such as elevated objects and flanking objects (Figure 2, Appendix M).
  4. IQA controls (Section 4.2, formerly Section 4.3): While subtle boundary or texture irregularities—especially in Thatcher faces—may still enter human ratings, the evaluated NR-IQA metrics correlate near zero with human judgments (median r=.087r = .087 on Thatcher, .009.009 on Illumination; Figure 3A), and generative models retain strong positive partial correlations after controlling for individual FR-IQA and NR-IQA metrics (Figure 3B).

We clarify this nuanced distinction in Section 4.2.

W4: Native prediction loss versus calibrated likelihood

We agree that uniformly averaged denoising and flow-matching losses are not calibrated image log-likelihoods. In the revision, we refine our phrasing in Sections 1 and 3.2 to describe paired loss differences (ΔL=Lmod−Lorig\Delta L = L_{\mathrm{mod}} - L_{\mathrm{orig}} under matched noise and timesteps) as relative plausibility surrogates under each model's learned generative objective, avoiding implying equivalence to calibrated densities.

Two empirical observations support the interpretation that native loss differences capture perceptually relevant learned regularities across varied modeling choices:

  • Human alignment is observed across diverse model families—spanning diffusion and flow formulations, pixel and latent spaces, and U-Net and transformer backbones (Figure 3A, Section 4.2; Appendix C, Table S1)—while model-wise comparisons also document Thatcher orientation dependence across different architectures (Appendix G, Figure S12). The selected top-five Illumination models additionally reproduce shape-dependent shadow tolerance (Figure 1B; Appendix J, Figures S15 and S17).
  • After controlling for violation sensitivity and model size, an exploratory comparison yielded positive partial-correlation estimates with external generation benchmarks (human-preference Elo and VBench Quality) for image illumination and video models (Section 4.3, Appendix L).

At the same time, we do not claim that generative priors can be separated from the architectures and objectives through which they are learned and evaluated; rather, we empirically probe these regularities through each model's learned objective, as now stated explicitly in Sections 1 and 3.2.

W5: Generalization beyond controlled stimuli

We agree that our findings do not transfer directly to unconstrained, multi-distortion images (Sections 1 and 5). Rather than proposing an unconstrained quality metric, our contribution is a scientific test of whether native generative losses capture graded human relational naturalness without supervised rescaling. As Appendix F shows, standalone single-image losses fail across differing scene compositions (Figure S11), and even paired differences fail when low-level statistics shift, as on CSIQ and shadow deletion (Figure S10). These observations motivated our use of content-preserving paired interventions to reduce low-level confounds and uncover higher-level relational regularities otherwise obscured in unconstrained evaluations. Accordingly, we frame extending these findings to unconstrained scenes as an explicit scope limitation (G3, W1; Section 5) rather than a gap in our intended contribution.

A brief follow-up on your comments about scope and generalization (W1 and W5). We have changed the title to "Do Generative Priors Align with Perceived Naturalness in Faces and Illumination?" and revised the abstract and main text to frame our conclusions explicitly around these two domains. We hope these revisions make the scope of our claims clearer.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to Reviewers (Part 1 of 2)

We sincerely thank all three reviewers (1b97, z5yQ, 6zwW) for their thoughtful, constructive feedback and for their positive assessments of our presentation, soundness, and informativeness—including recognizing the clarity of the writing and the rigor of our experimental design and 25-model evaluation. We have revised the manuscript and conducted new analyses to address your concerns. Below, we clarify three common points raised across the reviews; Part 2 (G4) summarizes the major manuscript revisions and where each new analysis appears.

G1. Separating violation sensitivity (SmS_m) and human alignment (rmr_m) (z5yQ Q2; 6zwW W1, Q1)

Reviewers asked how our study differs from prior paired-loss benchmarks such as LikePhys and YoCausal, a distinction we now state more explicitly in the revision (Sections 1, 2.2, and 3.2). While those benchmarks evaluate binary win/loss preferences on generated or unconstrained videos, we pair continuous loss differences (ΔL\Delta L) with item-level human naturalness ratings on content-preserving factorial pairs spanning facial configurations (Thatcher) and physical illumination factors, evaluated across 25 generators and their denoising schedules. This lets us measure violation sensitivity (SmS_m) and item-level human alignment (rmr_m) independently on the exact same stimuli, revealing that strong violation detection does not guarantee human-aligned perceptual weighting: across models, SmS_m and rmr_m decouple for light direction and surface reflections (Figure 4C), with larger models growing more sensitive to reflections without improving in human alignment (Appendix K, Figure S22); within models, alignment peaks earlier along the denoising schedule than sensitivity in nearly all models across both tasks (Figure 5).

G2. Diagnostic role of vision-encoder baselines (1b97 W2; z5yQ W2)

Reviewers questioned whether comparing directional generative losses against unsigned feature distances from frozen vision encoders is fair or informative (1b97 W2; z5yQ W2). We clarify in the revision (Sections 1, 3.4, 4.2; Appendix D.1) that we do not intend to claim that generative models outperform encoder representations; within our zero-shot scope, encoder distances serve strictly as diagnostic controls for change magnitude rather than upper bounds on encoder capacity. Still, to check whether lacking directionality makes distance an overly weak zero-shot baseline, we added directional text-prompt scoring across all seven vision–language encoders (Appendix Q, Table S10), which did not consistently outperform unsigned distance across tasks (detailed in our reply to 1b97 W2).

G3. Scope of evaluated domains and factorial design rationale (1b97 W1, W5; z5yQ W1)

Reviewers asked how our findings extend across perceptual dimensions and to unconstrained natural images (1b97 W1, W5; z5yQ W1). Our goal is not to build a general-purpose metric for unconstrained images, where unconditioned losses are confounded by low-level statistics (Section 1, Appendix F), or to claim generalization to untested domains (Section 5). Rather, our contribution is a controlled diagnostic framework that isolates higher-level relational plausibility to pinpoint where generative priors agree with or diverge from human perception. To this end, our benchmark spans upright and inverted facial configurations alongside three illumination paradigms systematically varied across 13 scene factors (44 levels) (Section 3.1, Appendix A), revealing concrete human–model agreements and divergences in Figure 1B and Sections 4.1 and 5. In particular, Appendix J now pairs factor profiles directly with stimulus examples illustrating key agreements (e.g., geometry-dependent rotation asymmetry and shadow softness in Figure S15) and divergences (e.g., shape swaps and subtle reflection hue shifts in Figure S16).

AuthorsOpenReview ↗

General Response (Part 2 of 2)

G4. Summary of major revisions

Major changes to the main paper (and Appendix J)

  • Reordered Section 4 & emphasized sensitivity–alignment dissociation (Abstract, §§1, 4.2–4.3, 5; Figs. 3–5):
    • Restructured Section 4 & figure order: Item-level alignment and baseline controls (§4.2, Fig. 3, formerly Fig. 5) now precede sensitivity–alignment coupling and dissociation (§4.3, Figs. 4–5, formerly Figs. 3–4), moving the denoising-schedule interpretation from the Discussion into §4.3 to avoid repetition.
    • Explicit dissociation framing & new Discussion paragraph: Removed the "broadly covaries" framing, added Fig. 4C to show that cross-model SmS_m–rmr_m coupling breaks down on light direction and reflections, and added a dedicated Discussion paragraph ("Dissociation between violation sensitivity and perceptual alignment", §5) tying together condition-level, model-scale (App. K), and schedule dissociations.
    • Unified terminology: Standardized violation sensitivity for SmS_m throughout (previously mixed with directional sensitivity).
  • Qualified item-level alignment claims (Abstract, §4.2): To ensure that headline figures do not overstate item-level agreement, we replaced the peak-correlation emphasis with full 25-model ranges (alongside top-five medians), explicitly noted the gap relative to human split-half reliability, and reported condition-centered and within-condition correlations alongside pooled metrics.
  • Clarified scope, baselines, and prior work (§§1, 2.1–2.2, 3.2, 3.4, 4.2; App. D.1): Clarified that paired losses are relative plausibility surrogates (not calibrated log-likelihoods), reframed frozen encoder distances as zero-shot diagnostic controls rather than capacity bounds, and discussed Hepburn et al. (2022) alongside LikePhys and YoCausal.
  • Reorganized Appendix J (Figs. S15–S19): Added stimulus-paired agreement/divergence summaries (Figs. S15–S16) and removed the secondary profile-MSE overview (shifting Figs. S20+ by +1+1).

New analyses added to the manuscript

AnalysisPurposeLocation
Directional text prompts (7 VLMs)Test zero-shot directional scoring vs. unsigned distance (1b97 W2)§3.4 summary; App. Q (Table S10)
Inside/Outside region-restricted lossVerify alignment in target areas and check outside signal (6zwW W4/Q4)§4.1 summary; App. R (Tables S11–S12)
Condition-centered baseline & SmS_m–rmr_m checksConfirm baseline advantage and SmS_m–rmr_m coupling hold within conditions (6zwW W2/Q2)§4.2 summary; App. S (Table S13)
Reflection family/factor diagnosticsDiagnose drivers of the nominal negative SmS_m–rmr_m trend on reflections (6zwW W3/Q3)App. H summary (details in 6zwW W3)

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.