- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper asks whether the native prediction losses of visual generative models align with human naturalness perception. Using content-preserving relational interventions (Thatcherized faces and swapped shadows, reflections, and lighting in rendered scenes), the authors compare paired loss differences from 25 image and video generators with human ratings. They report that these differences track item-level human judgments (up to r = .84 on faces and .64 on illumination) and reproduce effects such as the Thatcher illusion. They also find that the losses retain partial correlations with human judgments after controlling for vision encoders and IQA metrics, and that sensitivity and alignment peak at different points along the denoising schedule.
Critical strengths
S1. Design that directly addresses density confounds. The authors first show that native losses are dominated by low-level statistics: on CSIQ, loss-DMOS correlations reverse sign for blur and JPEG, and deleting a shadow lowers loss (App. F). Their content-preserving pixel swaps with paired loss differencing are a principled response to this failure mode, which makes the main results more credible.
S2. Rigorous experiments and reproducibility. The psychophysics uses within-subject paired ratings and counterbalanced assignment. Bootstrap intervals cover participants, scenes, and model/layer selection, and encoder baselines are favored by best-layer selection, which keeps the comparison conservative. The released code, stimuli, model scores, and anonymized ratings further support credibility and reproducibility.
S3. Broad model coverage. 25 open generators across 11 families span pixel/latent space, diffusion/flow, and image/video, so the conclusions do not hinge on a single architecture.
Critical weaknesses
W1. Limited technical contribution. The paper does not propose a new model, training objective, or method for improving human alignment. Its contribution is an evaluation protocol and analysis, and both its tool and its question have precedents. Paired denoising-loss contrasts have already been used to evaluate physical plausibility in video diffusion models [1]. The relation between probabilities under generative models and human perceptual sensitivity has also been studied [2], although this work is not cited. The main novel element therefore appears to be the combination of content-preserving interventions with item-level human ratings. This may provide useful insights, but the technical contribution appears limited, and the authors should discuss how their findings differ from [2].
[1] J. Yuan et al. LikePhys: Evaluating intuitive physics understanding in video diffusion models via likelihood preference. ICLR 2026. [2] A. Hepburn et al. On the relation between statistical learning and perceptual distances. ICLR 2022.
W2. Headline correlations overstate the degree of alignment. The reported pooled correlations (r = .84 for faces and r = .64 for illumination, from the best models) appear to be driven in part by mean differences across conditions (e.g., upright vs. inverted stimuli). After condition centering, the 25-model median decreases from .75 to .41 on faces and from .33 to .27 on illumination. More importantly, within-condition model–human alignment remains substantially below human split-half reliability (e.g., .26 vs. .96 for shadows; Fig. S12). These results suggest that the headline correlations may overstate item-level alignment. Claims that models “reliably track” human judgments should therefore be qualified, and within-condition alignment should be emphasized as the primary measure.
W3. Inconsistent coupling between sensitivity and human alignment. The association between model sensitivity and human alignment is not consistent across stimulus conditions. While the relationship is observed for inverted faces and shadows, it is negligible for light direction (r = .05) and negative for reflections (r = -.37; Fig. S13). Although Sec. 4.2 acknowledges this, the abstract and discussion still state that sensitivity broadly covaries with human alignment. This variability weakens that claim, and its scope should be narrowed accordingly in the abstract and discussion.
W4. Mismatch between human and model evaluation tasks. The human participants were explicitly instructed to attend to specific properties of the stimuli (App. B.2), making their task closer to targeted detection, whereas model responses are evaluated using whole-image denoising loss. This discrepancy complicates the interpretation of model–human alignment, as the model metric may be influenced by image regions unrelated to the manipulated property. Evaluating denoising loss within the manipulated regions, alongside the full-image metric, would provide a more controlled comparison.
Questions for authors
Q1. Could the authors clarify how the proposed paired-loss probing differs methodologically from LikePhys and YoCausal, beyond the comparison with human judgments? For example, are there design choices (e.g., content-preserving interventions, timestep and noise aggregation) that the authors consider essential and that were absent in prior work?
Q2. App. I and Fig. S13 report results within each condition, but the task-level analyses are not repeated on condition-centered scores. Do the main conclusions hold when the comparison with encoder and IQA baselines (Fig. 5) and the cross-model relation between sensitivity and alignment (Fig. 3) use condition-centered alignment?
Q3. What do the authors believe drives the negative sensitivity–alignment association for reflections? Is it concentrated in specific model families, or related to the models' response to hue differences noted in Sec. 4.1?
Q4. Do the model–human correlations hold when the loss difference is restricted to the manipulated regions, and how much signal comes from outside these regions? Since spatial loss maps are already available (Fig. 2, App. M), this analysis should be feasible without new model runs.
6 comments
Response to Reviewer 6zwW (2/4)
W2 / Q2: Pooled vs. within-condition alignment and condition-centered task analyses
We agree. In the revised Abstract and Section 4.2, we replace peak-correlation emphasis with the full 25-model ranges (Thatcher , Illumination ), remove "reliably track" phrasing, and explicitly note the remaining gap relative to human split-half reliability. Moreover, to separate between-condition mean differences from item-level agreement, Section 4.2 now reports condition-centered correlations directly alongside task-pooled metrics (all 25 models remain positive, with medians of on Thatcher and on Illumination, peaking at and ), complemented by full within-condition breakdowns in Appendices G–I (Figures S12–S14) and condition-centered task checks in Appendix S (Table S13).
We adopt this co-reporting strategy rather than dropping task-pooled alignment altogether because condition-level contrasts (such as orientation dependence in the Thatcher effect) are themselves a core part of the phenomena under study. Presenting pooled, condition-centered, and within-condition results side by side lets readers evaluate both condition-level structure and item-level ranking transparently.
With respect to the cast-shadow figure you cited ( vs. human reliability), we also note that the all-model median reflects wide spread across our 25-model panel ( to ), whereas the five top Illumination models reach a median of (and on condition-centered Illumination, peaking at ). Presenting both the full model range and top-five medians (Appendix I, Figure S14) clarifies that stronger generators show substantial within-condition agreement while still falling short of the human split-half reliability reference.
Condition-centered checks for main task-level conclusions (Q2): To test whether our main task-level conclusions hold after removing between-condition mean differences, we repeated both Figure 3 (baseline comparisons, formerly Figure 5) and Figure 4 (– coupling, formerly Figure 3) using condition-centered residuals (subtracting condition-specific means from human ratings and all model/baseline scores). Both conclusions remain supported:
-
Encoder and IQA baselines (Figure 3): Preserving the task-pooled selection procedure (repeating best-layer and top-five selection within each of 10,000 participant-and-scene bootstrap draws) and evaluating on condition-centered scores, generative losses retained positive partial correlations beyond encoders, FR-IQA, and NR-IQA across all six comparisons (all 95% CIs for median paired differences exclude zero):
Task & Baseline () Centered Centered Median Paired Diff. () Thatcher Vision encoders FR-IQA NR-IQA Illumination Vision encoders FR-IQA NR-IQA Reselecting encoder layers and top-five models directly on centered alignment yields consistent results—positive partial correlations beyond encoders ( Thatcher; Illumination) and continued superiority over FR/NR-IQA—except that paired generative-minus-encoder differences span zero ( and ).
-
Cross-model sensitivity–alignment relation (Figure 4): Using condition-centered alignment, cross-model correlations with remained positive under the joint family-, participant-, and scene-bootstrap: (Thatcher) and (Illumination), closely matching pooled correlations ( and ).
We summarize this in Section 4.2 and provide the full analysis in Appendix S (Table S13).
Response to Reviewer 6zwW (3/4)
W3 / Q3: Condition dependence of sensitivity–alignment coupling and reflection diagnostics
Thank you for highlighting this inconsistency. We agree that stating that sensitivity "broadly covaries" with human alignment overstates the cross-model relationship given its condition-level variability. As noted in General Response (G1), this condition-level dissociation, alongside schedule divergence and reflection scaling, is central to our main takeaway.
In the revision, we removed the "broadly covaries" framing from the Abstract, Introduction, Section 4.3 (formerly Section 4.2), and Discussion:
- Abstract & Introduction: We now state explicitly that while sensitivity and alignment associate positively across each full domain, cross-model coupling breaks down for specific violation types and decouples along denoising schedules.
- Section 4.3 & Figure 4C: We added a condition-resolved forest plot (Figure 4C) alongside the task-pooled scatters and report exact condition-level correlations and 95% joint family/participant/scene CIs in the main text (cast shadows: ; inverted faces: ; upright faces: ; light direction: ; surface reflections: ).
- Discussion: We replaced the opening summary with a unified discussion of sensitivity–alignment dissociation across conditions, model scale on reflections (Appendix K, Figure S22), and denoising schedules (Section 4.3; Figure 5, formerly Figure 4).
Drivers of the negative reflection association (Q3): Because the 95% CI spans zero (), we do not interpret as a reliable inverse law. New leave-one-family-out and factor-level diagnostics across the 96 reflection pairs (summarized in Appendix H) clarify what drives this point estimate:
- Model-family influence: Across the 11 model families, removing the three pixel-space JiT models (which combine low reflection sensitivity with relatively high alignment) shifts the correlation from to , whereas excluding any of the other 10 families yields to .
- Factor dependence and sensitivity balance: Excluding color manipulations moves the correlation only partway toward zero (), showing that hue insensitivity alone does not explain the trend. Evaluating alignment within each manipulation type (32 pairs each) shows that the relation with overall reflection sensitivity turns positive on
rotationtrials () while remaining negative onshape() andcolor(). Moreover, regressing out mean differences across the three reflection manipulation types from both human and model scores before computing alignment shifts the correlation to , and additionally removing the main effect of floor roughness shifts it to (95% CI ).
These diagnostics reinforce our central conclusion: average violation sensitivity cannot be treated as a general proxy for human alignment, as the coupling between the two depends on both the stimulus condition and how responses are distributed across manipulation factors.
Response to Reviewer 6zwW (1/4)
Thank you for your thorough review and concrete suggestions. We completed all requested analyses (condition-centered controls in W2, reflection diagnostics in W3, and inside/outside region evaluations in W4) and updated our claims, citations, and figures accordingly. Since Q1–Q4 map directly onto W1–W4 (thank you for organizing your review so clearly), we address each pair together below. Please also see our General Response for the distinction between Sm and rm (G1) and the revision summary (G4).
W1 / Q1: Technical contribution and methodological relation to prior work
Methodological comparison with LikePhys and YoCausal (W1, Q1): We agree that core probing mechanics (matched-noise sampling, timestep aggregation, and paired denoising contrasts) follow the same formulation as prior work (Section 3.2). Technically, our setup differs in two respects: retaining continuous loss differences (ΔL) instead of binary win/loss preferences, and using content-preserving factorial pairs rather than generated or unconstrained videos. As noted in our General Response (G1), these two choices suppress low-level confounds while spanning graded human tolerances, enabling us to measure violation sensitivity (Sm) and item-level human alignment (rm) independently on the exact same stimuli and uncover their dissociations across conditions, model scale, and denoising schedules (Figures 4–5, Appendix K). In addition, as highlighted in G3, our systematic factorial design enables a fine-grained analysis of human–model agreements and divergences across individual scene factors (Figure 1B, Sections 4.1 and 5, Appendix J, Figures S15–S16).
Comparison with Hepburn et al. [2]: Thank you for pointing out Hepburn et al. [2], which we now cite and discuss in Section 2.1. Although both works relate learned image statistics to human perception at a broad level, they address fundamentally different questions. Hepburn et al. [2] investigate how a reference image's probability modulates local sensitivity to perturbations, evaluating unsigned distances (induced by autoencoders) between reference and distorted images against human ratings of generic distortions. We instead test whether signed changes in native prediction losses directly track perceived naturalness under higher-level relational manipulations of facial configuration and scene illumination, and identify where stronger overall violation responses fail to translate into closer item-level agreement with human judgments.