acceptodds
Under review as a conference paper at ICLR 2027

Where the Examples Were: Local Loss Weighting Inherits the Spatial Layout of Scarce Supervision in Diffusion Models

Abstract

Local loss weighting emphasizes fine detail, but it applies to whatever the scarce detailed examples contain, including where they are. We test this in a controlled diffusion task where four texture classes, each in a cyan frame at one of four panel positions, share an identical low-resolution observation. Moving the same 32 highresolution photograph–detail pairs to another panel, without adding conditions or changing the budget, moves the extra frames that region-of-interest (ROI) weighting draws at unrequested panels. Their shift toward the new panel exceeds uniform weighting’s by +81.25, +70.31 and +98.44 percentage points in three SD1.5 training seeds, with paired intervals excluding zero; uniform weighting’s own shift is +1.56 in every seed. The effect is confined to this unrequested structure: the same contrast on joint class-and-location success crosses zero in all three seeds. Moving only the weighted region removes the extra frames instead of relocating them, in two seeds under each of two arrangements, while the layout code moves them consistently under neither. How the same examples are arranged also changes how much the weighting helps, in an interaction whose balanced half replicates. Our readouts are frame and classifier proxies rather than human judgments of texture, the target region is marked in the coarse observation, and native 4K is a boundary test.

Primary area:
generative models
Keywords:
diffusion models, local loss weighting, scarce supervision, spatial bias, controlled evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_vFfkOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates whether local loss weighting in diffusion models amplifies the spatial regularities of scarce high-resolution supervision. The authors construct framed texture targets whose four classes share an identical coarse observation, separating visible location from texture identity. Using LoRA adaptation with coarse conditioning, they compare uniform and region-of-interest weighting while reassigning the same photograph–detail pairs under a fixed training budget.

Critical strengths
  • Controlled observation construction. The zero-sum residual design makes coarse observations exactly identical across texture classes while preserving the location cue, enabling a focused study of detail learning without coarse-input class leakage.
  • Direct reassignment test. Moving unchanged photograph–detail pairs with matched budgets and random streams provides compelling evidence that unrequested frames follow supervision placement.
Critical weaknesses
  1. Experiments were conducted only on SD1.5, lacking results on more powerful or newer generative models.
  2. It does not read like a complete paper but rather a half-finished work. The paper’s insights into texture preferences are not distilled into a concrete method and applied to existing generative models; the experimental section contains only some ablation studies.
  3. The core hypothesis of the paper has limited value. The authors assume there exist positional preferences for textures in the data, causing models to overfit to those specific texture positions. However, this reads more like an intuitively likely assumption that does not require verification. Moreover, the scale of data used in the authors’ experiments is too small; overfitting at that scale is normal, whereas lack of overfitting would be the surprising result.
  4. Even if the paper’s hypothesis about texture positions holds, it would be difficult to formulate into a concrete method. Improving a generative model’s ability to produce textures sounds primarily like something that should be addressed during pretraining.
Questions for authors

Could the author explain the significance of the research question in this paper? What hypotheses were tested, what problems were addressed, and if the proposed hypotheses hold, what methods would result?

Reviewer_JSG6OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether spatially weighted losses cause diffusion models to learn the spatial distribution of scarce high-resolution training examples. The authors design a controlled texture-generation task in which different textures share identical low-resolution observations. They then relocate the high-resolution training examples to study the effects of ROI weighting. The results show that ROI weighting introduces extra frames at unrequested locations. These frames move with the training examples. The results also show that the effectiveness of ROI weighting depends on the spatial arrangement of the training examples.

Critical strengths
  1. The null-space residual construction produces different textures with identical low-resolution observations, allowing texture identity and spatial location to be studied separately.
  2. The authors preserve the 32 training pairs, optimization budget, initialization, and random sequences when relocating the examples, providing a relatively clean comparison of training-position effects.
  3. The experiments vary texture and frame weights and separately relocate ROI masks and layout codes, helping identify which components of the supervision contribute to the observed behavior.
Critical weaknesses
  1. The main experiments use 32 high-resolution training pairs, each visited 512 times. Since scarce supervision is central to the paper, experiments with different numbers of pairs (e.g., 8, 16, 32, 64, and 128) would help clarify how supervision size affects spatial inheritance.

  2. Section 3.2 introduces initial-spread and balanced-spread supports. Both assign eight conditions to each of the four positions, while the latter also balances content groups and detail classes. Section 6.1 reports improvements under both settings but lacks a direct paired comparison with confidence intervals. It remains unclear how much additional benefit content balancing provides beyond spatial coverage.

  3. Section 3.4 trains the fine generator on observed coarse latents but evaluates it on samples from a separate frozen coarse generator. Although all compared models use the same evaluation samples, this distribution shift may amplify the extra-frame effect.

Questions for authors

Please see Weaknesses.

Reviewer_pDSUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how region-of-interest (ROI) loss weighting affects spatial bias in diffusion models. Through controlled experiments, the authors show that locally weighted losses can learn the spatial positions of scarce high-resolution training examples, leading to unwanted structures at those positions. They also find that the spatial arrangement of training examples affects how much ROI weighting helps generation performance.

Critical strengths

It is interesting to analyze how local loss weighting can introduce positional bias in texture generation in a controlled manner.

Critical weaknesses
  1. The paper is difficult to follow, and the analysis's motivation and practical significance could be clarified further. Although the controlled experiments on texture demonstrate that local loss weighting can introduce positional shortcuts, the paper would benefit from a stronger discussion of the practical implications of these findings.
  2. The experiments mainly rely on relatively outdated diffusion models, such as SD1.5. It is unclear whether the findings generalize to more recent diffusion architectures, particularly diffusion transformers.
Questions for authors
  1. All experiments are conducted in a highly controlled setting in which textures are artificially inserted into fixed regions marked by cyan frames. Could authors discuss or provide evidence of whether this phenomenon also occurs in more realistic settings?
  2. Would training the texture classifier on VAE-reconstructed images, or combining original images with VAE-reconstructed images, improve its robustness to reconstruction artifacts?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.