acceptodds
Under review as a conference paper at ICLR 2027

Rare-Aware Autoencoding: Reconstructing Spatially Imbalanced Data

Abstract

Autoencoders are widely used for unsupervised and self-supervised learning when labels are scarce, but their reconstruction objectives can be biased by spatially imbalanced image data, favoring dominant background or frequent pixel patterns. This issue is common in scientific imaging including medical imaging, biology, physics, and astronomy, where relevant image content may appear sparsely or non-uniformly across spatial locations, and manual annotations are limited or expensive. We propose a reconstruction framework with two components: a self-information-based loss that upweights statistically uncommon pixel intensities conditioned on fixed spatial locations, and Sample Propagation, a replay strategy that re-exposes the model to hard-to-reconstruct samples across batches. Unlike most data-imbalance methods, which rely on task-specific annotations, our approach operates in the unsupervised setting. Nevertheless, we also adapt and benchmark methods from supervised learning. Experiments on a controlled simulated dataset and three real-world datasets spanning physical, biological, and astronomical domains show improved reconstruction over existing baselines, particularly in settings with strong spatial imbalance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
autoencoder, imbalance, physics, science, spatial-imbalance, reconstruction, representation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jpBCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies autoencoder reconstruction under spatial data imbalance. Here, rare intensities at fixed pixel locations get averaged out, causing blur. It proposes two components. The first is a self-information (SI) loss: MSE plus an L1 term weighted by batch level, per location surprisal. The second is Sample Propagation (SPP), which carries the hardest samples into the next batch. Experiments cover a synthetic pendulum, three real datasets and four autoencoder families. The paper reports lower MSE and higher PSNR, with competitive SSIM.

Critical strengths
  1. The paper frames location wise intensity imbalance as a distinct failure mode of autoencoders. This matters for scientific imaging, where content is sparse and labels are scarce.
  2. Neither component changes the architecture or inference cost. Both are tested on four autoencoder families and three real datasets.
Critical weaknesses
  1. The key ablation is missing. The SI loss is MSE with a L1 term. L1 alone reduces blur. Yet the paper never compares against uniformly weighted L1. It also omits L1 weighted by a simple proxy, such as variance or a foreground mask. So the gains cannot be attributed to the self-information weighting.
  2. The comparison may not be fair. Some baselines seem to fail. With MAE, VGG and Watson reach SSIM of only 0.05–0.11. CE reaches only 5–8 dB PSNR. Section 4 says hyperparameter ranges are in the appendix, but none are given. J and the baseline settings are also missing.
  3. The main text and appendix often disagree. Such as the captions of Tables 13 14 15 use a different memory definition from the main text. The Appendix part does not appear to have been carefully checked before submission. Such as several bold marks in Table 13 highlight worse results, not the best ones.
Questions for authors

Questions that could change my assessment: How does MSE + λ·L1 with uniform weights perform? How does variance or mask weighted L1 perform? Please report search ranges, selected values and J for all methods. Did baselines get the same tuning budget? Background pixels get weight α = 0.05 in the SI term. Does this cause false positives in the background? Please report background error separately. Clarifications: How do results change with a dataset level histogram? How do they change with B = 16 or 256? The main text defines M = B/k, so SPP2 has the largest memory. Tables 13 14 15 instead describe “memory of size k.” Which is correct?

1 comment
AuthorsOpenReview ↗

Response to Reviewer jpBC

We thank the reviewer for recognising spatial imbalance as a distinct failure mode and the architecture-agnostic design. Page (p.) and line (l.) numbers refer to the revised PDF. The new experiments are in Appendix B (pp. 27–33).

W1. Missing ablation: uniformly weighted L1, and L1 weighted by a simple proxy. We agree that this ablation was missing. We added the requested controls with the same code path, sample weighting, λ and training settings as SI; only the per-pixel L1 weight changes (Appendix B.1, Table 16, p. 27): (i) MSE + a·L1 with a equal to the mean SI weight, so the total L1 weight matches; (ii) the per-location batch variance; (iii) a foreground mask. SI beats the mask on every block. Against MSE + a·L1, over three seeds (Table 21, p. 31), SI is better on Galaxy and Pendulum in every seed and ties on Mitosis on average metrics. These metrics are dominated by the background; on the rare foreground content, SI lowers the error by 5–36% against the constant weight (Table 17, p. 27). So where the gain appears, it comes from where the weight is placed, not from adding L1. Variance is the strongest proxy but measures something else: it weights a location in every image, while SI weights the value in the current image. A star that appears in one image sits at a near-zero-variance location yet is the rarest value there (Appendix B.2, Fig. 15, p. 29); this is why SI is better on Galaxy and Mitosis. Qualitatively, SI recovers isolated stars that the control smooths away, while on Mitosis both are similar (Figs. 13–14, p. 28). A λ sweep on all 12 settings is in Table 18 (p. 30).

W2. Fairness of the comparison: failing baselines and tuning budget. We partly agree: the collapses are known failure modes of these losses, but the baselines deserved the same tuning budget, which we now give them. CE assumes Bernoulli targets and collapses on mostly dark images with continuous intensities (Galaxy, and MAE on all datasets). VGG and Watson are perceptual and do not constrain absolute intensity, so alone with an MAE they reproduce structure poorly. These are known failure modes, not under-training; all methods share the architecture, optimiser, epochs and batch size. To rule out a tuning artefact, we gave the baselines the same budget as SI (four values of their main hyperparameter; Appendix B.4, Table 22, p. 31): focal γ, LDAM margin, equalization threshold, and CE, VGG and Watson combined with L2 on the MAE. Adding L2 removes their collapse, but the best tuned baseline stays below SI on every MAE block.

W2 (cont.). Hyperparameter ranges, J and baseline settings are missing. We agree and apologise. We use J = 150, α = 0.05 and the λ per architecture in Table 8 (p. 19), now stated in Appendix A.3 (p. 15, l. 798–801). Table 22 (p. 31) lists the search grid and best configuration of every tuned method; all use learning rate 10⁻³, 1000 epochs and B = 128. We also added an ablation over J (see Questions).

W3. Disagreement between main text and appendix; wrong bold marks. We agree; these were errors in the appendix, now fixed. The main-text definition is correct: sppk propagates M = B/k samples, so spp2 has the largest memory; the captions of Tables 13–15 (pp. 24–26) are corrected. All appendix tables are now generated by a Python script from a single results file, with bold computed rather than marked by hand, which fixes the Table 13 errors.

Questions. Uniform and variance- or mask-weighted L1: see W1. Search ranges, selected values, J and tuning budget: see W2. We also added an ablation over J ∈ {16, 64, 150, 255} (Table 19, Fig. 16, p. 30): J = 16 is too coarse and 64–255 differ by at most 0.3 dB on Galaxy and Mitosis. Does the α = 0.05 floor cause false positives in the background? We respectfully disagree, and added a measurement to check it: background pixels keep the full MSE term plus at least α of the L1 term, so their error is still penalised; only their relative weight is lower. Table 17 (Appendix B.1, p. 27) splits the error into background and foreground: SI's background error rises by 19–30% relative to the uniform control but stays of the order of 10⁻⁴, while its foreground error drops by 5–36%. A foreground mask lowers foreground error further but raises background error 2.8–24× relative to SI, the failure the reviewer describes.

Reviewer_rnVLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies spatial data imbalance in unsupervised autoencoder reconstruction, where particular observations at fixed spatial locations are underrepresented. It proposes two components. First, the self-information loss (SI) estimates batch-level intensity distributions separately at each location and channel, applies per-image, per-channel normalized rarity weights to an L1 reconstruction term, and combines this term with MSE. Second, Sample Propagation (SPP) combines the current batch with previously retained high-loss samples, selects high-loss samples for training, and carries a subset forward to the next step. Both components modify the reconstruction objective and sample exposure without requiring additional task-specific annotations or architectural changes.

Critical strengths
  1. This paper clearly identifies the reconstruction errors that arise from latent representations in models such as AE and VAE. In particular, the observation that certain patterns occur rarely at specific spatial locations within the data (images) provides a useful perspective that is distinct from class imbalance.

  2. I believe the proposed components make clear contributions to addressing the problem. Specifically, the proposed SI adjusts the relative contribution of observations at different spatial locations to the loss, while SPP improves performance by increasing repeated exposure to images that are difficult to reconstruct. These components can be applied to various methods.

  3. The experiments include various autoencoder-based models, reconstruction losses, and sample selection methods. Although I believe some additional ablation studies are needed, the main results are encouraging.

Critical weaknesses
  1. The proposed approach appears to be somewhat heuristic and incremental in nature. The specific contributions of the method should be explained in the context of various datasets.

  2. Since the proposed SI method is estimated based on batches and histogram levels (J,BJ, B), further discussion on the configurations of JJ and BB is needed.

  3. As I understand it, SPP involves repeatedly training on difficult samples. Although this is a simple approach, I think it may limit learning from new samples. Could this lead to bias?

  4. I think assigning higher weights to observations that occur rarely at specific spatial locations may introduce another form of bias. I wonder whether this approach would also be applicable to data with complex structures, such as medical images.

  5. As I understand it, the paper computes histograms independently for each RGB channel. However, I believe the representative datasets are grayscale. While I think the proposed method could be effective for grayscale images, independently processing RGB channels may cause problems in capturing their joint combinations.

Questions for authors

Could the authors address the concerns raised in the Critical Weaknesses section with clarifications and supporting evidence?

2 comments
AuthorsOpenReview ↗

Response to Reviewer rnVL 1/2

We thank the reviewer for the careful reading and for recognising the problem formulation and the diagnostic analyses. Page (p.) and line (l.) numbers refer to the revised PDF, and the new experiments are in Appendix B (pp. 27–33).

W1. Heuristic and incremental; contributions in the context of various datasets. We partially agree that the contribution should be shown across datasets, however we consider the papers is: Not incremental. Spatial imbalance is a different problem from the class or sample imbalance that existing methods target, as the reviewer also notes (App. A.1, p. 14): an object can appear in every image while its appearance at a given location is rare. Existing methods therefore do not solve it: over the 12 dataset–autoencoder settings (Tables 10–15), none of focal, LDAM, ONBS, DCL or SVP improves on plain L2 on average, and tuned with the same budget as SI, focal and LDAM stay below it in 5 of 6 settings (Table 22). SI improves on L2 in all 12 settings (+4.5 dB on average) and SPP8 in all 12 (+3.1 dB). If our method were an increment over these, they would recover part of this gain; they do not. Not heuristic. Each choice follows from the problem: rarity is the self-information −log p̂ℓ(x) of a value at its location, the standard measure of how unexpected an observation is; it is estimated per location because the imbalance is per location, and per batch so that it reflects the current update; and it counters the MSE pull toward the frequent value (App. A.2, p. 15, l. 763–771). SPP follows from the same analysis at the sample level: a rare sample appears in few batches, so its correction is overwritten until it returns. The matched controls test the alternatives a heuristic would choose (Table 16, p. 27): only SI lowers the error on rare content in all three datasets without a large cost on the rest (Table 17; Figs. 13–14, 17–18). Our datasets cover distinct sources of spatial imbalance: a static background with rarely visited positions (Pendulum), centred objects with rare off-centre content (Galaxy Zoo) and rare events in time-lapse microscopy (Mitosis). We added two high-resolution datasets: sparse biological structures (fluorescent neurons) and rare defects on aligned objects (MVTec AD; App. B.5, pp. 32–33). In short: methods designed and tuned for imbalance do not improve on plain L2 when the imbalance is spatial, whereas SI and SPP improve on it in every one of the 12 settings, and SI also on both new high-resolution datasets. This gap is the contribution, not an increment.

W2. Configurations of B and J. We agree. We added sweeps of both (App. B.3, Tables 19–20 and Fig. 16, pp. 29–30). J = 16 is too coarse; from J = 64 to 255, PSNR changes by at most 0.3 dB on Galaxy and Mitosis (0.8 dB on Pendulum), so we use J = 150, inside this plateau. SI works for B from 16 to 256; smaller batches help on Mitosis. We use the batch size shared by all methods, so that every method sees the same batches. We also added the values (App. A.3, p. 15, l. 798–801) and a discussion of how B and J set the resolution of the counts (App. A.2, p. 15, l. 782–787).

AuthorsOpenReview ↗

Response to Reviewer rnVL 2/2

W3. Does SPP limit learning from new samples or introduce bias? We partly agree: too large a memory can reduce batch diversity, but SPP does not stop learning from new samples. SPP does not replace the new batch: the B samples of the update are the hardest of the current batch plus at most M = B/k propagated ones, so at least B − M (e.g., 75% for SPP4) of every update comes from new data, and each sample is still seen every epoch. A sample stays in the memory only while its error is high and leaves once it is reconstructed well. Empirically, SPP reduces the error gap between frequent and infrequent samples (Fig. 8b, p. 7, l. 341–349) and still improves overall MSE and PSNR, with SPP8 beating L2 in all 12 settings (Tables 13–15): the emphasis on hard samples does not come at the expense of the rest. We clarified the selection rule in Sec. 3 (p. 4, l. 215 to p. 5, l. 227).

W4. Could upweighting rare observations introduce another bias? Complex data such as medical images? We agree that this needed checking. Every pixel keeps the full MSE term plus at least a floor α of the L1 weight, so frequent content is still fitted. We added Table 17 (p. 27) to measure this: background error rises by 19–30% relative to a constant weight but stays of the order of 10⁻⁴, while foreground error drops by 5–36%; a foreground mask instead raises background error 2.8–24×. We note in the Discussion that rare noise or artifacts may also be emphasised (p. 9, l. 476–478) and that the method assumes aligned images (p. 9, l. 483–484). For complex biomedical data, Mitosis is cell microscopy and the new fluorescent neuron images contain sparse structures; Relative to L2, SI lowers the error on neurons by 11% and on defects by 35–46% while also lowering the background error, with the lowest error on neurons and defects in every seed (Tables 23–24). Rare content is recovered without degrading the rest, which is the opposite of a new bias.

W5. Independent RGB channels. We agree that this is a limitation. Pendulum and Galaxy Zoo are RGB (Mitosis is grayscale), as are the new neuron and MVTec datasets. A joint RGB histogram needs J³ bins per location (3.4 × 10⁶ at J = 150), which cannot be estimated from B samples. An unusual colour usually has an unusual value in at least one channel, which receives a high weight; only colours made entirely of individually common channel values are missed. To clarify it, we now state this limitation in Sec. 5 (p. 9, l. 483–484) and App. A.2 (p. 15, l. 770–771).

Summary. The reviewer noted that some additional ablation studies were needed. We added matched controls for the weight (App. B.1), its comparison with variance weighting (B.2), sensitivity to λ, J, B and seeds (B.3), baselines tuned with the same budget (B.4) and two high-resolution datasets (B.5). We hope these address the reviewer's concerns and would be grateful if the reviewer would reconsider the score.

Reviewer_5HSeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of reducing the reconstruction error of autoencoders in low resource fields, focusing on natural science domain. “spatial data imbalance” is introduced as a problem when image content is spatially translated across the datasets. A variant of class-reweighting method called self-information (SI) reconstruction loss is proposed, coupled with sample propagation (SPP), to increase diversity in the autoencoder reconstruction. Three natural science domains dataset is used in the evaluation against comparative methods, demonstrating mostly improved performance. Several downstream are also presented, demonstrating increases in performance in generation, anomaly detection and parameter estimation tasks.

Critical strengths
  1. The proposed method combing SI and SSP is interesting and works well for the proposed domain.

  2. The paper presents thorough evaluation of the method against the three datasets and varying downstream tasks. The paper applied domain specific analysis to some datasets (searching for temporal oscillations of intensity values in pendulum dataset), visualisations of reconstruction error and evidence of SPP’s impact on dataset selection. This makes it clear to the reader the impact of their method on the autoencoder’s learned distribution. Outside of this, the paper also demonstrated considerable improvement in performance on the downstream tasks.

Critical weaknesses
  1. The paper presents weak analysis of the mathematics of SI loss and how it addresses the proposed problem. Essentially, every pixel location is being normalised across each batch and discretised into a fixed number of bins. The log of the size of each bin proportionate to the entire distribution becomes the weighting for that training example. It is not clear that non- spatially aligned images would benefit from emphasising specific pixel locations in the training phase.

  2. There is little justification / discussion on the proposed method’s performance (refer to questions below)

  3. The paper can improve in its presentation e.g., although SPP and SI methods are prominently shown in figure 9, the other methods are impossible to distinguish from each other; clarify that k is a bin index in your method; Line 127: similar -> similarly

Questions for authors
  1. Why did you only evaluate the downstream tasks against (mostly) a single baseline model? Could you describe the decision process on which baseline models to compare against?

  2. Why does emphasising loss on pixels with rarer fluctuations improve the quality of generation and reduce blurriness, compared to training uniformly on each pixel?

  3. Does your justification of SI loss make any assumptions about the distribution of training examples?

  4. What makes SPP better than existing hard example detection algorithms?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 5HSe 1/2

We thank the reviewer for finding the method interesting and the evaluation thorough. Page (p.) and line (l.) numbers refer to the revised PDF.

W1. Mathematics of the SI loss and how it addresses the problem. We agree that this analysis was missing. Under MSE, the optimal reconstruction at a location is the conditional mean, which is pulled toward the frequent intensity, so rare intensities are blurred away. This blur comes from the reconstruction; SI does not remove blur already present in the input. SI adds an L1 term weighted by −log p̂ℓ(x), so pixels with a rare value at their location get a larger gradient and move toward that value instead of the average. Constants (1/B, 1/δ) cancel in the per-image min–max normalization. We added this to Appendix A.2 ("Reconstruction blur", p. 15, l. 763–771).

W1b. Non-spatially aligned images. We agree that this assumption should be explicit. SI assumes a shared coordinate frame, which holds for all our datasets and is common in scientific imaging. Without alignment, the per-location histograms flatten and SI reduces to a uniformly weighted MSE + λ·L1: it degrades gracefully rather than failing. We now state it in the limitations (Sec. 5, p. 9, l. 483–484) and in Appendix A.2 (p. 15, l. 769–770).

W2. Little discussion of performance. We agree. We expanded the discussion (Sec. 4.2, p. 7, l. 372–373): gains are largest where background or repeated states dominate a location (Pendulum, Galaxy) and smallest where intensities are already diverse (Mitosis). This pattern is what the mechanism predicts: SI acts on values that are rare at their location, and these datasets have more of them. Matched controls confirm that the weight, not the added L1 term, drives the gain where it appears: SI beats the same loss with a constant weight on Pendulum and Galaxy and ties on Mitosis (Table 16, p. 27), and against it lowers the error on rare foreground content by 5–36% (Table 17, p. 27). We also note in the Discussion that average metrics, dominated by the background, can hide erased rare structures and should be complemented by a qualitative evaluation (Sec. 5, p. 9, l. 473–476).

W3. Presentation. We agree. We replaced the radar plots by a figure that shows every method individually, with the best one framed (Fig. 9, p. 8). We clarified that k_{b,ℓ} is the bin index (Sec. 3, p. 4, l. 184) and fixed "similar → similarly".

AuthorsOpenReview ↗

Response to Reviewer 5HSe 2/2

Q1. Why a single baseline per downstream task, and how were baselines chosen? We agree that one baseline per task is limited evidence; Sec. 4.3 therefore calls these results indicative, not conclusive (p. 9, l. 457). For each task we followed the protocol of the method that introduced it (BEL for generation, AESc for anomaly detection, Delfys for parameter estimation) and compared against the strongest baseline reported there, to test whether our objective still adds a gain. These experiments support, rather than replace, the reconstruction results with 13 baselines.

Q2. Why does emphasising rare pixels reduce blur compared with uniform weighting? We agree; we added this explanation to Appendix A.2 (see W1). When a location is usually background but sometimes shows an object, a uniformly weighted loss is minimised by predicting close to the background value, because the object barely affects the total error; the object comes out faint and blurred. SI gives the object pixels a larger weight, so the decoder is pushed to reconstruct them sharply. On the samples hardest for MSE + a·L1, SI recovers isolated stars that the uniform loss smooths away (Fig. 13, p. 28). On the two new high-resolution datasets, L2 and the uniform weight erase neurons and capsule defects that SI recovers (Appendix B.5, Figs. 17–18, pp. 32–33).

Q3. Assumptions about the distribution of training examples. Yes, and we now state them explicitly. Two assumptions: images share a spatial frame, and a mini-batch is representative of the training set (Appendix A.2, p. 15, l. 769). We assume no particular form (e.g., Gaussian) for the intensity distribution at a location: the histogram estimates it from the batch counts, so multi-modal cases, such as a location that is usually background and occasionally shows the pendulum, are handled naturally.

Q4. What makes SPP better than existing hard-example methods? We agree that SPP is a hard-example method; it differs in how long a hard sample is kept. Existing hard-example methods (OHEM, focal weighting, ONBS) act within the current batch. Under strong imbalance, a rare sample may not reappear for many steps, so its correction is overwritten by batches of frequent samples. SPP carries the hardest samples into the next batches until they are reconstructed well, without a proxy model (SVP), a fixed schedule (DCL) or a larger gradient batch. Methods that keep up-weighting only the hardest samples can also over-focus on a few of them, which acts like fine-tuning at the expense of the frequent content. In SPP, a sample stays in the memory only while its loss is high, and the memory is a fraction of the batch, so frequent samples keep contributing to every update. We clarified the selection rule in Sec. 3 (p. 4, l. 215 to p. 5, l. 227) and compare against focal loss, ONBS, DCL and SVP in Tables 13–15 (pp. 24–26).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.