acceptodds
Under review as a conference paper at ICLR 2027

Bootstrap Your Own Noise: Denoising and Latent Prediction Cooperate Only When Noise Is Informative and Alignment Is Gated

Abstract

Reconstruction and latent prediction teach a vision encoder different things. Reconstruction from masked or noised inputs yields features that attend locally. Predicting the embedding of a clean view yields features that attend globally, so combining the two is a natural goal. We show that the combination is not additive. We pre-train a ViT-B for 400 epochs on ImageNet-1K. Adding Gaussian latent noise to masked image modeling leaves fine-tuning accuracy unchanged (82.86% against 82.89%), and adding a bootstrapped latent-prediction loss on top lowers it to 82.01% and costs 4.6 mIoU on ADE20K. We trace both failures to a single cause and remove it with two small changes. Image-adaptive noise sets the noise level of each image from its token saliency, and a reliability gate scales the latent-prediction loss by the current reconstruction error. The gate alone returns accuracy to 82.84%, adaptive noise alone reaches 83.16%, and the two together reach 83.56%. In other words, the latent-prediction loss helps only after the noise carries information about the image and the target it aligns to can be trusted. Under the same budget, the resulting model, Bootstrap Your Own Noise (BYON), improves over masked and denoising pre-training by 0.7 points on ImageNet-1K after fine-tuning and by 2.2 under a linear probe, by 1.4 mIoU on ADE20K, and by 1.7 box AP on COCO. It gains 0.8 to 3.5 points on six fine-grained benchmarks, and its margin over MAE does not shrink at 800 epochs or with ViT-L. Our evidence is empirical and covers models up to ViT-L trained on ImageNet-1K; code is provided in the supplementary material.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
self-supervised learning, masked image modeling, joint-embedding predictive architecture, denoising autoencoders, latent noise injection, bootstrapping, reconstruction versus latent prediction, implicit bias of reconstruction, vision transformers, representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9GE5OpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the problem of joining the features gained by reconstruction and prediction, which offer qualitatively different representations. The solution is to supplement the training objective with an image-adaptive noise and a reliability gate. The former is to address the shortcomings of a uniform noise used in other research, while the latter is to control the degree of prediction loss, acting as a curriculum. This proposed change results in consistent improvement of image-based prediction task performances over several related baselines.

Critical strengths
  • I think this paper identifies one of the fundamental questions that representation learning tries to answer: Is a representation good for reconstruction also good for prediction? It also goes on to clarify how existing approaches are not appropriate theoretically and explains why the proposed method is (somewhat) justified by the theoretical analysis as well. Furthermore, the experiments back up the claims by showing consistent increase in performance.
Critical weaknesses
  • (Not really critical) There are some minor notational issues in both propositions that make it hard for the reader to appreciate their importance. See questions below.
Questions for authors
  • Is there a reason why the authors focus on vision encoder instead of a general-purpose encoder?
  • Could the authors explain the following notations and concepts used in the propositions?
    • W(0): W is a map, but it's treated as a matrix.
    • What does τ\tau do?
    • How can the significance of these propositions extend beyond the linear setting?
  • The paper talks how an image-dependent noise is necessary to achieve the intended results, but will this be related to general augmentation techniques? Because many augmentation techniques can be thought of as being input-dependent perturbation, and it might bear some relevance to the main discussion.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 9GE5

Numbering follows the review.

W1 / Q2. Notation in both propositions. Fixed: we have revised Sec 3.2 so that every symbol is defined where it first appears.

SymbolMeaning, as we have now stated it in Sec 3.2
WW, W(0)W(0)WW is a d×dd\times d matrix, the weight of the linear restorer x^=Wx~\hat x=W\tilde x; "map" and "matrix" are the same object. W(0)=0W(0)=0 is the zero-matrix initialisation.
τ\tauTraining time of gradient flow, the continuous-time limit of gradient descent. W(τ)W(\tau) is the restorer after training for time τ\tau.
gig_i, sis_iIn W(τ)=Vdiag(gi(1−e−siτ))V⊤W(\tau)=V\mathrm{diag}\big(g_i(1-e^{-s_i\tau})\big)V^\top: gig_i is the value direction ii converges to, sis_i how fast; 1−e−siτ1-e^{-s_i\tau} is the fraction learned by time τ\tau.
Σω\Sigma_\omega, bˉ2\bar b^2Σω=E[ω(x) xx⊤]\Sigma_\omega=\mathbb{E}[\omega(x)\,xx^\top] is the data covariance with each image weighted by ω(x)\omega(x); we use ω=a\omega=a and ω=a2\omega=a^2. bˉ2=E[b(x)2]\bar b^2=\mathbb{E}[b(x)^2] is the mean noise power.

What each proposition says, in words. Proposition 1: with a noise level that ignores the image, directions of larger variance are learned faster and more completely, whatever the level. Noise of this kind cannot change what reconstruction prefers. Proposition 2: when the level depends on the image, reconstruction no longer sees Σ\Sigma but a covariance in which each image counts according to its signal rate, and that covariance can have a different order of directions. We have added these two sentences after the propositions.

Q1. Why a vision encoder and not a general-purpose encoder? Because our evidence is visual; the mechanism is not specific to vision. IAN needs only a token sequence: the saliency of Eq. (5) is computed from the affinity of token embeddings (L253–255). Our experiments cover ViT-B/L on ImageNet-1K and one vision-language model (CLIP, 83.35 → 85.28; App. B, L866–871), so we limit the claims to vision, as the abstract states (L030–031).

Q3. How can the significance of the propositions extend beyond the linear setting? As a prediction that we test in a ViT-B. The prediction: a noise level that ignores the image leaves the preference of reconstruction unchanged; only a level that depends on the image can change it.

Prediction of the linear analysisTest in a ViT-BResult
Uniform noise changes nothing (Prop. 1)Table 4, rows N vs. U82.89 vs. 82.86; CKA-top 0.423 vs. 0.420 (Table 9)
Image-dependent levels can change it (Prop. 2)rows U vs. ACKA-top 0.420 → 0.376; RankMe 37 → 54
Random levels independent of the image leave it unchanged (new Corollary 1)new shuffled-level control (Table 4b)CKA-top 0.407 vs. 0.417 (uniform) and 0.272 (BYON); IN-1K 82.94 vs. 82.84 and 83.56

The third row is the most direct answer: same noise levels, same distribution, only their link to the image removed. This control was requested by Reviewer m4GD; the full table is in our reply to that review. A derivation for nonlinear encoders remains open (Sec 5, L471–473).

Q4. Relation to general augmentation techniques, many of which are input-dependent perturbations. Yes, it is related, and the propositions add one distinction. The reviewer is right that the output of any augmentation depends on the input. What matters in Proposition 2 is whether its strength is chosen from the input.

  • Standard augmentations draw their strength independently of the image (crop scale, colour-jitter magnitude, a uniformly sampled noise level): the regime of Proposition 1 and Corollary 1.
  • Content-adaptive corruptions such as attention-guided or adversarial masking (L178–185) decide which tokens to remove from the image; IAN decides how strongly to perturb, and Table 6 (L838–842) shows the two are complementary.
  • Our proofs cover additive noise with scaling, so we have added this as a discussion paragraph in Sec 2 and make no claim for augmentations in general.
Reviewer_m4GDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies when reconstruction and bootstrapped latent prediction cooperate in visual self-supervised learning. The authors find that directly combining the objectives can hurt performance and propose image-adaptive noise (IAN) and a reconstruction-error-based gate (SUE). A linear analysis shows that content-dependent noise can change the learning preferences of reconstruction. Experiments report improvements on classification, segmentation, and detection, with ablations suggesting that the two components work better together.

Critical strengths
  1. The paper addresses a meaningful question: combining two useful self-supervised objectives does not necessarily improve representations. The negative results and component comparisons are informative.
  2. The full method improves performance across several downstream tasks, and gains persist with longer training and a larger backbone. Several main comparisons include five-seed results, supporting the effectiveness of the overall approach.
Critical weaknesses
  1. The experiments do not isolate whether the gains come from matching noise levels to image content. IAN changes both the noise-level distribution and its assignment to images. Without a distribution-matched baseline or shuffled assignments, the gains could reflect a better corruption-strength distribution rather than the content dependence central to the paper’s explanation.
  2. The theory shows that content-dependent noise can change learning preferences, but does not establish why the proposed saliency rule improves semantics or resolves objective conflict.
Questions for authors

Does IAN retain its gains if its noise levels are randomly reassigned across images within each batch, preserving the noise-level distribution? Does the benefit of IAN come from its dependence on image content? I also recommend further elaborations in the linking of the theory part with the saliency rule.

2 comments
AuthorsOpenReview ↗

Response to Reviewer m4GD

We have run the control exactly as the reviewer specified it (Table 4b of the revision).

W1/Q1. Are the gains retained when IAN's levels are reassigned at random within each batch? Is the benefit due to content dependence?

The benefit is due to content dependence. With IAN's levels reassigned at random, 86% of the gain on ImageNet-1K and 75% on ADE20K disappears; only 14% and 25% remain.

Protocol. t(x)t(x) is computed by Eq. (5) and then permuted within the batch, so every batch contains exactly IAN's levels: a shuffled assignment and a distribution-matched baseline at once. Nothing else changes. The mean signal rate E[αˉt]\mathbb{E}[\bar\alpha_t] is 0.481 for IAN and its shuffle and 0.458 for uniform noise, so overall noise strength is not the explanation.

Table R1 (new entries: mean of 3 seeds; BYON's ADE20K is a single run, five-seed mean 44.46 ± 0.21).

RowLevelsAssignmentLatentIN-1K FTLinearADE20KCKA-top ↓
Uuniformrandomnone82.8662.243.020.420
A-shufIANrandomnone82.8962.443.370.414
AIANby imagenone83.1663.243.660.376
U+gLuniformrandomgated82.8462.142.850.417
A-shuf+gLIANrandomgated82.94 ± 0.1562.643.310.407
A+gL (BYON)IANby imagegated83.56 ± 0.1664.544.690.272
  • This is the right test, and the claim holds under it: rows U and A of Table 4 differ in both distribution and assignment; Table R1 separates the two.
  • In the full model, content dependence is the larger share on every metric. Content dependence, (A+gL) − (A-shuf+gL): +0.62 / +1.9 / +1.38 on IN-1K / linear / ADE20K. Distribution alone, (A-shuf+gL) − (U+gL): +0.10 / +0.5 / +0.46. Without the latent loss the two are comparable on ADE20K (+0.35 vs. +0.29). The caption of Table 4b states this.
  • The latent loss needs the dependence: it is worth +0.05 / +0.2 / −0.06 with shuffled levels (neutral, as under uniform noise) and +0.40 / +1.3 / +1.03 with image-dependent levels. This is claim (ii) of the paper, now with the distribution fixed.
  • Same mechanism: with shuffled levels the features stay aligned with the high-variance pixel subspace (CKA-top 0.407; uniform 0.420; BYON 0.272).

W2. Why does the saliency rule (a) improve semantics and (b) resolve the objective conflict? Elaborate the link to the theory.

We have made the link explicit in three steps, with a new corollary, a table and a paragraph after Eq. (5). The propositions give the condition under which noise can change what reconstruction prefers (a necessary condition, as stated in L246–249); the steps show that the saliency rule meets it.

  1. The linear analysis predicts the outcome of the shuffled control (new Corollary 1). If (a,b)(a,b) is random but independent of xx, then Σa2=E[a2]Σ\Sigma_{a^2}=\mathbb{E}[a^2]\Sigma and Σa=E[a]Σ\Sigma_a=\mathbb{E}[a]\Sigma, so W⋆W^\star keeps the eigenvectors of Σ\Sigma, with gains E[a]λi/si\mathbb{E}[a]\lambda_i/s_i and rates si=E[a2]λi+bˉ2s_i=\mathbb{E}[a^2]\lambda_i+\bar b^2 that increase with λi\lambda_i as in Proposition 1 (independence factorises the moments of Eq. 4; App. A). Proposition 1 covered constant levels only. Only a covariation of level and image can change the order of learning, and Table R1 is consistent with this.

Table R2 (Table 10). (a) Moments of Proposition 2 on ImageNet-1K, 64×6464\times64 pixels; PtopP_{\rm top} spans 90% of pixel variance. (b) Cosine between the gradients of the reconstruction loss and the latent loss, blocks 9-12, epoch 200.

UniformIAN-ShuffleIAN
(a) Trace share of Σa2\Sigma_{a^2} in PtopP_{\rm top}0.900.900.82
(a) Spearman of rate order vs. Σ\Sigma1.001.000.90
(a) corr of ρ(x)\rho(x) with energy share in PtopP_{\rm top}–0+0.45
(b) Gradient cosine, gated−0.07−0.02+0.17
(b) Gradient cosine, ungated−0.19–−0.04
  1. (a) Semantics: IAN moves restoration away from the high-variance subspace. Restoration is governed by Σa2\Sigma_{a^2} (Prop. 2). ρ(x)\rho(x) correlates with an image's energy share in that subspace (r=0.45r=0.45), so IAN noises such images more and Σa2\Sigma_{a^2} carries 9% less trace there, on directions that carry little information for perception; with shuffled levels nothing changes. The features follow: CKA-top 0.420 → 0.376 → 0.272.
  2. (b) Objective conflict: the two gradients agree only under IAN. Their cosine is +0.17 under IAN, against −0.07 and −0.02 under image-independent levels (−0.19 ungated). The submission shows the consequence (Fig. 3b, Fig. 2): ungated under uniform noise, branch agreement falls to 0.11 within 20 epochs and the features collapse to rank 11; with gate and IAN it reaches 0.86 and rank 63.

In short, the requested control attributes most of the gain to content dependence, as the paper claims, and the linear analysis predicts this outcome. We are glad to clarify any remaining point.

AuthorsOpenReview ↗

Follow-up to Reviewer m4GD

Thank you again for your review. Your question about shuffling the noise levels was the experiment our paper was missing, and we are grateful you asked for it.

We know this is a busy week, so here is the short version of what we did:

  1. Shuffled-assignment control (W1/Q1). We ran it exactly as you described (Table 4b). With the distribution fixed and only the assignment shuffled, 75 to 86% of the gain disappears.
  2. Theory and the saliency rule (W2). We added Corollary 1, Table 10 and a short paragraph after Eq. (5).

We would really value your view on whether this resolves your concerns. If anything is still unclear, we will answer right away, before the discussion closes. Thank you very much for your time.

Reviewer_AXPxOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Problem: Restoring missing image parts and predicting the features of a clean image could teach a vision model complementary things, but combining them naively can make performance worse. Adding uniform random noise to the restoration task did not improve the model, and adding feature prediction on top reduced performance. The noise was unrelated to image content, while the model was still too poor at reconstructing corrupted images for its feature predictions to be reliable. The prediction task then competed with, rather than helped, reconstruction.

Authors' idea: The authors hypothesize that the two tasks can complement each other if the added noises are adaptive and the high-level prediction task is weighted according to how well the model can currently reconstruct the image.

How BYON works: BYON masks parts of an image and trains the model to restore them. It also adds noise to the image's internal representation, choosing the noise level from an attention-based estimate of how widely important content is spread across the image. A second task asks the model to predict a high-level representation of a clean view. BYON gives this task little weight when reconstruction error is high, then increases its weight as reconstruction improves.

Why it matters: The work suggests that reconstruction and high-level prediction need not compete. Choosing noise based on image content and delaying the extra prediction task until reconstruction improves may help one model learn both local details and broader visual structure.

Evaluation Setup: The authors test whether the learned visual features transfer to several tasks:

  • Image recognition: ImageNet top-1 measures the percentage of correctly classified images; a linear probe tests the features with a simple classifier while keeping the image model fixed. BYON improves on the strongest evaluated baseline by 0.7 and 2.2 percentage points, respectively.
  • Scene segmentation and object detection: ADE20K mIoU measures overlap between predicted and labeled regions; COCO box AP measures how well the model detects and locates objects. The reported gains are 1.4 mIoU and 1.7 box AP.
  • Fine-grained recognition: Six additional tests measure how well the model distinguishes similar categories; BYON gains 0.8 to 3.5 points over the strongest evaluated methods.

Overall, I find the paper well structured and the empirical study clearly presented.

Critical strengths
  • The method is clearly specified, and the factorial ablation tests the roles of adaptive noise and gating separately.
  • Evaluation spans multiple downstream tasks.
  • The paper reports the added training cost.
Critical weaknesses

The "matched budget" comparison matches epochs and GPU count, but BYON uses about 40% more FLOPs and 15% more GPU-hours than DiffMAE.

Questions for authors

Does BYON retain its gains against baselines trained at equal total compute or GPU-hours?

1 comment
AuthorsOpenReview ↗

Response to Reviewer AXPx

W1 / Q1. Does BYON retain its gains against baselines trained at equal total compute or GPU-hours?

Yes, under both criteria. At equal total FLOPs, which gives the baselines 12 to 29% more GPU-hours than BYON, BYON is ahead of every baseline in the table: on ImageNet-1K by +0.60 points over the strongest denoising baseline and +0.16 over MAE trained for twice as many epochs; on ADE20K by +1.05 mIoU; on COCO by +2.0 box AP; under the linear probe by +0.8.

Table R3. Equal compute (ViT-B/16; total FLOPs relative to BYON at 400 epochs; GPU-hours on 4×A100; new rows in bold).

MethodEpochsTotal FLOPsGPU-hoursIN-1K FTLinearADE20KCOCO AP$^{\rm box}$
DiffMAE4000.718482.60 ± 0.3362.042.68 ± 0.4844.9
DiffMAE5601.0012182.7862.342.9145.2
MaskDiT4000.688382.86 ± 0.0561.942.90 ± 0.1143.3
MaskDiT5841.0012582.9662.443.0844.6
MAE4000.505582.78 ± 0.2062.343.02 ± 0.1246.0
MAE8001.0010983.463.743.4146.5
BYON4001.009783.56 ± 0.1664.544.46 ± 0.2148.5
  • The margin narrows on classification and largely remains on dense prediction. Against MAE at 800 epochs the ImageNet-1K fine-tuning margin narrows from +0.78 to +0.16, within one seed standard deviation, and the linear-probe margin from +2.2 to +0.8. On dense prediction most of the margin remains: +1.05 mIoU on ADE20K (from +1.44) and +2.0 box AP on COCO (from +2.5).
  • Both criteria in one run per baseline. 560 epochs of DiffMAE equal BYON's total FLOPs (400×63.8/45.5400\times63.8/45.5) and exceed its GPU-hours (121 vs. 97); likewise 584 epochs of MaskDiT. A baseline matched on GPU-hours alone (about 462 epochs) has less compute than these rows.
  • The reviewer is right about the wording. Epochs, data, code base and hardware were matched; compute was not (63.8 vs. 45.5 GFLOPs, 97 vs. 84 GPU-hours; L464–467). We have revised "matched budget" to "matched epochs, data, code base and hardware".
  • Where the cost is. One extra forward pass without gradients during pre-training (L295–296). The reference branch is then discarded: fine-tuning and inference cost are identical for all methods.

We have added the two new rows to Table 8.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response (summary of our replies)

We thank the three reviewers for their careful reading and constructive feedback, and the area chair for handling our submission. The requested control and clarifications have made the evidence for the paper's claims stronger.

We have addressed all seven concerns raised in the three reviews, with completed experiments, and the method, the central claims, the conclusions and every number of the submission are unchanged. All three reviewers rate soundness "minor or fixable", and no review reports an error in a result or a proof. Every change in the revised PDF was requested by a reviewer and is additive: one small table for the requested control, two table rows, a two-line corollary, one appendix table and two short paragraphs of clarification (marked in blue). Details and tables are in the reply to each reviewer. Line numbers refer to the submitted version.

#ConcernReviewerWhat we didResult
1Is the gain of IAN due to content dependence or to the noise-level distribution?m4GD W1/Q1Ran the requested shuffled-assignment control (new Table 4b)With the distribution fixed and the assignment shuffled, 75 to 86% of the gain disappears
2Link between the propositions and the saliency rule; objective conflictm4GD W2Added Corollary 1, Table 10 and a paragraph after Eq. (5)Corollary 1 predicts result 1; reweighting and gradient agreement measured
3Gains at equal computeAXPx W1/Q1Trained the baselines to equal total FLOPs (two rows in Table 8)+0.60 pts over the strongest denoising baseline; +0.16 over MAE trained twice as long
4Notation of both propositions9GE5 W1/Q2Revised Sec 3.2Symbols defined in place; each proposition stated in words
5Significance beyond the linear setting9GE5 Q3Tested the linear prediction in a ViT-B in three waysConsistent in all three, including result 1
6Relation to augmentation9GE5 Q4Added a discussion paragraph in Sec 2The distinction is whether the strength is chosen from the input
7Vision vs. general-purpose encoder9GE5 Q1Clarified in the reply; no change to the paperClaims limited to vision, as the abstract states (L030–031)

Strengths noted by the reviewers. "a meaningful question: combining two useful self-supervised objectives does not necessarily improve representations" (m4GD); "one of the fundamental questions that representation learning tries to answer" (9GE5); "the factorial ablation tests the roles of adaptive noise and gating separately" (AXPx); "gains persist with longer training and a larger backbone" (m4GD); "The paper reports the added training cost" (AXPx).

What we have revised.

  • New Table 4b: the shuffled-assignment control requested by Reviewer m4GD. Table 8: two rows, DiffMAE at 560 epochs and MaskDiT at 584 epochs, with their top-1 column.
  • Sec 3.2: Corollary 1 (two lines; proof in App. A); symbols defined where they first appear; one sentence in words after each proposition.
  • One paragraph after Eq. (5) linking the propositions to the saliency rule; one discussion paragraph in Sec 2 on augmentation.
  • App. D: Table 10, second moments and gradient agreement.
  • Wording: "matched budget" reads "matched epochs, data, code base and hardware", or "matched protocol", throughout.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.