acceptodds
Under review as a conference paper at ICLR 2027

Decoding Register Tokens Preserves Global Scene Details

Abstract

Pretrained vision transformers learn structured image representations spanning spatial patch tokens and non-spatial tokens. Recent analyses suggest that non-spatial register tokens encode global scene properties such as illumination, blur, and style. Yet patch-based image decoders discard these outputs, leaving the decoder to recover image appearance from patches alone. We investigate whether retaining register features improves decoding when the complete spatial patch representation is already available. We study this question using representation autoencoders (RAEs) with frozen register-equipped encoders. Our decoder processes four register tokens together with all patch tokens through joint self-attention. Appearance probes, paired reconstruction-error analysis, and fixed-decoder register substitutions examine which global image properties are accessible from registers and how their values affect reconstructed pixels. For generation, we train a multi-head attention pooling (MHAP) predictor to recover register features from sampled patches, keeping the pretrained diffusion model and its samples unchanged. On ImageNet-1K, our ViT-B decoders achieve mean reconstruction FIDs of 0.472 with DINOv2 and 0.442 with DINOv3, reductions of 9.5% and 5.2% relative to their respective patch-only baselines. The corresponding PSNRs are 19.73 and 20.54 dB. For generation, our DINOv2 models achieve a mean FID of 1.641, 9.8% below the patch-only baseline on identical sampled patches.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
register tokens, vision transformers, image reconstruction, latent diffusion, representation learning

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_Z7SWOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed to use register tokens from pre-trained ViTs such as DINOv2 and v3 to generated images with better global scene details. To achieve this, the tokens are fed into representation-autoencoder decoders alongside all patch tokens. For image generation, an MHAP predictor estimates registers from sampled patches, leaving the pretrained DiT unchanged. The methods reports results on ImageNet-1K, where it achieves an rFID reduction across various ViT registers with also gFID improving on identical DiT samples.

Critical strengths
  • The proposed method is simple with low computational cost and can be used as a drop-in without model re-training
  • The paper is well written
  • The limitation section is transparent about where the method has shortcomings
  • The introduced controls are carefully designed (shuffled and mean registers, learned tokens, paired multi-seed statistics)
Critical weaknesses
  • The core finding of the proposed method is close to definitional: In their paper on register tokens, Darcet et al. point out that register tokens accumulate additional global information not captured by the CLS token. Therefore, these tokens naturally contain global appearance information
  • The presented ablations fail to clearly demonstrate that decoding registers adds significant benefit over simply using the CLS token (rFID 4.84 → 4.75 with CLS vs. 4.61 with registers). When evaluating the combination of CLS and registers combined, it barely improves on registers alone, which indicated information redundancy rather than a distinct information source
  • For generation, the predicted registers are a deterministic function of the patches, so the empirical gain might reflect added capacity and supervision rather than new information
Questions for authors
  • Can you make a compelling point why a CLS token cannot capture essentially the same benefit as apposed to additional register tokens?
  • Does the presented generation gain hold against a capacity-matched patch-only decoder and with noise-augmented decoders (τ > 0)?
5 comments

Thank you for the careful feedback. We agree that the manuscript should separate three questions more clearly: which type of global information registers carry, whether it is accessible for reconstruction, and whether different global representations are interchangeable. In response, we added CLS and register ablations, per-slot analyses, and noise-augmented generation results, and Section 5 now states the remaining attribution limits. Table numbers refer to the revised manuscript.

W1: Is the benefit already implied by prior work on registers?

We thank the reviewer for this point, which helps us clarify how our work differs from prior work. We agree that registers contain global information; this is our starting point, not our contribution. However, “global information” covers different things: semantic content such as object category, spatial layout, and image-wide appearance such as color, illumination, blur, and texture. Darcet et al. assessed global content through classification probes and discarded registers after encoding. Fel et al. identify appearance-related register concepts through dictionary learning. Decoding shows directly which global information is accessible from registers and whether a decoder uses it.

This is not implied by prior work. Patches are computed with registers present, so much of the register information may be recoverable from them. But image losses do not explicitly supervise recovery of the encoder’s register features. Our results show:

  • Registers expose image-wide appearance to the tested decoder. A decoder given only the four registers partly recovers color and texture while losing substantial object and spatial detail (Figures 2, 5). Adding registers to the full patch sequence raises mean-RGB R² from 0.573 to 0.663 but leaves nonzero-frequency targets essentially unchanged (Table 8). A new probe shows the gain persists with CLS included (0.588 → 0.636), while a shuffled control gives 0.589 (Table 9).
  • Reconstruction improves beyond a color shift. On ImageNet-1K, rFID drops by 9.5% (DINOv2) and 5.2% (DINOv3) and PSNR improves; every register seed beats every patch-only seed in rFID within each encoder. In the separate ImageNet-100 analysis, 57% of the MSE reduction is spatially varying, and the texture component accounts for 82% of the DISTS reduction (Section 4.2, Appendix G.1). The gains transfer to NINCO and Places365 without retraining.
  • Register values directly control decoded appearance. With patches and decoder fixed, registers from a color-transformed image shift the reconstructed color along the transformation (coefficient 0.499, 95% CI [0.475, 0.524]). New per-slot substitutions show that registers from a contrast-calibrated blurred copy (σ = 1) reduce high-frequency power by up to 20.6%, and that a +15° hue rotation transfers mainly through one slot per encoder (Table 29). Figure 6 provides a separate ordinary-blur illustration.

Our results thus identify image-wide appearance information accessible from registers. They also show that the decoder uses this information when it is supplied, rather than readily extracting it from patches.

W2: Does the small benefit of CLS on top of registers imply redundancy?

We thank the reviewer for the careful reading of the ablations. CLS does help, and CLS and registers share information, but their incremental benefits are asymmetric:

Added input (all patches retained)DINOv2 rFID reductionDINOv3 rFID reductionDINOv2 mean-RGB R² increase
Registers after CLS0.15420.04240.0477
CLS after registers0.01810.00380.0066

rFID from Table 12; R² from Table 9, using the 768-dimensional register mean to match the width of CLS.

Registers give 8.5× (DINOv2) and 11× (DINOv3) the incremental rFID reduction of CLS, and 7.3× the incremental gain in color prediction. CLS therefore adds little to the tested readouts given registers, whereas registers still improve them given CLS.

The gap is clearer in paired fidelity. On ImageNet-1K (three seeds), CLS reaches rFID 0.4968, MSE 0.01381, and DISTS 0.1080, compared with 0.4717, 0.01334, and 0.1044 for registers (Tables 2, 13, 19). Simply repeating CLS at extra token positions does not reproduce the register benefit either: four identical CLS copies perform like one (rFID 0.4931 vs. 0.4968, DISTS unchanged, MSE slightly worse; Table 13).

Q1: Could CLS capture the same benefit?

We cannot rule out that some CLS interface could close the gap, but none we tested does (W2). Our results suggest why: CLS and registers summarize different aspects of the image.

Each is better at a different task. We added an experiment on Waterbirds (Sagawa et al., 2020). In this dataset, landbirds and waterbirds appear on land or water backgrounds, and bird type correlates with background during training. With the same frozen DINOv2 encoder, CLS is the strongest bird classifier, while registers are the stronger reconstruction input (Tables 10–11):

PatchesCLSRegisters
Reconstruction DISTS ↓0.11180.11090.1072
Worst-group accuracy (%) ↑ (standalone readout)34.781.145.8

Reconstruction: ImageNet-1K-trained decoders, no adaptation, three seeds; inputs are patches, patches + CLS, and patches + four registers. Classification: linear heads with an equal update budget, three seeds; readouts are patch mean, CLS, and register mean. Worst-group accuracy is the minimum accuracy across the four bird/background groups.

Adding CLS even slightly worsens mean MSE and PSNR compared with patches alone, whereas registers outperform CLS on all four reconstruction metrics in every seed.

A plausible mechanism is the pretraining objective. DINOv2’s DINO loss on CLS compares views that differ in color jitter, blur, and solarization, which pushes CLS toward invariance to the appearance properties the decoder needs. Registers have no direct output loss. Consistent with this, the register mean aligns more closely with pooled patches than CLS does (CKA 0.858 vs. 0.658; Table 6). We present this as a hypothesis; the empirical conclusion does not depend on it.

W3: Do the generation gains reflect added capacity and supervision?

This point concerns generation only. In reconstruction, the decoder receives the encoder’s native registers. In generation, the registers are predicted from the sampled patches, so no new input information is added at generation time. What changes is the readout: MHAP is trained to predict the encoder’s native registers, and the decoder is trained to use them. MHAP receives explicit register-target supervision, while the decoder receives native registers as inputs and is trained with image-reconstruction losses. Two results support this interpretation:

  • The decoder relies on sample-specific register values. With decoder, predictor, and patches fixed, shuffling predictions within class worsens gFID from 1.656 to 1.798, and mean registers give 2.316 (Table 24). The gain therefore depends on registers predicted for each specific sample, not merely on additional tokens.
  • The tested alternatives with more parameters or internal register supervision do not reproduce the gain. A decoder trained to predict registers internally (88.2M parameters) reaches 1.840, worse than patch-only (1.819). A larger InfoNCE decoder (143.8M vs. 102.4M for the MHAP route) reaches 1.789, compared with 1.656 for MHAP (decoder seed 0; Table 15).

The MHAP route achieves the lowest gFID among these alternatives and improves gFID consistently: by 9.8% with clean decoders and 10.0% with noise-augmented decoders, with every decoder seed favoring it (Q2). The compared variants also differ in architecture, so they do not fully separate register supervision from additional processing (Section 5). The practical result is nonetheless clear: samples from an existing generator decode better without retraining the diffusion model.

Q2: Capacity-matched and noise-augmented decoders

Noise augmentation. We trained patch-only and register-input decoders with τ = 0.8. Each pair decodes the same 50,000 generated patch tensors:

Decoder trainingPatch-only gFID ↓MHAP + registers gFID ↓Reduction
τ = 01.8191 ± 0.00581.6414 ± 0.01339.77%
τ = 0.81.7516 ± 0.02991.5771 ± 0.01489.96%

Mean ± SD over three decoder seeds, with one fixed MHAP predictor and patch bank.

The relative gain remains approximately 10% under noise augmentation, and all seed pairs favor registers. With τ = 0.8, the ViT-B register decoder (1.577) approaches the much larger released ViT-XL decoder (1.557).

Capacity. The complete MHAP-assisted decoding route adds 16.0M parameters (+18%) and about 20% decoder latency (Table 5), so the comparison is not exactly capacity-matched. However, the larger InfoNCE decoder performs worse (W3). In the ImageNet-100 capacity study, a ViT-B decoder with CLS plus four registers also achieves lower rFID than patch-only ViT-L and ViT-XL decoders trained for the same 16-epoch budget (Table 14).

Dear Reviewer,

Thank you again for your time and feedback. We know this is a busy period and would appreciate hearing whether our rebuttal addresses the points you raised or whether any concerns remain. We would be happy to clarify anything further and discuss any open questions before the discussion closes.

If our updated evidence addresses your concerns, we would be grateful if you would consider raising your score.

Reviewer_1q9bOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript treats encoder-only image reconstruction and generation. The gap established in the introduction is that current decoder-only models discard register tokens, which have been shown to encode important image content in recent work. To address this, the work at hand proposes to a) train reconstruction decoders with register tokens attached, and b) predict register tokens from sampled patch tokens before reconstruction for latent-space image generation. Experiments show that global scene appearance can indeed be reconstructed from register tokens alone. Register tokens can also be used to apply aspects of global scene appearance from one image to another. Experiments on ImageNet show that the inclusion of registers improves reconstruction performance in terms of Fréchet Inception Distance. Predicting registers from patch tokens sampled from a pre-trained latent generative model also yields FID gains compared to the no-register baseline.

Critical strengths

The work is well-motivated and the overall goal and approach are clear. The manuscript tells a concise story from previous findings to the application to the specific problem of latent generation/reconstruction. The work provides some insight into what what register tokens encode and how the information can be read out, which is interesting in itself. While, as mentioned in the text, Into The Rabbit Hull has explored this before, the interpolation in Fig.3 is an interesting additional result. The same is true for the results presented in Fig. 2. In terms of reconstruction and generation performance, the quantitative results on ImageNet show clear improvements over the baseline method. I think that this would suffice to convince practitioners to use a similar approach, or motivate the field working on latent generation to directly incorporate register tokens in latent generative models. Therefore, the work could have some impact on the field.

Critical weaknesses

The overall scope of the paper is somewhat limited. The idea can be considered to be quite straightforward, and one could argue that the scope does not warrant a full publication at a highly competitive venue. This is corroborated by the inclusion of the weaker sections 4.1 and 4.2, which, in my opinion, do not contribute a lot of additional insight over the large-scale experiments that follow. For a small-scale introductory analysis I would expect a very illustrative toy example; in the current state I am not sure what to learn from it. Perhaps this is something the authors could clarify in the rebuttal. The analysis is limited to DINOv2 and DINOv3 which limits generalization, but due to the ubiquity of these models in the field I think this is defensible. A stronger criticism is that the quantitative gains are reflected very weakly by the qualitative examples. Considering Fig. 4, an effect is apparent for the reconstruction examples, but the generated examples do not look very convincing to me. I would have trouble to say which column is 'better here. When studying Fig. 6 (Appendix), the effect seems extremely small even for the reconstruction examples. Qualitatively, it seems as if the registers only contribute gains in cases where the background and/or object is white. Perhaps the authors could speak to the divergence between quantitative and qualitative results.

Overall, the paper is very borderline in my opinion. Quantitative results are good and relevant, but scope and qualitative performance gains are limited.

Questions for authors

No additional questions beyond the comments under 'Weaknesses'

5 comments

Thank you for recognizing the practical relevance of our results and for pointing out where their significance was unclear. In response, we tested the white-background explanation on the full evaluation set, updated the qualitative comparisons, and clarified what the introductory analyses establish. Figure and table numbers refer to the revised manuscript.

W1: Weak qualitative effects and possible concentration on white images

Your observation prompted us to examine the full evaluation set rather than selected examples. We grouped all 50,000 ImageNet-1K validation images by their coverage of white or light-gray pixels and compared reconstructions for both encoders across three decoder seeds:

Source imagesImagesMSE reduction (v2 / v3)DISTS reduction (v2 / v3)
All50,0003.28% / 2.96%4.65% / 4.41%
< 1% white29,6174.46% / 4.35%4.82% / 4.61%
≥ 1% white20,3831.92% / 1.35%4.41% / 4.09%
< 1% white or light gray9,7706.75% / 6.44%5.15% / 4.89%
≥ 1% white or light gray40,2302.64% / 2.33%4.53% / 4.28%

White: min RGB ≥ 230, channel spread ≤ 12. White or light gray: min RGB ≥ 160, spread ≤ 24. Pixel-coverage definitions, not semantic labels.

The gains are larger, not smaller, in images with less than 1% white-or-light-gray coverage. For these images, MSE drops by about twice the overall rate. All groups improve on average, although individual images can worsen. White regions may make the effect more visible: a slight color cast stands out on neutral surfaces but is masked by colorful content. The earlier Figure 4 (snow, a white Samoyed, a pale seashore) probably created this impression. The revised Figure 4(a) instead shows a red-lit guitar, a radio, and a green leaf.

Per-image gains are modest but systematic. PSNR improves by 0.24 dB. Such changes can be hard to see side by side (Figure 7, previously Figure 6). FID aggregates over 50,000 images, which makes small but consistent shifts in color and texture clearly measurable.

Generated samples have no reference image. We agree that neither column in Figure 4(b) is visibly better per sample, and we do not claim that. Both routes decode the same 50,000 generated patch tensors, and mean gFID improves from 1.82 to 1.64 (−9.8%), with improvement in all decoder seeds. The decoded images as a whole move closer to the distribution of real images.

W2: What do Sections 4.1 and 4.2 add, and would an illustrative example help?

We agree that their purpose should be clearer. The large-scale results show that reconstruction improves; Sections 4.1–4.3 explain what improves and why:

Section 4.1: Which information do registers expose? Image-wide appearance. The register-only decoder (Figure 2) shows this directly: color and texture are partly recovered, while substantial object and spatial detail is lost. Probes confirm that registers improve mean-color prediction even with the full patch sequence and CLS available (R² 0.588 → 0.636), while shuffled controls give no gain and nonzero-frequency prediction does not improve.

Section 4.2: Which errors change? On ImageNet-100, registers reduce mean RGB MSE by 8.5% in a paired, multi-seed comparison. 57% of this reduction is spatially varying rather than a uniform color shift.

Section 4.3: Does the decoder use register values? Yes. With patches and decoder fixed, changing only the registers changes the image, as the interpolation in Figure 3 shows. As a more illustrative controlled example, we added single-register substitutions using ordinary Gaussian blur (σ = 1; Figure 6, Appendix G.5). Replacing one register with its counterpart from a blurred copy of the image visibly blurs the reconstruction while all patches stay unchanged. In the separate contrast-calibrated σ = 1 study, across 256 evaluation images and three decoder seeds, blur-derived register substitutions reduce high-frequency power by up to 11.7% (DINOv2) and 20.6% (DINOv3). A +15° hue rotation shifts decoded hue by 2.3° and 2.6° through the same registers (Table 29).

W3: Limited scope and straightforward method

We see the simplicity as a strength. As you note, the results could convince practitioners to adopt the approach, because it requires no change to the encoder or diffusion model. Our contribution is the finding that these outputs, discarded by current RAE pipelines, should be retained, and this finding holds broadly:

  • across two encoder families, with every register seed beating every patch-only seed in ImageNet-1K rFID within each encoder;
  • out of distribution (NINCO, Places365) without retraining;
  • for generation, including with noise-augmented decoders (τ = 0.8: gFID 1.752 → 1.577, new in this rebuttal);
  • across decoder sizes from ViT-B to ViT-XL in the separate CLS-plus-four-register comparison.

Beyond performance, the paper identifies which global information registers carry and shows that the decoder uses it (W2).

W4: Generalization beyond DINOv2 and DINOv3

We agree that our results do not establish the same behavior for all vision-transformer families. The two encoders do, however, differ in training data, objective, patch size, and resolution, and they organize appearance information across different register slots (Table 29). That the gain holds for both suggests it does not depend on one specific encoder. Extending the analysis to other register-equipped models is a natural next step.

We hope these responses address your concerns. In particular, the register gains are not concentrated on white images (Relative MSE reductions are approximately twice the overall rate in images with less than 1% white-or-light-gray coverage), and the new single-register substitutions provide the illustrative example you asked for.

Dear Reviewer,

Thank you again for your time and feedback. We know this is a busy period and would appreciate hearing whether our rebuttal addresses the points you raised or whether any concerns remain. We would be happy to clarify anything further before the discussion closes.

Reviewer_LGq7OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tests whether four native register tokens improve RAE image decoding when the complete patch representation is retained. It combines appearance probes, paired reconstruction-error analysis, fixed-decoder interventions, and ImageNet-1K reconstruction with DINOv2 and DINOv3. For generation, a separately trained MHAP network predicts registers from sampled patches while leaving the pretrained diffusion generator unchanged. Additional experiments examine out-of-distribution reconstruction and several auxiliary-input alternatives.

Critical strengths
  1. Multiple complementary measurements support the reconstruction result. Native-register input reduces ImageNet-1K rFID from 0.5211 to 0.4717 with DINOv2 and from 0.4669 to 0.4424 with DINOv3, with paired pixel and perceptual improvements across three decoder seeds. The same ImageNet-1K-trained decoders improve rFID, PSNR, and DISTS on NINCO and Places365 without adaptation. Full-patch probes and fixed-decoder substitutions support an accessibility-to-readout interpretation without requiring registers to contain information absent from patches.

  2. The principal generation comparison is well controlled. All routes decode the same 50,000 stored DiT-DH/XL patch samples; gFID improves from 1.8191 to 1.6414 across decoder seeds. The within-class shuffled-register control preserves the class-conditional register-bank distribution while worsening gFID, strengthening the evidence that correspondence matters. The separate five-auxiliary-token DiT-S experiment provides additional, although not directly equivalent, evidence for the recovery route.

Critical weaknesses
  1. The register-target contribution is not fully isolated from an additional learned readout. The internal-supervision alternatives in Table 9 change architectures and objectives, while Table 10's learned-token and patch-summary controls use five tokens, one seed, and a different experiment from the principal four-register result. These are informative controls, but they do not provide an architecture-, token-, and training-budget-matched test of register supervision against a patch-derived auxiliary representation. This limits attribution of the gain, rather than disputing the reported same-size decoder improvement.

  2. Native-register reconstruction and predicted-register use should be distinguished more directly. Table 16 demonstrates that MHAP predictions outperform mean and shuffled replacements within a fixed register-aware decoder, but it does not include the corresponding patch-only decoder on those same encoded images. Predicted registers also retain a measurable gap to native registers. A matched patch-only versus MHAP reconstruction comparison would clarify the practical recovery pipeline, complementing the generation FID result. The primary four-register generation study further uses one predictor checkpoint and sample bank; three decoder seeds do not measure those sources of variation. The reported 20–24% latency increase is decoder-stage overhead, excluding diffusion sampling, and should not be interpreted as the same increase in end-to-end generation time.

Overall assessment. I lean toward weak acceptance. The reconstruction evidence is consistent across encoders, seeds, metrics, and two external datasets, and the fixed-sample generation experiment establishes a useful improvement in the principal setting. More closely matched auxiliary controls and a direct comparison of the predicted-register reconstruction pipeline would clarify the mechanism and practical scope. The larger ViT-XL reference does not invalidate the controlled ViT-B improvement.

Questions for authors
  1. Which existing control most closely matches the MHAP route in architecture, token count, and training budget? Can the remaining differences be stated explicitly, or can a matched patch-derived auxiliary control be reported if already available?

  2. On the same held-out encoded images and decoder seed used in Table 16, how does MHAP-predicted-register reconstruction compare with the patch-only decoder in MSE, PSNR, DISTS, and rFID? This would directly connect native-register reconstruction gains to the predicted-register pathway.

  3. What is the separate training cost of the 100-epoch predictor, and which variability sources are covered by the generation error bars? If another predictor checkpoint or sample bank is available, does it support the primary four-register result?

4 comments

Thank you for the constructive review and for recognizing the reconstruction and generation results. Your questions concern two distinctions that we have investigated further: whether the generation benefit arises specifically from register supervision, and how closely predicted registers recover the reconstruction utility of native registers.

W1 / Q1 — Which controls distinguish register supervision from additional learned processing?

We agree that the existing ablations do not fully isolate register supervision from additional learned processing, and we do not have a control matched in architecture, token count, and training budget. The two closest controls, and how they differ from the MHAP route, are: Learned patch summaries (Appendix C.5). This is the closest patch-derived auxiliary representation without register targets. Differences: five tokens instead of four; trained end-to-end with image losses rather than as a separate predictor with register targets; ImageNet-100, one seed, reconstruction only. Result: rFID drops by 0.012, compared with 0.228 for native registers in a separate ImageNet-100 run. Internal register MSE (Appendix C.4). This is the closest match in token count and setting: four slots, ImageNet-1K, the same 16-epoch decoder budget, and the same generated patches. Differences: it uses register targets but places the readout inside the decoder rather than in a separately trained predictor, and it has fewer parameters (88.2M vs. 102.4M) and no separate 100-epoch predictor training. Result: gFID 1.840, compared with 1.819 for patch-only and 1.656 for MHAP (decoder seed 0). The internal-register-MSE variant does not reproduce the MHAP generation gain, while learned patch summaries provide only a small reconstruction improvement in their separate experiment. Neither comparison isolates register supervision alone. Our generation claim therefore concerns the complete decoding pipeline, whose benefit is measured with the generator and sampled patches unchanged. Section 5 now states explicitly that these comparisons evaluate complete architectures and training objectives and do not isolate register-target supervision from additional learned processing.

W2 / Q2 — How does predicted-register reconstruction compare with patch-only decoding?

We completed the requested comparison on the same 50,000 ImageNet-1K validation images, using the corresponding decoder seed and clean-trained decoders (τ = 0).

DINOv2 decoder input, seed 0MSE ↓PSNR ↑DISTS ↓rFID ↓
Patches0.01377519.4920.1095820.525288
Patches + native registers0.01334019.7250.1044110.470804
Patches + MHAP-predicted registers0.01465119.1610.1120530.505178

The three-seed aggregate confirms the same pattern. MHAP-predicted registers reduce rFID by 3.08% relative to patch-only decoding, but increase paired MSE by 6.26% and DISTS by 2.33%.

MHAP therefore recovers useful register information, although it does not fully reproduce native-register reconstruction fidelity. Importantly, the current predictor is optimized for register-feature prediction, not final image reconstruction: training and checkpoint selection minimize register-feature MSE, independently of the receiving decoder. The remaining gap characterizes this predictor and objective, rather than establishing the best achievable recovery. Native and predicted registers use the same receiving decoder and patches, allowing their reconstruction difference to be measured directly.

Despite this incomplete recovery, the generation benefit persists against the newly completed noise-trained patch-only baseline: gFID improves from 1.7516 to 1.5771 (−9.96%), with improvement in all three decoder-seed comparisons. Thus, useful recovery for generation does not require perfect reproduction of native-register reconstruction fidelity.

Q3 — What are the predictor cost, latency and uncertainty?

MHAP was trained for 100 epochs (499,900 updates), requiring 64.43 A100-40GB GPU-hours across successful training workers, excluding failed attempts and setup. The deployed checkpoint was selected at epoch 96 using training-derived selection images.

The measured batch-one decoder-stage latency increases from 9.610 ms to 11.577 ms (+20.47%). This includes MHAP and decoding, from GPU-resident patches to RGB, but excludes diffusion sampling, encoding and I/O. It is therefore not a 20.47% increase in end-to-end generation time.

Which variability sources are covered by the generation error bars?

The principal generation result, gFID 1.8191 ± 0.0058 → 1.6414 ± 0.0133, reports mean ± SD across three decoder seeds, using one MHAP predictor checkpoint and the same 50,000 generated patch tensors. These error bars do not measure variation across independently trained predictors or generated sample banks. Additional predictor-seed evaluations are not yet complete, and no independent sample-bank replication is included in these estimates. Table 3 and Appendix F.4 specify that the reported variation is across decoder seeds, with one fixed predictor and the same generated patch samples.

Conclusion

The additional comparisons distinguish native-register reconstruction fidelity from predicted-register recovery. The current predictor already provides a generation benefit, including against a noise-trained patch-only baseline, without being optimized for final image reconstruction. The reconstruction comparison is reported in Appendix F.1, predictor cost in Appendix A.3, and the generation protocol and decoder-seed uncertainty in Appendix F.4. Section 5 explicitly attributes the measured improvement to the complete decoding pipeline rather than isolating register supervision.

Dear Reviewer,

Thank you again for your time and feedback. We know this is a busy period and would appreciate hearing whether our rebuttal addresses the points you raised or whether any concerns remain. We would be happy to clarify anything further before the discussion closes.

Dear Reviewer,

We hope you don’t mind us following up once more before the discussion closes. If you have a moment, we would really appreciate your thoughts on our response and the additional experiments. Please let us know if any of your concerns remain open; we would be happy to clarify anything further.

Thank you again for your time and feedback.

Reviewer_5AwHOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies the role of visual-encoder register tokens in image reconstruction and generation. Whereas patch-based decoding typically discards these non-spatial outputs, the proposed approach feeds four registers from a frozen DINOv2 or DINOv3 encoder to the decoder alongside the complete patch sequence. For generation, an independently trained attention-pooling predictor recovers registers from diffusion-generated patch features without modifying the diffusion model. Appearance probes, reconstruction-error analysis, and fixed-decoder register substitutions examine the role of these features, alongside reported improvements in reconstruction and generation quality. The main contribution is to establish the practical decoding value of existing register representations and provide a way to exploit them while retaining the original generator.

Critical strengths

1. The research question is concrete, and the practical gains are consistently supported. Retaining all spatial patches directly tests whether additionally reading registers remains useful, without creating an advantage by weakening the spatial input. Table 2 shows lower reconstruction FID and higher PSNR for both DINOv2 and DINOv3, consistently across decoder seeds. Section 4.2 further indicates that the benefit extends beyond correcting an image-wide color offset, while the cross-dataset results reported in Section 4.4 strengthen its practical relevance.

2. The key controls help establish the registers' functional role. Section 4.3 fixes the decoder and patches while changing register values, showing that the trained model uses these inputs. Table 3 compares generation from identical diffusion-sampled features, locating the observed differences in the subsequent decoding pipeline. The paper also reports the gap between predicted and native registers and explicitly distinguishes image differences from quality improvements, supporting a careful interpretation of the experiments.

Critical weaknesses

1. The contribution primarily concerns effective use of existing representations, with a limited conceptual advance. Section 2 acknowledges prior findings linking registers to global appearance, and neither token concatenation nor attention pooling is a new primitive. The contribution therefore rests mainly on the decoding experiments, rather than on a first discovery that registers contain appearance information. In the complete-patch probing setting of Section 4.1, the main improvement concerns mean color, while nonzero-frequency prediction is essentially unchanged. The paper should distinguish its useful decoding finding from broader representational interpretations and identify the specific conclusion that advances existing knowledge. This limits the extent of the contribution without negating the reconstruction gains.

2. The generation gains do not fully separate register supervision from the additional learned pathway. Sections 3.3 and 4.5 fix the generator and sampled features but introduce an independently trained MHAP predictor. Table 3 therefore supports an improvement of the complete pipeline over the tested patch-only ViT-B baseline, but does not by itself determine how much of the gain comes from register supervision rather than additional nonlinear processing and training. The internal-supervision, learned-token, and patch-summary controls mentioned in the main text are relevant; their ability to separate these factors should be clarified. Existing cost information for the complete decoding routes should be summarized alongside the main results, together with the attribution supported by these controls. This concerns the source of the gain and its practical trade-off, rather than the validity of the measured improvement in Table 3.

3. The appearance-control interpretation extends beyond what the interventions directly establish. The substitutions and interpolations in Section 4.3 show that registers can affect output appearance, but color and saturation change jointly, with additional off-target responses. Gradient energy in Figure 3 is also sensitive to texture and structure and cannot independently establish blur control. The style-transfer discussion in Section 5 is therefore better positioned as an exploratory application. Although some limitations are already reported, changing appearance should be distinguished from independently and reliably controlling a particular appearance attribute. This concerns the application claims rather than the principal reconstruction results.

Questions for authors

1. What is the most important new insight beyond the prior register-appearance studies discussed in Section 2? Which main-text results support an interpretation beyond improved decoding from reading existing registers? If the contribution is primarily this decoding finding, the contribution statement should focus accordingly.

2. Which controls mentioned in the main text vary the register-supervision target under comparable prediction pathways and training conditions? Please clarify which alternative explanations these controls exclude and summarize the quality–cost trade-off relative to patch-only decoding in Table 3 using the existing cost measurements.

3. Can the existing interventions distinguish changes in color and luminance from changes in blur, texture, or structure? If not, should the style-transfer discussion in Section 5 be restricted to exploratory appearance modulation, with its main limitations stated explicitly?

4 comments

Thank you for the careful review and for distinguishing the practical decoding result from questions about conceptual novelty, attribution, and appearance control. Your comments helped us clarify what each experiment establishes and how the contribution should be presented.

W1 / Q1 — What is the new insight beyond existing register-appearance studies?

The central finding is that registers improve image decoding even when the complete spatial patch representation is already available. The operation is simple, but its usefulness is not implied by knowing that registers contain global appearance information. Both decoding routes use the same register-equipped encoder; the patch-only route already benefits from register computation during encoding. We examine the additional value of supplying those registers to the decoder and recovering the information from generated patches without changing the diffusion model.

The analyses highlight the additional benefit of the register tokens. Adding the register mean to the complete patches and CLS increases mean-RGB prediction R² from 0.5884 to 0.6361, while shuffled registers remain at 0.5885. Nonzero-frequency prediction does not improve, identifying a specific advantage in accessible global color information rather than a general improvement in every image property.

Sections 4.1–4.3 address successive questions: which appearance properties are accessible from registers alongside patches, which reconstruction errors improve when registers are retained, and how changing individual registers affects decoded images. The new ablations connect the representation analysis to functional decoder responses: different registers influence luminance, color and fine-scale appearance differently, even with all patches fixed. Section 4.3 summarizes these unequal, overlapping influences; Appendix G.5 presents the complete results in Table 29 and a separate ordinary-blur illustration in Figure 6.

Our contribution is therefore a useful register-based decoding interface, a recovery route for generation, and an analysis connecting individual registers to decoded appearance.

W2 / Q2 — What do the controls establish about supervision versus additional processing, and what is the quality–cost trade-off?

The ablations separate two questions: whether the decoder uses sample-specific register predictions, and whether register supervision is better than an equally expressive learned alternative.

The first is supported by the fixed-decoder comparison. With one predictor–decoder pair and the same 50,000 generated patches, aligned predictions give gFID 1.6561, versus 1.7981 after within-class shuffling and 2.3155 with training-set means. Shuffling preserves the predicted register banks and their class association while breaking correspondence with individual patch samples. This establishes that the trained decoder uses sample-aligned predicted values; it does not isolate how much of the improvement over patch-only decoding comes from register supervision.

The five-learned-token and five-patch-summary ablations test additional decoder context without supplying encoder registers, while the internal-supervision ablation examines alternative uses of register targets. In the DINOv2 ImageNet-1K seed-0 comparison, internal register-MSE and InfoNCE variants achieve gFID 1.840 and 1.789, respectively, versus 1.656 for the MHAP route (Table 15). They differ from the principal MHAP route in token count, architecture or training procedure, so they do not isolate register supervision from additional learned processing. We therefore attribute the measured generation improvement to the complete decoding pipeline.

For the practical trade-off requested:

MeasurementPatch-only decodingMHAP + register decoding
gFID ↓1.8191 ± 0.00581.6414 ± 0.0133
Parameters, excluding encoder and diffusion model86.44M102.40M
Batch-one decoding latency9.610 ms11.577 ms
Batch-one peak allocated memory377.29 MiB439.25 MiB

This gives 9.77% lower gFID for 20.47% additional decoder-stage latency. Timings use an A100 40GB GPU with BF16, from GPU-resident patches to RGB outputs; they exclude encoding, diffusion sampling and I/O. The gFID values are means ± SD across three decoder fits, conditional on one predictor and the same generated patch bank.

The predictor required 64.43 A100-40GB GPU-hours across successful training workers, excluding failed attempts and setup. No diffusion retraining or additional diffusion pass is required. The generation benefit also persists with noise-trained decoders: gFID 1.7516 → 1.5771, a 9.96% reduction. The quality and cost measurements are summarized together in Appendix A.3.

W3 / Q3 — Do the interventions establish independently controllable appearance attributes?

We agree that changing appearance should be distinguished from independently controlling a particular attribute. Section 5 now describes the application as appearance modulation, with its coupled effects and limitations detailed in Appendix G.5.

Your question prompted us to examine individual registers rather than treating them only as a single bank. We evaluate 14 controlled transformations of hue, opponent color, illumination, contrast, blur and fine-scale noise on 512 NINCO and Places365 images, divided into 256 calibration and 256 evaluation images, with three decoder seeds per encoder. We replace one register at a time, keeping the original patches, remaining registers and decoder weights fixed. The study also includes norm-matched random register controls.

The responses differ substantially across slots. Under contrast-calibrated Gaussian blur (σ = 1), the principal high-frequency responses are:

Model / substituted slotChange in decoded high-frequency power [95% interval]
DINOv2 R4−11.71% [−13.68, −9.69]
DINOv3 R2−11.48% [−12.56, −10.43]
DINOv3 R3−20.56% [−22.55, −18.61]

Changes are relative to native reconstruction on the 256 evaluation images. Decoder seeds are averaged within each source before dataset-stratified source-family bootstrapping. Intervals are pointwise and exploratory on previously exposed cohorts, conditional on the trained decoders.

The same dominant slots also transmit color changes: a +15° source-hue rotation through DINOv2 R4 and DINOv3 R3 shifts decoded hue by +2.32° [1.63, 3.13] and +2.62° [1.90, 3.39], respectively. Thus, the influences are concentrated but overlapping rather than one attribute per register. Table 29 in Appendix G.5 reports all four slots and their accompanying effects; Figure 6 provides a separate ordinary-blur illustration at σ = 1.

Within DINOv2 R4 and DINOv3 R3, projecting the blur-derived change onto a calibration-fitted one-dimensional direction retains much of the high-frequency response, but also changes chroma.

We therefore report luminance, chroma and high-frequency power separately, including off-target changes. High-frequency power includes edges, texture and noise, so its reduction alone does not identify physical blur; unchanged patch inputs likewise do not guarantee unchanged decoded structure.

The supported application is targeted but coupled appearance modulation. The experiments establish transformation-related, slot-dependent responses, but not independent adjustment of brightness, color, blur or texture while preserving other properties. Appendix G.5 explicitly states these limits on attribute control, while Section 5 discusses appearance modulation separately from the principal reconstruction results.

Conclusion

The contribution is the demonstrated usefulness of registers for decoding with all patches retained, and a practical route to recovering that usefulness for generation. The added analyses identify which registers influence appearance and distinguish that influence from independent attribute control. The revised manuscript reflects this scope, with the principal generation result and measured computational cost reported together in Appendix A.3.

Dear Reviewer,

Thank you again for your time and feedback. We know this is a busy period and would appreciate hearing whether our rebuttal addresses the points you raised or whether any concerns remain. We would be happy to clarify anything further before the discussion closes.

Dear Reviewer,

Sorry to bother you again. As the discussion deadline approaches, we would be grateful if you could take a look at our response and the additional results. We hope these address the concerns you raised.

If any questions remain, please let us know. We would be happy to clarify anything further. Thank you again for your time.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of manuscript revisions

Summary of manuscript revisions

We thank all reviewers for their feedback. As stated before, we have uploaded a revised manuscript and provided individual responses. To help locate the changes, we summarize the principal additions and clarifications below. All section, figure and table numbers refer to the revised manuscript.

  • CLS and register comparisons. Section 4.1 and Appendix B.3 include the conditional appearance probe retaining all patches and CLS. Appendix C.2 reports the decoder-input comparisons and the four-copy CLS control. Tables 10–11 compare reconstruction and classification readouts on Waterbirds.

  • Individual-register responses and qualitative examples. Section 4.3 summarizes the expanded individual-register analysis, with the full transformation study and per-slot results in Appendix G.5 and Table 29. Figure 6 provides a separate ordinary-blur illustration. Figure 4 has also been updated with reconstruction and generation comparisons.

  • Predicted-register recovery, cost and uncertainty. Appendix F.1 includes the direct comparison of patch-only decoding, native registers and MHAP-predicted registers. Appendix A.3 summarizes generation quality, decoder-stage cost and predictor training cost. Appendices F.3–F.4 specify predictor training and the scope of the decoder-seed error bars.

  • Contribution and interpretation. The Introduction and Discussion clarify the distinction between information accessibility and information absence, and between the measured generation benefit of the complete pipeline and the unresolved contribution of register supervision alone. Section 5 and Appendix G.5 distinguish appearance modulation from independent attribute control.

The white/light-gray coverage analysis and the noise-augmented generation comparison at τ = 0.8 are reported as additional evidence in the reviewer-specific responses, rather than as additions to the uploaded manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.