- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposed to use register tokens from pre-trained ViTs such as DINOv2 and v3 to generated images with better global scene details. To achieve this, the tokens are fed into representation-autoencoder decoders alongside all patch tokens. For image generation, an MHAP predictor estimates registers from sampled patches, leaving the pretrained DiT unchanged. The methods reports results on ImageNet-1K, where it achieves an rFID reduction across various ViT registers with also gFID improving on identical DiT samples.
Critical strengths
- The proposed method is simple with low computational cost and can be used as a drop-in without model re-training
- The paper is well written
- The limitation section is transparent about where the method has shortcomings
- The introduced controls are carefully designed (shuffled and mean registers, learned tokens, paired multi-seed statistics)
Critical weaknesses
- The core finding of the proposed method is close to definitional: In their paper on register tokens, Darcet et al. point out that register tokens accumulate additional global information not captured by the CLS token. Therefore, these tokens naturally contain global appearance information
- The presented ablations fail to clearly demonstrate that decoding registers adds significant benefit over simply using the CLS token (rFID 4.84 → 4.75 with CLS vs. 4.61 with registers). When evaluating the combination of CLS and registers combined, it barely improves on registers alone, which indicated information redundancy rather than a distinct information source
- For generation, the predicted registers are a deterministic function of the patches, so the empirical gain might reflect added capacity and supervision rather than new information
Questions for authors
- Can you make a compelling point why a CLS token cannot capture essentially the same benefit as apposed to additional register tokens?
- Does the presented generation gain hold against a capacity-matched patch-only decoder and with noise-augmented decoders (τ > 0)?
5 comments
Q1: Could CLS capture the same benefit?
We cannot rule out that some CLS interface could close the gap, but none we tested does (W2). Our results suggest why: CLS and registers summarize different aspects of the image.
Each is better at a different task. We added an experiment on Waterbirds (Sagawa et al., 2020). In this dataset, landbirds and waterbirds appear on land or water backgrounds, and bird type correlates with background during training. With the same frozen DINOv2 encoder, CLS is the strongest bird classifier, while registers are the stronger reconstruction input (Tables 10–11):
| Patches | CLS | Registers | |
|---|---|---|---|
| Reconstruction DISTS ↓ | 0.1118 | 0.1109 | 0.1072 |
| Worst-group accuracy (%) ↑ (standalone readout) | 34.7 | 81.1 | 45.8 |
Reconstruction: ImageNet-1K-trained decoders, no adaptation, three seeds; inputs are patches, patches + CLS, and patches + four registers. Classification: linear heads with an equal update budget, three seeds; readouts are patch mean, CLS, and register mean. Worst-group accuracy is the minimum accuracy across the four bird/background groups.
Adding CLS even slightly worsens mean MSE and PSNR compared with patches alone, whereas registers outperform CLS on all four reconstruction metrics in every seed.
A plausible mechanism is the pretraining objective. DINOv2’s DINO loss on CLS compares views that differ in color jitter, blur, and solarization, which pushes CLS toward invariance to the appearance properties the decoder needs. Registers have no direct output loss. Consistent with this, the register mean aligns more closely with pooled patches than CLS does (CKA 0.858 vs. 0.658; Table 6). We present this as a hypothesis; the empirical conclusion does not depend on it.
W3: Do the generation gains reflect added capacity and supervision?
This point concerns generation only. In reconstruction, the decoder receives the encoder’s native registers. In generation, the registers are predicted from the sampled patches, so no new input information is added at generation time. What changes is the readout: MHAP is trained to predict the encoder’s native registers, and the decoder is trained to use them. MHAP receives explicit register-target supervision, while the decoder receives native registers as inputs and is trained with image-reconstruction losses. Two results support this interpretation:
- The decoder relies on sample-specific register values. With decoder, predictor, and patches fixed, shuffling predictions within class worsens gFID from 1.656 to 1.798, and mean registers give 2.316 (Table 24). The gain therefore depends on registers predicted for each specific sample, not merely on additional tokens.
- The tested alternatives with more parameters or internal register supervision do not reproduce the gain. A decoder trained to predict registers internally (88.2M parameters) reaches 1.840, worse than patch-only (1.819). A larger InfoNCE decoder (143.8M vs. 102.4M for the MHAP route) reaches 1.789, compared with 1.656 for MHAP (decoder seed 0; Table 15).
The MHAP route achieves the lowest gFID among these alternatives and improves gFID consistently: by 9.8% with clean decoders and 10.0% with noise-augmented decoders, with every decoder seed favoring it (Q2). The compared variants also differ in architecture, so they do not fully separate register supervision from additional processing (Section 5). The practical result is nonetheless clear: samples from an existing generator decode better without retraining the diffusion model.
Q2: Capacity-matched and noise-augmented decoders
Noise augmentation. We trained patch-only and register-input decoders with τ = 0.8. Each pair decodes the same 50,000 generated patch tensors:
| Decoder training | Patch-only gFID ↓ | MHAP + registers gFID ↓ | Reduction |
|---|---|---|---|
| τ = 0 | 1.8191 ± 0.0058 | 1.6414 ± 0.0133 | 9.77% |
| τ = 0.8 | 1.7516 ± 0.0299 | 1.5771 ± 0.0148 | 9.96% |
Mean ± SD over three decoder seeds, with one fixed MHAP predictor and patch bank.
The relative gain remains approximately 10% under noise augmentation, and all seed pairs favor registers. With τ = 0.8, the ViT-B register decoder (1.577) approaches the much larger released ViT-XL decoder (1.557).
Capacity. The complete MHAP-assisted decoding route adds 16.0M parameters (+18%) and about 20% decoder latency (Table 5), so the comparison is not exactly capacity-matched. However, the larger InfoNCE decoder performs worse (W3). In the ImageNet-100 capacity study, a ViT-B decoder with CLS plus four registers also achieves lower rFID than patch-only ViT-L and ViT-XL decoders trained for the same 16-epoch budget (Table 14).
Dear Reviewer,
Thank you again for your time and feedback. We know this is a busy period and would appreciate hearing whether our rebuttal addresses the points you raised or whether any concerns remain. We would be happy to clarify anything further and discuss any open questions before the discussion closes.
If our updated evidence addresses your concerns, we would be grateful if you would consider raising your score.
Thank you for the careful feedback. We agree that the manuscript should separate three questions more clearly: which type of global information registers carry, whether it is accessible for reconstruction, and whether different global representations are interchangeable. In response, we added CLS and register ablations, per-slot analyses, and noise-augmented generation results, and Section 5 now states the remaining attribution limits. Table numbers refer to the revised manuscript.
W1: Is the benefit already implied by prior work on registers?
We thank the reviewer for this point, which helps us clarify how our work differs from prior work. We agree that registers contain global information; this is our starting point, not our contribution. However, “global information” covers different things: semantic content such as object category, spatial layout, and image-wide appearance such as color, illumination, blur, and texture. Darcet et al. assessed global content through classification probes and discarded registers after encoding. Fel et al. identify appearance-related register concepts through dictionary learning. Decoding shows directly which global information is accessible from registers and whether a decoder uses it.
This is not implied by prior work. Patches are computed with registers present, so much of the register information may be recoverable from them. But image losses do not explicitly supervise recovery of the encoder’s register features. Our results show:
Our results thus identify image-wide appearance information accessible from registers. They also show that the decoder uses this information when it is supplied, rather than readily extracting it from patches.
W2: Does the small benefit of CLS on top of registers imply redundancy?
We thank the reviewer for the careful reading of the ablations. CLS does help, and CLS and registers share information, but their incremental benefits are asymmetric:
rFID from Table 12; R² from Table 9, using the 768-dimensional register mean to match the width of CLS.
Registers give 8.5× (DINOv2) and 11× (DINOv3) the incremental rFID reduction of CLS, and 7.3× the incremental gain in color prediction. CLS therefore adds little to the tested readouts given registers, whereas registers still improve them given CLS.
The gap is clearer in paired fidelity. On ImageNet-1K (three seeds), CLS reaches rFID 0.4968, MSE 0.01381, and DISTS 0.1080, compared with 0.4717, 0.01334, and 0.1044 for registers (Tables 2, 13, 19). Simply repeating CLS at extra token positions does not reproduce the register benefit either: four identical CLS copies perform like one (rFID 0.4931 vs. 0.4968, DISTS unchanged, MSE slightly worse; Table 13).