- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates the “post-release migration” problem in Representation Autoencoders (RAE). RAE uses a pre-trained visual feature grid (semantic field) as a generative latent variable, which is simultaneously read by downstream components such as the decoder and the retrieval gallery. After a producer upgrade, the shape of the field remains unchanged but the meaning of its coordinates changes, rendering the old components obsolete. The authors propose CTFM: under the premise of freezing all models, it learns a globally shared mapping to transform the new field into the old field space. This method first performs a baseline calibration, then uses the output responses of the old components to supervise a small residual, and strictly preserves the pooling coordinates read by the retriever through zero-mean projection (IQ-SAFE). Optionally, a low-rank refinement stage can be added on top of RAEv2. Experiments were conducted on two model pairs, DINOv3→DINOv2 and PE, and the results show that CTFM outperforms baselines of the same capacity in terms of recognition, fidelity, and distribution quality. It requires less training data, leaves retrieval decisions completely unchanged, can be transferred to distribution-shifted data without retraining, and incurs a deployment overhead of approximately 1% of the decoder’s runtime.
Critical strengths
1. Transparent experimental documentation. The paper reports data splits, checkpoint and initialization selection rules in unusual detail, and evaluates with independent observers excluded from fitting. This makes the claims verifiable and the experiments reproducible.
2. Practical feasibility. On RAEv2, a lightweight map (~1M learned parameters, ~1% added latency) lifts decoding from near failure under direct reuse close to the archived field, without modifying any released model. The same map also transfers to real-image distribution shifts without refitting. This shows that post-hoc migration of dense RAE fields can be done cheaply in practice.
Critical weaknesses
-
Weak motivation. Prior compatibility work retains consumers that are costly to replace, e.g., plugin ecosystems [1] or large indices [2]. Here, keeping the old decoder is justified by one sentence (“decoder replacement can require client updates”) without a concrete scenario. Moreover, the paper’s own results suggest that giving each consumer its own map would dominate: the unconstrained QUALITY map decodes better (e.g., PE ResNet-50 58.25 vs. 54.44, Tabs. 12 and 33), while retrieval only needs the base mean. The authors should clarify when a single shared field is actually required.
-
Limited novelty. The base calibration follows the latent translation recipe of [3], which is not cited. On PE, the residual objective is essentially the FFM objective of the VFM baseline (matching the frozen target's final hidden features).
-
Marginal gains over the strongest baseline. The pooled-coordinate guarantee is shared by the projected affine VFM baseline (Tab. 4), over which gains are marginal (53.28 vs. 52.70 Gallery Top-1, Tab. 2) or slightly negative (75.65 vs. 75.897 prospective ResNet-50, Tab. 10).
-
Weak identical-capacity control. A central piece of evidence is the gain over identical-capacity feature matching, but this control degrades the shared initialization (ResNet-50 74.88 → 71.84, PSNR 16.967 → 15.939, Tab. 2). Relative to the initialization, CTFM improves only modestly (to 75.96 and 17.816), so this comparison overstates the benefit of response supervision. Moreover, the reported intervals and seed repeats (Tabs. 11 and 24) only cover CTFM vs. feature matching, so it is unclear whether the 1-point gain over the initialization exceeds variance. Since the reduced-data comparison in Tab. 3 shares the same initialization, the initialization should be reported there as well.
Minor:
I found the main method’s pipeline (figure 2) hard to follow.
[1] Ran et al. X-Adapter: Adding Universal Compatibility of Plugins for Upgraded Diffusion Model. CVPR 2024.
[2] Vejendla. Drift-Adapter: A Practical Approach to Near Zero-Downtime Embedding Model Upgrades in Vector Databases. EMNLP 2025.
[3] Maiorca et al. Latent Space Translation via Semantic Alignment. NeurIPS 2023.
Questions for authors
-
In what concrete deployment scenario can the old decoder not be replaced, and when is a single shared field required instead of separate maps for the decoder and the gallery?
-
How does the base calibration differ from latent space translation, and how does the PE residual objective differ from VFM's FFM?
-
Given that projected affine VFM offers the same retrieval guarantee, what practical advantage justifies CTFM's small recognition gains over it?
-
How much does CTFM improve over the shared initialization on the prospective sets (Tab. 3), and is this gain significant?
6 comments
Response to Reviewer D7vh (2/2)
Additional control: continued fitting of projected affine VFM.
We continued VFM fitting on its own feature loss with the same update budget, selecting checkpoints by either feature loss or the declared response risk. This separates response information used for optimization from information used only for selection. App. D.2 documents the shared learning-rate grid and its further extension for this baseline.
Selected by that loss, it leaves ResNet-50 unchanged on prospective set 1 (75.90) and adds 0.62 dB. Selected by our response risk, it shows no significant difference from CTFM (affine) in 14 of 18 recognition comparisons across two generated sets and four real shifts, and is significantly higher in the other 4.
Starting CTFM's response refinement from this stronger feature fit improves PSNR by 0.15 to 1.07 dB and cycle error in all six populations. Recognition shows no significant change in 13 of 18 comparisons, is significantly better in 4, and significantly worse in 1. LPIPS, which neither response supervises, is 0.01 to 0.02 worse. Relative to continuing the feature fit to 2,400 updates, the two-stage map improves PSNR by 1.10 and 1.04 dB on the two generated sets. App. D.2 specifies its longer, 3,600-update budget.
These controls identify benefits from consumer-response information both during optimization and during checkpoint selection. Revised App. D.2 (Tabs. 13 and 14) reports both selection rules and the two-stage map, including the recognition and LPIPS trade-offs.
W1 and Q1. When is a single shared field required?
A shared field is relevant when one stored or broadcast tensor must remain executable by multiple retained readers. CTFM provides IQ-Safe for this requirement and Quality when a serving layer can deliver separate inputs (Sec. 2.2). Scale-RAE [1] illustrates readers beyond decoding: "the multimodal model can directly reason over generated latents".
On prospective set 1, the Quality affine map trained with the same response loss gains 1.83 ResNet-50 percentage points and 0.21 dB PSNR, and reduces LPIPS by 0.028 relative to the IQ-Safe map. The decoded-gallery difference (+0.09) is not significant, and cycle error slightly favors the shared field. These results quantify the decoder-quality trade-off when separate inputs can be routed to different consumers.
Decision stability under small margins. Unprojected affine VFM changes 99.78% and 99.95% of installed retrieval decisions on ImageNet-R and Sketch (App. H.10, Fig. 10). The smallest top-two score gaps are and (Tab. 41). Releasing 1% of the affine map's pooled displacement flips 0.9% and 1.0% of retrieval decisions while recovering 2.6% and 1.9% of the available PSNR gain. A soft penalty alone does not certify zero flips. The projected maps satisfy the retrieval-margin check on all 7,200 queries.
W2 and Q2. Base calibration and the PE objective.
Thank you for the pointer to Maiorca et al. [2]. Our base starts from an affine alignment between frozen latent spaces of the kind they study, then adds host-specific corrections fitted before repair (App. B.1). We have added this attribution in Sec. 3.1. Base calibration is shared by matched controls. The contribution evaluated here is subsequent response-guided repair under ownership constraints.
We have also made the PE objective explicit in Sec. 3.1: it is final-feature matching under the ownership constraint (App. B.2). PE tests the contract and actuator under this established functional (Tab. 38), gaining 9.06 ResNet-50 percentage points over the matched FFM-to-TLT procedure while preserving the pooled coordinate (Tab. 15). RAEv2 compares the response loss through decoder output and archived re-encoding with feature matching under matched conditions.
Minor. Fig. 2 is hard to follow.
We apologize for the lack of clarity in the previous figure. We have revised Fig. 2 to separate compact repair from affine refinement, highlight the protected pooled-coordinate path, and clarify each host's supervision.
References
[1] Tong et al. Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders. arXiv:2601.16208.
[2] Maiorca et al. Latent Space Translation via Semantic Alignment. NeurIPS 2023.
Thanks for the response. The added experiment has resolved some of my concerns. I still have some followup questions.
-
According to your own results, routing separate inputs to the decoder and the retrieval head is not worse than the shared IQ-Safe field. So when exactly do we need IQ-Safe? Could you give a concrete example in the real world that separate inputs cannot be routed and the archived decoder can't be replaced?
-
IQ-Safe will keep the retrieval input unchanged, but in the submitted version, this only gives only about 3% Top-1 for the linear pooled head, so most of it is wrong. Retrieval only gets good after the Procrustes calibration you added in the rebuttal. Is that calibration now part of your method?
Response to Reviewer D7vh (1/2)
Thank you for requesting a direct comparison with the shared initialization and a closer examination of projected affine VFM. We now report both, including validation-based retuning and uncertainty estimates.
On the same prospective states as Tab. 10, and on the second prospective set of Tab. 3, the submitted CTFM improves its compiled initialization on all seven endpoints, with statistically significant effects (recognition in percentage points):
Every entry is statistically significant under the class bootstrap of Tab. 11. The initialization is fixed before refinement and shared by CTFM and the feature-matching control, so these gains isolate the refinement. As a separate robustness check, Tab. 31 (Tab. 24 in the submission) refits the refinement with three data orders, and its paired effects against feature matching vary with standard deviation at most 0.143 percentage points and 0.006 dB. The revised Tabs. 3 and 10 include the initialization rows, and the new Tab. 12 marks the significance of every effect.
We agree that the margin over feature matching includes the control's own decline. We examined whether additional fitting or learning-rate selection explained this decline. In these runs, validation field error falls monotonically (1.000 to 0.856) and its selector picks the last step, while ResNet-50 and PSNR fall at every saved checkpoint (74.10 to 72.25, 16.90 to 15.89 dB).
Retuned over the same seven learning rates as our affine map, its best run by its own loss ends 7.05 ResNet-50 points below the initialization. Selected by the declared response risk instead, it returns the initialization at every tested rate. The table above therefore measures response refinement directly against the shared starting point.
The submitted prospective result slightly underperformed projected affine VFM on ResNet-50. We therefore repeated refinement with validation-based learning-rate selection. The submitted rate was 10−5. Selection over 10−5 to 10−3 chooses 10−4. The compiled CTFM then improves over projected affine VFM on all seven endpoints on both prospective sets (ResNet-50 +1.27 and +2.97, PSNR +1.41 dB on both sets), every gain significant. This retuning followed evaluation of the affine map to compare both at equal tuning. App. A.6 documents the sequence.
The full affine map trained with the same response loss yields larger gains over projected affine VFM (ResNet-50 +2.32 and +3.06, about +2.0 dB on both sets). It also outperforms projected VFM in all 24 endpoint and domain comparisons on four real shifts (revised Tab. 23). The revision adds per-population class-bootstrap comparisons alongside the submitted macro and observer analyses (Tabs. 22 and 37, numbered 19 and 30 in the submission).
Practical advantage under the same guarantee. Projected affine VFM and CTFM preserve the same pooled coordinate. The matched comparison shows that response supervision improves recognition, fidelity and cycle consistency while retaining this guarantee at the same weight count.