- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper introduces SAGE, a contrastive analysis framework that decomposes a frozen representation autoencoder’s spatial latent into common and target-specific salient factors using only background/target labels. The learned factors enable image reconstruction and attribute manipulation, and serve as conditions for diffusion-based generation. Experiments on Digits-ImageNet, FFHQ, and retinal OCT evaluate subtype discovery, factor manipulation, reconstruction quality, and conditional generation.
Critical strengths
- SAGE provides a unified framework for subtype discovery and image manipulation. Its decodable spatial factors allow discovered attributes to be inspected, removed, and transferred between images, making the representation useful beyond subtype prediction.
- The Digits-ImageNet results provide evidence that the gains extend beyond the use of a strong pretrained encoder. SAGE substantially outperforms unfactorized features and DINOv3 + SepCLR in subtype probe accuracy. The loss ablations also demonstrate that reconstruction quality alone does not ensure successful factor separation.
Critical weaknesses
- The comparisons do not fully isolate the contribution of the proposed factorization. SAGE benefits from a pretrained RAE representation and decoder, whereas the generative CA baselines are trained from scratch. Although DINOv3 + SepCLR provides a useful representation baseline, it is unclear whether its feature extraction exactly matches SAGE’s. Comparisons using alternative factorization objectives within the same frozen RAE would help establish how much of the improvement is attributable to SAGE itself.
- Validation of the claimed representation properties and design choices remains incomplete. Preservation of instance-specific shape, color, and location is supported mainly by qualitative examples rather than controlled attribute-level measurements. Furthermore, the reported experiments do not isolate GMP’s contribution or empirically establish the benefit of reconstruction-based discriminator real samples. The removal ablations are informative, but robustness to loss weighting remains unclear. These gaps limit confidence in both the claimed specificity of the factors and the robustness of the training procedure.
Questions for authors
The questions below concern issues that materially affect my assessment. Clear and convincing responses, supported by existing evidence, targeted experiments where feasible, or a well-substantiated explanation, could address these concerns and lead me to raise my score.
Comparability of the representation baselines
I would like to distinguish the contribution of SAGE’s factorization from differences in the input representation. The paper includes DINOv3 + SepCLR, which is a useful comparison. Does this baseline use exactly the same frozen features as SAGE, including RAEv2’s multi-layer summation?
A more direct control would apply alternative CA objectives to the same frozen RAE representation, using the same decoder wherever applicable. Are such results available?
Preservation of instance-specific attributes
I would like to understand how directly the claim of preserving instance-specific shape, color, and location has been validated. Figures 4 and 6 provide suggestive qualitative evidence, and Figure 7 demonstrates interpolation. However, predicting digit identity or distinguishing reading glasses from sunglasses does not establish preservation of finer instance-specific details.
Could the authors provide a controlled evaluation that varies one attribute while holding the others fixed? For example, synthetic digit images could retain the same background and digit identity while varying position, scale, or stroke shape. Attribute prediction from , together with preservation errors after swapping, would test whether these details are both encoded and transferred. Controlled eyewear examples varying frame shape or color on the same face would serve a similar purpose.
The specific contribution of GMP
I would like to separate the effects of factorization and pooling, particularly regarding location dependence. Section 3.2 explains that GMP retains channels activated at only a few spatial tokens. However, Table 2 compares with , so it does not isolate the benefit of GMP itself.
Could the authors compare spatial conditioning, global average pooling, and global max pooling for and , with conditioning capacity and training settings controlled where feasible? On synthetic digits with known positions, measuring location predictability before and after pooling, alongside subtype accuracy and generation diversity, would provide direct evidence.
Empirical support for avoiding decoder-artifact shortcuts
I would like to understand whether using reconstructions as discriminator real samples addresses an observed failure or an anticipated risk. Section 3.1 provides a plausible rationale: original images and decoded swaps could be distinguished through decoder artifacts rather than the intended attributes. What remains unclear is whether this shortcut materially affects factor learning in this setting.
Could the authors compare original-image and reconstruction-based real samples while keeping the remaining settings fixed? Subtype probe accuracy, common-content leakage, attribute removal and transfer quality, and training stability would help establish the practical effect of this choice.
Sensitivity to loss weights and performance trade-offs
I would like to assess whether successful factorization depends on a narrow balance among the training objectives. Equation 6 specifies unit weights apart from the dataset-specific target-sparsity coefficient , and Table 3 provides informative removal ablations. These results establish the effects of removing objectives, but do not show sensitivity around the chosen weights. Equal coefficients also need not imply comparable contributions when loss or gradient magnitudes differ.
Could the authors report a limited sensitivity study for the swap adversarial, target-sparsity, consistency, and Cycle NCE terms, for example using 0.1, 1, and 10 times their baseline weights? Reporting reconstruction quality, subtype probe and clustering performance, and common-content leakage together would reveal potential trade-offs. In particular, does stronger sparsity reduce leakage at the expense of instance-specific target detail?
7 comments
Response to Q2
Q2: Preservation of instance-specific attributes
A2: We agree that subtype accuracy alone does not establish the preservation of instance-specific details, and we added controlled evaluations on Digits-ImageNet and FFHQ using the SAGE Stage 1 models.
Digits-ImageNet: controlled setup. Following the Digits-ImageNet protocol (a white MNIST digit, 224 px, alpha-composited onto a 512 px ImageNet canvas), we sample 100 triplets of a source background , a host background (both digit-free) and an MNIST-test instance (10 per class). Within a triplet only the position varies: is pasted on at each cell of a grid spanning the training range, and its salient code is swapped onto , , giving 900 swaps. Since the data are synthetic, the ground-truth digit mask is known exactly. We recover the swapped digit by inverting the white compositing against the common-only decoding , and apply the same extraction to the plain reconstruction of the ideal composite ( pasted on ) as a reference for the error of decoding and extraction combined. Digit scale and color are fixed in Digits-ImageNet (224 px, white), so we vary position and stroke shape.
Location. We report the centroid distance (256 px images) between the swapped digit and its true position:
| Mean (px) ↓ | Median (px) ↓ | < 8 px ↑ | < 16 px (1 token) ↑ | |
|---|---|---|---|---|
| Reconstruction (reference) | 3.2 | 2.4 | 93.1% | 99.2% |
| Swap | 4.5 | 3.6 | 89.3% | 98.3% |
98.3% of swapped digits land within one 16 px token of the source position, only 1.3 px above reconstruction. Location is also directly readable from the code: the centroid of the token-norm map lies 3.9 px from the digit.
Shape. We paste the source instance and five other instances of the same class at the same position on , and assign each swap to the candidate whose ground-truth mask has the highest IoU with the swapped digit. The source instance is selected in 91.3% of cases (chance 16.7%), so the transferred shape is instance-specific rather than a class prototype.
FFHQ. We transfer 300 held-out eyewear donors (150 reading glasses, 150 sunglasses) onto 600 eyewear-free hosts. Beyond eyewear type, we evaluate same-subtype donor retrieval using SAM3 eyewear masks. For each swap, we rank 10 donors of the same subtype by mask IoU, with the true donor ranking first in 89.2% of cases (chance: 10%). While this metric can be affected by differences in head pose and facial geometry, it provides complementary evidence of donor-specific eyewear preservation beyond subtype identity.
| Dataset | Swaps | Subtype Acc. (source) ↑ | Subtype Acc. (swap) ↑ |
|---|---|---|---|
| Digits-ImageNet | 900 | 99.7% | 94.7% |
| FFHQ | 600 | 96.7% | 92.8% |
Subtype accuracy is measured using the same DINOv3-based judge from Table 2, trained only on real target-group images.
Our controlled experiments demonstrate substantial, though not exact, preservation of instance-specific location and shape. We also observe occasional variations in finer appearance details, such as saturated eyewear frame colors becoming darker after swapping, potentially due to reconstruction decoder limitations. Importantly, these variations do not contradict our primary claim of subtype-level attribute preservation. We will include these evaluations in the revised appendix.
Response to Q3
Q3 The specific contribution of GMP
A3 We agree that the original Table 2 does not isolate the contributions of salient factorization and GMP. We therefore conducted an additional 2 × 2 ablation on Digits-ImageNet, comparing GAP and GMP on both the original DINOv3 representation and the learned salient representation .
| Conditioning | Uncond. gFID ↓ | Cond. gFID ↓ | Subtype Acc. ↑ | Vendi ↑ | Cos-to-ref ↓ |
|---|---|---|---|---|---|
| Real images | – | – | 99.30 | 15.79 | – |
| Raw DINOv3 GMP() | 8.86 | 2.40 | 27.70 | 2.65 | 0.794 |
| Raw DINOv3 GAP() | 5.68 | 2.20 | 27.82 | 2.37 | 0.823 |
| SAGE Salient GAP() | 5.59 | 2.50 | 85.98 | 2.89 | 0.774 |
| SAGE Salient GMP() | 4.76 | 4.09 | 90.48 | 12.50 | 0.116 |
We draw two conclusions from the experiments.
First, salient factorization is essential for subtype preservation. Conditioning on substantially improves subtype accuracy over the original DINOv3 features under both GAP (27.82% → 85.98%) and GMP (27.70% → 90.48%). This confirms that the improvement primarily comes from the learned salient representation rather than pooling alone.
Second, GMP complements salient factorization by further reducing background dependence. We observe that although GAP() effectively preserves subtype information, the generated samples still exhibit some dependence on the reference background, as reflected by high reference similarity (0.774) and limited diversity (Vendi = 2.89). In contrast, GMP() substantially reduces reference similarity to 0.116 and increases Vendi to 12.50, while further improving subtype accuracy to 90.48%. Despite a higher conditional gFID, these results suggest that GMP better suppresses background-related dependencies while retaining subtype-specific information, enabling more diverse subtype-conditioned generation.
Regarding spatial conditioning and location predictability, our primary objective in Stage 2 is to preserve subtype identity while promoting generation diversity, rather than retaining the reference instance's spatial configuration. We have not included full spatial conditioning experiments at this stage due to computational constraints.
We greatly appreciate the reviewer's constructive feedback!
For W1 and W2, please refer to our detailed responses to Q1 and Q2–Q5 below.
Q1: Comparability of the representation baselines
A1: Yes, DINOv3 + SepCLR uses exactly the same frozen features as SAGE, including RAEv2's multi-layer summation. The remaining loss design follows the original SepCLR.
We also implemented a cVAE baseline using the same frozen RAE representation in our early exploratory experiments. Specifically, we kept the frozen RAE encoder, reconstruction decoder, and common/salient encoder architectures identical to SAGE, while training the common and salient encoder heads using the original cVAE ELBO objective. This allows us to directly assess the contribution of the factorization objective while controlling for differences in the representation and network architecture.
However, we encountered two main challenges:
These considerations motivated our use of a sparsity prior for the salient branch, together with swap-GAN and cycle-consistency losses to encourage factor disentanglement without explicitly estimating TC/MI.
We provide the results of our RAE-based cVAE implementation below for reference.
As shown in the table, cVAE-RAE substantially improves over the original cVAE baseline, demonstrating the benefits of pretrained RAE representations. However, it still falls considerably behind SAGE in salient factorization quality. For example, SAGE improves the BG/TG silhouette score from 0.173 to 0.820 on Digits-ImageNet and from 0.034 to 0.873on FFHQ, with similarly substantial improvements in clustering metrics.
Since the underlying RAE representation, reconstruction decoder, and encoder architectures are identical, these results provide direct evidence that SAGE's improvements arise not merely from stronger pretrained features, but also from its factorization objective.