acceptodds
Under review as a conference paper at ICLR 2027

SAGE: Salient Factor Discovery and Generation with Visual Foundation Representations

Abstract

Given a target dataset, such as faces with eyeglasses, and a background dataset, such as faces without, contrastive analysis separates salient factors specific to the target from common content shared by both. We aim for salient representations that capture target-specific detail in each image, such as the shape, color, and position of the glasses, so that they reveal subtypes without subtype labels and guide the generation of new examples of a discovered subtype, even one with no name or text description. We introduce SAGE, which learns both factors directly in the high-dimensional spatial latent of a frozen representation autoencoder and conditions a diffusion transformer on the learned salient representation of a reference image. On Digits-ImageNet and FFHQ eyeglasses, SAGE combines high-fidelity reconstruction (rFID below 2) with unsupervised subtype discovery, recovering the digits better than baselines (probe accuracy 0.950 vs. at most 0.281) and revealing eyewear types, finer sunglasses styles, and mislabeled images; salient-conditioned generation raises Digits-ImageNet subtype accuracy over the unfactorized latent (90.5% vs. 27.7%) and diversity on both datasets. On retinal OCT, SAGE's salient space separates three diseases using only normal/disease labels.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Contrastive Analysis, Latent-Space Factorization, Representation Autoencoders, Unsupervised Subtype Discovery, Salient-Conditioned Generation

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_nSLwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SAGE, a contrastive analysis framework that decomposes a frozen representation autoencoder’s spatial latent into common and target-specific salient factors using only background/target labels. The learned factors enable image reconstruction and attribute manipulation, and serve as conditions for diffusion-based generation. Experiments on Digits-ImageNet, FFHQ, and retinal OCT evaluate subtype discovery, factor manipulation, reconstruction quality, and conditional generation.

Critical strengths
  1. SAGE provides a unified framework for subtype discovery and image manipulation. Its decodable spatial factors allow discovered attributes to be inspected, removed, and transferred between images, making the representation useful beyond subtype prediction.

  1. The Digits-ImageNet results provide evidence that the gains extend beyond the use of a strong pretrained encoder. SAGE substantially outperforms unfactorized features and DINOv3 + SepCLR in subtype probe accuracy. The loss ablations also demonstrate that reconstruction quality alone does not ensure successful factor separation.
Critical weaknesses
  1. The comparisons do not fully isolate the contribution of the proposed factorization. SAGE benefits from a pretrained RAE representation and decoder, whereas the generative CA baselines are trained from scratch. Although DINOv3 + SepCLR provides a useful representation baseline, it is unclear whether its feature extraction exactly matches SAGE’s. Comparisons using alternative factorization objectives within the same frozen RAE would help establish how much of the improvement is attributable to SAGE itself.

  1. Validation of the claimed representation properties and design choices remains incomplete. Preservation of instance-specific shape, color, and location is supported mainly by qualitative examples rather than controlled attribute-level measurements. Furthermore, the reported experiments do not isolate GMP’s contribution or empirically establish the benefit of reconstruction-based discriminator real samples. The removal ablations are informative, but robustness to loss weighting remains unclear. These gaps limit confidence in both the claimed specificity of the factors and the robustness of the training procedure.
Questions for authors

The questions below concern issues that materially affect my assessment. Clear and convincing responses, supported by existing evidence, targeted experiments where feasible, or a well-substantiated explanation, could address these concerns and lead me to raise my score.


Comparability of the representation baselines

I would like to distinguish the contribution of SAGE’s factorization from differences in the input representation. The paper includes DINOv3 + SepCLR, which is a useful comparison. Does this baseline use exactly the same frozen features as SAGE, including RAEv2’s multi-layer summation?

A more direct control would apply alternative CA objectives to the same frozen RAE representation, using the same decoder wherever applicable. Are such results available?


Preservation of instance-specific attributes

I would like to understand how directly the claim of preserving instance-specific shape, color, and location has been validated. Figures 4 and 6 provide suggestive qualitative evidence, and Figure 7 demonstrates interpolation. However, predicting digit identity or distinguishing reading glasses from sunglasses does not establish preservation of finer instance-specific details.

Could the authors provide a controlled evaluation that varies one attribute while holding the others fixed? For example, synthetic digit images could retain the same background and digit identity while varying position, scale, or stroke shape. Attribute prediction from zsz_s, together with preservation errors after swapping, would test whether these details are both encoded and transferred. Controlled eyewear examples varying frame shape or color on the same face would serve a similar purpose.


The specific contribution of GMP

I would like to separate the effects of factorization and pooling, particularly regarding location dependence. Section 3.2 explains that GMP retains channels activated at only a few spatial tokens. However, Table 2 compares GMP(z)\mathrm{GMP}(z) with GMP(zs)\mathrm{GMP}(z_s), so it does not isolate the benefit of GMP itself.

Could the authors compare spatial conditioning, global average pooling, and global max pooling for zz and zsz_s, with conditioning capacity and training settings controlled where feasible? On synthetic digits with known positions, measuring location predictability before and after pooling, alongside subtype accuracy and generation diversity, would provide direct evidence.


Empirical support for avoiding decoder-artifact shortcuts

I would like to understand whether using reconstructions as discriminator real samples addresses an observed failure or an anticipated risk. Section 3.1 provides a plausible rationale: original images and decoded swaps could be distinguished through decoder artifacts rather than the intended attributes. What remains unclear is whether this shortcut materially affects factor learning in this setting.

Could the authors compare original-image and reconstruction-based real samples while keeping the remaining settings fixed? Subtype probe accuracy, common-content leakage, attribute removal and transfer quality, and training stability would help establish the practical effect of this choice.


Sensitivity to loss weights and performance trade-offs

I would like to assess whether successful factorization depends on a narrow balance among the training objectives. Equation 6 specifies unit weights apart from the dataset-specific target-sparsity coefficient λ\lambda, and Table 3 provides informative removal ablations. These results establish the effects of removing objectives, but do not show sensitivity around the chosen weights. Equal coefficients also need not imply comparable contributions when loss or gradient magnitudes differ.

Could the authors report a limited sensitivity study for the swap adversarial, target-sparsity, consistency, and Cycle NCE terms, for example using 0.1, 1, and 10 times their baseline weights? Reporting reconstruction quality, subtype probe and clustering performance, and common-content leakage together would reveal potential trade-offs. In particular, does stronger sparsity reduce leakage at the expense of instance-specific target detail?

7 comments
AuthorsOpenReview ↗

We greatly appreciate the reviewer's constructive feedback!

For W1 and W2, please refer to our detailed responses to Q1 and Q2–Q5 below.

Q1: Comparability of the representation baselines

A1: Yes, DINOv3 + SepCLR uses exactly the same frozen features as SAGE, including RAEv2's multi-layer summation. The remaining loss design follows the original SepCLR.

We also implemented a cVAE baseline using the same frozen RAE representation in our early exploratory experiments. Specifically, we kept the frozen RAE encoder, reconstruction decoder, and common/salient encoder architectures identical to SAGE, while training the common and salient encoder heads using the original cVAE ELBO objective. This allows us to directly assess the contribution of the factorization objective while controlling for differences in the representation and network architecture.

However, we encountered two main challenges:

  1. Gaussian prior regularization. cVAE imposes Gaussian priors on the learned salient and common representations, which may conflict with preserving the rich semantic structure of high-dimensional RAE features.
  2. TC/MI estimation. Methods such as SepVAE and cVAE employ discriminator-based density-ratio estimation to encourage independence between salient and common factors. Although SepVAE improves the optimization procedure, estimating these dependencies remains challenging for high-dimensional spatial features.

These considerations motivated our use of a sparsity prior for the salient branch, together with swap-GAN and cycle-consistency losses to encourage factor disentanglement without explicitly estimating TC/MI.

We provide the results of our RAE-based cVAE implementation below for reference.

Digits-ImageNetFFHQ
ReconstructionSalient representationReconstructionSalient representation
MethodrFID ↓PSNR ↑SSIM ↑LPIPS ↓BG/TG Sil. ↑LP Acc. ↑ARI/NMI ↑rFID ↓PSNR ↑SSIM ↑LPIPS ↓BG/TG Sil. ↑LP Acc. ↑ARI/NMI ↑
cVAE (Abid and Zou, 2019)151.018.330.4780.6730.0060.1350.001/0.003188.218.420.5670.5900.0270.8980.001/0.001
cVAE-RAE23.4123.300.6620.3420.1730.4530.000/0.00159.9922.710.7010.3930.0340.9540.000/0.002
SAGE (Ours)1.7820.560.5710.2160.8200.9500.337/0.4721.6523.020.7230.1630.8730.9830.939/0.865

As shown in the table, cVAE-RAE substantially improves over the original cVAE baseline, demonstrating the benefits of pretrained RAE representations. However, it still falls considerably behind SAGE in salient factorization quality. For example, SAGE improves the BG/TG silhouette score from 0.173 to 0.820 on Digits-ImageNet and from 0.034 to 0.873on FFHQ, with similarly substantial improvements in clustering metrics.

Since the underlying RAE representation, reconstruction decoder, and encoder architectures are identical, these results provide direct evidence that SAGE's improvements arise not merely from stronger pretrained features, but also from its factorization objective.

AuthorsOpenReview ↗

Response to Q2

Q2: Preservation of instance-specific attributes

A2: We agree that subtype accuracy alone does not establish the preservation of instance-specific details, and we added controlled evaluations on Digits-ImageNet and FFHQ using the SAGE Stage 1 models.

Digits-ImageNet: controlled setup. Following the Digits-ImageNet protocol (a white MNIST digit, 224 px, alpha-composited onto a 512 px ImageNet canvas), we sample 100 triplets (A,B,m)(A, B, m) of a source background AA, a host background BB (both digit-free) and an MNIST-test instance mm (10 per class). Within a triplet only the position varies: mm is pasted on AA at each cell of a 3×33\times3 grid spanning the training range, and its salient code is swapped onto BB, x^=D(zcB,zsT)\hat{x} = D(z_c^{B}, z_s^{T}), giving 900 swaps. Since the data are synthetic, the ground-truth digit mask is known exactly. We recover the swapped digit by inverting the white compositing against the common-only decoding D(zcB,0)D(z_c^{B}, 0), and apply the same extraction to the plain reconstruction of the ideal composite (mm pasted on BB) as a reference for the error of decoding and extraction combined. Digit scale and color are fixed in Digits-ImageNet (224 px, white), so we vary position and stroke shape.

Location. We report the centroid distance (256 px images) between the swapped digit and its true position:

Mean (px) ↓Median (px) ↓< 8 px ↑< 16 px (1 token) ↑
Reconstruction (reference)3.22.493.1%99.2%
Swap D(zcB,zsT)D(z_c^{B}, z_s^{T})4.53.689.3%98.3%

98.3% of swapped digits land within one 16 px token of the source position, only 1.3 px above reconstruction. Location is also directly readable from the code: the centroid of the zsz_s token-norm map lies 3.9 px from the digit.

Shape. We paste the source instance and five other instances of the same class at the same position on BB, and assign each swap to the candidate whose ground-truth mask has the highest IoU with the swapped digit. The source instance is selected in 91.3% of cases (chance 16.7%), so the transferred shape is instance-specific rather than a class prototype.

FFHQ. We transfer 300 held-out eyewear donors (150 reading glasses, 150 sunglasses) onto 600 eyewear-free hosts. Beyond eyewear type, we evaluate same-subtype donor retrieval using SAM3 eyewear masks. For each swap, we rank 10 donors of the same subtype by mask IoU, with the true donor ranking first in 89.2% of cases (chance: 10%). While this metric can be affected by differences in head pose and facial geometry, it provides complementary evidence of donor-specific eyewear preservation beyond subtype identity.

DatasetSwapsSubtype Acc. (source) ↑Subtype Acc. (swap) ↑
Digits-ImageNet90099.7%94.7%
FFHQ60096.7%92.8%

Subtype accuracy is measured using the same DINOv3-based judge from Table 2, trained only on real target-group images.

Our controlled experiments demonstrate substantial, though not exact, preservation of instance-specific location and shape. We also observe occasional variations in finer appearance details, such as saturated eyewear frame colors becoming darker after swapping, potentially due to reconstruction decoder limitations. Importantly, these variations do not contradict our primary claim of subtype-level attribute preservation. We will include these evaluations in the revised appendix.

AuthorsOpenReview ↗

Response to Q3

Q3 The specific contribution of GMP

A3 We agree that the original Table 2 does not isolate the contributions of salient factorization and GMP. We therefore conducted an additional 2 × 2 ablation on Digits-ImageNet, comparing GAP and GMP on both the original DINOv3 representation zz and the learned salient representation zsz_s.

ConditioningUncond. gFID ↓Cond. gFID ↓Subtype Acc. ↑Vendi ↑Cos-to-ref ↓
Real images––99.3015.79–
Raw DINOv3 GMP(zz)8.862.4027.702.650.794
Raw DINOv3 GAP(zz)5.682.2027.822.370.823
SAGE Salient GAP(zsz_s)5.592.5085.982.890.774
SAGE Salient GMP(zsz_s)4.764.0990.4812.500.116

We draw two conclusions from the experiments.

First, salient factorization is essential for subtype preservation. Conditioning on zsz_s substantially improves subtype accuracy over the original DINOv3 features under both GAP (27.82% → 85.98%) and GMP (27.70% → 90.48%). This confirms that the improvement primarily comes from the learned salient representation rather than pooling alone.

Second, GMP complements salient factorization by further reducing background dependence. We observe that although GAP(zsz_s) effectively preserves subtype information, the generated samples still exhibit some dependence on the reference background, as reflected by high reference similarity (0.774) and limited diversity (Vendi = 2.89). In contrast, GMP(zsz_s) substantially reduces reference similarity to 0.116 and increases Vendi to 12.50, while further improving subtype accuracy to 90.48%. Despite a higher conditional gFID, these results suggest that GMP better suppresses background-related dependencies while retaining subtype-specific information, enabling more diverse subtype-conditioned generation.

Regarding spatial conditioning and location predictability, our primary objective in Stage 2 is to preserve subtype identity while promoting generation diversity, rather than retaining the reference instance's spatial configuration. We have not included full spatial conditioning experiments at this stage due to computational constraints.

Reviewer_YXDiOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new, two-stage method (SAGE) to perform salient factor discovery and generations based on them. Specifically, in stage 1, it factorizes the high-dimensional spatial latent of a frozen representation autoencoder into two factors (salient representations which capture factors specific to target data, and common representations that hold common content shared by both target and background data) using two trainable encoders. Then, in stage 2, keeping the encoder learned in stage1 frozen, they train RAEv2-initialized diffusion transformer by conditioning it on the learned salient representations to generate new samples that keep the salient factors intact across varying common contents. They demonstrate the superior performance of their method over the baselines for reconstruction, common/salient-only decoding, discovering subtypes without labels, discovering mislabeled images, and factor manipulation, with qualitative as well as quantitative results using multiple datasets (Digits-ImageNet, FFHQ, OCT-Kermany).

Critical strengths
  1. The proposed method is able to both factorize the latent as well as perform reconstruction and generation well. Prior methods were not able to achieve both together.
  2. The method is able to reveal subtypes without being provided with their labels, as well as able to discover mislabeled images, which was interesting.
  3. Multiple qualitative and quantitative evaluations have been provided which makes it easier to understand the benefits of the proposed method.
Critical weaknesses
  1. DiffCA is included in related works, along with cVAE and others, under contrastive analysis; however, it is not included as one of the baselines. Since it is related to the diffusion model, it may be a more directly relevant or closer baseline than cVAE; so, including it would make the list of baselines more complete. If DiffCA representing salient factor as a low-dimensional vector or global token is the reason, doesn’t it apply to cVAE as well? If there is any other reason, it will be good to explicitly mention it in the paper for clarity.

  2. The full objective, eq(6), includes several types of losses. The authors have performed loss ablation in Section 4.7 considering Digits-ImageNet, and have shown that having all of them gives the best result overall (this is good). However, doesn’t balancing those losses during training become challenging? Did the authors experience one or more losses dominating the entire objective for any dataset? Along the same line, besides lambda on loss_TG-sp the value of which the authors mention in Appendix C.2, did the authors need to tune/apply weights on any of the other losses, or did the authors use weight 1 for all others in all datasets? Providing a discussion on this will be useful and can support its use for new datasets.

Once the aforementioned points are addressed, I am happy to increase my rating.

Questions for authors

Please refer to the Weaknesses section.

5 comments
AuthorsOpenReview ↗

We sincerely appreciate the reviewer's positive feedback and their recognition of our work!

Q1: Diff-CA Baseline.

A1: We could not find an open-source implementation of Diff-CA before our submission. However, we recently identified an anonymous repository (anonymous.4open.science/r/Diffusion-Contrastive-Analysis-0F66) that appears to provide a faithful implementation. Following its recipe, we trained Diff-CA on the same FFHQ and Digits-ImageNet datasets used in SAGE. The results are presented below.

Digits-ImageNetFFHQ
ReconstructionSalient representationReconstructionSalient representation
MethodrFID ↓PSNR ↑SSIM ↑LPIPS ↓BG/TG Sil. ↑LP Acc. ↑ARI/NMI ↑rFID ↓PSNR ↑SSIM ↑LPIPS ↓BG/TG Sil. ↑LP Acc. ↑ARI/NMI ↑
Diff-CA5.7819.220.4970.3470.8200.1750.005/0.0168.3519.710.5840.3130.9130.9650.768/0.618
SAGE (Ours)1.7820.560.5710.2160.8200.9500.337/0.4721.6523.020.7230.1630.8730.9830.939/0.865

SAGE consistently outperforms Diff-CA in reconstruction quality, salient-factor linear probing, and ARI/NMI clustering on both datasets, while achieving comparable background-target separation in terms of silhouette scores.

In particular, on Digits-ImageNet, SAGE achieves substantially higher salient-factor linear probing accuracy (0.950 vs. 0.175) and ARI (0.337 vs. 0.005), indicating that its learned salient representations more effectively capture the target-specific factors. Similar improvements are observed on FFHQ, where SAGE achieves an ARI of 0.939 compared with 0.768 for Diff-CA, alongside better reconstruction quality.

These results provide additional empirical evidence for the effectiveness of SAGE's spatial representation factorization compared with diffusion-based contrastive analysis using compact salient codes.

We will incorporate Diff-CA as an additional baseline and clarify the experimental protocols in the revised manuscript.

AuthorsOpenReview ↗

Response to Q2

Q2: Discussion on the loss weights and training balance.

A2: We did not observe any individual loss dominating the overall objective during training. Given the number of loss terms, an exhaustive joint grid search over their coefficients was beyond our computational budget. We therefore used unit coefficients as a simple default for all terms except TG-sp across all datasets. Only the TG-sp coefficient was adjusted on FFHQ and OCT.

We treated TG-sp as dataset-dependent because it encourages sparsity in the salient representation, while different salient factors may require different amounts of information to be retained. In particular, our intuition was that eyewear in FFHQ may require more representational capacity than digits in Digits-ImageNet and therefore benefit from weaker sparsity regularization.

We conducted an extensive set of additional training experiments to assess sensitivity to individual loss weights on Digits-ImageNet. Each multiplier below is relative to the corresponding baseline coefficient.

SettingrFID↓PSNR↑SSIM↑LPIPS↓Sil↑LP↑ARI/NMI↑
SAGE Baseline1.7820.560.5710.2160.8200.9500.337/0.472
cNCE ×0.11.6520.840.5820.2100.7890.9480.409/0.529
cNCE ×103.5919.310.5370.2560.5270.9380.371/0.484
TG-sp ×0.11.2620.760.5870.2010.8380.9420.290/0.422
TG-sp ×101.8720.190.5580.2300.6150.9330.357/0.475
L_cyc ×0.12.3219.280.5620.2270.3400.9440.341/0.480
L_cyc ×101.7619.970.5450.2350.7590.9430.199/0.342
L_lsc ×0.11.8020.170.5650.2230.6460.9490.457/0.558
L_lsc ×102.3419.520.5600.2250.3210.9470.339/0.469
swap ×0.11.3121.220.5950.1940.4830.9310.127/0.260
swap ×105.3917.970.4950.3000.7110.9540.503/0.613

Subtype probe performance remains high across the tested configurations (LP: 0.931–0.954), suggesting that retaining useful salient information does not require narrowly tuned individual coefficients. Reconstruction and clustering nevertheless exhibit weight-dependent trade-offs.

We additionally report the TG-sp tuning results on FFHQ. We initially used a coefficient of 1.0 and subsequently evaluated 0.1 and 0.01, selecting 0.01 for the final model.

TG-sp weightrFID ↓PSNR ↑SSIM ↑LPIPS ↓Sil. ↑LP ↑ARI/NMI ↑
1.0 (×100)2.6422.270.7010.1730.5840.9580.806/0.665
0.1 (×10)2.7722.990.7250.1630.8790.9830.916/0.827
0.01 (SAGE, ×1)1.6523.020.7230.1630.8730.9830.939/0.865

On FFHQ, reducing TG-sp from 1.0 to 0.01 improved both rFID (2.64 → 1.65) and ARI/NMI (0.806/0.665 → 0.939/0.865). On Digits-ImageNet, reducing it from 1.0 to 0.1 improved reconstruction but reduced ARI/NMI (0.337/0.472 → 0.290/0.422). These different trade-offs support dataset-dependent sparsity strengths and are consistent with our intuition that FFHQ benefits from allowing more information to be retained in the salient representation. We will include these results and the accompanying discussion in the appendix of the revised manuscript.

Thank you for the rebuttal. I appreciate the author's efforts on conducting additional experiments. The DiffCA baseline strengthens the evaluation, and the discussion on loss weights presents useful insights on tradeoffs and when narrowly tuned individual coefficients are not required. These address my concerns. I have decided to increase my score to Clear acceptance with the expectation that these new results and discussion will be incorporated in the revised manuscript as indicated in the comments by author.

Reviewer_vdhtOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper aims to identify the salient features in a data set obtain highly accurate reconstruction and a clear separation of subtypes. Using contrastive analysis, the latent produced by a frozen representation encoder is split into salient factors that capture shared content and target-specific variation. This separation is obtained by objectives that include swapping salient factors between background and target images, enforcing them to be sparce sparse. These separated features can then be applied in a generation process to modify a given image towards a specific subtype.

Critical strengths
  • The paper presents an interesting pipeline with a non-trivial approach to split up the latent into salient target features.
  • The presented results clearly can separate out the salient features of the target group in a given image.
  • The separated features can more easily be analyzed to obtain subclasses of the target.
  • The approach is likely applicable to a broad range of applications, here demonstrated for general images and OCT retinal scans.
Critical weaknesses
  • Currently, there are a couple of nice and convincing experiments on the feature separation and even removal of specific features. What is largely missing is a discussion of failure cases, or insights how good one can expect the method to work in a novel context. What happens if target and background are way more similar?
Questions for authors
  • As the entire pipeline is quite involved it would be nice to get an impression on the training effort and convergence.

  • Did you encounter a specific target group where the separation did not work as nicely?

2 comments
AuthorsOpenReview ↗

We sincerely appreciate the reviwer's positive feedback and their recognition of our work!

W1 Currently, there are a couple of nice and convincing experiments on the feature separation and even removal of specific features. What is largely missing is a discussion of failure cases, or insights how good one can expect the method to work in a novel context. What happens if target and background are way more similar?

R1: In our exploratory experiment using AI-edited images from ImgEdit (https://arxiv.org/abs/2505.20275) as the target and ImageNet images as the background, SAGE did not clearly disentangle editing-related factors from shared content. This suggests a limitation when target and background distributions are highly similar, particularly when their differences are subtle or insufficiently distinguishable in the DINOv3 feature space. SAGE's effectiveness depends on whether the pretrained feature space can sufficiently distinguish the salient differences between target and background distributions. We will include this failure case and discuss this applicability boundary in the revised manuscript.

Q1 As the entire pipeline is quite involved it would be nice to get an impression on the training effort and convergence.

A1: Stage 1 trains for 3 epochs on Digits-ImageNet (14.6 hrs on 4×H200, ~100hrs on 1xH200), 200 epochs on FFHQ (14.9 hrs on 4×H200), and 60 epochs on OCT-Kermany (48.6 hrs on 2×H100). Stage 2 trains on target images for 20 epochs on Digits-ImageNet (14.7 hrs on 4×H200) and 40 epochs on FFHQ (8.9 hrs on 4×H200). Stage 1 losses plateau: over the last 20% of training the total loss changes by less than 2.5% and the reconstruction terms by less than 2.5% on all three datasets.

The Stage 2 flow loss changes by 0.8% (Digits-ImageNet) and 4.1% (FFHQ) over the same interval. On FFHQ we deliberately stop Stage 2 early. In a preliminary 200-epoch run, the model began to memorize the pairing between conditions and target images: the mean pixel distance between two samples drawn with different seeds under the same condition fell from 55 at epoch 20 to about 17 at epoch 200, a 70% drop, whereas their distance to the input decreased by only 20%. Samples sharing a condition thus converged toward nearly the same image, reducing conditional diversity. We attribute this to the small FFHQ target set (about 10k images). With the 40-epoch budget, this diversity stays above 55 throughout training.

Q2 Did you encounter a specific target group where the separation did not work as nicely?

A2 Yes. AI-edited images were a challenging target group. To probe the generalization limits of SAGE, we conducted an additional experiment using edited images from ImgEdit(https://arxiv.org/abs/2505.20275) as the target dataset and real images from ImageNet as the background dataset. In this setting, SAGE did not clearly isolate editing-related factors from the shared image content in its learned salient representations. This highlights a limitation of the current method under this target–background configuration.

Extending SAGE and visual foundation model-based contrastive analysis to better capture and attribute AI-generated edits in latent space would be an interesting direction for future work. This approach might potentially complement pixel-level localization methods, which are the mainstream for AIGC content detection.

Thank you for your clarifications. I do keep my positive score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Discussion Period

Dear Reviewers and ACs,

We sincerely thank the reviewers and ACs for their constructive feedback, which has substantially strengthened our evaluation and presentation.

Positive Consensus

Unified factor discovery and generation. SAGE combines salient-factor discovery, high-fidelity reconstruction, attribute manipulation, and subtype-conditioned generation within a single framework.

Meaningful subtype discovery without subtype labels. The learned salient representations reveal target subtypes and mislabeled examples using only background/target labels. Reviewers recognized the effective factor separation and the substantial improvements over unfactorized features and representation-based baselines.

Extensive quantitative and qualitative evaluation. Experiments on Digits-ImageNet, FFHQ, and retinal OCT demonstrate effective salient-factor separation and subtype discovery, while reconstruction, factor manipulation, and salient-conditioned generation provide complementary evidence of the learned representations’ quality and utility.

Concerns

C1. Comparability of Representation Baselines (nSLw Q1). Separating the contribution of factorization from that of pretrained representations.

R1. DINOv3 + SepCLR uses exactly the same frozen RAEv2 features as SAGE, including multi-layer summation and latent normalization. With the frozen RAE encoder, decoder, and common/salient encoder architectures also held fixed, SAGE achieves a Digits-ImageNet probe accuracy of 0.950 versus 0.453 for cVAE-RAE, and an FFHQ ARI of 0.939 versus 0.000. These comparisons demonstrate the benefit of SAGE’s factorization objective within the same pretrained representation space.

C2. Diff-CA Baseline (YXDi Q1).

R2. SAGE outperforms the reproduced Diff-CA baseline in reconstruction, subtype probing, and clustering on both datasets. On Digits-ImageNet, SAGE achieves a probe accuracy of 0.950 versus 0.175 and an rFID of 1.78 versus 5.78; on FFHQ, it achieves an ARI of 0.939 versus 0.768 and an rFID of 1.65 versus 8.35. These results show stronger subtype discovery alongside higher reconstruction fidelity.

C3. Preservation of Instance-Specific Attributes (nSLw Q2).

R3. Controlled Stage 1 evaluations show that SAGE transfers instance-specific attributes alongside subtype identity. Across 900 digit swaps, mean location error is 4.5 px, close to the reconstruction reference of 3.2 px, and source-instance shape retrieval reaches 91.3% against 16.7% chance. Across 600 eyewear swaps, donor retrieval among same-subtype candidates reaches 89.2% against 10% chance. The spatial salient factors thus retain both positional and appearance information specific to the source instance.

C4. Contribution of GMP (nSLw Q3).

R4. A 2 × 2 comparison shows that factorization improves subtype control under either pooling operation: subtype accuracy increases from 27.82% to 85.98% with GAP and from 27.70% to 90.48% with GMP. On salient representations, GMP further increases Vendi diversity from 2.89 to 12.50 and reduces reference similarity from 0.774 to 0.116, with a trade-off in conditional gFID. Factorization provides subtype information, while GMP supports more diverse generation with less reference dependence.

C5. Empirical support for reconstruction-based supervision. (nSLw Q4).

R5. Using reconstructions as discriminator real samples improves subtype discovery with all other training settings fixed. ARI/NMI increases from 0.202/0.326 to 0.337/0.472 on Digits-ImageNet and from 0.122/0.129 to 0.415/0.415 on OCT. This consistent improvement supports reconstruction-based real samples as an effective choice for learning salient factors.

C6. Loss-Weight Sensitivity and Trade-offs (YXDi Q2, nSLw Q5).

R6. SAGE uses unit weights across datasets for all objectives except target-salient sparsity. Varying individual coefficients by ×0.1 and ×10 preserves high Digits-ImageNet probe accuracy (0.931–0.954), showing that subtype discovery does not require narrowly tuned weights within the tested range. The weights nevertheless affect reconstruction, separation, and clustering, while the FFHQ sparsity study illustrates the role of dataset-dependent sparsity strength.

C7. Failure Cases and Applicability Limits (vdht W1, Q2).

R7. In an exploratory experiment using ImgEdit images as the target and ImageNet as the background, SAGE does not clearly isolate editing-related factors. This failure suggests that subtle differences poorly captured by the frozen representation can limit salient-factor discovery.

C8. Training Effort and Convergence (vdht Q1).

R8. We report training epochs, hardware, wall-clock times, and loss changes near the end of training for both stages, documenting SAGE’s computational cost and convergence behavior.

We have incorporated the additional evaluations and clarifications into the revised manuscript, with changes highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.