acceptodds
Under review as a conference paper at ICLR 2027

Sphere Encoder 2

Abstract

Sphere Encoder is an autoencoder that generates images by decoding random points from a high-dimensional latent sphere. We identify two limitations of the original formulation that reduce its generation quality. First, random points concentrate near the equator relative to the pole on an encoded latent, but the training rotation never reaches this region, leaving a gap that limits one-step generation. Second, training for generation with pixel-wise reconstruction loss encourages the decoder to average over plausible images, producing blurry images that lack high-frequency details. We present Sphere Encoder 2 to address both limitations, substantially improving image generation quality while maintaining the speed and simplicity of a autoencoder.

Primary area:
generative models
Keywords:
Sphere Encoder, Autoencoder, Generative Models, Image Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_qNYEOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Sphere Encoder 2 (SE2), which considers two limitations of Sphere Encoder (SE): insufficient sphere coverage during training and the use of a per-pixel reconstruction loss for generation. Specifically, it reparameterizes the noise so that training covers the full [0°, 90°] angle range, and decouples the decoder's task into two regimes: low-noise reconstruction and high-noise generation. Experiments show improvements over SE.

Critical strengths
  1. Linking reconstruction and generation through the rotation angle is an interesting view.

  2. It demonstrates performance improvement over SE.

Critical weaknesses
  1. Potential self-contradictions and inconsistencies.
  • The paper identifies SE's [0°, 85°] angle range as a limitation and proposes a reparameterization that extends it to [0°, 90°]. However, Sec. 3.2 sets α = 85° by default, and Fig. 15 shows that 86° gives the best performance.
  • For Oxford Flowers 256, the default configuration of SE2-B yields FDr6 values ​​of 3.92/3.04 in Tab.3,4,6. However, the values ​​in Tab.1 are 2.67/2.17.
  • On ImageNet, the 4-step gFID for SE2-B is 3.66 in Fig.7, whereas it is 2.79 in Tab.2.
  • Sec.C.2 states that "...gives the best rFID" in Fig. 13. However, there seems to be no rFID in that figure.
  • Sec.C.3 mentions "Sphere-H to report the main results in Tab. 2". However, no Sphere-H results are reported there.
  1. Design and ablation issues.
  • The authors introduce SE in Sec.2, but lack sufficient background to demonstrate their differences and provide experimental advantages compared with the standard VAE.
  • The ablations of losses are merged. Detailed ablations of each component are needed.
  • Section 3.4.2 claims that "adversarial loss...prone to mode collapse". Please provide a comparison with adversarial loss or training curves.
  1. Overclaim and missing SOTA. SE2 significantly lags behind pMF-H and Drift-L at 1 NFE. Also, comparisons with strong baselines like PixelDiT are encouraged.

  2. There are many typos. For example, a autoencoder in the abstract, We replaces in L152.

Questions for authors
  1. In L342, the author claims that DINOv3-small has only 29M parameters with trivial FLOPs. Detailed training cost comparisons are necessary.

  2. I am not sure what the footnote "We release Sphere2-B checkpoints ... in our repo ..." on page 7 means. The supplementary materials contain no code or checkpoint files.

  3. Can this method extend to text-to-image scenarios?

  4. Tab. 2 only reports 1 and 4 steps on ImageNet. How about increasing steps to 8, 32, or more?

3 comments

We thank the reviewer qNYE for the comments and suggestions, and /appreciate the time spent on reviewing our paper.


Suggestions from weaknesses

  • For the discussion of max training angle and sampling angle, please see Q2&4 in our response to Reviewer jEUp.

  • We put the section of relate work in the main paper in the middle of Introduction and Preliminaries, to make the reader have background.

  • For the over claim, please see G1: Experiment claim in our general response at the top.

  • For accumulative ablation, please see Q3 in our response to Reviewer GEM3.

  • We added Section B.6 (page 23) on the training dynamics of Lscore\mathcal{L}_\mathrm{score}.

    Figure A13 (page 23) tracks Sphere2-B on Oxford Flowers 256×256256 \times 256 from epoch 1000, when Lscore\mathcal{L}_\mathrm{score} is switched on, to epoch 2000, total 1000 epochs of training.

    It plots: (1) the training loss and gradient norm of the 2-transformer-layer real and fake score models; and (2) Lscore\mathcal{L}_\mathrm{score} value and the magnitude of its transport direction g\mathbf{g}.


Q1: DINOv3-small+ with trivial FLOPs

This sentence in Section 4.4.3 (Latent Consistency) compares the encoder with the decoder, not Sphere Encoder 2 with other models. The frozen DINOv3-small+ encoder has only 29M parameters, against 161M and 487M in total for Sphere2-B and Sphere2-L (Table 2), and its weights are not updated. Its cost is therefore small relative to the decoder during training. We did not mean that the overall compute of Sphere Encoder 2 is trivial compared with other methods, and we removed the sentence.


Q2: FD-lite footnote

Sorry for the confusion. The footnote refers to the checkpoints we will release with the public version of the paper for the future. We have removed the footnote. In the meantime, we added large side-by-side comparisons of uncurated ImageNet 256×256256 \times 256 samples from Sphere2-B with and without the FD-lite loss, at 1 step in Figure A16 (page 26) and at 4 steps in Figure A17 (page 27).


Q3: Text-to-image generation

We added Section B.4 (Text-to-Image Generation on Flowers, page 21), a small-scale text-to-image experiment on Oxford Flowers 256×256256 \times 256. The model and all training hyperparameters stay the same as the class-conditional Sphere2-B, and only three things change:

  1. Captions. We caption its 8,189 images with Qwen2.5-VL-7B-Instruct, 20 captions per image ranging from 5 to 40 words.

  2. Decoder condition. We encode each caption with the SigLIP2 text tower (up to 64 tokens), project the token embeddings to the decoder width, and prepend them to the image tokens as in-context tokens, following Qwen-Image-2.1 and FLUX 3 Image Klein. The text tokens replace the class embedding, and the angle embedding is still processed through adaLN-Zero.

  3. Score model condition. The two score models of Lscore\mathcal{L}_\mathrm{score} are conditioned on the pooled SigLIP2 text embedding instead of the class embedding.

Figure A12 (page 23) shows uncurated 1-step and 4-step samples for 17 unseen, out-of-distribution prompts listed in Table A7. Already at 1 step, Sphere2-B renders colors and scenes that never appear in the training images, such as a blue water lily, a purple dandelion, a pink sunflower, and a water lily growing between stones. More details are in Section B.4 (page 21).


Q4: More sampling steps

Following this suggestion, we added Section B.3 (Sweeping Sampling Steps, page 21), which sweeps the number of sampling steps KK from 1 to 500 for three pixel-space base models on ImageNet 256×256256 \times 256.

Each model is pushed away from the budget it was designed for: the many-step diffusion model JiT-B (50 steps) toward few steps, and the one-step pMF-B and the few-step Sphere2-B (1 to 4 steps) toward many steps. All entries are FDr6\mathrm{FDr}^{6} ↓ in this compact version of Table A6 (page21).

ModelParamsK=1K=1448832326464100100200200500500
JiT-B131M222.8945.8423.1615.8915.5115.7915.8315.82
pMF-B118M13.8310.6910.6112.1712.6912.8212.9613.06
Sphere2-B161M14.508.628.969.719.919.9610.0010.05

Three observations follow:

  1. Each model is best near its design budget. JiT-B collapses below 8 steps and plateaus from 32 steps on, pMF-B peaks at 8 steps, and Sphere2-B peaks at 4.

  2. Sphere2-B degrades gracefully with many steps like pMF-B. Its FDr6\mathrm{FDr}^{6} rises from 8.62 at 4 steps to 10.05 at 500 and then saturates, so the encode–rotate–decode loop stays stable rather than drifting away. Same behavior is also observed in pMF-B, which rises from 10.61 at 8 steps to 13.06 at 500.

We also have a Figure A11 (page 22) to show the qualitative results at 512×512512 \times 512 with pMF-L and Sphere2-L.

Thanks for the response. My concerns remain:

  1. The numerical inconsistencies raised in my original review remain unexplained.
  2. Quantitative comparisons with standard VAEs and adversarial losses, as well as an analysis of DINO training costs, are still missing.
  3. Multiple inconsistencies exist between the original and revised versions. For example, Sphere2-B's 4-step FDr6/gFID changed from 7.82/2.79 to 7.95/3.07, while training epochs changed from 400 to 300, warmup from 5 to 10, and EMA decay from 0.9998 to 0.99985, without explanation.

We thank the reviewer for the response to our rebuttal.

For Oxford Flowers 256, the default configuration of SE2-B yields FDr6 values of 3.92/3.04 in Tab.3,4,6. However, the values in Tab.1 are 2.67/2.17.

This is explained in Q4 response to reviewer GEM3. Please forgive us not pointing this in the reponse.

On ImageNet, the 4-step gFID for SE2-B is 3.66 in Fig.7, whereas it is 2.79 in Tab.2.

This a typo in Fig.7. Thanks for catching this. We fix all numbers in Fig.7 (now Fig. A15) for both sides.

Multiple inconsistencies exist between the original and revised versions. For example, Sphere2-B's 4-step FDr6/gFID changed from 7.82/2.79 to 7.95/3.07, while training epochs changed from 400 to 300, warmup from 5 to 10, and EMA decay from 0.9998 to 0.99985, without explanation.

The old Table 7 in the original version (now Table A14) was prepared in a rush near the submission deadline and contained typos. The expanded Table A14 in the revised version is the final version, and it directly reflects the results in Table 2. We are happy to provide all checkpoints and code for verification if needed.

Warmup epoch numbers. The value of 5 epochs was a typo for ImageNet. We first used the hyperparameters on Oxford Flowers with 10 warmup epochs, then reused them on ImageNet. While debugging, we tried a shorter warmup of 5 epochs on ImageNet, since ImageNet is much larger than Oxford Flowers. However, this change was never applied on our main training cluster (we didn't share the same training script due to diffetent GPU environments between two clusters, one is Nvidia and the other is Amd). Then all final ImageNet models were still trained with 10 warmup epochs when we check the config to finish the Tabel A14.

EMA decay numbers. Behind the table, we swept EMA decay values of 0.9998, 0.99985, and 0.9999 for both the base and large models (by resuming from checkpoints at ep100 using 0.9998, to train the rest 300 epochs, total 400 epochs). As explained in General Response G1, not all evaluations were finished by the submission deadline. In the original version, we reported Sphere2-B with 0.9998 (400ep) and Sphere2-L with 0.99985 (300ep, because it didn't finish 400ep training yet when we submitted the paper) and the old Table 7 did not reflect this difference. In the revised version, we use a single EMA decay of 0.99985 for both model sizes. Sphere2-B therefore now uses 0.99985 (300ep), which gives slightly worse results than 0.9998 (400ep). We chose consistency across model sizes over the best per-model number, and our conclusions are unchanged.

Compare with adversarial loss

Table 2 already includes a modern GAN baseline: GAT-XL/2 (ICML 2026), a latent transformer GAN. It uses a large model (602M parameters) and optimal GAN training recipes, and it achieves sota FID on ImageNet.

We do not expect that we could have a GAN loss that outperforms theirs when applied to our decoder (487M parameters), so GAT-XL/2 already serves as a strong GAN baseline. Moreover, latent baselines generally outperform pixel-based baselines overall on FDr6 in Table 2. So we assume this modern GAN is already strong enough to present the best GAN we could get in pixel-base space.

Frozen encoder DINOv3-small-plus cost in training

ModuleImage sizeParamsGFLOPs/img fwdGFLOPs/img bwdGFLOPs/img totalms fwdms bwdms total (batch 4)Peak mem
DINOv3 ViT-S+ fwd, no grad25628.7M16.19–16.1911.90–11.900.14 GiB
DINOv3 ViT-S+ re-encode fwd+bwd (frozen) for Llat-con\mathcal{L}_{\text{lat-con}}25628.7M16.1918.0734.2614.1115.7429.850.39 GiB
Decoder base fwd+bwd256131.4M34.9770.69105.6626.4645.1871.641.46 GiB
Decoder large fwd+bwd256459.3M121.69245.59367.2851.9189.73141.644.37 GiB
DINOv3 ViT-S+ fwd, no grad51228.7M16.19–16.1911.96–11.960.15 GiB
Decoder base fwd+bwd512133.2M35.8772.50108.3726.7745.8472.611.52 GiB
Decoder large fwd+bwd512461.6M122.90248.01370.9152.5491.48144.024.43 GiB

We will add this table to the appendix. In short, the frozen encoder is DINOv3-small, and its computational cost is small compared with the decoder’s.

Compare with standard VAEs

Could you clarify this suggestion? Because standard VAEs are typically used for reconstruction, or as tokenizers for latent generative models (as with the latent baselines in Table 2), we are unsure how to set up an experiment that compares a standalone standard VAE with our method on sampling new images on ImageNet. The original sphere encoder (https://arxiv.org/abs/2602.15030) already tried the random sampling from standard VAEs in Figure 3, with sota VAEs such as FLUX, which is failed due to posterior hole. We would be happy to run experiments if you could say what setup you have in mind.

Reviewer_GEM3OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The original model. The paper revisits Sphere Encoder (Yue et al., 2026), a single-stage autoencoder that generates images by decoding random points on a high-dimensional latent sphere. It attributes the original model's blurry one-step samples to two causes:

  • training noise that stops at 85°, short of the equatorial band where random latents lie;
  • a pixel-wise reconstruction loss applied to generation.

What Sphere Encoder 2 changes.

  • (i) It replaces "spherify with noise" with a cos/sin rotation that reaches 90°.
  • (ii) It uses a frozen DINOv3-small encoder and trains only a ViT decoder conditioned on class and angle.
  • (iii) It splits the rotation arc at a 70° cutoff. Below the cutoff is a reconstruction regime trained with LPIPS + L1. Above it is a generation regime trained with ImageNet-logit alignment, latent consistency, a new teacher-free score-matching loss computed in frozen ConvNeXt V2 features, and a Fréchet-distance regularizer (FD-lite).
  • (iv) It adds spherical CFG.

Results claimed. On ImageNet 256/512 and Oxford Flowers, it reports large gains over Sphere Encoder. It also claims the best FDr6 among pixel-space generators at ≤ 8 NFE. Ablations are in the appendix.

Critical strengths
  1. A large, real improvement over the predecessor at a fraction of the cost.

    • On ImageNet 256 (Table 2), Sphere-XL scores FDr6 10.44 / gFID 4.02. It has 1.3B parameters and uses 977 GFLOPs per forward pass, at 4×2 NFE with CFG.
    • Sphere2-L scores 5.14 / 2.46 with 487M parameters and 115 GFLOPs, at 4 NFE without CFG.
    • Sphere2-B reaches 12.57 / 3.31 in a single step at 22 GFLOPs.
    • The uncurated Fig. 1 shows the sharpness gain directly.

    This is the main reason the paper matters to me. It shows that a single-stage autoencoder with no learned prior can approach few-step diffusion quality at very low sampling cost.

  2. The second diagnosis is supported, and the reporting is unusually candid.

    • The cutoff sweep (Fig. 4) and the rotation-arc visualization (Fig. 5) support the claim that removing the pixel loss from the high-angle regime matters.
    • The authors report that FD-lite changes the metrics but not the images, and they release checkpoints with and without it (footnote 2).
    • They also document the collapse at 90° (Fig. 14).

    This candor is what makes the problems below checkable, and it suggests they are fixable.

Critical weaknesses
  1. The main diagnosis, the "gap to the equator", misdescribes the prior method, is never tested, and is contradicted by the paper's own sampler.

    • Sec. 2.1 misdescribes the original. Sec. 2.1 says the original Sphere Encoder samples α only up to 85°, leaving "the residual gap of 5°" that limits one-step generation.
      • The original paper (arXiv 2602.15030, App. C.2 and Table 16) mixes in angles from [85°, 89°] for its ImageNet and Flowers models, so the gap was about 1°.
      • Its own ablation (Table 15) found that raising the maximum training angle from 85° to 88° made one-step gFID worse (15.40 → 23.27).
    • The rotation is not a new noise model. For every α < 90° it produces the same latents as Eq. (3), since Eq. (3) = F(Eq. (4)). It only adds the 90° endpoint.
    • Nothing tests the claim. No experiment varies α_max with everything else fixed.
    • The sampler avoids the 90° endpoint.
      • One-step generation feeds pure noise F(e) but conditions the decoder on α = 86° (B) or 84° (L) (Table 7).
      • It does so because conditioning on 90° collapses to near-identical images (App. C.3; Fig. 14: FID 20.6 at 90° vs. 3.2 at 86°).
      • This reintroduces the very train/test mismatch that Sec. 2.1 blames, and 84° lies inside the range the original already trained on.

    This matters because the abstract and Secs. 2–3 are built on this diagnosis; as written, readers would take away a design lesson the evidence does not support. The gains may well come from the regime split, the new losses or the frozen DINOv3 encoder instead. But with at least seven simultaneous changes and no cumulative ablation, the paper cannot say which (see Q3).

  2. The evaluation does not establish the headline claims.

    • FD-lite optimizes the reported metrics. It minimizes the Inception-space Fréchet distance to training-set statistics, and FID and FDr6 are then reported against that same training set.

      • By the paper's own account it "does not improve visual quality at all".
      • Yet it moves FID from 5.04 to 3.66 and FDr6 from 9.17 to 8.12 (Fig. 7).
    • The other generation losses work in feature spaces adjacent to FDr6's backbones.

      • L_sem uses ImageNet-1k classifier logits.
      • L_score uses an ImageNet-fine-tuned ConvNeXt V2, while FDr6 uses ConvNeXt-V2-Base.
      • L_lat-con uses DINOv3, while FDr6 uses DINOv2.
      • The largest FD reductions in Fig. 18 fall exactly in those spaces.
      • The sampling and guidance angles were tuned on FDr6.
      • FID is known to drop through ImageNet-feature alignment without any perceptual gain (Kynkäänniemi et al., ICLR 2023).
    • The headline claim is thin. "Best FDr6 among pixel-space generators" rests on:

      • 4.92 vs. 5.01 at 256, with FDr6-tuned CFG (5.14 without CFG);
      • single runs;
      • at 512, comparisons with only JiT-B and JiT-L.

      FDr6 is never defined, and the baseline values are not attributed.

    • The comparisons undercut the stronger claims.

      • At one step, Sphere2 trails every 1-NFE baseline in Table 2 on both metrics.
      • StyleGAN-XL, a one-step pixel-space model with FID 2.30 at 256 and 2.41 at 512, is missing.
      • The conclusion's "matching the image quality of far more expensive SOTA diffusions" is contradicted by Table 2: the best gFID here is 2.46, against 1.13–1.42 for REPA, LightningDiT, DDT and RAE.

    With no held-out metric, precision/recall or human study, I cannot separate real quality gains from metric targeting.

Questions for authors
  1. (Suggestion.) Keep everything else fixed. How does training Eq. (4) up to α_max = 85–86° compare with training up to 90°, with each model sampled at its best angle? The same comparison against the original's [85°, 89°] mixing would also help. And how does conditioning pure noise on 84–86° fit the argument of Sec. 2.1? A favorable answer would substantially change my view of the paper's main claim.
  2. (Suggestion.) What are the Table 2 numbers for checkpoints trained without FD-lite? Footnote 2 says such Sphere2-B checkpoints exist. Please report alongside them:
    • per-space FD in the spaces no training loss uses (MAE, SigLIP2, CLIP), which Yang et al. (2026) also report for the baselines;
    • precision/recall;
    • seed variation for the 256 margin.
  3. (Suggestion.)
    • Could you add a cumulative Flowers ablation from Sphere Encoder to Sphere Encoder 2? The steps would be: frozen encoder → angle conditioning → cutoff + L_sem → L_score → L_lat-con → FD-lite.
    • Could you compare L_score with a projected-GAN loss on the same ConvNeXt V2-N layers?
    • Please also position L_score's "without a pretrained teacher" novelty against earlier teacher-free score-difference training: SD flow (Weber, TMLR 2023) and Score-of-Mixture Training (Jayashankar et al., ICML 2025).
  4. (Clarification.)
    • How is FDr6 defined, and where does each baseline number come from? Several FDr6 and gFID values match the re-evaluation in Yang et al. (2026), while others match the original papers.
    • Why do Table 1's Sphere2-B Flowers numbers (1-step FDr6 2.67, gFID 7.62) differ from the "default" rows of Tables 3, 4 and 6 (3.92, 10.29)? All of them are trained for 2000 epochs.

What would move me to 3 (weak accept):

  • convincing answers to Q1 and Q2;
  • claims rewritten to match the evidence: the motivation reframed around the regime split and the losses, "best FDr6" scoped or dropped, and the conclusion toned down.

Minor (no effect on my rating):

  • The one-step 512 samples in Fig. 2 are curated, and no uncurated one-step ImageNet-512 samples appear.
  • Sec. 3.2 gives α = 85° as the default, while Table 7 uses 86° and 84°.
  • App. C.3 mentions a "Sphere-H" with no results.
  • Alg. 1 omits F(·) around E(x).
1 comment

We thank the reviewer GEM3 for the comments and suggestions, and appreciate the time spent on reviewing our paper.


Q1: Max training angle and sampling angle

The original Sphere Encoder does use a maximum training angle of 85∘85^\circ, and it also mixes angles in [85∘,89∘][85^\circ, 89^\circ], which brings training close to the equator at 90∘90^\circ. However, these high angles are mixed with probability only 0.1, so the decoder is undertrained at these angles in the original Sphere Encoder. Sphere Encoder 2 instead fully covers the range [0∘,90∘][0^\circ, 90^\circ] with uniform probability, so the decoder is trained on all angles and learns to behave differently at each angle.

Please see Q2&4: Max training angle and sampling angle in our response to Reviewer jEUp for the details of the maximum training angle sweep and the sampling angle sweep, as requested. In the paper, the full section is Section B.1 (Maximum Training Angle, page 17), and the results are in Figure A9 (page 18) and Table A4 (page 19).


Q2: FD-lite loss

As requested, Table 2 (page 10) now reports Sphere2-B and Sphere2-L both with and without the FD-lite loss. Section B.7 (Distribution Shift, page 24) further breaks down FDr6\mathrm{FDr}^{6} into its six feature spaces for both models on ImageNet 256×256256 \times 256 at 1, 2, and 4 sampling steps, with and without the FD-lite loss (Table A8 and Figure A14).

Section B.7 draws three conclusions:

  1. The drift is specific to the Inception space, where gFID is measured. Without the FD-lite loss, the statistics of generated images drift away from real ones only in this space; the FD-lite loss reduces the Inception term of FDr6\mathrm{FDr}^{6} by 34–46% for both models at every step count, which is the gFID improvement in Table 2.
  2. The other five modern feature spaces move far less, mostly by single digits, so the loss corrects an Inception-specific mismatch rather than a general one. The generated images are also visually indistinguishable with and without it (Figures A16 and A17).
  3. Model size matters. Sphere2-B improves in every space at every step count, while Sphere2-L improves at 2 and 4 steps but its 1-step FDr6\mathrm{FDr}^{6} worsens by 17%, which explains the 1-step Sphere2-L† entry in Table 2.

If having more ideas and suggestions, please let us know.


Q3: Accumulative ablation on important components

We added Section B.2 (Component Roadmap, page 18), an accumulative ablation with Sphere2-B on Oxford Flowers 256×256256 \times 256. Each row adds one component on top of the row above it:

ComponentFDr6\mathrm{FDr}^{6} ↓ (1 step)FDr6\mathrm{FDr}^{6} ↓ (4 steps)gFID ↓ (1 step)gFID ↓ (4 steps)
(a)frozen encoder51.9321.45332.27119.53
(b)+ angle conditioning35.6520.11219.34103.29
(c)+ Lsem\mathcal{L}_\mathrm{sem} with angle cutoff5.394.3815.2614.71
(d)+ Lscore\mathcal{L}_\mathrm{score}2.632.2210.019.56
(e)+ Llat-con\mathcal{L}_\mathrm{lat\text{-}con} (default)2.331.989.328.74

Every component improves both metrics at every step count. The angle cutoff (c) brings the largest gain, cutting 1-step FDr6\mathrm{FDr}^{6} from 35.65 to 5.39, and Lscore\mathcal{L}_\mathrm{score} (d) then halves it to 2.63. Figure A10 (page 20) shows 1-step samples from each row: (a) and (b) generate blurry mean images, and sharp flowers appear once the angle cutoff is added in (c).

We added StyleGAN-XL to Table 2 (page 10) and to the sampling efficiency plot in Section A.1 (page 17).

We cited and discussed SD flow (Weber, TMLR 2023) and Score-of-Mixture Training (SMT; Jayashankar et al., ICML 2025) in the distribution matching part of the related work (page 3, L136–L139).


Q4: Baseline FDr6\mathrm{FDr}^{6} numbers

For baselines with public code and checkpoints, we evaluate FDr6\mathrm{FDr}^{6} on ImageNet ourselves, keeping all sampling settings identical to their official release. We first verify that our run reproduces the FID reported in their paper, and only then report the FDr6\mathrm{FDr}^{6} from the same samples in Table 2. For example, the StyleGAN-XL as asked got 2.20 FID (256x256) and 2.39 FID (512x512) in our run, which is better than the 2.30 and 2.41 reported in the StyleGAN-XL paper. Then we report their FDr6\mathrm{FDr}^{6} numbers in Table 2, which are 14.39 and 15.57, respectively.

The gap arises because Tables A9, A11 and A13 are early ablations, run before the cutoff sweep with αcutoff=80∘\alpha_\mathrm{cutoff} = 80^\circ instead of the default 70∘70^\circ. All rows within each of these tables share this setting, so the comparisons within each table still hold. We added a note in Section B.8 (Angle Conditioning, page 28) to clarify this.

Reviewer_jEUpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Sphere Encoder 2, an improved version of Sphere Encoder for direct image generation from random points on a high-dimensional spherical latent space.

The paper identifies two limitations of the original Sphere Encoder. First, the original latent perturbation scheme only rotates encoded latents up to approximately (85∘85^\circ), whereas uniformly sampled random latents concentrate near the equator, corresponding to approximately (90∘90^\circ) relative to a fixed encoded latent. This creates a mismatch between training and generation. Sphere Encoder 2 replaces the original perturbation with an explicit spherical rotation, cos⁡(α)z+sin⁡(α)F(e)\cos(\alpha)z+\sin(\alpha)F(e), which allows training to reach the equator exactly.

Second, the paper argues that applying pixel-wise reconstruction losses over the entire rotation arc is inappropriate in the high-noise regime, where the latent no longer uniquely identifies the training image and pixel regression encourages blurry conditional averages. The authors therefore separate the latent arc into reconstruction and generation regimes. Reconstruction losses are retained at small angles, while the high-angle generation regime is trained using semantic alignment, latent consistency, a feature-space score-matching objective, and an optional Fréchet-distance regularizer.

A notable component is the proposed score-matching loss. Two lightweight online score models estimate real and generated feature distributions in a frozen feature space, and their difference provides a transport direction that updates the decoder toward the real feature distribution without requiring a pretrained diffusion teacher.

Experiments on Oxford Flowers and ImageNet at 256 and 512 resolution show substantial improvements over the original Sphere Encoder, particularly for one- and few-step generation.

Critical strengths
  1. The paper identifies a clear geometric train-test mismatch in the original Sphere Encoder.The observation that random points concentrate near the equator in high-dimensional spherical spaces, while the original training perturbation stops at (85∘85^\circ), provides a convincing explanation for degraded one-step generation. The explicit rotation formulation is simple and well motivated.

  2. Separating reconstruction and generation regimes is conceptually sound. Pixel reconstruction becomes increasingly ill-posed as the latent moves away from the encoded training example. Removing pixel-wise supervision in the high-noise region and replacing it with distribution-level and semantic objectives is a reasonable way to avoid regression toward blurry conditional averages.

  3. The proposed online feature-space score matching is interesting.Unlike DMD-style methods, the method does not require a pretrained diffusion teacher. Instead, lightweight real and fake score models are trained online in a frozen representation space. This component may potentially be useful beyond Sphere Encoder itself.

  4. The empirical study is extensive.The paper evaluates multiple resolutions, decoder scales, one- and few-step generation, different feature extractors, angle sampling strategies, guidance angles, score noise distributions, and latent encoders. The ablation section is substantially more comprehensive than is typical for this type of generative-modeling work.

  5. The inference procedure remains computationally attractive. In particular, one-step generation requires only decoding a random spherical latent, and even few-step refinement remains substantially cheaper than conventional many-step diffusion sampling.

Critical weaknesses
  1. The claimed simplicity of the method applies mainly to inference rather than training. The paper repeatedly emphasizes the simplicity of an autoencoder-based generator. However, training Sphere Encoder 2 requires several auxiliary components, including a frozen DINOv3 encoder, a ConvNeXt feature extractor, two online score models, semantic/perceptual feature networks, latent consistency, and multiple feature extractors for the FD-lite regularizer. Therefore, while the final sampling path is indeed simple, the complete training system is substantially more complex than a conventional autoencoder objective. I suggest clarifying this distinction throughout the paper.

  2. The reported FDr6 performance is partially coupled to an explicitly metric-aware regularizer. The proposed LFD−liteL_{\mathrm{FD-lite}} directly matches real and generated feature statistics in several frozen representation spaces, including feature extractors closely related to those used for FID/FDr evaluation. The paper itself notes that this regularizer can improve the metrics without visible changes in image quality. This makes the strongest FDr6 claims somewhat difficult to interpret, since part of the optimization objective directly targets the statistics measured by the evaluation metrics.

  3. The paired semantic-consistency objective becomes conceptually questionable close to 90∘90^\circ. At α=90∘\alpha=90^\circ, z^(90∘)=F(e)\hat{z}(90^\circ)=F(e), which is independent of the paired training image XX. Nevertheless, the generation regime still includes semantic and latent-consistency objectives that compare the generated image with that paired XX. Since the latent no longer contains instance-specific information about XX, it is unclear why matching the semantics of the particular paired image remains a statistically well-defined target near the equator. This may partly explain the reported tendency toward class-level mean predictions at 90∘90^\circ.

  4. The comparison against diffusion models is strongest under FDr6 but less so under conventional FID. Sphere Encoder 2 achieves strong FDr6 results with very few network evaluations. However, several latent-space and pixel-space baselines still obtain substantially better FID. Since hyperparameters such as sampling angle and guidance angle are selected according to FDr6, a more balanced discussion of the trade-off between FDr6, FID, compute, and visual preference would strengthen the experimental claims.

Questions for authors
  1. Why is semantic alignment with the paired training image still appropriate as α→90∘\alpha\rightarrow90^\circ? At the equator, the latent becomes independent of the paired image. What is the probabilistic interpretation of enforcing semantic or latent consistency with that particular image in this regime?

  2. Can the authors better reconcile the equator motivation with the actual sampling strategy? The method is motivated by enabling training exactly at 90∘90^\circ, yet the best sampling angles are around 84∘−86∘84^\circ-86^\circ, and the appendix reports collapse at 90∘90^\circ. Is the main benefit of reaching 90∘90^\circ therefore improved training coverage rather than making 90∘90^\circ itself a desirable sampling point?

  3. Could the authors report wall-clock or total FLOP comparisons for few-step generation? Each refinement step includes both re-encoding and decoding. It would be useful to clarify whether all encoder forwards are included in the compute comparison against diffusion and flow-based baselines.

  4. During training, an α<90∘\alpha<90^\circ latent is constructed as cos⁡αz+sin⁡αF(e)\cos\alpha z+\sin\alpha F(e), so it still contains information from the encoded image, while the decoder is also explicitly conditioned on α\alpha. In contrast, during one-step inference, the latent is always a pure random spherical sample F(e)F(e), regardless of the chosen α\alpha. Could the authors clarify why conditioning the decoder on α<90∘\alpha<90^\circ is statistically consistent in this setting? In particular, the inference latent no longer matches the conditional construction used during training. The observation that α=90∘\alpha=90^\circ leads to reduced diversity, while 84∘–86∘84^\circ–86^\circ works better, seems to suggest that α\alpha acts more like a learned generation-regime control variable at inference than a true geometric angle.

1 comment

We thank the reviewer jEUp for the comments and suggestions, and appreciate the time spent on reviewing our paper.


W1: Simplicity at inference vs. training

We removed the repeated simplicity claims throughout the paper, keeping only a single mention in the abstract. We added Section B.5 (Discussion and Limitations, page 22), which clarifies that the simplicity holds at inference, while training Sphere Encoder 2 requires additional components beyond a conventional autoencoder reconstruction objective.


W2: FD-lite loss

Please see the details in our response to Reviewer GEM3 (Q2).


Q1: Regression to mean at 90 degrees

At 90∘90^\circ, the latent is a random point on the equator and carries no information about the input image. Each random latent is therefore paired with an arbitrary training image of the given class. At this angle we apply the semantic loss Lsem\mathcal{L}_\mathrm{sem}, which matches the logits of the decoder output and the ground-truth image under an ImageNet-pretrained ConvNeXt classifier in LPIPS. The logits project the image feature onto the rows of the classifier weight W∈R1000×DW \in \mathbb{R}^{1000 \times D}, and each row acts as a prototype direction for one class, so images of the same class have similar logits. Minimizing this loss over all images of the class, the decoder learns to output an image whose logits match the class average, i.e., a mean image of the class at the semantic level rather than a pixel-wise blur.


Q3: Sampling efficiency

We added Section A (Sampling Efficiency, page 17). Below is a compact version of the sampling speed table, at 512×512512 \times 512 on a single H100 with batch size 16:

ModelSamplerNFEimg/s ↑50K images (min) ↓
Sphere2-Lloop ×11424.82
Sphere2-Lloop ×24 + 2 enc.145.46
Sphere2-Lloop ×48 + 6 enc.62.814
JiT-L/32Heun ×501984.5185
pMF-L/32MeanFlow ×11157.76
RAE-DiT-DH-XL/2Euler ×4949 + 49 guide0.9926

Section A also plots generation quality against sampling GFLOPs on ImageNet 256×256256 \times 256.


Q2&4: Max training angle and sampling angle

We agree with the explaination that the sampling angle is a generation-regime control knob for inference. Thanks for this input.

As requested by the reviewer GEM3, we added Section B.1 (Maximum Training Angle, page 17), which trains Sphere2-B on Oxford Flowers 256×256256 \times 256 with the maximum training angle αmax\alpha_\mathrm{max} from 80∘80^\circ to 90∘90^\circ in 2∘2^\circ increments, and evaluates each model with the sampling angle α\alpha from 70∘70^\circ to 86∘86^\circ (Figure A9 and Table A4). Below is the 1-step FDr6\mathrm{FDr}^{6} ↓ at the default sampling angle α=85∘\alpha = 85^\circ:

αmax\alpha_\mathrm{max}80∘80^\circ82∘82^\circ84∘84^\circ86∘86^\circ88∘88^\circ90∘90^\circ
1-step FDr6\mathrm{FDr}^{6} ↓7.796.435.023.362.472.32

Full results with dense sweeps of αmax\alpha_\mathrm{max} and α\alpha are in Table A4 (page 19).

The training angle must reach the equator. FDr6\mathrm{FDr}^{6} improves steadily as αmax\alpha_\mathrm{max} approaches 90∘90^\circ, from 7.79 at 80∘80^\circ to 2.32 at 90∘90^\circ, since the random points used for generation lie on the equator.

The sampling angle does not need to equal αmax\alpha_\mathrm{max}, because the two play different roles: αmax\alpha_\mathrm{max} decides which latents the decoder is trained on, while the angle conditioning tells the decoder how to treat a latent. At the equator (α=90∘\alpha = 90^\circ), the latent carries no information about the input, so the decoder learns to output the mean image of the class. Below 90∘90^\circ, the latent keeps some clean signal, so the decoder learns to output diverse, distinct images.

At inference, one-step generation still decodes a random point on the equator, but conditioning the decoder on a lower angle, e.g., α=85∘\alpha = 85^\circ, makes it render that point as a distinct image away from the mean. This works because the decoder learns a different behavior for each angle during training, so the sampling angle acts as a knob that selects which behavior to apply. The angle embedding thus acts like a guidance scale (learnable CFG embedding) that the model is conditioned on during training [1, 2, 3, 4] (away from uncondition): steering samples away from the mean. This explains why the sampling angle controls generation diversity, as shown in Section B.10 (Sampling Angle at Inference, page 29).

If having more ideas and suggestions, please let us know.

  • [1] On distillation of guided diffusion models. CVPR 2023.
  • [2] Visual generation without guidance. ICML 2025.
  • [3] Improved mean flows: On the challenges of fastforward generative models. CVPR 2026.
  • [4] One-step latent-free image generation with pixel mean flows. ICML 2026.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We thank all reviewers for their feedback. We address the shared concerns here and answer the individual points in each reviewer's thread.


G0: Paper structure changes

We moved two sections between the main paper and the appendix:

  1. Related work in the main paper. Following Reviewer qNYE's suggestion that readers need background before the preliminaries, we moved the related work from the appendix to Section 2 (page 2), between the introduction and the preliminaries.

  2. FD-lite discussion in the appendix. To stay within the 10-page limit, we moved the Distribution Shift section on the FD-lite loss from the main paper to Section B.7 (page 24). The main paper keeps a short summary, and Table 2 (page 10) now reports Sphere2 results both with and without the FD-lite loss, as Reviewer GEM3 suggested.


G1: Experiment claim

Due to limited GPU resources, we had not finished evaluating all baselines in the ImageNet Table 2 by the submission deadline, especially at 512×512512 \times 512. As a result, we overstated our results. We have now evaluated all baselines and updated Table 2 (page 10) with the complete results. Please forgive us for the overclaim.

We have added PixelDiT (as requested by Reviewer qNYE) and StyleGAN-XL (as requested by Reviewer GEM3) to the main Table.2, and the sampling efficiency plot in Section A (page 17) (as requested by Reviewer jEUp).

Below is the pixel-space part of Table 2 with FDr6\mathrm{FDr}^{6} ↓. As in Table 2, Total GFLOPs = GFLOPs × NFE, where GFLOPs is for a single forward and ×2 in NFE marks CFG. † marks Sphere2 models trained with the FD-lite loss LFD-lite\mathcal{L}_{\text{FD-lite}}. Bold marks the top three per resolution, taking each model at its best setting.

ModelGFLOPs x NFE256×256256 \times 256512×512512 \times 512
PixNerd-XL130 x 100×25.015.68
PixelDiT-XL151 x 100×24.464.88
JiT-B25 x 50×415.7816.31
JiT-L88 x 50×410.7311.35
JiT-H182 x 50×47.668.66
Sphere-XL977 x 4×210.44–
StyleGAN-XL1571 x 114.3915.57
Drift-L87 x 110.51–
pMF-B66 x 113.8314.26
pMF-L117 x 19.099.96
pMF-H271 x 16.878.24
Sphere2-B22 x 114.5315.17
Sphere2-B22 x 48.589.35
Sphere2-B22 x 4×27.838.61
Sphere2-L115 x 111.9613.52
Sphere2-L115 x 45.776.35
Sphere2-L115 x 4×25.466.27
Sphere2-B†22 x 112.7613.91
Sphere2-B†22 x 47.958.49
Sphere2-B†22 x 4×26.917.82
Sphere2-L†115 x 113.9711.65
Sphere2-L†115 x 45.146.60
Sphere2-L†115 x 4×24.925.99

With 115×4×2115 \times 4 \times 2 total GFLOPs, Sphere2-L† ranks second at 256×256256 \times 256, behind PixelDiT-XL, and third at 512×512512 \times 512, behind PixelDiT-XL and PixNerd-XL, which take 28×28\times to 33×33\times more sampling compute. Its gFID is still higher than these baselines, which we analyze in Section B.7 (Distribution Shift, page 24).

We revised the main claim accordingly in the introduction (page 2, L81–L86) and the experiments (page 9, L468–L474), and the full results are in Table 2 (page 10). For a visual comparison, we also added uncurated ImageNet 512×512512 \times 512 samples of Sphere2-L without CFG, side by side with pixel- and latent-space baselines:

FigureSphere2-L (no CFG)Baseline
A27 (page 39)NFE = 4RAE-DiT-DH-XL/2, NFE = 50×2 with AutoGuidance
A28 (page 40)NFE = 2JiT-L/32, NFE = 50×4 with CFG
A29 (page 41)NFE = 1pMF-L/32, NFE = 1 with CFG

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.