acceptodds
Under review as a conference paper at ICLR 2027

What Makes One-Step Inference Work? A Decomposition of Gradient Origin Networks

Abstract

Gradient Origin Networks remove the encoder, instead inferring a latent code with a single gradient step through the decoder with respect to the origin. We ask what a decoder must learn for such a restricted inference rule to work. We find that autoencoder decoders often reconstruct competitively when their latent codes are optimised directly, yet perform poorly under the same one-step inference rule. The difference therefore lies in inference rather than reconstruction capacity. In the linear case this has an exact explanation: one-step inference uses the decoder transpose while optimal inference uses its pseudoinverse, with the two coinciding when the decoder Gram matrix is isotropic up to scale. An untied autoencoder need not learn this parametrisation because its encoder can absorb arbitrary invertible changes of decoder coordinates. Nonlinear GONs similarly learn flatter origin-Jacobian spectra, but imposing this geometry on an autoencoder is not sufficient. Instead, successful one-step inference depends on training reconstruction through the decoder's own inference map. A decoder can reconstruct well and still be poorly suited to gradient inference.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Gradient Origin Networks, inference, autoencoders, representation learning, latent-variable models, optimisation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Tu9VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper compares one-step and optimised latent inference on the same fixed decoder. Linear theory and nonlinear interventions are used to explain the GON--autoencoder gap.

Critical strengths
  1. The within-decoder comparison separates reconstruction capacity from inference quality. Using the same decoder architectures also makes the family comparison meaningful.

  2. The linear analysis explains the role of decoder coordinates. The step-size, origin, frozen-code and decoder-transfer controls strengthen the empirical result.

Critical weaknesses
  1. The geometric statistics in Table 1 need clarification. The strong-regularisation row reports κ(M)=1.1\kappa(M)=1.1 and pr=9.7\mathrm{pr}=9.7 at nz=10n_z=10 (p. 7). These values do not match Equation 14 and Appendix B's full-spectrum definitions: \qquad \kappa(M)=\frac{\sigma_{\max}^2}{\sigma_{\min}^2}.$$ For ten singular values, $$\mathrm{pr}\ge\frac{40\sqrt{\kappa(M)}}{(1+\sqrt{\kappa(M)})^2}.$$ A mean $\kappa$ rounded to $1.1$ over three seeds gives $\kappa\le3\times1.15-2=1.45$ for each seed, since every $\kappa\ge1$. Thus each seed must have $\mathrm{pr}\ge9.914$, whose mean cannot round to $9.7$. Appendix I also defines conditioning over active directions.
Questions for authors
  1. Does Table 1 use full-spectrum or active-direction conditioning?

  2. It is unclear whether the identified effect survives in the regimes where it would matter in practice as the main claims rest largely on empirical observations from a narrow setup. This makes it difficult to tell whether the observed behavior is a general principle for popular transformer- or diffusion-based architectures.

1 comment

Table 1. You are right, thank you. The "pr" column reported the stable rank ∑σ2/σ12\sum\sigma^2/\sigma_1^2, not the participation ratio the paper defines; the figure of spectra used the right definition. Recomputed, every pr entry is high, 8.9-10.0 out of 10 (GON 9.5, AE 8.9; the freeze row 9.5 and the unfreeze row 9.1, previously 6.2 and 4.5). The spectral contrast in the table is therefore carried by κ(M)\kappa(M), which the text introducing the table now makes clear. The align column also changed under the correct definition (AE 0.36, GON 0.33), and we removed the sentence saying the transferred geometry drifts toward the AE regime. All changes are listed in CORRECTIONS_LOG.md in the revised supplementary material.

Q1, which κ\kappa? Every reported κ\kappa is full-spectrum. The "active latent directions" definition in the appendix was unused and is removed; at n_z=10 the GON's κ\kappa is below the AE's in all ten dataset-architecture cells at thresholds 10−310^{-3}, 10−210^{-2} and 10−110^{-1} as well.

Q2, other architectures. Our focus is the compatibility between a decoder and its one-step origin-gradient inference rule, rather than a comparison across generative architectures. The capacity–inference decomposition applies to differentiable decoders generally, and the linear theory does not depend on a particular network architecture. For the nonlinear experiments, we use MLP and convolutional decoders to investigate the proposed mechanisms under controlled conditions. Extending the experiments to transformers or diffusion models would require specifying comparable latent inference procedures and introduce additional factors beyond those we seek to isolate. We do not claim that these results generalise unchanged to these architectures, and clarify this scope in the revised limitations.

Reviewer_Y4JZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GONs skip the encoder entirely, and get a latent code via a single gradient step through the decoder, then decode from wherever that step lands. The paper asks what a decoder has to learn for such an inference trick to actually work.

They split reconstruction error into two parts. The first is how well the decoder could reconstruct if you searched hard for the best latent (what they call “capacity”). The second is how much extra error you pay for using the one-step code instead (the “inference gap”). The headline finding is that regular autoencoder decoders have about the same capacity as GON decoders, but they’re terrible under the one-step rule. The problem is inference, not reconstruction ability.

In the linear case there’s a clean explanation. The one-step trick only matches the optimal code when the decoder’s latent directions are orthogonal and equally scaled. An autoencoder has no reason to pick that nice basis, since its encoder can compensate for any reshuffling of the latent coordinates. A GON has no separate encoder, so it’s forced into the “right” basis.

Disclaimer: It is the first time I hear about GONs, so this will be a low-confidence review. However, I'm familiar with other, similar models, so I understand the math.

Critical strengths

I found the capacity vs. inference-gap decomposition to be clean and actually useful as a diagnostic. Also, the interventions seem to be well designed: the authors test one hypothesis at the time.

Critical weaknesses

Isn't the main finding is kind of expected? An AE never trained for one-step inference is bad at one-step inference. Nothing in AE training constrains the decoder’s coordinate system, so there’s no reason it would happen to suit the one-step rule.

The genuinely interesting part would be to understand how gradient descent finds a decoder whose local origin map sends inputs to codes where the decoder reconstructs well. That’s explicitly left open in the conclusion. The negative results (X isn’t sufficient, Y isn’t sufficient) are useful, but they narrow the search space rather than answer the question. As it stands, it feels like a solid first half of a paper.

Minor:

GONs in the abstract is not introduced

Questions for authors

Table 1 shows that detaching the one-step code during backprop raises the gap from 0.003 to 0.009. You frame this as the Jacobian-gradient pathway being necessary in the nonlinear case, and quote 1.5–11× increases across datasets. Bit it seems to me that simply computing the code with the decoder’s own transpose does most of the work, exactly as in the linear case. The extra gradient through the Jacobian is a refinement. Is this correct?

2 comments

Thank you. You raise three points; we agree with part of each.

"The result is expected." We agree that a GON's small inference gap is expected from its training objective (Section 3), and that the AE objective does not constrain the decoder coordinates required for one-step inference (Section 4). But these observations establish why the separation can occur, not how much of it is intrinsic to the trained decoder.

The new results (Section 7) show that much of the AE's apparent failure is surprisingly easy to repair. A single fixed linear map of the one-step code, fitted after training without changing the decoder, removes a median 90% of its excess inference error on held-out images. Thus, a decoder that initially performs very poorly under origin-gradient inference can often support substantially better inference through a simple correction to its latent read-out. The linear theory motivates this intervention, but does not establish that it will work so effectively for nonlinear decoders. The result identifies a substantial, practically recoverable component of the nonlinear inference gap that is not determined by the training objectives alone.

"Half a paper." We agree that a complete account of how gradient descent produces the GON's nonlinear parametrisation remains open. The revision instead strengthens the interventional explanation of what makes one-step inference work. Sections 5–6 show that matching the GON's origin-Jacobian conditioning or supplying its latent codes does not reproduce its inference performance, while forward tying already provides much of the benefit. Section 7 then identifies a training intervention: adding only the inference-gradient pathway to an ordinary AE brings its one-step error from 5–12× the GON's to 1.1–2.6×, generally preserving its reconstruction capacity. This isolates an effective component of the training signal without replacing the AE objective with the full GON objective.

The two approaches to improving inference are also not interchangeable. Post-hoc coordinate correction transfers more reliably to compressed sensing than the inference-gradient intervention, despite the latter often performing better with full observations. Thus, the revision goes beyond describing the gap: it identifies which simpler explanations are insufficient, demonstrates how much can be repaired after training, and isolates training mechanisms that improve inference. What remains open is a closed-form account of the nonlinear learning dynamics.

The detach ablation. We agree that forward tying accounts for much of the improvement, and our original interpretation overstated the distinction between linear and nonlinear models. We have rerun the nonlinear ablation from scratch with matched initialisations. Detaching z1z_1 worsens the inference gap in all 30 paired runs, by factors of 2.32.3--9.79.7 across configurations, although the detached model still performs substantially better than an ordinary AE. Thus, forward tying provides much of the benefit, while the gradient through z1z_1 produces an additional, consistent improvement. The linear/nonlinear distinction is therefore one of degree rather than kind. We have revised the discussion accordingly.

Minor. The abstract now introduces the GON acronym.

Thank you for your thoughtful answer. However, I still believe this work to be incremental for a prestigious conference like ICLR. I will hence keep my score.

Reviewer_udzDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents insightful analysis about Gradient Origin Networks. These are generative deep learning-based that do not require an encoder. GONs use the one-step inference rule to define the latent space for the generative model. This work specifically focuses on investigating a part of the one-step inference rule. The investigation divides into three main contributions: (1) Capacity-Inference Decomposition: This focuses on shows that a decoder capacity differs from the one-step inference quality. It tested this by comparing AE decoders with GON decoders under the one-step inference rule. (2) Exact Linear: It examines the one-step inference under the assumption of a linear decoder. It presents the isotropic gram matrix requirement and the autoencoder invariance under the decoder conditioning. (3) Nonlinear Geometric Characterization: This extends the analysis to nonlinear decoders and examines their Jacobian spectra. This work reports that trained GONs have flatter and better-conditioned origin Jacobian spectra than AEs.

Critical strengths

(1). The paper is well-organized. This makes it easier to link the main contributions of this work to the sections of the paper. Section 3 focuses on the capacity-inference decomposition, Section 4 addresses the exact linear contribution, and Section 5 deals with the nonlinear geometric characterization.

(2). The mathematical analysis is well-presented and easy to follow.

Critical weaknesses

(1). This work is not well-motivated. It is an insightful work on GONs and one-step inference work but does not provide enough information about importance and the implication of the main contributions of this work. As it is, it is difficult to establish the significance of this work.

(2). The definition of the loss function in Equation (11) is not clear. Are we still assuming squared error or does it generalize to any error function?

Questions for authors

(1). What is the exact loss function in Equation (11)?

(2). What are the benefits of the main contributions of this work to the design and use of GONs? This will help clarify the significance of this work.

(3) What would be the impact of having a nonlinear decoder on the exactness theory or the challenges associated with such analysis?

(4) How did you determine or ensure that the AE decoders for the simulation in Figure 1 have better decoder capacity? Is this always for AEs vs. GONs?

2 comments

Thank you for the careful review.

Motivation and significance. Our broader question is when a learned decoder can support efficient gradient-based latent inference. This matters when a generative decoder is reused without its original encoder, the case we tested was as a prior for an inverse problem. Reconstruction capacity alone does not guarantee that a one-step inference procedure can access that capacity. The distinction is therefore relevant to how decoders are evaluated and trained, not just to comparing GONs and AEs.

We demonstrate this practical consequence in compressed sensing (now Section 7): despite having lower optimised-code reconstruction errors in all 12 settings, AE decoders produce 2.1–7.9× worse one-step recovery than GONs. The analysis also suggests two ways to obtain inference-compatible decoders. After training, a fixed linear correction to the origin-gradient code removes a median 90% of the AE's excess one-step error without changing the decoder. During training, adding the inference-gradient pathway to an AE objective improves one-step performance from 5–12× to 1.1–2.6× that of a GON, generally with little loss of reconstruction capacity. These interventions demonstrate that inference compatibility can be improved without sacrificing the decoder's original purpose.

Finally, the two approaches behave differently when transferred to compressed sensing: the post-hoc correction retains its advantage more consistently, despite the inference-gradient intervention being stronger in several full-observation settings. Thus, our findings offer both a diagnostic for pretrained decoders and practical alternatives for improving their inference properties, with different costs and transfer characteristics.

Eq. (11). LAE\mathcal{L}_{AE} is the squared reconstruction error which the paper now makes clear. However, the invariance needs only that the set of reconstruction maps {WE}\{WE\} is unchanged under W↦WAW\mapsto WA, E↦A−1EE\mapsto A^{-1}E, so it holds for any loss that depends on (E,W)(E,W) only through the reconstruction. A separate penalty on EE or WW breaks it; with weight decay on both, only orthogonal AA remain symmetries (Kunin et al., 2019). This remark follows the proposition.

Q3, nonlinear exactness. In the linear case, the Gram matrix fully determines whether the one-step rule agrees with optimal latent inference. For a nonlinear decoder, this is no longer sufficient: J0J_0 determines the inferred code, but reconstruction depends on the decoder's behaviour away from the origin. For observations on the decoder's image, x=F(z)x=F(z), exact latent recovery requires αJ0⊤(F(z)−F(0))/c=z\alpha J_0^\top(F(z)-F(0))/c=z. If this holds throughout an open connected latent domain containing the origin, differentiation gives αJ0⊤JF(z)/c=I\alpha J_0^\top J_F(z)/c=I throughout that domain. The linear Gram condition is recovered at z=0z=0, but away from the origin the condition depends on the full nonlinear decoder. This does not rule out curved decoders. For instance, F(z)=(z,z2)F(z)=(z,z^2) admits exact one-step recovery, but local isotropy alone is insufficient. Our empirical setting concerns approximate reconstruction of general observations, for which characterising the required nonlinear compatibility remains open.

Q4, capacity. Figure 1 claims comparable capacity. The autoencoder floor is lower in 76 of 90 matched pairs (20 of 24 in Figure 1's subset), with median AE/GON floor ratio 0.91. The floors are optimised upper bounds and on dSprites depend on the start (whether from z0z_0 or from zaez_{ae}); the appendix on z⋆z^\star convergence reports both starts.

Reviewer_udzDOpenReview ↗

Response to Rebuttal

Dear Authors:

Thank you for your response. You have addressed the questions I raised. After considering all the discussions under this review, I will upgrade my rating.

Comments on the paper

1 comment

We would first like to thank the reviewers for their careful reading and constructive feedback. Their comments have helped us strengthen the paper, as detailed in our individual responses.

Separately, after submission we identified errors in our evaluation pipeline. We have corrected them, rerun every affected experiment, and revised the paper and supplementary material accordingly.

The main error: SmallNORB and dSprites were evaluated on random noise. The evaluation loader in the submitted code (run_zstar_recompute.py, lines 181–198, duplicated in run_phase1.py) wrapped each dataset in a fallback:

elif dataset == "smallnorb":
    try:
        ds = datasets.SmallNORB(str(DATASET_ROOT), train=False, download=False, transform=tf)
        imgs = torch.stack([ds[i][0] for i in range(n)])
    except Exception:
        imgs = torch.rand(n, channels, image_size, image_size)

The SmallNORB and dSprites evaluation loaders failed because of an unavailable dataset class and an incorrect file path, respectively. A broad exception handler silently substituted uniform random noise. Training used a separate, functioning loader, so all 180 reference decoders were trained on the intended data, but 72 were incorrectly evaluated on noise. This affected the submitted Figures 1–2, latent-cosine figure, Tables 2–3, the GON and AE entries of the generality table, the origin-anchor and choice-of-start analyses, and the reference-floor audits. The exact failure, affected scripts, and reproduction are documented in AUDIT_REPORT.md.

What we corrected.

  • Evaluation loader. A shared loader now uses the training preprocessing and splits, raises on failure, and logs data fingerprints.
  • Reruns and audits. We reran every affected experiment on held-out test images. Reference floors now take the better of two optimisation starts (the origin and the model's own code), since the origin start stalls for dSprites autoencoders. We repeated the floor audits on real data: their original conclusions did not hold, but the lowest floor found changes the central ratio by only 0.2% (67.2 → 67.1).
  • Other errors. We corrected inconsistent preprocessing (including the CelebA crop), unrepresentative dSprites subsets, three evaluations on training images, and a step-size search that clipped the optimum. Each is documented in the supplementary changelog.

Detach ablation. The submitted paper described training from scratch, but the code fine-tuned pretrained AE decoders. We reran the experiment from matched random initialisations. Detaching z1z_1 worsens the inference gap in all 30 paired runs, by 2.3–9.7× across configurations. The corrected fine-tuning experiment gives the same direction on all 30 pairs.

What changed in the results. The central decomposition holds. The AE/GON ratio of mean inference gaps increases from 5.0× to 67×, or 77× on the 61 decoders where every optimiser start plateaued. Refitting the step size reduces this to 30×, removing 55% of the AE gap—contrary to the submitted claim that rescaling barely mattered. dSprites now exhibits the same qualitative pattern as the other datasets. Table 1 corrections are addressed in our response to Reviewer 3.

Verification safeguards. New tests verify that evaluation inputs match the intended held-out splits, missing data raises errors, and experiment scripts use consistent images. Dataset fingerprints are logged, scoring errors are no longer silently suppressed, trajectory evaluations are checked against their reference images, and step-size searches flag boundary optima. verify.py checks the reported numbers and tables against their result files; 14 checkpoint re-evaluations reproduce the committed one-step errors, with floor discrepancies within the documented tolerances.

Supporting material. The supplement contains corrected code, per-defect diffs against the submission, an audit report documenting each error and its consequences, and a script reproducing the original loader failure.

Guide to the revision. Beyond the corrections described above, the revised paper adds new results on post-training correction of one-step inference, training with the inference-gradient pathway, and transfer to compressed sensing (Section 7, Table 2, Appendices I–J). We have also revised the linear/nonlinear interpretation and detach analysis (Sections 4–6). A summary of the substantive changes and their locations is provided in REVISION_SUMMARY.md in the supplementary material, alongside the detailed correction log.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.