acceptodds
Withdrawn from ICLR 2027

Hybrid SE-PG: Reliable Gradient Estimation in Pre-Convergence Differentiable Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Differentiable optimization enables end-to-end learning through iterative solvers, yet obtaining accurate gradients remains challenging when solvers operate in the pre-convergence regime. Existing approaches face a fundamental trade-off: Implicit Differentiation (IFT) provides accurate gradients only near solver fixed points, while truncated methods improve efficiency at the cost of missing future gradient propagation. We introduce Hybrid Spectral Extrapolation Phantom Gradient (Hybrid SE-PG), a gradient estimation framework for pre-convergence solver states. SE-PG exploits locally predictable gradient evolution to approximate missing gradient propagation, while an adaptive hybrid mechanism integrates IFT as solvers approach convergence. This enables efficient gradient estimation across different solver convergence states without requiring full convergence before gradient computation. We evaluate Hybrid SE-PG on differentiable optimization benchmarks, including classical control tasks and high-dimensional manipulation problems. Gradient fidelity experiments show that SE-PG provides more accurate approximations to fully unrolled gradients under pre-convergence solver states. Training experiments further demonstrate improved optimization efficiency compared with implicit differentiation baselines while maintaining accurate gradient estimation.

Primary area:
optimization
Keywords:
Differentiable Optimization, Trajectory Optimization, Bilevel Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_CLZLOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies differentiable optimization on nonlinear optimal control problems when the inner iterative solver operates in the pre-convergence (transient) regime. In this regime, implicit differentiation (IFT) is biased because the fixed-point condition does not hold, while truncated unrolling misses the remaining gradient tail beyond the truncation horizon. The authors propose Spectral Extrapolated Phantom Gradient (SE-PG). The method extends the transient state by a kk-step damped phantom rollout and approximates the infinite-horizon gradient tail via linear extrapolation utilizing geometric contraction. Furthermore, the authors present Hybrid SE-PG, which adaptively blends SE-PG with standard IFT using a sample-wise convergence mask and a batch-wise convergence fraction. The approach is evaluated on control tasks (Cartpole, Pendulum) and robot manipulation tasks (DiffTORI).

Critical strengths
  1. The problem is real and practically relevant. In differentiable MPC / trajectory optimization and DEQ-style models, inner solvers are routinely truncated under compute budgets or during early training when θ\theta is poor, and both IFT and truncated unrolling are biased in this regime. A cheap correction for the truncated tail is a reasonable thing to want.

  2. The core idea is lightweight in spirit: estimating a local contraction factor from two consecutive gradient increments (a power-iteration-like heuristic) and approximating the remaining infinite-horizon gradient tail via a simple geometric series (Eq. 11) does not require an explicit Jacobian inverse.

Critical weaknesses

Beyond presentation issues, the paper has internal inconsistencies between the method description, the algorithm, and the appendix derivations; the theory is close to tautological; the central gradient-fidelity experiment contains an unexplained failure mode; and the problem and experimental setup are essentially absent, making the work nearly impossible to reproduce.

W1. The core gradient quantity vtv_t is undefined in the main text, and both possible readings conflict with the paper's claims

  1. vtv_t is the core gradient variable heavily used in Sec. 4 and Algorithm 1, but is never formally defined in the main text. Only in Appendix B (Eq. 21) is it defined as vt:=dh(t)/dθ∈RN×Pv_t := dh^{(t)}/d\theta \in \mathbb{R}^{N \times P}, i.e., a full forward-mode Jacobian. Yet Algorithm 1 uses vT+kv_{T+k} plus the tail term directly to form the gradient ∇θL(h∗)\nabla_{\theta} \mathcal{L}(h^*) for parameter update. Either reading leads to a contradiction:
  2. If vtv_t really is the N×PN \times P Jacobian, propagating it costs O(min⁡(N,P))O(\min(N, P)) JVPs/VJPs per phantom step, compared to O(1)O(1) VJP per step for standard UPG/TBPTT. This contradicts the repeated claim that SE-PG "preserves the computational efficiency of truncated differentiation" (Sec. 4.4, Line 321).
  3. If instead vtv_t is the reverse-mode loss gradient ∇θL(h(t))∈RP\nabla_\theta \mathcal{L}(h^{(t)}) \in \mathbb{R}^P, then: (a) calling backward() on L(hT+k)\mathcal{L}(h_{T+k}) gives only vT+kv_{T+k}, so obtaining vT+k−1v_{T+k-1} and vT+k−2v_{T+k-2} for Eq. (10) requires running separate backward passes from steps T+k−1T+k-1 and T+k−2T+k-2; and (b) because h(t)h^{(t)} is still moving, the outer gradient ∂L(h(t))/∂h(t)\partial \mathcal{L}(h^{(t)}) / \partial h^{(t)} changes with tt, which breaks the constant-operator recursion Δvt=BΔvt−1\Delta v_t = B \Delta v_{t-1} indicated by Eq. (9).

W2. Algorithm 1 is internally inconsistent and not executable as written

  1. Algorithm 1 Line 8 instructs to "Compute J,gJ, g at anchor hT+k−1h_{T+k-1} to define operator BB and ganchorg_{\text{anchor}}". But these quantities are never used in Lines 9–13, and computing them directly contradicts Sec. 4.4 (Line 321) and Appendix B (Line 701), which state that SE-PG does not construct BB. Moreover, forming the full Jacobians JanchorJ_{\text{anchor}} and ganchorg_{\text{anchor}} in Line 8 already incurs the heavy Jacobian construction cost that truncated differentiation is meant to avoid.
  2. Conversely, the quantities that Line 9 actually needs (vT+kv_{T+k}, ΔvT+k\Delta v_{T+k}, ΔvT+k−1\Delta v_{T+k-1}) are never computed anywhere in the algorithm.
  3. Line 5 computes GIFT(i)G_{\text{IFT}}^{(i)} unconditionally for every sample before the check wi<1w_i < 1 in Line 6. For unconverged samples (mi=0⇒wi=0m_i = 0 \Rightarrow w_i = 0), this expensive IFT solve is computed only to be multiplied by zero in Line 11.

W3. The batch-coupled weight wi=miα2w_i = m_i \alpha^2 is not justified and depends on the batch size

  1. α2\alpha^2 only affects samples that already pass the convergence test (mi=1m_i = 1), for which IFT is already valid by the paper's own argument. Why should sample ii's gradient be diluted by (1−α2)GSE(i)(1 - \alpha^2) G_{\text{SE}}^{(i)} because other independent samples in the same mini-batch have not converged? If the concern is that mi=1m_i = 1 is a noisy indicator, the natural fix is a per-sample soft weight based on sample ii's own residual, not a batch statistic.
  2. The statistical properties of α2\alpha^2—in particular the conditional expectation E[wi∣mi=1]\mathbb{E}[w_i \mid m_i = 1]—depend on the batch size NN, so the batch size directly affects the resulting gradient estimator. Yet the paper provides neither a theoretical analysis of this dependence nor even reports the batch size in the experiments.
  3. The ablation in Appendix G (Fig. 8d) sweeps p∈1,2,3,4p \in \\{1, 2, 3, 4\\} in wi=miαpw_i = m_i \alpha^p but omits p=0p = 0 (i.e., wi=miw_i = m_i), the uncoupled baseline. Without p=0p = 0, the experiments cannot establish that introducing α\alpha is effective at all.

W4. Theorem 4.1 is essentially tautological, and its premise is violated in the paper's own experiments

  1. Theorem 4.1 replaces the matrix recursion by a scalar AR(1) model Δvt=ρΔvt−1+ϵt\Delta v_t = \rho \Delta v_{t-1} + \epsilon_t (Eq. 17). Any sequence satisfies this identity if ϵt\epsilon_t is defined as the residual. The theorem does not bound ∣ρ−ρ^∣|\rho - \hat{\rho}| (the accuracy of a one-step norm ratio with k=2k = 2) nor ∑t∥ϵt∥\sum_t \lVert \epsilon_t \rVert (which absorbs both the non-normal, multi-mode structure of BB and the infinite-horizon linearization drift), so it does not tell the reader when SE-PG is accurate. The "Interpretation" paragraph only restates the statement.
  2. The constants hide the dependence on δ\delta: since f(x)=x/(1−x)f(x) = x / (1 - x) has f′(x)=(1−x)−2f'(x) = (1 - x)^{-2} and ρ^\hat{\rho} is clipped at 1−δ1 - \delta in Eq. (10), C1=O(min⁡(1−ρ,δ)−2)C_1 = O(\min(1 - \rho, \delta)^{-2}) and C2=(1−∣ρ∣)−1C_2 = (1 - |\rho|)^{-1} (these are not even shown in the proof in Appendix C). Fig. 6 (Appendix D) shows the measured ρ\rho repeatedly reaching 1.001.00, where both constants blow up and the geometric series does not converge, so the theorem's premise is violated in the authors' own experiments.

W5. Fig. 3 reveals an unexplained failure mode of SE-PG, and the Hybrid variant's good behavior appears to be IFT in disguise

  1. In Fig. 3, from roughly epoch 40 to 150 the cosine similarity of pure SE-PG (and UPG) with the reference gradient is strongly negative (about −0.5-0.5 to −0.85-0.85), while mask_alpha is already at 90–100%. That is, SE-PG produces gradients pointing against the reference precisely when the solver is nearly converged after T=10T = 10 steps. This is not discussed. Fig. 2(a) stops at epoch 40, just before this regime, where a drop to about −0.25-0.25 is already visible.
  2. In the same regime α≈1\alpha \approx 1 implies wi≈1w_i \approx 1, so Hybrid SE-PG reduces to pure IFT, and comparing IFT against an IFT reference trivially yields cosine ≈1\approx 1. The paper's own evidence therefore does not show that the SE-PG component contributes positively beyond the first ~30 epochs.

W6. Problem setup, forward solver, and experimental details are essentially absent

Essential setup details are missing from both the main text and appendices, and no code is provided:

  1. The bilevel problem and its dimensions (dx,du,N,P=dim⁡θd_x, d_u, N, P = \dim \theta) are never written down: the outer loss L\mathcal{L}, the exact forms of gθg_\theta and fθf_\theta for Cartpole and Pendulum, and expert data generation are all omitted. For standard Pendulum (dx=2d_x = 2) and Cartpole (dx=4d_x = 4) with H=20H = 20, the trajectory dimension is only N=60N = 60–100100, making it unclear why Cartpole training in Fig. 4(a) takes ~16,000 s on an i9 + RTX 4060.
  2. The forward solver F\mathcal{F} used in the main control experiments (Sec. 5.2) is never specified.
  3. Critical hyperparameters that directly govern the hybrid switching behavior—most notably the batch size NN and convergence tolerance δconv\delta_{\text{conv}} in Eqs. (12)–(14)—are omitted, as are the optimizer, learning rate, ϵ\epsilon in Eq. (10), etc.

W7. Presentation and notation

  1. Notation collisions: TT (planning horizon in Eq. 1 vs. solver truncation step in Sec. 4), NN (state dimension in Eq. 2 vs. batch size in Sec. 4.3), and kk (iteration index in Eq. 2 vs. phantom horizon in Sec. 4) are all overloaded across sections.
  2. Sec. 3 spends a page on textbook IFT yet never writes down UPG (Geng et al., 2021), the method SE-PG directly builds on.
  3. The term "tail" is used in Sec. 4.1 before it is defined, and Sec. 4.2 presents BB and Eq. (9) as part of the method without clarifying until the end of Sec. 4.4 (Line 321) that BB is only an analysis device and is never constructed.
  4. In Appendix B, Eqs. (22)–(23) omit the evaluation point of the partial derivatives, and while Eq. (26) uses "≈\approx" when freezing the Jacobian at the anchor, Eqs. (28)–(30) and Eq. (9) in the main text silently revert to "==".
Questions for authors
  1. Is vtv_t in your implementation the Jacobian dh(t)/dθ∈RN×Pdh^{(t)}/d\theta \in \mathbb{R}^{N \times P} (Appendix B, Eq. 21) or the loss gradient ∇θL∈RP\nabla_\theta \mathcal{L} \in \mathbb{R}^P? In either case, how exactly are vT+k−2v_{T+k-2}, vT+k−1v_{T+k-1}, vT+kv_{T+k} obtained, and at what cost relative to standard UPG?
  2. In Algorithm 1, why does Line 8 compute JJ, gg, BB, and ganchorg_{\text{anchor}} when Sec. 4.4 states that BB is never constructed and Line 9 does not use them, and why is GIFTG_{\text{IFT}} computed unconditionally in Line 5 before the wi<1w_i < 1 check?
  3. What assumption in Appendix B justifies freezing the Jacobians (B,g)(B, g) as constants (Eqs. 24–30) when the solver is in the pre-convergence regime and h(t)h^{(t)} is still moving?
  4. In Theorem 4.1, under what conditions on the solver and trajectory are ∣ρ−ρ^∣|\rho - \hat{\rho}| (with k=2k = 2) and the infinite series ∑t=k+1∞∥ϵt∥\sum_{t=k+1}^\infty \lVert \epsilon_t \rVert guaranteed to be bounded, especially when ρB\rho_B reaches 1.001.00 as in Fig. 6?
  5. Why does cos_SEPG become strongly negative after epoch 40 in Fig. 3 while mask_alpha is at 90–100%? In that regime, is the Hybrid curve anything other than pure IFT?
  6. Why should a converged sample's weight depend on other samples through α2\alpha^2? Please report p=0p = 0 (wi=miw_i = m_i) in the ablation of Fig. 8d.
  7. Could you provide the key missing experimental setups for Sec. 5, such as the problem and parameter dimensions, the inner/outer objective formulations, and the forward solver F\mathcal{F}, together with the batch size NN and tolerance δconv\delta_{\text{conv}} used?
Reviewer_SgqJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to get good gradients when an iterative solver has not fully converged. The main idea is that the gradient changes may decay at a nearly fixed rate after a few steps. Based on this, SE-PG runs a few extra steps to estimate the decay rate and uses it to predict the remaining gradient updates. The method also switches toward IFT when the solver is close to convergence.

Critical strengths

The problem of getting good gradients with limited compute is useful for differentiable control is practical and important. And the method is simple and easy to understand.

Critical weaknesses
  1. My main concern is that the idea is not very new, and the reason why it should work better is not clear. Specifically, estimating the decay rate from a few steps and using it to predict the infinite sum is a common idea in numerical analysis. The paper does not have experiment results for comparing SE-PG with these existing methods.

  2. This paper’s motivation is also not clear. The authors argue that IFT gives wrong gradients when the solver has not converged. However, SE-PG still uses the Jacobian at a point that has not fully converged. Why should SE-PG give better gradients than IFT at the same point?

  3. The extrapolation assumption is too strong. Eq. (11) uses a single decay rate to estimate the remaining gradient. However, in general, the Jacobian can have many eigenvalues, and different directions may decay at different rates. The paper does not explain when this simple estimate works or fails. The algorithm of the hybrid mechanism seems very heuristic as well. In Algorithm 1, the weight of each sample depends on the convergence rate of the whole batch. Why should the gradient of one sample depend on whether other samples have converged? Also, the method computes IFT gradients for all samples, which adds cost even when SE-PG is used.

  4. In the experiments, most tasks are small and simple, and the paper does not compare with enough existing methods. On the high-dimensional task, the gain is very small. The theory also mainly relies on the assumed decay behavior, rather than explaining why SE-PG should work better than other methods.

Overall, the results look reasonable to me, but my main concerns are the single-rate extrapolation and the source of the efficiency gain. I would be more positive if the tail correction helps beyond the extra phantom steps under the same compute budget.

Minors: Use \citep instead of \citet.

Questions for authors

See weaknesses.

Reviewer_G5CwOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hybrid SE-PG for gradient estimation in differentiable optimization when the solver has not converged. SE-PG uses local spectral extrapolation to estimate the gradient tail omitted by a short phantom rollout. A hybrid mechanism combines this estimate with implicit differentiation (IFT) based on solver convergence. Experiments on Cartpole and Pendulum show that Hybrid SE-PG trains faster than fully converged DiLQR while reaching a comparable final loss. Gradient fidelity experiments further show that SE-PG improves directional alignment over UPG, using converged IFT gradients as the reference.

Critical strengths
  • The tail correction is simple and improves directional alignment over UPG. On Pendulum over five random seeds, the mean cosine similarity with the converged IFT reference increases from 0.4203 to 0.7735.
  • The control experiments and ablations make a good case for combining SE-PG and IFT. On Cartpole and Pendulum, the hybrid method reaches lower final losses than SE-PG alone and converges faster than Limited DiLQR.
Critical weaknesses
  • Figure 4 compares Hybrid SE-PG with UPG, changing both the tail correction and IFT fusion. Figure 5 does not include UPG with the same fusion rule. This leaves unclear how much of the training improvement comes from the spectral correction, a central part of the proposed method.
  • The paper does not report the deviations ε_t in Eq. (17) or the tail approximation error for the default k=2 setting. The accumulated deviations enter the bound in Theorem 4.1, while Appendix D compares the estimated contraction factor with the spectral radius. Figure 2 and Table 1 support better gradient directions, but do not establish how accurately the two-step extrapolation estimates the remaining tail on transient solver states.
Questions for authors
  • Could the authors include UPG with the same IFT fusion rule in Figure 5? I would like to compare its training loss and wall-clock time with Hybrid SE-PG while keeping the solver budget and phantom settings fixed.
  • How large are the deviations ε_t in Eq. (17), or the tail approximation error, for k=2 on the transient solver states used in the experiments?
  • Figure 2(b): please add the converged IFT gradient norm or the relative norm errors for UPG and SE-PG.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.