acceptodds
Under review as a conference paper at ICLR 2027

Non-convex Training using Imaginary Paths

Abstract

Most machine learning methods including deep learning are defined by non-convex optimization problems that are typically regularized for enhanced generalization yet fundamentally challenged by loss landscapes with local minima. We show that regularization by the and norms can be expressed in terms of a radial-phase decomposition for which we prove the existence of a complex path with a bounded loss leaving any nonzero real stationary point under the stated assumptions. We explore this by designing a novel complex lifting framework from to that introduces a bilinear skew-symmetric auxiliary loss. We impose a consensus constraint between the lifted and original parameterization and use the generic alternating direction methods of multipliers (ADMM) for the optimization defining the proposed Complex-Continuation ADMM (CC-ADMM) method for non-convex optimization. After finite time complex exploration, CC-ADMM projects the model onto the real parameter space and continues real-valued optimization, reducing its convergence analysis to that of the real optimization phase. We show that, under the stated assumptions, the CC-ADMM converges to a stationary point of the original loss with probability one. CC-ADMM uses only twice the number of parameters during training with no resulting inference overhead. Experiments on synthetic non-convex problems confirm the predicted exploration behavior and when applied on a variety of non-convex machine learning tasks including "grokking", as well as Vision Transformers and Llama-1B and 3B finetuning, we observe enhanced generalization performance. The CC-ADMM thereby provides a versatile and favorable framework for non-convex training.

Primary area:
learning theory
Keywords:
non-convex optimization, deep learning, complex lifting, local optima

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_asFmOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes CC-ADMM, which temporarily extends real parameters into the complex domain during training. A skew-symmetric bilinear auxiliary term drives motion into imaginary coordinates, while consensus, annealing, and projection return the model to real parameters. The paper presents conditional analyses of bounded-loss complex paths, departure from real stationary points, and convergence after projection, together with experiments on grokking, classification, rotation regression, language-model adaptation, and ViTs. The central idea is to improve optimization through additional training coordinates while retaining real-valued inference.

Critical strengths

1. The relationship between training-time exploration and final inference is clear. Section 4 connects complex exploration, real consensus, and final projection. The auxiliary term has a nonzero imaginary gradient when Mx≠0Mx\ne 0, giving a concrete local explanation for how motion can start from a real stationary point. Returning the final model to the real domain makes the approach practically appealing.

2. The ViT experiment provides a useful architecture-matched comparison. Table 3 uses the same forward architecture and parameter count, increasing final accuracy from 0.5964 to 0.6060 and reducing final test loss from 2.7572 to 2.0952. This supports the training method more directly than comparisons that also change the network architecture.

Critical weaknesses

1. The theory does not sufficiently establish the complex-path advantage emphasized for the practical algorithm. For the ℓ2\ell_2 case explicitly covered in the main text, Equations (1) and (2) and Theorem 1 imply L~real(γ(t))−Lreal(Z⋆cos⁡t)=μR2sin⁡2(t)/2≥0\widetilde L_{\mathrm{real}}(\gamma(t))-L_{\mathrm{real}}(Z^\star\cos t)=\mu R^2\sin^2(t)/2\ge0: the corresponding real path has pointwise no higher loss. Existence of a bounded-loss path therefore establishes no loss-barrier advantage. The exact primal update in Theorem 2 and real SGD after projection in Theorem 3 also do not directly cover the finite AdamW inner loops and implemented training procedure. Although the authors disclose some limitations, the explanation connecting theory to practical gains remains insufficient. This affects the central contribution rather than establishing that the individual theorems are incorrect.

2. The main experiments do not sufficiently isolate optimization, and some improvement directions cannot be interpreted reliably. The MLP/CNN and language-model comparisons also change activations, magnitude readout, or the adapter function class, preventing attribution of all gains to complex exploration. The architecture-matched ViT comparison in Table 3 is positive evidence but does not resolve attribution elsewhere. Furthermore, Section 5.2 lists MLP accuracies of 0.2153 vs 0.2613 in the stated order “CC-ADMM vs Baseline,” yet claims a gain of 4.60 percentage points; CNN and ResNet comparisons contain similar conflicts between ordering and gain direction. Both architectural controls and the actual direction of improvement need clarification to establish the claimed broad optimization value.

Questions for authors

1. Could you address the ell2ell_2 path comparison above? Since the real-part path has no higher loss than the complex path, is the key advantage attributed to path geometry or to the optimization dynamics induced by the auxiliary? Which part of the current argument distinguishes these explanations?

2. Could you specify which guarantees still apply to finite AdamW inner loops and which apply only to exact subproblem solutions or SGD after projection? I am not asking for a new complete theory during the response period, but I need to understand how far the current results support the implementation.

3. After projection to real parameters, does the language-model adapter retain a nonlinear function different from the QLoRA baseline? Beyond ViT, do any existing results compare training with and without complex exploration under the same final architecture? This directly affects whether I would attribute the main gains to the proposed training method.

4. Please check the method ordering and gain signs throughout Section 5.2. A corrected mapping would suffice, together with clarification of which classification results still show an improvement, particularly the ResNet results at the three training lengths.

2 comments

We thank the reviewer for the time and thoughtful comments. The CC-ADMM is a temporary exploration framework used with a suitable optimizer of the user's choice, such as AdamW or SGD. The extra training computation supports auxiliary exploration and consensus updates. It's designed and worth to be applied specifically to highly non-convex settings, where the trade off pays off as supported by our experiments. Under the stated conditions, the analyzed updates initially depart from real stationary points into imaginary directions. The projection guarantees a return to the real parameters, after which training inherits the chosen optimizer's convergence guarantees under their applicable assumptions. The case of AdamW, addressed in detail in Q2, is also the same case. Convergence of the method is not determined by ADMM phase, but rather by the final real phase (after back projection to real) and thus, inherited from the used optimizer for this stage, be it, SGD (Theorem 3 for completeness of the paper) AdamW or something else.

Our current Introduction does not make CC-ADMM's purpose and training-cost trade-off clear enough. We will amend it using the two-sentence revision in our response to Reviewer X4HZ.

Next, we address in the below your specific concerns one by one.

W1. Theory versus practical gains. Your real-path comment is correct and we agree that "existence of a bounded-loss path therefore establishes no loss-barrier advantage". However, it is rather the opposite that we use in the theory: Theorem 1 (importantly extended by Corollary 1.1) establishes that there exists a path with loss bounded from the top by 2Rϵ2R\epsilon. Then our method constructs and adds an auxiliary (imposed by anti-symmetric MM) that provides enough dynamics/energy to overcome this finite barrier.

Te below answer to the first question Q1, explains in detail the dynamics of the proposed method. Q2 separates the guarantees for ADMM with inexact steps during exploration, ordinary gradient steps and real stochastic-gradient descent.

W2. Attribution and inconsistent labels. Thank you for correctly identifying architectural confounds and reversed prose labels. We believe, they do not invalidate the existing matched ResNet/ViT controls. However, Q3 distinguishes their evidence from the original MLP/CNN/Alpaca comparisons. Q4 corrects the ordering, including the negative short-budget ResNet result.

Q1. Geometry or dynamics? Our theory and method address complementary parts of the escape mechanism. Theorem 1 constructs a complex path with a controlled increase in task loss, keeping the radial contribution constant. Corollary 1.1 establishes the existence of a path to a real point outside any bounded basin covered by its assumptions, with the explicit, radius-dependent loss bound in Appendix Eq. (15). These are existence results: they establish an available route and bound its loss, without prescribing an optimization trajectory.

CC-ADMM adds the driving mechanism through the skew auxiliary LM(x+iy)=2x⊤My\mathcal L_M(x+iy)=2x^\top My. This term is zero on the real subspace but can become negative during exploration. A sufficiently negative auxiliary contribution can compensate for a temporary increase in the extended task loss and the consensus penalty while the primal objective decreases (please see Remark 18 for details). Notably, this permits uphill movement in task loss during exploration. The auxiliary also explains how exploration can begin: although LM(x,0)=0L_M(x,0)=0, its imaginary gradient is −2Mx-2Mx, and its real gradient 2My2My couples the resulting imaginary motion back into real updates (Eq. (3), Lemma 16). Theorem 2 and Remark 18 establish conditional initial departure. The choice of MM, together with its weight and the current coordinates, determines the direction and available compensation.

Thus, the pointwise inequality Lreal(x)≤L~real(x+iy)\mathcal L_{\rm real}(x)\le\widetilde{\mathcal L}_{\rm real}(x+iy) for a non-decreasing radial term is consistent with the mechanism: the exploration updates optimize an augmented objective. Theorem 2 guarantees initial departure under its stated assumptions (geometry), while subsequent basin crossing depends on the trajectory (dynamics). The method need not follow the constructed path exactly. Especially beyond the barrier, gradient descent takes over and may descent off the bounded path.

To provide experimental evidence for the theory, we restarted Figure 1 at its strict real minimum, a saddle after lifting (gradient norm 2.2×10−162.2\times10^{-16}, Hessian eigenvalues 0.653,2.0000.653,2.000). With 2,500 updates and no added noise, CC-ADMM reaches real loss 5.0×10−95.0\times10^{-9}. Real gradient descent, zero-decay AdamW and the auxiliary-off controller remain near 11. This uses Figure 1's activation-based extension, distinct from Eq. (2) and the revised-paper appendix will contain the protocol and figure.

Reflecting your comments we clarified Section 4 immediately after the auxiliary definition: a negative auxiliary contribution can offset increases in task loss and consensus penalty, enabling exploration. We distinguish the available bounded-loss paths from the algorithm’s trajectory and point to the appendix restart experiment demonstrating escape to a lower-loss real endpoint.

Q2. Exact solves, finite AdamW and the real tail. Thank you for asking us to make this distinction explicit. Our guarantees concern different stages.

Departure. Theorem 2 guarantees a non-real exact primal minimizer under its stated curvature, non-degeneracy and initialization assumptions. Exact minimization is not the only setting in which departure is established: for Eq. (2), Remark 18 gives y+=2ηλMZ⋆y^+=2\eta\lambda MZ_\star for an ordinary gradient step from real stationary consensus with zero dual. This proves initial departure when η,λ>0\eta,\lambda>0 and MZ⋆≠0MZ_\star\ne0. Notably, it does not establish convergence of stochastic AdamW or complete basin escape.

After projection. The final projection to real parameters is exact independently of the inner solver. Theorem 3 permits a well-defined finite exploration phase, including finite AdamW updates. Once projection is performed and the auxiliary, consensus and dual updates are disabled, the stated real-SGD assumptions give almost-sure loss convergence, vanishing gradients and stationary limit points. Thus, this guarantee does not directly require or condition solver used during exploration. In particular inexact ADMM step solver is admitted. The experimental AdamW tail is just a standard prominent AdamW training and converges (or not) accordingly. It can be replaced by any optimizer of choice in general.

Practical implementation. Approximate primal updates are established in the inexact-ADMM literature, including nonconvex extensions (Boyd et al., 2011; Wang et al., 2019). Despite commonly used, their guarantees require specified error and regularity conditions, which a fixed number of AdamW steps does not itself explicitly verify. Our architecture-matched ViT and longer-budget ResNet gains were obtained with K=5K=5 finite stochastic AdamW steps between consensus updates, providing empirical support for practical benefits without requiring top solve primal ADMM step exactly.

We revised the paragraph after Theorem 3 in Section 4 to cite established inexact and proximal ADMM results and clarify that our real-SGD guarantee permits finite inexact exploration before exact projection. We also explicitly retain the applicable optimizer assumptions for convergence of the experimental AdamW tail.

Q3. Final architecture and attribution. Yes. After projection, the language-model adapter has real parameters but retains modReLU and magnitude pooling, so it remains nonlinear and differs from the linear QLoRA adapter. As with the original MLP/CNN comparisons, these results evaluate the combined architecture and training procedure.

Beyond ViT, the existing strong-recipe ResNet-18 comparisons in Section 5.2 use the same deployed architecture and parameter count (11.22M) for both methods, providing evidence for the training procedure with unchanged final architecture.

We additionally added new CIFAR-100 CNN experiments, also with identical final architecture in both methods. These show lower mean final test loss with a mean accuracy gain. The numerical results and consistency across seeds are detailed in our response to Reviewer exek, W1.

The existing ResNet/ViT comparisons use the finite-AdamW implementations discussed in Q2, while Q4 clarifies the method ordering and outcomes of the original Section 5.2 comparisons as suggested.

Q4. Corrected ordering and gains. Thank you for pointing out the reversed order. The order is Baseline →\to CC-ADMM: MLP 0.2153→0.26130.2153\to0.2613 (+4.60+4.60 percentage points), CNN 0.6104→0.65190.6104\to0.6519 (+4.14+4.14), ResNet300 0.7838→0.75990.7838\to0.7599 (−2.40-2.40), ResNet900 0.8019→0.80480.8019\to0.8048 (+0.28+0.28), ResNet2700 0.7944→0.80360.7944\to0.8036 (+0.92+0.92). Gains use unrounded means. ViT gains 0.950.95 points and reduces test loss 24.01%24.01\%. All three corresponding seeds improve accuracy and loss at both longer ResNet budgets and ViT. These equal-epoch results show consistent gains across the tested seeds in the longer-budget ResNet and ViT settings. Following the arguments above and the proposed clarifications the paper does not claim statistical significance or general superiority under arbitrary settings. For a more detailed summary of results we kindly refer to response W1 given to reviewer exek.

We hope that the provided updates and evidence support a reassessment of the paper.

Reviewer_exekOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes CC-ADMM, which adds temporary imaginary parameters during training. A skew auxiliary loss pushes the parameters away from the real subspace, while consensus updates encourage a return to real parameters. Training then continues on the real model. The paper gives path-existence and departure results and evaluates the method on several learning tasks.

Critical strengths
  1. The CC-ADMM design seems novel and interesting. It uses imaginary parameters for exploration during training, then returns to a real model for inference.
Critical weaknesses
  1. The gains seem weak and mixed, the improvement is quite small. I would like to see a real-only baseline with the same final architecture to understand the gain from complex training.

  2. I am not fully convinced by the explanation of why the complex updates help. Leaving a real stationary point does not necessarily lead to a better real solution after projection. The connection to the experimental gains is still unclear to me.

Questions for authors
  1. Table 16 reports four CC-ADMM runs out of five for Llama-3B. What happened to the remaining run? Was this a job interruption or a training failure?
2 comments

We thank the reviewer for the time and thoughtful comments. Let us start by addressing an apparent misconception we might have installed in the paper: The CC-ADMM is not a general-purpose optimizer replacement. It is a temporary exploration framework used with a suitable optimizer of the user's choice, such as AdamW or SGD. Extra training computation supports auxiliary exploration and consensus updates. It's designed and worth to be applied specifically to highly non-convex settings, where the trade off pays off as supported by our experiments. Under the stated conditions, the analyzed updates initially depart from the real stationary points into imaginary directions. The projection guarantees a return to the real parameters, after which training inherits the chosen optimizer's convergence guarantees under their applicable assumptions.

Our current Introduction does not make CC-ADMM's purpose and training-cost trade-off clear enough. We will amend it using the two-sentence revision in our response to Reviewer X4HZ.

Furthermore, we address your specific concerns one by one in the below.

W1. Gains with the same final architecture. We agree that differences may look absolutely small, but absolute accuracy changes should be considered alongside relative loss reductions. Our submitted ResNet-18 and ViT comparisons use real-only baselines with the same deployed architecture and parameter count (Section 5.2, Table 3):

  • ResNet-18, 900 epochs: accuracy improves by 0.280.28 percentage points and test loss falls from 1.53421.5342 to 1.45731.4573 (5.0%, (significant difference, i.e., outside of 3 SDs)).
  • ResNet-18, 2700 epochs: accuracy improves by 0.920.92 percentage points (significant difference, i.e., outside of 3 SDs). Test loss falls from 1.66071.6607 to 1.54711.5471 (6.8%).
  • ResNet-18, 300 epochs: - the same experiment as above, just shorter epochs, experiment accuracy decreases by 2.402.40 percentage points and test loss rises from 1.52231.5223 to 1.57371.5737 (3.4%). (significant difference, i.e., outside of 3 SDs, in the baseline's favor).
  • ViT, 200 epochs: accuracy improves by 0.950.95 percentage points and test loss falls from 2.75722.7572 to 2.09522.0952 (24.0% (significant difference, i.e., outside of 3 SDs).

3 SDs interval separation denotes non-overlapping mean ±3\pm3 sample-standard deviations bands. In each of the three favorable settings, all three corresponding seeds improve both accuracy and test loss.

The ResNet results show that the benefit depends on the training schedule. These comparisons demonstrate gains in the final-model quality at additional training cost, with unchanged inference architecture, rather than universal improvement or equal-time superiority.

For this rebuttal, we ran new CIFAR-100 CNN experiments as an additional architecture-matched experiment. With an identical final architecture, five paired seeds and 100 epochs per method, CC-ADMM reduces mean final test loss from 3.3753 to 2.0656 (38.8%38.8\%) and increases mean final accuracy from 65.32%65.32\% to 65.57%65.57\% (+0.25 percentage points), with lower loss in all five pairs and higher accuracy in four.

These findings are complemented by the language-model results, currently updated after adding 5th5^{th} seed run as per your question (Q1): CC-ADMM reduces final Alpaca loss from 1.29741.2974 to 1.26341.2634 and WikiText-103 perplexity from 10.31710.317 to 9.3869.386 (9.03%9.03\%) on Llama-3B. On Llama-1B, final perplexity decreases by 12.3%12.3\%. Both final metrics show 3 SDs separation from QLoRA at both scales. The original CIFAR-100 MLP and CNN comparisons also show accuracy gains of 4.604.60 and 4.144.14 percentage points, respectively, with 3 SDs separation (Section 5.2).

W2. Why departure can lead to a better real solution. We agree that departure alone is insufficient. Before presenting additional experimental support, let's provide theoretical argument: Our theory establishes complementary theory for CC-ADMM exploration dynamics. Theorem 1 constructs a path with controlled loss increase, and Corollary 1.1 guarantees a path to a real point outside every bounded basin covered by its assumptions, with an explicit finite loss bound. The skew auxiliary LM=2x⊤MyL_M=2x^\top My supplies the driving mechanism: its imaginary gradient −2Mx-2Mx initiates exploration, and its real gradient 2My2My couples imaginary motion back into real updates (Eq. (3), Lemma 16). Theorem 2 and Remark 18 establish conditional initial departure for exact primal minimization and an ordinary gradient step, respectively. Crucially, a decrease in the auxiliary can compensate for increases in task loss and the consensus penalty while the primal objective decreases (Remark 18), permitting uphill movement in task loss during exploration. Our response to Reviewer asFm, Q1 explains this connection between path geometry and exploration dynamics in detail. The restart supplies the additional empirical evidence that the specified updates reach a better real endpoint.

To provide direct experimental evidence (on top of existing results) of CC-ADMM's ability to find lower loss optimum even if initiated at the local optimum, we added the following experiment: a restart of Figure 1 from a verified strict real local minimum: gradient norm 2.2×10−162.2\times10^{-16} and Hessian eigenvalues 0.653,2.0000.653,2.000.

After 2,500 updates without added noise, CC-ADMM reaches real loss 5.0×10−95.0\times10^{-9} after projection/refinement, whereas real gradient descent, zero-weight-decay AdamW and the auxiliary-off controller remain near 11. Thus, the specified exploration succeeds where those unperturbed controls do not, on this objective and initialization. This uses Figure 1's activation-based extension, distinct from Eq. (2). We will add the figure and protocol to the revised paper's appendix.

Q1. The missing Llama-3B run. The run was interrupted during training and long enough to not have finished in due time. We apologize for incompleteness. We were able to rerun it in the meanwhile, obtaining final real-model Alpaca loss 1.26401.2640 and WikiText-103 perplexity 9.3759.375, consistent with the reported results. We will update the paper’s figure and table to include the completed run.

We hope the existing matched controls, direct restart and completed rerun support reconsideration.

Reviewer_X4HZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies nonconvex problem and proposes CC-ADMM, which introduces imagery as an additional parameter to escape local minimum. The main idea is using imagery as a temporary parameter to allow model explore imaginary directions then bypass loss barriers. The authors tests the approach across several benchmarks including CIFAR-100 CIFAR-10 Rotation, variations of Llama-3.2, Grokking benchmark, and shows some accuracy improvement.

Critical strengths
  1. The paper propose a novel method to escape the stationary point by introducing the imagery point to provide an addition direction.
  2. The proposed method is tested across different benchmark give a comprehensive evaluation of the method. It shows promising improvement in some benchmarks.
  3. The proposed imagery parameter is only introduced during training phrase so it would require extra effort during the inference phase.
Critical weaknesses
  1. The motivation to introduce the imagery parameter is to provide an addition path to escape the local minimum but I find the evidence of directly supporting such claim lacking. If the method is intend to be introduced as a general purpose training strategy, the current benchmarks do not fully establish a compelling advantage over conventional optimization methods. The paper may benefit from including more non-convex optimization benchmarks.
  2. The method leads to substantial computational overhead during training, according to table 13 it requires 1.49x - 1.84x more wall clock time compare to baseline method. The reported performance improvement in contrast is relatively modest or inconsistent.The paper along doesn't justify such additional computation cost as most comparison was done under matching epochs or optimizer steps rather than matched GPU hours or wall clock time.
  3. The paper does not sufficiently demonstrate whether the improvement showed can be attribute to imagery numbers alone. Since the complex value parameter can be represented with set of real numbers, it's unclear whether such benefits can only be achieved through imagery or using conventional real-valued auxiliary parameters can achieve similarly effect.
Questions for authors
  1. How does CC-ADMM differs from existing real-valued auxiliary-variable?
  2. How sensitive is CC-ADMM to its additional hyperparameters and computational budget?
  3. Could direct empirical evidence that CC-ADMM escapes suboptimal local minima be provided?
2 comments

We thank the reviewer for the time and thoughtful comments. Let's note, that CC-ADMM is a temporary exploration framework used with a suitable optimizer of the user's choice, such as AdamW or SGD. It is not a general-purpose optimizer replacement as we apparently wrongly gave the reviewer an impression of. Extra training computation is a natural cost to pay to support auxiliary exploration and consensus updates. Under the stated conditions, the analyzed updates initially depart from the real stationary points into imaginary directions. The projection guarantees a return to the real parameters, after which training inherits the chosen optimizer's convergence guarantees under their applicable assumptions. For the real SGD tail, Theorem 3 proves loss convergence and vanishing gradients under its stated assumptions.

Our current Introduction does not make CC-ADMM's purpose and training-cost trade-off clear enough. We will amend it with the following statement placed just before the existing paragraph: "Notably, the following three properties ... ... provide learnable additive shifts":

Proposed CC-ADMM is a temporary exploration framework that complements a suitable optimizer, spending additional training computation to seek better final real models. It combines conditional guarantees of initial departure into imaginary directions with an exact return to real parameters by projection, after which training inherits the chosen optimizer's convergence guarantees under their applicable assumptions. It's designed for highly non-convex settings.

The specific concerns raised are addressed in the below one by one.

W1. Direct evidence and benchmark breadth. As we noted above, the CC-ADMM method is not meant to be a "general purpose" optimizer and thus we believe it is particularly useful for applications dealing with highly non-convex problem solving. You rightly ask whether exploration actually escapes a suboptimal minimum. To make Figure 1's mechanism explicit, we performed a deterministic restart at its strict real minimum: gradient norm 2.2×10−162.2\times10^{-16} and Hessian eigenvalues 0.653,2.0000.653,2.000. With 2,500 updates and no added noise, CC-ADMM reaches the real loss 5.0×10−95.0\times10^{-9} after projection/refinement. Real parameter gradient descent, zero-weight-decay AdamW and the "auxiliary-off" controller remain near loss 11. This demonstrates escape and a better real endpoint for that objective and initialization. We will include the experiment's protocol and a new figure showing these results in the revised paper's appendix. The submitted paper already includes non-convex image-classification experiments with ResNet-18 and ViT, as well as rotation regression, grokking with transformers, and language-model adaptation supporting the theory.

W2. Training overhead versus benefit. Table 13 reports that CC-ADMM takes 1.49–1.84 times more training time, reflecting its additional exploration and consensus computation. Our comparisons match training epochs and demonstrate the reported improvements in final-model quality at this additional cost. Performance under an identical wall-clock budget is a separate comparison and depends on the settings, e.g., number of epochs, etc. In existing same-architecture, equal-epoch comparisons (Section 5.2, Table 3) mean final accuracy gains of +0.28+0.28 and +0.92+0.92 percentage points for ResNet-18 at 900 and 2700 epochs, respectively, and +0.95+0.95 for ViT are shown. ViT test loss decreases by 24.01%24.01\%. All three corresponding seeds improve both accuracy and test loss in each of these settings. At 300 epochs, however, ResNet-18 trained by CC-ADMM is worse by 2.402.40 p. points. The benefit therefore depends on the training schedule. In particular, these results support the intended quality–cost trade-off: improved final models in the reported longer-budget ResNet and ViT settings, at additional training cost. This is an observed trend and we report consistency across the tested seeds without general superiority guarantees. ResNet-18 and ViT retain the baseline inference architecture and parameter count.

W3. Imaginary versus real auxiliary coordinates. We agree that real-valued auxiliary formulation can reproduce the same effect if it implements the same lifted objective, skew coupling and consensus updates. It then becomes an equivalent representation of CC-ADMM. Our contribution is the explicit construction of these exploration dynamics and its theory.

The skew auxiliary LM=2x⊤MyL_M=2x^\top My supplies the imaginary gradient −2Mx-2Mx when the real task gradient vanishes, and couples subsequent imaginary motion back into real updates through 2My2My (Eq. (3), Lemma 16). Theorem 2 and Remark 18 establish initial departure under their respective assumptions. In our deterministic restart, disabling this auxiliary prevents escape despite retaining the additional coordinates, consensus and dual updates. Enabling it leads to a lower-loss real endpoint. This directly demonstrates the role of the specified coupling in that experiment. We do not rule out that other real-valued auxiliary constructions may achieve similar effects. However, none are currently known to the best of our knowledge.

Q1. Difference from real auxiliary-variable methods. The distinguishing mechanism of our method is a nonzero imaginary force even when the real task gradient vanishes. For Eq. (2), at real consensus with zero dual, the task has zero imaginary gradient and the auxiliary gives ∇yLaux=−2λMx\nabla_y\mathcal L_{\rm aux}=-2\lambda Mx, where xx is the real parameter, yy imaginary one, MM the skew coupling and λ\lambda its weight. When λ>0\lambda>0 and Mx≠0Mx\ne0, ordinary gradient descent initiates departure (Remark 18).

The differentiating element of CC-ADMM is thus the skew auxiliary term, that also permits temporary task-loss increases and thus invokes an 'escape' mechanism: a sufficiently negative contribution can offset those increases and the consensus penalty, while the primal objective decreases (Remark 18). Our response to Reviewer asFm, Q1 explains how this mechanism complements the bounded-loss paths established by Theorem 1 and Corollary 1.1.

Q2. Hyperparameter and budget sensitivity. To check whether the restart succeeds only for one particular parameter choice, we tested all 36 combinations of four auxiliary strengths, three consensus penalties and three auxiliary switch-off times. Every run used the same Figure 1 objective, initialization and 2,500-update budget. All reached a final real loss below 0.10.1, showing that the successful endpoint persists across these tested choices. We will report these results and settings in the revised paper's appendix. This evidence concerns the Figure 1 example. In Limitations we agree that neural-network sensitivity to CC-ADMM hyperparameters is a next step, yet it constitutes a significant research effort, as there are multiple possible paths including choice of matrix MM and its degrees of freedom (Theorem 2 only requires antisymmetry and bounded trace, leaving large degrees of freedom) and ADMM hyperparameters. Therefore we consider it being out of scope of this already long paper and leave it as future work. Never the less, W2 discusses the existing ResNet comparisons across training budgets.

Q3. Direct escape demonstration. Our theory establishes more than the availability of an initial direction. Specifically, the Corollary 1.1 guarantees a complex path from every strict real local minimum with a bounded basin, for a fixed real optimization rule, to a real point outside that basin, under its stated assumptions. For a basin contained within radius DD, the construction reaches a real endpoint at distance 2R>D2R>D, with an explicit finite loss bound. The MM-based auxiliary provides the complementary mechanism for initiating exploration. W1’s restart tests whether the resulting updates reach a better real endpoint on the stated example. We will include its protocol and figure in the revised appendix.

We hope these distinctions and direct evidence support reconsideration.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Updated manuscript and comments

Dear Reviewers,

Thank you for your careful and helpful feedback. We have addressed all your comments in our responses and revised the manuscript with clarifications (highlighted in blue font in revised manuscript) and additional experiments: a deterministic restart from a strict real local minimum, a 36 setting sensitivity study, and five-seed architecture-matched CNN controls, documented in a new appendix figure and tables.

We appreciate your time and consideration of the updated manuscript. Thank you.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.