- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CC-ADMM, which temporarily extends real parameters into the complex domain during training. A skew-symmetric bilinear auxiliary term drives motion into imaginary coordinates, while consensus, annealing, and projection return the model to real parameters. The paper presents conditional analyses of bounded-loss complex paths, departure from real stationary points, and convergence after projection, together with experiments on grokking, classification, rotation regression, language-model adaptation, and ViTs. The central idea is to improve optimization through additional training coordinates while retaining real-valued inference.
Critical strengths
1. The relationship between training-time exploration and final inference is clear. Section 4 connects complex exploration, real consensus, and final projection. The auxiliary term has a nonzero imaginary gradient when , giving a concrete local explanation for how motion can start from a real stationary point. Returning the final model to the real domain makes the approach practically appealing.
2. The ViT experiment provides a useful architecture-matched comparison. Table 3 uses the same forward architecture and parameter count, increasing final accuracy from 0.5964 to 0.6060 and reducing final test loss from 2.7572 to 2.0952. This supports the training method more directly than comparisons that also change the network architecture.
Critical weaknesses
1. The theory does not sufficiently establish the complex-path advantage emphasized for the practical algorithm. For the case explicitly covered in the main text, Equations (1) and (2) and Theorem 1 imply : the corresponding real path has pointwise no higher loss. Existence of a bounded-loss path therefore establishes no loss-barrier advantage. The exact primal update in Theorem 2 and real SGD after projection in Theorem 3 also do not directly cover the finite AdamW inner loops and implemented training procedure. Although the authors disclose some limitations, the explanation connecting theory to practical gains remains insufficient. This affects the central contribution rather than establishing that the individual theorems are incorrect.
2. The main experiments do not sufficiently isolate optimization, and some improvement directions cannot be interpreted reliably. The MLP/CNN and language-model comparisons also change activations, magnitude readout, or the adapter function class, preventing attribution of all gains to complex exploration. The architecture-matched ViT comparison in Table 3 is positive evidence but does not resolve attribution elsewhere. Furthermore, Section 5.2 lists MLP accuracies of 0.2153 vs 0.2613 in the stated order “CC-ADMM vs Baseline,” yet claims a gain of 4.60 percentage points; CNN and ResNet comparisons contain similar conflicts between ordering and gain direction. Both architectural controls and the actual direction of improvement need clarification to establish the claimed broad optimization value.
Questions for authors
1. Could you address the path comparison above? Since the real-part path has no higher loss than the complex path, is the key advantage attributed to path geometry or to the optimization dynamics induced by the auxiliary? Which part of the current argument distinguishes these explanations?
2. Could you specify which guarantees still apply to finite AdamW inner loops and which apply only to exact subproblem solutions or SGD after projection? I am not asking for a new complete theory during the response period, but I need to understand how far the current results support the implementation.
3. After projection to real parameters, does the language-model adapter retain a nonlinear function different from the QLoRA baseline? Beyond ViT, do any existing results compare training with and without complex exploration under the same final architecture? This directly affects whether I would attribute the main gains to the proposed training method.
4. Please check the method ordering and gain signs throughout Section 5.2. A corrected mapping would suffice, together with clarification of which classification results still show an improvement, particularly the ResNet results at the three training lengths.
2 comments
Reflecting your comments we clarified Section 4 immediately after the auxiliary definition: a negative auxiliary contribution can offset increases in task loss and consensus penalty, enabling exploration. We distinguish the available bounded-loss paths from the algorithm’s trajectory and point to the appendix restart experiment demonstrating escape to a lower-loss real endpoint.
Q2. Exact solves, finite AdamW and the real tail. Thank you for asking us to make this distinction explicit. Our guarantees concern different stages.
Departure. Theorem 2 guarantees a non-real exact primal minimizer under its stated curvature, non-degeneracy and initialization assumptions. Exact minimization is not the only setting in which departure is established: for Eq. (2), Remark 18 gives for an ordinary gradient step from real stationary consensus with zero dual. This proves initial departure when and . Notably, it does not establish convergence of stochastic AdamW or complete basin escape.
After projection. The final projection to real parameters is exact independently of the inner solver. Theorem 3 permits a well-defined finite exploration phase, including finite AdamW updates. Once projection is performed and the auxiliary, consensus and dual updates are disabled, the stated real-SGD assumptions give almost-sure loss convergence, vanishing gradients and stationary limit points. Thus, this guarantee does not directly require or condition solver used during exploration. In particular inexact ADMM step solver is admitted. The experimental AdamW tail is just a standard prominent AdamW training and converges (or not) accordingly. It can be replaced by any optimizer of choice in general.
Practical implementation. Approximate primal updates are established in the inexact-ADMM literature, including nonconvex extensions (Boyd et al., 2011; Wang et al., 2019). Despite commonly used, their guarantees require specified error and regularity conditions, which a fixed number of AdamW steps does not itself explicitly verify. Our architecture-matched ViT and longer-budget ResNet gains were obtained with finite stochastic AdamW steps between consensus updates, providing empirical support for practical benefits without requiring top solve primal ADMM step exactly.
We revised the paragraph after Theorem 3 in Section 4 to cite established inexact and proximal ADMM results and clarify that our real-SGD guarantee permits finite inexact exploration before exact projection. We also explicitly retain the applicable optimizer assumptions for convergence of the experimental AdamW tail.
Q3. Final architecture and attribution. Yes. After projection, the language-model adapter has real parameters but retains modReLU and magnitude pooling, so it remains nonlinear and differs from the linear QLoRA adapter. As with the original MLP/CNN comparisons, these results evaluate the combined architecture and training procedure.
Beyond ViT, the existing strong-recipe ResNet-18 comparisons in Section 5.2 use the same deployed architecture and parameter count (11.22M) for both methods, providing evidence for the training procedure with unchanged final architecture.
We additionally added new CIFAR-100 CNN experiments, also with identical final architecture in both methods. These show lower mean final test loss with a mean accuracy gain. The numerical results and consistency across seeds are detailed in our response to Reviewer exek, W1.
The existing ResNet/ViT comparisons use the finite-AdamW implementations discussed in Q2, while Q4 clarifies the method ordering and outcomes of the original Section 5.2 comparisons as suggested.
Q4. Corrected ordering and gains. Thank you for pointing out the reversed order. The order is Baseline CC-ADMM: MLP ( percentage points), CNN (), ResNet300 (), ResNet900 (), ResNet2700 (). Gains use unrounded means. ViT gains points and reduces test loss . All three corresponding seeds improve accuracy and loss at both longer ResNet budgets and ViT. These equal-epoch results show consistent gains across the tested seeds in the longer-budget ResNet and ViT settings. Following the arguments above and the proposed clarifications the paper does not claim statistical significance or general superiority under arbitrary settings. For a more detailed summary of results we kindly refer to response W1 given to reviewer exek.
We hope that the provided updates and evidence support a reassessment of the paper.
We thank the reviewer for the time and thoughtful comments. The CC-ADMM is a temporary exploration framework used with a suitable optimizer of the user's choice, such as AdamW or SGD. The extra training computation supports auxiliary exploration and consensus updates. It's designed and worth to be applied specifically to highly non-convex settings, where the trade off pays off as supported by our experiments. Under the stated conditions, the analyzed updates initially depart from real stationary points into imaginary directions. The projection guarantees a return to the real parameters, after which training inherits the chosen optimizer's convergence guarantees under their applicable assumptions. The case of AdamW, addressed in detail in Q2, is also the same case. Convergence of the method is not determined by ADMM phase, but rather by the final real phase (after back projection to real) and thus, inherited from the used optimizer for this stage, be it, SGD (Theorem 3 for completeness of the paper) AdamW or something else.
Our current Introduction does not make CC-ADMM's purpose and training-cost trade-off clear enough. We will amend it using the two-sentence revision in our response to Reviewer X4HZ.
Next, we address in the below your specific concerns one by one.
W1. Theory versus practical gains. Your real-path comment is correct and we agree that "existence of a bounded-loss path therefore establishes no loss-barrier advantage". However, it is rather the opposite that we use in the theory: Theorem 1 (importantly extended by Corollary 1.1) establishes that there exists a path with loss bounded from the top by 2Rϵ. Then our method constructs and adds an auxiliary (imposed by anti-symmetric M) that provides enough dynamics/energy to overcome this finite barrier.
Te below answer to the first question Q1, explains in detail the dynamics of the proposed method. Q2 separates the guarantees for ADMM with inexact steps during exploration, ordinary gradient steps and real stochastic-gradient descent.
W2. Attribution and inconsistent labels. Thank you for correctly identifying architectural confounds and reversed prose labels. We believe, they do not invalidate the existing matched ResNet/ViT controls. However, Q3 distinguishes their evidence from the original MLP/CNN/Alpaca comparisons. Q4 corrects the ordering, including the negative short-budget ResNet result.
Q1. Geometry or dynamics? Our theory and method address complementary parts of the escape mechanism. Theorem 1 constructs a complex path with a controlled increase in task loss, keeping the radial contribution constant. Corollary 1.1 establishes the existence of a path to a real point outside any bounded basin covered by its assumptions, with the explicit, radius-dependent loss bound in Appendix Eq. (15). These are existence results: they establish an available route and bound its loss, without prescribing an optimization trajectory.
CC-ADMM adds the driving mechanism through the skew auxiliary LM(x+iy)=2x⊤My. This term is zero on the real subspace but can become negative during exploration. A sufficiently negative auxiliary contribution can compensate for a temporary increase in the extended task loss and the consensus penalty while the primal objective decreases (please see Remark 18 for details). Notably, this permits uphill movement in task loss during exploration. The auxiliary also explains how exploration can begin: although LM(x,0)=0, its imaginary gradient is −2Mx, and its real gradient 2My couples the resulting imaginary motion back into real updates (Eq. (3), Lemma 16). Theorem 2 and Remark 18 establish conditional initial departure. The choice of M, together with its weight and the current coordinates, determines the direction and available compensation.
Thus, the pointwise inequality Lreal(x)≤Lreal(x+iy) for a non-decreasing radial term is consistent with the mechanism: the exploration updates optimize an augmented objective. Theorem 2 guarantees initial departure under its stated assumptions (geometry), while subsequent basin crossing depends on the trajectory (dynamics). The method need not follow the constructed path exactly. Especially beyond the barrier, gradient descent takes over and may descent off the bounded path.
To provide experimental evidence for the theory, we restarted Figure 1 at its strict real minimum, a saddle after lifting (gradient norm 2.2×10−16, Hessian eigenvalues 0.653,2.000). With 2,500 updates and no added noise, CC-ADMM reaches real loss 5.0×10−9. Real gradient descent, zero-decay AdamW and the auxiliary-off controller remain near 1. This uses Figure 1's activation-based extension, distinct from Eq. (2) and the revised-paper appendix will contain the protocol and figure.