acceptodds
Under review as a conference paper at ICLR 2027

Local proxies for memory-efficient training

Abstract

Local learning algorithms can reshape resource requirements of artificial intelligence. The memory bottleneck of standard end-to-end backpropagation lies in the need to store all network activations until the loss gradient has been propagated back through. Local learning avoids this, but typically at a cost in accuracy. We study how to achieve accurate and memory efficient local learning in transformers. For each transformer block, we train a local proxy network to predict the global network output via distillation. Gradients can be independently propagated through this local proxy while estimating well the backpropagation gradients. In theory, this approach is exact for deep linear networks; in non-linear residual models, the gradient error can be bounded when using the appropriate residual proxy. We verify this numerically and demonstrate that optimizing the distillation loss improves gradient estimation. Evaluating the performance of proxy propagation on CIFAR-100 in vision transformers, we find that it achieves a better accuracy/memory trade-off compared to other local algorithms. Finally, we show that the method is further relevant for fine-tuning large language models on a small memory budget. Our method trains faster and achieves higher performance than zeroth-order memory-efficient learning algorithms with comparable memory cost.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
local learning, memory efficiency, fine-tuning, residual connection

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_cjjROpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProxyProp, a local learning algorithm designed to reduce activation memory during the training of deep neural networks, particularly transformers. The central idea is to attach a trainable auxiliary proxy network to each block, which approximates the output of the remaining network through a distillation objective. Instead of propagating gradients through all subsequent layers, ProxyProp propagates the global output gradient through the local proxy to update each block independently.

The authors provide theoretical analyses showing that an optimal linear proxy recovers the expected downstream Jacobian under Gaussian assumptions and yields exact gradients in deep linear networks. For nonlinear residual networks, they establish a gradient approximation error bound under assumptions of small residual Jacobians.

Experimentally, the method is evaluated on CIFAR-100 with vision transformers and on several language understanding tasks with pretrained LLMs. The results suggest that ProxyProp achieves a favorable accuracy–memory trade-off compared with existing local learning methods and zeroth-order optimization baselines.

Critical strengths
  1. Practical motivation. The paper addresses an important problem in memory-efficient training. Reducing activation storage while preserving training performance is particularly relevant to large transformers and long-context fine-tuning.

  2. Promising empirical results. ProxyProp achieves competitive performance compared with end-to-end backpropagation while substantially reducing peak GPU memory. The evaluations on both vision transformers and pretrained language models demonstrate the potential applicability of the method across different architectures and tasks.

  3. Connection between local learning and gradient approximation. Rather than relying solely on auxiliary local classification objectives, the proposed approach explicitly aims to approximate global backpropagation gradients. The use of output distillation to construct a local gradient pathway is intuitive and provides a useful perspective on local learning.

  4. Empirical analysis of proxy design. The paper investigates several proxy architectures and training objectives, including linear and nonlinear proxies, residual parameterization, and gradient-based auxiliary objectives. These comparisons provide useful evidence about the effectiveness of the proposed design.

Critical weaknesses

1. Limited methodological novelty and insufficient justification of key design choices

Using auxiliary modules to approximate global gradient information is not new in local learning. ProxyProp appears closely related to Direct Feedback Alignment (DFA) and synthetic-gradient methods, mainly replacing fixed random feedback with a trainable nonlinear proxy learned through output distillation.

While this extension is practically useful, it remains unclear whether the improvements come from the trainable feedback pathway, proxy architecture/capacity, or distillation objective. More controlled ablations with matched auxiliary architectures and capacities, varying only the training objective or gradient propagation rule, would help clarify the methodological contribution.

2. Limited theoretical insight into the advantage of learned proxies

Proposition 3.4 establishes an O(ϵ)O(\epsilon) gradient approximation error under small residual Jacobians. However, this bound largely follows from the near-identity structure of residual networks and also holds for an identity proxy without training, as acknowledged in Appendix B. It therefore does not explain why a learned proxy should outperform identity feedback or how minimizing the distillation loss improves gradient accuracy.

Moreover, output approximation does not necessarily imply Jacobian approximation, and the theoretical connection in Proposition 3.1 relies on Gaussian assumptions and an optimal linear proxy. A stronger justification would connect distillation error to gradient approximation error, ideally supported by direct measurements of gradient alignment during training.

3. Baseline presentation and interpretation need clarification

Several baseline comparisons are not sufficiently clear:

  • Full BP: The full backpropagation result is missing from the main body of Table 1, although reported in its caption and Appendix Table A2.
  • Backprop (one block) vs. identity proxy: The former often performs substantially better, but their different objectives and gradient pathways are not clearly explained.
  • Local loss: Belilovsky et al. (2019) originally used convolutional architectures. It is unclear how its auxiliary classifier and training objective were adapted to transformers, and whether the baseline received comparable optimization and auxiliary-model capacity.

4. Insufficient gradient-matching evaluation and objective comparisons

Since the main motivation of ProxyProp is to approximate global backpropagation gradients, direct gradient-matching experiments on the target architectures are important. The paper reports gradient alignment in toy networks and a frozen pretrained ViT, but does not sufficiently evaluate gradient approximation quality throughout actual ProxyProp training, across blocks and training stages, or relate it to downstream performance.

Additional comparisons would also help isolate the role of the distillation objective:

  • Direct gradient supervision: Train the auxiliary module to predict global or local gradients rather than output representations. Although a DNI-style loss is considered, a simpler gradient-matching objective with detached targets may provide a useful comparison.
  • Direct local classification: Optimize L(pϕ(zℓ))\mathcal L(p_\phi(z_\ell)) rather than output distillation. Appendix Table A2 includes a related baseline, but its implementation and distinction from ProxyProp deserve clearer explanation.

5. Memory complexity analysis and comparison with gradient checkpointing are not sufficiently convincing

The memory analysis in Table A1 and Figure 5 requires further clarification. Training memory includes parameters, optimizer states, and activations. ProxyProp reduces activation storage but introduces additional proxy parameters and optimizer states.

First, Table A1 assigns O(L)O(\sqrt L) activation storage to BP with checkpointing and suggests that transformer-specific intermediate activations prevent optimal checkpointing. However, these intermediate activations primarily affect per-layer memory costs rather than necessarily changing the asymptotic dependence on depth. Different recomputation strategies also provide different memory–compute trade-offs.

Second, Figure 5 reports substantially lower memory usage for ProxyProp than BP + LoRA + checkpointing, but the checkpointing granularity and recomputation strategies are unclear. Standard transformer checkpointing retains layer-boundary activations while recomputing internal attention/MLP activations. For a ProxyProp block containing multiple layers, these internal activations may be considerably larger than the block input and should contribute substantially to peak memory during local backpropagation.

Questions for authors
  1. Novelty and relationship to DFA: What is the essential methodological distinction between ProxyProp and a DFA-style method equipped with a trainable nonlinear feedback module? Can the authors isolate the improvements due to proxy training, proxy architecture, and the output distillation objective?

  2. Proxy capacity: How sensitive is the performance to proxy depth, hidden dimension, parameter count, and computational cost? Would a stronger auxiliary network also substantially improve the local-loss or synthetic-gradient baselines?

  3. Training objective: What prevents directly training the proxy using the local task loss L(pϕ(zℓ))\mathcal L(p_\phi(z_\ell))? How does this compare with the proposed distillation objective when the proxy architecture and optimization budgets are matched? Please also clarify how the baseline in Appendix Table A2 differs from this formulation.

  4. Gradient supervision: Since the algorithm already computes the global output gradient during the initial forward pass, would directly matching the propagated gradient with a detached target provide a simpler alternative to output distillation? Is the large memory cost of the DNI-style objective inherent to gradient matching or specific to its implementation?

  5. Theoretical guarantee: Since the bound in Proposition 3.4 also applies to the identity proxy, what theoretical property of the learned proxy guarantees an improvement over identity feedback? Can the approximation bound be expressed in terms of the distillation error or the proxy Jacobian fitting error?

  6. Baseline implementations: What auxiliary architectures and optimization settings are used for local loss, DFA, and DNI in the transformer experiments? In particular, how was the convolutional auxiliary network in the original local-loss method adapted to ViT?

  7. Memory complexity: Why does Table A1 suggest that internal transformer activations prevent the standard checkpointing memory–compute trade-off? Could the authors distinguish the asymptotic dependence on network depth from the per-layer activation requirements?

  8. Checkpointing fairness: Does ProxyProp in Figure 5 use the same gradient checkpointing strategy as BP + LoRA + checkpointing? For a multi-layer ProxyProp block, how many internal activations are saved during the local backward pass, and what is their contribution to peak VRAM?

4 comments
AuthorsOpenReview ↗

Response to Weakness

Dear reviewer, thank you for the valuable feedback. We respond to your concerns and questions below:

Weakness 1

Even without a non-linear proxy, there are subtle and important differences even between linear ProxyProp and a learned DFA. To show this, we add Appendix Table A5, which focuses on the comparison between DFA and ProxyProp with linear proxies. We add a new baseline (non-residual linear proxy) which is arguably most similar to a learned DFA: a direct feedback initialized at random and learned with our distillation loss. It performs lower than our residual linear proxy baseline because it lacks the residual parametrization (init at identity and regularized to identity), which is crucial in residual networks as explained by Proposition 3.3 to 3.5. These differences are also visible empirically with transformer proxies in Fig. 3 and the performance gain on harder tasks as seen in Fig. 4. Also, to the best of our knowledge, there is no learned DFA variant that has theoretical analysis and approximation guarantees in residual networks (Proposition 3.1 to 3.5).

Regarding synthetic gradients, we have done two controls to verify that our gradient model (eq. 1) and distillation loss (eq. 2) are simpler and cheaper than the DNI without loss of performance.

Design choice justification: In addition to the 10 variants of ProxyProp from Table A4 and A5, we add a control and another table to isolate the distillation and the non-linear proxy capacity. We augment Local Loss with a non-linear classifier head parametrized like the proxy. It matches the capacity of non-linear ProxyProp, but trains local auxiliary networks with BP of local loss instead of equation 1 and 2. This control is consistently worse than ProxyProp with non-linear proxies seen in Table A6.

Weakness 2

The bound of 3.4 is valid for the identity proxy. We agree that it is important to show that it holds for learned proxies with better error bounds, thus adding a new Proposition 3.5 to generalize 3.4 to the learned proxy from Proposition 3.1. This bound is better than the bound for the identity proxy when the network is not too strongly non-linear.

In Appendix B, we relax the Gaussian assumption of Proposition 3.1 and obtain a slightly weaker relationship between the true Jacobian and linear proxy Jacobian. Propositions generalized without the Gaussian assumption are in the appendix now.

To confirm empirically the relationship between distillation loss and gradient approximation, we add analysis in the new Fig. A1. Across blocks, we observe a strong correlation between gradient alignment and distillation quality.

Weakness 3

The BP result is not inside the table because each column corresponds to a different local block partition, so we report BP above the table. We clarify the implementation of Backprop (one block) in Appendix E.1, with clearer naming in Table A4. The implementation detail of Local Loss is in Appendix D.2.1. Local Loss in Table 1 has a linear head per block, and we add a control with non-linear head matching the proxy in Table A6 (see weakness 1).

Weakness 4

  • Insufficient evaluation of gradient approximation: we refer to our answer for weakness 2 and new Fig. A1.
  • Additional comparisons: regarding the suggested “direct gradient supervision” baseline, we recall that computing the real gradient would require BP, so a “direct gradient supervision” with the true gradient undermines our goal to reduce memory. However, this is an instructive theoretical upper bound for our distillation objective. This was our approach in Fig. 2 with the “ideal linear projection”. In practice in ViT, we implemented DNI as the closest published algorithm in two flavors for comparison. Regarding the request for clarification on “direct local classification”, we refer to our answer for Weakness 1.

Weakness 5

We agree that ProxyProp reduces the activation storage but not the parameter and optimizer states and introduces the proxy parameters. We highlight, however, that in contemporary LLM finetuning using LoRA with long context, activations dominate the memory storage, and the overheads of the proxy become negligible.

To provide concrete numbers for Fig. 5 with Llama 8B, from BP to ProxyProp with 8 blocks, only 30Mb of additional memory is needed from rank-64 proxy parameters and optimizers, but activation storage drops from 17Gb to 2.9Gb for a sequence at length 10k. Additionally, we add a new experiment in Table 2 to demonstrate that this gain of ProxyProp in a long context is relevant at scale on contemporary benchmarks. We fine-tune Qwen2.5 32B on mathematical reasoning, where we compare training algorithms under a fixed total RAM capacity. Adding an epoch of ProxyProp training with longer context improves the BP baseline.

Regarding the asymptotic depth-dependence of checkpointing in transformers, we refer to Question 7. Regarding clarifications on checkpointing strategies with ProxyProp, we refer to Question 8.

AuthorsOpenReview ↗

Response to Questions

Question 1

We refer to our answer for Weakness 1

Question 2

The search space of the proxy architecture is potentially large, so we did not perform an exhaustive search in this search space. We identified that a single layer of transformer backbone with a smaller width (at least half of the main network) was cheap and performed relatively well for training a network from scratch. Deeper proxies are more costly in memory. Across the ViT and LLM fine-tuning tasks of Figure 4, this strategy provided stable performance with little RAM usage of the proxy. There is certainly headroom for studying the impact of the proxy capacity further in future work.

Question 3

We have studied the local loss with proxies in our response to Weakness 1. We have also clarified the implementation in Appendix D.2.1

Question 4

We refer to our answer for Weakness 4

Question 5

As we illustrate in our answer to Weakness 2, we now show that the learned proxy is better than the identity proxy in Proposition 3.5. We have not studied extensively the gradient approximation bound in terms of the distillation error beyond Proposition 3.1. Empirically, the new Figure A1 supports that lower distillation error correlates with better alignment.

Question 6

We refer to our answer for Weakness 3. Instead of augmenting the classifier head of every block with a convolutional layer as done in Belilovsky 2019, we augmented the classifier with a transformer layer matching our proxy in Table A6.

Question 7

Regarding the theoretical scaling analysis of Table A1, we added clarifications in the Table caption and in Appendix A.1. This is because the stored activation within each transformer layer is so large that the cost is close to storing input activations of all layers. We agreed with the reviewer that the Table caption was misleading because the asymptotic scaling remains true in theory although it is not favorable in practice.

Question 8

ProxyProp uses the same gradient checkpointing strategy as BP + LoRA + checkpointing. Since gradient checkpointing caches input activations of each transformer layer, in a transformer with L layers, ProxyProp + checkpointing with B blocks would cache L/B activations, whereas BP + checkpointing would cache L activations.

Thank you for the detailed and thoughtful responses. I particularly appreciate the additional ablations disentangling the contributions of the proxy architecture and the training signal, which demonstrate that both components contribute to the performance improvements.

The revised Appendix A.1 also clarifies my concerns regarding gradient checkpointing and memory efficiency. As a minor suggestion, I would recommend distinguishing standard checkpointing from nested checkpointing, as well as ProxyProp with and without checkpointing, in Table A1. Their forward computational costs are different: approximately (2L) forward operations for BP with standard checkpointing and ProxyProp without checkpointing, versus (3L) for BP with nested checkpointing and ProxyProp with checkpointing. This would make the memory–compute trade-off clearer.

Overall, the responses and revisions have addressed most of my concerns, and I will raise my rating to 3.

Dear reviewer, thank you for acknowledging our rebuttal and generously raising the score.

We also thank you for pointing out the nested checkpointing, which we have not discussed in detail. We agree with you that ProxyProp+checkpointing requires O(3L)O(3L) forward FLOPs. Nested checkpointing could also reduce activation memory to O(2L)O(2\sqrt{L}) by using O(3L)O(3L) forward FLOPs. We will clarify this comparison in Table A1 and Appendix A.1.

Reviewer_B2k3OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework that tries to lower the effort of back propagation. Instead of doing a full back propagation, the authors try to calculate the gradients via local proxies which are small modules whose gradients can be efficiently calculated. Under strict assumptions the authors prove that Jacobian expectations are close. The authors run experiments on experiments on ViT and LLM fine-tuning and show that the proposed framework can save substantial memory usage.

Critical strengths

I think the algorithm aims to address an important topic of how to train a neural network with less memory usage. The algorithm is somehow adaptive as it requires training along with the main network. Compared with baselines, the algorithm seems to be able to preserve more performance.

Critical weaknesses

The authors provide a theory trying to explain the reason why the proposed algorithm can mimic the learning behavior of full training. The theory seems to suggest that when the output is closed then the learning dynamics (e.g. gradient, jocabian) will be similar. However the theory only holds for very strong consumption. In real life when the outputs are similar the gradients are not necessarily. The authors' experiments also show this mismatch. A better explanation is more helpful.

I think the authors' work falls into the category of activation compression and gradient compression. There are a lot of related works out there which the authors did not discuss. For example, the authors should at least discuss Mesa, TinyTL, Back Razor, ActNN as related works. Discussing with DNI is not sufficient in my humble opinion.

Questions for authors

Better explanation of mechanism is desired. Current theory does not apply to the scenario of authors' experiments. For example, if p_l is a very wide non-linear network with frozen randomized weights, what would happen?

1 comment

Dear reviewer, thank you for the valuable feedback, which is very helpful for us to improve the paper. We respond to your concerns and questions below:

Weakness 1

The reviewer writes that the theory only holds for “very strong assumptions”. One key assumption to show that gradient alignment arises from distillation is the Gaussian assumption. We now add a remark with a proof in Appendix B, to explain that the Gaussian assumption of proposition 3.1 and 3.2 is not necessary.

The reviewer is concerned when “the outputs are similar yet the gradients not necessarily”, so we now empirically measure the distillation and gradient alignments in the new Figure A1. We confirm that the empirical simulations are coherent with the theory, and observe that distillation quality and alignment with the backpropagated gradient are strongly correlated.

Weakness 2

We thank the reviewer for pointing out insufficient literature review. In Section 2, we add a paragraph reviewing other memory-saving approaches and discussing how ProxyProp is complementary to many of them, including, but not limited to, the suggested activation compression methods.

However, we agree only partially that ProxyProp “falls in the category of activation and gradient compression”. ProxyProp replaces backpropagation as an algorithm to compute the gradient, but it can be combined with many other memory-efficient training techniques as BP does. In particular, activation compression methods reduce the memory of each activation, while ProxyProp reduces the number of stored activations. Additionally, we analyze how ProxyProp can be combined with activation checkpointing to reduce memory and fit in longer contexts in Figure 5 and Appendix A.1. New experiments in the last paragraph of Section 4 and Table 2 also show the practical benefit of fine-tuning with longer contexts via ProxyProp.

Question 1

To justify the “mechanism” of ProxyProp, we had already derived theoretical contributions which are not usually studied for local learning (we are not aware of comparable results for DFA, local losses or DNI). We now add Proposition 3.5 in the manuscript, which further shows that a learned proxy enjoys bounds of the gradient approximation error in a residual architecture. As an empirical analysis of the algorithm mechanism in addition to Figure 2 and 3, we add appendix Figure A1 as discussed above.

Regarding the study of a proxy network with random weights. As indicated in Table A4, we find random linear proxies not as effective as learned ones, which is also consistent with our theory. Increasing the width would undermine our purpose of memory saving. However, we are curious to learn more about the intuition of the reviewer.

Reviewer_k9TuOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This proposed work introduces a method for memory-efficient training using local proxy networks. The authors propose ProxyProp, which partitions a transformer into blocks, and introduces a proxy network in each block to approximate global backpropogation. Each proxy network is trained to predict the final layer representation of the transformer. The gradients are then propogated from the proxy networks instead of the downstream transformer layers, removing the need to store intermediate activations. The authors provide theoretical results on conditions under which proxy networks approximate or recover the gradients from standard backpropogation. They also empirically evaluate the method in two settings, demonstrating improved memory-efficiency trade-offs compared to other local learning methods.

Critical strengths
  • The writing is clear and easy to follow
  • This work addresses a crucial challenge in scaling up model training. Experimental results demonstrate that ProxyProp can achieve performance close to standard backpropogation while significantly reducing memory consumption.
Critical weaknesses
  • I would have liked more details on how distillation affects ProxyProp. Intuitively, it seems easier for proxies of later layers to predict the final representation than those of earlier layers. Additionally, since the proxies are trained alongside the main network, they must learn a continuously changing target. It is unclear how well the proxy networks approximate the true gradients in the empirical setting. It would be interesting to have a section on how distillation performance and gradient approximation quality vary across layers and throughout training.
Questions for authors
  • Would using more powerful proxy networks for earlier layers improve performance?
  • Since the downstream network is continuously changing during training, how effectively can the proxy networks track this changing target? While the theoretical results establish gradient approximation guarantees under certain assumptions, it is unclear how these guarantees translate to the setting where both the proxies and the main network are trained simultaneously.
  • Could the authors provide more explanation on why zeroth-order optimization methods perform significantly worse than ProxyProp in the experiments? Is this primarily due to noisy gradient estimates arising from random perturbations, or are there other factors contributing to the gap in performance?
1 comment

Dear reviewer, thank you for the valuable feedback, which is very helpful for us to improve the paper. We respond to your concerns and questions below:

Weakness:

Following the reviewer’s suggestion, we add Appendix Figure A1, which is a useful addition to the paper. We measure the gradient approximation along with the distillation alignment at different blocks and throughout training. This figure empirically confirms that better distillation quality correlates with better gradient alignment. In agreement with the reviewer’s intuition, we confirm that the alignment is higher for deeper layers. We also observe that, early in training, gradient alignment is high, as expected from propositions 3.3 to 3.5.

Question 1:

We think it is true that using a more powerful proxy network for earlier layers could improve alignment to the proxy network. Our current result on comparing linear and non-linear transformer proxies (row 1 and 3 in Table 1) supports this. However, from a memory perspective, the peak memory consumption is already paid when computing gradients for the block with the largest proxies. Hence, there are no resource benefits in having a proxy of different size. This is why we use equally powerful proxies across blocks.

Question 2

Indeed, the theory Proposition 3.1 assumes the proxy is optimized at each training step, so it does not track a moving target. As reported in Appendix Table A4, when using a linear proxy, we also implement a “closed-form” linear proxy, more consistent with the theory of Proposition 3.1 and 3.5, which works well. The deviation from the theory is that the expected Jacobian is estimated with a finite batch size. In practice, we find that the model has more stable performance when the proxy is updated online either via an exponential moving average of the closed-form solution or via a simple gradient descent step of the distillation loss (see Table A4). We therefore applied the simpler variant with a single step of gradient descent per iteration.

Question 3

In essence, the gradient estimates of ProxyProp have lower variance than with MeZo. As a zeroth-order method, MeZO could be unbiased in theory (converging statistically to the true gradient), but the variance is high and requires many repetitions of the gradient estimation. In comparison, ProxyProp uses a single sample with low variance (the proxy gradient has variance zero for a given input), though perhaps being biased in deep non-linear networks.

Reviewer_x6J5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes ProxyProp, a local learning algorithm for memory-efficient training of transformers. The network is partitioned into blocks, and each block is equipped with a small proxy network distilled to predict the final network output from the block's activation. The loss gradient is then backpropagated through the proxy instead of the downstream blocks, so that each block can be updated right after its forward pass and only one block's activations need to be stored. The authors show that the optimal linear proxy equals the expected Jacobian of the remaining layers, which makes the method exact for deep linear networks, and that a residual proxy gives exact gradients at initialization and a bounded error when skip connections dominate. Experiments on a small ViT on CIFAR-100 show a better accuracy–memory trade-off than other local learning methods, and LLM fine-tuning on OPT and LLaMAshows performance close to backpropagation, better than MeZO, with about half the peak memory of backpropagation.

Critical strengths
  1. A local learning method that approximates the true gradient. Unlike many local learning algorithms that optimize unrelated local objectives, ProxyProp explicitly aims to approximate the backpropagated gradient, and the connection between proxy distillation and the expected Jacobian gives a clear explanation of why this works. The residual proxy design is simple but effective, and the method clearly outperforms other local learning baselines such as local losses, DFA, and DNI in the ViT experiments.
Critical weaknesses
  1. Limited practical significance compared with standard memory-saving techniques. The main motivation is reducing activation memory, but this problem is already well addressed in practice by techniques that keep exact gradients, such as activation checkpointing, CPU offloading, and quantized fine-tuning. ProxyProp still requires a pre-forward pass over the full network, a second forward pass for each block, and the proxy computation, and it is about 1.5× slower than backpropagation, which is a cost comparable to checkpointing. In return, it gives only approximate gradients and lower accuracy. In Figure 5, even when combined with checkpointing, it only fits sequences about 50% longer with 4 blocks. Moreover, the main LLM baseline is MeZO, a zeroth-order method that is known to be much weaker, and its results are taken from prior papers rather than reproduced under the same setting. Without comparisons against stronger memory-efficient fine-tuning methods with exact or near-exact gradients, such as QLoRA, GaLore, LOMO, or more aggressive checkpointing and offloading, it is unclear in which practical scenario ProxyProp would be preferred.

  2. The theory does not cover the regime in which the method is used, and the evaluation is small. Proposition 3.1 assumes Gaussian activations, the exactness result only holds for deep linear networks or at initialization, and Proposition 3.4 requires residual branches to be small compared with skip connections. However, the paper itself shows that in trained transformers the gradient alignment is "much harder", with cosine similarity clearly below 1 in Figure 3. The theory therefore offers limited guarantees for the actual training setting. The empirical evaluation is also limited in scale: the ViT has an embedding dimension of only 128 and loses more than 8 points with 12 blocks, and the LLM experiments use only 1,000 examples on three relatively simple tasks. It remains unclear whether the method scales to more challenging fine-tuning tasks or to pretraining.

Questions for authors
  1. In which practical scenario would a practitioner choose ProxyProp over standard backpropagation combined with activation checkpointing, offloading, or quantization, given that ProxyProp also requires additional forward computation and only provides approximate gradients?
  2. How does ProxyProp compare with stronger memory-efficient fine-tuning methods, such as QLoRA, GaLore, or LOMO, in accuracy, memory, and wall-clock time under the same setting?
  3. How does the accuracy gap to backpropagation change on more challenging fine-tuning tasks, with full-size datasets, or for pretraining, where gradient accuracy may matter more?
  4. Since the main potential benefit beyond memory is model parallelism, can you provide any evidence, even preliminary, that ProxyProp brings an actual speedup in a multi-device setting?
1 comment

Dear reviewer, thank you for the valuable feedback. We respond to your concerns and questions below:

Weakness 1

The practical scenario where ProxyProp is preferred is when activation memory dominates. Long context fine-tuning of LLMs is a relevant scenario (Figure 5). We conduct a new experiment, evaluating AIME2024 performance on a Qwen2.5-32B model fine-tuned on long s1.1k reasoning traces. Under a fixed memory budget, ProxyProp is able to fit 60% longer contexts (both ProxyProp and BP use LoRA and checkpointing). Replacing one epoch of BP with ProxyProp using longer contexts, performance reaches 53% accuracy, compared to 43% when training with BP alone. Detailed results are in Table 2 and the last paragraph of section 4 in the updated PDF.

The reviewer critiques that ProxyProp allows only 50% longer context length for 50% longer training time. We argue that this gain is relevant, as shown by the 10 pp improvement on long-context reasoning benchmarks. As shown in figure 5, this gain is not small: already half of the maximal memory improvement (where only forward passes are performed).

Regarding the concern that MeZO performance was taken from literature, we clarify in Appendix D3: we use the exact same setup as MeZO, and we show that we could reproduce their number on the 13B baseline in the new Table A3.

ProxyProp is complementary to other suggested memory-saving methods. We refer to our replies to Question 1 and 2 for our justification why most techniques can be combined with ProxyProp in the same way as with backpropagation. We also added a paragraph in Section 2 to review these memory-saving methods and their relationship to ProxyProp.

Weakness 2

First, the Gaussian assumption of proposition 3.1 can be relaxed to obtain a slightly weaker relationship between the true Jacobian and the linear proxy Jacobian. We add this in Appendix B.

Second, we clarify that proposition 3.4 aims to show how ProxyProp can provide a good gradient approximation in residual networks. Removing the assumption of small residual branches makes the error difficult to analyze. However, showing an error bound with the current assumption is quite an improvement compared to other local methods such as DFA and DNI that enjoy no such guarantees.

Finally, we add Figure A1 which empirically reflects the link between distillation and gradient alignment. Gradient alignment early in training is high as predicted by 3.4, and throughout training, it consistently correlates with distillation quality.

To address the limitation on task difficulty, we refer to our answer for Question 3.

Question 1

We address the practical scenario in weakness 1. ProxyProp is not an alternative to checkpointing. In the new Appendix A.1, we analyzed why activation checkpointing can be combined with ProxyProp to further reduce memory compared to BP and checkpointing, shown also in Figure 5. ProxyProp can be combined with activation compression because it reduces the number of stored activations, while compression reduces the memory per activation. We do not compare ProxyProp with offloading, because it introduces a secondary memory level with different bandwidth.

Question 2

“Stronger” reflects a possible misunderstanding, clarified in Section 2. ProxyProp is not a competitor to these methods as it reduces activation memory; the other methods reduce the memory of the optimizer or gradient tensors. They may be used in conjunction.

Question 3

We evaluate with three larger and harder datasets.

First, we train a ViT with embedding dimension 192 on ImageNet. ProxyProp again approaches BP with better accuracy/memory tradeoff than other methods. We report the performance here, (peak RAM usage in parentheses).

BP: 72.3 (4.35 GB)

Number of layers per blockProxyPropLocal lossDFA
469.34 (1.73Gb)61.3 (1.73Gb)46.31 (1.80Gb)
262.11 (1.13Gb)44.43 (1.07Gb)29.97 (1.21Gb)
155.76 (0.93Gb)26.75 (0.86Gb)11.19 (1.00Gb)

Secondly, we evaluate ProxyProp on finetuning almost the full SST2, WIC, and Squad datasets. The gap to BP remains similar, with results reported in appendix E.2.1. Finally, the new experiment on Qwen2.5 32B with long context also shows scalability of ProxyProp.

Question 4

In the new Appendix A.2, we add an analysis comparing parallelized ProxyProp with pipeline parallelism, placing one block on one device where weights are not communicated across devices. Schedules such as 1F1B optimize wall-clock time by storing activation memory of multiple microbatches per device. ProxyProp instead computes a block’s gradient independently without waiting, so each device stores the activations of a single micro-batch and is never idle during the backward phase. Each micro-batch needs one additional forward pass, but the wall-clock time is lower than 1F1B when the number of micro-batches is small relative to the number of devices (Appendix Table A2). We leave empirical validation to future work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.