acceptodds
Under review as a conference paper at ICLR 2027

Backpropagated Output Momentum: Relocating Optimizer History from Parameters to Task Space

Abstract

Optimizer momentum is usually stored as a parameter-sized moving average of past gradients, which makes history costly and fixes each past signal in the coordinates in which it was computed. We introduce Backpropagated Output Momentum (BOM), which instead stores a compact moving average of prediction errors at the model output and reprojects that history through the current network at every step. A batch-level analysis characterizes the information retained and omitted by this relocation, while the implementation preserves the current supervised gradient and can replace the first-moment component of several adaptive optimizers. As a plug-in for momentum-based optimizers, including ones that already compress their state, BOM reduces parameter-shaped optimizer state by 49.7–99.8% in three compositions and, averaged over three language backbones, paired step time by 4.0%. It also improves mean validation performance across language and vision fine-tuning, by 1.42 points in the primary five-task comparison. Language and vision pretraining studies, together with matched mechanism controls, further test the construction across output spaces and model scales.

Primary area:
optimization
Keywords:
deep learning optimization, momentum, optimizer state, memory-efficient training, fine-tuning, pretraining

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_tCNgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper attempts to reduce the memory consumption of AdamW by not tracking the first moment in AdamW. In doing so, the authors first compute the batch-averaged gradients of the final output layer for a classification task, where the gradient for each sample in the batch is the predicted probability vector minus the one-hot vector. The authors then compute a moving average of batch-averaged gradients across iterations, which is referred to as 'backpropagated output momentum (BOM)'. BOM is then utilized when performing backpropagation for the current batch. The new method still computes and maintains the second moment by using the computed gradient as in AdamW. Finally, the model is updated following the AdamW style, replacing the first momentum in AdamW with only the computed gradient. As a result, the new method saves memory consumption and slightly reduces computational complexity. Experiments in natural language processing show that the new training method produces better validation performance.

,

Critical strengths

In brief, the paper is easy to follow. The authors propose an innovative method for computing backpropagated output momentum (BOM). BOM may carry additional output error information missing from AdamW, which could be helpful in training DNN models for classification tasks.

Critical weaknesses

While it is clear that the proposed training method reduces memory usage, it remains unclear why it also improves validation performance. Is this improvement genuinely driven by BOM carrying additional output error information absent from AdamW? The authors performed an ablation study in Table 2, which is helpful. However, I think further analysis or additional ablation studies are needed to clarify the exact cause of this performance gain.

Questions for authors

(1) When plugging BOM into AdamW, the framework uses three moving average factors (denoted as beta0, beta1, and beta2), where beta0 corresponds to BOM and (beta1, beta2) to AdamW. Could the authors provide additional ablation experiments to show how different values of beta0 affect performance while keeping (beta1, beta2) fixed? Furthermore, can they evaluate combinations of beta0 and beta1 to confirm whether BOM consistently yields performance gains across various settings of beta1 for different clasification tasks?

(3) Does the batch size affect the results discussed in question (1)?

(2) In this paper, the authors primarily focus on classification tasks. How well does the BOM method extend to regression tasks, or are there limitations in that setting?

Reviewer_Na3qOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents an alternative to AdamW optimizer called BOM that swaps the order of Jacobian-vector product computation and the state summarization. To implement this, the authors focus on the cross-entropy loss, a universal choice for LLM and other classification tasks, and suggest to calculate its surrogate loss to compute the parameter update from a single backpropagation. Experiments on a fair-sized (few hundred millions to 1.7B) LLM training, the suggested BOM optimization outperforms AdamW in both GLUE task with slightly reduced training time, with a significantly reduced memory requirement.

Critical strengths
  • The proposed method is elegant, and applicable to many real-world training scenarios that are relying on AdamW (and GaLore, Adam-mini) optimizer and CE loss. The contributions are significant.
  • The scope and scale of the experiments conducted in this paper is adequate to address the superiority of the proposed method.
  • Reduction of memory is achieved without deterioration of performance or training speed is a noteworthy contribution to the field.
Critical weaknesses

The following weakness is not critical in the assessment. I am positive about the paper.

  • The presentation of the paper can be fancier, for example, adding an algorithm block, will greatly help visualize where should we modify from the existing AdamW optimizer.
  • Many numbers reported in the paper, e.g., Tables 4 or 12, are more appropriately described as comparable or competitive results than a significant improvement, if we focus only on the score.
Questions for authors
  • Can the method be extended to other tasks that uses different losses than CE, for example, MSE or MAE losses used in vision tasks and regression tasks? Adding more experiments on different loss adaptation may enhance the applicability of the suggested method.
Reviewer_JmbzOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The momentum stored in the typical optimizers are consuming a lot of memory footprint.

This paper proposed a new momentum-based optimizer that stores an exponential moving average of prediction errors in the output space. This optimizer is called BOM (Back-propagated Output Momentum). At each optimization phase, the stored history is projected back into parameter space using the current model's Jacobian.

In the past, the other momentum optimizer typical store momentum in the parameter space. This BOM approach tries to reduce the optimizer memory footprint while allowing historical information to adapt to changes in the model parameters. AdamW stores past gradients, which tells the model how to change its parameters. In contrast, the BOM stores the past prediction errors and make it through Jacobian to convert to the parameter update directions.

BOM seems to be able to work with AdamW, Adam-mini and GaLore. Experiments show it has improvements over the baselines even on the VLA tasks.

Critical strengths

The core contribution in BOM is that the idea of maintaining momentum in output space over the parameter space. Reprojecting historical prediction errors through the current model's Jacobian is an interesting direction, compared to the traditional momentum accumulation.

The BOM optimizer is memory efficiency, reduces the parameter-shaped optimizer state and can be incorporated into existing optimizers.

There are some promising results with BOM. BOM shows better results on various language backbones in five tasks.

Critical weaknesses

The authors did not explain why conducting average computation on the output space through current model's Jacobian is superior than the traditional momentum. What's wrong with information loss or altered gradient smoothing compared to the BOM? It is not touched in the current paper about this explanation.

Fundamentally, it seems that the BOM is simply average errors across the samples. This removes some history information while save memory footprint. It is unclear if the removed information is just noise or critical information. In additional to the first point, I am not sure if the average expected prediction errors in BOM can keep the useful information without hurting the model performance by losing some critical information.

The paper demonstrated some useful evidences in NLP fine-tuning. How about the language pre-training, why this does not work well? What are the limitations of the BOM? Is this only good for fine-tuning?

Questions for authors

What is the main reason behind BOM's improved optimization performance? Is this improvement due to the use of re-projecting historical gradients through current Jacobian? Could the authors show more direct evidence that current-Jacobian re-projection is responsible for BOM's performance gains?

How much important information is lost by averaging prediction errors in the output space? Since the Jacobian–residual correlations are not preserved in the historical momentum, under what conditions is the batch-averaged residual sufficient to provide a useful optimization direction?

How should readers interpret the mixed results in language pre-training?

How robust is the BOM and how sensitive is to set up the BOM regarding the hyparameters? what tasks would it work, what tasks would it not work?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.