acceptodds
Desk rejected at ICLR 2027

Restoring Inputs Is Not Restoring Learning: Sparse Restoration in Multimodal Training

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Training on reduced visual views saves computation, yet restoring full images at evaluation need not recover the capability lost during training. Controlled continuations locate a recoverable contribution in the full-minus-reduced gradient at a shared learner state, beyond extra-compute and extra-update controls. Equal-norm remaining errors nevertheless produce opposite short-horizon loss changes. A local finite-horizon expansion explains this distinction through error transport across parameters and optimizer memory, followed by a signed terminal-task projection. GRAFT-V therefore restores the current raw reference-gradient mean with sparse same-state pairs and inverse-probability weighting before one optimizer update, rather than predicting a future task direction. Recovery repeats across six learner configurations; on the primary five-task experiment, fixed sampling closes 92.6% of the 2.44-point gap, while domain-stratified sampling closes 95.9% using 30.8% fewer final-training GPU-hours than reference training. The stronger-view and short-budget comparisons locate the operating boundary: preserving useful evidence directly can cost less than restoring its learning signal.

Primary area:
foundation or frontier models, including LLMs
Keywords:
multimodal large language models, vision-language models, efficient training, visual token pruning, gradient estimation, optimizer dynamics, sparse gradient restoration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_evfTOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Through experiments involving shared initialization, partial gradient recovery, directional interventions, optimizer state swapping, relevant content replacement, and comparisons under realistic training budgets, the paper systematically investigates how visual compression affects learning and demonstrates consistent recovery effects across multiple models.

Critical strengths

(1) The experimental design of this paper is particularly valuable. Rather than merely comparing final performance scores, the authors also branch from the same training state, keeping the data order, learning rate, and gradient clipping settings fixed, while varying the gradient contributions provided to the optimizer.

(2) The paper compares six learner configurations. In addition, the authors conduct short-horizon branching experiments with different sets of trainable modules, including projector-only, projector plus LoRA, a broader set of non-visual parameters, and the final few visual modules. These experiments provide evidence that the recovery effect generalizes to some extent across different models and scopes of trainable parameters.

(3) The paper clearly reports the conditions under which the proposed method does not outperform the alternatives.

Critical weaknesses

(1) The paper offers limited standalone algorithmic novelty, with its main contribution lying primarily in the empirical analysis.

(2) I did not find a comprehensive comparison with the full DualSpeed method under the same dataset, model, and computational budget.

Questions for authors

Could the authors clarify the relationship between “No inverse probability” and “View mixture”? If the implementation of “No inverse probability” is defined as g^=gc+A(gr−gc)=(1−A)gc+Agr\widehat{g} = g^c + A(g^r-g^c) = (1-A)g^c + Ag^r, then under the same training state, sampling outcomes, and optimizer settings, the gradient is gcg^c when A=0A=0 and grg^r when A=1A=1. Under ideal arithmetic, this appears to be mathematically equivalent to the gradient construction used in a standard view mixture. However, the paper reports different results for these two approaches: 80.03 for “View mixture” and 80.31 for “No inverse probability” (Tables C.2–C.3, Page 23). Could the authors explain what implementation or experimental differences account for this discrepancy?

Reviewer_kyBbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates why training with reduced visual inputs causes performance loss even when full inputs are restored at evaluation. It examines the effects of restoring full-minus-reduced gradients and analyzes how remaining gradient errors influence optimization. GRAFT-V uses sparse paired gradients with inverse-probability weighting to approximate the full-view reference gradient, recovering most of the aggressive-pruning performance gap at reduced training cost.

Critical strengths
  1. The controlled gradient-dose, error-direction, and optimizer-state experiments help distinguish the effects of missing gradient information from additional computation.

  2. Evaluations across six learner configurations, paired-seed analyses, and equal-update/equal-budget comparisons provide extensive empirical evidence.

Critical weaknesses
  1. The paper's writing and overall organization need improvement. For example, the introduction relies heavily on technical terminology, such as "recoverable learning contribution" and "signed terminal-task projection," without clearly establishing their roles in the main argument. Key notation (e.g., the gradients in Eq. (1)) is also introduced indirectly, and the connection between the mechanistic analysis and the proposed restoration method could be explained more clearly.

  2. The practical advantage is limited. Stronger visual-reduction methods achieve comparable performance at lower training cost in several settings. The sampling probabilities are insufficiently justified, and the reported efficiency gains do not fully account for calibration and preprocessing costs.

Questions for authors
  1. What new understanding of reduced-view learning does the mechanistic analysis provide beyond established optimization principles?

  2. How were the sampling probabilities chosen for Fixed (p = 0.19) and Stratified (the domain-specific probabilities)? Were they determined using calibration data or a principled allocation criterion? How sensitive are the results to these choices?

  3. How would accounting for calibration, caching, and additional reference-view access costs affect the end-to-end cost–performance comparison with stronger visual-reduction baselines?

Reviewer_1RMGOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper studies how reducing visual tokens during fine-tuning affects what vision-language models learn. It finds that supplying full visual inputs at evaluation does not eliminate the performance gap relative to full-view training. Experiments starting from shared checkpoints show that adding back part of the difference between full-view and reduced-view gradients can recover performance. Further experiments show that gradient errors of the same size can affect loss differently depending on their direction. A local Taylor expansion describes how these errors influence the final loss through subsequent parameter and optimizer updates. GRAFT-V uses an established gradient-estimation approach: it occasionally computes full-view and reduced-view gradients at the same model state and applies a scaled correction to the cheap gradient. Across several model configurations and visual question-answering tasks, this recovers most of the deficit caused by aggressive visual reduction while requiring less final-training computation than full-view training. The comparisons also show that better reduced views and short training budgets can make training without correction preferable.

Critical strengths
  1. The topic is relevant to the scope of the conference.
Critical weaknesses
  1. Poor presentation throughout the paper makes the method, experiments, and contribution difficult to follow. The introduction is extremely dense, difficult to follow, and discusses GRAFT-V's properties and results before clearly explaining its basic operation: use a reduced-view gradient at every step, occasionally compute a paired full-view gradient, and add a scaled correction before one optimizer update. Expressions such as "controlled continuations," "remaining errors," "signed terminal-task projection," and "sparse same-state pairs" appear before the reader has enough context to interpret them. Beyond the introduction, key definitions and experimental details are scattered across later sections and appendices, and several baseline methods are insufficiently explained. This repeatedly forces the reader to reconstruct the setup before being able to interpret the results.

  2. Key equal-time comparisons are missing, and important experimental choices are insufficiently documented. PriorTR and SFPruner are excluded from the equal-time comparison after configuration errors (Table 3 and Appendix F.4). Excluding invalid results is ok, but these comparisons should be rerun: both methods already achieve performance close to fixed GRAFT-V at lower cost in the equal-update experiment (Table 1). Their absence leaves an important part of the practical efficiency comparison unresolved. Table 3 does include other competitive alternatives, such as CLSE, but it cannot establish how GRAFT-V compares with these two inexpensive methods under a common time budget. The available comparisons are also difficult to reproduce. Some baseline adaptations are incompletely described, and training code and checkpoints are not released (Appendix I).

Questions for authors
  • Can PriorTR and SFPruner be rerun under the correct equal-time protocol (Table 3)?
  • How were the uniform probability 0.19 and the five domain-dependent probabilities chosen?
  • What is "VisionTrim + RESTORE" in Tables G.1-G.4 and H.6-H.7? How does it differ from VisionTrim with fixed correction in Table C.4?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.