acceptodds
Under review as a conference paper at ICLR 2027

Learning from the Past: Historical Representation Reuse with Negative Feedback for Vision-Language Models

Abstract

Vision-language models typically rely on cross-attention to capture interactions between visual and textual representations, yet intermediate representations generated during training are rarely reused across training units. Moreover, fluctuations in these representations may reduce the stability of cross-modal learning. To address these issues, we propose Historical Intermediate Representation Reuse and a Negative-Feedback Adapter. HIRR reuses the cross-attention Key and Value representations from the previous training unit, allowing historical intermediate information to participate in subsequent cross-modal interactions. The Negative-Feedback Adapter further regulates the current representation according to its deviation from a historical reference, suppressing unnecessary representation fluctuations. We analyze these mechanisms from the perspectives of statistical variance reduction and local perturbation propagation, linking historical representation reuse to effective sample complexity and feedback stability. Experimental results across multiple vision-language models and image-text retrieval benchmarks show that combining historical representation reuse with negative-feedback regulation consistently improves cross-modal learning performance and data efficiency, while demonstrating applicability across different cross-attention-based architectures.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-Language Models, Cross-Attention, Historical Representation Reuse, Negative Feedback

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_YSRFOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper suggests to reuse the cross-attention key and value representations from the previous training unit. Also, to regulate the deviation of current representation from its historical reference, the paper suggest to use the Negative-Feedback Adapter. The paper further analyze the effectiveness of the suggested technique from the perspective of variance reduction analysis of HIRR, which is linked to a reduced effective sample requirement, and a local perturbation analysis showing that the negative feedback regulation can reduce representation deviations. Experiments on image-text retrieval report improvements over the baselines, as well as improved data efficienty.

Critical strengths

The paper proposed a effective technique with potentially interesting idea to use the historical information to improve the cross-attention-based VLMs.

Critical weaknesses

Weaknesses

  • The manuscript does not currently meet the standards of academic writing expected for a conference paper.
    • The paper lacks of essential sections such as Related Work, Background, or Preliminaries. As a result, it is difficult to understand how the proposed work relates to prior literature, what gap it is intended to address, and how it should be positioned within the current research landscape.
  • More importantly, several citations do not appear to directly support the claims for which they are provided. I highlight two representative examples below:
    • For example, in the Introduction, the authors cite [1] and [2] to support the argument “such as mini-batch sampling, limited training data, and stochastic optimization.” However, these references do not appear to be directly related to this claim.
    • Similarly, the argument “their intermediate representations may therefore share statistical and cross-modal structural information even without sample-level correspondence.” is supported by citations to [3], [4], and [5], but it is difficult to identify a direct connection between these references and the stated assumption.

Given these issues, I believe the manuscript would require fundamental revisions before it could be considered suitable for acceptance as a conference paper. In particular, the citation issues described above also raise concerns regarding scholarly rigor and appropriate attribution. In its current form, I do not believe the manuscript has reached a sufficient level of academic maturity to support a meaningful technical evaluation.

[1] Li, Xiping, et al. "Cpgrec+: A balance-oriented framework for personalized video game recommendations." ACM Transactions on Information Systems 44.3 (2026): 1-44.

[2] Cheng, Zhiyong, et al. "Modeling Stage-wise Evolution of User Interests for News Recommendation." Proceedings of the ACM Web Conference 2026. 2026.

[3] Wang, Zening, et al. "ML-MIFD: Multi-Level Multimodal Invariant Feature Descriptor." The International Archives of Photogrammetry, Remote Sensing and Spatial Information Sciences 49 (2026): 79-85.

[4] Pu, Ruitao, et al. "Deep reversible consistency learning for cross-modal retrieval." IEEE Transactions on Multimedia 27 (2025): 4095-4106.

[5] Duan, Siyuan, et al. "Fuzzy multimodal learning for trusted cross-modal retrieval." 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2025.

Questions for authors

I have no question.

2 comments

Thank you for identifying the two passages. The cited studies provide relevant evidence for narrower claims, and we agree that the Introduction should make the scope of each citation explicit.

Lines 64–66 — representation variability.

Cheng et al. (2026, §4.4) explicitly discuss how short temporal windows can produce sparse subgraphs and unstable representations in news recommendation. This provides a task-specific example of a link between data sparsity and representation stability. Li et al. (2026) address over-smoothing in graph convolution, which is a different phenomenon. We will remove Li et al. from this passage, limit the attribution to Cheng et al. to its specific setting, and cite Bottou et al. (2018, §4.2) for variability in stochastic mini-batch gradients.

Lines 40–44 — shared structure and sample correspondence.

Pu et al. (2025, §I) provide a direct precedent for learning modality-invariant category representations without instance-paired training. Duan et al. (2025, §1) explain how a shared embedding space can preserve semantic relationships across modalities, and their consistency objective uses matched instances (§3.3). Wang et al. (2026) demonstrate shared structural cues in cross-sensor image matching. These works motivate the possibility of shared structure across modalities. The common expectation of successive training batches in Theorem 1 is an explicit local modeling assumption of our paper. We will revise the passage to make that distinction clear.

Technical positioning and evidence.

Section 2.1 introduces the cross-attention formulation, and a dedicated Related Work section will make the comparison to ARXSA explicit. ARXSA uses a recurrent key in self-attention. Our method uses normalized historical mixing for both cross-attention keys and values and a separate, learned output correction driven by the current-to-history discrepancy. The β-CLIP/MS COCO full-model entries in Tables 1 and 7 come from the same runs using both operations as formulated. Table 7 also tests them separately. I2T R@1 rises from 61.9 for the baseline to 62.3 with historical K/V reuse alone and 62.1 with the adapter alone, reaching 63.1 with both. The adapter-only result provides direct evidence for the added output correction with historical K/V mixing disabled.

I appreciate the authors' response about clarification. However, I'm not particularly convinced by the authors' response, and also, there are numerous citations that raise me similar concerns. I'll not change my decision.

Reviewer_T4QQOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Historical Intermediate Representation Reuse (HIRR) and a Negative-Feedback Adapter (NFA) for cross-attention-based vision-language models. The method incorporates historical attention representations to improve training stability and reports gains on image-text retrieval benchmarks.

Critical strengths

The method demonstrates consistent retrieval improvements across multiple datasets and architectures, with additional experiments on data efficiency and component ablations.

Critical weaknesses
  1. Limited novelty and lack of substantive insight: My concern is not the use or combination of existing techniques itself, but the lack of convincing new insights beyond prior work. The historical-attention mechanism closely resembles ARXSA (Findings of EMNLP 2025). More importantly, the supplied implementation uses the same recursive Key update and coefficients as ARXSA (1.3K - 0.3previous_K), making the actual implementation substantially closer to prior work than the manuscript suggests. The incremental contributions are not adequately established.

  2. Questionable semantic validity of cross-batch fusion: In image-text retrieval, consecutive mini-batches may contain entirely unrelated semantic content, yet the proposed method directly mixes their Key/Value representations without sample-level or token-level correspondence. It is unclear how such fusion preserves meaningful cross-modal information rather than introducing semantic interference. More importantly, the observed gains may simply result from feature smoothing or regularization rather than genuine historical knowledge reuse. The paper neither distinguishes these explanations experimentally nor justifies its assumption of shared underlying representations across heterogeneous batches.

  3. Serious discrepancies between the manuscript and supplementary code: The manuscript describes normalized, nonrecursive historical K/V averaging, whereas the supplied implementation uses recursive Key-only updates without historical Value fusion. The NFA formulation also differs from its implementation, and the provided training scripts do not consistently select the modified model by default. These inconsistencies make it difficult to determine which algorithm produced the reported results and whether the theoretical analysis applies to the supplied implementation.

Questions for authors

Please refer to the concerns detailed in the Critical Weaknesses.

2 comments

Which implementation produced the results?

The full-model β-CLIP results on MS COCO in Tables 1 and 7 come from the same runs. These runs used normalized historical mixing for both keys and values, as in Eq. (3), and the NFA took the current-to-history discrepancy as input and subtracted the learned correction, as in Eqs. (6)–(8). The tensor cited in the review is computed after K/V fusion. In ARXSA, the recurrent tensor itself supplies the key to self-attention. The two tensors therefore play different roles in their respective attention computations.

What does the method add beyond ARXSA?

We recognize ARXSA as the closest prior work. Our design introduces two independently testable changes to cross-attention. Historical keys affect the attention weights, and historical values affect the content aggregated by the current query. The NFA then learns a correction at the output. In Table 7, I2T R@1 increases from 61.9 for the baseline to 62.3 with historical reuse alone and 62.1 with NFA alone. The combined method reaches 63.1. In particular, the NFA-only gain provides evidence for the output correction with historical K/V fusion disabled.

Cross-batch semantics and smoothing.

We agree that consecutive batches can contain unrelated examples. The current query and supervised image–text pair come from the current batch, and the previous batch supplies a statistical reference in the same feature space. Theorem 1 formalizes this interpretation under a locally shared population expectation, which is compatible with different samples. Smoothing is a plausible contributor to the observed gains. Table 7 measures the contribution of each component to retrieval, and a statistics-matched replacement of the historical tensors would directly test whether the source of those tensors matters.

I appreciate the authors' response; however, the gains and insights offered by this paper are truly limited. In fact, the content itself is quite disappointing. Consequently, I am maintaining my original score.

Reviewer_Be1POpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Historical Intermediate Representation Reuse (HIRR) and a Negative-Feedback Adapter (NFA) for VLM. HIRR reuses key-value from the previous training unit as a historical prior without storing raw samples, and the NFA adjusts the current cross-attention output based on its deviation from a historical reference. The authors provide theoretical analyses of variance reduction and local perturbation attenuation. They evaluate the proposed method on different VLMs for image–text retrieval benchmarks across Flickr8k, Flickr30k, and MS COCO. The also provide multiple experiments such as variance analyses, the effect of different training data-ratio, generalization abilities, etc.

Critical strengths

This paper tries to explain why the two components work, rather than simply reporting empirical results. For example, HIRR is linked to finite-sample variance reduction, and NFA is analyzed by the local perturbation attenuation. These analyses offer some justification for why reusing historical K/V representations and regulating cross-attention outputs may help when training data are limited. The theoretical analysis makes the method more convincing than the one purely based on empirical experiments.

Critical weaknesses
  1. Theorem 1 assumes that the current and historical representations share the same latent H^*. However, the paper states that consecutive training units contain different samples, with no sample-level correspondence. It does not explain why the semantic differences between samples can be treated as zero-mean noise. Also, lower variance of a shared H^* does not mean better representations or better retrieval.

  2. Theorem 2 shows the output moves closer to the reference S_{t-1}. If S_{t-1} is noisy or biased, the adapter may reduce the distance to the reference while actually degrading the current representation. Thus, the theorem establishes a local contraction effect, but does not show that this effect improves cross-modal representations or retrieval performance.

Overall, the paper would be stronger if it explicitly connected these theoretical quantities to the final objective.

Questions for authors
  1. How can Theorem 1 assume a shared latent when consecutive training units contain different samples?
  2. The authors should analyze the effect of the proposed method when S_{t−1} is noisy or from a different sample?
  3. See the Critical Weaknesses.
1 comment

Why a shared target is compatible with different samples.

In Theorem 1, H⋆H^\star is a local population quantity. With fixed model parameters and batches drawn from the same distribution, one concrete interpretation is

H⋆=EB[hθ(B)]H^{\star} = \mathbb{E}_{B}[h_{\theta}(B)]

where hθ(B)h_\theta(B) is the compatible key or value representation. Two batches can contain entirely different samples and still have the same expectation. The centering is over batches. Per-example semantic content can differ substantially. During actual training, parameter drift or changes in batch composition can shift this expectation. If the historical representation has mean H⋆+bH^\star + b, the normalized estimator in Eq. (11) acquires the following additional squared-bias term in its MSE:

(α1+α)2∥b∥22.\left(\frac{\alpha}{1+\alpha}\right)^2 \lVert b \rVert_2^2.

Reuse therefore reduces MSE when the variance reduction exceeds this bias cost. This extends the original argument to the failure mode raised in the review.

When contraction is useful.

Let Z⋆Z^\star denote a task-relevant target representation, and let q<1q<1 denote the local contraction factor in Theorem 2. The triangle inequality gives

\le q\lVert Z-Z^\star \rVert_2 +(1+q)\lVert S-Z^\star \rVert_2.$$ Thus, a sufficient condition for feedback to improve proximity to $Z^\star$ is $$\lVert S-Z^\star \rVert_2 < \frac{1-q}{1+q}\lVert Z-Z^\star \rVert_2.$$ A biased or mismatched reference can violate this condition even when feedback contracts toward it. In Eq. (8), the correction is learned through the retrieval training objective. Its alignment with $Z^\star$ remains an empirical question. A direct diagnostic is to perturb or replace $S$ and track both the contraction ratio and the score margin between positive and negative retrieval candidates. **Evidence on the task objective.** Table 8 reports local attenuation, with $D_{\mathrm{before}}=27.1$ and $D_{\mathrm{after}}=24.4$. The retrieval evidence comes from held-out Recall@K. On MS COCO, Table 2 reports I2T R@1 of 62.5 with 50% of the training set, compared with 61.9 for the full-data β-CLIP baseline. At 90% of the training data, the reported means exceed the full-data baseline on all six retrieval metrics. The theory characterizes when the two operations can stabilize representations, and the data-fraction experiments quantify the observed retrieval benefit at the tested splits.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.