acceptodds
Under review as a conference paper at ICLR 2027

Mutual Equilibrium: Multimodal Representation Learning through Reciprocal Feedback

Abstract

This work proposes a mutual feedback architecture, MEQ, that refines the two inputs, of possibly different modalities, into a pair of coupled embeddings such that each embedding reflects the information of the other. The core idea is to incorporate continuous interchange of information between the two inputs. This idea leads to a mutual feedback architecture consisting of two components whose outputs are fed back into the other. The final output of this model is defined as the fixed point of this interaction. We provide theoretical analysis that offers interpretation of this model as well as design choices to prevent failure cases. We show the benefits of MEQ through classification and visual grounding tasks spanning various datasets. Quantitatively, our model outperforms or shows competitive performance on concatenation-based multimodal classification problems. Qualitatively, the proposed interactive mechanism allows the model to progressively refine the visual grounding when paired with complementary modality, thus demonstrating the power of mutual feedback.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
equilibrium, feedback, representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GNUMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes MEQ, a mutual feedback architecture in which two modality-specific networks repeatedly exchange their hidden states, the final representation being defined as the fixed point of the resulting coupled system. Building on tools from Deep Equilibrium Models (DEQ), the authors analyze the sensitivity of the equilibrium and a failure mode they call fixed point collapse. This analysis motivates an architectural template and two regularizers: a range penalty on the Jacobian norm and a fixed point residual penalty. Experiments on five multimodal benchmarks with frozen pretrained encoders (CLIP and BERT for the vision-language tasks) compare MEQ with concatenation and parameter-matched attention baselines, and are complemented by a visual grounding study on COD10K.

Critical strengths
  • The coupled formulation is clean, and Lemma 1 gives an interpretable decomposition of the total influence into self and cross-modal paths, even if it is a direct extension of the DEQ derivation of Bai et al. The cross-modal sensitivity of Eq. 5 is also useful in practice: on VQA v2, it shows that bounding the gate does not restore the coupling (Table 10).

• MEQ clearly improves over concatenation (e.g., +14.36 pp on VQA v2 and +6.57 pp on VCR), and matches or slightly outperforms parameter-matched attention baselines on SNLI-VE, VCR, VQA v2 and COD10K (Tables 1 and 8).

• The paper is transparent about its limitations, it uses paired 2σ criteria in the ablations, includes single-modality controls (which expose the text-only artifact of SNLI-VE and the weak fusion signal of CMU-MOSEI), examines the fixed-point behavior far beyond the read depth, and reports several negative results as such (e.g., the residual penalty backfiring on VQA v2).

Critical weaknesses
  • The main issue, from my point of view, is that the implemented model is not an equilibrium model. The representation is read at z(K)\mathbf{z}^{(K)} after K=10K=10 damped steps, at a nonzero residual, gradients are obtained by backpropagating through the unrolled steps, and the undamped map is expansive on four of the five benchmarks (Table 12). Moreover, iterating further towards the fixed point strongly degrades performance (from 62.8 to 27.4 on VQA v2 and from 75.0 to 53.2 on SNLI-VE at k=300k=300). In this regime, MEQ is effectively a weight-tied 10-step recurrent network: Lemma 1 and the DEQ guarantees invoked in Section 2 do not apply, and the distinction drawn with MulT reduces to weight sharing. The paper acknowledges these facts (Sec. 2 and App. C.1), but still frames the model and its analysis in terms of fixed points.

  • The collapse argument used to justify the lower bound ρℓ\rho_\ell (Sec. 2.1, Eq. 7) appears to be incorrect. Writing the total influence as (I−A)−1(I-A)^{-1} with A=Fzx+Fzy(I−Gzy)−1GzxA = F_{z_x} + F_{z_y}(I-G_{z_y})^{-1}G_{z_x}, one has 1≤∣I−A∣,∣(I−A)−1∣≤(1+∣A∣),∣(I−A)−1∣1 \le |I-A|,|(I-A)^{-1}| \le (1+|A|),|(I-A)^{-1}|, hence ∣(I−A)−1∣≥1/(1+∣A∣)>1/2|(I-A)^{-1}| \ge 1/(1+|A|) > 1/2 whenever ∣A∣<1|A| < 1. The total influence therefore cannot become small in the contractive regime the paper targets. As JT→0J_T \to 0, it tends to II, and Lemma 1 gives dz∗x/dx→∂F/∂xd\mathbf{z}^*x/d\mathbf{x} \to \partial F/\partial\mathbf{x} : the equilibrium reduces to a map that ignores the other modality. This is undesirable for the purposes of the paper, but it is not a collapse, which requires ∂F/∂x\partial F/\partial\mathbf{x} itself to vanish. In addition, the trained models lie outside the band [ρℓ,ρh]=[0.7,0.9][\rho\ell, \rho_h] = [0.7, 0.9], with spectral radii between 0.965 and 1.689 at the read depth (Table 12). Finally, Lfpc\mathcal{L}_{\mathrm{fpc}}, designed to keep the truncated iterate self-consistent, is trivially minimized by the identity map T(u,z)=zT(\mathbf{u},\mathbf{z}) = \mathbf{z}, for which every state is a fixed point and no information is exchanged. This is exactly what happened on VQA v2 (App. C.3), where the result reported in Table 1 is only obtained by setting λf=0\lambda_f = 0 (54.29 vs.\ 63.33, Table 9).

  • The gains over attention are small (+0.74+0.74 to +0.94+0.94,pp, with no dispersion reported in Table 1), MEQ trails attention by 2.31,pp on Hateful Memes, and it costs 2.13×2.13\times the wall-clock time of the fastest baseline (about 1.81.8-1.9×1.9\times that of the attention baselines, Table 13). On VQA v2, a single-iteration MEQ already matches the best attention baseline (62.39 in Tables 1 and 2) at essentially the same cost, so the advantage only appears at iterate. Since the baselines are matched in parameters but not in compute, the experiments cannot tell whether this gain comes from the mutual-equilibrium formulation or simply from additional depth. A compute-matched baseline (e.g., a stack of cross-attention layers with comparable FLOPs) is needed to attribute it to the proposed mechanism. Similarly, Figs. 3 and 4 read a model trained for K=10K=10 at shallower depths, which does not measure the value of iteration, this would require models trained at each depth.

Questions for authors

Most of my questions follow from the weaknesses above.

  1. What exactly is the “soft gate”? It is ablated in Table 2 and discussed in Sec. 3.3 and App. C.3, but it does not appear in Section 2. Please give the exact update that is implemented.
  2. Can you report a compute-matched baseline (e.g., a stack of cross-attention layers with FLOPs comparable to MEQ at K = 10)?
  3. Can you report the dispersion across seeds in Table 1?
  4. Does Ljac act on the damped or on the undamped map? How do you reconcile the band [0.7, 0.9] with the spectral radii of Table 12?
Reviewer_gACmOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Mutual Equilibrium (MEQ), a framework for multimodal representation learning based on iterative reciprocal feedback between modality-specific representations. Instead of combining features through a single fusion operation, MEQ employs two coupled neural modules that repeatedly update their representations using information from both modalities, with the interaction formally characterized through a fixed-point formulation. A theoretical analysis is provided of the coupled dynamics using Jacobian-based sensitivity and stability arguments, which motivate the architectural design and the introduction of regularization terms for controlling the Jacobian norm and the fixed-point residual.

The approach is evaluated on several multimodal classification benchmarks with comparisons against concatenation-based and attention-based fusion models. Additional experiments investigate iterative prediction refinement, visual grounding on camouflaged objects, architectural components, and the behavior of the proposed fixed-point regularization mechanisms.

Critical strengths

The paper presents a useful empirical investigation of iterative multimodal dynamics, going beyond final task performance to examine prediction refinement, cross-modal sensitivity, fixed-point residuals, and degenerate behaviors. In particular, the analysis of failure cases and the transparent reporting of negative results provide valuable insights into the practical challenges of recurrent multimodal architectures. These investigations represent a positive aspect of the work, even though their implications for the central equilibrium hypothesis remain insufficiently established.

Critical weaknesses

The paper presents several weaknesses that substantially affect the assessment of its scientific contribution and significance. The two most critical weaknesses concern the technical novelty and the experimental validation.

Limited theoretical novelty and insufficient justification. The proposed coupled dynamical system is mathematically a structured instance of an existing deep equilibrium formulation, and the theoretical analysis largely builds on established implicit differentiation and Jacobian-based sensitivity arguments. While these derivations characterize local interactions between the two states, they do not establish why the resulting equilibrium should produce more informative, semantically meaningful, or better-aligned multimodal representations. Moreover, the connection between the theoretical analysis and the architectural choices in Eq. (6) remains insufficiently justified, as the proposed operators largely combine established self-attention, cross-attention, and input-injection mechanisms. The discrepancy between the fixed-point formulation and the finite-depth unrolled implementation further limits the relevance of the theoretical results to the actual model. Consequently, the paper does not convincingly establish the distinctive theoretical contribution or necessity of the proposed mutual equilibrium framework.

Insufficient experimental validation. The experiments provide some evidence that repeated cross-modal interactions can improve predictive performance, but they do not establish that equilibrium itself is responsible for these improvements. A controlled comparison against a finite-depth recurrent multimodal architecture with comparable computational cost and without equilibrium-specific constraints is particularly important. Furthermore, the only useful analysis in that sense is reported in Appendix C.3 and shows that continuing the iterations beyond the selected read depth degrades performance across all benchmarks, raising questions about the desirability of the equilibrium state. The fixed-point residual penalty also induces a degenerate identity solution on VQA v2 and must be disabled, while additional experiments show that restoring cross-modal sensitivity does not necessarily improve predictive performance. Overall, the experimental evidence does not substantiate the proposed connection between dynamical convergence, cross-modal sensitivity, and improved semantic representations. These limitations directly affect the central hypothesis of the paper rather than merely its empirical competitiveness.

Additional concerns. The related work insufficiently addresses existing iterative cross-modal interaction methods, while the distinction from multimodal DEQ approaches remains unclear. The performance gains over attention-based baselines are generally modest relative to the additional computational cost. The visual grounding experiments provide limited evidence of improved semantic representations, particularly since the textual prompts contain ground-truth object labels. Furthermore, several findings essential to assessing the equilibrium hypothesis are relegated to the appendix, and the non-standard evaluation protocols used for VQA v2 and VCR limit the comparability and generalizability of the results.

Questions for authors

The following questions concern fundamental aspects of the proposed theoretical framework and its experimental validation that remain unclear after careful reading.

Q1. Theoretical contribution and representation significance. Could the authors clarify what distinguishes the proposed mutual equilibrium formulation from existing multimodal DEQ approaches? In particular, how does the Jacobian-based analysis support the claimed benefits in terms of meaningful cross-modal interactions and improved semantic representations?

Q2. Experimental evidence. How do the authors reconcile the claimed benefits of equilibrium with the finite-depth implementation and the performance degradation observed under prolonged iteration (Table 12)? Furthermore, how should the identity collapse induced by the residual penalty on VQA v2 and the lack of performance improvements when restoring cross-modal sensitivity be interpreted within the proposed theoretical framework?

Reviewer_S1SjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Mutual Equilibrium (MEQ), an architecture that iteratively refines multimodal representations through two coupled update modules. Each module incorporates its own hidden state, the other modality’s hidden state, and continuously injected input features. The joint fixed point is defined as the theoretical output representation. The authors analyze input sensitivity and cross-modal influence using block Jacobians and introduce a Jacobian norm range penalty and a fixed-point residual penalty. In practice, the model performs a fixed number of 10 damped iterations and is trained through unrolled backpropagation. Experiments cover five multimodal prediction benchmarks, together with qualitative visualizations and a constructed classification task on COD10K. MEQ outperforms simple concatenation on four benchmarks and slightly outperforms parameter-matched attention baselines on three. Ablations support the usefulness of cross-modal pathways and finite-step updates on some tasks. However, additional analyses show that residual regularization can induce degenerate behavior and that further iteration reduces task performance. The evidence therefore more directly supports the effectiveness of finite-step interaction than the benefit of reaching equilibrium itself.

Critical strengths
  1. MEQ maintains separate states for different modalities while allowing each to repeatedly access information from the other. In Table 2, the full model achieves 63.11 accuracy on VQA v2, compared with 58.90 after removing cross-attention and 62.39 for the single-iteration variant. These results support the usefulness of cross-modal pathways and repeated updates in this setting. The authors also compare alternative state layouts and mixing modules to investigate the sources of improvement, lending some credibility to the architectural contribution.
  2. The authors transparently report performance degradation under prolonged iteration, identity-map degeneration induced by residual regularization, and the inability of gate lower bounds and cross-modal sensitivity constraints to simultaneously restore interaction and accuracy. These diagnostics expose the limitations of evaluating effective fusion through residuals or sensitivity alone and constitute an informative part of the paper.
Critical weaknesses
  1. The implemented model is trained through a fixed unrolling of 10 iterations rather than through implicit differentiation at a solved fixed point. The results in Table 12 show that performance decreases on every benchmark when iteration continues to 300 steps. The concern is that the paper places fixed points at the center of its motivation without establishing a clear connection between the equilibrium objective and the observed benefits. Without contractivity or another suitable error bound, a small residual is also insufficient to establish that z(10)z^{(10)} is close to a task-relevant fixed point.
  2. MEQ improves over the stronger attention baseline by 0.74, 0.85, and 0.94 percentage points on SNLI-VE, VCR, and VQA v2, respectively, while showing no advantage on the other two benchmarks. Parameter matching does not eliminate the effect of additional computation. Direct comparisons are still needed against ordinary multi-step attention under matched computational budgets and against the same architecture without equilibrium-related regularization.
  3. “Best attn.” reports only the better result from self-attention and cross-attention, without clearly identifying the selected attention network architecture and its full configuration in different dataset. Self-attention over concatenated modalities already allows cross-modal interaction. Consequently, the experiments need to distinguish gains from the presence of interaction, repeated feedback, and differences in module configuration or computation.
Questions for authors
  1. Table 12 shows that continuing from the training readout depth to 300 iterations reduces performance on every benchmark. Do the authors regard the method’s central contribution as the final fixed point or the trained finite-step trajectory? Under the same architecture, iteration count, and training budget, what evidence separates the benefits of equilibrium-related regularization from those of an ordinary weight-tied recurrent network?
  2. Could the authors report stability measures for the actual damped updates, the fraction of individual examples that converge under an explicit convergence criterion, and sensitivity to initialization and inference readout depth? Beyond decreasing residuals, what evidence establishes that z(10)z^{(10)} is close to a task-relevant fixed point rather than merely an intermediate state that is useful for prediction?
  3. Please report the self-attention and cross-attention results separately and identify which corresponds to “Best attn.” on each dataset. Does MEQ retain its advantage under matched computational budgets, thereby ruling out differences in module configuration and additional computation as explanations for the gains?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.