- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes MEQ, a mutual feedback architecture in which two modality-specific networks repeatedly exchange their hidden states, the final representation being defined as the fixed point of the resulting coupled system. Building on tools from Deep Equilibrium Models (DEQ), the authors analyze the sensitivity of the equilibrium and a failure mode they call fixed point collapse. This analysis motivates an architectural template and two regularizers: a range penalty on the Jacobian norm and a fixed point residual penalty. Experiments on five multimodal benchmarks with frozen pretrained encoders (CLIP and BERT for the vision-language tasks) compare MEQ with concatenation and parameter-matched attention baselines, and are complemented by a visual grounding study on COD10K.
Critical strengths
- The coupled formulation is clean, and Lemma 1 gives an interpretable decomposition of the total influence into self and cross-modal paths, even if it is a direct extension of the DEQ derivation of Bai et al. The cross-modal sensitivity of Eq. 5 is also useful in practice: on VQA v2, it shows that bounding the gate does not restore the coupling (Table 10).
• MEQ clearly improves over concatenation (e.g., +14.36 pp on VQA v2 and +6.57 pp on VCR), and matches or slightly outperforms parameter-matched attention baselines on SNLI-VE, VCR, VQA v2 and COD10K (Tables 1 and 8).
• The paper is transparent about its limitations, it uses paired 2σ criteria in the ablations, includes single-modality controls (which expose the text-only artifact of SNLI-VE and the weak fusion signal of CMU-MOSEI), examines the fixed-point behavior far beyond the read depth, and reports several negative results as such (e.g., the residual penalty backfiring on VQA v2).
Critical weaknesses
-
The main issue, from my point of view, is that the implemented model is not an equilibrium model. The representation is read at after damped steps, at a nonzero residual, gradients are obtained by backpropagating through the unrolled steps, and the undamped map is expansive on four of the five benchmarks (Table 12). Moreover, iterating further towards the fixed point strongly degrades performance (from 62.8 to 27.4 on VQA v2 and from 75.0 to 53.2 on SNLI-VE at ). In this regime, MEQ is effectively a weight-tied 10-step recurrent network: Lemma 1 and the DEQ guarantees invoked in Section 2 do not apply, and the distinction drawn with MulT reduces to weight sharing. The paper acknowledges these facts (Sec. 2 and App. C.1), but still frames the model and its analysis in terms of fixed points.
-
The collapse argument used to justify the lower bound (Sec. 2.1, Eq. 7) appears to be incorrect. Writing the total influence as with , one has , hence whenever . The total influence therefore cannot become small in the contractive regime the paper targets. As , it tends to , and Lemma 1 gives : the equilibrium reduces to a map that ignores the other modality. This is undesirable for the purposes of the paper, but it is not a collapse, which requires itself to vanish. In addition, the trained models lie outside the band , with spectral radii between 0.965 and 1.689 at the read depth (Table 12). Finally, , designed to keep the truncated iterate self-consistent, is trivially minimized by the identity map , for which every state is a fixed point and no information is exchanged. This is exactly what happened on VQA v2 (App. C.3), where the result reported in Table 1 is only obtained by setting (54.29 vs.\ 63.33, Table 9).
-
The gains over attention are small ( to ,pp, with no dispersion reported in Table 1), MEQ trails attention by 2.31,pp on Hateful Memes, and it costs the wall-clock time of the fastest baseline (about - that of the attention baselines, Table 13). On VQA v2, a single-iteration MEQ already matches the best attention baseline (62.39 in Tables 1 and 2) at essentially the same cost, so the advantage only appears at iterate. Since the baselines are matched in parameters but not in compute, the experiments cannot tell whether this gain comes from the mutual-equilibrium formulation or simply from additional depth. A compute-matched baseline (e.g., a stack of cross-attention layers with comparable FLOPs) is needed to attribute it to the proposed mechanism. Similarly, Figs. 3 and 4 read a model trained for at shallower depths, which does not measure the value of iteration, this would require models trained at each depth.
Questions for authors
Most of my questions follow from the weaknesses above.
- What exactly is the “soft gate”? It is ablated in Table 2 and discussed in Sec. 3.3 and App. C.3, but it does not appear in Section 2. Please give the exact update that is implemented.
- Can you report a compute-matched baseline (e.g., a stack of cross-attention layers with FLOPs comparable to MEQ at K = 10)?
- Can you report the dispersion across seeds in Table 1?
- Does Ljac act on the damped or on the undamped map? How do you reconcile the band [0.7, 0.9] with the spectral radii of Table 12?