- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes HyWeCS, a method for improving adversarial robustness in multimodal fusion by combining two stochastic weight-generation mechanisms, ORaW, which integrates random weight perturbations with Layer-wise Orthogonal Training, and SHANW, which uses Euclidean and hyperbolic attention. The method also reuses stochastic weights for cross-modal interactions. Through experiments on eight multimodal benchmarks, the authors show improvements in certified and empirical robustness. The paper also reports reductions in parameter count and computational cost. The authors also provide theoretical robustness guarantees based on Lipschitz continuity and Randomized Smoothing.
Critical strengths
- The proposed method improves certified and empirical robustness across eight multimodal benchmarks.
- The method reduces parameters and computational cost compared with existing hybrid stochastic defenses.
Critical weaknesses
- The proposed method mainly combines various existing techniques. Although it seems empirically effective, the novelty and functional significance of this particular combination remain unclear.
- Whether the trained networks actually satisfy the assumptions underlying the theoretical analysis, including the required Lipschitz bounds, is unclear.
- Several design choices lack sufficient justification and validation. In particular, the contributions of self-modulated noise, hyperbolic attention, and reciprocal cross-modal interaction to robustness improvements remain unclear.
Questions for authors
- How are the Lipschitz bounds in Eq. (3) and Theorem 1 enforced or verified during training and inference? In particular, how is the input dependence of SHANW weights accounted for, and how is the non-expansiveness of reciprocal cross fusion ensured? Could the authors provide numerical evaluations or other evidence demonstrating that the implemented networks satisfy these assumptions?
- What motivates the self-modulated noise in Eq. (17), and what advantages does it offer over conventional Gaussian noise?
- Why is hyperbolic attention necessary in SHANW? What theoretical or empirical advantages does it offer over Euclidean attention alone?
- Does the reciprocal cross-modal interaction in Eq. (6) improve robustness beyond reducing computational cost? How does it compare with conventional multimodal fusion using the same ORaW and SHANW mechanisms?
3 comments
I sincerely appreciate the authors' efforts to address my previous concerns. In particular, I appreciate the detailed additional proofs about the Lipschitz guarantees, including the input dependence of SHANW and the effects of cross-modal fusion, while, to be honest, I have not been able to examine all the proofs in detail within this limited review period.
My remaining concern is whether the provided theoretical guarantee is fully applicable to the actual implementation and whether the overall contribution is sufficiently significant. Therefore, while I do not rule out possibility of raising my rating, I prefer to maintain my original rating at this stage.
Dear Reviewer 1ygQ,
Thank you very much for your thoughtful follow-up and for acknowledging the additional theoretical analysis. We understand the limited time available for reviewing the proofs and are grateful for your continued consideration. We would like to clarify the two remaining concerns.
1. Applicability of the theoretical guarantee
The revised analysis follows the computations described in the method. At the layer level, the calibration in Equation (3) accounts for how SHANW weights change with the input in the convolution defined by Equation (4). At the fusion level, it accounts for both the branch that determines the reused weights and the paired representation to which those weights are applied. The complete residual and fusion contributions, including any increase in sensitivity during fusion, are included when calibrating the final prediction mapping (App. D.5, Corollary 3, pp. 31–32, Eqs. (65)–(69); App. E.1, Proposition 3, pp. 33–35, Eqs. (70)–(74); App. E.2, pp. 35–36).
The corresponding execution procedure is specified in App. F, p. 38, ll. 2020–2051, which describes the calibrated evaluation used by Algorithms 1–3. It explains how bounds are updated for the current learned model, how calibration is applied to each stochastic realization, and how the complete forward computation enters the final calibration. Training, inference, and certification settings are detailed in App. H.2, pp. 43–45.
The guarantee assumes that internal draws follow an input-independent distribution, that expected predictions are well defined, and that finite bounds on inputs and intermediate representations hold throughout the stated evaluation domain, including inputs perturbed by Gaussian noise. The derivations establish the guarantee under these conditions; confirming that the evaluated code satisfies them requires separate verification (Sec. 2.4, Theorem 1, p. 7; App. F, p. 38, ll. 2046–2051).
We also distinguish the expected-logit certificate supported by Theorem 1 from the probability-based randomized-smoothing certificate supported by the Gaussian-shift argument and finite-sample probability estimation (Corollary 1, Eqs. (16)–(17); App. B.4, Lemma 3; App. H.2, p. 44, ll. 2349–2355).
2. Significance of the contribution
Distinct methodological construction. HyWeCS coordinates ORaW and SHANW at complementary, non-overlapping weight locations within each filter and reuses the terminal stochastic weights for reciprocal cross-modal interaction. This combines projection-driven diversity with input-conditioned Euclidean and hyperbolic attention noise. The comparison with CTRW, HyCAS, HySCAN, and LOT explains the distinctions motivating this organization (Sec. 2.3; App. A.5, Table 7).
Evidence for the coordinated design. On RAVDESS and UCF101, combining the two mechanisms improves RS-certified accuracy by 7.0 to 12.4 percentage points and EOTPGD-20 accuracy by 8.4 to 13.3 points over either mechanism alone. Comparisons with stochastic replacements additionally show gains of 3.1 to 6.4 points for ORaW and 1.3 to 4.7 points for SHANW (Apps. J.1–J.2, Tables 10–11).
Additional contribution of reciprocal fusion. With the encoders and both stochastic mechanisms retained, reciprocal fusion improves UCF101 accuracy over conventional late and intermediate fusion by 4.4 to 5.9 points under PGD-20, 4.6 to 5.3 points under EOTPGD-20, and 4.1 to 4.6 points for RS certification. Compared with same-branch weight reuse, the corresponding gains are 1.6, 1.8, and 1.9 points. We also report the accompanying clean-accuracy reductions of 1.7 to 2.4 points relative to conventional fusion and 0.8 points relative to same-branch reuse (App. J.5, Table 14).
Practical benefit. Across four natural-domain benchmarks, HyWeCS reduces training time by 29.6 to 50.5 percent, inference latency by 30.2 to 56.3 percent, memory consumption by 37.3 to 48.9 percent, and adversarial-evaluation time by 38.6 to 56.0 percent, relative to HyCAS-M and HySCAN-M (App. I.5, Table 9).
We respectfully believe these controlled comparisons support the significance of the coordinated design and its practical benefits, while the theoretical result should be assessed within its stated scope.
As the author discussion approaches its close, we would be grateful for any specific indication of an assumption, computational step, or comparison that would benefit from further clarification. We would be glad to respond promptly and carefully. We respectfully ask that these clarifications and the cited evidence be considered in your final assessment.
Thank you for your time and consideration.
With kind regards,
The Authors
Dear Reviewer 1ygQ,
We would like to sincerely thank you once again for the very careful reading of our manuscript and for your thoughtful, detailed, and constructive evaluation. As discussion closes, we respectfully invite feedback on the evidence below. References concern the revised manuscript; gains are percentage points. Tables 12–14 use UCF101.
W1 — Prior-defense distinctions. We explain the complementary limitations of CTRW, HyCAS, HySCAN, and LOT (Apps. A.2–A.3, A.5, pp. 16–19; Fig. 1; Table 7, p. 18).
W1 — Within-filter construction. We detail r/(n−r) allocation, ORaW’s projection resampling and LOT retraction, SHANW’s feature-conditioned attention noise, and reciprocal terminal-weight reuse (Secs. 2.2–2.3.3, pp. 4–7; App. B.2, p. 20; Sec. 3.1, p. 9, ll. 474–485).
W1 — Component and replacement evidence. Four configurations test individual/joint contributions; a separate ablation isolates random projection and LOT. ORaW is compared with RPF, CTRW, and SN-RPF, and SHANW with RANI-W and RWAN. On RAVDESS/UCF101, joint gains over either component alone are 7.0–12.4 certified / 8.4–13.3 EOTPGD-20 points. Replacement gains are 3.1–6.4 for ORaW and 1.3–4.7 points for SHANW (Tables 10–11; Apps. J.1–J.2, pp. 49–51).
W2/Q1 — SHANW’s input dependence. Lemma 6 bounds the full attention stack; Proposition 2 retains input-dependent attention and geometry-selection coefficients for a fixed draw (Apps. D.2–D.5; Lemma 6 and Proposition 2, pp. 28–30, Eqs. (60)–(63)).
W2/Q1 — Explicit layer calibration. Corollary 3 derives S + M K_SHANW, including the input-dependent term, and the coefficient-to-convolution conversion. Calibration gives Lip ≤ τ ≤ 1 on the stated input domain (Sec. 2.3, Eqs. (3)–(4); App. D.5, Corollary 3, pp. 31–32, Eqs. (64)–(69)).
W2/Q1 — Residual–fusion sensitivity. Proposition 3 retains the source branch’s weight-conditioning pathway and recipient-feature variation. Its 2 × 2 sensitivity matrix bounds the complete stage without assuming automatic non-expansiveness (App. E.1, pp. 33–35, Eqs. (70)–(74)).
W2/Q1 — End-to-end guarantee. Theorem 1 composes complete stage bounds and applies global calibration. Under the stated assumptions and uniform representation bounds, it establishes a 2-Lipschitz prediction mapping and expectation (Sec. 2.3.3; Sec. 2.4, Theorem 1, p. 7, Eq. (15); App. E.2, pp. 35–36).
W2/Q1 — Training, inference, and certification. We detail layer/global calibration, resampling, attack-phase and parameter-update draws, and EOT gradient averaging (App. D.4, p. 27, ll. 1439–1445; App. F, pp. 38–39; App. H.2, pp. 43–45).
W3/Q2 — Noise construction and analysis. Eqs. (6)–(7) distinguish noise construction from its integration into SHANW weights. Lemma 5 gives mean π, variance 3π², and second moment 4π²; Theorem 3 bounds the recursive weight update (Sec. 2.3.2; App. D.4, pp. 26–27; App. D.5, Lemma 5/Theorem 3, p. 28).
W3/Q2 — Isolated Gaussian comparison. HyWeCS-G changes only the noise rule. Self-modulation gains 0.5 / 1.5 / 1.6 / 2.9 points in clean / PGD-20 / EOTPGD-20 / RS-certified accuracy (Table 12; App. J.3, pp. 51–52).
W3/Q3 — Geometry and motivation. We explain Lorentz-geodesic relations to learned prototypes, complementary Euclidean local/channel responses, and their alignment with SHANW weights. We cite Nickel and Kiela (2018) and Park et al. (2023) (App. B.3, pp. 20–21; Apps. D.2–D.4, pp. 24–27).
W3/Q3 — Geometry ablation. Hybrid attention gains 0.9 / 1.2 / 1.6 / 1.7 points over Euclidean-only and 0.4 / 0.5 / 0.8 / 0.9 points over hyperbolic-only in clean / PGD-20 / EOTPGD-20 / RS-certified accuracy (Table 13; App. J.4, p. 52).
W3/Q4 — Reciprocal interaction. We explain fusion vulnerability and how each branch’s terminal stochastic weights transform the paired modality’s representation (Sec. 2.1; Sec. 2.3.3, Eq. (9), p. 6; App. A.1, p. 16; App. J.5, pp. 52–54).
W3/Q4 — Controlled fusion comparisons. With ORaW–SHANW retained, reciprocal fusion gains 4.4–5.9 / 4.6–5.3 / 4.1–4.6 points over conventional late/intermediate fusion and 1.6 / 1.8 / 1.9 points over same-branch reuse in PGD-20 / EOTPGD-20 / RS-certified accuracy. Clean-accuracy reductions are 1.7–2.4 and 0.8 points, respectively (Table 14; App. J.5, pp. 53–54).
Multimodal and efficiency evidence. We also report modality-specific attacks, robustness–complexity comparisons, and training, inference, memory, and adversarial-evaluation costs (Table 8; Figs. 7–9; Table 9; Apps. I.2–I.3 and I.5, pp. 45–48).
We would be very grateful if you could identify any remaining concerns and the specific argument, assumption, derivation, comparison, or result requiring clarification.
Thank you for your time and consideration.
With kind regards,
The Authors