- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Structured Predictive Distributions (SPD) for improving generalization and robustness in multimodal regression. Instead of conventional point-wise predictions, SPD introduces Normal-Inverse-Gamma (NIG) predictive representations with distribution-aware reweighting. The paper further provides a Rademacher complexity-based theoretical analysis to establish conditional generalization-bound tightening through NIG regularization. Experimental results across multiple multimodal regression tasks demonstrate improvements under both clean and noisy conditions.
Critical strengths
-
The paper presents an interesting approach to multimodal regression by introducing structured predictive representations. The combination of NIG-based predictive modeling, multimodal fusion, and distribution-aware reweighting provides a reasonable framework for handling noisy multimodal inputs.
-
The experimental evaluations cover multiple regression tasks, including image fusion, crowd counting, and depth estimation. The consistent improvements across different noise conditions provide strong empirical support for the effectiveness and general applicability of the proposed method.
Critical weaknesses
-
The theoretical results rely on several assumptions whose applicability to the actual training procedure remains unclear. For example, the analysis requires an explicit NIG regularization budget, bounded predictive variance, and a uniformly positive definite feature Gram matrix. However, the actual training procedure uses soft regularization, which does not necessarily guarantee these conditions. Therefore, it is unclear whether the theoretical guarantees directly apply to the learned models, especially when using complex nonlinear feature extractors.
-
Moreover, the theoretical contribution seems somewhat limited in explaining the specific generalization benefits of NIG-based predictive representations. The main result derives a parameter norm constraint from NIG regularization and then uses this constraint to establish a reduction in Rademacher complexity. However, the resulting bound tightening appears to follow primarily from conventional hypothesis-class restriction, rather than a property unique to NIG representations. This makes it difficult to distinguish the theoretical benefits of SPD from those of standard norm-based regularization. Furthermore, the analysis does not directly establish why the proposed method should be more robust under noisy multimodal conditions.
<I may raise the score if these concerns are resolved.>
Questions for authors
-
The theoretical analysis assumes an explicit NIG regularization budget and a uniformly positive definite feature Gram matrix. How are these conditions justified for the actual training procedure, which uses soft regularization and nonlinear feature extractors? Clarifying the connection between the theoretical assumptions and the implemented models would help establish the practical relevance of the guarantees.
-
The main theoretical result connects NIG regularization to a smaller parameter norm and consequently a reduced Rademacher complexity. What distinguishes this result from conventional norm-based capacity control? More specifically, does the NIG formulation provide any theoretical advantage that would not follow from applying a comparable norm constraint directly to the regression head?
-
From my perspective, there are some concerns about whether Rademacher complexity-based generalization analysis [1][2] can adequately characterize the behavior of modern overparameterized nonlinear models. Although the proposed bounds are mathematically valid under assumptions, it remains unclear how much of the actual generalization behavior of the learned nonlinear representations is captured by this hypothesis-class-based analysis. How should the theoretical bound tightening be interpreted in this context, especially given the known limitations of uniform convergence-based generalization analysis in deep learning?
<I may raise the score if these concerns are resolved.>
[1] Nagarajan, Vaishnavh, and J. Zico Kolter. "Uniform convergence may be unable to explain generalization in deep learning." NeurIPS'19
[2] Gastpar, Michael, et al. "Fantastic generalization measures are nowhere to be found." ICLR'24
4 comments
3. Interpretation for overparameterized models. Nagarajan and Kolter [1] and Gastpar et al. [2] identify limits of uniform convergence explanations. We interpret Corollary 1 as a relative comparison of two bounds for the same predictor. Its empirical risk stays fixed while the admissible parameter class changes. The audit isolates the capacity contribution of the location heads, with the feature extractor kept fixed.
Using valid pixel MSE in raw normalized log depth per image and equal averaging over 999 test images, the risk and radius class complexities were:
| Input | NIG empirical risk | C_r | C_R | Radius r complexity contribution | Radius R complexity contribution | Complexity contribution difference |
|---|---|---|---|---|---|---|
| Clean | 0.0126 | 46.77 | 51.34 | 3.257e+7 | 3.575e+7 | -3.181e+6 |
| Image Gaussian, σ = 30 | 0.0854 | 46.76 | 51.33 | 3.256e+7 | 3.574e+7 | -3.181e+6 |
| Event impulse, r = 0.25 | 0.0132 | 46.70 | 51.26 | 3.252e+7 | 3.569e+7 | -3.177e+6 |
Complexity estimates use 128 sign samples and 64 output coordinates with valid pixel weights. The radius ratio is approximately 0.911, representing an 8.90% reduction. Complexity contributions are and . This comparison quantifies the relative capacity reduction for the same predictor.
4. Connection to noise robustness. For the raw normalized log depth output, the parameter bound gives
\leq\lVert g_\theta(\widetilde x)-g_\theta(x)\rVert_2 +r_{\mathrm{ch}}\lVert T_\theta(\widetilde x)-T_\theta(x)\rVert_{\mathrm{op}}.$$ This identifies how the location constraint limits amplification of a given change in the prediction pathway. We assessed its empirical relevance with DAR disabled in the original counting ablation: | Benchmark | NIG regularizer | Clean RMSE ↓ | Gaussian RMSE ↓ | Increase from clean ↓ | |---|---|---:|---:|---:| | RGBT-CC | Off | 19.76 | 54.28 | 34.52 | | RGBT-CC | On | **18.74** | **49.61** | **30.87** | | DroneRGBT | Off | 10.24 | 28.44 | 18.20 | | DroneRGBT | On | **9.41** | **27.22** | **17.81** | NIG regularization reduces both Gaussian RMSE and the increase from clean inputs on both datasets. [1] Nagarajan, V., and Kolter, J. Z. Uniform convergence may be unable to explain generalization in deep learning. NeurIPS, 2019. [2] Gastpar, M., et al. Fantastic generalization measures are nowhere to be found. ICLR, 2024.Dear Authors,
Thanks for the responses. I really appreciate the work and additional efforts during the rebuttal period, which have largely resolved the concerns I raised. Thus, I will increase my score to weak accept
P.S. Just one more thing: I think there are some similar concepts that approach multimodal learning from a distributional perspective. So, from my perspective, it would be good to include some additional citations, such as:
- Xue, Yihao, et al. "Understanding the robustness of multi-modal contrastive learning to distribution shift." ICLR'24
- Qian, Chengxuan, et al. "Decalign: Hierarchical cross-modal alignment for decoupled multimodal representation learning." ICLR'26
- Lee, Jae-Jun, and Sung Whan Yoon. "Understanding Multimodal Learning: A Loss Landscape Smoothness Perspective." ICML'26
Dear reviewer Uu66,
Thank you for your thoughtful feedback and for acknowledging our additional efforts. We are glad that our responses have largely resolved your concerns.
We also appreciate the suggested references. We will cite all three papers in the revised manuscript, better positioning our work within the broader multimodal learning literature.
Response for Uu66
Thank you for your discretion. Our clarifications are listed below.
1. Connection between soft regularization and the analyzed constraint. We measured the fitted discrepancy and feature geometry on 1,998 reference images. All 384 channelwise weighted Grams were positive definite; the minimum unweighted patch eigenvalue was 7.96×10−5. The measured total, auxiliary, and location contributions were 0.01324, 0.01265, and 5.89×10−4.
We evaluate the complete output pathway across its location heads. The maximum observed aleatoric variance was 6557.43. The measurements are summarized below.
The architecture uses nonlinear feature extractors, with location predictions being linear in the learned features. Location biases are included through augmented features. Keeping inverse variance inside each Gram evaluates the location penalty directly.
With υ0=0.1 and Gh,j=EQ,position[ϕhϕh⊤/Vh,j], each bias-augmented location vector wh,j satisfies
\geq\frac{\upsilon_0\lambda_{\min}(G_{h,j})}{2}\|w_{h,j}\|^2.$$ This gives an explicit norm bound $r_{\mathrm{ch}}=459.09$ for the measured channel budgets. Those budgets define a subclass of the aggregate-budget class. The analysis evaluates the complete raw normalized log depth output $g_\theta+T_\theta W$ with the remaining parameters $\theta$ fixed; $g_\theta$ includes the fixed output bias. For the specified reference budget $R=503.94$, the radius-class complexity ratio is 0.911. **2. Difference from direct norm control.** While a direct norm constraint applies standard norm based capacity control, our approach derives this constraint from regularizing the predictive distribution. Before the scalar radius relaxation, the location term $\upsilon_0\gamma(x)^2/(2V(x))$ induces an inverse variance weighted geometry. In contrast, a scalar L2 penalty relies on isotropic parameter geometry. We compared NIG with the best L2 candidate ($10^{-5}$) and the candidate closest in head norm ($10^{-6}$), under image Gaussian noise σ = 30 and event impulse noise r = 0.25. | Input | Regularizer | MAE <10 m | MAE <20 m | MAE <30 m | AbsRel | RMSE log | SI log | |---|---|---:|---:|---:|---:|---:|---:| | Clean | NIG | **0.2654** | **0.6505** | **1.3367** | **0.1948** | **0.3816** | 0.1210 | | Clean | Best L2 | 0.3286 | 0.6541 | 1.3528 | 0.1958 | 0.4044 | **0.1203** | | Clean | L2 closest in norm | 0.3081 | 0.7568 | 1.4922 | 0.2077 | 0.4297 | 0.1279 | | Gaussian | NIG | **1.1165** | **2.2128** | **3.9644** | **0.4666** | **1.0755** | **0.7761** | | Gaussian | Best L2 | 1.1763 | 2.3927 | 4.2380 | 0.4865 | 1.1787 | 0.8921 | | Gaussian | L2 closest in norm | 1.3551 | 2.5545 | 4.3712 | 0.5110 | 1.1675 | 0.9078 | | Impulse | NIG | **0.2727** | **0.6546** | **1.3555** | **0.1945** | **0.3892** | **0.1205** | | Impulse | Best L2 | 0.2914 | 0.6758 | 1.3820 | 0.1968 | 0.4099 | 0.1286 | The measured head norms were 24.23, 23.96, and 24.19, respectively. NIG achieves lower Gaussian error on all six metrics. For independently trained models, NIG also consistently yields lower mean pixel risk across all three conditions: | Input | NIG mean pixel risk ↓ | Best L2 mean pixel risk ↓ | L2 closest in norm mean pixel risk ↓ | |---|---:|---:|---:| | Clean | **0.0126** | 0.0140 | 0.0156 | | Image Gaussian, σ = 30 | **0.0854** | 0.1024 | 0.1004 | | Event impulse, r = 0.25 | **0.0132** | 0.0144 | 0.0159 |