acceptodds
Under review as a conference paper at ICLR 2027

Tightening Multimodal Regression Generalization Bounds via Structured Predictive Distributions

Abstract

Multimodal regression is increasingly being deployed in real-world applications, yet its generalization performance remains limited, particularly under noisy conditions. Existing methods predominantly rely on point-wise predictions, leaving the role of structured predictive representations in generalization-bound tightening underexplored. In this work, we propose Structured Predictive Distributions (SPD) for tightening generalization bounds in multimodal regression, a theoretically grounded framework that reformulates multimodal regression in a structured predictive distribution space. Concretely, SPD combines Normal-Inverse-Gamma predictive representations with regularization to structure continuous regression outputs and performs multimodal fusion within this structured space. We further introduce a distribution-aware reweighting mechanism that adapts optimization to the predictive representation to improve robustness under noisy conditions. Using a common squared error risk, our analysis establishes tighter generalization bounds through NIG-induced capacity control. We further characterize the empirical-risk condition for comparisons with a matched reference predictor, relating empirical fit to the reduction in the complexity term. Extensive experiments on multiple multimodal regression benchmarks, covering image fusion, crowd counting, and depth estimation, demonstrate that SPD consistently improves robustness under diverse noise conditions. Our code is available in the supplementary material and https://anonymous.4open.science/r/SPD-A356/.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Robust Multimodal Learning, Model Robustness, Evidential Learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Uu66OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Structured Predictive Distributions (SPD) for improving generalization and robustness in multimodal regression. Instead of conventional point-wise predictions, SPD introduces Normal-Inverse-Gamma (NIG) predictive representations with distribution-aware reweighting. The paper further provides a Rademacher complexity-based theoretical analysis to establish conditional generalization-bound tightening through NIG regularization. Experimental results across multiple multimodal regression tasks demonstrate improvements under both clean and noisy conditions.

Critical strengths
  • The paper presents an interesting approach to multimodal regression by introducing structured predictive representations. The combination of NIG-based predictive modeling, multimodal fusion, and distribution-aware reweighting provides a reasonable framework for handling noisy multimodal inputs.

  • The experimental evaluations cover multiple regression tasks, including image fusion, crowd counting, and depth estimation. The consistent improvements across different noise conditions provide strong empirical support for the effectiveness and general applicability of the proposed method.

Critical weaknesses
  • The theoretical results rely on several assumptions whose applicability to the actual training procedure remains unclear. For example, the analysis requires an explicit NIG regularization budget, bounded predictive variance, and a uniformly positive definite feature Gram matrix. However, the actual training procedure uses soft regularization, which does not necessarily guarantee these conditions. Therefore, it is unclear whether the theoretical guarantees directly apply to the learned models, especially when using complex nonlinear feature extractors.

  • Moreover, the theoretical contribution seems somewhat limited in explaining the specific generalization benefits of NIG-based predictive representations. The main result derives a parameter norm constraint from NIG regularization and then uses this constraint to establish a reduction in Rademacher complexity. However, the resulting bound tightening appears to follow primarily from conventional hypothesis-class restriction, rather than a property unique to NIG representations. This makes it difficult to distinguish the theoretical benefits of SPD from those of standard norm-based regularization. Furthermore, the analysis does not directly establish why the proposed method should be more robust under noisy multimodal conditions.

<I may raise the score if these concerns are resolved.>

Questions for authors
  • The theoretical analysis assumes an explicit NIG regularization budget and a uniformly positive definite feature Gram matrix. How are these conditions justified for the actual training procedure, which uses soft regularization and nonlinear feature extractors? Clarifying the connection between the theoretical assumptions and the implemented models would help establish the practical relevance of the guarantees.

  • The main theoretical result connects NIG regularization to a smaller parameter norm and consequently a reduced Rademacher complexity. What distinguishes this result from conventional norm-based capacity control? More specifically, does the NIG formulation provide any theoretical advantage that would not follow from applying a comparable norm constraint directly to the regression head?

  • From my perspective, there are some concerns about whether Rademacher complexity-based generalization analysis [1][2] can adequately characterize the behavior of modern overparameterized nonlinear models. Although the proposed bounds are mathematically valid under assumptions, it remains unclear how much of the actual generalization behavior of the learned nonlinear representations is captured by this hypothesis-class-based analysis. How should the theoretical bound tightening be interpreted in this context, especially given the known limitations of uniform convergence-based generalization analysis in deep learning?

<I may raise the score if these concerns are resolved.>

[1] Nagarajan, Vaishnavh, and J. Zico Kolter. "Uniform convergence may be unable to explain generalization in deep learning." NeurIPS'19

[2] Gastpar, Michael, et al. "Fantastic generalization measures are nowhere to be found." ICLR'24

4 comments

Response for Uu66

Thank you for your discretion. Our clarifications are listed below.

1. Connection between soft regularization and the analyzed constraint. We measured the fitted discrepancy and feature geometry on 1,998 reference images. All 384 channelwise weighted Grams were positive definite; the minimum unweighted patch eigenvalue was 7.96×10−57.96\times10^{-5}. The measured total, auxiliary, and location contributions were 0.01324, 0.01265, and 5.89×10−45.89\times10^{-4}.

We evaluate the complete output pathway across its location heads. The maximum observed aleatoric variance was 6557.43. The measurements are summarized below.

HeadOutput channels / augmented feature dimensionMinimum unweighted Gram eigenvalueMinimum channelwise weighted Gram eigenvalueMean discrepancy DMean auxiliary term AMean location contribution b
int_head13128 / 1292.27e-41.58e-50.01130.01084.81e-4
int_head1464 / 651.32e-42.23e-50.01720.01648.11e-4
int_head23128 / 1291.13e-41.13e-50.01100.01064.46e-4
int_head2464 / 657.96e-58.83e-60.01340.01286.18e-4

The architecture uses nonlinear feature extractors, with location predictions being linear in the learned features. Location biases are included through augmented features. Keeping inverse variance inside each Gram evaluates the location penalty directly.

With υ0=0.1\upsilon_0=0.1 and Gh,j=EQ,position[ϕhϕh⊤/Vh,j]G_{h,j}=\mathbb E_{Q,\mathrm{position}}[\phi_h\phi_h^\top/V_{h,j}], each bias-augmented location vector wh,jw_{h,j} satisfies

\geq\frac{\upsilon_0\lambda_{\min}(G_{h,j})}{2}\|w_{h,j}\|^2.$$ This gives an explicit norm bound $r_{\mathrm{ch}}=459.09$ for the measured channel budgets. Those budgets define a subclass of the aggregate-budget class. The analysis evaluates the complete raw normalized log depth output $g_\theta+T_\theta W$ with the remaining parameters $\theta$ fixed; $g_\theta$ includes the fixed output bias. For the specified reference budget $R=503.94$, the radius-class complexity ratio is 0.911. **2. Difference from direct norm control.** While a direct norm constraint applies standard norm based capacity control, our approach derives this constraint from regularizing the predictive distribution. Before the scalar radius relaxation, the location term $\upsilon_0\gamma(x)^2/(2V(x))$ induces an inverse variance weighted geometry. In contrast, a scalar L2 penalty relies on isotropic parameter geometry. We compared NIG with the best L2 candidate ($10^{-5}$) and the candidate closest in head norm ($10^{-6}$), under image Gaussian noise σ = 30 and event impulse noise r = 0.25. | Input | Regularizer | MAE <10 m | MAE <20 m | MAE <30 m | AbsRel | RMSE log | SI log | |---|---|---:|---:|---:|---:|---:|---:| | Clean | NIG | **0.2654** | **0.6505** | **1.3367** | **0.1948** | **0.3816** | 0.1210 | | Clean | Best L2 | 0.3286 | 0.6541 | 1.3528 | 0.1958 | 0.4044 | **0.1203** | | Clean | L2 closest in norm | 0.3081 | 0.7568 | 1.4922 | 0.2077 | 0.4297 | 0.1279 | | Gaussian | NIG | **1.1165** | **2.2128** | **3.9644** | **0.4666** | **1.0755** | **0.7761** | | Gaussian | Best L2 | 1.1763 | 2.3927 | 4.2380 | 0.4865 | 1.1787 | 0.8921 | | Gaussian | L2 closest in norm | 1.3551 | 2.5545 | 4.3712 | 0.5110 | 1.1675 | 0.9078 | | Impulse | NIG | **0.2727** | **0.6546** | **1.3555** | **0.1945** | **0.3892** | **0.1205** | | Impulse | Best L2 | 0.2914 | 0.6758 | 1.3820 | 0.1968 | 0.4099 | 0.1286 | The measured head norms were 24.23, 23.96, and 24.19, respectively. NIG achieves lower Gaussian error on all six metrics. For independently trained models, NIG also consistently yields lower mean pixel risk across all three conditions: | Input | NIG mean pixel risk ↓ | Best L2 mean pixel risk ↓ | L2 closest in norm mean pixel risk ↓ | |---|---:|---:|---:| | Clean | **0.0126** | 0.0140 | 0.0156 | | Image Gaussian, σ = 30 | **0.0854** | 0.1024 | 0.1004 | | Event impulse, r = 0.25 | **0.0132** | 0.0144 | 0.0159 |

3. Interpretation for overparameterized models. Nagarajan and Kolter [1] and Gastpar et al. [2] identify limits of uniform convergence explanations. We interpret Corollary 1 as a relative comparison of two bounds for the same predictor. Its empirical risk stays fixed while the admissible parameter class changes. The audit isolates the capacity contribution of the location heads, with the feature extractor kept fixed.

Using valid pixel MSE in raw normalized log depth per image and equal averaging over 999 test images, the risk and radius class complexities were:

InputNIG empirical riskC_rC_RRadius r complexity contributionRadius R complexity contributionComplexity contribution difference
Clean0.012646.7751.343.257e+73.575e+7-3.181e+6
Image Gaussian, σ = 300.085446.7651.333.256e+73.574e+7-3.181e+6
Event impulse, r = 0.250.013246.7051.263.252e+73.569e+7-3.177e+6

Complexity estimates use 128 sign samples and 64 output coordinates with valid pixel weights. The radius ratio is approximately 0.911, representing an 8.90% reduction. Complexity contributions are 2LCr2LC_r and 2LCR2LC_R. This comparison quantifies the relative capacity reduction for the same predictor.

4. Connection to noise robustness. For the raw normalized log depth output, the parameter bound gives

\leq\lVert g_\theta(\widetilde x)-g_\theta(x)\rVert_2 +r_{\mathrm{ch}}\lVert T_\theta(\widetilde x)-T_\theta(x)\rVert_{\mathrm{op}}.$$ This identifies how the location constraint limits amplification of a given change in the prediction pathway. We assessed its empirical relevance with DAR disabled in the original counting ablation: | Benchmark | NIG regularizer | Clean RMSE ↓ | Gaussian RMSE ↓ | Increase from clean ↓ | |---|---|---:|---:|---:| | RGBT-CC | Off | 19.76 | 54.28 | 34.52 | | RGBT-CC | On | **18.74** | **49.61** | **30.87** | | DroneRGBT | Off | 10.24 | 28.44 | 18.20 | | DroneRGBT | On | **9.41** | **27.22** | **17.81** | NIG regularization reduces both Gaussian RMSE and the increase from clean inputs on both datasets. [1] Nagarajan, V., and Kolter, J. Z. Uniform convergence may be unable to explain generalization in deep learning. NeurIPS, 2019. [2] Gastpar, M., et al. Fantastic generalization measures are nowhere to be found. ICLR, 2024.

Dear Authors,

Thanks for the responses. I really appreciate the work and additional efforts during the rebuttal period, which have largely resolved the concerns I raised. Thus, I will increase my score to weak accept

P.S. Just one more thing: I think there are some similar concepts that approach multimodal learning from a distributional perspective. So, from my perspective, it would be good to include some additional citations, such as:

  1. Xue, Yihao, et al. "Understanding the robustness of multi-modal contrastive learning to distribution shift." ICLR'24
  2. Qian, Chengxuan, et al. "Decalign: Hierarchical cross-modal alignment for decoupled multimodal representation learning." ICLR'26
  3. Lee, Jae-Jun, and Sung Whan Yoon. "Understanding Multimodal Learning: A Loss Landscape Smoothness Perspective." ICML'26

Dear reviewer Uu66,

Thank you for your thoughtful feedback and for acknowledging our additional efforts. We are glad that our responses have largely resolved your concerns.

We also appreciate the suggested references. We will cite all three papers in the revised manuscript, better positioning our work within the broader multimodal learning literature.

Reviewer_FuhGOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper considers robust multimodal regression under noisy inputs. A Structured Predictive Distributions (SPD) framework is proposed to perform multimodal fusion using Normal-Inverse-Gamma representations, together with predictive regularization and distribution-aware gradient reweighting. The paper also provides a conditional generalization-bound analysis.

Critical strengths

-The idea of using structured predictive distributions to regularize multimodal regression is interesting. -The empirical evaluation covers three tasks and multiple corruption types, with promising improvements under noisy conditions. -Ablation studies show that both predictive regularization and gradient reweighting contribute to the performance.

Critical weaknesses

-There appears to be a mismatch between the training regularizer and the theoretical analysis. The location penalty in Eq. (32) uses the learned precision υ\upsilon, while Eq. (48) uses the fixed reference precision υ0\upsilon_0. Since the subsequent capacity bound depends on this term, the consistency of the proof needs clarification. -The main theoretical result requires bounded predictive variance, uniformly nondegenerate features, a bounded linear readout, and an explicit regularization budget. However, these conditions are not shown to hold for the experimental models. Without such evidence, it is difficult to assess whether the analysis explains the reported improvements. -The empirical “bound proxy” is defined as MSE plus twice MAE. A lower value indicates better prediction accuracy, but does not directly demonstrate reduced model complexity. Therefore, the plots provide limited evidence for the claimed generalization-bound tightening. -DAR averages uncertainty weights across the mini-batch and then scales the gradients of each modality-specific head. Its advantage over simpler gradient scaling or learning-rate adjustments is unclear from the current experiments.

Questions for authors

-Could the authors clarify the discrepancy between Eqs. (32) and (48), and specify which regularizer is used in the implementation? -Are the assumptions required by the theoretical analysis enforced or verified experimentally? Can the authors report the empirical-risk and complexity terms separately to demonstrate that the bound is tighter in practice? -How does DAR compare with fixed gradient scaling, tuned modality-specific learning rates, and sample-wise uncertainty weighting?

5 comments

Response for FuhG

Thank you for your response. Here are our clarification.

1. Reference precision in Eqs. (32) and (48). Both location penalties use the fixed reference precision υ0=0.1\upsilon_0=0.1. The learned precision υ(i)\upsilon^{(i)} is exclusively used in the separate precision ratio term. In the depth implementation, prior_v = 0.1. Setting γ0=0\gamma_0=0 yields (α−1)υ0γ2/(2β)(\alpha-1)\upsilon_0\gamma^2/(2\beta) in both equations. We will unify the notation by removing the sample index from the reference parameters in the revised version.

2. Assumptions, empirical risk, and complexity. Training uses the soft penalty in Eq. (7). We empirically verify the fitted budget and its induced parameter constraint on 1,998 reference images across int_head13, int_head14, int_head23, and int_head24, including location biases. The NIG parameterization guarantees positive variance. The maximum observed aleatoric variance is 6557.43, and all 384 weighted Grams are positive definite.

Table A. Gram and discrepancy measurements.

HeadOutput channels / augmented feature dimensionMinimum unweighted Gram eigenvalueMinimum channelwise weighted Gram eigenvalueMean discrepancy DMean auxiliary term AMean location contribution b
int_head13128 / 1292.27e-41.58e-50.01130.01084.81e-4
int_head1464 / 651.32e-42.23e-50.01720.01648.11e-4
int_head23128 / 1291.13e-41.13e-50.01100.01064.46e-4
int_head2464 / 657.96e-58.83e-60.01340.01286.18e-4

The feature extractors are nonlinear, and location predictions remain linear in their learned features. Location biases are incorporated through augmented features. Keeping inverse variance inside each Gram evaluates the location penalty exactly.

For Vh,j=βh,j/(αh,j−1)V_{h,j}=\beta_{h,j}/(\alpha_{h,j}-1) and υ0=0.1\upsilon_0=0.1,

G_{h,j}=\mathbb E_{Q,\mathrm{position}}\left[\frac{\phi_h\phi_h^\top}{V_{h,j}}\right].$$ The mean discrepancy is 0.01324, with an auxiliary contribution of 0.01265 and a location contribution of $5.89\times10^{-4}$. Using the observed location contributions as channel budgets gives $$\lVert W\rVert_F^2\leq\sum_{h,j}\frac{2b_{h,j}}{\upsilon_0\lambda_{\min}(G_{h,j})} =r_{\mathrm{ch}}^2, \qquad r_{\mathrm{ch}}=459.09.$$ **The fitted norm is 11.10**, which is well within the theoretical bound. The channel budgets define a subclass of the aggregate budget class, with $\tau_{\mathrm{agg}}=A_Q+\sum_{h,j}a_{h,j}b_{h,j}$ under the implemented averaging coefficients $a_{h,j}$. For fixed remaining parameters $\theta$, the complete raw normalized log depth output is $g_\theta+T_\theta W$, where $g_\theta$ includes the fixed output bias. Translation invariance and homogeneity imply $$C_S(\mathcal F_{\mathrm{ch},\theta}) \leq C_S(\mathcal F_\theta(r_{\mathrm{ch}})) =0.911\,C_S(\mathcal F_\theta(503.94)).$$ **Table B. Empirical risk and complexity.** The loss is valid pixel MSE in raw normalized log depth per image, averaged equally over 999 DENSE Town10 images. Complexity uses the same weights $v_{ij}/n_i$ inside the signed feature sums. We use 128 sign samples and 64 of 50,176 output coordinates. The table compares two radius classes for the same NIG prediction. | Input | NIG empirical risk | C_r | C_R | Radius r complexity contribution | Radius R complexity contribution | Complexity contribution difference | |---|---:|---:|---:|---:|---:|---:| | Clean | 0.0126 | 46.77 | 51.34 | 3.257e+7 | 3.575e+7 | -3.181e+6 | | Image Gaussian, σ = 30 | 0.0854 | 46.76 | 51.33 | 3.256e+7 | 3.574e+7 | -3.181e+6 | | Event impulse, r = 0.25 | 0.0132 | 46.70 | 51.26 | 3.252e+7 | 3.569e+7 | -3.177e+6 | The common squared loss constant is $L\approx3.482\times10^5$, and the normalized loss range envelope is $1.671\times10^8$. At $\delta=0.05$, the confidence contribution is $2.347\times10^7$, which cancels in the comparison. The complexity contributions are $2LC_r$ and $2LC_R$, demonstrating a clear relative capacity restriction.

Dear Reviewer FuhG,

Your comments helped us identify several points that needed stronger support, which we have addressed through the clarifications and additional experiments above. We would value your assessment of these additions. Is there any aspect you would like us to explain further while the discussion remains open?

We appreciate your time and engagement with our work.

Reviewer_FuhGOpenReview ↗

Response to provided clarifications

Thank you for the response. The reference precision and the empirical verification of the theoretical assumptions are clear now. But the problem with bound proxy and batch-level weighting remains. I therefore maintain my score.

Reviewer_vguWOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposes SPD, a unified framework combining NIG-based predictive representations, multimodal fusion, and regularization for robust multimodal regression. It also introduces distribution-aware reweighting (DAR) and provides a Rademacher-complexity analysis showing conditionally tighter generalization bounds. Experiments on image fusion, crowd counting, and depth estimation demonstrate promising performance, particularly under noisy conditions.

Critical strengths
  1. Unified framework and theoretical analysis. The paper integrates predictive distributions, regularization, and adaptive optimization into a coherent framework. The theoretical attempt to connect predictive regularization with generalization bounds is interesting and provides a useful perspective on robust multimodal regression.
  2. Promising empirical results. The method is evaluated across multiple tasks and noise settings, with consistent improvements in robustness. This broad evaluation supports the practical relevance of the proposed approach.
Critical weaknesses
  1. Limited novelty relative to prior work. NIG-based multimodal fusion is closely related to MoNIG, while uncertainty-aware gradient modulation has also been explored previously. Without direct comparisons with MoNIG and simpler regularization baselines, it is difficult to identify which components provide genuinely new contributions.
  2. Weak connection between theory and experiments. The generalization analysis relies on strong assumptions that are not verified for the implemented models. Moreover, the reported "bound proxy" combines MSE and MAE rather than directly measuring model complexity, so it does not convincingly validate the theoretical claims. DAR also applies batch-level gradient scaling, leaving its intended sample-level robustness mechanism unclear.
Questions for authors
  1. Could the authors clarify the key methodological differences from MoNIG and provide direct comparisons with MoNIG and simpler regularization baselines? This would help assess the novelty and necessity of the proposed components.
  2. How do the authors justify the theoretical assumptions for their actual implementations? Can they provide evidence that the predicted reduction in model complexity occurs in practice, beyond the reported error-based proxy?
  3. Since DAR scales gradients at the batch level, how does it specifically reduce the influence of unreliable samples? Could the authors clarify this mechanism or provide supporting evidence?
6 comments

Response for vguW

Thank you for your precious suggestions. We address your concerns in as follows.

1. Differences from MoNIG and simpler regularization. Both methods use NIG representations. MoNIG penalizes high evidence when a prediction is inaccurate. SPD regularizes each modality's NIG representation before fusion toward a fixed reference distribution (Eq. 7). MoNIG studies its fusion rule; our theory derives conditions under which this predictive regularizer restricts the final point-prediction class (Corollary 1).

We adapted MoNIG to all three tasks. SPD obtained higher scores on the four fusion metrics below and lower counting and depth errors in the reported comparisons.

BenchmarkInputMethodSSIM ↑MS-SSIM ↑QW ↑FMI ↑
LLVIPCleanMoNIG0.4300.8870.7200.886
LLVIPCleanSPD0.4690.9380.8900.896
LLVIPGaussianMoNIG0.1560.7580.6360.860
LLVIPGaussianSPD0.3950.8200.8300.879
MFFCleanMoNIG0.7090.9730.8650.877
MFFCleanSPD0.7140.9940.9270.885
MFFGaussianMoNIG0.3240.8510.7670.852
MFFGaussianSPD0.4820.8580.8340.861
MEFCleanMoNIG0.4050.9130.6500.891
MEFCleanSPD0.4430.9610.8290.901
MEFGaussianMoNIG0.1610.7850.5850.865
MEFGaussianSPD0.3420.8300.7410.866
BenchmarkInputMoNIG GAME(0) ↓SPD GAME(0) ↓MoNIG RMSE ↓SPD RMSE ↓
RGBT-CCClean10.319.8618.0213.04
RGBT-CCGaussian32.4622.3345.2542.82
DroneRGBTClean7.884.7610.547.24
DroneRGBTGaussian11.279.9921.4615.83
DENSE Town10MethodMAE <10 m ↓MAE <20 m ↓MAE <30 m ↓AbsRel ↓RMSE log ↓SI log ↓
CleanMoNIG0.5841.5912.3800.2360.6290.327
CleanSPD0.5811.5822.2030.1760.2990.070
Image GaussianMoNIG1.9153.6235.7090.5801.7011.135
Image GaussianSPD1.4503.3854.2660.4510.7450.465
Event impulseMoNIG1.4332.6814.3480.4751.1130.518
Event impulseSPD0.6311.7502.3990.1920.3310.083
MVSEC sequenceInputMoNIG AbsRel ↓SPD AbsRel ↓MoNIG RMSE log ↓SPD RMSE log ↓
night1Clean0.4860.3260.6930.457
night2Clean0.4080.2620.6530.403
night3Clean0.4060.2400.6710.405
night1Gaussian0.4200.3980.6910.633
night2Gaussian0.5660.4021.1600.562
night3Gaussian0.5930.3731.2300.563

We also compared NIG with tuned L2 regularization on the same location heads. NIG improved all six Gaussian depth metrics over both L2 candidates. Selected DENSE results are below (Gaussian image noise: σ=30\sigma=30; event impulse noise: r=0.25r=0.25).

RegularizerHead normClean AbsRelGaussian AbsRelImpulse AbsRel
NIG24.230.19480.46660.1945
Best L223.960.19580.48650.1968
L2 closest in norm24.190.20770.51100.2087

2. DAR mechanism and supporting evidence. Seitzer et al. [1] weight individual losses by predicted variance. ReGrad [2] uses modality uncertainty and gradient conflict to balance learning across modalities. DAR derives a multiplier from each modality's predictive spread, combining observation noise and uncertainty about the predicted mean, and applies it to that modality's NIG projection-weight gradient.

DAR acts at the batch level. Larger-spread samples contribute smaller coefficients to the batch average, reducing the whole head's update magnitude while preserving its direction and relative sample contributions (Eq. 10). Batch averaging preserves the relative contributions of difficult but informative examples while adjusting the update magnitude for the modality. With NIG regularization held fixed, DAR reduced Gaussian RMSE on RGBT-CC from 49.61 to 42.82. In the additional depth comparison, DAR achieved lower AbsRel than sample weighting in all 15 settings; on Gaussian MVSEC night1, the values were 0.3897 versus 0.4324.

[1] Seitzer, M., Tavakoli, A., Antić, D., and Martius, G. On the Pitfalls of Heteroscedastic Uncertainty Estimation with Probabilistic Neural Networks. ICLR, 2022.

[2] Lin, X., et al. Suppress and Rebalance: Towards Generalized Multi-Modal Face Anti-Spoofing. CVPR, 2024.

3. Evidence beyond the error proxy. Fig. 3 measures residuals through P=MSE+2MAEP=\mathrm{MSE}+2\mathrm{MAE}. Appendix C.8 relates it to the empirical-risk contribution through R^v,s=ζsMSE≤ζsP\widehat R_{v,s}=\zeta_s\mathrm{MSE}\leq\zeta_sP. We additionally measured variance-weighted Grams and Rademacher complexity directly.

On 1,998 reference images, all 384 channelwise weighted Grams were positive definite; the minimum unweighted patch Gram eigenvalue was 7.96×10−57.96\times10^{-5}. The maximum observed aleatoric variance was 6557.43. The discrepancy decomposed into auxiliary and location contributions of 0.01265 and 5.89×10−45.89\times10^{-4}. The location penalty agreed with its parameter-space quadratic form to relative error 1.91×10−141.91\times10^{-14}.

Table A. Measured feature geometry and NIG discrepancy.

HeadOutput channels / augmented feature dimensionMinimum unweighted Gram eigenvalueMinimum channelwise weighted Gram eigenvalueMean discrepancy DMean auxiliary term AMean location contribution b
Head1128 / 1292.27e-41.58e-50.01130.01084.81e-4
Head264 / 651.32e-42.23e-50.01720.01648.11e-4
Head3128 / 1291.13e-41.13e-50.01100.01064.46e-4
Head464 / 657.96e-58.83e-60.01340.01286.18e-4

The architecture uses nonlinear feature extractors, with location predictions being linear in the learned features. Location biases are included through augmented features. We evaluate the complete prediction pathway across its location heads. Keeping inverse variance inside each Gram evaluates the location penalty directly.

For the raw normalized log depth output fW,θ(x)=gθ(x)+Tθ(x)vec⁡(W)f_{W,\theta}(x)=g_\theta(x)+T_\theta(x)\operatorname{vec}(W), where gθg_{\theta} includes the fixed output bias, the measured channel budgets imply ∥W∥F≤rch=459.09\lVert W\rVert_F\leq r_{\mathrm{ch}}=459.09. Holding θ\theta at its fitted values and using the specified reference budget R=503.94R=503.94 gives rch/R=0.911r_{\mathrm{ch}}/R=0.911. The corresponding radius-class complexity contribution is 8.90% smaller. This parameter-level audit directly validates the capacity reduction in Corollary 1.

Table B. Empirical risk and complexity. Risk is valid-pixel MSE in raw normalized log depth per image, averaged over 999 test images. Complexity estimates use 128 sign samples and 64 output coordinates, with valid-pixel averaging included.

InputNIG empirical riskC_rC_RRadius-r complexity contributionRadius-R complexity contributionComplexity contribution difference
Clean0.012646.7751.343.257e+73.575e+7-3.181e+6
Image Gaussian, σ = 300.085446.7651.333.256e+73.574e+7-3.181e+6
Event impulse, r = 0.250.013246.7051.263.252e+73.569e+7-3.177e+6

The common squared-loss constant is L≈3.482×105L\approx3.482\times10^5; the confidence term cancels. These measurements demonstrate a clear relative reduction in complexity for the same predictor.Complexity contributions are 2LCr2LC_r and 2LCR2LC_R.

Dear Reviewer vguW,

As the discussion period is nearing its end, we would like to follow up on our responses and additional results above. We would appreciate any further feedback, particularly if any of your concerns remain unresolved or require additional clarification. We would be happy to address them.

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.