acceptodds
Under review as a conference paper at ICLR 2027

Gaussian Primitive Fusion for Any-Resolution Multimodal Images

Abstract

Multimodal image fusion integrates complementary observations from different modalities into a unified representation. Existing methods typically estimate modality contributions on a common discrete grid. When source modalities have different native resolutions, constructing such a grid generally involves resampling, which may introduce artifacts and affect subsequent fusion decisions. To address this problem, we propose Gaussian Primitive Fusion (GPF), which reformulates heterogeneous-resolution fusion in a shared continuous Gaussian space. Instead of first converting source images to a common resolution, GPF directly fits their native-resolution observations with shared Gaussian geometry and modality-specific content representations, and learns modality contributions within the resulting continuous representation. Once optimized, the fused representation is rendered at the finest native resolution without resampling the sources. With heterogeneous-resolution inputs, GPF produces clearer target contours and finer textures, with less blurring and blocking artifacts than resample-then-fuse pipelines. Quantitatively, it achieves the best results on multiple infrared-visible benchmarks against state-of-the-art fusion methods equipped with interpolation or super-resolution frontends.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Multimodal Image Fusion, Gaussian Primitives, Continuous Image Representation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_HZdNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work targets multi-modal image fusion under mismatched input resolutions, a common practical scenario where standard methods rely on pre-resampling and introduce interpolation artifacts. The authors propose GPF, a continuous representation built from 2D Gaussian primitives. The pipeline fits Gaussian parameters from raw source images at their native resolution in stage one, then only optimizes multi-scale fusion weights on these fixed primitives in stage two. Experiments are conducted on infrared-visible and PET-MRI datasets, with comparisons against many existing fusion approaches.

Critical strengths
  1. The problem setting is practically meaningful. Most existing multi-modal fusion methods assume perfectly aligned inputs of identical resolution. The proposed Gaussian primitive representation avoids upfront resampling of source images, which directly addresses the interpolation artifacts introduced by pre-upsampling low-resolution inputs.
  2. The overall framework is conceptually novel. Using 2D Gaussians as a continuous scene representation for fusion, instead of discrete pixel grids or coordinate MLPs, provides an alternative way to handle arbitrary output resolution. The authors perform extensive quantitative and qualitative tests including non-uniform downsampling cases.
Critical weaknesses
  1. There are too few evaluation indicators. More evaluation indicators are needed to measure the performance of the model. 2.The fusion result of the model is not competitive.
Questions for authors

1.There are too few evaluation metrics; more metrics need to be provided, like EN, AG, etc. Three metrics aren't enough to assess the model's performance. 2. From a visual perspective, the fusion results aren’t as good as a single image. The author needs to evaluate the model’s effectiveness on downstream tasks to verify if the model has any advantages in computer vision. Whether there’s an advantage in downstream tasks directly determines the significance of this work. 3.Why did you choose to freeze Gaussian geometry after stage 1 instead of jointly optimizing Gaussian parameters together with fusion weights? Can you add this baseline and compare the performance? 4.All experiments assume perfectly registered source images. How much performance drop will occur with small registration offsets between two modalities? Could you provide results under misalignment perturbations?

10 comments
AuthorsOpenReview ↗

Response (1/5): additional metrics, including EN and AG

We thank you for your time and effort in reviewing our paper and providing feedback. Please find our responses to the concerns you have raised below. All new figures (Figs. R1–R7) and tables are included in the appendix of the revised paper.

W1 & Q1. Additional evaluation metrics, including EN and AG

We thank the reviewer for this suggestion. For GPF and the 16 compared methods of the paper in the VIS×16 setting, we add Q_CB and N_abf and report EN and AG (Tables R8 and R9). For a comprehensive comparison, we rank all methods on all metrics, including these four metrics, the three metrics of the paper and the two downstream tasks (Table R10): GPF ranks first on seven of the nine and has a mean rank of 2.4, compared with at least 4.9 for the other methods.

(a) GPF preserves the most contrast and introduces very few artifacts. Q_abf measures how well the edges of the sources are transferred to the fused image, Q_CB measures how well their perceptually important local contrast is preserved, based on a model of human vision, and N_abf measures the artifacts introduced by fusion, where lower is better. On all five datasets, GPF achieves the best Q_abf (Table C.4) and the best Q_CB, 4.8% to 37.9% higher than the second-best method. It also achieves the lowest N_abf on four datasets and the second-lowest on the remaining one, and its N_abf is below one fourth of the average of the compared methods on every dataset.

(b) GPF introduces few artifacts because it fuses the sources at their native resolutions without any resampling frontend, and its fusion weights vary smoothly within the Gaussian supports, which follow the image structures, so no abrupt transitions appear between regions taken from different sources. Its band-wise targets select the source with the stronger local response (Eq. (9)), which preserves the contrast of both sources and leads to its high Q_CB.

(c) EN and AG, in contrast, are computed from the fused image alone: EN measures the entropy of its intensity histogram, and AG its average gradient magnitude. They therefore also increase with noise and artifacts, such as the ringing, halos and spurious textures introduced by resampling and fusion. On these metrics, GPF remains competitive: its EN and AG are higher than the average of the compared methods on all five datasets, and its EN is within 4.4% of the best. Moreover, on the same dataset, every compared method with a higher EN or AG than GPF also has a higher N_abf and a lower Q_CB, so its higher values come with more artifacts and less preserved contrast.

Table R8. Q_abf, VIF_Σ, S_Σ, Q_CB, N_abf, EN and AG of GPF and the 16 compared methods of the paper in the VIS×16 setting on LLVIP, MSRS and M3FD. Q_abf, VIF_Σ and S_Σ follow Table C.4 of the paper. The best result in each column is shown in bold.

LLVIP

MethodQ_abf↑VIF_Σ↑S_Σ↑Q_CB↑N_abf↓EN↑AG↑
DIDFuse0.1200.3350.8600.3630.00765.811.13
SeAFusion0.3720.7191.2510.3920.01327.362.32
CDDFuse0.3960.7491.2560.3940.00927.332.21
AGMFusion0.3660.7911.2240.3700.01027.532.24
MFDFuse0.3840.8161.2600.3750.00837.422.03
MLFuse0.3210.6141.2470.3100.00477.141.76
TDFusion0.3880.7591.2820.3950.01197.352.24
Deno-IF0.3400.7201.2740.3660.01057.442.02
TG-ECNet0.2450.4271.1500.3080.00337.101.81
RIS-Fuse0.3570.6291.2680.3620.00787.052.10
LUT-Fuse0.3550.7281.2620.3620.00497.251.96
TITA0.3880.7861.2720.3720.00457.321.99
MaeFuse0.2490.5501.2490.3760.00587.141.83
FDFuse0.4050.8501.2440.3850.01397.482.24
MFS-Fusion0.4160.8431.2720.4150.00627.372.31
SETFusion0.2150.5041.2420.3030.00266.621.42
GPF (ours)0.4401.0121.3180.4350.00107.202.02
AuthorsOpenReview ↗

Response (2/5): additional metrics (continued)

W1 & Q1 (continued)

Table R8 (continued).

MSRS

MethodQ_abf↑VIF_Σ↑S_Σ↑Q_CB↑N_abf↓EN↑AG↑
DIDFuse0.0790.2130.7060.3760.00754.541.01
SeAFusion0.2600.4361.1670.3550.02526.782.40
CDDFuse0.2350.4211.1570.3460.02586.812.20
AGMFusion0.2460.5371.1540.3320.01176.902.20
MFDFuse0.2020.4081.1640.2700.00696.841.58
MLFuse0.2090.4341.1750.2840.00696.581.66
TDFusion0.2860.5801.2070.3750.02046.692.25
Deno-IF0.1960.4601.1810.3090.00756.781.59
TG-ECNet0.1310.1891.0440.2700.00756.491.71
RIS-Fuse0.2160.4061.1640.2780.00676.781.63
LUT-Fuse0.2050.4511.1600.2810.00726.631.81
TITA0.1880.3751.2070.2740.00406.611.38
MaeFuse0.1990.4511.2090.3660.00666.591.76
FDFuse0.2770.5801.1480.3220.01196.872.10
MFS-Fusion0.2730.5011.2100.3710.01916.692.19
SETFusion0.1300.3291.1740.3000.00545.271.21
GPF (ours)0.3530.9831.3380.5190.00186.732.06

M3FD

MethodQ_abf↑VIF_Σ↑S_Σ↑Q_CB↑N_abf↓EN↑AG↑
DIDFuse0.2570.5971.2140.3860.01267.051.92
SeAFusion0.2700.4801.3060.3210.01676.802.20
CDDFuse0.2410.4031.2880.3130.01406.901.90
AGMFusion0.2810.6131.2790.3300.01116.942.12
MFDFuse0.2550.5321.2790.3110.01497.011.84
MLFuse0.2900.5741.3540.3530.00436.791.72
TDFusion0.3270.6861.3620.4050.01237.022.07
Deno-IF0.2550.5721.3650.3650.01056.821.72
TG-ECNet0.1680.2871.2430.3210.00236.791.47
RIS-Fuse0.1860.2371.3010.2270.00596.431.28
LUT-Fuse0.1860.3751.3070.2220.00496.541.45
TITA0.2350.4701.3660.2980.00146.461.26
MaeFuse0.2080.4761.3680.3960.00316.841.55
FDFuse0.2900.5441.3200.3290.02266.932.23
MFS-Fusion0.3110.6191.3560.3620.00556.761.90
SETFusion0.2150.5371.2730.3700.03137.002.34
GPF (ours)0.3760.9601.4200.5150.00056.892.01

Table R9. Q_abf, VIF_Σ, S_Σ, Q_CB, N_abf, EN and AG of GPF and the 16 compared methods of the paper in the VIS×16 setting on RoadScene and TNO, with the same layout as Table R8. The best result in each column is shown in bold.

RoadScene

MethodQ_abf↑VIF_Σ↑S_Σ↑Q_CB↑N_abf↓EN↑AG↑
DIDFuse0.4940.6101.2060.4310.02547.304.52
SeAFusion0.4970.6281.2370.4510.04667.375.45
CDDFuse0.4980.6011.2590.4400.04157.434.88
AGMFusion0.4230.5651.2150.3750.02727.124.61
MFDFuse0.5090.6841.2360.4350.03567.454.84
MLFuse0.4850.5951.3070.4200.00297.133.40
TDFusion0.5650.7521.2800.4670.03587.425.04
Deno-IF0.4540.5971.2980.4750.01917.304.03
TG-ECNet0.1670.2021.0870.3610.01637.022.50
RIS-Fuse0.3830.3251.2650.3090.01906.983.64
LUT-Fuse0.4240.5441.2960.3980.01236.943.41
TITA0.5290.6901.3580.4230.00447.063.79
MaeFuse0.4800.5681.3330.4820.00777.093.86
FDFuse0.5370.6671.2660.4490.04227.355.23
MFS-Fusion0.6150.8461.3310.5120.01217.254.78
SETFusion0.3660.4221.1760.3650.04377.014.61
GPF (ours)0.6560.9931.3800.5820.00017.264.56

TNO

MethodQ_abf↑VIF_Σ↑S_Σ↑Q_CB↑N_abf↓EN↑AG↑
DIDFuse0.3080.5251.1300.3840.03086.952.53
SeAFusion0.3610.5831.2120.3840.04857.173.22
CDDFuse0.3490.5341.2250.3770.03167.202.77
AGMFusion0.3370.5721.1980.3510.03587.053.04
MFDFuse0.3480.6371.1900.3710.03697.302.85
MLFuse0.3440.5521.2570.3890.01237.002.29
TDFusion0.4120.6721.2580.4130.03287.212.95
Deno-IF0.2880.5091.2560.4000.01507.162.15
TG-ECNet0.1840.2801.1350.3550.01586.861.94
RIS-Fuse0.2680.3461.2230.3020.01296.982.01
LUT-Fuse0.3040.4921.2330.3280.01016.782.14
TITA0.3420.5701.2860.3620.00436.791.96
MaeFuse0.3080.5141.2900.4370.00906.872.22
FDFuse0.3760.5931.2220.3890.04017.253.06
MFS-Fusion0.4360.7211.2910.4420.01346.942.71
SETFusion0.2880.4911.1780.3680.05896.802.94
GPF (ours)0.4920.9381.3210.5260.00547.132.89
AuthorsOpenReview ↗

Response (3/5): overall ranking and visual quality

W1 & Q1 (continued)

Table R10. Ranks of GPF and the 16 compared methods of the paper in the VIS×16 setting, where 1 is the best. The image metrics are averaged over the five datasets before ranking, the downstream tasks use the results without offset (Tables R11 and R12), and tied methods share the average rank. INR-Fuse is not included because Q_CB, N_abf, EN and AG were computed only for the 16 methods of the paper.

Reference-basedNo-referenceDownstream tasks
MethodQ_abfVIF_ΣS_ΣQ_CBN_abfENAGSeg.Det.Mean rank
GPF (ours)1111187112.44
TDFusion335413336.544.94
MFS-Fusion22328966.595.28
FDFuse441081522235.56
SeAFusion591271771427.11
AGMFusion9514111154357.44
Deno-IF118669610867.78
MaeFuse13134341111588.00
MFDFuse761310121812108.78
CDDFuse611119144510139.22
MLFuse1010712310131379.44
TITA872132131614119.56
LUT-Fuse121281651514111512.00
RIS-Fuse14169177141591713.11
DIDFuse1515175101712171413.56
TG-ECNet1717161561217151214.11
SETFusion1614151416169161614.67

W2 & Q2. Visual quality compared with a single image, and downstream detection and segmentation

(1) Visual quality compared with a single image. We thank the reviewer for this observation. A fused image is not meant to reproduce the higher-resolution source, but to add the complementary content of the other source, such as thermal targets, while keeping the details of the higher-resolution source. Since the other source is low-resolution, this content is coarse, so a fused image can look different from the higher-resolution source alone. The additional comparisons with zoomed-in regions (Figs. R2–R5) show that GPF keeps these details while adding the complementary content. Quantitatively, GPF also has the best mean rank over the seven metrics and two downstream tasks in Table R10 (see W1 & Q1).

(a) In Fig. R2 (IR×16), the compared methods spread the upsampled infrared response into blurred bright regions around the pedestrian and over the ground, whereas GPF keeps a clear contour of the pedestrian, with distinguishable head and limbs, and the sharp structure of the traffic light.

(b) In Fig. R3 (VIS×16), GPF keeps the sharp edges of the car and the pedestrian from the infrared image and places the colors of the visible image within them, while several compared methods blur the colored regions beyond the object boundaries.

(c) In Fig. R4 (VIS×16), the compared methods produce over-bright regions and distorted shapes around the vehicle and the roadside, whereas GPF keeps the outlines of the vehicle and the pedestrians and the texture of the road.

(d) In Fig. R5 (VIS×32), the compared methods lose most visible structures and surround the pedestrians with bright halos, while GPF still shows the leaves, the building facade and the pedestrians clearly, with these structures from the infrared image and the coarse content from the visible image.

Reviewer_uLGfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Gaussian Primitive Fusion (GPF) for registered multimodal images with heterogeneous input resolutions. GPF first fits modality-specific source attributes on a shared set of adaptive two-dimensional Gaussian primitives, using observation operators that compare predictions with each source at its native resolution. It then freezes the source representation and optimizes primitive-associated, band-specific fusion scores through a multiscale decomposition. The learned parameters can subsequently be reused for different output grids. Validation includes infrared–visible benchmarks, PET–MRI examples, comparisons with fusion methods equipped with reconstruction frontends, heterogeneous sampling configurations, and ablations of observation constraints, shared geometry, fusion weights, and representation capacity.

Critical strengths
  1. Learning separate fusion weights for each frequency band makes sense: coarse structure and fine details may benefit from different modalities. Table 2 supports this choice over using the same weights across scales.
  2. The native-observation formulation is a useful design choice. Fitting each modality through its own observation operator avoids treating an interpolated low-resolution image as observed data. Sharing geometry while retaining separate source attributes also gives the method a sensible way to allocate spatial capacity using both modalities. This is the aspect of the paper I find most promising.
Critical weaknesses
  1. The main concern is what the Gaussian formulation adds to the fusion itself. After fitting, the method evaluates both sources on a common grid and uses Laplacian bands with stronger-gradient targets. The learned scores appear to regularize these largely predetermined choices. Table 2 and Figure 7 help, but the capacity matching and the definition of fusion scores under independent geometries need clarification. Without these details or a direct INR-Fuse comparison, the gain over adaptive reconstruction followed by standard multiscale fusion remains unclear.
  2. The results do not show clearly how much the degraded modality contributes. Simply preserving the higher-resolution source could already give a sharp output and good source-based scores, especially when VIF and SSIM are summed using bicubically enlarged low-resolution references. Yet neither source alone is included as a baseline. In Tables C.3 and C.4, GPF’s scores barely change when visible degradation increases from ×4 to ×16. This could reflect robustness, but it could also mean limited use of the visible input. Single-source controls and separate per-modality scores are needed to distinguish the two.
Questions for authors
  1. In the independent-geometry ablation, how are fusion scores parameterized and associated with the two source dictionaries? Does this comparison preserve the effective capacity and spatial regularization of the shared-geometry model? This detail is necessary to interpret the large performance difference.
  2. Evidence needed to interpret the main results: How does GPF compare with returning the higher-resolution source alone, including its fitted Gaussian reconstruction? Please also separate the two source contributions to the reported VIF and SSIM sums. These controls would help determine whether the gains reflect complementary fusion or predominantly preservation of one modality.
  3. The Laplacian hierarchy is reconstructed on each requested output grid. Does changing that grid change the physical scales at which fusion decisions are applied? What supports interpreting the resulting outputs as consistent readouts of one fused continuous representation?
11 comments
AuthorsOpenReview ↗

Response (1/7): the Gaussian formulation and the independent-geometry ablation

We thank you for your time and effort in reviewing our paper and providing feedback. Please find our responses to the concerns you have raised below. All new figures (Figs. R1–R7) and tables are included in the appendix of the revised paper.

W1 & Q1. What the Gaussian formulation adds to fusion, the independent-geometry and other ablations, and comparison with INR-Fuse

(1) Contribution of the Gaussian formulation to the fusion itself. We thank the reviewer for this question. The Gaussian formulation provides a joint representation of the two sources and the fusion decisions. Instead of reconstructing each modality with its own primitives and then fusing the reconstructions pixel by pixel, GPF represents both sources and the fusion weights on one shared set of Gaussian primitives. With everything else unchanged, this raises Q_abf to 0.575, from 0.426 with separate primitives for the two modalities (Table 2B) and from 0.530 with pixel-wise weights (Table 2C). The difference lies in what each primitive contains.

(a) In an independent reconstruction, each modality mm has its own primitives, θk(m)=,pk(m),Σk(m),fk(m),,m=1,2,\theta^{(m)}_k = \\{\\, p^{(m)}_k,\\ \Sigma^{(m)}_k,\\ f^{(m)}_k \\,\\}, \quad m = 1, 2, with their own positions and shapes. The two sources then share no spatial unit, so a fusion decision cannot be attached to a primitive that describes both of them; it is either made per pixel or obtained by comparing two unrelated score fields at each pixel, as in the independent variant of (2).

(b) In GPF, each primitive carries the shared geometry, the content of both modalities and one fusion score per band, θk=,pk,Σk,fk(1),fk(2),ak,0,…,ak,L,,\theta_k = \\{\\, p_k,\\ \Sigma_k,\\ f^{(1)}_k,\\ f^{(2)}_k,\\ a_{k,0}, \ldots, a_{k,L} \\,\\}, and the sources and the fusion scores are expanded on the same basis functions ϕk\phi_k (Eqs. (2)–(3)), f(m)(x)=∑kϕk(x),fk(m)f^{(m)}(x) = \sum_k \phi_k(x)\\, f^{(m)}_k and qb(x)=∑kϕk(x),ak,bq_b(x) = \sum_k \phi_k(x)\\, a_{k,b}, where qbq_b gives the fusion weight of band bb through a sigmoid.

(c) As a result, the fusion compares the two sources on common units: the shared primitives are fitted to the native observations of both modalities (Eq. (5)), so each primitive describes the same region in both sources.

(d) In addition, the fusion decision for a region is stored in the same primitive as its content, so all pixels covered by a Gaussian share its scores. For this reason, the weights do not merely regularize the gradient-selected choices: the targets are fixed by gradient selection (Eq. (9)), but a pixel-wise rule follows every local gradient change, whereas GPF can realize the targets only through the scores of the shared primitives.

The ablations in Table 2 verify (c) and (d), and we apologize that the paper described them too briefly. Separate geometries for the two modalities, each with its own fusion scores, give Q_abf 0.426 (Table 2B), compared with 0.575 for GPF. Keeping the shared reconstruction, targets and losses of GPF but assigning two learnable weights to each pixel in each band (Pixel scores), which amounts to adaptive reconstruction followed by pixel-wise multiscale fusion, gives 0.530, and fixed equal weights give 0.467 (Table 2C). Attaching the scores to Gaussians on a fixed regular grid with the same number of parameters also couples neighboring pixels, but gives only 0.514, so the gain does not come from smoothing alone. Fig. 7 further keeps the fusion rule fixed and shows that adaptive Gaussians outperform linear grids, cubic B-splines, SIREN and fixed Gaussians of comparable capacity.

(2) Capacity and fusion weights in the independent-geometry ablation. We thank the reviewer for checking the fairness of this ablation. The comparison does not put the independent variant at a disadvantage: it has twice the capacity of GPF, uses the same fusion rule and objective, and its scores are also spatially regularized by Gaussian supports; the only difference is that the two modalities no longer share their Gaussians.

(a) It assigns K = 25,000 Gaussians to each modality, the same number as the shared Gaussians of GPF, and therefore uses twice as many Gaussians in total.

(b) Its fusion weights follow the same rule as in GPF, except that each modality has its own Gaussians and scores: with sb(m)s_b^{(m)} denoting the score field of modality mm averaged to the scale of band bb, the weight is αb=σ(sb(2)−sb(1))\alpha_b = \sigma(s_b^{(2)} - s_b^{(1)}), which normalizes the two modalities against each other.

(c) The scores of each modality are still expanded on its own Gaussians, so they vary smoothly within the Gaussian supports, and the fusion objective and its regularization terms are the same as in GPF.

Under these conditions, the independent variant reaches Q_abf 0.426, whereas GPF reaches 0.575, which shows the value of sharing the adaptive supports between the two modalities.

AuthorsOpenReview ↗

Response (2/7): other ablation settings and comparison with INR-Fuse

W1 & Q1 (continued)

(3) Settings of the other ablations in Table 2. All variants use the setting of the first column of Table 1, K = 25,000 Gaussians and the same optimization and losses, and each changes only one component; GPF outperforms all of them. The other variants are defined as follows.

(a) Resize targets (Table 2A) fits the lower-resolution source after enlarging it to the high-resolution grid by bicubic interpolation, instead of at its original resolution.

(b) IR-led and VIS-led (Table 2B) fit the geometry to the infrared or the visible image alone, freeze it, and then fit the attributes of the other modality on it.

(c) Without learned weights (Table 2C) uses fixed equal weights for the two sources in every band.

(d) Shared across scales (Table 2C) uses one score per Gaussian for all bands instead of one score per band.

(4) Comparison with INR-Fuse. We thank the reviewer for the suggestion. GPF outperforms INR-Fuse on all three metrics and all five datasets of Table 1.

(a) We run INR-Fuse with the settings described in its paper. GPF achieves Q_abf of 0.452 to 0.690, compared with 0.158 to 0.357 for INR-Fuse (Table R3), and INR-Fuse is also included in the runtime and downstream comparisons (Tables R1, R11 and R12).

(b) INR-Fuse produces over-smoothed results in which pedestrians, vehicles and vegetation are hard to recognize (Figs. R2–R5, panel (q)).

(c) The two methods use their continuous representations differently: INR-Fuse represents the fused image directly with a coordinate network trained with consistency losses to the sources, whereas GPF represents the two sources on shared Gaussians under their original observation constraints and learns the fusion weights on these Gaussians.

Table R3. Comparison with INR-Fuse in the setting of Table 1. The VIF_Σ and S_Σ of INR-Fuse are the sums of its per-source terms in Tables R4 and R5. The better result of each pair is shown in bold.

DatasetQ_abf INR-FuseQ_abf GPFVIF_Σ INR-FuseVIF_Σ GPFS_Σ INR-FuseS_Σ GPF
LLVIP0.1580.5750.2990.9351.2981.314
MSRS0.2150.6590.4660.9591.1411.367
M3FD0.3570.6640.3320.9721.3781.427
RoadScene0.2310.4520.3350.9651.3251.379
PET–MRI0.2930.6900.4760.7421.0851.329

Table R1. Runtime (ms) per image pair of all methods on an RTX 3090 for different image sizes. The runtime of the compared methods does not include their frontends. The peak GPU memory is reported in Fig. 8(b) of the main paper. Bold marks the fastest method at each size.

Method256×256512×512768×7681024×10241280×1024
DIDFuse6.622.146.782.7108.0
SeAFusion78.097.3129.6176.3209.9
CDDFuse128.3286.0555.2932.11150.2
AGMFusion4.212.426.239.556.5
MFDFuse124.0484.11103.81962.22461.7
MLFuse7.017.739.064.482.7
TDFusion45.8146.9333.3588.8742.7
Deno-IF64.6207.1466.6822.41028.8
TG-ECNet312.61057.52373.34230.75318.9
RIS-Fuse70.5175.6368.2632.9789.2
LUT-Fuse5.912.925.545.557.9
TITA924.73780.29033.115825.019775.3
MaeFuse267.5271.6274.3286.2290.7
FDFuse49.5177.2406.7722.5905.1
MFS-Fusion170.6388.7853.21511.01903.9
SETFusion78.6291.4689.51206.21515.1
INR-Fuse98121.597640.2100293.1116285.0100348.7
GPF (ours)29.775.4127.1208.6263.1
AuthorsOpenReview ↗

Response (3/7): single-source controls and per-source scores

W2 & Q2. Contribution of the degraded modality: single-source controls, per-modality scores, the ×4 to ×16 results and downstream verification

(1) Single-source controls and per-modality scores. We thank the reviewer for this valuable suggestion. The per-source scores show that GPF adds the information of the lower-resolution source while keeping the details of the higher-resolution source, rather than only preserving the higher-resolution one.

We add three controls, namely the higher-resolution source itself, its fitted Gaussian reconstruction and the bicubically enlarged lower-resolution source, and compute VIF and SSIM with each source separately; the controls give the scores obtained without any fusion. We report these scores for all methods in the setting of Table 1 (IR×16 and native PET–MRI; Tables R4 and R5), and for GPF and the controls at VIS×4 and VIS×16 (Tables R6 and R7).

(a) In the setting of Table 1, the VIF terms of GPF for the lower-resolution source are 1.6 to 7.2 times those of the higher-resolution source alone on all five datasets, and its SSIM terms are also higher, while its terms for the higher-resolution source remain high.

(b) When the visible image is the lower-resolution source, the fused image leans toward the infrared image in these scores, because the fine details at the output resolution exist only in the infrared image. It is nevertheless not the infrared image alone: its visible terms remain higher than those of the infrared image alone on all five datasets, its colors come from the visible image (Figs. R3–R5), and it outperforms the infrared input alone in downstream tasks with VIS×16 (see (3)).

Table R4. Per-source scores on LLVIP, MSRS and M3FD (IR×16, the setting of Table 1), where the infrared image is the lower-resolution source. Q_abf and the sums (Σ) follow Table 1, with INR-Fuse added, and VIF and SSIM are also given for each source. For the controls, only the per-source terms are reported. HR source: the higher-resolution source itself; HR Gaussian: its fitted Gaussian reconstruction; LR bicubic: the lower-resolution source enlarged by bicubic interpolation. Because of rounding, a sum may differ from its two terms by 0.001. Bold marks the best fusion result in Q_abf and the sums.

LLVIP

MethodQ_abfVIF ΣVIF IRVIF VISSSIM ΣSSIM IRSSIM VIS
DIDFuse0.2500.3660.1780.1880.9000.3550.545
SeAFusion0.4410.5160.2480.2681.2240.7130.511
CDDFuse0.3880.5020.2720.2301.2390.7470.492
AGMFusion0.4240.5750.3200.2551.1560.7050.451
MFDFuse0.3850.5100.3430.1671.2530.7660.487
MLFuse0.3780.5220.2660.2561.2420.7270.515
TDFusion0.5270.7340.2400.4941.2720.6710.601
Deno-IF0.3470.5170.2760.2411.0730.6370.436
TG-ECNet0.4110.5820.1620.4201.1010.5890.512
RIS-Fuse0.5350.8130.2850.5281.2760.6280.648
LUT-Fuse0.2830.3830.2900.0931.2040.7780.426
TITA0.4590.5870.3490.2381.2370.7410.496
MaeFuse0.1830.4080.2650.1431.2010.6980.503
FDFuse0.4680.6410.3750.2661.2120.7330.479
MFS-Fusion0.5130.6810.2770.4041.2340.7100.524
SETFusion0.1760.2510.2060.0451.1850.7430.442
INR-Fuse0.1580.2990.2580.0421.2980.7670.531
GPF (ours)0.5750.9350.2670.6681.3140.4190.895
HR source––0.0491.000–0.3541.000
HR Gaussian––0.0500.908–0.3540.997
LR bicubic––1.0000.046–1.0000.354

MSRS

MethodQ_abfVIF ΣVIF IRVIF VISSSIM ΣSSIM IRSSIM VIS
DIDFuse0.2230.3070.1150.1920.7010.3180.383
SeAFusion0.6040.8390.1450.6941.3300.3510.979
CDDFuse0.6040.9040.1350.7691.3180.3360.982
AGMFusion0.5670.8070.2150.5921.3060.3540.952
MFDFuse0.6120.8970.1900.7071.3370.3660.971
MLFuse0.5560.7180.1790.5391.2830.3750.908
TDFusion0.6280.9310.1560.7751.3330.3530.980
Deno-IF0.5420.7680.2220.5461.3320.3740.958
TG-ECNet0.4940.5270.1480.3791.1820.3240.858
RIS-Fuse0.6310.9460.1680.7781.3430.3520.991
LUT-Fuse0.5310.7010.1980.5031.3140.3860.928
TITA0.6200.8720.1830.6891.3510.3920.959
MaeFuse0.5020.6710.2210.4501.3140.4370.877
FDFuse0.6160.9480.2020.7461.3260.3550.971
MFS-Fusion0.6200.9300.1440.7861.3520.3540.998
SETFusion0.1910.2940.1370.1571.1640.5250.639
INR-Fuse0.2150.4660.3980.0681.1410.6360.505
GPF (ours)0.6590.9590.2070.7521.3670.4170.950
HR source––0.1311.000–0.3171.000
HR Gaussian––0.1310.992–0.3171.000
LR bicubic––1.0000.027–1.0000.317
Reviewer_LjwGOpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Gaussian Primitive Fusion (GPF) for fusing multimodal images with different resolutions. Rather than resizing all images to the same size, it uses shared adaptive Gaussian geometry to represent the inputs, while keeping the image content specific to each modality. It then learns how to combine information at different scales to produce a fused image at the desired resolution, without having to optimize the representation again. Experiments on several infrared-visible fusion datasets show that GPF performs well in heterogeneous-resolution settings.

Critical strengths
  1. The paper focuses on heterogeneous-resolution multimodal fusion, which is a practical problem. Simply resizing images to the same resolution may cause information loss or introduce artifacts.
  2. The Gaussian representation provides a unified way to handle images with different resolutions. It allows the method to use the original observations while generating fused images at different target resolutions.
  3. The experiments cover several datasets and compare GPF with existing fusion methods using different interpolation or super-resolution frontends. The ablation studies also examine the effects of native-resolution constraints, shared geometry, and multiscale fusion weights.
Critical weaknesses
  1. It is not entirely clear how much of the improvement comes from the Gaussian representation itself and how much comes from the fusion strategy. The paper uses the best tested frontend for each competing method, but the details of how these frontends are selected and tuned are not clear enough.
  2. GPF optimizes the Gaussian representation separately for each image pair, which may affect its practical efficiency. However, the paper does not provide enough information about runtime and memory usage for different image sizes and resolution ratios.
  3. Most of the evaluation focuses on infrared-visible image fusion, while the medical image experiments are relatively limited. It is also unclear how well GPF handles spatial misalignment, sensor noise, and other differences between real-world image sources.
Questions for authors
  1. How are the frontend configurations for competing methods selected? Could the authors provide comparisons under more consistent frontend settings to clarify whether the improvements mainly come from the Gaussian representation or the fusion strategy?
  2. Since GPF optimizes the representation for each image pair, how much time and memory does it require for images of different sizes and resolution ratios? A comparison with the competing methods would help clarify its practical efficiency.
  3. The method assumes that the source images are registered. How does its performance change when the images contain spatial misalignment or sensor noise? Would additional alignment or degradation modeling be needed to handle these cases?
3 comments
AuthorsOpenReview ↗

Response (1/3): frontend selection, source of the improvement, runtime and memory

We thank you for your time and effort in reviewing our paper and providing feedback. Please find our responses to the concerns you have raised below. All new figures (Figs. R1–R7) and tables are included in the appendix of the revised paper.

W1 & Q1. Frontend selection and the source of the improvement

(1) Frontend selection. We thank the reviewer for this comment, and we apologize that our description was unclear. GPF itself does not use any frontend. Frontends are used only for the compared methods, which require both inputs on a common grid, and GPF outperforms each of them even when it uses its best frontend.

(a) GPF fits a shared Gaussian geometry and modality-specific content directly to the source images at their original resolutions (Sec. 3.2), learns the fusion weights on the same Gaussians, and renders the fused image at the desired resolution.

(b) For each compared method, we tested interpolation frontends (nearest, bilinear, bicubic and Lanczos) and super-resolution frontends (HAT, SwinIR, Swin2SR and LIIF, with their official pretrained weights), and selected the one under which the fused image has the highest sum of SSIM values with the two sources (S_Σ). Each compared method is therefore evaluated with its strongest frontend, so the choice of frontend does not favor GPF. We will describe this selection in the revised paper.

(2) Source of the improvement. Both the Gaussian representation and the fusion strategy contribute to the improvement: in the ablations of Table 2, each of them accounts for a gain of about 0.05 in Q_abf.

(a) For the representation, fitting the lower-resolution source at its native resolution instead of its bicubically enlarged version raises Q_abf from 0.529 to 0.575 (Table 2A). Fig. 7 further keeps the native observations and the fusion rule fixed and changes only the representation: adaptive Gaussians outperform linear grids, cubic B-splines, SIREN and fixed Gaussians of comparable capacity.

(b) For the fusion strategy, defining the fusion weights on the Gaussian primitives instead of the pixel grid raises Q_abf from 0.530 to 0.575 (Table 2C).

(c) The compared methods, in contrast, depend strongly on the frontend: changing only the frontend noticeably changes their Q_abf (Fig. 5), so they are limited by the quality of the resampled inputs.

W2 & Q2. Runtime and memory

(1) Runtime. We thank the reviewer for this suggestion. At every image size, GPF is faster than at least 12 of the 17 compared methods (the 16 methods in the paper and INR-Fuse), even without counting their frontends.

(a) We add the runtime of all methods at five image sizes to the revised appendix (Table R1). On an RTX 3090, GPF fuses one image pair in about 0.03 s at 256×256 and about 0.26 s at 1280×1024.

(b) INR-Fuse, which also optimizes each image pair, takes about 100 s at every size, so GPF is about 380 times faster at 1280×1024.

(c) GPF requires no training data, and rendering at another output resolution reuses the fitted parameters without refitting.

Table R1. Runtime (ms) per image pair of all methods on an RTX 3090 for different image sizes. The runtime of the compared methods does not include their frontends. The peak GPU memory is reported in Fig. 8(b) of the main paper. Bold marks the fastest method at each size.

Method256×256512×512768×7681024×10241280×1024
DIDFuse6.622.146.782.7108.0
SeAFusion78.097.3129.6176.3209.9
CDDFuse128.3286.0555.2932.11150.2
AGMFusion4.212.426.239.556.5
MFDFuse124.0484.11103.81962.22461.7
MLFuse7.017.739.064.482.7
TDFusion45.8146.9333.3588.8742.7
Deno-IF64.6207.1466.6822.41028.8
TG-ECNet312.61057.52373.34230.75318.9
RIS-Fuse70.5175.6368.2632.9789.2
LUT-Fuse5.912.925.545.557.9
TITA924.73780.29033.115825.019775.3
MaeFuse267.5271.6274.3286.2290.7
FDFuse49.5177.2406.7722.5905.1
MFS-Fusion170.6388.7853.21511.01903.9
SETFusion78.6291.4689.51206.21515.1
INR-Fuse98121.597640.2100293.1116285.0100348.7
GPF (ours)29.775.4127.1208.6263.1

(2) Memory. The peak GPU memory of GPF is lower than that of all methods in Fig. 8(b) of the paper.

AuthorsOpenReview ↗

Response (2/3): real-world data and misalignment

W3 & Q3. Real-world data, misalignment and noise

(1) Real-world data. We thank the reviewer for raising these practical issues.

(a) The PET–MRI pairs have natively different resolutions and require no synthetic degradation, so they test GPF on real data.

(b) For infrared–visible fusion, to our knowledge, no public dataset currently provides registered pairs with natively different resolutions, so we construct heterogeneous inputs by downsampling with different ratios, following common practice in super-resolution.

(2) Misalignment. Under a 32-pixel offset between the two modalities, GPF still achieves the best or tied-best downstream results among all fusion methods (Tables R11 and R12), and it does so without an additional alignment step.

Most existing fusion methods, including all compared methods, assume registered inputs, and Table 1 follows the same setting. To test misalignment, we offset the two modalities by 32 pixels in a random direction (up, down, left or right) on the high-resolution grid, which corresponds to 2 pixels of the ×16 low-resolution input, and evaluate the fused images on segmentation and detection, since misalignment makes source-referenced metrics unreliable.

(a) In segmentation with VIS×16, GPF reaches an mIoU of 32.13, whereas the compared methods reach at most 28.09.

(b) In detection with IR×16, GPF reaches an AP50:95 of 20.62, whereas the compared methods reach at most 15.72.

The smooth and spatially extended support of the Gaussians contributes to this: an offset changes a Gaussian field only gradually, so the fitted sources and the fusion weights change smoothly rather than abruptly.

Table R11. Semantic segmentation on MSRS with a U-Net: mIoU (%) over the car, person and bike classes. 0 px and 32 px denote the results without and with a 32-pixel offset between the two modalities in a random direction (up, down, left or right) on the high-resolution grid. Infrared and Visible use the corresponding input alone. The best result in each column is shown in bold.

MethodIR×16 0 pxIR×16 32 pxVIS×16 0 pxVIS×16 32 px
DIDFuse38.2632.7732.2121.14
SeAFusion42.3538.8744.7721.94
CDDFuse42.1537.9443.1919.85
AGMFusion44.8841.8944.9727.91
MFDFuse45.2941.1542.5325.38
MLFuse45.0941.1242.3325.19
TDFusion38.9832.3244.6124.67
Deno-IF45.6142.0944.2327.89
TG-ECNet43.7739.8241.0425.22
RIS-Fuse46.2243.5643.9527.09
LUT-Fuse44.7040.6543.1124.81
TITA45.1641.8241.8926.08
MaeFuse44.2639.3144.6228.09
FDFuse45.7441.7644.9827.36
MFS-Fusion36.9728.0744.6124.67
SETFusion38.8435.8137.1724.11
INR-Fuse32.6026.8336.8220.54
GPF (ours)47.6745.0245.3732.13
Infrared14.71–42.08–
Visible45.90–13.75–

Table R12. Object detection on M3FD with Faster R-CNN: AP50:95 (%) of the person class. 0 px and 32 px denote the results without and with a 32-pixel offset between the two modalities in a random direction (up, down, left or right) on the high-resolution grid. Infrared and Visible use the corresponding input alone. The best result in each column is shown in bold.

MethodIR×16 0 pxIR×16 32 pxVIS×16 0 pxVIS×16 32 px
DIDFuse10.118.8324.6325.26
SeAFusion9.384.6229.6023.54
CDDFuse5.933.4325.8322.65
AGMFusion17.4412.9528.7727.03
MFDFuse15.8010.3127.3224.01
MLFuse14.5113.1028.3927.63
TDFusion7.861.9028.8326.67
Deno-IF20.6215.7228.5127.19
TG-ECNet14.697.8826.0123.47
RIS-Fuse5.946.9219.2317.49
LUT-Fuse11.266.4323.0621.25
TITA17.0512.4426.7623.40
MaeFuse14.525.9527.6726.54
FDFuse13.429.2129.4325.79
MFS-Fusion7.162.3627.3424.96
SETFusion12.059.2622.5922.82
INR-Fuse0.100.308.520.43
GPF (ours)26.8220.6230.4027.63
Infrared7.27–29.37–
Visible24.26–4.17–
AuthorsOpenReview ↗

Response (3/3): robustness to sensor noise

W3 & Q3 (continued). Noise

(3) Noise. GPF is robust to sensor noise without additional degradation modeling: when Gaussian noise is added to both the infrared image and the downsampled visible image, it is the only method that ranks in the top two on all three metrics at both noise levels (Table R2). This robustness comes from how GPF fits the sources.

(a) In the high-resolution infrared image, each Gaussian covers many pixels, so pixel-wise noise is averaged out.

(b) The low-resolution visible image is fitted at its native resolution instead of being enlarged together with its noise.

(c) The adaptive shapes of the Gaussians follow structural edges, so these edges are preserved while the noise is averaged out.

In Table R2, with a standard deviation of 20/255, GPF achieves the best Q_abf, VIF_Σ and S_Σ. With 40/255, it keeps the best S_Σ, 1.235 compared with at most 0.976 for the compared methods, and ranks second in Q_abf and VIF_Σ. Its S_Σ also decreases the least, by 6.6% from 20/255 to 40/255, compared with at least 18.9% for the compared methods.

Fig. R1 shows an example with a standard deviation of 35/255. Most compared methods enlarge the noise together with the image during resolution conversion and fusion, which produces speckles and blotches over the road and the vehicles, whereas GPF keeps the road surface clean and the car and the scooter sharp.

Table R2. Fusion results with Gaussian noise added to both the infrared image and the downsampled visible image, with standard deviations σ of 20/255 and 40/255. The best result in each column is shown in bold.

Methodσ = 20/255 Q_abfσ = 20/255 VIF_Σσ = 20/255 S_Σσ = 40/255 Q_abfσ = 40/255 VIF_Σσ = 40/255 S_Σ
DIDFuse0.1610.1590.6260.1240.1160.459
SeAFusion0.2370.2150.5010.1560.1360.252
CDDFuse0.2280.1900.3740.1480.1120.148
AGMFusion0.2440.2420.6070.1730.1600.329
MFDFuse0.2320.1860.4370.1560.1180.184
MLFuse0.2230.2030.6560.1570.1330.331
TDFusion0.2350.1920.4150.1520.1180.185
Deno-IF0.2600.2690.8550.1780.1770.485
TG-ECNet0.2650.3021.1100.2110.2370.861
RIS-Fuse0.2380.2020.3870.1570.1180.175
LUT-Fuse0.2010.2150.7660.1490.1450.392
TITA0.2410.2090.4780.1580.1240.182
MaeFuse0.1590.2590.9040.1280.1880.541
FDFuse0.2330.1890.3340.1520.1110.142
MFS-Fusion0.2350.1980.3930.1480.1240.168
SETFusion0.1420.1330.6990.1160.1010.393
INR-Fuse0.0500.1681.2050.0600.1710.976
GPF (ours)0.2790.3551.3220.1920.1891.235

Comments on the paper

5 comments

Dear Reviewers,

The authors have posted their rebuttal responses. I would kindly encourage you to review the authors' responses and continue the discussion by providing follow-up comments where appropriate. In particular, please indicate whether the authors' responses have adequately addressed your concerns and update your assessment if necessary.

Thank you for your time and continued efforts!

Your AC

Dear Reviewer uLGf,

There are fewer than four days left in the discussion period. The authors have submitted their rebuttal, but we have not yet received your feedback.

Please read the rebuttal and let the authors know if it answers your concerns. If you still have concerns, please share them soon so the authors have enough time to respond.

Thank you for your help.

Your AC

AuthorsOpenReview ↗

Summary of our responses

We sincerely thank you for your constructive review and for recognizing the practical significance of heterogeneous-resolution fusion, the unified Gaussian representation built directly on the original observations, and our experiments across multiple datasets, frontends and ablations. Your comments have helped us improve the paper, in particular the clarity of the comparison setting and the evaluation of efficiency and robustness. The main revisions are summarized below, and all new results are included in the revised paper:

  1. Frontend selection and source of the improvement: We describe how the strongest interpolation or super-resolution frontend was selected for each compared method, so the choice of frontend does not favor GPF. The ablations show that both the Gaussian representation and the fusion strategy contribute to the improvement.
  2. Runtime and memory: We report the runtime of GPF and 17 compared methods at multiple image sizes. GPF is faster than at least 12 of them at every size, and it also uses less peak GPU memory than the compared methods we measured.
  3. Misalignment: We evaluate downstream segmentation and detection under offsets between the modalities. Without an additional alignment step, GPF still achieves the best or tied-best results among all fusion methods.
  4. Sensor noise: We test Gaussian noise at different levels. Without additional degradation modeling, GPF is the only method that ranks in the top two on all three metrics at every tested noise level.

In addition, we have added further results to the revised paper:

  1. Role of the shared Gaussians: All variants that use separate Gaussians for the two modalities (even twice as many in total) or place the fusion weights on pixels or a fixed grid lower the fusion quality, showing that the gain comes from coupling reconstruction and fusion weights on the shared Gaussians.
  2. Additional metrics and overall ranking: We add four metrics, and over nine criteria including the downstream tasks, GPF ranks first on seven, showing that its advantage is not limited to the original metrics.
  3. Frozen versus joint optimization: Jointly optimizing the fitted representation with the fusion weights needs about five times as many steps and still falls behind GPF on Q_abf, VIF_Σ, S_Σ, Q_CB and N_abf, so freezing it improves both quality and efficiency.
  4. Visualization of the fusion weights: With the Gaussians, GPF learns the fusion on the image structures rather than on a fixed pixel grid. In Fig. R9, its weights stay consistent over the whole pedestrian and follow its contour, whereas the weights of other fusion methods vary from pixel to pixel and form square blocks.

We very much look forward to continuing the discussion with you to further improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.