- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work targets multi-modal image fusion under mismatched input resolutions, a common practical scenario where standard methods rely on pre-resampling and introduce interpolation artifacts. The authors propose GPF, a continuous representation built from 2D Gaussian primitives. The pipeline fits Gaussian parameters from raw source images at their native resolution in stage one, then only optimizes multi-scale fusion weights on these fixed primitives in stage two. Experiments are conducted on infrared-visible and PET-MRI datasets, with comparisons against many existing fusion approaches.
Critical strengths
- The problem setting is practically meaningful. Most existing multi-modal fusion methods assume perfectly aligned inputs of identical resolution. The proposed Gaussian primitive representation avoids upfront resampling of source images, which directly addresses the interpolation artifacts introduced by pre-upsampling low-resolution inputs.
- The overall framework is conceptually novel. Using 2D Gaussians as a continuous scene representation for fusion, instead of discrete pixel grids or coordinate MLPs, provides an alternative way to handle arbitrary output resolution. The authors perform extensive quantitative and qualitative tests including non-uniform downsampling cases.
Critical weaknesses
- There are too few evaluation indicators. More evaluation indicators are needed to measure the performance of the model. 2.The fusion result of the model is not competitive.
Questions for authors
1.There are too few evaluation metrics; more metrics need to be provided, like EN, AG, etc. Three metrics aren't enough to assess the model's performance. 2. From a visual perspective, the fusion results aren’t as good as a single image. The author needs to evaluate the model’s effectiveness on downstream tasks to verify if the model has any advantages in computer vision. Whether there’s an advantage in downstream tasks directly determines the significance of this work. 3.Why did you choose to freeze Gaussian geometry after stage 1 instead of jointly optimizing Gaussian parameters together with fusion weights? Can you add this baseline and compare the performance? 4.All experiments assume perfectly registered source images. How much performance drop will occur with small registration offsets between two modalities? Could you provide results under misalignment perturbations?
10 comments
Response (2/5): additional metrics (continued)
W1 & Q1 (continued)
Table R8 (continued).
MSRS
| Method | Q_abf↑ | VIF_Σ↑ | S_Σ↑ | Q_CB↑ | N_abf↓ | EN↑ | AG↑ |
|---|---|---|---|---|---|---|---|
| DIDFuse | 0.079 | 0.213 | 0.706 | 0.376 | 0.0075 | 4.54 | 1.01 |
| SeAFusion | 0.260 | 0.436 | 1.167 | 0.355 | 0.0252 | 6.78 | 2.40 |
| CDDFuse | 0.235 | 0.421 | 1.157 | 0.346 | 0.0258 | 6.81 | 2.20 |
| AGMFusion | 0.246 | 0.537 | 1.154 | 0.332 | 0.0117 | 6.90 | 2.20 |
| MFDFuse | 0.202 | 0.408 | 1.164 | 0.270 | 0.0069 | 6.84 | 1.58 |
| MLFuse | 0.209 | 0.434 | 1.175 | 0.284 | 0.0069 | 6.58 | 1.66 |
| TDFusion | 0.286 | 0.580 | 1.207 | 0.375 | 0.0204 | 6.69 | 2.25 |
| Deno-IF | 0.196 | 0.460 | 1.181 | 0.309 | 0.0075 | 6.78 | 1.59 |
| TG-ECNet | 0.131 | 0.189 | 1.044 | 0.270 | 0.0075 | 6.49 | 1.71 |
| RIS-Fuse | 0.216 | 0.406 | 1.164 | 0.278 | 0.0067 | 6.78 | 1.63 |
| LUT-Fuse | 0.205 | 0.451 | 1.160 | 0.281 | 0.0072 | 6.63 | 1.81 |
| TITA | 0.188 | 0.375 | 1.207 | 0.274 | 0.0040 | 6.61 | 1.38 |
| MaeFuse | 0.199 | 0.451 | 1.209 | 0.366 | 0.0066 | 6.59 | 1.76 |
| FDFuse | 0.277 | 0.580 | 1.148 | 0.322 | 0.0119 | 6.87 | 2.10 |
| MFS-Fusion | 0.273 | 0.501 | 1.210 | 0.371 | 0.0191 | 6.69 | 2.19 |
| SETFusion | 0.130 | 0.329 | 1.174 | 0.300 | 0.0054 | 5.27 | 1.21 |
| GPF (ours) | 0.353 | 0.983 | 1.338 | 0.519 | 0.0018 | 6.73 | 2.06 |
M3FD
| Method | Q_abf↑ | VIF_Σ↑ | S_Σ↑ | Q_CB↑ | N_abf↓ | EN↑ | AG↑ |
|---|---|---|---|---|---|---|---|
| DIDFuse | 0.257 | 0.597 | 1.214 | 0.386 | 0.0126 | 7.05 | 1.92 |
| SeAFusion | 0.270 | 0.480 | 1.306 | 0.321 | 0.0167 | 6.80 | 2.20 |
| CDDFuse | 0.241 | 0.403 | 1.288 | 0.313 | 0.0140 | 6.90 | 1.90 |
| AGMFusion | 0.281 | 0.613 | 1.279 | 0.330 | 0.0111 | 6.94 | 2.12 |
| MFDFuse | 0.255 | 0.532 | 1.279 | 0.311 | 0.0149 | 7.01 | 1.84 |
| MLFuse | 0.290 | 0.574 | 1.354 | 0.353 | 0.0043 | 6.79 | 1.72 |
| TDFusion | 0.327 | 0.686 | 1.362 | 0.405 | 0.0123 | 7.02 | 2.07 |
| Deno-IF | 0.255 | 0.572 | 1.365 | 0.365 | 0.0105 | 6.82 | 1.72 |
| TG-ECNet | 0.168 | 0.287 | 1.243 | 0.321 | 0.0023 | 6.79 | 1.47 |
| RIS-Fuse | 0.186 | 0.237 | 1.301 | 0.227 | 0.0059 | 6.43 | 1.28 |
| LUT-Fuse | 0.186 | 0.375 | 1.307 | 0.222 | 0.0049 | 6.54 | 1.45 |
| TITA | 0.235 | 0.470 | 1.366 | 0.298 | 0.0014 | 6.46 | 1.26 |
| MaeFuse | 0.208 | 0.476 | 1.368 | 0.396 | 0.0031 | 6.84 | 1.55 |
| FDFuse | 0.290 | 0.544 | 1.320 | 0.329 | 0.0226 | 6.93 | 2.23 |
| MFS-Fusion | 0.311 | 0.619 | 1.356 | 0.362 | 0.0055 | 6.76 | 1.90 |
| SETFusion | 0.215 | 0.537 | 1.273 | 0.370 | 0.0313 | 7.00 | 2.34 |
| GPF (ours) | 0.376 | 0.960 | 1.420 | 0.515 | 0.0005 | 6.89 | 2.01 |
Table R9. Q_abf, VIF_Σ, S_Σ, Q_CB, N_abf, EN and AG of GPF and the 16 compared methods of the paper in the VIS×16 setting on RoadScene and TNO, with the same layout as Table R8. The best result in each column is shown in bold.
RoadScene
| Method | Q_abf↑ | VIF_Σ↑ | S_Σ↑ | Q_CB↑ | N_abf↓ | EN↑ | AG↑ |
|---|---|---|---|---|---|---|---|
| DIDFuse | 0.494 | 0.610 | 1.206 | 0.431 | 0.0254 | 7.30 | 4.52 |
| SeAFusion | 0.497 | 0.628 | 1.237 | 0.451 | 0.0466 | 7.37 | 5.45 |
| CDDFuse | 0.498 | 0.601 | 1.259 | 0.440 | 0.0415 | 7.43 | 4.88 |
| AGMFusion | 0.423 | 0.565 | 1.215 | 0.375 | 0.0272 | 7.12 | 4.61 |
| MFDFuse | 0.509 | 0.684 | 1.236 | 0.435 | 0.0356 | 7.45 | 4.84 |
| MLFuse | 0.485 | 0.595 | 1.307 | 0.420 | 0.0029 | 7.13 | 3.40 |
| TDFusion | 0.565 | 0.752 | 1.280 | 0.467 | 0.0358 | 7.42 | 5.04 |
| Deno-IF | 0.454 | 0.597 | 1.298 | 0.475 | 0.0191 | 7.30 | 4.03 |
| TG-ECNet | 0.167 | 0.202 | 1.087 | 0.361 | 0.0163 | 7.02 | 2.50 |
| RIS-Fuse | 0.383 | 0.325 | 1.265 | 0.309 | 0.0190 | 6.98 | 3.64 |
| LUT-Fuse | 0.424 | 0.544 | 1.296 | 0.398 | 0.0123 | 6.94 | 3.41 |
| TITA | 0.529 | 0.690 | 1.358 | 0.423 | 0.0044 | 7.06 | 3.79 |
| MaeFuse | 0.480 | 0.568 | 1.333 | 0.482 | 0.0077 | 7.09 | 3.86 |
| FDFuse | 0.537 | 0.667 | 1.266 | 0.449 | 0.0422 | 7.35 | 5.23 |
| MFS-Fusion | 0.615 | 0.846 | 1.331 | 0.512 | 0.0121 | 7.25 | 4.78 |
| SETFusion | 0.366 | 0.422 | 1.176 | 0.365 | 0.0437 | 7.01 | 4.61 |
| GPF (ours) | 0.656 | 0.993 | 1.380 | 0.582 | 0.0001 | 7.26 | 4.56 |
TNO
| Method | Q_abf↑ | VIF_Σ↑ | S_Σ↑ | Q_CB↑ | N_abf↓ | EN↑ | AG↑ |
|---|---|---|---|---|---|---|---|
| DIDFuse | 0.308 | 0.525 | 1.130 | 0.384 | 0.0308 | 6.95 | 2.53 |
| SeAFusion | 0.361 | 0.583 | 1.212 | 0.384 | 0.0485 | 7.17 | 3.22 |
| CDDFuse | 0.349 | 0.534 | 1.225 | 0.377 | 0.0316 | 7.20 | 2.77 |
| AGMFusion | 0.337 | 0.572 | 1.198 | 0.351 | 0.0358 | 7.05 | 3.04 |
| MFDFuse | 0.348 | 0.637 | 1.190 | 0.371 | 0.0369 | 7.30 | 2.85 |
| MLFuse | 0.344 | 0.552 | 1.257 | 0.389 | 0.0123 | 7.00 | 2.29 |
| TDFusion | 0.412 | 0.672 | 1.258 | 0.413 | 0.0328 | 7.21 | 2.95 |
| Deno-IF | 0.288 | 0.509 | 1.256 | 0.400 | 0.0150 | 7.16 | 2.15 |
| TG-ECNet | 0.184 | 0.280 | 1.135 | 0.355 | 0.0158 | 6.86 | 1.94 |
| RIS-Fuse | 0.268 | 0.346 | 1.223 | 0.302 | 0.0129 | 6.98 | 2.01 |
| LUT-Fuse | 0.304 | 0.492 | 1.233 | 0.328 | 0.0101 | 6.78 | 2.14 |
| TITA | 0.342 | 0.570 | 1.286 | 0.362 | 0.0043 | 6.79 | 1.96 |
| MaeFuse | 0.308 | 0.514 | 1.290 | 0.437 | 0.0090 | 6.87 | 2.22 |
| FDFuse | 0.376 | 0.593 | 1.222 | 0.389 | 0.0401 | 7.25 | 3.06 |
| MFS-Fusion | 0.436 | 0.721 | 1.291 | 0.442 | 0.0134 | 6.94 | 2.71 |
| SETFusion | 0.288 | 0.491 | 1.178 | 0.368 | 0.0589 | 6.80 | 2.94 |
| GPF (ours) | 0.492 | 0.938 | 1.321 | 0.526 | 0.0054 | 7.13 | 2.89 |
Response (3/5): overall ranking and visual quality
W1 & Q1 (continued)
Table R10. Ranks of GPF and the 16 compared methods of the paper in the VIS×16 setting, where 1 is the best. The image metrics are averaged over the five datasets before ranking, the downstream tasks use the results without offset (Tables R11 and R12), and tied methods share the average rank. INR-Fuse is not included because Q_CB, N_abf, EN and AG were computed only for the 16 methods of the paper.
| Reference-based | No-reference | Downstream tasks | ||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| Method | Q_abf | VIF_Σ | S_Σ | Q_CB | N_abf | EN | AG | Seg. | Det. | Mean rank |
| GPF (ours) | 1 | 1 | 1 | 1 | 1 | 8 | 7 | 1 | 1 | 2.44 |
| TDFusion | 3 | 3 | 5 | 4 | 13 | 3 | 3 | 6.5 | 4 | 4.94 |
| MFS-Fusion | 2 | 2 | 3 | 2 | 8 | 9 | 6 | 6.5 | 9 | 5.28 |
| FDFuse | 4 | 4 | 10 | 8 | 15 | 2 | 2 | 2 | 3 | 5.56 |
| SeAFusion | 5 | 9 | 12 | 7 | 17 | 7 | 1 | 4 | 2 | 7.11 |
| AGMFusion | 9 | 5 | 14 | 11 | 11 | 5 | 4 | 3 | 5 | 7.44 |
| Deno-IF | 11 | 8 | 6 | 6 | 9 | 6 | 10 | 8 | 6 | 7.78 |
| MaeFuse | 13 | 13 | 4 | 3 | 4 | 11 | 11 | 5 | 8 | 8.00 |
| MFDFuse | 7 | 6 | 13 | 10 | 12 | 1 | 8 | 12 | 10 | 8.78 |
| CDDFuse | 6 | 11 | 11 | 9 | 14 | 4 | 5 | 10 | 13 | 9.22 |
| MLFuse | 10 | 10 | 7 | 12 | 3 | 10 | 13 | 13 | 7 | 9.44 |
| TITA | 8 | 7 | 2 | 13 | 2 | 13 | 16 | 14 | 11 | 9.56 |
| LUT-Fuse | 12 | 12 | 8 | 16 | 5 | 15 | 14 | 11 | 15 | 12.00 |
| RIS-Fuse | 14 | 16 | 9 | 17 | 7 | 14 | 15 | 9 | 17 | 13.11 |
| DIDFuse | 15 | 15 | 17 | 5 | 10 | 17 | 12 | 17 | 14 | 13.56 |
| TG-ECNet | 17 | 17 | 16 | 15 | 6 | 12 | 17 | 15 | 12 | 14.11 |
| SETFusion | 16 | 14 | 15 | 14 | 16 | 16 | 9 | 16 | 16 | 14.67 |
W2 & Q2. Visual quality compared with a single image, and downstream detection and segmentation
(1) Visual quality compared with a single image. We thank the reviewer for this observation. A fused image is not meant to reproduce the higher-resolution source, but to add the complementary content of the other source, such as thermal targets, while keeping the details of the higher-resolution source. Since the other source is low-resolution, this content is coarse, so a fused image can look different from the higher-resolution source alone. The additional comparisons with zoomed-in regions (Figs. R2–R5) show that GPF keeps these details while adding the complementary content. Quantitatively, GPF also has the best mean rank over the seven metrics and two downstream tasks in Table R10 (see W1 & Q1).
(a) In Fig. R2 (IR×16), the compared methods spread the upsampled infrared response into blurred bright regions around the pedestrian and over the ground, whereas GPF keeps a clear contour of the pedestrian, with distinguishable head and limbs, and the sharp structure of the traffic light.
(b) In Fig. R3 (VIS×16), GPF keeps the sharp edges of the car and the pedestrian from the infrared image and places the colors of the visible image within them, while several compared methods blur the colored regions beyond the object boundaries.
(c) In Fig. R4 (VIS×16), the compared methods produce over-bright regions and distorted shapes around the vehicle and the roadside, whereas GPF keeps the outlines of the vehicle and the pedestrians and the texture of the road.
(d) In Fig. R5 (VIS×32), the compared methods lose most visible structures and surround the pedestrians with bright halos, while GPF still shows the leaves, the building facade and the pedestrians clearly, with these structures from the infrared image and the coarse content from the visible image.
Response (1/5): additional metrics, including EN and AG
We thank you for your time and effort in reviewing our paper and providing feedback. Please find our responses to the concerns you have raised below. All new figures (Figs. R1–R7) and tables are included in the appendix of the revised paper.
W1 & Q1. Additional evaluation metrics, including EN and AG
We thank the reviewer for this suggestion. For GPF and the 16 compared methods of the paper in the VIS×16 setting, we add Q_CB and N_abf and report EN and AG (Tables R8 and R9). For a comprehensive comparison, we rank all methods on all metrics, including these four metrics, the three metrics of the paper and the two downstream tasks (Table R10): GPF ranks first on seven of the nine and has a mean rank of 2.4, compared with at least 4.9 for the other methods.
(a) GPF preserves the most contrast and introduces very few artifacts. Q_abf measures how well the edges of the sources are transferred to the fused image, Q_CB measures how well their perceptually important local contrast is preserved, based on a model of human vision, and N_abf measures the artifacts introduced by fusion, where lower is better. On all five datasets, GPF achieves the best Q_abf (Table C.4) and the best Q_CB, 4.8% to 37.9% higher than the second-best method. It also achieves the lowest N_abf on four datasets and the second-lowest on the remaining one, and its N_abf is below one fourth of the average of the compared methods on every dataset.
(b) GPF introduces few artifacts because it fuses the sources at their native resolutions without any resampling frontend, and its fusion weights vary smoothly within the Gaussian supports, which follow the image structures, so no abrupt transitions appear between regions taken from different sources. Its band-wise targets select the source with the stronger local response (Eq. (9)), which preserves the contrast of both sources and leads to its high Q_CB.
(c) EN and AG, in contrast, are computed from the fused image alone: EN measures the entropy of its intensity histogram, and AG its average gradient magnitude. They therefore also increase with noise and artifacts, such as the ringing, halos and spurious textures introduced by resampling and fusion. On these metrics, GPF remains competitive: its EN and AG are higher than the average of the compared methods on all five datasets, and its EN is within 4.4% of the best. Moreover, on the same dataset, every compared method with a higher EN or AG than GPF also has a higher N_abf and a lower Q_CB, so its higher values come with more artifacts and less preserved contrast.
Table R8. Q_abf, VIF_Σ, S_Σ, Q_CB, N_abf, EN and AG of GPF and the 16 compared methods of the paper in the VIS×16 setting on LLVIP, MSRS and M3FD. Q_abf, VIF_Σ and S_Σ follow Table C.4 of the paper. The best result in each column is shown in bold.
LLVIP