- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes T3 Fusion, a test-time training framework for task-general image fusion. The method learns bidirectional cross-source relations from each test image pair using closed-form ridge regression, while keeping the main network parameters fixed. A learned fusion readout uses the cross-source predictions, prediction residuals, and original features to refine the fused representation. The model is trained on MSRS and evaluated on unseen infrared-visible datasets as well as multi-exposure, multi-focus, medical, and polarization fusion tasks. The experiments also include downstream segmentation and detection to evaluate whether the fused images preserve useful information.
Critical strengths
- The paper introduces a clear test-time learning mechanism for image fusion. Pair-specific cross-source relations are fitted with closed-form regression, so the model can adapt to each input pair without test-time backpropagation or fused targets.
- The same MSRS-trained model transfers to several unseen fusion tasks. This gives strong evidence that the learned fusion process is not limited to one modality pair or one dataset.
Critical weaknesses
- Table 4 lists only small changes in the fusion metrics when current-pair relations replace shuffled relations. Table 5 also shows very similar results for static relations, the learned generator, and T3 Fusion on several datasets. The contribution of pair-specific closed-form fitting is therefore difficult to separate from the offline fusion network and readout.
Questions for authors
- Table 4 lists large reductions in held-out prediction error, while the fusion metrics change only slightly. How much do the pair-specific relations actually affect the final fused output?
- Does the cross-source fitting require pixel-level alignment between the two input images, and what spatial correspondence is assumed during inference?
4 comments
Q3. What spatial correspondence does cross-source fitting require between the two input images?
Response:
Cross-source fitting assumes corresponding locations on pre-aligned feature grids. A key from one source is paired with the target at the same coordinate in the other source. We reverse the sources for the other prediction direction, and the spatial folds keep each query location out of the fit used to predict it.
The 48 evaluations comprise 16 conditions × 3 training seeds. The conditions are 2 datasets (FMB, RoadScene) × 2 shift magnitudes (0.5, 1 pixel) × 2 shifted-source choices (infrared or visible, one at a time) × 2 axes (horizontal, vertical). Each condition is evaluated with seeds 3407, 3408, and 3409. Each evaluation summarizes test-image results for one condition and seed. The count excludes zero-shift inputs. The results are:
| Dataset | Shift (px) | MSE drop (%) | VIF | |
|---|---|---|---|---|
| FMB | 0.5 | 34.51 | +11.451 | +2.445 |
| FMB | 1 | 33.48 | +9.315 | +1.514 |
| RoadScene | 0.5 | 31.00 | +16.633 | +3.947 |
| RoadScene | 1 | 30.18 | +14.790 | +3.198 |
MSE reductions compare Current with Shuffle. Fusion gains compare Full with Base and are multiplied by . Each table row averages 12 evaluations at a fixed dataset and shift magnitude: 2 shifted-source choices × 2 axes × 3 training seeds.
Current reduces query MSE in 48/48 evaluations, with reductions of 24.4%-40.3%. In these same 48 evaluations, Full improves VIF over Base in 48/48 evaluations and in 44/48. Both tested shifts retain positive mean prediction and fusion gains. Larger displacement or nonrigid deformation requires image registration or a correspondence method to establish the shared grid.
All evaluations use a common eight-pixel valid crop and clean source references. The revised Sec. 3.2 (p. 4) clarifies the spatial-correspondence assumption, and Appendix G.4 (p. 23) describes the alignment tests in detail.
Thanks for the additional experiments and explanations. The new controls help clarify the contribution of pair-specific fitting, and the alignment assumption is now clear. While the gains remain modest, the response addresses my main concerns. I’m keeping my original score.
Thank you very much for your careful review and constructive feedback. We appreciate that our additional experiments and clarifications have addressed your main concerns.
Thank you for your careful review. We address each concern below with the relevant explanation and evidence.
Q1. The contribution of fitting relations for each pair needs to be separated from that of the fusion network and readout.
Response:
We separate the fitted maps, the readout, and learned map generation through three controls.
1. Change the maps with the network fixed. Current fits the 24 maps to the test pair. Shuffle replaces them with maps from other pairs, using identical inputs, network weights, and readout. Current improves Qabf in 24/24 evaluations (8 datasets × 3 training seeds: 3407, 3408, 3409). Each compares test-set mean scores for one dataset and seed. Current also improves VIF and Qabf in 54/54 noise evaluations. The 54 evaluations are 3 datasets (FMB, M3FD, RoadScene) × 2 noisy-source choices (infrared or visible, one at a time) × 3 training seeds (3407, 3408, 3409) × 3 noise realizations at σ=15/255. Each evaluation averages over the test images for one such combination. This isolates the maps' contribution within a fixed fusion model.
2. Retain fitting and change the readout. Full improves Qabf over a parameter-matched Shared reader in 24/24 evaluations, again 8 datasets × 3 training seeds. Representative Full-minus-Shared mean gains, scaled by 103, are:
These three-seed averages cover infrared-visible, multi-focus, and medical fusion. They test how the fitted predictions are converted into fusion updates.
3. Match the size of a learned map generator. Full and the generator follow the same training schedule, with 70,433 and 70,444 adaptation-module parameters. Representative Full-minus-generator results are:
These are individual trained-model comparisons on complete test sets. FMB infrared noise uses σ=15/255, with three noise realizations averaged. Full improves Qabf by 2.62% in that FMB example and 1.44% on clean DoLP-S0. Across the evaluated runs, mean Qabf favors Full on seven of eight clean datasets, and mean VIF favors Full in all eight noisy conditions: 4 datasets (FMB, M3FD, RoadScene, MFI-WHU) × 2 choices of which input source receives noise. These eight comparisons use means over the evaluated training seeds and noise realizations, rather than counting each repeat separately.
The fixed-network comparison isolates fitting, while the matched readout and generator controls test its use and an alternative implementation. The revised manuscript analyzes these controls alongside Tables 4-6 (pp. 9-10) and S9-S12 (pp. 22-23).
Q2. How much do the fitted relations change the final fused image and its quality?
Response:
The fitted relations produce measurable output corrections and improve fusion quality across all eight datasets. The table summarizes the range of dataset means across the eight datasets and gives RoadScene as a concrete example:
Output MAE uses the 0-255 luminance scale, and query MSE reductions compare Current with Shuffle. Because zero predictions exactly recover Base, the comparison with zero maps measures the correction produced by the relation branch.
These corrections improve fusion quality across individual images. Based on per-image gains averaged across runs, Current has higher Qabf than Shuffle for 58.7%-91.4% of images, and the mean, median and 10% trimmed mean gains are positive on all eight datasets. For the tested checkpoints and assignments of shuffled maps, the paired-image 95% intervals are also above zero on all eight datasets. The positive gains, medians and trimmed means show that the corrections are useful across the image set, linking better prediction to improved final fusion.
Output MAE measures the size of a correction, and positive fusion-score differences measure its benefit. The revised manuscript explains the link between output changes and fusion gains alongside Table 4 (p. 9) and Tables S9 and S11 (p. 22).