acceptodds
Under review as a conference paper at ICLR 2027

TFusion: Test-Time Training for Task-General Image Fusion

Abstract

Image fusion combines complementary observations into a more informative representation for subsequent perception and analysis. However, existing deep fusion models often generalize poorly to unseen datasets and tasks because fusion rules learned offline may not capture the relationships between sources in new inputs. To address this limitation, we propose TFusion, a test-time training framework that adapts fusion by learning cross-source relations from the current input pair during inference. Specifically, a fusion network extracts multiscale features from the two source images. We fit compact bidirectional mappings between these features through closed-form ridge regression so that each source predicts the other's features. A learned fusion readout combines these predictions and their residuals with the original features to refine the representations passed to the decoder for image reconstruction. The network weights remain fixed while the mappings are fitted to each test pair without fused ground truth or test-time backpropagation. Extensive experiments across five fusion tasks show that a single MSRS-trained model generalizes to unseen datasets and tasks without task-specific fine-tuning, achieving better fusion quality and downstream task performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Image Fusion, Test-Time Training, Deep Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xYhMOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes T3 Fusion, a test-time training framework for task-general image fusion. The method learns bidirectional cross-source relations from each test image pair using closed-form ridge regression, while keeping the main network parameters fixed. A learned fusion readout uses the cross-source predictions, prediction residuals, and original features to refine the fused representation. The model is trained on MSRS and evaluated on unseen infrared-visible datasets as well as multi-exposure, multi-focus, medical, and polarization fusion tasks. The experiments also include downstream segmentation and detection to evaluate whether the fused images preserve useful information.

Critical strengths
  1. The paper introduces a clear test-time learning mechanism for image fusion. Pair-specific cross-source relations are fitted with closed-form regression, so the model can adapt to each input pair without test-time backpropagation or fused targets.
  2. The same MSRS-trained model transfers to several unseen fusion tasks. This gives strong evidence that the learned fusion process is not limited to one modality pair or one dataset.
Critical weaknesses
  1. Table 4 lists only small changes in the fusion metrics when current-pair relations replace shuffled relations. Table 5 also shows very similar results for static relations, the learned generator, and T3 Fusion on several datasets. The contribution of pair-specific closed-form fitting is therefore difficult to separate from the offline fusion network and readout.
Questions for authors
  1. Table 4 lists large reductions in held-out prediction error, while the fusion metrics change only slightly. How much do the pair-specific relations actually affect the final fused output?
  2. Does the cross-source fitting require pixel-level alignment between the two input images, and what spatial correspondence is assumed during inference?
4 comments

Thank you for your careful review. We address each concern below with the relevant explanation and evidence.

Q1. The contribution of fitting relations for each pair needs to be separated from that of the fusion network and readout.

Response:

We separate the fitted maps, the readout, and learned map generation through three controls.

1. Change the maps with the network fixed. Current fits the 24 maps to the test pair. Shuffle replaces them with maps from other pairs, using identical inputs, network weights, and readout. Current improves QabfQ_{abf} in 24/24 evaluations (8 datasets × 3 training seeds: 3407, 3408, 3409). Each compares test-set mean scores for one dataset and seed. Current also improves VIF and QabfQ_{abf} in 54/54 noise evaluations. The 54 evaluations are 3 datasets (FMB, M3FD, RoadScene) × 2 noisy-source choices (infrared or visible, one at a time) × 3 training seeds (3407, 3408, 3409) × 3 noise realizations at σ=15/255\sigma=15/255. Each evaluation averages over the test images for one such combination. This isolates the maps' contribution within a fixed fusion model.

2. Retain fitting and change the readout. Full improves QabfQ_{abf} over a parameter-matched Shared reader in 24/24 evaluations, again 8 datasets × 3 training seeds. Representative Full-minus-Shared mean gains, scaled by 10310^3, are:

DatasetΔ\DeltaVIFΔQabf\Delta Q_{abf}ΔQcb\Delta Q_{cb}
RoadScene+6.469+4.223+2.297
MFI-WHU+0.265+3.133+3.896
PET-MRI+0.054+3.699+7.278

These three-seed averages cover infrared-visible, multi-focus, and medical fusion. They test how the fitted predictions are converted into fusion updates.

3. Match the size of a learned map generator. Full and the generator follow the same training schedule, with 70,433 and 70,444 adaptation-module parameters. Representative Full-minus-generator results are:

Dataset / inputΔ\DeltaVIFΔQabf\Delta Q_{abf}ΔQcb\Delta Q_{cb}
FMB / IR noise+0.012888+0.010916+0.003884
DoLP-S0 / clean+0.003917+0.009660+0.001932

These are individual trained-model comparisons on complete test sets. FMB infrared noise uses σ=15/255\sigma=15/255, with three noise realizations averaged. Full improves QabfQ_{abf} by 2.62% in that FMB example and 1.44% on clean DoLP-S0. Across the evaluated runs, mean QabfQ_{abf} favors Full on seven of eight clean datasets, and mean VIF favors Full in all eight noisy conditions: 4 datasets (FMB, M3FD, RoadScene, MFI-WHU) × 2 choices of which input source receives noise. These eight comparisons use means over the evaluated training seeds and noise realizations, rather than counting each repeat separately.

The fixed-network comparison isolates fitting, while the matched readout and generator controls test its use and an alternative implementation. The revised manuscript analyzes these controls alongside Tables 4-6 (pp. 9-10) and S9-S12 (pp. 22-23).

Q2. How much do the fitted relations change the final fused image and its quality?

Response:

The fitted relations produce measurable output corrections and improve fusion quality across all eight datasets. The table summarizes the range of dataset means across the eight datasets and gives RoadScene as a concrete example:

Evaluation scopeMAE vs zero mapsMAE vs ShuffleQuery MSE drop (%)
Eight-dataset range0.375-3.6560.231-0.8525.3-50.3
RoadScene3.6560.85230.3

Output MAE uses the 0-255 luminance scale, and query MSE reductions compare Current with Shuffle. Because zero predictions exactly recover Base, the comparison with zero maps measures the correction produced by the relation branch.

These corrections improve fusion quality across individual images. Based on per-image gains averaged across runs, Current has higher QabfQ_{abf} than Shuffle for 58.7%-91.4% of images, and the mean, median and 10% trimmed mean gains are positive on all eight datasets. For the tested checkpoints and assignments of shuffled maps, the paired-image 95% intervals are also above zero on all eight datasets. The positive gains, medians and trimmed means show that the corrections are useful across the image set, linking better prediction to improved final fusion.

Output MAE measures the size of a correction, and positive fusion-score differences measure its benefit. The revised manuscript explains the link between output changes and fusion gains alongside Table 4 (p. 9) and Tables S9 and S11 (p. 22).

Q3. What spatial correspondence does cross-source fitting require between the two input images?

Response:

Cross-source fitting assumes corresponding locations on pre-aligned feature grids. A key from one source is paired with the target at the same coordinate in the other source. We reverse the sources for the other prediction direction, and the spatial folds keep each query location out of the fit used to predict it.

The 48 evaluations comprise 16 conditions × 3 training seeds. The conditions are 2 datasets (FMB, RoadScene) × 2 shift magnitudes (0.5, 1 pixel) × 2 shifted-source choices (infrared or visible, one at a time) × 2 axes (horizontal, vertical). Each condition is evaluated with seeds 3407, 3408, and 3409. Each evaluation summarizes test-image results for one condition and seed. The count excludes zero-shift inputs. The results are:

DatasetShift (px)MSE drop (%)Δ\DeltaVIFΔQabf\Delta Q_{abf}
FMB0.534.51+11.451+2.445
FMB133.48+9.315+1.514
RoadScene0.531.00+16.633+3.947
RoadScene130.18+14.790+3.198

MSE reductions compare Current with Shuffle. Fusion gains compare Full with Base and are multiplied by 10310^3. Each table row averages 12 evaluations at a fixed dataset and shift magnitude: 2 shifted-source choices × 2 axes × 3 training seeds.

Current reduces query MSE in 48/48 evaluations, with reductions of 24.4%-40.3%. In these same 48 evaluations, Full improves VIF over Base in 48/48 evaluations and QabfQ_{abf} in 44/48. Both tested shifts retain positive mean prediction and fusion gains. Larger displacement or nonrigid deformation requires image registration or a correspondence method to establish the shared grid.

All evaluations use a common eight-pixel valid crop and clean source references. The revised Sec. 3.2 (p. 4) clarifies the spatial-correspondence assumption, and Appendix G.4 (p. 23) describes the alignment tests in detail.

Thanks for the additional experiments and explanations. The new controls help clarify the contribution of pair-specific fitting, and the alignment assumption is now clear. While the gains remain modest, the response addresses my main concerns. I’m keeping my original score.

Thank you very much for your careful review and constructive feedback. We appreciate that our additional experiments and clarifications have addressed your main concerns.

Reviewer_2ejPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This article proposes a test-time training framework T3Fusion for multiple image fusion tasks to address the problem of insufficient generalization ability of fusion models on new datasets and tasks. The method extracts multi-scale contextual features from two source images through a shared encoder, and then uses four fold cross fitting and closed-form ridge regression to learn bidirectional feature maps between the two sources on each test sample. The fusion readout module updates the representation through cross source prediction, prediction residuals, and original features, and generates results from a fixed network. The authors used the same model trained on the MSRS dataset to evaluate infrared and visible light, multi exposure, multi focus, medical and polarization fusion, and downstream perception tasks.

Critical strengths
  • The method represents the sample specific cross source relationships as small-scale ridge regression states, and uses closed form solving to avoid backpropagation problems during testing. The network parameters remain unchanged throughout the entire testing process.
  • The model simultaneously utilizes cross source prediction, prediction residuals, and original feature updates to fuse representations, while learning common information between two images and retaining their respective differences.
  • The experiment covered eight datasets and five types of fusion tasks, comprehensively verifying the generalization ability of the method.
Critical weaknesses
  • The specific implementation explanation of cross source mapping during testing in the paper is not centralized enough, and information such as feature standardization, support sample selection, and ridge regression coefficients is scattered in the main text and appendices.
  • Existing experiments mainly report overall average results on different datasets, lacking classification statistics for different image types or degradation conditions.
  • The paper assumes that the input image has completed pixel level alignment, but the applicability of this assumption and the potential impact of slight misalignment on cross source mapping have not been fully discussed.
Questions for authors
  • Although the article introduces test time regression, differentiable closed form solvers, and other test time learning methods, the connection and differences between T3Fusion and these methods are not fully explained. Suggest adding some relevant discussions to help readers better understand the specific positioning of T3Fusion.
  • The current method framework diagram contains a lot of modules and information, which makes it inconvenient to correspond to the main text and formulas. Suggest the author to add more detailed annotations in the figure to explain the data flow between closed-form fitting, cross source prediction, and fusion readout, in order to improve the readability of the framework diagram.
4 comments

Thank you for your careful review. We address each concern below with the relevant explanation and evidence.

Q1. The steps for test-time mapping, including standardization, support selection and ridge regression, need to be explained together.

Response:

The complete test-time procedure keeps the learned parameters fixed and solves only the current pair's maps.

Features and standardization. Encode the aligned sources at three scales. Row-normalized projections produce 16-channel keys from backbone features and target values from frozen Teacher features. Standardize them with shared per-scale means and channel-wise standard deviations calibrated once on training data. Append one to each key to represent the intercept.

Support and query locations. Divide the grid into four folds using g(i,j)=2(i mod 2)+(j mod 2)g(i,j)=2(i\bmod2)+(j\bmod2). For each query fold, deterministically sample at most 1,024 valid support locations from the other folds, excluding padding. For B→AB\to A, pair B's keys with A's target values at the same coordinates and solve

WB→A=VA,SKˉB,ST(KˉB,SKˉB,ST+nρI)−1.W_{B\to A}=V_{A,S}\bar K_{B,S}^{\mathsf T}(\bar K_{B,S}\bar K_{B,S}^{\mathsf T}+n\rho I)^{-1}.

Here SS is the support set, nn its size, and ρ>0\rho>0 the offline learned ridge coefficient, including regularization of the intercept. We evaluate the expression by a linear solve. The map predicts only its excluded fold. Reversing the sources and repeating across scales gives 24 temporary 16×1716\times17 maps. Query targets are excluded from the fits used to predict them.

Prediction to fusion. Assemble held-out predictions, compute residuals V−V^V-\hat V, and pass them with the original features to the zero-reference readout. Fuse the updated context and original details separately. Decode the context, concatenate it with the fused details, and reconstruct the image with the output head. Discard the maps after this pair. Sections 3.1-3.4 (pp. 3-6) of the revised manuscript provide a consolidated, detailed description of this procedure.

Q2. The experiments need results for different image types and degradation conditions.

Response:

The added analyses locate the gains across brightness levels and degradation conditions. We sort all 361 MSRS test pairs by mean visible-image brightness and divide them into four approximately equal groups, from darkest to brightest, containing 90, 90, 90, and 91 pairs. Using current-pair maps instead of shuffled maps gives the following QabfQ_{abf} improvements:

Brightness quartileImagesΔQabf×103\Delta Q_{abf}\times10^3
1 (darkest)902.697 ± 0.740
2901.914 ± 0.298
3900.751 ± 0.089
4910.674 ± 0.139

For each brightness group, we first average image-level gains within each training seed, then report the mean ± SD across seeds 3407, 3408, and 3409, scaled by 10310^3. All four brightness groups improve, with a gain about four times larger in the darkest group than in the brightest. This shows that adaptation is useful across lighting conditions and is especially helpful for darker inputs.

The FMB results also show higher VIF than Base and Static under each condition below:

FMB inputFull VIFBase VIFStatic VIF
Clean1.10581.08641.1022
VI noise 150.51140.50790.5097
VI blur 10.84140.82710.8375
IR blur 11.09261.07401.0894

Noise uses σ=15/255\sigma=15/255 and blur uses σ=1\sigma=1 pixel. All methods receive the same inputs and use the same clean references for evaluation. The mean gains over Base under blur are:

Blurred sourceΔ\DeltaVIFΔQabf\Delta Q_{abf}
Infrared+0.018681+0.003344
Visible+0.014302+0.003930

Both VIF and QabfQ_{abf} improve when either source is blurred.

To check the role of the fitted maps, we also fix all network weights and compare Current with Shuffle. Current improves VIF and QabfQ_{abf} in 54/54 noise evaluations. The 54 evaluations are 3 datasets (FMB, M3FD, RoadScene) × 2 noisy-source choices (infrared or visible, one at a time) × 3 training seeds (3407, 3408, 3409) × 3 noise realizations at σ=15/255\sigma=15/255. Each evaluation averages over the test images for one such combination. Together, these results show both where the gains occur and how the fitted relations contribute.

The revised manuscript analyzes the noise controls alongside Table 6 (p. 10), and the either-source noise and brightness groups alongside Tables S12 and S14 (p. 23).

Q3. The alignment assumption and the effect of small alignment errors need further discussion.

Response:

The method assumes pre-aligned feature grids. Each key in one source is paired with a target at the same coordinate in the other source. The fitted map predicts feature channels using this correspondence, so alignment establishes the locations used for fitting.

The 48 evaluations comprise 16 conditions × 3 training seeds. The conditions are 2 datasets (FMB, RoadScene) × 2 shift magnitudes (0.5, 1 pixel) × 2 shifted-source choices (infrared or visible, one at a time) × 2 axes (horizontal, vertical). Each condition is evaluated with seeds 3407, 3408, and 3409. Each evaluation summarizes test-image results for one condition and seed. The count excludes zero-shift inputs. The results are:

DatasetShift (px)MSE drop (%)Δ\DeltaVIFΔQabf\Delta Q_{abf}
FMB0.534.51+11.451+2.445
FMB133.48+9.315+1.514
RoadScene0.531.00+16.633+3.947
RoadScene130.18+14.790+3.198

Each table row averages 12 evaluations at a fixed dataset and shift magnitude: 2 shifted-source choices × 2 axes × 3 training seeds. MSE reductions compare Current with Shuffle, while the fusion gains compare Full with Base and are multiplied by 10310^3. Current reduces prediction MSE in 48/48 evaluations. In these same 48 evaluations, Full improves VIF over Base in 48/48, and QabfQ_{abf} in 44/48. Both datasets retain positive average prediction and fusion gains at each tested shift.

For larger displacement or nonrigid deformation, image registration or a correspondence method is needed to establish the shared feature grid. The spatial split then predicts each query from the other folds.

All evaluations use a common eight-pixel valid crop and clean source references. The revised Sec. 3.2 (p. 4) clarifies the alignment requirement, and Appendix G.4 (p. 23) describes these tests in detail.

Q4. T3Fusion's relation to test-time regression, differentiable solvers and other test-time learning methods needs clarification.

Response:

T3Fusion formulates fusion as held-out cross-source prediction and converts relations fitted to each image pair into fusion updates. Its connections to test-time regression, differentiable solvers, and other test-time learning methods are:

Related approachShared principleT3Fusion's use
Test-time regressionRegression over input tokens for memory and retrievalKeys from one image predict values in the other at held-out aligned locations
Differentiable solversLearn representations by differentiating through a fast fitOffline gradients through ridge fitting learn projections and regularization for prediction and fusion
TTTSelf-supervision from test inputsSolve temporary relation maps, with all learned network weights fixed and no test-time backpropagation
TTT layersA learner acts as the updated hidden stateSolve maps independently within each image pair, with no state carried to the next pair

Our design combines held-out cross-source prediction, per-pair relation fitting, and a zero-reference fusion readout. The readout receives predictions, residuals, and original features, then applies separate common and difference updates. Predictable information can guide fusion while complementary details remain available.

Two controls test the fitting and readout contributions separately. Current uses maps fitted to the test pair, while Shuffle uses maps from other pairs, with identical inputs and network weights. Full-Shared retains fitting but replaces the common/difference readout with a parameter-matched reader. The table reports ΔQabf×103\Delta Q_{abf}\times10^3, mean ± SD across three training seeds on the complete test sets.

DatasetCurrent - ShuffleFull - Shared reader
MSRS1.507 ± 0.2150.744 ± 0.063
FMB0.463 ± 0.2682.525 ± 0.402
M3FD1.320 ± 0.2213.111 ± 0.497
RoadScene1.242 ± 0.4194.223 ± 0.293
MEFB0.849 ± 0.3950.485 ± 0.419
MFI-WHU0.605 ± 0.0883.133 ± 0.086
PET-MRI0.099 ± 0.0073.699 ± 0.307
DoLP-S00.767 ± 0.2183.677 ± 1.347

Each control gives positive QabfQ_{abf} gains in 24/24 evaluations: 8 datasets × 3 training seeds (3407, 3408, 3409). Each evaluation compares test-set mean scores for one dataset and seed. Each table row summarizes the three seed-level differences. These comparisons separately support current-pair relation fitting and the common/difference readout that converts fitted predictions into fusion updates.

Gradient-based fitting and TTD comparisons. GD-2/8/32 optimize the same ridge objective with 2/8/32 gradient steps, using the checkpoint trained with closed-form fitting and its fixed projections and readout. TTD (NeurIPS 2024) derives fusion weights from relative dominability. We evaluate its full official IVF pipeline as an external method. The complete-test-set QabfQ_{abf} results are:

DatasetGD-2GD-8GD-32TTDT3Fusion
MSRS0.70860.72010.72320.63310.7258
FMB0.71020.70820.71150.63100.7179
M3FD0.67240.67240.67660.57360.6815
RoadScene0.59410.59790.60130.44250.6061
MEFB0.72060.72980.73140.54110.7332
MFI-WHU0.67900.67930.68040.62890.6821
PET-MRI0.75940.76170.76320.64480.7636
DoLP-S00.59020.66910.67480.52260.6821

The downstream results average the infrared- and visible-trained observers, as in Figure 6:

MethodFMB mIoU (%)M3FD mAP$_{50:95}$ (%)
GD-249.9737.14
GD-850.1336.78
GD-3250.0937.20
TTD47.7936.16
T3Fusion50.0537.57

T3Fusion achieves higher QabfQ_{abf} on all eight datasets and higher detection mAP than GD-2/8/32 and TTD. Against TTD, segmentation mIoU also increases from 47.79% to 50.05%.

The revised Sec. 2 (pp. 2-3) discusses the method connections, Tables S9-S10 (p. 22) analyze the fitting/readout controls, and Figure 6 (p. 8) with Sec. 4.5 (p. 9) reports the GD and TTD comparisons.

Q5. Figure 2 needs clearer annotations showing how fitting, cross-source prediction and fusion are connected.

Response:

In the revised manuscript, Figure 2 makes the connection between relation fitting, cross-source prediction, and fusion explicit. Its annotations and the descriptions in Secs. 3.1-3.4 (pp. 3-6) explain the following points.

Offline training and inference. Figure 2 distinguishes backbone training, TTT module learning, and inference. The annotations specify that the backbone is trained first and then frozen with the Teacher while the TTT projections, readout heads, and ridge coefficient are learned. At test time, all learned parameters remain fixed and only the current pair's relation maps are solved.

Fitting and prediction in panel (a). The panel labels the projected keys, target values from the frozen Teacher, support locations, Ridge Fit, and held-out predictions. The arrows show that fitting uses only support locations and each map predicts its excluded query fold. Four folds are assembled into V^A\hat V_A and V^B\hat V_B, with 24 maps across three scales and both directions.

Readout and feature updates in panel (b). The panel shows how predictions, residuals V−V^V-\hat V, target values, and original context features enter the readout. It separately labels the common and difference heads, their zero-prediction reference subtraction, and their ++/−- feature updates. This makes the independent heads and the use of cross-source predictions explicit.

Fusion and reconstruction in the upper workflow. The annotations specify that the Scorer uses FX′F'_X, the mean of the two updated features, and their absolute difference. The revised arrows distinguish fusion of updated context from fusion of original details. They also show that the pyramid decoder receives only the fused context, whose decoded output is concatenated with fused details before the output head. These annotations make each stage's inputs and outputs clear.

Reviewer_1vwZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a test-time training (TTT) framework for image fusion based on closed-form ridge regression. Although the test-time cross-source relation learning is conceptually appealing and demonstrates promising performance, the manuscript suffers from significant presentation issues, especially motivation and methodology.

Critical strengths
  1. The paper demonstrates promising performance, validating the effectiveness of the proposed TTT mechanism.

  2. The use of a closed-form solver for test-time adaptation avoids iterative backpropagation.

Critical weaknesses
  1. The manuscript is overly difficult to follow, requiring readers to infer the methodology from vague, repetitive statements (e.g., "A learned fusion readout combines these predictions... for image reconstruction"). A clear, step-by-step pipeline description should be provided early in the method section.

  2. Since the lack of fused ground truth is a universal challenge in image fusion, how exactly does this specific cross-reconstruction design address the generalization problem? Furthermore, the intuition behind key components (e.g., the necessity of the "Teacher" model and the "zero-prediction" mechanism) is insufficiently explained.

  3. Figure 2 is confusing. The inputs to the Scorer and Decoder are ambiguous, making it difficult to distinguish between the offline and test-time phases. And in Fig. 2(b), the algebraic representation is easily misinterpreted by readers: F_A+\Delta_c+\Delta_d=F_A+(R_{c,A}+R_{c,B})+(R_{c,A}-R_{c,B})=F_A+2R_{c,A}

Questions for authors
  1. Why specifically use ridge regression? How does ridge regression compare with other matching approaches like Sinkhorn or traditional representation decomposition?

  2. Is the \mathcal W in Equation (1) equivalent to the regression coefficient \beta in standard ridge regression? Also, is the loss \mathcal L_{cs} in Equation (1) identical to \mathcal L_q defined in Equation (8)? The manuscript should clarify these notations explicitly.

  3. In Sec. 3.1, the authors claim that the method can "suppress source-specific degradation." However, this claim is not empirically demonstrated or analyzed in the experimental section.

  4. Prior fusion methods [1] have already adopted "cross-reconstruction" design. The authors should properly cite and discuss the related works.

  5. The distinction between "offline training" and "test-time training (TTT)" is highly prone to confusion. It should be explicitly clarified.

  6. Why is the feature dimension set to r=16? Why is the spatial grid divided into a 4-fold split? The paper lacks experimental evidence or theoretical justification for these specific choices.

  7. While shuffling the pairs increases the reconstruction error (MSE), which is obvious, the actual gains in fusion metric are marginal. This does not provide compelling evidence to support the claim that pair matching improves the fusion.

  8. At Table 5, the Static relations method appears to perform worse than the Shuffled pairs results reported in Table 4. This seems counterintuitive and requires further explanation.

Ref: [1] FreeFusion: Infrared and visible image fusion via cross reconstruction learning

6 comments

Thank you for your careful review. We address each concern below with the relevant explanation and evidence.

Q1. The method needs a clear, step-by-step explanation of the complete pipeline.

Response:

Each test pair supplies its own cross-source prediction task, while the learned network stays fixed. The pipeline has four steps.

  1. Encode. A shared encoder extracts multiscale context features and full-resolution details. Learned projections produce keys from the backbone and target values from the frozen Teacher.
  2. Fit and predict. At each scale, fit a ridge map in each direction on support locations, then predict the spatial fold excluded from that fit. Three scales, four folds, and two directions give 24 maps.
  3. Read out. The readout uses predictions, residuals, and original features to compute common and difference updates of the context features. A zero-prediction reference makes the update vanish when the predictions vanish.
  4. Fuse and reconstruct. Fuse the updated context and original details separately. Decode the context pyramid, concatenate the decoded features with the fused details, and apply the output head.

Only the 24 temporary maps are computed anew at test time. No gradient update or state transfer between image pairs is used. The revised Figure 2 and Sec. 3.1 (pp. 3-4) explain this sequence in detail.

Q2. How does cross-source prediction help generalization, and why are the Teacher and zero-prediction reference needed?

Response:

The test pair provides its own supervision without a fused target. Fitting each pair allows its observed cross-source relations to guide fusion. Predictions are supplied together with residuals and original features, so information that is difficult to predict remains available to the readout.

The Teacher supplies a stable feature basis. It is a frozen copy of the pretrained encoder. The value projection is learned offline, so projected targets can change during training even though the Teacher is fixed.

The zero-prediction reference makes the correction depend on the predictions. For each head, Rj=hj(Z)−hj(Z0)R_j=h_j(Z)-h_j(Z^0), where Z0Z^0 replaces the prediction by zero and the residual by the target value. Zero predictions therefore give exactly zero update and recover Base. Without this subtraction, the zero-prediction output still differs from Base by 5.04-9.51 intensity levels.

The component controls on the complete FMB test set are:

FMB input/modelVIFQabfQ_{abf}QcbQ_{cb}
Clean / Full1.1057930.7178630.505700
Clean / random Teacher1.0928680.7152100.503096
IR noise / Full0.4551180.4272650.448389
IR noise / no reference0.4470310.4211400.445618

These are representative trained-model comparisons. Noise uses σ=15/255\sigma=15/255, averaged over three noise realizations. Both controls improve all three metrics in these examples. Sections 3.2-3.3 (pp. 4-5) of the revised manuscript clarify the Teacher and zero-prediction reference in detail.

Q3. Figure 2 needs to clarify the Scorer and Decoder inputs, the training/testing stages and the apparent cancellation of the feature updates.

Response:

The common and difference updates use independent heads, so the cross-source terms do not cancel algebraically.

The heads hch_c and hdh_d have different parameters. Their responses give

Δc=(Rc,A+Rc,B)/2,Δd=(Rd,A−Rd,B)/2.\Delta_c=(R_{c,A}+R_{c,B})/2,\qquad \Delta_d=(R_{d,A}-R_{d,B})/2.

Thus FA′=FA+Δc+ΔdF'_A=F_A+\Delta_c+\Delta_d contains the cross-source term (Rc,B−Rd,B)/2(R_{c,B}-R_{d,B})/2. It does not vanish by construction because Rc,BR_{c,B} and Rd,BR_{d,B} come from different heads. Each head shares its own weights across sources and between its actual and zero-reference branches. Swapping the sources preserves Δc\Delta_c and reverses Δd\Delta_d.

Scorer input. For source XX, the Scorer receives FX′F'_X, the mean of the two updated features, and their absolute difference. It computes fusion weights for the updated context. Original details are fused on a separate path.

Decoder input. The pyramid decoder receives only the fused multiscale context. Its output is concatenated with the fused original details, then passed to the output head.

Training versus inference. Offline training first learns the backbone, then freezes it and the Teacher while learning the TTT projections, heads, and regularization. All learned parameters stay fixed during inference, when only the 24 maps are solved from the current pair. In the revised manuscript, Figure 2 labels the common and difference heads separately and shows their distinct update paths. Sections 3.1 and 3.3 and the discussion alongside Eqs. (5)-(6) (pp. 3-5) explain the Scorer and decoder inputs and the non-cancellation above.

Q4. Why use ridge regression rather than Sinkhorn matching or representation decomposition?

Response:

Ridge regression directly fits the channel-prediction problem on our aligned feature grids. The positive regularizer gives a unique solution even when keys are correlated. With r=16r=16, fitting requires a 17×1717\times17 linear solve, and gradients can pass through it during offline training.

Sinkhorn-based matching estimates soft correspondences between locations, while representation decomposition separates feature components. Our inputs already supply spatial correspondences, and we need a predictor of one source's feature channels from the other. Ridge regression fits that objective directly.

To isolate regularization, we compare ridge regression with ordinary least squares. The mean gains on RoadScene are:

ComparisonΔ\DeltaVIFΔQabf\Delta Q_{abf}ΔQcb\Delta Q_{cb}
Ridge minus OLS+0.000155+0.000111+0.001018

Ridge regression improves all three metrics over OLS, supporting regularization in the channel predictor. The revised Sec. 3.2 explains this rationale after Eq. (3) (p. 4).

Q5. What does W\mathcal W represent, and how do Lcs\mathcal L_{cs} and Lq\mathcal L_q differ?

Response:

An individual WW is the regression coefficient matrix, including its intercept. W\mathcal W in Eq. (1) is the collection of 24 such matrices, from three scales, four spatial folds, and two prediction directions.

The two losses act on different locations and optimize different quantities. Lcs\mathcal L_{cs} is the regularized support-set prediction loss minimized over WW to fit a map for the current pair. Lq\mathcal L_q measures prediction error on query locations excluded from that fit, and is used offline to learn the projections and regularization strength by differentiating through the solve. A query target never enters the fit of its own predictor.

At test time, the learned projections and regularization remain fixed. We solve for WW using Lcs\mathcal L_{cs}, with no gradient update to the network. Sections 3.1-3.2 and the discussion around Eq. (8) (pp. 3-5) in the revised manuscript explain these definitions and their different roles in detail.

Q6. The claim about reducing the effects of degradation in one source needs experimental support.

Response:

The added controls test both fusion quality and sensitivity to noise. On FMB and M3FD, Full outperforms the model trained without the query loss in VIF and QabfQ_{abf} under infrared noise at σ=15/255\sigma=15/255:

DatasetΔ\DeltaVIFΔQabf\Delta Q_{abf}ΔQcb\Delta Q_{cb}
FMB9.541 ± 1.7893.313 ± 1.6671.844 ± 0.545
M3FD6.351 ± 1.7392.444 ± 1.3430.151 ± 0.715

Values are Full minus the no-query-loss model, scaled by 10310^3, as mean ± SD across three training runs. Each run averages three noise realizations. On FMB, MAE relative to each model's own clean output decreases from 18.29 to 14.68, a 19.8% reduction. Better fusion scores accompany a smaller noise-induced output change.

The fitted maps also help with all network weights fixed. Replacing shuffled maps by current-pair maps improves VIF and QabfQ_{abf} in 54/54 evaluations. The 54 evaluations are 3 datasets (FMB, M3FD, RoadScene) × 2 noisy-source choices (infrared or visible, one at a time) × 3 training seeds (3407, 3408, 3409) × 3 noise realizations at σ=15/255\sigma=15/255. Each evaluation averages over the test images for one such combination. All methods receive the same noisy inputs and are scored against clean source references.

These results support reduced sensitivity under the tested noise conditions. The revised manuscript analyzes these controls alongside Table 6 (p. 10) and Table S12 (p. 23).

Q7. Related work on cross reconstruction, including FreeFusion, needs to be cited and discussed.

Response:

The revised manuscript cites and discusses FreeFusion. FreeFusion learns fusion features by cross reconstruction. T3Fusion fits new cross-source maps from each test pair, predicts locations excluded from fitting, and turns those predictions into fusion updates with the network fixed.

The table uses FreeFusion's official WHU weights and identical test images and metrics. Each entry is FreeFusion/T3Fusion:

DatasetVIFQabfQ_{abf}QcbQ_{cb}
MSRS0.527/1.1350.421/0.7260.402/0.606
FMB1.049/1.1060.588/0.7180.470/0.506
M3FD0.939/0.9940.540/0.6810.482/0.519
RoadScene0.670/0.8200.421/0.6060.478/0.523
MEFB1.263/1.3680.484/0.7330.369/0.413
MFI-WHU1.031/1.1870.498/0.6820.496/0.686
PET-MRI0.532/0.7980.609/0.7640.509/0.439
DoLP-S00.737/1.1400.345/0.6820.357/0.357

T3Fusion has higher VIF and QabfQ_{abf} on all eight datasets. QcbQ_{cb} is higher on six datasets, while PET-MRI favors FreeFusion and DoLP-S0 is tied at the displayed precision. The QabfQ_{abf} gains are 44.0% on RoadScene, 51.5% on MEFB, and 37.0% on MFI-WHU. T3Fusion uses the same MSRS-trained weights across infrared-visible, multi-exposure, multi-focus, medical, and polarization fusion.

The revised manuscript discusses FreeFusion in Sec. 2 (p. 2), reports these comparisons in Tables 1-2 (pp. 6-7), and provides the downstream results in Table 3 (p. 8).

Q8. The distinction between offline training and test-time adaptation needs clearer explanation.

Response:

Offline training learns persistent parameters, while inference solves temporary maps for one pair. The two stages are:

  1. Backbone training. Train the encoder, context/detail fusion modules, pyramid decoder, and output head.
  2. TTT module training. Freeze the backbone and Teacher. Learn the key/value projections, readout heads, and positive ridge coefficient by backpropagating through the regression solve and fused output.

At test time, all these learned parameters remain fixed. Only the current pair's 24 temporary 16×1716\times17 regression matrices are solved. They guide the feature updates, are discarded after fusion, and are never passed to another pair. Inference uses no backpropagation. Figure 2 and Secs. 3.1 and 3.4 (pp. 3 and 5-6) of the revised manuscript distinguish these stages in detail.

Q9. Why use a feature dimension of 16 and a four-fold spatial split?

Response:

Dimension 16 with four folds balances fusion quality and cost. The four folds spread support locations across the image while keeping each query fold out of its own fit. With a dimension of 16, each linear system is only 17×1717\times17.

The sensitivity results are:

Rank/foldsFMB ΔQabf×103\Delta Q_{abf}\times10^3Time (ms)State (KiB)
8/4-0.06148.726.75
16/2-0.00846.0612.75
16/4+0.00048.9425.5
16/8-0.00854.2851
32/4+1.26350.0999

Quality differences use dimension 16 and four folds as the reference. Time measures the complete forward pass on an RTX 4090 at 640×480640\times480, with FP32 and a batch size of one. Dimension 16 uses 74.2% less temporary storage than dimension 32, which improves FMB quality. Eight folds take 10.9% longer than four, whereas changing the fold count changes mean QabfQ_{abf} by at most 0.000132 across the eight datasets. These results support the chosen setting as an efficient configuration with stable fusion quality.

The revised Sec. 3.2 (p. 4) explains the fold construction, and the discussion alongside Table S13 (p. 23) analyzes these sensitivity results.

Q10. The contribution of current-pair fitting needs to be demonstrated through final fusion quality.

Response:

With the input, network, and readout fixed, current-pair maps improve final fusion quality over shuffled maps. Representative mean gains are:

DatasetCurrent minus Shuffle: ΔQabf\Delta Q_{abf}
MSRS+0.001507
M3FD+0.001320
RoadScene+0.001242

Across all eight datasets, the mean, median and trimmed mean improvements are positive, and 58.7%-91.4% of images improve based on per-image gains averaged across runs. The benefit therefore extends across individual images as well as datasets.

The maps also change the final image. The average luminance MAE between Current and Shuffle is 0.231-0.852 on the 0-255 scale. Since this comparison changes only the fitted maps, the quality gains directly show the contribution of relations fitted to the current image pair.

The fixed-network comparison isolates the contribution of current-pair fitting to final fusion quality. The revised manuscript analyzes these gains alongside Table 4 (p. 9) and Tables S9 and S11 (p. 22).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response: Contributions, Additional Experiments, and Clarifications

We thank all reviewers for their careful reading and constructive feedback. We appreciate their recognition of closed-form adaptation to each image pair, prediction-guided fusion, and transfer across fusion tasks. Their comments help us clarify the method and strengthen the evidence for its individual components.

Contributions Recognized by Reviewers

  • Adaptation without test-time backpropagation. T3Fusion fits cross-source relations to each input pair while keeping the learned network fixed. [1vwZ, 2ejP, xYhM]
  • Fusion from predictions and complementary information. The readout combines cross-source predictions, residuals, and original features to refine fusion. [2ejP]
  • Transfer across fusion tasks. The same MSRS-trained model applies across different fusion tasks without target-task fine-tuning. [2ejP, xYhM]

Additional Experiments

  • Separating fitting from the network. Comparisons with the network fixed show that relations fitted to the current image pair improve final fusion quality. Image-level analyses examine how these benefits are distributed across the test data. [1vwZ, xYhM]
  • Testing the fusion readout. Controlled readout comparisons support the common/difference design. A matched map-generator comparison also examines per-pair fitting against an offline learned alternative. [1vwZ, 2ejP, xYhM]
  • Noise, blur, and illumination. Experiments examine fusion quality under input degradation and across brightness levels, supporting the usefulness of adaptation under the tested conditions. [1vwZ, 2ejP]
  • Residual alignment errors. Small-shift tests show that prediction and fusion benefits remain under mild misalignment. The revised manuscript also clarifies the method's pre-alignment requirement. [2ejP, xYhM]
  • Related methods and component choices. The revision includes comparisons with FreeFusion, gradient-based fitting, and TTD, together with analyses of the Teacher, zero-prediction reference, regularization, and fitting configuration. [1vwZ, 2ejP]

Method and Presentation Clarifications

  • A clearer framework figure and data flow. The revised Figure 2 and accompanying text distinguish offline training from inference and explain how fitting, prediction, readout, and image reconstruction connect. [1vwZ, 2ejP]
  • A precise account of test-time adaptation. The revision explains which quantities are fitted to each input pair, which learned parameters remain fixed, and how the method relates to test-time regression, differentiable solvers, and other test-time learning approaches. [1vwZ, 2ejP]

The responses below provide the detailed experimental settings, quantitative results, and corresponding revision locations.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.