acceptodds
Under review as a conference paper at ICLR 2027

Relation Matters: Learning Local Relations for Binary Infrared-Visible Image Fusion

Abstract

Infrared–Visible Image Fusion (IVIF) integrates thermal targets with spatial details, but full-precision models require substantial computation and storage. Binarization reduces these costs, yet binary IVIF faces three connected challenges: binary representation, where binarization can obscure local structural differences; feature propagation, where binary processing can degrade the magnitude information needed for reconstruction; and capacity-compatible training, where the binary network must learn complementary fusion patterns within its limited representational capacity. To address these challenges, we propose SDRTNet, a binary IVIF framework that jointly addresses binary representation, feature propagation, and capacity-compatible training. Sliding Directional Relation Transform (SDRT) makes local directional differences explicit by decomposing overlapping patches into a base component and four directional relation components for separate binary processing. Local Statistical Alignment (LSA) then uses the local means and variances of raw SDRT features to calibrate binary convolution outputs before spatial reconstruction. During training, Binary Vector-Quantized Knowledge Distillation (BVQKD) organizes a full-precision teacher's fusion knowledge into a shared codebook and guides the binary student through codeword-assignment distributions, without adding inference cost. Experiments on four fusion benchmarks and two downstream tasks show competitive fusion performance with reduced computational and storage costs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Image fusion, Infrared image, Visible image, Binary Neural Network

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vz6XOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper targets binarized infrared-visible image fusion (IVIF) and proposes SDRTNet, a hybrid-precision network in which a full-precision encoder and decoder wrap a multi-scale backbone of binary blocks. The authors motivate the design with three coupled challenges: retaining local structural distinctions under binarization, preserving magnitude information during binary propagation, and training a limited-capacity binary network to learn complementary fusion patterns. Three modules are introduced accordingly. (1) Sliding Directional Relation Transform (SDRT): from sliding, overlapping 2×22\times2 patches the method computes one base component (a+b+c+d)/2(a+b+c+d)/2 and four directional differences (horizontal, vertical, diagonal, anti-diagonal), processes the five components through independent binary branches, and recombines them with a fixed least-squares pseudo-inverse (ISDRT) plus overlap-averaging. The transform is shown to be invertible (A+A=I4A^{+}A = I_4, det⁡(A⊤A)=12\det(A^{\top}A) = 12), so information loss is attributed to binarization rather than to the transform. (2) Local Statistical Alignment (LSA): binary convolution features are standardized by their own local mean and variance, then re-scaled and re-shifted using the full-precision statistics of the raw SDRT features, and combined through a learnable gated residual. (3) Binary Vector-Quantized Knowledge Distillation (BVQKD): a full-precision teacher with the same SDRT-ISDRT topology is trained first, a VQ codebook and decoder are learned to reconstruct the teacher's fused output, and the binary student is distilled by matching teacher/student codeword-assignment distributions, which adds no inference cost. Validation: the model is trained on 2,000 LLVIP pairs and evaluated on LLVIP, TNO, M3FD and MSRS using six metrics (EN, MI, SF, EI, AG, CC) against five full-precision and five binary methods, plus downstream detection (M3FD, YOLOv11) and segmentation (MSRS, DeepLabV3+). Ablations cover the transform domain, normalization/alignment variants, knowledge-distillation objectives, codebook construction and residual-branch design. Reported efficiency is 0.38M parameters, 17.9 GFlops and 0.082 s inference at 1024×7681024\times768.

Critical strengths
  1. A genuinely under-explored problem, framed so that each claim is separately testable. Binary IVIF is largely untouched - the BNN literature concentrates on classification, super-resolution and restoration. Decomposing the difficulty into representation / propagation / training is not just rhetorical: it yields a modular design and, more importantly, an ablation suite (Tabs. 2-4, Fig. 6) in which every claim has a matching control. That discipline is above average for this venue.
Critical weaknesses
  1. The efficiency report should be made explicit rather than aggregate (W1, most worth fixing). The encoder and decoder are full precision, SDRT/ISDRT are computed in FP, and LSA injects full-precision local statistics plus a per-channel affine and gate into every block. This is a legitimate design, but the paper never states the binary-to-FP FLOP ratio, gives no module-level cost breakdown, and does not state the counting convention behind the 17.9 GFlops in Fig. 1 (are 1-bit MACs counted as 1 bit?). A short table would let readers attribute the advantage - the latency evidence in Fig. 1 already suggests the advantage is real, so this is a reporting gap, not a validity problem.
  2. Positioning against prior art would sharpen the contribution claim. Each module has a natural precedent: SDRT is a fixed over-complete 2×22\times2 difference/lifting transform; LSA is window normalization + affine + gated residual, which overlaps with ReActNet/IR-Net-style activation shaping; BVQKD combines VQ-VAE with KD. The paper does not discuss these links, and the Tab. 2 comparison (SDRT vs a critically-sampled DWT) changes both directionality and redundancy (≈5\approx5 coefficients/pixel vs ≈1\approx1), so the gain cannot yet be attributed to "directional relations" specifically. Two cheap additions would settle it: a redundancy-matched, non-directional 2×22\times2 transform, and a ReActNet/IR-Net-style baseline inside the same backbone.
Questions for authors
  1. Cost accounting. Could you give a module-level table of parameters, BOPs/MACs and latency (FP encoder, FP decoder, SDRT/ISDRT, LSA statistics, binary branches) and state the Fig. 1 counting convention? Even an approximate binary-to-FP FLOP ratio would resolve this.
  2. LSA: calibration or full-precision pathway? Since LSA reads full-precision μraw,σraw\mu_{\mathrm{raw}}, \sigma_{\mathrm{raw}}, a control that injects the same statistics without the standardization/affine/gating (e.g. Y=Xbin+μraw+σrawY = X_{\mathrm{bin}} + \mu_{\mathrm{raw}} + \sigma_{\mathrm{raw}}) would clarify which mechanism is responsible for the gain.
2 comments
AuthorsOpenReview ↗

Response to vz6X

Thanks for your recongnization. We’ll address Weaknesses(W) and Questions(Q) below.

1. Cost accounting(W1, Q1)

Thank you for pointing out that Figure 1 does not define its counting convention. For one 1024×7681024\times768 infrared–visible image pair, the SDRTNet has 376570 parameters (0.38 M). Its binary convolutions perform 8.4709 G MACs, and its full-precision convolutions perform 0.4894 G MACs, giving a binary-to-full-precision convolutional MAC ratio of 17.31:1. SDRTNet counts two operations per MAC for both binary and full-precision convolutions, with no bit-packing discount. This gives 2×(8.4709+0.4894)=17.92062\times(8.4709 + 0.4894) = 17.9206 G convolutional operations.

ModuleParametersBinary conv. GMACsFP conv. GMACsEstimated share of RTX 3090 latency (ms)
FP encoder and downsampling285300.14203.28
FP decoder, upsampling, and output layers690700.34747.10
Binary branches, excluding LSA3628358.4709028.60
LSA39750032.59
SDRT0001.99
ISDRT and overlap averaging0008.44
Total3765708.47090.489482.00

LSA has zero convolutional MACs because it computes local statistics through reductions and average pooling, then applies full-precision normalization, affine scaling, and gated residual mixing. These operations contribute to inference time.

2. LSA(W2, Q2)

Following your suggestion, we train a direct-injection control with Y=Xbin+μraw+σrawY=X_{\mathrm{bin}}+\mu_{\mathrm{raw}}+\sigma_{\mathrm{raw}}. It retains the binary branch's learnable activation shift and RPReLU, uses the same local statistics as LSA, and removes standardization, the channel-wise affine transformation, and the gate. Direct injection adds the same local offset to every channel. Full LSA also uses the raw local standard deviation to scale the standardized channel responses.

On TNO, direct injection achieves EI of 27.06 and AG of 3.01, above the No norm values of 19.87 and 1.73. Full LSA achieves 39.85 and 3.97, respectively. These results suggest that access to full-precision statistics accounts for part of the improvement, while the complete calibration procedure provides an additional benefit.

MethodEN ↑MI↑SF ↑EI ↑AG ↑CC ↑
No norm4.759.496.0819.871.730.526
Direct injection6.0312.077.9227.063.010.666
Full LSA6.6913.389.9439.853.970.681

3. Directionality and redundancy(W2, Q3)

Table 2 compares DWT and NSDRT on the same non-overlapping 2×22\times2 patch grid. DWT computes four wavelet coefficients per patch, while NSDRT encodes a base component and four directional differences. Even on this non-overlapping grid, NSDRT achieves higher SF (8.12 vs. 7.53), EI (34.21 vs. 30.24), and AG (3.46 vs. 3.03) on TNO. These gains therefore occur before overlap is introduced. With the same five SDRT components, switching to overlapping windows further increases SF to 9.94, EI to 39.85, and AG to 3.97.

We retain the fifth component Ra=c−bR^a=c-b because its sign records the ordering of the right and lower neighbors in a 2×22\times2 patch. Although Ra=Rv−RhR^a=R^v-R^h is redundant in full precision, this ordering can disappear after binarization. For example, the patches [0123]\left[\begin{smallmatrix}0&1&2&3\end{smallmatrix}\right] and [0213]\left[\begin{smallmatrix}0&2&1&3\end{smallmatrix}\right] give the same four signs (+,+,+,+)(+,+,+,+) for (M,Rh,Rv,Rd)(M,R^h,R^v,R^d) at zero threshold. The larger increase is downward in the first patch and rightward in the second, which RaR^a distinguishes through opposite signs. To test the value of this distinction, we remove RaR^a and use the exact inverse of the four-component transform, so both variants reconstruct unmodified patches exactly. On TNO, retaining RaR^a increases SF from 7.35 to 9.94 and EI from 25.99 to 39.85.

MethodEN ↑MI ↑SF ↑EI ↑AG ↑
Non-overlapping SDRT5.8611.718.1234.213.46
M,Rh,Rv,RdM, R^h, R^v, R^d6.0812.167.3525.993.61
Full SDRT6.6913.389.9439.853.97

4. Periodic artifacts (Q4)

We examine TNO image pairs using enlarged low-texture regions, detrended residuals, gradient maps, and Fourier spectra. Each comparison uses the same 32×3232\times32 region in the visible input, infrared input, and fused image. Nearest-neighbor enlargement preserves individual pixels. We also measure the combined energy at the horizontal, vertical, and checkerboard period-two frequencies, normalized by the total energy after removing a fitted intensity plane.

As shown in the uploaded supplementary material, the TNO fusion results exhibit no visible two-pixel-periodic grid artifacts near patch boundaries after overlap averaging. The period-two frequencies account for 0.018%–1.678% of the residual energy in the displayed fused regions. The region with the highest fraction contains a stronger period-two component in the visible input, at 5.317%. The enlarged regions, gradient maps, and Fourier spectra consistently support this observation.

5. Comparison methods (Q5)

We retain each baseline’s binary backbone and adapt its input and output interfaces to infrared–visible image fusion. All five binary baselines use a common training protocol: 2000 LLVIP training pairs, a 50-epoch schedule, a batch size of 4, and AdamW with an initial learning rate of 0.0002. They share the same base fusion loss as SDRTNet’s Stage 1. For the five full-precision methods, we use official checkpoints where available; otherwise, we train them using our Stage 1 protocol.

Regarding the E2FIF/BBCU LLVIP triples, Table 1 currently reports the same rounded (EN, MI, CC) values of (7.51,15.02,0.918). An independent re-evaluation of the archived LLVIP outputs with LinFeng Tang’s evaluation function gives (7.508452, 15.016904, 0.918442) for E2FIF and (7.510857, 15.021714, 0.918329) for BBCU.

6. Clarification of the format(Q6)

SDRTNet follows the binary-network formulation used by the compared baselines, including E2FIF, BBCU, FABNet, S2BNet, and BNNPan. Its binary branches use scaled sign-based weights and activations, represented as floating-point tensors and processed by standard convolution operators in our implementation, while the surrounding modules retain floating-point computation. Thus, our evaluation concerns binary representations rather than native packed 1-bit or INT8 execution. We focus on this setting because sign-based quantization removes magnitude information and can obscure local structural differences, which SDRT and LSA specifically address.

7. Deployment(Q7)

Given our limited budget for binary edge accelerators, we supplement inference time with a controlled resource-limited benchmark on a separate GPU platform(3090). We compare SDRTNet with five binary baselines using NVIDIA MPS active-thread limits of 100%, 50%, and 25%. All methods use FP32 execution, identical synthetic inputs at 1024×7681024\times768, batch size 1, and initialized weights for architecture-level timing. We report averages over 30 runs after 10 warm-up iterations. SDRTNet achieves the lowest latency and peak allocated GPU memory among these implementations.

MethodBaseline time (ms) ↓Estimated 50% limit (ms) ↓Estimated 25% limit (ms) ↓
E2FIF165.86267.31487.13
BBCU689.68899.291366.69
FABNet331.17485.13817.07
S2BNet137.70201.20358.22
BNNPAN281.00420.61740.16
Ours82.02111.12178.98
Reviewer_UPUjOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a binarized infrared–visible image fusion framework, termed SDRTNet. SDRTNet jointly addresses the challenges of binary representation, feature propagation, and capacity-compatible training within a unified framework. Experimental results demonstrate that SDRTNet maintains competitive fusion performance and downstream task performance with a relatively small number of parameters and low computational complexity.

Critical strengths

1.The proposed method presents a relatively comprehensive and well-structured technical framework. 2.The proposed method achieves competitive image fusion performance with a relatively small number of parameters and low computational complexity.

Critical weaknesses
  1. The local difference representation employed in SDRT shares similarities with existing techniques, such as Local Binary Patterns (LBP) and Pixel Difference Convolution (PDC). Furthermore, BVQKD builds upon established vector quantization and knowledge distillation techniques. Therefore, the methodological novelty of the proposed framework requires further clarification.
  2. The manuscript explicitly acknowledges the redundancy among the five components in SDRT. However, it does not sufficiently explain why five components, including a redundant one, are adopted instead of four independent components.
Questions for authors
  1. Could the authors further clarify the key methodological differences between SDRT and existing local difference representation techniques, such as LBP and PDC, as well as the novelty of BVQKD compared with existing knowledge distillation methods?
  2. Why do the authors adopt five components, including a redundant one, instead of four linearly independent components in SDRT? Could the authors provide further justification or ablation experiments to demonstrate the benefits of this design in terms of fusion performance?
4 comments
AuthorsOpenReview ↗

Response to UPUj

Thanks for your recongnization. We’ll address Weaknesses(W) and Questions(Q) below.

1. Key Innovation(W1, Q1)

LBP compares each neighbor with the center pixel to produce a binary code. PDC computes a learned weighted sum of selected pixel-pair differences, so binarizing a PDC response gives the sign of that sum. SDRT binarizes each directional difference in a separate branch, preserving the difference. For example, consider two neighborhoods that differ only by swapping the right and bottom neighbors of the center pixel:

\qquad X_2 = \begin{bmatrix} 2 & 2 & 2 \cr 2 & 1 & 3 \cr 2 & 2 & 4 \end{bmatrix}.$$ For LBP, the neighbor-to-center differences, starting from the right and proceeding clockwise, give $$\begin{aligned} X_1:\quad (1,3,2,1,1,1,1,1) &\longrightarrow (1,1,1,1,1,1,1,1), \cr X_2:\quad (2,3,1,1,1,1,1,1) &\longrightarrow (1,1,1,1,1,1,1,1). \end{aligned}$$ For a central PDC filter with unit weights, binarizing the aggregated response gives $$\begin{aligned} X_1:\quad \operatorname{sign}(1+3+2+1+1+1+1+1) &= +1, \cr X_2:\quad \operatorname{sign}(2+3+1+1+1+1+1+1) &= +1. \end{aligned}$$ SDRT operates on the lower-right $2\times2$ window of each neighborhood. With zero branch biases for this example, the five binary activations entering the binary convolution are $$\begin{aligned} X_1:\quad \operatorname{Sign}(5,1,2,3,1) &= (+1,+1,+1,+1,\mathbf{+1}), \cr X_2:\quad \operatorname{Sign}(5,2,1,3,-1) &= (+1,+1,+1,+1,\mathbf{-1}). \end{aligned}$$ The last bit encodes the anti-diagonal relation between the bottom and right neighbors. Swapping them changes this bit from $\operatorname{sign}(3-2)=+1$ to $\operatorname{sign}(2-3)=-1$. LBP assigns both neighbors the same bit because both exceed the center. In PDC, whether the swap changes the binary output depends on the learned weights. SDRT gives this ordering its own binary channel, and it can be reconstructed back to the feature spatial domain through ISDRT. We also assess the practical impact of this distinction on the feature maps presented in Table 2, adhering to the identical procedural methodology outlined therein. On these feature maps, reconstruction MSE was 0.2316 for LBP, 0.2194 for PDC, and 0.0187 for SDRT. The corresponding SSIM values are 0.2789, 0.3338, and 0.9410. Under these reconstruction protocols, SDRT recovers both feature values and local structure more faithfully. | Method | MSE ↓ | MAE ↓ | PSNR ↑ | Corr ↑ | SSIM ↑ | NMI ↑ | |---|---|---:|---:|---:|---:|---:| | LBP | 0.2316 | 0.3759 | 13.3312 | 0.4870 | 0.2789 | 0.1261 | | Central PDC |0.2194 | 0.3645 | 13.4160 | 0.5063 | 0.3338 | 0.1277 | | **SDRT** | **0.0187** | **0.0693** | **23.7540** | **0.9470** | **0.9410** | **0.4384** | The novelty of BVQKD lies in how we use vector quantization to train a binary fusion network under the guidance of a full-precision teacher. Direct feature matching asks the binary student to reproduce continuous teacher activations channel by channel, a demanding target given its limited representational capacity. We instead learn a codebook of local fusion patterns by reconstructing the teacher’s fused image and its gradients, so the codewords capture intensity and edge information relevant to fusion. During binary network training, we supplement the student’s fusion loss with a distillation loss that aligns its predicted distribution over these codewords with the teacher’s at each spatial location. This gives the binary student a way to learn which fusion patterns the teacher favors without reproducing its precise activation values. The teacher and codebook are used only during training and add no inference cost.

2. SDRT’s five components(W2, Q2)

We retain the anti-diagonal component RaR^a because its sign records which of the two off-diagonal values is larger. For a patch with top row (a,b)(a,b) and bottom row (c,d)(c,d), the horizontal and vertical differences are Rh=b−aR^h=b-a and Rv=c−aR^v=c-a. Since Ra=c−b=Rv−RhR^a=c-b=R^v-R^h, the four real-valued components (M,Rh,Rv,Rd)(M,R^h,R^v,R^d) suffice for exact reconstruction before binarization. After binarization, however, the signs of RhR^h and RvR^v alone cannot determine the sign of RaR^a.

For example, consider two patches that differ only by swapping 11 and 22:

\begin{bmatrix} 0 & 1 \cr 2 & 3 \end{bmatrix}, \qquad X_B= \begin{bmatrix} 0 & 2 \cr 1 & 3 \end{bmatrix}.$$ Starting from the top-left corner, both patches increase to the right and downward. Their base components and main-diagonal differences also agree. At zero threshold, $(M,R^h,R^v,R^d)$ therefore gives the same four signs $(+,+,+,+)$ for both patches. Yet **the larger increase is downward in $X_A$ and rightward in $X_B$**. The anti-diagonal component captures this distinction: $$R^a(X_A)=2-1=+1, \qquad R^a(X_B)=1-2=-1.$$ Thus, $R^a$ adds no linear degree of freedom, but its binary branch preserves a directional ordering that the four sign bits alone cannot determine. We test this design by removing $R^a$ from the full SDRTNet backbone. The resulting variant retains $(M,R^h,R^v,R^d)$ and reconstructs each patch with the exact inverse of its four-component transform. Both configurations therefore reconstruct unmodified patches exactly. We train the four-component variant from scratch for 50 epochs on 2000 LLVIP pairs using the same frozen teacher and codebook as in the reference training protocol. The table compares its results on TNO with the full SDRTNet scores reported in the manuscript. We measure computational cost at $1024\times768$ resolution. On TNO, the configuration retaining $R^a$ achieves higher edge intensity and average gradient. | Components retained in SDRT | Params (M) | GFlops | EN ↑ | MI ↑ | SF ↑ | EI ↑ | AG ↑ | |---|---:|---:|---:|---:|---:|---:|---:| | $M, R^h, R^v, R^d$ | 0.3032 | 14.53 | 6.08 | 12.16 | 7.35 | 25.99 | 3.61 | | $M, R^h, R^v, R^d, R^a$ | 0.3766 | 17.92 | **6.69** | **13.38** | **9.94** | **39.85** | **3.97** |
Reviewer_UPUjOpenReview ↗

Response to the Rebuttal

Thank you for the detailed explanations and responses. The additional analysis demonstrates certain advantages of SDRT under the specific comparison settings. However, the response does not sufficiently explain from a theoretical perspective why the proposed design is particularly suitable for image fusion tasks. Moreover, the comparison with PDC is limited to a specific configuration and does not establish an advantage over PDC with learnable weights. In addition, the explanation of BVQKD still does not sufficiently clarify the fundamental methodological distinctions from existing knowledge distillation approaches. Therefore, I will maintain my original assessment.

AuthorsOpenReview ↗

Response to UPUj

Thank you for clarifying your concerns. We would like to further clarify our contributions and address your concerns.

1. SDRT

The patch examples qualitatively illustrate what each representation retains after binarization. SDRT binarizes directional differences in separate branches. Its anti-diagonal branch records which of two off-diagonal feature values is larger, even when the other four binary components are identical across patches. This distinction matters for infrared–visible image fusion because local directional contrasts help describe object boundaries and fine textures that the fused image should preserve. In the TNO ablation, the five-component model achieves higher edge intensity (39.85 vs. 25.99) and average gradient (3.97 vs. 3.61) than the four-component variant, with a higher parameter count.

The reconstruction table below quantifies how much feature information each representation retains under the same protocol. SDRT achieves an MSE of 0.0187 and an SSIM of 0.9410, compared with 0.2316 and 0.2789 for LBP and 0.2194 and 0.3338 for central PDC. This quantitatively indicates that our method can retain more information, thereby improving the performance of image fusion.

MethodMSE ↓MAE ↓PSNR ↑Corr ↑SSIM ↑NMI ↑
LBP0.23160.375913.33120.48700.27890.1261
Central PDC0.21940.364513.41600.50630.33380.1277
SDRT0.01870.069323.75400.94700.94100.4384

2. BVQKD

BVQKD addresses Capacity-Compatible Training, one of the three challenges we identify in the paper. Direct feature matching asks the binary student to reproduce the full-precision teacher’s continuous activations channel by channel, which is chanllenging given the student’s limited representational capacity.

Inspired by vector-quantized representation learning, we adapt this established principle to binary image fusion. We learn a codebook by reconstructing the teacher’s fused image and its spatial gradients, then train the student to match the teacher’s codeword distribution at each location. The contribution lies in this use of vector quantization to design a distillation target that helps the binary student learn which local intensity and edge patterns to preserve.

Reviewer_4WcnOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes SDRTNet, a framework for infrared–visible image fusion. It introduces SDRT to encode local directional relations before binary processing, LSA to calibrate processed features using local statistics, and BVQKD to transfer knowledge from a full-precision teacher through codeword-assignment distributions. Experiments include four fusion datasets, component ablations, and downstream detection and segmentation tasks. The reported results suggest competitive fusion quality with reduced computational and storage costs.

Critical strengths

Evaluation covers four fusion datasets and two downstream tasks. This provides broader evidence of the usefulness of the fused images than fusion metrics alone.

Critical weaknesses
  1. The motivation is not clear, and I cannot see what research gap this paper is addressing. The abstract and related work repeatedly state that full-precision fusion models require substantial computation and storage. However, there are already many efficient and real-time infrared–visible image fusion methods. The paper does not sufficiently explain the limitations of these methods or why binarization is needed. The three proposed modules look reasonable, but much of the paper describes what they do rather than why they are necessary. It remains unclear whether these designs address a substantive research gap.

  2. The experimental comparisons are not sufficiently convincing. E2FIF, BBCU, and FABNet were not originally proposed for infrared–visible image fusion, but their adaptation to this task and the fairness of the comparisons are not clearly explained. In Figure 3, the results of TCMoA and ControlFusion look similar to those of the proposed method. The downstream improvements over the strongest competing methods are also small: 0.2 percentage points in detection and 0.4 in segmentation. Overall, the experiments do not clearly demonstrate the advantage of the proposed method over existing fusion approaches, particularly efficient ones.

Questions for authors
  1. What is the main motivation of this work? What are the main advantages of the proposed method over existing real-time image fusion method?

  2. How were E2FIF, BBCU, FABNet, S2BNet, and BNNPan adapted to IVIF? Were the backbone, training data, losses, and distillation settings controlled? Were the SDRT representation ablations matched in parameter count and computation?

  3. Was YOLOv11 trained independently for each fusion method using the same initialization and data split? Are the small downstream improvements consistent across repeated runs? Please also clarify the discrepancy between the segmentation improvement stated in the text and Table 5.

1 comment
AuthorsOpenReview ↗

Response to 4Wcn

Thanks for your recongnization. We’ll address Weaknesses(W) and Questions(Q) below.

1. Motivation(W1, Q1)

Some full-precision methods already achieve real-time infrared–visible fusion. We explore whether a one-bit backbone can further reduce the cost of high-resolution fusion while retaining the information needed for reconstruction. Pointwise binarization can map distinct local patterns to the same signs, motivating Binary Representation that preserves distinctions among local structures. These distinctions must then remain useful for reconstruction as features are transformed, motivating Feature Propagation that preserves the local feature levels and variations that shape image content and contrast. Even when this information is retained, the binary student must still learn to combine complementary infrared and visible cues within its limited capacity, which motivates Capacity-Compatible Training.

We compare SDRTNet with two recent full-precision real-time methods, GIFNet and PCDFusion. Table gives parameters and inference time per 1024×768 infrared–visible pair, along with results on MSRS. SDRTNet(0.38M) uses about 54% fewer parameters than GIFNet and 92% fewer than PCDFusion. On MSRS it exceeds GIFNet on EN, MI and CC. Compared with PCDFusion, its EN and MI are close.

MethodParam(M)Time(s)ENMIAGCC
GIFNet0.8230.2586.2812.573.950.513
PCDFusion4.6020.7697.0214.054.350.553
Ours0.380.0826.9713.953.790.553

2. Binary comparison methods(W2, Q2)

We retain each baseline’s binary backbone and adapt it to predict fused image from infrared and visible image. E2FIF retains its EDSR backbone, BBCU its BBCUM, and FABNet its wavelet-based FABNet. For these three methods, we adapt the input and output layers to accept two channels and predict one. S2BNet retains its binary U-Net and removes the original input upsampling. BNNPan retains its two input branches and auxiliary loss, and removes input upsampling. We feed visible image into its modified LMS branch and infrared into its PAN branch. We train IVIF-specific weights for all five methods. All five binary baselines use the same training configuration, which is the same as SDRTNet. We set a 50-epoch schedule, a batch size of 4, and an initial learning rate of 0.0002 with AdamW. We use the same rule to select 2000 LLVIP image pairs, and a common base fusion loss which is the same as SDRTNet’s Stage 1 loss.

For the four representations compared in Table 2, Plain binarization(0.38M, 18.1 GFLOPs) and overlapping SDRT(0.38M, 17.9 GFLOPs) have identical parameter counts and similar computation. DWT(0.30M, 4.4 GFLOPs) and non-overlapping SDRT (0.38M, 5.3 GFLOPs) require less computation. We consider the additional parameters of overlapping SDRT over other methods worthwhile given its gains in fusion quality and SDRTNet’s competitive performance against both binary and full-precision baselines.

3. Downstream experiments(Q3)

We train a separate YOLOv11 detector for each fusion method from the same pretrained weights, using the same M3FD train/validation/test split (7:1:2) and SGD for 200 epochs. For semantic segmentation, we train a separate DeepLabV3+ model for each method on the official MSRS split, using SGD with cross-entropy loss and a cosine learning-rate schedule for 1000 epochs. Both tasks use a batch size of 16 and an initial learning rate of 0.01. Within each task, only the fused input images change across methods. We report the mean of three runs per method. Our gains are comparable to those of other IVIF methods, where improvements over competing fusion baselines are usually like this on downstream tasks. On both tasks, the gains over visible-only and infrared-only inputs are more remarkable. The correct segmentation improvement over TCMoA is 0.4 percentage points, as shown in Table 5. The 1.1-point statement in Sec. 4.3 is a reporting error.

Pedestrian privacy

Thank you for raising the issue of pedestrian privacy. Our study uses images from four existing datasets (LLVIP, M3FD, MSRS, and TNO). The maintainers report that they blur faces and prohibit attempts to identify individuals. Before publication, we will review every image to avoid this problems.

Comments on the paper

3 comments

Dear Reviewers,

The authors have posted their rebuttal responses. I would kindly encourage you to review the authors' responses and continue the discussion by providing follow-up comments where appropriate. In particular, please indicate whether the authors' responses have adequately addressed your concerns and update your assessment if necessary.

Thank you for your time and continued efforts!

Your AC

Dear Reviewers UPUj and vz6X,

There are fewer than four days left in the discussion period. The authors have submitted their rebuttal, but we have not yet received your feedback.

Please read the rebuttal and let the authors know if it answers your concerns. If you still have concerns, please share them soon so the authors have enough time to respond.

Thank you for your help.

Your AC

Dear Reviewer vz6X,

There are fewer than 12 hours left in the author-reviewer discussion period, and we have not yet received your feedback on the rebuttal.

Please read the authors' responses and share your feedback as soon as possible. Please let the authors know whether your concerns have been addressed. If any concerns remain, please point them out so the authors have time to respond.

Thank you for your help.

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.