- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper targets binarized infrared-visible image fusion (IVIF) and proposes SDRTNet, a hybrid-precision network in which a full-precision encoder and decoder wrap a multi-scale backbone of binary blocks. The authors motivate the design with three coupled challenges: retaining local structural distinctions under binarization, preserving magnitude information during binary propagation, and training a limited-capacity binary network to learn complementary fusion patterns. Three modules are introduced accordingly. (1) Sliding Directional Relation Transform (SDRT): from sliding, overlapping patches the method computes one base component and four directional differences (horizontal, vertical, diagonal, anti-diagonal), processes the five components through independent binary branches, and recombines them with a fixed least-squares pseudo-inverse (ISDRT) plus overlap-averaging. The transform is shown to be invertible (, ), so information loss is attributed to binarization rather than to the transform. (2) Local Statistical Alignment (LSA): binary convolution features are standardized by their own local mean and variance, then re-scaled and re-shifted using the full-precision statistics of the raw SDRT features, and combined through a learnable gated residual. (3) Binary Vector-Quantized Knowledge Distillation (BVQKD): a full-precision teacher with the same SDRT-ISDRT topology is trained first, a VQ codebook and decoder are learned to reconstruct the teacher's fused output, and the binary student is distilled by matching teacher/student codeword-assignment distributions, which adds no inference cost. Validation: the model is trained on 2,000 LLVIP pairs and evaluated on LLVIP, TNO, M3FD and MSRS using six metrics (EN, MI, SF, EI, AG, CC) against five full-precision and five binary methods, plus downstream detection (M3FD, YOLOv11) and segmentation (MSRS, DeepLabV3+). Ablations cover the transform domain, normalization/alignment variants, knowledge-distillation objectives, codebook construction and residual-branch design. Reported efficiency is 0.38M parameters, 17.9 GFlops and 0.082 s inference at .
Critical strengths
- A genuinely under-explored problem, framed so that each claim is separately testable. Binary IVIF is largely untouched - the BNN literature concentrates on classification, super-resolution and restoration. Decomposing the difficulty into representation / propagation / training is not just rhetorical: it yields a modular design and, more importantly, an ablation suite (Tabs. 2-4, Fig. 6) in which every claim has a matching control. That discipline is above average for this venue.
Critical weaknesses
- The efficiency report should be made explicit rather than aggregate (W1, most worth fixing). The encoder and decoder are full precision, SDRT/ISDRT are computed in FP, and LSA injects full-precision local statistics plus a per-channel affine and gate into every block. This is a legitimate design, but the paper never states the binary-to-FP FLOP ratio, gives no module-level cost breakdown, and does not state the counting convention behind the 17.9 GFlops in Fig. 1 (are 1-bit MACs counted as 1 bit?). A short table would let readers attribute the advantage - the latency evidence in Fig. 1 already suggests the advantage is real, so this is a reporting gap, not a validity problem.
- Positioning against prior art would sharpen the contribution claim. Each module has a natural precedent: SDRT is a fixed over-complete difference/lifting transform; LSA is window normalization + affine + gated residual, which overlaps with ReActNet/IR-Net-style activation shaping; BVQKD combines VQ-VAE with KD. The paper does not discuss these links, and the Tab. 2 comparison (SDRT vs a critically-sampled DWT) changes both directionality and redundancy ( coefficients/pixel vs ), so the gain cannot yet be attributed to "directional relations" specifically. Two cheap additions would settle it: a redundancy-matched, non-directional transform, and a ReActNet/IR-Net-style baseline inside the same backbone.
Questions for authors
- Cost accounting. Could you give a module-level table of parameters, BOPs/MACs and latency (FP encoder, FP decoder, SDRT/ISDRT, LSA statistics, binary branches) and state the Fig. 1 counting convention? Even an approximate binary-to-FP FLOP ratio would resolve this.
- LSA: calibration or full-precision pathway? Since LSA reads full-precision , a control that injects the same statistics without the standardization/affine/gating (e.g. ) would clarify which mechanism is responsible for the gain.
2 comments
4. Periodic artifacts (Q4)
We examine TNO image pairs using enlarged low-texture regions, detrended residuals, gradient maps, and Fourier spectra. Each comparison uses the same region in the visible input, infrared input, and fused image. Nearest-neighbor enlargement preserves individual pixels. We also measure the combined energy at the horizontal, vertical, and checkerboard period-two frequencies, normalized by the total energy after removing a fitted intensity plane.
As shown in the uploaded supplementary material, the TNO fusion results exhibit no visible two-pixel-periodic grid artifacts near patch boundaries after overlap averaging. The period-two frequencies account for 0.018%–1.678% of the residual energy in the displayed fused regions. The region with the highest fraction contains a stronger period-two component in the visible input, at 5.317%. The enlarged regions, gradient maps, and Fourier spectra consistently support this observation.
5. Comparison methods (Q5)
We retain each baseline’s binary backbone and adapt its input and output interfaces to infrared–visible image fusion. All five binary baselines use a common training protocol: 2000 LLVIP training pairs, a 50-epoch schedule, a batch size of 4, and AdamW with an initial learning rate of 0.0002. They share the same base fusion loss as SDRTNet’s Stage 1. For the five full-precision methods, we use official checkpoints where available; otherwise, we train them using our Stage 1 protocol.
Regarding the E2FIF/BBCU LLVIP triples, Table 1 currently reports the same rounded (EN, MI, CC) values of (7.51,15.02,0.918). An independent re-evaluation of the archived LLVIP outputs with LinFeng Tang’s evaluation function gives (7.508452, 15.016904, 0.918442) for E2FIF and (7.510857, 15.021714, 0.918329) for BBCU.
6. Clarification of the format(Q6)
SDRTNet follows the binary-network formulation used by the compared baselines, including E2FIF, BBCU, FABNet, S2BNet, and BNNPan. Its binary branches use scaled sign-based weights and activations, represented as floating-point tensors and processed by standard convolution operators in our implementation, while the surrounding modules retain floating-point computation. Thus, our evaluation concerns binary representations rather than native packed 1-bit or INT8 execution. We focus on this setting because sign-based quantization removes magnitude information and can obscure local structural differences, which SDRT and LSA specifically address.
7. Deployment(Q7)
Given our limited budget for binary edge accelerators, we supplement inference time with a controlled resource-limited benchmark on a separate GPU platform(3090). We compare SDRTNet with five binary baselines using NVIDIA MPS active-thread limits of 100%, 50%, and 25%. All methods use FP32 execution, identical synthetic inputs at , batch size 1, and initialized weights for architecture-level timing. We report averages over 30 runs after 10 warm-up iterations. SDRTNet achieves the lowest latency and peak allocated GPU memory among these implementations.
| Method | Baseline time (ms) ↓ | Estimated 50% limit (ms) ↓ | Estimated 25% limit (ms) ↓ |
|---|---|---|---|
| E2FIF | 165.86 | 267.31 | 487.13 |
| BBCU | 689.68 | 899.29 | 1366.69 |
| FABNet | 331.17 | 485.13 | 817.07 |
| S2BNet | 137.70 | 201.20 | 358.22 |
| BNNPAN | 281.00 | 420.61 | 740.16 |
| Ours | 82.02 | 111.12 | 178.98 |
Response to vz6X
Thanks for your recongnization. We’ll address Weaknesses(W) and Questions(Q) below.
1. Cost accounting(W1, Q1)
Thank you for pointing out that Figure 1 does not define its counting convention. For one 1024×768 infrared–visible image pair, the SDRTNet has 376570 parameters (0.38 M). Its binary convolutions perform 8.4709 G MACs, and its full-precision convolutions perform 0.4894 G MACs, giving a binary-to-full-precision convolutional MAC ratio of 17.31:1. SDRTNet counts two operations per MAC for both binary and full-precision convolutions, with no bit-packing discount. This gives 2×(8.4709+0.4894)=17.9206 G convolutional operations.
LSA has zero convolutional MACs because it computes local statistics through reductions and average pooling, then applies full-precision normalization, affine scaling, and gated residual mixing. These operations contribute to inference time.
2. LSA(W2, Q2)
Following your suggestion, we train a direct-injection control with Y=Xbin+μraw+σraw. It retains the binary branch's learnable activation shift and RPReLU, uses the same local statistics as LSA, and removes standardization, the channel-wise affine transformation, and the gate. Direct injection adds the same local offset to every channel. Full LSA also uses the raw local standard deviation to scale the standardized channel responses.
On TNO, direct injection achieves EI of 27.06 and AG of 3.01, above the No norm values of 19.87 and 1.73. Full LSA achieves 39.85 and 3.97, respectively. These results suggest that access to full-precision statistics accounts for part of the improvement, while the complete calibration procedure provides an additional benefit.
3. Directionality and redundancy(W2, Q3)
Table 2 compares DWT and NSDRT on the same non-overlapping 2×2 patch grid. DWT computes four wavelet coefficients per patch, while NSDRT encodes a base component and four directional differences. Even on this non-overlapping grid, NSDRT achieves higher SF (8.12 vs. 7.53), EI (34.21 vs. 30.24), and AG (3.46 vs. 3.03) on TNO. These gains therefore occur before overlap is introduced. With the same five SDRT components, switching to overlapping windows further increases SF to 9.94, EI to 39.85, and AG to 3.97.
We retain the fifth component Ra=c−b because its sign records the ordering of the right and lower neighbors in a 2×2 patch. Although Ra=Rv−Rh is redundant in full precision, this ordering can disappear after binarization. For example, the patches [0123] and [0213] give the same four signs (+,+,+,+) for (M,Rh,Rv,Rd) at zero threshold. The larger increase is downward in the first patch and rightward in the second, which Ra distinguishes through opposite signs. To test the value of this distinction, we remove Ra and use the exact inverse of the four-component transform, so both variants reconstruct unmodified patches exactly. On TNO, retaining Ra increases SF from 7.35 to 9.94 and EI from 25.99 to 39.85.