- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors address the degradation caused by the combination of thermal persistence and rolling-shutter readout in uncooled infrared imaging. They define a spatio-temporal observation model that accounts for both phenomena and propose a framework that reconstructs sharp RS states at multiple time points from three consecutive thermal observations, subsequently rectifying them to global-shutter geometry. Additionally, they employ a Physical Reblur Loss to enforce consistency with the thermal response model. The authors construct a physics-based synthetic benchmark and collect separate real-world datasets using two uncooled infrared cameras, reporting performance improvements over existing reconstruction methods.
Critical strengths
- New synthetic benchmark was developed that simultaneously accounts for thermal persistence and RS distortion. By providing sharp RS/GS ground truth, it enables quantitative evaluation of reconstruction performance across multiple time points. The authors also collected real-world datasets using two different uncooled infrared cameras.
Critical weaknesses
-
Generalization across physical conditions has not been sufficiently verified. The quantitative evaluation uses a single configuration with a thermal time constant of 8 ms and an RS readout duration of 32 ms, and the test set is limited to four scenes. Furthermore, the evaluations on two real cameras are qualitative due to the lack of sharp GS ground truth. Therefore, it remains unclear whether the performance is maintained across sensors with different thermal responses or readout speeds.
-
The temporal and training protocols of the baselines are insufficiently explained. Although the authors state that all methods were retrained on the proposed dataset, the training schemes for the A+B pipelines, intermediate supervision, number of input frames, and handling of target query times remain unclear
Questions for authors
-
TRNeRF already models thermal persistence and RS distortion. Although it uses scene-specific optimization, could the authors clarify whether a comparison with TRNeRF, potentially adapted for temporal reconstruction, is feasible? If not, please explain why. Also, Appendix D reports only single-query inference time. Please provide peak GPU memory and total inference time for all six queries.
-
LPIPS typically uses features pretrained on RGB images, whereas thermal images are generally single-channel. Please clarify the LPIPS backbone and preprocessing used in the evaluation. Is there an established protocol or prior evidence that RGB-based LPIPS correlates with perceptual quality in thermal image restoration?
-
The method predicts only six of the 21 thermal history states, using GT for the remaining 15 during training. Beyond the 78% thermal-weight coverage, what justifies K=6? Please provide any existing K-sensitivity results, including K=21, and discuss the trade-off between reconstruction quality and computational cost.
-
Table 2 shows the benefit of Physical Reblur Loss, but does not isolate the contribution of the physical thermal model. Since the remaining 15 states use GT, the loss reduces to the L1 norm of an exponentially weighted sum of prediction errors over the six predicted states. How does it compare with uniform weights, mismatched thermal time constants, or weighted per-frame loss? Has this loss been evaluated on other reconstruction architectures?
10 comments
| Method | VIM384GS-USL | HIKMICRO HM-TM06-LF/ANT | ||||
|---|---|---|---|---|---|---|
| CLIP-IQA+ ↑ | EN ↑ | DOVER ↑ | CLIP-IQA+ ↑ | EN ↑ | DOVER ↑ | |
| PMBNet | 0.1595 | 6.1504 | 0.0116 | 0.2388 | 6.1442 | 0.0266 |
| DeMFI+JAMNet | 0.1616 | 6.2139 | 0.0120 | 0.2183 | 6.1700 | 0.0199 |
| DeMFI+DFRSC | 0.1672 | 6.2184 | 0.0121 | 0.2199 | 6.1697 | 0.0198 |
| BiT+JAMNet | 0.1414 | 6.1090 | 0.0112 | 0.2044 | 6.1460 | 0.0290 |
| BiT+DFRSC | 0.1459 | 6.1022 | 0.0109 | 0.2114 | 6.1502 | 0.0293 |
| BSST+UniVR | 0.1496 | 6.2306 | 0.0131 | 0.2174 | 6.1985 | 0.0373 |
| BSST+LBCNet | 0.1395 | 6.2512 | 0.0108 | 0.2176 | 6.2072 | 0.0307 |
| DST-Plus+UniVR | 0.1572 | 6.2222 | 0.0117 | 0.2441 | 6.1656 | 0.0296 |
| DST-Plus+LBCNet | 0.1534 | 6.2412 | 0.0094 | 0.2537 | 6.1671 | 0.0264 |
| PPFN+UniVR | 0.1487 | 6.1229 | 0.0116 | 0.1967 | 6.1213 | 0.0309 |
| PPFN+LBCNet | 0.1379 | 6.1468 | 0.0093 | 0.2037 | 6.1251 | 0.0265 |
| ECMR+UniVR | 0.1526 | 6.1176 | 0.0115 | 0.2120 | 6.1470 | 0.0258 |
| ECMR+LBCNet | 0.1481 | 6.1386 | 0.0097 | 0.2188 | 6.1496 | 0.0220 |
| Ours | 0.1965 | 6.2971 | 0.0137 | 0.2675 | 6.1590 | 0.0471 |
These real-world results provide complementary quantitative evidence that the proposed model remains effective across sensors with different thermal responses, readout speeds, resolutions, and imaging characteristics.
[1] Legal-Ayala H, et al. Entropy and contrast enhancement of infrared thermal images using the multiscale top-hat transform[J]. Entropy, 2019, 21(3): 244.
[2] Exploring clip for assessing the look and feel of images[C]//Proceedings of the AAAI conference on artificial intelligence. 2023, 37(2): 2555-2563.
[3] Exploring video quality assessment on user generated contents from aesthetic and technical perspectives[C]//2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023: 20087-20097.
W2: Training and temporal protocols of the compared baselines.
Response. We thank the reviewer for pointing out that the baseline training and temporal protocols were not sufficiently specified in the original manuscript. We have added a dedicated description in Appendix A.5 of the revised manuscript and clarify the complete protocol below.
All competing methods are retrained on our synthetic training set using their official implementations and original supervision strategies. PMBNet follows its original end-to-end training protocol, whereas all A+B pipelines are trained stage-wise, with explicit supervision for each component rather than jointly optimizing the two stages.
We preserve the native input configuration of each method whenever the number of input frames is fixed by its original architecture and official implementation, since forcing a common input length would require architectural modifications and deviate from the intended setting of the baseline. Specifically, PMBNet uses 2 input frames, BiT uses 3, DeMFI uses 4, JAMNet uses 2, DFRSC uses 3, PPFN and ECMR use 1, and UniVR and LBCNet use 2. For BSST and DST-Plus, whose input length is configurable, we use 3 consecutive frames, matching the three-frame input setting of our method.
The supervision and handling of target query times depend on the type of pipeline:
| Pipeline type | Stage A | Stage B | Training supervision | Handling of target query times |
|---|---|---|---|---|
| Sequence reconstruction + RSC | DeMFI / BiT | JAMNet / DFRSC | Stage A is supervised with temporally aligned sharp RS states; Stage B is independently supervised with the corresponding sharp GS targets | Stage A reconstructs the queried RS states, and Stage B performs RS-to-GS correction at the same temporal positions |
| Deblurring + RS interpolation | BSST / DST-Plus / PPFN / ECMR | UniVR / LBCNet | Stage A is supervised to recover the most recent sharp RS state; Stage B is independently supervised using temporally corresponding sharp RS endpoints and intermediate GS targets | Stage A provides the restored endpoint frames, and Stage B generates the required intermediate GS states at the same query times |
| End-to-end | PMBNet | -- | Original end-to-end supervision | Directly produces the requested reconstructed outputs |
For all methods, evaluation is performed at the same target times used by our method. Thus, although the compared methods have different native temporal architectures, their final outputs are evaluated against the same time-aligned GS references under an identical metric protocol.
These details have now been added to Appendix A.5, “Detailed Training Protocols of Compared Methods,” to make the comparison and reproduction protocol explicit.
Q1: Feasibility of comparison with TRNeRF and inference efficiency.
Response. We thank the reviewer for pointing out TRNeRF, which is indeed the closest prior work in terms of jointly modeling microbolometer thermal persistence and rolling-shutter readout. However, TRNeRF and our method solve substantially different inverse problems, which makes a direct adaptation non-trivial.
TRNeRF is a scene-specific neural scene reconstruction method. Given a sequence of degraded thermal images and a known camera trajectory, it incorporates thermal persistence and rolling-shutter readout into the NeRF rendering process, optimizes a scene-level radiance field, and then renders restored GS images from the learned scene representation. In contrast, our method is a feed-forward sequence reconstruction framework that directly takes three degraded thermal observations and reconstructs multiple sharp temporal states associated with the current observation, without constructing a 3D scene representation.
A key practical difference is the dependence on camera poses. TRNeRF assumes known per-frame camera poses and interpolates them into a continuous trajectory to determine the time-varying rays used for thermal-response integration and rolling-shutter rendering. TRNeRF itself reports that COLMAP fails on fast-motion thermal images due to motion blur and rolling-shutter distortion; it therefore estimates poses using additional monochrome cameras and transfers them to the thermal cameras through calibrated extrinsics. We observe the same difficulty on our real sequences: directly applying COLMAP to the degraded thermal observations does not recover a reliable trajectory. Our method instead operates directly on the thermal observations and requires neither camera poses nor auxiliary cameras, cross-sensor calibration, or scene-specific optimization.
Adapting TRNeRF to our temporal reconstruction setting would therefore require more than changing its output time. Its original formulation represents a scene through a radiance field and uses time primarily through the interpolated camera trajectory. To reconstruct multiple evolving scene states as required in our setting, particularly in the presence of independently moving objects, one would need to introduce a dynamic/time-conditioned scene representation in addition to removing or replacing the external pose requirement. These changes would substantially alter the original TRNeRF formulation rather than constitute a faithful baseline adaptation. We therefore regard TRNeRF as the closest physics-based related work, but not as a directly comparable baseline for our pose-free, feed-forward thermal sequence reconstruction setting.
We also thank the reviewer for requesting a more complete efficiency analysis. We have expanded the original single-query evaluation and now report single-query runtime, total runtime for all six reconstructed queries, and peak GPU memory for all compared methods under the same hardware and evaluation setting. The revised results are provided in Appendix F, Table 10 and reproduced below:
| Method | Single-Query Time (s) | Six-Query Time (s) | Peak Memory (MiB) |
|---|---|---|---|
| PMBNet | 0.3091 | 1.4767 | 1124 |
| DeMFI+JAMNet | 2.4975 | 2.7234 | 2524 |
| DeMFI+DFRSC | 0.5213 | 2.8235 | 5199 |
| BiT+JAMNet | 0.4218 | 1.4411 | 758 |
| BiT+DFRSC | 0.4568 | 1.5620 | 5982 |
| BSST+UniVR | 1.3290 | 1.3991 | 1288 |
| BSST+LBCNet | 1.3871 | 1.7430 | 2628 |
| DST-Plus+UniVR | 0.1930 | 0.2626 | 1708 |
| DST-Plus+LBCNet | 0.2508 | 0.6040 | 1916 |
| PPFN+UniVR | 0.5420 | 0.6204 | 3886 |
| PPFN+LBCNet | 0.5899 | 0.9471 | 3894 |
| ECMR+UniVR | 0.1240 | 0.1938 | 672 |
| ECMR+LBCNet | 0.1803 | 0.5450 | 1896 |
| Ours | 0.4979 | 1.6683 | 1954 |
Our method requires 0.4979 s for a single query and 1.6683 s for all six queries, with a peak GPU memory usage of 1954 MiB. The increase from single-query to six-query runtime differs across methods because their temporal-generation strategies are different. In particular, several deblurring + RS-interpolation pipelines perform the restoration stage once and subsequently generate multiple intermediate states through interpolation, resulting in a smaller additional cost for multi-query inference. We therefore report these measurements primarily as an efficiency reference rather than a strict architectural complexity ranking.
W1: Generalization across physical conditions and sensors.
Response. We thank the reviewer for highlighting this limitation. We agree that the original evaluation did not provide sufficient evidence for generalization across different physical imaging conditions and real sensors. In the revised manuscript, we strengthen the evaluation from three complementary aspects.
First, we add a controlled sensitivity study for the physical parameters. The model is trained at tau_th = 8 ms and T_r = 32 ms, and is directly evaluated on newly synthesized test sets with tau_th in {8, 10, 12} ms and T_r in {26, 32, 38} ms. We vary one parameter at a time while keeping the other fixed at its training value, and use the same trained model for all settings without retraining or fine-tuning. The results, added to Appendix B, Table 6 and Fig. 12 in the manuscript. We also reproduced the table below:
Here, * denotes the training setting. The reconstruction quality remains stable across these parameter variations, indicating that the model is not tightly restricted to the exact thermal-response and readout parameters used during training.
Second, regarding the limited number of test scenes, we clarify that the four reported scenes correspond to four top-level physical environments rather than four individual video sequences. Within these environments, the test set contains 21 independent sequences captured from different viewpoints and at different times, involving different moving targets, motion speeds, and motion patterns, including camera motion, object motion, and simultaneous camera-and-object motion. Therefore, although the benchmark contains four top-level environments, each environment covers multiple distinct motion and acquisition conditions. We nevertheless acknowledge that the number of top-level environments remains limited. To further verify that the reported average improvement is not dominated by a particular subset, we additionally provide per-scene results in Appendix E, Table 9.
Third, we substantially strengthen the real-world evaluation. The two real cameras operate under different physical conditions: VIM384GS-USL uses tau_th = 9 ms and T_r = 28 ms, whereas HIKMICRO HM-TM06-LF/ANT uses tau_th = 8 ms and T_r = 32 ms. The same model trained on the synthetic configuration is directly applied to both cameras without sensor-specific retraining or fine-tuning. Qualitative cross-sensor results are provided in Fig. 7 for VIM and Fig. 16 for HIKMICRO.
In addition, we add no-reference quantitative evaluation on both real-world datasets in Appendix H, Table 11, providing complementary assessment from infrared information content, frame-level perceptual quality, and video-level technical quality. Specifically, EN (Entropy) [1] is widely used in infrared image enhancement to characterize the information richness preserved in restored thermal images. CLIP-IQA+ [2] provides a no-reference assessment of per-frame perceptual quality; the original CLIP-IQA study explicitly evaluates fine-grained attributes such as sharpness and further validates the metric on real-world deblurring data, making it suitable for assessing the clarity and perceptual quality of the reconstructed frames. DOVER [3] provides a complementary video-level assessment from both aesthetic and technical perspectives, where the technical-quality branch is designed to capture degradations such as blur, artifacts, noise, and picture clearness. For CLIP-IQA+ and EN, we evaluate each reconstructed state independently and average the scores over the six states of each observation. For DOVER, the six reconstructed states are treated jointly as a short video sequence, thereby providing a sequence-level assessment of the reconstructed video quality. The results are reproduced below: