- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SHIFT proposes a novel approach to RGB-to-thermal adaptation with limited additional thermal supervision and thermal-only deployment. It constructs prediction changes from paired RGB-thermal data and projects them to a receiver response space orthogonal to a native training trajectory, and freeze a prediction family with weights estimated from thermal labels. Experiments cover various domains such as segmentation, detection, monocular depth estimation and stereo depth estimation.
Critical strengths
There is a well-motivated separation of direction construction and supervised adaptation. Constructing a restricted prediction family before accessing labels are a novel approach to adaptation with scarce additional supervision. Conditioning that family on selected receiver responses connect the transferred signal to the permitted update mechanism, while signed coefficient estimation allow the labels to determine how the candidate changes should be used. This combination offers a principled and potentially practical approach to adapting thermal models with scarce labels, while preserving thermal-only deployment, which is a current need in the thermal imaging community.
Critical weaknesses
1. The usefulness of the trajectory reference and the persistence of the selected geometry remain insufficiently established.
Equation 5 correctly enforces local reachability and orthogonality to the selected trajectory span E, but does not establish that excluding E improves adaptation. Useful corrections can follow directions already traversed during native training. Table 11b in the appendix supports including trajectory conditioning, but does not establish whether its benefit depends on the particular checkpoint selection or on reducing the available response dimension. Moreover, calibration uses the endpoint derivative DG, which can differ from the initially selected directions W. The paper therefore needs to establish whether the final updates preserve the geometric property used to motivate selection. Comparison of trajectory exclusion against rank-matched random reference subspaces, as well as quantification of final update trajectory overlap and deviation from u_beta (from equation 7) would provide further establishment, reporting task performance alongside both analysis.
2. The controls are insufficiently specified, and the reliability of the gains remains unclear.
Section 4.4 and table 4 matches direction count, weight estimation and Fisher energy, but does not report how the controls differ in effective endpoint rank or the response Gram matrix, which affects calibration through equation 12. This distinction matter because section 4.1 does report that the 7 detection coefficients yield only 3 independent endpoint responses. Section 4.4 also describes "sign-isometric" only as a fixed equal energy perturbation, leaving its precise construction unclear. The authors should specify the control transformations and report their effective endpoint ranks and Gram matrix from equation 10. Furthermore, table 2-4 show slight improvement in some metrics without uncertainty estimates, making the consistency of the improvements difficult to assess. Clarification of uncertainty estimates would be beneficial.
Questions for authors
- Given that section 3.1 treats the native receiver and its trajectory as supplied resources, how many labeled thermal scenes were used to construct them, and what dataset size defines the 10% label budget in tables 1–2? This would clarify the total supervision required for the reported results.
- How are the observer candidates in section 3.4 constructed from the observer predictions? Please distinguish their construction from that of the receiver response probes described in section 4.1, since these determine different parts of the prediction family.
- For each parameterized receiver, what precise update procedure defines G in equation 8? Please specify where the coefficients are injected, which data and losses drive the subsequent updates, and how many optimization steps are performed. These details determine what adaptation procedure the reported results actually evaluate.
- For monocular depth, do the 9 fixed prediction functions described in section 4.1 require separate model evaluations at inference, or do they share computation? What additional inference cost does this introduce relative to Native-H?
6 comments
Response to Reviewer zaLq (2/4)
W2. Control transformations, endpoint ranks, and gain reliability
Sample-shuffled permutes Fisher-whitened candidates by whole images. All positions, output components, and candidate columns of one image move together. We project the result back to the same , restore each column to its original Fisher energy, and recalibrate with the same support and regularization.
Sign-isometric uses an orthonormal basis of whitened and an orthogonal matrix to construct
This preserves the local Fisher Gram of the candidate directions. It changes the candidate subspace on MSRS. On M3FD, Mono, and Stereo, the complete prediction family is unchanged. The control tests the calibration prior induced by endpoint-column normalization. The M3FD comparison in W1 tests the trajectory reference at the same retained dimension and endpoint rank.
All three methods have the same effective endpoint ranks, and their normalized Gram spectra are similar. For Eq. (10), , we define and . The table reports the common rank and the nonzero spectral condition numbers of .
| Task | Rank | SHIFT | Shuffled | Isometric |
|---|---|---|---|---|
| MSRS | 9 | 1.66 | 1.69 | 1.72 |
| M3FD | 3 | 2.62 | 2.61 | 2.56 |
| Mono, fold 0 | 5 | 4.24 | 4.51 | 4.24 |
| Mono, fold 1 | 5 | 4.11 | 4.34 | 4.41 |
| Stereo | 4 | 8.27 | 7.70 | 8.99 |
For the seven detection coefficients, the three nonzero eigenvalues are [3.472, 2.203, 1.324], [3.500, 2.159, 1.341], and [3.443, 2.212, 1.345], in the same method order. Each method has three independent endpoint responses. Mono uses 792/791 and 791/792 calibration/evaluation inputs in its two folds and reports out-of-fold (OOF) results. The Full rank entry in Table 12 refers to the original nine-function Gram , with rank nine. The endpoint rank of five in Table 8 refers to the conditioned endpoint responses.
Recognition gains have positive paired 95% confidence intervals. We use 2,000 paired bootstrap resamples of evaluation predictions with fixed models and support, and recompute detection AP from the resampled detections. Full-FT denotes Native-H + Full-FT. Gains are in mIoU or AP points.
| Task | Control | Gain | 95% CI |
|---|---|---|---|
| MSRS mIoU | Full-FT | +2.97 | [2.43, 3.56] |
| M3FD AP | Full-FT | +1.99 | [1.25, 2.63] |
| MSRS mIoU | Sample-shuffled | +5.45 | [4.98, 5.91] |
| MSRS mIoU | Sign-isometric | +5.65 | [5.22, 6.09] |
| M3FD AP | Sample-shuffled | +3.96 | [3.26, 4.57] |
| M3FD AP | Sign-isometric | +3.40 | [2.68, 4.10] |
In the main depth evaluation, Mono SI-AbsRel improves by 0.0037 and Stereo EPE by 0.0039 px; 5-shot Mono improves by 0.0004. Against Sample-shuffled and Sign-isometric, Mono OOF improves by 0.0010/0.0013 and Stereo by 0.0047/0.0057 px.
Additional evaluation uses six held-out MS2 sequences disjoint from the original experimental sequences, with three day and three night sequences. Each contributes 64 uniformly sampled frames, giving 384 frames. Both tasks share these depth annotations, which count separately as evaluation GT. Native training remains at 64 unique labeled inputs and support at 1,583. Models and calibration coefficients remain fixed. All comparisons use the same frames, GT, and valid masks. We compute metrics per sequence and average the six sequences equally.
Mono uses per-image median scaling and compares with Native-H + LoRA-r8. Stereo compares with Native-H + Full-FT in the disparity coordinates of the paper. Drop is the relative error reduction; Improved counts sequences with lower error.
| Task | Baseline | SHIFT | Drop | Improved |
|---|---|---|---|---|
| Mono SI-AbsRel | 0.136705 | 0.131140 | 4.07% | 6/6 |
| Stereo EPE (px) | 0.411824 | 0.406073 | 1.40% | 5/6 |
These results show that the depth gains extend to additional independent sequences.
Response to Reviewer zaLq (3/4)
Q1. Native labels, trajectory construction, and the 10% denominator
Native-H uses 64 unique labeled inputs per task. MSRS uses nine-class segmentation labels and M3FD uses six-class detection labels. Mono and Stereo share depth labels from the same 64 timestamps in one native sequence; Stereo converts depth to disparity through . The trajectory uses checkpoints from the same native training. Table 7 reports 4,000 optimizer updates for MSRS/M3FD and 1,200 for Mono/Stereo.
The 5/20-shot and 10% budgets count additional labels after Native-H training. Pool denotes the calibration population, Support the 10% subset, and Dev the development scoring set. Each Train/Dev total includes the 64 native inputs.
| Task | Pool | Support | Dev | Extra | Train/Dev |
|---|---|---|---|---|---|
| MSRS | 911 | 91 | 108 | 0 | 263 |
| M3FD | 2,550 | 255 | 337 | 0 | 656 |
| Mono | 15,826 | 1,583 | 48 | 43 | 1,738 |
| Stereo | 15,826 | 1,583 | 48 | 55 | 1,750 |
Train/Dev is the deduplicated union of Native, Support, Dev, and Extra. Extra counts unique historical development or diagnostic inputs outside the other three sets. For Stereo, the 55 comprise 53 historical development inputs and two additional unique diagnostic inputs. Independent evaluation GT is counted separately, including the 384 shared depth annotations in W2. Native, support, and development scoring sets are disjoint, with scene groups also separated on M3FD. All methods share Native-H and support, with S_5\subset S_{20}\subset S_{10\\%}.
MSRS/M3FD RGB task training uses 884/2,768 labeled inputs; including RGB supervision gives deduplicated totals of 1,083/3,360. Shared depth labels are counted once across Mono and Stereo.
Q2. Observer candidates and receiver probes
Observer candidates describe changes proposed by RGB predictions. Receiver probes in Appendix B.2 specify allowed updates whose prediction responses form . The common observer displacement is . MSRS weights the complete by each of nine RGB class probabilities and uses 53 LoRA/head probes. M3FD uses six foreground weights and one background weight for its candidates, and seven class-conditioned probes.
Mono candidates use nine regions of a grid, with constant log-scale removed; its probes are the displacements of nine fixed thermal prediction functions. Stereo multiplies the common disparity displacement by seven stage-change weights and one global weight equal to one. Its eight stage probes correspond to the eight refinement updates in the public Fast-FoundationStereo (CVPR 2026) configuration. Parameter probes use negative mean gradients of unlabeled paired losses, followed by parameter-space L2 normalization. Candidate projection onto gives the prediction directions, and support labels determine their combination coefficients.
For Mono, collects the responses of the nine fixed functions. Projection of the regional candidates through Eq. (B.5) in Appendix B.3 gives . Discovery fixes , and support labels estimate nine-dimensional . The final family is
In Eq. (B.7) of Appendix B.4, the global combination feature is , and the function coefficient is . The endpoint response matrix has nine columns and rank five, as reported in Table 8.
Response to Reviewer zaLq (1/4)
Thank you for your careful review and insightful suggestions.
W1. Trajectory reference and geometry after finite updates
The native trajectory outperforms random references with the same rank. On M3FD, all comparisons use the same 255 support inputs. We construct five random four-dimensional constraints within the seven-dimensional receiver response space. Dim./Rank gives the retained response-space dimension and effective endpoint rank.
The five random references range from 24.72 to 30.94 AP. The native reference improves AP by 8.03 over the random mean and by 5.11 over the unconstrained case. Together with the trajectory ablation in Table 11(b), this comparison supports the contribution of trajectory information at a matched dimension.
Changing checkpoints preserves similar performance. The original checkpoints are at steps 0, 1,000, 2,000, 3,000, and 4,000. The alternative uses 0, 500, 2,000, 3,500, and 4,000.
We measure geometry on the actual final prediction at the calibrated coefficients. Let f⋆=LogqHGt(β⋆). Following Appendix A.5, we transport the trajectory and selected directions to the endpoint tangent space and compute
rE=∣f⋆∣gH2∣PEHf⋆∣gH2,eu=∣f⋆∣gH∣f⋆−WHβ⋆∣gH.
Here rE is the fraction of final Fisher energy in the native trajectory, and eu measures deviation from the selected direction combination. All rE values use the original native trajectory; each eu uses the direction combination of that configuration.
For MSRS, Mono, and Stereo, rE is 0.07%, approximately zero, and 0.31%, respectively, and eu is 7.23%, zero, and 9.09%. Mono combines fixed functions linearly in centered log-depth. These measurements quantify how the selected geometry persists through the complete receiver update.