acceptodds
Under review as a conference paper at ICLR 2027

SHIFT: Receiver-Conditioned Prediction-Space Transfer for Low-Data RGB-to-Thermal Adaptation

Abstract

Large-scale data and pretraining give RGB models rich semantic and geometric knowledge, whereas thermal perception remains limited by scarce and expensive task annotations. We study low-data RGB-to-thermal adaptation with a pretrained RGB model, abundant unlabeled RGB-thermal pairs, and only a small thermal support set, while keeping deployment thermal-only. Existing domain adaptation and cross-modal distillation methods transfer features, teacher predictions, or pseudo-labels, but they do not identify which transferred changes are new to the current thermal model and executable by its permitted update mechanism. We introduce SHIFT, a receiver-conditioned prediction-space transfer framework that selects RGB-induced changes that are novel relative to the native thermal trajectory and executable by the designated thermal receiver. Using predictive Fisher geometry, SHIFT removes task-invariant and trajectory-explained components, projects the remaining change onto the receiver's locally executable response space, and freezes a low-dimensional family of executable prediction directions. Thermal labels then estimate only their signed combination. On MSRS semantic segmentation, M3FD object detection, and MS2 monocular and stereo depth estimation, full SHIFT improves over four adaptation strategies initialized from the native thermal baseline under a 10% label budget. Shared-resource comparisons and 5/20-shot evaluations further support the value of receiver-conditioned filtering and low-dimensional calibration.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
RGB-to-thermal adaptation, infrared vision, low-data learning, cross-modal transfer

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_zaLqOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SHIFT proposes a novel approach to RGB-to-thermal adaptation with limited additional thermal supervision and thermal-only deployment. It constructs prediction changes from paired RGB-thermal data and projects them to a receiver response space orthogonal to a native training trajectory, and freeze a prediction family with weights estimated from thermal labels. Experiments cover various domains such as segmentation, detection, monocular depth estimation and stereo depth estimation.

Critical strengths

There is a well-motivated separation of direction construction and supervised adaptation. Constructing a restricted prediction family before accessing labels are a novel approach to adaptation with scarce additional supervision. Conditioning that family on selected receiver responses connect the transferred signal to the permitted update mechanism, while signed coefficient estimation allow the labels to determine how the candidate changes should be used. This combination offers a principled and potentially practical approach to adapting thermal models with scarce labels, while preserving thermal-only deployment, which is a current need in the thermal imaging community.

Critical weaknesses

1. The usefulness of the trajectory reference and the persistence of the selected geometry remain insufficiently established.

Equation 5 correctly enforces local reachability and orthogonality to the selected trajectory span E, but does not establish that excluding E improves adaptation. Useful corrections can follow directions already traversed during native training. Table 11b in the appendix supports including trajectory conditioning, but does not establish whether its benefit depends on the particular checkpoint selection or on reducing the available response dimension. Moreover, calibration uses the endpoint derivative DG, which can differ from the initially selected directions W. The paper therefore needs to establish whether the final updates preserve the geometric property used to motivate selection. Comparison of trajectory exclusion against rank-matched random reference subspaces, as well as quantification of final update trajectory overlap and deviation from u_beta (from equation 7) would provide further establishment, reporting task performance alongside both analysis.

2. The controls are insufficiently specified, and the reliability of the gains remains unclear.

Section 4.4 and table 4 matches direction count, weight estimation and Fisher energy, but does not report how the controls differ in effective endpoint rank or the response Gram matrix, which affects calibration through equation 12. This distinction matter because section 4.1 does report that the 7 detection coefficients yield only 3 independent endpoint responses. Section 4.4 also describes "sign-isometric" only as a fixed equal energy perturbation, leaving its precise construction unclear. The authors should specify the control transformations and report their effective endpoint ranks and Gram matrix from equation 10. Furthermore, table 2-4 show slight improvement in some metrics without uncertainty estimates, making the consistency of the improvements difficult to assess. Clarification of uncertainty estimates would be beneficial.

Questions for authors
  1. Given that section 3.1 treats the native receiver and its trajectory as supplied resources, how many labeled thermal scenes were used to construct them, and what dataset size defines the 10% label budget in tables 1–2? This would clarify the total supervision required for the reported results.
  2. How are the observer candidates in section 3.4 constructed from the observer predictions? Please distinguish their construction from that of the receiver response probes described in section 4.1, since these determine different parts of the prediction family.
  3. For each parameterized receiver, what precise update procedure defines G in equation 8? Please specify where the coefficients are injected, which data and losses drive the subsequent updates, and how many optimization steps are performed. These details determine what adaptation procedure the reported results actually evaluate.
  4. For monocular depth, do the 9 fixed prediction functions described in section 4.1 require separate model evaluations at inference, or do they share computation? What additional inference cost does this introduce relative to Native-H?
6 comments
AuthorsOpenReview ↗

Response to Reviewer zaLq (1/4)

Thank you for your careful review and insightful suggestions.

W1. Trajectory reference and geometry after finite updates

The native trajectory outperforms random references with the same rank. On M3FD, all comparisons use the same 255 support inputs. We construct five random four-dimensional constraints within the seven-dimensional receiver response space. Dim./Rank gives the retained response-space dimension and effective endpoint rank.

ReferenceAPDim./RankrEr_E (%)eue_u (%)
Native trajectory35.993/30.339.62
No constraint30.887/738.648.76
Random, mean of 527.963/360.898.05
Alternative checkpoints35.833/30.419.56

The five random references range from 24.72 to 30.94 AP. The native reference improves AP by 8.03 over the random mean and by 5.11 over the unconstrained case. Together with the trajectory ablation in Table 11(b), this comparison supports the contribution of trajectory information at a matched dimension.

Changing checkpoints preserves similar performance. The original checkpoints are at steps 0, 1,000, 2,000, 3,000, and 4,000. The alternative uses 0, 500, 2,000, 3,500, and 4,000.

We measure geometry on the actual final prediction at the calibrated coefficients. Let f⋆=Log⁡qHGt(β⋆)f^\star=\operatorname{Log}_{q_H}G_t(\beta^\star). Following Appendix A.5, we transport the trajectory and selected directions to the endpoint tangent space and compute

rE=∣PEHf⋆∣gH2∣f⋆∣gH2,eu=∣f⋆−WHβ⋆∣gH∣f⋆∣gH.r_E=\frac{\\|P_{E_H}f^\star\\|_{g_H}^2}{\\|f^\star\\|_{g_H}^2},\qquad e_u=\frac{\\|f^\star-W_H\beta^\star\\|_{g_H}}{\\|f^\star\\|_{g_H}}.

Here rEr_E is the fraction of final Fisher energy in the native trajectory, and eue_u measures deviation from the selected direction combination. All rEr_E values use the original native trajectory; each eue_u uses the direction combination of that configuration.

For MSRS, Mono, and Stereo, rEr_E is 0.07%, approximately zero, and 0.31%, respectively, and eue_u is 7.23%, zero, and 9.09%. Mono combines fixed functions linearly in centered log-depth. These measurements quantify how the selected geometry persists through the complete receiver update.

AuthorsOpenReview ↗

Response to Reviewer zaLq (2/4)

W2. Control transformations, endpoint ranks, and gain reliability

Sample-shuffled permutes Fisher-whitened candidates by whole images. All positions, output components, and candidate columns of one image move together. We project the result back to the same TT, restore each column to its original Fisher energy, and recalibrate with the same support and regularization.

Sign-isometric uses an orthonormal basis UU of whitened TT and an orthogonal matrix OO to construct

R=Odiag⁡(±1)O⊤,Wˉ′=URU⊤Wˉ.R=O\operatorname{diag}(\pm1)O^\top,\qquad \bar W'=URU^\top\bar W.

This preserves the local Fisher Gram of the candidate directions. It changes the candidate subspace on MSRS. On M3FD, Mono, and Stereo, the complete prediction family is unchanged. The control tests the calibration prior induced by endpoint-column normalization. The M3FD comparison in W1 tests the trajectory reference at the same retained dimension and endpoint rank.

All three methods have the same effective endpoint ranks, and their normalized Gram spectra are similar. For Eq. (10), G=B⊤FSBG=B^\top F_SB, we define Sjj=GjjS_{jj}=\sqrt{G_{jj}} and G~=S−1GS−1\widetilde G=S^{-1}GS^{-1}. The table reports the common rank and the nonzero spectral condition numbers of G~\widetilde G.

TaskRankSHIFTShuffledIsometric
MSRS91.661.691.72
M3FD32.622.612.56
Mono, fold 054.244.514.24
Mono, fold 154.114.344.41
Stereo48.277.708.99

For the seven detection coefficients, the three nonzero eigenvalues are [3.472, 2.203, 1.324], [3.500, 2.159, 1.341], and [3.443, 2.212, 1.345], in the same method order. Each method has three independent endpoint responses. Mono uses 792/791 and 791/792 calibration/evaluation inputs in its two folds and reports out-of-fold (OOF) results. The Full rank entry in Table 12 refers to the original nine-function Gram J⊤FPJJ^\top F_PJ, with rank nine. The endpoint rank of five in Table 8 refers to the conditioned endpoint responses.

Recognition gains have positive paired 95% confidence intervals. We use 2,000 paired bootstrap resamples of evaluation predictions with fixed models and support, and recompute detection AP from the resampled detections. Full-FT denotes Native-H + Full-FT. Gains are in mIoU or AP points.

TaskControlGain95% CI
MSRS mIoUFull-FT+2.97[2.43, 3.56]
M3FD APFull-FT+1.99[1.25, 2.63]
MSRS mIoUSample-shuffled+5.45[4.98, 5.91]
MSRS mIoUSign-isometric+5.65[5.22, 6.09]
M3FD APSample-shuffled+3.96[3.26, 4.57]
M3FD APSign-isometric+3.40[2.68, 4.10]

In the main depth evaluation, Mono SI-AbsRel improves by 0.0037 and Stereo EPE by 0.0039 px; 5-shot Mono improves by 0.0004. Against Sample-shuffled and Sign-isometric, Mono OOF improves by 0.0010/0.0013 and Stereo by 0.0047/0.0057 px.

Additional evaluation uses six held-out MS2 sequences disjoint from the original experimental sequences, with three day and three night sequences. Each contributes 64 uniformly sampled frames, giving 384 frames. Both tasks share these depth annotations, which count separately as evaluation GT. Native training remains at 64 unique labeled inputs and support at 1,583. Models and calibration coefficients remain fixed. All comparisons use the same frames, GT, and valid masks. We compute metrics per sequence and average the six sequences equally.

Mono uses per-image median scaling and compares with Native-H + LoRA-r8. Stereo compares with Native-H + Full-FT in the disparity coordinates of the paper. Drop is the relative error reduction; Improved counts sequences with lower error.

TaskBaselineSHIFTDropImproved
Mono SI-AbsRel0.1367050.1311404.07%6/6
Stereo EPE (px)0.4118240.4060731.40%5/6

These results show that the depth gains extend to additional independent sequences.

AuthorsOpenReview ↗

Response to Reviewer zaLq (3/4)

Q1. Native labels, trajectory construction, and the 10% denominator

Native-H uses 64 unique labeled inputs per task. MSRS uses nine-class segmentation labels and M3FD uses six-class detection labels. Mono and Stereo share depth labels from the same 64 timestamps in one native sequence; Stereo converts depth to disparity through d=fB/Zd=fB/Z. The trajectory uses checkpoints from the same native training. Table 7 reports 4,000 optimizer updates for MSRS/M3FD and 1,200 for Mono/Stereo.

The 5/20-shot and 10% budgets count additional labels after Native-H training. Pool denotes the calibration population, Support the 10% subset, and Dev the development scoring set. Each Train/Dev total includes the 64 native inputs.

TaskPoolSupportDevExtraTrain/Dev
MSRS911911080263
M3FD2,5502553370656
Mono15,8261,58348431,738
Stereo15,8261,58348551,750

Train/Dev is the deduplicated union of Native, Support, Dev, and Extra. Extra counts unique historical development or diagnostic inputs outside the other three sets. For Stereo, the 55 comprise 53 historical development inputs and two additional unique diagnostic inputs. Independent evaluation GT is counted separately, including the 384 shared depth annotations in W2. Native, support, and development scoring sets are disjoint, with scene groups also separated on M3FD. All methods share Native-H and support, with S_5\subset S_{20}\subset S_{10\\%}.

MSRS/M3FD RGB task training uses 884/2,768 labeled inputs; including RGB supervision gives deduplicated totals of 1,083/3,360. Shared depth labels are counted once across Mono and Stereo.

Q2. Observer candidates and receiver probes

Observer candidates describe changes proposed by RGB predictions. Receiver probes in Appendix B.2 specify allowed updates whose prediction responses form J=A0DJ=A_0D. The common observer displacement is Δ=QLog⁡p0(qO)\Delta=Q\operatorname{Log}_{p_0}(q_O). MSRS weights the complete Δ\Delta by each of nine RGB class probabilities and uses 53 LoRA/head probes. M3FD uses six foreground weights and one background weight for its candidates, and seven class-conditioned probes.

Mono candidates use nine regions of a 3×33\times3 grid, with constant log-scale removed; its probes are the displacements of nine fixed thermal prediction functions. Stereo multiplies the common disparity displacement by seven stage-change weights aj=∣dj−d0∣/mean⁡∣dj−d0∣a_j=|d_j-d_0|/\operatorname{mean}|d_j-d_0| and one global weight equal to one. Its eight stage probes correspond to the eight refinement updates in the public Fast-FoundationStereo (CVPR 2026) configuration. Parameter probes use negative mean gradients of unlabeled paired losses, followed by parameter-space L2 normalization. Candidate projection onto T=Im⁡(J)∩E⊥T=\operatorname{Im}(J)\cap E^\perp gives the prediction directions, and support labels determine their combination coefficients.

For Mono, JJ collects the responses of the nine fixed functions. Projection of the regional candidates through Eq. (B.5) in Appendix B.3 gives W=JCimplW=JC_{\mathrm{impl}}. Discovery fixes Cimpl∈R9×9C_{\mathrm{impl}}\in\mathbb R^{9\times9}, and support labels estimate nine-dimensional β\beta. The final family is

α=Cimplβ,log⁡Gt(β)(x)=log⁡qH(x)+∑ℓ=19αℓQx[log⁡qℓ(x)−log⁡qH(x)].\alpha=C_{\mathrm{impl}}\beta,\qquad \log G_t(\beta)(x)=\log q_H(x)+\sum_{\ell=1}^{9}\alpha_\ell Q_x[\log q_\ell(x)-\log q_H(x)].

In Eq. (B.7) of Appendix B.4, the global combination feature is ϕℓ1(x)≡1\phi_{\ell1}(x)\equiv1, and the function coefficient is αℓ=(Cimplβ)ℓ\alpha_\ell=(C_{\mathrm{impl}}\beta)_\ell. The endpoint response matrix BB has nine columns and rank five, as reported in Table 8.

Reviewer_gM1tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SHIFT, a framework for transferring knowledge from a pretrained RGB model to an already trained thermal model using unlabeled RGB–thermal pairs and a small number of additional thermal labels. Instead of directly copying RGB predictions, SHIFT removes changes already explained by the thermal model's training trajectory and selects directions that the receiver can realize through its allowed updates. These directions are identified using Fisher geometry, and their combination weights are calibrated with limited thermal supervision. Experiments on MSRS segmentation, M3FD detection, and MS2 monocular/stereo depth show improvements over Head, SSF, LoRA, and Full Fine-tuning baselines. Under the 10% additional label budget, SHIFT achieves 61.30 mIoU on MSRS and 35.99 AP on M3FD, compared with 58.33 and 34.00 for the strongest baselines. The paper also provides transfer-objective comparisons, direction controls, ablations, and theoretical analysis.

Critical strengths
  1. The paper introduces a meaningful receiver-conditioned transfer framework that selects prediction directions based on the thermal model's existing training trajectory and update capabilities, providing an interesting alternative to conventional knowledge distillation.

  2. The shuffled and isometric direction controls, together with the trajectory-conditioning and receiver-realization ablations, provide useful evidence that meaningful direction selection contributes to the improvements beyond low-dimensional calibration alone.

Critical weaknesses
  1. The total thermal supervision budget is not sufficiently clear because Native-H is already trained with labeled thermal data (4,000 steps for MSRS/M3FD and 1,200 for the depth tasks), while the reported 5/20-shot and 10% label-budget results only count additional adaptation labels, making it difficult to assess the overall label efficiency without knowing the number of labeled scenes used for native training.

  2. The claimed receiver executability is evaluated within a limited predefined probe space, while the construction of multiple observer candidates and sensitivity to probe design remain unclear, and relatively small improvements such as stereo EPE from 0.3918 to 0.3879 are reported without sufficient variance analysis or empirical validation of the selected directions after finite updates.

Questions for authors
  1. Could the authors report the number of unique labeled thermal scenes and annotations used to train Native-H for each task, and clarify the total supervision budget behind the reported low-data results?

  2. How are the multiple observer candidates (v_O^{(j)}) constructed, how is the calibration dimension (d) determined, and how does it relate to the receiver probes described in Appendix B.2? In particular, what are the nine fixed prediction functions used for MS2 Mono?

3 comments
AuthorsOpenReview ↗

Response to Reviewer gM1t (1/3)

Thank you for your thoughtful review and constructive feedback.

W1. Native supervision and the total label budget

Native-H uses 64 unique labeled inputs per task. MSRS uses 64 images with pixelwise labels for nine classes, and M3FD uses 64 images with boxes and labels for six classes. MS2 Mono and Stereo share metric depth annotations at 64 timestamps from one native sequence; Stereo converts depth to disparity as d=fB/Zd=fB/Z. The 4,000 updates for MSRS/M3FD and 1,200 for Mono/Stereo are optimizer steps, as specified in Table 7.

The 5/20-shot and 10% budgets count additional adaptation labels after Native-H. Pool denotes the calibration population, Support denotes the 10% subset, and Dev denotes development scoring inputs.

TaskPoolSupportDevExtraTrain/Dev
MSRS911911080263
M3FD2,5502553370656
Mono15,8261,58348431,738
Stereo15,8261,58348551,750

Train/Dev is the deduplicated union of Native, Support, Dev, and Extra. Extra counts unique historical development or diagnostic inputs outside the Native, Support, and Dev union. The Stereo Extra count comprises 53 additional historical development inputs and two new diagnostic inputs. Ground truth used only for independent evaluation is counted separately, including the 384 shared depth annotations in the supplementary evaluation described in W2.

All methods share Native-H and the same support set, with S_5\subset S_{20}\subset S_{10\\%}. Native, support, and development scoring inputs are disjoint; M3FD also separates scene groups. RGB task training uses 884 labeled inputs for MSRS and 2,768 for M3FD. Including RGB supervision, the corresponding deduplicated training and development totals are 1,083 and 3,360. Shared depth annotations are counted once across Mono and Stereo.

AuthorsOpenReview ↗

Response to Reviewer gM1t (2/3)

W2. Direction construction, probe sensitivity, and reliability

Probes define the update basis, and the geometric projection selects realizable prediction changes. With probes collected in DD, UR=Im⁡(D)U_R=\operatorname{Im}(D) and J=A0DJ=A_0D as in Appendix B.2. The selected space is T=Im⁡(J)∩E⊥T=\operatorname{Im}(J)\cap E^\perp. MSRS, M3FD, Mono, and Stereo use 53, 7, 9, and 8 probes, respectively. Observer candidates derive from RGB predictions through classes, spatial regions, or recurrent stages, and we project them onto TT. Q2 gives their construction and calibration dimensions.

Changing the M3FD probe design gives 36.00 AP compared with 35.99 AP for the original design. We retain seven class-conditioned probes and the same 255 support inputs, and change the objectness/class/box loss weights from 1:1:1 to 1:1:2. The principal angles between the two three-dimensional spaces TT are 8.175352, 15.367725, and 18.019691 degrees, with a projector Frobenius distance of 0.610159. This control changes the response space while maintaining detection performance.

Tables 1 to 3 show that SHIFT improves the primary metric over the compared adaptation methods on all four tasks at 5-shot, 20-shot, and 10% budgets. In the main 10% evaluation, Mono SI-AbsRel decreases from 0.1254 with Native-H + LoRA-r8 to 0.1217, a reduction of 0.0037, and Stereo EPE decreases from 0.3918 with Native-H + Full-FT to 0.3879 px, a reduction of 0.0039 px. At 5-shot, Mono SI-AbsRel decreases from 0.1300 with Native-H + SSF to 0.1296, a reduction of 0.0004.

A supplementary evaluation uses six independent MS2 sequences that are disjoint from the sequences in the original experiments. Three are daytime and three are nighttime; uniform sampling of 64 frames per sequence gives 384 frames. Mono and Stereo share the depth ground truth, which is counted as evaluation data. The 64 native inputs, 1,583 support inputs, trained models, and calibration coefficients remain fixed. All methods use the same frames, ground truth, and valid masks. We compute each sequence metric and then average equally across the six sequences. Mono uses per-image median scaling for SI-AbsRel; Stereo uses EPE in the disparity coordinates of the paper. Baselines are Native-H + LoRA-r8 for Mono and Native-H + Full-FT for Stereo. Drop is the relative error reduction, and Improved counts sequences with lower error under SHIFT.

TaskBaselineSHIFTDropImproved
Mono SI-AbsRel0.1367050.1311404.07%6/6
Stereo EPE (px)0.4118240.4060731.40%5/6

Recognition gains have positive paired 95% confidence intervals. We use 2,000 paired bootstrap resamples of evaluation results with fixed models and support, and recompute detection AP from the resampled detections. Both baselines are Native-H + Full-FT; gains are mIoU or AP points.

TaskBaselineSHIFTGain95% CI
MSRS mIoU58.3361.30+2.97[2.43, 3.56]
M3FD AP34.0035.99+1.99[1.25, 2.63]

We measure the geometry of the actual finite update. Let f⋆=Log⁡qHGt(β⋆)f^\star=\operatorname{Log}_{q_H}G_t(\beta^\star). Following Appendix A.5, we transport the trajectory and selected directions to the endpoint tangent space and compute

rE=∣PEHf⋆∣gH2∣f⋆∣gH2,eu=∣f⋆−WHβ⋆∣gH∣f⋆∣gH.r_E=\frac{\\|P_{E_H}f^\star\\|_{g_H}^2}{\\|f^\star\\|_{g_H}^2},\qquad e_u=\frac{\\|f^\star-W_H\beta^\star\\|_{g_H}}{\\|f^\star\\|_{g_H}}.

Here rEr_E measures the fraction of final Fisher energy in the native trajectory, and eue_u measures deviation from the selected direction combination. The task scores and geometric measurements are:

TaskMetricScorerEr_Eeue_u
MSRSmIoU61.300.07%7.23%
M3FDAP35.990.33%9.62%
MonoSI-AbsRel0.1217≈00
StereoEPE (px)0.38790.31%9.09%

The parameterized receivers retain low trajectory overlap with direction deviations from 7.23% to 9.62%. Mono combines fixed functions linearly in centered log-depth, so eu=0e_u=0.

AuthorsOpenReview ↗

Response to Reviewer gM1t (3/3)

Q1. Unique native annotations and total supervision

W1 specifies the 64 unique native inputs per task, their annotation types, and the deduplicated training and development budgets of 263, 656, 1,738, and 1,750 for MSRS, M3FD, Mono, and Stereo, respectively.

Q2. Observer candidates, calibration dimension, and the nine Mono functions

Observer candidates derive from the RGB displacement Δ=QLog⁡p0(qO)\Delta=Q\operatorname{Log}_{p_0}(q_O) at the common base point. Receiver probes specify allowed updates whose prediction responses form JJ. MSRS uses nine RGB class probabilities to weight Δ\Delta and forms probes from paired cross-entropy gradients restricted to 49 LoRA module groups and four head tensor groups. M3FD uses six foreground weights and one background weight to form candidates, with seven class-conditioned receiver probes.

Mono forms candidates from nine spatial regions in a 3×33\times3 grid and removes constant log-scale; its probes are the log-depth displacements of nine fixed thermal prediction functions. Stereo multiplies the common disparity displacement by seven stage-change weights aj=∣dj−d0∣/mean⁡∣dj−d0∣a_j=|d_j-d_0|/\operatorname{mean}|d_j-d_0| and one global weight of 1, with eight recurrent-stage disparity probes. Parameter probes use the negative mean gradient of the unlabeled paired loss, normalized in parameter-space L2 norm. The eight Stereo stage probes correspond to the eight recurrent refinements in the public implementation of Fast-FoundationStereo (CVPR 2026).

Appendix B.2 defines the receiver response matrix J=A0DJ=A_0D. Projecting observer candidates onto T=Im⁡(J)∩E⊥T=\operatorname{Im}(J)\cap E^\perp gives the calibration directions, with one coefficient per column. For MSRS, M3FD, Mono, and Stereo, dd is 9, 7, 9, and 8; probe counts are 53, 7, 9, and 8; and endpoint ranks are 9, 3, 5, and 4. Thus, the column count and independent endpoint response count have distinct roles. For example, M3FD uses seven coefficients for three independent endpoint responses.

The nine Mono functions use all combinations of λ∈0.1,0.3,1.0\lambda\in\\{0.1,0.3,1.0\\} and update counts m∈2,4,6m\in\\{2,4,6\\}. Each function starts independently from the same Native-H with a fresh AdamW state, reuses the native labels, and takes mm updates to minimize

Lnative centered-log L1+λ2Lpaired centered-log MSE.L_{\text{native centered-log L1}}+\frac{\lambda}{2}L_{\text{paired centered-log MSE}}.

The coefficients of the nine regional candidates map to global weights for the nine fixed functions through α=Cimplβ\alpha=C_{\mathrm{impl}}\beta. Discovery fixes Cimpl∈R9×9C_{\mathrm{impl}}\in\mathbb R^{9\times9} using the constrained projection in Appendix B.3, Eq. (B.5). Support labels estimate the nine-dimensional β\beta, and the conditioned endpoint response has rank 5 (Table 8). The final prediction is

log⁡Gt(β)=log⁡qH+∑ℓ=19(Cimplβ)ℓQ(log⁡qℓ−log⁡qH).\log G_t(\beta)=\log q_H+\sum_{\ell=1}^{9}(C_{\mathrm{impl}}\beta)_\ell Q(\log q_\ell-\log q_H).

All label budgets share Native-H, the 128 discovery pairs, and the frozen function family; support labels estimate the combination coefficients.

Reviewer_M7FnOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a method that adapts thermal models from frozen pretrained RGB models, through large scale unlabelled RGB-thermal pairs and a small set of thermal task labels. They demonstrate consistent gains over distillation and adaptation based baselines across segmentation, detection, monocular and stereo depth.

Critical strengths

The authors propose a new perspective to cross-modal adaptation, where the goal is to determine which of the observers proposed changes are new and achievable to the receiver. The supporting math is rigorous, with a predictive Fisher metric unifying discrete and continuous outputs, task typing map for output form, invariances and label complexity. the authors validate this approach on a wide range of tasks across multiple receiver types. demonstrating consistent gains and robustness of the method to varied tasks and frameworks

Critical weaknesses

None

Questions for authors

None

1 comment
AuthorsOpenReview ↗

Response to Reviewer M7Fn

Thank you for your thoughtful review and positive feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion and revisions

Dear Area Chair and Reviewers,

The discussion prompted a more direct test of whether the thermal model’s own training trajectory helps SHIFT choose what to transfer from RGB. On M3FD, we tuned regularization separately for each variant on the same development set. Test AP was 35.93 with the native trajectory, 30.88 without the trajectory constraint, and 30.29 on average across five rank-matched random references (Table 20).

We also clarified the label budget. Each native thermal model uses 64 labeled inputs. The 5-shot, 20-shot, and 10% budgets count additional adaptation labels. For segmentation and detection, the 128 unlabeled TartanRGBT pairs used to select transfer directions are disjoint from both RGB observers’ labeled training sets (Sec. 4.1).

We evaluated the fixed models on six additional MS2 sequences. SHIFT reduced monocular depth error on all six relative to LoRA-r8 and stereo error on five relative to full fine-tuning (Table 25). We welcome any further questions about these results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.