acceptodds
Under review as a conference paper at ICLR 2027

TF-DETR: Task-Factor Representation for RGB-X Object Detection

Abstract

RGB-X object detection typically represents RGB and auxiliary modalities as parallel feature streams and focuses on how their representations should be fused. However, given an already strong RGB representation, we observe that the additional task utility provided by X varies substantially across sensing modalities and detection tasks.Motivated by this observation, we propose TF-DETR (Task-Factor DETR), which preserves RGB as the primary semantic stream while representing the auxiliary modality through Task Factors that combine explicit detection-oriented states with flexible sensor-dependent latent evidence. To exploit these factors effectively, TF-DETR establishes cross-modal correspondence through global-anchored multi-scale task alignment, and incorporates aligned auxiliary evidence into the RGB encoder through reliability-aware query guidance and bounded low-rank feature refinement. This design allows X to guide and refine RGB representations without maintaining a second full semantic stream.Experiments on RGB-Thermal and RGB-Event object detection benchmarks demonstrate state-of-the-art performance with a lightweight auxiliary pathway. Code will be released publicly.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Computer Vision, Multi-modal Learning, Object Detection

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hNvYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed a framework for fusion of auxiliary information with RGB data based on their task-specific utility. The idea of using the auxiliary modality only if its adding value and ensuring that the powerful semantic cues obtained from RGB data are not corrupted by the auxiliary modality is interesting. This certainly has value in robotics and autonomous vehicles. Although the overall idea is novel there are several gaps in the theoretical formulation and in experimentation. Several architectural choices, variable definitions, operators used in equations and metric formulations are either left implicit or undefined in some cases. The experimental results with event camera and thermal camera as auxiliary modality show good results but the discussion on RGB-D (one of the most common case) is skipped. The ablation studies to study the effect of "Task-factor representation", "Task Reliability (g)" and "Low-Rank Refinement" could have been added to strengthen the claims.

Critical strengths

The main strength of the paper is the task-factor representation that allows to estimate the task reliability of the auxiliary modality. Combining the auxiliary features based on their task reliability is a good idea and seems to be working for noisy auxiliary modalities like (event cameras and thermal cameras).

Critical weaknesses

The weaknesses of the paper are:

  1. The explanation of sections 3.3 and 3.4 can be improved by defining all the variables used in the equations.
  2. The GAMTA module is not clear, why would you even need it if the task-factor representation for both RGB and the auxiliary module (X) is the same ?
  3. The paper fails to demonstrate the utility of the proposed method for RGB-D data. I checked the Appendix where the KITTI and SUN-D datasets are mentioned but I could not find a table of comparison of mAP with state of the art.
Questions for authors

Questions:

  1. In equation (1) the variables \Phi_X, E^l_X, \Phi_R, \Psi_R are not defined. Are these mapping functions multi-layer perceptron (MLP)? Do they share weights ?
  2. In equation (2) the vector concatenation is ambiguous, you use both square brackets and \oplus to represent concatenation. The dimensionality of Z_{p,m} and Z_{s,m} is not mentioned, I see that the total dimension of the task factor representation is 33 (from Appendix).
  3. Equation (3) is not clear, Is G^5 a rigid body transform ? What do you mean by multi-scale warping?
  4. Equation (4) is not clear. What is "p" in this equation? What is the value of parameter "K" ? How do you find the residual offset \delta_k ?
  5. What is H_{norm} in equation (6)?
  6. What is \sigma and S^l in equation (7) ?
  7. Why is equation (9) specifically reported for l=5 ? Will you not be doing for all the levels (l \in {3, 4, 5}) ?
  8. In the experiment tables can you also report the DETR output with the RGB path alone and vanilla fusion of auxiliary modality using standard cross attention? It will be good to see how much accuracy boost you are getting by fusing based on task utility.

Suggestions

  1. I would love to see more discussion on RGB-D datasets.
  2. You can look at the value of "task reliability (g)" and may be plot it for your datasets. It will be good to see how does it change for a given dataset.
  3. I am curious to see if you can get rid of GAMTA module completely, may be an ablation study with/without GAMTA might help.
  4. I am also curious to see if you can simplify the reliability estimation and compute it just from the objectness and geometry discrepancies.
5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

We sincerely thank the reviewer for the constructive comments and for recognizing the motivation and practical value of task-oriented auxiliary representation and reliability-aware fusion. We have clarified the mathematical definitions and architectural choices in Sections 3.2–3.4 (as shown in the revised paper), and strengthened the experiments with RGB-D comparisons, conventional-fusion baselines, component ablations, and robustness analyses.

We address all the weaknesses (W) and questions (Q) raised by the reviewer below.

1. Mapping functions and Task-Factor dimensions (Q1, Q2)

Q1: Definitions of the mapping functions in Eq. (1).

We have clarified the following definitions in Section 3.2:

  • EXlE_X^l denotes the level-specific mapping of the lightweight RepATE auxiliary encoder.

  • ψR\psi_R denotes the RGB task probe that projects the RGB semantic features into the Task-Factor space.

  • ΦX\Phi_X and ΦR\Phi_R denote the auxiliary and RGB Task-Factor projection heads, respectively.

These mappings are convolution-based rather than MLP-based, and their parameters are not shared across modalities or feature scales. The RGB task probe only provides a reference for task-space correspondence and reliability estimation; the original RGB semantic features remain intact in the primary detection pathway.

Q2: Concatenation and factor dimensions in Eq. (2).

We have explicitly defined ⊕\oplus as channel-wise concatenation and clarified that square brackets group the input cues supplied to subsequent modules.

The Task-Factor representation contains:

ComponentDimensionRole
Objectness (OO)1Object-presence evidence
Geometry (GG)4Local object geometry
Uncertainty (UU)4Geometric uncertainty
Positional latent (ZpZ_p)8Correspondence-related information
Residual latent (ZsZ_s)16Complementary task information
Total33

Both modalities use this 33-dimensional factor structure, providing a common task-level interface without requiring shared feature-extraction weights.

2. Necessity of GAMTA and clarification of Eqs. (3)–(7) (W2, Q3–Q6)

Why is GAMTA necessary if RGB and X share the same Task-Factor structure?

Sharing the same factor definitions does not guarantee spatial correspondence. RGB and X observe different physical signals, so their objectness responses, boundaries, and local geometric evidence may differ even when the original observations are approximately registered.

GAMTA establishes task-space correspondence before cross-modal discrepancies are evaluated and auxiliary evidence is incorporated into the RGB detector. We have clarified this motivation in Section 3.3 and strengthened the corresponding ablations in Section 4.4.

Q3: Global transformation and multi-scale warping in Eq. (3).

The predictor G5G^5 estimates a 2D similarity transformation comprising translation, rotation, and isotropic scaling, rather than a general 3D rigid-body transformation.

The transformation is estimated from P5 Task Factors and applied consistently to P3, P4, and P5. Each level then performs its own residual correspondence refinement.

Q4: Spatial location, candidate count, and residual offsets in Eq. (4).

Here, pp denotes a spatial location on the RGB feature grid. A lightweight convolutional matcher predicts K=4K=4 residual sampling offsets δkl(p)\delta_k^l(p) and their correspondence weights from paired RGB and globally warped auxiliary task cues.

The maximum offset ranges are 3, 2, and 1 feature-grid units at P3, P4, and P5, respectively. The residual displacement is further guided by local correspondence supervision. An additional Null-Match state allows the model to reject unsupported auxiliary correspondences.

These details are specified in the supplementary model configuration and alignment-supervision descriptions.

Q5: Definition of HnormH_{\mathrm{norm}} in Eq. (6).

HnormH_{\mathrm{norm}} is the normalized Shannon entropy of the non-Null matching distribution, divided by log⁡K\log K.

It measures correspondence ambiguity: a concentrated distribution indicates greater matching confidence, whereas a diffuse distribution indicates greater uncertainty. Alignment confidence combines this measure with the non-Null correspondence probability.

Q6: Definitions of σ\sigma and SlS^l in Eq. (7).

σ\sigma denotes the sigmoid activation function, and SlS^l is a learned, scale-specific convolutional reliability estimator.

The estimator uses alignment confidence, RGB-X objectness and geometry discrepancies, and predicted geometric uncertainties. Its output is further constrained by alignment confidence, auxiliary spatial validity, and modality presence.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

3. Why query guidance is applied only at P5 (Q7)

Eq. (9) operates at P5 because TF-DETR retains the RT-DETRv2 hybrid encoder, whose AIFI self-attention module is applied at the lowest-resolution P5 level.

Accordingly, Task-Factor guidance modulates the attention queries at P5, while the keys and values remain RGB-derived.

However, auxiliary information is not restricted to P5. After cross-scale fusion, the bounded low-rank refinement in Eqs. (10)–(11) incorporates aligned task evidence at P3, P4, and P5.

The two mechanisms are complementary: P5 query guidance influences attention-based semantic retrieval, whereas multi-scale refinement provides local, reliability-conditioned corrections.

The ablation in Table 5(a) supports this design: query guidance alone achieves 64.0 AP, feature refinement alone achieves 64.1 AP, and their combination achieves 64.6 AP on M3FD.

4. RGB-only and conventional-fusion comparisons (Q8)

Following the reviewer's suggestion, we have added single-modality and conventional-fusion baselines in the supplementary fusion analysis.

MethodAP50APParameters
RGB-only RT-DETR90.4061.0043.0M
Thermal-only RT-DETR88.3058.2043.0M
Add-Fuse92.5062.5067.0M
Concat-Fuse92.4063.4068.0M
Vanilla cross-attention (P5)92.0263.1043.48M
TF-DETR93.3064.6043.8M

TF-DETR improves over RGB-only detection by 3.6 AP, demonstrating the benefit of complementary Thermal information.

Compared with vanilla P5 cross-attention, TF-DETR achieves 1.5 higher AP with a similarly lightweight parameter count. It also exceeds Concat-Fuse by 1.2 AP while using 24.2M fewer parameters.

These comparisons support the effectiveness of task-guided auxiliary interaction rather than conventional feature fusion alone.

5. RGB-D evaluation and comparison with published results (W3, Suggestion 1)

We thank the reviewer for highlighting the importance of RGB-D evaluation.

In recent years, research involving KITTI and SUN RGB-D has increasingly focused on 3D object detection, while directly comparable RGB-D 2D detection studies remain relatively limited.

We would like to clarify that our primary purpose in including these datasets is to evaluate the cross-modal applicability of TF-DETR, rather than to establish a separate state-of-the-art benchmark for RGB-D detection.

Specifically, we investigate whether the same Task-Factor representation and interaction framework, originally evaluated with Thermal and Event observations, can also accommodate Depth as an auxiliary modality without changing the core architecture.

Nevertheless, we identified a recent published RGB-D 2D detection method, RGBX-DiffusionDet${}^{[1]}$, and have added its KITTI results to the supplementary RGB-D comparison.

MethodAPAP50
RGBX-DiffusionDet69.291.3
TF-DETR (Ours)79.695.9

TF-DETR reports 10.4 higher AP and 4.6 higher AP50 than the published RGBX-DiffusionDet results.

We have also clarified the RGB-D evaluation protocols:

  • KITTI RGB-D: eight detection categories, with 6,733 training and 748 validation images.

  • SUN RGB-D: 16 categories, with 5,285 training and 5,050 validation images.

Both datasets are evaluated using COCO 2D bounding-box metrics. On SUN RGB-D, TF-DETR achieves 54.7 AP and 69.1 AP50.

These experiments provide additional evidence that the proposed Task-Factor representation applies across heterogeneous auxiliary modalities, including Thermal, Event, and Depth, while preserving the same core detection framework.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

6. GAMTA and reliability ablations (Suggestions 2–4)

We appreciate the suggestions regarding the necessity of GAMTA and the reliability estimation mechanism.

We have added the following component-wise ablations in Section 4.4, Table 6(b):

ConfigurationAP50AP
Full TF-DETR93.3064.60
Without GAMTA92.7063.90
Without global alignment92.8063.87
Single-center matching92.6063.50
Without transport rejection92.9064.01
Without learned reliability gate93.0063.87
Discrepancy-only reliability93.0063.50

Necessity of GAMTA. Removing GAMTA reduces AP from 64.60 to 63.90. Removing global alignment yields 63.87 AP, and the single-center matching alternative achieves 63.50 AP. These results support the combined use of global anchoring and flexible local correspondence.

Reliability estimation. Removing the learned reliability estimator reduces AP to 63.87, while using only objectness and geometry discrepancies results in 63.50 AP. The full formulation therefore benefits from additional alignment and uncertainty-related reliability cues.

Robustness under unreliable auxiliary observations.

We have also added supplementary Thermal robustness evaluations on M3FD, covering spatial translations, Gaussian noise, and complete auxiliary-modality absence while keeping RGB unchanged.

Thermal perturbationTF-DETR AP dropConcat-Fuse AP drop
Shift 4 px0.996.61
Shift 8 px1.9313.50
Shift 16 px1.9121.15
Thermal missing0.2516.97
Gaussian noise (σ=0.2\sigma=0.2)0.3411.39

Under Thermal shifts of 4, 8, and 16 pixels, TF-DETR shows average AP reductions of 0.99, 1.93, and 1.91 points, respectively, compared with substantially larger reductions for Concat-Fuse.

When Thermal input is completely absent, TF-DETR loses only 0.25 AP points, compared with 16.97 points for Concat-Fuse. Under Gaussian Thermal noise with σ=0.2\sigma=0.2, the corresponding reductions are 0.34 and 11.39 AP points.

These results complement the component ablations by demonstrating reduced sensitivity to the evaluated spatially misaligned, missing, and noisy auxiliary observations.

Summary

We sincerely appreciate the reviewer's detailed questions and suggestions.

The revised manuscript clarifies the Task-Factor mappings, representation dimensions, GAMTA operations, reliability estimation, and the roles of P5 query guidance and multi-scale feature refinement.

The additional single-modality and conventional-fusion baselines, RGB-D evaluations, component-wise ablations, and robustness experiments strengthen the empirical evidence for the proposed design.

Together, these results support TF-DETR as a compact task-oriented framework that incorporates complementary auxiliary evidence while preserving the primary RGB semantic pathway.

We hope these revisions address the reviewer's concerns and clarify the methodological and experimental contributions of TF-DETR. Please let us know if you have any additional questions. We would be very glad to address them.

[1]. Orfaig E, Stainvas I, Bilik I. RGBX-DiffusionDet: a framework for multi-modal RGB-X object detection using DiffusionDet[J]. Pattern Recognition, 2025: 112460.

Reviewer_amKkOpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

TF-DETR proposes an efficient RGB-X object detection framework that treats RGB as the primary semantic stream and represents auxiliary modalities such as thermal, event, and depth data using compact Task Factors, including objectness, geometry, uncertainty, and sensor-specific latent information. It introduces multi-scale task alignment and reliability-aware guidance to incorporate auxiliary evidence into RGB features without building a second full semantic backbone. Experiments on six benchmarks show that TF-DETR achieves strong detection performance with fewer parameters and higher inference speed, demonstrating that task-oriented auxiliary representations can reduce computational cost while preserving multimodal detection accuracy.

Critical strengths

Experiments on six benchmarks show that TF-DETR achieves strong detection performance with fewer parameters and higher inference speed.

Critical weaknesses

1.The methodological novelty of the paper is open to question. Task Factors, cross-modal alignment, reliability gating, query guidance, and low-rank feature refinement all have clear connections to existing techniques in multimodal detection, feature alignment, uncertainty-aware fusion, and parameter-efficient adaptation. The main contribution appears to be the reorganization of these components into a unified framework with an RGB-primary and task-oriented auxiliary representation, rather than the introduction of a fundamentally new operation or learning paradigm. The authors should therefore provide more thorough component-level comparisons with existing modular fusion methods to demonstrate that the performance gains genuinely come from the Task-Factor representation itself.

2.The paper’s central assumption that Task Factors are a better way to represent auxiliary modalities is supported mainly by ablation studies and qualitative visualizations, with limited theoretical analysis or rigorous validation. Although Objectness and Geometry have intuitive meanings, the specific roles of the residual latent ZsZ_s and positional latent ZpZ_p remain unclear. They may still function as ordinary feature representations. Therefore, the paper does not yet fully demonstrate that Task Factors are more generally effective than other low-dimensional representations or simpler auxiliary-feature fusion methods.

3.The comparability of the experimental results and the practical applicability of the method have certain limitations. Some datasets are converted by the authors into a COCO-style evaluation protocol, and the KITTI and SUN RGB-D results do not correspond directly to their official evaluation settings. In addition, the reported speed mainly measures model forward inference and excludes data loading, preprocessing, cross-modal synchronization, and post-processing. The method also involves multiple alignment, gating, supervision, and consistency losses, which may make training and hyperparameter tuning relatively complex. Furthermore, it relies on reasonably accurate spatial or temporal correspondence between RGB and auxiliary modalities. Its robustness under severe sensor misalignment, large-scale missing data, or degraded RGB inputs remains insufficiently validated.

Questions for authors

See Weaknesses.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

We thank the reviewer for the detailed and constructive comments. We appreciate the concerns regarding methodological novelty, the functional validity of Task Factors, evaluation comparability, inference efficiency, and robustness.

We have strengthened the manuscript with controlled representation and fusion comparisons, component-wise ablations, quantitative analyses of the learned latents, clarified RGB-D and efficiency evaluation protocols, and supplementary robustness experiments (as detailed in the revised manuscript and supplementary material).

We address all the weaknesses (W) and questions raised by the reviewer below. For clarity, we divide our response to W3 into three parts and address the remaining weaknesses and all questions individually.

1. Methodological novelty and controlled component comparisons (W1)

We agree that cross-modal alignment, reliability gating, query guidance, and low-rank refinement have connections to existing techniques. The main contribution of TF-DETR is not the isolated introduction of these operations, but the task-grounded organization of auxiliary evidence and its integration into an RGB-primary detection framework.

Conventional feature-centric fusion generally combines modality-specific feature representations. In contrast, TF-DETR uses a lightweight auxiliary encoder to produce compact Task Factors comprising explicitly grounded objectness, geometry, and uncertainty states, together with learned positional and residual task latents.

These factors form a common task-level interface for cross-modal correspondence, reliability estimation, and constrained interaction with the RGB encoder. The auxiliary pathway does not need to maintain a second full semantic feature stream.

To examine whether the improvements arise from task-oriented representation rather than auxiliary feature dimensionality or conventional fusion alone, we have strengthened the comparisons in Tables 4–6 and the supplementary fusion-baseline analysis.

The results on M3FD include:

MethodAPParameters
Generic auxiliary features (32D)64.0043.3M
Add-Fuse62.5067.0M
Concat-Fuse63.4068.0M
Vanilla cross-attention (P5)63.1043.48M
TF-DETR (33D Task Factors)64.6043.8M

Under the same lightweight RepATE encoder and nearly matched representation dimensionality, TF-DETR achieves 0.6 higher AP than the generic-feature alternative.

Furthermore, Table 6(a) includes a task-supervised generic-feature variant, which also achieves 64.00 AP. This comparison supports the contribution of task-oriented representation beyond task supervision alone.

Compared with conventional Add-Fuse and Concat-Fuse, TF-DETR achieves stronger accuracy with substantially fewer parameters. In particular, it outperforms Concat-Fuse by 1.2 AP while using 24.2M fewer parameters.

The vanilla P5 cross-attention baseline achieves 63.10 AP, compared with 64.60 AP for TF-DETR, indicating that a similarly lightweight conventional interaction does not reproduce the full model's performance.

We also examine the two task-guided interaction paths in Table 5(a). Query guidance alone achieves 64.0 AP, while bounded feature refinement alone achieves 64.1 AP. Their combination reaches 64.6 AP, supporting their complementary contributions.

Together, these comparisons provide empirical evidence for the proposed design choice: organizing auxiliary information into detection-oriented Task Factors and incorporating it through structured, reliability-aware interaction offers an effective accuracy–efficiency trade-off.

2. Quantitative validation of Task Factors and latent roles (W2)

We agree that qualitative visualizations alone are insufficient to establish the functional roles of the learned latent factors.

We have therefore added quantitative analyses of the positional latent ZpZ_p and residual task latent ZsZ_s in the supplementary latent-factor evaluation.

Positional latent ZpZ_p.

Since GAMTA estimates global alignment at P5, we evaluate the correspondence information carried by ZpZ_p using frozen top-1 matching probes at this level.

The average matching recall across three evaluated spatial shifts is:

RepresentationTop-1 matching recall
Explicit objectness, geometry, uncertainty (O/G/U)26.00%
O/G/U + ZpZ_p31.00%
O/G/U + shuffled ZpZ_p21.02%

Adding the correctly paired ZpZ_p improves matching recall by 4.99 percentage points over the explicit task states and by 9.98 percentage points over the equal-dimensional shuffled-latent control.

These results provide quantitative evidence that ZpZ_p contains complementary correspondence-related information rather than merely increasing feature dimensionality.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

Residual task latent ZsZ_s.

We further investigate whether ZsZ_s contains task-decodable evidence beyond the explicitly grounded factors, positional factors, and alignment/reliability cues.

The evaluation adds the correctly paired cross-modal residual to a baseline probe and compares it with an equal-dimensional shuffled auxiliary-latent control.

Across P3–P5 (feature levels with strides of 8, 16, and 32), the correctly paired residual consistently reduces classification, objectness, and geometry probe losses.

At P5, for example, objectness loss decreases by 24.14% relative to the baseline and by 21.02% relative to the shuffled auxiliary-latent control.

These results support the interpretation of ZsZ_s as complementary task-related capacity that preserves information not exhausted by the explicitly grounded states.

The detection ablations further support the utility of both latents. Removing ZpZ_p decreases AP from 64.6 to 63.8, while removing ZsZ_s decreases AP to 63.7.

Together, the frozen-probe analyses, shuffled controls, and detection ablations provide functional evidence for the different roles of the two latent factors, beyond qualitative visualization alone.

3. RGB-D evaluation protocols and experimental comparability (W3)

We have clarified the RGB-D dataset protocols and the intended purpose of these experiments.

The primary objective of the RGB-D evaluations is to examine whether the same Task-Factor representation and interaction framework can accommodate Depth as an auxiliary modality, extending its applicability beyond Thermal and Event observations.

In recent years, research involving KITTI and SUN RGB-D has increasingly focused on 3D object detection, while directly comparable RGB-D 2D detection studies remain relatively limited. Accordingly, our RGB-D experiments are designed primarily as an assessment of cross-modal applicability rather than a separate state-of-the-art benchmark for RGB-D detection.

We have explicitly documented the adopted COCO 2D bounding-box evaluation protocol and dataset configurations:

  • KITTI RGB-D: eight object categories, with 6,733 training and 748 validation images.

  • SUN RGB-D: 16 object categories, with 5,285 training and 5,050 validation images.

We have also identified a recent published RGB-D 2D detection method, RGBX-DiffusionDet${}^{[1]}$, and added its reported KITTI results as a published reference in the supplementary RGB-D comparison.

MethodAPAP50
RGBX-DiffusionDet69.291.3
TF-DETR (Ours)79.695.9

TF-DETR reports 10.4 higher AP and 4.6 higher AP50 than this published reference.

On SUN RGB-D, TF-DETR additionally achieves 54.7 AP and 69.1 AP50.

These evaluations provide further evidence that the same core Task-Factor formulation can be applied across heterogeneous auxiliary sensing modalities, including Thermal, Event, and Depth.

The corresponding dataset descriptions, preprocessing details, and published KITTI comparison are provided in the supplementary evaluation protocols.

4. Inference efficiency and training complexity (W3)

Inference efficiency and measurement protocol.

We thank the reviewer for the comment on inference efficiency. As described in Section 4.3 and Appendix F, we adopt a unified benchmarking protocol for all compared methods to ensure consistent and meaningful efficiency comparisons.

Specifically, all models are evaluated on the same NVIDIA RTX 4090 GPU, with an input resolution of 640 × 640 and a batch size of 1.

The PyTorch measurements use the same execution environment and timing procedure, including GPU synchronization, warm-up iterations, and repeated inference measurements.

Under this unified setting, TF-DETR achieves 39.4 FPS on DSEC-Detection and 36.4 FPS on M3FD, demonstrating competitive computational efficiency with only 43.8M parameters.

The reported FPS measures model-forward inference throughput, excluding data loading, preprocessing, data transfer, and post-processing.

Thus, Table 3 provides a controlled comparison of the computational efficiency of the detection models, rather than the end-to-end throughput of a complete multimodal sensing and processing pipeline.

We additionally report 240.3 FPS with TensorRT 10.4 FP16 as a separate deployment-oriented engine-execution measurement, following the protocol detailed in Appendix F.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

Training complexity and supervision.

We have documented the implementation configurations, training objectives, and loss weights in the supplementary training details.

Importantly, the same core loss-weight configuration is used across all six benchmarks, rather than introducing a separate set of task-loss weights for every auxiliary modality.

To assess the contributions of the additional supervision objectives, Table 5(b) reports the following ablations on M3FD:

ConfigurationAP
Full supervision64.6
Without Task-Factor Loss63.6
Without Alignment Supervision63.8
Without Intervention Consistency64.2

Removing Task-Factor grounding decreases AP by 1.0 point, removing alignment supervision decreases AP by 0.8 points, and removing intervention consistency decreases AP by 0.4 points.

These results demonstrate measurable contributions from the additional objectives.

Furthermore, the supervision terms are used during training and do not require a second full auxiliary semantic backbone during inference, preserving the lightweight auxiliary pathway.

5. Robustness to misalignment and unreliable auxiliary observations (W3)

To address the concerns regarding cross-modal correspondence and auxiliary sensor reliability, we have added supplementary robustness experiments on M3FD.

We evaluate Thermal translations, Gaussian noise, and complete auxiliary-modality absence while keeping RGB unchanged. The results report AP reductions relative to each model's clean reference.

Thermal perturbationTF-DETR AP drop ↓Concat-Fuse AP drop ↓
Shift 4 px0.996.61
Shift 8 px1.9313.50
Shift 16 px1.9121.15
Thermal missing0.2516.97
Gaussian noise (σ=0.2\sigma=0.2)0.3411.39

Robustness analysis.

The robustness results reflect three complementary mechanisms in TF-DETR. For spatial misalignment, Concat-Fuse's AP drop increases from 6.61 to 21.15 points as the Thermal shift grows from 4 to 16 pixels, whereas TF-DETR maintains much smaller drops of 0.99, 1.93, and 1.91 points at 4, 8, and 16 pixels, respectively. This is consistent with GAMTA, which combines global alignment, multi-scale residual matching, and unsupported-correspondence rejection before auxiliary evidence interacts with RGB features. The same advantage holds across all 12 tested translation configurations.

For missing Thermal input, TF-DETR loses only 0.25 AP, compared with 16.97 AP for Concat-Fuse. The explicit modality-presence mask suppresses unavailable auxiliary evidence, while the RGB-primary pathway preserves the main detection representation. Under Gaussian Thermal noise (σ=0.2\sigma=0.2), TF-DETR drops only 0.34 AP, versus 11.39 AP for Concat-Fuse. Here, reliability estimation uses task discrepancies, alignment confidence, and geometric uncertainty to regulate auxiliary contributions, while bounded low-rank refinement limits their influence on RGB features.

These interpretations are further supported by Table 6: removing GAMTA, the learned reliability estimator, or the refinement norm bound reduces clean AP from 64.60 to 63.90, 63.87, and 63.60, respectively. Together, the results support the complementary roles of task-space alignment, reliability-aware interaction, and constrained RGB feature refinement.

The complete robustness results, including direction-wise translations, are provided in the supplementary fusion-baseline and Thermal robustness analysis.

Summary

We appreciate the reviewer's concerns regarding methodological novelty, Task-Factor validity, RGB-D comparability, inference efficiency, training complexity, and robustness.

The additional controlled comparisons support the contribution of task-grounded auxiliary representation; the quantitative probes provide functional evidence for ZpZ_p and ZsZ_s; the clarified experimental protocols establish the scope and consistency of the efficiency measurements; and the new robustness experiments demonstrate reduced sensitivity to misaligned, missing, and noisy auxiliary observations.

Together, these results strengthen the empirical foundation of TF-DETR as an efficient, task-oriented framework for RGB-X object detection.

We hope these revisions address the reviewer's concerns and clarify the methodological and experimental contributions of TF-DETR. Please let us know if you have any remaining questions. We would be happy to address them.

[1]. Orfaig E, Stainvas I, Bilik I. RGBX-DiffusionDet: a framework for multi-modal RGB-X object detection using DiffusionDet[J]. Pattern Recognition, 2025: 112460.

Reviewer_ssCtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TF‑DETR, an asymmetric RGB‑X object detection framework. Instead of building a full parallel semantic backbone for auxiliary modality X (thermal / event / depth), the authors design compact Task‑Factor representation to encode auxiliary inputs, paired with Global‑Anchored Multi‑Scale Task Alignment (GAMTA) and reliability‑aware RGB encoder interaction. The model is evaluated on six RGB‑X detection benchmarks.

  1. The paper observes that different auxiliary sensors contribute unequally across detection sub‑tasks (objectness, classification, geometry regression). The authors argue we do not need full semantic streams for every auxiliary modality; compact task‑oriented representation can serve as a cross‑sensor interface. This perspective is thought‑provoking for multimodal detection research.
  2. The same TF‑DETR architecture works for thermal, event‑based, and depth inputs, only changing input preprocessing. It demonstrates good cross‑modality generalization potential.
Critical strengths
  1. Well-Motivated Problem Formulation: The observation that dual-stream semantic representations in RGB-X detection often suffer from redundancy is insightful. Proposing a lightweight auxiliary representation is a meaningful direction for improving efficiency.

  2. Comprehensive Experimental Coverage: The authors evaluate the method on six diverse benchmarks covering three types of auxiliary modalities (Event, Thermal, Depth), demonstrating the generalizability of the proposed approach.

  3. Excellent Efficiency Profile: The model achieves a strong balance between accuracy and speed. With only 43.8M parameters and reaching up to 240.3 FPS in TensorRT deployment, it holds significant practical value for real-world applications.

  4. Intuitive Design Choices: The reliability-aware gating mechanism is a sensible design to prevent erroneous auxiliary information from degrading the primary RGB stream.

Critical weaknesses
  1. The core high‑level motivation — “avoid dual heavy backbones and use lightweight auxiliary branch guided by task requirements” — is not sufficiently novel. Prior multimodal detection and segmentation works (CMX, CMNeXt, Scene‑Specific Fusion, OneTracker) already adopt a primary‑auxiliary asymmetric design: keep the main RGB backbone intact and use lightweight modules for auxiliary modalities. The paper fails to clearly distinguish its contributions against these existing asymmetric multimodal paradigms.

  2. The conditional task utility analysis in the introduction only provides qualitative observations. There is no rigorous quantitative metric to measure task‑dependent modality contribution. The claimed “objectness‑dominant for event, classification‑dominant for thermal” remains observational without formal measurement.

  3. Task‑Factor factorization decomposes representation into explicit task states plus latent residuals. Similar decomposition (shared / private, disentangled task‑aware factors) has been widely explored in existing multimodal literature. The incremental novelty of Task‑Factor itself is limited.

  4. GAMTA (Global‑Anchored Multi‑Scale Task Alignment) complexity & limited ablation. GAMTA contains global similarity transform, multi‑scale residual offsets, null‑match rejection, alignment confidence, and reliability gating. The paper does not perform sufficient ablation to quantify the individual contribution of each sub‑component. It is unclear whether heavy alignment machinery is necessary or whether simpler cross‑modal alignment can achieve similar performance.

  5. Reliability gating is central to the whole pipeline, yet there is no standalone ablation to demonstrate how much performance originates from reliability‑aware gating. It is hard to tell whether reliability estimation brings tangible benefit or is redundant.

  6. Task‑Factor latent space interpretation is mostly supported by qualitative visualizations. Quantitative analysis measuring how well ZpZ_p encodes spatial correspondence and ZsZ_s encodes residual task evidence is absent. Visualization alone is insufficient to validate the functional decomposition claim.

  7. For the RGB‑Depth setting, the authors convert KITTI / SUN‑RGB‑D into a self‑defined COCO‑style 16‑category protocol, deviating from official benchmark protocols. Direct comparison against published SOTA RGB‑Depth detection results is not feasible, weakening the persuasiveness of the RGB‑Depth experiments.

Questions for authors
  1. You observe that event modalities show objectness‑dominant utility and thermal modalities show classification‑dominant utility, but the analysis is purely observational. Do you have a concrete quantitative metric to measure per‑subtask modality contribution that supports this claim? If yes, can you provide the computed numerical results from this metric?

  2. Why are these representative asymmetric baselines absent from your main comparisons? Have you run controlled ablations that swap the backbone/decoder to match your ResNet‑50vd + RT‑DETRv2 setup for those methods? If you have such results, please report them.

  3. Have you performed component‑wise ablations that isolate each individual sub‑module inside GAMTA? Specifically: removing global transform; removing multi‑scale residual offsets; disabling null‑match rejection; turning off reliability gating entirely, while keeping all other parts of TF‑DETR unchanged. Can you provide corresponding mAP / AP50 numbers?

  4. You re‑annotate datasets into your custom 16‑category COCO‑style protocol different from official benchmarks. Is there any published RGB‑Depth detector result that uses exactly your custom annotation split and label space for direct comparison? If yes, cite and include those numbers; if no, explain how we can verify the RGB‑Depth performance is meaningful.

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors (1/3)

We thank the reviewer for the constructive feedback and for recognizing the motivation, cross-sensor applicability, and computational efficiency of TF-DETR. We have strengthened the manuscript with quantitative task-utility analysis, controlled representation and fusion comparisons, component-wise ablations, quantitative latent-factor validation, and additional RGB-D results (as detailed in the revised manuscript and supplementary material).

We address all the weaknesses (W) and questions (Q) raised by the reviewer below, and hope that our revisions adequately address the reviewer's concerns and clarify the contributions of the proposed method.

1. Novelty and comparison with existing multimodal methods (W1, W3, Q2)

We agree that asymmetric multimodal processing and lightweight fusion have been explored in prior work. The main contribution of TF-DETR is not asymmetry itself, but how auxiliary information is represented according to detection requirements and subsequently used for correspondence and reliability-aware interaction.

The four methods highlighted by the reviewer differ from TF-DETR in their tasks or architectural designs:

  • CMX and CMNeXt primarily address RGB-X semantic segmentation through cross-modal feature rectification, fusion, and lightweight auxiliary information extraction.

  • OneTracker focuses on multimodal visual object tracking by adapting an RGB foundation tracker through lightweight prompting.

  • Scene-Specific Fusion is more closely related to our object detection setting. However, it retains two pretrained modality-specific EfficientDet feature extraction pathways, including separate backbones and feature pyramids, and combines them through lightweight scene-adaptive fusion modules.

In contrast, TF-DETR does not construct a second full semantic feature pathway for the auxiliary modality. Instead, a lightweight auxiliary encoder directly produces compact Task Factors comprising objectness, geometry, uncertainty, and learned positional and residual latents. These factors provide a task-level interface for cross-modal alignment, reliability estimation, and constrained interaction with the primary RGB detector.

We have strengthened the discussion of these distinctions in Related Work.

Regarding the requested comparisons, we have not directly reproduced the four named methods or conducted backbone/decoder-matched reimplementations. Instead, we provide additional controlled comparisons of auxiliary representations and fusion strategies within our detection framework.

The results on M3FD are:

MethodAPParameters
Generic auxiliary features (32D)64.043.3M
Add-Fuse62.567.0M
Concat-Fuse63.468.0M
Vanilla cross-attention (P5)63.143.48M
TF-DETR (33D Task Factors)64.643.8M

Under the same lightweight RepATE encoder, Task Factors outperform the nearly dimension-matched generic representation by 0.6 AP. Table 6 additionally reports a task-supervised generic-feature alternative achieving 64.0 AP, supporting the contribution of task-oriented representation beyond task supervision alone.

Compared with conventional feature fusion, TF-DETR improves over Concat-Fuse by 1.2 AP while using 24.2M fewer parameters. The lightweight P5 cross-attention baseline also obtains 63.1 AP, compared with 64.6 AP for TF-DETR.

These results support the effectiveness of organizing auxiliary evidence around detection-oriented factors rather than relying exclusively on generic auxiliary features or conventional fusion.

The corresponding comparisons are provided in Tables 4 and 6 and the supplementary fusion-baseline analysis.

AuthorsOpenReview ↗

Rebuttal by Authors (2/3)

2. Quantitative conditional task utility (W2, Q1)

We have added a quantitative conditional task utility analysis to support the motivation in Fig. 1(b).

RGB-only and auxiliary-modality encoders are trained independently and then frozen. Lightweight task probes compare predictions using RGB features alone with predictions using RGB and paired auxiliary features.

For each detection subtask, we calculate its relative validation-loss reduction, denoted by Δt\Delta_t, and its normalized utility share, denoted by UtU_t.

The results are:

DatasetSubtaskRelative loss reductionNormalized utility share
M3FD (RGB-Thermal)Classification30.31%43.72%
M3FD (RGB-Thermal)Objectness19.57%28.23%
M3FD (RGB-Thermal)Geometry19.45%28.05%
PKU-DAVIS-SOD (RGB-Event)Classification1.39%21.87%
PKU-DAVIS-SOD (RGB-Event)Objectness3.65%57.63%
PKU-DAVIS-SOD (RGB-Event)Geometry1.30%20.50%

On M3FD, classification has the largest measured utility share (43.72%), whereas objectness and geometry contribute 28.23% and 28.05%, respectively.

On PKU-DAVIS-SOD, objectness has the largest measured utility share (57.63%), compared with 21.87% for classification and 20.50% for geometry.

These measurements quantitatively support the different task-utility profiles observed in the two evaluated sensing settings.

The normalized shares characterize relative improvements in task-specific probe losses within each dataset, rather than absolute detection AP gains or fixed properties of the sensing modalities.

We have added the metric definitions, measurement procedure, and detailed results to Fig. 1(b) and the supplementary conditional task utility analysis.

3. GAMTA components and reliability modeling (W4, W5, Q3)

We have expanded the component-wise ablations in Section 4.4, Table 6(b) to examine global alignment, local correspondence, Null-Match rejection, and reliability estimation.

ConfigurationAP50AP
Full TF-DETR93.3064.60
Without GAMTA92.7063.90
Without global alignment92.8063.87
Single-center matching92.6063.50
Without transport rejection92.9064.01
Without learned reliability gate93.0063.87
Discrepancy-only reliability93.0063.50

Global and local correspondence. Removing global alignment reduces AP from 64.60 to 63.87, while removing GAMTA yields 63.90 AP. These results indicate the importance of establishing a shared global correspondence anchor.

The single-center matching alternative further decreases AP to 63.50, supporting the benefit of maintaining multiple local correspondence hypotheses instead of enforcing one potentially inaccurate match.

Null-Match rejection. Disabling transport rejection reduces AP to 64.01, indicating that rejecting unsupported auxiliary correspondences contributes to detection accuracy.

Reliability estimation. Removing the learned reliability estimator reduces AP to 63.87, while using only objectness and geometry discrepancies yields 63.50 AP.

These results indicate that alignment confidence, auxiliary validity, and uncertainty-related evidence provide useful additional information for regulating auxiliary interaction.

Overall, the expanded ablations demonstrate measurable contributions from GAMTA and reliability modeling rather than merely introducing additional architectural complexity.

AuthorsOpenReview ↗

Rebuttal by Authors (3/3)

4. Quantitative validation of Task-Factor latents (W6)

We have added quantitative analyses of the positional latent ZpZ_p and residual task latent ZsZ_s, complementing the qualitative visualizations and detection ablations.

Positional latent ZpZ_p.

Since GAMTA estimates global alignment at P5, we evaluate correspondence information using frozen matching probes at this level.

Across three spatial-shift settings, incorporating ZpZ_p improves top-1 matching recall by an average of 4.99 percentage points over the explicit objectness, geometry, and uncertainty factors.

Compared with an equal-dimensional shuffled-ZpZ_p control, the average improvement reaches 9.98 percentage points.

These results support the role of ZpZ_p as a complementary source of correspondence-related positional information.

Residual latent ZsZ_s.

We further evaluate whether the aligned cross-modal ZsZ_s residual preserves useful task information beyond explicit task states, positional factors, and alignment/reliability cues.

Adding the correctly paired residual consistently reduces classification, objectness, and geometry probe losses across P3–P5 (feature levels with strides of 8, 16, and 32).

For example, at P5, the objectness loss decreases by 24.14% relative to the baseline and by 21.02% relative to an equal-dimensional shuffled auxiliary-latent control.

These results indicate that the explicit Task Factors do not exhaust the information useful for downstream prediction. The residual latent provides complementary task-decodable evidence beyond the predefined states.

The detection ablations provide further support: removing ZpZ_p reduces AP from 64.6 to 63.8, while removing ZsZ_s reduces it to 63.7.

We have included the corresponding measurements in the supplementary quantitative analysis of Task-Factor latents.

5. RGB-D evaluation protocol and published comparison (W7, Q4)

We have clarified the RGB-D evaluation protocols and corrected the category-count error in the original manuscript.

For KITTI RGB-D, our evaluation uses eight detection categories, with 6,733 training and 748 validation images.

For SUN RGB-D, we use 16 categories, with 5,285 training and 5,050 validation images.

Both datasets are evaluated using COCO 2D bounding-box metrics.

We would also like to clarify the purpose of these experiments. Recent research involving KITTI and SUN RGB-D has increasingly focused on 3D detection, while directly comparable RGB-D 2D detection studies remain relatively limited.

Our primary goal in including RGB-D is to evaluate the cross-modal applicability of TF-DETR, examining whether the same Task-Factor representation and interaction framework can accommodate Depth in addition to Thermal and Event inputs.

Nevertheless, we identified a recent published RGB-D 2D detection method, RGBX-DiffusionDet${}^{[1]}$, and added its KITTI results to Table C.1.

MethodAPAP50
RGBX-DiffusionDet69.291.3
TF-DETR (Ours)79.695.9

TF-DETR reports 10.4 higher AP and 4.6 higher AP50 than the published RGBX-DiffusionDet results.

On SUN RGB-D, TF-DETR additionally achieves 54.7 AP and 69.1 AP50.

These results provide a published RGB-D reference and further support the applicability of the proposed Task-Factor representation across heterogeneous auxiliary sensing modalities.

The revised dataset descriptions and RGB-D comparison are included in the supplementary datasets and evaluation protocols section.

Summary

We appreciate the reviewer's concerns regarding novelty, quantitative motivation, component necessity, latent-factor interpretation, and evaluation comparability.

The added representation and fusion comparisons strengthen the evidence for task-grounded auxiliary modeling. The conditional task utility analysis provides quantitative support for our motivation, while the component ablations and latent-factor probes clarify the contributions of alignment, reliability, and the learned factors.

The additional RGB-D comparison further supports the applicability of the same framework across Thermal, Event, and Depth sensing.

Together, these results strengthen the empirical foundation of TF-DETR as an efficient, task-oriented RGB-X object detection framework.

We hope these revisions adequately address the reviewer's concerns and further clarify the contributions of TF-DETR. Please feel free to raise any additional questions. We would be glad to address them.

[1]. Orfaig E, Stainvas I, Bilik I. RGBX-DiffusionDet: a framework for multi-modal RGB-X object detection using DiffusionDet[J]. Pattern Recognition, 2025: 112460.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.