- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposed a framework for fusion of auxiliary information with RGB data based on their task-specific utility. The idea of using the auxiliary modality only if its adding value and ensuring that the powerful semantic cues obtained from RGB data are not corrupted by the auxiliary modality is interesting. This certainly has value in robotics and autonomous vehicles. Although the overall idea is novel there are several gaps in the theoretical formulation and in experimentation. Several architectural choices, variable definitions, operators used in equations and metric formulations are either left implicit or undefined in some cases. The experimental results with event camera and thermal camera as auxiliary modality show good results but the discussion on RGB-D (one of the most common case) is skipped. The ablation studies to study the effect of "Task-factor representation", "Task Reliability (g)" and "Low-Rank Refinement" could have been added to strengthen the claims.
Critical strengths
The main strength of the paper is the task-factor representation that allows to estimate the task reliability of the auxiliary modality. Combining the auxiliary features based on their task reliability is a good idea and seems to be working for noisy auxiliary modalities like (event cameras and thermal cameras).
Critical weaknesses
The weaknesses of the paper are:
- The explanation of sections 3.3 and 3.4 can be improved by defining all the variables used in the equations.
- The GAMTA module is not clear, why would you even need it if the task-factor representation for both RGB and the auxiliary module (X) is the same ?
- The paper fails to demonstrate the utility of the proposed method for RGB-D data. I checked the Appendix where the KITTI and SUN-D datasets are mentioned but I could not find a table of comparison of mAP with state of the art.
Questions for authors
Questions:
- In equation (1) the variables \Phi_X, E^l_X, \Phi_R, \Psi_R are not defined. Are these mapping functions multi-layer perceptron (MLP)? Do they share weights ?
- In equation (2) the vector concatenation is ambiguous, you use both square brackets and \oplus to represent concatenation. The dimensionality of Z_{p,m} and Z_{s,m} is not mentioned, I see that the total dimension of the task factor representation is 33 (from Appendix).
- Equation (3) is not clear, Is G^5 a rigid body transform ? What do you mean by multi-scale warping?
- Equation (4) is not clear. What is "p" in this equation? What is the value of parameter "K" ? How do you find the residual offset \delta_k ?
- What is H_{norm} in equation (6)?
- What is \sigma and S^l in equation (7) ?
- Why is equation (9) specifically reported for l=5 ? Will you not be doing for all the levels (l \in {3, 4, 5}) ?
- In the experiment tables can you also report the DETR output with the RGB path alone and vanilla fusion of auxiliary modality using standard cross attention? It will be good to see how much accuracy boost you are getting by fusing based on task utility.
Suggestions
- I would love to see more discussion on RGB-D datasets.
- You can look at the value of "task reliability (g)" and may be plot it for your datasets. It will be good to see how does it change for a given dataset.
- I am curious to see if you can get rid of GAMTA module completely, may be an ablation study with/without GAMTA might help.
- I am also curious to see if you can simplify the reliability estimation and compute it just from the objectness and geometry discrepancies.
5 comments
Rebuttal by Authors (2/3)
3. Why query guidance is applied only at P5 (Q7)
Eq. (9) operates at P5 because TF-DETR retains the RT-DETRv2 hybrid encoder, whose AIFI self-attention module is applied at the lowest-resolution P5 level.
Accordingly, Task-Factor guidance modulates the attention queries at P5, while the keys and values remain RGB-derived.
However, auxiliary information is not restricted to P5. After cross-scale fusion, the bounded low-rank refinement in Eqs. (10)–(11) incorporates aligned task evidence at P3, P4, and P5.
The two mechanisms are complementary: P5 query guidance influences attention-based semantic retrieval, whereas multi-scale refinement provides local, reliability-conditioned corrections.
The ablation in Table 5(a) supports this design: query guidance alone achieves 64.0 AP, feature refinement alone achieves 64.1 AP, and their combination achieves 64.6 AP on M3FD.
4. RGB-only and conventional-fusion comparisons (Q8)
Following the reviewer's suggestion, we have added single-modality and conventional-fusion baselines in the supplementary fusion analysis.
| Method | AP50 | AP | Parameters |
|---|---|---|---|
| RGB-only RT-DETR | 90.40 | 61.00 | 43.0M |
| Thermal-only RT-DETR | 88.30 | 58.20 | 43.0M |
| Add-Fuse | 92.50 | 62.50 | 67.0M |
| Concat-Fuse | 92.40 | 63.40 | 68.0M |
| Vanilla cross-attention (P5) | 92.02 | 63.10 | 43.48M |
| TF-DETR | 93.30 | 64.60 | 43.8M |
TF-DETR improves over RGB-only detection by 3.6 AP, demonstrating the benefit of complementary Thermal information.
Compared with vanilla P5 cross-attention, TF-DETR achieves 1.5 higher AP with a similarly lightweight parameter count. It also exceeds Concat-Fuse by 1.2 AP while using 24.2M fewer parameters.
These comparisons support the effectiveness of task-guided auxiliary interaction rather than conventional feature fusion alone.
5. RGB-D evaluation and comparison with published results (W3, Suggestion 1)
We thank the reviewer for highlighting the importance of RGB-D evaluation.
In recent years, research involving KITTI and SUN RGB-D has increasingly focused on 3D object detection, while directly comparable RGB-D 2D detection studies remain relatively limited.
We would like to clarify that our primary purpose in including these datasets is to evaluate the cross-modal applicability of TF-DETR, rather than to establish a separate state-of-the-art benchmark for RGB-D detection.
Specifically, we investigate whether the same Task-Factor representation and interaction framework, originally evaluated with Thermal and Event observations, can also accommodate Depth as an auxiliary modality without changing the core architecture.
Nevertheless, we identified a recent published RGB-D 2D detection method, RGBX-DiffusionDet${}^{[1]}$, and have added its KITTI results to the supplementary RGB-D comparison.
| Method | AP | AP50 |
|---|---|---|
| RGBX-DiffusionDet | 69.2 | 91.3 |
| TF-DETR (Ours) | 79.6 | 95.9 |
TF-DETR reports 10.4 higher AP and 4.6 higher AP50 than the published RGBX-DiffusionDet results.
We have also clarified the RGB-D evaluation protocols:
-
KITTI RGB-D: eight detection categories, with 6,733 training and 748 validation images.
-
SUN RGB-D: 16 categories, with 5,285 training and 5,050 validation images.
Both datasets are evaluated using COCO 2D bounding-box metrics. On SUN RGB-D, TF-DETR achieves 54.7 AP and 69.1 AP50.
These experiments provide additional evidence that the proposed Task-Factor representation applies across heterogeneous auxiliary modalities, including Thermal, Event, and Depth, while preserving the same core detection framework.
Rebuttal by Authors (3/3)
6. GAMTA and reliability ablations (Suggestions 2–4)
We appreciate the suggestions regarding the necessity of GAMTA and the reliability estimation mechanism.
We have added the following component-wise ablations in Section 4.4, Table 6(b):
| Configuration | AP50 | AP |
|---|---|---|
| Full TF-DETR | 93.30 | 64.60 |
| Without GAMTA | 92.70 | 63.90 |
| Without global alignment | 92.80 | 63.87 |
| Single-center matching | 92.60 | 63.50 |
| Without transport rejection | 92.90 | 64.01 |
| Without learned reliability gate | 93.00 | 63.87 |
| Discrepancy-only reliability | 93.00 | 63.50 |
Necessity of GAMTA. Removing GAMTA reduces AP from 64.60 to 63.90. Removing global alignment yields 63.87 AP, and the single-center matching alternative achieves 63.50 AP. These results support the combined use of global anchoring and flexible local correspondence.
Reliability estimation. Removing the learned reliability estimator reduces AP to 63.87, while using only objectness and geometry discrepancies results in 63.50 AP. The full formulation therefore benefits from additional alignment and uncertainty-related reliability cues.
Robustness under unreliable auxiliary observations.
We have also added supplementary Thermal robustness evaluations on M3FD, covering spatial translations, Gaussian noise, and complete auxiliary-modality absence while keeping RGB unchanged.
| Thermal perturbation | TF-DETR AP drop | Concat-Fuse AP drop |
|---|---|---|
| Shift 4 px | 0.99 | 6.61 |
| Shift 8 px | 1.93 | 13.50 |
| Shift 16 px | 1.91 | 21.15 |
| Thermal missing | 0.25 | 16.97 |
| Gaussian noise () | 0.34 | 11.39 |
Under Thermal shifts of 4, 8, and 16 pixels, TF-DETR shows average AP reductions of 0.99, 1.93, and 1.91 points, respectively, compared with substantially larger reductions for Concat-Fuse.
When Thermal input is completely absent, TF-DETR loses only 0.25 AP points, compared with 16.97 points for Concat-Fuse. Under Gaussian Thermal noise with , the corresponding reductions are 0.34 and 11.39 AP points.
These results complement the component ablations by demonstrating reduced sensitivity to the evaluated spatially misaligned, missing, and noisy auxiliary observations.
Summary
We sincerely appreciate the reviewer's detailed questions and suggestions.
The revised manuscript clarifies the Task-Factor mappings, representation dimensions, GAMTA operations, reliability estimation, and the roles of P5 query guidance and multi-scale feature refinement.
The additional single-modality and conventional-fusion baselines, RGB-D evaluations, component-wise ablations, and robustness experiments strengthen the empirical evidence for the proposed design.
Together, these results support TF-DETR as a compact task-oriented framework that incorporates complementary auxiliary evidence while preserving the primary RGB semantic pathway.
We hope these revisions address the reviewer's concerns and clarify the methodological and experimental contributions of TF-DETR. Please let us know if you have any additional questions. We would be very glad to address them.
[1]. Orfaig E, Stainvas I, Bilik I. RGBX-DiffusionDet: a framework for multi-modal RGB-X object detection using DiffusionDet[J]. Pattern Recognition, 2025: 112460.
Rebuttal by Authors (1/3)
We sincerely thank the reviewer for the constructive comments and for recognizing the motivation and practical value of task-oriented auxiliary representation and reliability-aware fusion. We have clarified the mathematical definitions and architectural choices in Sections 3.2–3.4 (as shown in the revised paper), and strengthened the experiments with RGB-D comparisons, conventional-fusion baselines, component ablations, and robustness analyses.
We address all the weaknesses (W) and questions (Q) raised by the reviewer below.
1. Mapping functions and Task-Factor dimensions (Q1, Q2)
Q1: Definitions of the mapping functions in Eq. (1).
We have clarified the following definitions in Section 3.2:
EXl denotes the level-specific mapping of the lightweight RepATE auxiliary encoder.
ψR denotes the RGB task probe that projects the RGB semantic features into the Task-Factor space.
ΦX and ΦR denote the auxiliary and RGB Task-Factor projection heads, respectively.
These mappings are convolution-based rather than MLP-based, and their parameters are not shared across modalities or feature scales. The RGB task probe only provides a reference for task-space correspondence and reliability estimation; the original RGB semantic features remain intact in the primary detection pathway.
Q2: Concatenation and factor dimensions in Eq. (2).
We have explicitly defined ⊕ as channel-wise concatenation and clarified that square brackets group the input cues supplied to subsequent modules.
The Task-Factor representation contains:
Both modalities use this 33-dimensional factor structure, providing a common task-level interface without requiring shared feature-extraction weights.
2. Necessity of GAMTA and clarification of Eqs. (3)–(7) (W2, Q3–Q6)
Why is GAMTA necessary if RGB and X share the same Task-Factor structure?
Sharing the same factor definitions does not guarantee spatial correspondence. RGB and X observe different physical signals, so their objectness responses, boundaries, and local geometric evidence may differ even when the original observations are approximately registered.
GAMTA establishes task-space correspondence before cross-modal discrepancies are evaluated and auxiliary evidence is incorporated into the RGB detector. We have clarified this motivation in Section 3.3 and strengthened the corresponding ablations in Section 4.4.
Q3: Global transformation and multi-scale warping in Eq. (3).
The predictor G5 estimates a 2D similarity transformation comprising translation, rotation, and isotropic scaling, rather than a general 3D rigid-body transformation.
The transformation is estimated from P5 Task Factors and applied consistently to P3, P4, and P5. Each level then performs its own residual correspondence refinement.
Q4: Spatial location, candidate count, and residual offsets in Eq. (4).
Here, p denotes a spatial location on the RGB feature grid. A lightweight convolutional matcher predicts K=4 residual sampling offsets δkl(p) and their correspondence weights from paired RGB and globally warped auxiliary task cues.
The maximum offset ranges are 3, 2, and 1 feature-grid units at P3, P4, and P5, respectively. The residual displacement is further guided by local correspondence supervision. An additional Null-Match state allows the model to reject unsupported auxiliary correspondences.
These details are specified in the supplementary model configuration and alignment-supervision descriptions.
Q5: Definition of Hnorm in Eq. (6).
Hnorm is the normalized Shannon entropy of the non-Null matching distribution, divided by logK.
It measures correspondence ambiguity: a concentrated distribution indicates greater matching confidence, whereas a diffuse distribution indicates greater uncertainty. Alignment confidence combines this measure with the non-Null correspondence probability.
Q6: Definitions of σ and Sl in Eq. (7).
σ denotes the sigmoid activation function, and Sl is a learned, scale-specific convolutional reliability estimator.
The estimator uses alignment confidence, RGB-X objectness and geometry discrepancies, and predicted geometric uncertainties. Its output is further constrained by alignment confidence, auxiliary spatial validity, and modality presence.