acceptodds
Under review as a conference paper at ICLR 2027

M2C: Matchability-Aware Mutual-Consistency Fusion for RGB–Infrared Object Detection

Abstract

We propose M2C, a matchability-aware mutual-consistency fusion method for RGB–infrared object detection. For each infrared query, M2C augments the local RGB candidate set with an explicit no-match state, measures ambiguity through the concentration of the conditional candidate distribution, and verifies the forward correspondence using reverse matchability and forward–reverse soft-offset agreement. The resulting confidence serves as a continuous evidence-admission signal that regulates RGB routing and gates target-aware cross-stage updates to the infrared stream. This formulation distinguishes finding a relative local match from determining whether its RGB evidence is reliable enough for fusion, while requiring neither cross-modal correspondence annotations nor an additional cycle-consistency loss. Experiments on DroneVehicle and LLVIP demonstrate the effectiveness of M2C across aerial and pedestrian scenes. On DroneVehicle, M2C achieves 63.67% , outperforming a controlled LocalMatch baseline by 0.62 percentage points. Under RGB replacement, the improvement increases to 5.45 points, while the clean-to-degraded performance drop decreases from 14.98 to 10.14 points. These results show that structured correspondence reliability enables more selective and robust RGB–infrared interaction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
RGB–Infrared Object Detection, Multimodal Fusion, Correspondence Reliability, Matchability-Aware Fusion, Robust Object Detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dA7hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose M2C, a matchability-aware mutual-consistency fusion method for RGB–infrared object detection. In detail, this method augments the local RGB candidate set with an explicit no-match state, measures ambiguity through the concentration of the conditional candidate distribution, and verifies the forward correspondence using reverse matchability and forward–reverse soft-offset agreement. The resulting confidence serves as a continuous evidence-admission signal that regulates RGB routing and gates target-aware cross-stage updates to the infrared stream. The experiments on both DroneVehicle and LLVIP is done to demonstrate the effectiveness of M2C across aerial and pedestrian scenes.

Critical strengths

Totally seen, there are some Critical Strengths in this paper. They are listed as follows.

  1. A matchability-aware mutual-consistency (i.e., M2C) fusion method for RGB–infrared object detection is proposed.
  2. In scheme, it distinguishs forced local correspondence from reliable evidence admission in RGB–IR detection, and combines no-match, conditional concentration, and reverse verification into a continuous confidence signal that controls RGB routing and infrared feedback.
  3. Extensive experiments on both both DroneVehicle and LLVIP are peformed to demonstrate the effectiveness of proposed M2c method.
Critical weaknesses

In this paper, there are some Critical Weaknesses.

  1. In Abstract / Introduction, it is unclear why the M2C is proposed for RGB–infrared object detection (research motivation?). What are the problems existing in current RGB–infrared object detection methods? What's the difference between M2C and others?
  2. Most figures in this paper, e.g., Fig 1 and Fig 2, are with low quality, even very difficult to see.
  3. Observed from Fig 1 and its explanations, the novelty of M2C framwork is limited, seeming incremental. The idea of Matchability-Aware could be seen C²Former‌ (2024) / FGSFNet‌, and the idea of mutual-consistency is also seen in EPNet++‌ (2021). It could be seen that many other methods adopt similar schemes, e.g., the MCF-YOLO‌ and the BGCF-Net‌.
  4. The perforamnce improvement is minor. In table 1 (on DroneVehicle), the gain of M2C is only 0.8 (84.4 vs. 83.6). And, in table 2 (on LLVIP), the mAP50 is only 96.7, less than the 97.0 obtained by Fusion-Mamba.
  5. Experiment visualization is weak. Only resultant visualization is demonstrated, ignoring the evidences of intermediate process.
Questions for authors

In this paper, there are some concern points to be posted to the authors. Please further clarify or explain or reply these concerned questions below.

1.Existing problems & motivation & differences

In Abstract / Introduction, it is unclear why the M2C is proposed for RGB–infrared object detection (research motivation?). What are the problems existing in current RGB–infrared object detection methods? What's the difference between M2C and others?

2.Figure Quality

Most figures in this paper, e.g., Fig 1 and Fig 2, are with low quality, even very difficult to see.

3.Novelty & Contriubtion

Observed from Fig 1 and its explanations, the novelty of M2C framwork is limited, seeming incremental. The idea of Matchability-Aware could be seen C²Former‌ (2024) / FGSFNet‌, and the idea of mutual-consistency is also seen in EPNet++‌ (2021). It could be seen that many other methods adopt similar schemes, e.g., the MCF-YOLO‌ and the BGCF-Net‌.

4.Perforamnce improvement

The perforamnce improvement is minor. In table 1 (on DroneVehicle), the gain of M2C is only 0.8 (84.4 vs. 83.6). And, in table 2 (on LLVIP), the mAP50 is only 96.7, less than the 97.0 obtained by Fusion-Mamba.

5.Visualization evidences

Experiment visualization is weak. Only resultant visualization is demonstrated, ignoring the evidences of intermediate process.

1 comment

Thank you for the comments. We have clarified the motivation and contribution and added intermediate evidence. AP below denotes mAP$_{50:95}$.

1. Motivation. A Softmax over real candidates assigns unit mass even when none provides informative RGB evidence. M2C separates real-versus-no-match support, conditional concentration, and reverse verification, then uses their confidence to regulate RGB routing and IR feedback (§§1, 3.2–3.3).

2. Contribution and related work. Appendix G compares the mechanism with C²Former/FGSFNet's attention-based fusion, EPNet++'s prediction-score consistency, MCF-YOLO's IR-prior regularization, and BGCF-Net's bidirectional gating. M2C's specific contribution is the explicit no-match/concentration/reverse-offset chain and its joint use in a routing log-prior and IR residual multiplier. The reverse branch verifies correspondence rather than updating a second feature stream. We credit matching and fusion precedents, including LightGlue.

3. Figure quality. Appendix B adds simplified vector schematics (Figures 6–7), complementing the original Figures 1–2. The main text now defines the projected features, logits, operators, and losses.

4. Performance. We acknowledge the small DroneVehicle mAP${50}$ gain over MDCFT-HSM (84.4 versus 83.6) and lower LLVIP mAP${50}$ than Fusion-Mamba (96.7 versus 97.0; mAP$_{50:95}$: 64.4 versus 64.3). In the controlled comparison, Full's clean advantage over LocalMatch is 0.62 AP points, increasing to 5.45 under RGB replacement. IR-only and joint γ=3\gamma=3 degradation yield advantages of 1.68 and 2.17 (Table 11). IR replacement remains a limitation: 8.78 versus 12.43 AP. We therefore claim selective RGB admission with a usable IR reference.

5. Intermediate evidence. Figures 8–10 and Tables 9–10 add actual maps, input overlays, and dataset-level statistics. Under RGB replacement, P3 foreground confidence falls 0.1772→0.0339, RGB weight 0.2024→0.0383, and actual IR-update RMS decreases 92.95%; individual cues need not all decrease. Routing-prior bypass further reduces RGB-drop AP from 53.53 to 23.43 (Table 12), demonstrating functional dependence within the trained model, not a retrained component's net gain.

Reviewer_cbJWOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper introduces M2C, a fusion module for RGB-IR object detection. The authors claim that standard multi-modal attention mechanisms can force misleading feature blends when one sensor is degraded. So here the authors treat the Infrared stream as a fixed spatial reference baseline and design a gating mechanism for the RGB stream. The M2C fusion module estimates the best match RGB feature for every IR feature and then fuses them only if the reliability score of the matched features is above certain threshold. This is also very similar to how LighGlue does matchability of features between image pairs.

Critical strengths

None

Critical weaknesses
  1. The paper is difficult to understand, especially section 3.2 which introduces the core M2C module. Figure 2 is confusing and very hard to follow, and the variables used in the figure do not always match with the text in sections 3.2 and 3.3.
  2. The corrospondence score used is an arbitrary combination of the three scores (m_f, q_f and s). There is no clear explanation of why this function is used. In fact one of the ablation studies (table 6) show that if you remove one of these (w/o concentration) from the score function the results improve.
  3. The results in table 1 and 2 do not show a major boost in accuracy. In fact for LLVIP dataset fusionMamba beats the
  4. The computation cost of the proposed method (M2C-E1) seems very high as compared to a strict baseline (which I am assuming is simple fusion).
Questions for authors

Questions:

  1. Why does w/o Concentration experiment (in table 6) gives best result? Can you provide an intuitive explanation?
  2. Can you clarify what is your baseline in the ablation studies?
  3. What is the soft Dice loss (L_{Dice}) and L_{det} in equation 7?
  4. In section 3.3 the semantic representation (Z) is introduced without any explanation, can you provide more information about this Z?
  5. What do you mean by modality logits (L_I, L_R) in section 3.3?
  6. Please define the variables and the operators in equation (6). I dont understand what you are trying to do there and how?

Suggestions:

  1. Please redraw figure 2 to make it consistent with respect to the text.
  2. Generate a qualitative figure (similar to attention maps) showing heatmaps of the variables matchability (m_{f}), concentration (q_{f}), and the final confidence score (c) overlaid onto the input images. Show a case under normal conditions versus a case under your "RGB drop" or spatial shift conditions. This will help in visualizing the importance of these parameters and will strengthen the claim of your gating logic.
2 comments

Thank you for identifying the unclear definitions. We have expanded Sections 3.2–3.4 and Appendices A–C. AP below denotes mAP$_{50:95}$.

1. Fusion rule and presentation. M2C uses continuous control: cc enters the RGB logit as log⁡max⁡(c,ϵ)\log\max(c,\epsilon) and multiplies IR feedback, without a hard acceptance threshold. We acknowledge LightGlue's matchability precedent. Appendix B adds simplified vector Figures 6–7 alongside the retained original diagrams.

2. ZZ, logits, and Equation (6). We now define

Z=ψ([Is,Rs,a,∣Is−Rs,a∣]),Z=\psi([I_s,R_{s,a},|I_s-R_{s,a}|]),

where Is,Rs,aI_s,R_{s,a} are encoded IR and aligned RGB, and ψ\psi is 1×11\times1 Conv–GroupNorm–SiLU. Spatial/channel heads produce unnormalized modality scores LI,LRL_I,L_R. Modality Softmax, with the RGB confidence prior, gives PI,PRP_I,P_R; wR=0.95PRw_R=0.95P_R, wI=1−wRw_I=1-w_R.

I^+=I+gtc\odot\phi(F-I),\qquad R^+=R.$$ $I,R$ are incoming IR/RGB features, and $R_a$ is aligned RGB; $\odot$ is elementwise multiplication; $\phi$ is pointwise Conv–GroupNorm; $g=\sigma(\gamma)$ is a learned stage gain; $t$ is the predicted target prior. Maps $t,c$ broadcast over channels. $F$ provides fused detection features; P3/P4 IR updates feed later backbone stages. **3. Loss definitions.** For predictions $p=\sigma(z)$ and box-derived targets $y$, soft Dice is the batch average of $$1-\frac{2\sum_i p_i y_i+1}{\sum_i p_i+\sum_i y_i+1}.$$ $\mathcal L_{\rm det}$ combines CIoU regression, classification BCE, and distribution focal loss with weights 7.5/0.5/1.5. Each scale's auxiliary focal-plus-Dice loss has weight 0.05 (§3.4; Appendix B). **4. Formula and concentration.** Geometric averaging balances product suppression against arithmetic averaging's compensation between cues. Table 8 adds five fixed-weight formulas across nine conditions; the original is best in eight, with IR replacement as the exception. This is sensitivity analysis, not a retrained ranking. Removing concentration improves clean AP (63.67→63.84) but reduces RGB-replacement AP (53.53→49.49), consistent with a tradeoff involving useful diffuse evidence (Table 6; Appendix F). **5. Baselines and cost.** Strict lacks E1 preparation/context refinement; E1-ADD adds these with direct addition; LocalMatch adds forced local matching; Full adds reliability control. The total Strict-to-Full gain thus includes supporting changes. Full costs 25.37 ms versus LocalMatch's 21.70 ms (+16.9%) and Strict's 11.49 ms on RTX 4090, FP32, batch 1, forward-only (Tables 3, 7; Appendix F). **6. Requested visual evidence and accuracy.** Figures 8–10 provide actual clean/shift/RGB-drop maps and input overlays. Under RGB replacement, P3 foreground $q_f$ falls 0.0552→0.0020, $c$ 0.1772→0.0339, and RGB weight 0.2024→0.0383; actual update RMS decreases 92.95%. Matchability remains high and reverse support increases (Tables 9–10). We acknowledge Fusion-Mamba's higher LLVIP mAP\$_{50}\$ (97.0 versus 96.7); our mAP\$_{50:95}\$ is 64.4 versus 64.3.

It is not clear to me if the authors have correctly addressed my questions. I dont understand how the answers (1) and (2) corresponds to my first and second question. Looks like (2) is an answer to my fourth question. Anyway, thanks for your response, I maintain my rating of 2.

Reviewer_FLhfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes M2C, a matchability-aware fusion method for RGB–infrared object detection. The core idea is to distinguish between “local correspondence estimation” and “whether the correspondence is reliable.” M2C constructs a continuous confidence score through an explicit no-match state, candidate distribution concentration, and reverse verification, and uses this confidence to control RGB information routing and infrared feature updates. Experiments on DroneVehicle and LLVIP show that the proposed method achieves competitive detection performance and exhibits better robustness when the RGB input is degraded.

Critical strengths

S1. The research motivation is clear and practically meaningful.

S2. The reliability modeling is intuitive and interpretable. The no-match state, concentration, and reverse verification play distinct roles.

Critical weaknesses

W1. The improvement brought by the core reliability mechanism under clean conditions is relatively limited. The effectiveness of the proposed reliability mechanism is not fully convincing.

W2. The robustness experiments mainly focus on RGB modality degradation. The robustness evaluation is not sufficiently convincing.

Questions for authors
  1. In the controlled comparison, E1-LocalMatch achieves 63.02 mAP50:95, while M2C achieves 63.72, corresponding to an improvement of only about 0.70 percentage points. The effectiveness of the proposed reliability mechanism is not fully convincing.
  2. I suggest further evaluating IR degradation, as well as simultaneous degradation of both RGB and IR, to better support the robustness claim.
  3. The paper defines the final confidence as c=m_f √(q_f s), but the justification for this specific formulation is currently limited. It would be helpful to provide further explanation or sensitivity analysis to better demonstrate the rationale behind this design.
  4. In the ablation study, the effectiveness of the Concentration module is not fully convincing. After removing this module, the clean performance increases from 63.67 to 63.84, suggesting that its actual contribution under normal conditions remains unclear.
1 comment

Thank you for the constructive comments. Below, AP denotes mAP$_{50:95}$; differences are percentage points.

1. Reliability gain. We agree that the clean gain is modest: 0.70 points in Table 3 and 0.62 under the shared evaluation pipeline (Table 11). Under RGB replacement, however, Full outperforms LocalMatch by 5.45 points (53.53 versus 48.07), with clean-to-degraded losses of 10.14 versus 14.98. This supports selective admission under unreliable RGB evidence.

2. IR and simultaneous degradation. Appendix D adds IR-only and joint gamma degradation on all 1,469 validation pairs. At γ=3\gamma=3, Full exceeds LocalMatch by 1.68 points for IR-only degradation and 2.17 for joint degradation; after subtracting the clean gap, the advantages are 1.07 and 1.55. Complete IR replacement remains a failure: 8.78 versus 12.43 AP. We therefore restrict the claim to selective RGB admission with a usable IR reference, without claiming uniform robustness.

3. Confidence formulation. Section 3.2 and Appendix A explain that mfm_f measures real-candidate support, qfq_f conditional concentration, and ss reverse support. The geometric mean suppresses less than a product while penalizing a low verification cue even when the other is high. Table 8 compares mf(qfs)αm_f(q_fs)^\alpha for α=0.25,0.5,0.75,1\alpha=0.25,0.5,0.75,1 and arithmetic averaging across nine conditions. The original formula performs best in eight; IR replacement is the exception. These fixed-weight results support the trained operating point, not optimality after independent retraining.

4. Concentration tradeoff. The trained ablation (Table 6) shows that removing concentration improves clean AP from 63.67 to 63.84 but reduces RGB-replacement AP from 53.53 to 49.49. A plausible explanation is suppression of useful diffuse RGB evidence on clean inputs, balanced against rejection of uninformative evidence. Appendix F explicitly discusses this tradeoff.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.