- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
DeCo jointly learns referring-expression localization and segmentation. It separates task-specific features from a shared vision–language encoder, then couples the branches using text features and a mask-derived spatial prior. A localization decoder predicts boxes and an auxiliary mask, while a UNet predicts the final segmentation mask. The training objective combines task losses with a mask-to-box loss. Experiments evaluate both tasks under dataset-specific and mixed training, including remote-sensing data and component comparisons.
Critical strengths
-
The model performs well on both localization and segmentation, including against methods built on the same BEiT3 backbone family.
-
The component comparisons show that decoupling and prior coupling each add value over naive joint training.
Critical weaknesses
- The claimed geometric feedback relies on a box extracted from a predicted binary mask. If that mask is hard-thresholded during training, the conversion blocks this gradient path and makes the mask-to-box loss ineffective for training the mask predictor; the paper gives no alternative backward rule.
Questions for authors
-
What tensor is passed to mask-to-box during training: probabilities or a thresholded mask? Please show the backward rule and clarify whether the localization and mask-to-box losses update the UNet through this conversion.
-
If the box is detached only before entering the localization decoder, while the separate mask-to-box loss is left unchanged, does segmentation accuracy change? This isolates feedback from localization rather than the benefit of using the mask as a forward prior.
-
Do the DoM and full-coupling comparisons use the same auxiliary instance-mask supervision and loss weights? If not, can you separate the gain from extra supervision from the gain due to coupling?
-
Do mask-derived boxes become accurate earlier than localization predictions during training? Curves for these two outputs would directly check the stated premise for the spatial prior.
4 comments
Official Comment by Authors of DeCo
Q3. Do the DoM and full-coupling comparisons use the same auxiliary instance-mask supervision and loss weights? If not, can you separate the gain from extra supervision from the gain due to coupling?
A: Thanks for raising this concern. Yes, all experiments use the same auxiliary instance-mask supervision and loss weights, expert for LoM and SoM, as listed in the following table.
| Models | ||||
|---|---|---|---|---|
| LoM | ✓ | ✓ | × | × |
| SoM | × | × | ✓ | ✓ |
| DoM | ✓ | ✓ | ✓ | ✓ |
| DoM w/ TSD | ✓ | ✓ | ✓ | ✓ |
| DoM w/ HPC | ✓ | ✓ | ✓ | ✓ |
| DeCo | ✓ | ✓ | ✓ | ✓ |
We have added the corresponding description in the revised Section 4.3 (Effects of Core Components).
Q4. Do mask-derived boxes become accurate earlier than localization predictions during training? Curves for these two outputs would directly check the stated premise for the spatial prior.
A: Thanks for highlighting this suggestion. We illustrate additional convergence curves, as shown in the revised Section 4.3 (Effects of Core Components) and Figure 3(a)-(c).
Specifically, Figure 3(a) compares the validation performance of different model variants, showing that DeCo achieves consistently strong performance across REC and RES, with TSD and HPC providing complementary improvements over DoM. Figure 3(b) directly compares the localization accuracy of REC-predicted boxes from LoM and mask-derived boxes from SoM throughout training. The results show that mask-derived boxes provide more accurate localization cues during early training, although LoM gradually catches up as training progresses. Figure 3(c) further illustrates the corresponding loss convergence of the two branches.
These observations support our motivation for incorporating dense segmentation cues as spatial priors into the localization query through HPC, facilitating faster convergence and more accurate localization.
Thanks for clarifying that M2B works as a forward prior. The gradient ablations and added training curves address my main concerns. I’m satisfied with the response and will keep my score of 3 (Weak Accept).
Official Comment by Authors of DeCo
We sincerely thank Reviewer tAKu for the positive feedback, and we are glad that our additional experiments and clarifications have addressed your primary concern. Once again, we sincerely appreciate your valuable time and effort, responsible and thorough review, and constructive suggestions.
Official Comment by Authors of DeCo
We sincerely thank you for the time and effort devoted to our paper and for your careful and constructive comments! We hope our point-by-point responses below can address your concerns, and we look forward to any further discussion.
Q1. What tensor is passed to mask-to-box during training: probabilities or a thresholded mask? Please show the backward rule and clarify whether the localization and mask-to-box losses update the UNet through this conversion.
A: Thanks for pointing out this issue. During training, we pass a thresholded binary mask rather than probabilities to the mask-to-box operation. We acknowledge that the original manuscript did not clearly distinguish this forward spatial prior, and have revised the relevant descriptions in Figure 1, Introduction, Sections 3.1, 3.3, and 3.4 accordingly.
To further investigate this issue, we conduct additional experiments with five M2B configurations (A–E), as reported in Section 4.3 (Effects of Mask-to-Box Variants), Table 9, and Figure 3(d). Specifically, we introduce a straight-through estimator (STE) based differentiable alternative that preserves hard box extraction in the forward pass while approximating backward gradients using probability-weighted spatial moments, as detailed in Appendix F. As shown in the following table, the original hard M2B (A) achieves the best performance (76.6%), while removing Lm2b yields comparable results (76.5%). The differentiable alternatives (C–E) achieve lower scores (75.0%–76.3%) and exhibit unstable convergence in Figure 3(d).
These findings support retaining M2B as a forward geometric prior that transfers spatial knowledge from RES to REC, facilitating more accurate localization. We have clarified the use of Lm2b as an optional objective in Section 3.4.
Q2. If the box is detached only before entering the localization decoder, while the separate mask-to-box loss is left unchanged, does segmentation accuracy change? This isolates feedback from localization rather than the benefit of using the mask as a forward prior.
A: Thanks for raising this concern. We conduct the requested isolation experiment using STE-based M2B. Specifically, both configurations retain Lm2b, while the box is detached only before entering the localization decoder in configuration D.
Thus, D removes the gradient from Ldet to RES through M2B while preserving the gradient from the mask-to-box loss. As shown below, detaching the localization input results in only marginal changes in segmentation performance: oIoU decreases from 71.4% to 71.0%, while mIoU decreases from 72.2% to 72.1%. These results suggest that localization-to-segmentation gradient feedback provides limited additional benefits.