acceptodds
Under review as a conference paper at ICLR 2027

Geometry Meets Semantics: Rethinking Bounding Box Regression with Semantic-Aware Fractional Gradients

Abstract

Bounding boxes are fundamental for object localization in visual detection tasks. Among them, oriented bounding boxes are widely used in visual detection tasks, which provide a more precise directional representation. Generally, IoU-based losses are widely adopted to optimize box regression. However, we observed that IoU-driven box optimization suffers from two key issues: (1) it relies solely on geometric properties while ignoring semantic cues; (2) orientation optimization suffers from unstable gradients, causing oscillations in orientation convergence. In this paper, we propose a Fractional Semantic IoU loss to achieve unified semantic-geometric learning with gradient stabilization. First, we design a semantic similarity metric to guide IoU optimization, building a Semantic IoU loss (SIoU loss) with an adaptive gradient gating mechanism. Then, we revisit the gradient instability issue in oriented box optimization and extend the SIoU loss to a fractional-order formulation to build the Fractional Semantic IoU loss (FrSIoU loss). The FrSIoU loss accumulates historical IoU states to regularize abnormal gradients during the bounding box optimization process. Extensive experiments demonstrate that our approach achieves stable performance gains across different bounding box formulations and diverse visual detection tasks. The code will be available on GitHub.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Remote Sensing, Object detection, FrFT

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XMNiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes FrSIoU for oriented bounding-box regression by combining semantic guidance with a fractional-order history-based modulation mechanism. The semantic component introduces feature-level similarity into IoU-based regression, while the fractional component is intended to stabilize abnormal angular gradients using historical semantic-geometric similarity. The method is evaluated on multiple oriented detection benchmarks and reports consistent improvements, including gains on high-precision localization metrics.

The overall direction is interesting and the empirical results are promising. However, I have several concerns regarding the theoretical analysis underlying the proposed gradient stabilization mechanism and whether the experiments sufficiently isolate the claimed contribution.

Critical strengths

The semantic-geometric coupling is intuitive and potentially useful. Instead of treating semantic supervision as an independent auxiliary objective, the method directly uses semantic consistency to modulate geometric regression. The evaluation also covers multiple datasets and detector architectures, which provides useful evidence of generalization.

Critical weaknesses

1. The angular-gradient analysis in Sec. 3.1 seems to rely on a geometric assumption that does not hold throughout the claimed optimization regime.

Eq. (2) is derived under the assumption that the intersection of two center-aligned OBBs is a parallelogram. However, as the angular error approaches zero, the intersection is no longer a parallelogram, which is also implicitly acknowledged by the boundary-condition analysis in Appendix C. Therefore, the statement that “the angular derivative grows as the angular error decreases” does not appear to hold generally over the whole convergence process. More specifically, the derived expression is valid only within a particular intersection topology and should not be extrapolated to the zero-angle limit. This issue weakens part of the motivation for the subsequent gradient-stabilization mechanism. The authors should clarify the valid angular range of the analysis and, ideally, verify the claimed gradient behavior using the exact rotated-IoU computation rather than the parallelogram approximation.

2. The Semantic IoU formulation may suppress useful geometric gradients in semantically ambiguous cases.

The proposed SIoU loss multiplicatively gates the geometric IoU term using the semantic score. Consequently, when the semantic score is close to one while the geometric IoU is still very low, the loss and its gradient with respect to box parameters can both become very small. This may be problematic in scenes containing repeated or visually similar objects, where a predicted box can have high semantic similarity to the target while being geometrically far from the correct instance. In such cases, the semantic term may prematurely suppress the localization gradient and prevent effective parameter updates. The paper should explain how this failure mode is avoided and provide empirical evidence, for example by analyzing cases with high semantic similarity but low IoU.

3. The interpretation of the fractional modulation as a gradient-stabilization mechanism needs further justification.

For some valid parameter settings, a stationary sIoU sequence can still produce a modulation weight smaller than one. This behavior may be desirable when several consecutive sIoU values are already close to one, because reducing the update magnitude near convergence is reasonable. However, the same behavior can also occur when several consecutive sIoU values remain consistently low, in which case further suppressing the update may be undesirable because stronger correction is arguably needed. Therefore, the current interpretation that the fractional signal distinguishes improving and deteriorating optimization states appears incomplete. The authors should clarify whether the method is intended to model temporal trend, similarity level, or both, and analyze how it behaves for high-quality and low-quality stationary trajectories

4. The comparison with existing gradient-stabilization methods is insufficient.

The main motivation of FrSIoU is to address abnormal gradients in rotated IoU optimization, but closely related methods such as Gradient Calibration Loss directly study and correct the same type of angular-gradient instability. GCL explicitly analyzes the negative correlation between rotated-IoU gradient magnitude and angular error and proposes gradient calibration to stabilize convergence. The paper would be significantly stronger with a controlled comparison against GCL and other dynamic reweighting approaches. Without such comparisons, it is difficult to determine whether the observed gain is specific to the proposed fractional-order mechanism or can be achieved by simpler gradient calibration or history-based weighting strategies.

5. The gradient derivation in Appendix A is unclear with respect to historical states.

Appendix A differentiates historical IoU states with respect to the current angular parameter and interprets the resulting expression as a weighted combination of historical geometric gradients. However, in standard iterative training, historical sIoU values are usually computed using previous network parameters and stored as detached values. If this is the actual implementation, their gradients with respect to the current parameters should be zero. If, instead, the method retains previous computation graphs and differentiates through the optimization history, this would imply a substantially different implementation and nontrivial memory cost. The paper should clearly explain how the historical states participate in automatic differentiation and ensure that the theoretical derivation is consistent with the actual implementation.

Questions for authors

Besides of the above weaknesses, there are two minor concerns:

1. Eq. (11) appears to contain an error.

As currently written, the coefficient recurrence in Eq. (11) does not generate the coefficients subsequently used in Eq. (12). The coefficients in Eq. (12) are consistent with the standard Grünwald–Letnikov recurrence, suggesting that Eq. (11) likely contains a typographical or algebraic error.

2. Two important hyperparameters of the fractional formulation are insufficiently specified and analyzed.

The fractional order and the number of historical steps directly determine the coefficients and temporal behavior of the proposed fractional operator, yet the paper does not clearly report their values in the experimental configuration and provides no ablation study for either parameter. Since both are central to the claimed memory and stabilization effects, their implementation values, sensitivity, and impact on performance should be explicitly reported.

Reviewer_XcQwOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes an IoU loss function that constructs a semantic IoU loss by exploiting the feature similarity inside predicted and ground-truth boxes as well as the feature similarity of their corresponding context regions, and guides bounding-box regression through the joint modeling of semantic and geometric information. On this basis, historical semantic–geometric similarity is further used to dynamically adjust the loss weight, aiming to alleviate gradient fluctuations during optimization. However, the paper has not yet provided sufficient evidence that semantic guidance can offer reliable and effective additional supervision for bounding-box localization. Meanwhile, whether the fractional-order mechanism genuinely mitigates angular gradient instability—rather than merely performing dynamic loss weighting—also requires stronger theoretical analysis and experimental support.

Critical strengths

1 The paper addresses the gradient instability problem in rotated bounding-box regression, which is a topic of research value for improving target localization accuracy. 2 The proposed loss function can be integrated into existing object detection frameworks without requiring substantial modifications to the detector architecture.

Critical weaknesses
  1. The authors construct a semantic score from the feature similarity inside predicted and ground-truth boxes and from the feature similarity of the corresponding context regions. However, when a predicted box is already close to the ground-truth box, a small localization error may not induce a noticeable change in semantic similarity; consequently, the semantic term may not provide effective additional localization information.
  2. The idea of using historical semantic and geometric similarity to dynamically adjust the loss weight is reasonable to some extent. Nevertheless, the paper has not yet adequately demonstrated that this mechanism fundamentally alleviates angular gradient instability. The authors are encouraged to provide more direct gradient analysis to substantiate the necessity of the fractional-order mechanism and its independent contribution.
  3. The authors are suggested to conduct ablation studies on the inside-box feature similarity, the context feature similarity, their combination, and the fractional-order modulation module separately, so as to clarify the necessity of each component. A direct comparison with related gradient stabilization methods would further strengthen the credibility of the experimental conclusions.
Questions for authors
  1. Can the authors demonstrate that, under small localization errors and weak feature contrast between the target and the background, the proposed semantic score still produces changes that are meaningful for localization guidance, and yields gradients with the correct direction?
  2. Can FrSIoU explicitly reduce the oscillation of the angular gradient? Could the authors provide direct evidence through gradient statistics, gradient curves, and controlled experiments?
Reviewer_gQLdOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper revisits the limitations of IoU-based loss functions in arbitrary-oriented object detection. Theoretical analysis in Section 3.1 concludes that there exists a severe gradient issue in angle prediction, especially when the predicted angle is close to the target one. To overcome that limitation, the paper proposed a Semantic IoU loss (SIoU loss), which is basically achieved by introducing the feature distance ss between the predict box and the target box. Coupled with the IoU loss term, SIoU loss = (1−s)(1−IoU)(1-s)(1-IoU), refer to Eq. 8. To further stabilize abnormal gradients, the paper proposes a fractional semantic IoU (FrSIoU) loss, which introduces a the G-L derivative term as a weight factor to SIoU loss.

Critical strengths

The paper proposes to use the G-L derivative to assist in smoothing the bbox loss, which is new.

Critical weaknesses
  1. The theoretical analysis is not rigorous enough. In Section 3.1, the authors claim that the angle gradient is not stable when the angles of the two boxes are close. However, the analysis are mostly based on the assumption that the intersection of the two boxes is a parallelogram. This requires that the aspect ratio of both boxes be large enough. If the aspect ratio is not large, the intersection of two boxes will probably not be a parallelogram, but be a octagon. The analysis of the IoU loss gradient becomes invalid. For many cases, the aspect ratios of boxes are not large, generally less than 5 in common datasets. Thus the assumption of the angle gradient issue in Section 3.1 is too strong, making the motivation less convincing.

  2. The experimental results are not convincing and the improvement is limited. Table 1 shows that SIoU loss can only improve 0.15 mAP50 over GIoU loss. And FrSIoU loss only improves 0.45 mAP50 over SIoU loss. The paper also did not directly compare with state-of-the-art loss functions in arbitrary-oriented object detection, e.g., KFIoU [1], ACM [2], COBB [3]. The validation is not sufficiently conducted.

  3. Another major concern is the lack of detail in the experimental implementation. The FrSIoU loss calculates the G-L derivative, which needs to record the bounding boxes or SIoUs in the historical iteration steps. This part is unclear and the paper did not provide the implementation details. Reproducibility is questionable.

  4. The proposed method introduces many hyper-parameters, some of which have not been experimentally verified, e.g., MM in Eq. 10.

Overall, considering the weak motivation, lack of implementation details, limited improvements, and insufficient evaluation, the paper requires quite a major overhaul to meet the acceptance bar of ICLR.

[1] Yang X, Zhou Y, Zhang G, et al. The KFIoU loss for rotated object detection. In ICLR 2023.

[2] Xu H, Liu X, Xu H, et al. Rethinking boundary discontinuity problem for oriented object detection. In CVPR 2024: 17406-17415.

[3] Xiao Z, Yang G, Yang X, et al. Theoretically achieving continuous representation of oriented bounding boxes. In CVPR 2024: 16912-16922.

Questions for authors

A detailed implementation description are required. The motivations need to be reorganized, weakening assumptions, or at least based on common characteristics of existing datasets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.