- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors proposed a sine-modulated BCE loss for object detection. They used YOLO8 and COCO and trained the model for 15 epochs from scratch and compared different loss functions like BCE, Focal Loss, Cost-sensitive losses like effective number of samples, and their proposed sine-modulated loss. The analysis shows that the sine-modulated loss favours the medium-difficulty samples and it produces low gradients for too easy and too hard examples. The authors conducted ablations and showed that the baseline loss is best for the 15-runs and the sine-modulated loss second best outperforming the Focal loss.
Critical strengths
The sine-modulated loss looks interesting and it is something that have never seen before for the object detection task. I like the analysis of the gradient and it shows that this design favours the medium-difficulty samples.
Critical weaknesses
The current experiments and analysis is insufficient. In detail, the experiments are run for 15 epochs not the full 300, and the results are suboptimal ~40 AP in COCO. There is little breadth, only one model and one dataset thus it is not easy to say whether the sine-modulated loss works in more general applications.
Questions for authors
I would like to give some suggestion to the authors to improve the quality of the paper. First, I would like to see comprehensive ablation at 300 epochs because the current study only uses 15 and it is too little training epochs to draw some valuable conclusions. Also, the authors could consider using this loss to 3 datasets in total e.g. they could use LVIS dataset [a1] that fits better with the rare-category narrative and the V3DET [a2] or OpenImages [a3]. Next, regarding the model, I think the authors could consider adding more models in the survey. There are many sources of imbalance in detection no class imbalance only, and there have been many architectures that tackle different imbalance problems [a4]. For example, I would like to see whether this loss can benefit Detr architectures, particularly their sampling through the Hungarian loss. Also, do we need this loss in RPN networks? Next, what is the cost of the sine-modulated function, is it slow during training or stable? Going beyond detection, there are sota papers like Deit3 that use BCE to train for image classification, so, does this modulating loss work for classification either balanced or imbalanced datasets like ImageNet-1K and/or ImageNet-LT, iNaturalist18 and Places-LT [a5]. Regarding the eval-metris, COCO mAP could be problematic as shown here [a6] so please consider giving more eval-metrics and visual and qualitative explanations of the loss and the behavior and the representation learned
[a1] Gupta et. al. 'LVIS: A Dataset for Large Vocabulary Instance Segmentation; in CVPR19 [a2] Wang et. al. 'V3Det: Vast Vocabulary Visual Detection Dataset' in ICCV2023 [a3] Kuznetsova et. al. 'The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scale' in IJCV 2020 [a4] Oksuz et. al. 'Imbalance Problems in Object Detection: A Review' tPAMI [a5] Liu et. al. 'Large-Scale Long-Tailed Recognition in an Open World' in CVPR2019 [a6] Dave et. al. 'Evaluating Large-Vocabulary Object Detectors: The Devil is in the Details' in Arxiv.
3 comments
Thank you for your answers
My concerns are not addressed yet. The LVIS results are too low to be informative, in the future please see EFL CVPR2022 [a1] for best practices of YOLO training in LVIS. The COCO finetuning is not convincing, because the COCO pretrained model used a sigmoid loss, not a sine-modulated, there could be a misalignment during the finetuning, training-from-scratch is a more convincing experiment. Regarding the breadth, I think there should be more experiments with DETR and some discussion beyond object detection makes the paper more complete. I will keep my score.
[a1] Li et. al. Equalized Focal Loss for Dense Long-Tailed Object Detection, CVPR2022
Thankyou for the review (Reviewer 4YQE )
Thank you for the further comments. This paper is an ablation study: it explains what each design choice contributes, and it does not aim to prove Sin-SMFL best. Below are all the results that bear on your three points.
1. LVIS and EFL. We now cite EFL and state that our LVIS numbers (YOLOv8s, from scratch, 50 epochs) are for comparing the three losses with each other; an EFL-style setup is future work. On rare categories, Sin-SMFL and Focal reach about 7 times BCE's AP and match each other:
| LVIS v1 val, from scratch, 50 epochs | AP | APr (rare) | APc | APf |
|---|---|---|---|---|
| BCE | 0.1419 | 0.0019 | 0.0903 | 0.2610 |
| Focal | 0.0951 | 0.0141 | 0.0671 | 0.1619 |
| Sin-SMFL | 0.0890 | 0.0136 | 0.0602 | 0.1543 |
2. Fine-tuning vs scratch. We trained from scratch as you suggest, and also fine-tuned for 15, 50 and 100 epochs and on two more model sizes:
| COCO val2017, mAP50-95 | BCE | Focal | Sin-SMFL |
|---|---|---|---|
| YOLOv8s fine-tune, 15 epochs | 0.4121 | 0.4042 | 0.4048 |
| YOLOv8s fine-tune, 50 epochs | 0.4312 | 0.4272 | 0.4278 |
| YOLOv8s fine-tune, 100 epochs | 0.4328 | 0.4322 | 0.4314 |
| YOLOv8s from scratch, 50 epochs | 0.3671 | 0.3500 | 0.3470 |
| YOLOv8n fine-tune, 15 epochs | 0.3422 | 0.3321 | 0.3305 |
| YOLOv8m fine-tune, 15 epochs | 0.4560 | 0.4472 | 0.4471 |
Sin-SMFL is within 0.003 of Focal in every setting. With longer fine-tuning the gap to BCE narrows from 0.0073 to 0.0014 (100 epochs, all three within 0.0014). The from-scratch gains were taken from the fine-tuning sweep and not retuned.
3. Beyond detection. We added CIFAR-100-LT (ResNet-32, 200 epochs, Appendix F). Without class weights BCE, Focal and the sine loss tie (38.3, 37.9, 38.2). With class weights, Sin-SMFL is ahead of Focal at every setting (+1.4 points at beta 0.999, +2.8 at beta 0.99; within seed spread):
| CIFAR-100-LT top-1 (%), 3 seeds | All | Many | Medium | Few |
|---|---|---|---|---|
| BCE (no class weights) | 38.33 | 65.99 | 36.10 | 8.66 |
| Focal (beta 0.999) | 24.64 | 48.59 | 19.70 | 2.46 |
| Sin-SMFL (beta 0.999) | 26.02 | 50.70 | 21.23 | 2.82 |
| Focal (beta 0.99) | 29.22 | 55.50 | 24.78 | 3.73 |
| Sin-SMFL (beta 0.99) | 31.99 | 59.64 | 27.99 | 4.40 |
| Focal (no class weights) | 37.89 | 65.90 | 35.49 | 8.01 |
| Sine (no class weights) | 38.17 | 65.19 | 36.33 | 8.80 |
This is a stand-in, not ImageNet-LT, and we make no claim for DETR.
Why we believe the paper is ready.
- It answers a practical question with controlled evidence: in this detector, gain calibration moves mAP by up to 0.005 (Focal at 100 epochs: 0.4258 to 0.4322), more than the choice of loss shape (under 0.0015 once each loss is tuned).
- The design is rigorous: a factorial ablation, paired per-class comparisons, a measured noise band, the official LVIS evaluator, and every number traced to a logged run.
- Breadth now covers COCO at 15, 50 and 100 epochs, from scratch, three model sizes, LVIS and CIFAR-100-LT.
- Your review and the others led us to match every claim to the evidence.
For these reasons we believe the paper is ready as an ablation study, and we respectfully ask you to reconsider your score.
Thank you for the detailed review. It led to the from-scratch comparison, the 100-epoch runs and the narrower, better-supported claims.
Response to Reviewer 4YQE
Thank you for your review. My reply to each point is below. All changes are in the revised PDF.
Summary, point 1: "trained from scratch". This is not correct, and the paper was not clear. The main runs start from the COCO-pretrained
yolov8s.pt, which scores 0.4409 mAP50-95 before any training. This also explains the AP of about 40. We now say this in Section 4. From scratch, 50 epochs: BCE 0.3671, Focal 0.3500, Sin-SMFL 0.3470 (Section 5.7).Summary, point 2: "sine-modulated loss second best, outperforming Focal". After we tuned the gain of every loss, sine and Focal are equal. Best results at 15 epochs (gains 0.25 to 4; Table 3): BCE 0.4121, Sin-SMFL 0.4048, Focal 0.4042. The difference between Sin-SMFL and Focal is 0.0006, and the run-to-run difference is up to 0.004. Per class, 42 better and 38 worse (p = 0.74). We corrected the abstract and the introduction.
W1. Only 15 epochs, not 300. AP is only about 40. We ran 50 epochs: BCE 0.4312, Sin-SMFL 0.4278, Focal 0.4272 (Table 5). The order and the gaps are the same as at 15 epochs. We did not run 300 epochs, and we cannot say what happens at 300. This is in the Limitations.
W2. Only one model and one dataset. We added YOLOv8n and YOLOv8m (15 epochs, one run each, BCE / Sin-SMFL / Focal): 0.3422 / 0.3305 / 0.3321 and 0.4560 / 0.4471 / 0.4472. BCE is best for both. We also added LVIS (see Q2). All models are still YOLOv8.
Q1. Ablation at 300 epochs. Not done. See W1. We have 50-epoch results only.
Q2. More datasets: LVIS, V3Det, OpenImages. We added LVIS (from scratch, 50 epochs, official LVIS evaluator, one run; Section 5.8, Table 6):
Q3. More models: DETR with Hungarian matching, RPN-based detectors, other kinds of imbalance. Not done. We only tested YOLOv8 (three sizes) and class imbalance. The paper makes no claim for other detectors. This is in the Limitations.
Q4. Cost of the sine function: slow, or stable in training? We did not measure it, so we give no number. This is in the Limitations.
Q5. Does the loss work for classification: ImageNet-1K, ImageNet-LT, iNaturalist18, Places-LT? Not tested. This is in the Limitations.
Q6. More metrics than COCO mAP, and visual or qualitative explanations. Metrics: done. We added per-class AP, AP by class frequency group, the official LVIS AP, APr, APc, APf, and intervals over classes for the rare-class comparison (Sections 5.6 and 5.8). Qualitative figures and the full protocol of Dave et al.: not done. Strength: sine favours medium-difficulty examples. We tested the strength of the modulation (15 epochs, gain 2). Weaker modulation scores higher: sine n = 1 / 2 / 3 gives 0.4100 / 0.4048 / 0.3959. Focal gamma 1 / 2 / 3 gives 0.4076 / 0.4042 / 0.3951. So we do not see an accuracy gain from the sine shape.
On seeds: the training code fixes its data-loader seed, so runs with different nominal seeds are not independent. We now report one run per configuration with a run-to-run difference of up to 0.004, and no standard deviations or p-values (Section 4).
Do these answers cover your concerns?