acceptodds
Under review as a conference paper at ICLR 2027

Theoretically Principled Balanced Deepfake Detection

Abstract

Driven by advances in deep neural networks (DNNs), modern generative AI has enabled highly realistic media synthesis while also giving rise to deepfakes that threaten social trust, making reliable deepfake detection essential. Although many DNN-based deepfake detectors have been developed, they overemphasize overall performance but largely ignore class imbalance, which can severely degrade performance and induce asymmetric accuracy between classes as the imbalance increases. Existing imbalance-aware deepfake detection methods mainly address class imbalance but overlook domain-level imbalance and sample hardness, leading to detectors fail to reliably detect fakes from specific domains in real-world deployments. In this work, we study a new bi-level imbalanced learning problem and propose the first balanced deepfake detection approach that jointly addresses class-level and domain-level imbalance by designing a bi-level balanced loss with a general guarantee of -consistency and a Bayes consistency result established for a special case. Extensive experiments demonstrate that our method consistently outperforms existing losses and can be used in various DNN-based deepfake detectors to improve balanced performance and further improve overall performance.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Deepfake Detection, Imbalanced Learning, Domain Generalization, Deep Neural Networks, Robust Learning, Consistency Theory, Generative AI

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wbsEOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper focuses on class-level and domain-level imbalance in deepfake detection. Authors propose a Bi-level Balanced (BB) Loss that combines class-aware and domain-aware logit adjustment with a top-k loss aggregation strategy to emphasize difficult samples. Experiments on five deepfake detection datasets show improvements in classification accuracy and balanced performance under different imbalance settings.

Critical strengths

1 Considering domain-level imbalance in addition to conventional real/fake class imbalance is well motivated.

2 The proposed loss can be easily used on existing deepfake detection frameworks.

3 The evaluation covers multiple datasets, model architectures, imbalance ratios, and robustness settings is soild and demonstrates improvements of proposed method. But implementation details need further explanation.

Critical weaknesses

Major comments:

1 The experiments primarily vary the real-to-fake ratio without independently controlling domain-level imbalance. I suggest evaluating different fake-domain proportions while keeping the class ratio fixed.

2 The ablation study in Table 6 is useful, but it is conducted on the original FF++ training distribution, where performance is already close to saturation. I suggest repeating the component ablations under severe imbalance setting.

Minor comments:

1 Table 6 reports percentages, whereas most other tables use values between 0 and 1. Please use a consistent format.

2 Implementation details like per-domain sample counts, the procedure used to construct imbalanced datasets, and random seeds would be good for reproducibility.

Questions for authors

1 Theorem 1 claims an equivalence between Eq. (5) and proposed domain-level surrogate. However, the original domain-balanced objective penalizes binary classification errors, whereas the proposed surrogate also penalizes incorrect predictions among different fake domains. Can authors explain how the claimed equivalence holds?

3 comments

WEAKNESSES

MAJOR COMMENTS

W1

In our original experiments, changing the class ratio also changes the relative proportions of the real and fake domains, and AI-Face is naturally imbalanced across generator families. Per your request, we conduct an additional experiment below. To isolate domain-level imbalance, we fix the class ratio at 1:1 and vary the share of one fake domain (Deepfakes, selected uniformly at random before running any experiments), which makes up only 1%, 5%, or 10% of the fake samples, with the remaining fakes split equally among the other four domains. All methods are trained on identical data.

Rare shareMethodDeepfakes accDegreeD\text{Degree}_DDegreeC\text{Degree}_CACC
1%BCE0.84830.86940.96000.9421
Base10.87700.88970.98310.9560
Base20.84690.85940.96950.9502
GLA0.87700.89080.99080.9554
BB0.90420.91640.99660.9613
5%BCE0.95960.95790.99260.9601
Base10.88730.89530.95640.9465
Base20.96320.95320.99290.9650
GLA0.91770.87680.94090.9380
BB0.96610.96610.99800.9719
10%BCE0.97350.92780.98210.9669
Base10.96240.92180.93720.9499
Base20.94680.90360.96080.9551
GLA0.96770.94190.99250.9663
BB0.97910.96420.99550.9687

Overall, under these extreme settings, our approach still performs best.

W2

The original FF++ training distribution is already imbalanced (17,218 real : 84,346 fake), so Table 6 was not run on balanced data. To test the components under more severe imbalance, we added a 1:10 real:fake setting.

MethodACCDegreeC\text{Degree}_CDegreeD\text{Degree}_D
Variant 194.74 ± 2.1495.98 ± 2.3090.59 ± 4.18
Variant 295.07 ± 1.3196.66 ± 1.8190.03 ± 4.05
Variant 394.60 ± 1.0395.97 ± 1.6889.82 ± 2.24
Variant 496.69 ± 1.6397.68 ± 0.0889.82 ± 7.67
Variant 596.76 ± 0.6798.13 ± 1.4594.83 ± 0.59
BB97.47 ± 0.3299.08 ± 0.4696.53 ± 0.49

The full objective is best on all three metrics, and the largest gain is on DegreeD\text{Degree}_D (Variant 5 → BB: 94.83 → 96.53, with smaller standard deviations).

MINOR COMMENTS

C1

Thank you for pointing this out. We will convert Table 6 to the same [0, 1] scale used in the other tables, so all results are reported in a consistent format. We will include this in the final manuscript.

C2

Thank you for this suggestion. We will add to Appendix D the per-domain sample counts for every training set, the procedure used to construct the imbalanced datasets (random subsampling of the real and fake training images to reach each target real:fake ratio, with a fixed sampling seed), and the random seeds used for each run. We will also release the scripts that construct these datasets. We will include this in the final manuscript.

QUESTIONS

Q1

Thank you for this question. Theorem 1 concerns the relationship between two surrogate formulations, not pointwise equality with the original objective in Eq. (3). Specifically, circle ① is replaced by the class surrogate in Eq. (4). For circle ②, we introduce the indicator-ratio decomposition and replace 1{g^≠g}/p(g)\mathbf{1}\{\hat{g}\ne g\}/p(g) with a domain-classification surrogate, yielding Eq. (5). Thus, Eq. (5) is already a surrogate. The argument for the claimed equivalence with the final domain surrogate is presented in Appendix C.1 through four cases, according to whether the true and predicted domains are real or fake. We will clarify these separate steps in the manuscript.

Thank you to the authors for their detailed responses and additional experiments. These have clarified some of my concerns. I will maintain my rating.

Thank you for your positive feedback and continued support of our work. We sincerely appreciate your time and effort in reviewing our responses and additional experiments.

Reviewer_y56mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies deepfake detection under simultaneous class-level and domain-level imbalance. It proposes a Bi-level Balanced (BB) loss that combines class- and domain-dependent logit adjustments with top-k loss aggregation to emphasize hard examples. The paper presents consistency analyses and evaluates the approach across multiple deepfake datasets, imbalance settings, detector architectures, and competing loss functions.

Critical strengths

A practically relevant problem setting. Balancing real and fake samples alone does not ensure reliable detection across different manipulation methods or generators. Explicitly considering underrepresented fake domains addresses an important limitation of aggregate detection performance.

Broad evidence of applicability. The experiments cover multiple imbalance ratios, spatial- and frequency-based detectors, and modern backbones. Comparisons with general imbalance-learning and DRO methods, together with component ablations, provide encouraging evidence that the proposed objective can improve class- and domain-level performance balance.

Critical weaknesses

The central theoretical guarantees are not sufficiently substantiated. The proposed domain surrogate distinguishes individual fake domains, whereas the stated target risk penalizes real/fake classification errors. These objectives are related but are not equivalent merely because each domain belongs to a particular class. The consistency arguments also require stronger justification when combining the class and domain losses and extending the analysis to top-k aggregation. Although the paper acknowledges that its Bayes-consistency result applies only to a special case, the broader guarantee for the practical objective remains unconvincing. Furthermore, the formulation uses domain-conditioned scores, but it is unclear how these produce binary predictions without domain labels at inference. These issues concern the foundation of the claimed contribution.

The evaluation does not sufficiently separate improved discrimination from changes in the operating point. Most conclusions rely on accuracy and worst-to-best accuracy ratios at a fixed threshold, while AUC improvements are inconsistent and sometimes negative, including in cross-dataset evaluation. The additional threshold sweep is useful, but applying the same thresholds to differently calibrated models does not establish superiority over baselines with independently validated thresholds. Moreover, the worst-to-best ratio measures performance parity rather than absolute detection quality. Stronger evidence using validation-selected operating points and absolute worst-domain performance is needed to support the claims of improved detection reliability.

Questions for authors
  1. Can you clarify why the domain-prediction surrogate is consistent with the intended domain-balanced binary classification risk, and which guarantees apply to the combined objective with top-k aggregation?
  2. How are the class and domain scores implemented, and how is the final real/fake prediction obtained when domain labels are unavailable at test time?
  3. Do the improvements persist when each baseline uses a threshold selected on a separate validation set? Please also report absolute worst-domain accuracy and detection rates at matched false-positive rates.
  4. How are the random training/test splits and hyperparameter selection performed? In particular, are related frames and their source/manipulated videos kept in the same split, and is model selection conducted independently of the test data?
9 comments

WEAKNESSES

W1

Thank you for this question.

Surrogate construction. Term ① in Eq. (3) is replaced by the class surrogate in Eq. (4); for term ②, we use the indicator-ratio decomposition and replace 1{g^≠g}/p(g)\mathbf{1}\{\hat g \neq g\}/p(g) with a domain-classification surrogate, giving Eq. (5). Theorem 1 relates Eq. (5) to the final domain surrogate ΨG\Psi_G, not pointwise to the original binary-error objective (Appendix C.1).

Combined objective and top-kk. We agree that the link to the binary risk, the consistency of ΨY+βΨG\Psi_Y + \beta\Psi_G, and the top-kk extension need stronger justification, and will add a joint conditional-risk analysis with explicit assumptions. Theorem 2 gives Bayes consistency only for α=1\alpha = 1; for 0<α<10 < \alpha < 1, training minimizes an upper-tail (CVaR) risk, which we will treat as a separate target from the expected risk.

Inference without domain labels. As stated in the Introduction (Line 70-71: "We assume domain labels are available during training but not required at test time…"), the domain label gg is used only during training to improve domain-level balanced learning. Our framework has no additional head for predicting gg; there is only a class-level prediction head that predicts yy (Section 3.2, Lines 205–210).

W2

Per your request, we conduct the following experiments:

(1) Independently validated operating points. For each method mm, we select a single global threshold on the validation set that maximizes the worst-domain balanced accuracy, and apply it unchanged to test:

τworst(m)=arg⁡max⁡τ min⁡d∈Dval BAval(m,d,τ),BAval(m,d,τ)=12(TNRval(m,real,τ)+TPRval(m,d,τ)).\tau^{\text{worst}}(m) = \underset{\tau}{\arg\max}\ \underset{d \in \mathcal{D}^{\text{val}}}{\min}\ \mathrm{BA}^{\text{val}}(m, d, \tau), \qquad \mathrm{BA}^{\text{val}}(m, d, \tau) = \frac{1}{2}\Big(\mathrm{TNR}^{\text{val}}(m, \text{real}, \tau) + \mathrm{TPR}^{\text{val}}(m, d, \tau)\Big).

where Dval\mathcal{D}^{\text{val}} is the set of fake domains; each domain dd is paired with the real validation images, so BAval(m,d,τ)\mathrm{BA}^{\text{val}}(m, d, \tau) measures how well method mm separates real images from domain dd at threshold τ\tau.

MethodThreshold (10:1)ACCWorstDegreeC\text{Degree}_CDegreeD\text{Degree}_DThreshold (100:1)ACCWorstDegreeC\text{Degree}_CDegreeD\text{Degree}_D
BCE0.07220.87090.78400.88270.84510.00640.73980.59370.77200.6749
Base10.05700.86750.75040.84230.78500.02590.46380.27980.53280.3683
Base20.31430.86560.80370.95710.87310.02610.74120.55680.71570.6638
BB0.11240.88300.82640.95750.89040.23480.74240.61480.80060.6725

With validated thresholds, BB has the highest worst-domain accuracy at both ratios (10:1: 0.8264 vs. 0.8037; 100:1: 0.6148 vs. 0.5937).

(2) Absolute worst-domain accuracy (threshold 0.5). Best = best-domain accuracy.

RatioMethodACCWorstBestDegreeC\text{Degree}_CDegreeD\text{Degree}_D
10:1BCE0.84130.71850.94040.87350.7641
Base10.81810.71850.93970.84470.7646
Base20.83150.73960.92890.87420.7962
BB0.88280.79490.94680.88290.8396
100:1BCE0.46560.06250.94300.39260.0662
Base10.23470.03420.99140.08430.0345
Base20.33430.07890.97040.21360.0813
BB0.52600.34650.94240.46990.3677

It is clear that BB has the highest ACC, Degree_C, Degree_D, and worst-domain accuracy at both ratios. 

(3) Detection rate at matched false-positive rates (All = all fakes; Worst = hardest domain). Thresholds are set so each method flags the same share of real images.

Method10:1 All@1%10:1 Worst@1%10:1 All@5%10:1 Worst@5%100:1 All@1%100:1 Worst@1%100:1 All@5%100:1 Worst@5%
BCE0.65750.53030.80890.69720.10180.00720.41780.1890
Base10.57570.47780.77760.69960.06100.01250.22090.1121
Base20.56830.46290.77150.67240.13060.02270.37870.1693
BB0.68450.56600.81120.71350.21230.04600.45110.2751

At matched FPR, BB detects the most fakes overall and on the hardest domain in all eight settings.

QUESTIONS

Q1

Eq. (5) is already a surrogate, and Theorem 1 relates it to the final ΨG\Psi_G; it does not claim pointwise equality with the binary-error objective. Bayes consistency holds only for α=1\alpha = 1. For 0<α<10 < \alpha < 1, training minimizes an upper-tail (CVaR) risk, and we will add a joint conditional-risk analysis of ΨY+βΨG\Psi_Y + \beta\Psi_G (see W1).

Q2

There is a single class-level (real/fake) head with no domain-prediction head. ΨY\Psi_Y and ΨG\Psi_G are both computed from its output, and at inference y^(x)\hat y(x) is obtained from this output alone, so no domain labels are needed (see "Inference without domain labels" in W1).

Q3

Yes. The improvement persists when each baseline uses a threshold selected on a separate validation set. See our response for W2.

Q4

  1. We follow the original training/test splits in FF++ and AI-Face. The hyperparameter selection depends on the test set.

  2. For the experiments in our submission, the related frames and their source/manipulated videos are not kept in the same split. For the new experiments that we added during this response phase for your questions, they are kept in the same split.

  3. For the experiments in our submission, the model selection is based on the test data. Per your request, we also include additional experiments with model selection based on the validation set (independently of the test data) in our response for W2.

Thank you for the clarification and additional results. My inference concern is resolved, but the theoretical justification and evaluation protocol remain insufficiently established. I therefore maintain my current assessment.

Reviewer_vpEcOpenReview ↗
Rating
3(+1)
Confidence
2(−2)
Informativeness
2
Presentation
2(−1)
Soundness
1
Read review
Paper summary

The paper proposes a Bi-level Balanced loss for deepfake detection to address both class-level imbalance between real and fake samples and domain-level imbalance across different generators. The method combines class-balanced and domain-balanced losses. A main advantage is its plug-and-play design, which allows the loss to replace the conventional BCE loss without modifying the underlying detector architecture. The method is evaluated on several face deepfake datasets, multiple existing detection models, different imbalance ratios, and cross-dataset settings.

Critical strengths
  • Plug-and-play integration with existing detectors: This is the paper's strongest practical contribution. The proposed method modifies the training objective rather than introducing a new detector architecture. The authors demonstrate integration with existing detectors spanning spatial-based and frequency-based approaches.

  • Theoretical analysis: The paper goes beyond empirical loss design by studying Bayes consistency and H-consistency. These results provide a formal foundation for the proposed objective, although their practical implications depend on the assumptions and the particular hyperparameter settings.

Critical weaknesses
  • Insufficient evaluation on recent models and different types of fake data: The experiments show promising improvements across several detectors, but the evaluation is still largely focused on face deepfakes. Most experiments use Xception and EfficientNet-B4, while the additional results on ConvNeXt-Base and ViT-B/16 are not sufficient to demonstrate the effectiveness of the method on a broader range of models. It would be helpful to evaluate the proposed loss with recent detection methods and investigate whether the improvements hold for different types of fake data, including non-face images and outputs from newer generative models (FLUX, VAR). The authors should also examine performance on unseen generators to better support their claims about generalization.

  • Hyperparameter tuning raises concerns about practical applicability and fairness: The proposed loss involves several hyperparameters, including qyq_y, qgq_g, β\beta, and α\alpha, and the appendix shows that different settings are used for different imbalance ratios and detector architectures. This makes it unclear whether the method can work reliably with a fixed configuration or requires tuning for each experimental setting. Such tuning limits its practical value as a plug-and-play solution. In addition, the paper does not clearly establish whether all baselines receive comparable hyperparameter-tuning budgets. The authors should evaluate the method using fixed hyperparameters across different datasets and backbones, and clarify the tuning protocols to ensure a fair comparison.

Questions for authors
  • Can the authors use one fixed configuration of qyq_y, qgq_g, β\beta, and α\alpha across all datasets, imbalance ratios, and backbones? How much performance is lost compared with per-setting tuning?

  • Can the authors evaluate the proposed loss on more contemporary deepfake detectors and suitable DiT- or VAR-based detection pipelines? Does the loss still improve performance when the underlying detector is substantially stronger than the older baselines? How does the method perform on non-face synthetic images?

  • How does the method behave when domain labels are noisy, incomplete, or unavailable for some training samples?


The rebuttal has addressed almost all of my concerns. But the plug-and-play applicability would be even more practical if the authors could demonstrate that a single configuration works for every dataset and backbone.

3 comments

WEAKNESSES

W1

We thank the reviewer. We added two recent detectors, UnivFD and Effort (ICML'25), trained on non-face data (ARForensics: ImageNet reals + LlamaGen fakes) at 100:1, with the same data and settings for every method, and only the loss changed. We test on ARForensics (VAR, Infinity, other autoregressive generators) and EvalGEN (FLUX, GoT, OmniGen, NOVA); training uses LlamaGen only, so every other generator is unseen.

UnivFD

LossACCAUCF1DegreeC\text{Degree}_CDegreeD\text{Degree}_D
BCE0.40670.99140.26180.15060.0305
Base10.30150.98990.00000.00000.0000
Base20.68720.99530.71180.55380.2813
BB0.96270.99580.97390.88560.8835

Effort

LossACCAUCF1DegreeC\text{Degree}_CDegreeD\text{Degree}_D
BCE0.49830.99770.43960.28170.0297
Base10.77980.99890.81290.68480.1488
Base20.62260.99850.62990.45980.1528
BB0.86400.99030.89220.80560.2615

Overall, our BB loss consistently outperforms the competing approaches when evaluated using recent detectors on non-face images.

W2

We thank the reviewer for this point. We use the following results to answer your questions:

  1. The submitted configurations are already shared across datasets and largely across backbones. Table 7 gives one configuration per backbone and ratio, used for both FF++ and AI-Face. At 100:1, Xception and EfficientNet-B4 use identical values (q_y, q_g, β, α) = (0.3, 0.5, 0.2, 0.9); at 10:1, they share (q_y, q_g, β) = (0.5, 0.4, 0.1) and differ only in α (0.5 vs. 0.3).
  2. We reuse the 10:1 values (q_y, q_g, β) = (0.5, 0.4, 0.1) with α = 0.2 for both backbones at 100:1 on FF++, with no per-backbone tuning:
BackboneMethodACCDegreeC\text{Degree}_CDegreeD\text{Degree}_D
XceptionBest baseline (BCE, Table 1)0.44560.33580.1172
BB (consistent configuration)0.49150.42090.1790
BB (tuned, Table 1)0.55260.47540.2754
EfficientNet-B4Best baseline (Base2, Table 1)0.21410.05230.0177
BB (consistent configuration)0.28790.14880.1019
BB (tuned, Table 1)0.33860.21580.1146

With the consistent configuration, BB still outperforms every baseline on all three metrics for both backbones. Per-setting tuning adds further gains (Table 1), so the consistent configuration is a strong default and tuning is an optional refinement.

  1. The submission used no separate validation set: for all methods, hyperparameters were chosen by one-at-a-time sweeps on the test partition. We will state this protocol in Appendix D and add the fixed-configuration results.

QUESTIONS

Q1

With one configuration transferred from 10:1 to 100:1 ((q_y, q_g, β) = (0.5, 0.4, 0.1), α = 0.2), Xception reaches ACC 0.4915 / DegreeC\text{Degree}_C 0.4209 / DegreeD\text{Degree}_D 0.1790 versus 0.5526 / 0.4754 / 0.2754 with per-setting tuning (a loss of 6.1, 5.5, and 9.6 points), and still beats the best baseline (BCE: 0.4456 / 0.3358 / 0.1172) on all three metrics. This tests transfer across ratios on FF++, not one configuration for every dataset and backbone; see W2.

Q2

Per your request, we evaluate two recent detectors (UnivFD and Effort) on non-face images, training on ARForensics (ImageNet reals + LlamaGen fakes) at 100:1 and testing on newer generators including VAR, Infinity, FLUX, GoT, OmniGen, and NOVA, with all generators except LlamaGen unseen at training. See W1 for details. In general, our loss can still improve performance with stronger deepfake detectors and work on non-face images.

Q3

To test this, we trained BB at 100:1 (Xception, FF++) under three corruptions of the training domain labels: (i) 20% of each fake domain relabeled as another fake domain, (ii) 20% of Deepfakes relabeled as FaceSwap, and (iii) 20% of fake samples with no domain label (plain cross-entropy). Real/fake labels are unchanged.

MethodDomain labelsACCDegreeC\text{Degree}_CDegreeD\text{Degree}_D
BCEnot used0.44560.33580.1172
Base1not used0.34130.21280.0413
Base2not used0.33610.20050.0315
BB20% flipped (all fake domains)0.47410.36980.2153
BB20% flipped (one domain)0.46020.35690.1801
BB20% missing0.53080.46240.2058
BBclean0.55260.47540.2754

Missing labels cost 2.2 points of ACC and 1.3 of DegreeC\text{Degree}_C, but 7.0 of DegreeD\text{Degree}_D; flipped labels cost 7.9-9.2 (ACC), 10.6-11.9 (DegreeC\text{Degree}_C), and 6.0-9.5 (DegreeD\text{Degree}_D). In every corrupted setting, BB still outperforms BCE, Base1, and Base2 on all three metrics, so it degrades gracefully but benefits from accurate domain labels.

Thank you for your responses. They have addressed almost all of my concerns. But the plug-and-play applicability would be even more practical if the authors could demonstrate that a single configuration works for every dataset and backbone. I will increase my score.

We sincerely thank you for the positive feedback, constructive suggestions, and the increased rating. We are pleased that our responses have addressed almost all of your concerns.

Regarding the plug-and-play applicability under a single hyperparameter configuration, we fully agree that using a unified configuration across different datasets and backbones would further improve the practical applicability of our approach.

Setup. To further address this suggestion, we conducted additional experiments across four datasets and three backbone architectures (Xception, EfficientNet-B4, and ConvNeXt-Base), using a single fixed BB configuration, (qy,qg,β,α)=(0.3,0.2,0.6,0.1)(q_y, q_g, \beta, \alpha) = (0.3, 0.2, 0.6, 0.1), for every dataset and backbone. All models are trained only on FF++ with its original 17,218:84,346 real:fake ratio, and the baselines (BCE, Base1, Base2) are trained on the same data. We evaluate them on the FF++ test split and, as unseen datasets, on the DFD, DFDC, and Celeb-DF test splits.

Results.

BackboneMethodDFD ACCDFD DegreeC\text{Degree}_CDFD AUCDFDC ACCDFDC DegreeC\text{Degree}_CDFDC AUCCeleb-DF ACCCeleb-DF DegreeC\text{Degree}_CCeleb-DF AUCFF++ ACCFF++ DegreeC\text{Degree}_CFF++ DegreeD\text{Degree}_DFF++ AUC
XceptionBCE0.76860.40810.74300.57700.72270.60790.73690.45350.63750.98130.94680.94190.9969
Base10.75760.56530.77260.55060.50930.58680.74440.53470.69030.95750.82030.81780.9891
Base20.78670.59580.78420.58460.81490.61600.73100.54090.66630.98310.94470.94080.9977
BB0.78500.68200.80410.57860.71760.60650.74020.58900.69870.98610.98410.97770.9975
EfficientNet-B4BCE0.79850.46280.80050.59330.92850.63860.86170.22020.74700.97990.94670.93890.9965
Base10.79280.45370.80480.58000.69420.61650.82890.33030.72740.97610.94470.93520.9960
Base20.81080.21670.81740.56380.53730.61760.83520.27330.69630.97380.87170.86920.9953
BB0.81470.57420.81650.59920.97050.64410.84220.34970.71970.98240.97600.96740.9959
ConvNeXt-BaseBCE0.79850.41520.81300.58890.65500.63010.82850.53610.78260.98270.93960.93510.9976
Base10.80230.50740.82980.59760.77430.64320.83630.55290.82010.98020.95030.94330.9973
Base20.79180.49830.79750.59640.70100.63640.83930.48060.79840.98420.92240.92080.9985
BB0.81120.68530.84460.60410.83880.64760.84230.57470.82220.98640.98370.97760.9981

Summary. With this fixed configuration, our method outperforms the compared methods across 4 datasets and 3 backbones.

Automatic hyperparameter selection. We would also like to emphasize that identifying a single hyperparameter configuration that performs optimally across all datasets and architectures is inherently challenging due to differences in data distributions, imbalance ratios, and model characteristics. Nevertheless, this challenge can potentially be alleviated through existing automatic hyperparameter optimization techniques. For example, Bayesian optimization (Snoek et al., NeurIPS 2012), gradient-based bilevel optimization (Franceschi et al., ICML 2018), and implicit differentiation-based hyperparameter optimization (Lorraine et al., AISTATS 2020) provide promising strategies for automatically determining suitable hyperparameters. Integrating such techniques with our proposed BB loss represents an interesting future direction.

Again, we sincerely appreciate your constructive feedback and positive reassessment of our work.

References

  1. Snoek, Jasper, Hugo Larochelle, and Ryan Adams. "Practical bayesian optimization of machine learning algorithms." Advances in neural information processing systems 25 (2012).
  2. Franceschi, Luca, et al. "Bilevel programming for hyperparameter optimization and meta-learning." International conference on machine learning. PMLR, 2018.
  3. Lorraine, Jonathan, Paul Vicol, and David Duvenaud. "Optimizing millions of hyperparameters by implicit differentiation." International conference on artificial intelligence and statistics. PMLR, 2021.

Comments on the paper

1 comment

Dear reviewers,

This is a kind reminder that the discussion with the authors ends on 18th Nov. Please take a moment to leave a comment to acknowledge the authors’ responses and engage in the discussion if you have further questions, particularly on points that may affect your assessment.

Thank you!

Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.