acceptodds
Under review as a conference paper at ICLR 2027

Calibration Is Grouping: VR-SAG with Intra-Group Variance Control and Logit-Cluster Evaluation

Abstract

Accurate click-through and conversion-rate estimates are pivotal for bid optimization in large-scale advertising, yet modern deep CTR/CVR models are often miscalibrated. Classical global calibrators (Platt scaling, isotonic regression) and feature-based binning struggle to capture latent user-item heterogeneity. We approach calibration through the lens of latent, calibration-aware groupings and propose Variance-Reduced Semantic-Aware Grouping (VR-SAG), a lightweight post-hoc layer over a frozen backbone that (i) learns calibration-aware groups in embedding space, (ii) fits per-group temperature+bias calibrators, and (iii) explicitly penalizes intra-group variance to tighten probability spreads. A group-wise decomposition of the Brier score guides our design. It separates within-group variance from score–label covariance and motivates a grouping regularizer. To decouple evaluation from training, we introduce Logit-Cluster Calibration Error (LCCE), an unsupervised fixed-partition metric obtained via -means in logit space; LCCE aligns with the reliability term of proper scores while avoiding pitfalls of trainable grouping heads used as metrics. Across offline CTR logs and AdAuction (an ad-auction dataset with simulator-defined oracle CTRs), VR-SAG improves the reported calibration metrics over the corresponding SAG variants. Its additional computation scales linearly with the number of grouping heads and groups.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
calibration, click-through rate prediction, post-hoc calibration, uncertainty quantification, semantic grouping, clustering, proper scoring rules, advertising systems

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_APM9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a lightweight method based on Semantic-Aware Grouping (SAG) for CTR/CVR calibration. In addition, it introduces Logit-Cluster Calibration Error (LCCE) to decouple evaluation from training and align with the reliability term of proper scoring rules. For evaluation, the paper conducts experiments on three real-world datasets, and the proposed method achieves the best performance.

Critical strengths
  1. This paper proposes VR-SAG, a lightweight post-hoc layer applied on top of a frozen backbone, which is easy to deploy.
  2. Comprehensive experiments are conducted to validate the effectiveness of VR-SAG, showing that VR-SAG outperforms numerous baselines across various calibration metrics.
Critical weaknesses
  1. There is substantial content overlap between Lines 54–62 and Lines 63–71.
  2. The paper claims the method can be applied to CVR prediction, yet it neither describes how the method is adapted for this task nor reports its CVR prediction performance.
  3. The baselines are somewhat outdated. With the exception of SAG, all other baselines adopted in this work were proposed prior to 2017.
Questions for authors
  1. How is the stability of LCCE ensured? As LCCE uses k-means for sample partitioning, and the appropriate K varies across different backbones, how can the authors verify that the selected K is well-suited for the backbone adopted in this work?
Reviewer_5GwyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper propose VR-SAG to solve CTR/CVR calibration problem in advertising system. Existing global calibration method ignore latent user-item heterogeneity. Paper use Brier-loss decomposition to show that intra-group variance is very important. So they add variance penalty to SAG model. Also, paper propose a new metric called LCCE. Because trainable group head can cheat traditional ECE metric, LCCE use fixed K-means cluster in logit space to provide fair evaluation. Experiments show VR-SAG can beat many baseline.

Critical strengths
  1. Decompose Brier score to separate variance and covariance is very smart. It give strong theoretical motivation to use variance penalty in the loss function. This make the method have good math foundation.

  2. Trainable group network can easy cheat traditional ECE metric by manipulate bin boundary. Use fixed K-means cluster to evaluate calibration error is a very objective way. I think this metric will be very useful for recommend system community.

Critical weaknesses
  1. In real-world advertising system, online inference latency is extremely strict. VR-SAG add soft grouping calculation and exponential operations for each group. But the paper only report offline training time. Without online inference latency report, we can not know if this model can be really deployed in real time system.

  2. In section 2.3, paper admit that delta can be negative. This mean reduce within-group variance do not strictly guarantee the decrease of Brier loss. So the variance penalty is just a engineering trick, not a strict mathematical upper bound. The paper over-claim the theoretical power of variance reduction.

  3. The paper heavily rely on AdAuction dataset to evaluate oracle CTR. Because this dataset is generated by internal simulator and is not public, so it is impossible for other researcher to check the data quality or reproduce the result.

Questions for authors
  1. Can you report the exact online inference latency?
  2. Because delta can be negative, did you ever observe that Brier loss actually increase when you force the model to minimize intra-group variance in your experiment?
Reviewer_gVyKOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose VR-SAG, an improved variant of SAG for CTR calibration. Specifically, building on top of the SAG architecture, VR-SAG adds intra-group variance regularization to enforce more coherent partitions. In addition, this paper further introduces LCCE, a fixed logit-space grouping metric to enable fairer evaluation. Finally, the author contribute a dataset AdAuction for CTR calibration task evaluation and conduct experiments to verify VR-SAG's performance.

Critical strengths

This paper conducts an extensive amount of ablation studies and utilizes three datasets for experiments.

Critical weaknesses

1.Limited Innovation: the main contribution of this paper is to add a within-group variance regularizer on top of the previously proposed SAG algorithm. Moreover, the performance of VR-SAG relies on hyper-parameter tuning rather than yielding stable improvements. This limits the overall novelty of this paper. 2.Performance gain is unclear: VR-SAG achieves only marginal improvements over traditional methods. Even on the AdAuction dataset, histogram binning performs better on the ECE metric, indicating that the performance of VR-SAG is limited. In addition, the results are not tested with statistical significance. 3.The presentation needs clear revision: some of terms used in the formula is not clearly defined, and some sentences are not complete. For example, in line 93, the expression "a proper scoring rule" has nothing to do with the preceding and following text; in Table 1, "The RMSE label is provisional: the archived routine takes a square root, but its correspondence to the final table requires verification", the experimental results seem to be unverified.

Questions for authors

See the weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.