acceptodds
Under review as a conference paper at ICLR 2027

Confidence-Aware Pareto-Evolving Auto-bidding via Uncertainty-Regularized Counterfactual Regret

Abstract

Constrained auto-bidding requires maximizing marketing value while strictly satisfying efficiency constraints such as Target Cost-Per-Action (CPA). Recent Decision-Transformer-based methods introduce a dual-stream constraint context and a counterfactual regret loss to actively push the policy toward the Pareto frontier. However, we identify two fundamental gaps that cap their performance: (i) the Pareto frontier is constructed once from historical data and remains static, anchoring the learning target at the empirical rather than the theoretical optimum; and (ii) the outcome predictor used for counterfactual evaluation is a deterministic point estimator, so out-of-distribution counterfactuals can hallucinate high utility and mislead policy updates. We propose CoPE-Bid, a Confidence-aware Pareto-Evolving auto-bidding framework built on two synergistic mechanisms: (i) Uncertainty-Regularized Counterfactual Regret (UR-CRO) replaces the point-estimate predictor with a lightweight ensemble and evaluates counterfactuals via a lower-confidence-bound utility, so predictor uncertainty automatically discounts hallucinated regret; and (ii) Frontier-Anchored Self-Improvement (FASI) periodically performs confidence-gated imagination rollouts from the current policy and admits only high-confidence, Pareto-improving trajectories into the frontier, letting the sampling anchor co-evolve with the policy. The two modules are coupled through a shared uncertainty signal—UR-CRO tells FASI what to trust, and FASI tells UR-CRO what to aim for. Extensive experiments on AuctionNet, its sparse variant, and online A/B tests demonstrate that CoPE-Bid consistently outperforms state-of-the-art baselines in both value acquisition and constraint satisfaction, with particularly large gains in the noise-heavy sparse regime.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_wnAaOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends PRO-Bid’s CDPR+CRO backbone with pessimistic counterfactual weighting (UR-CRO) and virtual frontier updates (FASI) for offline constrained bidding. It reports improvements on AuctionNet and in a seven-day online A/B test.

Critical strengths

The problem is practically important, and results across budgets and module ablations support the system’s effectiveness. FASI offers some design novelty by using virtual trajectories as sampling references for real logs, without using them as action supervision.

Critical weaknesses
  1. Limited methodological novelty. UR-CRO adds established pessimistic estimation to PRO-Bid’s CRO, while value-based action weighting has precedents such as GAVE. Its distinct benefits are insufficiently isolated. FASI offers an interesting design, but its evaluation mainly establishes improvements within PRO-Bid.

  2. Incomplete specification of FASI rollouts. The multi-step rollout procedure lacks complete definitions of state transitions and per-step feedback, limiting independent implementation.

  3. Insufficient validation of reliability claims. The utility reductions in Tables 4 and 8 demonstrate that the penalty operates, but do not independently justify the downweighting. Total uncertainty is not equivalent to error in the predicted mean, and the reliability of admitted virtual frontier points remains insufficiently validated.

  4. Unclear contribution of virtual frontier construction. Table 2 supports FASI’s effectiveness, but does not isolate its frontier design from generic dynamic resampling. Treating the static sampling frontier as a performance ceiling also requires justification.

  5. Ambiguous interpretation of ablation results. Some gains reflect improved compliance rather than increased value acquisition, and the configurations of related ablations are unclear.

  6. Missing reproducibility details. Data splitting, distance and uncertainty normalization, and the set used for frontier updates are insufficiently specified.

  7. Insufficient online evaluation details. The reported protocol and constraint definitions do not fully establish the reliability and scope of the online gains.

Questions for authors
  1. How does FASI generate next states and per-step feedback, update RTG/CTG, and determine rollout termination?

  2. Can independent evaluations establish how uncertainty relates to prediction error and constraint violations, and whether admitted virtual points genuinely improve the frontier?

  3. Can FASI be compared with a matched control that resamples logged trajectories using current predicted utility, to isolate the contribution of virtual frontier construction?

Reviewer_cRm3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a constrained auto-bidding problem in online advertising, where the objective is to maximize cumulative value subject to an overall budget constraint and efficiency constraints such as a target Cost Per Action (CPA). The bidder chooses a time-dependent bidding coefficient λt\lambda_t, which determines the bid for each impression through bi=λtvib_i=\lambda_t v_i. Existing approaches formulate this problem as autoregressive sequence modeling and apply Decision Transformers. In particular, CDPR incorporates both return-to-go and cost-to-go into the transformer and uses an empirical Pareto frontier to guide training. CRO further improves the policy by sampling counterfactual actions around the current policy, using a learned outcome predictor to estimate their rewards and costs, and performing regret-weighted regression toward actions with higher predicted utility.

However, the effectiveness of CRO heavily depends on the reliability of its outcome predictor, which may produce inaccurate estimates for out-of-distribution actions. This paper addresses this limitation through two extensions. First, UR-CRO replaces the point predictor with a Gaussian ensemble and adopts a pessimistic utility estimate, penalizing uncertain reward predictions and accounting conservatively for uncertain costs, in a manner similar to the LCB/UCB principle. Second, FASI extends the original CDPR framework by periodically updating its empirical Pareto frontier using simulated trajectories that pass uncertainty and constraint-compliance checks. Overall, the paper is primarily empirical, proposing uncertainty-aware extensions to an existing Decision Transformer-based auto-bidding framework.

Critical strengths

1, The paper studies constrained auto-bidding, an important practical application of sequential decision-making. The two proposed extensions address plausible limitations of previous approaches: unreliable counterfactual predictions and a static empirical Pareto frontier. In particular, using a shared uncertainty signal to regulate both policy improvement and frontier evolution is a reasonable and potentially useful design.

2, The proposed method achieves improvements over the strongest baseline across all ten dataset-budget configurations, supported by ablation and robustness experiments. The reported online A/B testing results further suggest potential practical value.

Critical weaknesses

My major concern is that the FASI mechanism is insufficiently specified and appears internally inconsistent. The outcome predictor in Eq. (8) estimates future accumulated rewards and costs, but FASI requires autoregressive generation of complete counterfactual trajectories. It is unclear how intermediate outcomes and state transitions are generated from these predictions. Moreover, Eq. (15) recomputes the Pareto frontier over the union of existing and imagined points, which can remove existing empirical frontier points when they become dominated. This appears inconsistent with the claim in Section 4.3 that FASI never removes empirical points. Additionally, the reliability of uncertainty estimation is not sufficiently justified or validated, as ensemble disagreement does not necessarily increase for out-of-distribution actions or provide calibrated confidence estimates. The paper appears to assume some sort of reliability without theoretical justification or direct empirical validation.

Questions for authors

Please see my concerns in "Weaknesses" listed above. I did not check the Appendix very carefully due to the short period of time for review, so please let me know if I missed anything.

Reviewer_bga8OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Authors tackle the problem of auto-bidding in offline environment optimizing the cost per conversion as per the advertiser’s budget. Towards this goal comparing the prior methods PRO-Bid / CDPR+CRO , the authors identify few gaps – 1) Pareto frontier is learned purely from historic logs which establishes a limitation and bias; 2) outcome predictor being a point estimator, authors suspect out of distribution led hallucinations. To circumvent these constraints, toward the prior constraint authors propose a 3 head Gaussian ensemble sharing the same backbone and for the second they propose a confidence-based gating of the trajectories for influencing the Pareto frontier by reweighing the sampling of the real data and not directly participate in the training.

Critical strengths

It's a well diagnosed problem identifying valid gaps in solution (PRO-Bid) - single point estimator underperforming on ODD and leveraging / capping performance on historical data with static Pareto frontier. Single signal (sigma_tot) filters risky counterfactuals and gates frontier updates. Imagined data only reweighs the real data choice rather than being used for training makes this proposal safe. Experimental rigour is good.

Critical weaknesses

Do OOD exist and how does one describe them in this domain? There seems to be no calibration or evidence and how ensemble vs point predictor works there. It is also not very clear that how just re weighting the real data exceeds the empirical Pareto frontier. I feel that proposal lack theoretical backing and the associated empiral result based claims are moderate. The method combines PRO-Bid, a standard deep-ensemble LCB (as in MOPO, MOReL, EDAC) and MBPO-style confidence-gated imagination and evaluates on one benchmark family without any theoretical proofs which sounds to me solid as an industry/applied track paper.

Questions for authors
  1. Elaborate what you mean by an OOD here?
  2. Removing UR-CRO alone (364.0) scores worse than removing both modules (371.9). So FASI without uncertainty is harmful, which the paper does not analyze.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.