acceptodds
Under review as a conference paper at ICLR 2027

ADX-Bench: A Benchmark for LLMs in Advertising Forecasting, Diagnosis, and Decision Making

Abstract

Large language models (LLMs) are increasingly used for domain-specific reasoning and decision support, yet their capabilities in real-world advertising remain insufficiently evaluated. Advertising decision making is inherently sequential: models must anticipate future outcomes, diagnose merchant-specific performance issues, select actions under explicit business constraints, and determine whether committed decisions should be revised as new information becomes available. Existing evaluations typically abstract these capabilities into isolated tasks and focus primarily on task-level correctness, providing limited insight into whether model behavior is grounded, operationally valid, and reliable throughout the decision process. To address these challenges, we introduce ADX-Bench, a decision-centered benchmark for evaluating LLMs in real-world advertising. ADX-Bench comprises three complementary benchmarks constructed around a shared data lineage. Action-Conditioned Outcome Forecasting (AOF) evaluates multivariate multi-step forecasting from historical advertising trajectories, campaign context, and observed actions. Evidence-Grounded Diagnosis (EGD) evaluates taxonomy-constrained root-cause identification, merchant-specific evidence grounding, and intervention recommendation. Constraint-Aware Action Selection (CAS) evaluates action selection under business objectives and operational constraints, together with post-commitment reasoning when additional evidence and critiques are revealed. Built from real-world operational advertising data, ADX-Bench preserves the sequential dependencies and evolving information states of advertising decisions while enabling independent evaluation of predictive, diagnostic, and decision-making capabilities.

Primary area:
datasets and benchmarks
Keywords:
Computational Advertising, Time-Series Forecasting, Evidence-Grounded Diagnosis, Constraint-Aware Decision Making

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FcAROpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces ADX-Bench, a benchmark built from real-world advertising operational data to evaluate LLMs for decision support. It links three tasks through shared data lineage: Action-Conditioned Outcome Forecasting (AOF), with 4,513 merchant–action instances; Evidence-Grounded Diagnosis (EGD), with 100 expert-annotated cases; and Constraint-Aware Action Selection (CAS), with 50 two-round decision cases. The evaluation covers forecasting accuracy, root-cause identification, intervention recommendations, evidence grounding, constraint compliance, forecast-based utility, and appropriate plan revision after additional evidence and critiques. Experiments compare conventional time-series models, time-series foundation models, and general-purpose LLMs. The results show that specialized temporal models perform better on several forecasting targets, auxiliary forecasts do not consistently improve diagnosis, and second-round feedback can introduce new decision errors.

Critical strengths
  1. A practically meaningful formulation of advertising decision support. ADX-Bench uses real-world operational data and shared data lineage to connect forecasting, diagnosis, and action selection, with expert annotations and explicit business constraints. This provides a valuable evaluation setting for studying how models integrate numerical observations, merchant context, and business objectives, constituting the paper’s main contribution.

  2. Reliability-oriented evaluation beyond answer correctness. The benchmark distinguishes merchant observations from conditional forecasts and general product knowledge, requires verifiable diagnostic evidence, and introduces a two-round protocol for assessing critique handling and plan revision. These designs expose failure modes that final-answer accuracy alone can miss, strengthening the benchmark’s value for research on reliable decision support.

Critical weaknesses
  1. The evaluation does not sufficiently substantiate the central positioning around sequential decision-making. AOF centers on individual actions and short observation windows, while the three tasks are largely scored independently. CAS introduces evidence and critiques rather than actual environmental feedback following action execution, and evaluates utility using frozen forecasts. Shared data lineage therefore does not establish a sustained state–action–outcome–revision trajectory. Moreover, forecasting accuracy on observed actions does not validate utility rankings for unexecuted alternatives. These limitations weaken the connection between benchmark performance and reliable, sustained advertising decision-making.

  2. Limited coverage and untested transferability constrain the benchmark’s broader relevance. AOF includes actions from a single date and merchants with exactly one recorded operation that day; EGD and CAS contain only 100 and 50 cases, respectively. Diagnostic labels, product mechanisms, and action scoring are closely tied to a particular platform’s operational setting. Without temporal or independent-platform evaluation, the paper does not establish whether its findings reflect transferable capabilities or adherence to platform-specific conventions. This leaves the scope and robustness of model rankings and auxiliary-information findings insufficiently characterized.

Questions for authors

How are sequential dependencies operationalized in the evaluation? Does CAS receive each model’s own EGD outputs or expert reference diagnoses? Could the authors provide an end-to-end experiment or controlled ablation showing how upstream forecasting and diagnostic errors affect final action selection, thereby demonstrating the evaluation value of shared data lineage?

What validates candidate-action forecasts as a basis for decision scoring? How are the frozen CAS forecasts generated, and are they shared across evaluated models? Since AOF validates outcomes only for observed actions, how are utility rankings among unexecuted alternatives validated? Without such validation, should CAS scores be interpreted as planning ability conditional on supplied forecasts or as actual advertising decision quality?

How stable are the main findings under distribution shifts? Could the authors evaluate additional action dates or an independent platform to test the stability of model rankings, the diagnostic effects of auxiliary forecasts, and second-round revision outcomes? Which fields, diagnostic labels, and scoring rules require platform-specific adaptation, and which evaluation mechanisms transfer directly?

9 comments

We sincerely apologize for our delayed response, and we greatly appreciate the reviewer's patience and thoughtful evaluation. We are particularly grateful for your recognition of ADX-Bench's practical relevance, its use of real-world advertising data, and its emphasis on reliability beyond final-answer correctness. Your insightful concerns regarding sequential dependencies, forecast-based decision scoring, and generalizability have motivated us to conduct substantial additional experiments and further clarify the scope of our benchmark. We address your questions individually below.

Q1

We sincerely thank the reviewer for raising this important concern. We fully agree that shared data lineage alone does not establish meaningful sequential dependencies. We apologize that the original manuscript did not sufficiently clarify how information propagates across the three evaluation stages.

First, we would like to clarify the model-specific information flow. In ADX-Bench, each evaluated LLM generates its own AOF forecasts, uses these forecasts to produce EGD diagnoses, and subsequently uses its own diagnostic outputs and candidate-action forecasts in CAS. Expert-reference diagnoses are not used to replace model-generated diagnoses in the default pipeline; they are introduced only in controlled ablation experiments. Although the three stages are scored separately for interpretability, their intermediate information is connected.

Following your suggestion, we conducted additional experiments to examine both the impact of upstream forecasts on diagnosis and the influence of forecasting and diagnostic information on downstream action selection.

1. Impact of Upstream Forecasting on EGD.

We conducted two controlled EGD experiments on the same 100 cases. In the first setting, all six LLMs receive GPT-5.6-Sol forecasts. In the second, they receive combined forecasts selected from the lowest-NRMSE LLM for each advertising outcome: GLM-5.2 for Charge, Gemini-3.1-Pro for Exposure, Clicks, and Conversions, GPT-5.6-Sol for Leads and GMV, and DeepSeek-V4-Pro for Orders.

The following table reports EGD performance.

ModelRCF1AF1EGS
LongCat-2.033.1544.6072.85
Kimi-K2.634.7450.9473.84
GLM-5.238.2752.6379.56
DeepSeek-V4-Pro39.2853.4680.85
Gemini-3.1-Pro40.5652.9778.29
GPT-5.6-Sol42.1558.5788.19
Average38.0352.2078.93

The following table reports the corresponding results using the combined forecasts.

ModelRCF1AF1EGS
LongCat-2.034.6346.4568.41
Kimi-K2.633.7951.2074.74
GLM-5.238.3350.1974.02
DeepSeek-V4-Pro39.3752.2680.26
Gemini-3.1-Pro39.7152.9975.39
GPT-5.6-Sol42.7257.2486.69
Average38.0951.7276.59

The results show that changing upstream forecasting inputs affects diagnostic performance, although the effects differ across models and metrics. The combined forecasts achieve a slightly higher average RCF1 (38.09 vs. 38.03), while GPT-5.6-Sol forecasts yield higher average AF1 (52.20 vs. 51.72) and EGS (78.93 vs. 76.59). These findings suggest that combining individually strong forecast sources does not necessarily improve all aspects of diagnosis, highlighting the nontrivial relationship between forecasting and diagnostic performance.

Q1

2. Impact of Upstream Forecasting and Diagnosis on CAS.

We further conducted a controlled cross-stage ablation involving all six LLMs on the same 50 CAS cases. We designed six configurations to distinguish the effects of forecasting inputs, diagnostic information, and the forecasts used for evaluation.

SettingDiagnosis InputForecast InputScoring Reference
S1Model-generated EGDModel-specific AOFCommon forecasts
S2Expert-reference diagnosisModel-specific AOFCommon forecasts
S3EGD regenerated using common forecastsCommon forecastsCommon forecasts
S4Expert-reference diagnosisCommon forecastsCommon forecasts
S1'Same decisions as S1Model-specific AOFModel-specific AOF
S2'Same decisions as S2Model-specific AOFModel-specific AOF

S1 preserves the original model-specific upstream information flow while applying a common scoring reference. S2 replaces only the diagnosis with expert annotations. S3 regenerates EGD diagnoses using common forecasts and propagates the updated information into CAS, whereas S4 combines expert diagnoses with common forecasts. S1' and S2' retain the exact decisions from S1 and S2, respectively, while changing only the scoring forecasts.

All configurations use the same 50 cases, candidate-action space, business objectives, and operational constraints. The common CAS reference combines GLM-5.2 for Charge, GPT-5.6-Sol for GMV, DeepSeek-V4-Pro for Orders, and Gemini-3.1-Pro for Exposure and Clicks. Original forecast values are preserved without monotonic calibration.

To quantify the impact of upstream substitutions, the following table reports paired differences in CAS utility gain (UG) with 95% confidence intervals estimated from 10,000 paired bootstrap resamples over the same 50 cases.

ModelS4−S3: DiagnosisS4−S2: ForecastS4−S1: Both
Kimi-K2.6+6.02 [−6.27, 18.12]+27.70 [13.66, 42.22]+30.27 [14.88, 45.43]
LongCat-2.0+6.70 [−6.55, 20.18]+36.24 [22.89, 49.61]+34.34 [19.25, 49.22]
DeepSeek-V4-Pro+13.17 [1.15, 25.13]+22.30 [10.12, 35.13]+25.17 [11.15, 39.40]
GLM-5.2+7.25 [−7.24, 21.72]+20.68 [3.28, 37.53]+22.34 [5.00, 39.25]
Gemini-3.1-Pro+2.85 [−7.39, 13.44]+38.38 [24.69, 51.52]+27.16 [14.14, 40.32]
GPT-5.6-Sol+5.38 [−4.76, 15.47]+35.40 [21.13, 49.72]+29.15 [16.24, 42.14]

The results provide three important observations.

First, forecasting information substantially affects downstream planning. With expert diagnoses held fixed (S4−S2), replacing model-specific forecasts with common forecasts increases UG by 20.68–38.38 points, with all six paired confidence intervals above zero.

Second, diagnostic information also influences downstream decisions, although its effect is less consistent. Under common forecasts (S4−S3), replacing model-generated diagnoses with expert annotations improves mean UG by 2.85–13.17 points. However, only DeepSeek-V4-Pro has a 95% confidence interval entirely above zero.

Third, jointly replacing forecasts and diagnoses changes downstream planning performance substantially. Relative to S1, S4 improves UG by 22.34–34.34 points, with all six paired confidence intervals above zero.

Importantly, the S1'/S2' controls reveal that forecast-reference disagreement also contributes substantially to these differences. With the selected actions unchanged, switching from common-reference scoring to model-specific scoring increases UG by 18.84–40.31 points for S1'−S1 and 22.91–45.79 points for S2'−S2. Therefore, the improvements observed under S4 cannot be attributed solely to better upstream predictions or decision-making.

For example, in DeepSeek-V4-Pro case L3-007, the model selects a 10% price increase under S2 because its own forecasts predict peak daily spending of 150 and ROI of 4.4603, satisfying the spending cap of 183 and ROI threshold of 4.18. However, the common reference predicts peak spending of 298 and ROI of 2.7004 for the same action, violating both constraints. After receiving common forecasts in S4, the model selects human escalation, increasing HC, UG, and AE from 0 to 100 under the shared reference.

We sincerely appreciate the reviewer's suggestion, which has helped us more rigorously characterize the sequential dependencies and evaluation boundaries of ADX-Bench.

Q2

We sincerely thank the reviewer for raising this important concern. We fully agree that forecasting accuracy on historically observed actions does not necessarily establish the validity of predicted outcomes for unexecuted alternatives. We appreciate the opportunity to clarify how candidate-action forecasts are generated and evaluated, as well as the intended interpretation of CAS scores.

1. Generation and Evaluation of Candidate-Action Forecasts.

We would first like to clarify that ADX-Bench evaluates each LLM independently, without sharing forecasts or diagnostic outputs across evaluated models in the original protocol.

For each evaluated LLM, the three stages follow a consistent model-specific information flow. Specifically, CAS receives the same LLM's own AOF forecasts and EGD diagnoses, rather than outputs generated by other LLMs or expert-reference diagnoses.

During CAS, each LLM can additionally request D0–D7 forecasts for admissible candidate actions before selecting its plan. These candidate-action forecasts are generated within that model's own forecasting process and represent predicted outcomes under hypothetical advertising interventions.

The forecasts used for scoring are also model-specific and remain frozen across the two CAS rounds. Consequently, each LLM is evaluated using its own fixed forecast information, and newly revealed R2 evidence or critiques do not modify the forecast values used to compute HC, UG, and AE.

Importantly, the original CAS evaluation does not employ a common forecast reference shared across different LLMs. The common forecasts introduced in our additional cross-stage experiments serve exclusively as controlled evaluation inputs for investigating the sensitivity of downstream decisions and scores to upstream forecasting information.

This distinction ensures that the original protocol preserves each model's independent forecasting–diagnosis–decision process, while the additional controlled experiments provide a separate analysis of forecast-reference sensitivity.

2. Validation Boundaries for Unexecuted Candidate Actions.

We fully agree with the reviewer that accurate forecasts for historically executed actions do not automatically validate the predicted outcomes or utility rankings of alternative actions that were never executed.

In AOF, forecasting accuracy is evaluated against observed outcomes associated with historically recorded advertising actions. However, for an alternative action that was not executed, its corresponding outcome is unobserved. Therefore, AOF forecasting accuracy alone cannot establish the counterfactual accuracy of candidate-action predictions or verify the true utility ranking among alternative interventions.

Accordingly, we do not treat the frozen CAS forecasts as ground-truth counterfactual outcomes. Instead, they define a model-specific, forecast-conditional evaluation environment.

Within this environment, CAS evaluates whether a model can identify feasible actions, satisfy operational constraints, select actions with favorable predicted utility, account for intervention costs, and appropriately revise its decisions when additional evidence becomes available.

Reviewer_64AWOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a benchmark suite (ADX-Bench) for evaluating the performance of LLMs in advertising decision-making compared to prior approaches including time-series and foundational models, and general-use LLMs. There are three well-constructed benchmarks including Action-Conditioned Outcome Forecasting (AOF, 4000+ instances), Evidence-Grounded Diagnosis (EGD, 100 instances), and Constraint-Aware Action Selection (CAS, 50 instances) which are constructed from real-world Ad data. The problem addressed is important, practically relevant, and framed well. Experiments comparing the performance to several baselines are reported to validate their approach.

The sequential trio of forecast, then diagnose, and then decide is realistic reflection of advertising decision making in practice. The paper's key contribution is to clearly articulate why evaluating the individual components of this trio separately doesn't work well and why reliability (evidence-grounding, constraint-satisfaction, and justifiable revision) may be equally or more useful that mere accuracy metrics.

Critical strengths

Problem framing and identifying the key aspects of decision-making in advertising and proposing a new way of addressing the problem holistically rather than in silos.

Well thought out benchmarks with real data from production systems and sound evaluation metrics is a big plus.

Critical weaknesses
  1. Small scale of data for EGD and CAS is a major concern, with no statistical significance attached to the results, which hurts the soundness.

  2. The sequential trio that are interconnected, which is an important aspect of their problem framing and conceptual contribution, does not appear to be adequately tested in the experiments so we are not sure how valuable this holistic analysis is, compared to prior approaches that handle them separately.

Questions for authors
  1. why are there no statistical significance values for the results, and also sometimes it appears that the evaluation metrics for EGD rely partly on LLM-based judgment, i.e. potential circularity of LLMs to evaluate LLM?
  2. Can we have experiments that quantify the importance of having an approach that takes the sequential dependence into account versus separately with the best baseline for each component.
5 comments

We sincerely apologize for our delayed response, and we greatly appreciate the reviewer's patience and thoughtful evaluation. We are particularly grateful for your recognition of the practical importance of ADX-Bench, its holistic problem formulation, and the value of constructing benchmarks from real-world advertising data. Your concerns regarding statistical reliability and sequential dependencies have motivated us to conduct substantial additional experiments. We address your questions individually below.

Q1

We sincerely thank the reviewer for highlighting these important concerns. We fully agree that statistical uncertainty is particularly important for relatively small evaluation sets, and that LLM-based evaluation requires careful consideration of potential judge bias.

1. Clarification of LLM-Based Evaluation and Potential Judge Bias.

We also sincerely appreciate the reviewer's concern regarding the potential circularity of using LLMs to evaluate LLM-generated diagnoses. We fully agree that LLM-based judgments should not automatically be regarded as independent ground truth.

We would like to clarify that EGD does not rely entirely on LLM-based evaluation. Its three metrics employ different evaluation mechanisms:

  • Root-Cause Identification F1 (RCF1): Measures agreement between predicted root causes and expert-annotated reference labels.
  • Action Recommendation F1 (AF1): Measures agreement between predicted actions and expert-annotated action types.
  • Evidence-Grounded Score (EGS): Evaluates evidence validity and semantic support for the predicted diagnosis.

For EGS, evidence validity is checked against the frozen merchant-specific input, including source references, field paths, and observed values. The evidence-support component evaluates entailment and diagnostic coherence through repeated LLM-based judgments with majority voting. Additionally, support is assigned zero when the predicted root-cause set has no overlap with the expert annotations. Therefore, RCF1 and AF1 are directly grounded in expert annotations, while LLM-based judgments are restricted to the semantic-support component of EGS.

Q1

2. Statistical Reliability of EGD and CAS.

To address the lack of statistical uncertainty analysis, we conducted ten independent evaluations for each of the six LLMs on the same 100 EGD cases, as well as ten complete two-round CAS evaluations on the same 50 merchant cases, yielding 3,000 paired R1/R2 CAS observations. We further performed 10,000 bootstrap resamples with replacement to estimate 95% confidence intervals, accounting for case-level sampling uncertainty and repeated-run variability. Importantly, repeated evaluations assess stability rather than increasing the number of distinct cases.

We first report the EGD results, including the mean ± standard deviation across ten runs and the corresponding 95% confidence intervals.

ModelRCF1AF1EGS
LongCat-2.028.82 ± 3.62; [24.66, 33.19]39.71 ± 6.02; [34.06, 45.31]55.37 ± 9.48; [48.30, 62.24]
Kimi-K2.632.78 ± 1.22; [28.38, 37.11]49.24 ± 1.83; [43.61, 54.77]71.57 ± 1.63; [67.46, 75.62]
GLM-5.237.14 ± 2.37; [33.70, 42.58]52.68 ± 1.74; [47.63, 57.64]78.35 ± 2.52; [73.44, 83.01]
Gemini-3.1-Pro38.10 ± 1.51; [33.76, 42.42]50.27 ± 1.70; [45.50, 54.93]76.75 ± 2.75; [72.64, 80.63]
DeepSeek-V4-Pro38.37 ± 1.17; [34.04, 42.75]50.30 ± 1.47; [45.48, 54.98]78.31 ± 1.85; [74.17, 82.22]
GPT-5.6-Sol39.27 ± 1.36; [35.14, 43.44]55.96 ± 1.51; [51.40, 60.31]86.21 ± 1.28; [82.02, 90.17]

The additional results show that GPT-5.6-Sol achieves the highest mean scores across all three EGD metrics. Meanwhile, the confidence intervals reveal uncertainty in several model comparisons, particularly for RCF1 and AF1. These results provide a more transparent characterization of EGD evaluation variability, rather than relying solely on point estimates.

We further report the corresponding CAS results, where each entry represents the ten-run mean ± standard deviation and 95% bootstrap confidence interval.

ModelHCUGAECD
Kimi-K2.685.20 ± 3.79 [77.40, 92.00]66.34 ± 3.95 [56.57, 75.60]72.71 ± 4.32 [62.94, 81.75]89.92 ± 2.35 [83.23, 95.17]
LongCat-2.083.60 ± 3.86 [75.80, 90.40]75.24 ± 4.20 [66.61, 83.28]74.84 ± 5.55 [65.39, 83.56]87.65 ± 4.33 [80.33, 93.37]
DeepSeek-V4-Pro88.40 ± 3.63 [81.40, 94.40]78.19 ± 4.36 [70.28, 85.60]85.22 ± 3.32 [77.99, 91.50]94.75 ± 2.35 [91.13, 97.28]
GLM-5.285.00 ± 2.16 [76.60, 92.20]72.63 ± 3.59 [62.97, 81.91]74.49 ± 3.67 [65.63, 82.85]88.31 ± 3.85 [81.43, 93.93]
Gemini-3.1-Pro91.60 ± 2.27 [84.00, 97.60]79.36 ± 1.85 [69.03, 88.84]79.14 ± 2.71 [69.19, 87.94]92.93 ± 3.44 [87.33, 96.96]
GPT-5.6-Sol92.40 ± 2.46 [86.40, 97.00]83.04 ± 4.14 [74.89, 90.25]85.31 ± 2.84 [78.06, 91.52]94.70 ± 1.53 [90.62, 97.60]

We further conducted paired permutation tests for all 15 model pairs across the four CAS metrics. Among the 60 comparisons, 15 attained nominal significance at the unadjusted threshold of p ≤ 0.05, including 1 for HC, 3 for UG, 6 for AE, and 5 for CD. The following table presents all 15 comparisons, including mean differences, 95% confidence intervals estimated from 10,000 paired bootstrap resamples, and unadjusted p-values.

MetricModel ComparisonMean Difference95% CIRaw p-value
HCGPT-5.6-Sol − LongCat-2.0+8.80[0.20, 17.60]0.0360
UGDeepSeek-V4-Pro − Kimi-K2.6+11.85[1.99, 21.59]0.0092
UGGemini-3.1-Pro − Kimi-K2.6+13.02[1.48, 24.80]0.0242
UGGPT-5.6-Sol − Kimi-K2.6+16.70[4.36, 29.05]0.0061
AEDeepSeek-V4-Pro − Kimi-K2.6+12.51[2.61, 22.54]0.0069
AEGPT-5.6-Sol − Kimi-K2.6+12.60[0.94, 23.96]0.0237
AEDeepSeek-V4-Pro − LongCat-2.0+10.38[0.30, 20.65]0.0220
AEGPT-5.6-Sol − LongCat-2.0+10.47[0.05, 20.61]0.0253
AEDeepSeek-V4-Pro − GLM-5.2+10.73[2.86, 19.11]0.0025
AEGPT-5.6-Sol − GLM-5.2+10.81[0.11, 21.49]0.0351
CDDeepSeek-V4-Pro − LongCat-2.0+7.10[0.50, 14.93]0.0216
CDGPT-5.6-Sol − LongCat-2.0+7.05[0.22, 14.77]0.0283
CDDeepSeek-V4-Pro − GLM-5.2+6.45[0.73, 13.22]0.0247
CDGemini-3.1-Pro − GLM-5.2+4.62[−0.56, 10.21]0.0362
CDGPT-5.6-Sol − GLM-5.2+6.40[1.36, 12.27]0.0078

These analyses provide quantitative evidence of performance variability and differences between models, which provide additional evidence regarding evaluation stability and help characterize uncertainty that was not adequately reported in the original manuscript.

Q2

We sincerely thank the reviewer for this insightful suggestion. We fully agree that sharing a common data lineage alone is insufficient to establish the importance of sequential dependencies. We apologize that the original experiments did not adequately quantify how upstream forecasting and diagnostic information affect downstream decision-making.

We would first like to clarify that ADX-Bench implements a model-specific forecasting–diagnosis–decision pipeline: each evaluated LLM generates its own AOF forecasts, uses these forecasts to produce EGD diagnoses, and subsequently uses its own diagnostic outputs and candidate-action forecasts in CAS. Although the three stages are scored separately to preserve stage-level interpretability, their model-generated information is propagated across stages.

Following your suggestion, we conducted additional controlled experiments to investigate how changes in upstream forecasting information affect EGD and how forecasting and diagnostic information subsequently influence CAS decisions.

1. Impact of Upstream Forecasting on EGD.

To examine the dependency between forecasting and diagnosis, we conducted two additional experiments on the same 100 EGD cases, providing all six evaluated LLMs with standardized upstream forecast inputs.

In the first setting, all models receive forecasts generated by GPT-5.6-Sol. In the second setting, all models receive a common combination of forecasts, where the source model for each advertising outcome is selected according to its AOF NRMSE. Specifically, GLM-5.2 is selected for Charge, Gemini-3.1-Pro for Exposure, Clicks, and Conversions, GPT-5.6-Sol for Leads and GMV, and DeepSeek-V4-Pro for Orders.

The following table reports EGD performance when all six models receive GPT-5.6-Sol forecasts.

ModelRCF1AF1EGS
LongCat-2.033.1544.6072.85
Kimi-K2.634.7450.9473.84
GLM-5.238.2752.6379.56
DeepSeek-V4-Pro39.2853.4680.85
Gemini-3.1-Pro40.5652.9778.29
GPT-5.6-Sol42.1558.5788.19
Average38.0352.2078.93

The following table presents the corresponding results when each model receives the combined forecasts selected for individual advertising outcomes.

ModelRCF1AF1EGS
LongCat-2.034.6346.4568.41
Kimi-K2.633.7951.2074.74
GLM-5.238.3350.1974.02
DeepSeek-V4-Pro39.3752.2680.26
Gemini-3.1-Pro39.7152.9975.39
GPT-5.6-Sol42.7257.2486.69
Average38.0951.7276.59

The results reveal that changing upstream forecasting information affects downstream diagnostic performance, although the effects vary across models and metrics.

The combined forecasts achieve a slightly higher average RCF1 (38.09 vs. 38.03), while GPT-5.6-Sol forecasts yield higher average AF1 (52.20 vs. 51.72) and EGS (78.93 vs. 76.59). The changes are not uniform across individual models. For example, LongCat-2.0 improves in RCF1 and AF1 under the combined forecasts, whereas its EGS decreases from 72.85 to 68.41.

Reviewer_dmACOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ADX-Bench, a benchmark for evaluating LLMs on advertising forecasting, evidence-grounded diagnosis, and constraint-aware decision making.

Critical strengths

S1. The problem is important. S2. The two-round CAS protocol is a meaningful contribution. S3. This paper makes a useful distinction between different information sources. S4. The evaluation reports several relevant dimensions rather than only a single aggregate score.

Critical weaknesses
  1. no open source codes and no demo.
Questions for authors

Q1. The shared data lineage is useful for traceability and scenario consistency, but the three benchmarks are still evaluated independently. The current experiments do not demonstrate how forecasting errors propagate into diagnosis and then into downstream decisions, nor do they evaluate state transitions in a closed loop.

Q2. In the CAS two round protocol, this paper states that aggregate scores may remain similar even when individual cases improve or deteriorate, using case 021 and case 052 as examples. However, it does not quantify how often these changes occur. At present, the evidence for this claim relies mainly on qualitative case studies.

Q3. The role of newly revealed evidence relative to the frozen forecasts in the second round is not clear. In case 021, positive evidence from peer merchants leads the model to increase the bid, but the frozen forecast predicts no additional orders from doing so, and the revision is therefore judged incorrect. This is internally consistent with the proposed protocol, but it raises an important question: can new evidence update the model’s belief about the action outcomes, or is the forecast treated as fixed ground truth?

Q4. Some failures reflect output-format compliance or search strategy and cannot be entirely attributed to decision reasoning. For example, a leading space can cause an action to become unparsable, while failing to query a feasible candidate action can affect the final decision score. These should be distinguished from errors in decision-making reasoning

Q5. Will you open the benchmark?

6 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback. We particularly appreciate your recognition of the practical relevance of ADX-Bench and the value of its two-round CAS evaluation. We also sincerely apologize for our delayed response, as we wanted to thoroughly address your valuable concerns through additional experiments, including cross-stage ablations, repeated evaluations, and detailed case-level analyses.

Q1.

We sincerely thank the reviewer for raising this important concern. We fully agree that sharing a common data lineage does not, by itself, establish meaningful sequential dependencies. We also apologize for not making the information flow sufficiently clear in the original manuscript.

ADX-Bench implements a model-specific forecasting–diagnosis–decision pipeline: each evaluated LLM generates its own AOF forecasts, uses these forecasts to produce EGD diagnoses, and subsequently uses its diagnostic outputs and candidate-action forecasts in CAS. Although the three stages are scored separately to preserve stage-level interpretability, their model-generated information is propagated across stages. Following your suggestion, we conducted additional controlled experiments to examine how changes in upstream forecasts affect diagnosis and how changes in forecasts and diagnoses subsequently influence decision-making.

we conducted additional controlled experiments to examine how changes in upstream forecasts affect diagnosis and how changes in forecasts and diagnoses subsequently influence decision-making.

1. Impact of Upstream Forecasts on EGD.

To investigate how forecasting information affects downstream diagnosis, we conducted two additional experiments on the same 100 EGD cases. In the first setting, all six LLMs receive the forecasts generated by GPT-5.6-Sol. In the second setting, all models receive a common combination of forecasts, where each advertising outcome is provided by the LLM achieving the lowest AOF NRMSE for that outcome.

The following table reports EGD performance when all models receive GPT-5.6-Sol forecasts.

ModelRCF1AF1EGS
LongCat-2.033.1544.6072.85
Kimi-K2.634.7450.9473.84
GLM-5.238.2752.6379.56
DeepSeek-V4-Pro39.2853.4680.85
Gemini-3.1-Pro40.5652.9778.29
GPT-5.6-Sol42.1558.5788.19
Average38.0352.2078.93

For the second setting, the common forecasts combine GLM-5.2 for Charge, Gemini-3.1-Pro for Exposure, Clicks, and Conversions, GPT-5.6-Sol for Leads and GMV, and DeepSeek-V4-Pro for Orders. The resulting EGD performance is reported below.

ModelRCF1AF1EGS
LongCat-2.034.6346.4568.41
Kimi-K2.633.7951.2074.74
GLM-5.238.3350.1974.02
DeepSeek-V4-Pro39.3752.2680.26
Gemini-3.1-Pro39.7152.9975.39
GPT-5.6-Sol42.7257.2486.69
Average38.0951.7276.59

The results reveal that changing upstream forecasting information affects downstream diagnostic performance, although the effects vary across models and metrics. While the combined forecasts achieve a slightly higher average RCF1 (38.09 vs. 38.03), GPT-5.6-Sol forecasts yield higher average AF1 (52.20 vs. 51.72) and EGS (78.93 vs. 76.59). These findings suggest that combining individually strong forecasting sources does not necessarily produce consistent improvements in diagnosis. More importantly, they demonstrate that EGD performance is sensitive to the forecasting information supplied by the upstream stage.

Q1

2. Impact of Upstream Forecasting and Diagnosis on CAS.

To further examine how upstream information propagates into downstream decisions, we conducted a controlled cross-stage ablation on the same 50 CAS cases across all six LLMs. We considered six configurations to distinguish the effects of forecasting and diagnostic inputs from differences in the forecasts used for evaluation.

SettingDiagnosis InputForecast InputEvaluation Reference
S1Model-generated EGDModel-specific AOFCommon forecasts
S2Expert-reference diagnosisModel-specific AOFCommon forecasts
S3EGD regenerated using common forecastsCommon forecastsCommon forecasts
S4Expert-reference diagnosisCommon forecastsCommon forecasts
S1'Same decisions as S1Model-specific AOFModel-specific AOF
S2'Same decisions as S2Model-specific AOFModel-specific AOF

S1 preserves the original model-specific upstream information flow while evaluating decisions under the common scoring reference. S2 replaces only the diagnosis with expert annotations. S3 regenerates EGD diagnoses using common forecasts and propagates the updated information to CAS, while S4 combines expert diagnoses with common forecasts. Importantly, S1' and S2' reuse exactly the same decisions as S1 and S2, respectively, changing only the scoring forecasts. These additional controls distinguish changes in decision-making from forecast-reference disagreement.

All configurations use the same 50 cases, candidate-action space, and operational constraints. S1–S4 are scored using a common reference constructed from the lowest-NRMSE LLM forecast sources for the corresponding advertising outcomes. The original predicted values are preserved without monotonic calibration.

To quantify the impact of upstream forecasting and diagnosis on downstream decision-making, the following table reports paired differences in CAS utility gain (UG), together with 95% confidence intervals estimated from 10,000 paired bootstrap resamples over the same 50 cases.

ModelS4−S3: DiagnosisS4−S2: ForecastS4−S1: Both
Kimi-K2.6+6.02 [−6.27, 18.12]+27.70 [13.66, 42.22]+30.27 [14.88, 45.43]
LongCat-2.0+6.70 [−6.55, 20.18]+36.24 [22.89, 49.61]+34.34 [19.25, 49.22]
DeepSeek-V4-Pro+13.17 [1.15, 25.13]+22.30 [10.12, 35.13]+25.17 [11.15, 39.40]
GLM-5.2+7.25 [−7.24, 21.72]+20.68 [3.28, 37.53]+22.34 [5.00, 39.25]
Gemini-3.1-Pro+2.85 [−7.39, 13.44]+38.38 [24.69, 51.52]+27.16 [14.14, 40.32]
GPT-5.6-Sol+5.38 [−4.76, 15.47]+35.40 [21.13, 49.72]+29.15 [16.24, 42.14]

The results provide several important observations.

First, forecasting information substantially affects downstream planning. With expert diagnoses held fixed (S4−S2), replacing model-specific forecasts with common-reference forecasts increases UG by 20.68–38.38 points across the six models, with all paired confidence intervals above zero. This demonstrates that changes in upstream forecasting inputs can substantially affect the actions selected by CAS and their evaluated utility.

Second, diagnostic information also affects downstream decisions, although its influence is less consistent. Under common forecasts (S4−S3), replacing model-generated diagnoses with expert annotations improves mean UG by 2.85–13.17 points across all six models. However, only DeepSeek-V4-Pro has a confidence interval entirely above zero. This suggests that improved diagnostic information does not necessarily translate into equally substantial decision improvements, as models must still correctly interpret the diagnosis and select appropriate actions.

Third, jointly replacing forecasts and diagnoses changes downstream planning performance substantially. Relative to S1, S4 improves UG by 22.34–34.34 points, with all six paired confidence intervals above zero. These results demonstrate that downstream planning is sensitive to upstream information that would not be fully characterized by evaluating each stage independently.

Importantly, forecast-reference disagreement also contributes substantially to the observed differences. When decisions remain unchanged and only the scoring reference is replaced with each model's own forecasts (S1'−S1), UG increases by 18.84–40.31 points across the six models. Similarly, S2'−S2 yields improvements of 22.91–45.79 points. Since the selected actions are identical within each comparison, these differences cannot be attributed to better decision-making. Furthermore, the paired confidence intervals for S4−S1' and S4−S2' include zero across all six models.

Q1

3. Case-Level Evidence of Information Propagation.

Beyond aggregate scores, we inspected individual action trajectories to examine how changes in upstream information affected downstream decisions.

For example, in DeepSeek-V4-Pro case L3-007, under S2 the model selected a 10% price increase based on its own forecasts, which predicted peak daily spending of 150 and ROI of 4.4603, satisfying the constraints of daily spending ≤183 and ROI ≥4.18. However, the common reference predicted peak spending of 298 and ROI of 2.7004 for the same action, violating both constraints. After replacing the forecast inputs in S4, the model instead selected human escalation, improving HC, UG, and AE from 0 to 100 under the shared evaluation reference.

Conversely, in Kimi-K2.6 case L3-013, replacing the model-generated diagnosis with an expert diagnosis changed the selected action from human escalation to creative optimization. However, the selected action lacked sufficient predicted outcomes to establish compliance with the ROI constraint, causing HC, UG, and AE to decrease from 100 to 0. This further illustrates that changing diagnostic information does not necessarily improve the final decision.

4. Clarification Regarding Closed-Loop Evaluation.

We sincerely appreciate the reviewer's important observation regarding closed-loop state transitions.

Our additional experiments directly examine sequential information dependencies across the forecasting–diagnosis–decision pipeline. The new EGD experiments evaluate how alternative upstream forecasting inputs affect diagnostic performance, while the cross-stage CAS ablations examine how changes in forecasting and diagnosis propagate into downstream action selection. Furthermore, the two-round CAS protocol evaluates how models respond to newly revealed evidence and critiques when deciding whether to retain or revise committed plans.

Together, these experiments provide a more comprehensive evaluation of information-dependent decision-making beyond isolated task-level assessment, while maintaining a clear distinction between sequential information propagation and closed-loop environmental interaction. We will clarify these findings and evaluation boundaries in the revised manuscript.

Comments on the paper

2 comments

Dear reviewers,

We sincerely thank you for your valuable feedback and apologize again for our delayed responses.

We have now provided detailed responses and additional experiments addressing the major concerns, including cross-stage ablations, statistical uncertainty, forecast-reference sensitivity, and temporal robustness.

As the discussion period is approaching its end, we would greatly appreciate any further feedback on whether our additional analyses have adequately addressed your concerns. We would be happy to clarify any remaining questions before the discussion closes.

Thank you again for your time and consideration.

Best regards,

The Authors

Dear Area Chair and reviewers,

We sincerely appreciate your time and efforts in coordinating the review process. We also apologize for submitting our rebuttal responses relatively late, as we conducted substantial additional experiments to address the reviewers' concerns.

We have now provided detailed responses covering cross-stage ablations, statistical uncertainty with bootstrap confidence intervals, forecast-reference sensitivity, and additional temporal evaluations. We believe these analyses provide important new evidence addressing several of the reviewers' major concerns regarding the evaluation validity, robustness, and interpretation of ADX-Bench.

As the discussion period is approaching its end, we would be very grateful if these additional results could be taken into consideration during the remaining discussion and meta-review process. We would also be happy to clarify any unresolved concerns.

Thank you again for your time and consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.