- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper introduces ADX-Bench, a benchmark built from real-world advertising operational data to evaluate LLMs for decision support. It links three tasks through shared data lineage: Action-Conditioned Outcome Forecasting (AOF), with 4,513 merchant–action instances; Evidence-Grounded Diagnosis (EGD), with 100 expert-annotated cases; and Constraint-Aware Action Selection (CAS), with 50 two-round decision cases. The evaluation covers forecasting accuracy, root-cause identification, intervention recommendations, evidence grounding, constraint compliance, forecast-based utility, and appropriate plan revision after additional evidence and critiques. Experiments compare conventional time-series models, time-series foundation models, and general-purpose LLMs. The results show that specialized temporal models perform better on several forecasting targets, auxiliary forecasts do not consistently improve diagnosis, and second-round feedback can introduce new decision errors.
Critical strengths
-
A practically meaningful formulation of advertising decision support. ADX-Bench uses real-world operational data and shared data lineage to connect forecasting, diagnosis, and action selection, with expert annotations and explicit business constraints. This provides a valuable evaluation setting for studying how models integrate numerical observations, merchant context, and business objectives, constituting the paper’s main contribution.
-
Reliability-oriented evaluation beyond answer correctness. The benchmark distinguishes merchant observations from conditional forecasts and general product knowledge, requires verifiable diagnostic evidence, and introduces a two-round protocol for assessing critique handling and plan revision. These designs expose failure modes that final-answer accuracy alone can miss, strengthening the benchmark’s value for research on reliable decision support.
Critical weaknesses
-
The evaluation does not sufficiently substantiate the central positioning around sequential decision-making. AOF centers on individual actions and short observation windows, while the three tasks are largely scored independently. CAS introduces evidence and critiques rather than actual environmental feedback following action execution, and evaluates utility using frozen forecasts. Shared data lineage therefore does not establish a sustained state–action–outcome–revision trajectory. Moreover, forecasting accuracy on observed actions does not validate utility rankings for unexecuted alternatives. These limitations weaken the connection between benchmark performance and reliable, sustained advertising decision-making.
-
Limited coverage and untested transferability constrain the benchmark’s broader relevance. AOF includes actions from a single date and merchants with exactly one recorded operation that day; EGD and CAS contain only 100 and 50 cases, respectively. Diagnostic labels, product mechanisms, and action scoring are closely tied to a particular platform’s operational setting. Without temporal or independent-platform evaluation, the paper does not establish whether its findings reflect transferable capabilities or adherence to platform-specific conventions. This leaves the scope and robustness of model rankings and auxiliary-information findings insufficiently characterized.
Questions for authors
How are sequential dependencies operationalized in the evaluation? Does CAS receive each model’s own EGD outputs or expert reference diagnoses? Could the authors provide an end-to-end experiment or controlled ablation showing how upstream forecasting and diagnostic errors affect final action selection, thereby demonstrating the evaluation value of shared data lineage?
What validates candidate-action forecasts as a basis for decision scoring? How are the frozen CAS forecasts generated, and are they shared across evaluated models? Since AOF validates outcomes only for observed actions, how are utility rankings among unexecuted alternatives validated? Without such validation, should CAS scores be interpreted as planning ability conditional on supplied forecasts or as actual advertising decision quality?
How stable are the main findings under distribution shifts? Could the authors evaluate additional action dates or an independent platform to test the stability of model rankings, the diagnostic effects of auxiliary forecasts, and second-round revision outcomes? Which fields, diagnostic labels, and scoring rules require platform-specific adaptation, and which evaluation mechanisms transfer directly?
9 comments
Q1
2. Impact of Upstream Forecasting and Diagnosis on CAS.
We further conducted a controlled cross-stage ablation involving all six LLMs on the same 50 CAS cases. We designed six configurations to distinguish the effects of forecasting inputs, diagnostic information, and the forecasts used for evaluation.
| Setting | Diagnosis Input | Forecast Input | Scoring Reference |
|---|---|---|---|
| S1 | Model-generated EGD | Model-specific AOF | Common forecasts |
| S2 | Expert-reference diagnosis | Model-specific AOF | Common forecasts |
| S3 | EGD regenerated using common forecasts | Common forecasts | Common forecasts |
| S4 | Expert-reference diagnosis | Common forecasts | Common forecasts |
| S1' | Same decisions as S1 | Model-specific AOF | Model-specific AOF |
| S2' | Same decisions as S2 | Model-specific AOF | Model-specific AOF |
S1 preserves the original model-specific upstream information flow while applying a common scoring reference. S2 replaces only the diagnosis with expert annotations. S3 regenerates EGD diagnoses using common forecasts and propagates the updated information into CAS, whereas S4 combines expert diagnoses with common forecasts. S1' and S2' retain the exact decisions from S1 and S2, respectively, while changing only the scoring forecasts.
All configurations use the same 50 cases, candidate-action space, business objectives, and operational constraints. The common CAS reference combines GLM-5.2 for Charge, GPT-5.6-Sol for GMV, DeepSeek-V4-Pro for Orders, and Gemini-3.1-Pro for Exposure and Clicks. Original forecast values are preserved without monotonic calibration.
To quantify the impact of upstream substitutions, the following table reports paired differences in CAS utility gain (UG) with 95% confidence intervals estimated from 10,000 paired bootstrap resamples over the same 50 cases.
| Model | S4−S3: Diagnosis | S4−S2: Forecast | S4−S1: Both |
|---|---|---|---|
| Kimi-K2.6 | +6.02 [−6.27, 18.12] | +27.70 [13.66, 42.22] | +30.27 [14.88, 45.43] |
| LongCat-2.0 | +6.70 [−6.55, 20.18] | +36.24 [22.89, 49.61] | +34.34 [19.25, 49.22] |
| DeepSeek-V4-Pro | +13.17 [1.15, 25.13] | +22.30 [10.12, 35.13] | +25.17 [11.15, 39.40] |
| GLM-5.2 | +7.25 [−7.24, 21.72] | +20.68 [3.28, 37.53] | +22.34 [5.00, 39.25] |
| Gemini-3.1-Pro | +2.85 [−7.39, 13.44] | +38.38 [24.69, 51.52] | +27.16 [14.14, 40.32] |
| GPT-5.6-Sol | +5.38 [−4.76, 15.47] | +35.40 [21.13, 49.72] | +29.15 [16.24, 42.14] |
The results provide three important observations.
First, forecasting information substantially affects downstream planning. With expert diagnoses held fixed (S4−S2), replacing model-specific forecasts with common forecasts increases UG by 20.68–38.38 points, with all six paired confidence intervals above zero.
Second, diagnostic information also influences downstream decisions, although its effect is less consistent. Under common forecasts (S4−S3), replacing model-generated diagnoses with expert annotations improves mean UG by 2.85–13.17 points. However, only DeepSeek-V4-Pro has a 95% confidence interval entirely above zero.
Third, jointly replacing forecasts and diagnoses changes downstream planning performance substantially. Relative to S1, S4 improves UG by 22.34–34.34 points, with all six paired confidence intervals above zero.
Importantly, the S1'/S2' controls reveal that forecast-reference disagreement also contributes substantially to these differences. With the selected actions unchanged, switching from common-reference scoring to model-specific scoring increases UG by 18.84–40.31 points for S1'−S1 and 22.91–45.79 points for S2'−S2. Therefore, the improvements observed under S4 cannot be attributed solely to better upstream predictions or decision-making.
For example, in DeepSeek-V4-Pro case L3-007, the model selects a 10% price increase under S2 because its own forecasts predict peak daily spending of 150 and ROI of 4.4603, satisfying the spending cap of 183 and ROI threshold of 4.18. However, the common reference predicts peak spending of 298 and ROI of 2.7004 for the same action, violating both constraints. After receiving common forecasts in S4, the model selects human escalation, increasing HC, UG, and AE from 0 to 100 under the shared reference.
We sincerely appreciate the reviewer's suggestion, which has helped us more rigorously characterize the sequential dependencies and evaluation boundaries of ADX-Bench.
Q2
We sincerely thank the reviewer for raising this important concern. We fully agree that forecasting accuracy on historically observed actions does not necessarily establish the validity of predicted outcomes for unexecuted alternatives. We appreciate the opportunity to clarify how candidate-action forecasts are generated and evaluated, as well as the intended interpretation of CAS scores.
1. Generation and Evaluation of Candidate-Action Forecasts.
We would first like to clarify that ADX-Bench evaluates each LLM independently, without sharing forecasts or diagnostic outputs across evaluated models in the original protocol.
For each evaluated LLM, the three stages follow a consistent model-specific information flow. Specifically, CAS receives the same LLM's own AOF forecasts and EGD diagnoses, rather than outputs generated by other LLMs or expert-reference diagnoses.
During CAS, each LLM can additionally request D0–D7 forecasts for admissible candidate actions before selecting its plan. These candidate-action forecasts are generated within that model's own forecasting process and represent predicted outcomes under hypothetical advertising interventions.
The forecasts used for scoring are also model-specific and remain frozen across the two CAS rounds. Consequently, each LLM is evaluated using its own fixed forecast information, and newly revealed R2 evidence or critiques do not modify the forecast values used to compute HC, UG, and AE.
Importantly, the original CAS evaluation does not employ a common forecast reference shared across different LLMs. The common forecasts introduced in our additional cross-stage experiments serve exclusively as controlled evaluation inputs for investigating the sensitivity of downstream decisions and scores to upstream forecasting information.
This distinction ensures that the original protocol preserves each model's independent forecasting–diagnosis–decision process, while the additional controlled experiments provide a separate analysis of forecast-reference sensitivity.
2. Validation Boundaries for Unexecuted Candidate Actions.
We fully agree with the reviewer that accurate forecasts for historically executed actions do not automatically validate the predicted outcomes or utility rankings of alternative actions that were never executed.
In AOF, forecasting accuracy is evaluated against observed outcomes associated with historically recorded advertising actions. However, for an alternative action that was not executed, its corresponding outcome is unobserved. Therefore, AOF forecasting accuracy alone cannot establish the counterfactual accuracy of candidate-action predictions or verify the true utility ranking among alternative interventions.
Accordingly, we do not treat the frozen CAS forecasts as ground-truth counterfactual outcomes. Instead, they define a model-specific, forecast-conditional evaluation environment.
Within this environment, CAS evaluates whether a model can identify feasible actions, satisfy operational constraints, select actions with favorable predicted utility, account for intervention costs, and appropriately revise its decisions when additional evidence becomes available.
We sincerely apologize for our delayed response, and we greatly appreciate the reviewer's patience and thoughtful evaluation. We are particularly grateful for your recognition of ADX-Bench's practical relevance, its use of real-world advertising data, and its emphasis on reliability beyond final-answer correctness. Your insightful concerns regarding sequential dependencies, forecast-based decision scoring, and generalizability have motivated us to conduct substantial additional experiments and further clarify the scope of our benchmark. We address your questions individually below.
Q1
We sincerely thank the reviewer for raising this important concern. We fully agree that shared data lineage alone does not establish meaningful sequential dependencies. We apologize that the original manuscript did not sufficiently clarify how information propagates across the three evaluation stages.
First, we would like to clarify the model-specific information flow. In ADX-Bench, each evaluated LLM generates its own AOF forecasts, uses these forecasts to produce EGD diagnoses, and subsequently uses its own diagnostic outputs and candidate-action forecasts in CAS. Expert-reference diagnoses are not used to replace model-generated diagnoses in the default pipeline; they are introduced only in controlled ablation experiments. Although the three stages are scored separately for interpretability, their intermediate information is connected.
Following your suggestion, we conducted additional experiments to examine both the impact of upstream forecasts on diagnosis and the influence of forecasting and diagnostic information on downstream action selection.
1. Impact of Upstream Forecasting on EGD.
We conducted two controlled EGD experiments on the same 100 cases. In the first setting, all six LLMs receive GPT-5.6-Sol forecasts. In the second, they receive combined forecasts selected from the lowest-NRMSE LLM for each advertising outcome: GLM-5.2 for Charge, Gemini-3.1-Pro for Exposure, Clicks, and Conversions, GPT-5.6-Sol for Leads and GMV, and DeepSeek-V4-Pro for Orders.
The following table reports EGD performance.
The following table reports the corresponding results using the combined forecasts.
The results show that changing upstream forecasting inputs affects diagnostic performance, although the effects differ across models and metrics. The combined forecasts achieve a slightly higher average RCF1 (38.09 vs. 38.03), while GPT-5.6-Sol forecasts yield higher average AF1 (52.20 vs. 51.72) and EGS (78.93 vs. 76.59). These findings suggest that combining individually strong forecast sources does not necessarily improve all aspects of diagnosis, highlighting the nontrivial relationship between forecasting and diagnostic performance.