- Rating
- 4(+2)
- Confidence
- 4(+1)
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies high-level action selection for customer simulation as behavioral cloning over ten actions spanning search, product engagement, conversion, navigation, and session termination. The proposed StructBC score model combines a causal Transformer with slate self-attention, windowed temporal attention, et. It is trained on 452,296 sessions from 6,824 shoppers. The paper additionally fits nine class-specific logit offsets on validation data to maximize macro-F1. On the customer-disjoint test split, the complete system obtains 0.5065 macro-F1 and 59.0% accuracy over ten actions; experiments also cover nine directly observed actions, recurrent and attention-based baselines, an adapted LoRA-fine-tuned LLM, feature audits, and action-frequency diagnostics. The evaluation is explicitly limited to open-loop next-action prediction on logged histories.
Critical strengths
The paper is unusually transparent about its scope and experimental limitations. It uses a large real-world dataset with customer-disjoint splits, reports per-action behavior rather than only aggregate accuracy, repeats the main architecture comparison across recurrent and attentive backbones, and checks that the conclusions persist after excluding the constructed termination label. The feature audit is particularly valuable: the authors identify and remove two future-dependent customer features, document the remaining transductive features, and report relevant ablations instead of hiding these complications. Separating the score model from a simple validation-fitted decision rule is also practically useful, and the action-distribution diagnostics make clear what the rule changes.
Critical weaknesses
-
The reported final policy is a deterministic argmax classifier evaluated only on logged states, whereas the motivating application requires a model of customer action propensities. Class-weighted training and macro-F1-fitted offsets are designed to improve hard classification decisions, but the paper does not establish that the resulting conditional probabilities are calibrated or that sampling from them produces realistic behavior. Matching the marginal action frequencies over logged states cannot show that conversion or termination rates remain faithful after the model generates its own histories. Consequently, the experiments validate open-loop next-action classification, but not the fidelity of a customer simulator or its proposed use for evaluating unseen interfaces.
-
The headline architecture comparison is not conducted with a strictly training-only, temporally available state. Product popularity is aggregated over the complete corpus window, including validation/test customers and events later than the state being predicted, while the promotion features use scaling constants from a calendar interval extending beyond the extraction window. This matters for the architectural claim: removing the popularity channel costs 0.0037 test macro-F1, and in the only popularity-ablated attentive comparison, StructBC’s advantage over the bare Transformer falls from 0.0023 to 0.0005. So the paper does not establish that the already small structural gain survives without this feature. In addition, the authors state that the documented formula does not recover the exact popularity artifact used in the experiments, preventing independent reproduction of the main result from the provided specification.
Questions for authors
-
Can the authors evaluate a stochastic version of the policy, including calibration of the reported past-only model and sampled action distributions, and provide at least a limited closed-loop evaluation of trajectory-level termination, cart-add, and purchase rates? The answer would determine whether the model supports the simulator interpretation rather than only next-action classification.
-
Can the authors reconstruct product popularity using training customers and temporally prior events only, normalize promotion variables without future calendar information, and rerun the bare-Transformer versus StructBC comparison at both reported seeds? Please also clarify whether the exact popularity artifact or sufficient code to reproduce it can be released.
Thank you for your explanation and I decide to change my choice from weak reject to weak accept