- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes CADET, a framework for predicting human actions in e-commerce setting. It replaces autoregressive action generation with a classification head on a pretrained LLM backbone and uses log or Brier loss to learn calibrated action probabilities. Experiments on OPeRA and ShopCART show improved calibration over exact-match reward optimization, but inconsistent accuracy gains and weaker full-action prediction than autoregressive SFT. The authors conclude that direct classification with proper scoring rules offers a more calibrated approach to user simulation.
Critical strengths
N/A
Critical weaknesses
- The paper is formatted and written poorly, see Table 2.
- The related work section discuss about computer use agents, but the actual paper is about user action simulation. It's unclear what is the motivation for the work.
- The paper lacks novelty, as it essentially trains a LLM backbone with classification head.
- The paper claims that exact match loss can't learn probability distribution, and proposed using cross entropy or Brier. This is a well known knowledge in machine learning.
Questions for authors
N/A
Response to "Reviewer LdwN"
We thank the reviewer and respond point by point. We have uploaded a revised version addressing each point.
1. Motivation and related work (agents vs. simulators). The submitted Related Work already discusses shopping simulators (Customer-R1, Shop-R1) and OPeRA; we agree the motivation was under-stated. User simulators are used to evaluate site, search, and ranking changes offline, pre-screen A/B tests, and generate synthetic sessions. These uses need how often people take each action, not only the most likely one: a mode-seeking simulator always takes the modal action and essentially never simulates rare behavior such as abandonment.
This is where simulation differs from the computer-use agents we cited (Mind2Web, WebLINX), which we cited only for their shared action format. An agent should output the best action, so mode-seeking is fine; a simulator must match the human distribution. Shopping simulators (Customer-R1, Shop-R1) inherit agent-style training that rewards agreement with the recorded action; we study what this discards. Revision: The introduction now opens with these use cases; Related Work now leads with user simulation and states this contrast.
2. "That exact match cannot learn a distribution is well known." We agree. We cite Gneiting & Raftery and present Proposition 1 as a specialization, not a new theorem. Our contribution is its consequence for simulators trained with exact-match rewards:
Regularization adds a weight to tune; proper scores do not. Entropy- and KL-regularized exact-match objectives identify the distribution only through an inverse link (App. A.3), so their native probabilities depend on the regularizer weight; proper scores have none. With LoRA on Gemma-3-1B at the default weight (0.10), CADET (log) reaches 47.01% leaf accuracy against 46.86% for entropy-regularized exact-match, with ECE 0.064 versus 0.167 (Table 4). At a tuned weight, chosen post hoc on the test set, regularized exact-match attains lower native ECE than CADET in the cases we tested, and native CADET can still need recalibration (see our response to Reviewer tUoS).
Rare actions are lost during optimization. Proposition 2 shows exact-match gradients vanish on low-probability labels. With LoRA on four backbones (Appendix Table 15), exact-match assigns 0.0001-0.0026x the true termination mass, its KL- and entropy-regularized variants (weight 0.10) 0.17-0.58x, and CADET (log and Brier) 0.75-0.93x (ideal 1.0). Termination recall is zero for every arm.
Equal accuracy is expected. Corollary 1 shows both objectives reach Bayes accuracy, so differences appear in the probabilities.
Breadth. At the default weight, the native ECE gain over the best regularized baseline holds on all 7 frozen (-27-58%) and all 4 LoRA backbones (-71-79%) across Gemma, Llama, and ModernBERT.
3. "Just a classification head on an LLM." The architecture is intentionally simple. The contribution is changing the simulator's interface from generating action strings to a typed decision. This (i) yields a valid type-subtype action by construction in one forward pass; (ii) gives an explicit, sampleable probability for every action; and (iii) calls a text generator only when an action needs text.
We do not claim CADET beats autoregressive SFT on full actions (27.6% vs. 17.4%, scored under different output contracts). Target-element selection, not the typed decision, is the bottleneck: always-click with the same selector reaches 16.9%. The text branch is a modular interface, not an accuracy gain: under identical routing, copying the previous query matches generation (17.41% vs. 17.37%). Our experiments evaluate single-step forecasting of recorded human actions; the abstract and conclusion now state that multi-step simulation fidelity is untested.
4. Presentation and Table 2. We agree. We have: (a) split Tables 1 and 2 into separate floats with self-contained captions, no blank cells, and protocol notes moved to the appendix; (b) moved the separately fitted browse heads, whose setup differs, from Table 2 to the appendix; (c) defined "Public decision model API," "Luna," and "leaf" before first use; (d) fixed caption errors (e.g., Table 3); and (e) added an appendix roadmap (appendix letters are unchanged, so existing references hold).
5. Soundness. No specific soundness issue was listed, so we summarize the evidence. The propositions have proofs and numerical checks (App. A). Table 4 fixes each backbone's configuration and varies only the loss; its ECE values reproduce when recomputed from saved per-seed outputs. Table 3's ShopCART panel re-scores exactly from saved per-seed probabilities; its OPeRA panel is provisional, because one of five seeds' predictions survives (see our response to Reviewer qseM). With a known ground truth and Bayes accuracy held fixed, CADET halves the distance to the true distribution (mean total variation 0.33 versus 0.66 for bare exact-match).