acceptodds
Under review as a conference paper at ICLR 2027

From Generation to Decision: Proper Scoring Rules for Type-Safe User Action Prediction

Abstract

User simulation must reproduce behavior on interfaces that mix discrete choices with open-ended text. Existing language-model simulators generate action strings for both, coupling finite choices to token decoding. We introduce CADET (Calibrated Decisions over Types), a hybrid simulator that matches its computation to the interaction. A pre-trained LM encoder feeds a typed decision head that predicts joint operation-subtype probabilities in one forward pass. An element selector resolves the target, and the predicted operation gates a language generator for text-bearing actions. Proper scoring rules train these probabilities to recover behavioral distributions beyond the most likely action. We evaluate search and category-based browse on two e-commerce benchmarks. On one benchmark, CADET-Brier reduces expected calibration error by 27–58% across seven frozen backbones relative to the best regularized exact-match baseline, with reductions of 71–79% across four LoRA-adapted backbones. On the second benchmark, LoRA CADET achieves 46.7% leaf accuracy, 9.7 percentage points above the majority baseline.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
scoring rules, calibration, uncertainty quantification, reward design, reinforcement learning for language models, structured prediction, type safety, next-action prediction, user behavior modeling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LdwNOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CADET, a framework for predicting human actions in e-commerce setting. It replaces autoregressive action generation with a classification head on a pretrained LLM backbone and uses log or Brier loss to learn calibrated action probabilities. Experiments on OPeRA and ShopCART show improved calibration over exact-match reward optimization, but inconsistent accuracy gains and weaker full-action prediction than autoregressive SFT. The authors conclude that direct classification with proper scoring rules offers a more calibrated approach to user simulation.

Critical strengths

N/A

Critical weaknesses
  • The paper is formatted and written poorly, see Table 2.
  • The related work section discuss about computer use agents, but the actual paper is about user action simulation. It's unclear what is the motivation for the work.
  • The paper lacks novelty, as it essentially trains a LLM backbone with classification head.
  • The paper claims that exact match loss can't learn probability distribution, and proposed using cross entropy or Brier. This is a well known knowledge in machine learning.
Questions for authors

N/A

1 comment
AuthorsOpenReview ↗

Response to "Reviewer LdwN"

We thank the reviewer and respond point by point. We have uploaded a revised version addressing each point.

1. Motivation and related work (agents vs. simulators). The submitted Related Work already discusses shopping simulators (Customer-R1, Shop-R1) and OPeRA; we agree the motivation was under-stated. User simulators are used to evaluate site, search, and ranking changes offline, pre-screen A/B tests, and generate synthetic sessions. These uses need how often people take each action, not only the most likely one: a mode-seeking simulator always takes the modal action and essentially never simulates rare behavior such as abandonment.

This is where simulation differs from the computer-use agents we cited (Mind2Web, WebLINX), which we cited only for their shared action format. An agent should output the best action, so mode-seeking is fine; a simulator must match the human distribution. Shopping simulators (Customer-R1, Shop-R1) inherit agent-style training that rewards agreement with the recorded action; we study what this discards. Revision: The introduction now opens with these use cases; Related Work now leads with user simulation and states this contrast.

2. "That exact match cannot learn a distribution is well known." We agree. We cite Gneiting & Raftery and present Proposition 1 as a specialization, not a new theorem. Our contribution is its consequence for simulators trained with exact-match rewards:

  • Regularization adds a weight to tune; proper scores do not. Entropy- and KL-regularized exact-match objectives identify the distribution only through an inverse link (App. A.3), so their native probabilities depend on the regularizer weight; proper scores have none. With LoRA on Gemma-3-1B at the default weight (0.10), CADET (log) reaches 47.01% leaf accuracy against 46.86% for entropy-regularized exact-match, with ECE 0.064 versus 0.167 (Table 4). At a tuned weight, chosen post hoc on the test set, regularized exact-match attains lower native ECE than CADET in the cases we tested, and native CADET can still need recalibration (see our response to Reviewer tUoS).

  • Rare actions are lost during optimization. Proposition 2 shows exact-match gradients vanish on low-probability labels. With LoRA on four backbones (Appendix Table 15), exact-match assigns 0.0001-0.0026x the true termination mass, its KL- and entropy-regularized variants (weight 0.10) 0.17-0.58x, and CADET (log and Brier) 0.75-0.93x (ideal 1.0). Termination recall is zero for every arm.

  • Equal accuracy is expected. Corollary 1 shows both objectives reach Bayes accuracy, so differences appear in the probabilities.

  • Breadth. At the default weight, the native ECE gain over the best regularized baseline holds on all 7 frozen (-27-58%) and all 4 LoRA backbones (-71-79%) across Gemma, Llama, and ModernBERT.

3. "Just a classification head on an LLM." The architecture is intentionally simple. The contribution is changing the simulator's interface from generating action strings to a typed decision. This (i) yields a valid type-subtype action by construction in one forward pass; (ii) gives an explicit, sampleable probability for every action; and (iii) calls a text generator only when an action needs text.

We do not claim CADET beats autoregressive SFT on full actions (27.6% vs. 17.4%, scored under different output contracts). Target-element selection, not the typed decision, is the bottleneck: always-click with the same selector reaches 16.9%. The text branch is a modular interface, not an accuracy gain: under identical routing, copying the previous query matches generation (17.41% vs. 17.37%). Our experiments evaluate single-step forecasting of recorded human actions; the abstract and conclusion now state that multi-step simulation fidelity is untested.

4. Presentation and Table 2. We agree. We have: (a) split Tables 1 and 2 into separate floats with self-contained captions, no blank cells, and protocol notes moved to the appendix; (b) moved the separately fitted browse heads, whose setup differs, from Table 2 to the appendix; (c) defined "Public decision model API," "Luna," and "leaf" before first use; (d) fixed caption errors (e.g., Table 3); and (e) added an appendix roadmap (appendix letters are unchanged, so existing references hold).

5. Soundness. No specific soundness issue was listed, so we summarize the evidence. The propositions have proofs and numerical checks (App. A). Table 4 fixes each backbone's configuration and varies only the loss; its ECE values reproduce when recomputed from saved per-seed outputs. Table 3's ShopCART panel re-scores exactly from saved per-seed probabilities; its OPeRA panel is provisional, because one of five seeds' predictions survives (see our response to Reviewer qseM). With a known ground truth and Bayes accuracy held fixed, CADET halves the distance to the true distribution (mean total variation 0.33 versus 0.66 for bare exact-match).

Reviewer_qseMOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies next-action prediction for web shopping simulators. Instead of generating each action as text, it predicts the action type with a classification head on a pretrained language-model encoder. A separate selector picks the target on the page. A frozen language model is called only when text must be typed. The paper argues that the classification head should be trained with proper scoring rules (log or Brier loss) instead of an exact-match reward. Experiments on two shopping benchmarks report similar accuracy and better calibration for proper scoring rules.

Critical strengths
  • The paper clearly separates two goals of a simulator: predicting the most likely action, and reporting how often each action occurs.
  • The comparison across encoders is broad. The calibration advantage of proper scoring rules holds on all of them.
Critical weaknesses
  • The paper does not offer a new method or insight. Its main finding is that log or Brier loss gives better-calibrated probabilities than an exact-match objective. This follows from standard results on proper scoring rules, which the paper cites for its first proposition. The method itself is a standard classification head on a pretrained encoder, trained with cross-entropy.
  • The typed head does not improve the end task. With the same element selector, always predicting a click gives about the same full-action accuracy as CADET. Adding generated search text changes full-action accuracy by about half a point.
  • The main probability results cannot be checked. The authors state that the trained-head ECE, NLL, and AUROC values are “provisional diagnostics retained from the original summaries,” and that their definitions remain unverified (Appendix P). The reproducibility statement says the code, splits, configurations, and per-example predictions needed to reproduce these tables are missing.
Questions for authors

Can you release the scorer and per-example predictions behind Table 3, so that its ECE, NLL, and AUROC values can be recomputed?

1 comment
AuthorsOpenReview ↗

Response to "Reviewer qseM"

W1. "The paper does not offer a new method or insight."

We agree that log and Brier losses and the classification head are standard; the paper claims neither a new loss nor a new architecture. We disagree that it offers no insight: its contribution is at the level of the application. For a user simulator whose actions are sampled, the predicted distribution, not its mode, is the output. We show what an exact-match reward does to that distribution and measure it on eleven backbones. It also introduces category-choice browse tasks with constructed category labels.

An exact-match action reward (Customer-R1 additionally weights it by action difficulty) is maximized by a point mass on one action, so for the unweighted objective top-1 accuracy is already at the Bayes rate (Proposition 1, Corollary 1) and does not show whether the predicted probabilities are right.

With LoRA on four backbones (Appendix Table 15), exact-match assigns 0.0001-0.0026x the true termination mass, its KL- and entropy-regularized variants (weight 0.10) 0.17-0.58x, and CADET 0.75-0.93x; a CADET arm has the highest mean leaf accuracy on all four. Termination recall is zero for every arm. Across eleven backbones (Table 4), proper scores give the lowest native ECE at the default regularizer weight (0.10). At a tuned weight, chosen post hoc on the test set, regularized exact-match attains lower native ECE than CADET in the cases we tested, and native CADET can still need recalibration (see our response to Reviewer tUoS); proper scores have no regularizer weight to tune.

We evaluate single-step forecasting of recorded human actions, not sampled multi-step behavior, and the revised paper says so.

W2. "This follows from standard results on proper scoring rules, which the paper cites for its first proposition."

Agreed for Proposition 1, which the paper presents as a finite-action specialization of the cited result. Proposition 2 is a separate, optimization-level statement that this citation does not cover: in the low-probability limit, the exact-match logit-gradient norm is Theta(pi(a*)) and vanishes, while the log score's gradient norm is bounded below by 1 - pi(a*). In the paper's synthetic study with a known conditional and Bayes accuracy held fixed, the mean total-variation distance to truth is 0.3294 for CADET and 0.6575 for bare exact-match. "Type-safe" means the head cannot emit an out-of-schema type-subtype leaf (not targets, text or complete actions), and the revised paper defines it that way.

W3. "The typed head does not improve the end task."

We do not claim it does: the paper's claim for the typed head concerns probability recovery, calibration and rare-action learning. In Table 1, always-click with the same element selector gives 16.93%, CADET (log) 16.85+/-0.29%, and the hybrid with generated text 17.37+/-0.23%. Under identical routing and element selection, copying the previous query gives 17.406% versus generation's 17.369% (difference -0.037 points, 95% CI [-0.092, 0.000]), so the revised paper calls query generation exploratory.

W4. "The main probability results cannot be checked."

Appendix P and the Reproducibility Statement understated what was retained; we correct both. (1) Table 4, the source of the abstract's ECE reductions, has per-seed scorer outputs for every cell; recomputing from them reproduces the abstract's 27-58% (frozen) and 71-79% (LoRA), CADET-Brier vs. the best regularized baseline at weight 0.10. (2) Table 3, OPeRA panel (provisional): the script and configurations were retained, with the per-example output of one of five seeds. Its 902 gold labels match the benchmark cohort sorted by user and time exactly, which is strong evidence of row identity, not proof. Re-scored with 15 bins it gives CADET-log 47.78% accuracy, ECE .078 and NLL 1.64, against reported five-seed means of 47.52%, .0845 and 1.67. One array does not verify the means, so the panel and the text that depends on it are provisional. (3) Table 3, ShopCART panel: per-seed probabilities for every arm were retained and bind to the benchmark packet by row order; re-scoring reproduces every accuracy and 15-bin ECE exactly. The table's NLL is floored at 1e-12 (exact-match's raw NLL is infinite in two of three seeds); the revised paper reports both, states the ECE bins, and adds AUROC.

Q1. "Can you release the scorer and per-example predictions behind Table 3, so that its ECE, NLL, and AUROC values can be recomputed?"

Not during review: the code repository cannot be released at this stage. What exists today: the scorer and metric code; training scripts and configurations for both Table 3 panels; per-seed metrics for every Table 4 cell; and per-example probabilities where retained (Table 4's four LoRA backbones, the ShopCART panel, and one OPeRA seed). The revised Reproducibility Statement and Appendix P record what is retained, that the OPeRA panel is provisional, and each metric's definition.

Reviewer_tUoSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CADET, a hybrid framework for user action prediction that combines a typed decision head, an element selector, and an on-demand text generator. The decision head is trained with logarithmic or Brier scoring rules to estimate behavioral probabilities. The paper analyzes why sampled exact-match rewards favor modal actions rather than recover the full conditional distribution. Experiments on synthetic data and two e-commerce benchmarks show improved probability quality across several frozen and LoRA-adapted backbones, although improvements in complete-action prediction remain limited.

Critical strengths
  1. The paper provides a useful explanation of why predicting the most likely action and recovering behavioral probabilities are different objectives. Its analysis also distinguishes distribution identification from optimization difficulties associated with low-probability labels.

  2. The objective comparisons hold the architecture and representations fixed within each setting. The consistent improvements in probability metrics across seven frozen and four LoRA-adapted backbones strengthen the empirical findings. Synthetic experiments with known ground-truth distributions further support the analysis.

  3. Separating finite action decisions from open-ended text generation is a sensible design for heterogeneous interaction interfaces. The typed head ensures valid type–subtype outputs, while the generator is invoked only when textual input is required.

Critical weaknesses
  1. The approach primarily combines established proper scoring rules, pretrained encoders, and classification heads. Its methodological contribution beyond this combination appears incremental. Moreover, the empirical evaluation is restricted to e-commerce, leaving its applicability to broader interaction tasks insufficiently demonstrated.

  2. The evaluation mainly concerns single-step predictions conditioned on recorded human trajectories, without validating behavioral fidelity in autonomous multi-step interactions. Even for complete-action prediction, adding text generation improves accuracy only from 16.85% to 17.37%, while always predicting a click with the same element selector achieves 16.93%. These results do not yet establish that improved action probabilities yield substantial and reliable improvements in complete user simulation.

  3. Appendix P acknowledges that aspects of the ECE and NLL computations in the original trained-head study remain unverified, while test-instance identifiers and per-example predictions are unavailable. Since probability quality is central to the paper’s contribution, these gaps weaken confidence in the affected results. The corresponding evaluation implementations and records should be provided, and the metrics recomputed.

Questions for authors

How much of CADET’s calibration advantage remains after applying temperature scaling to the strongest entropy- and KL-regularized baselines using the same held-out calibration data?

Could the hybrid model be compared against copying the previous query under identical predicted routing and element selection? Reporting paired confidence intervals for the difference in complete-action accuracy would help isolate the contribution of text generation.

1 comment
AuthorsOpenReview ↗

Response to Reviewer tUoS

We thank the reviewer for a careful review and for proposing two concrete controls; we ran both (points 2 and 3). We have uploaded a revised PDF.

1. Contribution and scope. We agree that log and Brier losses and the classification head are standard; the paper claims neither a new loss nor a new architecture. We respectfully disagree that the contribution is only incremental: it is at the level of the application. For a user simulator whose actions are sampled, the predicted distribution, not its mode, is the output. We show what an exact-match reward does to that distribution and measure it on eleven backbones.

We also agree with the limits the reviewer identifies. The seven frozen and four LoRA backbones broaden encoder coverage, not domain coverage. Both datasets are about shopping, and we do not claim transfer beyond commerce. All experiments are single-step predictions on recorded human states, so they do not show that better probabilities improve multi-step simulation. Type safety covers only the type-subtype pair, not targets or text.

2. Previous-query copy control. On the same 902 OPeRA decisions, we fixed the saved type and routing predictions and used the same three element-selector seeds for both text methods. No text, generation, and copying the previous query reach 16.85%, 17.37% and 17.41% complete-action accuracy. Averaging paired row differences over the three seeds and taking 2,000 user-cluster bootstrap resamples gives generation minus copy of -0.04 points (95% interval [-0.09, 0.00]). The interval is conditional on the saved upstream models. Generation shows no advantage over copying. We have described the text branch as an exploratory interface, not a source of accuracy, and now report this control next to the hybrid result.

3. Temperature scaling of regularized baselines. We ran the requested control on frozen OPeRA for six of seven backbones (3 seeds; Gemma-3-12B was dropped for time). Every head, CADET included, was trained without 10 training users (349 decisions), and one temperature per head was fitted on those users. Baselines were entropy and KL at our weight 0.10 and a tuned 0.3; the best baseline and the better CADET arm were both selected post hoc on the test set. After scaling, ECE no longer separates the objectives: best baseline versus best CADET is 0.088/0.105 (Gemma-1B), 0.106/0.102 (Gemma-4B), 0.065/0.084 (Llama-1B), 0.102/0.099 (Llama-3B), 0.083/0.075 (ModernBERT-base) and 0.059/0.070 (ModernBERT-large), and every user-cluster bootstrap interval includes zero. Held-out NLL still favors proper scoring on all six (2.013, 1.995, 2.014, 2.003, 1.887, 1.887 vs. 2.056, 2.058, 2.099, 2.043, 1.994, 1.975), with intervals excluding zero on five (Llama-3B: -0.040 [-0.104, 0.002]). Brier stays within 0.013 and accuracy is unchanged. Without scaling, tuning the entropy weight to 0.3 post hoc on test data gives lower native ECE than CADET on the causal backbones, where CADET is overconfident; proper scores have no such weight to tune. The abstract's ECE reductions are therefore native comparisons at the default weight; the revised paper says so and adds this table (Appendix Table 16).

4. Metric provenance. We agree this is central. Our earlier statement that code and predictions were unavailable was too broad. The evidence differs by table (details in our response to Reviewer qseM). Table 4 comes from retained per-seed scorer outputs for every cell, and all 60 LoRA cells re-score exactly from saved probabilities. The ShopCART panel of Table 3 re-scores exactly from retained per-seed probabilities for every arm. For the five-seed OPeRA panel of Table 3, the training script, selected configurations and one seed's per-example probabilities survive. Rescoring that array gives CADET (log) 47.78% accuracy, 15-bin ECE 0.078 and raw NLL 1.64, against entropy-regularized exact-match's 47.67%, 0.231 and 2.07. Its 902 gold labels match the benchmark cohort sorted by user and time, which is strong evidence of row identity but not proof; one seed does not verify five-seed means. We therefore label that panel provisional, and state the ECE bins, raw and floored NLL, and AUROC definition in the Table 3 and 4 captions and Appendix P.

5. Revisions. We have revised the paper to (i) state in the introduction that we forecast recorded human actions, not autonomous behavior; (ii) add the copy control and call query generation exploratory; (iii) restrict "type-safe" to the type-subtype pair; (iv) add the matched-recalibration table and qualify the abstract's ECE reductions as native, default-weight comparisons without recalibration; and (v) list multi-step fidelity and non-commerce transfer as untested.

We hope these clarifications and revisions address the reviewer's concerns.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.