acceptodds
Under review as a conference paper at ICLR 2027

ConsumerGym: Training LLM Agents for Real-World Conversational Commerce via Multi-Turn Reinforcement Learning

Abstract

Conversational commerce requires agents to track user needs that emerge and change during dialogue, making both environment design and credit assignment important for policy learning. We introduce \CG, a unified training and evaluation environment combining controllable user-state evolution, catalog-grounded tools, decomposed rewards, and replayable interaction snapshots. The environment separates latent user requirements, revealed oracle states, and agent-maintained states without exposing oracle inputs to the policy. We further propose \RG, which combines failure-aware trajectory resampling with user-state graph and boundary-critic estimation, context-matched counterfactual comparisons, and credit assignment to state updates, plans, tool actions, and responses. Evaluations on two internal protocols and three external benchmarks cover Qwen3.5-4B, 9B, and 27B. Relative to the strongest evaluated non-RUS baseline at each scale, the PPO-based variant improves fixed-context Avg. P@1 by 4.8–5.4 points and ConsumerGym success by 3.3–6.1 percentage points. Improvements with a GRPO-based implementation show that the gains are not confined to the primary PPO configuration.

Primary area:
datasets and benchmarks
Keywords:
LLM, Conversational Commerce, Conversational Recommender Systems

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_2kiqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ConsumerGym is an executable environment for conversational commerce across shopping, food ordering, and travel/hospitality. User requirements can be gradually disclosed, revised, or switched during a conversation. The environment separates hidden user requirements, disclosed information, and the agent's own state, and provides decomposed rewards and restorable interaction snapshots. The authors report 30,000 training scenarios and 200,000 SFT trajectories. They also propose RUS-GPO, which combines failure-aware sampling, graph/critic-based estimation, selective replay, and credit assignment across state, plan, action, and response units. Evaluation covers three Qwen3.5 model sizes, internal and external benchmarks, environment-quality audits, and PPO controls for the environment design.

Critical strengths
  1. The environment is the strongest part of the paper. Separating hidden user requirements from disclosed information and the agent's own state makes the supervision better suited to conversations where requirements change. The authors also evaluate the environment independently of RUS-GPO. With ordinary PPO and all RUS enhancements disabled, the full environment achieves 37.8% success, compared with 33.8% for free-form role-play. The execution checks and human evaluation further support the environment's quality.

  2. The evaluation covers three model sizes, both PPO and GRPO implementations, and external commerce/service benchmarks. The external results are particularly useful because they show that the training gains extend beyond ConsumerGym's own evaluation.

Critical weaknesses
  1. I could not determine how the dynamic credit allocator R(E) is actually computed. Section 4.4 and Appendix D.5 describe constraints on its output, while D.9 leaves its inputs and implementation unspecified. The relevant ablation also removes routing and unit-balanced loss together, so it does not tell us how much dynamic allocation contributes. This makes it difficult to assess or reproduce this part of the method.

  2. Counterfactual replay provides a modest additional gain in task success, removing it reduces success from 44.10% to 43.57%. It also requires additional interactions, and the no-replay variant does not use the saved budget for more actor rollouts. The current comparison therefore leaves open whether replay is a better use of the available interaction budget. The authors should explain when replay is most useful and whether its gains justify the added cost.

Questions for authors
  1. How is R(E) computed? Please specify its inputs and allocation rule, and clarify what evidence separates dynamic routing from a fixed allocation with the same unit-balanced loss.

  2. Does the replay advantage remain when the no-replay learner uses the saved interaction budget for additional actor rollouts? Given the modest task-success increment, when is replay worth its added cost?

1 comment

For Weakness 1:
Thank you for identifying the need for a more explicit description of the responsibility allocator.

Implementation of R(E). We clarify that R(E) is implemented using GPT-5.5 as a responsibility-assessment judge, through a call separate from reward scoring. Its input E(i,t) is the interaction prefix preceding the current assistant turn. For each of the 12 reward components, the judge produces a separate four-dimensional numerical weight vector over State, Plan, Action, and Response, resulting in a 12 × 4 responsibility matrix.

The judge is instructed to produce non-negative weights that sum to one within each row. The implementation additionally validates and normalizes the output. For decision units without valid actor tokens in the current turn, the corresponding weights are set to zero, and the remaining active-unit weights are renormalized. Malformed outputs, negative weights, and all-zero rows trigger re-scoring rather than being accepted.

The resulting matrix M(i,t) satisfies:

M(i,t,k,d) >= 0,

sum(d in D+) M(i,t,k,d) = 1.

The component-level advantages are subsequently routed to the corresponding decision units according to Eq. (37):

A(i,t,d) = sum(k in K) w(k) M(i,t,k,d) A_tilde(i,t,k).

Rationale for dynamic allocation. Unlike a fixed global responsibility mapping, the judge produces context-conditioned weights based on the interaction prefix. This allows the same reward component to assign different responsibility across interactions. For example, the responsibility associated with task success may be distributed differently among state tracking, tool actions, and response generation depending on the preceding interaction context.

Additional supervision and validation scope. We acknowledge that judge-generated responsibility weights provide additional training supervision beyond the reward scores themselves. The separate GPT-5.5 call also incurs additional inference cost. Thus, our method is more accurately characterized as judge-assisted structured credit assignment, rather than a purely estimator-level modification under identical supervision.

Regarding the ablation, our existing w/o Structured Unit Credit variant jointly removes dynamic responsibility allocation and unit-balanced optimization. As reported in Table 8, removing this combined module decreases Session from 0.547 to 0.502 and ConsumerGym task success from 44.10% to 40.82%.

For Weakness 2:

Interaction-budget clarification. In our current ablation, the actor rollout budget is kept identical between the full RUS-GPO configuration and the no-replay variant. However, the environment interactions saved by removing counterfactual replay are not reallocated to additional actor rollouts. Therefore, this comparison measures the marginal contribution of replay under a fixed actor rollout budget, rather than comparing replay against additional on-policy rollouts under an identical total environment-interaction budget.

Existing empirical evidence. As reported in Table 8, removing counterfactual replay decreases ConsumerGym task success from 44.10% to 43.57%, corresponding to a 0.53-percentage-point difference. The impact is also observed on external benchmarks: ShopSimulator Rsucc decreases from 0.530 to 0.514, while VitaBench In-store decreases from 66.5 to 62.3 and OTA decreases from 54.8 to 52.4, as reported in Table 9. These results indicate that counterfactual replay provides additional gains under the current training configuration, although the improvement in overall task success is modest.

Reviewer_RqDyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper targets multi-turn RL for conversational-commerce agents whose users reveal, revise, or switch requirements during the dialogue.

ConsumerGym is a simulated environment with:

  • GPT-5.5-generated scenarios and sandboxed tools;
  • a two-layer user simulator: a rule+LLM controller selects user events, which Kimi K2.6 realizes as utterances;
  • three state views (latent, revealed oracle srevs^{rev}, agent estimate);
  • a 12-component reward, mostly judged by GPT-5.5;
  • turn-level snapshot/replay.

RUS-GPO combines failure-aware scenario resampling, per-component advantages from an srevs^{rev}-keyed state graph fused with a 12-head privileged critic, selective restore-and-branch comparisons, and a responsibility matrix R(E)R(E) that routes credit to the state, plan, action, and response tokens.

Results: on Qwen3.5-4B/9B/27B, RUS-GPO improves fixed-context P@1 by 4.8–5.4 and success by 3.3–6.1 pp over the best baseline (PPO, GRPO, GiGPO, HGPO, G2PO, ReBel), and also leads on three external benchmarks.

Critical strengths

S1. The environment design is working. Table 11 fixes the learner (plain PPO) and varies only the environment. Prompt-only role-play, static goals, and terminal-only reward all underperform the full environment, including on external benchmarks. This is the right evidence for an environment contribution.

S2. Performance gains hold on external benchmarks with their native scoring (e.g., ShopSimulator RsuccR_{succ} 0.60→0.71 at 27B), across three model scales and two optimizers.

Critical weaknesses
  • Environment: no code is released, and the simulator is described only at the interface level. It is not clear whether LLM drives the controller, who checks the utterances, or how the user decides to reject or accept.

  • Benchmark: the scenarios are "generated with GPT-5.5", with no information on catalog provenance, diversity, or construction of the fixed-context set. The test split is in-distribution. If ConsumerGym is to be released as a benchmark-style Gym with scenarios, it should clearly describe the scenario diversity, fidelity and quality etc.

  • Algorithm:

    • The novel routing R(E)R(E) is undefined.
    • The novel parts depend on ConsumerGym-only interfaces (12 reward components, 4 units), and the method is not tested on ALFWorld or WebShop, where its baselines report results.
  • Reward validity: I was only able to find 3 of the 12 scorers specified in the paper. Human agreement is reported only for the averages of the turn-level and session-level components (Spearman ρ), not per-component, and with no inter-annotator agreement. There is no test of judge stability, of length bias, or of agreement with another judge.

  • Circularity: internal evaluation uses the same rubric, the same GPT-5.5 judge, and the same Kimi simulator as training.

Questions for authors
  1. Please report inter-annotator agreement.
  2. How stable is the judge under repeated scoring, and does it agree with a non-GPT judge?
  3. Do the internal gains hold with a different judge and user simulator than those used in training?
  4. Please define R(Ei,t)R(E_{i,t}) and Ei,tE_{i,t}. Is RR learned, rule-based, or LLM-based, and how is it validated?
  5. What is the schema of qstateq^{state}: fixed slots or free-form? Does it match srevs^{rev}? How accurate is qstateq^{state} against srevs^{rev} before and after RL?
  6. What is the critic's architecture? Is it a separate LLM or does it share the actor's backbone, and what is its size?
  7. Simulator: which LLM drives FstateF_{state}, and who verifies the realized utterances?
  8. What are the scenario and catalog sources and the diversity statistics?
  9. Can you evaluate frontier models on the benchmark?
  10. External benchmarks: at inference, does the policy still emit the state and plan segments, and how is it adapted to each benchmark's tool and interaction format? Did you check for overlap between the GPT-5.5/Gemini-generated training data and the external test sets?
3 comments

For Weakness 1:
Thank you for raising the important question of human annotation reliability. We use three independent annotators (A, B, and C) to evaluate the same audit samples and calculate inter-annotator agreement (IAA) by averaging the pairwise Spearman correlations across the three annotator pairs (A–B, A–C, and B–C).

Specifically, the Turn-level and Session-level IAA scores are:

DomainTurn-level IAASession-level IAA
E-commerce0.830.87
Food Delivery0.850.88
Travel & Hospitality0.810.80

The IAA is calculated as:

IAA=ρA,B+ρA,C+ρB,C3.\mathrm{IAA}=\frac{\rho_{A,B}+\rho_{A,C}+\rho_{B,C}}{3}.

These results indicate consistent rank agreement among the three annotators when assessing interaction quality across the three domains.

Additionally, Appendix C.2 reports Spearman correlations between automatic evaluator scores and mean human judgments. Specifically, the Turn-level correlations range from 0.88 to 0.91, while the Session-level correlations range from 0.83 to 0.97. These two sets of results measure different aspects: agreement among human annotators and agreement between automatic evaluation and human judgments, respectively.

For Weakness 2:
Our reward-evaluation framework primarily uses GPT-5.5 for semantic assessment, complemented by deterministic verifiers for executable properties. Specifically, factuality and price accuracy combine GPT-5.5 judgments with deterministic checks; state tracking is assessed by GPT-5.5 at the session boundary; and task success combines execution evidence with judge assessment. The remaining reward components are also evaluated by GPT-5.5. Regarding length bias, we conducted paired checks comparing longer and shorter responses while keeping their underlying content equivalent. We did not observe an obvious length bias in these checks.

For Weakness 3:
Thank you for identifying the underspecified implementation of the responsibility allocator. We clarify that R(E) uses GPT-5.5 as a responsibility-assessment judge, generating numerical responsibility weights through a call separate from reward scoring.

Specifically, E_{i,t} is the interaction prefix preceding the current assistant turn. For each of the 12 reward components, the judge produces a separate four-dimensional numerical weight vector over State, Plan, Action, and Response, rather than using a single shared four-dimensional vector across all components. The resulting output corresponds to a 12 × 4 responsibility matrix.

The judge is instructed to produce non-negative weights that sum to one within each row. The implementation additionally validates the output format and numerical values and performs normalization. For decision units without valid actor tokens in the current turn, such as an Action unit when no tool call occurs, the corresponding weights are set to zero, and the remaining active-unit weights are renormalized. Malformed outputs, negative weights, and all-zero rows trigger re-scoring rather than being accepted.

The resulting matrix M_{i,t} satisfies:

[ M_{i,t}^{k,d}\geq 0,\qquad \sum_{d\in D_{i,t}^{+}}M_{i,t}^{k,d}=1. ]

Component-level advantages are then converted into decision-unit-level advantages according to Eq. (37):

\sum_{k\in K} w_k M_{i,t}^{k,d}\widetilde A_{i,t}^{k}. ]

For Weakness 4:

ConsumerGym explicitly distinguishes the Latent User State, Revealed Oracle State, and Agent-maintained State.

The actor records Agent State using JSON with fixed fields, and Agent State shares the same field definitions as Revealed Oracle State, allowing direct field-by-field comparison. Their main difference lies in their information sources: Agent State is generated by the policy from visible interaction history and tool feedback, whereas Revealed Oracle State is maintained by the environment for training supervision and failure diagnosis.

Regarding state accuracy, two existing measurements should be distinguished.

First, the oracle-state slot F1 scores reported in Table 6 of Appendix C.2 are 92.4%, 93.1%, and 95.7%, respectively. These measure the agreement between the environment-maintained Revealed Oracle State and its reference state, rather than the prediction accuracy of Agent State.

Second, the State diagnostic scores in Table 1 show consistent improvements of RUS-GPO (PPO) over SFT across all three model scales:

  • Qwen3.5-4B: 0.473 → 0.611
  • Qwen3.5-9B: 0.524 → 0.650
  • Qwen3.5-27B: 0.578 → 0.740

These results support improvements in state-related decision quality, but they are rubric-based diagnostic scores rather than direct slot-F1 measurements between Agent State and Revealed Oracle State.

For Weakness 5:
Our critic uses a model from the same backbone family and of the same scale as its corresponding actor. Specifically, the Qwen3.5-4B, 9B, and 27B actors are paired with critics of the corresponding sizes. The actor and critic are independent model copies with separately updated parameters; they do not share parameters. The critic uses a linear value projection to output 12 scalar values, corresponding to the 12 reward components. The critic's forward input includes the interaction prefix preceding the current decision boundary, together with training-side information such as scenario and user-state information. Rewards and returns are used only as training targets, not as forward-input features. This is consistent with Appendix D.3, which excludes future realized rewards, future tool outcomes, and counterfactual returns from critic inputs. Furthermore, critic-based baselines have access to the same interaction-prefix and training-side scenario/user-state information. RUS-GPO therefore does not have exclusive access to these additional critic inputs.

For Weakness 6:
ConsumerGym separates user-state event decisions from natural-language realization into two stages. First, F_state uses Kimi K2.6 together with rule-based control to determine the next user event based on the latent session plan, current revealed state, and assistant decision. Supported events include information disclosure, requirement revision, candidate rejection, and intent switching. Then, F_role also uses Kimi K2.6 to convert the selected structured event into a natural-language user utterance. To ensure consistency between the generated utterance and the intended event, the system performs online verification using both rules and Kimi K2.6. If the utterance fails to express the intended event, it is regenerated, with at most five attempts. Samples that still fail are marked as simulator failures rather than accepted as successful state transitions. Additionally, the simulated user's acceptance or rejection of a candidate is determined by an LLM based on user requirements, interaction history, and the recommendation, rather than simply following a predetermined acceptance/rejection sequence.

Thank you for the clarifications on the simulator, critic, responsibility allocator, state schema, and annotator agreement; these are very helpful. However, several central questions remain unaddressed:

  • External-benchmark transfer and its fairness. The external benchmarks are the only independent evidence that the proposed gym works beyond its own judge system. How is a ConsumerGym-trained checkpoint adapted to the three external benchmarks (e.g., tool-schema mapping, system prompt, and whether the state/plan segments are still emitted)?
  • Data provenance and potential leakage. What sources were used for scenario generation and for the product catalog? This matters because the SFT/RL data was generated by GPT-5.5, so exposure to or leakage from the external benchmarks cannot be ruled out. Please report basic statistics of the scenario data (e.g., diversity and difficulty) and a decontamination analysis of the training data against the three external test sets.
Reviewer_eygVOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ConsumerGym, a training and evaluation environment, and RUS-GPO, a training method that up-samples scenarios where the policy performs poorly and assigns per-turn credit to different decision units.

Critical strengths
  1. ConsumerGym is a strength of the paper. The setup of evolving user requirements is realistic. Separating latent requirements, revealed requirements, and the agent's own state estimate allows direct measurement of how well the agent tracks requirement changes. Table 11 also shows that the environment design itself benefits training.

  2. The experimental setup is comprehensive, and the improvements are consistent across settings.

Critical weaknesses
  1. My main concern is that the comparison with baselines is not controlled under same compute budget. RUS-GPO uses two additional components: failure-aware resampling and counterfactual replay, neither of which the baselines receive. Counterfactual replay adds environment interactions, but the computational cost is not reported in the paper. So it is unclear how much of the improvement comes from the credit assignment itself from current results.

  2. A core component of the method is underspecified. The dynamic responsibility matrix Mi,t=R(Ei,t)M_{i,t} = R(E_{i,t}) is central to the claimed within-turn credit assignment, but Section 4.4 and Appendix D.5 only state its constraints: non-negative entries, row normalization over active units, and recomputation at every turn. The paper does not specify

  • (i) what the evidence Ei,tE_{i,t} contains

  • (ii) the functional form of RR (learned, or rubric-based).

    And if RR relies on an LLM judge, it provides extra supervision that the baselines do not have. In addition, the "w/o structured unit credit" ablation (Appendix E.2) removes dynamic allocation and unit-balanced loss together, and no ablation compares the dynamic matrix with a fixed matrix. The benefit of the "dynamic" allocation is therefore not isolated.

Questions for authors
  1. Could the authors apply the same resampling to the baselines, or compare under matched environment interactions, and report the computational cost of each method?

  2. Please define EE and RR, and clarify whether RR uses an LLM judge.

  3. Could the authors add an ablation comparing a dynamic responsibility matrix with a fixed one?

2 comments

We thank the reviewer for recognizing the environment design, the modeling of evolving user requirements, and the breadth of our evaluation. Below, we clarify the training-budget comparison, the implementation of the responsibility allocator, and the conclusions supported by the existing ablations.

For weakness 1:
We first clarify that the main experiments match the number of ordinary actor-rollout trajectories, rather than the total computation of all auxiliary mechanisms. The RL methods use the same 12 reward components and training weights. For critic-based baselines, the critic also has access to the same interaction-prefix, scenario, and user-state information as in RUS-GPO.

Failure-aware resampling reuses ordinary training rollouts for failure diagnosis and does not generate additional diagnostic trajectories. It changes the scenario sampling distribution while collecting fresh on-policy trajectories, rather than retraining on historical failures. Its primary effect is therefore to change experience allocation under a fixed trajectory count.

Counterfactual replay, however, introduces additional environment interactions, and responsibility allocation uses separate GPT-5.5 calls. We therefore do not interpret the current results as a strictly total-compute-matched comparison or claim computational-efficiency superiority. Appendix E.2 explicitly states that interactions saved by removing replay are not reallocated to additional actor rollouts.

The existing results nevertheless provide a useful distinction: the no-replay variant achieves 63.25 fixed-context P@1 and 43.57% ConsumerGym success, compared with 59.97 and 39.17% for ReBel. Thus, the performance advantage does not depend entirely on additional replay interactions. This does not attribute the remaining advantage exclusively to credit assignment or replace a direct comparison against baselines augmented with the same resampling mechanism.

For Weakness 2:

Thank you for identifying this implementation gap. R(E) uses GPT-5.5 as a responsibility-assessment judge, with a call separate from reward scoring.

Its input E(i,t) is the interaction prefix preceding the current assistant turn. For each of the 12 reward components, the judge produces a separate four-dimensional numerical weight vector over State, Plan, Action, and Response, rather than a single vector shared across components.

The judge is instructed to produce non-negative weights summing to one within each row. The implementation additionally validates and normalizes the output. Weights for units without valid actor tokens in the current turn are set to zero, and the remaining active-unit weights are renormalized. Invalid outputs, including malformed responses, negative weights, and all-zero rows, trigger re-scoring.

The resulting matrix M(i,t) belongs to R(12×4) and routes component-level advantages to decision units according to Eq. (37):

A(i,t,d) = Σ(k∈K) w(k) M(i,t,k,d) Ã(i,t,k).

Because the judge receives only the pre-turn prefix, this mechanism provides context-conditioned credit allocation rather than retrospective causal diagnosis of errors in the current turn. It assigns component-specific weights to decision units based on the interaction context, instead of applying the same fixed mapping to every turn.

For Weakness 3:
The structured-unit-credit ablation in Appendix E.2 jointly removes responsibility allocation and unit-balanced optimization. It therefore evaluates the combined module rather than the independent advantage of dynamic routing.

In Table 8, removing this module decreases Session from 0.547 to 0.502 and ConsumerGym success from 44.10% to 40.82%. These results support the overall contribution of structured unit credit under the reported configuration, but do not disentangle dynamic weighting, additional judge supervision, and loss balancing.

Thank you for the clarification. The definitions of E and R address Q2. Q1 and Q3 remain largely unaddressed: the computational cost is still not reported, no baseline is evaluated with the same resampling or under matched environment interactions, and no ablation compares the dynamic matrix with a fixed one. I will therefore maintain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.