ConsumerGym: Training LLM Agents for Real-World Conversational Commerce via Multi-Turn Reinforcement Learning

Conversational commerce requires agents to track user needs that emerge and change during dialogue, making both environment design and credit assignment important for policy learning. We introduce \CG, a unified training and evaluation environment combining controllable user-state evolution, catalog-grounded tools, decomposed rewards, and replayable interaction snapshots. The environment separates latent user requirements, revealed oracle states, and agent-maintained states without exposing oracle inputs to the policy. We further propose \RG, which combines failure-aware trajectory resampling with user-state graph and boundary-critic estimation, context-matched counterfactual comparisons, and credit assignment to state updates, plans, tool actions, and responses. Evaluations on two internal protocols and three external benchmarks cover Qwen3.5-4B, 9B, and 27B. Relative to the strongest evaluated non-RUS baseline at each scale, the PPO-based variant improves fixed-context Avg. P@1 by 4.8–5.4 points and ConsumerGym success by 3.3–6.1 percentage points. Improvements with a GRPO-based implementation show that the gains are not confined to the primary PPO configuration.