- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
ConsumerGym is an executable environment for conversational commerce across shopping, food ordering, and travel/hospitality. User requirements can be gradually disclosed, revised, or switched during a conversation. The environment separates hidden user requirements, disclosed information, and the agent's own state, and provides decomposed rewards and restorable interaction snapshots. The authors report 30,000 training scenarios and 200,000 SFT trajectories. They also propose RUS-GPO, which combines failure-aware sampling, graph/critic-based estimation, selective replay, and credit assignment across state, plan, action, and response units. Evaluation covers three Qwen3.5 model sizes, internal and external benchmarks, environment-quality audits, and PPO controls for the environment design.
Critical strengths
-
The environment is the strongest part of the paper. Separating hidden user requirements from disclosed information and the agent's own state makes the supervision better suited to conversations where requirements change. The authors also evaluate the environment independently of RUS-GPO. With ordinary PPO and all RUS enhancements disabled, the full environment achieves 37.8% success, compared with 33.8% for free-form role-play. The execution checks and human evaluation further support the environment's quality.
-
The evaluation covers three model sizes, both PPO and GRPO implementations, and external commerce/service benchmarks. The external results are particularly useful because they show that the training gains extend beyond ConsumerGym's own evaluation.
Critical weaknesses
-
I could not determine how the dynamic credit allocator R(E) is actually computed. Section 4.4 and Appendix D.5 describe constraints on its output, while D.9 leaves its inputs and implementation unspecified. The relevant ablation also removes routing and unit-balanced loss together, so it does not tell us how much dynamic allocation contributes. This makes it difficult to assess or reproduce this part of the method.
-
Counterfactual replay provides a modest additional gain in task success, removing it reduces success from 44.10% to 43.57%. It also requires additional interactions, and the no-replay variant does not use the saved budget for more actor rollouts. The current comparison therefore leaves open whether replay is a better use of the available interaction budget. The authors should explain when replay is most useful and whether its gains justify the added cost.
Questions for authors
-
How is R(E) computed? Please specify its inputs and allocation rule, and clarify what evidence separates dynamic routing from a fixed allocation with the same unit-balanced loss.
-
Does the replay advantage remain when the no-replay learner uses the saved interaction budget for additional actor rollouts? Given the modest task-success increment, when is replay worth its added cost?
For Weakness 1:
Thank you for identifying the need for a more explicit description of the responsibility allocator.
Implementation of R(E). We clarify that R(E) is implemented using GPT-5.5 as a responsibility-assessment judge, through a call separate from reward scoring. Its input E(i,t) is the interaction prefix preceding the current assistant turn. For each of the 12 reward components, the judge produces a separate four-dimensional numerical weight vector over State, Plan, Action, and Response, resulting in a 12 × 4 responsibility matrix.
The judge is instructed to produce non-negative weights that sum to one within each row. The implementation additionally validates and normalizes the output. For decision units without valid actor tokens in the current turn, the corresponding weights are set to zero, and the remaining active-unit weights are renormalized. Malformed outputs, negative weights, and all-zero rows trigger re-scoring rather than being accepted.
The resulting matrix M(i,t) satisfies:
M(i,t,k,d) >= 0,
sum(d in D+) M(i,t,k,d) = 1.
The component-level advantages are subsequently routed to the corresponding decision units according to Eq. (37):
A(i,t,d) = sum(k in K) w(k) M(i,t,k,d) A_tilde(i,t,k).
Rationale for dynamic allocation. Unlike a fixed global responsibility mapping, the judge produces context-conditioned weights based on the interaction prefix. This allows the same reward component to assign different responsibility across interactions. For example, the responsibility associated with task success may be distributed differently among state tracking, tool actions, and response generation depending on the preceding interaction context.
Additional supervision and validation scope. We acknowledge that judge-generated responsibility weights provide additional training supervision beyond the reward scores themselves. The separate GPT-5.5 call also incurs additional inference cost. Thus, our method is more accurately characterized as judge-assisted structured credit assignment, rather than a purely estimator-level modification under identical supervision.
Regarding the ablation, our existing w/o Structured Unit Credit variant jointly removes dynamic responsibility allocation and unit-balanced optimization. As reported in Table 8, removing this combined module decreases Session from 0.547 to 0.502 and ConsumerGym task success from 44.10% to 40.82%.
For Weakness 2:
Interaction-budget clarification. In our current ablation, the actor rollout budget is kept identical between the full RUS-GPO configuration and the no-replay variant. However, the environment interactions saved by removing counterfactual replay are not reallocated to additional actor rollouts. Therefore, this comparison measures the marginal contribution of replay under a fixed actor rollout budget, rather than comparing replay against additional on-policy rollouts under an identical total environment-interaction budget.
Existing empirical evidence. As reported in Table 8, removing counterfactual replay decreases ConsumerGym task success from 44.10% to 43.57%, corresponding to a 0.53-percentage-point difference. The impact is also observed on external benchmarks: ShopSimulator Rsucc decreases from 0.530 to 0.514, while VitaBench In-store decreases from 66.5 to 62.3 and OTA decreases from 54.8 to 52.4, as reported in Table 9. These results indicate that counterfactual replay provides additional gains under the current training configuration, although the improvement in overall task success is modest.