- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VHD-Play constructs agent training environments from mathematical decision problems that have already been solved. A language-model setter turns each sampled problem into a stateful environment with tools and partially hidden information, while the solver's reference values determine the terminal reward. The pipeline produces 3,300 environments, with 2,200 used for training a Qwen3.6-35B-A3B policy. Evaluation covers held-out instances, eight unseen mechanism families, and external tool-use and planning tasks. A useful part of the study compares the same problem in written-out, fully informed interactive, and partially informed interactive forms. Across five optimization families, training raises the mean partially informed score from 0.204 to 0.815.
Critical strengths
-
The construction gives the reward a concrete basis. Both the environment and its score start from the same sampled decision problem, and the reward endpoints come from a solver rather than a model's judgment of the final trajectory. This makes it possible to inspect what a high score means and generate further instances by changing parameters. The separation between the mechanism and the interface is also useful: the authors can vary what the agent knows without changing the underlying objective. I see this as the main methodological contribution.
-
The informed-agentic experiment is more informative than a simple before/after training comparison. In Table 2, the base model scores 0.962 on written-out problems but only 0.231 when all parameters are still given and it must act through the environment. Training raises the latter to 0.875. This shows that hiding parameters is not the only source of difficulty; carrying a plan through changing states is itself a substantial problem. Gains on unseen families and external tasks provide additional evidence that the training is teaching something useful beyond the sampled training instances.
Critical weaknesses
-
The verification supports a narrower claim than correctness of the generated environment in general. Appendix B audits 80 environments from eight families, using oracle, default, and idle trajectories. Passing these checks is useful, but a generated transition function could agree on these paths and still behave incorrectly after other legal decisions. This matters because Appendix C.2 allows the setter to implement dynamics when a reusable implementation is unavailable. Negotiation is also a training family, but it is outside the listed replay coverage. I would like to know which parts of each family are fixed code, which are generated, and how their agreement with the mathematical mechanism is checked away from the reference trajectories. The reward reference can be correct even when the environment used to earn that reward is not.
-
The broader transfer claims need a clearer account of uncertainty. TravelBench contains 20 tasks, the storefront experiment has five runs per arm, and the generated-family results use one training run and one evaluation seed. The storefront improvement over the base model is large, but the reported mean of 182,844 versus 165,224 for the larger reference model is much less decisive given the dispersion in Figure 4. I would treat that ranking as preliminary. The paper also trains a separate checkpoint at each scale in Figure 5; this supports training at larger scales, but does not establish that one trained policy generalizes to longer horizons. These limits do not erase the main result, but they narrow what the experiments establish.
Questions for authors
-
Please give a family-by-family breakdown of fixed and generated dynamics, including the checks used for negotiation. Have you compared intermediate states and terminal utilities against an independent mechanism implementation on legal action sequences other than the oracle, default, and idle paths?
-
Are the storefront runs paired by simulator seed? Please report the individual outcomes and uncertainty for the differences between models, together with task-level uncertainty for TravelBench and the BFCL subset. Were any generated-family results repeated with a second training seed?
4 comments
W2. Longer horizons
Section 5.1 evaluates the same trained checkpoint on the 365-day storefront. Generated-family episodes have median 31 and mean 59.7 assistant turns, while storefront runs span 1,147 to 1,825 and average 1,359 (Appendices F and H.4). All five finish without bankruptcy. This gives external transfer over roughly the generated-family mean horizon. Figure 5 addresses whether larger mechanisms remain constructible and learnable, using a checkpoint trained at each scale. The storefront results demonstrate transfer of the same trained policy to longer interactions with a different objective and interface.
We thank the reviewer for recognizing the contributions of our mechanism-first approach and experimental findings. We are grateful for the opportunity to clarify the experimental settings and address a potential misunderstanding regarding longer-horizon generalization. We would be happy to provide further clarification.
Thank you again for your constructive review. We hope our responses have helped clarify your remaining concerns, particularly regarding longer-horizon generalization.
We would welcome any further feedback and, if you find our clarifications satisfactory, would greatly appreciate your reconsideration of the assessment.
Thank you for your time and consideration!
W1 & Q1. Fixed and generated components
We thank the reviewer for the valuable comments. The mathematical mechanism and reference are fixed before environment realization in every family. Eight optimization families use sampler-provided dynamics with refinement disabled. For Negotiation, sequential stopping, and sequential search, the setter implements the dynamics from prescribed equations and templates (Appendix C.2). For these three families, we manually checked 30 environments per family to verify that the generated code and its execution followed the prescribed mathematical rules. The checks covered intermediate states and terminal utilities, and we found no departures from the specified mechanisms.
We manually inspected 30 environments per family. This inspection includes Negotiation, sequential stopping, and sequential search, covering their generated dynamics as well as intermediate states and terminal utilities. We checked the code and observed trajectories for incorrect transitions and access to hidden parameters. We found neither mechanism departures nor hidden-parameter access in the 330 inspected environments. The player-facing interface exposes declared observations and actions, while the executable dynamics retain latent parameters and references internally. The separate computation utility cannot access these internals (Appendix C.4). The automatic audit had 480 successful executions with repeatable outcomes and no above-reference utility. Training-time monitoring also found no raw utility above its reference before clipping.
W2 & Q2. Repeated runs and uncertainty
All arms use the benchmark's default simulator setting with five repeated runs. We compare the groups using an unpaired Welch analysis. The plotted ending balances are reconstructed below in thousands. Entries follow plotting order.
The balance gain over Base is about 128.6k, with an approximate 95% interval of [42.0k, 215.1k] and p≈0.009. All five trained-policy runs complete without bankruptcy. Against Max, the observed mean is 10.7% higher and sample variance is 77.5% lower, with a mean-difference interval of [−147.3k,182.5k]. Generated-family results use one evaluation seed and the step-34 training checkpoint.