acceptodds
Under review as a conference paper at ICLR 2027

Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms

Abstract

Language-model agents increasingly face long-horizon tasks with evolving state, interdependent decisions, and delayed outcomes. Scaling their training requires diverse agentic environments, dependable outcome signals, and low extension cost. Existing generation pipelines commonly construct an environment before defining its outcome rule or annotating its trajectories, leaving dynamics and evaluation to be aligned post hoc. VHD-Paly reverses this dependency by sampling and solving a mathematical model before a corpus-grounded setter renders its decision process as stateful tools. The executable dynamics and trajectory-scoring reference are inherited from the same solved model. The pipeline produces 3,300 diverse agentic environments at a cost of a few cents each. Training Qwen3.6-35B-A3B on three families raises its mean agentic score from 0.204 to 0.815 in a five-family diagnostic. Gains also appear on held-out instances from all three training families and eight unseen mechanism families, then extend beyond the generated substrate to external benchmarks for general function calling, travel planning, and 365-day e-commerce. On E-Commerce Bench, the trained checkpoint completes every run without bankruptcy and exceeds Qwen3.7-Max. We compare written-out problems with stateful versions that reveal or hide their parameters. The comparison shows that most of the learnable gap lies in stateful interaction rather than underlying problem solving. A frozen 35B setter realizes larger environments, and scale-matched training retains gains as mechanism size and horizon grow, indicating the potential for an evolving training substrate.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic RL, Env Scaling

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_dTFyOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VHD-Play constructs agent training environments from mathematical decision problems that have already been solved. A language-model setter turns each sampled problem into a stateful environment with tools and partially hidden information, while the solver's reference values determine the terminal reward. The pipeline produces 3,300 environments, with 2,200 used for training a Qwen3.6-35B-A3B policy. Evaluation covers held-out instances, eight unseen mechanism families, and external tool-use and planning tasks. A useful part of the study compares the same problem in written-out, fully informed interactive, and partially informed interactive forms. Across five optimization families, training raises the mean partially informed score from 0.204 to 0.815.

Critical strengths
  1. The construction gives the reward a concrete basis. Both the environment and its score start from the same sampled decision problem, and the reward endpoints come from a solver rather than a model's judgment of the final trajectory. This makes it possible to inspect what a high score means and generate further instances by changing parameters. The separation between the mechanism and the interface is also useful: the authors can vary what the agent knows without changing the underlying objective. I see this as the main methodological contribution.

  2. The informed-agentic experiment is more informative than a simple before/after training comparison. In Table 2, the base model scores 0.962 on written-out problems but only 0.231 when all parameters are still given and it must act through the environment. Training raises the latter to 0.875. This shows that hiding parameters is not the only source of difficulty; carrying a plan through changing states is itself a substantial problem. Gains on unseen families and external tasks provide additional evidence that the training is teaching something useful beyond the sampled training instances.

Critical weaknesses
  1. The verification supports a narrower claim than correctness of the generated environment in general. Appendix B audits 80 environments from eight families, using oracle, default, and idle trajectories. Passing these checks is useful, but a generated transition function could agree on these paths and still behave incorrectly after other legal decisions. This matters because Appendix C.2 allows the setter to implement dynamics when a reusable implementation is unavailable. Negotiation is also a training family, but it is outside the listed replay coverage. I would like to know which parts of each family are fixed code, which are generated, and how their agreement with the mathematical mechanism is checked away from the reference trajectories. The reward reference can be correct even when the environment used to earn that reward is not.

  2. The broader transfer claims need a clearer account of uncertainty. TravelBench contains 20 tasks, the storefront experiment has five runs per arm, and the generated-family results use one training run and one evaluation seed. The storefront improvement over the base model is large, but the reported mean of 182,844 versus 165,224 for the larger reference model is much less decisive given the dispersion in Figure 4. I would treat that ranking as preliminary. The paper also trains a separate checkpoint at each scale in Figure 5; this supports training at larger scales, but does not establish that one trained policy generalizes to longer horizons. These limits do not erase the main result, but they narrow what the experiments establish.

Questions for authors
  1. Please give a family-by-family breakdown of fixed and generated dynamics, including the checks used for negotiation. Have you compared intermediate states and terminal utilities against an independent mechanism implementation on legal action sequences other than the oracle, default, and idle paths?

  2. Are the storefront runs paired by simulator seed? Please report the individual outcomes and uncertainty for the differences between models, together with task-level uncertainty for TravelBench and the BFCL subset. Were any generated-family results repeated with a second training seed?

4 comments

W1 & Q1. Fixed and generated components

We thank the reviewer for the valuable comments. The mathematical mechanism and reference are fixed before environment realization in every family. Eight optimization families use sampler-provided dynamics with refinement disabled. For Negotiation, sequential stopping, and sequential search, the setter implements the dynamics from prescribed equations and templates (Appendix C.2). For these three families, we manually checked 30 environments per family to verify that the generated code and its execution followed the prescribed mathematical rules. The checks covered intermediate states and terminal utilities, and we found no departures from the specified mechanisms.

FamilyDynamics realizationAppendix B replayManual inspection
Inventory DPInjected codeIncluded30 environments
RoutingInjected codeIncluded30 environments
KnapsackInjected codeIncluded30 environments
Linear programmingInjected codeIncluded30 environments
Quadratic programmingInjected codeIncluded30 environments
Facility locationInjected codeIncluded30 environments
SchedulingInjected codeIncluded30 environments
Scheduling (tardiness)Injected codeIncluded30 environments
NegotiationSetter implementation of prescribed counterpart equations and session templateNot included30 environments
Sequential stoppingSetter implementation of prescribed reveal/commit templateNot included30 environments
Sequential searchSetter implementation of prescribed inspect/claim templateNot included30 environments

We manually inspected 30 environments per family. This inspection includes Negotiation, sequential stopping, and sequential search, covering their generated dynamics as well as intermediate states and terminal utilities. We checked the code and observed trajectories for incorrect transitions and access to hidden parameters. We found neither mechanism departures nor hidden-parameter access in the 330 inspected environments. The player-facing interface exposes declared observations and actions, while the executable dynamics retain latent parameters and references internally. The separate computation utility cannot access these internals (Appendix C.4). The automatic audit had 480 successful executions with repeatable outcomes and no above-reference utility. Training-time monitoring also found no raw utility above its reference before clipping.

W2 & Q2. Repeated runs and uncertainty

All arms use the benchmark's default simulator setting with five repeated runs. We compare the groups using an unpaired Welch analysis. The plotted ending balances are reconstructed below in thousands. Entries follow plotting order.

ModelFive balancesMeanSample SD
Base105.0, 5.4, 111.7, 53.2, −3.8-3.854.353.9
VHD-Play266.8, 227.5, 170.1, 138.1, 111.6182.863.8
Qwen3.7-Max215.6, 81.1, 164.3, 358.8, 6.3165.2134.5

The balance gain over Base is about 128.6k, with an approximate 95%95\% interval of [42.0k, 215.1k] and p≈0.009p \approx 0.009. All five trained-policy runs complete without bankruptcy. Against Max, the observed mean is 10.7%10.7\% higher and sample variance is 77.5%77.5\% lower, with a mean-difference interval of [−147.3k,182.5k][-147.3\mathrm{k}, 182.5\mathrm{k}]. Generated-family results use one evaluation seed and the step-34 training checkpoint.

W2. Longer horizons

Section 5.1 evaluates the same trained checkpoint on the 365-day storefront. Generated-family episodes have median 31 and mean 59.7 assistant turns, while storefront runs span 1,147 to 1,825 and average 1,359 (Appendices F and H.4). All five finish without bankruptcy. This gives external transfer over roughly 23×23\times the generated-family mean horizon. Figure 5 addresses whether larger mechanisms remain constructible and learnable, using a checkpoint trained at each scale. The storefront results demonstrate transfer of the same trained policy to longer interactions with a different objective and interface.

We thank the reviewer for recognizing the contributions of our mechanism-first approach and experimental findings. We are grateful for the opportunity to clarify the experimental settings and address a potential misunderstanding regarding longer-horizon generalization. We would be happy to provide further clarification.

Thank you again for your constructive review. We hope our responses have helped clarify your remaining concerns, particularly regarding longer-horizon generalization.

We would welcome any further feedback and, if you find our clarifications satisfactory, would greatly appreciate your reconsideration of the assessment.

Thank you for your time and consideration!

Reviewer_qYk2OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes to set the model and the environment as a constraint before starting a long-horizon tasks for the agents. Having even a sampled approximate to set a model will help to limit the potential scope of the long trajectories. Often without at least a lose setup of the model and environment, the trajectory drift can lead to errors. Doing well in benchmarks does not reveal the problems.

Critical strengths

This idea of setting the model ahead is a good one for long horizon tasks. Helps to limit the drift.

Critical weaknesses

One has to have a good model and also may miss out on the scope of obtainable results as the trajectories are now constrained by the models. We will get to have better performance for the defined system but we may completely miss out on what is outside of the model that was provided.

Questions for authors

What are your thoughts on the sampling based models, this is an ancient technique from game and control and system identification communities that play long horizon games or interactions. These may give you good results but I am afraid that the results are limited. How will you get out of what you start with? I do like the idea and have experimented myself with these types of modeling but could not relax to generalize later!

3 comments

W1 & Q1. Family coverage

We thank the reviewer for the valuable comments. The eleven families are an initial selection from established mathematical and operations-research models. New families can build on or be selected from established mathematical models and solvers. Once a family is registered, new parameter and corpus draws produce further environments automatically (Appendix C.1). Training uses three families, and Section 5.1 reports gains on eight unseen families. It also reports gains on external function-calling and travel-planning tasks, as well as the 365-day storefront task.

The model already performs well on written-out problems. We think its gains on unseen families and real-world tasks suggest that training helps translate this knowledge into effective policies for stateful agentic tasks.

Q1. Mathematical mechanisms and LLM wrapping

VHD-Play combines solved mathematical mechanisms with corpus-grounded environment generation. This hybrid pipeline can provide verifiable reference solutions while using an LLM to generate diverse environments.

Based on our experiments, our educated guess is that incorporating the suggested sampling-based methods into our pipeline, or using them to replace the mathematical model, could help. In particular, the diverse corpus could support the generation of varied training samples for better generalization, rather than quickly overfits on a small amount of simliar sampling-based methods related cases. Another educated guess is that sampling-based models could help the model learn to handle uncertainty.

We thank the reviewer for this exciting question and would like to expand the discussion in the revision!

We thank the reviewer for recognizing the value of our mechanism-first approach for long-horizon tasks and for the thoughtful discussion on generalization. We are grateful to answer any follow-up questions.

Thank you for your clarifications. None of my questions were to ask for new computations or improvement comparisons. I will maintain the scores.

Comments on the paper

3 comments

W1 & Q1. Verification

We thank the reviewer for the valuable comments. The mechanism defines the transition rules and feasible actions before realization, together with the evaluation references. The player-facing interface exposes designated observations and actions. The executable dynamics retain the sampled parameters and reference values internally. The computation tool is state-independent and cannot access environment internals (Appendix C.4). Stopping uses the largest option value as its hindsight ceiling, and search uses the best single-inspection net value. Both are computed from the full draw. Negotiation uses prescribed counterpart equations and a solver-derived reference. Each family has a fixed specification against which its realization can be inspected.

Appendix B reports 480 executions on 80 environments from eight families. During the training and evaluation, all executions completed with matching outcomes on repeated runs. We did not observe raw utility exceeded its reference. We additionally inspected 30 environments from each of all 11 families, 330 in total. The inspection covered the code and trajectories, including intermediate states and terminal outcomes, and checked for incorrect transitions and access to hidden parameters. We found neither mechanism departures nor hidden-parameter access in the inspected environments across all 11 families, as we acknowledge that it could happen without careful design and the potential tendency from base policy. During training, we also monitored sampled policy-generated trajectories and found no raw utility above its reference before reward clipping. We thank the reviewer again for mentioning this point and will include these details in appendix in next reversion.

W2 & Q2. Evaluation uncertainty

We follow E-Commerce Bench's protocol with five runs per arm under the shared default simulator setting. Mean ending balance rises from 54,294 to 182,844, or 3.4×3.4\times. The trained checkpoint completes all five runs without bankruptcy, while Base completes four. An unpaired Welch analysis of the plotted outcomes gives a balance gain of about 128.6k, a 95%95\% interval of [42.0k, 215.1k], and p≈0.009p \approx 0.009. Its observed mean exceeds Qwen3.7-Max's 165,224 by 10.7%10.7\%, with 77.5%77.5\% lower sample variance. The mean-difference interval is [−147.3k,182.5k][-147.3\mathrm{k}, 182.5\mathrm{k}]. Generated-family results use one evaluation seed and the reported step-34 training run.

W2 & Q3. Construction order

We thank the reviewer for the comments. EnvScaler is a representative released environment-first pipeline. We compare performance after training on data built by their pipeline and ours, starting from the same 35B checkpoint and using the same GRPO configuration and 34-step budget. We also report performance advantages on external instances under shared inference settings. These results help show the gap between training on data built using the two construction orders.

Section 5.2 complements this comparison by holding the mathematical problem and scoring reference fixed while varying its presentation. It shows a gap between the model's knowledge of the problem and its ability to act as a policy in the environment. Carrying out a solution through valid tool calls is part of this operating gap. The model must also respect shared constraints and persistent commitments. The I−AI-A comparison then varies parameter exposure under the same interactive form. BFCL's improvement from 61.25 to 64.08 provides external evidence of transfer to tool interaction. F−IF-I measures the combined planning and execution demands of acting as a policy. The experiments compare the resulting construction approaches and characterize the policy improvement enabled by our realization.

We thank the reviewer for recognizing the contributions of our mechanism-first construction and the value of our evaluations on unseen families and external tasks. We are grateful to provide any further clarification.

Thank you for the detailed response. It addressed the questions I raised in my initial review. I have decided to maintain my original rating. Best of luck!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.