acceptodds
Under review as a conference paper at ICLR 2027

Prax: Programmable Synthesis of Verifiable Environments for Working Agents

10 views

Abstract

Training capable agents for real-world work requires large collections of diverse, executable, and verifiable environments, yet such environments remain expensive to author and difficult to synthesize reliably. Unlike text-only data, a work environment forms a self-contained system: task instructions, underlying state, tool interactions, and verification logic must remain mutually consistent. Directly generating these components with LLMs often produces environments that appear plausible but are actually unsolvable, inconsistent, or incorrectly scored. We introduce **Prax**, a framework for **programmable synthesis of verifiable work environments**. Prax represents an environment as a structured specification that separates invariant task logic from generative parameters, executable ground-truth derivations, rendering rules, and reference instances. This representation enables scaling along two complementary axes: **(i)instantiating the same task structure across new scenarios, entities, and states**, and **(ii)applying correctness-constrained transformations that systematically alter reasoning structure, difficulty, and failure modes.** To preserve verifiability, Prax generates environment facts before deriving expected outcomes, and validates each synthesized instance with programmatic checks for structural validity, tool-call closure, state–verifier consistency, answer leakage, and runtime compatibility. We further execute synthesized environments with agents and reject trajectories that fail execution or quality criteria before training. Starting from over 20k synthesized task intents, Prax constructs 25k reusable specifications and over 65k environment instances spanning diverse workflows. Training on the verified trajectories yields state-of-the-art performance on general knowledge tasks at 35B size, demonstrating that Prax can turn a small collection of expert-designed tasks into scalable, high-quality supervision for agents that perform real-world professional work.

Primary area:
foundation or frontier models, including LLMs
Keywords:
synthetic data generation, working agent

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jfnxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Prax, a pipeline for synthesizing verifiable work environments for agent training via decision programs that co-generate tasks, environments, and verifiers, plus formal results on verification error and large benchmark improvements. It includes a formal treatment of post-hoc vs. co-compiled verification error, a large-scale synthesis pipeline (25k specs, 65k instances), and large benchmark gains for 35B-A3B and 4B models on Claw-Eval, AutomationBench, JobBench, and WorkspaceBench.

Critical strengths

The problem is important to study: naive LLM-generated environments are full of silent correctness bugs, and the paper's taxonomy of those bugs (unsolvable, inconsistent, mis-scored) rings true. The co-compilation idea is sound, the theory section is a genuine attempt at formal grounding rather than decoration, and the engineering is thorough (demand/solve modes, fact ledgers, counterfactual replay with per-assertion failure vectors). The ablation against post-hoc-verifier-plus-rejection-sampling is the right experiment and it favors Prax. Results are good in absolute terms.

Critical weaknesses

Novelty is the main concern. The theoretical contribution is thinner than it appears. Theorem 1 is essentially the observation that if two different ground truths render to the same surface task, a verifier that only sees the surface can't distinguish them. It is true, but this is a fairly direct information-theoretic point, and the tightness claim is over unrestricted acceptance rules, which is a weak notion of tightness. Theorem 2 proves exact verification under four assumptions that include "rendering preserves primitive semantics" and "assertion predicates cover every component without extra constraints", i.e., assuming the pipeline works, the pipeline works. The actual hard problem (does the LLM rendering preserve semantics? are the assertions complete?) is handled by "automated checks covering common violations," which is engineering, not theory.

Besides, I'm not sure about reproducibility and complexity. This is a very long pipeline with skill graphs, intent sampling with quotas and fingerprints, a typed Python DSL with four primitive families, demand/solve compilation, service adapters, constrained LLM rendering, static checks, replay-based acceptance, trajectory rejection sampling, with LLM calls at multiple stages and many hand-designed components (the primitives, the adjudication rules, the assertion contracts). Appendix B (which supposedly details the primitives) wasn't sufficient for me to assess whether the DSL is doing the real work. When the hand-designed DSL encodes this much task structure, the synthesis starts to look like elaborate templating, and the generality of the approach is unclear: how much human effort goes into designing primitives for a new domain?

Questions for authors

how much human effort goes into designing primitives for a new domain?

Reviewer_k7QWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Prax, which synthesizes executable work environments and verifiers from shared typed decision programs. The authors construct 25k reusable specifications and over 65k environment instances, then train 4B and 35B-A3B agents on verified trajectories, reporting improvements across four work-related benchmarks.

Critical strengths
  1. The paper presents a scalable pipeline for constructing consistent work environments. Generating environments, reference trajectories, and verifiers from a shared typed decision program helps reduce inconsistencies among these components. This provides a useful design for building executable and verifiable agent training tasks at scale.

  2. The trained models achieve strong empirical results. The 35B model improves over its base model across all four evaluated benchmarks, and the synthesis ablation also shows an advantage over forward synthesis with post-hoc verification. These results support the practical training value of the proposed pipeline.

Critical weaknesses
  1. Important implementation details are insufficiently specified, limiting the assessment of reproducibility. The origins of the initial task seeds and skill graph, the generation of decision programs, and the materialization of ledger requirements into concrete workspace contents require clearer explanations. The model used to generate training trajectories, and whether it is held fixed across the synthesis ablations, are also unclear. These omissions make it difficult to independently reproduce the pipeline and assess the contribution of the proposed synthesis method.

  2. The scope and generalization of the method remain insufficiently established. The framework relies on a fixed set of 23 primitives, but their coverage of realistic work tasks is not quantified. Although the aggregate gains across four benchmarks are encouraging, they do not establish whether improvements are concentrated on tasks closely aligned with these primitives or extend to tasks requiring more open-ended document understanding and complex file operations.

Questions for authors
  1. The paper reports constructing 25k reusable specifications and over 65k environment instances from over 20k synthesized task intents. What exactly are the initial seeds underlying these intents, and where do they come from? How are occupation/domain information, the skill–artifact graph, and user personas used to construct these seeds and generate such a large collection of task intents and corresponding decision programs?
  2. Section 2.4 describes structured instantiation and constrained LLM rendering, and Appendix B.3 mentions background distractors. How are these mechanisms implemented to construct the complete workspace? In particular, what additional files and background information are included beyond the evidence required by the ledger, and how is their consistency with the task facts checked? Could you provide two or three complete examples showing the initial task inputs, decision programs, ledgers, actual workspace contents, and verifiers?
  3. Which capabilities required by realistic work tasks are covered by the current 23 primitives, and which fall outside their expressive scope? Separately, how do base-to-trained performance gains differ across evaluation tasks fully covered, partially covered, and not covered by these primitives?
  4. Which teacher model is used to generate the training trajectories? Is the same teacher model used across the synthesis ablations to distinguish the contribution of the proposed method from that of teacher model capabilities?
  5. Do the authors plan to release the training pipeline and generated workspaces? If so, which components and data will be included? For a paper whose central contribution is a data-generation pipeline, access to the pipeline implementation and generated data is crucial for verifying reproducibility and supporting further research. If the authors provide sufficient implementation details and supporting artifacts to establish reproducibility, I would be willing to raise my rating to at least 3.
Reviewer_CM3vOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This work introduces Prax, a framework for programmable synthesis of verifiable work environments. The main insight of this paper is that verification of task outcomes can be more effective if the verifier builder has access to seed/intent that generates the task (which the paper term 'co-compiled verifiers'), instead of the task description itself (which the paper terms 'post-hoc verifiers'). The paper also formalizes the ambiguity of post-hoc verification (Theorem 1) and fidelity of co-compiled verifiers under strong assumptions (Theorem 2).

As for empirical verification, the paper compares 2 variants of Prax against 12 models on 4 benchmarks.

Critical strengths

The idea that verifiers should be built from hidden intents of tasks makes sense. Empirical results in Table 1 seem to suggest Prax models are competitive.

Critical weaknesses
  1. Novelty: the idea that verifiers should be built from compact task descriptions does not seem to be new, as procedural benchmarks (e.g., Reasoning Gym) already does this.
  2. Theoretical content seems thin: theorem 1 is quite straightforward and there is no discussion on why it should have real world implication. On the other hand, the assumptions of Theorem 2 are fairly strong; and there is no study on whether optimal verifiers can be learned empirically.
  3. Unclear experiment setup / lacking ablation studies: there is no description on how Prax models are trained (or were they not trained models? In that case the authors should experiment with more base models). The paper would also be more convincing if there is a large scale co-compiling vs post-hoc verification comparison (say using the same 20k intents).
  4. Writing is confusing at times: for example the formalization in Sec 2.1 seems unnecessarily complicated for conveying the main idea that the verifier constructor should be a function of intent. Some running examples of what intents / tasks are may help.
Questions for authors

How will you extend Prax to support recursion and unbounded loops?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.