- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents Prax, a pipeline for synthesizing verifiable work environments for agent training via decision programs that co-generate tasks, environments, and verifiers, plus formal results on verification error and large benchmark improvements. It includes a formal treatment of post-hoc vs. co-compiled verification error, a large-scale synthesis pipeline (25k specs, 65k instances), and large benchmark gains for 35B-A3B and 4B models on Claw-Eval, AutomationBench, JobBench, and WorkspaceBench.
Critical strengths
The problem is important to study: naive LLM-generated environments are full of silent correctness bugs, and the paper's taxonomy of those bugs (unsolvable, inconsistent, mis-scored) rings true. The co-compilation idea is sound, the theory section is a genuine attempt at formal grounding rather than decoration, and the engineering is thorough (demand/solve modes, fact ledgers, counterfactual replay with per-assertion failure vectors). The ablation against post-hoc-verifier-plus-rejection-sampling is the right experiment and it favors Prax. Results are good in absolute terms.
Critical weaknesses
Novelty is the main concern. The theoretical contribution is thinner than it appears. Theorem 1 is essentially the observation that if two different ground truths render to the same surface task, a verifier that only sees the surface can't distinguish them. It is true, but this is a fairly direct information-theoretic point, and the tightness claim is over unrestricted acceptance rules, which is a weak notion of tightness. Theorem 2 proves exact verification under four assumptions that include "rendering preserves primitive semantics" and "assertion predicates cover every component without extra constraints", i.e., assuming the pipeline works, the pipeline works. The actual hard problem (does the LLM rendering preserve semantics? are the assertions complete?) is handled by "automated checks covering common violations," which is engineering, not theory.
Besides, I'm not sure about reproducibility and complexity. This is a very long pipeline with skill graphs, intent sampling with quotas and fingerprints, a typed Python DSL with four primitive families, demand/solve compilation, service adapters, constrained LLM rendering, static checks, replay-based acceptance, trajectory rejection sampling, with LLM calls at multiple stages and many hand-designed components (the primitives, the adjudication rules, the assertion contracts). Appendix B (which supposedly details the primitives) wasn't sufficient for me to assess whether the DSL is doing the real work. When the hand-designed DSL encodes this much task structure, the synthesis starts to look like elaborate templating, and the generality of the approach is unclear: how much human effort goes into designing primitives for a new domain?
Questions for authors
how much human effort goes into designing primitives for a new domain?