- Rating
- 3
- Confidence
- 4
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper aims to address an important problem of current research on CUA is not standardized in terms of model, environment, and data. The authors introduced Lite.Gym (for unified environment) and Lite.Sample (for unified trajectory format) with standardized model harness execution. The paper includes comprehensive experiments showing that their implementation results in similar performance with original ones, and the authors performed evaluation on 15 models and 12 benchmarks.
Critical strengths
- The authors integrated several important aspects of CUA, including unifying environments and data, while standardizing model harness execution. This has important practical value to the CUA research community for easier plugin and standardized evaluation.
- The resulting unified environment enables similar performance compared to the raw performance.
- The evaluation is very comprehensive on 12 benchmarks and 15 models.
Critical weaknesses
- The paper didn't demonstrate whether there is actually benefit in unifying environment and training data, despite making evaluation more standardized and easier. It would be great if the authors could show that without standardization, it is difficult to train a good model, etc.
- Missing references: Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents. AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning.
Questions for authors
N/A
2 comments
Thanks for the response. I will maintain my positive score.
We thank reviewer AmfM for recognizing the practical value of unifying environments, data, and harnesses, and the breadth of our evaluation. In response to your questions:
While we have not demonstrated that unification is required to train a good model, it lets any model train on data from any source, whether public datasets or other models' rollouts. For example, the distillation ablations from two teachers (GPT-5.5 and Qwen3.5-27B) in Section 4.1, under different reasoning and screenshot settings across three platforms (Table G.1), would otherwise be much more difficult to run. The same format also makes it easy to combine data from many sources. We have already curated 1.26M samples from 9 public datasets and 82K verified rollouts in this format (Section 2.2), which we believe will facilitate future model training.
Thank you for pointing out these works. We have cited Agent Data Protocol [1] and AgentOhana [2] in the related work of the updated PDF.
If these answers do not fully address your concerns, we are happy to clarify further. We will integrate the new results into the revised version.
[1] Yueqi Song, Ketan Ramaneti, Zaid Sheikh, et al. Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents. ICLR 2026.
[2] Jianguo Zhang, Tian Lan, Rithesh Murthy, et al. AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learning. arXiv:2402.15506, 2024.