- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposed Earis, a framework to let agent reason in the synthetic environment first before final actions in real environments. It comprises two components, a environment synthesis pipeline and a inference-time augmentation method. Results across 6 benchmarks on 2 models suggest simple checks, rather than the proposed method, suffice.
Critical strengths
- Explored a training-free way to improve agent performance by inference in the synthetic environments before real actions.
- Proposed a lightweight pipeline to synthesize and validate environments.
Critical weaknesses
- Environment synthesis pipeline in section 2.2 is not convincing. The author claims the verification "cannot establish agreement with the real environment" and the refinement "does not guarantee correct behavior on untested calls".
- Experiment results in section 5.3 show that the Earis doens't outperform basic ReAct agent nor simply policy-given ReAct agent in 3 out of 4 evaluations, suggesting the advantage of Earis mostly from more compute and access to the withheld rules.
Questions for authors
- The customter service example in the introduction section is not clear enough to show the importance of the order of the action sequence. Why editing size first leads to the failure of editing address? Is it the intrinsic limitation of the environment?
- It's remains unclear why creating new synthetic environments instead of copying and resuing the tested environment for the inference-time reasoning. If it's because the agent doesn't have access to the tested environment at inference time, the synthesis is also impossible. If because of effiency, author should provide environment effiency-related results in the evaluation section.