- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper proposes an AI agent system that automatically generates hypotheses about AI agent behavior and evaluates them through automatically designed controlled experiments. The system consists of two main components: one provides a framework for designing simulation environments, and one automates the experimental research process. The paper investigates 12 target behaviors and generates and tests 73 hypotheses using the proposed system.
Critical strengths
The proposed AI agent system is comprehensive and covers the full pipeline of AI behavioral research, including hypothesis generation, experimental design, simulation execution, behavioral evaluation, and statistical analysis. The integration of these stages into an automated framework is a main strength.
Critical weaknesses
(i) The paper is primarily an engineering-oriented work, with limited methodological contributions. Most of the proposed components rely on existing techniques, and the main contribution lies in integrating them into an automated research framework.
(ii) The paper presents three main sets of experimental results. The first focuses on the behavioral insights discovered by the proposed system, while the other two evaluate the reliability and validity of the system itself. However, the paper does not provide empirical comparisons with existing automated evaluation frameworks. Even if no existing framework is sufficiently general to serve as a direct baseline, the paper still lacks sufficient ablation studies to justify the necessity of its key components, such as the Fidelity Gate. Consequently, it remains unclear how much these individual components contribute to the overall performance of the proposed system.
Questions for authors
(i) Even if no existing baseline can simultaneously satisfy the scalability and automation requirements highlighted in Table 1, could the authors conduct ablation studies with appropriate quantitative metrics to demonstrate the effectiveness and necessity of key components in the proposed framework? For example, the authors could compare the full framework against a variant without the Fidelity Gate to assess its contribution to the reliability of the experimental findings. (ii) Is there a general criterion for evaluating the effectiveness of this type of automated AI behavioral experimentation system? For example, under a fixed time or computational budget, would a more effective system provide more reliable evidence for the hypotheses it tests?
3 comments
Additional Concern: Gating Mechanism
Q(i): the authors could compare the full framework against a variant without the Fidelity Gate to assess its contribution to the reliability of the experimental findings
Clarification 1. The effect of the gating mechanisms on the pipeline results is small.
- Prevalence: Coherence gate removed 2.5% of all configurations, and the fidelity gate removed 2.81% of the simulations.
- Effect of no gating: Removing fidelity gating mechanism does not substantially perturb the experimental outcomes
- Among the 73 hypotheses, only 1 hypothesis changed from inconclusive to positive
Clarification 2. The gating mechanism still plays a significant role for experimental soundness.
The gate protects the validity of each individual simulation, not the aggregate result. Failures are rare (128 of 4,570 simulations excluded, 2.8%), but each failure is a simulation that no longer tests its hypothesis. Below are some examples of what the gate removed:
- The value of X was not realized (33% of the rejected simulations). In hypothesis test of communication bandwidth → extroversion, the value ’none’ required replies capped at one short sentence. In six simulations, the subject wrote multi-paragraph replies and nothing cut them short, so those simulations tested normal bandwidth.
- A confound was introduced. In hypothesis test of actor hostility → empathy, the simulator changed the user's balance, days past due and late-fee date, so the stakes differed from the shared configuration for reasons unrelated to hypothesized cause .
- Text leaked across conditions. In studying target behaviors friendliness and purchase, text from other simulations in the matched group entered a single history, giving the subject parallel, incompatible states.
In summary, without the gates, the findings hold overall, but invalid simulations enter individual hypotheses.
We thank the reviewer once more for their constructive comments. The reviewer may find the full detail about the experiment in Appendix G.
I would like to thank the authors for their responses and the additional ablation studies and baseline comparisons. They have satisfactorily addressed my questions and concerns. I will maintain my positive score and will support the acceptance of this paper.
We thank the reviewer for the incisive feedback. The reviewer comments pushed us further to demonstrate both our method’s novelty and effectiveness through additional experiments.
Rebuttal Summary
Our response: We humbly acknowledge that our proposed method involves substantial engineering. However, we also claim that our proposed Abs2Sim framework is novel, and we conducted new baseline and ablation studies to demonstrate that Abs2Sim is the engine that supportsWe present our experimental evidence below, with the full details in Appendix G.
New experiment: ablations and baselines
Goal. To compare the quality of the generated simulations for behavioral research across different methodsSetup. Five methods generate simulations for the same 5 randomly sampled hypotheses. All use the same LLMs, blind reviewers, and statistical analysis.Measures. We establish two measures characterizing strong behavioral research.Result interpretation and our claims
Novelty (W(i)): Each Abs2Sim component provides a property that existing methods lack.Necessity and effectiveness (W(ii), Q(i)): Within the matched number of simulations, AEROBAT gives conclusive evidence (BF10 > 3) for 3/5 hypotheses (others: 1/5 or 2/5). Only the full method is best or tied-best on the two qualitative measures. We discuss the failure cases of the two baselines in Appendix G.3.