- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
BLUE proposes a framework for building agent benchmarks by combining atomic tasks into more complex scenarios. It uses a graph to guide composition and evaluates both task completion and interaction quality via rule-based checks and LLM judges. The paper evaluates multiple agent frameworks and models. It also reports various task-generation comparisons, ablations, and human–LLM agreement analyses.
Critical strengths
-
The paper addresses a meaningful problem: building more complex agent benchmarks from existing atomic tasks. The task-relation graph provides a structured way to combine tasks through dependencies and shared context.
-
The evaluation covers multiple agent frameworks and models. What's more, it considers both task completion and interaction quality. This broader scope can help identify limitations that final-answer evaluation alone may miss.
Critical weaknesses
-
There is a big problem: the reported human evaluation is suspicious. Section 4.4 states that three experts independently scored the outputs. However, the released repository includes
Human_Eval/Human_Consistency_Eval_1/_generate_human_scores.py, which generates “human” scores by perturbing LLM scores and writes them into the annotation files. I reproduced this procedure and found exact matches for all 350 answer-quality scores and 150 interaction-quality scores in that directory. Aggregating the three released annotation sets also reproduces the InteractionQuality statistics in Table 5. These findings directly conflict with the claim of independent human evaluation and raise a serious concern about integrity. This concern is a primary reason for my recommendation for rejection. -
Some rule-based checks do not verify substantive task completion. A fabricated report received 20/20 from
Ops/scene_1.pysimply by including expected headings and keywords, despite explicitly stating that no inspection was performed. It raises concerns about the reliability of the benchmark’s scores.
Questions for authors
-
The released scores contradict the claim of independent human evaluation. All 500 scores in
Human_Consistency_Eval_1exactly match the script that generates scores by perturbing LLM ratings, and aggregating the released annotation sets reproduces the InteractionQuality statistics in Table 5. How do the authors reconcile this evidence with Section 4.4? -
Clarification on evaluation validity: How does the full evaluator handle a fabricated report that receives 20/20 from
Ops/scene_1.pywithout performing the task? Does the LLM judge reliably reject such outputs?
Dear Reviewer,
Thank you very much for bringing this important issue to our attention.
Regarding your concerns about the human evaluation data, we would like to provide a complete and truthful explanation.
Due to inadequate coordination and scheduling during the research process, the expert responsible for Eval_1 was unable to submit their final evaluation results by the scheduled deadline. While waiting for this expert to complete the evaluation, we temporarily used placeholder data generated from LLM-based scoring to facilitate the drafting, formatting, and presentation of the preliminary manuscript.
However, due to my personal oversight and the approaching submission deadline, I failed to replace these placeholder results with the actual scores from the third expert. Consequently, I also performed statistical analyses based on the preliminary placeholder data, which led to the inclusion of results in the submitted manuscript that had not been evaluated by the actual human expert. I fully acknowledge that this constitutes a serious lack of rigor in our experimental methodology and reporting.
We sincerely appreciate your identification of this issue. In the interest of maintaining scientific integrity and research rigor, all co-authors have jointly decided to voluntarily withdraw the current submission. Furthermore, my academic advisor and other co-authors have instructed me to conduct a systematic and comprehensive re-examination of all experimental data, evaluation procedures, and statistical analyses presented in the manuscript to ensure their accuracy and reliability.
Once again, I would like to express my sincere gratitude for your careful and rigorous review, as well as the time and effort you devoted to investigating and verifying these concerns. Your feedback has made me deeply aware of my shortcomings.
Sincerely,
First author