acceptodds
Withdrawn from ICLR 2027

BLUE: Benchmarks, Let Us Evolve with Agent

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

10 views

Abstract

As LLM agents acquire new skills, tools, and other actions, their capability space continues to expand. However, existing benchmarks are either static or generate more complex tasks only within a previously defined capability space, creating a growing mismatch between what agents can do and what benchmarks can evaluate. To address this gap, we introduce BLUE, an evolvable benchmark generator for agent evaluation. For each newly introduced capability, BLUE requires only a simple user-supplied atomic task and its corresponding evaluation. It groups atomic tasks into atomic scenarios, constructs a scenario graph to generate complex tasks, and simultaneously composes atomic evaluations into evaluators for these complex tasks. BLUE combines rule-based metrics with LLM judges and supports both single-turn and multi-turn evaluation. We construct BLUE-Bench, which covers diverse multi-capability scenarios, to systematically evaluate five representative coding agents paired with three foundation models. We further conduct a detailed analysis of BLUE key components to validate the effectiveness of the framework. All code and data are available at https://anonymous.4open.science/r/BLUE-ICLR.

Primary area:
datasets and benchmarks
Keywords:
Benchmarks, Benchmark Generation, LLM Agents, Agent Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dAZROpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

BLUE proposes a framework for building agent benchmarks by combining atomic tasks into more complex scenarios. It uses a graph to guide composition and evaluates both task completion and interaction quality via rule-based checks and LLM judges. The paper evaluates multiple agent frameworks and models. It also reports various task-generation comparisons, ablations, and human–LLM agreement analyses.

Critical strengths
  1. The paper addresses a meaningful problem: building more complex agent benchmarks from existing atomic tasks. The task-relation graph provides a structured way to combine tasks through dependencies and shared context.

  2. The evaluation covers multiple agent frameworks and models. What's more, it considers both task completion and interaction quality. This broader scope can help identify limitations that final-answer evaluation alone may miss.

Critical weaknesses
  1. There is a big problem: the reported human evaluation is suspicious. Section 4.4 states that three experts independently scored the outputs. However, the released repository includes Human_Eval/Human_Consistency_Eval_1/_generate_human_scores.py, which generates “human” scores by perturbing LLM scores and writes them into the annotation files. I reproduced this procedure and found exact matches for all 350 answer-quality scores and 150 interaction-quality scores in that directory. Aggregating the three released annotation sets also reproduces the InteractionQuality statistics in Table 5. These findings directly conflict with the claim of independent human evaluation and raise a serious concern about integrity. This concern is a primary reason for my recommendation for rejection.

  2. Some rule-based checks do not verify substantive task completion. A fabricated report received 20/20 from Ops/scene_1.py simply by including expected headings and keywords, despite explicitly stating that no inspection was performed. It raises concerns about the reliability of the benchmark’s scores.

Questions for authors
  1. The released scores contradict the claim of independent human evaluation. All 500 scores in Human_Consistency_Eval_1 exactly match the script that generates scores by perturbing LLM ratings, and aggregating the released annotation sets reproduces the InteractionQuality statistics in Table 5. How do the authors reconcile this evidence with Section 4.4?

  2. Clarification on evaluation validity: How does the full evaluator handle a fabricated report that receives 20/20 from Ops/scene_1.py without performing the task? Does the LLM judge reliably reject such outputs?

1 comment

Dear Reviewer,

Thank you very much for bringing this important issue to our attention.

Regarding your concerns about the human evaluation data, we would like to provide a complete and truthful explanation.

Due to inadequate coordination and scheduling during the research process, the expert responsible for Eval_1 was unable to submit their final evaluation results by the scheduled deadline. While waiting for this expert to complete the evaluation, we temporarily used placeholder data generated from LLM-based scoring to facilitate the drafting, formatting, and presentation of the preliminary manuscript.

However, due to my personal oversight and the approaching submission deadline, I failed to replace these placeholder results with the actual scores from the third expert. Consequently, I also performed statistical analyses based on the preliminary placeholder data, which led to the inclusion of results in the submitted manuscript that had not been evaluated by the actual human expert. I fully acknowledge that this constitutes a serious lack of rigor in our experimental methodology and reporting.

We sincerely appreciate your identification of this issue. In the interest of maintaining scientific integrity and research rigor, all co-authors have jointly decided to voluntarily withdraw the current submission. Furthermore, my academic advisor and other co-authors have instructed me to conduct a systematic and comprehensive re-examination of all experimental data, evaluation procedures, and statistical analyses presented in the manuscript to ensure their accuracy and reliability.

Once again, I would like to express my sincere gratitude for your careful and rigorous review, as well as the time and effort you devoted to investigating and verifying these concerns. Your feedback has made me deeply aware of my shortcomings.

Sincerely,
First author

Reviewer_Rq6jOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper observes that existing benchmarks are challenging to fully capture the capabilities of real-world agents due to a fixed capability space.

This paper proposed BLUE, an evolvable benchmark generator, for agent evaluation by constructing complex and realistic tasks from atomic tasks.

A benchmark, BLUE-Bench, was constructed to cover diverse scenarios and used to evaluate 5 coding agents with 3 foundation models.

A detailed analysis shows that BLUE is an effective benchmark generator framework for capability composition expansion of both tasks and evaluations.

Critical strengths

1, The argument that existing benchmarks defined over a fixed capability space are challenging to evaluate real-world agents. The idea of introducing an evolvable benchmark generator framework is interesting with capability composition expansion for agent evaluation.

2, BLUE proposes a hierarchical composition scheme to compose atomic tasks to realistic tasks with end-to-end evaluators.

3, Ablation studies are conducted to validate the effectiveness of BLUE for high-quality tasks generation.

Critical weaknesses

1, lack experimental results for supporting that existing LLM-based generators such as Continuous Benchmark Generation failed to evaluate agent capability and BLUE can address this issue.

2, natural-language evidence h_ij seems not needed since the relation and confidence score for the Scenario Graph is provided. More support needed to show why natural-language evidence is necessary.

3, Main result section is not clear. For example, line 351-355, Table 1 seems not supporting that agent harness increases from 3.5 in single-turn evaluation to 6.71 in multi-turn evaluation, and the gap across foundation models shrinks from 2.96 to 0.75 points.

No explanation why LLM-judge score is quite low compared to rule-based score even for single-turn evaluation in Table 1. Take Claude Code + GLM5 for example, For easy and hard tasks in single-turn evaluation, the performance is 17 point gap, which may make the LLM-judge multi-turn evaluation not needed since LLM score is already too low to reflect the agent capability in multi-turn evaluation.

For line 362-367, Codex+Kimi-2.6 has much high interaction score than codex+deepseek-v4-pro/glm-5 with reasonable multi-turn score. I do not understand why it means good interaction alone does not guarantee task success. Is it because Table 1 mutli-turn evaluation overall score is much better indicator than interaction score?

4, How can we interpret Table 2. Table 2 is not referenced in the main paper.

5, For Fig. 6, why semantic coverage suddenly changes the expansion direction for k=70 to k=100 compared to k=10 to k = 40 and k=40 to k=70.

Questions for authors

See Weakness.

Reviewer_hDDxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

As agent capabilities develop, more complex and customized evaluations are needed to properly evaluate the agent, while prior benchmarks are fixed capabilities that may not adapt to the evaluation requirements. The paper introduces BLUE, an adaptive evaluation generation workflow that extrapolates complex task construction into graphs to generate customized evaluations based on single sample scenarios, combining both rule-based and LLM checks. The workflow also supports multi-turn evaluation scenarios with a user proxy. The paper also introduces BLUE-Bench and evaluates five agent harnesses paired with three models.

Critical strengths
  1. The paper addresses a timely problem of automating evaluation generation for newly introduced capabilities. The motivation to solve the problem would be meaningful to the community as agentic evaluation becomes pivotal in more deployed use cases.
  2. The presentation is clear, intuitive, with coherent motivations and organized results.
Critical weaknesses
  1. The framework lacks critical evaluations on the final task quality. Although the authors does provide indirect evidences including human preferences study and scenario construction ablation on human annotated sets, the scale of 10 human annotations is very restricted. Direct evidence by a certain amount of end to end human auditing of the task generated is pretty innegotiable for task validty.
  2. The main claim about evolving with new capabilities is also briefly evaluated in section 4.5 with claims about increasing atomic task pool introducing BLUE's task diversity. However, it would be much more direct if the author can give several qualitative examples of scenarios with newly introduced capabilities and evaluate the resulting benchmark behaviors.
Questions for authors
  1. Corresponding to W1, could the author provide direct evidence of task quality?
  2. Corresponding to W2, could the author provide direct evidence of evolving with new capabilities?
  3. It will be great if the authors can report an estimated time and money cost of developing BLUE-bench or certain benchmarks versus BLUE to demostrate its economic impacts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.