- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a method called Rubric-guided Harness Evolution (RHE). It introduces a "rubric layer" that serves as a behavioral guide, positioned between the analysis of execution traces and the modification of the test harness. This layer is decoupled from the underlying implementation details. The system first identifies behavioral flaws within task execution traces and abstracts these flaws into a set of rubrics, which encompass expected behaviors, applicable scenarios, and boundary constraints. Subsequently, the system employs a structural editor and a guided editor to generate modifications for the test harness, selecting the most effective version—based on task evaluation results—for the next iteration. Furthermore, the method incorporates a hierarchical experience memory that stores the relationships between execution evidence (rubrics) and historical modifications, enabling the reuse of past experience across multiple iterations. The authors evaluated the approach using three backbone models on the SWE-bench Verified and AppWorld datasets, including comparisons with AHE, ReasoningBank, and ACE. Ablation studies were conducted to assess the behavioral abstraction module and the experience memory module. The study also analyzed different modification spaces and the transferability of rubrics across diverse environments. Experimental results demonstrate that RHE achieves state-of-the-art performance across all six designated test settings.
Critical strengths
- The Behavioral Rubric provides a valuable intermediate abstraction of behaviors for Harness Evolution. Instead of generating modifications directly from failures observed in execution traces, the approach first distills behavioral requirements, which are then used to optimize Harness components (specifically Prompts, Memory, Tools, and Middleware). This design decouples behavioral objectives from their concrete implementations. The Rubric can be reused, refined, or expanded based on new evidence; when combined with a hierarchical experiential memory, it forms a continuously updated guidance mechanism—a key contribution of this work.
- The experimental evaluation is comprehensive, providing evidence of effectiveness across different models, tasks, and modification mechanisms. Across two benchmarks and six distinct test scenarios, covering three backbone models, RHE consistently outperforms baseline methods. Through Tables 2 and 3, the authors validate the impact of behavioral abstraction, experiential reuse, and the design space of Harness implementations; the paper also includes iterative execution results, evolutionary curves, and case studies. Furthermore, cross-environment transfer experiments provide preliminary evidence that the Rubric is reusable. Collectively, these results substantiate the overall efficacy of the proposed approach.
Critical weaknesses
- Insufficient validation of the independent contributions of the "Behavioral Rubric" and the hierarchical experiential memory. Existing frameworks like AHE and ReasoningBank already leverage trajectory-based experience to optimize the harness. Since the diagnostic phase already identifies generalization deficits, the Rubric merely provides supplementary information regarding specific scenarios and boundary conditions. Furthermore, the two sets of ablation experiments fail to account for differences in API call volume and token consumption. Removing the memory module results in only marginal performance gains, and the study lacks a control group using a "flat" (non-hierarchical) memory structure. Consequently, the current experimental design cannot isolate the individual contributions of each mechanism.
- Limited evidence for cross-environment generalization; lack of analysis regarding the computational costs of the evolutionary process. The performance improvements gained from transferring the Rubric across environments are modest, and importing the entire set of criteria at once actually degrades performance. The absence of control groups—such as those using randomized criteria or general behavioral benchmarks—makes it difficult to verify whether the observed improvements stem from behavioral knowledge acquired in the source environment. Moreover, the paper reports results based on a fixed budget of ten iterations, without analyzing the impact of iteration-count variability or the total cost of the evolutionary process.
Questions for authors
- Validation of the contributions of the Rubric and memory modules. Within the constraints of a fixed token budget for alignment analysis, could we compare the full Behavioral Rubric (containing comprehensive behavioral summaries) against a simplified version that retains only the behavioral objectives? Additionally, by including a baseline that utilizes a flat memory retrieval mechanism (based on the same historical information), we could determine whether the hierarchical, relational structure itself provides an independent performance gain.
- Alignment between the Rubric and the proposed modifications. Could we employ independent behavioral evaluations to verify whether the modifications made to the Harness satisfy the requirements specified in the corresponding Rubric? Furthermore, by comparing the outcomes when the Rubric is correctly matched versus when it is randomly mismatched, we can confirm whether performance improvements are truly derived from effective behavioral guidance.
- Cross-environment generalization. Could we incorporate a comparable set of randomized Rubrics or generic behavioral guidelines as a control group for transfer experiments? Specifically, if a Rubric learned in a source environment is used to guide modifications to a Harness in a target environment—without undergoing further iterative evolution—can we still observe performance gains resulting from this transfer?
- Evolutionary overhead and result robustness. It should be clarified whether the task agent and the evolutionary editor share the same backbone model; additionally, the total token consumption and model invocation costs for the entire evolutionary process should be reported. Furthermore, by varying the iteration budget and conducting multiple independent runs, we could analyze the trends and stability of the performance improvements.
A Kind Reminder Before the Discussion Period Ends (Reviewer mTLh)
Dear Reviewer mTLh, Thank you again for your detailed review. In our replies, we reported human ratings of rubric quality and evaluated retrieval using human judgments of relevance (Recall@3 = 0.72). We also tested whether rubrics selected automatically by an LLM could improve performance on AppWorld and compared RHE with AHE under the same 2.40B-token cap. If your schedule permits, we would appreciate hearing whether these results address your main concerns. We would be glad to clarify any remaining points while the discussion is open.