- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a method for evolving multi-objective agents called Harmony Harness. This method integrates multi-objective evolutionary algorithms (MOEA/D), factorized evolution, and dynamic workflow composition. Experiments on SecRepoBench, BaxBench, and CWEval demonstrate that this method improves functionality, security, and transferability.
Critical strengths
- Optimizing both functionality and security is of great value. Both of these objectives are evaluated through executable tests.
- This method outperformed the baseline in two benchmark tests and demonstrated encouraging transfer performance. The results of the ablation experiments support the proposed components.
Critical weaknesses
- The Pareto front is narrow and eventually converges to a dominant solution (Appendix G.1). Compared to weighted sum optimization, it yields a gain of 6.7% (Table 2), and this does not include uncertainty estimates. Appendix D also erroneously claims that the TChebyshev substitution cannot reduce the value of a single objective function.
- Each partition contains only 28-30 tasks, 3 runs were conducted, and no confidence intervals are provided. The effective sample size also fluctuates (Appendix C). Furthermore, Appendix K describes an overfitting threshold but does not report its detection accuracy or the threshold value.
Questions for authors
- Can the authors provide the results, confidence intervals, and effective sample size for each run?
- Can the authors demonstrate that MOEA/D has a clear advantage when the objective function is conflicting?
- Was an overfitting threshold used in the main experiments? If so, how was it calibrated and evaluated?
We thank the reviewer for their valuable time and constructive feedback.
Weakness 1, Q1 and Q2: The Pareto front is narrow and eventually converges to a dominant solution (Appendix G.1).
Compared to weighted sum optimization, it yields a gain of 6.7% (Table 2), and this does not include uncertainty estimates. Appendix D also erroneously claims that the TChebyshev substitution cannot reduce the value of a single objective function. Can the authors provide the results, confidence intervals, and effective sample size for each run? Can the authors demonstrate that MOEA/D has a clear advantage when the objective function is conflicting?
Response: We politely address it from two perspectives.
1. Functionality and security are not always aligned.
Although functionality and security can improve together during some stages of evolution, they are not universally aligned. The complete Pareto fronts in Figures 5 and 6 show solutions for which improving security reduces functionality, and vice versa. Thus, the search does encounter genuine objective conflicts rather than optimizing two equivalent metrics.
We agree that our original discussion did not make this distinction sufficiently clear. In the revised manuscript, we will explicitly discuss the trade-offs visible in Figures 5 and 6 and clarify that the purpose of MOEA/D is to preserve multiple solutions representing different functionality–security preferences.
2. Additional functionality–time-efficiency experiment
Evaluating an objective related to computational efficiency provides a stronger test of the proposed multi-objective framework. Therefore we conducted an additional experiment that jointly considers task performance and time efficiency.
All methods in the controlled comparison start from the same seed harness and use the same search budget. Results are reported on the 20-task search split. Higher values are better for both pass rate and time efficiency.
Table R4. Joint optimization of task performance and time efficiency.
tb_wf_seedwf_s2_diffwf_s2_toolwf_s2_genCompared with the unevolved seed harness, HH improves the pass rate from 58.3 to 72.0 and time efficiency from 49.6 to 65.2. Under our matched experimental setting, HH also outperforms our Meta-Harness run on both pass rate (72.0 vs. 69.2) and time efficiency (65.2 vs. 63.9). Weighted-sum evolution improves time efficiency but provides almost no pass-rate improvement, illustrating the difficulty of balancing the two objectives through a single linear scalarization.
This experiment extends our evaluation beyond functionality and security and demonstrates that the proposed framework can also optimize task performance jointly with computational efficiency.
And yes we would correct the saying in Appendix D.
###Weakness 2 and Q3: Each partition contains only 28-30 tasks, 3 runs were conducted, and no confidence intervals are provided.
The effective sample size also fluctuates (Appendix C). Furthermore, Appendix K describes an overfitting threshold but does not report its detection accuracy or the threshold value. Was an overfitting threshold used in the main experiments? If so, how was it calibrated and evaluated?
Response: Evaluation size. Our setting is actually a fair amount of testing (30+30+30), compared to the 5 harness papers evolving on the TerminalBench2, which has 89 tasks. The difference is that they evolve and test on the same set, evolved on 89 and tested on 89. After a second thought, we authors decided to separate them into 3 sets: 30 for evolving, 30 for in-distribution test, and 30 for out-of-distribution test. So, the difference is that we evolve on a smaller set, but for evaluation, we are the same, at a ~90 level.
Gate. Appendix K is designed to prevent reward hacking by rejecting invalid modifications that exploit the evaluator rather than genuinely improving functionality or security. There's no threshold or calibrated related to the overfitting gate, it's applied on every harness child, and if rejected, this harness is considered to be hacking and invalid.