- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes BeNav, a behavior-space navigation framework for LLM-driven automatic heuristic design. BeNav maps each candidate algorithm to a 4D interpretable behavioral coordinate, covering search style, information usage, uncertainty, and adaptivity, and uses this coordinate to steer algorithm generation via three mechanisms: behavioral composition, behavioral diversion, and local exploitation. Experiments across four AHD settings show consistent improvements over recent LLM-AHD baselines, with ablation studies supporting the contributions of the proposed components.
Critical strengths
The paper presents a clear, timely, and intuitive perspective on algorithm search. Moving from treating algorithm behavior as a descriptive property to explicitly using it as a search signal is an interesting and potentially useful idea.
Critical weaknesses
-
The paper does not sufficiently establish that the proposed behavior space is actually controllable for algorithm generation. The key premise of BeNav is that a behavioral representation can serve not only as a description of an existing algorithm, but also as a target for generating new algorithms. However, the paper mainly demonstrates the forward mapping from an algorithm to its behavioral coordinate. The reverse direction is much less clear: given a target behavior, can the LLM reliably generate an algorithm that exhibits that behavior? For example, Behavioral Composition defines the offspring behavior as an intended interpolation between two parents, but the paper does not directly measure how closely the generated offspring reaches the intended behavioral target. The reported performance improvements and behavioral clustering results therefore show that BeNav is effective, but do not clearly establish that the proposed behavior space is controllable or that the intended behavioral transitions actually occur.
-
The experimental comparison does not fully resolve search-effort fairness, making the role of behavioral navigation difficult to isolate. All methods are constrained by the same 500 function evaluations, but BeNav additionally performs behavior extraction and behavioral navigation operations. The paper does not clearly report the number of generated candidate algorithms, LLM generation calls, behavior-extraction calls, reflection calls, or total LLM token/computational cost under this budget. Consequently, it is unclear whether BeNav achieves better results through more effective navigation or partly through a different amount of algorithm-generation/search effort. Reporting these quantities would be important for establishing a fair comparison.
-
The paper's distinction from selection-based search is not entirely convincing. The paper criticizes existing approaches for primarily using performance or other signals to select promising parents, arguing that this only indirectly determines the direction of search. However, Behavioral Composition in BeNav also first selects two parents based on fitness and then uses their behavioral representations to guide offspring generation. Thus, the proposed method still relies on parent selection as an important mechanism for determining the search direction. It would be useful to clarify what fundamentally distinguishes this process from existing selection-guided evolutionary search, beyond additionally providing behavioral information to the LLM.
Questions for authors
-
Can the authors provide direct evidence that a specified behavioral target leads to a generated algorithm whose measured behavior moves toward that target? More importantly, what is the fundamental advantage of using a behavioral representation as a target over simply using performance as the target when influencing the algorithm search space? Does the behavioral representation actually provide stronger or more controllable guidance for generation, and can this improvement in controllability be quantitatively measured?
-
Under the 500-FE budget, how many candidate algorithms and LLM calls are made by BeNav versus each baseline? In particular, how many calls are used for behavior extraction, reflection, and algorithm generation, and what is the corresponding total token/computational cost?
2 comments
I believe the central controllability concern is only partially addressed. The reported adherence criteria measure relatively coarse behavioral constraints rather than how closely offspring reach their intended targets. For example, satisfying the expanded coordinate intervals in Composition does not necessarily indicate proximity to the intended interpolated behavior, while a positive directional projection in Diversion does not establish meaningful progress toward the target. These results support a tendency toward the intended behavioral regions or directions, but do not yet establish reliable target-conditioned generation.
A more direct experiment would be to take two parent algorithms with behavioral vectors A and B, specify an expected offspring behavior C (e.g., an interpolated point between A and B), and measure the deviation between the generated offspring's actual behavioral vector and this target. Repeating this experiment across multiple parent pairs (e.g., 10 pairs), with multiple offspring generated for each pair, and reporting the average deviation would provide a more direct and intuitive assessment of the controllability of the mapping from behavior space to algorithm space.
Regarding search-effort fairness, I appreciate the authors' transparency. Nevertheless, equal FE budgets only establish fairness in algorithm-evaluation effort, not in total LLM inference resources. Without the corresponding generation, extraction, and reflection call counts or token costs, the resource efficiency of BeNav remains unclear.
My overall assessment remains unchanged, and I maintain my original rating.
Response: Navigation Evidence, Search Effort, and Parent Selection
The reviewer's thoughtful assessment and constructive suggestions are greatly appreciated. In response to the concerns about controllability, search effort, and parent selection, we have clarified the method and provided additional empirical evidence.
1. Direct evidence of behavioral guidance and its advantage over a performance target
Appendix B.1, Table 7 reports a supplementary CVRP-ACO run recording operator inputs and re-extracted offspring coordinates. Its Diversion trigger threshold is set to zero to increase observations specifically for this diagnostic run.
Among successfully executed offspring with evaluable coordinates, explicit post-hoc criteria yield:
For Diversion, the criterion is
(b_child - b_ref) · (b_target - b_ref) > 0, where the dot denotes the vector inner product. The case study supports a directional tendency and directly measures behavioral combination, local preservation, and directional guidance, complementing performance and clustering results. These measurements characterize adherence to behavioral regions and directions rather than exact target-coordinate matching.Appendix B.1, Table 5 also reports ICC(2,1) values of 0.765-0.910 across 160 algorithms with five extractions each. This supports extraction repeatability under the evaluated prompt and model configuration. Table 6 connects coordinates to execution measurements for 500 CVRP-ACO and 495 TSP-GLS algorithms; UN concordance reaches 73.9% and 74.8% at minimum coordinate separation 0.4.
Performance specifies how candidate quality is evaluated. Behavioral guidance additionally specifies characteristics of the modification to pursue: combining existing mechanisms, preserving a local behavioral pattern, or exploring another behavioral region. BeNav uses performance to guide quality-based selection and behavioral information to condition generation.
To examine numerical-coordinate contributions, Section 4.3 and Table 4 add BeNav-Random. It retains strategy extraction and semantic descriptions but replaces coordinates used for storage and downstream navigation with uniformly sampled vectors in the four-dimensional unit cube, fixed for each distinct implementation within a run.
Three-run MKP-ACO objectives (mean +/- standard deviation) are:
Random replacement lowers mean performance at all three sizes and substantially increases run-to-run variability. This control supports the usefulness of numerical behavioral information within the generation pipeline, while the offspring analysis measures adherence to the intended operations.
2. Search-effort fairness and LLM overhead
The shared backbone and nominal 500-FE budget in Section 4.1 control model choice and algorithm-evaluation effort. We agree that equal evaluation budgets do not imply equal total inference cost. Generated candidates, generation requests, extraction requests, reflection requests, and token or timing costs are distinct quantities.
BeNav introduces additional overhead through behavioral extraction and behavioral context in generation and reflection prompts. Each extraction request jointly produces the semantic description and all four coordinates; coordinate operations, retrieval, and pruning are local computations. We have not completed a systematic inference-cost comparison covering all baselines. Our performance claims concern solution quality under a common algorithm-evaluation budget, rather than superiority under equal token or wall-clock budgets. The random-coordinate experiment provides a complementary control for coordinate information while retaining extraction and semantic descriptions.
3. Distinction from selection-based search
We agree that parent selection remains important. As described in Section 3.3, Composition provides parent implementations and behavioral representations to guide a transition between their behavioral patterns. Diversion constructs a target behavioral region and provides a feasible reference implementation. Selection identifies starting points; behavioral navigation adds a specification for subsequent generation. These operations are complementary, and the contribution lies in the additional behavioral guidance.