acceptodds
Withdrawn from ICLR 2027

LLM-Based Evaluation Planning for Heterogeneous Optimization

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Heterogeneous optimization problems require an optimizer to decide not only which candidate solutions to evaluate, but also which objectives and constraints should be evaluated under different costs and fidelities. Existing SOTA methods such as MFE-NSGA-III address this setting with manually designed selection metrics, but their behavior is largely determined by predefined heuristic rules and hyperparameters. In this work, we propose REP-HO, a reasoning-based evaluation planner that replaces the rule-based mixed-fidelity selection step with a large language model (LLM) planner. The LLM receives the same primitive optimization-state information as the heuristic planner, such as surrogate predictions, uncertainties, evaluation costs, and budget status. Each decision is also conditioned on an adaptive nearest-neighbor summary of nearby high-fidelity infill points, and the same planner is instantiated with two open-weight backbones Qwen and Deepseek. We evaluate REP-HO across five heterogeneous regimes covering cost imbalance, constraint difficulty, surrogate fidelity, budget level, and objective conflict. Across these regimes, one of the two variants obtains the best mean ranking score in every regime, REP-HO(Deepseek) is first or second throughout, and the planner shows adaptive allocation behavior, such as prioritizing constraint evaluations under tighter budgets and reallocating constraint evaluation budget when surrogate reliability changes. These results suggest that LLMs can serve as flexible evaluation planners for heterogeneous optimization, offering a general alternative to fixed hand-designed metrics.

Primary area:
optimization
Keywords:
Heterogeneous Optimization, Large Language Model, Multi-fidelity, Multi-Objectives

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_vV13OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The work proposes an LLM-based approach to evaluation-planning in heterogeneous optimization. Evaluation-planning is a step in an evolutionary framework for solving constrained multi-objective optimization problems, which determines which objectives and constraints to evaluate. This is typically done with pre-defined rules. This paper instead replaces that with an LLM prompt. Benchmarking shows that the LLM approach outperforms a recently proposed method MFE-NSGA-III.

Critical strengths
  1. The experimental design and results appear to be sound.
Critical weaknesses
  1. The approach is a direct application of LLM prompting to evaluation-planning in heterogeneous optimization. There is no methodological analysis or insight. As such, it may be out of scope of ICLR.
  2. The work is focus on a narrow application, and as such it is hard to interpret the generality of the approach.
Questions for authors
  1. Address the weaknesses
Reviewer_ywgaOpenReview ↗
Rating
1(−1)
Confidence
3(+1)
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes REP-HO, which replaces the hand-designed selection rule in MFE-NSGA-III with an LLM that decides which objectives and constraints to evaluate at high fidelity, and reports better mean rankings than MFE-NSGA-III and NSGA-III baselines across five synthetic heterogeneity regimes on standard benchmarks.

Critical strengths

This paper addresses a well-motivated problem and offers a simple, general idea, using an off-the-shelf LLM with a fixed prompt as an evaluation planner at a low reported API cost, and it tests it across five controlled heterogeneity regimes with interpretable allocation analyses, which makes the question of whether LLMs can replace hand-designed selection metrics worth studying.

Critical weaknesses
  1. The empirical evidence is unreliable. Many gaps lie within one standard deviation with no significance tests, the reported mean ranks cannot be reproduced from the paper's own tables (Regime 2 and 3 scores sum to 13.69 and 9.81 instead of 15 and 10), and several results are internally inconsistent (e.g., TNK-easy has HV = 0 for SA-NSGA-III while all methods have IGD+ = 0, and Regime 4 is described with different budgets and problems in §3.3, Table 4 and the appendix). The headline claim rests entirely on these rankings, so I cannot trust that REP-HO outperforms MFE-NSGA-III.

  2. The claim that LLM reasoning drives the gains is unsupported. Reasoning is disabled ('reasoning-effort=none'), a budget-matched random allocator matches or beats REP-HO in HV on several Regime 1 settings, and MFE-NSGA-III shows the same adaptive trends with untuned hyperparameters. Without ablations that isolate the planner's decision quality, the gains cannot be attributed to LLM planning.

  3. The code link in Appendix A is dead ("The repository was not found on GitHub"), so the results cannot be verified. Please fix it.

  4. Figure 1 is not a vector graphic, and the prompt example in Appendix J overflows the page margin so that its lines are cut off, which makes it unreadable.

Questions for authors

The code link in Appendix A is not accessible (the repository is not found); could you provide working open-source code, together with easy-to-run reproduction scripts?

Reviewer_mZZPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes REP-HO (Reasoning-based Evaluation Planning for Heterogeneous Optimization), an LLM-based evaluation planner for constrained multi-objective optimization problems with heterogeneous evaluation costs and fidelities. The proposed method replaces the manually designed mixed-fidelity selection mechanism of MFE-NSGA-III with an LLM-based planner while retaining the surrogate-assisted evolutionary optimization backbone. At each generation, the LLM receives structured optimization-state information, including surrogate predictions, uncertainties, nondomination probabilities, constraint feasibility estimates, evaluation costs, remaining budgets, and nearby historical high-fidelity evaluations. Based on this information, it determines which objectives and constraints should be evaluated at high fidelity for each candidate solution. The authors evaluate REP-HO using two LLM backbones, Qwen and DeepSeek, across five experimental regimes covering cost imbalance, constraint difficulty, surrogate fidelity, evaluation budget, and objective conflict. The experimental results show that at least one REP-HO variant achieves the best average ranking in each regime, suggesting that LLMs can provide a competitive alternative to manually designed evaluation-allocation heuristics.

Critical strengths
  1. Relevant and practically motivated problem formulation. The paper addresses an important problem in heterogeneous multi-objective optimization, where objectives and constraints have different evaluation costs and fidelities. Efficiently deciding which evaluations to perform under a limited budget is a meaningful research direction, particularly for expensive black-box optimization tasks.
  2. Interesting integration of LLM-based planning into an established optimization framework. Rather than using an LLM to directly generate candidate solutions, the authors investigate its role as an evaluation-allocation planner. Replacing the manually designed mixed-fidelity selection step of MFE-NSGA-III provides a relatively focused investigation of LLM-based decision-making within an optimization algorithm.
  3. Broad evaluation across heterogeneous optimization regimes. The experiments investigate five distinct factors, including cost imbalance, constraint difficulty, surrogate fidelity, budget variation, and objective conflict. Using two different LLM backbones and reporting results over 20 random seeds per setting strengthens the empirical coverage. The results demonstrate competitive performance across multiple configurations
Critical weaknesses
  1. Limited evidence that LLM reasoning is necessary for the observed improvements. The central contribution is replacing a manually designed evaluation-allocation strategy with an LLM. However, the experiments do not sufficiently establish whether the improvement requires an LLM or could be achieved by a simpler adaptive decision policy using the same optimization-state information. Although MFE-NSGA-III and a budget-matched random allocator are included, stronger comparisons with lightweight cost-aware heuristics, uncertainty-based adaptive policies, or learned allocation strategies are missing. Consequently, it remains unclear whether the LLM provides a meaningful advantage over simpler alternatives.
  2. Performance improvements over the strongest baseline are often relatively small. Although REP-HO achieves favorable average rankings, its numerical improvements over MFE-NSGA-III are frequently modest and are not consistent across individual benchmarks. For example, in Regime 1, REP-HO(DeepSeek) obtains an average ranking of 1.958 compared with 2.188 for MFE-NSGA-III. In Regime 5, the corresponding rankings are 1.958 and 2.062. Moreover, MFE-NSGA-III achieves better hypervolume results on all three Carside cost configurations in Regime 1. Given the reported standard deviations, the paper would benefit from paired statistical tests, confidence intervals for performance differences, and a clearer analysis of when the proposed method provides practically meaningful improvements.
  3. Insufficient isolation of the adaptive nearest-neighbor context. The proposed LLM planner receives additional historical high-fidelity information through the adaptive nearest-neighbor context. This information is not explicitly provided to the original MFE-NSGA-III selection rule in the same form, making it difficult to attribute the observed improvements solely to the LLM-based planning mechanism. Although Appendix D includes results without nearest-neighbor context, these results involve a different set of compared methods, partially different budget configurations, and a single LLM backbone. Therefore, they do not constitute a fully controlled comparison with the main method. A direct ablation under identical settings for both LLM backbones would be necessary to establish the contribution of the nearest-neighbor context.
  4. Limited validation of computational efficiency and scalability. Although the estimated monetary cost of LLM inference is relatively low, Appendix F reports substantial wall-clock times, averaging approximately 203 seconds for DeepSeek and 293 seconds for Qwen per run. Since the experiments mainly rely on analytical benchmark problems and normalized evaluation budgets, the practical trade-off between optimization quality and LLM inference overhead remains insufficiently established. Furthermore, the planner makes multiple LLM calls per generation, raising questions about scalability as the number of candidates, objectives, and constraints increases. More comprehensive end-to-end runtime comparisons and experiments involving genuinely expensive evaluations would strengthen the practical claims.
  5. Experimental inconsistencies and limitations in benchmark characterization. Several inconsistencies require clarification. Section 3.3 and Appendix C.4 describe Regime 4 using budgets of {200, 300, 400} and the problems TNK, MW3, and Carside, whereas Tables 4 and 11 report budgets of {200, 400, 600} and replace Carside with Welded Beam. Additionally, the high-correlation surrogate settings do not reach the stated mean objective Spearman correlation target of above 0.8 in Table 9. The characterization of objective conflict also deserves further justification. Pareto-front curvature measures the geometric nonlinearity of the trade-off surface, but it does not necessarily quantify the degree of objective conflict. Thus, the conclusions regarding adaptation to objective conflict are not yet fully convincing.
Questions for authors

Q1. What evidence demonstrates that the performance improvements specifically require LLM-based reasoning? Could the authors compare REP-HO against additional lightweight adaptive allocation strategies, such as cost-normalized uncertainty selection, uncertainty-and-feasibility heuristics, or a simple learned allocation policy using exactly the same input features and budget constraints? Q2. How much improvement is directly attributable to the adaptive nearest-neighbor context? Could the authors provide controlled ablations of REP-HO with and without nearest-neighbor context for both LLM backbones, using identical random seeds, benchmark configurations, and evaluation budgets? It would also be informative to evaluate a non-LLM baseline that explicitly uses the same historical information. Q3. Are the performance differences statistically significant? Given that several improvements over MFE-NSGA-III are relatively small, could the authors provide paired statistical significance tests or confidence intervals for the differences in HV and IGD+? In particular, which benchmark configurations demonstrate statistically and practically meaningful improvements? Q4. How does the method handle global budget consistency across multiple LLM calls? Since each generation is divided into batches of ten candidates, how does REP-HO coordinate budget allocation across different batches? Is the remaining budget updated after each batch, and can the order in which batches are processed influence the final evaluation decisions? Have the authors examined sensitivity to candidate ordering? Q5. Could the authors clarify the experimental configurations and the evaluation protocol? Why do the budget values and benchmark problems in Regime 4 differ between Section 3.3, Appendix C.4, and Tables 4 and 11? Additionally, when computing the final HV and IGD+ scores, are all objective and constraint values verified using high-fidelity evaluations, or can surrogate-estimated values remain in the final approximation set? If verification is performed, how is its cost accounted for? Q6. How does REP-HO scale to larger and more realistic heterogeneous optimization problems? Could the authors report end-to-end runtime comparisons with the strongest baseline and discuss the impact of increasing the number of objectives, constraints, candidates, and historical evaluations? Experiments involving genuinely expensive evaluations would be valuable for establishing whether the quality improvements justify LLM inference overhead.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.