acceptodds
Under review as a conference paper at ICLR 2027

SemanticOpt: Towards LLM-Based Semantic Black-Box Optimization

Abstract

Optimizing expensive black-box systems often requires both domain knowledge and effective sequential decision making. Bayesian optimization (BO) provides strong numerical refinement and uncertainty-aware search, but typically cannot directly use broader semantic information such as expert knowledge, scientific documentation, or prior experiments. Large language models (LLMs) can exploit such information to propose strong initial configurations, but their optimization performance often plateaus as numerical observations accumulate. We introduce SemanticOpt, an LLM-based optimizer that learns model-based optimization behavior from BO trajectories. We construct more than 100,000 optimization trajectories combining context-informed LLM initialization with multiple BO methods across diverse synthetic and real-world-like function spaces, and fine-tune an LLM to jointly propose candidate configurations and predict objective value, uncertainty, and failure probability. Across 42 scientific and engineering black-box optimization problems, SemanticOpt achieves a strong semantic initialization relative to BO while substantially improving iterative refinement over longer optimization trajectories. Its predictions also become more accurate and better calibrated as observations accumulate, and the model is more robust than the base LLM to misleading semantic information. These results suggest that supervision from numerical optimizers can teach LLMs to better incorporate accumulating numerical evidence while preserving their ability to exploit natural-language priors.

Primary area:
foundation or frontier models, including LLMs
Keywords:
black-box optimization, Bayesian optimization, large language models, LLM fine-tuning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_Ch9UOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces SemanticOpt, an LLM-based optimizer for black-box optimization. An LLM (Qwen 3.5 27B) is directly fine-tuned with LoRA on more than 100,000100,000 optimization trajectories. Given the semantic context and observation history, it jointly proposes candidate configurations and predict objective values with uncertainty, and failure probability. The trajectories are collected on two types of function spaces: GP-based functions, and LLM-generated function spaces with semantic context, where the initializations are also LLM-generated. The remaining queries come from the best-performing HEBO method for given budget. At inference, SemanticOpt gives strong context-informed initializations while being able to continually improve as observations accumulate, outperforming the base model on average at all context levels, and is more robust to counterfactual, mismatched and adversarial semantic information.

Critical strengths
  1. The paper is well written and easy to follow.
  2. The function space generation part is especially interesting and useful.
  3. The experiments answer meaningful questions, including how the proposed method performs at different context levels, and how it reacts to imperfect semantic information compared to the base model.
Critical weaknesses
  1. The function spaces with semantic information are generated by an LLM. It’s unclear: 1) whether the generated problems are realistic and diverse, and 2) whether training and test problems overlap.
  2. SemanticOpt is supervised fine-tuned on HEBO trajectories, without on-policy data. Therefore, apart from the context-informed initializations, it is not expected to exceed the teachers. Also, since it conditions on its own queries which are likely to differ from the teachers’, the errors might occur and accumulate. Additionally, fine-tuning appears to degrade the base models’ initialization quality.
Questions for authors
  1. See Weakness 1: could you clarify on how you ensure the training data to be realistic and diverse, and how you check for overlap between training and test problems?
  2. See Weakness 2: did you consider on-policy training, e.g., DAgger or RL fine-tuning?
  3. How does SemanticOpt perform against the baselines on standard BO benchmarks without semantic information?
  4. Even though the authors consider it as out-of-scope to compare against other LLM-based baselines, to me it’s still worthy of learning whether the fine-tuning cost brings clear gains over training-free LLM-based methods with a base model of comparable strength.
  5. Do the LLM-generated training problems include counterfactual, mismatched, and adversarial semantic information? If not, where does the robustness come from?
  6. SemanticOpt is trained on trajectories with 10, 25, 50, and 100 total steps. How does it perform with longer budgets (e.g., 200 steps)? Given the maximum sequence length, what are the longest histories that SemanticOpt can handle?
4 comments
AuthorsOpenReview ↗

Update Overview

Thank you for your careful reading and constructive comments. We have run several new experiments and revised the paper based on your useful feedback. All new or changed text is marked in blue within the updated paper.

Key new experiments:

  1. LLM-based optimization methods. We have added suggested methods, including LLAMBO, LGBO, LLINBO-Transient, LLINBO-Justify, LABO and GOLLuM, on all 42 benchmarks, with Qwen 3.5 27B as the LLM (GOLLuM uses its own T5 encoder). Overall results are available in Figure 9.
  2. Same initial configurations. SemanticOpt, LGBO and the three HEBO teachers continue from the same five Qwen 3.5 27B configurations, which separates refinement from initialization (Appendix G).
  3. Misleading context. We show SemanticOpt has less degradation to adversarial context than LGBO and the base model (Figure 10).
  4. Longer budgets and standard benchmarks. We test with 100 evaluations (Appendix I), and standard test problems and BBOB test functions without context (Appendix H).

Core claims:

  • Refinement without a GP. LLMs propose strong initial configurations from semantic context, but their search stalls as observations accumulate. Methods such as LGBO address this by fitting a GP around the LLM. SemanticOpt instead learns this refinement within the LLM: from the same starts, it refines as well as LGBO and the GP and RF teachers, while the base model plateaus (Appendix G).
  • Robustness. Misleading context costs SemanticOpt far less than LGBO or the base model, because its initial proposals are degraded far less (Figure 10, Appendix L).
AuthorsOpenReview ↗

Response to Specific Questions

We hope to address your specific comments below.

Q1. Realism, diversity and overlap of the training problems

The 1,800 training problems span a diverse set of problems, dimensions, and parameter types. Their objectives are surrogates fitted to generated points, so no benchmark data is used in training. We evaluate the dataset and benchmark using textual similarity scores and find that the gain over the base model is unrelated to similarity (Appendix J).

Q2. Off-policy training, error accumulation and initialization

We agree that, from the same starts, SemanticOpt matches rather than exceeds its teachers. Its contribution is to reach their level within the LLM, where the base model plateaus. We see no error accumulation within the trained length: between 50 and 100 evaluations, SemanticOpt keeps improving and stays level with the teachers (Appendix I).

We also agree on initialization. SemanticOpt's first proposals are weaker than the base model's, likely because the training trajectories started from a smaller model's proposals, and given the base model's starts instead, SemanticOpt scores higher (Appendix G). On-policy training such as DAgger or RL is a principled fix and could be an interesting opportunity for future work.

Q3. Standard BO benchmarks without semantic information

We evaluated SemanticOpt, the base model and the HEBO teachers on 33 standard test functions and the 24 BBOB functions, with anonymized parameters and no context (Appendix H):

Standard (33)BBOB (24)
SemanticOpt0.5780.843
Qwen 3.5 27B0.4710.713
HEBO teachers0.524–0.6080.771–0.881

The learned refinement carries over without context. SemanticOpt improves clearly on its base model and falls within the range of its teachers.

Q4. Comparison with training-free LLM methods on a comparable base model

We have added the suggested methods, LLAMBO, LGBO, LLINBO-Transient, LLINBO-Justify, LABO and GOLLuM, on all 42 benchmarks and the same context. Overall results are available in Figure 9. Scores after 50 evaluations, each method from its own initial configurations, are summarized below:

MethodScore
LGBO0.854
SemanticOpt0.827
LLINBO-Transient0.809
LLINBO-Justify0.808
LABO0.786
Qwen 3.5 27B0.769
LLAMBO0.767
GOLLuM0.713

Fine-tuning brings a clear gain over prompting the same base model (0.827 against 0.769), and SemanticOpt also outperforms LLAMBO and GOLLuM. It is level with both LLINBO variants and LABO, and only LGBO ends higher (0.854).

Additionally, LGBO’s lead over SemanticOpt comes from initialization rather than refinement. LGBO takes its five initial configurations from the base model, and these are stronger than SemanticOpt’s own (0.61 against 0.48 after five evaluations). When both methods continue from the same five configurations, SemanticOpt refines as well as LGBO (0.836 against 0.844, p = 0.49; Appendix G).

SemanticOpt therefore matches LGBO's refinement within a single model, without fitting a GP. LGBO queries the base model throughout the search, and under misleading context it loses 0.14, against 0.04 for SemanticOpt (Figure 10). The remaining gap is SemanticOpt's weaker initialization, which is the main step for future work.

Q5. Source of robustness to misleading context

No, the training data contains no specified counterfactual, mismatched or adversarial context. However, its contexts are generated by an LLM together with the objective and are never checked against it, and in each trajectory the BO teacher explores beyond the context-informed initial proposals. We believe the model learns to treat context as a prior rather than an instruction. Consistent with this, its initial proposals lose far less under adversarial context than the base model’s (Figure 10), and it keeps exploring where the base model does not (Appendix D).

Q6. Longer budgets

With a budget of 100 evaluations (Figure 24, Appendix I), SemanticOpt is level with its teachers, while the base model barely improves after 50. Training used trajectories of up to 100 evaluations and 24,576 tokens. As SemanticOpt was only trained up to 100 evaluations and many of our training functions reach near-optimum after this long, we do not expect the performance to generalize beyond this range. Further training and dataset generation could help alleviate this issue.

We hope these additions address your concerns, and we would be happy to answer any further questions.

Thank you for the rebuttal and the additional results.

The finding that the gain over the base model is not correlated with the TF-IDF similarity between training and test problems is interesting. A future investigation using a metric that reflects semantic similarity between tasks, or a principled criterion for designing the LLM-generated training priors, would be insightful.

The new results on standard BO benchmarks without semantic information, and the comparison with LLM-based optimizers based on a frozen LLM, better position the proposed method: SemanticOpt does not outperform the strongest BO teacher or the frozen-LLM method (LGBO), but it offers an alternative approach that does not rely on an external surrogate, and it is more robust to adversarial context than frozen-LLM baselines and the base model.

I believe the additional results and the revised contributions and scope in the updated manuscript improve the work, and I am happy to increase my score.

Thank you for raising your score and for your thoughtful engagement. We agree that a semantic similarity measure between tasks, or a principled criterion for designing the LLM-generated training problems, would be a valuable follow-up.

Reviewer_vGBhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SemanticOpt, an LLM-based semantic black-box optimization framework that leverages BO-generated trajectories to improve iterative optimization while preserving the ability to utilize semantic prior knowledge. The authors fine-tune Qwen 3.5 27B on over 100000 optimization trajectories generated using multiple BO methods, training the model to jointly propose candidate configurations and predict objective values, uncertainties, and failure probabilities. Experiments on 42 scientific and engineering benchmarks show that SemanticOpt improves iterative optimization performance, uncertainty calibration, and robustness to misleading semantic information compared to the base LLM, while achieving refinement performance comparable to its BO teachers.

Critical strengths
  1. I appreciate the shared-initialization experiments in Figure 8, which help separate the effects of initialization quality from subsequent optimization performance. This makes the reported improvements more convincing.
  2. The evaluation across different semantic context levels and misleading contexts provides useful insights into how LLM-based optimizers utilize and adapt semantic prior knowledge.
Critical weaknesses
  1. Several design choices are introduced together, including BO trajectory supervision, surrogate prediction targets, and multi-teacher selection. It remains unclear which components contribute most to the observed improvements.
  2. While the paper focuses on the benefits of BO trajectory supervision, the evaluation mainly compares SemanticOpt with the base Qwen model and BO teachers. Since methods such as LLAMBO and LGBO also address LLM-based black-box optimization, comparisons with these approaches would help establish the competitiveness of SemanticOpt beyond the improvements over its base model.
Questions for authors

How much of the performance improvement comes from BO action imitation versus surrogate prediction supervision? Would action-only fine-tuning achieve similar results? How does SemanticOpt compare with existing LLM-based black-box optimizers, such as LLAMBO and LGBO, under comparable evaluation budgets and semantic context? The appendix shows that LLM initialization followed by conventional BO already achieves competitive performance. What additional advantages does SemanticOpt offer over this approach, particularly in terms of optimization performance and computational cost?

4 comments
AuthorsOpenReview ↗

Update Overview

Thank you for your careful reading and constructive comments. We have run several new experiments and revised the paper based on your useful feedback. All new or changed text is marked in blue within the updated paper.

Key new experiments:

  1. LLM-based optimization methods. We have added suggested methods, including LLAMBO, LGBO, LLINBO-Transient, LLINBO-Justify, LABO and GOLLuM, on all 42 benchmarks, with Qwen 3.5 27B as the LLM (GOLLuM uses its own T5 encoder). Overall results are available in Figure 9.
  2. Same initial configurations. SemanticOpt, LGBO and the three HEBO teachers continue from the same five Qwen 3.5 27B configurations, which separates refinement from initialization (Appendix G).
  3. Misleading context. We show SemanticOpt has less degradation to adversarial context than LGBO and the base model (Figure 10).
  4. Longer budgets and standard benchmarks. We test with 100 evaluations (Appendix I), and standard test problems and BBOB test functions without context (Appendix H).

Core claims:

  • Refinement without a GP. LLMs propose strong initial configurations from semantic context, but their search stalls as observations accumulate. Methods such as LGBO address this by fitting a GP around the LLM. SemanticOpt instead learns this refinement within the LLM: from the same starts, it refines as well as LGBO and the GP and RF teachers, while the base model plateaus (Appendix G).
  • Robustness. Misleading context costs SemanticOpt far less than LGBO or the base model, because its initial proposals are degraded far less (Figure 10, Appendix L).
AuthorsOpenReview ↗

Response to Specific Questions

We hope to address your specific comments below.

Q1. Contribution of the supervision components

A full ablation requires retraining (about 2.5 days on 4 GPUs per run), which we have not had the resources to complete in this rebuttal phase. We wish to show that a plausible data generation and fine-tuning scheme can yield improvements in iterative search capabilities, which base LLMs and other LLM-based harnesses struggle with. We agree that further ablations of these design choices would be beneficial, but were not within the scope of our resources.

Q2. Comparison with LLAMBO and LGBO

We have added the suggested methods, LLAMBO, LGBO, LLINBO-Transient, LLINBO-Justify, LABO and GOLLuM, on all 42 benchmarks and the same context. Overall results are available in Figure 9. Scores after 50 evaluations, each method from its own initial configurations, are summarized below:

MethodScore
LGBO0.854
SemanticOpt0.827
LLINBO-Transient0.809
LLINBO-Justify0.808
LABO0.786
Qwen 3.5 27B0.769
LLAMBO0.767
GOLLuM0.713

Fine-tuning brings a clear gain over prompting the same base model (0.827 against 0.769), and SemanticOpt also outperforms LLAMBO and GOLLuM. It is level with both LLINBO variants and LABO, and only LGBO ends higher (0.854).

Additionally, LGBO’s lead over SemanticOpt comes from initialization rather than refinement. LGBO takes its five initial configurations from the base model, and these are stronger than SemanticOpt’s own (0.61 against 0.48 after five evaluations). When both methods continue from the same five configurations, SemanticOpt refines as well as LGBO (0.836 against 0.844, p = 0.49; Appendix G).

SemanticOpt therefore matches LGBO's refinement within a single model, without fitting a GP. LGBO queries the base model throughout the search, and under misleading context it loses 0.14, against 0.04 for SemanticOpt (Figure 10). The remaining gap is SemanticOpt's weaker initialization, which is the main step for future work.

Q3. Benefit over initialization followed by BO

We agree that LLM-initialized BO reaches at least the same final score. Our contribution is an alternative that performs this refinement within the LLM. From the same starts, SemanticOpt refines as well as LGBO and the GP and RF teachers and ends slightly below SVIDKL, while the base model plateaus (Appendix G).

SemanticOpt also provides its own mean and uncertainty predictions, which improve as observations accumulate (Figure 11), and it keeps up with the teachers at 100 evaluations (Appendix I). Under misleading context, it loses far less than LGBO, because its initial proposals are degraded far less (Figure 10).

We hope these additions address your concerns, and we would be happy to answer any further questions.

Thank you for your response. However, I will maintain my original score, as I believe the work does not yet meet the acceptance standard for ICLR. So sorry.

Thank you for considering our response. Your feedback led to the comparison with LLAMBO, LGBO and the other LLM-based optimizers, which strengthened the paper.

Reviewer_o9jHOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SemanticOpt, a method that fine-tunes a 27B LLM to act as an evidence-aware black-box optimizer by distilling BO trajectories. The model outputs a predictive distribution plus a failure probability, and at inference it samples multiple candidates and selects via success-weighted expected improvement. Experiments on 42 benchmarks show improvements in iterative refinement and uncertainty calibration.

Critical strengths

The design is systematic and reasonable. It places data construction, label generation, fine tuning and the selection rule at inference in one pipeline and outputs a predictive distribution.

Good evaluation coverage. The paper includes full-context, counterfactual, mismatched, and adversarial settings, and also separates shared-initialization refinement from warm-start effects via replay.

Critical weaknesses
  1. The paper’s “semantic optimization” narrative is over-claimed. While the method leverages natural-language context, the experiments do not convincingly disentangle semantic understanding from simpler effects such as parameter identity/formatting or improved warm-start and refinement dynamics. The current evidence is not sufficient to support the paper’s claim.

  2. The contribution is a combination of two existing lines, and neither line is tested on its own. ZeroShotOpt trains a transformer to imitate BO trajectories on synthetic functions, and LLAMBO, LGBO and GOLLuM put language model context inside a Bayesian optimization pipeline. The paper runs neither as a comparison arm, so its novelty rests on the combination alone.

Questions for authors
  1. The trajectory kept per function space is the best of the three teachers, while the refinement gap uses their mean. How should teacher level performance be read?

  2. How many model calls and how much wall clock time does one full run take, against one teacher run?

  3. The base model nearly repeats an earlier proposal in most steps. Could the difference on misleading context follow from this alone?

  4. Initialization followed by the numerical teacher ends at least as high as the model alone. What does the end to end model add?

3 comments
AuthorsOpenReview ↗

Update Overview

Thank you for your careful reading and constructive comments. We have run several new experiments and revised the paper based on your useful feedback. All new or changed text is marked in blue within the updated paper.

Key new experiments:

  1. LLM-based optimization methods. We have added suggested methods, including LLAMBO, LGBO, LLINBO-Transient, LLINBO-Justify, LABO and GOLLuM, on all 42 benchmarks, with Qwen 3.5 27B as the LLM (GOLLuM uses its own T5 encoder). Overall results are available in Figure 9.
  2. Same initial configurations. SemanticOpt, LGBO and the three HEBO teachers continue from the same five Qwen 3.5 27B configurations, which separates refinement from initialization (Appendix G).
  3. Misleading context. We show SemanticOpt has less degradation to adversarial context than LGBO and the base model (Figure 10).
  4. Longer budgets and standard benchmarks. We test with 100 evaluations (Appendix I), and standard test problems and BBOB test functions without context (Appendix H).

Core claims:

  • Refinement without a GP. LLMs propose strong initial configurations from semantic context, but their search stalls as observations accumulate. Methods such as LGBO address this by fitting a GP around the LLM. SemanticOpt instead learns this refinement within the LLM: from the same starts, it refines as well as LGBO and the GP and RF teachers, while the base model plateaus (Appendix G).
  • Robustness. Misleading context costs SemanticOpt far less than LGBO or the base model, because its initial proposals are degraded far less (Figure 10, Appendix L).
AuthorsOpenReview ↗

Response to Specific Questions

We hope to address your specific questions below.

W1. Scope of the semantic optimization claim

Based on your advice, we have narrowed the claim to match the evidence. Semantic context mainly drives initialization and robustness: full context beats no context on 33 of 42 benchmarks, a counterfactual context performs just as well, and the gain comes from the descriptions rather than the parameter names (Appendices E and L). Fine-tuning then provides the refinement. From the same starts, SemanticOpt refines as well as LGBO and its teachers, while the base model plateaus (Appendix G).

W2. Comparison with LLM-based and learned optimizers

We have added the suggested methods, LLAMBO, LGBO, LLINBO-Transient, LLINBO-Justify, LABO, and GOLLuM, on all 42 benchmarks and the same context. Overall results are available in Figure 9. Scores after 50 evaluations, each method from its own initial configurations, are summarized below:

MethodScore
LGBO0.854
SemanticOpt0.827
LLINBO-Transient0.809
LLINBO-Justify0.808
LABO0.786
Qwen 3.5 27B0.769
LLAMBO0.767
GOLLuM0.713

Fine-tuning brings a clear gain over prompting the same base model (0.827 against 0.769), and SemanticOpt also outperforms LLAMBO and GOLLuM. It is level with both LLINBO variants and LABO, and only LGBO ends higher (0.854).

Additionally, LGBO’s lead over SemanticOpt comes from initialization rather than refinement. LGBO takes its five initial configurations from the base model, and these are stronger than SemanticOpt’s own (0.61 against 0.48 after five evaluations). When both methods continue from the same five configurations, SemanticOpt refines as well as LGBO (0.836 against 0.844, p = 0.49; Appendix G).

SemanticOpt therefore matches LGBO's refinement within a single model, without fitting a GP. LGBO queries the base model throughout the search, and under misleading context it loses 0.14, against 0.04 for SemanticOpt (Figure 10). The remaining gap is SemanticOpt's weaker initialization, which is the main step for future work.

Q1. Interpretation of teacher-level performance

“Teacher-level” refers to the individual teachers, not to the best teacher chosen in hindsight. Training keeps the best teacher for each function space, but making that choice at test time requires knowing the results in advance, which is not possible, so the refinement gap uses the mean of the three teachers. From the same starts, SemanticOpt is also level with GP and RF individually and slightly below SVIDKL. Against the hindsight-best teacher in Appendix G (six problems per benchmark), it is level when each method uses its own starts, random configurations for the teachers (0.819 against 0.825), and behind when all start from the same configurations (0.836 against 0.878).

Q2. Model calls and wall-clock time

Based on your advice, we have added wall-clock tests, reported as the median time per run (Appendix K), tested on one H200 for the LLM-based methods and 15 CPU cores for the teachers:

MethodLLM requestsMinutes
SemanticOpt462.4
HEBO teachers01.6–4.3
LGBO4712.1
LLINBO-Transient40.8
LLINBO-Justify462.1
LABO696.4
LLAMBO81916.2
GOLLuM0 (T5 encoder)11.3

SemanticOpt makes one LLM request for its initial configurations and one per evaluation, the same as direct prompting.

Q3. Repeated proposals under misleading context

The base model repeats about as often with full as with adversarial context, so repetition limits its refinement under every context and, under misleading context, keeps it near its poor starts. From the same adversarial starts, as shown in Appendix L, SemanticOpt reaches 0.756 after 50 evaluations and the HEBO teachers 0.784, while the base model stays at 0.455. SemanticOpt's remaining advantage is its own initial proposals, which adversarial context degrades far less (0.37 against 0.10 after five evaluations).

Q4. Benefit over initialization followed by a teacher

We agree that an LLM-initialized hybrid BO approach reaches at least the same final score. SemanticOpt is an alternative that learns this refinement within the LLM, so the context keeps shaping its proposals after initialization, whereas a hand-off to BO discards it. From the same starts, it is ahead of the GP and RF teachers after 10 evaluations and ends level with them and slightly below SVIDKL, as a single model with no surrogate to fit (Appendix G). Its own mean and uncertainty predictions are close to those of the teachers' surrogates (Figure 11), and its refinement carries over to benchmarks without context (Appendix H). Against base-model initialization followed by BO and with adversarial-context, it starts far ahead (0.37 against 0.10 after five evaluations) and is level after 50 (Appendix L).

We hope these additions address your concerns, and we would be happy to answer any further questions.

Thank you again for your detailed review. We hope the added LLM-based baselines (Figure 9), the same-start comparison (Appendix G), the adversarial-start comparison (Appendix L) and the cost comparison (Appendix K) address your questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.