acceptodds
Under review as a conference paper at ICLR 2027

MOSAIC: Modular Search Architecture Evolution for LLM-Driven Heuristic Design in Multi-Objective Combinatorial Optimization

Abstract

Large language model (LLM)-driven automated heuristic design (AHD) has attracted increasing attention for its potential to reduce labor-intensive manual design. However, existing LLM-driven multi-operator AHD methods typically optimize search behaviors within predefined operator organizations, which constrains the exploration of potential synergies among different search behaviors. To address this limitation, we propose MOSAIC, a modular search architecture evolution framework for LLM-driven AHD. MOSAIC represents each heuristic as an ordered, variable-length architecture composed of multiple search modules, where each module implements a search behavior and the architecture determines how these behaviors are organized. Based on this representation, MOSAIC jointly evolves modules and architectures through module rewriting and architecture variation. The former rewrites modules to explore and refine search behaviors, while the latter evolves module composition, execution order, and architecture length, enabling broader exploration of potential synergies among search behaviors. Experiments on multiple multi-objective combinatorial optimization problems show that MOSAIC outperforms advanced LLM-driven AHD methods and exhibits strong cross-scale generalization.

Primary area:
optimization
Keywords:
Combinatorial Optimization, Heuristic Design, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_dnkXOpenReview ↗
Rating
3
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MOSAIC is an LLM-driven automated heuristic design (AHD) framework for multi-objective combinatorial optimization (MCOP). A heuristic is represented as an ordered, variable-length sequence of executable "search modules", each implementing one search behavior. MOSAIC jointly evolves both the module behaviors (via LLM module rewriting: Behavioral Rewriting and Adaptive Refinement) and their organization (composition, order, and length, via architecture crossover plus structural mutation), with NSGA-II/MOEA/D survival selection over two HV-based objectives (mean and worst-case performance across instance types). Candidates are always evaluated as complete heuristics. The framework is evaluated on three bi-objective problems (Bi-TSP, Bi-KP, Bi-GVRP) plus a three-objective MOFJSP in the appendix, against multi-operator (E2OC, G-LNS) and single-operator (EoH, FunSearch, ReEvo, MCTS-AHD) baselines, with Wilcoxon + Holm significance tests, ablations, LLM-usage accounting, and cross-scale generalization tests.

Critical strengths
  • Making operator organization (composition, order, length) an explicit, jointly-evolved design dimension is a well-motivated step over E2OC's fixed slots and G-LNS's fixed Destroy–Repair structure, and the permutation analysis of the four-module architecture provides concrete evidence of order-dependent synergy rather than a bare claim.
  • The evaluation is unusually rigorous for this field: Wilcoxon + Holm significance tests, a matched evaluation budget, explicit LLM-usage accounting (Table 6) showing the gains are not bought with more tokens, ablations isolating the two rewriting modes and architecture variation, and released code.
Critical weaknesses
  • The domain where the largest advantage is claimed, Bi-GVRP, is the one where statistical significance is absent (0/2 size groups vs all four single-operator baselines, 1/1 vs multi-operator). The "substantially outperforms" language is therefore stronger than the evidence supports where it matters most, and only three of five Bi-TSP size groups are testable because two contain a single instance.
  • The contribution is an incremental assembly of established ingredients (LLM code generation, NEAT-style evolutionary architecture search, MOEA). The paper itself flags the semantic-drift and scalability limits, and the evaluation covers only three synthetic benchmark MCOPs (one more in the appendix), so the breadth of the "MCOP" claim is not yet established.
Questions for authors
  1. How is the reference Pareto front constructed for each instance, and does it include solutions produced by MOSAIC or any baseline? This determines whether the HV-gap and IGD metrics are fair and achievable.
  2. On Bi-GVRP, MOSAIC shows its largest HV-gap advantage (16.7% vs ~33% at the 100–199 scale) yet Table 3 reports no significant win against any single-operator baseline (0/2). Why is that margin not significant there, and is "substantially outperforms" the right characterization?
  3. MOSAIC's initial architectures already far outrank E2OC's initial designs on Bi-KP (49.9% vs 95.4% gap). How much of the final advantage is attributable to module-pool initialization versus the joint evolution, beyond what the Fixed-Arch ablation isolates?
  4. G-LNS's initial Destroy–Repair design scores 0.00 HV (99.7–100.0% gap) on all three Bi-KP test scales, so its "substantial improvement on Bi-KP" is recovery from a non-functional initialization. Is that initial configuration correct, and does the multi-operator comparison remain meaningful given this?
  5. What explains MOSAIC's advantage over E2OC? The paper reports the result but offers little analysis of the cause; please add a short discussion after the experiments.
9 comments
AuthorsOpenReview ↗

Response (1/7)

W1: Concerns about statistical evidence and performance claims. The domain where the largest advantage is claimed, Bi-GVRP, is the one where statistical significance is absent (0/2 size groups vs all four single-operator baselines, 1/1 vs multi-operator). The "substantially outperforms" language is therefore stronger than the evidence supports where it matters most, and only three of five Bi-TSP size groups are testable because two contain a single instance.

Thank you for highlighting the gap between the performance claims and the statistical evidence. We agree that mean performance improvements should be distinguished from statistical significance, and that the scope of the statistical conclusions should be explicit. We respond as follows.

  1. Statistical evidence on Bi-GVRP. Table 3 shows that MOSAIC’s differences from the four single-operator design methods are not statistically significant in either Bi-GVRP size group. Against E2OC and G-LNS, the differences are significant in one of the two groups for each method. Although Tables 1 and 2 report higher mean HV and lower mean IGD for MOSAIC, these mean differences do not establish statistically significant superiority over all baselines on Bi-GVRP.

  2. Revised performance descriptions. We have revised the phrases “substantially outperforms” and “the largest advantage” concerning Bi-GVRP in the “Comparison with LLM-driven Single-Operator Design Methods” paragraph of Section 3.2, explicitly limiting these descriptions to differences in mean performance.

  3. Statistical testing and its scope. The “Statistical Significance Analysis” paragraph in Section 3.2 and Appendix C.5 already specify that we pair MOSAIC’s performance with each baseline’s performance on the same test instance, after averaging over five runs, and apply two-sided Wilcoxon signed-rank tests with Holm correction. Among the five Bi-TSP size groups, the 150- and 200-city groups each contain only one instance and therefore provide only one paired difference, which is insufficient for an instance-level significance assessment. These two groups were already excluded from statistical testing, with their p-values marked as “–” in Appendix Table 15. The statistical significance conclusions concern only the 100-, 300-, and 500-city groups.

In summary, we have revised the Bi-GVRP descriptions to distinguish mean performance differences from statistical significance. For Bi-TSP, Section 3.2 and Appendix C.5 already state the exclusion of single-instance groups, and the significance conclusions have consistently concerned the remaining three groups.

AuthorsOpenReview ↗

Response (2/7)

W2: Concerns about methodological contribution, limitations, and generality. The contribution is an incremental assembly of established ingredients (LLM code generation, NEAT-style evolutionary architecture search, MOEA). The paper itself flags the semantic-drift and scalability limits, and the evaluation covers only three synthetic benchmark MCOPs (one more in the appendix), so the breadth of the "MCOP" claim is not yet established.

Thank you for examining the methodological contribution and scope of applicability. We respond in terms of the core contribution, the limitations already discussed in the paper, and supplementary evidence across more diverse optimization settings.

  1. Core contribution. Our motivation is to move beyond predefined operator organizations by incorporating module composition, execution order, and architecture length into automated search and optimizing them jointly with module behaviors. This enables the discovery of operator synergies while reducing reliance on manual configuration. Crossover, mutation, and MOEAs serve as the underlying tools; MOSAIC’s methodological contribution lies in the joint design of module behaviors and architecture organization. The motivation and core mechanisms are detailed in our response to Reviewer bDsc’s W1. MOSAIC’s architecture evolution mechanism offers advantages in exploring operator synergies, runtime efficiency, and token consumption. The runtime comparisons are detailed in our response to Reviewer bDsc’s W2, and the token consumption analysis is provided in our response to Reviewer bDsc’s Q2.

  2. Semantic drift and scalability. Sections 4.2 and 4.3 and Appendix D.6 discuss these limitations and directions for improvement.

    • Semantic drift. Current LLM-based automated heuristic design methods commonly face this risk: natural-language prompts cannot strictly constrain the behavior of generated code, and executability and valid outputs do not guarantee faithful implementation of the intended design. By directly reorganizing existing modules, MOSAIC allows part of the search to proceed without regenerating code, reducing the number of generation steps that may introduce semantic drift. Future work will incorporate semantic consistency checks for intended search behaviors and investigate architecture search over verified module pools.

    • Scalability. MOSAIC allows the search space to be adjusted through the module pool size and maximum architecture length, while reusing existing code to explore different module combinations. In Section 4.2, the fixed-module-pool experiment with module rewriting disabled shows that performance generally improves as the pool grows from 200 to 1,000 modules, but declines at 2,000. Appendix D.5 likewise shows that increasing the architecture length upper bound does not consistently yield improvements. These results indicate that larger search spaces require more effective use of a limited evaluation budget. Future work will investigate module selection, redundancy control, and more efficient architecture search to better exploit larger search spaces.

  3. Supplementary evidence of broader applicability. To further assess the framework’s applicability, we add experiments on MONRP, MOSCP, three-objective MOQAP, and five-objective PPP. Together with Bi-TSP, Bi-KP, and Bi-GVRP in the main text and three-objective MOFJSP in the appendix, the evaluation now covers eight classes of combinatorial optimization problems.

    Under the same evaluation budget, MOSAIC achieves higher mean HV than all six baselines in all 13 test-scale groups of the additional problems, providing further evidence of its applicability across different problem types, numbers of objectives, and test scales. The complete data sources, experimental settings, and results are provided in our response to Reviewer bDsc’s W3.

In summary, MOSAIC jointly searches module behaviors and architecture organization to discover operator synergies, while using module reuse to reduce reliance on LLM code generation. The paper explicitly discusses the limitations of semantic drift and architecture-search scalability, together with directions for improvement. The supplementary experiments extend the evaluation to eight classes of combinatorial optimization problems; MOSAIC achieves the highest mean HV in all 13 additional test-scale groups, providing further support for its applicability to broader MCOP settings.

AuthorsOpenReview ↗

Response (3/7)

Q1: Concerns about reference Pareto front construction. How is the reference Pareto front constructed for each instance, and does it include solutions produced by MOSAIC or any baseline? This determines whether the HV-gap and IGD metrics are fair and achievable.

Thank you for highlighting the need to clarify how the reference fronts are constructed. Public reference sets are available for Bi-TSP, Bi-KP, and Bi-GVRP, which we use to compute HV-gap and IGD. For the other problems, we report only HV. The following clarification concerns these three problems.

  1. Construction of the reference front for each instance.

    • Bi-TSP: We select the corresponding public results from the Paquete benchmark page. For euclid, random, and mixed instances, we use the files labeled “best non-dominated set found.” For kro instances, we use specified PD-TPLS results. For example, kroAB100 uses the result corresponding to 2,000 scalarizations and the 3.first.ils configuration.
    • Bi-KP: We use the non dominated points/*.max files accompanying the Xavier instances. We verify solution feasibility and objective values, remove invalid, duplicate, and dominated points, and use the remaining objective vectors as the reference set.
    • Bi-GVRP: We use Approximate_Pareto_Fronts from the publicly available supplementary materials, remove invalid, duplicate, and dominated points, and use the remaining objective vectors as the reference set.

    The reference files for all test instances are provided in the test_sets/ directory at this anonymous code link.

  2. Whether solutions generated by MOSAIC or the baselines are included. No. The reference sets do not include solutions generated by MOSAIC or any baseline in our experiments.

  3. Fairness and attainability of HV-gap and IGD. For each instance, all methods use the same reference set, normalization bounds, and HV reference point, ensuring consistent comparisons. Every objective vector in the reference set corresponds to a feasible solution satisfying that instance’s constraints. Thus, the HV-gap and IGD comparisons are fair, and their reference targets are attainable.

Reviewer_bDscOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MOSAIC, an LLM-driven automated heuristic design framework that jointly evolves search modules and their architectures. It combines module rewriting with architecture crossover and structural mutation to optimize search behaviors, module composition, execution order, and architecture length. Experiments on multi-objective combinatorial optimization problems demonstrate competitive performance against existing single-operator and multi-operator design methods, with promising cross-scale generalization.

Critical strengths
  1. The paper introduces a meaningful extension to automated heuristic design by explicitly optimizing module organization rather than relying on predefined operator structures.
  2. The experimental evaluation is comprehensive, covering multiple optimization problems, competitive baselines, two evolutionary frameworks, and cross-scale generalization.
  3. The ablation studies and module-ordering analysis provide convincing evidence that architecture evolution contributes beyond individual module optimization.
Critical weaknesses
  1. The architecture evolution mechanism primarily relies on conventional crossover and mutation operators. Its algorithmic novelty beyond extending the search space to module organization could be better established.
  2. Although LLM usage is analyzed, the end-to-end computational overhead of architecture evaluation and evolution is not sufficiently quantified through wall-clock comparisons.
  3. The main experiments focus on three bi-objective problems. Despite an additional three-objective scheduling experiment, the generality of the framework to more diverse optimization settings remains insufficiently established.
Questions for authors
  1. How does MOSAIC compare with the strongest baselines under matched end-to-end computational budgets, including architecture evaluation time?
  2. Can the authors clarify which aspects of the architecture evolution mechanism provide advantages beyond conventional evolutionary search?
  3. How sensitive is MOSAIC to the module pool size and architecture length when applied to larger or more complex problems?
8 comments
AuthorsOpenReview ↗

Response (1/6)

W1: Concerns about contribution clarity. The architecture evolution mechanism primarily relies on conventional crossover and mutation operators. Its algorithmic novelty beyond extending the search space to module organization could be better established.

Thank you for highlighting the need to clarify our methodological contribution. We explain its motivation, mechanisms, and empirical support below.

  1. Research motivation. Methods with predefined operator structures optimize search behaviors within specified roles and execution sequences. For example, E2OC starts with Or-opt → 2-opt → 3-opt for TSP, while G-LNS adopts Destroy → Repair. These configurations restrict the search over operator organization and require problem-specific choices: E2OC uses route-based operators for TSP but operation-sequencing and machine-assignment operators for FJSP. Our motivation is to automate these organizational choices while exploring operator synergies.

  2. Core innovation. MOSAIC represents heuristics as variable-length, ordered architectures of executable modules and jointly evolves their behaviors and organization through two mechanisms:

    • Complementary behavior search. Behavioral Rewriting (BR) explores new search behaviors, while Adaptive Refinement (AR) refines existing behaviors.
    • Architecture evolution. Module composition, execution order, and architecture length evolve to identify organizations suited to the generated modules and target problem.

    Crossover and mutation implement structural search; the methodological contribution is their integration with behavior search to jointly design executable modules and their organization.

  3. Empirical evidence. We provide evidence from two perspectives:

    (1) Discovered module synergies. Section 4.1 and Appendix D.1 analyze the best Bi-KP architecture obtained with seed 2027: O782 → O181 → O103 → O181. Its executable code is available in trained_best_architectures/ at this anonymous code link.

    Holding module code and all other settings unchanged, we compare its 12 unique execution orders. The following results reproduce Appendix Table 22, reporting mean ± standard deviation over five independent runs and ranked by mean HV. The best mean for each metric is shown in bold.

    RankExecution orderHV ↑IGD ↓
    1O782 → O103 → O181 → O1810.590 ± 0.0250.219 ± 0.020
    2 (Evolved)O782 → O181 → O103 → O1810.585 ± 0.0260.219 ± 0.022
    3O103 → O782 → O181 → O1810.577 ± 0.0310.219 ± 0.024
    4O181 → O782 → O103 → O1810.576 ± 0.0280.222 ± 0.022
    5O782 → O181 → O181 → O1030.567 ± 0.0360.217 ± 0.023
    6O181 → O782 → O181 → O1030.556 ± 0.0390.223 ± 0.025
    7O103 → O181 → O782 → O1810.550 ± 0.0390.226 ± 0.022
    8O181 → O181 → O782 → O1030.546 ± 0.0430.226 ± 0.022
    9O181 → O103 → O782 → O1810.543 ± 0.0430.224 ± 0.024
    10O103 → O181 → O181 → O7820.523 ± 0.0500.232 ± 0.028
    11O181 → O103 → O181 → O7820.513 ± 0.0550.237 ± 0.029
    12O181 → O181 → O103 → O7820.501 ± 0.0550.238 ± 0.031

    Changing only execution order produces mean HV values of 0.501–0.590; the evolved order ranks second at 0.585. All three orders placing the information-recombination module O782 first rank among the top five, whereas those placing it last occupy the bottom three and have the highest IGD values. This supports an order-dependent synergy between information recombination and subsequent mutation.

    (2) Adaptation to different problems. Given problem-specific representations, constraints, and evaluation information, MOSAIC searches module behaviors and organization without prescribing concrete operator types or execution sequences. Tables 1 and 2 in the main paper report higher mean test HV and lower mean IGD than the compared multi-operator and single-operator methods on Bi-TSP, Bi-KP, and Bi-GVRP, supporting its ability to discover effective configurations for the evaluated problems.

In summary, MOSAIC’s contribution is the joint discovery of module behaviors and organization. The performance comparisons and execution-order analysis support its value for discovering operator synergies and adapting configurations to different problems.

AuthorsOpenReview ↗

Response (2/6)

W2: Concerns about end-to-end computational cost. Although LLM usage is analyzed, the end-to-end computational overhead of architecture evaluation and evolution is not sufficiently quantified through wall-clock comparisons.

Thank you for pointing out the lack of quantitative comparisons of end-to-end computational costs. We provide wall-clock time comparisons and use Bi-TSP to analyze the runtime of individual components in MOSAIC.

Experimental settings. All experiments run on a 10-core Apple M5 CPU with 24 GB of unified memory, using the same budget of 253,000 objective function evaluations per training instance. Details of the datasets and other experimental settings are provided in Section 3.1 and Appendices C.1–C.2.

  1. End-to-end wall-clock time. The following table reports the end-to-end runtime of MOSAIC and the baseline methods on Bi-TSP, Bi-KP, Bi-GVRP, and MOFJSP. The shortest runtime for each problem is shown in bold.

    MethodBi-TSPBi-KPBi-GVRPMOFJSP
    EoH5h 31m 8s3h 33m 10s14h 20m 15s5h 55m 5s
    FunSearch18h 5m 56s11h 5m 13s49h 41m 1s18h 25m 23s
    ReEvo5h 54m 52s3h 53m 38s15h 34m 16s6h 11m 25s
    MCTS-AHD10h 7m 48s6h 18m 7s28h 12m 53s10h 28m 41s
    E2OC3h 29m 54s2h 19m 47s8h 56m 53s3h 28m 29s
    G-LNS10h 14m 0s6h 28m 14s28h 15m 23s10h 15m 33s
    MOSAIC2h 44m 38s1h 44m 43s7h 20m 29s2h 49m 6s

    MOSAIC achieves the shortest end-to-end runtime on all four problems. Compared with E2OC, the second-fastest method, MOSAIC reduces runtime by approximately 21.6%, 25.1%, 18.0%, and 18.9% on Bi-TSP, Bi-KP, Bi-GVRP, and MOFJSP, respectively. These results complement the comparisons under matched evaluation budgets and demonstrate MOSAIC’s competitiveness in wall clock time.

  2. Runtime breakdown. We use Bi-TSP to break down MOSAIC’s end-to-end runtime as follows.

    ComponentIncluded operationsBi-TSP runtime
    LLM-driven module generation and rewritingLLM requests, response waiting, and retries1h 46m 16s
    Candidate architecture execution and evaluationExecuting candidate modules or architectures and evaluating training performance56m 40s
    Architecture operations and population maintenanceArchitecture sampling, crossover, mutation, sorting, and selection23s
    Other framework and logging overheadProcess startup, some code checks, logging, and file I/O1m 19s
    Total2h 44m 38s

    LLM-driven module generation and rewriting account for approximately 64.6% of the total runtime, while candidate architecture execution and evaluation account for approximately 34.4%. Architecture operations and population maintenance take only 23 seconds, approximately 0.23% of the total. Thus, under this experimental setting, computational costs are dominated by LLM calls and candidate evaluation, while architecture operations and population maintenance incur little direct overhead.

In summary, the end-to-end comparison shows that MOSAIC completes the search in less time on all four problems. The Bi-TSP breakdown further shows that architecture operations and population maintenance account for only a small fraction of the total runtime. These supplementary results quantify the computational costs of the complete search process and support MOSAIC’s competitiveness in practical runtime efficiency.

AuthorsOpenReview ↗

Response (3/6)

W3: Concerns about generality across diverse optimization settings. The main experiments focus on three bi-objective problems. Despite an additional three-objective scheduling experiment, the generality of the framework to more diverse optimization settings remains insufficiently established.

Thank you for highlighting the limited scope of the evaluation. To further assess MOSAIC’s applicability across different problem types and numbers of objectives, we provide supplementary experiments on four additional multi-objective combinatorial optimization problems.

  1. Datasets. We include the multi-objective next release problem (MONRP) [1], multi-objective set covering problem (MOSCP) [2], multi-objective quadratic assignment problem (MOQAP) [3], and multi-objective production planning problem (PPP) [4]. MOQAP has three objectives, and PPP has five. The datasets are available in the Supplementary_Datasets/ directory at this anonymous code link.

  2. Experimental settings. All experiments run on a 10-core Apple M5 CPU with 24 GB of unified memory, using the same budget of 253,000 objective function evaluations per training instance. Other experimental settings are detailed in Section 3.1 and Appendix C.2.

    Test performance is measured by hypervolume (HV), where higher values indicate better performance. The table reports the mean over five independent runs. The highest mean HV in each row is shown in bold.

  3. Experimental results. The table below presents the results across the four additional problems and their test scales. MOSAIC achieves higher mean HV than all six baselines in all 13 test-scale groups, further supporting its applicability across different problem types, numbers of objectives, and test scales.

    ProblemTest scaleInstancesEoHFunSearchReEvoMCTS-AHDE2OCG-LNSMOSAIC
    MONRP200 requirements60.66400.61350.64220.67300.58240.56640.7388
    MONRP300 requirements20.61480.57440.60030.61880.56230.55670.6354
    MOSCP100 (nominal size)480.89980.88610.89670.90040.84520.85430.9233
    MOSCP150 (nominal size)800.91990.90010.92300.91250.90150.91240.9308
    MOSCP200 (nominal size)800.92760.92470.92340.92960.92120.92080.9403
    MOQAP30 locations20.31590.31420.32340.32530.31680.31640.3534
    MOQAP40 locations60.28700.28860.28780.29130.28740.28810.3123
    MOQAP50 locations10.37350.37510.37550.38230.37640.37780.4023
    PPP5 periods40.48870.49230.49450.50140.50230.50160.5322
    PPP6 periods100.56910.57240.57620.58030.58140.58100.6122
    PPP8 periods100.64070.64310.64380.65310.65430.65360.6774
    PPP10 periods100.70010.71220.72380.73670.74120.74100.7884
    PPP12 periods100.74950.75310.75690.76230.76590.76530.7912

In summary, the main text evaluates MOSAIC on three bi-objective problems, Bi-TSP, Bi-KP, and Bi-GVRP, and the appendix further examines the three-objective scheduling problem MOFJSP. In response to your suggestion, we add experiments on MONRP, MOSCP, three-objective MOQAP, and five-objective PPP. These experiments extend the evaluation to eight problem classes and provide further evidence of MOSAIC’s applicability to more diverse optimization settings.

References

[1] Pérez-Piqueras, V., Bermejo, P., and Gámez, J. A. “FEDA-NRP: A fixed-structure multivariate estimation of distribution algorithm to solve the multi-objective Next Release Problem with requirements interactions.” Engineering Applications of Artificial Intelligence, 124:106555, 2023.

[2] Jabs, C., et al. “From single-objective to bi-objective maximum satisfiability solving.” Journal of Artificial Intelligence Research, 80:1223–1269, 2024.

[3] Fernandes, I. F. C., et al. “Multi- and many-objective path-relinking: A taxonomy and decomposition approach.” Computers & Operations Research, 133:105370, 2021.

[4] Forget, N., Gadegaard, S. L., and Nielsen, L. R. “Warm-starting lower bound set computations for branch-and-bound algorithms for multi-objective integer linear programs.” European Journal of Operational Research, 302(3):909–924, 2022.

Reviewer_bx3gOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MOSAIC searches for heuristics represented as variable-length sequences of executable modules. An LLM rewrites individual modules, while ordinary crossover and mutation change their composition, order, and number. Complete architectures are selected using average and worst-type training hypervolume. Experiments compare the resulting heuristics with six LLM-based design methods on bi-objective TSP, knapsack, and vehicle routing, with additional scheduling experiments. The paper also studies module rewriting, fixed architectures, module order, and architecture length.

Critical strengths
  1. The modular representation is a useful extension of heuristic design with predefined operator roles. It permits reuse and reordering without another LLM call, while still allowing the modules themselves to change. The Bi-KP order experiment is particularly helpful: holding the four modules fixed gives HV values from 0.501 to 0.590, and the evolved ordering reaches 0.585. This directly supports the claim that organization matters for these modules.

  2. The evaluation goes beyond a single favorable task. There are held-out instances, larger test sizes, two underlying evolutionary frameworks, five independent runs, and comparisons with both singl and multi operator methods. The multi-operator gains are substantial; on Bi-TSP500, for example, Table 1 reports HV 0.89 for MOSAIC with NSGA-II, versus 0.72 for G-LNS and 0.55 for E2OC. Tables 3 and 15 also distinguish significant improvements from the inconclusive Bi-GVRP comparisons with single-operator methods.

Critical weaknesses
  1. Some code presented as the executable modules of the best architectures cannot be used as printed. Figure 7 creates and modifies result but returns new_candidates, which is not defined in the displayed function. Figure 12 initializes row_vals but subsequently reads row, again without a definition. These may be transcription errors, but they are difficult to reconcile with the validity checks in Appendix B and leave uncertainty about which implementations produced the reported results. Establishing that connection matters for a method whose output is executable code.

  2. The architecture ablation is not specified well enough to isolate the benefit claimed for it. Sec. 3.3 and Appendix C.6 say that Fixed-Arch retains module rewriting, but do not state its fixed length, how its initial composition is chosen, or how the evaluation budget is reassigned when architecture variation is removed.

Questions for authors
  1. Are the undefined variables in Figures 7 and 12 errors introduced when preparing the figures? Please identify the exact evaluated modules and provide the corresponding executable versions and a minimal validation check. Does correcting the displayed code require any change to the reported results?

  2. What exactly is held fixed in Fixed-Arch, and how are its length, initialization, number of candidate evaluations, and rewriting opportunities matched to MOSAIC? If it does not already use a four-module architecture with the same total evaluation budget, a matched version on the existing benchmarks would help establish how much of the gain comes from architecture evolution.

6 comments
AuthorsOpenReview ↗

Response (1/4)

W1: Concerns about code presentation. Some code presented as the executable modules of the best architectures cannot be used as printed. Figure 7 creates and modifies result but returns new_candidates, which is not defined in the displayed function. Figure 12 initializes row_vals but subsequently reads row, again without a definition. These may be transcription errors, but they are difficult to reconcile with the validity checks in Appendix B and leave uncertainty about which implementations produced the reported results. Establishing that connection matters for a method whose output is executable code.

Thank you for carefully checking the displayed code and raising these concerns. We address them below.

  1. Code presentation. Figure 7 contains a transcription error: return new_candidates should be return result. The implementation used in our experiments correctly returns result. We correct the displayed return statement in the revision.

    In Figure 12, row is initialized on line 12 through the Python assignment row = candidates[i].copy(), before its subsequent use. We will make this initialization clearer in the revised figure to avoid ambiguity.

  2. Connection between executable code and reported results. We agree that this connection must be explicit for a method whose output is executable code. For each problem, the reported results are averaged over five independent runs. Due to space constraints, the appendix displays the module code for the best architecture from one run per problem. For each of the three problems, we provide the actual executable code for the best architecture obtained under each of the five random seeds, giving 15 architectures in total. The code is available in the trained_best_architectures/ directory at this anonymous code link. The table below lists the best architecture obtained for each problem and seed; arrows indicate module execution order.

    SeedBi-TSPBi-KPBi-GVRP
    2026O506 → O326 → O326O504 → O421 → O504 → O421O726 → O753 → O735 → O510
    2027O766 → O521 → O671O782 → O181 → O103 → O181O385 → O491 → O123 → O201
    2028O634 → O392O577 → O910 → O93 → O161O463 → O698 → O599 → O599
    2029O768 → O730 → O745 → O377O312 → O924 → O391 → O450O8 → O383 → O767 → O674
    2030O751 → O815 → O743 → O855O105 → O243 → O660 → O113O623 → O574 → O340 → O340
  3. Reconciliation with the validity checks in Appendix B. The validity checks were applied to the actual executable modules used in the experiments. The error in Figure 7 was introduced when transferring the code into the manuscript figure, after the experimental implementation had been validated. This explains the discrepancy between the displayed return statement and the validity checks. We also provide the runnable test script test_architectures.py and datasets in test_sets/ at the same anonymous link. The required dependencies are listed in requirements.txt, and the testing procedure is described in README.md.

In summary, the reported results were obtained using the validated executable implementations of the best architectures listed above for each problem and seed. The transcription error in Figure 7 has been corrected in the revised manuscript.

AuthorsOpenReview ↗

Response (2/4)

W2: Concerns about the architecture ablation. The architecture ablation is not specified well enough to isolate the benefit claimed for it. Sec. 3.3 and Appendix C.6 say that Fixed-Arch retains module rewriting, but do not state its fixed length, how its initial composition is chosen, or how the evaluation budget is reassigned when architecture variation is removed.

Thank you for pointing out that the ablation settings are insufficiently described. We clarify the initialization, structural constraints, and evaluation budget allocation below.

  1. Shared initialization. All four ablation variants in Section 3.3 (w/o AR, w/o BR, w/o AR+BR, and Fixed-Arch) use exactly the same initial architecture population as the full MOSAIC for each random seed, including identical architecture lengths, module compositions, and execution orders. This controls for initialization effects when assessing individual components. We provide the initial architecture populations and their complete module code for all five seeds and three problems through this anonymous code link. The module code is stored in initial_architecture_populations/, while architecture_order.md lists the module execution sequence for each of the 20 architectures in every initial population.

  2. Architecture initialization and constraints. The initial architecture population is generated by randomly composing modules from the initial module pool. For each candidate architecture, we randomly choose a length between 1 and 4 and sample a module for each position, allowing the same module to appear multiple times. After removing duplicate architectures, we evaluate the candidates on the training instances and select 20 architectures using nondominated sorting and crowding distance, based on mean HV and worst-case instance-type HV. Further details are provided in Section 2.2 and Appendix B.1.

    In Fixed-Arch, architecture crossover and structural mutation are disabled, while module rewriting is retained. Each rewritten offspring preserves its parent’s architecture length and positional structure; only the implementation of the module at the selected position is updated. Thus, structural constraints are maintained within each evolutionary lineage, while module behaviors can continue to evolve.

  3. Evaluation budget allocation. All ablation variants use the same total evaluation budget as the full MOSAIC: 253,000 objective function evaluations per training instance, with 20 candidate architecture evaluation slots per generation. When a mechanism is disabled, its candidate evaluation slots are reassigned to the remaining mechanisms. The allocation per generation is shown below.

    VariantARBRArchitecture variationTotal
    w/o AR0101020
    w/o BR1001020
    w/o AR+BR002020
    Fixed-Arch1010020
    MOSAIC551020

    Specifically, Fixed-Arch reallocates the 10 architecture-variation slots to module rewriting, so that AR and BR each generate 10 candidates per generation. Fixed-Arch and MOSAIC therefore have the same total evaluation budget and number of candidate evaluation slots per generation, while Fixed-Arch receives 20 module-rewriting slots compared with 10 in MOSAIC.

In summary, the four ablation variants share MOSAIC’s initial architecture populations, and Fixed-Arch retains module rewriting while disabling architecture crossover and structural mutation. These results support the conclusion that MOSAIC’s advantage arises from jointly evolving module behaviors and their organization, including architecture length, module composition, and execution order, rather than from differences in the initial architecture population or the total evaluation budget.

AuthorsOpenReview ↗

Response (3/4)

Q1: Concerns about code presentation and verification. Are the undefined variables in Figures 7 and 12 errors introduced when preparing the figures? Please identify the exact evaluated modules and provide the corresponding executable versions and a minimal validation check. Does correcting the displayed code require any change to the reported results?

Thank you for requesting these clarifications. We respond to each question in turn.

  1. Figure preparation. As clarified in our response to Reviewer bx3g’s W1, the error in Figure 7 was introduced when preparing the code figure: return new_candidates should be return result. In Figure 12, row is initialized on line 12 through the Python assignment row = candidates[i].copy() before its subsequent use.

  2. Evaluated implementations and testing materials. We provide the best architecture from each of the five independent runs for Bi-TSP, Bi-KP, and Bi-GVRP, yielding five architectures per problem and 15 in total. The actual executable implementations of their constituent modules are available in the trained_best_architectures/ directory at this anonymous code link. The runnable test script test_architectures.py uses the datasets in test_sets/. Dependencies are listed in requirements.txt, and the testing procedure is described in README.md.

  3. Effect on the reported results. Correcting Figure 7 does not require any changes to the reported results. The implementation used in our experiments correctly returns result; the transcription error is confined to the manuscript figure.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.