acceptodds
Under review as a conference paper at ICLR 2027

TRACE-EA: Adaptive Trajectory Observation for Automatic Operator Design in Multi-Objective Combinatorial Optimization

Abstract

Large Language Models (LLMs) have enabled automatic heuristic design (AHD) for multi-objective combinatorial optimization problems (MOCOPs), but execution feedback used to guide heuristic revision remains centered on aggregate performance measures. Such summaries provide limited information about how progress and stagnation are distributed across trade-off regions or how previously attained coverage is lost during search. We present TRACE-EA, an LLM-driven AHD method that couples heuristic evolution with selective trajectory observation and observation-policy adaptation. An LLM observer selects tools and query arguments to examine trajectories recorded during candidate evaluation within a fixed multi-objective evolutionary solver. The retrieved evidence supports a diagnosis that guides the selection and generation of heuristic variations. A second feedback loop uses the history of observations, variation actions, and candidate outcomes to adapt the observation policy's temporal window, query-round allowance, and tool priors. Policy updates are retained or reverted based on subsequent candidate outcomes. Experiments across 18 multi-objective settings show that TRACE-EA achieves competitive performance, with advantages that carry over to unseen instances and larger problem scales.

Primary area:
optimization
Keywords:
Large Language Models, Automatic Heuristic Design, Multi-Objective Combinatorial Optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2MMaOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TRACE-EA is an LLM-driven automatic heuristic design framework for multi-objective combinatorial optimization. It queries NSGA-II/SEMO search trajectories through 11 diagnostic tools to guide operator variation; a second loop adjusts the observation window, query rounds, and tool priors. Experiments on bi-/tri-objective TSP, CVRP, and FJSP across scales and evaluation budgets show substantial gains on several TSP/CVRP tasks, with mixed FJSP results.

Critical strengths
  1. The paper asks what execution evidence beyond aggregate HV should drive heuristic revision in multi-objective search. Using the additive ε-indicator in both directions to separate coverage gained from coverage lost is a nice touch. The tool interface keeps prompts compact, and the tool-usage analysis in Fig. 3 is informative.

  2. The paper has good protocol choices. Comparisons are made within a shared inner solver. HV reference points are fixed and independent of any method’s output, and HV is computed per run. Cross-scale transfer is evaluated.

Critical weaknesses
  1. Limited methodological novelty. DASH (Wang et al., 2026) and AutoEP (Xu et al., 2026) already leverage search dynamics for LLM-guided algorithm design/adaptation; MoH (Shi et al., 2026) and MeLA (Qiu et al., 2025) explore feedback-based adaptation of heuristic generation. TRACE-EA mainly exposes established Pareto indicators via 11 tools and adjusts a small observation policy using fixed thresholds and multipliers (Appendix C.3). Its distinct information-selection or policy-learning advance over these precedents is insufficiently established.

  2. Experimental evidence does not adequately isolate the proposed mechanisms. The main ablations are confined to Bi-TSP50/NSGA-II (Table 2), without comparison against a tuned fixed-observation strategy or validation on another problem/solver. Moreover, the NSGA-II baselines and TRACE-EA evolve different operator interfaces (mutation operator, operator chain, and batch operator), while TRACE-EA–SEMO uses a 360 s candidate limit versus 120 s for MPaGE/MEoH (Table 7). These design differences complicate attribution of the reported gains specifically to trajectory observation and policy adaptation.

  3. The paper argues that aggregate HV masks search dynamics, yet the second feedback loop assigns credit using only a difference in mean HV and citations self-reported by the LLM. As a result, an edit that fixes the diagnosed problem but does not raise HV may be counted as a failure, while an HV gain unrelated to the diagnosis may reward the cited tools. Whether this feedback actually improves observation remains unsupported.

  4. The observation process in TRACE-EA is very expensive. The paper reports 78,074 tool calls over 6,813 observed search steps across the 36 TRACE-EA searches.

Questions for authors
  1. What new diagnostic or information-selection principle does TRACE-EA offer beyond the dynamics-aware feedback in DASH/AutoEP and querying established Pareto indicators? Can the authors illustrate a distinctive decision?

  2. How does the second loop differ methodologically from MoH/MeLA? What motivates its thresholds, 1.2/0.9 tool-prior multipliers, and short accept/revert horizon? Why should cited evidence predict causal utility?

  3. Can TRACE-EA and the strongest NSGA-II baseline be compared using the same editable operator interface and candidate-evaluation rules?

  4. On Bi-CVRP50/NSGA-II, can the authors compare the adaptive observation policy with a tuned fixed-observation policy under the same observation budget, to establish the added value of online adaptation?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 2MMa

We greatly appreciate your insightful comments and rigorous assessment of our experimental design. Your feedback directly guided the additional controlled experiments presented below and in the revised manuscript (highlighted in blue).

W1 (Methodological distinction from AutoEP, DASH, MoH, and MeLA)

We agree that individual Pareto indicators and bounded update rules are not new standalone algorithms. What none of these four works addresses is the core bottleneck in multi-objective multi-operator code design: when a multi-function program stalls on a set-valued Pareto front, how to localize which time window, which trade-off region, and which operator function caused the stall, and adjust observation online without extra solver runs.

Prior workHow it works & core limitationHow TRACE-EA differsWhy this matters for MOCOP operator design
AutoEP (Indicators)Computes fixed online ELA scalars (fitness dispersion, correlation, diversity) within one run to tune numeric hyperparameters; no code edits, assumes a 1D scalar fitness order, and logs no per-operator stats.Computes reference-free set-geometry dynamics (ε±\varepsilon^\pm gain/loss, per-direction R2 progress, front retention sts_t) paired with per-operator improvement rates and HV contributions.On a partially ordered Pareto front, scalar diversity and total HV can look healthy while the front slides and loses covered regions (ε+>0,ε−>0\varepsilon^+>0, \varepsilon^->0) or stalls on interior directions; without per-operator attribution, feedback cannot identify which function to rewrite.
DASH (Observation)Compresses a 1D quality curve into one scalar decay rate (tLDR) to passively score the whole solver for early-stopping schedules and library retrieval; no interactive slicing or component breakdown.Interactive coarse-to-fine querying: the Observer slices specific generation windows, trade-off directions, and per-operator logs conditioned on earlier findings.A whole-solver decay scalar tells whether progress slows, not when, in which direction, or which operator failed. On Bi-TSP50, scalar HV curves alone trail set-valued diagnostics (Search HV 0.6132 → 0.6282; HV@2s 0.4942 → 0.5930, Appendix F.5).
MoH & MeLA (Adaptation)Treat the solver as a black box returning an endpoint scalar score and adapt the generator/optimizer: MoH meta-evolves outer optimizers across training tasks (extra outer runs); MeLA rewrites prompts via reflection (risking prompt drift).Leaves the generator untouched and adapts only the observation policy φ=(w,nrounds,θ)\varphi = (w, n_{\mathrm{rounds}}, \boldsymbol{\theta}) online with 0 extra solver runs, action-adjusted residuals, and 4-step rollback.Under a single-task 100–200 candidate budget (~25–50 steps), multi-task meta-training is unaffordable and open-ended prompt rewrites drift on noisy steps; adapting a 3-component observation policy with rollback stays sample-safe (Q2, Q4).

We will incorporate a concise comparison with these four works into Section 2 (Related Work) in the final version.

W2 (Isolating mechanisms: NSGA-II interfaces, SEMO 360 s vs. 120 s cap, and ablation scope)

We thank the reviewer for catching these protocol differences and ran dedicated controls to isolate each factor:

  1. Same-interface NSGA-II baseline (Q3, Appendix G.1): Upgrading official EoH to evolve the exact same multi-operator batch module under the identical evaluator closes a large part of Table 1's raw NSGA-II gap; under this aligned setup, TRACE-EA still outperforms EoH across all seen and unseen Bi-TSP50 and Bi-CVRP50 settings with markedly lower across-seed variance (Q3).
  2. Aligned 120 s SEMO search timeout (Appendix G.3): Note that on SEMO, TRACE-EA already shares the exact single-function select_neighbor interface of MPaGE and MEoH. We re-ran all 18 TRACE-EA (SEMO) searches (6 tasks × 3 seeds) from scratch under their exact 120 s per-candidate cap, updating the equal-wall-time data in Tables 1, 3, and 7 and Appendix D.1 of the revised PDF. At 30 s, TRACE-EA (SEMO) ranks 1st by mean HV in 18/24 settings within the SEMO group, including 5/6 seen settings (table below; all 24 settings in Appendix G.3) and all 16/16 settings on Bi-TSP, Tri-TSP, Tri-CVRP, and Tri-FJSP. The corresponding fixed-evaluation-budget runs (3,600 s per-run safety cap, Appendix D.2) for these 120 s SEMO operators are currently running, and we will immediately update both the PDF and this rebuttal as soon as they finish.
AuthorsOpenReview ↗

Response to Reviewer 2MMa

Seen Setting (HV at 30 s)TRACE-EA (SEMO, 120 s)Previous 360 sMPaGE (120 s)MEoH (120 s)
Bi-TSP500.6377 ± 0.00340.63710.60590.6150
Tri-TSP500.4359 ± 0.00350.44310.36580.3729
Bi-CVRP500.5400 ± 0.00130.54130.53610.5355
Tri-CVRP500.5249 ± 0.00050.52560.51050.4996
Bi-FJSP10j5m0.3668 ± 0.00770.36580.36190.3725
Tri-FJSP10j5m0.1363 ± 0.00060.13590.13200.1323
  1. Additional problem and solver ablations (Q4, Appendix G.2): Across 3 search seeds (100-candidate budget) on unseen medium-scale instances, we added: (i) Bi-CVRP50 with NSGA-II, comparing online policy adaptation against a pre-tuned fixed policy (0.5096 ± 0.0167 vs. 0.4839 ± 0.0043 HV at 30 s; Q4), and (ii) Bi-TSP50 with SEMO (120 s cap), comparing selective tool observation against fixed trajectory summaries (0.6420 ± 0.0045 vs. 0.6374 ± 0.0034 HV at 30 s; 0.6425 ± 0.0049 vs. 0.6376 ± 0.0038 at 60 s).

W3 (Credit assignment in Loop 2: relationship between fixing the diagnosed behavior and HV gain)

Single-step mismatches indeed occur in stochastic heuristic search. Using the same-parent Structural Mutation (M1) protocol (Appendix E.4), we measured this directly on Bi-TSP50 across 14 parents from early, middle, and late search stages (4 independent children per parent, 56 edits total) against each parent's diagnosed runtime metric (stalled-direction R2 progress, occupied front cells, or target-operator late improvement rate):

Same-parent M1 edits on Bi-TSP50 (56 edits across 14 parents)Mean HV increased (Δq>0\Delta q > 0)Mean HV did not increase (Δq≤0\Delta q \le 0)Conditional HV-gain rate
Targeted runtime metric improved (n=25n = 25)18 (agreement)7 (12.5% mismatch)72.0% (18/25)
Targeted runtime metric did not improve (n=31n = 31)6 (10.7% mismatch)25 (agreement)19.4% (6/31)

Repairing the targeted behavior raises HV-gain probability by 3.7× (72.0% vs. 19.4%; 76.8% overall agreement vs. 23.2% step-level mismatches). Averaging action-adjusted residuals over 4 steps with automatic rollback (Q2) smooths out these step-level mismatches, enabling online adaptation to outperform a task-tuned fixed policy on unseen Bi-CVRP50 (Q4; documented in Appendix E.4).

W4 (Tool-querying overhead and future directions)

Note that the 76,763 tool calls across the 36 main searches (~11.3 per observed step; 78,074 in the initial submission before the 120 s SEMO rerun) are local, read-only Python slices over already-recorded trajectory arrays at 0 extra solver evaluations, batched into 2–4 Observer LLM turns per step rather than separate LLM calls per tool. Even so, multi-round LLM querying adds outer-search interaction cost. Once what set-valued trajectory evidence matters is established, a natural next step is distilling recorded TRACE-EA trajectories and diagnoses into a lightweight specialized tool-selection module for MOCOPs to predict informative windows and indicators directly from compact solver states—cutting querying latency and token cost in future work, which we will note in the final version.

AuthorsOpenReview ↗

Response to Reviewer 2MMa

Q1 (Information-selection principle beyond DASH/AutoEP, and a distinctive decision)

  • Principle (3D fault localization across time window × trade-off region × operator): TRACE-EA narrows down where and why front expansion stalls in three steps: (1) cross-run consistency (scan_findings, compare_repeats) filters single-run noise; (2) window and region slicing (region_progress, front_shape) separates benign trade-off shifts (ε−=0\varepsilon^- = 0) from stalled directions or coverage loss (ε−>0\varepsilon^- > 0); and (3) time-aligned operator_stats compares sibling operators on the same population to isolate the failing function (Appendix E.2–E.3).
  • Distinctive decision on Bi-TSP50: On parent 958bf3350013 (defining 2-opt, 3-opt, and Or-opt), overall HV rises smoothly to 0.5482 ± 0.0088 with ε−=0\varepsilon^- = 0 and high diversity (0.96), so scalar curves look healthy. Zooming into generations 20–30 reveals that progress is skewed toward Objective 1 (+0.055 utility at λ=(1,0)\lambda=(1,0) vs. +0.011–0.016 on central/Objective-2 weights across 5 cells) and that 2-opt and 3-opt still gain HV (1.54 and 2.90) whereas Or-opt is nearly dead (2.1% late improvement rate, 0.03 HV gain). Guided by this diagnosis, the LLM leaves 2-opt and 3-opt intact and rewrites only Or-opt from random relocation to weighted high-cost edge relocation biased toward central/Objective-2 trade-offs—lifting Or-opt's late improvement rate up to 19.8% and improving HV in 3/4 children, whereas all 4 no-feedback children mutate the already-strong 3-opt and regress (Appendix E.4).

Q2 (Loop 2 vs. MoH/MeLA, parameter motivation, and why cited evidence is used)

  • Difference from MoH / MeLA: MoH and MeLA adapt how heuristics are generated from endpoint black-box scores; Loop 2 leaves the generator untouched and adapts how trajectories are observed (w,nrounds,θw, n_{\mathrm{rounds}}, \boldsymbol{\theta}) online at zero extra solver runs (W1).
  • Parameter motivation: With only ~25–50 steps over 100–200 candidates, rules must be deconfounded, reversible, and bounded: (1) action-adjusted residuals (ri=Δqi−Δqˉair_i = \Delta q_i - \bar{\Delta q}_{a_i}) remove baseline gain differences across variation actions (e.g., E1 vs. M2); (2) the 4-step window spans one 4-action outer generation and hard-reverts any update whose mean residual drops; and (3) 0.5 / 0.9 citation gates (over [2,6][2, 6] rounds) and 1.2 / 0.9 prior multipliers (in [0.2,5][0.2, 5]) trim unused rounds without disabling (≥0.2\ge 0.2) or locking in (≤5\le 5) any tool.
  • Why cited evidence: The code generator never sees the Observer's raw tool transcript—only cited [Ek][\text{E}_k] slices enter the variation prompt (Appendix E.1). Uncited exploratory calls (overview, scan_findings, 37% of calls) orient where to zoom, whereas cited calls (zoom 98%, region_progress 80%) are the sole trajectory inputs conditioning code edits. A citation rate <0.5< 0.5 thus directly signals over-querying (e.g., trimming nroundsn_{\mathrm{rounds}} from 4 to 2 on Bi-TSP50), and prior updates apply only to tools that actually conditioned the edit.

Q3 (Comparison with the strongest NSGA-II baseline under identical interface and evaluation)

We adapted official EoH—the strongest NSGA-II baseline on most Table 1 settings, and E2OC's inner evolver—to evolve the exact same multi-operator batch module under the exact same candidate evaluator, instances, and 200-candidate budget (Appendix G.1):

Setting (HV at 30 s, mean ± SD)EoH (single-operator)EoH (batch interface)TRACE-EA (batch interface)
Bi-TSP50 (Seen instances)0.5215 ± 0.00430.6265 ± 0.00310.6366 ± 0.0015
Bi-TSP50 (Unseen instances)0.5253 ± 0.01990.6340 ± 0.00380.6403 ± 0.0022
Bi-CVRP50 (Seen instances)0.3068 ± 0.00500.4632 ± 0.06450.5094 ± 0.0028
Bi-CVRP50 (Unseen instances)0.3406 ± 0.00120.4633 ± 0.04380.5263 ± 0.0087

Upgrading EoH to the batch interface indeed accounts for a substantial part of Table 1's raw NSGA-II gap (+0.11 to +0.16 HV; Appendix G.1). At the same time, under identical interface and evaluation, TRACE-EA still outperforms batch-EoH on every setting with zero seed overlap on Bi-TSP50 and 5×–23× smaller across-seed SD on Bi-CVRP50 (±0.0028 / ±0.0087 vs. ±0.0645 / ±0.0438)—because without per-operator trajectory attribution, EoH mutates multi-function bundles blindly, whereas TRACE-EA localizes edits to the bottleneck operator (complementing SEMO, where all methods already share a single-function interface; W2).

Reviewer_t1MHOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TRACE-EA, a framework that uses multi-objective search trajectories to guide LLM-driven automatic operator design. It records changes in approximation fronts, directional progress, and operator behavior during candidate evaluation. An LLM selectively queries this information through a tool interface, produces a diagnosis, and uses it to guide subsequent program modifications. The framework also adapts its observation policy based on observation history and candidate outcomes. Experiments cover TSP, CVRP, and FJSP with NSGA-II and SEMO, demonstrating competitive performance in several settings and some transfer to unseen instances and larger problem scales.

Critical strengths
  1. The paper addresses an important limitation of performance-only feedback in automatic algorithm design. Using search trajectories to identify uneven directional progress, stagnation, and coverage changes is a promising research direction. The revised Appendix E substantially strengthens the methodological explanation by providing screening rules, prompts, a tool-query trace, and concrete code modifications. These additions clarify how trajectory evidence can inform targeted operator changes.

  2. The method achieves competitive results in several TSP/CVRP settings, with some advantages extending to unseen instances and larger scales. The additional comparisons against scalar feedback and fixed trajectory summaries, together with cross-problem and cross-solver ablations and aligned baseline interfaces, strengthen the empirical support for the approach.

Critical weaknesses

Update after the author response: My initial concerns focused on the insufficient explanation of how trajectory features are identified, communicated, interpreted, and used. The revised Appendix E substantially addresses these concerns through explicit screening rules, prompts, tool-interaction records, and before-and-after code examples. I no longer consider the absence of a concrete trajectory-to-code explanation a primary reason for rejection. My remaining concerns are narrower:

  1. The detailed mechanism validation remains concentrated on Bi-TSP50 and Structural Mutation. The complete case study and the 14-parent/56-edit behavioral analysis provide useful evidence, but do not establish the same level of diagnostic reliability across other problem families and variation actions. The broader ablations support overall performance, while claims about the generality of the diagnostic mechanism should remain appropriately qualified.

  2. Some diagnostic-effectiveness statistics need more complete reporting. The response reports edit-concentration and behavioral-repair percentages, but I could not locate their full counts, denominators, and measurement definitions in the supplied revision. The behavioral-agreement analysis in Appendix E.4 is informative, but agreement between behavioral improvement and HV improvement is distinct from diagnosis accuracy or repair success. A compact breakdown of these statistics and their control conditions would improve clarity.

Overall, the new material substantially alleviates my main original concerns. Given the promising motivation, competitive results, and improved explanation and validation of the central mechanism, I am updating my recommendation from 2 (weak rejection) to 3 (weak acceptance). The remaining concerns limit the strength of my recommendation but do not outweigh these contributions.

Questions for authors

Most of my initial questions have been substantially addressed by Appendix E and the additional experiments. The remaining clarification concerns the diagnostic-effectiveness statistics.

Could you provide the counts, denominators, measurement definitions, and control conditions underlying the reported edit concentration of approximately 96%–97%, overall repair rates of 45.8% versus 37.5%, and operator-bottleneck repair rates of 50% versus 33%? Please clarify how these analyses relate to the 14-parent/56-edit study in Appendix E.4, including how the parents were selected and how the target behavioral improvement criterion was defined.

As a presentation suggestion, I encourage the authors to incorporate a concise observation-to-modification workflow and a representative example into the main text.

7 comments
AuthorsOpenReview ↗

Response to Reviewer t1MH

Thank you for your systematic and thorough review. We fully understand the concerns you raised and provide responses and new experimental evidence below (with full details highlighted in blue in the revised manuscript).

Main concern:

Thank you for the constructive review. Below is how the four stages (identifying → communicating → interpreting → using trajectory features) connect in a single outer step:

  1. Evaluate & log: Each candidate runs from scratch on fixed instances and seeds, recording per-generation multi-objective front features and per-operator move outcomes.
  2. Select action & parent(s): From recent per-action score deltas and the previous step's diagnosis, the outer controller picks one variation action and parent(s) (or adapts the observation policy φ\varphi without spending a candidate).
  3. Query & diagnose: Under φ\varphi, the Observer queries read-only tools on the selected parent(s)' trajectories (receiving tagged slices [E1], [E2], ...) and returns a JSON diagnosis, suggested target function, and cited evidence IDs.
  4. Modify & re-evaluate: The generator receives the parent code, action instruction, and cited evidence slices, synthesizes a new program, and evaluates it from scratch.

We give a concrete end-to-end trace in Q1, provide verbatim prompts, screening rules, tool outputs, and code diffs in Appendix E (E.1–E.4) of the revised PDF, and will add a single-step workflow diagram to Section 4 in the final version.

W1 (Identifying actionable patterns from trajectories and converting them into observations)

Computing raw indicators and deciding what warrants code modification are distinct steps: in multi-objective search, slow gain in one direction can reflect benign trade-off reallocation or genuine operator failure, while raw multi-run logs are noisy. TRACE-EA turns raw trajectories into reliable, meaningful, and actionable observations through three mechanisms (Q2, Appendix E.1–E.2):

  • Reliable: Deterministic rules (scan_findings) and compare_repeats check all repeated runs using relative thresholds (k/Nk/N support), filtering out single-run fluctuations before LLM interpretation.
  • Meaningful: Rather than reading one directional curve in isolation, the Observer cross-checks region_progress with coverage loss (ε−\varepsilon^-) and diversity to separate benign shifts (ε−=0\varepsilon^- = 0) from front erosion, and checks sibling operators (operator_stats) or other parents (compare_candidates) as relative baselines when the true Pareto front is unknown.
  • Actionable: Only cited tool slices ([E_k]), a concise mechanism diagnosis, and one editable target function enter the variation prompt.

W2 (Communicating trajectories, cross-run summaries, and policy parameters)

Complete prompts and multi-turn outputs are in Appendix E.1 and E.3 (complementing Appendix C.2). Specifically:

  • Observer system prompt & policy parameters ({rounds} = nroundsn_{\mathrm{rounds}}; {funcs} = editable operator names; {priors} = soft hint of top-3 weighted tools from θ\boldsymbol{\theta}; tool outputs return as [E1], [E2], ...):
You are the trajectory observer of an automated algorithm-design system. Candidates are multi-objective evolutionary optimizer programs; each was run on several instances and repeats, and every inner generation was recorded (hypervolume, genotype diversity, R2 by trade-off direction, epsilon-progress vs the previous front, front survival, HV-contribution concentration, operator statistics, the full Pareto front). Multi-objective trajectories are not scalar curves: the front can slide (gains and losses at once), collapse onto a few extreme points, or progress in one trade-off direction only. Use the tools to decide what to inspect and at which resolution; you have at most {rounds} tool rounds and each round costs tokens, so zoom only where it changes the diagnosis. Candidate functions that can be edited: {funcs}.{priors} When done, answer with JSON only:
{"diagnosis": "<2-5 sentences: mechanism, when it starts, which trade-off region>",
  "suggested_target": "<one function name from the list or null>",
  "evidence_ids": ["E1", ...]}
  • Diagnosis block injected into the heuristic variation prompt (placed between parent code and the variation instruction; uncited tool outputs are never passed):
### Observer diagnosis
{diagnosis}
Suggested target: `{suggested_target}`
Evidence:
[E{k}] {tool_name}({short_args}):
{tool_output_text}
AuthorsOpenReview ↗

Response to Reviewer t1MH

  • Query flow: In Q1, Round 1 calls overview, scan_findings, and compare_repeats, seeing HV slow after gen 20 across 9/9 runs while rule flags trigger on all three operators. Round 2 then contrasts operator_stats(0..30) vs. operator_stats(20..30) alongside region_progress(20..30) and front_shape(30), isolating near-zero late gain in tsp_oropt and central/Objective-2 stagnation; Round 3 outputs the JSON diagnosis (Appendix E.3).
  • Cross-run summarization: Time-series and front tools show the representative run closest to the mean HV (with all-run mean ± std in the header) so cross-instance averaging does not blur sudden events, whereas scan_findings and compare_repeats aggregate all runs (k/9k/9 counts and per-run summaries).

W3 (Validating LLM interpretation and attributing behavior to code)

Within each setting, locking the instances, solver selection rules, population size, and seeds ensures that operator code is the sole variable changing between parent and child. Within that fixed solver, checking genotype diversity alongside time-aligned operator_stats on the same population separates population collapse from an individual operator stalling while sibling operators still gain HV. Because citing [E_k] does not by itself guarantee causal correctness, each diagnosis acts as a testable modification hypothesis verified on child runtime behavior (Q3, Q4; Appendix E.4).

W4 (How diagnoses guide action choice, code edits, and behavioral improvement)

Diagnoses act at two levels (controlled parent-to-child tests and code diffs in Q3, Q4, Appendix E.4):

  • Action scheduling: Pairing recent per-action score deltas with the latest diagnosis shifts search from Parameter Retuning (weight/count tweaks) to Structural Mutation or multi-parent synthesis once retuning stalls under structural bottlenecks.
  • Localized code repair & behavioral gain: Injecting the diagnosis and suggested_target focuses ~96%–97% of Structural Mutation edits onto the diagnosed weak operator (vs. 63%–75% without feedback, which frequently mutates working sibling operators) and increases targeted runtime-behavior improvement rates (Q4).

Q1 (Complete end-to-end Bi-TSP50 example)

In Bi-TSP50 with NSGA-II (outer gen 7), parent 958bf3350013 defines tsp_2opt, tsp_3opt, and tsp_oropt over 9 runs:

  1. Trajectory & pre-scan: Overall HV climbs smoothly (0.5482 ± 0.0088) with zero coverage loss (ε−=0\varepsilon^- = 0) and high diversity (0.96), while scan_findings flags low final-step improvement across all operators.
  2. Tool queries: compare_repeats ([E3]) confirms consistency across all 9 runs; Round-2 window queries isolate the failing operator and stalled trade-off directions, producing the Round-3 diagnosis:
[E5] operator_stats({"cand": 0, "g0": 20, "g1": 30})
operator statistics averaged over gens 20-30:
- tsp_2opt: improve_ratio=0.0536, worse_ratio=0.0000, invalid_ratio=0.0000, hv_delta=1.5357
- tsp_3opt: improve_ratio=0.0464, worse_ratio=0.0000, invalid_ratio=0.0000, hv_delta=2.8999
- tsp_oropt: improve_ratio=0.0200, worse_ratio=0.0000, invalid_ratio=0.0000, hv_delta=0.0323

[E6] region_progress({"cand": 0, "g0": 20, "g1": 30})
gens 20->30: R2 0.2192->0.1953; per-direction utility change (+ = better)
- λ=(1.00, 0.00): +0.0547
- λ=(0.90, 0.10): +0.0492
- λ=(0.80, 0.20): +0.0437
- λ=(0.70, 0.30): +0.0213
- λ=(0.50, 0.50): +0.0160
- λ=(0.00, 1.00): +0.0159
- λ=(0.10, 0.90): +0.0143
- λ=(0.40, 0.60): +0.0128
- λ=(0.20, 0.80): +0.0127
- λ=(0.60, 0.40): +0.0115
- λ=(0.30, 0.70): +0.0111
grid cells: 4 -> 5 (gained 1, lost 0)

Returned Observer JSON:
{"diagnosis": "After generation 20, progress becomes mutation-limited: all local-search operators have low improvement rates, and tsp_oropt contributes essentially no late HV despite retained diversity. Remaining gains are concentrated toward the first-objective extreme (λ=1.0–0.8), while the final front occupies only 5 cells and has concentrated HV contributions, leaving central and second-objective trade-offs comparatively underexplored.",
 "suggested_target": "tsp_oropt",
 "evidence_ids": ["E1", "E3", "E5", "E6", "E7"]}
AuthorsOpenReview ↗

Response to Reviewer t1MH

  1. Code edit: Whereas tsp_3opt already cuts high weighted-cost edges, parent tsp_oropt cuts and reinserts segments uniformly at random. Structural Mutation (Q3) leaves tsp_2opt and tsp_3opt untouched and rewrites only tsp_oropt:
# Parent tsp_oropt (core loop: uniform random weight + random cut & reinsertion)
weight = float(np.random.random())
for _ in range(8):
    length = int(np.random.randint(1, max_segment_length + 1))
    start = int(np.random.randint(0, n - length + 1))
    end = start + length
    segment, remainder = tour[start:end], np.concatenate((tour[:start], tour[end:]))
    insert_at = int(np.random.randint(remainder.size))
    ...

# Diagnosis-guided M1 child tsp_oropt (core loop: central/obj-2 weights + high-cost edge priority)
target_weights = np.array((0.20, 0.35, 0.50, 0.65), dtype=np.float64)
...
weight = float(target_weights[np.random.randint(target_weights.size)])
edge_priority = weight * d1[tour, nxt] / scale[0] + (1.0 - weight) * d2[tour, nxt] / scale[1]
source_edges = np.argsort(-edge_priority)[:min(2, n)]
for source_rank, source_edge in enumerate(source_edges):
    ...
    for insertion_edge in np.argsort(-reinsertion_priority)[:min(4, remainder.size)]:
        ...
  1. Trajectory change: Re-evaluated on the identical 9 runs, tsp_oropt's late improvement rate jumps from 2.1% to up to 19.8%, directional R2 improves, and design HV rises from 0.5482 to 0.5530–0.5610 in 3/4 independently generated children (full trace and both working and unsuccessful/no-feedback code in Appendix E.3–E.4).

Q2 (Determining what is worth fixing: tool rules vs. LLM inference & prompts)

  • Deterministic screening rules (no LLM): scan_findings evaluates 9 scale-free relative rules across all runs (Appendix E.2)—e.g., second-half HV gain <10%<10\% of total (premature convergence), flat HV for ≥1/3\ge 1/3 of the run (stagnation), last-quarter diversity <0.35×<0.35\times first-quarter (diversity collapse), min-objective gain >3×>3\times another's (imbalance), or final operator improvement <0.5×<0.5\times its run-mean or best sibling (operator ineffective), plus front shrinkage, low spread, oscillation, and peak-HV loss. Because operators naturally slow down late in search, k/Nk/N rule flags serve only as coarse cues rather than direct edit commands.
  • LLM cross-checking: Guided by the Observer prompt (W2, Appendix E.1), the LLM judges whether a cue warrants intervention by testing (i) whether slow directional progress coincides with zero coverage loss (ε−=0\varepsilon^- = 0, benign trade-off shift) vs. coverage erosion or global stagnation under healthy diversity, and (ii) relative saturation—whether sibling operators still improve the same late population (operator_stats) or other parents reached better directional attainment (compare_candidates).

Q3 (Influence on the 4 actions; same-parent edits with vs. without diagnosis)

  • Action selection & conditioning: The outer action prompt (Appendix E.1) combines recent per-action score deltas with the latest diagnosis to schedule Parent-guided Synthesis (E1), Backbone Diversification (E2), Structural Mutation (M1), Parameter Retuning (M2), or policy adaptation—moving from M2 to M1/E1/E2 when parameter tweaks stall. For the chosen parent(s), cited [E_k] slices (including compare_candidates for E1/E2) and suggested_target enter the variation prompt to guide which strengths to recombine or which function to rewrite.
  • Controlled same-parent test: Holding parent code, M1 prompt template, instances, and seeds fixed on Bi-TSP50 while toggling only the diagnosis block, diagnosis focuses ~96%–97% of edits onto the suggested weak function (vs. 63%–75% without feedback). On parent 958bf3350013 (Q1, Appendix E.4), all 4 diagnosis-guided children rewrite tsp_oropt toward central/Objective-2 edge priority (3/4 improving HV), whereas all 4 no-feedback children mutate the already-strong tsp_3opt while leaving tsp_oropt random, and all 4 regress in HV.

Q4 (Do diagnosed behaviors improve beyond HV? Comparison vs. fixed summaries & non-adaptive querying)

Yes—at both the single-step parent-to-child level and the full-search level:

  • Step-level behavioral repair: Re-evaluating independently generated M1 children on the identical 9 runs against the exact runtime metric named in each parent's diagnosis (stalled-direction R2 progress, occupied front cells, or target-operator late improvement rate) shows higher single-shot repair rates with diagnosis than without feedback (45.8% vs. 37.5% overall across already-evolved parents, and 50% vs. 33% on operator-specific bottlenecks over both fixed summaries and no feedback)—where population selection only needs one of the generated offspring to succeed while leaving working sibling operators intact.
Reviewer_R78BOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Presents TRACE-EA, an LLM-driven automatic heuristic design method for multi-objective combinatorial optimization that couples heuristic evolution with selective trajectory observation. An LLM observer queries recorded NSGA-II/SEMO solver trajectories through eleven read-only tools (overview, zoom, front, region_progress, operator_stats, and others) and synthesizes the retrieved evidence into a diagnosis that guides one of four variation actions. A second loop adapts the observation policy (temporal window, query-round allowance, tool priors) with bounded deterministic rules, retaining or reverting changes by comparing action-adjusted residual quality over four subsequent steps. Evaluated on 18 settings across TSP, CVRP, and FJSP with bi- and tri-objective variants at three scales. TRACE-EA leads on TSP and CVRP but trails existing methods on FJSP.

Critical strengths
  • Broad evaluation across 18 settings, two solvers, six baselines, and three protocols (30s, 60s, fixed-budget), with honest reporting of where TRACE-EA loses.
  • The trajectory feature set (ε-indicator gain/loss, weighted-Tchebycheff directional progress, front retention, exclusive-HV contribution concentration) is well-defined and documented in Appendix C.1.
  • The negative results on FJSP and on transfer to smaller instances are reported rather than omitted.
Critical weaknesses
  • The novelty is incremental: the trajectory features are standard EMO indicators, the observer is an LLM calling read-only functions over them, and the policy adaptation is a set of hand-set bounded rules, so the contribution reads as system integration whose academic value rests mainly on the empirical demonstration rather than a new algorithmic insight.
  • The comparison with E2OC, the closest prior method in the same NSGA-II group, is treated as a single table entry: the paper does not delineate what TRACE-EA adds over E2OC's MCTS-based operator search or analyze what causes the performance differences, so the relative contribution is not established.
  • The claim that advantages carry over to larger scales does not hold for FJSP: at the largest setting TRACE-EA is below the classical NSGA-II baseline and every learned baseline, so the claim holds for TSP/CVRP only.
  • The primary 30s wall-clock budget conflates heuristic speed with search quality and is hardware-dependent; the fixed-budget results rank methods differently, but use a single cap in solver-specific units rather than a sweep over evaluation counts.
  • Results report point means with no standard deviation or significance test, and only three outer search seeds are used, so the winning margins may not be statistically meaningful.
  • The core claim that selective observation drives the gains is not isolated: the ablation has no control that feeds all trajectory features directly to the LLM, and both loops are ablated only on Bi-TSP50 with NSGA-II.
  • Hypervolume uses hand-selected, per-scale reference points with no sensitivity analysis, so the method ranking may depend on the reference choice.
Questions for authors
  1. The trajectory features are standard EMO indicators and the policy adaptation is a set of hand-set bounded rules. What is the distinct algorithmic contribution beyond wrapping these in a tool-calling LLM loop, and what does it add over feeding the features to the LLM directly?
  2. What does TRACE-EA add over E2OC's MCTS-based operator search, and what mechanism accounts for TRACE-EA leading on TSP/CVRP but trailing on some FJSP settings? A more careful head-to-head comparison is needed.
  3. Why is wall-clock time the primary budget rather than function evaluations, given that it conflates heuristic speed with search quality and is hardware-dependent? Would the advantages persist across a sweep of evaluation budgets?
  4. The ablation lacks a control that passes all trajectory features directly into the prompt without the tool interface. What evidence isolates selective observation from simply providing trajectory evidence?
  5. On the largest FJSP setting, TRACE-EA is below the classical NSGA-II baseline and every learned baseline. How should the "advantages carry over to larger scales" claim be read when it fails for the scheduling family?
  6. Only three outer search seeds are used and no standard deviation or significance test is reported. What is the across-seed variance, and do the close margins survive a significance test?
7 comments
AuthorsOpenReview ↗

Response to Reviewer R78B

We thank you for your detailed and constructive review. Below we provide our responses and new experimental results, with full details included in the revised manuscript (highlighted in blue).

W1 (Novelty and algorithmic contribution vs. system integration)

Rather than proposing new EMO indicators, TRACE-EA changes what feedback drives LLM-based heuristic design for MOCOPs. Existing MOCOP AHD methods (MEoH, MPaGE, E2OC) treat the solver as a black box returning only an endpoint aggregate score, whereas TRACE-EA exposes set-valued search trajectories to guide function-level program edits.

Using established EMO indicators is a deliberate design choice: they are Pareto-compliant, require no ground-truth reference front during search, and ensure that every number cited in a diagnosis can be audited directly against the recorded run. Likewise, the bounded policy-adaptation rules provide lightweight guardrails against observation drift without fitting a data-hungry meta-learner over 25–50 steps. The algorithmic contribution lies in turning set-valued process dynamics into operator-grounded synthesis feedback (isolated in Q1). More broadly, each search step logs a grounded tuple of (trajectory evidence, diagnosis, code edit, quality delta), providing natural supervision or reward signals for future SFT/RL-trained MOCOP design agents.

W2 (Delineation and comparison with E2OC)

We added a dedicated comparison with E2OC in Appendix F.4. Conceptually, E2OC uses MCTS over design thoughts to search combinations of separately evolved operator slots from final scalar HV alone, whereas TRACE-EA evolves a unified multi-operator program guided by process evidence (coverage gain/loss, directional progress, front retention, and per-operator stats) to repair failing functions when final HV masks directional stalls. Outer combinatorial slot search and inner trajectory diagnosis are thus complementary. See Q2 (Appendix F.4) for the routing-versus-scheduling runtime mechanism and 30–180 s FJSP head-to-head results.

W3 (Scope of cross-scale transfer claims on FJSP)

We agree that consistent equal-time (30 s) gains on unseen and larger-scale instances hold on TSP and CVRP, whereas FJSP transfer is runtime-budget-dependent (as noted in Section 5.2 and analyzed across 30–180 s budgets in Appendix F.4 of the revised manuscript; see Q2 and Q5). We will refine the summary phrasing in the final version to reflect this distinction more precisely.

W4 (30 s wall-clock budget, hardware dependence, and fixed-budget rankings)

All methods were executed on the exact same machine (Mac Studio, Apple M4 Max, 16-core CPU, 64 GB RAM) under identical concurrency, ensuring internally consistent wall-clock comparisons. Rather than treating differences between equal-time and fixed-evaluation rankings as an inconsistency, we report both protocols specifically to separate per-evaluation search quality from per-step operator runtime across 30 s (main text), 60 s (Appendix D.1), fixed evaluation budgets (Appendix D.2), and 30–180 s FJSP budgets (Q2, Appendix F.4); see Q3 for why wall-clock time is the primary metric for LLM-generated operators.

W5 (Reporting variance, significance testing, and sample size of three search seeds)

We added an across-seed statistical analysis in Appendix F.1, treating each outer search seed (n=3n = 3) as the independent experimental unit rather than inner solver runs. Across all 96 shared-solver peer comparisons (4 baselines × 24 settings; summary table in Q6, per-setting mean ± SD in Appendix F.1), treating overlapping seed ranges as statistical ties consolidates raw-mean counts of 75 ahead vs. 21 behind into 58 zero-overlap wins (permutation p=0.10p = 0.10), 31 statistical ties, and only 7 losses, raising non-inferiority (wins + ties) from 78.1% (75/96) to 92.7% (89/96) and widening our win-to-loss ratio from 3.6:1 to 8.3:1.

W6 (Isolating selective observation vs. full-trajectory prompting, and ablation task scope)

  1. Direct full-trajectory control: Bypassing tools to dump raw trajectories directly into the prompt overflows the 1M-token context window on multi-parent steps (~340k input tokens per parent), and even a forced single-parent workaround consumes 13.6× more total tokens (41.21M vs. 3.03M) without improving quality (0.6210 vs. 0.6239 HV′@30s; full table in Q4, Appendix F.3).
  2. Ablation scope: Beyond Table 2, we added trajectory feedback-content ablations on Bi-TSP50 (Q1, Appendix F.5) as well as cross-problem (Bi-CVRP50) and cross-solver (SEMO) ablations (Appendix G.2).
AuthorsOpenReview ↗

Response to Reviewer R78B

W7 (Sensitivity to hand-selected HV reference points)

In AHD, the hypervolume reference point rs\mathbf{r}_s is part of the shared optimization objective maximized by every compared method during outer candidate search—not a post-hoc reporting parameter. Because rs\mathbf{r}_s determines the HV-optimal weighting between extreme and knee regions (Auger et al., 2009; Shang et al., 2020), rescaling rs\mathbf{r}_s post hoc without re-running outer search conflates heuristic quality with a search–evaluation preference mismatch (Ishibuchi et al., 2018).

Before freezing our protocol, we ran a two-stage baseline convergence pilot across all 18 settings using NSGA-II and MOEA/D (population 300, 2 seeds per solver; 144 runs up to 1,000–3,000 generations, Appendix F.2). Requiring pooled fronts to sustain ≥99.5%\ge 99.5\% of terminal HV and normalized unary additive ε≤0.01\varepsilon \le 0.01, only 7/18 settings stabilized within 1,000 generations, 3 more by 2,000–2,500 generations, and 8/18 medium- and large-scale routing settings remained unconverged at 3,000 generations—confirming that solver-derived nadir bounds are stopping-cutoff-dependent (Knowles et al., 2006; Li & Yao, 2019). Following standard MOCO and automatic MOEA design practice (Bezerra et al., 2015; Guerreiro et al., 2021), we therefore pre-froze one shared pair (zs=0,rs)(\mathbf{z}_s = \mathbf{0}, \mathbf{r}_s) per setting (Table 6): Bi-/Tri-TSP (N∈{20,50,100}N \in \{20, 50, 100\}) and Bi-CVRP (N∈{50,100}N \in \{50, 100\}) adopt Lin et al. (2022), Chen et al. (2023), and Hieu et al. (2026), while Bi-CVRP20, Tri-CVRP, and generated FJSP (Song et al., 2022; Qiu et al., 2026) use fixed pre-evaluation engineering bounds verified for front coverage alongside reference-point-free IGD (Tables 1 and 3; clarified in Appendix A.4 and Appendix F.2).

Q1 (Distinct algorithmic contribution and comparison to feeding features directly)

  1. Algorithmic contribution beyond wrapping indicators: A candidate's final HV cannot reveal whether front coverage eroded in one region while rising overall, or which function inside a multi-operator program stalled. Linking set-valued trajectory diagnostics to specific editable functions drives consistent gains over both no feedback and scalar HV trajectories on Bi-TSP50 (Table 2 and Appendix F.5), especially in search efficiency and early anytime convergence:
Bi-TSP50 Variant (NSGA-II, seen instances)Search Best HVHV @2 sHV @5 sHV @30 s
No feedback (Table 2)0.59030.40470.54930.6116
Scalar HV dynamics only (Appendix F.5)0.61320.49420.59150.6313
Fixed trajectory summaries (Appendix F.5)0.61500.51530.60220.6308
TRACE-EA (selective set-valued feedback, Table 2)0.62820.59300.63170.6366
  1. Comparison against feeding all trajectory features directly: See Q4 (Appendix F.3).

Q2 (Added value over E2OC and mechanism behind TSP/CVRP vs. FJSP performance)

Beyond the architectural difference in W2 (combinatorial MCTS over final scalar HV in E2OC vs. trajectory-grounded function repair in TRACE-EA), the routing-versus-scheduling gap stems from per-move evaluation cost interacting with the 30 s wall-clock cap (Appendix F.4):

  • NSGA-II mechanism: On TSP and CVRP, local move checks are O(1)O(1) edge updates, completing many generations within 30 s. On FJSP, candidate search runs 30 generations on medium-scale 10j5m instances (retaining quality–efficiency trade-offs in the population; Section 4.2), where TRACE-EA evolves multi-trial local search operators that invoke full schedule decoding per trial move. These operators lead every FJSP setting per evaluation under fixed budgets (Appendix D.2), but complete fewer generations within 30 s when transferred zero-shot to larger 25j5m instances. Extending the wall-clock cap to 120–180 s closes nearly the entire 25j5m deficit (+0.0155 Bi-FJSP and +0.0109 Tri-FJSP HV vs. +0.0042 and +0.0005 for the best baseline, shrinking the gap from 0.0133 → 0.0020 and 0.0112 → 0.0008):
  • SEMO mechanism: On Tri-FJSP (~8–17 front points), directional parent selection pays off and TRACE-EA leads the SEMO group on all four settings at 30 s (0.1703, 0.1363, 0.0784, 0.0551). On Bi-FJSP (~1–3 front points), TRACE-EA ties MPaGE across all four settings at 30 s (+0.0010 task-mean HV) while trailing MEoH, whose lightweight single-parent mutation executes more neighbor attempts within 30 s on a compact 2D front.
AuthorsOpenReview ↗

Response to Reviewer R78B

Bi-FJSP (HV across wall-clock budgets)

TypeMethod5j5m 30s5j5m 120s5j5m 180s10j5m seen 30s10j5m seen 120s10j5m seen 180s10j5m unseen 30s10j5m unseen 120s10j5m unseen 180s25j5m 30s25j5m 120s25j5m 180s
TraditionalNSGA-II0.50910.50940.50960.37970.38080.38080.30760.30820.30820.20680.20750.2075
TraditionalMOEA/D0.49790.49940.50050.37250.37390.37440.29690.29840.29870.19950.20020.2003
NSGA-II-basedE2OC0.50450.50480.50490.38600.38710.38750.31530.31680.31720.21310.21690.2173
NSGA-II-basedEoH0.50620.50710.50720.38030.38200.38200.30720.30790.30820.20740.20830.2087
NSGA-II-basedTRACE-EA (NSGA-II)0.51270.51350.51390.38440.38650.38690.31300.31600.31660.19980.21370.2153
SEMO-basedMEoH0.50460.50780.50800.37250.37550.37620.30420.30800.30910.20250.20710.2082
SEMO-basedMPaGE0.49610.49970.50090.36190.36420.36500.28970.29360.29460.19550.19860.1993
SEMO-basedTRACE-EA (SEMO)0.49570.49810.49940.36680.36930.36960.29090.29460.29660.19380.19560.1960

Tri-FJSP (HV across wall-clock budgets)

TypeMethod5j5m 30s5j5m 120s5j5m 180s10j5m seen 30s10j5m seen 120s10j5m seen 180s10j5m unseen 30s10j5m unseen 120s10j5m unseen 180s25j5m 30s25j5m 120s25j5m 180s
TraditionalNSGA-II0.17260.17310.17320.13740.13860.13890.07970.08040.08080.05540.05590.0559
TraditionalMOEA/D0.17010.17050.17060.13570.13670.13690.07750.07830.07840.05320.05450.0547
NSGA-II-basedE2OC0.17120.17140.17140.13670.13740.13750.07850.07910.07920.05430.05470.0548
NSGA-II-basedEoH0.17290.17320.17320.13830.13910.13920.08000.08050.08070.05540.05580.0559
NSGA-II-basedTRACE-EA (NSGA-II)0.17200.17210.17210.13790.13910.13930.07970.08050.08060.04420.05390.0551
SEMO-basedMEoH0.16770.16850.16860.13230.13310.13330.07540.07630.07650.05230.05340.0537
SEMO-basedMPaGE0.16830.16900.16910.13200.13410.13450.07530.07690.07720.05020.05270.0531
SEMO-basedTRACE-EA (SEMO)0.17030.17080.17120.13630.13740.13750.07840.07890.07900.05510.05540.0554

Comments on the paper

2 comments

Dear Area Chair and Reviewers,

We sincerely thank all three reviewers (t1MH, R78B, and 2MMa) for their thorough feedback. All reviewers recognized the relevance of bringing search-process information into LLM-driven multi-objective heuristic design, the breadth of our 18-setting / 2-solver benchmark, and the empirical strength of TRACE-EA. Their comments converged on four constructive themes: (1) concrete diagnostic mechanics and prompts (t1MH, 2MMa), (2) protocol alignment and broader ablations (2MMa, R78B), (3) across-seed statistical validation and reference-point justification (R78B), and (4) runtime-budget dynamics on FJSP and outer querying cost (R78B, 2MMa).

During the discussion period, we ran targeted experiments and analyses addressing every point:

  1. Across-seed statistical validation (Reviewer R78B; Appendix F.1, B.4): Treating each outer search seed (n=3n = 3) as the independent experimental unit across all 96 shared-solver peer comparisons (4 baselines × 24 settings), exact permutation tests consolidate raw-mean counts (75 ahead vs. 21 behind) into 58 zero-overlap wins (p=0.10p = 0.10), 31 statistical ties, and only 7 losses—raising non-inferiority from 78.1% to 92.7% and widening our win-to-loss ratio from 3.6:1 to 8.3:1 (16:1 vs. E2OC, 17:4 vs. EoH, 14:0 vs. MPaGE, 11:2 vs. MEoH).
  2. Aligned baselines & cross-setting ablations (Reviewer 2MMa; Appendix G.1–G.3):
    • NSGA-II batch interface (G.1): Under the identical multi-operator batch interface and evaluator, TRACE-EA still consistently outperforms batch-EoH on seen and unseen Bi-TSP50 and Bi-CVRP50, with 5×–23× lower across-seed variance on Bi-CVRP50.
    • SEMO candidate timeout (G.3): Re-running all 18 TRACE-EA (SEMO) searches under the exact 120 s cap of MPaGE and MEoH preserves our lead in 18/24 settings (including 5/6 seen settings and 16/16 on Bi-TSP, Tri-TSP, Tri-CVRP, and Tri-FJSP; updated in Tables 1, 3, 7 and Appendix D.1). The corresponding fixed-budget evaluation (3,600 s per-run safety cap, Appendix D.2) for these 120 s SEMO operators is currently running and will be updated in the PDF and rebuttal immediately upon completion.
    • New problem/solver controls (G.2): Online policy adaptation outperforms an offline-tuned fixed policy φ∗\varphi^* on unseen Bi-CVRP50 (+0.0257 HV at 30 s under equal LLM calls, while saving 200 tuning candidates), and selective observation outperforms fixed summaries on SEMO Bi-TSP50.
  3. Diagnostic mechanism, raw-prompting wall & step-level attribution (Reviewers t1MH, R78B, 2MMa; Appendix E.1–E.4, F.3, F.5):
    • Verbatim prompts & parent-to-child causality (E.1–E.4): Controlled same-parent M1 tests show diagnosis focuses ~96%–97% of edits onto the failing operator (vs. 63%–75% without feedback), lifts operator-bottleneck repair rates (50% vs. 33%), and exhibits 76.8% agreement (43/56) between targeted behavioral repair and HV gain (72.0% vs. 19.4% conditional HV-gain rate).
    • Full-trajectory & scalar-feedback controls (F.3, F.5): Raw trajectory dumping overflows 1M-token context on multi-parent steps and costs 13.6× more tokens even in single-parent mode without improving quality, while set-valued diagnostics substantially outperform both scalar HV curves and fixed digests (HV@2s: 0.5930 vs. 0.4942 and 0.5153).
  4. FJSP runtime scaling & reference-point convergence pilot (Reviewer R78B; Appendix F.2, F.4): Extending FJSP evaluation to 120–180 s shrinks the 25j5m NSGA-II schedule-decoding gap from 0.0133 → 0.0020 (Bi) and 0.0112 → 0.0008 (Tri), matching our fixed-FE lead (Appendix D.2). A 144-run baseline convergence pilot (up to 3,000 generations) shows 8/18 routing settings remain unconverged, justifying pre-frozen shared reference points alongside reference-free IGD.

Summary of Revised PDF: To keep the manuscript clean and self-consistent, we left the main-text prose unchanged and updated only the aligned 120 s SEMO table entries (Tables 1, 3, 7 and Appendix D.1), while organizing the new prompts, case studies, statistical tests, and controlled experiments into Appendices E–G (with brief notes in Appendices A.4 and B.4).

Best regards,

The Authors

AuthorsOpenReview ↗

Closing Summary of Discussion and Final Revision Updates

Dear Area Chair and Reviewers,

We sincerely thank Area Chair JKAo and Reviewers t1MH, R78B, and 2MMa for the constructive discussion, and thank Reviewers t1MH and 2MMa for raising their scores following our revision. Below is a brief update on the items completed since our November 16 summary:

  1. Aligned 120 s SEMO fixed-work-budget evaluation (Reviewer 2MMa; Appendix D.2 Tables 12–13, G.3): Now complete in the revised PDF—within the SEMO group, TRACE-EA ranks 1st in 17/24 settings by mean HV and 16/24 by mean IGD (including 5/6 largest-scale settings on both metrics), consistent with our 30 s and 60 s wall-clock results.
  2. Step-level diagnostic breakdown (Reviewer t1MH; Nov 19 follow-up, Appendix E.4): Reported exact counts, denominators, and control conditions across the 24-edit and 72-edit Bi-TSP50 M1 sets against Fixed trajectory summaries and No feedback (95.8%/97.2% target-function edit concentration; 12/24 vs. 8/24 operator-bottleneck repair).
  3. HV reference-point sensitivity check (Reviewer R78B; Appendix F.2): Added to the revised PDF alongside our across-seed statistical tests (F.1, B.4), raw-prompting and scalar-feedback controls (F.3, F.5), and E2OC / 30–180 s FJSP analyses (F.4)—rescoring all stored 30 s fronts under rα=αrs\mathbf{r} _ {\alpha} = \alpha \mathbf{r} _ s (α∈{1.1,1.2,1.5}\alpha \in \{1.1, 1.2, 1.5\}) leaves group-best and overall HV winners unchanged in all 24/24 settings (8-method mean Kendall's τb\tau _ b of 0.9643–0.9911).

Final-Version Updates: Beyond Appendices E–G, the final manuscript will incorporate: (i) a Section 2 comparison with AutoEP, DASH, MoH, and MeLA and a querying-efficiency note (2MMa); (ii) a Section 4 workflow diagram and Bi-TSP50 example, qualification of step-level claims to Bi-TSP50/M1, and the full count table in Appendix E.4 (t1MH); and (iii) refined cross-scale phrasing scoping consistent 30 s gains to TSP/CVRP vs. runtime-dependent FJSP transfer (R78B, t1MH).

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.