- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
MAPLE is an agent for optimization under changing natural-language requirements. It maintains an executable optimization program, event history, representative plans, solution archives, and evolutionary populations across successive user revisions. Three core components handle typed search-space scaffolding (TSS), live problem decomposition (LPD), and live state memory (LSM). A restart selector decides between warm and full initialization. The paper also introduces NLDO, a benchmark of 15 planning trajectories (180 updates) across five task types. MAPLE completes all 15 trajectories with mean scalar quality 0.951 and Pareto HV ratio 0.875, substantially outperforming six adapted baselines.
Critical strengths
- The formulation of "live dynamic optimization" is new at this intersection. Prior work addresses parts but not their combination. The separation of event history for entity lookup from representative plans for assignment retrieval is a clean architectural insight.
Critical weaknesses
-
MAPLE is a complete system. The baselines generate solver code from scratch at each step without any of these components designed in the paper. The enormous gap on dynamic Pareto tasks (MAPLE 0.875 vs. next-best 0.084) primarily measures whether the method maintains evolutionary state, not the marginal contribution of MAPLE's language-understanding components. A fairer comparison would give baselines access to the same NSGA-II solver with fresh initialization at each step, isolating MAPLE's NL understanding and state-management contributions.
-
NLDO and MAPLE were developed together, and the benchmark's update patterns match MAPLE's TSS design well. The problem instances are small. The three episodes per domain share significant structural repetition. Independent validation on externally-designed dynamic optimization benchmarks would strengthen the claims.
Questions for authors
Future work
How would MAPLE handle updates that require changing the decision-space structure (e.g., adding a new decision type not in the TSS catalog)? The current TSS catalog is fixed, and is this a fundamental limitation?
3 comments
I thank the authors for the response. The Always Full ablation partially addresses my first concern by showing that fresh initialization retains most of MAPLE’s advantage (0.825 vs. 0.875). However, it still uses MAPLE’s formulations. The large gap to the best baseline therefore still conflates formulation quality, solver backend, and state reuse. The independent-validation concern also remains: NLDO is co-developed with MAPLE, and the authors agree that external dynamic benchmarks are future work. I maintain my rating of 2
We thank the reviewer for the follow-up and for acknowledging that fresh initialization retains most of MAPLE's advantage.
1. Separating formulation quality, solver backend, and state reuse
We agree that these factors should be separated. Our existing controls do so to a large extent:
| Comparison | What differs | Mean HV ratio |
|---|---|---|
| MAPLE vs. Always Full | State reuse | 0.875 vs. 0.825 |
| MAPLE vs. MAPLE w/o TSS | Formulation (same NSGA-II backend) | 0.875 vs. 0.665 |
| MAPLE vs. baselines at t00 | Formulation and backend, no state reuse | 0.832 vs. 0.440 |
| MAPLE vs. baselines over t00-t12 | All factors, across updates | 0.875 vs. 0.084 |
MAPLE w/o TSS is close to the comparison the reviewer suggested: the model writes its own program from scratch, without MAPLE's formulations, and runs it with the same NSGA-II backend. It still reaches 0.665, far above the baselines. The baselines, in turn, drop from 0.440 at t00 to 0.084 over the trajectories, so most of the remaining gap arises from maintaining the problem across updates, which is the capability MAPLE targets.
2. Independent validation
We agree that external validation is desirable, and Appendix A.4 states this as a limitation. However, no existing benchmark expresses dynamic optimization through successive natural-language updates, so such a benchmark can currently only be built by extending existing ones. NLDO follows this route: its problem formulations and instances come from established benchmarks designed independently of MAPLE (OR-Library, Brandimarte, INRC-II, green vehicle routing, and energy-aware cloud placement), rather than from our own design. We added only the natural-language descriptions and update streams on top of them, which preserves the generality of the underlying problems as far as possible in the current setting.
We respect the reviewer's assessment and thank the reviewer for the detailed engagement.
We thank the reviewer for recognizing the novelty of the setting. We have revised and uploaded the paper in response to the reviewer’s feedback.
1. Question 1
This is not a fundamental limitation. The fixed TSS catalog limits what the current TSS runtime can represent directly, but developers can extend it to support new decision types. The catalog covers common decision types in evolutionary computation, including binary, bounded integer, bounded real, categorical, permutation, assignment, and optional assignment. Segments of these types can be combined in any number and order. When an update changes the decision space, LPD can add, remove, or resize the corresponding segments. All NLDO tasks were expressible using the existing catalog.
Some additional decision structures can be represented through indirect encoding. For example, before evaluating a candidate, MAPLE can decode a real-valued vector into an ordering by sorting its entries or into binary choices by rounding values in [0,1].
When a decision structure requires a dedicated type, a developer can implement its initialization, crossover, mutation, and repair operators and register the type in the TSS catalog. If both indirect encoding and extending the catalog are impractical, the untyped Workbench provides an alternative configuration. We evaluated this configuration in the w/o TSS ablation.
2. Weakness 1
MAPLE constructs an executable optimization problem from an initial natural-language request and revises it as subsequent requests arrive. Existing optimization agents primarily target one-shot problem formulation, and we are unaware of a directly comparable agent designed for successive natural-language updates.
To assess whether MAPLE’s formulations remain effective when search starts from scratch, we use the Always Full ablation. MAPLE interprets each request and updates the executable problem as usual, while NSGA-II initializes a fresh population at every update.
All three results cover the initial state and all 12 updates in each of the six Pareto trajectories. Most of MAPLE’s observed advantage over the best-performing baseline persists with fresh initialization, showing that the optimization problems generated and revised by MAPLE support effective search without reusing earlier candidates. A shared NSGA-II backend, as the reviewer suggests, would allow a direct comparison of the formulations produced by different agents under the same search procedure.
3. Weakness 2
We agree that evaluation on independently designed dynamic optimization tasks would provide further evidence of MAPLE’s generalizability. As noted in Appendix A.4, such validation remains future work.
The underlying optimization problems in NLDO come from established benchmarks designed independently of MAPLE. We draw on their problem formulations and instance formats for selection and allocation (OR-Library, Beasley, 1990), flexible job-shop scheduling (Brandimarte, 1993), nurse rostering (Ceschia et al., 2018), green vehicle routing (Erdoğan & Miller-Hooks, 2012, and Bektaş & Laporte, 2011), and energy-aware cloud placement (Beloglazov et al., 2012). The nurse-rostering instances come from the public INRC-II datasets. For NLDO, we added the natural-language descriptions and 12 cumulative requirement updates per trajectory.
We are unaware of an independently designed benchmark that provides successive natural-language updates to an optimization problem. One next step is to express the changes specified by an existing dynamic routing benchmark as natural-language updates and evaluate MAPLE on the resulting trajectories.
We thank the reviewer for the feedback and look forward to further discussion.