- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper presents EvoTune, an agentic Bayesian optimization framework for system configuration tuning. It represents coordinated parameter changes as semantic Actions, uses an Action–Metric–Objective Graph to guide Action selection, and applies BO to optimize their execution strengths. Trial evidence and feasibility checks govern updates to Action directions and graph relationships. Evaluation covers eight static workloads and three workload-switch experiments across vLLM, PostgreSQL, and Spark, together with component ablations on Spark.
Critical strengths
-
A coherent combination of semantic reasoning and numerical optimization. Separating coordinated adjustment directions from their execution strengths provides a useful way to incorporate system knowledge into BO. Graph-guided selection and evidence-gated representation updates give the framework a concrete mechanism for responding to changing bottlenecks.
-
Broad and encouraging empirical validation. Experiments span three system types, both static and changing workloads, and several relevant baselines. The Spark ablations provide additional evidence for the roles of Action updates, graph updates, and selection. Explicitly including validation and failed executions in the trial budget also strengthens the evaluation protocol.
Critical weaknesses
-
The statistical robustness of the reported improvements is insufficiently characterized. Section 5.2 mentions experiments with different initial configurations, but does not report their counts or aggregate variability. Appendix F.3 clarifies that the main static results are best-observed endpoints rather than independently confirmed performance. Reporting uncertainty and confirmation results would help establish whether the improvements over the strongest baselines are consistent rather than sensitive to measurement noise or initialization.
-
The relationship between the specified selection rule and the implementation needs clarification. Equation 7 uses average absolute GP posterior gradients to measure Action sensitivity. However, Appendix K.3 states that the inspected v2 diagnosis code instead estimates positive posterior objective gain and combines it with confidence and repeated response support. These are distinct scores, and it is unclear which produced the reported results. This matters for reproducing the method and interpreting the Action-selection ablation.
Questions for authors
-
Which Action-scoring rule was used for each system in Figures 5–8: the gradient-based rule in Equation 7 or the posterior-gain-based implementation described in Appendix K.3? Does this distinction materially affect the results or ablation conclusions?
-
How many independent optimization runs and initialization settings support the reported comparisons? Please provide variability estimates and the independent confirmation results described in Appendix F.3, particularly for comparisons against the strongest baselines.
3 comments
Dear Reviewer WVVR,
With just over one day remaining before the author–reviewer discussion deadline on November 18 (AoE), we would appreciate your feedback on our response. We clarified that all three systems used the absolute posterior-gradient rule in Eq. (7) whenever Action screening was enabled, and corrected Appendix K.3 accordingly. To address your concerns about repeatability and endpoint confirmation, we added five independent searches per static workload and four additional measurements of each selected configuration, with paired 95% confidence intervals reported in Appendices I.4–I.5. The confirmation results favor EvoTune on all eight static workloads.
Could you please share your assessment of these clarifications and additional results? We would be happy to address any remaining questions before the deadline. Thank you for your time.
Dear Reviewer WVVR, we have devoted considerable effort to strengthening the statistical evaluation through repeated searches, independent endpoint confirmation, and confidence intervals, while also clarifying the Action-scoring rule used in the experiments. As the discussion period draws to a close, we would greatly appreciate it if you could review our response and revised manuscript and share your assessment. Thank you for your thoughtful feedback and time.
Response to Reviewer WVVR
Thank you for your review. We address all your concerns below.
vLLM, PostgreSQL, and Spark all used the absolute posterior-gradient rule in Eq. (7) whenever Action screening was enabled. The positive posterior-gain score in the previous Appendix K.3 came from an older implementation. The revised K.3 now specifies Eq. (7), the rule used in these experiments.
Each method completed five independent searches per static workload, with a common first-trial configuration across methods for each seed. The advantage persisted after independent remeasurement on all eight static workloads. Figures 5–6 now show mean best-so-far curves across the five seeds, with pointwise 95% confidence bands for EvoTune and the strongest baseline. After the budget was exhausted, we locked the configuration with the best valid observed objective from each run and measured it four more times. The four repeats contribute only to the confirmation statistics. Relative to the strongest remeasured baseline, EvoTune reduced TPC-H latency by 1.17 s (paired 95% CI 0.67–1.66 s) and increased PageRank throughput by 3.98 thousand ranks/s (2.14–5.81). Appendix I.4 gives the seed mapping and endpoint sample variances. Appendix I.5 reports the paired confirmation interval for each of the eight workloads. These replication counts and uncertainty estimates apply to the static experiments in Figs. 5–6.