- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3(+1)
Read review
Paper summary
The paper proposes PerfEvolve, which turns database tuning guidance into procedures containing measurements, decision rules, and checks. Offline profiling ranks PostgreSQL parameters, screens ranges, and identifies interacting groups. The resulting knowledge is injected into GPTuner and E2ETune. On 3 PostgreSQL workloads, GPTuner gains 1.4–6.1 percentage points over its original version on matched VMs and 45.8–58.9 points on a different server; valid trials rise from 68%–81% to 100% on the VMs. The server gains are relative to a very weak transferred GPTuner baseline: 2 of the 3 resulting configurations still underperform default PostgreSQL. E2ETune gains are smaller and mixed.
Critical strengths
- The paper frames a concrete problem: tuning documentation rarely specifies how to validate its advice on a new deployment. The proposed skill structure makes that missing decision process explicit
- Evaluation includes 2 different tuners and a hardware transfer setting. The sharp failure of original GPTuner after transfer makes the practical value of empirical parameter selection visible.
Critical weaknesses
- The central attribution is unresolved. The GPTuner comparison changes the parameter set from 57 to 15 and changes the allowed ranges as well as the knowledge representation . The authors acknowledge that these effects are bundled . In E2ETune, procedural steps without reference data underperform the vanilla model, while the full package exceeds it by only 0.7–1.6 points on the server. A matched-information comparison is needed to establish the claimed advantage of procedural documentation itself.
- The evidence for reliability and cost effectiveness is incomplete. The main tables give means but no run-level variation or paired intervals for gains as small as 0.7–1.4 points. Building the offline profile uses about 11,000 benchmark runs, with no measured break-even analysis . Moreover, the evaluated integrations do not demonstrate full agent execution of the skill DAG.
Questions for authors
- With the same 15 parameters, ranges, optimizer, and online trial budget, how much does the procedural form add over an equally informed declarative baseline? Can you report a full skill-DAG execution condition separately from knowledge injection?
- What are the run-level results or paired uncertainty intervals for Table 4 and Table 5, and how many deployments are needed to amortize the approximately 11,000 offline runs under realistic recalibration?
3 comments
Thank you for the additional clarifications. I have no further questions at this point, I will raise my soundness score to 3, but maintain my overall rating.
Thank you very much for your thoughtful follow-up and for reconsidering the soundness assessment. We sincerely appreciate your recognition of the additional experimental evidence.
We would like to emphasize that our new information-matched, real-system experiments directly address the central concern regarding procedural knowledge: beyond knowledge injection, PerfEvolve enables agents to execute verifiable tuning procedures, detect invalid assumptions, and adapt their decisions through validation and fallback.
We believe this strengthens not only the technical soundness but also the distinctive methodological contribution of our work. We sincerely appreciate your consideration and hope these clarifications may also help inform your final overall assessment.
Thank you again for your valuable feedback!
Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your feedback. Below, we respectfully provide our detailed responses to address your concerns.
Q1: "What does procedural form add under identical information and budget? Is full DAG execution demonstrated?"
1. Information-matched evaluation: isolating procedural knowledge.
We conducted a pre-registered experiment on real PostgreSQL 16, controlling all empirical information and execution resources:
The procedural version adds measurement, branching, verification, and fallback instructions—not additional empirical facts.
Results (mean ± SD, five seeds):
Although mean differences are not statistically significant, procedural knowledge produces distinct verification and fallback behavior:
2. Full Skill-DAG execution on real PostgreSQL.
Unlike the original knowledge-injection experiments, our new study includes 30 complete agent sessions, with all execution traces available.
Workflow: Context Check → Measurement → Reference Validation → Parameter Testing → A/B Verification → Accept/Revert.
A representative TPC-C-w trace demonstrates measurement-driven adaptation:
This directly demonstrates executable procedures that guide agents through validation, conditional branching, and recovery from unsuccessful tuning attempts.
3. Distinguishing procedural execution from search-space optimization.
GPTuner's SMAC optimizer consumes search spaces, so procedural representation cannot affect its behavior when parameters and ranges are identical.
Nevertheless, empirical screening provides an independent benefit: documentation-derived ranges produced 30–68 penalty trials per 100, versus zero with screened ranges in observed VM runs.
Q2: "What is the uncertainty and realistic amortization?"
1. Strengthened experimental reliability.
2. Measured execution cost and amortization.
The new experiment provides a directly measured procedural-execution budget of 12–20 benchmark calls per session.
Even assuming 11,000 offline runs and treating avoided penalty trials as equivalent savings, amortization would require approximately 160–370 deployments, excluding recalibration.
Importantly, offline profiling constructs reusable knowledge, while online procedural execution validates its applicability. Our new study demonstrates that such validation is feasible within a bounded benchmark budget.
At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!