acceptodds
Under review as a conference paper at ICLR 2027

Towards Agent-Friendly Documentation for Database Tuning

Abstract

System-tuning documentation provides parameter recommendations but often leaves their deployment-specific validation implicit, limiting their relevance across environments. We present PerfEvolve, a framework that augments static tuning guidance with executable procedural knowledge for LLM-based tuning agents. Offline, PerfEvolve ranks parameters by sensitivity, screens candidate ranges, and identifies interacting groups, compiling the results into procedures with measurements, decision rules, and validation checks. At deployment, these procedures guide local experiments or supply empirical priors to existing tuners. On PostgreSQL 16 with TPC-C and TPC-H workloads, integrating PerfEvolve into GPTuner improves matched-environment performance by 1.4%–6.1%, increases the valid-trial rate from 68%–81% to 100%, and yields gains of 45.8%–58.9% under cross-hardware transfer. With E2ETune, the full procedural representation achieves a 35.2% improvement over default PostgreSQL. These results support combining executable procedures with empirical reference data to improve tuning reliability and adaptation across deployments. Code and artifacts are available at https://anonymous.4open.science/r/PerfEvolve-DF10.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Large Language Model Agents, Procedural Knowledge, Executable Skills, Agentic System Optimization, Automated Database Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_FveSOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

The paper proposes PerfEvolve, which turns database tuning guidance into procedures containing measurements, decision rules, and checks. Offline profiling ranks PostgreSQL parameters, screens ranges, and identifies interacting groups. The resulting knowledge is injected into GPTuner and E2ETune. On 3 PostgreSQL workloads, GPTuner gains 1.4–6.1 percentage points over its original version on matched VMs and 45.8–58.9 points on a different server; valid trials rise from 68%–81% to 100% on the VMs. The server gains are relative to a very weak transferred GPTuner baseline: 2 of the 3 resulting configurations still underperform default PostgreSQL. E2ETune gains are smaller and mixed.

Critical strengths
  1. The paper frames a concrete problem: tuning documentation rarely specifies how to validate its advice on a new deployment. The proposed skill structure makes that missing decision process explicit
  2. Evaluation includes 2 different tuners and a hardware transfer setting. The sharp failure of original GPTuner after transfer makes the practical value of empirical parameter selection visible.
Critical weaknesses
  1. The central attribution is unresolved. The GPTuner comparison changes the parameter set from 57 to 15 and changes the allowed ranges as well as the knowledge representation . The authors acknowledge that these effects are bundled . In E2ETune, procedural steps without reference data underperform the vanilla model, while the full package exceeds it by only 0.7–1.6 points on the server. A matched-information comparison is needed to establish the claimed advantage of procedural documentation itself.
  2. The evidence for reliability and cost effectiveness is incomplete. The main tables give means but no run-level variation or paired intervals for gains as small as 0.7–1.4 points. Building the offline profile uses about 11,000 benchmark runs, with no measured break-even analysis . Moreover, the evaluated integrations do not demonstrate full agent execution of the skill DAG.
Questions for authors
  1. With the same 15 parameters, ranges, optimizer, and online trial budget, how much does the procedural form add over an equally informed declarative baseline? Can you report a full skill-DAG execution condition separately from knowledge injection?
  2. What are the run-level results or paired uncertainty intervals for Table 4 and Table 5, and how many deployments are needed to amortize the approximately 11,000 offline runs under realistic recalibration?
3 comments

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your feedback. Below, we respectfully provide our detailed responses to address your concerns.


Q1: "What does procedural form add under identical information and budget? Is full DAG execution demonstrated?"

1. Information-matched evaluation: isolating procedural knowledge.

We conducted a pre-registered experiment on real PostgreSQL 16, controlling all empirical information and execution resources:

FactorExperimental Setting
AgentDeepSeek-V4-Flash
KnowledgeSame 15 parameters, ranges, and 150 numerical facts
Tools & BudgetIdentical interfaces; ≤20 benchmark calls
Evaluation2 workloads × 5 independent seeds/condition
ValidationIndependent remeasurement against default

The procedural version adds measurement, branching, verification, and fallback instructions—not additional empirical facts.

Results (mean ± SD, five seeds):

WorkloadProceduralDeclarativeDifference (95% CI; p)
TPC-C-r+0.6 ± 0.8%−0.9 ± 4.2%+1.5 pp [−1.1, +5.3]; p=0.72
TPC-C-w−3.6 ± 2.5%−14.3 ± 22.4%+10.7 pp [−0.8, +31.3]; p=0.30

Although mean differences are not statistically significant, procedural knowledge produces distinct verification and fallback behavior:

Behavioral EvidenceProceduralDeclarative
Validated and retained default8/101/10
Worst observed regression−6.8%−54.5%

2. Full Skill-DAG execution on real PostgreSQL.

Unlike the original knowledge-injection experiments, our new study includes 30 complete agent sessions, with all execution traces available.

Workflow: Context Check → Measurement → Reference Validation → Parameter Testing → A/B Verification → Accept/Revert.

A representative TPC-C-w trace demonstrates measurement-driven adaptation:

StepObservationAgent Decision
Transfer checkLocal/reference ratio 0.77, below 0.8Reassess offline assumptions
Parameter verificationFour values degrade by 14–17%Reject harmful configurations
Interleaved A/BCandidate 764/765 vs. default 897/845Reject candidate; retain default

This directly demonstrates executable procedures that guide agents through validation, conditional branching, and recovery from unsuccessful tuning attempts.

3. Distinguishing procedural execution from search-space optimization.

GPTuner's SMAC optimizer consumes search spaces, so procedural representation cannot affect its behavior when parameters and ranges are identical.

Nevertheless, empirical screening provides an independent benefit: documentation-derived ranges produced 30–68 penalty trials per 100, versus zero with screened ranges in observed VM runs.


Q2: "What is the uncertainty and realistic amortization?"

1. Strengthened experimental reliability.

  • Independent evaluation: The new study reports five independent seeds per condition, SDs, bootstrap CIs, and permutation tests.
  • Original results: GPTuner used one tuning run per cell; E2ETune used one sampled configuration measured three times. These cannot establish end-to-end tuning variability.
  • Statistical interpretation: E2ETune Full–Vanilla yielded p≥0.48; we therefore focus on the directly validated procedural behavior.

2. Measured execution cost and amortization.

The new experiment provides a directly measured procedural-execution budget of 12–20 benchmark calls per session.

Cost ComponentEvidence
Offline profiling~17,300 recorded evaluations
Online procedural execution12–20 calls/session
Empirical screening30–68 fewer penalty trials per 100
Break-evenDepends on reuse/recalibration; not measured

Even assuming 11,000 offline runs and treating avoided penalty trials as equivalent savings, amortization would require approximately 160–370 deployments, excluding recalibration.

Importantly, offline profiling constructs reusable knowledge, while online procedural execution validates its applicability. Our new study demonstrates that such validation is feasible within a bounded benchmark budget.


At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!

Thank you for the additional clarifications. I have no further questions at this point, I will raise my soundness score to 3, but maintain my overall rating.

Thank you very much for your thoughtful follow-up and for reconsidering the soundness assessment. We sincerely appreciate your recognition of the additional experimental evidence.

We would like to emphasize that our new information-matched, real-system experiments directly address the central concern regarding procedural knowledge: beyond knowledge injection, PerfEvolve enables agents to execute verifiable tuning procedures, detect invalid assumptions, and adapt their decisions through validation and fallback.

We believe this strengthens not only the technical soundness but also the distinctive methodological contribution of our work. We sincerely appreciate your consideration and hope these clarifications may also help inform your final overall assessment.

Thank you again for your valuable feedback!

Reviewer_SPrfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper points out that database tuning documentation usually tells you what value to set (e.g., "shared_buffers = 25% of RAM''), but not how to check whether that value is right for your own machine and workload. This matters because recent LLM-based tuners use the documentation as their main source of knowledge. Using PostgreSQL, the authors show that the official advice and PGTune make seven TPC-H queries 4--27% slower, and that a configuration tuned on read-heavy TPC-C (+35.2%) loses performance on write-heavy TPC-C and TPC-H.

To address this, they propose PerfEvolve. Offline, it changes one setting at a time to find which settings matter (15 of 116 for PostgreSQL) and which ranges are safe, then tests pairs of settings to find the ones that should be tuned together. The results are written as step-by-step procedures ("skills'') for an LLM agent. This takes about 11,000 benchmark runs per database version.

They add this knowledge to two existing tuners. For GPTuner, the 57 settings it normally takes from the documentation are replaced by PerfEvolve's 15 settings and measured ranges. This improves results by 1.4--6.1 points on the VMs used for profiling, makes all trials valid (up from 68--81%), and on a larger server takes GPTuner from about 55% below the default to around the default. For E2ETune, which produces a configuration in one shot, the gains are small (up to about 2 points).

Critical strengths
  • The problem is well motivated. The examples in Section 2 are convincing, for instance tuning two settings separately gives −13.4%-13.4\% while tuning them together gives +35.2%+35.2\%.

  • The idea of documenting the tuning process, not only the final values, is simple and sensible, and fits well with LLM agents that can run experiments.

  • The offline pipeline uses standard and well-understood methods (sensitivity scans, factorial experiments, ANOVA with multiple-testing correction), which makes it easy to follow.

  • The approach works with two quite different tuners without changing their optimizer or model. The GPTuner results on the larger server and the 100% valid-trial rate are a clear practical improvement.

  • The paper is open about its limitations (Appendix G), and the code is released.

Critical weaknesses
  • The paper is framed around procedures that an agent follows, but as far as I understand, the experiments give the knowledge to existing tuners instead: GPTuner gets the selected settings, ranges and interactions as JSON, and E2ETune gets them in its input. The authors also mention in Appendix A.2 that this does not show an agent following the procedure graph. An experiment where an agent actually runs the procedures would make the main idea much more convincing.

  • The comparison with GPTuner doesn't include the offline cost. PerfEvolve uses about 11,000 runs on the same VMs used for evaluation, while GPTuner gets 100 trials. It would help to see how much of the gain comes simply from having these extra measurements, for example by giving the same data to an existing method for selecting important settings (such as the knob ranking in OtterTune, or Kanellis et al., 2020, which the paper cites).

  • In Table 5, procedures with reference data beat the other variants, but the variants also differ in what information they include, not only in format (Appendix E.1 says this too). So I'm not sure the table shows that the procedural format itself is what helps.

  • The results use 3 repetitions (5 in Table~5) without error bars or standard deviations. Several differences are small, e.g.\ E2ETune's −0.7-0.7 to +2.0+2.0 points on the VMs, so it's hard to tell how reliable they are. Also, the introduction says that the shared_buffers ×\times {work_mem} interaction explains 39% of performance variance, but I could not find where this is shown in the paper.

  • The evaluation is limited to PostgreSQL, three workloads (TPC-H at scale factor 1) and one transfer server. On that server, GPTuner with PerfEvolve is still below the default on both TPC-C workloads (−3.3%-3.3\% and −10.6%-10.6\%). Results on another system, such as MySQL, would make the claims more general. It might also be worth discussing DB-BERT (Trummer, SIGMOD 2022), which also uses database manuals for tuning.

Questions for authors

Please address the weaknesses and possible mitigations.

3 comments

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your feedback. Below, we respectfully provide our detailed responses to address your concerns.


W1: "Does an agent actually follow the proposed procedures?"

Yes. We now demonstrate full procedural execution on real PostgreSQL 16, beyond the original knowledge-injection experiments.

  • Setup: DeepSeek-V4-Flash; 3 knowledge conditions (Procedural/Declarative/No Knowledge); 2 workloads (TPC-C-r/TPC-C-w); 5 independent seeds each.
  • Execution: Real configuration/benchmark tools, verified through pg_settings; ≤20 benchmark calls.
  • Workflow: Context Check → Measurement → Reference Validation → Parameter Testing → A/B Verification → Accept/Revert.

Representative TPC-C-w execution:

StepObservationAgent Decision
Reference checkLocal ratio 0.77, outside [0.8, 1.25]Reassess offline knowledge
Parameter testingFour recommended values cause 14–17% degradationReject harmful values
A/B validationCandidate 764/765 vs. default 897/845Reject candidate; retain default

This demonstrates that PerfEvolve enables agents to validate assumptions, branch on real measurements, and recover from failed recommendations—not merely consume static hints.


W2: "Is the benefit only additional profiling or knob selection?"

PerfEvolve contributes at three complementary levels:

LevelContribution
Knowledge AcquisitionIdentify useful parameters and empirical constraints
Knowledge RepresentationEncode measurements, conditions, and actions
Agent ExecutionValidate, adapt, and safely revert decisions

Empirical range screening provides practical value: documentation-derived ranges produced 30–68 penalty trials per 100, versus zero with screened ranges in observed VM runs.

Thus, PerfEvolve's distinctive contribution extends beyond knob selection: it enables deployment-time verification of empirical knowledge through executable procedures.


W3: "Does representation itself matter when information is held fixed?"

Our new information-matched experiment provides direct evidence of different agent behavior. Both conditions contain identical 15 parameters, ranges, and 150 numerical facts (automatically verified), with identical LLMs, tools, and budgets.

Performance relative to default (mean ± SD, five seeds):

DeploymentProceduralDeclarativeDifference (95% CI)
TPC-C-r+0.6 ± 0.8%−0.9 ± 4.2%+1.5 pp [−1.1, +5.3]
TPC-C-w−3.6 ± 2.5%−14.3 ± 22.4%+10.7 pp [−0.8, +31.3]

While mean differences are not statistically significant, procedural instructions produce distinct validation and fallback behavior:

Behavioral evidenceProceduralDeclarative
Validated and retained default8/101/10
Worst observed regression−6.8%−54.5%

As a result, procedural knowledge changes not merely what an agent knows, but how it acts—encouraging evidence-based validation rather than unverified configuration changes.**


W4: "What about uncertainty and the 39% interaction claim?"

We strengthened experimental rigor through independent runs, uncertainty estimates, and a comprehensive audit.

1. Statistical reliability.

  • Our new study reports five independent seeds per condition, SDs, bootstrap CIs, and permutation tests.
  • Original GPTuner/E2ETune repetitions did not establish independent end-to-end tuning variability; E2ETune Full–Vanilla yielded p≥0.48. We therefore narrow the corresponding performance claims.

2. Interaction verification.

  • The original 39% corresponded to partial η²=0.394 (uncorrected p=0.040), but did not survive FDR correction (q=0.12).
  • Our 5,038-run replicated study found no FDR-significant interaction edges; we remove the unsupported topology conclusions.

This supports the practical importance of validating parameter combinations, without asserting a confirmed global interaction map.


W5–W6: "Generalization and comparison with DB-BERT"

1. Workload adaptation. While our experiments focus on PostgreSQL 16, the TPC-C-r → TPC-C-w setting demonstrates adaptation under workload shift: the agent identifies invalid offline assumptions, rejects harmful configurations, and falls back safely.

2. Relationship to DB-BERT.

AspectDB-BERTPerfEvolve
KnowledgeExisting manual hintsEmpirically enriched knowledge
MechanismLearn to weight/select hintsGenerate executable procedures
FocusBenchmark-guided hint selectionValidation, branching, fallback

At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!

Thanks for the rebuttal. Did you add the results of the responses to W4 somewhere in the paper? Without that, it is not clear how the results become statistically reliable after the new experiments. In addition, the response to W5-W6 are only textual. Providing details results will help us to evaluate better.

Thank you very much for your constructive follow-up. Our new experiments do more than re-evaluate the original results: they provide an independently replicated, information-matched evaluation of actual procedural execution, directly addressing the central concern about PerfEvolve's contribution.

1. W4: Statistical Reliability and New Experimental Evidence

New controlled evaluation on real PostgreSQL 16:

  • 30 independent agent sessions: 3 knowledge conditions × 2 workloads × 5 seeds.
  • Controlled information: Identical 15 parameters, ranges, and 150 numerical facts across procedural and declarative conditions.
  • Evaluation: Independent final-configuration remeasurement, reporting mean ± SD, bootstrap 95% CIs, and permutation tests.
WorkloadProceduralDeclarativeDifference (95% CI; p)
TPC-C-r+0.6 ± 0.8%−0.9 ± 4.2%+1.5 pp [−1.1, +5.3]; p=0.72
TPC-C-w−3.6 ± 2.5%−14.3 ± 22.4%+10.7 pp [−0.8, +31.3]; p=0.30

Performance relative to default; mean ± SD across five seeds.

Procedural execution produces directly observable verification behavior: 8/10 procedural sessions validated and retained default, compared with 1/10 declarative sessions. The worst observed regressions were −6.8% and −54.5%, respectively.

These results provide stronger evidence for our primary methodological claim: executable procedures change how an agent validates and applies identical empirical knowledge. They do not establish a statistically significant throughput or population-level risk advantage.

We also clarify that the original GPTuner/E2ETune measurements cannot establish independent end-to-end variability. Accordingly, the revised evaluation will distinguish those results from the new independently replicated study rather than treating the latter as retroactive validation of the former.


2. W5: Quantitative Evidence for Workload-Shift Adaptation

We appreciate the request for concrete results. Our new experiment includes a TPC-C-r → TPC-C-w workload shift, explicitly testing whether the agent can detect when offline knowledge becomes unreliable.

A representative real execution shows:

Validation StageQuantitative EvidenceAgent Behavior
Reference validationLocal/reference ratio 0.77, below 0.8Detect mismatch
Parameter verificationFour recommendations degrade throughput by 14–17%Reject harmful priors
Interleaved A/B testingCandidate 764/765 vs. default 897/845Revert to default
Final outcomeDefault retained after failed validationAvoid deploying the rejected candidate

Across five shifted-workload seeds, the procedural agent retained default in 5/5 sessions.

This is a meaningful distinction from static documentation: PerfEvolve enables agents to verify when previously useful knowledge no longer applies and change their decisions accordingly.

Furthermore, our audit revealed that BenchBase's reported throughput can include failed requests (observed failure rates of 60–71% under SERIALIZABLE isolation). Our new evaluation therefore uses goodput and verified applied configurations, reducing the risk of treating failed transactions as successful tuning outcomes.


3. W6: Clarifying the Contribution Beyond DB-BERT

Thank you for identifying DB-BERT as an important related work.

DimensionDB-BERTPerfEvolve
KnowledgeManual-derived tuning hintsEmpirically enriched procedures
MechanismLearn hint selection/weightingExplicit measurement and decision rules
AdaptationBenchmark-feedback-driven tuningValidation-conditioned branching and fallback
Main evidenceTuning optimizationReal agent execution and decision traces

DB-BERT demonstrates the value of learning from documentation and benchmark feedback. PerfEvolve addresses a complementary problem: making the validation process itself explicit, reusable, and directly executable by agents.

We do not claim quantitative superiority over DB-BERT without a matched experimental comparison.


Summary

Thank you for requesting more concrete evidence. Our central advance is not merely better parameter recommendations, but an agent's ability to determine when empirical knowledge should be trusted, verified, or rejected.

We hope these additional results may address your concerns and strengthen the evidence for PerfEvolve's contribution. We would be grateful if you could reconsider your assessment in light of the new evaluation.

Reviewer_GSDhOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes PerfEvolve, a framework for converting database-tuning knowledge into procedures that specify measurements, actions, decision rules, and validation checks. Offline, it profiles individual parameters, screens their ranges, confirms candidate pairwise interactions using replicated factorial experiments and FDR-corrected ANOVA, and uses the resulting graph to organize joint search. These results inform a skill DAG or are injected as knowledge into existing tuners. The evaluation uses PostgreSQL 16, two modified TPC-C workloads, TPC-H, a VM environment, and a larger server. With 100 GPTuner trials, empirical knowledge improves matched-environment results by 1.4–6.1 percentage points over the original tuner and increases the reported valid-trial rate from 68%–81% to 100%. Server improvements are 45.8–58.9 percentage points, although both TPC-C results remain below default PostgreSQL. E2ETune receives smaller server gains of 0.7–1.6 percentage points; its TPC-H result increases from 33.6% to 35.2% above default. Knowledge-form comparisons and component ablations provide additional evidence, with approximately 11,000 offline benchmark runs reported. The principal validation concerns knowledge injection; execution of the proposed procedural workflow is not comparably established.

Critical strengths

A useful formulation of deployment-specific tuning knowledge. Preserving measurements, decision rules, and applicability conditions is a meaningful improvement over recording a selected configuration alone. Figure 1b makes the motivation concrete: a jointly tuned configuration improves its source workload by 35.2%, yet loses 4.3% and 18.6% on the other workloads. This supports the importance of preserving workload context and checking whether recommendations transfer.

A practically relevant improvement in search reliability. At a fixed optimizer and 100-trial online budget, GPTuner’s reported valid-trial rate increases from 68%–81% to 100%, alongside improvements on all three matched-environment workloads (Table 4). Reducing unsuccessful evaluations is valuable when database experiments are expensive. Integration with a second tuner also demonstrates that the empirical knowledge can be consumed through more than one interface, even though E2ETune’s incremental gains are modest.

Critical weaknesses

The central contribution of executable procedural documentation is not isolated or directly validated. GPTuner receives selected parameters, ranges, and interaction annotations while its search dimension changes from 57 to 15. E2ETune receives additional context and still produces a configuration in one inference. Appendix A.2 explicitly notes that this does not demonstrate benchmark-driven traversal of the skill DAG, and Appendix D supplies a conceptual specification rather than an actual exported procedure or execution trace. These experiments support the usefulness of empirically informed search spaces and prompts, but do not establish the benefit of executing procedural documentation. Furthermore, Table 5 changes both representation and information content, as Appendix E.1 acknowledges. A baseline containing the same measurements, ranges, and interaction information without procedural control flow is missing. This distinction matters because parameter selection and range optimization already appear in the original GPTuner; the proposed procedural layer needs its own evidence.

The quantitative evidence leaves the reliability and practical value of the gains unresolved. E2ETune’s server improvements are only 0.7–1.6 percentage points over Vanilla, but Tables 4–7 provide no run-level dispersion or confidence intervals. The reported three/five repetitions are not sufficiently resolved into independent configuration-generation/tuning runs versus repeated measurements of a fixed configuration. GPTuner’s much larger transfer gains are measured against a baseline more than 54% below default; the manuscript does not clearly establish how target-hardware constraints, the default configuration, and incumbent selection were handled. These details could materially change the interpretation. Cost evidence also needs reconciliation: approximately 11,000 offline runs are required, while Appendix F gives conditional amortization rather than a measured break-even result. Section 3.2 reports components of at most three to four parameters, but Figure 6 appears to depict a connected region substantially larger than four. Since component size determines the joint-search cost, this is consequential to the claimed efficiency, even if it ultimately proves to be a figure error.

Questions for authors
  1. Clarification of implemented execution. Which reported experiment actually runs an agent through the skill DAG, including measurements, branching, and postcondition checks? Please identify the corresponding results and provide an actual generated skill/prompt and a representative trace, ideally showing how a failed transfer check changes subsequent actions. If the evaluated integrations only perform knowledge injection, what evidence supports the full-execution claim and its 30–120-run budget?

  2. Controlled evidence for the main contribution. Can you compare the full procedural representation with a reference-data/constraint representation that contains the same parameter set, ranges, interaction graph, and numerical observations, with a comparable context and evaluation budget? This is a control needed to distinguish the paper’s central contribution from the benefit of additional empirical information, rather than a suggestion for a new application.

  3. Clarification of the reported improvements. Please provide run-level results or uncertainty estimates, distinguish independent tuning/generation runs from repeated benchmark measurements, and define the TPC-H performance metric and “valid trial.” For server GPTuner, were the 100 trials run on the target server with target-aware constraints, was default PostgreSQL an eligible incumbent, and is the reported result the best validated configuration? What accounts for the original tuner’s greater-than-54% degradation?

  4. Clarification of topology and cost accounting. Please reconcile Figure 6’s apparent larger connected component with the three-to-four-parameter maximum in Section 3.2. What are the actual retained parameter set, confirmed edges, and component sizes for each workload, and how do the executed screening/confirmation/component-search counts produce the approximately 11,000-run total? Please distinguish measured counts from illustrative calculations and specify any target-deployment recalibration cost.

1 comment

Thank you very much for your time and effort in reviewing our paper. We sincerely appreciate your feedback. Below, we respectfully provide our detailed responses to address your concerns.


Q1. Is the skill DAG truly executed, including failed-transfer branching?

We agree that the original GPTuner/E2ETune integrations demonstrated knowledge injection, not full procedural execution. To address this gap, we conducted a new experiment on real PostgreSQL 16 (2 vCPU, 8 GB).

  • Setup: DeepSeek-V4-Flash agent; 3 knowledge conditions × 2 workloads × 5 seeds = 30 sessions.
  • Execution: The agent applies configurations, verifies settings through pg_settings, and runs actual benchmarks.
  • Budget: 20 benchmark calls maximum; procedural sessions used 12–20 calls. All 30 execution traces are available.

Representative execution trace (TPC-C-w, seed 1):

Executed skillObservationDecision
Check applicabilityLocal baseline 977 / offline reference 1,276 = 0.77, outside [0.8, 1.25]Invalidate direct transfer; test hypotheses
Verify alternativesFour suggested settings cause 14–17% regressionsReject harmful values
Validate candidateCandidate 764/765 vs. interleaved default 897/845Revert to default

This provides direct evidence that a failed validation check changes the agent's subsequent actions, rather than merely producing a static recommendation.


Q2. Does procedural representation help when empirical information is identical?

GPTuner's SMAC optimizer consumes search spaces, so identical parameters and ranges make procedural/declarative representations indistinguishable at its interface.

We therefore conducted a controlled LLM-agent comparison using:

  • Identical information: 15 parameters, ranges, reference observations, and all 150 numerical values, verified automatically.
  • Identical conditions: Model, tools, 20-call budget, and evaluation harness.
  • Independent evaluation: Five seeds per condition and final configuration remeasurement.

New results (mean ± SD improvement over default):

WorkloadProceduralDeclarativeDifference, 95% CI
TPC-C-r+0.6 ± 0.8%−0.9 ± 4.2%+1.5 pp [−1.1, +5.3]
TPC-C-w−3.6 ± 2.5%−14.3 ± 22.4%+10.7 pp [−0.8, +31.3]

The clearest result is decision behavior: 8/10 procedural sessions validated and retained default versus 1/10 declarative sessions. The worst observed regression was −6.8% versus −54.5%. This isolates auditable validation actions, not an assumed benefit from extra facts.


Q3. How dependable are the original gains and cross-hardware results?

We rechecked the experimental units and metrics. Each original GPTuner performance cell was one tuning run (30 trials, not 100); each E2ETune condition was one generated configuration measured three times, not three independent generations. E2ETune Full–Vanilla differences have p ≥ 0.48. Accordingly, the revised paper does not rely on those performance differences as established gains; the new controlled study supplies seed-level uncertainty instead.

On the server, trials ran on target hardware but inherited VM ranges; default was absent from SMAC's initial design, final best trials were not independently revalidated, and one TPC-H cell used an inconsistent latency metric. A narrower measured finding remains: documentation-derived ranges caused 30–68 penalty trials per 100 in individual VM runs versus none with screened ranges, under the stated VM criterion.


Q4. What is the verified topology and cost?

Figure 6 used an unreplicated graph (16 vertices; components of 12 and 4). Replicated confirmation (5,038 runs) found no FDR-significant edges. We remove the topology, 39% claim, and joint-search efficiency inference. A separate measured joint-versus-independent example remains (48.6 pp, n=3, p=0.037), illustrating the value of testing combinations without claiming a general interaction graph.


At last, we sincerely appreciate your valuable feedback. We have carefully considered all your suggestions and we would be deeply grateful if you could kindly reconsider raising the score to 3 or above. Thank you very much!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.