acceptodds
Under review as a conference paper at ICLR 2027

Language-Guided Tuning: Configuration Optimization for Automated ML Research

Abstract

Configuration optimization remains a critical bottleneck in machine learning, requiring coordinated tuning across model architecture, training strategy, feature engineering, and hyperparameters. Traditional approaches treat these dimensions independently and lack interpretability, while recent automated methods struggle with dynamic adaptability and semantic reasoning about optimization decisions. We introduce Language-Guided Tuning (LGT), a framework that employs multi-agent Large Language Models to automatically optimize configurations through natural language reasoning. We apply textual feedback signals that complement numerical optimization by providing semantic understanding of training dynamics and configuration interdependencies. LGT coordinates three specialized agents: an Advisor that proposes configuration changes, an Evaluator that assesses progress, and an Optimizer that refines the decision-making process, creating a self-improving feedback loop. Through comprehensive evaluation on seven diverse datasets, LGT demonstrates substantial improvements over traditional optimization methods while maintaining high interpretability.

Primary area:
optimization
Keywords:
AutoResearch, LLM, ML Optimization, Hyperparameter Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_V9vDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Language-Guided Tuning (LGT), a configuration optimization framework where an Advisor proposes edits, an Evaluator assesses outcomes, and a Prompt Optimizer generates guidance for subsequent proposals. It couples inter-cycle architecture changes with epoch-level adjustments to features, training strategies, and hyperparameters using a validation-based retention criterion. Experiments demonstrate favorable mean performance across seven tabular/text datasets and a MiniViT vision extension.

Critical strengths
  1. Unified search framework and traceability: LGT unifies model architecture, feature engineering, training strategies, and hyperparameters into a coherent optimization workflow. The closed-loop synergy between the Advisor, Evaluator, and Prompt Optimizer, alongside natural-language recording of trial decisions, improves the transparency and traceability of the optimization trajectory.
  2. Broad experimental coverage and diagnostic transparency: The paper covers diverse task types (classification, regression, and character-level generation), includes domain ablations, and evaluates alternative LLM backbones. Detailed prompt templates and illustrative success/failure logs (Tables 2–5; Appendices F–H) facilitate inspection of the agent workflow.
Critical weaknesses
  1. Insufficient controlled comparisons and agent-role validation: LGT is benchmarked against baselines restricted to narrower search spaces, without establishing comparable total computational budgets or evaluating against matched LLM-based optimizers. Furthermore, the ablation studies only ablate editable search domains rather than agent roles, leaving the actual utility of the Advisor–Evaluator–Optimizer multi-agent decomposition unverified. As a result, the observed gains cannot be definitively attributed to the proposed architecture rather than differences in available intervention levers or expanded computational budgets.
  2. Confounded configuration attribution and evaluation discrepancies: In Algorithm 1, an edited candidate configuration evaluated after an additional training epoch is compared against the incumbent checkpoint without an unchanged-configuration continuation control. This design fails to disentangle the benefit of the proposed edit from ordinary continuous training progress. Additionally, internally inconsistent reporting of MNIST AUC mean and standard deviation, unreconciled split descriptions, and unverified test-set independence weaken confidence in the empirical findings.
Questions for authors
  1. Could you reconcile the MNIST AUC statistics between Table 2 and Table 5 by providing the individual values across the 10 runs and the exact aggregation procedure? Please also clarify the exact train/validation/test splits and explicitly confirm whether test set metrics influenced any configuration-selection decisions.
  2. How does LGT compare against classical and LLM-based optimizers when evaluated under a strictly shared search space and comparable total compute budgets? Can you provide a matched single-agent baseline to demonstrate whether the multi-role decomposition drives the performance improvement?
  3. How does an edited candidate compare against the unchanged configuration continued for the exact same number of epochs starting from the same incumbent checkpoint? Furthermore, does evaluating over a longer horizon materially alter which edits are retained?
Reviewer_pRMsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Language Guided Tuning (LGT), which organizes ML configuration experiments into Advisor, Evaluator and Prompt Optimizer roles. It proposes architecture changes between training cycles and compatible feature, training-strategy, and hyperparameter changes at epoch boundaries, with a numerical validation gate for retaining candidates. Validation includes seven compact datasets, comparisons with classical tuning methods, single-domain ablations, and a MiniViT extension.

Critical strengths
  1. This paper proposes a three-role tuning framework (proposal, assessment, guidance) is practical and covers architecture, features, training strategy and hyperparameters through one interface.
Critical weaknesses
  1. The paper seems to be largely AI-generated. It repeatedly negates or hedges its own claims and contradicts itself, mainly in the experimental sections. For example, it reports both a 72/8/20 and an 80/20 split, and it calls CIFAR-10 balanced in Fig.4 but later appeals to class imbalance.
  2. The method is technically simple, and similar structures have already appeard in prior work (e.g., OPRO[1], TextGrad[2], AgentHPO[3]). The authors also state that none of these components alone distinguishes their approach.
  3. Some of the key terms lack clear definitions. For example, “training cycle” is never defined and should be distinguished from “step” and “epoch”. Also, “iteration” and “attempt” are used inconsistently, and “baseline” refers both to No Tuning and to the current incumbent configuration.
  4. The citations are insufficient. Some of the closely related LLM-based HPO work, such as LLAMBO[4], is not mentioned in paper.
  5. The experiments lack comparisons with LLM-based baselines (e.g., AgentHPO[3], OptiMindTune[5]), so the reported gains cannot be attributed to the proposed mechanism.
  6. The AI use statement on page 10 is problematic. It contains a broken reference (“Appendix ??”) which is incomplete.

[1] Yang, Chengrun, et al. "Large language models as optimizers." International Conference on Learning Representations. Vol. 2024. 2024.

[2] Yuksekgonul, Mert, et al. "Textgrad: Automatic" differentiation" via text." arXiv preprint arXiv:2406.07496 (2024).

[3] Liu, Siyi, Chen Gao, and Yong Li. "Large language model agent for hyper-parameter optimization." arXiv preprint arXiv:2402.01881 (2024).

[4] Liu, Tennison, et al. "Large language models to enhance bayesian optimization." International Conference on Learning Representations. Vol. 2024. 2024.

[5] Madiraju, Meher Bhaskar, and Meher Sai Preetam Madiraju. "Optimindtune: A multi-agent framework for intelligent hyperparameter optimization." arXiv preprint arXiv:2505.19205 (2025).

Questions for authors
  1. Is the Evaluator’s output signal determined solely by performance metrics?
  2. Is the history in section 3.3 simply concatenated across iterations? How is it compressed and how to handle cases when the context becomes too long?
  3. During agent interaction, does each round modify a single parameter or jointly change multiple parameters? If multiple parameters change together, how to determine which change caused an improvement and how is a wrong attribution corrected?
  4. Tables 2 and 5 report an MNIST AUC of 99.99 ± 0.10. Is this plausible? With a mean of 99.99 on a [0, 100] scale, a standard deviation of 0.10 over ten runs seems impossible.
Reviewer_PTeoOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes Language-Guided Tuning (LGT), in which three LLM roles (Advisor, Evaluator and Optimizer) propose, assess and guide edits to a model's architecture, feature engineering, training strategy and hyperparameters during training. Compatible edits are tried for one epoch from the current checkpoint. They are kept only if the Evaluator approves and validation improves by a margin; architecture changes happen between training cycles. On seven datasets, LGT has the best reported mean results against untuned training and ten classical tuners, with further experiments on MiniViT and four LLM backbones. The stated contributions are the two timescales, conditional validation-gated retention and a shared interface across the four edit domains.

Critical strengths
  1. An inspectable design for LLM-driven configuration: The two timescales follow from how each edit affects training state. Under the formal rule, a candidate is kept only with both a measured validation gain and Evaluator approval, so persuasive text alone cannot retain it. Logged examples include failed attempts and even an incorrect agent rationale (Appendix G). This design is useful even apart from the headline results.

  2. Honest scoping of the analysis: The theory appendix separates an accounting identity from results that need extra assumptions, which it calls "conditional analyses of an idealized acceptance process, not verified guarantees for the reported implementation". It does not claim that LGT converges, so the retention rule can be judged on its stated scope.

Critical weaknesses
  1. By the paper's own account, the aggregation, implementation and split behind its main results still need verification. Appendix J says these open details "limit conclusions from the current configuration experiments":

    • The MNIST AUC of 99.99 ± 0.10 cannot be a ten-run mean and standard deviation. Appendix J shows this, yet retains the entry "pending verification". The Iris accuracy standard deviations cannot all come from ten runs on a 30-item test set either, under either standard-deviation formula, and the paper does not flag them.
    • The implemented comparison target, margin and state restoration of the retention rule, the core of the method, still "require code-level verification".
    • Section 4.2 describes a 72/8/20 split, Appendix A an 80/20 split without a validation set, and Figure 2 plots test-loss curves, so test independence from adaptive selection "remains unestablished".
    • On Tiny Shakespeare, LGT's perplexity of 1.72 is far below the roughly 4.35 of the public nanoGPT shakespeare_char example on a contiguous 90/10 split. This is a calibration point, not a matched comparison or a bound, but until the split and windowing are specified, it is hard to reconcile with held-out next-character prediction.

    The paper is open about most of these gaps, but openness does not replace verification. Until the checks are done, with reruns where the implementation or split differs from the description, Tables 2–3 cannot support the headline comparison. On the submitted record, this is the main reason I recommend clear rejection.

  2. The paper does not establish the added value over simpler LLM-based tuning. It states that "neither online adaptation, LLM-based proposals, nor multiple agents alone distinguishes our approach", so the novelty rests on their combination. The comparisons do not isolate it:

    • LGT edits four domains, while the classical baselines tune narrower ones.
    • The 50-configuration and 10-epoch caps do not establish matched budgets once rejected candidates, restarts and LLM calls are counted (Section 5.4).
    • There is no matched LLM-based baseline. Single-agent, TextGrad/GEPA and autoresearch comparisons are listed as missing (Appendix I), and Table 5 varies edit domains, not agent roles.
    • As written, Algorithm 1 trains a candidate one more epoch before comparing it with the incumbent's earlier score. While training still improves, a neutral edit can pass the numerical gate on that extra epoch alone.

    Even taking the tables at face value, the paper therefore does not test whether LGT improves on a simpler history-conditioned LLM tuner.

Questions for authors
  1. How does LGT compare with a simpler history-conditioned LLM tuner that shares its backbone, edit interface, retention rule and total budget?
  2. From the same checkpoints, how often does a no-edit continuation pass the numerical gate, compared with the proposed edits?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.