- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Language-Guided Tuning (LGT), a configuration optimization framework where an Advisor proposes edits, an Evaluator assesses outcomes, and a Prompt Optimizer generates guidance for subsequent proposals. It couples inter-cycle architecture changes with epoch-level adjustments to features, training strategies, and hyperparameters using a validation-based retention criterion. Experiments demonstrate favorable mean performance across seven tabular/text datasets and a MiniViT vision extension.
Critical strengths
- Unified search framework and traceability: LGT unifies model architecture, feature engineering, training strategies, and hyperparameters into a coherent optimization workflow. The closed-loop synergy between the Advisor, Evaluator, and Prompt Optimizer, alongside natural-language recording of trial decisions, improves the transparency and traceability of the optimization trajectory.
- Broad experimental coverage and diagnostic transparency: The paper covers diverse task types (classification, regression, and character-level generation), includes domain ablations, and evaluates alternative LLM backbones. Detailed prompt templates and illustrative success/failure logs (Tables 2–5; Appendices F–H) facilitate inspection of the agent workflow.
Critical weaknesses
- Insufficient controlled comparisons and agent-role validation: LGT is benchmarked against baselines restricted to narrower search spaces, without establishing comparable total computational budgets or evaluating against matched LLM-based optimizers. Furthermore, the ablation studies only ablate editable search domains rather than agent roles, leaving the actual utility of the Advisor–Evaluator–Optimizer multi-agent decomposition unverified. As a result, the observed gains cannot be definitively attributed to the proposed architecture rather than differences in available intervention levers or expanded computational budgets.
- Confounded configuration attribution and evaluation discrepancies: In Algorithm 1, an edited candidate configuration evaluated after an additional training epoch is compared against the incumbent checkpoint without an unchanged-configuration continuation control. This design fails to disentangle the benefit of the proposed edit from ordinary continuous training progress. Additionally, internally inconsistent reporting of MNIST AUC mean and standard deviation, unreconciled split descriptions, and unverified test-set independence weaken confidence in the empirical findings.
Questions for authors
- Could you reconcile the MNIST AUC statistics between Table 2 and Table 5 by providing the individual values across the 10 runs and the exact aggregation procedure? Please also clarify the exact train/validation/test splits and explicitly confirm whether test set metrics influenced any configuration-selection decisions.
- How does LGT compare against classical and LLM-based optimizers when evaluated under a strictly shared search space and comparable total compute budgets? Can you provide a matched single-agent baseline to demonstrate whether the multi-role decomposition drives the performance improvement?
- How does an edited candidate compare against the unchanged configuration continued for the exact same number of epochs starting from the same incumbent checkpoint? Furthermore, does evaluating over a longer horizon materially alter which edits are retained?