- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes AttriOpt, an attribution-guided evolutionary framework for optimizing LLM-generated API workflows. It uses augmented inverse probability weighting (AIPW) and hierarchical shrinkage to estimate the contributions of API calls, parameter configurations, and interaction patterns from historical evaluations. These attribution scores then guide workflow evolution. Experiments on MILP, TSP, and logic synthesis report improvements over existing API-calling and evolutionary methods.
Critical strengths
- The paper addresses an important credit assignment problem in evolutionary API workflow optimization. Extracting component-level feedback from historical evaluations offers a promising way to guide workflow refinement beyond scalar rewards.
- The evaluation covers three distinct optimization domains. AttriOpt achieves the best mean solving time across all four MILP benchmarks, with notable gains on Cauctions and Facility. These results support the potential value of attribution-guided evolution.
Critical weaknesses
- The reliability of the attribution estimates is not sufficiently established. AIPW is applied to adaptively generated workflows with correlated, non-randomly selected components. It is unclear whether the observed context adequately accounts for selection bias. The reported 78.8% agreement between attribution signs and observed improvements is encouraging but does not establish reliable component-level credit assignment.
- The experimental evidence has several important gaps. The TSP200 results in Tables 2, 4, and 5 use different metrics and report values whose relationship is unclear, while some standard deviations in Table 2 appear unusually large for the stated three-run protocol. The TSP500 runtime advantage may partly reflect a reduced solver search budget (RUNS=2). Standard algorithm configuration baselines such as SMAC or irace are also missing. Finally, the component ablation is limited to TSP200, where removing attribution changes tour length only from 10.69 to 10.75, without variance estimates.
Questions for authors
- How are the assumptions underlying AIPW justified for adaptively generated workflows? Can controlled component interventions validate the attribution estimates?
- Does attribution-guided evolution consistently improve performance across all three domains? How does AIPW compare with simpler attribution methods under matched budgets?
- The same 20 EPFL circuits are used for optimization and evaluation. Can the selected workflows be tested on held-out circuits?
- Can the authors clarify the TSP evaluation settings, particularly the instances, reference tour lengths, and standard deviations in Tables 2, 4, and 5? Are the solver and workflow evaluation budgets consistent across methods?