- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper presents TRACE, a framework that extends LLM-based automated heuristic design for dynamic resource assignment from scalar fitness feedback to structured knowledge extracted from system execution logs. An agentic workflow pairs a Reasoner agent, which formulates hypotheses about unknown system dynamics in light of the optimization objective, with a Coder agent, which writes and executes schema-specific code to produce textual insights or executable surrogate models (e.g., processing-time predictors); these inform an evolutionary loop that generates assignment heuristics. Evaluated on a synthetic cloud benchmark and a measurement-driven 5G vRAN scenario, TRACE outperforms FunSearch, EoH, ReEvo, and HSEvo, reduces energy use versus an industry-standard policy while maintaining zero task loss, and yields more auditable heuristics at under 2% overhead.
Critical strengths
The first strength is a clearly identified and plausible limitation of existing LLM-based heuristic design: scalar fitness cannot explain why one heuristic beats another when system dynamics are unknown, and execution logs are an obvious, largely untapped source of that explanation. The second strength is the 5G case study grounded in industrial testbed measurements and operational traces is also a stronger form of evaluation than purely synthetic benchmarks.
Critical weaknesses
The first and decisive weakness is that the absolute quality of the solutions TRACE produces is unknown, which limits the scientific value of the claimed gains. No optimality gap, lower bound, or even gap-to-oracle is reported in either scenario; all headline numbers ("36× lower cost than EoH," "65% less energy than HA-Only") are purely relative. Because the cost is dominated by the drop penalty, these numbers largely encode a threshold effect — TRACE achieves zero loss while baselines do not — rather than a measured distance from good solutions. Combined with the fact that the best privileged oracle (Oracle-Time) already achieves zero loss in the 5G case and near-zero loss with 1.30 ms delay in Scenario 1, it is impossible to tell how much headroom TRACE leaves on the table or whether a simple informed rule would match it.
The second weakness is that the baseline set cannot support the paper's own framing. The intro positions the work against "learning-based schedulers [that] excel in research," yet no DRL scheduler (Mao et al. 2016; Li et al. 2023, both cited), no classical load-balancing policy (join-shortest-queue, least-loaded), and — most critically — no "surrogate + simple rule" control is evaluated.
Questions for authors
Q1. What is the quality of TRACE's solutions in absolute terms? Specifically: (a) in the cloud scenarios, what is the gap between TRACE's cost/delay and the best achievable policy with known dynamics (e.g., an oracle that knows the true processing-time functions and optimizes the assignment), or a simulated lower bound; (b) in the 5G scenario, how close is TRACE's energy consumption to the minimum energy among all zero-loss policies? Without such a reference, the reported gains cannot be interpreted.
Q2. The most direct test of your mechanism would be a hand-written baseline that receives the same extracted knowledge (the processing-time surrogate, per-processor speed and energy statistics) and applies a simple transparent rule, e.g., assign to the feasible processor minimizing predicted waiting plus processing time, or minimizing energy subject to the deadline. How does TRACE compare against such a surrogate-plus-greedy rule? If the gap is small, what justifies the evolutionary loop and the LLM machinery? If it is large, the comparison would substantially strengthen the paper.
Q3. The paper cites DRL schedulers as the learning-based state of the art for resource assignment but does not compare against any. Why? Even accounting for differences in observability and interface, could you adapt one DRL baseline (e.g., a policy trained with full state information, or evaluated under your restricted interface) so that readers can position TRACE against the paradigm the introduction argues it should replace in deployment?
The authors have responded to your reviews. Please read their response and reply. Please specify which concerns are resolved and which remain. Please update your review and score if the assessment has changed.
AC