- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes EvoDAGRec, a multi-agent, LLM-driven code evolution framework for the cascaded training pipeline of generative recommendation. The authors argue that existing evolution frameworks may suffer from cost blindness and short-sighted evaluation in this setting, and address these issues by combining a Code DAG-based search with stage-aware cost planning and active downstream co-adaptation. The framework is evaluated on the Amazon Office dataset against general-purpose and recommender-specific evolution frameworks, and is further validated through an industrial offline evaluation and an online A/B test.
Critical strengths
-
Problem formulation is well motivated. The observation that general-purpose and recommender-specific evolution frameworks have complementary strengths and limitations is convincing. The argument that asymmetric costs per each stage and deferred gains in cascaded training pipelines may violate implicit assumptions of existing search strategies is also reasonable.
-
Authors validate the evolved architecture in a real deployment setting on a large-scale commercial platform.
Critical weaknesses
-
Two mechanisms presented as the core contribution are insufficiently described and validated. The introduction frames cost blindness and short-sighted evaluation as the central problems, and stage-aware cost planning and active downstream co-adaptation as the key solutions. Yet Section 4 describes each mechanism in roughly one paragraph: how the expected gain is estimated in cost planning is not specified, and co-adaptation is described only as conditioning subsequent prompts on a tag. The experiments also do not focus on these mechanisms. A dedicated research question (RQ5) analyzes the diagnostic feedback, which corresponds to the recommender-specific evolution component. In contrast, the two mechanisms are examined mainly through the narrative of a single trajectory and a coarse ablation, in which co-adaptation is introduced together with multi-branch search and is not isolated. No quantitative evidence shows how cost planning changes the per-stage budget allocation. For co-adaptation, cost planning postpones upstream edits until the downstream stages mature, so an upstream edit is ultimately applied on top of downstream stages refined under the previous upstream configuration. Whether its effect holds regardless of this downstream state, and whether the subsequent re-adaptation is effective and cost-efficient, is not examined.
-
Main claims rely largely on a single case study. The evidence for RQ1–RQ3 comes mostly from a post hoc description of one 39-round trajectory, obtained on a single dataset with a single coding LLM, and no repeated runs or variance are reported. It is thus unclear whether the observed behavior is typical, and whether the reported margins over the strongest baselines exceed run-to-run variability. In addition, the industrial evaluation compares the evolved model only with the pre-evolution setting, not with models produced by other evolution frameworks. It therefore does not establish the specific advantage of EvoDAGRec.
-
Novelty of the individual components is limited. Cost-aware optimization that exploits the structure of multi-stage pipelines, such as the higher cost of updating early stages, has been studied in Bayesian optimization [1][2]. DAG-structured evolution histories with merges anchored at the common ancestor have been explored in LLM-based code evolution [3]. The diagnostic feedback follows existing recommender-specific evolution frameworks. The contribution therefore lies mainly in the problem setting and the integration of these components. This makes the lack of direct validation noted above, particularly for the co-adaptation rule, more consequential.
[1] Lin, Chi-Heng, Joseph D. Miano, and Eva L. Dyer. "Bayesian optimization for modular black-box systems with switching costs." Uncertainty in Artificial Intelligence. PMLR, 2021.
[2] Buathong, Poompol, et al. “Bayesian optimization of function networks with partial evaluations.” International Conference on Machine Learning. PMLR, 2024.
[3] Huang, Beichen, Ran Cheng, and Kay Chen Tan. "EvoGit: Decentralized code evolution via git-based multi-agent collaboration." arXiv preprint arXiv:2506.02049 (2025).
Questions for authors
-
Which components are enabled in Variants A, B, and C in Figure 4? In particular, does Variant B include cost planning and co-adaptation tagging, and how is the uniform budget across stages enforced in Variant C?
-
How is the expected gain estimated and traded off against the expected cost? Can the authors provide quantitative evidence of how cost planning changes the search, including the per-stage budget allocation and how often upstream edits are deferred?
-
Can the authors provide quantitative evidence of the role of co-adaptation? This evidence should cover whether the effect of a deferred upstream edit holds after the downstream stages have changed, and how effective and costly the re-adaptation is.
6 comments
Does a deferred upstream edit remain effective after downstream stages have changed, and how effective and costly is downstream re-adaptation?
Thank you for pointing this out. We agree that an upstream edit cannot be assumed to work independently of the downstream state, and the gains of two branches cannot simply be added together. Our co-adaptation mechanism is designed for exactly this situation: an upstream proposal can perform well on its own branch but still require downstream adjustment when integrated with an already optimized downstream model.
We illustrate the changes before and after adaptation through a specific merge of a tokenizer and a pre-training architecture.
Before merging: the two branches optimize different components.
There are two candidate branches:
- Downstream architecture branch: It uses the original tokenizer and optimizes mechanisms including fine-level contrastive learning, user injection into the decoder, and recency bias, reaching Hit@10 of 0.0824.
- New tokenizer branch: It uses Repr-LLM representations and PSRQ quantization, with RMS repair and parameter adjustments completed. It reaches 0.0773 with its own pre-training configuration.
The score of 0.0773 is obtained jointly by the new tokenizer and the pre-training model on that branch. It shows that this upstream proposal has produced a usable recommendation candidate, but does not guarantee that it will remain effective when connected to a different downstream architecture.
Direct merge: the gains of the two branches do not add up.
The agent then merges the new tokenizer with the optimized downstream architecture above and reruns pre-training on the new semantic IDs. At this point, it still uses the original downstream objective settings and has not adjusted the fine-margin for the new tokenizer.
- The combined candidate reaches Hit@10 of 0.0775, only 0.0002 above the new tokenizer branch's 0.0773.
- Compared with the architecture champion using the original tokenizer (0.0824), this is a drop of 0.0049.
Thus, connecting the new tokenizer to the optimized downstream model does not directly preserve the original architecture champion's performance. An effective upstream branch should not be interpreted as an upstream proposal that works with any downstream model.
Downstream adaptation: retain the new tokenizer and adjust the related pre-training objective.
The agent does not immediately abandon the combination. Instead, it keeps the new tokenizer fixed and adjusts the fine-margin twice:
- It sets the margin to 0.5 and reruns pre-training, reaching Hit@10 of 0.0846.
- It further sets the margin to 0.7 and reruns pre-training, reaching Hit@10 of 0.0858.
Both adjustments use the modified tokenizer and reuse the existing embeddings and semantic IDs, without extracting features or running quantization again.
In terms of cost, the two adaptation steps rerun only pre-training and evaluation. They do not pay the tokenizer construction cost again. These limited downstream adjustments recover and exceed the performance before upstream integration, without rerunning the full pipeline.
The industrial evaluation does not compare against models produced by other evolution frameworks.
Thank you for this helpful suggestion. In the industrial setting, we transferred the architecture evolved by Arbor + Diag. to the industrial dataset and evaluated it offline. The results below show that the architecture evolved by EvoDAGRec performs better after transfer.
| Method (Industrial Offline Evaluation) | Hit@1 | Hit@5 | Hit@10 | Hit@20 | Hit@50 |
|---|---|---|---|---|---|
| Baseline | 0.3616 | 0.7188 | 0.8599 | 0.9454 | 0.9940 |
| Arbor + Diag. (Transferred) | 0.3685 | 0.7246 | 0.8628 | 0.9465 | 0.9943 |
| EvoDAGRec (Ours, Transferred) | 0.4056 | 0.7535 | 0.8730 | 0.9508 | 0.9965 |
Which components are enabled in Variants A, B, and C?
Thank you for your question. The three variants are run independently with the following configurations:
| Component | A | B | C | Full |
|---|---|---|---|---|
| Multi-branch exploration | No | Yes | Yes | Yes |
| Co-adaptation tags | No | Yes | Yes | Yes |
| Cross-branch MERGE | No | No | Yes | Yes |
| Cost table and planning instructions | Yes | Yes | No | Yes |
| Free selection of training scope | Yes | Yes | Yes | Yes |
| Reuse of compatible checkpoints | Yes | Yes | Yes | Yes |
| Charging actual machine occupancy time | Yes | Yes | Yes | Yes |
B retains cost planning and co-adaptation and disables only MERGE. C retains the full search capabilities and removes only the cost table and planning instructions. In the experiments above, we also added a variant that disables only co-adaptation tagging. “Uniform budget across stages” means replacing the cost-table-based planning instructions with instructions to consider all stages equally, while allowing the agent to decide freely.
Individual components have limited novelty: cost-aware multi-stage optimization has been studied in Bayesian optimization, DAG-based code evolution in EvoGit, and diagnostics in recommender evolution frameworks.
Thank you for raising this concern, and for recognizing our problem setting. Above, we have added the details of the two core mechanisms: how stage-aware cost planning dynamically affects the allocation of the agent's experimental budget, and how active downstream co-adaptation helps deferred upstream edits adapt efficiently to downstream tasks. We have also added controlled ablations that isolate co-adaptation. Our experiments further demonstrate that these two mechanisms can serve as plug-and-play modules that significantly improve existing general-purpose evolution frameworks.
Although asymmetric costs in cascaded systems have been explored in traditional Bayesian optimization, almost all previous LLM-agent-based code evolution methods are limited to single-stage, end-to-end systems. No prior work has explored how agents can evolve a tightly coupled, multi-stage cascaded system such as generative recommendation (GR). We want to share with the community the key benefits of these two mechanisms for evolving cascaded systems. In the future, we will also extend them to compute-sensitive cascaded tasks such as multi-stage LLM training.
We would also like to emphasize that, beyond these two cascade-specific mechanisms, our Code DAG evolution framework itself has fundamental search advantages over previous methods such as EvoGit and hypothesis-tree methods such as Arbor:
- Compared with EvoGit: EvoGit relies heavily on human experts to periodically inspect branches and manually prune them. It randomly selects code segments of up to 128 lines for unguided edits without long-term memory and uses random strategies to resolve merge conflicts. In contrast, EvoDAGRec is an end-to-end automated loop. It uses lesson distillation to block previously falsified directions and performs diagnosis-driven, targeted hypothesis edits. During merging, agents resolve semantic conflicts using diffs against the lowest common ancestor (LCA), and the Reviewer checks that the mechanisms from each branch remain functionally active.
- Compared with hypothesis-tree methods such as Arbor: Arbor follows a “hypothesis-tree exploration + greedy acceptance into a single main branch” pattern. New experiments start from the current best main branch by default. A change is accepted through a standard Git merge only if its offline metrics immediately exceed those of the main branch; code conflicts cause the merge to fail and stop. It cannot directly combine code from multiple historical branches as joint parents. In contrast, EvoDAGRec retains historical code snapshots as states that can be further expanded and merged, without requiring a branch to first become the main-branch champion. It also defines an explicit multi-parent semantic code-merging operator, MERGE. Thus, complementary branches whose individual scores do not exceed their parents are not greedily discarded. Instead, they can be directly recombined across branches to achieve further gains.
We further compare Arbor + Diag. and EvoDAGRec with a fixed tokenizer and evolution restricted to the architecture used in pre-training. As shown below, even with exactly the same diagnostic feedback and evolution limited to a single stage, EvoDAGRec still shows a clear advantage:
| Method (Fixed Tokenizer, Pre-training-Only Evolution) | Hit@10 | Hit@20 | NDCG@10 | NDCG@20 |
|---|---|---|---|---|
| Arbor + Diag. | 0.0775 | 0.1193 | 0.0394 | 0.0499 |
| EvoDAGRec (Ours) | 0.0824 | 0.1211 | 0.0436 | 0.0533 |
These discussions and experiments have greatly improved the rigor and depth of our paper. We will include them in full in the main text of the revised version. Thank you again for your valuable suggestions!
We appreciate your comments and feedback. We address your concerns below.
Thank you for pointing this out. Section 4 currently does not explain the execution details sufficiently. We will add the following explanations to the main text.
Enabling stage-aware cost planning means explicitly prompting the agent, before an experiment, to use historical results, diagnostics, and accumulated lessons to explain the problem the edit aims to solve and the metrics it is expected to improve. The agent judges the expected gain itself; it is not a numerical prediction or a fixed gain-to-cost formula.
The agent then uses the estimated per-stage cost table (the average training time of each baseline stage), actual durations of historical experiments, reusable training artifacts, and the remaining budget to select the target stage and training scope. Cost depends not only on which stage is edited, but also on which stages need to be rerun. For example:
Estimated cost informs the decision, while actual machine occupancy time measured by the platform is charged against the budget. All experiments use a single L20 GPU with a total budget of 24 GPU-hours, including training and evaluation but excluding the root baseline.
Enabling active downstream co-adaptation means explicitly prompting the agent responsible for topology adjustment to check the following after each experiment:
When diagnostics for an upstream module improve but downstream recommendation performance drops, or when the pre-trained model performs well but post-training worsens its performance, the agent should assign a downstream-adaptation tag. This tag distinguishes an ineffective proposal from a proposal that needs downstream adaptation. It reminds subsequent experiments to consider modifying downstream components rather than abandoning the upstream proposal based only on immediate regression. The tag does not force the next round to execute a specified edit. The agent still uses diagnostics, history, and the budget to decide whether to continue adaptation and which downstream module to modify.
Thank you for pointing this out. The original Variant A disables multi-branch exploration, co-adaptation, and merging together, so it cannot isolate the contribution of co-adaptation. We added three controls: Full without co-adaptation, A with co-adaptation, and Arbor + Diag. with co-adaptation.
These results show that co-adaptation is a plug-and-play design that improves performance, including in linear search and other frameworks. However, it works best with an architecture that supports multi-branch exploration and merging. In a linear structure, even if adaptation of a regressed proposal is allowed, exploration can continue only through a limited set of states. A multi-branch structure can retain other candidates at the same time and continue repairing, comparing, and combining different historical versions. Thank you again for helping us clarify our contribution. Multi-branch search provides a more flexible search space for co-adaptation, and the full framework uses both.