acceptodds
Under review as a conference paper at ICLR 2027

EvoDAGRec: A Multi-Agent Code DAG Evolution Framework for Generative Recommendation

Abstract

Large Language Model (LLM) agents have enabled automated code evolution, offering a promising paradigm for algorithm discovery. However, current recursive self-improvement (RSI) research for sequential recommendation faces a critical dilemma: general-purpose code evolution frameworks possess flexible search mechanisms but lack domain-specific feedback, whereas recommender-specific evolution frameworks utilize rich feedback but rely on linear evolution. More critically, naively integrating them fails when applied to Generative Recommendation (GR). Unlike traditional end-to-end models, GR relies on a module-cascaded training pipeline (i.e., tokenization, pre-training, and alignment). In this paradigm, existing naive evolution methods inevitably suffer from i) cost blindness: squandering budgets by ignoring stage-asymmetric retraining overheads; and ii) short-sighted evaluation: discarding promising upstream innovations that temporarily regress metrics before downstream adaptation. To bridge these gaps, we propose EvoDAGRec, a multi-agent code evolution framework explicitly tailored for GR. EvoDAGRec organizes code evolution as a Directed Acyclic Graph (DAG) that explicitly maintains multiple evolution branches and merges complementary innovations, and harmonizes an advanced search mechanism with rich GR-specific diagnostic feedback. We further introduce stage-aware cost planning to optimize real-world GPU-hour budgets, and active downstream co-adaptation to navigate temporary regressions and unlock deferred cross-stage gains. On public Amazon datasets, EvoDAGRec marks the first autonomous evolution of a complete GR pipeline, improving offline metrics by up to 37.4% and boosting Agent4Rec simulated user satisfaction by 19.7% over the strongest GR baseline. Furthermore, in industrial settings, the evolved GR achieves a 12.16% relative improvement in offline Hit@1 and, when deployed as an additional retrieval channel on a large-scale commercial media platform, a statistically significant 0.40% increase in online click-through rate (CTR).

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Auto Research Agent, Self Evolving, Generative Recommendation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_vWmsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes EvoDAGRec, a multi-agent, LLM-driven code evolution framework for the cascaded training pipeline of generative recommendation. The authors argue that existing evolution frameworks may suffer from cost blindness and short-sighted evaluation in this setting, and address these issues by combining a Code DAG-based search with stage-aware cost planning and active downstream co-adaptation. The framework is evaluated on the Amazon Office dataset against general-purpose and recommender-specific evolution frameworks, and is further validated through an industrial offline evaluation and an online A/B test.

Critical strengths
  1. Problem formulation is well motivated. The observation that general-purpose and recommender-specific evolution frameworks have complementary strengths and limitations is convincing. The argument that asymmetric costs per each stage and deferred gains in cascaded training pipelines may violate implicit assumptions of existing search strategies is also reasonable.

  2. Authors validate the evolved architecture in a real deployment setting on a large-scale commercial platform.

Critical weaknesses
  1. Two mechanisms presented as the core contribution are insufficiently described and validated. The introduction frames cost blindness and short-sighted evaluation as the central problems, and stage-aware cost planning and active downstream co-adaptation as the key solutions. Yet Section 4 describes each mechanism in roughly one paragraph: how the expected gain is estimated in cost planning is not specified, and co-adaptation is described only as conditioning subsequent prompts on a tag. The experiments also do not focus on these mechanisms. A dedicated research question (RQ5) analyzes the diagnostic feedback, which corresponds to the recommender-specific evolution component. In contrast, the two mechanisms are examined mainly through the narrative of a single trajectory and a coarse ablation, in which co-adaptation is introduced together with multi-branch search and is not isolated. No quantitative evidence shows how cost planning changes the per-stage budget allocation. For co-adaptation, cost planning postpones upstream edits until the downstream stages mature, so an upstream edit is ultimately applied on top of downstream stages refined under the previous upstream configuration. Whether its effect holds regardless of this downstream state, and whether the subsequent re-adaptation is effective and cost-efficient, is not examined.

  2. Main claims rely largely on a single case study. The evidence for RQ1–RQ3 comes mostly from a post hoc description of one 39-round trajectory, obtained on a single dataset with a single coding LLM, and no repeated runs or variance are reported. It is thus unclear whether the observed behavior is typical, and whether the reported margins over the strongest baselines exceed run-to-run variability. In addition, the industrial evaluation compares the evolved model only with the pre-evolution setting, not with models produced by other evolution frameworks. It therefore does not establish the specific advantage of EvoDAGRec.

  3. Novelty of the individual components is limited. Cost-aware optimization that exploits the structure of multi-stage pipelines, such as the higher cost of updating early stages, has been studied in Bayesian optimization [1][2]. DAG-structured evolution histories with merges anchored at the common ancestor have been explored in LLM-based code evolution [3]. The diagnostic feedback follows existing recommender-specific evolution frameworks. The contribution therefore lies mainly in the problem setting and the integration of these components. This makes the lack of direct validation noted above, particularly for the co-adaptation rule, more consequential.

[1] Lin, Chi-Heng, Joseph D. Miano, and Eva L. Dyer. "Bayesian optimization for modular black-box systems with switching costs." Uncertainty in Artificial Intelligence. PMLR, 2021.

[2] Buathong, Poompol, et al. “Bayesian optimization of function networks with partial evaluations.” International Conference on Machine Learning. PMLR, 2024.

[3] Huang, Beichen, Ran Cheng, and Kay Chen Tan. "EvoGit: Decentralized code evolution via git-based multi-agent collaboration." arXiv preprint arXiv:2506.02049 (2025).

Questions for authors
  1. Which components are enabled in Variants A, B, and C in Figure 4? In particular, does Variant B include cost planning and co-adaptation tagging, and how is the uniform budget across stages enforced in Variant C?

  2. How is the expected gain estimated and traded off against the expected cost? Can the authors provide quantitative evidence of how cost planning changes the search, including the per-stage budget allocation and how often upstream edits are deferred?

  3. Can the authors provide quantitative evidence of the role of co-adaptation? This evidence should cover whether the effect of a deferred upstream edit holds after the downstream stages have changed, and how effective and costly the re-adaptation is.

6 comments

We appreciate your comments and feedback. We address your concerns below.

The two core mechanisms need more implementation detail: how is expected gain estimated and traded off against cost, and how does the co-adaptation tag guide subsequent edits?

Thank you for pointing this out. Section 4 currently does not explain the execution details sufficiently. We will add the following explanations to the main text.

Enabling stage-aware cost planning means explicitly prompting the agent, before an experiment, to use historical results, diagnostics, and accumulated lessons to explain the problem the edit aims to solve and the metrics it is expected to improve. The agent judges the expected gain itself; it is not a numerical prediction or a fixed gain-to-cost formula.

The agent then uses the estimated per-stage cost table (the average training time of each baseline stage), actual durations of historical experiments, reusable training artifacts, and the remaining budget to select the target stage and training scope. Cost depends not only on which stage is edited, but also on which stages need to be rerun. For example:

  • With a compatible pre-training checkpoint, the agent can run post-training only.
  • After modifying the tokenizer, it can first evaluate pre-training without immediately running post-training.
  • When modifying quantization parameters, it can reuse embeddings and only rerun quantization and pre-training.

Estimated cost informs the decision, while actual machine occupancy time measured by the platform is charged against the budget. All experiments use a single L20 GPU with a total budget of 24 GPU-hours, including training and evaluation but excluding the root baseline.

Enabling active downstream co-adaptation means explicitly prompting the agent responsible for topology adjustment to check the following after each experiment:

When diagnostics for an upstream module improve but downstream recommendation performance drops, or when the pre-trained model performs well but post-training worsens its performance, the agent should assign a downstream-adaptation tag. This tag distinguishes an ineffective proposal from a proposal that needs downstream adaptation. It reminds subsequent experiments to consider modifying downstream components rather than abandoning the upstream proposal based only on immediate regression. The tag does not force the next round to execute a specified edit. The agent still uses diagnostics, history, and the budget to decide whether to continue adaptation and which downstream module to modify.

Co-adaptation is not isolated from multi-branch search in the original ablation. What is its independent contribution?

Thank you for pointing this out. The original Variant A disables multi-branch exploration, co-adaptation, and merging together, so it cannot isolate the contribution of co-adaptation. We added three controls: Full without co-adaptation, A with co-adaptation, and Arbor + Diag. with co-adaptation.

These results show that co-adaptation is a plug-and-play design that improves performance, including in linear search and other frameworks. However, it works best with an architecture that supports multi-branch exploration and merging. In a linear structure, even if adaptation of a regressed proposal is allowed, exploration can continue only through a limited set of states. A multi-branch structure can retain other candidates at the same time and continue repairing, comparing, and combining different historical versions. Thank you again for helping us clarify our contribution. Multi-branch search provides a more flexible search space for co-adaptation, and the full framework uses both.

Method / VariantHit@10Hit@20NDCG@10NDCG@20
A (Greedy)0.06730.10890.03340.0438
A + Active downstream co-adaptation0.06950.11230.03470.0454
Arbor + Diag. (Baseline)0.08140.12720.04090.0550
+ Stage-aware cost planning0.08260.12930.04410.0558
+ Active downstream co-adaptation0.08320.13050.04460.0565
+ Both mechanisms0.08480.13350.04610.0583
Full without co-adaptation0.08300.12930.04390.0555
Full EvoDAGRec0.08690.13890.04780.0609

Does a deferred upstream edit remain effective after downstream stages have changed, and how effective and costly is downstream re-adaptation?

Thank you for pointing this out. We agree that an upstream edit cannot be assumed to work independently of the downstream state, and the gains of two branches cannot simply be added together. Our co-adaptation mechanism is designed for exactly this situation: an upstream proposal can perform well on its own branch but still require downstream adjustment when integrated with an already optimized downstream model.

We illustrate the changes before and after adaptation through a specific merge of a tokenizer and a pre-training architecture.

Before merging: the two branches optimize different components.

There are two candidate branches:

  • Downstream architecture branch: It uses the original tokenizer and optimizes mechanisms including fine-level contrastive learning, user injection into the decoder, and recency bias, reaching Hit@10 of 0.0824.
  • New tokenizer branch: It uses Repr-LLM representations and PSRQ quantization, with RMS repair and parameter adjustments completed. It reaches 0.0773 with its own pre-training configuration.

The score of 0.0773 is obtained jointly by the new tokenizer and the pre-training model on that branch. It shows that this upstream proposal has produced a usable recommendation candidate, but does not guarantee that it will remain effective when connected to a different downstream architecture.

Direct merge: the gains of the two branches do not add up.

The agent then merges the new tokenizer with the optimized downstream architecture above and reruns pre-training on the new semantic IDs. At this point, it still uses the original downstream objective settings and has not adjusted the fine-margin for the new tokenizer.

  • The combined candidate reaches Hit@10 of 0.0775, only 0.0002 above the new tokenizer branch's 0.0773.
  • Compared with the architecture champion using the original tokenizer (0.0824), this is a drop of 0.0049.

Thus, connecting the new tokenizer to the optimized downstream model does not directly preserve the original architecture champion's performance. An effective upstream branch should not be interpreted as an upstream proposal that works with any downstream model.

Downstream adaptation: retain the new tokenizer and adjust the related pre-training objective.

The agent does not immediately abandon the combination. Instead, it keeps the new tokenizer fixed and adjusts the fine-margin twice:

  • It sets the margin to 0.5 and reruns pre-training, reaching Hit@10 of 0.0846.
  • It further sets the margin to 0.7 and reruns pre-training, reaching Hit@10 of 0.0858.

Both adjustments use the modified tokenizer and reuse the existing embeddings and semantic IDs, without extracting features or running quantization again.

In terms of cost, the two adaptation steps rerun only pre-training and evaluation. They do not pay the tokenizer construction cost again. These limited downstream adjustments recover and exceed the performance before upstream integration, without rerunning the full pipeline.

The industrial evaluation does not compare against models produced by other evolution frameworks.

Thank you for this helpful suggestion. In the industrial setting, we transferred the architecture evolved by Arbor + Diag. to the industrial dataset and evaluated it offline. The results below show that the architecture evolved by EvoDAGRec performs better after transfer.

Method (Industrial Offline Evaluation)Hit@1Hit@5Hit@10Hit@20Hit@50
Baseline0.36160.71880.85990.94540.9940
Arbor + Diag. (Transferred)0.36850.72460.86280.94650.9943
EvoDAGRec (Ours, Transferred)0.40560.75350.87300.95080.9965

Which components are enabled in Variants A, B, and C?

Thank you for your question. The three variants are run independently with the following configurations:

ComponentABCFull
Multi-branch explorationNoYesYesYes
Co-adaptation tagsNoYesYesYes
Cross-branch MERGENoNoYesYes
Cost table and planning instructionsYesYesNoYes
Free selection of training scopeYesYesYesYes
Reuse of compatible checkpointsYesYesYesYes
Charging actual machine occupancy timeYesYesYesYes

B retains cost planning and co-adaptation and disables only MERGE. C retains the full search capabilities and removes only the cost table and planning instructions. In the experiments above, we also added a variant that disables only co-adaptation tagging. “Uniform budget across stages” means replacing the cost-table-based planning instructions with instructions to consider all stages equally, while allowing the agent to decide freely.

Individual components have limited novelty: cost-aware multi-stage optimization has been studied in Bayesian optimization, DAG-based code evolution in EvoGit, and diagnostics in recommender evolution frameworks.

Thank you for raising this concern, and for recognizing our problem setting. Above, we have added the details of the two core mechanisms: how stage-aware cost planning dynamically affects the allocation of the agent's experimental budget, and how active downstream co-adaptation helps deferred upstream edits adapt efficiently to downstream tasks. We have also added controlled ablations that isolate co-adaptation. Our experiments further demonstrate that these two mechanisms can serve as plug-and-play modules that significantly improve existing general-purpose evolution frameworks.

Although asymmetric costs in cascaded systems have been explored in traditional Bayesian optimization, almost all previous LLM-agent-based code evolution methods are limited to single-stage, end-to-end systems. No prior work has explored how agents can evolve a tightly coupled, multi-stage cascaded system such as generative recommendation (GR). We want to share with the community the key benefits of these two mechanisms for evolving cascaded systems. In the future, we will also extend them to compute-sensitive cascaded tasks such as multi-stage LLM training.

We would also like to emphasize that, beyond these two cascade-specific mechanisms, our Code DAG evolution framework itself has fundamental search advantages over previous methods such as EvoGit and hypothesis-tree methods such as Arbor:

  • Compared with EvoGit: EvoGit relies heavily on human experts to periodically inspect branches and manually prune them. It randomly selects code segments of up to 128 lines for unguided edits without long-term memory and uses random strategies to resolve merge conflicts. In contrast, EvoDAGRec is an end-to-end automated loop. It uses lesson distillation to block previously falsified directions and performs diagnosis-driven, targeted hypothesis edits. During merging, agents resolve semantic conflicts using diffs against the lowest common ancestor (LCA), and the Reviewer checks that the mechanisms from each branch remain functionally active.
  • Compared with hypothesis-tree methods such as Arbor: Arbor follows a “hypothesis-tree exploration + greedy acceptance into a single main branch” pattern. New experiments start from the current best main branch by default. A change is accepted through a standard Git merge only if its offline metrics immediately exceed those of the main branch; code conflicts cause the merge to fail and stop. It cannot directly combine code from multiple historical branches as joint parents. In contrast, EvoDAGRec retains historical code snapshots as states that can be further expanded and merged, without requiring a branch to first become the main-branch champion. It also defines an explicit multi-parent semantic code-merging operator, MERGE. Thus, complementary branches whose individual scores do not exceed their parents are not greedily discarded. Instead, they can be directly recombined across branches to achieve further gains.

We further compare Arbor + Diag. and EvoDAGRec with a fixed tokenizer and evolution restricted to the architecture used in pre-training. As shown below, even with exactly the same diagnostic feedback and evolution limited to a single stage, EvoDAGRec still shows a clear advantage:

Method (Fixed Tokenizer, Pre-training-Only Evolution)Hit@10Hit@20NDCG@10NDCG@20
Arbor + Diag.0.07750.11930.03940.0499
EvoDAGRec (Ours)0.08240.12110.04360.0533

These discussions and experiments have greatly improved the rigor and depth of our paper. We will include them in full in the main text of the revised version. Thank you again for your valuable suggestions!

Reviewer_w2zLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

EvoDAGRec uses LLM agents to improve the code of a generative recommendation pipeline through branching search, merging, and experimental feedback. Its DAG search preserves and merges branches, accounts for stage-dependent retraining costs, and adapts downstream components when upstream changes initially hurt performance.

Critical strengths

A concrete, well-motivated problem. Cascaded pipelines have unequal edit costs and delayed benefits, making immediate-score greedy search a poor fit.

Practical evidence beyond simulation. The reported industrial transfer and 14-day A/B test provide useful evidence of real-world value.

Critical weaknesses

The novelty (for example distinction from Arbor) needs stronger justification. Previous work already supports branching, experimental feedback, refinement, and merging. The novelty therefore rests mainly on stage-aware planning and downstream adaptation, and the paper should show whether these mechanisms outperform benchmark, e.g. Arbor, given the same pipeline information and explicit adaptation instructions.

The evaluation is narrow. The experiments focus only on Amazon Office despite broader wording in the abstract. More datasets are needed to show that the gains are consistent rather than dependent on a particularly successful search trajectory.

Some practical claims exceed what the controls isolate. The GPU budget excludes LLM-token costs, while adding a retrieval channel online does not by itself isolate the benefit of the evolutionary method from the benefit of adding a channel.

Questions for authors

see weaknesses

4 comments

We appreciate your comments and feedback. We address your concerns below.

The novelty, particularly the distinction from Arbor, needs stronger justification.

Thank you for your suggestion. We would like to emphasize that almost all previous LLM-agent-based code evolution methods are limited to single-stage, end-to-end systems. No prior work has explored how agents can evolve a tightly coupled, multi-stage cascaded system such as generative recommendation (GR). We want to share with the community the key benefits of these two mechanisms for evolving cascaded systems. In the future, we will also extend them to compute-sensitive cascaded tasks such as multi-stage LLM training.

As shown below, adding these two mechanisms to Arbor, a previous state-of-the-art hypothesis-tree evolution framework, also brings immediate improvements:

Method / VariantHit@10Hit@20NDCG@10NDCG@20
Arbor + Diag. (Baseline)0.08140.12720.04090.0550
+ Stage-aware cost planning0.08260.12930.04410.0558
+ Active downstream co-adaptation0.08320.13050.04460.0565
+ Both mechanisms0.08480.13350.04610.0583
Full EvoDAGRec0.08690.13890.04780.0609

We would also like to emphasize that, beyond these two cascade-specific mechanisms, our Code DAG evolution framework itself has fundamental search advantages over previous DAG-based methods such as EvoGit and hypothesis-tree methods such as Arbor:

  • Compared with EvoGit: EvoGit relies heavily on human experts to periodically inspect branches and manually prune them. It randomly selects code segments of up to 128 lines for unguided edits without long-term memory and uses random strategies to resolve merge conflicts. In contrast, EvoDAGRec is an end-to-end automated loop. It uses lesson distillation to block previously falsified directions and performs diagnosis-driven, targeted hypothesis edits. During merging, agents resolve semantic conflicts using diffs against the lowest common ancestor (LCA), and the Reviewer checks that the mechanisms from each branch remain functionally active.
  • Compared with hypothesis-tree methods such as Arbor: Arbor follows a “hypothesis-tree exploration + greedy acceptance into a single main branch” pattern. New experiments start from the current best main branch by default. A change is accepted through a standard Git merge only if its offline metrics immediately exceed those of the main branch; code conflicts cause the merge to fail and stop. It cannot directly combine code from multiple historical branches as joint parents. In contrast, EvoDAGRec retains historical code snapshots as states that can be further expanded and merged, without requiring a branch to first become the main-branch champion. It also defines an explicit multi-parent semantic code-merging operator, MERGE. Thus, complementary branches whose individual scores do not exceed their parents are not greedily discarded. Instead, they can be directly recombined across branches to achieve further gains.

We further compare Arbor + Diag. and EvoDAGRec with a fixed tokenizer and evolution restricted to the architecture used in pre-training. As shown below, even with exactly the same diagnostic feedback and evolution limited to a single stage, EvoDAGRec still shows a clear advantage:

Method (Fixed Tokenizer, Pre-training-Only Evolution)Hit@10Hit@20NDCG@10NDCG@20
Arbor + Diag.0.07750.11930.03940.0499
EvoDAGRec (Ours)0.08240.12110.04360.0533

These discussions and experiments have greatly improved the rigor and depth of our paper. We will include them in full in the main text of the revised version. Thank you again for your valuable suggestions!

The evaluation is narrow.

Thank you for your suggestion. It is important to test evolution on more datasets, examine its randomness, and try more LLMs.

We first changed the random seed and then upgraded the coding LLM to GLM-5.3. The results are shown below.

Changing the random seed has little effect on the evolution results. This is because, in our environment, the mathematical feedback is computed deterministically, while the simulated user diagnosis uses fixed user profiles and an LLM temperature of 0. These low-randomness signals expose consistent major failure modes to the agents, such as the initial coarse-level collapse and diluted recent intent, leading to similar evolution paths.

Upgrading the LLM further improves performance. With stronger reasoning and code synthesis capabilities, GLM-5.3 tends to implement a complete mechanism in one step after reading a diagnostic report. For example, it directly injects temporal bias into attention across all layers, or combines RMS normalization with a low weight coefficient, pw = 0.1, when repairing PSRQ. This avoids step-by-step trial and error and allows more attempts within the limited experimental budget. We will add these results to the main text.

Configuration (Amazon Office)Total RoundsHit@10Hit@20NDCG@10NDCG@20
EvoDAGRec (GLM-5.2, Seed 42, Original Paper)390.08690.13890.04780.0609
EvoDAGRec (GLM-5.2, Seed 2026)390.08640.13820.04750.0605
EvoDAGRec (GLM-5.2, Seed 2027)380.08710.13920.04800.0611
EvoDAGRec (GLM-5.2, Mean ± Std)38.7 ± 0.60.0868 ± 0.00040.1388 ± 0.00050.0478 ± 0.00030.0608 ± 0.0003
EvoDAGRec (Upgraded to GLM-5.3, Seed 42)430.08850.14130.04890.0622

We also ran evolution on the Yelp dataset. As shown below, our evolution framework is effective on different datasets. Given the limited time available for rebuttal, we ask for your understanding and will test more datasets in future work.

Method (Yelp Dataset)Hit@10Hit@20NDCG@10NDCG@20
TIGER (Pre-train)0.03930.05690.02090.0253
APAO0.04260.06170.02250.0273
EvoDAGRec (Ours)0.05210.07760.02930.0357

LLM token consumption should also be included in the evolution budget.

Thank you for your suggestion! We did not initially include LLM token costs because we use the centrally deployed open-source model GLM-5.2. Token capacity is abundant, whereas the GPU-hours available to recommendation engineers for experiments are limited. Your suggestion is valuable, however. We tried accounting for LLM token costs during evolution and observed a clear change in the agents' operator choices: less frequent MERGE operations, fewer aggressive EXPLORE operations, and a preference for low-token-cost MUTATE operations.

Among the three search operators, MERGE has the highest token cost—more than 2.5 times that of a typical MUTATE operation. It needs to serialize context diffs and historical lessons from multiple parent branches, and the Coder–Reviewer loop often requires several rounds to resolve cross-branch code conflicts and interface dependencies. Similarly, large EXPLORE edits can trigger multiple compilation and repair rounds.

When aware of token costs, the Operator Selector postpones and reduces tentative merges of intermediate states. For example, it combines changes in a single merge after the branches have matured, rather than performing two successive merges. It also allocates a larger share of operations to local MUTATE edits, which have a high first-pass success rate and very low token consumption.

SettingTotal RoundsOperators: Explore / Mutate / MergeCandidate Stages: Tokenizer / Pre-training / Post-trainingHit@10Hit@20NDCG@10NDCG@20
Default3921 / 14 / 46 / 27 / 60.08690.13890.04780.0609
With token consumption considered3517 / 15 / 35 / 25 / 50.08560.13680.04680.0596

These results show that the cost definition affects the search strategy. For deployments using expensive commercial LLM APIs, such as GPT models, token costs should be included in cost planning.

Adding a retrieval channel online does not by itself isolate the benefit of the evolutionary method from the benefit of adding a channel.

Thank you for your suggestion. We agree that this is the most appropriate control. In fact, we have already conducted two experiments against the same baseline—the online system without an additional retrieval channel:

Added Retrieval ChannelExperiment DurationCTR ChangeClick PV ChangeChannel Unique Exposure ShareChannel CTR
Model before evolution18 days+0.096% (p = 0.128)+0.048% (p = 0.509)15.42%12.41%
Model after evolution20 days+0.331% (p = 1.6 × 10⁻⁸)+0.337% (p = 3.3 × 10⁻⁷)24.10%13.21%

Unique exposure share = exposure PV originating exclusively from this retrieval channel / total exposure PV. Channel CTR is the click-through rate of exposures from this retrieval channel. Click PV refers to the number of clicks.

The results show that adding the model before evolution as a retrieval channel does not produce a significant improvement. Adding the evolved model, however, significantly improves both CTR and click PV. The evolved channel also has a higher unique exposure share (24.10% vs. 15.42%) and a higher channel CTR (13.21% vs. 12.41%).

We should clarify that the two experiments were conducted in different time periods, rather than concurrently. The paper reports the first 14 days of the experiment with the evolved model (CTR +0.40%, p = 0.019). We subsequently extended this experiment to 20 days. The improvement settled at +0.331%, while statistical significance increased further (p = 1.6 × 10⁻⁸).

Dear Reviewer w2zL,

Thank you again for your detailed review. As the discussion period is closing, we would like to confirm whether we have addressed your concerns. We would be happy to clarify anything further before the discussion closes.

Reviewer_KmTrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EvoDAGRec, a multi-agent code evolution framework for generative recommendation. It organizes code evolution through a DAG structure and introduces stage-aware cost planning and active downstream co-adaptation to handle the challenges of multi-stage GR pipelines. Experiments on Amazon Office and an industrial recommendation platform report improvements over existing evolutionary frameworks and recommendation baselines.

Critical strengths
  1. The paper addresses a practical problem in automated evolution of multi-stage generative recommendation pipelines, particularly the asymmetric training costs and dependencies between stages.
  2. The experimental evaluation includes comparisons with several existing evolutionary frameworks, ablation studies, and real-world industrial validation, including an online A/B test.
Critical weaknesses
  1. (major) The methodological novelty is limited. The proposed framework mainly combines existing techniques such as DAG-based evolution and recommendation-specific diagnostics. The two main contributions, stage-aware cost planning and active downstream co-adaptation, are largely heuristic and rely on LLM-driven decisions without sufficient theoretical justification.
  2. Although the paper emphasizes cost-aware evolution, its budget only accounts for GPU-hours while excluding LLM inference costs (Appendix A.1). However, token consumption incurs either monetary costs or additional inference compute, and therefore should also be taken into account in b(t)b^{(t)}.
Questions for authors

see weakness.

4 comments

We appreciate your comments and feedback. We address your concerns below.

Limited methodological novelty; the two main mechanisms are LLM-driven heuristics without sufficient theoretical justification.

Thank you for raising this concern. We would like to emphasize that almost all previous LLM-agent-based code evolution methods are limited to single-stage, end-to-end systems. No prior work has explored how agents can evolve a tightly coupled, multi-stage cascaded system such as generative recommendation (GR). We want to share with the community the key benefits of these two mechanisms for evolving cascaded systems. In the future, we will also extend them to compute-sensitive cascaded tasks such as multi-stage LLM training.

As shown below, adding these two mechanisms to Arbor, a previous state-of-the-art hypothesis-tree evolution framework, also brings immediate improvements:

Method / VariantHit@10Hit@20NDCG@10NDCG@20
Arbor + Diag. (Baseline)0.08140.12720.04090.0550
+ Stage-aware cost planning0.08260.12930.04410.0558
+ Active downstream co-adaptation0.08320.13050.04460.0565
+ Both mechanisms0.08480.13350.04610.0583
Full EvoDAGRec0.08690.13890.04780.0609

These results demonstrate that the two mechanisms are plug-and-play. Indeed, they are LLM-guided heuristics rather than theoretically derived policies, but we still want to share their importance with the community. Your suggestion is valuable. In future work, we will use a Bayesian optimization framework to formally explain the effects of our mechanisms and provide further theoretical guarantees for our method.

We would also like to emphasize that, beyond these two cascade-specific mechanisms, our Code DAG evolution framework itself has fundamental search advantages over previous DAG-based methods such as EvoGit and hypothesis-tree methods such as Arbor:

  • Compared with EvoGit: EvoGit relies heavily on human experts to periodically inspect branches and manually prune them. It randomly selects code segments of up to 128 lines for unguided edits without long-term memory and uses random strategies to resolve merge conflicts. In contrast, EvoDAGRec is an end-to-end automated loop. It uses lesson distillation to block previously falsified directions and performs diagnosis-driven, targeted hypothesis edits. During merging, agents resolve semantic conflicts using diffs against the lowest common ancestor (LCA), and the Reviewer checks that the mechanisms from each branch remain functionally active.
  • Compared with hypothesis-tree methods such as Arbor: Arbor follows a “hypothesis-tree exploration + greedy acceptance into a single main branch” pattern. New experiments start from the current best main branch by default. A change is accepted through a standard Git merge only if its offline metrics immediately exceed those of the main branch; code conflicts cause the merge to fail and stop. It cannot directly combine code from multiple historical branches as joint parents. In contrast, EvoDAGRec retains historical code snapshots as states that can be further expanded and merged, without requiring a branch to first become the main-branch champion. It also defines an explicit multi-parent semantic code-merging operator, MERGE. Thus, complementary branches whose individual scores do not exceed their parents are not greedily discarded. Instead, they can be directly recombined across branches to achieve further gains.

We further compare Arbor + Diag. and EvoDAGRec with a fixed tokenizer and evolution restricted to the architecture used in pre-training. As shown below, even with exactly the same diagnostic feedback and evolution limited to a single stage, EvoDAGRec still shows a clear advantage:

Method (Fixed Tokenizer, Pre-training-Only Evolution)Hit@10Hit@20NDCG@10NDCG@20
Arbor + Diag.0.07750.11930.03940.0499
EvoDAGRec (Ours)0.08240.12110.04360.0533

These discussions and experiments have greatly improved the rigor and depth of our paper. We will include them in full in the main text of the revised version. Thank you again for your valuable suggestions!

LLM token consumption should also be included in the evolution budget.

Thank you for your suggestion! We did not initially include LLM token costs because we use the centrally deployed open-source model GLM-5.2. Token capacity is abundant, whereas the GPU-hours available to recommendation engineers for experiments are limited. Your suggestion is valuable, however. We tried accounting for LLM token costs during evolution and observed a clear change in the agents' operator choices: less frequent MERGE operations, fewer aggressive EXPLORE operations, and a preference for low-token-cost MUTATE operations.

Among the three search operators, MERGE has the highest token cost—more than 2.5 times that of a typical MUTATE operation. It needs to serialize context diffs and historical lessons from multiple parent branches, and the Coder–Reviewer loop often requires several rounds to resolve cross-branch code conflicts and interface dependencies. Similarly, large EXPLORE edits can trigger multiple compilation and repair rounds.

When aware of token costs, the Operator Selector postpones and reduces tentative merges of intermediate states. For example, it combines changes in a single merge after the branches have matured, rather than performing two successive merges. It also allocates a larger share of operations to local MUTATE edits, which have a high first-pass success rate and very low token consumption.

SettingTotal RoundsOperators: Explore / Mutate / MergeCandidate Stages: Tokenizer / Pre-training / Post-trainingHit@10Hit@20NDCG@10NDCG@20
Default3921 / 14 / 46 / 27 / 60.08690.13890.04780.0609
With token consumption considered3517 / 15 / 35 / 25 / 50.08560.13680.04680.0596

These results show that the cost definition affects the search strategy. For deployments using expensive commercial LLM APIs, such as GPT models, token costs should be included in cost planning.

We appreciate the authors' responses and additional experiments, which have addressed most of our concerns. However, the first concern is not fully resolved, as Bayesian optimization remains future work and the current method still relies on heuristic strategies. We therefore increase our rating to 3. We believe 2.5 would be a more appropriate score, but since no intermediate rating is available, we assign a score of 3.

Thank you again for your time and effort. Your suggestions have helped us substantially improve the quality of our paper!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.