- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LLM agents are used to generate advertising content, which are then shown in production and accumulate various kinds of feedback that might be used as a signal to improve the system. Although a data-rich environment, this is a difficult problem as it requires credit assignment over time and materials. To address this challenge, the authors of this paper propose TIDE. TIDE is a training-free method for managing an external memory store that will be given in-context to the generator. TIDE relies on an evolutionary prompting pipeline to manage a capacity-constrained population of experiences. In real-world deployment A/B tests, TIDE improves over baselines in click-through and activation rate.
Critical strengths
I think that this problem setting is very interesting, and understudied relative to the more static and structured environments of many continual learning and memory papers. I really like the aspect of the formulation where “the value of a memory update is determined by subsequent tasks rather than by the example that triggered the update”; this represents an extremely important and tough to crack class of problems.
Critical weaknesses
TIDE is heavily engineering-based and heuristic. The methods section mixes business rules and machine learning methods, and it is unclear what general principles can be extracted and applied to other tasks of interest to the community. No motivation is given for the particular solution; for example, why the method is training-free when there is plentiful historical data, or why an evolutionary approach might be appropriate. This issue extends to the MEG metric. The experiments on MemoryCD seem like they could help demonstrate the generality of the system, but that benchmark is never described or illustrated so that doesn’t happen.
I also wonder whether the results in Table 1 are significant and/or reliable. They seem to rely on an unvalidated LLM judge.
Questions for authors
- How does this solution map onto problems outside of this domain?
- Why is training-free best, when it seems like you could collect data here and fine-tune some of these parts?
- Why is genetic approach the right/principled approach to take here?
7 comments
W2 / Q2: Why is it training-free?
W2 / Q2: Why is it training-free?
Thank you for your interest in the efficiency of our method. We believe this choice depends on the LLM "brain" behind the agent: since our service utilizes LLMs larger than 27B, a training-free approach is a highly efficient choice. We have also discussed and explored in the main text whether training-free methods perform worse than training-based ones:
First, theoretically, our investigation in the Introduction, Appendix, and Related Work sections shows that training-free methods possess immense potential for agent evolution.
Second, in empirical comparisons, Section 4.2 contrasts TIDE with sample-score-based fine-tuning methods such as BERT and Distilled Recommenders. Although BERT scores higher on certain pre-release prediction metrics, its advantages fail to generalize to future tasks, showing a tendency to overfit historical score targets. In contrast, TIDE achieves the highest Future MEG (+0.180) and win rate (60.7%), along with the lowest negative transfer rate (25.0%). This demonstrates its robustness in converting feedback into memory updates that are effective for future generation.
Third, TIDE can also be combined with training. Table 1 shows that combining TIDE with parameter backpropagation yields better results than backpropagation alone. We further compared the effects of introducing TIDE memories during the training phase versus the inference phase: inference-stage injection performs better on fundamental metrics like compliance and formatting, likely because raw sample signals have a stronger influence on these metrics than memory. Training-stage injection, however, excels in content efficiency and diversity. We believe TIDE memories indirectly enrich the prompt information in training samples, making the model's output more aligned with actual production needs rather than simply memorizing training samples.
| Method | Compliance ↑ | Format ↑ | Efficiency ↑ | Diversity ↑ |
|---|---|---|---|---|
| Training | 55.0 | 67.5 | 40.0 | 65.0 |
| Inference | 70.0 | 75.0 | 37.5 | 52.5 |
Table: TIDE combined with backpropagation; experimental settings follow Table 1 in the main text.
In summary, for models with large enough parameter scales to understand memory, a training-free approach is the most cost-effective choice. For smaller models, we suggest synergistically integrating TIDE memories into the training and inference pipeline.
W3 / Q3: Why Choose an Evolutionary Design?
W3 / Q3: Why Choose an Evolutionary Design?
We appreciate the reviewer raising this pivotal question. Our choice of an evolutionary design is primarily dictated by the structural characteristics of the problem itself.
First, we optimize a set of discrete, interpretable natural language memories rather than continuous, differentiable model parameters. Feedback in recommendation tasks is often delayed, context-dependent, multi-objective, and non-differentiable; consequently, there is no "target memory" that can be derived directly via gradients. More importantly, as you noted, the value of a memory update can only be validated by its performance on subsequent tasks, rather than the current sample that triggered the update. Therefore, directly overwriting memory based on a single feedback instance risks erroneous updates, while simple appending leads to uncontrolled memory bloat. Maintaining multiple candidates supported by different evidence and selecting them based on future feedback within a fixed capacity is a more robust approach.
Second, preliminary evidence is provided in Appendix Table 4. After ten rounds of feedback, TIDE achieved the highest Mean Utility (0.809) and Joint Pass (0.617) while consuming only 2,121 memory tokens—fewer than Full History, Append-only, and Random Replacement. Notably, 9 out of 10 "EVICT" operations in TIDE correctly targeted injected noisy memories. This demonstrates that TIDE’s performance gains stem from selective evolution and denoising rather than the mere accumulation of historical information.
Third, inspired by your question, we further investigated other memory evolution methods: ReasoningBank [1] (extracting reusable strategies from successes and failures), KARMA [2] (generating local atomic modifications and using a rollback mechanism), and EvolveR [3] (distilling interactions into general principles). We have added a comparative experiment against these methods, which we will report in the Appendix.
| Method | Mean Utility ↑ | MEG ↑ | Joint Pass ↑ | Negative Transfer ↓ |
|---|---|---|---|---|
| Frozen Memory | 0.8353 | 0.0000 | 13.3% | 0.0% |
| ReasoningBank-style [1] | 0.8522 | +0.0169 | 13.3% | 36.7% |
| KARMA-style [2] | 0.8572 | +0.0219 | 3.3% | 33.3% |
| EvolveR-Memory-style [3] | 0.8222 | -0.0131 | 16.7% | 50.0% |
| TIDE | 0.8600 | +0.0247 | 16.7% | 33.3% |
Table: Comparison of memory evolution methods. We conducted a targeted experiment across 30 tasks with three random seeds (13, 42, 97) over 50 evolution rounds. While each method managed its own memory context, all shared the same generator, task inputs, and output requirements. We evaluated results using a frozen DeepSeek evaluator. Mean Utility averages Compliance, Format, and Efficiency, while Joint Pass also incorporates Diversity. TIDE achieved the highest Mean Utility (0.8600) and MEG (+0.0247), while matching the best results in Joint Pass and Negative Transfer, further justifying our evolutionary design.
References
-
Ouyang et al. ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memory. ICLR 2026.
-
Miao et al. KARMA: Training-Free Knowledge-Atomic Harness Evolution for Industrial Advertising Agents. 2026.
-
Wu et al. EvolveR: Self-Evolving LLM Agents through an Experience-Driven Lifecycle. ICML 2026.
Generalizability concerns
We sincerely thank the reviewers for recognizing that our research is innovative, interesting, and addresses a current research gap. To the best of our knowledge, TIDE is among the earliest to systematically study the self-evolution of content-generation agents driven by delayed and context-dependent recommendation feedback.
We note the reviewer’s focus on generalizability. Like the reviewer, we are also committed to providing new insights into agent self-evolution through our research. Below are our responses to the issues regarding generalizability:
W1 / Q1: Generalizability: How can our approach be utilized by others?
1. From the Perspective of Application Scenarios
Our framework can be directly applied to recommendation and similar scenarios. For other tasks, the framework is structured into two layers:
In summary, as long as a task's feedback cannot be instantly attributed and summarized into memory like a simple math problem, our logic is applicable. As shown in Figure 1, the CREATE, SERVE, and LEARN components form a closed loop, corresponding to LLM output, feedback acquisition, and memory updates. To put it simply: as long as an LLM can generate output, receive feedback, and continue to optimize based on it, our framework can be utilized.
2. From the Perspective of Application Methods
As explained in W2, the memories evolved via TIDE can be used not only directly during the inference stage but also to help construct training data. Experimental results show that incorporating these memories during the training phase improves the efficiency and diversity of the generated content. We believe this is because evolved memories enrich prompt templates and experience expressions in the training data, reducing the model's reliance on fixed sample patterns and thereby enhancing training diversity and robustness.
W4: Clarification on Generalization Experiments
We have actually introduced the MemoryCD benchmark in the main text (as shown below), and we detailed the specific modifications in the "Public delayed-label benchmark" section of the Appendix.
We appreciate your reminder that the citations in the experimental section were not prominent enough. We will add the corresponding references accordingly 🔧🔨.