- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Natural-Language Policy Gradients (NLPG), a method for improving compound LLM agents while keeping the model weights, program structure and base prompts frozen. Instead of rewriting prompts, NLPG keeps an external "policy memory" of short natural-language rules. Each rule is filed under a route, which is either a program module or a task family. At inference time, the top-K rules for a route are appended to that module's prompt. To build this memory, NLPG diagnoses failed runs with benchmark-specific adapters. It then passes textual feedback backward through the module dependency graph, as TextGrad does, to produce a local correction for each module. Corrections are grouped across runs and scored by how many failures support them and by the critic's confidence. They are turned into rules, and a new memory is accepted only if it improves a held-out set without regressions. The method is evaluated on memory benchmarks (LoCoMo, HaluMem, LongMemEval) against memory-agent baselines. It is also evaluated on the GEPA suite (HotpotQA, IFBench, HoVer) against MIPROv2, TextGrad, Trace and GEPA, using Qwen3-8B and GPT-4.1 Mini. It reports the best results in every comparison, for example 59.00 vs. 51.09 for GEPA with Qwen3-8B.
Critical strengths
- NLPG adds short rules tied to specific modules or task families and never rewrites prompts or code. This makes each change easy to inspect, attribute and roll back. It also avoids prompt drift, a risk with methods that rewrite whole instructions, such as GEPA or Meta-Harness.
- It joins TextGrad-style backward credit assignment, voting across runs and held-out validation into a single procedure. Together these address a known weakness of reflective optimizers: overfitting to fixes that come from a single example.
- NLPG outperforms GEPA, GEPA+Merge, TextGrad and MIPROv2 on all three transfer tasks with both backbones. It also outperforms memory-agent baselines on LoCoMo, HaluMem and LongMemEval. Because routing can be by task family, the same procedure works both for multi-module programs and for memory-QA tasks.
Critical weaknesses
- The backward credit assignment is taken directly from TextGrad, which the authors acknowledge. Module-level, reflection-driven instruction optimization with validation is the core of GEPA. Storing reusable lessons from experience resembles prior experiential-memory work such as ExpeL and Agent Workflow Memory. The main new elements are storing fixes as additive rules and support-weighted aggregation. The paper should state more clearly which part is new and isolate its effect, for example by comparing against GEPA with an add-rules-only constraint, or TextGrad with cross-run aggregation. Recent harness-optimization work such as Meta-Harness [1], which optimizes the full harness code, is neither cited nor discussed.
- Appendix C says that auxiliary models used for reflection and optimization are "reported separately," but I could not find them in the paper. This matters a lot. If a stronger model writes the rules, part of NLPG's gain on Qwen3-8B may come from transferring that model's knowledge. A fair comparison needs the same reflection model as GEPA, or an analysis of how sensitive the results are to this choice.
- The paper matches inference budgets but does not report optimization cost relative to GEPA and TextGrad: rollouts, critic calls, tokens or wall-clock time. A per-module backward pass on every failure, plus held-out validation each round, could cost much more than GEPA.
- The backbones (GPT-4.1 Mini, GPT-4o-mini, Qwen3-8B) are relatively old and small. It is unclear whether rule-based addenda still help stronger current models that may already follow such procedures.
References
[1] Meta-Harness: End-to-End Optimization of Model Harnesses, 2026, Lee et al.
Questions for authors
- Which models serve as the critic and optimizer, and did GEPA and TextGrad use the same reflection model?
- What is the total optimization cost (LLM calls and tokens) of NLPG compared with GEPA?
- How does NLPG perform with a stronger backbone (e.g., GPT-6-class or Qwen3-32B+)?
9 comments
The Response to Weakness3 and Question2
The Response to Weakness3 and Question2
Thank you for highlighting optimization cost beyond inference. We have added the optimization rollout budgets and reflection-call counts for NLPG and GEPA,using qwen3-8b for task execution, criticism, backward reflection, and optimization.:
| Benchmark | NLPG reflection calls | GEPA reflection calls | NLPG rollouts | GEPA rollouts |
|---|---|---|---|---|
| HotpotQA | 42 / 62 | 69 / 64 | 420 | 6,871 |
| IFBench | 21 / 27 | 21 / 17 | 360 | 3,593 |
| HoVer | 86 / 47 | 92 / 50 | 420 | 7,051 |
The following costs exclude preflight and final testing:
| Benchmark | Task rollouts | Total API requests | Reported tokens |
|---|---|---|---|
| IFBench | 360 | 580 | 439,892 |
| HotpotQA | 420 | 1,527 | 1,410,649 |
| HoVer | 420 | 3,203 | 3,673,576 |
Auxiliary stages account for 33.9%, 25.2%, and 46.5% of these token budgets, respectively. Observed offline elapsed times were 18.7, 19.5, and 21.8 minutes, respectively.
NLPG uses approximately 16.4×, 10.0×, and 16.8× fewer optimization rollouts on HotpotQA, IFBench, and HoVer, respectively. These budgets include policy construction and parent/candidate held-out validation; final-test rollouts are excluded.
GEPA’s larger rollout count arises from repeated candidate evaluation during evolutionary search. Each reflective update is first evaluated on a training minibatch, and candidates that improve are subsequently evaluated on the larger selection set. These repeated executions accumulate rollouts even when the number of distinct training examples is small. NLPG instead constructs a candidate policy memory from batch-level feedback and uses a fixed parent–candidate validation schedule, resulting in fewer optimization rollouts in our reported experiments.
Thus, module-local feedback does not translate into a larger rollout budget in these experiments: reflection counts are broadly comparable, while NLPG requires substantially fewer agent executions.
The Response to Weakness4 and Question3
The Response to Weakness4 and Question3
We thank the reviewer for the suggestion. We additionally evaluated NLPG with GPT-6.1 Sol and Qwen3-32B under the same data splits and evaluation settings:
| Backbone | HotpotQA | IFBench | HoVer |
|---|---|---|---|
| GPT-6.1 Sol | 83.00% | 82.00% | 73.33% |
| Qwen3-32B | 77.67% | 42.00% | 58.67% |
Together with the backbone results already reported in the paper, these additional experiments show that NLPG remains effective across different model families and scales. In particular, its strong performance with GPT-6.1 Sol supports its applicability to more capable backbones.
The Response to Weakness2 and Question1
We thank the reviewer for their thoughtful assessment and recognition of NLPG’s interpretable policy updates. and for raising the questions which have helped us improve the paper.
All auxiliary roles run on the same model as the task backbone of their block. For the Qwen3-8B block the critic and the optimizer are Qwen3-8B; for the GPT-4.1 Mini block they are GPT-4.1 Mini. The same weights that answer the task also diagnose the failure and write, score and filter the rules; no external model is introduced at any stage. GEPA and TextGrad use the same reflection model as their task backbone. GEPA instantiates each system once per model with all modules on that model, and its reflection LM is that model. TextGrad's optimizer engine is set to the same model, run in the Trace framework under GEPA's budget, metric and feedback function. We have revised the writing in the Appendix B.2