- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper revisits the effectiveness of LLM-based context engineering (LCE), which aims to improve model performance by automatically optimizing prompts or extracting reusable experiences without updating model parameters. The authors conduct a systematic evaluation of representative LCE methods.
Critical strengths
- The paper investigates an important question
- Easy to follow.
- comprehensive experimental results.
Critical weaknesses
The primary conclusions are largely intuitive and consistent with existing knowledge. For example, different context engineering methods performing inconsistently across tasks, larger models not necessarily producing better optimized contexts, and diverse failures being harder to summarize into reusable experiences are all reasonably expected observations.
Questions for authors
please see weakness
2 comments
Kind follow-up before discussion window closes
Dear Reviewer,
Many thanks again for your time and careful review of our paper. Please let us know if there are any remaining concerns and questions that require further details. We greatly appreciate the chance to further discuss our work with you before the discussion window closes.
Author Response [1/1]
Dear Reviewer,
Thank you for your time and valuable feedback. Below, we address your concerns about the intuitiveness and their relationship to existing knowledge.
Intuitive conclusions: We agree that some conclusions are intuitive, but humbly argue that not all conclusions are necessarily intuitive, to our best knowledge.
(1) Benchmark of LCEs. We agree that different LCE methods may intuitively exhibit varying performance across tasks, but their effectiveness has not been systematically compared under a unified evaluation. Therefore, we designed a unified benchmark, revealing that no LCE method consistently outperforms the naive CoT baseline across tasks, and some methods can even degrade performance. These findings provide a more comprehensive and balanced view of LCE effectiveness than previous studies, showing that most LCE methods are surprisingly much less effective in some types of tasks.
(2) Scaling experiments. We argue that the failure of scaling laws with LCE is not intuitive. Previous studies have extensively demonstrated scaling laws across various research fields [1][2]. In contrast, our study shows, in a rigorous way, that increasing model size or using newer model generations does not consistently improve LCE effectiveness.
(3) Analysis of failure diversity. Though it is intuitive that diverse failures are harder to summarize into reusable experiences, our work is the first to provide empirical evidence. Our work also isolated other factors like model sizes and shed light on future research directions toward improving LCES.
Relation to existing knowledge: To the best of our knowledge, we are the first work to systematically benchmark different LCE methods under a unified evaluation framework. Thus, we gained novel insights into the underlying causes for their inconsistent effectiveness.
(1) Benchmark of LCEs. As summarized in Table 4, we have surveyed representative LCE methods (TextGrad, GEPA, TF-GRPO, and ACE) and found that they used different evaluation protocols, e.g., ACE reports improvements with DeepSeek-V3.1-671B on agentic and financial tasks, whereas GEPA reports improvements with Qwen3-8B and GPT4.1-mini on math and multi-hop reasoning, making direct comparisons difficult. In contrast, our work provides a unified benchmark and reveals that no method can consistently outperform the naive CoT baseline and can even degrade performance.
(2) Scaling experiments. We conduct three scaling experiments, including the relationship between reflector and generator, parameter scaling, and generation scaling. Closely related to our finding, Zhang et al. [3] investigate whether contexts optimized on smaller models can also benefit larger models. Our scaling experiments differ from theirs in comparing LCE methods instead of the context itself. We tested larger models for both optimizing context (which is not in Zhang et al.’s work) and inference. Through the new experiments, we void the scaling assumption for the LCE methods.
(3) Analysis of failure diversity. One related work is Ashury-Tahan et al. [4], who constructed a taxonomy of common failure modes in reasoning trajectories across benchmarks and models. Distinct from their work, our work focuses on context engineering methods and reveals the relationship between failure-mode diversity and LCE effectiveness for the first time. Plus, we find that the failure-mode diversity mainly comes from the datasets, and our further ablation studies identify limited extraction capability as a key bottleneck in turning diverse failures into broadly useful context. These insights provide essential new knowledge to improve LCE methods in the future.
[1] Scaling Laws for Neural Language Models, Kaplan J, McCandlish S, Henighan T, et al., arXiv 2001.08361.
[2] Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models, Wang X, Tan S, Xu S, et al., ICML2026
[3] SPRIG: Improving Large Language Model Performance by System Prompt Optimization, Zhang L, Ergen T, Logeswaran L, et al., ICLR2026
[4] ErrorMap and ErrorAtlas: Charting the failure landscape of large language models, Ashury-Tahan S, Mai Y, Bandel E, et al., arXiv:2601.15812.