acceptodds
Under review as a conference paper at ICLR 2027

Revisiting LLMs as Context Engineers

Abstract

As large language models (LLMs) are used in more and more new tasks, context engineering becomes a fundamental method to efficiently steer LLMs by constructing instruction prompts or metadata without intensive training. A growing number of LLM context engineering (LCE) approaches automate the task by casting the LLM itself as the engineer that draws generalizable instructions or experiences from offline trials as context. Although these methods resemble how humans learn from experiences, it remains unclear LLMs can engineer effective context from data. To answer the question, we first establish a benchmark composed of datasets with different reasoning patterns. We compared different LCE methods with different LLM families in 8B size, showing that there is no single winner for all tasks, even compared with simple baselines like chain-of-thought. Even when scaling model capabilities via more parameters or generation iterations, LCEs do not consistently boost existing performance. Across all datasets and models, the performance gains of LCEs are negatively correlated () with the diversity of the failure modes of reasoning trajectories. Investigating how LCEs work, we find that (1) the improvement in instruction adherence can explain some success of LCEs; (2) one major bottleneck is whether LLMs can extract effective experiences out of natively diverse failures. Through this study, we uncover the limited capabilities of LCEs in abstracting principled contexts from diverse failure modes and understand how LCEs work, providing guidelines for future work.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Prompt Optimization, Context Engineering, LLMs

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_uBVeOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits the effectiveness of LLM-based context engineering (LCE), which aims to improve model performance by automatically optimizing prompts or extracting reusable experiences without updating model parameters. The authors conduct a systematic evaluation of representative LCE methods.

Critical strengths
  1. The paper investigates an important question
  2. Easy to follow.
  3. comprehensive experimental results.
Critical weaknesses

The primary conclusions are largely intuitive and consistent with existing knowledge. For example, different context engineering methods performing inconsistently across tasks, larger models not necessarily producing better optimized contexts, and diverse failures being harder to summarize into reusable experiences are all reasonably expected observations.

Questions for authors

please see weakness

2 comments
AuthorsOpenReview ↗

Author Response [1/1]

Dear Reviewer,

Thank you for your time and valuable feedback. Below, we address your concerns about the intuitiveness and their relationship to existing knowledge.

[w1] The primary conclusions are largely intuitive and consistent with existing knowledge.

Intuitive conclusions: We agree that some conclusions are intuitive, but humbly argue that not all conclusions are necessarily intuitive, to our best knowledge.

(1) Benchmark of LCEs. We agree that different LCE methods may intuitively exhibit varying performance across tasks, but their effectiveness has not been systematically compared under a unified evaluation. Therefore, we designed a unified benchmark, revealing that no LCE method consistently outperforms the naive CoT baseline across tasks, and some methods can even degrade performance. These findings provide a more comprehensive and balanced view of LCE effectiveness than previous studies, showing that most LCE methods are surprisingly much less effective in some types of tasks.

(2) Scaling experiments. We argue that the failure of scaling laws with LCE is not intuitive. Previous studies have extensively demonstrated scaling laws across various research fields [1][2]. In contrast, our study shows, in a rigorous way, that increasing model size or using newer model generations does not consistently improve LCE effectiveness.

(3) Analysis of failure diversity. Though it is intuitive that diverse failures are harder to summarize into reusable experiences, our work is the first to provide empirical evidence. Our work also isolated other factors like model sizes and shed light on future research directions toward improving LCES.

Relation to existing knowledge: To the best of our knowledge, we are the first work to systematically benchmark different LCE methods under a unified evaluation framework. Thus, we gained novel insights into the underlying causes for their inconsistent effectiveness.

(1) Benchmark of LCEs. As summarized in Table 4, we have surveyed representative LCE methods (TextGrad, GEPA, TF-GRPO, and ACE) and found that they used different evaluation protocols, e.g., ACE reports improvements with DeepSeek-V3.1-671B on agentic and financial tasks, whereas GEPA reports improvements with Qwen3-8B and GPT4.1-mini on math and multi-hop reasoning, making direct comparisons difficult. In contrast, our work provides a unified benchmark and reveals that no method can consistently outperform the naive CoT baseline and can even degrade performance.

(2) Scaling experiments. We conduct three scaling experiments, including the relationship between reflector and generator, parameter scaling, and generation scaling. Closely related to our finding, Zhang et al. [3] investigate whether contexts optimized on smaller models can also benefit larger models. Our scaling experiments differ from theirs in comparing LCE methods instead of the context itself. We tested larger models for both optimizing context (which is not in Zhang et al.’s work) and inference. Through the new experiments, we void the scaling assumption for the LCE methods.

(3) Analysis of failure diversity. One related work is Ashury-Tahan et al. [4], who constructed a taxonomy of common failure modes in reasoning trajectories across benchmarks and models. Distinct from their work, our work focuses on context engineering methods and reveals the relationship between failure-mode diversity and LCE effectiveness for the first time. Plus, we find that the failure-mode diversity mainly comes from the datasets, and our further ablation studies identify limited extraction capability as a key bottleneck in turning diverse failures into broadly useful context. These insights provide essential new knowledge to improve LCE methods in the future.

[1] Scaling Laws for Neural Language Models, Kaplan J, McCandlish S, Henighan T, et al., arXiv 2001.08361.

[2] Finding the Minimal Parameter Budget for Implicit Reasoning: A Data Complexity Driven Scaling Law for Language Models, Wang X, Tan S, Xu S, et al., ICML2026

[3] SPRIG: Improving Large Language Model Performance by System Prompt Optimization, Zhang L, Ergen T, Logeswaran L, et al., ICLR2026

[4] ErrorMap and ErrorAtlas: Charting the failure landscape of large language models, Ashury-Tahan S, Mai Y, Bandel E, et al., arXiv:2601.15812.

AuthorsOpenReview ↗

Kind follow-up before discussion window closes

Dear Reviewer,

Many thanks again for your time and careful review of our paper. Please let us know if there are any remaining concerns and questions that require further details. We greatly appreciate the chance to further discuss our work with you before the discussion window closes.

Reviewer_vEUoOpenReview ↗
Rating
4
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Paper compares four methods for LLM as Context Engineer (LCE) along with two control methods using a unified benchmark composed by joining datasets for math reasoning, multi-Hop reasoning, instruction following, retrieval, and agentic tasks. Paper identifies that existing evaluations for the LCE method are not unified and cannot be used to draw conclusions about the general effectiveness of LCE (Table 4). Main contributions of the paper are:

  1. Defines unified benchmark for LCE
  2. No combination of LCE method and reflector model outperforms trivial CoT prompting in testing across all datasets.
  3. Scaling (bigger or newer model) the task solver/generator model improves baseline performance on tasks but scaling the reflector does not result in clear trend in gain from LCE.
  4. Uses LLM review of failed generations and Vendi score to measure diversity of failure modes. Analysis supports the hypotheses that LCE works better when the baseline generation has less diverse failure modes.
Critical strengths

This paper defines the first benchmark for LCE that incorporates a variety of tasks. Moreover it identifies that current state of the art LCE methods don't show consistent gains over trivial CoT baseline across models and tasks and gives evidence for the hypothesis that LCE works better for tasks with less diverse failure modes. This hypothesis suggests clear directions for further investigation and can be used to improve LCE methods.

Paper's careful analysis supports the following hypotheses that can give clear direction for future research:

  1. Diveristy in failure modes comes primarily from the task, not from the model used or the LCE method.
  2. LCE can improve on tasks with high failure mode diversity in the following ways: (a) in HotpotQA and MATH500 LCE added prompting to adhere to task instructions (b) in all cases LCE improved performance only if it extracted useful experiences from rollouts
Critical weaknesses

No weakness.

Questions for authors

Did you consider adding a trivial baseline like "Answer only what the question asks. Avoid extraneous details."? Do you expect this to significantly decrease gains of most LCE methods?

You compared LCE gains to model size. How do you expect LCE gains compare to baseline task accuracy or to (model size, baseline task accuracy)?

Does the literature contain a systematic study of ICL context length and demonstration quality. Based on your work in Section F do you have suggest improvements to ICL or GRPO-ICL or further experiments you would like conducted?

2 comments
AuthorsOpenReview ↗

Author Response [1/2]

Thank you for your time and support of our work. We greatly appreciate your insightful comments and suggestions.

To make sure we understand your first question correctly, could you please kindly clarify whether you mean using “Answer only what the question asks. Avoid extraneous details.” as an additional baseline without context engineering, or using it as the initial prompt for LCE methods? This would help us ensure that the additional experiment addresses your suggestion.

AuthorsOpenReview ↗

Author Response [2/2]

Dear Reviewer, Thanks for your supportive and thoughtful review. We have carefully considered your questions and addressed each of them below.

[q1]: Did you consider adding a trivial baseline like "Answer only what the question asks. Avoid extraneous details."? Do you expect this to significantly decrease gains of most LCE methods?

Thank you for your thoughtful comment. We add new experiments using the additional baseline in Tab. 9 of Section H.2. As an additional baseline, we append the prompt, 'Answer only what the question asks. Avoid extraneous details.' to both the initial prompts and each LCE's optimized contexts across three models and four datasets. In the table below, we show that there is no significant difference in LCE performance gains between the settings w/ and w/o this prompt.

ModelDatasetGEPATF-GRPOACE
w/ Promptw/o Promptw/ Promptw/o Promptw/ Promptw/o Prompt
Qwen3MATH500-0.80-0.20+0.80-1.10-5.70-7.30
HotpotQA-0.33-1.54+1.27+0.00-2.07-3.26
IFBench-1.16-0.37-1.16-0.14+1.12+0.64
MiddleMatch+12.93+12.53+6.60+6.47+5.40+4.73
Gemma4MATH500+2.20+3.50+9.00+6.10+1.10+1.60
HotpotQA-0.53+0.73-0.33-0.13-0.47-0.20
IFBench-1.94-1.22+0.54+1.87+1.77+1.13
MiddleMatch+5.27+7.40+3.60+10.80+3.40+0.73
Min3MATH500+0.60-1.30-1.60+0.80-6.90-3.00
HotpotQA-0.27+2.80+0.47+1.07+0.93+3.13
IFBench+2.55+0.82-0.20+0.07-0.03+0.27
MiddleMatch+14.80+22.20+0.33+9.40+5.13-1.07

To test whether this prompt significantly reduces LCE gains, we performed a paired t-test comparing the performance gains with and without this specific prompt over all benchmarks, datasets, and methods. The p-value is larger than 0.05. That means there is no evidence to support that the simplified explanations can impact the effectiveness of LCEs.

[q2]: You compared LCE gains to model size. How do you expect LCE gains compare to baseline task accuracy or to (model size, baseline task accuracy)?

Thanks for pointing it out. We find there is no significant relation between LCE performance gains and either baseline task accuracy alone or (model size, baseline task accuracy). Just as larger models do not consistently result in more LCE gains, improvements in baseline task accuracy brought by model scales do not necessarily contribute to better effectiveness of LCEs.

In the left panel of Fig. 20 in Section H.3, we conduct a Pearson correlation analysis and observe no significant correlation between baseline task accuracy and LCE performance gains (pearson’s r = -0.100, n.s.). Meanwhile, in the right panel of Fig. 20 in Section H.3, we conduct a multiple linear regression analysis with LCE performance gains as the outcome, while model size and baseline task accuracy are independent variables. Then we use the F-test to assess the joint association of (model size, baseline task accuracy) with LCE performance gains and find that this association is not significant (F = 1.175, n.s.).

[q3] A systematic study of ICL context length and demonstration quality. Based on Sec. F do you have suggestions for improving ICL/GRPO-ICL or further experiments?

Thank you for your comment. We have added a more extensive review of ICL in Section F, when our original version focused on LCEs.

The most relevant literature is: Zhang et al. [1] observed that performance improves with a few-shot ICL setup but declines with many-shot ICL, while Hong et al. [2] showed that lengthy contexts can hurt performance. Different from these works, our analysis considers both context length and context quality, which is directly related to the effectiveness of LCE methods. To the best of our knowledge, existing studies have not jointly investigated these two aspects.

Based on our work in Section F, we find that GRPO-ICL uses longer demonstrations than standard ICL, leading to worse performance, while reducing the number of demonstrations can quickly restore performance. Meanwhile, even gold contexts can hurt performance when they are overly long. Therefore, we suggest that reducing redundancy in the context while preserving the most informative items may help improve the effectiveness of in-context examples/experiences and thereby the context engineering methods.

[1] More is not always better? Enhancing many-shot in-context learning with differentiated and reweighting objectives, Zhang X, Lv A, Liu Y, et al., ACL2025

[2] Context Rot: How Increasing Input Tokens Impacts LLM Performance, Hong K, Troynikov A, Huber J, https://www.trychroma.com/research/context-rot

Reviewer_RsaKOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether LLMs can learn useful context from training examples without updating model parameters. It compares several representative LLM context engineering methods, including TextGrad, GEPA, TF-GRPO, and ACE, across multiple tasks and model families. The main finding is that these methods can improve performance in some settings, but no method consistently outperforms simple CoT or ICL baselines across tasks and models. Increasing model size or using newer models also does not reliably solve this problem. The paper further finds that LCE methods tend to work better when the model’s failures are relatively homogeneous. Its case studies suggest that some gains come from improved instruction or output-format adherence rather than better reasoning.

Critical strengths
  1. The paper provides a useful unified evaluation of several LCE methods. Previous methods are often evaluated using different models and benchmarks, making their results difficult to compare. Evaluating them under a shared set of models, datasets, splits, and metrics gives a more realistic picture of their strengths and limitations.

  2. The analysis goes beyond reporting average performance. The connection between failure-mode diversity and LCE performance is interesting, and the case studies show that some improvements come from better format or instruction adherence rather than new reasoning ability. This distinction is useful for understanding what these methods actually learn.

Critical weaknesses
  1. The failure-mode diversity result is based mainly on correlation, and dataset identity appears to explain much of the diversity. It is therefore unclear whether failure diversity itself explains LCE performance, or whether the correlation is driven by other differences between datasets, such as task difficulty and structure. A controlled or within-dataset analysis would make this claim more convincing. Without a within-dataset or mixed-effects analysis controlling for dataset, model, method, and baseline performance, the evidence supports an association but not the stronger claim that failure-mode diversity itself explains LCE effectiveness.

  2. The methods are not fully comparable in terms of prompt structure, context length, and computation budget. The authors reasonably preserve each method’s original design, but this makes it difficult to attribute performance differences specifically to the quality of the optimization method. The results more strongly support conclusions about the end-to-end systems than conclusions about which algorithm is intrinsically better.

Questions for authors
  1. Does the negative relationship between failure-mode diversity and LCE performance remain significant after controlling for dataset identity, model, method, and initial performance? A within-dataset or mixed-effects analysis would help clarify this.

  2. Could the authors report the number of LLM calls or total generated tokens used by each method? This would help readers understand how different optimization budgets affect the comparison.

  3. For the cases attributed to improved format adherence, could the authors re-evaluate outputs after normalizing formatting or extracting the intended answer with a common evaluator? How much of the reported gain remains?

5 comments
AuthorsOpenReview ↗

Author Response [1/2]

Dear Reviewer,

Thanks for your constructive feedback. Below, we respond to each of your comments to clarify our findings and address your questions and concerns.

[w1][q1] Does the negative relationship between failure-mode diversity and LCE performance remain significant after controlling for dataset identity, model, method, and initial performance? A within-dataset or mixed-effects analysis would help clarify this.

Thank you for your insightful comment. We did extra experiments and found that the negative relationship between failure-mode diversity and LCE performance remains significant in a mixed-effects analysis (where many samples can be used), though it is not significant in the within-dataset analysis (due to very limited sample size). The results have been added to Section G in our revised paper. We examined the relation in two ways.

Mixed-effects Analysis: We use a mixed-effects analysis to test our hypothesized negative association between failure-mode diversity and performance gain. Our primary hypothesis is that β\beta (the coefficient for the diversity term) is negative. We therefore formulate the null hypothesis as (H0:β≥0H_0: \beta \geq 0), use a linear mixed-effects model, and report the β\beta and corresponding p-value by the t-test in Tab. 8 of Sec. G. The performance gain is defined as the outcome variable, with failure-mode diversity, dataset choice, and initial performance as fixed effects, and models and LCE methods as random effects.

Through the mixed-effects analysis, we observe that the coefficient for failure-mode diversity is negative (β=−0.793\beta=-0.793, p<0.05). That implies that LCEs work better when facing less failure-mode diversity. In comparison, the initial performance coefficient is small with non-significant correlation (β=−0.038\beta=-0.038, n.s.). Moreover, the coefficients of the four datasets relative to AppWorld are (βHotpotQA=−2.070\beta_{HotpotQA}=-2.070, βIFBench=−5.427\beta_{IFBench}=-5.427, βMATH500=−1.698\beta_{MATH500}=-1.698, and βMiddleMatch=+2.658\beta_{MiddleMatch}=+2.658). None of these coefficients is significant.

Within-dataset Analysis: We group the experiments by dataset and then independently assess the correlation between failure-mode diversity and performance gain within each dataset. For each dataset, we test the combinations of 3 models and 3 LCE methods.

DatasetPearson’s rp-value
MiddleMatch-0.380n.s.
IFBench0.054n.s.
MATH500-0.775p<0.01
HotpotQA-0.276n.s.
AppWorld-0.400n.s.

In the above table, our within-dataset analyses did not reveal a significant negative correlation (n.s.) in most cases, except for the MATH500. However, the non-significant results are largely due to the fact that the failure-mode diversity values within each dataset are concentrated around the mean.

In conclusion, the LCE gains can be mostly explained by the failure-mode diversity. Though the dataset identity and diversity are related (Fig. 6), the dataset identity itself does not explain the LCE gains in the mixed-effect model.

AuthorsOpenReview ↗

Author Response [2/2]

[w2&q2] Could the authors report the number of LLM calls or total generated tokens used by each method? This would help readers understand how different optimization budgets affect the comparison.

Thanks for this detailed comment. We compared the optimization budget of different LCE methods across four tasks using Qwen3, and found there is no significant correlation between performance gains and either token consumption (input + output tokens, Pearson’s r=-0.448, n.s.) or the number of LLM calls (Pearson’s r=-0.395, n.s.) in Fig. 19 of Section H.1.

Individual results are presented in the table below. Here, an LLM call refers to one request sent to the LLM, while the input and output token counts represent the total number of tokens input to and generated by the model, respectively.

DatasetMethodLLM CallsInput (M)Output (M)Performance gains
MiddleMatchGEPA1,887.21.8650.739+12.53
MiddleMatchTF-GRPO2,859.02.8992.525+6.47
MiddleMatchACE2,246.83.1190.748+4.73
IFBenchGEPA3,791.85.5578.326-0.37
IFBenchTF-GRPO4,084.27.3553.419-0.14
IFBenchACE3,921.611.6032.715+0.64
MATH500GEPA2,573.62.3331.864-0.20
MATH500TF-GRPO2,442.61.4462.205-1.10
MATH500ACE2,423.45.7711.081-7.30
HotpotQAGEPA3,200.87.7700.432-1.54
HotpotQATF-GRPO3,520.26.1260.5160.00
HotpotQAACE3,602.815.1150.883-3.26

Different cases in the above table further illustrate this inconsistency. For example, GEPA achieves the largest improvement on MiddleMatch with the smallest token consumption, while ACE on IFBench achieves the largest improvement using the largest token consumption. A potential explanation for these non-significant correlations between optimization budget and LCE performance gains is that additional calls and tokens may be spent evaluating ineffective candidates or repeatedly refining similar instructions, thus more optimization budget is insufficient to improve the context.

[q3]: For the cases attributed to improved format adherence, could the authors re-evaluate outputs after normalizing formatting or extracting the intended answer with a common evaluator? How much of the reported gain remains?

Thanks. We added new experiments in Tab. 10 of Section I, and we found that the gains substantially decrease and almost vanish after applying LLM-Extractor. That implies the importance of format adherence, as we have argued in our paper.

In the table below, we use Qwen3.8-27B to extract the final answers in the failures and re-evaluate the final score for Gemma4 on MATH500. We focus on this Gemma4+MATH500 because Fig. 8 shows that Gemma4’s improvements are attributed to improved format adherence, while LCEs can not improve the performance of the other models. We report the initial and final performance, along with the corresponding performance gains (final - initial), both with and without LLM-Extractor.

MethodInitial (w/)Final (w/)Gain (w/)Initial (w/o)Final (w/o)Gain (w/o)
GEPA91.7091.20-0.5067.7071.20+3.50
TF-GRPO91.1092.60+1.5067.7073.80+6.10
ACE88.8088.00-0.8067.8069.40+1.60
AuthorsOpenReview ↗

Kind follow-up before discussion window closes

Dear Reviewer,

Thank you again for your time and effort in reviewing our paper. If there are any remaining questions that require further clarification, please feel free to let us know. We sincerely appreciate the opportunity to have an in-depth discussion with you before the discussion window closes.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.