acceptodds
Under review as a conference paper at ICLR 2027

Demonstration Conditioning Generally Improves Jagged Frontier Model Performance on Digital and Physical Agentic Tasks

Abstract

We measure how much an in-context demonstration improves frontier models, and how much of that improvement comes from the demonstration containing the an- swer. Benchmark averages hide both: they show neither where inside a benchmark an agent is weak nor whether a demonstration taught a procedure or supplied the item’s answer. Holding the prompt fixed, we vary only the demonstration for seven models, including GPT Astra and Claude Opus 5.5, on tasks across web automation, manipulation, assembly, industrial logistics, and self-driving. A demonstration from the same situation raises the score in all five domains and for all seven web models. With the answer removed, the gain survives on driving and manipulation for all three models and disappears on assembly. Across the seven models the gain falls as the baseline rises, narrowing the gap between the best and worst model while the unevenness inside each benchmark remains.

Primary area:
foundation or frontier models, including LLMs
Keywords:
frontier models, in-context learning, multimodal agents, agent evaluation, procedural memory, retrieval-augmented generation, jagged intelligence, vision-language models

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_WD72OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper performs experiments on a number of VLM agent tasks to study the effect of ICL on performance. Specifically, the paper designs experiments to study this compared to other baselines such as having unrelated demos or having things in context that do not give the answer but provide additional domain context. The paper finds there is a narrowing of the gap due to other effects.

Critical strengths

None

Critical weaknesses

The elephant in the room is that this paper seems to be majority, if not fully AI generated. The AI disclosure acknowledges the use of AI to run experiments and write major sections of the paper. And having read it, the quality of the writing, analysis and choice of research problem reflects this. I think this alone should make this paper ineligible for acceptance.

If I were to completely ignore the AI part, the scope of the paper is extremely narrow and I do not believe has enough of a contribution to be accepted as an ICLR paper. The focus is very narrow: just on this set of questions about teasing out the effects of ICL from other effects by prompting different demonstrations into context and comparing numbers. There is no methodology, really, this is just doing really well-known prompting techniques and analyzing the results. This is the kind of experiments that I would expect as a subsection of an ICLR paper that also did something else: introduce a new method, create a new datasets. Even ICLR papers that are just analysis will introduce some interesting new method of analysis or tackle a really unique or unknown phenomenon and do a really good job of this. But there has been so much work at this point looking at in-context learning and prompting in LLMs, VLMs and agents, that even the choice of problem is not very unique.

Questions for authors

None

Reviewer_J34jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies how demonstration conditioning affects frontier models across five multimodal domains. It finds that demonstrations generally improve average performance, but their effects vary across task categories. By recording whether demonstrations contain the target answer and analyzing copying behavior, the paper finds evidence of transfer beyond answer copying in some domains, alongside failures caused by misleading demonstrations.

Critical strengths
  1. The paper addresses an important evaluation problem: improvements from demonstrations can mix procedural transfer with access to task-specific answers. Recording answer availability makes these gains easier to interpret.

  2. The study covers multiple models and domains and examines both improvements and regressions. This provides a useful picture of when demonstration conditioning helps and when it misleads the model.

Critical weaknesses
  1. The main findings have limited significance. The paper shows that demonstrations can help beyond answer copying and that their effects vary across tasks. However, it offers limited novelty into why transfer succeeds in some settings and fails in others, or how these observations can improve demonstration selection or agent reliability.

  2. The smaller effects are difficult to assess reliably. Main comparisons use a single pass, while the repeat-run reference covers only one model with a shorter step budget.

Questions for authors
  1. What actionable conclusions follow from the observed differences in demonstration gains across tasks and domains?
Reviewer_ZVg3OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper explores whether demo improve performance across web and physical tasks. It compares different types of demos and examines whether improvements come from ICL.

Critical strengths

The paper evaluates several frontier models. It also examines both successful and harmful demos, which gives a more complete picture.

Critical weaknesses
  1. The writing feels heavily AI-generated and needs more careful editing. The paper spends too much text describing numbers without making clear claim each number shows. Eg in Table 1 presents different transformations of the same underlying outcomes: pp gain, net tasks changed, and “gap closed”. This makes the main claim hard to follow. The authors should clarify whether AI helped interpret the results and how those interpretations were checked (they exclude results verification and analysis drafting in AI-use statement). Undisclosed AI interpretation would conflict with ICLR 2027 policy.
  2. The results do not fully support the broad conclusions, and the contribution is limited. On the web domain, demos of the original task give gain of 9.3 pp, while demos from other tasks give 0.9. In driving, demo with 5 of 6 answers matching is still counted as “answer-absent". Copying could therefore explain the pp improvement. Overall, this paper reports too many score changes but offers limited new insight when demos help models solve new tasks.
Questions for authors
  1. Would the web gain remain if the retriever excluded the original task’s demonstration?
Reviewer_1AVbOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an empirical study of demonstration conditioning across multiple models and different multimodal agentic benchmarks. The authors compare performance with and without demonstrations from the same, related, or unrelated situations, and analyze whether improvements arise from procedural transfer or answer copying.

Critical strengths

Broad empirical evaluation with useful diagnostic analysis: The paper evaluates multiple frontier models across five diverse domains and distinguishes improvements from answer-present versus answer-absent demonstrations. This analysis provides useful evidence about the extent to which demonstration conditioning supports procedural transfer rather than simply supplying answers.

Critical weaknesses

Insufficient new insights:

The main findings are largely consistent with existing research on in-context demonstrations, procedural memory, and agent reliability. It does not establish a sufficiently novel or generalizable understanding of demonstration conditioning. In particular, the analysis primarily reports empirical trends without adequately explaining why demonstrations help certain tasks or models but fail on others, or identifying principles that could guide more effective demonstration selection.

Limited evidence supporting generalizable procedural transfer:

A substantial portion of the reported improvements comes from demonstrations containing the test task's answer. Meaningful answer-absent improvements are observed in only two of the four physical domains (which are evaluated through offline symbolic predictions rather than interactive execution). These results raise concerns about how much of the improvement reflects transferable procedural knowledge and how well the conclusions generalize to unseen agentic tasks.

Questions for authors

What are the key new insights beyond prior studies on in-context demonstrations, procedural memory, and agent reliability? Can the authors provide a more systematic explanation of what determines whether a demonstration improves or degrades performance, rather than primarily reporting the observed variations?

Why does answer-absent demonstration conditioning improve performance on Bench2Drive-VL and VLABench but not on Assembly101 or IndEgo? Can the authors provide evidence identifying what procedural information is actually transferred and which task characteristics determine its usefulness?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.