acceptodds
Under review as a conference paper at ICLR 2027

Learning from Physical Experience:Context-Space Adaptation forRobotic Manipulation

12 views

Abstract

Recent multimodal foundation models exhibit strong perception, reasoning, and in-context learning capabilities, but translating these capabilities into physical action remains challenging. Robotic manipulation demands four critical capabilities: spatial reasoning, dexterous interaction, tool use, and adaptation to execution failures. Through real-world experiments across diverse manipulation tasks, we find that ChatGPT-6-Astra, a state-of-the-art general-purpose reasoner, can serve as a physically actionable, context-responsive manipulation policy, demonstrating effective fine manipulation and spatial reasoning. Building on these findings, we introduce a recursive self-improvement harness for learning from physical experience. A context compiler integrates current observations, system-level guidance, task instructions, and retrieved experience to generate subtask sequences and executable motion plans. Plans, execution results, and task outcomes are accumulated as experience, with optional human feedback providing additional guidance. Reflection on this evidence produces context updates that inform subsequent planning and execution, forming a recursive loop of interaction, experience accumulation, and context adaptation without task-specific parameter updates. Our experiments show that learning from experience enables manipulation adaptation in context space. On challenging tool-use and potato-chip picking tasks where direct reasoning alone yields zero success, successive experience-driven updates support progressive skill consolidation and improved execution. These findings connect general-purpose reasoning with physical skill acquisition and demonstrate how accumulated interaction experience can improve robotic manipulation while keeping model parameters fixed.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robot Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_8joGOpenReview ↗
Rating
1(−1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper presents FreeMani, a context-based adaptation harness that keeps its actor and reflection models fixed while turning robot observations and execution outcomes into reusable guidance. In a bread-transfer demonstration, the robot takes 144 decisions to complete the first transfer and 26 for the second, illustrating within-session reuse of a tool-use strategy.

Critical strengths
  1. The work studies physical robot interaction without updating model weights.
  2. Its interaction traces show how tool discovery, failure recovery, and strategy reuse can unfold in practice.
Critical weaknesses
  1. Most experiments focus on GPT-6 Astra; Opus 5.5 and GPT-5.6 Sol appear only in small preliminary comparisons, and FreeMani lacks a clearly available implementation and controlled evaluation.
  2. The paper gives little detail about its data and relies on a few demonstrations, so the evidence feels thin for its broad claims about learning from experience.
  3. The paper lacks matched post-reset experiments to establish persistent learning or measure a reliable success-rate improvement.
Questions for authors

The second transfer retains the tongs, physical setup, and conversation history, so its lower cost does not isolate learning from those factors?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 8joG

Dear Reviewer 8joG,

Thank you for highlighting how the physical interaction traces make tool discovery, failure recovery, and strategy reuse visible in a frozen-model setting. We address your concerns below.

(W2,W3,Q1) Experiments on Learning from Physical Experience

We thank the reviewer for raising the question regarding the claim of learning from physical experience.

Figure 8 and project page Sec “From physical experience to context space adaptation” illustrate an epistemic action experience across three bread-transfer episodes. Each begins with a fresh actor conversation, empty grippers, and both the bread and the tongs outside the tray. In Episode 1, the robot retracts the tongs at 333 s to see whether the tips have captured the bread. In Episode 2, it uses the free wrist camera at 88 s, and the system subsequently retains this strategy. In Episode 3, the actor request contains the retained guidance to reposition the free wrist camera when the tips are obscured. The robot executes that action at 83 s. This trace shows which guidance carried across the reset and how the actor used it.

As suggested by the reviewer,we expanded the evaluation to 12 real-world tasks. Each trial begins with a scene and conversation reset. Overall task success was 5.6% without memory, 52.8% with memory alone, and 63.9% with the proposed method (Figure 5). On three representative tasks, we also compared the proposed method with a condition that retained planning, interaction history, and immediate feedback but disabled lesson retention. Across these tasks, task success was 33.3% without retained lessons and 77.8% with the proposed method (Figure 6). Figure 6 measures the contribution of retaining lessons across the evaluated trials, including reuse within an episode. Figure 8 traces one instance of reuse across episodes. Appendix B defines the physical success criteria, and Appendix E reports the task-level results.

(W1) Implementation Details

We thank the reviewer for asking about implementation details. Section 3 and Figure 4 describe how LEXP connects planning, experience, and action selection. The planner proposes a method for the current subtask. The experience module uses that proposal, recent physical evidence, and retained lessons to generate feedback, which the actor receives alongside its task instruction, current observation, and interaction memory. Section 3.2 describes how the actor’s context is constructed and how action requests are executed. Appendix D.1 specifies the experience-entry fields and update rules, and Appendix F provides the module prompts and structured output formats.

Sincerely,

The Authors

Reviewer_J2NLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces FreeMani, which keeps the actor, reflection model, and action interface fixed while a context compiler combines observations, task instructions, history, and structured experience into an execution-reflection-context update loop. Evaluation support context responsiveness and local strategy reuse.

Critical strengths

S1. Small-object pickup succeeds in lots of cases. Language-feedback examples also document corresponding changes in direction, arm selection, and action order. These results provide useful preliminary empirical evidence for using a frozen general-purpose reasoner in fine manipulation.

S2. Both framework decomposition evidential limits are clear and explicit.

Critical weaknesses

W1. Bread transfer decreases from 144 to 26 calls, but the second phase starts with the tongs already held, the inspection configuration retained, and different target geometry. Without matched scene-and-conversation reset controls, this gain cannot be attributed to experience learning rather than favorable physical state or continued history.

W2. The evidence linking structured experience to execution is incomplete. The method is mainly formalized at the reflection/update level, without a traceable execution-evidence to experience-entry to actor-request example. The introduction also notes that nominal experience conditions do not guarantee delivery to the actor. This prevents verification and reproduction of the proposed mechanism’s actual contribution.

W3. The incremental contribution lacks targeted comparisons. Section 5 acknowledges that memory, retained corrections, and validation are established components. Yet the evaluation primarily compares history/no-history conditions, without comparing existing reflection or experience methods under a matched actor, interface, and budget. FreeMani’s contribution beyond combining existing components therefore remains unclear.

W4. The abstract presents tool use and potato-chip picking as improving from zero success through repeated updates. However, the body provides no potato-chip task setup, trial denominator, or results. The authors should supply verifiable evidence for this claim or remove it from the abstract.

Questions for authors

Q1. Which calls in Figure 4 actually received which model-extracted experiences? Please provide at least one complete trace linking execution evidence, reflection output, the actor request, and subsequent action, and specify whether raw history, human feedback, and retrieval were enabled concurrently.

Q2. Can the current form compare no persistent correction, expert correction, and model-extracted correction after resetting both the scene and conversation, with matched interaction budgets and controlled raw history? Please report repeated-trial success rates, confidence intervals, and physical execution/model-call costs.

Q3. Which experiments, trial denominators, and successive-update results support the abstract’s potato-chip picking and zero-success direct-reasoning claims? Please identify the corresponding evidence in the paper or Appendix.

1 comment
AuthorsOpenReview ↗

Response to Reviewer J2NL

Dear Reviewer J2NL,

Thank you for highlighting the small-object pickup results and the examples in which language feedback changed the robot’s motion direction, arm selection, and action order. And we address your concerns below.

(W1,W2,Q1) Experiments on Learning from Physical Experience

As suggested by the reviewer,to examine what carries across episodes, we reset the physical scene and actor conversation and inspect the experience text actually delivered to the actor.

In Figure 8 and project page Sec “From physical experience to context space adaptation”, each of the three bread-transfer episodes begins with a fresh actor conversation, neither gripper holding the tongs, and both the bread and the tongs outside the tray. Episode 1 starts without retained experience. Later episodes can draw on lessons retained from earlier interactions. In Episode 1, the robot cannot tell whether the bread is held between the tong tips. At 333 s, it retracts the tongs to bring the tips into view. In Episode 2, it repositions the free wrist camera at 88 s to inspect the tips from another angle and subsequently incorporates this inspection strategy into the lesson. In Episode 3, the actor receives the revised guidance before repositioning the free wrist camera at 83 s.

(W3,Q2) Ablations on Retained Experience and Feedback delivery

Following the reviewer’s suggestion to control the source of correction, we compared conditions with and without retained lessons on bread transfer, drawer retrieval, and item sorting. Each trial began with a reset scene and a fresh conversation. None of these ablation trials received human feedback or expert corrections.

The no-retained-experience condition is our comparison for no persistent correction. The experience module can still generate immediate feedback from the current observation, recent interaction history, and planner output, but it cannot retain or reuse lessons. Full method adds model-extracted lessons that the module can retain and apply when generating subsequent feedback. A separate no-planner condition preserves experience retention and feedback while removing the planner’s current subtask and proposed method.

Across three tasks, the results were as follows (Figure 6).

ConditionTask SuccessSubtask Completion
Memory alone33.3%48.1%
Our method without the planner44.4%59.3%
Our method without retained experience33.3%44.4%
Full method77.8%88.9%

With planning and immediate feedback available in both conditions, enabling lesson retention increased observed task success from 33.3% to 77.8%. For these pooled rates, the descriptive 95% Wilson intervals are 12.1%–64.6% and 45.3%–93.7%, respectively. The intervals reflect the small trial count. This comparison measures the contribution of lesson retention across the evaluated executions, which includes reuse within an episode.

(W4,Q3) Potato-Chip Protocol and Results

In Figure 5b and Appendix E, potato-chip pickup is one of the 12 evaluation tasks. The chip starts on the tabletop, and success requires lifting it clear of the surface while keeping it intact without breaking it(Appendix B, Table 1). Across three trials per condition, success was 0% without memory, 33.3% with memory alone, and 33.3% with the full method.

Sincerely,

The Authors

Reviewer_d89vOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents FreeMani, a proposed framework for adapting a frozen multimodal actor through physical interaction. A context compiler combines observations, task instructions, system guidance, interaction history, and retrieved experience to generate subtasks and executable robot actions. A separate reflection process is intended to convert execution outcomes into structured experience that can be reused in later contexts. The paper reports preliminary history/no-history comparisons, qualitative examples of behavior changes under demonstrations and language feedback, and one within-session bread-transfer trajectory in which the second transfer uses fewer model calls than the first. The problem is relevant, and the real-world interaction traces are potentially useful. However, the current experiments do not establish that model-extracted experience causes persistent or recursive improvement.

Critical strengths
  1. The paper studies a practically important setting: using a frozen general-purpose multimodal model for real-world robot interaction without task-specific parameter updates. The behavior–experience–context abstraction is clearly presented.
  2. The paper includes physically grounded traces involving active perception, tool regrasping, retention checks, and local action correction. The limitations section is also relatively transparent about the lack of fresh-reset validation.
Critical weaknesses
  1. The central learning claim is not directly evaluated. The first bread-transfer phase requires 144 calls and 1,365 seconds, whereas the second requires 26 calls and 240 seconds. However, the second phase retains the tool, inspection configuration, physical setup, session/conversation history, and presumably previously accumulated context. It also uses a different target. There is no fresh-scene/fresh-conversation control, no no-update or raw-history-only comparison, and no repeated independent episodes. Thus, the observed reduction in calls cannot be attributed to model-extracted experience, recursive self-improvement, or skill consolidation. Moreover, the paper does not show the actual reflection output, structured experience entry, retrieval result, or actor context that would establish that the proposed FreeMani loop was responsible for the change.
  2. The current evidence does not support the scope of the claims. Figure 2 uses small and heterogeneous task-level counts with incompletely matched video/history conditions. Figure 3 is qualitative and includes explicit human feedback. Figure 4 is a single non-reset session. The abstract also refers to potato-chip picking and improvement from zero direct-reasoning success, but the submitted manuscript contains no corresponding protocol, result, figure, table, or trial-level evidence. In addition, the method is underspecified with respect to prompts, experience schema, retrieval, context limits, model/API configuration, action contract, success criteria, reset protocol, and operator intervention. These omissions prevent the reader from determining whether the reported behavior changes come from FreeMani, ordinary conversation history, human feedback, retained physical state, or task familiarity.
Questions for authors
  1. Is the potato-chip picking result part of this submission? If so, where are the corresponding protocol, data, and quantitative results? If not, will the abstract claim be removed?
  2. Did the second bread phase receive a reflection-generated experience entry through retrieval? Please clarify whether the conversation history, tool state, inspection pose, and physical scene were retained, and identify the actual information delivered to the actor.
  3. Are there existing episode-level logs that show the reflection output, stored experience, retrieval hit, and actor context for the reported bread trajectory?
  4. How were video availability, history, model identity, task order, and human feedback assigned in Figure 2? Please clarify the success definition and whether the reported trials are independent episodes.
  5. What are the actor/reflection model versions, prompt format, experience schema, retrieval settings, context limits, API parameters, action interface, and operator-intervention protocol?
1 comment
AuthorsOpenReview ↗

Response to Reviewer d89v

Dear Reviewer d89v,

Thank you for recognizing the practical value of adapting a frozen multimodal actor through physical interaction and for highlighting the real-world traces of active perception and recovery. We sincerely appreciate the opportunity to clarify our contributions and address your concerns. Below we respond to each of your questions point by point.

(W1,W2,Q2,Q3) Experiments on Learning from Physical Experience

We thank the reviewer for raising the question regarding the claim of learning from physical experience. We have therefore added experiments to isolate the contribution of extracted experience, and examined how lessons are extracted, revised, and reused across episodes, as suggested by the reviewer.

In Figure 8 and project page Sec “From physical experience to context space adaptation”, each of the three bread-transfer episodes begins with neither gripper holding the tongs and with both the bread and the tongs outside the tray. The first actor request in Episode 1 contains no interaction history or retained experience. As the robot interacts with the bread and tong, the system extracts lessons from physical interaction, revises them as new evidence becomes available, and retains them for reuse. At 333 s, the robot cannot tell whether the bread is held between the tong tips, so it retracts the tongs to bring the tips into view. Episode 2 draws on lessons retained from Episode 1 and updates them through further interaction. It repositions the free wrist camera at 88 s to inspect the tips from another angle and subsequently incorporates this inspection strategy into the lesson. Episode 3, in turn, reuses and updates lessons carried forward from Episode 2. The actor receives the revised guidance before repositioning the free wrist camera at 83 s.

(W2,Q4) Experimental Conditions and Evaluation Protocol

Memory comparison. Figure 2 compares performance with and without interaction memory under the same no-video condition. We report results separately for Astra, Opus, and Sol so that each comparison uses the same actor.

Sources of guidance (Figure 3). Figure 3 examines two forms of context steering. A towel-folding example shows how a visual demonstration guides grasp verification and recovery. A barbecue-transfer example shows how online human feedback changes motion direction, arm selection, and action order. In the subsequent our proposed method performance comparison, the experience module generates the actor’s feedback without online human guidance.

Trial initialization and independence. For the 12-task performance comparison, we run three trials per task and condition. Before each trial, we reset the scene and start a fresh actor conversation. In the experience-reuse analysis, retained lessons can carry across episodes, as described above.

Success criteria. Human annotators judge success from the observed physical outcome. For example, bottle reorientation succeeds when the bottle stands upright on its base after both grippers withdraw. Appendix A describes the evaluation protocol, and Table 1 in Appendix B gives the initial configuration and success criterion for each task.

(W2,Q2,Q5) System Specification

We thank the reviewer for asking us to clarify how experience reaches the actor and to document the implementation details needed to interpret the experiments.

Information delivered to the actor. The experience module uses the planner’s current subtask and proposed method, recent interaction evidence, and relevant retained lessons to generate feedback for the next actor decision. The actor receives this feedback alongside the task instruction, current observation, and interaction memory (Section 3 and Figure 4).

Implementation details. Section 3.2 describes how the actor’s context is constructed and how action requests are executed. Appendix D.1 specifies the experience-entry fields and update rules, and Appendix F provides the module prompts and structured output formats. Appendix A describes the evaluation protocol, and Appendix C explains how demonstrations and online human feedback were supplied in the preliminary experiments.

(W2,Q1) Potato-Chip Results

Yes. Potato-chip pickup is one of the 12 evaluation tasks. The chip starts on the tabletop, and success requires lifting it clear of the surface while keeping it intact without breaking it(Appendix B, Table 1). Across three trials per condition, success was 0% without memory, 33.3% with memory alone, and 33.3% with the full method (Figure 5b and Appendix E).

Sincerely,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.