acceptodds
Under review as a conference paper at ICLR 2027

GRIP: A Generalist In-Context Reward Model for Robotic Manipulation

Abstract

Dense rewards provide step-level feedback for assessing progress, selecting demonstrations, and training policies, yet specifying them separately for every task is costly. We present GRIP, an in-context reward model that predicts a task's dense reward from a small set of labeled observations supplied at inference time. Its main configuration combines a frozen visual encoder with a frozen in-context regressor, allowing tasks to change through context rather than parameter updates. On MetaWorld and ManiSkill, GRIP improves dense-reward prediction over regressors fitted on the same examples. Its predictions improve with more context, depend on the observation–reward pairing, and remain stable when a shared context contains up to 50 tasks. On ProcVQA, a lightweight adapter trained on global visual features raises VOC from 0.6287 to 0.8133. GRIP's predicted rewards also select demonstrations that improve behavior-cloning success. Together, these results show how context-specified rewards can support progress assessment and data selection across robotic manipulation and embodied procedural tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Robotic Manipulation, In-Context Learning, Generalist In-Context Reward Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_jpBHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GRIP, a model that predicts dense rewards for robotic manipulation from a small set of labeled visual examples. Its main configuration combines a frozen visual encoder with a frozen pretrained regressor, allowing tasks to be specified through context without additional parameter updates. The paper also investigates optional feature adaptation and uses predicted rewards together with visual diversity to select demonstrations for behavior cloning. The evaluations demonstrated improved reward prediction and more effective demonstration selection.

Critical strengths
  • The paper provides a clear and intuitive way to specify what progress means for a task. The connection between task specification and reward prediction is interesting.

  • The framework is simple and makes useful use of existing pretrained models. Keeping the main encoder and regressor frozen allows the model to switch tasks without additional optimization.

Critical weaknesses
  • The first and biggest concern is that the method assumes that a task’s dense reward can be inferred from a single image, but from my experience, successful manipulation often depends on small geometric differences and physical conditions that are quite difficult to observe visually. This concern is particularly relevant to the main configuration, which uses a global visual embedding without action history or contact measurements, since additional labeled examples cannot resolve distinctions that the input does not capture.

-Also the connection between dense reward prediction and the downstream application needs stronger justification. Demonstration selection summarizes predicted rewards near the end of a trajectory, so its usefulness may largely come from recognizing proximity to completion.

Questions for authors

In the shared context setting, do CLIP text features enable useful transfer between related tasks, or would replacing them with arbitrary task identifiers produce similar behavior?

What motivated representing trajectory diversity using only the first observed frame, and how would the selection rule distinguish demonstrations that start similarly but follow different manipulation strategies?

Why using only the first frame overlooks differences in manipulation strategies, many demonstration have similar or even identical first frames. They might be treated as redundant despite containing useful info.

5 comments
AuthorsOpenReview ↗

Response to Reviewer jpBH

Thank you for your careful review and questions about visual reward prediction and demonstration selection. We respond to each point below.

W1. Can a single image capture the geometry and physical conditions needed to infer dense rewards?

A single frame often shows how far a manipulation task has progressed, and labeled examples tell GRIP how to score those visible changes. Estimating progress from visual observations is also the basis of GVL [1], which uses Gemini-1.5-Pro to predict per-frame completion from a set of shuffled frames.

In Table 2, with the same frozen CLIP features and 100 labeled frames, GRIP achieves Spearman correlations of 0.6811 on MetaWorld and 0.8276 on ManiSkill. The strongest fitted regressors reach 0.6187 (kNN) and 0.8135 (Ridge), respectively. These results establish the usefulness of single-frame reward prediction on these benchmarks.

We also tested whether a short history adds useful information. Averaging the current and two preceding observation features improves Spearman from 0.7507 to 0.7582 on MetaWorld and from 0.8455 to 0.8574 on ManiSkill, while keeping the feature dimension unchanged. These paired evaluations use the same 100 context labels, three seeds, and up to 10 uniformly sampled query frames per trajectory. Single-frame inputs already provide useful progress estimates, and recent observations improve them further. Contact conditions that are absent from the visual input remain a limitation.

W2. Does demonstration selection validate dense rewards, or mainly recognition of proximity to completion?

GRIP is designed to predict reward at each observation, giving downstream users a progress curve from which to define selection criteria. Our current BC application is an initial use of these predictions. We score the final portion of each trajectory with a low reward quantile to favor sustained completion and reduce the influence of isolated high predictions. At K=20 on MetaWorld MT10, adding GRIP scores to visual diversity raises BC success from 36.8% to 94.6% (Table 4).

Dense predictions also make other selection criteria possible. For example, future selectors could require sufficient completion and then favor smoother progress, penalize substantial reward drops, or assess progress across different phases of the task. These are directions for further exploration. The same predicted reward curve lets users choose criteria suited to their downstream objective without retraining the reward model.

Q1. Do CLIP text features enable transfer between related tasks, or would arbitrary task IDs work similarly?

Arbitrary task IDs can support shared-context prediction. TabPFN receives numerical task features, and an ID identifies which task each example belongs to. We replaced CLIP text features with one-hot IDs for all 50 MetaWorld tasks, using the same 512-dimensional input width, 100 labeled context frames per task, and three seeds.

Task encodingSpearman ↑C-index ↑NRMSE ↓
One-hot ID0.62150.74020.0968
CLIP text0.68090.76750.0909

We also tested random normalized 512-dimensional IDs and shuffled the assignment of text embeddings to tasks. In this additional evaluation using 10 query frames per trajectory, Spearman is 0.7455 for CLIP text, 0.7422 for random dense IDs, and 0.7394 for shuffled text. Arbitrary dense IDs thus recover nearly the same prediction accuracy. Task identity is sufficient for effective shared-context prediction in this setting, with CLIP text providing a convenient encoding.

Q2. Why use the first observed frame for diversity, and how does selection distinguish different strategies from similar starts?

The first observed frame helps a fixed demonstration budget cover different initial conditions. Selecting by reward alone can retain many high-quality demonstrations from similar starting scenes. First-frame diversity reduces that redundancy, while the GRIP score favors demonstrations that reach higher completion.

Demonstrations with similar starts can still receive different GRIP quality scores, but the first-frame diversity term does not distinguish their manipulation strategies. We will describe its role as initial-condition coverage.

Q3. Can similar or identical starting frames cause useful demonstrations to be treated as redundant?

The selector combines reward quality with diversity, so similar starting frames do not automatically exclude a demonstration. We also tested whether adding later observations to the diversity representation improves downstream selection. We compared the first frame with two alternatives: a mean over all frame features, and a concatenation of four segment means that retains coarse temporal order. Only the diversity representation changes; GRIP scores, the selection objective, and BC training remain fixed. The table reports BC success (%) at K=20 over three paired seeds.

Diversity featuresMetaWorld MT10ManiSkill
First observed frame93.00 ± 5.1934.42 ± 2.65
Whole-trajectory mean88.07 ± 3.4932.29 ± 3.82
Four-phase features84.47 ± 11.3236.08 ± 1.94

Using the whole-trajectory mean lowers success on both benchmarks. Four-phase features raise the ManiSkill mean from 34.42% to 36.08%, while the first frame remains best on MetaWorld. The first frame therefore provides a useful, simple coverage measure, and phase-aware diversity is a promising extension for retaining differences later in the trajectory.

Conclusion

GRIP turns a few labeled observations into a dense visual reward for a new task, without fitting a task-specific model. The same predictor can then score unlabeled demonstrations for policy training.

[1] Ma, Yecheng Jason, et al. "Vision language models are in-context value learners." ICLR. 2025.

Thank the authors for the detailed clarification!
Aopologies, this is a bit late but could author provide real-robot expirement results? Does the model maintain it's capability in real lifte setting? does the real life expierment show the generalization aspect of the model?

Reviewer_gc4VOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

In this paper, authors proposed GRIP, an in-context reward model framework that utilizes frozen visual encoder and frozen regressor to conduct in-context style robotics reward modeling. GRIP aims to predict task-specific dense rewards from a small set of labeled in-context examples, without requiring task-specific model training. Experiments also show that GRIP can help select better demos for down-stream policy training.

Critical strengths

In-context learning to achieve training-free reward models is well motivated

Critical weaknesses
  1. The generalization is mainly intra-task generalization but not at cross-task level, which only saves training compute but not data or annotation requirement. Also, author should point out this clearly at problem definition section, otherwise the paper has serious over-claim issues.

  2. It is unclear whether GRIP can generalize from successful demos to fail queries, no visualization/demonstrations provided.

  3. The paper did not compare with any training-free general robot reward models like TOPReward [1].

  4. All experiments with both reward model and policy are limited to simulation benchmark, it is unclear whether the method can work on real robot settings.

Reference [1] Liang, Anthony, et al. "Robometer: Scaling general-purpose robotic reward models via trajectory comparisons." arXiv preprint arXiv:2603.02115 (2026).

Questions for authors

Same as weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer gc4V

Thank you for reviewing our work and raising questions about generalization and practical applicability. We respond to each point below.

W1. Labeling and training on a new task

Figure 3 shows where the labeling savings come from. For a new task, we only label the frames supplied as context. With 10 labeled frames from one demonstration, training-free GRIP reaches a Spearman correlation of 0.3787 on MetaWorld and 0.6708 on ManiSkill. With 50 labeled frames from five demonstrations, these scores rise to 0.6134 and 0.7937 (Table 8). This reduces the amount of target-task annotation compared with preparing densely labeled trajectories to train a reward model.

Both versions switch to a new task by changing the context. The training-free version uses frozen CLIP and TabPFN throughout. The adapter version trains once on source tasks, then uses the same adapter on held-out tasks without updating it (Table 6). We will clarify in the abstract and Sec. 3.1 that the main benchmark tests generalization to unseen trajectories of the same task.

W2. Can successful demonstrations tell us how a failed attempt progressed?

GRIP can predict rewards on failed attempts even when every demonstration in its context succeeds. On MetaWorld, we gave all five methods the same 100 labeled frames from 10 successful demonstrations per task and scored the same held-out failures. GRIP achieves the best Spearman correlation and lowest NRMSE among all five methods, while its C-index is close to the best result.

MethodFailure-frame Spearman ↑C-index ↑NRMSE ↓
GRIP0.43390.65660.5968
kNN0.42490.65880.7150
Ridge0.25320.58330.9818
GP0.17880.54200.9987
XGBoost0.27360.59141.0090

Successful demonstrations are therefore sufficient to obtain useful predictions on failed attempts. GRIP can also use labeled frames from failures in context. Our main experiments include both outcomes in context and achieve 0.6811 Spearman and 0.7663 C-index on held-out successful and failed trajectories (Table 2).

W3. TOPReward

We added a comparison with TOPReward on 120 held-out trajectories across three MetaWorld tasks, including 30 successful and 90 failed attempts. Both methods are evaluated at the same 10 time points per trajectory. On this full test set, standard training-free GRIP with mixed context achieves 0.5965 Spearman and 0.7284 C-index, compared with 0.3112 and 0.6069 for TOPReward. We use TOPReward's released Qwen3-VL-8B-Instruct implementation with task text and video prefixes. GRIP uses a single query image and 100 labeled context frames per task.

W4. Real-robot data

Table 5 reports offline progress prediction on ProcVQA. Training-free GRIP reaches a VOC of 0.6287, and the adapter version reaches 0.8133. These results extend the evaluation to progress estimation on recorded demonstrations.

Conclusion

GRIP specifies a task’s dense visual reward at inference time through a small set of labeled examples. By changing these examples, the same frozen predictor can score unseen trajectories of a new task without task-specific training.

Reviewer_gc4VOpenReview ↗

Respond to author

Thank you for clarification, I will mark 2 & 3 as resolved.

  1. how many tasks included in training-free arm on earth, how many tasks it saw previously? Can you add a visualization reward curve of its performance on an unseen task, compared with GT.

  2. I still believe that a reward model is not persuasive enough without real robot policy improvement results.

Thank you for the follow-up and for confirming that the successful-context generalization experiment and the TOPReward comparison resolve your related concerns. We address your remaining questions below.

1. Task coverage and reward curves for training-free GRIP

Training-free GRIP is evaluated on 50 MetaWorld tasks and 8 ManiSkill tasks, for 58 robotic manipulation tasks in total. None of these tasks is used for parameter training in GRIP: we directly use frozen CLIP and TabPFN without fine-tuning them on robotics tasks, and TabPFN's regression capability comes from pretraining on synthetic tasks. When evaluating a target task, GRIP receives a small set of image–reward pairs from that task as context and predicts rewards on disjoint trajectories. Thus, the same frozen model handles different tasks by changing its context, without task-specific model training.

The optional adapter version is a separate experiment, with train/validation/test task splits of 20/5/25 for MetaWorld and 3/2/3 for ManiSkill.

We have added comparisons between ground-truth rewards and GRIP predictions at the end of the appendix, allowing a direct comparison of the ground-truth and predicted rewards on query trajectories.

2. Evaluation on real trajectories and real-robot policy training

Our paper already evaluates progress prediction on real-robot trajectories: Table 5 follows ProcVQA's official train/test split and evaluation protocol and uses the full ProcVQA-ID evaluation set, covering real-robot trajectories from DROID, Bridge, Table30, and selected OXE datasets. Training-free GRIP achieves 0.6287 VOC, increasing to 0.8133 with the adapter. These results support GRIP's progress estimation capability on unseen real-robot trajectories. Downstream policy benefits are evaluated through demonstration selection and BC on MetaWorld and ManiSkill.

We would also like to emphasize the methodological contribution. To our knowledge, GRIP is the first to use a PFN to infer per-frame dense robotic rewards from a small set of labeled visual examples. The context defines both which visual changes count as progress and how those changes map to numerical scores. Changing the reference examples lets the same frozen predictor implement a different reward specification. This offers a new way to design rewards through a few examples, with the predicted progress supporting trajectory assessment, data selection, and subsequent policy learning.

Combining reward evaluation on real trajectories with policy validation in simulation has direct precedent. For example, VICtoR [1] evaluates reward curves on real-robot XSkill videos, while validating downstream policy improvements in simulation, without reporting benefits from real-robot policy training. In addition, RoboCLIP [2] and VLM-RM [3] also validate new visual reward specification methods through simulation experiments. We believe GRIP's methodological contribution should be assessed through its reward inference mechanism, prediction performance on real trajectories, and demonstrated downstream policy benefits.

[1] Hung, Kuo-Han, et al. "VICtoR: Learning hierarchical vision-instruction correlation rewards for long-horizon manipulation." ICLR. 2025.

[2] Sontakke, Sumedh, et al. "RoboCLIP: One demonstration is enough to learn robot policies." NeurIPS. 2023.

[3] Rocamonde, Juan, et al. "Vision-language models are zero-shot reward models for reinforcement learning." ICLR. 2024.

Reviewer_cPaiOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

GRIP combines frozen CLIP embeddings with the pretrained TabPFN regressor to predict visual rewards from labeled context examples, without task-specific parameter updates. Experiments evaluate reward prediction, context dependence, optional feature adaptation, and demonstration selection for behavior cloning on simulated manipulation tasks and procedural videos.

Critical strengths
  • Context-based reward specification is simple and flexible. Matched visual features and labeled examples make the prediction comparisons informative, while disjoint trajectory/source-group splits and context interventions strengthen the evidence that the model uses the supplied reward mapping.
  • Predicted rewards improve demonstration selection and downstream behavior cloning under paired training conditions, with especially large gains on MetaWorld. Context-size and shared-context analyses further characterize when the approach works. These results provide useful empirical support for changing reward specifications without retraining the model.
Critical weaknesses
  • The methodological contribution is mainly the application of existing components: CLIP and TabPFN, with the trained robot adapter following V0. Adapter gains on held-out robot tasks are small and confounded by additional patch inputs. Downstream selection also omits kNN and Ridge, despite their close prediction results on ManiSkill, leaving the policy benefit over simpler regressors unresolved.
  • Practical annotation savings remain uncertain. Main experiments use 100 precise environment-reward labels per task, and shared-context scaling retains that per-task budget. Human-label feasibility and robustness to realistic annotation errors are untested. The downstream evidence establishes value for offline data filtering; suitability as a dense RL reward remains open.
Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer cPai

Thank you for recognizing the flexibility of context-based reward specification and its downstream benefits. Your comments helped us strengthen the baseline comparisons and examine adapter training and annotation errors more directly. We address the two weaknesses below.

W1. Contribution, adapter training, and downstream comparisons

  • Specifying task rewards through examples. GRIP places the task's reward definition in labeled context: changing the examples lets the same frozen model score another task without fitting a task-specific model.

  • Adapter training helps with inputs held fixed. Table 5 keeps ProcVQA inputs and label budgets fixed; adapter training raises VOC from 0.6287 to 0.8133. To address the patch-input confound in the robot experiment, we added a matched-input control. On three held-out ManiSkill tasks, the same adapter at initialization and step 20,000 gives mean Spearman of 0.8165 and 0.8306, and NRMSE of 0.1641 and 0.1553, over three paired context seeds. Thus this gain is obtained without changing patch inputs.

  • Downstream comparison with simple regressors. We added kNN and Ridge selection and BC experiments using the same features, label budget, selection rule, and training protocol. Values are success (%) as mean ± SD over three seeds; bold marks the highest mean.

    BenchmarkReward predictorK=20K=40K=60
    MetaWorld MT10GRIP94.6 ± 3.695.7 ± 1.596.2 ± 0.2
    MetaWorld MT10kNN91.5 ± 7.794.3 ± 2.792.3 ± 2.8
    MetaWorld MT10Ridge92.9 ± 4.995.9 ± 3.194.4 ± 4.5
    ManiSkillGRIP35.7 ± 3.440.4 ± 3.144.1 ± 3.5
    ManiSkillkNN32.3 ± 2.438.4 ± 2.243.7 ± 2.2
    ManiSkillRidge32.2 ± 1.639.0 ± 0.840.7 ± 6.2

    GRIP has the highest mean in five of six settings, exceeding the strongest simple regressor by 3.4 percentage points on ManiSkill at K=20. Ridge is slightly higher on MetaWorld at K=40.

W2. Annotation requirements and the scope of downstream evidence

  • A small context budget without reward-model training. GRIP uses 100 labeled reference frames to score 320 candidate demonstrations per task (Table 4); with only 50 labels, Spearman reaches 0.6134 on MetaWorld and 0.7937 on ManiSkill (Table 8). The training-free configuration needs neither adapter training as in V0 nor autoregressive MLLM scoring as in GVL, avoiding these additional training and generative-inference costs.

  • Some robustness to noisy labels. In a new experiment, Gaussian noise with σ=0.05 on the [0,1] label scale changes NRMSE only from 0.1162 to 0.1185 on MetaWorld and from 0.1671 to 0.1678 on ManiSkill. Spearman changes from 0.7507 to 0.5842 and from 0.8455 to 0.8368, respectively. Both benchmarks retain useful predictions, showing some tolerance to imperfect context labels. Noisy labels are clipped to [0,1] and query labels remain clean; evaluation uses three paired seeds and up to 10 uniformly sampled query frames per trajectory.

  • RL as a future direction. We plan to explore GRIP as dense feedback during RL optimization; the current policy experiments concern demonstration selection and BC.

Conclusion

GRIP uses a small labeled context to score unlabeled trajectories without task-specific reward-model training. The added regressor comparisons strengthen the evidence for its downstream selection value, while the matched-input adapter control shows that training can improve predictions with visual inputs held fixed.

Thank you for your review! We have added the relevant comparison experiments and addressed your questions in our response. We would appreciate it if you could take a look when convenient. We would be happy to clarify any further questions. Thank you again for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.