acceptodds
Withdrawn from ICLR 2027

NDB-JEPA: Label-Efficient Grasp Detection from Frozen I-JEPA Features

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

As robots become increasingly prevalent in real-world settings, their perception models must become label-efficient and robust to distributional shifts not represented in their labeled training data which poses a considerable challenge for end-to-end grasp detection models typically trained on limited or synthetic datasets. We introduce NDB-JEPA, a label-efficient grasp point detection perception model centered around I-JEPA. Our proposed model takes advantage of the robust, generalized feature extraction of I-JEPA, a self-supervised joint-embedding predictive foundation model. Our model trains lightweight grasp heads on top of the frozen image patch outputs of a 632M-parameter I-JEPA encoder, predicting grasp confidence, orientation, and width per embedded patch. We ask whether these frozen features can replace end-to-end training for the grasping problem. On the Cornell Grasping Dataset, NDB-JEPA reaches 86.9% Top-1 rectangle success, within 1.5 points of an end-to-end GR-ConvNet baseline, and among large pretrained encoders, I-JEPA features outperform MAE, supervised ViT, and DINOv2 alternatives. The frozen representation is also markedly more label-efficient and robust: with only 10% of the training data, NDB-JEPA attains roughly 74% success versus 40% for GR-ConvNet trained from scratch, and it retains about 94% of its clean accuracy under severe darkening while the end-to-end baseline collapses to 1.1%. While end-to-end models retain a small edge in clean accuracy, our results indicate that frozen joint-embedding predictive representations offer a data-efficient, robust basis for grasp perception.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robotic grasp detection, object manipulation, grasp point detection, vision foundation models, JEPA, I-JEPA, self-supervised learning, label efficiency, robustness to distribution shift, robustness to out-of-distribution, vision transformers

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_EPCuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper adapts a frozen I-JEPA vision encoder with lightweight prediction heads for grasp detection. The main claim is that frozen representations generalize better than end-to-end trained models. While the empirical setup has some carefully designed controls, several problems with method novelty, experimental fairness.

Critical strengths
  1. This paper explores the feasibility of using the frozen large-scale pre-trained model (I-JEPA) for object detection and demonstrates its good robustness in out-of-distribution scenarios.Useful for real-world deployment.

  2. The experimental setup was comprehensive.

Critical weaknesses
  1. This paper merely added a lightweight double-tower structure on the basis of the I-JEPA framework, without adapting the features for the grasping tasks. The innovation of the method is insufficient.

  2. The experimental comparison is unfair. The abstract and introduction state that frozen representation is better than end-to-end training, but the main text does not conduct the same architecture I-JEPA end-to-end fine-tuning experiment. Additionally, the 632M ViT-H is compared with a small CNN of only a few M, and the parameter quantity difference is more than an order of magnitude. This merely indicates that large models have a large number of parameters, but it cannot prove that frozen representation is necessarily better than end-to-end training.

  3. The empirical evidence for generalization ability is scarce. Most test objects have already appeared from other perspectives in the training set, so the generalization ability for unseen objects has not been fully verified. Moreover, the experiments were conducted only on the Cornell single dataset, lacking cross-dataset validation, and no real robot closed-loop tests were carried out.

  4. Appendix A and B are contradictory. A.4 states that without performance enhancement, it drops to 61.8%; enhancement is necessary. B.3 re-ran the same experiment, with or without enhancement, it remained around 85%, which fails to demonstrate the necessity of enhancement.

  5. The relevant work has not been written in a comprehensive manner.

Questions for authors

See the Weaknesses.

1 comment

Thank you for the review. We respond to each weakness below.

Regarding method novelty. We agree that our contribution is empirical: the head is deliberately lightweight, and we do not adapt the I-JEPA features to the grasping task. The next version will present the work as a controlled study of frozen representations rather than as a new method.

Regarding tairness of the comparison. We would like to clarify one point. Section 5.4 includes a control with the same ViT-H/14 architecture, the same 632M parameters, and the same head as NDB-JEPA, but with random frozen weights. This control reaches only 60.7% on clean images against 86.9% for I-JEPA, and it fails completely under darkening (0.0%) according to our experiments. We have made this code available to test out as well. In Table 1, DINOv2 ViT-g/14 is the largest encoder at approximately 1.1B parameters, yet it remains below both I-JEPA and the supervised ViT-H/14. We believe that model size alone does not explain our results, but would like to do more experimentation to have an even stronger argument for this. However, you are right that we do not fine-tune I-JEPA end-to-end, because this was infeasible on our available compute, and Section 5.4 states that our controls do not separate freezing from fine-tuning. As Section 5.3 notes, NDB-JEPA and GR-ConvNet also differ in architecture, capacity, and training regime, so that comparison does not isolate the effect of freezing a pretrained encoder. Our abstract and introduction should not suggest that frozen representations are better than end-to-end training in general, and we will revise both to claim only what the controls support.

Regarding the generalization evidence. Our evidence for generalization is limited: most test objects appear in the training set from other viewpoints, as we state in Section 4.2, so our results do not establish generalization to unseen objects. We will add an object-wise split and a controlled cross-dataset evaluation. We will also revise our wordings to better match the results of these additional tests.

The contradiction in our appendices is valid. The A.4 ablation applied augmentation once, when the dataset was constructed, to every split. Appendix B.3 repeats the ablation on the validation split with three seeds and with augmentation applied to the training split only, and there augmentation has no measurable effect. B.3 says that the earlier result does not reproduce, but we left A.4's statement that augmentation is vital in place, which was a mistake; the next version will report only the B.3 result. None of the backbone, low-shot, or corruption experiments use data augmentation, so our main results are not affected. This wording was an artifact of our iteratively working through the experiments and we should have been diligent to clean up any obsolete findings.

Regarding our related work. Our related work is not comprehensive, in particular for grasp detection; the next version will cover recent foundation model grasp detectors and robot control methods.

Reviewer_KLTJOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper trains two lightweight 1×1 convolutional towers on frozen I-JEPA ViT-H/14 patch embeddings for planar parallel-jaw grasping detection on the Cornell grasping dataset. One tower predicts a grasp confidence map over a 16×16 anchor grid, while the other predicts (cos 2θ, sin 2θ, width). On an image-wise split 80/10/10 dataset, the model achieves a top-1 rectangular success rate of 86.9% at IoU > 0.25; end-to-end RGB-D GR-ConvNet reaches 88.4%, and RGB-only GR-ConvNet achieves 82.0%. With the same prediction head, frozen I-JEPA outperforms frozen MAE-H, supervised ViT-H, and DINOv2-g. With 10% of the training data, NDB-JEPA reaches about 74%, versus about 40% for GR-ConvNet trained from scratch. On Cornell-C , it is far more robust to darkening and occlusion. Architecture controls suggest that the gains come from pretraining rather than from the architecture.

Critical strengths
  1. The design of the representation probe is clean. The head consists solely of 1×1 convolutions, effectively functioning as a patch-wise probe; the code for backbone contrastive partitioning, loss computation, decoding, and evaluation is fully unified.
  2. The architectural alignment comparison is informative. By contrasting a randomly initialized, frozen ViT-H model with a ViT-B model trained from scratch, the explanation that "it’s simply the ViT architecture at work" is effectively ruled out.
  3. Three seeds, a shared evaluation implementation, an oracle analysis of the anchor-grid ceiling, and a modality-matched RGB-only baseline for the corruption study.
Critical weaknesses
  1. The evaluation scale is small, resulting in weak statistical power. The test set contains only 89 images, with each image representing approximately 1.1 percentage points. Since the data is split by image, most test objects have appeared in the training set from different viewpoints. There is no object-level splitting, no evaluation on Jacquard—Jacquard was used solely for hyperparameter search—and no real-world robot grasping experiments were conducted. A difference of 3.4 percentage points between I-JEPA and MAE, for instance, does not seem to be conclusive.
  2. All few-shot models should be trained until convergence—either by using the same number of gradient update steps across all data proportions, or by scaling the number of epochs inversely proportional to the data fraction and selecting checkpoints based on validation set performance. At a minimum, results for GR-ConvNet trained for 100 epochs (following the ViT-B setup) should be provided, along with training and validation curves for the 5% and 10% data settings. With a frozen head, the model undergoes only about 40 update steps at the 10% data level and roughly 20 steps at the 5% level; consequently, the relative ranking of different backbones at 5% data—for instance, the ~17% performance of MAE—may be skewed by insufficient training.
Questions for authors

1.Could you add tests broken down by object, as well as cross-dataset evaluations—for instance, training on the Cornell dataset and testing on Jacquard (or vice versa)—and, if feasible, conduct a small-scale evaluation on a real-world robot? 2. Could you report GR-ConvNet trained to convergence in the low-shot setting? Possible options include a fixed number of updates across fractions, epochs scaled by 1/fraction, or simply 100 epochs as for ViT-B, with checkpoints selected on validation. Training and validation curves at 5% and 10% would also help. As a smaller point, the frozen heads receive only about 20 updates at 5%, so it may be worth checking whether MAE's drop to about 17% persists with a longer budget.

1 comment

Thank you for a careful and insightful review. Your summary describes the probe design and the architecture-matched controls as we intended them.

Regarding our evaluation scale and the gap between I-JEPA and MAE. The test set is small, and we agree that a 3.4-point gap is not conclusive on 89 images. Our section 5.1 describes this gap as comparable to the seed-to-seed variation of the two models; however, our abstract states the ordering more strongly than that, and we will correct it by running larger experiments and rewording the abstract to match the rest of the paper. An object-wise split with cross-validation folds or more seeds should give a more reliable estimate than a single test set.

Low-shot training budget question; this is an important point, and we agree with it: every data fraction was trained for the same 20 epochs, so the smallest fractions received few updates. GR-ConvNet was trained at batch size 8 and the frozen heads at batch size 32, which means the heads received even fewer updates per epoch. The ordering at 5% may therefore reflect insufficient training rather than representation quality. We will run the low-shot study with epochs scaled inversely to the data fraction for every system, including GR-ConvNet, keeping the checkpoint selection by validation loss that we already use, and we will report training and validation curves at 5% and 10%. This will show whether MAE's drop to approximately 17% at 5% persists with a longer budget. We also note that the low-shot runs for MAE, supervised ViT, and DINOv2 used the Base-size checkpoints rather than the larger encoders of Table 1, so the rerun will use the size-matched encoders throughout.

Object-wise, cross-dataset, and real-robot evaluation (Q1). We plan to add both the object-wise split and a cross-dataset evaluation. An earlier Jacquard-to-Cornell transfer run of ours used a single seed and was not strong enough to report, and we would rather present a properly controlled version than a rushed one. A real-robot evaluation is something we may be able to run, but at the time of the intial experimentation wasn't an option.

Reviewer_gAHXOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies label-efficient grasp detection using frozen pretrained visual representations. The authors propose NDB-JEPA, which uses a frozen I-JEPA encoder to extract image features and trains lightweight prediction heads to predict grasp confidence and geometry. Experiments on the Cornell Grasping Dataset compare different pretrained encoders and evaluate performance under limited training data and image corruptions. The results show that I-JEPA features achieve competitive grasp detection performance and provide better label efficiency and robustness than several baselines.

Critical strengths
  • The paper studies a meaningful problem of improving label efficiency and robustness in robotic grasp detection.

  • The experiments compare different pretrained visual encoders and evaluate performance under different amounts of labeled data and image corruptions. The authors also provide several controlled comparisons to investigate the experimental results.

Critical weaknesses

Insufficient motivation and unclear novelty. The Introduction discusses the challenges of label efficiency and generalization in grasp detection, but it does not clearly establish the limitations of existing methods that this work aims to address. The proposed method mainly uses a frozen pretrained I-JEPA encoder with lightweight prediction heads. It is unclear what the main technical contribution is beyond applying existing pretrained representations to grasp detection.

Unclear organization and methodology presentation. The organization of the paper could be improved. Section 3 provides only a brief description of the proposed architecture, while important implementation details and the training objective are introduced in Section 4. The methodology would be easier to understand if the prediction process and training objectives were described more systematically in Section 3.

Questions for authors
  1. What is the main novelty of NDB-JEPA compared with existing methods that use frozen pretrained visual encoders for downstream robotic tasks? The authors should more clearly explain which limitations of existing methods are addressed by the proposed framework.

  2. The experiments are conducted only on the Cornell Grasping Dataset, using an image-wise split. Could the authors provide experiments on additional datasets or object-wise splits to better evaluate generalization to unseen objects?

  3. Could the authors provide a more complete description of the proposed methodology in Section 3, including how the grasp predictions are generated and how the prediction heads are trained?

1 comment

Thank you for the review and for the suggestions on how the paper is organized.

Regarding our motivation and novelty. We agree that the introduction does not make clear which limitation of existing work we address, as we focus too much on methodologies from 3-4 years ago. Existing work on frozen visual encoders for robotics, such as R3M, MVP, and VC-1, largely evaluates representations through downstream policy learning. We instead evaluate them on grasp rectangle prediction, a dense perception task with a standardized metric, which isolates representation quality from policy-optimization effects. Within this setting we hold the grasp head, data split, and evaluation fixed and vary only the pretrained encoder. Our contribution is this controlled comparison of pretraining objectives, and in our rework of this paper we will build the introduction around it.

About the additional datasets and object-wise splits question. This is the main gap in our evaluation. Our image-wise split tests new views of objects that also appear in the training set, which we state in Section 4.2 and in our limitations, and we will add an object-wise split. This was an oversight on our part as to how vital this would be to the generalization claim and study.

Regarding the methodology in Section 3. This is a fair criticism of how the paper is organized. Section 3 describes the architecture only briefly, while the head topology and the training objective are given in Section 4.4. In the next version, Section 3 will give the complete method: how the patch tokens are reshaped into the anchor grid, what each tower predicts, how the top-1 grasp is decoded, and how the heads are trained. Section 4 will then cover only the experimental protocol. At first Section 3 was larger, but we cut down on this to save space.

Reviewer_wmEVOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper presents NDB-JEPA, a grasp detection model that trains lightweight confidence and geometry heads on a frozen, pretrained I-JEPA encoder. The heads predict grasp scores, orientations, and gripper opening widths at image-patch locations. On the Cornell Grasping Dataset, the authors compare grasp-detection performance using frozen I-JEPA, MAE, supervised ViT, and DINOv2 encoders with a common grasp-head architecture, alongside GR-ConvNet trained from scratch. They report competitive clean-image accuracy, improved label efficiency, and greater robustness to darkening and occlusion than the tested GR-ConvNet baselines. The paper’s main contribution is an empirical investigation of frozen pretrained representations for grasp detection under limited supervision and synthetic image corruptions.

Critical strengths
  1. The paper is easy to follow, with a straightforward description of the proposed method and experimental setup.

  2. Within the Cornell benchmark, the authors compare multiple pretrained encoders, examine label efficiency and corruption robustness, and include architecture controls and design ablations.

Critical weaknesses
  1. It combines an existing frozen I-JEPA encoder with straightforward confidence and geometry heads. Its contribution is mainly empirical.

  2. The paper does not explain convincingly why independent towers are preferable to a joint head, or test alternative decoder designs.

  3. Although Table 1 compares multiple frozen pretrained encoders on clean images, the Cornell-C evaluation does not extend this comparison to corrupted images.

  4. Results come from one small Cornell split, where most test objects also appear in training. Generalization to unseen objects, other datasets, and physical grasping remains unverified.

Questions for authors
  1. Could the authors clarify the main novel insight beyond combining a frozen I-JEPA encoder with lightweight grasp heads?

  2. What motivates using independent confidence and geometry towers? Could the authors compare this design with a joint prediction head or a small spatial decoder under a comparable parameter budget?

  3. Could the Cornell-C evaluation include frozen MAE and DINOv2 encoders to determine whether the robustness gains are specific to I-JEPA or shared by other pretrained representations?

  4. Could the authors evaluate an object-disjoint Cornell split or an additional dataset?

1 comment

Thank you for the clear review and for the specific questions.

Regarding your first question of novelty. We agree that our contribution is empirical rather than architectural. The work is structured as a controlled study: holding the grasp head, data split, and evaluation fixed. We only change the pretrained encoder, and then we measure how each representation behaves as labeled data shrinks and amount of appearance shift. The finding we aim to establish is that the pretraining objective governs how useful frozen features are for dense grasp prediction, and that encoder size or pretraining by itself does not. The introduction describes the work this way, but the abstract and our first contribution present NDB-JEPA which muddies the water on this; the next version will be organized around the study instead.

Next for the towers, we do not have evidence that independent towers are preferable to a joint head, because we did not compare the two. The two towers keep the confidence prediction, which is trained at every anchor, apart from the geometry prediction, which is trained only at positive grasp anchors. We will add a joint head and a small spatial decoder at a comparable parameter budget.

Regarding Cornell-C for the other encoders. This comparison is key to knowing whether the robustness gains are specific to I-JEPA or shared by other pretrained representations, and the paper unfortunately does not include it. The Cornell-C corruptions are applied at evaluation time only, so the frozen MAE, supervised ViT, and DINOv2 models can be evaluated on the full suite under the same protocol, and we will report all of them.

Finally, our current split does not test generalization to unseen objects: as we state in Section 4.2, Cornell's 885 images show 240 objects, so most test objects also appear in the training set from other viewpoints. The next version will add an object-wise split to address this shortcoming.

Comments on the paper

2 comments

Dear reviewers and authors,

As the discussion period is limited, please follow up on the latest responses and comments in a timely manner to address any remaining concerns. Thank you!

We thank all the reviewers for their time and the care put into reviewing our paper. The reviews agree on several points, and we agree with them. Our evaluation uses a single image-wise Cornell split, so it measures generalization to new views of seen objects rather than to unseen objects. The Cornell-C evaluation also does not include the other frozen encoders, which leaves open whether the robustness is specific to I-JEPA or shared by other pretrained representations. The low-shot runs train for the same 20 epochs at every data fraction, and the smallest fractions receive very few updates as a result. Finally, the paper should present and make clear that our findings are a controlled study of frozen representations rather than a new architecture. Addressing these points properly requires more experiments than we can run on our available compute within the discussion period; therefore, we are withdrawing the submission. The next version will address these points, such as with an object-wise split, Cornell-C results for every backbone, low-shot runs with a longer training budget and their training and validation curves, a comparison of our two-tower head against a joint head, a complete description of the method in Section 3, and a broader related work section. We are grateful for this feedback, which has made it clear what the work needs next.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.