acceptodds
Withdrawn from ICLR 2027

RN-VLA: Reference-Point Normalized Vision-Language-Action Models for Sub-Centimeter Fine-Grained Manipulation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Fine-grained robotic manipulation—such as pressing a specific key on a dense keyboard or performing micro-adjustments on household dials—remains challenging, as moderate per-step errors compound exponentially over long-horizon sequences. Monolithic Vision-Language-Action (VLA) models struggle here due to instance-level semantic ambiguity, geometric attention dispersion, and coordinate scale mismatch between reaching and fine manipulation. We propose RN-VLA, which grounds task-specified targets via Semantic Query Grounding (SQG), suppresses background clutter through FiLM-based Cross-Modal Gated Modulation (CMGM), and re-centers the action space onto the target part via Reference Point Normalization (RPN), enabling a diffusion policy to achieve sub-centimeter precision. Experiments in Isaac Sim and on a physical xArm6 robot show RN-VLA achieves over 80% atomic success across seven household tasks, outperforming state-of-the-art VLA and 3D diffusion baselines, with strong robustness under out-of-distribution placements. By eliminating atomic-level errors, reliable single-step policies compose zero-shot into long-horizon tasks—without LLM planning or online verification—achieving up to a 68% relative improvement over the strongest baselines (up to 1.68× higher success rate).

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action (VLA) Models, Long-Horizon Manipulation, Fine-Grained Manipulation, Diffusion Policy, Spatial Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EoXBOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RN-VLA addresses fine-grained robotic manipulation through a modular pipeline combining Semantic Query Grounding, Cross-Modal Gated Modulation, and Reference Point Normalization. A task-conditioned perception frontend predicts a target-part bounding box. Its centroid is used to re-center geometric observations and proprioception, while visual and task features modulate point-cloud features through FiLM. A conditional diffusion policy generates action sequences.

The paper reports substantial improvements on seven atomic manipulation tasks, real-robot demonstrations, and improved success when atomic policies are composed into longer sequences. My main concerns are whether the improvements are sufficiently attributed to the proposed mechanisms, whether the input and ablation protocols isolate their contributions, and whether the evidence supports the stated precision and long-horizon claims.

Critical strengths

S1. Meaningful tasks and substantial reported improvements.

The tasks are interesting and practically relevant, particularly selecting and pressing a specific key among densely arranged alternatives. The reported average atomic success rate of 81.6% substantially exceeds the evaluated baselines. Real-robot results, including 70.0% success on Keyboard and 67.6% on Remote with predicted bounding boxes, provide useful evidence that the system can perform these interactions beyond simulation.

S2. The internal ablations support the usefulness of several design choices.

The target-centered representation and cross-modal modulation have substantial effects within the proposed system. On Keyboard, replacing the target reference with a workspace-fixed reference reduces success from 80.8% to 53.4%, while removing CMGM reduces it to 56.1%. These results support the utility of the components, although they do not fully establish the mechanisms or comparative advantages claimed.

Critical weaknesses

W1. Limited methodological novelty and insufficient attribution of the improvements.

The method primarily integrates established representation extraction, conditional feature fusion, coordinate re-centering, and diffusion-based control. These choices have reasonable motivations, but the paper does not sufficiently establish what new methodological insight the combination provides or why its particular feature-processing mechanisms are responsible for the reported fine-grained manipulation advantage.

Table 3 is especially relevant: compared with FVP-DP3, target-selection accuracy increases from 58.6% to 94.0%, whereas execution success conditional on correct selection increases from 84.0% to 86.8%. This suggests that improved target specification or localization is central to the overall gain. Since the correctly selected subsets may differ, these statistics are not a controlled causal decomposition, but they do call for a more careful distinction between selection and execution improvements.

The evidence does not clearly demonstrate that SigLIP-based fine-grained visual discrimination or the proposed visual-to-geometric modulation produces more accurate manipulation. Aggregate selection accuracy and attention maps do not establish whether the model reads distinguishing target details, exploits familiar spatial layouts, or primarily benefits from explicit target-box conditioning. Similarly, channel-wise FiLM modulation is not itself evidence of precise point-level spatial selection. The contribution should be positioned more carefully and supported by controls that isolate these explanations.

W2. The “without RGB” ablation and baseline input protocols are insufficiently specified.

Removing RGB reduces success only from 80.8% to 76.8% on Keyboard and from 83.9% to 79.8% on Safe. These relatively small differences require explanation given the central role assigned to visual semantics and cross-modal fusion.

RGB information can reach the policy both through the visual-fusion branch and indirectly through SQG-generated bounding boxes. The paper does not clearly specify whether “without RGB” removes both paths, whether SQG still processes RGB, or whether predicted, cached, or oracle boxes remain available. Consequently, the ablation does not establish the end-to-end contribution of RGB or SigLIP. Moreover, CMGM receives task and bounding-box information, so its larger ablation effect cannot be attributed specifically to RGB-driven modulation.

The side branch also directly supplies task, bounding-box, and proprioceptive information, and training explicitly includes cases with the fused visual condition removed. Its standalone capability is therefore an important missing reference.

Finally, each baseline's access to the intended target identity, language, bounding boxes, point clouds, and normalization should be specified. Otherwise, comparisons may combine differences in available information with differences in policy design. The box conditions also need reconciliation: Table 5 is labeled “Oracle BBox” but reports 80.8% for Keyboard, while Table 6 reports 80.8% with predicted boxes and 88.2% with oracle boxes.

W3. The precision and long-horizon claims are stronger than the evidence supports.

Successful interaction with small targets is meaningful, but task-success thresholds do not directly quantify localization or control precision. For example, depressing the correct key by 5 mm measures successful actuation, not a 5 mm targeting error. The paper lacks sufficiently direct measurements of target-centroid error, contact-point error, or end-effector accuracy to substantiate the claimed fine-grained precision and its source.

The long-horizon results show improvement, but do not establish reliable execution. Table 7 reports seven-step success rates of 22.5% and 33.6%, versus 13.4% and 22.3% for the strongest baselines. These gains are useful, but most sequences still fail. Statements about eliminating atomic errors or resolving error compounding should therefore be substantially qualified. The expression p^T is an independence-based reference, not a generally justified conservative lower bound.

The execution protocol also requires clarification. The main text states that composition requires no LLM planning or online verification, while Appendix A.1 and Appendix D describe an LLM planner and verification procedures. These may be offline or evaluation-only components, but their roles in the reported experiments must be made explicit.

Overall, I lean toward weak rejection. The tasks and performance gains are encouraging, but the current evidence does not sufficiently isolate the proposed contribution or support the breadth of the precision and composability claims. Clearer protocols and focused matched-information controls could materially change this assessment.

Questions for authors

Q1. Contribution and improvement attribution — corresponding to W1.

What is the principal methodological contribution beyond combining task-conditioned grounding, feature modulation, target-relative representations, and a diffusion policy? Please explain how the evidence separates improved target selection from improved control.

In particular, can you provide task-specific grounding diagnostics, such as target-confusion rates or localization errors on Keyboard, that support the claimed fine-grained visual discrimination? Please clarify whether evaluation distinguishes recognition of target details from reliance on familiar layouts. A focused comparison of the proposed modulation against simple fusion under the same target-box conditions would help establish its specific contribution.

Q2. RGB ablation and input protocol — corresponding to W2.

Please specify exactly what is removed in the “without RGB” experiment, including whether the model is retrained, whether SQG still receives RGB, how its bounding boxes are obtained, and what generates the FiLM conditions after RGB removal.

Please provide a per-baseline input inventory covering target identity, language, bounding boxes, RGB, point clouds, and coordinate/action normalization, and reconcile the oracle/predicted-box conditions in Tables 5 and 6.

How much performance is achievable from the side branch alone, or from a simple target-box-conditioned 3D policy? A representative matched-information control would help determine whether the additional visual-fusion machinery provides a meaningful advantage beyond the supplied target information.

Q3. Precision and long-horizon interpretation — corresponding to W3.

Beyond binary task success, what measurements establish the claimed spatial precision? Please report available localization or execution-error statistics, together with evaluation counts and uncertainty, and distinguish perception errors from errors after correct target localization.

For the long-horizon experiments, please specify whether sub-goal lists are predetermined, whether the LLM or verifier influences switching, stopping, or retries, and how the 5,800 long-horizon trajectories are used relative to the claim of atomic-only training. Please also explain the lower-bound interpretation of p^T and qualify the reliability claims in light of the measured 22.5% and 33.6% full-sequence success rates.

Reviewer_iEbTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

RN-VLA combines language-conditioned target grounding, visual modulation of point-cloud features, target-centered observations, and a diffusion action policy for fine-grained manipulation. It reports seven simulated task categories, physical robot experiments, placement generalization, and sequential composition of atomic policies. The central claim is that reference-point normalization improves precise execution while semantic grounding identifies the intended target.

Critical strengths
  1. The paper addresses a relevant failure mode in fine-grained manipulation: selecting the right object is different from placing the end effector accurately. Its separate semantic and precision analyses are useful for understanding where a policy fails.
  2. The combination of language-conditioned grounding, visual point-cloud features, local coordinates, and diffusion control is a practical system design. The task set includes small targets and cluttered scenes that are more demanding than coarse pick-and-place demonstrations.
  3. The reported simulation and physical robot results suggest that the full system can outperform several baselines. Component, perception-noise, and failure analyses also provide potentially useful evidence about robustness, subject to clarifying the evaluation configurations.
Critical weaknesses
  1. The experimental protocol is difficult to reconcile across the manuscript. The main text says that the method excludes LLM planning, online verification, and long-horizon-specific training, while the appendices describe an LLM planner, GPT-4o verification, and long-horizon trajectories. It is unclear which components and data were used for each headline result.
  2. Several reported settings are insufficiently aligned for reliable comparison. Oracle and predicted bounding-box results carry different labels and success rates across evaluations, without a clear explanation of whether the task sets or protocols changed. Trial counts, evaluation splits, and baseline implementation details are also incomplete, limiting confidence in the size of the gains.
  3. The reference-point mechanism is not isolated from other normalization choices. The method recenters observations and state, but the actions are already displacement increments and are standardized separately. Absolute target-box information also enters the policy. Thus, translation invariance of the complete policy does not follow directly from recentering, and the ablations do not separate these effects.
  4. The generalization evidence relies heavily on oracle target boxes. Results under shifted or rotated placements do not establish how the complete system behaves when the proposed grounding module must predict the target in those settings. Direct measurements of contact or end-effector error would also better support the sub-centimeter claim.
Questions for authors
  1. Which experiments used an LLM planner, GPT-4o verification, retries, or the reported long-horizon trajectories? Please map the exact training and evaluation configuration to each main result and clarify whether the appendix describes a separate system.
  2. Please reconcile the oracle and predicted bounding-box labels and success rates across the placement and real-robot experiments. Were these different cohorts or protocols? Please report success counts, trial counts, split definitions, and baseline reproduction settings.
  3. With action standardization and all other components held fixed, what benefit comes specifically from reference-point recentering? How are action coordinates transformed, and what property of the complete policy is actually invariant?
  4. What is out-of-distribution performance with predicted target boxes rather than oracle boxes? Can the authors report grounding errors and physical target or end-effector error distributions for the sub-centimeter tasks?
Reviewer_NZ8ZOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes RN-VLA, which grounds task-specified targets via Semantic Query Grounding (SQG), suppresses background clutter through FiLM-based Cross-Modal Gated Modulation (CMGM), and re-centers the action space onto the target part via Reference Point Normalization (RPN), enabling a diffusion policy to achieve sub-centimeter precision. Experiments in Isaac Sim and on a physical xArm6 robot show RN-VLA achieves over 80% atomic success across seven household tasks, outperforming state-of-the-art VLA and 3D diffusion baselines, with strong robustness under out-of-distribution placements. By eliminating atomic-level errors, reliable single-step policies compose zero-shot into long-horizon tasks—without LLM planning or online verification—achieving up to a 68% relative improvement over the strongest baselines (up to 1.68× higher success rate).

Critical strengths

None

Critical weaknesses
  1. The manuscript has several issues with writing and referencing that affect its clarity and readability. For example, FVP-DP3 is presented as a key baseline for comparison, yet I could not find the corresponding reference in the paper. This omission makes it difficult to identify the original method and verify its implementation and reported performance. The authors should carefully check the manuscript to ensure that all cited methods are properly referenced and that the terminology and descriptions are presented consistently and unambiguously.

  2. The experimental results reported in Tables 2, 3, and 5 do not include comparisons with the latest methods published in 2026. Given the rapid progress in this research area, comparisons with relatively recent approaches may not be sufficient to establish the competitiveness of the proposed method. The authors should incorporate relevant state-of-the-art (SOTA) methods published in 2026 and provide a comprehensive comparison under consistent experimental settings. Without such comparisons, it is difficult to determine whether the proposed approach offers meaningful advantages over the current state of the art.

  3. The manuscript does not provide a comprehensive comparison of model complexity between the proposed method and existing SOTA approaches. In addition to task performance, computational efficiency and model complexity are important factors in assessing the practical applicability of the proposed approach. The authors should report relevant metrics, such as the number of parameters, computational cost (e.g., FLOPs), inference latency, and, where applicable, memory consumption. These results should be compared against competitive baselines under consistent evaluation settings to clarify the trade-offs between performance and computational overhead and to demonstrate whether the proposed method provides a favorable balance between accuracy and efficiency.

  4. The manuscript makes several specific claims regarding the functional benefits of its proposed components, but the current experimental results do not provide sufficient visual evidence to substantiate these claims. Specifically, the authors state that Semantic Query Grounding (SQG) can resolve instance-level ambiguity, Cross-Modal Gated Modulation (CMGM) can suppress geometric attention dispersion, and Reference Point Normalization (RPN) can decouple global reaching from local micro-manipulation. However, no qualitative visualizations are provided to clearly demonstrate these effects. The authors should include representative visual results, such as attention maps, intermediate feature visualizations, spatial grounding results, and manipulation trajectories, to illustrate how each component influences the model's behavior. Furthermore, comparisons between the full model and variants with individual components removed would help establish the specific contribution of each design. Such qualitative evidence, together with appropriate ablation studies, is necessary to verify the proposed mechanisms and strengthen the credibility of the paper's technical claims.

Questions for authors
  1. The manuscript has several issues with writing and referencing that affect its clarity and readability. For example, FVP-DP3 is presented as a key baseline for comparison, yet I could not find the corresponding reference in the paper. This omission makes it difficult to identify the original method and verify its implementation and reported performance. The authors should carefully check the manuscript to ensure that all cited methods are properly referenced and that the terminology and descriptions are presented consistently and unambiguously.

  2. The experimental results reported in Tables 2, 3, and 5 do not include comparisons with the latest methods published in 2026. Given the rapid progress in this research area, comparisons with relatively recent approaches may not be sufficient to establish the competitiveness of the proposed method. The authors should incorporate relevant state-of-the-art (SOTA) methods published in 2026 and provide a comprehensive comparison under consistent experimental settings. Without such comparisons, it is difficult to determine whether the proposed approach offers meaningful advantages over the current state of the art.

  3. The manuscript does not provide a comprehensive comparison of model complexity between the proposed method and existing SOTA approaches. In addition to task performance, computational efficiency and model complexity are important factors in assessing the practical applicability of the proposed approach. The authors should report relevant metrics, such as the number of parameters, computational cost (e.g., FLOPs), inference latency, and, where applicable, memory consumption. These results should be compared against competitive baselines under consistent evaluation settings to clarify the trade-offs between performance and computational overhead and to demonstrate whether the proposed method provides a favorable balance between accuracy and efficiency.

  4. The manuscript makes several specific claims regarding the functional benefits of its proposed components, but the current experimental results do not provide sufficient visual evidence to substantiate these claims. Specifically, the authors state that Semantic Query Grounding (SQG) can resolve instance-level ambiguity, Cross-Modal Gated Modulation (CMGM) can suppress geometric attention dispersion, and Reference Point Normalization (RPN) can decouple global reaching from local micro-manipulation. However, no qualitative visualizations are provided to clearly demonstrate these effects. The authors should include representative visual results, such as attention maps, intermediate feature visualizations, spatial grounding results, and manipulation trajectories, to illustrate how each component influences the model's behavior. Furthermore, comparisons between the full model and variants with individual components removed would help establish the specific contribution of each design. Such qualitative evidence, together with appropriate ablation studies, is necessary to verify the proposed mechanisms and strengthen the credibility of the paper's technical claims.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.