acceptodds
Under review as a conference paper at ICLR 2027

Peregrine: Latent-Future-Based Action Correction for Vision-Language-Action Policies in Contact-Rich Dynamic Manipulation

Abstract

Action chunks amortize the online inference cost of vision-language-action (VLA) policies by generating multistep action sequences, but robot and object motion can render later actions in a chunk stale before execution. We propose Peregrine, a lightweight framework that conditions action correction on latent representations predicted for the target execution time. A Dynamic Tokenizer learns structured historical dynamics through robot-motion and region supervision, encoding observed changes in the robot and objects into compact dynamics tokens. A Future Predictor combines these tokens with the latest observation, the action timeline, and temporal conditions to predict latent representations at the target action’s execution time. A Residual Corrector uses the execution-time prediction and the corresponding base action to generate a bounded residual correction, without requiring futureimage generation or complete action replanning. The pretrained flow-based VLA remains frozen. All auxiliary modules are trained offline, and their weights remain fixed during deployment, while predictions and corrections are updated as new observations arrive.We evaluate Peregrine in dynamic simulations and real-robot manipulation, assessing task success rates and future latent prediction accuracy, and examining component contributions through ablation studies.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Vision-Language-Action Models, Dynamic Robotic Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_AL6uOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Peregrine augments a frozen VLA with structured historical dynamics, execution-time latent prediction, and bounded residual action correction. Its central objective is to align the predicted pre-action state with the application time of a specific pending command. The paper presents evaluations across dynamic simulation conditions and Franka tasks.

Critical strengths
  1. The prediction horizon covers the complete observation-to-correction pipeline, including correction computation and command adoption.
  2. Compact historical tokens, joint visual–pose forecasting, and bounded correction provide a plausible way to improve responsiveness while preserving the frozen base policy.
Critical weaknesses
  1. The title shown in the PDF is inconsistent with the title entered in the submission system, and both titles are confusing to the reviewer.
  2. The motivation in the Introduction and the description of the proposed approach in the Method section are difficult to follow.
  3. The manuscript lacks sufficient technical novelty and contribution. The proposed approach is simple and intuitive, and appears closer to a training-free engineering refinement than a substantive methodological advance. The presentation of Figs. 1 and 2 also requires further clarification and improvement. For example, it is unclear where the “latest visual observation” and state information originate, and how they are obtained and updated within the framework.
  4. The simulation experiments are conducted only on Isaac Sim, which provides insufficient evidence to substantiate the effectiveness and generalizability of the proposed method.
  5. Essential details of the real-world experimental setup and qualitative visualizations of real-world task executions are missing.
Questions for authors

The authors should thoroughly address all concerns raised in the Weaknesses section.

Reviewer_AB5ROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper addresses action-time mismatch in VLA models by introducing action correction conditioned on latent representations of the state at the target execution time. A Dynamic Tokenizer encodes historical dynamics, and a Future Predictor predicts future latent representations and robot pose. A Residual Corrector then uses these predictions and the corresponding base VLA action to correct the action before execution. The modules are trained sequentially offline. The paper evaluates the method across controlled simulation delays, real-robot manipulation tasks, and component ablations.

Critical strengths
  1. The paper proposes an interesting approach that uses future latent representations for action correction without requiring future-image prediction.

  2. The evaluation design covers different settings, including controlled simulation delays and real-robot manipulation tasks.

Critical weaknesses
  1. The paper appears not to be ready for submission. Appendix B, p.19, lines 1008–1012, states: “The supplied values have not been independently verified as results of completed evaluations; their provenance remains to be confirmed.” Furthermore, Appendix B, p.20, lines 1031–1036, states: “The median placeholder values previously marked E have been removed from the two main tables and replaced with the supplied revised values. This replacement alone does not establish that methods not previously measured have now been evaluated.” These statements indicate that the provenance and completion status of the reported evaluations still require confirmation.

  2. The correspondence between the experimental results and task definitions remains unclear. Appendix B, p.19, lines 1019–1021, states that the meanings of “Volle and the final Ball column still require confirmation” before they can be associated with the task descriptions or images. This makes it difficult to determine which tasks the reported results actually evaluate.

  3. The training details are insufficiently specified, including the actual training configurations, datasets, and implementation details. In particular, the current corrector architecture, residual bounds, loss weights, and training budgets need clarification.

  4. The experiments do not clearly isolate the contributions of the model design. Although controlled latency experiments are relevant to action-execution mismatch, they do not establish the benefit under the actual inference and execution latencies of Peregrine and the VLA baselines. The fairness and practical relevance of the comparison therefore require further clarification. Additionally, Peregrine introduces auxiliary dynamics supervision and additional training. Comparisons that account for these differences would help determine whether the improvements come from execution-time prediction and correction or from the additional supervision and training.

  5. Peregrine requires multiple stages of offline training, but the associated cost and practical operating limits are insufficiently characterized. The paper reports moving-target tasks, but does not clearly quantify how performance varies with target speed or changes in speed. What happens when the target moves beyond the gripper’s reachable region or cannot be caught through a bounded correction? Although the paper discusses qualitative limitations, the operating limits and failure cases need stronger experimental characterization.

Questions for authors
  1. Which entries in Tables 1–4 were obtained from completed evaluations of the current method? The statements in Appendix B make their status unclear.

  2. Can you provide the actual training configurations, dataset descriptions, and implementation details for both Peregrine and the VLA baselines?

  3. Do the loss weights or auxiliary-module configurations need to change when switching VLA backbones or checkpoints? Which modules must be retrained?

  4. What are Peregrine’s measured end-to-end latency and effective correction frequency? Can you provide a comparison with VLA baselines on dynamic-object grasping under each method’s actual runtime conditions, with clearly specified target speeds and trial counts?

Reviewer_ifMJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses the problem of “action staleness” in action chunking VLAs: action chunking policies predicted a sequence of actions given the current observation; these actions are usually executed in an open loop manner, and external factors, such as changes in the environment and tracking errors end up making later actions in a chunk “stale” by the time they are executed. The paper proposes a method to generate residual corrections for actions in a chunk at execution time by conditioning on future state prediction provided by an ad-hoc model, the Future State Predictor, which the work introduces. The future state predictor conditions on the latest observations and latent history dynamics encoded by another ad-hoc model, the “Dynamics Tokenizer”. The proposed method is evaluated both in simulated and real-world robot manipulation environments.

Critical strengths

None

Critical weaknesses
  • Extremely poor presentation. The paper reads like an unfinished agent-written draft, with many artifacts left in the manuscript. Here are some examples:

    • Line 299: “[…] the exact GR00T release and checkpoint underlying the supplied results remain to be specified”
    • Line 317: “Existing simulation and Franka values are retained from supplied tables; previously estimated entries still require verification against trial records. Trial counts are not inferred from percentages. A dash marks a pending measurement”
    • Appendix B

    In general, many sections of the manuscript are rather cryptic and hard to understand. While AI assistance in the writing process is allowed and disclosed, the final result was clearly insufficiently reviewed by the authors and is, in my assessment, not fit for submission to a conference in the current state.

  • Lack of information on simulation benchmark. Evaluations are done on an unspecified simulation benchmark, with not enough information about the tasks and environments. The paper provides no explanation of why commonly evaluated simulation benchmarks, such as LIBERO and LIBERO+, for which the evaluated baselines report numbers, are not used instead. This makes it hard to get a sense of the genuinity of the reported results.

  • No comparison with other residual correction approaches. The work does not compare against methods that operate in the same regime, i.e. applying residual corrections to predicted action chunks between inference steps based on the latest observations. For example, A2C2, which is also mentioned in the related works section and shares many a few similarities, is not compared with.

  • Lack of information on training data and evaluation protocols. There is no mention of the training data used for policy training, such as number of demonstrations, mode of collection, and generation of labels for the region supervision. Key information about the evaluation environments and protocol, such as number of episodes per task, environment variations, robot setup details, is also absent.

Questions for authors
  • Clarify the experimental setup for the simulation experiments
  • Elaborate on the choice of the compared methods

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.