acceptodds
Withdrawn from ICLR 2027

Astra on Call: When and How GPT-6 Should Intervene in a VLA Policy

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A new generation of large models, exemplified by GPT-6, demonstrates strong multimodal understanding and embodied reasoning, enabling them to analyze robot behavior using visual scenes, task goals, and execution feedback. These capabilities offer opportunities to improve existing policies through intervention, but how to allocate control remains an open question: which actions should remain with the policy, when should the model intervene, and how should it make corrections? We investigate these questions in *Astra on Call*, an empirical study of how GPT-6 should share control with a frozen VLA policy, . Using the Inspect-EEF harness, we compare consultation schedules, optional versus required takeover, and corrections expressed as absolute end-effector targets or bounded waypoint edits. Our results show that selective GPT-6 intervention can improve task outcomes while leaving most control steps to the base policy. On RoboDojo, Fixed-15, which consults GPT-6 every 15 policy steps with optional takeover, raises the weak-set Score from 18.1 to 42.9 while matching the policy's aggregate strong-set Score of 76.7. With GPT-6 inference taking approximately 16s per call, Fixed-15 reduces wall-clock time per weak-set episode by 27.6% relative to full GPT-6 control. Our project webpage (https://astra-on-call-anonymous.pages.dev) and code (https://anonymous.4open.science/r/astra_on_call) are available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLA, test-time, GPT-6

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_hRntOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

Astra on Call presents a study which examines how to incorporate Astra with a frozen VLA policy. The authors conduct their study by exploring the effects of allowing Astra to (a) optionally take over, (b) decide how long to execute the frozen VLA’s inferred action chunk. They compare to policy-only baselines as well as a GPT-6 only baseline. They find that in their setting, consulting Astra every 15 steps works best, with performance degrading as they increase the consultation interval. A fixed schedule also outperforms allowing Astra to decide when it is next consulted. The authors also explore two different strategies of Astra takeover: bounded waypoint editing, which enforces a maximum deviation from the frozen VLA and segment takeover, which replaces the policy’s actions with absolute end-effector targets.

The authors evaluate in simulation using RoboDojo’s ARX X5 embodiment and use Codex on medium reasoning effort, pause the simulator during inference, and allow 100 consultations per episode. They also evaluate on the RoboLab benchmark. Tasks are split into a weak set and a strong set, which represent the sets for which the frozen policy is weak and strong, respectively.

Primarily, the authors answer five key questions.

  1. Can Astra improve the policy and maintain its strengths?
  2. When and how should Astra be consulted?
  3. How should Astra correct the policy?
  4. What are the costs of interfacing with Astra?
  5. Do the findings translate to other tasks and benchmarks?

In summary, the authors find that Astra can improve the frozen policy on the weak set while maintaining its strong-set performance, thereby by preserving the frozen policy’s successes. To achieve this, the authors find that Astra should be consulted on a fixed schedule with the option to let the frozen policy continue (Fixed-K), which outperforms forced takeover and a model-chosen schedule. When Astra does intervene, absolute end-effector targets generally outperform bounded waypoint editing, under which Astra tends to give up earlier. While this selective intervention is faster than the GPT-6 only baseline, each consultation costs around 16 seconds. Finally, the benefits of selective takeover hold on additional RoboDojo tasks and on RoboLab.

Critical strengths

The authors are fairly comprehensive in their justification of Fixed-K as a method within the regime of running a frozen policy and having Astra evaluate whether or not to intervene. Their ablations (policy only, Astra only, optional vs. forced, and dynamic choice of when to consult Astra) are fairly natural comparisons to make. Thus, in their setting, the choice of Fixed-K is rational.

Critical weaknesses

It is unclear to me how useful the findings presented in the paper are. The findings that are general, such as, for example, letting the policy continue when it is succeeding and intervening when it is not, are self-evident. Specifically, it is not surprising that Fixed-15’s performance over Always-Segment, comes from strong set performance where it is deferring to a well-trained model. Similarly, on the weak set, where the policy is expected to have worse contribution, the score/success of Always-Segment (37.7±6.0,17.1%±6.437.7 \pm 6.0, 17.1\% \pm 6.4) are within the margin of error of Fixed-15 (42.9±6.4,22.9%±7.142.9 \pm 6.4, 22.9\% \pm 7.1), as the performance gains come from Astra in either case. On the other hand, the specific findings, such as the 15 step consultation schedule or the takeover/editing comparison seem tied to factors such as the model, its latency, and the harness. For example, the optimal consultation interval likely would vary as the inference speed of Astra changes, and, as discussed below, the takeover/editing comparison seems tied to the design of the interface. Since the implicit assumption of this strategy is that eventually inference calls will increase in speed to be viable for robotics tasks, it could be called to question whether Fixed-15’s optimality is still maintained when this assumption holds true.

Interfacing with Astra raises questions of implementation. Specifically, the method imposes what are seemingly arbitrary constraints, such as a 5-waypoint summary of the proposal and ±0.05\pm 0.05 meter bounds on edits. It’s unclear whether the results are a reflection of the fundamental properties of Astra’s intervention or whether these are artifacts of prompting or engineering choices. Take the fact that Fixed-15 editing performs close to Fixed-15 takeover in RoboLab (76.5 vs. 80.8) according to Table 14, in contrast to RoboDojo, where editing does much worse (25.1 vs. 42.9). While the authors do not overclaim, this suggests that Section 4.3 results may depend on how well-engineered the solution was, which detracts from the results.

Questions for authors
  1. Does Fixed-KK monotonically improve performance as KK decreases on the weak set (e.g., is Fixed-1010 more performant)? Why was 15 the chosen minimum? Practically, 15 already seems infeasible in a real environment, so it seems an arbitrary choice.
  2. What prompts were used to conduct experiments? How were the tools, descriptions, and formatting passed to Astra? On a related note, why was Inspect-EEF chosen as the methodology of choice?
  3. How does Fixed-15 perform on completely out-of-distribution tasks? Does it perform similarly to Always-Segment?
Reviewer_9Sc5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Authors present a study on utilizing GPT 6 Astra along with a frozen VLA policy as controllers for robot tasks. On RoboDojo, authors use Pi-0.5 and Astra in various modes: (1) policy only, (2) fixed-interval consultations, (3) segment-level consultations and evaluate performance on a set of robot pick-and-place tasks.

The authors provide a comprehensive coverage of various strategies in how to use Astra to intervene in a VLA policy to improve task success rate. The experiments show a consistent story of Fixed-15 being the preferred tradeoff between a trained robot foundation model and Astra.

Critical strengths
  • Authors cover a substantially broad combinations of the base VLA policy and Astra, and demonstrated results provide useful insights into the most optimal configuration. The implemented Fixed-15 is a strong setup for robust pick-and-place task performance
  • The latency analysis provided through Table 4 is thorough and highlights the gaps in deploying various studied combinations of Astra and the VLA policy.
Critical weaknesses
  • My biggest concern is around the narrow setting of the deployed setup: the primary experiments are conducted on a single simulation - RoboDojo's ARX X5 robot with just Pi-0.5 as the VLA policy. Its not established that these results will hold across other VLAs. Atleast one more VLA policy on a smaller subset is necessary to understand if Astra is compatible with other models too
  • From a robotics standpoint the paper does not provide insights into how does astra actually improve the task success rate. From table 2 its clear that on weaker tasks (determined by the VLA policy), astra when used in Fixed-15 setup substantially improves the scores. But its not clear how the performance improved - did astra use a different strategy, change the manipulation pose, recover from offsets, etc.
Questions for authors
  • Can the authors include atleast one more VLA on a smaller subset of tasks? Without that its not possible to claim how GPT-6 should intervene in a generalist VLA policy.
  • Refer to Weakness #2: Please give more qualitative and quantitative metrics on how the motion of the robot changed with and without Astra's intervention.
  • Can the authors also provide a better failure analysis of the Fixed-15 policy? Its surprising to observe that its performance does not improve over Base-25 on the stronger set of RoboDojo tasks - does that imply the performance of these integrated systems are bounded by the performance of the VLA?
Reviewer_QucNOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper, Astra on Call proposes a training-free framework for calling a LLM agent during the rollout process of a pretrained robot policy. Instead of continuously allowing the agent to control the robot, the method proposes to periodically call GPT-6 Astra, which can decide if it’ s better to continue executing the base policy or intervene with corrective actions. The method is evaluated with π0.5 policy on simulated robotics benchmarks. The paper shows on RoboDojo benchmark weak-task set, the proposed fixed-15 controller improves the average score from 18.1 to 42.9 and the success rate from 5.7% to 22.9%, while maintaining the base policy's performance on the strong-task set.

Critical strengths
  1. Simple, training-free framework for augmenting pretrained VLA policies. The method requires neither retraining nor modifying the VLA, which makes it a practical way to improve an already-deployed policy with current LLM agents. Compared with giving GPT-6 full control, Fixed-15 uses fewer model calls while achieving higher success rates in the reported experiments.
  2. Preservation of already-correct VLA behavior. The results show that selective intervention outperforms giving GPT-6 full control: it corrects some failures while preserving more of the VLA’s existing successful behavior.
Critical weaknesses
  1. Limited model diversity. The evaluation currently only considers GPT-6 Astra supervising π0.5. Experiments with multiple base VLAs and LLM agents are needed to show that the gains come from the proposed strategy rather than these specific models.
  2. Lack of real-world experiments. All experiments are conducted in simulation, with no evaluation on physical robots. I am therefore unsure whether the approach remains effective under real-world perception, dynamics, and deployment errors.
  3. Lack of comparison with other fine-tuned VLAs The authors should evaluate other VLAs fine-tuned on the same benchmark, both with and without agent intervention. This would help show whether the strategy benefits different adapted policies rather than only π0.5.
  4. The evaluation is small. The main results use only five trials per task, with 35 episodes on the weak set and 15 on the strong set. Substantially more trials are needed to draw conclusions.
Questions for authors

Could the authors provide a broader evaluation to show that intervention strategy generalizes beyond GPT-6 Astra + π0.5? In particular, it would be important to compare at least 2–3 different LLM agents across multiple VLAs fine-tuned on the same benchmark, evaluating each VLA both with and without agent intervention. Given the robotics setting, I would also like to see evaluation on multiple real-world tasks with substantially more than five trials per task. These experiments would help distinguish whether the improvements come from the proposed intervention strategy, the strength of GPT-6, or weaknesses of the π0.5 policy, while also demonstrating whether the findings transfer to real-world robot execution.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.