- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Astra on Call presents a study which examines how to incorporate Astra with a frozen VLA policy. The authors conduct their study by exploring the effects of allowing Astra to (a) optionally take over, (b) decide how long to execute the frozen VLA’s inferred action chunk. They compare to policy-only baselines as well as a GPT-6 only baseline. They find that in their setting, consulting Astra every 15 steps works best, with performance degrading as they increase the consultation interval. A fixed schedule also outperforms allowing Astra to decide when it is next consulted. The authors also explore two different strategies of Astra takeover: bounded waypoint editing, which enforces a maximum deviation from the frozen VLA and segment takeover, which replaces the policy’s actions with absolute end-effector targets.
The authors evaluate in simulation using RoboDojo’s ARX X5 embodiment and use Codex on medium reasoning effort, pause the simulator during inference, and allow 100 consultations per episode. They also evaluate on the RoboLab benchmark. Tasks are split into a weak set and a strong set, which represent the sets for which the frozen policy is weak and strong, respectively.
Primarily, the authors answer five key questions.
- Can Astra improve the policy and maintain its strengths?
- When and how should Astra be consulted?
- How should Astra correct the policy?
- What are the costs of interfacing with Astra?
- Do the findings translate to other tasks and benchmarks?
In summary, the authors find that Astra can improve the frozen policy on the weak set while maintaining its strong-set performance, thereby by preserving the frozen policy’s successes. To achieve this, the authors find that Astra should be consulted on a fixed schedule with the option to let the frozen policy continue (Fixed-K), which outperforms forced takeover and a model-chosen schedule. When Astra does intervene, absolute end-effector targets generally outperform bounded waypoint editing, under which Astra tends to give up earlier. While this selective intervention is faster than the GPT-6 only baseline, each consultation costs around 16 seconds. Finally, the benefits of selective takeover hold on additional RoboDojo tasks and on RoboLab.
Critical strengths
The authors are fairly comprehensive in their justification of Fixed-K as a method within the regime of running a frozen policy and having Astra evaluate whether or not to intervene. Their ablations (policy only, Astra only, optional vs. forced, and dynamic choice of when to consult Astra) are fairly natural comparisons to make. Thus, in their setting, the choice of Fixed-K is rational.
Critical weaknesses
It is unclear to me how useful the findings presented in the paper are. The findings that are general, such as, for example, letting the policy continue when it is succeeding and intervening when it is not, are self-evident. Specifically, it is not surprising that Fixed-15’s performance over Always-Segment, comes from strong set performance where it is deferring to a well-trained model. Similarly, on the weak set, where the policy is expected to have worse contribution, the score/success of Always-Segment () are within the margin of error of Fixed-15 (), as the performance gains come from Astra in either case. On the other hand, the specific findings, such as the 15 step consultation schedule or the takeover/editing comparison seem tied to factors such as the model, its latency, and the harness. For example, the optimal consultation interval likely would vary as the inference speed of Astra changes, and, as discussed below, the takeover/editing comparison seems tied to the design of the interface. Since the implicit assumption of this strategy is that eventually inference calls will increase in speed to be viable for robotics tasks, it could be called to question whether Fixed-15’s optimality is still maintained when this assumption holds true.
Interfacing with Astra raises questions of implementation. Specifically, the method imposes what are seemingly arbitrary constraints, such as a 5-waypoint summary of the proposal and meter bounds on edits. It’s unclear whether the results are a reflection of the fundamental properties of Astra’s intervention or whether these are artifacts of prompting or engineering choices. Take the fact that Fixed-15 editing performs close to Fixed-15 takeover in RoboLab (76.5 vs. 80.8) according to Table 14, in contrast to RoboDojo, where editing does much worse (25.1 vs. 42.9). While the authors do not overclaim, this suggests that Section 4.3 results may depend on how well-engineered the solution was, which detracts from the results.
Questions for authors
- Does Fixed- monotonically improve performance as decreases on the weak set (e.g., is Fixed- more performant)? Why was 15 the chosen minimum? Practically, 15 already seems infeasible in a real environment, so it seems an arbitrary choice.
- What prompts were used to conduct experiments? How were the tools, descriptions, and formatting passed to Astra? On a related note, why was Inspect-EEF chosen as the methodology of choice?
- How does Fixed-15 perform on completely out-of-distribution tasks? Does it perform similarly to Always-Segment?