acceptodds
Under review as a conference paper at ICLR 2027

MetricFly: Numerical Instruction Following via Action-Conditioned Calibration for UAV Vision-Language-Action Models

Abstract

Aerial vision-language-action (VLA) policies can generate task-relevant UAV motion, but often fail to meet numerical requirements such as a specified orbit radius or flight altitude. This paper studies how to improve numerical execution by exposing the current metric error to the policy and calibrating the motion it actually generates. We introduce MetricFly, a two-stage framework that first conditions a policy on the target, current task metric, and signed residual, then uses a lightweight external calibrator to apply bounded geometric corrections to the sampled action chunk. The calibrator is trained with supervision from an analytic geometric teacher while the fine-tuned policy remains frozen during calibrator training. We evaluate MetricFly on HUGE-Bench-derived Orbit-R and Orbit-H tasks with 3DGS-rendered visual feedback and externally supplied reference geometry. Explicit numerical conditioning reduces trajectory task-metric MAE by and , respectively, compared with text-only conditioning. Adding action-conditioned calibration raises full-orbit completion from to over the metric-conditioned policy alone. Replacing the calibrator's input with a mismatched action sample reduces calibration precision, showing the value of conditioning corrections on the motion being executed.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Vision-Language-Action Models, Robot Learning, Numerical Instruction Following, Action-Conditioned Trajectory Calibration, UAV Navigation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JdUdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies numerical instruction following for UAV vision-language-action models, where policies can execute semantic tasks but often fail to regulate physical quantities specified in instructions. The authors introduce METRICFLY, a two-stage framework that first provides explicit target-state-residual numerical conditioning to a π0-based policy and then applies an external action-conditioned geometric calibrator to refine generated action chunks. Experiments on controlled Orbit-R and Orbit-H tasks show that numerical conditioning improves metric accuracy and action-conditioned calibration substantially improves full-orbit completion under closed-loop evaluation.

Critical strengths
  1. The formulation of numerical instruction following is a meaningful contribution.

  2. The target–state–residual interface is supported by carefully controlled ablations.

  3. The action-conditioned calibration analysis is one of the strongest parts of the paper.

Critical weaknesses
  1. The contribution of the learned calibrator versus the underlying geometric controller remains difficult to separate. Since the calibrator is supervised by an analytic geometric teacher and applies bounded geometric corrections, a stronger comparison against simpler analytical correction rules or lightweight non-learning controllers would help clarify what is learned by METRICFLY.

  2. The current evaluation focuses on Orbit-R and Orbit-H, where the desired behavior can be described using relatively simple geometric constraints. It would be valuable to evaluate whether the target-state-residual interface extends to other numerical instructions, such as distance-to-object, approach angle, or multi-constraint manipulation tasks.

  3. The paper argues that observing the sampled action chunk is important for calibration, but the ablation suggests that some compressed representations (e.g., first action or shuffled chunk) already capture much of the benefit. A deeper analysis of which information in the action sequence is actually necessary would make the action-conditioned design more convincing.

  4. Minor issue:

  • Some terminology, such as "calibration" and "correction", overlaps with classical control terminology; a clearer discussion of the relationship to residual control or model-based correction would improve positioning.
Questions for authors

See weakness

Reviewer_4KExOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies numerical instruction execution for UAV vision-language-action policies, focusing on orbit radius and reference-relative altitude. MetricFly first fine-tunes a HUGE-Bench π₀ policy with structured target, current-metric, and signed-residual inputs. It then freezes the policy and trains an action-conditioned MLP to predict bounded geometric corrections using an analytic teacher. Evaluation uses a landmark-disjoint dataset with 3DGS visual feedback, externally supplied geometry, and additive world-frame kinematics. Numerical conditioning reduces short-horizon task-metric MAE by 12.1% and 19.1%, while calibration increases nominal full-lap success from 2.78% to 93.33%. Matched-action conditioning improves numerical precision, although mismatched or shuffled inputs sometimes achieve higher completion rates.

Critical strengths
  1. The paper distinguishes numerical precision from complete maneuver execution. Same-state target interventions and separate maintenance and signed-lap metrics provide a more informative evaluation than a single completion score. The paper openly reports cases where better precision does not yield higher completion.

  2. The evaluation includes meaningful controls. Condition-only calibration, same-context action mismatch, alternative representations, channel masks, and an analytic controller using the same direction-estimation rule strengthen the evidence beyond comparison with an uncalibrated policy. Landmark-disjoint splits further support the numerical-precision findings within the defined setting.

Critical weaknesses
  1. The methodological contribution is small. Conditioning residual corrections on the base action has already been studied in Policy Decorator. Here, the teacher’s correction target explicitly depends on the sampled action, so the benefit of providing that action is expected. The experiments demonstrate a useful combination of existing ideas, but offer limited new insight into numerical instruction execution.

  2. The reported gains have limited support for actual UAV execution. There are no real-world UAV experiments, and the evaluator directly applies requested displacements without modeling inertia, aerodynamics, or actuator dynamics. A corrected trajectory can therefore score well without demonstrating that a drone could actually track it. The 93.33% completion rate shows success in this simplified setting, but leaves numerical accuracy during physical flight unverified.

Questions for authors

N/A

Reviewer_G6egOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how aerial VLA policies follow numerical parts of an instruction, such as an orbit radius or a relative altitude. It combines a π0\pi_0 policy fine-tuned with numerical target, current task metric, and signed-residual inputs with an external calibrator trained to predict bounded corrections from an analytic geometric teacher.

Experiments use HUGE-Bench-derived orbit tasks with 3DGS-rendered visual feedback, kinematic execution, and externally supplied reference geometry. This paper reports improved numerical accuracy from explicit conditioning and substantially higher full-orbit completion after calibration. Matched–mismatched action interventions and supplementary ablations investigate the contribution of action information, correction channels, and numerical-target generalization.

Critical strengths

1.The diagnostic is useful. The RQ1 intervention holds observation, state and semantics fixed and varies only the commanded value. It shows two separate failure modes: near-zero response (TinyVLA and Octo at 100% near-zero rate) and wrong-direction response (π0\pi_0 at 54–68% on the orbit tasks). This is a clean, reusable probe that the VLA community would benefit from.

2.The evaluation design is careful in several ways. It uses landmark-disjoint splits, a confirmation split frozen until model selection is done, paired action-sampling noise, cluster bootstraps over landmarks, and a clear split between precision metrics (Emaint,qE_{maint,q}) and completion metrics (MlapjointM_{lap}^{joint}).

3.Negative and nuanced results are reported honestly. Examples:

  • A-Mismatch has a higher M_lap than A (95.56% vs 93.33%, Table 2) and is higher in 9 of 11 perturbation conditions (Table 23).
  • A shuffled chunk matches or beats the ordered chunk on completion (93.33% vs 92.78%, Table 11).
  • The tangential channel provides no additional observed completion benefit (Table 13).
  • The four-landmark intervals are labelled "descriptive."
Critical weaknesses

1.The calibrator mostly overrides the VLA, so it is unclear what the VLA contributes to the numerical part of the task

The analytic teacher (App. E.1) builds each correction to cancel the base action's error exactly:

  • Vertical (Eq. 40): so the next height equals z∗z* exactly.
  • Radial (Eqs. 38–39): the smallest root that puts the next-step radius exactly on r∗r*.
  • Tangential (Eq. 37): vτ∗−Πxy(aS)⊤uτv_\tau^*-\Pi_{xy}(a^S)^\top u^\tau inside the band, which replaces the base tangential speed with a fixed 1 m/step; outside the band it cancels the base tangential motion.
  • Yaw (Eq. 41): sets the heading to the reference tangent.

The per-step bound is 1 m per channel, the same size as the nominal motion. Table 19/20 shows that doubling the chunk budget changes almost nothing (Δ ≈ 0.000–0.006 m), which suggests the chunk budget seldom binds. So whenever the bounds don't bind, the executed action of ANALYTIC is the output of a hand-designed controller given ground-truth geometry, and the sampled VLA action has no influence on it. This matches ANALYTIC (oracle) reaching 100% MlapM_{lap} with EmaintE_{maint} < 0.001 m (Table 2).

2.The scope is narrow and the setting is strongly idealised

  • Two closely related tasks. Only Orbit-R and Orbit-H are studied, each with one varied metric. The teacher, the radial/tangential basis and ActionDir are all orbit-specific. Landing appears in RQ1 but not in the method. It is unclear how MetricFly would handle other numerical instructions (fly X m, turn Y°, hold speed Z) without designing a new analytic teacher for each.
  • Ground-truth geometry. The reference point PrP_r is externally supplied ground truth, and both r_t and h_t are computed from it. In this setting the numerical-regulation problem is largely solved by geometry, as ANALYTIC = 100% shows. The perception problem that makes this hard for a VLA, estimating the orbit centre from images, is deferred to future work.
  • Idealised execution and evaluation. Execution is additive kinematic replay, with no dynamics, latency (beyond the one-step-delay perturbation) or collisions. Initial offsets are only ±4 m. Evaluation covers 4 test landmarks, which the authors themselves describe as giving descriptive rather than inferential intervals.

3.The precision gains are small next to the success tolerances, and completion results do not favour A

Within the learned variants, A's advantage is mostly centimetre-level precision under a 1 m success band and a kinematic simulator. For example, Emaint,HE_{maint,H} is 0.028 for A vs 0.032 for A-Mismatch (Table 2), and the ordered-vs-shuffled difference is about 0.02 m (App. G.3). On completion:

  • A-Mismatch ≥ A in Table 2 and in 9 of 11 perturbations;
  • A ≈ A-RH+YAW (Δ = 0.56 pp, CI [−1.11, 2.22]);
  • shuffled ≈ ordered.

The paper acknowledges this, but the abstract and contribution list still present action conditioning as the key ingredient. The practical significance of a few centimetres in this setting should be argued, not assumed.

4.Undefined baselines, promised results that never appear, and inconsistent numbers

  • Undefined baselines. Fig. 3(b) shows "Fixed-Gain" and "Error-only affine" baselines that are not defined or discussed anywhere in the paper.
  • Results stated but not reported: G.1 says official HUGE-Bench metrics (TCR, nDTW, NTP, SR) are "additionally reported"; they are not
  • Unspecified thresholds. The tolerances ε_ρ, ε_a and τ_q used in RQ1 and App. C are "pre-specified" but never given. For RQ1, the number of matched pairs per task and the fine-tuning status of each baseline VLA are also not stated. The Landing percentages (12%, 25%, 75%) suggest about 8 pairs.
  • The same method gives different offline H24 numbers in different tables. A's radius MAE is 1.192 (Table 10), 0.961 (Table 16) and 1.057 (Table 21). STATE and ANALYTIC are identical in Tables 16 and 21, which suggests the learned rows differ by direction source (expert vs ActionDir), but this is never stated.
  • The expert-referenced metric conflicts across protocols. STATE has the lowest expert-referenced XYZ RMSE in Tables 16 and 21 (2.564 m, vs 3.152–4.146 m for every calibrated method), while RQ3 uses an expert-referenced metric, Etraj3DE_{traj3D}, as its main precision evidence and there Base is worst. Because the teacher is not the expert, an expert-referenced error is a questionable primary metric for a calibrator trained to match the teacher.
  • Leftovers from revision. Table 15 duplicates Table 13.
Questions for authors

1.What fraction of executed displacement comes from the calibrator versus the base chunk, per channel? How often do the step and chunk budgets bind in closed loop? How does A (or ANALYTIC-AD) perform on top of a trivial or non-fine-tuned base policy?

2.Why build a UAV control method on π0\pi_0 rather than on a UAV-specific vision-language navigation or action model? To my knowledge, π0\pi_0's pretraining data is mostly manipulation and includes no aerial data, so its UAV behaviour comes only from the HUGE-Bench fine-tuning. Several UAV-native models are cited in Sec. 2. Was π0 chosen mainly because HUGE-Bench released a checkpoint, or because the calibrator needs continuous action chunks? Would the RQ1 failures and MetricFly's gains still hold with a base model pretrained on aerial data?

3.In RQ1, were all five VLAs fine-tuned on HUGE-Bench UAV data?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.