acceptodds
Under review as a conference paper at ICLR 2027

Elastic Queries Reinforcement Learning: Self-Aware Policy Execution for VLA Models

Abstract

A generative policy acting in a closed loop must decide what action to produce, how much inference to spend producing it, and when to observe again. Flow-based vision-language-action (VLA) policies typically fix the latter two choices across an entire task, even as the value of computation and feedback changes across states. We formulate each policy query as a variable-duration decision over latent input, denoising steps, and execution length, and introduce Elastic Queries Reinforcement Learning (EQRL) to learn these choices while keeping the VLA frozen. A lightweight actor jointly learns latent steering and two schedule residuals around a difficulty-guided reference, using duration-aware query-level RL with a number-of-function-evaluations (NFE) objective. A local compute–feedback analysis distinguishes schedules with equal amortized inference cost. On the long-horizon LIBERO-10 task of putting both moka pots on the stove, EQRL raises five-seed success from 0.411 to 0.861 while reducing requested NFE from 0.500 to 0.441. On four LIBERO-90 tasks and ALOHA Cube Transfer, EQRL improves aggregate success and learning AUC while reducing requested NFE by 39.4% and 14.0% relative to DSRL. Scheduling ablations support the contributions of joint control and difficulty guidance, and physical-robot evaluations match or exceed DSRL's success while reducing inference cost and execution time across five tasks.

Primary area:
applications to robotics, autonomy, planning
Keywords:
robot learning, vision-language-action models, reinforcement learning, adaptive inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_WS92OpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EQRL, which uses a lightweight RL adaptor to select the number of denoising steps K and the execution chunk length C for a frozen flow-based VLA model. The authors evaluate EQRL in both simulation and real-world experiments, comparing it with other RL adaptation methods like DSRL.

Critical strengths
  1. The motivation is clear. The authors investigate how to adapt the number of denoising steps and the execution chunk length at inference time while keeping the VLA frozen, motivated by variation in difficulty across different stages of task execution. Fig 4 provides an illustration consistent with this motivation.

  2. The evaluation demonstrate that the method can be deployed in real-world settings.

Critical weaknesses

However, I still have the following concerns:

  1. The presentation of experimental results does not adequately convey uncertainty. Fig 3 shows learning curves for six simulation tasks. The first five report the run with the highest full-training AUC among five seeds, whereas the last one shows seed 1 with a trailing average over five evaluations. However, reporting the mean success rate across all five seeds at each evaluation checkpoint, with a shaded band indicating ±1 standard deviation, would better convey typical learning behavior and variability across seeds. In addition, with only 20 trials per method and task, the real-world evaluations provide limited evidence for differences in success rate between DSRL and EQRL.

  2. Limited comparison with and discussion of related methods. The main RL baselines in this paper are DSRL and RLinf-SDE/Noise. However, some relevant methods are not included in the experimental comparisons. RLT[1] and the first stage of PLD [2] also freezes the VLA backbone, and training a small adaptorusing RL. AAC [3] adapts the execution chunk length at inference time in a training-free method, addressing a closely related aspect of the problem studied in this paper. The authors should clarify how EQRL differs from these methods and discuss the relevance of experimental comparisons.

[1] Charles Xu et al. RL Token: Bootstrapping Online RL with Vision-Language-Action Models. arXiv:2604.23073, 2026. https://arxiv.org/abs/2604.23073

[2] Wenli Xiao et al. Self-Improving Vision-Language-Action Models with Data Generation via Residual RL. arXiv:2511.00091, 2025. https://arxiv.org/abs/2511.00091

[3] Yuanchang Liang et al. Adaptive Action Chunking at Inference-time for Vision-Language-Action Models. CVPR, 2026. https://arxiv.org/abs/2604.04161

Questions for authors
  1. The reward formulation is complex, and the real-world experiments use different training objectives from those used in simulation. This raises a question about the manual design and tuning effort required to apply the method to a new setting. Could the authors describe the manual design and tuning effort involved, and quantify it where possible?

  2. How long does online training on the physical robot take?

  3. Have the authors evaluated the method on tasks requiring finer manipulation precision?

6 comments
AuthorsOpenReview ↗

Response to Reviewer WS92 (Part 1/2): New five-seed results and robot uncertainty

Thank you for your careful review. We address your comments in 2 parts.

W1. Five-seed statistics and real-robot uncertainty

We ran a new set of experiments on all four tasks, with five seeds per method. The table below reports mean success and standard deviation at 50k, 100k, 250k, and 500k SAC updates. Means are percentages and SDs are percentage points, showing the variability across seeds.

Table 1 | New four-task, five-seed checkpoint statistics, also reported in our response to rFA8 W3

TaskMethod50k100k250k500k
Black BowlDSRL72.46 ± 5.7487.60 ± 10.4192.43 ± 8.1898.77 ± 1.68
Black BowlEQRL25.83 ± 7.5583.34 ± 18.7296.36 ± 6.30100.00 ± 0.00
Alphabet Soup to TrayDSRL84.83 ± 4.8692.65 ± 8.2895.40 ± 7.5195.73 ± 6.43
Alphabet Soup to TrayEQRL70.99 ± 8.6496.76 ± 5.5893.56 ± 9.8397.86 ± 3.71
Cream Cheese to TrayDSRL65.55 ± 5.9596.80 ± 5.4693.49 ± 8.4393.22 ± 7.71
Cream Cheese to TrayEQRL93.71 ± 4.0297.06 ± 4.8396.00 ± 6.6094.63 ± 7.00
Tomato Sauce to TrayDSRL63.49 ± 6.6582.47 ± 13.0886.82 ± 10.9494.97 ± 6.93
Tomato Sauce to TrayEQRL92.90 ± 4.3098.94 ± 1.7090.02 ± 12.3393.85 ± 8.16

At 500k updates, both methods achieve mean success above 93% on all four tasks. EQRL has higher mean success on three tasks and is about one percentage point lower on Tomato Sauce. The per-task, per-checkpoint SDs make the seed-to-seed variability explicit.

Robot evaluation covers five tasks with 20 complete trials per method and task, totaling 200 trials. Failure at any stage counts as failure of the entire trial. Nominal 95% Wilson intervals computed from the observed success counts are:

TaskDSRL successes; rate and 95% CIEQRL successes; rate and 95% CI
Bimanual pouring11/20; 55%, 34.2–74.2%15/20; 75%, 53.1–88.8%
Fruit into bowl17/20; 85%, 64.0–94.8%17/20; 85%, 64.0–94.8%
Cucumber handover into bowl18/20; 90%, 69.9–97.2%19/20; 95%, 76.4–99.1%
Lemon into bowl17/20; 85%, 64.0–94.8%18/20; 90%, 69.9–97.2%
Cup-to-Bowl and Bowl Lift18/20; 90%, 69.9–97.2%19/20; 95%, 76.4–99.1%

We provide the success counts and 95% Wilson confidence intervals for each task to show the estimation uncertainty with a limited number of trials. In this evaluation, EQRL's success rate is higher than DSRL's on four tasks and tied on one, while requested NFE is 31–34% lower and mean execution time is 11–23% shorter on all five tasks (Table 5 of the paper). These experiments therefore demonstrate lower computation cost and faster execution alongside comparable or higher observed success rates.

AuthorsOpenReview ↗

Response to Reviewer WS92 (Part 2/2): Related work, tuning, time, and precision

Response to Reviewer WS92, Part 2/2.

W2. Relation to RLT, PLD, and AAC

EQRL focuses on jointly learning w,K,Cw,K,C for a frozen VLA: latent behavior adjustment, denoising computation, and feedback timing. Related methods overlap with different parts of this design.

RLT first adapts an RL-token representation, then freezes the VLA and that representation during online RL, using a lightweight actor–critic to adjust action chunks while regularizing toward the VLA reference. PLD first learns an action-space residual with a frozen VLA, then collects data for distillation into the base policy. AAC selects an execution prefix using action entropy and motion magnitude across candidate chunks, without additional scheduler training. EQRL instead learns latent steering, denoising steps, and execution length jointly before invoking the frozen generator.

DSRL shares EQRL's frozen generator and latent-adaptation interface and differs in whether K/C scheduling is learned, so it is the direct baseline for testing our contribution. RLT and the first stage of PLD change how actions are adapted, and AAC adjusts only the execution length; none of them learns K and C jointly, so they are complementary to EQRL's schedule learning and we did not run protocol-matched comparisons. Within our framework, the C-only variant, which learns only the execution length, needs requested NFE 0.41, whereas full EQRL uses 0.32.

Q1. Reward design and task-specific tuning

New tasks reuse the same core training hyperparameters and need no manual reward design: the task reward is 1 for success and 0 otherwise, without separate dense rewards for grasping, moving, and placing. The schedule-related settings (K/C ranges, the NFE band, and the difficulty-related weights) give similar success across their basic range; about 3–5 runs within a small range are enough to reach the best result.

Simulation uses an episode-level compute regularizer. Online robot training uses the query-local cost Rj=rjtask−λstepLj−λNFEKjR_j=r_j^{\rm task}-\lambda_{\rm step}L_j-\lambda_{\rm NFE}K_j when the efficiency term is enabled, where L counts executed actions. Offline training lacks interaction feedback for alternative K/C choices, so it uses expert-action matching for scheduling supervision together with compute regularization. These differences reflect the data and feedback available in each setting.

The four LIBERO-90 tasks share batch size 256, discount 0.999, 10 critics, and actor/critic/temperature learning rates 10−4/3×10−4/3×10−410^{-4}/3\times10^{-4}/3\times10^{-4}. The settings chosen per task are the K/C ranges, the NFE band and its weights, and the difficulty weight and prior scale (Table 7 of the paper). The additional three-seed sensitivity study supports this: when the alignment and cost weights vary between 0.5× and 2×, success stays between 78.67% and 87.33%, within one SD of the default.

Q2. Online robot training time

Online robot training takes approximately 1–2 hours, including robot interaction, model updates, and routine resets, but excluding collection of the initial 20 demonstrations. The complete online stage uses approximately 15k executed robot actions and 500k SAC gradient updates, with 1,000 updates after each collection round. This is the total online-training time, distinct from the final policy's approximately 11-second trajectory execution time.

Q3. Higher-precision tasks

We have not evaluated tight-tolerance insertion or screw assembly, or measured millimeter-level pose error.

The current robot tasks do include precision-demanding stages: bottle–cup alignment for bimanual pouring, bimanual cucumber handover, and placing a cup into a bowl before lifting the bowl; a trial succeeds only when all stages are completed. EQRL's largest gain is on Bimanual Coke Pouring, which requires bottle–cup alignment (15/20 versus 11/20), and its requested NFE peaks near alignment and grasping (Figure 6 of the paper), so elastic scheduling spends more computation and feedback where precision is needed.

Thank you for the response. It partially addresses my concerns, but I think a few points still need clarification. I have not seen an uploaded revision, so the comments below are based on your response and the original PDF.

W1 — Simulation. Thank you for providing the new five-seed results on these four tasks. My main concern, however, was about how the results were presented. The "Evaluation and metrics" paragraph in Section 4.1 already states: "Unless otherwise stated, simulation experiments use five random seeds." I was therefore asking for Figure 3 to show the mean and standard deviation across those existing runs. I'm not require new training experiments, if the evaluation logs are available.

W1 — Real-world evaluation. I may not have expressed this clearly enough. By "provide limited evidence," I meant that the number of trials makes it difficult to establish a success-rate advantage. I'm not asking for CIs. Apart from the pouring task, the differences are either zero or five percentage points, corresponding to at most one successful trial out of twenty. More trials—for example, 40–60 per method and task—would help improve the precision of the estimates. Given the remaining time, I am not asking you to add these experiments now. I am comfortable with the more cautious description in your response, "comparable or higher observed success rates," and would appreciate having this qualification reflected in the paper.

W2 — Related methods and comparisons. I would also like to clarify my point here. I agree that DSRL is the most directly matched baseline for isolating the contribution of learned scheduling. However, I do not think this removes the value of discussing or comparing other closely related approaches. In particular, I still think a comparison with AAC would be useful. It directly addresses the choice of execution length C without additional scheduler training. The C-only variant in EQRL is not equivalent to AAC.

More broadly, the introduction moves from learning a lightweight latent-space policy for a frozen VLA to jointly learning (w,K,C)(w,K,C), with the observation that latent steering alone leaves the execution schedule fixed. This explains what EQRL adds, but does not by itself establish why the additional complexity is worthwhile. If a simpler adaptation or scheduling method already performs well, what additional benefit does joint learning provide?

For PLD and RLT, my concern is also about positioning the work. If these approaches are complementary to EQRL, as you suggest, it would help to explain more clearly how their action-adaptation mechanisms relate to the latent steering and scheduling used here, and whether these components could be combined. (Since these methods involve different adaptation mechanisms and training objectives, it is not immediately clear to me how they could be combined.) Most importantly, these closely related methods should be adequately discussed in the introduction or related work. PLD is cited in the original paper, but its differences from latent steering are not explained clearly. I could not find a corresponding discussion of RLT or AAC. If there is insufficient time to add experiments comparing EQRL with PLD, RLT, or other lightweight VLA adaptation methods, please at least discuss these methods in the introduction or related work and incorporate the relevant explanations into the paper.

Q1-3. These responses largely resolve my questions. Please include the training-time information and its accounting scope in the paper or appendix.

Thank you again for the clarifications and additional results. The points above are my remaining concerns. I see value in the problem addressed by the paper, and a convincing resolution of these concerns could lead me to raise my score. For now, I am keeping my current score.

Reviewer_obfLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Elastic Queries Reinforcement Learning (EQRL), which adapts a frozen flow-based VLA by jointly selecting latent steering, the denoising-step count K, and the execution-prefix length C. Separate learned schedule residuals modify a critic-disagreement-based reference schedule. Query-level learning uses duration-dependent bootstrapping, while a requested-NFE regularizer and an allocation-alignment loss encourage efficient execution. Experiments on six simulation tasks and five real-world manipulation tasks demonstrate the effectiveness and efficiency of EQRL.

Critical strengths

The paper addresses the important problem of adapting inference computation and feedback frequency to varying task demands. This is particularly relevant to closed-loop robotic manipulation, where fixed inference and execution schedules may limit both efficiency and performance. The proposed EQRL framework jointly optimizes denoising steps and action execution length through query-level reinforcement learning while keeping the pretrained VLA frozen. The compute-feedback analysis provides useful insights into the trade-off between inference cost and feedback frequency.

Critical weaknesses
  1. The evidence for adaptive scheduling is thin. The central claim is that choosing K and C per state beats a fixed schedule at the same compute. Yet the main ablations use a single task, the equal-compute comparison includes only one fixed schedule, and the paper does not say why these four LIBERO-90 tasks were chosen. Fixed combinations of K and C with the same compute can behave quite differently, so the paper needs a comparison against several of them on more tasks. This is the main reason for my score.
  2. The method is hard to follow, and its parts are hard to separate. The reference schedule, the learned residuals, the gates, the compute band and the alignment loss all interact, and the paper never shows which of them matter. There is no sensitivity study of the main loss weights either. The abstract puts the problem formulation and the learning method into one long sentence, which hides the contribution.
Questions for authors
  1. Learning the denoising budget with RL and adapting the chunk length have both been studied before, for example in D3P. Which part of the joint formulation do you see as new, and how does EQRL compare with these approaches?
  2. How do K and C change over the course of a task? Do they match the intuition in Figure 1, with more compute and more frequent feedback near contact?
  3. Does a schedule learned on one task transfer to another, or does it have to be relearned for every task?
  4. How stable is online learning on the real robot, and how much robot interaction does it take before the learned schedule starts to help?
3 comments
AuthorsOpenReview ↗

Response to Reviewer obfL (Part 1/2): Fixed-schedule controls, ablations, and sensitivity

Thank you for your careful review. We address your comments in 2 parts.

W1. Fixed-schedule controls and task selection

Cream Cheese to Tray follows the task evaluated by DSRL. The other three LIBERO-90 tasks were randomly selected. We added Moka to test longer-horizon manipulation requiring consecutive placement of two pots.

We added a fixed-K/C comparison on Moka with five seeds per configuration. It includes adaptive EQRL, four fixed schedules near its requested NFE, and two fixed schedules with requested NFE 0.5:

Table 2 | Additional fixed-K/C comparison on Moka

MethodFixed K/CSuccess, mean ± SDRequested NFEExecuted NFE
Full EQRLAdaptive86.34% ± 1.73 pp0.43750.4481
Fixed8 / 1846.41% ± 8.85 pp0.44440.4488
Fixed9 / 2038.33% ± 7.15 pp0.45000.4607
Fixed10 / 2244.24% ± 8.78 pp0.45450.4630
Fixed11 / 2441.29% ± 8.56 pp0.45830.4698
Fixed20 / 4032.36% ± 7.54 pp0.50000.5052
Standard DSRL10 / 2042.78% ± 8.45 pp0.50000.5110

Fixed (K,C)=(10,20) and (20,40) have the same requested NFE but success rates of 42.78% and 32.36%, respectively, showing that equal NFE can still yield different performance.

All six fixed schedules reach 32–46% success, and none approaches the 86% of adaptive scheduling, so the gain comes from adjusting K/C by state rather than from a single better fixed combination. Together with the equal-NFE fixed-schedule comparison on Cream Cheese in Figure 5(a) of the paper, these comparisons cover a short-horizon and a long-horizon task. This table is an additional experiment and is reported separately from the main Moka results in Table 2 of the paper.

W2. Component ablations and weight sensitivity

EQRL jointly selects latent steering, denoising computation, and feedback timing for a frozen VLA. Critic disagreement supplies a reference K/C schedule; learned residuals adjust it using task value, followed by clipping and rounding. Gates control activation and strength, the NFE band regularizes total compute, and alignment guides state-dependent allocation. We test these components below.

Reference schedule and learned residuals. The original five-seed Cream Cheese ablation is below. Prior-only disables learned scheduling residuals, while SAC-only disables the difficulty prior; both retain latent-policy learning and critic updates.

ConfigurationSuccess, mean ± SDFull AUC, mean ± SDRequested NFE, mean ± SD
Prior-only96.00% ± 5.48 pp0.860 ± 0.0160.344 ± 0.011
SAC-only86.00% ± 5.48 pp0.834 ± 0.0230.400 ± 0.016
Full EQRL100.00% ± 0.00 pp0.886 ± 0.0150.320 ± 0.007

Full EQRL is best on all three metrics, supporting complementary roles for the reference and the residuals.

K, C, and the budget term. In Figure 5(b) of the paper, the K-only, C-only, and Budget-only variants need requested NFE of 0.40, 0.41, and 0.35, and Budget-only shows larger drops in success; full EQRL uses 0.32.

Sensitivity analysis. We added a separate three-seed sensitivity study on one LIBERO-90 task, with 500k updates and 50 final evaluation episodes per seed. NFE averages seed-level ratios.

ConfigurationFinal success, mean ± SDFull AUC, mean ± SDRequested NFEExecuted NFE, mean ± SD
Full87.33% ± 10.26 pp0.8125 ± 0.02640.34360.3615 ± 0.0560
No alignment82.67% ± 4.16 pp0.8064 ± 0.01660.41440.4316 ± 0.0440
alignment ×0.581.33% ± 13.61 pp0.7793 ± 0.04200.38670.4042 ± 0.0294
alignment ×280.67% ± 9.24 pp0.7844 ± 0.04100.37190.3881 ± 0.0315
cost ×0.586.00% ± 10.00 pp0.8162 ± 0.02340.38590.4073 ± 0.0666
cost ×278.67% ± 13.61 pp0.7056 ± 0.10080.35140.3694 ± 0.0243

Removing alignment raises executed NFE from 0.3615 to 0.4316 (the default is 16.2% lower) and lowers success by 4.67 percentage points; all three alignment weights give executed NFE of 0.36–0.40, below the 0.43 without alignment. When the alignment and cost weights vary between 0.5× and 2×, success stays between 78.67% and 87.33%, within one SD of the default. The default has the highest final success and lowest NFE; its AUC is within 0.004 of the best (cost ×0.5). Gates were not separately ablated. The five- and three-seed studies are separate cohorts. Overall, the main weights work across a wide range, and the default needs no fine tuning.

For the abstract, the revised version will state the problem formulation and the learning method in two separate sentences so that the contribution is clearer.

AuthorsOpenReview ↗

Response to Reviewer obfL (Part 2/2): Related work, schedules, transfer, and online learning

Response to Reviewer obfL, Part 2/2.

Q1. Relation to D3P and the joint formulation

D3P uses RL to adapt denoising stride and includes joint training of an adaptor and diffusion policy. EQRL keeps the VLA frozen during adaptation and jointly learns latent steering ww, denoising steps KK, and execution length CC. C determines when to observe again and changes the duration of a query transition.

The joint formulation distinguishes generated behavior, computation, and feedback timing. For example, K=10, C=20 and K=5, C=10 both have requested NFE 0.5, but obtain feedback every 20 versus 10 requested actions. Our local analysis separates sampling error from feedback delay. D3P trains the diffusion policy itself jointly with its adaptor, unlike our frozen-VLA setting, so we did not run a protocol-matched comparison. Within our framework, the closest counterpart is the K-only variant, which learns only the denoising steps: it needs requested NFE 0.40, whereas full EQRL uses 0.32, 20% less; the C-only variant, which learns only the execution length, needs 0.41. Learning feedback timing together with denoising steps therefore gives an additional gain.

Q2. How K and C change during a task

Yes, the learned schedules match the intuition in Figure 1: near object interaction and contact, EQRL uses more denoising steps and shorter chunks, and it does the opposite during free-space motion.

In an Alphabet Soup to Tray episode, the schedule starts at K=7, C=31, changes to K=10, C=21 and K=12, C=17 when the gripper approaches the object and attempts the grasp, and then returns to K=8, C=30.

On the long-horizon Moka task (500k updates, seed 1), requested NFE is 0.428, 0.518, and 0.424 in the early, middle, and late thirds of an episode. The middle third is approximately 21–22% higher, from both more denoising steps and shorter chunks, and it is also the phase with the highest critic disagreement (Section 4.2 and Figure 4 of the paper). The same pattern appears on the real robot: in Bimanual Coke Pouring, requested NFE peaks near alignment and grasping and falls during steadier transfer and placement (Figure 6 of the paper).

State interventions show that this allocation is useful. Across three mature Cream Cheese runs, in the highest-difficulty quartile of states, shortening C improves success by 7.3–30.0 percentage points and increasing K gives a pooled improvement of 4.4 points. Earlier feedback gives the larger and more consistent benefit, in line with the schedule's direction of shortening C and increasing K as difficulty rises.

Q3. Cross-task transfer

Our lightweight adaptor is trained per task, with the pretrained VLA frozen. We added a Cream Cheese → Tomato Sauce zero-shot evaluation, applying final adaptors from three source-task seeds to the target instruction without target-task training or recalibration. Each seed was evaluated for 20 episodes per condition.

Transfer conditionSuccessesSuccess, mean ± SDRequested NFE, mean ± SDExecuted NFE, mean ± SD
Whole adaptor, dynamic K/C3/605.00% ± 8.66 pp0.4648 ± 0.12330.4754 ± 0.1236
Same adaptor, fixed K=10, C=2010/6016.67% ± 24.66 pp0.5000 ± 0.00000.5029 ± 0.0040

This result does not demonstrate reliable zero-shot transfer; our conclusions remain based on task-specific adaptation. The whole adaptor is transferred, and the fixed condition is not target-trained DSRL, so failure cannot be attributed to the scheduler alone. Lower NFE under dynamic scheduling is not an efficiency improvement at preserved success. Per-task adaptation is inexpensive: only the lightweight adaptor is trained, and online robot training takes approximately 1–2 hours. Core training hyperparameters are shared across tasks; the schedule-related settings give similar success across their basic range, and about 3–5 runs within a small range are enough to reach the best result.

Q4. Online robot learning and interaction requirements

Online Cup-to-Bowl and Bowl Lift starts from 20 demonstrations. In this training run, the learned schedule starts to help around the 10th collection–training round, with 1,000 gradient updates per round, corresponding to approximately 10k online-stage gradient updates.

The complete online stage uses approximately 15k executed robot actions and 500k SAC gradient updates and takes 1–2 hours, including interaction, updates, and routine resets but excluding demonstration collection. Final success is 19/20 for EQRL versus 18/20 for DSRL, with requested NFE 0.341 versus 0.500. Execution time over the 20 trials is 11.310 ± 0.773 seconds versus 13.255 ± 1.007 seconds, respectively: EQRL is faster and its execution time varies less. These statistics describe repeated execution of the learned policy, rather than variability across independent training runs.

Thank you again for your careful review. If you have a chance to read our rebuttal, we would appreciate hearing your thoughts. Please let us know if any of your concerns remain or if there is anything we can clarify further.

Reviewer_rFA8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Elastic Queries Reinforcement Learning (EQRL) for adapting a frozen flow-based VLA. A lightweight SAC adaptor jointly selects latent steering, the number of denoising steps K, and the executed action-chunk length C. Critic disagreement provides a difficulty-conditioned reference schedule, while learned residuals adjust K and C, and query-level targets use gamma raised to the actual executed duration.

Critical strengths
  • Jointly adapting inference effort and feedback frequency is an important and underexplored problem. The paper correctly emphasizes that schedules with the same amortized K/C can behave differently because C changes when the policy receives new observations.
  • The long-horizon Moka Pots result is a consequential positive result: across five seeds, success increases from approximately 0.41 to 0.87, a difference that is large relative to the reported seed variation.
Critical weaknesses
  • The central compute-efficiency claim is based on requested rather than executed compute. The reported metric is sum(K)/sum(C), although actual execution uses L<=C and the paper itself shows that sum(K)/sum(L) is larger. No main experiment reports executed NFE, and NFE also omits fixed per-query costs such as vision/language encoding, observation acquisition, adaptor execution, and synchronization.
  • The duration-aware SMDP objective is not fully specified or internally consistent. The final success reward is assigned directly to the terminal query rather than discounted within the final chunk, while a future-dependent episode compute penalty is redistributed uniformly across queries. With variable gamma^L, that redistribution does not preserve the original discounted objective, and the critic state does not include accumulated budget or debt. The exact penalty function is also missing.
  • The comparison budget and statistical presentation are insufficient. Equal optimizer updates can correspond to different primitive actions, observations, episodes, and wall-clock because C is adaptive. Several main curves select the best full-AUC seed; task-wise five-seed uncertainty is incomplete. Moreover, the offline real-robot variant uses fixed-stride transitions and future expert actions to create scheduling labels, so candidate C does not causally change the transition duration.
Questions for authors

See above

4 comments
AuthorsOpenReview ↗

Response to Reviewer rFA8 (Part 1/2): Executed NFE and training objective

Thank you for your careful review. We address your comments in 2 parts.

W1. Executed-action NFE and inference overhead

We recomputed NFE for both methods on the same runs underlying the original four-task LIBERO results, using actually executed actions as the denominator. EQRL's reduction relative to DSRL changes from 39.4% under requested-action accounting to 35.7% under executed-action accounting. Thus, the savings remain above one-third after accounting for early termination. The two accountings differ only in the final chunk of an episode, when it terminates early, and this applies to both methods, so the conclusions of the other experiments are unchanged. On Moka, for example, the reduction is 12.5% under requested-action accounting and 12.3% under executed-action accounting (Table 2 in our response to Reviewer obfL).

For the fixed per-query overhead, we measured complete policy-inference time on a LIBERO task, including preprocessing, encoding, actor, critic, scheduling, VLA sampling, and synchronization; both methods use the same GPU and the same inference optimizations. Inference time per executed action decreases from 2.491 ± 0.035 ms to 1.731 ± 0.114 ms, approximately 30.5%. The fixed overhead is paid once per query, and EQRL executes more actions per query on average, so this overhead is amortized as well and does not offset the savings. Results are mean ± SD across three seeds, with 10 episodes per seed and 30/30 successes for each method. Timing starts from an available observation and excludes sensor acquisition, environment execution, and resets.

For the online robot task, mean trajectory execution time across all 20 attempts per method, including failures, decreases from 13.255 to 11.310 seconds, a 14.7% reduction.

W2. Query-level objective and compute regularization

EQRL uses a sparse task reward: 1 for success and 0 otherwise, without stage-specific dense reward design. Success is assigned to the terminal query, and subsequent value is discounted by the actual executed duration, γLj\gamma^{L_j}. This objective is defined at the query level, where the adaptor acts. With γ=0.999\gamma=0.999 and at most 50 actions per chunk, the discount omitted within the terminal chunk is at least 0.99950≈0.950.999^{50}\approx0.95, so it changes the success reward by at most 5%. The episode compute cost enters the training reward as a query-level regularizer, controlling overall computation while allowing more compute at difficult states. Uniform redistribution preserves its undiscounted total; we do not claim equivalence to the original primitive-action discounted objective.

Let SK=∑jKjS_K=\sum_jK_j, SC=max⁡(∑jCj,1)S_C=\max(\sum_jC_j,1), and ρ=SK/SC\rho=S_K/S_C. The compute regularizer is

P_{\rm ep}={}&g_o\frac{[S_K-uS_C-D]_+}{s} +g_u w_u\frac{[\ell S_C-S_K]_+}{s}\\\\ &-g_s w_s\mathbf 1_{\ell\leq\rho\leq u} \operatorname{clip}\\!\left(\frac{u-\rho}{\max(u-\ell,10^{-6})},0,1\right). \end{aligned}$$ The terms penalize overspending, penalize underspending, and reward savings within the active NFE band $[\ell,u]$. Here, $[x]_+=\max(x,0)$, $D$ is a fixed overspend tolerance, $s>0$ is a normalization scale, and $w_u,w_s$ are weights. Successful episodes use $g_u=1$ and $g_o=g_s=g_{\rm success}$; failed episodes use $g_u=g_s=0$ and $g_o=g_{\rm failed}$. The implementation uses $g_{\rm success}=c_{\min}+(1-c_{\min})p$ and $g_{\rm failed}=c_{\min}+(1-c_{\min})p w_{\rm fail}$, where $p$ gates training progress and additionally incorporates a success-rate gate when configured. The training reward and critic target are $$R_j=\mathbf1[j=N]\mathbf1[\mathrm{success}]-\lambda_{\rm cost}P_{\rm ep}/N, \qquad y_j=R_j+\gamma^{L_j}(1-d_j)\bar Q_{\rm targ}(o',\tilde u').$$ Here, $N$ is the number of episode queries, $1-d_j$ is the bootstrap mask, and $\bar Q_{\rm targ}$ is the target-critic ensemble mean. The target has no entropy term; entropy regularization is applied to the actor. The critic does not explicitly receive accumulated budget, so its treatment of this history-dependent term is approximate. The regularizer encourages overall NFE to fall within the target band, and the reported values lie within their respective bands: requested NFE is 0.320 on Cream Cheese with band [0.28, 0.425], and 0.172 on ALOHA with band [0.12, 0.18]. This is the query-level training regularizer described in Section 3.4 of the paper, given here in full; the per-task bands $[\ell,u]$, including the annealed upper bound, and the weights $w_u,w_s$ are listed in Appendix Tables 7 and 8. These equations specify simulation training.
AuthorsOpenReview ↗

Response to Reviewer rFA8 (Part 2/2): Budgets, new five-seed results, and offline scheduling

Response to Reviewer rFA8, Part 2/2.

W3. Comparison budgets, five-seed statistics, and offline scheduling

We use SAC update count as the training budget and keep the same update ratio: 20 gradient updates per newly collected query transition (Table 7 of the paper). The two methods are therefore compared under the same optimization budget. Because execution length C is learned, the same number of queries can correspond to different numbers of primitive actions. This also reflects the purpose of elastic execution: in suitable states, a longer execution prefix allows one policy query to cover more actions, amortizing inference cost per action. We assess this trade-off jointly through task success and NFE.

We ran a new set of experiments on all four tasks, with five seeds per method. The table below reports mean success and standard deviation at 50k, 100k, 250k, and 500k SAC updates. Means are percentages and SDs are percentage points, showing the variability across seeds.

Table 1 | New four-task, five-seed checkpoint statistics

TaskMethod50k100k250k500k
Black BowlDSRL72.46 ± 5.7487.60 ± 10.4192.43 ± 8.1898.77 ± 1.68
Black BowlEQRL25.83 ± 7.5583.34 ± 18.7296.36 ± 6.30100.00 ± 0.00
Alphabet Soup to TrayDSRL84.83 ± 4.8692.65 ± 8.2895.40 ± 7.5195.73 ± 6.43
Alphabet Soup to TrayEQRL70.99 ± 8.6496.76 ± 5.5893.56 ± 9.8397.86 ± 3.71
Cream Cheese to TrayDSRL65.55 ± 5.9596.80 ± 5.4693.49 ± 8.4393.22 ± 7.71
Cream Cheese to TrayEQRL93.71 ± 4.0297.06 ± 4.8396.00 ± 6.6094.63 ± 7.00
Tomato Sauce to TrayDSRL63.49 ± 6.6582.47 ± 13.0886.82 ± 10.9494.97 ± 6.93
Tomato Sauce to TrayEQRL92.90 ± 4.3098.94 ± 1.7090.02 ± 12.3393.85 ± 8.16

At 500k updates, both methods achieve mean success above 93% on all four tasks. EQRL has higher mean success on three tasks and is about one percentage point lower on Tomato Sauce. The per-task, per-checkpoint SDs make the seed-to-seed variability explicit.

Offline robot scheduling uses expert-action supervision. At the same observation and noise, we compare action prefixes generated with candidate K/C choices against expert actions to obtain scheduling labels, and train with compute regularization. Fixed demonstrations cannot provide transitions under alternative C, as stated in Appendix A.1; the causal effect of C on transition duration is learned in simulation and online robot training. The offline-adapted policies are then evaluated in closed-loop execution on the robot, where C does determine when feedback arrives: they succeed in 69/80 trials across the four offline tasks versus 63/80 for DSRL, with requested NFE 0.338 versus 0.500.

Thank you for the thorough rebuttal, the executed-NFE and wall-clock latency measurements, the explicit PepP_{\text{ep}} formulation, and the new five-seed LIBERO-90 runs. These substantially address several of my concerns.

Before deciding whether to raise my score, could you briefly clarify three remaining points (and confirm they will be incorporated into the revised PDF)?

  1. W1 & W3 (Primitive Action Budget & Latency When C≤CDSRLC \le C_{\text{DSRL}}): In Table 7, DSRL fixes C=20C=20 on LIBERO-90 whereas EQRL allows CC up to 40–4440\text{–}44. With a fixed 20 gradient updates per query transition, larger CC both consumes more primitive environment steps per update and amortizes fixed per-query overhead. Could you report (a) total primitive environment steps alongside 500k SAC updates for DSRL vs. EQRL on LIBERO-90 and Moka Pots, and (b) wall-clock inference time per executed action on ALOHA or Moka Pots, where CC cannot exceed CDSRLC_{\text{DSRL}} to offset the extra 10-critic qstdq_{\text{std}} overhead? (Also, please reconcile Moka requested-NFE reduction: 11.8% in Table 2 vs. 12.5% in the rebuttal.)
  2. W3 (Five-Seed Aggregates for Figure 5 Ablations): While Figure 3 shows five-seed mean ±\pm SD, Figure 5 (Matched-NFE DSRL (0.32), K-only, C-only, and Budget-only) still plots only each variant's single highest-AUC seed out of five. Could you provide the five-seed mean ±\pm SD (final success, full AUC, and executed NFE) for all variants in Figure 5?
  3. W2 (Per-Task Schedule/Band Tuning): Table 7 uses task-specific [Cmin⁡,Cmax⁡][C_{\min}, C_{\max}], NFE target bands, and under/saving weights even across the four LIBERO-90 tasks. How sensitive is EQRL to using a single shared NFE band and chunk range across tasks within the same suite?

Clarifying these concise points will help me finalize my assessment.

AuthorsOpenReview ↗

Follow-up response to Reviewer rFA8

Thank you for following up. We address the three points below.

Q1. Training interactions, inference latency, and Moka NFE

(a) Training environment steps. We count actual training steps, with evaluation and reset steps counted separately.

TaskSAC updatesDSRL stepsEQRL stepsChange
LIBERO-90 (four-task mean)500k487,915.25545,225.00+11.7%
Moka Pots (supplemental, three-seed mean)500k502,520511,334+1.75%
ALOHA Cube Transfer1M2,398,3392,281,913-4.85%

Updates are scheduled per policy query, so actual training steps depend on how many actions each query executes. On LIBERO-90, DSRL uses C=20 and EQRL can select longer chunks. On ALOHA, DSRL uses C=50 and EQRL can select shorter chunks. The observed step counts follow this pattern.

(b) Latency with EQRL's chunk length capped at DSRL's. On ALOHA, DSRL requested C=50 and EQRL requested C=45–50. We used the same optimized inference path on one H800 and evaluated 10 paired initial states. Timing included observation preprocessing, the adaptor, all 10 EQRL critics, schedule selection, and synchronized policy outputs. We divided total inference time by the number of executed actions. DSRL took 0.999 ms per executed action and EQRL took 0.992 ms. EQRL's executed NFE was 0.2019, about 4% below DSRL's 0.2103. EQRL succeeded in 10/10 trials and DSRL in 9/10.

(c) Moka requested NFE. The 11.8% reduction is from the main Table 2 experiment (0.500 to 0.441). The 12.5% reduction is from the supplementary fixed-schedule comparison (0.5000 to 0.4375). Appendix A.2 now reports the interaction budgets, controlled latency, and both Moka NFE comparisons.

Q2. Figure 5 ablations

We updated both panels of Figure 5 to show the checkpoint-wise mean and ±1 sample SD across five training seeds. Appendix Table 10 reports these summaries (mean ± sample SD).

The table below summarizes the five seeds per variant in Figure 5's K/C scheduling study, whose Full EQRL runs were trained independently of those in the prior/residual ablation in Table 3.

Method500k success (%)Full AUCExecuted NFE at 500k
Matched-NFE DSRL94.0 ± 8.940.86392 ± 0.028670.33335 ± 0.01949
Full EQRL100.0 ± 0.000.88275 ± 0.014960.34326 ± 0.02951
K-only100.0 ± 0.000.79686 ± 0.002970.41809 ± 0.03342
C-only86.0 ± 11.400.82588 ± 0.024030.42856 ± 0.01743
Budget-only74.0 ± 5.480.74902 ± 0.020330.37331 ± 0.01836

Compared with DSRL matched on requested NFE, EQRL has higher mean AUC and slightly higher executed NFE. Compared with the three scheduling ablations, Full EQRL has higher mean AUC and lower executed NFE. Full EQRL and K-only both reach 100% final success.

Q3. Sensitivity to a shared NFE band and chunk range (W2)

For this rebuttal, we trained separate EQRL policies for the four LIBERO-90 tasks, with three seeds per task and 50 final evaluation episodes per seed at 500k SAC updates. All tasks used K=3–15, C=16–44, and the same requested-NFE band. Its lower bound was fixed at 0.30, and its upper bound annealed from 0.50 to 0.40 between 150k and 350k SAC updates. The under-budget and saving weights were 0.45 and 0.04. Each policy still selected K and C from its current observation. Results for the shared configuration are means ± sample SD.

TaskTuned success (%)Shared success (%)Shared requested NFEShared executed NFE
Black Bowl~9889.33 ± 4.160.3610 ± 0.01780.3871 ± 0.0199
Alphabet Soup~9691.33 ± 4.160.3691 ± 0.02340.3994 ± 0.0261
Cream Cheese10096.67 ± 5.770.3603 ± 0.03700.3891 ± 0.0332
Tomato Sauce~9293.33 ± 5.770.3820 ± 0.02550.4098 ± 0.0359

Mean success was 92.67%, about 3 percentage points below the 96% reported for task-specific tuning in Table 1 of the manuscript. Three tasks stayed within about 5 points of their tuned results. Black Bowl had the largest drop. Mean executed NFE was 0.3964. Fixed-schedule DSRL executes at most C actions per query, so its executed NFE is at least K/C=0.500. The shared configuration thus saved at least 20.7% of denoising evaluations per executed action. Our earlier rebuttal reported a 35.7% saving for task-specific EQRL under the same executed-action accounting. Appendix A.2 and Table 13 give the shared configuration and task-level results.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Author summary of the discussion

EQRL jointly learns latent steering, denoising steps (K), and how many actions to execute before observing again (C) for a frozen vision–language–action model. Even at the same denoising cost per action, schedules can differ in feedback timing and task success. In our interventions, earlier feedback helped most consistently in difficult states. Reviewers recognized the importance of this problem and the practical value of the approach. Across five robot tasks, EQRL shortened mean trajectory time by 11–23%, with comparable or higher observed success.

WS92 · Statistical reporting, related methods, and robot evaluation. We updated Figure 3 to show five-seed means and standard deviations for all six simulation tasks. We qualified the robot success claims given 20 trials per method and task, expanded the discussion of RLT, PLD, and AAC, and reported a preliminary test using AAC’s released selector. We also clarified the tuning effort, the scope of the precision evaluation, and the 1–2 hours of online training, which includes interaction, updates, and resets and excludes initial demonstration collection. WS92 said the revisions largely addressed their concerns and raised their rating from 2 to 3, with presentation as their main remaining concern. WS92’s follow-up

rFA8 · Compute accounting, training objective, and robustness. Our initial rebuttal showed 35.7% fewer denoising evaluations per executed action across four LIBERO-90 tasks and clarified the compute regularizer, duration-aware bootstrapping, and offline schedule supervision. rFA8 said this substantially addressed several concerns and requested three follow-ups before reconsidering the rating.

  1. Training steps and latency. We counted actual environment steps. At 500k updates, EQRL used 11.7% more on LIBERO-90. With the same chunk-length cap on ALOHA, inference took 0.992 vs. 0.999 ms per executed action over ten paired initial states, including EQRL’s ten critics. The 11.8% and 12.5% reductions in requested NFE on Moka come from the main and supplemental experiments, respectively.
  2. Five-seed ablations. At equal update budgets, EQRL achieved higher mean learning-curve AUC than DSRL matched on requested NFE, with slightly higher executed NFE. It also had higher mean AUC and lower executed NFE than the K-only, C-only, and budget-only variants (Figure 5 and Appendix Table 10).
  3. Shared scheduling settings. With one shared chunk range and compute band, separately trained policies averaged 92.7% success across four tasks versus 96.0% with task-specific tuning, while using at least 20.7% fewer denoising evaluations per executed action than fixed-schedule DSRL.

We posted these results on November 19 and have yet to receive a follow-up on them. Our follow-up

obfL · Fixed-schedule controls, component ablations, and transfer.

  1. Fixed schedules and task selection. We explained how the tasks were selected and compared EQRL against six fixed schedules on long-horizon Moka Pots. Across five seeds, EQRL achieved 86.3% success versus 46.4% for the best tested fixed schedule at nearly identical executed NFE (0.4481 vs. 0.4488).
  2. What each component contributes. We summarized the prior and residual ablations and added a sensitivity study for the alignment and compute-cost weights.
  3. D3P, scheduling behavior, transfer, and online learning. We explained how EQRL relates to D3P’s denoising adaptation, showed how schedules change near contact, and tested their effects through state interventions. We reported limited zero-shot transfer and when the schedule began helping in the online robot run.

We have yet to receive a follow-up from obfL. Fixed-schedule and ablation results · Responses to the four questions

Given the time available during discussion, we focused the PDF update on Figures 3 and 5 and selected results in Appendix A.2. If the paper is accepted, we will integrate the full set of rebuttal experiments into the paper and appendix, rework the introduction to explain why joint scheduling matters, and organize the experimental text around the main findings. We will also revise the layout of the figures and tables, including Table 2. Current PDF

Whatever the final ratings, we are grateful for the time and care the reviewers put into their feedback. Their suggestions led to stronger controls and helped us address most of the points raised. We are eager to present the full results in a final paper that shows when joint scheduling helps in practice, and we hope the new evidence earns your support.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.