acceptodds
Withdrawn from ICLR 2027

StructRL: Online Structured Reinforcement Learning for Long-Horizon Vision-Language-Action Tasks

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision-language-action (VLA) models perform well on shorter-horizon manipulation tasks but still struggle with long-horizon tasks that require multiple dependent manipulations from a single command. Online reinforcement learning (RL) can improve these policies through environment interaction, yet many existing methods provide reward only after the complete task succeeds. However, such terminal supervision is sparse and does not distinguish early failures from rollouts that make substantial partial progress. We propose StructRL, an online RL framework that constructs structured intermediate supervision from verifiable subtask completions. StructRL decomposes each task into verifiable subtasks, grants intermediate rewards only after the prerequisite subtasks have been completed, and scales each reward according to completion pace. Across RoboCasa365 and LIBERO-Long with GR00T-N1.5 and pi 0.5, StructRL consistently outperforms evaluated online RL baselines. These results show that verifiable, structured intermediate rewards improve long-horizon VLA post-training. Code is included in the supplementary material.

Primary area:
reinforcement learning
Keywords:
Vision-Language-Action (VLA), Online Reinforcement Learning, Long-Horizon Manipulation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oGR8OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

StructRL uses an LLM to decompose long-horizon tasks into subtasks with dependency stages, grounds each subtask to a binary predicate on simulator state, and rewards each completion once, only after prerequisites are met, scaled by completion speed relative to demonstrations. Trained with PPO on RoboCasa365 and LIBERO-Long with GR00T-N1.5 and π0.5, it beats the strongest online RL baseline by 2.2 to 7.6 points.

Critical strengths
  1. The paper is clearly written and easy to follow.
  2. Reasonable breadth in simulation: two benchmarks, two backbones, GRPO variant, and a decomposition-density is a useful practical finding.
Critical weaknesses
  1. No real-world experiments, and the method depends on privileged simulator state. Every intermediate reward is a predicate over the benchmark's native state and success-check utilities.
  2. The core argument is that scalar progress estimates cannot tell which prerequisite makes a completion valid. Stage-aware reward models already address this: VICtoR (ICLR 2025) and SARM (ICLR 2026) both predict the current stage plus within-stage progress. Neither is cited or compared.
  3. Table 1 reports single runs. Figure 3's "three evaluation runs" measure evaluation noise, not training variance. Many deltas (+0.4, +0.5, +0.8, +0.9) and both the gating and pacing contributions are of this magnitude.
  4. The method rely on privileged state and success checks which is unavailable on real environment
Questions for authors
  1. How would subtask verification work on a real robot?
  2. Why were VICtoR, SARM not compared, at least on RoboCasa365?
  3. How many training seeds, and what are the standard deviations?
Reviewer_1ExJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

StructRL addresses sparse rewards in online reinforcement learning for long-horizon vision-language-action tasks. It decomposes tasks into verifiable subtasks, constructs intermediate rewards, gates rewards according to prerequisite dependencies, and adjusts their magnitude using completion time relative to demonstrations. On RoboCasa365, an LLM proposes the decompositions, which are grounded to simulator predicates; on LIBERO, the method uses native task-goal conditions. The framework is evaluated with GR00T-N1.5 and π0.5 on RoboCasa365 and LIBERO-Long.

Critical strengths

The paper studies an important problem: terminal-only rewards provide little feedback when a policy makes progress but fails before completing a long task. The reward construction is intuitive and compatible with existing VLA training frameworks, and the paper is generally clear and easy to follow.

The simulation experiments are substantial, covering two long-horizon benchmarks and two policy backbones. StructRL improves over both SFT and the evaluated online RL baselines. With GR00T-N1.5, success increases over SimpleVLA-RL from 41.5% to 49.1% on RoboCasa365 and from 92.4% to 96.6% on LIBERO-Long. The subtask-level analysis suggests that intermediate supervision helps policies progress through stages where SFT policies tend to fail. The component ablations are clearly presented, and the decomposition-density analysis provides useful evidence that finer reward decomposition is not always better.

Critical weaknesses
  1. The methodological contribution could be distinguished more clearly from existing reward-design approaches. Subtask rewards, dependency-based task structure, and LLM-assisted reward construction have precedents in work such as Reward Machines and Text2Reward. Figure 3 shows that fixed subtask rewards account for most of the improvement on RoboCasa365, increasing success from 41.3% to 47.4%, while pacing and gating further raise it to 49.2%. The ablation supports the usefulness of these mechanisms, but does not by itself resolve how they differ conceptually from earlier structured reward methods. My main reservation is that the paper could explain this distinction more clearly and establish what the framework contributes beyond an effective integration of existing ideas for VLA training.

  2. The practical requirements and limits of broader applicability deserve further discussion. As acknowledged in Appendix G, the current experiments rely on simulator predicates and do not include physical-robot training or evaluation. On real robots, identifying subtask completions may require manual annotation, training completion detectors, or developing task-specific sensing and detection rules. This affects both estimating demonstration reference durations T_d and generating rewards during online RL, and could introduce substantial human effort as the task set grows. Moreover, a missed prerequisite completion may prevent later subtasks from receiving rewards. Discussing the cost and reliability of supplying these signals would help readers understand the requirements for extending the framework beyond simulation. Transfer to unseen task compositions also remains limited: on RoboCasa365 Composite-Unseen, StructRL achieves 4.8% success, compared with 4.3% for SimpleVLA-RL. On Composite-Seen, the corresponding rates are 49.1% and 41.5%. The suites differ in difficulty, so the difference in gains alone does not establish the cause of limited transfer. However, the low absolute unseen-task success leaves open how much the improved subtask execution carries over to unseen compositions. These results and the requirements for real-world completion detection help define the method's demonstrated scope.

Questions for authors
  1. Could you clarify the main methodological distinction from earlier approaches using subtask rewards and task dependencies? Given the large gain from fixed subtask rewards in Figure 3, what do you consider the key new contribution of gating and pacing?

  2. How do you envision obtaining the required completion signals on real robots, both for estimating T_d from demonstrations and for assigning rewards during online RL? It would be helpful to clarify which parts could use direct sensor feedback and which may require human annotation or detector development.

  3. The subtask-level results suggest improved execution of intermediate skills, while Composite-Unseen success remains low. Based on the existing evaluations and observed failures, do you have insight into what limits the transfer of these improvements to unseen task compositions?

Reviewer_8kUXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StructRL proposes structured intermediate rewards for online reinforcement learning of vision-language-action policies on long-horizon manipulation tasks. It represents progress as verifiable subtask completions, gates rewards by prerequisite completion, and scales their magnitude using completion intervals relative to demonstrations. Rewards are computed from simulator predicates and used with PPO, with an additional GRPO experiment. RoboCasa uses an LLM-assisted decomposition and rule-based grounding pipeline; LIBERO uses goal predicates from task specifications. Experiments on RoboCasa365 and LIBERO-Long with GR00T-N1.5 and π0.5 report improvements over the evaluated online RL baselines, supported by component and reward-density analyses.

Critical strengths
  1. The method addresses a concrete limitation of terminal-only rewards, and the overall empirical gains are meaningful. Under matched starting checkpoints and interaction budgets, StructRL improves over the strongest evaluated online RL baseline on both benchmarks and both backbones. For example, GR00T-N1.5 improves from 41.5% to 49.1% on RoboCasa365 and from 92.4% to 96.6% on LIBERO-Long (Table 1). These results support the practical value of verifiable intermediate supervision.
  2. The reward-design analyses provide useful information beyond the aggregate success rates. The component ablation identifies intermediate subtask rewards as the main source of improvement. The decomposition-density study and the timing and reward-budget controls in Appendix C also help explain why adding more progress signals does not necessarily improve learning.
Critical weaknesses
  1. The incremental methodological contribution is not yet sufficiently established. Structured reward specification and task decomposition have established precedents, including Reward Machines (Icarte et al., ICML 2018). In Figure 3, fixed subtask rewards improve RoboCasa365 success from 41.3% to 47.4%, while pacing adds 0.5 percentage points and gating adds a further 1.3 points. The corresponding increments on LIBERO-Long are 3.5, 1.2, and 0.5 points. Most of the gain therefore comes from rewarding verifiable subtask completions. The evidence for the added value of the two proposed mechanisms is more limited: the ablation omits fixed subtask rewards with gating but without pacing, and the reported repeats are described as evaluation runs, leaving variability across independently trained models unclear. I find the VLA application useful, but the current evidence does not yet establish a compelling advance from the specific gating and pacing design.
  2. The pacing reward needs a precise and internally consistent operational definition. Section 3.2.2 measures the rollout interval from the previous rewarded subtask, whereas the demonstration interval starts when the current subtask’s prerequisites become complete. These origins can differ for subtasks in the same unordered stage. In addition, Algorithm 1, lines 10–13, updates the last-completion timestamp inside the loop over subtasks. Read literally, if two eligible events are detected in one action chunk, the second receives a zero interval and hence the maximum reward. With an elapsed interval of 10 chunks, reference intervals of 5 and 20, and reward scale 0.6, processing the first event before the second yields a total reward of 0.8; reversing the order yields 1.0. Thus, the same rollout may receive different rewards depending on enumeration order. This is a concern about the stated algorithm; whether it occurs in the implementation remains to be clarified. It directly affects interpretation and reproducibility of a claimed contribution.
Questions for authors
  1. How are the two timing origins reconciled in the implementation? How are multiple eligible completions within one action chunk processed, and is the reward invariant to subtask enumeration order? Please explain the numerical example above and provide the precise timing and tie-handling rule, with the relevant implementation reference.
  2. Does pacing still help when compared with fixed subtask rewards plus gating, using the same predicates and training budget? Please also clarify whether the repeated runs in Figure 3 and Appendix D.1 include independent training seeds. Evidence on variability across training runs would help assess the modest component gains.
  3. For reproducibility, please identify the code and configuration files implementing the grounding and timing pipeline in Appendix B.2, and clarify any task-specific additions beyond benchmark utilities and shared templates. This includes the different construction used for LIBERO. The reproducibility statement says code and static graphs are supplied; pointers to the relevant files would help resolve this clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.