acceptodds
Under review as a conference paper at ICLR 2027

Mixture-of-Checkpoints: Improving Initial-State Coverage of Vision-Language-Action Policies via Online Routing

Abstract

Post-training for vision-language-action policies usually keeps the checkpoint with the highest aggregate success rate and discards the rest. This selection ignores whether different checkpoints cover different initial states. We study whether checkpoints that do not improve aggregate success can still improve initial-state coverage. We post-train OpenVLA-7B with Direct Preference Optimization on action preference pairs derived from Monte Carlo tree search value estimates; each post-training run yields one checkpoint. The resulting checkpoints match or underperform the base model on LIBERO. However, a two-round per-state audit shows that they succeed on different initial states. Together, they cover most of the initial states where the base model fails. Therefore, we deploy the base model and all checkpoints as a policy portfolio, which we call Mixture-of-Checkpoints, and add a router that selects one policy per episode. The router is a per-state contextual bandit: the initial-state index is the context, episode success is the reward, and an exponential moving average updates a per-state score table. Across eight LIBERO tasks, routing matches or outperforms the base model. On stable tasks with headroom, it adds 8 to 19 successful episodes per 100 episodes. Two tasks selected after the method was fixed confirm predictions recorded in advance. Ablations show that the gain requires online per-state measurement, and a two-round audit predicts which tasks benefit.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language-Action Models, Robot Learning, Post-Training, Direct Preference Optimization, Checkpoint Routing, Policy Portfolios, Contextual Bandits, Online Routing, Initial-State Coverage, LIBERO

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EMacOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether post-training checkpoints with similar or even worse aggregate performance can still provide complementary coverage over different initial states. It keeps the base VLA policy and several post-training checkpoints. Then, it uses an online per-state contextual bandit to select one policy for each episode. The router uses the initial-state index as context and updates a simple score table from episode success. Experiments on LIBERO show that this routing strategy can improve success on tasks where policy outcomes are stable and repeatable for the same initial states.

Critical strengths

I do not see a strong contribution here. The main positive aspect is that the paper makes a useful empirical observation that checkpoints with similar aggregate performance can still cover different initial states. And the paper also evaluates several routing baselines and alternatives and includes additional analyses on state-level stability and cross-process repeatability.

Critical weaknesses

My main concern is that the setting is quite narrow. The router directly uses the initial-state ID and assumes that the same states will be revisited and behave consistently. This is a strong assumption, and the paper itself shows that the state-level outcomes can change substantially across processes and even GPU types. I also find the methodological contribution too limited for a full ICLR paper. Despite the "Mixture-of-Checkpoints" framing, the actual method is basically a per-state lookup table with standard epsilon-greedy routing. The observation is interesting, but the method itself feels too lightweight.

Questions for authors

See above weaknesses.

Reviewer_dsZ7OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies whether post-training checkpoints of an OpenVLA policy can provide complementary success coverage over fixed initial states even when they do not improve aggregate LIBERO success rates. The authors construct a portfolio of a base policy and several post-trained checkpoints, then apply an online per-state ϵ\epsilon-greedy contextual-bandit router that maintains an exponentially updated policy-success table indexed by initial-state ID. Experiments on eight LIBERO tasks show that the approach improves final-round success on tasks with low two-round “split” counts.

Critical strengths
  1. The high-level idea of leveraging otherwise underperforming checkpoints is interesting. Instead of simply discarding checkpoints with lower overall success rates, the paper explores how their complementary capabilities can be exploited to improve performance.
Critical weaknesses
  1. Unrealistic state assumptions and a problematic evaluation protocol. First, the method assumes a fixed, discrete set of identifiable initial states, which poorly reflects continuously varying real-world configurations. Second, Algorithm 1 obtains the state ID by taking the episode index modulo the number of states, relying on a predetermined evaluation order rather than identifying the state from observations. Third, the evaluation protocol does not demonstrate generalizable online adaptation, as the router repeatedly receives feedback on the same initial states used for final evaluation. It therefore only learns which checkpoint solves each recurring instance, without establishing transfer to unseen initial states, let alone other tasks.

  2. Insufficient experimental coverage and unconvincing baseline comparisons. The experimental evaluation is limited to only eight LIBERO tasks. Comparisons rely mainly on conventional bandit algorithms, such as UCB1 and Thompson Sampling, alongside a simplified AgentNet-inspired method. Comparisons with more recent and competitive routing methods are needed. Given the restrictive evaluation setting, limited task diversity, and insufficient baseline coverage, the empirical evidence is unconvincing.

Questions for authors
  1. How would this method work in realistic robotic scenarios with continuously varying initial configurations, where exact state IDs are unavailable and states rarely recur?

  2. Is it possible to provide additional evaluations across more diverse tasks, other VLA architectures, and different initial-state distributions to support the generality of their conclusions?

  3. Is my understanding correct that each individual LIBERO task is independently trained multiple checkpoints using data collected from it and then evaluated on the exact same task? If so, does this method have cross-task generalization? Or is it able to train all tasks as a whole ?

  4. Why does t07 use only three members trained with different objectives or preference-pair construction rules, while most other tasks use five DPO members trained with different random seeds?

Reviewer_3URQOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether VLA checkpoints with weak aggregate scores can still be useful because they succeed on different initial states. It keeps a base OpenVLA policy and its post-trained variants as a portfolio. A tabular contextual bandit uses the initial-state index and observed episode outcomes to select one policy per episode. Experiments on eight LIBERO tasks show substantial final-round gains on tasks with stable per-state outcomes, with little consistent benefit on the less repeatable tasks.

Critical strengths
  1. The checkpoint complementarity is demonstrated convincingly. On obj3, all five members score below the base policy, yet at least one member succeeds twice on 11 of the 12 states where the base fails twice. This is a concrete reason to reconsider discarding checkpoints based solely on their average score. The obj2/obj3 evaluations made after fixing the method, together with the reported prospective predictions, strengthen the empirical case.

  2. The analysis of when routing works is another useful part of the paper. Removing state context largely removes the gain, and the failure cases on unstable tasks are reported alongside the positive results. The random-LoRA control is informative too: it suggests that policy diversity itself accounts for much of the benefit, while DPO adds a further gain on obj3. These controls help explain the result rather than merely present a higher success rate.

Critical weaknesses
  1. The demonstrated deployment setting is unusually favorable to a lookup-table router. It receives an exact simulator-provided state ID, revisits the same 50 starts, and is scored after three earlier visits to each one. The additional novel-state experiment likewise repeats a set of 20 identified states ten times. This establishes online adaptation to recurring, identifiable starts, but leaves the practical state-matching problem unresolved. The authors acknowledge this limitation; it nevertheless substantially restricts the significance for VLA deployment. Evidence using observation-based matching, small perturbations of the initial states, or less frequent recurrence would make the application much more convincing.

  2. The main result measures performance at the end of adaptation, while the cost of learning the routing table is paid during deployment. Figure 3 reports the four-round trajectory, but Table 1 uses only the last 50 episodes. Summing the listed round counts gives 87% and 83% over all 200 episodes on obj3, compared with the reported final-round 94% and 90%. Those cumulative scores still improve on the separately measured base score of 75%, but they quantify a smaller deployment-wide benefit. With only two routing runs per setting and base evaluations performed in separate processes, the size and reliability of that benefit remain less well established than the headline final-round gains. A matched full-horizon comparison is needed to assess the online trade-off.

Questions for authors

How much of the gain survives observation-based state matching or small initial-state perturbations, and how does it vary with revisit frequency? Could the main comparison include cumulative success over the complete 200-episode budget, with matched fixed-policy controls and more independent runs? Given the random-LoRA results, how consistently do post-trained portfolios outperform equally sized random portfolios across tasks, and what additional construction cost do they require?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.