acceptodds
Withdrawn from ICLR 2027

From Association to Decision Value: Testing Paired Instruction Dependence for VLA Corpus Selection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Choosing among robot demonstration corpora is a ranking problem: an upstream diagnostic must preserve which candidates produce better downstream policies, not merely correlate with average performance. We study *paired instruction dependence*, the rate at which a demonstrator changes its first decision when only the goal changes. Across 136 independently seeded quadruped-navigation corpora, predictions developed on 56 corpora are evaluated prospectively on 80 held-out corpora. The audit is reliable and associated with downstream paired correctness, and lowers mean absolute prediction error when added to a correctness-aware baseline, but the primary proper-log-score increment is +0.059 nats (95% corpus-bootstrap CI [-0.027,0.133]) and the prespecified top-20% comparison does not establish an incremental selection gain. An exploratory response decomposition suggests a structural source of this predictive-ranking mismatch: under the tested predictor specification, separating both-correct and both-wrong responses yields larger held-out gains than their unsigned switch statistic. Seed-to-seed agreement indicates that the held-out endpoint retains recoverable rank signal. These results show that counterfactual response structure can be reliable and predictive while providing limited incremental decision value in this setting when compressed into an unsigned statistic, motivating diagnostics that preserve outcome-relevant structure and are validated against the decisions they are intended to support.

Primary area:
applications to robotics, autonomy, planning
Keywords:
vision-language-action models, robot demonstration data, corpus selection, dataset auditing, instruction following, counterfactual evaluation, data valuation, prospective evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.40 over 5 reviews, above 9% of the others.
Reviewer_N6FzOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • The authors investigated approaches for ranking ahead of time how effective of time a set of proposed datasets will be for model training. They collect 56 datasets to train their ranking predictors on, and evaluate them on an additional 80 datasets.
  • They explore three dataset metrics used to inform dataset choice - (1) if the agent has two different actions when presented with two different goals (2) if both the actions are correct, (3) if both the actions are incorrect.
  • They find that the different-goals metric did not significantly improve prediction, but the both-correct and both-incorrect metrics did improve these
Critical strengths
  • This is an interesting and understudied problem to tackle, as predicting dataset efficacy before training could lead to significant and more efficient data collection leading to higher downstream performance. The authors indeed found that two of the three metrics were able to significantly improve predictive performance
Critical weaknesses
  • The paper is missing a strong theoretical or intuitive justification in the choices of the three dataset metrics, they mostly seem like arbitrary choices done as exploratory work (which is how the latter two are framed by the authors).
  • It's unclear how this methodology is intended to be operationalized in other domains. In the paper, only one domain is analyzed, and even if the metrics analyzed produced significant improvements for ranking datasets in this domain, which they did not, the appropriate test should be if these metrics improve predictive performance and ranking in multiple completely unrelated domains. Otherwise, we would have to collect multiple datasets and run these experiments for each new domain, defeating the purpose of identifying datasets.
Questions for authors
  • How do you see this methodology being used across domains?
  • What theoretical and empirical justifications do you have for the three metrics you identified in this paper generalizing, and what insights can you draw from this?
Reviewer_cvCrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

I am unable to say for certain what this paper is about, even at broad strokes. My best estimation is that it aims to develop a measure of corpus quality for fine-tuning an off-the-shelf vision-language-action policy for navigation by measuring language-level counterfactual perturbations that change the goal target. It is possible that the changing goal is audited against a "demonstrator" policy that is providing the rollouts for later fine-tuning. The paper is thick with LLM-generated sentences, ambiguous or undefined terminology and notation, and does not ever answer in plain language important questions like "What did you do?" or "Why is what was done important?"

Critical strengths

None

Critical weaknesses

The abstract and Figure 1 on page 1 are so thick with undefined terminology that it's really difficult to parse what this paper is doing, even in broad strokes. The title suggests VLAs are involved, but page 1 includes no language instructions or mention of VLAs aside from the title itself. Long LLM-generated sentences further obfuscate the point, such as the Abstract's "An exploratory response decomposition suggests a structural source of this predictive–ranking mismatch: under the tested predictor specification, separating both-correct and both-wrong responses yields larger held-out gains than their unsigned switch statistic."

[TODO] "We call the resulting switch rate paired instruction dependence, A_d. Unlike ordinary correctness, it uses a controlled same-state intervention and tests whether the response depends on the instruction." > Why rename existing techniques? This "switch rate paired instruction dependence" sounds like language-perturbation contrast sets, explored in robotics for VLAs (COLOSSEUM), VLN (Contrast Sets), and in natural language for evaluation generally (Gardner).

The introduction makes frequent use of terminology that has not yet been defined, which again makes it hard to tease out what exactly this paper is doing. For example, "With binary left/right responses, A_d is exactly the sum of these both-correct and both-wrong cases; the observed HALT action makes that identity approximate rather than exact in our data." The "observed HALD action" is not something that the reader has been told anything about yet.

Another LLM-thick sentence in the introduction mentions "improve selection and prediction" but the reader still has not been told what is being selected or predicted. "The relevant question is therefore narrower than whether language sensitivity matters in general: does unsigned same-state response structure improve prediction and selection after marginal demonstrator correctness is known?"

Introduction contributions 1 and 3 are composed entirely of undefined and ambiguous terms. I think I am able to get the broad strokes idea of contribution 2, though "held-out gains" are not defined (gains in what???). Contribution 1: "A prospective decision-value evaluation: we separate reliability, association, proper probabilistic prediction, and top-k selection for a same-state robot-data diagnostic, using the corpus as the unit of inference." Contribution 3: "Decision resolution and practical alternatives: a cross-seed resolution benchmark establishes recoverable corpus ordering, while demonstrator and proxy baselines locate the audit relative to realistic selectors."

The Abstract and Introduction read like they may have been entirely or heavily LLM-generated, with terminology that is not standard in the field, vague or undefined research outcomes, and semantics that seem to orbit a missing central point (for example, the answer to the question "What did you do?")

Section 3 should clearly explain the method to the reader. Currently, it defers to Figure 2, which contains many of the same pieces of terminology we've encournered so far that aren't defined. Then 3.1 begins immediately with "For each audit state, we query the demonstrator twice with an identical visual observation and action history." At this time, "audit state" is not defined, where an audit state comes from is not defined (each over what), and the "demonstrator" is not defined, which means there's no way for us to interpret what querying it means. It is very difficult to tell what is happening in this paper and what the authors want to convey. The very next sentences continues the issue, "Only the object goal changes, and the two goals require opposite branches." with "object goal" not defined, and "branches" not defined or clear as a term in literature (either VLA or VLN).

Section 3.2 ends with another iteration of the LLM-thick statement of a research question that is composed of undefined and ambiguous terms, "The decomposition makes the primary question precise: does unsigned joint response structure improve prediction beyond marginal correctness and corpus composition?"

Equations (3,4,5) start making the paper feel completely illegible. Alpha and beta are never defined but might be meant to come from the binomial draw (that is not clear). A weight matrix(?) W_d somehow "contains marginal per-goal demonstrator correctness, training-goal entropy, and coarse trainingaction entropy." A value η appears in equation (5) but is never defined and does not appear again in the paper at all. γ_s is defined as a "cross-seed effect" and "nuisance term", neither of which have obvious meaning in context.

3.4 opens with more technical jargon that feels like it borders on gibberish to describe equation (6). "The primary estimand is the equal-corpus improvement in proper log score". What is "proper log score"?? The primary estimand of what?? Section 4.3 later describes equation (6) as "the primary metric".

After equation (6) there are some inscrtutable sentences " The prespecified primary rule requires its lower bound to exceed zero. The 0.105 ≈ − log(0.9)-nat magnitude reference corresponds to multiplying the geometric-mean assigned probability by approximately 1.11; it is not a universal utility threshold. " What do these correspond to? Are these about soemthing in equation (6)?? These values are not present anywhere else in the main paper, but the mention of "0.105 nats" happens again in the Appendix.

I feel that reading through the Method section multiple times I am still unable to determine what exactly this paper is aiming to do, what it did do, and what the authors are trying to communicate.

The idea of a "demonstrator" is finally defined in section 4.1, over halfway through the paper's main text.

5.1 through 5.5 throw a lot of results at us densely and quickly, almost with every sentence mixing in-line, table, and figure results. I still cannot tell with any certainty what is being measured or why. The language reads again and again like dense LLM output summarizing some kind of database of experiments.

The conclusion is an opportunity to repeat and reiterate findings in clear language, but in this paper it includes run-on LLM sentences thick with jargon, such as "The switch rate is measured with substantial reliability and associated with learner behavior, but its unsigned form mixes both-correct and both-wrong responses; correctness resolved parameterizations yield larger exploratory held-out gains under the tested predictor specification." making it hard for the reader to determine, even in summary, what the paper is about.

The AI statement "Language models assisted with code refinement and grammar review. The authors verified all suggested changes and remain fully responsible for the manuscript content." seems unlikely to be true to this reviewer. There are many, many sentence structures in this paper that LLMs are known to overuse and that are not popular or natural for most human writers. An LLM did not just conduct "grammar review" of this paper; this reviewer asserts that much of this paper was copy-pasted from an LLM output box.

Questions for authors

The introduction establishes the paper goal as another LLM-thick sentence, "The practical question is consequently decision-focused: can a measurement rank candidate corpora well enough to select the ones that will produce better policies?" But is this a question that folks doing VLA training actually face? In my experience in robotics, the availability of corpora at all is the issue, not the subselection from among candidate trajectories or whole corpora. Later, in Section 3, this limitation is made more explicit with "It is therefore applicable in settings such as automated simulation or online data generation, where the same state can be replayed under alternate goals before committing to a large collection. The study therefore concerns this access model rather than arbitrary human demonstrations." What data settings for VLN or VLA training practically look like this? The availability of human demonstration data is typically the bottleneck in these settings within robotics.

Reviewer_cjzPOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether a simple diagnostic on a demonstration dataset can predict which dataset is best for fine-tuning a vision-language-action (VLA) model. The proposed diagnostic measures how often the demonstrator changes its action when only the goal is swapped. In a simulated quadruped navigation task, they find that this measure is reliable and correlates with downstream performance, but it does not significantly improve prediction or dataset selection over a correctness-based baseline. The paper presents this as a carefully pre-planned evaluation with a mostly negative result.

Critical strengths
  • The paper studies an important and timely problem for VLAs.
  • The paper has rigorous evaluation design.
  • It also reports null primary result and limitation honestly.
Critical weaknesses

1. The headline result 'predictive-ranking mismatch' is itself not statistically established

According to table 6, the primary log-score gain CI is [−.027, .133] and the selection difference CI is [−.099, +.023]; the resampled selection difference is +.0019. No CI is given for the Spearman difference (.589 vs .490). Fig 1 and the abstract build the narrative 'lower error, weaker ordering' on point estimates that are all consistent with zero effect.

2. Toy, non-standard setting with very limited external validity

Demonstrators are tabular Q-learners over 24 discrete states (Table 17); corpora are ~300 steps; the endpoint is a single binary first turn; the 'VLA' is a LoRA'd Qwen VLM emitting discrete commands. No standard VLA benchmark (LIBERO, CALVIN, SimplerEnv), no Open X-Embodiment-scale corpora, no manipulation, no human teleoperation data. The corpus population is largely driven by an artificial maturity knob (Sec 6, Table 4).

Questions for authors
  • What is the bootstrap CI for the Spearman difference (M1M_1−M0M_0) and for seed-to-seed selection vs M0M_0? Is the 'ranking mismatch' distinguishable from noise?
  • Can you test the diagnostic on a standard benchmark (e.g., LIBERO or SimplerEnv with scripted/planner demonstrators that allow counterfactual queries) and a standard VLA such as OpenVLA, Octo, or π0\pi_0?
  • How would the audit apply to human-teleoperated corpora, the dominant real use case?
Reviewer_ShwoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies paired instruction dependence as a diagnostic for choosing among robot demonstration corpora before training downstream policies. It takes the rate at which a demonstrator changes its first decision when only the goal changes, and uses it, together with demonstrator correctness, to select corpora and train VLAs on them. After detailed analysis, the conclusion is that this diagnostic does not significantly help select better corpora beyond demonstrator correctness to enable better policy behavior.

Critical strengths
  • Having good data selection before downstream VLA training is a good motivation, as it can avoid the cost of training a policy on every candidate corpus and help identify higher-quality data.
  • The paper conducts a good and honest analysis of the relationship between the diagnostic and downstream policy performance, clearly reports that it does not improve selection, and discusses its limitations.
Critical weaknesses
  • As the authors report, although the diagnostic is analyzed in detail, it does not bring a clear performance improvement. Adding it to the baseline gives only +0.059 nats (95% CI [−0.027, 0.133], crossing zero), and top-20% selection does not improve (.838 → .811). The paper's own results also show that other selectors work better, such as training-trajectory correctness (+.156 nats) and a small proxy model (Spearman .715).
  • The task setting is very limited and narrow: a simple simulated navigation task, tabular Q-learning demonstrators, and only 48 short trajectories per corpus. The evaluation also focuses on correlation and first-step correctness instead of generalization or task success rate. It is unclear whether the findings transfer to other VLA data and more diverse tasks.
Questions for authors
  • Why choose the first decision to judge data quality? In practice, more diverse data may actually be better for current policies, and judging by the first decision alone may miss this. Is there a reason or evidence for this choice?
  • The paper includes a controlled shift, but the scene, objects, camera, and instructions stay the same. Could the method be tested on more task settings, e.g., similar tasks that require more generalization ability?
  • The data here is self-created, and whether a corpus is better or worse is designed in a very specific way (mainly by demonstrator training level). The authors note that the audit requires re-querying the demonstrator under different goals (Sec. 3.1). Can this be applied to real-world VLA data and tasks, e.g., human demonstrations?
  • Since correct labels are available in simulation, why not just filter out the failed demonstrations instead of selecting whole corpora?
Reviewer_u6SXOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a VLA data selection strategy and evaluates a particular design choice to see if it improves the Qwen-based VLA policy's performance for Unitree Go2 in a navigation task. This data selection strategy is to learn a predictor that takes 3 statics of the demonstration and output how often a VLA trained on that corpus will choose the correct first action.

The particular design choice the authors test is whether to add a new "unsigned goal-swap rate" as part of the predictor's input. This rate measures how often the demonstrator changes its first action when only the goal changes. The intuition is that, at the current state, when two goals require opposite actions, a change in action may indicate that the demonstrator is responding to the instruction changes.

However, over the empirical results in simulation, they finds no clear gain of this new input.

Critical strengths
  • The data selection strategy direction the paper investigates is worth exploring for robot learning.
  • The empirical experimental design is sound and reported in detail. The confidence intervals are included for main results.
Critical weaknesses
  • The main criticism I have is that it has very unclear motivation. The paper spends much effort to show the limitations of a particular design choice of a particular data selection strategy. What I could not find in the paper is an example where anyone uses this design to select demonstration data, in the related robot learning settings. Without a concrete use case, I find the impact of the negative result quite limited.
  • The state relies "opposite first action", such as turning left or right. It is unclear how this would generalize to continuous action spaces in manipulation or whole-body control.
  • The mathematical explanation in the paper is fairly straight-forward, with most of the derivations are already known.
Questions for authors
  1. Is there a prior work in which the "unsigned goal-swap rate" is used to rank demonstration corpora? If this is a new use of the statistic, what motivated it in a setting where correctness/success labels are already available?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.