- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
LogicVLA introduces a task-conditioned neural logic consistency critic (NLC critic) beyond behavior cloning, evaluating whether predicted actions are compatible with language instructions, visual observations, and robot state. Trained on matched and in-batch mismatched samples from the original demonstrations and removed after training, the critic adds no deployment-time parameters or computation. Using OpenVLA-7B and OFT fine-tuning, the authors evaluate on LIBERO-Spatial with three seeds, plus ablations and paired statistical tests. At 15K steps, overall success rises from 94.4% to 97.0%, and success on the fixed hardest (stove) task from 68% to 94%. LIBERO-Long experiments mainly verify co-training stability, without a complete closed-loop comparison.
Critical strengths
-
Task–action compatibility serves as auxiliary supervision for direct policy optimization. The key contribution is applying neural logic to supervise VLA-generated actions, so the objective accounts for action–context relations and supplements pointwise behavior cloning. This supervision learns directly from existing demonstrations without extra annotations, reward design, or environment interaction, offering a valuable direction for better demonstration-data utilization.
-
Meaningful early-training gains are achieved without deployment burden. The critic is used only during fine-tuning, with the original policy structure retained after deployment. In Table 1, success at 4K steps rises from 84.6% to 91.6%, with main results covering three seeds, indicating potential to improve performance under limited training budgets in the current setting.
Critical weaknesses
-
Key controls are missing, so the logical structure cannot yet be confirmed as the main source of improvement. Table 2 mainly compares the full method with removing the entire consistency supervision; it neither compares against an ordinary scoring network of similar parameter count nor separately removes the logic-algebra regularization (Eq. 9). The controls are not fine-grained enough to isolate the contributions of logical composition/operators versus logic-algebra regularization.
-
The added critic introduces training sensitivity, with insufficient evaluation of its stability and efficiency cost. Table 2 shows success dropping to 1.8% without warmup and to 0% with consistency weight 1.0, indicating the auxiliary supervision may substantially interfere with policy learning. Existing ablations use only a single seed and do not verify each stabilization mechanism separately. Moreover, without reported training time or total computation, fewer training steps do not directly establish a proportional reduction in wall-clock time or total computation.
Questions for authors
- With the same inputs, negative-sample construction, warmup, and stabilization mechanisms, how does a plain MLP scoring network with a comparable parameter count perform compared with LogicVLA? How does performance change when the logical-identity penalty in Eq. (9) is removed while retaining the latent-magnitude regularization term?
- Why is an expert action chunk from another sample in the same minibatch treated as a valid negative for the current context? Has there been any analysis of “mismatched but still reasonable” actions and their potential false-negative effect on critic learning?
- Can ablations of warmup, consistency weight , and each stabilization component be provided across multiple training seeds, to show whether the results depend on a specific random seed or a narrow hyperparameter setting?
- Please clarify how the McNemar tests in Table 5 account for the three training seeds: are they computed per seed, on pooled episodes across seeds, or using a single selected training seed? Please also report the discordant counts (b, c) or more complete statistical results.
- On the same hardware (4×A800-80GB), what are the wall-clock time/GPU hours required to reach comparable success rates, and what is the per-step training overhead of the auxiliary critic?
2 comments
Dear Reviewer Reqs,
We hope the above clarifications and the additional experiments in the revised draft sufficiently addressed your concerns. We remain committed to addressing any remaining points you may have during the discussion phase.
If you are satisfied with our responses, we would greatly appreciate your consideration in adjusting the evaluation scores accordingly.
W1: Separating logical composition from regularization
Removing consistency supervision alone does not isolate logical composition from algebraic regularization, so we added two controls — a plain MLP (structured design versus a generic scorer) and removal of the identity penalty. Both underperform the full model at 15K under the same stabilized setup; Q1 gives the design, table, and reference.
W2: Training sensitivity and critic overhead
The 1.8% (no-warmup) and 0% (λ = 1.0) results in Table 2 used the earlier recipe, before three-layer stabilization; they show that recipe was sensitive to how the critic was introduced, not whether it still collapses once stabilized — with all three mechanisms the full model is repeatable across seeds (Q3). The added cost is ~+23% per step, so a tenfold step reduction is not a tenfold compute reduction (Q5).
Q1: Plain MLP and Eq. (9) ablations
The plain MLP replaces the learned logical composition and truth-vector readout with a feed-forward scoring network over the same four projected modality vectors (736,873 parameters versus 736,833 for the full critic). We held the backbone, input projections, pairs, ranking margin, warmup, schedule, and all three stabilization mechanisms fixed; for the Eq. (9) control we set α = 0 while retaining the latent-magnitude term weighted by β, keeping the architecture unchanged.
Counts and continuity-corrected McNemar p-values are from 500 paired rollouts for one usable seed. The MLP tests whether generic compatibility scoring recovers the full result under matched conditions; the Eq. (9) control isolates removing the identity penalty. All rows use stabilization, distinct from the pre-stabilization Table 2 results.
Q2: Plausible negatives and false-negative supervision
We treat each permuted expert chunk as a candidate negative because it was not demonstrated for the current context, not because it is known invalid. Reassigning in-batch chunks by ρ(b)=(b+⌊B/2⌋)modB preserves the action distribution and avoids a real-versus-noise shortcut, but can produce plausible negatives when two contexts admit similar actions.
The ranking hinge max(0,m−(s+−s−)) yields zero loss once the positive exceeds the negative by at least the margin; if a plausible negative scores high and the gap is below the margin the pair is still penalized, so the hinge does not eliminate false-negative supervision. The objective scores relative compatibility, not action validity.
Q3: Stabilization and repeatability across seeds
We evaluated the full three-layer stabilization on two training seeds: at 15K the full model achieves 99.2% and 98.4% for seeds 2 and 42. The warmup and consistency-weight results in Table 2 characterize the sensitivity of the earlier recipe; the two stabilized runs show similar success across those seeds but do not isolate the effects of warmup, λ, or any single mechanism.
Q4: Per-seed McNemar tests and discordant counts
We applied McNemar's test separately to each of the three training seeds in Table 5: for a given seed, the 500 matched rollout conditions for LogicVLA and its baseline form one paired contingency table — not pooled across seeds, not a single selected seed. For the new single-seed critic controls, b = paired rollouts where the full critic succeeds and the control fails, c = the opposite (the (b, c) columns of the Q1 table); these address paired evaluation outcomes given the trained checkpoints, not uncertainty across training seeds.
Q5: Four-GPU timing and per-step overhead
On four A800-80GB GPUs at batch size 16 per GPU, the no-critic OFT run averages 2.70 s per step versus ~3.31 s for LogicVLA — about 0.6 s more, or roughly 23% additional training cost per step. Because the logged runs use different GPU counts, we compare GPU-hours, not raw wall-clock.
At the four-GPU step rates, 15K LogicVLA ≈ 55 GPU-hours and 150K OFT ≈ 451 (table) — estimates at the respective recipe lengths, not timed runs to an identical threshold. With reported Spatial outcomes of 97.0% and 97.6%, LogicVLA uses roughly one-eighth of the published OFT recipe's GPU-hours at similar success; the critic's cost is training-only.