acceptodds
Under review as a conference paper at ICLR 2027

Do World-Action Models Need Dense World Context? Progressive Sparse Forcing for Action DiTs

Abstract

World-Action Models (WAMs) built on Diffusion Transformers (DiTs) have become a powerful framework for embodied control, generating actions by denoising action tokens conditioned on visual world states. Across diverse visual conditioning layouts, existing WAMs typically ground action prediction through dense action-to-world attention at every DiT layer. This raises an underexplored question: which visual tokens do noisy action tokens attend to during denoising? Analyzing WAMs under mainstream conditioning layouts, we find that dense conditioning often becomes homogenized across depth and task phases, and frequently anchors on task-irrelevant regions rather than action-relevant physical cues. To make visual grounding adaptive, we introduce Progressive Sparse Forcing (PSF), a general action-to-world conditioning framework that converts dense visual-context aggregation into active visual focusing. PSF couples a depth-decaying sparsity schedule with an end-to-end learned action-world router, forcing noisy action tokens to select denoising-relevant visual context from the original training objective. On robotic manipulation, PSF improves closed-loop success on LIBERO and substantially strengthens zero-shot spatial robustness on LIBERO-Plus. On autonomous driving, it improves long-horizon planning performance across 4K-100K training clips from the PhysicalAI-AV dataset. Beyond accuracy, PSF reaches strong performance and reduces inference computation via sparse conditioning. Code will be publicly released.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World-Action Models, Diffusion Transformers, Visual Grounding, Progressive Sparse Forcing, Robotic Manipulation, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_ezYkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The reviewer uses "human" to denote my hand-written opinions in the review and the reviewer uses "Opus" to denote Opus 5.5. The reviewer uses the incognito mode so no data will be shared to Claude.

Human: This paper proposes a progressive sparse forcing mechanism to reconfigure the action-to-world conditioning. The proposed approach DAAR scores the usefulness of the visual token w.r.t. the action token. It outputs an action-world relevance matrix (eq.3). It then gathers only the most relevant attention values (eq. 5), which is called the sparse-forcing attention. Then it uses a progressive schedule to ensure the later stages contain increasingly compact visual visibility.

Critical strengths

Human:

  1. The proposed mechanism is somewhat novel.

  2. The DARR is simple and probably at low cost. It could generally benefit the domain.

  3. It demonstrates strong performance on two distinct domains: robots and AD.

Critical weaknesses

Human:

  1. No video demos are provided. This is required if the authors claim advancement in closed-loop simulation.

  2. The author claims they have done experiments in autonomous driving scenarios. However, no video demos or figure visualizations are provided. Only presenting numbers is not sufficient to demonstrate the superiority of the proposed approach. In AD scenarios, the reviewer wishes to see whether the proposed approach could change agent behavior or not.

  3. The closed loop simulation performance are saturated (>98%). It's hard to say whether the proposed mechanism is really better than the baseline.

  4. It's hard to see whether the proposed mechanism would deteroiate the base model performance because of the proposed mechanism.

  5. Figure 1 isn't clear and doesn't make much sense. It's hard to know how the "attention values" are computed and what kind of normalization is applied. And it's hard to say whether larger values mean "stronger attention". It's hard to say whether the visualization really discovers an issue. Probably manipulating the features would harm the base model performance. It's hard to say whether a mistake may exist in such a visualization, which may reverse the conclusion.

  6. Opus finds a lot of unresolved issues.

Opus (verified by the reviewer):

The LIBERO gains are within noise, and no variance is reported anywhere. LIBERO is near saturation: the overall gains are +0.5 (IDM), +0.6 (JD), and +1.0 (FastWAM), from 50 trials per task and, as far as I can tell, a single training seed. JD-WAM even drops on Spatial (99.0 → 98.6). Without seeds or confidence intervals, these results show parity at best. The same concern applies to the per-axis LIBERO-Plus numbers, where some axes contain only a few hundred episodes, and to the Table 5 schedule ablation, where all variants fall within 96.6–97.6. The driving results also have no variance across training runs.

The control needed to support "learned visual focusing" is missing. The central claim is that PSF works because the router learns to select action-relevant visual tokens. But restricting visual visibility in deep layers is also a form of structured token dropout, which could plausibly explain the OOD gains on its own, since it reduces reliance on spurious visual context. The ablation (Table 5, left) compares only schedules of the same learned router against the dense baseline.

The driving evaluation is limited, and the efficiency analysis is incomplete. Driving is open-loop ADE/FDE only, with no collision, off-road, or closed-loop metrics. "Long-horizon planning" is a stretch for 3–4 s horizons with the cache refreshed from ground-truth observations at every chunk.

Questions for authors

Opus:

How does PSF compare with random Top-K at the same schedule, with score-bias-only (no sparsity), and with Top-K from backbone attention logits, on LIBERO-Plus?

Can you quantify grounding, for example as attention mass on object, gripper, and target masks in LIBERO, for dense vs. PSF?

What is the latency of dense DriveWAM at v = 1, a = 2, and which single configuration does "+PSF-b" correspond to?

How many training seeds were used, and what is the variance on LIBERO, LIBERO-Plus, and driving?

How do the reproduced baselines compare with the published FastWAM, Lingbot-VA, DreamZero, and DriveWAM numbers, and how were PSF's hyperparameters selected?

How does performance change as the deepest-stage retention drops below 0.4?

8 comments

We sincerely appreciate Reviewer ezYk’s recognition of the mechanism’s simplicity, potential usefulness, and results across manipulation and driving. Your detailed comments have helped us identify where additional evidence and clearer explanations are needed.

Several concerns are addressed in our shared official comments:

  • G1: Controlled Ablations and Sources of Improvement addresses the reviewer-verified Opus concern about structured token dropout and Q1. We add random Top-K, dense score bias, backbone-attention Top-K, and constant-sparsity controls.
  • G2: Training-Run Variability and Layout Dependence addresses the reviewer’s W3–4, the Opus concern about unreported variance, and Q4. We report three-run mean ± SD for baselines and PSF variants on LIBERO, LIBERO-Plus, and offline driving.
  • G3: Design Motivation and Evidence for Learned Visual Conditioning addresses the interpretation of attention maps raised in the reviewer’s W5 and provides additional evidence relevant to Q2 through a matched selected-versus-unselected intervention.

Revisions addressing the shared concerns are marked in red in the revised manuscript; their locations are listed in our general official comments.

Below, we address the remaining concerns individually, including the visualization procedure and the requested experimental details.

1. Video demonstrations

Addresses the reviewer’s Weaknesses 1–2.

We apologize for not providing sufficient video demonstrations in the original submission. We have added closed-loop manipulation videos on LIBERO, rollouts under LIBERO-Plus perturbations, and open-loop driving trajectory visualizations to the supplementary material.

1.1 Additional closed-loop driving evaluation

Addresses the reviewer’s Weakness 2 and the concern about open-loop driving evaluation.

Following your suggestion to examine whether PSF changes driving behavior, we adapt DriveWAM and DriveWAM+PSF to closed-loop simulation on the official local AlpaSim validation set, following the DriveWAM baseline settings and adapting its standard outputs to the simulator. Both variants use the same evaluation protocol.

MetricDriveWAMDriveWAM + PSF
Close Encounter, all ↓40.0%36.1%
Close Encounter, at-fault ↓16.0%14.0%
Off-road Rate ↓22.0%19.0%
Progress ↑40.9%41.2%
AlpaSim Score, all ↑0.06000.0970
AlpaSim Score, at-fault ↑0.15400.1958

Compared with the adapted baseline, PSF reduces close encounters and off-road rate while improving progress and both AlpaSim scores. The closed-loop videos further illustrate the resulting driving behavior. We have added the evaluation protocol and results to the appendix, with reviewer-specific additions marked in purple,

Because the baseline model DriveWAM was not originally evaluated on this closed-loop benchmark, we treat this as an exploratory comparison of matched adaptations, rather than a comparison with specialized closed-loop driving systems.

2. Attention computation and visualization

Addresses the reviewer’s Weakness 5 and the visualization aspect of Q2.

How are the attention values computed?
The displayed values originate from post-softmax attention weights. Softmax is computed over all allowed video and action keys. For each map, we average over attention heads and the first 10 action queries, corresponding to the first 0.5 seconds of actions. We then display the primary-camera visual tokens from the first conditioning frame as a 7×7 grid.

What do the colors and gray grids represent?
Each heatmap is independently normalized for display. Colors therefore indicate relative attention within a panel, rather than comparable absolute attention magnitudes across layers or models. In the PSF visualizations, gray grids denote visual tokens excluded by the router; they are not inferred from low attention values.

What does this visualization establish?
The maps illustrate spatial attention patterns in the displayed examples; they do not independently establish causal importance or object-level grounding. G3 provides complementary behavioral evidence through matched selected-versus-unselected feature perturbations. We have clarified the visualization procedure in Appendix D.4 of the revised manuscript.

3. Matched-step driving latency and the definition of “+PSF-b”

Addresses Q3 and the reviewer-verified Opus concern about inference efficiency.

Driving performance and latency under explicit denoising configurations.
We provide the requested dense v=1,a=2v=1,a=2 result alongside PSF at the same budget and the selected DriveWAM-best configuration.

MethodVideo stepsAction stepsLatency (ms) ↓ADE@3s ↓FDE@3s ↓ADE@4s ↓FDE@4s ↓
DriveWAM12409.10.5971.4480.9452.465
DriveWAM-best15936.90.4801.3100.8302.380
DriveWAM + PSF12436.70.4661.2580.7932.239

At the same two-step action-denoising budget, PSF improves all four trajectory metrics with an additional 27.6 ms (6.7%) of latency. As detailed in “Latency breakdown and speedup attribution,” of our response to Q6bN, routing and selection overhead can outweigh sparse-attention savings for small, single-camera inputs, whereas computational benefits emerge with larger visual contexts, including higher-resolution and multi-view inputs.

Our primary goal is to improve action prediction. Compared with DriveWAM-best, two-step PSF achieves lower errors on all four metrics while reducing latency from 936.9 to 436.7 ms (2.15× speedup), demonstrating improved accuracy with fewer denoising steps.

What does “+PSF-b” denote?
We apologize for the ambiguous notation. “b” denotes the selected best-performing configuration, with one video-denoising step and three action-denoising steps. Its reported latency (573.1 ms), ADE@4s (0.76), and FDE@4s (2.21) all correspond to this single configuration, rather than separately selected settings for different metrics. We have clarified these video/action step counts in Section 4 and Tables 2–3 of the revised manuscript, with reviewer-specific changes marked in purple.

4. Baseline reproduction and comparison with reported results

Addresses the baseline-comparison part of Q5.

Implementation and comparability.
We reproduce FastWAM using its official code. For the other manipulation layouts, we follow the same codebase to train a non-autoregressive IDM-WAM following the Lingbot-VA architectural design and a JD-WAM following the DreamZero-style joint-denoising design. These are controlled layout implementations. In particular, Lingbot-VA uses an autoregressive rollout framework, and DreamZero does not provide corresponding LIBERO results for a direct comparison.

Below, we compare reference scores with our reproductions. The IDM-WAM and JD-WAM reference rows concern the corresponding layout variants, not the complete original models.

The reference results do not report multi-seed statistics, whereas our reproductions report mean ± SD across three independent training runs. Training-run variability may therefore contribute to the differences between the reported and reproduced scores, although the available reference results do not allow us to quantify this contribution.

LIBERO success rates (%).

MethodResultSpatialObjectGoalLongOverall
FastWAMReported98.2100.097.095.297.6
FastWAMOur reproduction95.7 ± 0.999.4 ± 0.597.1 ± 0.892.9 ± 0.896.3 ± 0.7
IDM-WAMReference variant98.897.897.897.698.0
IDM-WAMOur reproduction98.8 ± 0.598.1 ± 0.597.7 ± 0.397.7 ± 0.598.1 ± 0.4
JD-WAMReference variant99.699.498.296.898.5
JD-WAMOur reproduction99.2 ± 0.399.7 ± 0.398.2 ± 0.296.1 ± 0.698.3 ± 0.2

We closely follow the released code and experimental settings for each reproduced baseline. Our overall results differ from the reference scores by −1.3 percentage points for FastWAM, +0.1 for IDM-WAM, and −0.2 for JD-WAM. PSF is evaluated against these reproduced baselines using the same training hardware, training seeds, data, optimization settings, and evaluation protocol.

DriveWAM: offline trajectory errors.

ResultADE@3sFDE@3sADE@4sFDE@4s
Reported0.471.350.832.47
Our reproduction0.47 ± 0.011.34 ± 0.010.84 ± 0.012.49 ± 0.01

The reproduced DriveWAM results closely match the reported values, with differences of at most 0.02 across these metrics.

LIBERO-Plus: comparison with a third-party FastWAM reproduction.
For additional context, we compare with the FastWAM results reported by FasterWAM. These are a third-party reproduction, not original FastWAM results.

FastWAM resultBackgroundCameraLanguageLightingObject layoutInitial statesSensor noiseOverall
Reported by FasterWAM45.7018.7570.0883.2362.7345.6929.8449.14
Our reproduction51.1 ± 5.518.3 ± 2.070.0 ± 0.684.8 ± 1.563.6 ± 0.945.8 ± 1.736.9 ± 6.451.4 ± 1.9

Our reproduction has a higher overall mean (51.4% vs. 49.14%), although this cross-publication comparison is contextual rather than controlled. The main evidence for PSF remains the matched baseline-versus-PSF comparisons in G1 and G2.

We have clarified the baseline implementations in Section 4 and Appendix D.1 of the revised manuscript (marked in purple).

How were PSF's hyperparameters selected?
We selected the three-router configuration based on the stage-count ablation in Table 5 of the original manuscript and fixed the default retention schedule across layouts and benchmarks. The additional comparison with the 0.3-decrement schedule, [0.9, 0.6, 0.3], further supports this choice: it reaches 97.2% overall LIBERO success, compared with 97.6% for the default schedule.

Reviewer_Q6bNOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Progressive Sparse Forcing (PSF), which uses an action-conditioned router to progressively restrict action-to-world attention in deep DiT layers. Experiments evaluate manipulation performance, robustness, and offline driving trajectory prediction.

Critical strengths
  1. Matched comparisons preserve the backbone, training data, and optimization settings, supporting the usefulness of the overall module.

  2. Evaluation spans several WAM layouts and two domains, with substantial robustness improvements for Fast-WAM.

  3. The paper reports routing overhead and includes matched denoising-step sweeps.

Critical weaknesses
  1. Progressive sparsification and learned token selection are established ideas, while PSF’s distinction lies mainly in action-conditioned, per-query selection during denoising. Comparisons with attention-based Top-K selection, fixed sparsity, and dense attention with router biases are needed to isolate this contribution.

  2. Dense visibility does not imply uniform attention, and concentrated attention maps do not establish more effective use of relevant visual information. The current ablations cannot distinguish benefits from sparse selection, additional attention reweighting, or the progressive schedule.

  3. The approximately 3× speedup includes fewer denoising steps; fixed-forward acceleration is smaller, and PSF is slower at 480p. Robustness gains also depend on the conditioning layout: full LIBERO-Plus improves substantially for Fast-WAM but remains essentially unchanged for JD-WAM.

  4. Evaluation is limited to simulated manipulation and offline driving trajectory prediction, with no physical-robot experiments.

Questions for authors
  1. Under matched token and computation budgets, how does PSF compare with attention-based Top-K selection and a fixed sparsity schedule?

  2. Does adding router biases to dense attention recover similar gains without token selection?

  3. What fraction of end-to-end latency comes from routing, Top-K selection, and gathering, and how stable are the smaller performance gains across training runs?

Other Suggestions for Improvement

  • Position PSF more explicitly against progressive token sparsification and action-aware pruning, including DynamicViT and SpecPrune-VLA.
  • Separate acceleration from sparse attention from acceleration through reduced denoising steps.
  • Describe the driving evaluation as offline trajectory prediction and state that the theoretical bounds do not guarantee grounding or monotonic improvement across layers.
8 comments

We sincerely appreciate Reviewer Q6bN’s recognition of our matched experimental settings, evaluation across layouts and domains, and transparent reporting of routing overhead. Your detailed feedback has helped us better isolate PSF’s contribution and clarify the scope of our evidence.

As these concerns are shared across the reviews, we consolidate the corresponding experiments in our shared official comments:

  • Weaknesses 1–2 / Questions 1–2 → G1: Controlled Ablations and Sources of Improvement. We compare PSF against dense router bias, backbone-attention Top-K, random selection, and approximately budget-matched constant sparsity to examine the sources of improvement.
  • Weakness 3 / Question 3 (layout dependence and training stability) → G2: Training-Run Variability and Layout Dependence. We report three-run mean ± SD for both baselines and PSF variants on LIBERO, LIBERO-Plus, and offline driving.
  • Weakness 2 / Other Suggestion 3 (attention interpretation and theoretical limits) → G3: Design Motivation and Evidence for Learned Visual Conditioning. We clarify the role of attention visualization and add a matched intervention showing greater action-error increases when selected visual information is perturbed.

Revisions addressing the shared concerns are marked in red in the revised manuscript; their locations are listed in our general official comments.

Below, we address two further points: (1) latency breakdown and speedup attribution, and (2) positioning relative to DynamicViT and SpecPrune-VLA.

1. Latency breakdown and speedup attribution

Addresses Weakness 3, Question 3 (latency), and Other Suggestion 2.

Separating reduced-step inference from sparse-attention acceleration.
Our intent in reporting the 2.94× speedup over the default DriveWAM configuration is to highlight that PSF achieves lower FDE@4s with only two action-denoising steps (2.24 vs. 2.48), consistent with more effective visual conditioning for action refinement. This comparison demonstrates improved accuracy with fewer denoising steps, rather than a 2.94× acceleration from sparse attention alone. We will revise the wording to make this distinction explicit.

Latency breakdown of the PSF-modified attention path.
We profile the final 20 modified layers during one action-only forward pass. To reduce overhead, we develop a Triton kernel that fuses gathering with attention; these operations are therefore measured jointly. (We will release this implementation with the code.) The table reports cumulative latency across the profiled layers, in milliseconds.

Component480p1080p
Dense attention9.5950.12
PSF: fused gather + attention11.5923.31
PSF: router input processing + scoring3.633.81
PSF: Top-K + mask + bias4.587.37
PSF total19.8034.49
PSF minus dense+10.21−15.63

At 480p, even fused gathering and attention take longer than dense attention (11.59 vs. 9.59 ms). Including routing and selection increases the modified-path latency by 10.21 ms, so PSF provides no latency benefit in this short-context setting. At 1080p, fused gathering and attention save 26.81 ms, outweighing the 11.18 ms routing and selection overhead and yielding a net saving of 15.63 ms. These results show that PSF's computational advantage emerges with longer visual sequences, motivating its use in high-resolution and multi-view settings common in autonomous driving and robotic manipulation.

Moreover, PSF is designed primarily to improve action prediction, with acceleration as an additional benefit rather than its sole objective. We clarify this distinction from efficiency-oriented pruning methods in the positioning discussion below.

2. Positioning relative to DynamicViT and SpecPrune-VLA

Addresses Weakness 1 (novelty) and Other Suggestion 1.

Thank you for suggesting these connections; the table summarizes the shared ideas and differences in selection, training, and objectives.

MethodToken selectionTrainingMain objective
DynamicViTLearned, progressive visual-token keep/drop masksTask loss + teacher distillation + retention-ratio lossAccuracy–efficiency trade-off
SpecPrune-VLAHistorical/local information and layer-specific importance, with action-speed controlPost-hoc pruning without additional trainingInference acceleration
PSFLearned per-action-query selection under a progressive token budgetOriginal action loss through router-score biasBetter action prediction and robustness

DynamicViT already learns progressive token selection. Our distinction is therefore not simply that importance scores are learned. DynamicViT predicts which visual tokens remain in subsequent backbone processing, whereas PSF learns which visual tokens each action query can access under its current denoising state.

SpecPrune-VLA already incorporates action awareness. Its controller uses action speed and granularity to adjust pruning aggressiveness, alongside historical and current information for token selection. PSF instead trains a dedicated scorer conditioned on each noisy action query, rather than applying post-hoc pruning rules.

PSF uses progressive sparsity as a training constraint on action-conditioned visual access. Selected router scores determine visibility and enter attention logits, allowing the original denoising objective to train the router without auxiliary distillation or retention-ratio losses. The goal is improved action prediction and robustness, with computational savings as an additional benefit. G1 supports this combined design: neither dense score bias alone nor learned routing with constant sparsity recovers PSF's OOD gains.

3. Additional closed-loop driving evaluation

Addresses Weakness 4.

To extend our evaluation beyond offline trajectory prediction, we adapt DriveWAM and DriveWAM+PSF to the official local AlpaSim validation set and evaluate both in closed-loop simulation under the same protocol. Since DriveWAM was not originally evaluated on this closed-loop benchmark, we regard this as an exploratory comparison between matched adaptations, rather than a comparison with closed-loop driving systems designed and trained for this benchmark.

MetricDriveWAMDriveWAM + PSF
Close Encounter, all ↓40.0%36.1%
Close Encounter, at-fault ↓16.0%14.0%
Off-road Rate ↓22.0%19.0%
Progress ↑40.9%41.2%
AlpaSim Score, all ↑0.06000.0970
AlpaSim Score, at-fault ↑0.15400.1958

PSF reduces close encounters and off-road rate while improving progress and both AlpaSim scores. These results provide an initial closed-loop assessment of how PSF affects driving behavior, complementing the offline ADE/FDE evaluation.

We have added the closed-loop evaluation protocol and full results to the appendix, together with driving videos in the supplementary material.

Changes in the revised PDF.
In Section 4 and Tables 3 and 6, we clarify the denoising-step configurations and distinguish reduced-step speedup from fixed-forward efficiency. In Section 5, “Related Work and Discussion,” and Appendix B, we clarify PSF’s relationship to progressive sparsification and action-aware pruning, including DynamicViT and SpecPrune-VLA. Changes addressing the shared official comments are marked in red; changes addressing your individual suggestions are marked in green.

Reviewer_zNHFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces Progressive Sparse Forcing (PSF), which reduces action-to-visual attention through depth in world-action diffusion transformers. A learned router selects visual tokens for each action token, and its selected scores enter the attention logits so that the original denoising loss trains the router. The method is tested across first-frame, inverse-dynamics, and joint-denoising layouts, with manipulation, spatial-robustness, and open-loop driving evaluations.

Critical strengths

1、PSF is a concrete modification to a shared conditioning pathway and does not require object masks or auxiliary grounding supervision. The distinction between discrete Top-K indices and gradients through selected score values is clearly explained (Section 3 and Appendix A.3). 2、The strongest evidence comes from OOD manipulation. FastWAM improves from 57.3% to 71.1% on the LIBERO-Plus spatial suite and from 53.2% to 62.4% on the full suite. IDM-WAM also improves on the full suite (Tables 4 and 9). These results are more informative than the small gains on saturated in-distribution tasks. 3、The driving study includes matched denoising-budget comparisons across three data scales and an extensive step sweep. The efficiency analysis also reports the small-context slowdown rather than presenting sparsity as a universal latency improvement (Tables 2, 5, 7 and 8).

Critical weaknesses
  1. The controls do not isolate progressive sparsity from the router's additional attention bias. Equation 5 changes both which tokens are available and the logits assigned to retained tokens. The stage-count ablation does not distinguish these effects or control average retained-token budget. A dense model with the same learned score bias, together with a constant-sparsity model at matched budget, would directly test whether progressive restriction is responsible for the gains. The conditional depth bound motivates the design but does not establish this empirical causal claim.
  2. Uncertainty is important for several headline improvements. The overall LIBERO gains are 0.5-1.0 percentage points, and Table 6 specifies 50 trials per task without reporting training-seed variability or confidence intervals. The full LIBERO-Plus result for JD-WAM is essentially unchanged, 65.7% versus 65.6%, despite its spatial-suite improvement. These results support layout-dependent benefits rather than a reliable improvement in every setting. Paired outcomes and independent-seed information are needed to interpret the smaller differences.
  3. The interpretation as improved physical grounding remains partly inferential. Attention maps and successful rollout visualizations show where the model attends, but do not establish which information causes better actions. The paper itself notes that apparent attention sinks can contain globally aggregated information. Fewer denoising steps are also compatible with improved optimization or regularization. Neither Proposition 1 nor Proposition 2 establishes semantic grounding; the latter is local and compares against a score-biased full-context reference, not the original dense baseline. A limited intervention on selected versus unselected visual information would make the mechanism claim more convincing.
Questions for authors
  1. Can you compare progressive PSF with dense score-biased attention and constant sparsity at a matched retained-token budget, using one existing manipulation configuration?
  2. How many independently trained checkpoints underlie Tables 1 and 4? Can you provide paired uncertainty estimates for the manipulation gains and explain the near-parity JD-WAM result on the full suite?
  3. On an existing evaluation subset, does perturbing the selected task-relevant information impair actions more than a size-matched perturbation of unselected information? Alternatively, can you provide another controlled test of the proposed grounding mechanism?
2 comments

We sincerely appreciate Reviewer zNHF’s encouraging recognition of our routing mechanism and OOD results. Your careful, constructive feedback has helped us strengthen the evidence and improve the clarity of our manuscript.

As these concerns are shared across the reviews, we consolidate the additional experiments in three shared official comments:

Weakness 1 / Question 1 → G1: Controlled Ablations and Sources of Improvement.
We add the requested dense score-biased attention and approximately budget-matched constant-sparsity controls, alongside random and backbone-attention Top-K selection. Dense router bias alone does not recover PSF’s OOD gains, while progressive PSF outperforms constant sparsity with the learned router and score bias unchanged. G1 provides the full results and experimental controls.

Weakness 2 / Question 2 → G2: Training-Run Variability and Layout Dependence.
We extend the original single-seed experiments to three independently trained runs for both the baselines and their PSF variants, reporting mean ± SD on LIBERO, LIBERO-Plus, and offline driving. Overall mean gains remain positive across manipulation layouts. For JD-WAM, full LIBERO-Plus success changes from near parity in the original run to a positive three-run mean difference of 0.5 percentage points. G2 also reports lower overall SDs for JD-WAM on both LIBERO (0.2 → 0.1) and LIBERO-Plus (1.4 → 1.1), alongside its positive mean gains.

Weakness 3 / Question 3 → G3: Design Motivation and Evidence for Learned Visual Conditioning.
Inspired by your suggestion, we conduct a matched selected-versus-unselected intervention in G3. Across two token counts and three Gaussian noise levels, perturbing selected information causes larger increases in action error than perturbing unselected information. G3 details the protocol and results, and distinguishes this behavioral evidence from semantic-grounding guarantees. We also clarify that Proposition 2 concerns a local comparison with score-biased full-context attention, not a proof of superiority over the original dense baseline.

Your three main concerns were also highlighted by the other reviewers, so we address them together in our shared official comments G1–G3. The corresponding revisions are marked in red in the revised manuscript, with their locations listed in the shared comments.

AuthorsOpenReview ↗

Follow-up on Our Response and Revised Manuscript

Thank you again for your time, thoughtful review, and constructive suggestions. We wanted to follow up on our response and briefly highlight the additional evidence we provided to address your main concerns.

  • What drives the gains (G1): We added dense router-bias, random and attention-based Top-K, and budget-matched constant-sparsity controls. Among these controls, only PSF achieves a substantial LIBERO-Plus improvement, supporting the combined role of learned routing and progressive sparsity. These results appear in revised Table 5 and Appendix D.1.

  • Training Variability and layout dependence (G2): We added two independent training runs for each baseline and PSF variant and now report mean ± SD across three runs. PSF shows positive overall mean gains for all three manipulation layouts on both LIBERO and LIBERO-Plus. The results appear in revised Tables 1 and 4.

  • Visual-conditioning evidence (G3): Following your suggestion, we conducted a matched intervention comparing perturbations to router-selected and unselected visual features. Perturbing selected features caused larger increases in action error across the tested settings, providing behavioral evidence beyond attention visualizations. The protocol and results appear in revised Appendix D.4.

We hope these responses help address your concerns and would greatly welcome any further questions or discussion. Thank you again for your time and careful consideration.

Comments on the paper

1 comment

Dear Reviewers,

Please kindly review the authors' rebuttal and engage in discussions if you have further questions.

Thanks, AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.