- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies robotic manipulation under smoke and other airborne scattering media, where RGB observations are degraded and active depth sensing becomes unreliable. The authors introduce SmokeBot, consisting of two components: SmokeManip, a perception-to-manipulation system, and SmokeLab, a simulation and benchmark platform for manipulation under smoke. The motivation is practically meaningful: existing manipulation pipelines generally assume clean RGB/depth observations, while smoke can degrade both modalities substantially.
SmokeManip decomposes perception into SmokeScene for target localization/segmentation and SmokeAfford for action-conditioned affordance prediction. The predicted affordances are combined with a smoke-adapted monocular depth estimator to construct 6-DoF interaction candidates, which are executed using a ProMP-based primitive library and hierarchical retry/re-observation mechanism.
Critical strengths
The paper identifies a practically relevant failure mode that is largely neglected by mainstream robot-learning benchmarks. Smoke is particularly interesting because it is spatially non-uniform and dynamic and can simultaneously corrupt visual appearance and active depth sensing. The motivation is convincing and substantially different from standard photometric corruption benchmarks.
2. Strong perception-to-execution system design.
Rather than treating smoke robustness purely as image restoration, the authors propagate the problem all the way from segmentation and affordance prediction to pose generation and manipulation recovery. The global-to-local re-observation strategy and hierarchical retry mechanism are sensible engineering choices for manipulation under unreliable perception.
The execution ablation is particularly informative. At medium smoke density, full SmokeManip achieves 60.0% primitive and 46.7% compositional success, while removing the smoke-adapted depth estimator, pose generation, or closed-loop recovery reduces performance substantially.
3. SmokeLab could be a useful benchmark contribution.
The benchmark contains configurable volumetric smoke rather than simply applying an image-space haze operator. Smoke varies spatially, temporally, and according to viewpoint, which makes it substantially more realistic for embodied interaction.
The inclusion of multiple abstraction levels—scene understanding, affordance prediction, primitive execution, and compositional manipulation—is also a positive design choice.
4. Real-robot experiments are included.
The work does not remain purely in simulation. The authors deploy the complete pipeline on a RealMan robot using only RGB observations at deployment and evaluate five primitives together with multi-stage tasks.
Critical weaknesses
- The manipulation comparison is not fully apples-to-apples This is my largest concern. SmokeManip is not simply a perception model or robot policy. It receives substantial task structure from:
- an object localization module,
- an explicit affordance predictor,
- a fine-tuned monocular depth model,
- explicit 6-DoF geometric pose generation,
- a handcrafted/learned primitive library,
- feasibility constraints,
- object-state success predicates,
- retries,
- local re-observation,
- global re-observation.
In contrast, the VLA baselines are trained as action policies and explicitly do not receive SmokeScene masks, affordances, smoke-adapted depth, or retry logic.
Therefore, Table 2 does not cleanly demonstrate: SmokeManip learns better manipulation under smoke than VLA methods.
It demonstrates something closer to: A highly structured perception/planning/recovery pipeline is more reliable than end-to-end policies in this relatively low-data smoke benchmark.
That result is useful, but it is substantially different.
For example, the ablation shows that simply removing pose generation drops compositional SR from 46.7% to 13.3%, while removing closed-loop recovery drops it to 31.1%.
Thus, a large portion of the advantage over the VLA baselines may arise from classical geometric planning and retry rather than the proposed smoke-robust learning mechanism.
I would strongly encourage an additional comparison where baselines receive comparable closed-loop execution infrastructure, e.g.:
VLA perception/action proposal + the same success predicates/re-observation policy,
or alternatively: SmokeScene/SmokeAfford + a standard policy without handcrafted primitives.
This would isolate where the improvement actually originates. 2. There is a striking inconsistency between the predicted-mask and GT-mask affordance results Table 1 evaluates affordance prediction with SmokeScene-predicted masks. For GLOVER++, the reported results are:
- Low: KLD 1.243 / SIM 0.523 / NSS 12.449
- Medium: 1.397 / 0.468 / 11.532
- High: 1.572 / 0.409 / 10.617.
However, Table 5 gives GLOVER++ ground-truth masks, and performance becomes dramatically worse: - Low: KLD 3.710 / SIM 0.216 / NSS 5.809
- Medium: 3.181 / 0.240 / 6.166
- High: 3.173 / 0.191 / 4.658.
Providing a ground-truth object mask should normally improve performance, or at least not cause this enormous degradation. This discrepancy needs a clear explanation. As currently written, it raises concerns about either baseline implementation, mask preprocessing, evaluation configuration, or table correctness. This is sufficiently large that I would want it resolved before trusting the affordance comparison.
- The perception ablation protocol is ambiguous Table 6 reports the full SmokeAfford model as: KLD = 0.9300, SIM = 0.5420, NSS = 16.0800.
Interestingly, these values are exactly the averages of the ground-truth-mask results in Table 5, rather than the predicted-mask results from Table 1. For example,
This strongly suggests that the SmokeAfford ablations are performed using GT masks. That is reasonable for isolating the affordance network, but it should be stated explicitly.
As currently presented, the paper says that the ablation entries and metrics “match those used in the main experimental analysis,” which makes the protocol unnecessarily ambiguous.
4. The claim of “depth-free deployment” is somewhat misleading
The contribution section describes the system as supporting “depth-free deployment.”
However, the execution pipeline explicitly uses a fine-tuned DepthAnythingV2 model to predict monocular depth and lift 2D affordance locations into 3D.
The appendix further states:
“The smoke-adapted depth estimator lifts ranked 2D affordance candidates into 3D grasp or contact hypotheses.”
Thus, the method is not depth-free in an algorithmic sense; it is active-depth-sensor-free. I recommend changing this terminology throughout the manuscript because the present wording may lead readers to conclude that no depth inference is required during deployment. 5. Much of the learning methodology is composed of familiar components From an ICLR perspective, I am less convinced by the algorithmic novelty. SmokeScene essentially combines a segmentation backbone with geometry-teacher supervision and spatial reweighting. SmokeAfford combines:
- DINO features,
- DepthAnything features,
- mask encoding/refinement,
- cross-attention,
- object tokens,
- gated residual modulation,
- FiLM conditioning,
- standard BCE/KLD/edge/consistency losses.
The main formulations for depth-guided modulation and object-conditioned decoding are variants of fairly standard feature fusion and conditional modulation mechanisms.
Likewise, execution relies on ProMP primitives, explicit geometry, and retry logic. The combination is well engineered, but it is unclear what broad learning principle emerges from the method beyond the specific smoke setting. For ICLR, I would like a clearer answer to: What algorithmic insight from SmokeBot is expected to generalize to other partially observable/degraded perception settings?
At present, I view the benchmark/platform contribution as stronger than the methodological one.
6. Simulation-specific privileged supervision may limit the claimed generalization
SmokeScene training uses privileged quantities including clean depth and smoke-density-derived severity weights.
The severity term explicitly depends on clean depth and the smoke coefficient .
This is fine for simulation-trained policies, but it tightly couples the method to a simulator capable of producing:
- clean/smoke image pairs,
- accurate depth,
- known smoke parameters. I would like to see whether SmokeScene trained on SmokeLab transfers to smoke distributions whose scattering statistics are substantially different from the renderer. The real-world result partially addresses this, but the real-world experiment appears to involve one smoke generator and a single tabletop setup.
- Real-world evaluation is too narrow to establish robust smoke generalization The real evaluation is useful, but limited. It uses:
- one RealMan robot,
- one tabletop environment,
- one RGB camera,
- one smoke generator,
- medium smoke density.
This leaves several unanswered generalization questions: - unseen environments,
- unseen object classes,
- multiple smoke generators,
- steam versus dry smoke versus mist,
- different particle sizes/colors,
- low/high real smoke density,
- different cameras,
- outdoor or emergency-response conditions.
The paper explicitly groups smoke, fog, fume, steam, mist, and dust under one term, despite their different physical properties.
Consequently, demonstrating robustness to one artificial-smoke setup is insufficient to justify robustness to the broader family of scattering media suggested by the introduction.
- Evaluation episode counts are relatively small and uncertainty is not reported
The appendix states that each primitive is evaluated on only 10 simulation episodes per method, while each compositional task uses 20 episodes.
The real-world primitive evaluation uses three rounds of five episodes.
For success-rate comparisons, particularly where numbers such as 44.5%, 36.6%, etc. are reported, I would expect:
- number of total trials,
- standard error or confidence intervals,
- seeds,
- statistical significance. Without these, it is difficult to assess whether some differences are robust.
- Reproducibility is promised rather than fully provided in the paper
The authors state that source code, SmokeLab, and weights will be released.
However, several important hyperparameters—query count, feature resolution, Gaussian radius, loss weights, curriculum schedule, optimizer settings—are deferred to future released configuration files rather than specified in the manuscript.
For an anonymous submission, the paper should ideally contain enough information to reproduce the central results independently of a future release.
Questions for authors
Questions for the Authors
- Why does GLOVER++ perform dramatically worse with ground-truth masks than with SmokeScene-predicted masks? Is Table 1 or Table 5 using a different preprocessing/training protocol?
- Are the SmokeAfford ablations in Table 6 performed with ground-truth masks? The reported full-model averages appear to exactly match Table 5.
- How much of Table 2's improvement remains if all methods receive the same retry/re-observation mechanism and object-state success detector?
- Can the authors report a baseline of: SmokeScene + SmokeAfford + pose generation but no primitive/recovery system? This would help isolate the perception contribution.
- Similarly, could an existing perception method be combined with the exact same primitive/recovery system?
- Are the VLA models trained with comparable numbers of trajectories to the demonstrations used in the SmokeManip primitive system? More details about total training data per baseline would help assess fairness.
- What fraction of failures originates from:
- scene localization,
- affordance prediction,
- monocular-depth error,
- pose generation,
- motion planning,
- contact execution?
- How well does SmokeManip generalize to unseen smoke statistics, e.g. real steam or mist rather than smoke-generator output?
- Why is the real-world evaluation restricted to medium smoke? Results across controlled low/medium/high physical smoke levels would substantially strengthen the paper.
- Could the paper report confidence intervals over the success rates?
7 comments
To W3 and Q2:
The SmokeAfford ablations in Table 6 are conducted using ground-truth masks. To complete the ablation study, we supplement additional ablation results using SmokeScene-predicted masks across Low, Medium, and High smoke densities.
Table Ablation study of SmokeAfford with SmokeScene-predicted masks.
| Method | KLD | SIM | NSS |
|---|---|---|---|
| w/o object mask refinement and encoding | 2.1538 | 0.2887 | 10.4389 |
| w/o depth-guided feature modulation | 0.8866 | 0.5444 | 13.3394 |
| w/o object-centric representation modeling | 2.0199 | 0.2994 | 11.0979 |
| w/o object-conditioned affordance decoding | 0.9233 | 0.5149 | 14.0105 |
| SmokeAfford(Full) | 0.8733 | 0.5780 | 14.3087 |
To W4:
We have updated the corresponding statements in the revised PDF, with modified passages highlighted in red for easy identification.
To W5:
SmokeManip's core insight is to turn robust perception in smoke scenes into robust manipulation in smoke scenes, without relying on active depth sensing. The algorithmic insight of SmokeScene and SmokeAfford is to leverage geometric cues that retain valid spatial-structural information under visual degradation.
Given the scarcity of open-source datasets in this specialized domain, we compare our approach with GraspALL[1] and conduct evaluation on its proposed MIGG dataset, which targets low-light garment segmentation and grasping. To ensure fair comparison, we report results with our own backbone and results using the identical backbone from GraspALL.
We also compare against OS-AGDO[2] on its proposed AGDDO15 dataset, an affordance prediction dataset, with both methods sharing the same visual backbone.
The experimental results demonstrate that the algorithmic insight from SmokeBot holds the potential to generalize to other partially observable or degraded perception settings.
Table Comparison with GraspALL on the MIGG dataset
| Model | 90-120 | 60-90 | 30-60 | 0-30 |
|---|---|---|---|---|
| GraspALL | 84.8% | 84.3% | 83.4% | 82.8% |
| SmokeScene(backbone used in GraspALL) | 85.1% | 84.1% | 83.6% | 83.1% |
| SmokeScene(backbone used in SmokeScene) | 88.0% | 86.0% | 84.3% | 83.4% |
Table Comparison with OS-AGDO on the AGDDO15 dataset
| Model | KLD | SIM | NSS |
|---|---|---|---|
| OS-AGDO[2] (one-shot) | 0.856 | 0.549 | 2.236 |
| SmokeAfford(one-shot) | 0.804 | 0.569 | 2.535 |
| OS-AGDO[2] (fully-supervised) | 0.629 | 0.610 | 2.512 |
| SmokeAfford(fully-supervised) | 0.465 | 0.696 | 2.942 |
[1] Zhong et al. GraspALL: Adaptive Structural Compensation from Illumination Variation for Robotic Garment Grasping in Any Low-Light Conditions. CVPR 2026.
[2] Jia et al. One-Shot Affordance Grounding of Deformable Objects in Egocentric Organizing Scenes. IROS 2025.
To W6, W8, Q9 and Q10:
(1) To W6, Q9:
We provide additional evaluation in both simulation and the real world, covering varying smoke densities and different tabletop setups. In the real world, each task is evaluated across the three distinct tabletops illustrated in Fig. 7.
For every action primitive and compositional task, we conduct 3 rounds with 15 trials per round in the real-world setting, and 3 rounds with 45 trials per round in simulation. The random seeds for the three rounds are 42, 3407, and 106202, respectively. The results are shown in the table below.
(2) To W8, Q10:
To quantify finite-trial uncertainty, we report Wilson score 95% confidence intervals (z = 1.96). Each entry is formatted as k/n (SR %) [lower, upper], where k and n denote successful and total trials, respectively, and the interval bounds are expressed as percentages.
Table Manipulation success rate and Wilson 95% confidence intervals in simulation
| Method | Low Pri. | Low Com. | Medium Pri. | Medium Com. | High Pri. | High Com. |
|---|---|---|---|---|---|---|
| 63/135 (46.7) [38.5, 55.1] | 33/135 (24.4) [18.0, 32.3] | 36/135 (26.7) [19.9, 34.7] | 16/135 (11.9) [7.4, 18.4] | 13/135 (9.6) [5.7, 15.8] | 6/135 (4.4) [2.1, 9.4] | |
| OpenVLA-OFT | 56/135 (41.5) [33.5, 49.9] | 28/135 (20.7) [14.8, 28.3] | 30/135 (22.2) [16.0, 29.9] | 13/135 (9.6) [5.7, 15.8] | 10/135 (7.4) [4.1, 13.1] | 4/135 (3.0) [1.2, 7.4] |
| CRT-VLA | 83/135 (61.5) [53.1, 69.3] | 46/135 (34.1) [26.6, 42.4] | 49/135 (36.3) [28.7, 44.7] | 29/135 (21.5) [15.4, 29.1] | 27/135 (20.0) [14.1, 27.5] | 14/135 (10.4) [6.3, 16.7] |
| StableVLA | 72/135 (53.3) [44.9, 61.5] | 40/135 (29.6) [22.6, 37.8] | 43/135 (31.9) [24.6, 40.1] | 23/135 (17.0) [11.6, 24.3] | 20/135 (14.8) [9.8, 21.8] | 11/135 (8.1) [4.6, 14.0] |
| DehazeFormer + DP | 47/135 (34.8) [27.3, 43.2] | 23/135 (17.0) [11.6, 24.3] | 24/135 (17.8) [12.2, 25.1] | 10/135 (7.4) [4.1, 13.1] | 8/135 (5.9) [3.0, 11.3] | 3/135 (2.2) [0.8, 6.3] |
| DehazeNet + DP | 44/135 (32.6) [25.3, 40.9] | 20/135 (14.8) [9.8, 21.8] | 22/135 (16.3) [11.0, 23.4] | 9/135 (6.7) [3.5, 12.2] | 7/135 (5.2) [2.5, 10.3] | 3/135 (2.2) [0.8, 6.3] |
| SmokeManip | 96/135 (71.1) [63.0, 78.1] | 72/135 (53.3) [44.9, 61.5] | 81/135 (60.0) [51.6, 67.9] | 63/135 (46.7) [38.5, 55.1] | 60/135 (44.4) [36.3, 52.9] | 45/135 (33.3) [25.9, 41.6] |
Table Manipulation success rate and Wilson 95% confidence intervals in real-world
| Method | Low Pri. | Low Com. | Medium Pri. | Medium Com. | High Pri. | High Com. |
|---|---|---|---|---|---|---|
| 26/45 (57.8) [43.3, 71.0] | 9/45 (20.0) [10.9, 33.8] | 7/45 (15.6) [7.7, 28.8] | 2/45 (4.4) [1.2, 14.8] | 5/45 (11.1) [4.8, 23.5] | 1/45 (2.2) [0.4, 11.6] | |
| OpenVLA-OFT | 21/45 (46.7) [32.9, 60.9] | 6/45 (13.3) [6.3, 26.2] | 5/45 (11.1) [4.8, 23.5] | 1/45 (2.2) [0.4, 11.6] | 4/45 (8.9) [3.5, 20.7] | 1/45 (2.2) [0.4, 11.6] |
| CRT-VLA | 41/45 (91.1) [79.3, 96.5] | 15/45 (33.3) [21.4, 47.9] | 11/45 (24.4) [14.2, 38.7] | 4/45 (8.9) [3.5, 20.7] | 14/45 (31.1) [19.5, 45.7] | 5/45 (11.1) [4.8, 23.5] |
| StableVLA | 34/45 (75.6) [61.3, 85.8] | 12/45 (26.7) [16.0, 41.0] | 9/45 (20.0) [10.9, 33.8] | 3/45 (6.7) [2.3, 17.9] | 6/45 (13.3) [6.3, 26.2] | 2/45 (4.4) [1.2, 14.8] |
| DehazeFormer + DP | 18/45 (40.0) [27.0, 54.5] | 5/45 (11.1) [4.8, 23.5] | 4/45 (8.9) [3.5, 20.7] | 1/45 (2.2) [0.4, 11.6] | 3/45 (6.7) [2.3, 17.9] | 1/45 (2.2) [0.4, 11.6] |
| DehazeNet + DP | 17/45 (37.8) [25.1, 52.4] | 4/45 (8.9) [3.5, 20.7] | 4/45 (8.9) [3.5, 20.7] | 1/45 (2.2) [0.4, 11.6] | 3/45 (6.7) [2.3, 17.9] | 1/45 (2.2) [0.4, 11.6] |
| SmokeManip | 31/45 (68.9) [54.3, 80.5] | 22/45 (48.9) [35.0, 63.0] | 20/45 (44.4) [30.9, 58.8] | 13/45 (28.9) [17.7, 43.4] | 17/45 (37.8) [25.1, 52.4] | 12/45 (26.7) [16.0, 41.0] |
Thanks for your helpful and meaningful reviews!
To Weakness1(W1) , Question(Q3) , Q4 and Q5:
(1) To W1 and Q3:
We provide supplementary results where baselines receive the same retry/re-observation closed-loop recovery (CR). These results show closed-loop recovery yields limited gains for baselines. In SmokeManip, pose generation and closed-loop recovery convert robust smoke perception into robust manipulation under smoke. The closed-loop pipeline works for SmokeManip because retries and re-observation integrate naturally with its interpretable modular design. In contrast, end-to-end action proposals from baseline policies benefit little from these components.
(2) To Q4:
The variant "SmokeScene + SmokeAfford + pose generation without primitive/recovery system" cannot produce trajectories after grasping and thus cannot perform manipulation following a grasp. To isolate the perception contribution, we adopt Mask DINO and GLOVER++ (the strongest alternatives to SmokeScene and SmokeAfford) paired with the identical pose generation/primitive/recovery pipeline, alongside "SmokeScene + SmokeAfford + DP". The results demonstrate the effectiveness of our framework design methodology, as well as the necessity of the smoke-robust design for SmokeScene/SmokeAfford.
(3) To Q5:
The results of "MaskDINO + GLOVER++ + PG+Pri. + CR" further demonstrate that existing perception models can plug into this exact same primitive/recovery system.
Table Baselines with closed-loop execution infrastructure and new baselines in simulation
To W2 and Q1:
We greatly appreciate this comment! There are indeed some unfair implementation choices in Table 5. We faithfully reproduce GLOVER++ following its official codebase and conduct multi-seed experiments.
Table Manipulation affordance prediction under smoke using ground-truth target-object masks