acceptodds
Under review as a conference paper at ICLR 2027

SmokeBot: A Perception-to-Manipulation Framework and Simulation Platform for Robotic Manipulation in Smoke Scenes

Abstract

Robotic manipulation in smoke scenes is common yet highly challenging, as airborne scattering media degrade RGB observations and make active depth sensing unreliable. Existing robotic manipulation systems are primarily optimized for unobscured visual conditions, yet they overlook the complex multi-modal degradation induced by dynamic and spatially non-uniform smoke, which can severely impair perceptual reliability and thereby lead to a significant drop in manipulation performance. To address this challenge, we propose SmokeManip, a perception-to-manipulation framework for smoke-robust robotic manipulation without relying on active depth sensors. It decouples robotic perception into a scene understanding model and an affordance prediction model, both trained with geometric supervision to enable adaptive geometric and visual feature modeling under smoke. The former parses the task scene and localizes target objects, conditioning the latter to predict object-centric affordances that generate task-aware and physically feasible end-effector poses. For manipulation, SmokeManip further leverages a reusable action primitive library and hierarchical closed-loop recovery mechanism to compensate for perception inaccuracies and maintain stable manipulation. Beyond the algorithmic challenges, smoke-scene data collection and robotic manipulation evaluation lack a suitable platform. Therefore, we further construct SmokeLab, a dedicated simulation platform for robotic manipulation in smoke scenes. SmokeLab provides high-fidelity dynamic smoke simulation, automated data collection, and unified evaluation protocols for robotic perception and manipulation. Extensive experiments in SmokeLab and real-world demonstrate that our approach outperforms existing baselines across varying smoke densities. The project page is available at [https://smokebot-web.github.io/](https://smokebot-web.github.io/).

Primary area:
applications to robotics, autonomy, planning
Keywords:
applications to robotics, robotic manipulation framework, simulation platform, robust perception in smoke scenes

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_1BeZOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3(+1)
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies robotic manipulation under smoke and other airborne scattering media, where RGB observations are degraded and active depth sensing becomes unreliable. The authors introduce SmokeBot, consisting of two components: SmokeManip, a perception-to-manipulation system, and SmokeLab, a simulation and benchmark platform for manipulation under smoke. The motivation is practically meaningful: existing manipulation pipelines generally assume clean RGB/depth observations, while smoke can degrade both modalities substantially.
SmokeManip decomposes perception into SmokeScene for target localization/segmentation and SmokeAfford for action-conditioned affordance prediction. The predicted affordances are combined with a smoke-adapted monocular depth estimator to construct 6-DoF interaction candidates, which are executed using a ProMP-based primitive library and hierarchical retry/re-observation mechanism.

Critical strengths

The paper identifies a practically relevant failure mode that is largely neglected by mainstream robot-learning benchmarks. Smoke is particularly interesting because it is spatially non-uniform and dynamic and can simultaneously corrupt visual appearance and active depth sensing. The motivation is convincing and substantially different from standard photometric corruption benchmarks.
2. Strong perception-to-execution system design. Rather than treating smoke robustness purely as image restoration, the authors propagate the problem all the way from segmentation and affordance prediction to pose generation and manipulation recovery. The global-to-local re-observation strategy and hierarchical retry mechanism are sensible engineering choices for manipulation under unreliable perception. The execution ablation is particularly informative. At medium smoke density, full SmokeManip achieves 60.0% primitive and 46.7% compositional success, while removing the smoke-adapted depth estimator, pose generation, or closed-loop recovery reduces performance substantially.
3. SmokeLab could be a useful benchmark contribution. The benchmark contains configurable volumetric smoke rather than simply applying an image-space haze operator. Smoke varies spatially, temporally, and according to viewpoint, which makes it substantially more realistic for embodied interaction.
The inclusion of multiple abstraction levels—scene understanding, affordance prediction, primitive execution, and compositional manipulation—is also a positive design choice. 4. Real-robot experiments are included. The work does not remain purely in simulation. The authors deploy the complete pipeline on a RealMan robot using only RGB observations at deployment and evaluate five primitives together with multi-stage tasks.

Critical weaknesses
  1. The manipulation comparison is not fully apples-to-apples This is my largest concern. SmokeManip is not simply a perception model or robot policy. It receives substantial task structure from:
  • an object localization module,
  • an explicit affordance predictor,
  • a fine-tuned monocular depth model,
  • explicit 6-DoF geometric pose generation,
  • a handcrafted/learned primitive library,
  • feasibility constraints,
  • object-state success predicates,
  • retries,
  • local re-observation,
  • global re-observation. In contrast, the VLA baselines are trained as action policies and explicitly do not receive SmokeScene masks, affordances, smoke-adapted depth, or retry logic.
    Therefore, Table 2 does not cleanly demonstrate: SmokeManip learns better manipulation under smoke than VLA methods.

It demonstrates something closer to: A highly structured perception/planning/recovery pipeline is more reliable than end-to-end policies in this relatively low-data smoke benchmark.

That result is useful, but it is substantially different. For example, the ablation shows that simply removing pose generation drops compositional SR from 46.7% to 13.3%, while removing closed-loop recovery drops it to 31.1%.
Thus, a large portion of the advantage over the VLA baselines may arise from classical geometric planning and retry rather than the proposed smoke-robust learning mechanism. I would strongly encourage an additional comparison where baselines receive comparable closed-loop execution infrastructure, e.g.: VLA perception/action proposal + the same success predicates/re-observation policy,

or alternatively: SmokeScene/SmokeAfford + a standard policy without handcrafted primitives.

This would isolate where the improvement actually originates. 2. There is a striking inconsistency between the predicted-mask and GT-mask affordance results Table 1 evaluates affordance prediction with SmokeScene-predicted masks. For GLOVER++, the reported results are:

  • Low: KLD 1.243 / SIM 0.523 / NSS 12.449
  • Medium: 1.397 / 0.468 / 11.532
  • High: 1.572 / 0.409 / 10.617.
    However, Table 5 gives GLOVER++ ground-truth masks, and performance becomes dramatically worse:
  • Low: KLD 3.710 / SIM 0.216 / NSS 5.809
  • Medium: 3.181 / 0.240 / 6.166
  • High: 3.173 / 0.191 / 4.658.
    Providing a ground-truth object mask should normally improve performance, or at least not cause this enormous degradation. This discrepancy needs a clear explanation. As currently written, it raises concerns about either baseline implementation, mask preprocessing, evaluation configuration, or table correctness. This is sufficiently large that I would want it resolved before trusting the affordance comparison.
  1. The perception ablation protocol is ambiguous Table 6 reports the full SmokeAfford model as: KLD = 0.9300, SIM = 0.5420, NSS = 16.0800.

Interestingly, these values are exactly the averages of the ground-truth-mask results in Table 5, rather than the predicted-mask results from Table 1. For example,

(1.128+0.944+0.718)/3=0.930.(1.128 + 0.944 + 0.718)/3 = 0.930.

This strongly suggests that the SmokeAfford ablations are performed using GT masks. That is reasonable for isolating the affordance network, but it should be stated explicitly. As currently presented, the paper says that the ablation entries and metrics “match those used in the main experimental analysis,” which makes the protocol unnecessarily ambiguous.
4. The claim of “depth-free deployment” is somewhat misleading The contribution section describes the system as supporting “depth-free deployment.”
However, the execution pipeline explicitly uses a fine-tuned DepthAnythingV2 model to predict monocular depth and lift 2D affordance locations into 3D.
The appendix further states: “The smoke-adapted depth estimator lifts ranked 2D affordance candidates into 3D grasp or contact hypotheses.”

Thus, the method is not depth-free in an algorithmic sense; it is active-depth-sensor-free. I recommend changing this terminology throughout the manuscript because the present wording may lead readers to conclude that no depth inference is required during deployment. 5. Much of the learning methodology is composed of familiar components From an ICLR perspective, I am less convinced by the algorithmic novelty. SmokeScene essentially combines a segmentation backbone with geometry-teacher supervision and spatial reweighting. SmokeAfford combines:

  • DINO features,
  • DepthAnything features,
  • mask encoding/refinement,
  • cross-attention,
  • object tokens,
  • gated residual modulation,
  • FiLM conditioning,
  • standard BCE/KLD/edge/consistency losses. The main formulations for depth-guided modulation and object-conditioned decoding are variants of fairly standard feature fusion and conditional modulation mechanisms.
    Likewise, execution relies on ProMP primitives, explicit geometry, and retry logic. The combination is well engineered, but it is unclear what broad learning principle emerges from the method beyond the specific smoke setting. For ICLR, I would like a clearer answer to: What algorithmic insight from SmokeBot is expected to generalize to other partially observable/degraded perception settings?

At present, I view the benchmark/platform contribution as stronger than the methodological one. 6. Simulation-specific privileged supervision may limit the claimed generalization SmokeScene training uses privileged quantities including clean depth and smoke-density-derived severity weights.
The severity term explicitly depends on clean depth and the smoke coefficient β\beta.
This is fine for simulation-trained policies, but it tightly couples the method to a simulator capable of producing:

  • clean/smoke image pairs,
  • accurate depth,
  • known smoke parameters. I would like to see whether SmokeScene trained on SmokeLab transfers to smoke distributions whose scattering statistics are substantially different from the renderer. The real-world result partially addresses this, but the real-world experiment appears to involve one smoke generator and a single tabletop setup.
  1. Real-world evaluation is too narrow to establish robust smoke generalization The real evaluation is useful, but limited. It uses:
  • one RealMan robot,
  • one tabletop environment,
  • one RGB camera,
  • one smoke generator,
  • medium smoke density.
    This leaves several unanswered generalization questions:
  • unseen environments,
  • unseen object classes,
  • multiple smoke generators,
  • steam versus dry smoke versus mist,
  • different particle sizes/colors,
  • low/high real smoke density,
  • different cameras,
  • outdoor or emergency-response conditions. The paper explicitly groups smoke, fog, fume, steam, mist, and dust under one term, despite their different physical properties.
    Consequently, demonstrating robustness to one artificial-smoke setup is insufficient to justify robustness to the broader family of scattering media suggested by the introduction.
  1. Evaluation episode counts are relatively small and uncertainty is not reported The appendix states that each primitive is evaluated on only 10 simulation episodes per method, while each compositional task uses 20 episodes.
    The real-world primitive evaluation uses three rounds of five episodes.
    For success-rate comparisons, particularly where numbers such as 44.5%, 36.6%, etc. are reported, I would expect:
  • number of total trials,
  • standard error or confidence intervals,
  • seeds,
  • statistical significance. Without these, it is difficult to assess whether some differences are robust.
  1. Reproducibility is promised rather than fully provided in the paper The authors state that source code, SmokeLab, and weights will be released.
    However, several important hyperparameters—query count, feature resolution, Gaussian radius, loss weights, curriculum schedule, optimizer settings—are deferred to future released configuration files rather than specified in the manuscript.
    For an anonymous submission, the paper should ideally contain enough information to reproduce the central results independently of a future release.
Questions for authors

Questions for the Authors

  1. Why does GLOVER++ perform dramatically worse with ground-truth masks than with SmokeScene-predicted masks? Is Table 1 or Table 5 using a different preprocessing/training protocol?
  2. Are the SmokeAfford ablations in Table 6 performed with ground-truth masks? The reported full-model averages appear to exactly match Table 5.
  3. How much of Table 2's improvement remains if all methods receive the same retry/re-observation mechanism and object-state success detector?
  4. Can the authors report a baseline of: SmokeScene + SmokeAfford + pose generation but no primitive/recovery system? This would help isolate the perception contribution.
  5. Similarly, could an existing perception method be combined with the exact same primitive/recovery system?
  6. Are the VLA models trained with comparable numbers of trajectories to the demonstrations used in the SmokeManip primitive system? More details about total training data per baseline would help assess fairness.
  7. What fraction of failures originates from:
    • scene localization,
    • affordance prediction,
    • monocular-depth error,
    • pose generation,
    • motion planning,
    • contact execution?
  8. How well does SmokeManip generalize to unseen smoke statistics, e.g. real steam or mist rather than smoke-generator output?
  9. Why is the real-world evaluation restricted to medium smoke? Results across controlled low/medium/high physical smoke levels would substantially strengthen the paper.
  10. Could the paper report confidence intervals over the success rates?
7 comments

Thanks for your helpful and meaningful reviews!

To Weakness1(W1) , Question(Q3) , Q4 and Q5:

(1) To W1 and Q3:

We provide supplementary results where baselines receive the same retry/re-observation closed-loop recovery (CR). These results show closed-loop recovery yields limited gains for baselines. In SmokeManip, pose generation and closed-loop recovery convert robust smoke perception into robust manipulation under smoke. The closed-loop pipeline works for SmokeManip because retries and re-observation integrate naturally with its interpretable modular design. In contrast, end-to-end action proposals from baseline policies benefit little from these components.

(2) To Q4:

The variant "SmokeScene + SmokeAfford + pose generation without primitive/recovery system" cannot produce trajectories after grasping and thus cannot perform manipulation following a grasp. To isolate the perception contribution, we adopt Mask DINO and GLOVER++ (the strongest alternatives to SmokeScene and SmokeAfford) paired with the identical pose generation/primitive/recovery pipeline, alongside "SmokeScene + SmokeAfford + DP". The results demonstrate the effectiveness of our framework design methodology, as well as the necessity of the smoke-robust design for SmokeScene/SmokeAfford.

(3) To Q5:

The results of "MaskDINO + GLOVER++ + PG+Pri. + CR" further demonstrate that existing perception models can plug into this exact same primitive/recovery system.

Table Baselines with closed-loop execution infrastructure and new baselines in simulation

MethodLow Pri.Low Com.Medium Pri.Medium Com.High Pri.High Com.
π0.5\pi_{0.5} + CR65/135 (+2)34/135 (+1)42/135 (+6)23/135 (+7)21/135 (+8)11/135 (+5)
OpenVLA-OFT + CR58/135 (+2)29/135 (+1)37/135 (+7)21/135 (+8)17/135 (+7)12/135 (+8)
CRT-VLA + CR84/135 (+1)49/135 (+3)57/135 (+8)37/135 (+8)42/135 (+15)24/135 (+10)
StableVLA + CR74/135 (+2)42/135 (+2)52/135 (+9)31/135 (+8)31/135 (+11)18/135 (+7)
DehazeFormer + DP + CR49/135 (+2)26/135 (+3)32/135 (+8)18/135 (+8)19/135 (+11)13/135 (+10)
DehazeNet + DP + CR45/135 (+1)21/135 (+1)30/135 (+8)16/135 (+7)15/135 (+8)10/135 (+7)
SmokeScene + SmokeAfford + DP72/13560/13553/13539/13541/13529/135
MaskDINO + GLOVER++ + PG + Pri. + CR78/13559/13542/13522/13536/13518/135
SmokeManip96/13572/13581/13563/13560/13545/135

To W2 and Q1:

We greatly appreciate this comment! There are indeed some unfair implementation choices in Table 5. We faithfully reproduce GLOVER++ following its official codebase and conduct multi-seed experiments.

Table Manipulation affordance prediction under smoke using ground-truth target-object masks

MethodLow KLD ↓\downarrowLow SIM ↑\uparrowLow NSS ↑\uparrowMedium KLD ↓\downarrowMedium SIM ↑\uparrowMedium NSS ↑\uparrowHigh KLD ↓\downarrowHigh SIM ↑\uparrowHigh NSS ↑\uparrow
GLOVER++ (seed 42)1.3420.44415.4851.1010.50415.2160.7680.55615.380
GLOVER++ (seed 3407)1.3100.45415.8651.0980.50415.3790.7590.55115.048
GLOVER++ (seed 106202)1.4630.44415.3461.1870.49915.2550.8880.56015.394
GLOVER++ (average of three seeds)1.3720.44715.5651.1290.50215.2830.8050.55615.274
SmokeAfford1.1280.51616.6880.9440.54016.0910.7180.57015.460

To W3 and Q2:

The SmokeAfford ablations in Table 6 are conducted using ground-truth masks. To complete the ablation study, we supplement additional ablation results using SmokeScene-predicted masks across Low, Medium, and High smoke densities.

Table Ablation study of SmokeAfford with SmokeScene-predicted masks.

MethodKLD ↓\downarrowSIM ↑\uparrowNSS ↑\uparrow
w/o object mask refinement and encoding2.15380.288710.4389
w/o depth-guided feature modulation0.88660.544413.3394
w/o object-centric representation modeling2.01990.299411.0979
w/o object-conditioned affordance decoding0.92330.514914.0105
SmokeAfford(Full)0.87330.578014.3087

To W4:

We have updated the corresponding statements in the revised PDF, with modified passages highlighted in red for easy identification.

To W5:

SmokeManip's core insight is to turn robust perception in smoke scenes into robust manipulation in smoke scenes, without relying on active depth sensing. The algorithmic insight of SmokeScene and SmokeAfford is to leverage geometric cues that retain valid spatial-structural information under visual degradation.

Given the scarcity of open-source datasets in this specialized domain, we compare our approach with GraspALL[1] and conduct evaluation on its proposed MIGG dataset, which targets low-light garment segmentation and grasping. To ensure fair comparison, we report results with our own backbone and results using the identical backbone from GraspALL.

We also compare against OS-AGDO[2] on its proposed AGDDO15 dataset, an affordance prediction dataset, with both methods sharing the same visual backbone.

The experimental results demonstrate that the algorithmic insight from SmokeBot holds the potential to generalize to other partially observable or degraded perception settings.

Table Comparison with GraspALL on the MIGG dataset

Model90-12060-9030-600-30
GraspALL84.8%84.3%83.4%82.8%
SmokeScene(backbone used in GraspALL)85.1%84.1%83.6%83.1%
SmokeScene(backbone used in SmokeScene)88.0%86.0%84.3%83.4%

Table Comparison with OS-AGDO on the AGDDO15 dataset

ModelKLD ↓\downarrowSIM ↑\uparrowNSS ↑\uparrow
OS-AGDO[2] (one-shot)0.8560.5492.236
SmokeAfford(one-shot)0.8040.5692.535
OS-AGDO[2] (fully-supervised)0.6290.6102.512
SmokeAfford(fully-supervised)0.4650.6962.942

[1] Zhong et al. GraspALL: Adaptive Structural Compensation from Illumination Variation for Robotic Garment Grasping in Any Low-Light Conditions. CVPR 2026.

[2] Jia et al. One-Shot Affordance Grounding of Deformable Objects in Egocentric Organizing Scenes. IROS 2025.

To W6, W8, Q9 and Q10:

(1) To W6, Q9:

We provide additional evaluation in both simulation and the real world, covering varying smoke densities and different tabletop setups. In the real world, each task is evaluated across the three distinct tabletops illustrated in Fig. 7.

For every action primitive and compositional task, we conduct 3 rounds with 15 trials per round in the real-world setting, and 3 rounds with 45 trials per round in simulation. The random seeds for the three rounds are 42, 3407, and 106202, respectively. The results are shown in the table below.

(2) To W8, Q10:

To quantify finite-trial uncertainty, we report Wilson score 95% confidence intervals (z = 1.96). Each entry is formatted as k/n (SR %) [lower, upper], where k and n denote successful and total trials, respectively, and the interval bounds are expressed as percentages.

Table Manipulation success rate and Wilson 95% confidence intervals in simulation

MethodLow Pri.Low Com.Medium Pri.Medium Com.High Pri.High Com.
π0.5\pi_{0.5}63/135 (46.7) [38.5, 55.1]33/135 (24.4) [18.0, 32.3]36/135 (26.7) [19.9, 34.7]16/135 (11.9) [7.4, 18.4]13/135 (9.6) [5.7, 15.8]6/135 (4.4) [2.1, 9.4]
OpenVLA-OFT56/135 (41.5) [33.5, 49.9]28/135 (20.7) [14.8, 28.3]30/135 (22.2) [16.0, 29.9]13/135 (9.6) [5.7, 15.8]10/135 (7.4) [4.1, 13.1]4/135 (3.0) [1.2, 7.4]
CRT-VLA83/135 (61.5) [53.1, 69.3]46/135 (34.1) [26.6, 42.4]49/135 (36.3) [28.7, 44.7]29/135 (21.5) [15.4, 29.1]27/135 (20.0) [14.1, 27.5]14/135 (10.4) [6.3, 16.7]
StableVLA72/135 (53.3) [44.9, 61.5]40/135 (29.6) [22.6, 37.8]43/135 (31.9) [24.6, 40.1]23/135 (17.0) [11.6, 24.3]20/135 (14.8) [9.8, 21.8]11/135 (8.1) [4.6, 14.0]
DehazeFormer + DP47/135 (34.8) [27.3, 43.2]23/135 (17.0) [11.6, 24.3]24/135 (17.8) [12.2, 25.1]10/135 (7.4) [4.1, 13.1]8/135 (5.9) [3.0, 11.3]3/135 (2.2) [0.8, 6.3]
DehazeNet + DP44/135 (32.6) [25.3, 40.9]20/135 (14.8) [9.8, 21.8]22/135 (16.3) [11.0, 23.4]9/135 (6.7) [3.5, 12.2]7/135 (5.2) [2.5, 10.3]3/135 (2.2) [0.8, 6.3]
SmokeManip96/135 (71.1) [63.0, 78.1]72/135 (53.3) [44.9, 61.5]81/135 (60.0) [51.6, 67.9]63/135 (46.7) [38.5, 55.1]60/135 (44.4) [36.3, 52.9]45/135 (33.3) [25.9, 41.6]

Table Manipulation success rate and Wilson 95% confidence intervals in real-world

MethodLow Pri.Low Com.Medium Pri.Medium Com.High Pri.High Com.
π0.5\pi_{0.5}26/45 (57.8) [43.3, 71.0]9/45 (20.0) [10.9, 33.8]7/45 (15.6) [7.7, 28.8]2/45 (4.4) [1.2, 14.8]5/45 (11.1) [4.8, 23.5]1/45 (2.2) [0.4, 11.6]
OpenVLA-OFT21/45 (46.7) [32.9, 60.9]6/45 (13.3) [6.3, 26.2]5/45 (11.1) [4.8, 23.5]1/45 (2.2) [0.4, 11.6]4/45 (8.9) [3.5, 20.7]1/45 (2.2) [0.4, 11.6]
CRT-VLA41/45 (91.1) [79.3, 96.5]15/45 (33.3) [21.4, 47.9]11/45 (24.4) [14.2, 38.7]4/45 (8.9) [3.5, 20.7]14/45 (31.1) [19.5, 45.7]5/45 (11.1) [4.8, 23.5]
StableVLA34/45 (75.6) [61.3, 85.8]12/45 (26.7) [16.0, 41.0]9/45 (20.0) [10.9, 33.8]3/45 (6.7) [2.3, 17.9]6/45 (13.3) [6.3, 26.2]2/45 (4.4) [1.2, 14.8]
DehazeFormer + DP18/45 (40.0) [27.0, 54.5]5/45 (11.1) [4.8, 23.5]4/45 (8.9) [3.5, 20.7]1/45 (2.2) [0.4, 11.6]3/45 (6.7) [2.3, 17.9]1/45 (2.2) [0.4, 11.6]
DehazeNet + DP17/45 (37.8) [25.1, 52.4]4/45 (8.9) [3.5, 20.7]4/45 (8.9) [3.5, 20.7]1/45 (2.2) [0.4, 11.6]3/45 (6.7) [2.3, 17.9]1/45 (2.2) [0.4, 11.6]
SmokeManip31/45 (68.9) [54.3, 80.5]22/45 (48.9) [35.0, 63.0]20/45 (44.4) [30.9, 58.8]13/45 (28.9) [17.7, 43.4]17/45 (37.8) [25.1, 52.4]12/45 (26.7) [16.0, 41.0]
Reviewer_vT6COpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents SmokeBot, a robotic manipulation system designed for visually degraded smoke environments. It consists of SmokeManip, which combines smoke-aware scene understanding, object-conditioned affordance prediction, monocular depth-based pose generation, motion primitives, and closed-loop recovery, and SmokeLab, a simulation platform for data generation and evaluation under configurable volumetric smoke. The system is evaluated on perception, primitive actions, and compositional tasks in both simulation and real-world settings.

Critical strengths

The work provides a complete perception-to-execution pipeline together with a dedicated simulation platform, and evaluates both individual perception components and full manipulation performance in simulation and real-world experiments.

Critical weaknesses

The practical motivation is not sufficiently substantiated. Although SmokeLab contains several smoke-related environments, the paper does not clearly describe what manipulation tasks are performed in each environment or why these tasks specifically require operation under smoke. Without a clear mapping between scenes, tasks, and real-world use cases, it is difficult to assess whether SmokeLab captures meaningful smoke-specific manipulation challenges.

Questions for authors
  1. The technical novelty is limited. SmokeScene and SmokeAfford mainly build on existing components, including Depth Anything v2, DINOv2, target masks, AnyGrasp-style representations, and ProMP primitives. Although several smoke-specific mechanisms are introduced, the overall contribution is closer to domain adaptation and system integration than an independent methodological advance. SmokeManip appears to be an incremental work. Please provide a detailed analysis of its contributions and novelty beyond fine-tuning on a domain-specific dataset.
  2. The system-level comparison does not clearly isolate the source of the performance gains. End-to-end baselines such as pi0.5 are compared with SmokeManip, which additionally uses explicit masks, affordances, monocular depth, pose generation, motion primitives, and closed-loop retries. Table 3 partially addresses this issue by replacing SmokeScene and SmokeAfford within the same execution framework. However, it includes only one alternative for each module and evaluates them only at medium smoke density, so the gains in Table 2 cannot be attributed solely to smoke-robust perception.
  3. SmokeLab is insufficiently documented. Although Figures 4 and 5 show the supported environments and mention configurable smoke, camera settings, and random object placement, the paper does not specify the manipulation tasks and interactive objects in each scene. It also omits the sampling ranges, distributions, and train/test protocols for most randomized factors, making the benchmark’s coverage, reproducibility, and support for generalization difficult to assess. Please present SmokeLab in as much detail as possible to better demonstrate its value and role.
  4. The sim-to-real evidence is insufficient. The paper does not quantify the gap between simulated and real smoke or evaluate generalization to unseen scenes, unseen objects, or multiple real-world smoke densities. It also omits the amount and composition of the data used to fine-tune pi0.5, making its low real-world success rate difficult to interpret. Please provide a detailed analysis of the baseline training and carefully explain why pi0.5 performs poorly. In particular, the smoke concentration in the videos does not appear very high, and pi0.5 should be capable of performing these tasks under such conditions.
  5. The real-world evaluation uses only one robot, one tabletop workspace, and one medium-smoke condition. The relevance of generic tasks such as opening cabinets and drawers to smoke-specific applications is also insufficiently justified. These experiments demonstrate feasibility in a controlled setting but do not establish stable or general real-world performance. Please extend the evaluation to cover more scenarios. Otherwise, the motivation of the paper remains questionable, as robotic manipulation under smoke may be relevant only to a limited range of scenarios rather than being broadly applicable.
7 comments

To Question1(Q1):

(1) The novelty of SmokeBot. The core novelty of SmokeManip is a modular framework that translates robust perception into robust manipulation in smoke scenes. This design offers greater interpretability, and comparisons with VLA-based and DP-based methods support its soundness and effectiveness.

We also introduce a novel simulation platform for robotic perception and manipulation in smoke scenes. It addresses the lack of a unified platform combining high-fidelity dynamic smoke and smoke-specific manipulation tasks. The platform and SmokeManip, SmokeScene, and SmokeAfford complement each other. Beyond providing the data required by VLA models and imitation learning, the platform supports an alternative perspective: first establish robust perception under degraded visual conditions such as smoke, then translate that perception into effective manipulation.

As shown in the table below, combining SmokeScene and SmokeAfford with Diffusion Policy (DP) substantially improves performance over "DehazeNet/DehazeFormer + DP" . We also pair Mask DINO and GLOVER++, the strongest alternatives to SmokeScene and SmokeAfford, with the same pose generation, action primitive, and recovery pipeline used in SmokeManip. These results support the effectiveness of perception-to-manipulation frameworks and the importance of the smoke-robust designs of SmokeScene and SmokeAfford.

Table Supplementary baselines in simulation

MethodLow Pri.Low Com.Medium Pri.Medium Com.High Pri.High Com.
SmokeScene + SmokeAfford + DP72/13560/13553/13539/13541/13529/135
MaskDINO + GLOVER++ + PG+Pri. + CR78/13559/13542/13522/13536/13518/135
SmokeManip96/13572/13581/13563/13560/13545/135

(2) The potential to generalize to other settings. We compare our approach with GraspALL and conduct evaluation on its proposed MIGG dataset, which targets low-light garment segmentation and grasping. To ensure fair comparison, we report results with our own backbone and results using the identical backbone from GraspALL. We also compare against OS-AGDO [2] on its proposed AGDDO15 dataset, an affordance prediction dataset, with both methods sharing the same visual backbone. The experimental results demonstrate that the algorithmic insight from SmokeBot holds the potential to generalize to other settings.

Table Comparison with GraspALL on the MIGG dataset

Model90-12060-9030-600-30
GraspALL84.8%84.3%83.4%82.8%
SmokeScene(GraspALL Backbone)85.1%84.1%83.6%83.1%
SmokeScene(Swin-L)88.0%86.0%84.3%83.4%

Table Comparison with OS-AGDO on the AGDDO15 dataset

ModelKLD ↓\downarrowSIM ↑\uparrowNSS ↑\uparrow
OS-AGDO[2] (one-shot)0.8560.5492.236
SmokeAfford(one-shot)0.8040.5692.535
OS-AGDO[2] (fully-supervised)0.6290.6102.512
SmokeAfford(fully-supervised)0.4650.6962.942

[1] Zhong et al. GraspALL: Adaptive Structural Compensation from Illumination Variation for Robotic Garment Grasping in Any Low-Light Conditions. CVPR 2026.

[2] Jia et al. One-Shot Affordance Grounding of Deformable Objects in Egocentric Organizing Scenes. IROS 2025.

To Q2:

For a more comprehensive comparison, we provide supplementary results where baselines receive the same retry/re-observation closed-loop recovery (CR). These results show closed-loop recovery yields limited gains for baselines. In SmokeManip, pose generation and closed-loop recovery convert robust smoke perception into robust manipulation under smoke. The closed-loop pipeline works for SmokeManip because retries and re-observation integrate naturally with its interpretable modular design. In contrast, end-to-end action proposals from baseline policies benefit little from these components.

To clearly isolate the source of the performance gains, we adopt Mask DINO and GLOVER++ (the strongest alternatives to SmokeScene and SmokeAfford) paired with the identical pose generation/primitive/recovery pipeline, alongside "SmokeScene + SmokeAfford + DP". The results demonstrate that a modular perception-to-manipulation framework benefits more from adopting "explicit masks, affordances, monocular depth, pose generation, motion primitives, and closed-loop retries."

Table Baselines with closed-loop execution infrastructure and new baselines in simulation

MethodLow Pri.Low Com.Medium Pri.Medium Com.High Pri.High Com.
π0.5\pi_{0.5} + CR65/135 (+2)34/135 (+1)42/135 (+6)23/135 (+7)21/135 (+8)11/135 (+5)
OpenVLA-OFT + CR58/135 (+2)29/135 (+1)37/135 (+7)21/135 (+8)17/135 (+7)12/135 (+8)
CRT-VLA + CR84/135 (+1)49/135 (+3)57/135 (+8)37/135 (+8)42/135 (+15)24/135 (+10)
StableVLA + CR74/135 (+2)42/135 (+2)52/135 (+9)31/135 (+8)31/135 (+11)18/135 (+7)
DehazeFormer + DP + CR49/135 (+2)26/135 (+3)32/135 (+8)18/135 (+8)19/135 (+11)13/135 (+10)
DehazeNet + DP + CR45/135 (+1)21/135 (+1)30/135 (+8)16/135 (+7)15/135 (+8)10/135 (+7)
SmokeScene + SmokeAfford + DP72/13560/13553/13539/13541/13529/135
MaskDINO + GLOVER++ + PG + Pri. + CR78/13559/13542/13522/13536/13518/135
SmokeManip96/13572/13581/13563/13560/13545/135

To Q3:

(1) Details of scene, object category and task. The table below reports the scenes, object categories, and manipulation tasks covered in SmokeLab. Manipulation tasks span common ones like object grasping and cabinet door pushing/pulling, alongside smoke-scenario-specific tasks such as pressing the fire alarm, rotating the cooker knob, and pressing the range hood button.

Table mapping between scene, object category and task

SceneCategoryTask
Kitchenaxe, cooker, detergent, barrel detergent, dishwasher, bottled drinks, drink can, fire extinguisher, fire cabinet, fork, frying pan, hammer, knife, mug, pot, oven, screwdriver, shampoo, soap, spatula, spoon, squeeze bottle condiments, range hood , fridge, microwave, outlet, drawer, cabinet, shelfpress the range hood button, grasp the axe, pull/push the dishwasher door, rotate the cooker knob, pull/push the drawer, grasp the spoon
Workshopdesk,security inspection machine, cutting board, woodworking machinery, axe, turning lathe, trim router, small rack, medium rack, cabinet, wooden box, board stack, forklift, workbench, hammer, fire cabinet, fire extinguisher, toolbox, shelf, card box, palletgrasp the axe, pull the fire cabinet door, grasp the fire extinguisher
Bathroomshelf, shampoo, shower hand, mop, curtain, toilet trash can, mirror, wash basin, cabinet, brush, soapgrasp the shampoo, grasp the brush
Laboratoryshaker, friability tester, fire extinguisher, chair computer, chemicals, tube rack, localized titrator, centrifuge, gas evolution apparatus, oscillator, fume extractor arm, pure water machine, incubator, lab cabinet,poison cabinet, workbench, door, fire alarm, lab bench, dry bathgrasp the fume extractor arm, pull/push the fume extractor arm, grasp the fire extinguisher, pull/push the door, press the fire alarm, press the dry bath button
Storesqueeze bottle condiments, bottled drinks, barrel detergent, fork, frying pan, knife, mug, shampoo, soap, spatula, canned drinks, boxed snacks, bagged snacks, counter, cash register, scanner, steamer, shelf, refrigerator, freezer, self-checkout, oden cooker, display chillerpull/push the freezer door, grasp the bottled drinks

(2) The train/test protocols. We split the dataset at the scene level. SmokeLab provides 28 scenes spanning 5 categories (Kitchen x 20 + Bathroom x 2 + Laboratory x 2 + Store x 2 + Workshop x 2). For each category, half of the scenes are allocated to the training set and the remaining half to the test set. Scenes of the same category use the same set of object instances, while object positions are randomized.

For all scenes, the scene understanding dataset contains 18 randomly-sampled robot-centric viewpoints: 12 local viewpoints and 6 global viewpoints. The affordance prediction dataset reuses the same 12 randomly-sampled robot-centric local viewpoints from the scene-understanding dataset.

Reviewer_FSfJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SmokeBot includes the robot manipulation framework SmokeManip in smoke environments and the simulation platform SmokeLab based on volumetric smoke rendering. SmokeManip combines geometrically supervised scene understanding, object-conditioned interaction region prediction, adapted monocular depth estimation, and action primitives with hierarchical recovery mechanisms. The paper reports perception and manipulation results under three smoke densities, component ablation, and real-world robot experiments deployed without active depth sensing.

Critical strengths
  1. The paper connects perception under smoke with actual robot manipulation and provides a platform supporting dynamic volumetric smoke, automatic annotation, action primitives, and combinatorial tasks. This system's contribution has practical value: the evaluation of the complete interaction process can reveal failures that cannot be measured by image-level perception metrics alone.

  2. The paper reports significant performance improvements under different smoke densities. At medium density, the success rates for primitive and composition tasks in Table 2 are 60.0% and 46.7%, respectively, compared to 36.6% and 21.7% for CRT-VLA. Table 3 also provides meaningful component replacement and removal experiments, supporting the role of perception, depth adaptation, pose generation, and recovery mechanisms in the system.

Critical weaknesses
  1. The evaluation in the main text does not adequately explain the generalization range of the performance improvements. Section 5.3 provides the dataset size but does not clarify whether the scene, object instances, and related viewpoints are isolated between training and testing; Section 6.1 also does not clearly state the amount of demonstration data and the training resources used for baseline adaptation. Table 2 lacks a consistent smoke-free control, which would help distinguish the difference between smoke robustness and basic task capabilities. While the tests conducted at varying densities and the ablation study presented in Table 3 are valuable, they do not fully resolve the aforementioned issues regarding the interpretation of benchmark comparison results.

  2. The main text fails to clearly articulate the critical information required for autonomous execution. Section 3.3 describes triggering recovery mechanisms and advancing the task based on the verification of object states, yet it does not specify how these states are estimated during deployment on a physical robot. Removing the recovery mechanism causes the success rate of primitives at medium density to drop from 60.0% to 42.2%; thus, this detail directly influences the assessment of the system's capabilities. Similarly, while Table 3 demonstrates the importance of depth adaptation, Sections 3.2 and 6.4 do not quantify the resulting pose accuracy under actual smoky conditions. Consequently, based solely on the reported success rates, it remains difficult to ascertain the level of state perception and geometric precision necessary to replicate the complete system.

Questions for authors
  1. Please clarify the train/test split for scenes and objects, as well as the data used for baseline adaptation. If available, please also provide success rates in smoke-free conditions for a consistent comparison.

  2. What measurements does the physical robot use to determine the success of an action and trigger recovery? What level of accuracy—in terms of metric depth or contact point localization—can the deployed estimator achieve?

4 comments

Thanks for your helpful and meaningful reviews!

To Weakness1 and Question1:

(1) To " ...clarify the train/test split for scenes and objects..."

We split the dataset at the scene level. SmokeLab provides 28 scenes spanning 5 categories (Kitchen x 20 + Bathroom x 2 + Laboratory x 2 + Store x 2 + Workshop x 2). For each category, half of the scenes are allocated to the training set and the remaining half to the test set. Scenes of the same category use the same set of object instances, while object positions are randomized.

For all scenes, the scene understanding dataset contains 18 randomly-sampled robot-centric viewpoints: 12 local viewpoints and 6 global viewpoints. The affordance prediction dataset reuses the same 12 randomly-sampled robot-centric local viewpoints from the scene-understanding dataset.

(2) To "...the amount of demonstration data and the training resources used for baseline adaptation..."

For all baselines (VLA-based and DP-based), we use 30 demonstration episodes per primitive and 100 demonstration episodes per compositional task. We train all baselines using four NVIDIA RTX 4090 GPUs on Ubuntu 20.04. In addition, all baselines share consistent configurations that comply with each method's original requirements.

(3) To "...provide success rates in smoke-free conditions..."

We provide success rates under smoke-free conditions as supplementary results. For each primitive action and composite task, we conducted 3 rounds of experiments--with 15 trials per round in the real-world, and 45 trials per round in simulation--as shown in the table below. These results demonstrate that our method achieves a balance between smoke robustness and fundamental task-execution capabilities.

Table Manipulation success rate in simulation

MethodSmoke-Free Pri.Smoke-Free Com.Low Pri.Low Com.Medium Pri.Medium Com.High Pri.High Com.
π0.5\pi_{0.5}106/13592/13563/13533/13536/13516/13513/1356/135
OpenVLA-OFT106/13582/13556/13528/13530/13513/13510/1354/135
CRT-VLA100/13576/13583/13546/13549/13529/13527/13514/135
StableVLA94/13570/13572/13540/13543/13523/13520/13511/135
DehazeFormer + DP76/13554/13547/13523/13524/13510/1358/1353/135
DehazeNet + DP70/13548/13544/13520/13522/1359/1357/1353/135
SmokeManip110/13586/13596/13572/13581/13563/13560/13545/135

Table Manipulation success rate in real-world

MethodSmoke-Free Pri.Smoke-Free Com.Low Pri.Low Com.Medium Pri.Medium Com.High Pri.High Com.
π0.5\pi_{0.5}38/4531/4526/459/457/452/455/451/45
OpenVLA-OFT35/4527/4521/456/455/451/454/451/45
CRT-VLA33/4525/4521/4515/4511/454/4514/455/45
StableVLA31/4523/4524/4512/459/453/456/452/45
DehazeFormer + DP24/4517/4518/455/454/451/453/451/45
DehazeNet + DP22/4515/4517/454/454/451/453/451/45
SmokeManip38/4529/4531/4522/4520/4513/4517/4512/45

To Weakness2 and Question2:

(1) To "...physical robot use to determine the success of an action and trigger recovery?"

In real-robot experiments, after executing each action primitive, the system judges whether the current manipulation satisfies the execution conditions for the next stage by combining the gripper state and joint-current feedback from the RealMan robot, as well as scene-understanding outputs from SmokeScene. For fairness, the simulation setting adopts a similar approach (SmokeScene outputs and robot proprioceptive information), without leveraging any privileged simulation-side information.

The judgment schemes for different action primitives are as follows:

Grasp: During execution, the opening-closing position and gripping-force feedback of the RealMan gripper are used to determine whether valid grasping is achieved. After gripper closure, if the feedback does not fall within the predefined valid gripping range, an empty grasp is preliminarily identified. RGB observations from SmokeScene are then used to verify whether the target object remains at its original position or only undergoes minor displacement. If these conditions hold, local re-observation or grasp-pose retry is triggered. If the gripper opening-closing position and gripping force lie within the predefined valid range, the robot proceeds along the predefined motion trajectory, and the gripping state is continuously monitored alongside trajectory-execution progress. When the robot finishes the corresponding motion while maintaining a valid grip, the current grasp stage passes validation, and the next action primitive can be executed.

Push/Pull: The completion of push-pull actions is determined based on motion trajectories generated by ProMP and the robot's end-effector state. Target-segmentation results provided by SmokeScene are adopted to examine target-position changes before and after the action. For operations requiring gripping, the gripping-state-checking mechanism defined in the Grasp primitive is reused. The current action passes stage-wise validation and enables the subsequent primitive only if trajectory execution finishes, the target exhibits the expected observable change, and all relevant feedback remains normal.

Rotate: The gripping-state-checking mechanism of the Grasp primitive is first reused to confirm whether the gripper holds the target validly. Afterwards, rotation trajectories generated by ProMP and actual pose feedback from the robot end-effector are used to verify whether the end-effector achieves the expected angular variation. The current action passes stage-wise validation and enables the subsequent primitive if the grip remains valid and the end-effector completes the predefined rotation.

Press: Completion of the predefined press motion is judged using press trajectories from ProMP and displacement feedback of the robot end-effector. Meanwhile, joint-current feedback from the RealMan manipulator detects contact status during pressing. The action passes stage-wise validation and enables the subsequent primitive when the end-effector reaches the predefined pressing displacement, joint currents exhibit contact variations consistent with expectations, and no abnormal resistance is detected.

It should be noted that the above mechanism only supports simple stage-wise state verification for each action primitive and cannot identify all execution anomalies. For detectable initial failures such as an empty grasp in the grasping phase, the system triggers local re-observation or retry. For anomalies occurring during subsequent primitive execution that cannot be reliably identified automatically (including object dropping, slipping and other conditions) the trial is marked as failed and terminated by human without further automatic-recovery attempts.

To Weakness2 and Question2:

(2) To "What level of accuracy ... can the deployed estimator achieve?"

(i) Relative-depth prediction accuracy. Below we report the prediction accuracy of smoke-adapted DepthAnythingv2 [1] (using metrics from its original paper). Though sufficient for pose generation and contact-point localization, inaccurate depth still leads to experimental failures. This limitation will be discussed in the limitations section.

Table Relative-depth prediction accuracy of smoke-adapted DepthAnythingv2

EnvironmentDistance (m)LowLowLowMediumMediumMediumHighHighHigh
δ₁AbsRelRMSE (m)δ₁AbsRelRMSE (m)δ₁AbsRelRMSE (m)
w/o smoke-adapted
Simulation0.5-20.0470.3180.3230.0480.4180.3570.0520.4850.407
Simulation<0.50.1760.4150.1640.1860.3170.1810.1900.3150.178
Real-World0.5-20.0810.3670.2950.0780.4580.4030.0930.5540.437
Real-World<0.50.2760.4720.1520.2910.5410.2120.2960.4960.227
smoke-adapted
Simulation0.5-20.9880.0230.0160.9850.0280.0190.9810.0410.025
Simulation<0.50.9890.0230.0130.9860.0270.0150.9830.0320.019
Real-World0.5-20.9840.0260.0180.9810.0370.0230.9790.0450.028
Real-World<0.50.9850.0280.0150.9840.0350.0180.9810.0420.023

[1] Yang et al. Depth Anything V2. NeurIPS 2024.

(ii) Deployment error of smoke-adapted depth estimator. We conduct supplementary experiments after deploying this depth estimator: 3 rounds with 15 trials per round in the real-world setting, and 3 rounds with 45 trials per round in simulation. We tabulate the concrete errors (distance, in centimeters, between predicted and target poses) during primitive execution in these supplementary experiments, as shown in the table below.

Table Deployment error of smoke-adapted depth estimator (units in cm)

EnvironmentLow Pri.Low Pri.Medium Pri.Medium Pri.High Pri.High Pri.
min error, max errormean errormin error, max errormean errormin error, max errormean error
Simulation0.02, 3.120.640.02, 4.500.980.27, 5.111.52
Real-World0.01, 2.920.540.06, 5.501.110.19, 6.381.47

To Weakness2 and Question2:

(2) To "What level of accuracy ... can the deployed estimator achieve?"

(iii) Fraction of failures. Failure statistics in the table below are derived from additional experiments. The results show that monocular-depth error remain one of the main sources of failure. We will discuss this limitation in the limitations section.

Table Fraction of failures in simulation

FailurePri. (fail/total)FractionCom. (fail/total)Fraction
scene localization31/40518%27/40512%
affordance prediction17/40510%27/40512%
monocular-depth error43/40526%46/40520%
pose generation39/40523%44/40520%
ProMP-based primitive13/4058%19/4058%
motion planning4/4052%14/4056%
contact execution6/4054%8/4054%
closed-loop recovery15/4059%17/4058%
primitive transitionN/AN/A23/40510%

Table Fraction of failures in real-world

FailurePri. (fail/total)FractionCom. (fail/total)Fraction
scene localization9/13513%10/13511%
affordance prediction7/13510%9/13510%
monocular-depth error14/13521%16/13518%
pose generation15/13522%17/13519%
ProMP-based primitive4/1356%7/1358%
motion planning5/1357%4/1355%
contact execution5/1357%5/1356%
closed-loop recovery8/13512%10/13511%
primitive transitionN/AN/A10/13511%

Thanks again! We welcome any further questions and would be happy to continue the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.