acceptodds
Under review as a conference paper at ICLR 2027

EMBODIED XAI: TRACING HIERARCHICAL AGENT DECISIONS VIA 4D SPATIO-TEMPORAL SALIENCY

Abstract

Robots steered by vision-language models are being proposed for spaces people occupy, where an unaccountable choice carries real cost. Yet when these physical AI agents silently pick a target, per-frame 2D saliency cannot say what in the physical world drove them: the planner is re-queried continuously while the robot drives, so each map sits in its own moving camera frame, leaving thousands of disconnected images that cannot be pooled, sliced, or compared. We present 4D spatiotemporal saliency, a model-agnostic framework that relocates attribution from the camera to the scene: an attribution stage computes importance scores with respect to the planner's sampled steering token, each attributed pixel is lifted into world coordinates through depth and pose, and the results accumulate across queries and trials into one heat field growing through time. The explanation thereby becomes an object that can be added up, grouped by outcome, differenced against a reference, replayed, and physically intervened upon. We instantiate it with chosen-action integrated gradients on a quadruped driven by ambiguous instructions, and use the field to certify scene-level attribution patterns as repeatable, separate the signatures of success and failure, watch a decision harden, and run closed-loop occlusion experiments that persistently target the same physical region as the camera moves.

Primary area:
interpretability and explainable AI
Keywords:
explainable AI, embodied AI, vision-language models, spatiotemporal saliency, hierarchical reinforcement learning, integrated gradients, bias auditing

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_VJn4OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes tracing hierarchical agent decisions via 4d spatio-temporal saliency. The motivation behind the research is that per-frame 2D saliency does not successfully handle the rich 3D environment. The authors design a testbed with a VLM-PPO agent to verify the motivation. By analyzing the results of the testbed, the paper presents a 4D (3D + time) heat map.

Critical strengths
  1. Motivation: the paper presents a clear motivation. The storyline in the introduction section that motivates the proposed study is straightforward.

  2. Testbed and analysis for method design: the authors present the testbed and experimental results before illustrating the proposed method for better understanding the motivation.

Critical weaknesses
  1. The experiment: the test conditions and tasks are too simple to generalize over multiple situations and there is no comparative study demonstrating the effectiveness of the proposed method.

  2. The method: accumulating saliency over multiple regions does not necessarily represent the true saliency. Feeding the overall map or other information and querying it for saliency estimation would simply be able to substitute for the proposed method.

Questions for authors
  1. Can you compare the proposed method against 3D VLMs and geometric VLMs? Comparing the proposed method with related areas could present the contributions in a more effective manner.

  2. Can the proposed method enhance the performance of robot tasks by tracing saliency?

  3. Does reasoning over a top-view map replace the proposed method?

4 comments
AuthorsOpenReview ↗

Response to Reviewer VJn4 (Weaknesses W1 & W2)

W1. "The tasks are too simple, and there is no comparative study."

The simple arena is a deliberate choice. Our contribution is not a navigation policy. It is a way to place attribution in the world, so that the decisions of a moving robot can be audited. To test an explanation method, we first need a setting where the factors behind a decision are clear. Section 4 builds one. The bare two-cube arena, the mirrored layouts and the same-color control let us measure the agent's biases exactly (Table 1). Clutter would hide them. Section 4 states this design choice.

The framework is not tied to this arena. The paper already tests it beyond the main setup:

  • a moving target in a dynamic arena (Appendix L);
  • three further planners: InternVL3-2B, SmolVLM2-2.2B (Appendix K), and an API-only Gemini planner (Appendix I);
  • four other attributors, including black-box RISE (Appendix I);
  • the low-level controller on their own (Appendix J).

On a comparative study: we know of no prior method that gives world-anchored attribution for a robot that moves while it decides. The natural baseline is per-frame 2D saliency. The new Figure 9 (page 10) compares the two on the same 20 episodes. The 2D baseline gives the auditor 1,084 maps from 301 queries, each in its own camera pose. These maps cannot be added, sliced or compared. Our method turns the same maps into one field. Every analysis in the paper runs on that field: pooling (Figure 4), outcome splits (Figure 5) and closed-loop masking (Section 6). None of these has a per-frame version. In particular, a 2D image mask cannot follow a physical region while the camera moves, so occlusion auditing of a driving robot is not even defined in 2D (Section 1, contribution 3). Appendix M shows how the 2D pile grows with the memory window.

W2. "Accumulated saliency is not true saliency. Feeding an overall map and querying it could replace the method."

We agree that accumulated attribution is not "true" causal saliency, and we do not claim it is. Section 1 ("Scope of the claim") states that we do not propose a new definition of pixel importance. Section 5.3 states that our guarantees establish neither semantic correctness nor causal faithfulness. Our attributor, Integrated Gradients, satisfies known axioms such as sensitivity and implementation invariance. It has the same standing as 2D saliency in computer vision: it shows what drove the output, not a causal proof. We point it at the exact steering token the robot sampled (Section 5.1). We then test faithfulness separately, with closed-loop occlusion (Section 6) and with matched random regions (Appendix Q).

The accumulation is also not a raw sum. Each world bin shows the mean attribution per time the bin was seen, so how often a region was seen is divided out (Appendix O). Proposition 1 shows that the pooled field converges as trials are added.

On feeding an overall map: the robot never has one. Its planner sees only its onboard camera (Section 3, Figure 3a). Our goal is to audit the robot as deployed, using only what it actually saw. A model queried with a top-down map is a different system with different inputs. Its answer would describe that model, not the policy that drove our robot. Asking a model what it found important also yields a self-report, which is not attribution (Jacovi & Goldberg, 2020). Adding aerial or map data, for example from a drone, is an interesting direction, but it is outside the scope of this paper. We have listed it as future work in Section 8.

I sincerely appreciate the authors for taking the time to prepare the response.

I still question the significance of the technical contribution and particularly the generality of the method. In addition, as the authors have not answered my questions, my concerns about those issues remain unresolved.

AuthorsOpenReview ↗

Response to Reviewer VJn4 (Questions and Contribution)

Thank you for your reply, and we apologize for not answering your three questions directly. We answer them first, then return to the technical contribution and its generality.

Q1. Can you compare the proposed method against 3D VLMs and geometric VLMs?

Our method does not compete with 3D or geometric VLMs. Those are planners, the part of the robot that makes decisions. Our method is an audit layer that explains what any planner relied on. So the two do not replace each other. They combine, and the combination gives a richer heat field.

3D VLMs. These models take depth or point clouds as input, often alongside images. Integrated Gradients applies to depth the same way it applies to pixels. The path grows each depth value from zero to its true value. Each 3D point then receives its own attribution score. These scores already live in 3D, so they enter the world-anchored field directly. Pixel and point attributions then share one world frame and can be compared. This shows whether the two inputs drive the decision from the same regions of space or from different ones. A clear division of labor would be useful in practice: each input could be limited to the regions it relies on, which would cut input size and latency.

Geometric VLMs. These models read several camera views at once and reason about the 3D scene behind them. Here our method applies with almost no change. Each view gets its own attribution map, and each map is unprojected with its own camera pose (Eq. 2). Without our method, an auditor would face one map per camera per query, which multiplies the pile of Appendix M by the number of cameras. With our method, all views fuse into one shared field. The auditor can also keep one field per camera to compare them.

Our framework needs only a per-pixel or per-point score, plus depth and pose (Section 1, contribution 1). This is why it already transfers across four attributors and three further planners (Appendices I and K).

Q2. Can the proposed method improve the performance of robot tasks by tracing saliency?

This paper focuses on auditing, not on improving performance, and we do not claim performance gains. But the field gives a natural training signal. A human or a larger teacher VLM can mark the task-relevant objects as 3D bounding boxes, static or moving. The gap between these boxes and the agent's own heat field can then serve as a loss for fine-tuning. This follows the idea of saliency supervision: penalize attribution that falls outside the relevant regions.

World-anchoring makes this practical. A 3D box is labeled once and projected into every frame automatically. In 2D, the same label would have to be drawn again in every frame. Our own agent shows where this could help. Its field holds a persistent lobe on the background wall in every outcome group (Figure 4, Appendix O). Supervision of this kind could steer attention back toward the targets.

Q3. Does reasoning over a top-view map replace the proposed method?

No. A top-view map changes the robot's inputs, for example to a camera on a drone instead of the robot's own camera. Our method still applies to that system. It would then explain which parts of the drone's view drove the decision, instead of the robot's view.

This is a promising setting. In search and rescue, ground robots may work with aerial drones. Many agents then move and decide at the same time. A per-frame explanation would give one stack of maps per agent. A world-anchored field puts every agent's attribution onto one shared map of the terrain. An auditor can rotate that map and replay it over time to see which places influenced the team's decisions.

Technical contribution and generality:

To our knowledge, no prior method anchors attribution in the world for an agent that moves while it decides. Our contribution turns a pile of camera-tied maps into one object that can be pooled, sliced, replayed and intervened on.

The method is general by construction. It needs three things: a per-pixel score, depth and camera pose. Any attributor provides the first (Appendix I). Most mobile robots already estimate the other two for navigation, through SLAM or lidar-inertial odometry. Appendix G shows the reading holds under pose and depth noise. The paper also runs it on two further open planners (Appendix K), a black-box API planner (Appendix I) and a moving target (Appendix L).

The same idea extends beyond vision. Robots increasingly read sound, touch and other sensors along with video. Any signal with a known position in space can be anchored in the same field. Finding the best way to attribute and visualize such mixed signals will take work across XAI, robotics and visualization. We see world-anchored attribution as a first, necessary step toward that goal.

I sincerely appreciate the authors for taking the time to prepare this response.

I still think that other previous research works can be applied to the proposed problem setting with a slight modification. The exact match of the input format is not always possible, so some related baselines I mentioned in the previous comments should be incorporated and compared. Furthermore, I am not certain about the significance of the technical contribution. The auditing should deliver some value for other systems. Without showing the benefit of the proposed system, I would question the need to formulate one. Lastly, I think the proposed method can be easily replaced by other conventional methods, such as top-view maps, as I pointed out in Q3. The authors could do an extensive literature survey and compare related methods in a table or some effective way to demonstrate the superiority of the proposed method.

Reviewer_ivU3OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the interpretability and auditability of decisions made by VLM-steered embodied agents. Because the robot re-queries its planner continuously while it drives, per-frame saliency maps each correspond to a different, constantly moving camera pose. In this setting they are difficult to aggregate and compare across time and trials. The authors propose relocates attribution from the camera to the scene; unprojects each attributed pixel into world coordinates through depth and pose and accumulating them over time and trials into a unified 4D spatiotemporal heat field. The field supports convergence sweeps, outcome-split fields, time-resolved accumulation, and closed-loop occlusion auditing.

Critical strengths
  1. The paper addresses an important and practically relevant problem. As VLM-steered robots move toward human-shared spaces, the auditability of their decisions deserves attention. The motivation for moving attribution from image space into a shared world frame is clearly presented.

  2. The paper places attributions from every query of every trial into one shared world frame. Results from many trials can then be accumulated to certify scene-level patterns as repeatable rather than single-trial noise. They can also be compared across outcome groups, such as reached vs. drifted, or replayed over time to show how a decision forms. This offers a practical way to analyze the decisions of a moving robot.

  3. The framework does not depend on a particular attribution method, and many existing methods can be plugged in directly. The authors also demonstrate it on a model that is only accessible through an API, which suggests it could be applied to different types of models.

Critical weaknesses
  1. The behavioral probe identifies a positional bias as the dominant effect, most clearly in the same-color control (87% left, Table 1). The heat fields help explain the secondary color effect (the nearer-blue group concentrates heat on the blue cube in Fig. 5), but no heat field is shown for the same-color control. It is therefore not entirely clear how the heat field explains the positional bias. Showing the heat field for the control condition would help connect the explanation to the main behavioral finding.

  2. The heat field is obtained by directly summing attributions, without accounting for how often each region is seen. The target the robot walks toward appears more often in view and may therefore collect more heat. As a result, the regions where heat concentrates are not necessarily the ones the model relies on more.

Questions for authors
  1. How much computation time and resources are needed to produce a 4D heat field? Would the approach remain practical for longer tasks or larger models?

  2. If the heat field is normalized by how often each region is seen, do the differences between outcome groups in Fig. 5 still remain?

3 comments
AuthorsOpenReview ↗

Response to Reviewer ivU3 (Weaknesses W1 & W2)

Thank you for a clear and constructive review. Both of your points led to new appendices.

W1. "No heat field is shown for the same-color control."

The new Appendix R (Figure 38) adds it. We drove the both-red control for 20 trials per instruction and sliced each field by the side the agent ended nearer and by success. Friendly trials ended nearer the left target in 14 of 20 (70%), and hostile trials in 11 of 20 (55%) for this smaller-scale trial.

With color removed, the left cube carries more heat than the right. However, the strongest heat sits on the back wall behind the left target. It appears in every slice, even in trials that ended nearer the right target. This links the positional bias more to the left-side background region. Appendix O finds the same back-wall lobe in every outcome group of the main condition.

W2. "Summing attributions ignores how often each region is seen."

Thank you, this is an important point, and our main text did not explain it well. Figure 5 does not show a raw sum. Each world bin shows the summed attribution divided by the number of times the bin was seen. Exposure is therefore already divided out. A patch seen 300 times and a patch seen 10 times show the same value if their average attribution is the same.

The new Appendix O checks this directly. We re-drove the condition until every outcome group had 10 trials. From the same points we built four views (Figure 36): summed attribution, normalized attribution, raw exposure and normalized exposure. Normalized exposure is flat, so the normalization does remove exposure. Summed attribution looks much like exposure, as you expected. Normalized attribution does not: the most-seen surfaces, including the cube being approached, are not where attribution per view is highest.

We now state this exposure normalization explicitly in Section 5.2.

AuthorsOpenReview ↗

Response to Reviewer ivU3 (Questions)

Q1. How much computation is needed, and does it scale to longer tasks or larger models?

The cost depends mostly on the attribution method, not on our framework. In the main setup, Integrated Gradients with 8 steps needs 8 forward and backward passes per query. This is the same cost as making ordinary per-frame 2D saliency maps. Anchoring the maps in the world adds very little. It uses only depth and camera pose, with one small matrix product per pixel (Eq. 2). It never calls the model, so its cost does not depend on model size.

For longer tasks, the cost grows linearly with the number of queries. The field itself stays the same size, because it lives on a fixed grid of world bins. For larger models, only the attribution step gets more expensive. Cheaper attributors can be swapped in without changing anything else (Appendix I).

Q2. If the field is normalized by exposure, do the differences between outcome groups in Figure 5 remain?

Figure 5 is already normalized by exposure, as explained in our response to W2. Appendix O (Figure 36) repeats it with equal groups of 10 trials. Therefore, differences between outcome groups in Fig. 5 still remain.

Thanks for the response. It addresses my concerns and answers my questions. I’ve raised my score to 3.

Reviewer_M195OpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper maps image attributions for a VLM planner’s steering commands into a shared 3D scene and accumulates them over time. Experiments with a simulated quadruped examine spatial preferences, trajectory-level patterns, responses to visual occlusion, and sensitivity to localization noise.

Critical strengths
  1. The paper addresses a practical problem: comparing attribution across changing camera viewpoints. A shared spatial representation makes these comparisons easier to interpret.

  2. Separating attribution from spatial aggregation allows different attribution methods and planners to use the same analysis pipeline.

Critical weaknesses
  1. The diagnostic interpretation is not sufficiently separated from differences in observation history.

    1. Figure 5 compares accumulated fields from different trajectories and unequal outcome groups, including 5 nearer-red and 15 nearer-blue trials. Because Section 5.2 accumulates attribution over observations, repeatedly visible regions have more opportunities to contribute. The paper does not establish whether the outcome-specific patterns remain after accounting for this exposure. Clarifying the group normalization and comparing against uniformly weighted projections of the same camera histories would help identify what the attribution scores add.

    2. Section 5.5 contrasts 2 direction reversals in the reached trial with 5 in the drifted trial. However, Table 4 reports durations of 3.1 and 7.9 seconds, giving nearly identical reversal rates of approximately 0.65 and 0.63 per second. These raw counts alone provide limited support for greater directional instability in the failed trial. Comparing reversal rates across multiple trajectories would strengthen this part of the failure diagnosis.

  2. The occlusion experiments provide limited validation of the selected attribution hotspots.

    1. Section 6 and Figure 7 include useful controls, but BLACK-COLD expands to cover almost everything outside HOT. Appendix E explicitly acknowledges that this comparison does not localize the decision. The realized masking areas also differ in the mirrored experiment: 20.5% for BLACK-COLD versus 32.1% for BLACK-HOT in Table 6. These differences complicate attributing the behavioral effects specifically to the selection of high-saliency regions. Several low-saliency or random control regions matched for projected area and spatial structure would provide a stronger comparison.

    2. Figure 8 shows that attribution shifts after masking, but this does not distinguish the redundant-cue interpretation proposed in Section 6 from changes in trajectories or fallback behavior. The authors appropriately describe the study as exploratory. Its results demonstrate the feasibility of the auditing procedure more clearly than they establish that the selected hotspots explain the agent’s decisions.

Questions for authors
  1. Normalization and comparability

    1. Beyond the per-query normalization specified in Table 2, how are observations and trials weighted when constructing the outcome-specific fields in Figure 5?

    2. Do the panels in Figure 5 share a common numerical color scale, or is each field rescaled independently for display?

  2. Diagnostic evidence

    1. Using the cached trajectories, could you compare the fields in Figure 5 with projections that assign uniform weights to the same valid pixel observations? Which outcome-specific patterns remain distinctive to the attribution scores?

    2. Could you compare direction-reversal rates between reached and drifted trials across the recorded trajectories, normalized by duration or query count and reported with confidence intervals?

  3. Occlusion controls and interpretation

    1. Beyond the controls in Figure 7, could you compare HOT against several low-saliency or randomly selected regions calibrated to cover a similar image area on the same clean observations?

    2. What evidence beyond the attribution shifts in Figure 8 supports the redundant-cue interpretation over explanations involving changed trajectories or fallback behavior?

  4. Real-world validation

    1. Beyond the reported simulation experiments, are any preliminary physical-robot results available? If so, how stable are the main spatial attribution patterns when depth and camera poses are estimated from real sensors?
2 comments
AuthorsOpenReview ↗

Response to Reviewer M195 (Weaknesses W1 & W2)

Thank you for a careful and detailed review. Each of your four points led to a new analysis.

W1: 1.1 "Figure 5 compares unequal groups, and exposure is not controlled."

Appendix O addresses this. We re-drove the condition until each outcome group had 10 trials. We also ran the comparison you proposed: the same camera histories projected with a uniform weight of 1. This gives exposure, the number of times each place was seen (Figure 36c). Figure 5 divides summed attribution by this exposure (Figure 36b), so group size changes noise and coverage, not brightness. The result: summed attribution mostly tracks exposure. Normalized attribution does not, and normalized exposure is flat. This normalization is what was used for Figure 5. What the normalized attribution scores mainly reveal is a persistent back-wall lobe that appears in every group.

1.2 "The reversal counts in Section 5.5 are not rates."

You are right. Appendix P (Table 10) conducts 10 drifted and 10 reached trials. Per second, the rates are similar: 0.63 vs. 0.51 (rate ratio 1.22, exact Poisson test p=0.29). Drifted trials reverse more in every matched opening window, but the gap is significant only at T = 1.5 s (p=0.03). Reversal rate alone is therefore a weak failure signal at this sample size. We have added these rate comparisons to Section 5.5.

W2: 2.1 "BLACK-COLD does not localize, and the mask areas differ."

We agree, and Appendix E already notes it. The new Appendix Q replaces this control. We compare HOT with four random regions that keep HOT's shape and match its projected area. Each is a fixed world region, re-projected into every frame. All arms run on the same 10 seeds and are scored against each seed's own clean rollout (Table 11).

HOT disrupts the robot the most. It gives the lowest reach (30% vs. 40-70%), the largest path divergence (0.91 m vs. 0.73-0.89 m) and the largest end displacement (2.69 m vs. 2.02-2.57 m). It does this while hiding less of the frame during the actual drive (realized mask of 29.1%) than three of the four random regions. It does not cause more choice flips than the random regions. With 10 seeds per arm, the reach gap is not significant (Fisher p=0.22). We therefore report it as directional, not conclusive.

2.2 "The shift in Figure 8 does not prove the redundant-cue reading."

We agree. In a closed loop, a mask changes the path, and a new path changes what the camera sees. Both redundant cues and a changed trajectory can move the attribution, and our data cannot yet separate them. We have reworded Section 6 to state that this shift "is consistent with redundant cues or with a changed trajectory." Appendix Q now reports path divergence for the hot region and the random regions, in a separate 10-seed run, so the size of the trajectory change is visible. We also accept your overall reading. The occlusion study shows that the audit can be run in a closed loop. It does not yet show that the selected hotspots explain the decisions. We now state this plainly at the end of Section 6.

AuthorsOpenReview ↗

Response to Reviewer M195 (Questions)

Thank you for the questions. Most are answered in our first response, so we keep this short.

  1. Normalization and comparability. Trials are pooled with no weighting by group size. Each world bin shows the mean attribution per time it was seen, so exposure is divided out. Each panel is scaled to its own 99th percentile, so panels compare in pattern, not in magnitude. Appendix O gives the full details, and Section 5.2 now states them.

  2. Diagnostic evidence. See our response to W1 (points 1.1 and 1.2), Appendix O (uniform-weight projections) and Appendix P (reversal rates).

  3. Occlusion controls and interpretation. See our response to W2 (points 2.1 and 2.2) and Appendix Q (area-matched random regions).

  4. Real-world validation. We have not yet run the framework on a physical robot. As a first step, Appendix G adds pose, yaw and depth noise to the simulated drives. A hardware port is our next step (Section 8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.