acceptodds
Under review as a conference paper at ICLR 2027

SIGMA: Interaction-Conditioned Geometry Tokenization for Cross-Shape Humanoid Manipulation

11 views

Abstract

Humanoid interaction with diverse objects depends on local surface structures that determine feasible contacts and motion strategies. Yet representing such geometry for policy learning remains challenging: compact global descriptors such as bounding boxes discard interaction-relevant surface details, while raw point-cloud inputs preserve fine-grained geometry but leave local surface structure implicit, increasing the burden on the policy to learn transferable control across shapes. To address this challenge, we propose **SIGMA** (**S**tructured **I**nteraction **G**eometry for humanoid **MA**nipulation), a geometry interface combining Structured Surface Attributes (SSA) with Interaction-Conditioned Geometry Compression (ICGC). SSA organizes object geometry into fixed-size local surface elements that encode position, normal, and represented area. ICGC aggregates these elements according to task compatibility, humanoid body-surface proximity, and surface-area priors, producing a geometry token that adapts to the current interaction state. We integrate SIGMA into a unified multi-task humanoid controller and evaluate frozen-policy transfer without target-object adaptation on objects with identical global extents but different local structures, unseen local geometric perturbations, and diverse furniture geometries. At 50k iterations, SIGMA achieves 58.33% average full-task success across the three evaluation settings, compared with 22.67% for PointNet and 6.76% for AABB-shape-token. On three held-out Sit object groups, SIGMA averages 89.03% target-entry success, compared with 82.52% for PointNet and 61.00% for AABB-shape-token.

Primary area:
applications to robotics, autonomy, planning
Keywords:
humanoid manipulation, geometric representation, surface tokens, reinforcement learning, observational aliasing, zero-shot generalization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mCiROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to encode object geometry for frozen-policy transfer of a unified humanoid controller to unseen shapes. Its proposed interface, SIGMA, combines Structured Surface Attributes (SSA) with Interaction-Conditioned Geometry Compression (ICGC). SSA samples an object mesh and selects local representatives using a joint position-normal distance; each representative stores a position, an aggregated normal, and an estimated surface-area fraction. The implementation uses 26 valid surface elements selected from 2,048 dense samples. A shared encoder processes these descriptors in a humanoid-relative coordinate frame. ICGC pools them into a 64-dimensional token using a learned task query, distances to task-associated body keypoints, and an area prior. The interface is integrated into a TokenHSI-based Transformer policy trained jointly with PPO and AMP on Trajectory Tracking, Sit, Carry, and Climb.

The main evaluation measures object-macro Carry success, requiring pickup, transport, and stable goal completion, on Alias, Notch, and Furniture suites without target-object adaptation. At 50k iterations, SIGMA reports 74.24%, 72.89%, and 27.85%, respectively, compared with 72.53%, 67.82%, and 22.12% for independently trained SSA + Area Pooling. The equal average across the three suites is 58.33% for SIGMA, versus 22.67% for the implemented PointNet and 6.76% for AABB-shape-token; the corresponding SSA + Area Pooling average is 54.16%, calculated from Table 1. The paper also reports frozen-policy interventions, object-wise and initial-state robustness analyses, and a held-out Sit evaluation with 89.03% average target-entry success. The Sit metric measures whether the root enters the target region at any step, and is distinct from sustained supported sitting.

Critical strengths
  1. A useful problem formulation with a well-targeted benchmark and substantial observed representation gains. Alias holds global extent, mass, and inertia fixed while varying local interaction geometry, providing a controlled test of geometry information that a bounding-box token cannot distinguish. The results extend to eight Notch objects and 49 Furniture objects. SSA + Area Pooling improves over the implemented PointNet by 46.07, 34.99, and 13.39 percentage points on these three suites, respectively, under the reported common policy framework and training budget. These are consequential empirical gains, particularly because Carry success requires a complete pickup-transport-placement sequence. Frozen transfer without target-object adaptation is also a useful contribution alongside the shape-adaptation setting in TokenHSI (Pan et al., 2025), where the Carry tokenizer is fine-tuned for new object categories (TokenHSI, Sec. 3.4). The evidence makes compact geometry interfaces a worthwhile subject of study even though the source of the gains needs further isolation.

  2. A coherent compact interface with meaningful controls and robustness checks. Orientation-aware anchor selection and explicit position/normal/area descriptors have a clear geometric motivation, and the area-biased, task/body-conditioned pooling rule is simple and precisely specified. The independently trained SSA + Area Pooling arm is a meaningful control for ICGC as a whole. The positive Furniture completion gap persists under a second initial-state sample (+3.90 points), under the additional stable-termination criterion (+5.47 and +3.14 points for the two samples), and after excluding the source model shared with Climb training. The two available continued seeds also favor SIGMA at the evaluated 55k-70k checkpoints, although those curves use episode-micro rather than object-macro scores. These checks strengthen the case that the observed improvement extends beyond one aggregate evaluation. The reported 10.8% additional geometry-tokenization latency is modest relative to static pooling in the measured setup, with the timing scope explicitly limited to the encoder.

Critical weaknesses
  1. The specific contribution of the proposed geometry design is insufficiently isolated from richer inputs and alternative pooling. The PointNet comparator uses 64 area-uniform xyz-only samples with max pooling. SSA simultaneously introduces preprocessing from 2,048 samples, position-normal anchor selection, explicit normals and area fractions, centroid-based coordinate organization, and different pooling. This establishes a useful complete-interface comparison, but leaves open whether a comparably compact point encoder with normals and matched preprocessing would recover much of the gain. Seen-object Carry results would also help distinguish optimization from transfer: PointNet seed 1 achieves only 0.13%, 0.00%, and 0.07% on the three held-out suites, but its seen-object Carry performance and continuation curves are not reported.

    The independently trained area-pooling control isolates ICGC's combined addition, but its mean benefits are smaller: +1.71, +5.08, and +5.73 points using the authors' unrounded rates, with a negative difference for one of the three training seeds in every suite. The favorable robustness checks deserve credit, while the magnitude and consistency of this incremental effect remain less certain than the headline comparisons. No Task and No Body are frozen-policy interventions, so they establish reliance on those signals rather than the benefit of each component when trained independently. This matters for positioning against CORN (Cho et al., 2024), which already uses hand-conditioned point-cloud representations and task-state cross-attention. A compact learned conditioning baseline and matched-feature controls would clarify the additional value of SIGMA's particular surface construction and explicit distance/area biases. This uncertainty is the main reason for my recommendation.

  2. The evidence beyond Carry does not establish stable supported interaction or an ICGC-specific cross-task benefit. The held-out Sit metric is satisfied when the humanoid root comes within 0.20 m of the target at any simulation step; it does not require sustained seated support, a stable posture, or absence of a subsequent fall . The 89.03% result therefore supports Sit target entry, while the conclusion's claim about whole-body supporting balance is stronger than this measurement establishes. Moreover, Table 5 omits SSA + Area Pooling. The authors correctly acknowledge that this is an interface-level comparison, but it leaves the value of ICGC on the second interaction task unresolved. Near-saturated performance on the other jointly trained tasks in Sec. 4.6 establishes compatibility with that evaluation, not the missing held-out stable-Sit result. Because interaction-dependent geometry across tasks is central to the paper's framing, a sustained-support Sit evaluation with the static-pooling control could materially strengthen the acceptance case.

Questions for authors
  1. Additional evidence for the representation claim: How does a compact point encoder perform with explicit normals and coordinate preprocessing matched to SSA, using the same mesh source and a comparable representation/compute budget? A controlled comparison using the same samples or anchors would help separate descriptor content, anchor selection, and pooling. Please also report seen-object Carry performance for the existing PointNet seeds, especially seed 1, so that poor transfer can be distinguished from incomplete acquisition of Carry within the training budget. These results would directly address Weakness 1.

  2. Additional evidence for ICGC's specific contribution: What happens when No Task and No Body variants are trained from scratch, or when ordinary learned task/body-conditioned pooling is used on the same SSA descriptors without the explicit distance/area biases? Please quantify uncertainty in SIGMA-minus-area-pooling performance across independent training runs and use a common aggregation metric when comparing later checkpoints. I recognize the existing independent area-pooling arm and positive continuation results; the unresolved issue is which conditioning components are responsible for the added benefit.

  3. Additional evidence for the cross-task claim: Do the held-out Sit gains persist under a criterion requiring sustained supported sitting and no fall, and how does SSA + Area Pooling perform under that same criterion? Please specify the dwell, support/contact, and termination conditions used. This would determine whether the current target-entry results support the stronger stable-support interpretation and whether ICGC contributes beyond Carry.

  4. Clarification of the existing experimental protocol: Please identify the exact observation/reward and benchmark configuration files in the described archive, or provide their specification. In particular, which task observations or rewards contain geometry-derived quantities, what surface-anchor set is used in the pickup evaluator, and is that evaluator common across methods? Please also provide the Alias/Notch construction parameters and the asset IDs, counts, and split definitions for the held-out Sit groups. The PDF provides substantial encoder and PPO detail, but these remaining specifications affect independent reproduction and interpretation of the controlled geometry comparison. This is a request to resolve missing information, not an allegation of unequal inputs or data leakage.

3 comments

We address the four questions through expanded experimental specifications and additional analyses in the revised manuscript (changes highlighted in red).

Q1. Matched PointNet and Seen-Object Performance

To test whether PointNet's lower performance stems from raw point inputs or symmetric max pooling, we use Matched PointNet, which feeds the exact same SSA descriptors and preprocessing into a PointNet-style max-pooling aggregator.

Held-out Carry full-task success (%):

MethodAliasNotchFurniture
Matched PointNet67.5869.7926.05
Learned Conditioning45.7727.284.51
SIGMA (Ours)81.9080.4732.71

Giving PointNet structured SSA inputs substantially improves its performance over raw point clouds, but SIGMA still outperforms Matched PointNet by +14.32, +10.68, and +6.66 pp across the three benchmarks, isolating the gain from interaction-conditioned aggregation (ICGC).

To separate optimization failure during training from cross-shape transfer failure, we also evaluate the original PointNet policies on training-seen Carry objects across Seeds 0–2, yielding 22.98%, 0.07%, and 58.40% full-task success, respectively. This shows that Seed 1 fails to acquire the task during training itself, whereas Seeds 0 and 2 learn functional policies on seen objects (up to 58.40%) that subsequently degrade on held-out geometries.

Q2. Contribution of ICGC

We assess ICGC's contribution from two angles: comparing against unconstrained learned attention, and examining paired multi-seed deltas over static area pooling.

First, we use Learned Conditioning on the same SSA features—generating attention queries from task and body states, but omitting SIGMA's explicit body-to-surface distance bias and area prior. As shown in the table above, it reaches only 45.77% (Alias), 27.28% (Notch), and 4.51% (Furniture). Diagnostics across 5 checkpoints and 448,327 valid states confirm that its attention weights become increasingly selective without collapse or numerical instability; however, without explicit spatial and area priors, learned attention fails to ground contact surfaces that generalize to unseen geometries.

Second, we report the paired 3-seed improvements of SIGMA over independently trained SSA + Area Pooling:

MetricAliasNotchFurniture
Paired Difference (pp, Mean ± SD)+1.71 ± 4.69+5.08 ± 12.83+5.73 ± 6.80

In addition, across both available continuation seeds from 55k to 70k iterations, SIGMA remains consistently ahead of SSA + Area Pooling at every evaluated checkpoint under the same episode-micro metric.

Q3. Cross-Task Generalization on Held-Out Sit

We apply a stricter held-out Sit evaluation by requiring the downward root ray to intersect the annotated seat surface within 0.10 m for 30 consecutive control steps, and evaluate SSA + Area Pooling under the same 3-seed protocol. (We note in the text that this measures sustained kinematic seat proximity rather than contact forces or dynamic balance.)

Held-out Sit success (%), mean ± SD over 3 seeds:

MethodStraightChairStoolCoffee TableAverage
PointNet75.35 ± 6.4553.82 ± 8.1665.97 ± 13.4265.05
SSA + Area Pooling76.04 ± 5.7562.85 ± 7.1274.31 ± 8.6571.07
SIGMA (Ours)77.43 ± 5.1267.01 ± 6.3880.90 ± 5.4775.11

SIGMA leads across all three categories, improving average sustained sitting success by +4.04 pp over SSA + Area Pooling and +10.06 pp over PointNet, with the clearest margins on Stool (+4.16 pp) and Coffee Table (+6.59 pp).

Q4. Experimental Protocol and Reproducibility

Appendices B and G.1 now provide expanded documentation of the exact training and evaluation protocols in one place:

  • Shared Setup: All methods use identical PPO/AMP hyperparameters, non-geometric proprioceptive/task inputs, and reward functions.
  • Pickup & Evaluation Protocol: Carry pickup is evaluated using precomputed discrete mesh-bank surface anchors under a unified success criterion across all baselines.
  • Held-Out Splits: We explicitly list all evaluation splits—4 Alias configurations, 8 Notch objects, 49 Furniture objects, and the 3/9/13 held-out Sit objects (StraightChair / Stool / Coffee Table)—along with per-episode sampling and rollout budgets.

Thank you very much for your thorough review and helpful suggestions. With the discussion period closing soon, we wanted to kindly check whether our additional experiments and revisions (highlighted in red) have resolved your concerns. Please let us know if anything further needs clarification in the remaining time. Thank you again for your time and effort!

Thanks for the additional experiments to clarify my concerns. I would like to maintain my score.

Reviewer_1CbXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses on humanoid interaction. The authors note that generalizing across object shapes remains a significant challenge in this domain. They argue that the key to addressing this challenge lies in a structured representation of interaction geometry, and accordingly propose SIGMA, a geometric interface that combines Structured Surface Attributes (SSA) with Interaction-Conditioned Geometry Compression (ICGC). The proposed method is integrated into a unified multi-task humanoid controller, and its effectiveness is validated across various evaluation settings.

Critical strengths
  1. The motivation of this work is commendable; for human-object interaction, coarse-grained object representations are suboptimal for downstream manipulation tasks. More detailed, structured geometric representations for interaction warrant further exploration.

  2. The authors provide a thorough description of the proposed method. Specifically, the Structured Surface Attributes (SSA), and the technical approach is well-integrated and self-contained.

Critical weaknesses

The presentation of experiments and analysis is unclear and confusing, specifically regarding the following parts:

  1. The selection and analysis of "baselines" are overly abstract. Even if the comparisons are conducted at a "paradigm level," it is still necessary to clarify, or even supplement the analysis with comparisons to specific methods, rather than relying on broad, generalized categories like "AABB" and "Point Cloud." Furthermore, the authors should use concrete experimental results to clarify the "technical-level" differences between the proposed method and the baselines, that is, precisely in what aspects the proposed method demonstrates superior performance (with specific examples).
  2. Many experimental configurations are not described intuitively. Even for simulation-based experiments, it is essential to provide visualizations (such as illustrations of the datasets or the task workflow) rather than relying solely on text and quantitative results.
  3. The experimental section lacks visualizations of the robot actions (similar to the input-output examples shown in Figure 2). Given the constraints on space, these should at least be included in the appendix; such visual evidence is essential for a paper of robotics.
  4. The paper does not mention real-world experiments. The authors should discuss whether the proposed method is deployable in real-world settings and identify the primary gaps between the simulation environment and the real world.
  5. The description of results in Section 4.6 is confusing. In cases where two experiments yield the same value (99.67%), the authors should clarify this by stating that both results are 99.67%, rather than using "respectively."
Questions for authors

Please explain the limitations in "Weaknesses."

3 comments

We address the main points through additional baseline comparisons, qualitative visualizations, and clarifications in the revised manuscript (changes highlighted in red).

1. Baseline Comparisons

Appendix G now provides full baseline implementation details, and we report two additional controlled experiments on held-out Carry benchmarks using identical SSA inputs: Matched PointNet (replacing ICGC with symmetric max pooling) and Learned Conditioning (task/body-conditioned attention without explicit distance or area priors).

Held-out Carry full-task success (%, Seed 0):

MethodAliasNotchFurniture
Matched PointNet67.5869.7926.05
Learned Conditioning45.7727.284.51
SIGMA (Ours)81.9080.4732.71

Even with identical surface descriptors, SIGMA consistently outperforms both static max pooling and unconstrained learned attention, confirming the necessity of explicit interaction-conditioned geometric priors.

2. Benchmark and Policy Visualizations

We include two new figures to make our evaluation tasks and policy behaviors concrete:

  • Figure B.1 (Appendix B) illustrates representative unseen objects across the Alias, Notch, Furniture, and held-out Sit benchmarks.
  • Figure 5 shows side-by-side Carry and Sit rollouts comparing SIGMA and PointNet from identical initial states and objects, visually highlighting how SIGMA adapts its grasp and seating approach to local surface geometry.

3. Real-World Applicability

Architecturally, SIGMA's interface does not rely on simulation-only privileged states. In a real-world pipeline, SSA surface descriptors can be extracted online from reconstructed object meshes (via RGB-D depth fusion or pre-scanned object models), while the task and body-configuration queries come directly from standard onboard state estimation and whole-body controllers.

Bridging the sim-to-real gap would primarily require handling partial point cloud occlusion, mesh reconstruction noise, and real-world contact dynamics discrepancies. In this work, we focus on isolating how structured, interaction-conditioned geometry drives cross-shape generalization in controlled simulation, laying the representational foundation for future hardware deployment.

4. Section 4.6 Clarification

Section 4.6 now accurately states that both SIGMA and SSA + Area Pooling achieve 99.67% success on Trajectory Tracking.

Thank you again for your valuable feedback. As the discussion period is ending soon, we wanted to kindly follow up to check if our response and revised manuscript (with changes highlighted in red) have addressed your concerns.
Please let us know if you have any remaining questions before the deadline. We truly appreciate your time and consideration!

Thanks for the supplementary experiments and further elaboration on this work. I would like to maintain my score.

Reviewer_uDx9OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SIGMA represents object geometry through compact surface elements containing positions, normals, and area weights. Task and body-proximity cues guide their aggregation into a geometry token for a unified humanoid controller. Simulation experiments evaluate transfer to unseen shapes in carrying and sitting.

Critical strengths
  • The controlled Alias benchmark changes local interaction geometry while holding global bounds, mass, and inertia fixed, directly testing information that bounding boxes miss.
  • The structured representation substantially improves over bounding-box and PointNet baselines. A separately trained static-pooling baseline, varied held-out furniture, multiple seeds, and targeted interventions strengthen the evaluation.
Critical weaknesses
  • The methodological advance appears incremental: familiar surface descriptors and attention pooling are combined with task and body-distance biases within TokenHSI. Most gains already arise with static pooling; interaction conditioning adds modest, variable improvements, although longer-training results on two seeds are encouraging. Its advantage over stronger matched alternatives remains unclear: PointNet receives sparse XYZ-only samples, while SIGMA uses normals and area attributes derived from denser samples. Frozen-policy interventions establish reliance on conditioning, rather than superiority over separately trained alternatives.
  • Evidence for reliable whole-body support remains limited. Furniture completion improves under stricter termination checks, but falls also increase. Held-out sitting only measures entry into a target region, which does not establish stable seated contact or balance.
Questions for authors
  1. How does SIGMA compare with a stronger point encoder given matched surface attributes and sampling, and with learned attention pooling trained without the proposed task/body-distance biases?
  2. Do the held-out sitting gains persist when success requires sustained seated contact and balance?
2 comments

We address both questions through new controlled baselines and a stricter Sit evaluation.

Q1. Stronger Point Encoders and Learned Attention Pooling

To isolate the contribution of ICGC under identical surface inputs, we compare two additional baselines using the exact same SSA surface descriptors and preprocessing:

  • Matched PointNet: Replaces ICGC with symmetric max pooling over SSA descriptors, testing whether a standard point-style aggregator suffices when given structured surface features.
  • Learned Conditioning: Replaces ICGC with an independently trained task/body-conditioned cross-attention module without our explicit body-to-surface distance bias or area prior, testing whether generic learned attention can discover interaction-relevant geometry on its own.

Full-task Carry success (%, Seed 0) on held-out benchmarks:

MethodAliasNotchFurniture
Matched PointNet67.5869.7926.05
Learned Conditioning45.7727.284.51
SIGMA (Ours)81.9080.4732.71

SIGMA outperforms Matched PointNet by +14.32, +10.68, and +6.66 pp on Alias, Notch, and Furniture, respectively, showing that static symmetric pooling loses critical interaction context even when provided with SSA features.

Furthermore, Learned Conditioning underperforms significantly, dropping to 4.51% on Furniture. To verify that this is not caused by optimization failure, we examine its attention distributions across 5 checkpoints and 448,327 valid states: the baseline develops increasingly selective attention over training without entropy collapse or numerical instability. Its lower task performance indicates that without explicit geometric inductive biases (body-to-surface distance and patch area), unconstrained attention struggles to ground contact-relevant surfaces that generalize to unseen shapes.

Q2. Sustained Sitting on Unseen Objects

We use a stricter held-out Sit criterion requiring the humanoid's downward root ray to intersect the annotated seat surface within 0.10 m for 30 consecutive control steps (serving as a strict kinematic proxy for stable seating rather than instantaneous target entry, though distinct from direct contact-force measurement). We also evaluate SSA + Area Pooling under this exact protocol across all three seeds.

Held-out Sit success (%), mean ± SD over 3 seeds:

MethodStraightChairStoolCoffee TableAverage
PointNet75.35 ± 6.4553.82 ± 8.1665.97 ± 13.4265.05
SSA + Area Pooling76.04 ± 5.7562.85 ± 7.1274.31 ± 8.6571.07
SIGMA (Ours)77.43 ± 5.1267.01 ± 6.3880.90 ± 5.4775.11

SIGMA achieves the highest success across all three object categories, outperforming PointNet by +10.06 pp and SSA + Area Pooling by +4.04 pp on average (with the largest gains on Stool and Coffee Table, where seat heights and geometries vary more widely). Section 4.5 and Table 5 now report these results, while Figure 5 shows paired Carry/Sit rollouts.

Thank the authors for the response. Most of my concerns are addressed. I have updated my rating.

Comments on the paper

1 comment

We thank the reviewers for their positive and constructive feedback. We address their comments in the revised manuscript, with changes highlighted in red.

To directly address the reviewers' questions, we present new controlled baseline experiments, apply stricter cross-task evaluation criteria, and provide expanded visualizations and reproducibility details. Key updates include:

Reviewer(s)Core TopicNew Evidence & Manuscript Updates
uDx9, 1CbX, mCiRFairer & stronger geometric baselinesWe compare two new controlled baselines on identical SSA inputs: Matched PointNet (symmetric max pooling) and Learned Conditioning (task/body cross-attention without explicit distance or area priors). SIGMA outperforms both across all Carry benchmarks (detailed in individual responses).
uDx9, mCiRIsolating ICGC's contributionWe combine the new Learned Conditioning diagnostics (448,327 states across 5 checkpoints), paired 3-seed deltas over Area Pooling, and 55k–70k continuation curves to isolate the role of explicit geometric priors (Sections 4.3–4.4, Tables 1–4, Appendix D.2).
uDx9, mCiRSustained Sit & cross-task transferWe apply a strict 30-step sustained seat-proximity criterion and include SSA + Area Pooling under the same protocol. SIGMA achieves the highest average success at 75.11% vs. 71.07% for Area Pooling and 65.05% for PointNet (Sections 4.1 & 4.5, Table 5, Appendix C).
1CbXVisual evidence & baseline clarityWe include a held-out object gallery (Figure B.1 in Appendix B), paired Carry/Sit rollout comparisons under identical initial states (Figure 5), baseline implementation details (Appendix G), and clarify the Trajectory Tracking text (Section 4.6).
mCiRPointNet training & reproducibilityWe evaluate PointNet on training-seen objects to separate optimization failure from transfer failure, and provide full protocol specifications (Appendices B–G).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.