acceptodds
Under review as a conference paper at ICLR 2027

DexBeyond: Scaling Dexterous Manipulation Through Iterative Hand-Object Interaction Generation

Abstract

Learning dexterous manipulation depends on continuous hand-object interaction data that capture fine-grained finger coordination. Many existing datasets emphasize grasping and placement, with limited coverage of both in-hand motion diversity and object variety. We present , a unified framework that combines a large-scale hand-object interaction dataset, a generative expansion model, and an iterative data flywheel to scale dexterous hand-object interaction data. We first construct , a large-scale hand-object interaction dataset comprising 20,891 sequences and 58.03 hours of valid hand-object interaction. It covers 219 object instances from 77 object categories and 30 manipulation behaviors, and provides synchronized 3D hand and object poses with the corresponding object meshes. Because exhaustive real-world capture is impractical, we develop to expand interaction diversity for captured objects and augment the dataset with generated motions, thereby improving downstream controller performance. Although DexBeyond-G can propose interactions for unseen objects by replacing its mesh condition, the resulting motions often exhibit penetration, lost contacts, and other physical inconsistencies. We therefore introduce , an iterative, self-improving data flywheel that turns these proposals into feasible interaction data, enabling generalization to unseen objects without additional real-world collection. Experiments demonstrate that DexBeyond scales interaction synthesis across motions and objects, with downstream gains in simulation and the real world and progressive performance improvements on unseen objects.

Primary area:
generative models
Keywords:
Hand-Object Interaction Generation, Dexterous Manipulation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JrAGOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents DexBeyond, a framework for scaling dexterous manipulation data through a combination of large-scale human hand-object interaction collection, motion generation, and iterative simulation-based data expansion. The framework consists of three components: DexBeyond-D, a real-world hand-object interaction dataset; DexBeyond-G, a conditional diffusion model for generating interaction trajectories conditioned on object geometry and motion descriptions; and DexBeyond-FW, an iterative flywheel that refines, filters, and expands generated interactions for unseen objects through simulation rollouts.

Critical strengths
  1. The scale and coverage of DexBeyond-D are impressive for dexterous hand-object interaction data. Compared with prior datasets focused on grasping or smaller-scale in-hand manipulation, the dataset substantially expands both object diversity and continuous interaction diversity. The combination of long-horizon interaction sequences, reconstructed MANO trajectories, and object meshes makes the resource potentially useful for multiple downstream problems beyond the specific experiments in the paper.

  2. The paper presents a fairly comprehensive end-to-end system for scaling dexterous manipulation data. In addition to collecting real interaction data, the authors integrate motion generation, geometry-aware refinement, feasibility filtering, and iterative rollout expansion into a unified pipeline. While many components build on existing ideas, the overall system is well engineered and the experimental section is reasonably thorough, including downstream controller evaluation in both simulation and limited real-world deployment.

Critical weaknesses
  1. The methodological novelty appears relatively limited compared with the scale of the system. Many core components, including conditional diffusion-based HOI generation, geometry-aware contact and penetration losses, simulation-based feasibility filtering, and iterative rollout expansion, have strong precedents in prior work on hand-object interaction generation and robotic data flywheels. As a result, the main contribution seems to come more from engineering integration, scaling, and dataset construction rather than from fundamentally new modeling or learning ideas. While the resulting system is comprehensive, it is less clear what the key conceptual advance is beyond combining existing ingredients at larger scale.

  2. The claims around unseen-object generalization rely heavily on simulation-based refinement and evaluation. DexBeyond-FW generates candidate interactions, filters them through Isaac Gym-based physical checks, and improves performance through iterative simulation rollouts, but the paper provides only limited real-world validation. Given the well-known sim-to-real challenges in dexterous in-hand manipulation, especially for contact-rich finger coordination and friction-sensitive interactions, it remains unclear whether the generated “physically feasible” interactions correspond to robust real-world behaviors. This concern is important because a central claim of the paper is scalable generalization to unseen objects without additional demonstrations, yet this capability is not convincingly validated outside simulation.

Questions for authors
  1. The paper positions DexBeyond-FW as enabling generalization to unseen objects without additional demonstrations. However, all iterative refinement and rollout expansion are performed in simulation. Do the authors have any evidence that the synthesized interactions for unseen objects transfer to real-world execution, even qualitatively? This would substantially affect the strength of the paper’s central claim.

  2. The text prompts are automatically constructed from discretized object translation and rotation statistics extracted from the target trajectories. Could the authors clarify how much the generation task depends on these tightly coupled motion descriptors? In particular, how different is this setting from conditioning directly on low-level motion summaries rather than semantic language-guided interaction generation?

  3. Many components of the framework appear closely related to prior HOI generation and robotic data flywheel pipelines. Could the authors clarify which components they believe provide the primary conceptual advance beyond scaling and system integration?

This is still a relatively underexplored area. I would like to raise my score if clearer evidence around novelty are offered.

5 comments
AuthorsOpenReview ↗

Response to W1 and Q3: Methodological Novelty, Conceptual Contributions, and the Data-Centric Flywheel Framework [1/3]

We thank the reviewer for the detailed review. We clarify the conceptual contributions of DexBeyond and provide additional experimental evidence below.

W1 & Q3: Methodological Novelty and Conceptual Contributions

W1: The methodological novelty appears relatively limited compared with the scale of the system.

Q3: Many components of the framework appear closely related to prior HOI generation and robotic data flywheel pipelines. Could the authors clarify which components they believe provide the primary conceptual advance beyond scaling and system integration?

R1 (W1/Q3). We would like to emphasize that DexBeyond is not simply an engineering integration of existing modules, but a unified, data-centric framework for scaling dexterous manipulation across interaction and object diversity.

Conceptual distinction and design rationale. Existing HOI generation methods (e.g., Text2HOI, LatentHOI) primarily focus on motion synthesis, while robotic data flywheels (e.g., DexFlyWheel) emphasize policy improvement through iterative learning and rollouts. In contrast, DexBeyond investigates how to systematically scale dexterous manipulation data itself.

We argue that a framework with genuine scaling and self-improvement potential must address three interdependent challenges: (1) establishing a high-quality data foundation, (2) expanding interaction diversity beyond captured demonstrations, and (3) generalizing to unseen objects through iterative feedback. These requirements motivate DexBeyond-D, DexBeyond-G, and DexBeyond-FW, respectively. The components are therefore designed to overcome successive bottlenecks, rather than arbitrarily assembled.

(1) DexBeyond-D: Establishing the data foundation. Scaling first requires high-quality, diverse interaction data, which existing datasets provide only to a limited extent. We construct DexBeyond-D with 58.03 hours of interactions, 219 objects across 77 categories, and 30 behaviors, providing 46.8× more interaction duration and 12.8× more object categories than DexterHand, the closest dataset in scope.

(2) DexBeyond-G: Expanding interaction diversity. However, even large-scale collection cannot exhaustively cover all valid interactions, while additional demonstrations are costly. We therefore develop DexBeyond-G to generate diverse interactions conditioned on object geometry and motion descriptions, improving downstream controller SR from 60.2% to 69.1% (Table 2).

(3) DexBeyond-FW: Generalizing to unseen objects. Although DexBeyond-G can propose interactions for unseen objects, these often contain penetration, missing contacts, and self-collisions, making them unsuitable for direct controller training. This motivates DexBeyond-FW, which uses geometric refinement (GR), physical feasibility detection (PFD), cross-round merge (CRM), and iterative rollouts to transform these proposals into feasible interaction data. Successful rollouts are fed back into the generator, enabling self-improvement without additional demonstrations for unseen objects.

To isolate the contribution of generator updates, we introduce a fixed-generator baseline, where the generator remains frozen after round 1, with matched controller-training budgets and identical test references:

SettingFixed GeneratorUpdated Generator
SR↑ (%)12.240.7

The full flywheel substantially outperforms the fixed-generator baseline (40.7% vs. 12.2%). Furthermore, removing GR+PFD or CRM reduces SR from 44.3% to 19.2% or 14.3%, respectively, in the separate three-object ablation (Table 5), demonstrating their essential roles in effective data expansion.

In summary, our primary conceptual contribution is a closed-loop framework that connects real interaction data, geometry-conditioned generation, and physically grounded self-improvement to progressively scale both interaction and object coverage.

AuthorsOpenReview ↗

Response to W2 and Q1: Real-World Generalization, Physical Feasibility, and Sim-to-Real Transfer of DexBeyond-FW [2/3]

W2 & Q1: Real-World Generalization to Unseen Objects

W2: DexBeyond-FW generates candidate interactions, filters them through Isaac Gym-based physical checks, and improves performance through iterative simulation rollouts, but the paper provides only limited real-world validation.

Q1: Do the authors have any evidence that the synthesized interactions for unseen objects transfer to real-world execution, even qualitatively?

R2 (W2/Q1). We provide additional real-world experiments to address your concern regarding the transferability of DexBeyond-FW.

(1) Why real-world FW results were not originally presented. We clarify that DexBeyond-FW is evaluated using real captured interaction trajectories, rather than generated trajectories. In DexBeyond-FW, all target-object demonstrations are excluded from generator training, controller training, and flywheel updates, and are reserved for evaluation. Thus, our original simulation evaluation already measures the ability to track real captured interactions on unseen objects, with the sim-to-real gap being the primary additional challenge in physical deployment.

(2) Additional real-world validation. We deploy DexBeyond-FW on the SharpaWave robotic hand and compare the reported sim-to-real SR retention across three training strategies:

Training DataSimulation SR↑Real-World SR↑Retention↑
Real Data60.2%42.1%69.9%
Real + Generated Data69.1%54.4%78.7%
DexBeyond-FW40.7%29.8%73.2%

DexBeyond-FW retains approximately 73.2% of its simulation SR, comparable in magnitude to the reported ratios of real-data-based controllers. Although the evaluation object sets differ, these results suggest that FW-generated interactions do not introduce disproportionately severe sim-to-real degradation.

(3) Qualitative comparison. In DexBeyond-FW evaluation, we compare ring-object manipulation under three settings: (1) FW-trained controller in simulation, (2) real-data-trained controller in the real world, and (3) FW-trained controller in the real world. The two real-world controllers exhibit similar manipulation patterns, supporting the real-world transferability of FW-generated interactions.

Together, these results demonstrate that DexBeyond-FW supports real-world manipulation of unseen objects without additional target-object demonstrations for training, with comparable relative sim-to-real transfer performance.

W2: Physical Feasibility and Sim-to-Real Transfer

W2: Given the well-known sim-to-real challenges in dexterous in-hand manipulation, especially for contact-rich finger coordination and friction-sensitive interactions, it remains unclear whether the generated "physically feasible" interactions correspond to robust real-world behaviors.

R3 (W2). We clarify the role of simulation-based feasibility evaluation and our sim-to-real transfer strategy.

(1) Simulation-based dexterous manipulation. Training dexterous controllers in simulation and transferring them to real robots is a widely established approach, particularly because collecting large-scale real-world demonstrations for high-DoF in-hand manipulation is challenging.

Our PFD follows this paradigm, combining geometric checks with 10–20 physics simulation steps to detect penetration, slipping, or ejection. Its purpose is to provide physically feasible and trackable reference subgoals for controller training and exploration, rather than guarantee real-world success.

(2) Preparing for sim-to-real transfer. To address actuator latency, joint noise, friction, and object-property variations, we apply domain randomization during controller training, including joint noise, external forces, wrist orientation perturbations, and variations in object density and friction. These measures improve robustness to simulation inaccuracies.

(3) Real-world transfer results. As shown in R2, DexBeyond-FW achieves 40.7% SR in simulation and 29.8% in the real world, retaining approximately 73.2% of its simulation SR. Additional demonstrations are provided in real-world evaluation and DexBeyond-FW.

Therefore, PFD provides feasible reference subgoals rather than guaranteed real-world execution, while our additional experiments demonstrate that the resulting interactions can successfully transfer to unseen objects despite the sim-to-real gap.

AuthorsOpenReview ↗

Response to Q2: Motion-Intensity vs. Semantic Conditioning and Language-Based Prompt Ablation [3/3]

Q2: Motion-Intensity Prompts versus Semantic Language Conditioning

Q2: The text prompts are automatically constructed from discretized object translation and rotation statistics extracted from the target trajectories. Could the authors clarify how much the generation task depends on these tightly coupled motion descriptors? In particular, how different is this setting from conditioning directly on low-level motion summaries rather than semantic language-guided interaction generation?

R4 (Q2). We emphasize that motion-intensity conditioning is a deliberate choice aligned with our goal of expanding interaction diversity, rather than generating predefined semantic behaviors.

(1) Why motion-intensity prompts? Compared with high-level behavior labels, motion-intensity descriptions directly characterize object translation and rotation magnitudes over two temporal segments, providing finer control over in-hand motion diversity. These descriptions are derived from captured trajectories during training, while the desired conditions can be specified at inference time.

(2) Comparison with semantic conditioning. We additionally train a separate DexBeyond-G variant using behavior-label prompts and compare both strategies:

MethodPen.↓Cont.↑Cells↑RS↑FS↑FST↑PCT/s↑CMT/s↑
Motion-intensity levels0.1192.5133.04154.8426.1231.457.284.98
Behavioral semantics0.1295.5418.9156.7418.7424.706.033.06

Motion-intensity conditioning outperforms behavioral conditioning on all six interaction-diversity metrics, supporting our choice for diversity expansion. In contrast, behavioral conditioning enables specified semantic interactions but offers less direct control over motion characteristics. Qualitative results are provided in DexBeyond-G (template) and DexBeyond-G (behavior).

(3) Why language-based prompts instead of discrete values? We compare T5-based motion-intensity prompts with direct discrete conditioning using Prompt-acc. Specifically, we extract the maximum object translation and rotation from each of the two segments of a generated trajectory, discretize them using the same intensity thresholds as the input prompts, and compute the matching accuracy for translation, rotation, and their joint agreement (Both).

MethodTranslation↑Rotation↑Both↑
Discrete44.62%71.82%33.18%
Ours (T5)46.81%81.22%39.94%

Our T5-based prompts outperform discrete conditioning across all three metrics. Moreover, natural-language conditioning can be readily extended to higher-level semantic interaction generation, offering greater flexibility than task-specific discrete codes.

Thus, our prompts are intentionally structured motion descriptions rather than unrestricted semantic instructions, better serving our objective of scaling interaction diversity while retaining the flexibility of language-based conditioning.

We hope these additional experiments and clarifications address your concerns regarding methodological novelty, real-world generalization, physical feasibility, and motion conditioning. We would appreciate your reconsideration of the assessment.

Reviewer_zoHuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces DexBeyond-D, a 58-hour motion-capture dataset of right-hand in-hand interactions with 219 rigid objects across 77 categories, released with MANO hand poses, object poses, and meshes. On top of it, the authors train DexBeyond-G, a diffusion model conditioned on object geometry and a templated motion-intensity prompt, with contact and penetration losses. They further propose DexBeyond-FW, a simulation loop that generates proposals for unseen objects, refines and filters them for physical feasibility, trains a tracking controller on the survivors, and fine-tunes the generator on successful rollouts. Adding generated data to real data improves tracking success in simulation and on a real 22-DoF hand, and the flywheel reaches about 40% success on four unseen objects after four rounds.

Critical strengths
  1. Dataset scale and quality. The dataset is roughly an order of magnitude larger than the closest in-hand manipulation dataset, covers far more object geometries, and uses a careful reconstruction pipeline (per-subject shape calibration, robust marker fitting, contact-aware post-optimization) that yields low-penetration, high-contact trajectories.
  2. The flywheel concept, with real-world evidence that generated data helps. Closing the loop between a geometry-conditioned generator, geometric refinement, physics-based feasibility filtering, and controller rollouts is a sound and practical way to obtain robot-space interaction data for objects with no demonstrations. The ablations show that refinement, feasibility detection, and cross-round merging each contribute, and augmenting captured data with generated data improves tracking success on a physical hand, not only in simulation, even though that evaluation is narrow.
Critical weaknesses
  1. The method uses only a small part of what the dataset advertises, and several claims are overstated. The 30 behavior labels are never used as conditions; the "text" condition is a template generated automatically from the trajectory's own translation and rotation magnitudes (L201-203, L902-913), so the model cannot be prompted for a behavior. This also weakens the comparison to Text2HOI and LatentHOI, which are evaluated on prompts they were not designed for. Articulated objects are captured and modeled as single rigid bodies with one 6-DoF pose and one mesh, and the paper does not explain how their articulation is handled. Including nuts and bolts that cannot be unscrewed, or scissors that cannot be closed, defeats the purpose of the category. Finally, the real-world evaluation uses five handle-shaped objects held palm-up by the handle, which is effectively one grasp family tested five times rather than a test of object diversity. Together these make the framing around 30 behaviors, articulated objects, and text-conditioned generation materially stronger than what is demonstrated.
  2. The flywheel claim is not isolated and is vulnerable to mode collapse. The success curve across rounds comes from a single run without mean and standard deviation. The controller is warm-started and trained on pooled data each round, and the test references are regenerated each round, so the improvement could come from controller accumulation or from the generator drifting toward motions the controller already tracks. No fixed-generator control is reported. The generator is fine-tuned only on its own filtered rollouts, and the selection keeps the extremes of translation and rotation, which is a recipe for collapse. The diversity metrics are computed per rollout and averaged, so they cannot detect collapse across the set of rollouts. In addition, the feasibility check is a per-frame static equilibrium test, which excludes dynamic behaviors by construction and limits the kinds of manipulation the flywheel can ever produce. Without these controls, the central self-improvement claim is unsupported.
Questions for authors
  1. Is the flywheel success curve reproducible across seeds? Please report mean and standard deviation, and a control where the generator is frozen after round one while the controller is trained for the same number of rounds.
  2. Did you observe mode collapse? Please report set-level diversity per round if possible.
  3. How are articulated objects represented? Is a per-part pose stored, and was the articulation actuated during capture?
  4. What is the exact prompt template, how many intensity levels are used, and how does the model perform with no text or with the raw discretized values instead of T5?
  5. Can the generator produce different behaviors for the same object, for example hammering versus in-hand reorientation of the hammer? If not, what is the role of the behavior labels in the release?
  6. How many trials per object were run in the real-world experiment, and what was the success criterion?
7 comments
AuthorsOpenReview ↗

Response to W1 and Q5: Motion-Intensity vs. Behavior Conditioning and Fairness of Baseline Comparisons [1/5]

We sincerely thank you for the thorough and constructive review. We address each concern below.

W1 & Q5: Behavior Labels and Motion Conditioning

W1: The 30 behavior labels are never used as conditions; the "text" condition is a template generated automatically from the trajectory's own translation and rotation magnitudes.

Q5: Can the generator produce different behaviors for the same object? If not, what is the role of the behavior labels in the release?

R1 (W1/Q5). We would like to clarify that using motion-intensity prompts rather than behavior labels is a deliberate choice aligned with our task objective, not a limitation of DexBeyond-G.

(1) Role of behavior labels. We define 30 manipulation behaviors in DexBeyond-D to cover diverse everyday interactions and support downstream applications, including behavior-conditioned generation.

(2) Why motion-intensity prompts? DexBeyond-G and DexBeyond-FW aim to expand interaction diversity, rather than generate predefined semantic behaviors. Compared with high-level behavior labels, motion-intensity prompts directly characterize object translation and rotation magnitudes, providing finer control over in-hand motion diversity.

(3) Behavior-conditioned generation. Although the original DexBeyond-G is conditioned on motion intensity, it can generate different interaction behaviors for the same object, as demonstrated in DexBeyond-G (motion-intensity). We additionally train a separate DexBeyond-G variant conditioned on behavior labels, with results shown in DexBeyond-G (behavior). We compare the two conditioning strategies below:

MethodPen.↓Cont.↑Cells↑RS↑FS↑FST↑PCT/s↑CMT/s↑
Motion-intensity levels0.1192.5133.04154.8426.1231.457.284.98
Behavioral semantics0.1295.5418.9156.7418.7424.706.033.06

Motion-intensity conditioning achieves higher scores on all six interaction-diversity metrics, supporting our choice for diversity expansion. These results also demonstrate that the original DexBeyond-G can generate diverse behaviors for the same object, while a separately trained variant supports explicit semantic behavior conditioning.

W1: Fairness of Baseline Comparisons

W1: This also weakens the comparison to Text2HOI and LatentHOI, which are evaluated on prompts they were not designed for.

R2 (W1). We would like to clarify that Table 3 does not use the original pretrained weights of Text2HOI or LatentHOI. Instead, we retrain both methods on DexBeyond-D using identical motion-intensity prompts and matched computational budgets, ensuring a consistent comparison.

We additionally retrain the baselines using behavior-label conditions and compare them with our separately trained behavior-conditioned variant:

MethodPen.↓Cont.↑Cells↑RS↑FS↑FST↑PCT/s↑CMT/s↑
Text2HOI0.0870.7516.6337.193.795.770.780.64
LatentHOI0.1098.9119.4451.7517.7724.406.052.99
Ours0.1295.5418.9156.7418.7424.706.033.06

Our method achieves the best results on four interaction-diversity metrics (RS, FS, FST, and CMT/s), while remaining competitive on contact-related metrics.

AuthorsOpenReview ↗

Response to W1 and Q3: Articulated Object Representation and Real-World Evaluation Diversity [2/5]

W1 & Q3: Articulated Object Representation

W1: Articulated objects are captured and modeled as single rigid bodies with one 6-DoF pose and one mesh.

Q3: How are articulated objects represented? Is a per-part pose stored, and was the articulation actuated during capture?

R3 (W1/Q3). We apologize for the insufficient description. DexBeyond-D retains separate meshes and motion-capture-derived poses for articulated object components, including their relative motion during capture, rather than treating them as single rigid bodies. Representative demonstrations are provided in DexBeyond-D.

Given the root and child poses, we compute their relative transformation as Trel(t)=Tr(t)−1Tc(t)T_{rel}(t)=T_r(t)^{-1}T_c(t). Using multiple sequences of the same object instance, we estimate a shared joint axis, reference transformation T0T_0, and per-frame joint coordinates q(t)q(t).

For revolute joints (e.g., scissors), we model rotation around the calibrated axis. For screw joints (e.g., bolt and nut), we additionally model coupled axial displacement with pitch h=P/(2π)h=P/(2π), where PP is the signed axial displacement per revolution.

Thus, DexBeyond-D retains per-component pose trajectories and meshes, together with fitted joint parameters and per-frame joint coordinates. Both the original component poses and fitted joint representations will be included in the dataset release, and we will clarify this in the revised manuscript.

W1: Real-World Evaluation and Object Diversity

W1: The real-world evaluation uses five handle-shaped objects held palm-up by the handle, which is effectively one grasp family tested five times rather than a test of object diversity.

R4 (W1): Real-World Evaluation and Object Diversity. We respectfully believe this concern partly stems from a misunderstanding of our experimental setup and research scope.

(1) Experimental setup. Our real-world experiments use a palm-down, not palm-up, configuration, which is more challenging because the hand must prevent the object from falling under gravity. Moreover, the five objects are not all handle-shaped: they include a sixprism, a cylinder, and three everyday tools. We selected these objects to cover common everyday manipulation scenarios and evaluated two fundamental in-hand motions: translation and reorientation.

(2) Research scope. DexBeyond focuses on constructing, diversifying, and scaling dexterous manipulation data across interaction and object diversity, rather than developing a general-purpose controller or solving the sim-to-real gap. We include real-world evaluation to strengthen our experimental evidence. This validation is an additional strength, not a prerequisite for establishing our core contributions. Therefore, its limited coverage should not be considered evidence against the contributions of our data-centric framework.

(3) Interaction diversity. DexBeyond-D covers 219 object instances, 77 categories, and 30 manipulation behaviors, as demonstrated in DexBeyond-D. Importantly, dataset diversity should not be conflated with downstream controller capabilities, which are limited by controller performance and the sim-to-real gap.

(4) Additional demonstrations. To further address your concern, we provide additional controller demonstrations covering diverse grasp configurations and manipulation patterns in both simulation and the real world.

Overall, the limited real-world evaluation does not undermine the independently evaluated diversity of DexBeyond-D. We hope these clarifications better convey the scope and contributions of our work.

AuthorsOpenReview ↗

Response to W2, Q1, and Q2: Flywheel Reproducibility, Fixed-Generator Ablation, and Mode Collapse Analysis [3/5]

W2 & Q1: Reproducibility and Fixed-Generator Control

W2/Q1: Please report mean and standard deviation, and a control where the generator is frozen after round one while the controller is trained for the same number of rounds.

R5 (W2/Q1). We conduct additional experiments with four independent random seeds, reporting the mean and standard deviation across flywheel rounds. We also introduce a fixed-generator baseline, where the generator is frozen after round 1 while the controller receives the same total training budget. All evaluations use identical test reference trajectories.

SettingFixed GeneratorRound 1Round 2Round 3Round 4
SR↑ (%)12.210.76±5.4318.76±5.2126.10±3.1042.00±3.50

SR improves consistently across rounds, reaching 42.0±3.5%, compared with only 12.2% for the fixed-generator baseline. This demonstrates that the improvement cannot be attributed solely to controller training accumulation, supporting the contribution of iterative generator updates.

W2 & Q2: Mode Collapse, Set-Level Diversity, and Generator Drift

W2/Q2: Did you observe mode collapse? Please report set-level diversity per round if possible.

W2: The generator is fine-tuned only on its own filtered rollouts, and the selection keeps the extremes of translation and rotation, which is a recipe for collapse. The diversity metrics are computed per rollout and averaged, so they cannot detect collapse across the set of rollouts.

R6 (W2/Q2). To assess mode collapse, we evaluate Prompt-acc and FID across all objects for steerability and distributional fidelity. Prompt-acc measures agreement between input prompts and motion descriptions reconstructed from generated trajectories using the same discretization rules (Both: joint translation-rotation accuracy). For target objects, we measure set-level cosine similarity (CS↓) using a DexBeyond-D-trained encoder, alongside RS, FST, PCT/s, and CMT/s.

RoundPrompt-acc(Both)↑FID↓CS↓RS↑FST↑PCT/s↑CMT/s↑
136.69%1.800.522148.5113.182.632.14
233.83%3.290.446155.3832.597.754.44
333.90%4.320.409153.4534.968.344.59
433.93%4.140.407152.8038.309.255.16

Although FID initially increases, it slightly decreases in the final round, while Prompt-acc remains stable at approximately 34% from rounds 2 to 4. For target objects, CS consistently decreases from 0.522 to 0.407, with substantial improvements in interaction diversity. These results suggest that our approach mitigates excessive degradation and sustained drift while reducing the risk of mode collapse.

This benefits from three complementary designs:

(1) Generator updates. Each round initializes from the original checkpoint GR(0)\mathcal{G}_{\mathrm{R}}^{(0)} and uses the top 15 successful rollouts ranked by translation diversity and the top 15 by rotation diversity, together with retained data from previous rounds. This helps limit cumulative drift.

(2) Cross-Round Merge (CRM). We retain diverse controller-training trajectories across rounds, preventing previously successful interaction patterns from being discarded.

(3) Rollout diversification. We introduce cross-trajectory subgoal switching during controller training and vary starting subgoals, gravity, friction, and success criteria during rollouts. For example, reducing gravity to 2.0 m/s² enables additional interactions beyond the original training conditions.

W2: The improvement could come from the generator drifting toward motions the controller already tracks.

We further address this concern from two perspectives:

(1) Generator diversity. As shown above, CS consistently decreases, while RS remains high and FST, PCT/s, and CMT/s increase substantially. These results indicate that the generator maintains broad motion coverage and increasingly diverse contact transitions, rather than converging toward repetitive interactions.

(2) Controller performance across motion categories. We classify object motions into left/right translation and clockwise/counterclockwise rotation using 60-frame windows in the wrist coordinate system, and evaluate controller success rates:

RoundLeft↑Right↑Clockwise↑Counterclockwise↑
17.6933.3042.906.60
223.1033.3071.4060.00
346.2050.0092.8060.00
446.2050.00100.0093.30

Success rates improve or remain unchanged across all four motion categories, rather than concentrating on a single preferred direction. Together with the generator-diversity results, this provides evidence against the concern that the flywheel merely drifts toward a narrow set of easily tracked motions.

Reviewer_DBJpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces DexBeyond, a framework for scaling up in-hand object interaction data for downstream applications such as training tracking controllers. The authors first present a large-scale motion-capture dataset of hand-object interactions, substantially larger in scale compared to related datasets. They then use this dataset to train a transformer-based diffusion model that generates new interaction trajectories conditioned on object geometry and prompts describing the desired translation or rotation. Finally, they introduce a flywheel system to extend the approach to unseen objects without collecting additional human demonstrations. This system refines and filters generated trajectories, uses them to train a tracking controller in simulation, and feeds successful controller rollouts back into the generative model to increase output quality.

Critical strengths

The main strength of this work is the collection of in-hand hand-object interaction data at a substantially larger scale than existing datasets. Collecting such data requires considerable effort, and increasing its availability could be valuable for dexterous manipulation research. The capture setup described in the appendix also appears to reflect a high-quality data collection effort.

Generating hand-object interaction trajectories is also a promising direction, particularly when collecting additional demonstrations is difficult. A steerable generator could supplement limited training data or produce reference trajectories for guiding a controller at test time. Conditioning generation on object geometry allows the model to transfer patterns learned from existing interactions to objects.

Critical weaknesses

The dataset is not sufficiently demonstrated for reviewers to assess its value. I view the dataset as the paper’s main contribution, but the provided material does not give enough visibility into the collected interactions to judge their quality or coverage. The capture setup looks impressive, and the authors show a few trajectories and real-world demonstrations, but these do not provide a sufficiently representative view of the dataset. For a contribution centered on collecting interaction data, it is important to show what that data actually contains through clear examples and a more in-depth showcase of the collected interactions. The reported scale and quality metrics are promising, but they do not by themselves establish the fidelity or usefulness of the dataset. For example, the highlighted contact ratio measures the presence of contact rather than whether the reconstructed contact configuration matches the captured interaction. Since the reconstruction procedure explicitly encourages contact, a high score alone cannot establish that this optimization preserves the original interaction.

The evaluation does not adequately establish the generator’s steerability or the flywheel’s claimed improvement. Prompt conditioning is a central motivation for the generator, yet the paper does not demonstrate whether generated motions follow the requested translation and rotation, either through explicit prompt–output examples or a quantitative evaluation of adherence. Since these conditions describe measurable motion properties, this is a basic gap in evaluating the proposed capability. The flywheel adapts to the target objects using selected successful controller rollouts and reports increasing controller success on those same objects. Repeatedly updating the generator in this way raises concerns about model collapse, particularly the potential loss of diversity and steerability over successive rounds. The presented results do not provide sufficient evidence that degradation is prevented.

Questions for authors
  1. Can the authors provide a substantially more extensive and in-depth showcase of the dataset beyond the few examples currently shown? The reported metrics alone do not give enough visibility into the collected interactions to assess their quality, coverage, and usefulness.

  2. Can the authors demonstrate whether generated interactions follow the requested translation and rotation, through both explicit prompt–output examples and quantitative evaluation? This is central to assessing the usefulness of the proposed steerable generator.

  3. How much generated data is added to the real data for each controller in the augmentation experiments? For each flywheel round, how many trajectories are generated, survive filtering, and are used to train the controller and update the generator?

  4. How is the generator retrained across flywheel rounds, and what training data is retained during these updates? Beyond increased controller success, what evidence shows that repeatedly training on selected successful rollouts does not degrade the generator’s diversity, steerability, or original capabilities?

12 comments
AuthorsOpenReview ↗

Response to W1 and Q1:Dataset Visualization, Downstream Utility, and Reconstruction Fidelity [1/3]

W1 & Q1: Dataset Showcase, Utility, and Fidelity

W1: The dataset is not sufficiently demonstrated for reviewers to assess its value. Q1: Can the authors provide a substantially more extensive and in-depth showcase of the dataset beyond the few examples currently shown?

R1 (W1/Q1): Dataset Showcase. To comprehensively showcase DexBeyond-D, we provide an anonymous DexBeyond-D visualization featuring representative interactions across different wrist orientations, manipulation behaviors, and articulated objects, illustrating the dataset's diversity, coverage, and reconstruction quality.

W1: The reported scale and quality metrics are promising, but they do not by themselves establish the fidelity or usefulness of the dataset.

R2 (W1): Dataset Usefulness and Reconstruction Fidelity.

Regarding usefulness, we validate DexBeyond-D through two downstream tasks:

(1) Hand-object interaction generation. We train DexBeyond-G on DexBeyond-D (Table 3), with additional visualizations under DexBeyond-G (motion-intensity) and DexBeyond-G (behavior), the latter using a separately trained behavior-conditioned DexBeyond-G variant.

(2) Robotic dexterous manipulation. We train controllers using DexBeyond-D and evaluate them in simulation (Table 2; SimDemos) and the real world (Table 4; RealworldDemos). These results establish the dataset's utility for downstream controller learning beyond visual plausibility.

Regarding fidelity, we understand your concern that contact-aware post-optimization may alter the captured interactions. Our reconstruction includes the marker-fitting loss LmarkerD\mathcal{L}_{\mathrm{marker}}^{\mathrm{D}} to preserve the original motion. We also provide three additional evaluations:

(1) Marker reconstruction accuracy. The mean absolute error between raw mocap markers (blue) and reconstructed MANO markers (red) is 1.0056 mm, as visualized in DexBeyond-D.

(2) Human Evaluation of Contact Fidelity. We additionally collected 20 minutes of synchronized mocap and RGB recordings using the same capture protocol. Expert evaluators compared the reconstructed contact configurations against the RGB videos in 2-second segments. 95.7% of the evaluated segments were judged to preserve the raw contact configurations. Synchronized comparisons are available in HumanEval.

(3) Reconstruction ablation. We remove the penetration and contact losses during reconstruction to assess their influence:

DatasetDur.↑Cat.↑Inst.↑Pen.↓Cont.↑RS↑FST↑PCT/s↑CMT/s↑
OAKINK29.2148740.0396.7440.705.021.150.68
HOI4D7.6793730.3276.3572.7913.493.042.76
ARCTIC2.0211110.0775.72109.1818.113.982.89
DexterHand1.24680.0897.92110.1529.397.303.22
Ours58.03772190.0699.85104.2928.377.053.88
w/o pen+contact58.03772190.0395.75104.0125.996.703.21

Even without geometric losses, DexBeyond-D maintains competitive physical-plausibility and interaction-diversity metrics. Together with marker reconstruction accuracy and human evaluation, these results provide additional evidence of reconstruction fidelity.

Finally, we will release both raw capture data and reconstructed hand-object trajectories, enabling independent verification and flexible downstream use.

AuthorsOpenReview ↗

Response to W2, Q2, and Q4: Generator Steerability, Diversity Preservation, and Model Collapse Analysis [2/3]

W2 & Q2: Generator Steerability

W2/Q2: The paper does not demonstrate whether generated motions follow the requested translation and rotation, either through explicit prompt–output examples or a quantitative evaluation of adherence.

R3 (W2/Q2). We further evaluate DexBeyond-G's steerability through qualitative and quantitative analyses.

(1) Qualitative demonstrations. We provide generated interactions conditioned on DexBeyond-G (motion-intensity). These examples illustrate how generated motions follow their respective conditions.

(2) Quantitative prompt adherence. We introduce Prompt-acc, which converts generated trajectories back into two-segment motion descriptions using the original prompt construction rules, then measures their agreement with input conditions for translation, rotation, and both jointly.

MethodTranslation↑Rotation↑Both↑
Text2HOI22.72%37.98%12.59%
LatentHOI44.54%83.23%36.28%
Ours46.81%81.22%39.94%

All methods are trained on DexBeyond-D and evaluated using the same protocol. DexBeyond-G achieves the highest translation (46.81%) and joint accuracy (39.94%), with competitive rotation accuracy (81.22%), demonstrating its ability to follow motion-intensity conditions.

W2 & Q4: Generator Diversity and Model Collapse

W2: Repeatedly updating the generator in this way raises concerns about model collapse, particularly the potential loss of diversity and steerability over successive rounds.

Q4: Beyond increased controller success, what evidence shows that repeatedly training on selected successful rollouts does not degrade the generator's diversity, steerability, or original capabilities?

R4 (W2/Q4). We conduct additional evaluations across four flywheel rounds:

(1) Steerability and distributional fidelity. We measure joint Prompt-acc (Both) and FID against real interactions across all objects.

(2) Set-level and interaction diversity. For target objects, we measure inter-trajectory cosine similarity (CS↓) using a feature encoder trained on DexBeyond-D, alongside RS, FST, PCT/s, and CMT/s.

RoundPrompt-acc↑FID↓CS↓RS↑FST↑PCT/s↑CMT/s↑
136.69%1.800.522148.5113.182.632.14
233.83%3.290.446155.3832.597.754.44
333.90%4.320.409153.4534.968.344.59
433.93%4.140.407152.8038.309.255.16

Although FID initially increases, it decreases slightly in the final round, while Prompt-acc remains stable at approximately 34% from rounds 2 to 4. For target objects, CS consistently decreases from 0.522 to 0.407, with substantial improvements in FST, PCT/s, and CMT/s. These results suggest that our approach mitigates excessive degradation and sustained drift while reducing the risk of mode collapse, retaining steerability across all objects while expanding interaction diversity on target objects.

We attribute this to three complementary designs:

(1) Generator updates. Each round starts from the original checkpoint GR(0)\mathcal{G}_{\mathrm{R}}^{(0)}. We update it using the top 15 successful rollouts ranked by translation diversity and the top 15 by rotation diversity, together with retained data from earlier rounds. This reduces cumulative drift and preserves learned capabilities.

(2) Cross-round controller training. Cross-Round Merge (CRM) retains diverse trajectories from multiple rounds, preventing previously successful interaction patterns from being discarded.

(3) Controller training and rollout diversification. We introduce cross-trajectory subgoal switching during training and vary starting subgoals, gravity, friction, tracking tolerances, and success criteria during rollouts. For example, gravity of 2.0 m/s² enables additional motions beyond the original training conditions, further expanding interaction diversity.

Together, these mechanisms and evaluations provide evidence that our flywheel mitigates mode collapse while progressively expanding interaction diversity.

AuthorsOpenReview ↗

Response to Q3 and Q4: Data Quantities, Cross-Round Allocation, and Generator Update Protocol [3/3]

Q3: Data Quantity and Flywheel Training Protocol

Q3: How much generated data is added to the real data for each controller in the augmentation experiments? For each flywheel round, how many trajectories are generated, survive filtering, and are used to train the controller and update the generator?

R5 (Q3). We apologize for the unclear description of data quantities and selection procedures.

For data augmentation, we add generated sequences equal to 50% of the real training sequences for each controller.

For DexBeyond-FW, we initially generate 200 candidate trajectories per round, followed by geometric refinement (GR) and physical feasibility detection (PFD). Valid trajectories are ranked by translation and rotation diversity. If the current round cannot satisfy its Cross-Round Merge (CRM) quota, we generate another 200 candidates with a new random seed and resampled prompts, repeating the process until the quota is met.

Each controller uses 80 selected trajectories per round, allocated across rounds as follows:

Training RoundRound 1 DataRound 2 DataRound 3 DataRound 4 DataTotal
18000080
240400080
3252530080
42020202080

For example, round 3 selects 25, 25, and 30 diversity-ranked trajectories from rounds 1, 2, and 3, respectively. Generator updates use separately selected successful rollouts, as detailed in R6. We will clarify this protocol in the revised manuscript.

Q4: Generator Update Across Flywheel Rounds

Q4: How is the generator retrained across flywheel rounds, and what training data is retained during these updates?

R6 (Q4). At each round, we initialize the generator from the original checkpoint GR(0)\mathcal{G}_{\mathrm{R}}^{(0)}. Then we select the top 15 successful rollouts by translation diversity and the top 15 by rotation diversity from the current round, combining them with retained data from earlier rounds to construct DR(r+1)\mathcal{D}_{\mathrm{R}}^{(r+1)}.

We then train for 2,000 iterations to obtain GR(r+1)\mathcal{G}_{\mathrm{R}}^{(r+1)}. Reinitializing from the original checkpoint and retaining diverse cross-round data help limit cumulative drift and mitigate mode collapse.

We sincerely hope these additional experiments and clarifications address your concerns regarding dataset quality, generator steerability, and flywheel stability. We would greatly appreciate your reconsideration of your assessment in light of this additional evidence.

Comments on the paper

3 comments
AuthorsOpenReview ↗

Quick Summary of Contributions and Rebuttal Updates

Dear Area Chair,

We sincerely thank you and the reviewers for your time and constructive feedback. To facilitate your assessment, we briefly summarize the recognized strengths, key rebuttal updates, and manuscript revisions.

1. Recognized Strengths

  • Dataset Scale and Value (DBJp, zoHu, JrAG). All three reviewers recognized the substantial scale and value of DexBeyond-D, comprising 58.03 hours, 219 objects, 77 categories, and 30 behaviors.
  • Generation and Practical Utility (DBJp, zoHu). DBJp recognized hand-object interaction generation as a promising direction, while zoHu highlighted the practical value of our iterative flywheel.
  • Comprehensive Framework (zoHu, JrAG). Reviewers recognized the well-engineered pipeline and downstream validation in both simulation and the real world.

2. Key Rebuttal Updates

  • Dataset Fidelity and Visualization (DBJp, zoHu). We provided extensive visualizations, reported 1.0056 mm marker error and 95.7% contact-configuration agreement, and clarified articulated-object representations.
  • Generator Steerability and Fairness (DBJp, zoHu, JrAG). We introduced Prompt-acc (39.94% joint accuracy), conducted conditioning ablations, and provided behavior-conditioned comparisons with retrained baselines.
  • Flywheel Reproducibility and Diversity (DBJp, zoHu). Four-seed evaluation achieves 42.0±3.5% SR, compared with 12.2% for a fixed-generator control. Set-level cosine similarity decreases from 0.522 to 0.407. We also clarified generator updates and cross-round data retention.
  • Real-World Generalization (zoHu, JrAG). Additional DexBeyond-FW experiments achieve 29.8% real-world SR on unseen objects. We further clarified the distinction between feasible reference subgoals and dynamic execution.
  • Conceptual Contributions (JrAG). We clarified how DexBeyond-D, DexBeyond-G, and DexBeyond-FW address three successive challenges in scaling dexterous manipulation data, rather than merely combining existing components.

3. Manuscript Revisions

We have expanded the main text from 9 to 10 pages, with revisions highlighted in blue. Specifically:

  • Sec. 3.1 & 3.3: Clarified articulated-object representations and the generator update mechanism.
  • Sec. 4.2: Added reconstruction fidelity evaluations, including marker accuracy and human evaluation.
  • Sec. 4.3: Added Prompt-acc evaluations and conditioning ablations.
  • Sec. 4.5: Added four-seed reproducibility, fixed-generator control, generator stability analysis, and real-world generalization experiments.
  • Appendix: Expanded implementation details, evaluation protocols, and additional ablation studies.

Finally, we wish to emphasize that DexBeyond is a data-centric framework for scaling interaction and object diversity through generation and iterative feedback. Our real-world experiments evaluate the transferability and downstream utility of generated interactions, while developing a new sim-to-real transfer method is not our primary objective. We appreciate the reviewers' important questions regarding generator effectiveness, flywheel self-improvement, and experimental evaluation. We have provided additional evidence and clarifications to address these concerns and respectfully hope the AC can facilitate further discussion and consider the revised manuscript in the final assessment.

Thank you again for your time and consideration.

Best regards,

The Authors

Two corrections regarding my review.

I did not acknowledge "the practical value of our iterative flywheel". My review says generating hand-object interactions is a promising direction. The flywheel is listed under weaknesses.

I also don't think I misunderstood the scope. I never evaluated this as a general-purpose manipulation controller. My major concerns are with the generator and the flywheel.

Thank you for pointing this out. We apologize for the inaccurate attribution of your views on the flywheel and have corrected the relevant statement in our AC summary.

We also recognize that your concerns focus specifically on the generator and flywheel, and we did not intend to suggest that you misunderstood our research scope. We have revised the scope statement to clarify that developing a new sim-to-real transfer method is not our primary objective. This clarification concerns the overall positioning of our work and is not directed at your review.

Thank you again for the clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.