acceptodds
Under review as a conference paper at ICLR 2027

InsideArt: Benchmarking Articulated 3D Generation with Usable Interiors

Abstract

Usable articulated containers require accessible interior space, not just accurate exterior geometry and joints. We introduce InsideArt, a dataset and benchmark linking articulated 3D containers to support surfaces, interior free space, ordered and alternative access paths, and language-described item arrangements. Our data-construction pipeline combines vision–language proposals, human verification, geometric constraints, and rigid-body simulation to produce over 100,000 placement scenes across 654 containers with 1,981 annotated free-space regions. The dataset supports three complementary tasks: image-conditioned articulated container generation, language-guided interior synthesis, and language-guided item placement. We evaluate interior-volume recovery and access relations beyond surface similarity, and define a placement protocol covering containment, collision freedom, stability, and orientation compliance. Reconstruction and free-space evaluations reveal that accurate surfaces can coexist with missing interior regions and incorrect access relations, highlighting the need to evaluate functionality beyond appearance and joint accuracy.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3d generation, articulated 3d

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_ySEfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces InsideArt, a dataset and benchmark for generative models of 3D articulated objects, and focuses on the usability of the generated objects’ interior spaces. The paper utilizes articulated assets from PartNet-Mobility and Artiverse, using large VLMs to generate structured descriptions for each object, with particular emphasis on potential placement positions inside these articulated objects. The annotation pipeline has two data streams: one for describing the original object and the other for preparing candidate items to be placed.

The final dataset provides 654 articulated containers with 12,441 placeable items and more than 100,000 placements. The benchmark evaluates generative models in terms of three aspects, i.e., reconstruction quality, free-space matching and access-topology correctness, and finally assesses item placement.

Critical strengths
  1. The motivation for considering the usability of generated objects’ interior spaces in 3D generation is solid, especially in articulated object generation, as these objects are intended to be operated and interacted with. The paper builds on this central motivation, providing an effective annotation pipeline and a dedicated benchmark for 3D generative models.
  2. The benchmark assesses item placement not merely by reconstruction metrics (i.e., target-region-, position-, and collision-based metrics), but also by physical stability in a simulated environment, which makes the benchmark more realistic and useful for downstream applications.
Critical weaknesses
  1. The contribution of the data processing and annotation may be limited. As illustrated in figure 5, most of the annotations can be accessed directly from the original datasets via rule-based methods. Two parts of the pipeline require VLM inference, but the tasks are also relatively simple, involving descriptions of the target object or placements. The placeable items are also acquired from open datasets. Although human verification and edits make the annotated data trustworthy and ready to use, the relatively straightforward annotation tasks and pipeline leave me uncertain about this work’s contribution beyond existing datasets such as PartNet Mobility and Artiverse.

  2. Following up on W1, the limited pool of articulated containers makes the benchmark’s coverage and reliability for downstream applications uncertain. The containers shown in Figure 3 are mostly storage units, tables, and refrigerators, yet the distribution across different container types seems uneven. Section 3.2 reports that 444 of the 654 containers are storage furniture. Although the dataset also includes kitchen appliances and Figure 8 shows oven and refrigerator examples, I remain uncertain whether its category distribution and structural diversity sufficiently support the claimed downstream applicability.

    In practice, robotic real-to-sim modeling emphasizes accurate and robust reconstruction of a wide range of interactable real-world objects, such as those in RoboCasa: ovens of various shapes and sizes, toasters with realistic interior spaces, and coffee machines that can accommodate cups. However, in the examples presented by the authors, I do not see evaluations involving a more diverse range of articulated containers, which makes me question the benchmark's reliability in specific downstream applications.

Questions for authors
  1. During stage 2 of the annotation pipeline, why is a VLM required to provide candidate support points? It seems these positions can be identified directly from the original models through collision detection (using tools like CoACD and FCL). What are the advantages of using a VLM to propose candidates over directly searching for horizontal surfaces, given that both approaches require manual verification?
  2. Is the example shown in the bottom-right panel of Figure 1 drawn from the actual annotated data? It illustrates various placement configurations, such as books placed flat or upright and nested placements involving a pen holder and the pens inside it. Are these configurations included in the dataset proposed in the paper?
  3. As mentioned previously, could the authors provide more varied examples to demonstrate the benchmark’s ability to evaluate a wider range of container geometries and placement configurations beyond cabinet scenarios? These could include evaluating the quality of generated interior spaces in appliances such as refrigerators, ovens, dishwashers, toasters, and coffee machines; assessing different placement orientations for objects such as books and plates (e.g., placed flat or on their sides).
  4. How does the benchmark address data balance and coverage across all the assets used? For example, each “container category–object–placement configuration” combination may require a minimum number of samples and sufficient diversity to ensure reliable evaluation. Could the authors provide statistics on the sample counts and proportions for these combinations, or at least explain how the benchmark handles imbalances in sample counts across container categories?
  5. Given that most articulated object generation methods cannot generate contained items themselves, is the applicability of the evaluation in Section 5.3 limited? For articulated object generation, even in robotics-related applications, models are not necessarily required to generate a container, an item, and the corresponding placement all at once. Could the item placement evaluation therefore be extended to a broader range of test scenarios to assess the quality of generated containers, rather than requiring models to output placements directly?
6 comments
AuthorsOpenReview ↗

The response to Reviewer ySEf (Part 1)

We thank the reviewer for the careful and constructive evaluation of InsideArt. Below, we clarify the substantive contributions of our annotations and benchmark, discuss container coverage and data balance, compare VLM-based support-surface proposals with geometric alternatives, verify the provenance and diversity of the illustrated placements, and explain how the benchmark evaluates generated containers.

W1 The contribution of data processing and annotation

Our annotation effort is substantial, involving extensive manual curation, correction, and multi-stage verification throughout the dataset construction process. The source datasets do not specify which surfaces can support objects, the usable volume above each surface, the parts and order of operations required for access, or the poses and metric scales at which items can be placed. Nor can this information be obtained directly through rules. InsideArt contributes three types of annotation:

1. Containers

  • 1,9811{,}981 named and manually verified storage regions, each with a metric-scale free-space volume;
  • the support surface and articulated part associated with each region;
  • complete access conditions, including ordered dependencies, such as opening a door before pulling out a drawer behind it, and alternative access paths; 64.264.2% of containers have one or two distinct opening paths;
  • a unified real-world metric scale and image-aligned multi-state observations, including RGB, part masks, and depth; and
  • corrections to erroneous interior mesh structures and rendering materials.

2. Items

  • 12,44112{,}441 items in 198198 categories and 1616 semantic groups;
  • a canonical vertical axis, front direction, stable resting pose, and real-world dimensions for every item;
  • deduplicated assets, with multi-item assets split into individual objects and candidate poses determined through gravity settling; and
  • completed geometry for assets that originally contain only surface meshes.

3. Placements

  • more than 100,000100{,}000 placement scenes in easy, medium, and hard splits;
  • item-container compatibility rules, including disallowed categories, non-stackable pairs, and food-vessel compatibility;
  • links from every item instance to its target region, supporting surface or object, and the articulated part that must be opened to reach it;
  • constraint checks and rigid-body simulation, followed by human screening, with no more than 5 cm5\,\mathrm{cm} of sliding or 20∘20^\circ of rotation between placement and rest; and
  • language annotations consistent with this structure, covering spatial relations, pose changes, and opening operations at both qualitative and metric levels.

Rules and VLMs provide only initial proposals in our pipeline. The final structure is produced through multiple human-verified rounds of proposal, correction, and re-proposal. Every articulated-container sample is cross-checked by at least two annotators, and the complete annotation process requires approximately 300300 person-hours.

The correction rate is substantial. Annotators add or remove entries in the support-surface inventories of 27.127.1% of containers and update the corresponding free-space annotations accordingly. When changes to names, location descriptions, access instructions, and associated parts are also included, 38.438.4% of containers are modified. VLM-proposed surfaces have the highest deletion rates for ovens (33.333.3%), tables (26.526.5%), and refrigerators (23.323.3%). The initial joint and access graph is generated from the URDF, VLM output, and rules, but annotators still modify 20.820.8% of containers and 15.115.1% of surface-state associations.

The source assets themselves are also not directly usable. Our interior-cavity check finds that 18.618.6% of candidate containers are solid inside, lack a floor, or have walls that are invisible from the interior. Item assets lack canonical orientations, stable poses, and metric scales, and also include duplicates and unsuitable models. We manually discard unsuitable 3D assets, redefine their front directions, correct their scales, repair incomplete meshes, and determine stable poses for all assets through gravity settling. Item-container compatibility rules are likewise not predefined. Through a generate-review-generalize-regenerate loop, annotators identify 174174 positional, oriented, and semantic problems in 303303 layout variants and convert all of them into generation constraints, such as no food in wardrobes, no cup-on-cup stacking, upright milkshakes, and no mixing of raw and cooked food on the same refrigerator shelf.

AuthorsOpenReview ↗

The response to Reviewer ySEf (Part 2)

W2 Articulated container coverage and reliability for downstream applications

We thank the reviewer for highlighting the relationship between category coverage and downstream applicability. InsideArt is designed primarily to evaluate text-and-image-conditioned 3D generation of containers with internal partitions, rather than to provide uniformly balanced coverage across all object categories. Therefore, its category distribution is determined by the relevance and complexity of internal structures instead of equal sampling. Storage furniture, such as cabinets, drawers, and shelving units, naturally exhibits greater variation in compartments and hierarchical layouts, making it especially suitable for evaluating internal spatial reasoning and conditional generation. By contrast, many other container categories have relatively simple or homogeneous interiors. Enforcing strict category balance would consequently introduce more structurally uninformative examples and dilute the benchmark’s diagnostic value. Thus, the imbalance is an intentional, task-driven design choice rather than an oversight.

Following the reviewer's suggestion, we have expanded the dataset with 301 additional containers with enclosed internal cavity from appliances (refrigerators, ovens, washers/dryers, freezers, dishwashers, toaster ovens, and others), tables (desks and coffee tables), and boxes or cases (safes, trash cans, tackle boxes, instrument cases, mailboxes, and others) to better support downstream robotic tasks. The resulting distribution is:

SupercategoryContainersProportionFine-grained categories
Storage furniture44446.5%storage cabinet; sideboard; nightstand; chest of drawers; wardrobe; TV cabinet; storage bench; storage chest; sink cabinet
Tables / desks21522.5%desk ; table with drawers or cabinet doors; coffee table
Appliances21522.5%Refrigerator; oven; microwave; washer/dryer; freezer; toaster oven; dishwasher; air fryer; bread maker ; grill press; rice cooker
Boxes / cases818.5%box; safe; trash can; tackle box; instrument case; mailbox; storage case; toolbox; sewing box
Total955100%32 fine-grained categories

Moreover, to our knowledge, InsideArt is the first benchmark to explicitly model the correspondence between internal regions and external articulated parts. For evaluating the generation and usability of container interiors, even though the difference of shape and materials is significant, the diversity of internal structures and access topologies is our target: how the interior is partitioned, which movable parts are associated with each region, and which sequence of operations is required to access it. Containers in the same semantic category can still differ substantially, including independent drawers, one compartment shared by two doors, and nested configurations in which a door must be opened before a drawer can be pulled out. These differences directly determine the difficulty of internal-structure generation and access-relation reasoning.

We quantify this dimension for both InsideArt and RoboCasa. We represent each container as a labeled access graph: its nodes are internal regions and access parts, while its edges specify which part must be opened to reach each region. Opening directions and required operation orders are also recorded. Two containers count as the same structural type only when their access graphs are isomorphic. We apply the same construction to RoboCasa: the five shared enclosed-container categories below contain 200 RoboCasa fixtures.

CategoryInsideArt containersInsideArt structural typesRoboCasa containersRoboCasa structural types
Refrigerator94625021
Oven6554253
Microwave187501
Toaster oven555013
Dishwasher55254
Storage furniture / tables / boxes and remaining appliances768276——
All95539437543

Across the five categories represented in both datasets, InsideArt obtains approximately three times as many structural types with fewer containers: 133 types from 187 containers, compared with 42 types from 200 RoboCasa fixtures. RoboCasa containers are often appearance variants of a shared template: its 50 microwaves contain only one structural type, and its 25 ovens contain three. Only 5 of RoboCasa's 43 structural types also occur in InsideArt; the remaining types mainly arise from categories such as stoves and sinks, which are not containers intended to hold placed objects. These results show that, relative to RoboCasa, InsideArt provides substantially greater diversity in internal structures and articulated access topologies. Thus, InsideArt includes wider range of interactable real-world objects.

AuthorsOpenReview ↗

The response to Reviewer ySEf (Part 3)

Q1 Horizontal surfaces with geometry method (CoACD or FCL)

The most important limitation is that geometry provides neither semantic names nor access structures. We therefore use a VLM to obtain semantic information for every articulated part and free space and to express their positional relationships in language, allowing a one-to-one correspondence between the textual information and the physical 3D structure. This information is required by the subsequent placement stage.

To compare the geometric and VLM method, we further add a purely geometric baseline and compare it with the original VLM method. A surface is considered found when its height differs by at most 44 cm and its top-view IoU is at least 0.250.25. We evaluate three approaches:

  1. voxel support-surface extractor for generated assets, applied directly to the ground-truth URDFs
  2. per-part extraction of near-horizontal mesh faces with overhead clearance in the open pose
  3. a CoACD plus ray-casting pilot.
Geometric methodPrecisionRecallF1
Voxel support-surface extractor0.7940.9280.855
Open-pose per-part horizontal-face extraction0.6290.8600.726
CoACD + ray casting0.7140.1160.200

If the container is not partitioned into parts, for example when an entire drawer is treated as a single mesh, CoACD's convex decomposition fills enclosed cavities and finds almost no support surfaces, with a recall of 0.120.12. Conversely, if the mesh is partitioned, a complete support surface may be split across multiple subparts. Recovering the original support region then requires correctly identifying and merging these local surface patches, which is difficult.

The best geometric method finds most support surfaces, with a recall of 0.930.93, but its precision is only 0.790.79. This is comparable to the 27.127.1% human intervention rate for VLM-proposed geometry, indicating that VLM and geometric methods require similar amounts of subsequent manual annotation at the support-surface proposal stage. The main problem is that geometry has difficulty distinguishing a nearly horizontal face from a genuinely usable support surface: it mistakes gaps between layers, rails, and handle tops for supports, while missing valid storage regions with absent floors, uneven surfaces, or open tops.

Q2 Is the bottom-right example in Figure 1, including flat and upright books and pens inside a pen holder, drawn from actual annotated data?

The bottom-right example in Figure 1 comes from real annotated data. It is a formal rendering of an annotated scene, with materials recolored consistently by object type for visualization. Flat and upright books, as well as pen-holder-like nested arrangements, are all placements actually generated in the dataset.

Alternative placement poses are annotated as item properties during item annotation, Stage 4 of Figure 5. We group items according to their three axis lengths:

  1. Items with three similar axis lengths, such as apples, grapes, and potatoes, retain one pose.
  2. Items with two similar axes and one distinct axis, such as pens, plates, and bottles, retain one or two semantically plausible and gravity-stable poses. Pens and plates are placed flat, while bottles may be horizontal or upright.
  3. Items with three distinct axis lengths, such as books, retain one to three plausible poses.

During layout generation, we sample from the selected item's candidate poses.

We show object stacking and nested placements, such as items supported by vessels, in our response to Q3.

Q3 Can the paper show more container types and placement orientations?

We have added 18 annotated placement examples to Appendix L Fig. 16, organized around three aspects. First, they cover more container categories, including refrigerators, ovens, microwaves, dishwashers, toaster ovens, washing machines, safes, and large and small boxes. Second, they show diverse item orientations, such as plates standing on edge in a dish rack, books standing upright in a row, and books lying flat in a drawer. Third, they include stacked, nested, and densely packed arrangements, such as upside-down bowls placed between dishwasher-rack tines, piles of flat items, food supported by plates, flat book stacks, densely packed drawers, and pens inserted into a pen holder. These are generated and annotated dataset scenes rather than schematic illustrations.

Reviewer_izaoOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work presents InsideArt, a dataset and benchmark for usable articulated contianers, linking the containers to support surfaces, interior free space and access paths, and language-described item arrangements. The paper proposes a data-annotaion pipeline on existing 3D assets from previous papers, and deliver over 100K places across 654 containers with 1981 annotated free-space regions.

Critical strengths
  1. Extend existing dataset to evaluting interior structure and placements is going to have a positive impact on robot training and evaluation in simulations.

  2. The dataset contains rich annotaions about a large number of containers and items. Though it is a repurpose of existing 3D assets from existing work.

Critical weaknesses
  1. Annotation pipeline requires human in the loop, raising question to how scalable it is beyond the current dataset. How much time does it require from human annotation? What is the intervention rate?

  2. Practical usability. The reviewer believes the core use case of interactable 3D assets is in robotic simulation. How do the interior annotations and placements connnect to actual usefulness. If an object is placed stably in a container in InsideArt's evaluation, does that imply that the object is reliably retrievable by a robot? Although the paper includes robot manipulation demonstrations, it lacks systematic quantitative functional validation to establish how well the benchmark metrics reflect usability in opening and retrieval tasks.

Questions for authors

Appendix J demonstrates InsideArt scenes in MuJoCo. What additional conversion or manual setup would be required to integrate the assets and their annotated placements into existing robotics simulation evaluations, such as RoboCasa or RoboTwin? Will the release include tools or documentation supporting this integration?

4 comments
AuthorsOpenReview ↗

The response to Reviewer izao (Part 1)

We thank the reviewer for the careful and constructive evaluation of InsideArt. Below, we clarify annotation effort and scalability, provide full-benchmark geometric and kinematic retrieval analyses, and describe automatic integration with robotics simulators.

W1 The annotation pipeline requires human intervention. Is it scalable, how much annotation time does it require, and what is the intervention rate?

Human effort. Ten annotators participate in the complete annotation process. Approximately three annotators work on articulated-container annotations and pipeline modifications, three annotators filter and annotate item assets, and the remainder summarize and iterate placement rules and perform final data screening. The total effort is approximately 300300 person-hours.

For articulated containers, annotators check each interior space, rendering, open and closed state, metric scale, and load-bearing capacity. They also verify that semantics match locations, that each free space is accurate and associated with the correct joint, and that its graph representation is correct. Every sample is cross-checked by at least two annotators. Item preparation involves removing unsuitable assets, annotating orientations and poses, adjusting scale, assigning mass, and settling each item under gravity on a virtual horizontal plane so that it can be inserted directly into a container. Placement annotation focuses on iterating semantic and geometric rules, refining the gravity-simulation rules, and screening the benchmark layouts for plausibility.

Intervention rate. Annotators manually add to or remove from the support-surface inventory for 27.127.1% of containers and update the free space associated with every added or removed surface. Among retained proposals, edits to the name, location description, access instruction, or associated part affect 43.343.3% of containers. VLM-proposed surfaces have the highest deletion rates for ovens (33.333.3%), tables (26.526.5%), and refrigerators (23.323.3%), and lower rates for storage furniture (10.310.3%) and microwaves (6.76.7%).

AuthorsOpenReview ↗

The response to Reviewer izao (Part 2)

W2 Systematic quantitative validation of opening and retrieval.

To better reflect usability in retrieval tasks, we directly conduct two levels of systematic validation over the full benchmark: geometric retrievability checks whether a collision-free exit path exists, while robot kinematic retrievability further tests whether a Franka Panda with the specified end effectors can grasp and extract the item.

(1) Is the item geometrically retrievable? At the annotated open configuration of the container, we keep all other items fixed. If the target is at the bottom of a stack, we move the target together with all items above it as a single group and search for a collision-free extraction path. “Axial straight” permits translation only vertically upward or along one of the four horizontal axes. “Any-direction straight” permits straight-line translation in any direction in the upper hemisphere. “Container-only straight” is a relaxed control that removes all other items and treats only the container as an obstacle. If no straight-line path exists, the GRR evaluation further invokes an SE(3)SE(3) search that permits both translation and rotation.

DifficultyScenesItemsAxial straightAny-direction straightContainer-only straightGRR
Easy3,2704,65271.2%96.4%97.0%99.1%
Medium3,25612,50061.9%93.4%96.9%99.0%
Hard3,24524,55359.6%88.3%94.1%97.1%
All9,77141,70561.6%90.7%95.3%97.9%

The results show that most layouts are not completely sealed by surrounding items or container geometry: 90.7%90.7\% of items can be extracted along some straight direction in the upper hemisphere, and retrievability rises to 97.9%97.9\% with a full SE(3)SE(3) search. By contrast, only 61.6%61.6\% can be removed along the vertical or horizontal axes, indicating that many layouts do contain an exit but require oblique motion or a change in orientation during extraction; this is especially common for items at the bottom of a stack.

(2) Is the item kinematically retrievable by a robot? At the annotated open configuration, we test all items using a Franka Panda mounted on a mobile, height-adjustable base, with either a parallel-jaw gripper or a long-reach suction tool. With the base outside the container on the extraction side, we evaluate 60 base poses spanning different distances, lateral offsets, and heights. For each pose, we check IK feasibility and FCL mesh collisions at the grasp and at 1-cm intervals along the item-carrying extraction path. An item is considered retrievable if any candidate passes all checks. Otherwise, its failure reason is assigned according to the furthest stage reached by any candidate. This test uses the relaxed setting that removes other items and treats only the container as an obstacle. “Suction-retrievable” denotes items that cannot be retrieved by the gripper but can be retrieved by the suction tool.

DifficultyItemsNo straight exitNo valid graspOut of reachCollision at graspFailure during extractionGripper-retrievableSuction-retrievableTotal retrievable
Easy4,6523.0%19.0%0.7%32.5%9.4%16.0%19.4%35.4%
Medium12,5003.1%18.1%0.8%33.1%10.2%16.7%18.1%34.8%
Hard24,5535.9%14.0%0.8%26.6%10.7%15.0%26.8%41.9%
All41,7054.7%15.8%0.8%29.2%10.4%15.6%23.4%39.1%

At object level, 39.1% of objects can be acquired successfully. Besides, at the scene level, 80.2% of hard scenes contain at least one kinematically retrievable item, while all items are retrievable in 7.5% of hard scenes. Thus, most scenes contain at least one retrievable item. The dominant failure mode is collision at the grasp pose rather than the robot being unable to reach the item and we visualize these cases in Appendix N.

Although some items cannot be retrieved under the current Franka Panda and end-effector configuration, we keep them because retrieval difficulty caused by the complexity and diversity of real-world container contents. To facilitate downstream use, we provide per-item retrievability annotations for every item placed in a container, recording whether it can be retrieved by the Franka Panda and, if not, the specific failure reason. Downstream tasks can therefore select only items labeled as retrievable when conducting manipulation experiments.

AuthorsOpenReview ↗

The response to Reviewer izao (Part 3)

Q1 What additional conversion or manual setup is required to integrate InsideArt with robotics simulators such as RoboCasa or RoboTwin? Will the tools and documentation be released?

Conversion. We implement a fully automatic converter from an InsideArt scene (the container URDF and part meshes, item poses, and free-space boxes) to a native RoboCasa fixture in MJCF. The output includes textured visual meshes, CoACD convex-decomposition collision geometries, and joints. Revolute joints become MuJoCo hinge joints (radians) and prismatic joints become slide joints (metres, after the container's metric scale is applied), each with its annotated range [qclose,qopen][q_{\mathrm{close}}, q_{\mathrm{open}}], clipped to the free swept opening where our kinematic check finds an obstruction. Free-space boxes become RoboCasa placement regions named reg_* and are attached to the moving part associated with the corresponding support surface. Each item is converted into a RoboCasa-style object and placed at its annotated pose.

Validation. We convert six scenes from five categories: cabinet, oven, microwave, refrigerator, and nightstand. The conversion requires no per-scene manual steps, and all scenes compile in MuJoCo. After 33 seconds of settling, all items remain inside their regions. We then drive every joint first to its closed state and then to its upper limit. All joints reach their targets without numerical instability.

Manual setup. Per-joint simulator settings, including effective opening limits, collision-exclusion pairs, and opening order, are derived automatically from our existing kinematic checks. No manual setting is needed for 97.097.0% of access joints. The remaining 4848 joints are blocked by the container mesh or contain errors in the source URDF, and the converter explicitly flags them. We only need to implement the converter and choose global defaults, reusing RoboCasa's native fixture parameters whenever possible.

Inside RoboCasa kitchens. Insertion uses only RoboCasa's public configuration, with no change to its code. A converted appliance replaces the kitchen's own one through update_fxtr_cfg_dict, and further assets are added as an extra fixture group in the kitchen layout, either at an absolute pose or on a counter chosen by RoboCasa's own placement sampler. The kitchen shown in the appendix (replaced oven, box on the counter, table against the wall) is built as follows:

env = robosuite.make("PickPlaceCounterToOven",
  robots="PandaOmron", layout_ids=[12], style_ids=[11],
  update_fxtr_cfg_dict={"oven": {
    "xml": "insideart/102018_oven",
    "size": [0.75, 0.60, 0.68]}})          # replace the oven

# extra layout group "insideart":
{"name": "insideart_box",                  # box on a counter
 "xml": "insideart/102379_generic",
 "placement": {"fixture": "counter_1_main_group",
               "size": [1.0, 0.6]}}
{"name": "insideart_table",                # table at a fixed pose
 "xml": "insideart/20411_generic",
 "pos": [4.93, -6.00, 0.34], "z_rot": -1.57}

RoboCasa's own task code then opens our oven door and samples its baking tray into our exported region reg_rack0 (see the appendix K).

Release. We will release the converter, documentation, and converted example scenes with the dataset.

Thank you for addressing my questions and concerns. I do not have any more questions.

Reviewer_1rDROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents a dataset and benchmark for automated synthesis of valid sim-usable placements of assets inside containers (desks, drawers, etc.) – by carefully annotating and modeling space available, placement surfaces, object placement grammar, access paths etc. The paper includes an annotation pipeline to synthesize various placement scenarios and benchmarks existing techniques on this task of interior object placement.

Critical strengths
  • I think this is a novel setting in articulated 3D asset generation, and the paper does a good job at clearly articulating various aspects of procedurally synthesizing such interior object placements.
  • The proposed dataset and benchmark could be an useful asset for the community to improve methods and quantify performance, especially with growing interest in real2sim pipelines.
Critical weaknesses
  • It is unclear how the placement synthesis procedure handles tightly constrained configurations. This is especially important to generate scenarios with clutter.
    • Stage 6 description doesn’t detail how the procedure fairs with more constraints and items placed inside (does the sampling time increase? Does the method encourage higher diversity in placement poses?) The paper should provide more analysis on the computational costs, diversity, and scalability as the number of objects and constraints increases.
    • While the paper does factor some object-on-object placement it is not clear if it handles highly constrained arrangements, like typical real-world drawers, well enough.
    • In Stage 4, the authors mention annotating a single canonical axis and resting pose – but for objects there might be several stable resting poses that arise from symmetries or static supports from other objects in the container e.g. a book can be rested upright or facedown. Does the sampling procedure explore these alternatives or does it just prefer the specified canonical pose?
  • I’m not convinced that the proposed metrics adequately capture relevant axes for downstream usability in simulation. Procgen methods should ideally capture diversity, fidelity, feasibility, and complexity. While the metrics capture geometric correctness they probably don’t test functional usability for manipulation interactions.
    • In particular, can the authors clarify if the annotated free-space and access path provide sufficient margins for manipulability/retrieving objects.
    • It would be useful to quantify some notion of retrievability of objects, perhaps even just kinematic feasibility for a specified downstream manipulator (like a Franka arm)
Questions for authors
  1. Can the authors quantify the annotator workloads? Have there been efforts to minimize this?
    1. How often do annotators have to carefully refine VLM proposed suggestions in articulation/surface/language annotations?
  2. Could the reliance on VLM generated proposals introduce annotation bias, particularly if human annotators primarily verify proposals rather than carefully identify additional/diverse annotations.
  3. For the rule base annotation synthesis, could the authors suggest how these rules were declared/iterated on and also mention any shortcomings?
  4. For the GPT-6-Astra test, was this just a zero-shot evaluation without any feedback to improve placement etc.,? Could the authors include an evaluation with iterative feedback to see if this improves some shortcomings?
  5. Could the authors report computational / monetary cost statistics for the synthesis pipeline?
6 comments
AuthorsOpenReview ↗

The response to Reviewer 1rDR (Part 1)

We thank the reviewer for the careful and constructive evaluation of InsideArt. Below, we provide additional analyses and experiments on scalability, diversity, cluttered layouts, robot retrievability, annotation, rule design, iterative feedback, and computational cost.

W1.1 How do sampling time, diversity, and scalability change as the numbers of constraints and objects increase?

We discuss the effects of increasing the number of constraints and the number of objects separately, beginning with constraints.

Constraints specify disallowed combinations during placement. They are used either to reject a sampled combination or to remove impossible combinations before sampling. Examples include the following:

  • Semantic constraints: (1) refrigerators may contain only food. (2) raw and cooked food cannot be placed together on the same support surface.
  • Geometric constraints: (1) when vessels are stacked, the lower vessel must have a larger support area than the upper vessel. (2) a vessel may contain only food, and the food itself must not cover the vessel.

Adding a constraint directly changes the sampling distribution by setting the probability of an invalid object or combination to zero. It therefore does not increase sampling time. Constraints necessarily reduce diversity by excluding some arrangements, but the excluded arrangements are primarily unrealistic ones, so this reduction has little effect on the utility of the dataset.

For an increasing number of objects, we observe that the table below reports the generation time (item selection, placement, and gravity checking) of all scenes, grouped by the number of items per scene:

Items1234567891011121314$\geq$15
Median time (s)0.380.641.191.461.852.332.482.823.403.843.594.804.254.304.97

Generation time grows approximately linearly with the number of items, but the absolute cost is small. Over the main range of 11 to 1414 items, the median generation time rises from 0.380.38 to 4.304.30. The scenes with 1515 or more items have a median of 4.974.97, close to the 4.304.30 at 1414 items, so generation does not suddenly accelerate. Moreover, more than 9999% of the scattered-layout scenes in the current data contain at most 1515 items, so these statistics cover the vast majority of typical scenes, and the additional generation cost of more items is acceptable overall.

Increasing the number of objects does not noticeably reduce diversity. The medium split, with 3.803.80 objects per scene on average, and the hard split, with 7.307.30, both cover 198198 item categories. Their normalized category entropies (the Shannon entropy of item-category frequencies divided by its maximum possible value where a larger value indicates a more uniform distribution) are 0.970.97 and 0.920.92, respectively, indicating that both splits maintain broad category coverage with relatively balanced distributions. Across different variants of the same free space, 97.797.7% and 94.994.9% of scenes have unique category combinations. This reflects low compositional redundancy.

Object positions cover all cells of a 10×1010 \times 10 grid. Layout modes include scattered, row, grid, shelf-style, and stacked arrangements. The corresponding occupancy heatmaps are reported below. For each free space, we normalize its support surface and divide it into a 10×1010 \times 10 grid. A cell is marked as occupied when an object's horizontal footprint overlaps the cell by more than 55% of the cell width along both axes. Each entry is the percentage of scenes in which that cell is occupied. Rows run from the back wall to the opening, and columns run from left to right.

Medium split

Depth \\backslash width0.00.10.20.30.40.50.60.70.80.9
0.920303131303131312818
0.830444747464747464329
0.734505355535252534933
0.634525758565555565034
0.534515758575756565034
0.434515757565656565034
0.335525757575657565135
0.234505455545453544934
0.129444848474747464330
0.018283233323232312819
AuthorsOpenReview ↗

The response to Reviewer 1rDR (Part 2)

W1.1 (continued): Sampling time, diversity, and scalability

Hard split

Depth \\backslash width0.00.10.20.30.40.50.60.70.80.9
0.927394342424242413827
0.842586362616161595639
0.748657070706968676143
0.649677174747472696345
0.549677073747573716646
0.450677375757573716646
0.350667273737372706546
0.249657070717069686345
0.142586262626161605539
0.028394244444343413725

The two tables show that, as the number of objects increases, occupancy becomes somewhat more concentrated near the center, without a substantial loss of spatial diversity. The ratio between the most- and least-occupied cells is 3.1×3.1\times for the medium split (57.657.6% vs. 18.418.4%) and 3.0×3.0\times for the hard split (75.075.0% vs. 25.325.3%). Thus, although the number of items approximately doubles, the relative degree of spatial non-uniformity remains almost unchanged.

Overall, increasing the numbers of constraints and objects does not create a scalability bottleneck in our pipeline. We have converted the above analysis and tables into line plots and heatmaps in Appendix M.

W1.2 Can the procedure handle densely cluttered arrangements such as real drawers?

In terms of placement complexity, InsideArt goes beyond other benchmarks (e.g. RoboCasa): whereas RoboCasa typically samples objects into predefined regions, we explicitly model multiple coexisting objects, inter-object support, and diverse poses and orientations. In the hard split, 28.3%28.3\% of objects are placed on top of other objects. Same-category objects can be stacked up to 1212 layers, and the data also include mixed-category stacks of pens, paper, and books. When multiple categories are arranged side by side, the footprint fill ratio can reach 0.700.70 to 0.750.75. For objects such as books and notebooks, we additionally sample from multiple stable poses and different in-plane orientations. InsideArt therefore covers more complex layouts involving multi-object stacking, densely packed side-by-side placement, and diverse orientations. However, because all objects are rigid bodies, we currently cannot represent effects such as bending paper or interleaved arrangements in which objects mutually support one another. We will further emphasize both this advantage and this limitation in the main paper.

W1.3 Does sampling explore alternative stable poses?

We jointly sample an object's identity and its candidate poses. During item annotation, objects are divided into three groups according to their dimensions:

  1. When the lengths of all three axes are similar, as for an apple, a grape, or a potato, we retain only 1 candidate pose.
  2. When two axes have similar lengths and the third differs, as for a pen, plate, or bottle, we retain 1 or 2 candidate poses according to semantic plausibility and stability under gravity. For example, pens and plates are placed flat, whereas a bottle may be either upright or horizontal.
  3. When all three axis lengths differ, as for a book, we retain 1 to 3 semantically and physically plausible poses.

The candidate-pose set is stored as an item property and is produced in Stage 4 of Figure 5. During layout sampling, we simply sample from the selected item's candidate poses.

AuthorsOpenReview ↗

The response to Reviewer 1rDR (Part 3)

W2 Do the annotated free spaces and access paths provide sufficient manipulation margins, and can retrievability be quantified for a downstream robot?

Whether an item can be retrieved from a container depends on its geometry, the grasping method, robot reachability, and collision constraints. To make our dataset more useful for downstream tasks, we do not infer manipulability indirectly from a single margin. Instead, we directly conduct two levels of systematic validation over the full benchmark: geometric retrievability checks whether a collision-free exit path exists, while robot kinematic retrievability further tests whether a Franka Panda with the specified end effectors can grasp and extract the item.

(1) Is the item geometrically retrievable? At the annotated open configuration of the container, we keep all other items fixed. If the target is at the bottom of a stack, we move the target together with all items above it as a single group and search for a collision-free extraction path. “Axial straight” permits translation only vertically upward or along one of the four horizontal axes. “Any-direction straight” permits straight-line translation in any direction in the upper hemisphere. “Container-only straight” is a relaxed control that removes all other items and treats only the container as an obstacle. If no straight-line path exists, the GRR evaluation further invokes an SE(3)SE(3) search that permits both translation and rotation.

DifficultyScenesItemsAxial straightAny-direction straightContainer-only straightGRR
Easy3,2704,65271.2%96.4%97.0%99.1%
Medium3,25612,50061.9%93.4%96.9%99.0%
Hard3,24524,55359.6%88.3%94.1%97.1%
All9,77141,70561.6%90.7%95.3%97.9%

The results show that most layouts are not completely sealed by surrounding items or container geometry: 90.7%90.7\% of items can be extracted along some straight direction in the upper hemisphere, and retrievability rises to 97.9%97.9\% with a full SE(3)SE(3) search. By contrast, only 61.6%61.6\% can be removed along the vertical or horizontal axes, indicating that many layouts do contain an exit but require oblique motion or a change in orientation during extraction; this is especially common for items at the bottom of a stack.

(2) Is the item kinematically retrievable by a robot? At the annotated open configuration, we test all items using a Franka Panda mounted on a mobile, height-adjustable base, with either a parallel-jaw gripper or a long-reach suction tool. With the base outside the container on the extraction side, we evaluate 60 base poses spanning different distances, lateral offsets, and heights. For each pose, we check IK feasibility and FCL mesh collisions at the grasp and at 1-cm intervals along the item-carrying extraction path. An item is considered retrievable if any candidate passes all checks. Otherwise, its failure reason is assigned according to the furthest stage reached by any candidate. This test uses the relaxed setting that removes other items and treats only the container as an obstacle. “Suction-retrievable” denotes items that cannot be retrieved by the gripper but can be retrieved by the suction tool.

DifficultyItemsNo straight exitNo valid graspOut of reachCollision at graspFailure during extractionGripper-retrievableSuction-retrievableTotal retrievable
Easy4,6523.0%19.0%0.7%32.5%9.4%16.0%19.4%35.4%
Medium12,5003.1%18.1%0.8%33.1%10.2%16.7%18.1%34.8%
Hard24,5535.9%14.0%0.8%26.6%10.7%15.0%26.8%41.9%
All41,7054.7%15.8%0.8%29.2%10.4%15.6%23.4%39.1%

At object level, 39.1% of objects can be acquired successfully. Besides, at the scene level, 80.2% of hard scenes contain at least one kinematically retrievable item, while all items are retrievable in 7.5% of hard scenes. Thus, most scenes contain at least one retrievable item. The dominant failure mode is collision at the grasp pose rather than the robot being unable to reach the item and we visualize these cases in Appendix N.

Although some items cannot be retrieved under the current Franka Panda and end-effector configuration, we keep them because retrieval difficulty caused by the complexity and diversity of real-world container contents. To facilitate downstream use, we provide per-item retrievability annotations for every item placed in a container, recording whether it can be retrieved by the Franka Panda and, if not, the specific failure reason. Downstream tasks can therefore select only items labeled as retrievable when conducting manipulation experiments.

Comments on the paper

1 comment

We thank the reviewers for their insightful comments, constructive feedback, and recognition of the strengths of our work:

  • Novel and Important Problem: Reviewers recognized the significance of evaluating interior usability in articulated 3D generation, describing it as a "novel setting" (1rDR) with "solid" motivation (ySEf).
  • Valuable Dataset and Benchmark: Reviewers appreciated the "rich annotations" (izao), "effective annotation pipeline" (ySEf), and systematic modeling of interior structures and placements (1rDR).
  • Meaningful Evaluation and Downstream Impact: Reviewers highlighted our physically grounded evaluation (ySEf) and its potential value for real-to-sim pipelines and robotic applications (1rDR, izao).

We summarize the major updates and additional experiments provided in response to the reviewers:

  • Annotation Quality and Efficiency (1rDR-Q1,Q2; izao-W1; ySEf-W1): Clarified annotation workload, human intervention rates, quality control, and iterative rule refinement.
  • Dataset Coverage and Diversity (ySEf-W2,Q3,Q4): Expanded container categories, quantified structural diversity against RoboCasa, and added diverse placement examples and data-balance analyses (Appendices L, O).
  • Geometric Baselines (ySEf-Q1): Added comparisons with geometric support-surface extraction methods and clarified the advantages of semantic annotations.
  • Placement Complexity and Scalability (1rDR-W1; ySEf-Q2,Q3): Analyzed sampling efficiency, spatial diversity, cluttered arrangements, and alternative stable poses (Appendix M).
  • Robotic Usability (1rDR-W2; izao-W2,Q1): Added geometric and robot kinematic retrievability evaluations, failure analysis, and automatic RoboCasa integration (Section 5.4 and Appendices K, N).
  • Extended Benchmark Evaluation (ySEf-Q5): Introduced a ground-truth layout insertion test to evaluate generated container interiors independently of item generation.
  • Iterative Refinement and Computational Costs (1rDR-Q3–Q5): Evaluated feedback-based placement refinement and clarified rule limitations and computational requirements.

We appreciate the reviewers' constructive feedback. We are encouraged that ySEf raised their rating to Weak Accept and 1rDR / izao maintained a positive assessment. We thank all reviewers again for their valuable suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.