acceptodds
Withdrawn from ICLR 2027

RIGOR: CONSTRAINT-AWARE LAYOUT PLANNING AND GUI GROUNDING FOR INDUSTRIAL SCENES

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Constructing an industrial scene through a graphical interface requires both a coherent spatial target and actions that realize its device parameters. Accurate interaction cannot compensate for an unsuitable layout, while a suitable layout can still be misapplied to the wrong interface field. We introduce Rigor, a dataset and framework for studying these complementary requirements in static Unity workcells. The dataset links 1,050 rule-validated layouts from 291 task seeds to 222,306 GUI action units from 938 verified expert trajectories. The framework connects a supervised coordinate planner, relation-informed candidate generation, and bounded local search to a separately trained action model through explicit device-level targets. Refinement uses deterministic task feedback at inference time. Stage-wise evaluation separates layout quality from next-action correctness on frozen GUI states with shared targets. Rigor achieves 87.04 in layout quality and 79.69 in relation satisfaction, compared with 54.14 and 29.17 for the best respective baselines, and 81.14% step success under the given-target protocol. Planning ablations improve layout quality from 25.85 with coordinate supervision to 87.04 with both refinement stages. These results support explicit constraint coordination and distinguish action recognition from parameter-grounded progress in GUI-based construction.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Industrial Scene Construction, GUI Grounding, Constraint-Aware Planning, Vision-Language Models, Layout Refinement, Digital Twin

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_HmJ3OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper looks at building static industrial scenes in the Unity Editor through the GUI, and argues that this requires two separate things: a good layout, and actions that actually put the right values into the right fields. To study this, the authors build a dataset of 1,050 rule-checked layouts (291 task seeds, all from a single kitting-to-assembly workcell with 11 fixed devices), paired with about 222K action units from 938 trajectories recorded by a scripted expert.

The proposed system has two separately trained parts. A Qwen3-VL-8B planner is fine-tuned to output device coordinates, and its output is then refined at test time by a relation-based candidate generator (R) and a bounded beam search (L), both driven by a deterministic "Authority" score. The chosen layout is compiled into per-device field/value targets, and a second LoRA-tuned Qwen3-VL-8B predicts the next GUI action from the screenshot, target, and action history.

Planning and execution are evaluated separately. PLAN uses 24 questions with a soft layout score and a relation-satisfaction score. ACT uses frozen GUI states with targets taken from RIGOR's own refined layouts. The two are combined into a composite score (OCS). RIGOR reports 87.04 layout score, 79.69 relation satisfaction, 100% action-type accuracy and 81.14% step success, ahead of 13 baselines. Most of the planning gain comes from refinement: SFT alone gives 25.85.

Critical strengths
  1. The idea of evaluating layout quality and action correctness separately is, in my view, the most valuable part of the paper. The results make the point fairly convincingly: the API models get around 95% action-type accuracy but only about 70% step success, and Qwen3.5-9B has the best baseline layout score while almost never navigating fields correctly. The Position X vs. Rotation X example captures the problem well. Picking the right type of action is not the same as editing the right parameter, and I think this observation is useful beyond this particular setting.

  2. The dataset ties layout targets to GUI trajectories through shared IDs, and the validation is careful (state continuity, screenshot checks, saved-scene consistency). Keeping failed or malformed outputs in a fixed denominator instead of dropping them is also good practice. GUI agents for professional 3D software have received relatively little attention, so a resource like this could be useful if it is released.

Critical weaknesses
  1. My main concern is that the planning results may largely reflect search against the evaluation metric itself. Nearly all of the improvement (25.85 to 87.04 in layout score, 26.56 to 79.69 in relation satisfaction) comes from R and L, and both choose candidates by maximizing SAuthS_{Auth}. I could not find a definition of SAuthS_{Auth} anywhere in the paper. Appendix C mentions that layouts are ranked with "the same frozen soft-score implementation used throughout data construction," which suggests it may be close to, or the same as, the S1S_1/S2S_2 scores and relation checks used for evaluation. If so, the 87.04 mostly shows that beam search can optimize a known objective. None of the baselines get this kind of access or search budget, so the comparison in Table 2 is not really fair.

    Two other things make it hard for me to tell whether the refined layouts are actually good. First, the hard-constraint pass rate is not reported; the paper itself says Stotal,24S_{total,24} is not the fraction of layouts satisfying all hard constraints. Second, RIGOR's Structural Completeness is only 43.8, lower than most baselines (GPT-5.5 gets 97.9). Given how much Figure 1 leans on the safety angle, I expected to see whether the outputs are valid at all.

  2. The ACT comparison mostly measures how well the model fits the training distribution. The ACT model is trained on ~222K steps from the same scripted expert, in the same custom JSON action format, on the same Unity interface. The baselines are all zero-shot on this format, and GUI agents like UI-TARS and OpenCUA have to give up their native action spaces, which probably explains a lot of their near-zero numbers. Under these conditions, 100% action-type accuracy is not surprising. It is also unclear what "held-out ACT interface profiles" means; if the panel layout is the same at train and test time, the model could simply memorize where fields are. Appendix B also reports RIGOR's Field Grounding at 40.0, the same as several baselines, which does not fit well with Figure 5 or the overall claims about grounding.

    On top of this, the test sets are small: 24 PLAN questions, and the ACT set size is never stated. No variance or confidence intervals are given. There is no end-to-end run, even though connecting planning and execution is the main claim. The human study behind Figure 6 is not described, and the correlation is given as 0.9773 in the text but 0.9924 in the figure.

    Finally, the scope is narrow: one workcell template with exactly the same 11 devices. The method is essentially SFT, scorer-guided beam search, and an SFT action model, so the technical novelty is limited.

Questions for authors
  1. What exactly is SAuthS_{Auth}, and how much does it overlap with the metrics used for evaluation (S1S_1, S2S_2, relation checks)? If they overlap, could you show results with a separate scorer for refinement, or apply the same R+L search to a couple of baselines?
  2. How many of the 24 test layouts pass all hard constraints? Why is RIGOR's Structural Completeness (43.8) so low when its layout score is 87.04?
  3. How are the ACT train and test sets separated? Does the Unity panel layout change between them? How many units are in the ACT test set? Did the baselines see any examples of the action format?
  4. Which correlation in Figure 6 is correct (0.9773 or 0.9924)? Who were the human raters?
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer HmJ3 (Part 1/3)

We thank you for asking which planning gains survive when every planner can query the same evaluator, and whether ACT’s advantage extends to actions with the same targets and histories.

Q1. Authority, final scoring, and a common search budget (W1&Q1).

A1: Yes, Authority and the final PLAN metrics use the same geometric distances. PLAN predicts coordinates from the task and public asset geometry. At inference, Authority checks each proposed layout against the specified relations (Section 3.3). Following your suggestion, we gave all six initializations the same evaluator, refinement operators, and cap on evaluator calls.

(1) Search and scoring. Search ranks candidates by soft score, hard violations, or a composite. The search soft score averages six geometric components.

SearchSoft⁡=sprocess+stool+srack+scontainer+sdesk+ssign6.\operatorname{SearchSoft}=\frac{s_{\mathrm{process}}+s_{\mathrm{tool}}+s_{\mathrm{rack}}+s_{\mathrm{container}}+s_{\mathrm{desk}}+s_{\mathrm{sign}}}{6}.

SearchComposite⁡=SearchSoft⁡100−NormalizedHardViolation⁡.\operatorname{SearchComposite}=\frac{\operatorname{SearchSoft}}{100}-\operatorname{NormalizedHardViolation}.

A replacement must satisfy the output protocol, improve at least one component, preserve or increase SearchSoft, and introduce no new or larger hard violation. Final evaluation weights the process-distance score S1S_1 at 60% and the mean of the five service-placement scores S2S_2 at 40%. Both are normalized to [0,1][0,1]. Relation counts eight binary distance checks per question. All 24 questions remain in both denominators, with non-scorable layouts contributing zero (Appendix F.1).

Relation((2)∗∗Matchedrefinement.∗∗Everyinitializationusedbeamwidth4,depthatmost4,andaceilingof2,000distinctevaluationsperquestion,includingtheinitiallayout.Countsexcludethecostofgeneratingthestartinglayout.Arrowsgiveresultsbeforeandafterrefinement.∣R/LStartingPoint∣SoftLayoutScore∣Relation∣HardPass∣Eval.Calls∣∣−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−∣−−−−−−−−−−−−−−−−:∣−−−−−−−−−−−−−−−−−−:∣−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−:∣−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−:∣∣RIGOR(SFTinitialization;finalvalueonly)∣87.04∣79.69∣Base(finalvalueonly)∣98.88∣89.06∣Search−only(deterministicgrid)∣39.34→60.82∣12.50∣DeepSeek−V4.1−Flash∣40.07→54.64∣29.17∣GPT−5.5∣26.71→45.05∣22.92∣Qwen3.5−9B∣54.14→∗∗99.30∗∗∣21.88Beforesearch,coordinatesupervisionimprovestheBasemodelfrom∗∗17.25to25.85∗∗onLayout(Table3).Aftersearch,Qwen3.5−9Breaches99.30andtheuntunedBasemodelreaches98.88,bothaboveRIGOR’s87.04.DeepSeekpassesallhardconstraintson∗∗22/24∗∗questionsdespiteitslowerLayoutof54.64.ThecontrolledgainfromSFTisthereforethe8.60−pointimprovementbeforesearch.Thepost−searchscoresalsodependontheevaluator−guidedrefinement.\mathrm{Relation}(\\%)=\frac{100}{24\times8}\sum_{q=1}^{24}\sum_{r=1}^{8}h(q,r).$$ (2) **Matched refinement.** Every initialization used beam width 4, depth at most 4, and a ceiling of 2,000 distinct evaluations per question, including the initial layout. Counts exclude the cost of generating the starting layout. Arrows give results before and after refinement. | R/L Starting Point | Soft Layout Score | Relation | Hard Pass | Eval. Calls | | -------------------------------------------- | ----------------: | ------------------: | -------------------------------: | ------------------------------------------------: | | RIGOR (SFT initialization; final value only) | 87.04 | 79.69% | 0/24 | 39,036 | | Base (final value only) | 98.88 | 89.06% | 0/24 | 44,694 | | Search-only (deterministic grid) | 39.34 → 60.82 | 12.50% → 25.00% | 0/24 | 48,000 | | DeepSeek-V4.1-Flash | 40.07 → 54.64 | 29.17% → 44.27% | **22/24** | 47,942 | | GPT-5.5 | 26.71 → 45.05 | 22.92% → 39.06% | 9/24 | 47,403 | | Qwen3.5-9B | 54.14 → **99.30** | 21.88% → **90.63%** | 0/24 | 46,374 | Before search, coordinate supervision improves the Base model from **17.25 to 25.85** on Layout (Table 3). After search, Qwen3.5-9B reaches 99.30 and the untuned Base model reaches 98.88, both above RIGOR’s 87.04. DeepSeek passes all hard constraints on **22/24** questions despite its lower Layout of 54.64. The controlled gain from SFT is therefore the 8.60-point improvement before search. The post-search scores also depend on the evaluator-guided refinement.
AuthorsOpenReview ↗

Author Response to Reviewer HmJ3 (Part 2/3)

Q2. Hard-constraint passes and Structural Completeness (W1&Q2).

A2: None of the 24 original RIGOR layouts passed every hard constraint. All 24 contained all 11 devices, but none passed the parent–child attachment check. As a separate experiment, we applied a bounded rule-based repair to the attachments and positions using device bounding boxes and workspace constraints.

24 PLAN QuestionsAll Devices PresentScorableParent PassHard PassSoft Layout ScoreRelation(%)
Original RIGOR outputs (additional hard-constraint statistics)24/2421/240/240/24 (0.00%)87.0479.69%
Original outputs with additional bounded rule-based repair24/2424/2424/2424/24 (100.00%)38.9326.56%

Eq. (24) gives equal weight to device coverage and attachment. For each scorable layout,

Structural(q)=100[0.5ndevice(q)11+0.5 1(parent(q)=PASS)].\mathrm{Structural}(q)=100\left[0.5\frac{n_{\mathrm{device}}(q)}{11}+0.5\,\mathbf{1}\bigl(\mathrm{parent}(q)=\mathrm{PASS}\bigr)\right].

Each of the 21 scorable layouts earned 50 points for device coverage. The other three contributed zero. Thus (21×50+3×0)/24=43.75(21\times50+3\times0)/24=43.75, reported as 43.8 in Figure 8. GPT-5.5 scores 97.9 on the same diagnostic. RIGOR’s 87.04 Layout score measures process distances and service placement. Repair raised hard passes to 24/24 while reducing Layout to 38.93 and Relation to 26.56%. We will report these outcomes together in Section 5.3.

Q3. ACT split, interface variation, and matched baseline comparisons (W2&Q3).

A3: ACT fine-tuning used 8,000 unique actions. Table 1’s 222,306 raw actions are the recording corpus from 938 non-test trajectories. The training and test splits are disjoint by task seed, task, and layout identity.

ScopeTasks and Action Counts
Final ACT fine-tuning36 tasks, 8,000 unique action units, presented 9,600 times in total
Frozen ACT test24 seeds × 2 interface configurations × 200 actions, totaling 48 trajectories and 9,600 actions
Additional matched-sample comparison2,400 matched actions (6 seeds, 12 trajectories); a smaller subset of 600 actions is included (3 seeds, 3 trajectories)

(1) Interface and action format. Among records with explicit interface labels, training uses C1, C4, C8, and C10. Held-out test profiles are C2, C3, C5, C6, C7, C9, and C11–C16. General-purpose models receive the action vocabulary, required fields, coordinate convention, and a generic JSON example. UI-TARS and OpenCUA use native outputs with fixed scoring adapters. RIGOR receives task-specific training, and the external baselines are evaluated primarily zero-shot.

(2) Parameter-level execution. On the full frozen test, RIGOR’s 100.00% measures action-type accuracy. Its 81.14% strict Step Success also requires correct parameters and target progress. For the direct comparisons below, we matched action IDs, screenshots, targets, and history prefixes. The 600-action sample is contained in the 2,400-action sample.

Additional Matched SampleRIGOR Step SuccessStrongest Baseline on the Same SampleDifference
2,400 actions (6 seeds, 12 trajectories)1,947/2,400 (81.13%)DeepSeek-V4.1-Flash, 1,680/2,400 (70.00%)+11.13 percentage points
600 of those actions (3 seeds, 3 trajectories)491/600 (81.83%)Claude Sonnet 5, 439/600 (73.17%)+8.67 percentage points

RIGOR’s Step Success gain over DeepSeek was positive on each of the six matched seeds (9.50–13.00 percentage points). The gain over Claude ranged from 8.50 to 9.00 points across three seeds. These are observed per-seed differences. We do not estimate a population confidence interval from these samples.

AuthorsOpenReview ↗

Author Response to Reviewer HmJ3 (Part 3/3)

Q4. Figure 6 correlation and human evaluation (W2&Q4).

A4: The Pearson correlation for Figure 6 is r=0.9924r=0.9924 across 14 models. Section 5.6 retained the first-round value of 0.9773. We will update the text and caption to report the second-round result consistently.

Two doctoral students and one master’s student rated the same model outputs as the three blinded LLM raters. Each panel scored the outputs separately on the 0–5 scale. We aggregated scores by model before computing Pearson’s rr, so each point is a pair of human and LLM scores for one model. The correlation measures agreement at the model level.

Q5. The PLAN–ACT interface and sequential execution (W2).

A5: A layout can specify the correct desk coordinate while a GUI action enters it into the wrong field. RIGOR makes this distinction testable by compiling the selected layout into an explicit target for each device (Eq. 5).

T_j^{\mathrm{device}}=\bigl(\mathrm{id}_j,\lbrace(f,\nu_{j,f})\rbrace_{f\in\mathcal F_j}\bigr).$$ The dataset contains **1,050 validated layouts**. Task and layout IDs link **222,306 expert action units** from 938 verified trajectories to their source layouts. This makes it possible to check a planning error against the scene requirements and a GUI error against the specific device field the plan calls for. We also tested consecutive actions for a single device. ACT used its own action history and incrementally updated textual state, with the interface supplying the fixed device ID at each step. Success required creating the device, writing exact target values, saving, and verifying them by readback. ACT completed **22/46 tasks (47.83%) across 23 seeds**. Full 11-device scene completion remains unmeasured. **Q6. Field grounding and generalization across workcell conditions (W2).** A6: The 40.0 Field Grounding score and Figure 5 evaluate different fields. The diagnostic covers 144 clicks on Rotation Y, Scale Y, and Scale Z. RIGOR selected the right action type every time and achieved strict Step Success on none. | Result | Evaluation Target | Observation | | ---------------- | ----------------------------------------------------- | ------------------------------------------------------------ | | Field Grounding | 144 clicks targeting Rotation Y, Scale Y, and Scale Z | Correct action type: 144/144; strict Step Success: 0/144 | | Figure 5 example | Desk Position X | Predicted point [1721, 256] lies inside the target input field | By Eq. (23), $\mathrm{Field\ Grounding}=100[0.4(144/144)+0.6(0/144)]=40.0$. The desk’s Position X target in Figure 5 spans x = [1692, 1749] and y = [249, 262]. The illustrated click lands inside that field. On the three diagnostic fields, all 144 actions failed the full step criterion. The full dataset contains 97 condition groups and 291 task seeds. The 24 PLAN questions cover eight groups with three seeds each, one fixed 11-device workcell, and a Z-oriented main aisle. Within this workcell we test variation in aisle width, equipment clearance, and boundary clearance. Transfer to different workcells or device inventories remains open. We would be grateful if you would **reconsider your evaluation** in light of the common-budget planner comparisons, the explicit hard-pass counts, and the matched ACT results.
Reviewer_VUEHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents RIGOR, a framework for industrial scene layout planning and GUI execution. It combines supervised planning, constraint-based refinement, and GUI action prediction. The authors also introduce a dataset and report improvements over several baselines.

Critical strengths

1.The problem is interesting. Separating layout planning from GUI execution makes sense, and the dataset could be useful for future work. 2.The experiments cover a reasonable set of baselines, and the ablations show clear gains from the refinement stages.

Critical weaknesses
  1. I'm not fully convinced by the novelty here. The main components, including supervised planning, constraint-based search, and GUI action prediction, are mostly existing techniques. The improvement over prior work such as SceneGenAgent is not very clear.

  2. The large planning gains mainly come from test-time refinement using the task-specific evaluator. so how much improvement comes from the learned model itself? A search-only baseline with a matched compute budget is missing. Also, ACT is evaluated on frozen states with given targets and reference histories. There is no end-to-end execution evaluation, so the reported 81.14% step success does not really demonstrate reliable scene construction.

Minor Comments

1.The evaluation only covers a fixed 11-device industrial scenario and 24 planning questions. Generalization is unclear.

2.Figure 6 reports different correlation values in the figure (0.9924) and the main text (0.9773).

3.The main text is only around 8.5 pages. Some figures take up considerable space, and the overall presentation and formatting could be improved.

Questions for authors

1.Could the authors compare against a search-only baseline with the same inference budget? 2.What is the actual hard-constraint satisfaction rate? The Structural Completeness score of 43.8 seems quite low. 3.Are there any end-to-end results without ground-truth action histories?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer VUEH (Part 1/2)

We thank you for asking how RIGOR extends previous industrial scene generators and what learned planning contributes when search is held fixed.

Q1. Grounding a planned layout in the Unity interface (W1).

A1: SceneGenAgent (Xia et al., 2025) and RIGOR both plan industrial layouts and check their geometry. RIGOR additionally tests whether an agent can realize a specified device transform through the visible Unity interface. Figure 5 illustrates the distinction. The desk’s Position X is specified, yet a click just below its input box selects Rotation X. The same numeric entry would then change a rotation instead of a position.

MethodHow the Layout Is Realized in the Scene
SceneGenAgentGenerates C# code to construct industrial scenes, with iterative optimization through layout checks.
RIGORCompiles layouts into device identities, fields, and target values, from which ACT predicts the next action based on interface screenshots.

Eq. (5) specifies what the planner passes to the action model.

T_j^{\mathrm{device}}=\left(\mathrm{id}_j,\lbrace(f,\nu_{j,f})\rbrace_{f\in\mathcal{F}_j}\right).$$ Each target binds a device ID to its fields and requested values. Across the corpus of 1,050 validated layouts, task and layout IDs link 222,306 expert action units from 938 verified trajectories to their source layouts (Table 1). On 2,400 matched actions, RIGOR achieves **81.13% strict Step Success**, compared with **70.00%** for DeepSeek-V4.1-Flash, the strongest baseline on those actions. **Q2. Learned initialization and search under a common budget (W2&Q1).** A2: With the same search procedure, the deterministic grid reaches **60.82** in layout quality and the SFT initialization reaches **87.04**. The grid uses only the public workspace and device geometry and the attachment rules to construct a $3\times3$ starting layout. For all six initializations we used the same 24 questions, Authority evaluator, relation-informed refinement (R), and bounded local search (L). Beam width and maximum depth were both 4, with a cap of 2,000 distinct layout evaluations per question. Counts include the initial evaluation and exclude the cost of generating the initial layout. | R/L Starting Point | Soft Layout Score (Before → After Search) | Relation (Before → After Search) | Actual Candidate Evaluations (Including Initial Evaluation) | | ---------------------------------------------- | ----------------------------------------: | -------------------------------: | ----------------------------------------------------------: | | RIGOR SFT initialization | 25.85 → 87.04 | 26.56% → 79.69% | 39,036 | | Base (additional comparison; final value only) | 98.88 | 89.06% | 44,694 | | Search-only | 39.34 → 60.82 | 12.50% → 25.00% | 48,000 | | DeepSeek-V4.1-Flash | 40.07 → 54.64 | 29.17% → 44.27% | 47,942 | | GPT-5.5 | 26.71 → 45.05 | 22.92% → 39.06% | 47,403 | | Qwen3.5-9B | 54.14 → **99.30** | 21.88% → **90.63%** | 46,374 | Table 3 isolates the gain before search. Coordinate supervision raises layout quality from 17.25 to 25.85 (**+8.60**). R/L then adds **61.19** points. The SFT initialization reaches its final score in 39,036 evaluations, while the grid uses all 48,000. Under the same cap, Base and Qwen3.5-9B reach higher final scores of 98.88 and 99.30. The contribution of SFT is measurable before search. The larger gain in the RIGOR pipeline comes from refinement.
AuthorsOpenReview ↗

Author Response to Reviewer VUEH (Part 2/2)

Q3. Hard-constraint satisfaction and structural completeness (Q2).

A3: All 24 original layouts contain the 11 required devices, but 0/24 pass the parent–child attachment check, so none pass all hard constraints. A separate bounded repair adjusts attachments and coordinates using device bounding boxes and workspace constraints.

24 PLAN QuestionsAll Devices PresentScorableParent–Child Attachments PassedAll Hard Constraints PassedSoft Layout ScoreRelation
Original RIGOR outputs (additional hard-constraint statistics)24/2421/240/240/24 (0.00%)87.0479.69%
Original outputs with additional bounded rule-based repair24/2424/2424/2424/24 (100.00%)38.9326.56%

Eq. (24) in Appendix F.5 assigns equal weight to device presence and attachment.

Structural⁡(q)=100[0.5ndevice(q)11+0.5,1{parent⁡(q)=PASS}].\operatorname{Structural}(q)=100\left[0.5\frac{n_{\mathrm{device}}(q)}{11}+0.5\\,\mathbf{1}\lbrace\operatorname{parent}(q)=\mathrm{PASS}\rbrace\right].

Each of the 21 scorable layouts earns 50 points for device presence. The other three contribute zero. The mean is (21×50+3×0)/24=43.75(21\times50+3\times0)/24=43.75, reported as 43.8. The soft score measures process distances and service-equipment placement. Repair raises the all-constraint pass count to 24/24 while lowering the soft score from 87.04 to 38.93. We will report both measures together in Section 5.3.

Q4. Sequential execution with the model’s own action history (W2&Q3).

A4: ACT completed 22/46 single-device tasks (47.83%) across 23 seeds. At each step the interface supplied a fixed device ID, and ACT chose its next action from its own history and the updated textual state. Success required creating the device, writing exact values to the specified fields, saving the scene, and verifying those values by reading them back. The 81.14% in Table 2 is strict next-action success on frozen GUI states with recorded histories. The new experiment measures verified completion for one device. Autonomous construction of the full 11-device scene remains untested.

Q5. Variation across workcell conditions (M1).

A5: We hold the kitting-to-assembly workcell and its 11 devices fixed. The held-out PLAN questions vary aisle width, equipment clearance, and boundary clearance.

ScopeActual Coverage
Full dataset97 condition groups, 291 task seeds, and 1,050 layouts. The varying factors are aisle width, equipment clearance, boundary clearance, and main-aisle orientation.
Frozen PLAN evaluationEight condition groups, three questions per group, with all main aisles along the Z axis. Aisle widths are 1.30–1.60 m, required equipment clearances are 0.35–0.50 m, and boundary-clearance requirements are 0.40–0.60 m.

We will specify this held-out range in Sections 4.1 and 5.1. Transfer to a different workcell or device inventory remains untested.

Q6. Human–LLM correlation in Figure 6 (M2).

A6: Pearson’s r=0.9924r=0.9924 is the second-round result plotted in Figure 6. Section 5.6 retained the first-round coefficient. We will correct the text and specify in the caption that the 14 observations are paired human and LLM mean scores, one per model.

Q7. Main-text presentation (M3).

A7: We will condense the repeated interface illustrations, use the space in Section 3.3 for the search settings and scoring rule, and add the search-only comparison to Section 5.4. The main results table will identify the PLAN question and ACT action counts. We will bring the structural-completeness calculation from Appendix F.5 into the main-text discussion of hard feasibility.

We would be grateful if you would reconsider your evaluation in light of the search-only control and the single-device completion results using the model’s own action history.

Reviewer_VSbnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RIGOR for constraint-aware layout planning and GUI action prediction in industrial scenes. It combines supervised coordinate planning, relation-informed candidate generation, and bounded local search, connecting the resulting layouts to a separately trained ACT model through explicit device-level targets. The paper also introduces a paired layout–GUI dataset and demonstrates improvements in layout quality and next-action prediction through stage-wise evaluation.

Critical strengths

1. Practical relevance. The paper distinguishes spatial constraint satisfaction from precise GUI parameter manipulation in industrial scene construction. This provides a meaningful setting for studying the connection between planning and execution.

2. Valuable dataset and evaluation design. The dataset contains 1,050 rule-validated layouts and 222,306 GUI action units. The stage-wise evaluation distinguishes action selection, field grounding, and parameter entry, enabling fine-grained analysis of execution errors.

3. Supporting experimental evidence. Ablations demonstrate complementary benefits from relation-informed refinement and local search. RIGOR achieves competitive results in layout quality, relation satisfaction, and next-action success. (Tables 2 and 3)

Critical weaknesses

1. High layout scores do not sufficiently establish hard-constraint compliance.

The reported layout quality of 87.04 primarily measures soft objectives, such as process distance and service-equipment placement, rather than the percentage of layouts satisfying all hard constraints. Meanwhile, structural completeness is only 43.8 in Figure 8, substantially below several baselines. Given the metric definitions in the appendix, this discrepancy particularly warrants clarification of the number of scorable layouts and parent-relation passes. Without reporting the overall hard-constraint pass rate, the practical feasibility of the generated layouts remains unclear.

2. The contributions of the learned planner and inference-time search are not sufficiently isolated.

Table 3 shows that SFT improves layout quality from 17.25 to 25.85, while adding R+L further increases it to 87.04, indicating that most gains arise during inference-time refinement. However, the paper does not evaluate Base+R+L or alternative planners under the same scoring and search budgets. Therefore, the independent contribution of learned coordinate initialization remains unclear.

3. The effectiveness of the PLAN–ACT interface is not directly validated in execution.

ACT is evaluated on frozen GUI states with recorded histories and shared A+R+L targets. Although RIGOR achieves 81.14% step success, this protocol does not capture error accumulation or demonstrate that the agent can fully realize its own planned layouts in Unity. Moreover, the interaction success rate is only 44.74%, suggesting substantial remaining challenges for complete scene construction. (Table 2)

4. Key search and evaluation details are insufficient for reproducibility.

Although R and L are formally described, the paper does not sufficiently specify relation-informed candidate generation, search neighborhoods, actual beam width, search depth, or the complete Authority scoring implementation. In addition, the ACT evaluation-set size and operation-specific sample counts are not explicitly reported, and the relationship between the 24 PLAN evaluation questions and 112 test layouts requires clarification. These details are important for reproducing the results and assessing their statistical reliability.

Questions for authors
  1. Could the authors report the overall hard-constraint pass rate and explain the structural completeness score of 43.8? In particular, how many layouts pass the parent-relation checks?
  2. How do Base+R+L and alternative planners perform under the same Authority feedback and search budgets? This would help isolate the contribution of learned initialization.
  3. Could the authors provide a small closed-loop evaluation reporting full-scene completion when ACT sequentially executes its own actions to realize PLAN-generated layouts?
  4. Could the authors provide the concrete R/L search configurations, candidate-generation and scoring rules, and the actual sizes and composition of the PLAN and ACT evaluation sets?
2 comments
AuthorsOpenReview ↗

Author Response to Reviewer VSbn (Part 1/2)

We thank you for asking how often RIGOR produces a feasible layout, how much the planner contributes before search, and what ACT can execute from its own history.

Q1. Hard-constraint satisfaction and structural completeness (W1&Q1).

A1: All 24 original layouts contained all 11 devices, but none passed the parent–child attachment check or the complete set of hard constraints. We applied a separate bounded rule-based repair that adjusts attachments and coordinates using device bounding boxes and workspace constraints.

24 PLAN QuestionsAll Devices PresentScorableParent–Child Attachments PassedAll Hard Constraints PassedSoft Layout ScoreRelation
Original RIGOR outputs (additional hard-constraint statistics)24/2421/240/240/24 (0.00%)87.0479.69%
Original outputs with additional bounded rule-based repair24/2424/2424/2424/24 (100.00%)38.9326.56%

All 24 repaired layouts passed the encoded hard checks. Those changes reduced Layout from 87.04 to 38.93 and Relation from 79.69% to 26.56%. The soft objectives reward process distances and service-equipment placement even when a required attachment is incorrect.

Eq. (24) gives equal weight to device count and attachment. All 21 scorable original layouts earned the 50 device-count points alone. The remaining three contributed zero in the 24-question average (Appendix F.5).

Structural⁡(q)=100[0.5ndevice(q)11+0.5 1(parent⁡(q)=PASS)]\operatorname{Structural}(q)=100\left[0.5\frac{n_{\mathrm{device}}(q)}{11}+0.5\,\mathbf{1}\bigl(\operatorname{parent}(q)=\mathrm{PASS}\bigr)\right]

Structural Completeness=21×50+3×024=43.75≈43.8\text{Structural Completeness}=\frac{21\times50+3\times0}{24}=43.75\approx43.8

We will report the original and repaired hard-pass rates alongside their soft scores in Section 5.3.

Q2. Contributions of coordinate supervision and R/L refinement (W2&Q2).

A2: Supervised fine-tuning (SFT) on layout coordinates improves the initial Layout score by 8.60 points (17.25 to 25.85), and R/L adds 61.19 points (25.85 to 87.04). Relation rises from 7.81% to 26.56% and then to 79.69% (Table 3). We then applied the same deterministic evaluator (Authority), relation-informed refinement (R), and bounded local search (L) to all six starting points, with the same per-question evaluation cap.

R/L Starting PointInitial Soft Layout ScoreFinal Soft Layout ScoreInitial RelationFinal RelationFinal All-Hard-Constraint PassesActual Evaluations (Including Initial Evaluation)
RIGOR SFT planner—87.04—79.69%0/2439,036
Base—98.88—89.06%0/2444,694
Search-only39.3460.8212.50%25.00%0/2448,000
DeepSeek-V4.1-Flash40.0754.6429.17%44.27%22/2447,942
GPT-5.526.7145.0522.92%39.06%9/2447,403
Qwen3.5-9B54.1499.3021.88%90.63%0/2446,374

For the two rows with dashes, Table 3 gives the pre-search scores reported above. Starting from a deterministic 3×33\times3 grid, search reaches 60.82. It reaches 87.04 from the SFT planner, 98.88 from Base, and 99.30 from Qwen3.5-9B. Thus SFT improves the unrefined prediction, while the largest gain in the RIGOR pipeline comes from search. DeepSeek-V4.1-Flash, in turn, passes all hard constraints on 22/24 cases, compared with 0/24 for the three highest soft scores. The evaluation counts include the initial layout check and exclude the cost of generating the starting layout.

AuthorsOpenReview ↗

Author Response to Reviewer VSbn (Part 2/2)

Q3. Sequential execution of PLAN targets (W3&Q3).

A3: ACT completed 22 of 46 single-device tasks (47.83%) from 23 seeds using its own action history. The compiler binds each device to its target fields and requested values (Eq. 5). During the rollout the interface supplied the fixed device ID and updated the textual state after each action. Success requires creating the device, entering the exact values, saving, and verifying those values by reading them back.

The 81.14% in Table 2 measures individual steps on frozen screenshots with recorded histories. The new evaluation follows a single device through save and readback. We have not measured completion of an entire 11-device workcell.

Q4. Search configuration, scoring, and evaluation units (W4&Q4).

A4: R proposes coordinated moves for unsatisfied device relations. L performs bounded local beam search around the layout selected by R (Section 3.3).

ItemSetting
R candidatesTranslate devices according to the directions and target distances of unsatisfied relations; child devices move with their parent. Generate at most 128 candidates per expanded search state.
L neighborhoodTranslate along the X/Z axes by ±0.05, ±0.10, ±0.20, or ±0.40 m; also includes constrained yaw rotations, pairwise device separation, or coordinated translation.
Beam width and depthBeam width 4; maximum depth 4.
Budget and stopping rulesEvaluate at most 2,000 distinct layouts per question (including the initial evaluation), with random seed 42 and a 600 s time limit checked within the search loop. Stop when candidates are exhausted, no valid improvement remains, or the depth, evaluation, or time limit is reached.

(1) Candidate ranking and acceptance. Authority ranks candidates by soft score, hard violations, or a composite of the two. We accept a protocol-valid candidate only if it preserves or improves the soft score, improves at least one component, and neither introduces nor worsens a hard violation. Otherwise we retain the incumbent.

SearchSoft⁡=sprocess+stool+srack+scontainer+sdesk+ssign6\operatorname{SearchSoft}=\frac{s_{\mathrm{process}}+s_{\mathrm{tool}}+s_{\mathrm{rack}}+s_{\mathrm{container}}+s_{\mathrm{desk}}+s_{\mathrm{sign}}}{6}

SearchComposite⁡=SearchSoft⁡100−NormalizedHardViolation⁡\operatorname{SearchComposite}=\frac{\operatorname{SearchSoft}}{100}-\operatorname{NormalizedHardViolation}

(2) Final evaluation. Final scoring reuses the geometric distances but weights them differently. Appendix F.1 weights the process-distance score S1S_1 by 0.60 and the mean of the five placement scores S2S_2 by 0.40, both on [0,1][0,1]. Relation counts satisfied requirements out of 24×8=19224\times8=192, with h(q,r)=1h(q,r)=1 for a satisfied requirement. A non-scorable layout contributes zero to both metrics. Appendix F.4 combines them into PLANScore.

Stotal(q)=0.60 S1(q)+0.40 S2(q)S_{\mathrm{total}}(q)=0.60\,S_1(q)+0.40\,S_2(q)

Layout=10024∑q=124S~total(q)\mathrm{Layout}=\frac{100}{24}\sum_{q=1}^{24}\widetilde{S}_{\mathrm{total}}(q)

Relation(\mathrm{Relation}(\\%)=\frac{100}{192}\sum_{q=1}^{24}\sum_{r=1}^{8}h(q,r)

PLANScore⁡=0.7 Layout+0.3 Relation(\operatorname{PLANScore}=0.7\,\mathrm{Layout}+0.3\,\mathrm{Relation}(\\%)

(3) Evaluation units. The 112 test layouts in Table 1 count individual layouts, several of which can instantiate one task seed. The frozen PLAN set contains 24 questions from eight condition groups, with three seeds per group. The frozen ACT set contains 9,600 actions across 24 seeds, two interface configurations per seed, and 200 actions per configuration.

ACT Action TypeCountACT Action TypeCount
FOCUS_PROJECT_SEARCH432CLICK_TRANSFORM_PX288
TYPE_PREFAB_QUERY432INPUT_TRANSFORM_VALUE3,888
SELECT_PROJECT_RESULT432TAB_FIELD_TRANSITION3,168
DRAG_PREFAB_TO_HIERARCHY432SAVE_SCENE_ONCE48
CLICK_START_TRANSFORM_FIELD432ASSERT_SCENE_CLEAN_AFTER_SAVE48

We will specify the distance thresholds and hard-violation normalization in Section 3.3 and the appendix, and give the ACT action counts in Section 5.1.

We would be grateful if you would reconsider your evaluation in light of the matched search comparisons and the verified single-device execution results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.