- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper looks at building static industrial scenes in the Unity Editor through the GUI, and argues that this requires two separate things: a good layout, and actions that actually put the right values into the right fields. To study this, the authors build a dataset of 1,050 rule-checked layouts (291 task seeds, all from a single kitting-to-assembly workcell with 11 fixed devices), paired with about 222K action units from 938 trajectories recorded by a scripted expert.
The proposed system has two separately trained parts. A Qwen3-VL-8B planner is fine-tuned to output device coordinates, and its output is then refined at test time by a relation-based candidate generator (R) and a bounded beam search (L), both driven by a deterministic "Authority" score. The chosen layout is compiled into per-device field/value targets, and a second LoRA-tuned Qwen3-VL-8B predicts the next GUI action from the screenshot, target, and action history.
Planning and execution are evaluated separately. PLAN uses 24 questions with a soft layout score and a relation-satisfaction score. ACT uses frozen GUI states with targets taken from RIGOR's own refined layouts. The two are combined into a composite score (OCS). RIGOR reports 87.04 layout score, 79.69 relation satisfaction, 100% action-type accuracy and 81.14% step success, ahead of 13 baselines. Most of the planning gain comes from refinement: SFT alone gives 25.85.
Critical strengths
-
The idea of evaluating layout quality and action correctness separately is, in my view, the most valuable part of the paper. The results make the point fairly convincingly: the API models get around 95% action-type accuracy but only about 70% step success, and Qwen3.5-9B has the best baseline layout score while almost never navigating fields correctly. The Position X vs. Rotation X example captures the problem well. Picking the right type of action is not the same as editing the right parameter, and I think this observation is useful beyond this particular setting.
-
The dataset ties layout targets to GUI trajectories through shared IDs, and the validation is careful (state continuity, screenshot checks, saved-scene consistency). Keeping failed or malformed outputs in a fixed denominator instead of dropping them is also good practice. GUI agents for professional 3D software have received relatively little attention, so a resource like this could be useful if it is released.
Critical weaknesses
-
My main concern is that the planning results may largely reflect search against the evaluation metric itself. Nearly all of the improvement (25.85 to 87.04 in layout score, 26.56 to 79.69 in relation satisfaction) comes from R and L, and both choose candidates by maximizing . I could not find a definition of anywhere in the paper. Appendix C mentions that layouts are ranked with "the same frozen soft-score implementation used throughout data construction," which suggests it may be close to, or the same as, the / scores and relation checks used for evaluation. If so, the 87.04 mostly shows that beam search can optimize a known objective. None of the baselines get this kind of access or search budget, so the comparison in Table 2 is not really fair.
Two other things make it hard for me to tell whether the refined layouts are actually good. First, the hard-constraint pass rate is not reported; the paper itself says is not the fraction of layouts satisfying all hard constraints. Second, RIGOR's Structural Completeness is only 43.8, lower than most baselines (GPT-5.5 gets 97.9). Given how much Figure 1 leans on the safety angle, I expected to see whether the outputs are valid at all.
-
The ACT comparison mostly measures how well the model fits the training distribution. The ACT model is trained on ~222K steps from the same scripted expert, in the same custom JSON action format, on the same Unity interface. The baselines are all zero-shot on this format, and GUI agents like UI-TARS and OpenCUA have to give up their native action spaces, which probably explains a lot of their near-zero numbers. Under these conditions, 100% action-type accuracy is not surprising. It is also unclear what "held-out ACT interface profiles" means; if the panel layout is the same at train and test time, the model could simply memorize where fields are. Appendix B also reports RIGOR's Field Grounding at 40.0, the same as several baselines, which does not fit well with Figure 5 or the overall claims about grounding.
On top of this, the test sets are small: 24 PLAN questions, and the ACT set size is never stated. No variance or confidence intervals are given. There is no end-to-end run, even though connecting planning and execution is the main claim. The human study behind Figure 6 is not described, and the correlation is given as 0.9773 in the text but 0.9924 in the figure.
Finally, the scope is narrow: one workcell template with exactly the same 11 devices. The method is essentially SFT, scorer-guided beam search, and an SFT action model, so the technical novelty is limited.
Questions for authors
- What exactly is , and how much does it overlap with the metrics used for evaluation (, , relation checks)? If they overlap, could you show results with a separate scorer for refinement, or apply the same R+L search to a couple of baselines?
- How many of the 24 test layouts pass all hard constraints? Why is RIGOR's Structural Completeness (43.8) so low when its layout score is 87.04?
- How are the ACT train and test sets separated? Does the Unity panel layout change between them? How many units are in the ACT test set? Did the baselines see any examples of the action format?
- Which correlation in Figure 6 is correct (0.9773 or 0.9924)? Who were the human raters?
3 comments
Author Response to Reviewer HmJ3 (Part 2/3)
Q2. Hard-constraint passes and Structural Completeness (W1&Q2).
A2: None of the 24 original RIGOR layouts passed every hard constraint. All 24 contained all 11 devices, but none passed the parent–child attachment check. As a separate experiment, we applied a bounded rule-based repair to the attachments and positions using device bounding boxes and workspace constraints.
| 24 PLAN Questions | All Devices Present | Scorable | Parent Pass | Hard Pass | Soft Layout Score | Relation(%) |
|---|---|---|---|---|---|---|
| Original RIGOR outputs (additional hard-constraint statistics) | 24/24 | 21/24 | 0/24 | 0/24 (0.00%) | 87.04 | 79.69% |
| Original outputs with additional bounded rule-based repair | 24/24 | 24/24 | 24/24 | 24/24 (100.00%) | 38.93 | 26.56% |
Eq. (24) gives equal weight to device coverage and attachment. For each scorable layout,
Each of the 21 scorable layouts earned 50 points for device coverage. The other three contributed zero. Thus , reported as 43.8 in Figure 8. GPT-5.5 scores 97.9 on the same diagnostic. RIGOR’s 87.04 Layout score measures process distances and service placement. Repair raised hard passes to 24/24 while reducing Layout to 38.93 and Relation to 26.56%. We will report these outcomes together in Section 5.3.
Q3. ACT split, interface variation, and matched baseline comparisons (W2&Q3).
A3: ACT fine-tuning used 8,000 unique actions. Table 1’s 222,306 raw actions are the recording corpus from 938 non-test trajectories. The training and test splits are disjoint by task seed, task, and layout identity.
| Scope | Tasks and Action Counts |
|---|---|
| Final ACT fine-tuning | 36 tasks, 8,000 unique action units, presented 9,600 times in total |
| Frozen ACT test | 24 seeds × 2 interface configurations × 200 actions, totaling 48 trajectories and 9,600 actions |
| Additional matched-sample comparison | 2,400 matched actions (6 seeds, 12 trajectories); a smaller subset of 600 actions is included (3 seeds, 3 trajectories) |
(1) Interface and action format. Among records with explicit interface labels, training uses C1, C4, C8, and C10. Held-out test profiles are C2, C3, C5, C6, C7, C9, and C11–C16. General-purpose models receive the action vocabulary, required fields, coordinate convention, and a generic JSON example. UI-TARS and OpenCUA use native outputs with fixed scoring adapters. RIGOR receives task-specific training, and the external baselines are evaluated primarily zero-shot.
(2) Parameter-level execution. On the full frozen test, RIGOR’s 100.00% measures action-type accuracy. Its 81.14% strict Step Success also requires correct parameters and target progress. For the direct comparisons below, we matched action IDs, screenshots, targets, and history prefixes. The 600-action sample is contained in the 2,400-action sample.
| Additional Matched Sample | RIGOR Step Success | Strongest Baseline on the Same Sample | Difference |
|---|---|---|---|
| 2,400 actions (6 seeds, 12 trajectories) | 1,947/2,400 (81.13%) | DeepSeek-V4.1-Flash, 1,680/2,400 (70.00%) | +11.13 percentage points |
| 600 of those actions (3 seeds, 3 trajectories) | 491/600 (81.83%) | Claude Sonnet 5, 439/600 (73.17%) | +8.67 percentage points |
RIGOR’s Step Success gain over DeepSeek was positive on each of the six matched seeds (9.50–13.00 percentage points). The gain over Claude ranged from 8.50 to 9.00 points across three seeds. These are observed per-seed differences. We do not estimate a population confidence interval from these samples.
Author Response to Reviewer HmJ3 (Part 3/3)
Q4. Figure 6 correlation and human evaluation (W2&Q4).
A4: The Pearson correlation for Figure 6 is across 14 models. Section 5.6 retained the first-round value of 0.9773. We will update the text and caption to report the second-round result consistently.
Two doctoral students and one master’s student rated the same model outputs as the three blinded LLM raters. Each panel scored the outputs separately on the 0–5 scale. We aggregated scores by model before computing Pearson’s , so each point is a pair of human and LLM scores for one model. The correlation measures agreement at the model level.
Q5. The PLAN–ACT interface and sequential execution (W2).
A5: A layout can specify the correct desk coordinate while a GUI action enters it into the wrong field. RIGOR makes this distinction testable by compiling the selected layout into an explicit target for each device (Eq. 5).
T_j^{\mathrm{device}}=\bigl(\mathrm{id}_j,\lbrace(f,\nu_{j,f})\rbrace_{f\in\mathcal F_j}\bigr).$$ The dataset contains **1,050 validated layouts**. Task and layout IDs link **222,306 expert action units** from 938 verified trajectories to their source layouts. This makes it possible to check a planning error against the scene requirements and a GUI error against the specific device field the plan calls for. We also tested consecutive actions for a single device. ACT used its own action history and incrementally updated textual state, with the interface supplying the fixed device ID at each step. Success required creating the device, writing exact target values, saving, and verifying them by readback. ACT completed **22/46 tasks (47.83%) across 23 seeds**. Full 11-device scene completion remains unmeasured. **Q6. Field grounding and generalization across workcell conditions (W2).** A6: The 40.0 Field Grounding score and Figure 5 evaluate different fields. The diagnostic covers 144 clicks on Rotation Y, Scale Y, and Scale Z. RIGOR selected the right action type every time and achieved strict Step Success on none. | Result | Evaluation Target | Observation | | ---------------- | ----------------------------------------------------- | ------------------------------------------------------------ | | Field Grounding | 144 clicks targeting Rotation Y, Scale Y, and Scale Z | Correct action type: 144/144; strict Step Success: 0/144 | | Figure 5 example | Desk Position X | Predicted point [1721, 256] lies inside the target input field | By Eq. (23), $\mathrm{Field\ Grounding}=100[0.4(144/144)+0.6(0/144)]=40.0$. The desk’s Position X target in Figure 5 spans x = [1692, 1749] and y = [249, 262]. The illustrated click lands inside that field. On the three diagnostic fields, all 144 actions failed the full step criterion. The full dataset contains 97 condition groups and 291 task seeds. The 24 PLAN questions cover eight groups with three seeds each, one fixed 11-device workcell, and a Z-oriented main aisle. Within this workcell we test variation in aisle width, equipment clearance, and boundary clearance. Transfer to different workcells or device inventories remains open. We would be grateful if you would **reconsider your evaluation** in light of the common-budget planner comparisons, the explicit hard-pass counts, and the matched ACT results.
Author Response to Reviewer HmJ3 (Part 1/3)
We thank you for asking which planning gains survive when every planner can query the same evaluator, and whether ACT’s advantage extends to actions with the same targets and histories.
Q1. Authority, final scoring, and a common search budget (W1&Q1).
A1: Yes, Authority and the final PLAN metrics use the same geometric distances. PLAN predicts coordinates from the task and public asset geometry. At inference, Authority checks each proposed layout against the specified relations (Section 3.3). Following your suggestion, we gave all six initializations the same evaluator, refinement operators, and cap on evaluator calls.
(1) Search and scoring. Search ranks candidates by soft score, hard violations, or a composite. The search soft score averages six geometric components.
SearchSoft=6sprocess+stool+srack+scontainer+sdesk+ssign.
SearchComposite=100SearchSoft−NormalizedHardViolation.
A replacement must satisfy the output protocol, improve at least one component, preserve or increase SearchSoft, and introduce no new or larger hard violation. Final evaluation weights the process-distance score S1 at 60% and the mean of the five service-placement scores S2 at 40%. Both are normalized to [0,1]. Relation counts eight binary distance checks per question. All 24 questions remain in both denominators, with non-scorable layouts contributing zero (Appendix F.1).
Relation((2)∗∗Matchedrefinement.∗∗Everyinitializationusedbeamwidth4,depthatmost4,andaceilingof2,000distinctevaluationsperquestion,includingtheinitiallayout.Countsexcludethecostofgeneratingthestartinglayout.Arrowsgiveresultsbeforeandafterrefinement.∣R/LStartingPoint∣SoftLayoutScore∣Relation∣HardPass∣Eval.Calls∣∣−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−∣−−−−−−−−−−−−−−−−:∣−−−−−−−−−−−−−−−−−−:∣−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−:∣−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−:∣∣RIGOR(SFTinitialization;finalvalueonly)∣87.04∣79.69∣Base(finalvalueonly)∣98.88∣89.06∣Search−only(deterministicgrid)∣39.34→60.82∣12.50∣DeepSeek−V4.1−Flash∣40.07→54.64∣29.17∣GPT−5.5∣26.71→45.05∣22.92∣Qwen3.5−9B∣54.14→∗∗99.30∗∗∣21.88Beforesearch,coordinatesupervisionimprovestheBasemodelfrom∗∗17.25to25.85∗∗onLayout(Table3).Aftersearch,Qwen3.5−9Breaches99.30andtheuntunedBasemodelreaches98.88,bothaboveRIGOR’s87.04.DeepSeekpassesallhardconstraintson∗∗22/24∗∗questionsdespiteitslowerLayoutof54.64.ThecontrolledgainfromSFTisthereforethe8.60−pointimprovementbeforesearch.Thepost−searchscoresalsodependontheevaluator−guidedrefinement.