acceptodds
Withdrawn from ICLR 2027

MAGAR: Multimodal Agents for GUI-Based 3D Mechanical Assembly and Minimal Rework

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A GUI command can be correct at the interface level yet still produce an incorrect assembly. An agent must identify the relevant components, respect their struc- tural relationships, and determine which parts should move during recovery. We introduce MAGAR-Bench, a benchmark of 4,000 multimodal samples from 26 assembly families that separates Assembly Decision, Atomic Action, and Min- imal Rework. For rework, the objective counts distinct moved parts within the validated recovery candidate set for each state, rather than the number of actions. We also introduce MAGAR, a selective correction framework built on a frozen policy obtained by supervised fine-tuning. Its key design matches correction gran- ularity to output dependencies. Structured candidates revise assembly fields and rework scope, while complete action candidates keep native GUI primitives cou- pled with their parameters. A learned selector retains the original prediction un- less a candidate passes a task-specific intervention threshold. On recorded states from held-out assembly families, MAGAR improves Overall, the task-balanced aggregate, from 37.50 to 41.39, with gains on all three tasks. Across 209 interven- tions, 70 correct an error and 22 introduce one. These results support task-specific correction while highlighting the remaining tradeoff between fixing errors and preserving reliable predictions.

Primary area:
datasets and benchmarks
Keywords:
multimodal GUI agents, vision-language models, 3D mechanical assembly, assembly decision, atomic action grounding, minimal rework, selective correction, multimodal benchmark

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_h3KHOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MAGAR-Bench, which contains 4K multimodal examples from 26families, organized into Assembly Decision, Atomic Action, and Minimal Rework tasks. Minimal Rework measures the number of distinct moved parts relative to a state’s validated recovery candidates. The paper also proposes MAGAR, a correction framework built around a frozen, supervised-fine-tuned Qwen3-VL-8B model. Its modules generate structured assembly/rework corrections and complete GUI-action replacements, while learned selectors determine when to retain the original prediction.

Critical strengths
  1. It provides a useful diagnostic perspective for multimodal agents to distinguish assembly intent, GUI-action grounding, and recovery scope. Preserving already assembled components is a meaningful objective beyond action accuracy.

  2. Matching correction granularity to output dependencies is well motivated. Preserving reliable fields while keeping action primitives coupled to their parameters is a sensible design.

  3. Comparing against the same SFT anchor helps distinguish the correction framework’s contribution from supervised adaptation. In particular, Table 2 reports gains across all three tasks, including an increase in action-parameter correctness from 29.37 to 37.39.

Critical weaknesses
  1. The Minimal Rework task needs a clearer explanation of how it is constructed and how difficult it actually is. For example, how are recovery candidates generated, how many candidates are provided, and what criteria are used to determine whether a recovery is valid? The paper should also explain how it handles cases where multiple moved-part sets are equally optimal. Without these details, it is hard to tell whether the task evaluates actual recovery reasoning or simply rewards matching a particular reference answer. Since the prompt already provides recovery candidates, the authors should clarify what information is available for each candidate. If action-to-part mappings are included, a simple baseline that minimizes the number of distinct moved parts would be an important comparison. Otherwise, the paper should explain what information the model needs to infer from the visual input.

  2. I am concerned that the correction mechanism in Eq. 9 could produce inconsistent recovery outputs. It updates the moved-part set and count but keeps the original route and action sequence unchanged. This means the reported moved parts may not match what the actions actually move. Neither MPA nor the fieldwise RCA score appears to check for this inconsistency. While the authors acknowledge that execution is not evaluated, this makes it difficult to know whether the improved recovery predictions would actually lead to better recovery behavior. I would suggest adding a consistency check between the actions and moved parts, ideally along with execution replay for a subset of corrected recoveries. The authors should also justify using exact matching against a single reference when multiple valid recovery solutions may exist.

  3. The current ablation studies do not fully support the main claim about correction granularity. Table 2 shows that task-specific modules improve performance on their corresponding tasks, but it is still unclear which parts of the correction design are responsible for these improvements. For example, how much of the gain comes from preserving individual fields rather than replacing the entire action? Does plan conditioning actually help, and how important is the intervention gate? I would recommend comparing whole-output replacement against field-preserving correction using the same candidate budget. It would also be useful to compare independent and joint action-parameter correction, as well as always replacing predictions versus using the intervention gate.

  4. Several implementation details are still missing, making the proposed method difficult to reproduce. Table 4 explains the backbone adaptation, but provides limited information about how the proposed modules are trained and used. The authors should report the candidate budgets, training objectives and schedules, selector architecture, utility weights, intervention thresholds, and calibration procedures. The description of "family-level cross-fitting" also needs to be more specific. Which components are retrained in each fold, and which data are used to determine the thresholds? Finally, the five rework grounders introduce additional model capacity and computational overhead, but their individual contributions and costs are not evaluated. An analysis of these trade-offs would help readers better understand whether the added complexity is justified.

Questions for authors
  1. What exact test examples and denominators were used for each Table 1 model? Can you reconcile these with Table 5 and explain the different Overall values in Figure 8?

  2. How are recovery candidates generated and validated? How many candidates typically exist per state, and are all equally optimal moved-part sets accepted?

  3. What recovery information is visible at inference? How well does a symbolic candidate-selection baseline perform using that same information?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer h3KH (Part 1/2)

We thank you for requesting the evaluation denominators, recovery validation, and controls for the correction mechanism.

Q1. Evaluation examples, denominators, and Figure 8.

A1: The correct Overall scores are 41.39 for MAGAR and 37.50 for Vanilla-SFT (Table 2). Eq. (13) gives equal weight to the task means. For MAGAR these are 38.18 for Assembly Decision (AD), 46.61 for Atomic Action (AA), and 39.38 for Minimal Rework (MR). We will replace the values of 38.67 and 29.97 in Figure 8 and align its model labels with the result tables.

We evaluated MAGAR and Vanilla-SFT on all 1,000 examples from six held-out families. The table gives the full-test denominator for each metric.

TaskMetricFull-Test denominator / unit
ADOPG214 eligible instances
ADOBA6 operation classes; macro-average
ADSGF / IBC238 instances
AAATA662 instances
AAPCA1,035 required parameter components
AAWCS76 recorded workflows
MRMPA / RCA100 instances

OPG excludes empty reference part sets. PCA pools required parameter components. WCS averages components per step, steps per workflow, then workflows equally. Invalid outputs remain in the denominators. Overall equally weights the three unrounded task means. The separate judge sample has 120 cases, 40 per task (Appendix C.3.4).

Table 5 counts predictions changed from Vanilla-SFT on this split. MAGAR intervened on 14/238 AD, 184/662 AA, and 11/100 MR cases, or 209/1,000 in total. For the other Table 1 models, we cannot confirm the exact per-model example lists or denominators.

Q2. Recovery construction, alternative optima, and execution consistency (W1&W2&Q2).

A2: We select the validated recovery that restores the target while moving the fewest distinct parts (Appendix C.2).

(1) Construction and minimality. We perturb translation, rotation, or both, then generate no-op, direct-restoration, and additional-clearance proposals. Before validation, 40% of records have two proposals and 60% have three. We check each proposed route’s transitions and verify that the final poses fall within target tolerances.

Let DD contain the parts outside their fixed target-pose tolerances. Every successful route must move all of DD. A feasible route moving exactly these parts attains the lower bound ∣D∣|D|, even if several action orders attain it. The reference sets we checked matched DD. In one Unity case, five actions restored the target while moving exactly the four necessary parts. The route passed 396 part-pair/state mesh checks at post-action and terminal states.

(2) Equivalent solutions. When a feasible route moves exactly DD, its moved-part set is uniquely minimal. MPA accepts that set in any order with the correct count. A different equally minimal set would receive no credit under single-reference scoring. We found no annotated ties between distinct sets. RCA also scores the route ID and ordered action IDs, so a valid alternative sequence can lose sequence credit. Unity replay tests restoration independently of reference order.

(3) Prediction–execution consistency. We separately replayed all 11 changed MR predictions from paired failed states. Both methods passed the transition checks in 11/11 cases and restored 6/11 targets. Exact trace agreement on the moved-part set and count improved from 1/11 to 5/11, and restoration with matching scope from 0/11 to 4/11. The trajectories are unchanged under Eq. (9).

Q3. Model-visible information and symbolic candidate selection (W1&Q3).

A3: The model sees the failed and target images, candidate-part references, public state, interaction history, and numerical Inspector transforms. Recovery proposals are used during annotation. Figure 11’s phrase “within the provided recovery candidates” obscures that distinction, and we will correct it.

The part, route, and action IDs specify the permitted outputs. The model receives no list of candidate routes, costs, or action-to-part mappings, and neither a complete table of paired current and target poses nor labels identifying the parts outside tolerance. A symbolic method using these inputs must first infer which parts are displaced and which movements can restore them. We did not evaluate a symbolic baseline with that visual inference step.

AuthorsOpenReview ↗

Author Response to Reviewer h3KH (Part 1/2)

Q4. Correction granularity, plan conditioning, and the intervention gate (W3).

A4: With the selected candidate held fixed, preserving individual fields retained 80/238 fully correct decisions, compared with 19/238 for whole-output replacement.

(1) Field preservation. Both conditions used the same SFT predictions, 12 candidate slots, and selected candidate indices. Both accepted the replacement. We varied only which fields were overwritten.

ConfigurationOPGOBASGFIBCFully correct
Vanilla-SFT36.4532.7642.8633.6180/238
Whole-output replacement6.5433.4540.347.9819/238
Field-preserving correction36.4532.7642.8633.6180/238

Whole-output replacement corrupted 115 previously correct part sets. Field preservation repaired one set without damaging any correct field and matched Vanilla-SFT on fully correct decisions.

(2) Plan conditioning and joint actions. Table 2 reports 37.50 Overall for SFT, 39.63 with Module A, 39.26 with Module B, and 41.39 with both. Module B conditions routing and parameter experts on zAz_A and selects a primitive together with its parameters. PCA rises from 29.37 to 37.39 (+8.02) and WCS from 50.64 to 54.24 (+3.60). These gains measure Module B as a whole. We did not separately ablate plan conditioning or compare joint with independent parameter correction.

(3) Gate versus forced replacement. With the MR candidates and ranking fixed, always replacing yielded 10.00 MPA / 42.75 RCA and turned 16 previously correct MPA predictions into errors. In this diagnostic, the gate rejected every change under its training-calibrated thresholds, preserving SFT’s 24.00 / 45.50. In the main evaluation MAGAR intervened 11 times and scored 30.00 / 48.75 (Tables 2 and 5).

Q5. Training, candidate budgets, calibration, and grounder cost (W4).

A5: We freeze SFT and calibrate thresholds on held-out Train families.

(1) Training and candidate budgets. Module A’s structural head, base action experts, and plan residuals train for 2/2/1 epochs at 10−410^{-4}/2×10−42\times10^{-4}/5×10−55\times10^{-5} (effective batch 2); Module A and base experts are frozen during residual training. Each grounder trains for 35 epochs (4×10−44\times10^{-4}, batch 16) with weighted BCE, count CE, and margin losses. AD generates 12 full candidates with up to seven field variants (17–51 non-anchor after deduplication). AA shares 18–28 complete-action candidates; five MR grounders yield 40–265 source-tagged records, not necessarily distinct.

(2) Selector and gate. AD/MR selectors have two heads: predicted metric gains Δ^i(c)\widehat{\boldsymbol\Delta}_i(c) over SFT and sigmoid benefit p^i+(c)\widehat p_i^+(c) (Eqs. 4–6). The weights wt\mathbf w_t are (1,1,1,2)(1,1,1,2) for OPG/OBA/SGF/IBC and (4,1)(4,1) for MPA/RCA. With bi(c)=I[wt⊤Δi(c)>0]b_i(c)=\mathbb I[\mathbf w_t^\top\boldsymbol\Delta_i(c)>0], we minimize

Lsel=∑i,c[∥Δ^i(c)−Δi(c)∥22+λrelBCE⁡(p^i+(c),bi(c))].\mathcal L_{\mathrm{sel}}=\sum_{i,c}\left[\|\widehat{\boldsymbol\Delta}_i(c)-\boldsymbol\Delta_i(c)\|_2^2+\lambda_{\mathrm{rel}}\operatorname{BCE}(\widehat p_i^+(c),b_i(c))\right].

The highest-utility candidate U‾t(c)=wt⊤Δ^t(c)\overline U_t(c)=\mathbf w_t^\top\widehat{\boldsymbol\Delta}_t(c) replaces SFT only if U‾t(c)≥τt\overline U_t(c)\geq\tau_t and p^t+(c)≥τt+\widehat p_t^+(c)\geq\tau_t^+. Calibration criteria follow; fitted thresholds and λrel\lambda_{\mathrm{rel}} values are not in the manuscript.

(3) Cross-fitting. Only selectors are refit; SFT/LoRA, generators, and grounders remain frozen. Across five folds, AD/MR fit on 16 Train families and predict on four; pooled out-of-fold results calibrate thresholds. AD requires ≥4 interventions, positive cumulative gains in all four metrics, and Ngain≥3max⁡(Nharm,1)N_{\mathrm{gain}}\geq3\max(N_{\mathrm{harm}},1). MR requires ≥2 interventions, gains in both metrics, ≤1 regression, and Ngain≥2max⁡(Nharm,1)N_{\mathrm{gain}}\geq2\max(N_{\mathrm{harm}},1). Final selectors refit on all 20 families with thresholds fixed before testing. AA uses a 20-member leave-one-family-out ridge committee, each calibrated/refit on 19 families; ≥0.5 vote share accepts a complete action.

(4) Grounder coverage and cost. On Train500, individual correct-set/count coverage is 295/268/305/282/283; the union is 430/500 (86%), versus 305/500 (61%) for the best single head. Removing any head loses 8–15 unique cases. Together they use 32.62M parameters (124.43 MiB FP32). Sequential median inference is 15.632 ms (one Xeon Platinum 8358 CPU thread, cached features, nine parts, batch 1), excluding backbone, enumeration, and selection.

We would be grateful if you would reconsider your evaluation in light of the execution checks, matched replacement control, and clarified evaluation protocol.

Reviewer_WEcMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MAGAR-Bench separates assembly intent, atomic GUI actions, and minimal-rework prediction using recorded Unity states. MAGAR augments a frozen supervised-fine-tuned Qwen3-VL-8B policy with structured candidate generators and learned intervention selectors. It improves the task-balanced aggregate from 37.50 to 41.39, with 70 strict corrections and 22 introduced errors across 209 interventions.

Critical strengths

The distinction between assembly intent, GUI grounding, and recovery scope is useful.

Reporting introduced errors alongside corrections provides a more honest assessment than aggregate gains alone.

Critical weaknesses

The rework module changes the moved-part set and count while retaining the anchor’s route and action sequence. These fields can become mutually incompatible. The paper acknowledges this, but executable recovery validation is needed to substantiate the assembly-agent framing.

The reaonable of building a GUI benchmark, I would like to see analysis whether through api/cli the tasks can be done with coding agent (or a hybrid coding plus GUI agent)

Questions for authors

Do corrected outputs produce valid Unity transitions and successful recovery when executed?

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer WEcM

We thank you for asking whether the recovery actions executed in Unity move the parts identified by the model.

Q1. Compatibility between predicted recovery scope and Unity execution (W1&Q1).

A1: Of the six trajectories that restored the target, MAGAR identified exactly the parts moved in four, versus none for Vanilla-SFT. We replayed all 11 changed predictions among the 100 held-out rework cases (Table 5). Each pair used the same failed state, with no reference-action substitution, replanning, or post-hoc filtering.

Execution measureVanilla-SFTMAGAR
Recorded transition checks passed11/1111/11
Target restored6/116/11
Moved-part consistency (MPC)1/115/11
Recovery with consistent scope (JRC)0/114/11

Moved-part consistency (MPC) requires the predicted set and count to match the trace. Recovery with consistent scope (JRC) also requires valid transitions and target restoration. Eq. (9) preserves the route and actions, so both methods restore the target in the same six cases. The gain from 0/11 to 4/11 comes from correctly identifying what moved during successful recovery.

Q2. Assembly through GUI, API/CLI, and hybrid agents (W2).

A2: Yes. A coding agent could use an API or CLI where the software exposes the necessary operations. A hybrid agent could use the GUI for the remaining steps. Passing a part ID to a tool bypasses the gesture, but the agent must still decide which part to move, when to move it, and which parts to leave in place.

Assembly Decision and Minimal Rework test those structural decisions. Atomic Action tests their native GUI implementation. An API exposing exact poses or dependencies also changes what the agent can observe. Our experiments evaluate the GUI setting.

We hope the paired execution results address your concern about compatibility and would be grateful if you would reconsider your evaluation.

Reviewer_z1QaOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MAGAR-Bench, a benchmark of 4,000 multimodal samples from 26 assembly families that separates Assembly Decision, Atomic Action, and Minimal Rework. It also introduces MAGAR, a selective correction framework built on a frozen policy obtained by supervised fine-tuning. Structured candidates revise assembly fields and rework scope, while complete action candidates keep native GUI primitives coupled with their parameters. A learned selector retains the original prediction unless a candidate passes a task-specific intervention threshold.

Critical strengths

The paper explores 3D mechanical assembly, a relatively underexplored application domain for GUI agents. Its emphasis on structural dependencies and minimal rework offers an setting for studying assembly reasoning and error recovery beyond basic interface grounding.

Critical weaknesses
  1. The task setup and motivation need further clarification a. Why is a task-specific GUI agent model necessary? Can existing state-of-the-art GUI agents, such as UI-Venus2, or general-purpose multimodal models, such as GPT-astra or GPT sol already perform these tasks effectively? b. What is the practical significance of this task, and why is GUI-based interaction required? Could the same functionality be achieved through an MCP interface provided by the software?

  2. The evaluation is limited to an offline setting. GUI benchmarks such as OSWorld and AndroidWorld emphasize the importance of evaluating agents through interaction with the environment, where errors can accumulate and affect subsequent decisions. Consequently, it remains unclear whether the reported improvements translate into successful end-to-end completion of tasks.

  3. Limited comparison with SOTA methods, such as UI-venus 2.0 and OPUS.

  4. Many equations lack sufficient explanation. For example, Eq. (8) introduces the p_P without specifying how it is computed.

Questions for authors

See Weakness

1 comment
AuthorsOpenReview ↗

Author Response to Reviewer z1Qa

We thank you for asking what an assembly-specific GUI benchmark adds to existing evaluations.

Q1. Assembly decisions and their grounding in the GUI (W1&Q1).

A1: Figure 5 shows two failures that better click localization alone cannot fix. One prediction selects a nearby cap too early in the assembly order. Another proposes clicking Rotate when the shaft must be dragged.

Assembly Decision tests which operation, parts, and subgoal the assembly requires. Atomic Action tests the native command and its parameters. Minimal Rework tests which parts must move to restore a failed assembly (Section 3.1). An API exposed through MCP could achieve the same goal. Passing a part ID bypasses gesture grounding, but the caller must still choose the part and operation. The GUI setting tests both what to change and how to change it through the visible interface.

Q2. Recorded-state evaluation and execution in Unity (W2&Q1).

A2: We replayed all 11 predictions that MAGAR changed among the 100 held-out Minimal Rework cases. Each pair started from the same failed state and executed the predicted actions without reference substitution. Both methods passed the recorded transition checks in 11/11 cases and restored the target in 6/11. Exact agreement with the executed moved-part set and count rose from 1/11 to 5/11. Successful recovery with matching scope rose from 0/11 to 4/11.

Eq. (9) preserves the SFT action sequence, so both methods restore the target in the same six cases. The change is in how accurately the predicted scope describes the executed actions. The Workflow Consistency Score (WCS) measures action-component correctness within recorded workflows. Measuring error accumulation would require the agent to observe each new state and choose its next action.

Q3. Comparison with specialized GUI agents and general-purpose models (W3&Q1).

A3: Following your suggestion, we evaluated two specialized GUI agents and two general-purpose multimodal models on MAGAR-Bench. Overall weights the three tasks equally (Eq. 13). All scores are percentages.

ModelOPGOBASGFIBCATAPCAWCSMPARCAOverall
UI-Venus-2-9B0.0012.5010.420.0028.0311.0612.5310.006.2510.35
UI-Mate-9B13.6444.9218.754.1717.420.005.2515.0011.2513.68
GPT-6.1 Sol15.9123.4941.6712.5036.3622.1233.6185.0042.5039.28
Claude Opus 5.522.7343.1533.1916.6746.9728.0437.3764.0045.0040.30
Vanilla-SFT36.4532.7642.8633.6143.9629.3750.6424.0045.5037.50
MAGAR (Ours)39.2533.6244.5435.2948.1937.3954.2430.0048.7541.39

Claude Opus 5.5 scores 40.30 Overall. Its moved-part accuracy is 64.00, while its action-parameter correctness is 28.04. GPT-6.1 Sol shows the same pattern (85.00 and 22.12). Both models identify which parts must move more reliably than they predict the parameters of a GUI action.

On the full test split, MAGAR raises Overall from 37.50 to 41.39 (+3.89 points) over Vanilla-SFT on the same Qwen3-VL-8B-Instruct backbone. The largest gain is in action-parameter correctness, which rises from 29.37 to 37.39 (+8.02 points).

Q4. Computation of the part-set term in Eq. (8) (W4&Q1).

A4: The part-set score is the weakest inclusion or exclusion confidence. For operation oo, we project the assembly representation zAz_A with weights WqW_q and bias bqb_q, add the operation embedding eoe_o, and score each candidate part jj.

s_j^{(o)}=\sigma(\ell_j^{(o)}).$$ The key $k_j$ encodes the part reference, current–target difference, and target-state evidence. The scale $d_h$ is the query/key dimension. For a proposed set $P$, $$p_P(P\mid o,X,z_A)=\min_j\left[\mathbb I(j\in P)s_j^{(o)}+\mathbb I(j\notin P)(1-s_j^{(o)})\right].$$ Selected parts contribute inclusion confidence and omitted parts contribute exclusion confidence. The minimum over all candidate parts is unnormalized over subsets. A separate head $p_K$ predicts cardinality. We use the learned `rank_score` to order complete structural proposals. Eqs. (3)–(4) retain SFT unless the highest-utility correction clears both acceptance thresholds. We would be grateful if you would **reconsider your evaluation** in light of the same-backbone comparison and the paired execution results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.