- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MAGAR-Bench, which contains 4K multimodal examples from 26families, organized into Assembly Decision, Atomic Action, and Minimal Rework tasks. Minimal Rework measures the number of distinct moved parts relative to a state’s validated recovery candidates. The paper also proposes MAGAR, a correction framework built around a frozen, supervised-fine-tuned Qwen3-VL-8B model. Its modules generate structured assembly/rework corrections and complete GUI-action replacements, while learned selectors determine when to retain the original prediction.
Critical strengths
-
It provides a useful diagnostic perspective for multimodal agents to distinguish assembly intent, GUI-action grounding, and recovery scope. Preserving already assembled components is a meaningful objective beyond action accuracy.
-
Matching correction granularity to output dependencies is well motivated. Preserving reliable fields while keeping action primitives coupled to their parameters is a sensible design.
-
Comparing against the same SFT anchor helps distinguish the correction framework’s contribution from supervised adaptation. In particular, Table 2 reports gains across all three tasks, including an increase in action-parameter correctness from 29.37 to 37.39.
Critical weaknesses
-
The Minimal Rework task needs a clearer explanation of how it is constructed and how difficult it actually is. For example, how are recovery candidates generated, how many candidates are provided, and what criteria are used to determine whether a recovery is valid? The paper should also explain how it handles cases where multiple moved-part sets are equally optimal. Without these details, it is hard to tell whether the task evaluates actual recovery reasoning or simply rewards matching a particular reference answer. Since the prompt already provides recovery candidates, the authors should clarify what information is available for each candidate. If action-to-part mappings are included, a simple baseline that minimizes the number of distinct moved parts would be an important comparison. Otherwise, the paper should explain what information the model needs to infer from the visual input.
-
I am concerned that the correction mechanism in Eq. 9 could produce inconsistent recovery outputs. It updates the moved-part set and count but keeps the original route and action sequence unchanged. This means the reported moved parts may not match what the actions actually move. Neither MPA nor the fieldwise RCA score appears to check for this inconsistency. While the authors acknowledge that execution is not evaluated, this makes it difficult to know whether the improved recovery predictions would actually lead to better recovery behavior. I would suggest adding a consistency check between the actions and moved parts, ideally along with execution replay for a subset of corrected recoveries. The authors should also justify using exact matching against a single reference when multiple valid recovery solutions may exist.
-
The current ablation studies do not fully support the main claim about correction granularity. Table 2 shows that task-specific modules improve performance on their corresponding tasks, but it is still unclear which parts of the correction design are responsible for these improvements. For example, how much of the gain comes from preserving individual fields rather than replacing the entire action? Does plan conditioning actually help, and how important is the intervention gate? I would recommend comparing whole-output replacement against field-preserving correction using the same candidate budget. It would also be useful to compare independent and joint action-parameter correction, as well as always replacing predictions versus using the intervention gate.
-
Several implementation details are still missing, making the proposed method difficult to reproduce. Table 4 explains the backbone adaptation, but provides limited information about how the proposed modules are trained and used. The authors should report the candidate budgets, training objectives and schedules, selector architecture, utility weights, intervention thresholds, and calibration procedures. The description of "family-level cross-fitting" also needs to be more specific. Which components are retrained in each fold, and which data are used to determine the thresholds? Finally, the five rework grounders introduce additional model capacity and computational overhead, but their individual contributions and costs are not evaluated. An analysis of these trade-offs would help readers better understand whether the added complexity is justified.
Questions for authors
-
What exact test examples and denominators were used for each Table 1 model? Can you reconcile these with Table 5 and explain the different Overall values in Figure 8?
-
How are recovery candidates generated and validated? How many candidates typically exist per state, and are all equally optimal moved-part sets accepted?
-
What recovery information is visible at inference? How well does a symbolic candidate-selection baseline perform using that same information?
2 comments
Author Response to Reviewer h3KH (Part 1/2)
Q4. Correction granularity, plan conditioning, and the intervention gate (W3).
A4: With the selected candidate held fixed, preserving individual fields retained 80/238 fully correct decisions, compared with 19/238 for whole-output replacement.
(1) Field preservation. Both conditions used the same SFT predictions, 12 candidate slots, and selected candidate indices. Both accepted the replacement. We varied only which fields were overwritten.
| Configuration | OPG | OBA | SGF | IBC | Fully correct |
|---|---|---|---|---|---|
| Vanilla-SFT | 36.45 | 32.76 | 42.86 | 33.61 | 80/238 |
| Whole-output replacement | 6.54 | 33.45 | 40.34 | 7.98 | 19/238 |
| Field-preserving correction | 36.45 | 32.76 | 42.86 | 33.61 | 80/238 |
Whole-output replacement corrupted 115 previously correct part sets. Field preservation repaired one set without damaging any correct field and matched Vanilla-SFT on fully correct decisions.
(2) Plan conditioning and joint actions. Table 2 reports 37.50 Overall for SFT, 39.63 with Module A, 39.26 with Module B, and 41.39 with both. Module B conditions routing and parameter experts on and selects a primitive together with its parameters. PCA rises from 29.37 to 37.39 (+8.02) and WCS from 50.64 to 54.24 (+3.60). These gains measure Module B as a whole. We did not separately ablate plan conditioning or compare joint with independent parameter correction.
(3) Gate versus forced replacement. With the MR candidates and ranking fixed, always replacing yielded 10.00 MPA / 42.75 RCA and turned 16 previously correct MPA predictions into errors. In this diagnostic, the gate rejected every change under its training-calibrated thresholds, preserving SFT’s 24.00 / 45.50. In the main evaluation MAGAR intervened 11 times and scored 30.00 / 48.75 (Tables 2 and 5).
Q5. Training, candidate budgets, calibration, and grounder cost (W4).
A5: We freeze SFT and calibrate thresholds on held-out Train families.
(1) Training and candidate budgets. Module A’s structural head, base action experts, and plan residuals train for 2/2/1 epochs at // (effective batch 2); Module A and base experts are frozen during residual training. Each grounder trains for 35 epochs (, batch 16) with weighted BCE, count CE, and margin losses. AD generates 12 full candidates with up to seven field variants (17–51 non-anchor after deduplication). AA shares 18–28 complete-action candidates; five MR grounders yield 40–265 source-tagged records, not necessarily distinct.
(2) Selector and gate. AD/MR selectors have two heads: predicted metric gains over SFT and sigmoid benefit (Eqs. 4–6). The weights are for OPG/OBA/SGF/IBC and for MPA/RCA. With , we minimize
The highest-utility candidate replaces SFT only if and . Calibration criteria follow; fitted thresholds and values are not in the manuscript.
(3) Cross-fitting. Only selectors are refit; SFT/LoRA, generators, and grounders remain frozen. Across five folds, AD/MR fit on 16 Train families and predict on four; pooled out-of-fold results calibrate thresholds. AD requires ≥4 interventions, positive cumulative gains in all four metrics, and . MR requires ≥2 interventions, gains in both metrics, ≤1 regression, and . Final selectors refit on all 20 families with thresholds fixed before testing. AA uses a 20-member leave-one-family-out ridge committee, each calibrated/refit on 19 families; ≥0.5 vote share accepts a complete action.
(4) Grounder coverage and cost. On Train500, individual correct-set/count coverage is 295/268/305/282/283; the union is 430/500 (86%), versus 305/500 (61%) for the best single head. Removing any head loses 8–15 unique cases. Together they use 32.62M parameters (124.43 MiB FP32). Sequential median inference is 15.632 ms (one Xeon Platinum 8358 CPU thread, cached features, nine parts, batch 1), excluding backbone, enumeration, and selection.
We would be grateful if you would reconsider your evaluation in light of the execution checks, matched replacement control, and clarified evaluation protocol.
Author Response to Reviewer h3KH (Part 1/2)
We thank you for requesting the evaluation denominators, recovery validation, and controls for the correction mechanism.
Q1. Evaluation examples, denominators, and Figure 8.
A1: The correct Overall scores are 41.39 for MAGAR and 37.50 for Vanilla-SFT (Table 2). Eq. (13) gives equal weight to the task means. For MAGAR these are 38.18 for Assembly Decision (AD), 46.61 for Atomic Action (AA), and 39.38 for Minimal Rework (MR). We will replace the values of 38.67 and 29.97 in Figure 8 and align its model labels with the result tables.
We evaluated MAGAR and Vanilla-SFT on all 1,000 examples from six held-out families. The table gives the full-test denominator for each metric.
OPG excludes empty reference part sets. PCA pools required parameter components. WCS averages components per step, steps per workflow, then workflows equally. Invalid outputs remain in the denominators. Overall equally weights the three unrounded task means. The separate judge sample has 120 cases, 40 per task (Appendix C.3.4).
Table 5 counts predictions changed from Vanilla-SFT on this split. MAGAR intervened on 14/238 AD, 184/662 AA, and 11/100 MR cases, or 209/1,000 in total. For the other Table 1 models, we cannot confirm the exact per-model example lists or denominators.
Q2. Recovery construction, alternative optima, and execution consistency (W1&W2&Q2).
A2: We select the validated recovery that restores the target while moving the fewest distinct parts (Appendix C.2).
(1) Construction and minimality. We perturb translation, rotation, or both, then generate no-op, direct-restoration, and additional-clearance proposals. Before validation, 40% of records have two proposals and 60% have three. We check each proposed route’s transitions and verify that the final poses fall within target tolerances.
Let D contain the parts outside their fixed target-pose tolerances. Every successful route must move all of D. A feasible route moving exactly these parts attains the lower bound ∣D∣, even if several action orders attain it. The reference sets we checked matched D. In one Unity case, five actions restored the target while moving exactly the four necessary parts. The route passed 396 part-pair/state mesh checks at post-action and terminal states.
(2) Equivalent solutions. When a feasible route moves exactly D, its moved-part set is uniquely minimal. MPA accepts that set in any order with the correct count. A different equally minimal set would receive no credit under single-reference scoring. We found no annotated ties between distinct sets. RCA also scores the route ID and ordered action IDs, so a valid alternative sequence can lose sequence credit. Unity replay tests restoration independently of reference order.
(3) Prediction–execution consistency. We separately replayed all 11 changed MR predictions from paired failed states. Both methods passed the transition checks in 11/11 cases and restored 6/11 targets. Exact trace agreement on the moved-part set and count improved from 1/11 to 5/11, and restoration with matching scope from 0/11 to 4/11. The trajectories are unchanged under Eq. (9).
Q3. Model-visible information and symbolic candidate selection (W1&Q3).
A3: The model sees the failed and target images, candidate-part references, public state, interaction history, and numerical Inspector transforms. Recovery proposals are used during annotation. Figure 11’s phrase “within the provided recovery candidates” obscures that distinction, and we will correct it.
The part, route, and action IDs specify the permitted outputs. The model receives no list of candidate routes, costs, or action-to-part mappings, and neither a complete table of paired current and target poses nor labels identifying the parts outside tolerance. A symbolic method using these inputs must first infer which parts are displaced and which movements can restore them. We did not evaluate a symbolic baseline with that visual inference step.