- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper tackles the problem of fault-conditioned executable rework generation. The authors introduce a benchmark suite (FORGE-Bench) and FORGE methodology to address this challenge. The FORGE methodology incorporates rework structuring and action grounding.
Critical strengths
Overall, the technique seems plausible (Figure 2) and the details look decent. Program fixing is easy to validate, so I have no issue with the validation.
Critical weaknesses
The paper is very difficult to follow.
Assembly: I thought this referred to assembly language for the first 5 pages, and only then figured that it doesn't. I'm still not sure what assemblies refers to. Are these physical structures that need to be assembled? Fault definition: This is also something I didn't capture. Typically faults refer to hardware defects or issues, that may propagate as errors. What other types of faults do the authors intend (i.e., what is a software fault; as opposed to a software error?). The problem statement: I still don't understand the objective of the work. How is the stakeholder, and how do they benefit from this solution. ForgeBench: what influences the 3000/1000 total forgebench tasks? Is there something special about 3000? Could more have been generated? Or, conversely, how many of the 3000 are redundant and can be deduplicated? Also: what is fault? Assembly: what assembly language? or hardware was used? This is not included in the experimental setup, which makes me further question the definition of assembly. Metrics should not be put in an appendix. These are quite important.. Line 356 main results (table 2) has abbreviations not previously explained. For human judges, did the authors obtain IRB approvals?
Questions for authors
Significant work has to be done to convince me of the problem statement at a conceptual level. Definitions are missing. Assembly; Fault; lots of acronyms. See Weaknesses above.
4 comments
Follow-up on Task Definitions and Evaluation Clarifications
Thank you for your valuable feedback and for highlighting areas where greater clarity was needed.
We hope our explanations of the mechanical assembly task, fault definitions, evaluation metrics, and expert assessment protocol have helped clarify the scope and objectives of our work.
We would sincerely appreciate your consideration of these clarifications in your overall assessment. If any concerns remain, we would be glad to provide further explanation.
Thank you again for your time and thoughtful review.
Request for Feedback on the Task and Evaluation Clarifications
Dear Reviewer REyV,
Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our explanations of the mechanical assembly task, fault definitions, intended use, and evaluation and expert assessment protocols have addressed your concerns.
If any points remain unresolved, please let us know so that we can provide targeted clarification during the discussion.
Thank you for your consideration.
Best regards, The authors of Paper 6660
Request for Rating Reconsideration
Dear Reviewer REyV,
Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our explanations of the mechanical assembly task, fault definitions, intended use, and evaluation and expert assessment protocols have addressed your concerns.
If any points remain unresolved, please let us know so that we can provide targeted clarification during the discussion.
Thank you for your consideration.
Best regards, The authors of Paper 6660
Response to Reviewer REyV
Reviewer REyV:
We thank you for pointing out that the opening leaves the meaning of assembly unclear. We will introduce the mechanical repair task in Section 1 with the bayonet example from Appendix A.2.
Q1 & Q2. Definition of a mechanical assembly fault and intended use of the repair program (W1&W2&Q1).
A1 & A2 : (1) Task definition. An assembly is a three-dimensional structure of interlocking mechanical parts. A fault is a discrepancy between the observed assembly and its target, such as a missing or incorrect part, insufficient insertion, or a pose outside tolerance. In the bayonet example, the lock is present but its pin is missing. The diagnosis is
missing_part, and oneinsertaction restores the target. If access is blocked, the repair must also clear and restore the obstructing part. An assembly already within tolerance requires a no-fault diagnosis and an empty plan.(2) Intended use. Assembly engineers and researchers can use these Unity simulations to test whether a repair planner restores the entire assembly. Given initial, observed, and target states, parts, constraints, and available actions, the model returns a JSON diagnosis and repair program. Symbolic replay checks whether each step is legal and whether any part remains out of place at the end.
Q3. Rationale for the 3,000/1,000 split and duplicate training inputs (W3).
A3: The 3,000 training and 1,000 test tasks define the reported evaluation, not a generation limit. An assembly-disjoint split evaluates generalization to unseen structures. Additional tasks can be generated by varying faulty parts, pose perturbations, and fault combinations, with each reference repair verified through symbolic replay (Section 4.1).
Of the 3,000 training records, 109 (3.63%) repeat a canonicalized public input, leaving 2,891 distinct inputs. The diagnosis and repair labels agree within every duplicate group. All reported results use the original split.
Q4. Definitions of the six core metrics before the main results (W4&Q1).
A4: We will define the six metrics before Table 2. For example, a correctly diagnosed repair that legally restores the target via a different action sequence earns DGTR credit. The sequence difference alone rules out SSE and MVE credit.
For MVE, reference-level cost allows at most as many actions as the verified reference (Eq. 15). Overall-Core averages the six unrounded scores with equal weight (Eq. 18). ETR measures legal target restoration regardless of diagnosis or action order. ETR, LLM-as-Judge scores, and expert ratings are reported separately. Appendix D gives the full definitions and tolerances.
Q5. Expert evaluation protocol and IRB applicability (W5&E1).
A5: In our assessment, this evaluation did not require human-subjects institutional review board (IRB) approval. The experts judged the technical quality of model-generated diagnoses and repair programs.
Following LLM-as-Judge scoring, three experts in mechanical assembly, symbolic planning, or structured-program evaluation independently assessed model-generated diagnoses and repair programs. Both evaluations used the same 0–5 rubric for diagnosis consistency, recovery completeness, action executability, and target-state restoration. Expert ratings were averaged with equal weight and aggregated into model-level scores, separately from Overall-Core. We will clarify the protocol and the basis for our IRB assessment in Section 5.5 and Appendix F.
We hope the worked example and metric definitions clarify what the benchmark measures, and we would be grateful if you would reconsider your evaluation.