acceptodds
Under review as a conference paper at ICLR 2027

Forge: Fault-Oriented Rework Code Generation for Executable Interlocking Assembly

Abstract

Large language models have demonstrated strong capabilities in structured rea- soning and code generation, yet identifying an assembly fault does not guarantee the ability to generate executable rework programs. In interlocking assemblies, a locally plausible repair program may violate blocking dependencies, omit required recovery operations, or terminate in an incorrect configuration. We identify this challenge as fault-conditioned executable rework generation, where models must jointly reason about structural failures and generate programs that restore con- strained assembly states. To benchmark this capability, we introduce FORGE- Bench, containing 4,000 tasks across 26 interlocking assemblies with five as- semblies held out for evaluating generalization to unseen structures. We further propose FORGE (Fault-Oriented Rework Code Generation for Executable In- terlocking Assembly), a structured state-to-program framework that decomposes rework generation into Rework Structuring and Action Grounding. Rework Structuring captures fault-induced recovery dependencies, while Action Ground- ing maps these dependencies into state-conditioned executable actions. Experi- ments show that locally reasonable repair code is insufficient for reliable assembly recovery, and that connecting structural recovery reasoning with executable action realization is essential for verified target restoration.

Primary area:
datasets and benchmarks
Keywords:
Large Language Models, Assembly Rework, Fault Diagnosis, Executable Program Generation, Interlocking Assemblies, Symbolic Verification, State-to-Program Generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_REyVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper tackles the problem of fault-conditioned executable rework generation. The authors introduce a benchmark suite (FORGE-Bench) and FORGE methodology to address this challenge. The FORGE methodology incorporates rework structuring and action grounding.

Critical strengths

Overall, the technique seems plausible (Figure 2) and the details look decent. Program fixing is easy to validate, so I have no issue with the validation.

Critical weaknesses

The paper is very difficult to follow.

Assembly: I thought this referred to assembly language for the first 5 pages, and only then figured that it doesn't. I'm still not sure what assemblies refers to. Are these physical structures that need to be assembled? Fault definition: This is also something I didn't capture. Typically faults refer to hardware defects or issues, that may propagate as errors. What other types of faults do the authors intend (i.e., what is a software fault; as opposed to a software error?). The problem statement: I still don't understand the objective of the work. How is the stakeholder, and how do they benefit from this solution. ForgeBench: what influences the 3000/1000 total forgebench tasks? Is there something special about 3000? Could more have been generated? Or, conversely, how many of the 3000 are redundant and can be deduplicated? Also: what is fault? Assembly: what assembly language? or hardware was used? This is not included in the experimental setup, which makes me further question the definition of assembly. Metrics should not be put in an appendix. These are quite important.. Line 356 main results (table 2) has abbreviations not previously explained. For human judges, did the authors obtain IRB approvals?

Questions for authors

Significant work has to be done to convince me of the problem statement at a conceptual level. Definitions are missing. Assembly; Fault; lots of acronyms. See Weaknesses above.

4 comments
AuthorsOpenReview ↗

Response to Reviewer REyV

Reviewer REyV:

We thank you for pointing out that the opening leaves the meaning of assembly unclear. We will introduce the mechanical repair task in Section 1 with the bayonet example from Appendix A.2.

Q1 & Q2. Definition of a mechanical assembly fault and intended use of the repair program (W1&W2&Q1).

A1 & A2 : (1) Task definition. An assembly is a three-dimensional structure of interlocking mechanical parts. A fault is a discrepancy between the observed assembly and its target, such as a missing or incorrect part, insufficient insertion, or a pose outside tolerance. In the bayonet example, the lock is present but its pin is missing. The diagnosis is missing_part, and one insert action restores the target. If access is blocked, the repair must also clear and restore the obstructing part. An assembly already within tolerance requires a no-fault diagnosis and an empty plan.

(2) Intended use. Assembly engineers and researchers can use these Unity simulations to test whether a repair planner restores the entire assembly. Given initial, observed, and target states, parts, constraints, and available actions, the model returns a JSON diagnosis and repair program. Symbolic replay checks whether each step is legal and whether any part remains out of place at the end.

Q3. Rationale for the 3,000/1,000 split and duplicate training inputs (W3).

A3: The 3,000 training and 1,000 test tasks define the reported evaluation, not a generation limit. An assembly-disjoint split evaluates generalization to unseen structures. Additional tasks can be generated by varying faulty parts, pose perturbations, and fault combinations, with each reference repair verified through symbolic replay (Section 4.1).

Of the 3,000 training records, 109 (3.63%) repeat a canonicalized public input, leaving 2,891 distinct inputs. The diagnosis and repair labels agree within every duplicate group. All reported results use the original split.

Q4. Definitions of the six core metrics before the main results (W4&Q1).

A4: We will define the six metrics before Table 2. For example, a correctly diagnosed repair that legally restores the target via a different action sequence earns DGTR credit. The sequence difference alone rules out SSE and MVE credit.

MetricFull NameEvaluation Focus
SSEStructural Sequence ExactExact agreement with the reference action/part sequence
SAPFState–Action Parameter FidelityRequired diagnostic-state fields and action parameters
VMREVerified Minimal Repair EfficiencyReference-relative action efficiency after diagnosis, execution, and parameter checks
DGTRDiagnosis-Grounded Target RestorationCorrect diagnosis and legal target restoration without reference-order matching
MVEMinimal Verified ExecutionCorrect diagnosis, verified restoration, valid parameters, reference-sequence agreement, and reference-level cost
HRQHard Reference QualityReference similarity and parameter fidelity on hard cases passing diagnosis and restoration checks

For MVE, reference-level cost allows at most as many actions as the verified reference (Eq. 15). Overall-Core averages the six unrounded scores with equal weight (Eq. 18). ETR measures legal target restoration regardless of diagnosis or action order. ETR, LLM-as-Judge scores, and expert ratings are reported separately. Appendix D gives the full definitions and tolerances.

Q5. Expert evaluation protocol and IRB applicability (W5&E1).

A5: In our assessment, this evaluation did not require human-subjects institutional review board (IRB) approval. The experts judged the technical quality of model-generated diagnoses and repair programs.

Following LLM-as-Judge scoring, three experts in mechanical assembly, symbolic planning, or structured-program evaluation independently assessed model-generated diagnoses and repair programs. Both evaluations used the same 0–5 rubric for diagnosis consistency, recovery completeness, action executability, and target-state restoration. Expert ratings were averaged with equal weight and aggregated into model-level scores, separately from Overall-Core. We will clarify the protocol and the basis for our IRB assessment in Section 5.5 and Appendix F.

We hope the worked example and metric definitions clarify what the benchmark measures, and we would be grateful if you would reconsider your evaluation.

AuthorsOpenReview ↗

Follow-up on Task Definitions and Evaluation Clarifications

Thank you for your valuable feedback and for highlighting areas where greater clarity was needed.

We hope our explanations of the mechanical assembly task, fault definitions, evaluation metrics, and expert assessment protocol have helped clarify the scope and objectives of our work.

We would sincerely appreciate your consideration of these clarifications in your overall assessment. If any concerns remain, we would be glad to provide further explanation.

Thank you again for your time and thoughtful review.

AuthorsOpenReview ↗

Request for Feedback on the Task and Evaluation Clarifications

Dear Reviewer REyV,

Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our explanations of the mechanical assembly task, fault definitions, intended use, and evaluation and expert assessment protocols have addressed your concerns.

If any points remain unresolved, please let us know so that we can provide targeted clarification during the discussion.

Thank you for your consideration.

Best regards, The authors of Paper 6660

AuthorsOpenReview ↗

Request for Rating Reconsideration

Dear Reviewer REyV,

Thank you again for your time and thoughtful review. We would appreciate your feedback on whether our explanations of the mechanical assembly task, fault definitions, intended use, and evaluation and expert assessment protocols have addressed your concerns.

If any points remain unresolved, please let us know so that we can provide targeted clarification during the discussion.

Thank you for your consideration.

Best regards, The authors of Paper 6660

Reviewer_YX62OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces FORGE-Bench, a benchmark for generating executable rework programs for faulty interlocking assemblies. Given the observed and target states, structural relations, and a discrete action library, a model must diagnose faults and produce actions that are legal at each step and restore the target configuration. The authors also propose FORGE, which organizes fault-dependent recovery sequences and grounds actions in symbolic states when constructing supervision for a Qwen3.5-9B model. The benchmark contains 4,000 tasks across 26 assemblies, with five assembly identities held out for testing. FORGE obtains an Overall-Core score of 72.75, compared with 67.53 for the same-backbone vanilla SFT model.

Critical strengths
  1. The benchmark checks both stepwise action legality and final-state restoration, directly testing whether a plausible repair program actually works under its symbolic rules.
  2. The assembly-disjoint test split and same-backbone vanilla SFT comparison make the evaluation more informative than a random task split or comparisons with unadapted models alone.
Critical weaknesses
  1. The method's added mechanism is unclear. Sections 3.2–3.4 describe supervision organization, standard SFT, and replay after generation, but do not specify whether or how intermediate symbolic states enter generation. Figure 2 introduces further operations without corresponding algorithms. Table 3 does not say exactly what each ablation changes. The 5.22-point Overall-Core gain over vanilla SFT therefore cannot be attributed confidently to the proposed modules.
  2. The headline evaluation mixes distinct capabilities. Appendix B.3 defines minimality relative to a prescribed route; SSE and MVE require the reference action order, and HRQ rewards similarity to it. A different legal repair can therefore lose credit. DGTR does not require the same sequence and improves by 4.37 points over vanilla SFT, so that result should be reported separately from reference matching. The main tables also omit false-intervention or empty-program rates for the 1,115 no-fault tasks, despite the paper's no-intervention objective.
Questions for authors
  1. Can you show one task's exact training targets and inference steps under vanilla SFT, both ablations, and full FORGE? Does generation receive the state resulting from previous actions?
  2. What are the reference-sequence-independent restoration rate and the no-fault false-intervention rate for each method? How often are valid alternative action orders penalized by Overall-Core?
  3. How does FORGE compare with a graph-dependency or symbolic-search baseline using the same public target state, blocking graph, and action library?
6 comments
AuthorsOpenReview ↗

Response to Reviewer YX62 — Q1: Training Targets and Ablations

We thank you for asking whether FORGE’s restoration gains persist when the evaluator accepts alternative repairs.

Q1. Training targets, intermediate states, and the source of the ablation gain (W1&Q1).

A1: (1) Same-task targets. For luban_3_piece_burr_train_000792, all four saved records share the public input, diagnosis, and final plan. The following field values are literal excerpts.

  • Vanilla SFT supervises annotation and rework_plan.
  • Rework Structuring only adds an fcrc obligation assigning "role":"CLEAR" to ThreePieceBurr_Part01.
  • Action Grounding only adds a cctg position transition for ThreePieceBurr_Part02 with {"before":[0.0,1.3,0.0],"after":[0.0,1.0,0.0]}.
  • Full FORGE includes both fields. The joint target links a transition to the corresponding clearance obligation through certificate_obligation={"obligation_id":"v0001","role":"CLEAR","part_id":"ThreePieceBurr_Part01","reference_action_index":0,"certificate_depth":0}. Training also includes diagnosis-and-plan-only records.

(2) Training and inference. We checked token supervision by reconstructing the labels under the recorded training configuration. In these records, every auxiliary token was retained and contributed to the SFT loss (Eq. 7). We will expand Eq. (6) and Appendix C.2 beyond the two-field target and include all four complete records. At inference, each model generates annotation and rework_plan in one autoregressive pass. No executor state enters decoding. We will clarify the operations labeled in Figure 2 by identifying which construct training targets and where post-decoding replay begins.

(3) Ablation. We matched each Table 3 result to its training configuration. All four runs use the same backbone and SFT objective. w/o Rework Structuring retains Grounding alone, and w/o Action Grounding retains Structuring alone.

MethodSSE ↑SAPF ↑VMRE ↑DGTR ↑MVE ↑HRQ ↑Overall-Core ↑LLM-as-Judge ↑
Vanilla SFT69.2361.1163.2371.5662.6077.4467.534.03
w/o Rework Structuring62.3841.6762.1968.1960.1073.6061.354.12
w/o Action Grounding65.4366.6764.0567.3863.1180.6767.894.30
FORGE (Ours)71.5461.1172.6075.9372.1383.1772.754.37

Joint supervision gains 5.22 Overall-Core points over Vanilla SFT. Structuring alone gains 0.36 points and Grounding alone loses 6.18. The largest gains for the full model are in verified execution (MVE, +9.53) and repair efficiency (VMRE, +9.37). Parameter fidelity (SAPF) remains at 61.11.

AuthorsOpenReview ↗

Response to Reviewer YX62 — Q2–Q3: Restoration and Planning Baselines

Q2. Restoration under alternative repair sequences and interventions on no-fault cases (W2&Q2).

A2: (1) Restoration under alternative sequences. FORGE gains 6.62 percentage points over Vanilla SFT on ETR and 4.37 on DGTR. Execution Target Restoration (ETR) credits a schema-valid program that executes legally and restores the target. Diagnosis-Grounded Target Restoration (DGTR) also requires exact diagnosis. Both scores are macro-averaged over fault types within each assembly, then over assemblies. ETR is reported separately from Overall-Core.

MethodETR (%) ↑DGTR (%) ↑Overall-Core ↑
Vanilla SFT74.8971.5667.53
FORGE (Ours)81.5175.9372.75
Improvement+6.62+4.37+5.22

(2) Effect of reference matching. SSE and MVE require exact agreement with the reference action/part sequence. HRQ grades sequence similarity on hard cases. Appendix B.3 uses the verified reference route as its action-count baseline. Global shortest-path optimality is untested. We audited 642 successful FORGE repairs to determine how often sequence agreement alone cost a repair MVE credit.

Outcome in the prediction auditRepairsShare of 642
Successful repairs audited642100.00%
Program differed from the reference10816.82%
Failed MVE solely on reference-sequence agreement60.93%
Differed solely by reordering the same actions00.00%

Of the 108 successful non-reference repairs, six met every MVE condition except sequence agreement. None of the 642 differed solely by reordering the same actions. Estimating the effect of those six cases on Overall-Core would require recomputing the macro-averaged metrics.

(3) No-fault interventions. FORGE and Vanilla SFT returned valid empty plans on every evaluated no-fault case. Table 1 counts 1,115 no-fault tasks across both splits. Below, ETR uses faulty cases. The other columns partition each method’s evaluated no-fault cases into valid empty plans, valid nonempty plans (false interventions), and invalid outputs. All entries are percentages.

MethodETR ↑Valid Empty Plan ↑False Intervention ↓Invalid Output ↓
Qwen3.5-9B Base46.7460.0040.000.00
Gemma-3-12B-IT9.1761.9038.100.00
GLM-4-9B-0414-Chat6.6766.6733.330.00
InternLM3-8B-Instruct20.000.00100.000.00
Ministral-3-8B-Instruct44.7420.0080.000.00
Llama-3.1-8B-Instruct1.610.9595.243.81
GPT-5.4 Mini47.3165.2434.760.00
Claude Sonnet 4.678.5175.7124.290.00
Qwen3.6-Plus81.021.4398.570.00
Gemini 3.5 Flash82.50100.000.000.00
AceCoder-7B48.14100.000.000.00
OpenCodeReasoning-Nemotron-7B15.21100.000.000.00
Seed-Coder-8B-Reasoning19.38100.000.000.00
Vanilla SFT74.89100.000.000.00
w/o Action Grounding71.52100.000.000.00
w/o Rework Structuring71.08100.000.000.00
FORGE (Ours)81.51100.000.000.00

Qwen3.6-Plus achieved 81.02% ETR on faulty cases yet intervened on 98.57% of its evaluated no-fault cases. Every audited FORGE empty plan was accompanied by a correct no-fault diagnosis.

Q3. Comparison with graph-dependency and symbolic-search planners.

A3: Both symbolic planners restored the target on every planning case they evaluated. Uniform-Cost Search (UCS) and the Graph Dependency Planner (GDP) use the public observed and target states, blocking graph, and action library. Both derive a rule-based diagnosis from state differences and generate plans without test reference programs.

MethodETR (%) ↑DGTR (%) ↑Overall-Core ↑
Vanilla SFT74.8971.5667.53
UCS + rule-based diagnosis100.0026.6740.08
GDP + rule-based diagnosis100.0026.6739.37
FORGE (Ours)81.5175.9372.75

Requiring exact diagnosis reduced both planners’ scores to 26.67% DGTR. The audit found omitted fault subtypes, incorrectly grouped simultaneous faults, and incorrect replacement labels. The planner and model rows use their respective case sets and tool-access protocols, which limits direct comparison. The controlled comparison is between FORGE and same-backbone Vanilla SFT.

We would be grateful if you would reconsider your evaluation in light of the restoration results and the analysis of valid alternative repairs.

AuthorsOpenReview ↗

Follow-up on Additional Evidence and Clarifications

Thank you for your constructive feedback and careful evaluation.

We hope the additional evidence regarding the training targets, reference-order-independent restoration results, and symbolic planning comparisons has helped address your concerns.

If you find these clarifications satisfactory, we would appreciate your reconsideration of our work in light of the additional evidence. We would also be happy to clarify any remaining questions.

Thank you again for your time and consideration.

Reviewer_o3jLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces FORGE-Bench and FORGE for generating rework programs that repair faults in interlocking assemblies while respecting action constraints and restoring the target state. The benchmark contains 4,000 tasks and uses a symbolic transition engine for evaluation; the proposed method builds on supervised fine-tuning and two additional modules.

Critical strengths

The task is practically relevant: models must diagnose faults, handle blocking and recovery dependencies, and restore the target state.The benchmark includes no-fault, single-fault, and multi-fault cases, with assembly-disjoint splits and verifier-based evaluation.Experiments compare diverse models and include ablations, case studies, and human evaluation. FORGE outperforms Vanilla SFT.

Critical weaknesses

1.The novelty and scope need stronger support. The papers’ approach relies on standard SFT, but the two main algorithmic modules are currently closer to supervision organization and action-constraint design than to clearly established algorithmic contributions. The paper should provide more examples of assemblies and cases demonstrating structural diversity.The related-work discussion would also benefit from positioning FORGE relative to CADMate: Generating CAD Assembly Plan with Geometric Chain-of-Thought and Spatial Physical Rewards (Chen et al., ACL 2026), particularly in terms of structured assembly reasoning and the use of geometric and physical constraints.

  1. The two core modules are underspecified. Rework Structuring is described as organizing the supervision target, but the paper does not clearly show how its labels differ from those used in Vanilla SFT. Action Grounding requires actions to depend on the state produced by preceding operations, but the paper does not explain whether this changes the input representation, training labels, or generation process.

  2. Some figures require clarification. Figure 4 lists interface operations such as Keypress, Click, Type Text, and Hotkey as part of FORGE-Bench, which appears inconsistent with the assembly task and action space described in the paper. The images labeled Target State and Actual State in Figure 2 may also be ambiguous or mislabeled and should be checked. In Figure 5, Vanilla SFT and Gemini 3.5 Flash have different rework plans, yet their final assembly images appear identical. The authors should clarify whether these images show the actual terminal states produced by each model or are illustrative.

Questions for authors

See above

6 comments
AuthorsOpenReview ↗

Response to Reviewer o3jL

We thank you for recognizing the practical value of fault-conditioned rework and execution-based verification.

Q1. Contribution of the two modules, relation to CADMate, and structural diversity (W1&Q1).

A1: FORGE uses standard supervised fine-tuning (SFT). Its contribution is to link each recovery obligation to the action that fulfills it. Rework Structuring follows assembly dependencies from the fault to identify what must be cleared, corrected, restored, or relocked (Eq. 4). Action Grounding supplies the corresponding action, parameters, and state transition.

(1) Controlled ablation. With the Qwen3.5-9B backbone and objective fixed, joint supervision raises Overall-Core from 67.53 to 72.75 (+5.22). Structuring alone scores 67.89 and Grounding alone 61.35 (Table 3). The gain persists when any legal repair route is accepted. Execution Target Restoration (ETR) rises from 74.89% to 81.51%, and Diagnosis-Grounded Target Restoration (DGTR), which also requires exact diagnosis, rises from 71.56% to 75.93%.

(2) Relation to CADMate. CADMate [1] predicts 6-DoF placements from disassembled parts and a reference image using geometric chain-of-thought and spatial-physical rewards. FORGE starts from a faulty assembly. A correctly installed part may have to be removed to reach the fault and reinstalled to restore the target. The verifier checks each action against the current state and the final assembly against the target. We will add this distinction to Section 2.

(3) Structural diversity. FORGE-Bench contains 4,000 tasks across 26 interlocking assemblies. The Unity programs and task records include multi-axis, spherical/radial, dovetail/sliding, compound mortise-and-tenon, and cylindrical/rotational interlocks. In the bayonet example, one insert action restores the missing pin (Appendix A.2). In the three-piece burr example, the program clears Part01, corrects Part02, inserts the missing Part03, restores Part01, and relocks Part03.

Q2. Supervision targets and the role of intermediate states in generation (W2&Q1).

A2: All four configurations use the same public input. The fields supervised for a given task differ as shown below.

ConfigurationTraining-target fields
Vanilla SFTannotation + rework_plan
Rework Structuring onlyannotation + fcrc + rework_plan
Action Grounding onlyannotation + cctg + rework_plan
Full FORGE, joint-target recordsannotation + fcrc + cctg + rework_plan

In joint records, fcrc encodes recovery obligations and dependencies, cctg records action-conditioned state transitions, and certificate_obligation links each transition to its obligation. Full FORGE mixes joint records with diagnosis-and-plan-only records. Eq. (6) and Appendix C.2 describe only the latter. We will include the augmented format and all four targets for the same task.

We reconstructed the training labels under the recorded configuration. In the audited records, all auxiliary tokens were retained and contributed to the loss. At inference, the model generates only annotation and rework_plan in one autoregressive pass. Replay starts after decoding, so no executor state enters generation.

Q3. State labels, action categories, and terminal-state images in Figures 2, 4, and 5 (W3&Q1).

A3: (1) Figure 2. We corrected the state labels. Initial State supplies the starting context, Actual State is the observed configuration where diagnosis and replay begin, and Target State is the configuration to restore.

(2) Figure 4. We replaced the GUI operations with fault counts across both splits (1,115 no-fault, 2,353 single-fault, and 532 multi-fault tasks). Table 5 gives the mechanical action library.

(3) Figure 5. The Vanilla SFT and Gemini 3.5 Flash panels had reused an illustration. We replaced each with the terminal state obtained by replaying that model’s program and verified the part poses against its replay record.

[1] CADMate: Generating CAD Assembly Plan with Geometric Chain-of-Thought and Spatial Physical Rewards. ACL 2026.

We hope the controlled ablation and corrected figures address your concerns, and we would be grateful if you would reconsider your evaluation.

AuthorsOpenReview ↗

Follow-up on Rebuttal Clarifications

Thank you for your thoughtful feedback and consideration of our work.

We hope our clarifications regarding the methodological contributions, structural diversity, and the corrected figures have addressed your concerns. If any questions remain, we would be happy to provide further clarification.

We sincerely appreciate your time and valuable feedback.

AuthorsOpenReview ↗

Request for Feedback on Our Rebuttal

Dear Reviewer o3jL,

Thank you again for your time and thoughtful review. We would appreciate a brief update on whether our responses have addressed your concerns regarding the supervision design, structural diversity, and revised figures.

If any points remain unresolved, please let us know so that we can provide targeted clarification during the discussion.

Thank you for your consideration.

Best regards, The authors of Paper 6660

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.