acceptodds
Withdrawn from ICLR 2027

FROM RULE DIAGNOSTICS TO GUARDED COMPILA- TION FOR INFRASTRUCTURE-AS-CODE REPAIR

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Repairing an infrastructure rule violation can require constructing a resource that is itself subject to further rules. We use this rule structure to compile reusable resource obligations and execute them under a structural guard before model revision. Our diagnostic study shows why both steps matter: adding resource locations to feedback raises scanner acceptance by percentage points, concentrated on bucket-bearing templates, while joint scanner-and-lint acceptance stays flat. The intervention study evaluates a frozen compiler on matched CloudFormation tasks. With Haiku 4.5 and a six-call refinement cap, guarded compilation reaches joint acceptance versus Reflexion's , using versus calls and k versus k input tokens per task. Executing the library adds points over supplying it as context. Rule diagnostics thus motivate a concrete intervention: resolve supported obligations with checked code and reserve model revision for the residual failures. The benchmark was used during development, and the acceptance advantage does not persist with the tested stronger runtimes.

Primary area:
foundation or frontier models, including LLMs
Keywords:
verifier-guided repair, code generation, large language models, feedback localization, infrastructure-as-code

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ZkNoOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates a class of resource dependency problems when large language models repair Infrastructure-as-Code (IaC): satisfying the rule requirements of a certain resource may require creating another resource, which in turn is subject to further rule constraints. Using CloudFormation as an example, the authors propose pre-compiling supported resource constructs into reusable libraries, executing them under structure protection checks, and then having the model handle remaining errors.

The paper's three main contributions are:

  1. Diagnosing the discrepancy between feedback and repair quality. Through feedback ablation, the authors found that resource location can improve scanner pass rates, but not the joint pass rate of scanner and lint. Further comparison of resource identifiers and types, and S3 resource layering, links this phenomenon to similar resource obligations in log buckets.

  2. Proposing a repair process from rules to protected constructs. The method combines rule metadata, resource schemas, and manual mapping to construct six types of resource builders. These constructs are executed at runtime, checking protected paths and new structure diagnoses, and leaving residual issues to the model for revision.

  3. Validating improvements in joint pass rates and runtime consumption. The author provides a relatively complete repair process and distinguishes the role of the constructor library context and the actual execution of the constructor library through the ablation of fixed prompt words.

On 153 shared draft tasks, Haiku 4.5 achieved a joint pass rate of 88.2%, higher than Reflexion's 80.4%, with the average additional calls decreasing from 3.42 to 1.78. Ablation analysis of fixed libraries and prompt words further shows a 4.6 percentage point increase in execution compared to providing only library context. The acceptance rate advantage did not persist on stronger models tested.

Critical strengths
  1. Direct ablation evidence for execution gain. On 152 co-scored tasks, after fixing the construct library and prompt words, execution increased the joint pass rate by 4.6 percentage points, with a 95% CI of [1.3, 8.6], while reducing 0.52 calls and 35.6k input tokens. This is the strongest evidence supporting the core intervention.

  2. Diagnostic experiments reveal meaningful evaluation bias. Location feedback increased the scanner pass rate by 8.8 percentage points, while the joint pass rate remained essentially unchanged, and circular dependencies increased from 7.2% to 14.8%. Further ablation of resource identifiers and types enhances the interpretability of the results.

  3. Simultaneous improvement in quality and runtime costs, and more standardized experimental reporting. Using the same initial draft and call limits in the main comparison, the method improved the joint pass rate while reducing the number of input tokens from 230.1k in Reflexion to 120.4k. Clear descriptions of paired statistics, handling of unknown results, and cost boundaries improve the reliability of the results.

Critical weaknesses
  1. Lack of task validation independent of development process. All 153 requirements were previously developed, and manual mapping, constructors, and prompts may have been affected by this task distribution. Freezing the process before runtime cannot eliminate previous adaptation biases, and bootstrap cannot measure these biases. The lack of evaluation of new requirements after freezing the system means the core value of "cross-task reusability" has not been fully validated.

  2. The reusability and adaptation value of compilation interfaces has not been quantified. Existing ablation studies demonstrate the execution value of a given constructor library, but do not adequately explain which constructors are automatically derived from rules, or how much manual modification is required to adapt to new rule combinations. Coverage, success, and rejection statistics by constructor family are also insufficient. Furthermore, the lack of a Reflexion+ comparison with the same library limits the assessment of the source of the full method's advantages.

  3. Static pass is insufficient to confirm task repair completion. S3 log construction relies on the caller providing existing external buckets and permissions, but experiments did not verify whether these conditions are guaranteed by the original requirements. The Joint gate also does not check the preservation of full requirements. Therefore, it is currently unclear how many of the newly added successful repairs are effective fixes and how many still depend on unmet external prerequisites.

Questions for authors
  1. After freezing all components, how does it perform on new requirements not developed by the developers? How many tasks do the six types of constructors cover, and what are the benefits for uncovered tasks?

  2. Which constructors are automatically derived, and which rely on manual addition? How many modifications are needed to adapt to new rule combinations? What are the results of Reflexion after obtaining the same library?

  3. How many new external resources or permission prerequisites are added to the accepted templates? After verifying these prerequisites and the original requirements, do the remediation benefits still hold?

  4. The main body guard checks the protected paths and adds lint diagnostics; how does the actual implementation establish the stronger prerequisite of "maintaining previously satisfied checker predicates" required by Appendix B.3?

Reviewer_Wnr8OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the task of repairing infrastructure-as-code tasks that are derived from AWS CloudFormation scenarios (that are specific to AWS services). The authors propose a pipeline consisting of a compilation and runtime mechanism that outperforms techniques such as self refine, and reflexion. Additional ablations show that adding locations of faults improves the performance of the pipeline.

Critical strengths

None

Critical weaknesses

The paper is extremely difficult to follow.

I have the following concerns:

  1. Writing and experiments: The paper studies context curation for infrastructure-as-code tasks. I would like to understand the task setting, with clear notation and research questions that are answered through the experimental pipeline. In the current version of the paper, the experiments in the paper are disjoint and don't provide clarity into the contributions. Tables 2 and 3 talk about the importance of providing location information for repair, and Table 4 shows the comparison of the pipeline with the established baselines without providing insights if the location information is used only by the propose technique or also by the baselines.
  2. There is leakage of the dataset used to develop and evaluate the technique since the same 153 tasks are used for both stages. Further, the final metric is also flawed since it does not use successful AWS deployment as its principal acceptance condition. Its main endpoint is joint Checkov + cfn-lint acceptance.
  3. There is limited evaluation. The paper only measures the performance of Claude Haiku in the main paper and adds results with missing statistics (in terms of token costs) in the appendix (Table 9, Page 18) without any mention of it in the main paper. Additionally, baseline results using Opus and Fable (on page 18) beat the proposed technique. Since they show positive results only for Haiku, and worse results for LLMs such as Opus and Fable, it is difficult to judge the validity of the proposed approach.
Questions for authors

n/a

Reviewer_T9ETOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the limitations of LLM-based Infrastructure-as-Code (IaC) repair and proposes a guarded compilation framework that converts security rule metadata and resource schemas into reusable resource constructors. The authors first conduct a diagnostic study showing that providing resource-level location information improves scanner acceptance but does not improve joint scanner-and-lint acceptance, partly due to structural errors such as circular dependencies. Based on these observations, the proposed framework executes precompiled resource constructions under structural guards before delegating remaining violations to LLM-based refinement. Experiments on 153 AWS CloudFormation tasks evaluate the proposed method against existing LLM-based repair approaches in terms of joint scanner-and-lint acceptance and refinement efficiency.

Critical strengths
  1. The paper provides a systematic diagnostic study showing how feedback localization affects scanner acceptance and structural validity. The proposed framework combines rule-based resource construction with structural guards to address IaC repair failures. The evaluation includes controlled ablation studies that distinguish the effects of library context and execution, demonstrating improvements in joint acceptance and reduced LLM calls.
Critical weaknesses
  1. Development and evaluation separation All 153 evaluation tasks were inspected during development, and the compiler relies on manually authored mappings and patches. Since the same tasks are used for evaluation, it is unclear whether the reported improvements generalize beyond the development benchmark or reflect benchmark-specific engineering. Evaluation on unseen tasks with a frozen pipeline is needed.

  2. Incomplete efficiency evaluation scalability evaluation The efficiency analysis excludes manual library authoring and end-to-end execution overhead. Equal model calls do not ensure comparable computational budgets, and the absence of token matched or price-adjusted comparisons limits the assessment of efficiency gains. Furthermore, the compiler supports only six manually authored constructor families, without reporting their coverage of blocking findings or the effort required to extend the library to new rules and platforms. These omissions make it difficult to assess the practical efficiency and scalability of the approach.

Questions for authors
  1. Can the authors evaluate the frozen pipeline on unseen tasks to assess generalization beyond the development benchmark?
  2. Can the authors report the coverage of the six constructor families, the effort required to extend the library, and end-to-end latency and price adjusted efficiency comparisons under matched computational budgets?
  3. Can the authors compare against Reflexion with the same constructor library and a structured IaC repair baseline, and provide constructor family-level results to clarify whether the gains primarily come from resolving same type resource obligations?
  4. Can the authors extend the evaluation to commercial LLMs, to assess whether the proposed approach generalizes to stronger proprietary models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.