- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates a class of resource dependency problems when large language models repair Infrastructure-as-Code (IaC): satisfying the rule requirements of a certain resource may require creating another resource, which in turn is subject to further rule constraints. Using CloudFormation as an example, the authors propose pre-compiling supported resource constructs into reusable libraries, executing them under structure protection checks, and then having the model handle remaining errors.
The paper's three main contributions are:
-
Diagnosing the discrepancy between feedback and repair quality. Through feedback ablation, the authors found that resource location can improve scanner pass rates, but not the joint pass rate of scanner and lint. Further comparison of resource identifiers and types, and S3 resource layering, links this phenomenon to similar resource obligations in log buckets.
-
Proposing a repair process from rules to protected constructs. The method combines rule metadata, resource schemas, and manual mapping to construct six types of resource builders. These constructs are executed at runtime, checking protected paths and new structure diagnoses, and leaving residual issues to the model for revision.
-
Validating improvements in joint pass rates and runtime consumption. The author provides a relatively complete repair process and distinguishes the role of the constructor library context and the actual execution of the constructor library through the ablation of fixed prompt words.
On 153 shared draft tasks, Haiku 4.5 achieved a joint pass rate of 88.2%, higher than Reflexion's 80.4%, with the average additional calls decreasing from 3.42 to 1.78. Ablation analysis of fixed libraries and prompt words further shows a 4.6 percentage point increase in execution compared to providing only library context. The acceptance rate advantage did not persist on stronger models tested.
Critical strengths
-
Direct ablation evidence for execution gain. On 152 co-scored tasks, after fixing the construct library and prompt words, execution increased the joint pass rate by 4.6 percentage points, with a 95% CI of [1.3, 8.6], while reducing 0.52 calls and 35.6k input tokens. This is the strongest evidence supporting the core intervention.
-
Diagnostic experiments reveal meaningful evaluation bias. Location feedback increased the scanner pass rate by 8.8 percentage points, while the joint pass rate remained essentially unchanged, and circular dependencies increased from 7.2% to 14.8%. Further ablation of resource identifiers and types enhances the interpretability of the results.
-
Simultaneous improvement in quality and runtime costs, and more standardized experimental reporting. Using the same initial draft and call limits in the main comparison, the method improved the joint pass rate while reducing the number of input tokens from 230.1k in Reflexion to 120.4k. Clear descriptions of paired statistics, handling of unknown results, and cost boundaries improve the reliability of the results.
Critical weaknesses
-
Lack of task validation independent of development process. All 153 requirements were previously developed, and manual mapping, constructors, and prompts may have been affected by this task distribution. Freezing the process before runtime cannot eliminate previous adaptation biases, and bootstrap cannot measure these biases. The lack of evaluation of new requirements after freezing the system means the core value of "cross-task reusability" has not been fully validated.
-
The reusability and adaptation value of compilation interfaces has not been quantified. Existing ablation studies demonstrate the execution value of a given constructor library, but do not adequately explain which constructors are automatically derived from rules, or how much manual modification is required to adapt to new rule combinations. Coverage, success, and rejection statistics by constructor family are also insufficient. Furthermore, the lack of a Reflexion+ comparison with the same library limits the assessment of the source of the full method's advantages.
-
Static pass is insufficient to confirm task repair completion. S3 log construction relies on the caller providing existing external buckets and permissions, but experiments did not verify whether these conditions are guaranteed by the original requirements. The Joint gate also does not check the preservation of full requirements. Therefore, it is currently unclear how many of the newly added successful repairs are effective fixes and how many still depend on unmet external prerequisites.
Questions for authors
-
After freezing all components, how does it perform on new requirements not developed by the developers? How many tasks do the six types of constructors cover, and what are the benefits for uncovered tasks?
-
Which constructors are automatically derived, and which rely on manual addition? How many modifications are needed to adapt to new rule combinations? What are the results of Reflexion after obtaining the same library?
-
How many new external resources or permission prerequisites are added to the accepted templates? After verifying these prerequisites and the original requirements, do the remediation benefits still hold?
-
The main body guard checks the protected paths and adds lint diagnostics; how does the actual implementation establish the stronger prerequisite of "maintaining previously satisfied checker predicates" required by Appendix B.3?