- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper clarifies ambiguous infrastructure requests through binary questions about candidate resources, topologies and attributes, followed by pruning and regeneration. AMBIG-IAC contains tasks with plan-checked reference configurations; experts audit ambiguous requests. Experiments evaluate configuration recovery and cost across models and question budgets.
Critical strengths
- Clarifying infrastructure intent is a useful problem, and the resource/topology/attribute decomposition gives question selection an explicit, interpretable basis.
- Structural gains and lower token use and latency than ATD at support the framework’s usefulness. Multiple models and budgets, output-generation controls, and simulated-user audits strengthen the evaluation.
Critical weaknesses
- The metrics can miss incorrect attribute-to-resource assignments. pools items across instances, while Structure Score ignores attributes (§5; Appendix C). Swapping port/CIDR settings between same-type security groups attached to different roles can therefore yield perfect scores for an incorrect configuration. This is a constructed blind spot; its frequency in actual outputs is unknown. An instance-aligned assessment is needed to substantiate recovery of the target’s attributes.
- Structural consensus can end clarification before attributes are resolved. Section 4 deduplicates candidates by resource types and topology and stops when structurally distinct candidates cannot be generated. This can remove the attribute-only alternatives needed for further questions. The tradeoff is acknowledged, but its impact on attribute recovery has not been isolated, leaving uncertainty about whether the design supports clarification across all three intended axes.
Questions for authors
- How do attribute comparisons change under a topology-consistent resource-instance alignment, particularly for repeated resource types?
- How are unresolved attribute-only alternatives retained once candidates agree structurally? What changes if deduplication preserves meaningful attribute differences?
3 comments
Response to Reviewer jz7C (Part 2)
Q2. Impact of Attribute-Aware Candidate Preservation
We tested the impact of preserving attribute-only alternatives. To investigate Q2, we implemented a variant that preserves candidates with meaningful per-instance attribute differences, while retaining the existing exclusions for names and identifiers. Disagreement extraction considers attribute values across all instances of each resource type and ranks them by partition entropy. The remaining framework is unchanged. On GPT-4o-mini, the mean pool sizes increase from 5.8/5.2/5.0 to 6.9/6.8/6.8, containing more candidates with attribute differences.
Table R2. GPT-4o-mini results under the instance-aligned metric. Bold marks the highest mean across all methods, including Submitted.
| Method | Structure K=5 | K=10 | K=15 | Aligned R+A F1 K=5 | K=10 | K=15 |
|---|---|---|---|---|---|---|
| Direct Q Generation | 43.08 | 44.38 | 46.31 | 16.73 | 17.52 | 17.21 |
| Best-of-N | 41.28 | 43.52 | 43.12 | 17.54 | 17.73 | 18.71 |
| Self-Consistency | 38.59 | 38.41 | 38.19 | 14.88 | 15.44 | 15.52 |
| ATD | 42.23 | 41.89 | — | 21.15 | 22.68 | — |
| Ours (attribute-preserving) | 49.08 | 52.68 | 53.69 | 21.46 | 22.36 | 24.46 |
| Ours (submitted) | 48.95 | 50.98 | 54.84 | 20.81 | 22.64 | 24.78 |
The main comparative findings remain, but overall attribute gains are limited. The variant still exceeds all baselines on Structure and the three lightweight baselines on aligned R+A F1 at every budget. Differences from ATD on R+A F1 are not significant. Compared with Submitted, among reference attribute items missed by the variant, 62–68% belong to resource types absent from the final output, suggesting that resource recovery remains an important limitation.
Takeaway. Attribute-aware candidate preservation restores attribute-level clarification without significantly improving overall R+A F1. We suspect that better attribute question selection alone may be insufficient: attribute recovery also depends on generating the intended resources and attribute values. Moreover, even with attribute-aware candidate preservation, attribute questions still compete with resource and topology questions under a fixed interaction budget. A more adaptive strategy could prioritize structural clarification until a convergence threshold is reached, then allocate more questions to unresolved attributes. We see this as a promising direction for future work.
Friendly Reminder
Dear Reviewer,
Thank you for recognizing the importance of the problem and our contributions to both the method and the benchmark. We have conducted additional experiments to address your concerns and hope you will have a chance to review our rebuttal. We would greatly appreciate any further feedback!
Response to Reviewer jz7C
Thank you for the thoughtful review. We investigated both concerns through implementation changes and additional evaluations. For Q1, we corrected the metric and re-scored all submitted outputs and baselines. The revised metric detects the reported attribute-matching errors, while our structural and attribute gains remain. For Q2, we implemented and tested an attribute-preserving variant across question budgets to quantify the impact of structural deduplication. The variant enables more attribute-level clarification, but does not significantly improve overall R+A F1.
Q1. Topology-consistent attribute comparisons
We revised the metric and re-scored all results. R+A F1 now credits an attribute only on the generated resource instance aligned to the corresponding reference instance. We retain the item definitions of Appendix C.4, but replace pooled matching with instance-aligned matching and account for every repeated nested-block element.
The alignment starts from the node correspondence used by the Structure Score's graph edit distance (GED), ensuring topology consistency. Since multiple correspondences may have the same edit cost, we break ties in favor of attribute agreement. Specifically, we iteratively swap the matched node-pairs of same-type resource instances whenever doing so increases the number of matched attributes without increasing the edit cost, until no further improvement is possible. Within each aligned resource pair, repeated nested-block elements, such as ingress rules, are matched one-to-one to maximize matched attribute leaves, recursively. The submitted implementation retained only one element per repeated block; the revised implementation counts every element. Re-scoring actual outputs lowers each method's R+A F1 by 0.6–2.8 points, primarily by removing credit previously assigned across resource instances or nested-block elements.
The structural advantage and attribute gains over the three lightweight baselines remain. Table R1 reports the submitted GPT-4o-mini runs under the revised metric. Structure Score is unchanged. Table R1. Submitted runs under the revised metric. The difference between ATD and Ours is not significant under paired Wilcoxon tests with Holm correction within each setting.
Our method significantly exceeds Direct Question Generation, Best-of-N, and Self-Consistency on both metrics at every budget shown. The differences from ATD on aligned R+A F1 are not statistically significant, but at a much higher cost. This is consistent with the original findings in the paper.