- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies synthesizing a skill’s precondition and postcondition in a supplied typed DSL, using execution feedback while withholding the reference contract and environment source from the proposing model. Active-CEGIS repeatedly partitions available reachable states into predicted accepts and rejects, selects approximately equal numbers from each partition, executes the target skill, and revises the contract using structured counterexamples. A relational-diversity heuristic optionally guides selection within partitions.
Critical strengths
The paper investigates important failure modes. The identifiable Shopping experiment demonstrates that perfect relaxed-fixture performance need not imply recovery of the underlying policy. The W2 retail experiment reveals order-specific overfitting and a supplied-predicate defect. The postcondition audit identifies semantically incomplete contracts that nevertheless satisfy the primary exactness criterion.
Follow-ups meaningfully address several weaknesses. Candidate-pool sensitivity includes evaluation on the full closure after synthesis from smaller pools. Two additional Deployment configurations reproduce Gemma’s advantage. Strengthened retail effects preserve the 19/20 versus 3/20 Active–Fixed-pool result. These are useful additions, although their interpretation must remain protocol-specific.
Critical weaknesses
The methodological advance is relatively narrow, and the need for an LLM is insufficiently established. The synthesis loop follows established CEGIS, while the main effective addition is sampling both sides of the current Boolean prediction. Structured transition differences are useful engineering, but their contribution is evaluated jointly with the loop. Coverage has mixed effects and often provides no improvement over partitioned uniform selection.
Several headline baselines are informative controls but do not isolate the claimed mechanism fairly.
The execution budget relies on a strong, costly state-access assumption. Active selection scans candidate predictions over an already enumerated reachable-state pool. Selected immutable states can then be supplied directly to the target skill, without replaying the trajectories needed to reach them. This is a legitimate experimental setup, but materially easier than obtaining useful states online in a deployed agent environment.
Balancing predicted partitions does not ensure coverage of the errors that matter. Allocating half the batch to predicted rejects is helpful relative to ignoring that region, but offers little protection when the region is large and contains only a few false rejects. The residual-error analysis documents this problem in both local and hosted cohorts. Sampling can therefore satisfy a candidate while missing rare successful states.
Questions for authors
My concerns are the incremental methodological contribution and the gap between successful contract assembly under supplied structure and reliable inference of undisclosed rules. The identifiable Shopping failure, permissive primary postcondition criterion, and pre-enumerated state-access assumption make that gap consequential.
2 comments
Response
Thank the author for the rebuttal. After reading the rebuttal, I maintain my score, as I am still not convinced by the novelty.
Response to NMCy: Candidate Partitioning, Symbolic Synthesis, and State Access
Our contribution is candidate-partitioned state selection within CEGIS.
The selector samples both candidate-predicted accepts and rejects without reference labels; coverage is auxiliary.
The common-loop factorial fixes initial candidates and the structured-feedback interface, disables coverage, and crosses partitioned/uniform selection with stopping on a clean batch/auditing the remaining budget. Requested batch sizes match; realized sizes and timing follow each policy's trajectory. For Gemma-4-31B (20 trials/fixed environment, failures included):
All four Gemma comparisons have adjusted p<0.001 within the 16-comparison Holm family constructed during analysis. Qwen3.8 comparisons are non-significant after correction. These comparisons support the selection-policy effect for Gemma under the tested trajectories.
We replayed Gemma's delivered state transitions in restricted Deployment with both valid-option lists and postcondition examples hidden (V−P−): 20 Active traces and 20 Fixed-pool traces, spanning 139 request boundaries. Fixed-pool supplies a median of 1 observation; 4 traces supply none.
The symbolic learner seeks a minimal observation-consistent precondition in a finite grammar using public types and the full option vocabulary. Its postcondition grammar assigns an expression to each of 7 primitive fields, excluding derived
quota_sizeinvariants. Development checks confirmed expressibility of the reference's 8 precondition atoms and 7 postcondition equations. Synthesis receives delivered observations and public vocabulary, with the reference contract and full-closure labels withheld. Exhaustive scoring follows synthesis.This replay compares generalization from fixed, LLM-collected transitions. Supporting information and hypothesis spaces differ: diagnostics, prior candidates, and pretrained priors versus a finite grammar.
Symbolic synthesis is consistent with all 40 observation traces, including the 4 empty traces, yet over-accepts unseen states (Active-trace median FA: 5,951 vs. 110 for the LLM). The LLM generalizes preconditions better (41.61% vs. 1.46% IoU). Under its 7-field grammar, symbolic Active-trace postconditions distinguish all 88 normal successful transitions from 2,464 audited faulty effects; the LLM's invalid
quota_sizeinvariant instead fails normal-transition checks, yielding CFR 0%. Symbolic observation fitting succeeds; applicability generalization and effect synthesis fail differently.Gemma's restricted Deployment Active has 100% IoU with options disclosed (V+) and about 42% hidden (V−), but Exact 0/20 across all four original V/P conditions because of the postcondition invariant.
Retail retains effect examples/predicates: original Active IoU/CFR is 21.11% (V+) vs. 43.61% (V−), and 47.54% vs. 46.47% after post-hoc budget completion; Exact remains 0/20. The original advantage therefore favors V+ in Deployment and V− in Retail.
Separately, removing only Deployment's postcondition examples while retaining the original DSL preserves Active 20/20 Exact on normal transitions. Faulty-effect rejection drops 100%→82.14%, and joint passage drops 20/20→0/20. In identifiable Shopping, new Full Active remains 0/20 Exact; budget completion raises mean IoU/CFR from 24.81%/19.27% to 31.52%/27.12%.
We assume a pre-enumerated finite pool and direct execution from selected states. The intended use is offline contract synthesis in a resettable sandbox before agent deployment. B counts distinct feedback states. Pool construction, candidate-prediction scans, repeated re-scoring, and online reachability/reset costs are separate; original Deployment/Gemma Active performs 96 audits over B=48 distinct states.
Rare successful states can still be missed inside a large predicted-reject region. Completing 7 Retail runs with unsent final requests delivered the first false-reject counterexample in 2 runs: FR fell 8→4 while FA rose 0→6. The other 5 retained their errors. This separates finding evidence, delivering it, and revising the contract consistently.
We will clarify this offline scope in Section 3 and add the generator comparison to the appendix.
Materials and code