acceptodds
Under review as a conference paper at ICLR 2027

Counterexample-Guided Synthesis of Skill Applicability Contracts with Large Language Models

Abstract

Agents reusing tool-calling skills need to know when they succeed and how they change the environment. We synthesize such applicability contracts from a supplied DSL and execution feedback, with the reference contract withheld. Our counterexample-guided inductive synthesis (CEGIS) loop partitions reachable states by the proposed contract's accept/reject predictions, executes the skill on both sides, and revises the contract from structured counterexamples. We evaluate synthesized contracts exhaustively over four core domains, three synthetic and one built on third-party tool code; a run counts as exact if its final contract is correct on every reachable state. Gemma-4-31B is exact in 20 of 20 seeds on Deployment and Calendar, against 0 and 3 for direct generation. On a finite -bench retail fixture it is exact in 19 of 20, against 0 for direct generation and 3 for fixed-pool refinement. A common-loop factorial study of selection and termination retains the partitioning advantage for Gemma under both stopping rules after multiplicity correction; Qwen3.8 shows weaker, non-significant differences. Relational coverage has mixed effects across conditions. A Shopping difficulty study distinguishes recovery on a deliberately relaxed variant from failure when hidden rules become independently identifiable. Muse-Glimmer-30B is also exact in 20 of 20 seeds on Deployment and Calendar under its native low-reasoning setting. For local models, evidence chosen without the candidate fails mostly by false accepts; in local and hosted cohorts alike, the partitioned loop's residual failures are mostly false rejects hidden in the candidate's large predicted-reject region.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
LLM Agents, Tool Use, Skill Applicability, Counterexample-Guided Inductive Synthesis, Program Synthesis, Domain-Specific Languages

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_NMCyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies synthesizing a skill’s precondition and postcondition in a supplied typed DSL, using execution feedback while withholding the reference contract and environment source from the proposing model. Active-CEGIS repeatedly partitions available reachable states into predicted accepts and rejects, selects approximately equal numbers from each partition, executes the target skill, and revises the contract using structured counterexamples. A relational-diversity heuristic optionally guides selection within partitions.

Critical strengths

The paper investigates important failure modes. The identifiable Shopping experiment demonstrates that perfect relaxed-fixture performance need not imply recovery of the underlying policy. The W2 retail experiment reveals order-specific overfitting and a supplied-predicate defect. The postcondition audit identifies semantically incomplete contracts that nevertheless satisfy the primary exactness criterion.

Follow-ups meaningfully address several weaknesses. Candidate-pool sensitivity includes evaluation on the full closure after synthesis from smaller pools. Two additional Deployment configurations reproduce Gemma’s advantage. Strengthened retail effects preserve the 19/20 versus 3/20 Active–Fixed-pool result. These are useful additions, although their interpretation must remain protocol-specific.

Critical weaknesses

The methodological advance is relatively narrow, and the need for an LLM is insufficiently established. The synthesis loop follows established CEGIS, while the main effective addition is sampling both sides of the current Boolean prediction. Structured transition differences are useful engineering, but their contribution is evaluated jointly with the loop. Coverage has mixed effects and often provides no improvement over partitioned uniform selection.

Several headline baselines are informative controls but do not isolate the claimed mechanism fairly.

The execution budget relies on a strong, costly state-access assumption. Active selection scans candidate predictions over an already enumerated reachable-state pool. Selected immutable states can then be supplied directly to the target skill, without replaying the trajectories needed to reach them. This is a legitimate experimental setup, but materially easier than obtaining useful states online in a deployed agent environment.

Balancing predicted partitions does not ensure coverage of the errors that matter. Allocating half the batch to predicted rejects is helpful relative to ignoring that region, but offers little protection when the region is large and contains only a few false rejects. The residual-error analysis documents this problem in both local and hosted cohorts. Sampling can therefore satisfy a candidate while missing rare successful states.

Questions for authors

My concerns are the incremental methodological contribution and the gap between successful contract assembly under supplied structure and reliable inference of undisclosed rules. The identifiable Shopping failure, permissive primary postcondition criterion, and pre-enumerated state-access assumption make that gap consequential.

2 comments
AuthorsOpenReview ↗

Response to NMCy: Candidate Partitioning, Symbolic Synthesis, and State Access

Our contribution is candidate-partitioned state selection within CEGIS.

  1. Selection mechanism

The selector samples both candidate-predicted accepts and rejects without reference labels; coverage is auxiliary.

The common-loop factorial fixes initial candidates and the structured-feedback interface, disables coverage, and crosses partitioned/uniform selection with stopping on a clean batch/auditing the remaining budget. Requested batch sizes match; realized sizes and timing follow each policy's trajectory. For Gemma-4-31B (20 trials/fixed environment, failures included):

  • Deployment: partitioned vs. uniform reaches 20/20 vs. 1/20 under both stopping rules.
  • Retail: 19/20 vs. 2/20 (stop), and 19/20 vs. 3/20 (continue).

All four Gemma comparisons have adjusted p<0.001 within the 16-comparison Holm family constructed during analysis. Qwen3.8 comparisons are non-significant after correction. These comparisons support the selection-policy effect for Gemma under the tested trajectories.

  1. Non-LLM synthesis comparison

We replayed Gemma's delivered state transitions in restricted Deployment with both valid-option lists and postcondition examples hidden (V−P−): 20 Active traces and 20 Fixed-pool traces, spanning 139 request boundaries. Fixed-pool supplies a median of 1 observation; 4 traces supply none.

Trace sourceGeneratorMean IoU (%)Mean CFR (%)Exact
ActiveLLM41.610.000/20
ActiveSymbolic1.461.460/20
Fixed-poolLLM37.780.000/20
Fixed-poolSymbolic0.640.290/20

The symbolic learner seeks a minimal observation-consistent precondition in a finite grammar using public types and the full option vocabulary. Its postcondition grammar assigns an expression to each of 7 primitive fields, excluding derived quota_size invariants. Development checks confirmed expressibility of the reference's 8 precondition atoms and 7 postcondition equations. Synthesis receives delivered observations and public vocabulary, with the reference contract and full-closure labels withheld. Exhaustive scoring follows synthesis.

This replay compares generalization from fixed, LLM-collected transitions. Supporting information and hypothesis spaces differ: diagnostics, prior candidates, and pretrained priors versus a finite grammar.

Symbolic synthesis is consistent with all 40 observation traces, including the 4 empty traces, yet over-accepts unseen states (Active-trace median FA: 5,951 vs. 110 for the LLM). The LLM generalizes preconditions better (41.61% vs. 1.46% IoU). Under its 7-field grammar, symbolic Active-trace postconditions distinguish all 88 normal successful transitions from 2,464 audited faulty effects; the LLM's invalid quota_size invariant instead fails normal-transition checks, yielding CFR 0%. Symbolic observation fitting succeeds; applicability generalization and effect synthesis fail differently.

  1. Supplied structure and information removal

Gemma's restricted Deployment Active has 100% IoU with options disclosed (V+) and about 42% hidden (V−), but Exact 0/20 across all four original V/P conditions because of the postcondition invariant.

Retail retains effect examples/predicates: original Active IoU/CFR is 21.11% (V+) vs. 43.61% (V−), and 47.54% vs. 46.47% after post-hoc budget completion; Exact remains 0/20. The original advantage therefore favors V+ in Deployment and V− in Retail.

Separately, removing only Deployment's postcondition examples while retaining the original DSL preserves Active 20/20 Exact on normal transitions. Faulty-effect rejection drops 100%→82.14%, and joint passage drops 20/20→0/20. In identifiable Shopping, new Full Active remains 0/20 Exact; budget completion raises mean IoU/CFR from 24.81%/19.27% to 31.52%/27.12%.

  1. State access and rare errors

We assume a pre-enumerated finite pool and direct execution from selected states. The intended use is offline contract synthesis in a resettable sandbox before agent deployment. B counts distinct feedback states. Pool construction, candidate-prediction scans, repeated re-scoring, and online reachability/reset costs are separate; original Deployment/Gemma Active performs 96 audits over B=48 distinct states.

Rare successful states can still be missed inside a large predicted-reject region. Completing 7 Retail runs with unsent final requests delivered the first false-reject counterexample in 2 runs: FR fell 8→4 while FA rose 0→6. The other 5 retained their errors. This separates finding evidence, delivering it, and revising the contract consistently.

We will clarify this offline scope in Section 3 and add the generator comparison to the appendix.

Materials and code

Reviewer_NMCyOpenReview ↗

Response

Thank the author for the rebuttal. After reading the rebuttal, I maintain my score, as I am still not convinced by the novelty.

Reviewer_UDy3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper synthesizes pre/postcondition "applicability contracts" for tool-calling skills with a CEGIS loop: an LLM proposes a contract in a typed JSON DSL; the loop partitions reachable states by the contract's accept/reject predictions, executes the skill on a sample from both sides (B = 48 states), and revises from up to three transition-diff counterexamples (Q = 4 calls). Exactness is checked exhaustively over finite deterministic closures in four domains (Deployment, Calendar, relaxed Shopping, τ-bench retail). Across eight local and four hosted models, the paper finds that candidate partitioning — not coverage — drives exact recovery (Table 5, Tables 11–13), reports Gemma-4-31B exact in 20/20 seeds on Deployment and Calendar, and documents a failure case (identifiable Shopping variant, Sec. 7). An 11,119-run artifact with a hash manifest and verification scripts accompanies the submission.

Critical strengths
  1. The artifact reproduces the paper, and the main-path evaluation is oracle-isolated. From the raw run JSONs I recomputed Tables 2–5, 8, 11–14, 17, 18, 20–22, 28 and Appendices A.5/D/J/N/O/Q/R; all match. All 11,440 manifest hashes verify. All 136 main-table cells record oracle_feedback_queries = 0, and the truncation-retry set (58/2,720 and 52/1,080 runs) is exactly the set of runs with finish_reason = length, with limits matching App. D. This is the most verifiable submission I have reviewed; the selector claim (Tables 2–5, 11–13) stands on it.

  2. Candid limitation reporting. App. S classifies every reference clause by prompt support, App. T shows relaxed Shopping cannot distinguish the reference from a six-clause weakening, and Sec. 7 is a well-designed negative result. This makes the weaknesses below fixable rather than fundamental.

Critical weaknesses
  1. The basic-Shopping "ceiling" comparison is not the method. Sec. 6.4 (p.8): "Active reaches 20/20 versus at most 6/20 for the reduced literature-inspired controls"; Table 9 (p.23) lists "Active-CEGIS 20" and "Active-CEGIS, Qwen2.5-14B 13." The authors' analysis/cell_manifest_2026-09-04.json maps both rows to method counterexample_guided; experiment/modeling/runner.py score() (L507–515) is documented as "Run the oracle evaluator over the whole enumerated state set," and every artifact in those cells records states_checked = 4752, oracle_feedback_queries = 2–4, stopped_because = "contract_is_exact", while the controls in the same table have oracle_feedback_queries = 0. The paper never identifies these rows as oracle-guided. This is the only comparison with literature baselines in the paper, so it is consequential; the mitigation is that a sampled Active-CEGIS run at B = 32 also reaches 20/20 for Qwen2.5-32B (no sampled counterpart exists for 14B).

  2. "Reference contract withheld" (abstract, Sec. 1/3) is stronger than what the prompts do. POSTCONDITION_PATTERNS in experiment/deployment/prompts.py L55–67 lists, as "patterns," the five clauses of the reference postcondition in App. A.3; calendar/prompts.py L41–51 and taubench_retail/prompts.py L45–55 do the same, and the Self-Refine critique prompt (L150–157) names all six precondition topics. App. S (p.38) acknowledges that "substantial effect structure is supplied," but the main text does not, and it changes how the exact/CFR metrics should be read in three of four domains. Related completeness gaps: the Sec. 7 negative result was run with the No-Coverage ablation (active_cegis_no_coverage, whose config calls it "the method's main arm"); the "doubling budgets 0/20" and "Qwen3.8 0/20" cells and the Figure 5 budget sweep (B = 8/16/24) have no artifacts despite p.10's "every experimental artifact cited" statement; and the FA median reported as 12 recomputes to 20.

Questions for authors

Clarification (answers would change my assessment):

  1. Are the Table 9 "Active-CEGIS" rows the counterexample_guided method with full-closure oracle feedback? What are sampled Active-CEGIS results at matched budget (B = 24) for Qwen2.5-32B and 14B?
  2. Has any main-domain cell been run without the POSTCONDITION_PATTERNS block? How do the Table 2/3 exact counts and CFR change?
  3. Was full Active-CEGIS run on the identifiable variant? Please provide the 2× budget and Qwen3.8 artifacts and the FA-median computation.
  4. Were the Qwen2.5-32B Deployment Direct/Self-Refine cells served on the vLLM 0.8.5/V100 stack while Active/Random/Fixed-pool used 0.19.1/A100 (as the artifacts record)? The paper mentions this caveat only for 14B.

Suggestions: add the Figure 5 sweep runs and cited analysis files to the archive; mark retry-affected cells in the main tables (e.g., Qwen3.6 Deployment Active 14→20) and complete Table 16 with the Qwen3.6 OFF rows (released: Active 3/20, Random 4/20); flag the nine runner_error crashes entering Table 3 as IoU = CFR = 0; name Figure 7's domain; check that the anonymous repository's commit metadata is anonymized.

2 comments
AuthorsOpenReview ↗

Response to UDy3: Table 9 Correction, New Controls, and Artifact Audit

Thank you for identifying the labeling and archive issues. We checked the raw records and added the requested controls. Each cell below has 20 trials on a fixed environment, with failures in the denominator.

  1. Table 9 and sampled B=24

The two Table 9 rows labeled Active are historical counterexample_guided runs using full-closure oracle feedback during synthesis: 2 queries/run for Qwen2.5-32B and 2–4 for 14B. With the same full-oracle access, both this arm and contractskill_full_oracle reach 20/20 on 32B.

New sampled Active B=24 runs in basic Shopping make zero full-closure oracle queries in all 40 trials:

ModelExactMean IoU (%)Mean CFR (%)
Qwen2.5-14B13/2073.3371.83
Qwen2.5-32B20/20100.00100.00

14B ran on V100/FP16/TP4/vLLM 0.8.5.post1; 32B on A100/BF16/TP1/vLLM 0.19.1. Each row measures oracle-free recovery within that model's serving setup, including budget/truncation failures.

  1. Removing POSTCONDITION_PATTERNS

For Gemma-4-31B in Deployment, we removed only the example block, retaining the original DSL, public operator semantics, and environment. Cells show Exact / mean CFR:

PatternsDirectFixed-poolActive
Present0/20 / 70.97%3/20 / 75.32%20/20 / 100.00%
Removed0/20 / 56.44%3/20 / 75.32%20/20 / 100.00%

Normal-transition Exact remains 20/20 for Active, while effect specificity decreases. Contracts omit the clause clearing allocated resources; rejection of audited faulty effects falls from 100% to 82.14% (2,024/2,464). Contracts passing normal transitions AND all audited faulty effects fall from 20/20 to 0/20. The two evaluations distinguish valid-execution consistency from postcondition strength.

  1. Identifiable Shopping

The historical arm is No-Coverage. FA median is 20 at the base budget and 12 at twice the budget, computed over 18/20 valid contracts in each cohort; Exact is 0/20 over all trials.

We added Full Active and No-Coverage, 20 Gemma trials each. Post-hoc continuation reused initial candidates and prior responses, holding B48/Q4 fixed. Total token budget increased from 16,000 to 32,768; the 3 remaining final-response truncations then received a 3,072 output cap, preserving the other 37 expanded-budget results.

Method/policyValidExactMean IoU (%)Mean CFR (%)
Full Active, original19/200/2024.8119.27
Full Active, completed20/200/2031.5227.12
No-Coverage, original20/200/2024.9418.50
No-Coverage, completed20/200/2030.9025.46

Full Active reproduces the Exact failure; budget completion improves CFR by 7.85 and 6.96 percentage points, respectively.

  1. Materials and audit corrections
  • All 542 missing files are now published: 20 historical Gemma doubled-budget identifiable runs, 40 Qwen3.8 identifiable Direct/No-Coverage runs, 480 Figure 5 B8/16/24 runs, and all_cells.csv/metrics_detail.pdf. The 60 older duplicate Retail Sampled runs are excluded. We verified source/release hashes, run aggregates, and the downloaded ZIP.
  • 32B Deployment: Direct/Self-Refine used vLLM 0.8.5.post1 without initial-request response_format; Random/Fixed-pool/Active used 0.19.1 with it. Fixed-pool and Active match on these serving/request settings; Direct comparisons include that difference.
  • Post-hoc length retries cover 58 runs/24 main cells and 52 runs/8 ablation cells, retaining 20 trials/cell. Table 5 Active reuses main-table replacements; Qwen3.6 ON Deployment Active changes 14→20.
  • Table 16, historical Qwen3.6 OFF Deployment: Direct/Self-Refine/Random/Fixed-pool/Active/No-Balance/No-Coverage/Uniform Exact counts are 0/0/4/0/3/1/4/1, each out of 20.
  • Of 9 historical runner errors, 4 are in Table 3's methods: Direct seed13; Self-Refine/Fixed-pool/Active seed15. These count as failures with IoU=CFR=0; 5 belong to Random/old Sampled. Missing cost records mean unlogged cost.
  • Figure 7 shows reasoning-OFF relaxed Shopping. The 3 commits from 16815af to root have Anonymous author/committer names and example.invalid emails.

In the revised manuscript, the Abstract and Sections 1/3 will specify the supplied DSL, option information, and effect patterns alongside the withheld reference contract. Table 9 and Section 6.4 will distinguish historical oracle-guided runs from the new sampled B=24 results. We will document the 32B serving differences and incorporate the Table 16, Figure 7, and runner-error corrections above.

Materials and code

Thank you for the comprehensive response and for publishing the missing artifacts. The relabeling of Table 9 together with the new sampled B=24 runs (14B 13/20, 32B 20/20, zero oracle queries) resolves my first concern: the comparison now stands on the method as described. The pattern-removal experiment is a candid and informative result — normal-transition exactness is unchanged, but the postcondition-strength audit drops from 20/20 to 0/20, which confirms that the supplied effect patterns carry real information; the planned rewording of the abstract and Sections 1/3 is the right fix. The Full-Active runs on the identifiable variant, the FA-median clarification, and the serving-stack, Table 16, and runner-error corrections address my remaining questions.

Given that both critical weaknesses have been answered with new experiments and the artifact is now complete, I am raising my rating to weak acceptance.

Reviewer_M5qAOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

An LLM writes a contract for a skill. The contract states when the skill succeeds and what it changes. The LLM gets a rule language, not the reference contract. A loop then tests the contract. The loop splits states by predicted success. It runs the skill on both groups. The LLM revises the contract from wrong predictions. The experiments use four domains: Deployment, Calendar, a slice of tau-bench retail and price-aware Shopping. Gemma-4-31B gets 20 of 20 exact runs on Deployment and Calendar. Without the loop, it gets 0 and 3.

Critical strengths

Final scoring covers every reachable state.

The split is the most important part of picking states. In Table 5, removing the split lowers exact runs in every nonzero row without a star. In Figure 2, Gemma-4-31B gets 20 of 20 on Deployment with the split. Random picking (Uniform) gets 1 of 20 under both stopping rules.

Critical weaknesses

On Deployment, Calendar and retail, exact runs show assembly of given parts. They do not show discovery of hidden rules. Appendix S makes the same distinction. These three domains have no hidden rules. Their prompts list the valid options. Calendar and retail also give ready-made checks for four rules each. Only price-aware Shopping has hidden rules. There, exact runs happen only on the relaxed version. The relaxed version also counts a looser contract as exact. This weakness counts most for my rating. The main claim needs the loop to find hidden rules.

The authors made two choices after seeing the first results. They retried cut-off runs. They grouped some tests for p-value correction. On relaxed Shopping, two Qwen3.8 tests become significant only after the retry. Fixed-pool reuses one set of states. Against Fixed-pool, the corrected p-value falls from 0.227 (Table 10) to 0.0386 (Section 6.3). Against Uniform, it falls from 0.359 to 0.043. Tables 13 and 28 report later runs with tests chosen in advance. There, Qwen3.8 shows no significant gain. Gemma-4-31B still beats Uniform and Fixed-pool. Many cells carry labels such as historical, legacy, starred or retry. These labels make the paper hard to follow.

Questions for authors

The authors should hide the valid options in Deployment and retail. They should then count exact runs again. This test answers the first weakness.

The paper should mark which cells in Tables 2 to 5 changed after the retry.

1 comment
AuthorsOpenReview ↗

Response to M5qA: Hidden-Option Experiments and Retry Accounting

Thank you for highlighting the distinction between assembly from supplied structure and discovery of undisclosed rules. We tested option hiding in Deployment and Retail and audited retry accounting.

  1. Hidden valid options

Gemma-4-31B receives allowed/disallowed lists in V+ and only the full option vocabulary in V−. We use neutral identifiers, block DSL access to private valid-option sets and policy-dependent selector features, and retain public types, operator semantics, and execution feedback. Each cell has 20 trials on a fixed environment, including parsing and truncation failures.

In Deployment, we crossed option disclosure (V) with postcondition-example disclosure (P), holding the restricted DSL, environment, and selector fixed. Original Active results:

ConditionMean IoU (%)Mean CFR (%)Exact
V+P+100.000.000/20
V+P−100.000.000/20
V−P+42.000.000/20
V−P−41.610.000/20

Option disclosure recovers the applicability boundary; all four conditions retain an erroneous quota_size postcondition invariant. Truncation affected 36/40 V+ Active runs and 0/40 V− runs. A separate higher-budget continuation of those 36 resolved all truncation/parsing failures: 1 became Exact, while 35 retained the invariant error. All adjusted p-values in the predeclared Exact-testing families equal 1. The information effect appears in descriptive precondition IoU, alongside a common postcondition error.

Retail retains effect examples and domain-specific effect predicates while varying V. Its post-hoc budget-completed cohort continues 15 truncated runs and 7 runs with an unsent refinement request, preserving the other 98 runs. Every cell remains at Exact 0/20; mean IoU equals CFR:

ConditionDirect (%)Fixed-pool (%)Active (%)
V+, original14.7126.4321.11
V−, original14.7127.0843.61
V+, budget-completed14.7131.4347.54
V−, budget-completed14.7132.0846.47

In the original Retail Active cohort, V+ had 10 truncated runs versus 1 for V−; unsent requests numbered 2 versus 5. The original IoU advantage favors V+ in Deployment but V− in Retail. After budget completion, Retail Active scores are close (47.54% vs. 46.47%). These experiments separate applicability recovery from postcondition correctness and show domain-specific responses to information removal. The restricted-DSL Deployment comparison is separate from the pattern-only experiment, which retains the original DSL.

  1. Retry accounting and selection evidence

The retry policy was introduced post hoc. Within its selected scope, every finish_reason=length run was replaced regardless of outcome: 58 runs across 24 main-table cells and 52 across 8 ablation cells. Denominators remain 20; Table 5 Active reuses main-table replacements. The main table has 16 Exact gains and 2 regressions; ablations have 11 gains and 0 regressions. Changes include reasoning-ON Deployment Active: Qwen3.6 14→20 and Gemma-26B 6→11; Qwen3.8 relaxed Shopping Active: 11→14. Under this post-hoc policy, Qwen3.8 reaches adjusted p=0.0386 against Fixed-pool and p=0.043 against Uniform. The release includes cellwise replacements and before/after metrics.

The separate common-loop factorial shares initial candidates, fixes the structured-feedback interface, disables coverage, and crosses partitioned/uniform selection with stopping rule. For Gemma, partitioned vs. uniform reaches 20/20 vs. 1/20 in Deployment under both rules; Retail reaches 19/20 vs. 2/20 when stopping on a clean batch and 19/20 vs. 3/20 when auditing the remaining budget. All four Gemma comparisons have adjusted p<0.001 in the 16-comparison Holm family constructed during analysis; Qwen3.8 comparisons are non-significant after correction.

Requested batch sizes match; realized sizes and feedback timing follow each policy's trajectory. Under these conditions, partitioned selection outperforms uniform selection for Gemma in both domains and both stopping rules.

In the revised manuscript, we will mark retry-replaced cells in Tables 2–5, report pre-retry Exact counts alongside the updated counts in the appendix, and consolidate label definitions while distinguishing reasoning settings and retry policies.

Materials and code

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.