acceptodds
Under review as a conference paper at ICLR 2027

ArgGYM: A Procedural, Engine-Verified Benchmark for Structured Defeasible Reasoning

Abstract

Recent progress in large language model reasoning has been driven by benchmarks and reinforcement learning environments with automatically verifiable rewards, particularly in mathematics, code, and formal logic. These settings make model accuracy easier to evaluate and optimize, but it remains unclear how far success under fixed problem specifications and stable evaluation criteria transfers to reasoning outside such domains. Real-world reasoning often proceeds under incomplete and revisable information: conclusions may be supported provisionally, defeated by counter-evidence, reinstated by further arguments, or revised when stronger reasons become available. Reasoning of this kind is generally referred to as defeasible reasoning. We introduce ArgGYM, a procedural benchmark and RLVR-compatible training environment for structured defeasible reasoning. ArgGYM decomposes this reasoning into twelve tasks and grounds task-specific scoring in a symbolic argumentation engine that computes the formal states used to evaluate model outputs. It includes a frozen benchmark of 1,440 verified instances across fifteen curriculum configurations, two argument preference orderings (weakest-link and last-link), and two set orderings (elitist and democratic), while the same generators and verifiers can produce fresh instances for evaluation that reduces dependence on static test sets and for verifiable-reward training. On the frozen benchmark, frontier and open-weight models show sharply different reasoning profiles: they can recover substantial parts of structured answers without solving the complete task, and performance declines in later curriculum configurations with longer dependencies and more interacting structures. We release the benchmark, generators, and verifiers for reproducible evaluation and RLVR training.

Primary area:
datasets and benchmarks
Keywords:
Defeasible reasoning, structured argumentation, computational argumentation, large language models, large reasoning models, reasoning benchmarks, procedural generation, reinforcement learning with verifiable rewards (RLVR)

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8aubOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents ArgGYM, a procedural benchmark for structured defeasible reasoning with engine-verified evaluation. Built on ASPIC+ and PyArg, it covers 12 tasks and provides 1,440 frozen instances plus a generator for new samples. Experiments across 18 model variants and multiple reasoning settings show that models can handle some local reasoning but struggle to recover fully correct argumentative states in complex structures. Performance also varies across preference regimes and intervention settings. The paper positions ArgGYM as a foundation for RLVR training but reports no actual RLVR training experiments.

Critical strengths

The experiments cover multiple models, different reasoning settings, 12 tasks, and varied structural configurations. The evaluation is fairly thorough. ArgGYM also has practical value as a benchmark for structured defeasible reasoning.

Critical weaknesses

The paper repeatedly calls ArgGYM an “RLVR-compatible training environment,” yet reports no RLVR training experiments. A verifier that provides rewards is not enough. Does training with ArgGYM actually improve defeasible reasoning? The paper provides no evidence.

Questions for authors
  • Why does the main text spend so much space on discussion and explanation, while many experimental results that directly support the conclusions are placed in the appendix? The current organization feels unbalanced.
  • ArgGYM still relies on “procedural generation + symbolic verification” and builds directly on existing ASPIC+ and PyArg. Compared with prior work (eg. ProverQA, Reasoning Gym, DeFAb, and DPG), what exactly does ArgGYM add beyond extending this approach to structured defeasible argumentation and introducing 12 tasks?
  • Curriculum levels change several structural factors at once (eg. depth, branching, and attackers/defenders). The last-link and weakest-link instances are also generated independently. How do the authors rule out these confounding factors and identify which specific factor drives the performance differences? What supports interpreting these results as more than descriptive correlations?
1 comment

Thank you for the thorough reading and for asking for stronger evidence and clearer positioning.

1. RLVR training. We trained Qwen3.5-4B with GRPO using ArgGYM’s verifiers as rewards on fresh problems from levels 1–12 (Section 6, Table 3, Appendices E and F). On 1,440 held-out instances excluded from training and selection, success rises from 20.4% to 53.3% (+32.9 percentage points, 95% CI [30.5, 35.3]). Gains extend to unsampled task–level combinations (+29.6 points) and levels beyond the training range (+32.2).

Transfer is mixed. Accuracy gains are 1.1 points on BoardgameQA, 1.4 on LogicNMR, and 2.0 on δ-NLI. Belief revision on Belief-R shows no observable improvement, and LogicBench’s nonmonotonic subset declines by 2.0 points. GSM8K, MATH-500, and MMLU-Pro remain within 0.6 points of baseline.

2. Manuscript organization (Q1). The main text now presents the training results, the analysis of all-or-nothing scoring, the structural regression, the matched link-principle evaluation, and the shared-model benchmark comparison. It also reports that 82.6% of analysis-task failures contain incorrect judgments and 0.1% contain correct content in a noncanonical format. We shortened the Discussion to two paragraphs and moved the extended discussion to Appendix H.

3. Contribution relative to prior work (Q2). Section 2 now distinguishes ProverQA’s prover verification for first-order logic, Reasoning Gym’s procedural RLVR environments, DeFAb’s focus on defeasible abduction, and DPG’s generation of DeLP knowledge bases. ArgGYM brings 12 analysis and intervention tasks together on a shared ASPIC+ substrate, so failures can be compared across operations within one formalism. It also compares two intervention contracts on the same theories. Executing a construction lets the scorer credit different answers that produce the required state. The same verification contracts apply to frozen and fresh instances and supply rewards during RLVR.

The training experiment shows what those rewards can achieve on held-out items. The shared-model comparison shows that ArgGYM distinguishes configurations that score similarly on some existing benchmarks, with rank correlations ranging from −0.38 to 0.82.

4. Confounds in level and link principle (Q3). The benchmark-wide comparisons are descriptive. We added three analyses (Section 5, Appendices D.3, D.5, and E).

  • Level and structure. Level alone predicts lower success in 11 of 12 tasks. After adjusting for recorded features including depth, attackers, junctions, argument and defeat counts, and ordering, the level coefficient’s interval includes zero in 11 of 12 tasks. The exception is counter_argument_strict. On status_query, errors are associated with longer derivations and undecided status. The number of queried claims has no detectable negative association with judgment accuracy. These results are consistent with level serving as a proxy for structural complexity.
  • Engine-computed statuses. Switching only the link principle changes status maps for 107/120 status_query theories, 101/120 defeat_diagnosis theories, and all 120 theories in each of preference_construction, claim_chain, and perturbation.
  • Matched model evaluation. We re-evaluate 120 frozen status_query theories with only the link principle changed. Changes in success are in percentage points, with 95% CIs.
ModelWeakest-link → last-linkLast-link → weakest-link
Base+10.0 [1.7, 20.0]−3.3 [−15.0, 10.0]
Trained+18.3 [6.7, 31.7]−25.0 [−38.3, −11.7]

The trained model performs better under last-link in both matched comparisons. This result applies to status_query. Comparisons for other tasks remain descriptive (Section 5, Appendix I).

Reviewer_u2dcOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This work presents ArgGYM, a benchmark and generator, as well as RLVR-compatible training environment for structured defeasible reasoning based on ASPIC+. It consists of 12 tasks (formalization, claim chain, defeat diagnosis, status query, semantics query, perturbation, preference construction, counterargument, counterargument strict, attack, defense, attack-defense), each generated and evaluated using a symbolic argumentation engine. The work comes with a public release of a 1440-instance benchmark, a generator to generate additional instances, and a verifier for evaluation and RLVR training. Experiments with modern LLMs on the benchmark show local correctness but failure to solve global, complete tasks.

Critical strengths
  • This work addresses reasoning evaluation in defeasible reasoning, which has many real-world applications and are highly overlooked compared to code, math, etc.
  • The benchmark design is comprehensive, covering 12 tasks.
  • This work provides both frozen benchmark and an instance generator allowing RLVR training.
Critical weaknesses
  • The benchmark primarily operates in a symbolic setting. It is unclear how the results translate to realistic defeasible reasoning in natural language, where linguistic ambiguity and world knowledge as context will pose additional challenges.
  • ArgGYM is presented as a training environment, but its effectiveness in improving performance on broader reasoning tasks has not been demonstrated.
  • Similarly, RLVR-compatibility is stressed as a contribution, but it has not been demonstrated in the paper.
Questions for authors
  • How sensitive are the results to the choice of argumentation framework, ASPIC+?
  • Does performance on ArgGYM correlate with performance on other defeasible reasoning benchmarks?
1 comment

Thank you for the positive assessment and for asking how far the results generalize beyond ArgGYM.

1. RLVR and broader reasoning (W2, W3). We trained Qwen3.5-4B with GRPO using rewards from ArgGYM’s verifiers (Section 6, Table 3, Appendices E and F). On 1,440 held-out instances excluded from training and selection, success rises from 20.4% to 53.3% (+32.9 percentage points, 95% CI [30.5, 35.3]). Gains extend to unsampled task–level combinations (+29.6 points) and levels above the training range (+32.2).

We evaluate the full test sets of 13 external benchmarks and pool three inference runs of one trained checkpoint. GSM8K, MATH-500, and MMLU-Pro each change by at most 0.6 points. GPQA Diamond, LiveCodeBench, and BBEH improve by 5.4, 5.3, and 2.6 points, respectively, with lower truncation on all three. In the run-1 comparison on items neither model truncated, LiveCodeBench gains 2.1 points (95% CI [0.2, 3.9]).

2. Transfer to natural-language defeasible reasoning (W1). Transfer is mixed (Section 6, Appendix F). BoardgameQA and LogicNMR, which make rules, exceptions, or priorities explicit, gain 1.1 and 1.4 points from baselines above 92%. BoardgameQA ranges from −1.2 points at depth 1 to +4.4 at depth 3. δ-NLI gains 2.0 points, partly because the trained model gives readable answers more often. Both models rarely revise earlier conclusions on Belief-R. On LogicBench’s 440 nonmonotonic yes/no questions, accuracy falls by 2.0 points, with the largest declines on one exception-reasoning category (−6.7) and priority reasoning (−6.3). ArgGYM’s template-generated formalization task offers a controlled natural-language setting. Transfer to more varied language remains open.

3. Sensitivity to the argumentation framework (Q1). A controlled comparison across frameworks remains open (Section 8, Appendix I). Within ASPIC+, ArgGYM varies last-link versus weakest-link and elitist versus democratic set comparison. Its semantics queries cover grounded, eager, and stable semantics and credulous and skeptical preferred acceptance. On matched status_query theories, the trained model loses 25.0 points switching to weakest-link (95% CI [−38.3, −11.7]) and gains 18.3 switching to last-link ([6.7, 31.7]). Arg2P independently reproduces every stored grounded status across 11,400 theory evaluations (Appendix A.1). The transfer results also cover benchmarks with different representations, such as default rules in LogicNMR and information-source preferences in BoardgameQA.

4. Correlation with existing benchmarks (Q2). We compare 12 shared configurations across seven benchmarks (Section 5, Appendix G). Spearman correlations with ArgGYM are 0.82 for δ-NLI, 0.70 for LogicNMR, 0.52 for LogicBench, 0.46 for BoardgameQA, 0.17 for Multi-LogiEval, and −0.38 for DEFREASING. We omit Belief-R’s correlation given its narrow range (48.2–50.6 BREU). LogicBench scores span 81.7%–85.5%, while ArgGYM success spans 1.3%–39.3%, showing that ArgGYM distinguishes configurations with similar LogicBench scores.

Reviewer_frNGOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ArgGYM, a procedural benchmark and RLVR-compatible environment for structured defeasible reasoning, built on the ASPIC+ argumentation framework. The motivating claim is that math, code, and formal-logic benchmarks have mechanically verifiable rewards but do not capture reasoning under incomplete, conflicting, and revisable information, and that existing defeasible-reasoning benchmarks either test particular inference problems or are not decomposed into separately verifiable operations. ArgGYM represents each problem as an ASPIC+ theory: a knowledge base, strict and defeasible rules, conflict relations, and preferences. Arguments are built recursively, attacks are classified as undermining, rebutting, or undercutting, preferences determine which attacks become defeats, and grounded semantics determines which claims are accepted. The benchmark crosses two argument preference orderings (last-link and weakest-link) with two set orderings (elitist and democratic). Its twelve tasks cover representation and support recovery (formalization, claim chain), diagnosis and revision (defeat diagnosis, status query, semantics query, perturbation), and constructive intervention (preference construction, counter argument, strict-rule-enabled counter argument, attack, defence, attack-and-defence). Instances are produced by task-specific generators that build candidate theories and pass them through structural and formal gates; the pinned PyArg engine computes the reference argumentative state, and constructive interventions are evaluated by executing them and recomputing the resulting state. The frozen benchmark contains 1,440 instances (12 tasks x 15 curriculum configurations x 2 preference orderings x 2 set orderings x 2 instances), with the same generators able to produce fresh verified instances for evaluation and for mechanically rewarded training. The evaluation covers 18 model variants in 42 model-reasoning configurations across five model families, scored by complete-task success and a task-native partial score that is never averaged across tasks. Results range from near zero to 82.4% complete success (Gemini 3.8 Flash at high effort). The paper's main empirical claims are that partial argumentative competence does not compose into complete solutions, that later curriculum bands with longer dependencies and more interacting structures reduce complete success before local competence disappears, that preference regimes change behavior without a universal difficulty ordering, and that providing a cheaper intervention route does not imply models can exploit it.

Critical strengths
  1. The shared formal substrate is a genuine design contribution, not just a collection of tasks.
  2. Six tasks are constructive, and any intervention that satisfies the postconditions after execution is accepted, so the benchmark does not penalize a valid solution for differing from the generator's own. This is the right contract for a search problem and is what makes the counter-argument versus strict-rule comparison meaningful.
  3. The evaluation is broad and the reporting is disciplined.
  4. Limitations are stated candidly and match the evidence.
Critical weaknesses
  1. The paper advertises an RLVR training environment but reports no training experiment. The abstract, contributions, and Section 3.4 describe ArgGYM as "RLVR-compatible", with verifier outputs that "can also serve as mechanically computed rewards", and the environment is presented as both an evaluation setting and a source of training problems.
  2. The key "local v.s. global" finding depends on a strict complete-task metric whose failure modes are not decomposed. Complete-task success requires exact recovery of the structured output for the six analysis tasks, while the native score gives partial credit; the paper interprets the gap as evidence that partial competence does not compose (lines 293–305; lines 347–362).
  3. Everything in the benchmark rests on one third-party engine, and the engine itself is not validated. Reference states are computed by PyArg 2.0.2, pinned, and the paper treats it as the formal-semantic oracle. No cross-check against a second ASPIC+ implementation, no hand-verified sample, and no agreement study is reported. For a benchmark whose entire value proposition is exact verification, an error or ambiguity in the oracle would propagate into every task and every reported model score, and readers cannot distinguish a model failure from an engine disagreement. This is not acknowledged in the Limitations section.
  4. The evaluation has no shared-model comparison against the defeasible-reasoning benchmarks the paper positions itself against. The paper argues that δ-NLI, LogicNMR, BoardgameQA, Belief-R, DEFREASING, DeFAb, DeReLab, and DPG each cover particular inference problems, and that ArgGYM's contribution is to unify separately verifiable operations over one substrate. That positioning is about complementarity, so a head-to-head is not strictly required; but without evaluating even a subset of the same models on one or two of those benchmarks, the paper cannot show that its operation-level profile reveals failures those benchmarks miss. The claim that the decomposition is diagnostically valuable is therefore supported only within ArgGYM's own task suite.
Questions for authors
  1. Can you run one small RLVR experiment? For example, train one open-weight model with the verifier reward on two tasks, with and without the curriculum ordering, and report frozen-benchmark performance before and after. Does training on fresh instances transfer to the frozen suite, and does the curriculum band ordering predict learning difficulty?
  2. What fraction of complete-task failures are semantic versus format/schema errors? Please report a stratified error analysis on a sample of failing responses for the six analysis tasks, separating (a) wrong argumentative judgments, (b) correct judgments in a non-canonical schema, and (c) incomplete multi-part answers.
1 comment

Thank you for the detailed feedback and for highlighting ArgGYM’s shared formal substrate and execution-based scoring.

1. RLVR and curriculum (W1, Q1). We trained Qwen3.5-4B with GRPO and LoRA on fresh problems from 82 task–level combinations across all 12 tasks at levels 1–12. ArgGYM’s verifiers reward complete success, give partial credit on four analysis tasks, and give zero for truncated answers. We selected step 200 on a separate development split (Section 6, Table 3, Appendix E).

On 1,440 held-out instances excluded from training and selection, success rises from 20.4% to 53.3% (+32.9 percentage points, 95% CI [30.5, 35.3]). On the frozen benchmark, it rises from 20.8% to 52.7% (+31.8 points, [29.9, 33.9]). Frozen base-model scores informed training-cell selection, so the separate held-out set is our primary test. Gains extend to unsampled task–level combinations (+29.6) and levels 13–15 (+32.2). Eleven of 12 task intervals exclude zero in the direction of improvement. Constructive success rises from 24.4% to 80.0%, and analysis gains 10.2 points. Even crediting every truncated base answer as correct leaves 78% of the constructive gain.

Base success falls across L1–5, L6–10, and L11–15 (36.5%, 17.1%, 7.7%), while endpoint gains are similar (+35.6, +31.9, +31.2 points). These comparisons leave learning speed open. A matched fixed-sampler ablation was beyond our compute budget, so the contribution of adaptive sampling remains open (Appendix I).

2. Partial correctness and all-or-nothing scoring (W2, Q2). We now describe the finding as high partial correctness alongside low complete-task success. Across all six analysis tasks, we classified 11,671 failures from 20 untrained configurations (Section 5, Appendix D.3). Incorrect judgments account for 82.6% (74.2%–95.1% across configurations). Another 6.9% are incomplete with all stated judgments correct. Six answers fail on layout alone (0.1%) and three on ordering alone (<0.1%). We could not assess semantic correctness for the 10.4% that were unreadable (6.7%), lacked an answer region (3.0%), or were truncated (0.7%).

On status_query, observed exact success exceeds the prediction under independent per-claim errors in 45 of the 47 configuration–band combinations with at least one solved instance, by up to 12.5×. This is consistent with errors clustering by instance. Longer derivations and undecided statuses are associated with lower judgment accuracy in 19 and 16 of 20 configurations, respectively. We find no detectable negative association with the number of queried claims.

3. Independent engine validation (W3). Arg2P 0.25.1 reproduces all stored grounded statuses across 11,400 theory evaluations and all answers in the 600 semantics_query records. A separate implementation of Dung’s semantics also reproduces the grounded statuses from the exported defeat graphs (Section 3.1, Appendix A.1).

PyArg can merge separately named rules with identical content, whereas Arg2P preserves them. All checked gold labels agree. The difference can affect scoring when duplicates are present in the base theory or introduced by a model. Across 21 archived runs, we screened 17,640 answers and independently re-scored the 625 flagged for this risk. Three verdicts changed, all from failure to success. No reported overall score in these runs changed by more than 0.1 percentage points.

4. Shared-model comparison (W4). We evaluate the same 12 configurations on ArgGYM and seven external benchmarks (Section 5, Appendix G). Spearman correlations with ArgGYM are 0.82 for δ-NLI, 0.70 for LogicNMR, 0.52 for LogicBench, 0.46 for BoardgameQA, 0.17 for Multi-LogiEval, and −0.38 for DEFREASING. Belief-R scores cluster at 48.2–50.6 BREU, so we omit its correlation.

Qwen3.5-4B and Qwen3.5-27B score 84.4% and 84.8% on LogicBench and both score 50.0% on ArgGYM’s claim_chain, but score 17.5% and 44.2% on attack. This illustrates what the task-level breakdown adds to aggregate comparisons.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.