- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper presents ArgGYM, a procedural benchmark for structured defeasible reasoning with engine-verified evaluation. Built on ASPIC+ and PyArg, it covers 12 tasks and provides 1,440 frozen instances plus a generator for new samples. Experiments across 18 model variants and multiple reasoning settings show that models can handle some local reasoning but struggle to recover fully correct argumentative states in complex structures. Performance also varies across preference regimes and intervention settings. The paper positions ArgGYM as a foundation for RLVR training but reports no actual RLVR training experiments.
Critical strengths
The experiments cover multiple models, different reasoning settings, 12 tasks, and varied structural configurations. The evaluation is fairly thorough. ArgGYM also has practical value as a benchmark for structured defeasible reasoning.
Critical weaknesses
The paper repeatedly calls ArgGYM an “RLVR-compatible training environment,” yet reports no RLVR training experiments. A verifier that provides rewards is not enough. Does training with ArgGYM actually improve defeasible reasoning? The paper provides no evidence.
Questions for authors
- Why does the main text spend so much space on discussion and explanation, while many experimental results that directly support the conclusions are placed in the appendix? The current organization feels unbalanced.
- ArgGYM still relies on “procedural generation + symbolic verification” and builds directly on existing ASPIC+ and PyArg. Compared with prior work (eg. ProverQA, Reasoning Gym, DeFAb, and DPG), what exactly does ArgGYM add beyond extending this approach to structured defeasible argumentation and introducing 12 tasks?
- Curriculum levels change several structural factors at once (eg. depth, branching, and attackers/defenders). The last-link and weakest-link instances are also generated independently. How do the authors rule out these confounding factors and identify which specific factor drives the performance differences? What supports interpreting these results as more than descriptive correlations?
Thank you for the thorough reading and for asking for stronger evidence and clearer positioning.
1. RLVR training. We trained Qwen3.5-4B with GRPO using ArgGYM’s verifiers as rewards on fresh problems from levels 1–12 (Section 6, Table 3, Appendices E and F). On 1,440 held-out instances excluded from training and selection, success rises from 20.4% to 53.3% (+32.9 percentage points, 95% CI [30.5, 35.3]). Gains extend to unsampled task–level combinations (+29.6 points) and levels beyond the training range (+32.2).
Transfer is mixed. Accuracy gains are 1.1 points on BoardgameQA, 1.4 on LogicNMR, and 2.0 on δ-NLI. Belief revision on Belief-R shows no observable improvement, and LogicBench’s nonmonotonic subset declines by 2.0 points. GSM8K, MATH-500, and MMLU-Pro remain within 0.6 points of baseline.
2. Manuscript organization (Q1). The main text now presents the training results, the analysis of all-or-nothing scoring, the structural regression, the matched link-principle evaluation, and the shared-model benchmark comparison. It also reports that 82.6% of analysis-task failures contain incorrect judgments and 0.1% contain correct content in a noncanonical format. We shortened the Discussion to two paragraphs and moved the extended discussion to Appendix H.
3. Contribution relative to prior work (Q2). Section 2 now distinguishes ProverQA’s prover verification for first-order logic, Reasoning Gym’s procedural RLVR environments, DeFAb’s focus on defeasible abduction, and DPG’s generation of DeLP knowledge bases. ArgGYM brings 12 analysis and intervention tasks together on a shared ASPIC+ substrate, so failures can be compared across operations within one formalism. It also compares two intervention contracts on the same theories. Executing a construction lets the scorer credit different answers that produce the required state. The same verification contracts apply to frozen and fresh instances and supply rewards during RLVR.
The training experiment shows what those rewards can achieve on held-out items. The shared-model comparison shows that ArgGYM distinguishes configurations that score similarly on some existing benchmarks, with rank correlations ranging from −0.38 to 0.82.
4. Confounds in level and link principle (Q3). The benchmark-wide comparisons are descriptive. We added three analyses (Section 5, Appendices D.3, D.5, and E).
counter_argument_strict. Onstatus_query, errors are associated with longer derivations and undecided status. The number of queried claims has no detectable negative association with judgment accuracy. These results are consistent with level serving as a proxy for structural complexity.status_querytheories, 101/120defeat_diagnosistheories, and all 120 theories in each ofpreference_construction,claim_chain, andperturbation.status_querytheories with only the link principle changed. Changes in success are in percentage points, with 95% CIs.The trained model performs better under last-link in both matched comparisons. This result applies to
status_query. Comparisons for other tasks remain descriptive (Section 5, Appendix I).