acceptodds
Under review as a conference paper at ICLR 2027

UltraLogic: Enhancing LLM Reasoning through Large-Scale Data Synthesis and Bipolar Float Reward

Abstract

While Large Language Models (LLMs) have demonstrated significant potential in natural language processing, complex general-purpose reasoning—requiring multi-step logic, planning, and verification—remains a critical bottleneck. Although Reinforcement Learning with Verifiable Rewards (RLVR) has succeeded in specific domains, the field lacks large-scale, high-quality, and difficulty-calibrated data for general reasoning. To address this, we propose UltraLogic, a framework that decouples the logical core of a problem from its natural language expression through a Code-based Solving methodology to automate high-quality data production. The framework comprises hundreds of unique task types and an automated calibration pipeline across ten difficulty levels, which is dynamically adjustable to remain effective and future-proof as base model capabilities evolve. Through feedback-guided task refinement and difficulty recalibration, it supports a data-centric approach to recursive self-improvement (RSI), producing verifiable training data matched to evolving model capabilities. Furthermore, to mitigate binary reward sparsity and the Non-negative Reward Trap, we introduce the Bipolar Float Reward (BFR) mechanism, utilizing graded penalties to effectively distinguish perfect responses from those with logical flaws. Our experiments demonstrate that BFR, combined with a difficulty matching strategy, significantly improves training efficiency, guiding models toward consistent logical policies.

Primary area:
reinforcement learning
Keywords:
Logic Reasoning, Synthetic Data Generation, Reinforcement Learning with Verifiable Rewards, Reward Shaping

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_jZW3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents UltraLogic, a pipeline that generates verifiable logic-puzzle data with a ten-level difficulty ladder. It also proposes Bipolar Float Reward (BFR): reward 1 for a fully correct answer, and S−1 otherwise, where S is a partial score. Qwen3-8B and 14B are trained with GRPO. The paper reports gains on AIME, HMMT, BBH, BBEH, and ARC-AGI. It also claims that the best training difficulty grows with model size, and that BFR beats binary and float rewards.

Critical strengths

The data problem is real. Separating the logic core from the text is a sound design. The pipeline is largely automated. The appendix is detailed. The evaluation includes out-of-domain benchmarks. The authors plan to release the code.

Critical weaknesses

BFR has little novelty. The paper shows that R_BFR + 1 = S + 1[S=1]. GRPO normalization ignores the shift. So the "bipolar" part and the "Non-negative Reward Trap" do nothing by themselves. The trap is never defined or tested. What remains is partial credit plus a bonus. Appendix F does not help choose c. The simulation shows nearly the same gap for c from 0.5 to 2. The reward ablation is weak. Graded Float is much worse than Binary on AIME24 (76.9 vs 81.7). The paper does not explain why. The BFR gains are small. The efficiency claim compares epoch-1 and epoch-2 checkpoints, not equal compute. Reward curves with different scales are not comparable. "Difficulty Matching" is over-claimed. The "scaling law" uses only two model sizes and three bands. Difficulty is calibrated on other strong models, not on the trained ones. The trained model's pass rate per band is never measured. Hard bands also have longer outputs, so truncation at 32K could explain the instability. This is not ruled out. Baselines and transfer evidence are thin. The only baseline is LogicPro, which gives almost no gain, and the reason is untested. There are no results for SynLogic or Reasoning Gym. There is no control with generic math RL data. RL on any data might fix length or format and raise AIME scores. The benchmark-overlap cleaning is not described. Changes of 0.1–0.6 on ARC-AGI are noise. The headline claims go beyond the experiments. The paper claims "recursive self-improvement", but all runs use fixed datasets. The abstract says "hundreds of task types", but only 100 are released and 50 are used. LLM-written solvers have no measured error rate. The paper itself reports collapse when only 1–3 tasks were wrong.

Questions for authors
  1. Can you add a generic RL-data baseline and SynLogic under the same budget?
  2. Why does Graded Float hurt? Can you ablate the bonus weight in actual training?
Reviewer_9VHsOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper makes two contributions. First, it introduces UltraLogic, a data synthesis framework that selects a source task, matches it to a template, and programmatically generates a concrete problem instance with a controlled difficulty level from 1 to 10. Second, it proposes Bipolar Float Reward, a reward mechanism for GRPO that combines partial correctness rewards with a bonus for full correctness to address the sparsity of binary rewards. Experiments with Qwen3 8B and 14B models yield better performance than baseline on common benchmarks such as AIME24/25 and BBH.

Critical strengths
  • Overall the paper organization is clear; the figures are well made and easy to interpret.
Critical weaknesses
  • The paper lacks scientific rigor. The manuscript presents a collection of techniques or "tricks" for data synthesis and training efficiency, but their motivations and connections are insufficiently developed. A more convincing study would establish a coherent contribution supported by detailed experiments and thorough ablations. For example, the manuscript states, “For suitable tasks, an automated LLM workflow builds Chinese and English templates for bilingual training,” without clearly motivating bilingual training. Why is bilingual training necessary? Is there any ablation study without bilingual training?

  • Experiments are only done with small Qwen models with little competitive RL baselines. Several design choices appear arbitrary.

  • The methods have substantial limitations. Tasks are generated from a fixed set of templates, and the examples presented appear to be toy problems that lack the complexity of realistic tasks, such as those in modern tasks (e.g., GDPval [1] and WorkBuddy Bench [2]). Difficulty is also adjusted within these fixed templates. Although this may produce harder instances, greater difficulty alone does not establish that the resulting tasks are meaningful or realistic. The manuscript provides insufficient evidence that this approach can generate tasks that are both challenging and practically relevant. While such an approach might have been compelling two years ago, it seems largely outdated. It is unclear to me whether the method and conclusions would be insightful to the community.

[1] https://arxiv.org/abs/2510.04374 [2] https://arxiv.org/abs/2607.20911

Questions for authors

See above

Reviewer_dnD6OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UltraLogic is a data-centric framework for synthesizing verifiable training data for general-purpose reasoning. Each of its 100 task types (50 used in training) is defined programmatically: an input function generates slot-filling parameters at a specified difficulty, and a solution function deterministically computes the ground-truth answer, while LLM-written bilingual (ZH/EN) templates render the logical core into natural language. A validation gate (flagship-model solvability checks with ReAct-style revision) and a ten-level difficulty ladder calibrated to measured model success rates (approx. 100% at L1 down to 0% at L10) close the production loop. The second contribution is the Bipolar Float Reward (BFR): imperfect answers receive S-1, with S in [0,1] a task-specific partial-correctness score over the structured final answer (accuracy / F1 / similarity / absolute-difference), while fully correct answers receive 1; per Eq. (2), under GRPO group normalization this is equivalent to a graded score plus a unit full-correctness bonus. Experiments train Qwen3-8B/14B with GRPO on 10k samples and evaluate on AIME24/25, HMMT25, BBH, BBEH, and ARC-AGI (avg@64). The paper reports (i) a difficulty-matching effect -- the optimal band shifts from Levels 1-4 at 8B to Levels 4-7 at 14B; (ii) BFR outperforming binary and graded-float rewards, matching binary's two-epoch results after one epoch on five of six benchmarks; and (iii) gains over LogicPro under matched settings.

Critical strengths
  1. A well-engineered, reusable data-generation pipeline. The decoupled input/solution functions, the automated validation gate, and the ten-level difficulty ladder calibrated to measured model success rates are concrete, adoptable pieces of infrastructure, and the promised release of data and code extends their value. Given the shortage of difficulty-controlled verifiable data for general-reasoning RLVR, this is the strongest reason to accept.

  2. Quantified evidence that RL data difficulty must track model capacity. The 8B vs. 14B contrast (Table 2, Panels A/B), together with LogicPro's near-zero gains on a current model, makes the "recalibrate as models improve" point concrete rather than anecdotal; the data-quality sensitivity observation (Sec. 6.3) is a useful negative result. These findings are actionable beyond this paper's setup.

Critical weaknesses
  1. The empirical scope does not match the breadth of the claims. All results come from one model family (Qwen3-8B/14B, dense only; the MoE divergence claim in Sec. 6.3 is asserted without data), so the "scaling law of difficulty" (Sec. 6.1) rests on two model sizes from a single family, and whether the calibrated ladder transfers is unknown. The sole numerical baseline is LogicPro, which the paper itself argues is obsolete for current models. Missing are an SFT baseline on the same data, training on the most related programmatic frameworks (SynLogic, Reasoning Gym) under identical GRPO settings, a math-domain RLVR control to interpret the AIME/HMMT gains, and any in-domain held-out evaluation. There is also no training-seed variance: avg@64 controls evaluation sampling noise, but with one run per cell, gaps such as AIME24 +0.9 for BFR over binary (Table 2, Panel C) are within typical seed-to-seed variation, and the BBH / BBEH / ARC-AGI gains are small (+0.9 / +1.4 / +0.1).

  2. BFR's evidence lags its framing. Eq. (2) shows BFR is equivalent to a graded float reward plus a unit bonus for perfect answers, and Appendix F shows the scaling factor matters only when a group contains partially correct responses -- the mechanism is a reweighting, useful but modest. More importantly, the correctness of the graded signal is assumed rather than established: S is produced by per-task heuristics over the final answer, with no audit of scoring accuracy, no reported distribution of S over rollouts, and no reward-hacking analysis. The ablation covers only Qwen3-8B at Levels 1-4 (Table 2, Panel C; Table 7), and the motivating "Non-negative Reward Trap" is never defined. Without evidence that S tracks reasoning quality across the 50 task types, the reliability of the reward -- the paper's second main contribution -- remains unclear.

Questions for authors
  1. Can you add at least one non-Qwen model (e.g., Llama-3.x-8B)? Which specific models were used for the pass-rate calibration, and does the resulting ladder transfer across families? Please also substantiate or soften the MoE divergence claim in Sec. 6.3.

  2. Can you add (a) SFT on the same UltraLogic data and (b) GRPO training on SynLogic / Reasoning Gym data under the identical setup? A math-domain RLVR control would further help interpret the AIME/HMMT improvements.

  3. How are the per-task BFR scoring functions selected and validated? What fraction of rollouts receive intermediate S, and is there any audit or reward-hacking stress test showing that S reflects reasoning quality rather than surface answer overlap? Does BFR's advantage hold at 14B and on other difficulty bands?

  4. How many training seeds underlie each cell of Table 2? Please report mean +/- std for the BFR vs. binary comparison (Table 7).

  5. Please define the "Non-negative Reward Trap"; justify the overlapping difficulty bands (1-4 / 4-7 / 7-10); and specify how "tasks similar to existing benchmarks" (Sec. 3.2) was measured for decontamination.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.