acceptodds
Under review as a conference paper at ICLR 2027

LURE: Pursuit-Evasion Self-Play for Zero-Data Multi-Step Reasoning

Abstract

Reinforcement learning with verifiable rewards (RLVR) has substantially improved large language model reasoning, but its scalability remains tied to externally supplied tasks whose difficulty must keep pace with an improving policy. Zero-data self-play addresses this bottleneck by generating its own curriculum, yet existing approaches either estimate task difficulty from model-derived proxies or rely on executable task specifications, while multi-step solvers remain largely trained from sparse outcome supervision. We introduce LURE, a verifier-grounded pursuit-evasion framework for zero-data self-play, in which a task-generating Challenger and a planner-executor Solver co-evolve through a shared environment verifier. This formulation turns the verifier into a common learning interface for both curriculum construction and policy improvement. Specifically, LURE uses verifier-confirmed Solver success to position generated tasks near a capture frontier, and converts stepwise verifier progress into capture-anchored dense process credit for the Solver. Round-anchored regularization further stabilizes the two adaptive policies during co-evolution. Across three heterogeneous reasoning environments and three backbone families, LURE outperforms advanced baselines under both unified and specialist settings, and across out-of-domain benchmarks it delivers the strongest near-domain transfer and the best overall average.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Zero-data RL, Curriculum learning, Multi-step reasoning, Dense credit assignment, Large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_yJg9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LURE studies self-generated curricula for verifiable reasoning. A Challenger selects difficulty parameters for programmatic task generators, while separate Planner and Executor policies learn from verifier feedback. The Planner receives trajectory-level verified progress; the Executor receives immediate progress rewards and terminal correctness. All three policies use clipped GRPO with round-start KL regularization. Experiments show promising in-domain results across three environments and backbone families. Experiments across three environments and three backbone families demonstrate the effectiveness of the proposed method.

Critical strengths
  1. Using the same programmatic verifier for curriculum feedback and process supervision provides a coherent training framework without a separately trained process reward model.
  2. The PhantomWiki improvements are promising, and the in-domain experiments cover three environments, three backbone families, and unified/specialist settings.
Critical weaknesses
  1. LURE uses different learning-rate, epoch, and KL settings from the baselines and trains on more retained tasks per round. Moreover, the “w/o planner credit” ablation changes the Planner’s reward but retains the model, leaving the benefit of a separate Planner over a single Solver untested.

  2. The evaluation protocol does not explain several reported scores, and uncertainty is insufficiently documented. ZebraLogic reportedly uses 300 puzzles with one greedy rollout each, but Table 1 values such as 11.6% and 16.9% cannot directly arise under that protocol. The authors should clarify sample counts and aggregation, and report independent training repetitions and confidence intervals to establish whether small advantages, particularly on OOD benchmarks, are reliable.

Questions for authors
  1. Under comparable token/GPU cost, what benefit comes from separate Planner and Executor models over a single or parameter-sharing Solver?
  2. What exact sample counts, aggregation rules and independent training repetitions produce the reported ZebraLogic scores? Please provide per-run results and uncertainty for the principal comparisons.
  3. Which role’s weights are used for OOD evaluation: Planner or Executor? How are checkpoints selected, and why is OOD evaluation restricted to Qwen2.5-7B-Instruct despite the in-domain Llama and Gemma experiments?
  4. Will the code be publicly released?
3 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1/2)

Thank you for the valuable feedback. We address each point below.


W1 & Q1 [Optimization settings, retained tasks, and the separate Planner]

LURE still outperforms R-Zero under the shared recipe, and R-Zero equipped with our verifier needs four times the candidates and five times the rollouts to match LURE. The separate Planner leads the single Solver by 2.7 points at equal rounds and by 0.5 points at matched GPU hours. We address this in two parts: the optimization recipe with task exposure, and the separate Planner.

(1) Recipe and exposure. In the paper, every method receives 144 candidates per round, GRPO trains on the same 144 tasks per round as LURE, and R-Zero's lower count comes from its own filter (App. A, Fig. 7). We retrain R-Zero under LURE's recipe (learning rate 5×10−65\times10^{-6}, one PPO epoch, KL 0.10.1 to the round-start snapshot). R-Zero-V is R-Zero equipped with our generators and verifiers: its Challenger proposes task specifications, the shared generator and validity gate instantiate them, and verified capture replaces majority agreement in its reward and filter. R-Zero-V with four times as many candidates and subsampled LURE match exposure in both directions. Table R1 reports the results.

Table R1. Shared-recipe, verifier and exposure comparisons on the Qwen2.5-7B-Instruct unified model. Task counts are eight-round means.

MethodRecipeTraining tasks per roundAvg. (%)
R-ZeroBaseline3641.0
R-ZeroLURE40.644.2
R-Zero-VLURE37.444.4
R-Zero-V, 4× candidatesLURE150.148.2
LURE, subsampled tasksLURE37.445.9
LURELURE14448.0

Under the shared recipe, LURE leads R-Zero by 3.8 points. At matched exposure, LURE reaches 45.9 versus 44.4 for R-Zero-V. To reach LURE's accuracy, R-Zero-V uses approximately 4608+8×150.1≈58094608+8\times150.1\approx5809 Solver rollouts per round for probing and training, and LURE uses 1152, which is 19.8% as many.

(2) Separate Planner versus a single Solver. LURE uses separate parameters θplan\theta_{\mathrm{plan}} and θexec\theta_{\mathrm{exec}}. The single Solver generates a plan and then executes it, with planner credit on plan tokens and executor credit on action tokens. The parameter-sharing variant uses one set of weights with role-specific prompts. All variants retain LURE's curriculum, verifier feedback, and training recipe. We also extend the single Solver's training to match LURE's GPU hours. Table R2 reports the results.

Table R2. Solver variants of the Qwen2.5-7B-Instruct unified model.

SolverRoundsAvg. (%)Generated tokens per round (M)GPU-hour budget
Single Solver845.31.2120.20
Single Solver, matched GPU hours9.7347.51.2124.56
Parameter-sharing Planner and Executor847.41.2921.03
Separate Planner and Executor (LURE)848.01.2824.56

Separate parameters give the highest accuracy in every comparison. Parameter sharing uses fewer GPU hours at comparable token counts and offers a lower-cost alternative with 0.6 points lower accuracy. We have added Table R2 to App. C.


W2 & Q2 [ZebraLogic sample counts, aggregation, and uncertainty]

Each ZebraLogic score averages three evaluation passes over the same 300 puzzles, so 11.6 and 16.9 are 104/900 and 152/900. App. B omitted this aggregation step, and we have corrected it. With krk_r solved puzzles in pass rr, A‾=13∑r=13100kr300\overline{A}=\frac{1}{3}\sum_{r=1}^{3}\frac{100k_r}{300}, rounded after averaging, for all ZebraLogic results in Tabs. 1, 10 and 11. Each pass uses one greedy output per puzzle, and outputs can differ across passes since small numerical differences in the logits can change near-tied choices. The passes re-evaluate one checkpoint, and Table R3 reports three independent training runs.

Table R3. Three independent training runs of the Qwen2.5-7B-Instruct unified model.

MethodTraining runPhantomWikiIFEvalZebraLogic k1,k2,k3k_1,k_2,k_3 (mean acc.)OOD Overall
R-Zero1, original50.857.9[43, 43, 42] (14.2)36.4
R-Zero249.256.4[56, 55, 56] (18.6)36.7
R-Zero350.057.9[45, 45, 45] (15.0)36.3
R-ZeroMean ± std50.0 ± 0.857.4 ± 0.915.9 ± 2.336.5 ± 0.2
LURE1, original65.861.2[51, 50, 51] (16.9)36.9
LURE270.059.9[55, 55, 55] (18.3)36.8
LURE367.559.1[56, 56, 56] (18.7)35.8
LUREMean ± std67.8 ± 2.160.1 ± 1.018.0 ± 0.936.5 ± 0.6
LURE minus R-Zero, 95% CI/[13.3, 22.3][0.5, 4.9][-2.9, 7.0][-1.3, 1.4]

We report the sample std and Welch-tt intervals. In-domain means are 48.6 ± 0.7 versus 41.1 ± 0.2, with gap interval [5.9, 9.1], and the PhantomWiki, IFEval and in-domain intervals exclude zero. These runs use the configurations of Tab. 1, and with R-Zero under LURE's recipe the in-domain gap is 3.8 points (Table R1 above), more than five times LURE's run-to-run std.


Please see the subsequent Authors' Rebuttal (2/2) for remaining responses.

AuthorsOpenReview ↗

Authors' Rebuttal (2/2)

Continued: Authors’ Rebuttal (2/2)


Q3 [Weights, checkpoints, and backbones for OOD evaluation]

OOD evaluation uses the Executor weights, with checkpoints selected on a held-out validation set and never on OOD scores. The Executor produces complete answers, whereas the Planner outputs subgoals, each tied to a verifier predicate. The Tab. 3 caption describes the prompting: benchmark-specific prompts without a role prompt. The validation set comprises 240 generated PhantomWiki instances, 320 IFEval-style prompts generated by the same constraint builder with 16 constraint types, and 300 generated ZebraLogic puzzles. It is a fixed held-out set, separate from the 120, 541 and 300 reported test instances. Generator seed ranges are disjoint across training, validation and test, and training tasks identical to a validation or test instance are removed before training. Validation instances never enter Solver updates or Challenger feedback.

All eight-round comparisons, including the shared-recipe and verifier controls, use the same validation instances and the eight end-of-round checkpoints. Unified models maximize the mean of the three validation accuracies, and specialists use their own environment. Ties are broken in favor of the earlier round. Validation uses the same number of passes as the test evaluation. All 13 OOD benchmarks use the one selected unified checkpoint. We have added these details to App. B. The original OOD table uses Qwen because Qwen is the backbone of the component analyses (Figs. 4 and 5). Table R4 adds both other backbones on all 13 OOD benchmarks.

Table R4. OOD group averages (%) for Llama-3.1-8B-Instruct and Gemma-2-9B-it unified models. Overall is the mean over the 13 benchmarks.

BackboneMethodNear-DomainMathematicsLogicOther ReasoningOverall
Llama-3.1-8BBase26.824.018.624.424.1
Llama-3.1-8BR-Zero29.127.119.223.926.0
Llama-3.1-8BLURE28.824.819.822.524.9
Gemma-2-9BBase35.628.724.226.929.9
Gemma-2-9BR-Zero37.628.324.728.430.6
Gemma-2-9BLURE36.128.026.027.330.1

On both added backbones, LURE remains above the base model overall and has the highest far-transfer logic score. In the Qwen three-run result, both methods reach 36.5 on OOD and LURE is 7.5 points higher in-domain. LURE's in-domain gains therefore come without loss of overall OOD performance relative to the base model, with transfer varying across benchmarks and backbones. We have revised the transfer claims in the abstract, the introduction, Sec. 4.2 and the conclusion, and added Table R4 to App. C.


Q4 [Code release]

Yes, we will release our code with the final version.


We hope this fully addresses your questions and comments, and remain happy to engage in further discussion for the rest of the rebuttal period.

Thank you for the detailed rebuttal, which addresses my concerns. However, I will maintain my score of 2 because LURE requires substantial environment-specific engineering and its practical gains remain limited. In my view, an agentic setting could broaden LURE’s practical applicability, and may be an interesting direction for future work.

Reviewer_DGGSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces LURE, a verifier-grounded self-play framework for multi-step reasoning. A Challenger selects task difficulty, while a planner–executor Solver learns from terminal success and intermediate verified progress. The framework combines a curriculum objective targeting approximately 50% Solver success, dense process rewards, and round-anchored KL regularization. Experiments cover three reasoning environments and three backbone families, with additional transfer evaluations.

Critical strengths
  • Using a shared verifier for curriculum selection and process supervision provides a coherent training framework. The paper also gives a mathematical justification for targeting intermediate success rates: under its stated assumptions, 50% success maximizes the probability of obtaining non-degenerate outcome-based group-relative advantages.
  • The reported improvements on PhantomWiki are substantial. For the Qwen2.5-7B unified model, success increases from 50.8% with R-Zero to 65.8% with LURE. The component ablations also provide useful evidence about the behavior of the complete system.
Critical weaknesses
  1. The headline improvement is confounded by unequal optimization settings, and the paper’s own ablation demonstrates that this materially changes the comparison. Table 7 assigns the baselines a learning rate of 10−510^{-5}, two PPO epochs, and no KL regularization, whereas LURE uses 5×10−65\times10^{-6}, one epoch, and KL coefficient 0.1. In Table 11, running LURE with the baseline optimization recipe reduces its three-environment average from 48.0 to 40.3, a 7.7-point decrease. R-Zero scores 41.0 in Table 1, so LURE’s reported 7.0-point advantage disappears under that common recipe. This does not establish that optimization accounts for every improvement, but it directly prevents attributing the headline advantage to the proposed curriculum and credit-assignment mechanisms. A convincing comparison must also evaluate the baselines with LURE’s stabilization recipe.

  2. The reported ZebraLogic puzzle accuracies cannot be reconstructed from the stated evaluation protocol. Table 8 specifies 300 puzzles, and Appendix B specifies greedy decoding with one rollout per instance and all-or-nothing puzzle correctness. Under this protocol, accuracy is 100k/300100k/300 for an integer number of solved puzzles kk. However, Table 1 reports 16.9% for the Qwen unified LURE model: 50 solved puzzles yield 16.7% after rounding, while 51 yield 17.0%. Likewise, the reported 15.4% for GRPO cannot arise from this denominator. These are discrepancies in the central results, not merely missing error bars. Any additional averaging, weighting, or different evaluation denominator must be explicitly documented and reconciled with the protocol before these results are reproducible.

  3. The principal self-play comparison does not isolate the proposed curriculum mechanism from changes in supervision and training-task exposure. Table 6 specifies that R-Zero learns from majority-vote pseudo-labels, whereas LURE receives verifier-confirmed success and intermediate progress. The curriculum-cost analysis further reports that, from 144 candidates per round, R-Zero retains 36 training tasks, while LURE trains on all 144, giving LURE four times as many distinct training tasks in that comparison. Consequently, the performance difference jointly reflects curriculum design, supervision quality, and task exposure. The within-LURE ablations do not provide the missing comparison against a self-play baseline with the same verifier access and matched task exposure. The experiments therefore establish a difference between complete training systems, but do not establish the specific advantage of the proposed pursuit–evasion formulation over a comparably supervised curriculum method.

Questions for authors

Refer to weakness

2 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1/2)

Thank you for the valuable feedback. We address each point below.


W1 [Unequal optimization settings]

Under a shared recipe, LURE leads every baseline, and the stabilized recipe benefits LURE most because round-anchored KL is a component of the method. Baselines follow R-Zero's settings in the paper. We retrain GRPO, GRPO-Zero, and R-Zero with learning rate 5×10−65\times10^{-6}, one PPO epoch, and KL coefficient 0.10.1 to the round-start snapshot. Table R1 reports the results.

Table R1. Qwen2.5-7B-Instruct unified accuracy (%) under the two recipes.

MethodRecipePhantomWikiIFEvalZebraLogic (Puzzle)Avg.
GRPOBaseline43.360.615.439.8
GRPOLURE51.761.215.042.6
GRPO-ZeroBaseline37.558.413.936.6
GRPO-ZeroLURE47.560.114.040.5
R-ZeroBaseline50.857.914.241.0
R-ZeroLURE56.759.916.044.2
LUREBaseline52.555.512.940.3
LURELURE65.861.216.948.0

LURE's recipe raises every baseline by 2.8 to 3.9 points, and LURE still outperforms the strongest baseline, R-Zero, by 3.8 points on average. This margin is more than five times LURE's run-to-run std (0.7, response to yJg9). The same recipe raises LURE by 7.7 points, and Tab. 11 attributes 6.3 of them to the round-anchored KL alone and only 1.4 to learning rate and epoch count. Round-anchored KL is a stated component of LURE that stabilizes the co-evolution of three policies (Prop. 3). The w/o stabilized recipe row is therefore an ablation of LURE and not a shared-recipe comparison. The shared-recipe comparison is Table R1, with 3.8 of the original 7.0 points remaining. The controller comparison in W3 separately isolates the difficulty selector.

We have added Table R1 to App. C.


W2 [ZebraLogic puzzle accuracies and the evaluation protocol]

ZebraLogic scores are means over three evaluation passes of the same 300 puzzles, so the denominator is 900. App. B omitted this aggregation step, and we have corrected it. Its “greedy decoding with a single rollout per instance” describes each pass, and the aggregation applies to all ZebraLogic results in Tabs. 1, 10 and 11. With krk_r solved puzzles in pass rr, A‾=13∑r=13100kr300\overline{A}=\frac{1}{3}\sum_{r=1}^{3}\frac{100k_r}{300}, rounded after averaging. For the two entries cited in the comment, LURE's per-pass counts [51, 50, 51] give 16.9% and GRPO's [46, 46, 47] give 15.4%. Greedy outputs can differ across passes, since small numerical differences in the logits can change near-tied choices. Table R2 lists all per-pass counts of Tab. 1. We have added Table R2 to App. C.

Table R2. Per-pass solved counts for the ZebraLogic (Puzzle) entries of Tab. 1, with the reported mean (%).

SettingMethodQwen2.5-7BLlama-3.1-8BGemma-2-9B
Base/[35, 35, 34] (11.6)[39, 38, 38] (12.8)[39, 40, 39] (13.1)
UnifiedGRPO[46, 46, 47] (15.4)[43, 42, 42] (14.1)[47, 47, 47] (15.7)
UnifiedGRPO-Zero[41, 42, 42] (13.9)[39, 39, 39] (13.0)[43, 43, 43] (14.3)
UnifiedR-Zero[43, 43, 42] (14.2)[44, 43, 44] (14.6)[45, 45, 46] (15.1)
UnifiedLURE[51, 50, 51] (16.9)[48, 47, 48] (15.9)[55, 54, 55] (18.2)
SpecialistGRPO[48, 48, 49] (16.1)[41, 41, 40] (13.6)[47, 47, 47] (15.7)
SpecialistGRPO-Zero[38, 38, 38] (12.7)[39, 39, 38] (12.9)[43, 43, 43] (14.3)
SpecialistR-Zero[45, 45, 45] (15.0)[41, 41, 40] (13.6)[45, 45, 45] (15.0)
SpecialistLURE[55, 55, 55] (18.3)[46, 45, 46] (15.2)[52, 53, 53] (17.6)

W3 [Isolating the pursuit-evasion formulation from supervision and task exposure]

Against curriculum methods with identical supervision and task exposure, the learned Challenger is 2.0 to 3.7 more accurate, and R-Zero with the same verifier access needs five times the rollouts to match LURE. We address this in three parts below:

(1) Comparably supervised curriculum methods. The two adaptive controllers of Table R3 share LURE's generator, competence summary, verifier, dense credit, recipe and 144 tasks per round, and differ from LURE only in the curriculum method. LURE exceeds the cycling controller by 2.0 points (before rounding) and the competence-summary controller by 3.7 points. Within LURE, replacing only the capture-frontier reward reduces the average by 5.8 points (Tab. 11).

(2) Self-play baseline with the same verifier access. R-Zero-V is R-Zero equipped with our generators and verifiers. Its Challenger outputs a task specification (environment, difficulty parameters, generator seed), and the environment generator instantiates the task together with the information required by the verifier. Gold answers remain hidden from the models. R-Zero-V keeps R-Zero's proposer-solver loop and acceptance band and uses verified capture wherever R-Zero used majority agreement.


Please see the subsequent Authors' Rebuttal (2/2) for remaining responses.

AuthorsOpenReview ↗

Authors' Rebuttal (2/2)

Continued: Authors’ Rebuttal (2/2)


(3) Task exposure. In the paper, both methods already receive 144 candidates per round and R-Zero uses more Solver rollouts. The 36-versus-144 difference comes from R-Zero's own filter (App. A, Fig. 7). We match exposure in both directions. R-Zero-V with four times as many candidates reaches LURE's number of training tasks. Subsampled LURE generates only as many candidates per round as R-Zero-V retains and computes its Challenger rewards from those same training rollouts. Table R3 reports all comparisons.

Table R3. Qwen2.5-7B-Instruct unified comparisons. Task counts are eight-round means, and total rollouts are probe rollouts plus eight training rollouts per retained task. LURE needs no probes because it reuses its training rollouts for Challenger rewards (Alg. 1).

MethodRecipeSolver supervisionTasks/roundProbes/roundTotal rollouts/roundAvg. (%)
R-ZeroBaselineMajority vote361152144041.0
R-ZeroLUREMajority vote40.61152147744.2
R-Zero-VLUREVerified capture37.41152145144.4
R-Zero-V, 4× candidatesLUREVerified capture150.14608580948.2
LURE, subsampled tasksLURECapture and progress37.4029945.9
Competence-summary controllerLURECapture and progress1440115244.3
Cycling controllerLURECapture and progress1440115245.9
LURELURECapture and progress1440115248.0

At matched exposure, LURE reaches 45.9 versus 44.4 for R-Zero-V with 299 instead of 1451 Solver rollouts per round. To reach LURE's accuracy, R-Zero-V needs four times as many candidates and approximately five times as many Solver rollouts per round.

The pursuit-evasion formulation therefore contributes beyond supervision and task exposure, through its learned Challenger and through reusing one set of verified rollouts for both curriculum and credit. We have added Table R3 to App. C.


We hope this fully addresses your questions and comments, and remain happy to engage in further discussion for the rest of the rebuttal period.

Reviewer_z7kAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LURE uses an environment verifier to drive both adaptive curriculum construction and multi-step Solver training. A Challenger selects task difficulty, a planner–executor Solver learns from terminal capture and stepwise verified progress, and round-start snapshots stabilize updates to the three policies. The evaluation covers three reasoning environments, three backbone families, difficulty extrapolation, cross-environment transfer, and OOD benchmarks.

Critical strengths
  1. Adaptive curricula and multi-step credit assignment are worthwhile problems in zero-data training. Addressing task difficulty mismatch and sparse outcome supervision within the same training process gives the paper a clear motivation.

  2. Connecting the Challenger and Solver through one verifier is a coherent design. The capture-frontier reward, verified progress, and round-anchored updates each address a specific training issue, with explicit objectives.

Critical weaknesses
  1. The scope of zero-data scalability depends on the available environment infrastructure. Section 3 assumes a task generator with adjustable difficulty and a stepwise progress verifier, both of which still require construction. The introduction should explain which environment design burdens LURE reduces relative to executor-grounded self-play, and which tasks can benefit from this change.

  2. The benefits for broader reasoning are uneven. In Table 3, LURE has an overall average of 36.9 versus 36.4 for R-Zero, but scores 31.0 versus 38.3 on IFBench and 39.6 versus 40.3 on the mathematics average. These differences call for a more specific account of the method's strongest use cases and capability trade-offs.

  3. The main learned decision in the pursuit–evasion formulation is a scalar difficulty, while a fixed generator produces task content. Removing the learned Challenger in Figure 4 supports adaptive task placement, but leaves its value over a simple adaptive controller unresolved. A difficulty controller using the same competence summary would be a direct alternative for assessing the formulation's added contribution.

  4. Proposition 1 establishes q = 0.5 as optimal for obtaining a nondegenerate outcome-only group-relative signal, whereas the actual Solver uses dense turn-level credit. The paper also explains that this credit remains informative when all rollouts fail. The connection between the capture frontier and actual learning progress therefore needs validation, for example by comparing target capture rates while retaining dense credit.

  5. Eqs. 6 and 12 use each turn's immediate reward as its advantage, with the reward for eventual success assigned only to the final turn. Necessary preparation or search can temporarily produce no verified progress and still incur the zero-progress penalty. This design depends on the progress verifier recognizing useful actions promptly, so it would help to examine credit for steps with delayed benefits in successful trajectories.

  6. The capture-preserving condition establishes an ordering of total raw trajectory credit, while the update uses normalization across turns and a token-level objective. With variable trajectory lengths, subtracting a common mean changes cumulative advantage according to length, so the raw-credit ordering does not directly carry over. The paper should explain how length handling and loss weighting maintain the priority of terminal success.

Questions for authors

None.

3 comments
AuthorsOpenReview ↗

Authors' Rebuttal (1/2)

Thank you for the valuable feedback. We address each point below.


W1 [Environment design burden relative to executor-grounded self-play]

LURE reduces the model's task-construction burden by assigning task instantiation to a reusable generator. In Absolute Zero, each model proposal must be executable code (Fig. 1b). In LURE, the Challenger selects a difficulty, the generator instantiates the task, and the Validity Gate checks it. Each environment needs a generator and a verifier with per-unit progress. Our verifiers check each hop, constraint, or grid cell, and IFEval reuses the official constraint builder (App. A). This setting covers knowledge-base multi-hop retrieval and logic-grid constraint satisfaction. PhantomWiki shows the largest gain, at 65.8 versus 50.8 for R-Zero. We have revised the introduction accordingly.


W2 [Uneven gains on broader reasoning]

LURE's gains concentrate on skills shared with the verifiable training environments, and IFBench tests constraint types outside its generator. IFBench includes 58 new constraint types, whereas LURE's generator uses 16 IFEval types (App. A) and R-Zero's Challenger generates free-form problem text. LURE leads R-Zero on IFEval, which uses the trained constraint types (Tab. 1), and on LiveBench-IF (Tab. 3). Mathematics is far transfer, since no training environment involves mathematical problem solving, and LURE ranks second in this group, above the base model. On 8-hop PhantomWiki, beyond the training range, LURE reaches 19.0% versus 9.0% for the best baseline (Tab. 2). Across three training runs, LURE's Qwen OOD overall equals that of R-Zero (36.5, response to mHXZ, Table R3) and its in-domain average is 7.5 points higher. LURE therefore yields its largest gains on multi-hop and logic-grid reasoning and on harder instances of them, with overall OOD performance above the base model. We have revised the transfer claims in the abstract, the introduction, Sec. 4.2 and the conclusion.


W3 [Learned Challenger versus a simple adaptive controller]

The competence-summary controller recovers part of the gain, and the learned Challenger outperforms both adaptive controllers. For each task, the competence-summary controller selects the level with the capture rate closest to 1/21/2 in the previous-round summary ζe\zeta_e and samples dd uniformly inside that level. The cycling variant rotates through valid levels within one step of the target and samples inside each selected level in the same way. Table R1 compares the selectors.

Table R1. Difficulty selectors under the same Solver, dense credit, recipe and 48 candidates per environment. Signatures (App. A) are counted within each environment, summed, and averaged over eight rounds.

Difficulty selectorAvg. (%)Distinct signatures per round
Programmatic sampling, no learned Challenger37.415.4
Competence-summary controller44.35.6
Controller cycling within one step of the target45.916.6
Learned Challenger (LURE)48.025.8

The learned Challenger exceeds the two adaptive controllers by 3.7 and 2.0 points (the latter computed before rounding). Among these three selectors, higher accuracy accompanies greater signature diversity. We have added Table R1 to App. C.


W4 [The capture frontier when dense credit is retained]

With dense credit retained, q⋆=1/2q^\star=1/2 gives the highest average among the tested targets. The paper already tests the frontier with dense credit retained: replacing the capture-frontier reward with a monotone one reduces the average by 5.8 points (Tab. 11). Prop. 1 concerns mixed terminal outcomes. Under its independence assumption their probability is ψG(q)=1−qG−(1−q)G\psi_G(q)=1-q^G-(1-q)^G, equal to 0.9920.992 at q=1/2q=1/2 and 0.5700.570 at q=0.1q=0.1 for G=8G=8 (Cor. 1). Groups with identical outcomes still provide turn-level signal: all-failure groups through varying progress, and all-success groups through terminal versus nonterminal rewards. We test the dense-credit objective directly by replacing min⁡(p,1−p)\min(p,1-p) in Eq. 4 with 12min⁡(p/q⋆,(1−p)/(1−q⋆))\tfrac12\min\bigl(p/q^\star,(1-p)/(1-q^\star)\bigr), which keeps the peak reward unchanged and recovers the original reward at q⋆=1/2q^\star=1/2. Table R2 reports the results.

Table R2. Target capture rates with dense credit retained.

Target q⋆q^\starAvg. (%)Groups with identical outcomes (%)
0.347.437.4
0.5 (LURE)48.026.0
0.746.237.6

Both alternative targets lower accuracy and raise the share of identical-outcome groups. We have added Table R2 to App. C.


Please see the subsequent Authors' Rebuttal (2/2) for remaining responses.

AuthorsOpenReview ↗

Authors' Rebuttal (2/2)

Continued: Authors’ Rebuttal (2/2)


W5 [Credit for steps with delayed benefits]

LURE separates trajectory-level planning credit from turn-level execution credit, and the cost of a stalled turn is small by design. The Planner receives rplan(τ)=∑iΦi=ϕTτr^{\mathrm{plan}}(\tau)=\sum_i\Phi_i=\phi_{T_\tau} (Eq. 9), which includes progress realized later in its segments. For the Executor, the zero-progress cost is assessed after a completed turn, and all tokens of that turn inherit one advantage (Eq. 12). The cost is λcred=0.0125\lambda c_{\mathrm{red}}=0.0125 per stalled turn and at most 0.23750.2375 per trajectory, compared with a capture reward of 1 (Cor. 2). It removes the incentive to lengthen trajectories with stalled turns (Prop. 2(iii)), and removing it lowers accuracy by 3.6 points (Tab. 11). As a delayed-credit control, we replace immediate Executor rewards with return-to-go ∑t′≥tγt′−trt′\sum_{t'\ge t}\gamma^{t'-t}r_{t'} before group normalization and keep the Challenger and Planner objectives unchanged. Table R3 reports the results.

Table R3. Zero-progress turns in capturing trajectories of LURE (Qwen2.5-7B-Instruct unified) and accuracy with return-to-go credit.

QuantityPhantomWikiIFEvalZebraLogic
Zero-progress turns in capturing trajectories (%)26.632.012.8
Followed by progress within two turns (%)97.996.5100.0
Accuracy (%), immediate credit (LURE)65.861.216.9
Accuracy (%), return-to-go credit, γ=0.9\gamma=0.969.260.816.3

Return-to-go raises PhantomWiki by 3.3 points (computed before rounding) and lowers IFEval and ZebraLogic by 0.4 and 0.6. Return-to-go credit can therefore replace immediate credit with no other change to LURE, and it further improves multi-hop reasoning. Immediate credit remains our standard setting for all environments because it localizes supervision to verified progress and needs no discount hyperparameter. We have added Table R3 to App. C.


W6 [Normalization, trajectory length, and the priority of terminal success]

The token objective retains the higher advantage of the tokens in a capturing final turn, and the cumulative trajectory order additionally depends on length. We address this in three parts: priority in the update, trajectory length, and token weighting. Let μ,σ\mu,\sigma be the mean and std of rtr_t over G(x)\mathcal{G}(x) and At=(rt−μ)/σA_t=(r_t-\mu)/\sigma (Eq. 12).

(1) Priority in the update. A capturing final turn has rT≥1−λcred=0.9875r_T\ge 1-\lambda c_{\mathrm{red}}=0.9875, and a non-capturing turn has rt≤λ=0.25r_t\le\lambda=0.25 (Eq. 6). The shared affine map preserves this order (Prop. 4(i)). Each generated token inherits its turn advantage. At the round-start snapshot, clipping is inactive and the KL gradient is zero (Prop. 3), so each token's policy-gradient coefficient is its advantage divided by the total number of tokens in the batch. Tokens in a capturing final turn therefore retain higher coefficients than non-capturing tokens in their group.

(2) Trajectory length. Summing gives ∑tAt=(U(τ)−Tτμ)/σ\sum_t A_t=(U(\tau)-T_\tau\mu)/\sigma. For capturing τ\tau and non-capturing τ′\tau', Prop. 2(ii) gives ∑tAt(τ)−∑tAt(τ′)≥[0.7625−μ(Tτ−Tτ′)]/σ\sum_t A_t(\tau)-\sum_t A_t(\tau')\ge[0.7625-\mu(T_\tau-T_{\tau'})]/\sigma, with 0.7625=1−λcred(Tmax⁡−1)0.7625=1-\lambda c_{\mathrm{red}}(T_{\max}-1). The cumulative order therefore holds for μ(Tτ−Tτ′)<0.7625\mu(T_\tau-T_{\tau'})<0.7625. Under the task horizons Te≤8T_e\le 8 (App. A) the margin is 0.91250.9125 and ∣μ∣<0.130|\mu|<0.130 suffices.

(3) Token weighting. Let ntn_t count tokens in turn tt, Nτ=∑tntN_\tau=\sum_t n_t, and Utok(τ)=∑tntrtU_{\mathrm{tok}}(\tau)=\sum_t n_t r_t. Then Stok(τ)=[Utok(τ)−μNτ]/σS_{\mathrm{tok}}(\tau)=[U_{\mathrm{tok}}(\tau)-\mu N_\tau]/\sigma. Dividing by the total number of tokens in the batch rescales all terms equally, so the length effect remains. Non-capturing stalled turns attain the minimum raw reward −λcred-\lambda c_{\mathrm{red}} and receive (−λcred−μ)/σ<0(-\lambda c_{\mathrm{red}}-\mu)/\sigma<0 in non-degenerate groups, so each added stalled turn contributes a negative advantage. Table R4 reports the measured order.

Table R4. Share (%) of within-group pairs of capturing and non-capturing trajectories with preserved order, by length gap Tτ−Tτ′T_\tau-T_{\tau'}.

Summation≤0\le 01 to 45 to 9All
Turn-level sums100.0098.5586.3399.28
Token-weighted sums99.9593.4256.8396.94
Trajectory pairs (n)6365385813910362

Under both summations the order is preserved for more than 96% of pairs, and the 5 to 9 bin holds only 1.3% of them. The theory guarantees that every capturing trajectory receives a larger total raw reward than any non-capturing trajectory, and that its final-turn tokens receive larger advantages in the update. We have added this analysis to App. D.


We hope this addresses your concerns and are happy to discuss further.

Thank you for the additional experiments. Table R2 addresses my empirical concern about the capture frontier, and Table R1 provides the requested controller comparison. I still have some questions. In your response to W5, you acknowledge that return-to-go “further improves multi-hop reasoning.” Table R3 also gives a higher three-task average, approximately 48.77 versus 47.97. How does retaining immediate credit address the original delayed-credit concern? Regarding W6, the argument that “the cost of a stalled turn is small by design” concerns raw rewards. How large are the penalties on useful zero-progress turns after normalization? Table R4 also shows substantial ordering reversals in the small 5–9 length-gap subset, leaving the length-related concern unresolved.

Reviewer_mHXZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes LURE, a pursuit-evasion self-play framework for zero-data multi-step reasoning. It uses environment verifiers to adapt task difficulty and provides dense training signals through intermediate progress. Experiments across three task environments and three model backbones show that LURE improves multi-step reasoning performance and offers some gains in cross-task generalization.

Critical strengths
  1. Unified and coherent environment-feedback mechanism. LURE uses environment verifiers for both task difficulty adjustment and solver optimization. The Solver's success rate guides the Challenger in generating appropriately challenging tasks, while verifiable intermediate progress provides dense rewards for the Solver. This design integrates task generation and solving without requiring a separately trained reward model.
  2. Comprehensive experimental evaluation from multiple perspectives. The paper evaluates LURE across three different task environments and three model backbones, with additional ablation studies, difficulty extrapolation, cross-environment transfer, and evaluation on 13 OOD benchmarks. These experiments demonstrate LURE's effectiveness in multi-step reasoning and provide evidence of its generalization potential.
Critical weaknesses
  1. Strong dependence on pre-engineered programmable environments. Although LURE is presented as a zero-data training framework, its implementation relies on task-specific structured generators, difficulty parameters, and environment verification logic. These environments must provide adjustable task difficulty and verifiable intermediate progress, requiring substantial prior human design and engineering effort. This limits the applicability of LURE to open-ended tasks without structured generation mechanisms or explicit verification criteria, narrowing the scope of its zero-data setting.
  2. Insufficient empirical evidence for generalization. Although the paper includes cross-environment transfer and OOD evaluations, the results do not consistently demonstrate robust improvements. For example, LURE outperforms R-Zero by only 0.5 points on average across 13 OOD benchmarks and performs worse on mathematical tasks. Cross-environment transfer results are also mixed. Furthermore, the absence of variance estimates or confidence intervals across multiple random seeds makes it difficult to assess the statistical reliability of the reported improvements. These limitations weaken the claims regarding LURE's generalization capabilities.
Questions for authors
  1. LURE's Dense Credit relies on the environment providing monotonic and verifiable intermediate progress signals (\phi_t). Would LURE remain effective if the environment could only provide coarse-grained feedback or terminal binary success/failure signals? Have the authors conducted controlled experiments that remove intermediate progress feedback and retain only terminal rewards?
  2. Task generation in LURE relies on predefined structural difficulty parameters, such as the number of hops in PhantomWiki and the number of constraints in IFEval. For unstructured tasks without explicit structural parameters, how would the Challenger determine task difficulty and construct tasks? Could the authors clarify the requirements that the current framework imposes on task generators and environment structures, and whether the experiments provide evidence of its applicability to such tasks?
1 comment
AuthorsOpenReview ↗

Authors' Rebuttal

Thank you for the valuable feedback. We address each point below.


W1 & Q2 [Dependence on programmable environments and unstructured tasks]

LURE requires two components per environment, a generator that accepts a difficulty input and a verifier, and the difficulty input needs no known order. Zero-data means that training uses no human-curated tasks, process annotations, or learned reward model, as in R-Zero and Absolute Zero, and Absolute Zero likewise relies on a code executor (Fig. 1b). We address this in three parts: environment requirements, the difficulty input, and scope.

(1) Environment requirements. Each environment supplies a generator x=ge(d,ω)x=g_e(d,\omega) and a verifier that returns capture R(τ)R(\tau) and progress ϕt\phi_t, both implemented once per environment rather than per task. Our generators map dd to a structural parameter with five levels (Tab. 5), and ϕt\phi_t counts verified hops, constraints, or grid cells. The IFEval generator reuses the official constraint builder (App. A).

(2) The difficulty input need not be ordered. The frontier term min⁡(p,1−p)\min(p,1-p) (Eq. 4) depends only on the measured capture rate, so it does not assume that a larger dd yields a harder task. We randomly permute the five levels of each generator and remove the order hint from the Challenger prompt. Table R1 reports the results.

Table R1. Qwen2.5-7B-Instruct unified accuracy (%) with ordered and permuted levels.

Difficulty mappingPhantomWikiIFEvalZebraLogic (Puzzle)Avg.
Ordered levels (LURE)65.861.216.948.0
Randomly permuted levels65.060.416.747.4
Programmatic sampling, no learned Challenger40.058.014.137.4

Permutation reduces the average by only 0.6 points, so the Challenger locates the frontier from capture statistics, not from a manually ordered parameter. We have added Table R1 to App. C.

(3) Scope. LURE targets RLVR settings with a task generator and an automatic verifier. Without an explicit structural parameter, any generator input that changes the capture rate can serve as dd, and its order need not be known in advance. Applicability to unstructured tasks depends on providing these two components.


Q1 [Coarse-grained or terminal-only progress signals]

LURE remains effective with coarse or terminal-only feedback, and its curriculum uses no progress signal. The Challenger is trained from binary capture alone (Eq. 4), and only Solver credit uses ϕt\phi_t. The w/o dense credit row of Tab. 11 sets λ=0\lambda=0, and a second terminal-only variant also replaces the Planner reward (Eq. 9) with R(τ)R(\tau). The three-level variant quantizes ϕt\phi_t to {0,0.5,1}\{0,0.5,1\}. Table R2 reports the results.

Table R2. Qwen2.5-7B-Instruct unified accuracy (%) under different progress signals.

Progress signalPhantomWikiIFEvalZebraLogic (Puzzle)Avg.
Full ϕt\phi_t (LURE)65.861.216.948.0
Three-level ϕt\phi_t63.360.416.346.7
Terminal-only Executor, λ=0\lambda=057.557.915.143.5
Terminal-only Executor and Planner55.058.815.743.1

Three-level progress lowers the average by only 1.3 points, so a fine-grained ϕt\phi_t is not required. With terminal-only signals, LURE remains approximately 10 points above the base model. A coarse verifier therefore retains most of the benefit of progress feedback. We have added Table R2 to App. C.


W2 [Generalization evidence and statistical reliability]

LURE's in-domain advantage is reliable across independent training runs, and it is obtained without loss of OOD performance. In the Qwen comparison, LURE has the highest near-domain, logic and other-reasoning averages, and ranks second in mathematics, above the base model (Tab. 3). Across specialists, LURE is best or second-best in every cross-environment cell, and its ZebraLogic and IFEval specialists improve both other environments. The cells below the base model come from PhantomWiki specialization, which lowers IFEval for every method (Fig. 3).Table R3 summarizes three training runs per method, the original and two new independent runs, with per-run entries in the response to yJg9.

Table R3. Qwen2.5-7B-Instruct unified accuracy (%) over three training runs, mean ± std.

MethodPhantomWikiIFEvalZebraLogic (Puzzle)Avg.OOD OverallGap to LURE (Avg.), 95% CI
R-Zero50.0 ± 0.857.4 ± 0.915.9 ± 2.341.1 ± 0.236.5 ± 0.2[5.9, 9.1]
LURE67.8 ± 2.160.1 ± 1.018.0 ± 0.948.6 ± 0.736.5 ± 0.6/

The in-domain gap of 7.5 points has an interval that excludes zero, and on the OOD benchmarks both methods reach 36.5, above the base model. We have revised the transfer claims in the abstract, the introduction, Sec. 4.2 and the conclusion, and added Table R3 to App. C.


We hope this addresses your concerns and are happy to discuss further.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

Dear all,

Thank you for handling our submission. We have posted responses to each reviewer and summarize the main points below.


Paper Overview and Strengths

[Paper Overview]: This paper proposes LURE, a pursuit-evasion self-play framework for zero-data multi-step reasoning, in which one environment verifier drives both sides of the game. The Challenger places tasks at the Solver's capture frontier using verified success, and the planner-executor Solver learns from capture-anchored dense credit derived from verified progress. Across three environments and three backbones, LURE achieves the best multi-step reasoning average in every setting, extrapolates beyond its training difficulty, and improves OOD performance over the base model.

[Strengths]: We are encouraged that the reviewers find the shared-verifier design coherent (mHXZ, z7kA, DGGS, yJg9), the problem well motivated (z7kA), the capture frontier mathematically justified (DGGS), and the evaluation broad (mHXZ, yJg9), and recognize the gains on PhantomWiki (DGGS, yJg9).


Additional Experiments for Concerns

  • [Shared Recipe] (DGGS W1, yJg9 W1): With all baselines retrained under LURE's optimizer settings (Table R1 for DGGS, R1 for yJg9), LURE leads the strongest baseline by 3.8 points, more than five times its run-to-run std.
  • [Verifier Access and Exposure] (DGGS W3, yJg9 W1): R-Zero with our generators and verifiers (Table R3 for DGGS, R1 for yJg9) reaches 44.4 against 45.9 for LURE at matched exposure, and needs approximately five times the rollouts to match LURE.
  • [Adaptive Controllers] (z7kA W3, DGGS W3): The learned Challenger exceeds two adaptive controllers with identical supervision by 3.7 and 2.0 points (Table R1 for z7kA, R3 for DGGS).
  • [Target Capture Rate] (z7kA W4): With dense credit retained, q⋆=1/2q^\star=1/2 gives the highest average among the targets 0.3, 0.5 and 0.7 (Table R2 for z7kA).
  • [Unordered Difficulty] (mHXZ W1 & Q2): Randomly permuting the generator levels changes the average by only 0.6 points (Table R1 for mHXZ).
  • [Coarse Progress Signals] (mHXZ Q1): Three-level progress lowers the average by only 1.3 points, and with terminal-only signals LURE remains approximately 10 points above the base model (Table R2 for mHXZ).
  • [Delayed Credit] (z7kA W5): At least 96.5% of zero-progress turns in capturing trajectories are followed by progress within two turns, and return-to-go credit raises PhantomWiki by 3.3 points (Table R3 for z7kA).
  • [Separate Planner] (yJg9 W1 & Q1): Separate parameters lead a single Solver by 2.7 points at equal rounds and by 0.5 points at matched GPU hours (Table R2 for yJg9).
  • [Training Runs] (mHXZ W2, yJg9 W2 & Q2): Over three training runs (Table R3 for mHXZ, R3 for yJg9), LURE's in-domain average is 48.6 ± 0.7 against 41.1 ± 0.2 for R-Zero, a 7.5-point gap with 95% CI [5.9, 9.1], and both reach 36.5 on OOD.
  • [OOD Backbones] (yJg9 Q3): On Llama and Gemma, LURE remains above the base model overall and attains the highest logic score (Table R4 for yJg9).

Summary of Responses to Reviewers' Questions

  • [ZebraLogic Protocol] (DGGS W2, yJg9 W2 & Q2): Each score averages three evaluation passes over the same 300 puzzles, so 16.9% and 11.6% are 152/900 and 104/900. App. B omitted this aggregation step, and we have corrected it. Per-pass counts are listed (Table R2 for DGGS).
  • [Environment Requirements] (mHXZ W1 & Q2, z7kA W1): LURE needs one generator and one verifier per environment, not per task. The Challenger only selects a difficulty, whereas executor-grounded self-play requires every proposal to be executable code.
  • [Transfer] (mHXZ W2, z7kA W2): Gains concentrate on skills shared with the training environments and on harder instances of them, and the in-domain advantage is obtained without loss of OOD performance.
  • [Normalization] (z7kA W6): The shared affine map preserves the higher advantage of capturing final-turn tokens, and the trajectory order is preserved for more than 96% of pairs (Table R4 for z7kA).
  • [OOD Evaluation] (yJg9 Q3): We use the Executor weights of one checkpoint selected on a held-out validation set, never on OOD scores.

Revision During Rebuttal

All changes are highlighted in blue in the revised PDF: the transfer claims (abstract, Sec. 1, Sec. 4.2, conclusion), the comparison with executor-grounded self-play (Sec. 1), an overview of the new analyses (Sec. 4.3), the evaluation protocol (App. B), all new experiments (Tabs. 12 to 22, App. C), and capture priority under group normalization (Tab. 23, App. D).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.