acceptodds
Under review as a conference paper at ICLR 2027

Learning Retrieval Plans from Sibling Execution Outcomes

Abstract

Multi-hop question answering requires agents to retrieve evidence and use earlier findings to guide later searches and reasoning. Yet different workflows can yield the same correct answer, so answer labels alone do not define a unique target for supervising intermediate actions. To address this problem, we propose SIPPO (Sibling-Informed Planner Preference Optimization), which learns executable workflows through execution-grounded supervision and final-answer rewards. A shared role-conditioned policy generates a directed acyclic graph whose dependencies resolve against node memory, connecting actions to states from earlier execution. We construct execution-grounded supervision by executing teacher-proposed workflows on multi-hop question-answering data, verifying evidence support and answer correctness, and pairing each recorded action with the state available at that step. We instantiate RLOO at workflow granularity (Workflow-RLOO), assigning advantages from final-answer Exact Match (EM) rewards to every generated role action in a workflow. Sibling Execution Outcomes are the terminal EM rewards for on-policy workflows sampled for the same question. Planner Sibling Contrast (PSC) uses reward differences to compare planner responses from EM-correct and EM-incorrect workflows, without additional intermediate labels. Across four multi-hop benchmarks comprising 22,517 answerable questions, SIPPO achieves the highest macro EM and macro F1 in the main comparison in both backbone settings. Its macro EM/F1 are 0.3362/0.4212 on Qwen2.5-3B and 0.3813/0.4701 on Qwen3-4B. Adding PSC to Workflow-RLOO improves macro EM by 2.72 and 7.01 percentage points, respectively; the paired 95% confidence interval for the 3B gain excludes zero. The 3B gain accompanies fewer generated tokens and more retrieval calls. Frozen-policy diagnostics show that PSC gradients respond to the terminal EM reward labels of same-question workflows.

Primary area:
reinforcement learning
Keywords:
agentic RAG, multi-hop question answering, reinforcement learning, workflow learning, retrieval-augmented generation, credit assignment, LLM agents

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ohW5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes SIPPO for learning retrieval workflows in multi-hop question answering. A shared model generates and executes graph-based plans. Training combines supervised workflow examples, final-answer reinforcement learning, and planner preferences from successful and unsuccessful sibling executions. Experiments cover four benchmarks and two backbones, with reported improvements in answer accuracy.

Critical strengths

The workflow representation and execution procedure are clearly described. The matched training comparisons help separate reinforcement learning gains from the added planner objective. Evaluation covers 22,517 questions, and the paper reports paired confidence intervals and retrieval costs. I also appreciate that the authors discuss uneven gains across the different benchmarks.

Critical weaknesses

The main concern is training robustness. Appendix G repeats decoding, but does not repeat model training. The 3B micro EM improvement is only 0.71 points, while much of the macro gain comes from Bamboogle. Frozen-policy gradient checks also do not show whether the planner contrast is better than simpler alternatives.

Questions for authors

Could the authors repeat both training conditions with several random seeds? Would the improvement remain under equal retrieval budgets? Please compare PSC with stronger weighting of the existing planner loss. Could you also freeze the downstream policy and compare plans, to test whether planner changes alone improve final answer accuracy?

3 comments

Q1: Repeat both training conditions with multiple random seeds

We conducted three independent RL training runs per condition, rather than repeated decoding of one checkpoint. Overall original-order EM is comparable, while PSC improves option-order robustness on comparison questions in every seed.

Changes from submission. We rebuilt execution-grounded SFT after identifying mismatches between training inputs and execution states and repeated supervision of earlier assistant turns. We re-rendered targets from executed states, masked earlier responses, added stronger-teacher HotpotQA/MuSiQue trajectories, and re-screened RL questions (4,477 retained). We increased siblings from four to eight, localized PSC to differing plan steps after alignment, and scaled its auxiliary gradient relative to Planner-RLOO rather than using a fixed 0.02 coefficient. Same-question comparisons, terminal EM-derived supervision, and the softplus loss remain.

Matched setup. Both conditions share the updated SFT initialization, 4,477 RL questions, eight siblings per question, and 300 training steps (16 questions/128 workflows per step). Each seed independently controls question order, rollout sampling, and process randomness. All models are evaluated on the full 22,517-question suite.

Micro EMSeed 1Seed 2Seed 3MeanRange
Workflow-RLOO0.39860.41990.39440.4042.6 pp
PSC0.40550.40210.40250.4030.3 pp

Both methods improve over SFT (0.349) by approximately 5.5 pp. PSC has a smaller observed range across training seeds and uses 11% fewer retrievals.

Order-balanced evaluation. We identified seed-dependent option-position preferences in Workflow-RLOO. For 4,897 eligible comparison questions, we evaluate both original and swapped option orders, keeping the gold answer fixed, and average their EM. Other questions retain original-order EM, preserving the full-suite weighting.

Full-suite micro EM (%)Seed 1Seed 2Seed 3Mean
RLOO, original39.8641.9939.4440.43
PSC, original40.5540.2140.2540.34
RLOO, order-balanced38.9241.3739.9540.08
PSC, order-balanced40.7640.3040.3140.46
PSC − RLOO (pp)+1.84−1.07+0.36+0.38

The mean full-suite order-balanced gain is +0.38 pp (95% CI [0.10, 0.66]), but it is not consistent across seeds: seed 2 favors RLOO (−1.07 pp). On the 4,897 comparison questions, PSC improves order-balanced EM by +1.47 [0.44, 2.55], +2.66 [1.62, 3.71], and +3.12 [2.15, 4.08] pp across seeds. All three paired CIs exclude zero. These CIs use 10,000 paired bootstrap resamples of test questions, not training-seed uncertainty.

Conclusion. The independent runs support PSC's option-order robustness on comparison questions and lower retrieval cost at comparable overall accuracy, not a consistent general full-suite EM improvement.

Revised paper. Table 2; Section 4.4, Tables 4–5 and Figure 4; Appendix G, Table 14.

Q2: Does the improvement remain under the same retrieval budget?

PSC achieves comparable EM with fewer realized retrieval calls, although we did not enforce an identical hard retrieval cap. Both methods use the same training budget and Wiki18/E5 top-3 retrieval environment.

Three-seed mean (22,517 questions)Micro EMRetrievals/questionModel calls/question
SFT0.3492.576.57
Workflow-RLOO0.4042.376.17
PSC0.4032.11 (−11.0%)5.53 (−10.4%)

Conclusion. PSC maintains comparable accuracy while making 11% fewer retrieval calls and 10.4% fewer model calls. This establishes lower realized inference cost, not performance under an identical hard retrieval cap.

Revised paper. Section 4.4, Figure 3; Appendix J, Table 18.

Q3: Compare against more strongly weighting the existing Planner loss

Setup. The control adds an extra term that amplifies Workflow-RLOO's own Planner outcome loss. Its gradient norm is set in each minibatch by the same rule as PSC's extra term (the same proportion of the Planner-RLOO gradient norm, times the same coverage factor), added before gradient clipping, followed by identical clipping and optimizer updates. Only the direction differs.

Seed 1, all 22,517 questions (micro)EMEM difference from Workflow-RLOO (pp)F1
Workflow-RLOO0.399-0.429
+ Increased Planner loss (budget matched)0.388-1.08 [-1.72, -0.50]0.430
PSC0.406+0.69 [0.11, 1.27]0.433
PSC - increased Planner loss+1.78 [1.22, 2.31]+0.29 [-0.16, +0.74]

Conclusion. Under the same extra gradient budget, the reweighting control does not improve on Workflow-RLOO's EM, and PSC exceeds it by 1.78 points. Simply amplifying Planner outcome supervision does not reproduce PSC's effect.

Revised paper. Section 4.4, Table 3.

Q4: Freeze the downstream executor and compare plans alone

Setup. We fix the executor to the SFT model, the common starting point of both RL conditions, and use it to execute the plans generated by Workflow-RLOO and PSC (three seeds each) on all 22,517 questions with greedy decoding. The two plans for a question use the same hardware and retrieval service.

Executor fixed to SFT: PSC plans - Workflow-RLOO plans (EM pp)Seed 1Seed 2Seed 3Mean
All 22,517 questions-1.66-2.07+0.42-1.10
Non-comparison questions (15,239)+0.61-0.81-1.35-0.52
Comparison questions (7,278)-6.40-4.71+4.14-2.33

The frozen-executor analysis includes 7,278 comparison questions; the option-order test evaluates 4,897 eligible comparison questions. On non-comparison questions, the gap between the two kinds of plans stays below 1.5 points in every seed and changes direction across seeds. On comparison questions, the difference comes from how the plans are written. For example, on a 2Wiki bridge-comparison question:

Workflow-RLOO's plan: 1. What nationality is the director of film A? 2. What nationality is the director of film B? 3. Are the two directors from the same country?

PSC's plan: 1. Who directed film A? 2. Who directed film B? 3. Are #1# and #2# from the same country?

PSC's plan is more concise and leaves the attribute lookup to the final step for the executor. The executor trained jointly with PSC completes this step; the SFT executor does not:

2Wiki bridge-comparison EM (%)Seed 1Seed 2Seed 3
PSC plans, PSC's own executor51.155.052.6
PSC plans, SFT executor35.935.948.7

Interpretation. The frozen-executor experiment answers an important but distinct question: whether plans learned under joint RL training transfer to the shared pre-RL SFT executor. Under this intervention, PSC plans do not outperform Workflow-RLOO plans, and we report this negative result explicitly.

However, this result should not be interpreted as evidence that PSC fails to improve the jointly trained workflow policy. In SIPPO, the Planner and execution roles are optimized together within a shared model. The comparison-question analysis illustrates this distinction: PSC plans can delegate attribute resolution to later execution steps, requiring execution behavior learned during joint training. On exactly these comparison questions, the complete PSC-trained policy is the more robust one: when the two options are swapped, it achieves significantly higher order-balanced EM than Workflow-RLOO in all three independent training seeds (+1.47, +2.66, and +3.12 pp; all 95% CIs exclude zero; see W).

We therefore distinguish two claims: PSC does not demonstrate superior standalone plan transfer to a frozen SFT executor, but the complete PSC-trained policy exhibits more robust option-order behavior and comparable overall accuracy with fewer retrievals. Our empirical contribution concerns the latter, not an independently improved Planner under an unchanged executor. We report this experiment, the question-type breakdown, and the plan example in Section 4.4 and Appendix G, Table 15.

W: Frozen-gradient checks do not establish that PSC is better than simpler methods

We agree that gradient diagnostics alone are insufficient, and we added two direct tests.

Training control. Under a matched extra gradient budget (same rule as in Q3), amplifying the Planner outcome loss does not reproduce PSC's EM:

Seed 1, all 22,517 questions (micro)EMEM difference from Workflow-RLOO (pp)
Workflow-RLOO0.399-
+ Increased Planner loss (budget matched)0.388-1.08 [-1.72, -0.50]
PSC0.406+0.69 [0.11, 1.27]
PSC - increased Planner loss+1.78 [1.22, 2.31]

Behavioral test. The two trained models reach comparable overall EM (PSC 40.3%, Workflow-RLOO 40.4%), so we examined how their decisions differ. Workflow-RLOO shows a strong option-position preference on comparison questions that changes with the training seed: on 2Wiki bridge-comparison questions it selects the second-listed option in 90.0%, 71.8%, and 11.7% of cases across the three seeds, versus 55% for SFT and 37.5%, 43.1%, and 27.9% for PSC. We therefore swap the two options in 4,897 comparison questions of the form "..., A or B?", keeping the answer unchanged, and evaluate each model under both orders (95% paired bootstrap CIs):

SeedWorkflow-RLOO: original → swapped EM (%)PSC: original → swapped EM (%)Same answer under both orders: Workflow-RLOO / PSCOrder-balanced EM: PSC - Workflow-RLOO (pp)
163.5 → 54.8 (-8.70)59.6 → 61.5 (+1.92)42.0% / 56.4%+1.47 [0.44, 2.55]
265.3 → 59.6 (-5.72)64.7 → 65.6 (+0.84)58.1% / 69.8%+2.66 [1.62, 3.71]
360.4 → 65.1 (+4.70)65.5 → 66.1 (+0.61)48.9% / 64.0%+3.12 [2.15, 4.08]

Although PSC and Workflow-RLOO achieve comparable overall EM (40.3% vs. 40.4%), PSC consistently achieves higher order-balanced EM on comparison questions across all three independent training seeds (+1.47, +2.66, and +3.12 percentage points; all 95% confidence intervals exclude zero), while exhibiting substantially lower sensitivity to option order.

Conclusion. The two tests answer your concern from complementary sides, without relying on frozen-gradient diagnostics: in training, PSC outperforms the matched simpler alternative (stronger Planner outcome loss) by 1.78 EM points; in behavior, PSC achieves significantly higher mean order-balanced accuracy on the full evaluation suite (+0.38 pp, 95% paired bootstrap CI [0.10, 0.66]), with consistent gains across all three seeds on comparison questions. This complements the matched Planner-reweighting control and demonstrates benefits beyond simply strengthening outcome supervision. The full-suite gain is a mean over seeds rather than a per-seed result: in seed 2, Workflow-RLOO remains ahead on the full suite (-1.07 pp; see Q1).

Revised paper. Control: Section 4.4, Table 3. Option-order test and order-balanced EM: Section 4.4, Tables 4–5 and Figure 4.

Summary of the requested experiments. We have now conducted the independent training repetitions, the retrieval-cost analysis, the gradient-budget-matched Planner-reweighting control, and the frozen-executor evaluation requested in your review. The results show that (i) PSC's order-balanced accuracy advantage on comparison questions holds across all three independent training seeds, and its mean full-suite order-balanced EM is higher by +0.38 pp (95% paired bootstrap CI over test questions [0.10, 0.66]), although this gain is not consistent across training seeds; (ii) PSC maintains comparable overall accuracy with fewer retrieval and model calls; (iii) simply strengthening the existing Planner loss does not reproduce PSC's seed-1 improvement; and (iv) PSC plans do not independently outperform Workflow-RLOO plans under a frozen SFT executor, clarifying the role of Planner-executor co-adaptation.

We have revised our interpretation accordingly and hope this more complete evidence addresses your primary soundness concerns.

Reviewer_aNnpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SIPPO trains executable retrieval plans using final-answer outcomes. A shared policy handles planning and execution roles, first learning from execution-grounded demonstrations. Workflow-RLOO then applies terminal EM supervision across roles, and Planner Sibling Contrast adds a preference loss between plans associated with successful and unsuccessful executions of the same question.

Critical strengths

The executable workflow is a real system contribution: dependent queries use earlier node conclusions, and training actions are paired with the states actually available when they occurred. The internal comparison shares SFT initialization, data, rollout exposure, and execution protocol. PSC improves macro EM over Workflow-RLOO by 2.72 points on 3B, with a paired 95% interval of [1.00, 4.49], and by 7.01 points on 4B. The paper also reports dataset-level effects and cost trade-offs, and appropriately limits its causal interpretation of workflow patterns.

Critical weaknesses

The main question is whether PSC's pairwise structure adds value beyond simply strengthening existing planner supervision. Workflow-RLOO already updates planner tokens using the same final EM labels. The frozen-gradient diagnostics show sensitivity to those labels, but do not rule out this simpler explanation. The implemented objectives are not identical: weighting, normalization, and clipping matter. That is precisely why a planner-only outcome-loss reweighting control, with comparable normalization and gradient budget, would be informative.

Training robustness remains unclear. Appendix G's three seeds vary stochastic decoding, rather than independently retraining both conditions. The paired intervals support differences between the evaluated checkpoints, but do not measure training variation. This matters on 3B: Bamboogle contributes 1.80 of the 2.72 macro-EM points through 9 net extra correct answers, while the full-suite micro-EM gain is about 0.71 points. The larger 4B gain is encouraging, but does not replace training-repeat evidence. This uncertainty does not negate the reported positive results.

Questions for authors

Could you compare PSC with a Workflow-RLOO variant that only increases the planner outcome-loss contribution, matching normalization and gradient budget under the same rollout budget? How much of the EM/F1 gain remains?

How many independent training runs are available for each condition? Please report existing matched repeats; if none are available, could a small paired replication test whether the main gain survives training randomness?

3 comments

Thank you for pointing out that gradient checks cannot replace training controls and for specifying the Planner-reweighting comparison.

Q1: Does PSC's pairwise structure offer value beyond increasing Planner outcome loss? How much of the EM/F1 gain remains?

Yes for EM in the matched seed-1 control; not conclusively for F1 or across all seeds. PSC exceeds gradient-budget-matched Planner outcome-loss reweighting by +1.78 EM percentage points (95% paired bootstrap CI [1.22, 2.31]), while its F1 advantage is +0.29 pp (CI [-0.16, 0.74]).

Updated experimental setup (changes from submission). We rebuilt execution-grounded SFT after finding repeated supervision of earlier assistant turns and mismatches between SFT inputs and executed states. Examples were re-rendered from actual runtime states with subquestion-step plans, earlier assistant responses were loss-masked, and stronger-teacher trajectories were added for HotpotQA and MuSiQue. We re-screened the RL questions using this SFT (4,477 retained) and retrained every condition from the same initialization. We also increased siblings per question from four to eight (up to 16 rather than four success–failure pairs), replaced whole-plan PSC scoring with aligned differing-step scoring, and set auxiliary strength relative to the Planner-RLOO gradient rather than a fixed 0.02 coefficient. Same-question pairing, terminal EM labels, and softplus preference loss remain unchanged. Each run uses 16 questions (128 workflows) per step for 300 steps; all compared conditions share the same setup.

On frozen training batches, whole-plan PSC is nearly parallel to the Planner-RLOO gradient (median cosine 0.895), whereas differing-step PSC is not (0.682), which is why the refined form is the right one to test against reweighting.

We trained the control you describe, "planner-only outcome-loss reweighting, with comparable normalization and gradient budget":

Both the control and PSC add an extra Planner-only gradient term to Workflow-RLOO. The norm of each extra term follows the same rule in every minibatch: the same proportion multiplied by the Workflow-RLOO gradient norm on the Planner, times the same coverage factor. The term is added before gradient clipping; subsequent global clipping, learning rate, and optimizer are identical. The only difference is the direction: the control uses Workflow-RLOO's own Planner outcome loss, while PSC uses same-question sibling preferences localized to differing plan steps. The control was run once, on seed 1, together with the seed-1 Workflow-RLOO and PSC runs.

Scores (seed 1, all 22,517 questions, micro averages).

MethodEMF1
Workflow-RLOO0.3990.429
+ Increased Planner outcome loss (budget matched)0.3880.430
Differing-step PSC0.4060.433

Paired differences (pp, 95% paired bootstrap CIs).

ComparisonEMF1
PSC - increased Planner loss+1.78 [1.22, 2.31]+0.29 [-0.16, +0.74]
PSC - Workflow-RLOO+0.69 [0.11, 1.27]+0.38 [-0.17, +0.93]
Increased Planner loss - Workflow-RLOO-1.08 [-1.72, -0.50]+0.08 [-0.47, +0.64]

Update direction (frozen training batches).

QuantityValue
Median gradient cosine, differing-step PSC vs. Planner-RLOO0.682

Interpretation. In this single matched seed, increasing Planner outcome-loss weight does not reproduce PSC’s EM gain; the F1 difference is not significant. This control does not establish cross-seed superiority.

Revised paper. Control: Section 4.4, Table 3. Gradient cosine: Section 4.5. Updated setup: Section 3.2, Section 4.2, and Appendix A, Table 6.

Q2a: How many independent training runs are there per condition?

Seven independent RL runs: three training seeds each for Workflow-RLOO and differing-step PSC, plus one gradient-budget-matched Planner-reweighting control. Each seed independently controls question order, rollout sampling, and process randomness; the submitted version had one training run per condition (its repeated evaluation varied decoding only; Appendix H in the revision). Per-seed results: Table 2 and Appendix G, Table 14. All runs share the updated SFT initialization, 4,477 training questions, eight siblings per question, 300 steps, and retrieval environment; each is evaluated on all 22,517 questions.

Q2b: Do the improvements withstand training randomness?

Yes for option-order robustness on comparison questions, but not for a general full-suite EM gain. We independently retrained Workflow-RLOO and PSC with three seeds each, using the same SFT initialization, training questions, rollout budget, and evaluation protocol. All models were evaluated on 22,517 questions.

Micro EMSeed 1Seed 2Seed 3MeanRange
Workflow-RLOO0.39860.41990.39440.4042.6 pp
PSC0.40550.40210.40250.4030.3 pp

Both methods improve over SFT (0.349) by approximately 5.5 pp. Overall EM is comparable, while PSC shows a smaller observed range across seeds and uses 11% fewer retrievals (2.11 vs. 2.37 per question).

Position preference. On 2Wiki bridge-comparison questions, Workflow-RLOO selects the second-listed option in 90.0%, 71.8%, and 11.7% of cases across seeds, compared with 55% for SFT and 37.5%, 43.1%, and 27.9% for PSC. This reveals a seed-dependent position preference in Workflow-RLOO.

Option-order test. We swapped the two answer options in 4,897 eligible comparison questions ("A or B?"), keeping the gold answer fixed. We evaluated both orders and computed order-balanced EM by equally weighting original and swapped accuracy.

SeedRLOO original → swapped EM (%)PSC original → swapped EM (%)Same answer: RLOO / PSCPSC − RLOO order-balanced EM (pp, 95% CI)
163.5 → 54.859.6 → 61.542.0% / 56.4%+1.47 [0.44, 2.55]
265.3 → 59.664.7 → 65.658.1% / 69.8%+2.66 [1.62, 3.71]
360.4 → 65.165.5 → 66.148.9% / 64.0%+3.12 [2.15, 4.08]

PSC achieves higher order-balanced EM on comparison questions in all three independently trained seeds, with every paired 95% CI excluding zero. Its answers are also more consistent across option orders.

Full-suite order-balanced evaluation. For the 4,897 eligible questions, original and swapped orders receive equal weight; all other questions retain their original evaluation. This preserves the weighting of the complete 22,517-question benchmark.

Full-suite micro EM (%)Seed 1Seed 2Seed 3Mean
RLOO, original39.8641.9939.4440.43
PSC, original40.5540.2140.2540.34
RLOO, order-balanced38.9241.3739.9540.08
PSC, order-balanced40.7640.3040.3140.46
PSC − RLOO (pp, 95% CI)+1.84 [1.35, 2.34]−1.07 [−1.52, −0.62]+0.36 [−0.11, 0.84]+0.38 [0.10, 0.66]

The mean full-suite order-balanced advantage is +0.38 pp, but it is not consistent across seeds: seed 2 favors Workflow-RLOO. Confidence intervals use 10,000 paired bootstrap resamples of test questions, not training-seed uncertainty.

Conclusion. The revised evidence supports reproducible option-order robustness on comparison questions and lower retrieval cost at comparable overall EM, rather than a general accuracy improvement across all seeds.

Revised paper. Section 4.4, Tables 4–5 and Figure 4; Appendix G, Table 14.

W2: Of the 3B macro-average gain, 1.80 points come from Bamboogle (nine questions)

The original Bamboogle-driven macro gain does not persist in the revised three-seed results. We therefore use full-suite micro EM as the primary accuracy summary and report all datasets and the macro average excluding the 125-question Bamboogle (three-seed means):

MethodHotpotQA2WikiMuSiQueBamboogleFour-dataset averageAverage excluding BamboogleMicro EM
SFT0.3230.3990.1720.3360.3070.2980.349
Workflow-RLOO0.3620.4720.1870.3200.3350.3400.404
PSC0.3600.4730.1830.2990.3280.3390.403

Without Bamboogle, PSC and Workflow-RLOO are essentially tied (0.339 vs. 0.340). The Bamboogle gap of 2.1 pp amounts to roughly two to three questions per seed. We therefore do not base the claim on aggregate accuracy gains; the positive evidence is the all-seed order-balanced improvement on comparison questions and 11% fewer retrievals at comparable overall EM.

Revised paper. Per-dataset, per-seed EM and F1: Appendix G, Table 14.

We hope these matched training controls and independent-seed evaluations address your principal concerns, while clarifying that our strongest evidence is for option-order robustness rather than a general overall-EM improvement.

Reviewer_2WkWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies structured multi-hop retrieval when final answers are verifiable but there is no unique ground-truth intermediate retrieval/reasoning trajectory. It proposes SIPPO, which represents retrieval as an executable dependency-resolved DAG, uses execution-grounded SFT to initialize a shared multi-role policy, applies RLOO at the whole-workflow level, and introduces Planner Sibling Contrast (PSC) to prefer planner outputs from successful over unsuccessful same-question sibling executions. Experiments on four multi-hop QA benchmarks and two backbone settings show consistent improvements over the matched SFT and Workflow-RLOO baselines, with SIPPO achieving the best macro EM/F1 in the main comparison.

Critical strengths
  1. The paper provides a coherent execution-grounded formulation of structured retrieval. The workflow representation explicitly connects planning, retrieval, reasoning, and synthesis through actual runtime states rather than treating the intermediate trajectory as a purely textual trace. This provides a clean interface for both supervised initialization and outcome-based RL and makes the overall system reasonably interpretable and reproducible.

  2. The empirical evaluation is relatively careful and the proposed PSC objective produces non-trivial gains over a matched Workflow-RLOO baseline. The comparison controls initialization, rollout exposure, retrieval environment, and training configuration, and the paper additionally reports paired statistics, compute accounting, and gradient diagnostics. The consistent gains across both backbones provide evidence that PSC is not merely an artifact of a single training run or model family.

Critical weaknesses
  1. The paper provides insufficient theoretical and empirical justification for its specific structural design. While SIPPO combines executable DAG workflows, execution-grounded supervision, workflow-level RLOO, and planner-specific sibling contrast, the paper does not convincingly establish that this particular combination is necessary or preferable to alternative credit-assignment strategies. In particular, the evidence that local process validity does not reliably predict terminal correctness only shows that naive intermediate signals can be misleading, but it does not rule out more carefully constructed process supervision. In fact, the appendix reports that a counterfactual subgraph signal achieves 89.13% direction accuracy and improves replay-selection EM by 8.64 points over random selection, suggesting that useful intermediate credit signals may exist. Moreover, the claimed benefit of dependency-resolved runtime states is not isolated by a matched ablation against static or non-grounded workflow supervision. As a result, the paper demonstrates that the full system works, but does not clearly identify which structural choices are essential or why they should be preferred over competing formulations.

  2. The empirical evidence is not sufficiently strong to compensate for the limited methodological justification. PSC provides a training signal for only 2,906 of 14,976 question groups (19.4%), and the paper does not compare planner-only contrast against alternative role-specific, step-level, or counterfactual credit mechanisms. The gains over Workflow-RLOO are also not uniformly significant across benchmarks, and SIPPO does not achieve the best performance under every benchmark/backbone configuration (e.g., other methods outperform it on HotpotQA in both backbone settings). Overall, while the proposed configuration is effective, the current theoretical motivation and empirical evidence do not yet establish that its particular structural choices are generally preferable for multi-hop retrieval credit assignment.

Questions for authors
  1. Can the authors provide stronger theoretical, conceptual, or empirical evidence that the proposed structural choices are important or necessary for addressing the stated credit-assignment problem, rather than constituting one effective combination among several plausible alternatives? In particular, I would be interested in a clearer mechanism-level justification for why execution-grounded DAG workflows together with whole-workflow RLOO and planner-specific sibling contrast should be preferred in this setting, ideally supported by experiments that directly test the corresponding hypothesis. Additional component-wise ablations alone may not fully resolve this concern unless they clarify why these design choices are essential or systematically advantageous.

  2. Can the authors compare PSC with at least one alternative intermediate-credit mechanism under the same rollout and training budget. For example, applying contrast to other roles, using a step/subgraph-level signal, or using the counterfactual signal studied in the appendix? The current comparison against Workflow-RLOO establishes that PSC helps, but does not establish whether planner-only sibling contrast is the key design choice.

6 comments

Thank you for recognizing the system-level value of executable workflows and for asking us to explain the design choices further.

Q1: Why this combination of components?

The four components solve complementary problems: executable structure, state-aligned supervision, outcome-based learning, and Planner-specific credit.

Executable DAG. References such as #1# and #2# are resolved from earlier node conclusions during execution, so later steps act on observed evidence rather than assumed answers. Execution-grounded SFT. Each role is supervised on the actual state visible when its action was executed, which prevents a teacher's hidden evidence from leaking into student targets. Workflow-RLOO. Final-answer EM advantages are assigned to the actions in sampled complete workflows, training planning and execution jointly without step-level labels. Differing-step PSC. Sibling plans are grouped into classes of fully aligned plans; plans from a class with a higher EM success rate are compared with plans from a class with a lower rate, scoring only the steps that differ after alignment; this adds a Planner-specific preference beyond the shared trajectory-level reward. Changes from the submission and their rationale. Two issues motivated the updated setup.

SFT correction. First, supervision-state mismatch: an SFT audit revealed repeated supervision of earlier assistant turns and examples whose training inputs did not match the states visible during execution. We therefore rebuilt SFT from executed states, masked earlier responses, added stronger-teacher trajectories for HotpotQA and MuSiQue, and re-screened RL questions using the new initialization (4,477 retained).

More informative sibling comparisons. Second, richer within-question contrast: both PSC pairs and Workflow-RLOO advantages require groups containing both correct and incorrect executions. We increased sampling from four to eight siblings to make such contrasts more likely (up to 16 success–failure pairs per question, versus four). In these larger sibling groups, successful and failed plans often share most steps; scoring the whole plan would also assign preference to these common steps. We therefore localized PSC to aligned differing steps so that its preference focuses on the plan decisions that separate the paired workflows. We also set its auxiliary gradient strength relative to Planner-RLOO rather than using a fixed 0.02 coefficient. The same-question pairing, terminal EM labels, and softplus loss remain unchanged.

Matched evaluation. Following the reviewers' suggestions, we conducted seven new independent RL training runs from this same updated SFT: three Workflow-RLOO seeds, three differing-step PSC seeds, and one gradient-budget-matched Planner-reweighting control. All runs use 16 questions (128 workflows) per step for 300 steps and are evaluated on the full set of 22,517 questions, preserving matched comparisons within the revised setup.

If PSC merely strengthened the Planner part of Workflow-RLOO, two predictions would follow: increasing the Planner outcome loss under the same extra gradient budget should reproduce PSC's effect, and the two objectives should produce similar behavior. We tested both.

Matched Planner-reweighting control. Both the control and PSC add an extra Planner gradient term to Workflow-RLOO. In every minibatch, the norm of the extra term is set by the same rule (the same proportion of the Planner-RLOO gradient norm, times the same coverage factor, before gradient clipping), followed by identical clipping and optimizer updates. The only difference is the direction: the control amplifies Workflow-RLOO's own Planner outcome loss, while PSC uses differing-step sibling preferences.

Method (seed 1, all 22,517 questions, micro)EMEM difference from Workflow-RLOO (pp)F1
Workflow-RLOO0.399-0.429
+ Increased Planner outcome loss (budget matched)0.388-1.08 [-1.72, -0.50]0.430
Differing-step PSC0.406+0.69 [0.11, 1.27]0.433
PSC - increased Planner loss+1.78 [1.22, 2.31]+0.29 [-0.16, +0.74]

Option-order test: PSC learns different and more robust decisions. Overall, PSC and Workflow-RLOO reach comparable EM on the full set (40.3% vs. 40.4%, three-seed means). Inspecting their answers, we found that Workflow-RLOO develops a strong option-position preference on comparison questions, and the preferred position changes with the training seed: on 2Wiki bridge-comparison questions it selects the second-listed option in 90.0%, 71.8%, and 11.7% of cases across the three seeds, versus 55% for SFT and 37.5%, 43.1%, and 27.9% for PSC. To test this directly, we swap the two options in 4,897 comparison questions of the form "..., A or B?", keeping the gold answer unchanged, and evaluate each model under both orders (95% paired bootstrap CIs).

SeedWorkflow-RLOO: original → swapped EM (%)PSC: original → swapped EM (%)Same answer under both orders: Workflow-RLOO / PSCOrder-balanced EM: PSC - Workflow-RLOO (pp)
163.5 → 54.8 (-8.70)59.6 → 61.5 (+1.92)42.0% / 56.4%+1.47 [0.44, 2.55]
265.3 → 59.6 (-5.72)64.7 → 65.6 (+0.84)58.1% / 69.8%+2.66 [1.62, 3.71]
360.4 → 65.1 (+4.70)65.5 → 66.1 (+0.61)48.9% / 64.0%+3.12 [2.15, 4.08]

Although PSC and Workflow-RLOO achieve comparable overall EM (40.3% vs. 40.4%), PSC consistently achieves higher order-balanced EM on comparison questions across all three independent training seeds (+1.47, +2.66, and +3.12 percentage points; all 95% confidence intervals exclude zero), while exhibiting substantially lower sensitivity to option order. A stronger Planner gradient along Workflow-RLOO's own direction would be expected to inherit Workflow-RLOO's behavior; instead, PSC produces different and more robust decisions.

These results do not support the simpler explanation that PSC merely increases Planner outcome-loss weight: in the matched seed-1 control, reweighting performs worse, and PSC produces more order-robust comparison decisions across all three seeds. Original-order full-suite EM remains comparable; the mean full-suite order-balanced difference is +0.38 pp (95% paired bootstrap CI [0.10, 0.66]), with seed 2 favoring Workflow-RLOO.

Revised paper. Updated setup: Section 3.2, Section 4.2, and Appendix A, Table 6. Matched Planner-reweighting control: Section 4.4, Table 3. Option-order test and order-balanced EM: Section 4.4, Tables 4–5 and Figure 4.

W1b: The counterfactual subgraph signal in the appendix suggests that useful intermediate credit signals exist

The appendix result is correct for the setting it measures: when counterfactual replay branches are constructed and re-executed, the subgraph signal predicts which branch has the better terminal outcome with 89.13% direction accuracy [80.85, 96.59], and signal-guided replay selection improves EM by 8.64 points [6.05, 11.44] over random selection. Training, however, requires credit for actions in naturally generated trajectories, so we tested that use directly.

We first found that the explicit missing-information marker used in the original deletion intervention triggered a default "No" shortcut on some yes/no questions. We therefore replaced intermediate responses with valid alternatives sampled from each action's original input, and re-executed all affected later nodes and the final summarization.

Replacement auditValid replacementsChange the intermediate conclusionChange final-answer correctness
On-policy Search/Reason replacements, byte-distinct (Qwen3-4B)18441 (22.3%)0
Local replacement audit (paper, Figure 1b)100% remain valid26.1%0%

In both audits, valid replacements frequently change intermediate conclusions but never change whether the final answer is correct. The same holds for local attribution: reason-premise attribution reaches AUC 0.869 on controlled local-support cases but only 0.238 when ranking natural trajectories by final correctness (paper, Figure 1a).

These audits therefore provide no terminal-outcome signal that distinguishes the actions in natural trajectories, so we did not use this estimator for training. We retain the appendix's replay-selection finding as a validation tool. PSC instead derives Planner preferences from naturally sampled sibling outcomes of the same question, without additional counterfactual executions.

Revised paper. Local attribution and the local replacement audit: Figure 1. Replay-selection validation: Appendix K, Table 19.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.