- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes SIPPO for learning retrieval workflows in multi-hop question answering. A shared model generates and executes graph-based plans. Training combines supervised workflow examples, final-answer reinforcement learning, and planner preferences from successful and unsuccessful sibling executions. Experiments cover four benchmarks and two backbones, with reported improvements in answer accuracy.
Critical strengths
The workflow representation and execution procedure are clearly described. The matched training comparisons help separate reinforcement learning gains from the added planner objective. Evaluation covers 22,517 questions, and the paper reports paired confidence intervals and retrieval costs. I also appreciate that the authors discuss uneven gains across the different benchmarks.
Critical weaknesses
The main concern is training robustness. Appendix G repeats decoding, but does not repeat model training. The 3B micro EM improvement is only 0.71 points, while much of the macro gain comes from Bamboogle. Frozen-policy gradient checks also do not show whether the planner contrast is better than simpler alternatives.
Questions for authors
Could the authors repeat both training conditions with several random seeds? Would the improvement remain under equal retrieval budgets? Please compare PSC with stronger weighting of the existing planner loss. Could you also freeze the downstream policy and compare plans, to test whether planner changes alone improve final answer accuracy?
3 comments
Q3: Compare against more strongly weighting the existing Planner loss
Setup. The control adds an extra term that amplifies Workflow-RLOO's own Planner outcome loss. Its gradient norm is set in each minibatch by the same rule as PSC's extra term (the same proportion of the Planner-RLOO gradient norm, times the same coverage factor), added before gradient clipping, followed by identical clipping and optimizer updates. Only the direction differs.
| Seed 1, all 22,517 questions (micro) | EM | EM difference from Workflow-RLOO (pp) | F1 |
|---|---|---|---|
| Workflow-RLOO | 0.399 | - | 0.429 |
| + Increased Planner loss (budget matched) | 0.388 | -1.08 [-1.72, -0.50] | 0.430 |
| PSC | 0.406 | +0.69 [0.11, 1.27] | 0.433 |
| PSC - increased Planner loss | +1.78 [1.22, 2.31] | +0.29 [-0.16, +0.74] |
Conclusion. Under the same extra gradient budget, the reweighting control does not improve on Workflow-RLOO's EM, and PSC exceeds it by 1.78 points. Simply amplifying Planner outcome supervision does not reproduce PSC's effect.
Revised paper. Section 4.4, Table 3.
Q4: Freeze the downstream executor and compare plans alone
Setup. We fix the executor to the SFT model, the common starting point of both RL conditions, and use it to execute the plans generated by Workflow-RLOO and PSC (three seeds each) on all 22,517 questions with greedy decoding. The two plans for a question use the same hardware and retrieval service.
| Executor fixed to SFT: PSC plans - Workflow-RLOO plans (EM pp) | Seed 1 | Seed 2 | Seed 3 | Mean |
|---|---|---|---|---|
| All 22,517 questions | -1.66 | -2.07 | +0.42 | -1.10 |
| Non-comparison questions (15,239) | +0.61 | -0.81 | -1.35 | -0.52 |
| Comparison questions (7,278) | -6.40 | -4.71 | +4.14 | -2.33 |
The frozen-executor analysis includes 7,278 comparison questions; the option-order test evaluates 4,897 eligible comparison questions. On non-comparison questions, the gap between the two kinds of plans stays below 1.5 points in every seed and changes direction across seeds. On comparison questions, the difference comes from how the plans are written. For example, on a 2Wiki bridge-comparison question:
Workflow-RLOO's plan: 1. What nationality is the director of film A? 2. What nationality is the director of film B? 3. Are the two directors from the same country?
PSC's plan: 1. Who directed film A? 2. Who directed film B? 3. Are #1# and #2# from the same country?
PSC's plan is more concise and leaves the attribute lookup to the final step for the executor. The executor trained jointly with PSC completes this step; the SFT executor does not:
| 2Wiki bridge-comparison EM (%) | Seed 1 | Seed 2 | Seed 3 |
|---|---|---|---|
| PSC plans, PSC's own executor | 51.1 | 55.0 | 52.6 |
| PSC plans, SFT executor | 35.9 | 35.9 | 48.7 |
Interpretation. The frozen-executor experiment answers an important but distinct question: whether plans learned under joint RL training transfer to the shared pre-RL SFT executor. Under this intervention, PSC plans do not outperform Workflow-RLOO plans, and we report this negative result explicitly.
However, this result should not be interpreted as evidence that PSC fails to improve the jointly trained workflow policy. In SIPPO, the Planner and execution roles are optimized together within a shared model. The comparison-question analysis illustrates this distinction: PSC plans can delegate attribute resolution to later execution steps, requiring execution behavior learned during joint training. On exactly these comparison questions, the complete PSC-trained policy is the more robust one: when the two options are swapped, it achieves significantly higher order-balanced EM than Workflow-RLOO in all three independent training seeds (+1.47, +2.66, and +3.12 pp; all 95% CIs exclude zero; see W).
We therefore distinguish two claims: PSC does not demonstrate superior standalone plan transfer to a frozen SFT executor, but the complete PSC-trained policy exhibits more robust option-order behavior and comparable overall accuracy with fewer retrievals. Our empirical contribution concerns the latter, not an independently improved Planner under an unchanged executor. We report this experiment, the question-type breakdown, and the plan example in Section 4.4 and Appendix G, Table 15.
W: Frozen-gradient checks do not establish that PSC is better than simpler methods
We agree that gradient diagnostics alone are insufficient, and we added two direct tests.
Training control. Under a matched extra gradient budget (same rule as in Q3), amplifying the Planner outcome loss does not reproduce PSC's EM:
| Seed 1, all 22,517 questions (micro) | EM | EM difference from Workflow-RLOO (pp) |
|---|---|---|
| Workflow-RLOO | 0.399 | - |
| + Increased Planner loss (budget matched) | 0.388 | -1.08 [-1.72, -0.50] |
| PSC | 0.406 | +0.69 [0.11, 1.27] |
| PSC - increased Planner loss | +1.78 [1.22, 2.31] |
Behavioral test. The two trained models reach comparable overall EM (PSC 40.3%, Workflow-RLOO 40.4%), so we examined how their decisions differ. Workflow-RLOO shows a strong option-position preference on comparison questions that changes with the training seed: on 2Wiki bridge-comparison questions it selects the second-listed option in 90.0%, 71.8%, and 11.7% of cases across the three seeds, versus 55% for SFT and 37.5%, 43.1%, and 27.9% for PSC. We therefore swap the two options in 4,897 comparison questions of the form "..., A or B?", keeping the answer unchanged, and evaluate each model under both orders (95% paired bootstrap CIs):
| Seed | Workflow-RLOO: original → swapped EM (%) | PSC: original → swapped EM (%) | Same answer under both orders: Workflow-RLOO / PSC | Order-balanced EM: PSC - Workflow-RLOO (pp) |
|---|---|---|---|---|
| 1 | 63.5 → 54.8 (-8.70) | 59.6 → 61.5 (+1.92) | 42.0% / 56.4% | +1.47 [0.44, 2.55] |
| 2 | 65.3 → 59.6 (-5.72) | 64.7 → 65.6 (+0.84) | 58.1% / 69.8% | +2.66 [1.62, 3.71] |
| 3 | 60.4 → 65.1 (+4.70) | 65.5 → 66.1 (+0.61) | 48.9% / 64.0% | +3.12 [2.15, 4.08] |
Although PSC and Workflow-RLOO achieve comparable overall EM (40.3% vs. 40.4%), PSC consistently achieves higher order-balanced EM on comparison questions across all three independent training seeds (+1.47, +2.66, and +3.12 percentage points; all 95% confidence intervals exclude zero), while exhibiting substantially lower sensitivity to option order.
Conclusion. The two tests answer your concern from complementary sides, without relying on frozen-gradient diagnostics: in training, PSC outperforms the matched simpler alternative (stronger Planner outcome loss) by 1.78 EM points; in behavior, PSC achieves significantly higher mean order-balanced accuracy on the full evaluation suite (+0.38 pp, 95% paired bootstrap CI [0.10, 0.66]), with consistent gains across all three seeds on comparison questions. This complements the matched Planner-reweighting control and demonstrates benefits beyond simply strengthening outcome supervision. The full-suite gain is a mean over seeds rather than a per-seed result: in seed 2, Workflow-RLOO remains ahead on the full suite (-1.07 pp; see Q1).
Revised paper. Control: Section 4.4, Table 3. Option-order test and order-balanced EM: Section 4.4, Tables 4–5 and Figure 4.
Summary of the requested experiments. We have now conducted the independent training repetitions, the retrieval-cost analysis, the gradient-budget-matched Planner-reweighting control, and the frozen-executor evaluation requested in your review. The results show that (i) PSC's order-balanced accuracy advantage on comparison questions holds across all three independent training seeds, and its mean full-suite order-balanced EM is higher by +0.38 pp (95% paired bootstrap CI over test questions [0.10, 0.66]), although this gain is not consistent across training seeds; (ii) PSC maintains comparable overall accuracy with fewer retrieval and model calls; (iii) simply strengthening the existing Planner loss does not reproduce PSC's seed-1 improvement; and (iv) PSC plans do not independently outperform Workflow-RLOO plans under a frozen SFT executor, clarifying the role of Planner-executor co-adaptation.
We have revised our interpretation accordingly and hope this more complete evidence addresses your primary soundness concerns.
Q1: Repeat both training conditions with multiple random seeds
We conducted three independent RL training runs per condition, rather than repeated decoding of one checkpoint. Overall original-order EM is comparable, while PSC improves option-order robustness on comparison questions in every seed.
Changes from submission. We rebuilt execution-grounded SFT after identifying mismatches between training inputs and execution states and repeated supervision of earlier assistant turns. We re-rendered targets from executed states, masked earlier responses, added stronger-teacher HotpotQA/MuSiQue trajectories, and re-screened RL questions (4,477 retained). We increased siblings from four to eight, localized PSC to differing plan steps after alignment, and scaled its auxiliary gradient relative to Planner-RLOO rather than using a fixed 0.02 coefficient. Same-question comparisons, terminal EM-derived supervision, and the softplus loss remain.
Matched setup. Both conditions share the updated SFT initialization, 4,477 RL questions, eight siblings per question, and 300 training steps (16 questions/128 workflows per step). Each seed independently controls question order, rollout sampling, and process randomness. All models are evaluated on the full 22,517-question suite.
Both methods improve over SFT (0.349) by approximately 5.5 pp. PSC has a smaller observed range across training seeds and uses 11% fewer retrievals.
Order-balanced evaluation. We identified seed-dependent option-position preferences in Workflow-RLOO. For 4,897 eligible comparison questions, we evaluate both original and swapped option orders, keeping the gold answer fixed, and average their EM. Other questions retain original-order EM, preserving the full-suite weighting.
The mean full-suite order-balanced gain is +0.38 pp (95% CI [0.10, 0.66]), but it is not consistent across seeds: seed 2 favors RLOO (−1.07 pp). On the 4,897 comparison questions, PSC improves order-balanced EM by +1.47 [0.44, 2.55], +2.66 [1.62, 3.71], and +3.12 [2.15, 4.08] pp across seeds. All three paired CIs exclude zero. These CIs use 10,000 paired bootstrap resamples of test questions, not training-seed uncertainty.
Conclusion. The independent runs support PSC's option-order robustness on comparison questions and lower retrieval cost at comparable overall accuracy, not a consistent general full-suite EM improvement.
Revised paper. Table 2; Section 4.4, Tables 4–5 and Figure 4; Appendix G, Table 14.
Q2: Does the improvement remain under the same retrieval budget?
PSC achieves comparable EM with fewer realized retrieval calls, although we did not enforce an identical hard retrieval cap. Both methods use the same training budget and Wiki18/E5 top-3 retrieval environment.
Conclusion. PSC maintains comparable accuracy while making 11% fewer retrieval calls and 10.4% fewer model calls. This establishes lower realized inference cost, not performance under an identical hard retrieval cap.
Revised paper. Section 4.4, Figure 3; Appendix J, Table 18.