acceptodds
Under review as a conference paper at ICLR 2027

CAGE: Blockwise Policy Gradients for Explicitly Routed Language-Model Workflows

Abstract

Learning from terminal rewards in routed language-model workflows requires two distinct comparisons: between routes to learn which workflow to select, and within routes to learn how to execute it. Small execution budgets make both comparisons difficult: independent sampling can omit routes, while global centering mixes route-level reward differences into execution credit. We introduce Credit Assignment with Graph-aware Estimation (CAGE), which allocates executions across routes before observing rewards. It sums the Router gradient over the finite route set and uses a route-local leave-one-out baseline for the Specialist. Live-policy weights correct for the allocated sample counts. Under independent conditional sampling and sampling-consistent scores, both raw estimators are conditionally unbiased for the original current-policy objective. We establish deterministic route coverage, invariance of Specialist credit to route-specific reward offsets, and removal of the additional categorical sampling variance of a same-batch sampled-Router estimator. An exact covariance decomposition quantifies the noise added by estimating the Specialist's local baselines from finite samples. A fixed-policy diagnostic confirms the coverage mechanism: IID sampling misses routes on 29 of 32 questions, whereas CAGE covers every route. A one-question full-LoRA-gradient check on a frozen 4B model also verifies samplewise route-shift invariance. These diagnostics support the coverage and local-credit mechanisms; an end-to-end reward advantage remains unestablished. CAGE estimates both route-selection and route-execution gradients from the same fixed complete-workflow budget while preserving the current-policy objective.

Primary area:
reinforcement learning
Keywords:
reinforcement learning, large language models, policy gradient estimation, credit assignment, routed workflows, stratified sampling

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_wyXLOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes CAGE, a policy-gradient method for LLM workflows that separates Router and Specialist gradient estimation. It ensures that every route is sampled, computes Router gradients across routes, and uses route-specific baselines for Specialist gradients.

Critical strengths
  1. The paper provides some mathematical proofs for the proposed gradient estimators, including their unbiasedness under the stated assumptions.

  2. The paper honestly reports its experimental limitations and acknowledges that the current results do not demonstrate a significant improvement over the baseline.

Critical weaknesses
  1. Figure 1 has overlapping lines and labels and does not clearly show the key idea of CAGE or how it improves upon existing methods. It strongly resembles an AI-generated image.

  2. The paper is difficult to follow because the writing is unnecessarily complicated and the explanations lack a clear logical flow.

  3. The experiments are very limited, and the results do not show that CAGE improves learning performance or that its components are effective.

  4. The novelty is unclear because CAGE mainly combines existing techniques and does not actually use graph structure for credit assignment.

  5. The theoretical proofs assume that the sampling and gradient calculations use the same policy distribution, but the reported experiments do not fully satisfy this assumption.

  6. The paper does not explain why forcing every route to be sampled improves learning, especially when this may waste samples on rarely selected routes without producing useful gradients.

  7. The supplementary materials appear to contain only the LaTeX source of the paper, rather than the code or additional experimental evidence needed to verify and reproduce the results. This was weird.

Questions for authors

Please answer each question listed in the Weakness.

Reviewer_RuGKOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CAGE, a policy-gradient estimator for explicitly routed language-model workflows. CAGE allocates a fixed execution budget across all routes, exactly marginalizes the Router gradient over the finite route set, and applies a route-local leave-one-out baseline to Specialist trajectories. The paper proves conditional unbiasedness under independent within-route sampling and sampling-consistent policy scores, as well as route coverage and route-shift invariance. The empirical study verifies the coverage mechanism and samplewise route-shift invariance through fixed-policy diagnostics. However, the historical multi-step training runs used target-dependent support repair, and additional checks exposed Router reconstruction discrepancies. These runs therefore do not establish sampling-consistent execution of the theoretically analyzed estimator. The external Omni-MATH evaluation also does not show a statistically significant or seed-consistent reward improvement over IID/global RLOO. The paper provides theoretical and diagnostic analysis, while explicitly leaving end-to-end efficacy unresolved.

Critical strengths
  1. CAGE correctly recognizes that Router learning must preserve between-route reward differences, whereas Specialist credit can remove route-specific offsets. The exact Router marginalization and route-local LOO construction are conceptually well motivated.

  2. The paper provides conditional unbiasedness results, a route-shift invariance result, and an explicit covariance decomposition for the finite-sample local baseline. The authors also clearly disclose that the reported training implementation does not fully match the sampling-consistent estimator required by the theory.

Critical weaknesses
  1. Section 6.3 and Appendix C.1 report target-dependent support expansion in every historical Stage-B run and cross-process Router reconstruction discrepancies. The authors report that support expansion affected approximately 0.024% of rescored tokens across the four arms (0.02311%, 0.02533%, 0.02482%, and 0.02465%), but this frequency alone does not bound the resulting gradient bias. The subsequent repairs were checked numerically, but no complete amended multi-step training comparison is available. Consequently, the reported learning experiments cannot establish the practical effectiveness of the estimator analyzed in the theory.

  2. Table 2 reports a selected-checkpoint improvement of only +0.100 percentage points, with a 95% interval of [-0.592, +0.808]. The three seed effects have mixed signs, and the fixed-final difference is -0.225 percentage points. The intervals are conditional on these trained seeds and do not estimate uncertainty across training runs (Section 6.3). These results do not establish superiority or equivalence. Although the authors acknowledge this limitation, the practical benefit remains unsupported even for the historical implementation.

  3. With N=8N=8 and K=3K=3, six executions are committed to the two-sample floor, leaving only two to follow the Router probabilities. Equation 13 also identifies additional covariance from estimating local baselines. However, no parameter-gradient variance or MSE comparison against IID/global RLOO is reported. The coverage and route-shift checks verify their respective properties, but do not show whether the combined estimator improves estimation accuracy at the same budget.

Questions for authors
  1. The key missing evidence is a complete multi-step comparison using the amended backend and sampling-consistent scores. If such a repaired paired-seed experiment is available, please report it under the same training and held-out protocol, including individual seed effects and a distinction between question-level uncertainty and uncertainty across training runs. If it is unavailable, the practical effectiveness of CAGE remains unresolved.

  2. The effect of the historical scoring discrepancies on the actual updates is unclear. For identical saved batches and parameters, please compare historical and reference Router probabilities and gradient vectors, including relative norm differences and cosine similarities for both policy blocks. Appendix C.1 correctly notes that the low support-expansion frequency (approximately 0.024%) gives no gradient-bias bound; the relevant evidence here is quantitative update sensitivity to assess whether the historical runs remain informative.

  3. The current ablations do not isolate the contribution of route-local Specialist centering. Please compare CAGE with the same covered allocation and exact Router estimator, but a weighted global LOO Specialist baseline. With independent trajectories and stop-gradient coefficients, this comparator can retain the pr/nrp_r/n_r correction and the same target objective, thereby isolating local centering more directly than the current sampled-Router ablation.

Reviewer_GaviOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CAGE, a policy-gradient estimator for routed LLM workflows. CAGE allocates at least two executions to every route and weights each route by p_r/n_r. It computes the Router gradient by exact summation over the finite route set and gives the Specialist a route-local leave-one-out baseline. The authors prove conditional unbiasedness, route-shift invariance and coverage. Experiments on HardMath and Omni-MATH with Qwen3.5-4B show no significant reward difference from IID/global RLOO.

Critical strengths
  1. The problem is clearly formulated. The two-route example and the counterexample in Sec. 4.3 make clear why the Router and the Specialist need different comparisons.
  2. The derivations appear correct. The finite-sample covariance decomposition (Eq. 13) usefully quantifies the cost of estimating local baselines.
  3. The authors are transparent about limitations, implementation issues and negative results.
Critical weaknesses
  1. Under the paper’s budget setting, the two-sample floor takes most of the executions, so allocation is close to uniform and the policy-dependent part has little effect. In the four-arm check, uniform-covered allocation actually scores higher than CAGE. Simpler alternatives discussed in the paper, such as IID sampling with a route-local LOO baseline, are not compared.
  2. The core motivation is that global centering injects between-route noise into Specialist credit, but the paper never measures gradient variance or MSE against IID/global RLOO. The coverage diagnostic and the route-shift check only confirm properties that hold by construction.
  3. On the external test set, CAGE shows no significant improvement over the baseline, is slightly worse at final checkpoints, and is inconsistent across seeds. All training runs also used support repair and had Router reconstruction inconsistencies, so the reported results do not test CAGE as analyzed.
  4. The experiments are small in scale, with few update steps and a single model. The three routes differ only in how many times the same Specialist is called, and there are no base-model or single-agent controls. Heavy hedging, defensive phrasing and process terminology throughout the text obscure the actual contribution.
Questions for authors
  • Can you report gradient variance or MSE at a fixed policy for CAGE versus IID/global RLOO?
  • Since uniform-covered allocation outperformed CAGE, what does live-policy allocation actually contribute?
  • Are there multi-step training results under a sampling-consistent implementation?
  • How do the trained checkpoints compare with the frozen base model?
Reviewer_PM3fOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the problem of policy-gradient training of routed LM workflows, where there is a router that selects one from K routes, a specialist executes that route’s chain of LM calls, and a verifier that returns a terminal reward. It introduces Credit Assignment with Graph-aware Estimation (CAGE), which allocates executions across routes before observing rewards. It estimates the router gradient by summing over all routes and estimates the Specialist gradient using route-local leave-one-out baselines. The central idea is to distinguish between-route credit for selecting a workflow from within-route credit for executing it.

In theoretical analysis, the authors established five mathematical properties of CAGE. These results demonstrate that CAGE is an unbiased policy-gradient estimator with guaranteed route coverage and certain variance-reduction properties.

In experimental session, CAGE is evaluated on mathematical reasoning tasks based on Qwen3.5-4B, with comparisons with standard IID and global RLOO. The experiments verified that CAGE’s proposed mechanisms are theoretically expected, but do not demonstrate better gradient estimation, learning efficiency, or downstream performance.

Critical strengths
  • The problem of credit assignment in routed language-model workflows is important in building multi-agent LLM systems. The method of CAGE is intuitive and the theoretical analysis is solid.
  • The reports on experimentation are clear and transparent. The authors openly discloses the support-repair scorer, training-data defects, the schedule deviation and limitations of the experimental results.
Critical weaknesses
  • The paper did not show any empirical evidence that CAGE improves learning, or reduces gradient variance comparing to existing methods. This is a clear gap for a paper proposing a new methodology, as the practical advantages over existing methods are unclear.
  • In CAGE, at least execution per route is guaranteed, regardless of how unlikely the route is to be selected. This could waste a significant amount of training budget on rarely selected routes. The paper did not sufficiently investigate the tradeoff and whether guaranteed route coverage justify additional sampling cost.
Questions for authors

Q1: The theoretical analysis establishes unbiasedness, guaranteed coverage, and variance reduction relative to a same-batch sampled-Router estimator. However, it does not establish an overall variance advantage over IID/global RLOO under the actual CAGE allocation. Can you clarify under what conditions CAGE is expected to provide lower gradient variance or MSE than IID/global RLOO, and ideally validate those conditions empirically?

Q2: Given that uniform-covered allocation outperforms CAGE in the four-arm extension, is there any setting in your data where live-policy residual allocation outperforms uniform allocation?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.