acceptodds
Under review as a conference paper at ICLR 2027

Orchestration within LLMs via Discovering and Orchestrating Internal Operators

Abstract

Large Language Models (LLMs) are increasingly composed into agentic systems, where orchestration has become a key factor in system performance. Rather than relying on ever-larger models, agentic systems can solve complex tasks with models of bounded size, which makes orchestration a practical alternative to scaling. This motivates internalizing such orchestration within a single model, which holds the promise to improve both performance and parameter efficiency. To achieve it, we propose a framework for internally orchestrating LLMs, which performs orchestration inside a single LLM through two modules. The first module identifies patterns of coordinated activity among attention heads, which we call modes, and measures their causal effects on subsequent reasoning. We find that modes fall into three functionally distinct classes, corresponding to exploration, effort, and monitoring. Building on these findings, we introduce a Mode-Guided Orchestrator that uses mode-derived signals to determine when to intervene, adaptively invokes mode-specific interventions, and aggregates results through sequential consensus. Together, these modules dynamically construct query-specific computation graphs at inference time. Across five backbones and multiple benchmarks spanning mathematical, scientific, medical, and multi-hop reasoning, our method lies on the accuracy-compute Pareto frontier compared to the selected baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, Orchestration

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_xq3bOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies whether orchestration can be internalized within a single frozen LLM. The authors first identify low-rank patterns in attention-head writes to the residual stream, termed modes, and use controlled interventions to characterize them as Exploration, Effort, or Monitoring modes. They then build a Mode-Guided Orchestrator that uses internal signals to decide when to branch, applies mode-specific interventions to generate alternative trajectories, and aggregates the resulting answers using sequential consensus.

The framework is evaluated on five backbones and five reasoning benchmarks, with comparisons to self-consistency, adaptive inference-time reasoning, latent reasoning, and multi-agent baselines. The method achieves strong average accuracy and is reported to lie on the accuracy-compute Pareto frontier.

Critical strengths
  1. I find the central idea interesting. Treating recurring internal computation patterns as manipulable operators provides a different perspective on inference-time orchestration, and the paper goes beyond purely observational analysis by explicitly intervening on the discovered modes.

  2. The empirical scope is fairly strong. The authors evaluate five different backbones and multiple reasoning domains, and the mode subspaces appear reproducible across samples and reasonably consistent across benchmarks. The comparison also covers a broad range of inference-time reasoning and multi-agent baselines.

Critical weaknesses
  1. The end-to-end gains are not cleanly attributed to the discovered mode structure.

My main concern is that the final system changes several things at once: it detects risky trajectories, forks multiple continuations from an intermediate state, perturbs them, and aggregates the answers through sequential consensus. The paper shows that mode-aligned perturbations affect entropy and trajectory length differently from a permutation control, but this does not establish that the mode structure is necessary for the final accuracy gains.

I would have expected a compute-matched control that keeps the same risk gate, fork point, branch budget, and consensus rule, but replaces the mode-specific interventions with random or permuted-head interventions. Without such a control, a simpler explanation remains plausible: much of the gain may come from generating diverse branches and voting over them, rather than from the discovered Exploration/Effort/Monitoring operators themselves.

This matters because the central claim is not merely that branching improves reasoning, but that meaningful internal operators can be discovered and orchestrated.

  1. The current causal control does not fully match the actual intervention magnitude.

The permutation control preserves the multiset of head-wise gain factors, but the appendix explicitly notes that it does not preserve the norm of the resulting residual perturbation because different heads have different write magnitudes. Therefore, a stronger response under the mode-aligned intervention could partly reflect a larger effective perturbation rather than the specific mode structure.

I think this weakens the causal interpretation of the probing results. A cleaner control would match the actual residual perturbation norm, ideally layer-wise as well, rather than only matching the gain coefficients assigned to heads.

Questions for authors
  1. Do the authors have a compute-matched end-to-end control in which the same risk gate, fork point, branch budget, and sequential consensus are used, but the mode-specific interventions are replaced by random or permuted-head perturbations? This would materially affect my confidence that the accuracy improvement is specifically due to the discovered internal operators.

  2. Since the permutation control does not preserve the norm of the residual perturbation, can the authors quantify the perturbation-energy difference between the mode intervention and its control? Have the authors tested a norm-matched control?

  3. The paper states that the orchestrator adaptively determines which operators to invoke. However, the appendix describes a fixed role-based schedule (Exploration first, then Effort, then Monitoring, with random order within each role). Could the authors clarify which part of operator selection is query-adaptive in the current implementation?

  4. How much does the Monitoring-mode signal contribute beyond ordinary confidence, trajectory length, and the step-ceiling indicator in the risk model? An existing ablation of the risk predictor without Monitoring features would help clarify the value of the proposed monitoring interpretation.

2 comments
AuthorsOpenReview ↗

Response to Reviewer xq3b

We thank the reviewer for finding the central idea interesting and the valuable review. We address each concern below. Corresponding revisions will be provided in the revised manuscript.

W1 / Q1: Compute-matched end-to-end comparisons with random and permuted-head interventions under otherwise identical settings.

Thank you for your insightful suggestion. We added compute-matched end-to-end comparisons in which only the intervention operators are replaced. All conditions use the same evaluation items, risk-gating configuration, branching point, branch budget, and sequential-consensus rule. We evaluate two replacements: permuted heads, which randomly reassigns each operator's gain values across attention heads; and random direction, which replaces the mode gains with randomly sampled gains matched to the mode's per-layer gain norm. These controls retain the branching-and-consensus procedure while replacing the mode-specific gain pattern. As shown in Table 1, the proposed design achieves higher accuracy.

Table 1. End-to-end accuracy (%) with mode operators and replacement controls at matched compute.

ConfigurationGSM8KARC-CARC-EMedQAHotpotQA
Proposed design84.288.295.264.071.2
Permuted heads81.386.192.061.368.4
Random direction81.685.391.661.667.2

W2 / Q2: Comparison with controls matched in residual-perturbation norm.

Thank you for your insightful suggestion. We added a norm-matched perturbation control: the permuted-head gains are rescaled at each layer and decoding step to match the residual-perturbation norm induced by the corresponding mode operator. As shown in Table 2, the proposed design outperforms the norm-matched control. The accuracy advantage therefore persists when the residual-perturbation norm is matched at each layer and decoding step, suggesting that perturbation magnitude alone does not account for this advantage.

Table 2. End-to-end accuracy (%) with mode operators and residual-norm-matched perturbations.

ConfigurationGSM8KARC-CARC-EMedQAHotpotQA
Proposed design84.288.295.264.071.2
A3: norm-matched perturbation81.785.892.461.967.4

Q3: Query-adaptive components in the current implementation.

Thank you for the valuable question. In the current implementation, three aspects of branching vary across queries:

  1. Whether to branch, determined by the risk trigger in Eq. 8.
  2. How many branches to generate, determined by sequential consensus, which can stop after two branches when answers agree or continue for up to 16 when agreement is insufficient.
  3. Where to branch, determined by a sentence-boundary rule whose resulting position depends on the trajectory content.

Operator selection order currently follows a fixed schedule, but could be made query-adaptive in several ways:

  • State-conditioned ordering. Use read-outs from the main trajectory, including confidence, length, and Monitoring activity, to determine which operator role is selected first. For example, prioritizing Exploration on high-risk items.
  • Feedback-driven ordering. Select the next operator based on the outcomes of previous branches. For example, switching to Exploration when a branch disagrees with the main answer.
  • Learned selection. Train a lightweight policy that maps the trajectory state to a distribution over operators.

We currently use a fixed schedule because it requires no additional training or labels, and sequential consensus already allocates more branches to items with persistent disagreement, allowing them to draw from a larger portion of the operator pool regardless of ordering. Query-adaptive operator selection could further improve accuracy, and we will investigate this direction in future work.

Q4: How much does the Monitoring signal contribute beyond confidence, trajectory length and the ceiling indicator in the risk model?

Thank you for the insightful question. We compare a conventional trigger, using confidence, trajectory length, and the ceiling indicator, with a full trigger that additionally uses Monitoring read-outs. Both are fitted with the same training protocol. As shown in Table 3, the full trigger achieves higher accuracy than the conventional trigger. These results support the incremental benefit of incorporating Monitoring read-outs into the trigger for downstream controller accuracy.

Table 3. Accuracy (%) of the gated controller.

TriggerGSM8KARC-CARC-EMedQAHotpotQA
Conventional trigger83.387.694.762.866.7
Full trigger84.288.295.264.071.2

Thank you for the additional experiments. They address my concerns about branching and perturbation magnitude. However, the evidence still does not establish the necessity of the proposed operator roles, and operator selection follows a fixed schedule. I maintain my score of 2

Reviewer_TEWROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes discovering and orchestrating internal operators within a frozen LLM. It extracts low-rank modes from attention-head contributions, characterizes their responses to interventions, and assigns Exploration, Effort, and Monitoring roles. A mode-guided controller then selectively generates intervened continuations and aggregates their answers through sequential consensus. Experiments across five backbones and five benchmarks examine mode stability and report accuracy and computational-efficiency improvements.

Critical strengths
  1. The paper connects internal representation analysis to a concrete inference-time intervention mechanism. The extracted head-loading vectors specify how attention-head contributions should be rescaled, providing an actionable interface to a frozen model. Bidirectional interventions and permutation controls are more informative than purely correlational observations. The stability analysis is also substantive: Table 1 reports split-sample mean principal cosines of at least 0.987, and Appendix A.3 examines individual-mode correspondence and nearly degenerate singular values.

  2. The evaluation provides encouraging evidence across several model families and task types. Experiments cover five backbones and mathematical, scientific, medical, and multi-hop question answering, with comparisons against sampling, adaptive consistency, latent reasoning, and multi-agent methods. Table 2 reports the highest backbone-averaged accuracy on four of five benchmarks. These results support further investigation of mode-guided inference as a practical way to improve frozen models.

Critical weaknesses
  1. The cross-method efficiency interpretation requires a common compute reference. Section 3.2 (p. 8, lines 403–419) and Appendix A.6 (p. 17, lines 899–907) normalize text-space methods by the length of a greedy text trajectory, while normalizing the proposed method by the length of a greedy latent trajectory. The reported approximately 45.7% reduction is consistent with these separately normalized ratios. However, because their denominators need not represent equal computational costs, this does not by itself establish 45.7% less inference computation in a shared unit. In particular, the description “4.38× the CoT baseline” at line 409 needs reconciliation with the stated latent-trajectory denominator.

Figure 5 therefore shows a tradeoff under the reported normalization, while the broader computational-efficiency interpretation needs calibration to a common reference. Reporting absolute reference lengths and costs would help; FLOPs or latency under comparable execution conditions would provide stronger practical evidence. Appendix A.6 makes this particularly important for HotpotQA, where the text reference contains only 4–7 tokens. This concern does not dispute the reported accuracy measurements or the arithmetic of the normalized ratios.

  1. The contribution of mode structure to end-to-end accuracy remains insufficiently isolated. The complete method combines soft-feedback decoding, selective branching, prefix reuse, operator scheduling, and sequential voting. I could not find a comparison of the complete controller with an otherwise identical version using permuted head-gain assignments. Equation 8 also combines Monitoring features with confidence, trajectory length, and a ceiling indicator. Appendix A.4 reports that Monitoring-only features recover the all-eight-mode AUROC on four backbones; this is useful evidence about which mode features retain diagnostic information, but does not quantify their incremental benefit over the conventional features in Equation 8.

The paper does include permutation-controlled probes in Section 2.1.2 and Appendix A.4. Their measured outcomes are entropy and continuation length, rather than downstream accuracy within the complete controller. Soft Thinking is a relevant standalone baseline, but is not a matched component ablation because its mixture size, temperature, and stopping rule differ (Appendices A.1–A.2). A focused comparison on representative evaluated tasks, such as GSM8K and HotpotQA, retaining the same latent loop and controller at matched expected compute, would help separate the contribution of mode structure from perturbation and aggregation more generally.

Questions for authors
  1. Can Figure 5 be re-expressed using a common compute reference, with the absolute greedy text and latent trajectory costs reported? Please clarify what the accounting includes, particularly prefill, answer readout, and intervention overhead.

  2. Under matched expected compute and identical branching and consensus rules, how does the method compare with permuted head-gain assignments and with a trigger using only confidence, trajectory length, and the ceiling indicator?

  3. Appendix A.4 specifies balanced GSM8K training boundaries for the Monitoring probe. What held-out data and numerical AUROCs support its reported comparison, and how is the composite risk trigger in Equation 8 fitted and thresholded? Please also provide per-backbone accuracy, since Tables 7–11 currently report compute rather than accuracy.

4 comments
AuthorsOpenReview ↗

Response to Reviewer TEWR (Part 1/2)

We thank the reviewer for the valuable review. We address each concern below, and will add corresponding revisions in the revised manuscript.

W1 / Q1: Common compute reference.

Thank you for the valuable suggestion. We now report FLOPs per item for all methods using a consistent accounting protocol. As shown in Table 1, our method remains on the empirical accuracy–compute Pareto frontier.

Table 1. Inference compute and accuracy under a common FLOP accounting protocol.

MethodTFLOPs / itemAccuracy (%)
CoT7.677.1
SC@826.179.4
Early-Stopping SC18.879.4
Adaptive-Consistency18.579.4
DeepConf26.179.7
Soft Thinking8.278.1
LTPO58.076.6
Multi-Agent Debate72.379.1
TextMAS, sequential70.175.8
TextMAS, hierarchical45.874.7
LatentMAS, sequential27.872.8
LatentMAS, hierarchical25.373.9
Ours14.380.6

W2-a / Q2-a: Under matched expected compute and identical branching and consensus, how does the method compare with permuted head-gain assignments?

Thank you for your insightuful suggestion. We added an end-to-end comparison with the requested permuted-head variant, together with two additional controls. All variants retain the complete controller, and differ only in the head-gain vectors used for intervention. They are evaluated on the same items at matched compute, with all other evaluation settings held fixed.

Permuted heads. We randomly permute each operator's gains across attention heads, preserving the gain values while changing their head assignments.

Norm-matched perturbation. We rescale the permuted-head gains at each layer and decoding step to match the residual-stream perturbation norm of the corresponding mode operator. This tests whether the mode operator provides a benefit beyond perturbation magnitude.

Random direction. We replace the mode gains with randomly sampled gains matched to the mode's per-layer gain norm. This control matches the norm of the gain vector rather than the resulting residual-stream perturbation.

Table 2. End-to-end accuracy (%) of the complete controller and three operator-replacement variants at matched compute.

ConfigurationGSM8KARC-CARC-EMedQAHotpotQA
Proposed design84.288.295.264.071.2
Permuted heads81.386.192.061.368.4
Norm-matched perturbation81.785.892.461.967.4
Random direction81.685.391.661.667.2

The proposed design achieves higher accuracy than the permuted-head variant, with gains of 2.1–3.2 percentage points. It also outperforms the norm-matched and random-direction controls by 2.1–3.8 and 2.4–4.0 points, respectively. In particular, the advantage over Norm-matched perturbation persists after matching the residual-stream perturbation norm, suggesting that perturbation magnitude alone does not account for the improvement. These comparisons support the benefit of mode-specific head-gain assignments within the complete controller over the tested alternatives.

AuthorsOpenReview ↗

Response to Reviewer TEWR (Part 2/2)

W2-b / Q2-b: Incremental benefits over a trigger that uses only confidence, trajectory length and the ceiling indicator.

Thank you for the insightful question. We compare a conventional trigger, using confidence, trajectory length, and the ceiling indicator, with a full trigger that additionally uses Monitoring read-outs. Both are fitted with the same training protocol. As shown in Table 3, the full trigger achieves higher accuracy than the conventional trigger. These results support the incremental benefit of incorporating Monitoring read-outs into the trigger for downstream controller accuracy.

Table 3. Accuracy (%) of the gated controller.

TriggerGSM8KARC-CARC-EMedQAHotpotQA
Conventional trigger83.387.694.762.866.7
Full trigger84.288.295.264.071.2

Q3-a: What held-out data and AUROCs support the Monitoring-probe comparison? How is the trigger fitted and thresholded?

Thank you for the valuable question.

Held-out evaluation and AUROCs. For each backbone, we evaluate prediction of an incorrect main-trajectory answer on held-out test items from the five benchmarks. Table 4 compares a probe using only Monitoring read-outs with the full risk model, which additionally incorporates confidence, trajectory length, and the ceiling indicator.

Table 4. Held-out AUROC for predicting an incorrect main-trajectory answer.

BackboneMonitoring onlyFull design
Qwen3-1.7B0.8100.842
Qwen3-4B0.8370.854
Qwen3-8B0.8690.875
Mistral-7B0.7930.801
Llama-3.1-8B0.8020.814

Monitoring read-outs alone achieve AUROCs close to the full risk model. It indicates that Monitoring read-outs contain information about answer correctness and that combining them with the conventional features improves error prediction relative to using Monitoring read-outs alone.

Fitting and thresholds. For each backbone, the trigger is fitted on 500 trajectories from the GSM8K training set, using the boundary-balancing procedure in Appendix A.4. The target is whether the main trajectory's final answer, a0a_0, is incorrect. The first stage maps Monitoring read-outs to a scalar logistic score; the second fits a logistic regression using this score together with confidence, trajectory length, and the ceiling indicator. Features are standardized, and L2 regularization uses a prespecified inverse regularization strength of C=0.3C=0.3. We set the risk threshold τ\tau to the 30th percentile of the cross-validated risk scores on the training trajectories. If r<τr < \tau, we return a0a_0 without branching. Otherwise, we generate branches sequentially and vote over their answers together with a0a_0, stopping when the leading answer is at least three votes ahead of the runner-up (ℓ=3\ell=3) or the branch budget is exhausted.

Q3-b: Per-backbone accuracy.

Thank you for the suggestion. We will include them in the revised manuscript.

Reviewer_TEWROpenReview ↗

Response to Authors

Thank you for the clarifications and additional experiments. The new FLOP estimates and operator-replacement controls partially address my concerns. However, the FLOP table does not specify whether prefill, answer readout, and intervention overhead are included, leaving the compute accounting insufficiently transparent. The trigger comparison also reports accuracy without corresponding compute costs, so the incremental benefit of Monitoring at matched compute remains unclear. Furthermore, the AUROC comparison evaluates Monitoring-only against the composite risk model, rather than the Monitoring-only versus all-eight-mode comparison originally questioned. Per-backbone accuracies are still deferred to a future revision, and no revised manuscript is available for assessing the promised updates. While I appreciate the additional evidence, these remaining gaps prevent me from considering my concerns fully resolved. I therefore maintain my original rating.

Thank you for the follow-up and for pointing out exactly what remained unclear. We address each point below. The revised manuscript is now uploaded, with changes marked in green.

1. What the FLOP accounting includes.

We cover prompt prefill, decoding of every sample and branch, answer read-out of every latent trajectory, and the overhead of our intervention. This accounting scope is now explicitly documented in Appendix A.6.

2. Trigger comparison at matched compute.

We now compare the conventional trigger (using confidence, trajectory length, and the step-ceiling indicator) with the full trigger that additionally incorporates Monitoring readouts. Both triggers are evaluated at the same compute budget within each benchmark.

Table 1. Accuracy (%) on each benchmark.

BenchmarkMatched computeConventional trigger+ Monitoring (Eq. 8)Gain
GSM8K2.04×80.1082.06+1.96
ARC-C1.60×85.4687.40+1.94
ARC-E1.51×91.0294.97+3.95
MedQA1.94×60.4563.38+2.93
HotpotQA2.33×68.9271.07+2.15

Adding Monitoring improves accuracy on every benchmark. These results support the incremental value of Monitoring beyond conventional confidence and trajectory features at matched compute.

3. Monitoring-only vs all eight modes.

We now conduct a direct comparison of held-out probe AUROC between Monitoring-only and all eight modes across five backbones, as shown in Table 2.

Table 2. Held-out AUROC for predicting an incorrect main-trajectory answer.

BackboneMonitoring onlyAll eight modes
Qwen3-1.7B0.8100.816
Qwen3-4B0.8370.841
Qwen3-8B0.8690.872
Mistral-7B0.7930.791
Llama-3.1-8B0.8020.805

The results show that Monitoring-only achieves AUROC scores close to those obtained using all eight modes. The differences are small, with Monitoring-only slightly outperforming all eight modes on Mistral-7B. This suggests that Monitoring captures most of the predictive signal for detecting incorrect main-trajectory answers, while the additional modes offer limited incremental benefit.

4. Per-backbone accuracy and revised manuscript.

The revised manuscript has now been uploaded, with changes marked in green. Appendix A.6 reports per-backbone accuracy and TFLOPs for all methods and benchmarks.

Reviewer_w473OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper propose mode-guided orchestrator that leverages the internal modes from the hidden states to manipulate the LLM's dedocing process and achieve test-time scaling. Experimental results show that the proposed method demonstrate the best performance-compute trade-off.

Critical strengths
  • This paper presents an elegant formulation, namely internal orchestration, which migrates the orchestration in multi-agent systems into the single-agent setting and offers valuable insights.

  • This paper has a good writing and is easy to follow.

Critical weaknesses

I have some concerns about the method design of mode-guided orchestrator:

  • The authors propose the Exploration mode and Effort mode. However, in my view, these two modes appear to overlap to some extent, since a longer trajectory also implies stronger exploration at each token dimension. This weakens the independence among different operators; The definition of the Monitoring mode is somewhat ambiguous; The mechanism for triggering intervention is not clearly described.

The experimental section of this paper requires further strengthening:

  • Ablation studies, sensitivity analysis, case studies and related experiments are missing. As a result, the design, behavior, and module contributions of the method remain unclear; Results with a single backbone are not provided, so the concrete performance of the proposed method across different model families is unknown; Although the authors claim that the proposed method achieves the best performance-compute trade-off, the compute metrics are inconsistent across different baselines. In my opinion, using a more consistent wall-clock time would be more appropriate.
Questions for authors
  • How will the model’s decoding behavior change when we select the Monitoring mode to intervene?

  • What is the architecture of network gg? How is it trained? What are the details of its inputs, and how are these inputs converted into features?

  • What results would be obtained if we simply randomly select several sentence endings to perform branching?

  • In table 2, why the MAS-based baselines are weak than single-model baselines?

  • Could you provide the time-based not the compute-based trade-off figure 5?

5 comments
AuthorsOpenReview ↗

Response to Reviewer w473 (Part 1/4)

We thank the reviewer for finding internal orchestration an elegant formulation and for the constructive review. We address each concern below. Corresponding revisions will be provided in the revised manuscript.

W1-a: Differentiate the Exploration and Effort modes.

Thank you for the insightful question. Exploration and Effort are distinguished by their effects on the entropy of the next-step distribution and trajectory length, respectively. The model can take more steps without necessarily increasing next-step entropy. Conversely, it can also increase next-step entropy without increasing trajectory length. To better examine their distinction, we added two experiments: E1 assesses whether Exploration's entropy response persists after accounting for trajectory-length differences; E2 quantifies the correlation between entropy and length responses across different modes.

E1. Entropy response after accounting for trajectory length

For both Exploration and Effort modes, we pair each mode-intervention branch with a control branch generated from the same branching point, using the same head gains randomly permuted across heads. We measure ΔH\Delta H, the difference between the branches' mean next-step entropy after the branching point, expressed as a percentage of baseline entropy. Entropy is averaged over decoding steps. For each mode class, we report three analyses of these paired comparisons. All reports the unadjusted mean ΔH\Delta H across all branch pairs. Length-matched restricts the analysis to pairs whose trajectory lengths differ by at most eight steps. Length-adjusted reports the mean response after subtracting the fitted length-dependent component, estimated separately for each backbone by regressing ΔH\Delta H on trajectory-length change. The latter two analyses assess whether the entropy response persists within pairs of comparable length and after accounting for its linear association with length change, respectively.

Table 1. Entropy response ΔH\Delta H (%) relative to the permuted-head control.

DatasetMode classAll (unadjusted)Length-matchedLength-adjusted
GSM8KExploration27.414.626.5
GSM8KEffort6.45.86.2
ARC-CExploration17.911.817.4
ARC-CEffort4.82.55.2
ARC-EExploration19.113.519.7
ARC-EEffort6.14.37.4
MedQAExploration16.04.913.0
MedQAEffort2.32.72.8
HotpotQAExploration48.824.941.1
HotpotQAEffort22.913.921.4

As shown in Table 1, Exploration mode retains a positive entropy response in the length-matched analysis on all datasets.

E2. Correlation between entropy and length responses across modes.

We further quantify the association between entropy and length responses across modes. For each mode, EE denotes its entropy response relative to the permuted-head control, expressed as a percentage of baseline entropy. Its length response, ∣A∣|A|, is half the absolute difference between the trajectory-length changes under the +α+\alpha and −α-\alpha interventions, each measured relative to its permuted control, in steps. We assess their association using Pearson, Spearman, and Kendall correlations: Pearson r=0.17r=0.17 (p=0.31p=0.31), Spearman ρ=0.23\rho=0.23 (p=0.16p=0.16), and Kendall τ=0.16\tau=0.16 (p=0.16p=0.16). These results indicate a weak association among modes.

Together, E1 and E2 provide empirical support for the distinction between Exploration and Effort.

W1-b. The definition of the Monitoring mode.

We apologize for the unclear definition. A mode is Monitoring if (a) driving it with ±α\pm\alpha changes neither entropy nor length beyond its permuted control, and (b) reading its activity predicts whether the trajectory will be correct. We will provide clearer explanation in the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer w473 (Part 2/4)

W1-c / Q2. The detailed mechanism for triggering model g.

We apologize for the unclear description. Intervention is triggered by a risk score r=g(zMon,c0,τ0,κ0)r=g(z^{\mathrm{Mon}},c_0,\tau_0,\kappa_0).

Inputs. The Monitoring features zMonz^{\mathrm{Mon}} are obtained by projecting attention-head outputs onto each mode's output direction and combining them across heads. This produces three sentence-level activity measures, differing in their use of mode loadings and head-output magnitudes. These measures are recorded at the saved early fork boundary tft_f. The remaining inputs are the mean log-probability of the answer tokens (c0c_0), the log-scaled latent-step count (τ0\tau_0), and an indicator that the trajectory reached the step cap without terminating (κ0\kappa_0). All features use training-set standardization.

Architecture and training. gg is a two-stage, L2-regularized logistic regression. The first stage maps zMonz^{\mathrm{Mon}} to a scalar sMons^{\mathrm{Mon}}; the second outputs r=σ(w⊤[sMon,c0,τ0,κ0]+b)r=\sigma\left(w^\top[s^{\mathrm{Mon}},c_0,\tau_0,\kappa_0]+b\right). Both stages are fitted by regularized maximum likelihood with C=0.3C=0.3, once per backbone, using trajectories labeled by whether a0a_0 is wrong.

Trigger. We set the risk threshold τ\tau to the 30th percentile of the cross-validated risk scores on the training trajectories. If r<τr < \tau, we return a0a_0 without branching. Otherwise, we generate branches sequentially and vote over their answers together with a0a_0, stopping when the leading answer is at least three votes ahead of the runner-up (ℓ=3\ell=3) or the branch budget is exhausted.

Branch generation. Each branch resumes decoding from the KV-cache snapshot saved at the branching point. A single mode operator with signed gain ±α\pm\alpha is applied during the first k=8k=8 latent steps of the continuation, after which decoding proceeds without further intervention until an answer is produced. The main answer a0a_0 counts as the first vote. We stop when the leading answer reaches the required vote margin, or after 16 branches; the cap is four branches when the main trajectory hits its step ceiling.

AuthorsOpenReview ↗

Response to Reviewer w473 (Part 3/4)

W2-a / Q3: Ablation studies, sensitivity analysis and case studies.

Thank you for the constructive suggestion. We added three sets of analyses:

  • Ablation studies. We examine the effects of branching position (A1) and operator design (A2–A4).
  • Sensitivity analysis. We vary the intervention gain α\alpha (S1), intervention duration kk (S2), and branching position (S3).
  • Case studies. We examine decoded main trajectories and their branches, which will be included in the revised manuscript.

Ablation studies. Specifically, we conduct the following comparisons:

  • A1. Random branching position (Q3). We randomly select the branching point from sentence boundaries along the decoding trajectory. This evaluates the choice of branching position (Table 2).
  • A2. Permuted heads. We randomly permute each operator's gains across attention heads, preserving the gain values while changing their head assignments. This evaluates the importance of the mode-specific assignment of gains to heads (Table 3).
  • A3. Norm-matched perturbation. We rescale the permuted-head gains so that the resulting residual-stream perturbation matches the mode operator's perturbation norm at every layer and decoding step. This assesses whether the mode operator provides a benefit beyond perturbation magnitude (Table 3).
  • A4. Random direction. We replace the mode gains with randomly sampled gains matched to the mode's per-layer gain norm. This compares the mode-specific gain pattern with an unstructured alternative of the same gain magnitude (Table 3).

Each variant in A1–A4 is paired with our originally proposed design on the same evaluation items. A2–A4 use matched compute and otherwise identical evaluation protocols.

Table 2. A1: branching position. Accuracy (%).

ConfigurationGSM8KARC-CARC-EMedQAHotpotQA
Proposed design84.288.295.264.071.2
A1: random branching position82.685.892.761.266.9

The results show that random branching underperforms the proposed method.

Table 3. A2–A4: operator replacements. Accuracy (%).

ConfigurationGSM8KARC-CARC-EMedQAHotpotQA
Proposed design84.288.295.264.071.2
A2: permuted heads81.386.192.061.368.4
A3: norm-matched perturbation81.785.892.461.967.4
A4: random direction81.685.391.661.667.2

As shown in Table 3, the proposed design achieves higher accuracy than all three operator replacements on every benchmark at matched compute. The gains are 2.1–3.2 percentage points over permuted heads (A2), 2.1–3.8 points over norm-matched perturbations (A3), and 2.4–4.0 points over random directions (A4). In particular, the advantage over A3 persists after matching the residual-stream perturbation norm at each layer and decoding step, suggesting that perturbation magnitude alone does not account for the improvement. Together, these comparisons support the benefit of the mode-specific gain assignment over the tested alternatives.

Sensitivity analysis. As shown in Table 4, we evaluate sensitivity to the following three parameters:

  • S1. Intervention gain α\alpha. We evaluate α=0.5\alpha=0.5 and α=2\alpha=2, compared with the default α=1\alpha=1.
  • S2. Intervention duration kk. We apply the gain for 32 latent steps, compared with the default of 8 steps.
  • S3. Branching position. We branch at the first sentence boundary after latent step 64, rather than after step 24.

Table 4. Sensitivity analysis: accuracy (%).

SettingGSM8KARC-CARC-EMedQAHotpotQA
Default84.288.295.264.071.2
S1: α=0.5\alpha=0.584.588.495.063.771.1
S1: α=2\alpha=284.388.194.863.971.4
S2: k=32k=3284.187.995.163.771.0
S3: branching position83.987.694.863.869.5

Table 4 shows limited sensitivity to intervention gain and duration within the tested ranges, whereas delaying the branching point reduces accuracy. These results support the default early branching position and brief intervention.

W2-b: Per-backbone accuracy.

Thank you for the suggestion. We will include them in the revised manuscript.

W3-c / Q5: Use a consistent compute measure and provide a time-based version of Fig. 5.

Thank you for the valuable suggestion. We will use wall-clock time as the consistent compute measurement for each method and update the Fig. 5 in the revision.

Comments on the paper

1 comment

We thank the PC, SAC, AC, and all reviewers for their time and attention. Our main contributions are as follows:

  • A framework for internal orchestration within a frozen LLM. We formulate internal orchestration as two connected problems: discovering reusable internal operators and coordinating them during inference. This framework turns recurring computational patterns into an actionable interface for directing the model’s computation.

  • Discovery of reproducible modes with distinct functional profiles. We extract low-rank modes from attention-head contributions to the residual stream and characterize them through Causal Mode Probing, identifying Exploration and Effort responses alongside predictive Monitoring signals. Their subspaces are reproducible across samples and largely consistent across tasks within each backbone, with similar functional profiles across model families.

  • Mode-guided inference-time scaling. Our orchestrator uses modes as signals for allocating compute and as operators for steering reasoning, improving the accuracy–compute trade-off.

Main reviewer concerns and our responses:

  • Contribution of mode structure. Compute-matched controls retain the complete controller and replace only the operators. Mode-specific interventions outperform permuted-head, residual-norm-matched, and random-direction controls on all five benchmarks, addressing concerns about gain attribution and perturbation magnitude.

  • Consistent compute accounting and per-backbone results. We now use a common FLOP protocol covering prefill, decoding, answer readout, and intervention overhead, with per-backbone accuracy and compute reported in Appendix A.6. Our method remains on the empirical accuracy–compute Pareto frontier under this accounting.

  • Functional roles and adaptivity. Length-matched and length-adjusted analyses support distinct Exploration and Effort effects.

  • Monitoring’s incremental and diagnostic value. At matched compute, adding Monitoring improves accuracy on all five benchmarks. The requested direct comparison shows that Monitoring-only probes achieve AUROC scores close to those using all eight modes. Trigger fitting and thresholding are now explicitly documented.

Our rebuttal addresses the reviewers’ main concerns about the contribution of the discovered operators, the fairness of compute comparisons, Monitoring mode’s added value, and the details of the proposed orchestrator.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.