acceptodds
Under review as a conference paper at ICLR 2027

Which Models to Mix Depends on How You Mix Them: Composition-Aware Portfolio Selection for Mixture-of-Models

Abstract

Given a large pool of candidate language models, which small subset should be composed into a Mixture-of-Models? Existing rules score candidates by individual accuracy, error diversity, or complementary coverage, and implicitly value a pool by the probability that at least one member is correct. This assumption fails for an entire class of deployed systems. We distinguish *selection*-type composers, which output a member's answer, from *assembly*-type composers, which synthesize an answer from member evidence, and show that the two induce structurally different portfolio objectives. Under a subgoal decomposition of difficulty, the gap between their ceilings equals the probability that the members solve a query collectively while none solves it individually. This yields a dichotomy: the ceilings coincide exactly when this event is impossible; otherwise, the optimal portfolios can be disjoint, and using either in place of the other incurs a constant loss. Tractability is similarly asymmetric: the selection objective is monotone submodular at every information level, whereas the assembly objective exhibits increasing returns and admits no polynomial-time constant-factor approximation unless P = NP. Therefore, we recover a guarantee through a submodular logarithmic surrogate. Graders observe competence only after a member's upstream errors have propagated, so we build the surrogate on intrinsic competence, recovered in closed form by inversion over the prerequisite order. Without it, a selector can discard specialists that make assembly worthwhile. Our method performs best on six benchmarks with a 42-model pool, and its diagnostic indicates whether a homogeneous or heterogeneous portfolio performs better before composition runs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mixture-of-Models, Portfolio Selection, Model Routing

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_sYQTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper examines the problem of selecting a portfolio of models in Mixture-of-Models systems and claims that the optimal set of models depends strongly on the way in which the chosen models are combined. It makes a distinction between selection-type composers, which simply return the result of one of the selected models, and assembly-type composers, which are able to produce an answer by combining partial evidence from several models. Using a decomposition of task difficulty into subgoals, the authors introduce separate value functions for the two different cases and describe the difference between their upper bounds in terms of the probability of distributed solvability: instances in which the entire portfolio manages to solve all the required subgoals even though no single model is able to solve the complete query. The paper also demonstrates that selection objectives are monotone submodular, while the assembly objective may show increasing returns and is not polynomial-time constant-factor approximable unless P=NPP=NP. In order to deal with this, the authors suggest a logarithmic submodular surrogate and also derive a propagation correction that allows the intrinsic subgoal competence to be recovered from graded competence that has been affected by failures at upstream stages.

Based on these theoretical findings, the paper introduces CAPS, a portfolio selection framework that is aware of composition. CAPS combines sparse subgoal probing with low-rank tensor completion, incorporates conditional competence prediction, carries out the separate optimization of the various composition branches, includes held-out branch calibration, and uses a cross-fitted estimator in order to reduce the bias resulting from the maxima of the estimated model scores. The empirical evaluation involves six benchmarks-MMLU-Pro, MATH-500, HotpotQA, MuSiQue, EvalPlus, and LiveCodeBench-together with a live pool of 42 models drawn from nine families, as well as several routing and composition baselines. The experiments show that CAPS achieves the highest realized accuracy on all six benchmarks, with greater improvements observed on those benchmarks where the estimated decomposability gap is larger.

While I acknowledge the merit in distinguishing between selection and assembly, I am unconvinced that the current experiments sufficiently demonstrate the breadth and scalability implied by the paper's approach. Specifically, the fully composition-aware pipeline is evaluated only on a pool of 42 models, whereas experiments with thousands of candidate models rely on precomputed correctness matrices and focus exclusively on selection due to the absence of subgoal annotations.

Critical strengths
  1. The paper points out a significant conceptual gap in the current approaches to model portfolio selection.

A key strength of the paper is the argument that portfolio selection cannot generally be separated from the composition operator which consumes the portfolio. The distributed-solvability identity provides a clear account of the cases in which assembly can benefit from models that would seem useless when assessed using standard end-to-end correctness metrics. Moreover, the separation result demonstrates that selection-optimal and assembly-optimal portfolios can be entirely different, showing that the problem goes beyond a minor adjustment to existing coverage-based methods.

This is important since a large portion of the current literature on model selection assesses the candidates according to individual accuracy, diversity, or complementary end-to-end coverage. The paper gives a sound explanation for why such criteria can systematically exclude specialists who prove to be useful only when partial solutions are combined.

  1. The theoretical treatment goes beyond a single algorithmic contribution.

The paper presents a number of related findings: the submodularity of the selection branch, the difficulty of the assembly branch, a logarithmic surrogate, a decomposition of the assembly gains into distributed solvability and propagation repair, and a correction for upstream-error masking. This endows the work with greater conceptual depth than a purely empirical approach to model selection. The fact that propagated grading is able to suppress late-stage specialists is especially interesting and could be of use outside of the particular CAPS implementation.

Critical weaknesses
  1. The empirical validation is too limited compared with the extent of the claims made in the paper.

Even though the paper presents its results on six benchmarks and involves several ablation studies, the complete composition-aware CAPS pipeline is only assessed on a live pool consisting of 42 models. The experiments with thousands of models make use of the RouterBench, LLMRouterBench, and RouterEval correctness matrices, which fail to provide subgoal labels and therefore only test the selection aspect of the framework.

It is important to make this distinction since one of the main aims of the paper is to carry out portfolio selection from very large model pools, whereas the most innovative elements-assembly-aware selection, intrinsic-competence recovery, and propagation-aware portfolio selection-are not subjected to end-to-end testing at that scale. The experiments carried out so far therefore show that CAPS functions in a moderately sized and structured environment, but have not yet proven that the complete approach can be scaled up to the wider deployment scenario stressed in the paper.

The fact that it is computationally expensive also means that this question is nontrivial rather than just one that has been omitted experimentally. The paper states that at a probe rate of ρ=0.04\rho=0.04, it takes approximately 1.9K A100 GPU-hours to construct the graded tensor for the 42-model live pool, together with an extra cost from the prefix-forced probes. Although the downstream optimization is relatively cheap, it is exactly the probing stage that would have to scale as the candidate pool increases.

  1. The experimental study of the mechanisms of composition is limited.

The theory is based on three broad categories of selection-type, splice-free assembly, and splice-capable assembly operators, but the empirical study only includes one specific composer for each category-a pre-answer selector, a synthesis-based assembly composer, and a splice-capable composer which explicitly chooses subgoal sources and then re-executes the downstream computation.

Consequently, it is hard to tell if the portfolio differences observed are reliable when applied across the wider range of systems covered by the theoretical definitions. For instance, the evaluation does not show that the same conclusions apply in the case of alternative summarizers, verifier-guided synthesis, multi-round debate, iterative refinement, different prompts, or agent architectures that have dynamically generated subgoals.

This distinction is important because the practical relationship between portfolio composition and realized accuracy can depend greatly on the composition method. Although the theory distinguishes the classes in principle, the current experiments provide only limited evidence that CAPS generalizes robustly across different implementations. The method depends on subgoal-level competence, a prerequisite structure, and, in some cases, prefix-forced probing. While these requirements are natural for unit-test-based code benchmarks and can be implemented for step-graded mathematics and annotated multi-hop QA, they are much less straightforward for open-ended generation tasks, where subgoal decomposition may be ambiguous and correctness may not be conjunctive.

The paper makes it clear that its framework does not define the correct combination of rubric items for open-ended generation and that prefix-forced probing has no obvious equivalent in that case.

Although the benchmarks selected are suitable for verifying the theoretical mechanism under controlled conditions, they also greatly favour the assumptions needed by the proposed method. It would be important to carry out further experiments using noisy, learned, or imperfect subgoal decompositions in order to show robustness outside of these structured situations.

The robustness analysis does not place adequate stress on the key structural assumptions.

The paper includes some useful generalisation and stability experiments, such as a model split in which the data sets are disjoint across families and a shifted task distribution. CAPS is still stronger than the baseline methods in both cases, though its performance decreases when there is a distribution shift.

Yet these experiments do not directly examine the assumptions which are most particular to the method-namely, the local independence of intrinsic competence, the correctness of the prerequisite DAG, the reliability of subgoal grading, and the accuracy of the deconvolution procedure. Because breaches of these assumptions can have a direct effect on the inferred complementarity between the models, I think this is an important aspect that has so far been lacking empirically.

Questions for authors
  1. Clarification / possibly altering the assessment The end-to-end assembly-aware experiments have been carried out on 42 live models, while the studies involving thousands of candidate models involve only model selection. Could the authors provide end-to-end assembly-aware experiments on a much larger set of models, or at least a realistic cost extrapolation that shows how the probing and subgoal-grading stages scale with KK? Since the present experiment with 42 models already demands a significant amount of GPU time, it appears to be important for assessing the practical scalability claim.

  2. A clarification / one which could change the assessment
    How dependent are the empirical conclusions on the specific composer implementations employed in the paper? Although the theoretical distinction refers to broad categories of selection and assembly operators, only one particular composer is actually used for each category. The paper would be greatly strengthened if it showed that similar portfolio rankings or gains are obtained with multiple summarizers, different prompts, verifier-based composers, or with multi-round composition schemes.

  3. A clarification / one that could change the assessment
    Can the authors assess the method's robustness when the prerequisite structure is imperfect? For instance, what would occur if edges in the prerequisite DAG were randomly deleted or inserted, or if the DAG was automatically inferred rather than being obtained from the benchmark metadata? Because the propagation correction relies directly on this structure, such an experiment would help in measuring the sensitivity of the method to incorrect specification of the structure.

  4. A clarification / one that could change the assessment Can the authors give empirical evidence in support of the local-independence assumption which underlies the propagation correction? More specifically, once the learned latent difficulty representation has been taken into account, what level of residual correlation remains between the intrinsic competence variables across the models, particularly among those models which belong to the same family? A sensitivity analysis involving controlled violations of this assumption would help to determine whether the recovered intrinsic competence is reliable.

4 comments
AuthorsOpenReview ↗

Response to Reviewer sYQT (Part 1)

W1: The empirical validation is too limited compared with the extent of the claims made in the paper. ...

A1: This is a fair description of the gap, and we have narrowed it in three ways rather than argue it away. All three are collected in the new Appendix E.7, with a pointer from the experiments section.

First, we now report the selection-branch results on the public corpora explicitly instead of only alluding to them: new Table 9 gives RouterBench (K=11K=11), LLMRouterBench (K=52K=52) and RouterEval (K=8,567K=8{,}567) at k=4k=4, with the probe rate used in each. This does not test the assembly machinery, as the reviewer says, and we label it as such.

Second, we have run an end-to-end assembly-aware experiment at a larger pool. We expanded the live pool to K=126K=126 by pairing each checkpoint with three decoding and prompt configurations, which are genuinely distinct members from the portfolio's point of view (they differ in subgoal competence and in cost) and whose grading reuses the same harness; Appendix F documents the construction and notes that it adds no new weights. On MuSiQue and LiveCodeBench, CAPS improves from .465.465 to .478.478 and from .461.461 to .474.474 at k=4k=4 when the pool grows, while the probe cost grows only from 1.91.9 to 2.22.2 thousand A100 hours (new Table 10).

Third, and most to the point of (3), we now give the cost model instead of leaving the extrapolation to the reader. The probe budget needed for a fixed completion error is O((K+nL)r)O((K+nL)r) cells, not O(KnL)O(KnL), so the required rate falls roughly as 1/K1/K and the total cost is dominated by the item side once K≪nLK \ll nL. Holding completion error fixed and extrapolating from our measured per-cell cost, a 1,0001{,}000-member pool costs about 2.22.2 thousand A100 hours, within about 15%15\% of the 4242-member run; the independently measured ρ=0.008\rho = 0.008 needed at K=8,567K = 8{,}567 on RouterEval is consistent with the same scaling. The marginal cost of placing one newly released checkpoint is on the order of rTr T graded generations, which is what makes the model-level embedding of Eq. (11) worth having. Appendix E.7 states explicitly that the extrapolation holds completion error and the benchmark suite fixed, and says nothing about how many candidates are worth grading in the first place.

W2: The experimental study of the mechanisms of composition is limited. [...] only one specific composer for each category [...] it is hard to tell if the portfolio differences observed are reliable when applied across the wider range of systems covered by the theoretical definitions [...] It would be important to carry out further experiments using noisy, learned, or imperfect subgoal decompositions in order to show robustness outside of these structured situations. [...] these experiments do not directly examine the assumptions which are most particular to the method.

A2: We have added the three studies this asks for, as Appendices E.6, E.8 and E.9; the details are in A4, A5 and A6 below, and we summarize the design here.

On composer mechanisms, the theory is a statement about classes, so the right test is whether the portfolio is stable within a class and changes across classes. We now instantiate thirteen composers spanning the three classes: router head, confidence cascade and judge picker for selection; two different summarizers, a two-layer Mixture-of-Agents, debate followed by synthesis, verifier-guided synthesis and an alternative prompt for splice-free assembly; and subgoal splicing, iterative per-subgoal repair and verifier-guided splicing for splice-capable assembly, with self-consistency over repeated draws of one member as a selection-type control. The selected portfolio has mean pairwise Jaccard .86.86 within a class and .38.38 across classes, and our portfolio beats the coverage portfolio under every one of the thirteen (Table 8).

On imperfect structure, we perturb the prerequisite graph by random edge deletion and insertion at rates up to 30%30\%, and we also replace the annotated graph with one the anchor infers from the query alone. Degradation is gradual, not sharp (Table 11).

On the assumptions specific to the method, we now measure the residual dependence that Assumption 9 forbids, as a function of the completion rank and separately within and across model families, and we inject controlled violations and report their effect on both Δ^\widehat\Delta and realized accuracy (Table 12).

We have also added one sentence to the extended limitations: because the splice-free branch needs no prerequisite structure, a deployment without usable decompositions still has a two-branch version of the method available, which is the honest statement of how far the framework reaches into open-ended generation.

AuthorsOpenReview ↗

Response to Reviewer sYQT (Part 2)

(Following above Part 1)

Q1: Clarification / possibly altering the assessment. The end-to-end assembly-aware experiments have been carried out on 42 live models, while the studies involving thousands of candidate models involve only model selection. Could the authors provide end-to-end assembly-aware experiments on a much larger set of ...

A3: Both, as described in A1 and reported in Appendix E.7: a 126126-member end-to-end assembly-aware run (Table 10), and an explicit cost model with the extrapolation to K=1,000K=1{,}000. The key quantitative point is that the probe requirement for fixed completion error is O((K+nL)r)O((K+nL)r) rather than O(KnL)O(KnL), so per-model probe cost falls as the pool grows and total cost is close to flat in KK in the regime where K≪nLK \ll nL: about 2.22.2 thousand A100 hours at K=1,000K=1{,}000 against 1.91.9 thousand at K=42K=42. We also give the marginal cost of adding one checkpoint, which is the number a deployment actually faces. We make no claim beyond what the measurement and the scaling support.

Q2: A clarification / one which could change the assessment. How dependent are the empirical conclusions on the specific composer implementations employed in the paper?

A4: New Appendix E.6 (Table 8) reports all thirteen instantiations listed in A2, on MuSiQue and LiveCodeBench, each with the portfolio our method selects, the portfolio greedy coverage selects, and the Jaccard overlap against the reference composer of the same class. Three findings:

  1. Within a class the portfolio is stable. Mean pairwise Jaccard .86.86 within a class. Changing the summarizer, the prompt, or moving from a single summarizer to a two-layer stack, to debate-then-synthesis or to verifier-guided synthesis moves realized accuracy by at most 2.22.2 points and the portfolio by at most one member.
  2. Across classes it changes. Mean Jaccard .38.38 between classes, which is the dichotomy showing up where the theory says it should.
  3. The ordering is invariant. Our portfolio beats the coverage portfolio under every instantiation, by 1.21.2 to 5.35.3 points, including under verifier-guided synthesis and iterative refinement, which are the two variants furthest from the composer we tuned on.

Q3: A clarification / one that could change the assessment. Can the authors assess the method's robustness when the prerequisite structure is imperfect?

A5: New Appendix E.8 (Table 11) reports exactly the experiment described: random perturbation of the prerequisite graph at rates 10%10\%, 20%20\% and 30%30\% (each perturbation deletes or inserts an edge, subject to acyclicity), an anchor-inferred graph built from the query alone, and a no-graph condition in which only the splice-free branch is available. On MATH-500 and MuSiQue, accuracy falls by 0.30.3, 0.80.8 and 1.61.6 points at the three perturbation rates and the selected specialist share falls from .46.46 to .37.37, which is the mechanism Proposition 20 predicts: a corrupted ancestor set mis-scales exactly the members whose value sits at deep subgoals. The inferred graph costs 0.60.6 points, so most of the benefit survives without annotations, and the prompt that produces it is now in Appendix G.6. With no graph at all, the method falls back to the splice-free branch, where it matches the strongest baseline on MATH-500 and still leads on MuSiQue. The limitations section now points at this curve where it says that a misspecified reachability matrix enters the inversion of Eq. (15) directly.

Q4: A clarification / one that could change the assessment. Can the authors give empirical evidence in support of the local-independence assumption which underlies the propagation correction?

A6: New Appendix E.9 (Table 12) reports both halves of what the reviewer asks for.

Residual correlation. After removing the fitted item factors, the mean absolute residual correlation of intrinsic competence between model pairs is .07.07, .04.04 and .03.03 at completion ranks 88, 1616 and 3232. As the reviewer anticipates, it is concentrated within families: .19.19, .11.11 and .07.07 at the same ranks, against .03.03 to .06.06 across families. So the assumption is approximately right at the rank we use and is violated mostly between siblings, which is also the regime in which the deconvolution would overstate complementarity.

Controlled violation. We inject a shared misconception into a randomly chosen family at a fraction qq of subgoals. At q=.1,.2,.3q = .1, .2, .3 the decomposability gap is inflated by .009.009, .021.021 and .038.038 and realized accuracy falls by 0.40.4, 0.70.7 and 1.31.3 points. The direction is the one to worry about, since inflating Δ\Delta makes the assembly branch look better than it is, so we now recommend reporting the within-family residual correlation alongside the gaps and treating a large value as a reason to increase the rank before trusting the diagnostic. We also note there that the realized comparison does not depend on the assumption.

AuthorsOpenReview ↗

Follow-up on Rebuttal and Revision

Dear Reviewer sYQT,

Thank you again for your constructive suggestions. We would appreciate your thoughts on whether the new experiments with 13 composer instantiations, the expanded 126-member pool and cost analysis, and the robustness tests for imperfect prerequisite graphs and dependence violations have addressed your concerns about generality and scalability. Please let us know if any concern remains or further clarification would help. If the added evidence has resolved your concerns, would you consider raising your rating?

Best regards,

The Authors

Dear Reviewer sYQT,

With one day remaining, we wanted to ask whether the new experiments address your questions about generality and scalability. Table 8 compares our selected portfolio with the coverage portfolio across 13 composer instantiations, and Table 10 reports an end-to-end evaluation with 126 members. If you feel that an important setting is still missing, we would welcome your feedback before the discussion closes. If the added evidence resolves your concerns, would you consider raising your rating?

Best regards,

The Authors

Reviewer_2UV2OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

CAPS selects model portfolios according to the downstream composition operator, distinguishing answer selection from subgoal assembly and accounting for error propagation.

Critical strengths

The paper addresses an important question: how model complementarity depends on the composition rule.

The restricted subgoal model yields useful insights, and the reported evaluation covers 42 models and six benchmarks.

Critical weaknesses

The assignment-based assembly oracle does not automatically characterize unrestricted generative synthesis, weakening the connection between the theory and the evaluated systems.

The reported ceiling estimates are difficult to reconcile with their definitions. For MMLU-Pro (L=1, no prerequisite edges), the three ceilings are algebraically identical for every fixed portfolio, yet Table 1 reports different values and nonzero gaps. It is unclear whether these differences arise from inconsistent estimators or different portfolios. Moreover, the multi-draw procedure appears to conflate the expectation of a maximum with the maximum of expected success probabilities, potentially changing the coverage estimand.

The plain gain/cost greedy procedure does not inherit the claimed knapsack approximation guarantee.

Questions for authors

Please specify the ceiling estimators, including the ordering of expectations and maxima, and explain why the MMLU-Pro ceilings differ despite their algebraic equivalence.

Under what conditions do the implemented composers satisfy the subgoal-assignment model, and are those conditions enforced or validated in the experiments?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 2UV2 (Part 1)

W1: The assignment-based assembly oracle does not automatically characterize unrestricted generative synthesis, weakening the connection between the theory and the evaluated systems.

A1: We agree, and we have narrowed the claim in the paper and measured the residual.

The precise scope is this: V†V^{\dagger} and V‡V^{\ddagger} upper-bound any composer whose output at each subgoal is either adopted from a member's evidence or produced by the composer itself. The second clause is why Φ\Phi is a member of SΦS_\Phi and carries its own competence, which covers the common case where a summarizer repairs a step none of the proposers got right. What the oracle does not cover is correctness at a subgoal that arises from the interaction of several wrong candidates, with neither any member nor the anchor acting alone being right. We now (a) state this scope explicitly after Eq. (5) in the main text, and (b) measure the uncovered mass. On probe items we check, per subgoal, whether the composed output is graded correct while no member's and not the anchor's own solo output is; this emergent synthesis rate is 0.7%0.7\% to 3.1%3.1\% across the six benchmarks (new Table 7, Appendix E.5). So the ceilings are valid up to that mass, and since realized accuracies sit 1919 to 4949 points below them, the ceilings are nowhere near binding in our experiments.

We would also add the point that matters most for the method rather than for the theory: the optimizer never needs the ceiling to be attained, only the objective to order portfolios the way the real composer does. We now test that directly. Over 5050 random size-4 portfolios per benchmark, the Spearman correlation between an objective and the realized accuracy of the matching composer is, across the five structured benchmarks, .36.36 to .46.46 for coverage and .74.74 to .84.84 for our two log-surrogates (Table 7); on the atomic control all three agree. That is the property the greedy relies on, and it is measured rather than assumed.

W2: The reported ceiling estimates are difficult to reconcile with their definitions. For MMLU-Pro (L=1, no prerequisite edges), the three ceilings are algebraically identical for every fixed portfolio, yet Table 1 reports different values and nonzero gaps. It is unclear whether these differences arise from inconsistent estimators or different portfolios. Moreover, the multi-draw procedure appears to conflate the expectation of a maximum with the maximum of expected success probabilities, potentially changing the coverage estimand.

A2: The reviewer has diagnosed this exactly, and the second sentence is the cause of the first. Our C^\widehat C took a union over draws as well as over members, while the two assembly rungs took a maximum over members of per-cell success probabilities; at L=1L=1 the difference between these two estimands is precisely the spurious .021.021. The portfolio was not the issue; it was the same at all three rungs.

We have adopted a single convention: the per-cell success probability is the primitive, and every rung takes the maximum over members inside the item average (Eq. (19), Appendix G.3). Consequences, all now in the paper: at L=1L=1 the three estimators are identical term by term, so MMLU-Pro's ceilings coincide and both gaps are exactly zero at every probe item; the union-over-draws convention is named and explicitly rejected, with a note that it is what makes single-draw grading inflate CC (the 4.64.6-point effect in Table 4); and the identities of Section 4 now hold for the estimates, not only for the estimands, which lets us use the atomic benchmark as a pipeline check. The same correction, combined with the new Corollary 4 on chain-structured items, also zeroes the MATH-500 decomposability gap and moves its assembly advantage to the propagation term.

W3: The plain gain/cost greedy procedure does not inherit the claimed knapsack approximation guarantee.

A3: Correct, and Reviewer vb9E gives a two-item witness. Theorem 1 now distinguishes the cardinality case (1−1/e1-1/e for greedy) from the knapsack case, where density greedy plus the best feasible singleton gives (1−1/e)/2(1-1/e)/2 and only Sviridenko's partial enumeration gives 1−1/e1-1/e, and states that density greedy alone has no constant-factor guarantee. Algorithm 1 implements the singleton safeguard by default and notes the enumeration variant. The witness is recorded as Remark 22. All reported experiments use the cardinality budget, so no number changes.

AuthorsOpenReview ↗

Response to Reviewer 2UV2 (Part 2)

(Following above Part 1)

Q1: Please specify the ceiling estimators, including the ordering of expectations and maxima, and explain why the MMLU-Pro ceilings differ despite their algebraic equivalence.

A4: The estimators are displayed in A2 above and as Eq. (19) of the revision: in all three, the maximum over members is taken inside the item average, at the level of per-cell success probabilities. The MMLU-Pro ceilings differed because the coverage rung alone also took a union over the TT draws, which is a different estimand; under the unified convention they coincide exactly and the gaps are identically zero. Appendix G.3 states the portfolio at which the ladder is evaluated, the number of labeled probe items behind each limit, and how the limits are computed (item bootstrap, with the Clopper and Pearson limit on the hard-graded indicator reported alongside for comparability). Table 1 now carries the sample size as a column so that every limit is reproducible from the table.

Q2: Under what conditions do the implemented composers satisfy the subgoal-assignment model, and are those conditions enforced or validated in the experiments?

A5: Enforced for one class, validated for the others, and we now report both in Appendix E.5.

Enforced. For code, splicing is mechanical: the splice-capable composer names a source candidate per module and the program is assembled programmatically, so the assignment model holds by construction. The selection composer returns a member's answer verbatim, so its output lies in the member set by construction.

Validated. For mathematics and multi-hop the splice-capable composer is instrumented: it must emit SUBGOAL k: candidate j lines, and we check whether the subgoal content it then produces actually matches the named candidate. This compliance rate is .95.95 to .96.96 on the two code suites, .91.91 on MATH-500 and .87.87 to .88.88 on the two multi-hop sets (Table 7). The complement is the composer paraphrasing or silently rewriting a sub-solution, which is also where the emergent-synthesis mass of W1 lives.

Not enforced anywhere. The splice-free composer is free to write anything; we do not constrain it, and the 0.7%0.7\% to 3.1%3.1\% emergent-synthesis rate is our measurement of how far outside the assignment model it actually goes. We now state in the main text that Eq. (5) is a ceiling for adoption-or-anchor composers and that the residual mass is measured rather than assumed.

AuthorsOpenReview ↗

Follow-up on Rebuttal and Revision

Dear Reviewer 2UV2,

Thank you again for your thoughtful feedback. We wanted to follow up on whether the clarified scope of the assembly oracle, the new composer-validation results, and the corrections to the ceiling estimators and knapsack guarantee have addressed your concerns. We would be happy to discuss any remaining questions about the theory or its connection to the evaluated systems. If you find that the revision has resolved your concerns, we would be grateful if you would consider raising your rating.

Best regards,

The Authors

Reviewer_vb9EOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks which small subset of a model pool to compose into a Mixture-of-Models. It separates selection-type composers, which return a member's answer, from assembly-type composers, which synthesize one. Under a conjunctive subgoal model with propagation over a prerequisite DAG (Eqs. (1), (2)), the gap between assembly and coverage ceilings splits into a decomposability gap Δ\Delta and a propagation gap Δprop\Delta_{prop} (Eq. (10)). Theory covers ceiling coincidence (Thm. 3), disjoint optima (Thm. 4), submodular selection (Thm. 1), hard assembly (Thm. 7) and a deconvolved log surrogate (Thm. 9, Prop. 17). The method, CAPS, adds low-rank completion, per-branch greedy and calibration, tested with 42 open models on six benchmarks.

Critical strengths

S1. A clean formulation of a real design question. Eq. (10) ties Δ\Delta to keeping complementary members and Δprop\Delta_{prop} to repairing intermediate results (l.279-281). Thm. 4 gives an explicit, hand-checkable instance (l.196-200). Why this matters to my assessment: making the composer part of the objective is the paper's main value.

S2. Claims are carefully scoped. Eq. (9) certifies the portfolio "rather than claiming near-optimality" (l.268-269). Why this matters to my assessment: it makes the theory easier to trust.

Critical weaknesses

W1. Reported numbers contradict the paper's own identities and tables, and the gap estimator is not defined.

  • MMLU-Pro has L=1L = 1 and no edges (Table 7). Thm. 3 holds "whenever L=1L = 1" (l.191) and B=AB = A without edges (l.133), so both gaps are identically zero. Table 1 reports .021 and .004, called "consistent with Theorem 3" (l.410). This is an identity, not a test.
  • MATH-500 is a chain (l.1403). Under Eq. (2) each graded vector is a prefix of ones, so the vectors form a chain, which Thm. 3 states is sufficient for Δ=0\Delta = 0 (l.190). Table 1 reports .094, above τ=0.05\tau = 0.05 (Table 8).
  • HotpotQA has Lˉ=2.4\bar L = 2.4, dˉ=1.4\bar d = 1.4, with depth the edge count of the longest path (l.1388-1389). Depth is at most L−1L - 1 and items at that bound are chains, so allowing for rounding Δ≤0.10\Delta \le 0.10. Table 1 reports .168.
  • Table 3 contradicts Table 2. The Greedy coverage mean over the four benchmarks with Δ^>0.15\hat\Delta > 0.15 in Table 2 is (.521 + .413 + .628 + .396)/4 = .4895, but the "Coverage ceiling C" row of Table 3 reports .481 and .501 (l.436). The MMLU-Pro column of Table 3 copies Table 2 exactly (.679, .685), so the two tables should agree. The 98%, 96% and 51% at l.460-462 reproduce only with the .4895 baseline: with Table 3's own .501, random branch choice recovers (.516 - .501)/(.541 - .501) = 37.5%, and with .481 it recovers 58%.
  • The pool is described as "ranging from 0.5B to 72B parameters" (l.361), while Table 6 lists Mixtral-8x22B at 141B (l.1373).
  • No formula is given for the estimates, the portfolio used, or the counts behind the Clopper-Pearson limits (l.327-328).

The Branch column, Table 3 grouping and Fig. 2 x-axis rest on the Table 1 values. Why this matters to my assessment: I cannot verify that "the decomposability gap tracks assembly gains" (l.377).

W2. Lack of novelty, and the main comparison does not isolate what is new.

  • In the authors' words, Thm. 1 "contains the coverage result of Shihab et al. (2026)" (l.169), and "the knapsack extension is classical (Sviridenko, 2004) and we claim no novelty for it" (l.172). The completion model is that of Song et al. (2025) and Zhuang et al. (2025) (l.302). Individually weak proposers that are jointly sufficient, the effect behind Thm. 4, already appear in a counterexample of Zhang et al. (2026) (l.061-064), the closest work, which also proposes marginal-contribution greedy (l.103-105).
  • "Model-first greedy" is the strongest baseline in every column of Table 2 but is never defined, and its relation to "the greedy variants of Zhang et al. (2026)" (l.368) is not stated.
  • G.6 says all methods share the portfolio size, "so differences are attributable to the selection objective alone" (l.1530-1531), yet calibration "stops CAPS between 3 and 5" (l.470-471) and removing it changes size by +3 (Table 4).
  • Branch choice is attributed to the diagnostic (Table 1 caption, l.423-424) and to calibrated comparison (l.319-322, Alg. 1). The CAPS mean over the four high-gap benchmarks, (.559 + .463 + .673 + .461)/4 = .539, matches the "Diagnostic as a shortcut" row of Table 3, not "CAPS enumerate and compare" (.540).

Why this matters to my assessment: with most parts credited to prior work, the contribution is the objective, and Table 2 does not separate it from calibration and protocol.

Questions for authors
  1. What are the exact estimators behind Table 1? Give formulas, the portfolio and the Clopper-Pearson nn; explain the nonzero MMLU-Pro and MATH-500 values; say whether any branch changes. Which baseline produces the 98%, 96% and 51% at l.460-462, and which Table 3 entries are correct?
  2. What does the splice-capable composer see on test items? It "additionally receives the per-subgoal decomposition" and re-runs "the test suite" (l.1459-1462). Are these the grading subgoals and tests? l.1432 covers only probes.
  3. Which algorithm does the knapsack claim of Thm. 1 cover? Alg. 1 (line 7) uses the plain ratio. With anchor value 0, budget 1, and members (cost, accuracy) (0.01, 0.02) and (1, 1), it returns 0.02 against an optimum of 1.
  4. Where are the cited results? The two scoring modes (l.1339), the cheapest-model baseline (l.365), RouterBench results (l.359-361), and the specialist share 0.46 to 0.18 attributed to Table 4 (l.419-420), which has no such column.
7 comments
AuthorsOpenReview ↗

Response to Reviewer vb9E (Part 1)

W1. Reported numbers contradict the paper's own identities and tables, and the gap estimator is not defined.

A1: Thanks for pointing out these items. We have fixed all six, and in doing so we found that two of them share a single root cause: the ladder was estimated with two different orderings of maximum and expectation. We walk through them in order and then list the resulting edits.

(i) The two structural zeros, and their common cause. The reviewer is right that MMLU-Pro cannot have a nonzero gap: at L=1L=1 the three ceilings are the same functional of the same quantities, for every portfolio, so no sample can separate them. The reason our table reported .021.021 is that CC was estimated as the probability that at least one draw of at least one member was correct (a union over the TT draws as well as over members), while V†V^{\dagger} and V‡V^{\ddagger} were estimated as item averages of a maximum over members of per-cell success probabilities. Those are different estimands, and their difference at L=1L=1 is exactly the spurious .021.021. We have adopted one convention everywhere: the per-cell success probability p^miℓ\hat p_{mi\ell} (and the end-to-end rate c^mi\hat c_{mi}) is the primitive, and all three rungs take the maximum over members inside the item average. Under this convention the three estimators are algebraically identical at L=1L=1, the estimated gaps are exactly 00 at every probe item, and we now say so and use the atomic benchmark as a pipeline check rather than as evidence. This is also Reviewer 2UV2's W2, and the formulas are now written out as Eq. (19) in the new Appendix G.3.

(ii) Chains. The reviewer's derivation is exactly right and, we think, worth putting in the paper rather than only in a reply: under Eq. (2) a prerequisite graph with a path through all LL subgoals forces every graded vector to be a prefix of ones, prefixes of ones are totally ordered componentwise, and Theorem 3 then gives Δ=0\Delta = 0. We have added this as Corollary 4 (Structural zeros), with a proof in Appendix C.5, and we draw the two consequences the reviewer's argument implies:

  • Δ(S)≤P[the query’s graph has no path through all subgoals]\Delta(S) \le \mathbb{P}[\text{the query's graph has no path through all subgoals}], a bound computable from benchmark metadata before any grading. We now report it per benchmark in the new Table 16 as an identity check, and it dominates every reported Δ\Delta.
  • MATH-500 is a chain, so its decomposability gap is identically zero and its entire assembly advantage is propagation repair. Recomputing under the unified convention gives V†=C=.804V^{\dagger} = C = .804 and V‡=.961V^{\ddagger} = .961, i.e. Δ^=.000\widehat\Delta = .000 and Δ^prop=.157\widehat\Delta_{\mathrm{prop}} = .157 in place of the previous .094.094 and .063.063.

This correction strengthens rather than weakens the "two gaps imply different interventions" paragraph: MATH-500 is now the pure propagation-repair case and EvalPlus the pure distributed-solvability case, and the two are separated by depth exactly as Eq. (17) predicts. We have rewritten that paragraph accordingly.

(iii) HotpotQA and MuSiQue depths. The reviewer's bound Δ≤P[d<L−1]\Delta \le \mathbb{P}[d < L-1] is correct, and our reported depths violated it. The cause is a reporting error in the depth column: it averaged the longest-path edge count over the items that have at least one prerequisite edge, not over all items. This affects only the two multi-hop sets, where a substantial fraction of items have mutually independent supporting facts; on MATH-500 and the code suites every item has at least one edge, so those entries are unchanged, and MMLU-Pro has no edges at all. Averaging over all items gives dˉ=1.0\bar d = 1.0 for HotpotQA and dˉ=1.3\bar d = 1.3 for MuSiQue. With the structural bound now reported alongside (P[non-chain]=.40\mathbb{P}[\text{non-chain}] = .40 and .62.62), the estimated gaps .168.168 and .213.213 sit well inside it. Table 1 and the subgoal-structure table (now Table 15, Appendix G.2) are corrected, and the latter's caption states that averaging is over all items.

(iv) Table 2 versus Table 3. The reviewer reverse-engineered the baseline correctly. The three percentages were computed against greedy coverage deployed under the matching composer, i.e. the .4895.4895 four-benchmark mean of Table 2, and the .501.501 entry in Table 3 was mis-transcribed. The correct entry is .490.490, which reproduces 98%98\%, 96%96\% and 51%51\% from the unrounded .4895.4895 exactly. We have corrected the entry, stated the baseline explicitly in the caption of Table 3, and updated the dependent sentence: the mismatch cost of coverage under assembly is 4.94.9 points, not 3.83.8 (the 2.62.6 points for the assembly surrogate under selection is unaffected). The caption also notes that on MMLU-Pro the deployed class is selection, which is why that column necessarily repeats Table 2.

(v) Parameter range. Corrected to 0.50.5B to 141141B, with a note in Appendix F that the counts are totals rather than activated parameters.

AuthorsOpenReview ↗

Response to Reviewer vb9E (Part 2)

(Following Part 1)

(vi) Estimators. The new Appendix G.3 gives, in display, the three ceiling estimators, the two gap estimators, the portfolio at which the ladder of Table 1 is evaluated (the size-4 portfolio selected by the splice-capable branch on the probe split; pool-level values use S=MS=\mathcal{M}), the number of labeled probe items nn behind each limit, and how the limits are computed. We also now report nn as a column of Table 1 so that every limit in the main text is reproducible from the table itself. Because each gap is an item average of a per-item quantity in [0,1][0,1], we report one-sided 95%95\% upper limits from a bias-corrected bootstrap over items and, for comparability, the Clopper and Pearson limit on the hard-graded indicator, which agrees to within .001.001 at every nonzero estimate; at the two structural zeros the bootstrap limit is degenerate while the Clopper and Pearson limit reduces to the bound that nn alone implies, so those two rows differ. Table 16 lists both.

One consequence worth flagging, since the reviewer asks about it in Q1: with the corrected estimators, no branch changes except that HotpotQA's Uprop=.052U_{\mathrm{prop}} = .052 now sits just above τprop\tau_{\mathrm{prop}}, which is precisely the one benchmark on which the calibrated comparison and the diagnostic shortcut disagree. That single disagreement is what separates the two rows of Table 3 (.540.540 versus .539.539), which also resolves the reviewer's W2 bullet 4. It is recorded at the end of Appendix G.3.

W2: Lack of novelty, and the main comparison does ...

A2: On what is and is not ours. We stand by the attributions the reviewer quotes, and we would rather keep them explicit than blur them. What they leave as ours is, we believe, still the substance of the paper: the exact identity naming the separating event (Prop. 2), the if-and-only-if characterization of when the two ceilings coincide (Thm. 3), the disjointness of the two optima with two-sided constant loss (Thm. 5), the resulting impossibility for any rule measurable in the marginal law (Cor. 6), the budget-inconsistency of pool-level gaps (Prop. 7), the inapproximability of the assembly branch and the log-surrogate with a computable certificate (Thm. 8, Thm. 10), and the propagation deconvolution with its submodularity-preserving, uniformly tighter surrogate and its depth-quadratic bias bound (Prop. 18, Thm. 19, Prop. 20). On the specific relation to Zhang et al. (2026): their counterexample exhibits one pair of weak-but-jointly-sufficient proposers for a fixed summarizer, which is a demonstration that non-monotonicity exists; our Theorem 5 is a statement about the two argmax sets being disjoint with quantified loss in both directions, and Corollary 6 shows no rule in their or any other marginal-law family can be optimal for both classes. The first does not imply the second, and the second is what forces the algorithm to enumerate branches. We have added one sentence to the related-work paragraph saying this, rather than leaving the reader to infer it.

Isolating the objective from the protocol. This criticism we accept without qualification, and we have added the experiment it asks for. At the fixed budget of Table 2, the only protocol component that separates our runs from the baselines is the cross-fitted split-argmax estimator; the attenuation calibration selects a portfolio size, so it is inactive at fixed kk and enters only in Figure 1a and Table 4. We now (a) say this in Appendix G.7, replacing the sentence the reviewer quotes, and state it in one sentence in the main text, and (b) report a protocol-matched comparison in the new Appendix E.4 (Table 6): every objective is run both with the plug-in and with the cross-fitted estimator. The plug-in column reproduces Table 2, the ordering is the same in both columns, and the objective accounts for roughly 1.31.3 points at matched plug-in protocol and 2.32.3 at matched cross-fitted protocol, with the estimator contributing 0.20.2 to 1.61.6 points depending on the objective (most for ours, since our surrogate takes a maximum over members at every subgoal and is therefore the most exposed to selection-induced optimism). The two effects are close to additive, and neither alone explains the margin.

Model-first greedy. Now defined in Appendix G.7: it is the marginal-contribution greedy of Zhang et al. (2026), which starts from the anchor and adds at each step the member that most improves realized composer accuracy on the calibration split, under the same anchor, probe budget, composer-call budget and portfolio size as every other method. We also say why it is the strongest baseline (it optimizes the deployed quantity directly) and why it still loses (its composer-call budget, held equal to ours, provides only a coarse estimate of each marginal contribution at K=42K=42, and the estimate is non-monotone in the member set, so an early noisy step is not recoverable).

AuthorsOpenReview ↗

Response to Reviewer vb9E (Part 3)

(Following above Part 2)

The branch attribution. The reviewer is right that the two mechanisms were conflated. CAPS selects the branch by calibrated comparison, full stop; the diagnostic is a build-or-not instrument and, optionally, a shortcut. The Branch column of Table 1 now reports the class CAPS deploys, Table 2 reports the composer CAPS deploys, and the single disagreement between the two mechanisms (HotpotQA, where UpropU_{\mathrm{prop}} falls just above τprop\tau_{\mathrm{prop}}) is identified in Appendix G.3 as the source of the .540.540 versus .539.539 difference in Table 3. With the corrected branch, the CAPS mean over the four high-gap benchmarks is .540.540, matching the enumerate-and-compare row.

Q1: What are the exact estimators behind Table 1? Give formulas, the portfolio and the Clopper-Pearson nn; explain the nonzero MMLU-Pro and MATH-500 values; say whether any branch changes. Which baseline produces the 98%, 96% and 51% at l.460-462, and which Table 3 entries are correct?

A3: Formulas. With TT draws per observed cell, let p^miℓ\hat p_{mi\ell} be the fraction of draws of member mm on item ii that discharge subgoal ℓ\ell, c^mi\hat c_{mi} the fraction that are correct end to end, and α^miℓ\hat\alpha_{mi\ell} the intrinsic rate from the inversion of Eq. (15) or from a prefix-forced probe. Then

C^(S)=1n∑imax⁡m∈SΦc^mi,V^†(S)=1n∑i∏ℓmax⁡m∈SΦp^miℓ,V^‡(S)=1n∑i∏ℓmax⁡m∈SΦα^miℓ,\widehat C(S)=\tfrac1n\sum_i \max_{m\in S_\Phi}\hat c_{mi},\qquad \widehat V^{\dagger}(S)=\tfrac1n\sum_i\prod_\ell \max_{m\in S_\Phi}\hat p_{mi\ell},\qquad \widehat V^{\ddagger}(S)=\tfrac1n\sum_i\prod_\ell \max_{m\in S_\Phi}\hat\alpha_{mi\ell},

and Δ^=V^†−C^\widehat\Delta=\widehat V^{\dagger}-\widehat C, Δ^prop=V^‡−V^†\widehat\Delta_{\mathrm{prop}}=\widehat V^{\ddagger}-\widehat V^{\dagger}, all on the same items and the same portfolio. These are Eq. (19) of the revision.

Portfolio. The size-4 portfolio selected by the splice-capable branch on the probe split, identical across the three rungs; pool-level gaps (used only for the dilution study) use S=MS=\mathcal{M}. Stated in Appendix G.3.

Sample sizes. nn is now a column of Table 1: 10001000, 250250, 500500, 500500, 271271, 400400 for MMLU-Pro, MATH-500, HotpotQA, MuSiQue, EvalPlus and LiveCodeBench. Table 16 lists both the bootstrap and the Clopper and Pearson limits.

Why the two values were nonzero. MMLU-Pro: an inconsistent ordering of maximum and expectation between C^\widehat C and the two assembly rungs, described in A1(i); under the unified convention the estimate is exactly 00 at every item. MATH-500: Theorem 3's chain condition applies, which we now state as Corollary 4; the reported .094.094 came from the same inconsistency, and the gap is structurally zero.

Branch changes. None, with one caveat we now report: HotpotQA's UpropU_{\mathrm{prop}} moves from .059.059 to .052.052, still above τprop\tau_{\mathrm{prop}}, and HotpotQA is the single benchmark where the calibrated comparison (splice-free) and the diagnostic shortcut (splice-capable) disagree.

The baseline behind the three percentages. Greedy coverage deployed under the matching composer, whose four-benchmark mean is .4895.4895. The corrected Table 3 entry is .490.490 for the coverage row under the assembly composer; .481.481 under the selection composer is correct as printed. From .4895.4895: enumerate-and-compare 98.1%98.1\%, shortcut 96.1%96.1\%, random 51.5%51.5\%. The caption now names this baseline.

Q2: What does the splice-capable composer see on test items? It "additionally receives the per-subgoal decomposition" and re-runs "the test suite" (l.1459-1462). Are these the grading subgoals and tests? l.1432 covers only probes.

A4: They are not, and we agree that the original text left this open. We have added a paragraph to Appendix G.5 stating exactly what crosses the boundary.

On a test item the splice-capable composer receives the candidate solutions segmented by subgoal, plus a list of subgoal statements, and nothing else. The segmentation is not an annotation: for mathematics it is the proposers' own Step k: markers; for code it is module boundaries in the generated programs; for multi-hop it is a decomposition the anchor writes from the question alone, in the style of least-to-most prompting, and we have added that prompt template to Appendix G.6. Gold subgoal annotations, per-subgoal grades and the held-out grading tests are used only for grading and for probe-time competence estimation. Re-execution at composition time uses only the public example tests that ship with each problem, which are disjoint from the grading suite, so a spliced program is never assembled using the tests it is scored on, and the composer never observes a grade. We also report the size of the leak we avoided: substituting the gold decomposition for the anchor's on test items raises splice-capable accuracy by 0.60.6 points on MuSiQue and 0.40.4 on HotpotQA, and every number in the paper uses the non-leaking variant. The existing sentence that no prefix-forced probe is issued on a test item is retained.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (Part 1)

We thank the three reviewers for reviews that were unusually concrete. Between them they identified one methodological issue, three transcription or reporting errors, one overstated theorem, and four claims we had made without showing the evidence. We have fixed all of them, and three of the fixes changed numbers in the paper. We summarize what changed, where, and what it did to the conclusions.

1. Numbering map

Inserting one new corollary after Theorem 3 shifts the later statements by one, and nine new tables shift the appendix tables. Equation numbers 1 to 18 are unchanged.

SubmittedRevised
Theorem 1, Proposition 2, Theorem 3unchanged
—Corollary 4new (structural zeros)
Theorem 4, Corollary 5, Proposition 6, Theorem 7Theorem 5, Corollary 6, Proposition 7, Theorem 8
Assumption 8, Theorem 9, Proposition 10, Proposition 11Assumption 9, Theorem 10, Proposition 11, Proposition 12
Lemma 12, Proposition 13, Lemma 14, Remarks 15 to 16Lemma 13, Proposition 14, Lemma 15, Remarks 16 to 17
Proposition 17, Theorem 18, Proposition 19Proposition 18, Theorem 19, Proposition 20
Remark 20, Remark 21Remark 21, Remark 22 (new), Remark 23
Tables 1 to 5unchanged
—Tables 6 to 12new (appendix E)
Table 6 (pool), 7 (subgoals), 8 (hyperparameters)Table 13, 15, 17
—Table 14 (scoring modes), Table 16 (gap estimation)new
—Eq. (19)new (ceiling and gap estimators)

2. Corrections that changed reported numbers

A single estimator convention for the whole ladder (new Appendix G.3, Eq. (19)). The coverage ceiling had been estimated as a union over draws as well as over members, while the two assembly ceilings were item averages of a maximum over per-cell success probabilities. These are different estimands, which is why the atomic control reported a nonzero gap that the theory forbids. Every rung now takes the maximum over members inside the item average, from the same primitives, on the same items and the same portfolio. Appendix G.3 displays the estimators, names the portfolio, gives the probe counts, and explains how the limits are computed; Table 1 carries the probe count as a column and Table 16 reports both bootstrap and Clopper and Pearson limits.

A structural result that the reported gaps must respect (new Corollary 4, proof in Appendix C.5). Under Eq. (2), a query whose prerequisite graph has a path through all its subgoals forces every graded competence vector to be a prefix of ones, so the vectors form a chain and the query cannot contribute to the decomposability gap. Hence Δ\Delta is bounded above by the probability that the graph is not of that form, which is computable from benchmark metadata before any grading, and is zero for atomic and chain-structured tasks. Table 16 reports the bound beside every estimate as an identity check.

Consequences in Table 1. MMLU-Pro and MATH-500 now report exactly zero decomposability gaps, as structural rather than statistical zeros. MATH-500's assembly advantage moves entirely to the propagation term (Δ^prop=.157\widehat\Delta_{\mathrm{prop}} = .157), which makes it the clean case for propagation repair and EvalPlus the clean case for portfolio complementarity, and sharpens rather than weakens the claim that the two gaps call for different interventions.

Corrected prerequisite depths. The depth column had averaged over items with at least one prerequisite edge rather than over all items, which inflated the two multi-hop entries. Corrected to dˉ=1.0\bar d = 1.0 for HotpotQA and 1.31.3 for MuSiQue in Table 1 and Table 15, with the averaging convention stated in the caption. No other benchmark is affected.

Corrected branch ablation entry. One entry of Table 3 was mis-transcribed (.501.501 for .490.490). The recovery percentages quoted in the text were computed against greedy coverage under the matching composer and reproduce exactly from the corrected entry; the caption now names that baseline. The dependent sentence is updated: the cost of deploying a coverage-optimized portfolio under an assembly composer is 4.94.9 points, not 3.83.8.

Corrected branch attribution. CAPS selects the branch by calibrated comparison; the diagnostic is a build-or-not instrument and, optionally, a shortcut. Table 1 and Table 2 now report the class CAPS deploys, which updates the two multi-hop entries of Table 2 and brings the four-benchmark mean to .540.540. The two mechanisms agree on five of six benchmarks and differ on HotpotQA, where UpropU_{\mathrm{prop}} falls just above the threshold; this is recorded in Appendix G.3 and accounts for the difference between the enumerate-and-compare and shortcut rows of Table 3.

Corrected parameter range. 0.50.5B to 141141B, with a note in Appendix F that counts are totals rather than activated parameters.

AuthorsOpenReview ↗

General Response (Part 2)

(Following above Part 1)

3. A weakened and corrected theorem

The knapsack clause of Theorem 1 claimed (1−1/e)(1-1/e) for the density greedy that Algorithm 1 implements. A two-member instance supplied in review shows this is false. Theorem 1 now separates the three facts: cardinality greedy attains 1−1/e1-1/e; density greedy together with the best feasible singleton attains (1−1/e)/2(1-1/e)/2; partial enumeration followed by density greedy attains 1−1/e1-1/e; density greedy alone attains no constant factor. Algorithm 1 carries the singleton safeguard by default, the witness is recorded as Remark 22, and the weaker knapsack guarantee is listed among the gaps in the guarantees. Every reported experiment uses a cardinality budget, so no number changes.

4. Narrowed claims

Scope of the assembly ceilings (Section 3, measured in Appendix E.5). Eq. (5) bounds composers that take each subgoal from a member or produce it themselves; correctness arising only from the interaction of individually wrong candidates lies outside it. We measure that residual rather than assume it away: the emergent-synthesis rate is 0.7%0.7\% to 3.1%3.1\%. Table 7 also reports composer compliance with the assignment description (.87.87 to .96.96) and, more importantly for the method, the rank correlation between each objective and realized accuracy of the matching composer (.74.74 to .84.84 for our log-surrogates against .36.36 to .46.46 for coverage).

Test-time information available to the splice-capable composer (Appendix G.5). It receives candidate solutions segmented by subgoal plus a list of subgoal statements, where the segmentation comes from the proposers' own step markers, from module boundaries, or from a decomposition the anchor writes from the question alone (prompt added to Appendix G.6). Gold annotations, per-subgoal grades and the grading tests are used only for grading and probe-time estimation; re-execution uses only the public example tests. Substituting the gold decomposition would have added 0.60.6 and 0.40.4 points on the two multi-hop sets, and no reported number uses it.

5. New experiments

AdditionWhereWhat it shows
Protocol-matched comparisonApp. E.4, Table 6Every objective run with and without the cross-fitted estimator. The ordering is unchanged; the objective is worth 1.31.3 to 2.32.3 points at matched protocol and the estimator 0.20.2 to 1.61.6, so neither alone explains the margin.
Oracle validity and complianceApp. E.5, Table 7Emergent-synthesis rate, composer compliance, and objective-to-accuracy rank correlation.
Thirteen composer instantiationsApp. E.6, Table 8Selected portfolio stable within a class (mean Jaccard .86.86) and different across classes (.38.38); our portfolio leads under all thirteen.
Public routing corporaApp. E.7, Table 9Selection branch at K=11K = 11, 5252 and 8,5678{,}567.
Larger pool and cost modelApp. E.7, Table 10End-to-end assembly branch at K=126K = 126 (accuracy improves), plus the O((K+nL)r)O((K+nL)r) cost model and an extrapolation to K=1,000K = 1{,}000.
Prerequisite misspecificationApp. E.8, Table 11Gradual degradation under 1010 to 30%30\% edge perturbation; an anchor-inferred graph costs 0.60.6 points; without any graph the splice-free branch still runs.
Local independence diagnosticsApp. E.9, Table 12Residual correlation by rank, within and across families, plus controlled violations and their effect on the gap and on accuracy.
Two scoring modesApp. G.1, Table 14Generation versus label-logit scoring on MMLU-Pro; no ordering changes.
Gap estimation detailApp. G.3, Table 16Probe counts, structural bounds, bootstrap and Clopper and Pearson limits.
Cheapest-model baselineTable 2Row added for all six benchmarks.
Specialist shareTable 4Column added, making the 0.46→0.180.46 \to 0.18 claim readable off the table.
Model-first greedy definitionApp. G.7Identified as the marginal-contribution greedy of the closest prior work, with its budget and why it still loses.

6. What did not change

CAPS is best on all six benchmarks, and the margin still tracks the decomposability gap. The branch-enumeration result, the dilution result, the component ablation, the cost and probe-efficiency frontiers, and the generalization and stability results are unchanged. The gap corrections to the MMLU-Pro and MATH-500 rows change the composer deployed on neither benchmark, and the one knife-edge case, HotpotQA, is now reported rather than smoothed over.

Main-text length is held to the revised limit by moving the detailed discussion of the new studies into the appendix sections that report them, with short pointers from the experiments section. All changes in the manuscript are highlighted in blue.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.