- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper examines the problem of selecting a portfolio of models in Mixture-of-Models systems and claims that the optimal set of models depends strongly on the way in which the chosen models are combined. It makes a distinction between selection-type composers, which simply return the result of one of the selected models, and assembly-type composers, which are able to produce an answer by combining partial evidence from several models. Using a decomposition of task difficulty into subgoals, the authors introduce separate value functions for the two different cases and describe the difference between their upper bounds in terms of the probability of distributed solvability: instances in which the entire portfolio manages to solve all the required subgoals even though no single model is able to solve the complete query. The paper also demonstrates that selection objectives are monotone submodular, while the assembly objective may show increasing returns and is not polynomial-time constant-factor approximable unless . In order to deal with this, the authors suggest a logarithmic submodular surrogate and also derive a propagation correction that allows the intrinsic subgoal competence to be recovered from graded competence that has been affected by failures at upstream stages.
Based on these theoretical findings, the paper introduces CAPS, a portfolio selection framework that is aware of composition. CAPS combines sparse subgoal probing with low-rank tensor completion, incorporates conditional competence prediction, carries out the separate optimization of the various composition branches, includes held-out branch calibration, and uses a cross-fitted estimator in order to reduce the bias resulting from the maxima of the estimated model scores. The empirical evaluation involves six benchmarks-MMLU-Pro, MATH-500, HotpotQA, MuSiQue, EvalPlus, and LiveCodeBench-together with a live pool of 42 models drawn from nine families, as well as several routing and composition baselines. The experiments show that CAPS achieves the highest realized accuracy on all six benchmarks, with greater improvements observed on those benchmarks where the estimated decomposability gap is larger.
While I acknowledge the merit in distinguishing between selection and assembly, I am unconvinced that the current experiments sufficiently demonstrate the breadth and scalability implied by the paper's approach. Specifically, the fully composition-aware pipeline is evaluated only on a pool of 42 models, whereas experiments with thousands of candidate models rely on precomputed correctness matrices and focus exclusively on selection due to the absence of subgoal annotations.
Critical strengths
- The paper points out a significant conceptual gap in the current approaches to model portfolio selection.
A key strength of the paper is the argument that portfolio selection cannot generally be separated from the composition operator which consumes the portfolio. The distributed-solvability identity provides a clear account of the cases in which assembly can benefit from models that would seem useless when assessed using standard end-to-end correctness metrics. Moreover, the separation result demonstrates that selection-optimal and assembly-optimal portfolios can be entirely different, showing that the problem goes beyond a minor adjustment to existing coverage-based methods.
This is important since a large portion of the current literature on model selection assesses the candidates according to individual accuracy, diversity, or complementary end-to-end coverage. The paper gives a sound explanation for why such criteria can systematically exclude specialists who prove to be useful only when partial solutions are combined.
- The theoretical treatment goes beyond a single algorithmic contribution.
The paper presents a number of related findings: the submodularity of the selection branch, the difficulty of the assembly branch, a logarithmic surrogate, a decomposition of the assembly gains into distributed solvability and propagation repair, and a correction for upstream-error masking. This endows the work with greater conceptual depth than a purely empirical approach to model selection. The fact that propagated grading is able to suppress late-stage specialists is especially interesting and could be of use outside of the particular CAPS implementation.
Critical weaknesses
- The empirical validation is too limited compared with the extent of the claims made in the paper.
Even though the paper presents its results on six benchmarks and involves several ablation studies, the complete composition-aware CAPS pipeline is only assessed on a live pool consisting of 42 models. The experiments with thousands of models make use of the RouterBench, LLMRouterBench, and RouterEval correctness matrices, which fail to provide subgoal labels and therefore only test the selection aspect of the framework.
It is important to make this distinction since one of the main aims of the paper is to carry out portfolio selection from very large model pools, whereas the most innovative elements-assembly-aware selection, intrinsic-competence recovery, and propagation-aware portfolio selection-are not subjected to end-to-end testing at that scale. The experiments carried out so far therefore show that CAPS functions in a moderately sized and structured environment, but have not yet proven that the complete approach can be scaled up to the wider deployment scenario stressed in the paper.
The fact that it is computationally expensive also means that this question is nontrivial rather than just one that has been omitted experimentally. The paper states that at a probe rate of , it takes approximately 1.9K A100 GPU-hours to construct the graded tensor for the 42-model live pool, together with an extra cost from the prefix-forced probes. Although the downstream optimization is relatively cheap, it is exactly the probing stage that would have to scale as the candidate pool increases.
- The experimental study of the mechanisms of composition is limited.
The theory is based on three broad categories of selection-type, splice-free assembly, and splice-capable assembly operators, but the empirical study only includes one specific composer for each category-a pre-answer selector, a synthesis-based assembly composer, and a splice-capable composer which explicitly chooses subgoal sources and then re-executes the downstream computation.
Consequently, it is hard to tell if the portfolio differences observed are reliable when applied across the wider range of systems covered by the theoretical definitions. For instance, the evaluation does not show that the same conclusions apply in the case of alternative summarizers, verifier-guided synthesis, multi-round debate, iterative refinement, different prompts, or agent architectures that have dynamically generated subgoals.
This distinction is important because the practical relationship between portfolio composition and realized accuracy can depend greatly on the composition method. Although the theory distinguishes the classes in principle, the current experiments provide only limited evidence that CAPS generalizes robustly across different implementations. The method depends on subgoal-level competence, a prerequisite structure, and, in some cases, prefix-forced probing. While these requirements are natural for unit-test-based code benchmarks and can be implemented for step-graded mathematics and annotated multi-hop QA, they are much less straightforward for open-ended generation tasks, where subgoal decomposition may be ambiguous and correctness may not be conjunctive.
The paper makes it clear that its framework does not define the correct combination of rubric items for open-ended generation and that prefix-forced probing has no obvious equivalent in that case.
Although the benchmarks selected are suitable for verifying the theoretical mechanism under controlled conditions, they also greatly favour the assumptions needed by the proposed method. It would be important to carry out further experiments using noisy, learned, or imperfect subgoal decompositions in order to show robustness outside of these structured situations.
The robustness analysis does not place adequate stress on the key structural assumptions.
The paper includes some useful generalisation and stability experiments, such as a model split in which the data sets are disjoint across families and a shifted task distribution. CAPS is still stronger than the baseline methods in both cases, though its performance decreases when there is a distribution shift.
Yet these experiments do not directly examine the assumptions which are most particular to the method-namely, the local independence of intrinsic competence, the correctness of the prerequisite DAG, the reliability of subgoal grading, and the accuracy of the deconvolution procedure. Because breaches of these assumptions can have a direct effect on the inferred complementarity between the models, I think this is an important aspect that has so far been lacking empirically.
Questions for authors
-
Clarification / possibly altering the assessment The end-to-end assembly-aware experiments have been carried out on 42 live models, while the studies involving thousands of candidate models involve only model selection. Could the authors provide end-to-end assembly-aware experiments on a much larger set of models, or at least a realistic cost extrapolation that shows how the probing and subgoal-grading stages scale with ? Since the present experiment with 42 models already demands a significant amount of GPU time, it appears to be important for assessing the practical scalability claim.
-
A clarification / one which could change the assessment
How dependent are the empirical conclusions on the specific composer implementations employed in the paper? Although the theoretical distinction refers to broad categories of selection and assembly operators, only one particular composer is actually used for each category. The paper would be greatly strengthened if it showed that similar portfolio rankings or gains are obtained with multiple summarizers, different prompts, verifier-based composers, or with multi-round composition schemes. -
A clarification / one that could change the assessment
Can the authors assess the method's robustness when the prerequisite structure is imperfect? For instance, what would occur if edges in the prerequisite DAG were randomly deleted or inserted, or if the DAG was automatically inferred rather than being obtained from the benchmark metadata? Because the propagation correction relies directly on this structure, such an experiment would help in measuring the sensitivity of the method to incorrect specification of the structure. -
A clarification / one that could change the assessment Can the authors give empirical evidence in support of the local-independence assumption which underlies the propagation correction? More specifically, once the learned latent difficulty representation has been taken into account, what level of residual correlation remains between the intrinsic competence variables across the models, particularly among those models which belong to the same family? A sensitivity analysis involving controlled violations of this assumption would help to determine whether the recovered intrinsic competence is reliable.
4 comments
Response to Reviewer sYQT (Part 2)
(Following above Part 1)
Q1: Clarification / possibly altering the assessment. The end-to-end assembly-aware experiments have been carried out on 42 live models, while the studies involving thousands of candidate models involve only model selection. Could the authors provide end-to-end assembly-aware experiments on a much larger set of ...
A3: Both, as described in A1 and reported in Appendix E.7: a -member end-to-end assembly-aware run (Table 10), and an explicit cost model with the extrapolation to . The key quantitative point is that the probe requirement for fixed completion error is rather than , so per-model probe cost falls as the pool grows and total cost is close to flat in in the regime where : about thousand A100 hours at against thousand at . We also give the marginal cost of adding one checkpoint, which is the number a deployment actually faces. We make no claim beyond what the measurement and the scaling support.
Q2: A clarification / one which could change the assessment. How dependent are the empirical conclusions on the specific composer implementations employed in the paper?
A4: New Appendix E.6 (Table 8) reports all thirteen instantiations listed in A2, on MuSiQue and LiveCodeBench, each with the portfolio our method selects, the portfolio greedy coverage selects, and the Jaccard overlap against the reference composer of the same class. Three findings:
- Within a class the portfolio is stable. Mean pairwise Jaccard within a class. Changing the summarizer, the prompt, or moving from a single summarizer to a two-layer stack, to debate-then-synthesis or to verifier-guided synthesis moves realized accuracy by at most points and the portfolio by at most one member.
- Across classes it changes. Mean Jaccard between classes, which is the dichotomy showing up where the theory says it should.
- The ordering is invariant. Our portfolio beats the coverage portfolio under every instantiation, by to points, including under verifier-guided synthesis and iterative refinement, which are the two variants furthest from the composer we tuned on.
Q3: A clarification / one that could change the assessment. Can the authors assess the method's robustness when the prerequisite structure is imperfect?
A5: New Appendix E.8 (Table 11) reports exactly the experiment described: random perturbation of the prerequisite graph at rates , and (each perturbation deletes or inserts an edge, subject to acyclicity), an anchor-inferred graph built from the query alone, and a no-graph condition in which only the splice-free branch is available. On MATH-500 and MuSiQue, accuracy falls by , and points at the three perturbation rates and the selected specialist share falls from to , which is the mechanism Proposition 20 predicts: a corrupted ancestor set mis-scales exactly the members whose value sits at deep subgoals. The inferred graph costs points, so most of the benefit survives without annotations, and the prompt that produces it is now in Appendix G.6. With no graph at all, the method falls back to the splice-free branch, where it matches the strongest baseline on MATH-500 and still leads on MuSiQue. The limitations section now points at this curve where it says that a misspecified reachability matrix enters the inversion of Eq. (15) directly.
Q4: A clarification / one that could change the assessment. Can the authors give empirical evidence in support of the local-independence assumption which underlies the propagation correction?
A6: New Appendix E.9 (Table 12) reports both halves of what the reviewer asks for.
Residual correlation. After removing the fitted item factors, the mean absolute residual correlation of intrinsic competence between model pairs is , and at completion ranks , and . As the reviewer anticipates, it is concentrated within families: , and at the same ranks, against to across families. So the assumption is approximately right at the rank we use and is violated mostly between siblings, which is also the regime in which the deconvolution would overstate complementarity.
Controlled violation. We inject a shared misconception into a randomly chosen family at a fraction of subgoals. At the decomposability gap is inflated by , and and realized accuracy falls by , and points. The direction is the one to worry about, since inflating makes the assembly branch look better than it is, so we now recommend reporting the within-family residual correlation alongside the gaps and treating a large value as a reason to increase the rank before trusting the diagnostic. We also note there that the realized comparison does not depend on the assumption.
Follow-up on Rebuttal and Revision
Dear Reviewer sYQT,
Thank you again for your constructive suggestions. We would appreciate your thoughts on whether the new experiments with 13 composer instantiations, the expanded 126-member pool and cost analysis, and the robustness tests for imperfect prerequisite graphs and dependence violations have addressed your concerns about generality and scalability. Please let us know if any concern remains or further clarification would help. If the added evidence has resolved your concerns, would you consider raising your rating?
Best regards,
The Authors
Dear Reviewer sYQT,
With one day remaining, we wanted to ask whether the new experiments address your questions about generality and scalability. Table 8 compares our selected portfolio with the coverage portfolio across 13 composer instantiations, and Table 10 reports an end-to-end evaluation with 126 members. If you feel that an important setting is still missing, we would welcome your feedback before the discussion closes. If the added evidence resolves your concerns, would you consider raising your rating?
Best regards,
The Authors
Response to Reviewer sYQT (Part 1)
W1: The empirical validation is too limited compared with the extent of the claims made in the paper. ...
A1: This is a fair description of the gap, and we have narrowed it in three ways rather than argue it away. All three are collected in the new Appendix E.7, with a pointer from the experiments section.
First, we now report the selection-branch results on the public corpora explicitly instead of only alluding to them: new Table 9 gives RouterBench (K=11), LLMRouterBench (K=52) and RouterEval (K=8,567) at k=4, with the probe rate used in each. This does not test the assembly machinery, as the reviewer says, and we label it as such.
Second, we have run an end-to-end assembly-aware experiment at a larger pool. We expanded the live pool to K=126 by pairing each checkpoint with three decoding and prompt configurations, which are genuinely distinct members from the portfolio's point of view (they differ in subgoal competence and in cost) and whose grading reuses the same harness; Appendix F documents the construction and notes that it adds no new weights. On MuSiQue and LiveCodeBench, CAPS improves from .465 to .478 and from .461 to .474 at k=4 when the pool grows, while the probe cost grows only from 1.9 to 2.2 thousand A100 hours (new Table 10).
Third, and most to the point of (3), we now give the cost model instead of leaving the extrapolation to the reader. The probe budget needed for a fixed completion error is O((K+nL)r) cells, not O(KnL), so the required rate falls roughly as 1/K and the total cost is dominated by the item side once K≪nL. Holding completion error fixed and extrapolating from our measured per-cell cost, a 1,000-member pool costs about 2.2 thousand A100 hours, within about 15% of the 42-member run; the independently measured ρ=0.008 needed at K=8,567 on RouterEval is consistent with the same scaling. The marginal cost of placing one newly released checkpoint is on the order of rT graded generations, which is what makes the model-level embedding of Eq. (11) worth having. Appendix E.7 states explicitly that the extrapolation holds completion error and the benchmark suite fixed, and says nothing about how many candidates are worth grading in the first place.
W2: The experimental study of the mechanisms of composition is limited. [...] only one specific composer for each category [...] it is hard to tell if the portfolio differences observed are reliable when applied across the wider range of systems covered by the theoretical definitions [...] It would be important to carry out further experiments using noisy, learned, or imperfect subgoal decompositions in order to show robustness outside of these structured situations. [...] these experiments do not directly examine the assumptions which are most particular to the method.
A2: We have added the three studies this asks for, as Appendices E.6, E.8 and E.9; the details are in A4, A5 and A6 below, and we summarize the design here.
On composer mechanisms, the theory is a statement about classes, so the right test is whether the portfolio is stable within a class and changes across classes. We now instantiate thirteen composers spanning the three classes: router head, confidence cascade and judge picker for selection; two different summarizers, a two-layer Mixture-of-Agents, debate followed by synthesis, verifier-guided synthesis and an alternative prompt for splice-free assembly; and subgoal splicing, iterative per-subgoal repair and verifier-guided splicing for splice-capable assembly, with self-consistency over repeated draws of one member as a selection-type control. The selected portfolio has mean pairwise Jaccard .86 within a class and .38 across classes, and our portfolio beats the coverage portfolio under every one of the thirteen (Table 8).
On imperfect structure, we perturb the prerequisite graph by random edge deletion and insertion at rates up to 30%, and we also replace the annotated graph with one the anchor infers from the query alone. Degradation is gradual, not sharp (Table 11).
On the assumptions specific to the method, we now measure the residual dependence that Assumption 9 forbids, as a function of the completion rank and separately within and across model families, and we inject controlled violations and report their effect on both Δ and realized accuracy (Table 12).
We have also added one sentence to the extended limitations: because the splice-free branch needs no prerequisite structure, a deployment without usable decompositions still has a two-branch version of the method available, which is the honest statement of how far the framework reaches into open-ended generation.