- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposed an expert-state reusing method, Factorized Counterfactual Training (FCT), for training-plan search in multi-expert adaptation. FCT exactly reuses expert-state (parameters and optimizer state) when exploring different assignments with the same ordered training history of that expert. FCT reduces repetitive computation and speeds up the search process, making it possible to compare more training plans within a fixed search budget.
Critical strengths
- The proposed method is simple and effective for multi-expert adaptation. FCT substantially outperforms whole-prefix reuse baseline in the specific settings studied in the paper, in which experts are updated independently and the target is to search over training plans.
Critical weaknesses
-
The significance and novelty are unclear. FCT is designed for training-plan search settings which require experts are updated independently. This narrow scenario leaves its broader significance unclear. The main idea of FCT appears straightforward, raising concerns about its novelty.
-
The presentation needs to be polished. It would be better to introduce concrete task scenarios before presenting the proposed method.
Questions for authors
-
What are the practical applications of training-plan search? Could the authors provide more concrete examples of real-world scenarios in which such a search is necessary?
-
What is the key technical novelty of FCT beyond caching and reusing identical expert training histories?
-
How well does FCT generalize to more realistic training settings where experts are not updated independently or the shared backbone is also updated?
-
How does the efficiency improvement of FCT scale with the number of experts and the length of training plans? Does the additional memory cost of caching expert states becomes a bottleneck?
Thank you for the review. New material is blue in the revised PDF.
Presentation: "introduce concrete task scenarios before presenting the method."
Done. The Introduction now opens with three concrete operations on a sharded adapter bank (assignment, deletion, valuation), then states the cost, the observation, and the contributions. Two figures and two tables moved to the appendix to make room.
Q1. Practical applications of training-plan search.
A bank of isolated modules with an aggregated readout is the architecture of exact machine unlearning (SISA), and with adapters on a frozen backbone (SAFE, S3T) it is our setting. Three operations on such a bank compare training plans. First, which shard receives which data changes accuracy and retraining cost (SISA Sec. VIII-B, ARCANE, GraphEraser); this is the search in our SciQ and ARC experiments. Second, each unlearning request needs the bank trained without some events. Third, Data Shapley, Data Banzhaf and datamodels need models trained on many subsets.
For 32 events on 4 shards, 16 deletion requests need 188 updates with whole-prefix reuse and 42 with FCT, the same as SISA's own checkpoints, and 10,000 sampled subsets need 99,337 and 1,020 (Table 3). On Qwen3-0.6B the stored states equal direct retraining in every bit (156 of 156 tensors). On one A100, 10,000 sampled subsets of 32 events took 9.5 minutes with FCT and 6.2 hours with whole-prefix reuse, with identical results (General Response 2/2).
Q2. Key technical novelty beyond caching identical histories.
The rule is memoization, and the Introduction now says so. We claim two things beyond it. First, we store a state under its expert's own history, while prior training reuse follows a global prefix (Hippo's stage tree, PBT) or one trained history (SISA). This allows reuse after two plans diverge and turns (K+1)^T-1 updates into K(2^T-1). Second, we state when reuse is exact. A stored state equals a retrained one only if the update is computed the same way. Table 63 shows that on Apple MPS the same update is bitwise identical in another slot of a four-slot batch but changes in its last bits in a batch of one or eight slots, and that on CPU Qwen2.5 changes even in its own slot. The native LoRAFusion path changes the selected plan in 3 of 3 runs. FCT keeps optimizer state, treats the batch arrangement as part of a state's identity, and runs batches of one fixed size.
Q3. Experts that are not updated independently, or an updated shared backbone.
Appendix O extends the result (Proposition 2). Divide the experts into groups so that every update reads and writes only its own group. The state to reuse is then the state of a group, identified by the group's ordered history. Independent experts are groups of one, which is FCT. A shared component updated between blocks, such as a backbone or router refreshed when the bank is committed, is already covered, because every commitment starts a fresh set of histories; all 24 public-task runs commit four successive banks this way. Partly coupled experts keep part of the saving. With 8 experts coupled in pairs, 38-46% of whole-prefix updates are still avoided, and in groups of four, 14-22% (Table 62). A backbone updated at every event, or a soft mixture that sends gradient to every expert, makes all experts one group. FCT then becomes whole-prefix reuse, no better and no worse.
Q4. Scaling with the number of experts and plan length; is memory a bottleneck?
The share of whole-prefix updates avoided grows with the number of experts (35-42% at 4, 45-60% at 8, 57-67% at 16) and with beam width (about 20% at width 2, 62-70% at width 16), and falls slowly with block length (35-42% at 8 events, 26-40% at 32). Real searches on Qwen3-0.6B fall inside these ranges (Table 58). For deletion and valuation requests the saving grows with the number of requests. On one A100 the time ratio rose from 14.1-14.8x at 1,000 sampled subsets to 107x at 10,000, and 8 shards with 64 events gave 16.1x at 3,000 subsets.
Memory was not a bottleneck in our workloads. For the paper's searches FCT stores less, 119 MB against 1,198 MB for 180 plans on the A100 (Table 60), and a beam of width w over K experts needs at most 2wK states without recomputation. For valuation requests served depth first FCT stores more. Whole-prefix reuse held at most 249 states, while FCT keeps every state it trains, 1,020 states of 2.29 MB each (rank 2), or 2.3 GB, for 4 shards of 8 events. The bound does not grow with the number of requests but doubles with each further event per shard; a state grows linearly with rank.
We also withdrew one claim. Compared at equal compute, assignment search gives no clear advantage over tuned training without search, so the paper no longer claims an accuracy gain (General Response 1/2).
We hope these answers and the new presentation address your concerns, and we would be grateful if you would reconsider your rating.