acceptodds
Under review as a conference paper at ICLR 2027

Exact Expert-State Reuse for Efficient Training-Plan Search

Abstract

In multi-expert adaptation, comparing assignments of training examples to experts is costly because each candidate requires a sequence of parameter updates. Different assignments can give an expert the same ordered training history, even after their global plans diverge. Factorized Counterfactual Training (FCT) exploits this repetition by computing each required expert update once and retaining both parameters and optimizer state. With independent expert updates and a fixed numerical program, it preserves candidate states, scores, and deterministic search decisions for the same requests. A controlled four-expert workload reduces updates from 3,124 to 124, with a warm speedup over whole-prefix reuse. On SciQ and ARC-Challenge across four language models, FCT avoids 36.8–46.9% of prefix-cached updates and preserves every test prediction. All 24 paired runs are faster, with median speedups of to including graph preparation and full testing on a resident model. A separate 120-second search on Qwen3-0.6B more than doubles mean candidate counts and improves mean test accuracy by 0.93 and 0.20 percentage points, respectively. These savings make it possible to compare more training plans within a fixed search budget.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Training Systems, Computation Reuse, Multi-Expert Learning, Low-Rank Adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_p4uWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposed an expert-state reusing method, Factorized Counterfactual Training (FCT), for training-plan search in multi-expert adaptation. FCT exactly reuses expert-state (parameters and optimizer state) when exploring different assignments with the same ordered training history of that expert. FCT reduces repetitive computation and speeds up the search process, making it possible to compare more training plans within a fixed search budget.

Critical strengths
  1. The proposed method is simple and effective for multi-expert adaptation. FCT substantially outperforms whole-prefix reuse baseline in the specific settings studied in the paper, in which experts are updated independently and the target is to search over training plans.
Critical weaknesses
  1. The significance and novelty are unclear. FCT is designed for training-plan search settings which require experts are updated independently. This narrow scenario leaves its broader significance unclear. The main idea of FCT appears straightforward, raising concerns about its novelty.

  2. The presentation needs to be polished. It would be better to introduce concrete task scenarios before presenting the proposed method.

Questions for authors
  1. What are the practical applications of training-plan search? Could the authors provide more concrete examples of real-world scenarios in which such a search is necessary?

  2. What is the key technical novelty of FCT beyond caching and reusing identical expert training histories?

  3. How well does FCT generalize to more realistic training settings where experts are not updated independently or the shared backbone is also updated?

  4. How does the efficiency improvement of FCT scale with the number of experts and the length of training plans? Does the additional memory cost of caching expert states becomes a bottleneck?

1 comment

Thank you for the review. New material is blue in the revised PDF.

Presentation: "introduce concrete task scenarios before presenting the method."

Done. The Introduction now opens with three concrete operations on a sharded adapter bank (assignment, deletion, valuation), then states the cost, the observation, and the contributions. Two figures and two tables moved to the appendix to make room.

Q1. Practical applications of training-plan search.

A bank of isolated modules with an aggregated readout is the architecture of exact machine unlearning (SISA), and with adapters on a frozen backbone (SAFE, S3T) it is our setting. Three operations on such a bank compare training plans. First, which shard receives which data changes accuracy and retraining cost (SISA Sec. VIII-B, ARCANE, GraphEraser); this is the search in our SciQ and ARC experiments. Second, each unlearning request needs the bank trained without some events. Third, Data Shapley, Data Banzhaf and datamodels need models trained on many subsets.

For 32 events on 4 shards, 16 deletion requests need 188 updates with whole-prefix reuse and 42 with FCT, the same as SISA's own checkpoints, and 10,000 sampled subsets need 99,337 and 1,020 (Table 3). On Qwen3-0.6B the stored states equal direct retraining in every bit (156 of 156 tensors). On one A100, 10,000 sampled subsets of 32 events took 9.5 minutes with FCT and 6.2 hours with whole-prefix reuse, with identical results (General Response 2/2).

Q2. Key technical novelty beyond caching identical histories.

The rule is memoization, and the Introduction now says so. We claim two things beyond it. First, we store a state under its expert's own history, while prior training reuse follows a global prefix (Hippo's stage tree, PBT) or one trained history (SISA). This allows reuse after two plans diverge and turns (K+1)^T-1 updates into K(2^T-1). Second, we state when reuse is exact. A stored state equals a retrained one only if the update is computed the same way. Table 63 shows that on Apple MPS the same update is bitwise identical in another slot of a four-slot batch but changes in its last bits in a batch of one or eight slots, and that on CPU Qwen2.5 changes even in its own slot. The native LoRAFusion path changes the selected plan in 3 of 3 runs. FCT keeps optimizer state, treats the batch arrangement as part of a state's identity, and runs batches of one fixed size.

Q3. Experts that are not updated independently, or an updated shared backbone.

Appendix O extends the result (Proposition 2). Divide the experts into groups so that every update reads and writes only its own group. The state to reuse is then the state of a group, identified by the group's ordered history. Independent experts are groups of one, which is FCT. A shared component updated between blocks, such as a backbone or router refreshed when the bank is committed, is already covered, because every commitment starts a fresh set of histories; all 24 public-task runs commit four successive banks this way. Partly coupled experts keep part of the saving. With 8 experts coupled in pairs, 38-46% of whole-prefix updates are still avoided, and in groups of four, 14-22% (Table 62). A backbone updated at every event, or a soft mixture that sends gradient to every expert, makes all experts one group. FCT then becomes whole-prefix reuse, no better and no worse.

Q4. Scaling with the number of experts and plan length; is memory a bottleneck?

The share of whole-prefix updates avoided grows with the number of experts (35-42% at 4, 45-60% at 8, 57-67% at 16) and with beam width (about 20% at width 2, 62-70% at width 16), and falls slowly with block length (35-42% at 8 events, 26-40% at 32). Real searches on Qwen3-0.6B fall inside these ranges (Table 58). For deletion and valuation requests the saving grows with the number of requests. On one A100 the time ratio rose from 14.1-14.8x at 1,000 sampled subsets to 107x at 10,000, and 8 shards with 64 events gave 16.1x at 3,000 subsets.

Memory was not a bottleneck in our workloads. For the paper's searches FCT stores less, 119 MB against 1,198 MB for 180 plans on the A100 (Table 60), and a beam of width w over K experts needs at most 2wK states without recomputation. For valuation requests served depth first FCT stores more. Whole-prefix reuse held at most 249 states, while FCT keeps every state it trains, 1,020 states of 2.29 MB each (rank 2), or 2.3 GB, for 4 shards of 8 events. The bound does not grow with the number of requests but doubles with each further event per shard; a state grows linearly with rank.

We also withdrew one claim. Compared at equal compute, assignment search gives no clear advantage over tuned training without search, so the paper no longer claims an accuracy gain (General Response 1/2).

We hope these answers and the new presentation address your concerns, and we would be grateful if you would reconsider your rating.

Reviewer_Ej8WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Factorized Counterfactual Training (FCT), which accelerates multi-expert training-plan search by reusing complete expert states associated with identical local training histories. Under independent updates and fixed numerical execution, FCT preserves candidate outcomes and deterministic search decisions. The paper provides correctness arguments, update-count analysis, and a batched implementation. Public-task experiments report median speedups of 1.40–1.63× over whole-prefix caching with identical predictions. Equal-time search evaluates more candidates, although the resulting accuracy gains remain uncertain.

Critical strengths
  • Clear correctness conditions. The paper carefully identifies the dependencies required for exact reuse, including optimizer state, randomness, and numerical execution, and extends its argument to deterministic adaptive search.
  • Well-designed execution comparisons. Batched replay, whole-prefix caching, fixed-quota workloads, and single-plan controls help isolate the benefits of local-history reuse from batching and differences in training exposure.
  • Consistent efficiency gains and transparent reporting. All 24 paired public-task runs favor FCT. The paper distinguishes timing scopes and acknowledges cross-backend numerical differences and uncertainty in accuracy improvements.
Critical weaknesses
  • The necessity of training-plan search is insufficiently motivated. When adapters correspond to predefined domains or tasks, data assignments may already be known. The experiments instead search data partitions for a uniformly averaged adapter ensemble. The paper should better establish when this workflow is practically valuable.
  • Novelty beyond complete-state memoization needs clarification. The central observation follows from independent, deterministic updates. While the implementation addresses meaningful challenges, the paper could more clearly distinguish its nontrivial technical contributions from a direct local-history cache.
  • Executor speedups do not establish the competitiveness of the overall learning workflow. Round-robin comparisons match committed updates rather than total compute, while equal-time experiments compare executors for the same search procedure. These results do not show whether searching is preferable to spending the budget on direct training or other adaptation strategies.
  • Limited evidence for broader applicability and quality gains. Experiments use relatively small models, rank-2 adapters, limited adaptation data, and two related tasks. Equal-time accuracy improvements rely on three seeds and have confidence intervals containing zero. Cache scalability and performance under longer training horizons also remain insufficiently characterized.
Questions for authors
  • What practical application naturally requires searching over expert assignments, rather than using predefined domains, random partitions, or clustering? How much reuse occurs in that application?
  • Under equal total compute, how does FCT-based search compare with a single adapter and strong multi-adapter baselines without assignment search, accounting for parameter count and inference cost?
  • Which benefits come from local-state reuse, fixed numerical blocks, validation caching, and scheduling? Could targeted ablations clarify their respective contributions?
  • How do memory usage, transfer overhead, and speedups scale with training horizon, expert count, and LoRA rank, particularly when cache eviction becomes necessary?
  • Do equal-time quality gains persist across more seeds, diverse tasks, and search budgets?
3 comments

Thank you for the detailed review and the five questions. Each led to a new experiment, and one of them made us withdraw a claim. New material is blue in the revised PDF.

Q2 / W3. Equal total compute against a single adapter and multi-adapter baselines without search.

We ran this comparison on Qwen3-0.6B with new seeds (Appendix Q, Table 65). Every method sees the same 288 training and 64 validation examples. Selected baselines choose the learning rate from {0.02, 0.01, 0.005}, and where stated the number of passes (up to 10), on those validation examples. Test labels select nothing. Updates and validation evaluations (SciQ means) count everything a method spends, including discarded candidates and learning rates. Accuracy is the mean over seeds on the full test set, with the range in brackets.

MethodAdapters per queryUpdatesVal. evalsSciQ, 3 seeds (%)ARC-Challenge, 3 seeds (%)
Round-robin, one pass436473.8 (69.4-78.5)55.8 (53.6-56.9)
Random balanced partition, one pass436464.7 (40.9-80.2)56.4 (56.0-56.9)
k-means partition, averaged readout436074.1 (70.8-79.2)55.2 (54.3-57.0)
Round-robin, rate selected41081278.2 (77.1-78.9)56.8 (56.7-56.9)
Single rank-8 adapter, rate and passes selected15041479.9 (79.4-80.2)54.3 (45.7-59.5)
Round-robin bank, rate and passes selected43964479.2 (77.9-80.3)57.8 (56.7-58.4)
Assignment search, FCT418920179.0 (78.2-79.9)57.3 (55.7-59.1)
Same search, whole-prefix reuse4326338identicalidentical

All rows have 1.15M trainable parameters (one rank-8 adapter or four rank-2 experts).

We read three things from the table. First, search is not better than training without search. On both tasks it is within about one point of the best baseline that does no search, never leads by a clear margin, and costs more than selecting a learning rate (row 4). Second, the assignment matters only when the learning rate is too large. At the paper's rate of 0.02, round-robin, a random balanced partition and k-means vary by many points from seed to seed on SciQ, while the search stays within 78.2-79.9%. At 0.01 the same seeds give 78.6% for round-robin, 78.4% for the random partition and 78.4% for the search (Table 67). The search guards against a poor learning rate, which selecting the rate on validation does more cheaply, and FCT avoids the same share of updates at both rates. Third, the benchmark saturates early. Qwen3-0.6B answers 60.1% of SciQ without adaptation and 76.3-78.1% after a single update per expert. The 32 further events add about one point at learning rates 0.01 and 0.005 and lower accuracy on average at 0.02 (Table 66).

We therefore changed the paper. The abstract, Sec. 4.5 and the conclusion no longer claim an accuracy gain; they claim that the same budget evaluates more candidates. The Introduction states that FCT changes the cost of evaluating requested plans, not which plans are worth requesting, and Sec. 4.7 reports this comparison.

Q5 / W4. Do the equal-time gains persist across seeds, tasks and budgets?

Not as a general result, and we no longer claim them. In the submission they were +0.93 and +0.20 points over three seeds, with intervals that include zero. We repeated the equal-budget search on four tasks (adding OpenBookQA and CommonsenseQA), 3 new seeds each, with budgets projected from exact counts and measured unit costs (Appendix Q.1). FCT evaluated 2.0x as many candidates, and its selected plan scored +0.72 points higher on average over 12 task-seed pairs (5 gains, 6 ties, 1 loss). The gain comes from CommonsenseQA (+1.8 points, all 3 seeds), where the search keeps finding better plans; on the other three tasks it averages +0.36. What holds everywhere is the cost result. Both executors follow the same request sequence, so within one budget FCT completes more requests and its best validation objective is never lower.

W2. Novelty beyond complete-state memoization. We agree that the observation follows from independent deterministic updates, and the Introduction now says so. We claim three things beyond it, namely what a state is stored under (prior training reuse follows a global prefix, as in Hippo's stage tree, or one trained history, as in SISA), the conditions under which a stored state is exactly the retrained one on batched hardware, and identical decisions with bounded memory.

Our answers to Q1, Q3 and Q4 follow in part 2.

Q1 / W1. Which application needs to compare training plans, and how much reuse occurs there?

A bank of isolated modules with an aggregated readout is the architecture of exact unlearning (SISA; with adapters, SISA-A, SAFE, S3T). Two operations there are defined by retraining, so retraining cannot be avoided. One is answering a deletion request, and the other is valuing data by retraining on subsets (Data Shapley, Data Banzhaf, datamodels). For 32 events on 4 shards (Table 3), 16 deletion requests need 188 updates with whole-prefix reuse and 42 with FCT, the same as SISA's own checkpoints, and 10,000 sampled subsets need 99,337 and 1,020. On Qwen3-0.6B the stored states equal direct retraining in every bit (156 of 156 tensors), and on one A100, 10,000 sampled subsets of 32 events took 9.5 minutes with FCT and 6.2 hours with whole-prefix reuse (General Response 2/2). The third operation is choosing the shard assignment (SISA Sec. VIII-B, ARCANE, GraphEraser).

Q3. Which benefits come from local-state reuse, fixed numerical blocks, validation caching, and scheduling?

We re-ran the paper's public-task search (four committed blocks, beam width 4) with both executors on Qwen3-0.6B and SciQ, on Apple MPS with nothing else running (Appendix P.2, Table 64, means over 3 seeds).

ExecutorUpdatesValidation evaluationsTrain (s)Eval (s)Wall (s)
Whole-prefix reuse32233860187247
FCT18520138119156

Local-state reuse removes 40.6-43.7% of the updates, in line with the 36.8-46.9% measured on the A100, and 38.7-41.6% of the validation evaluations, because a state that two plans share is evaluated once. Validation caching matters because validation takes 74-78% of the baseline's time. Without it the same search would need 444-469 evaluations with FCT and 605-651 with whole-prefix reuse. FCT is 1.52-1.61x faster here, and the A100 runs in the paper gave 1.40-1.63x for the same search. Fixed numerical blocks add no speed; they make the reuse exact. In every seed both executors select the same plan and give bitwise identical states and outputs on all 1,000 test questions. Without fixed blocks this fails. On Apple MPS an update keeps its bits in another slot of a four-slot batch but not in a batch of one or eight slots, and the same round-robin plan run both ways ended at 62.1% and 69.4% test accuracy on one seed (Table 63). Scheduling is the weakest evidence. Keeping captured CUDA graphs cut captures from 449 to 104 and one invocation from 554.7 s to 168.5 s with identical outputs, but this comes from two invocations and is a diagnostic, not a controlled comparison (Appendix C).

Q4. Memory, transfer overhead and speedups against horizon, expert count and rank; eviction.

The share of whole-prefix updates avoided is 35-42% with 4 experts, 45-60% with 8 and 57-67% with 16; about 20% at beam width 2 and 62-70% at width 16; and 35-42% for 8 events per block and 26-40% for 32 (Table 57). Nine searches on Qwen3-0.6B fall inside these ranges (Table 58). Rank does not change the counts (39.7%, 39.5% and 39.7% avoided at ranks 2, 8 and 32), and two experts with no skipping save nothing. For valuation requests the speedup grows with the number of requests. On one A100 it rose from 14.1-14.8x at 1,000 subsets to 107x at 10,000 (4 shards of 6 events), and it was 39x for 10,000 subsets of 4 shards of 8 events and 16.1x for 8 shards of 8 events.

A state grows linearly with rank, from 2.3 MB at rank 2 to 36.7 MB at rank 32. In the paper's searches FCT stores less than whole-prefix reuse. For 180 plans on the A100 it stores 52 states (119 MB) against 522 (1,198 MB), and copying them to the host takes 0.04 s against 0.41 s, next to 3.4 s and 31.4 s of model time (Table 60). A beam of width w over K experts can drop every state that no kept partial plan reads, which bounds memory at 2wK states with no recomputation. In four searches this rule changed no result and held at most 14-35 states instead of 44-143 (Appendix N). For valuation requests served depth first FCT stores more. Whole-prefix reuse held at most 249 states, while FCT keeps all 1,020 histories of a 4 x 8 bank, 2.3 GB, a bound that does not grow with the number of requests but doubles with each further event per shard.

Thank you again for the questions. We hope the new results, including the negative one, give a clearer picture of what the method is for, and we would be grateful if you would reconsider your rating.

Thank you for the additional experiments and for candidly revising the paper’s claims. These additions have alleviated my main concerns. However, considering the revised scope of the contribution, I will maintain my current rating.

Reviewer_sEmrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the high computational cost of evaluating alternative training plans in multi-expert adaptation, where different plans may require identical expert updates even after their global training histories diverge. It proposes Factorized Counterfactual Training (FCT), which decomposes training plans into independent expert histories and caches complete expert states, including model parameters and optimizer states, to eliminate redundant updates. By organizing these states into a shared dependency graph, FCT reduces training computation while preserving exact training outcomes and enabling more candidate plans to be evaluated within a fixed time budget.

Critical strengths
  1. The paper identifies redundant expert updates across different training plans and provides a well-defined state factorization framework with theoretical guarantees for exact reuse.
  2. FCT combines complete expert-state caching, demand-driven execution, and GPU batching to eliminate redundant computation while preserving identical training states, evaluation scores, and search decisions.
  3. The paper evaluates multiple model architectures and workloads, demonstrating substantial computational savings and consistent speedups over strong baselines, with additional evidence that the saved computation enables broader training-plan exploration.
Critical weaknesses
  1. The core idea of FCT largely follows from standard memoization of independent, deterministic state transitions. The paper does not sufficiently clarify its technical novelty beyond existing computation-reuse frameworks
  2. The experiments primarily consider small numbers of independent LoRA experts and relatively constrained training-plan search workloads. It remains unclear whether similar computational redundancy and speedups exist in realistic large-scale adaptation systems.
Questions for authors

Can the authors provide evidence that real-world multi-expert adaptation systems frequently require evaluating overlapping training plans, and demonstrate that FCT remains effective under realistic memory constraints?

1 comment

Thank you for the review and the accurate summary. New material is blue in the revised PDF.

W1. Technical novelty beyond existing computation-reuse frameworks.

We agree that the rule is memoization of deterministic state transitions, and the revised Introduction says so. What differs from existing frameworks is what is reused and what it is stored under (new text in Sec. 2).

SystemWhat is reusedStored underReuse after two plans diverge
Nautilusoutputs of frozen layersthe frozen subgraphno trained state is shared
Helixworkflow intermediatesoperator and inputsno partially trained state
Hippo (stage tree), PBTwhole-model checkpointsthe trial's own sequence so farno
SISA slicingone shard's parametersa prefix of its one trained historyon that history only
mLoRA, LoRAFusion, tLoRA, ALTOkernels, batches, early stopping-no state reuse across candidates
FCTone expert's parameters and optimizer statethat expert's ordered history, and the batch arrangementyes

Two parts go beyond a direct cache. The first is counting. Storing states per expert turns (K+1)^T-1 updates into K(2^T-1), with closed forms for fixed budgets and for subsets and an exact count for adaptive searches. The second is exactness on batched hardware. A stored state equals a retrained one only if the update is computed the same way. On Apple MPS the same update is bitwise identical in another slot of a four-slot batch, but its last bits change in a batch of one or eight slots, and on CPU Qwen2.5 changes even in its own slot (Table 63). The native LoRAFusion path changes the selected plan in 3 of 3 runs (Sec. 4.6). FCT keeps optimizer state, treats the batch arrangement as part of a state's identity, and runs batches of one fixed size.

W2. Small numbers of experts and constrained search workloads.

Appendix M covers larger settings. The table gives the share of whole-prefix updates that FCT avoids in the paper's beam search, as ranges over three simple random score models with 20 searches each (Table 57). For 4 experts and width 4 the models give 35-42%, and the 24 real searches in the paper gave 36.8-46.9%.

ExpertsEvents per blockBeam widthAvoided
482 / 4 / 8 / 1616-22% / 35-42% / 48-56% / 62-70%
88 to 32445-60%
1616 to 32457-67%
2any40%

Nine searches on Qwen3-0.6B fall inside these ranges, with 17.9%, 39.7%, 54.9% and 67.7% at widths 2, 4, 8 and 16 and 52.8-57.5% with 8 experts (Table 58). Two experts with no skipping give no saving; we now prove this and state it as a limit. For deletion and valuation requests the saving also grows with the number of requests. On one A100 the time ratio rose from 14.1-14.8x at 1,000 sampled subsets to 107x at 10,000, and 8 shards with 64 events gave 16.1x at 3,000 subsets (General Response 2/2).

Q. Do real multi-expert systems evaluate overlapping plans, and does FCT remain effective under realistic memory constraints?

The setting is the architecture of exact unlearning (SISA; with adapters, SISA-A, SAFE, S3T). There, shard assignment is a design problem, and every deletion or data-valuation request is another training plan. Table 3 gives exact counts for 32 events on 4 shards; sliced checkpoints are the reuse SISA already has.

RequestsWhole prefixSliced checkpointsFCT
16 deletion requests1884242
10,000 sampled subsets (Banzhaf, datamodels)99,337120,4541,020
1,000 Shapley permutations173,444286,4961,020

FCT equals SISA for plain deletion and needs 97-170x fewer updates than whole-prefix reuse once plans branch. On Qwen3-0.6B the stored states equal direct retraining in every bit (156 of 156 tensors), and on one A100, 10,000 subsets took 9.5 minutes with FCT and 6.2 hours with whole-prefix reuse.

How much memory FCT needs depends on the workload. In searches it needs less than its baseline. For 180 plans on the A100, whole-prefix reuse stores 522 states (1,198 MB on Qwen3) and FCT 52 (119 MB; Table 60). A beam of width w over K experts needs at most 2wK states and never recomputes an update; in four searches on Qwen3-0.6B it held at most 35 states. With room for only 256 states (0.6 GB), best-first search still needs 1.5-1.6x fewer updates than whole-prefix reuse with unlimited memory, and below about 64 states this simple rule loses its advantage (Table 61). For valuation requests served depth first the order reverses. Whole-prefix reuse then holds at most 249 states, while FCT keeps all 1,020 histories of a 4 x 8 bank, 2.3 GB. That bound does not grow with the number of requests, but it doubles with each further event per shard.

We hope the positioning and the new scaling and memory results address your concerns, and we would be grateful if you would reconsider your rating.

Reviewer_3Zg8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper presents Factorized Counterfactual Training (FCT), a method whose core mechanism is memoization, and shows that FCT can speed up a problem that the authors construct as searching over training examples assignments to independent LoRA experts. In this problem, memoization works well because each expert’s state depends only on its own ordered training history, so different plans that share identical expert updates that can be computed once and reused. FCT thus gives speedups over baselines such as whole-prefix reuse.

Critical strengths

The paper evaluates multiple models, workloads, numerical configurations and provides detailed implementation details, derivations, timing details, and experimental results.

Critical weaknesses

The method has very limited novelty as the reuse rule itself is standard memoization under an independence assumption.

The paper does not convincingly establish why searching over assignments of training examples to independent LoRA experts is an important or representative of real world workloads. I feel the problem formulation is contrived to create opportunities for reuse.

Questions for authors

What would be real world use case for the method?

1 comment

Thank you for the review. We answer your question first and then the two weaknesses. New material is blue in the revised PDF.

Q. What would be a real-world use case?

Sharded adapter banks, as used for exact machine unlearning. SISA (S&P 2021) trains isolated models on disjoint shards of the data and aggregates their predictions. Adapter variants keep a frozen backbone and train adapters on shards or slices (SISA-A; SAFE, ICCV 2023; S3T, ICLR 2025). That is our setting, independent experts on a frozen model with an aggregated readout. Running such a bank means comparing training plans, in which each training event goes to one module or is skipped, and it happens in three ways. First, which shard gets which data is a studied design problem (SISA Sec. VIII-B on distribution-aware sharding; ARCANE, where partitioning by class "reduces the accuracy loss"; GraphEraser, where random partitioning "could severely damage the resulting model utility"). Our SciQ and ARC experiments are this case. Second, every unlearning request asks for the bank trained without some events. Third, Data Shapley, Data Banzhaf and datamodels need models trained on many subsets; datamodels train 300,000 models for one CIFAR-10 setting.

The second and third are defined by retraining, so their cost is set by how retraining is executed. Table 3 gives exact update counts for 32 events on 4 shards. Sliced checkpoints are the reuse SISA already has, checkpoints along the one history each shard was trained on.

RequestsReplayWhole prefixSliced checkpointsFCT
16 deletion requests3761884242
Leave-one-out1,024528144144
10,000 sampled subsets160,39999,337120,4541,020
1,000 Shapley permutations496,032173,444286,4961,020
All 2^32 subsets6.9e104.3e9-1,020

For plain deletion FCT does what SISA already does, and we now say so. Its advantage appears once plans branch, as in valuation or in comparing shardings. On Qwen3-0.6B with 24 events, 252 updates cover all 16.8 million subset plans, and the stored states equal direct retraining in every bit (156 of 156 tensors). On one A100, 10,000 sampled subsets of 32 events took 9.5 minutes with FCT and 6.2 hours with whole-prefix reuse, with identical results (Appendix R).

W1. "The reuse rule itself is standard memoization under an independence assumption."

We agree, and the revised Introduction says so. Our claims concern what makes this memoization correct and worthwhile. First, we store a state under its expert's own history, while training reuse in prior systems follows a global prefix (Hippo's stage tree, population-based training) or one trained history (SISA). This allows reuse after two plans diverge. For K experts, T events and the option to skip, it turns (K+1)^T-1 updates into K(2^T-1), or 3,124 into 124 in Sec. 4.2. Second, we state when reuse is exact. A stored state may replace a recomputed one only if the two are equal, and on batched hardware they are not by default. On Apple MPS the same update is bitwise identical in another slot of a four-slot batch, but its last bits change if the batch has one or eight slots, and on CPU Qwen2.5 updates change even in their own slot (Table 63). Training amplifies such differences. The same plan run in one-slot and four-slot batches ended at 62.1% and 69.4% test accuracy on one seed, and with the native LoRAFusion path the selected plan changed in 3 of 3 runs (Sec. 4.6). FCT therefore stores optimizer state, treats the batch arrangement as part of a state's identity, and runs batches of one fixed size. This gives equality in every bit of states, scores, decisions and test predictions in all paired runs. Third, we measure how much it saves against whole-prefix reuse on one backend, with low-reuse controls that stay near 1.0x.

W2. "The problem formulation is contrived to create opportunities for reuse."

The submission introduced the setting abstractly and showed only assignment search, so we understand this reading. We did not choose independence to create reuse. Exact unlearning isolates modules on purpose, because isolation is what makes retraining affordable. The revised Introduction starts from the three scenarios above, and Sec. 4.7 measures deletion and valuation requests.

We should also say what we do not claim. FCT saves nothing for a single plan, for a search that follows one path, or for two experts when no event may be skipped (Appendix M). Following Reviewer Ej8W, we compared assignment search with tuned training that does no search. The search gives no clear advantage (79.0% against 79.9% on SciQ), so we withdrew the accuracy claim from the abstract and conclusion.

We hope the concrete setting and the new measurements address your concerns, and we would be grateful if you would reconsider your rating.

Comments on the paper

2 comments

We thank the four reviewers. Two concerns are shared. The reuse rule is memoization, so what is the contribution? And is comparing assignments of training events to independent experts a real workload? We answer both, report new experiments, and withdraw one claim. All changes are blue in the revised PDF. They include a rewritten Introduction, additions to Sec. 2 and 3, a new Sec. 4.7, and new Appendices L to R. New code and result files are in the supplement. Part 2 of this response covers scale and memory.

Where does the setting occur?

A bank of isolated trainable modules on a frozen backbone, each trained on its own share of the data and aggregated at inference, is the architecture of exact machine unlearning (SISA; with adapters, SISA-A, SAFE and S3T). Three routine operations on such a bank compare training plans, in which each training event goes to one module or is skipped. The first is deciding which shard gets which data, a studied design question (SISA Sec. VIII-B, ARCANE). The second is deletion, since every unlearning request asks for the bank trained without the removed events. The third is data valuation, since Data Shapley, Data Banzhaf and datamodels need models trained on many subsets. Table 3 counts the updates for 32 events on 4 shards. Sliced checkpoints are the reuse SISA already has, checkpoints along the one history each shard was trained on.

RequestsReplayWhole prefixSliced checkpointsFCT
16 deletion requests3761884242
10,000 sampled subsets160,39999,337120,4541,020
All 2^32 subsets6.9e104.3e9-1,020

For plain deletion FCT does what SISA already does, and the paper now says so. Once plans branch, FCT stops at the number of distinct expert histories. On Qwen3-0.6B the stored states equal direct retraining in every bit (156 of 156 tensors). On one A100, 10,000 sampled subsets of 32 events took 9.5 minutes with FCT and 6.2 hours with whole-prefix reuse, with identical results (part 2).

What is new beyond memoization?

We agree that the rule is memoization, and the Introduction now says so. We claim three narrower things. First, we store a state under its expert's own history. Prior training reuse follows a global prefix (Hippo's stage tree, PBT) or one trained history (SISA), and multi-LoRA systems (mLoRA, LoRAFusion, tLoRA, ALTO) batch jobs but, to our knowledge, do not reuse adapter state across candidates. Storing states this way turns (K+1)^T-1 updates into K(2^T-1). Second, exact reuse is not automatic. On Apple MPS the same update is bitwise identical in another slot of a four-slot batch, but its last bits change in a batch of one or eight slots, and on CPU Qwen2.5 changes even in its own slot (Table 63). Training amplifies this. The same plan run in one-slot and four-slot batches ended at 62.1% and 69.4% test accuracy on one seed, and the native LoRAFusion path changed the selected plan in 3 of 3 runs. FCT therefore stores optimizer state, treats the batch arrangement as part of a state's identity, and runs batches of one fixed size. Third, adaptive searches make identical decisions, and a beam search needs at most 2wK stored states and never recomputes.

Is search a better use of compute than training longer? (Ej8W)

Table 65 compares them on Qwen3-0.6B and SciQ, with 3 new seeds and the same data for every method.

MethodUpdatesVal. evalsTest acc. (%)
Round-robin, one pass36473.8 (69.4-78.5)
Random balanced partition36464.7 (40.9-80.2)
Single rank-8 adapter, rate and passes selected5041479.9
Round-robin bank, rate and passes selected3964479.2
Assignment search, FCT18920179.0 (78.2-79.9)
Same search, whole prefix326338identical

The search gives no clear accuracy advantage. At the paper's learning rate of 0.02 the result depends on the assignment (rows 1 and 2), and the search avoids the bad ones. At 0.01, round-robin, the random partition and the search all reach 78.0-79.0%. Tuned training without search comes within a point of the search or beats it, at lower cost. We therefore withdraw the accuracy claim. The abstract, Sec. 4.5 and the conclusion now claim only that a fixed budget evaluates more candidates. FCT changes the cost of evaluating requested plans, not which plans are worth requesting.

FCT saves nothing for a single plan, for a search that follows one path, or for two experts when no event may be skipped. The new experiments use a second implementation in plain PyTorch, on Apple MPS and on one A100, not the paper's CUDA-graph executor. Update counts are exact, and timings were taken with nothing else running on the device.

Three reviewers asked how the saving scales and whether memory limits it (sEmr W2 and Q, Ej8W Q4, p4uW Q4). We answer with a new measurement at scale (Appendix R) and with the scaling and memory results of Appendices M and N.

First, we timed valuation requests on one A100 (80 GB), with Qwen3-0.6B in FP32, rank-2 experts, and SciQ events of 8 examples assigned to the shards in rotation. Each requested plan trains the bank on one subset of the events and is scored on 64 validation questions. We gave whole-prefix reuse its best case. Plans arrive sorted, so it walks the stage tree depth first, frees a state as soon as no later plan needs it, and never recomputes an update.

Shards x events per shardRequestsUpdates, whole prefix / FCTTime, whole prefix / FCTRatio
4 x 61,000 subsets, 3 seeds7,569-7,618 / 25232.5-35.6 min / 2.2-2.5 min14.1-14.8x
4 x 610,000 subsets58,988 / 2524.7 h / 2.6 min107x
4 x 810,000 subsets99,038 / 1,0206.2 h / 9.5 min39x
4 x 8200 Shapley permutations (5,782 plans)39,469 / 1,0082.9 h / 12.7 min13.5x
8 x 83,000 subsets80,196 / 2,0404.3 h / 16.2 min16.1x

In all 7 runs both executors return the same utility in every bit for all 31,788 plans. The arithmetic behind the table is simple. A batched training call costs 0.35-0.36 s and an evaluation call 0.48-0.56 s, the same for both executors to within 1.0%, so time is the number of calls times these costs. With four shards, whole-prefix reuse pays 5.9-9.9 updates and 3.1-3.9 evaluations for each new subset, about 1.7-2.2 s. FCT pays once for each expert history, at most K(2^c - 1) updates for K shards of c events. Its whole bill for the 4 x 8 bank equals what whole-prefix reuse pays for 255 subsets, and every further subset costs FCT only the mixture readout. For the 4 x 6 bank, ten times as many subsets multiplied the time of whole-prefix reuse by 8.7 and that of FCT by 1.15, so the ratio grows with the number of requests. It stays below the update ratio (39x against 97x for 4 x 8) because whole-prefix reuse evaluates only plan endpoints and because the last FCT updates fill fewer of the four lanes of a call (Table 70).

Second, the paper's beam search saves more as the beam widens and as experts are added. Over three simple score models, the share of whole-prefix updates that FCT avoids grows from about 20% at width 2 to 35-42% at width 4 and 62-70% at width 16, and reaches 45-60% with 8 experts and 57-67% with 16. Nine searches on Qwen3-0.6B fall inside these ranges (Tables 57 and 58). Two experts with no skipping save nothing, which we now prove. When experts are coupled, the coupled group becomes the unit of reuse. Pairs among 8 experts still avoid 38-46%, and full coupling reduces FCT to whole-prefix reuse (Appendix O).

Third, how much memory FCT needs depends on the workload. In the paper's searches FCT stores less than whole-prefix reuse, 52 states (119 MB) against 522 (1,198 MB) for 180 plans on the A100 (Table 60), and a beam of width w needs at most 2wK states without any recomputation. For valuation requests served depth first it is the other way round. Whole-prefix reuse held at most 249 states, while FCT keeps every state it trains, 1,020 states or 2.3 GB for 4 x 8 and 4.7 GB for 8 x 8, at 2.29 MB per rank-2 state. This bound does not grow with the number of requests, but it doubles with each further event per shard. Four shards of 12 events would need about 38 GB, and states would then have to go to host memory or be recomputed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.