acceptodds
Under review as a conference paper at ICLR 2027

Which Host Versions Should a Removable Branch Learn From?

Abstract

A branch trained beside a continuing host should be removable without changing the host's training. Optimization-Recoverable Augmentation () enforces this through complete-state isolation, with conditional guarantees and Qwen/BERT recovery tests. Which host versions should the branch observe before deployment on the final host? In one-block Qwen2.5-0.5B, reader-progress observation () exceeds terminal-only training by on GSM8K/SQuAD with only the version sequence changed. The advantage survives terminal learning-rate and checkpoint selection: on a separate estimation half of the historical holdout. Two prospective comparisons with latest-version observation give positive paired differences. The benefit is conditional on the branch's data. Independent branch orders attenuate the gain over latest-version observation. In a preregistered equal-size data control, exceeds terminal-only by when host and branch share a training stream, but trails by when host and branch train on group-disjoint data. A retained-seed extension separates order and data contrasts: independent ordering reduces the shared-data advantage, while SQuAD retains a positive gap. The tested full-model recipe also favors terminal-only training. These results establish a repeatable shared-stream observation effect, rather than a general advantage of earlier host versions, and show why observation policies must be evaluated jointly with branch data and the deployment host.

Primary area:
optimization
Keywords:
optimization recoverability, training dynamics, state isolation, branch training, parameter-efficient fine-tuning, reproducible training

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_o3Z9OpenReview ↗
Rating
2
Confidence
1(−2)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to train a removable branch alongside a continuing host without changing the host’s standalone training trajectory. It proposes ORA, which achieves this through complete-state isolation, and compares policies for selecting host versions for branch training. Formal and recovery tests validate host isolation; on Qwen’s GSM8K and SQuAD tasks, reader-progress outperforms terminal-only and latest-version observation in the tested one-block, shared-data setting, but the advantage varies and can reverse under other data and training conditions. The central finding is therefore conditional rather than universal.

Critical strengths
  1. A clear, empirically verified recovery contract for the host. ORA isolates the host’s and branch’s training state, and state-equality checks plus branch removal and restoration tests show that, under the stated conditions, the host follows the same trajectory as standalone training. This makes removability a verifiable systems property and provides a reliable basis for the quality comparisons.
  2. Carefully controlled experiments that test the limits of the main finding. The authors compare observation policies while holding the final host fixed and support the shared-data result with paired experiments and prospective replications. They also show that the advantage reverses with host-disjoint data and in full-model training. This makes the contribution more than a single performance result: it clarifies when the effect holds and when it does not.
Critical weaknesses
  1. The external validity of the central quality finding is limited. The evaluation focuses on one Qwen2.5-0.5B-Instruct model, one-block training, and two QA tasks, using prompts without the model’s chat template. Without robustness results across model scales, a Base checkpoint, or the official chat template, it is difficult to determine whether the observed advantage generalizes beyond this particular model, training recipe, and input format.

  2. The proposed policy does not show a stable advantage over strong alternatives. Reader-progress outperforms terminal-only and latest-version observation in the shared-data setting, but the paper does not establish joint noninferiority to clean-plus-joint across both tasks. Its advantage also reverses with host-disjoint data and full-model training. This limits the paper’s ability to recommend a generally applicable policy and weakens its guidance for practical deployment choices.

Questions for authors

see weaknesses

1 comment
Reviewer_o3Z9OpenReview ↗

Response to the authors

Thanks for the authors’ response. I’ll keep my rating but lower my confidence in it, good luck.

Reviewer_YVNNOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method, ORA, to observe host versions of an LLM branch before deployment to the final host. The paper compares latest version observation with ORA observation on Qwen2.5-0.5B evaluated on GSM8K and SQuAD.

Critical strengths

This paper proposes a practical method for tracking and reproducibility for progress observation, and there seem to be some gains over vanilla models (freezing, latest-version observation). Within the small set of conditions that it is evaluated on (one small model, two benchmarks), experiments are quite extensive.

Critical weaknesses

Concerns:

  • this is a very minimal contribution for an ICLR full conference paper: an accounting method for the observation branch that preserves the host trajectory through tracking and source control
  • for a simple method whose main contribution is that is it works well, the experimental work is fairly light (one tiny model, two very basic benchmarks, a small set of branch data configurations)
Questions for authors

As an empirical method, how robust is this to other benchmarks? What about other models? Qwen2.5-0.5B is extremely small and can have different training behavior/saturation vs larger models or models with different architectures (eg MoEs or models with memory banks).

Reviewer_fCFAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies which host versions a removable branch should learn from while its host continues training independently. It proposes ORA to preserve the host's training trajectory through complete-state isolation, and a reader-progress policy (P) that selects host checkpoints according to the branch's own progress. Experiments on Qwen2.5-0.5B show that P outperforms terminal-only and latest-version observation in a one-block, shared-stream setting, but these results reverse with disjoint data and full-model training.

Critical strengths
  1. In the restricted experimental setting, P improves over terminal-only training by 4.67/13.87 percentage points on GSM8K/SQuAD, retaining an advantage after expanding the baseline selection search.
  2. The paper proposes ORA, a training scheme that isolates the host and branch.
Critical weaknesses
  1. The main positive finding depends on a restrictive parameter partition, with insufficient explanation. Only decoder layer 23 and final normalization are updated in the host; LoRA attaches to Q/V projections in layers 0–22, whose base weights remain frozen. The branch therefore adapts to a changing downstream block while its underlying base weights stay fixed, limiting how well this setting represents an evolving backbone. Full-model training reverses the advantage, but also changes the host learning rate, preventing the effects of parameter partition, optimization configuration, and fitting behavior from being separated.
  2. The baseline configuration is questionable. Restricting host updates to the final block may give the host less capacity to adapt than the added branch, potentially reversing their intended roles and weakening the interpretation of the results. Overfitting is a plausible hypothesis, but it could also be related to the host's limited adaptation capacity. The reversal under full-model training is consistent with this possibility.
  3. Independent evaluation of the headline policy remains incomplete. P was developed after observing earlier outcomes, and confirmations reuse existing partitions. Splitting the historical holdout protects subsequent terminal selection, not earlier adaptive reuse. Official-split results in Table 27 concern ORApost and its original comparator, rather than the final P-versus-tuned-terminal comparison. Fresh seeds establish repeatability conditional on the same data.
Questions for authors
  1. Can controlled comparisons vary the host's trainable layers and their overlap with LoRA locations while keeping the host learning rate fixed?
  2. Is there a concrete use case for this training strategy?
  3. Can retained P and tuned terminal-only checkpoints be compared on the official GSM8K test and SQuAD validation splits without further selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.