- Rating
- 2
- Confidence
- 1(−2)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to train a removable branch alongside a continuing host without changing the host’s standalone training trajectory. It proposes ORA, which achieves this through complete-state isolation, and compares policies for selecting host versions for branch training. Formal and recovery tests validate host isolation; on Qwen’s GSM8K and SQuAD tasks, reader-progress outperforms terminal-only and latest-version observation in the tested one-block, shared-data setting, but the advantage varies and can reverse under other data and training conditions. The central finding is therefore conditional rather than universal.
Critical strengths
- A clear, empirically verified recovery contract for the host. ORA isolates the host’s and branch’s training state, and state-equality checks plus branch removal and restoration tests show that, under the stated conditions, the host follows the same trajectory as standalone training. This makes removability a verifiable systems property and provides a reliable basis for the quality comparisons.
- Carefully controlled experiments that test the limits of the main finding. The authors compare observation policies while holding the final host fixed and support the shared-data result with paired experiments and prospective replications. They also show that the advantage reverses with host-disjoint data and in full-model training. This makes the contribution more than a single performance result: it clarifies when the effect holds and when it does not.
Critical weaknesses
-
The external validity of the central quality finding is limited. The evaluation focuses on one Qwen2.5-0.5B-Instruct model, one-block training, and two QA tasks, using prompts without the model’s chat template. Without robustness results across model scales, a Base checkpoint, or the official chat template, it is difficult to determine whether the observed advantage generalizes beyond this particular model, training recipe, and input format.
-
The proposed policy does not show a stable advantage over strong alternatives. Reader-progress outperforms terminal-only and latest-version observation in the shared-data setting, but the paper does not establish joint noninferiority to clean-plus-joint across both tasks. Its advantage also reverses with host-disjoint data and full-model training. This limits the paper’s ability to recommend a generally applicable policy and weakens its guidance for practical deployment choices.
Questions for authors
see weaknesses
Response to the authors
Thanks for the authors’ response. I’ll keep my rating but lower my confidence in it, good luck.