- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper studies how an update candidate for a language model can be selected before running any of the available ones. The presented framework, Counterfactual Plasticity, predicts each candidate’s acquisition gain and collateral damage using only information available before updating. It excludes candidates based on a calibrated damage criterion, and then selects the highest-gain ranking one to realize. They find that gain is easier to predict than damage, and evaluate on a small controlled GPT-2 experiment, locked evaluations on TinyLlama and Gemma, and Qwen2.5-7B.
Critical strengths
S1: The problem is clearly defined, and the importance of the problem is well motivated.
Critical weaknesses
W1 (Engagement with literature): While the paper cites seven works in the related work section, there is no engagement with the literature in any other part of the paper, making it hard to set the contribution into context. The introduction’s motivating claim, that model updates are evaluated mostly retrospectively, is stated without providing evidence or supporting references.
W2 (Presentation): The decision-level calibration (lines 153-161) is hard to follow without prior knowledge of conformal risk control: is not defined, and the main inequality is not derived or referenced. It is unclear to me, how choices for parameters like 90 disjoint calibration edits, =.10, or the 61 numerical values for lambda are justified. Section 3 mixes the method with experimental settings for E1, splitting it and restructuring the paper would help make the paper clearer. The numbers in Section 4 and Figure 1 do not align for the Original menu.
W3 (Main paper is not self-contained): The forecaster is the component that determines whether the proposed framework is useful. However, the forecaster that predicts gain and damage from pre-update information is not described in the main paper. The guarantee holds only for one fixed training set and pipeline, and it is unclear to me how it applies in other settings.
W4 (Experiments):
- The controlled GPT-2 experiments use a 10-fold CV-CRC procedure that does not inherit the E1 split-CRC guarantee, so it does not support any previous safety claims.
- The regimes are not formally defined, making it unclear to me what “harder experiences” or “denser strenghtst” refer to in practice.
- The primary comparator, B1, shares CP’s six-candidate action space and forecasting machinery and lacks only the calibrated exclusion, so the experiments isolate the effect of calibration, not the value of pre-execution forecasting itself, which is the paper’s motivating question.
- R_CRC is undefined.
- There is no reference point for how much any update could improve the model, and baselines from the literature are missing, so the numbers are hard to interpret and it is difficult to assess the achievement of the proposed method.
Questions for authors
Q1: Are the 90 calibration edits representative of the test edits?
Q2: How was = 0.10 chosen, and what happens to harm and gain if you use = 0.05 or 0.20?
Q3: Calibration requires running every candidate on every one of the 90 calibration edits. How many update runs is that in total, and after how many future edits does forecasting become cheaper than simply running all candidates each time?
Q4: How do the results transfer to other training data or forecasters? Which assumptions does the forecaster have to fulfill?
Q5: Which loss defines gain and damage, and in what units? What does a gain difference of 1 mean in practice?
Q6: What is the gain of the best possible choice for each edit (including abstaining when no candidate is safe), and what percentage of it does CP reach? How does a simple rule that needs no forecaster, such as always using the same fixed setting, compare?