- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper studies how text-only interaction traces from a stronger closed-source model can be used to construct activation steering for a frozen open-source student. The student replays teacher trajectories offline, extracts states before and after feedback and at the beginning and end of responses, and computes U (update magnitude), C (direction retention), V (a J-lens-based ranking of content tokens), and S (cross-turn consistency). The method matches local events from successful and failed trajectories of the same task using U/V, filters them by differences in C/S, and averages the success-minus-failure differences in the student's original response-state transitions separately by layer to obtain steering directions. Online, the corresponding direction is added at a fixed layer before each response. The authors report aggregate improvements for three student models on Terminal-Bench 2.1 and a tool-enabled HLE subset, and compare against SFT on the same trajectory data in the Qwen/Terminal-Bench experiment. The evaluation is explicitly described as within-benchmark adaptation, without a task-level held-out split.
Critical strengths
-
The method requires teacher text traces rather than access to the closed-source model's internal states, and does not update student parameters online. Turning offline state analysis into reusable activation directions is practically appealing. The paper also clearly distinguishes offline construction from online deployment.
-
The aggregate scores in all six model-benchmark settings in Table 1 improve over the listed controls. Table 2 additionally provides a Qwen SFT comparison using the same 124 trajectories and reports fewer online tokens and interactions. These results provide initial empirical support. The paper also discloses task overlap, the rank-based interpretation of V, and the scope of its online cost accounting.
Critical weaknesses
-
The evaluation design and incremental results do not sufficiently establish the method's reusable value. Section 3.3 explicitly uses outcome-labeled teacher trajectories from the evaluation task collection, without a task-level held-out split. For Terminal-Bench, construction uses 124 trajectories from 33 evaluation tasks, while the final scores are reported over 89 tasks. This setting is disclosed and should not be characterized as undisclosed data leakage. Nevertheless, the aggregate scores do not separate adaptation gains on construction tasks from gains on the remaining tasks. Relative to the most relevant REVEAL-Base control, the full method yields net gains of only 3, 1, and 3 tasks for Qwen, Gemma, and GPT-OSS, respectively, and the main performance table does not report uncertainty across repeated runs. Qwen's Hard score also falls from 6/30 (20%) for Base to 5/30 (16.67%). This is a Qwen-specific regression relative to Base, not a claim that every model regresses relative to its original version. It makes the stability of the complex event-selection procedure on difficult tasks particularly unclear. Table 2 does include a same-data SFT comparison, but only for Qwen/Terminal-Bench; training details, handling of failed trajectories, and the adequacy of baseline tuning still need clarification. The reported cost advantage covers only online execution after adaptation, excluding teacher sampling, offline replay, and metric computation, and therefore does not establish end-to-end cost savings.
-
The measurement validity, selection rationale, and necessity of J-lens/V lack direct experimental support. V is a vocabulary-ranking score obtained by applying a fixed readout to the pre-response state, not an actual generation probability, content-recovery rate, or correctness measure. That distinction alone does not invalidate the method. Appendices A.4, B.3, and D.4 provide ranking-property and content-coverage diagnostics and acknowledge systematic preferences for frequent tokens and task vocabulary; these diagnostics do not establish superiority over alternative readouts. The substantive questions are whether V remains a reliable, task-relevant matching signal after controlling for lexical preferences, and whether final performance depends on the particular fitted lens, supported layers, and tokenization. V directly affects event pairing and thus the samples used to compute directions; it is not merely a visualization metric. I did not find controlled online performance comparisons that remove V and match using U alone, or replace J-lens with a direct or alternative readout. The aggregate gains cannot be attributed specifically to this measurement and filtering design.
Questions for authors
The following questions could materially affect my assessment:
-
Please report per-task results and aggregate gains separately for the 33 Terminal-Bench tasks used for direction construction and the remaining 56 tasks, and identify which tasks were used to select metric statistics, intervention layers, strengths, and filtering thresholds. The remaining 56 tasks should not automatically be treated as a strictly held-out set. Can you provide an independent task-level split with configurations selected only on construction/development tasks?
-
Are the net gains of 3, 1, and 3 tasks over Base stable across repeated runs? Please provide paired per-task success/failure outcomes and repeated-run intervals, and explain which task outcomes account for the Qwen Hard decrease from 6/30 to 5/30.
-
Please compare online performance after removing V and matching only on U, and after replacing the J-lens readout. Control the number of selected events, intervention layer, strength, and tuning budget. Do controls for frequent tokens, task vocabulary, and tokenization differences change V-based matching or final gains?
-
For the SFT baseline in Table 2, please provide training configurations, supervision/loss masks, failed-trajectory handling, and the tuning procedure, as well as a successful-trajectories-only result. Does the comparison hold for other students or HLE? If practical cost savings are claimed, please also report teacher-trajectory sampling, offline state replay, and metric-computation costs and explain their amortization.
9 comments
Response to Reviewer 5xJ8 (Part 2 of 5)
Table A. Qwen3.6-27B on Terminal-Bench 2.1, by role of each task in REVEAL's construction. Base uses trajectories from all 89 tasks.
| Task group | N | Vanilla | Base (Outcome-CAA) | REVEAL | REVEAL − Base |
|---|---|---|---|---|---|
| Vector + statistics | 33 | 9/33 (27.27%) | 14/33 (42.42%) | 12/33 (36.36%) | −6.06 pp |
| Statistics only | 2 | 0/2 (0.00%) | 2/2 (100.00%) | 1/2 (50.00%) | −50.00 pp |
| Not used by REVEAL | 54 | 21/54 (38.89%) | 16/54 (29.63%) | 22/54 (40.74%) | +11.11 pp |
| All | 89 | 30/89 (33.71%) | 32/89 (35.96%) | 35/89 (39.33%) | +3.37 pp |
Table B. Vector rebuilt from the 56 non-contributing tasks (Terminal-Bench 2.1, Qwen3.6-27B, accuracy %).
| Method | Easy | Medium | Hard | All |
|---|---|---|---|---|
| REVEAL-Base | 75.00 | 41.82 | 20.00 | 35.96 |
| REVEAL (rebuilt from 56 tasks) | 75.00 | 49.09 | 13.33 | 38.20 |
| REVEAL (original vector) | 100.00 | 47.27 | 16.67 | 39.33 |
W1 (part 2). Relative to the most relevant REVEAL-Base control, the full method yields net gains of only 3, 1, and 3 tasks for Qwen, Gemma, and GPT-OSS, respectively, and the main performance table does not report uncertainty across repeated runs. Qwen's Hard score also falls from 6/30 (20%) for Base to 5/30 (16.67%). This is a Qwen-specific regression relative to Base, not a claim that every model regresses relative to its original version. It makes the stability of the complex event-selection procedure on difficult tasks particularly unclear.
Q2. Are the net gains of 3, 1, and 3 tasks over Base stable across repeated runs? Please provide paired per-task success/failure outcomes and repeated-run intervals, and explain which task outcomes account for the Qwen Hard decrease from 6/30 to 5/30.
Response to W1 (part 2) and Q2: We repeat the Qwen3.6-27B evaluation on Terminal-Bench 2.1 four times at temperature 1.0 for both REVEAL-Base and REVEAL, with the same vectors and protocol (Table C). REVEAL reaches 38.20% ± 0.92 pp (mean ± SD; 95% CI 36.74%–39.66%), versus 33.43% ± 0.56 pp for Base. The gain of 4.77 pp has a 95% CI of [3.38, 6.16] and is significant (Welch's t-test across runs, p < 0.001).
The Hard decrease does not recur across repeated runs: REVEAL averages 15.83% on Hard tasks, versus 8.33% for Base. In the original run, REVEAL and Base differ by a net of one Hard task (5/30 vs. 6/30), and Table H lists the per-task outcomes. A one-task net difference on 30 Hard tasks falls within run-to-run variation, as the repeated runs confirm.
Table C. Repeated evaluation of Qwen3.6-27B on Terminal-Bench 2.1 (accuracy %, mean ± SD over four runs, temperature 1.0).
| Method | Easy | Medium | Hard | All |
|---|---|---|---|---|
| REVEAL-Base | 56.25 ± 12.50 | 45.45 ± 1.48 | 8.33 ± 1.92 | 33.43 ± 0.56 |
| REVEAL | 87.50 ± 14.43 | 46.82 ± 2.29 | 15.83 ± 3.19 | 38.20 ± 0.92 |
Response to Reviewer 5xJ8 (Part 3 of 5)
W2. The measurement validity, selection rationale, and necessity of J-lens/V lack direct experimental support. V is a vocabulary-ranking score obtained by applying a fixed readout to the pre-response state, not an actual generation probability, content-recovery rate, or correctness measure. That distinction alone does not invalidate the method. Appendices A.4, B.3, and D.4 provide ranking-property and content-coverage diagnostics and acknowledge systematic preferences for frequent tokens and task vocabulary; these diagnostics do not establish superiority over alternative readouts. The substantive questions are whether V remains a reliable, task-relevant matching signal after controlling for lexical preferences, and whether final performance depends on the particular fitted lens, supported layers, and tokenization. V directly affects event pairing and thus the samples used to compute directions; it is not merely a visualization metric. I did not find controlled online performance comparisons that remove V and match using U alone, or replace J-lens with a direct or alternative readout. The aggregate gains cannot be attributed specifically to this measurement and filtering design.
Q3. Please compare online performance after removing V and matching only on U, and after replacing the J-lens readout. Control the number of selected events, intervention layer, strength, and tuning budget. Do controls for frequent tokens, task vocabulary, and tokenization differences change V-based matching or final gains?
Response to W2 and Q3: Table D reports both requested controls under the same candidate pool, intervention layer (40), strength (α = 0.5), and selection rule, with no variant retuned. w/o V matches events on U alone, and V (logit lens) replaces J-lens transport with the identity map while keeping everything else. The number of selected events is also identical across variants, since each rollout pair yields min(#success events, #failure events) matched pairs and the top half is kept regardless of the matching features. Removing V lowers accuracy from 39.33% to 38.20%, and the logit lens lowers it further to 37.08%, below not using V at all, with the entire loss on Hard tasks (10.00% vs. 16.67%). V's contribution therefore comes from the J-lens readout itself: an unsuitable readout misguides event pairing rather than merely failing to help.
The reason lies in how V is used. V is a rank statistic, not a generation probability (Proposition 2, App. B.3), and REVEAL uses it only to decide which events are comparable; the direction itself is computed from the target's raw response-state transitions (App. A.3, B.4). V must therefore measure how accessible the content of the whole upcoming response is from the boundary state. A logit lens reads an intermediate state as if it were already the final state, skipping the transformation applied by later layers, and a tuned lens is fitted for next-token fidelity. The J-lens averages the Jacobian transport from the source layer to the output layer (App. A.1), and thus reads what the boundary state contributes to the downstream response, which is exactly the quantity matching requires.
The lexical factors you raise are controlled by the pairing itself. Both events in a matched pair come from the same task and teacher, and are measured with the same observer, tokenizer, layer, and lens. Task vocabulary, tokenization, and the lens's preference for frequent tokens are therefore shared within each comparison and largely cancel in the matching distance. In addition, V counts each unique content token once, so repeated tokens add no weight, and as a rank statistic it is invariant to any strictly increasing transformation of the scores.
Table D. V and readout controls (Terminal-Bench 2.1, Qwen3.6-27B, accuracy %).
| Method | Easy | Medium | Hard | All |
|---|---|---|---|---|
| Vanilla | 75.00 | 43.64 | 10.00 | 33.71 |
| REVEAL-Base | 75.00 | 41.82 | 20.00 | 35.96 |
| w/o V (U-only matching) | 75.00 | 49.09 | 13.33 | 38.20 |
| V (logit lens) | 100.00 | 47.27 | 10.00 | 37.08 |
| REVEAL (J-lens) | 100.00 | 47.27 | 16.67 | 39.33 |
Response to Reviewer 5xJ8 (Part 1 of 5)
Thank you for the careful and detailed review, and for recognizing the practical appeal of converting teacher text into reusable activation controls without access to teacher internals, as well as the gains across all six model–benchmark settings and over same-data SFT. Your questions pinpoint the evidence needed to assess REVEAL's reusable value, and we have run the corresponding experiments. We address each point below.
Response to W1 (part 1) and Q1: We answer both parts of the question: which tasks enter each stage of construction, and how the configuration is chosen.
Which tasks are used. Table H at the end of this response (App. E.1, Table 12 in the revision) lists, for each of the 89 tasks, whether it contributes to the steering vector, whether it enters the U, V, C, and S statistics, and the outcome of each method. The tasks fall into three groups. 33 tasks (37.1%) contribute to both the statistics and the vector; 2 tasks (2.2%;
cancel-async-tasks,git-multibranch) enter only the statistics; and 54 tasks (60.7%) are never used by REVEAL. No task enters the vector without also entering the statistics. In trajectory terms, REVEAL computes its statistics on 131 trajectories from 35 tasks and builds its vector from 124 trajectories of 33 tasks (424 matched event pairs), whereas Base averages over 338 trajectories covering all 89 tasks, drawn from the same sampled pool. The per-task counts are similar (3.76 vs. 3.80), so REVEAL uses less than 40% of Base's trajectories because it keeps only tasks with both successful and failed rollouts. Base has therefore seen every evaluation task during construction, while REVEAL has never seen 54 of them. Table A summarizes the outcomes by group. On these 54 tasks, REVEAL solves 22, versus 21 for Vanilla and 16 for Base, gaining six tasks over Base and losing none, even though Base's direction was built partly from these same tasks. On the 33 contributing tasks, Base solves 14 and REVEAL 12, so REVEAL's overall gain over Base comes entirely from tasks it never used.How the configuration is chosen. No task is used to select the intervention layer, strength, or selection threshold. Steering performance depends jointly on layer and strength and is sensitive to the target model, so searching this space per task or per benchmark would both require held-out data and risk overfitting. We therefore fix one setting per target before evaluation, following common activation-steering practice: an intermediate layer at roughly half to two-thirds of the network depth (layers 40, 30, and 15 for Qwen3.6-27B, Gemma-4-31B-IT, and GPT-OSS-20B) and a single strength of α = 0.5 for all three targets. The selection rule has no threshold to tune: matching is one-to-one within each rollout pair, and the top half of matched pairs is kept.
This fixed configuration also makes the comparisons controlled. REVEAL and Base share the same layer and strength for each target, so they differ only in which success–failure contrasts enter the vector, and the gain over Base isolates the contribution of event selection. Because the setting is shared and not tuned for either method, the reported gains are a conservative estimate.
For an independent split, we rebuild the vector solely from the teacher trajectories of the 56 tasks outside the vector's construction set, recomputing all standardization statistics on these tasks and reusing the original layer, strength, and selection rule without any retuning (Table B). The rebuilt vector reaches 38.20% on all 89 tasks, 2.25 pp above Base and within 1.12 pp of the original vector. The gain thus holds with a different construction set and fixed configurations.