acceptodds
Under review as a conference paper at ICLR 2027

Probing System-Level Performance Representations in Code Models

Abstract

Can a code language model represent a system-level property more reliably than it reports that property in its output? We study capacity bottlenecks in rendered request handlers, where the correct stage is derived exactly from visit counts, per-visit costs, and the runtime sharing regime. Across six Qwen2.5-Coder scales from 0.5B to 32B on 1,200 instances, linear probes predict the bottleneck with 67.9–87.1% held-out accuracy, while surface-feature and shuffled-label controls remain near chance. A teacher-forced output read-out on the same held-out instances reaches 22.1–45.0%, leaving a 37.5–58.8 point gap. At 0.5B and 1.5B, the rank of the correct stage under the output logits is indistinguishable from chance; from 3B upward, the rank is significantly better than chance, indicating that the output distribution carries some bottleneck information even though it remains substantially less accurate than the probe. Layerwise profiles distinguish lexically explicit properties, which become decodable early, from composed properties, which emerge later; bottleneck decodability peaks around mid-depth and then decreases. The probe result also holds for DeepSeek-Coder 1.3B and StarCoder2 3B. Finally, steering with fitted probe directions is unstable across data splits: refitting changes the selected layer and reverses intervention effects. These results separate linear decodability, output accessibility, and intervention stability when probing system-level reasoning in code models.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, linear probing, code language models, representation analysis, system-level reasoning, activation steering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7PYjOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether language models represent properties of a system more reliably internally than in their output. Specifically, they study a setting with 1,200 Python request handlers that a model is given, and it is asked to find capacity bottlenecks. They find that a simple linear probe on the model's internal activations relatively reliably predicts the bottleneck, while when checking the model outputs, the model gets it right at a much lower rate. In more detail, they find that the middle layers predict this best. Furthermore, steering along the probe direction is not stable.

Critical strengths

The setup of the Python request handlers is well-designed to elicit whether a model internally represents a property that has to be deduced in multi-step reasoning (i.e., it has to be worked out based on multiple numbers the code specifies).

Critical weaknesses

Generally, I believe the paper does not offer significant new insights. Generating a new probing task, comparing probes across scales and families, and comparing internal decodability with the model's output are things that have been done relatively often in and I do not see how this work offers significant insight beyond that.

Questions for authors
  1. It seems to me like there is a potential issue that the probe is trained on labeled instances from the distribution we are testing on, while the direct output test is zero-shot. Do I understand this correctly? Is this an issue? Have you thought about testing few-shot prompting?
  2. Furthermore, the probe might just directly be doing work that the model struggles with internally. How do you view this?
2 comments
AuthorsOpenReview ↗

Response to Reviewer 7PYj

Thank you for the constructive review and questions.

Novelty. Most work showing that internal states know more than outputs studies targets the model may have stored: truthfulness, factual knowledge and hallucination signals (Burns et al., 2023; Kadavath et al., 2022; Liu et al., 2023; Orgad et al., 2025; Gekhman et al., 2025). The closest studies use numbers in the input: probes recover the sum of three-digit additions whether or not the answer is right (Sun et al., 2025) and rank two numerals better than the model says (Yuchi et al., 2026). All are now cited and discussed. Our target composes several printed quantities by an exact law, a product per stage and then a maximum over the stages that hold a server, and this allows three analyses that the revision now reports:

  • First, a set in which no input-local shortcut predicts the label. With realistic costs a rule that never reads a cost finds the bottleneck on 92% of instances; on cs no cost-free rule or linear model exceeds 48.3%, yet the 7B answer-token probe reaches 67.9% and randomly initialized networks at most 40.8%, so a probe above that level must use the printed costs.
  • Second, an answer read-out with the probe's supervision, and a gap that widens with scale. With the same token, forward pass and 720 labels, a logistic read-out of the answer logits reaches 38.3% at 7B, against 67.9%. It fits an offset per candidate, the correction with which concurrent work on logic verification (Villuri et al., 2026) removes most of the probe–answer gap and finds the remainder shrinking with scale. Here it stays at 35.8–38.8% at every Qwen scale while the probe improves, and the gap is 5.8 points at 0.5B and 29.6 at 7B. A probe on the final state hLh_L, which the unembedding reads, still reaches 64.6%. LoRA on the same labels raises the answer to 92.1%, so the answer format can express the label.
  • Third, which directions steer. Class-mean directions move the answer toward a chosen stage in eight of ten model–set pairs, more often than probe weight rows. This extends findings for truthfulness (Li et al., 2023b; Marks & Tegmark, 2024) to a multi-class target computed from the input, and fixing the layer shows that the instability of probe-based steering came largely from the reselected layer and the probe row.

Q1 (supervision of the two read-outs). Yes, the submission had this asymmetry, and the revision removes it in three ways (Tables 4 and 9): three-shot prompting with solved training instances gives 21.2% at 7B on cs, a logistic read-out of the answer logits trained on the same 720 labels as the probe gives 38.3%, and LoRA fine-tuning on those 720 instances gives 92.1%. The first two leave the model unchanged and stay far below the probe at the same token (67.9%). LoRA retrains the model, so it is not a read-out of the pretrained representation; it shows that the answer format can express the label, so the gap is not an artifact of the read-out. Below 7B, an adapter trained on orig reaches only 34.6–50.4% on cs, so fine-tuning, too, takes the cost-free shortcut when the training data allow it.

Q2 (whether the probe does the computation). Four controls address this. (1) The same probes on the same architectures with random weights reach 34.6–40.8% on cs at the answer token (Table 7), below the best cost-free baseline (48.3%), against 56.2% at 1.5B and 67.9% at 7B for the trained networks, and Figure 2b shows the random 7B network below that baseline at every layer; the trained network computes the composition. (2) Cost-free rules and linear models reach at most 48.3% on cs, and paired with the best of them on the same 240 instances the answer-token probe is ahead at 1.5B, 3B, 7B and 14B (exact McNemar test, one-sided p=0.02p = 0.02, 0.030.03, 2×10−72\times10^{-7} and 2×10−42\times10^{-4}), as is the probing-prompt probe of DeepSeek-Coder and StarCoder2 (p=0.04p = 0.04 and 0.0060.006). (3) The probe trained on shuffled labels reaches 16.2–20.4% across the Qwen scales (18.3% at 7B), against a majority class of 22.5%. (4) The bag-of-tokens probe, trained on the mean input embedding of the code tokens, sees every token but no contextual computation and reaches only 31.2% on cs. Finally, adding the class-mean direction at the answer token moves the answer toward the chosen stage in eight of ten model–set pairs, so this direction of the probed representation is causally connected to the answer.

We would be glad to answer further questions during the discussion.

AuthorsOpenReview ↗

Remaining questions on the new controls

Dear Reviewer 7PYj, if you have a moment before the discussion closes, we would be glad to clarify any of the new experiments: the read-outs of the answer with the probe's supervision, whose gap to the probe widens to 29.6 points at 7B (Table 4), or the random-weight, bag-of-tokens and cost-shuffled controls, which show that the trained network, not the probe, does the computation (Table 7, Figure 2b). Thank you again for your time.

Reviewer_ok1zOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Paper asks if code models internally represent a system-level property of code, or just process tokens. Property is: which stage of rendered request handler becomes capacity bottleneck when request rate grows. Labels computed from queueing law (visit count x per-visit cost), no human annotation, so exact. 1200 generated instances, linear logistic probes on hidden states, six Qwen2.5-Coder scales 0.5B to 32B.

Main results. Probes get 67.9-87.1% test accuracy, controls much lower (majority 30.4%, surface features 24.6%, shuffled labels around 20-27%). Lexical targets (sharing regime, unindexed query) decoded at ~100% in early layers. Composed bottleneck target peaks mid-network, goes down toward last layer. Output readout with teacher forcing reaches only 22.1-45.0%, gold rank at chance for 0.5B and 1.5B. Same picture on DeepSeek-Coder 1.3B and StarCoder2 3B. Steering with probe directions looks good on one split, flips after refit on other splits.

Overall, its a good paper

Critical strengths
  • OK design. Analytic labels, companion lexical targets under same renderer, shortcut check built in: largest per-visit cost correct on only 27.9% instances, authors measured it.
  • Controls are proper. Both near chance/majority, like they should be.
  • Output readout careful: teacher forcing, rank statistic, check that logit spread is not tied.
  • Honest negative steering result. People usually hide such things.
  • Statistics above what I normally see: binomial CIs, exact tests. Numbers look consistent.
Critical weaknesses
  • Everything synthetic. One renderer, one template. No paraphrase test, nothing on real repositories. This bounds conclusions.
  • Probe error (12-32 points) never analyzed. Appendix A hints it is near-tie instances, but no real error decomposition.
  • Only linear probes. Without MLP probe I cannot tell representation limit from readout limit.
  • Output accessibility tested with one prompt, one answer format.
  • Steering experiment is one setup (rank-one add, one scale, two alphas). Maybe instability comes from this choice, cannot say.
  • Minor: surface control below majority, no explanation why
Questions for authors
  1. How are ties in labels broken? What fraction of instances is near-ties, do probe errors sit there?
  2. Would small MLP probe beat linear one at selected layer?
  3. Steering with fixed class rows or averaged directions, still unstable? Is layer moving or direction?
  4. Why surface model below majority baseline?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ok1z (1/2)

Thank you for the detailed and constructive review. This reply covers W1 to W3; W4 to W6 follow in part 2/2.

W1 (renderings, paraphrases and realistic code). The revision varies both the code and the question. The code appears without comments, with costs as decorator arguments, and, for the cc instances, as TypeScript/Express routes with different identifiers, loop syntax, admission mechanism and layout. The question appears reworded, as lettered multiple choice, with three solved examples, and with chain of thought. Two new samplers, including cs, make the cost-free shortcuts fail; on cs no cost-free rule exceeds 48.3%.

On every rendering the answer-token probe stays far above every read-out of the model's answer (Table 5); at 7B it reaches 82.1, 84.2 and 58.3% without comments, with decorator costs and in TypeScript, against 15.8, 21.7 and 21.2% for the forced choice. In TypeScript the probe at the end of the probing prompt reaches 50.4–55.0% across the Qwen scales, against 56.2–58.8% for the Python rendering of the same instances. The shortcut audit and the cost-shuffled set need exact labels, which a generator provides; we release the generator, renderers and baselines so that the protocol can be run on repositories with measured bottlenecks.

W2 and Q1 (ties and near-ties). No test instance is a tie: the smallest relative gap (D(1)−D(2))/D(1)(D_{(1)} - D_{(2)})/D_{(1)} between the two largest demands on the test split is 0.18% on orig and 6.0% on cs. In exact rational arithmetic, each set of 1,200 contains one exact tie, in the training or validation split, and on orig the two-decimal rounding of the printed costs reorders the top two demands of one further training instance; no test number depends on either, and the appendix now states this. Probe errors concentrate on near-ties (Appendix A, Table 3). At 7B, the accuracy of the probe at the end of the probing prompt rises from 66.7% in the smallest-gap quartile to 93.3% in the largest on orig, and from 48.3% to 63.3% on cs. This is the pattern expected from demands estimated with finite precision: if every demand is within relative error ϵ\epsilon, the leader stays correctly ordered whenever the gap exceeds about 2ϵ2\epsilon.

W3 and Q2 (MLP probe). A nonlinear probe does not do better. A one-hidden-layer MLP probe (256 units, ℓ2\ell_2 penalty 10−310^{-3}, early stopping) at the selected layer at the end of the probing prompt reaches 77.9% on orig and 41.7% on cs at 7B, against 86.7% and 53.8% for the linear probe, and it is below the linear probe for every model on both orig and cs (Appendix A, Table 3). With 720 training instances and up to 5,120 features, a nonlinear read-out recovers nothing beyond the linear probe at this sample size, so the comparison with the answer uses the strongest read-out we fit.

Continued in part 2/2 below.

AuthorsOpenReview ↗

Response to Reviewer ok1z (2/2)

W4 (prompts and answer formats). We now read the frozen model's answer in six ways: the original question; a reworded question that asks for the bare number; a lettered multiple-choice version that lists each stage with its helper and loop count; a three-shot version with solved training instances; zero-shot chain of thought, in which the model is asked to work out each stage's per-request server time before answering and the reply is started with a step-by-step cue; and a logistic read-out of the answer logits trained on the probe's 720 labels. At 7B on cs these give 16.2, 18.3, 30.4, 21.2, 33.3 and 38.3%, against 67.9% for the probe at the same token (Table 4). None of them reaches the probe, on cs or on any of the other six sets (Table 5), while the same forced choice reaches 92.1% after LoRA fine-tuning on the 720 training instances (Table 9).

W5 and Q3 (steering: layer versus direction). Both move, and the revision separates them. Across refits the selected layer moves by up to 10 layers (Table 6, last column), so we steer both at one fixed layer per model and set and at each refit's own layer. With the layer fixed, the sign of the margin change is the same in every refit for 47 of the 60 combinations of pair, strength and estimator; when each refit picks its own layer, it is the same for 29. At the fixed layer, the class-mean direction estimated on different refits has mean pairwise cosine 0.97–0.99, the probe row only 0.64–0.83. On fixed rows versus averaged directions, we compare the probe's weight row for the target class, a discriminative boundary normal, with the class-mean difference μc−μ\mu_c-\mu, a class-level average. The class-mean direction beats both the unsteered rate and a random direction in eight of ten pairs, with a positive margin change in every refit in six, and it moves the answer more often than the probe row in eight of the ten pairs. The instability in the submission therefore came largely from moving the layer and from using the probe row as the direction. The result holds across strengths: at α=0.5\alpha = 0.5, 1 and 2, the class-mean direction beats both references in eight, eight and eight of the ten pairs (Appendix D, Table 8).

W6 and Q4 (the index-encoded surface control). The submitted surface control gave a linear model the index of the largest cost and of the largest visit count as integers. A multinomial logistic model cannot express "class kk when this integer equals kk" with one weight per feature, so it underfit and scored below the class prior. Encoding the kind and loop count of every stage one-hot fixes this (Appendix B) and makes the control much stronger, 91.2% on orig, which is why the composition claim rests on cs, where it reaches 48.3% and the answer-token probe exceeds it from 1.5B on.

We would be glad to answer further questions during the discussion.

AuthorsOpenReview ↗

Checking in before the discussion closes

Dear Reviewer ok1z, please let us know if anything in our two-part reply above remains unclear: the TypeScript rendering, the MLP probes and the near-tie breakdown (Table 3), the six read-outs of the answer (Tables 4 and 5), or the fixed-layer steering (Table 6). Thank you again for the detailed review.

Reviewer_kJiBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2(+1)
Read review
Paper summary

The paper analysis how well coding agents can reason about bottlenecks in python request handlers with 3-5 stages. It shows that the correct answer of the bottleneck can be extracted from intermediate layers of the LLM even if a direct readout from the LLM via teacher forcing (which works quite poorly) fails. The authors also analyze wether the response of the LLM can be altered via steering but come to the conclusion that this is unstable. Yet, the paper shows that such models might not answer correctly even though they know the answer internally for more complex code-level reasoning problems.

Critical strengths
  • The analysis contains interesting insights for the LLM for coding community. However, I am not an expert in the field to fully judge this.
Critical weaknesses
  • Writing is quite obscure, in particularly for a non-expert. An example of what a probe is and how the code examples look like would have helped tremendously (its somewhere hidden in the appendix, I would suggest to move that to the main paper)
  • All examples have the costs and visit counts in the comments, hence, the task might be much easier than system level reasoning. It would be good to ablate the results without these comments
  • The probe vs output comparison is not fair. The probe is learned using 700 examples while the output is zero shot. Would this result still hold if we would finetune the LLM with LORA?
  • The steering results are rather thin and only presented on one model (1.5B).
Questions for authors
  • How would the probe and output perform without the code comments?
  • Would LORA fine-tuning of the full model for the teacher forcing (can be dne only on a smaller subset of models) change the results?
4 comments
AuthorsOpenReview ↗

Response to Reviewer kJiB

Thank you for the careful review. We answer your four points in order.

W1 (example and definition of a probe in the main text). We moved both into the main paper. Figure 1(a) shows a rendered handler with its per-stage demands written next to each loop: stage 1 (20×2.6220 \times 2.62 ms, or 52.4 ms) is the bottleneck, although stage 3, the unindexed query, has the largest per-visit cost. Figure 1(b) shows the probe and the model's answer reading the same forward pass. Section 4 opens with a plain definition: a probe is a linear classifier that takes the hidden-state vector of one layer at one token and outputs a score for each stage, while the model's own answer is read from the logits of the stage tokens at the same token. Section 3 explains which resources hold a server and what each instance set tests.

W2 and Q1 (removing the comments). We ran the ablation in two forms. orig_nocomment deletes every comment, so no cost appears in the text, and the probe stays at 63.3–75.8% across the Qwen scales. orig_costcode deletes every comment and writes each cost as a decorator argument, and the probe stays at 64.6–82.5%. In both, the model's own answer stays below the 30.4% majority class (15.0–22.9% and 20.8–27.9%).

With realistic costs, the resource kind already predicts the bottleneck: a rule that never reads a cost is right on 92% of orig. We therefore built cs, where every server-holding resource draws its cost from one range, so no cost-free rule or linear model exceeds 48.3% and the costs must be read and multiplied by the loop counts. On cs the probe reaches 67.9% at 7B and 62.9% at 14B, while the model's answer is 16.2%, below the majority class, as it is without comments. The gap holds on every rendering (Table 5); without comments at 7B the answer-token probe reaches 82.1%, against 15.8% for the forced choice and 32.9% for the trained read-out.

W3 and Q2 (matching the supervision, and LoRA). We added both kinds of supervision-matched control. With the model unchanged, a logistic read-out trained on the answer logits with the same 720 labels, at the same token and forward pass as the probe, reaches 38.3% at 7B on cs, against 67.9% for the probe; three-shot prompting gives 21.2% (Table 4). A probe on the final state hLh_L, which the unembedding reads, still reaches 64.6%, and Figure 2b shows the bottleneck decodable at every layer from mid-depth to that state. The 720 labels therefore do not account for the gap.

With the model retrained, LoRA (rank 16 on all attention and MLP projections, epoch selected on validation) on Qwen 0.5B, 1.5B, 3B and 7B, and DeepSeek-Coder 1.3B, on orig and on cs, brings the same forced-choice read-out to 71.7–93.8% on cs and 92.1–95.4% on orig (Table 9). The answer format can thus express the label, and the low accuracy of the pretrained model's answer reflects what that model reports. Because LoRA also retrains the representations, we report it separately from the read-outs of the frozen model.

LoRA also bears on your comments question. Below 7B, an adapter trained on orig reaches only 34.6–50.4% on cs, the level of rules that read no cost, so fine-tuned models also need the cost-shuffled set. At 7B the orig-trained adapter reaches 69.2% on cs, in line with the larger lead of the frozen-state probe at 7B, and adapters trained on cs reach 84.6–95.8% on orig at every scale.

W4 (steering beyond 1.5B). Steering now covers Qwen 0.5B, 1.5B, 3B and 7B, and DeepSeek-Coder 1.3B, each on orig and cs, with three direction estimators (the probe's weight row, the class-mean difference μc−μ\mu_c-\mu and a random direction), three strengths, five refits on random 60/20/20 partitions, and both a fixed and a reselected layer (Table 6, Appendix D). With the layer fixed, adding the class-mean direction moves the answer to the target stage more often than both the unsteered model and a random direction in eight of the ten model–set pairs, and in six of them the target's margin rises in every refit. The probe's weight row is the weaker target: the class-mean direction moves the answer more often in eight of the ten pairs. The submission steered along this row at a reselected layer; at the fixed layer the probe row varies across refits (cosine 0.64–0.83) while the class-mean direction does not (0.97–0.99).

We would be glad to answer further questions during the discussion.

AuthorsOpenReview ↗

Follow-up on the revision

Dear Reviewer kJiB, before the author discussion closes, we would be glad to answer any follow-up question on the revision. It moves the handler example and the definition of a probe into Figure 1 and Section 4, adds the comment-free and cost-shuffled sets (Table 5), compares the probe with a read-out trained on the same 720 labels and with LoRA (Tables 4 and 9), and steers five models (Table 6). Many thanks again for your feedback.

I thank the authors for the thorough comment ablation and the cost-shuffled set, but to my understanding these experiments narrow what the paper shows: on the realistic renderings (orig, orig_nocomment), the probe (63–87%) stays below a cost-free logistic model on resource kind and loop count (91.2%), so the hidden states seem to beat cost-free baselines only on cs. On cs the task, as I understand it, reduces to multiplying two printed numbers per stage and taking the largest, which in my view supports a claim about the arithmetic composition of printed numbers rather than the ‘system-level performance representations’ of the title. The steering evidence appears to follow the same pattern: on cs the class-mean direction raises the target’s margin in every refit only at 0.5B and 1.5B, and at 7B it moves the answer to the target less often than no steering at all (16% vs. 27%, Table 6), so the causal evidence also seems to come mostly from the shortcut-solvable sets. I may be missing nuances given my limited familiarity with this literature, but for these reasons I keep my score, but raise my score for presentation and soundness.

AuthorsOpenReview ↗

Clarifying system-level semantics on cs, realistic priors, and 7B steering

Thank you for reading the revision closely and for raising the presentation and soundness scores. The paper contains the evidence on the three points you raise, and one further number computed from the released cs set makes the first point concrete.

(1) Why cs requires system-level semantics and not bare arithmetic. Most cs handlers (986 of 1,200) also call an outbound HTTP service, with its own loop count and printed per-visit cost, and that call waits for a remote service and holds only a request slot, not a server of limited capacity (Section 3). Multiplying the two printed numbers of every stage and taking the largest therefore finds the bottleneck on only 34.2% of the cs test instances, below the best cost-free baseline (48.3%): on 158 of the 240, the outbound call has the largest product. Excluding it gives 100%. To reach the label, a model has to tell from the code which helpers hold a database connection, a process-pool worker or a cache connection and which only wait on a remote service, read each visit count from a loop bound or a single call, and attach the printed cost. This is the operational analysis of a request handler (Denning & Buzen, 1978), not a comparison of printed products.

(2) How the realistic renderings and cs complement each other. The logistic model at 91.2% on orig is handed the parsed resource kind and loop count of every stage and learns from the 720 labels how costly each kind is; it is system-level performance knowledge supplied by a parser. The probe has to recover that knowledge from raw code: at the answer token it reaches 87.5% at 7B and 89.6% at 14B on orig (Table 5; the 14B interval, 85.1–92.8%, contains 91.2%), while the best read-out of the model's own answer is 38.3% and 44.2%. On orig_nocomment no cost appears anywhere in the input, and the 7B probe still reaches 82.1%, against 15.8% for the forced choice; that accuracy must come from what the model knows about the operations in the code, such as an index lookup, a table scan or a cache read. Together, orig_nocomment shows that the hidden states encode the typical cost of each operation, and cs shows that when profiled costs override those typical costs, the hidden states compose the printed costs with the loop counts over the stages that hold a server. Both are parts of the system-level performance representation that the title refers to.

(3) Steering on cs across scales and at 7B. At the fixed layer (Table 6, α=1\alpha = 1), the class-mean direction moves the answer to the target more often than both no steering and a random direction in four of the five models on cs: 42% against 28% and 32% at 0.5B, 33% against 20% and 24% at 1.5B, 35% against 26% and 32% at 3B, and 39% against 24% and 26% for DeepSeek-Coder 1.3B; at 3B with α=0.5\alpha = 0.5 it reaches 53% against 26%, with a positive margin change in 4 of 5 refits (Table 8). At 7B the fixed layer is the one selected on the first refit (layer 20), while the other refits select layers 20–26. When each refit steers at its own validation-selected layer, the class-mean direction reaches 36% at α=1\alpha = 1 and 39% at α=2\alpha = 2, above both no steering (27%) and a random direction (25% and 22%), with a positive margin change in 3 of 5 refits (Table 8, Appendix D). On cs, the class-mean direction therefore moves the answer toward the chosen stage in every model we steered.

If these clarifications address the points above, we would appreciate it if you would reconsider your overall rating, and thank you again for engaging closely with our response.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: shortcut-free sets, supervision-matched read-outs, steering, and re-extraction

We thank the reviewers for detailed and constructive reviews. The revision adds comment ablations, LoRA, MLP probes, more prompts and read-outs, chain of thought, steering on five models and a layer-by-layer profile of the probe. The updated supplementary material (all sets, scripts and per-instance outputs) regenerates every set, table and figure.

(1) Evaluation sets on which cost-free shortcuts fail (kJiB W2, ok1z W1, 7PYj Q2). Beyond kJiB's comment ablation, we tested rules and linear models that ignore the printed costs. With realistic costs, a rule that multiplies each stage's loop count by the typical cost of its kind finds the bottleneck on 92% of test instances (Table 1). We therefore added four sets. In orig_nocomment every comment is removed, so no cost appears in the text; in orig_costcode each cost is written in code as @profiled(ms=...). In the cost-shuffled set cs, 1,200 new instances, every server-holding resource draws its cost from one range, so no rule or linear model that ignores the printed costs exceeds 48.3%, while the same model given the parsed costs reaches 91.7%. cc_ts renders instances as TypeScript/Express routes with different identifiers and layout.

(2) The composition is in the hidden states (all reviewers). On cs, the probe at the answer token reaches 44.6% at 0.5B, 56.2% at 1.5B, 55.8% at 3B, 67.9% at 7B and 62.9% at 14B. A bag-of-tokens probe (every input token, no contextual computation) stays at 31.2%, and the same probe on randomly initialized networks at 34.6–40.8%. From 1.5B on, the hidden states encode the bottleneck above every cost-free baseline (paired exact McNemar test, p≤0.03p \le 0.03; all four significant after Holm correction), with the largest lead at 7B and 14B (19.6 and 14.6 points, against 7.9 and 7.5 at 1.5B and 3B; Figure 2).

(3) The comparison with the model's answer is supervision-matched, and the gap widens with scale (kJiB W3, 7PYj Q1). The probe and the model's answer are now read at the same token of the same forward pass. At 7B on cs:

read-outaccuracy (%)
probe at the selected layer67.9
probe on the final state hLh_L, which the unembedding reads64.6
forced choice, original question16.2
reworded question18.3
lettered multiple choice30.4
three-shot21.2
logit read-out trained on the probe's 720 labels38.3
reasoning before answering33.3

On all seven sets and at every scale, the probe exceeds every read-out of the frozen model's answer (Tables 4 and 5). The trained read-out fits an offset per candidate and so corrects any miscalibrated threshold; on cs it stays at 35.8–38.8% at every scale while the probe improves, and the gap is 5.8 points at 0.5B and 29.6 and 27.1 points at 7B and 14B. LoRA on the same 720 instances changes the weights and is reported separately (Table 9); with the same forced-choice read-out it reaches 71.7–93.8% on cs, so the answer format can express the label. Below 7B, an adapter trained on orig reaches only 34.6–50.4% on cs, the level of the cost-free rules, so fine-tuned models need cs as well; at 7B it transfers (69.2%).

(4) Steering at four Qwen scales and in DeepSeek-Coder (kJiB W4, ok1z W5/Q3). The revision holds the layer fixed, which separates a change of layer from a change of direction, and compares the probe's weight row with the class-mean direction μc−μ\mu_c-\mu and a random direction. The class-mean direction moves the answer toward the target stage more often than both the unsteered model and the random direction in eight of the ten model–set pairs at each of α=0.5\alpha = 0.5, 1 and 2, and in six of them the target's margin rises in every refit. It also moves the answer more often than the probe's weight row in eight of the ten pairs. With the layer fixed, the sign of the effect is the same in every refit for 47 of 60 combinations, against 29 when the layer moves (Table 6, Appendix D).

(5) Re-extraction. All activations were re-extracted with unpadded single-sequence forward passes (the submission used batched left padding). On orig, probe accuracy from 0.5B to 14B moved from 67.9/80.8/77.9/86.2/85.8% to 74.6/78.3/79.2/86.7/85.0%, and the forced-choice accuracies changed by at most 2.1 points. Every number in the revision comes from the new extraction; the 32B model will be re-extracted for the final version.

(6) Presentation (kJiB W1, ok1z). The label is defined exactly as the stage with the largest demand among the stages that hold a server; the runtime affects throughput and leaves the bottleneck unchanged, and we corrected the submitted wording that suggested otherwise. Figure 1 shows a rendered handler and the three read-outs, Section 4 defines a probe in plain terms, Section 3 says which resources hold a server and what each set tests, and the appendix explains the surface control and breaks down the probe's errors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.