- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper traces how price sensitivity forms during training, using the public checkpoints and data of OLMo 2 7B. It separately measures 1. numerical representation (i.e. linear probes on hidden states), 2. price comparison accuracy, and 3. price weighting (i.e. a conjoint-style hotel-choice task with shuffled prices). The authors report several findings. The paper reports that the comparison accuracy does not determine how strongly price influences choices. Also, a cheaper-option preference appears during pretraining, even before preference labels. Annealing and SFT lower the relative price weight, mainly by sharpening non-price choices, and DPO partly reverses this. Both the price response and the internal number scale are closer to logarithmic than linear. The weighting study is rerun on five other models and four DPO retraining runs, and the authors frame the full pipeline as a reusable audit template.
Critical strengths
- The motivation is timely as purchasing agents are being deployed and tracing agents' economic preference is a reasonable question.
- The author provide careful and transparent measurement design. Several design choices, such as separating price weight from overall decisiveness, can go beyond typical LLM preference studies
- The code release give the work some practical values. The authors also report limitations candidly, including invalid stage-1 measurements, checkpoint-to-checkpoint variance beyond bootstrap intervals, run-to-run variation, and dependence on normalization.
Critical weaknesses
- The evidence does not yet support the claims about the “origins” of the preference. The trajectory analysis covers one model lineage, one training run per phase, and one task domain (binary hotel choices in New York City). “Training stage” is also not a standardized notion across model families; stages such as mid-training, annealing and cool-down differ in content and schedule. The stage-wise conclusions may therefore not transfer beyond the OLMo 2 recipe. OLMo 2 is widely used, but a detailed analysis of a single model has limited implications for how price sensitivity forms in general.
- Early checkpoints have little question-answering ability. The early findings such as comparison preference of the early checkpoints, are therefore measured on log-odds between letters the model rarely produces. In my reading, they reflect a mix of emerging task-following ability and preference, which the design cannot separate.
- Neighboring checkpoints differ far more than task-bootstrap intervals predict. For example, two DPO retraining seeds on the same recipe give a plain-prompt increase of 1.45× and 2.84×, with non-overlapping intervals.
- Also, elicitation through first-token A/B log-odds mixes preference with the model’s general capability, such as instruction following, sensitivity to context. The strong dependence of the results on prompt format supports this concern. For example, the SFT→DPO step is 2.21× in the plain prompt, 1.07× in the chat template, 1.55× with a system message, and not detectable in the cue format.
- The findings rely on strong modeling assumption. The price weight relies on a specific discrete-choice specification which might not naturally satisfy, such as a log-price term, additive effects and no interactions with other attributes. Also, key conclusions also depend on the normalization chosen.
- Crucially, if the framework is meant to be general, it is not clear why it focuses on price rather than LLM choice criteria broadly. If price is the special focus, the paper should show what is specific to price.
- As I understand it, the comparison task is basically the accuracy when asked to pick the lower of two prices, while the weight of price in a multi-attribute choice is a trade-off. It is not surprising that the first does not determine the second, as first task is closer to math comparison. Therefore the claim that “a test of a capability does not certify a preference” seems close to expected and its novelty is limited.
[minor]
- The introduction section is very dense and previews nearly every result and number from the later sections. A shorter introduction focused on the question, the approach and two or three key findings would be easier to read.
- The paper’s framing is primarily economic and regulatory. And I have a small concern whether there would be a better fit for behavior analysis on economic. Emphasizing what it contributes to ICLR-relevant questions, such as training dynamics, interpretability of numerical representations, or how post-training changes model decisions, would strengthen its fit for this audience.
Questions for authors
- By the paper’s own validity rule (Appendix A.3), 11 of 15 stage-1 checkpoints would be excluded. How should the stage-1 results be interpreted, given that most stage-1 checkpoints fail your own validity rule?
- To what extent are the findings specific to price? If the analysis were rerun with, for example, distance as the focal attribute, or even some known bias such as gender bias, would the same stage-wise story hold?
- Given that the log-vs-linear gain in held-out MSE is modest (3.6–10.3%) and the five-bin model fits better after post-training, how strongly do you interpret the logarithmic shape?
- Regarding the untrained-network results, what does it mean concretely for a random-weight network to exhibit “full-range logarithmic spacing”? Different random initializations could behave differently, and with only two initializations it is unclear how stable this reference is. Because untrained networks put almost no probability on A/B, how meaningful are the behavioral metrics for them?
- Extending the weighting study to other domains (e.g., e-commerce products) and to more than two options would help establish generality.
4 comments
(2/3)
W4. Elicitation through letter log-odds. We agree that first-token log-odds mix preference with format compliance. Table A14 therefore scores the SFT→DPO step in four prompt formats. DPO raises the price weight in all four: 3.54-fold in the plain prompt, 1.91-fold in the chat template, 1.68-fold with the answer cue inside the chat template and 2.96-fold with a system message. It raises decisiveness in all four as well, by 1.60- to 1.91-fold. The price preference therefore strengthens at DPO in every format we scored. So do the preferences over the other attributes.
Relative price weight is the ratio of the two. It doubles in the plain prompt, where the price weight rises about twice as much as decisiveness. It barely moves in the chat template, where the two rise by about the same factor. The format changes how far the price weight outpaces the other attributes. The two formats also agree on where DPO ends: relative price weight is 0.258 in the plain prompt and 0.289 in the chat template (Table 2). The same holds across models. DPO raises the price weight in 19 of 20 steps and SFT and DPO raise decisiveness in all 24, while the change in the ratio ranges from 0.73× to 2.21×. This is why the paper reports the price weight, decisiveness and their ratio separately for each format, with slot-lock beside every number. It also asks audits to name the format and any system message (Section 7).
The scores are also read from tokens the model would produce. At every released stage the higher-scored letter holds more probability than all non-letter tokens combined on 98.6% to 100% of prompts, so greedy decoding would begin with that letter (App. D.2). Scoring generated recommendations on the same tasks is the natural next step and is named in the Limitations.
W5 and Q3. Modeling assumptions and the log shape. The concern is that the price weight depends on one regression form. Table A4 refits it under six specifications: the six-control fit, price-by-attribute interactions, hotel fixed effects, a price window of 100 to 600 dollars, a 5% trim and 71 controls. All six keep log price. The post-training contrasts are stable across them:
| Contrast (ratio of r) | Range across the six specifications |
|---|---|
| Base→SFT, plain | 0.29 to 0.38 |
| SFT→DPO, plain | 2.19 to 2.29 |
| SFT→DPO, chat | 1.06 to 1.09 |
The interactions do show that higher-starred hotels are less price-sensitive after post-training, which the additive fit averages over. One contrast is sensitive to the specification: the single step from 3.896T to +5B, within the price window. Stated as the comparison between phases in W3, the annealing result is robust. On normalization, the two post-training steps keep their direction under all four denominators we tried. Whether DPO ends below the pretraining levels does depend on the denominator (App. D.1).
On the log shape, we read the response as concave and far from linear, with the logarithm as an approximation. In behavior the gain over a linear fit is modest, as you note. A power-transformation fit, in which exponent 0 is logarithmic and 1 is linear, puts the exponent at −0.10 [−0.30, 0.05] at the end of stage 1 and between −0.40 and −0.80 after annealing and post-training (Table A5). The response is logarithmic at first and more concave than logarithmic later. The five-bin curve picks up that extra concavity. In the representation the evidence is stronger. The logarithmic relation beats the linear one in all 176 readout cells (median held-out R² 0.84 against 0.39). It does so again within equal digit counts (0.71 against 0.19; App. B.2). Section 3 now calls the logarithmic form an approximation.
Q4. Untrained networks. Random weights can preserve how a number is written, including its length. Length is a coarse logarithmic cue, because each extra digit means a tenfold larger value. A readout fitted on a random-weight network recovers that cue. This is all that "full-range logarithmic spacing" means there.
We therefore compare only numbers with the same digit count, such as 200 and 800, which length cannot tell apart. On those pairs the untrained readouts do worse than predicting the mean (R² −0.07 and −0.12). The trained model reaches 0.39 at 399B, the first checkpoint probed. It stays above the untrained values at every later checkpoint (0.65 to 0.74 across annealing and Base). Training therefore adds magnitude information that digit count cannot supply.
On stability, the two initializations differ by at most 0.04 in full-range R² and 0.08 in same-digit R². The paper now describes the reference as the two tested initializations (App. A.4).
You are right that the behavioral metrics of untrained networks do not measure behavior. These networks put less than 0.0001 of their probability on A or B. Their accuracy, w and r are therefore estimator floors. Section 3 now labels them as such.
(3/3)
W6 and Q2. What is specific to price. Your question is whether the stage-wise story is about price or about how any attribute comes to be weighted. We tested this by rerunning the trajectory with star rating as the focal attribute (App. C.6, Table A10). If the story were generic, star rating would follow the same path as price. It does not:
| Price weight w | Star-rating coefficient | |
|---|---|---|
| Detectable at 399B, the first checkpoint | No | Yes |
| Trend over stage 1 (Kendall τ) | Rises (+0.73) | Falls (−0.51) |
| Annealing | Tracked-run mean falls to 0.59× the late-stage-1 mean (Table A9) | Rises 4.2× by +5B and keeps rising |
| Base→SFT, plain | 1.26× | 2.5× |
| SFT→DPO, plain | 3.5× | 1.3× |
The price preference is absent at the first checkpoint, builds over stage 1 and falls across annealing. The star-rating preference is present from the first checkpoint, weakens over stage 1 and strengthens across annealing. The two pretraining findings, a preference that builds over stage 1 and declines across annealing, are therefore specific to price among the attributes we can test. Post-training sharpening is shared. SFT and DPO raise every attribute coefficient, SFT the quality attributes most (2.5× to 8.2×) and DPO in the plain prompt price most. The comparison is descriptive, because star rating is listed rather than shuffled and the early checkpoints fail the validity rule.
On distance, your example, we have a smaller test. A separate instrument varies seven quantities one at a time at six checkpoints (App. D.4). Price has the largest normalized coefficient at every released stage. Distance is the only other quantity with a consistent sign, favoring the closer hotel (−0.65 at 3.775T, −0.27 at Base, −0.12 at SFT, −0.16 at DPO; App. C.6). Social biases such as gender are outside this instrument.
Why price? It is the attribute a purchasing agent trades against everything else. It is also the one where the capability (comparing numbers) and the preference (favoring the cheaper option) can be measured separately on the same checkpoints. The method itself, a randomized attribute with a decisiveness denominator, applies to any attribute.
W7. Novelty. We agree that comparing two prices and trading price against quality are different constructs and that saying so is not a finding. The finding is empirical. It runs against the simple expectation that a model which compares prices better also weighs price more.
We tested that expectation in two steps. First, 18 pretraining checkpoints all pick the lower price at least 97% of the time. Their relative price weight still ranges from 0.09 to 0.87. Equal accuracy therefore comes with very different weights on price. Second, accuracy at this level cannot rank the checkpoints, so we use a finer measure of the same skill. The comparison slope records how strongly the answer responds to the size of the price gap. Unlike accuracy, it does not saturate. If better comparison meant more weight on price, the checkpoints with a steeper slope would have a higher relative price weight. Across the 18 checkpoints they tend to have a lower one (Spearman −0.64). The slope instead follows decisiveness, the strength of the model's choices on the other attributes (0.69; Section 4). Checkpoint noise of the size we measured does not produce this reversal. If one shared factor drove both measures, the correlation would still be positive, about 0.57 (App. C.4).
The practical point is that an evaluation can show that a released agent compares prices correctly and still say nothing about the weight it puts on price. The paper supplies the measurement that does. The dissociation stays as a contribution, now stated as the measured result above. The other contributions are measured results as well. A lower-price preference is measurable on valid pretraining checkpoints, before any preference labels. SFT lowers the relative price weight by raising decisiveness. DPO raises the price weight in every format. The size of the relative change depends on the format and the run (Sections 4 and 5).
Q5. Other domains, more than two options. Both remain limitations. The Limitations now name other products, wordings and larger choice sets as future work.
Minor. Presentation and fit. The introduction is rewritten around the training-dynamics question, with almost every number moved out. The market motivation is one paragraph. The regulatory quote moves to the Discussion.
We would welcome further questions.
I thank the authors for the detailed response and the revisions. I appreciate applying the validity rule throughout the trajectory, retesting against checkpoint noise, and moving the cross-model results into the main text, which I believe add more clarity and reliability to the contribution. These changes address my concerns about checkpoint noise and the robustness of the annealing result. However, the following concerns remain.
- The claims may still go beyond what the valid evidence supports. I appreciate that the revised paper now acknowledges this directly. However, the checkpoints that pass the validity rule all lie at the very end of stage 1. As I read it, they show that a preference exists by the end of pretraining, but not that it develops during pretraining, since the rising trend relies mostly on checkpoints that fail the rule. Separation from the untrained band also seems to hold only after normalization: on the raw price weight, only 3 of the 15 stage-1 checkpoints lie above the band (Table A8). Finally, cross-model evidence for a pre-post-training preference comes only from OLMo-family base models. I suggest aligning the title, abstract and Section 4 heading with this narrower evidence.
- The comparison–weighting dissociation may partly reflect a phase transition. I appreciate that the authors moved from a conceptual argument to an empirical test, however I am not yet fully convinced by the evidence. Within stage 1, the comparison slope and relative price weight move together. The overall negative correlation therefore appears to come mainly from the transition into annealing. The slope rise appears in only one of the two annealing runs, it coincides with a change in answer format (App. C.4), and some of the included checkpoints fail the validity rule. Could the authors report this relationship within phases or on valid checkpoints only, and clarify whether the one-factor simulation accounts for this phase structure?
- I thank the authors for clarifying that the logarithmic form is an approximation. The new power fits, however, seem to raise a related question. After annealing and post-training, the behavioral exponent is −0.40 to −0.80, while the internal readout exponent stays near 0.04. If I understand correctly, the abstract’s statement that the internal scale has “the same shape the price response takes” would then hold mainly at the end of stage 1. It may help to qualify this statement or explain its basis.
- The headline post-training results depend on format. The response shows that the raw price weight rises at DPO in all four formats. However, the paper’s main metric, relative price weight, changes for example by 1.07× in chat template, 1.55× in system message)and 2.21× in plain prompt. The abstract’s “partly reverses” appears to rest mainly on the plain prompt. That is not the format the model was trained on, and the post-trained checkpoints are slot-locked in it. With two retraining seeds per arm giving 1.45× and 2.84×, the size of the DPO step remains uncertain. I suggest presenting the DPO result in the abstract and conclusion as format- and run-dependent, rather than as a stage-level finding.
(1/3)
Thank you for the thoughtful review. Your comments on early-checkpoint validity, checkpoint noise and generality led to three changes. The validity rule is now applied to the pretraining checkpoints. The annealing decline is tested across groups of checkpoints. The cross-model results move into the main text as Table 3.
W1. One lineage, one run per phase, one domain. We agree that a single trajectory cannot show how price sensitivity forms in general. The paper no longer claims that. "Origin" now means where in training the preference becomes measurable and which later stages change it. The paper suggests sources but does not isolate causes.
The cross-model results, previously an appendix table, are now Table 3 in Section 5. Because training stages are not standardized across families, Table 3 compares each model's own released SFT and DPO checkpoints. Three patterns recur across five models and four retraining runs. A lower-price preference is present before post-training in all three base models that pass the validity rule. SFT and DPO raise decisiveness in all 24 steps. DPO raises the raw price weight in 19 of 20. What varies is how much the relative price weight changes at DPO, because decisiveness rises alongside the price weight. The step is 2.21× in OLMo 2 7B's plain prompt and 0.73× to 1.10× in the released chat-template pairs (W4). The trajectory remains a single lineage. We therefore claim its timing for OLMo 2 7B only and rest the cross-model claims on the three patterns above. On other domains, see Q5.
W2 and Q1. Early checkpoints and measurement validity. We agree that the submission placed too much weight on checkpoints that fail our own validity rule: 11 of the 15 stage-1 checkpoints fail at least one criterion. The rule is now applied throughout the trajectory (App. A.3). The failing checkpoints are shaded in Figure 3 and their scores are reported as descriptive.
The central finding survives, because it needs only that a lower-price preference is measurable before any preference labels. Its timing becomes less precise. On the four valid checkpoints, all between 3.880T and 3.896T, the price weight is positive with its interval above zero. Relative price weight there is 0.60 to 0.92, well above the untrained band (upper edge 0.13). We keep that conclusion and move the onset from 3.146T to "by 3.880T". The checkpoints that dated it earlier compare prices accurately but put too little probability on the answer letters.
Two further checks show that the stage-1 scores respond to the displayed price rather than to answer position or to the regression specification. First, refitting on the half of tasks with less position bias changes the estimates at the nine checkpoints from 3.146T onward by at most 11% (Table A7). Second, a measure that uses no regression agrees. It counts how often the model's choice reverses when the shuffle reverses which option is cheaper, net of how often it reverses when the cheaper option stays the same. It tracks relative price weight across stage 1 (Spearman 0.95; App. C.1). Neither check separates preference from emerging task-following. Your reading that early scores mix the two is now stated in Section 4.
W3. Checkpoint noise. You are right. Our intervals capture one source of uncertainty, the sample of 600 tasks. Appendix A.4 now quantifies what they miss. Within a training phase, relative price weight varies across checkpoints 2.7 to 3.4 times as much as task sampling predicts.
The annealing result in the submission rested on a single step between neighboring checkpoints, so we retested it as a comparison between groups of checkpoints (Table A9). Against the average of the last five stage-1 checkpoints, all 11 annealing states have a lower relative price weight in every bootstrap draw. The highest annealing value (0.40) is below the lowest late-stage-1 value (0.60), a gap of three to five between-checkpoint standard deviations. Averaged over the tracked annealing run, relative price weight is about a third of its late-stage-1 level (0.32 [0.25, 0.40]), because decisiveness rises 1.77-fold while the price weight itself falls to 0.59 times its level. The result is the same against the four late checkpoints that pass the validity rule (0.34 [0.27, 0.42]; App. C.5) and under the price-window and trimmed fits (0.40 and 0.35; App. B.3). A gap this large and this consistent is well beyond the checkpoint noise we measured.
On the two DPO seeds (1.45× and 2.84×), we agree that one run cannot fix the size of the DPO effect. Section 5 now reports both. The direction recurs. All four retrained runs raise the relative price weight in the plain prompt. DPO raises the price weight in 19 of the 20 steps we measured.